docs(decisions): Crafty sign-in handoff — the gate was fine, the local login was the wall

Records what refused (Crafty's own 401, not oauth2-proxy), that the existing
bootstrap credential is still live (users.last_update == users.created), the
separate cookie-name collision fixed alongside, the traps measured rather
than assumed, the default.json seeding approach considered and rejected with
the entrypoint `ls -A` landmine that kills it, and — plainly — that no browser
round-trip was performed and the gate was rc 3, not green.

Written as its OWN FILE under decisions/ rather than appended to the tail of
DECISIONS-LOG.md. The single-file append is what made this PR (#22) and three
others unmergeable on 2026-09-29: every branch collided on the same last hunk,
with DECISIONS-LOG.md the only conflicting path in all four. The one-file-per-
entry layout and its ci/decisions-log.sh guard land in #24; this entry is
already written in that format, and validates against that checker (rc 0,
"1 entry well-formed"). This branch now touches DECISIONS-LOG.md not at all.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

decisions/2026-09-29-conformance-trigger-kubeconfig.md

@@ -1,125 +0,0 @@
----
-date: 2026-09-29
-branch: feat/conformance-trigger-kubeconfig
-title: The trigger credential script — it REFUSES, on purpose, and it is UNEXECUTED
----
-
-## What changed
-
-Delivered `infra/gitea/conformance-trigger-kubeconfig.sh` —
-`docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 6 — plus its section in
-`infra/gitea/README.md`. **The script is UNEXECUTED.** Minting a cluster credential and
-changing Gitea variables are operator actions; nothing was minted, no Gitea object was
-written, no `kubectl` mutation was made. Everything below is a claim about the *script*,
-not about the platform.
-
-*What it does.* Mints a token for `conformance/conformance-trigger` (TokenRequest), wraps
-**only that token** with the cluster CA and server into a kubeconfig, stores it as the
-Actions secret `CONFORMANCE_TRIGGER_KUBECONFIG` (legal name — **not** the reserved `GITEA_`
-prefix), and sets `vars.CLUSTER_AVAILABLE` only as a separately-requested,
-precondition-gated, TTY-confirmed final step. The token is never printed and never reaches
-argv: it goes into a 0600 file and into the PUT **body file**. The script refuses to run
-under `set -x`.
-
-*The design decision worth reviewing: TWO precondition classes.*
-
-- **mint-blocking (cluster-side):** SA/Role/RoleBinding present, and ClusterPolicy
-  `conformance-trigger-job-shape` present + `Enforce` + `failurePolicy: Fail` + Ready. If any
-  fails, **nothing is minted**. Rationale: Kubernetes puts no `use` check on a Job's
-  `serviceAccountName`, so a `create jobs` token in that namespace IS the privileged
-  `conformance-runner` identity unless the admission policy bounds it. RBAC cannot fix that
-  (plan §4); minting before the boundary is live would hand CI the escalation the boundary
-  exists to close.
-- **arm-blocking (repo-side):** digest-pinned `CONFORMANCE_RUNNER_IMAGE`, an init-container
-  plus the admitted image in `chainsaw-job.yaml`, and `CONFORMANCE_TRIGGER_KUBECONFIG`
-  referenced in `ci/conformance.sh` **code rather than a comment**. These block only the
-  variable, because storing the secret early is harmless and is the plan's own order.
-
-**Measured in the tree at the time of writing: all three arm-blocking checks FAIL** (no
-`CONFORMANCE_RUNNER_IMAGE` in `conformance/versions.env`; `chainsaw-job.yaml` is still the
-skeleton with an empty `workspace` emptyDir and the stock chainsaw image; `ci/conformance.sh`
-names the trigger kubeconfig only in its TODO comment). So on a live run today the script
-would provision the secret and then **refuse to arm `CLUSTER_AVAILABLE`**, naming each
-missing item — which is the intended behaviour, not a defect. §3 items 1, 2 and 4 are the
-work that unblocks it.
-
-*Both writes are verified the way `set-ci-token.sh` learned to.* `curl -sS` exits 0 on an
-HTTP error, so status alone proves nothing: each write checks `%{http_code}` **and** reads
-the object back. For the secret that is presence-by-name only (Gitea never returns secret
-values) and the script says so rather than implying the content was checked; for the variable
-it asserts the value reads back as exactly `true`.
-
-*Noted, not fixed at the time (since fixed on this branch).* `infra/gitea/README.md`'s
-detector table still named `GITEA_SETTINGS_TOKEN` — the illegal name PR #15 replaced with
-`SETTINGS_API_TOKEN`. The second commit on this branch corrects the README's table and its
-closing "remaining step" line, and records what `ci/gitea-settings.sh` actually does
-(`TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"` — the old name survives only as a
-local-run environment variable, never as CI wiring, because the secret that would feed it
-cannot be created at all). Documentation only, no behaviour change. Still NOT fixed, because
-it is outside this PR's diff: `ci/gitea-settings.sh:45` instructs the reader to create a repo
-secret `GITEA_SETTINGS_TOKEN`, which is impossible.
-
-*Rework, 2026-09-29.* This branch originally appended ~99 lines to the tail of
-`DECISIONS-LOG.md`. That is the hunk every concurrent branch collided on and the sole reason
-PR #19 was `mergeable: false`. The branch was rebuilt on today's `main` with the
-`DECISIONS-LOG.md` change dropped entirely, and this file is that handoff moved into
-`decisions/`. The changes to `infra/gitea/**` are byte-identical to the original three
-commits; the commit that contained only the log append was dropped and its substance folded
-in below.
-
-## What I verified
-
-*Of the script, before it stores anything:* it PROVES the credential narrow — `auth whoami`
-must return `system:serviceaccount:conformance:conformance-trigger`; a real `get jobs`
-round-trip must succeed; then an `auth can-i` matrix asserts BOTH halves — granted
-(`create/get jobs`, `get pods`, `get pods/log`) and **denied** (`delete jobs`, `create
-pods/exec`, `get secrets`, `create jobs` in `default`, `get nodes`,
-`'*' '*' --all-namespaces`). A mismatch aborts before anything is stored, and the message
-says explicitly **not** to widen `conformance/incluster/rbac.yaml` to make it pass. That is a
-property of the code as read, not of an execution.
-
-*Of PR #19's CI, measured rather than assumed — and it contradicts the brief.* The brief for
-this work stated this PR's CI would FAIL `approval-policy` because `infra/gitea/**` is
-security-relevant. Measured on the PR head commit's `/statuses`:
-
-    gate / gate (pull_request)          success   Successful in 1m34s
-    gate / conformance (pull_request)   pending   Blocked by required conditions
-
-`.gitea/workflows/` on `main` contains only `gate.yml` and the dormant `merge-gate.yaml`.
-**No path-scoped approval policy exists**, so nothing machine-enforces "security-relevant
-paths need a human"; that rule rests entirely on `required_approvals: 1` plus a reviewer
-actually reading the diff. Another instance of the CLAUDE.md §2b failure mode — a control
-believed to exist because a brief said so. If the policy is intended, it is missing and needs
-its own change. And per this repo's own rule, that green `gate` tick is **not** evidence this
-script works: the gate never executes `infra/gitea/**`, and the L2/L3 stage skipped.
-
-*Gate on the reworked branch (pinned toolbox, from the agent worktree, `DEV_KUBECONFIG=/dev/null`
-plus the `ENGINE` bind-mount shim per CLAUDE.md §3):* `MERGE GATE: UNDETERMINED`, **rc 3**.
-Diff base `origin/main @ c59fe9f923c4` ... this branch's head; touched requirement resolved as
-`R-ENV-01`. `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden` and `l1` all
-green; the gate decision names exactly one stage that did not run — `gitea-settings`
-(`SETTINGS_API_TOKEN` is a credential this agent does not hold, unwired repo-wide). The
-`conformance` stage also reported SKIPPED (`CLUSTER_AVAILABLE!=true`, Phase 0). The new
-`decisions-log` stage — merged to `main` as PR #24 after this rework began — validated this very
-file: "archive frozen, 2 entries well-formed", with `DECISIONS-LOG.md` unchanged and matching
-`decisions/ARCHIVE.sha256`. rc 3 is UNDETERMINED, not green, and is not reported as green here.
-
-## What I did NOT verify
-
-**The script has never executed.** Not one line of it has run against Gitea or against the
-cluster. Only `bash -n` was run (clean); `shellcheck` is not on the host and there is no
-shellcheck stage in the gate.
-
-- The Gitea variables API shape (POST-creates / PUT-updates, body field `value` not `data`)
-  is from **documentation, not measurement**. The script tries POST then falls back to PUT,
-  prints the response body on failure, and treats the read-back as the authority.
-- The Kyverno readiness field (`.status.ready` vs. a Ready condition) is handled both ways
-  and treats "neither present" as a block rather than assuming ready — but which form this
-  cluster actually reports is **unmeasured**.
-- Whether the kubeconfig's embedded server URL is reachable from the runner pod **cannot be
-  known from here** — hence `--server`, and a hard refusal on a loopback address.
-- No credential was minted, no Actions secret was created, `vars.CLUSTER_AVAILABLE` was not
-  set, and no `can-i` matrix was ever run against a real token. The narrowness argument above
-  is read from the RBAC and the script, not observed.
-- The gate proves only its L0/L1 offline subset. L2/L3 conformance did not run, so nothing
-  here is evidence about runtime behaviour.

decisions/2026-09-29-crafty-oauth2-proxy-signin.md

@@ -0,0 +1,123 @@
+---
+date: 2026-09-29
+branch: fix/crafty-oauth2-proxy-signin
+title: Crafty "won't sign in after Keycloak" — the gate was fine, Crafty's own login was the wall
+---
+
+## What changed
+
+Reported as *"crafty utilizing the keycloak oauth2-proxy does not sign in to the service after
+signing into keycloak; there is no way to set the service up without credentials."* The sidecar's
+own access log settles which layer refused, and it is not the one the report names:
+
+    GET /                            -> 302 to Keycloak
+    [AuthSuccess] ... groups:[platform-admins]        <- oauth2-proxy did its job
+    GET /oauth2/callback             -> 302
+    GET /panel/dashboard             -> 302
+    GET /login?next=/panel/dashboard -> 200           <- CRAFTY's own login page
+    POST /api/v2/auth/login/         -> 401           <- CRAFTY refused
+
+**The Keycloak gate was never broken.** Crafty 4 has no OIDC and cannot consume the sidecar's
+identity headers, so it presents its own local login to an already-authenticated user. That second
+factor is intended. What was not intended — and what made the service genuinely impossible to set
+up — is that **nothing documented where that password comes from**. Crafty generates it **once, on
+first boot only** (users table empty → `DatabaseBuilder.default_settings()`) into
+`/crafty/app/config/default-creds.txt`, mode `0600`, on the PVC's `config` subPath. Never in git; a
+fresh PVC generates a new one and the old is gone with the volume. `platform/crafty/README.md`
+asserted the double login was "expected, not a bug" and then left the reader with no way forward —
+the assertion was true and useless in the same sentence.
+
+So this branch:
+
+- Adds `runbooks/crafty-admin-credential.md` — tell the two auth layers apart from the sidecar log,
+  retrieve and rotate the bootstrap credential, and reset a lost one via argon2. Registered in the
+  runbooks `configMapGenerator`/`kustomization.yaml` and listed in `runbooks/README.md`, so the URL
+  actually resolves rather than the file merely existing.
+- Documents the credential's provenance and the two auth layers in `platform/crafty/README.md`.
+- **Fixes a second, real defect found while proving the first.** Both oauth2-proxy instances emitted
+  oauth2-proxy's **default** cookie names (`_oauth2_proxy`, `_oauth2_proxy_csrf`) scoped to
+  `.home.keithjackson.dev`, while holding **different** cookie-secrets. One cookie pair, two
+  proxies: whichever authenticated last clobbered the other's session; the loser logged `No valid
+  authentication in request`, bounced to Keycloak, and — SSO session still live — was redirected
+  straight back, clobbering it again. **That mutual eviction presents exactly as the reported
+  symptom**, intermittently, and would have been blamed on Keycloak indefinitely. Fixed on the
+  crafty side only (renaming the shared proxy's cookie would sign out every other gated app): a
+  unique `--cookie-name=_oauth2_proxy_crafty` plus a **host-scoped** `--cookie-domain`.
+  `--whitelist-domain` stays zone-wide — it governs post-login *redirect* targets, not cookie scope.
+
+**Considered and rejected: seeding `default.json` from a SOPS Secret.** Crafty does ship the hook —
+`find_default_password()` reads `app/config/default.json` (`username`, `password`, min 8 chars) and
+`delete_default_json` controls its removal — so an initContainer could make a rebuilt Crafty's admin
+password deterministic and git-managed. Rejected: the image entrypoint pulls its default config
+**only when `app/config` is empty** (`if [ ! "$(ls -A --ignore=.gitkeep ./app/config)" ]; then cp -r
+./app/config_original/* ./app/config/`). Writing `default.json` there pre-boot makes the directory
+non-empty and **suppresses the entire config bootstrap** — no `config.json`, no `credits.json`.
+Doing it safely means replicating the entrypoint's copy inside the initContainer, i.e. co-owning
+image-internal bootstrap logic that a 4.11 bump can change silently, to replace a one-command
+retrieval. Not worth it. If it is ever revisited, the `ls -A` guard is the landmine to design around.
+
+## What I verified
+
+- **Which layer refused**, from the sidecar's own access log: `[AuthSuccess] ...
+  groups:[platform-admins]` followed by Crafty's `/login` page and `POST /api/v2/auth/login/ → 401`.
+  oauth2-proxy admitted the user; Crafty rejected the local credential.
+- **The existing bootstrap credential is still usable**, so no DB surgery is needed to get in:
+  `users.last_update` **equals** `users.created` for `admin` (`09/05/2026, 08:10:04`) — the password
+  has never been changed — and admin's `last_login` is 09/14 from `127.0.0.1`, the API login that
+  created the "homelab" server. `default-creds.txt` is therefore live.
+- **The two cookie-secrets differ** — compared the two live Secrets (key presence/values compared
+  in-cluster, not printed), so neither proxy can decrypt the other's cookie; and both instances were
+  emitting the default cookie names on the shared `.home.keithjackson.dev` scope.
+- **The 3-strikes cooldown cannot fire behind this sidecar.** Crafty *records* failed attempts under
+  `get_remote_ip()` (honours `X-Forwarded-For` — the live tracker key is the chain
+  `"192.168.2.132, 10.0.0.123"`, with 1 attempt: the reported 401) but *checks* the cooldown under
+  `self.request.remote_ip`, always `127.0.0.1` here. The halves key differently, so
+  `get_recent_attempts()` always reads empty and `max_login_attempts: 3` never fires. Good
+  (**repeated guesses cannot lock the operator out**) and bad (that throttle is not a control we
+  have; the `platform-admins` gate is the only thing limiting who reaches the form). I had written
+  the opposite into the README from the config value alone before measuring it — the config key
+  existing is not the same claim as the code path running.
+- **`set_passwd` is unreachable in k8s**: it calls `getpass.getpass()` and needs a TTY; the container
+  runs `main.py -d -i`. Recovery must go through the DB.
+- **argon2 is installed only in `/crafty/.venv/bin/python`** — the system `python3` in that container
+  cannot import it, so the obvious reset one-liner fails.
+- **`dev.sh` does not forward stdin.** A `kubectl exec -i` through it hangs forever (provoked, not
+  assumed). So the runbook's reset generates its password **in-pod** into a `0600` file — Crafty's own
+  `default-creds.txt` idiom — rather than passing a secret through argv, stdin or shell history.
+- **Gate:** toolbox `./ci/gate.sh` — see the next section for the exit code. Green stages:
+  `spec-immutability`, `lint`, `l0`, `render-golden`, `l1`.
+
+## What I did NOT verify
+
+- **No browser round-trip.** The sign-in fix is **UNVERIFIED end-to-end**. The cookie change is
+  reasoned from measured facts (default cookie names on both, shared domain, differing secrets) and
+  renders correctly, but nobody has signed in through it. The human test is in
+  `platform/crafty/README.md`.
+- **Nothing was applied.** No `kubectl apply/edit/patch`, no `flux reconcile`. The running pod is
+  still the pre-fix spec (`crafty-64f7b6d78-pkpgp`, 2/2, 15d, 0 restarts — unchanged by this
+  session's read-only probes). **Merging is what deploys, and it will restart the pod** —
+  `strategy: Recreate` on an RWO volume, so the "homelab" Minecraft server stops and restarts. Pick
+  the moment.
+- **Gate rc 3, not 0.** Green: `spec-immutability`, `lint`, `l0`, `render-golden`, `l1`. **Skipped:**
+  `gitea-settings` (no `SETTINGS_API_TOKEN`) and **L2/L3 conformance** (`CLUSTER_AVAILABLE` unset) —
+  touched requirements `R-DNS-01 R-NET-01 R-OBS-04 R-PLAT-02 R-STATE-01` are **not certified**.
+  Nothing failed; nothing runtime was proven either. **rc 3 is UNDETERMINED, not green.**
+- **The admin password was not rotated**, and the bootstrap credential is still the standing admin
+  credential. The runbook says to rotate on first sign-in; that is an operator action and is still
+  open.
+- **The cookie collision's symptom was never reproduced in a browser.** This is the part of the
+  change with no observed failure to point at — the sidecar trace above shows a *working* session,
+  because that browser had not visited another gated app first. It is a latent-bug fix.
+
+### Follow-ups
+
+1. **Sign in and rotate** — `runbooks/crafty-admin-credential.md`, then confirm `last_update` moves
+   off `created`. Until then the panel's admin password is a value generated on 2026-09-05 that has
+   sat unchanged and undocumented.
+2. **Decide whether the inert brute-force cooldown matters.** Fixing it is upstream (Crafty keys the
+   two halves differently); the alternative is to accept the `platform-admins` gate as the only
+   throttle and say so.
+3. **Any further per-app reverse-proxy sidecar needs a unique `--cookie-name` too.** The collision is
+   a property of the pattern, not of Crafty. Worth an L0 check asserting that every oauth2-proxy
+   instance in the tree declares a distinct cookie name — the failure is silent and intermittent,
+   which is exactly the shape this repo keeps getting burned by.

infra/gitea/README.md

@@ -102,19 +102,7 @@ Behaviour, proven by test rather than assumed:
 |---|---|---|
 | live matches the declaration | 0 | green |
 | any key drifted, or no runner online | 1 | **blocks the merge** |
-| no `SETTINGS_API_TOKEN`, or Gitea unreachable | 3 | skips **loudly**; never a pass, never a block |
-
-> **The secret is `SETTINGS_API_TOKEN`.** An earlier draft of this table named
-> `GITEA_SETTINGS_TOKEN`, which **cannot exist as a Gitea secret at all** — Gitea reserves the
-> `GITEA_` prefix and returns HTTP 400 `invalid secret name`. That is the whole 2026-09-29
-> inert-detector story, so the wrong name surviving here is worth correcting rather than
-> reading past. Measured in `ci/gitea-settings.sh`:
->
->     TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
->
-> so `GITEA_SETTINGS_TOKEN` is still honoured — but **only as an environment variable for a
-> LOCAL run**. It can never be the CI wiring, because the secret that would feed it cannot be
-> created. In CI, only `SETTINGS_API_TOKEN` exists.
+| no `GITEA_SETTINGS_TOKEN`, or Gitea unreachable | 3 | skips **loudly**; never a pass, never a block |
 
 The skip is deliberately *not* a block: an unwired credential blocking every PR is the
 lock-yourself-out failure this repo keeps hitting. Wire the token with `set-ci-token.sh`, confirm
@@ -124,59 +112,6 @@ the stage goes **green on a real PR**, and only then set `REQUIRE_GITEA_SETTINGS
 **Changing the declaration is how you change policy.** If the live state is what you now want,
 edit this JSON in the same commit — deliberately, in review — rather than letting the two drift.
 
-## `conformance-trigger-kubeconfig.sh` — the narrow L2/L3 trigger credential
-
-Implements `docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 6 (= §5 step 7): mint a
-ServiceAccount token for `conformance/conformance-trigger`, wrap **only that token** in a
-kubeconfig, store it as the Actions secret `CONFORMANCE_TRIGGER_KUBECONFIG`, and — separately,
-and only on request — set `vars.CLUSTER_AVAILABLE=true`.
-
-    ./infra/gitea/conformance-trigger-kubeconfig.sh --dry-run   # measure only; writes nothing
-    ./infra/gitea/conformance-trigger-kubeconfig.sh             # mint + store the secret
-    ./infra/gitea/conformance-trigger-kubeconfig.sh --arm-cluster-available   # + the variable
-
-Never a cluster-admin kubeconfig, never a copy of the operator's own. The token is never
-printed: it goes from the TokenRequest API into a 0600 file, into the request **body file**
-(`--data @file`, so not even into `ps`), and nowhere else. The script refuses to run under
-`set -x`.
-
-**Its preconditions are split in two, and the split is the point.**
-
-| Class | Checks | Blocks |
-|---|---|---|
-| **mint-blocking** (cluster) | SA/Role/RoleBinding exist; ClusterPolicy `conformance-trigger-job-shape` is present, `Enforce`, `failurePolicy: Fail`, Ready | the **mint** — nothing is created |
-| **arm-blocking** (repo) | digest-pinned `CONFORMANCE_RUNNER_IMAGE`; `chainsaw-job.yaml` has an init-container and the admitted image; `ci/conformance.sh` names the trigger kubeconfig in **code, not a comment** | only `CLUSTER_AVAILABLE` |
-
-Why mint-blocking: Kubernetes puts **no `use` check on a Job's `serviceAccountName`**, so a
-token that can `create jobs` in `conformance` can run arbitrary code as the privileged
-`conformance-runner` SA. RBAC cannot close that; the Kyverno policy is the boundary. Minting
-while it is absent or in `Audit` hands CI the privileged identity, so the script will not do it.
-
-Why arm-blocking is separate: storing the secret before the runner image / Job template / CI
-trigger flow exist is harmless (the job is gated off), and is the plan's intended order. So
-running this today provisions the secret and then **refuses to arm the variable**, naming the
-missing item.
-
-**`CLUSTER_AVAILABLE` is not a "turn conformance on" switch.** Measured 2026-09-29: set with no
-trigger path, `run.sh` scaffold-skipped all 54 tests at rc 0 and `ci/conformance.sh` printed
-"L2/L3 conformance passed" — a fake green on an enforcing required check. `ci/conformance.sh`
-now blocks that case, so arming early does not enable L2/L3; it only breaks every PR. Hence:
-`--arm-cluster-available` plus all preconditions green plus a typed confirmation on a TTY, or
-nothing happens.
-
-Both writes are verified the way `set-ci-token.sh` learned to: check `%{http_code}` **and**
-read the object back. For the secret that proves presence by name only (Gitea never returns
-secret values); for the variable it proves the value is exactly `true`, which is what
-`gate.yml` compares against.
-
-The token **expires** — it is a TokenRequest token, and the script decodes the JWT `exp` claim
-to report the real expiry rather than trusting `--duration` (the API server caps it). Re-run to
-rotate. After expiry the trigger gets 401 and the stage fails loudly rather than skipping.
-
-`--server` exists because the embedded API-server URL must be reachable **from the CI runner
-pod** — the VIP — which is not necessarily the address your workstation uses. The script
-refuses a loopback server outright.
-
 ### Why these scripts exist at all, and what is still missing
 
 Gitea's repo settings live in **its own database**, not in anything Flux reconciles or the
@@ -189,9 +124,8 @@ These scripts encode the desired state and can re-assert it; `ci/gitea-settings.
 **detector** that turns divergence from a silent, invisible failure into a loud one. It closes the
 root cause of the 2026-09-29 session.
 
-It is only as good as its wiring: **until `SETTINGS_API_TOKEN` exists it skips**, and a skipped
-detector detects nothing. (That secret was stored and verified in PR #15; this sentence predates
-it. The name matters — see the note under the behaviour table above.)
+It is only as good as its wiring: **until `GITEA_SETTINGS_TOKEN` exists it skips**, and a skipped
+detector detects nothing. That is the remaining step.
 
 ## Historical sequence (kept for context)
 

infra/gitea/conformance-trigger-kubeconfig.sh

@@ -1,575 +0,0 @@
-#!/usr/bin/env bash
-# infra/gitea/conformance-trigger-kubeconfig.sh
-#
-# OPERATOR script for docs/plans/l2-l3-conformance-ci-wiring.md §3 item 6 (= §5 step 7):
-# provision the NARROW CI trigger credential for L2/L3 conformance, and — only when the
-# rest of the trigger path actually exists — arm `vars.CLUSTER_AVAILABLE`.
-#
-# What it does, in order:
-#   1. measures the preconditions (cluster-side AND repo-side) instead of assuming them
-#   2. mints a bounded-lifetime ServiceAccount token for `conformance-trigger` in ns
-#      `conformance` (kubectl create token -> TokenRequest API)
-#   3. assembles a kubeconfig containing ONLY that token + the cluster CA + the API server
-#   4. PROVES the kubeconfig is the narrow identity (auth whoami + an allow/deny matrix)
-#   5. stores it as the Gitea Actions repo secret CONFORMANCE_TRIGGER_KUBECONFIG, and
-#      VERIFIES the store by reading the secret list back
-#   6. sets vars.CLUSTER_AVAILABLE=true ONLY under --arm-cluster-available, only with every
-#      precondition green, and only after a typed confirmation. Otherwise it REFUSES.
-#
-# NOTHING SECRET IS EVER PRINTED. The token exists in a shell variable and in a 0600 file
-# under a 0700 temp dir that is removed on exit; it reaches the Gitea API in a request-body
-# FILE (`--data @file`), never in argv, never on stdout. This script prints names, byte
-# lengths, HTTP status codes, the token's expiry timestamp, and yes/no authorization answers.
-#
-# ----------------------------------------------------------------------------------------
-# THREE HARD-WON LESSONS THIS SCRIPT IS BUILT AROUND
-# ----------------------------------------------------------------------------------------
-# 1. GITEA RESERVES THE `GITEA_` PREFIX FOR SECRET NAMES.
-#    `PUT .../actions/secrets/GITEA_ANYTHING` returns HTTP 400 {"message":"invalid secret
-#    name"}. That is why set-ci-token.sh stores SETTINGS_API_TOKEN and not
-#    GITEA_SETTINGS_TOKEN. `CONFORMANCE_TRIGGER_KUBECONFIG` is legal, and is the exact name
-#    .gitea/workflows/{gate.yml,merge-gate.yaml} already reference — do not "tidy" it.
-#
-# 2. `curl -sS` EXITS 0 ON AN HTTP ERROR. The first version of set-ci-token.sh sent its PUT
-#    to /dev/null and printed "stored" unconditionally; the secret never existed and nobody
-#    knew for hours. So every write here captures `-w '%{http_code}'`, checks it, AND reads
-#    the object back afterwards. A store is not claimed unless the read-back shows it.
-#
-# 3. `CLUSTER_AVAILABLE` IS A LOADED GUN — SEE THE 2026-09-29 CORRECTION IN THE PLAN.
-#    `conformance/run.sh` exits 0 in scaffold mode with all 54 tests `<skipped>` when it
-#    cannot reach a cluster. Measured on 2026-09-29: with CLUSTER_AVAILABLE=true and no
-#    trigger path, ci/conformance.sh printed "[ ok ] L2/L3 conformance passed" at rc 0 — a
-#    FAKE GREEN on an enforcing required status check. ci/conformance.sh now BLOCKS that
-#    case (exit 1), so today the variable does not enable L2/L3: it only changes which way
-#    the stage fails. Arming it before §3 items 1-4 land therefore buys nothing and breaks
-#    every PR. This script refuses to do it.
-#
-# ----------------------------------------------------------------------------------------
-# WHY THE PRECONDITIONS ARE SPLIT INTO TWO CLASSES (deliberate, and load-bearing)
-# ----------------------------------------------------------------------------------------
-# MINT-BLOCKING (cluster-side). Kubernetes places NO `use` check on a Job's
-# `serviceAccountName`, so a token with plain `create jobs` in ns `conformance` can create a
-# Job running as the PRIVILEGED `conformance-runner` SA with an arbitrary command. The
-# trigger Role is only the necessary half; the sufficient half is the Kyverno ClusterPolicy
-# `conformance-trigger-job-shape` (platform/kyverno/admission/), which must be present AND
-# Enforce AND Ready. Minting this token while that policy is absent or in Audit hands CI a
-# privileged identity. So those checks block the MINT itself — nothing is created if they
-# fail. (See conformance/incluster/rbac.yaml's trigger header and plan §4.)
-#
-# ARM-BLOCKING (repo-side). The runner image, the Job template's init-container, and
-# ci/conformance.sh's in-cluster trigger flow (plan §3 items 1, 2, 4) are what make a real
-# L2/L3 run possible. Storing the kubeconfig secret without them is harmless — the
-# conformance job is gated on the variable and never starts — and the plan explicitly lands
-# the secret before the switch. So these checks block ONLY the variable.
-#
-# Net effect: you can safely run this script today to provision the secret. It will then
-# REFUSE to arm the variable, and tell you exactly which repo-side item is missing.
-#
-# ----------------------------------------------------------------------------------------
-# USAGE
-# ----------------------------------------------------------------------------------------
-#   ./infra/gitea/conformance-trigger-kubeconfig.sh --dry-run
-#         Read-only. Measures every precondition, mints nothing, writes nothing.
-#   ./infra/gitea/conformance-trigger-kubeconfig.sh [--server URL]
-#         Mint + store the secret. Does NOT touch CLUSTER_AVAILABLE.
-#   ./infra/gitea/conformance-trigger-kubeconfig.sh --arm-cluster-available
-#         The above, then set vars.CLUSTER_AVAILABLE=true after a typed confirmation —
-#         refused unless every precondition is green.
-#
-# Options:
-#   --dry-run                 measure only; no TokenRequest, no PUT/POST
-#   --arm-cluster-available   additionally set vars.CLUSTER_AVAILABLE=true (guarded)
-#   --server URL              API server URL to embed (default: the operator context's own)
-#   --context NAME            kube context to read the CA/server from (default: k3s-prod)
-#   --duration D              requested token lifetime (default: 720h = 30d)
-#   -h | --help               this text
-#
-# ON --server: the embedded server URL must be reachable FROM THE CI RUNNER POD, which is
-# not necessarily the address your workstation uses. The runner has broad egress and reaches
-# the kube-API VIP (per the plan: 192.168.1.49:6443). If your operator kubeconfig points at
-# a workstation-only name or a localhost tunnel, pass --server explicitly; the script warns
-# on a loopback server but cannot know your topology for you.
-#
-# ON TOKEN LIFETIME: this is a TokenRequest token and it EXPIRES. The API server may cap the
-# requested duration, so the script decodes the JWT `exp` claim and reports the ACTUAL
-# expiry rather than trusting --duration. Re-run this script to rotate. After expiry the
-# trigger gets 401 and the conformance stage FAILS (loudly) rather than skipping — which is
-# the correct direction, but put the expiry date in your calendar.
-#
-# DELIBERATELY NOT DONE: SOPS-sealing the token into secrets/conformance-trigger.sops.yaml
-# (plan §3 item 6's "for reproducibility/rotation"). A bearer token with a 30-day life
-# cannot usefully survive the branch -> PR -> review -> merge flow (CLAUDE.md §6), and
-# committing one buys re-issuance we get for free by re-running this script against the
-# live SA. If the operator still wants it sealed, that is a separate reviewed change.
-#
-# FORBIDDEN HERE, ON PURPOSE: no cluster-admin kubeconfig, no copy of the operator's own
-# kubeconfig, no widening of conformance/incluster/rbac.yaml, no mutating kubectl beyond the
-# TokenRequest itself, no `kubectl apply`.
-set -euo pipefail
-
-# xtrace would print the token. Refuse rather than leak.
-case "$-" in
-  *x*) echo "FATAL: refusing to run under 'set -x' — xtrace would print the token" >&2; exit 1 ;;
-esac
-umask 077
-
-HOST=gitea.home.keithjackson.dev
-OWNER=Home-Network
-REPO=home-network-k3s-impl
-# Legal secret name: NOT prefixed GITEA_ (lesson 1), and the name the workflows already use.
-SECRET=CONFORMANCE_TRIGGER_KUBECONFIG
-VARIABLE=CLUSTER_AVAILABLE
-NS=conformance
-SA=conformance-trigger
-RUNNER_SA=conformance-runner
-POLICY=conformance-trigger-job-shape
-# The image prefix the admission policy admits. Kept in sync with
-# platform/kyverno/admission/conformance-trigger-job-shape.yaml by the P2 check below.
-RUNNER_IMAGE_PREFIX='gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:'
-
-KCTX=k3s-prod
-DURATION=720h
-DRY_RUN=false
-ARM=false
-SERVER_OVERRIDE=""
-
-REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
-
-step() { printf '\n==> %s\n' "$*"; }
-info() { printf '    %s\n' "$*"; }
-warn() { printf '    WARN: %s\n' "$*" >&2; }
-die()  { printf 'FATAL: %s\n' "$*" >&2; exit 1; }
-
-usage() {
-  cat <<'USAGE'
-conformance-trigger-kubeconfig.sh — provision the narrow L2/L3 conformance CI trigger credential.
-
-  --dry-run                 measure every precondition; mint nothing, write nothing (read-only)
-  --arm-cluster-available   additionally set vars.CLUSTER_AVAILABLE=true (guarded + confirmed)
-  --server URL              API server URL to embed (default: the operator context's own;
-                            must be reachable FROM THE CI RUNNER POD — the kube-API VIP)
-  --context NAME            kube context to read the CA/server from (default: k3s-prod)
-  --duration D              requested token lifetime (default: 720h)
-  -h | --help               this text
-
-Read the header of this file before running it: the CLUSTER_AVAILABLE guard, the split between
-mint-blocking and arm-blocking preconditions, and why nothing secret is ever printed.
-USAGE
-}
-
-while [ $# -gt 0 ]; do
-  case "$1" in
-    --dry-run)               DRY_RUN=true ;;
-    --arm-cluster-available) ARM=true ;;
-    --server)                SERVER_OVERRIDE="${2:?--server needs a URL}"; shift ;;
-    --context)               KCTX="${2:?--context needs a name}"; shift ;;
-    --duration)              DURATION="${2:?--duration needs a value}"; shift ;;
-    -h|--help)               usage; exit 0 ;;
-    *)                       die "unknown argument: $1 (try --help)" ;;
-  esac
-  shift
-done
-if $DRY_RUN && $ARM; then die "--dry-run and --arm-cluster-available are contradictory"; fi
-
-for t in kubectl jq curl git base64; do
-  command -v "$t" >/dev/null || die "missing required tool: $t"
-done
-
-WORK="$(mktemp -d)"
-KC="${WORK}/conformance-trigger.kubeconfig"
-cleanup() { rm -rf "$WORK"; }
-trap cleanup EXIT
-trap 'cleanup; exit 130' INT TERM
-
-# ========================================================================================
-# 0. Gitea credentials — from the credential helper git already has for this host, exactly
-#    as branch-protection.sh / set-ci-token.sh / require-gate.sh do. Never in argv: curl
-#    reads them from a --config file descriptor.
-# ========================================================================================
-creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
-U="$(sed -n 's/^username=//p' <<<"$creds")"
-P="$(sed -n 's/^password=//p' <<<"$creds")"
-unset creds
-[ -n "$U" ] && [ -n "$P" ] || die "no stored credential for ${HOST} (git credential fill returned nothing)"
-basic() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
-API="https://${HOST}/api/v1/repos/${OWNER}/${REPO}"
-
-# ========================================================================================
-# 1. PRECONDITIONS — measured, never assumed (CLAUDE.md §2b: query the thing itself).
-#    Two independent tallies, per the split documented in the header.
-# ========================================================================================
-MINT_FAIL=0
-ARM_FAIL=0
-pass()      { printf '    [ ok ]   %s\n' "$*"; }
-fail_mint() { printf '    [MINT-BLOCK] %s\n' "$*" >&2; MINT_FAIL=$((MINT_FAIL + 1)); }
-fail_arm()  { printf '    [ARM-BLOCK]  %s\n' "$*" >&2; ARM_FAIL=$((ARM_FAIL + 1)); }
-
-step "preconditions — cluster side (these block the MINT: without them the token IS privileged)"
-
-kubectl --context "$KCTX" get --raw /version >/dev/null 2>&1 \
-  || die "cannot reach the API server via context '${KCTX}' — nothing below can be measured"
-pass "API server reachable via context ${KCTX}"
-
-# C1: the trigger identity exists (created by Flux from conformance/incluster/rbac.yaml via
-# clusters/base/conformance.yaml). We print object names only, never contents.
-if kubectl --context "$KCTX" -n "$NS" get serviceaccount "$SA" -o name >/dev/null 2>&1; then
-  pass "ServiceAccount ${NS}/${SA} exists"
-else
-  fail_mint "ServiceAccount ${NS}/${SA} is ABSENT — is the 'conformance' Flux Kustomization reconciled?"
-fi
-for kind in role rolebinding; do
-  if kubectl --context "$KCTX" -n "$NS" get "$kind" "$SA" -o name >/dev/null 2>&1; then
-    pass "${kind} ${NS}/${SA} exists"
-  else
-    fail_mint "${kind} ${NS}/${SA} is ABSENT — the token would authenticate but authorize nothing"
-  fi
-done
-
-# C2: THE SECURITY BOUNDARY. Present, Enforce, and Ready. Anything else and we do not mint.
-# `validationFailureAction: Audit` is not a boundary; an unready policy is not a boundary.
-if pol="$(kubectl --context "$KCTX" get clusterpolicy "$POLICY" -o json 2>/dev/null)"; then
-  action="$(jq -r '.spec.validationFailureAction // .spec.rules[0].validate.failureAction // "UNSET"' <<<"$pol")"
-  failpol="$(jq -r '.spec.failurePolicy // "UNSET"' <<<"$pol")"
-  # Kyverno moved readiness from .status.ready to a Ready condition; accept either, and
-  # treat "neither key present" as UNMEASURABLE -> block (fail closed, do not assume ready).
-  ready="$(jq -r '
-      if (.status.ready != null) then (.status.ready | tostring)
-      elif (.status.conditions != null) then
-        ((.status.conditions[] | select(.type == "Ready") | .status) // "UNKNOWN")
-      else "UNKNOWN" end' <<<"$pol")"
-  info "ClusterPolicy ${POLICY}: validationFailureAction=${action} failurePolicy=${failpol} ready=${ready}"
-  [ "$action" = "Enforce" ] || fail_mint "ClusterPolicy ${POLICY} is '${action}', not Enforce — the trigger would be unbounded"
-  [ "$failpol" = "Fail" ]   || fail_mint "ClusterPolicy ${POLICY} failurePolicy='${failpol}', not Fail — a Kyverno outage would admit arbitrary Jobs"
-  case "$ready" in
-    true|True) pass "ClusterPolicy ${POLICY} reports Ready" ;;
-    *)         fail_mint "ClusterPolicy ${POLICY} does not report Ready (got '${ready}') — refusing to assume it is loaded" ;;
-  esac
-else
-  fail_mint "ClusterPolicy ${POLICY} is ABSENT — plan §4's boundary does not exist; a token with 'create jobs' here can run arbitrary code as ${RUNNER_SA}"
-fi
-
-step "preconditions — repo side (these block ONLY arming ${VARIABLE}; the secret may be stored first)"
-
-# R1: the runner image is built and digest-pinned. Without it the Job cannot run run.sh at
-# all (the stock chainsaw image has no bash/python3/kubectl/git) — plan §3 item 1.
-VERSIONS="${REPO_ROOT}/conformance/versions.env"
-if grep -Eq '^[[:space:]]*CONFORMANCE_RUNNER_IMAGE=.*@sha256:[0-9a-f]{64}' "$VERSIONS" 2>/dev/null; then
-  pass "CONFORMANCE_RUNNER_IMAGE is digest-pinned in conformance/versions.env"
-else
-  fail_arm "conformance/versions.env has no digest-pinned CONFORMANCE_RUNNER_IMAGE (plan §3 item 1) — the Job cannot run ./run.sh"
-fi
-
-# R2: the Job template is the real thing, not the skeleton: the admitted image AND an
-# init-container that populates the empty `workspace` volume with the suite at the PR SHA
-# (plan §3 item 2). A template whose image the policy will not admit is a guaranteed denial.
-JOBTPL="${REPO_ROOT}/conformance/incluster/chainsaw-job.yaml"
-if grep -q 'initContainers' "$JOBTPL" 2>/dev/null; then
-  pass "chainsaw-job.yaml declares an initContainers block (workspace population)"
-else
-  fail_arm "chainsaw-job.yaml has no initContainers — 'workspace' is an empty emptyDir and the suite is never delivered (plan §3 item 2)"
-fi
-if grep -q "$RUNNER_IMAGE_PREFIX" "$JOBTPL" 2>/dev/null; then
-  pass "chainsaw-job.yaml uses the digest-pinned image the admission policy admits"
-else
-  fail_arm "chainsaw-job.yaml does not use ${RUNNER_IMAGE_PREFIX}… — every triggered Job would be DENIED by ${POLICY}"
-fi
-
-# R3: ci/conformance.sh actually drives the in-cluster Job. Today it mentions
-# CONFORMANCE_TRIGGER_KUBECONFIG only in a TODO comment and otherwise runs run.sh LOCALLY in
-# the CI job — where there is no credential at all (the runner sets
-# automountServiceAccountToken: false by design), so it can only scaffold-skip. Hence the
-# check must ignore comment lines: a mention in prose is not an implementation.
-#
-# ONE awk, NOT `grep -v … | grep -q …`: under `set -o pipefail` a successful `grep -q` exits
-# early, the upstream grep takes EPIPE, and the pipeline reports non-zero — which would read
-# here as "unwired" exactly when the seam IS wired. A single process cannot have that bug.
-CONF_STAGE="${REPO_ROOT}/ci/conformance.sh"
-if awk '!/^[[:space:]]*#/ && /CONFORMANCE_TRIGGER_KUBECONFIG/ { found = 1 } END { exit !found }' \
-     "$CONF_STAGE" 2>/dev/null; then
-  pass "ci/conformance.sh references CONFORMANCE_TRIGGER_KUBECONFIG in code (in-cluster trigger path wired)"
-else
-  fail_arm "ci/conformance.sh mentions CONFORMANCE_TRIGGER_KUBECONFIG only in comments — the trigger path is UNIMPLEMENTED (plan §3 item 4); arming ${VARIABLE} now only changes which way the stage fails"
-fi
-
-info ""
-info "precondition tally: mint-blocking failures=${MINT_FAIL}  arm-blocking failures=${ARM_FAIL}"
-
-if [ "$MINT_FAIL" -gt 0 ]; then
-  cat >&2 <<EOF
-
-REFUSING TO MINT. ${MINT_FAIL} cluster-side precondition(s) failed.
-
-Minting a token for ${NS}/${SA} while the admission boundary is absent/Audit/unready hands CI
-an identity that can create a Job running as ${RUNNER_SA} (nodes/patch, pods/exec,
-namespaces/patch) with an arbitrary command. That is precisely the escalation
-docs/plans/l2-l3-conformance-ci-wiring.md §4 exists to close, and RBAC cannot close it.
-
-Fix the cluster side first (reconcile platform/kyverno/admission/ and conformance/incluster/,
-confirm the ClusterPolicy is Enforce + Ready), then re-run. Nothing was created.
-EOF
-  exit 1
-fi
-
-if $ARM && [ "$ARM_FAIL" -gt 0 ]; then
-  cat >&2 <<EOF
-
-REFUSING TO ARM ${VARIABLE}. ${ARM_FAIL} repo-side precondition(s) failed.
-
-Measured 2026-09-29: with ${VARIABLE}=true and no working trigger path, conformance/run.sh
-scaffold-skips all 54 tests at rc 0 and ci/conformance.sh used to print
-"[ ok ] L2/L3 conformance passed" — a fake green on an enforcing required status check. The
-guard now in ci/conformance.sh turns that into exit 1 instead, so arming today does not
-enable L2/L3: it breaks every PR while proving nothing.
-
-Land plan §3 items 1, 2 and 4 first. Re-run WITHOUT --arm-cluster-available to provision the
-secret in the meantime — that part is safe and is step 6's intended order.
-EOF
-  exit 1
-fi
-
-if $DRY_RUN; then
-  step "DRY RUN — stopping here. Nothing was minted, nothing was written."
-  info "secret that WOULD be written: ${SECRET} (repo ${OWNER}/${REPO})"
-  info "variable that WOULD be set:   none — ${VARIABLE} is never touched in --dry-run"
-  # Read-only visibility into what is already there. Gitea never returns secret VALUES.
-  names="$(basic "${API}/actions/secrets" | jq -rc '[.. | objects | select(has("name")) | .name] | unique')" || true
-  info "existing Actions secret names: ${names:-<unreadable>}"
-  cur="$(basic -o "${WORK}/var.json" -w '%{http_code}' "${API}/actions/variables/${VARIABLE}")" || true
-  if [ "$cur" = 200 ]; then
-    info "existing ${VARIABLE} = $(jq -r '.data // "<no data field>"' "${WORK}/var.json")"
-  else
-    info "existing ${VARIABLE}: not set (HTTP ${cur})"
-  fi
-  exit 0
-fi
-
-# ========================================================================================
-# 2. MINT the ServiceAccount token (TokenRequest). This is the only write this script makes
-#    against the cluster, and it creates no durable object.
-# ========================================================================================
-step "minting a ServiceAccount token for ${NS}/${SA} (requested duration ${DURATION})"
-TOKEN="$(kubectl --context "$KCTX" -n "$NS" create token "$SA" --duration="$DURATION")" \
-  || die "kubectl create token failed for ${NS}/${SA}"
-[ -n "$TOKEN" ] || die "kubectl create token returned an empty token"
-info "got a ${#TOKEN}-character token (value NOT shown)"
-
-# Report the ACTUAL expiry from the JWT rather than trusting --duration: the API server caps
-# TokenRequest lifetimes (--service-account-max-token-expiration) and silently shortens.
-jwt_exp() {
-  local payload pad b64
-  payload="$(cut -d. -f2 <<<"$1")"
-  pad=$(( (4 - ${#payload} % 4) % 4 ))
-  b64="$(tr '_-' '/+' <<<"$payload")$(printf '%*s' "$pad" '' | tr ' ' '=')"
-  printf '%s' "$b64" | base64 -d 2>/dev/null | jq -r '.exp // empty' 2>/dev/null || true
-}
-exp="$(jwt_exp "$TOKEN")"
-if [ -n "$exp" ]; then
-  info "token expires: $(date -u -d "@${exp}" '+%Y-%m-%dT%H:%M:%SZ' 2>/dev/null || echo "epoch ${exp}") — re-run this script to rotate"
-else
-  warn "could not decode the token's exp claim; lifetime is UNVERIFIED (requested ${DURATION})"
-fi
-
-# ========================================================================================
-# 3. ASSEMBLE the kubeconfig — SA token + cluster CA + server. Nothing else.
-#
-#    The CA and the server URL are read from the operator's context, but the operator's
-#    client certificate/key are NEVER copied: only .cluster fields are extracted. `config
-#    view --raw` output is held in a variable and discarded, never printed.
-#    Built with jq (not `kubectl config set-credentials --token=…`) so the token never
-#    appears in argv where `ps` can see it.
-# ========================================================================================
-step "assembling the kubeconfig (SA token only — never cluster-admin, never the operator's)"
-raw="$(kubectl --context "$KCTX" config view --raw --minify -o json)" \
-  || die "could not read context '${KCTX}' from the operator kubeconfig"
-SERVER="$(jq -r '.clusters[0].cluster.server // empty' <<<"$raw")"
-CADATA="$(jq -r '.clusters[0].cluster["certificate-authority-data"] // empty' <<<"$raw")"
-CAFILE="$(jq -r '.clusters[0].cluster["certificate-authority"] // empty' <<<"$raw")"
-INSECURE="$(jq -r '.clusters[0].cluster["insecure-skip-tls-verify"] // false | tostring' <<<"$raw")"
-unset raw
-
-[ -n "$SERVER" ] || die "context '${KCTX}' has no cluster.server"
-if [ -n "$SERVER_OVERRIDE" ]; then
-  info "overriding server ${SERVER} -> ${SERVER_OVERRIDE}"
-  SERVER="$SERVER_OVERRIDE"
-fi
-case "$SERVER" in
-  *localhost*|*127.0.0.1*|*://0.0.0.0*)
-    die "server '${SERVER}' is loopback — the CI runner pod cannot reach it. Pass --server with the address the runner uses (the kube-API VIP)." ;;
-esac
-if [ -z "$CADATA" ] && [ -n "$CAFILE" ]; then
-  [ -r "$CAFILE" ] || die "certificate-authority file '${CAFILE}' is not readable"
-  CADATA="$(base64 -w0 <"$CAFILE")"
-fi
-[ -n "$CADATA" ] || die "no cluster CA available for '${KCTX}' (insecure-skip-tls-verify=${INSECURE}); refusing to emit a kubeconfig that trusts anything"
-[ "$INSECURE" = "false" ] || die "context '${KCTX}' sets insecure-skip-tls-verify; refusing to propagate that into CI"
-info "server: ${SERVER}"
-info "CA: ${#CADATA} base64 characters embedded (public certificate, not a secret)"
-
-jq -n --arg srv "$SERVER" --arg ca "$CADATA" --arg tok "$TOKEN" --arg ns "$NS" --arg sa "$SA" '
-{
-  apiVersion: "v1",
-  kind: "Config",
-  clusters:  [ { name: "conformance", cluster: { server: $srv, "certificate-authority-data": $ca } } ],
-  users:     [ { name: $sa, user: { token: $tok } } ],
-  contexts:  [ { name: "conformance", context: { cluster: "conformance", user: $sa, namespace: $ns } } ],
-  "current-context": "conformance"
-}' >"$KC"
-chmod 600 "$KC"
-info "kubeconfig written to the temp dir: $(wc -c <"$KC") bytes (content NOT shown)"
-
-# Prove kubectl can PARSE the file we generated (JSON is valid YAML, but assert it rather
-# than assume it) — this prints only the server we already printed.
-parsed="$(kubectl --kubeconfig "$KC" config view -o jsonpath='{.clusters[0].cluster.server}')" \
-  || die "kubectl cannot parse the generated kubeconfig"
-[ "$parsed" = "$SERVER" ] || die "generated kubeconfig parses to server '${parsed}', expected '${SERVER}'"
-pass "kubectl parses the generated kubeconfig"
-
-# ========================================================================================
-# 4. PROVE it is the narrow identity. Two independent proofs:
-#      a) who the token authenticates AS (SelfSubjectReview)
-#      b) an allow/deny matrix (SelfSubjectAccessReview) — both halves matter: "can create
-#         jobs" proves it works, "cannot get secrets / delete jobs / exec" proves it is not
-#         something wider that happens to include the trigger rights.
-#    All read-only. `auth can-i` answers from the API server's own authorizer.
-# ========================================================================================
-step "verifying the credential is the NARROW trigger identity"
-EXPECT_USER="system:serviceaccount:${NS}:${SA}"
-if who="$(kubectl --kubeconfig "$KC" auth whoami -o jsonpath='{.status.userInfo.username}' 2>/dev/null)"; then
-  [ "$who" = "$EXPECT_USER" ] || die "token authenticates as '${who}', expected '${EXPECT_USER}' — WRONG CREDENTIAL, nothing stored"
-  pass "authenticates as ${who}"
-else
-  warn "'kubectl auth whoami' unavailable (older kubectl / SelfSubjectReview denied) — identity UNVERIFIED by this check; the matrix below still constrains it"
-fi
-
-# A real authorization round-trip, not just a review: `list jobs` is granted by the Role.
-kubectl --kubeconfig "$KC" -n "$NS" get jobs -o name >/dev/null \
-  || die "the minted credential cannot list Jobs in ns ${NS} — the RoleBinding is not effective; nothing stored"
-pass "can list Jobs in ns ${NS} (end-to-end authentication + authorization)"
-
-cani() { # cani <expected yes|no> <verb> <resource...>
-  local want="$1"; shift
-  local got
-  got="$(kubectl --kubeconfig "$KC" auth can-i "$@" 2>/dev/null || true)"
-  case "$got" in
-    yes|no) : ;;
-    *) die "auth can-i '$*' returned '${got:-<nothing>}' — cannot measure the identity's rights; refusing to store an unverified credential" ;;
-  esac
-  if [ "$got" = "$want" ]; then
-    pass "can-i $* -> ${got} (expected)"
-  else
-    die "can-i $* -> ${got}, expected ${want}. This credential is NOT the narrow trigger identity described in conformance/incluster/rbac.yaml. Nothing stored. Do NOT widen that Role to make this pass — investigate what it is actually bound to."
-  fi
-}
-# Granted by the trigger Role (create/get/list/watch jobs; get/list/watch pods + pods/log).
-cani yes create jobs.batch -n "$NS"
-cani yes get    pods       -n "$NS"
-cani yes get    pods/log   -n "$NS"
-# Explicitly NOT granted — each is a specific escape route the Role's comments call out.
-cani no  delete jobs.batch -n "$NS"          # a trigger that can delete can hide a failed run
-cani no  create pods/exec  -n "$NS"          # no execution channel into the runner pod
-cani no  get    secrets    -n "$NS"
-cani no  create jobs.batch -n default        # namespaced Role: no reach outside ns conformance
-cani no  get    nodes                        # the RUNNER has nodes/patch; the trigger must not
-cani no  '*' '*' --all-namespaces            # not cluster-admin, stated as a measurement
-
-# ========================================================================================
-# 5. STORE as the Gitea Actions repo secret, then VERIFY BY READING IT BACK (lesson 2).
-#    The body goes in a 0600 file so the token is not in argv.
-# ========================================================================================
-step "storing the kubeconfig as repo Actions secret ${SECRET}"
-jq -n --rawfile d "$KC" '{data:$d}' >"${WORK}/body.json"
-chmod 600 "${WORK}/body.json"
-code="$(basic -o "${WORK}/put.out" -w '%{http_code}' -X PUT \
-          -H 'Content-Type: application/json' --data @"${WORK}/body.json" \
-          "${API}/actions/secrets/${SECRET}")"
-unset TOKEN
-rm -f "${WORK}/body.json" "$KC"
-if [ "$code" != 201 ] && [ "$code" != 204 ]; then
-  # Safe to echo: the response body is Gitea's error message, not our payload. HTTP 400
-  # "invalid secret name" here means the name hit the reserved GITEA_ prefix (lesson 1).
-  die "PUT ${SECRET} failed: HTTP ${code} -- $(head -c 300 "${WORK}/put.out" 2>/dev/null)"
-fi
-info "PUT returned HTTP ${code}"
-
-# curl -sS exits 0 on an HTTP error, so the status check above is necessary but NOT
-# sufficient: read the object back. Gitea never returns secret VALUES, so this proves
-# PRESENCE BY NAME only — say so rather than implying the content was checked.
-names="$(basic "${API}/actions/secrets" | jq -rc '[.. | objects | select(has("name")) | .name] | unique')"
-grep -q "$SECRET" <<<"$names" \
-  || die "PUT reported HTTP ${code} but ${SECRET} is NOT in ${names} — refusing to claim success"
-pass "stored and VERIFIED PRESENT by read-back: ${names}"
-info "(read-back proves the secret EXISTS; Gitea never returns secret values, so its CONTENT"
-info " is proven only by the first CI run that uses it — see the NOTE below.)"
-info ""
-info "FOR WHOEVER WRITES plan §3 item 4: this secret holds the kubeconfig CONTENT, not a path."
-info "The plan's sketch writes 'kubectl --kubeconfig=\$CONFORMANCE_TRIGGER_KUBECONFIG', which"
-info "would treat it as a filename. ci/conformance.sh must write the env var into a 0600 temp"
-info "file (umask 077, trap-removed) and pass THAT path — an Actions secret is a string, so a"
-info "kubeconfig cannot reach CI any other way."
-
-# ========================================================================================
-# 6. vars.CLUSTER_AVAILABLE — the loaded gun. Only here, only on request, only confirmed.
-# ========================================================================================
-if ! $ARM; then
-  cat <<EOF
-
-==> ${VARIABLE} NOT TOUCHED (no --arm-cluster-available). This is the safe default.
-
-The secret is in place; the conformance job stays gated off until the variable is set. That
-is plan §5's order: land items 1-6, prove them, and only then flip step 7's switch.
-EOF
-else
-  step "arming vars.${VARIABLE}=true — every precondition above is green"
-  cat <<EOF
-    This makes .gitea/workflows/gate.yml's conformance job RUN on every PR. If the
-    in-cluster trigger path misbehaves, ci/conformance.sh will BLOCK (exit 1) rather than
-    fake a green — so the failure mode is loud PRs, not silent false confidence. Undo with:
-      curl -X DELETE .../repos/${OWNER}/${REPO}/actions/variables/${VARIABLE}
-EOF
-  [ -t 0 ] || die "refusing to arm ${VARIABLE} non-interactively — this needs a human confirmation"
-  printf '    Type exactly "ARM %s" to proceed: ' "$VARIABLE"
-  read -r confirm
-  [ "$confirm" = "ARM ${VARIABLE}" ] || die "not confirmed (got '${confirm}') — ${VARIABLE} unchanged. The secret IS stored."
-
-  # Gitea's variables API: POST creates, PUT updates, and the body field is `value` — NOT
-  # `data` as in the secrets API above. Try create, fall back to update. The value "true" is
-  # not a secret, so response bodies are printed here to make a failure diagnosable.
-  vcode="$(basic -o "${WORK}/var.out" -w '%{http_code}' -X POST \
-             -H 'Content-Type: application/json' --data '{"value":"true"}' \
-             "${API}/actions/variables/${VARIABLE}")"
-  if [ "$vcode" != 201 ] && [ "$vcode" != 204 ]; then
-    info "POST returned HTTP ${vcode} ($(head -c 200 "${WORK}/var.out" 2>/dev/null)); trying PUT (variable may already exist)"
-    vcode="$(basic -o "${WORK}/var.out" -w '%{http_code}' -X PUT \
-               -H 'Content-Type: application/json' \
-               --data "$(jq -n --arg n "$VARIABLE" '{name:$n,value:"true"}')" \
-               "${API}/actions/variables/${VARIABLE}")"
-  fi
-  [ "$vcode" = 201 ] || [ "$vcode" = 204 ] \
-    || die "could not set ${VARIABLE}: HTTP ${vcode} -- $(head -c 300 "${WORK}/var.out" 2>/dev/null)"
-  info "write returned HTTP ${vcode}"
-
-  # Unlike a secret, a variable's value IS readable — so verify the VALUE, not just presence.
-  # gate.yml compares `vars.CLUSTER_AVAILABLE == 'true'`, so it must be exactly "true".
-  gcode="$(basic -o "${WORK}/varget.json" -w '%{http_code}' "${API}/actions/variables/${VARIABLE}")"
-  [ "$gcode" = 200 ] || die "stored HTTP ${vcode} but GET ${VARIABLE} returned HTTP ${gcode} — refusing to claim success"
-  val="$(jq -r '.data // .value // empty' "${WORK}/varget.json")"
-  [ "$val" = "true" ] || die "${VARIABLE} reads back as '${val}', not 'true' — gate.yml compares against 'true' exactly"
-  pass "${VARIABLE}=${val} stored and VERIFIED by read-back"
-fi
-
-cat <<EOF
-
-NEXT — and none of it is proven until you watch it happen:
-  1. Open a THROWAWAY PR. Read the conformance job's LOG, not its tick: a skipped Gitea job
-     reports success ("Has been skipped"), and a green tick is not evidence anything ran.
-  2. Confirm a Job appeared in ns ${NS}, ran the touched --requirements subset, and that its
-     artifacts came back. Then break an L3 test deliberately and confirm the gate BLOCKS.
-  3. Only after seeing BOTH directions should the conformance check be added to branch
-     protection (infra/gitea/require-gate.sh's status_check_contexts).
-  4. Diary the token expiry printed above. Re-run this script to rotate; CI fails loudly
-     (401 on the trigger) rather than skipping when it lapses.
-
-ROLLBACK: DELETE the ${VARIABLE} variable (instant SKIP, gate reverts to L0/L1-only) and/or
-DELETE the ${SECRET} secret. The token itself is revoked by deleting the ${NS}/${SA}
-ServiceAccount, which Flux will recreate with a fresh identity.
-EOF

platform/crafty/README.md

@@ -49,9 +49,49 @@ Crafty's UI is gated by a **dedicated oauth2-proxy sidecar in the Crafty pod** (
 `crafty.home.keithjackson.dev`). The `crafty` Service targets the sidecar, never Crafty's `:8443`, so
 the gate is **in the data path** (fail-closed: no valid `platform-admins` session ⇒ no upstream).
 
-> **Note:** Crafty ALSO has its own local admin login (it has no OIDC). After oauth2-proxy admits the
-> user, they still see Crafty's own login page — a second, defense-in-depth factor. The operator logs
-> in with Crafty's local admin credentials (set on first boot). This is expected, not a bug.
+#### Cookie isolation (why the session used to vanish)
+
+This sidecar and the **shared** forward-auth oauth2-proxy are independent instances with **different
+`cookie-secret` values** (measured in-cluster — neither can decrypt the other's cookie). Both used to
+emit oauth2-proxy's *default* cookie names (`_oauth2_proxy`, `_oauth2_proxy_csrf`) scoped to
+`.home.keithjackson.dev`, so the browser held **one** cookie pair for **both**: whichever proxy
+authenticated last clobbered the other's session, and the loser logged
+`No valid authentication in request`, bounced to Keycloak, and — because the Keycloak SSO session was
+still live — was redirected straight back, clobbering the cookie again. That mutual eviction presents
+exactly as *"I signed in to Keycloak but I'm still not signed in."*
+
+Fixed on this side only (renaming the shared proxy's cookie would sign out every other gated app):
+`--cookie-name=_oauth2_proxy_crafty` and a **host-scoped** `--cookie-domain=crafty.home.keithjackson.dev`.
+`--whitelist-domain` stays zone-wide — it governs post-login *redirect* targets, not cookie scope.
+
+#### Crafty's own local login — this is where setup actually blocks
+
+Crafty has **no OIDC**, so after oauth2-proxy admits the user they still meet Crafty's own
+username/password page. That second factor is intended. What was **not** intended — and what made the
+service impossible to set up — is that **nothing documented where that password comes from**.
+
+Crafty generates it **once, on first boot only**, when the users table is empty
+(`main.py` `FRESH_INSTALL` → `DatabaseBuilder.default_settings()`), and writes it to
+**`/crafty/app/config/default-creds.txt`** (mode `0600`, on the `config` subPath of the PVC). It is
+**never** in git and **cannot** be recovered from git: a rebuild onto a fresh PVC generates a *new*
+random password. Submitting anything else yields `POST /api/v2/auth/login/` → **401**, which is exactly
+what a browser sees with no hint as to why.
+
+**→ Retrieval, rotation, and lost-password recovery: [`../../runbooks/crafty-admin-credential.md`](../../runbooks/crafty-admin-credential.md).**
+
+Two traps that runbook covers:
+
+- Crafty's `set_passwd` console command is **unreachable** in Kubernetes — it calls
+  `getpass.getpass()` and so needs a TTY, while the container runs `main.py -d -i` (daemon).
+  Recovery therefore has to go through the DB with Crafty's own argon2 hasher.
+- Crafty's own 3-strikes brute-force cooldown is **inert behind this sidecar**, verified live. Crafty
+  *records* failed attempts under `get_remote_ip()` (which honours `X-Forwarded-For` — the live
+  tracker key is the chain `"192.168.2.132, 10.0.0.123"`) but *checks* for a cooldown under
+  `self.request.remote_ip`, which behind the sidecar is always `127.0.0.1`. The two halves key
+  differently, so `get_recent_attempts()` always reads an empty list and `max_login_attempts: 3`
+  never fires. Good news for the operator (**repeated guesses cannot lock you out**), but it means
+  the only thing actually limiting who can reach the login form is the oauth2-proxy
+  `platform-admins` gate — worth knowing before anyone treats that cooldown as a control.
 
 ### Keycloak client
 

platform/crafty/deployment.yaml

@@ -150,7 +150,27 @@ spec:
             # Single provider => 302 straight to Keycloak, no oauth2-proxy interstitial button.
             - --skip-provider-button=true
             - --redirect-url=https://crafty.${BASE_DOMAIN}/oauth2/callback
-            - --cookie-domain=.${BASE_DOMAIN}
+            # ── COOKIE ISOLATION (bug fix 2026-09-29) ────────────────────────────────────────────
+            # This sidecar and the SHARED forward-auth oauth2-proxy (platform/oauth2-proxy) are
+            # INDEPENDENT instances holding DIFFERENT cookie-secrets (verified in-cluster: the two
+            # `cookie-secret` values differ, so neither can decrypt the other's cookie). Both used
+            # to emit oauth2-proxy's DEFAULT cookie names (`_oauth2_proxy` and `_oauth2_proxy_csrf`)
+            # scoped to `.${BASE_DOMAIN}`, so the browser kept ONE pair of cookies for BOTH proxies:
+            # whichever authenticated last overwrote the other's session, and the loser then logged
+            # "No valid authentication in request" and bounced to Keycloak -- which, thanks to the
+            # live Keycloak SSO session, silently redirected back and overwrote the cookie again.
+            # That mutual eviction presents EXACTLY as the reported symptom: "I signed in to
+            # Keycloak but I am still not signed in to the app."
+            # Two independent fixes, both applied here (only this side changes -- renaming the
+            # SHARED proxy's cookie would sign every other gated app out):
+            #   1. --cookie-name: a unique name, so the two sessions cannot alias at all.
+            #   2. --cookie-domain: HOST-scoped. A per-app reverse proxy has no reason to broadcast
+            #      its session to every other host in the zone; the shared proxy keeps the wide
+            #      scope because cross-host forward-auth is precisely its job.
+            # --whitelist-domain deliberately stays zone-wide: it governs permitted post-login
+            # REDIRECT targets, not cookie scope.
+            - --cookie-name=_oauth2_proxy_crafty
+            - --cookie-domain=crafty.${BASE_DOMAIN}
             - --whitelist-domain=.${BASE_DOMAIN}
             - --cookie-secure=true
             - --pass-access-token=true

runbooks/README.md

@@ -115,6 +115,12 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
   plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
   token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
   Includes why a changed Secret does not restart anything, and why an expired token is harmless.
+- **`crafty-admin-credential`** — you signed into Keycloak and Crafty *still* asks for a password.
+  Expected: Crafty 4 has no OIDC, so its own local login sits behind the oauth2-proxy gate as a second
+  factor. The page tells the two layers apart from the sidecar log, and explains that the bootstrap
+  password exists **only** on the PVC (`default-creds.txt`, generated once on first boot, never in git).
+  Includes the argon2 reset for a lost password, why the 3-strikes lockout cannot fire behind the
+  sidecar, and why deleting `crafty.sqlite` to force a new password destroys every server definition.
 
 ### Virtualisation
 - **`kubevirt-control-plane`** — VM lifecycle operations will fail; running VMs keep running.

runbooks/crafty-admin-credential.md

@@ -0,0 +1,179 @@
+# crafty-admin-credential — Crafty's own login rejects you after Keycloak sign-in
+
+**Not alert-driven.** This is an operator procedure, linked from
+[`platform/crafty/README.md`](../platform/crafty/README.md). It covers getting into the Crafty panel the
+first time, rotating that credential, and recovering when it is lost.
+
+> **Everything below prints or sets a LIVE credential.** Do not paste the output into a PR, an issue, a
+> chat, or an AI agent transcript. Read it, use it, rotate it. (Repo rule §6: secrets never enter a
+> transcript.)
+
+## What this means here
+
+`crafty.home.keithjackson.dev` sits behind **two** independent authentications, and only the first is
+Keycloak's:
+
+1. the **oauth2-proxy sidecar** in the Crafty pod — Keycloak `homelab` realm, group `platform-admins`;
+2. **Crafty's own local username/password login** — Crafty 4 has no OIDC and cannot consume the
+   sidecar's identity headers, so it shows its own login page to an already-authenticated user.
+
+So "I signed into Keycloak and I'm still not signed in" is the **expected** shape of this service, and
+the real question is only ever *which* of the two rejected you. They fail differently:
+
+| Where you are | What you see | Which layer |
+|---|---|---|
+| Bounced to Keycloak repeatedly, never reach Crafty | sidecar log: `No valid authentication in request` | oauth2-proxy |
+| Authenticated but `403` | not in `platform-admins` | oauth2-proxy |
+| **Crafty's own login page, credentials refused** | `POST /api/v2/auth/login/` → **401** | **Crafty local** |
+
+The third row is the one that blocks setup, because **nothing in the UI tells you where that password
+comes from**. Crafty generates it **once, on first boot only** — when the users table is empty
+(`main.py` `FRESH_INSTALL` → `DatabaseBuilder.default_settings()`) — and writes it to
+`/crafty/app/config/default-creds.txt` (mode `0600`) on the `config` subPath of the `crafty-data` PVC.
+It is **never in git** and **cannot be recovered from git**: a rebuild onto a fresh PVC generates a new
+random password, and the old one is gone with the old volume.
+
+Two things that are *not* the problem, both verified live:
+
+- **Repeated wrong guesses cannot lock you out.** Crafty *records* failed attempts under
+  `get_remote_ip()` (honours `X-Forwarded-For`) but *checks* the 3-strikes cooldown under
+  `self.request.remote_ip`, which behind the sidecar is always `127.0.0.1`. The halves key
+  differently, so `max_login_attempts: 3` never fires. (It also means that throttle is not a control
+  you have — the `platform-admins` gate is what limits who reaches the form at all.)
+- **Crafty's `set_passwd` console command is unreachable here.** It calls `getpass.getpass()` and needs
+  a TTY; the container runs `main.py -d -i` as a daemon. Recovery goes through the DB instead.
+
+## First commands
+
+`kubectl` does not exist on the workstation — the toolchain lives only in the pinned dev container, and
+`dev.sh` picks a current-context **per invocation**, so always name the context.
+
+**1. Is the pod healthy, and is the password still the generated one?**
+
+```bash
+./infra/devcontainer/dev.sh run bash -c '
+kubectl --context k3s-prod -n crafty get pods -l app.kubernetes.io/name=crafty
+kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- python3 -c "
+import sqlite3
+c=sqlite3.connect(\"file:/crafty/app/config/db/crafty.sqlite?mode=ro\",uri=True)
+for r in c.execute(\"SELECT username,created,last_update,last_login FROM users WHERE user_id=1\"):
+    print(\"created    :\",r[1]); print(\"last_update:\",r[2]); print(\"last_login :\",r[3])
+"'
+```
+
+Read `last_update` against `created`: **equal ⇒ the password has never been changed**, so
+`default-creds.txt` is still live. Different ⇒ someone rotated it and that file is stale — skip to
+*Reset*.
+
+**2. Read the bootstrap credential** (prints a live secret — see the warning at the top):
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- cat /crafty/app/config/default-creds.txt'
+```
+
+The username is `admin`. Sign in at `https://crafty.home.keithjackson.dev` (Keycloak first, then this).
+
+**3. Rotate it immediately**, in the UI: user menu → your profile → change password. That is the only
+supported path that keeps Crafty's own bookkeeping consistent. Then confirm it took — re-run command 1
+and check `last_update` has moved away from `created`.
+
+## Likely causes, most likely first
+
+1. **You never had the password.** By far the most common, and the reason this runbook exists. The
+   credential was only ever on the PVC. → commands 1–3 above.
+2. **The password was rotated and not recorded.** `last_update` ≠ `created` while nobody knows the
+   value. `default-creds.txt` is now misleading, not authoritative. → *Reset* below.
+3. **The PVC was rebuilt.** A fresh volume means a fresh random password *and* an empty panel — no
+   servers, no worlds (`platform/crafty/README.md`, "the servers themselves are NOT in git"). Read
+   `default-creds.txt` from the new pod; do not expect the old credential to work.
+4. **You are actually stuck at the oauth2-proxy layer, not Crafty's.** Check which layer is talking
+   before touching any password:
+   ```bash
+   ./infra/devcontainer/dev.sh run bash -c \
+     'kubectl --context k3s-prod -n crafty logs deploy/crafty -c oauth2-proxy --tail=200 | grep -v kube-probe'
+   ```
+   `[AuthSuccess] ... groups:[platform-admins]` followed by a `200` on `/login` means oauth2-proxy did
+   its job and the 401 is Crafty's. Repeated `No valid authentication in request` with no `AuthSuccess`
+   means the session cookie is not surviving — see *cookie isolation* in
+   `platform/crafty/README.md`; this sidecar must keep its own `--cookie-name` distinct from the shared
+   proxy's, or the two evict each other.
+5. **`platform-admins` membership lapsed.** A `403` rather than a login page. Fix in Keycloak, not here.
+
+## Reset when the password is unknown
+
+Crafty stores an **argon2** hash in `users.password`. This writes a fresh random password, invalidates
+existing sessions, and drops the new value into a `0600` file **inside the pod** — deliberately
+mirroring Crafty's own `default-creds.txt` idiom so the secret never crosses argv, stdin or shell
+history. (`dev.sh` does **not** forward stdin — a `kubectl exec -i` here hangs forever, which is why the
+password is generated in-pod rather than passed in.)
+
+Two details that will otherwise cost you the afternoon: argon2 is installed **only** in Crafty's venv,
+so it must be `/crafty/.venv/bin/python`, not the system `python3`; and this is a **mutating operator
+action**, not an agent action.
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+ 'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- /crafty/.venv/bin/python -c "
+import os, secrets, sqlite3, string, datetime
+from argon2 import PasswordHasher
+pw  = \"\".join(secrets.choice(string.ascii_letters + string.digits) for _ in range(24))
+now = datetime.datetime.now()
+c   = sqlite3.connect(\"/crafty/app/config/db/crafty.sqlite\", timeout=30)
+cur = c.execute(
+    \"UPDATE users SET password=?, last_update=?, valid_tokens_from=? WHERE username=?\",
+    (PasswordHasher().hash(pw), str(now),
+     str(datetime.datetime.now(datetime.timezone.utc)), \"admin\"),
+)
+c.commit()
+print(\"rows updated:\", cur.rowcount)
+out = \"/crafty/app/config/reset-creds.txt\"
+open(out, \"w\").write(pw + chr(10))
+os.chmod(out, 0o600)
+print(\"new password written to\", out)
+"'
+```
+
+Expect `rows updated: 1`. Then read it, sign in, rotate in the UI, and delete the file:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- cat /crafty/app/config/reset-creds.txt'
+# after signing in and rotating:
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- rm -f /crafty/app/config/reset-creds.txt'
+```
+
+`rows updated: 0` means the username is not `admin` — list the accounts (names only, never the
+`password` column) and retry against the right one:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+ 'kubectl --context k3s-prod -n crafty exec deploy/crafty -c crafty -- python3 -c "
+import sqlite3
+c=sqlite3.connect(\"file:/crafty/app/config/db/crafty.sqlite?mode=ro\",uri=True)
+print([r[0] for r in c.execute(\"SELECT username FROM users\")])
+"'
+```
+
+`database is locked` means Crafty held the write lock; the `timeout=30` usually rides it out, otherwise
+retry. The panel picks the new hash up on the next login attempt — no restart needed, and a restart
+costs a Minecraft-server stop (`strategy: Recreate` on an RWO volume), so do not restart reflexively.
+
+## When to escalate or stop
+
+- **Stop** as soon as you can sign in, and **rotate immediately** — do not leave the generated
+  bootstrap password in place as the standing admin credential, and do not leave `reset-creds.txt` on
+  the PVC.
+- **Do not `kubectl edit`/`patch`/`apply`** the Deployment to work around a login problem. Flux
+  reconciles `main`; hand-applied changes are drift. Manifest changes go through a branch and a PR.
+- **Do not delete `crafty.sqlite`** to force a "fresh install" and a new generated password. It holds
+  every server definition — jar, ports, heap, `auto_start` — and losing it means recreating each server
+  by hand, re-accepting the Mojang EULA, and re-`PATCH`ing `auto_start`. The worlds under
+  `/crafty/servers/<uuid>/` are not in git and not recreatable. The reset above is always cheaper.
+- **Escalate to the Keycloak side** (`runbooks/keycloak-down.md`,
+  `runbooks/keycloak-client-secrets.md`) if there is no `AuthSuccess` in the sidecar log at all — that
+  is an IdP or client-secret problem and no Crafty password will help.
+- **If `default-creds.txt` is absent on a pod whose DB says the password was never changed**, stop and
+  hand off: that combination should not occur, and it suggests the `config` subPath is not the volume
+  you think it is. Record what you measured rather than guessing.

runbooks/kustomization.yaml

@@ -60,6 +60,9 @@ configMapGenerator:
       - cnpg-exporter=cnpg-exporter.md
       - cnpg-primary-spread=cnpg-primary-spread.md
       - cnpg-wal-growth=cnpg-wal-growth.md
+      # Operator procedure, not alert-linked (Crafty's local login is a second factor behind
+      # the Keycloak gate and its bootstrap password lives only on the PVC).
+      - crafty-admin-credential=crafty-admin-credential.md
       - denied-flows=denied-flows.md
       - flux-reconcile=flux-reconcile.md
       - flux-suspended=flux-suspended.md