fix(crafty): the Keycloak gate works — Crafty's own login was the wall #22

2026-09-30T00:36:11Z

gitea-admin commented

2026-09-30 00:36:11 +00:00

Reworked 2026-09-29: the handoff moved out of `DECISIONS-LOG.md`

This PR was mergeable: false, and the only conflicting path was DECISIONS-LOG.md. Nothing
was wrong with the change itself. DECISIONS-LOG.md was an append-only single file and every branch
appended to its tail, so any two concurrent branches collided on the same last hunk. Measured
against the live Gitea on 2026-09-29, four of the five open PRs were unmergeable and this file was
the sole conflicting path in all four.

So the branch was rebased onto main and its DECISIONS-LOG.md append dropped entirely — this
branch no longer touches that file at all. The handoff prose is unchanged in substance and now lives
in its own file:

decisions/2026-09-29-crafty-oauth2-proxy-signin.md

That is the layout introduced by #24 (chore/decisions-log-conflict-free): one file per handoff,
DECISIONS-LOG.md frozen, enforced by a new gate stage ci/decisions-log.sh. #24 merged while this
rework was in progress, so the branch is rebased onto the post-#24 main (c59fe9f) and that stage
runs in-tree and passes — [ ok ] stage decisions-log: pass.

This PR depends on nothing and blocks nothing. git merge-tree --write-tree origin/main HEAD
reports no conflict and the Gitea API reports mergeable: true. Because it does not touch
DECISIONS-LOG.md, it cannot invalidate #24's decisions/ARCHIVE.sha256 pin.

No change to platform/crafty/** or runbooks/** was altered in the rebase.

The report, and what actually refused

"crafty utilizing the keycloak oauth2-proxy does not sign in to the service after signing into keycloak. There is no way to setup the service without credentials."

The sidecar's own access log settles which layer refused — and it is not the one the report names:

GET /                            -> 302 to Keycloak
[AuthSuccess] ... groups:[platform-admins]        <- oauth2-proxy did its job
GET /oauth2/callback             -> 302
GET /panel/dashboard             -> 302
GET /login?next=/panel/dashboard -> 200           <- CRAFTY's own login page
POST /api/v2/auth/login/         -> 401           <- CRAFTY refused

The Keycloak gate was never broken. Crafty 4 has no OIDC and cannot consume the sidecar's identity headers, so it shows its own local login to an already-authenticated user. That second factor is intended. What was not intended — and what made setup genuinely impossible — is that nothing documented where that password comes from.

Crafty generates it once, on first boot only (users table empty → DatabaseBuilder.default_settings()) into /crafty/app/config/default-creds.txt, mode 0600, on the PVC's config subPath. Never in git; a fresh PVC generates a new one. The README asserted the double login was "expected, not a bug" and then left the reader with no way forward — true and useless in the same sentence.

Good news: no DB surgery is needed to get in. Verified in-cluster that users.last_update equals users.created for admin (09/05/2026, 08:10:04) — the password has never been changed — and admin's last_login is 09/14 from 127.0.0.1, the API login that created the "homelab" server. default-creds.txt is live. Retrieval is one command, now in the runbook.

A second, real defect found while proving the first

Both oauth2-proxy instances emitted oauth2-proxy's default cookie names (_oauth2_proxy, _oauth2_proxy_csrf) scoped to .home.keithjackson.dev, while holding different cookie-secrets (compared the two live Secrets — they differ, so neither can decrypt the other's cookie). One cookie pair, two proxies: whichever authenticated last clobbered the other's session; the loser logged No valid authentication in request, bounced to Keycloak, and — SSO session still live — was redirected straight back, clobbering it again.

That mutual eviction presents exactly as the reported symptom, intermittently, and would have been blamed on Keycloak indefinitely.

Fixed on the crafty side only (renaming the shared proxy's cookie would sign out every other gated app): a unique --cookie-name plus a host-scoped --cookie-domain. --whitelist-domain stays zone-wide — it governs post-login redirect targets, not cookie scope.

This half is a latent-bug fix with no observed failure to point at — the trace above shows a working session, because that browser had not visited another gated app first.

Traps measured rather than assumed

The 3-strikes cooldown cannot fire behind this sidecar. Crafty records attempts under get_remote_ip() (honours X-Forwarded-For — live tracker key is the chain "192.168.2.132, 10.0.0.123", 1 attempt: the reported 401) but checks the cooldown under self.request.remote_ip, always 127.0.0.1 here. The halves key differently, so max_login_attempts: 3 never fires. Good (repeated guesses cannot lock you out) and bad (that throttle is not a control we have). I had written the opposite into the README from the config value alone before measuring it.
set_passwd is unreachable in k8s — it calls getpass.getpass() and needs a TTY; the container runs main.py -d -i.
argon2 is installed only in /crafty/.venv/bin/python — the system python3 cannot import it, so the obvious reset one-liner fails.
dev.sh does not forward stdin — a kubectl exec -i through it hangs forever (provoked, not assumed). So the reset generates its password in-pod into a 0600 file rather than passing a secret through argv, stdin or history.

Considered and rejected: seeding `default.json` from a SOPS Secret

Crafty ships the hook (find_default_password() reads app/config/default.json, min 8 chars), so an initContainer could make a rebuilt Crafty's password deterministic. Rejected: the entrypoint pulls its default config only when app/config is empty —

if [ ! "$(ls -A --ignore=.gitkeep ./app/config)" ]; then cp -r ./app/config_original/* ./app/config/

— so writing a file there pre-boot suppresses the entire config bootstrap. Doing it safely means co-owning image-internal bootstrap logic that a 4.11 bump could change silently, to replace a one-command retrieval.

Changes

File	What
`platform/crafty/deployment.yaml`	unique `--cookie-name`, host-scoped `--cookie-domain`
`platform/crafty/README.md`	replaces the "expected, not a bug" dead end with the real credential lifecycle + cookie-isolation rationale
`runbooks/crafty-admin-credential.md`	new — tell the two auth layers apart, retrieve/rotate, argon2 reset for a lost password
`runbooks/kustomization.yaml`	generator entry, so the URL resolves and not just the file exists
`runbooks/README.md`	index entry under "Operator procedures"
`decisions/2026-09-29-crafty-oauth2-proxy-signin.md`	new — the handoff, as its own file (see Reworked above)

Gate

./infra/devcontainer/dev.sh run bash -c './ci/gate.sh' → rc 3 (UNDETERMINED), not 0. Re-run on
the rebased branch (01679a0), measured exit code 3.

Green: spec-immutability, decisions-log, lint, l0, render-golden, l1
Skipped: gitea-settings (no SETTINGS_API_TOKEN), L2/L3 conformance (CLUSTER_AVAILABLE unset)
Touched requirements R-DNS-01 R-NET-01 R-OBS-04 R-PLAT-02 R-STATE-01 are not certified

Nothing failed; nothing runtime was proven either.

What is NOT verified

No browser round-trip. The sign-in fix is UNVERIFIED end-to-end. The cookie change is reasoned from measured facts and renders correctly, but nobody has signed in through it. Human test is in the README.
Nothing was applied — no kubectl apply/edit/patch, no flux reconcile. The running pod is still the pre-fix spec (crafty-64f7b6d78-pkpgp, 2/2, 15d, 0 restarts).
Merging restarts the pod — strategy: Recreate on an RWO volume, so the "homelab" Minecraft server stops and restarts. Pick the moment.
The admin password is still the value generated 2026-09-05 and is still the standing admin credential. Rotating it is an operator action and remains open.

🤖 Generated with Claude Code

## Reworked 2026-09-29: the handoff moved out of `DECISIONS-LOG.md` **This PR was `mergeable: false`, and the *only* conflicting path was `DECISIONS-LOG.md`.** Nothing was wrong with the change itself. `DECISIONS-LOG.md` was an append-only single file and every branch appended to its **tail**, so any two concurrent branches collided on the same last hunk. Measured against the live Gitea on 2026-09-29, four of the five open PRs were unmergeable and this file was the sole conflicting path in all four. So the branch was rebased onto `main` and its `DECISIONS-LOG.md` append **dropped entirely** — this branch no longer touches that file at all. The handoff prose is unchanged in substance and now lives in its own file: decisions/2026-09-29-crafty-oauth2-proxy-signin.md That is the layout introduced by **#24** (`chore/decisions-log-conflict-free`): one file per handoff, `DECISIONS-LOG.md` frozen, enforced by a new gate stage `ci/decisions-log.sh`. #24 merged while this rework was in progress, so the branch is rebased onto the post-#24 `main` (`c59fe9f`) and that stage **runs in-tree and passes** — `[ ok ] stage decisions-log: pass`. **This PR depends on nothing and blocks nothing.** `git merge-tree --write-tree origin/main HEAD` reports no conflict and the Gitea API reports `mergeable: true`. Because it does not touch `DECISIONS-LOG.md`, it cannot invalidate #24's `decisions/ARCHIVE.sha256` pin. No change to `platform/crafty/**` or `runbooks/**` was altered in the rebase. --- ## The report, and what actually refused > "crafty utilizing the keycloak oauth2-proxy does not sign in to the service after signing into keycloak. There is no way to setup the service without credentials." The sidecar's own access log settles which layer refused — and it is not the one the report names: ``` GET / -> 302 to Keycloak [AuthSuccess] ... groups:[platform-admins] <- oauth2-proxy did its job GET /oauth2/callback -> 302 GET /panel/dashboard -> 302 GET /login?next=/panel/dashboard -> 200 <- CRAFTY's own login page POST /api/v2/auth/login/ -> 401 <- CRAFTY refused ``` **The Keycloak gate was never broken.** Crafty 4 has no OIDC and cannot consume the sidecar's identity headers, so it shows its own local login to an already-authenticated user. That second factor is intended. What was *not* intended — and what made setup genuinely impossible — is that **nothing documented where that password comes from**. Crafty generates it **once, on first boot only** (users table empty → `DatabaseBuilder.default_settings()`) into `/crafty/app/config/default-creds.txt`, mode `0600`, on the PVC's `config` subPath. Never in git; a fresh PVC generates a new one. The README asserted the double login was *"expected, not a bug"* and then left the reader with no way forward — true and useless in the same sentence. **Good news: no DB surgery is needed to get in.** Verified in-cluster that `users.last_update` **equals** `users.created` for `admin` (`09/05/2026, 08:10:04`) — the password has never been changed — and admin's `last_login` is 09/14 from `127.0.0.1`, the API login that created the "homelab" server. `default-creds.txt` is live. Retrieval is one command, now in the runbook. ## A second, real defect found while proving the first Both oauth2-proxy instances emitted oauth2-proxy's **default** cookie names (`_oauth2_proxy`, `_oauth2_proxy_csrf`) scoped to `.home.keithjackson.dev`, while holding **different** cookie-secrets (compared the two live Secrets — they differ, so neither can decrypt the other's cookie). One cookie pair, two proxies: whichever authenticated last clobbered the other's session; the loser logged `No valid authentication in request`, bounced to Keycloak, and — SSO session still live — was redirected straight back, clobbering it again. **That mutual eviction presents exactly as the reported symptom**, intermittently, and would have been blamed on Keycloak indefinitely. Fixed on the crafty side only (renaming the shared proxy's cookie would sign out every other gated app): a unique `--cookie-name` plus a **host-scoped** `--cookie-domain`. `--whitelist-domain` stays zone-wide — it governs post-login *redirect* targets, not cookie scope. > This half is a **latent-bug fix with no observed failure to point at** — the trace above shows a working session, because that browser had not visited another gated app first. ## Traps measured rather than assumed - **The 3-strikes cooldown cannot fire behind this sidecar.** Crafty *records* attempts under `get_remote_ip()` (honours `X-Forwarded-For` — live tracker key is the chain `"192.168.2.132, 10.0.0.123"`, 1 attempt: the reported 401) but *checks* the cooldown under `self.request.remote_ip`, always `127.0.0.1` here. The halves key differently, so `max_login_attempts: 3` never fires. Good (**repeated guesses cannot lock you out**) and bad (that throttle is not a control we have). I had written the opposite into the README from the config value alone before measuring it. - **`set_passwd` is unreachable in k8s** — it calls `getpass.getpass()` and needs a TTY; the container runs `main.py -d -i`. - **argon2 is installed only in `/crafty/.venv/bin/python`** — the system `python3` cannot import it, so the obvious reset one-liner fails. - **`dev.sh` does not forward stdin** — a `kubectl exec -i` through it hangs forever (provoked, not assumed). So the reset generates its password **in-pod** into a `0600` file rather than passing a secret through argv, stdin or history. ## Considered and rejected: seeding `default.json` from a SOPS Secret Crafty ships the hook (`find_default_password()` reads `app/config/default.json`, min 8 chars), so an initContainer could make a rebuilt Crafty's password deterministic. **Rejected:** the entrypoint pulls its default config *only when `app/config` is empty* — ```sh if [ ! "$(ls -A --ignore=.gitkeep ./app/config)" ]; then cp -r ./app/config_original/* ./app/config/ ``` — so writing a file there pre-boot **suppresses the entire config bootstrap**. Doing it safely means co-owning image-internal bootstrap logic that a 4.11 bump could change silently, to replace a one-command retrieval. ## Changes | File | What | |---|---| | `platform/crafty/deployment.yaml` | unique `--cookie-name`, host-scoped `--cookie-domain` | | `platform/crafty/README.md` | replaces the "expected, not a bug" dead end with the real credential lifecycle + cookie-isolation rationale | | `runbooks/crafty-admin-credential.md` | **new** — tell the two auth layers apart, retrieve/rotate, argon2 reset for a lost password | | `runbooks/kustomization.yaml` | generator entry, so the URL resolves and not just the file exists | | `runbooks/README.md` | index entry under "Operator procedures" | | `decisions/2026-09-29-crafty-oauth2-proxy-signin.md` | **new** — the handoff, as its own file (see *Reworked* above) | ## Gate `./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'` → **rc 3 (UNDETERMINED), not 0.** Re-run on the rebased branch (`01679a0`), measured exit code `3`. - **Green:** `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` - **Skipped:** `gitea-settings` (no `SETTINGS_API_TOKEN`), **L2/L3 conformance** (`CLUSTER_AVAILABLE` unset) - Touched requirements `R-DNS-01 R-NET-01 R-OBS-04 R-PLAT-02 R-STATE-01` are **not certified** Nothing failed; nothing runtime was proven either. ## What is NOT verified - **No browser round-trip.** The sign-in fix is **UNVERIFIED end-to-end**. The cookie change is reasoned from measured facts and renders correctly, but nobody has signed in through it. Human test is in the README. - **Nothing was applied** — no `kubectl apply/edit/patch`, no `flux reconcile`. The running pod is still the pre-fix spec (`crafty-64f7b6d78-pkpgp`, 2/2, 15d, 0 restarts). - **Merging restarts the pod** — `strategy: Recreate` on an RWO volume, so the "homelab" Minecraft server stops and restarts. Pick the moment. - **The admin password is still the value generated 2026-09-05** and is still the standing admin credential. Rotating it is an operator action and remains open. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 2 commits 2026-09-30 00:36:11 +00:00

fix(crafty): the Keycloak gate works -- Crafty's own login was the wall dd1d5a6d0a

Reported: "crafty does not sign in after signing into keycloak, and there is
no way to set the service up without credentials."

MEASURED, not inferred. The sidecar log settles which layer refused:

  GET /                        -> 302 to Keycloak
  [AuthSuccess] ... groups:[platform-admins]        <- oauth2-proxy is FINE
  GET /oauth2/callback         -> 302
  GET /login?next=/panel/dashboard -> 200           <- Crafty's OWN login page
  POST /api/v2/auth/login/     -> 401               <- Crafty refused

So the Keycloak gate was never broken. Crafty 4 has no OIDC and cannot consume
the sidecar's identity headers, so it shows its own local login to an already
authenticated user. That second factor is intended. What was NOT intended is
that nothing documented where its password comes from, which is exactly why
setup was impossible: Crafty generates it ONCE on first boot (users table
empty -> DatabaseBuilder.default_settings()) into /crafty/app/config/
default-creds.txt, mode 0600, on the PVC's `config` subPath. Never in git;
a fresh PVC generates a new one. The README asserted the double login was
"expected, not a bug" and then left the reader with no way forward.

Verified in-cluster that the existing credential is still usable, so the
service can be set up without touching the DB: users.last_update equals
users.created for `admin` (09/05/2026, 08:10:04), i.e. the password has never
been changed, and admin's last_login is 09/14 from 127.0.0.1 -- the API login
that created the "homelab" server. default-creds.txt is therefore live.

ALSO FIXED -- a real, separate defect found while proving the above, which
produces the reported symptom intermittently and would have been blamed on
Keycloak forever: this sidecar and the SHARED forward-auth oauth2-proxy both
emitted oauth2-proxy's DEFAULT cookie names (_oauth2_proxy,
_oauth2_proxy_csrf) scoped to .home.keithjackson.dev, while holding DIFFERENT
cookie-secrets (compared the two live Secrets: they differ, so neither can
decrypt the other's cookie). One cookie pair, two proxies: whichever
authenticated last clobbered the other's session; the loser logged "No valid
authentication in request", bounced to Keycloak, and -- SSO session still live
-- was redirected straight back, clobbering it again. Fixed on this side only
(renaming the shared proxy's cookie would sign out every other gated app):
a unique --cookie-name plus a host-scoped --cookie-domain, since a per-app
reverse proxy has no reason to broadcast its session zone-wide.
--whitelist-domain stays zone-wide: it governs redirect targets, not cookies.

Two traps measured rather than assumed, both now documented:
  - The 3-strikes cooldown CANNOT fire behind this sidecar. Crafty records
    attempts under get_remote_ip() (honours X-Forwarded-For -- the live
    tracker key is the chain "192.168.2.132, 10.0.0.123") but checks the
    cooldown under request.remote_ip, which is always 127.0.0.1 here. The
    halves key differently, so get_recent_attempts() always reads empty.
    Good (no lockout risk) and bad (that throttle is not a control we have).
  - Crafty's `set_passwd` console command is unreachable in k8s: it calls
    getpass.getpass() and needs a TTY; the container runs `main.py -d -i`.
    Recovery has to go through the DB, and argon2 is installed ONLY in
    /crafty/.venv/bin/python -- the system python3 lacks it.

Deliberately NOT done: an initContainer seeding default.json from a SOPS
Secret to make a rebuilt Crafty's password deterministic. Crafty does support
that hook (find_default_password reads app/config/default.json, min 8 chars),
but the image entrypoint pulls its default config ONLY when app/config is
EMPTY -- so writing a file there pre-boot suppresses the whole config
bootstrap. Buying reproducibility means co-owning image-internal bootstrap
logic that a 4.11 bump could silently change, to replace a one-command
retrieval. Recorded in DECISIONS-LOG as considered and rejected.

New runbook runbooks/crafty-admin-credential.md: tell the two auth layers
apart from the sidecar log, retrieve and rotate the bootstrap credential,
and reset a lost one via argon2 -- generating the password in-pod into a 0600
file rather than passing it in, because dev.sh does not forward stdin (a
`kubectl exec -i` through it hangs; verified). Registered in the configMapGenerator
so the URL actually resolves, not just the file exists.

Gate: toolbox ./ci/gate.sh -- lint/l0/render-golden/l1 green, spec-immutability
green. rc 3 overall: gitea-settings SKIPPED (no SETTINGS_API_TOKEN) and L2/L3
NOT RUN (CLUSTER_AVAILABLE unset). No runtime browser round-trip performed --
the sign-in fix is UNVERIFIED end-to-end and needs the human test in the README.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

docs(decisions-log): Crafty sign-in handoff — the gate was fine, the local login was the wall

gate / gate (pull_request) Successful in 1m36s

Details

gate / conformance (pull_request) Has been skipped

Details

854d8668bd

Records what refused (Crafty's own 401, not oauth2-proxy), that the existing
bootstrap credential is still live (users.last_update == users.created), the
separate cookie-name collision fixed alongside, the four traps measured rather
than assumed, the default.json seeding approach considered and rejected with
the entrypoint `ls -A` landmine that kills it, and — plainly — that no browser
round-trip was performed and the gate was rc 3, not green.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

jakman85 approved these changes

2026-09-30 03:20:50 +00:00

Dismissed

gitea-admin referenced this issue from a commit

2026-09-30 03:37:39 +00:00

chore(log): the handoff log was serialising every parallel branch

gitea-admin referenced this pull request

2026-09-30 03:38:12 +00:00

chore(log): the handoff log was serialising every parallel branch #24

gitea-admin referenced this issue from a commit

2026-09-30 03:47:20 +00:00

docs(decisions): Crafty sign-in handoff — the gate was fine, the local login was the wall