fix(crafty): the Keycloak gate works -- Crafty's own login was the wall

Reported: "crafty does not sign in after signing into keycloak, and there is
no way to set the service up without credentials."

MEASURED, not inferred. The sidecar log settles which layer refused:

  GET /                        -> 302 to Keycloak
  [AuthSuccess] ... groups:[platform-admins]        <- oauth2-proxy is FINE
  GET /oauth2/callback         -> 302
  GET /login?next=/panel/dashboard -> 200           <- Crafty's OWN login page
  POST /api/v2/auth/login/     -> 401               <- Crafty refused

So the Keycloak gate was never broken. Crafty 4 has no OIDC and cannot consume
the sidecar's identity headers, so it shows its own local login to an already
authenticated user. That second factor is intended. What was NOT intended is
that nothing documented where its password comes from, which is exactly why
setup was impossible: Crafty generates it ONCE on first boot (users table
empty -> DatabaseBuilder.default_settings()) into /crafty/app/config/
default-creds.txt, mode 0600, on the PVC's `config` subPath. Never in git;
a fresh PVC generates a new one. The README asserted the double login was
"expected, not a bug" and then left the reader with no way forward.

Verified in-cluster that the existing credential is still usable, so the
service can be set up without touching the DB: users.last_update equals
users.created for `admin` (09/05/2026, 08:10:04), i.e. the password has never
been changed, and admin's last_login is 09/14 from 127.0.0.1 -- the API login
that created the "homelab" server. default-creds.txt is therefore live.

ALSO FIXED -- a real, separate defect found while proving the above, which
produces the reported symptom intermittently and would have been blamed on
Keycloak forever: this sidecar and the SHARED forward-auth oauth2-proxy both
emitted oauth2-proxy's DEFAULT cookie names (_oauth2_proxy,
_oauth2_proxy_csrf) scoped to .home.keithjackson.dev, while holding DIFFERENT
cookie-secrets (compared the two live Secrets: they differ, so neither can
decrypt the other's cookie). One cookie pair, two proxies: whichever
authenticated last clobbered the other's session; the loser logged "No valid
authentication in request", bounced to Keycloak, and -- SSO session still live
-- was redirected straight back, clobbering it again. Fixed on this side only
(renaming the shared proxy's cookie would sign out every other gated app):
a unique --cookie-name plus a host-scoped --cookie-domain, since a per-app
reverse proxy has no reason to broadcast its session zone-wide.
--whitelist-domain stays zone-wide: it governs redirect targets, not cookies.

Two traps measured rather than assumed, both now documented:
  - The 3-strikes cooldown CANNOT fire behind this sidecar. Crafty records
    attempts under get_remote_ip() (honours X-Forwarded-For -- the live
    tracker key is the chain "192.168.2.132, 10.0.0.123") but checks the
    cooldown under request.remote_ip, which is always 127.0.0.1 here. The
    halves key differently, so get_recent_attempts() always reads empty.
    Good (no lockout risk) and bad (that throttle is not a control we have).
  - Crafty's `set_passwd` console command is unreachable in k8s: it calls
    getpass.getpass() and needs a TTY; the container runs `main.py -d -i`.
    Recovery has to go through the DB, and argon2 is installed ONLY in
    /crafty/.venv/bin/python -- the system python3 lacks it.

Deliberately NOT done: an initContainer seeding default.json from a SOPS
Secret to make a rebuilt Crafty's password deterministic. Crafty does support
that hook (find_default_password reads app/config/default.json, min 8 chars),
but the image entrypoint pulls its default config ONLY when app/config is
EMPTY -- so writing a file there pre-boot suppresses the whole config
bootstrap. Buying reproducibility means co-owning image-internal bootstrap
logic that a 4.11 bump could silently change, to replace a one-command
retrieval. Recorded in DECISIONS-LOG as considered and rejected.

New runbook runbooks/crafty-admin-credential.md: tell the two auth layers
apart from the sidecar log, retrieve and rotate the bootstrap credential,
and reset a lost one via argon2 -- generating the password in-pod into a 0600
file rather than passing it in, because dev.sh does not forward stdin (a
`kubectl exec -i` through it hangs; verified). Registered in the configMapGenerator
so the URL actually resolves, not just the file exists.

Gate: toolbox ./ci/gate.sh -- lint/l0/render-golden/l1 green, spec-immutability
green. rc 3 overall: gitea-settings SKIPPED (no SETTINGS_API_TOKEN) and L2/L3
NOT RUN (CLUSTER_AVAILABLE unset). No runtime browser round-trip performed --
the sign-in fix is UNVERIFIED end-to-end and needs the human test in the README.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

platform/crafty/README.md

@@ -49,9 +49,49 @@ Crafty's UI is gated by a **dedicated oauth2-proxy sidecar in the Crafty pod** (
 `crafty.home.keithjackson.dev`). The `crafty` Service targets the sidecar, never Crafty's `:8443`, so
 the gate is **in the data path** (fail-closed: no valid `platform-admins` session ⇒ no upstream).
 
-> **Note:** Crafty ALSO has its own local admin login (it has no OIDC). After oauth2-proxy admits the
-> user, they still see Crafty's own login page — a second, defense-in-depth factor. The operator logs
-> in with Crafty's local admin credentials (set on first boot). This is expected, not a bug.
+#### Cookie isolation (why the session used to vanish)
+
+This sidecar and the **shared** forward-auth oauth2-proxy are independent instances with **different
+`cookie-secret` values** (measured in-cluster — neither can decrypt the other's cookie). Both used to
+emit oauth2-proxy's *default* cookie names (`_oauth2_proxy`, `_oauth2_proxy_csrf`) scoped to
+`.home.keithjackson.dev`, so the browser held **one** cookie pair for **both**: whichever proxy
+authenticated last clobbered the other's session, and the loser logged
+`No valid authentication in request`, bounced to Keycloak, and — because the Keycloak SSO session was
+still live — was redirected straight back, clobbering the cookie again. That mutual eviction presents
+exactly as *"I signed in to Keycloak but I'm still not signed in."*
+
+Fixed on this side only (renaming the shared proxy's cookie would sign out every other gated app):
+`--cookie-name=_oauth2_proxy_crafty` and a **host-scoped** `--cookie-domain=crafty.home.keithjackson.dev`.
+`--whitelist-domain` stays zone-wide — it governs post-login *redirect* targets, not cookie scope.
+
+#### Crafty's own local login — this is where setup actually blocks
+
+Crafty has **no OIDC**, so after oauth2-proxy admits the user they still meet Crafty's own
+username/password page. That second factor is intended. What was **not** intended — and what made the
+service impossible to set up — is that **nothing documented where that password comes from**.
+
+Crafty generates it **once, on first boot only**, when the users table is empty
+(`main.py` `FRESH_INSTALL` → `DatabaseBuilder.default_settings()`), and writes it to
+**`/crafty/app/config/default-creds.txt`** (mode `0600`, on the `config` subPath of the PVC). It is
+**never** in git and **cannot** be recovered from git: a rebuild onto a fresh PVC generates a *new*
+random password. Submitting anything else yields `POST /api/v2/auth/login/` → **401**, which is exactly
+what a browser sees with no hint as to why.
+
+**→ Retrieval, rotation, and lost-password recovery: [`../../runbooks/crafty-admin-credential.md`](../../runbooks/crafty-admin-credential.md).**
+
+Two traps that runbook covers:
+
+- Crafty's `set_passwd` console command is **unreachable** in Kubernetes — it calls
+  `getpass.getpass()` and so needs a TTY, while the container runs `main.py -d -i` (daemon).
+  Recovery therefore has to go through the DB with Crafty's own argon2 hasher.
+- Crafty's own 3-strikes brute-force cooldown is **inert behind this sidecar**, verified live. Crafty
+  *records* failed attempts under `get_remote_ip()` (which honours `X-Forwarded-For` — the live
+  tracker key is the chain `"192.168.2.132, 10.0.0.123"`) but *checks* for a cooldown under
+  `self.request.remote_ip`, which behind the sidecar is always `127.0.0.1`. The two halves key
+  differently, so `get_recent_attempts()` always reads an empty list and `max_login_attempts: 3`
+  never fires. Good news for the operator (**repeated guesses cannot lock you out**), but it means
+  the only thing actually limiting who can reach the login form is the oauth2-proxy
+  `platform-admins` gate — worth knowing before anyone treats that cooldown as a control.
 
 ### Keycloak client
 

platform/crafty/deployment.yaml

@@ -150,7 +150,27 @@ spec:
             # Single provider => 302 straight to Keycloak, no oauth2-proxy interstitial button.
             - --skip-provider-button=true
             - --redirect-url=https://crafty.${BASE_DOMAIN}/oauth2/callback
-            - --cookie-domain=.${BASE_DOMAIN}
+            # ── COOKIE ISOLATION (bug fix 2026-09-29) ────────────────────────────────────────────
+            # This sidecar and the SHARED forward-auth oauth2-proxy (platform/oauth2-proxy) are
+            # INDEPENDENT instances holding DIFFERENT cookie-secrets (verified in-cluster: the two
+            # `cookie-secret` values differ, so neither can decrypt the other's cookie). Both used
+            # to emit oauth2-proxy's DEFAULT cookie names (`_oauth2_proxy` and `_oauth2_proxy_csrf`)
+            # scoped to `.${BASE_DOMAIN}`, so the browser kept ONE pair of cookies for BOTH proxies:
+            # whichever authenticated last overwrote the other's session, and the loser then logged
+            # "No valid authentication in request" and bounced to Keycloak -- which, thanks to the
+            # live Keycloak SSO session, silently redirected back and overwrote the cookie again.
+            # That mutual eviction presents EXACTLY as the reported symptom: "I signed in to
+            # Keycloak but I am still not signed in to the app."
+            # Two independent fixes, both applied here (only this side changes -- renaming the
+            # SHARED proxy's cookie would sign every other gated app out):
+            #   1. --cookie-name: a unique name, so the two sessions cannot alias at all.
+            #   2. --cookie-domain: HOST-scoped. A per-app reverse proxy has no reason to broadcast
+            #      its session to every other host in the zone; the shared proxy keeps the wide
+            #      scope because cross-host forward-auth is precisely its job.
+            # --whitelist-domain deliberately stays zone-wide: it governs permitted post-login
+            # REDIRECT targets, not cookie scope.
+            - --cookie-name=_oauth2_proxy_crafty
+            - --cookie-domain=crafty.${BASE_DOMAIN}
             - --whitelist-domain=.${BASE_DOMAIN}
             - --cookie-secure=true
             - --pass-access-token=true