4e1b169256b70f1ffbb39f4e252b1b7caccd285f
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
ea0c0632d2 |
fix(crafty): the Keycloak gate works -- Crafty's own login was the wall
Reported: "crafty does not sign in after signing into keycloak, and there is
no way to set the service up without credentials."
MEASURED, not inferred. The sidecar log settles which layer refused:
GET / -> 302 to Keycloak
[AuthSuccess] ... groups:[platform-admins] <- oauth2-proxy is FINE
GET /oauth2/callback -> 302
GET /login?next=/panel/dashboard -> 200 <- Crafty's OWN login page
POST /api/v2/auth/login/ -> 401 <- Crafty refused
So the Keycloak gate was never broken. Crafty 4 has no OIDC and cannot consume
the sidecar's identity headers, so it shows its own local login to an already
authenticated user. That second factor is intended. What was NOT intended is
that nothing documented where its password comes from, which is exactly why
setup was impossible: Crafty generates it ONCE on first boot (users table
empty -> DatabaseBuilder.default_settings()) into /crafty/app/config/
default-creds.txt, mode 0600, on the PVC's `config` subPath. Never in git;
a fresh PVC generates a new one. The README asserted the double login was
"expected, not a bug" and then left the reader with no way forward.
Verified in-cluster that the existing credential is still usable, so the
service can be set up without touching the DB: users.last_update equals
users.created for `admin` (09/05/2026, 08:10:04), i.e. the password has never
been changed, and admin's last_login is 09/14 from 127.0.0.1 -- the API login
that created the "homelab" server. default-creds.txt is therefore live.
ALSO FIXED -- a real, separate defect found while proving the above, which
produces the reported symptom intermittently and would have been blamed on
Keycloak forever: this sidecar and the SHARED forward-auth oauth2-proxy both
emitted oauth2-proxy's DEFAULT cookie names (_oauth2_proxy,
_oauth2_proxy_csrf) scoped to .home.keithjackson.dev, while holding DIFFERENT
cookie-secrets (compared the two live Secrets: they differ, so neither can
decrypt the other's cookie). One cookie pair, two proxies: whichever
authenticated last clobbered the other's session; the loser logged "No valid
authentication in request", bounced to Keycloak, and -- SSO session still live
-- was redirected straight back, clobbering it again. Fixed on this side only
(renaming the shared proxy's cookie would sign out every other gated app):
a unique --cookie-name plus a host-scoped --cookie-domain, since a per-app
reverse proxy has no reason to broadcast its session zone-wide.
--whitelist-domain stays zone-wide: it governs redirect targets, not cookies.
Two traps measured rather than assumed, both now documented:
- The 3-strikes cooldown CANNOT fire behind this sidecar. Crafty records
attempts under get_remote_ip() (honours X-Forwarded-For -- the live
tracker key is the chain "192.168.2.132, 10.0.0.123") but checks the
cooldown under request.remote_ip, which is always 127.0.0.1 here. The
halves key differently, so get_recent_attempts() always reads empty.
Good (no lockout risk) and bad (that throttle is not a control we have).
- Crafty's `set_passwd` console command is unreachable in k8s: it calls
getpass.getpass() and needs a TTY; the container runs `main.py -d -i`.
Recovery has to go through the DB, and argon2 is installed ONLY in
/crafty/.venv/bin/python -- the system python3 lacks it.
Deliberately NOT done: an initContainer seeding default.json from a SOPS
Secret to make a rebuilt Crafty's password deterministic. Crafty does support
that hook (find_default_password reads app/config/default.json, min 8 chars),
but the image entrypoint pulls its default config ONLY when app/config is
EMPTY -- so writing a file there pre-boot suppresses the whole config
bootstrap. Buying reproducibility means co-owning image-internal bootstrap
logic that a 4.11 bump could silently change, to replace a one-command
retrieval. Recorded in DECISIONS-LOG as considered and rejected.
New runbook runbooks/crafty-admin-credential.md: tell the two auth layers
apart from the sidecar log, retrieve and rotate the bootstrap credential,
and reset a lost one via argon2 -- generating the password in-pod into a 0600
file rather than passing it in, because dev.sh does not forward stdin (a
`kubectl exec -i` through it hangs; verified). Registered in the configMapGenerator
so the URL actually resolves, not just the file exists.
Gate: toolbox ./ci/gate.sh -- lint/l0/render-golden/l1 green, spec-immutability
green. rc 3 overall: gitea-settings SKIPPED (no SETTINGS_API_TOKEN) and L2/L3
NOT RUN (CLUSTER_AVAILABLE unset). No runtime browser round-trip performed --
the sign-in fix is UNVERIFIED end-to-end and needs the human test in the README.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
c59a4b76bb |
docs(crafty): a player has joined -- the allowlist chain is closed, recommendation now PROVEN
Operator added one gamertag and the Bedrock client connected and joined. The three-pass investigation closes with a working server, no manifest change, no transport change, no network change. Moved out of UNVERIFIED: * A successful join -- OPERATOR-REPORTED (I did not watch it; the distinction is kept deliberately). Corroborated directly on disk, which is stronger than the report alone: allowlist.json went [] -> one entry, 75 bytes, and that entry's xuid is POPULATED (mtime 22:03:17). audit.log shows the operator wrote only the name (Edited file .../allowlist.json, 22:02:13). Vendor doc: xuid "will be populated when someone with a matching name connects". So the server wrote the XUID on connect, ~64s after the edit -- server-authored evidence of an XBL-authenticated join. * The reasoned mapping of "You are not invited" to the empty allowlist is CONFIRMED: one gamertag added, one successful join. allow-list=true with allowlist.json == [] was the wall, and nothing else was. * transport=raknet survives a restart: the fix involved one (pid 47006 -> 47875); afterwards server.properties mtime is still 21:33:31 with transport=raknet, UDP 4ABC bound, still NO TCP 4ABC, and a fresh pong from 192.168.2.22:19132 at 22:06. * The operator's actual route was file-edit + restart (route 2), not the console `allowlist add` I listed first. Both proven; console still preferred (no restart). README: join-gate section marked RESOLVED; decision heading and option 1 promoted from provisional to PROVEN; Bedrock human-test 4 rewritten as a 5-step verified connect procedure, each step with its distinct failure signature, stating plainly that the TRANSPORT TYPE ERROR banner and the "too few fields" warning BOTH appear on a fully working server -- that pair misled two consecutive sessions. NOT weakened, and sharpened where possible: * auto_start=0 / crash_detection=0 STILL 0, re-measured at 22:06 after the join and after the operator's 22:02:28 "modified the server" action, which did not change them. The failure mode is now worse, not better: "worked, then vanished". Merging fix/crafty-oauth2-proxy-signin is such a restart. * Silent-reset now precisely characterised: a RESTART preserves both files (observed) but leaves the server down (auto_start=0); a server RECREATE or reinstall silently resets server.properties to transport=nethernet AND allowlist.json to []. * L2/L3 conformance still uncertified; multi-player SNAT still untested (exactly one player has joined -- that is precisely the evidence that does not settle it); BDS console output still never seen (log_path is ''). No manifest touched, in any of the three passes. Read-only throughout. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
cd2617e5b3 |
docs(crafty): "You are not invited" is an empty allowlist -- and it proves RakNet works
New operator observation: a Bedrock client got "Unable to connect / You are not invited to play on this server". That was put to me as evidence that NetherNet signalling is working, that the refusal may have come from Microsoft rather than our server, and that BDS players therefore never traverse crafty-l4-udp. Measured read-only at 21:56, all three are wrong. allow-list=true (server.properties line 37) allowlist.json == [] (3 bytes, mtime 20:51, never edited since create) online-mode=true transport=raknet (line 71; file mtime STILL 21:33:31) UDP 19132 bound (/proc/net/udp 00000000:4ABC, pid 47006) TCP 19132 ABSENT (no 4ABC in /proc/net/tcp or tcp6) bedrock_server pid 47006, same process since 21:34, never restarted 1. transport is still raknet -- nobody reverted it, the binary did not rewrite it (mtime unchanged, audit.log records no later edit, same pid). NetherNet is NOT in use and cannot be what served this client. 2. NetherNet could not have served it in principle: it signals over TCP on server-port, and there is no TCP 19132 listener in the pod at all. 3. So the refusal came from OUR server, over RakNet, over 19132/udp, through the VIP. The 19132-19134/UDP exposure is the opposite of inert -- it carried both the earlier pong and this refusal. 4. Cause is certain and trivial: allow-list=true with an empty allowlist.json refuses every player. The shipped how_to says so, and it keys on Xbox Gamertag. The requested log correlation cannot be done from disk: K3s-Bedrock's log_path in crafty.sqlite is the empty string and Dedicated_Server.txt is 0 bytes -- Crafty persists no BDS stdout at all, the console is panel-UI-only. commander.log 21:39-21:56 holds only homelab crash-detection and raknet_ping lines. That silence is evidence about Crafty's logging, not about the refusal's origin. Adds a README section on the join authorization model: the measured table, the proof the refusal is ours, the "console is not on disk" caveat, gamertag/XUID identity, and three operator routes (console `allowlist add`, edit-then-`allowlist reload`, or allow-list=false -- discouraged, since the game port is deliberately un-gated so the allowlist is the only join control). Notes allowlist.json is PVC state that resets to [] on a server recreate. Options table and human test 4 now check the allowlist before blaming the network. Ranking unchanged at the top: keep transport=raknet, add gamertags. Still zero manifest change. NetherNet would ADD work (TCP 19132 in crafty-l4-tcp and in crafty-allow-ingress, plus server-udp-ports) and admit not one extra player. Read-only throughout: allowlist.json, server.properties and the DB were read; no console command issued, no kubectl mutation, no flux reconcile. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
d29a73cad8 |
docs(crafty): 19132/udp IS bound and answering -- the ERROR banner was advisory
The prior entry's operator fix (transport=raknet) was labelled REASONED, NOT
OBSERVED. The operator applied it, BDS 1.26.52 printed
"================ TRANSPORT TYPE ERROR ===================
NetherNet is the only supported transport type", and that was read as a refusal.
It was not a refusal. The process kept running and bound the RakNet sockets:
/proc/net/udp 00000000:4ABC and /proc/net/udp6 ...:4ABD, both owned by
bedrock_server pid 47006 (fd/25, fd/26). The earlier TCP bind on 4ABC is gone.
Crafty's raknet_ping warning changed kind at 21:34 -- last "Got timeout"
21:34:04,306, first "too few fields" 21:34:09,854 -- and per raknet_ping.py the
latter is reachable only after a 0x1C Unconnected Pong arrives with valid magic.
Proven end-to-end from off-cluster: a RakNet Unconnected Ping to the prod VIP
192.168.2.22:19132 returns a 139-byte 0x1c pong in ~12ms:
MCPE;Dedicated Server;2193;1.26.52;0;10;...;Bedrock level;Survival;1;19132;19133;0;0;0;
The vendor's own shipped bedrock_server_how_to.html documents raknet as a
supported value ("the server listens directly on server-port"), and RakNet is
still compiled in (RakNetConnector, createUniqueRakPeer, RakNetSocket2.cpp).
"Only supported transport type" is a support statement, not a capability one.
Also explains why nethernet could never have worked here: it signals over TCP on
server-port, and this repo exposes no TCP 19132 -- crafty-l4-tcp is 25565-25567
and crafty-allow-ingress allows 19132 UDP only. packet-statistics.txt for that
run: Total Received: 0, 0B over 2293s.
Decision: keep transport=raknet, NO manifest change. The 19132-19134/UDP
exposure is correct and live, not inert. Geyser+Floodgate documented as the
reserve migration if RakNet is ever really removed (also zero manifest change).
Other corrections: the "too few fields" warning is the success signature, not a
failure; enable-lan-visibility upgraded to OBSERVED (19133 answers with the same
server_guid, so slot 1 is NOT free -- previous text said it was unreachable);
Game VIP corrected from 192.168.1.55 (the base/dev value) to the live prod
192.168.2.22; noted transport=raknet is PVC state that silently regresses to
nethernet on a server recreate. K3s-Bedrock auto_start=0/crash_detection=0
re-confirmed from crafty.sqlite and still outstanding.
Read-only throughout: no kubectl mutation, no flux reconcile, no write inside
the container. server.properties was read, never touched.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
07f03536b3 |
docs(crafty): Bedrock 19132 refuses nothing -- BDS ships transport=nethernet
Reported as "users cannot connect to Bedrock on UDP 19132". Measured read-only on prod; the platform delivered the packets and the server never opened the socket. Server `K3s-Bedrock` (c5837301-..., type=minecraft-bedrock, server_port=19132) exists and `./bedrock_server` is alive, but /proc/net/udp and /proc/net/udp6 in the crafty container hold NO socket on 19132 (0x4ABC) in either family -- port 19132 is bound as TCP (/proc/net/tcp6 ...:4ABC state 0A). Cause, from the server's own server.properties: transport=nethernet, the value Microsoft's BDS archive ships. Its inline docs: `Allowed values: "raknet" or "nethernet"`, and against server-portv6, `This property is ignored when transport=nethernet`. NetherNet is the WebRTC/Xbox-signalled transport and opens no RakNet UDP listener. Crafty does not manage the property (no `transport` anywhere under /crafty/app/classes/) and its only Bedrock probe is raknet_ping.py, hence commander.log spamming "Got timeout while issuing bedrock ping to 127.0.0.1:19132" -- over loopback, which indicts the server, not the network. The path below it is sound and was measured as such: crafty-l4-udp live as externalTrafficPolicy: Cluster with the VIP in status.loadBalancer, a non-empty EndpointSlice, `cilium-dbg service list` showing 192.168.2.22:19132/UDP => 10.0.0.130:19132/UDP (active) on BOTH announcing nodes, the datapath policy map carrying `Allow Ingress reserved:world 19132/UDP` with 26 packets / 27660 bytes already counted through it, pihole answering crafty-mc -> 192.168.2.22, and Flux at main@sha1:20ea70d == the GitRepository artifact. So this is not a platform or manifest defect. No manifest is touched: adding ports or widening policy for this would be manufacturing work. The repo-side defect is documentation -- the README told the operator that creating a server on 19132 was sufficient. README: add "Bedrock on UDP 19132 -- the transport trap" (the measurement chain, and the raknet_ping warning as its signature) and "Bedrock realities that are NOT platform defects" (no LAN/Friends discovery behind a VIP, consoles cannot enter an address, enable-lan-visibility collides with slots 1-2, server-portv6 is not exposed). Correct three stale claims: "the operator has not created any servers yet" (two exist), the human test implying a started Bedrock server is reachable, and the single-server Runtime state section (now a two-row table recording K3s-Bedrock's auto_start=0 / crash_detection=0 -- the create-API trap, live again, so it will not survive a pod restart). Also measured, reported, and deliberately NOT changed: the shared VIP's two leases sit on different nodes (crafty-l4-tcp on k3s-prod-3, crafty-l4-udp on k3s-prod-1), so two nodes ARP-announce 192.168.2.22. Benign under `Cluster` -- both nodes carry the full active frontend set -- and the existing L0 check l2-announced-vips-serve-from-any-node already forbids the `Local` that would make it fatal. A "shared VIPs must co-locate" check would assert runtime lease placement a static check cannot see. Read-only throughout: no apply/edit/patch, no flux reconcile, no write inside the container. Operator actions (set transport=raknet and verify the UDP bind; PATCH auto_start; tell players to Add Server by hand) are in the DECISIONS-LOG entry, along with what is UNVERIFIED -- chiefly that `raknet` fixes it is reasoned from the shipped docs, not observed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
b4bc8e1cb9 |
feat(crafty): first Minecraft server — Paper 26.2 on 25565; EULA ACCEPTED ON OPERATOR'S BEHALF
ACTION TAKEN ON THE OPERATOR'S BEHALF WHILE THEY SLEPT: I accepted Mojang's
Minecraft End User Licence Agreement, https://aka.ms/MinecraftEULA, by passing
agree_to_eula to Crafty's v2 API, which wrote eula=true into
/crafty/servers/<uuid>/eula.txt. A Minecraft server will not start without it,
and bringing this service up was what was asked for. Please review. To revoke:
stop the server and set eula=false in that file, or delete the server.
Crafty was healthy but had never had a server created, so 192.168.2.22:25565
refused connections. Created "homelab" via the API (not UI automation):
Paper 26.2, TCP 25565, heap -Xms2000M -Xmx2000M, autostart on
uuid 13c7b85d-00bb-4a58-8c86-2433ca47891c
Paper over Vanilla for tick performance and plugins, with no cost to players
(vanilla clients connect unchanged). 26.2 is Mojang's latest.release, out
2026-06-16; the 26.3-rc-2 snapshot and 26.2-rc-2 were deliberately avoided.
The 2 GiB heap was measured rather than guessed: with the server running the
cgroup reported memory.current 2.86Gi against the 4Gi limit, so a 3 GiB heap
would have risked an OOMKill.
Proven playable, all real output: Crafty reports running with 'Done (12.737s)!';
Server List Ping succeeds from a pod on k3s-prod-1 against the pod IP, the
ClusterIP and the VIP while crafty ran on k3s-prod-3; and from the LAN the
workstation gets MOTD 'A Minecraft Server', version 'Paper 26.2', protocol 776,
players 0/20 over a full handshake plus ping/pong. Deleting the pod moved Crafty
to another node; it auto-started with no manual action and the LAN ping still
succeeded, re-confirming the externalTrafficPolicy: Cluster fix from
|
||
|
jakman85andClaude Opus 4.8
|
0437871807 |
Onboard Crafty Controller (Minecraft panel) with Keycloak-gated web UI
Add Crafty Controller 4 (registry.gitlab.com/crafty-controller/crafty-4:4.10.8) as a platform component: a web panel that runs Java + Bedrock Minecraft servers as subprocesses in one pod + a 20Gi Longhorn PVC (mounted 5x by subPath). Auth = Option B (per-app oauth2-proxy reverse-proxy sidecar), since Cilium 1.16.5 cannot do ext_authz on the shared Gateway: - oauth2-proxy v7.8.1 sidecar (:4180) reverse-proxies to Crafty's self-signed HTTPS UI (127.0.0.1:8443), gated to Keycloak homelab realm group platform-admins. - Confidential `crafty` client provisioned live via kcadm + captured in the realm import; client secret + cookie secret via SOPS/age (mirrored in realm-clients). - Web UI: HTTPRoute crafty.home.keithjackson.dev -> Service :80 -> sidecar :4180. - Game ports: Cilium L2 LoadBalancer on VIP 192.168.1.55 (next free after .53/.54), starter set TCP 25565-25567 (Java) + UDP 19132-19134 (Bedrock), NOT auth-gated. - fsGroup:0 / runAsGroup:0 / runAsUser:1000 for the image's crafty:root identity on the root:root Longhorn PVC (the seerr fsGroup lesson). - Own non-gating Flux tier clusters/base/crafty.yaml (dependsOn platform-config, wait:false) + sops-decryption patch in clusters/dev — a leaf UI must never gate the platform. - Explicit default-deny NetworkPolicy + CiliumNetworkPolicies (platform namespace, not app-namespace, so Kyverno does not auto-generate the baseline). make -C ci gate green; kustomize build of platform/crafty, clusters/base, clusters/dev clean. Co-Authored-By: Claude Opus 4.8 <[email protected]> |