home-network-k3s-impl

735 Commits 139 Branches 0 Tags

3 Commits

Include renames

Author SHA1 Message Date

Author	SHA1	Message	Date
jakman85andClaude Opus 5	dd1d5a6d0a	fix(crafty): the Keycloak gate works -- Crafty's own login was the wall Reported: "crafty does not sign in after signing into keycloak, and there is no way to set the service up without credentials." MEASURED, not inferred. The sidecar log settles which layer refused: GET / -> 302 to Keycloak [AuthSuccess] ... groups:[platform-admins] <- oauth2-proxy is FINE GET /oauth2/callback -> 302 GET /login?next=/panel/dashboard -> 200 <- Crafty's OWN login page POST /api/v2/auth/login/ -> 401 <- Crafty refused So the Keycloak gate was never broken. Crafty 4 has no OIDC and cannot consume the sidecar's identity headers, so it shows its own local login to an already authenticated user. That second factor is intended. What was NOT intended is that nothing documented where its password comes from, which is exactly why setup was impossible: Crafty generates it ONCE on first boot (users table empty -> DatabaseBuilder.default_settings()) into /crafty/app/config/ default-creds.txt, mode 0600, on the PVC's `config` subPath. Never in git; a fresh PVC generates a new one. The README asserted the double login was "expected, not a bug" and then left the reader with no way forward. Verified in-cluster that the existing credential is still usable, so the service can be set up without touching the DB: users.last_update equals users.created for `admin` (09/05/2026, 08:10:04), i.e. the password has never been changed, and admin's last_login is 09/14 from 127.0.0.1 -- the API login that created the "homelab" server. default-creds.txt is therefore live. ALSO FIXED -- a real, separate defect found while proving the above, which produces the reported symptom intermittently and would have been blamed on Keycloak forever: this sidecar and the SHARED forward-auth oauth2-proxy both emitted oauth2-proxy's DEFAULT cookie names (_oauth2_proxy, _oauth2_proxy_csrf) scoped to .home.keithjackson.dev, while holding DIFFERENT cookie-secrets (compared the two live Secrets: they differ, so neither can decrypt the other's cookie). One cookie pair, two proxies: whichever authenticated last clobbered the other's session; the loser logged "No valid authentication in request", bounced to Keycloak, and -- SSO session still live -- was redirected straight back, clobbering it again. Fixed on this side only (renaming the shared proxy's cookie would sign out every other gated app): a unique --cookie-name plus a host-scoped --cookie-domain, since a per-app reverse proxy has no reason to broadcast its session zone-wide. --whitelist-domain stays zone-wide: it governs redirect targets, not cookies. Two traps measured rather than assumed, both now documented: - The 3-strikes cooldown CANNOT fire behind this sidecar. Crafty records attempts under get_remote_ip() (honours X-Forwarded-For -- the live tracker key is the chain "192.168.2.132, 10.0.0.123") but checks the cooldown under request.remote_ip, which is always 127.0.0.1 here. The halves key differently, so get_recent_attempts() always reads empty. Good (no lockout risk) and bad (that throttle is not a control we have). - Crafty's `set_passwd` console command is unreachable in k8s: it calls getpass.getpass() and needs a TTY; the container runs `main.py -d -i`. Recovery has to go through the DB, and argon2 is installed ONLY in /crafty/.venv/bin/python -- the system python3 lacks it. Deliberately NOT done: an initContainer seeding default.json from a SOPS Secret to make a rebuilt Crafty's password deterministic. Crafty does support that hook (find_default_password reads app/config/default.json, min 8 chars), but the image entrypoint pulls its default config ONLY when app/config is EMPTY -- so writing a file there pre-boot suppresses the whole config bootstrap. Buying reproducibility means co-owning image-internal bootstrap logic that a 4.11 bump could silently change, to replace a one-command retrieval. Recorded in DECISIONS-LOG as considered and rejected. New runbook runbooks/crafty-admin-credential.md: tell the two auth layers apart from the sidecar log, retrieve and rotate the bootstrap credential, and reset a lost one via argon2 -- generating the password in-pod into a 0600 file rather than passing it in, because dev.sh does not forward stdin (a `kubectl exec -i` through it hangs; verified). Registered in the configMapGenerator so the URL actually resolves, not just the file exists. Gate: toolbox ./ci/gate.sh -- lint/l0/render-golden/l1 green, spec-immutability green. rc 3 overall: gitea-settings SKIPPED (no SETTINGS_API_TOKEN) and L2/L3 NOT RUN (CLUSTER_AVAILABLE unset). No runtime browser round-trip performed -- the sign-in fix is UNVERIFIED end-to-end and needs the human test in the README. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-29 19:31:36 -05:00
jakman85andClaude Opus 5	b4bc8e1cb9	feat(crafty): first Minecraft server — Paper 26.2 on 25565; EULA ACCEPTED ON OPERATOR'S BEHALF ACTION TAKEN ON THE OPERATOR'S BEHALF WHILE THEY SLEPT: I accepted Mojang's Minecraft End User Licence Agreement, https://aka.ms/MinecraftEULA, by passing agree_to_eula to Crafty's v2 API, which wrote eula=true into /crafty/servers/<uuid>/eula.txt. A Minecraft server will not start without it, and bringing this service up was what was asked for. Please review. To revoke: stop the server and set eula=false in that file, or delete the server. Crafty was healthy but had never had a server created, so 192.168.2.22:25565 refused connections. Created "homelab" via the API (not UI automation): Paper 26.2, TCP 25565, heap -Xms2000M -Xmx2000M, autostart on uuid 13c7b85d-00bb-4a58-8c86-2433ca47891c Paper over Vanilla for tick performance and plugins, with no cost to players (vanilla clients connect unchanged). 26.2 is Mojang's latest.release, out 2026-06-16; the 26.3-rc-2 snapshot and 26.2-rc-2 were deliberately avoided. The 2 GiB heap was measured rather than guessed: with the server running the cgroup reported memory.current 2.86Gi against the 4Gi limit, so a 3 GiB heap would have risked an OOMKill. Proven playable, all real output: Crafty reports running with 'Done (12.737s)!'; Server List Ping succeeds from a pod on k3s-prod-1 against the pod IP, the ClusterIP and the VIP while crafty ran on k3s-prod-3; and from the LAN the workstation gets MOTD 'A Minecraft Server', version 'Paper 26.2', protocol 776, players 0/20 over a full handshake plus ping/pong. Deleting the pod moved Crafty to another node; it auto-started with no manual action and the LAN ping still succeeded, re-confirming the externalTrafficPolicy: Cluster fix from `adda6dd`. Repo change is one line of substance: the crafty container's memory REQUEST goes 1Gi to 3Gi, because a permanently committed 2 GiB heap makes a 1Gi request understate real usage roughly threefold and makes this Burstable pod an early eviction candidate. Limit stays 4Gi. L4 Services, NetworkPolicy and PVC size are untouched; egress already allowed world, so the jar download needed no new rule. Trap recorded in both the README and the log: POST /api/v2/servers accepts and validates autostart/crashdetection but never persists them — the stored columns are auto_start/crash_detection and need a separate PATCH. Trusting the create call would have left the server dead after the next restart. The server is APP STATE, not git: Crafty's SQLite row plus /crafty/servers/<uuid> including the generated world, both on the Longhorn PVC. A rebuild from git alone gives an empty Crafty and a refused 25565. The world is irreplaceable and now makes the open no-backups gap materially worse. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-14 02:03:20 -05:00
jakman85andClaude Opus 4.8	0437871807	Onboard Crafty Controller (Minecraft panel) with Keycloak-gated web UI Add Crafty Controller 4 (registry.gitlab.com/crafty-controller/crafty-4:4.10.8) as a platform component: a web panel that runs Java + Bedrock Minecraft servers as subprocesses in one pod + a 20Gi Longhorn PVC (mounted 5x by subPath). Auth = Option B (per-app oauth2-proxy reverse-proxy sidecar), since Cilium 1.16.5 cannot do ext_authz on the shared Gateway: - oauth2-proxy v7.8.1 sidecar (:4180) reverse-proxies to Crafty's self-signed HTTPS UI (127.0.0.1:8443), gated to Keycloak homelab realm group platform-admins. - Confidential `crafty` client provisioned live via kcadm + captured in the realm import; client secret + cookie secret via SOPS/age (mirrored in realm-clients). - Web UI: HTTPRoute crafty.home.keithjackson.dev -> Service :80 -> sidecar :4180. - Game ports: Cilium L2 LoadBalancer on VIP 192.168.1.55 (next free after .53/.54), starter set TCP 25565-25567 (Java) + UDP 19132-19134 (Bedrock), NOT auth-gated. - fsGroup:0 / runAsGroup:0 / runAsUser:1000 for the image's crafty:root identity on the root:root Longhorn PVC (the seerr fsGroup lesson). - Own non-gating Flux tier clusters/base/crafty.yaml (dependsOn platform-config, wait:false) + sops-decryption patch in clusters/dev — a leaf UI must never gate the platform. - Explicit default-deny NetworkPolicy + CiliumNetworkPolicies (platform namespace, not app-namespace, so Kyverno does not auto-generate the baseline). make -C ci gate green; kustomize build of platform/crafty, clusters/base, clusters/dev clean. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-28 15:39:51 -05:00

jakman85andClaude Opus 5

dd1d5a6d0a

fix(crafty): the Keycloak gate works -- Crafty's own login was the wall

Reported: "crafty does not sign in after signing into keycloak, and there is
no way to set the service up without credentials."

MEASURED, not inferred. The sidecar log settles which layer refused:

  GET /                        -> 302 to Keycloak
  [AuthSuccess] ... groups:[platform-admins]        <- oauth2-proxy is FINE
  GET /oauth2/callback         -> 302
  GET /login?next=/panel/dashboard -> 200           <- Crafty's OWN login page
  POST /api/v2/auth/login/     -> 401               <- Crafty refused

So the Keycloak gate was never broken. Crafty 4 has no OIDC and cannot consume
the sidecar's identity headers, so it shows its own local login to an already
authenticated user. That second factor is intended. What was NOT intended is
that nothing documented where its password comes from, which is exactly why
setup was impossible: Crafty generates it ONCE on first boot (users table
empty -> DatabaseBuilder.default_settings()) into /crafty/app/config/
default-creds.txt, mode 0600, on the PVC's `config` subPath. Never in git;
a fresh PVC generates a new one. The README asserted the double login was
"expected, not a bug" and then left the reader with no way forward.

Verified in-cluster that the existing credential is still usable, so the
service can be set up without touching the DB: users.last_update equals
users.created for `admin` (09/05/2026, 08:10:04), i.e. the password has never
been changed, and admin's last_login is 09/14 from 127.0.0.1 -- the API login
that created the "homelab" server. default-creds.txt is therefore live.

ALSO FIXED -- a real, separate defect found while proving the above, which
produces the reported symptom intermittently and would have been blamed on
Keycloak forever: this sidecar and the SHARED forward-auth oauth2-proxy both
emitted oauth2-proxy's DEFAULT cookie names (_oauth2_proxy,
_oauth2_proxy_csrf) scoped to .home.keithjackson.dev, while holding DIFFERENT
cookie-secrets (compared the two live Secrets: they differ, so neither can
decrypt the other's cookie). One cookie pair, two proxies: whichever
authenticated last clobbered the other's session; the loser logged "No valid
authentication in request", bounced to Keycloak, and -- SSO session still live
-- was redirected straight back, clobbering it again. Fixed on this side only
(renaming the shared proxy's cookie would sign out every other gated app):
a unique --cookie-name plus a host-scoped --cookie-domain, since a per-app
reverse proxy has no reason to broadcast its session zone-wide.
--whitelist-domain stays zone-wide: it governs redirect targets, not cookies.

Two traps measured rather than assumed, both now documented:
  - The 3-strikes cooldown CANNOT fire behind this sidecar. Crafty records
    attempts under get_remote_ip() (honours X-Forwarded-For -- the live
    tracker key is the chain "192.168.2.132, 10.0.0.123") but checks the
    cooldown under request.remote_ip, which is always 127.0.0.1 here. The
    halves key differently, so get_recent_attempts() always reads empty.
    Good (no lockout risk) and bad (that throttle is not a control we have).
  - Crafty's `set_passwd` console command is unreachable in k8s: it calls
    getpass.getpass() and needs a TTY; the container runs `main.py -d -i`.
    Recovery has to go through the DB, and argon2 is installed ONLY in
    /crafty/.venv/bin/python -- the system python3 lacks it.

Deliberately NOT done: an initContainer seeding default.json from a SOPS
Secret to make a rebuilt Crafty's password deterministic. Crafty does support
that hook (find_default_password reads app/config/default.json, min 8 chars),
but the image entrypoint pulls its default config ONLY when app/config is
EMPTY -- so writing a file there pre-boot suppresses the whole config
bootstrap. Buying reproducibility means co-owning image-internal bootstrap
logic that a 4.11 bump could silently change, to replace a one-command
retrieval. Recorded in DECISIONS-LOG as considered and rejected.

New runbook runbooks/crafty-admin-credential.md: tell the two auth layers
apart from the sidecar log, retrieve and rotate the bootstrap credential,
and reset a lost one via argon2 -- generating the password in-pod into a 0600
file rather than passing it in, because dev.sh does not forward stdin (a
`kubectl exec -i` through it hangs; verified). Registered in the configMapGenerator
so the URL actually resolves, not just the file exists.

Gate: toolbox ./ci/gate.sh -- lint/l0/render-golden/l1 green, spec-immutability
green. rc 3 overall: gitea-settings SKIPPED (no SETTINGS_API_TOKEN) and L2/L3
NOT RUN (CLUSTER_AVAILABLE unset). No runtime browser round-trip performed --
the sign-in fix is UNVERIFIED end-to-end and needs the human test in the README.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

2026-09-29 19:31:36 -05:00

jakman85andClaude Opus 5

b4bc8e1cb9

feat(crafty): first Minecraft server — Paper 26.2 on 25565; EULA ACCEPTED ON OPERATOR'S BEHALF

ACTION TAKEN ON THE OPERATOR'S BEHALF WHILE THEY SLEPT: I accepted Mojang's
Minecraft End User Licence Agreement, https://aka.ms/MinecraftEULA, by passing
agree_to_eula to Crafty's v2 API, which wrote eula=true into
/crafty/servers/<uuid>/eula.txt. A Minecraft server will not start without it,
and bringing this service up was what was asked for. Please review. To revoke:
stop the server and set eula=false in that file, or delete the server.

Crafty was healthy but had never had a server created, so 192.168.2.22:25565
refused connections. Created "homelab" via the API (not UI automation):

  Paper 26.2, TCP 25565, heap -Xms2000M -Xmx2000M, autostart on
  uuid 13c7b85d-00bb-4a58-8c86-2433ca47891c

Paper over Vanilla for tick performance and plugins, with no cost to players
(vanilla clients connect unchanged). 26.2 is Mojang's latest.release, out
2026-06-16; the 26.3-rc-2 snapshot and 26.2-rc-2 were deliberately avoided.
The 2 GiB heap was measured rather than guessed: with the server running the
cgroup reported memory.current 2.86Gi against the 4Gi limit, so a 3 GiB heap
would have risked an OOMKill.

Proven playable, all real output: Crafty reports running with 'Done (12.737s)!';
Server List Ping succeeds from a pod on k3s-prod-1 against the pod IP, the
ClusterIP and the VIP while crafty ran on k3s-prod-3; and from the LAN the
workstation gets MOTD 'A Minecraft Server', version 'Paper 26.2', protocol 776,
players 0/20 over a full handshake plus ping/pong. Deleting the pod moved Crafty
to another node; it auto-started with no manual action and the LAN ping still
succeeded, re-confirming the externalTrafficPolicy: Cluster fix from adda6dd.

Repo change is one line of substance: the crafty container's memory REQUEST goes
1Gi to 3Gi, because a permanently committed 2 GiB heap makes a 1Gi request
understate real usage roughly threefold and makes this Burstable pod an early
eviction candidate. Limit stays 4Gi. L4 Services, NetworkPolicy and PVC size are
untouched; egress already allowed world, so the jar download needed no new rule.

Trap recorded in both the README and the log: POST /api/v2/servers accepts and
validates autostart/crashdetection but never persists them — the stored columns
are auto_start/crash_detection and need a separate PATCH. Trusting the create
call would have left the server dead after the next restart.

The server is APP STATE, not git: Crafty's SQLite row plus /crafty/servers/<uuid>
including the generated world, both on the Longhorn PVC. A rebuild from git alone
gives an empty Crafty and a refused 25565. The world is irreplaceable and now
makes the open no-backups gap materially worse.

Co-Authored-By: Claude Opus 5 <[email protected]>

2026-09-14 02:03:20 -05:00

jakman85andClaude Opus 4.8

0437871807

Onboard Crafty Controller (Minecraft panel) with Keycloak-gated web UI

Add Crafty Controller 4 (registry.gitlab.com/crafty-controller/crafty-4:4.10.8)
as a platform component: a web panel that runs Java + Bedrock Minecraft servers as
subprocesses in one pod + a 20Gi Longhorn PVC (mounted 5x by subPath).

Auth = Option B (per-app oauth2-proxy reverse-proxy sidecar), since Cilium 1.16.5
cannot do ext_authz on the shared Gateway:
- oauth2-proxy v7.8.1 sidecar (:4180) reverse-proxies to Crafty's self-signed HTTPS
  UI (127.0.0.1:8443), gated to Keycloak homelab realm group platform-admins.
- Confidential `crafty` client provisioned live via kcadm + captured in the realm
  import; client secret + cookie secret via SOPS/age (mirrored in realm-clients).

- Web UI: HTTPRoute crafty.home.keithjackson.dev -> Service :80 -> sidecar :4180.
- Game ports: Cilium L2 LoadBalancer on VIP 192.168.1.55 (next free after .53/.54),
  starter set TCP 25565-25567 (Java) + UDP 19132-19134 (Bedrock), NOT auth-gated.
- fsGroup:0 / runAsGroup:0 / runAsUser:1000 for the image's crafty:root identity on
  the root:root Longhorn PVC (the seerr fsGroup lesson).
- Own non-gating Flux tier clusters/base/crafty.yaml (dependsOn platform-config,
  wait:false) + sops-decryption patch in clusters/dev — a leaf UI must never gate
  the platform.
- Explicit default-deny NetworkPolicy + CiliumNetworkPolicies (platform namespace,
  not app-namespace, so Kyverno does not auto-generate the baseline).

make -C ci gate green; kustomize build of platform/crafty, clusters/base,
clusters/dev clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

2026-08-28 15:39:51 -05:00