jakman85andClaude Opus 5
4664b41db3
Cilium's L2 announcer elects one node per Service to answer ARP for its LoadBalancer VIP, and that election is NOT backend-aware. Combined with externalTrafficPolicy: Local -- which lets a node deliver only to a backend on itself -- an elected announcer holding no backend answers ARP for the address and then has nowhere to send the packet, so the client gets ICMP port unreachable. It never self-heals: the lease holder keeps renewing after the backend moves away. Measured on prod 2026-09-14: unifi's VM ran on k3s-prod-1 with both leases on k3s-prod-2, and crafty's pod ran on k3s-prod-1 with the -tcp lease on k3s-prod-2 while its identical -udp twin happened to draw k3s-prod-1 and worked. Confirmed causally on the announcing node with `cilium-dbg service list`: under Local the frontend 192.168.2.22:25565 is absent from that node's service table entirely; under Cluster it appears as `1 => 10.0.0.11:25565 (active)`; flipped back, it vanishes again. Durability confirmed by cordoning k3s-prod-1 and moving the crafty pod to k3s-prod-3, after which the announcing node re-pointed to the new backend with no intervention. Both files also carried a comment claiming Local "pins the VIP to the node running the pod", which is the precise thing that is false here; both are rewritten. apps/wave-0/pihole-dns.yaml is the reference implementation. The client-IP cost is stated per service rather than waved away. For unifi it is ZERO: the VM's KubeVirt `masquerade` binding already SNATs everything into the guest, so Local was never preserving a client address the controller could see. For crafty it is real and accepted -- player identity is a Mojang/Xbox UUID so /ban and the whitelist are unaffected, but /ban-ip would hit every player at once and must not be used; no IP ban or per-IP rate limit is configured today. No loadBalancerSourceRanges here, deliberately departing from pihole-dns.yaml: these Services never had a source restriction (their CNPs admit `world`), so Cluster removes no protection, and adding a LAN-only filter during an outage fix could cut off off-LAN players or a remote site's device informing. No CiliumNetworkPolicy change is needed, and that was verified rather than assumed: Cilium's `cluster` entity subsumes `remote-node`, confirmed by dumping the crafty endpoint's datapath policy map with the CNP unmodified (`reserved:remote-node` was already an Allow on the game ports). Unlike pihole, whose policy omits `cluster` and must name remote-node explicitly. Adds an L0 whole-repo guard for the class. It fails on the pre-fix tree with exactly these four Services named and passes after. whole-repo scope is necessary: ci/touched-requirements.sh has no arm for apps/* or platform/crafty/*, so the change most likely to reintroduce this would attribute no requirement and be gated by nothing. HONEST LIMIT -- this does NOT end the reported outage. All four ports also have no listening backend: UniFi OS was never installed in the VM (its README flags this as the open blocker) and /crafty/servers is empty, so no Minecraft server exists. Probed directly from the node, bypassing the VIP entirely, 10.0.0.74:8443/:8080/:11443 and 10.0.0.11:25565 all refuse, while the two controls answer. This commit fixes a real latent defect that would have blackholed those VIPs once the applications exist; it cannot make an absent application answer. Co-Authored-By: Claude Opus 5 <[email protected]>
platform/crafty
Crafty Controller 4 — a web panel that creates and manages multiple Minecraft servers (Java +
Bedrock) as subprocesses inside its own container. Its web UI is Keycloak-gated via a per-pod
oauth2-proxy reverse-proxy sidecar (the decided Option B, see
../oauth2-proxy/README.md). Game ports are exposed un-gated on a
Cilium L2 LoadBalancer (game protocols authenticate players themselves).
- Owner: WS-APP (auth wiring shared with WS-SEC)
- Flux tier: own non-gating
clusters/base/crafty.yaml(dependsOn platform-config,wait:false) - Namespace:
crafty(PLATFORM namespace —gateway-access=shared, NOTapp-namespace=true)
Deploy summary
| Thing | Value |
|---|---|
| Image | registry.gitlab.com/crafty-controller/crafty-4:4.10.8 (current stable; bare-semver tags, no v) |
| Web UI | HTTPS self-signed on :8443 (reached only via the sidecar / same-pod) |
| Sidecar | quay.io/oauth2-proxy/oauth2-proxy:v7.8.1, reverse-proxy on :4180 |
| Data | ONE Longhorn RWO PVC crafty-data (20Gi), mounted 5× by subPath |
| PVC subPaths | servers→/crafty/servers, backups→/crafty/backups, logs→/crafty/logs, import→/crafty/import, config→/crafty/app/config |
| Identity | runAsUser:1000 runAsGroup:0 fsGroup:0 runAsNonRoot:true (see below) |
| Web host | crafty.home.keithjackson.dev → shared Gateway → crafty Service :80 → sidecar :4180 → Crafty :8443 |
| Game VIP | 192.168.1.55 (Cilium L2 LoadBalancer; .53=Gateway, .54=unifi, .55=crafty) |
| Game ports | starter set: TCP 25565-25567 (Java), UDP 19132-19134 (Bedrock) |
fsGroup / user decision (the seerr lesson, applied)
Crafty's Dockerfile does useradd -g root -M crafty (UID 1000, primary group root/GID 0),
chown -R crafty:root /crafty, and makes /crafty group-writable — the OpenShift/K8s
"arbitrary-UID, root-group-perms" pattern. A fresh Longhorn PVC is root:root and not group-writable,
so the non-root crafty process could not create /crafty/servers/* etc. and would CrashLoop (exactly
seerr's failure at UID 1000 on a root:root PVC). Fix: fsGroup: 0 makes the kubelet group-own
the mounted volume to GID 0, mark it group-writable+SGID, and add GID 0 as a supplemental group;
runAsGroup: 0 + runAsUser: 1000 pin the image's own crafty:root identity. Longhorn's
CSIDriver fsGroupPolicy: ReadWriteOnceWithFSType applies fsGroup to this RWO filesystem volume.
HUMAN runtime check: confirm the pod writes its SQLite DB and can start a server. If the image's
craftyUID is ever not 1000, adjustrunAsUser(the launcher's non-root path runs directly as the current UID, so any non-root UID with GID 0 works, but 1000 matchesgetpwuidcleanly).
Auth = Option B (per-app oauth2-proxy reverse-proxy)
Cilium 1.16.5 cannot inject ext_authz into the shared Gateway's machine-generated Envoy config, and
native Gateway-API ExternalAuth ships in no released Cilium yet (see ../oauth2-proxy/README.md). So
Crafty's UI is gated by a dedicated oauth2-proxy sidecar in the Crafty pod (--reverse-proxy,
--upstream=https://127.0.0.1:8443, --ssl-upstream-insecure-skip-verify=true,
--allowed-groups=platform-admins, --skip-provider-button, cookie/redirect for
crafty.home.keithjackson.dev). The crafty Service targets the sidecar, never Crafty's :8443, so
the gate is in the data path (fail-closed: no valid platform-admins session ⇒ no upstream).
Note: Crafty ALSO has its own local admin login (it has no OIDC). After oauth2-proxy admits the user, they still see Crafty's own login page — a second, defense-in-depth factor. The operator logs in with Crafty's local admin credentials (set on first boot). This is expected, not a bug.
Keycloak client
Confidential auth-code client crafty in the homelab realm — redirect
https://crafty.home.keithjackson.dev/oauth2/callback, webOrigins the same host, a groups
membership mapper (bare names). Provisioned live via kcadm at authoring time (2026-08-28) and
captured as the durable GitOps source of truth in
../keycloak/keycloak-realm-import.yaml (secret injected via
the CRAFTY_CLIENT_SECRET placeholder).
Secret locations (SOPS/age)
| Secret | File | Key |
|---|---|---|
| oauth2-proxy client id/secret + cookie secret | crafty-oauth2-proxy-credentials.sops.yaml (this dir) |
client-id, client-secret, cookie-secret |
| Keycloak client secret (mirror) | ../keycloak/keycloak-realm-clients.sops.yaml |
crafty-client-secret |
client-secret here must equal crafty-client-secret there (cross-namespace mirror — a Secret
can't span namespaces). Both are age-encrypted to the repo sops-age recipient and Flux-decryptable
(verified: both decrypt and the two values match).
Game-port L4 exposure
The Gateway is L7-HTTPS-only; raw TCP/UDP game ports can't ride it. service-l4.yaml exposes them as
two type=LoadBalancer Services (TCP + UDP) sharing the pinned VIP 192.168.1.55
(lbipam.cilium.io/sharing-key: crafty), ARP-announced by the existing default-l2 policy. This path
is NOT auth-gated — Minecraft Java (Mojang/Microsoft) and Bedrock (Xbox Live) authenticate players
themselves. external-dns publishes crafty-mc.home.keithjackson.dev → 192.168.1.55.
Adding more game ports (when the operator creates servers)
The operator has not created any servers yet; the starter set covers 3 Java + 3 Bedrock slots. To add capacity:
- In the Crafty UI, create the server and bind it to a NEW port (Java TCP or Bedrock UDP).
- Add that port to the matching Service in
service-l4.yaml(one{ name, port, targetPort, protocol }line — K8s Services enumerate ports, no ranges). - Add it to the game-port ingress rule in
networkpolicy.yaml(crafty-allow-ingress). - (Optional) add a
containerPortindeployment.yamlfor documentation. - Commit → Flux reconciles.
Policies
namespace.yaml sets only gateway-access=shared (platform namespace, like headlamp/homepage), so
Kyverno does not auto-generate the app baseline. networkpolicy.yaml ships it explicitly (defense in
depth): default-deny (DNS-only egress) + Cilium allows — Gateway→sidecar :4180, world/LAN→game
ports, host→probe ports, and egress world (server-jar downloads + the OIDC path to Keycloak via the
Gateway VIP). The sidecar→Crafty hop is pod-local (127.0.0.1) and not subject to network policy.
Verification
make -C ci gate— green (hand-authored platform manifests; render-golden unaffected).kustomize buildofplatform/crafty,clusters/base,clusters/dev— clean.
Human tests (post-deploy)
- Browser OIDC round-trip: incognito →
https://crafty.home.keithjackson.dev→ expect a 302 tohttps://keycloak.home.keithjackson.dev/realms/homelab/...(NOT the Crafty UI). Log in as a platform-admins user (MFA/OTP enforced) → land on Crafty's own login → sign in with Crafty's local admin. A non-platform-admins user → 403 (authenticated but not authorized). - Pod health / fsGroup:
kubectl -n crafty get podReady 2/2; Crafty writes its DB under/crafty/app/configand/crafty/serverswithout EACCES. - Server-create + game reach: in the Crafty UI create a Java server bound to
25565; start it; connect a Minecraft client tocrafty-mc.home.keithjackson.dev:25565(or192.168.1.55:25565) from the LAN and confirm the world loads. (Bedrock: a server on19132/udpreached at the VIP.)