jakman85andClaude Opus 5
c8c1d2935d
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).
THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.
Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.
PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.
BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.
TWO THAT CANNOT BE TOKENIZED, flagged not forced:
- headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
anyway — client secrets are realm-scoped.
- kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
prod.home.
EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.
I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.
DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.
L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.
PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.
Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
platform/crafty
Crafty Controller 4 — a web panel that creates and manages multiple Minecraft servers (Java +
Bedrock) as subprocesses inside its own container. Its web UI is Keycloak-gated via a per-pod
oauth2-proxy reverse-proxy sidecar (the decided Option B, see
../oauth2-proxy/README.md). Game ports are exposed un-gated on a
Cilium L2 LoadBalancer (game protocols authenticate players themselves).
- Owner: WS-APP (auth wiring shared with WS-SEC)
- Flux tier: own non-gating
clusters/base/crafty.yaml(dependsOn platform-config,wait:false) - Namespace:
crafty(PLATFORM namespace —gateway-access=shared, NOTapp-namespace=true)
Deploy summary
| Thing | Value |
|---|---|
| Image | registry.gitlab.com/crafty-controller/crafty-4:4.10.8 (current stable; bare-semver tags, no v) |
| Web UI | HTTPS self-signed on :8443 (reached only via the sidecar / same-pod) |
| Sidecar | quay.io/oauth2-proxy/oauth2-proxy:v7.8.1, reverse-proxy on :4180 |
| Data | ONE Longhorn RWO PVC crafty-data (20Gi), mounted 5× by subPath |
| PVC subPaths | servers→/crafty/servers, backups→/crafty/backups, logs→/crafty/logs, import→/crafty/import, config→/crafty/app/config |
| Identity | runAsUser:1000 runAsGroup:0 fsGroup:0 runAsNonRoot:true (see below) |
| Web host | crafty.home.keithjackson.dev → shared Gateway → crafty Service :80 → sidecar :4180 → Crafty :8443 |
| Game VIP | 192.168.1.55 (Cilium L2 LoadBalancer; .53=Gateway, .54=unifi, .55=crafty) |
| Game ports | starter set: TCP 25565-25567 (Java), UDP 19132-19134 (Bedrock) |
fsGroup / user decision (the seerr lesson, applied)
Crafty's Dockerfile does useradd -g root -M crafty (UID 1000, primary group root/GID 0),
chown -R crafty:root /crafty, and makes /crafty group-writable — the OpenShift/K8s
"arbitrary-UID, root-group-perms" pattern. A fresh Longhorn PVC is root:root and not group-writable,
so the non-root crafty process could not create /crafty/servers/* etc. and would CrashLoop (exactly
seerr's failure at UID 1000 on a root:root PVC). Fix: fsGroup: 0 makes the kubelet group-own
the mounted volume to GID 0, mark it group-writable+SGID, and add GID 0 as a supplemental group;
runAsGroup: 0 + runAsUser: 1000 pin the image's own crafty:root identity. Longhorn's
CSIDriver fsGroupPolicy: ReadWriteOnceWithFSType applies fsGroup to this RWO filesystem volume.
HUMAN runtime check: confirm the pod writes its SQLite DB and can start a server. If the image's
craftyUID is ever not 1000, adjustrunAsUser(the launcher's non-root path runs directly as the current UID, so any non-root UID with GID 0 works, but 1000 matchesgetpwuidcleanly).
Auth = Option B (per-app oauth2-proxy reverse-proxy)
Cilium 1.16.5 cannot inject ext_authz into the shared Gateway's machine-generated Envoy config, and
native Gateway-API ExternalAuth ships in no released Cilium yet (see ../oauth2-proxy/README.md). So
Crafty's UI is gated by a dedicated oauth2-proxy sidecar in the Crafty pod (--reverse-proxy,
--upstream=https://127.0.0.1:8443, --ssl-upstream-insecure-skip-verify=true,
--allowed-groups=platform-admins, --skip-provider-button, cookie/redirect for
crafty.home.keithjackson.dev). The crafty Service targets the sidecar, never Crafty's :8443, so
the gate is in the data path (fail-closed: no valid platform-admins session ⇒ no upstream).
Note: Crafty ALSO has its own local admin login (it has no OIDC). After oauth2-proxy admits the user, they still see Crafty's own login page — a second, defense-in-depth factor. The operator logs in with Crafty's local admin credentials (set on first boot). This is expected, not a bug.
Keycloak client
Confidential auth-code client crafty in the homelab realm — redirect
https://crafty.home.keithjackson.dev/oauth2/callback, webOrigins the same host, a groups
membership mapper (bare names). Provisioned live via kcadm at authoring time (2026-08-28) and
captured as the durable GitOps source of truth in
../keycloak/keycloak-realm-import.yaml (secret injected via
the CRAFTY_CLIENT_SECRET placeholder).
Secret locations (SOPS/age)
| Secret | File | Key |
|---|---|---|
| oauth2-proxy client id/secret + cookie secret | crafty-oauth2-proxy-credentials.sops.yaml (this dir) |
client-id, client-secret, cookie-secret |
| Keycloak client secret (mirror) | ../keycloak/keycloak-realm-clients.sops.yaml |
crafty-client-secret |
client-secret here must equal crafty-client-secret there (cross-namespace mirror — a Secret
can't span namespaces). Both are age-encrypted to the repo sops-age recipient and Flux-decryptable
(verified: both decrypt and the two values match).
Game-port L4 exposure
The Gateway is L7-HTTPS-only; raw TCP/UDP game ports can't ride it. service-l4.yaml exposes them as
two type=LoadBalancer Services (TCP + UDP) sharing the pinned VIP 192.168.1.55
(lbipam.cilium.io/sharing-key: crafty), ARP-announced by the existing default-l2 policy. This path
is NOT auth-gated — Minecraft Java (Mojang/Microsoft) and Bedrock (Xbox Live) authenticate players
themselves. external-dns publishes crafty-mc.home.keithjackson.dev → 192.168.1.55.
Adding more game ports (when the operator creates servers)
The operator has not created any servers yet; the starter set covers 3 Java + 3 Bedrock slots. To add capacity:
- In the Crafty UI, create the server and bind it to a NEW port (Java TCP or Bedrock UDP).
- Add that port to the matching Service in
service-l4.yaml(one{ name, port, targetPort, protocol }line — K8s Services enumerate ports, no ranges). - Add it to the game-port ingress rule in
networkpolicy.yaml(crafty-allow-ingress). - (Optional) add a
containerPortindeployment.yamlfor documentation. - Commit → Flux reconciles.
Policies
namespace.yaml sets only gateway-access=shared (platform namespace, like headlamp/homepage), so
Kyverno does not auto-generate the app baseline. networkpolicy.yaml ships it explicitly (defense in
depth): default-deny (DNS-only egress) + Cilium allows — Gateway→sidecar :4180, world/LAN→game
ports, host→probe ports, and egress world (server-jar downloads + the OIDC path to Keycloak via the
Gateway VIP). The sidecar→Crafty hop is pod-local (127.0.0.1) and not subject to network policy.
Verification
make -C ci gate— green (hand-authored platform manifests; render-golden unaffected).kustomize buildofplatform/crafty,clusters/base,clusters/dev— clean.
Human tests (post-deploy)
- Browser OIDC round-trip: incognito →
https://crafty.home.keithjackson.dev→ expect a 302 tohttps://keycloak.home.keithjackson.dev/realms/homelab/...(NOT the Crafty UI). Log in as a platform-admins user (MFA/OTP enforced) → land on Crafty's own login → sign in with Crafty's local admin. A non-platform-admins user → 403 (authenticated but not authorized). - Pod health / fsGroup:
kubectl -n crafty get podReady 2/2; Crafty writes its DB under/crafty/app/configand/crafty/serverswithout EACCES. - Server-create + game reach: in the Crafty UI create a Java server bound to
25565; start it; connect a Minecraft client tocrafty-mc.home.keithjackson.dev:25565(or192.168.1.55:25565) from the LAN and confirm the world loads. (Bedrock: a server on19132/udpreached at the VIP.)