377fa205ea0aca09ccbdf75773f38d43dae52647
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
c8c1d2935d |
feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).
THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.
Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.
PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.
BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.
TWO THAT CANNOT BE TOKENIZED, flagged not forced:
- headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
anyway — client secrets are realm-scoped.
- kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
prod.home.
EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.
I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.
DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.
L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.
PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.
Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
3f980d2a17 |
feat(env): tokenize hardcoded dev LAN addresses into cluster-vars (R-ENV-02)
R-ENV-02 promises dev and prod differ ONLY by scoped cluster-vars overrides. In fact
dev LAN addresses were hardcoded into base manifests, so clusters/prod would have come
up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN (192.168.2.x).
New cluster-vars knobs (base default = dev literal, per the file's existing convention):
K3S_API_HOST platform/cilium/helmrelease.yaml (+ the values.yaml mirror)
CILIUM_LB_POOL_START platform-config/cilium/l2-ip-pool.yaml
CILIUM_LB_POOL_STOP "
UNIFI_L4_VIP apps/vms/unifi/50-service-l4.yaml
CRAFTY_L4_VIP platform/crafty/service-l4.yaml
S3_BACKUP_ENDPOINT platform/keycloak/postgres-cluster.yaml, apps/wave-0/{nextcloud,
semaphore}-db.yaml, and platform/velero/helmrelease.yaml
Also fixes GATEWAY_VIP's base default, which was still the RETIRED libvirt address
10.73.0.240, and adds the GATEWAY_VIP patch that clusters/prod was missing entirely.
clusters/base/apps-vms.yaml gains postBuild.substituteFrom (it was the one consuming
Kustomization without it, so a ${TOKEN} there would have shipped as a literal string).
The envsubst contract is documented there: ${VAR} substitutes and escapes as $${VAR};
$VAR and $(cmd) are left alone; ${ARR[0]} hard-errors. The three UOS_URL expansions in
the unifi cloud-init are escaped accordingly.
PROD ADDRESS PLAN (operator, 2026-08-31):
nodes .10/.11/.12 · control-plane VIP .9 (Ansible-side) · GATEWAY_VIP .20
Cilium LB pool .20-.30
CONTAINMENT INVARIANT, now documented at the pool and in the prod overlay: GATEWAY_VIP
MUST equal CILIUM_LB_POOL_START. The shared Gateway pins no address, so Cilium LB-IPAM
gives it the first free address of a matching pool block. An initial prod proposal put
.20 outside a .21-.30 pool; the Gateway would have come up on .21 while split-horizon
coredns-custom kept answering *.home.keithjackson.dev with .20 — an address nothing
announces — blackholing in-cluster OIDC hairpin traffic while reporting healthy. The
pool was widened to .20-.30. The control-plane VIP (.9) stays outside the pool so
kube-vip and LB-IPAM can never contend.
Values prod still cannot know stay INVALID `TBD-PROD-*` placeholders on purpose: base
defaults are dev values, so an omitted patch means prod silently inherits a dev address
— far worse than a loud failed apply.
DEV IS BYTE-IDENTICAL. Verified by kustomize build over 13 paths before/after, then
flux envsubst with the dev cluster-vars values applied to both sides: zero drift. The
only deltas are the intended additive keys in the overlay ConfigMap itself. No
unresolved ${TOKEN} survives. render-golden green (10 goldens); yamllint clean.
Serves: R-ENV-02. Task: T-7.1 (prerequisite). Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
68690e5019 |
fix(net): allow SSO apps egress to keycloak backend (Cilium 1.20 CVE-2025-30162)
App SSO broke cluster-wide after the Cilium 1.16.5->1.20.1 upgrade: the CVE-2025-30162 fix (in
>=1.16.8/1.17.2) makes the shared Gateway's L7 proxy re-evaluate a pod's egress policy against
Keycloak's REAL backend cluster identity (enforce_policy_on_l7lb) when it hairpins to keycloak.home.
'toEntities: world' matches the Gateway VIP at L3/L4 but not the resolved keycloak identity at L7, so
OIDC discovery/token 403'd ('Access denied') for every app with a NetworkPolicy (gitea/nextcloud/
semaphore/headlamp/crafty...); Grafana survived only because monitoring has no NetworkPolicy.
Fix: explicit toEndpoints egress allow to the keycloak backend for SSO apps.
- catalog/render.py: emit it for auth.mode in {oidc,forward-auth}; re-rendered 8 app goldens.
- platform/headlamp + platform/crafty: hand-written policies get the same allow.
Split-horizon design unchanged. Verified live: headlamp-ns -> keycloak discovery 403 -> 200.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
748e9ca55d |
fix(oauth2-proxy): --allowed-groups → --allowed-group (correct oauth2-proxy v7.8.1 flag)
The plural --allowed-groups does not exist in oauth2-proxy v7.8.1; the shared oauth2-proxy new pod and the crafty auth sidecar both crashlooped on 'unknown flag'. The flag is the singular, repeatable --allowed-group. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
0437871807 |
Onboard Crafty Controller (Minecraft panel) with Keycloak-gated web UI
Add Crafty Controller 4 (registry.gitlab.com/crafty-controller/crafty-4:4.10.8) as a platform component: a web panel that runs Java + Bedrock Minecraft servers as subprocesses in one pod + a 20Gi Longhorn PVC (mounted 5x by subPath). Auth = Option B (per-app oauth2-proxy reverse-proxy sidecar), since Cilium 1.16.5 cannot do ext_authz on the shared Gateway: - oauth2-proxy v7.8.1 sidecar (:4180) reverse-proxies to Crafty's self-signed HTTPS UI (127.0.0.1:8443), gated to Keycloak homelab realm group platform-admins. - Confidential `crafty` client provisioned live via kcadm + captured in the realm import; client secret + cookie secret via SOPS/age (mirrored in realm-clients). - Web UI: HTTPRoute crafty.home.keithjackson.dev -> Service :80 -> sidecar :4180. - Game ports: Cilium L2 LoadBalancer on VIP 192.168.1.55 (next free after .53/.54), starter set TCP 25565-25567 (Java) + UDP 19132-19134 (Bedrock), NOT auth-gated. - fsGroup:0 / runAsGroup:0 / runAsUser:1000 for the image's crafty:root identity on the root:root Longhorn PVC (the seerr fsGroup lesson). - Own non-gating Flux tier clusters/base/crafty.yaml (dependsOn platform-config, wait:false) + sops-decryption patch in clusters/dev — a leaf UI must never gate the platform. - Explicit default-deny NetworkPolicy + CiliumNetworkPolicies (platform namespace, not app-namespace, so Kyverno does not auto-generate the baseline). make -C ci gate green; kustomize build of platform/crafty, clusters/base, clusters/dev clean. Co-Authored-By: Claude Opus 4.8 <[email protected]> |