feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain

INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).

THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.

Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.

PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.

BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.

TWO THAT CANNOT BE TOKENIZED, flagged not forced:
  - headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
    over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
    so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
    anyway — client secrets are realm-scoped.
  - kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
    reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
    overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
    prod.home.

EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.

I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.

DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.

L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.

PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.

Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

infra/PROD-PROVISIONING.md

@@ -1172,6 +1172,7 @@ no extra vars is byte-for-byte the dev behaviour it always had (verified by a be
 | `192.168.1.55` (crafty) | `platform/crafty/service-l4.yaml` | prod-LAN address |
 | `192.168.1.54` (UniFi VM) | `apps/vms/unifi/50-service-l4.yaml` | prod-LAN address |
 | `192.168.1.9:9000` (S3 backup) | `platform/keycloak/postgres-cluster.yaml`, `apps/wave-0/{nextcloud,semaphore}-db.yaml` | reachable prod backup target |
+| ~~`home.keithjackson.dev` (every hostname)~~ | ~~HTTPRoutes, Gateway, wildcard cert, coredns-custom, Keycloak realm, oauth2-proxy, Homepage, `catalog/`~~ | **CLOSED 2026-09-04 — see §10.** This one was not merely "would build wrong": it **already destroyed dev's public DNS**. `BASE_DOMAIN` was a declared-but-DEAD knob; it is now real, and prod serves `prod.home.keithjackson.dev`. |
 
 - [ ] **Tokenize each into a `cluster-vars` knob** with a base default, then set the value in *both*
       overlays. Adding a knob before referencing `${KNOB}` is the sanctioned mechanism (see the
@@ -1282,3 +1283,152 @@ single-server-etcd window, and avoids a later member-add dance.
       specifically.
 - [ ] Monitoring and alerts live on prod.
 - [ ] The conformance suite passes green **against the prod cluster**, not just dev (R-TEST-03).
+      Blocked on parameterizing the L2 tests that hardcode `home.keithjackson.dev` — see §10 phase 2.
+- [ ] **Prod serves its own DNS/TLS identity** and dev's records are provably untouched (§10):
+      distinct `EXTERNALDNS_TXT_OWNER_ID`, prod wildcard cert `*.prod.home.keithjackson.dev`, and
+      prod's own Keycloak realm + per-service client secrets.
+
+## 10. DNS/TLS environment split — **the 2026-09-04 incident, and phase 1 of the fix**
+
+### What happened
+
+Prod's `external-dns` **deleted every one of dev's public DNS records.**
+
+Both clusters ran `policy: sync` against the **same Cloudflare zone**, served the **same
+hostnames**, and — because neither overlay set `txtOwnerId` — both inherited the external-dns
+chart's default owner id **`default`**. ExternalDNS decides what it may delete by reading its own
+owner id back out of the TXT registry. Prod read dev's TXT entries, saw its own owner id, found no
+local resource claiming those names, and garbage-collected the records behind them. Dev recovered
+once prod was stopped. Prod's `platform` tier and everything downstream were then **suspended**, and
+remain suspended, so it cannot recur before this change is reviewed.
+
+`policy: sync` was not the bug and has not been changed. What was missing was **identity**.
+
+### The second bug, queued directly behind it
+
+`platform-config` would have requested a `*.home.keithjackson.dev` wildcard `Certificate` from
+**letsencrypt-prod** — for a cert **dev already holds** — and both clusters would then have raced
+each other's `_acme-challenge` TXT records on that one zone. Neither would reliably solve DNS-01.
+This never fired only because the tier was suspended first.
+
+### Root cause, stated precisely
+
+Not external-dns. **Every hostname in the repo was a hardcoded literal.** `BASE_DOMAIN` existed in
+`clusters/base/cluster-vars.yaml` but was referenced by **no reconciled manifest** — the repo-derived
+status report flagged it as a dead knob. R-ENV-02 promises dev and prod differ only by `cluster-vars`
+values; for DNS and TLS that promise was false, so `clusters/prod` was *authored* to serve dev's
+public identity. Two clusters that describe the same DNS name will fight over it no matter how the
+DNS controller is configured.
+
+### The architecture decision (operator)
+
+```
+dev   ->  *.dev.home.keithjackson.dev        gitea.dev.home.keithjackson.dev
+prod  ->  *.prod.home.keithjackson.dev       gitea.prod.home.keithjackson.dev
+apex  ->  CNAME <svc>.home.keithjackson.dev  ->  <svc>.prod.home.keithjackson.dev
+```
+
+### PHASE 1 — **prod only. Dev did not move.** (this change)
+
+`BASE_DOMAIN` is now real: every hostname in every Flux-reconciled manifest derives from it.
+
+| | `BASE_DOMAIN` | `EXTERNALDNS_TXT_OWNER_ID` | `EXTERNALDNS_DOMAIN_FILTER` |
+|---|---|---|---|
+| base default | `home.keithjackson.dev` | `unset-see-cluster-vars` (non-colliding sentinel) | `keithjackson.dev` |
+| **dev** | `home.keithjackson.dev` (**unchanged**) | `default` (**must not change**) | `keithjackson.dev` |
+| **prod** | `prod.home.keithjackson.dev` | `prod` | `prod.home.keithjackson.dev` |
+
+**Dev's effective hostname is deliberately still the apex `home.keithjackson.dev`.** Moving prod
+alone ends the collision and unblocks prod, and it is the whole of the risk that had to be taken
+while the operator is away. Dev's rendered output is **byte-identical** except for one added line,
+`txtOwnerId: default`, which is the value dev's external-dns was already using implicitly — verified
+by rendering all 12 `substituteFrom` paths through `kubectl kustomize | flux envsubst --strict` for
+both overlays and diffing.
+
+**Two independent guards now, either sufficient alone:**
+
+1. **`txtOwnerId`** — per-environment identity. A cluster only ever touches records it owns. This is
+   the direct fix for the incident. Dev **must** render exactly `default`: its live records are
+   already registered under that owner, and any other value would make dev's own external-dns stop
+   recognising them and re-create duplicates beside the orphans.
+2. **`domainFilters`** — per-environment name scope. Prod cannot see a name outside its own subtree.
+   Defence in depth, added because the first guard is a single string in a single file.
+
+The base default for the owner id is a **non-colliding sentinel, deliberately not `default`**: a
+future environment that forgets the knob must not silently inherit dev's identity — that is exactly
+what happened here. With the sentinel such a cluster owns nothing, so it can delete nothing.
+
+### What could NOT be tokenized, and why
+
+- **`OIDC_ISSUER_URL` / `OIDC_CALLBACK_URL` in `platform/headlamp/headlamp-oidc-credentials.sops.yaml`**
+  — SOPS-encrypted (`encrypted_regex: ^(data|stringData)$`), so the two hostnames are ciphertext.
+  Re-encrypting needs the operator's age key, rewrites the MAC over every value in the file, and
+  **cannot be verified by the offline render harness** (`kustomize` does not decrypt), so a mistake
+  would break Headlamp SSO on the live dev cluster with no pre-flight proof. Left as-is deliberately.
+  **Prod action required:** prod's Headlamp will otherwise authenticate against **dev's** Keycloak.
+  Prod needs its own copy of this Secret with `keycloak.prod.home.keithjackson.dev` and a client
+  secret from prod's own realm — which it needs regardless, since every `*.sops.yaml` client secret
+  is per-realm.
+- **`oidc-issuer-url` in the kube-apiserver flags** (`roles/k3s_server/tasks/main.yml`) — a k3s
+  **server flag** in `/etc/rancher/k3s/config.yaml`, not a manifest, so Flux post-build substitution
+  can never reach it. Correctly tokenized on the **Ansible** side instead: new role default
+  `k3s_base_domain` (dev value) + derived `k3s_oidc_issuer_url`, overridden for prod in
+  `inventory/group_vars/k3s_prod/vars.yml` — the same inventory seam as `k3s_data_dir`/`k3s_api_host`.
+  `[parity]`: `k3s_base_domain` **must** equal the same cluster's `BASE_DOMAIN`, or the apiserver
+  validates tokens against a different Keycloak than the cluster runs and kubectl SSO fails while the
+  cert-based admin kubeconfig keeps working (so it looks like "SSO is broken", not like config skew).
+
+### A pre-existing prod problem this incidentally FIXED
+
+§4 recorded that prod nodes use **public** resolvers, so the kube-apiserver would resolve its OIDC
+issuer publicly and get **dev's** Gateway VIP (192.168.1.53) — the wrong cluster on the wrong LAN,
+making prod kubectl SSO impossible, and requiring an `/etc/hosts` pin as a workaround. With prod on
+its own subdomain the public record for `keycloak.prod.home.keithjackson.dev` is published by
+**prod's own** external-dns pointing at **prod's** Gateway VIP (192.168.2.20), so the public answer
+and the split-horizon answer agree. **No `/etc/hosts` pin is needed.** This holds only once prod's
+external-dns has actually published the record; until then use the break-glass cert kubeconfig.
+
+### Verify on prod's first reconcile
+
+- [ ] **`domainFilters` zone selection.** The Cloudflare provider uses this value for *zone* lookup
+      as well as record matching, and in 2026-08 a strict-subdomain filter dropped the parent zone
+      ("no hosted zone matching record DNS Name"). If prod's external-dns logs no matching zone it
+      will create **nothing** — fail-closed, never destructive. Fallback: set prod's
+      `EXTERNALDNS_DOMAIN_FILTER` back to `keithjackson.dev` and rely on `txtOwnerId`, which is
+      sufficient on its own.
+- [ ] **Prod's wildcard cert is `*.prod.home.keithjackson.dev`** and solves DNS-01 without touching
+      dev's `_acme-challenge` records.
+- [ ] **Dev's records are untouched** after prod's tiers are resumed. Confirm dev's TXT registry
+      entries still read owner `default` and that prod's read owner `prod`.
+- [ ] **Prod's own Keycloak realm + per-service client secrets** exist (every `*.sops.yaml` client
+      secret is realm-scoped; prod cannot reuse dev's).
+
+### PHASE 2 — dev migration + apex CNAMEs (**deferred, not done**)
+
+Deliberately not attempted here: dev is **live** (18 Longhorn volumes, working SSO) and it hosts the
+Gitea that is currently this repo's GitOps source. Moving it is a migration, not a variable change.
+Required, roughly in order:
+
+1. **`BASE_DOMAIN` → `dev.home.keithjackson.dev`** in `clusters/dev`, plus `k3s_base_domain` for the
+   dev inventory group.
+2. **A new `*.dev.home.keithjackson.dev` wildcard cert** issued and Ready *before* cutover.
+3. **Keycloak issuer change.** The realm's issuer URL is baked into every token; every OIDC client
+   (`grafana`, `gitea`, `nextcloud`, `semaphore`, `headlamp`, `crafty`, `pihole`, `jackett`,
+   `qbittorrent`, `radarr`, `sonarr`, and the `kubernetes` public client) needs its redirect URIs
+   updated in the same change, and the k3s apiserver `oidc-issuer-url` with it — which means a
+   **rolling k3s restart** on the dev servers.
+4. **kubectl SSO re-issue** for every user (issuer changed).
+5. **`gitea.dev.home.keithjackson.dev`** — this is the GitOps source URL in
+   `clusters/dev/kustomization.yaml`. Flux must be able to resolve and trust the new host *before*
+   the URL is switched, or dev stops reconciling and the change cannot be pushed to fix itself.
+   Sequence this last, and keep the bootstrap Gitea available as the recovery path (docs/05).
+6. **`EXTERNALDNS_TXT_OWNER_ID` for dev.** Once dev's names move, its existing `default`-owned
+   records are for names dev no longer serves; retire them deliberately rather than letting a
+   renamed owner orphan them.
+7. **Apex CNAMEs** `<svc>.home.keithjackson.dev` → `<svc>.prod.home.keithjackson.dev`. This is what
+   actually **moves user traffic to prod**, and it must not land before prod serves those services.
+8. **The L2 behavioural conformance tests hardcode `home.keithjackson.dev`** (`dns-exposure`,
+   `dns-internal-resolution`, `gateway-ingress-reachable`, `oidc-sso-grafana`, `sso-forward-auth`,
+   `trusted-tls`). They are test inputs, not reconciled manifests, so they were left alone in phase 1
+   — but they will fail against prod and must be parameterized before §9's "suite passes green
+   against the prod cluster" can be claimed.

infra/ansible/inventory/group_vars/k3s_prod/vars.yml

@@ -289,19 +289,40 @@ node_baseline_dns_servers:
   #   LAN clients       -> PiHole. Not a cluster concern.
   #   nodes             -> THIS FILE. Public resolvers, so the zone resolves PUBLICLY.
   #
-  # THE ONE THING THAT BREAKS: the k3s API SERVER is a node process and carries
-  # `oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab`
-  # (roles/k3s_server/tasks/main.yml). It reaches the issuer through the NODE resolver, so on
-  # prod it would get the PUBLIC record instead of the prod Gateway VIP — breaking kubectl
-  # OIDC SSO. Dev avoids this because its node resolver is PiHole, which answers the zone
-  # with the dev Gateway VIP.
+  # THE k3s API SERVER is a node process and carries `oidc-issuer-url=...` (built from
+  # k3s_base_domain, below). It reaches the issuer through the NODE resolver, i.e. through
+  # these PUBLIC upstreams — not through CoreDNS's split-horizon answer.
   #
-  # NOT blocking today (prod has no cluster and no Keycloak yet). Options, in
-  # PROD-PROVISIONING.md §4: a narrow Ansible-managed /etc/hosts entry pinning
-  # keycloak.home.keithjackson.dev -> 192.168.2.20 on prod nodes; or a local resolver on the
-  # prod LAN serving the zone; or run prod on the cert kubeconfig and defer OIDC.
+  # THIS USED TO BE BROKEN, AND THE 2026-09-04 SUBDOMAIN SPLIT FIXED IT. While prod was
+  # authored to serve dev's hostnames, a prod node resolving keycloak.home.keithjackson.dev
+  # publicly got DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN, so
+  # kubectl OIDC SSO on prod could not work at all. Now prod's issuer is
+  # keycloak.prod.home.keithjackson.dev, and the public record for that name is published by
+  # PROD's own external-dns pointing at PROD's Gateway VIP (192.168.2.20). The public answer
+  # and the split-horizon answer therefore AGREE, and no /etc/hosts pin is needed.
+  #
+  # STILL TO VERIFY on prod bring-up (PROD-PROVISIONING.md §10): this holds only once prod's
+  # external-dns has actually published the record. Until then the issuer does not resolve
+  # from the nodes and kubectl SSO fails — the cert-based admin kubeconfig is the break-glass
+  # path, and it always works. Ordering, not a design gap.
 node_baseline_dns_search: []
 
+# --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
+# PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/
+# kustomization.yaml — the same one-fact-in-two-places contract as k3s_data_dir and
+# k3s_api_host. The Flux side cannot supply this: the kube-apiserver's oidc-issuer-url is a
+# k3s server FLAG written to /etc/rancher/k3s/config.yaml, not a manifest, so
+# postBuild.substituteFrom never sees it. Full rationale in
+# roles/k3s_server/defaults/main.yml, where k3s_oidc_issuer_url is derived from this.
+#
+# WHY PROD MOVED (infra/PROD-PROVISIONING.md §10): both clusters were authored to serve the
+# SAME hostnames from the SAME Cloudflare zone, and on 2026-09-04 prod's external-dns
+# deleted every one of dev's public records. Dev stays on home.keithjackson.dev (it is LIVE);
+# prod takes prod.home.keithjackson.dev. Consequence here: prod's Keycloak issuer is
+# https://keycloak.prod.home.keithjackson.dev/realms/homelab, so the `kubernetes` OIDC client
+# in prod's OWN Keycloak realm is what prod's kubectl SSO authenticates against.
+k3s_base_domain: "prod.home.keithjackson.dev"
+
 # --- Flux bootstrap: WHICH overlay this environment reconciles from (R-ENV-02) -----
 # roles/flux_bootstrap defaults flux_bootstrap_env_path to `clusters/dev` (dev is LIVE
 # and group_vars/all.yml carries no override, so dev keeps resolving exactly that).

infra/ansible/roles/cilium/defaults/main.yml

@@ -15,9 +15,20 @@ cilium_namespace: "kube-system"
 # Cilium 1.20.1 targets Gateway API v1.6.1.
 gateway_api_version: "v1.6.1"
 gateway_api_crd_urls:
+  # ONE URL, deliberately. As of v1.6.0 the STANDARD channel bundle already contains
+  # TLSRoute (plus TCPRoute, UDPRoute, ListenerSet and BackendTLSPolicy), so the separate
+  # experimental TLSRoute manifest this list used to carry is redundant — and worse than
+  # redundant: v1.6 also ships a ValidatingAdmissionPolicy,
+  # `safe-upgrades.gateway.networking.k8s.io`, which REJECTS an experimental-channel CRD
+  # applied over a standard-channel one. Applying the old second URL therefore fails hard:
+  #   ValidatingAdmissionPolicy 'safe-upgrades.gateway.networking.k8s.io' ... denied request:
+  #   Installing experimental CRDs on top of standard channel CRDs is prohibited by default.
+  # Verified 2026-09-04 on the prod bootstrap: standard-install.yaml v1.6.1 alone produces the
+  # exact 10 gateway.networking.k8s.io CRDs the LIVE dev cluster is running, tlsroutes included
+  # and annotated channel=standard, bundle-version=v1.6.1 — so this is parity with dev, not a
+  # reduction. Dev needs no reapply; it already has them. If gateway_api_version is ever moved
+  # BACK below v1.6.0, the experimental TLSRoute URL has to come back with it.
   - "https://github.com/kubernetes-sigs/gateway-api/releases/download/{{ gateway_api_version }}/standard-install.yaml"
-  # TLSRoute (experimental) is required by Cilium's Gateway API support.
-  - "https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/{{ gateway_api_version }}/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml"
 
 # Local kubeconfig fetched from the primary. *.kubeconfig is gitignored. The FILENAME
 # comes from cluster_kubeconfig_name (group_vars) so a prod run cannot overwrite the dev

infra/ansible/roles/k3s_server/defaults/main.yml

@@ -66,3 +66,33 @@ k3s_tls_sans: >-
   {{ ([k3s_api_host, kube_vip_address, '127.0.0.1']
       + (groups[k3s_servers_group] | map('extract', hostvars, 'k3s_node_ip') | list))
      | unique }}
+
+# --- Keycloak OIDC issuer for kubectl SSO (R-IDP-03, R-ENV-02) -----------------------
+# THE ANSIBLE-SIDE HALF OF ${BASE_DOMAIN}, and it cannot be the Flux-side half.
+#
+# The kube-apiserver's `oidc-issuer-url` is a k3s SERVER FLAG written into
+# /etc/rancher/k3s/config.yaml by this role — it is NOT a Kubernetes manifest, so Flux never
+# sees it and `postBuild.substituteFrom: cluster-vars` cannot reach it. The platform hostname
+# suffix therefore exists as ONE FACT IN TWO PLACES, and they must agree:
+#
+#   Flux side     clusters/base/cluster-vars.yaml   BASE_DOMAIN      (per-overlay)
+#   Ansible side  THIS variable                     k3s_base_domain  (per-inventory-group)
+#
+# [parity] k3s_base_domain MUST equal the BASE_DOMAIN the SAME cluster's Flux overlay sets.
+# If they disagree, the apiserver validates ID tokens against a DIFFERENT Keycloak than the
+# one the cluster actually runs: every `kubectl` SSO login fails with an issuer mismatch,
+# while the cert-based admin kubeconfig keeps working — so it looks like "SSO is broken",
+# not like a config skew. The break-glass cert kubeconfig is why this is recoverable.
+#
+# THE DEFAULT IS THE DEV VALUE, restated explicitly rather than left implicit — the same
+# convention as k3s_data_dir above and cluster-vars' GATEWAY_VIP / LONGHORN_DATA_PATH: the
+# base default is the known-good LIVE value, so dev's rendered config.yaml is byte-for-byte
+# what it already is and the Restart handler does NOT fire on a re-run.
+#
+# PROD OVERRIDES IT in inventory/group_vars/k3s_prod/vars.yml — the same inventory seam
+# k3s_data_dir / k3s_api_host / kube_vip_address use, so a FOURTH prod server inherits it
+# from group membership rather than from anyone remembering.
+k3s_base_domain: "home.keithjackson.dev"
+
+# Derived, so the realm path is stated once. Override k3s_base_domain, not this.
+k3s_oidc_issuer_url: "https://keycloak.{{ k3s_base_domain }}/realms/homelab"

infra/ansible/roles/k3s_server/tasks/main.yml

@@ -75,13 +75,19 @@
       # OIDC (Keycloak SSO for kubectl). The API server trusts ID tokens from the homelab realm;
       # the `kubernetes` PUBLIC client (PKCE) + kubelogin mint them, and RBAC binds the `groups`
       # claim (platform/keycloak/k8s-oidc-rbac.yaml: oidc:platform-admins->cluster-admin,
-      # oidc:viewers->view). Nodes resolve keycloak.home.keithjackson.dev to the Cilium gateway VIP
-      # (192.168.1.53) and trust its public LE-prod cert, so NO oidc-ca-file is needed. The `oidc:`
+      # oidc:viewers->view). Nodes resolve keycloak.<k3s_base_domain> to the Cilium gateway VIP
+      # and trust its public LE-prod cert, so NO oidc-ca-file is needed. The `oidc:`
       # prefixes namespace federated identities away from cert / service-account subjects.
       # ADDITIVE + break-glass-safe: the cert-based admin kubeconfig keeps full access with or
       # without Keycloak, so a bad realm/OIDC state can never lock you out of the cluster.
+      #
+      # NOT a Flux ${BASE_DOMAIN} token: this is a k3s SERVER FLAG, not a manifest, so
+      # postBuild.substituteFrom never touches it. k3s_oidc_issuer_url derives from
+      # k3s_base_domain (defaults/main.yml), whose [parity] contract with the cluster's
+      # BASE_DOMAIN is documented there. Dev's default renders the value it already runs, so
+      # config.yaml is unchanged and the Restart handler does not fire.
       kube-apiserver-arg:
-        - "oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab"
+        - "oidc-issuer-url={{ k3s_oidc_issuer_url }}"
         - "oidc-client-id=kubernetes"
         - "oidc-username-claim=email"
         - "oidc-username-prefix=oidc:"
@@ -98,15 +104,20 @@
     INSTALL_K3S_VERSION: "{{ k3s_version }}"
   when: k3s_server_mode == 'primary'
 
+# PATH FOLLOWS --data-dir. k3s writes the node-token to <data-dir>/server/token, NOT to
+# the literal /var/lib/rancher/k3s. On dev k3s_data_dir IS /var/lib/rancher/k3s (the role
+# default, restated explicitly above) so this is byte-for-byte the old path; on prod
+# --data-dir is /data/k3s (group_vars/k3s_prod) and the hardcoded path did not exist at all
+# — this task would have timed out after 180s and the two secondaries could never join.
 - name: Wait for the k3s node-token to exist (primary)
   ansible.builtin.wait_for:
-    path: /var/lib/rancher/k3s/server/token
+    path: "{{ k3s_data_dir }}/server/token"
     timeout: 180
   when: k3s_server_mode == 'primary'
 
 - name: Read the k3s node-token (primary)
   ansible.builtin.slurp:
-    src: /var/lib/rancher/k3s/server/token
+    src: "{{ k3s_data_dir }}/server/token"
   register: k3s_token_raw
   when: k3s_server_mode == 'primary'
 
@@ -126,9 +137,11 @@
 # "The primary" is resolved through k3s_primary_group (group_vars: `k3s_primary` for
 # dev, `k3s_prod_primary` for prod), never a literal group name — so a prod secondary
 # delegates to the PROD primary using this same role, unchanged (R-ENV-02).
+# k3s_data_dir is a CLUSTER-WIDE value (group_vars for prod, role default for dev), so the
+# secondary's own copy of it is also the primary's data dir — correct for the delegated read.
 - name: Fetch the k3s node-token directly from the primary (secondary)
   ansible.builtin.slurp:
-    src: /var/lib/rancher/k3s/server/token
+    src: "{{ k3s_data_dir }}/server/token"
   delegate_to: "{{ groups[k3s_primary_group][0] }}"
   register: k3s_primary_token_raw
   when:

infra/ansible/roles/kube_vip/defaults/main.yml

@@ -31,4 +31,14 @@ kube_vip_retry_period: 2
 
 # k3s auto-deploy manifests dir: k3s applies anything here on start, so kube-vip
 # is reproducible and independent of Flux/Helm.
-kube_vip_manifest_path: "/var/lib/rancher/k3s/server/manifests/kube-vip.yaml"
+#
+# THE PATH FOLLOWS --data-dir. k3s reads auto-deploy manifests from
+# <data-dir>/server/manifests, so this must be derived from the cluster-wide
+# k3s_data_dir (roles/k3s_server turns the same variable into --data-dir), not from the
+# literal /var/lib/rancher/k3s. Dev leaves k3s_data_dir unset outside roles/k3s_server's
+# own defaults, so the fallback below reproduces dev's existing path exactly; prod sets
+# k3s_data_dir: /data/k3s in group_vars/k3s_prod, where the old hardcoded path was simply
+# not the directory k3s watches — the manifest would have been written and silently never
+# applied, and the control-plane VIP would never have come up.
+kube_vip_manifest_path: >-
+  {{ k3s_data_dir | default('/var/lib/rancher/k3s') }}/server/manifests/kube-vip.yaml

infra/ansible/roles/kube_vip/tasks/main.yml

@@ -1,13 +1,22 @@
 ---
 # Deploy kube-vip (control-plane HA VIP) via the k3s auto-deploy manifests dir.
-# k3s applies /var/lib/rancher/k3s/server/manifests/*.yaml on start, so this is
-# reproducible and independent of Flux. The same manifest is written on every
+# k3s applies <data-dir>/server/manifests/*.yaml on start, so this is
+# reproducible and independent of Flux (see kube_vip_manifest_path in defaults/main.yml —
+# the dir follows k3s's --data-dir, which prod moves to /data/k3s). The same manifest is written on every
 # server so the DaemonSet survives the loss of any single node.
 
-- name: Detect the LAN interface holding the node's 192.168.1.5x address
+# WHICH INTERFACE kube-vip ARPs THE VIP ON. Detected from this node's OWN k3s_node_ip
+# (inventory, per host) rather than a hardcoded subnet: the previous awk matched the
+# literal 192.168.1.5[0-2], i.e. DEV's node addresses, so on prod (192.168.2.10/.11/.12)
+# it returned nothing and the assert below aborted the play. k3s_node_ip is the same
+# address k3s advertises with --node-ip and is defined for dev and prod alike, so this
+# resolves eno1 on prod and the identical dev interface it always resolved before.
+# `ip addr show to <ip>/32` does the matching in iproute2 instead of a regex over dotted
+# quads (where `.` would match any character).
+- name: Detect the LAN interface holding this node's k3s_node_ip
   ansible.builtin.shell: >-
     set -o pipefail;
-    ip -o -4 addr show | awk '/192\.168\.1\.5[0-2]\//{print $2; exit}'
+    ip -o -4 addr show to {{ k3s_node_ip }}/32 | awk '{print $2; exit}'
   args:
     executable: /bin/bash
   register: kube_vip_iface_detect
@@ -25,7 +34,7 @@
       - kube_vip_vip_interface is defined
       - kube_vip_vip_interface | length > 0
     fail_msg: >-
-      Could not detect the 192.168.1.5x LAN interface for kube-vip on
+      Could not detect the interface holding {{ k3s_node_ip }} for kube-vip on
       {{ inventory_hostname }} — set kube_vip_interface explicitly.
 
 - name: Ensure the k3s auto-deploy manifests directory exists