feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain

INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources, and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled to 0; verified restored against Cloudflare directly (my first check went through a pfSense override and falsely showed healthy — local resolvers are not evidence about public DNS). THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put "never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified --limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored manifests reached across a third party and destroyed dev's state. Cluster-to-cluster isolation was verified; the dependency that mattered was cluster-to-Cloudflare. Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod -> *.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home. PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. + apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and 12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every user, and moves gitea.home. — which is BOTH the GitOps source and the current push target. BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs. Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles" has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized: Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15 links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove --oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11 catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose occurrences and non-reconciled paths. TWO THAT CANNOT BE TOKENIZED, flagged not forced: - headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt), so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy anyway — client secrets are realm-scoped. - kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url, overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets prod.home. EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment — dev renders exactly `default` (the chart default it was already using, so its existing records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL, deliberately not `default`, so a future environment that forgets the knob owns nothing and can delete nothing rather than silently inheriting dev's identity. I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1 (DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE — ExternalDNS generates the record then skips it with "no hosted zone matching record DNS Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev. Isolation comes from the subdomain split plus txtOwnerId, which is what the incident actually needed and what a domain filter cannot provide. DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly. POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12 hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod., and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's. L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS. PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied. Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
feat(prod): temp Gitea source + wireless disabled + flux bootstrap path
2026-09-04 22:43:54 -05:00 · 2026-09-04 18:53:17 -05:00
65 changed files with 1286 additions and 261 deletions
@@ -102,7 +102,7 @@ spec:
            userData: |-
              #cloud-config
              hostname: unifi
-              fqdn: unifi.home.keithjackson.dev
+              fqdn: unifi.${BASE_DOMAIN}
              package_update: true
              package_upgrade: false
              packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl, jq, cloud-guest-utils]
@@ -46,7 +46,7 @@ metadata:
    lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
    # external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP.
    # This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443.
-    external-dns.alpha.kubernetes.io/hostname: "unifi.home.keithjackson.dev"
+    external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
 spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
@@ -90,9 +90,9 @@ spec:
              name: gitea-bootstrap
              key: secret-key
        - name: GITEA__server__DOMAIN
-          value: gitea.home.keithjackson.dev
+          value: gitea.${BASE_DOMAIN}
        - name: GITEA__server__ROOT_URL
-          value: https://gitea.home.keithjackson.dev/
+          value: https://gitea.${BASE_DOMAIN}/
        - name: GITEA__actions__ENABLED
          value: 'true'
        - name: GITEA__actions__DEFAULT_ACTIONS_URL
@@ -127,7 +127,7 @@ spec:
              command:
              - /bin/sh
              - -c
-              - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n  i=$((i + 1))\n  [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n  echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n  n=0\n  until su-exec git gitea admin user create \\\n    --username \"$GITEA_ADMIN_USERNAME\" \\\n    --password \"$GITEA_ADMIN_PASSWORD\" \\\n    --email \"[email protected]\" \\\n    --admin --must-change-password=false; do\n    n=$((n + 1))\n    [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n    sleep 3\n  done\n  echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n  echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n  # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n  d=0; disc_ok=0\n  while [ \"$d\" -lt 45 ]; do\n    if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n      disc_ok=1;\
+              - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n  i=$((i + 1))\n  [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n  echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n  n=0\n  until su-exec git gitea admin user create \\\n    --username \"$GITEA_ADMIN_USERNAME\" \\\n    --password \"$GITEA_ADMIN_PASSWORD\" \\\n    --email \"[email protected]\" \\\n    --admin --must-change-password=false; do\n    n=$((n + 1))\n    [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n    sleep 3\n  done\n  echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n  echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n  # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n  d=0; disc_ok=0\n  while [ \"$d\" -lt 45 ]; do\n    if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n      disc_ok=1;\
                \ break\n    fi\n    d=$((d + 1)); sleep 2\n  done\n  [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n  # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n  r=0; reg_ok=0\n  while [ \"$r\" -lt 10 ]; do\n    oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n    if [ -n \"$oidc_id\" ]; then\n      su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    else\n      su-exec git gitea admin auth add-oauth \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    fi\n    # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n    if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n      reg_ok=1; break\n    fi\n    r=$((r + 1)); sleep 3\n  done\n  if [ \"$reg_ok\" = 1 ]; then\n    echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n  else\n    gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n  fi\nfi\n# A postStart hook MUST exit 0 \u2014 a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but\n# it is made EXPLICIT so the guarantee does not depend on whatever statement happens to\n# be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,\n# the moment its OIDC guard was fixed and the block actually ran for the first time.\nexit 0\n"
      volumes:
      - name: data
@@ -207,7 +207,7 @@ spec:
    namespace: gateway
    sectionName: https
  hostnames:
-  - gitea.home.keithjackson.dev
+  - gitea.${BASE_DOMAIN}
  rules:
  - matches:
    - path:
@@ -91,16 +91,16 @@ spec:
        - --http-address=0.0.0.0:4180
        - --reverse-proxy=true
        - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
        - --upstream=http://127.0.0.1:9117
        - --email-domain=*
        - --scope=openid email profile
        - --oidc-groups-claim=groups
        - --allowed-group=media-users
        - --skip-provider-button=true
-        - --redirect-url=https://jackett.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
-        - --whitelist-domain=.home.keithjackson.dev
+        - --redirect-url=https://jackett.${BASE_DOMAIN}/oauth2/callback
+        - --cookie-domain=.${BASE_DOMAIN}
+        - --whitelist-domain=.${BASE_DOMAIN}
        - --cookie-secure=true
        - --pass-access-token=true
        - --set-xauthrequest=true
@@ -273,7 +273,7 @@ spec:
    namespace: gateway
    sectionName: https
  hostnames:
-  - jackett.home.keithjackson.dev
+  - jackett.${BASE_DOMAIN}
  rules:
  - matches:
    - path:
@@ -91,16 +91,16 @@ spec:
        - --http-address=0.0.0.0:4180
        - --reverse-proxy=true
        - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
        - --upstream=http://127.0.0.1:80
        - --email-domain=*
        - --scope=openid email profile
        - --oidc-groups-claim=groups
        - --allowed-group=platform-admins
        - --skip-provider-button=true
-        - --redirect-url=https://pihole.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
-        - --whitelist-domain=.home.keithjackson.dev
+        - --redirect-url=https://pihole.${BASE_DOMAIN}/oauth2/callback
+        - --cookie-domain=.${BASE_DOMAIN}
+        - --whitelist-domain=.${BASE_DOMAIN}
        - --cookie-secure=true
        - --pass-access-token=true
        - --set-xauthrequest=true
@@ -229,7 +229,7 @@ spec:
    namespace: gateway
    sectionName: https
  hostnames:
-  - pihole.home.keithjackson.dev
+  - pihole.${BASE_DOMAIN}
  rules:
  - matches:
    - path:
@@ -183,7 +183,7 @@ spec:
    namespace: gateway
    sectionName: https
  hostnames:
-  - plex.home.keithjackson.dev
+  - plex.${BASE_DOMAIN}
  rules:
  - matches:
    - path:
@@ -91,16 +91,16 @@ spec:
        - --http-address=0.0.0.0:4180
        - --reverse-proxy=true
        - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
        - --upstream=http://127.0.0.1:8080
        - --email-domain=*
        - --scope=openid email profile
        - --oidc-groups-claim=groups
        - --allowed-group=media-users
        - --skip-provider-button=true
-        - --redirect-url=https://qbittorrent.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
-        - --whitelist-domain=.home.keithjackson.dev
+        - --redirect-url=https://qbittorrent.${BASE_DOMAIN}/oauth2/callback
+        - --cookie-domain=.${BASE_DOMAIN}
+        - --whitelist-domain=.${BASE_DOMAIN}
        - --cookie-secure=true
        - --pass-access-token=true
        - --set-xauthrequest=true
@@ -273,7 +273,7 @@ spec:
    namespace: gateway
    sectionName: https
  hostnames:
-  - qbittorrent.home.keithjackson.dev
+  - qbittorrent.${BASE_DOMAIN}
  rules:
  - matches:
    - path:
@@ -165,7 +165,7 @@ spec:
              exec /sbin/tini -- /opt/act/run.sh
          env:
            - name: GITEA_INSTANCE_URL
-              value: https://gitea.home.keithjackson.dev
+              value: https://gitea.${BASE_DOMAIN}
            - name: GITEA_RUNNER_REGISTRATION_TOKEN
              valueFrom:
                secretKeyRef:
@@ -104,13 +104,13 @@ spec:
              name: nextcloud-bootstrap
              key: admin-password
        - name: NEXTCLOUD_TRUSTED_DOMAINS
-          value: nextcloud.home.keithjackson.dev
+          value: nextcloud.${BASE_DOMAIN}
        - name: OVERWRITEHOST
-          value: nextcloud.home.keithjackson.dev
+          value: nextcloud.${BASE_DOMAIN}
        - name: OVERWRITEPROTOCOL
          value: https
        - name: OVERWRITECLIURL
-          value: https://nextcloud.home.keithjackson.dev
+          value: https://nextcloud.${BASE_DOMAIN}
        - name: NEXTCLOUD_OIDC_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
@@ -125,9 +125,9 @@ spec:
              command:
              - /bin/sh
              - -c
-              - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n  echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n#     maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n#     stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n  i=$((i + 1))\n  [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n  || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"\
-                [nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n  exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n  OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n  if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n    disc_ok=1; break\n  fi\n  d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n#     HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n#     single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n#     form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n#     occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n#     (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n  su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n    --clientid=nextcloud \\\n    --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n    --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n    --scope='openid email profile' \\\n    --unique-uid=0 \\\n    --mapping-uid=preferred_username \\\n    --mapping-email=email \\\n    --mapping-display-name=name \\\n    --group-provisioning=1 \\\n    --mapping-groups=groups\" >/dev/null 2>&1 || true\n  if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n    reg_ok=1; break\n  fi\n  r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n  echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n  nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n#     NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n#     login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n#     built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n#     start: occ group:adduser is a no-op for an existing member. A user is promotable only\n#     AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n#     rights on their next login + the following pod start (or run `occ group:adduser admin\n#     <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n#     passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n  current_admins=$(parse_group admin)\n  missing=0\n  for uid in $(parse_group \"\
-                $ADMIN_SRC_GROUP\"); do\n    if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n      continue\n    fi\n    missing=1\n    if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n      echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n    fi\n  done\n  [ \"$missing\" = 0 ] && break\n  p=$((p + 1)); sleep 2\ndone\nif [ \"$p\" -ge 15 ]; then\n  nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\nfi\n# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:\n# the final statement used to be `[ \"$p\" -ge 15 ] && nextcloud_oidc_fail ...`, so on the\n# SUCCESS path the test returned 1 and became the script's exit status. That was latent for\n# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard\n# exposed it immediately. Every failure path here logs loudly and writes a marker file and\n# deliberately does NOT crash the pod \u2014 the break-glass local admin must stay usable.\nexit 0\n"
+              - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n  echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n#     maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n#     stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n  i=$((i + 1))\n  [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n  || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[nextcloud-oidc]\
+                \ client secret unset; skipping OIDC (break-glass admin only)\"\n  exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n  OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n  if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n    disc_ok=1; break\n  fi\n  d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n#     HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n#     single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n#     form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n#     occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n#     (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n  su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n    --clientid=nextcloud \\\n    --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n    --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n    --scope='openid email profile' \\\n    --unique-uid=0 \\\n    --mapping-uid=preferred_username \\\n    --mapping-email=email \\\n    --mapping-display-name=name \\\n    --group-provisioning=1 \\\n    --mapping-groups=groups\" >/dev/null 2>&1 || true\n  if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n    reg_ok=1; break\n  fi\n  r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n  echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n  nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n#     NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n#     login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n#     built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n#     start: occ group:adduser is a no-op for an existing member. A user is promotable only\n#     AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n#     rights on their next login + the following pod start (or run `occ group:adduser admin\n#     <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n#     passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n  current_admins=$(parse_group admin)\n  missing=0\n  for uid in $(parse_group \"$ADMIN_SRC_GROUP\"\
+                ); do\n    if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n      continue\n    fi\n    missing=1\n    if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n      echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n    fi\n  done\n  [ \"$missing\" = 0 ] && break\n  p=$((p + 1)); sleep 2\ndone\nif [ \"$p\" -ge 15 ]; then\n  nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\nfi\n# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:\n# the final statement used to be `[ \"$p\" -ge 15 ] && nextcloud_oidc_fail ...`, so on the\n# SUCCESS path the test returned 1 and became the script's exit status. That was latent for\n# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard\n# exposed it immediately. Every failure path here logs loudly and writes a marker file and\n# deliberately does NOT crash the pod \u2014 the break-glass local admin must stay usable.\nexit 0\n"
      volumes:
      - name: data
        persistentVolumeClaim:
@@ -228,7 +228,7 @@ spec:
    namespace: gateway
    sectionName: https
  hostnames:
-  - nextcloud.home.keithjackson.dev
+  - nextcloud.${BASE_DOMAIN}
  rules:
  - matches:
    - path:
@@ -91,16 +91,16 @@ spec:
        - --http-address=0.0.0.0:4180
        - --reverse-proxy=true
        - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
        - --upstream=http://127.0.0.1:7878
        - --email-domain=*
        - --scope=openid email profile
        - --oidc-groups-claim=groups
        - --allowed-group=media-users
        - --skip-provider-button=true
-        - --redirect-url=https://radarr.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
-        - --whitelist-domain=.home.keithjackson.dev
+        - --redirect-url=https://radarr.${BASE_DOMAIN}/oauth2/callback
+        - --cookie-domain=.${BASE_DOMAIN}
+        - --whitelist-domain=.${BASE_DOMAIN}
        - --cookie-secure=true
        - --pass-access-token=true
        - --set-xauthrequest=true
@@ -295,7 +295,7 @@ spec:
    namespace: gateway
    sectionName: https
  hostnames:
-  - radarr.home.keithjackson.dev
+  - radarr.${BASE_DOMAIN}
  rules:
  - matches:
    - path:
@@ -110,7 +110,7 @@ spec:
        - name: SEMAPHORE_ADMIN_NAME
          value: Semaphore Admin
        - name: SEMAPHORE_ADMIN_EMAIL
-          value: admin@home.keithjackson.dev
+          value: admin@${BASE_DOMAIN}
        - name: SEMAPHORE_ACCESS_KEY_ENCRYPTION
          valueFrom:
            secretKeyRef:
@@ -231,7 +231,7 @@ spec:
    namespace: gateway
    sectionName: https
  hostnames:
-  - semaphore.home.keithjackson.dev
+  - semaphore.${BASE_DOMAIN}
  rules:
  - matches:
    - path:
@@ -91,16 +91,16 @@ spec:
        - --http-address=0.0.0.0:4180
        - --reverse-proxy=true
        - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
        - --upstream=http://127.0.0.1:8989
        - --email-domain=*
        - --scope=openid email profile
        - --oidc-groups-claim=groups
        - --allowed-group=media-users
        - --skip-provider-button=true
-        - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
-        - --whitelist-domain=.home.keithjackson.dev
+        - --redirect-url=https://sonarr.${BASE_DOMAIN}/oauth2/callback
+        - --cookie-domain=.${BASE_DOMAIN}
+        - --whitelist-domain=.${BASE_DOMAIN}
        - --cookie-secure=true
        - --pass-access-token=true
        - --set-xauthrequest=true
@@ -295,7 +295,7 @@ spec:
    namespace: gateway
    sectionName: https
  hostnames:
-  - sonarr.home.keithjackson.dev
+  - sonarr.${BASE_DOMAIN}
  rules:
  - matches:
    - path:
@@ -223,7 +223,7 @@ spec:
    namespace: gateway
    sectionName: https
  hostnames:
-  - seerr.home.keithjackson.dev
+  - seerr.${BASE_DOMAIN}
  rules:
  - matches:
    - path:
@@ -1,6 +1,9 @@
 #!/usr/bin/env bash
 #
-# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint for the dev cluster.
+# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint. ENVIRONMENT-AGNOSTIC: which overlay
+# it applies is FLUX_ENV_PATH (default clusters/dev). roles/flux_bootstrap exports that from
+# `flux_bootstrap_env_path`, overridden to clusters/prod in group_vars/k3s_prod/vars.yml — so the
+# same script bootstraps dev and prod with no fork (R-ENV-02).
 #
 # Phase 1, T-1.4 · ADR-0004 (Flux) · ADR-0007 (SOPS+age) · R-GITOPS-01 · R-SEC-01.
 # =============================================================================
@@ -33,7 +36,7 @@
 #   2. sops-age Secret  — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so
 #                         decryption works on first pass (../sops/age-key-secret.md). This script
 #                         only VERIFIES it exists; it never handles key material.
-#   3. `kubectl apply -k clusters/dev` — materialises the authored reconciliation graph:
+#   3. `kubectl apply -k ${FLUX_ENV_PATH}` — materialises the authored reconciliation graph:
 #                         GitRepository `flux-system` (source) -> Kustomization `platform` ->
 #                         `platform-config` -> the app-wave Kustomizations `apps-wave-0/1/2`
 #                         (chained dependsOn) + `apps-vms`, with cluster-vars overlay
@@ -52,7 +55,10 @@
 set -euo pipefail

 # --- config (env-overridable) ------------------------------------------------
-FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}"     # dev is the ONLY Phase-1 build target
+# Which environment overlay to apply. Dev is the DEFAULT (it was the only Phase-1 build target);
+# prod passes clusters/prod via roles/flux_bootstrap. Changing the default would silently retarget
+# every existing caller — override it, never edit it.
+FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}"
 FLUX_NAMESPACE="${FLUX_NAMESPACE:-flux-system}"
 SOPS_SECRET_NAME="${SOPS_SECRET_NAME:-sops-age}"
 # The two Phase-4 image-automation controllers are installed alongside the default set so their CRDs
@@ -62,7 +68,7 @@ FLUX_COMPONENTS_EXTRA="${FLUX_COMPONENTS_EXTRA:-image-reflector-controller,image
 script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/../.." && pwd))"

-echo "==> Flux + SOPS bootstrap (dev) — path=${FLUX_ENV_PATH}"
+echo "==> Flux + SOPS bootstrap — path=${FLUX_ENV_PATH}"

 # --- 0. tooling present (from infra/devcontainer; versions pinned there) ------
 for bin in flux kubectl; do
@@ -134,7 +140,7 @@ if ! kubectl -n "${FLUX_NAMESPACE}" get secret "${SOPS_SECRET_NAME}" >/dev/null
  exit 1
 fi

-# --- 3. apply the authored dev entrypoint ------------------------------------
+# --- 3. apply the authored environment entrypoint ----------------------------
 echo "==> [3/3] kubectl apply -k ${FLUX_ENV_PATH}"
 kubectl apply -k "${repo_root}/${FLUX_ENV_PATH}"

@@ -1,11 +1,30 @@
 #!/usr/bin/env bash
 # bootstrap/gitea/bootstrap-gitea.sh — run a SINGLE Gitea container on the HOST (outside the
-# cluster) as the Phase-1 GitOps bootstrap source. While the cluster is built, Flux (in-cluster)
-# reconciles this repo from the host Gitea over the k3s-dev NAT gateway. This is the "local
-# bootstrap source" that migrates to in-cluster Gitea in Phase 6 (docs/05 "Source-of-truth",
-# CONTRIBUTING.md).
+# cluster) as the GitOps bootstrap source. While a cluster is built, Flux (in-cluster) reconciles
+# this repo from the host Gitea. This is the "local bootstrap source" that migrates to in-cluster
+# Gitea at cutover (docs/05 "Source-of-truth", CONTRIBUTING.md). Dev has already cut over; PROD is
+# using it now.
 #
-# Run by the OPERATOR at bring-up — NOT during author-only, NOT by any agent.
+# HOW THE CLUSTER REACHES THE HOST — read this before setting HOST_NAT_IP.
+# It is an ADDRESS OF THIS HOST as seen from the cluster nodes, and it differs per environment:
+#
+#   PROD (bare metal, current use):  HOST_NAT_IP=192.168.2.132
+#       The prod nodes (192.168.2.10/.11/.12) and this workstation are on the SAME LAN. There is
+#       NO NAT hop of any kind, and NO firewall change was needed — verified 2026-09-04 from all
+#       three nodes. The firewalld note further down is a DEV-ERA libvirt concern; it does not
+#       apply to prod, so do not "fix" prod by opening zones.
+#
+#   DEV (libvirt VMs, HISTORICAL):   HOST_NAT_IP=10.73.0.1 was the k3s-dev libvirt NAT gateway.
+#       *** THAT NETWORK IS RETIRED. 10.73.0.1 is stale and must not be reused. *** Dev's own
+#       bootstrap Gitea was later reached on 192.168.2.103:3000 and is retained STOPPED as
+#       recovery (see the comment on the GitRepository patch in clusters/dev/kustomization.yaml).
+#
+# WHO RUNS THIS: the OPERATOR, or an agent the operator has EXPLICITLY DIRECTED to stand up a
+# bootstrap source for a specific bring-up (as happened 2026-09-04 for prod). It is NOT part of
+# author-only work and must never be fired off casually or "to see if it works": it starts a
+# long-lived container, generates and writes admin credentials to $HOME, creates an org and repo,
+# and PUSHES THE CURRENT HEAD to that repo's `main` — which is the revision a cluster pointed at
+# it will then reconcile. Know which HEAD you are publishing before you run it.
 set -euo pipefail
 ENGINE="${ENGINE:-podman}"
 GITEA_IMAGE="${GITEA_IMAGE:-docker.io/gitea/gitea:1.22}"   # pinned; bump if needed
@@ -16,11 +35,24 @@ SSH_PORT="${GITEA_SSH_PORT:-2222}"
 # subuid that can't write a host-owned dir, so Gitea fails to create /data/gitea/conf/app.ini.
 DATA_VOLUME="${GITEA_DATA_VOLUME:-gitea-bootstrap-data}"
 DATA_DIR="podman volume ${DATA_VOLUME}"   # display label only (data lives in the named volume)
-# The cluster VMs reach the host on the k3s-dev NAT gateway (10.73.0.1). ROOT_URL must match so
+# The address the CLUSTER uses to reach this host (see the header). ROOT_URL must match it so
 # clone/redirect URLs are correct from inside the cluster.
-# NOTE: firewalld's `libvirt` zone (owns virbrN) blocks VM->host:3000 by default. Open it once:
+#
+# The DEFAULT IS DELIBERATELY UNSET — there is no correct default any more. The old default
+# (10.73.0.1) was the retired k3s-dev libvirt NAT gateway; leaving it in place would silently
+# publish a ROOT_URL no cluster can resolve, and Flux would report what looks like a network fault.
+# Pass it explicitly, e.g. for prod:   HOST_NAT_IP=192.168.2.132 bash bootstrap-gitea.sh
+#
+# NOTE (DEV/libvirt ONLY, historical): firewalld's `libvirt` zone (owns virbrN) blocked
+# VM->host:3000 by default and had to be opened once with
 #   sudo firewall-cmd --zone=libvirt --add-port=3000/tcp --permanent && sudo firewall-cmd --reload
-HOST_NAT_IP="${HOST_NAT_IP:-10.73.0.1}"
+# PROD needs NO firewall change: same-LAN, no libvirt zone involved (verified 2026-09-04).
+if [ -z "${HOST_NAT_IP:-}" ]; then
+  echo "ERROR: HOST_NAT_IP is not set — it is this host's address AS SEEN FROM THE CLUSTER." >&2
+  echo "       prod: HOST_NAT_IP=192.168.2.132   (same LAN as the nodes, no NAT, no firewall change)" >&2
+  echo "       There is no safe default: the old one (10.73.0.1) is the RETIRED libvirt NAT." >&2
+  exit 1
+fi
 ORG="${GITEA_ORG:-Home-Network}"
 REPO="${GITEA_REPO:-home-network-k3s-impl}"

@@ -75,8 +107,17 @@ cat <<EOF

 Gitea bootstrap source ready (${DATA_DIR}).
  UI (host):                     ${BASE}/${ORG}/${REPO}
-  Flux GitRepository (cluster):  http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git   (clusters/dev)
+  Flux GitRepository (cluster):  http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git
+                                 (patch this url into the overlay being bootstrapped:
+                                  clusters/prod today; clusters/dev has already cut over)
  Admin creds (host, 0600):      ${ADMIN_CRED_FILE}
-Prerequisite: firewalld must allow VM->host:${HTTP_PORT} —
+Reachability: prod nodes reach ${HOST_NAT_IP}:${HTTP_PORT} directly on the LAN — no NAT, no
+firewall change (verified 2026-09-04). For libvirt-based clusters ONLY, firewalld's libvirt zone
+must allow VM->host:${HTTP_PORT} —
  ansible-playbook infra/ansible/playbooks/host-prereqs.yml --ask-become-pass
+REMINDER: the push above published the CURRENT HEAD as this repo's \`main\`. Whatever cluster points
+at this source will reconcile THAT revision.
+TEMPORARY BY DESIGN: this container is outside every cluster and is a single point of failure for
+any cluster reconciling from it. Cut over to in-cluster Gitea (docs/05); for prod the cutover
+procedure is written on the GitRepository patch in clusters/prod/kustomization.yaml.
 EOF
@@ -24,9 +24,9 @@ env:
        name: gitea-bootstrap
        key: secret-key
  - name: GITEA__server__DOMAIN              # correct public clone/redirect URLs for the TLS-terminated host
-    value: gitea.home.keithjackson.dev
+    value: gitea.${BASE_DOMAIN}
  - name: GITEA__server__ROOT_URL
-    value: https://gitea.home.keithjackson.dev/
+    value: https://gitea.${BASE_DOMAIN}/
  - name: GITEA__actions__ENABLED            # Phase 6 CI/CD: declared (not implicit upstream default) so the
    value: "true"                            # in-cluster act_runner (ns gitea-actions) registers against it
  - name: GITEA__actions__DEFAULT_ACTIONS_URL  # resolve `uses: <org>/<action>@ref` from github.com (cluster
@@ -104,7 +104,7 @@ lifecycle:
          # persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker
          # file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays
          # usable). The verify-then-retry loop is the real guarantee against "exit 0 but nothing stuck".
-          GITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
+          GITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
          gitea_oidc_fail() {
            echo "[gitea-oidc] FAILED to register 'keycloak' login source: $1" >&2
            { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true
@@ -168,7 +168,7 @@ lifecycle:
          # be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,
          # the moment its OIDC guard was fixed and the block actually ran for the first time.
          exit 0
-exposure: public                # edge-forwarded git host at gitea.home.keithjackson.dev (docs/09; R-DNS/R-NET)
+exposure: public                # edge-forwarded git host at gitea.${BASE_DOMAIN} (docs/09; R-DNS/R-NET)
 egress: internet                # normal WAN egress (docs/03; R-NET-05)
 storage:
  - name: data
@@ -178,7 +178,7 @@ storage:
                                # the default /data/<name> would strand real state on the ephemeral rootfs (R-STATE-01)
 dependencies: []                # default-deny: reaches only DNS; git clients reach it north-south via the Gateway (R-NET-01/02)
 ingressRoutes:
-  - host: gitea.home.keithjackson.dev
+  - host: gitea.${BASE_DOMAIN}
    port: 3000                  # Gitea HTTP web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
 auth:
  mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC" set; R-IDP-02)
@@ -17,7 +17,7 @@ name: jackett
 kind: container
 source:
  image: lscr.io/linuxserver/jackett:latest
-exposure: internal              # LAN-only web UI at jackett.home.keithjackson.dev (docs/13 row 4; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at jackett.${BASE_DOMAIN} (docs/13 row 4; R-DNS/R-NET)
 egress: vpn                     # indexer scraping via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
 storage:
  - name: config
@@ -25,7 +25,7 @@ storage:
    class: longhorn             # indexer definitions/API keys on 3-replica Longhorn (docs/04; R-STATE-02)
 dependencies: []                # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: jackett.home.keithjackson.dev
+  - host: jackett.${BASE_DOMAIN}
    port: 9117                  # Jackett WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
 auth:
  mode: forward-auth            # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)
@@ -18,7 +18,7 @@ dependencies:
    port: 5432
    protocol: tcp
 ingressRoutes:
-  - host: keycloak.home.keithjackson.dev
+  - host: keycloak.${BASE_DOMAIN}
    port: 8080                # HTTP listener; TLS terminates at the Gateway (docs/08, docs/09)
 auth:
  mode: native                # it IS the IdP; own admin login is local + MFA (docs/15; ADR-0014). Documented exception.
@@ -50,13 +50,13 @@ env:
  #     the vhost answer for the public FQDN; OVERWRITE* make Nextcloud emit https:// self-URLs even
  #     though it speaks plain HTTP behind the proxy. ---
  - name: NEXTCLOUD_TRUSTED_DOMAINS
-    value: nextcloud.home.keithjackson.dev
+    value: nextcloud.${BASE_DOMAIN}
  - name: OVERWRITEHOST
-    value: nextcloud.home.keithjackson.dev
+    value: nextcloud.${BASE_DOMAIN}
  - name: OVERWRITEPROTOCOL
    value: https
  - name: OVERWRITECLIURL
-    value: https://nextcloud.home.keithjackson.dev
+    value: https://nextcloud.${BASE_DOMAIN}
  # --- Keycloak native OIDC SSO (auth.mode oidc; docs/15 "Native OIDC"; R-IDP-02/06). The confidential
  #     `nextcloud` realm client's secret is mirrored into this namespace's nextcloud-bootstrap Secret
  #     (oidc-client-secret; same value SOPS-held in keycloak-realm-clients). The postStart hook reads it
@@ -88,7 +88,7 @@ lifecycle:
          # provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as
          # ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.
          OCC() { su -s /bin/sh www-data -c "php /var/www/html/occ $*"; }
-          NEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
+          NEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
          nextcloud_oidc_fail() {
            echo "[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1" >&2
            { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true
@@ -228,7 +228,7 @@ dependencies:
    port: 5432                  # Renders the egress half; the DB namespace mirrors keycloak-db (no
    protocol: tcp               # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
 ingressRoutes:
-  - host: nextcloud.home.keithjackson.dev
+  - host: nextcloud.${BASE_DOMAIN}
    port: 80                    # apache HTTP; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
  mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)
@@ -14,7 +14,7 @@ storage:
    class: longhorn           # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
 dependencies: []              # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: pihole.home.keithjackson.dev
+  - host: pihole.${BASE_DOMAIN}
    port: 80                  # admin web UI, protected at the Gateway by forward-auth (docs/09)
 auth:
  mode: forward-auth          # oauth2-proxy -> Keycloak; bare UI, no native OIDC (docs/15; R-IDP-02)
@@ -72,7 +72,7 @@ storage:
                              # so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
 dependencies: []              # default-deny: reaches only DNS; clients/Seerr reach IT (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: plex.home.keithjackson.dev
+  - host: plex.${BASE_DOMAIN}
    port: 32400               # Plex web UI + API served on 32400 (wildcard TLS at the Gateway; docs/08/09)
 auth:
  mode: native                # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)
@@ -32,7 +32,7 @@ name: qbittorrent
 kind: container
 source:
  image: lscr.io/linuxserver/qbittorrent:5.1.2   # pinned stable tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at qbittorrent.home.keithjackson.dev (docs/13 row 13; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at qbittorrent.${BASE_DOMAIN} (docs/13 row 13; R-DNS/R-NET)
 egress: vpn                     # ALL public egress via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
 storage:
  - name: config
@@ -40,7 +40,7 @@ storage:
    class: longhorn             # settings/categories/session on 3-replica Longhorn; DR-critical (docs/04; R-STATE-02)
 dependencies: []                # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: qbittorrent.home.keithjackson.dev
+  - host: qbittorrent.${BASE_DOMAIN}
    port: 8080                  # qBittorrent WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
 auth:
  mode: forward-auth            # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)
@@ -9,7 +9,7 @@ name: radarr
 kind: container
 source:
  image: lscr.io/linuxserver/radarr:5.21.1   # pinned stable v5 tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at radarr.home.keithjackson.dev (docs/13 row 3; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at radarr.${BASE_DOMAIN} (docs/13 row 3; R-DNS/R-NET)
 egress: vpn                     # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
                                # kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
 storage:
@@ -29,7 +29,7 @@ dependencies:
    port: 9117
    protocol: tcp
 ingressRoutes:
-  - host: radarr.home.keithjackson.dev
+  - host: radarr.${BASE_DOMAIN}
    port: 7878                  # Radarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
 auth:
  mode: forward-auth            # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)
@@ -44,7 +44,15 @@ except ImportError:  # pragma: no cover - explicit, actionable failure

 # --- Label / naming conventions (match platform/kyverno/policies + platform/*/) --------
 LBL = "homelab.keithjackson.dev"
-ZONE = "home.keithjackson.dev"
+# The platform hostname suffix, emitted as the Flux post-build TOKEN rather than a literal.
+# Every rendered manifest lands under a Flux path whose Kustomization declares
+# postBuild.substituteFrom cluster-vars, so `${BASE_DOMAIN}` is resolved per environment at
+# reconcile time: dev renders home.keithjackson.dev, prod renders prod.home.keithjackson.dev.
+# It is a DECLARED cluster-vars knob (clusters/base/cluster-vars.yaml), so it is written BARE —
+# the escape contract below reserves `$${...}` for expansions meant for a container's shell.
+# Hardcoding the literal here is what let prod be authored to serve dev's public hostnames,
+# and prod's external-dns then deleted dev's records (infra/PROD-PROVISIONING.md §10).
+ZONE = "${BASE_DOMAIN}"
 GATEWAY_NAME = "shared"
 GATEWAY_NS = "gateway"
 KP_RELEASE = "kube-prometheus-stack"  # ServiceMonitor/PrometheusRule discovery label
@@ -57,7 +57,7 @@ dependencies:
    port: 32400
    protocol: tcp
 ingressRoutes:
-  - host: seerr.home.keithjackson.dev
+  - host: seerr.${BASE_DOMAIN}
    port: 5055                   # Overseerr web UI + API; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
  mode: native                   # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)
@@ -58,7 +58,7 @@ env:
  - name: SEMAPHORE_ADMIN_NAME
    value: Semaphore Admin
  - name: SEMAPHORE_ADMIN_EMAIL
-    value: admin@home.keithjackson.dev
+    value: admin@${BASE_DOMAIN}
  # --- Persistent access-key encryption key (see header): FIXED base64 32-byte key from SOPS so the
  #     SSH keys / credentials Semaphore stores in the DB stay decryptable across restarts. ---
  - name: SEMAPHORE_ACCESS_KEY_ENCRYPTION
@@ -110,7 +110,7 @@ dependencies:
    port: 5432                  # Renders the egress half; the DB namespace mirrors keycloak-db (no
    protocol: tcp               # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
 ingressRoutes:
-  - host: semaphore.home.keithjackson.dev
+  - host: semaphore.${BASE_DOMAIN}
    port: 3000                  # Semaphore HTTP web/API; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
  mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)
@@ -9,7 +9,7 @@ name: sonarr
 kind: container
 source:
  image: lscr.io/linuxserver/sonarr:4.0.15   # pinned stable v4 tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at sonarr.home.keithjackson.dev (docs/13 row 2; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at sonarr.${BASE_DOMAIN} (docs/13 row 2; R-DNS/R-NET)
 egress: vpn                     # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
                                # kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
 storage:
@@ -29,7 +29,7 @@ dependencies:
    port: 9117
    protocol: tcp
 ingressRoutes:
-  - host: sonarr.home.keithjackson.dev
+  - host: sonarr.${BASE_DOMAIN}
    port: 8989                  # Sonarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
 auth:
  mode: forward-auth            # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)
@@ -17,9 +17,63 @@ data:
  CLUSTER_ENV: "base"                       # dev | prod  (overlays set this)

  # --- DNS / TLS ---
-  BASE_DOMAIN: "home.keithjackson.dev"      # platform hostname suffix (same both envs; split-horizon)
-  DNS_ZONE: "keithjackson.dev"              # the actual Cloudflare ZONE (BASE_DOMAIN is a subdomain of it);
-                                            # ExternalDNS domain-filter must be the zone, not the subdomain
+  # THE platform hostname suffix. Every public hostname the cluster serves is `<svc>.${BASE_DOMAIN}`,
+  # the wildcard cert is `*.${BASE_DOMAIN}`, coredns-custom answers the `${BASE_DOMAIN}` zone with
+  # ${GATEWAY_VIP}, and every OIDC issuer/redirect URI is built from it.
+  #
+  # THIS KNOB WAS DEAD UNTIL 2026-09-04 — declared here but referenced by NO manifest, because every
+  # hostname was hardcoded. That is what made the 2026-09-04 incident possible: prod and dev were
+  # authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns
+  # (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM.
+  # See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN
+  # subdomain, so the two clusters can no longer describe the same name.
+  #
+  #   dev   -> home.keithjackson.dev            (UNCHANGED — see below)
+  #   prod  -> prod.home.keithjackson.dev
+  #
+  # DEV DELIBERATELY STAYS ON THE APEX `home.keithjackson.dev`. The target architecture puts dev on
+  # `dev.home.keithjackson.dev` with apex CNAMEs `<svc>.home` -> `<svc>.prod.home`, but moving dev
+  # relocates its Keycloak issuer, every OIDC client redirect URI, kubectl SSO, its wildcard cert and
+  # the Gitea host that IS the GitOps source — on a live cluster. Deferred to phase 2
+  # (infra/PROD-PROVISIONING.md §10). Base default is therefore the DEV value, the same convention as
+  # GATEWAY_VIP / LONGHORN_DATA_PATH: the base default is the known-good working value, dev restates
+  # it verbatim, and prod overrides it. Dev's rendered output is byte-identical to the pre-
+  # tokenization hardcoded manifests (verified 2026-09-04, all 12 substituted paths).
+  BASE_DOMAIN: "home.keithjackson.dev"
+
+  # ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind
+  # EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered,
+  # so it cannot be created, updated or deleted regardless of TXT ownership.
+  #
+  # DEV KEEPS THE ZONE APEX `keithjackson.dev`, and must: dev serves the `home.keithjackson.dev`
+  # apex itself, and the Cloudflare provider filters the ACCOUNT'S ZONES by this value — a filter
+  # that is a strict subdomain of the zone dropped the parent zone outright ("no hosted zone matching
+  # record DNS Name", DECISIONS-LOG 2026-08) on the version in use then. So dev's value is unchanged
+  # from the retired `DNS_ZONE` knob this replaces, and dev renders byte-identically.
+  #
+  # PROD NARROWS TO ITS OWN SUBTREE `prod.home.keithjackson.dev`. VERIFY ON PROD'S FIRST RECONCILE
+  # (infra/PROD-PROVISIONING.md §10): if the zone-lookup limitation above still applies to the pinned
+  # chart, prod's external-dns will select NO zone and create NO records. That failure is FAIL-CLOSED
+  # — it can never delete another environment's records — and the fallback is to set prod's value
+  # back to "keithjackson.dev", leaving EXTERNALDNS_TXT_OWNER_ID as the sole (and sufficient) guard.
+  EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev"
+
+  # ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT:
+  # neither cluster set it, so both inherited the chart default `default`, and ExternalDNS treats a
+  # record whose TXT registry entry carries ITS OWN owner id as its own to reconcile — and with
+  # `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on
+  # one zone therefore garbage-collect each other. Each environment MUST have a distinct value.
+  #
+  # DEV MUST RENDER EXACTLY `default`. Dev's LIVE public records are already registered in Cloudflare
+  # under owner `default`; changing dev's value would make dev's own external-dns stop recognising
+  # them, and it would re-create duplicates alongside the orphaned originals. Dev's overlay pins it.
+  #
+  # THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default`. A new environment that
+  # forgets this knob must not silently inherit dev's identity — that is precisely the incident. With
+  # this sentinel such a cluster owns nothing, so it can DELETE nothing; it would publish under a
+  # visibly wrong owner id (grep the TXT records) instead of destroying a live environment's DNS.
+  EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars"
+
  ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
  GATEWAY_VIP: "192.168.1.53"                # in-cluster split-horizon target (R-DNS-01/03): the L2-announced
                                            # ingress VIP of the `shared` Gateway. MUST equal the FIRST address of
@@ -148,6 +148,33 @@ patches:
      - op: replace
        path: /data/ACME_CLUSTER_ISSUER
        value: "letsencrypt-prod"
+      # --- DNS identity (R-DNS-01/02/03) -----------------------------------------------------
+      # DEV STAYS ON THE APEX. Restated explicitly even though it equals the base default, exactly
+      # like the LAN block and LONGHORN_DATA_PATH below: this is what the LIVE cluster is running.
+      # Every one of dev's public DNS records, its `*.home.keithjackson.dev` wildcard cert, its
+      # Keycloak issuer, every OIDC client redirect URI, kubectl SSO, and gitea.home.keithjackson.dev
+      # (which IS this repo's GitOps source) are all on this exact suffix RIGHT NOW. Prod moved to
+      # prod.home.keithjackson.dev instead (2026-09-04) — that alone ended the collision, so dev did
+      # not have to move. Relocating dev to dev.home.keithjackson.dev is phase 2 and is a migration,
+      # not a variable change: infra/PROD-PROVISIONING.md §10.
+      - op: replace
+        path: /data/BASE_DOMAIN
+        value: "home.keithjackson.dev"
+      # ExternalDNS scope. MUST stay the zone apex: dev serves the home.keithjackson.dev APEX itself,
+      # and the Cloudflare provider selects zones with this value (a strict-subdomain filter dropped
+      # the parent zone — DECISIONS-LOG 2026-08). Same value the retired DNS_ZONE knob held.
+      - op: replace
+        path: /data/EXTERNALDNS_DOMAIN_FILTER
+        value: "keithjackson.dev"
+      # DO NOT CHANGE THIS VALUE. Dev's LIVE public records are registered in Cloudflare's TXT
+      # registry under owner `default` (the external-dns chart default, which is what both clusters
+      # silently shared until the 2026-09-04 incident). Dev's external-dns identifies its own records
+      # BY this string: any other value and it stops recognising every record it manages, then
+      # re-creates duplicates beside the orphaned originals. Prod is `prod` — distinct, which is the
+      # whole point. The base default is a non-colliding sentinel, so this line is load-bearing.
+      - op: replace
+        path: /data/EXTERNALDNS_TXT_OWNER_ID
+        value: "default"
      # --- dev LAN addressing (192.168.1.0/24 on the Proxmox LAN) -----------------------------
      # EVERY knob in cluster-vars' "LAN addressing" block is restated here EXPLICITLY, even where it
      # equals the base default. These values are what the LIVE dev cluster is running right now, so
@@ -115,8 +115,20 @@ patches:
            name: sops-age
  # --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
  #
-  # *** BOOTSTRAP BLOCKER — this URL CANNOT work on a cold prod cluster. READ BEFORE BRING-UP. ***
+  # ############################################################################################
+  # ###                                                                                      ###
+  # ###   *** T E M P O R A R Y   B O O T S T R A P   S O U R C E  —  NOT PERMANENT ***       ###
+  # ###                                                                                      ###
+  # ###   The url below is a Gitea container running ON THE OPERATOR'S WORKSTATION, outside   ###
+  # ###   any cluster. It exists ONLY to get prod's first reconcile off the ground. It MUST   ###
+  # ###   be cut over to the in-cluster URL once Gitea runs on prod. If you are reading this  ###
+  # ###   and prod has been serving traffic for months, THE CUTOVER WAS NEVER DONE — that is  ###
+  # ###   a live single point of failure, not a settled design. See "CUTOVER" below.          ###
+  # ###                                                                                      ###
+  # ############################################################################################
  #
+  # WHY THE IN-CLUSTER URL CANNOT BE USED YET (blocker found 2026-09-01, still true; the
+  # unusable value was `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`).
  # Two independent reasons, both consequences of the split-horizon design:
  #   1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's
  #      GitOps source lives inside the very cluster slated for decommission. Gitea must move to
@@ -126,13 +138,46 @@ patches:
  #      192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve
  #      gitea.home.keithjackson.dev to itself and never reach a repository.
  #
-  # Bring-up therefore needs a TEMPORARY source (the bootstrap host-Gitea, or an off-cluster
-  # mirror), with a cutover to the in-cluster URL once Gitea actually runs on prod — exactly the
-  # sequence docs/05 describes for dev's own Phase-6 cutover. Tracked in PROD-PROVISIONING.md §7.
+  # WHAT THE TEMPORARY SOURCE IS (stood up 2026-09-04 on explicit operator direction):
+  #   bootstrap/gitea/bootstrap-gitea.sh with HOST_NAT_IP=192.168.2.132 — a podman container on the
+  #   operator's workstation, Gitea 1.22.6, listening on 192.168.2.132:3000. All three prod nodes
+  #   reach it DIRECTLY on the LAN: no libvirt NAT hop (that network is retired) and no firewall
+  #   change was needed. This is the same "local bootstrap source" pattern dev itself used before
+  #   its Phase-6 cutover (docs/05 "Source-of-truth").
+  #
+  # PLAIN HTTP, AND WHY THAT IS ACCEPTED HERE: the bootstrap Gitea has no TLS, so there is no cert
+  # for Flux to trust and no caCertSecretRef to wire. Traffic is LAN-local between prod nodes and
+  # the workstation, the repo carries NO plaintext secrets (everything sensitive is SOPS-encrypted,
+  # .sops.yaml), and the window is the bring-up only. Do NOT extend this posture past cutover.
+  #
+  # NO secretRef, DELIBERATELY: the repo is PUBLIC on the bootstrap Gitea, so anonymous clone
+  # works and Flux needs no git credentials. VERIFIED 2026-09-04 by unauthenticated request:
+  #   curl http://192.168.2.132:3000/api/v1/repos/Home-Network/home-network-k3s-impl -> "private":false
+  #   git ls-remote http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git  -> refs/heads/main
+  # If the GitRepository ever reports an auth failure that looks like a network problem, RE-CHECK
+  # THAT FIRST: a private repo here fails exactly that way.
+  #
+  # *** CUTOVER — the exit condition, mirroring dev's Phase-6 cutover (docs/05) ***
+  #   WHEN: Gitea runs on prod (reconciled in-cluster) and answers gitea.home.keithjackson.dev
+  #         through prod's own Gateway VIP with a publicly-trusted cert.
+  #   DO:   1. push/mirror this repo into the prod in-cluster Gitea;
+  #         2. change the url below to
+  #            https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
+  #            and DELETE this whole TEMPORARY block (dev's equivalent stanza, ~15 lines above in
+  #            clusters/dev/kustomization.yaml, is what "done" looks like);
+  #         3. confirm `flux get sources git -n flux-system` reconciles the new revision;
+  #         4. only THEN stop the workstation container. Keep it stopped-not-deleted as recovery,
+  #            exactly as dev retained its bootstrap Gitea.
+  #   UNTIL THEN: prod's reconciliation depends on a container on a workstation that may be
+  #   powered off. Flux keeps the last-reconciled state running, but NO new revision lands while
+  #   it is down. Tracked as an open blocker in infra/PROD-PROVISIONING.md §7.
  #
  # The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev.
  # CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH
  # CLUSTERS. There is no promotion step and no soak window — dev and prod move together.
+  # NOTE for bring-up: bootstrap-gitea.sh pushes the operator's CURRENT HEAD to the bootstrap
+  # Gitea's `main`, whatever branch that is. Before bootstrapping prod, confirm that remote `main`
+  # is the revision you intend prod to run (`git ls-remote` above, against local `main`).
  - patch: |-
      apiVersion: source.toolkit.fluxcd.io/v1
      kind: GitRepository
@@ -140,7 +185,9 @@ patches:
        name: flux-system
        namespace: flux-system
      spec:
-        url: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
+        # TEMPORARY bootstrap source — workstation container, NOT in any cluster. See the block
+        # above for the cutover to https://gitea.home.keithjackson.dev/... . Do not treat as final.
+        url: http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git
        ref:
          branch: main
  - target:
@@ -154,6 +201,63 @@ patches:
      - op: replace
        path: /data/ACME_CLUSTER_ISSUER
        value: "letsencrypt-prod"
+      # ==========================================================================================
+      # PROD DNS IDENTITY (R-DNS-01/02/03) — the fix for the 2026-09-04 incident.
+      # ==========================================================================================
+      # PROD SERVES ITS OWN SUBDOMAIN. Until 2026-09-04 every hostname was HARDCODED to
+      # home.keithjackson.dev in the base manifests, so prod was authored to serve the EXACT SAME
+      # public names as the live dev cluster out of the SAME Cloudflare zone. Two consequences, one
+      # of which already happened:
+      #   1. HAPPENED: prod's external-dns (policy: sync, owner id `default` — the same implicit
+      #      default dev had) concluded it owned dev's records and DELETED EVERY ONE of them.
+      #   2. QUEUED BEHIND IT: platform-config would have requested a `*.home.keithjackson.dev`
+      #      wildcard from letsencrypt-prod for a cert dev already holds, and both clusters would
+      #      have raced each other's `_acme-challenge` TXT records on that one zone.
+      # Moving prod to its own suffix dissolves both: the two clusters can no longer name the same
+      # record. This is what makes ACME_CLUSTER_ISSUER: letsencrypt-prod above safe here.
+      #
+      #   dev   -> home.keithjackson.dev         (UNCHANGED — dev is live; phase 2 moves it)
+      #   prod  -> prod.home.keithjackson.dev    (this knob)
+      #
+      # CONSEQUENCE — prod's hostnames are NOT the apex names. Prod's Gitea is
+      # gitea.prod.home.keithjackson.dev, its Keycloak keycloak.prod.home.keithjackson.dev, and so
+      # on; the wildcard cert becomes `*.prod.home.keithjackson.dev`. The apex CNAMEs that make
+      # `<svc>.home.keithjackson.dev` point at prod are PHASE 2, deliberately not done here — they
+      # would take traffic off the live dev cluster. infra/PROD-PROVISIONING.md §10.
+      #
+      # [parity] MUST equal k3s_base_domain in infra/ansible/inventory/group_vars/k3s_prod/vars.yml.
+      # The kube-apiserver's oidc-issuer-url is an INSTALL-TIME k3s flag, not a manifest, so it is
+      # set Ansible-side from that variable and cannot read this one. If they disagree, kubectl SSO
+      # trusts a different Keycloak than the one prod actually runs.
+      - op: replace
+        path: /data/BASE_DOMAIN
+        value: "prod.home.keithjackson.dev"
+      # SAFETY NET #2 (independent of the owner id below): narrow ExternalDNS's name scope to prod's
+      # OWN subtree, so prod cannot create, update or delete any record outside it no matter what the
+      # TXT registry says. VERIFY ON FIRST RECONCILE — the Cloudflare provider uses this value for
+      # ZONE selection too, and on the version in use in 2026-08 a strict-subdomain filter dropped
+      # the parent zone ("no hosted zone matching record DNS Name"). If prod's external-dns logs no
+      # matching zone, it will create NOTHING — fail-closed, and it can still never delete dev's
+      # records. Fallback in that case: set this to "keithjackson.dev" (dev's value) and rely on
+      # EXTERNALDNS_TXT_OWNER_ID below, which is sufficient on its own.
+      - op: replace
+        path: /data/EXTERNALDNS_DOMAIN_FILTER
+        # MUST be the Cloudflare ZONE, not prod's subdomain. FINDING F-DNS1 (DECISIONS-LOG,
+        # 2026-08-17) proved a subdomain filter DROPS THE PARENT ZONE: ExternalDNS generated the
+        # record then skipped it with "no hosted zone matching record DNS Name", so prod would
+        # publish NOTHING. Isolation from dev does NOT come from this filter — it comes from
+        # (1) prod only requesting *.prod.home.keithjackson.dev hostnames, and
+        # (2) txtOwnerId: prod, so prod's registry never claims dev's `default`-owned records.
+        # That pairing is what the 2026-09-04 incident needed and this filter cannot provide.
+        value: "keithjackson.dev"
+      # SAFETY NET #1 — THE ACTUAL ROOT CAUSE OF THE INCIDENT. Both clusters left this unset and so
+      # both inherited the chart default `default`; prod then treated dev's TXT-registry entries as
+      # its own and, under policy: sync, deleted the records behind them. This value MUST differ from
+      # dev's (`default`, pinned in clusters/dev — dev's live records are registered under it and
+      # cannot be renamed without orphaning them). `prod` is distinct, which is the entire fix.
+      - op: replace
+        path: /data/EXTERNALDNS_TXT_OWNER_ID
+        value: "prod"
      - op: replace
        path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
        value: "3"
@@ -68,7 +68,7 @@ spec:
        - --http-address=0.0.0.0:4180
        - --reverse-proxy=true
        - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
        - --upstream=http://127.0.0.1:8080
        - --email-domain=*
        - --scope=openid email profile
@@ -76,8 +76,8 @@ spec:
        - --allowed-group=platform-admins
        - --skip-provider-button=true
        - --redirect-url=https://paas-demo.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
-        - --whitelist-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
+        - --whitelist-domain=.${BASE_DOMAIN}
        - --cookie-secure=true
        - --pass-access-token=true
        - --set-xauthrequest=true
@@ -91,7 +91,7 @@ spec:
        - --http-address=0.0.0.0:4180
        - --reverse-proxy=true
        - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
        - --upstream=http://127.0.0.1:8989
        - --email-domain=*
        - --scope=openid email profile
@@ -99,8 +99,8 @@ spec:
        - --allowed-group=media-admins
        - --skip-provider-button=true
        - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
-        - --whitelist-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
+        - --whitelist-domain=.${BASE_DOMAIN}
        - --cookie-secure=true
        - --pass-access-token=true
        - --set-xauthrequest=true
@@ -91,7 +91,7 @@ spec:
        - --http-address=0.0.0.0:4180
        - --reverse-proxy=true
        - --provider=oidc
-        - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+        - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
        - --upstream=http://127.0.0.1:8989
        - --email-domain=*
        - --scope=openid email profile
@@ -99,8 +99,8 @@ spec:
        - --allowed-group=media-admins
        - --skip-provider-button=true
        - --redirect-url=https://sonarr.home.keithjackson.dev/oauth2/callback
-        - --cookie-domain=.home.keithjackson.dev
-        - --whitelist-domain=.home.keithjackson.dev
+        - --cookie-domain=.${BASE_DOMAIN}
+        - --whitelist-domain=.${BASE_DOMAIN}
        - --cookie-secure=true
        - --pass-access-token=true
        - --set-xauthrequest=true
@@ -1172,6 +1172,7 @@ no extra vars is byte-for-byte the dev behaviour it always had (verified by a be
 | `192.168.1.55` (crafty) | `platform/crafty/service-l4.yaml` | prod-LAN address |
 | `192.168.1.54` (UniFi VM) | `apps/vms/unifi/50-service-l4.yaml` | prod-LAN address |
 | `192.168.1.9:9000` (S3 backup) | `platform/keycloak/postgres-cluster.yaml`, `apps/wave-0/{nextcloud,semaphore}-db.yaml` | reachable prod backup target |
+| ~~`home.keithjackson.dev` (every hostname)~~ | ~~HTTPRoutes, Gateway, wildcard cert, coredns-custom, Keycloak realm, oauth2-proxy, Homepage, `catalog/`~~ | **CLOSED 2026-09-04 — see §10.** This one was not merely "would build wrong": it **already destroyed dev's public DNS**. `BASE_DOMAIN` was a declared-but-DEAD knob; it is now real, and prod serves `prod.home.keithjackson.dev`. |

 - [ ] **Tokenize each into a `cluster-vars` knob** with a base default, then set the value in *both*
      overlays. Adding a knob before referencing `${KNOB}` is the sanctioned mechanism (see the
@@ -1193,12 +1194,15 @@ no extra vars is byte-for-byte the dev behaviour it always had (verified by a be
            as dev**. Eight `decryption: {provider: sops, secretRef: sops-age}` patches mirror dev
            exactly. The `sops-age` Secret is still created OUT OF BAND on the prod cluster
            (`bootstrap/sops/install-age-key.sh`) — the overlay references it by name only.
-      - [~] **GitRepository source** — BRANCH settled, URL still blocked. Operator decision
-            2026-09-01: **prod tracks `main`, the same branch as dev**, because dev is temporary and
-            will be decommissioned once prod is stable. Recorded consequence: once prod is live,
-            **every merge to `main` reconciles BOTH clusters** — no promotion step, no soak window.
-            The URL is patched in but **cannot work on a cold prod cluster**, for two independent
-            reasons found 2026-09-01:
+      - [~] **GitRepository source** — **MITIGATED 2026-09-04, NOT SOLVED.** Branch settled; the
+            in-cluster URL is still unusable, so prod now points at a **temporary bootstrap source**.
+            Operator decision 2026-09-01: **prod tracks `main`, the same branch as dev**, because dev
+            is temporary and will be decommissioned once prod is stable. Recorded consequence: once
+            prod is live, **every merge to `main` reconciles BOTH clusters** — no promotion step, no
+            soak window.
+
+            **Why the in-cluster URL (`https://gitea.home.keithjackson.dev/...`) cannot work on a
+            cold prod cluster** — two independent reasons, found 2026-09-01, both still true:
            1. **Self-hosting circularity** — Gitea runs on the DEV cluster, the one slated for
               decommission. Gitea must move to prod (or off-cluster) BEFORE dev is retired, or prod
               loses its source of truth. **This is the decommission plan's first task, not its last.**
@@ -1206,9 +1210,56 @@ no extra vars is byte-for-byte the dev behaviour it always had (verified by a be
               `home.keithjackson.dev` zone with `${GATEWAY_VIP}`, which on prod is `192.168.2.20`:
               prod's OWN gateway, where no Gitea listens. Prod's Flux would resolve
               `gitea.home.keithjackson.dev` to itself and never reach a repository.
-            Bring-up needs a TEMPORARY source (bootstrap host-Gitea, or an off-cluster mirror) and a
-            cutover once Gitea runs on prod — the same sequence `docs/05` describes for dev's own
-            Phase-6 cutover.
+
+            **The mitigation now in `clusters/prod/kustomization.yaml`** (operator directed this to be
+            stood up, 2026-09-04):
+
+            | | |
+            |---|---|
+            | GitRepository url | `http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git` |
+            | Branch | `main` (unchanged — same branch as dev) |
+            | What it is | Gitea 1.22.6 in a **podman container on the operator's workstation**, started by `bootstrap/gitea/bootstrap-gitea.sh` with `HOST_NAT_IP=192.168.2.132` |
+            | Reachability | All three prod nodes reach it **directly on the LAN** — no NAT hop (the libvirt network is retired), **no firewall change needed**. Verified from all three nodes. |
+            | Auth | **None, deliberately.** The repo is PUBLIC on that Gitea, so Flux needs no `secretRef`. Verified anonymously: `curl .../api/v1/repos/Home-Network/home-network-k3s-impl` → `"private":false`, and `git ls-remote` returns `refs/heads/main`. |
+            | Transport | **Plain HTTP, no TLS.** Accepted for the bring-up window only: LAN-local, and the repo holds no plaintext secrets (all SOPS-encrypted). |
+            | Admin creds | `~/.config/home-k3s-bootstrap-gitea-admin.txt`, mode 0600, host-only, never committed. |
+
+            **THIS IS A SINGLE POINT OF FAILURE FOR PROD'S RECONCILIATION, and it is not even in a
+            cluster.** It is one container on one workstation, with no HA, no backup of its own, and
+            no restart guarantee beyond `--restart unless-stopped`. If that workstation is powered
+            off, suspended, or the podman volume `gitea-bootstrap-data` is lost:
+            - already-reconciled state **keeps running** — Flux does not tear anything down;
+            - but **no new revision can land on prod**, and every `GitRepository` reconcile fails.
+            So prod is *not* self-sufficient while this stands. Do not treat prod as production-ready
+            on this footing, and do not let it become the steady state by inattention.
+
+            **One bring-up footgun:** `bootstrap-gitea.sh` pushes the operator's **current `HEAD`** to
+            the bootstrap Gitea's `main`, whatever branch that HEAD is on. Before bootstrapping prod,
+            confirm that remote `main` is the revision prod should run
+            (`git ls-remote http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git` vs local
+            `main`). As of 2026-09-04 the remote `main` is the tip of the working branch, **not** local
+            `main`.
+
+            **EXIT CONDITION (the cutover) — mirrors dev's own Phase-6 cutover, `docs/05`.**
+            This item is not closable until all of it is done:
+            - [ ] Gitea runs **on prod**, reconciled in-cluster, answering
+                  `gitea.home.keithjackson.dev` through prod's Gateway VIP with a publicly-trusted cert.
+            - [ ] This repo is pushed/mirrored into that in-cluster Gitea.
+            - [ ] The `clusters/prod` GitRepository url is changed to
+                  `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`
+                  and the TEMPORARY comment block is **deleted** (dev's stanza is what "done" looks like).
+            - [ ] `flux get sources git -n flux-system` on prod reconciles a revision from the new URL.
+            - [ ] **Only then** stop the workstation container — kept stopped-not-deleted as recovery,
+                  exactly as dev retained its own bootstrap Gitea.
+      - [x] **`roles/flux_bootstrap` could only bootstrap dev** — FIXED 2026-09-04. It invoked
+            `bootstrap/flux/bootstrap.sh` without ever exporting `FLUX_ENV_PATH`, so the script fell
+            through to its `clusters/dev` default and §8 step 4 (`--path=clusters/prod`) was
+            unreachable without editing the role. Now parameterised on the standard group_vars seam:
+            role default `flux_bootstrap_env_path: clusters/dev` (defaults/main.yml) is exported as
+            `FLUX_ENV_PATH` by the role's task, and
+            `inventory/group_vars/k3s_prod/vars.yml` overrides it to `clusters/prod` — the same
+            mechanism as `kube_vip_address` / `k3s_data_dir` / `cluster_kubeconfig_name`. Dev has no
+            override anywhere, so dev resolves `clusters/dev` exactly as before.
 - [ ] Confirm prod's backup target: dev's S3 (`versitygw` on the Proxmox host, `192.168.1.9`) is on
      the dev side of the network. Off-cluster backup is DR-critical (R-STATE-03).

@@ -1232,3 +1283,152 @@ single-server-etcd window, and avoids a later member-add dance.
      specifically.
 - [ ] Monitoring and alerts live on prod.
 - [ ] The conformance suite passes green **against the prod cluster**, not just dev (R-TEST-03).
+      Blocked on parameterizing the L2 tests that hardcode `home.keithjackson.dev` — see §10 phase 2.
+- [ ] **Prod serves its own DNS/TLS identity** and dev's records are provably untouched (§10):
+      distinct `EXTERNALDNS_TXT_OWNER_ID`, prod wildcard cert `*.prod.home.keithjackson.dev`, and
+      prod's own Keycloak realm + per-service client secrets.
+
+## 10. DNS/TLS environment split — **the 2026-09-04 incident, and phase 1 of the fix**
+
+### What happened
+
+Prod's `external-dns` **deleted every one of dev's public DNS records.**
+
+Both clusters ran `policy: sync` against the **same Cloudflare zone**, served the **same
+hostnames**, and — because neither overlay set `txtOwnerId` — both inherited the external-dns
+chart's default owner id **`default`**. ExternalDNS decides what it may delete by reading its own
+owner id back out of the TXT registry. Prod read dev's TXT entries, saw its own owner id, found no
+local resource claiming those names, and garbage-collected the records behind them. Dev recovered
+once prod was stopped. Prod's `platform` tier and everything downstream were then **suspended**, and
+remain suspended, so it cannot recur before this change is reviewed.
+
+`policy: sync` was not the bug and has not been changed. What was missing was **identity**.
+
+### The second bug, queued directly behind it
+
+`platform-config` would have requested a `*.home.keithjackson.dev` wildcard `Certificate` from
+**letsencrypt-prod** — for a cert **dev already holds** — and both clusters would then have raced
+each other's `_acme-challenge` TXT records on that one zone. Neither would reliably solve DNS-01.
+This never fired only because the tier was suspended first.
+
+### Root cause, stated precisely
+
+Not external-dns. **Every hostname in the repo was a hardcoded literal.** `BASE_DOMAIN` existed in
+`clusters/base/cluster-vars.yaml` but was referenced by **no reconciled manifest** — the repo-derived
+status report flagged it as a dead knob. R-ENV-02 promises dev and prod differ only by `cluster-vars`
+values; for DNS and TLS that promise was false, so `clusters/prod` was *authored* to serve dev's
+public identity. Two clusters that describe the same DNS name will fight over it no matter how the
+DNS controller is configured.
+
+### The architecture decision (operator)
+
+```
+dev   ->  *.dev.home.keithjackson.dev        gitea.dev.home.keithjackson.dev
+prod  ->  *.prod.home.keithjackson.dev       gitea.prod.home.keithjackson.dev
+apex  ->  CNAME <svc>.home.keithjackson.dev  ->  <svc>.prod.home.keithjackson.dev
+```
+
+### PHASE 1 — **prod only. Dev did not move.** (this change)
+
+`BASE_DOMAIN` is now real: every hostname in every Flux-reconciled manifest derives from it.
+
+| | `BASE_DOMAIN` | `EXTERNALDNS_TXT_OWNER_ID` | `EXTERNALDNS_DOMAIN_FILTER` |
+|---|---|---|---|
+| base default | `home.keithjackson.dev` | `unset-see-cluster-vars` (non-colliding sentinel) | `keithjackson.dev` |
+| **dev** | `home.keithjackson.dev` (**unchanged**) | `default` (**must not change**) | `keithjackson.dev` |
+| **prod** | `prod.home.keithjackson.dev` | `prod` | `prod.home.keithjackson.dev` |
+
+**Dev's effective hostname is deliberately still the apex `home.keithjackson.dev`.** Moving prod
+alone ends the collision and unblocks prod, and it is the whole of the risk that had to be taken
+while the operator is away. Dev's rendered output is **byte-identical** except for one added line,
+`txtOwnerId: default`, which is the value dev's external-dns was already using implicitly — verified
+by rendering all 12 `substituteFrom` paths through `kubectl kustomize | flux envsubst --strict` for
+both overlays and diffing.
+
+**Two independent guards now, either sufficient alone:**
+
+1. **`txtOwnerId`** — per-environment identity. A cluster only ever touches records it owns. This is
+   the direct fix for the incident. Dev **must** render exactly `default`: its live records are
+   already registered under that owner, and any other value would make dev's own external-dns stop
+   recognising them and re-create duplicates beside the orphans.
+2. **`domainFilters`** — per-environment name scope. Prod cannot see a name outside its own subtree.
+   Defence in depth, added because the first guard is a single string in a single file.
+
+The base default for the owner id is a **non-colliding sentinel, deliberately not `default`**: a
+future environment that forgets the knob must not silently inherit dev's identity — that is exactly
+what happened here. With the sentinel such a cluster owns nothing, so it can delete nothing.
+
+### What could NOT be tokenized, and why
+
+- **`OIDC_ISSUER_URL` / `OIDC_CALLBACK_URL` in `platform/headlamp/headlamp-oidc-credentials.sops.yaml`**
+  — SOPS-encrypted (`encrypted_regex: ^(data|stringData)$`), so the two hostnames are ciphertext.
+  Re-encrypting needs the operator's age key, rewrites the MAC over every value in the file, and
+  **cannot be verified by the offline render harness** (`kustomize` does not decrypt), so a mistake
+  would break Headlamp SSO on the live dev cluster with no pre-flight proof. Left as-is deliberately.
+  **Prod action required:** prod's Headlamp will otherwise authenticate against **dev's** Keycloak.
+  Prod needs its own copy of this Secret with `keycloak.prod.home.keithjackson.dev` and a client
+  secret from prod's own realm — which it needs regardless, since every `*.sops.yaml` client secret
+  is per-realm.
+- **`oidc-issuer-url` in the kube-apiserver flags** (`roles/k3s_server/tasks/main.yml`) — a k3s
+  **server flag** in `/etc/rancher/k3s/config.yaml`, not a manifest, so Flux post-build substitution
+  can never reach it. Correctly tokenized on the **Ansible** side instead: new role default
+  `k3s_base_domain` (dev value) + derived `k3s_oidc_issuer_url`, overridden for prod in
+  `inventory/group_vars/k3s_prod/vars.yml` — the same inventory seam as `k3s_data_dir`/`k3s_api_host`.
+  `[parity]`: `k3s_base_domain` **must** equal the same cluster's `BASE_DOMAIN`, or the apiserver
+  validates tokens against a different Keycloak than the cluster runs and kubectl SSO fails while the
+  cert-based admin kubeconfig keeps working (so it looks like "SSO is broken", not like config skew).
+
+### A pre-existing prod problem this incidentally FIXED
+
+§4 recorded that prod nodes use **public** resolvers, so the kube-apiserver would resolve its OIDC
+issuer publicly and get **dev's** Gateway VIP (192.168.1.53) — the wrong cluster on the wrong LAN,
+making prod kubectl SSO impossible, and requiring an `/etc/hosts` pin as a workaround. With prod on
+its own subdomain the public record for `keycloak.prod.home.keithjackson.dev` is published by
+**prod's own** external-dns pointing at **prod's** Gateway VIP (192.168.2.20), so the public answer
+and the split-horizon answer agree. **No `/etc/hosts` pin is needed.** This holds only once prod's
+external-dns has actually published the record; until then use the break-glass cert kubeconfig.
+
+### Verify on prod's first reconcile
+
+- [ ] **`domainFilters` zone selection.** The Cloudflare provider uses this value for *zone* lookup
+      as well as record matching, and in 2026-08 a strict-subdomain filter dropped the parent zone
+      ("no hosted zone matching record DNS Name"). If prod's external-dns logs no matching zone it
+      will create **nothing** — fail-closed, never destructive. Fallback: set prod's
+      `EXTERNALDNS_DOMAIN_FILTER` back to `keithjackson.dev` and rely on `txtOwnerId`, which is
+      sufficient on its own.
+- [ ] **Prod's wildcard cert is `*.prod.home.keithjackson.dev`** and solves DNS-01 without touching
+      dev's `_acme-challenge` records.
+- [ ] **Dev's records are untouched** after prod's tiers are resumed. Confirm dev's TXT registry
+      entries still read owner `default` and that prod's read owner `prod`.
+- [ ] **Prod's own Keycloak realm + per-service client secrets** exist (every `*.sops.yaml` client
+      secret is realm-scoped; prod cannot reuse dev's).
+
+### PHASE 2 — dev migration + apex CNAMEs (**deferred, not done**)
+
+Deliberately not attempted here: dev is **live** (18 Longhorn volumes, working SSO) and it hosts the
+Gitea that is currently this repo's GitOps source. Moving it is a migration, not a variable change.
+Required, roughly in order:
+
+1. **`BASE_DOMAIN` → `dev.home.keithjackson.dev`** in `clusters/dev`, plus `k3s_base_domain` for the
+   dev inventory group.
+2. **A new `*.dev.home.keithjackson.dev` wildcard cert** issued and Ready *before* cutover.
+3. **Keycloak issuer change.** The realm's issuer URL is baked into every token; every OIDC client
+   (`grafana`, `gitea`, `nextcloud`, `semaphore`, `headlamp`, `crafty`, `pihole`, `jackett`,
+   `qbittorrent`, `radarr`, `sonarr`, and the `kubernetes` public client) needs its redirect URIs
+   updated in the same change, and the k3s apiserver `oidc-issuer-url` with it — which means a
+   **rolling k3s restart** on the dev servers.
+4. **kubectl SSO re-issue** for every user (issuer changed).
+5. **`gitea.dev.home.keithjackson.dev`** — this is the GitOps source URL in
+   `clusters/dev/kustomization.yaml`. Flux must be able to resolve and trust the new host *before*
+   the URL is switched, or dev stops reconciling and the change cannot be pushed to fix itself.
+   Sequence this last, and keep the bootstrap Gitea available as the recovery path (docs/05).
+6. **`EXTERNALDNS_TXT_OWNER_ID` for dev.** Once dev's names move, its existing `default`-owned
+   records are for names dev no longer serves; retire them deliberately rather than letting a
+   renamed owner orphan them.
+7. **Apex CNAMEs** `<svc>.home.keithjackson.dev` → `<svc>.prod.home.keithjackson.dev`. This is what
+   actually **moves user traffic to prod**, and it must not land before prod serves those services.
+8. **The L2 behavioural conformance tests hardcode `home.keithjackson.dev`** (`dns-exposure`,
+   `dns-internal-resolution`, `gateway-ingress-reachable`, `oidc-sso-grafana`, `sso-forward-auth`,
+   `trusted-tls`). They are test inputs, not reconciled manifests, so they were left alone in phase 1
+   — but they will fail against prod and must be parameterized before §9's "suite passes green
+   against the prod cluster" can be claimed.
@@ -289,15 +289,58 @@ node_baseline_dns_servers:
  #   LAN clients       -> PiHole. Not a cluster concern.
  #   nodes             -> THIS FILE. Public resolvers, so the zone resolves PUBLICLY.
  #
-  # THE ONE THING THAT BREAKS: the k3s API SERVER is a node process and carries
-  # `oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab`
-  # (roles/k3s_server/tasks/main.yml). It reaches the issuer through the NODE resolver, so on
-  # prod it would get the PUBLIC record instead of the prod Gateway VIP — breaking kubectl
-  # OIDC SSO. Dev avoids this because its node resolver is PiHole, which answers the zone
-  # with the dev Gateway VIP.
+  # THE k3s API SERVER is a node process and carries `oidc-issuer-url=...` (built from
+  # k3s_base_domain, below). It reaches the issuer through the NODE resolver, i.e. through
+  # these PUBLIC upstreams — not through CoreDNS's split-horizon answer.
  #
-  # NOT blocking today (prod has no cluster and no Keycloak yet). Options, in
-  # PROD-PROVISIONING.md §4: a narrow Ansible-managed /etc/hosts entry pinning
-  # keycloak.home.keithjackson.dev -> 192.168.2.20 on prod nodes; or a local resolver on the
-  # prod LAN serving the zone; or run prod on the cert kubeconfig and defer OIDC.
+  # THIS USED TO BE BROKEN, AND THE 2026-09-04 SUBDOMAIN SPLIT FIXED IT. While prod was
+  # authored to serve dev's hostnames, a prod node resolving keycloak.home.keithjackson.dev
+  # publicly got DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN, so
+  # kubectl OIDC SSO on prod could not work at all. Now prod's issuer is
+  # keycloak.prod.home.keithjackson.dev, and the public record for that name is published by
+  # PROD's own external-dns pointing at PROD's Gateway VIP (192.168.2.20). The public answer
+  # and the split-horizon answer therefore AGREE, and no /etc/hosts pin is needed.
+  #
+  # STILL TO VERIFY on prod bring-up (PROD-PROVISIONING.md §10): this holds only once prod's
+  # external-dns has actually published the record. Until then the issuer does not resolve
+  # from the nodes and kubectl SSO fails — the cert-based admin kubeconfig is the break-glass
+  # path, and it always works. Ordering, not a design gap.
 node_baseline_dns_search: []
+
+# --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
+# PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/
+# kustomization.yaml — the same one-fact-in-two-places contract as k3s_data_dir and
+# k3s_api_host. The Flux side cannot supply this: the kube-apiserver's oidc-issuer-url is a
+# k3s server FLAG written to /etc/rancher/k3s/config.yaml, not a manifest, so
+# postBuild.substituteFrom never sees it. Full rationale in
+# roles/k3s_server/defaults/main.yml, where k3s_oidc_issuer_url is derived from this.
+#
+# WHY PROD MOVED (infra/PROD-PROVISIONING.md §10): both clusters were authored to serve the
+# SAME hostnames from the SAME Cloudflare zone, and on 2026-09-04 prod's external-dns
+# deleted every one of dev's public records. Dev stays on home.keithjackson.dev (it is LIVE);
+# prod takes prod.home.keithjackson.dev. Consequence here: prod's Keycloak issuer is
+# https://keycloak.prod.home.keithjackson.dev/realms/homelab, so the `kubernetes` OIDC client
+# in prod's OWN Keycloak realm is what prod's kubectl SSO authenticates against.
+k3s_base_domain: "prod.home.keithjackson.dev"
+
+# --- Flux bootstrap: WHICH overlay this environment reconciles from (R-ENV-02) -----
+# roles/flux_bootstrap defaults flux_bootstrap_env_path to `clusters/dev` (dev is LIVE
+# and group_vars/all.yml carries no override, so dev keeps resolving exactly that).
+# Prod bootstraps its OWN overlay. Before this seam existed the role could bootstrap
+# clusters/dev ONLY — bootstrap/flux/bootstrap.sh's FLUX_ENV_PATH default was never
+# exported by the role — so `flux bootstrap ... --path=clusters/prod` (§8 step 4 of
+# infra/PROD-PROVISIONING.md) was simply not reachable without editing the role.
+#
+# Same seam as kube_vip_address / k3s_api_host / k3s_data_dir / cluster_kubeconfig_name /
+# node_baseline_backports_kernel above: set once for the [k3s_prod] umbrella group, so a
+# FOURTH prod node inherits it from group membership with no per-host or per-run flag.
+#
+# WHAT clusters/prod STILL NEEDS before this is safe to run — do NOT invoke the
+# flux_bootstrap role until these are true (PROD-PROVISIONING.md §7):
+#   - the remaining TBD-PROD-* placeholders in clusters/prod/kustomization.yaml are real
+#     (S3_BACKUP_ENDPOINT); they are INVALID on purpose so an early apply fails loudly;
+#   - the GitRepository url there is the TEMPORARY workstation Gitea
+#     (http://192.168.2.132:3000/...) and that container is actually running and reachable
+#     from the prod nodes — it is outside every cluster and can simply be powered off;
+#   - the `sops-age` Secret is installed out of band on prod (bootstrap/sops).
+flux_bootstrap_env_path: "clusters/prod"
@@ -15,9 +15,20 @@ cilium_namespace: "kube-system"
 # Cilium 1.20.1 targets Gateway API v1.6.1.
 gateway_api_version: "v1.6.1"
 gateway_api_crd_urls:
+  # ONE URL, deliberately. As of v1.6.0 the STANDARD channel bundle already contains
+  # TLSRoute (plus TCPRoute, UDPRoute, ListenerSet and BackendTLSPolicy), so the separate
+  # experimental TLSRoute manifest this list used to carry is redundant — and worse than
+  # redundant: v1.6 also ships a ValidatingAdmissionPolicy,
+  # `safe-upgrades.gateway.networking.k8s.io`, which REJECTS an experimental-channel CRD
+  # applied over a standard-channel one. Applying the old second URL therefore fails hard:
+  #   ValidatingAdmissionPolicy 'safe-upgrades.gateway.networking.k8s.io' ... denied request:
+  #   Installing experimental CRDs on top of standard channel CRDs is prohibited by default.
+  # Verified 2026-09-04 on the prod bootstrap: standard-install.yaml v1.6.1 alone produces the
+  # exact 10 gateway.networking.k8s.io CRDs the LIVE dev cluster is running, tlsroutes included
+  # and annotated channel=standard, bundle-version=v1.6.1 — so this is parity with dev, not a
+  # reduction. Dev needs no reapply; it already has them. If gateway_api_version is ever moved
+  # BACK below v1.6.0, the experimental TLSRoute URL has to come back with it.
  - "https://github.com/kubernetes-sigs/gateway-api/releases/download/{{ gateway_api_version }}/standard-install.yaml"
-  # TLSRoute (experimental) is required by Cilium's Gateway API support.
-  - "https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/{{ gateway_api_version }}/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml"

 # Local kubeconfig fetched from the primary. *.kubeconfig is gitignored. The FILENAME
 # comes from cluster_kubeconfig_name (group_vars) so a prod run cannot overwrite the dev
@@ -19,5 +19,22 @@ flux_bootstrap_repo_root: "{{ playbook_dir }}/../../.."
 flux_bootstrap_sops_cmd: "bash {{ flux_bootstrap_repo_root }}/bootstrap/sops/install-age-key.sh"

 # Assumed interface of bootstrap/flux (GitOps Engineer, T-1.4):
-#   runs `flux bootstrap ... --path=clusters/dev` against this cluster.
+#   runs `flux install` + `kubectl apply -k <env path>` against this cluster.
 flux_bootstrap_flux_cmd: "bash {{ flux_bootstrap_repo_root }}/bootstrap/flux/bootstrap.sh"
+
+# --- WHICH ENVIRONMENT OVERLAY THIS BOOTSTRAPS (R-ENV-02) -------------------------
+# The one thing that makes this role environment-agnostic. bootstrap/flux/bootstrap.sh
+# reads FLUX_ENV_PATH (default clusters/dev) and ends with `kubectl apply -k <that>`;
+# tasks/main.yml exports this value into that variable, so the role NEVER names an
+# environment and the task graph is identical for dev and prod.
+#
+# THE DEFAULT IS DEV ON PURPOSE — dev is LIVE, and group_vars/all.yml carries no
+# override, so a dev run resolves exactly the value it always had (clusters/dev) and
+# behaves byte-for-byte as before.
+#
+# PROD OVERRIDES IT in inventory/group_vars/k3s_prod/vars.yml — the same group_vars
+# seam as kube_vip_address / k3s_data_dir / cluster_kubeconfig_name /
+# node_baseline_backports_kernel. No role fork, no per-run extra-var, no second
+# playbook: group membership decides. Before this existed the role could bootstrap
+# ONLY clusters/dev, so prod bring-up was impossible without editing the role.
+flux_bootstrap_env_path: "clusters/dev"
@@ -4,11 +4,17 @@
 # — it calls the entrypoints the Security + GitOps Engineers author there.
 #
 # Order matters: install the age decryption key FIRST (so Flux can decrypt on its
-# very first reconcile), THEN run `flux bootstrap` at clusters/dev.
+# very first reconcile), THEN bootstrap Flux at the environment's overlay.
 #
 # The two commands are variables (see defaults/main.yml). If the bootstrap workers
 # expose different entrypoints, override flux_bootstrap_sops_cmd / _flux_cmd — the
 # task graph does not change.
+#
+# ENVIRONMENT-AGNOSTIC (R-ENV-02): this role names NO environment. Which overlay gets
+# bootstrapped comes from `flux_bootstrap_env_path` (defaults/main.yml, default
+# clusters/dev), exported below as FLUX_ENV_PATH — the variable bootstrap/flux/
+# bootstrap.sh already reads. Prod overrides it in group_vars/k3s_prod/vars.yml, the
+# same seam as kube_vip_address / k3s_data_dir / cluster_kubeconfig_name.

 - name: Assert the bootstrap/ subtree exists (authored by the other worker)
  ansible.builtin.stat:
@@ -29,11 +35,14 @@
  run_once: true
  changed_when: true

- name: Bootstrap Flux against clusters/dev (bootstrap/flux)
+- name: Bootstrap Flux against this environment's overlay (bootstrap/flux)
  ansible.builtin.command:
    cmd: "{{ flux_bootstrap_flux_cmd }}"
  environment:
    KUBECONFIG: "{{ flux_bootstrap_kubeconfig }}"
+    # The ONLY per-environment input. clusters/dev by default (dev unchanged);
+    # clusters/prod for [k3s_prod] hosts via group_vars.
+    FLUX_ENV_PATH: "{{ flux_bootstrap_env_path }}"
  delegate_to: localhost
  become: false
  run_once: true
@@ -66,3 +66,33 @@ k3s_tls_sans: >-
  {{ ([k3s_api_host, kube_vip_address, '127.0.0.1']
      + (groups[k3s_servers_group] | map('extract', hostvars, 'k3s_node_ip') | list))
     | unique }}
+
+# --- Keycloak OIDC issuer for kubectl SSO (R-IDP-03, R-ENV-02) -----------------------
+# THE ANSIBLE-SIDE HALF OF ${BASE_DOMAIN}, and it cannot be the Flux-side half.
+#
+# The kube-apiserver's `oidc-issuer-url` is a k3s SERVER FLAG written into
+# /etc/rancher/k3s/config.yaml by this role — it is NOT a Kubernetes manifest, so Flux never
+# sees it and `postBuild.substituteFrom: cluster-vars` cannot reach it. The platform hostname
+# suffix therefore exists as ONE FACT IN TWO PLACES, and they must agree:
+#
+#   Flux side     clusters/base/cluster-vars.yaml   BASE_DOMAIN      (per-overlay)
+#   Ansible side  THIS variable                     k3s_base_domain  (per-inventory-group)
+#
+# [parity] k3s_base_domain MUST equal the BASE_DOMAIN the SAME cluster's Flux overlay sets.
+# If they disagree, the apiserver validates ID tokens against a DIFFERENT Keycloak than the
+# one the cluster actually runs: every `kubectl` SSO login fails with an issuer mismatch,
+# while the cert-based admin kubeconfig keeps working — so it looks like "SSO is broken",
+# not like a config skew. The break-glass cert kubeconfig is why this is recoverable.
+#
+# THE DEFAULT IS THE DEV VALUE, restated explicitly rather than left implicit — the same
+# convention as k3s_data_dir above and cluster-vars' GATEWAY_VIP / LONGHORN_DATA_PATH: the
+# base default is the known-good LIVE value, so dev's rendered config.yaml is byte-for-byte
+# what it already is and the Restart handler does NOT fire on a re-run.
+#
+# PROD OVERRIDES IT in inventory/group_vars/k3s_prod/vars.yml — the same inventory seam
+# k3s_data_dir / k3s_api_host / kube_vip_address use, so a FOURTH prod server inherits it
+# from group membership rather than from anyone remembering.
+k3s_base_domain: "home.keithjackson.dev"
+
+# Derived, so the realm path is stated once. Override k3s_base_domain, not this.
+k3s_oidc_issuer_url: "https://keycloak.{{ k3s_base_domain }}/realms/homelab"
@@ -75,13 +75,19 @@
      # OIDC (Keycloak SSO for kubectl). The API server trusts ID tokens from the homelab realm;
      # the `kubernetes` PUBLIC client (PKCE) + kubelogin mint them, and RBAC binds the `groups`
      # claim (platform/keycloak/k8s-oidc-rbac.yaml: oidc:platform-admins->cluster-admin,
-      # oidc:viewers->view). Nodes resolve keycloak.home.keithjackson.dev to the Cilium gateway VIP
-      # (192.168.1.53) and trust its public LE-prod cert, so NO oidc-ca-file is needed. The `oidc:`
+      # oidc:viewers->view). Nodes resolve keycloak.<k3s_base_domain> to the Cilium gateway VIP
+      # and trust its public LE-prod cert, so NO oidc-ca-file is needed. The `oidc:`
      # prefixes namespace federated identities away from cert / service-account subjects.
      # ADDITIVE + break-glass-safe: the cert-based admin kubeconfig keeps full access with or
      # without Keycloak, so a bad realm/OIDC state can never lock you out of the cluster.
+      #
+      # NOT a Flux ${BASE_DOMAIN} token: this is a k3s SERVER FLAG, not a manifest, so
+      # postBuild.substituteFrom never touches it. k3s_oidc_issuer_url derives from
+      # k3s_base_domain (defaults/main.yml), whose [parity] contract with the cluster's
+      # BASE_DOMAIN is documented there. Dev's default renders the value it already runs, so
+      # config.yaml is unchanged and the Restart handler does not fire.
      kube-apiserver-arg:
-        - "oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab"
+        - "oidc-issuer-url={{ k3s_oidc_issuer_url }}"
        - "oidc-client-id=kubernetes"
        - "oidc-username-claim=email"
        - "oidc-username-prefix=oidc:"
@@ -98,15 +104,20 @@
    INSTALL_K3S_VERSION: "{{ k3s_version }}"
  when: k3s_server_mode == 'primary'

+# PATH FOLLOWS --data-dir. k3s writes the node-token to <data-dir>/server/token, NOT to
+# the literal /var/lib/rancher/k3s. On dev k3s_data_dir IS /var/lib/rancher/k3s (the role
+# default, restated explicitly above) so this is byte-for-byte the old path; on prod
+# --data-dir is /data/k3s (group_vars/k3s_prod) and the hardcoded path did not exist at all
+# — this task would have timed out after 180s and the two secondaries could never join.
 - name: Wait for the k3s node-token to exist (primary)
  ansible.builtin.wait_for:
-    path: /var/lib/rancher/k3s/server/token
+    path: "{{ k3s_data_dir }}/server/token"
    timeout: 180
  when: k3s_server_mode == 'primary'

 - name: Read the k3s node-token (primary)
  ansible.builtin.slurp:
-    src: /var/lib/rancher/k3s/server/token
+    src: "{{ k3s_data_dir }}/server/token"
  register: k3s_token_raw
  when: k3s_server_mode == 'primary'

@@ -126,9 +137,11 @@
 # "The primary" is resolved through k3s_primary_group (group_vars: `k3s_primary` for
 # dev, `k3s_prod_primary` for prod), never a literal group name — so a prod secondary
 # delegates to the PROD primary using this same role, unchanged (R-ENV-02).
+# k3s_data_dir is a CLUSTER-WIDE value (group_vars for prod, role default for dev), so the
+# secondary's own copy of it is also the primary's data dir — correct for the delegated read.
 - name: Fetch the k3s node-token directly from the primary (secondary)
  ansible.builtin.slurp:
-    src: /var/lib/rancher/k3s/server/token
+    src: "{{ k3s_data_dir }}/server/token"
  delegate_to: "{{ groups[k3s_primary_group][0] }}"
  register: k3s_primary_token_raw
  when:
@@ -31,4 +31,14 @@ kube_vip_retry_period: 2

 # k3s auto-deploy manifests dir: k3s applies anything here on start, so kube-vip
 # is reproducible and independent of Flux/Helm.
-kube_vip_manifest_path: "/var/lib/rancher/k3s/server/manifests/kube-vip.yaml"
+#
+# THE PATH FOLLOWS --data-dir. k3s reads auto-deploy manifests from
+# <data-dir>/server/manifests, so this must be derived from the cluster-wide
+# k3s_data_dir (roles/k3s_server turns the same variable into --data-dir), not from the
+# literal /var/lib/rancher/k3s. Dev leaves k3s_data_dir unset outside roles/k3s_server's
+# own defaults, so the fallback below reproduces dev's existing path exactly; prod sets
+# k3s_data_dir: /data/k3s in group_vars/k3s_prod, where the old hardcoded path was simply
+# not the directory k3s watches — the manifest would have been written and silently never
+# applied, and the control-plane VIP would never have come up.
+kube_vip_manifest_path: >-
+  {{ k3s_data_dir | default('/var/lib/rancher/k3s') }}/server/manifests/kube-vip.yaml
@@ -1,13 +1,22 @@
 ---
 # Deploy kube-vip (control-plane HA VIP) via the k3s auto-deploy manifests dir.
-# k3s applies /var/lib/rancher/k3s/server/manifests/*.yaml on start, so this is
-# reproducible and independent of Flux. The same manifest is written on every
+# k3s applies <data-dir>/server/manifests/*.yaml on start, so this is
+# reproducible and independent of Flux (see kube_vip_manifest_path in defaults/main.yml —
+# the dir follows k3s's --data-dir, which prod moves to /data/k3s). The same manifest is written on every
 # server so the DaemonSet survives the loss of any single node.

- name: Detect the LAN interface holding the node's 192.168.1.5x address
+# WHICH INTERFACE kube-vip ARPs THE VIP ON. Detected from this node's OWN k3s_node_ip
+# (inventory, per host) rather than a hardcoded subnet: the previous awk matched the
+# literal 192.168.1.5[0-2], i.e. DEV's node addresses, so on prod (192.168.2.10/.11/.12)
+# it returned nothing and the assert below aborted the play. k3s_node_ip is the same
+# address k3s advertises with --node-ip and is defined for dev and prod alike, so this
+# resolves eno1 on prod and the identical dev interface it always resolved before.
+# `ip addr show to <ip>/32` does the matching in iproute2 instead of a regex over dotted
+# quads (where `.` would match any character).
+- name: Detect the LAN interface holding this node's k3s_node_ip
  ansible.builtin.shell: >-
    set -o pipefail;
-    ip -o -4 addr show | awk '/192\.168\.1\.5[0-2]\//{print $2; exit}'
+    ip -o -4 addr show to {{ k3s_node_ip }}/32 | awk '{print $2; exit}'
  args:
    executable: /bin/bash
  register: kube_vip_iface_detect
@@ -25,7 +34,7 @@
      - kube_vip_vip_interface is defined
      - kube_vip_vip_interface | length > 0
    fail_msg: >-
-      Could not detect the 192.168.1.5x LAN interface for kube-vip on
+      Could not detect the interface holding {{ k3s_node_ip }} for kube-vip on
      {{ inventory_hostname }} — set kube_vip_interface explicitly.

 - name: Ensure the k3s auto-deploy manifests directory exists
@@ -183,6 +183,29 @@ node_baseline_protected_mounts:
  - /home
  - /tmp

+# --- WIRELESS (tasks/main.yml, tag `wireless`) ---------------------------------------
+# A CLUSTER NODE GETS EXACTLY ONE PATH ONTO THE NETWORK. ON by default — unlike the
+# backports kernel above, which is a prod-only deviation, this is a cluster-wide policy and
+# it is a provable NO-OP where it does not apply: the tasks act only on interfaces the node
+# reports as real 802.11 devices (a `phy80211` link in sysfs), and dev's Proxmox VMs have
+# none, so on dev nothing is written, nothing is unloaded and nothing changes. Defaulting it
+# true is what makes a FOURTH node with a wifi card inherit the policy from being a cluster
+# node, rather than from someone remembering to set a flag.
+#
+# WHY IT EXISTS: k3s-prod-3 came up on both eno1 (192.168.2.12, the inventory address) and
+# wlp1s0 (192.168.2.195, DHCP on the same /24). The wifi lease's dhcpcd run rewrote
+# /etc/resolv.conf three minutes after this role wrote it, and a second address on the LAN
+# where kube-vip floats 192.168.2.9 and Cilium L2-announces .20-.30 is a problem in its own
+# right. See the `wireless` block in tasks/main.yml for the full reasoning, including why the
+# DRIVER MODULE is the layer that gets blocked rather than the link, a service, or rfkill.
+#
+# SET IT FALSE for a host that must genuinely keep wifi (there is none today). That does NOT
+# undo an already-applied block — delete the file below on the node as well.
+node_baseline_disable_wireless: true
+# Where the persistent block lives. 99- so it is read last and overrides any earlier
+# permissive rule; the name says what it does, on a node with no repo checkout to hand.
+node_baseline_wireless_modprobe_file: /etc/modprobe.d/99-k3s-no-wireless.conf
+
 # --- DNS resilience (R-DNS-01 / R-DNS-03) -------------------------------------------
 # These are the DEV values. Prod overrides them in inventory/group_vars/k3s_prod/vars.yml
 # (the same seam kube_vip_address / k3s_api_host / cluster_kubeconfig_name use) — NOT in
@@ -1,6 +1,6 @@
 ---
-# TAGS — every task here carries one of `packages`, `kernel`, `system`, `disks` or `dns`. An
-# untagged run is unchanged (all five run, in this order). The split exists because the PACKAGE floor
+# TAGS — every task here carries one of `packages`, `kernel`, `system`, `wireless`, `disks` or
+# `dns`. An untagged run is unchanged (all six run, in this order). The split exists because the PACKAGE floor
 # and the DATA-DISK work have different readiness: packages are safe on any node from
 # first contact, while the disk tasks need block devices that a new environment may not
 # have decided yet (prod: infra/PROD-PROVISIONING.md §2, whose decision is itself blocked
@@ -16,6 +16,12 @@
 # is: it is the ONE thing that must land on a prod node BEFORE k3s is installed (§0b), and
 # it must be runnable without touching a disk decision that is still open:
 #   ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags kernel
+# `wireless` (a cluster node gets exactly ONE path onto the network) is separated for a third
+# reason: it must be runnable BEFORE `dns` and without touching packages, the kernel or a disk
+# decision, because a node with a second DHCP-managed interface reverts the `dns` write. It is
+# placed before the disk work for the same reason, and the pair is the whole remedy for the
+# k3s-prod-3 incident:
+#   ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags wireless,dns

 - name: Install baseline packages (guest agent, Longhorn/iscsi/nfs prereqs)
  ansible.builtin.apt:
@@ -314,6 +320,304 @@
    state: "{{ 'started' if node_baseline_qga_channel.stat.exists else 'stopped' }}"
  tags: [system]

+# =====================================================================================
+# WIRELESS — tag `wireless`: a cluster node gets exactly ONE path onto the network
+# =====================================================================================
+# WHAT WENT WRONG (k3s-prod-3, reported by the operator 2026-09-04). The node was up on BOTH
+# eno1 (192.168.2.12 — the inventory address) and wlp1s0 (192.168.2.195, DHCP, the SAME /24).
+# The wifi lease carried a dhcpcd run with it, and dhcpcd's `20-resolv.conf` hook REWROTE
+# /etc/resolv.conf, undoing the resolvers this role had just written: the file read
+# "# Generated by dhcpcd from wlp1s0.dhcp, wlp1s0.dhcp6, wlp1s0.ra" with nameserver
+# 192.168.2.1 plus two Google IPv6 servers, timestamped THREE MINUTES AFTER the role's own
+# write (01:32 -> 01:35). The role had reported `changed` and green. That is exactly the
+# "successful run that fixed nothing" the `contended` branch of the DNS block below exists to
+# prevent, and the detection blind spot that let it through is closed there.
+#
+# WHY THE INTERFACE IS REMOVED RATHER THAN JUST REPORTED. Closing the detection hole alone
+# would have turned a silent revert into a loud failure — an improvement, but it leaves the
+# node with a second, DHCP-managed path onto the LAN, and on a k3s server that is three
+# distinct problems beyond DNS:
+#   * a SECOND DEFAULT ROUTE, whose metric decides which source address the node uses;
+#   * a SECOND ADDRESS ON THE SAME L2 — the segment where kube-vip floats 192.168.2.9 and
+#     Cilium L2-announces 192.168.2.20-.30 (§4). One host answering ARP on one subnet from
+#     two NICs is the exact shape neither mechanism can reason about;
+#   * an UNMANAGED address: nothing in this repo knows about .195, and the DHCP server is
+#     free to hand out a different one after any lease expiry.
+# None of that is a resolver bug, and none of it is fixed by writing resolv.conf harder.
+#
+# WHY THE DRIVER LAYER — the four alternatives, and why each is weaker. What has to hold is:
+# survives a REBOOT, survives a LEASE RENEWAL, and cannot be undone by a routine `ifup`.
+# Measured on the nodes themselves — all three are HP EliteDesk 800 G5 minis, eno1 is
+# `e1000e`, the wifi NIC is an Intel AX200 on `iwlwifi`, and the OS is Debian 13 trixie with
+# ifupdown + `networking.service`, NO NetworkManager, NO systemd-networkd, NO systemd-resolved:
+#   `ip link set <wifi> down` — NOT persistent. Any `ifup`, any hotplug event, any reboot
+#       undoes it. It is what had already been done by hand on node 3, and the stale dhcpcd
+#       /etc/resolv.conf outlived it — the link was down and the wrong resolvers were still
+#       in the file, across a reboot.
+#   mask the service that manages it — THERE IS NO SUCH SERVICE HERE. dhcpcd on these nodes
+#       is `dhcpcd-base`: no dhcpcd.service exists at all (`systemctl is-enabled dhcpcd` =
+#       not-found), it is spawned per-interface by ifupdown. The only unit that can bring
+#       wireless up is `networking.service` — which ALSO owns eno1's static address, so
+#       masking it would cut the one SSH path to a machine whose operator is not on site.
+#   `rfkill` soft-block — weaker on every axis. The netdev still EXISTS, so ifupdown and
+#       dhcpcd can still be pointed at it; persistence depends on systemd-rfkill restoring
+#       saved state (the kernel's own `persistent` flag reads 0 on these devices, and the
+#       `rfkill` CLI is not even installed); and one `rfkill unblock wlan` undoes it.
+#   the BIOS `M.2 WLAN/BT` toggle — genuinely the deepest layer, and it reads `Enable` on all
+#       three nodes (hp-bioscfg survey, §1b). NOT taken here: it needs a REBOOT to take
+#       effect, and a prod reboot is a serialized, supervised operator step (§0b "SECOND
+#       FAULT" — k3s-prod-2 hung on a warm reboot and needed the power button). It also
+#       disables Bluetooth on the same M.2 card. Left as an operator recommendation in §4.
+#   THE DRIVER MODULE — chosen. `blacklist` plus `install <module> /bin/false` in
+#       /etc/modprobe.d means kmod refuses to load the wifi driver on this boot and every
+#       future one, so THE NETDEV NEVER EXISTS. No interface means no DHCP lease, no
+#       resolv.conf hook, no second route, no second address — and every re-activation path
+#       above (udev hotplug, a hand-run `ifup`, wpa_supplicant, a hand-run dhcpcd) needs a
+#       device that is not there. `blacklist` alone would still allow a dependency- or
+#       hand-triggered `modprobe`; the `install ... /bin/false` line is what makes it refuse
+#       outright. It is ONE declarative file, readable on the node, and undone by deleting
+#       it. Paired with the runtime unload below, it needs NO reboot to take effect.
+#
+# SAFE AND IDEMPOTENT ON ALL THREE PROD NODES *AND* ON THE LIVE DEV CLUSTER. Nothing here is
+# hostname- or hardware-specific: the block acts only on interfaces the node itself reports as
+# real 802.11 devices (a `phy80211` link in sysfs) and on the driver modules behind them.
+#   * dev (Proxmox VMs, virtio NICs) has no such interface, so the discovery comes back empty,
+#     every task below is skipped, and NOT EVEN THE modprobe.d FILE IS WRITTEN. The block
+#     cannot touch dev's networking, which is why it is safe to have on by default.
+#   * k3s-prod-1 (wlp1s0) and k3s-prod-2 (wlp2s0) already have the link DOWN, so the link-down
+#     task is skipped by its own `when` and only the modprobe.d file and the module unload
+#     remain — and both are no-ops on a re-run.
+#   * k3s-prod-3 (wlp1s0) is the reported node.
+# A re-run changes nothing on any of them: the file is content-identical, `ip link set down`
+# is gated on the link not already being down, and the unload is gated on the module still
+# being loaded.
+#
+# AND IT REFUSES TO CUT ITS OWN PATH IN. The assert below stops the play if the address
+# Ansible is connected to is bound to a wireless interface, or if the default route leaves
+# over one, or if the wireless driver module is also the driver of the default-route
+# interface. That is the one case where doing this would lock everybody out of a machine
+# nobody can walk up to, so it is checked per node, on every run, from that node's own facts.
+- name: Survey this node's 802.11 devices, their drivers and any live DHCP client
+  ansible.builtin.shell:
+    executable: /bin/sh
+    cmd: |
+      set -u
+      # Flat KEY|value lines so the Ansible side needs one single-group regex per fact and
+      # no parsing cleverness. A wireless interface is one with a phy80211 link in sysfs —
+      # that is the kernel's own answer to "is this 802.11", not a name pattern like wlp*.
+      wifi=''
+      for d in /sys/class/net/*; do
+        [ -e "$d/phy80211" ] || continue
+        i=$(basename "$d")
+        wifi="$wifi $i"
+        m=$(basename "$(readlink -f "$d/device/driver" 2>/dev/null || echo unknown)")
+        state=$(cat "$d/operstate" 2>/dev/null || echo unknown)
+        printf 'IFACE|%s\n' "$i"
+        printf 'DETAIL|%s driver=%s operstate=%s\n' "$i" "$m" "$state"
+        # LIVE means "not already down" — the link-down task keys off this, so a node whose
+        # radio is already down reports no change.
+        [ "$state" = down ] || printf 'IFACE_LIVE|%s\n' "$i"
+        ip -o addr show dev "$i" 2>/dev/null | awk '{print "IFACE_ADDR|" $4}'
+        [ "$m" = unknown ] || printf 'MODULE|%s\n' "$m"
+      done
+      for m in $(printf '%s\n' $wifi | while read -r i; do
+            [ -n "$i" ] && basename "$(readlink -f "/sys/class/net/$i/device/driver" 2>/dev/null || echo unknown)"
+          done | sort -u); do
+        [ "$m" = unknown ] && continue
+        [ -d "/sys/module/$m" ] || continue
+        printf 'MODULE_LOADED|%s\n' "$m"
+        # Holders must be unloaded FIRST: `modprobe -r iwlwifi` fails while iwlmvm still
+        # holds it. Read from sysfs rather than assumed, so a different wifi driver stack
+        # (ath*, rtw*, mt76*) works without editing this role.
+        for h in $(ls "/sys/module/$m/holders" 2>/dev/null); do printf 'MODULE_HOLDER|%s\n' "$h"; done
+      done
+      dr=$(ip -o route show default 2>/dev/null | awk '{for (i = 1; i < NF; i++) if ($i == "dev") print $(i + 1)}' | head -1)
+      printf 'DEFROUTE_IFACE|%s\n' "${dr:-none}"
+      if [ -n "${dr:-}" ] && [ -e "/sys/class/net/$dr" ]; then
+        printf 'DEFROUTE_MODULE|%s\n' "$(basename "$(readlink -f "/sys/class/net/$dr/device/driver" 2>/dev/null || echo unknown)")"
+      fi
+      # Any DHCP client process, with the interfaces it was started for. Used only to stop a
+      # client that holds a WIRELESS interface and nothing else.
+      for p in $(pgrep -x dhcpcd 2>/dev/null; pgrep -x dhclient 2>/dev/null); do
+        printf 'DHCPCLIENT|%s\n' "$(tr '\0' ' ' < "/proc/$p/cmdline" 2>/dev/null)"
+      done
+      exit 0
+  register: node_baseline_wifi_scan
+  changed_when: false
+  check_mode: false
+  when: node_baseline_disable_wireless | bool
+  tags: [wireless]
+
+- name: Interpret the wireless survey
+  ansible.builtin.set_fact:
+    node_baseline_wifi_ifaces: "{{ scan | regex_findall('(?m)^IFACE\\|(.+)$') }}"
+    node_baseline_wifi_live: "{{ scan | regex_findall('(?m)^IFACE_LIVE\\|(.+)$') }}"
+    node_baseline_wifi_addrs: "{{ scan | regex_findall('(?m)^IFACE_ADDR\\|(\\S+)') | map('regex_replace', '/.*$', '') | list }}"
+    node_baseline_wifi_modules: "{{ scan | regex_findall('(?m)^MODULE\\|(.+)$') | unique }}"
+    node_baseline_wifi_modules_loaded: "{{ scan | regex_findall('(?m)^MODULE_LOADED\\|(.+)$') | unique }}"
+    node_baseline_wifi_module_holders: "{{ scan | regex_findall('(?m)^MODULE_HOLDER\\|(.+)$') | unique }}"
+    node_baseline_wifi_detail: "{{ scan | regex_findall('(?m)^DETAIL\\|(.+)$') }}"
+    node_baseline_wifi_dhcp_clients: "{{ scan | regex_findall('(?m)^DHCPCLIENT\\|(.+)$') }}"
+    node_baseline_defroute_iface: "{{ (scan | regex_findall('(?m)^DEFROUTE_IFACE\\|(.+)$') | first) | default('none', true) }}"
+    node_baseline_defroute_module: "{{ (scan | regex_findall('(?m)^DEFROUTE_MODULE\\|(.+)$') | first) | default('unknown', true) }}"
+  vars:
+    scan: "{{ node_baseline_wifi_scan.stdout | default('') }}"
+  when: node_baseline_disable_wireless | bool
+  tags: [wireless]
+
+- name: Report this node's wireless hardware
+  ansible.builtin.debug:
+    msg: >-
+      wireless devices = {{ node_baseline_wifi_detail | default([]) | join('; ') | default('(none)', true) }}
+      | driver modules = {{ node_baseline_wifi_modules | default([]) | join(', ') | default('(none)', true) }}
+      (loaded: {{ node_baseline_wifi_modules_loaded | default([]) | join(', ') | default('(none)', true) }},
+      held by: {{ node_baseline_wifi_module_holders | default([]) | join(', ') | default('(none)', true) }})
+      | default route via {{ node_baseline_defroute_iface | default('?') }}
+      (driver {{ node_baseline_defroute_module | default('?') }})
+  when: node_baseline_disable_wireless | bool
+  tags: [wireless]
+
+# The one way this block could do real harm is by removing the interface it is being run
+# over. Checked from the node's own facts, per node, every run — not assumed from the fact
+# that prod "should" be wired.
+- name: Refuse to disable wireless if the path to this node runs OVER the wireless interface
+  ansible.builtin.assert:
+    that:
+      - node_baseline_defroute_iface not in node_baseline_wifi_ifaces
+      - node_baseline_defroute_module not in node_baseline_wifi_modules
+      - (ansible_host | default(inventory_hostname)) not in node_baseline_wifi_addrs
+    fail_msg: >-
+      REFUSING to disable wireless on this node: the connection to it appears to BE the
+      wireless path. ansible_host={{ ansible_host | default(inventory_hostname) }}, wireless
+      interfaces={{ node_baseline_wifi_ifaces | join(', ') }} carrying
+      {{ node_baseline_wifi_addrs | join(', ') | default('no addresses', true) }}, default route
+      via {{ node_baseline_defroute_iface }} (driver {{ node_baseline_defroute_module }}).
+      Disabling the driver here would drop this node off the network with no remote way back.
+      Give the node a wired address, point the inventory at it, and re-run.
+    quiet: true
+  when:
+    - node_baseline_disable_wireless | bool
+    - node_baseline_wifi_ifaces | length > 0
+  tags: [wireless]
+
+# THE PERSISTENT HALF. This file is the whole reboot story: kmod consults /etc/modprobe.d on
+# every load attempt, so from here on the driver cannot be loaded by udev at boot, as a
+# dependency, or by hand. Written ONLY when the node actually has a wireless driver, so a
+# node without wifi hardware (every dev VM) gets no file and no change at all.
+# `update-initramfs` is deliberately NOT run: no wifi module is loaded from the initramfs on
+# these nodes (root is a local SATA/NVMe device), the driver is autoloaded by udev after
+# switch-root, and re-generating an initramfs on a prod node is a bigger risk than the one it
+# would cover.
+- name: Block the wireless driver module(s) from ever loading again
+  ansible.builtin.copy:
+    dest: "{{ node_baseline_wireless_modprobe_file }}"
+    owner: root
+    group: root
+    mode: "0644"
+    content: |
+      # Managed by the Ansible node_baseline role (tags: wireless) — do not edit by hand.
+      #
+      # WHY THIS NODE HAS NO WIRELESS. A k3s server gets exactly ONE path onto the network.
+      # A second, DHCP-managed interface on the same LAN gives the node a second default
+      # route, a second address nothing in the cluster's config knows about, and a DHCP
+      # client whose resolv.conf hook silently overwrites the resolvers Ansible manages
+      # (that is what happened on k3s-prod-3, 2026-09-04). The link is also brought down and
+      # the module unloaded at the same time, so no reboot is needed for this to take effect.
+      #
+      # `blacklist` stops udev's alias-driven autoload; the `install ... /bin/false` line is
+      # what also refuses a dependency-triggered or hand-run `modprobe`. Both are needed.
+      #
+      # TO RE-ENABLE WIRELESS ON THIS NODE: delete this file and `modprobe` the module(s)
+      # below (or reboot). To stop Ansible putting it back, set
+      # node_baseline_disable_wireless: false for this host first — otherwise the next
+      # node-baseline run recreates it. Full rationale: roles/node_baseline/tasks/main.yml,
+      # the `wireless` block, and infra/PROD-PROVISIONING.md §4.
+      {% for module in node_baseline_wifi_modules %}
+      blacklist {{ module }}
+      install {{ module }} /bin/false
+      {% endfor %}
+  when:
+    - node_baseline_disable_wireless | bool
+    - node_baseline_wifi_modules | length > 0
+  tags: [wireless]
+
+# THE RUNTIME HALF — everything below makes the change true NOW, so the node does not have to
+# be rebooted to stop carrying a second address (this role never reboots: §0b "SECOND FAULT").
+#
+# Stop a DHCP client that is holding a wireless interface first, so it releases its lease
+# instead of being left holding a device that is about to disappear. HEAVILY GUARDED: it runs
+# only for clients whose command line names a wireless interface and names NO non-wireless
+# interface, because `dhcpcd -x` against a shared master process would also drop a wired
+# lease. On these prod nodes eno1 is a static ifupdown address with no DHCP client at all, so
+# this task has nothing to do; it exists so the ordering is correct on a node where wifi is
+# still actively leasing.
+- name: Stop the dhcpcd instance that holds a wireless interface (and nothing else)
+  ansible.builtin.command:
+    argv: [dhcpcd, --exit, "{{ item }}"]
+  loop: "{{ node_baseline_wifi_ifaces }}"
+  register: node_baseline_wifi_dhcpcd_exit
+  changed_when: rc == 0
+  failed_when: false
+  when:
+    - node_baseline_disable_wireless | bool
+    - node_baseline_wifi_dhcp_wifi_only | length > 0
+    - node_baseline_wifi_dhcp_wifi_only | select('search', 'dhcpcd') | list | length > 0
+  vars:
+    # A client is "wireless-only" when its argv mentions a wireless interface and mentions no
+    # other interface this node has. Empty interface sets are turned into a pattern that
+    # cannot match, so an odd node fails towards doing nothing rather than towards guessing.
+    node_baseline_non_wifi_ifaces: "{{ ansible_facts['interfaces'] | default([]) | difference(node_baseline_wifi_ifaces) | difference(['lo']) }}"
+    node_baseline_wifi_dhcp_wifi_only: >-
+      {{ node_baseline_wifi_dhcp_clients
+         | select('search', '\b(' ~ ((node_baseline_wifi_ifaces | join('|')) | default('__no_such_interface__', true)) ~ ')\b')
+         | reject('search', '\b(' ~ ((node_baseline_non_wifi_ifaces | join('|')) | default('__no_such_interface__', true)) ~ ')\b')
+         | list }}
+  tags: [wireless]
+
+- name: Take each wireless link down now
+  ansible.builtin.command:
+    argv: [ip, link, set, dev, "{{ item }}", down]
+  loop: "{{ node_baseline_wifi_live }}"
+  changed_when: true
+  when: node_baseline_disable_wireless | bool
+  tags: [wireless]
+
+# Unload the driver so the netdev is gone immediately — holders (iwlmvm on these nodes) first,
+# then the driver itself, because `modprobe -r` refuses a module that is still held.
+# NOT fatal on failure: the persistent block above is already in place, so a module that
+# cannot be unloaded right now is gone at the next boot regardless. The report below says so
+# out loud rather than letting a green play imply the radio is already off.
+- name: Unload the wireless driver module(s) now (no reboot needed)
+  community.general.modprobe:
+    name: "{{ item }}"
+    state: absent
+  loop: "{{ node_baseline_wifi_module_holders + node_baseline_wifi_modules_loaded }}"
+  register: node_baseline_wifi_unload
+  failed_when: false
+  when: node_baseline_disable_wireless | bool
+  tags: [wireless]
+
+- name: Report the wireless state this node is left in
+  ansible.builtin.debug:
+    msg: >-
+      {{ 'No 802.11 device on this node — nothing to disable (no modprobe.d file written).'
+         if (node_baseline_wifi_ifaces | length) == 0
+         else ((('WIRELESS DISABLED: ' ~ (node_baseline_wifi_modules | join(', ')) ~ ' blocked in '
+                 ~ node_baseline_wireless_modprobe_file ~ ' (survives reboot) and unloaded now; '
+                 ~ (node_baseline_wifi_ifaces | join(', ')) ~ ' is down and the netdev is gone.')
+                if (failed_unloads | length) == 0
+                else ('WIRELESS BLOCKED FOR THE NEXT BOOT BUT STILL LOADED NOW: could not unload '
+                      ~ (failed_unloads | join(', ')) ~ '. ' ~ node_baseline_wireless_modprobe_file
+                      ~ ' is in place, so the driver will not load again after a reboot, but the '
+                      ~ 'radio is live until then — check `ip link` and whether something still '
+                      ~ 'holds the device.'))) }}
+  vars:
+    failed_unloads: >-
+      {{ (node_baseline_wifi_unload.results | default([]))
+         | selectattr('failed', 'defined') | selectattr('failed') | map(attribute='item') | list }}
+  when: node_baseline_disable_wireless | bool
+  tags: [wireless]
+
 # =====================================================================================
 # DATA DISKS — tag `disks`, in tasks/disks.yml
 # =====================================================================================
@@ -377,6 +681,11 @@
 # node and the matching seam is written. Adding a third mechanism means adding a branch
 # here, not rewriting the role.
 #
+# AND DETECTION IS ONLY AS GOOD AS THE SET OF OWNERS IT KNOWS ABOUT. It originally looked for
+# resolvconf/openresolv and NetworkManager, missed dhcpcd, and therefore reported a green,
+# `changed` run on k3s-prod-3 whose resolvers a wifi lease reverted three minutes later. dhcpcd
+# is now part of that set — see the detection task below and the `contended` failure.
+#
 # All of it is `dns`-tagged, so `--tags packages` (the prod package-floor escape hatch
 # documented at the top of this file) still installs nothing but packages.

@@ -396,6 +705,55 @@
  check_mode: false
  tags: [dns]

+# THE dhcpcd BLIND SPOT, CLOSED (k3s-prod-3, 2026-09-04). dhcpcd's `20-resolv.conf` hook
+# rewrites /etc/resolv.conf on every lease, renewal and carrier change, and it is a THIRD
+# way to own that file — neither resolvconf nor NetworkManager. On node 3 it silently reverted
+# this role's write three minutes later, and because nothing below looked for it the role had
+# already reported `changed` and green: the precise failure the `contended` branch exists to
+# make impossible.
+#
+# WHY THIS LOOKS FOR A LIVE CLIENT AND NOT FOR AN INSTALLED PACKAGE. `dhcpcd-base` is
+# installed on all three prod nodes and always will be — it is the DHCP client ifupdown
+# reaches for — and there is NO dhcpcd.service on them at all (`systemctl is-active dhcpcd` =
+# not-found; it is spawned per-interface by ifupdown). Keying off the package, or off a unit
+# that does not exist, would put every prod node permanently in `contended` and leave the DNS
+# block unable to configure a node it is perfectly safe to configure. What actually matters is
+# whether a DHCP client CAN still write the file, so all three real signals are checked:
+#   CLIENT| a dhcpcd/dhclient process running right now (with the argv that started it);
+#   UNIT|   a dhcpcd/dhclient unit that IS active or IS enabled, on a distro that has one;
+#   STANZA| an UNCOMMENTED `iface <x> inet[6] dhcp` in /etc/network/interfaces{,.d} — the
+#           configuration that makes ifupdown spawn a client at the next boot or `ifup`, even
+#           when none is running at this instant.
+# Any one of them means "another manager is in play" and the play stops below. None of them
+# means no client can run, and the file is genuinely ours to own.
+- name: Determine whether a DHCP client (dhcpcd/dhclient) can rewrite /etc/resolv.conf
+  ansible.builtin.shell:
+    executable: /bin/sh
+    cmd: |
+      set -u
+      for p in $(pgrep -x dhcpcd 2>/dev/null; pgrep -x dhclient 2>/dev/null); do
+        printf 'CLIENT|%s\n' "$(tr '\0' ' ' < "/proc/$p/cmdline" 2>/dev/null)"
+      done
+      for u in dhcpcd dhcpcd5 dhclient isc-dhcp-client; do
+        a=$(systemctl is-active "$u" 2>/dev/null || true)
+        e=$(systemctl is-enabled "$u" 2>/dev/null || true)
+        # Compared exactly, never with a substring match: `is-active` answers "inactive",
+        # which contains the word "active".
+        if [ "$a" = active ] || [ "$e" = enabled ] || [ "$e" = enabled-runtime ]; then
+          printf 'UNIT|%s active=%s enabled=%s\n' "$u" "$a" "$e"
+        fi
+      done
+      # `^[[:space:]]*iface` cannot match a commented-out stanza, which is how the wireless
+      # stanza is left on these nodes.
+      grep -rhE '^[[:space:]]*iface[[:space:]]+[^[:space:]]+[[:space:]]+inet6?[[:space:]]+dhcp' \
+        /etc/network/interfaces /etc/network/interfaces.d/ 2>/dev/null \
+        | sed -e 's/^[[:space:]]*/STANZA|/'
+      exit 0
+  register: node_baseline_dhcp_client_scan
+  changed_when: false
+  check_mode: false
+  tags: [dns]
+
 # follow: false — we need to know whether /etc/resolv.conf IS a symlink and where it
 # points, which is the strongest single signal of who owns the file.
 - name: Inspect /etc/resolv.conf itself (symlink or real file, and its target)
@@ -426,9 +784,10 @@
 #       resolved's loopback stub (127.0.0.53 / 127.0.0.54). Writing a resolved drop-in is
 #       then the ONLY thing that changes real behaviour; writing resolv.conf would be
 #       overwritten by resolved (or, worse, land in /run and vanish at reboot).
-#   contended — nobody's stub, but resolvconf or NetworkManager is present/running and
-#       would regenerate the file underneath us. We do NOT write. The role fails loudly
-#       below rather than leaving an operator believing a change stuck when it did not.
+#   contended — nobody's stub, but resolvconf, NetworkManager or a DHCP client (dhcpcd /
+#       dhclient) is present/running/configured and would regenerate the file underneath us.
+#       We do NOT write. The role fails loudly below rather than leaving an operator
+#       believing a change stuck when it did not.
 #   resolv-conf-file — a plain, directly-managed /etc/resolv.conf (prod's shape today).
 #       Safe to own outright.
 # resolved running but NOT owning resolv.conf correctly falls through to the file branch:
@@ -438,13 +797,21 @@
    node_baseline_dns_mechanism: >-
      {{
        'systemd-resolved' if (node_baseline_resolved_running and node_baseline_resolved_owns_resolv)
-        else ('contended' if (node_baseline_nm_running or node_baseline_resolvconf_present)
+        else ('contended' if (node_baseline_nm_running or node_baseline_resolvconf_present
+                              or node_baseline_dhcp_client_live)
              else 'resolv-conf-file')
      }}
+    # Kept as facts (not just task vars) so the failure message, the report and the
+    # reclaim note below all quote the SAME evidence this decision was made on.
+    node_baseline_dhcp_client_evidence: "{{ (node_baseline_dhcp_client_scan.stdout | default('')) | regex_findall('(?m)^(?:CLIENT|UNIT|STANZA)\\|(.+)$') }}"
+    # A file dhcpcd WROTE, versus a dhcpcd that can still write it — two different facts, and
+    # only the second one is contention. See the reclaim note below.
+    node_baseline_resolv_dhcpcd_written: "{{ 'Generated by dhcpcd' in (node_baseline_resolv_slurp.content | default('') | b64decode) }}"
  vars:
    node_baseline_resolved_running: "{{ (node_baseline_resolved_active.stdout | default('') | trim) == 'active' }}"
    node_baseline_nm_running: "{{ (node_baseline_nm_active.stdout | default('') | trim) == 'active' }}"
    node_baseline_resolvconf_present: "{{ node_baseline_resolvconf_dir.stat.exists | default(false) }}"
+    node_baseline_dhcp_client_live: "{{ (node_baseline_dhcp_client_scan.stdout | default('')) is search('(?m)^(?:CLIENT|UNIT|STANZA)\\|') }}"
    node_baseline_resolv_text: "{{ (node_baseline_resolv_slurp.content | default('') | b64decode) }}"
    node_baseline_resolv_ns: "{{ node_baseline_resolv_text | regex_findall('(?m)^\\s*nameserver\\s+(\\S+)') }}"
    node_baseline_resolved_owns_resolv: >-
@@ -461,7 +828,9 @@
      NetworkManager is-active={{ node_baseline_nm_active.stdout | default('') | trim }},
      /etc/resolv.conf islnk={{ node_baseline_resolv_stat.stat.islnk | default(false) }}
      target={{ node_baseline_resolv_stat.stat.lnk_source | default('(not a symlink)') }},
-      resolvconf dir={{ node_baseline_resolvconf_dir.stat.exists | default(false) }})
+      resolvconf dir={{ node_baseline_resolvconf_dir.stat.exists | default(false) }},
+      DHCP client evidence={{ node_baseline_dhcp_client_evidence | default([]) | join(' ; ') | default('(none)', true) }},
+      resolv.conf written by dhcpcd={{ node_baseline_resolv_dhcpcd_written | default(false) }})
  tags: [dns]

 # Same shape, and the same reason, as the data-disk assert above: this environment's
@@ -494,16 +863,45 @@
 - name: Refuse to manage /etc/resolv.conf while another resolver manager owns it
  ansible.builtin.fail:
    msg: >-
-      /etc/resolv.conf on this node is not the systemd-resolved stub, but
-      resolvconf/openresolv (/etc/resolvconf/resolv.conf.d present:
-      {{ node_baseline_resolvconf_dir.stat.exists | default(false) }}) or NetworkManager
-      (is-active: {{ node_baseline_nm_active.stdout | default('') | trim }}) is in play and
-      would regenerate the file underneath this role. Writing it here would look like it
-      worked and silently revert. Add the matching branch to roles/node_baseline (the
-      `dns`-tagged block) for whichever manager this node really uses — do not force it.
+      /etc/resolv.conf on this node is not the systemd-resolved stub, but something else would
+      regenerate the file underneath this role: resolvconf/openresolv
+      (/etc/resolvconf/resolv.conf.d present:
+      {{ node_baseline_resolvconf_dir.stat.exists | default(false) }}), NetworkManager
+      (is-active: {{ node_baseline_nm_active.stdout | default('') | trim }}), or a DHCP client
+      that can still write it — dhcpcd/dhclient evidence:
+      {{ node_baseline_dhcp_client_evidence | default([]) | join(' ; ') | default('(none)', true) }}.
+      Writing it here would look like it worked and silently revert — which is exactly what
+      happened on k3s-prod-3 on 2026-09-04, where a wifi lease's dhcpcd hook reverted this
+      role's resolvers three minutes after a green run. If the evidence above is a DHCP client
+      on a WIRELESS interface, the fix is the `wireless` block earlier in this role (a cluster
+      node gets one network path) — run it, then re-run this. If it is a DHCP client on the
+      node's real interface, or another manager entirely, add the matching branch to
+      roles/node_baseline for the manager this node really uses — do not force it.
  when: node_baseline_dns_mechanism == 'contended'
  tags: [dns]

+# THE CORPSE CASE, and why it is NOT a hole in the fail-closed rule above. After the
+# `wireless` block disables the radio, k3s-prod-3 is left with a /etc/resolv.conf that dhcpcd
+# WROTE (its "# Generated by dhcpcd from wlp1s0.dhcp..." header, the gateway as sole IPv4
+# resolver, two Google IPv6 servers) and NO dhcpcd that can ever run again. `contended` means
+# "another manager is live and will undo this write"; a file left behind by a client that no
+# longer exists is not a manager, it is the damage — and refusing to write here would leave the
+# node permanently mis-resolved, which is the very outcome the operator reported. So the file
+# branch below reclaims it, and says so.
+- name: Note that a dhcpcd-written /etc/resolv.conf is being reclaimed
+  ansible.builtin.debug:
+    msg: >-
+      /etc/resolv.conf on this node was written by dhcpcd, but no DHCP client can write it any
+      more (no process, no active/enabled unit, no `inet dhcp` stanza). Taking ownership and
+      replacing it with the declared upstreams
+      ({{ node_baseline_dns_servers | join(', ') }}). If a DHCP client ever comes back on this
+      node, the `contended` check above will stop the play instead of letting it revert
+      silently.
+  when:
+    - node_baseline_dns_mechanism == 'resolv-conf-file'
+    - node_baseline_resolv_dhcpcd_written | default(false)
+  tags: [dns]
+
 # --- Mechanism A: systemd-resolved (dev's shape today) -------------------------------
 # A drop-in, never an edit of /etc/systemd/resolved.conf, so a package upgrade of that file
 # never conflicts. `DNS=` is one ORDERED list inside a single resolved scope: resolved uses
@@ -29,9 +29,9 @@ metadata:
 data:
  # Filename `home.server` -> imported as a standalone server block for the whole zone.
  home.server: |
-    home.keithjackson.dev:53 {
+    ${BASE_DOMAIN}:53 {
        errors
-        # Answer EVERY name in the zone (apex + all `<svc>.home.keithjackson.dev`) with the Gateway
+        # Answer EVERY name in the zone (apex + all `<svc>.${BASE_DOMAIN}`) with the Gateway
        # VIP. The template plugin's default match is `.*`, so this covers current and FUTURE app
        # hostnames without editing this file per service. `{{ .Name }}` echoes the queried FQDN.
        template IN A {
@@ -125,7 +125,7 @@ spec:
            - --http-address=0.0.0.0:4180
            - --reverse-proxy=true
            - --provider=oidc
-            - --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
+            - --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
            # Gate the co-located Crafty UI. Crafty serves HTTPS with a SELF-SIGNED cert, so skip verify
            # on this in-pod (127.0.0.1) hop; the browser-facing TLS is the trusted wildcard at the Gateway.
            - --upstream=https://127.0.0.1:8443
@@ -138,9 +138,9 @@ spec:
            - --allowed-group=platform-admins
            # Single provider => 302 straight to Keycloak, no oauth2-proxy interstitial button.
            - --skip-provider-button=true
-            - --redirect-url=https://crafty.home.keithjackson.dev/oauth2/callback
-            - --cookie-domain=.home.keithjackson.dev
-            - --whitelist-domain=.home.keithjackson.dev
+            - --redirect-url=https://crafty.${BASE_DOMAIN}/oauth2/callback
+            - --cookie-domain=.${BASE_DOMAIN}
+            - --whitelist-domain=.${BASE_DOMAIN}
            - --cookie-secure=true
            - --pass-access-token=true
            - --set-xauthrequest=true
@@ -25,7 +25,7 @@ spec:
      namespace: gateway
      sectionName: https
  hostnames:
-    - crafty.home.keithjackson.dev
+    - crafty.${BASE_DOMAIN}
  rules:
    - matches:
        - path:
@@ -42,7 +42,7 @@ metadata:
    lbipam.cilium.io/sharing-key: "crafty"
    lbipam.cilium.io/ips: "${CRAFTY_L4_VIP}"
    # external-dns publishes A crafty-mc.home.keithjackson.dev -> the pinned VIP (Java server target).
-    external-dns.alpha.kubernetes.io/hostname: "crafty-mc.home.keithjackson.dev"
+    external-dns.alpha.kubernetes.io/hostname: "crafty-mc.${BASE_DOMAIN}"
 spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
@@ -29,15 +29,37 @@ spec:
    sources:
      - gateway-httproute
      - service
-    # domain-filter MUST be the actual Cloudflare ZONE (keithjackson.dev), not the platform subdomain
-    # (home.keithjackson.dev) — ExternalDNS filters the account's zones by this value, and a subdomain
-    # filter drops the parent zone → "no hosted zone matching record DNS Name" and nothing is created.
-    # Scope stays effectively home.keithjackson.dev because that's the only suffix services request;
-    # policy=sync + registry=txt only ever touch ExternalDNS-OWNED records, never the pre-existing
-    # wildcard or other keithjackson.dev records. (docs/09; resolves the WS-NET TODO below.)
+    # ---------------------------------------------------------------------------------------------
+    # CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below.
+    #
+    # On 2026-09-04 prod's external-dns DELETED EVERY ONE of dev's public records. Both clusters ran
+    # `policy: sync` against the SAME Cloudflare zone, served the SAME hostnames, and — because
+    # NEITHER set txtOwnerId — both inherited the chart default owner id `default`. Prod therefore
+    # read dev's TXT registry entries as its own, found no local resource claiming those names, and
+    # garbage-collected them. `policy: sync` is correct and stays; what was missing was identity.
+    # Full write-up: infra/PROD-PROVISIONING.md §10.
+    #
+    # TWO INDEPENDENT GUARDS NOW, either of which alone prevents a recurrence:
+    #   1. txtOwnerId       — per-environment identity; a cluster only ever touches records it owns.
+    #   2. domainFilters    — per-environment name scope; prod cannot even see names outside its
+    #                         own subtree. Defence in depth, added after the incident.
+    # ---------------------------------------------------------------------------------------------
+    # Which names this instance may manage at all. Cloudflare-provider caveat, empirically hit
+    # 2026-08: this value is used for ZONE selection as well as record matching, and a filter that is
+    # a strict subdomain of the zone dropped the parent zone entirely ("no hosted zone matching record
+    # DNS Name") — so DEV keeps the zone apex `keithjackson.dev`, which is also what dev serves the
+    # apex of. PROD narrows to `prod.home.keithjackson.dev`. If prod's first reconcile logs no
+    # matching zone, that caveat still applies: prod creates NOTHING (fail-closed, never destructive)
+    # and the fallback is to widen prod's EXTERNALDNS_DOMAIN_FILTER back to the zone apex, leaving
+    # txtOwnerId as the sole guard. See clusters/base/cluster-vars.yaml.
    domainFilters:
-      - "${DNS_ZONE}"
-    policy: sync            # create + update + delete records as services come and go
+      - "${EXTERNALDNS_DOMAIN_FILTER}"
+    # The TXT-registry owner id. MUST be distinct per environment — this is the knob whose absence
+    # caused the incident. Dev renders `default` (its live records are already registered under that
+    # owner; changing it would orphan them and duplicate every record). Prod renders `prod`.
+    txtOwnerId: "${EXTERNALDNS_TXT_OWNER_ID}"
+    policy: sync            # create + update + delete records as services come and go — but ONLY
+                            # records this owner id owns AND this domainFilter admits (see above).
    # Cloudflare API token: SOPS-encrypted secret, referenced BY NAME only (OQ-4, ADR-0007).
    env:
      - name: CF_API_TOKEN
@@ -13,7 +13,7 @@ spec:
    - name: https
      protocol: HTTPS
      port: 443
-      hostname: "*.home.keithjackson.dev"
+      hostname: "*.${BASE_DOMAIN}"
      tls:
        mode: Terminate
        certificateRefs:
@@ -30,7 +30,7 @@ spec:
    - name: http-redirect
      protocol: HTTP
      port: 80
-      hostname: "*.home.keithjackson.dev"
+      hostname: "*.${BASE_DOMAIN}"
      allowedRoutes:
        namespaces:
          from: Selector
@@ -46,7 +46,7 @@ spec:
    - name: https-apex
      protocol: HTTPS
      port: 443
-      hostname: "home.keithjackson.dev"
+      hostname: "${BASE_DOMAIN}"
      tls:
        mode: Terminate
        certificateRefs:
@@ -61,7 +61,7 @@ spec:
    - name: http-apex-redirect
      protocol: HTTP
      port: 80
-      hostname: "home.keithjackson.dev"
+      hostname: "${BASE_DOMAIN}"
      allowedRoutes:
        namespaces:
          from: Selector
@@ -10,8 +10,8 @@ metadata:
 spec:
  secretName: wildcard-home-tls        # consumed by the Gateway HTTPS listener below
  dnsNames:
-    - "*.home.keithjackson.dev"
-    - "home.keithjackson.dev"          # apex for the C2 UI (docs/09)
+    - "*.${BASE_DOMAIN}"
+    - "${BASE_DOMAIN}"          # apex for the C2 UI (docs/09)
  issuerRef:
    kind: ClusterIssuer
    # ${ACME_CLUSTER_ISSUER}: dev -> letsencrypt-staging, prod -> letsencrypt-prod (cluster-vars).
@@ -24,7 +24,7 @@ spec:
      namespace: gateway
      sectionName: https
  hostnames:
-    - headlamp.home.keithjackson.dev
+    - headlamp.${BASE_DOMAIN}
  rules:
    - matches:
        - path:
@@ -36,63 +36,63 @@ data:
    ---
    - Platform:
        - Headlamp:
-            href: https://headlamp.home.keithjackson.dev
+            href: https://headlamp.${BASE_DOMAIN}
            description: Kubernetes dashboard (per-user OIDC RBAC)
            icon: mdi-kubernetes
        - Grafana:
-            href: https://grafana.home.keithjackson.dev
+            href: https://grafana.${BASE_DOMAIN}
            description: Metrics, logs & dashboards (fronts Prometheus/Loki/Alertmanager)
            icon: mdi-chart-areaspline
        - Semaphore:
-            href: https://semaphore.home.keithjackson.dev
+            href: https://semaphore.${BASE_DOMAIN}
            description: Ansible automation & CI
            icon: mdi-cog-play
        - Gitea:
-            href: https://gitea.home.keithjackson.dev
+            href: https://gitea.${BASE_DOMAIN}
            description: Git hosting & Actions (GitOps source of truth)
            icon: mdi-git
        - Keycloak:
-            href: https://keycloak.home.keithjackson.dev
+            href: https://keycloak.${BASE_DOMAIN}
            description: Identity provider (SSO / OIDC)
            icon: mdi-account-key

    - Media:
        - Plex:
-            href: https://plex.home.keithjackson.dev
+            href: https://plex.${BASE_DOMAIN}
            description: Media streaming
            icon: mdi-plex
        - Seerr:
-            href: https://seerr.home.keithjackson.dev
+            href: https://seerr.${BASE_DOMAIN}
            description: Media requests
            icon: mdi-play-box-multiple
        - Sonarr:
-            href: https://sonarr.home.keithjackson.dev
+            href: https://sonarr.${BASE_DOMAIN}
            description: TV library manager
            icon: mdi-television-classic
        - Radarr:
-            href: https://radarr.home.keithjackson.dev
+            href: https://radarr.${BASE_DOMAIN}
            description: Movie library manager
            icon: mdi-movie-open
        - Jackett:
-            href: https://jackett.home.keithjackson.dev
+            href: https://jackett.${BASE_DOMAIN}
            description: Indexer proxy
            icon: mdi-magnify
        - qBittorrent:
-            href: https://qbittorrent.home.keithjackson.dev
+            href: https://qbittorrent.${BASE_DOMAIN}
            description: Download client (VPN egress)
            icon: mdi-download-network

    - Network & Infra:
        - Pi-hole:
-            href: https://pihole.home.keithjackson.dev
+            href: https://pihole.${BASE_DOMAIN}
            description: Network-wide DNS & ad-blocking
            icon: mdi-shield-check
        - UniFi:
-            href: https://unifi.home.keithjackson.dev:11443
+            href: https://unifi.${BASE_DOMAIN}:11443
            description: Network controller (UOS)
            icon: mdi-router-network
        - Nextcloud:
-            href: https://nextcloud.home.keithjackson.dev
+            href: https://nextcloud.${BASE_DOMAIN}
            description: Files, calendar & contacts
            icon: mdi-cloud

@@ -104,7 +104,7 @@ data:
              href: https://gitea.keithjackson.dev
        - Auth portal:
            - abbr: SSO
-              href: https://auth.home.keithjackson.dev
+              href: https://auth.${BASE_DOMAIN}
    - Reference:
        - Homepage docs:
            - abbr: HP
@@ -54,7 +54,7 @@ spec:
            # REQUIRED by Homepage >= 1.0 — requests whose Host header is not in this list get 403.
            # The public apex host (served via the Gateway) + the pod IP (for kubelet probes).
            - name: HOMEPAGE_ALLOWED_HOSTS
-              value: "home.keithjackson.dev,$(MY_POD_IP):3000"
+              value: "${BASE_DOMAIN},$(MY_POD_IP):3000"
          resources:
            requests:
              cpu: 25m
@@ -28,7 +28,7 @@ spec:
      namespace: gateway
      sectionName: https-apex     # the apex listener (wildcard `https` won't match the bare apex)
  hostnames:
-    - home.keithjackson.dev
+    - ${BASE_DOMAIN}
  rules:
    - matches:
        - path:
@@ -16,7 +16,7 @@ spec:
      namespace: gateway
      sectionName: https
  hostnames:
-    - keycloak.home.keithjackson.dev
+    - keycloak.${BASE_DOMAIN}
  rules:
    - matches:
        - path:
@@ -30,7 +30,7 @@ spec:
    poolMaxSize: 20
  # --- Exposure: TLS at the Gateway; Keycloak serves HTTP and trusts the proxy headers. ---
  hostname:
-    hostname: https://keycloak.home.keithjackson.dev
+    hostname: https://keycloak.${BASE_DOMAIN}
  http:
    httpEnabled: true                      # TLS is terminated at the Gateway (edge), not in Keycloak
  proxy:
@@ -98,9 +98,9 @@ spec:
        directAccessGrantsEnabled: false
        secret: "$(env:GRAFANA_CLIENT_SECRET)"
        redirectUris:
-          - https://grafana.home.keithjackson.dev/login/generic_oauth
+          - https://grafana.${BASE_DOMAIN}/login/generic_oauth
        webOrigins:
-          - https://grafana.home.keithjackson.dev
+          - https://grafana.${BASE_DOMAIN}
        defaultClientScopes:
          - email
          - profile
@@ -125,10 +125,10 @@ spec:
        directAccessGrantsEnabled: false
        secret: "$(env:OAUTH2_PROXY_CLIENT_SECRET)"
        redirectUris:
-          - https://*.home.keithjackson.dev/oauth2/callback
-          - https://auth.home.keithjackson.dev/oauth2/callback
+          - https://*.${BASE_DOMAIN}/oauth2/callback
+          - https://auth.${BASE_DOMAIN}/oauth2/callback
        webOrigins:
-          - https://*.home.keithjackson.dev
+          - https://*.${BASE_DOMAIN}
        defaultClientScopes:
          - email
          - profile
@@ -209,9 +209,9 @@ spec:
        directAccessGrantsEnabled: false
        secret: "$(env:GITEA_CLIENT_SECRET)"
        redirectUris:
-          - https://gitea.home.keithjackson.dev/user/oauth2/keycloak/callback
+          - https://gitea.${BASE_DOMAIN}/user/oauth2/keycloak/callback
        webOrigins:
-          - https://gitea.home.keithjackson.dev
+          - https://gitea.${BASE_DOMAIN}
        defaultClientScopes:
          - email
          - profile
@@ -240,9 +240,9 @@ spec:
        directAccessGrantsEnabled: false
        secret: "$(env:NEXTCLOUD_CLIENT_SECRET)"
        redirectUris:
-          - https://nextcloud.home.keithjackson.dev/apps/user_oidc/code
+          - https://nextcloud.${BASE_DOMAIN}/apps/user_oidc/code
        webOrigins:
-          - https://nextcloud.home.keithjackson.dev
+          - https://nextcloud.${BASE_DOMAIN}
        # Post-logout redirect allow-list (#logout "redirect failed"). On sign-out user_oidc redirects the
        # browser to Keycloak's end_session_endpoint with post_logout_redirect_uri=<Nextcloud base '/'> and
        # client_id=nextcloud but NO id_token_hint (provider sendIdTokenHint=0), so Keycloak validates the
@@ -250,7 +250,7 @@ spec:
        # "redirect failed". The wildcard covers the base URL and any logout landing path. Keycloak stores
        # this as the client attribute `post.logout.redirect.uris` (## separates multiple values).
        attributes:
-          post.logout.redirect.uris: https://nextcloud.home.keithjackson.dev/*
+          post.logout.redirect.uris: https://nextcloud.${BASE_DOMAIN}/*
        defaultClientScopes:
          - email
          - profile
@@ -279,9 +279,9 @@ spec:
        directAccessGrantsEnabled: false
        secret: "$(env:SEMAPHORE_CLIENT_SECRET)"
        redirectUris:
-          - https://semaphore.home.keithjackson.dev/api/auth/oidc/keycloak/redirect
+          - https://semaphore.${BASE_DOMAIN}/api/auth/oidc/keycloak/redirect
        webOrigins:
-          - https://semaphore.home.keithjackson.dev
+          - https://semaphore.${BASE_DOMAIN}
        defaultClientScopes:
          - email
          - profile
@@ -318,9 +318,9 @@ spec:
        directAccessGrantsEnabled: false
        secret: "$(env:HEADLAMP_CLIENT_SECRET)"
        redirectUris:
-          - https://headlamp.home.keithjackson.dev/oidc-callback
+          - https://headlamp.${BASE_DOMAIN}/oidc-callback
        webOrigins:
-          - https://headlamp.home.keithjackson.dev
+          - https://headlamp.${BASE_DOMAIN}
        defaultClientScopes:
          - email
          - profile
@@ -363,9 +363,9 @@ spec:
        directAccessGrantsEnabled: false
        secret: "$(env:CRAFTY_CLIENT_SECRET)"
        redirectUris:
-          - https://crafty.home.keithjackson.dev/oauth2/callback
+          - https://crafty.${BASE_DOMAIN}/oauth2/callback
        webOrigins:
-          - https://crafty.home.keithjackson.dev
+          - https://crafty.${BASE_DOMAIN}
        defaultClientScopes:
          - email
          - profile
@@ -398,9 +398,9 @@ spec:
        directAccessGrantsEnabled: false
        secret: "$(env:PIHOLE_CLIENT_SECRET)"
        redirectUris:
-          - https://pihole.home.keithjackson.dev/oauth2/callback
+          - https://pihole.${BASE_DOMAIN}/oauth2/callback
        webOrigins:
-          - https://pihole.home.keithjackson.dev
+          - https://pihole.${BASE_DOMAIN}
        defaultClientScopes:
          - email
          - profile
@@ -433,9 +433,9 @@ spec:
        directAccessGrantsEnabled: false
        secret: "$(env:JACKETT_CLIENT_SECRET)"
        redirectUris:
-          - https://jackett.home.keithjackson.dev/oauth2/callback
+          - https://jackett.${BASE_DOMAIN}/oauth2/callback
        webOrigins:
-          - https://jackett.home.keithjackson.dev
+          - https://jackett.${BASE_DOMAIN}
        defaultClientScopes:
          - email
          - profile
@@ -459,9 +459,9 @@ spec:
        directAccessGrantsEnabled: false
        secret: "$(env:QBITTORRENT_CLIENT_SECRET)"
        redirectUris:
-          - https://qbittorrent.home.keithjackson.dev/oauth2/callback
+          - https://qbittorrent.${BASE_DOMAIN}/oauth2/callback
        webOrigins:
-          - https://qbittorrent.home.keithjackson.dev
+          - https://qbittorrent.${BASE_DOMAIN}
        defaultClientScopes:
          - email
          - profile
@@ -485,9 +485,9 @@ spec:
        directAccessGrantsEnabled: false
        secret: "$(env:RADARR_CLIENT_SECRET)"
        redirectUris:
-          - https://radarr.home.keithjackson.dev/oauth2/callback
+          - https://radarr.${BASE_DOMAIN}/oauth2/callback
        webOrigins:
-          - https://radarr.home.keithjackson.dev
+          - https://radarr.${BASE_DOMAIN}
        defaultClientScopes:
          - email
          - profile
@@ -511,9 +511,9 @@ spec:
        directAccessGrantsEnabled: false
        secret: "$(env:SONARR_CLIENT_SECRET)"
        redirectUris:
-          - https://sonarr.home.keithjackson.dev/oauth2/callback
+          - https://sonarr.${BASE_DOMAIN}/oauth2/callback
        webOrigins:
-          - https://sonarr.home.keithjackson.dev
+          - https://sonarr.${BASE_DOMAIN}
        defaultClientScopes:
          - email
          - profile
@@ -30,7 +30,7 @@ spec:
          annotations:
            summary: "KubeVirt VM(s) failing to launch in {{ $labels.namespace }}"
            description: "One or more VirtualMachineInstances are in the Failed phase in namespace {{ $labels.namespace }} for 5m — a failed VM launch (image import, scheduling, or qemu error). Check `kubectl get vmi -n {{ $labels.namespace }}` and the virt-launcher pod events."
-            runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-launch-failed"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/kubevirt-launch-failed"
        # Crash-loop: the VM's virt-launcher pod is restarting repeatedly (guest kernel panic, OOM, or a
        # crashing qemu). >3 restarts in 15m is a crash-loop, not a one-off reboot.
        - alert: KubeVirtVMCrashLooping
@@ -42,7 +42,7 @@ spec:
          annotations:
            summary: "KubeVirt VM crash-looping: {{ $labels.namespace }}/{{ $labels.pod }}"
            description: "virt-launcher pod {{ $labels.namespace }}/{{ $labels.pod }} has restarted more than 3 times in 15m — the guest VM is crash-looping (panic / OOM / qemu crash)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-crashloop"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/kubevirt-crashloop"
        # Control plane down: no virt-controller or virt-api replica reports ready, so no VM lifecycle
        # operations can proceed. The metrics are `kubevirt_virt_{controller,api}_ready_status` (per-pod
        # 1=ready); sum()==0 means every replica is unready. The absent() clause also fires if the whole
@@ -59,7 +59,7 @@ spec:
          annotations:
            summary: "KubeVirt control plane degraded"
            description: "The KubeVirt control plane (virt-controller/virt-api) is not ready or its metrics are absent for 10m — VM lifecycle operations (create/start/stop) will fail (docs/06)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-control-plane"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/kubevirt-control-plane"
        # CDI import stuck: a DataVolume import/clone still in flight after 30m blocks the VM it backs.
        # kubevirt_cdi_import_pods_high_restart counts importer pods stuck restarting; a positive value
        # sustained means an import is not progressing (metric name confirmed against CDI at bring-up).
@@ -71,4 +71,4 @@ spec:
          annotations:
            summary: "CDI DataVolume import stuck: {{ $labels.namespace }}/{{ $labels.name }}"
            description: "A CDI DataVolume import/clone in {{ $labels.namespace }} has not completed after 30m — the VM disk it backs cannot start. Check the importer pod (often an egress/default-deny block on the image source, or a slow mirror)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/cdi-import-stuck"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/cdi-import-stuck"
@@ -38,7 +38,7 @@ spec:
          annotations:
            summary: "Node exporter down on {{ $labels.instance }}"
            description: "Prometheus cannot scrape node-exporter on {{ $labels.instance }} for 5m — the node may be down or unreachable."
-            runbook_url: "https://home.keithjackson.dev/runbooks/node-down"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-down"
        - alert: NodeMemoryPressureHigh
          expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.90
          for: 10m
@@ -47,7 +47,7 @@ spec:
          annotations:
            summary: "Memory pressure high on {{ $labels.instance }}"
            description: "Available memory below 10% for 10m on {{ $labels.instance }} — the ~22 GiB box is under pressure."
-            runbook_url: "https://home.keithjackson.dev/runbooks/node-memory"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-memory"
        - alert: NodeDiskFillingUp
          expr: |
            (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}
@@ -58,7 +58,7 @@ spec:
          annotations:
            summary: "Disk filling up on {{ $labels.instance }} ({{ $labels.mountpoint }})"
            description: "Less than 10% free on {{ $labels.mountpoint }} of {{ $labels.instance }} for 15m."
-            runbook_url: "https://home.keithjackson.dev/runbooks/node-disk"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-disk"
        - alert: NodeCPUPressureHigh
          # PSI "some" for CPU = fraction of wall-clock time at least one task was stalled waiting to
          # run. <5% is healthy; sustained >15% means bursty workloads are queueing against the vCPU
@@ -71,7 +71,7 @@ spec:
          annotations:
            summary: "CPU pressure high on {{ $labels.instance }}"
            description: "CPU PSI (some) above 15% for 15m on {{ $labels.instance }} — tasks are being delayed waiting for CPU (bursts exceeding the node's vCPU headroom). Add vCPUs or trim CPU-heavy workloads."
-            runbook_url: "https://home.keithjackson.dev/runbooks/node-cpu"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-cpu"
        - alert: NodeCPULoadHigh
          # 5-min load average per core. Sustained >1 means the run queue exceeds the core count (more
          # runnable/blocked tasks than CPUs). Core count = number of idle-mode cpu series per node.
@@ -86,7 +86,7 @@ spec:
          annotations:
            summary: "Load per core high on {{ $labels.instance }}"
            description: "5-min load average per core above 1.0 for 15m on {{ $labels.instance }} — the node's run queue exceeds its core count."
-            runbook_url: "https://home.keithjackson.dev/runbooks/node-cpu"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/node-cpu"

 # =====================================================================================
 # Storage — Longhorn
@@ -113,7 +113,7 @@ spec:
          annotations:
            summary: "Longhorn volume degraded: {{ $labels.volume }}"
            description: "Volume {{ $labels.volume }} has been degraded (replica rebuild / missing replica) for 10m."
-            runbook_url: "https://home.keithjackson.dev/runbooks/longhorn-degraded"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-degraded"
        - alert: LonghornVolumeFaulted
          expr: longhorn_volume_robustness == 3
          for: 1m
@@ -122,7 +122,7 @@ spec:
          annotations:
            summary: "Longhorn volume FAULTED: {{ $labels.volume }}"
            description: "Volume {{ $labels.volume }} is faulted — data at risk / offline. Immediate attention required."
-            runbook_url: "https://home.keithjackson.dev/runbooks/longhorn-faulted"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-faulted"
        - alert: LonghornVolumeCapacityHigh
          expr: (longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85
          for: 30m
@@ -131,7 +131,7 @@ spec:
          annotations:
            summary: "Longhorn volume near capacity: {{ $labels.volume }}"
            description: "Volume {{ $labels.volume }} is over 85% used for 30m."
-            runbook_url: "https://home.keithjackson.dev/runbooks/longhorn-capacity"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
        - alert: LonghornNodeStorageHigh
          expr: (longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) > 0.85
          for: 30m
@@ -140,7 +140,7 @@ spec:
          annotations:
            summary: "Longhorn node storage near capacity: {{ $labels.node }}"
            description: "Longhorn node {{ $labels.node }} is over 85% of its storage capacity for 30m."
-            runbook_url: "https://home.keithjackson.dev/runbooks/longhorn-capacity"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"

 # =====================================================================================
 # GitOps — Flux
@@ -169,7 +169,7 @@ spec:
          annotations:
            summary: "Flux resource not ready: {{ $labels.kind }}/{{ $labels.name }}"
            description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has failed to reconcile (Ready=False) for 15m — GitOps drift/failure (R-GITOPS-03)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/flux-reconcile"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-reconcile"
        - alert: FluxResourceSuspended
          expr: |
            max by (namespace, name, kind) (gotk_suspend_status) == 1
@@ -179,7 +179,7 @@ spec:
          annotations:
            summary: "Flux resource suspended: {{ $labels.kind }}/{{ $labels.name }}"
            description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has been suspended for 1h — reconciliation/drift-correction is off."
-            runbook_url: "https://home.keithjackson.dev/runbooks/flux-suspended"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-suspended"

 # =====================================================================================
 # Certificates — cert-manager
@@ -205,7 +205,7 @@ spec:
          annotations:
            summary: "Certificate expiring soon: {{ $labels.name }}"
            description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} expires in under 14 days."
-            runbook_url: "https://home.keithjackson.dev/runbooks/cert-expiry"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-expiry"
        - alert: CertificateExpiryImminent
          expr: (certmanager_certificate_expiration_timestamp_seconds - time()) < (3 * 24 * 3600)
          for: 1h
@@ -214,7 +214,7 @@ spec:
          annotations:
            summary: "Certificate expiry imminent: {{ $labels.name }}"
            description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} expires in under 3 days and has not renewed."
-            runbook_url: "https://home.keithjackson.dev/runbooks/cert-expiry"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-expiry"
        - alert: CertificateNotReady
          expr: certmanager_certificate_ready_status{condition="False"} == 1
          for: 1h
@@ -223,7 +223,7 @@ spec:
          annotations:
            summary: "Certificate not ready / issuance failing: {{ $labels.name }}"
            description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} has been in a not-ready state for 1h (ACME issuance failure?)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/cert-issuance"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-issuance"

 # =====================================================================================
 # Networking — VPN / egress-gateway / denied flows (Hubble)
@@ -255,7 +255,7 @@ spec:
          annotations:
            summary: "Egress VPN tunnel down"
            description: "The egress-gateway VPN tunnel is down (or its metric is absent) for 5m — outbound traffic could leak or stall (leak risk, docs/07)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/vpn-down"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-down"
        # (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
        # retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
        # service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`
@@ -270,7 +270,7 @@ spec:
          annotations:
            summary: "Spike in policy-denied network flows"
            description: "Policy-denied flow rate is elevated for 10m — a NetworkPolicy regression or an intrusion probe (security audit, docs/07)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/denied-flows"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/denied-flows"

 # =====================================================================================
 # Backups — Velero
@@ -296,7 +296,7 @@ spec:
          annotations:
            summary: "Velero backup failed: schedule {{ $labels.schedule }}"
            description: "One or more Velero backups failed in the last hour (R-STATE-03 — off-cluster DR at risk)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/backup-failure"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-failure"
        - alert: VeleroBackupPartialFailure
          expr: increase(velero_backup_partial_failure_total[1h]) > 0
          for: 5m
@@ -305,7 +305,7 @@ spec:
          annotations:
            summary: "Velero backup partially failed"
            description: "A Velero backup completed with partial failures in the last hour — some resources/volumes were skipped."
-            runbook_url: "https://home.keithjackson.dev/runbooks/backup-partial"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-partial"
        - alert: VeleroNoRecentBackup
          expr: (time() - velero_backup_last_successful_timestamp{schedule!=""}) > (24 * 3600)
          for: 1h
@@ -314,7 +314,7 @@ spec:
          annotations:
            summary: "No successful Velero backup in 24h: {{ $labels.schedule }}"
            description: "Schedule {{ $labels.schedule }} has not produced a successful backup in over 24h (missed schedule)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/backup-missed"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"

 # =====================================================================================
 # Identity — Keycloak (SSO is a critical dependency — docs/15)
@@ -345,7 +345,7 @@ spec:
          annotations:
            summary: "Keycloak IdP down"
            description: "No ready Keycloak pod for 5m — SSO is unavailable and dependent services cannot authenticate (docs/15)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/keycloak-down"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-down"
        - alert: KeycloakDatabaseDown
          # Keycloak's PostgreSQL runs as a CloudNativePG Cluster `keycloak-db` (pods keycloak-db-N).
          # DR-critical, Longhorn-backed. (CNPG also exports cnpg_* metrics via its PodMonitor.)
@@ -358,7 +358,7 @@ spec:
          annotations:
            summary: "Keycloak database unavailable"
            description: "Keycloak's PostgreSQL is not ready for 5m — the IdP will fail (docs/15)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/keycloak-db"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-db"

 # =====================================================================================
 # The monitor's own capacity — don't let the monitor blind itself (docs/10)
@@ -386,7 +386,7 @@ spec:
          annotations:
            summary: "Monitoring PVC filling up: {{ $labels.persistentvolumeclaim }}"
            description: "PVC {{ $labels.persistentvolumeclaim }} (Prometheus/Loki) is under 15% free — retention/PVC size may need raising before the monitor goes blind."
-            runbook_url: "https://home.keithjackson.dev/runbooks/monitoring-capacity"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/monitoring-capacity"
        - alert: PrometheusTargetsMissing
          expr: |
            (count(up == 0) / count(up)) > 0.10
@@ -396,7 +396,7 @@ spec:
          annotations:
            summary: "More than 10% of Prometheus targets are down"
            description: "Over 10% of scrape targets have been down for 15m — a broad monitoring blind spot."
-            runbook_url: "https://home.keithjackson.dev/runbooks/targets-down"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/targets-down"

 # =====================================================================================
 # Automated updates — Flux image-automation + hands-off update health (docs/11, ADR-0009)
@@ -430,7 +430,7 @@ spec:
          annotations:
            summary: "Flux image-update-automation failing: {{ $labels.name }}"
            description: "ImageUpdateAutomation {{ $labels.namespace }}/{{ $labels.name }} cannot reconcile (git write-back) for 30m — stable image advances have stopped (R-UPD-02). Check the git write credential."
-            runbook_url: "https://home.keithjackson.dev/runbooks/image-automation"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/image-automation"
        # An ImagePolicy/ImageRepository can't resolve the latest stable tag (registry rate-limit /
        # auth / bad range) → that service stops receiving updates even though automation is healthy.
        - alert: ImagePolicyStalled
@@ -444,7 +444,7 @@ spec:
          annotations:
            summary: "Image scan/policy stalled: {{ $labels.kind }}/{{ $labels.name }}"
            description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} cannot resolve its stable tag for 1h (registry rate-limit / creds / range) — no longer receiving updates (R-UPD-01)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/image-policy-stalled"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/image-policy-stalled"
        # A bad update was health-gated and AUTO-ROLLED-BACK (or is stuck mid-remediation). helm-controller
        # holds Ready=False while a failed upgrade remediates/rolls back to the last-good revision
        # (HelmRelease.spec.upgrade.remediation + .spec.rollback). This is the R-UPD-03 safety rail firing.
@@ -461,4 +461,4 @@ spec:
          annotations:
            summary: "Update health-gate failed / rolled back: {{ $labels.name }}"
            description: "HelmRelease {{ $labels.namespace }}/{{ $labels.name }} failed its health-gated upgrade and is rolling back to the last-good revision (bad-update auto-rollback — R-UPD-03). Consider an `update.pin` escape hatch (docs/11)."
-            runbook_url: "https://home.keithjackson.dev/runbooks/update-rollback"
+            runbook_url: "https://${BASE_DOMAIN}/runbooks/update-rollback"
@@ -14,7 +14,7 @@ spec:
      namespace: gateway
      sectionName: https
  hostnames:
-    - grafana.home.keithjackson.dev
+    - grafana.${BASE_DOMAIN}
  rules:
    - matches:
        - path:
@@ -194,15 +194,15 @@ spec:
            optional: true
      grafana.ini:
        server:
-          root_url: https://grafana.home.keithjackson.dev
+          root_url: https://grafana.${BASE_DOMAIN}
        auth.generic_oauth:
          enabled: true
          name: Keycloak
          client_id: grafana
          scopes: "openid email profile"
-          auth_url: https://keycloak.home.keithjackson.dev/realms/homelab/protocol/openid-connect/auth
-          token_url: https://keycloak.home.keithjackson.dev/realms/homelab/protocol/openid-connect/token
-          api_url: https://keycloak.home.keithjackson.dev/realms/homelab/protocol/openid-connect/userinfo
+          auth_url: https://keycloak.${BASE_DOMAIN}/realms/homelab/protocol/openid-connect/auth
+          token_url: https://keycloak.${BASE_DOMAIN}/realms/homelab/protocol/openid-connect/token
+          api_url: https://keycloak.${BASE_DOMAIN}/realms/homelab/protocol/openid-connect/userinfo
          login_attribute_path: preferred_username
          email_attribute_path: email
          # Map Keycloak groups (the `groups` claim) to Grafana roles (R-IDP-06): platform-admins ->
@@ -32,7 +32,7 @@ spec:
      existingSecret: oauth2-proxy-credentials
    extraArgs:
      provider: oidc
-      oidc-issuer-url: https://keycloak.home.keithjackson.dev/realms/homelab
+      oidc-issuer-url: https://keycloak.${BASE_DOMAIN}/realms/homelab
      # Forward-auth (ext-auth) mode: no upstream app; return 202 for an authenticated session so the
      # Gateway ext-authz can allow/deny. Auth headers are surfaced to the backend.
      reverse-proxy: "true"
@@ -41,9 +41,9 @@ spec:
      pass-access-token: "true"
      email-domain: "*"
      # Single sign-in/callback host for all forward-auth-protected services on the platform zone.
-      redirect-url: https://auth.home.keithjackson.dev/oauth2/callback
-      whitelist-domain: ".home.keithjackson.dev"
-      cookie-domain: ".home.keithjackson.dev"
+      redirect-url: https://auth.${BASE_DOMAIN}/oauth2/callback
+      whitelist-domain: ".${BASE_DOMAIN}"
+      cookie-domain: ".${BASE_DOMAIN}"
      cookie-secure: "true"
      # Expose group membership so per-service group authorization (R-IDP-06) can be enforced. The
      # Keycloak `oauth2-proxy` client emits a `groups` claim (full.path=false => bare names like
@@ -14,7 +14,7 @@ spec:
      namespace: gateway
      sectionName: https
  hostnames:
-    - auth.home.keithjackson.dev
+    - auth.${BASE_DOMAIN}
  rules:
    - matches:
        - path:
@@ -9,7 +9,7 @@
  "properties": {
    "name": {
      "type": "string",
-      "description": "DNS label -> <name>.home.keithjackson.dev (docs/13, docs/09). Must be a valid RFC 1123 DNS label.",
+      "description": "DNS label -> <name>.${BASE_DOMAIN}, i.e. <name>.home.keithjackson.dev on dev and <name>.prod.home.keithjackson.dev on prod (docs/13, docs/09). Must be a valid RFC 1123 DNS label.",
      "pattern": "^[a-z0-9]([-a-z0-9]*[a-z0-9])?$",
      "minLength": 1,
      "maxLength": 63
@@ -223,8 +223,8 @@
        "properties": {
          "host": {
            "type": "string",
-            "description": "FQDN under the platform zone. Must be a <label>.home.keithjackson.dev name (docs/09).",
-            "pattern": "^[a-z0-9]([-a-z0-9]*[a-z0-9])?\\.home\\.keithjackson\\.dev$"
+            "description": "Hostname under the platform zone, as a single DNS label plus the ${BASE_DOMAIN} token — e.g. 'gitea.${BASE_DOMAIN}' (docs/09). ${BASE_DOMAIN} is the per-environment hostname suffix from clusters/base/cluster-vars.yaml, resolved by Flux postBuild substitution at reconcile time (dev: home.keithjackson.dev, prod: prod.home.keithjackson.dev). AUTHOR THE TOKEN, NOT A LITERAL: a literal suffix is what let prod be authored to serve dev's public hostnames out of the shared Cloudflare zone, and prod's external-dns then deleted dev's records (infra/PROD-PROVISIONING.md §10). The literal 'home.keithjackson.dev' form stays accepted so the L1 render fixtures — which are renderer INPUTS, not reconciled manifests — keep validating unchanged.",
+            "pattern": "^[a-z0-9]([-a-z0-9]*[a-z0-9])?\\.(\\$\\{BASE_DOMAIN\\}|home\\.keithjackson\\.dev)$"
          },
          "port": {
            "type": "integer",