feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain

INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources, and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled to 0; verified restored against Cloudflare directly (my first check went through a pfSense override and falsely showed healthy — local resolvers are not evidence about public DNS). THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put "never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified --limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored manifests reached across a third party and destroyed dev's state. Cluster-to-cluster isolation was verified; the dependency that mattered was cluster-to-Cloudflare. Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod -> *.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home. PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. + apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and 12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every user, and moves gitea.home. — which is BOTH the GitOps source and the current push target. BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs. Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles" has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized: Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15 links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove --oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11 catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose occurrences and non-reconciled paths. TWO THAT CANNOT BE TOKENIZED, flagged not forced: - headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt), so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy anyway — client secrets are realm-scoped. - kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url, overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets prod.home. EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment — dev renders exactly `default` (the chart default it was already using, so its existing records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL, deliberately not `default`, so a future environment that forgets the knob owns nothing and can delete nothing rather than silently inheriting dev's identity. I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1 (DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE — ExternalDNS generates the record then skips it with "no hosted zone matching record DNS Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev. Isolation comes from the subdomain split plus txtOwnerId, which is what the incident actually needed and what a domain filter cannot provide. DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly. POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12 hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod., and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's. L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS. PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied. Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-04 22:43:54 -05:00
1 parent 9b8979a427
commit c8c1d2935d
@@ -24,9 +24,9 @@ env:
        name: gitea-bootstrap
        key: secret-key
  - name: GITEA__server__DOMAIN              # correct public clone/redirect URLs for the TLS-terminated host
-    value: gitea.home.keithjackson.dev
+    value: gitea.${BASE_DOMAIN}
  - name: GITEA__server__ROOT_URL
-    value: https://gitea.home.keithjackson.dev/
+    value: https://gitea.${BASE_DOMAIN}/
  - name: GITEA__actions__ENABLED            # Phase 6 CI/CD: declared (not implicit upstream default) so the
    value: "true"                            # in-cluster act_runner (ns gitea-actions) registers against it
  - name: GITEA__actions__DEFAULT_ACTIONS_URL  # resolve `uses: <org>/<action>@ref` from github.com (cluster
@@ -104,7 +104,7 @@ lifecycle:
          # persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker
          # file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays
          # usable). The verify-then-retry loop is the real guarantee against "exit 0 but nothing stuck".
-          GITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
+          GITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
          gitea_oidc_fail() {
            echo "[gitea-oidc] FAILED to register 'keycloak' login source: $1" >&2
            { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true
@@ -168,7 +168,7 @@ lifecycle:
          # be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,
          # the moment its OIDC guard was fixed and the block actually ran for the first time.
          exit 0
-exposure: public                # edge-forwarded git host at gitea.home.keithjackson.dev (docs/09; R-DNS/R-NET)
+exposure: public                # edge-forwarded git host at gitea.${BASE_DOMAIN} (docs/09; R-DNS/R-NET)
 egress: internet                # normal WAN egress (docs/03; R-NET-05)
 storage:
  - name: data
@@ -178,7 +178,7 @@ storage:
                                # the default /data/<name> would strand real state on the ephemeral rootfs (R-STATE-01)
 dependencies: []                # default-deny: reaches only DNS; git clients reach it north-south via the Gateway (R-NET-01/02)
 ingressRoutes:
-  - host: gitea.home.keithjackson.dev
+  - host: gitea.${BASE_DOMAIN}
    port: 3000                  # Gitea HTTP web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
 auth:
  mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC" set; R-IDP-02)
@@ -17,7 +17,7 @@ name: jackett
 kind: container
 source:
  image: lscr.io/linuxserver/jackett:latest
-exposure: internal              # LAN-only web UI at jackett.home.keithjackson.dev (docs/13 row 4; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at jackett.${BASE_DOMAIN} (docs/13 row 4; R-DNS/R-NET)
 egress: vpn                     # indexer scraping via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
 storage:
  - name: config
@@ -25,7 +25,7 @@ storage:
    class: longhorn             # indexer definitions/API keys on 3-replica Longhorn (docs/04; R-STATE-02)
 dependencies: []                # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: jackett.home.keithjackson.dev
+  - host: jackett.${BASE_DOMAIN}
    port: 9117                  # Jackett WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
 auth:
  mode: forward-auth            # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)
@@ -18,7 +18,7 @@ dependencies:
    port: 5432
    protocol: tcp
 ingressRoutes:
-  - host: keycloak.home.keithjackson.dev
+  - host: keycloak.${BASE_DOMAIN}
    port: 8080                # HTTP listener; TLS terminates at the Gateway (docs/08, docs/09)
 auth:
  mode: native                # it IS the IdP; own admin login is local + MFA (docs/15; ADR-0014). Documented exception.
@@ -50,13 +50,13 @@ env:
  #     the vhost answer for the public FQDN; OVERWRITE* make Nextcloud emit https:// self-URLs even
  #     though it speaks plain HTTP behind the proxy. ---
  - name: NEXTCLOUD_TRUSTED_DOMAINS
-    value: nextcloud.home.keithjackson.dev
+    value: nextcloud.${BASE_DOMAIN}
  - name: OVERWRITEHOST
-    value: nextcloud.home.keithjackson.dev
+    value: nextcloud.${BASE_DOMAIN}
  - name: OVERWRITEPROTOCOL
    value: https
  - name: OVERWRITECLIURL
-    value: https://nextcloud.home.keithjackson.dev
+    value: https://nextcloud.${BASE_DOMAIN}
  # --- Keycloak native OIDC SSO (auth.mode oidc; docs/15 "Native OIDC"; R-IDP-02/06). The confidential
  #     `nextcloud` realm client's secret is mirrored into this namespace's nextcloud-bootstrap Secret
  #     (oidc-client-secret; same value SOPS-held in keycloak-realm-clients). The postStart hook reads it
@@ -88,7 +88,7 @@ lifecycle:
          # provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as
          # ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.
          OCC() { su -s /bin/sh www-data -c "php /var/www/html/occ $*"; }
-          NEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
+          NEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
          nextcloud_oidc_fail() {
            echo "[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1" >&2
            { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true
@@ -228,7 +228,7 @@ dependencies:
    port: 5432                  # Renders the egress half; the DB namespace mirrors keycloak-db (no
    protocol: tcp               # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
 ingressRoutes:
-  - host: nextcloud.home.keithjackson.dev
+  - host: nextcloud.${BASE_DOMAIN}
    port: 80                    # apache HTTP; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
  mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)
@@ -14,7 +14,7 @@ storage:
    class: longhorn           # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
 dependencies: []              # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: pihole.home.keithjackson.dev
+  - host: pihole.${BASE_DOMAIN}
    port: 80                  # admin web UI, protected at the Gateway by forward-auth (docs/09)
 auth:
  mode: forward-auth          # oauth2-proxy -> Keycloak; bare UI, no native OIDC (docs/15; R-IDP-02)
@@ -72,7 +72,7 @@ storage:
                              # so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
 dependencies: []              # default-deny: reaches only DNS; clients/Seerr reach IT (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: plex.home.keithjackson.dev
+  - host: plex.${BASE_DOMAIN}
    port: 32400               # Plex web UI + API served on 32400 (wildcard TLS at the Gateway; docs/08/09)
 auth:
  mode: native                # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)
@@ -32,7 +32,7 @@ name: qbittorrent
 kind: container
 source:
  image: lscr.io/linuxserver/qbittorrent:5.1.2   # pinned stable tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at qbittorrent.home.keithjackson.dev (docs/13 row 13; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at qbittorrent.${BASE_DOMAIN} (docs/13 row 13; R-DNS/R-NET)
 egress: vpn                     # ALL public egress via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
 storage:
  - name: config
@@ -40,7 +40,7 @@ storage:
    class: longhorn             # settings/categories/session on 3-replica Longhorn; DR-critical (docs/04; R-STATE-02)
 dependencies: []                # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
 ingressRoutes:
-  - host: qbittorrent.home.keithjackson.dev
+  - host: qbittorrent.${BASE_DOMAIN}
    port: 8080                  # qBittorrent WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
 auth:
  mode: forward-auth            # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)
@@ -9,7 +9,7 @@ name: radarr
 kind: container
 source:
  image: lscr.io/linuxserver/radarr:5.21.1   # pinned stable v5 tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at radarr.home.keithjackson.dev (docs/13 row 3; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at radarr.${BASE_DOMAIN} (docs/13 row 3; R-DNS/R-NET)
 egress: vpn                     # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
                                # kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
 storage:
@@ -29,7 +29,7 @@ dependencies:
    port: 9117
    protocol: tcp
 ingressRoutes:
-  - host: radarr.home.keithjackson.dev
+  - host: radarr.${BASE_DOMAIN}
    port: 7878                  # Radarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
 auth:
  mode: forward-auth            # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)
@@ -44,7 +44,15 @@ except ImportError:  # pragma: no cover - explicit, actionable failure

 # --- Label / naming conventions (match platform/kyverno/policies + platform/*/) --------
 LBL = "homelab.keithjackson.dev"
-ZONE = "home.keithjackson.dev"
+# The platform hostname suffix, emitted as the Flux post-build TOKEN rather than a literal.
+# Every rendered manifest lands under a Flux path whose Kustomization declares
+# postBuild.substituteFrom cluster-vars, so `${BASE_DOMAIN}` is resolved per environment at
+# reconcile time: dev renders home.keithjackson.dev, prod renders prod.home.keithjackson.dev.
+# It is a DECLARED cluster-vars knob (clusters/base/cluster-vars.yaml), so it is written BARE —
+# the escape contract below reserves `$${...}` for expansions meant for a container's shell.
+# Hardcoding the literal here is what let prod be authored to serve dev's public hostnames,
+# and prod's external-dns then deleted dev's records (infra/PROD-PROVISIONING.md §10).
+ZONE = "${BASE_DOMAIN}"
 GATEWAY_NAME = "shared"
 GATEWAY_NS = "gateway"
 KP_RELEASE = "kube-prometheus-stack"  # ServiceMonitor/PrometheusRule discovery label
@@ -57,7 +57,7 @@ dependencies:
    port: 32400
    protocol: tcp
 ingressRoutes:
-  - host: seerr.home.keithjackson.dev
+  - host: seerr.${BASE_DOMAIN}
    port: 5055                   # Overseerr web UI + API; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
  mode: native                   # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)
@@ -58,7 +58,7 @@ env:
  - name: SEMAPHORE_ADMIN_NAME
    value: Semaphore Admin
  - name: SEMAPHORE_ADMIN_EMAIL
-    value: admin@home.keithjackson.dev
+    value: admin@${BASE_DOMAIN}
  # --- Persistent access-key encryption key (see header): FIXED base64 32-byte key from SOPS so the
  #     SSH keys / credentials Semaphore stores in the DB stay decryptable across restarts. ---
  - name: SEMAPHORE_ACCESS_KEY_ENCRYPTION
@@ -110,7 +110,7 @@ dependencies:
    port: 5432                  # Renders the egress half; the DB namespace mirrors keycloak-db (no
    protocol: tcp               # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
 ingressRoutes:
-  - host: semaphore.home.keithjackson.dev
+  - host: semaphore.${BASE_DOMAIN}
    port: 3000                  # Semaphore HTTP web/API; wildcard TLS terminates at the shared Gateway (docs/08/09)
 auth:
  mode: oidc                    # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)
@@ -9,7 +9,7 @@ name: sonarr
 kind: container
 source:
  image: lscr.io/linuxserver/sonarr:4.0.15   # pinned stable v4 tag (R-UPD-01); update.range governs advancement
-exposure: internal              # LAN-only web UI at sonarr.home.keithjackson.dev (docs/13 row 2; R-DNS/R-NET)
+exposure: internal              # LAN-only web UI at sonarr.${BASE_DOMAIN} (docs/13 row 2; R-DNS/R-NET)
 egress: vpn                     # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
                                # kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
 storage:
@@ -29,7 +29,7 @@ dependencies:
    port: 9117
    protocol: tcp
 ingressRoutes:
-  - host: sonarr.home.keithjackson.dev
+  - host: sonarr.${BASE_DOMAIN}
    port: 8989                  # Sonarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
 auth:
  mode: forward-auth            # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)