--- # clusters/prod — DESIGN-ONLY overlay. NOT a build target until roadmap Phase 7 opens prod # (spec roadmap/phases.md, AGENTS.md §6: "Never build directly on production"). # # When Phase 7 is opened, `flux bootstrap ... --path=clusters/prod` points the prod cluster's Flux # here. Like dev, this overlay adds NO resources — it ONLY patches cluster-vars with prod-scoped # values (R-ENV-02). Promotion to prod is a change to THESE variables, never a base edit. # # *** PHASE-7 BLOCKER — READ BEFORE BOOTSTRAPPING PROD *** # `clusters/base/platform.yaml` now carries `dependsOn: [flux-system]` (the 2026-09-01 substitution- # race fix). That names the ROOT Kustomization, which clusters/dev supplies as `dev/sync.yaml` # (path ./clusters/dev). THIS overlay has no sync.yaml yet, so as written a prod bootstrap would # leave `platform` stuck on "dependency 'flux-system/flux-system' is not ready" FOREVER and nothing # downstream would reconcile. Phase 7 MUST add `clusters/prod/sync.yaml` — a copy of dev/sync.yaml # with `path: ./clusters/prod` — as part of opening this overlay. (Adding it now is deliberately NOT # done here: prod is design-only and its root object is the prod bring-up owner's call.) apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../base - sync.yaml # self-managing root Kustomization -> ./clusters/prod (REQUIRED: see below) patches: # --- SOPS decryption (operator decision 2026-09-01: prod uses the SAME age key as dev) --- # Mirrors clusters/dev exactly, one stanza per Kustomization that carries encrypted Secrets. # The `sops-age` Secret is created OUT OF BAND on the prod cluster # (bootstrap/sops/install-age-key.sh) — referenced by NAME only, never inlined. # Add a stanza here whenever `render.py --emit-waves` introduces a new deepest wave. - patch: |- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: platform namespace: flux-system spec: decryption: provider: sops secretRef: name: sops-age - patch: |- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: apps-wave-0 namespace: flux-system spec: decryption: provider: sops secretRef: name: sops-age - patch: |- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: apps-wave-1 namespace: flux-system spec: decryption: provider: sops secretRef: name: sops-age - patch: |- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: apps-wave-2 namespace: flux-system spec: decryption: provider: sops secretRef: name: sops-age - patch: |- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: oauth2-proxy namespace: flux-system spec: decryption: provider: sops secretRef: name: sops-age - patch: |- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: headlamp namespace: flux-system spec: decryption: provider: sops secretRef: name: sops-age - patch: |- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: crafty namespace: flux-system spec: decryption: provider: sops secretRef: name: sops-age - patch: |- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: velero namespace: flux-system spec: decryption: provider: sops secretRef: name: sops-age # --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) --- # # ############################################################################################ # ### ### # ### *** T E M P O R A R Y B O O T S T R A P S O U R C E — NOT PERMANENT *** ### # ### ### # ### The url below is a Gitea container running ON THE OPERATOR'S WORKSTATION, outside ### # ### any cluster. It exists ONLY to get prod's first reconcile off the ground. It MUST ### # ### be cut over to the in-cluster URL once Gitea runs on prod. If you are reading this ### # ### and prod has been serving traffic for months, THE CUTOVER WAS NEVER DONE — that is ### # ### a live single point of failure, not a settled design. See "CUTOVER" below. ### # ### ### # ############################################################################################ # # WHY THE IN-CLUSTER URL CANNOT BE USED YET (blocker found 2026-09-01, still true; the # unusable value was `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`). # REASONING UPDATED 2026-09-04 for the new suffix direction (prod = the apex, dev = dev.home.), # which changes WHY the blocker holds but not THAT it holds: # 1. SELF-HOSTING CIRCULARITY — still true, and now the only reason. The Gitea this repo is # pushed to runs ON THE DEV CLUSTER. Prod syncing from it would put prod's GitOps source # inside the very cluster slated for decommission. Gitea must move to prod (or off-cluster) # BEFORE dev is retired, or prod loses its source of truth. # NOTE the suffix change makes the two Giteas DIFFERENT HOSTS: dev's is now # gitea.dev.home.keithjackson.dev and prod's is gitea.home.keithjackson.dev. Prod pointing at # the apex name therefore means "prod's OWN Gitea", never dev's — which is what makes the # cutover below a one-line change once prod actually runs Gitea. # 2. SPLIT-HORIZON DNS — NO LONGER A BLOCKER, and the note is kept because the old reasoning is # easy to re-derive incorrectly. platform/coredns/coredns-custom.yaml answers the whole # ${BASE_DOMAIN} zone with ${GATEWAY_VIP}; when prod's BASE_DOMAIN was `prod.home.` that meant # prod resolved gitea.home.keithjackson.dev to PROD's gateway (192.168.2.20) where no Gitea # listened — a misdirection. Now that prod IS the apex, resolving gitea.home.keithjackson.dev # to prod's own Gateway is CORRECT: that is where prod's Gitea will be served from. What # remains is purely reason 1 — the Gitea is not there YET. # # WHAT THE TEMPORARY SOURCE IS (stood up 2026-09-04 on explicit operator direction): # bootstrap/gitea/bootstrap-gitea.sh with HOST_NAT_IP=192.168.2.132 — a podman container on the # operator's workstation, Gitea 1.22.6, listening on 192.168.2.132:3000. All three prod nodes # reach it DIRECTLY on the LAN: no libvirt NAT hop (that network is retired) and no firewall # change was needed. This is the same "local bootstrap source" pattern dev itself used before # its Phase-6 cutover (docs/05 "Source-of-truth"). # # PLAIN HTTP, AND WHY THAT IS ACCEPTED HERE: the bootstrap Gitea has no TLS, so there is no cert # for Flux to trust and no caCertSecretRef to wire. Traffic is LAN-local between prod nodes and # the workstation, the repo carries NO plaintext secrets (everything sensitive is SOPS-encrypted, # .sops.yaml), and the window is the bring-up only. Do NOT extend this posture past cutover. # # NO secretRef, DELIBERATELY: the repo is PUBLIC on the bootstrap Gitea, so anonymous clone # works and Flux needs no git credentials. VERIFIED 2026-09-04 by unauthenticated request: # curl http://192.168.2.132:3000/api/v1/repos/Home-Network/home-network-k3s-impl -> "private":false # git ls-remote http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git -> refs/heads/main # If the GitRepository ever reports an auth failure that looks like a network problem, RE-CHECK # THAT FIRST: a private repo here fails exactly that way. # # *** CUTOVER — the exit condition, mirroring dev's Phase-6 cutover (docs/05) *** # WHEN: Gitea runs on prod (reconciled in-cluster) and answers gitea.home.keithjackson.dev # through prod's own Gateway VIP with a publicly-trusted cert. # DO: 1. push/mirror this repo into the prod in-cluster Gitea; # 2. change the url below to # https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git # and DELETE this whole TEMPORARY block (dev's equivalent stanza, ~15 lines above in # clusters/dev/kustomization.yaml, is what "done" looks like); # 3. confirm `flux get sources git -n flux-system` reconciles the new revision; # 4. only THEN stop the workstation container. Keep it stopped-not-deleted as recovery, # exactly as dev retained its bootstrap Gitea. # UNTIL THEN: prod's reconciliation depends on a container on a workstation that may be # powered off. Flux keeps the last-reconciled state running, but NO new revision lands while # it is down. Tracked as an open blocker in infra/PROD-PROVISIONING.md §7. # # The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev. # CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH # CLUSTERS. There is no promotion step and no soak window — dev and prod move together. # NOTE for bring-up: bootstrap-gitea.sh pushes the operator's CURRENT HEAD to the bootstrap # Gitea's `main`, whatever branch that is. Before bootstrapping prod, confirm that remote `main` # is the revision you intend prod to run (`git ls-remote` above, against local `main`). - patch: |- apiVersion: source.toolkit.fluxcd.io/v1 kind: GitRepository metadata: name: flux-system namespace: flux-system spec: # TEMPORARY bootstrap source — workstation container, NOT in any cluster. See the block # above for the cutover to https://gitea.home.keithjackson.dev/... . Do not treat as final. url: http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git ref: branch: main - target: kind: ConfigMap name: cluster-vars patch: |- - op: replace path: /data/CLUSTER_ENV value: "prod" # trusted, rate-limited real certs in prod — ADR-0006. - op: replace path: /data/ACME_CLUSTER_ISSUER value: "letsencrypt-prod" # ========================================================================================== # PROD DNS IDENTITY (R-DNS-01/02/03) — PROD **IS** THE APEX. # ========================================================================================== # OPERATOR DECISION 2026-09-04, superseding the interim split committed earlier the same day # (prod on `prod.home.` behind apex CNAMEs). The simpler target, and the one authored here: # # dev -> dev.home.keithjackson.dev (dev carries the label; clusters/dev) # prod -> home.keithjackson.dev (THE APEX — this knob) # # There is NO `prod.` label and NO CNAME layer. `.home.keithjackson.dev` is prod's REAL # name, so nothing has to alias anything: prod's Gitea is gitea.home.keithjackson.dev, its # Keycloak keycloak.home.keithjackson.dev, and the wildcard cert is `*.home.keithjackson.dev`. # # BACKGROUND — WHY THE SUFFIXES HAD TO SPLIT AT ALL. Until 2026-09-04 every hostname was # HARDCODED to home.keithjackson.dev in the base manifests, so prod was authored to serve the # EXACT SAME public names as the live dev cluster out of the SAME Cloudflare zone. Two # consequences, one of which already happened: # 1. HAPPENED: prod's external-dns (policy: sync, owner id `default` — the same implicit # default dev had) concluded it owned dev's records and DELETED EVERY ONE of them. # 2. QUEUED BEHIND IT: platform-config would have requested a `*.home.keithjackson.dev` # wildcard from letsencrypt-prod for a cert dev already holds, and both clusters would # have raced each other's `_acme-challenge` TXT records on that one zone. # # *** THE ORDERING CONSTRAINT THIS VALUE CREATES — READ BEFORE UNSUSPENDING ANYTHING. *** # Prod now claims the very names the LIVE dev cluster is serving today. Both hazards above are # therefore still armed until DEV HAS VACATED THE APEX. Prod's `platform` and its 10 downstream # tiers are SUSPENDED for exactly this reason. The gate is not "review is done", it is: # (a) dev is reconciling on *.dev.home.keithjackson.dev with a Ready wildcard cert, AND # (b) dev's old apex A/CNAME records and their TXT registry entries are GONE from Cloudflare # (verified against Cloudflare directly — the workstation's dnsmasq is NOT evidence about public # DNS; that mistake was made on 2026-09-04), AND # (c) dev's old `*.home.keithjackson.dev` cert/Certificate is released, because prod ordering # that same name set while dev's still exists counts against LE's duplicate-certificate # limit (5 per week per identical name set). # Full ordered runbook: infra/PROD-PROVISIONING.md §10. Unsuspending before (a)-(c) re-creates # the 2026-09-04 incident with the roles reversed. # # [parity] MUST equal k3s_base_domain in infra/ansible/inventory/group_vars/k3s_prod/vars.yml. # The kube-apiserver's oidc-issuer-url is an INSTALL-TIME k3s flag, not a manifest, so it is # set Ansible-side from that variable and cannot read this one. If they disagree, kubectl SSO # trusts a different Keycloak than the one prod actually runs. - op: replace path: /data/BASE_DOMAIN value: "home.keithjackson.dev" - op: replace path: /data/EXTERNALDNS_DOMAIN_FILTER # MUST be the Cloudflare ZONE. Prod now serves the apex, so `home.keithjackson.dev` looks # like a tighter, more honest filter than the zone — IT IS NOT, IT IS AN OUTAGE. FINDING # F-DNS1 (DECISIONS-LOG 2026-08-17) proved a STRICT SUBDOMAIN of the hosted zone drops the # zone entirely: ExternalDNS generates the record then skips it with "no hosted zone matching # record DNS Name", so prod publishes NOTHING. An agent narrowed this to # prod.home.keithjackson.dev on 2026-09-04 and it was reverted in review; do not retry it. # Isolation from dev does NOT come from this filter — it comes from # (1) the two clusters requesting DISJOINT hostname sets (apex vs dev.home.), and # (2) txtOwnerId: prod vs dev, so neither registry ever claims the other's records. # That pairing is what the 2026-09-04 incident needed and this filter cannot provide. value: "keithjackson.dev" # SAFETY NET #1 — THE ACTUAL ROOT CAUSE OF THE INCIDENT. Both clusters left this unset and so # both inherited the chart default `default`; prod then treated dev's TXT-registry entries as # its own and, under policy: sync, deleted the records behind them. # # PROD MUST NOT BE GIVEN `default`, however tempting it now looks. Prod is taking over the apex # NAMES, and dev's existing apex records are registered under owner `default` — so `default` is # the one value that would let prod adopt them. It would also let prod SYNC-DELETE the ones no # prod source claims, on a cluster dev is still serving. Prod inherits the apex names; it must # never inherit the apex OWNER ID. `prod` keeps prod fail-safe: records owned by `default` are # invisible to it, so an early unsuspend cannot destroy dev's DNS the way 2026-09-04 did — it # simply fails to publish until those records are retired, which is the correct behaviour. # dev is `dev` (clusters/dev, changed with dev's migration); the base default is a # non-colliding sentinel. All three differ. - op: replace path: /data/EXTERNALDNS_TXT_OWNER_ID value: "prod" - op: replace path: /data/LONGHORN_DEFAULT_REPLICA_COUNT value: "3" # Longhorn's on-node replica directory: a SUBDIRECTORY of the claimed data partition, not # the mount root. Prod's ~424 GB partition (LABEL=k3s-data) is mounted at the neutral /data # and serves TWO tenants — /data/longhorn here, and /data/k3s for k3s's entire data dir # (embedded etcd + the containerd image store), which had to move off the 11.8 GiB /var: # measured on dev, containerd alone is 21-25 GB per node, so prod would have filled /var on # its first image pulls and etcd could not have written its WAL. infra/PROD-PROVISIONING.md §2. # # [parity] MUST equal node_baseline_longhorn_data_path in # infra/ansible/inventory/group_vars/k3s_prod/vars.yml — Ansible creates and mounts the # directory, this tells Longhorn to use it. If they disagree, Longhorn writes replicas to a # path on the OS filesystem instead of the data partition and /var fills up. - op: replace path: /data/LONGHORN_DATA_PATH value: "/data/longhorn" # longer retention / larger volumes on prod NVMe. - op: replace path: /data/MONITORING_PROMETHEUS_RETENTION value: "30d" - op: replace path: /data/MONITORING_PROMETHEUS_PVC_SIZE value: "50Gi" # Size-cap below the 50Gi PVC (headroom for WAL + head + compaction scratch): prod may hit the # size cap before 30d if series volume is high, keeping the disk from filling either way. - op: replace path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE value: "45GiB" - op: replace path: /data/MONITORING_LOKI_PVC_SIZE value: "50Gi" # HA control-plane addons in prod. - op: replace path: /data/KEYCLOAK_REPLICAS value: "2" - op: replace path: /data/OAUTH2_PROXY_REPLICAS value: "2" - op: replace path: /data/VELERO_RETENTION value: "720h" # ========================================================================================== # PROD LAN ADDRESSING (192.168.2.0/24). Nodes: k3s-prod-1/2/3 = .10 / .11 / .12. # ========================================================================================== # These knobs were HARDCODED to the dev LAN inside base manifests until 2026-08-31; prod would # have come up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN. They are knobs now, and # this overlay supplies every one of them. Anything the operator has NOT yet decided stays an # INVALID `TBD-PROD-*` placeholder on purpose: base defaults are the DEV values, so an OMITTED # op means prod silently inherits a dev address — far worse than a failed apply. `TBD-PROD-*` # is not a valid IP/URL, so Cilium CRD validation / LB-IPAM / CNPG / Velero reject it LOUDLY. # # ------------------------------------------------------------------------------------------ # ADDRESS PLAN (operator, 2026-08-31) — see infra/PROD-PROVISIONING.md §4 for the full table. # ------------------------------------------------------------------------------------------ # nodes 192.168.2.10 / .11 / .12 (Ansible inventory) # control-plane VIP 192.168.2.9 (Ansible group_vars/k3s_prod — NOT here) # ingress Gateway VIP 192.168.2.20 (GATEWAY_VIP, below) # Cilium LB pool 192.168.2.20 - .30 (CILIUM_LB_POOL_*, below) # # CONTAINMENT INVARIANT SATISFIED: GATEWAY_VIP (.20) IS CILIUM_LB_POOL_START, mirroring dev # (.53 of .53-.60). This matters because the `shared` Gateway (platform/gateway/gateway.yaml) # pins NOTHING — no `lbipam.cilium.io/ips`, no spec.addresses (unlike platform/crafty and # apps/vms/unifi, which DO pin) — so Cilium LB-IPAM assigns it the FIRST FREE address of a # matching pool block. Had GATEWAY_VIP stayed outside every pool (the original .21-.30 # proposal), the Gateway would have come up on .21 while split-horizon coredns-custom kept # answering *.home.keithjackson.dev with .20 — an address nothing announces — blackholing # in-cluster hairpin traffic (OIDC clients -> Keycloak/Grafana) while looking healthy. # KEEP GATEWAY_VIP == CILIUM_LB_POOL_START when changing either. # The control-plane VIP (.9) is deliberately OUTSIDE this pool: kube-vip floats it, and # LB-IPAM must never be able to hand it to a Service. - op: replace path: /data/GATEWAY_VIP value: "192.168.2.20" # Cilium LB IP pool, operator-supplied: 192.168.2.20-.30 (START == GATEWAY_VIP, see above). # Mirror this range into the prod tofu reservation (cilium_l2_vip_range) if prod ever gets one. - op: replace path: /data/CILIUM_LB_POOL_START value: "192.168.2.20" - op: replace path: /data/CILIUM_LB_POOL_STOP value: "192.168.2.30" # --- still UNDECIDED -> loud placeholders (must be real before `flux bootstrap`) ------------ # k3s API endpoint Cilium's kube-proxy replacement dials. This is the prod CONTROL-PLANE VIP # (kube-vip), NOT .20 (the ingress VIP) and NOT the single node address .10. It is owned # Ansible-side (prod inventory k3s_api_host) and MUST equal that value — [parity] contract in # platform/cilium/helmrelease.yaml. Not yet decided. - op: replace path: /data/K3S_API_HOST value: "192.168.2.10" # Pinned L4 VIPs, both carved out of the pool by lbipam.cilium.io/ips. Not yet assigned for # prod. Leaving the dev literals in place would hand a prod LoadBalancer a dev-LAN address, so # they stay placeholders; whatever is chosen MUST fall inside [POOL_START, POOL_STOP] and not # collide with the Gateway VIP or the node IPs (.10/.11/.12). - op: replace path: /data/UNIFI_L4_VIP value: "192.168.2.21" - op: replace path: /data/CRAFTY_L4_VIP value: "192.168.2.22" # ------------------------------------------------------------------------------------------ # OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05). # # THERE IS DELIBERATELY NO `S3_BACKUP_ENDPOINT` OP HERE, and that is a reversal of the rule # stated at the top of this overlay ("an OMITTED op means prod silently inherits a dev # address"). That rule exists because base defaults are dev's LAN addresses. R2 is not one: it # is a single shared, environment-independent endpoint, so under R-ENV-02 it belongs in base and # overriding it with an identical value would only create two places to change on a rotation — # the exact drift this repo has been bitten by before. Isolation does NOT come from the # endpoint; it comes from the two path knobs below. # # WHY R2 AND NOT THE MinIO the operator first named: prod is on the SOHO network # 192.168.2.0/24, which is pfSense's WAN, while the MinIO sits inside pfSense's LAN # 192.168.1.0/24. Measured from a pod INSIDE prod (the workstation proves nothing about pod # egress): `curl http://192.168.1.9:9000/minio/health/live` → connection timed out after 15s. # Reaching it would need a WAN→LAN rule that does not exist. R2 needs no firewall change, and # unlike that MinIO — which runs on the same Proxmox host as dev's VMs — it is genuinely # off-site. Verify prod can reach the new endpoint before resuming velero: # bootstrap/standup.sh ... --s3-endpoint s3check # # THE PATH KNOBS ARE WHAT MAKE ONE SHARED BUCKET SAFE. Nothing about the paths was # per-environment before 2026-09-05: Velero used one unprefixed bucket, and every CNPG Cluster # used `s3://cnpg` with serverName defaulting to the CLUSTER NAME — and both environments have # keycloak-db / nextcloud-db / semaphore-db. Without these knobs prod would (a) adopt and # TTL-delete dev's Velero backups and (b) interleave two Postgres servers' WAL into ONE Barman # stream, corrupting the chain and destroying PITR for both. See clusters/base/cluster-vars.yaml. # ------------------------------------------------------------------------------------------ - op: replace path: /data/VELERO_BACKUP_PREFIX value: "prod" # Barman objects land at s3://cnpg/prod//; dev's at s3://cnpg/dev//. - op: replace path: /data/CNPG_BACKUP_PATH value: "s3://cnpg/prod" # TODO(WS-INFRA, Phase 7): pin prod node placement once the bare-metal role labels exist.