diff --git a/clusters/prod/kustomization.yaml b/clusters/prod/kustomization.yaml index 171eda3..ec34c1a 100644 --- a/clusters/prod/kustomization.yaml +++ b/clusters/prod/kustomization.yaml @@ -20,6 +20,129 @@ resources: - ../base - sync.yaml # self-managing root Kustomization -> ./clusters/prod (REQUIRED: see below) patches: + # --- SOPS decryption (operator decision 2026-09-01: prod uses the SAME age key as dev) --- + # Mirrors clusters/dev exactly, one stanza per Kustomization that carries encrypted Secrets. + # The `sops-age` Secret is created OUT OF BAND on the prod cluster + # (bootstrap/sops/install-age-key.sh) — referenced by NAME only, never inlined. + # Add a stanza here whenever `render.py --emit-waves` introduces a new deepest wave. + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: platform + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: apps-wave-0 + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: apps-wave-1 + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: apps-wave-2 + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: oauth2-proxy + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: headlamp + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: crafty + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: velero + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + # --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) --- + # + # *** BOOTSTRAP BLOCKER — this URL CANNOT work on a cold prod cluster. READ BEFORE BRING-UP. *** + # + # Two independent reasons, both consequences of the split-horizon design: + # 1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's + # GitOps source lives inside the very cluster slated for decommission. Gitea must move to + # prod (or off-cluster) BEFORE dev is retired, or prod loses its source of truth. + # 2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml + # answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP}. On prod that is + # 192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve + # gitea.home.keithjackson.dev to itself and never reach a repository. + # + # Bring-up therefore needs a TEMPORARY source (the bootstrap host-Gitea, or an off-cluster + # mirror), with a cutover to the in-cluster URL once Gitea actually runs on prod — exactly the + # sequence docs/05 describes for dev's own Phase-6 cutover. Tracked in PROD-PROVISIONING.md §7. + # + # The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev. + # CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH + # CLUSTERS. There is no promotion step and no soak window — dev and prod move together. + - patch: |- + apiVersion: source.toolkit.fluxcd.io/v1 + kind: GitRepository + metadata: + name: flux-system + namespace: flux-system + spec: + url: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git + ref: + branch: main - target: kind: ConfigMap name: cluster-vars diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index a337754..55202bf 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -317,13 +317,26 @@ no extra vars is byte-for-byte the dev behaviour it always had (verified by a be *load-bearing*, not cosmetic: `clusters/base/platform.yaml` now carries `dependsOn: [flux-system]`, so without a root object on prod, `platform` would block forever on a dependency that does not exist and nothing downstream would reconcile. - - [ ] **SOPS decryption** — `clusters/dev/kustomization.yaml` patches every Kustomization with - `decryption: {provider: sops, secretRef: sops-age}`. Prod has no equivalent, so every - SOPS-encrypted Secret would fail to decrypt. Needs the prod age key decision first - (same key as dev, or a prod-specific key — a real security choice, not a copy). - - [ ] **GitRepository source URL** — dev patches in the real Gitea URL. Prod needs its own, - plus a decision on whether prod tracks the same branch as dev or a separate one. - Tracking the same branch means a dev-facing merge reconciles prod. + - [x] **SOPS decryption** — DONE. Operator decision 2026-09-01: **prod uses the SAME age key + as dev**. Eight `decryption: {provider: sops, secretRef: sops-age}` patches mirror dev + exactly. The `sops-age` Secret is still created OUT OF BAND on the prod cluster + (`bootstrap/sops/install-age-key.sh`) — the overlay references it by name only. + - [~] **GitRepository source** — BRANCH settled, URL still blocked. Operator decision + 2026-09-01: **prod tracks `main`, the same branch as dev**, because dev is temporary and + will be decommissioned once prod is stable. Recorded consequence: once prod is live, + **every merge to `main` reconciles BOTH clusters** — no promotion step, no soak window. + The URL is patched in but **cannot work on a cold prod cluster**, for two independent + reasons found 2026-09-01: + 1. **Self-hosting circularity** — Gitea runs on the DEV cluster, the one slated for + decommission. Gitea must move to prod (or off-cluster) BEFORE dev is retired, or prod + loses its source of truth. **This is the decommission plan's first task, not its last.** + 2. **Split-horizon sends it to the wrong place** — `coredns-custom` answers the entire + `home.keithjackson.dev` zone with `${GATEWAY_VIP}`, which on prod is `192.168.2.20`: + prod's OWN gateway, where no Gitea listens. Prod's Flux would resolve + `gitea.home.keithjackson.dev` to itself and never reach a repository. + Bring-up needs a TEMPORARY source (bootstrap host-Gitea, or an off-cluster mirror) and a + cutover once Gitea runs on prod — the same sequence `docs/05` describes for dev's own + Phase-6 cutover. - [ ] Confirm prod's backup target: dev's S3 (`versitygw` on the Proxmox host, `192.168.1.9`) is on the dev side of the network. Off-cluster backup is DR-critical (R-STATE-03).