From 6b4da27f841a8193ebb68f95336ea49750a3b336 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 1 Sep 2026 16:27:32 -0500 Subject: [PATCH] feat(prod): SOPS decryption (same age key as dev) + GitOps source on `main` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator decisions, 2026-09-01: 1. Prod uses the SAME SOPS age key as dev. 2. Prod tracks the SAME BRANCH as dev (`main`). Dev is a temporary instance and will be decommissioned once prod is deemed stable. SOPS: eight `decryption: {provider: sops, secretRef: sops-age}` patches mirroring dev exactly — platform, apps-wave-0/1/2, oauth2-proxy, headlamp, crafty, velero. The sops-age Secret is still created OUT OF BAND on prod (bootstrap/sops/install-age-key.sh); the overlay references it by name only, never inlines key material. RECORDED CONSEQUENCE of the shared branch: once prod is live, EVERY MERGE TO main RECONCILES BOTH CLUSTERS. There is no promotion step and no soak window — dev and prod move together. That is a deliberate consequence of dev being temporary, not an oversight, but it removes the "prove it on dev first" safety net that the last few days have leaned on heavily. Worth revisiting if dev outlives its expected life. *** BOOTSTRAP BLOCKER recorded, NOT papered over *** — the source URL cannot work on a cold prod cluster, for two independent reasons found while wiring this: 1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER — the one slated for decommission. Prod syncing from it means prod's source of truth lives inside the cluster being retired. Gitea must move to prod (or off-cluster) BEFORE dev is torn down. This makes Gitea migration the FIRST task of the decommission plan, not the last. 2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP} via a template plugin. On prod that is 192.168.2.20 — PROD's own gateway, where no Gitea listens. Prod Flux would resolve gitea.home.keithjackson.dev to itself and never reach a repository. This is the same split-horizon mechanism that makes dev work, inverted against us. Bring-up therefore needs a TEMPORARY source (the bootstrap host-Gitea, or an off-cluster mirror) with a cutover to the in-cluster URL once Gitea actually runs on prod — precisely the sequence docs/05 describes for dev's own Phase-6 cutover. The branch is settled and correct; only the host is blocked. Documented at the patch site and in PROD-PROVISIONING.md §7. Gate: MERGE GATE: PASS. Serves: R-SEC-01, R-GITOPS-01, R-ENV-02, R-ENV-03. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) --- clusters/prod/kustomization.yaml | 123 +++++++++++++++++++++++++++++++ infra/PROD-PROVISIONING.md | 27 +++++-- 2 files changed, 143 insertions(+), 7 deletions(-) diff --git a/clusters/prod/kustomization.yaml b/clusters/prod/kustomization.yaml index 171eda3..ec34c1a 100644 --- a/clusters/prod/kustomization.yaml +++ b/clusters/prod/kustomization.yaml @@ -20,6 +20,129 @@ resources: - ../base - sync.yaml # self-managing root Kustomization -> ./clusters/prod (REQUIRED: see below) patches: + # --- SOPS decryption (operator decision 2026-09-01: prod uses the SAME age key as dev) --- + # Mirrors clusters/dev exactly, one stanza per Kustomization that carries encrypted Secrets. + # The `sops-age` Secret is created OUT OF BAND on the prod cluster + # (bootstrap/sops/install-age-key.sh) — referenced by NAME only, never inlined. + # Add a stanza here whenever `render.py --emit-waves` introduces a new deepest wave. + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: platform + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: apps-wave-0 + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: apps-wave-1 + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: apps-wave-2 + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: oauth2-proxy + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: headlamp + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: crafty + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: velero + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age + # --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) --- + # + # *** BOOTSTRAP BLOCKER — this URL CANNOT work on a cold prod cluster. READ BEFORE BRING-UP. *** + # + # Two independent reasons, both consequences of the split-horizon design: + # 1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's + # GitOps source lives inside the very cluster slated for decommission. Gitea must move to + # prod (or off-cluster) BEFORE dev is retired, or prod loses its source of truth. + # 2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml + # answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP}. On prod that is + # 192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve + # gitea.home.keithjackson.dev to itself and never reach a repository. + # + # Bring-up therefore needs a TEMPORARY source (the bootstrap host-Gitea, or an off-cluster + # mirror), with a cutover to the in-cluster URL once Gitea actually runs on prod — exactly the + # sequence docs/05 describes for dev's own Phase-6 cutover. Tracked in PROD-PROVISIONING.md §7. + # + # The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev. + # CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH + # CLUSTERS. There is no promotion step and no soak window — dev and prod move together. + - patch: |- + apiVersion: source.toolkit.fluxcd.io/v1 + kind: GitRepository + metadata: + name: flux-system + namespace: flux-system + spec: + url: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git + ref: + branch: main - target: kind: ConfigMap name: cluster-vars diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index a337754..55202bf 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -317,13 +317,26 @@ no extra vars is byte-for-byte the dev behaviour it always had (verified by a be *load-bearing*, not cosmetic: `clusters/base/platform.yaml` now carries `dependsOn: [flux-system]`, so without a root object on prod, `platform` would block forever on a dependency that does not exist and nothing downstream would reconcile. - - [ ] **SOPS decryption** — `clusters/dev/kustomization.yaml` patches every Kustomization with - `decryption: {provider: sops, secretRef: sops-age}`. Prod has no equivalent, so every - SOPS-encrypted Secret would fail to decrypt. Needs the prod age key decision first - (same key as dev, or a prod-specific key — a real security choice, not a copy). - - [ ] **GitRepository source URL** — dev patches in the real Gitea URL. Prod needs its own, - plus a decision on whether prod tracks the same branch as dev or a separate one. - Tracking the same branch means a dev-facing merge reconciles prod. + - [x] **SOPS decryption** — DONE. Operator decision 2026-09-01: **prod uses the SAME age key + as dev**. Eight `decryption: {provider: sops, secretRef: sops-age}` patches mirror dev + exactly. The `sops-age` Secret is still created OUT OF BAND on the prod cluster + (`bootstrap/sops/install-age-key.sh`) — the overlay references it by name only. + - [~] **GitRepository source** — BRANCH settled, URL still blocked. Operator decision + 2026-09-01: **prod tracks `main`, the same branch as dev**, because dev is temporary and + will be decommissioned once prod is stable. Recorded consequence: once prod is live, + **every merge to `main` reconciles BOTH clusters** — no promotion step, no soak window. + The URL is patched in but **cannot work on a cold prod cluster**, for two independent + reasons found 2026-09-01: + 1. **Self-hosting circularity** — Gitea runs on the DEV cluster, the one slated for + decommission. Gitea must move to prod (or off-cluster) BEFORE dev is retired, or prod + loses its source of truth. **This is the decommission plan's first task, not its last.** + 2. **Split-horizon sends it to the wrong place** — `coredns-custom` answers the entire + `home.keithjackson.dev` zone with `${GATEWAY_VIP}`, which on prod is `192.168.2.20`: + prod's OWN gateway, where no Gitea listens. Prod's Flux would resolve + `gitea.home.keithjackson.dev` to itself and never reach a repository. + Bring-up needs a TEMPORARY source (bootstrap host-Gitea, or an off-cluster mirror) and a + cutover once Gitea runs on prod — the same sequence `docs/05` describes for dev's own + Phase-6 cutover. - [ ] Confirm prod's backup target: dev's S3 (`versitygw` on the Proxmox host, `192.168.1.9`) is on the dev side of the network. Off-cluster backup is DR-critical (R-STATE-03).