diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index f91ab5d..f043169 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -1563,10 +1563,64 @@ been done. remains. **This is the gate on the whole P sequence.** A pfSense/PiHole override is not evidence about public DNS. +### D0/D1 AS EXECUTED (2026-09-04) — and the self-reference that would have undone it + +D1 was taken via the **repoint** option, not the suspend option, because it is strictly stronger: +suspending only defers the strand, while repointing removes the dependency that causes it. Dev's +`GitRepository` now reads **`http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git`, +branch `dev-live`** — IP-addressed, plain HTTP, so it depends on **neither DNS nor TLS**, which is +exactly the property needed during a DNS/TLS migration. **A merge to `main` is now inert for dev.** + +**THE TRAP, and it is not obvious: a bare copy of `main@4809199` on the bootstrap Gitea would NOT +have held.** The root Kustomization renders `./clusters/dev`, and `clusters/dev/kustomization.yaml` +*contains the patch that sets `GitRepository.spec.url`*. So Flux would have fetched from the +bootstrap Gitea, applied the manifests it found there, and those manifests would have set the url +straight back to `gitea.home.keithjackson.dev` — reverting the repoint on the next reconcile and +re-arming the strand. **The branch must declare its own source.** `dev-live` (`a4a3c2e`) therefore +carries exactly one content change on top of `4809199`: `clusters/dev/kustomization.yaml`'s url + +`ref.branch`. Proven by `kubectl kustomize clusters/dev` at both revisions differing in **exactly +two lines** out of 357. The same property makes the revert self-sustaining in reverse, since +`main@4809199` carries the `gitea.home…` url. Revert command: see `REVERT.txt` in the run scratchpad. + +Verified after the cutover: **13/13 Kustomizations Ready** on `dev-live@a4a3c2e`, `cluster-vars` +byte-identical to the D0 snapshot (27/27 keys, `BASE_DOMAIN` still `home.keithjackson.dev`), +**18/18 Longhorn volumes attached+healthy**, all 16 HTTPRoute hostnames still on the apex, +`wildcard-home` still Ready, and the Cloudflare record set re-queried and diffed **byte-for-byte +against the D0 snapshot — identical** (57 records, 36 TXT all `owner=default`). + +> **NEW DEPENDENCY, AND IT IS THE WEAK LINK.** `192.168.2.132` is a **DHCP lease on the +> workstation's wifi** (`wlp0s20f3`, NetworkManager `ipv4.method=auto` — confirmed, not assumed). A +> lease change, a wifi drop, or the workstation going offline **stalls dev's fetching**. The cluster +> keeps running and serving; only reconciliation pauses, and the fix is a `kubectl patch` of the url. +> Two consequences: (a) a **static reservation** for this host is worth having before any repeat of +> this procedure, and (b) **keep the window short** — the exit is D4, which returns dev to its own +> in-cluster Gitea at `gitea.dev.home.keithjackson.dev`. +> +> Pointing Flux at dev's Gitea **Service** instead (`gitea.gitea.svc.cluster.local:3000`) would drop +> the DNS, TLS *and* workstation dependencies all at once, and is the better long-term shape. It was +> **not** done here: namespace `gitea` carries a `default-deny` NetworkPolicy plus a +> `gitea-allow-gateway-ingress` CiliumNetworkPolicy admitting **only** the Gateway, so +> source-controller cannot reach the Service without a new authored policy. Worth doing deliberately, +> not mid-migration. + ### P — prod takes the apex -- [ ] **P1. Only now unsuspend prod**, in dependency order: `flux resume kustomization platform`, - then the 10 downstream tiers. Confirm D8 first, in Cloudflare, not from a local resolver. +- [ ] **P1. Unsuspend prod**, in dependency order: `flux resume kustomization platform`, then the + 10 downstream tiers. + **CORRECTION (2026-09-04) — the D8 gate is on prod taking the APEX, not on prod coming up.** + This step originally read "only now unsuspend prod", which silently assumed prod would be + serving `*.home.keithjackson.dev`. It is not: prod runs revision `c8c1d29`, where + `BASE_DOMAIN=prod.home.keithjackson.dev` and `EXTERNALDNS_TXT_OWNER_ID=prod`. Prod's names are + therefore a **disjoint subtree** under a **distinct registry owner**, so neither incident mode + of §10a applies: it cannot claim dev's `default`-owned records, and its wildcard + `*.prod.home.keithjackson.dev` is a **different name set** from dev's, so it has its own + Let's Encrypt duplicate-certificate bucket and races no `_acme-challenge` with dev. + **Prod may therefore be brought up in full while dev still holds the apex** — and should be, + because it proves the entire platform stack end-to-end *before* the apex flip, turning the + cutover into a DNS/TLS change on a known-good cluster instead of a bring-up and a cutover at + once. What D8 gates is **only** the later move of prod's `BASE_DOMAIN` to the bare apex. + Verify the two guard values live in prod's applied `cluster-vars` before unsuspending; if + `EXTERNALDNS_TXT_OWNER_ID` is not `prod`, STOP — that was the root cause of the incident. - [ ] **P2. Prod's external-dns publishes under owner `prod`.** Check the logs for a matching hosted zone (F-DNS1 fail-closed mode: no zone selected ⇒ nothing created — which is safe, but means the filter was narrowed against instruction). Confirm the new TXT registry entries read owner