docs(prod): D0/D1 as executed, and CORRECT the P1 gate — it guards the apex, not prod

Two runbook corrections from executing step A of the dev migration.

1. P1 said "only NOW unsuspend prod", gated on D8 (dev's apex vacated). That
   silently assumed prod would be serving the bare apex. IT IS NOT. Prod runs
   c8c1d29, where BASE_DOMAIN=prod.home.keithjackson.dev and
   EXTERNALDNS_TXT_OWNER_ID=prod — verified live in prod's applied cluster-vars.
   Prod's names are a DISJOINT SUBTREE under a DISTINCT registry owner, so
   neither §10a incident mode applies: it cannot claim dev's `default`-owned
   records, and *.prod.home.keithjackson.dev is a DIFFERENT name set from dev's,
   so it has its own Let's Encrypt duplicate-cert bucket and races no
   _acme-challenge with dev. Prod can therefore be brought up IN FULL while dev
   still holds the apex — and should be, because it proves the whole stack
   before the flip, turning the cutover into a DNS/TLS change on a known-good
   cluster rather than a bring-up and a cutover simultaneously. D8 gates ONLY
   the later move of prod's BASE_DOMAIN to the apex.

2. D1 was executed via the REPOINT option, not suspend — repointing removes the
   dependency that causes the strand instead of merely deferring it. Dev now
   fetches from http://192.168.2.132:3000/... branch `dev-live`: IP-addressed,
   plain HTTP, dependent on NEITHER DNS NOR TLS, which is the property a DNS/TLS
   migration needs. A merge to `main` is now inert for dev.

   THE TRAP, recorded because it is not obvious: a bare copy of main@4809199 on
   the bootstrap Gitea would NOT have held. The root renders ./clusters/dev, and
   clusters/dev/kustomization.yaml CONTAINS the patch that sets
   GitRepository.spec.url — so Flux would have fetched from the bootstrap Gitea
   and then applied manifests that set the url straight back, reverting the
   repoint and re-arming the strand. The branch must declare its own source.
   dev-live (a4a3c2e) carries exactly one content change on 4809199, proven by
   `kubectl kustomize clusters/dev` differing in EXACTLY TWO of 357 lines.

   NEW DEPENDENCY, and it is the weak link: 192.168.2.132 is a DHCP lease on the
   workstation's WIFI (wlp0s20f3, NetworkManager ipv4.method=auto — confirmed by
   inspection, not assumed). A lease change or the workstation going offline
   stalls dev's FETCHING; the cluster keeps serving and the fix is a kubectl
   patch. Hence: a static reservation is worth having, and the window should be
   kept short — the exit is D4. Pointing Flux at gitea.gitea.svc.cluster.local
   would drop the DNS, TLS and workstation dependencies at once and is the better
   long-term shape, but namespace `gitea` has default-deny plus a Cilium policy
   admitting only the Gateway, so it needs a new authored policy — deliberately,
   not mid-migration.

Post-cutover verification: 13/13 Kustomizations Ready on dev-live@a4a3c2e,
cluster-vars byte-identical to the D0 snapshot (BASE_DOMAIN still the apex),
18/18 Longhorn volumes attached+healthy, all 16 HTTPRoute hostnames unchanged,
and the Cloudflare record set re-queried and diffed byte-for-byte against D0 —
identical, 57 records, 36 TXT all owner=default.

Refs R-ENV-02, R-GITOPS-01.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-04 23:39:02 -05:00
1 parent 13081a863f
commit 6b88670129
1 file changed
+56 -2
+56 -2
View File
@@ -1563,10 +1563,64 @@ been done.
remains. **This is the gate on the whole P sequence.** A pfSense/PiHole override is not
evidence about public DNS.
### D0/D1 AS EXECUTED (2026-09-04) — and the self-reference that would have undone it
D1 was taken via the **repoint** option, not the suspend option, because it is strictly stronger:
suspending only defers the strand, while repointing removes the dependency that causes it. Dev's
`GitRepository` now reads **`http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git`,
branch `dev-live`** — IP-addressed, plain HTTP, so it depends on **neither DNS nor TLS**, which is
exactly the property needed during a DNS/TLS migration. **A merge to `main` is now inert for dev.**
**THE TRAP, and it is not obvious: a bare copy of `main@4809199` on the bootstrap Gitea would NOT
have held.** The root Kustomization renders `./clusters/dev`, and `clusters/dev/kustomization.yaml`
*contains the patch that sets `GitRepository.spec.url`*. So Flux would have fetched from the
bootstrap Gitea, applied the manifests it found there, and those manifests would have set the url
straight back to `gitea.home.keithjackson.dev` — reverting the repoint on the next reconcile and
re-arming the strand. **The branch must declare its own source.** `dev-live` (`a4a3c2e`) therefore
carries exactly one content change on top of `4809199`: `clusters/dev/kustomization.yaml`'s url +
`ref.branch`. Proven by `kubectl kustomize clusters/dev` at both revisions differing in **exactly
two lines** out of 357. The same property makes the revert self-sustaining in reverse, since
`main@4809199` carries the `gitea.home…` url. Revert command: see `REVERT.txt` in the run scratchpad.
Verified after the cutover: **13/13 Kustomizations Ready** on `dev-live@a4a3c2e`, `cluster-vars`
byte-identical to the D0 snapshot (27/27 keys, `BASE_DOMAIN` still `home.keithjackson.dev`),
**18/18 Longhorn volumes attached+healthy**, all 16 HTTPRoute hostnames still on the apex,
`wildcard-home` still Ready, and the Cloudflare record set re-queried and diffed **byte-for-byte
against the D0 snapshot — identical** (57 records, 36 TXT all `owner=default`).
> **NEW DEPENDENCY, AND IT IS THE WEAK LINK.** `192.168.2.132` is a **DHCP lease on the
> workstation's wifi** (`wlp0s20f3`, NetworkManager `ipv4.method=auto` — confirmed, not assumed). A
> lease change, a wifi drop, or the workstation going offline **stalls dev's fetching**. The cluster
> keeps running and serving; only reconciliation pauses, and the fix is a `kubectl patch` of the url.
> Two consequences: (a) a **static reservation** for this host is worth having before any repeat of
> this procedure, and (b) **keep the window short** — the exit is D4, which returns dev to its own
> in-cluster Gitea at `gitea.dev.home.keithjackson.dev`.
>
> Pointing Flux at dev's Gitea **Service** instead (`gitea.gitea.svc.cluster.local:3000`) would drop
> the DNS, TLS *and* workstation dependencies all at once, and is the better long-term shape. It was
> **not** done here: namespace `gitea` carries a `default-deny` NetworkPolicy plus a
> `gitea-allow-gateway-ingress` CiliumNetworkPolicy admitting **only** the Gateway, so
> source-controller cannot reach the Service without a new authored policy. Worth doing deliberately,
> not mid-migration.
### P — prod takes the apex
- [ ] **P1. Only now unsuspend prod**, in dependency order: `flux resume kustomization platform`,
then the 10 downstream tiers. Confirm D8 first, in Cloudflare, not from a local resolver.
- [ ] **P1. Unsuspend prod**, in dependency order: `flux resume kustomization platform`, then the
10 downstream tiers.
**CORRECTION (2026-09-04) — the D8 gate is on prod taking the APEX, not on prod coming up.**
This step originally read "only now unsuspend prod", which silently assumed prod would be
serving `*.home.keithjackson.dev`. It is not: prod runs revision `c8c1d29`, where
`BASE_DOMAIN=prod.home.keithjackson.dev` and `EXTERNALDNS_TXT_OWNER_ID=prod`. Prod's names are
therefore a **disjoint subtree** under a **distinct registry owner**, so neither incident mode
of §10a applies: it cannot claim dev's `default`-owned records, and its wildcard
`*.prod.home.keithjackson.dev` is a **different name set** from dev's, so it has its own
Let's Encrypt duplicate-certificate bucket and races no `_acme-challenge` with dev.
**Prod may therefore be brought up in full while dev still holds the apex** — and should be,
because it proves the entire platform stack end-to-end *before* the apex flip, turning the
cutover into a DNS/TLS change on a known-good cluster instead of a bring-up and a cutover at
once. What D8 gates is **only** the later move of prod's `BASE_DOMAIN` to the bare apex.
Verify the two guard values live in prod's applied `cluster-vars` before unsuspending; if
`EXTERNALDNS_TXT_OWNER_ID` is not `prod`, STOP — that was the root cause of the incident.
- [ ] **P2. Prod's external-dns publishes under owner `prod`.** Check the logs for a matching hosted
zone (F-DNS1 fail-closed mode: no zone selected ⇒ nothing created — which is safe, but means
the filter was narrowed against instruction). Confirm the new TXT registry entries read owner