home-network-k3s-impl/infra/PROD-STANDUP.md

Prod cluster standup — the complete command sequence

Every command below is copy-pasteable and runs from the repo root, through the pinned dev container. Nothing here needs a command invented on the spot; that is the point. If a step is not in this file it is not part of standup.

State as of 2026-09-06: phases A, B and C are DONE. The cluster is live — k3s v1.36.4 HA (3 etcd members), kube-vip, Cilium 1.20.1, Longhorn — with 12/13 Flux tiers Ready (velero deliberately suspended), 112 pods Running, an issued Let's Encrypt wildcard, and all three nodes addressing their full 32 GB (node 2's DIMM upgrade completed 2026-09-06 and is confirmed at ~32638256Ki, matching the other two to within 4 KiB — the §0b memattr bug is not biting on the backports kernel).

Storage survived two unplanned node events (an ungraceful power-down of prod-3, then prod-2's memory upgrade) and converged to 17/17 volumes healthy, replicas symmetric across all three nodes — though only after a leftover cordon was cleared; read §Node maintenance before doing node work, that cordon silently prevents Longhorn from rebuilding.

What remains is in §Blockers, and every item there needs an operator decision: no backup coverage (R2 credentials), prod still on prod.home. rather than the apex, and — found 2026-09-05 — prod's UniFi VM has no disk because Kyverno blocks CDI's generated importer pod.

The one guard that matters. Dev is LIVE and shares the keithjackson.dev Cloudflare zone with prod. On 2026-09-04 prod deleted every one of dev's public DNS records. bootstrap/standup.sh therefore requires --expect-server and compares it against the live API server before running anything. There is no override flag. Do not add one.

Constants

kube-API (kube-vip VIP)	https://192.168.2.9:6443
Gateway / ingress VIP	192.168.2.20
Nodes	k3s-prod-1/2/3 — 192.168.2.10 / .11 / .12
Break-glass kubeconfig	infra/ansible/.cache/prod.kubeconfig (cert-based; survives OIDC/DNS failure)
GitOps source (interim)	http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git branch main
BASE_DOMAIN	prod.home.keithjackson.dev — not the apex; see §Blockers

Container paths: the repo is mounted at /work, so kubeconfig paths passed to the script are repo-relative (infra/ansible/.cache/prod.kubeconfig).

---

Phase A — bare metal to a joined cluster (Ansible)

Prerequisites that are not commands: firmware/BIOS (§1), disk layout (§2), OS install (§3), network (§4), and SSH + the vaulted ansible_become_password in inventory/group_vars/k3s_prod/vault.yml (§5). Ansible cannot escalate on prod without that vault file. See infra/PROD-PROVISIONING.md.

Install the backports kernel first. All three prod nodes need ≥6.13 because HP's R21 firmware publishes a corrupt EFI Memory Attributes Table and Linux 6.12's efi_memattr_init() does not validate it, reserving everything above the table (fixed upstream in 6.13 by 8fbe4c49c0cc, absent from linux-6.12.y). Prod runs 7.1.8+deb13-amd64 from trixie-backports.

timeout 3600 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
  ansible-playbook /work/infra/ansible/playbooks/node-baseline.yml \
  -i /work/infra/ansible/inventory/hosts.ini -e k3s_hosts=k3s_prod --tags kernel

Then the full chain — baseline, k3s HA, Cilium, verify, Flux hand-off. All three -e flags are required: a play's hosts: is templated before any host exists, so group_vars cannot retarget it.

timeout 3600 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
  ansible-playbook /work/infra/ansible/playbooks/site.yml \
  -i /work/infra/ansible/inventory/hosts.ini \
  -e k3s_hosts=k3s_prod -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary

kube-vip is a separate play (not imported by site.yml):

timeout 1800 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
  ansible-playbook /work/infra/ansible/playbooks/kube-vip.yml \
  -i /work/infra/ansible/inventory/hosts.ini -e k3s_hosts=k3s_prod

Hardware health (read-only; node_health asserts the memattr fix took and that all RAM is addressed).

The flag is node_health_hosts, NOT k3s_hosts. Diagnostic plays deliberately keep their own target var so they stay independently pointable while a bring-up run is scoped elsewhere (node-health.yml → node_health_hosts, memattr-survey.yml → memattr_survey_hosts, vpn.egress.gateway.yml → vpn_egress_gateway_hosts). The consequence is a silent one: passing k3s_hosts=k3s_prod here does not error, it falls through to the k3s_servers default and runs against DEV's three nodes. Verified — this exact mistake was made while writing this file and caught only by --list-hosts. Always dry-run the target first:

timeout 200 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
  ansible-playbook /work/infra/ansible/playbooks/node-health.yml \
  -i /work/infra/ansible/inventory/hosts.ini -e node_health_hosts=k3s_prod --list-hosts

timeout 900 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
  ansible-playbook /work/infra/ansible/playbooks/node-health.yml \
  -i /work/infra/ansible/inventory/hosts.ini -e node_health_hosts=k3s_prod

Phase B — Flux bootstrap

flux-bootstrap.yml (last play of site.yml) shells to bootstrap/flux/bootstrap.sh, the single documented entrypoint. That script now runs the CRD pre-apply as step [2/4], so a cold bootstrap no longer strands platform. The age private key Secret must exist first — bootstrap verifies it and refuses to continue rather than silently skipping decryption (bootstrap/sops/age-key-secret.md).

Phase C — bring the reconciliation graph up

One script, six steps, each independently runnable and idempotent. Run them in this order.

K=infra/ansible/.cache/prod.kubeconfig
S=https://192.168.2.9:6443

C1 — preflight (read-only, changes nothing)

Asserts the guards that prevent a repeat of the DNS incident and reports every tier's state. Fails closed if EXTERNALDNS_TXT_OWNER_ID is empty, still the base sentinel, or default (dev's owner).

timeout 300 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 preflight

C2 — pre-apply chart CRDs, and break the CDI cold-start deadlock (idempotent)

timeout 600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 crds

Two cold-start problems, same root cause: Flux server-side dry-runs every object in a Kustomization before applying any of them, so one unknown kind aborts the entire tier.

1. Kyverno — its ~5.6 MiB CRD bundle exceeds kubectl's 256 KiB annotation limit, so the chart sets crds.install: false and nothing else installs them. bootstrap/flux/preapply-crds.sh applies them server-side, deriving the version from the HelmRelease Flux reconciles (no second pin).

2. CDI — a true deadlock. platform-config applies the CDI CR and platform/kubevirt/cdi-storageprofiles.yaml in the same pass. But StorageProfile's CRD does not exist until cdi-operator has reconciled that CR. So the dry-run fails on the unknown kind → the CR is never applied → the CRD is never created → platform-config can never converge, and every tier below it stays blocked. Invisible on an established cluster (dev's CRDs have been there for months) and fatal on a cold one. Observed on prod 2026-09-05: cdi-operator Available for 12h with no CDI CR and no storageprofiles CRD.

   The step breaks it by applying only the CR — the minimum — and leaving everything else to Flux. It carries no ${} tokens, so applying it here cannot diverge from what Flux would render. The CRD appeared in 10s.

   REPAIRED IN THE GRAPH (2026-09-05). cdi-storageprofiles.yaml now lives in its own tier, ./platform-config-cdi, with dependsOn: platform-config and wait: true. Because dependsOn is revision-gated and platform-config runs wait: true, the CDI CR must be applied and healthy — which is precisely the event that creates the StorageProfile CRD — before the profiles are applied. A cold cluster now orders itself. The crds step is retained as a fallback for a cluster already stuck in the broken state, and because it is idempotent.

   The generalisable rule, now recorded in clusters/README.md: a CR whose CRD is created by a controller at runtime is second-order and needs its own pass. cdi-operator.yaml registers exactly one CRD (cdis.cdi.kubevirt.io); storageprofiles, datavolumes, datasources and dataimportcrons appear only after cdi-operator reconciles the CDI CR. An audit of platform-config found StorageProfile is the only second-order kind there.

   apps-vms gained dependsOn: platform-config-cdi — load-bearing, not cosmetic: apps/vms/unifi/20-datavolume.yaml provisions on storageClassName: longhorn-vm, and its comment previously claimed platform-config guaranteed the profiles, which the move would have silently falsified.

Merge precondition and the one-time transient

platform-config has prune: true, so on the first reconcile after merge it deletes both StorageProfiles, CDI recreates them within seconds with an empty spec, and platform-config-cdi re-applies the overrides ~30–60s later (kustomize-controller's dependency requeue).

Measured on dev, so the exposure is known rather than assumed — comparing our override against what CDI infers unaided for the same Longhorn provisioner:

profile	spec (ours)	CDI-derived status
longhorn-vm (we override)	RWX / Block, snapshot	ReadWriteMany / Block, snapshot
longhorn-static (no override)	unset	ReadWriteOnce / Block, snapshot

So cloneStrategy is not at risk — CDI infers snapshot for Longhorn anyway. accessModes is: unaided, CDI produces RWO. A DataVolume provisioned on longhorn-vm inside the window would be stamped RWO/Block and be permanently non-live-migratable. Flux can no longer cause that (apps-vms now gates on the new tier); do not start a VM disk import, clone or DataVolume by hand during the window.

• Before merging, confirm flux get kustomization platform-config is Ready on the target. If it is not, the profiles are pruned and sit at CDI defaults until it recovers — and prod was in exactly that state hours earlier. The fallback is the two-step the Cilium move used: merge with platform-config.prune: false, confirm the ownership label transferred, restore prune: true in a second commit.
• After: both profiles carry spec.cloneStrategy: snapshot, longhorn-vm carries claimPropertySets RWX/Block, and both are labelled kustomize.toolkit.fluxcd.io/name=platform-config-cdi.

C3 — unwedge stuck Helm releases

A release left in uninstalling / pending-* cannot be advanced by helm-controller: it reports StateError forever, so platform (wait: true) never goes Ready and all 10 tiers below it stall. Dry run by default — it prints what it would do and changes nothing:

timeout 600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 unwedge

To act (this deletes the release's Kubernetes objects and lets Flux install cold). Pre-applied CRDs live outside the Helm release and survive, which is exactly why C2 comes first:

timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 \
  --confirm-destructive unwedge

C4 — external-dns safe upgrade

The most dangerous step in the repo, and the ordering is the entire mitigation. A Helm upgrade resets replicas to the chart default (1) and applies the new args in ONE operation — there is no window in which to check the result first. If --txt-owner-id were missing, the pod starts as owner default, claims dev's TXT registry entries, and under policy: sync deletes the records behind them.

So this step proves the outcome before causing it: it renders the chart with the live HelmRelease values and asserts --txt-owner-id=<owner> appears, aborting if not. Rendering is deterministic and touches nothing, which converts a race into a precondition. It then snapshots the Cloudflare zone, upgrades, re-verifies the live args (scaling to 0 and failing if they are wrong), and diffs the zone.

timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 externaldns

Expected diff: additions only, all under prod.home.keithjackson.dev, all TXT entries owner prod. Any change to a name outside that subtree, or to a TXT entry owned by another cluster, is a regression — stop and investigate.

C5 — resume the tiers, in dependency order

platform → platform-config → {oauth2-proxy, headlamp, homepage, crafty, apps-wave-0, apps-vms} → apps-wave-1 → apps-wave-2. Waits for each to go Ready and stops at the first that does not — a partial, understood bring-up beats a forced green. velero is skipped while S3_BACKUP_ENDPOINT is a placeholder (see §Blockers). Long-running; expect it to background.

timeout 3600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 resume

C6 — verify

timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 verify

Everything at once

Runs C1→C6 with unwedge in dry-run mode (no --confirm-destructive), so it stops short of the destructive step by design:

timeout 3600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 all

---

VERIFIED WORKING — 2026-09-05, standup complete

Kustomizations	12/13 Ready (velero deliberately suspended — see §Blockers)
Pods	113 total, 0 not Running/Completed
Wildcard cert	wildcard-home Ready, Let's Encrypt prod, *.prod.home.keithjackson.dev
Gateway VIP	192.168.2.20 · crafty L4 192.168.2.22 · unifi L4 192.168.2.21
Published DNS	36 records under prod.home.keithjackson.dev, every TXT registry entry owner=prod
Dev's records	57, byte-identical to the pre-standup baseline — nothing added, changed or removed

End-to-end HTTPS through the workstation's own resolver — no --resolve override, no -k (after installing the dnsmasq profile below). Every request landed on 192.168.2.20:

prod.home.keithjackson.dev  200   headlamp.  200   keycloak.  302 (login)   nextcloud. 302
gitea.       200            semaphore. 200        grafana.   302 (SSO)      pihole.    302

The served certificate is prod's own, confirmed by SNI — not dev's wildcard, which is what the resolver bug previously made it look like:

subject = CN=*.prod.home.keithjackson.dev
issuer  = C=US, O=Let's Encrypt, CN=YR1          (LE production)
SAN     = *.prod.home.keithjackson.dev, prod.home.keithjackson.dev
valid   = 2026-09-05 .. 2026-12-04

Dev, checked in the same pass, still serves CN=*.home.keithjackson.dev — unchanged.

Ignore curl: (91) No OCSP response received if you pass --cert-status: Let's Encrypt retired OCSP responders in 2025, so there is nothing to staple. It is not a certificate fault, and the plain requests above all validated normally.

Workstation DNS — install the prod dnsmasq profile

CORRECTION (operator, 2026-09-05): this is the workstation's own dnsmasq, NOT pfSense. Earlier notes in this repo called it a "pfSense override"; that was wrong and the topology below is why.

/etc/NetworkManager/dnsmasq.d/10-home.keithjackson.dev.conf (the dev profile) contains:

address=/.home.keithjackson.dev/192.168.2.50

dnsmasq's address= matches the named domain and every subdomain, so that one line also captures the whole prod.home.keithjackson.dev tree and answers it with dev's address, while Cloudflare correctly returns prod's 192.168.2.20.

Fix — one command, already authored in this repo:

sudo infra/workstation/dnsmasq/install.sh

It installs 20-prod.home.keithjackson.dev.conf and reloads the live resolver. dnsmasq resolves address= by longest match, so the prod rules win without editing or reordering the dev profile. Verified against a throwaway dnsmasq loaded with both profiles:

query	answer
prod.home.keithjackson.dev, gitea.prod…, keycloak.prod…	192.168.2.20 (Gateway)
unifi.prod.home.keithjackson.dev	192.168.2.21 (L4)
crafty-mc.prod.home.keithjackson.dev	192.168.2.22 (L4)
gitea.home.keithjackson.dev, home.keithjackson.dev	192.168.2.50 — dev, unchanged

Install to /etc/NetworkManager/dnsmasq.d/, never /etc/dnsmasq.d/. The standalone dnsmasq.service is inactive; the live resolver is the one NetworkManager spawns (dns=dnsmasq) with --conf-dir=/etc/NetworkManager/dnsmasq.d. A file in the other directory changes nothing and looks like it should. install.sh pins the destination for exactly this reason.

Until it is installed, verify prod only with an explicit resolve — and note why this trap is convincing: dev serves a *.home.keithjackson.dev wildcard cert, which matches prod.home.keithjackson.dev (one label), so the prod apex returns 200 over valid TLS while keycloak.prod.home.keithjackson.dev (two labels) fails with a certificate-name error. It reads like a broken prod certificate. It is the wrong server answering.

curl -sS --resolve gitea.prod.home.keithjackson.dev:443:192.168.2.20 \
  https://gitea.prod.home.keithjackson.dev/

NODE MAINTENANCE — draining, rebooting, and the cordon that stops self-healing

Two unplanned node events on 2026-09-05/06 (an ungraceful power-down of prod-3, then a memory upgrade on prod-2) produced one finding worth more than either incident.

A failed kubectl drain still cordons — and on this cluster a cordon stops storage self-healing

kubectl drain k3s-prod-2 fails on this cluster with the usual two objections (emptyDir pods, DaemonSet pods) — but it cordons the node first, and the cordon persists after the failure. Longhorn's disable-scheduling-on-cordoned-node is true, so a cordoned node receives no replica scheduling. Observed consequence: after prod-2's reboot, all 17 volumes sat attached/degraded with prod-2 holding 2 of 18 replicas and ZERO rebuilds in flight. Nothing was driving recovery, every pod was Running, and no alert fired. kubectl uncordon started the rebuilds immediately; they converged 17/17 healthy in ~6 minutes.

So: after any drain — successful or not — kubectl uncordon <node> is not tidying up, it is what re-enables storage redundancy. Check kubectl get nodes -o custom-columns=N:.metadata.name,\ U:.spec.unschedulable before believing a green pod list.

drain cannot complete on this cluster, and the two usual flags do not fix it

--ignore-daemonsets --delete-emptydir-data clears the two reported objections and then hangs indefinitely: all three instance-manager-* PDBs report ALLOWED DISRUPTIONS = 0 and node-drain-policy is block-if-contains-last-replica. Longhorn is refusing to let an instance-manager be evicted while it runs volume engines. That is data protection, not a bug.

Requesting replica eviction does not rescue it either: with 3 nodes and 3 replicas there is no fourth node to move a replica to, so eviction can never complete. This is structural, not a setting.

Options, in order of preference:

1. Just reboot it (cordon, reboot, uncordon). Proven twice on this cluster: Longhorn resyncs on return, and replica-replenishment-wait-interval is 600s, so a reboot completed inside ~10 minutes does not even trigger full rebuilds. This is the normal path for firmware, memory or kernel work.
2. If the node must be genuinely empty, first scale to 0 the workloads holding engines on it — that detaches them, the PDBs go to 1 allowed disruption, and the drain completes. Find them with: kubectl -n longhorn-system get engines.longhorn.io -o json filtered on .spec.nodeID.
3. Changing node-drain-policy to always-allow is not advised at 3 nodes/3 replicas — it trades the protection for exactly the data risk the default exists to prevent.

Check the kube-vip leader before rebooting a server

kube-vip holds the API VIP (192.168.2.9) via a leader lease, and it is a DaemonSet — so --ignore-daemonsets leaves it running and a drain does not move the VIP. A reboot does: the lease fails over and kubectl blips for a few seconds. Harmless, but do not mistake it for a fault. Identify the holder: kubectl -n kube-system logs -l app.kubernetes.io/name=kube-vip-ds | grep -i leader.

Post-maintenance checklist

• kubectl uncordon <node> — and confirm .spec.unschedulable is clear on all three.
• Longhorn converged: kubectl -n longhorn-system get volumes.longhorn.io shows 0 degraded. Rebuilds are serial, so allow minutes, and watch rather than assume.
• Replica symmetry: each node runs the same count (get replicas.longhorn.io grouped by .spec.nodeID / .status.currentState).
• All three nodes report full memory — prod's HP firmware publishes a corrupt EFI memattr table and a wrong kernel silently strands RAM (§0b). k3s-prod-* should each show ~32638256Ki.
• Pods: none Running-but-not-Ready.

KNOWN, NOT FIXED — pod distribution does not rebalance. After prod-2's cordon+reboot the spread was 52 / 10 / 50. Kubernetes does not move running pods back, and for the many SINGLE-REPLICA apps here a spread constraint would change nothing (one replica lives on one node either way). It self-corrects as pods restart. It is recorded because it concentrates failure: at the time of writing all three CNPG databases run on prod-1, so losing that node takes Keycloak, Nextcloud and Semaphore together. The real fix is database HA (instances: 2), which is a resource decision the operator owns — see §Blockers.

Blockers

1. Backups: endpoint DECIDED, but prod cannot reach it — prod has no backup coverage

Operator ruling 2026-09-05: "the S3 backup location is the same as Dev." Prod's S3_BACKUP_ENDPOINT is now http://192.168.1.9:9000, and the TBD-PROD-* placeholder is retired.

It is not reachable from prod. Measured from a throwaway pod inside the prod cluster — not from the workstation, which proves nothing about pod egress:

curl http://192.168.1.9:9000/minio/health/live   →  connection timed out after 15s

This is expected topology, not a misconfiguration (operator, 2026-09-05): the k3s prod cluster is not behind pfSense. It sits on the SOHO router network 192.168.2.0/24, which is pfSense's WAN side; pfSense contains 192.168.1.0/24, the dev LAN where MinIO lives. Prod reaching 192.168.1.9 therefore means crossing into pfSense from its WAN — which requires a deliberate pfSense rule or port-forward that does not exist. Operator decision required, and worth weighing: a prod backup that lands inside the dev LAN is arguably not off-cluster DR at all.

Re-test with standup.sh --s3-endpoint http://192.168.1.9:9000 s3check — it fails closed and says so.

Do not confuse this with the SOHO router's inbound NAT. That router forwards inbound 443 to the pfSense WAN address 192.168.2.102 to serve the existing, non-k3s production environment, which must stay live on 443 until the k3s cutover is deliberately made. Nothing in this runbook touches that NAT, and prod's Gateway on 192.168.2.20:443 does not contend with it.

Until that routing exists, velero stays suspended and apps-wave-0 still applies GREEN — the token is defined, so strict substitution succeeds — while every write to that endpoint fails at runtime. A green apps-wave-0 does not mean backups work. Treat prod as having no backup coverage.

Sharing one MinIO is only safe because of two new knobs

Nothing about the backup paths was per-environment before 2026-09-05. Enabling prod against dev's MinIO without these would have been the external-dns incident again, in a worse place:

	before	after
Velero	bucket: velero, no prefix, both clusters run schedules named daily	prefix: ${VELERO_BACKUP_PREFIX} — dev "", prod prod
CNPG	destinationPath: s3://cnpg, serverName defaults to the Cluster name	${CNPG_BACKUP_PATH} — dev s3://cnpg, prod s3://cnpg/prod

• Velero syncs backups it finds in its BSL into itself as Backup CRs and garbage-collects them when their TTL expires. Prod would have adopted dev's backups and deleted them on schedule.
• CNPG is the worse one. Both environments have keycloak-db, nextcloud-db and semaphore-db, so prod's Postgres would archive WAL into the exact prefix dev's does. Two servers interleaving WAL into one Barman stream corrupts the chain and destroys PITR — and dev's restore is proven (R-STATE-03).

The base values are dev's current layout on purpose and must stay that way. Dev's existing backups live at the bucket root and at s3://cnpg; giving dev a prefix would orphan the very backups its proven restore depends on — the same reasoning that keeps dev's EXTERNALDNS_TXT_OWNER_ID at default. Prod has no history to preserve, so it takes the nested paths.

Delivery note. Prod reconciles the bootstrap Gitea's main at c8c1d29. This change sits on a later branch that also carries the DNS inversion, which must not reach prod (it would move prod to the apex and collide with dev). Delivering it needs a c8c1d29-based branch carrying only these commits. No urgency: velero cannot be enabled until the routing above is fixed.

2. Prod is on prod.home.keithjackson.dev, not the apex

The operator's target layout is dev under *.dev.home.keithjackson.dev and prod on the apex. Prod cannot take the apex until dev vacates it, because both clusters would request a letsencrypt-prod wildcard for the same name set and race each other's _acme-challenge TXT records, so neither reliably solves DNS-01.

This does not block phase C. Prod on its own subtree with txtOwnerId: prod collides with nothing, and standing it up first is strictly better: it proves the whole stack before the flip, so the cutover becomes a DNS/TLS change on a known-good cluster instead of a bring-up and a cutover at once. The apex move is infra/PROD-PROVISIONING.md §10a, sequence D (dev vacates) then P.

THE APEX FLIP IS ALSO THE LEGACY-PROD CUTOVER — they are the same event. The SOHO router NATs inbound 443 to the pfSense WAN address 192.168.2.102, serving the existing, non-k3s production environment, which the operator requires to stay live until the switch is deliberately made. So §10a sequence P is not merely a DNS/TLS change on the k3s side: taking home.keithjackson.dev means public traffic must also stop resolving to / being forwarded to the legacy stack. Sequence P as written does not cover repointing that NAT, and the two must be planned together or the apex will resolve to k3s while inbound 443 still lands on the legacy box.

Nothing in phase C touches it: prod's Gateway is 192.168.2.20:443, a different address from the NAT target, and prod's wildcard cert is issued by DNS-01, which needs no inbound 443 at all.

---

Not covered here

• Dev's migration off the apex — §10a sequence D. Touches Keycloak's issuer, ~12 OIDC client redirect URIs, and a rolling k3s restart that invalidates every OIDC kubeconfig. Operator-present.
• Prod's GitOps cutover (P5) from the workstation bootstrap Gitea to prod's own in-cluster Gitea. clusters/prod/kustomization.yaml owns that procedure.
• Six L2 conformance tests hardcode the apex and must be parameterized per environment before §9's "suite passes green against the prod cluster" can be claimed.