home-network-k3s-impl/infra/PROD-STANDUP.md

Prod cluster standup — the complete command sequence

Every command below is copy-pasteable and runs from the repo root, through the pinned dev container. Nothing here needs a command invented on the spot; that is the point. If a step is not in this file it is not part of standup.

State as of 2026-09-05: phases A and B are DONE on the current hardware. The cluster is live (k3s v1.36.4 HA, 3 etcd members, kube-vip, Cilium 1.20.1, Longhorn) with platform reconciling and 8/9 HelmReleases Ready, 76/76 pods Running. Outstanding work is phase C, blocked on the two items in §Blockers.

The one guard that matters. Dev is LIVE and shares the keithjackson.dev Cloudflare zone with prod. On 2026-09-04 prod deleted every one of dev's public DNS records. bootstrap/standup.sh therefore requires --expect-server and compares it against the live API server before running anything. There is no override flag. Do not add one.

Constants

kube-API (kube-vip VIP)	https://192.168.2.9:6443
Gateway / ingress VIP	192.168.2.20
Nodes	k3s-prod-1/2/3 — 192.168.2.10 / .11 / .12
Break-glass kubeconfig	infra/ansible/.cache/prod.kubeconfig (cert-based; survives OIDC/DNS failure)
GitOps source (interim)	http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git branch main
BASE_DOMAIN	prod.home.keithjackson.dev — not the apex; see §Blockers

Container paths: the repo is mounted at /work, so kubeconfig paths passed to the script are repo-relative (infra/ansible/.cache/prod.kubeconfig).

---

Phase A — bare metal to a joined cluster (Ansible)

Prerequisites that are not commands: firmware/BIOS (§1), disk layout (§2), OS install (§3), network (§4), and SSH + the vaulted ansible_become_password in inventory/group_vars/k3s_prod/vault.yml (§5). Ansible cannot escalate on prod without that vault file. See infra/PROD-PROVISIONING.md.

Install the backports kernel first. All three prod nodes need ≥6.13 because HP's R21 firmware publishes a corrupt EFI Memory Attributes Table and Linux 6.12's efi_memattr_init() does not validate it, reserving everything above the table (fixed upstream in 6.13 by 8fbe4c49c0cc, absent from linux-6.12.y). Prod runs 7.1.8+deb13-amd64 from trixie-backports.

timeout 3600 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
  ansible-playbook /work/infra/ansible/playbooks/node-baseline.yml \
  -i /work/infra/ansible/inventory/hosts.ini -e k3s_hosts=k3s_prod --tags kernel

Then the full chain — baseline, k3s HA, Cilium, verify, Flux hand-off. All three -e flags are required: a play's hosts: is templated before any host exists, so group_vars cannot retarget it.

timeout 3600 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
  ansible-playbook /work/infra/ansible/playbooks/site.yml \
  -i /work/infra/ansible/inventory/hosts.ini \
  -e k3s_hosts=k3s_prod -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary

kube-vip is a separate play (not imported by site.yml):

timeout 1800 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
  ansible-playbook /work/infra/ansible/playbooks/kube-vip.yml \
  -i /work/infra/ansible/inventory/hosts.ini -e k3s_hosts=k3s_prod

Hardware health (read-only; node_health asserts the memattr fix took and that all RAM is addressed).

The flag is node_health_hosts, NOT k3s_hosts. Diagnostic plays deliberately keep their own target var so they stay independently pointable while a bring-up run is scoped elsewhere (node-health.yml → node_health_hosts, memattr-survey.yml → memattr_survey_hosts, vpn.egress.gateway.yml → vpn_egress_gateway_hosts). The consequence is a silent one: passing k3s_hosts=k3s_prod here does not error, it falls through to the k3s_servers default and runs against DEV's three nodes. Verified — this exact mistake was made while writing this file and caught only by --list-hosts. Always dry-run the target first:

timeout 200 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
  ansible-playbook /work/infra/ansible/playbooks/node-health.yml \
  -i /work/infra/ansible/inventory/hosts.ini -e node_health_hosts=k3s_prod --list-hosts

timeout 900 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
  ansible-playbook /work/infra/ansible/playbooks/node-health.yml \
  -i /work/infra/ansible/inventory/hosts.ini -e node_health_hosts=k3s_prod

Phase B — Flux bootstrap

flux-bootstrap.yml (last play of site.yml) shells to bootstrap/flux/bootstrap.sh, the single documented entrypoint. That script now runs the CRD pre-apply as step [2/4], so a cold bootstrap no longer strands platform. The age private key Secret must exist first — bootstrap verifies it and refuses to continue rather than silently skipping decryption (bootstrap/sops/age-key-secret.md).

Phase C — bring the reconciliation graph up

One script, six steps, each independently runnable and idempotent. Run them in this order.

K=infra/ansible/.cache/prod.kubeconfig
S=https://192.168.2.9:6443

C1 — preflight (read-only, changes nothing)

Asserts the guards that prevent a repeat of the DNS incident and reports every tier's state. Fails closed if EXTERNALDNS_TXT_OWNER_ID is empty, still the base sentinel, or default (dev's owner).

timeout 300 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 preflight

C2 — pre-apply chart CRDs, and break the CDI cold-start deadlock (idempotent)

timeout 600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 crds

Two cold-start problems, same root cause: Flux server-side dry-runs every object in a Kustomization before applying any of them, so one unknown kind aborts the entire tier.

1. Kyverno — its ~5.6 MiB CRD bundle exceeds kubectl's 256 KiB annotation limit, so the chart sets crds.install: false and nothing else installs them. bootstrap/flux/preapply-crds.sh applies them server-side, deriving the version from the HelmRelease Flux reconciles (no second pin).

2. CDI — a true deadlock. platform-config applies the CDI CR and platform/kubevirt/cdi-storageprofiles.yaml in the same pass. But StorageProfile's CRD does not exist until cdi-operator has reconciled that CR. So the dry-run fails on the unknown kind → the CR is never applied → the CRD is never created → platform-config can never converge, and every tier below it stays blocked. Invisible on an established cluster (dev's CRDs have been there for months) and fatal on a cold one. Observed on prod 2026-09-05: cdi-operator Available for 12h with no CDI CR and no storageprofiles CRD.

   The step breaks it by applying only the CR — the minimum — and leaving everything else to Flux. It carries no ${} tokens, so applying it here cannot diverge from what Flux would render. The CRD appeared in 10s.

   The durable fix is a repo change, not this script: move cdi-storageprofiles.yaml out of platform-config into its own Kustomization with dependsOn: platform-config, so the CR lands one reconcile before the resources that need its CRDs. Until that lands, C2 is a prerequisite for C5 on any cold cluster.

C3 — unwedge stuck Helm releases

A release left in uninstalling / pending-* cannot be advanced by helm-controller: it reports StateError forever, so platform (wait: true) never goes Ready and all 10 tiers below it stall. Dry run by default — it prints what it would do and changes nothing:

timeout 600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 unwedge

To act (this deletes the release's Kubernetes objects and lets Flux install cold). Pre-applied CRDs live outside the Helm release and survive, which is exactly why C2 comes first:

timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 \
  --confirm-destructive unwedge

C4 — external-dns safe upgrade

The most dangerous step in the repo, and the ordering is the entire mitigation. A Helm upgrade resets replicas to the chart default (1) and applies the new args in ONE operation — there is no window in which to check the result first. If --txt-owner-id were missing, the pod starts as owner default, claims dev's TXT registry entries, and under policy: sync deletes the records behind them.

So this step proves the outcome before causing it: it renders the chart with the live HelmRelease values and asserts --txt-owner-id=<owner> appears, aborting if not. Rendering is deterministic and touches nothing, which converts a race into a precondition. It then snapshots the Cloudflare zone, upgrades, re-verifies the live args (scaling to 0 and failing if they are wrong), and diffs the zone.

timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 externaldns

Expected diff: additions only, all under prod.home.keithjackson.dev, all TXT entries owner prod. Any change to a name outside that subtree, or to a TXT entry owned by another cluster, is a regression — stop and investigate.

C5 — resume the tiers, in dependency order

platform → platform-config → {oauth2-proxy, headlamp, homepage, crafty, apps-wave-0, apps-vms} → apps-wave-1 → apps-wave-2. Waits for each to go Ready and stops at the first that does not — a partial, understood bring-up beats a forced green. velero is skipped while S3_BACKUP_ENDPOINT is a placeholder (see §Blockers). Long-running; expect it to background.

timeout 3600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 resume

C6 — verify

timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 verify

Everything at once

Runs C1→C6 with unwedge in dry-run mode (no --confirm-destructive), so it stops short of the destructive step by design:

timeout 3600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 all

---

VERIFIED WORKING — 2026-09-05, standup complete

Kustomizations	12/13 Ready (velero deliberately suspended — see §Blockers)
Pods	113 total, 0 not Running/Completed
Wildcard cert	wildcard-home Ready, Let's Encrypt prod, *.prod.home.keithjackson.dev
Gateway VIP	192.168.2.20 · crafty L4 192.168.2.22 · unifi L4 192.168.2.21
Published DNS	36 records under prod.home.keithjackson.dev, every TXT registry entry owner=prod
Dev's records	57, byte-identical to the pre-standup baseline — nothing added, changed or removed

End-to-end HTTPS, certificate validated (no -k), pinned to prod's gateway:

prod.home.keithjackson.dev            200     keycloak.…  302 (login redirect)
gitea.prod.home.keithjackson.dev      200     grafana.…   302 (SSO redirect)

ACTION REQUIRED — the pfSense override shadows prod from the LAN

getent hosts on the workstation returns 192.168.2.50 (dev) for every *.home.keithjackson.dev name — and the new prod.home.… subtree falls inside that wildcard. Cloudflare correctly returns 192.168.2.20. So prod is reachable from anywhere using public DNS, but from this workstation and anything else behind pfSense, prod hostnames resolve to dev.

Narrow the override so it does not cover prod.home.keithjackson.dev, or add a matching override pointing that subtree at 192.168.2.20. Until then, test prod with an explicit resolve:

curl -sS --resolve gitea.prod.home.keithjackson.dev:443:192.168.2.20 \
  https://gitea.prod.home.keithjackson.dev/

This is also the trap that produced a false healthy reading during the 2026-09-04 incident. It is subtle because the apex appears to work: dev's *.home.keithjackson.dev wildcard matches prod.home.keithjackson.dev (one label) so TLS succeeds, while keycloak.prod.home.keithjackson.dev (two labels) fails with a certificate-name error. A partial success is what makes it convincing. Never verify DNS or TLS through a local resolver here.

Blockers

1. Backups: endpoint DECIDED, but prod cannot reach it — prod has no backup coverage

Operator ruling 2026-09-05: "the S3 backup location is the same as Dev." Prod's S3_BACKUP_ENDPOINT is now http://192.168.1.9:9000, and the TBD-PROD-* placeholder is retired.

It is not reachable from prod. Measured from a throwaway pod inside the prod cluster — not from the workstation, which proves nothing about pod egress:

curl http://192.168.1.9:9000/minio/health/live   →  connection timed out after 15s

pfSense does not route prod's pod network to the dev LAN's MinIO. Operator action required. Re-test with standup.sh --s3-endpoint http://192.168.1.9:9000 s3check — it fails closed and says so.

Until that routing exists, velero stays suspended and apps-wave-0 still applies GREEN — the token is defined, so strict substitution succeeds — while every write to that endpoint fails at runtime. A green apps-wave-0 does not mean backups work. Treat prod as having no backup coverage.

Sharing one MinIO is only safe because of two new knobs

Nothing about the backup paths was per-environment before 2026-09-05. Enabling prod against dev's MinIO without these would have been the external-dns incident again, in a worse place:

	before	after
Velero	bucket: velero, no prefix, both clusters run schedules named daily	prefix: ${VELERO_BACKUP_PREFIX} — dev "", prod prod
CNPG	destinationPath: s3://cnpg, serverName defaults to the Cluster name	${CNPG_BACKUP_PATH} — dev s3://cnpg, prod s3://cnpg/prod

• Velero syncs backups it finds in its BSL into itself as Backup CRs and garbage-collects them when their TTL expires. Prod would have adopted dev's backups and deleted them on schedule.
• CNPG is the worse one. Both environments have keycloak-db, nextcloud-db and semaphore-db, so prod's Postgres would archive WAL into the exact prefix dev's does. Two servers interleaving WAL into one Barman stream corrupts the chain and destroys PITR — and dev's restore is proven (R-STATE-03).

The base values are dev's current layout on purpose and must stay that way. Dev's existing backups live at the bucket root and at s3://cnpg; giving dev a prefix would orphan the very backups its proven restore depends on — the same reasoning that keeps dev's EXTERNALDNS_TXT_OWNER_ID at default. Prod has no history to preserve, so it takes the nested paths.

Delivery note. Prod reconciles the bootstrap Gitea's main at c8c1d29. This change sits on a later branch that also carries the DNS inversion, which must not reach prod (it would move prod to the apex and collide with dev). Delivering it needs a c8c1d29-based branch carrying only these commits. No urgency: velero cannot be enabled until the routing above is fixed.

2. Prod is on prod.home.keithjackson.dev, not the apex

The operator's target layout is dev under *.dev.home.keithjackson.dev and prod on the apex. Prod cannot take the apex until dev vacates it, because both clusters would request a letsencrypt-prod wildcard for the same name set and race each other's _acme-challenge TXT records, so neither reliably solves DNS-01.

This does not block phase C. Prod on its own subtree with txtOwnerId: prod collides with nothing, and standing it up first is strictly better: it proves the whole stack before the flip, so the cutover becomes a DNS/TLS change on a known-good cluster instead of a bring-up and a cutover at once. The apex move is infra/PROD-PROVISIONING.md §10a, sequence D (dev vacates) then P.

---

Not covered here

• Dev's migration off the apex — §10a sequence D. Touches Keycloak's issuer, ~12 OIDC client redirect URIs, and a rolling k3s restart that invalidates every OIDC kubeconfig. Operator-present.
• Prod's GitOps cutover (P5) from the workstation bootstrap Gitea to prod's own in-cluster Gitea. clusters/prod/kustomization.yaml owns that procedure.
• Six L2 conformance tests hardcode the apex and must be parameterized per environment before §9's "suite passes green against the prod cluster" can be claimed.