home-network-k3s-impl/infra/PROD-STANDUP.md

Prod cluster standup — the complete command sequence

Every command below is copy-pasteable and runs from the repo root, through the pinned dev container. Nothing here needs a command invented on the spot; that is the point. If a step is not in this file it is not part of standup.

State as of 2026-09-05: phases A and B are DONE on the current hardware. The cluster is live (k3s v1.36.4 HA, 3 etcd members, kube-vip, Cilium 1.20.1, Longhorn) with platform reconciling and 8/9 HelmReleases Ready, 76/76 pods Running. Outstanding work is phase C, blocked on the two items in §Blockers.

The one guard that matters. Dev is LIVE and shares the keithjackson.dev Cloudflare zone with prod. On 2026-09-04 prod deleted every one of dev's public DNS records. bootstrap/standup.sh therefore requires --expect-server and compares it against the live API server before running anything. There is no override flag. Do not add one.

Constants

kube-API (kube-vip VIP)	https://192.168.2.9:6443
Gateway / ingress VIP	192.168.2.20
Nodes	k3s-prod-1/2/3 — 192.168.2.10 / .11 / .12
Break-glass kubeconfig	infra/ansible/.cache/prod.kubeconfig (cert-based; survives OIDC/DNS failure)
GitOps source (interim)	http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git branch main
BASE_DOMAIN	prod.home.keithjackson.dev — not the apex; see §Blockers

Container paths: the repo is mounted at /work, so kubeconfig paths passed to the script are repo-relative (infra/ansible/.cache/prod.kubeconfig).

---

Phase A — bare metal to a joined cluster (Ansible)

Prerequisites that are not commands: firmware/BIOS (§1), disk layout (§2), OS install (§3), network (§4), and SSH + the vaulted ansible_become_password in inventory/group_vars/k3s_prod/vault.yml (§5). Ansible cannot escalate on prod without that vault file. See infra/PROD-PROVISIONING.md.

Install the backports kernel first. All three prod nodes need ≥6.13 because HP's R21 firmware publishes a corrupt EFI Memory Attributes Table and Linux 6.12's efi_memattr_init() does not validate it, reserving everything above the table (fixed upstream in 6.13 by 8fbe4c49c0cc, absent from linux-6.12.y). Prod runs 7.1.8+deb13-amd64 from trixie-backports.

timeout 3600 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
  ansible-playbook /work/infra/ansible/playbooks/node-baseline.yml \
  -i /work/infra/ansible/inventory/hosts.ini -e k3s_hosts=k3s_prod --tags kernel

Then the full chain — baseline, k3s HA, Cilium, verify, Flux hand-off. All three -e flags are required: a play's hosts: is templated before any host exists, so group_vars cannot retarget it.

timeout 3600 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
  ansible-playbook /work/infra/ansible/playbooks/site.yml \
  -i /work/infra/ansible/inventory/hosts.ini \
  -e k3s_hosts=k3s_prod -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary

kube-vip is a separate play (not imported by site.yml):

timeout 1800 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
  ansible-playbook /work/infra/ansible/playbooks/kube-vip.yml \
  -i /work/infra/ansible/inventory/hosts.ini -e k3s_hosts=k3s_prod

Hardware health (read-only; node_health asserts the memattr fix took and that all RAM is addressed).

The flag is node_health_hosts, NOT k3s_hosts. Diagnostic plays deliberately keep their own target var so they stay independently pointable while a bring-up run is scoped elsewhere (node-health.yml → node_health_hosts, memattr-survey.yml → memattr_survey_hosts, vpn.egress.gateway.yml → vpn_egress_gateway_hosts). The consequence is a silent one: passing k3s_hosts=k3s_prod here does not error, it falls through to the k3s_servers default and runs against DEV's three nodes. Verified — this exact mistake was made while writing this file and caught only by --list-hosts. Always dry-run the target first:

timeout 200 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
  ansible-playbook /work/infra/ansible/playbooks/node-health.yml \
  -i /work/infra/ansible/inventory/hosts.ini -e node_health_hosts=k3s_prod --list-hosts

timeout 900 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
  ansible-playbook /work/infra/ansible/playbooks/node-health.yml \
  -i /work/infra/ansible/inventory/hosts.ini -e node_health_hosts=k3s_prod

Phase B — Flux bootstrap

flux-bootstrap.yml (last play of site.yml) shells to bootstrap/flux/bootstrap.sh, the single documented entrypoint. That script now runs the CRD pre-apply as step [2/4], so a cold bootstrap no longer strands platform. The age private key Secret must exist first — bootstrap verifies it and refuses to continue rather than silently skipping decryption (bootstrap/sops/age-key-secret.md).

Phase C — bring the reconciliation graph up

One script, six steps, each independently runnable and idempotent. Run them in this order.

K=infra/ansible/.cache/prod.kubeconfig
S=https://192.168.2.9:6443

C1 — preflight (read-only, changes nothing)

Asserts the guards that prevent a repeat of the DNS incident and reports every tier's state. Fails closed if EXTERNALDNS_TXT_OWNER_ID is empty, still the base sentinel, or default (dev's owner).

timeout 300 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 preflight

C2 — pre-apply chart CRDs, and break the CDI cold-start deadlock (idempotent)

timeout 600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 crds

Two cold-start problems, same root cause: Flux server-side dry-runs every object in a Kustomization before applying any of them, so one unknown kind aborts the entire tier.

1. Kyverno — its ~5.6 MiB CRD bundle exceeds kubectl's 256 KiB annotation limit, so the chart sets crds.install: false and nothing else installs them. bootstrap/flux/preapply-crds.sh applies them server-side, deriving the version from the HelmRelease Flux reconciles (no second pin).

2. CDI — a true deadlock. platform-config applies the CDI CR and platform/kubevirt/cdi-storageprofiles.yaml in the same pass. But StorageProfile's CRD does not exist until cdi-operator has reconciled that CR. So the dry-run fails on the unknown kind → the CR is never applied → the CRD is never created → platform-config can never converge, and every tier below it stays blocked. Invisible on an established cluster (dev's CRDs have been there for months) and fatal on a cold one. Observed on prod 2026-09-05: cdi-operator Available for 12h with no CDI CR and no storageprofiles CRD.

   The step breaks it by applying only the CR — the minimum — and leaving everything else to Flux. It carries no ${} tokens, so applying it here cannot diverge from what Flux would render. The CRD appeared in 10s.

   The durable fix is a repo change, not this script: move cdi-storageprofiles.yaml out of platform-config into its own Kustomization with dependsOn: platform-config, so the CR lands one reconcile before the resources that need its CRDs. Until that lands, C2 is a prerequisite for C5 on any cold cluster.

C3 — unwedge stuck Helm releases

A release left in uninstalling / pending-* cannot be advanced by helm-controller: it reports StateError forever, so platform (wait: true) never goes Ready and all 10 tiers below it stall. Dry run by default — it prints what it would do and changes nothing:

timeout 600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 unwedge

To act (this deletes the release's Kubernetes objects and lets Flux install cold). Pre-applied CRDs live outside the Helm release and survive, which is exactly why C2 comes first:

timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 \
  --confirm-destructive unwedge

C4 — external-dns safe upgrade

The most dangerous step in the repo, and the ordering is the entire mitigation. A Helm upgrade resets replicas to the chart default (1) and applies the new args in ONE operation — there is no window in which to check the result first. If --txt-owner-id were missing, the pod starts as owner default, claims dev's TXT registry entries, and under policy: sync deletes the records behind them.

So this step proves the outcome before causing it: it renders the chart with the live HelmRelease values and asserts --txt-owner-id=<owner> appears, aborting if not. Rendering is deterministic and touches nothing, which converts a race into a precondition. It then snapshots the Cloudflare zone, upgrades, re-verifies the live args (scaling to 0 and failing if they are wrong), and diffs the zone.

timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 externaldns

Expected diff: additions only, all under prod.home.keithjackson.dev, all TXT entries owner prod. Any change to a name outside that subtree, or to a TXT entry owned by another cluster, is a regression — stop and investigate.

C5 — resume the tiers, in dependency order

platform → platform-config → {oauth2-proxy, headlamp, homepage, crafty, apps-wave-0, apps-vms} → apps-wave-1 → apps-wave-2. Waits for each to go Ready and stops at the first that does not — a partial, understood bring-up beats a forced green. velero is skipped while S3_BACKUP_ENDPOINT is a placeholder (see §Blockers). Long-running; expect it to background.

timeout 3600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 resume

C6 — verify

timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 verify

Everything at once

Runs C1→C6 with unwedge in dry-run mode (no --confirm-destructive), so it stops short of the destructive step by design:

timeout 3600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 all

---

VERIFIED WORKING — 2026-09-05, standup complete

Kustomizations	12/13 Ready (velero deliberately suspended — see §Blockers)
Pods	113 total, 0 not Running/Completed
Wildcard cert	wildcard-home Ready, Let's Encrypt prod, *.prod.home.keithjackson.dev
Gateway VIP	192.168.2.20 · crafty L4 192.168.2.22 · unifi L4 192.168.2.21
Published DNS	36 records under prod.home.keithjackson.dev, every TXT registry entry owner=prod
Dev's records	57, byte-identical to the pre-standup baseline — nothing added, changed or removed

End-to-end HTTPS through the workstation's own resolver — no --resolve override, no -k (after installing the dnsmasq profile below). Every request landed on 192.168.2.20:

prod.home.keithjackson.dev  200   headlamp.  200   keycloak.  302 (login)   nextcloud. 302
gitea.       200            semaphore. 200        grafana.   302 (SSO)      pihole.    302

The served certificate is prod's own, confirmed by SNI — not dev's wildcard, which is what the resolver bug previously made it look like:

subject = CN=*.prod.home.keithjackson.dev
issuer  = C=US, O=Let's Encrypt, CN=YR1          (LE production)
SAN     = *.prod.home.keithjackson.dev, prod.home.keithjackson.dev
valid   = 2026-09-05 .. 2026-12-04

Dev, checked in the same pass, still serves CN=*.home.keithjackson.dev — unchanged.

Ignore curl: (91) No OCSP response received if you pass --cert-status: Let's Encrypt retired OCSP responders in 2025, so there is nothing to staple. It is not a certificate fault, and the plain requests above all validated normally.

Workstation DNS — install the prod dnsmasq profile

CORRECTION (operator, 2026-09-05): this is the workstation's own dnsmasq, NOT pfSense. Earlier notes in this repo called it a "pfSense override"; that was wrong and the topology below is why.

/etc/NetworkManager/dnsmasq.d/10-home.keithjackson.dev.conf (the dev profile) contains:

address=/.home.keithjackson.dev/192.168.2.50

dnsmasq's address= matches the named domain and every subdomain, so that one line also captures the whole prod.home.keithjackson.dev tree and answers it with dev's address, while Cloudflare correctly returns prod's 192.168.2.20.

Fix — one command, already authored in this repo:

sudo infra/workstation/dnsmasq/install.sh

It installs 20-prod.home.keithjackson.dev.conf and reloads the live resolver. dnsmasq resolves address= by longest match, so the prod rules win without editing or reordering the dev profile. Verified against a throwaway dnsmasq loaded with both profiles:

query	answer
prod.home.keithjackson.dev, gitea.prod…, keycloak.prod…	192.168.2.20 (Gateway)
unifi.prod.home.keithjackson.dev	192.168.2.21 (L4)
crafty-mc.prod.home.keithjackson.dev	192.168.2.22 (L4)
gitea.home.keithjackson.dev, home.keithjackson.dev	192.168.2.50 — dev, unchanged

Install to /etc/NetworkManager/dnsmasq.d/, never /etc/dnsmasq.d/. The standalone dnsmasq.service is inactive; the live resolver is the one NetworkManager spawns (dns=dnsmasq) with --conf-dir=/etc/NetworkManager/dnsmasq.d. A file in the other directory changes nothing and looks like it should. install.sh pins the destination for exactly this reason.

Until it is installed, verify prod only with an explicit resolve — and note why this trap is convincing: dev serves a *.home.keithjackson.dev wildcard cert, which matches prod.home.keithjackson.dev (one label), so the prod apex returns 200 over valid TLS while keycloak.prod.home.keithjackson.dev (two labels) fails with a certificate-name error. It reads like a broken prod certificate. It is the wrong server answering.

curl -sS --resolve gitea.prod.home.keithjackson.dev:443:192.168.2.20 \
  https://gitea.prod.home.keithjackson.dev/

Blockers

1. Backups: endpoint DECIDED, but prod cannot reach it — prod has no backup coverage

Operator ruling 2026-09-05: "the S3 backup location is the same as Dev." Prod's S3_BACKUP_ENDPOINT is now http://192.168.1.9:9000, and the TBD-PROD-* placeholder is retired.

It is not reachable from prod. Measured from a throwaway pod inside the prod cluster — not from the workstation, which proves nothing about pod egress:

curl http://192.168.1.9:9000/minio/health/live   →  connection timed out after 15s

This is expected topology, not a misconfiguration (operator, 2026-09-05): the k3s prod cluster is not behind pfSense. It sits on the SOHO router network 192.168.2.0/24, which is pfSense's WAN side; pfSense contains 192.168.1.0/24, the dev LAN where MinIO lives. Prod reaching 192.168.1.9 therefore means crossing into pfSense from its WAN — which requires a deliberate pfSense rule or port-forward that does not exist. Operator decision required, and worth weighing: a prod backup that lands inside the dev LAN is arguably not off-cluster DR at all.

Re-test with standup.sh --s3-endpoint http://192.168.1.9:9000 s3check — it fails closed and says so.

Do not confuse this with the SOHO router's inbound NAT. That router forwards inbound 443 to the pfSense WAN address 192.168.2.102 to serve the existing, non-k3s production environment, which must stay live on 443 until the k3s cutover is deliberately made. Nothing in this runbook touches that NAT, and prod's Gateway on 192.168.2.20:443 does not contend with it.

Until that routing exists, velero stays suspended and apps-wave-0 still applies GREEN — the token is defined, so strict substitution succeeds — while every write to that endpoint fails at runtime. A green apps-wave-0 does not mean backups work. Treat prod as having no backup coverage.

Sharing one MinIO is only safe because of two new knobs

Nothing about the backup paths was per-environment before 2026-09-05. Enabling prod against dev's MinIO without these would have been the external-dns incident again, in a worse place:

	before	after
Velero	bucket: velero, no prefix, both clusters run schedules named daily	prefix: ${VELERO_BACKUP_PREFIX} — dev "", prod prod
CNPG	destinationPath: s3://cnpg, serverName defaults to the Cluster name	${CNPG_BACKUP_PATH} — dev s3://cnpg, prod s3://cnpg/prod

• Velero syncs backups it finds in its BSL into itself as Backup CRs and garbage-collects them when their TTL expires. Prod would have adopted dev's backups and deleted them on schedule.
• CNPG is the worse one. Both environments have keycloak-db, nextcloud-db and semaphore-db, so prod's Postgres would archive WAL into the exact prefix dev's does. Two servers interleaving WAL into one Barman stream corrupts the chain and destroys PITR — and dev's restore is proven (R-STATE-03).

The base values are dev's current layout on purpose and must stay that way. Dev's existing backups live at the bucket root and at s3://cnpg; giving dev a prefix would orphan the very backups its proven restore depends on — the same reasoning that keeps dev's EXTERNALDNS_TXT_OWNER_ID at default. Prod has no history to preserve, so it takes the nested paths.

Delivery note. Prod reconciles the bootstrap Gitea's main at c8c1d29. This change sits on a later branch that also carries the DNS inversion, which must not reach prod (it would move prod to the apex and collide with dev). Delivering it needs a c8c1d29-based branch carrying only these commits. No urgency: velero cannot be enabled until the routing above is fixed.

2. Prod is on prod.home.keithjackson.dev, not the apex

The operator's target layout is dev under *.dev.home.keithjackson.dev and prod on the apex. Prod cannot take the apex until dev vacates it, because both clusters would request a letsencrypt-prod wildcard for the same name set and race each other's _acme-challenge TXT records, so neither reliably solves DNS-01.

This does not block phase C. Prod on its own subtree with txtOwnerId: prod collides with nothing, and standing it up first is strictly better: it proves the whole stack before the flip, so the cutover becomes a DNS/TLS change on a known-good cluster instead of a bring-up and a cutover at once. The apex move is infra/PROD-PROVISIONING.md §10a, sequence D (dev vacates) then P.

THE APEX FLIP IS ALSO THE LEGACY-PROD CUTOVER — they are the same event. The SOHO router NATs inbound 443 to the pfSense WAN address 192.168.2.102, serving the existing, non-k3s production environment, which the operator requires to stay live until the switch is deliberately made. So §10a sequence P is not merely a DNS/TLS change on the k3s side: taking home.keithjackson.dev means public traffic must also stop resolving to / being forwarded to the legacy stack. Sequence P as written does not cover repointing that NAT, and the two must be planned together or the apex will resolve to k3s while inbound 443 still lands on the legacy box.

Nothing in phase C touches it: prod's Gateway is 192.168.2.20:443, a different address from the NAT target, and prod's wildcard cert is issued by DNS-01, which needs no inbound 443 at all.

---

Not covered here

• Dev's migration off the apex — §10a sequence D. Touches Keycloak's issuer, ~12 OIDC client redirect URIs, and a rolling k3s restart that invalidates every OIDC kubeconfig. Operator-present.
• Prod's GitOps cutover (P5) from the workstation bootstrap Gitea to prod's own in-cluster Gitea. clusters/prod/kustomization.yaml owns that procedure.
• Six L2 conformance tests hardcode the apex and must be parameterized per environment before §9's "suite passes green against the prod cluster" can be claimed.