home-network-k3s-impl/infra/PROD-STANDUP.md

# Prod cluster standup — the complete command sequence

**Every command below is copy-pasteable and runs from the repo root**, through the pinned dev
container. Nothing here needs a command invented on the spot; that is the point. If a step is not in
this file it is not part of standup.

State as of **2026-09-05**: phases **A** and **B** are DONE on the current hardware. The cluster is
live (k3s v1.36.4 HA, 3 etcd members, kube-vip, Cilium 1.20.1, Longhorn) with `platform` reconciling
and **8/9 HelmReleases Ready, 76/76 pods Running**. Outstanding work is **phase C**, blocked on the
two items in [§Blockers](#blockers).

> **The one guard that matters.** Dev is LIVE and shares the `keithjackson.dev` Cloudflare zone with
> prod. On 2026-09-04 prod deleted every one of dev's public DNS records. `bootstrap/standup.sh`
> therefore **requires** `--expect-server` and compares it against the live API server before running
> anything. There is no override flag. Do not add one.

## Constants

| | |
|---|---|
| kube-API (kube-vip VIP) | `https://192.168.2.9:6443` |
| Gateway / ingress VIP | `192.168.2.20` |
| Nodes | `k3s-prod-1/2/3` — `192.168.2.10 / .11 / .12` |
| Break-glass kubeconfig | `infra/ansible/.cache/prod.kubeconfig` (cert-based; survives OIDC/DNS failure) |
| GitOps source (interim) | `http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git` branch `main` |
| `BASE_DOMAIN` | `prod.home.keithjackson.dev` — **not** the apex; see §Blockers |

Container paths: the repo is mounted at `/work`, so kubeconfig paths passed to the script are
**repo-relative** (`infra/ansible/.cache/prod.kubeconfig`).

---

## Phase A — bare metal to a joined cluster (Ansible)

Prerequisites that are **not** commands: firmware/BIOS (§1), disk layout (§2), OS install (§3),
network (§4), and SSH + the vaulted `ansible_become_password` in
`inventory/group_vars/k3s_prod/vault.yml` (§5). Ansible cannot escalate on prod without that vault
file. See `infra/PROD-PROVISIONING.md`.

**Install the backports kernel first.** All three prod nodes need ≥6.13 because HP's R21 firmware
publishes a corrupt EFI Memory Attributes Table and Linux 6.12's `efi_memattr_init()` does not
validate it, reserving everything above the table (fixed upstream in 6.13 by `8fbe4c49c0cc`, absent
from `linux-6.12.y`). Prod runs `7.1.8+deb13-amd64` from trixie-backports.

```bash
timeout 3600 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
  ansible-playbook /work/infra/ansible/playbooks/node-baseline.yml \
  -i /work/infra/ansible/inventory/hosts.ini -e k3s_hosts=k3s_prod --tags kernel
```

Then the full chain — baseline, k3s HA, Cilium, verify, Flux hand-off. **All three `-e` flags are
required**: a play's `hosts:` is templated before any host exists, so `group_vars` cannot retarget it.

```bash
timeout 3600 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
  ansible-playbook /work/infra/ansible/playbooks/site.yml \
  -i /work/infra/ansible/inventory/hosts.ini \
  -e k3s_hosts=k3s_prod -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary
```

kube-vip is a separate play (not imported by `site.yml`):

```bash
timeout 1800 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
  ansible-playbook /work/infra/ansible/playbooks/kube-vip.yml \
  -i /work/infra/ansible/inventory/hosts.ini -e k3s_hosts=k3s_prod
```

Hardware health (read-only; `node_health` asserts the memattr fix took and that all RAM is addressed).

> **The flag is `node_health_hosts`, NOT `k3s_hosts`.** Diagnostic plays deliberately keep their own
> target var so they stay independently pointable while a bring-up run is scoped elsewhere
> (`node-health.yml` → `node_health_hosts`, `memattr-survey.yml` → `memattr_survey_hosts`,
> `vpn.egress.gateway.yml` → `vpn_egress_gateway_hosts`). The consequence is a **silent** one: passing
> `k3s_hosts=k3s_prod` here does not error, it falls through to the `k3s_servers` default and runs
> against **DEV's three nodes**. Verified — this exact mistake was made while writing this file and
> caught only by `--list-hosts`. **Always dry-run the target first:**
>
> ```bash
> timeout 200 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
>   ansible-playbook /work/infra/ansible/playbooks/node-health.yml \
>   -i /work/infra/ansible/inventory/hosts.ini -e node_health_hosts=k3s_prod --list-hosts
> ```

```bash
timeout 900 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
  ansible-playbook /work/infra/ansible/playbooks/node-health.yml \
  -i /work/infra/ansible/inventory/hosts.ini -e node_health_hosts=k3s_prod
```

## Phase B — Flux bootstrap

`flux-bootstrap.yml` (last play of `site.yml`) shells to `bootstrap/flux/bootstrap.sh`, the single
documented entrypoint. That script now runs the CRD pre-apply as step **[2/4]**, so a cold bootstrap
no longer strands `platform`. The **age private key Secret must exist first** — bootstrap verifies it
and refuses to continue rather than silently skipping decryption
(`bootstrap/sops/age-key-secret.md`).

## Phase C — bring the reconciliation graph up

One script, six steps, each independently runnable and idempotent. **Run them in this order.**

```bash
K=infra/ansible/.cache/prod.kubeconfig
S=https://192.168.2.9:6443
```

### C1 — preflight (read-only, changes nothing)

Asserts the guards that prevent a repeat of the DNS incident and reports every tier's state. **Fails
closed** if `EXTERNALDNS_TXT_OWNER_ID` is empty, still the base sentinel, or `default` (dev's owner).

```bash
timeout 300 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 preflight
```

### C2 — pre-apply chart CRDs, and break the CDI cold-start deadlock (idempotent)

```bash
timeout 600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 crds
```

Two cold-start problems, same root cause: **Flux server-side dry-runs every object in a Kustomization
before applying any of them, so one unknown `kind` aborts the entire tier.**

1. **Kyverno** — its ~5.6 MiB CRD bundle exceeds kubectl's 256 KiB annotation limit, so the chart sets
   `crds.install: false` and nothing else installs them. `bootstrap/flux/preapply-crds.sh` applies them
   server-side, deriving the version from the HelmRelease Flux reconciles (no second pin).

2. **CDI — a true deadlock.** `platform-config` applies the CDI CR *and*
   `platform/kubevirt/cdi-storageprofiles.yaml` in the **same pass**. But `StorageProfile`'s CRD does
   not exist until `cdi-operator` has reconciled that CR. So the dry-run fails on the unknown kind →
   the CR is never applied → the CRD is never created → `platform-config` can never converge, and
   every tier below it stays blocked. **Invisible on an established cluster** (dev's CRDs have been
   there for months) **and fatal on a cold one.** Observed on prod 2026-09-05: `cdi-operator`
   Available for 12h with **no CDI CR** and no `storageprofiles` CRD.

   The step breaks it by applying **only the CR** — the minimum — and leaving everything else to
   Flux. It carries no `${}` tokens, so applying it here cannot diverge from what Flux would render.
   The CRD appeared in 10s.

   **The durable fix is a repo change, not this script:** move `cdi-storageprofiles.yaml` out of
   `platform-config` into its own Kustomization with `dependsOn: platform-config`, so the CR lands one
   reconcile before the resources that need its CRDs. Until that lands, C2 is a prerequisite for C5 on
   any cold cluster.

### C3 — unwedge stuck Helm releases

A release left in `uninstalling` / `pending-*` cannot be advanced by helm-controller: it reports
`StateError` forever, so `platform` (`wait: true`) never goes Ready and all 10 tiers below it stall.
**Dry run by default** — it prints what it would do and changes nothing:

```bash
timeout 600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 unwedge
```

To act (this **deletes** the release's Kubernetes objects and lets Flux install cold). Pre-applied
CRDs live **outside** the Helm release and survive, which is exactly why C2 comes first:

```bash
timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 \
  --confirm-destructive unwedge
```

### C4 — external-dns safe upgrade

**The most dangerous step in the repo, and the ordering is the entire mitigation.** A Helm upgrade
resets `replicas` to the chart default (1) and applies the new args in ONE operation — there is no
window in which to check the result first. If `--txt-owner-id` were missing, the pod starts as owner
`default`, claims dev's TXT registry entries, and under `policy: sync` **deletes the records behind
them**.

So this step **proves the outcome before causing it**: it renders the chart with the *live*
HelmRelease values and asserts `--txt-owner-id=<owner>` appears, aborting if not. Rendering is
deterministic and touches nothing, which converts a race into a precondition. It then snapshots the
Cloudflare zone, upgrades, re-verifies the live args (scaling to 0 and failing if they are wrong),
and diffs the zone.

```bash
timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 externaldns
```

Expected diff: **additions only, all under `prod.home.keithjackson.dev`, all TXT entries owner
`prod`.** Any change to a name outside that subtree, or to a TXT entry owned by another cluster, is a
regression — stop and investigate.

### C5 — resume the tiers, in dependency order

`platform → platform-config → {oauth2-proxy, headlamp, homepage, crafty, apps-wave-0, apps-vms} →
apps-wave-1 → apps-wave-2`. Waits for each to go Ready and **stops at the first that does not** — a
partial, understood bring-up beats a forced green. `velero` is skipped while `S3_BACKUP_ENDPOINT` is
a placeholder (see §Blockers). Long-running; expect it to background.

```bash
timeout 3600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 resume
```

### C6 — verify

```bash
timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 verify
```

### Everything at once

Runs C1→C6 with `unwedge` in **dry-run** mode (no `--confirm-destructive`), so it stops short of the
destructive step by design:

```bash
timeout 3600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 all
```

---

## VERIFIED WORKING — 2026-09-05, standup complete

| | |
|---|---|
| Kustomizations | **12/13 Ready** (`velero` deliberately suspended — see §Blockers) |
| Pods | **113 total, 0 not Running/Completed** |
| Wildcard cert | `wildcard-home` **Ready**, Let's Encrypt **prod**, `*.prod.home.keithjackson.dev` |
| Gateway VIP | `192.168.2.20` · crafty L4 `192.168.2.22` · unifi L4 `192.168.2.21` |
| Published DNS | **36 records** under `prod.home.keithjackson.dev`, every TXT registry entry `owner=prod` |
| Dev's records | **57, byte-identical to the pre-standup baseline — nothing added, changed or removed** |

End-to-end HTTPS, certificate validated (no `-k`), pinned to prod's gateway:

```
prod.home.keithjackson.dev            200     keycloak.…  302 (login redirect)
gitea.prod.home.keithjackson.dev      200     grafana.…   302 (SSO redirect)
```

### ACTION REQUIRED — the pfSense override shadows prod from the LAN

`getent hosts` on the workstation returns **192.168.2.50** (dev) for **every** `*.home.keithjackson.dev`
name — and the new `prod.home.…` subtree falls inside that wildcard. Cloudflare correctly returns
`192.168.2.20`. So **prod is reachable from anywhere using public DNS, but from this workstation and
anything else behind pfSense, prod hostnames resolve to dev.**

Narrow the override so it does not cover `prod.home.keithjackson.dev`, or add a matching override
pointing that subtree at `192.168.2.20`. Until then, test prod with an explicit resolve:

```bash
curl -sS --resolve gitea.prod.home.keithjackson.dev:443:192.168.2.20 \
  https://gitea.prod.home.keithjackson.dev/
```

**This is also the trap that produced a false healthy reading during the 2026-09-04 incident.** It is
subtle because the *apex* appears to work: dev's `*.home.keithjackson.dev` wildcard matches
`prod.home.keithjackson.dev` (one label) so TLS succeeds, while `keycloak.prod.home.keithjackson.dev`
(two labels) fails with a certificate-name error. A partial success is what makes it convincing.
**Never verify DNS or TLS through a local resolver here.**

## Blockers

### 1. Backups: endpoint DECIDED, but prod cannot reach it — **prod has no backup coverage**

**Operator ruling 2026-09-05: "the S3 backup location is the same as Dev."** Prod's
`S3_BACKUP_ENDPOINT` is now `http://192.168.1.9:9000`, and the `TBD-PROD-*` placeholder is retired.

**It is not reachable from prod.** Measured from a throwaway pod **inside the prod cluster** — not
from the workstation, which proves nothing about pod egress:

```
curl http://192.168.1.9:9000/minio/health/live   →  connection timed out after 15s
```

pfSense does not route prod's pod network to the dev LAN's MinIO. **Operator action required.**
Re-test with `standup.sh --s3-endpoint http://192.168.1.9:9000 s3check` — it fails closed and says so.

Until that routing exists, `velero` stays suspended and **`apps-wave-0` still applies GREEN** — the
token *is* defined, so strict substitution succeeds — while every write to that endpoint fails at
**runtime**. A green `apps-wave-0` does not mean backups work. **Treat prod as having no backup
coverage.**

#### Sharing one MinIO is only safe because of two new knobs

Nothing about the backup *paths* was per-environment before 2026-09-05. Enabling prod against dev's
MinIO without these would have been the external-dns incident again, in a worse place:

| | before | after |
|---|---|---|
| Velero | `bucket: velero`, **no prefix**, both clusters run schedules named `daily` | `prefix: ${VELERO_BACKUP_PREFIX}` — dev `""`, prod `prod` |
| CNPG | `destinationPath: s3://cnpg`, serverName defaults to the **Cluster name** | `${CNPG_BACKUP_PATH}` — dev `s3://cnpg`, prod `s3://cnpg/prod` |

- **Velero** syncs backups it finds in its BSL into itself as `Backup` CRs and **garbage-collects them
  when their TTL expires**. Prod would have adopted dev's backups and deleted them on schedule.
- **CNPG is the worse one.** Both environments have `keycloak-db`, `nextcloud-db` and `semaphore-db`,
  so prod's Postgres would archive WAL into the *exact prefix* dev's does. Two servers interleaving
  WAL into one Barman stream **corrupts the chain and destroys PITR** — and dev's restore is
  **proven** (R-STATE-03).

**The base values are dev's current layout on purpose and must stay that way.** Dev's existing
backups live at the bucket root and at `s3://cnpg`; giving dev a prefix would **orphan the very
backups its proven restore depends on** — the same reasoning that keeps dev's
`EXTERNALDNS_TXT_OWNER_ID` at `default`. Prod has no history to preserve, so it takes the nested paths.

> **Delivery note.** Prod reconciles the bootstrap Gitea's `main` at `c8c1d29`. This change sits on a
> later branch that also carries the DNS inversion, which must **not** reach prod (it would move prod
> to the apex and collide with dev). Delivering it needs a `c8c1d29`-based branch carrying only these
> commits. No urgency: velero cannot be enabled until the routing above is fixed.

### 2. Prod is on `prod.home.keithjackson.dev`, not the apex

The operator's target layout is dev under `*.dev.home.keithjackson.dev` and prod on the **apex**.
Prod cannot take the apex until dev vacates it, because both clusters would request a
`letsencrypt-prod` wildcard for the *same name set* and race each other's `_acme-challenge` TXT
records, so neither reliably solves DNS-01.

**This does not block phase C.** Prod on its own subtree with `txtOwnerId: prod` collides with
nothing, and standing it up first is strictly better: it proves the whole stack before the flip, so
the cutover becomes a DNS/TLS change on a known-good cluster instead of a bring-up and a cutover at
once. The apex move is `infra/PROD-PROVISIONING.md` §10a, sequence **D** (dev vacates) then **P**.

---

## Not covered here

- **Dev's migration off the apex** — §10a sequence D. Touches Keycloak's issuer, ~12 OIDC client
  redirect URIs, and a rolling k3s restart that invalidates every OIDC kubeconfig. Operator-present.
- **Prod's GitOps cutover** (P5) from the workstation bootstrap Gitea to prod's own in-cluster Gitea.
  `clusters/prod/kustomization.yaml` owns that procedure.
- **Six L2 conformance tests hardcode the apex** and must be parameterized per environment before
  §9's "suite passes green against the prod cluster" can be claimed.