home-network-k3s-impl/infra/PROD-STANDUP.md

# Prod cluster standup — the complete command sequence

**Every command below is copy-pasteable and runs from the repo root**, through the pinned dev
container. Nothing here needs a command invented on the spot; that is the point. If a step is not in
this file it is not part of standup.

State as of **2026-09-06**: phases **A**, **B** and **C** are DONE. The cluster is live — k3s
v1.36.4 HA (3 etcd members), kube-vip, Cilium 1.20.1, Longhorn — with **12/13 Flux tiers Ready**
(`velero` deliberately suspended), **112 pods Running**, an issued Let's Encrypt wildcard, and
**all three nodes addressing their full 32 GB** (node 2's DIMM upgrade completed 2026-09-06 and is
confirmed at ~`32638256Ki`, matching the other two to within 4 KiB — the §0b memattr bug is not
biting on the backports kernel).

Storage survived two unplanned node events (an ungraceful power-down of prod-3, then prod-2's memory
upgrade) and converged to **17/17 volumes healthy, replicas symmetric across all three nodes** —
though only after a leftover cordon was cleared; read
[§Node maintenance](#node-maintenance--draining-rebooting-and-the-cordon-that-stops-self-healing)
before doing node work, that cordon silently prevents Longhorn from rebuilding.

What remains is in [§Blockers](#blockers), and every item there needs an operator decision:
no backup coverage (R2 credentials), prod still on `prod.home.` rather than the apex, and — found
2026-09-05 — **prod's UniFi VM has no disk** because Kyverno blocks CDI's generated importer pod.

> **The one guard that matters.** Dev is LIVE and shares the `keithjackson.dev` Cloudflare zone with
> prod. On 2026-09-04 prod deleted every one of dev's public DNS records. `bootstrap/standup.sh`
> therefore **requires** `--expect-server` and compares it against the live API server before running
> anything. There is no override flag. Do not add one.

## Constants

| | |
|---|---|
| kube-API (kube-vip VIP) | `https://192.168.2.9:6443` |
| Gateway / ingress VIP | `192.168.2.20` |
| Nodes | `k3s-prod-1/2/3` — `192.168.2.10 / .11 / .12` |
| Break-glass kubeconfig | `infra/ansible/.cache/prod.kubeconfig` (cert-based; survives OIDC/DNS failure) |
| GitOps source (interim) | `http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git` branch `main` |
| `BASE_DOMAIN` | `prod.home.keithjackson.dev` — **not** the apex; see §Blockers |

Container paths: the repo is mounted at `/work`, so kubeconfig paths passed to the script are
**repo-relative** (`infra/ansible/.cache/prod.kubeconfig`).

---

## Phase A — bare metal to a joined cluster (Ansible)

Prerequisites that are **not** commands: firmware/BIOS (§1), disk layout (§2), OS install (§3),
network (§4), and SSH + the vaulted `ansible_become_password` in
`inventory/group_vars/k3s_prod/vault.yml` (§5). Ansible cannot escalate on prod without that vault
file. See `infra/PROD-PROVISIONING.md`.

**Install the backports kernel first.** All three prod nodes need ≥6.13 because HP's R21 firmware
publishes a corrupt EFI Memory Attributes Table and Linux 6.12's `efi_memattr_init()` does not
validate it, reserving everything above the table (fixed upstream in 6.13 by `8fbe4c49c0cc`, absent
from `linux-6.12.y`). Prod runs `7.1.8+deb13-amd64` from trixie-backports.

```bash
timeout 3600 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
  ansible-playbook /work/infra/ansible/playbooks/node-baseline.yml \
  -i /work/infra/ansible/inventory/hosts.ini -e k3s_hosts=k3s_prod --tags kernel
```

Then the full chain — baseline, k3s HA, Cilium, verify, Flux hand-off. **All three `-e` flags are
required**: a play's `hosts:` is templated before any host exists, so `group_vars` cannot retarget it.

```bash
timeout 3600 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
  ansible-playbook /work/infra/ansible/playbooks/site.yml \
  -i /work/infra/ansible/inventory/hosts.ini \
  -e k3s_hosts=k3s_prod -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary
```

kube-vip is a separate play (not imported by `site.yml`):

```bash
timeout 1800 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
  ansible-playbook /work/infra/ansible/playbooks/kube-vip.yml \
  -i /work/infra/ansible/inventory/hosts.ini -e k3s_hosts=k3s_prod
```

Hardware health (read-only; `node_health` asserts the memattr fix took and that all RAM is addressed).

> **The flag is `node_health_hosts`, NOT `k3s_hosts`.** Diagnostic plays deliberately keep their own
> target var so they stay independently pointable while a bring-up run is scoped elsewhere
> (`node-health.yml` → `node_health_hosts`, `memattr-survey.yml` → `memattr_survey_hosts`,
> `vpn.egress.gateway.yml` → `vpn_egress_gateway_hosts`). The consequence is a **silent** one: passing
> `k3s_hosts=k3s_prod` here does not error, it falls through to the `k3s_servers` default and runs
> against **DEV's three nodes**. Verified — this exact mistake was made while writing this file and
> caught only by `--list-hosts`. **Always dry-run the target first:**
>
> ```bash
> timeout 200 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
>   ansible-playbook /work/infra/ansible/playbooks/node-health.yml \
>   -i /work/infra/ansible/inventory/hosts.ini -e node_health_hosts=k3s_prod --list-hosts
> ```

```bash
timeout 900 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
  ansible-playbook /work/infra/ansible/playbooks/node-health.yml \
  -i /work/infra/ansible/inventory/hosts.ini -e node_health_hosts=k3s_prod
```

## Phase B — Flux bootstrap

`flux-bootstrap.yml` (last play of `site.yml`) shells to `bootstrap/flux/bootstrap.sh`, the single
documented entrypoint. That script now runs the CRD pre-apply as step **[2/4]**, so a cold bootstrap
no longer strands `platform`. The **age private key Secret must exist first** — bootstrap verifies it
and refuses to continue rather than silently skipping decryption
(`bootstrap/sops/age-key-secret.md`).

## Phase C — bring the reconciliation graph up

One script, six steps, each independently runnable and idempotent. **Run them in this order.**

```bash
K=infra/ansible/.cache/prod.kubeconfig
S=https://192.168.2.9:6443
```

### C1 — preflight (read-only, changes nothing)

Asserts the guards that prevent a repeat of the DNS incident and reports every tier's state. **Fails
closed** if `EXTERNALDNS_TXT_OWNER_ID` is empty, still the base sentinel, or `default` (dev's owner).

```bash
timeout 300 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 preflight
```

### C2 — pre-apply chart CRDs, and break the CDI cold-start deadlock (idempotent)

```bash
timeout 600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 crds
```

Two cold-start problems, same root cause: **Flux server-side dry-runs every object in a Kustomization
before applying any of them, so one unknown `kind` aborts the entire tier.**

1. **Kyverno** — its ~5.6 MiB CRD bundle exceeds kubectl's 256 KiB annotation limit, so the chart sets
   `crds.install: false` and nothing else installs them. `bootstrap/flux/preapply-crds.sh` applies them
   server-side, deriving the version from the HelmRelease Flux reconciles (no second pin).

2. **CDI — a true deadlock.** `platform-config` applies the CDI CR *and*
   `platform/kubevirt/cdi-storageprofiles.yaml` in the **same pass**. But `StorageProfile`'s CRD does
   not exist until `cdi-operator` has reconciled that CR. So the dry-run fails on the unknown kind →
   the CR is never applied → the CRD is never created → `platform-config` can never converge, and
   every tier below it stays blocked. **Invisible on an established cluster** (dev's CRDs have been
   there for months) **and fatal on a cold one.** Observed on prod 2026-09-05: `cdi-operator`
   Available for 12h with **no CDI CR** and no `storageprofiles` CRD.

   The step breaks it by applying **only the CR** — the minimum — and leaving everything else to
   Flux. It carries no `${}` tokens, so applying it here cannot diverge from what Flux would render.
   The CRD appeared in 10s.

   **REPAIRED IN THE GRAPH (2026-09-05).** `cdi-storageprofiles.yaml` now lives in its own tier,
   `./platform-config-cdi`, with `dependsOn: platform-config` and `wait: true`. Because `dependsOn` is
   **revision**-gated and `platform-config` runs `wait: true`, the CDI CR must be applied *and healthy*
   — which is precisely the event that creates the `StorageProfile` CRD — before the profiles are
   applied. A cold cluster now orders itself. The `crds` step is retained as a fallback for a cluster
   already stuck in the broken state, and because it is idempotent.

   **The generalisable rule, now recorded in `clusters/README.md`:** a CR whose CRD is created by a
   *controller at runtime* is **second-order** and needs its own pass. `cdi-operator.yaml` registers
   exactly one CRD (`cdis.cdi.kubevirt.io`); `storageprofiles`, `datavolumes`, `datasources` and
   `dataimportcrons` appear only after cdi-operator reconciles the CDI CR. An audit of
   `platform-config` found `StorageProfile` is the only second-order kind there.

   `apps-vms` gained `dependsOn: platform-config-cdi` — load-bearing, not cosmetic:
   `apps/vms/unifi/20-datavolume.yaml` provisions on `storageClassName: longhorn-vm`, and its comment
   previously claimed `platform-config` guaranteed the profiles, which the move would have silently
   falsified.

#### Merge precondition and the one-time transient

`platform-config` has `prune: true`, so on the first reconcile after merge it **deletes** both
StorageProfiles, CDI recreates them within seconds with an empty spec, and `platform-config-cdi`
re-applies the overrides ~30–60s later (kustomize-controller's dependency requeue).

**Measured on dev, so the exposure is known rather than assumed** — comparing our override against
what CDI infers unaided for the same Longhorn provisioner:

| profile | `spec` (ours) | CDI-derived `status` |
|---|---|---|
| `longhorn-vm` (we override) | RWX / Block, `snapshot` | **ReadWriteMany** / Block, `snapshot` |
| `longhorn-static` (no override) | *unset* | **ReadWriteOnce** / Block, `snapshot` |

So **`cloneStrategy` is not at risk** — CDI infers `snapshot` for Longhorn anyway. **`accessModes`
is:** unaided, CDI produces **RWO**. A DataVolume provisioned on `longhorn-vm` *inside the window*
would be stamped RWO/Block and be **permanently non-live-migratable**. Flux can no longer cause that
(`apps-vms` now gates on the new tier); **do not start a VM disk import, clone or DataVolume by hand
during the window.**

- [ ] **Before merging, confirm `flux get kustomization platform-config` is Ready on the target.** If
      it is not, the profiles are pruned and sit at CDI defaults until it recovers — and prod was in
      exactly that state hours earlier. The fallback is the two-step the Cilium move used: merge with
      `platform-config.prune: false`, confirm the ownership label transferred, restore `prune: true` in
      a second commit.
- [ ] After: both profiles carry `spec.cloneStrategy: snapshot`, `longhorn-vm` carries
      `claimPropertySets` RWX/Block, and both are labelled
      `kustomize.toolkit.fluxcd.io/name=platform-config-cdi`.

### C3 — unwedge stuck Helm releases

A release left in `uninstalling` / `pending-*` cannot be advanced by helm-controller: it reports
`StateError` forever, so `platform` (`wait: true`) never goes Ready and all 10 tiers below it stall.
**Dry run by default** — it prints what it would do and changes nothing:

```bash
timeout 600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 unwedge
```

To act (this **deletes** the release's Kubernetes objects and lets Flux install cold). Pre-applied
CRDs live **outside** the Helm release and survive, which is exactly why C2 comes first:

```bash
timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 \
  --confirm-destructive unwedge
```

### C4 — external-dns safe upgrade

**The most dangerous step in the repo, and the ordering is the entire mitigation.** A Helm upgrade
resets `replicas` to the chart default (1) and applies the new args in ONE operation — there is no
window in which to check the result first. If `--txt-owner-id` were missing, the pod starts as owner
`default`, claims dev's TXT registry entries, and under `policy: sync` **deletes the records behind
them**.

So this step **proves the outcome before causing it**: it renders the chart with the *live*
HelmRelease values and asserts `--txt-owner-id=<owner>` appears, aborting if not. Rendering is
deterministic and touches nothing, which converts a race into a precondition. It then snapshots the
Cloudflare zone, upgrades, re-verifies the live args (scaling to 0 and failing if they are wrong),
and diffs the zone.

```bash
timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 externaldns
```

Expected diff: **additions only, all under `prod.home.keithjackson.dev`, all TXT entries owner
`prod`.** Any change to a name outside that subtree, or to a TXT entry owned by another cluster, is a
regression — stop and investigate.

### C5 — resume the tiers, in dependency order

`platform → platform-config → {oauth2-proxy, headlamp, homepage, crafty, apps-wave-0, apps-vms} →
apps-wave-1 → apps-wave-2`. Waits for each to go Ready and **stops at the first that does not** — a
partial, understood bring-up beats a forced green. `velero` is skipped while `S3_BACKUP_ENDPOINT` is
a placeholder (see §Blockers). Long-running; expect it to background.

```bash
timeout 3600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 resume
```

### C6 — verify

```bash
timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 verify
```

### Everything at once

Runs C1→C6 with `unwedge` in **dry-run** mode (no `--confirm-destructive`), so it stops short of the
destructive step by design:

```bash
timeout 3600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
  --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 all
```

---

## VERIFIED WORKING — 2026-09-05, standup complete

| | |
|---|---|
| Kustomizations | **12/13 Ready** (`velero` deliberately suspended — see §Blockers) |
| Pods | **113 total, 0 not Running/Completed** |
| Wildcard cert | `wildcard-home` **Ready**, Let's Encrypt **prod**, `*.prod.home.keithjackson.dev` |
| Gateway VIP | `192.168.2.20` · crafty L4 `192.168.2.22` · unifi L4 `192.168.2.21` |
| Published DNS | **36 records** under `prod.home.keithjackson.dev`, every TXT registry entry `owner=prod` |
| Dev's records | **57, byte-identical to the pre-standup baseline — nothing added, changed or removed** |

End-to-end HTTPS **through the workstation's own resolver — no `--resolve` override, no `-k`** (after
installing the dnsmasq profile below). Every request landed on `192.168.2.20`:

```
prod.home.keithjackson.dev  200   headlamp.  200   keycloak.  302 (login)   nextcloud. 302
gitea.       200            semaphore. 200        grafana.   302 (SSO)      pihole.    302
```

The served certificate is prod's own, confirmed by SNI — **not** dev's wildcard, which is what the
resolver bug previously made it look like:

```
subject = CN=*.prod.home.keithjackson.dev
issuer  = C=US, O=Let's Encrypt, CN=YR1          (LE production)
SAN     = *.prod.home.keithjackson.dev, prod.home.keithjackson.dev
valid   = 2026-09-05 .. 2026-12-04
```

Dev, checked in the same pass, still serves `CN=*.home.keithjackson.dev` — **unchanged**.

> Ignore `curl: (91) No OCSP response received` if you pass `--cert-status`: Let's Encrypt retired
> OCSP responders in 2025, so there is nothing to staple. It is not a certificate fault, and the plain
> requests above all validated normally.

### Workstation DNS — install the prod dnsmasq profile

**CORRECTION (operator, 2026-09-05): this is the workstation's own dnsmasq, NOT pfSense.** Earlier
notes in this repo called it a "pfSense override"; that was wrong and the topology below is why.

`/etc/NetworkManager/dnsmasq.d/10-home.keithjackson.dev.conf` (the dev profile) contains:

```
address=/.home.keithjackson.dev/192.168.2.50
```

dnsmasq's `address=` matches the named domain **and every subdomain**, so that one line also captures
the whole `prod.home.keithjackson.dev` tree and answers it with **dev's** address, while Cloudflare
correctly returns prod's `192.168.2.20`.

**Fix — one command, already authored in this repo:**

```bash
sudo infra/workstation/dnsmasq/install.sh
```

It installs `20-prod.home.keithjackson.dev.conf` and reloads the live resolver. dnsmasq resolves
`address=` by **longest match**, so the prod rules win without editing or reordering the dev profile.
Verified against a throwaway dnsmasq loaded with both profiles:

| query | answer |
|---|---|
| `prod.home.keithjackson.dev`, `gitea.prod…`, `keycloak.prod…` | `192.168.2.20` (Gateway) |
| `unifi.prod.home.keithjackson.dev` | `192.168.2.21` (L4) |
| `crafty-mc.prod.home.keithjackson.dev` | `192.168.2.22` (L4) |
| `gitea.home.keithjackson.dev`, `home.keithjackson.dev` | `192.168.2.50` — **dev, unchanged** |

> **Install to `/etc/NetworkManager/dnsmasq.d/`, never `/etc/dnsmasq.d/`.** The standalone
> `dnsmasq.service` is **inactive**; the live resolver is the one NetworkManager spawns
> (`dns=dnsmasq`) with `--conf-dir=/etc/NetworkManager/dnsmasq.d`. A file in the other directory
> changes nothing and looks like it should. `install.sh` pins the destination for exactly this reason.

**Until it is installed, verify prod only with an explicit resolve** — and note *why* this trap is
convincing: dev serves a `*.home.keithjackson.dev` wildcard cert, which matches
`prod.home.keithjackson.dev` (one label), so the prod **apex returns 200 over valid TLS** while
`keycloak.prod.home.keithjackson.dev` (two labels) fails with a certificate-name error. It reads like
a broken prod certificate. It is the wrong server answering.

```bash
curl -sS --resolve gitea.prod.home.keithjackson.dev:443:192.168.2.20 \
  https://gitea.prod.home.keithjackson.dev/
```

## NODE MAINTENANCE — draining, rebooting, and the cordon that stops self-healing

Two unplanned node events on 2026-09-05/06 (an ungraceful power-down of prod-3, then a memory upgrade
on prod-2) produced one finding worth more than either incident.

### A failed `kubectl drain` still cordons — and on this cluster a cordon stops storage self-healing

`kubectl drain k3s-prod-2` fails on this cluster with the usual two objections (emptyDir pods,
DaemonSet pods) — **but it cordons the node first, and the cordon persists after the failure.**
Longhorn's `disable-scheduling-on-cordoned-node` is **`true`**, so a cordoned node receives no replica
scheduling. Observed consequence: after prod-2's reboot, **all 17 volumes sat `attached/degraded` with
prod-2 holding 2 of 18 replicas and ZERO rebuilds in flight.** Nothing was driving recovery, every pod
was `Running`, and no alert fired. `kubectl uncordon` started the rebuilds immediately; they converged
17/17 healthy in ~6 minutes.

**So: after any drain — successful or not — `kubectl uncordon <node>` is not tidying up, it is what
re-enables storage redundancy.** Check `kubectl get nodes -o custom-columns=N:.metadata.name,\
U:.spec.unschedulable` before believing a green pod list.

### `drain` cannot complete on this cluster, and the two usual flags do not fix it

`--ignore-daemonsets --delete-emptydir-data` clears the two reported objections and then **hangs
indefinitely**: all three `instance-manager-*` PDBs report **`ALLOWED DISRUPTIONS = 0`** and
`node-drain-policy` is `block-if-contains-last-replica`. Longhorn is refusing to let an
instance-manager be evicted while it runs volume engines. That is data protection, not a bug.

**Requesting replica eviction does not rescue it either:** with **3 nodes and 3 replicas** there is no
fourth node to move a replica to, so eviction can never complete. This is structural, not a setting.

Options, in order of preference:

1. **Just reboot it (cordon, reboot, uncordon).** Proven twice on this cluster: Longhorn resyncs on
   return, and `replica-replenishment-wait-interval` is `600`s, so a reboot completed inside ~10
   minutes does not even trigger full rebuilds. This is the normal path for firmware, memory or kernel
   work.
2. **If the node must be genuinely empty**, first scale to 0 the workloads holding engines on it —
   that detaches them, the PDBs go to 1 allowed disruption, and the drain completes. Find them with:
   `kubectl -n longhorn-system get engines.longhorn.io -o json` filtered on `.spec.nodeID`.
3. Changing `node-drain-policy` to `always-allow` is **not** advised at 3 nodes/3 replicas — it trades
   the protection for exactly the data risk the default exists to prevent.

### Check the kube-vip leader before rebooting a server

`kube-vip` holds the API VIP (`192.168.2.9`) via a leader lease, and it is a **DaemonSet** — so
`--ignore-daemonsets` leaves it running and a *drain* does not move the VIP. A **reboot** does: the
lease fails over and `kubectl` blips for a few seconds. Harmless, but do not mistake it for a fault.
Identify the holder: `kubectl -n kube-system logs -l app.kubernetes.io/name=kube-vip-ds | grep -i leader`.

### Post-maintenance checklist

- [ ] `kubectl uncordon <node>` — **and confirm** `.spec.unschedulable` is clear on all three.
- [ ] Longhorn converged: `kubectl -n longhorn-system get volumes.longhorn.io` shows **0 degraded**.
      Rebuilds are serial, so allow minutes, and watch rather than assume.
- [ ] Replica symmetry: each node runs the same count
      (`get replicas.longhorn.io` grouped by `.spec.nodeID` / `.status.currentState`).
- [ ] All three nodes report full memory — prod's HP firmware publishes a corrupt EFI memattr table and
      a wrong kernel silently strands RAM (§0b). `k3s-prod-*` should each show ~`32638256Ki`.
- [ ] Pods: none `Running`-but-not-`Ready`.

> **KNOWN, NOT FIXED — pod distribution does not rebalance.** After prod-2's cordon+reboot the spread
> was **52 / 10 / 50**. Kubernetes does not move running pods back, and for the many SINGLE-REPLICA
> apps here a spread constraint would change nothing (one replica lives on one node either way). It
> self-corrects as pods restart. It is recorded because it concentrates failure: at the time of writing
> **all three CNPG databases run on prod-1**, so losing that node takes Keycloak, Nextcloud and
> Semaphore together. The real fix is database HA (`instances: 2`), which is a resource decision the
> operator owns — see §Blockers.

## Blockers

### 1. Backups: endpoint DECIDED, but prod cannot reach it — **prod has no backup coverage**

**Operator ruling 2026-09-05: "the S3 backup location is the same as Dev."** Prod's
`S3_BACKUP_ENDPOINT` is now `http://192.168.1.9:9000`, and the `TBD-PROD-*` placeholder is retired.

**It is not reachable from prod.** Measured from a throwaway pod **inside the prod cluster** — not
from the workstation, which proves nothing about pod egress:

```
curl http://192.168.1.9:9000/minio/health/live   →  connection timed out after 15s
```

**This is expected topology, not a misconfiguration** (operator, 2026-09-05): the k3s prod cluster is
**not behind pfSense**. It sits on the SOHO router network `192.168.2.0/24`, which is pfSense's
**WAN** side; pfSense *contains* `192.168.1.0/24`, the dev LAN where MinIO lives. Prod reaching
`192.168.1.9` therefore means crossing **into** pfSense from its WAN — which requires a deliberate
pfSense rule or port-forward that does not exist. **Operator decision required**, and worth weighing:
a prod backup that lands inside the dev LAN is arguably not off-cluster DR at all.

Re-test with `standup.sh --s3-endpoint http://192.168.1.9:9000 s3check` — it fails closed and says so.

> **Do not confuse this with the SOHO router's inbound NAT.** That router forwards inbound **443** to
> the pfSense WAN address `192.168.2.102` to serve the **existing, non-k3s production environment**,
> which must stay live on 443 until the k3s cutover is deliberately made. Nothing in this runbook
> touches that NAT, and prod's Gateway on `192.168.2.20:443` does not contend with it.

Until that routing exists, `velero` stays suspended and **`apps-wave-0` still applies GREEN** — the
token *is* defined, so strict substitution succeeds — while every write to that endpoint fails at
**runtime**. A green `apps-wave-0` does not mean backups work. **Treat prod as having no backup
coverage.**

#### Sharing one MinIO is only safe because of two new knobs

Nothing about the backup *paths* was per-environment before 2026-09-05. Enabling prod against dev's
MinIO without these would have been the external-dns incident again, in a worse place:

| | before | after |
|---|---|---|
| Velero | `bucket: velero`, **no prefix**, both clusters run schedules named `daily` | `prefix: ${VELERO_BACKUP_PREFIX}` — dev `""`, prod `prod` |
| CNPG | `destinationPath: s3://cnpg`, serverName defaults to the **Cluster name** | `${CNPG_BACKUP_PATH}` — dev `s3://cnpg`, prod `s3://cnpg/prod` |

- **Velero** syncs backups it finds in its BSL into itself as `Backup` CRs and **garbage-collects them
  when their TTL expires**. Prod would have adopted dev's backups and deleted them on schedule.
- **CNPG is the worse one.** Both environments have `keycloak-db`, `nextcloud-db` and `semaphore-db`,
  so prod's Postgres would archive WAL into the *exact prefix* dev's does. Two servers interleaving
  WAL into one Barman stream **corrupts the chain and destroys PITR** — and dev's restore is
  **proven** (R-STATE-03).

**The base values are dev's current layout on purpose and must stay that way.** Dev's existing
backups live at the bucket root and at `s3://cnpg`; giving dev a prefix would **orphan the very
backups its proven restore depends on** — the same reasoning that keeps dev's
`EXTERNALDNS_TXT_OWNER_ID` at `default`. Prod has no history to preserve, so it takes the nested paths.

> **Delivery note.** Prod reconciles the bootstrap Gitea's `main` at `c8c1d29`. This change sits on a
> later branch that also carries the DNS inversion, which must **not** reach prod (it would move prod
> to the apex and collide with dev). Delivering it needs a `c8c1d29`-based branch carrying only these
> commits. No urgency: velero cannot be enabled until the routing above is fixed.

### 2. Prod is on `prod.home.keithjackson.dev`, not the apex

The operator's target layout is dev under `*.dev.home.keithjackson.dev` and prod on the **apex**.
Prod cannot take the apex until dev vacates it, because both clusters would request a
`letsencrypt-prod` wildcard for the *same name set* and race each other's `_acme-challenge` TXT
records, so neither reliably solves DNS-01.

**This does not block phase C.** Prod on its own subtree with `txtOwnerId: prod` collides with
nothing, and standing it up first is strictly better: it proves the whole stack before the flip, so
the cutover becomes a DNS/TLS change on a known-good cluster instead of a bring-up and a cutover at
once. The apex move is `infra/PROD-PROVISIONING.md` §10a, sequence **D** (dev vacates) then **P**.

> **THE APEX FLIP IS ALSO THE LEGACY-PROD CUTOVER — they are the same event.** The SOHO router NATs
> inbound **443** to the pfSense WAN address `192.168.2.102`, serving the **existing, non-k3s
> production environment**, which the operator requires to stay live until the switch is deliberately
> made. So §10a sequence **P** is not merely a DNS/TLS change on the k3s side: taking
> `home.keithjackson.dev` means public traffic must also stop resolving to / being forwarded to the
> legacy stack. Sequence **P** as written does **not** cover repointing that NAT, and the two must be
> planned together or the apex will resolve to k3s while inbound 443 still lands on the legacy box.
>
> Nothing in phase C touches it: prod's Gateway is `192.168.2.20:443`, a different address from the
> NAT target, and prod's wildcard cert is issued by **DNS-01**, which needs no inbound 443 at all.

---

## Not covered here

- **Dev's migration off the apex** — §10a sequence D. Touches Keycloak's issuer, ~12 OIDC client
  redirect URIs, and a rolling k3s restart that invalidates every OIDC kubeconfig. Operator-present.
- **Prod's GitOps cutover** (P5) from the workstation bootstrap Gitea to prod's own in-cluster Gitea.
  `clusters/prod/kustomization.yaml` owns that procedure.
- **Six L2 conformance tests hardcode the apex** and must be parameterized per environment before
  §9's "suite passes green against the prod cluster" can be claimed.