# Prod cluster standup — the complete command sequence **Every command below is copy-pasteable and runs from the repo root**, through the pinned dev container. Nothing here needs a command invented on the spot; that is the point. If a step is not in this file it is not part of standup. State as of **2026-09-06**: phases **A**, **B** and **C** are DONE. The cluster is live — k3s v1.36.4 HA (3 etcd members), kube-vip, Cilium 1.20.1, Longhorn — with **12/13 Flux tiers Ready** (`velero` deliberately suspended), **112 pods Running**, an issued Let's Encrypt wildcard, and **all three nodes addressing their full 32 GB** (node 2's DIMM upgrade completed 2026-09-06 and is confirmed at ~`32638256Ki`, matching the other two to within 4 KiB — the §0b memattr bug is not biting on the backports kernel). Storage survived two unplanned node events (an ungraceful power-down of prod-3, then prod-2's memory upgrade) and converged to **17/17 volumes healthy, replicas symmetric across all three nodes** — though only after a leftover cordon was cleared; read [§Node maintenance](#node-maintenance--draining-rebooting-and-the-cordon-that-stops-self-healing) before doing node work, that cordon silently prevents Longhorn from rebuilding. What remains is in [§Blockers](#blockers), and every item there needs an operator decision: no backup coverage (R2 credentials), prod still on `prod.home.` rather than the apex, and — found 2026-09-05 — **prod's UniFi VM has no disk** because Kyverno blocks CDI's generated importer pod. > **The one guard that matters.** Dev is LIVE and shares the `keithjackson.dev` Cloudflare zone with > prod. On 2026-09-04 prod deleted every one of dev's public DNS records. `bootstrap/standup.sh` > therefore **requires** `--expect-server` and compares it against the live API server before running > anything. There is no override flag. Do not add one. ## Constants | | | |---|---| | kube-API (kube-vip VIP) | `https://192.168.2.9:6443` | | Gateway / ingress VIP | `192.168.2.20` | | Nodes | `k3s-prod-1/2/3` — `192.168.2.10 / .11 / .12` | | Break-glass kubeconfig | `infra/ansible/.cache/prod.kubeconfig` (cert-based; survives OIDC/DNS failure) | | GitOps source (interim) | `http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git` branch `main` | | `BASE_DOMAIN` | `prod.home.keithjackson.dev` — **not** the apex; see §Blockers | Container paths: the repo is mounted at `/work`, so kubeconfig paths passed to the script are **repo-relative** (`infra/ansible/.cache/prod.kubeconfig`). --- ## Phase A — bare metal to a joined cluster (Ansible) Prerequisites that are **not** commands: firmware/BIOS (§1), disk layout (§2), OS install (§3), network (§4), and SSH + the vaulted `ansible_become_password` in `inventory/group_vars/k3s_prod/vault.yml` (§5). Ansible cannot escalate on prod without that vault file. See `infra/PROD-PROVISIONING.md`. **Install the backports kernel first.** All three prod nodes need ≥6.13 because HP's R21 firmware publishes a corrupt EFI Memory Attributes Table and Linux 6.12's `efi_memattr_init()` does not validate it, reserving everything above the table (fixed upstream in 6.13 by `8fbe4c49c0cc`, absent from `linux-6.12.y`). Prod runs `7.1.8+deb13-amd64` from trixie-backports. ```bash timeout 3600 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \ ansible-playbook /work/infra/ansible/playbooks/node-baseline.yml \ -i /work/infra/ansible/inventory/hosts.ini -e k3s_hosts=k3s_prod --tags kernel ``` Then the full chain — baseline, k3s HA, Cilium, verify, Flux hand-off. **All three `-e` flags are required**: a play's `hosts:` is templated before any host exists, so `group_vars` cannot retarget it. ```bash timeout 3600 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \ ansible-playbook /work/infra/ansible/playbooks/site.yml \ -i /work/infra/ansible/inventory/hosts.ini \ -e k3s_hosts=k3s_prod -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary ``` kube-vip is a separate play (not imported by `site.yml`): ```bash timeout 1800 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \ ansible-playbook /work/infra/ansible/playbooks/kube-vip.yml \ -i /work/infra/ansible/inventory/hosts.ini -e k3s_hosts=k3s_prod ``` Hardware health (read-only; `node_health` asserts the memattr fix took and that all RAM is addressed). > **The flag is `node_health_hosts`, NOT `k3s_hosts`.** Diagnostic plays deliberately keep their own > target var so they stay independently pointable while a bring-up run is scoped elsewhere > (`node-health.yml` → `node_health_hosts`, `memattr-survey.yml` → `memattr_survey_hosts`, > `vpn.egress.gateway.yml` → `vpn_egress_gateway_hosts`). The consequence is a **silent** one: passing > `k3s_hosts=k3s_prod` here does not error, it falls through to the `k3s_servers` default and runs > against **DEV's three nodes**. Verified — this exact mistake was made while writing this file and > caught only by `--list-hosts`. **Always dry-run the target first:** > > ```bash > timeout 200 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \ > ansible-playbook /work/infra/ansible/playbooks/node-health.yml \ > -i /work/infra/ansible/inventory/hosts.ini -e node_health_hosts=k3s_prod --list-hosts > ``` ```bash timeout 900 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \ ansible-playbook /work/infra/ansible/playbooks/node-health.yml \ -i /work/infra/ansible/inventory/hosts.ini -e node_health_hosts=k3s_prod ``` ## Phase B — Flux bootstrap `flux-bootstrap.yml` (last play of `site.yml`) shells to `bootstrap/flux/bootstrap.sh`, the single documented entrypoint. That script now runs the CRD pre-apply as step **[2/4]**, so a cold bootstrap no longer strands `platform`. The **age private key Secret must exist first** — bootstrap verifies it and refuses to continue rather than silently skipping decryption (`bootstrap/sops/age-key-secret.md`). ## Phase C — bring the reconciliation graph up One script, six steps, each independently runnable and idempotent. **Run them in this order.** ```bash K=infra/ansible/.cache/prod.kubeconfig S=https://192.168.2.9:6443 ``` ### C1 — preflight (read-only, changes nothing) Asserts the guards that prevent a repeat of the DNS incident and reports every tier's state. **Fails closed** if `EXTERNALDNS_TXT_OWNER_ID` is empty, still the base sentinel, or `default` (dev's owner). ```bash timeout 300 infra/devcontainer/dev.sh run bootstrap/standup.sh \ --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 preflight ``` ### C2 — pre-apply chart CRDs, and break the CDI cold-start deadlock (idempotent) ```bash timeout 600 infra/devcontainer/dev.sh run bootstrap/standup.sh \ --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 crds ``` Two cold-start problems, same root cause: **Flux server-side dry-runs every object in a Kustomization before applying any of them, so one unknown `kind` aborts the entire tier.** 1. **Kyverno** — its ~5.6 MiB CRD bundle exceeds kubectl's 256 KiB annotation limit, so the chart sets `crds.install: false` and nothing else installs them. `bootstrap/flux/preapply-crds.sh` applies them server-side, deriving the version from the HelmRelease Flux reconciles (no second pin). 2. **CDI — a true deadlock.** `platform-config` applies the CDI CR *and* `platform/kubevirt/cdi-storageprofiles.yaml` in the **same pass**. But `StorageProfile`'s CRD does not exist until `cdi-operator` has reconciled that CR. So the dry-run fails on the unknown kind → the CR is never applied → the CRD is never created → `platform-config` can never converge, and every tier below it stays blocked. **Invisible on an established cluster** (dev's CRDs have been there for months) **and fatal on a cold one.** Observed on prod 2026-09-05: `cdi-operator` Available for 12h with **no CDI CR** and no `storageprofiles` CRD. The step breaks it by applying **only the CR** — the minimum — and leaving everything else to Flux. It carries no `${}` tokens, so applying it here cannot diverge from what Flux would render. The CRD appeared in 10s. **REPAIRED IN THE GRAPH (2026-09-05).** `cdi-storageprofiles.yaml` now lives in its own tier, `./platform-config-cdi`, with `dependsOn: platform-config` and `wait: true`. Because `dependsOn` is **revision**-gated and `platform-config` runs `wait: true`, the CDI CR must be applied *and healthy* — which is precisely the event that creates the `StorageProfile` CRD — before the profiles are applied. A cold cluster now orders itself. The `crds` step is retained as a fallback for a cluster already stuck in the broken state, and because it is idempotent. **The generalisable rule, now recorded in `clusters/README.md`:** a CR whose CRD is created by a *controller at runtime* is **second-order** and needs its own pass. `cdi-operator.yaml` registers exactly one CRD (`cdis.cdi.kubevirt.io`); `storageprofiles`, `datavolumes`, `datasources` and `dataimportcrons` appear only after cdi-operator reconciles the CDI CR. An audit of `platform-config` found `StorageProfile` is the only second-order kind there. `apps-vms` gained `dependsOn: platform-config-cdi` — load-bearing, not cosmetic: `apps/vms/unifi/20-datavolume.yaml` provisions on `storageClassName: longhorn-vm`, and its comment previously claimed `platform-config` guaranteed the profiles, which the move would have silently falsified. #### Merge precondition and the one-time transient `platform-config` has `prune: true`, so on the first reconcile after merge it **deletes** both StorageProfiles, CDI recreates them within seconds with an empty spec, and `platform-config-cdi` re-applies the overrides ~30–60s later (kustomize-controller's dependency requeue). **Measured on dev, so the exposure is known rather than assumed** — comparing our override against what CDI infers unaided for the same Longhorn provisioner: | profile | `spec` (ours) | CDI-derived `status` | |---|---|---| | `longhorn-vm` (we override) | RWX / Block, `snapshot` | **ReadWriteMany** / Block, `snapshot` | | `longhorn-static` (no override) | *unset* | **ReadWriteOnce** / Block, `snapshot` | So **`cloneStrategy` is not at risk** — CDI infers `snapshot` for Longhorn anyway. **`accessModes` is:** unaided, CDI produces **RWO**. A DataVolume provisioned on `longhorn-vm` *inside the window* would be stamped RWO/Block and be **permanently non-live-migratable**. Flux can no longer cause that (`apps-vms` now gates on the new tier); **do not start a VM disk import, clone or DataVolume by hand during the window.** - [ ] **Before merging, confirm `flux get kustomization platform-config` is Ready on the target.** If it is not, the profiles are pruned and sit at CDI defaults until it recovers — and prod was in exactly that state hours earlier. The fallback is the two-step the Cilium move used: merge with `platform-config.prune: false`, confirm the ownership label transferred, restore `prune: true` in a second commit. - [ ] After: both profiles carry `spec.cloneStrategy: snapshot`, `longhorn-vm` carries `claimPropertySets` RWX/Block, and both are labelled `kustomize.toolkit.fluxcd.io/name=platform-config-cdi`. ### C3 — unwedge stuck Helm releases A release left in `uninstalling` / `pending-*` cannot be advanced by helm-controller: it reports `StateError` forever, so `platform` (`wait: true`) never goes Ready and all 10 tiers below it stall. **Dry run by default** — it prints what it would do and changes nothing: ```bash timeout 600 infra/devcontainer/dev.sh run bootstrap/standup.sh \ --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 unwedge ``` To act (this **deletes** the release's Kubernetes objects and lets Flux install cold). Pre-applied CRDs live **outside** the Helm release and survive, which is exactly why C2 comes first: ```bash timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \ --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 \ --confirm-destructive unwedge ``` ### C4 — external-dns safe upgrade **The most dangerous step in the repo, and the ordering is the entire mitigation.** A Helm upgrade resets `replicas` to the chart default (1) and applies the new args in ONE operation — there is no window in which to check the result first. If `--txt-owner-id` were missing, the pod starts as owner `default`, claims dev's TXT registry entries, and under `policy: sync` **deletes the records behind them**. So this step **proves the outcome before causing it**: it renders the chart with the *live* HelmRelease values and asserts `--txt-owner-id=` appears, aborting if not. Rendering is deterministic and touches nothing, which converts a race into a precondition. It then snapshots the Cloudflare zone, upgrades, re-verifies the live args (scaling to 0 and failing if they are wrong), and diffs the zone. ```bash timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \ --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 externaldns ``` Expected diff: **additions only, all under `prod.home.keithjackson.dev`, all TXT entries owner `prod`.** Any change to a name outside that subtree, or to a TXT entry owned by another cluster, is a regression — stop and investigate. ### C5 — resume the tiers, in dependency order `platform → platform-config → {oauth2-proxy, headlamp, homepage, crafty, apps-wave-0, apps-vms} → apps-wave-1 → apps-wave-2`. Waits for each to go Ready and **stops at the first that does not** — a partial, understood bring-up beats a forced green. `velero` is skipped while `S3_BACKUP_ENDPOINT` is a placeholder (see §Blockers). Long-running; expect it to background. ```bash timeout 3600 infra/devcontainer/dev.sh run bootstrap/standup.sh \ --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 resume ``` ### C6 — verify ```bash timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \ --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 verify ``` ### Everything at once Runs C1→C6 with `unwedge` in **dry-run** mode (no `--confirm-destructive`), so it stops short of the destructive step by design: ```bash timeout 3600 infra/devcontainer/dev.sh run bootstrap/standup.sh \ --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 all ``` --- ## VERIFIED WORKING — 2026-09-05, standup complete | | | |---|---| | Kustomizations | **12/13 Ready** (`velero` deliberately suspended — see §Blockers) | | Pods | **113 total, 0 not Running/Completed** | | Wildcard cert | `wildcard-home` **Ready**, Let's Encrypt **prod**, `*.prod.home.keithjackson.dev` | | Gateway VIP | `192.168.2.20` · crafty L4 `192.168.2.22` · unifi L4 `192.168.2.21` | | Published DNS | **36 records** under `prod.home.keithjackson.dev`, every TXT registry entry `owner=prod` | | Dev's records | **57, byte-identical to the pre-standup baseline — nothing added, changed or removed** | End-to-end HTTPS **through the workstation's own resolver — no `--resolve` override, no `-k`** (after installing the dnsmasq profile below). Every request landed on `192.168.2.20`: ``` prod.home.keithjackson.dev 200 headlamp. 200 keycloak. 302 (login) nextcloud. 302 gitea. 200 semaphore. 200 grafana. 302 (SSO) pihole. 302 ``` The served certificate is prod's own, confirmed by SNI — **not** dev's wildcard, which is what the resolver bug previously made it look like: ``` subject = CN=*.prod.home.keithjackson.dev issuer = C=US, O=Let's Encrypt, CN=YR1 (LE production) SAN = *.prod.home.keithjackson.dev, prod.home.keithjackson.dev valid = 2026-09-05 .. 2026-12-04 ``` Dev, checked in the same pass, still serves `CN=*.home.keithjackson.dev` — **unchanged**. > Ignore `curl: (91) No OCSP response received` if you pass `--cert-status`: Let's Encrypt retired > OCSP responders in 2025, so there is nothing to staple. It is not a certificate fault, and the plain > requests above all validated normally. ### Workstation DNS — install the prod dnsmasq profile **CORRECTION (operator, 2026-09-05): this is the workstation's own dnsmasq, NOT pfSense.** Earlier notes in this repo called it a "pfSense override"; that was wrong and the topology below is why. `/etc/NetworkManager/dnsmasq.d/10-home.keithjackson.dev.conf` (the dev profile) contains: ``` address=/.home.keithjackson.dev/192.168.2.50 ``` dnsmasq's `address=` matches the named domain **and every subdomain**, so that one line also captures the whole `prod.home.keithjackson.dev` tree and answers it with **dev's** address, while Cloudflare correctly returns prod's `192.168.2.20`. **Fix — one command, already authored in this repo:** ```bash sudo infra/workstation/dnsmasq/install.sh ``` It installs `20-prod.home.keithjackson.dev.conf` and reloads the live resolver. dnsmasq resolves `address=` by **longest match**, so the prod rules win without editing or reordering the dev profile. Verified against a throwaway dnsmasq loaded with both profiles: | query | answer | |---|---| | `prod.home.keithjackson.dev`, `gitea.prod…`, `keycloak.prod…` | `192.168.2.20` (Gateway) | | `unifi.prod.home.keithjackson.dev` | `192.168.2.21` (L4) | | `crafty-mc.prod.home.keithjackson.dev` | `192.168.2.22` (L4) | | `gitea.home.keithjackson.dev`, `home.keithjackson.dev` | `192.168.2.50` — **dev, unchanged** | > **Install to `/etc/NetworkManager/dnsmasq.d/`, never `/etc/dnsmasq.d/`.** The standalone > `dnsmasq.service` is **inactive**; the live resolver is the one NetworkManager spawns > (`dns=dnsmasq`) with `--conf-dir=/etc/NetworkManager/dnsmasq.d`. A file in the other directory > changes nothing and looks like it should. `install.sh` pins the destination for exactly this reason. **Until it is installed, verify prod only with an explicit resolve** — and note *why* this trap is convincing: dev serves a `*.home.keithjackson.dev` wildcard cert, which matches `prod.home.keithjackson.dev` (one label), so the prod **apex returns 200 over valid TLS** while `keycloak.prod.home.keithjackson.dev` (two labels) fails with a certificate-name error. It reads like a broken prod certificate. It is the wrong server answering. ```bash curl -sS --resolve gitea.prod.home.keithjackson.dev:443:192.168.2.20 \ https://gitea.prod.home.keithjackson.dev/ ``` ## NODE MAINTENANCE — draining, rebooting, and the cordon that stops self-healing Two unplanned node events on 2026-09-05/06 (an ungraceful power-down of prod-3, then a memory upgrade on prod-2) produced one finding worth more than either incident. ### A failed `kubectl drain` still cordons — and on this cluster a cordon stops storage self-healing `kubectl drain k3s-prod-2` fails on this cluster with the usual two objections (emptyDir pods, DaemonSet pods) — **but it cordons the node first, and the cordon persists after the failure.** Longhorn's `disable-scheduling-on-cordoned-node` is **`true`**, so a cordoned node receives no replica scheduling. Observed consequence: after prod-2's reboot, **all 17 volumes sat `attached/degraded` with prod-2 holding 2 of 18 replicas and ZERO rebuilds in flight.** Nothing was driving recovery, every pod was `Running`, and no alert fired. `kubectl uncordon` started the rebuilds immediately; they converged 17/17 healthy in ~6 minutes. **So: after any drain — successful or not — `kubectl uncordon ` is not tidying up, it is what re-enables storage redundancy.** Check `kubectl get nodes -o custom-columns=N:.metadata.name,\ U:.spec.unschedulable` before believing a green pod list. ### `drain` cannot complete on this cluster, and the two usual flags do not fix it `--ignore-daemonsets --delete-emptydir-data` clears the two reported objections and then **hangs indefinitely**: all three `instance-manager-*` PDBs report **`ALLOWED DISRUPTIONS = 0`** and `node-drain-policy` is `block-if-contains-last-replica`. Longhorn is refusing to let an instance-manager be evicted while it runs volume engines. That is data protection, not a bug. **Requesting replica eviction does not rescue it either:** with **3 nodes and 3 replicas** there is no fourth node to move a replica to, so eviction can never complete. This is structural, not a setting. Options, in order of preference: 1. **Just reboot it (cordon, reboot, uncordon).** Proven twice on this cluster: Longhorn resyncs on return, and `replica-replenishment-wait-interval` is `600`s, so a reboot completed inside ~10 minutes does not even trigger full rebuilds. This is the normal path for firmware, memory or kernel work. 2. **If the node must be genuinely empty**, first scale to 0 the workloads holding engines on it — that detaches them, the PDBs go to 1 allowed disruption, and the drain completes. Find them with: `kubectl -n longhorn-system get engines.longhorn.io -o json` filtered on `.spec.nodeID`. 3. Changing `node-drain-policy` to `always-allow` is **not** advised at 3 nodes/3 replicas — it trades the protection for exactly the data risk the default exists to prevent. ### Check the kube-vip leader before rebooting a server `kube-vip` holds the API VIP (`192.168.2.9`) via a leader lease, and it is a **DaemonSet** — so `--ignore-daemonsets` leaves it running and a *drain* does not move the VIP. A **reboot** does: the lease fails over and `kubectl` blips for a few seconds. Harmless, but do not mistake it for a fault. Identify the holder: `kubectl -n kube-system logs -l app.kubernetes.io/name=kube-vip-ds | grep -i leader`. ### Post-maintenance checklist - [ ] `kubectl uncordon ` — **and confirm** `.spec.unschedulable` is clear on all three. - [ ] Longhorn converged: `kubectl -n longhorn-system get volumes.longhorn.io` shows **0 degraded**. Rebuilds are serial, so allow minutes, and watch rather than assume. - [ ] Replica symmetry: each node runs the same count (`get replicas.longhorn.io` grouped by `.spec.nodeID` / `.status.currentState`). - [ ] All three nodes report full memory — prod's HP firmware publishes a corrupt EFI memattr table and a wrong kernel silently strands RAM (§0b). `k3s-prod-*` should each show ~`32638256Ki`. - [ ] Pods: none `Running`-but-not-`Ready`. > **KNOWN, NOT FIXED — pod distribution does not rebalance.** After prod-2's cordon+reboot the spread > was **52 / 10 / 50**. Kubernetes does not move running pods back, and for the many SINGLE-REPLICA > apps here a spread constraint would change nothing (one replica lives on one node either way). It > self-corrects as pods restart. It is recorded because it concentrates failure: at the time of writing > **all three CNPG databases run on prod-1**, so losing that node takes Keycloak, Nextcloud and > Semaphore together. The real fix is database HA (`instances: 2`), which is a resource decision the > operator owns — see §Blockers. ## Blockers ### 1. Backups: endpoint DECIDED, but prod cannot reach it — **prod has no backup coverage** **Operator ruling 2026-09-05: "the S3 backup location is the same as Dev."** Prod's `S3_BACKUP_ENDPOINT` is now `http://192.168.1.9:9000`, and the `TBD-PROD-*` placeholder is retired. **It is not reachable from prod.** Measured from a throwaway pod **inside the prod cluster** — not from the workstation, which proves nothing about pod egress: ``` curl http://192.168.1.9:9000/minio/health/live → connection timed out after 15s ``` **This is expected topology, not a misconfiguration** (operator, 2026-09-05): the k3s prod cluster is **not behind pfSense**. It sits on the SOHO router network `192.168.2.0/24`, which is pfSense's **WAN** side; pfSense *contains* `192.168.1.0/24`, the dev LAN where MinIO lives. Prod reaching `192.168.1.9` therefore means crossing **into** pfSense from its WAN — which requires a deliberate pfSense rule or port-forward that does not exist. **Operator decision required**, and worth weighing: a prod backup that lands inside the dev LAN is arguably not off-cluster DR at all. Re-test with `standup.sh --s3-endpoint http://192.168.1.9:9000 s3check` — it fails closed and says so. > **Do not confuse this with the SOHO router's inbound NAT.** That router forwards inbound **443** to > the pfSense WAN address `192.168.2.102` to serve the **existing, non-k3s production environment**, > which must stay live on 443 until the k3s cutover is deliberately made. Nothing in this runbook > touches that NAT, and prod's Gateway on `192.168.2.20:443` does not contend with it. Until that routing exists, `velero` stays suspended and **`apps-wave-0` still applies GREEN** — the token *is* defined, so strict substitution succeeds — while every write to that endpoint fails at **runtime**. A green `apps-wave-0` does not mean backups work. **Treat prod as having no backup coverage.** #### Sharing one MinIO is only safe because of two new knobs Nothing about the backup *paths* was per-environment before 2026-09-05. Enabling prod against dev's MinIO without these would have been the external-dns incident again, in a worse place: | | before | after | |---|---|---| | Velero | `bucket: velero`, **no prefix**, both clusters run schedules named `daily` | `prefix: ${VELERO_BACKUP_PREFIX}` — dev `""`, prod `prod` | | CNPG | `destinationPath: s3://cnpg`, serverName defaults to the **Cluster name** | `${CNPG_BACKUP_PATH}` — dev `s3://cnpg`, prod `s3://cnpg/prod` | - **Velero** syncs backups it finds in its BSL into itself as `Backup` CRs and **garbage-collects them when their TTL expires**. Prod would have adopted dev's backups and deleted them on schedule. - **CNPG is the worse one.** Both environments have `keycloak-db`, `nextcloud-db` and `semaphore-db`, so prod's Postgres would archive WAL into the *exact prefix* dev's does. Two servers interleaving WAL into one Barman stream **corrupts the chain and destroys PITR** — and dev's restore is **proven** (R-STATE-03). **The base values are dev's current layout on purpose and must stay that way.** Dev's existing backups live at the bucket root and at `s3://cnpg`; giving dev a prefix would **orphan the very backups its proven restore depends on** — the same reasoning that keeps dev's `EXTERNALDNS_TXT_OWNER_ID` at `default`. Prod has no history to preserve, so it takes the nested paths. > **Delivery note.** Prod reconciles the bootstrap Gitea's `main` at `c8c1d29`. This change sits on a > later branch that also carries the DNS inversion, which must **not** reach prod (it would move prod > to the apex and collide with dev). Delivering it needs a `c8c1d29`-based branch carrying only these > commits. No urgency: velero cannot be enabled until the routing above is fixed. ### 2. Prod is on `prod.home.keithjackson.dev`, not the apex The operator's target layout is dev under `*.dev.home.keithjackson.dev` and prod on the **apex**. Prod cannot take the apex until dev vacates it, because both clusters would request a `letsencrypt-prod` wildcard for the *same name set* and race each other's `_acme-challenge` TXT records, so neither reliably solves DNS-01. **This does not block phase C.** Prod on its own subtree with `txtOwnerId: prod` collides with nothing, and standing it up first is strictly better: it proves the whole stack before the flip, so the cutover becomes a DNS/TLS change on a known-good cluster instead of a bring-up and a cutover at once. The apex move is `infra/PROD-PROVISIONING.md` §10a, sequence **D** (dev vacates) then **P**. > **THE APEX FLIP IS ALSO THE LEGACY-PROD CUTOVER — they are the same event.** The SOHO router NATs > inbound **443** to the pfSense WAN address `192.168.2.102`, serving the **existing, non-k3s > production environment**, which the operator requires to stay live until the switch is deliberately > made. So §10a sequence **P** is not merely a DNS/TLS change on the k3s side: taking > `home.keithjackson.dev` means public traffic must also stop resolving to / being forwarded to the > legacy stack. Sequence **P** as written does **not** cover repointing that NAT, and the two must be > planned together or the apex will resolve to k3s while inbound 443 still lands on the legacy box. > > Nothing in phase C touches it: prod's Gateway is `192.168.2.20:443`, a different address from the > NAT target, and prod's wildcard cert is issued by **DNS-01**, which needs no inbound 443 at all. --- ## Not covered here - **Dev's migration off the apex** — §10a sequence D. Touches Keycloak's issuer, ~12 OIDC client redirect URIs, and a rolling k3s restart that invalidates every OIDC kubeconfig. Operator-present. - **Prod's GitOps cutover** (P5) from the workstation bootstrap Gitea to prod's own in-cluster Gitea. `clusters/prod/kustomization.yaml` owns that procedure. - **Six L2 conformance tests hardcode the apex** and must be parameterized per environment before §9's "suite passes green against the prod cluster" can be claimed.