jakman85andClaude Opus 5
765ad032db
`platform-config` applied the CDI CR *and* cdi-storageprofiles.yaml in ONE pass.
StorageProfile's CRD does not exist until cdi-operator reconciles that CR, and
Flux server-side dry-runs every object before applying any — so the unknown kind
aborted the tier, the CR was never applied, the CRD was never created, and the
tier could NEVER converge. Observed on prod: cdi-operator Available 12h, no CDI
CR, no storageprofiles CRD, every downstream tier blocked.
The profiles now live in their own tier ./platform-config-cdi with
dependsOn: platform-config and wait: true. Because dependsOn is REVISION-gated
and platform-config runs wait: true, the CDI CR must be applied AND healthy —
exactly the event that creates the CRD — before the profiles are applied. Cold
clusters order themselves. `standup.sh crds` is retained as a fallback for
clusters already stuck, and because it is idempotent.
THE GENERALISABLE RULE (recorded in clusters/README.md): a CR whose CRD is
created by a CONTROLLER AT RUNTIME is second-order and needs its own pass.
cdi-operator.yaml registers exactly ONE CRD (cdis.cdi.kubevirt.io); storageprofiles,
datavolumes, datasources and dataimportcrons appear only after it reconciles the
CDI CR. An audit found StorageProfile is the only second-order kind in
platform-config — KubeVirt, CDI, KeycloakRealmImport, CNPG Cluster,
VolumeSnapshotClass, ClusterIssuer, PrometheusRule, ServiceMonitor and the Cilium
L2 CRs all have their CRD in git or in a chart.
Moving the CDI CR up into `platform` was rejected: that would put CDI's health on
the tier the CNI gates and everything waits on — the mistake the oauth2-proxy and
headlamp splits exist to undo.
apps-vms gains dependsOn: platform-config-cdi. LOAD-BEARING, not cosmetic:
apps/vms/unifi/20-datavolume.yaml provisions on storageClassName: longhorn-vm and
its comment claimed platform-config guaranteed the profiles — which this move
would have silently falsified.
THE PRUNE TRANSIENT, MEASURED RATHER THAN ASSUMED. platform-config has
prune: true, so it deletes both profiles once, CDI recreates them with empty spec,
and the new tier re-applies the overrides ~30-60s later. Compared our override
against CDI's unaided inference for the same Longhorn provisioner on dev:
longhorn-vm (we override) spec RWX/Block -> status ReadWriteMany/Block
longhorn-static (no override) spec unset -> status ReadWriteOnce/Block
Both infer cloneStrategy: snapshot. So cloneStrategy is NOT at risk; accessModes
IS — unaided CDI gives RWO. A DataVolume provisioned on longhorn-vm inside the
window would be stamped RWO/Block and be PERMANENTLY non-live-migratable. Flux can
no longer cause it (apps-vms gates on the new tier); a human must not do it by hand.
Corroborating evidence that the recreate happens at all: dev has 5 StorageClasses
and 5 StorageProfiles, three with fluxOwner=NONE, spec unset and status populated
— CDI creating and inferring them with no Flux involvement.
MERGE PRECONDITION: `flux get kustomization platform-config` must be Ready on the
target first. If it is not, the profiles are pruned and sit at CDI defaults until
it recovers — and prod was in that state hours earlier. Fallback is the two-step
the Cilium move used (merge with prune: false, verify the ownership label moved,
restore prune: true).
substituteFrom: cluster-vars is declared even though the manifest has no ${TOKEN},
deliberately: conformance/tests/l0-static/flux-postbuild-substitution only audits
paths whose Kustomization declares it, so omitting it would move the file OUT of
the audited set — a blind spot for a future literal-token ship. Gate confirms it
stayed in (path=./platform-config-cdi files=2 defined=0 hazards=0). No SOPS
decryption stanza: the moved file is two plain CRs, so no overlay change was needed.
VERIFIED INDEPENDENTLY OF THE AGENT: platform-config renders 0 StorageProfile
while keeping the CDI and KubeVirt CRs; platform-config-cdi renders exactly the
two profiles; the graph is byte-identical across dev and prod with apps-vms
carrying both edges and no cycle. Full CI gate PASS.
KNOWN FOLLOW-UP: bootstrap/standup.sh TIER_ORDER does not yet include
platform-config-cdi, so `resume` will not wait on it. Another agent holds that
file; the PM will add it.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
441 lines
23 KiB
Markdown
441 lines
23 KiB
Markdown
# Prod cluster standup — the complete command sequence
|
||
|
||
**Every command below is copy-pasteable and runs from the repo root**, through the pinned dev
|
||
container. Nothing here needs a command invented on the spot; that is the point. If a step is not in
|
||
this file it is not part of standup.
|
||
|
||
State as of **2026-09-05**: phases **A** and **B** are DONE on the current hardware. The cluster is
|
||
live (k3s v1.36.4 HA, 3 etcd members, kube-vip, Cilium 1.20.1, Longhorn) with `platform` reconciling
|
||
and **8/9 HelmReleases Ready, 76/76 pods Running**. Outstanding work is **phase C**, blocked on the
|
||
two items in [§Blockers](#blockers).
|
||
|
||
> **The one guard that matters.** Dev is LIVE and shares the `keithjackson.dev` Cloudflare zone with
|
||
> prod. On 2026-09-04 prod deleted every one of dev's public DNS records. `bootstrap/standup.sh`
|
||
> therefore **requires** `--expect-server` and compares it against the live API server before running
|
||
> anything. There is no override flag. Do not add one.
|
||
|
||
## Constants
|
||
|
||
| | |
|
||
|---|---|
|
||
| kube-API (kube-vip VIP) | `https://192.168.2.9:6443` |
|
||
| Gateway / ingress VIP | `192.168.2.20` |
|
||
| Nodes | `k3s-prod-1/2/3` — `192.168.2.10 / .11 / .12` |
|
||
| Break-glass kubeconfig | `infra/ansible/.cache/prod.kubeconfig` (cert-based; survives OIDC/DNS failure) |
|
||
| GitOps source (interim) | `http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git` branch `main` |
|
||
| `BASE_DOMAIN` | `prod.home.keithjackson.dev` — **not** the apex; see §Blockers |
|
||
|
||
Container paths: the repo is mounted at `/work`, so kubeconfig paths passed to the script are
|
||
**repo-relative** (`infra/ansible/.cache/prod.kubeconfig`).
|
||
|
||
---
|
||
|
||
## Phase A — bare metal to a joined cluster (Ansible)
|
||
|
||
Prerequisites that are **not** commands: firmware/BIOS (§1), disk layout (§2), OS install (§3),
|
||
network (§4), and SSH + the vaulted `ansible_become_password` in
|
||
`inventory/group_vars/k3s_prod/vault.yml` (§5). Ansible cannot escalate on prod without that vault
|
||
file. See `infra/PROD-PROVISIONING.md`.
|
||
|
||
**Install the backports kernel first.** All three prod nodes need ≥6.13 because HP's R21 firmware
|
||
publishes a corrupt EFI Memory Attributes Table and Linux 6.12's `efi_memattr_init()` does not
|
||
validate it, reserving everything above the table (fixed upstream in 6.13 by `8fbe4c49c0cc`, absent
|
||
from `linux-6.12.y`). Prod runs `7.1.8+deb13-amd64` from trixie-backports.
|
||
|
||
```bash
|
||
timeout 3600 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
|
||
ansible-playbook /work/infra/ansible/playbooks/node-baseline.yml \
|
||
-i /work/infra/ansible/inventory/hosts.ini -e k3s_hosts=k3s_prod --tags kernel
|
||
```
|
||
|
||
Then the full chain — baseline, k3s HA, Cilium, verify, Flux hand-off. **All three `-e` flags are
|
||
required**: a play's `hosts:` is templated before any host exists, so `group_vars` cannot retarget it.
|
||
|
||
```bash
|
||
timeout 3600 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
|
||
ansible-playbook /work/infra/ansible/playbooks/site.yml \
|
||
-i /work/infra/ansible/inventory/hosts.ini \
|
||
-e k3s_hosts=k3s_prod -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary
|
||
```
|
||
|
||
kube-vip is a separate play (not imported by `site.yml`):
|
||
|
||
```bash
|
||
timeout 1800 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
|
||
ansible-playbook /work/infra/ansible/playbooks/kube-vip.yml \
|
||
-i /work/infra/ansible/inventory/hosts.ini -e k3s_hosts=k3s_prod
|
||
```
|
||
|
||
Hardware health (read-only; `node_health` asserts the memattr fix took and that all RAM is addressed).
|
||
|
||
> **The flag is `node_health_hosts`, NOT `k3s_hosts`.** Diagnostic plays deliberately keep their own
|
||
> target var so they stay independently pointable while a bring-up run is scoped elsewhere
|
||
> (`node-health.yml` → `node_health_hosts`, `memattr-survey.yml` → `memattr_survey_hosts`,
|
||
> `vpn.egress.gateway.yml` → `vpn_egress_gateway_hosts`). The consequence is a **silent** one: passing
|
||
> `k3s_hosts=k3s_prod` here does not error, it falls through to the `k3s_servers` default and runs
|
||
> against **DEV's three nodes**. Verified — this exact mistake was made while writing this file and
|
||
> caught only by `--list-hosts`. **Always dry-run the target first:**
|
||
>
|
||
> ```bash
|
||
> timeout 200 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
|
||
> ansible-playbook /work/infra/ansible/playbooks/node-health.yml \
|
||
> -i /work/infra/ansible/inventory/hosts.ini -e node_health_hosts=k3s_prod --list-hosts
|
||
> ```
|
||
|
||
```bash
|
||
timeout 900 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
|
||
ansible-playbook /work/infra/ansible/playbooks/node-health.yml \
|
||
-i /work/infra/ansible/inventory/hosts.ini -e node_health_hosts=k3s_prod
|
||
```
|
||
|
||
## Phase B — Flux bootstrap
|
||
|
||
`flux-bootstrap.yml` (last play of `site.yml`) shells to `bootstrap/flux/bootstrap.sh`, the single
|
||
documented entrypoint. That script now runs the CRD pre-apply as step **[2/4]**, so a cold bootstrap
|
||
no longer strands `platform`. The **age private key Secret must exist first** — bootstrap verifies it
|
||
and refuses to continue rather than silently skipping decryption
|
||
(`bootstrap/sops/age-key-secret.md`).
|
||
|
||
## Phase C — bring the reconciliation graph up
|
||
|
||
One script, six steps, each independently runnable and idempotent. **Run them in this order.**
|
||
|
||
```bash
|
||
K=infra/ansible/.cache/prod.kubeconfig
|
||
S=https://192.168.2.9:6443
|
||
```
|
||
|
||
### C1 — preflight (read-only, changes nothing)
|
||
|
||
Asserts the guards that prevent a repeat of the DNS incident and reports every tier's state. **Fails
|
||
closed** if `EXTERNALDNS_TXT_OWNER_ID` is empty, still the base sentinel, or `default` (dev's owner).
|
||
|
||
```bash
|
||
timeout 300 infra/devcontainer/dev.sh run bootstrap/standup.sh \
|
||
--kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 preflight
|
||
```
|
||
|
||
### C2 — pre-apply chart CRDs, and break the CDI cold-start deadlock (idempotent)
|
||
|
||
```bash
|
||
timeout 600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
|
||
--kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 crds
|
||
```
|
||
|
||
Two cold-start problems, same root cause: **Flux server-side dry-runs every object in a Kustomization
|
||
before applying any of them, so one unknown `kind` aborts the entire tier.**
|
||
|
||
1. **Kyverno** — its ~5.6 MiB CRD bundle exceeds kubectl's 256 KiB annotation limit, so the chart sets
|
||
`crds.install: false` and nothing else installs them. `bootstrap/flux/preapply-crds.sh` applies them
|
||
server-side, deriving the version from the HelmRelease Flux reconciles (no second pin).
|
||
|
||
2. **CDI — a true deadlock.** `platform-config` applies the CDI CR *and*
|
||
`platform/kubevirt/cdi-storageprofiles.yaml` in the **same pass**. But `StorageProfile`'s CRD does
|
||
not exist until `cdi-operator` has reconciled that CR. So the dry-run fails on the unknown kind →
|
||
the CR is never applied → the CRD is never created → `platform-config` can never converge, and
|
||
every tier below it stays blocked. **Invisible on an established cluster** (dev's CRDs have been
|
||
there for months) **and fatal on a cold one.** Observed on prod 2026-09-05: `cdi-operator`
|
||
Available for 12h with **no CDI CR** and no `storageprofiles` CRD.
|
||
|
||
The step breaks it by applying **only the CR** — the minimum — and leaving everything else to
|
||
Flux. It carries no `${}` tokens, so applying it here cannot diverge from what Flux would render.
|
||
The CRD appeared in 10s.
|
||
|
||
**REPAIRED IN THE GRAPH (2026-09-05).** `cdi-storageprofiles.yaml` now lives in its own tier,
|
||
`./platform-config-cdi`, with `dependsOn: platform-config` and `wait: true`. Because `dependsOn` is
|
||
**revision**-gated and `platform-config` runs `wait: true`, the CDI CR must be applied *and healthy*
|
||
— which is precisely the event that creates the `StorageProfile` CRD — before the profiles are
|
||
applied. A cold cluster now orders itself. The `crds` step is retained as a fallback for a cluster
|
||
already stuck in the broken state, and because it is idempotent.
|
||
|
||
**The generalisable rule, now recorded in `clusters/README.md`:** a CR whose CRD is created by a
|
||
*controller at runtime* is **second-order** and needs its own pass. `cdi-operator.yaml` registers
|
||
exactly one CRD (`cdis.cdi.kubevirt.io`); `storageprofiles`, `datavolumes`, `datasources` and
|
||
`dataimportcrons` appear only after cdi-operator reconciles the CDI CR. An audit of
|
||
`platform-config` found `StorageProfile` is the only second-order kind there.
|
||
|
||
`apps-vms` gained `dependsOn: platform-config-cdi` — load-bearing, not cosmetic:
|
||
`apps/vms/unifi/20-datavolume.yaml` provisions on `storageClassName: longhorn-vm`, and its comment
|
||
previously claimed `platform-config` guaranteed the profiles, which the move would have silently
|
||
falsified.
|
||
|
||
#### Merge precondition and the one-time transient
|
||
|
||
`platform-config` has `prune: true`, so on the first reconcile after merge it **deletes** both
|
||
StorageProfiles, CDI recreates them within seconds with an empty spec, and `platform-config-cdi`
|
||
re-applies the overrides ~30–60s later (kustomize-controller's dependency requeue).
|
||
|
||
**Measured on dev, so the exposure is known rather than assumed** — comparing our override against
|
||
what CDI infers unaided for the same Longhorn provisioner:
|
||
|
||
| profile | `spec` (ours) | CDI-derived `status` |
|
||
|---|---|---|
|
||
| `longhorn-vm` (we override) | RWX / Block, `snapshot` | **ReadWriteMany** / Block, `snapshot` |
|
||
| `longhorn-static` (no override) | *unset* | **ReadWriteOnce** / Block, `snapshot` |
|
||
|
||
So **`cloneStrategy` is not at risk** — CDI infers `snapshot` for Longhorn anyway. **`accessModes`
|
||
is:** unaided, CDI produces **RWO**. A DataVolume provisioned on `longhorn-vm` *inside the window*
|
||
would be stamped RWO/Block and be **permanently non-live-migratable**. Flux can no longer cause that
|
||
(`apps-vms` now gates on the new tier); **do not start a VM disk import, clone or DataVolume by hand
|
||
during the window.**
|
||
|
||
- [ ] **Before merging, confirm `flux get kustomization platform-config` is Ready on the target.** If
|
||
it is not, the profiles are pruned and sit at CDI defaults until it recovers — and prod was in
|
||
exactly that state hours earlier. The fallback is the two-step the Cilium move used: merge with
|
||
`platform-config.prune: false`, confirm the ownership label transferred, restore `prune: true` in
|
||
a second commit.
|
||
- [ ] After: both profiles carry `spec.cloneStrategy: snapshot`, `longhorn-vm` carries
|
||
`claimPropertySets` RWX/Block, and both are labelled
|
||
`kustomize.toolkit.fluxcd.io/name=platform-config-cdi`.
|
||
|
||
### C3 — unwedge stuck Helm releases
|
||
|
||
A release left in `uninstalling` / `pending-*` cannot be advanced by helm-controller: it reports
|
||
`StateError` forever, so `platform` (`wait: true`) never goes Ready and all 10 tiers below it stall.
|
||
**Dry run by default** — it prints what it would do and changes nothing:
|
||
|
||
```bash
|
||
timeout 600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
|
||
--kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 unwedge
|
||
```
|
||
|
||
To act (this **deletes** the release's Kubernetes objects and lets Flux install cold). Pre-applied
|
||
CRDs live **outside** the Helm release and survive, which is exactly why C2 comes first:
|
||
|
||
```bash
|
||
timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
|
||
--kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 \
|
||
--confirm-destructive unwedge
|
||
```
|
||
|
||
### C4 — external-dns safe upgrade
|
||
|
||
**The most dangerous step in the repo, and the ordering is the entire mitigation.** A Helm upgrade
|
||
resets `replicas` to the chart default (1) and applies the new args in ONE operation — there is no
|
||
window in which to check the result first. If `--txt-owner-id` were missing, the pod starts as owner
|
||
`default`, claims dev's TXT registry entries, and under `policy: sync` **deletes the records behind
|
||
them**.
|
||
|
||
So this step **proves the outcome before causing it**: it renders the chart with the *live*
|
||
HelmRelease values and asserts `--txt-owner-id=<owner>` appears, aborting if not. Rendering is
|
||
deterministic and touches nothing, which converts a race into a precondition. It then snapshots the
|
||
Cloudflare zone, upgrades, re-verifies the live args (scaling to 0 and failing if they are wrong),
|
||
and diffs the zone.
|
||
|
||
```bash
|
||
timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
|
||
--kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 externaldns
|
||
```
|
||
|
||
Expected diff: **additions only, all under `prod.home.keithjackson.dev`, all TXT entries owner
|
||
`prod`.** Any change to a name outside that subtree, or to a TXT entry owned by another cluster, is a
|
||
regression — stop and investigate.
|
||
|
||
### C5 — resume the tiers, in dependency order
|
||
|
||
`platform → platform-config → {oauth2-proxy, headlamp, homepage, crafty, apps-wave-0, apps-vms} →
|
||
apps-wave-1 → apps-wave-2`. Waits for each to go Ready and **stops at the first that does not** — a
|
||
partial, understood bring-up beats a forced green. `velero` is skipped while `S3_BACKUP_ENDPOINT` is
|
||
a placeholder (see §Blockers). Long-running; expect it to background.
|
||
|
||
```bash
|
||
timeout 3600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
|
||
--kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 resume
|
||
```
|
||
|
||
### C6 — verify
|
||
|
||
```bash
|
||
timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
|
||
--kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 verify
|
||
```
|
||
|
||
### Everything at once
|
||
|
||
Runs C1→C6 with `unwedge` in **dry-run** mode (no `--confirm-destructive`), so it stops short of the
|
||
destructive step by design:
|
||
|
||
```bash
|
||
timeout 3600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
|
||
--kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 all
|
||
```
|
||
|
||
---
|
||
|
||
## VERIFIED WORKING — 2026-09-05, standup complete
|
||
|
||
| | |
|
||
|---|---|
|
||
| Kustomizations | **12/13 Ready** (`velero` deliberately suspended — see §Blockers) |
|
||
| Pods | **113 total, 0 not Running/Completed** |
|
||
| Wildcard cert | `wildcard-home` **Ready**, Let's Encrypt **prod**, `*.prod.home.keithjackson.dev` |
|
||
| Gateway VIP | `192.168.2.20` · crafty L4 `192.168.2.22` · unifi L4 `192.168.2.21` |
|
||
| Published DNS | **36 records** under `prod.home.keithjackson.dev`, every TXT registry entry `owner=prod` |
|
||
| Dev's records | **57, byte-identical to the pre-standup baseline — nothing added, changed or removed** |
|
||
|
||
End-to-end HTTPS **through the workstation's own resolver — no `--resolve` override, no `-k`** (after
|
||
installing the dnsmasq profile below). Every request landed on `192.168.2.20`:
|
||
|
||
```
|
||
prod.home.keithjackson.dev 200 headlamp. 200 keycloak. 302 (login) nextcloud. 302
|
||
gitea. 200 semaphore. 200 grafana. 302 (SSO) pihole. 302
|
||
```
|
||
|
||
The served certificate is prod's own, confirmed by SNI — **not** dev's wildcard, which is what the
|
||
resolver bug previously made it look like:
|
||
|
||
```
|
||
subject = CN=*.prod.home.keithjackson.dev
|
||
issuer = C=US, O=Let's Encrypt, CN=YR1 (LE production)
|
||
SAN = *.prod.home.keithjackson.dev, prod.home.keithjackson.dev
|
||
valid = 2026-09-05 .. 2026-12-04
|
||
```
|
||
|
||
Dev, checked in the same pass, still serves `CN=*.home.keithjackson.dev` — **unchanged**.
|
||
|
||
> Ignore `curl: (91) No OCSP response received` if you pass `--cert-status`: Let's Encrypt retired
|
||
> OCSP responders in 2025, so there is nothing to staple. It is not a certificate fault, and the plain
|
||
> requests above all validated normally.
|
||
|
||
### Workstation DNS — install the prod dnsmasq profile
|
||
|
||
**CORRECTION (operator, 2026-09-05): this is the workstation's own dnsmasq, NOT pfSense.** Earlier
|
||
notes in this repo called it a "pfSense override"; that was wrong and the topology below is why.
|
||
|
||
`/etc/NetworkManager/dnsmasq.d/10-home.keithjackson.dev.conf` (the dev profile) contains:
|
||
|
||
```
|
||
address=/.home.keithjackson.dev/192.168.2.50
|
||
```
|
||
|
||
dnsmasq's `address=` matches the named domain **and every subdomain**, so that one line also captures
|
||
the whole `prod.home.keithjackson.dev` tree and answers it with **dev's** address, while Cloudflare
|
||
correctly returns prod's `192.168.2.20`.
|
||
|
||
**Fix — one command, already authored in this repo:**
|
||
|
||
```bash
|
||
sudo infra/workstation/dnsmasq/install.sh
|
||
```
|
||
|
||
It installs `20-prod.home.keithjackson.dev.conf` and reloads the live resolver. dnsmasq resolves
|
||
`address=` by **longest match**, so the prod rules win without editing or reordering the dev profile.
|
||
Verified against a throwaway dnsmasq loaded with both profiles:
|
||
|
||
| query | answer |
|
||
|---|---|
|
||
| `prod.home.keithjackson.dev`, `gitea.prod…`, `keycloak.prod…` | `192.168.2.20` (Gateway) |
|
||
| `unifi.prod.home.keithjackson.dev` | `192.168.2.21` (L4) |
|
||
| `crafty-mc.prod.home.keithjackson.dev` | `192.168.2.22` (L4) |
|
||
| `gitea.home.keithjackson.dev`, `home.keithjackson.dev` | `192.168.2.50` — **dev, unchanged** |
|
||
|
||
> **Install to `/etc/NetworkManager/dnsmasq.d/`, never `/etc/dnsmasq.d/`.** The standalone
|
||
> `dnsmasq.service` is **inactive**; the live resolver is the one NetworkManager spawns
|
||
> (`dns=dnsmasq`) with `--conf-dir=/etc/NetworkManager/dnsmasq.d`. A file in the other directory
|
||
> changes nothing and looks like it should. `install.sh` pins the destination for exactly this reason.
|
||
|
||
**Until it is installed, verify prod only with an explicit resolve** — and note *why* this trap is
|
||
convincing: dev serves a `*.home.keithjackson.dev` wildcard cert, which matches
|
||
`prod.home.keithjackson.dev` (one label), so the prod **apex returns 200 over valid TLS** while
|
||
`keycloak.prod.home.keithjackson.dev` (two labels) fails with a certificate-name error. It reads like
|
||
a broken prod certificate. It is the wrong server answering.
|
||
|
||
```bash
|
||
curl -sS --resolve gitea.prod.home.keithjackson.dev:443:192.168.2.20 \
|
||
https://gitea.prod.home.keithjackson.dev/
|
||
```
|
||
|
||
## Blockers
|
||
|
||
### 1. Backups: endpoint DECIDED, but prod cannot reach it — **prod has no backup coverage**
|
||
|
||
**Operator ruling 2026-09-05: "the S3 backup location is the same as Dev."** Prod's
|
||
`S3_BACKUP_ENDPOINT` is now `http://192.168.1.9:9000`, and the `TBD-PROD-*` placeholder is retired.
|
||
|
||
**It is not reachable from prod.** Measured from a throwaway pod **inside the prod cluster** — not
|
||
from the workstation, which proves nothing about pod egress:
|
||
|
||
```
|
||
curl http://192.168.1.9:9000/minio/health/live → connection timed out after 15s
|
||
```
|
||
|
||
**This is expected topology, not a misconfiguration** (operator, 2026-09-05): the k3s prod cluster is
|
||
**not behind pfSense**. It sits on the SOHO router network `192.168.2.0/24`, which is pfSense's
|
||
**WAN** side; pfSense *contains* `192.168.1.0/24`, the dev LAN where MinIO lives. Prod reaching
|
||
`192.168.1.9` therefore means crossing **into** pfSense from its WAN — which requires a deliberate
|
||
pfSense rule or port-forward that does not exist. **Operator decision required**, and worth weighing:
|
||
a prod backup that lands inside the dev LAN is arguably not off-cluster DR at all.
|
||
|
||
Re-test with `standup.sh --s3-endpoint http://192.168.1.9:9000 s3check` — it fails closed and says so.
|
||
|
||
> **Do not confuse this with the SOHO router's inbound NAT.** That router forwards inbound **443** to
|
||
> the pfSense WAN address `192.168.2.102` to serve the **existing, non-k3s production environment**,
|
||
> which must stay live on 443 until the k3s cutover is deliberately made. Nothing in this runbook
|
||
> touches that NAT, and prod's Gateway on `192.168.2.20:443` does not contend with it.
|
||
|
||
Until that routing exists, `velero` stays suspended and **`apps-wave-0` still applies GREEN** — the
|
||
token *is* defined, so strict substitution succeeds — while every write to that endpoint fails at
|
||
**runtime**. A green `apps-wave-0` does not mean backups work. **Treat prod as having no backup
|
||
coverage.**
|
||
|
||
#### Sharing one MinIO is only safe because of two new knobs
|
||
|
||
Nothing about the backup *paths* was per-environment before 2026-09-05. Enabling prod against dev's
|
||
MinIO without these would have been the external-dns incident again, in a worse place:
|
||
|
||
| | before | after |
|
||
|---|---|---|
|
||
| Velero | `bucket: velero`, **no prefix**, both clusters run schedules named `daily` | `prefix: ${VELERO_BACKUP_PREFIX}` — dev `""`, prod `prod` |
|
||
| CNPG | `destinationPath: s3://cnpg`, serverName defaults to the **Cluster name** | `${CNPG_BACKUP_PATH}` — dev `s3://cnpg`, prod `s3://cnpg/prod` |
|
||
|
||
- **Velero** syncs backups it finds in its BSL into itself as `Backup` CRs and **garbage-collects them
|
||
when their TTL expires**. Prod would have adopted dev's backups and deleted them on schedule.
|
||
- **CNPG is the worse one.** Both environments have `keycloak-db`, `nextcloud-db` and `semaphore-db`,
|
||
so prod's Postgres would archive WAL into the *exact prefix* dev's does. Two servers interleaving
|
||
WAL into one Barman stream **corrupts the chain and destroys PITR** — and dev's restore is
|
||
**proven** (R-STATE-03).
|
||
|
||
**The base values are dev's current layout on purpose and must stay that way.** Dev's existing
|
||
backups live at the bucket root and at `s3://cnpg`; giving dev a prefix would **orphan the very
|
||
backups its proven restore depends on** — the same reasoning that keeps dev's
|
||
`EXTERNALDNS_TXT_OWNER_ID` at `default`. Prod has no history to preserve, so it takes the nested paths.
|
||
|
||
> **Delivery note.** Prod reconciles the bootstrap Gitea's `main` at `c8c1d29`. This change sits on a
|
||
> later branch that also carries the DNS inversion, which must **not** reach prod (it would move prod
|
||
> to the apex and collide with dev). Delivering it needs a `c8c1d29`-based branch carrying only these
|
||
> commits. No urgency: velero cannot be enabled until the routing above is fixed.
|
||
|
||
### 2. Prod is on `prod.home.keithjackson.dev`, not the apex
|
||
|
||
The operator's target layout is dev under `*.dev.home.keithjackson.dev` and prod on the **apex**.
|
||
Prod cannot take the apex until dev vacates it, because both clusters would request a
|
||
`letsencrypt-prod` wildcard for the *same name set* and race each other's `_acme-challenge` TXT
|
||
records, so neither reliably solves DNS-01.
|
||
|
||
**This does not block phase C.** Prod on its own subtree with `txtOwnerId: prod` collides with
|
||
nothing, and standing it up first is strictly better: it proves the whole stack before the flip, so
|
||
the cutover becomes a DNS/TLS change on a known-good cluster instead of a bring-up and a cutover at
|
||
once. The apex move is `infra/PROD-PROVISIONING.md` §10a, sequence **D** (dev vacates) then **P**.
|
||
|
||
> **THE APEX FLIP IS ALSO THE LEGACY-PROD CUTOVER — they are the same event.** The SOHO router NATs
|
||
> inbound **443** to the pfSense WAN address `192.168.2.102`, serving the **existing, non-k3s
|
||
> production environment**, which the operator requires to stay live until the switch is deliberately
|
||
> made. So §10a sequence **P** is not merely a DNS/TLS change on the k3s side: taking
|
||
> `home.keithjackson.dev` means public traffic must also stop resolving to / being forwarded to the
|
||
> legacy stack. Sequence **P** as written does **not** cover repointing that NAT, and the two must be
|
||
> planned together or the apex will resolve to k3s while inbound 443 still lands on the legacy box.
|
||
>
|
||
> Nothing in phase C touches it: prod's Gateway is `192.168.2.20:443`, a different address from the
|
||
> NAT target, and prod's wildcard cert is issued by **DNS-01**, which needs no inbound 443 at all.
|
||
|
||
---
|
||
|
||
## Not covered here
|
||
|
||
- **Dev's migration off the apex** — §10a sequence D. Touches Keycloak's issuer, ~12 OIDC client
|
||
redirect URIs, and a rolling k3s restart that invalidates every OIDC kubeconfig. Operator-present.
|
||
- **Prod's GitOps cutover** (P5) from the workstation bootstrap Gitea to prod's own in-cluster Gitea.
|
||
`clusters/prod/kustomization.yaml` owns that procedure.
|
||
- **Six L2 conformance tests hardcode the apex** and must be parameterized per environment before
|
||
§9's "suite passes green against the prod cluster" can be claimed.
|