feat(prod): standup.sh + PROD-STANDUP.md — the complete standup sequence, in git
The last steps of a prod bring-up were living in shell history: resume the tiers in the right order, clear a wedged HelmRelease, fix external-dns without deleting the other cluster's DNS. That is the same failure the Kyverno CRD pre-apply had, so it gets the same treatment. bootstrap/standup.sh — environment-agnostic (R-ENV-02), no dev/prod fork. Six idempotent steps: preflight, crds, unwedge, externaldns, resume, verify. THE GUARD IS THE POINT. --expect-server is MANDATORY and is compared against the live API server before any step runs; there is no override flag. Dev is live and shares the keithjackson.dev Cloudflare zone with prod, and on 2026-09-04 prod deleted every one of dev's public records. Negative control run: prod's kubeconfig with dev's server URL refuses and changes nothing. preflight fails closed if EXTERNALDNS_TXT_OWNER_ID is empty, still the base sentinel, or `default` (dev's owner) — the exact condition that caused the incident. It also surfaces SUSPENDED HelmReleases separately, because a suspended release keeps its last Ready condition and a stale `Ready=True` otherwise hides that Flux is not managing it at all (external-dns today). externaldns is the dangerous step, and the ordering IS the mitigation. A Helm upgrade resets replicas to 1 and applies new args in ONE operation, so there is no window to check the result first; if --txt-owner-id were missing the pod starts as owner `default` and syncs away dev's records. So the step PROVES THE OUTCOME BEFORE CAUSING IT: render the chart with the LIVE HelmRelease values and assert --txt-owner-id=<owner> appears, aborting if not. Rendering is deterministic and touches nothing, which turns a race into a precondition. Then snapshot Cloudflare, upgrade, re-verify live args (scale to 0 and fail if wrong), and diff the zone. resume walks the dependency order and STOPS at the first tier that does not go Ready — a partial understood bring-up beats a forced green. velero is skipped while S3_BACKUP_ENDPOINT is the TBD placeholder. TWO BUGS CAUGHT WHILE VALIDATING THIS, both by measuring instead of assuming: 1. preflight counted Kyverno CRDs with grep 'kyverno.io' and reported 20 of 22 — the policy-report types are in the wgpolicyk8s.io group. A preflight that under-reports reads as "the pre-apply failed" when it succeeded. Fixed and re-run: 22. 2. The documented node-health invocation used -e k3s_hosts=k3s_prod and SILENTLY RAN AGAINST DEV'S THREE NODES. Diagnostic plays deliberately keep their own target var (node_health_hosts, memattr_survey_hosts, vpn_egress_gateway_hosts) so they stay pointable while a bring-up is scoped elsewhere — a good design I did NOT change. The hazard is that the wrong flag does not error, it falls through to the dev default. Caught only by --list-hosts; the doc now leads with that warning and a dry-run command. Verified after fixing: node-health and all six site.yml plays target prod. .claude/settings.local.json — permission rules for the above. Every entry hardcodes prod's kubeconfig AND prod's --expect-server (or prod's ansible groups), so the permission strings are themselves prod-scoped and cannot be turned against dev. Deliberately NO rule for `dev.sh run bash -c *`, which would have been blanket cluster access wearing a narrow-looking mask. Refs R-ENV-02, R-GITOPS-01, R-SEC-02. Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
7432012713
commit
4160fee6ce
3 files changed
+598
No files matched your search
@@ -10,6 +10,11 @@ hardware until the blockers in §7 are closed and `make -C ci gate` is green.
|
||||
> **R-ENV-01/02/03** — the very requirements this document serves. We proceed with prep work
|
||||
> knowingly; see `DECISIONS-LOG.md` (2026-08-31 Phase-7 entry).
|
||||
|
||||
> **Looking for the commands?** This document is the *decision record* — why prod is built the way it
|
||||
> is, what was measured, and which hypotheses are dead. The **executable sequence** lives in
|
||||
> [`PROD-STANDUP.md`](PROD-STANDUP.md): every command, in order, driven by `bootstrap/standup.sh`.
|
||||
> Start there; come back here for the reasoning behind any step.
|
||||
|
||||
Prod is **3× mini-PC bare metal** ([spec docs/02](../../home-network-k3s/docs/02-environments-and-provisioning.md)).
|
||||
The dev path (OpenTofu → Proxmox VM → cloud-init) **does not apply**: there is no hypervisor API to
|
||||
call. Stage 1 "Create" is a human imaging a box. Stages 2–4 (baseline, cluster-join, manage) reuse
|
||||
|
||||
@@ -0,0 +1,233 @@
|
||||
# Prod cluster standup — the complete command sequence
|
||||
|
||||
**Every command below is copy-pasteable and runs from the repo root**, through the pinned dev
|
||||
container. Nothing here needs a command invented on the spot; that is the point. If a step is not in
|
||||
this file it is not part of standup.
|
||||
|
||||
State as of **2026-09-05**: phases **A** and **B** are DONE on the current hardware. The cluster is
|
||||
live (k3s v1.36.4 HA, 3 etcd members, kube-vip, Cilium 1.20.1, Longhorn) with `platform` reconciling
|
||||
and **8/9 HelmReleases Ready, 76/76 pods Running**. Outstanding work is **phase C**, blocked on the
|
||||
two items in [§Blockers](#blockers).
|
||||
|
||||
> **The one guard that matters.** Dev is LIVE and shares the `keithjackson.dev` Cloudflare zone with
|
||||
> prod. On 2026-09-04 prod deleted every one of dev's public DNS records. `bootstrap/standup.sh`
|
||||
> therefore **requires** `--expect-server` and compares it against the live API server before running
|
||||
> anything. There is no override flag. Do not add one.
|
||||
|
||||
## Constants
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| kube-API (kube-vip VIP) | `https://192.168.2.9:6443` |
|
||||
| Gateway / ingress VIP | `192.168.2.20` |
|
||||
| Nodes | `k3s-prod-1/2/3` — `192.168.2.10 / .11 / .12` |
|
||||
| Break-glass kubeconfig | `infra/ansible/.cache/prod.kubeconfig` (cert-based; survives OIDC/DNS failure) |
|
||||
| GitOps source (interim) | `http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git` branch `main` |
|
||||
| `BASE_DOMAIN` | `prod.home.keithjackson.dev` — **not** the apex; see §Blockers |
|
||||
|
||||
Container paths: the repo is mounted at `/work`, so kubeconfig paths passed to the script are
|
||||
**repo-relative** (`infra/ansible/.cache/prod.kubeconfig`).
|
||||
|
||||
---
|
||||
|
||||
## Phase A — bare metal to a joined cluster (Ansible)
|
||||
|
||||
Prerequisites that are **not** commands: firmware/BIOS (§1), disk layout (§2), OS install (§3),
|
||||
network (§4), and SSH + the vaulted `ansible_become_password` in
|
||||
`inventory/group_vars/k3s_prod/vault.yml` (§5). Ansible cannot escalate on prod without that vault
|
||||
file. See `infra/PROD-PROVISIONING.md`.
|
||||
|
||||
**Install the backports kernel first.** All three prod nodes need ≥6.13 because HP's R21 firmware
|
||||
publishes a corrupt EFI Memory Attributes Table and Linux 6.12's `efi_memattr_init()` does not
|
||||
validate it, reserving everything above the table (fixed upstream in 6.13 by `8fbe4c49c0cc`, absent
|
||||
from `linux-6.12.y`). Prod runs `7.1.8+deb13-amd64` from trixie-backports.
|
||||
|
||||
```bash
|
||||
timeout 3600 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
|
||||
ansible-playbook /work/infra/ansible/playbooks/node-baseline.yml \
|
||||
-i /work/infra/ansible/inventory/hosts.ini -e k3s_hosts=k3s_prod --tags kernel
|
||||
```
|
||||
|
||||
Then the full chain — baseline, k3s HA, Cilium, verify, Flux hand-off. **All three `-e` flags are
|
||||
required**: a play's `hosts:` is templated before any host exists, so `group_vars` cannot retarget it.
|
||||
|
||||
```bash
|
||||
timeout 3600 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
|
||||
ansible-playbook /work/infra/ansible/playbooks/site.yml \
|
||||
-i /work/infra/ansible/inventory/hosts.ini \
|
||||
-e k3s_hosts=k3s_prod -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary
|
||||
```
|
||||
|
||||
kube-vip is a separate play (not imported by `site.yml`):
|
||||
|
||||
```bash
|
||||
timeout 1800 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
|
||||
ansible-playbook /work/infra/ansible/playbooks/kube-vip.yml \
|
||||
-i /work/infra/ansible/inventory/hosts.ini -e k3s_hosts=k3s_prod
|
||||
```
|
||||
|
||||
Hardware health (read-only; `node_health` asserts the memattr fix took and that all RAM is addressed).
|
||||
|
||||
> **The flag is `node_health_hosts`, NOT `k3s_hosts`.** Diagnostic plays deliberately keep their own
|
||||
> target var so they stay independently pointable while a bring-up run is scoped elsewhere
|
||||
> (`node-health.yml` → `node_health_hosts`, `memattr-survey.yml` → `memattr_survey_hosts`,
|
||||
> `vpn.egress.gateway.yml` → `vpn_egress_gateway_hosts`). The consequence is a **silent** one: passing
|
||||
> `k3s_hosts=k3s_prod` here does not error, it falls through to the `k3s_servers` default and runs
|
||||
> against **DEV's three nodes**. Verified — this exact mistake was made while writing this file and
|
||||
> caught only by `--list-hosts`. **Always dry-run the target first:**
|
||||
>
|
||||
> ```bash
|
||||
> timeout 200 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
|
||||
> ansible-playbook /work/infra/ansible/playbooks/node-health.yml \
|
||||
> -i /work/infra/ansible/inventory/hosts.ini -e node_health_hosts=k3s_prod --list-hosts
|
||||
> ```
|
||||
|
||||
```bash
|
||||
timeout 900 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \
|
||||
ansible-playbook /work/infra/ansible/playbooks/node-health.yml \
|
||||
-i /work/infra/ansible/inventory/hosts.ini -e node_health_hosts=k3s_prod
|
||||
```
|
||||
|
||||
## Phase B — Flux bootstrap
|
||||
|
||||
`flux-bootstrap.yml` (last play of `site.yml`) shells to `bootstrap/flux/bootstrap.sh`, the single
|
||||
documented entrypoint. That script now runs the CRD pre-apply as step **[2/4]**, so a cold bootstrap
|
||||
no longer strands `platform`. The **age private key Secret must exist first** — bootstrap verifies it
|
||||
and refuses to continue rather than silently skipping decryption
|
||||
(`bootstrap/sops/age-key-secret.md`).
|
||||
|
||||
## Phase C — bring the reconciliation graph up
|
||||
|
||||
One script, six steps, each independently runnable and idempotent. **Run them in this order.**
|
||||
|
||||
```bash
|
||||
K=infra/ansible/.cache/prod.kubeconfig
|
||||
S=https://192.168.2.9:6443
|
||||
```
|
||||
|
||||
### C1 — preflight (read-only, changes nothing)
|
||||
|
||||
Asserts the guards that prevent a repeat of the DNS incident and reports every tier's state. **Fails
|
||||
closed** if `EXTERNALDNS_TXT_OWNER_ID` is empty, still the base sentinel, or `default` (dev's owner).
|
||||
|
||||
```bash
|
||||
timeout 300 infra/devcontainer/dev.sh run bootstrap/standup.sh \
|
||||
--kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 preflight
|
||||
```
|
||||
|
||||
### C2 — pre-apply chart CRDs (idempotent)
|
||||
|
||||
```bash
|
||||
timeout 600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
|
||||
--kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 crds
|
||||
```
|
||||
|
||||
### C3 — unwedge stuck Helm releases
|
||||
|
||||
A release left in `uninstalling` / `pending-*` cannot be advanced by helm-controller: it reports
|
||||
`StateError` forever, so `platform` (`wait: true`) never goes Ready and all 10 tiers below it stall.
|
||||
**Dry run by default** — it prints what it would do and changes nothing:
|
||||
|
||||
```bash
|
||||
timeout 600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
|
||||
--kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 unwedge
|
||||
```
|
||||
|
||||
To act (this **deletes** the release's Kubernetes objects and lets Flux install cold). Pre-applied
|
||||
CRDs live **outside** the Helm release and survive, which is exactly why C2 comes first:
|
||||
|
||||
```bash
|
||||
timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
|
||||
--kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 \
|
||||
--confirm-destructive unwedge
|
||||
```
|
||||
|
||||
### C4 — external-dns safe upgrade
|
||||
|
||||
**The most dangerous step in the repo, and the ordering is the entire mitigation.** A Helm upgrade
|
||||
resets `replicas` to the chart default (1) and applies the new args in ONE operation — there is no
|
||||
window in which to check the result first. If `--txt-owner-id` were missing, the pod starts as owner
|
||||
`default`, claims dev's TXT registry entries, and under `policy: sync` **deletes the records behind
|
||||
them**.
|
||||
|
||||
So this step **proves the outcome before causing it**: it renders the chart with the *live*
|
||||
HelmRelease values and asserts `--txt-owner-id=<owner>` appears, aborting if not. Rendering is
|
||||
deterministic and touches nothing, which converts a race into a precondition. It then snapshots the
|
||||
Cloudflare zone, upgrades, re-verifies the live args (scaling to 0 and failing if they are wrong),
|
||||
and diffs the zone.
|
||||
|
||||
```bash
|
||||
timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
|
||||
--kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 externaldns
|
||||
```
|
||||
|
||||
Expected diff: **additions only, all under `prod.home.keithjackson.dev`, all TXT entries owner
|
||||
`prod`.** Any change to a name outside that subtree, or to a TXT entry owned by another cluster, is a
|
||||
regression — stop and investigate.
|
||||
|
||||
### C5 — resume the tiers, in dependency order
|
||||
|
||||
`platform → platform-config → {oauth2-proxy, headlamp, homepage, crafty, apps-wave-0, apps-vms} →
|
||||
apps-wave-1 → apps-wave-2`. Waits for each to go Ready and **stops at the first that does not** — a
|
||||
partial, understood bring-up beats a forced green. `velero` is skipped while `S3_BACKUP_ENDPOINT` is
|
||||
a placeholder (see §Blockers). Long-running; expect it to background.
|
||||
|
||||
```bash
|
||||
timeout 3600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
|
||||
--kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 resume
|
||||
```
|
||||
|
||||
### C6 — verify
|
||||
|
||||
```bash
|
||||
timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \
|
||||
--kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 verify
|
||||
```
|
||||
|
||||
### Everything at once
|
||||
|
||||
Runs C1→C6 with `unwedge` in **dry-run** mode (no `--confirm-destructive`), so it stops short of the
|
||||
destructive step by design:
|
||||
|
||||
```bash
|
||||
timeout 3600 infra/devcontainer/dev.sh run bootstrap/standup.sh \
|
||||
--kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 all
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Blockers
|
||||
|
||||
### 1. `S3_BACKUP_ENDPOINT` is an intentional placeholder — operator decision required
|
||||
|
||||
Prod sets `TBD-PROD-S3-BACKUP-ENDPOINT` on purpose: base defaults are the **dev** values, so an
|
||||
omitted override would make prod silently inherit dev's address (`http://192.168.1.9:9000`), which
|
||||
prod cannot reach anyway. A loud placeholder beats a silent wrong answer.
|
||||
|
||||
It is consumed by **`velero` and `apps-wave-0`**. `velero` is skipped by C5. **`apps-wave-0` still
|
||||
applies** — the token *is* defined, so strict substitution succeeds — but anything writing to that
|
||||
endpoint fails at **runtime**, not at apply time. **A green `apps-wave-0` does not mean backups
|
||||
work.** Until a real prod endpoint is set, treat prod as having **no** backup coverage.
|
||||
|
||||
### 2. Prod is on `prod.home.keithjackson.dev`, not the apex
|
||||
|
||||
The operator's target layout is dev under `*.dev.home.keithjackson.dev` and prod on the **apex**.
|
||||
Prod cannot take the apex until dev vacates it, because both clusters would request a
|
||||
`letsencrypt-prod` wildcard for the *same name set* and race each other's `_acme-challenge` TXT
|
||||
records, so neither reliably solves DNS-01.
|
||||
|
||||
**This does not block phase C.** Prod on its own subtree with `txtOwnerId: prod` collides with
|
||||
nothing, and standing it up first is strictly better: it proves the whole stack before the flip, so
|
||||
the cutover becomes a DNS/TLS change on a known-good cluster instead of a bring-up and a cutover at
|
||||
once. The apex move is `infra/PROD-PROVISIONING.md` §10a, sequence **D** (dev vacates) then **P**.
|
||||
|
||||
---
|
||||
|
||||
## Not covered here
|
||||
|
||||
- **Dev's migration off the apex** — §10a sequence D. Touches Keycloak's issuer, ~12 OIDC client
|
||||
redirect URIs, and a rolling k3s restart that invalidates every OIDC kubeconfig. Operator-present.
|
||||
- **Prod's GitOps cutover** (P5) from the workstation bootstrap Gitea to prod's own in-cluster Gitea.
|
||||
`clusters/prod/kustomization.yaml` owns that procedure.
|
||||
- **Six L2 conformance tests hardcode the apex** and must be parameterized per environment before
|
||||
§9's "suite passes green against the prod cluster" can be claimed.
|
||||
Reference in new issue
Block a user