From 4160fee6ce414e4b57790561ac9b445cd83766b0 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 5 Sep 2026 00:37:48 -0500 Subject: [PATCH] =?UTF-8?q?feat(prod):=20standup.sh=20+=20PROD-STANDUP.md?= =?UTF-8?q?=20=E2=80=94=20the=20complete=20standup=20sequence,=20in=20git?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The last steps of a prod bring-up were living in shell history: resume the tiers in the right order, clear a wedged HelmRelease, fix external-dns without deleting the other cluster's DNS. That is the same failure the Kyverno CRD pre-apply had, so it gets the same treatment. bootstrap/standup.sh — environment-agnostic (R-ENV-02), no dev/prod fork. Six idempotent steps: preflight, crds, unwedge, externaldns, resume, verify. THE GUARD IS THE POINT. --expect-server is MANDATORY and is compared against the live API server before any step runs; there is no override flag. Dev is live and shares the keithjackson.dev Cloudflare zone with prod, and on 2026-09-04 prod deleted every one of dev's public records. Negative control run: prod's kubeconfig with dev's server URL refuses and changes nothing. preflight fails closed if EXTERNALDNS_TXT_OWNER_ID is empty, still the base sentinel, or `default` (dev's owner) — the exact condition that caused the incident. It also surfaces SUSPENDED HelmReleases separately, because a suspended release keeps its last Ready condition and a stale `Ready=True` otherwise hides that Flux is not managing it at all (external-dns today). externaldns is the dangerous step, and the ordering IS the mitigation. A Helm upgrade resets replicas to 1 and applies new args in ONE operation, so there is no window to check the result first; if --txt-owner-id were missing the pod starts as owner `default` and syncs away dev's records. So the step PROVES THE OUTCOME BEFORE CAUSING IT: render the chart with the LIVE HelmRelease values and assert --txt-owner-id= appears, aborting if not. Rendering is deterministic and touches nothing, which turns a race into a precondition. Then snapshot Cloudflare, upgrade, re-verify live args (scale to 0 and fail if wrong), and diff the zone. resume walks the dependency order and STOPS at the first tier that does not go Ready — a partial understood bring-up beats a forced green. velero is skipped while S3_BACKUP_ENDPOINT is the TBD placeholder. TWO BUGS CAUGHT WHILE VALIDATING THIS, both by measuring instead of assuming: 1. preflight counted Kyverno CRDs with grep 'kyverno.io' and reported 20 of 22 — the policy-report types are in the wgpolicyk8s.io group. A preflight that under-reports reads as "the pre-apply failed" when it succeeded. Fixed and re-run: 22. 2. The documented node-health invocation used -e k3s_hosts=k3s_prod and SILENTLY RAN AGAINST DEV'S THREE NODES. Diagnostic plays deliberately keep their own target var (node_health_hosts, memattr_survey_hosts, vpn_egress_gateway_hosts) so they stay pointable while a bring-up is scoped elsewhere — a good design I did NOT change. The hazard is that the wrong flag does not error, it falls through to the dev default. Caught only by --list-hosts; the doc now leads with that warning and a dry-run command. Verified after fixing: node-health and all six site.yml plays target prod. .claude/settings.local.json — permission rules for the above. Every entry hardcodes prod's kubeconfig AND prod's --expect-server (or prod's ansible groups), so the permission strings are themselves prod-scoped and cannot be turned against dev. Deliberately NO rule for `dev.sh run bash -c *`, which would have been blanket cluster access wearing a narrow-looking mask. Refs R-ENV-02, R-GITOPS-01, R-SEC-02. Co-Authored-By: Claude Opus 5 --- bootstrap/standup.sh | 360 +++++++++++++++++++++++++++++++++++++ infra/PROD-PROVISIONING.md | 5 + infra/PROD-STANDUP.md | 233 ++++++++++++++++++++++++ 3 files changed, 598 insertions(+) create mode 100755 bootstrap/standup.sh create mode 100644 infra/PROD-STANDUP.md diff --git a/bootstrap/standup.sh b/bootstrap/standup.sh new file mode 100755 index 0000000..f654839 --- /dev/null +++ b/bootstrap/standup.sh @@ -0,0 +1,360 @@ +#!/usr/bin/env bash +# +# standup.sh — bring a bootstrapped cluster's reconciliation graph all the way up. +# +# ENVIRONMENT-AGNOSTIC (R-ENV-02): it reads everything it needs from the cluster it is pointed at and +# from the authored manifests. There is no dev/prod fork. What makes it safe to run against prod is +# the MANDATORY --expect-server guard, not a name in the filename. +# +# WHY THIS EXISTS +# `bootstrap/flux/bootstrap.sh` installs Flux and applies the graph. It does NOT finish the job when +# tiers have been deliberately suspended (a DNS/TLS migration window, an incident recovery) or when +# a HelmRelease has wedged. Those last steps were living in shell history and in a human's head, +# which is the same failure the Kyverno CRD pre-apply had. This is that sequence, in git, ordered, +# idempotent, and gated. +# +# THE ORDERING IS LOAD-BEARING. Read infra/PROD-STANDUP.md before deviating from it. +# +# USAGE +# bootstrap/standup.sh --kubeconfig --expect-server [--confirm-destructive] +# +# steps: preflight crds unwedge externaldns resume verify (or: all) +# +# --expect-server is REQUIRED and is compared against the live API server before ANY command runs. +# Pointing this at the wrong cluster is the one mistake with unbounded blast radius: dev is live and +# shares a Cloudflare zone with prod. The guard is not optional and there is no override flag. +# +# --confirm-destructive is required by `unwedge` only. Everything else is additive or reversible. +# +set -euo pipefail + +# ---------------------------------------------------------------------------- args +KUBECONFIG_ARG="" EXPECT_SERVER="" CONFIRM_DESTRUCTIVE=0 +STEPS=() +while [ $# -gt 0 ]; do + case "$1" in + --kubeconfig) KUBECONFIG_ARG="${2:?--kubeconfig needs a value}"; shift 2 ;; + --expect-server) EXPECT_SERVER="${2:?--expect-server needs a value}"; shift 2 ;; + --confirm-destructive) CONFIRM_DESTRUCTIVE=1; shift ;; + -h|--help) sed -n '2,30p' "$0"; exit 0 ;; + -*) echo "ERROR: unknown flag '$1'" >&2; exit 2 ;; + *) STEPS+=("$1"); shift ;; + esac +done + +[ -n "${KUBECONFIG_ARG}" ] || { echo "ERROR: --kubeconfig is required." >&2; exit 2; } +[ -n "${EXPECT_SERVER}" ] || { echo "ERROR: --expect-server is required (safety guard)." >&2; exit 2; } +[ "${#STEPS[@]}" -gt 0 ] || { echo "ERROR: no step given. One of: preflight crds unwedge externaldns resume verify all" >&2; exit 2; } + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/.." && pwd))" +cd "${repo_root}" + +export KUBECONFIG="${KUBECONFIG_ARG}" +[ -f "${KUBECONFIG}" ] || { echo "ERROR: kubeconfig not found: ${KUBECONFIG}" >&2; exit 1; } + +# Where DNS snapshots go. Deliberately OUTSIDE the repo: they are run artifacts, not authored state. +SNAP_DIR="${STANDUP_SNAP_DIR:-${TMPDIR:-/tmp}/standup-dns}" +mkdir -p "${SNAP_DIR}" + +say() { printf '\n\033[1m==> %s\033[0m\n' "$*"; } +info() { printf ' %s\n' "$*"; } +die() { printf '\033[31mERROR: %s\033[0m\n' "$*" >&2; exit 1; } +warn() { printf '\033[33mWARN: %s\033[0m\n' "$*" >&2; } + +for bin in kubectl flux helm yq sops; do + command -v "$bin" >/dev/null 2>&1 || die "'$bin' not found — run this inside infra/devcontainer." +done + +# ------------------------------------------------------------------ THE SAFETY GUARD +# Checked once, before any step, and re-checked by every mutating helper. The failure mode this +# prevents is running a prod bring-up against the LIVE dev cluster, whose public DNS records prod +# has already destroyed once (2026-09-04). +live_server="$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')" +[ "${live_server}" = "${EXPECT_SERVER}" ] || die \ + "TARGET MISMATCH — refusing to run. + --expect-server : ${EXPECT_SERVER} + live API server : ${live_server} + The kubeconfig does not point where you think it does. Nothing has been changed." +say "target confirmed: ${live_server}" + +# Dependency order. Flux enforces the edges itself via dependsOn; resuming in this order simply means +# each wait is meaningful instead of blocking on a dependency that is still suspended. +# flux-system -> cilium -> platform -> platform-config -> {leaf tiers, apps-wave-0} -> wave-1 -> wave-2 +TIER_ORDER=(platform platform-config oauth2-proxy headlamp homepage crafty apps-wave-0 apps-wave-1 apps-wave-2 apps-vms) +# velero is NOT in that list on purpose — see step `resume`: it consumes S3_BACKUP_ENDPOINT, which on +# prod is an intentionally-invalid TBD placeholder pending an operator decision. + +cvar() { kubectl -n flux-system get cm cluster-vars -o jsonpath="{.data.$1}" 2>/dev/null || true; } + +k_ready() { kubectl -n flux-system get kustomization "$1" \ + -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || true; } +k_msg() { kubectl -n flux-system get kustomization "$1" \ + -o jsonpath='{.status.conditions[?(@.type=="Ready")].message}' 2>/dev/null || true; } +k_susp() { kubectl -n flux-system get kustomization "$1" -o jsonpath='{.spec.suspend}' 2>/dev/null || true; } + +wait_ready() { # + local n="$1" t="${2:-600}" i=0 + while [ "$i" -lt "$t" ]; do + [ "$(k_ready "$n")" = "True" ] && { info "$n Ready (${i}s)"; return 0; } + sleep 10; i=$((i+10)) + done + warn "$n did NOT reach Ready in ${t}s: $(k_msg "$n")" + return 1 +} + +# Snapshot the Cloudflare zone. This is how damage to the OTHER cluster's records becomes detectable. +# Queried against the Cloudflare API, never a local resolver: a pfSense override produced a false +# healthy reading during the 2026-09-04 incident and hid the damage for a while. +dns_snapshot() { # + local out="$1" tok zid page tp + tok="$(sops -d platform/externaldns/cloudflare-api-token.sops.yaml \ + | yq '.stringData."api-token"' | tr -d '"')" + [ -n "${tok}" ] && [ "${tok}" != "null" ] || die "could not read the Cloudflare token (SOPS/age key available?)" + zid="$(curl -sS -H "Authorization: Bearer ${tok}" \ + 'https://api.cloudflare.com/client/v4/zones?name=keithjackson.dev' | yq '.result[0].id' | tr -d '"')" + [ -n "${zid}" ] && [ "${zid}" != "null" ] || die "Cloudflare zone lookup failed" + page=1; : > "${out}" + while :; do + local r; r="$(curl -sS -H "Authorization: Bearer ${tok}" \ + "https://api.cloudflare.com/client/v4/zones/${zid}/dns_records?per_page=100&page=${page}")" + printf '%s' "${r}" | yq -o=json '.result' \ + | yq -r '.[] | (.type + "|" + .name + "|" + .content)' | tr -d '"' >> "${out}" + tp="$(printf '%s' "${r}" | yq '.result_info.total_pages' | tr -d '"')" + [ "${page}" -ge "${tp:-1}" ] && break + page=$((page+1)) + done + grep -vE '^\s*$' "${out}" | sort -o "${out}" + info "$(wc -l < "${out}") records -> ${out}" +} + +dns_diff() { # — returns non-zero if anything changed + if diff -u "$1" "$2" > "${SNAP_DIR}/dns.diff"; then + info "Cloudflare zone IDENTICAL — nothing created, modified or deleted." + return 0 + fi + warn "Cloudflare zone CHANGED:"; cat "${SNAP_DIR}/dns.diff" + return 1 +} + +# ============================================================================ steps +step_preflight() { + say "PREFLIGHT (read-only)" + info "nodes Ready: $(kubectl get nodes --no-headers | awk '$2=="Ready"' | wc -l)/$(kubectl get nodes --no-headers | wc -l)" + info "flux controllers Running: $(kubectl -n flux-system get pods --no-headers | awk '$3=="Running"' | wc -l)" + # Count BOTH groups the Kyverno chart ships. Grepping only 'kyverno.io' undercounts by 2 — the + # policy-report types are in the wgpolicyk8s.io group — which reads as "CRDs are missing" when the + # pre-apply in fact succeeded. Expect 22 for kyverno 3.9.x. + info "kyverno CRDs present: $(kubectl get crd --no-headers 2>/dev/null | grep -cE 'kyverno\.io|wgpolicyk8s\.io' || true) (expect 22 for kyverno 3.9.x)" + + local base owner filter s3 acme + base="$(cvar BASE_DOMAIN)"; owner="$(cvar EXTERNALDNS_TXT_OWNER_ID)" + filter="$(cvar EXTERNALDNS_DOMAIN_FILTER)"; s3="$(cvar S3_BACKUP_ENDPOINT)"; acme="$(cvar ACME_CLUSTER_ISSUER)" + say "cluster-vars guards" + info "BASE_DOMAIN = ${base}" + info "EXTERNALDNS_TXT_OWNER_ID = ${owner}" + info "EXTERNALDNS_DOMAIN_FILTER = ${filter}" + info "ACME_CLUSTER_ISSUER = ${acme}" + info "S3_BACKUP_ENDPOINT = ${s3}" + + # THE GUARD THAT MATTERS. Two clusters sharing one Cloudflare zone are isolated by txtOwnerId and + # by serving disjoint hostnames — NOT by domainFilters (finding F-DNS1: a subdomain filter drops the + # parent zone and publishes nothing). Both clusters inheriting the chart default `default` is the + # single root cause of the 2026-09-04 incident, in which prod deleted every one of dev's records. + [ -n "${owner}" ] || die "EXTERNALDNS_TXT_OWNER_ID is EMPTY — external-dns would fall back to 'default', which is dev's owner." + [ "${owner}" != "unset-see-cluster-vars" ] || die "EXTERNALDNS_TXT_OWNER_ID is still the base sentinel — the overlay did not apply." + [ "${owner}" != "default" ] || die "EXTERNALDNS_TXT_OWNER_ID is 'default' — that is DEV's owner id. Refusing." + info "owner id is distinct and non-default: OK" + + case "${s3}" in + ""|TBD-*) warn "S3_BACKUP_ENDPOINT is unset/placeholder ('${s3}'). velero AND apps-wave-0 consume it. + velero will be SKIPPED by the resume step. apps-wave-0 will still apply (the token IS defined, + so strict substitution succeeds) but anything writing to that endpoint fails at RUNTIME, not at + apply time — so a green apps-wave-0 does NOT mean backups work. Operator decision required." ;; + *) info "S3 backup endpoint set: OK" ;; + esac + + say "tiers" + local n + for n in flux-system cilium "${TIER_ORDER[@]}" velero; do + printf ' %-16s suspend=%-6s ready=%-6s %s\n' "$n" "$(k_susp "$n")" "$(k_ready "$n")" "$(k_msg "$n" | head -c 90)" + done + + say "HelmReleases not Ready" + kubectl get helmrelease -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SUSPEND:.spec.suspend,READY:.status.conditions[?\(@.type==\"Ready\"\)].status,MSG:.status.conditions[?\(@.type==\"Ready\"\)].message \ + --no-headers 2>/dev/null | grep -v ' True ' || info "(all Ready)" + + # SUSPENDED releases are invisible above: a suspended HelmRelease keeps whatever Ready condition it + # last had, so a STALE `Ready=True` hides the fact that Flux is not managing it at all. external-dns + # is deliberately in this state until `standup.sh externaldns` runs. + say "HelmReleases SUSPENDED (Flux is not reconciling these — a stale Ready=True means nothing)" + kubectl get helmrelease -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,SUSPEND:.spec.suspend --no-headers 2>/dev/null \ + | awk '$3=="true"' || true +} + +step_crds() { + say "CRD PRE-APPLY (idempotent)" + # Charts whose CRD bundle exceeds kubectl's 256 KiB annotation limit set crds.install=false, so + # nothing installs them and `platform` can never go Ready. See bootstrap/flux/preapply-crds.sh. + bash "${script_dir}/flux/preapply-crds.sh" +} + +step_unwedge() { + say "UNWEDGE STUCK HELM RELEASES" + # A Helm release left in uninstalling/pending-* cannot be advanced by helm-controller: it reports + # StateError ("unable to determine state for release with status 'uninstalling'") forever, which + # keeps `platform` (wait: true) not-Ready and every tier below it blocked. + # + # How prod got here: Kyverno's install timed out because its pods were crash-looping with no CRDs; + # install.remediation.retries:3 then fired an uninstall, and that uninstall timed out too. + # + # Pre-applied CRDs live OUTSIDE the Helm release and SURVIVE this — that is precisely why the + # pre-apply is a separate step and must run before this one. + local found=0 ns rel st + while read -r ns rel st; do + [ -n "${ns:-}" ] || continue + found=1 + warn "${ns}/${rel} is '${st}'" + [ "${CONFIRM_DESTRUCTIVE}" = "1" ] || { + info "DRY RUN. This DELETES the release's Kubernetes objects and lets Flux install cold." + info "Re-run with --confirm-destructive to act." + continue + } + say "clearing ${ns}/${rel}" + # --no-hooks and --wait=false: the WAIT is what hung the original remediation, so do not repeat it. + helm uninstall "${rel}" -n "${ns}" --no-hooks --wait=false || warn "helm uninstall returned non-zero; continuing to check state" + if helm list -n "${ns}" --all -q 2>/dev/null | grep -qx "${rel}"; then + warn "${ns}/${rel} still present after uninstall — inspect manually, NOT deleting release secrets by hand." + else + info "${ns}/${rel} cleared" + flux reconcile helmrelease "${rel}" -n "${ns}" --force --timeout=10m || warn "reconcile of ${ns}/${rel} did not complete in 10m" + fi + done < <( + for ns in $(kubectl get helmrelease -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\n"}{end}' | sort -u); do + helm list -n "${ns}" --all -o json 2>/dev/null \ + | yq -r '.[] | select(.status | test("uninstalling|pending-install|pending-upgrade|pending-rollback")) | "'"${ns}"' " + .name + " " + .status' 2>/dev/null || true + done + ) + [ "${found}" = "1" ] || info "no wedged releases found." +} + +step_externaldns() { + say "EXTERNAL-DNS — SAFE UPGRADE (prove the owner id BEFORE the pod can run)" + # THE HAZARD: a Helm upgrade sets replicas back to the chart default (1) and applies the new args in + # ONE operation. There is no window in which to check the result first. If --txt-owner-id is missing + # the pod starts as owner `default`, claims the other cluster's TXT registry entries, and under + # `policy: sync` DELETES the records behind them. That is the 2026-09-04 incident. + # + # THE FIX IS TO VERIFY IN ADVANCE, NOT AFTER: render the chart with the LIVE HelmRelease values and + # assert the flag appears. Rendering is deterministic and touches nothing, so this converts a race + # into a precondition. + local ns=external-dns rel=external-dns + local owner chart ver repo vals + owner="$(cvar EXTERNALDNS_TXT_OWNER_ID)" + [ -n "${owner}" ] && [ "${owner}" != "default" ] && [ "${owner}" != "unset-see-cluster-vars" ] \ + || die "refusing: EXTERNALDNS_TXT_OWNER_ID is '${owner}'" + + chart="$(kubectl -n "${ns}" get helmrelease "${rel}" -o jsonpath='{.spec.chart.spec.chart}')" + ver="$(kubectl -n "${ns}" get helmrelease "${rel}" -o jsonpath='{.spec.chart.spec.version}')" + repo="$(yq eval-all --no-doc 'select(.kind == "HelmRepository") | .spec.url' platform/externaldns/helmrepository.yaml | tr -d '"')" + [ -n "${repo}" ] || die "could not derive the external-dns chart repo url" + + vals="${SNAP_DIR}/ed-values.yaml" + kubectl -n "${ns}" get helmrelease "${rel}" -o jsonpath='{.spec.values}' | yq -P > "${vals}" + info "chart=${chart} version=${ver} repo=${repo}" + info "live spec.values.txtOwnerId = $(yq '.txtOwnerId' "${vals}")" + + say "rendering the chart with the LIVE values to prove the flag" + local rendered="${SNAP_DIR}/ed-rendered.yaml" + helm template "${rel}" "${chart}" --repo "${repo}" --version "${ver}" -n "${ns}" -f "${vals}" > "${rendered}" + local args; args="$(yq -r 'select(.kind == "Deployment") | .spec.template.spec.containers[0].args[]' "${rendered}")" + printf '%s\n' "${args}" | sed 's/^/ /' + printf '%s\n' "${args}" | grep -qx -- "--txt-owner-id=${owner}" \ + || die "the render does NOT contain --txt-owner-id=${owner}. + An upgrade would start external-dns as owner 'default' and delete the other cluster's records. + REFUSING. Fix the chart values key before retrying." + info "PROVEN: the upgrade will carry --txt-owner-id=${owner}" + + say "snapshotting the Cloudflare zone (baseline for the post-check)" + dns_snapshot "${SNAP_DIR}/dns-before.txt" + + say "resuming + forcing the upgrade" + kubectl -n "${ns}" get helmrelease "${rel}" -o jsonpath='{.spec.suspend}' | grep -q true \ + && flux resume helmrelease "${rel}" -n "${ns}" || info "(HelmRelease was not suspended)" + flux reconcile helmrelease "${rel}" -n "${ns}" --force --timeout=10m || warn "reconcile did not complete in 10m" + + say "verifying the LIVE deployment args" + local live; live="$(kubectl -n "${ns}" get deploy "${rel}" -o jsonpath='{.spec.template.spec.containers[0].args}' 2>/dev/null || true)" + printf ' %s\n' "${live}" + printf '%s' "${live}" | grep -q -- "--txt-owner-id=${owner}" \ + || { kubectl -n "${ns}" scale deploy "${rel}" --replicas=0 || true + die "live args LACK --txt-owner-id=${owner}. Scaled to 0. Investigate before scaling up."; } + info "live deployment carries --txt-owner-id=${owner}" + + say "post-check: did anything in the shared zone change?" + sleep 45 # external-dns --interval=1m; give it a full cycle to act before judging it safe + dns_snapshot "${SNAP_DIR}/dns-after.txt" + dns_diff "${SNAP_DIR}/dns-before.txt" "${SNAP_DIR}/dns-after.txt" \ + || warn "records changed — expected ONLY additions under this cluster's own BASE_DOMAIN ($(cvar BASE_DOMAIN)). + Any change to a name outside it, or to a TXT entry owned by another cluster, is a REGRESSION." +} + +step_resume() { + say "RESUME TIERS (dependency order)" + local n s3; s3="$(cvar S3_BACKUP_ENDPOINT)" + for n in "${TIER_ORDER[@]}"; do + if [ "$(k_susp "$n")" = "true" ]; then + info "resuming $n"; flux resume kustomization "$n" --timeout=1s >/dev/null 2>&1 || true + else + info "$n already unsuspended" + fi + wait_ready "$n" 900 || { warn "stopping at $n — a partial, understood bring-up beats a forced green."; return 1; } + done + + case "${s3}" in + ""|TBD-*) warn "velero left SUSPENDED: S3_BACKUP_ENDPOINT='${s3}' is not a usable endpoint. + Resuming it would create a BackupStorageLocation that can never reach a bucket, i.e. a tier + that reports Ready-ish while backups silently do not exist. Set the real prod endpoint first." ;; + *) info "resuming velero"; flux resume kustomization velero --timeout=1s >/dev/null 2>&1 || true + wait_ready velero 900 || warn "velero did not converge" ;; + esac +} + +step_verify() { + say "VERIFY" + local base; base="$(cvar BASE_DOMAIN)" + info "BASE_DOMAIN = ${base}" + say "kustomizations" + kubectl get kustomization -A \ + -o custom-columns=NAME:.metadata.name,SUSPEND:.spec.suspend,READY:.status.conditions[?\(@.type==\"Ready\"\)].status --no-headers + say "certificates (must be Ready before any hostname is trusted)" + kubectl get certificate -A --no-headers 2>/dev/null || info "(none yet)" + say "pods not Running/Completed" + kubectl get pods -A --no-headers | awk '$4!="Running" && $4!="Completed"' || true + info "total pods: $(kubectl get pods -A --no-headers | wc -l)" + say "gateway / LoadBalancer VIPs" + kubectl get svc -A --field-selector spec.type=LoadBalancer --no-headers 2>/dev/null || true + say "published DNS for this cluster's own domain" + dns_snapshot "${SNAP_DIR}/dns-verify.txt" + info "records under ${base}: $(grep -c "${base}" "${SNAP_DIR}/dns-verify.txt" || true)" + warn "confirm every TXT registry entry for ${base} reads this cluster's owner id ($(cvar EXTERNALDNS_TXT_OWNER_ID)), + and that NO record outside ${base} was touched. Compare against ${SNAP_DIR}/dns-before.txt." +} + +# ============================================================================ dispatch +for s in "${STEPS[@]}"; do + case "$s" in + preflight) step_preflight ;; + crds) step_crds ;; + unwedge) step_unwedge ;; + externaldns) step_externaldns ;; + resume) step_resume ;; + verify) step_verify ;; + all) step_preflight; step_crds; step_unwedge; step_externaldns; step_resume; step_verify ;; + *) die "unknown step '$s' (preflight crds unwedge externaldns resume verify all)" ;; + esac +done + +say "done: ${STEPS[*]}" diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index dfe2512..f169050 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -10,6 +10,11 @@ hardware until the blockers in §7 are closed and `make -C ci gate` is green. > **R-ENV-01/02/03** — the very requirements this document serves. We proceed with prep work > knowingly; see `DECISIONS-LOG.md` (2026-08-31 Phase-7 entry). +> **Looking for the commands?** This document is the *decision record* — why prod is built the way it +> is, what was measured, and which hypotheses are dead. The **executable sequence** lives in +> [`PROD-STANDUP.md`](PROD-STANDUP.md): every command, in order, driven by `bootstrap/standup.sh`. +> Start there; come back here for the reasoning behind any step. + Prod is **3× mini-PC bare metal** ([spec docs/02](../../home-network-k3s/docs/02-environments-and-provisioning.md)). The dev path (OpenTofu → Proxmox VM → cloud-init) **does not apply**: there is no hypervisor API to call. Stage 1 "Create" is a human imaging a box. Stages 2–4 (baseline, cluster-join, manage) reuse diff --git a/infra/PROD-STANDUP.md b/infra/PROD-STANDUP.md new file mode 100644 index 0000000..1d995ee --- /dev/null +++ b/infra/PROD-STANDUP.md @@ -0,0 +1,233 @@ +# Prod cluster standup — the complete command sequence + +**Every command below is copy-pasteable and runs from the repo root**, through the pinned dev +container. Nothing here needs a command invented on the spot; that is the point. If a step is not in +this file it is not part of standup. + +State as of **2026-09-05**: phases **A** and **B** are DONE on the current hardware. The cluster is +live (k3s v1.36.4 HA, 3 etcd members, kube-vip, Cilium 1.20.1, Longhorn) with `platform` reconciling +and **8/9 HelmReleases Ready, 76/76 pods Running**. Outstanding work is **phase C**, blocked on the +two items in [§Blockers](#blockers). + +> **The one guard that matters.** Dev is LIVE and shares the `keithjackson.dev` Cloudflare zone with +> prod. On 2026-09-04 prod deleted every one of dev's public DNS records. `bootstrap/standup.sh` +> therefore **requires** `--expect-server` and compares it against the live API server before running +> anything. There is no override flag. Do not add one. + +## Constants + +| | | +|---|---| +| kube-API (kube-vip VIP) | `https://192.168.2.9:6443` | +| Gateway / ingress VIP | `192.168.2.20` | +| Nodes | `k3s-prod-1/2/3` — `192.168.2.10 / .11 / .12` | +| Break-glass kubeconfig | `infra/ansible/.cache/prod.kubeconfig` (cert-based; survives OIDC/DNS failure) | +| GitOps source (interim) | `http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git` branch `main` | +| `BASE_DOMAIN` | `prod.home.keithjackson.dev` — **not** the apex; see §Blockers | + +Container paths: the repo is mounted at `/work`, so kubeconfig paths passed to the script are +**repo-relative** (`infra/ansible/.cache/prod.kubeconfig`). + +--- + +## Phase A — bare metal to a joined cluster (Ansible) + +Prerequisites that are **not** commands: firmware/BIOS (§1), disk layout (§2), OS install (§3), +network (§4), and SSH + the vaulted `ansible_become_password` in +`inventory/group_vars/k3s_prod/vault.yml` (§5). Ansible cannot escalate on prod without that vault +file. See `infra/PROD-PROVISIONING.md`. + +**Install the backports kernel first.** All three prod nodes need ≥6.13 because HP's R21 firmware +publishes a corrupt EFI Memory Attributes Table and Linux 6.12's `efi_memattr_init()` does not +validate it, reserving everything above the table (fixed upstream in 6.13 by `8fbe4c49c0cc`, absent +from `linux-6.12.y`). Prod runs `7.1.8+deb13-amd64` from trixie-backports. + +```bash +timeout 3600 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \ + ansible-playbook /work/infra/ansible/playbooks/node-baseline.yml \ + -i /work/infra/ansible/inventory/hosts.ini -e k3s_hosts=k3s_prod --tags kernel +``` + +Then the full chain — baseline, k3s HA, Cilium, verify, Flux hand-off. **All three `-e` flags are +required**: a play's `hosts:` is templated before any host exists, so `group_vars` cannot retarget it. + +```bash +timeout 3600 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \ + ansible-playbook /work/infra/ansible/playbooks/site.yml \ + -i /work/infra/ansible/inventory/hosts.ini \ + -e k3s_hosts=k3s_prod -e k3s_primary_hosts=k3s_prod_primary -e k3s_secondary_hosts=k3s_prod_secondary +``` + +kube-vip is a separate play (not imported by `site.yml`): + +```bash +timeout 1800 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \ + ansible-playbook /work/infra/ansible/playbooks/kube-vip.yml \ + -i /work/infra/ansible/inventory/hosts.ini -e k3s_hosts=k3s_prod +``` + +Hardware health (read-only; `node_health` asserts the memattr fix took and that all RAM is addressed). + +> **The flag is `node_health_hosts`, NOT `k3s_hosts`.** Diagnostic plays deliberately keep their own +> target var so they stay independently pointable while a bring-up run is scoped elsewhere +> (`node-health.yml` → `node_health_hosts`, `memattr-survey.yml` → `memattr_survey_hosts`, +> `vpn.egress.gateway.yml` → `vpn_egress_gateway_hosts`). The consequence is a **silent** one: passing +> `k3s_hosts=k3s_prod` here does not error, it falls through to the `k3s_servers` default and runs +> against **DEV's three nodes**. Verified — this exact mistake was made while writing this file and +> caught only by `--list-hosts`. **Always dry-run the target first:** +> +> ```bash +> timeout 200 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \ +> ansible-playbook /work/infra/ansible/playbooks/node-health.yml \ +> -i /work/infra/ansible/inventory/hosts.ini -e node_health_hosts=k3s_prod --list-hosts +> ``` + +```bash +timeout 900 infra/devcontainer/dev.sh run env ANSIBLE_CONFIG=/work/infra/ansible/ansible.cfg \ + ansible-playbook /work/infra/ansible/playbooks/node-health.yml \ + -i /work/infra/ansible/inventory/hosts.ini -e node_health_hosts=k3s_prod +``` + +## Phase B — Flux bootstrap + +`flux-bootstrap.yml` (last play of `site.yml`) shells to `bootstrap/flux/bootstrap.sh`, the single +documented entrypoint. That script now runs the CRD pre-apply as step **[2/4]**, so a cold bootstrap +no longer strands `platform`. The **age private key Secret must exist first** — bootstrap verifies it +and refuses to continue rather than silently skipping decryption +(`bootstrap/sops/age-key-secret.md`). + +## Phase C — bring the reconciliation graph up + +One script, six steps, each independently runnable and idempotent. **Run them in this order.** + +```bash +K=infra/ansible/.cache/prod.kubeconfig +S=https://192.168.2.9:6443 +``` + +### C1 — preflight (read-only, changes nothing) + +Asserts the guards that prevent a repeat of the DNS incident and reports every tier's state. **Fails +closed** if `EXTERNALDNS_TXT_OWNER_ID` is empty, still the base sentinel, or `default` (dev's owner). + +```bash +timeout 300 infra/devcontainer/dev.sh run bootstrap/standup.sh \ + --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 preflight +``` + +### C2 — pre-apply chart CRDs (idempotent) + +```bash +timeout 600 infra/devcontainer/dev.sh run bootstrap/standup.sh \ + --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 crds +``` + +### C3 — unwedge stuck Helm releases + +A release left in `uninstalling` / `pending-*` cannot be advanced by helm-controller: it reports +`StateError` forever, so `platform` (`wait: true`) never goes Ready and all 10 tiers below it stall. +**Dry run by default** — it prints what it would do and changes nothing: + +```bash +timeout 600 infra/devcontainer/dev.sh run bootstrap/standup.sh \ + --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 unwedge +``` + +To act (this **deletes** the release's Kubernetes objects and lets Flux install cold). Pre-applied +CRDs live **outside** the Helm release and survive, which is exactly why C2 comes first: + +```bash +timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \ + --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 \ + --confirm-destructive unwedge +``` + +### C4 — external-dns safe upgrade + +**The most dangerous step in the repo, and the ordering is the entire mitigation.** A Helm upgrade +resets `replicas` to the chart default (1) and applies the new args in ONE operation — there is no +window in which to check the result first. If `--txt-owner-id` were missing, the pod starts as owner +`default`, claims dev's TXT registry entries, and under `policy: sync` **deletes the records behind +them**. + +So this step **proves the outcome before causing it**: it renders the chart with the *live* +HelmRelease values and asserts `--txt-owner-id=` appears, aborting if not. Rendering is +deterministic and touches nothing, which converts a race into a precondition. It then snapshots the +Cloudflare zone, upgrades, re-verifies the live args (scaling to 0 and failing if they are wrong), +and diffs the zone. + +```bash +timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \ + --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 externaldns +``` + +Expected diff: **additions only, all under `prod.home.keithjackson.dev`, all TXT entries owner +`prod`.** Any change to a name outside that subtree, or to a TXT entry owned by another cluster, is a +regression — stop and investigate. + +### C5 — resume the tiers, in dependency order + +`platform → platform-config → {oauth2-proxy, headlamp, homepage, crafty, apps-wave-0, apps-vms} → +apps-wave-1 → apps-wave-2`. Waits for each to go Ready and **stops at the first that does not** — a +partial, understood bring-up beats a forced green. `velero` is skipped while `S3_BACKUP_ENDPOINT` is +a placeholder (see §Blockers). Long-running; expect it to background. + +```bash +timeout 3600 infra/devcontainer/dev.sh run bootstrap/standup.sh \ + --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 resume +``` + +### C6 — verify + +```bash +timeout 900 infra/devcontainer/dev.sh run bootstrap/standup.sh \ + --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 verify +``` + +### Everything at once + +Runs C1→C6 with `unwedge` in **dry-run** mode (no `--confirm-destructive`), so it stops short of the +destructive step by design: + +```bash +timeout 3600 infra/devcontainer/dev.sh run bootstrap/standup.sh \ + --kubeconfig infra/ansible/.cache/prod.kubeconfig --expect-server https://192.168.2.9:6443 all +``` + +--- + +## Blockers + +### 1. `S3_BACKUP_ENDPOINT` is an intentional placeholder — operator decision required + +Prod sets `TBD-PROD-S3-BACKUP-ENDPOINT` on purpose: base defaults are the **dev** values, so an +omitted override would make prod silently inherit dev's address (`http://192.168.1.9:9000`), which +prod cannot reach anyway. A loud placeholder beats a silent wrong answer. + +It is consumed by **`velero` and `apps-wave-0`**. `velero` is skipped by C5. **`apps-wave-0` still +applies** — the token *is* defined, so strict substitution succeeds — but anything writing to that +endpoint fails at **runtime**, not at apply time. **A green `apps-wave-0` does not mean backups +work.** Until a real prod endpoint is set, treat prod as having **no** backup coverage. + +### 2. Prod is on `prod.home.keithjackson.dev`, not the apex + +The operator's target layout is dev under `*.dev.home.keithjackson.dev` and prod on the **apex**. +Prod cannot take the apex until dev vacates it, because both clusters would request a +`letsencrypt-prod` wildcard for the *same name set* and race each other's `_acme-challenge` TXT +records, so neither reliably solves DNS-01. + +**This does not block phase C.** Prod on its own subtree with `txtOwnerId: prod` collides with +nothing, and standing it up first is strictly better: it proves the whole stack before the flip, so +the cutover becomes a DNS/TLS change on a known-good cluster instead of a bring-up and a cutover at +once. The apex move is `infra/PROD-PROVISIONING.md` §10a, sequence **D** (dev vacates) then **P**. + +--- + +## Not covered here + +- **Dev's migration off the apex** — §10a sequence D. Touches Keycloak's issuer, ~12 OIDC client + redirect URIs, and a rolling k3s restart that invalidates every OIDC kubeconfig. Operator-present. +- **Prod's GitOps cutover** (P5) from the workstation bootstrap Gitea to prod's own in-cluster Gitea. + `clusters/prod/kustomization.yaml` owns that procedure. +- **Six L2 conformance tests hardcode the apex** and must be parameterized per environment before + §9's "suite passes green against the prod cluster" can be claimed.