jakman85andClaude Opus 5
4160fee6ce
The last steps of a prod bring-up were living in shell history: resume the tiers in the right order, clear a wedged HelmRelease, fix external-dns without deleting the other cluster's DNS. That is the same failure the Kyverno CRD pre-apply had, so it gets the same treatment. bootstrap/standup.sh — environment-agnostic (R-ENV-02), no dev/prod fork. Six idempotent steps: preflight, crds, unwedge, externaldns, resume, verify. THE GUARD IS THE POINT. --expect-server is MANDATORY and is compared against the live API server before any step runs; there is no override flag. Dev is live and shares the keithjackson.dev Cloudflare zone with prod, and on 2026-09-04 prod deleted every one of dev's public records. Negative control run: prod's kubeconfig with dev's server URL refuses and changes nothing. preflight fails closed if EXTERNALDNS_TXT_OWNER_ID is empty, still the base sentinel, or `default` (dev's owner) — the exact condition that caused the incident. It also surfaces SUSPENDED HelmReleases separately, because a suspended release keeps its last Ready condition and a stale `Ready=True` otherwise hides that Flux is not managing it at all (external-dns today). externaldns is the dangerous step, and the ordering IS the mitigation. A Helm upgrade resets replicas to 1 and applies new args in ONE operation, so there is no window to check the result first; if --txt-owner-id were missing the pod starts as owner `default` and syncs away dev's records. So the step PROVES THE OUTCOME BEFORE CAUSING IT: render the chart with the LIVE HelmRelease values and assert --txt-owner-id=<owner> appears, aborting if not. Rendering is deterministic and touches nothing, which turns a race into a precondition. Then snapshot Cloudflare, upgrade, re-verify live args (scale to 0 and fail if wrong), and diff the zone. resume walks the dependency order and STOPS at the first tier that does not go Ready — a partial understood bring-up beats a forced green. velero is skipped while S3_BACKUP_ENDPOINT is the TBD placeholder. TWO BUGS CAUGHT WHILE VALIDATING THIS, both by measuring instead of assuming: 1. preflight counted Kyverno CRDs with grep 'kyverno.io' and reported 20 of 22 — the policy-report types are in the wgpolicyk8s.io group. A preflight that under-reports reads as "the pre-apply failed" when it succeeded. Fixed and re-run: 22. 2. The documented node-health invocation used -e k3s_hosts=k3s_prod and SILENTLY RAN AGAINST DEV'S THREE NODES. Diagnostic plays deliberately keep their own target var (node_health_hosts, memattr_survey_hosts, vpn_egress_gateway_hosts) so they stay pointable while a bring-up is scoped elsewhere — a good design I did NOT change. The hazard is that the wrong flag does not error, it falls through to the dev default. Caught only by --list-hosts; the doc now leads with that warning and a dry-run command. Verified after fixing: node-health and all six site.yml plays target prod. .claude/settings.local.json — permission rules for the above. Every entry hardcodes prod's kubeconfig AND prod's --expect-server (or prod's ansible groups), so the permission strings are themselves prod-scoped and cannot be turned against dev. Deliberately NO rule for `dev.sh run bash -c *`, which would have been blanket cluster access wearing a narrow-looking mask. Refs R-ENV-02, R-GITOPS-01, R-SEC-02. Co-Authored-By: Claude Opus 5 <[email protected]>
infra/ — node provisioning and the toolchain
OpenTofu (Proxmox VE) + cloud-init + Ansible bring up the 3-node k3s/Cilium dev cluster.
Spec: ../home-network-k3s/docs/02-environments-and-provisioning.md.
| Path | What |
|---|---|
tofu/ |
The "Create" stage: 3 cloud-init'd VMs on the Proxmox box (bpg/proxmox provider) |
cloud-init/ |
First-boot baseline shipped as a Proxmox user-data snippet |
ansible/ |
The "Baseline"/"Configure" stages: k3s HA, Cilium, kube-vip, the VPN egress gateway |
devcontainer/ |
The pinned toolchain (tofu, ansible, sops, age, flux, kubectl, helm and the whole CI gate toolchain) |
cluster.sh |
RETIRED — drove the old libvirt VMs; does not work against Proxmox |
PHASE1-STATUS.md |
Current state of the cluster and how to reconnect to it |
PHASE1-SCOPE.md |
Historical Phase-1 scoping doc (libvirt era — superseded) |
Where the cluster actually lives
The dev cluster is not on this workstation. It runs on a Proxmox VE host; the nodes sit on the Proxmox LAN and are reached across a pfSense NAT:
| Nodes | k3s-dev-1/2/3 at 192.168.1.50/.51/.52 — 6 vCPU, 24 GiB, Debian 13, k3s v1.36.4 |
| kube-API | kube-vip VIP 192.168.1.49:6443, reached from the workstation via NAT 192.168.2.50:6443 |
| Node SSH | pfSense NAT 192.168.2.102:2250/2251/2252 → node :22 (Ansible only) |
| Ingress VIP | 192.168.1.53 (Cilium L2, shared Gateway); .53–.60 reserved for LB VIPs |
The earlier libvirt/KVM cluster on 10.73.0.0/24 is retired. Any doc or command referencing
virsh, qemu:///system, or 10.73.0.x for the cluster is historical.