2 Commits

Include renames

Author	SHA1	Message	Date
jakman85andClaude Opus 5	1b056dc9d9	fix(prod): break the CDI cold-start deadlock, and make preflight catch its class ... `platform-config` stopped dead on prod: StorageProfile/longhorn dry-run failed: no matches for kind "StorageProfile" in version "cdi.kubevirt.io/v1beta1" A TRUE DEADLOCK, not a transient. platform-config applies the CDI CR *and* platform/kubevirt/cdi-storageprofiles.yaml in the SAME pass. StorageProfile's CRD does not exist until cdi-operator has reconciled that CR — and Flux server-side dry-runs EVERY object before applying ANY of them, so the unknown kind aborts the whole tier, the CR is never applied, the CRD is never created, and platform-config can never converge. Every tier below it stays blocked. Same root cause as the Kyverno CRD bundle, different operator: Flux cannot order a resource against a CRD that one of its own siblings is responsible for creating. It is INVISIBLE on an established cluster — dev's CDI CRDs have been there for months — and fatal on a cold one. Observed on prod: cdi-operator Available for 12h, no CDI CR, no storageprofiles CRD. `crds` now breaks it by applying ONLY the CDI CR — the minimum — and leaving every other object to Flux. The CR carries no ${} tokens, so applying it here cannot diverge from what Flux would render. It refuses if the `cdi` namespace is absent (platform has not run yet) rather than applying into nothing, waits for the CRD, and requires it to reach Established. On prod the CRD appeared in 10s and platform-config was unblocked. `preflight` now reports the CRDs that later tiers apply resources against (storageprofiles/cdis/kubevirts/volumesnapshotclasses) and dumps the cdi namespace when one is missing, so this is caught in seconds instead of after `resume` burns its 900s wait discovering it. It also names the stale-discovery- cache variant, which presents identically but clears on a forced reconcile. THE DURABLE FIX IS A REPO CHANGE, deliberately NOT made here: move cdi-storageprofiles.yaml out of platform-config into its own Kustomization with dependsOn: platform-config, so the CR lands one reconcile before the resources needing its CRDs. That edits the reconciliation graph both clusters share, and prod currently reads a pinned revision from the bootstrap Gitea while dev is mid-migration on a separate branch — so it wants doing deliberately, not at 02:00 to unblock a bring-up. Until it lands, C2 is a prerequisite for C5 on any cold cluster, and PROD-STANDUP.md says so. Refs R-GITOPS-01, R-ENV-02. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-05 08:05:46 -05:00
jakman85andClaude Opus 5	4160fee6ce	feat(prod): standup.sh + PROD-STANDUP.md — the complete standup sequence, in git ... The last steps of a prod bring-up were living in shell history: resume the tiers in the right order, clear a wedged HelmRelease, fix external-dns without deleting the other cluster's DNS. That is the same failure the Kyverno CRD pre-apply had, so it gets the same treatment. bootstrap/standup.sh — environment-agnostic (R-ENV-02), no dev/prod fork. Six idempotent steps: preflight, crds, unwedge, externaldns, resume, verify. THE GUARD IS THE POINT. --expect-server is MANDATORY and is compared against the live API server before any step runs; there is no override flag. Dev is live and shares the keithjackson.dev Cloudflare zone with prod, and on 2026-09-04 prod deleted every one of dev's public records. Negative control run: prod's kubeconfig with dev's server URL refuses and changes nothing. preflight fails closed if EXTERNALDNS_TXT_OWNER_ID is empty, still the base sentinel, or `default` (dev's owner) — the exact condition that caused the incident. It also surfaces SUSPENDED HelmReleases separately, because a suspended release keeps its last Ready condition and a stale `Ready=True` otherwise hides that Flux is not managing it at all (external-dns today). externaldns is the dangerous step, and the ordering IS the mitigation. A Helm upgrade resets replicas to 1 and applies new args in ONE operation, so there is no window to check the result first; if --txt-owner-id were missing the pod starts as owner `default` and syncs away dev's records. So the step PROVES THE OUTCOME BEFORE CAUSING IT: render the chart with the LIVE HelmRelease values and assert --txt-owner-id=<owner> appears, aborting if not. Rendering is deterministic and touches nothing, which turns a race into a precondition. Then snapshot Cloudflare, upgrade, re-verify live args (scale to 0 and fail if wrong), and diff the zone. resume walks the dependency order and STOPS at the first tier that does not go Ready — a partial understood bring-up beats a forced green. velero is skipped while S3_BACKUP_ENDPOINT is the TBD placeholder. TWO BUGS CAUGHT WHILE VALIDATING THIS, both by measuring instead of assuming: 1. preflight counted Kyverno CRDs with grep 'kyverno.io' and reported 20 of 22 — the policy-report types are in the wgpolicyk8s.io group. A preflight that under-reports reads as "the pre-apply failed" when it succeeded. Fixed and re-run: 22. 2. The documented node-health invocation used -e k3s_hosts=k3s_prod and SILENTLY RAN AGAINST DEV'S THREE NODES. Diagnostic plays deliberately keep their own target var (node_health_hosts, memattr_survey_hosts, vpn_egress_gateway_hosts) so they stay pointable while a bring-up is scoped elsewhere — a good design I did NOT change. The hazard is that the wrong flag does not error, it falls through to the dev default. Caught only by --list-hosts; the doc now leads with that warning and a dry-run command. Verified after fixing: node-health and all six site.yml plays target prod. .claude/settings.local.json — permission rules for the above. Every entry hardcodes prod's kubeconfig AND prod's --expect-server (or prod's ansible groups), so the permission strings are themselves prod-scoped and cannot be turned against dev. Deliberately NO rule for `dev.sh run bash -c *`, which would have been blanket cluster access wearing a narrow-looking mask. Refs R-ENV-02, R-GITOPS-01, R-SEC-02. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-05 00:37:48 -05:00