`platform-config` stopped dead on prod:
StorageProfile/longhorn dry-run failed: no matches for kind
"StorageProfile" in version "cdi.kubevirt.io/v1beta1"
A TRUE DEADLOCK, not a transient. platform-config applies the CDI CR *and*
platform/kubevirt/cdi-storageprofiles.yaml in the SAME pass. StorageProfile's
CRD does not exist until cdi-operator has reconciled that CR — and Flux
server-side dry-runs EVERY object before applying ANY of them, so the unknown
kind aborts the whole tier, the CR is never applied, the CRD is never created,
and platform-config can never converge. Every tier below it stays blocked.
Same root cause as the Kyverno CRD bundle, different operator: Flux cannot order
a resource against a CRD that one of its own siblings is responsible for
creating. It is INVISIBLE on an established cluster — dev's CDI CRDs have been
there for months — and fatal on a cold one. Observed on prod: cdi-operator
Available for 12h, no CDI CR, no storageprofiles CRD.
`crds` now breaks it by applying ONLY the CDI CR — the minimum — and leaving
every other object to Flux. The CR carries no ${} tokens, so applying it here
cannot diverge from what Flux would render. It refuses if the `cdi` namespace is
absent (platform has not run yet) rather than applying into nothing, waits for
the CRD, and requires it to reach Established. On prod the CRD appeared in 10s
and platform-config was unblocked.
`preflight` now reports the CRDs that later tiers apply resources against
(storageprofiles/cdis/kubevirts/volumesnapshotclasses) and dumps the cdi
namespace when one is missing, so this is caught in seconds instead of after
`resume` burns its 900s wait discovering it. It also names the stale-discovery-
cache variant, which presents identically but clears on a forced reconcile.
THE DURABLE FIX IS A REPO CHANGE, deliberately NOT made here: move
cdi-storageprofiles.yaml out of platform-config into its own Kustomization with
dependsOn: platform-config, so the CR lands one reconcile before the resources
needing its CRDs. That edits the reconciliation graph both clusters share, and
prod currently reads a pinned revision from the bootstrap Gitea while dev is
mid-migration on a separate branch — so it wants doing deliberately, not at
02:00 to unblock a bring-up. Until it lands, C2 is a prerequisite for C5 on any
cold cluster, and PROD-STANDUP.md says so.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
The last steps of a prod bring-up were living in shell history: resume the
tiers in the right order, clear a wedged HelmRelease, fix external-dns without
deleting the other cluster's DNS. That is the same failure the Kyverno CRD
pre-apply had, so it gets the same treatment.
bootstrap/standup.sh — environment-agnostic (R-ENV-02), no dev/prod fork. Six
idempotent steps: preflight, crds, unwedge, externaldns, resume, verify.
THE GUARD IS THE POINT. --expect-server is MANDATORY and is compared against
the live API server before any step runs; there is no override flag. Dev is
live and shares the keithjackson.dev Cloudflare zone with prod, and on
2026-09-04 prod deleted every one of dev's public records. Negative control
run: prod's kubeconfig with dev's server URL refuses and changes nothing.
preflight fails closed if EXTERNALDNS_TXT_OWNER_ID is empty, still the base
sentinel, or `default` (dev's owner) — the exact condition that caused the
incident. It also surfaces SUSPENDED HelmReleases separately, because a
suspended release keeps its last Ready condition and a stale `Ready=True`
otherwise hides that Flux is not managing it at all (external-dns today).
externaldns is the dangerous step, and the ordering IS the mitigation. A Helm
upgrade resets replicas to 1 and applies new args in ONE operation, so there is
no window to check the result first; if --txt-owner-id were missing the pod
starts as owner `default` and syncs away dev's records. So the step PROVES THE
OUTCOME BEFORE CAUSING IT: render the chart with the LIVE HelmRelease values and
assert --txt-owner-id=<owner> appears, aborting if not. Rendering is
deterministic and touches nothing, which turns a race into a precondition. Then
snapshot Cloudflare, upgrade, re-verify live args (scale to 0 and fail if
wrong), and diff the zone.
resume walks the dependency order and STOPS at the first tier that does not go
Ready — a partial understood bring-up beats a forced green. velero is skipped
while S3_BACKUP_ENDPOINT is the TBD placeholder.
TWO BUGS CAUGHT WHILE VALIDATING THIS, both by measuring instead of assuming:
1. preflight counted Kyverno CRDs with grep 'kyverno.io' and reported 20 of 22
— the policy-report types are in the wgpolicyk8s.io group. A preflight that
under-reports reads as "the pre-apply failed" when it succeeded. Fixed and
re-run: 22.
2. The documented node-health invocation used -e k3s_hosts=k3s_prod and
SILENTLY RAN AGAINST DEV'S THREE NODES. Diagnostic plays deliberately keep
their own target var (node_health_hosts, memattr_survey_hosts,
vpn_egress_gateway_hosts) so they stay pointable while a bring-up is scoped
elsewhere — a good design I did NOT change. The hazard is that the wrong flag
does not error, it falls through to the dev default. Caught only by
--list-hosts; the doc now leads with that warning and a dry-run command.
Verified after fixing: node-health and all six site.yml plays target prod.
.claude/settings.local.json — permission rules for the above. Every entry
hardcodes prod's kubeconfig AND prod's --expect-server (or prod's ansible
groups), so the permission strings are themselves prod-scoped and cannot be
turned against dev. Deliberately NO rule for `dev.sh run bash -c *`, which
would have been blanket cluster access wearing a narrow-looking mask.
Refs R-ENV-02, R-GITOPS-01, R-SEC-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
`platform/kyverno/admission/helmrelease.yaml` sets `crds: {install: false}`
because the CRD bundle is 5,636,155 bytes, far past kubectl's 256 KiB
last-applied-configuration limit — a client-side apply fails with
"metadata.annotations: Too long" and STALLS the HelmRelease.
The consequence nobody automated: with CRD install off, NOTHING installed
them. `platform` has `wait: true` and every other tier chains transitively
through it, so on a cold cluster Kyverno's API types never exist, `platform`
never goes Ready, and all 10 downstream tiers stall. Dev only ever had these
CRDs because they were applied BY HAND — exactly the undocumented manual step
this repo exists to eliminate.
New `bootstrap/flux/preapply-crds.sh`, wired into bootstrap.sh as step [2/4]
(after `flux install`, before the graph is applied) and standalone-safe so an
already-bootstrapped cluster can be unblocked without re-running bootstrap.
NO SECOND PIN: all four chart coordinates are read with yq out of the same
HelmRelease/HelmRepository pair Flux reconciles. The version is passed through
as the CONSTRAINT ("3.9.x"), not a resolved tag — helm and helm-controller both
use Masterminds/semver, so the render is the version Flux installs. Renovate
bumps one line and this step follows; there is nothing here to drift from.
Only `kind: CustomResourceDefinition` documents are applied. The by-hand
command recorded in the HelmRelease comment piped the WHOLE template into
kubectl, which would create Deployments/webhooks outside Helm's ownership and
then fight the HelmRelease over field managers.
THREE GUARDS, because a silent skip here reproduces a failure mode this repo
has been bitten by repeatedly:
1. any derived coordinate that is empty/null/multi-valued aborts;
2. rendered CRD count must be > 0 — this caught a real bug in the first
draft: on an empty render `yq .metadata.name` emits one BLANK line, which
counted as 1 CRD and sailed past the guard into an apply of an empty file
returning 0, i.e. the exact silent-success trap. Blank/null lines are now
dropped and a chart with no CRDs exits 1;
3. every applied CRD is waited on for condition=Established, so "kubectl
returned 0" is not the only evidence.
No opt-out flag, deliberately: a skipped CRD apply yields a cluster that looks
bootstrapped and reconciles nothing.
`roles/flux_bootstrap` is untouched — it already shells to bootstrap.sh, the
documented single entrypoint, so it inherits the step with no dev/prod fork
(R-ENV-02). A second role task would double-apply.
MEASURED, not assumed — reproduced under the PINNED toolchain (helm 3.16.0,
yq 4.44.6) in infra/devcontainer, matching an independent run on helm 3.20.0
byte-for-byte: 3.9.x resolves to 3.9.0, 22 CRDs, 5,636,155 bytes CRD-only
(5,870,019 full render). Largest single CRD document is 1,378,544 bytes —
under etcd's 1.5 MiB default, but not by much; worth watching on chart bumps.
NOT YET EXERCISED against a live API server: the server-side apply and the
Established wait were tested with a stub kubectl. velero and
kube-prometheus-stack also say "CRDs pre-applied" but both still get theirs
from Helm's native crds/ dir on FIRST install, so neither is a cold-start
blocker; CRD_HELMRELEASE makes extending this a one-liner if that changes.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
Autonomous prod bring-up, operator away. Three changes that together clear the last
blockers before `flux bootstrap`.
=== 1. TEMPORARY BOOTSTRAP GITEA — the last hard blocker, cleared ===
clusters/prod's source URL could not work: Gitea runs ON the dev cluster (the one slated for
decommission) and split-horizon DNS resolves gitea.home.keithjackson.dev to PROD's OWN
gateway. Stood up the existing bootstrap/gitea/bootstrap-gitea.sh on the operator's
workstation with HOST_NAT_IP=192.168.2.132; all three prod nodes verified reachable
(Gitea 1.22.6 via the API), no firewall change needed — prod is on the same /24, no NAT.
clusters/prod now points at http://192.168.2.132:3000/... under a boxed TEMPORARY banner
with a 5-step cutover checklist ending 'delete this whole block — dev's stanza is what done
looks like'. Repo verified PUBLIC with anonymous read two independent ways (API
"private":false; git ls-remote unauthenticated with GIT_TERMINAL_PROMPT=0), so NO secretRef
is needed — documented so a future auth failure is not misread as a network fault.
bootstrap-gitea.sh's header corrected: it documented HOST_NAT_IP=10.73.0.1, the RETIRED
libvirt NAT, and forbade agent use; the operator has now directed this for prod.
=== 2. WIRELESS DISABLED ON PROD (operator-reported) ===
k3s-prod-3 was on ethernet AND wireless, and the wifi DHCP client was overwriting
/etc/resolv.conf. Root cause of the silence: node_baseline's DNS `contended` detection
checked resolvconf and NetworkManager but NOT dhcpcd, so the role wrote resolvers, reported
success, and dhcpcd reverted them on lease renewal — the exact 'green run that fixed
nothing' the contended branch exists to prevent.
Now IaC, not a one-off: survey wireless hardware, **REFUSE to disable it if the path to the
node runs over the wireless interface** (the guard that meant an agent dying mid-apply cost
nothing), blacklist the driver so it never reloads, stop dhcpcd only where it holds a
wireless interface and nothing else, take links down and unload modules with NO reboot, then
report the resulting state. dhcpcd is now in the contended set without weakening fail-closed.
Applied and verified: node 3's wlp1s0 is GONE from /sys/class/net, dhcpcd inactive and its
unit not-found, ONE default route per node, all three on 8.8.8.8/1.1.1.1 resolving public
names. Safe and idempotent on nodes 1 and 2, whose wifi hardware was already down.
=== 3. flux_bootstrap PATH PARAMETERISED ===
bootstrap/flux/bootstrap.sh already read FLUX_ENV_PATH but the role never exported it, so it
ALWAYS bootstrapped clusters/dev — prod was impossible. Two behaviour-bearing lines: role
default clusters/dev + `environment: FLUX_ENV_PATH`, overridden to clusters/prod in
group_vars/k3s_prod. Proven per host with a read-only Ansible probe: prod hosts resolve
clusters/prod, localhost/hypervisor resolves clusters/dev. Only two definitions repo-wide.
DEV PROVEN BYTE-IDENTICAL for the source change: all 12 substituting Flux paths rendered for
both trees, 27/27 SHA-256 artefacts identical, with TWO POSITIVE CONTROLS (an overlay-level
canary and a deep-path content canary) confirming the harness detects change — and both
canaries proven restored. Honest scope note recorded: kustomize strips comments, so the
harness proves the built stream, not source comments.
L0 substitution check: 106 assertions, 0 hazards. ci/l0.sh and ci/lint.sh exit 0.
=== STILL BLOCKING A FULL PROD RECONCILE (operator decisions) ===
- S3_BACKUP_ENDPOINT remains TBD-PROD-*: prod CANNOT reach dev's versitygw at
192.168.1.9:9000 (verified UNREACHABLE from all three — the pfSense boundary), and a prod
backup target is a deferred operator decision. Deliberately NOT invented: a DR target on a
workstation container would create false confidence. Velero/CNPG backups will fail loudly.
- Prod's public resolvers mean the API server's oidc-issuer-url resolves PUBLICLY, so kubectl
SSO will not work on prod until split-horizon is solved for it (§4).
- The temp Gitea is a single container on the operator's workstation — a single point of
failure for prod reconciliation, acceptable for bring-up only.
Serves: R-GITOPS-01, R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
P0 — clusters/base/platform.yaml gains `dependsOn: [flux-system]`. This is the actual fix
for the incident, not the feature gate. Verified against kustomize-controller v1.4.0
`kustomization_controller.go :: checkDependencies` (our version): dependsOn is REVISION-
gated, not merely Ready-gated — when the dependency shares a GitRepository, the dependent
cannot reconcile revision N until the dependency has APPLIED revision N. Applying revision
N is exactly what writes a newly-added cluster-vars knob.
Graph verified mechanically, not by eye: built clusters/dev and clusters/prod and walked
the resulting objects. `platform` was the ONLY Kustomization without a dependsOn, and all
11 substituting Kustomizations chain transitively through it (platform-config -> platform;
the waves, apps-vms, oauth2-proxy, headlamp, homepage, crafty, velero -> platform-config).
The "not chaining through platform" set is EMPTY, so this one edge protects the whole
graph. DFS over the full edge set after the edit: no cycles. The root (clusters/dev/
sync.yaml) runs wait: false, which is what makes the edge deadlock-free — it reaches Ready
on apply success without health-gating its children.
Accepted trade-off, documented in-file: the root becomes a gate. A malformed manifest
anywhere under clusters/<env> — including clusters/base — now stalls platform and
everything downstream, where today it would not. Fail-closed, per doctrine.
P1 — StrictPostBuildSubstitutions. MY PREMISE WAS WRONG about where this belongs and it
matters: this repo deliberately does NOT use `flux bootstrap` (see bootstrap/flux/README).
There is no gotk-components.yaml in git, and the live kustomize-controller Deployment
carries NO Flux ownership labels — it was installed imperatively by `flux install` from
bootstrap.sh and is not in Flux's inventory. `flux-system` self-manages the CR graph, NOT
the controllers. Nothing under clusters/** can reach a controller flag.
So it uses Flux's documented mechanism for hand-installed clusters: a new
bootstrap/flux/components/kustomization.yaml wrapping `gotk-components.yaml` with a
JSON-6902 patch appending --feature-gates=StrictPostBuildSubstitutions=true to
kustomize-controller. JSON-6902 `add /args/-` deliberately, NOT strategic merge — a
container's args is atomic and a merge patch would replace all five install-generated
args. bootstrap.sh step 1 now renders (`flux install --export` into a temp dir, apply the
patch, ASSERT the gate string is present so a future Flux bump that breaks the patch
target fails loudly) then applies. FLUX_INSTALL_PLAIN=1 is a loud escape hatch. Nothing is
vendored, so components can never drift from the pinned FLUX_VERSION.
Rejected: vendoring gotk-components.yaml under the root's path (the root's prune: true
would put Flux's own controllers and CRDs in a prune inventory), and shipping a partial
Deployment as a Flux-managed resource (a future prune would delete kustomize-controller).
ROLLOUT ORDER MATTERS. P0 arrives automatically on the next reconcile. P1 does NOT arrive
on a git push — the controller Deployment is outside Flux's inventory, so an operator must
re-run bootstrap.sh step 1, which restarts kustomize-controller (~10-20s; Flux is
level-triggered so an interrupted reconcile is simply retried). P0 must be reconciled
BEFORE P1 is rolled out: with the gate on but the ordering edge absent, the same race
turns a transient empty render into a transient hard Kustomization failure.
ALSO FIXES A PHASE-7 BLOCKER THIS CHANGE WOULD OTHERWISE HAVE CREATED. clusters/prod had
no root Kustomization — dev/sync.yaml even carried the note "a prod overlay needs its own
sync.yaml -> ./clusters/prod". With the new dependsOn, a prod bootstrap would have left
platform blocked forever on a flux-system Kustomization that does not exist, and nothing
downstream would ever reconcile. clusters/prod/sync.yaml is now authored, mirroring dev.
Two further prod-overlay gaps found and DOCUMENTED, not guessed (PROD-PROVISIONING §7):
prod has no SOPS `decryption` patch (every encrypted Secret would fail to decrypt — needs
the prod age-key decision, a security choice not a copy), and no GitRepository source URL
patch (needs a decision on whether prod tracks the same branch as dev, which would make a
dev-facing merge reconcile prod).
Substitution gate re-run: check OK, 102 assertions, 0 hazards. Gate: MERGE GATE: PASS.
Serves: R-ENV-02, R-GITOPS-01, R-PLAT-01, R-ENV-03.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Text-only cleanup after the upgrade campaign. Updates stale version
strings, prose, and comments to the verified current facts; no
Kubernetes behavior or version pins changed except the kubectl
toolchain bump (below).
Verified facts applied: k3s v1.36.4+k3s1, Debian 13 trixie,
Cilium 1.20.1 / Gateway API v1.6.1, Longhorn 1.12.1, node RAM 24 GiB,
dedicated virtio2 SSD etcd disk per node, Velero chart 12.x / v1.18,
Grafana 13.2.0.
- Version strings across infra/ and platform/ READMEs + comments
- kubectl toolchain 1.31.4 -> 1.36.4 (devcontainer/ci/conformance
versions.env; 1.31 vs a 1.36 cluster exceeded the +-1 skew)
- Retired-single-apps-Kustomization refs -> apps-wave-0/1/2 + apps-vms
with per-wave SOPS decryption (flux/clusters docs + bootstrap.sh)
- COMPLETED/historical banners on the three executed plan docs
- Resolved TODO(WS-SEC) markers -> Renovate-maintained ranges
- Appended dated DECISIONS-LOG entry (2026-08-30) superseding the
2026-08-26 LIVE STATE line
Left as revalidation items (not blind-edited): external-snapshotter
v8.2.0 vs 1.36, Cilium ext_authz/forward-auth relevance under 1.20.1,
conformance ImageVolume feature-gate assertions under 1.36.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Replaces the single `apps` Flux Kustomization with dependency-ordered waves
(apps-wave-0..2) so a crashlooping workload isolates to its wave + downstream
instead of stalling the entire app tier. Parallel within a wave, ordered across.
- catalog/render.py: compute waves by longest dependency-path depth over the
ServiceDefinition `dependencies:` edges (a `<svc>-db` is a leaf, so its
consumer lands strictly higher — the "DB below consumer" rule falls out of
longest-path, no special case). `--emit-waves` generates the per-wave
kustomizations + Flux CRs; `--wave-layout` places renders into the right dir.
wave 0: pihole gitea jackett plex qbittorrent nextcloud-db semaphore-db
wave 1: sonarr radarr nextcloud semaphore gitea-actions
wave 2: seerr
- apps/*.yaml -> apps/wave-{0,1,2}/ (byte-identical moves; SOPS files intact).
- clusters/base: apps.yaml removed; apps-wave-{0,1,2}.yaml added (dependsOn
chain: platform-config -> wave-0 -> wave-1 -> wave-2, prune+wait each).
- clusters/dev: one SOPS decryption patch per wave (load-bearing).
- DESIGN-waves.md: wave table, topology, cutover runbook.
Cutover: preceded by disabling prune on `apps` (1bfb0db) so removing that
object here does not GC its resources; the wave Kustomizations SSA-adopt the
byte-identical manifests in place (no restart, no PVC churn).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Install image-reflector-controller + image-automation-controller as part of the
flux-system install (bootstrap.sh --components-extra) so the image.toolkit CRDs
pre-exist before reconcile — no F-MON1 split needed for the ImageUpdateAutomation CR.
Refine the cluster-wide ImageUpdateAutomation for machine-authored git write-back
(setter strategy, path ./apps) reusing the flux-system GitRepository secretRef, and
document the operator git-write-credential hand-off + SOPS secret template
(gitea-image-automation-credentials). image-automation stays deferred in
platform/kustomization.yaml until that credential is provisioned.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
PIA-provider server selection is flaky here (random dead IPs); the operator's .ovpn
uses sweden.privacy.network (load-balanced, reliable). Switch to custom mode
(VPN_SERVICE_PROVIDER=custom, OPENVPN_CUSTOM_CONFIG=/config/pia.ovpn), mount the
pia-custom-ovpn Secret, keep creds + kill-switch + tun-mtu 1300. Drops PIA-provider
server list/updater and native PF (Phase-6 concern). Pending the operator-provisioned
pia-custom-ovpn Secret (not merged until it exists).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Per ADR-0013: platform/vpn-egress/ = a single shared Gluetun gateway (PIA over WireGuard,
built-in kill-switch FIREWALL=on, native port forwarding), namespace (non-app, NET_ADMIN +
/dev/net/tun), control-API Service, and a pre-wired (commented) pia-credentials SOPS line.
Defined the PIA secret location (platform/vpn-egress, keys username/password → OPENVPN_USER/
OPENVPN_PASSWORD) and updated the SOPS template accordingly.
NOT wired into platform/kustomization.yaml (Phase 3, author-first) — deploy needs PIA creds +
validation of the egress-steering mechanism (Cilium Egress Gateway → the Gluetun node's wg0;
documented in README with options + a fail-closed rationale, deliberately NOT committed as an
untested CiliumEgressGatewayPolicy). WireGuard chosen (UDP, protocol != TCP); PIA supports it on
all regions and the secret stays username/password.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The terse 'Zone->DNS->Edit + Zone->Read' sent the operator to the Account
permission scope. Spell out switching the scope dropdown to Zone + the exact
two Zone-scoped rows + Specific-zone resource.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Per operator: the firewalld rule (VM->host:3000 for the Flux<-Gitea source) must be
IaC, not a manual command (R-ENV-01, R-PLAT-04).
- infra/ansible/playbooks/host-prereqs.yml: idempotent firewalld open on the libvirt
zone (localhost play, become; no extra collection).
- bootstrap/gitea/bootstrap-gitea.sh: now fully automates admin + public repo + push
(was manual instructions) — idempotent; creds stay 0600 on host, never committed.
Applying host-prereqs still needs host sudo (I lack non-interactive sudo); with
passwordless become a destroy/rebuild is fully unattended.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Run a single Gitea container on the host (bootstrap/gitea/), outside the cluster,
while the cluster is built; Flux pulls this repo from it over the k3s-dev NAT
gateway (http://10.73.0.1:3000/Home-Network/home-network-k3s-impl.git, public repo
=> no creds, NAT-only). Wired the flux-system GitRepository url in clusters/dev.
Migrates to in-cluster Gitea in Phase 6. Pins confirmed. Author-only; gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
jakman85andClaude Opus 5