Un-suspend deadlocked: the single platform Kustomization applies HelmReleases
(which register CRDs) AND custom resources of those CRDs (ClusterIssuer,
Certificate, Kyverno ClusterPolicies) in ONE pass. Flux's server-side apply
fails atomically ('no matches for kind ClusterIssuer') → nothing installs, so
the CRD-registering HelmReleases never run either. Classic ordering deadlock.
Fix (flux2 controllers/configs pattern, applies to dev+prod equally — no
R-ENV-02 divergence): add a platform-config Flux Kustomization (dependsOn
platform, wait) carrying the CRs; platform keeps only the controllers.
- New clusters/base/platform-config.yaml (path ./platform-config, substituteFrom
cluster-vars, no decryption — no SOPS content); wired into base + apps now
dependsOn platform-config (transitively waits for platform).
- platform-config/ references the CR files IN PLACE via ../platform/... (Flux
builds LoadRestrictionsNone; Kyverno policies must stay under platform/kyverno/
for the CI L0 gate). CR resources dropped from the platform
component kustomizations (cert-manager, gateway, kyverno).
- Authored the missing Cilium L2 resources (were only a TODO): CiliumLoadBalancerIPPool
(dev NAT 10.73.0.240-.250) + CiliumL2AnnouncementPolicy (ens*), so Gateway
LoadBalancer listeners get a VIP. Prod range must be parameterized later.
Verified: platform build has zero CRs; platform-config build has 2 ClusterIssuers,
6 ClusterPolicies, Certificate, Gateway, both Cilium L2 CRs; dev/prod build; gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The un-suspend reconcile stalled: platform Kustomization Ready=False with
'StorageClass/longhorn ... .parameters.numberOfReplicas: expected string, got 3'.
StorageClass.parameters is map[string]string, but kustomize strips the quotes
around the bare "${LONGHORN_DEFAULT_REPLICA_COUNT}" placeholder, so Flux postBuild
substitution produced an unquoted numeric that server-side apply rejects as int.
Hardcode the quoted numeric literal "3" (kustomize preserves those, as "30" shows);
topology is locked at 3 nodes. The replica knob still drives the global default via
defaultSettings.defaultReplicaCount in the helmrelease (a Helm int value, unaffected).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
- Rename the committed encrypted secrets to canonical cloudflare-api-token.sops.yaml
(cert-manager + external-dns namespaces) and uncomment the resource lines in both
kustomizations, so cert-manager DNS-01 + ExternalDNS get the token at reconcile.
- Exempt **/*.sops.yaml from ci/yamllint.yaml: SOPS re-emits encrypted files with its
own indentation + long ENC[...] lines (never hand-edited), which was failing the lint
gate on main after the secrets were committed. Same treatment as **/vault.yml.
Both Secrets confirmed present + encrypted in the platform build; gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The terse 'Zone->DNS->Edit + Zone->Read' sent the operator to the Account
permission scope. Spell out switching the scope dropdown to Zone + the exact
two Zone-scoped rows + Specific-zone resource.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Two dev.sh gaps surfaced while running bootstrap/sops/install-age-key.sh:
- kubectl defaulted to localhost:8080 (connection refused) because KUBECONFIG
was never set in the toolbox. Set KUBECONFIG=/work/infra/ansible/.cache/dev.kubeconfig
(override via DEV_KUBECONFIG); --network host already reaches 10.73.0.11:6443.
- age-keygen.sh writes the PRIVATE key to ~/.config/sops/age/keys.txt, but that
path was not mounted in the --rm container, so the key was lost on exit. Mount
${HOME}/.config/sops -> /root/.config/sops (created first) so the key persists
on the host and SOPS_AGE_KEY_FILE resolves to it.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Prep only — nothing deployed; running cluster untouched.
Reconciliation graph (operator-approved Option A): author clusters/dev/rollout/
= 4 per-component Flux Kustomizations (kyverno->cilium->longhorn->monitoring,
monitoring dependsOn longhorn). Dev overlay $patch:delete's the monolithic base
platform/apps and adds ./rollout, so only the no-secret tier reconciles; the
secret-blocked components are simply not created until hand-offs arrive. The
no-secret tier carries no SOPS content, so it needs no sops-age key. prod is
untouched (still builds the monolith). Convergence path coded in (delete rollout/
+ the two deletes) — R-ENV-02 divergence flagged for architect review.
Component hardening via cluster-vars (prod keeps strict posture):
- KYVERNO_VALIDATION_ACTION Enforce->Audit(dev), _WEBHOOK_FAILURE_POLICY
Fail->Ignore(dev), _ADMISSION_REPLICAS 3->1(dev); webhook namespaceSelector
excludes system+platform namespaces.
- MONITORING_STORAGE_CLASS var; GRAFANA_ADMIN_SECRET var (dev ''=chart default);
Alertmanager SMTP/secret deferred to Phase-4 with verbatim re-enable markers.
Cilium parity verified against the live CNI (no-op adopt). Validated:
kubectl kustomize dev/prod/all-components, envsubst->YAML parse, make -C ci gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Implements T-1.5 findings F1/F2 (author-first, no cluster change).
F1 (tofu): refactor libvirt node resources count -> for_each keyed by
node name so single-node -replace/destroy cannot cascade to siblings.
Add ordered-list/primary helper locals; update outputs.tf off the map;
add moved.tf (12 moved blocks). tofu plan proves a pure state-address
migration: 0 to add, 0 to change, 0 to destroy (all 12 'has moved to').
State-persist via a 0-change apply is operator-permission-gated.
F2 (ansible): k3s_server secondary fetches the join token via
delegate_to the primary (published fact as fast path), so a single-node
rejoin with --limit k3s-dev-N is self-contained and no longer depends on
a fact an earlier play had to set. Syntax-check + CI gate green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
T-1.5 destructive IaC-validity test PASSED: dev cluster torn down and
reconstituted from IaC + preserved disks + Flux git source with zero
state loss (3/3 Ready, Cilium OK, Flux READY=True). Post-rebuild
conformance L2 R-NET-01 + R-NET-02 both PASS.
Two IaC-hardening findings for Phase 2:
- F1: tofu destroy -target on a count-indexed resource over-destroyed
all 3 domains; refactor libvirt_domain.node count -> for_each keyed by
node name; never use destroy -target for single-node lifecycle.
- F2: single-node ansible rejoin needs the primary in --limit to publish
the k3s join-token fact; make the join self-contained via delegate_to
the primary so --limit k3s-dev-N is hands-off.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Flux postBuild.substituteFrom is case-sensitive; platform/ components reference
${UPPERCASE} tokens but cluster-vars keys were lowercase -> none would resolve
(literal ${...} would reach the HelmReleases). Uppercased the ConfigMap data
keys + the dev/prod overlay patch paths. Every component ref now resolves;
clusters/dev & clusters/prod build; gate green. Clears the K1 landmine before
any platform un-suspend.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
host-prereqs.yml opened the firewall (via the operator's ansible-vault become
password); Flux GitRepository READY=True (main@sha1:a2f3bdc). Commit the
encrypted hypervisor vault (ansible_become_password) so rebuild is fully
hands-off; passphrase stays out-of-repo. platform/apps still not reconciled.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Where the sudo password goes, codified:
- inventory: new [hypervisor] group (localhost, connection=local).
- host-prereqs.yml targets 'hypervisor'; become password from vaulted
group_vars/hypervisor/vault.yml (ansible_become_password) — user creates it
encrypted (safe to commit); README documents the exact commands.
- ansible.cfg vault_password_file = ~/.config/home-k3s/ansible-vault-pass
(passphrase kept OUTSIDE the repo, never committed).
- ci/yamllint.yaml ignores **/vault.yml (vault blob isn't parseable YAML).
Nodes keep NOPASSWD sudo (group_vars/all) — vault scoped to the host only.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Per operator: the firewalld rule (VM->host:3000 for the Flux<-Gitea source) must be
IaC, not a manual command (R-ENV-01, R-PLAT-04).
- infra/ansible/playbooks/host-prereqs.yml: idempotent firewalld open on the libvirt
zone (localhost play, become; no extra collection).
- bootstrap/gitea/bootstrap-gitea.sh: now fully automates admin + public repo + push
(was manual instructions) — idempotent; creds stay 0600 on host, never committed.
Applying host-prereqs still needs host sudo (I lack non-interactive sudo); with
passwordless become a destroy/rebuild is fully unattended.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Cluster is UP (3/3 Ready, Cilium OK); Phase 1 paused before Flux bootstrap
(Option A: suspend platform/apps until Phase 2). Add infra/PHASE1-STATUS.md
(what's done, state file locations, reconnect steps, remaining T-1.4/1.5,
Phase-2 platform prereq), infra/cluster.sh (status/start/stop/kubeconfig), and
document that tofu+ansible run on the host (libvirt provider limitation).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Run a single Gitea container on the host (bootstrap/gitea/), outside the cluster,
while the cluster is built; Flux pulls this repo from it over the k3s-dev NAT
gateway (http://10.73.0.1:3000/Home-Network/home-network-k3s-impl.git, public repo
=> no creds, NAT-only). Wired the flux-system GitRepository url in clusters/dev.
Migrates to in-cluster Gitea in Phase 6. Pins confirmed. Author-only; gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Per operator rule: OpenTofu/age/sops/flux are Go binaries, so bake the toolchain
into a pinned Fedora-44 dev container (matches host libvirt 12) instead of a venv.
infra/devcontainer/: Containerfile + versions.env (tofu 1.9.0, age 1.2.1, sops
3.9.4, flux 2.4.0, kubectl 1.32.0, helm 3.16.0) + dev.sh (runs with host libvirt
socket + repo + ssh mounted) + README. Built & validated: all tools resolve;
virsh reaches host libvirt from inside the container.
Also record resolved Phase 1 decisions in PHASE1-SCOPE.md (6GiB/4vCPU x3;
k3s-dev NAT 10.73.0.0/24; container toolchain). Gate green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
jakman85
