jakman85andClaude Opus 5
9b8979a427
Autonomous prod bring-up, operator away. Three changes that together clear the last blockers before `flux bootstrap`. === 1. TEMPORARY BOOTSTRAP GITEA — the last hard blocker, cleared === clusters/prod's source URL could not work: Gitea runs ON the dev cluster (the one slated for decommission) and split-horizon DNS resolves gitea.home.keithjackson.dev to PROD's OWN gateway. Stood up the existing bootstrap/gitea/bootstrap-gitea.sh on the operator's workstation with HOST_NAT_IP=192.168.2.132; all three prod nodes verified reachable (Gitea 1.22.6 via the API), no firewall change needed — prod is on the same /24, no NAT. clusters/prod now points at http://192.168.2.132:3000/... under a boxed TEMPORARY banner with a 5-step cutover checklist ending 'delete this whole block — dev's stanza is what done looks like'. Repo verified PUBLIC with anonymous read two independent ways (API "private":false; git ls-remote unauthenticated with GIT_TERMINAL_PROMPT=0), so NO secretRef is needed — documented so a future auth failure is not misread as a network fault. bootstrap-gitea.sh's header corrected: it documented HOST_NAT_IP=10.73.0.1, the RETIRED libvirt NAT, and forbade agent use; the operator has now directed this for prod. === 2. WIRELESS DISABLED ON PROD (operator-reported) === k3s-prod-3 was on ethernet AND wireless, and the wifi DHCP client was overwriting /etc/resolv.conf. Root cause of the silence: node_baseline's DNS `contended` detection checked resolvconf and NetworkManager but NOT dhcpcd, so the role wrote resolvers, reported success, and dhcpcd reverted them on lease renewal — the exact 'green run that fixed nothing' the contended branch exists to prevent. Now IaC, not a one-off: survey wireless hardware, **REFUSE to disable it if the path to the node runs over the wireless interface** (the guard that meant an agent dying mid-apply cost nothing), blacklist the driver so it never reloads, stop dhcpcd only where it holds a wireless interface and nothing else, take links down and unload modules with NO reboot, then report the resulting state. dhcpcd is now in the contended set without weakening fail-closed. Applied and verified: node 3's wlp1s0 is GONE from /sys/class/net, dhcpcd inactive and its unit not-found, ONE default route per node, all three on 8.8.8.8/1.1.1.1 resolving public names. Safe and idempotent on nodes 1 and 2, whose wifi hardware was already down. === 3. flux_bootstrap PATH PARAMETERISED === bootstrap/flux/bootstrap.sh already read FLUX_ENV_PATH but the role never exported it, so it ALWAYS bootstrapped clusters/dev — prod was impossible. Two behaviour-bearing lines: role default clusters/dev + `environment: FLUX_ENV_PATH`, overridden to clusters/prod in group_vars/k3s_prod. Proven per host with a read-only Ansible probe: prod hosts resolve clusters/prod, localhost/hypervisor resolves clusters/dev. Only two definitions repo-wide. DEV PROVEN BYTE-IDENTICAL for the source change: all 12 substituting Flux paths rendered for both trees, 27/27 SHA-256 artefacts identical, with TWO POSITIVE CONTROLS (an overlay-level canary and a deep-path content canary) confirming the harness detects change — and both canaries proven restored. Honest scope note recorded: kustomize strips comments, so the harness proves the built stream, not source comments. L0 substitution check: 106 assertions, 0 hazards. ci/l0.sh and ci/lint.sh exit 0. === STILL BLOCKING A FULL PROD RECONCILE (operator decisions) === - S3_BACKUP_ENDPOINT remains TBD-PROD-*: prod CANNOT reach dev's versitygw at 192.168.1.9:9000 (verified UNREACHABLE from all three — the pfSense boundary), and a prod backup target is a deferred operator decision. Deliberately NOT invented: a DR target on a workstation container would create false confidence. Velero/CNPG backups will fail loudly. - Prod's public resolvers mean the API server's oidc-issuer-url resolves PUBLICLY, so kubectl SSO will not work on prod until split-horizon is solved for it (§4). - The temp Gitea is a single container on the operator's workstation — a single point of failure for prod reconciliation, acceptable for bring-up only. Serves: R-GITOPS-01, R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
bootstrap/flux — Flux bootstrap → clusters/dev (Phase 1, T-1.4)
Points Flux at the authored clusters/dev entrypoint and turns on SOPS decryption.
Spec: ADR-0004 (Flux),
docs/05 · Requirements: R-GITOPS-01
(pull-based reconcile), R-GITOPS-03 (drift correction), R-SEC-01 (SOPS decryption).
AUTHOR-ONLY. Nothing in this directory has been executed. No
flux,kubectl, key-gen, or cluster contact was run. These are files + runbooks the PM/operator validates and runs.
Files
| File | Purpose |
|---|---|
bootstrap.sh |
The single bootstrap entrypoint the Infra worker's final Ansible play calls. render+apply the Flux controllers (components/) → verify sops-age Secret → kubectl apply -k clusters/dev. |
components/kustomization.yaml |
Customization of the Flux CONTROLLERS themselves. Overlays flux install --export output; currently enables StrictPostBuildSubstitutions on kustomize-controller. See "Customizing the controllers" below. |
patches/platform-sops-decryption.yaml |
Strategic-merge patch enabling decryption.provider: sops + secretRef: sops-age on the platform Kustomization (realises the base TODO). |
patches/apps-sops-decryption.yaml |
Same, for the app-wave Kustomizations. There is no single apps Kustomization any more — the apps tier is now apps-wave-0/1/2 + apps-vms, and SOPS decryption is layered on per wave (each wave Kustomization is patched by name in clusters/dev). |
patches/flux-system-source-url.yaml |
Overrides the flux-system GitRepository url placeholder with the real bootstrap source (template — fill at bootstrap). |
Customizing the controllers (components/) — and how it reaches a RUNNING cluster
Because this repo does not use flux bootstrap, the Flux controller Deployments are not in git
and not reconciled by Flux. Verified on the live dev cluster (2026-09-01):
$ kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}'
{"app.kubernetes.io/component":"kustomize-controller","app.kubernetes.io/instance":"flux-system",
"app.kubernetes.io/part-of":"flux","app.kubernetes.io/version":"v2.4.0","control-plane":"controller"}
# no kustomize.toolkit.fluxcd.io/name label; deployment.kubernetes.io/revision: 1
No kustomize.toolkit.fluxcd.io/* label => no Flux Kustomization owns it. Nothing under
clusters/** can change a controller flag. So bootstrap.sh step 1 uses the mechanism Flux
documents for hand-installed clusters (Customize Flux
manifests):
flux install --export … -> <tmp>/gotk-components.yaml # regenerated from the pinned flux CLI
components/kustomization.yaml -> <tmp>/kustomization.yaml # the only hand-authored part (in git)
kubectl kustomize <tmp> | kubectl apply -f -
Nothing is vendored into the repo, so the component manifests can never drift from FLUX_VERSION
(infra/devcontainer/versions.env). Every future controller flag (vertical scaling, tolerations,
more feature gates) is another entry in components/kustomization.yaml.
Rolling this out to the ALREADY-RUNNING dev cluster (operator/PM step — NOT automatic)
A
git pushdoes not deliver this. Merging the change reconcilesclusters/**only; the controller Deployment is outside Flux's inventory. Someone must run step 1.
Re-running bootstrap.sh is safe and idempotent, but if you want only step 1:
# inside infra/devcontainer (pinned flux 2.4.0 / kubectl 1.36.4), from the repo root
d=$(mktemp -d)
flux install --export --namespace=flux-system \
--components-extra=image-reflector-controller,image-automation-controller > "$d/gotk-components.yaml"
cp bootstrap/flux/components/kustomization.yaml "$d/kustomization.yaml"
kubectl kustomize "$d" | kubectl apply -f - # add --server-side --force-conflicts on a field-manager conflict
kubectl -n flux-system rollout status deploy/kustomize-controller
Expect kustomize-controller to restart (new arg => new pod template => new ReplicaSet). Any reconcile in flight is interrupted and retried by the new pod; Flux is level-triggered, so nothing is lost. Confirm afterwards:
kubectl -n flux-system get deploy kustomize-controller \
-o jsonpath='{.spec.template.spec.containers[0].args}' # must contain --feature-gates=StrictPostBuildSubstitutions=true
flux get kustomizations -A # all 11 must return to Ready
Rollback is the same command with FLUX_INSTALL_PLAIN=1 bash bootstrap/flux/bootstrap.sh, or
simply flux install --namespace=flux-system --components-extra=… (drops the arg back off).
Why flux install + apply, not flux bootstrap git (Phase 1)
The reconciliation graph is already authored in clusters/base (read-only to this worker):
flux-system.yaml is the flux-system GitRepository; platform.yaml, platform-config.yaml
and the app-wave Kustomizations (apps-wave-0/1/2.yaml + apps-vms.yaml — the single apps.yaml
was retired) are the Flux Kustomizations; clusters/dev is the overlay that patches
cluster-vars (R-ENV-02) and layers SOPS decryption onto each Kustomization by name.
flux bootstrap git generates and owns its own gotk-sync GitRepository + Kustomization named
flux-system and commits them into the repo — which would duplicate/conflict with the authored
base. It also requires a writable remote git host, which does not exist in Phase 1 (spec docs/05
"Source-of-truth", CONTRIBUTING.md: bring-up is from a local/out-of-band bootstrap source; Gitea
arrives in Phase 6). So Phase 1 uses flux install (controllers only) + kubectl apply -k clusters/dev (materialise the authored graph). This reconciles with clusters/base instead of
regenerating it.
Bootstrap order (what runs, in sequence)
tofu apply → cloud-init → Ansible (k3s + Cilium) [Infra worker, T-1.1..1.3]
│
▼ (final Ansible play calls the entrypoints below)
1. bootstrap/sops/age-keygen.sh # operator generates age key OUTSIDE repo
2. bootstrap/sops/age-key-secret.md # operator installs private key as sops-age Secret
3. bootstrap/flux/bootstrap.sh # flux install → verify sops-age → apply -k clusters/dev
│
▼
Flux reconciles: GitRepository(flux-system) → Kustomization(platform) → Kustomization(platform-config)
→ Kustomization(apps-wave-0 → apps-wave-1 → apps-wave-2) + Kustomization(apps-vms),
with per-Kustomization SOPS decryption enabled (patches/) and cluster-vars overlay (R-ENV-02).
Interface note — for the Infra worker (T-1.3 Ansible)
Single entrypoint your final play calls: bootstrap/flux/bootstrap.sh (run inside the
infra/devcontainer toolchain so pinned flux/kubectl are on PATH; FLUX_VERSION/SOPS_VERSION
live in infra/devcontainer/versions.env).
Order your play must enforce (the age key is a hard dependency of a working decrypt):
bootstrap/sops/age-keygen.sh(operator step — generates key outside repo).- Install the
sops-ageSecret perbootstrap/sops/age-key-secret.md(thekubectl create secrettemplate).bootstrap.shfails fast if this Secret is absent. bootstrap/flux/bootstrap.sh(this entrypoint).
bootstrap.sh is idempotent-friendly (flux install and kubectl apply -k re-run safely) and
never handles key material — it only verifies the Secret exists.
Handoff — clusters owner / PM (files this worker may not edit)
clusters/ is out of this worker's write scope, so three wirings are delivered as patches here and
must be applied into the clusters/dev overlay (clusters/dev/kustomization.yaml patches:):
patches:
- path: ../../bootstrap/flux/patches/flux-system-source-url.yaml # real source URL
- path: ../../bootstrap/flux/patches/platform-sops-decryption.yaml # SOPS on platform
- path: ../../bootstrap/flux/patches/apps-sops-decryption.yaml # SOPS on apps
Alternatively the base TODOs in clusters/base/platform.yaml and each app-wave Kustomization
(apps-wave-0/1/2.yaml, apps-vms.yaml) and the url in
clusters/base/flux-system.yaml can be filled in directly (same effect) — the clusters owner's call.
Either way the sops-age secretRef name and flux-system source name are kept in lockstep with
the authored base.
Phase-6 cutover (WS-GITOPS — NOT now)
Once Gitea is stable, migrate the source to Gitea (docs/05, CONTRIBUTING.md). At that point
flux bootstrap git becomes appropriate (there is a writable remote to own):
# ILLUSTRATIVE — Phase 6, not run in Phase 1.
flux bootstrap git \
--url=https://gitea.home.keithjackson.dev/<org>/home-network-k3s-impl.git \
--branch=main \
--path=clusters/dev
Keep an off-cluster mirror so a source-of-truth outlives the cluster (docs/05 "Circularity & DR").