jakman85andClaude Opus 5
9b8979a427
Autonomous prod bring-up, operator away. Three changes that together clear the last blockers before `flux bootstrap`. === 1. TEMPORARY BOOTSTRAP GITEA — the last hard blocker, cleared === clusters/prod's source URL could not work: Gitea runs ON the dev cluster (the one slated for decommission) and split-horizon DNS resolves gitea.home.keithjackson.dev to PROD's OWN gateway. Stood up the existing bootstrap/gitea/bootstrap-gitea.sh on the operator's workstation with HOST_NAT_IP=192.168.2.132; all three prod nodes verified reachable (Gitea 1.22.6 via the API), no firewall change needed — prod is on the same /24, no NAT. clusters/prod now points at http://192.168.2.132:3000/... under a boxed TEMPORARY banner with a 5-step cutover checklist ending 'delete this whole block — dev's stanza is what done looks like'. Repo verified PUBLIC with anonymous read two independent ways (API "private":false; git ls-remote unauthenticated with GIT_TERMINAL_PROMPT=0), so NO secretRef is needed — documented so a future auth failure is not misread as a network fault. bootstrap-gitea.sh's header corrected: it documented HOST_NAT_IP=10.73.0.1, the RETIRED libvirt NAT, and forbade agent use; the operator has now directed this for prod. === 2. WIRELESS DISABLED ON PROD (operator-reported) === k3s-prod-3 was on ethernet AND wireless, and the wifi DHCP client was overwriting /etc/resolv.conf. Root cause of the silence: node_baseline's DNS `contended` detection checked resolvconf and NetworkManager but NOT dhcpcd, so the role wrote resolvers, reported success, and dhcpcd reverted them on lease renewal — the exact 'green run that fixed nothing' the contended branch exists to prevent. Now IaC, not a one-off: survey wireless hardware, **REFUSE to disable it if the path to the node runs over the wireless interface** (the guard that meant an agent dying mid-apply cost nothing), blacklist the driver so it never reloads, stop dhcpcd only where it holds a wireless interface and nothing else, take links down and unload modules with NO reboot, then report the resulting state. dhcpcd is now in the contended set without weakening fail-closed. Applied and verified: node 3's wlp1s0 is GONE from /sys/class/net, dhcpcd inactive and its unit not-found, ONE default route per node, all three on 8.8.8.8/1.1.1.1 resolving public names. Safe and idempotent on nodes 1 and 2, whose wifi hardware was already down. === 3. flux_bootstrap PATH PARAMETERISED === bootstrap/flux/bootstrap.sh already read FLUX_ENV_PATH but the role never exported it, so it ALWAYS bootstrapped clusters/dev — prod was impossible. Two behaviour-bearing lines: role default clusters/dev + `environment: FLUX_ENV_PATH`, overridden to clusters/prod in group_vars/k3s_prod. Proven per host with a read-only Ansible probe: prod hosts resolve clusters/prod, localhost/hypervisor resolves clusters/dev. Only two definitions repo-wide. DEV PROVEN BYTE-IDENTICAL for the source change: all 12 substituting Flux paths rendered for both trees, 27/27 SHA-256 artefacts identical, with TWO POSITIVE CONTROLS (an overlay-level canary and a deep-path content canary) confirming the harness detects change — and both canaries proven restored. Honest scope note recorded: kustomize strips comments, so the harness proves the built stream, not source comments. L0 substitution check: 106 assertions, 0 hazards. ci/l0.sh and ci/lint.sh exit 0. === STILL BLOCKING A FULL PROD RECONCILE (operator decisions) === - S3_BACKUP_ENDPOINT remains TBD-PROD-*: prod CANNOT reach dev's versitygw at 192.168.1.9:9000 (verified UNREACHABLE from all three — the pfSense boundary), and a prod backup target is a deferred operator decision. Deliberately NOT invented: a DR target on a workstation container would create false confidence. Velero/CNPG backups will fail loudly. - Prod's public resolvers mean the API server's oidc-issuer-url resolves PUBLICLY, so kubectl SSO will not work on prod until split-horizon is solved for it (§4). - The temp Gitea is a single container on the operator's workstation — a single point of failure for prod reconciliation, acceptable for bring-up only. Serves: R-GITOPS-01, R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
clusters/
Flux entrypoints and the environment-overlay model (R-ENV-02, R-GITOPS-01/02/03).
Governing spec: docs/05-gitops-and-delivery.md,
docs/02-environments-and-provisioning.md,
ADR-0004 (Flux).
Phase 0 scaffolding only. These are structurally-correct stubs. No cluster exists — nothing here reconciles until WS-INFRA delivers a cluster and bootstraps Flux (roadmap Phase 1).
Layout — base + overlay (never fork the base)
clusters/
base/ shared Flux objects (one authoring, both environments)
flux-system.yaml GitRepository — the reconciliation source (bootstrap → Gitea)
platform.yaml Flux Kustomization → ./platform (reconciles first)
apps-wave-0.yaml Flux Kustomization → ./apps/wave-0 (dependsOn platform-config)
apps-wave-1.yaml Flux Kustomization → ./apps/wave-1 (dependsOn apps-wave-0)
apps-wave-2.yaml Flux Kustomization → ./apps/wave-N (dependsOn apps-wave-N-1)
cluster-vars.yaml ConfigMap of overlay knobs (BASE defaults)
kustomization.yaml
dev/ ACTIVE build target — patches cluster-vars with dev values only
kustomization.yaml
prod/ DESIGN-ONLY until roadmap Phase 7 — patches cluster-vars with prod values only
kustomization.yaml
How the overlay model satisfies R-ENV-02
The base authors the reconciliation graph once. platform.yaml and the app-wave Kustomizations
(apps-wave-0/1/2.yaml — the single apps.yaml was retired; apps-vms.yaml deliberately omits it,
see below) declare postBuild.substituteFrom the cluster-vars ConfigMap. Platform and
app manifests use ${TOKEN} placeholders for every value that legitimately differs between
environments (issuer, replica counts, storage sizes, retention, node placement).
clusters/dev and clusters/prod differ ONLY by the values they patch into cluster-vars —
they add no resources of their own and never copy/fork a base manifest. Promotion to prod
(docs/02) is a change to the
prod overlay's variables, nothing else.
LAN addressing is a knob, not a literal (fixed 2026-08-31)
Until 2026-08-31 the base manifests hardcoded the dev LAN: the k3s API host Cilium dials, the
Cilium LB IP pool range, the pinned crafty/unifi L4 VIPs and the off-cluster S3 backup endpoint. That
broke R-ENV-02 outright — clusters/prod would have rendered dev addresses (192.168.1.x) onto the
prod LAN (192.168.2.x). They are now the cluster-vars "LAN addressing" block
(K3S_API_HOST, CILIUM_LB_POOL_START/_STOP, UNIFI_L4_VIP, CRAFTY_L4_VIP,
S3_BACKUP_ENDPOINT, alongside the pre-existing GATEWAY_VIP), set explicitly by BOTH overlays.
Two rules when touching that block:
- Substitution coverage. A
${TOKEN}only resolves in a path whose Flux Kustomization declarespostBuild.substituteFrom: cluster-vars; anywhere else the literal string${TOKEN}ships to the cluster.apps-vmswas the one path without it and now has it — which means every shell${...}in a VM manifest's cloud-init must be escaped$${...}(seeclusters/base/apps-vms.yamlfor the full envsubst escape contract). GATEWAY_VIPmust be insideCILIUM_LB_POOL_START..STOP, and in practice equal to START: thesharedGateway pins no address, so Cilium LB-IPAM gives it the pool's first free IP while split-horizon CoreDNS answers with${GATEWAY_VIP}. A VIP outside the pool blackholes in-cluster hairpin traffic while the Gateway still looks healthy.
Values the operator has not decided are TBD-PROD-* placeholders, not guesses: prod inherits the
DEV base defaults if an overlay op is simply omitted, so an invalid value that fails loudly is
strictly safer than a plausible one that quietly comes up on the wrong network.
How Flux is pointed at an environment
flux bootstrap (run by WS-INFRA in Phase 1) targets one environment's path per cluster:
# dev cluster
flux bootstrap ... --path=clusters/dev
# prod cluster (Phase 7 only — design-only until then)
flux bootstrap ... --path=clusters/prod
Bootstrap materializes the flux-system Kustomization that renders the chosen overlay, which pulls
in base/ and applies the env-scoped cluster-vars.
Reconciliation order
GitRepository (flux-system)
│
▼
Kustomization: platform (./platform, substituteFrom cluster-vars)
│ healthy
▼
Kustomization: apps-wave-0 (./apps/wave-0, dependsOn platform-config, substituteFrom cluster-vars)
│ healthy
▼
Kustomization: apps-wave-1 (./apps/wave-1, dependsOn apps-wave-0)
│ healthy
▼
Kustomization: apps-wave-N (./apps/wave-N, dependsOn apps-wave-N-1)
Wave 0 dependsOn platform-config (transitively platform) so CNI/storage/TLS/DNS/gateway
exist before any workload reconciles. The app tier is split into dependency-ordered WAVES
(catalog/render.py --emit-waves): each apps-wave-<N> health-gates (wait:true) ONLY its own
wave, so a crashloop in a later wave can no longer stall an earlier wave or block new revisions.
See DESIGN-waves.md for the wave table + cutover runbook.
TODOs (later phases — do not action in Phase 0)
- Source cutover (WS-GITOPS, Phase 6):
base/flux-system.yamlcurrently points at the bootstrap source. After Gitea is onboarded, repointspec.urltogitea.home.keithjackson.devand keep an off-cluster mirror (specdocs/05"Source-of-truth",CONTRIBUTING.md). - SOPS decryption (WS-SEC, Phase 1): add
spec.decryption.provider: sops(age secret name only, never plaintext) to the platform/apps Kustomizations once the age key is bootstrapped (ADR-0007). - Split ordered Flux Kustomizations if intra-platform ordering needs to be explicit (e.g. a
separate
cilium/longhornKustomization othersdependsOn).