feat(prod): temp Gitea source + wireless disabled + flux bootstrap path

Autonomous prod bring-up, operator away. Three changes that together clear the last
blockers before `flux bootstrap`.

=== 1. TEMPORARY BOOTSTRAP GITEA — the last hard blocker, cleared ===
clusters/prod's source URL could not work: Gitea runs ON the dev cluster (the one slated for
decommission) and split-horizon DNS resolves gitea.home.keithjackson.dev to PROD's OWN
gateway. Stood up the existing bootstrap/gitea/bootstrap-gitea.sh on the operator's
workstation with HOST_NAT_IP=192.168.2.132; all three prod nodes verified reachable
(Gitea 1.22.6 via the API), no firewall change needed — prod is on the same /24, no NAT.

clusters/prod now points at http://192.168.2.132:3000/... under a boxed TEMPORARY banner
with a 5-step cutover checklist ending 'delete this whole block — dev's stanza is what done
looks like'. Repo verified PUBLIC with anonymous read two independent ways (API
"private":false; git ls-remote unauthenticated with GIT_TERMINAL_PROMPT=0), so NO secretRef
is needed — documented so a future auth failure is not misread as a network fault.

bootstrap-gitea.sh's header corrected: it documented HOST_NAT_IP=10.73.0.1, the RETIRED
libvirt NAT, and forbade agent use; the operator has now directed this for prod.

=== 2. WIRELESS DISABLED ON PROD (operator-reported) ===
k3s-prod-3 was on ethernet AND wireless, and the wifi DHCP client was overwriting
/etc/resolv.conf. Root cause of the silence: node_baseline's DNS `contended` detection
checked resolvconf and NetworkManager but NOT dhcpcd, so the role wrote resolvers, reported
success, and dhcpcd reverted them on lease renewal — the exact 'green run that fixed
nothing' the contended branch exists to prevent.

Now IaC, not a one-off: survey wireless hardware, **REFUSE to disable it if the path to the
node runs over the wireless interface** (the guard that meant an agent dying mid-apply cost
nothing), blacklist the driver so it never reloads, stop dhcpcd only where it holds a
wireless interface and nothing else, take links down and unload modules with NO reboot, then
report the resulting state. dhcpcd is now in the contended set without weakening fail-closed.

Applied and verified: node 3's wlp1s0 is GONE from /sys/class/net, dhcpcd inactive and its
unit not-found, ONE default route per node, all three on 8.8.8.8/1.1.1.1 resolving public
names. Safe and idempotent on nodes 1 and 2, whose wifi hardware was already down.

=== 3. flux_bootstrap PATH PARAMETERISED ===
bootstrap/flux/bootstrap.sh already read FLUX_ENV_PATH but the role never exported it, so it
ALWAYS bootstrapped clusters/dev — prod was impossible. Two behaviour-bearing lines: role
default clusters/dev + `environment: FLUX_ENV_PATH`, overridden to clusters/prod in
group_vars/k3s_prod. Proven per host with a read-only Ansible probe: prod hosts resolve
clusters/prod, localhost/hypervisor resolves clusters/dev. Only two definitions repo-wide.

DEV PROVEN BYTE-IDENTICAL for the source change: all 12 substituting Flux paths rendered for
both trees, 27/27 SHA-256 artefacts identical, with TWO POSITIVE CONTROLS (an overlay-level
canary and a deep-path content canary) confirming the harness detects change — and both
canaries proven restored. Honest scope note recorded: kustomize strips comments, so the
harness proves the built stream, not source comments.

L0 substitution check: 106 assertions, 0 hazards. ci/l0.sh and ci/lint.sh exit 0.

=== STILL BLOCKING A FULL PROD RECONCILE (operator decisions) ===
- S3_BACKUP_ENDPOINT remains TBD-PROD-*: prod CANNOT reach dev's versitygw at
  192.168.1.9:9000 (verified UNREACHABLE from all three — the pfSense boundary), and a prod
  backup target is a deferred operator decision. Deliberately NOT invented: a DR target on a
  workstation container would create false confidence. Velero/CNPG backups will fail loudly.
- Prod's public resolvers mean the API server's oidc-issuer-url resolves PUBLICLY, so kubectl
  SSO will not work on prod until split-horizon is solved for it (§4).
- The temp Gitea is a single container on the operator's workstation — a single point of
  failure for prod reconciliation, acceptable for bring-up only.

Serves: R-GITOPS-01, R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-04 18:53:17 -05:00
1 parent 05baae90ec
commit 9b8979a427
9 files changed
+659 -46

No files matched your search

@@ -183,6 +183,29 @@ node_baseline_protected_mounts:
- /home
- /tmp
# --- WIRELESS (tasks/main.yml, tag `wireless`) ---------------------------------------
# A CLUSTER NODE GETS EXACTLY ONE PATH ONTO THE NETWORK. ON by default — unlike the
# backports kernel above, which is a prod-only deviation, this is a cluster-wide policy and
# it is a provable NO-OP where it does not apply: the tasks act only on interfaces the node
# reports as real 802.11 devices (a `phy80211` link in sysfs), and dev's Proxmox VMs have
# none, so on dev nothing is written, nothing is unloaded and nothing changes. Defaulting it
# true is what makes a FOURTH node with a wifi card inherit the policy from being a cluster
# node, rather than from someone remembering to set a flag.
#
# WHY IT EXISTS: k3s-prod-3 came up on both eno1 (192.168.2.12, the inventory address) and
# wlp1s0 (192.168.2.195, DHCP on the same /24). The wifi lease's dhcpcd run rewrote
# /etc/resolv.conf three minutes after this role wrote it, and a second address on the LAN
# where kube-vip floats 192.168.2.9 and Cilium L2-announces .20-.30 is a problem in its own
# right. See the `wireless` block in tasks/main.yml for the full reasoning, including why the
# DRIVER MODULE is the layer that gets blocked rather than the link, a service, or rfkill.
#
# SET IT FALSE for a host that must genuinely keep wifi (there is none today). That does NOT
# undo an already-applied block — delete the file below on the node as well.
node_baseline_disable_wireless: true
# Where the persistent block lives. 99- so it is read last and overrides any earlier
# permissive rule; the name says what it does, on a node with no repo checkout to hand.
node_baseline_wireless_modprobe_file: /etc/modprobe.d/99-k3s-no-wireless.conf
# --- DNS resilience (R-DNS-01 / R-DNS-03) -------------------------------------------
# These are the DEV values. Prod overrides them in inventory/group_vars/k3s_prod/vars.yml
# (the same seam kube_vip_address / k3s_api_host / cluster_kubeconfig_name use) — NOT in