feat(prod): temp Gitea source + wireless disabled + flux bootstrap path
Autonomous prod bring-up, operator away. Three changes that together clear the last blockers before `flux bootstrap`. === 1. TEMPORARY BOOTSTRAP GITEA — the last hard blocker, cleared === clusters/prod's source URL could not work: Gitea runs ON the dev cluster (the one slated for decommission) and split-horizon DNS resolves gitea.home.keithjackson.dev to PROD's OWN gateway. Stood up the existing bootstrap/gitea/bootstrap-gitea.sh on the operator's workstation with HOST_NAT_IP=192.168.2.132; all three prod nodes verified reachable (Gitea 1.22.6 via the API), no firewall change needed — prod is on the same /24, no NAT. clusters/prod now points at http://192.168.2.132:3000/... under a boxed TEMPORARY banner with a 5-step cutover checklist ending 'delete this whole block — dev's stanza is what done looks like'. Repo verified PUBLIC with anonymous read two independent ways (API "private":false; git ls-remote unauthenticated with GIT_TERMINAL_PROMPT=0), so NO secretRef is needed — documented so a future auth failure is not misread as a network fault. bootstrap-gitea.sh's header corrected: it documented HOST_NAT_IP=10.73.0.1, the RETIRED libvirt NAT, and forbade agent use; the operator has now directed this for prod. === 2. WIRELESS DISABLED ON PROD (operator-reported) === k3s-prod-3 was on ethernet AND wireless, and the wifi DHCP client was overwriting /etc/resolv.conf. Root cause of the silence: node_baseline's DNS `contended` detection checked resolvconf and NetworkManager but NOT dhcpcd, so the role wrote resolvers, reported success, and dhcpcd reverted them on lease renewal — the exact 'green run that fixed nothing' the contended branch exists to prevent. Now IaC, not a one-off: survey wireless hardware, **REFUSE to disable it if the path to the node runs over the wireless interface** (the guard that meant an agent dying mid-apply cost nothing), blacklist the driver so it never reloads, stop dhcpcd only where it holds a wireless interface and nothing else, take links down and unload modules with NO reboot, then report the resulting state. dhcpcd is now in the contended set without weakening fail-closed. Applied and verified: node 3's wlp1s0 is GONE from /sys/class/net, dhcpcd inactive and its unit not-found, ONE default route per node, all three on 8.8.8.8/1.1.1.1 resolving public names. Safe and idempotent on nodes 1 and 2, whose wifi hardware was already down. === 3. flux_bootstrap PATH PARAMETERISED === bootstrap/flux/bootstrap.sh already read FLUX_ENV_PATH but the role never exported it, so it ALWAYS bootstrapped clusters/dev — prod was impossible. Two behaviour-bearing lines: role default clusters/dev + `environment: FLUX_ENV_PATH`, overridden to clusters/prod in group_vars/k3s_prod. Proven per host with a read-only Ansible probe: prod hosts resolve clusters/prod, localhost/hypervisor resolves clusters/dev. Only two definitions repo-wide. DEV PROVEN BYTE-IDENTICAL for the source change: all 12 substituting Flux paths rendered for both trees, 27/27 SHA-256 artefacts identical, with TWO POSITIVE CONTROLS (an overlay-level canary and a deep-path content canary) confirming the harness detects change — and both canaries proven restored. Honest scope note recorded: kustomize strips comments, so the harness proves the built stream, not source comments. L0 substitution check: 106 assertions, 0 hazards. ci/l0.sh and ci/lint.sh exit 0. === STILL BLOCKING A FULL PROD RECONCILE (operator decisions) === - S3_BACKUP_ENDPOINT remains TBD-PROD-*: prod CANNOT reach dev's versitygw at 192.168.1.9:9000 (verified UNREACHABLE from all three — the pfSense boundary), and a prod backup target is a deferred operator decision. Deliberately NOT invented: a DR target on a workstation container would create false confidence. Velero/CNPG backups will fail loudly. - Prod's public resolvers mean the API server's oidc-issuer-url resolves PUBLICLY, so kubectl SSO will not work on prod until split-horizon is solved for it (§4). - The temp Gitea is a single container on the operator's workstation — a single point of failure for prod reconciliation, acceptable for bring-up only. Serves: R-GITOPS-01, R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
05baae90ec
commit
9b8979a427
9 files changed
+659
-46
No files matched your search
@@ -19,5 +19,22 @@ flux_bootstrap_repo_root: "{{ playbook_dir }}/../../.."
|
||||
flux_bootstrap_sops_cmd: "bash {{ flux_bootstrap_repo_root }}/bootstrap/sops/install-age-key.sh"
|
||||
|
||||
# Assumed interface of bootstrap/flux (GitOps Engineer, T-1.4):
|
||||
# runs `flux bootstrap ... --path=clusters/dev` against this cluster.
|
||||
# runs `flux install` + `kubectl apply -k <env path>` against this cluster.
|
||||
flux_bootstrap_flux_cmd: "bash {{ flux_bootstrap_repo_root }}/bootstrap/flux/bootstrap.sh"
|
||||
|
||||
# --- WHICH ENVIRONMENT OVERLAY THIS BOOTSTRAPS (R-ENV-02) -------------------------
|
||||
# The one thing that makes this role environment-agnostic. bootstrap/flux/bootstrap.sh
|
||||
# reads FLUX_ENV_PATH (default clusters/dev) and ends with `kubectl apply -k <that>`;
|
||||
# tasks/main.yml exports this value into that variable, so the role NEVER names an
|
||||
# environment and the task graph is identical for dev and prod.
|
||||
#
|
||||
# THE DEFAULT IS DEV ON PURPOSE — dev is LIVE, and group_vars/all.yml carries no
|
||||
# override, so a dev run resolves exactly the value it always had (clusters/dev) and
|
||||
# behaves byte-for-byte as before.
|
||||
#
|
||||
# PROD OVERRIDES IT in inventory/group_vars/k3s_prod/vars.yml — the same group_vars
|
||||
# seam as kube_vip_address / k3s_data_dir / cluster_kubeconfig_name /
|
||||
# node_baseline_backports_kernel. No role fork, no per-run extra-var, no second
|
||||
# playbook: group membership decides. Before this existed the role could bootstrap
|
||||
# ONLY clusters/dev, so prod bring-up was impossible without editing the role.
|
||||
flux_bootstrap_env_path: "clusters/dev"
|
||||
@@ -4,11 +4,17 @@
|
||||
# — it calls the entrypoints the Security + GitOps Engineers author there.
|
||||
#
|
||||
# Order matters: install the age decryption key FIRST (so Flux can decrypt on its
|
||||
# very first reconcile), THEN run `flux bootstrap` at clusters/dev.
|
||||
# very first reconcile), THEN bootstrap Flux at the environment's overlay.
|
||||
#
|
||||
# The two commands are variables (see defaults/main.yml). If the bootstrap workers
|
||||
# expose different entrypoints, override flux_bootstrap_sops_cmd / _flux_cmd — the
|
||||
# task graph does not change.
|
||||
#
|
||||
# ENVIRONMENT-AGNOSTIC (R-ENV-02): this role names NO environment. Which overlay gets
|
||||
# bootstrapped comes from `flux_bootstrap_env_path` (defaults/main.yml, default
|
||||
# clusters/dev), exported below as FLUX_ENV_PATH — the variable bootstrap/flux/
|
||||
# bootstrap.sh already reads. Prod overrides it in group_vars/k3s_prod/vars.yml, the
|
||||
# same seam as kube_vip_address / k3s_data_dir / cluster_kubeconfig_name.
|
||||
|
||||
- name: Assert the bootstrap/ subtree exists (authored by the other worker)
|
||||
ansible.builtin.stat:
|
||||
@@ -29,11 +35,14 @@
|
||||
run_once: true
|
||||
changed_when: true
|
||||
|
||||
- name: Bootstrap Flux against clusters/dev (bootstrap/flux)
|
||||
- name: Bootstrap Flux against this environment's overlay (bootstrap/flux)
|
||||
ansible.builtin.command:
|
||||
cmd: "{{ flux_bootstrap_flux_cmd }}"
|
||||
environment:
|
||||
KUBECONFIG: "{{ flux_bootstrap_kubeconfig }}"
|
||||
# The ONLY per-environment input. clusters/dev by default (dev unchanged);
|
||||
# clusters/prod for [k3s_prod] hosts via group_vars.
|
||||
FLUX_ENV_PATH: "{{ flux_bootstrap_env_path }}"
|
||||
delegate_to: localhost
|
||||
become: false
|
||||
run_once: true
|
||||
|
||||
Reference in new issue
Block a user