feat(prod): temp Gitea source + wireless disabled + flux bootstrap path
Autonomous prod bring-up, operator away. Three changes that together clear the last blockers before `flux bootstrap`. === 1. TEMPORARY BOOTSTRAP GITEA — the last hard blocker, cleared === clusters/prod's source URL could not work: Gitea runs ON the dev cluster (the one slated for decommission) and split-horizon DNS resolves gitea.home.keithjackson.dev to PROD's OWN gateway. Stood up the existing bootstrap/gitea/bootstrap-gitea.sh on the operator's workstation with HOST_NAT_IP=192.168.2.132; all three prod nodes verified reachable (Gitea 1.22.6 via the API), no firewall change needed — prod is on the same /24, no NAT. clusters/prod now points at http://192.168.2.132:3000/... under a boxed TEMPORARY banner with a 5-step cutover checklist ending 'delete this whole block — dev's stanza is what done looks like'. Repo verified PUBLIC with anonymous read two independent ways (API "private":false; git ls-remote unauthenticated with GIT_TERMINAL_PROMPT=0), so NO secretRef is needed — documented so a future auth failure is not misread as a network fault. bootstrap-gitea.sh's header corrected: it documented HOST_NAT_IP=10.73.0.1, the RETIRED libvirt NAT, and forbade agent use; the operator has now directed this for prod. === 2. WIRELESS DISABLED ON PROD (operator-reported) === k3s-prod-3 was on ethernet AND wireless, and the wifi DHCP client was overwriting /etc/resolv.conf. Root cause of the silence: node_baseline's DNS `contended` detection checked resolvconf and NetworkManager but NOT dhcpcd, so the role wrote resolvers, reported success, and dhcpcd reverted them on lease renewal — the exact 'green run that fixed nothing' the contended branch exists to prevent. Now IaC, not a one-off: survey wireless hardware, **REFUSE to disable it if the path to the node runs over the wireless interface** (the guard that meant an agent dying mid-apply cost nothing), blacklist the driver so it never reloads, stop dhcpcd only where it holds a wireless interface and nothing else, take links down and unload modules with NO reboot, then report the resulting state. dhcpcd is now in the contended set without weakening fail-closed. Applied and verified: node 3's wlp1s0 is GONE from /sys/class/net, dhcpcd inactive and its unit not-found, ONE default route per node, all three on 8.8.8.8/1.1.1.1 resolving public names. Safe and idempotent on nodes 1 and 2, whose wifi hardware was already down. === 3. flux_bootstrap PATH PARAMETERISED === bootstrap/flux/bootstrap.sh already read FLUX_ENV_PATH but the role never exported it, so it ALWAYS bootstrapped clusters/dev — prod was impossible. Two behaviour-bearing lines: role default clusters/dev + `environment: FLUX_ENV_PATH`, overridden to clusters/prod in group_vars/k3s_prod. Proven per host with a read-only Ansible probe: prod hosts resolve clusters/prod, localhost/hypervisor resolves clusters/dev. Only two definitions repo-wide. DEV PROVEN BYTE-IDENTICAL for the source change: all 12 substituting Flux paths rendered for both trees, 27/27 SHA-256 artefacts identical, with TWO POSITIVE CONTROLS (an overlay-level canary and a deep-path content canary) confirming the harness detects change — and both canaries proven restored. Honest scope note recorded: kustomize strips comments, so the harness proves the built stream, not source comments. L0 substitution check: 106 assertions, 0 hazards. ci/l0.sh and ci/lint.sh exit 0. === STILL BLOCKING A FULL PROD RECONCILE (operator decisions) === - S3_BACKUP_ENDPOINT remains TBD-PROD-*: prod CANNOT reach dev's versitygw at 192.168.1.9:9000 (verified UNREACHABLE from all three — the pfSense boundary), and a prod backup target is a deferred operator decision. Deliberately NOT invented: a DR target on a workstation container would create false confidence. Velero/CNPG backups will fail loudly. - Prod's public resolvers mean the API server's oidc-issuer-url resolves PUBLICLY, so kubectl SSO will not work on prod until split-horizon is solved for it (§4). - The temp Gitea is a single container on the operator's workstation — a single point of failure for prod reconciliation, acceptable for bring-up only. Serves: R-GITOPS-01, R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
05baae90ec
commit
9b8979a427
9 files changed
+659
-46
No files matched your search
@@ -1193,12 +1193,15 @@ no extra vars is byte-for-byte the dev behaviour it always had (verified by a be
|
||||
as dev**. Eight `decryption: {provider: sops, secretRef: sops-age}` patches mirror dev
|
||||
exactly. The `sops-age` Secret is still created OUT OF BAND on the prod cluster
|
||||
(`bootstrap/sops/install-age-key.sh`) — the overlay references it by name only.
|
||||
- [~] **GitRepository source** — BRANCH settled, URL still blocked. Operator decision
|
||||
2026-09-01: **prod tracks `main`, the same branch as dev**, because dev is temporary and
|
||||
will be decommissioned once prod is stable. Recorded consequence: once prod is live,
|
||||
**every merge to `main` reconciles BOTH clusters** — no promotion step, no soak window.
|
||||
The URL is patched in but **cannot work on a cold prod cluster**, for two independent
|
||||
reasons found 2026-09-01:
|
||||
- [~] **GitRepository source** — **MITIGATED 2026-09-04, NOT SOLVED.** Branch settled; the
|
||||
in-cluster URL is still unusable, so prod now points at a **temporary bootstrap source**.
|
||||
Operator decision 2026-09-01: **prod tracks `main`, the same branch as dev**, because dev
|
||||
is temporary and will be decommissioned once prod is stable. Recorded consequence: once
|
||||
prod is live, **every merge to `main` reconciles BOTH clusters** — no promotion step, no
|
||||
soak window.
|
||||
|
||||
**Why the in-cluster URL (`https://gitea.home.keithjackson.dev/...`) cannot work on a
|
||||
cold prod cluster** — two independent reasons, found 2026-09-01, both still true:
|
||||
1. **Self-hosting circularity** — Gitea runs on the DEV cluster, the one slated for
|
||||
decommission. Gitea must move to prod (or off-cluster) BEFORE dev is retired, or prod
|
||||
loses its source of truth. **This is the decommission plan's first task, not its last.**
|
||||
@@ -1206,9 +1209,56 @@ no extra vars is byte-for-byte the dev behaviour it always had (verified by a be
|
||||
`home.keithjackson.dev` zone with `${GATEWAY_VIP}`, which on prod is `192.168.2.20`:
|
||||
prod's OWN gateway, where no Gitea listens. Prod's Flux would resolve
|
||||
`gitea.home.keithjackson.dev` to itself and never reach a repository.
|
||||
Bring-up needs a TEMPORARY source (bootstrap host-Gitea, or an off-cluster mirror) and a
|
||||
cutover once Gitea runs on prod — the same sequence `docs/05` describes for dev's own
|
||||
Phase-6 cutover.
|
||||
|
||||
**The mitigation now in `clusters/prod/kustomization.yaml`** (operator directed this to be
|
||||
stood up, 2026-09-04):
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| GitRepository url | `http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git` |
|
||||
| Branch | `main` (unchanged — same branch as dev) |
|
||||
| What it is | Gitea 1.22.6 in a **podman container on the operator's workstation**, started by `bootstrap/gitea/bootstrap-gitea.sh` with `HOST_NAT_IP=192.168.2.132` |
|
||||
| Reachability | All three prod nodes reach it **directly on the LAN** — no NAT hop (the libvirt network is retired), **no firewall change needed**. Verified from all three nodes. |
|
||||
| Auth | **None, deliberately.** The repo is PUBLIC on that Gitea, so Flux needs no `secretRef`. Verified anonymously: `curl .../api/v1/repos/Home-Network/home-network-k3s-impl` → `"private":false`, and `git ls-remote` returns `refs/heads/main`. |
|
||||
| Transport | **Plain HTTP, no TLS.** Accepted for the bring-up window only: LAN-local, and the repo holds no plaintext secrets (all SOPS-encrypted). |
|
||||
| Admin creds | `~/.config/home-k3s-bootstrap-gitea-admin.txt`, mode 0600, host-only, never committed. |
|
||||
|
||||
**THIS IS A SINGLE POINT OF FAILURE FOR PROD'S RECONCILIATION, and it is not even in a
|
||||
cluster.** It is one container on one workstation, with no HA, no backup of its own, and
|
||||
no restart guarantee beyond `--restart unless-stopped`. If that workstation is powered
|
||||
off, suspended, or the podman volume `gitea-bootstrap-data` is lost:
|
||||
- already-reconciled state **keeps running** — Flux does not tear anything down;
|
||||
- but **no new revision can land on prod**, and every `GitRepository` reconcile fails.
|
||||
So prod is *not* self-sufficient while this stands. Do not treat prod as production-ready
|
||||
on this footing, and do not let it become the steady state by inattention.
|
||||
|
||||
**One bring-up footgun:** `bootstrap-gitea.sh` pushes the operator's **current `HEAD`** to
|
||||
the bootstrap Gitea's `main`, whatever branch that HEAD is on. Before bootstrapping prod,
|
||||
confirm that remote `main` is the revision prod should run
|
||||
(`git ls-remote http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git` vs local
|
||||
`main`). As of 2026-09-04 the remote `main` is the tip of the working branch, **not** local
|
||||
`main`.
|
||||
|
||||
**EXIT CONDITION (the cutover) — mirrors dev's own Phase-6 cutover, `docs/05`.**
|
||||
This item is not closable until all of it is done:
|
||||
- [ ] Gitea runs **on prod**, reconciled in-cluster, answering
|
||||
`gitea.home.keithjackson.dev` through prod's Gateway VIP with a publicly-trusted cert.
|
||||
- [ ] This repo is pushed/mirrored into that in-cluster Gitea.
|
||||
- [ ] The `clusters/prod` GitRepository url is changed to
|
||||
`https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`
|
||||
and the TEMPORARY comment block is **deleted** (dev's stanza is what "done" looks like).
|
||||
- [ ] `flux get sources git -n flux-system` on prod reconciles a revision from the new URL.
|
||||
- [ ] **Only then** stop the workstation container — kept stopped-not-deleted as recovery,
|
||||
exactly as dev retained its own bootstrap Gitea.
|
||||
- [x] **`roles/flux_bootstrap` could only bootstrap dev** — FIXED 2026-09-04. It invoked
|
||||
`bootstrap/flux/bootstrap.sh` without ever exporting `FLUX_ENV_PATH`, so the script fell
|
||||
through to its `clusters/dev` default and §8 step 4 (`--path=clusters/prod`) was
|
||||
unreachable without editing the role. Now parameterised on the standard group_vars seam:
|
||||
role default `flux_bootstrap_env_path: clusters/dev` (defaults/main.yml) is exported as
|
||||
`FLUX_ENV_PATH` by the role's task, and
|
||||
`inventory/group_vars/k3s_prod/vars.yml` overrides it to `clusters/prod` — the same
|
||||
mechanism as `kube_vip_address` / `k3s_data_dir` / `cluster_kubeconfig_name`. Dev has no
|
||||
override anywhere, so dev resolves `clusters/dev` exactly as before.
|
||||
- [ ] Confirm prod's backup target: dev's S3 (`versitygw` on the Proxmox host, `192.168.1.9`) is on
|
||||
the dev side of the network. Off-cluster backup is DR-critical (R-STATE-03).
|
||||
|
||||
|
||||
Reference in new issue
Block a user