Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

feat(prod): temp Gitea source + wireless disabled + flux bootstrap path

Browse Source 
Autonomous prod bring-up, operator away. Three changes that together clear the last
blockers before `flux bootstrap`.

=== 1. TEMPORARY BOOTSTRAP GITEA — the last hard blocker, cleared ===
clusters/prod's source URL could not work: Gitea runs ON the dev cluster (the one slated for
decommission) and split-horizon DNS resolves gitea.home.keithjackson.dev to PROD's OWN
gateway. Stood up the existing bootstrap/gitea/bootstrap-gitea.sh on the operator's
workstation with HOST_NAT_IP=192.168.2.132; all three prod nodes verified reachable
(Gitea 1.22.6 via the API), no firewall change needed — prod is on the same /24, no NAT.

clusters/prod now points at http://192.168.2.132:3000/... under a boxed TEMPORARY banner
with a 5-step cutover checklist ending 'delete this whole block — dev's stanza is what done
looks like'. Repo verified PUBLIC with anonymous read two independent ways (API
"private":false; git ls-remote unauthenticated with GIT_TERMINAL_PROMPT=0), so NO secretRef
is needed — documented so a future auth failure is not misread as a network fault.

bootstrap-gitea.sh's header corrected: it documented HOST_NAT_IP=10.73.0.1, the RETIRED
libvirt NAT, and forbade agent use; the operator has now directed this for prod.

=== 2. WIRELESS DISABLED ON PROD (operator-reported) ===
k3s-prod-3 was on ethernet AND wireless, and the wifi DHCP client was overwriting
/etc/resolv.conf. Root cause of the silence: node_baseline's DNS `contended` detection
checked resolvconf and NetworkManager but NOT dhcpcd, so the role wrote resolvers, reported
success, and dhcpcd reverted them on lease renewal — the exact 'green run that fixed
nothing' the contended branch exists to prevent.

Now IaC, not a one-off: survey wireless hardware, **REFUSE to disable it if the path to the
node runs over the wireless interface** (the guard that meant an agent dying mid-apply cost
nothing), blacklist the driver so it never reloads, stop dhcpcd only where it holds a
wireless interface and nothing else, take links down and unload modules with NO reboot, then
report the resulting state. dhcpcd is now in the contended set without weakening fail-closed.

Applied and verified: node 3's wlp1s0 is GONE from /sys/class/net, dhcpcd inactive and its
unit not-found, ONE default route per node, all three on 8.8.8.8/1.1.1.1 resolving public
names. Safe and idempotent on nodes 1 and 2, whose wifi hardware was already down.

=== 3. flux_bootstrap PATH PARAMETERISED ===
bootstrap/flux/bootstrap.sh already read FLUX_ENV_PATH but the role never exported it, so it
ALWAYS bootstrapped clusters/dev — prod was impossible. Two behaviour-bearing lines: role
default clusters/dev + `environment: FLUX_ENV_PATH`, overridden to clusters/prod in
group_vars/k3s_prod. Proven per host with a read-only Ansible probe: prod hosts resolve
clusters/prod, localhost/hypervisor resolves clusters/dev. Only two definitions repo-wide.

DEV PROVEN BYTE-IDENTICAL for the source change: all 12 substituting Flux paths rendered for
both trees, 27/27 SHA-256 artefacts identical, with TWO POSITIVE CONTROLS (an overlay-level
canary and a deep-path content canary) confirming the harness detects change — and both
canaries proven restored. Honest scope note recorded: kustomize strips comments, so the
harness proves the built stream, not source comments.

L0 substitution check: 106 assertions, 0 hazards. ci/l0.sh and ci/lint.sh exit 0.

=== STILL BLOCKING A FULL PROD RECONCILE (operator decisions) ===
- S3_BACKUP_ENDPOINT remains TBD-PROD-*: prod CANNOT reach dev's versitygw at
  192.168.1.9:9000 (verified UNREACHABLE from all three — the pfSense boundary), and a prod
  backup target is a deferred operator decision. Deliberately NOT invented: a DR target on a
  workstation container would create false confidence. Velero/CNPG backups will fail loudly.
- Prod's public resolvers mean the API server's oidc-issuer-url resolves PUBLICLY, so kubectl
  SSO will not work on prod until split-horizon is solved for it (§4).
- The temp Gitea is a single container on the operator's workstation — a single point of
  failure for prod reconciliation, acceptable for bring-up only.

Serves: R-GITOPS-01, R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-04 18:53:17 -05:00
1 parent 05baae90ec
commit 9b8979a427
9 files changed
+659 -46
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

clusters/prod/kustomization.yaml
+52 -5
View File
@@ -115,8 +115,20 @@ patches:
name: sops-age
# --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
#
# *** BOOTSTRAP BLOCKER — this URL CANNOT work on a cold prod cluster. READ BEFORE BRING-UP. ***
# ############################################################################################
# ### ###
# ### *** T E M P O R A R Y B O O T S T R A P S O U R C E — NOT PERMANENT *** ###
# ### ###
# ### The url below is a Gitea container running ON THE OPERATOR'S WORKSTATION, outside ###
# ### any cluster. It exists ONLY to get prod's first reconcile off the ground. It MUST ###
# ### be cut over to the in-cluster URL once Gitea runs on prod. If you are reading this ###
# ### and prod has been serving traffic for months, THE CUTOVER WAS NEVER DONE — that is ###
# ### a live single point of failure, not a settled design. See "CUTOVER" below. ###
# ### ###
# ############################################################################################
#
# WHY THE IN-CLUSTER URL CANNOT BE USED YET (blocker found 2026-09-01, still true; the
# unusable value was `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`).
# Two independent reasons, both consequences of the split-horizon design:
# 1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's
# GitOps source lives inside the very cluster slated for decommission. Gitea must move to
@@ -126,13 +138,46 @@ patches:
# 192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve
# gitea.home.keithjackson.dev to itself and never reach a repository.
#
# Bring-up therefore needs a TEMPORARY source (the bootstrap host-Gitea, or an off-cluster
# mirror), with a cutover to the in-cluster URL once Gitea actually runs on prod — exactly the
# sequence docs/05 describes for dev's own Phase-6 cutover. Tracked in PROD-PROVISIONING.md §7.
# WHAT THE TEMPORARY SOURCE IS (stood up 2026-09-04 on explicit operator direction):
# bootstrap/gitea/bootstrap-gitea.sh with HOST_NAT_IP=192.168.2.132 — a podman container on the
# operator's workstation, Gitea 1.22.6, listening on 192.168.2.132:3000. All three prod nodes
# reach it DIRECTLY on the LAN: no libvirt NAT hop (that network is retired) and no firewall
# change was needed. This is the same "local bootstrap source" pattern dev itself used before
# its Phase-6 cutover (docs/05 "Source-of-truth").
#
# PLAIN HTTP, AND WHY THAT IS ACCEPTED HERE: the bootstrap Gitea has no TLS, so there is no cert
# for Flux to trust and no caCertSecretRef to wire. Traffic is LAN-local between prod nodes and
# the workstation, the repo carries NO plaintext secrets (everything sensitive is SOPS-encrypted,
# .sops.yaml), and the window is the bring-up only. Do NOT extend this posture past cutover.
#
# NO secretRef, DELIBERATELY: the repo is PUBLIC on the bootstrap Gitea, so anonymous clone
# works and Flux needs no git credentials. VERIFIED 2026-09-04 by unauthenticated request:
# curl http://192.168.2.132:3000/api/v1/repos/Home-Network/home-network-k3s-impl -> "private":false
# git ls-remote http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git -> refs/heads/main
# If the GitRepository ever reports an auth failure that looks like a network problem, RE-CHECK
# THAT FIRST: a private repo here fails exactly that way.
#
# *** CUTOVER — the exit condition, mirroring dev's Phase-6 cutover (docs/05) ***
# WHEN: Gitea runs on prod (reconciled in-cluster) and answers gitea.home.keithjackson.dev
# through prod's own Gateway VIP with a publicly-trusted cert.
# DO: 1. push/mirror this repo into the prod in-cluster Gitea;
# 2. change the url below to
# https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
# and DELETE this whole TEMPORARY block (dev's equivalent stanza, ~15 lines above in
# clusters/dev/kustomization.yaml, is what "done" looks like);
# 3. confirm `flux get sources git -n flux-system` reconciles the new revision;
# 4. only THEN stop the workstation container. Keep it stopped-not-deleted as recovery,
# exactly as dev retained its bootstrap Gitea.
# UNTIL THEN: prod's reconciliation depends on a container on a workstation that may be
# powered off. Flux keeps the last-reconciled state running, but NO new revision lands while
# it is down. Tracked as an open blocker in infra/PROD-PROVISIONING.md §7.
#
# The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev.
# CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH
# CLUSTERS. There is no promotion step and no soak window — dev and prod move together.
# NOTE for bring-up: bootstrap-gitea.sh pushes the operator's CURRENT HEAD to the bootstrap
# Gitea's `main`, whatever branch that is. Before bootstrapping prod, confirm that remote `main`
# is the revision you intend prod to run (`git ls-remote` above, against local `main`).
- patch: |-
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
@@ -140,7 +185,9 @@ patches:
name: flux-system
namespace: flux-system
spec:
url: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
# TEMPORARY bootstrap source — workstation container, NOT in any cluster. See the block
# above for the cutover to https://gitea.home.keithjackson.dev/... . Do not treat as final.
url: http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git
ref:
branch: main
- target: