Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

feat(prod): temp Gitea source + wireless disabled + flux bootstrap path

Browse Source 
Autonomous prod bring-up, operator away. Three changes that together clear the last
blockers before `flux bootstrap`.

=== 1. TEMPORARY BOOTSTRAP GITEA — the last hard blocker, cleared ===
clusters/prod's source URL could not work: Gitea runs ON the dev cluster (the one slated for
decommission) and split-horizon DNS resolves gitea.home.keithjackson.dev to PROD's OWN
gateway. Stood up the existing bootstrap/gitea/bootstrap-gitea.sh on the operator's
workstation with HOST_NAT_IP=192.168.2.132; all three prod nodes verified reachable
(Gitea 1.22.6 via the API), no firewall change needed — prod is on the same /24, no NAT.

clusters/prod now points at http://192.168.2.132:3000/... under a boxed TEMPORARY banner
with a 5-step cutover checklist ending 'delete this whole block — dev's stanza is what done
looks like'. Repo verified PUBLIC with anonymous read two independent ways (API
"private":false; git ls-remote unauthenticated with GIT_TERMINAL_PROMPT=0), so NO secretRef
is needed — documented so a future auth failure is not misread as a network fault.

bootstrap-gitea.sh's header corrected: it documented HOST_NAT_IP=10.73.0.1, the RETIRED
libvirt NAT, and forbade agent use; the operator has now directed this for prod.

=== 2. WIRELESS DISABLED ON PROD (operator-reported) ===
k3s-prod-3 was on ethernet AND wireless, and the wifi DHCP client was overwriting
/etc/resolv.conf. Root cause of the silence: node_baseline's DNS `contended` detection
checked resolvconf and NetworkManager but NOT dhcpcd, so the role wrote resolvers, reported
success, and dhcpcd reverted them on lease renewal — the exact 'green run that fixed
nothing' the contended branch exists to prevent.

Now IaC, not a one-off: survey wireless hardware, **REFUSE to disable it if the path to the
node runs over the wireless interface** (the guard that meant an agent dying mid-apply cost
nothing), blacklist the driver so it never reloads, stop dhcpcd only where it holds a
wireless interface and nothing else, take links down and unload modules with NO reboot, then
report the resulting state. dhcpcd is now in the contended set without weakening fail-closed.

Applied and verified: node 3's wlp1s0 is GONE from /sys/class/net, dhcpcd inactive and its
unit not-found, ONE default route per node, all three on 8.8.8.8/1.1.1.1 resolving public
names. Safe and idempotent on nodes 1 and 2, whose wifi hardware was already down.

=== 3. flux_bootstrap PATH PARAMETERISED ===
bootstrap/flux/bootstrap.sh already read FLUX_ENV_PATH but the role never exported it, so it
ALWAYS bootstrapped clusters/dev — prod was impossible. Two behaviour-bearing lines: role
default clusters/dev + `environment: FLUX_ENV_PATH`, overridden to clusters/prod in
group_vars/k3s_prod. Proven per host with a read-only Ansible probe: prod hosts resolve
clusters/prod, localhost/hypervisor resolves clusters/dev. Only two definitions repo-wide.

DEV PROVEN BYTE-IDENTICAL for the source change: all 12 substituting Flux paths rendered for
both trees, 27/27 SHA-256 artefacts identical, with TWO POSITIVE CONTROLS (an overlay-level
canary and a deep-path content canary) confirming the harness detects change — and both
canaries proven restored. Honest scope note recorded: kustomize strips comments, so the
harness proves the built stream, not source comments.

L0 substitution check: 106 assertions, 0 hazards. ci/l0.sh and ci/lint.sh exit 0.

=== STILL BLOCKING A FULL PROD RECONCILE (operator decisions) ===
- S3_BACKUP_ENDPOINT remains TBD-PROD-*: prod CANNOT reach dev's versitygw at
  192.168.1.9:9000 (verified UNREACHABLE from all three — the pfSense boundary), and a prod
  backup target is a deferred operator decision. Deliberately NOT invented: a DR target on a
  workstation container would create false confidence. Velero/CNPG backups will fail loudly.
- Prod's public resolvers mean the API server's oidc-issuer-url resolves PUBLICLY, so kubectl
  SSO will not work on prod until split-horizon is solved for it (§4).
- The temp Gitea is a single container on the operator's workstation — a single point of
  failure for prod reconciliation, acceptable for bring-up only.

Serves: R-GITOPS-01, R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-04 18:53:17 -05:00
1 parent 05baae90ec
commit 9b8979a427
9 files changed
+659 -46
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

bootstrap/flux/bootstrap.sh
+11 -5
View File
@@ -1,6 +1,9 @@
#!/usr/bin/env bash
#
# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint for the dev cluster.
# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint. ENVIRONMENT-AGNOSTIC: which overlay
# it applies is FLUX_ENV_PATH (default clusters/dev). roles/flux_bootstrap exports that from
# `flux_bootstrap_env_path`, overridden to clusters/prod in group_vars/k3s_prod/vars.yml — so the
# same script bootstraps dev and prod with no fork (R-ENV-02).
#
# Phase 1, T-1.4 · ADR-0004 (Flux) · ADR-0007 (SOPS+age) · R-GITOPS-01 · R-SEC-01.
# =============================================================================
@@ -33,7 +36,7 @@
# 2. sops-age Secret — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so
# decryption works on first pass (../sops/age-key-secret.md). This script
# only VERIFIES it exists; it never handles key material.
# 3. `kubectl apply -k clusters/dev` — materialises the authored reconciliation graph:
# 3. `kubectl apply -k ${FLUX_ENV_PATH}` — materialises the authored reconciliation graph:
# GitRepository `flux-system` (source) -> Kustomization `platform` ->
# `platform-config` -> the app-wave Kustomizations `apps-wave-0/1/2`
# (chained dependsOn) + `apps-vms`, with cluster-vars overlay
@@ -52,7 +55,10 @@
set -euo pipefail
# --- config (env-overridable) ------------------------------------------------
FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}" # dev is the ONLY Phase-1 build target
# Which environment overlay to apply. Dev is the DEFAULT (it was the only Phase-1 build target);
# prod passes clusters/prod via roles/flux_bootstrap. Changing the default would silently retarget
# every existing caller — override it, never edit it.
FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}"
FLUX_NAMESPACE="${FLUX_NAMESPACE:-flux-system}"
SOPS_SECRET_NAME="${SOPS_SECRET_NAME:-sops-age}"
# The two Phase-4 image-automation controllers are installed alongside the default set so their CRDs
@@ -62,7 +68,7 @@ FLUX_COMPONENTS_EXTRA="${FLUX_COMPONENTS_EXTRA:-image-reflector-controller,image
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/../.." && pwd))"
echo "==> Flux + SOPS bootstrap (dev) — path=${FLUX_ENV_PATH}"
echo "==> Flux + SOPS bootstrap — path=${FLUX_ENV_PATH}"
# --- 0. tooling present (from infra/devcontainer; versions pinned there) ------
for bin in flux kubectl; do
@@ -134,7 +140,7 @@ if ! kubectl -n "${FLUX_NAMESPACE}" get secret "${SOPS_SECRET_NAME}" >/dev/null
exit 1
fi
# --- 3. apply the authored dev entrypoint ------------------------------------
# --- 3. apply the authored environment entrypoint ----------------------------
echo "==> [3/3] kubectl apply -k ${FLUX_ENV_PATH}"
kubectl apply -k "${repo_root}/${FLUX_ENV_PATH}"
bootstrap/gitea/bootstrap-gitea.sh
+51 -10
View File
@@ -1,11 +1,30 @@
#!/usr/bin/env bash
# bootstrap/gitea/bootstrap-gitea.sh — run a SINGLE Gitea container on the HOST (outside the
# cluster) as the Phase-1 GitOps bootstrap source. While the cluster is built, Flux (in-cluster)
# reconciles this repo from the host Gitea over the k3s-dev NAT gateway. This is the "local
# bootstrap source" that migrates to in-cluster Gitea in Phase 6 (docs/05 "Source-of-truth",
# CONTRIBUTING.md).
# cluster) as the GitOps bootstrap source. While a cluster is built, Flux (in-cluster) reconciles
# this repo from the host Gitea. This is the "local bootstrap source" that migrates to in-cluster
# Gitea at cutover (docs/05 "Source-of-truth", CONTRIBUTING.md). Dev has already cut over; PROD is
# using it now.
#
# Run by the OPERATOR at bring-up — NOT during author-only, NOT by any agent.
# HOW THE CLUSTER REACHES THE HOST — read this before setting HOST_NAT_IP.
# It is an ADDRESS OF THIS HOST as seen from the cluster nodes, and it differs per environment:
#
# PROD (bare metal, current use): HOST_NAT_IP=192.168.2.132
# The prod nodes (192.168.2.10/.11/.12) and this workstation are on the SAME LAN. There is
# NO NAT hop of any kind, and NO firewall change was needed — verified 2026-09-04 from all
# three nodes. The firewalld note further down is a DEV-ERA libvirt concern; it does not
# apply to prod, so do not "fix" prod by opening zones.
#
# DEV (libvirt VMs, HISTORICAL): HOST_NAT_IP=10.73.0.1 was the k3s-dev libvirt NAT gateway.
# *** THAT NETWORK IS RETIRED. 10.73.0.1 is stale and must not be reused. *** Dev's own
# bootstrap Gitea was later reached on 192.168.2.103:3000 and is retained STOPPED as
# recovery (see the comment on the GitRepository patch in clusters/dev/kustomization.yaml).
#
# WHO RUNS THIS: the OPERATOR, or an agent the operator has EXPLICITLY DIRECTED to stand up a
# bootstrap source for a specific bring-up (as happened 2026-09-04 for prod). It is NOT part of
# author-only work and must never be fired off casually or "to see if it works": it starts a
# long-lived container, generates and writes admin credentials to $HOME, creates an org and repo,
# and PUSHES THE CURRENT HEAD to that repo's `main` — which is the revision a cluster pointed at
# it will then reconcile. Know which HEAD you are publishing before you run it.
set -euo pipefail
ENGINE="${ENGINE:-podman}"
GITEA_IMAGE="${GITEA_IMAGE:-docker.io/gitea/gitea:1.22}" # pinned; bump if needed
@@ -16,11 +35,24 @@ SSH_PORT="${GITEA_SSH_PORT:-2222}"
# subuid that can't write a host-owned dir, so Gitea fails to create /data/gitea/conf/app.ini.
DATA_VOLUME="${GITEA_DATA_VOLUME:-gitea-bootstrap-data}"
DATA_DIR="podman volume ${DATA_VOLUME}" # display label only (data lives in the named volume)
# The cluster VMs reach the host on the k3s-dev NAT gateway (10.73.0.1). ROOT_URL must match so
# The address the CLUSTER uses to reach this host (see the header). ROOT_URL must match it so
# clone/redirect URLs are correct from inside the cluster.
# NOTE: firewalld's `libvirt` zone (owns virbrN) blocks VM->host:3000 by default. Open it once:
#
# The DEFAULT IS DELIBERATELY UNSET — there is no correct default any more. The old default
# (10.73.0.1) was the retired k3s-dev libvirt NAT gateway; leaving it in place would silently
# publish a ROOT_URL no cluster can resolve, and Flux would report what looks like a network fault.
# Pass it explicitly, e.g. for prod: HOST_NAT_IP=192.168.2.132 bash bootstrap-gitea.sh
#
# NOTE (DEV/libvirt ONLY, historical): firewalld's `libvirt` zone (owns virbrN) blocked
# VM->host:3000 by default and had to be opened once with
# sudo firewall-cmd --zone=libvirt --add-port=3000/tcp --permanent && sudo firewall-cmd --reload
HOST_NAT_IP="${HOST_NAT_IP:-10.73.0.1}"
# PROD needs NO firewall change: same-LAN, no libvirt zone involved (verified 2026-09-04).
if [ -z "${HOST_NAT_IP:-}" ]; then
echo "ERROR: HOST_NAT_IP is not set — it is this host's address AS SEEN FROM THE CLUSTER." >&2
echo " prod: HOST_NAT_IP=192.168.2.132 (same LAN as the nodes, no NAT, no firewall change)" >&2
echo " There is no safe default: the old one (10.73.0.1) is the RETIRED libvirt NAT." >&2
exit 1
fi
ORG="${GITEA_ORG:-Home-Network}"
REPO="${GITEA_REPO:-home-network-k3s-impl}"
@@ -75,8 +107,17 @@ cat <<EOF
Gitea bootstrap source ready (${DATA_DIR}).
UI (host): ${BASE}/${ORG}/${REPO}
Flux GitRepository (cluster): http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git (clusters/dev)
Flux GitRepository (cluster): http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git
(patch this url into the overlay being bootstrapped:
clusters/prod today; clusters/dev has already cut over)
Admin creds (host, 0600): ${ADMIN_CRED_FILE}
Prerequisite: firewalld must allow VM->host:${HTTP_PORT} —
Reachability: prod nodes reach ${HOST_NAT_IP}:${HTTP_PORT} directly on the LAN — no NAT, no
firewall change (verified 2026-09-04). For libvirt-based clusters ONLY, firewalld's libvirt zone
must allow VM->host:${HTTP_PORT} —
ansible-playbook infra/ansible/playbooks/host-prereqs.yml --ask-become-pass
REMINDER: the push above published the CURRENT HEAD as this repo's \`main\`. Whatever cluster points
at this source will reconcile THAT revision.
TEMPORARY BY DESIGN: this container is outside every cluster and is a single point of failure for
any cluster reconciling from it. Cut over to in-cluster Gitea (docs/05); for prod the cutover
procedure is written on the GitRepository patch in clusters/prod/kustomization.yaml.
EOF