Files
jakman85andClaude Opus 5 9b8979a427 feat(prod): temp Gitea source + wireless disabled + flux bootstrap path
Autonomous prod bring-up, operator away. Three changes that together clear the last
blockers before `flux bootstrap`.

=== 1. TEMPORARY BOOTSTRAP GITEA — the last hard blocker, cleared ===
clusters/prod's source URL could not work: Gitea runs ON the dev cluster (the one slated for
decommission) and split-horizon DNS resolves gitea.home.keithjackson.dev to PROD's OWN
gateway. Stood up the existing bootstrap/gitea/bootstrap-gitea.sh on the operator's
workstation with HOST_NAT_IP=192.168.2.132; all three prod nodes verified reachable
(Gitea 1.22.6 via the API), no firewall change needed — prod is on the same /24, no NAT.

clusters/prod now points at http://192.168.2.132:3000/... under a boxed TEMPORARY banner
with a 5-step cutover checklist ending 'delete this whole block — dev's stanza is what done
looks like'. Repo verified PUBLIC with anonymous read two independent ways (API
"private":false; git ls-remote unauthenticated with GIT_TERMINAL_PROMPT=0), so NO secretRef
is needed — documented so a future auth failure is not misread as a network fault.

bootstrap-gitea.sh's header corrected: it documented HOST_NAT_IP=10.73.0.1, the RETIRED
libvirt NAT, and forbade agent use; the operator has now directed this for prod.

=== 2. WIRELESS DISABLED ON PROD (operator-reported) ===
k3s-prod-3 was on ethernet AND wireless, and the wifi DHCP client was overwriting
/etc/resolv.conf. Root cause of the silence: node_baseline's DNS `contended` detection
checked resolvconf and NetworkManager but NOT dhcpcd, so the role wrote resolvers, reported
success, and dhcpcd reverted them on lease renewal — the exact 'green run that fixed
nothing' the contended branch exists to prevent.

Now IaC, not a one-off: survey wireless hardware, **REFUSE to disable it if the path to the
node runs over the wireless interface** (the guard that meant an agent dying mid-apply cost
nothing), blacklist the driver so it never reloads, stop dhcpcd only where it holds a
wireless interface and nothing else, take links down and unload modules with NO reboot, then
report the resulting state. dhcpcd is now in the contended set without weakening fail-closed.

Applied and verified: node 3's wlp1s0 is GONE from /sys/class/net, dhcpcd inactive and its
unit not-found, ONE default route per node, all three on 8.8.8.8/1.1.1.1 resolving public
names. Safe and idempotent on nodes 1 and 2, whose wifi hardware was already down.

=== 3. flux_bootstrap PATH PARAMETERISED ===
bootstrap/flux/bootstrap.sh already read FLUX_ENV_PATH but the role never exported it, so it
ALWAYS bootstrapped clusters/dev — prod was impossible. Two behaviour-bearing lines: role
default clusters/dev + `environment: FLUX_ENV_PATH`, overridden to clusters/prod in
group_vars/k3s_prod. Proven per host with a read-only Ansible probe: prod hosts resolve
clusters/prod, localhost/hypervisor resolves clusters/dev. Only two definitions repo-wide.

DEV PROVEN BYTE-IDENTICAL for the source change: all 12 substituting Flux paths rendered for
both trees, 27/27 SHA-256 artefacts identical, with TWO POSITIVE CONTROLS (an overlay-level
canary and a deep-path content canary) confirming the harness detects change — and both
canaries proven restored. Honest scope note recorded: kustomize strips comments, so the
harness proves the built stream, not source comments.

L0 substitution check: 106 assertions, 0 hazards. ci/l0.sh and ci/lint.sh exit 0.

=== STILL BLOCKING A FULL PROD RECONCILE (operator decisions) ===
- S3_BACKUP_ENDPOINT remains TBD-PROD-*: prod CANNOT reach dev's versitygw at
  192.168.1.9:9000 (verified UNREACHABLE from all three — the pfSense boundary), and a prod
  backup target is a deferred operator decision. Deliberately NOT invented: a DR target on a
  workstation container would create false confidence. Velero/CNPG backups will fail loudly.
- Prod's public resolvers mean the API server's oidc-issuer-url resolves PUBLICLY, so kubectl
  SSO will not work on prod until split-horizon is solved for it (§4).
- The temp Gitea is a single container on the operator's workstation — a single point of
  failure for prod reconciliation, acceptable for bring-up only.

Serves: R-GITOPS-01, R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-04 18:53:17 -05:00

124 lines
7.3 KiB
Bash
Executable File

#!/usr/bin/env bash
# bootstrap/gitea/bootstrap-gitea.sh — run a SINGLE Gitea container on the HOST (outside the
# cluster) as the GitOps bootstrap source. While a cluster is built, Flux (in-cluster) reconciles
# this repo from the host Gitea. This is the "local bootstrap source" that migrates to in-cluster
# Gitea at cutover (docs/05 "Source-of-truth", CONTRIBUTING.md). Dev has already cut over; PROD is
# using it now.
#
# HOW THE CLUSTER REACHES THE HOST — read this before setting HOST_NAT_IP.
# It is an ADDRESS OF THIS HOST as seen from the cluster nodes, and it differs per environment:
#
# PROD (bare metal, current use): HOST_NAT_IP=192.168.2.132
# The prod nodes (192.168.2.10/.11/.12) and this workstation are on the SAME LAN. There is
# NO NAT hop of any kind, and NO firewall change was needed — verified 2026-09-04 from all
# three nodes. The firewalld note further down is a DEV-ERA libvirt concern; it does not
# apply to prod, so do not "fix" prod by opening zones.
#
# DEV (libvirt VMs, HISTORICAL): HOST_NAT_IP=10.73.0.1 was the k3s-dev libvirt NAT gateway.
# *** THAT NETWORK IS RETIRED. 10.73.0.1 is stale and must not be reused. *** Dev's own
# bootstrap Gitea was later reached on 192.168.2.103:3000 and is retained STOPPED as
# recovery (see the comment on the GitRepository patch in clusters/dev/kustomization.yaml).
#
# WHO RUNS THIS: the OPERATOR, or an agent the operator has EXPLICITLY DIRECTED to stand up a
# bootstrap source for a specific bring-up (as happened 2026-09-04 for prod). It is NOT part of
# author-only work and must never be fired off casually or "to see if it works": it starts a
# long-lived container, generates and writes admin credentials to $HOME, creates an org and repo,
# and PUSHES THE CURRENT HEAD to that repo's `main` — which is the revision a cluster pointed at
# it will then reconcile. Know which HEAD you are publishing before you run it.
set -euo pipefail
ENGINE="${ENGINE:-podman}"
GITEA_IMAGE="${GITEA_IMAGE:-docker.io/gitea/gitea:1.22}" # pinned; bump if needed
NAME="${GITEA_NAME:-home-k3s-bootstrap-gitea}"
HTTP_PORT="${GITEA_HTTP_PORT:-3000}"
SSH_PORT="${GITEA_SSH_PORT:-2222}"
# Use a NAMED podman volume, not a host bind-mount: rootless podman maps the container's uid to a
# subuid that can't write a host-owned dir, so Gitea fails to create /data/gitea/conf/app.ini.
DATA_VOLUME="${GITEA_DATA_VOLUME:-gitea-bootstrap-data}"
DATA_DIR="podman volume ${DATA_VOLUME}" # display label only (data lives in the named volume)
# The address the CLUSTER uses to reach this host (see the header). ROOT_URL must match it so
# clone/redirect URLs are correct from inside the cluster.
#
# The DEFAULT IS DELIBERATELY UNSET — there is no correct default any more. The old default
# (10.73.0.1) was the retired k3s-dev libvirt NAT gateway; leaving it in place would silently
# publish a ROOT_URL no cluster can resolve, and Flux would report what looks like a network fault.
# Pass it explicitly, e.g. for prod: HOST_NAT_IP=192.168.2.132 bash bootstrap-gitea.sh
#
# NOTE (DEV/libvirt ONLY, historical): firewalld's `libvirt` zone (owns virbrN) blocked
# VM->host:3000 by default and had to be opened once with
# sudo firewall-cmd --zone=libvirt --add-port=3000/tcp --permanent && sudo firewall-cmd --reload
# PROD needs NO firewall change: same-LAN, no libvirt zone involved (verified 2026-09-04).
if [ -z "${HOST_NAT_IP:-}" ]; then
echo "ERROR: HOST_NAT_IP is not set — it is this host's address AS SEEN FROM THE CLUSTER." >&2
echo " prod: HOST_NAT_IP=192.168.2.132 (same LAN as the nodes, no NAT, no firewall change)" >&2
echo " There is no safe default: the old one (10.73.0.1) is the RETIRED libvirt NAT." >&2
exit 1
fi
ORG="${GITEA_ORG:-Home-Network}"
REPO="${GITEA_REPO:-home-network-k3s-impl}"
"$ENGINE" volume create "${DATA_VOLUME}" >/dev/null 2>&1 || true
if "$ENGINE" container exists "$NAME" 2>/dev/null; then
echo "Gitea container '${NAME}' already exists — starting it."
"$ENGINE" start "$NAME" >/dev/null
else
"$ENGINE" run -d --name "$NAME" --restart unless-stopped \
-p "${HTTP_PORT}:3000" -p "${SSH_PORT}:22" \
-v "${DATA_VOLUME}:/data" \
-e GITEA__server__ROOT_URL="http://${HOST_NAT_IP}:${HTTP_PORT}/" \
-e GITEA__server__SSH_PORT="${SSH_PORT}" \
-e GITEA__security__INSTALL_LOCK=true \
"$GITEA_IMAGE"
fi
# --- automated, idempotent setup (admin + PUBLIC repo + push) — NO manual steps (R-ENV-01) ---
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
ADMIN_CRED_FILE="${GITEA_ADMIN_CRED_FILE:-${HOME}/.config/home-k3s-bootstrap-gitea-admin.txt}"
BASE="http://localhost:${HTTP_PORT}" # host-side mgmt via localhost; the cluster uses HOST_NAT_IP
echo "Waiting for Gitea to be healthy..."
curl -fsS --retry 40 --retry-delay 2 --retry-all-errors "${BASE}/api/healthz" >/dev/null
# Admin credentials: generated once, stored 0600 on the host (never committed, never printed).
if [ ! -s "$ADMIN_CRED_FILE" ]; then
mkdir -p "$(dirname "$ADMIN_CRED_FILE")"
printf 'user=gitops\npass=%s\n' "$(openssl rand -base64 18 | tr -dc 'A-Za-z0-9' | head -c 22)" > "$ADMIN_CRED_FILE"
chmod 600 "$ADMIN_CRED_FILE"
fi
ADMIN_USER="$(sed -n 's/^user=//p' "$ADMIN_CRED_FILE")"
ADMIN_PW="$(sed -n 's/^pass=//p' "$ADMIN_CRED_FILE")"
# Create the admin user if missing (idempotent).
if ! "$ENGINE" exec -u git "$NAME" gitea admin user list 2>/dev/null | grep -qw "$ADMIN_USER"; then
"$ENGINE" exec -u git "$NAME" gitea admin user create \
--username "$ADMIN_USER" --password "$ADMIN_PW" --email "${ADMIN_USER}@home.local" \
--admin --must-change-password=false
fi
# Org + PUBLIC repo (idempotent — 'already exists' is fine; public => Flux needs no git creds).
curl -fsS -u "${ADMIN_USER}:${ADMIN_PW}" -H 'Content-Type: application/json' -X POST \
"${BASE}/api/v1/orgs" -d "{\"username\":\"${ORG}\"}" -o /dev/null 2>/dev/null || true
curl -fsS -u "${ADMIN_USER}:${ADMIN_PW}" -H 'Content-Type: application/json' -X POST \
"${BASE}/api/v1/orgs/${ORG}/repos" -d "{\"name\":\"${REPO}\",\"private\":false}" -o /dev/null 2>/dev/null || true
# Push current HEAD -> the bootstrap source's main.
git -C "$REPO_ROOT" push "http://${ADMIN_USER}:${ADMIN_PW}@localhost:${HTTP_PORT}/${ORG}/${REPO}.git" HEAD:main
cat <<EOF
Gitea bootstrap source ready (${DATA_DIR}).
UI (host): ${BASE}/${ORG}/${REPO}
Flux GitRepository (cluster): http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git
(patch this url into the overlay being bootstrapped:
clusters/prod today; clusters/dev has already cut over)
Admin creds (host, 0600): ${ADMIN_CRED_FILE}
Reachability: prod nodes reach ${HOST_NAT_IP}:${HTTP_PORT} directly on the LAN — no NAT, no
firewall change (verified 2026-09-04). For libvirt-based clusters ONLY, firewalld's libvirt zone
must allow VM->host:${HTTP_PORT} —
ansible-playbook infra/ansible/playbooks/host-prereqs.yml --ask-become-pass
REMINDER: the push above published the CURRENT HEAD as this repo's \`main\`. Whatever cluster points
at this source will reconcile THAT revision.
TEMPORARY BY DESIGN: this container is outside every cluster and is a single point of failure for
any cluster reconciling from it. Cut over to in-cluster Gitea (docs/05); for prod the cutover
procedure is written on the GitRepository patch in clusters/prod/kustomization.yaml.
EOF