Autonomous prod bring-up, operator away. Three changes that together clear the last blockers before `flux bootstrap`. === 1. TEMPORARY BOOTSTRAP GITEA — the last hard blocker, cleared === clusters/prod's source URL could not work: Gitea runs ON the dev cluster (the one slated for decommission) and split-horizon DNS resolves gitea.home.keithjackson.dev to PROD's OWN gateway. Stood up the existing bootstrap/gitea/bootstrap-gitea.sh on the operator's workstation with HOST_NAT_IP=192.168.2.132; all three prod nodes verified reachable (Gitea 1.22.6 via the API), no firewall change needed — prod is on the same /24, no NAT. clusters/prod now points at http://192.168.2.132:3000/... under a boxed TEMPORARY banner with a 5-step cutover checklist ending 'delete this whole block — dev's stanza is what done looks like'. Repo verified PUBLIC with anonymous read two independent ways (API "private":false; git ls-remote unauthenticated with GIT_TERMINAL_PROMPT=0), so NO secretRef is needed — documented so a future auth failure is not misread as a network fault. bootstrap-gitea.sh's header corrected: it documented HOST_NAT_IP=10.73.0.1, the RETIRED libvirt NAT, and forbade agent use; the operator has now directed this for prod. === 2. WIRELESS DISABLED ON PROD (operator-reported) === k3s-prod-3 was on ethernet AND wireless, and the wifi DHCP client was overwriting /etc/resolv.conf. Root cause of the silence: node_baseline's DNS `contended` detection checked resolvconf and NetworkManager but NOT dhcpcd, so the role wrote resolvers, reported success, and dhcpcd reverted them on lease renewal — the exact 'green run that fixed nothing' the contended branch exists to prevent. Now IaC, not a one-off: survey wireless hardware, **REFUSE to disable it if the path to the node runs over the wireless interface** (the guard that meant an agent dying mid-apply cost nothing), blacklist the driver so it never reloads, stop dhcpcd only where it holds a wireless interface and nothing else, take links down and unload modules with NO reboot, then report the resulting state. dhcpcd is now in the contended set without weakening fail-closed. Applied and verified: node 3's wlp1s0 is GONE from /sys/class/net, dhcpcd inactive and its unit not-found, ONE default route per node, all three on 8.8.8.8/1.1.1.1 resolving public names. Safe and idempotent on nodes 1 and 2, whose wifi hardware was already down. === 3. flux_bootstrap PATH PARAMETERISED === bootstrap/flux/bootstrap.sh already read FLUX_ENV_PATH but the role never exported it, so it ALWAYS bootstrapped clusters/dev — prod was impossible. Two behaviour-bearing lines: role default clusters/dev + `environment: FLUX_ENV_PATH`, overridden to clusters/prod in group_vars/k3s_prod. Proven per host with a read-only Ansible probe: prod hosts resolve clusters/prod, localhost/hypervisor resolves clusters/dev. Only two definitions repo-wide. DEV PROVEN BYTE-IDENTICAL for the source change: all 12 substituting Flux paths rendered for both trees, 27/27 SHA-256 artefacts identical, with TWO POSITIVE CONTROLS (an overlay-level canary and a deep-path content canary) confirming the harness detects change — and both canaries proven restored. Honest scope note recorded: kustomize strips comments, so the harness proves the built stream, not source comments. L0 substitution check: 106 assertions, 0 hazards. ci/l0.sh and ci/lint.sh exit 0. === STILL BLOCKING A FULL PROD RECONCILE (operator decisions) === - S3_BACKUP_ENDPOINT remains TBD-PROD-*: prod CANNOT reach dev's versitygw at 192.168.1.9:9000 (verified UNREACHABLE from all three — the pfSense boundary), and a prod backup target is a deferred operator decision. Deliberately NOT invented: a DR target on a workstation container would create false confidence. Velero/CNPG backups will fail loudly. - Prod's public resolvers mean the API server's oidc-issuer-url resolves PUBLICLY, so kubectl SSO will not work on prod until split-horizon is solved for it (§4). - The temp Gitea is a single container on the operator's workstation — a single point of failure for prod reconciliation, acceptable for bring-up only. Serves: R-GITOPS-01, R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
124 lines
7.3 KiB
Bash
Executable File
124 lines
7.3 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# bootstrap/gitea/bootstrap-gitea.sh — run a SINGLE Gitea container on the HOST (outside the
|
|
# cluster) as the GitOps bootstrap source. While a cluster is built, Flux (in-cluster) reconciles
|
|
# this repo from the host Gitea. This is the "local bootstrap source" that migrates to in-cluster
|
|
# Gitea at cutover (docs/05 "Source-of-truth", CONTRIBUTING.md). Dev has already cut over; PROD is
|
|
# using it now.
|
|
#
|
|
# HOW THE CLUSTER REACHES THE HOST — read this before setting HOST_NAT_IP.
|
|
# It is an ADDRESS OF THIS HOST as seen from the cluster nodes, and it differs per environment:
|
|
#
|
|
# PROD (bare metal, current use): HOST_NAT_IP=192.168.2.132
|
|
# The prod nodes (192.168.2.10/.11/.12) and this workstation are on the SAME LAN. There is
|
|
# NO NAT hop of any kind, and NO firewall change was needed — verified 2026-09-04 from all
|
|
# three nodes. The firewalld note further down is a DEV-ERA libvirt concern; it does not
|
|
# apply to prod, so do not "fix" prod by opening zones.
|
|
#
|
|
# DEV (libvirt VMs, HISTORICAL): HOST_NAT_IP=10.73.0.1 was the k3s-dev libvirt NAT gateway.
|
|
# *** THAT NETWORK IS RETIRED. 10.73.0.1 is stale and must not be reused. *** Dev's own
|
|
# bootstrap Gitea was later reached on 192.168.2.103:3000 and is retained STOPPED as
|
|
# recovery (see the comment on the GitRepository patch in clusters/dev/kustomization.yaml).
|
|
#
|
|
# WHO RUNS THIS: the OPERATOR, or an agent the operator has EXPLICITLY DIRECTED to stand up a
|
|
# bootstrap source for a specific bring-up (as happened 2026-09-04 for prod). It is NOT part of
|
|
# author-only work and must never be fired off casually or "to see if it works": it starts a
|
|
# long-lived container, generates and writes admin credentials to $HOME, creates an org and repo,
|
|
# and PUSHES THE CURRENT HEAD to that repo's `main` — which is the revision a cluster pointed at
|
|
# it will then reconcile. Know which HEAD you are publishing before you run it.
|
|
set -euo pipefail
|
|
ENGINE="${ENGINE:-podman}"
|
|
GITEA_IMAGE="${GITEA_IMAGE:-docker.io/gitea/gitea:1.22}" # pinned; bump if needed
|
|
NAME="${GITEA_NAME:-home-k3s-bootstrap-gitea}"
|
|
HTTP_PORT="${GITEA_HTTP_PORT:-3000}"
|
|
SSH_PORT="${GITEA_SSH_PORT:-2222}"
|
|
# Use a NAMED podman volume, not a host bind-mount: rootless podman maps the container's uid to a
|
|
# subuid that can't write a host-owned dir, so Gitea fails to create /data/gitea/conf/app.ini.
|
|
DATA_VOLUME="${GITEA_DATA_VOLUME:-gitea-bootstrap-data}"
|
|
DATA_DIR="podman volume ${DATA_VOLUME}" # display label only (data lives in the named volume)
|
|
# The address the CLUSTER uses to reach this host (see the header). ROOT_URL must match it so
|
|
# clone/redirect URLs are correct from inside the cluster.
|
|
#
|
|
# The DEFAULT IS DELIBERATELY UNSET — there is no correct default any more. The old default
|
|
# (10.73.0.1) was the retired k3s-dev libvirt NAT gateway; leaving it in place would silently
|
|
# publish a ROOT_URL no cluster can resolve, and Flux would report what looks like a network fault.
|
|
# Pass it explicitly, e.g. for prod: HOST_NAT_IP=192.168.2.132 bash bootstrap-gitea.sh
|
|
#
|
|
# NOTE (DEV/libvirt ONLY, historical): firewalld's `libvirt` zone (owns virbrN) blocked
|
|
# VM->host:3000 by default and had to be opened once with
|
|
# sudo firewall-cmd --zone=libvirt --add-port=3000/tcp --permanent && sudo firewall-cmd --reload
|
|
# PROD needs NO firewall change: same-LAN, no libvirt zone involved (verified 2026-09-04).
|
|
if [ -z "${HOST_NAT_IP:-}" ]; then
|
|
echo "ERROR: HOST_NAT_IP is not set — it is this host's address AS SEEN FROM THE CLUSTER." >&2
|
|
echo " prod: HOST_NAT_IP=192.168.2.132 (same LAN as the nodes, no NAT, no firewall change)" >&2
|
|
echo " There is no safe default: the old one (10.73.0.1) is the RETIRED libvirt NAT." >&2
|
|
exit 1
|
|
fi
|
|
ORG="${GITEA_ORG:-Home-Network}"
|
|
REPO="${GITEA_REPO:-home-network-k3s-impl}"
|
|
|
|
"$ENGINE" volume create "${DATA_VOLUME}" >/dev/null 2>&1 || true
|
|
if "$ENGINE" container exists "$NAME" 2>/dev/null; then
|
|
echo "Gitea container '${NAME}' already exists — starting it."
|
|
"$ENGINE" start "$NAME" >/dev/null
|
|
else
|
|
"$ENGINE" run -d --name "$NAME" --restart unless-stopped \
|
|
-p "${HTTP_PORT}:3000" -p "${SSH_PORT}:22" \
|
|
-v "${DATA_VOLUME}:/data" \
|
|
-e GITEA__server__ROOT_URL="http://${HOST_NAT_IP}:${HTTP_PORT}/" \
|
|
-e GITEA__server__SSH_PORT="${SSH_PORT}" \
|
|
-e GITEA__security__INSTALL_LOCK=true \
|
|
"$GITEA_IMAGE"
|
|
fi
|
|
|
|
# --- automated, idempotent setup (admin + PUBLIC repo + push) — NO manual steps (R-ENV-01) ---
|
|
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
|
ADMIN_CRED_FILE="${GITEA_ADMIN_CRED_FILE:-${HOME}/.config/home-k3s-bootstrap-gitea-admin.txt}"
|
|
BASE="http://localhost:${HTTP_PORT}" # host-side mgmt via localhost; the cluster uses HOST_NAT_IP
|
|
|
|
echo "Waiting for Gitea to be healthy..."
|
|
curl -fsS --retry 40 --retry-delay 2 --retry-all-errors "${BASE}/api/healthz" >/dev/null
|
|
|
|
# Admin credentials: generated once, stored 0600 on the host (never committed, never printed).
|
|
if [ ! -s "$ADMIN_CRED_FILE" ]; then
|
|
mkdir -p "$(dirname "$ADMIN_CRED_FILE")"
|
|
printf 'user=gitops\npass=%s\n' "$(openssl rand -base64 18 | tr -dc 'A-Za-z0-9' | head -c 22)" > "$ADMIN_CRED_FILE"
|
|
chmod 600 "$ADMIN_CRED_FILE"
|
|
fi
|
|
ADMIN_USER="$(sed -n 's/^user=//p' "$ADMIN_CRED_FILE")"
|
|
ADMIN_PW="$(sed -n 's/^pass=//p' "$ADMIN_CRED_FILE")"
|
|
|
|
# Create the admin user if missing (idempotent).
|
|
if ! "$ENGINE" exec -u git "$NAME" gitea admin user list 2>/dev/null | grep -qw "$ADMIN_USER"; then
|
|
"$ENGINE" exec -u git "$NAME" gitea admin user create \
|
|
--username "$ADMIN_USER" --password "$ADMIN_PW" --email "${ADMIN_USER}@home.local" \
|
|
--admin --must-change-password=false
|
|
fi
|
|
|
|
# Org + PUBLIC repo (idempotent — 'already exists' is fine; public => Flux needs no git creds).
|
|
curl -fsS -u "${ADMIN_USER}:${ADMIN_PW}" -H 'Content-Type: application/json' -X POST \
|
|
"${BASE}/api/v1/orgs" -d "{\"username\":\"${ORG}\"}" -o /dev/null 2>/dev/null || true
|
|
curl -fsS -u "${ADMIN_USER}:${ADMIN_PW}" -H 'Content-Type: application/json' -X POST \
|
|
"${BASE}/api/v1/orgs/${ORG}/repos" -d "{\"name\":\"${REPO}\",\"private\":false}" -o /dev/null 2>/dev/null || true
|
|
|
|
# Push current HEAD -> the bootstrap source's main.
|
|
git -C "$REPO_ROOT" push "http://${ADMIN_USER}:${ADMIN_PW}@localhost:${HTTP_PORT}/${ORG}/${REPO}.git" HEAD:main
|
|
|
|
cat <<EOF
|
|
|
|
Gitea bootstrap source ready (${DATA_DIR}).
|
|
UI (host): ${BASE}/${ORG}/${REPO}
|
|
Flux GitRepository (cluster): http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git
|
|
(patch this url into the overlay being bootstrapped:
|
|
clusters/prod today; clusters/dev has already cut over)
|
|
Admin creds (host, 0600): ${ADMIN_CRED_FILE}
|
|
Reachability: prod nodes reach ${HOST_NAT_IP}:${HTTP_PORT} directly on the LAN — no NAT, no
|
|
firewall change (verified 2026-09-04). For libvirt-based clusters ONLY, firewalld's libvirt zone
|
|
must allow VM->host:${HTTP_PORT} —
|
|
ansible-playbook infra/ansible/playbooks/host-prereqs.yml --ask-become-pass
|
|
REMINDER: the push above published the CURRENT HEAD as this repo's \`main\`. Whatever cluster points
|
|
at this source will reconcile THAT revision.
|
|
TEMPORARY BY DESIGN: this container is outside every cluster and is a single point of failure for
|
|
any cluster reconciling from it. Cut over to in-cluster Gitea (docs/05); for prod the cutover
|
|
procedure is written on the GitRepository patch in clusters/prod/kustomization.yaml.
|
|
EOF
|