jakman85andClaude Opus 5
12d4e0ae40
The operator's UOS bring-up becomes one field. apps/vms/unifi/unifi-uos-installer.sops.yaml
(SOPS/age, with a .example sibling per the crafty/headlamp convention) carries `installer-url`
and `console-password`. KubeVirt mounts it as a read-only iso9660 disk labelled UOSDATA and an
in-guest `uos-install.service` reads it on EVERY boot, so a later URL change needs only a commit
and a VM restart.
Mechanism verified against the DEPLOYED KubeVirt v1.9.0, not from memory:
* inline userData cap is exactly 2048 bytes — server-side dry-run apply accepted 2048 and
rejected 2049; the payload here is 1454.
* `cloudInitNoCloud.secretRef` and `secret.volumeLabel` both exist in the live CRD; the secret
disk + third disk were accepted by a server-side dry-run of the real manifest.
* the NoCloud seed's instance-id IS spec.template.spec.domain.firmware.uuid (read out of
noCloud.iso inside virt-launcher), stamped on the VM OBJECT, so it survives restarts. Editing
userData alone therefore changes NOTHING — cloud-init restores its cached datasource and skips
write_files/runcmd. The UUID is now pinned in git and bumped once so the new payload lands.
secretRef was deliberately NOT used for the whole cloud-config: it would put the shell in
ciphertext, out of reach of the L0 dollar-collapse guard, while Flux's envsubst still runs over
decrypted resources — the 2026-09-13 nextcloud-db shape. Script in plaintext, operator values in
SOPS. apps-vms had no `decryption` stanza (first encrypted content under apps/vms/), so one was
added to both overlays.
CORRECTION, and it is the headline: UNIFI OS SERVER IS ALREADY INSTALLED on the live guest. The
serial console log starts uosserver.service and uosserver-updater.service, and / is 6.3 GB of
40 GB used. The controller is unreachable because THE GUEST HAS NO IP: .status.interfaces shows
enp1s0 up with no address, and `ip neigh` in virt-launcher reports 10.0.2.2 FAILED. 11443/8443/
8080 and 22 all RST, from the pod IP and from 10.0.2.2 directly, so the VIP and the CNP are
exonerated. install-uos therefore bails out the moment systemd knows a uosserver.service —
without that guard this change would have reinstalled over a working controller.
There was also no way into the guest at all: no key, no password, and 22 is not a declared
masquerade port so SSH cannot work by construction. Hence console-password.
README corrections (it doubles as the rebuild runbook and was misdirecting):
* VIP 192.168.1.54 -> 192.168.2.21; pool 192.168.1.53-.60/.53 -> 192.168.2.20-.30 with the
Gateway on .20; device subnet 192.168.1.0/24 -> 192.168.2.0/24; node k3s-dev-1 -> k3s-prod-1.
All re-read live (pool blocks, cluster-vars, the Service EXTERNAL-IPs, LAN_CIDR).
* "UOS is NOT installed" -> it is; the real fault is the lost IP, with the evidence table.
* "apps-vms must NOT use postBuild.substituteFrom" -> it has since 2026-08-31 and the escape
contract is in force.
* "install-uos is no longer baked into the guest" -> it was baked in, and it was broken twice
over (an unverified firmware API, and no `install` subcommand).
* "SSH in, cloud-init installs the operator's key" -> it installs none, and 22 is not forwarded.
* externalTrafficPolicy Local -> Cluster (the 8-day outage fix already in 50-service-l4.yaml).
* 60 GB -> 40Gi (the DataVolume and the Bound PVC); "10 GB minimum" -> ~25 GB.
* userDataSecretRef -> the YAML key is `secretRef`.
* 90-httproute-optional.yaml referenced as if it existed; it was dropped.
Unattended install, honestly: Ubiquiti documents no silent flag, no answer file, no env vars.
`yes y | <installer> install` is the convergent community practice and is what ships here, with a
bare-invocation fallback. It has NOT been run end-to-end on our guest. The installer URL also
does not look signed or time-limited — it is account-derived and per-release; it is in SOPS
anyway, but nothing should depend on expiry.
The running VM was not touched: no restart, no migration, no re-import. Every cluster call was a
read or an explicit --dry-run=server; the live VM still carries firmware uuid ...c07 and the
34h-old virt-launcher pod is untouched. The change takes effect on the operator's next VM restart.
Co-Authored-By: Claude Opus 5 <[email protected]>
628 lines
40 KiB
YAML
628 lines
40 KiB
YAML
---
|
|
# clusters/prod — DESIGN-ONLY overlay. NOT a build target until roadmap Phase 7 opens prod
|
|
# (spec roadmap/phases.md, AGENTS.md §6: "Never build directly on production").
|
|
#
|
|
# When Phase 7 is opened, `flux bootstrap ... --path=clusters/prod` points the prod cluster's Flux
|
|
# here. Like dev, this overlay adds NO resources — it ONLY patches cluster-vars with prod-scoped
|
|
# values (R-ENV-02). Promotion to prod is a change to THESE variables, never a base edit.
|
|
#
|
|
# *** PHASE-7 BLOCKER — READ BEFORE BOOTSTRAPPING PROD ***
|
|
# `clusters/base/platform.yaml` now carries `dependsOn: [flux-system]` (the 2026-09-01 substitution-
|
|
# race fix). That names the ROOT Kustomization, which clusters/dev supplies as `dev/sync.yaml`
|
|
# (path ./clusters/dev). THIS overlay has no sync.yaml yet, so as written a prod bootstrap would
|
|
# leave `platform` stuck on "dependency 'flux-system/flux-system' is not ready" FOREVER and nothing
|
|
# downstream would reconcile. Phase 7 MUST add `clusters/prod/sync.yaml` — a copy of dev/sync.yaml
|
|
# with `path: ./clusters/prod` — as part of opening this overlay. (Adding it now is deliberately NOT
|
|
# done here: prod is design-only and its root object is the prod bring-up owner's call.)
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
resources:
|
|
- ../base
|
|
- sync.yaml # self-managing root Kustomization -> ./clusters/prod (REQUIRED: see below)
|
|
patches:
|
|
# --- SOPS decryption (operator decision 2026-09-01: prod uses the SAME age key as dev) ---
|
|
# Mirrors clusters/dev exactly, one stanza per Kustomization that carries encrypted Secrets.
|
|
# The `sops-age` Secret is created OUT OF BAND on the prod cluster
|
|
# (bootstrap/sops/install-age-key.sh) — referenced by NAME only, never inlined.
|
|
# Add a stanza here whenever `render.py --emit-waves` introduces a new deepest wave.
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: platform
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: apps-wave-0
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: apps-wave-1
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: apps-wave-2
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: oauth2-proxy
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: headlamp
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: crafty
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: velero
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
# apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret
|
|
# on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL,
|
|
# which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize
|
|
# would hand Flux the raw SOPS document and the VM would mount a disk full of ciphertext — the
|
|
# failure is silent from the cluster's point of view (the Secret applies fine), so the stanza is
|
|
# as load-bearing as the file itself. Mirrors clusters/dev. `sops-age` is out-of-band (name only).
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: apps-vms
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
# --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
|
|
#
|
|
# ############################################################################################
|
|
# ### ###
|
|
# ### *** T E M P O R A R Y B O O T S T R A P S O U R C E — NOT PERMANENT *** ###
|
|
# ### ###
|
|
# ### The url below is a Gitea container running ON THE OPERATOR'S WORKSTATION, outside ###
|
|
# ### any cluster. It exists ONLY to get prod's first reconcile off the ground. It MUST ###
|
|
# ### be cut over to the in-cluster URL once Gitea runs on prod. If you are reading this ###
|
|
# ### and prod has been serving traffic for months, THE CUTOVER WAS NEVER DONE — that is ###
|
|
# ### a live single point of failure, not a settled design. See "CUTOVER" below. ###
|
|
# ### ###
|
|
# ############################################################################################
|
|
#
|
|
# WHY THE IN-CLUSTER URL CANNOT BE USED YET (blocker found 2026-09-01, still true; the
|
|
# unusable value was `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`).
|
|
# Two independent reasons, both consequences of the split-horizon design:
|
|
# 1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's
|
|
# GitOps source lives inside the very cluster slated for decommission. Gitea must move to
|
|
# prod (or off-cluster) BEFORE dev is retired, or prod loses its source of truth.
|
|
# 2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml
|
|
# answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP}. On prod that is
|
|
# 192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve
|
|
# gitea.home.keithjackson.dev to itself and never reach a repository.
|
|
#
|
|
# WHAT THE TEMPORARY SOURCE IS (stood up 2026-09-04 on explicit operator direction):
|
|
# bootstrap/gitea/bootstrap-gitea.sh with HOST_NAT_IP=192.168.2.132 — a podman container on the
|
|
# operator's workstation, Gitea 1.22.6, listening on 192.168.2.132:3000. All three prod nodes
|
|
# reach it DIRECTLY on the LAN: no libvirt NAT hop (that network is retired) and no firewall
|
|
# change was needed. This is the same "local bootstrap source" pattern dev itself used before
|
|
# its Phase-6 cutover (docs/05 "Source-of-truth").
|
|
#
|
|
# PLAIN HTTP, AND WHY THAT IS ACCEPTED HERE: the bootstrap Gitea has no TLS, so there is no cert
|
|
# for Flux to trust and no caCertSecretRef to wire. Traffic is LAN-local between prod nodes and
|
|
# the workstation, the repo carries NO plaintext secrets (everything sensitive is SOPS-encrypted,
|
|
# .sops.yaml), and the window is the bring-up only. Do NOT extend this posture past cutover.
|
|
#
|
|
# NO secretRef, DELIBERATELY: the repo is PUBLIC on the bootstrap Gitea, so anonymous clone
|
|
# works and Flux needs no git credentials. VERIFIED 2026-09-04 by unauthenticated request:
|
|
# curl http://192.168.2.132:3000/api/v1/repos/Home-Network/home-network-k3s-impl -> "private":false
|
|
# git ls-remote http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git -> refs/heads/main
|
|
# If the GitRepository ever reports an auth failure that looks like a network problem, RE-CHECK
|
|
# THAT FIRST: a private repo here fails exactly that way.
|
|
#
|
|
# *** CUTOVER — the exit condition, mirroring dev's Phase-6 cutover (docs/05) ***
|
|
# WHEN: Gitea runs on prod (reconciled in-cluster) and answers gitea.home.keithjackson.dev
|
|
# through prod's own Gateway VIP with a publicly-trusted cert.
|
|
# DO: 1. push/mirror this repo into the prod in-cluster Gitea;
|
|
# 2. change the url below to
|
|
# https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
|
|
# and DELETE this whole TEMPORARY block (dev's equivalent stanza, ~15 lines above in
|
|
# clusters/dev/kustomization.yaml, is what "done" looks like);
|
|
# 3. confirm `flux get sources git -n flux-system` reconciles the new revision;
|
|
# 4. only THEN stop the workstation container. Keep it stopped-not-deleted as recovery,
|
|
# exactly as dev retained its bootstrap Gitea.
|
|
# UNTIL THEN: prod's reconciliation depends on a container on a workstation that may be
|
|
# powered off. Flux keeps the last-reconciled state running, but NO new revision lands while
|
|
# it is down. Tracked as an open blocker in infra/PROD-PROVISIONING.md §7.
|
|
#
|
|
# The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev.
|
|
# CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH
|
|
# CLUSTERS. There is no promotion step and no soak window — dev and prod move together.
|
|
# NOTE for bring-up: bootstrap-gitea.sh pushes the operator's CURRENT HEAD to the bootstrap
|
|
# Gitea's `main`, whatever branch that is. Before bootstrapping prod, confirm that remote `main`
|
|
# is the revision you intend prod to run (`git ls-remote` above, against local `main`).
|
|
- patch: |-
|
|
apiVersion: source.toolkit.fluxcd.io/v1
|
|
kind: GitRepository
|
|
metadata:
|
|
name: flux-system
|
|
namespace: flux-system
|
|
spec:
|
|
# CUTOVER DONE 2026-09-06 (operator direction: "move the repo to prod"). Prod now reconciles
|
|
# from PROD'S OWN in-cluster Gitea. The workstation bootstrap container
|
|
# (http://192.168.2.132:3000) is retired as prod's source — it was a podman container on the
|
|
# operator's workstation holding a DHCP lease on WIFI, i.e. production's source of truth was a
|
|
# laptop. It is kept only as a recovery path (bootstrap/gitea/bootstrap-gitea.sh).
|
|
#
|
|
# THIS LINE IS SELF-REFERENTIAL AND THAT IS THE WHOLE DIFFICULTY. The root Kustomization
|
|
# renders ./clusters/prod, which contains this patch — so a live `kubectl patch` of
|
|
# GitRepository.spec.url is REVERTED on the next reconcile unless the authored value moves
|
|
# too. The cutover therefore had to be: (1) push the repo to prod's Gitea, (2) fast-forward
|
|
# its `main` to the exact commit prod was already running so the source swap changed nothing
|
|
# else, (3) live-patch the url to open a window, (4) land THIS commit through that window so
|
|
# the authored value agrees. Dev hit the identical trap on 2026-09-05
|
|
# (infra/PROD-PROVISIONING.md §10a "D0/D1 as executed").
|
|
#
|
|
# Verified before the switch: a throwaway pod INSIDE prod resolved this host through
|
|
# in-cluster CoreDNS to the Gateway VIP and cloned over the public LE cert with no CA
|
|
# supplied. The workstation being able to reach it proves nothing about pod egress.
|
|
# MOVED WITH THE APEX FLIP (2026-09-13) — and it had to move in the SAME fetch.
|
|
# This URL is a hardcoded literal, NOT templated on ${BASE_DOMAIN}, so the flip did not
|
|
# carry it. But the flip DOES rename prod's own HTTPRoute hostnames, so:
|
|
# - external-dns (policy=sync) deletes gitea.prod.home.keithjackson.dev, and
|
|
# - platform/coredns/coredns-custom.yaml answers the whole ${BASE_DOMAIN} zone with
|
|
# ${GATEWAY_VIP}, so in-cluster resolution moves from prod.home. to home.
|
|
# Leaving this on the old name would therefore have severed prod from its OWN GitOps
|
|
# source moments after the flip reconciled — self-reference, the same class of trap as
|
|
# patching this url live and having the root Kustomization revert it. The fix has to land
|
|
# THROUGH THE OLD SOURCE, which is why it is in the same push as the flip rather than
|
|
# after it.
|
|
# FOLLOW-UP worth doing: point this at the in-cluster Service
|
|
# (gitea-http.gitea.svc.cluster.local) so prod's GitOps stops depending on public DNS and
|
|
# on its own Gateway at all. Deferred: it needs the TLS/credential path rethought.
|
|
# IN-CLUSTER SERVICE, NOT A PUBLIC HOSTNAME (2026-09-13). This is the durable fix for a
|
|
# deadlock the apex flip walked straight into, and it is worth stating precisely.
|
|
#
|
|
# The url used to be a public name. The flip renamed prod's HTTPRoute hostnames, so the
|
|
# name this GitRepository pointed at stopped serving — but the HTTPRoute that WOULD have
|
|
# served the new name lives in a DOWNSTREAM tier, which cannot reconcile while the source
|
|
# is broken. Source down -> routes never rename -> source stays down. Observed live:
|
|
# `srcReady=False`, "unable to list remote for https://gitea.home.keithjackson.dev".
|
|
#
|
|
# Pointing at the Service breaks the cycle permanently: prod's GitOps no longer depends on
|
|
# public DNS, on Cloudflare, on its own Gateway, or on ${BASE_DOMAIN} at all. A future
|
|
# domain change cannot sever prod from its own source again.
|
|
#
|
|
# Plain http is deliberate and safe here: it is cluster-internal traffic to a ClusterIP,
|
|
# TLS terminates at the Gateway for external clients only, the GitRepository has no
|
|
# secretRef (the repo is cloned anonymously), and every secret in the tree is SOPS-encrypted
|
|
# at rest regardless of transport.
|
|
url: http://gitea.gitea.svc.cluster.local:3000/Home-Network/home-network-k3s-impl.git
|
|
ref:
|
|
branch: main
|
|
- target:
|
|
kind: ConfigMap
|
|
name: cluster-vars
|
|
patch: |-
|
|
- op: replace
|
|
path: /data/CLUSTER_ENV
|
|
value: "prod"
|
|
# trusted, rate-limited real certs in prod — ADR-0006.
|
|
- op: replace
|
|
path: /data/ACME_CLUSTER_ISSUER
|
|
value: "letsencrypt-prod"
|
|
# ==========================================================================================
|
|
# PROD DNS IDENTITY (R-DNS-01/02/03) — the fix for the 2026-09-04 incident.
|
|
# ==========================================================================================
|
|
# PROD SERVES ITS OWN SUBDOMAIN. Until 2026-09-04 every hostname was HARDCODED to
|
|
# home.keithjackson.dev in the base manifests, so prod was authored to serve the EXACT SAME
|
|
# public names as the live dev cluster out of the SAME Cloudflare zone. Two consequences, one
|
|
# of which already happened:
|
|
# 1. HAPPENED: prod's external-dns (policy: sync, owner id `default` — the same implicit
|
|
# default dev had) concluded it owned dev's records and DELETED EVERY ONE of them.
|
|
# 2. QUEUED BEHIND IT: platform-config would have requested a `*.home.keithjackson.dev`
|
|
# wildcard from letsencrypt-prod for a cert dev already holds, and both clusters would
|
|
# have raced each other's `_acme-challenge` TXT records on that one zone.
|
|
# Moving prod to its own suffix dissolves both: the two clusters can no longer name the same
|
|
# record. This is what makes ACME_CLUSTER_ISSUER: letsencrypt-prod above safe here.
|
|
#
|
|
# dev -> home.keithjackson.dev (UNCHANGED — dev is live; phase 2 moves it)
|
|
# prod -> prod.home.keithjackson.dev (this knob)
|
|
#
|
|
# CONSEQUENCE — prod's hostnames are NOT the apex names. Prod's Gitea is
|
|
# gitea.prod.home.keithjackson.dev, its Keycloak keycloak.prod.home.keithjackson.dev, and so
|
|
# on; the wildcard cert becomes `*.prod.home.keithjackson.dev`. The apex CNAMEs that make
|
|
# `<svc>.home.keithjackson.dev` point at prod are PHASE 2, deliberately not done here — they
|
|
# would take traffic off the live dev cluster. infra/PROD-PROVISIONING.md §10.
|
|
#
|
|
# [parity] MUST equal k3s_base_domain in infra/ansible/inventory/group_vars/k3s_prod/vars.yml.
|
|
# The kube-apiserver's oidc-issuer-url is an INSTALL-TIME k3s flag, not a manifest, so it is
|
|
# set Ansible-side from that variable and cannot read this one. If they disagree, kubectl SSO
|
|
# trusts a different Keycloak than the one prod actually runs.
|
|
- op: replace
|
|
path: /data/BASE_DOMAIN
|
|
# APEX AS OF 2026-09-13. Prod now serves home.keithjackson.dev directly; dev has been
|
|
# decommissioned and released the name. Prerequisites completed and VERIFIED before this
|
|
# flip, in this order:
|
|
# 1. dev's 18 A records released by its own external-dns (policy=sync, sources deleted)
|
|
# 2. dev's 18 LEGACY UNPREFIXED TXT registry entries deleted from Cloudflare by hand —
|
|
# external-dns v0.21.0 removed only the `a-` prefixed form and then reported "All
|
|
# records are already up to date", so it would NEVER have cleared them. They carried
|
|
# `owner=default`, and prod runs `owner=prod`, so prod could not have claimed these
|
|
# names while they existed. Confirmed gone at the AUTHORITY (betty.ns.cloudflare.com);
|
|
# 1.1.1.1 served stale cache for minutes afterwards and must not be trusted here.
|
|
# 3. dev's `*.home.keithjackson.dev` + apex Certificate deleted, so it stops renewing.
|
|
# It was issued ~18d ago, so prod's request is #1 this week against LE's duplicate
|
|
# limit of 5 per identical name set per week.
|
|
# NOTE the zone has wildcards `*.keithjackson.dev` AND `*.home.keithjackson.dev` pointing
|
|
# at the public IP 136.51.115.252 (legacy non-k3s prod). Dev's specific records were
|
|
# SHADOWING that wildcard; prod's new specific records shadow it again. The legacy stack
|
|
# lives on `*.keithjackson.dev` and is untouched by any of this.
|
|
value: "home.keithjackson.dev"
|
|
# SAFETY NET #2 (independent of the owner id below): narrow ExternalDNS's name scope to prod's
|
|
# OWN subtree, so prod cannot create, update or delete any record outside it no matter what the
|
|
# TXT registry says. VERIFY ON FIRST RECONCILE — the Cloudflare provider uses this value for
|
|
# ZONE selection too, and on the version in use in 2026-08 a strict-subdomain filter dropped
|
|
# the parent zone ("no hosted zone matching record DNS Name"). If prod's external-dns logs no
|
|
# matching zone, it will create NOTHING — fail-closed, and it can still never delete dev's
|
|
# records. Fallback in that case: set this to "keithjackson.dev" (dev's value) and rely on
|
|
# EXTERNALDNS_TXT_OWNER_ID below, which is sufficient on its own.
|
|
- op: replace
|
|
path: /data/EXTERNALDNS_DOMAIN_FILTER
|
|
# MUST be the Cloudflare ZONE, not prod's subdomain. FINDING F-DNS1 (DECISIONS-LOG,
|
|
# 2026-08-17) proved a subdomain filter DROPS THE PARENT ZONE: ExternalDNS generated the
|
|
# record then skipped it with "no hosted zone matching record DNS Name", so prod would
|
|
# publish NOTHING. Isolation from dev does NOT come from this filter — it comes from
|
|
# (1) prod only requesting *.prod.home.keithjackson.dev hostnames, and
|
|
# (2) txtOwnerId: prod, so prod's registry never claims dev's `default`-owned records.
|
|
# That pairing is what the 2026-09-04 incident needed and this filter cannot provide.
|
|
value: "keithjackson.dev"
|
|
# SAFETY NET #1 — THE ACTUAL ROOT CAUSE OF THE INCIDENT. Both clusters left this unset and so
|
|
# both inherited the chart default `default`; prod then treated dev's TXT-registry entries as
|
|
# its own and, under policy: sync, deleted the records behind them. This value MUST differ from
|
|
# dev's (`default`, pinned in clusters/dev — dev's live records are registered under it and
|
|
# cannot be renamed without orphaning them). `prod` is distinct, which is the entire fix.
|
|
- op: replace
|
|
path: /data/EXTERNALDNS_TXT_OWNER_ID
|
|
value: "prod"
|
|
- op: replace
|
|
path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
|
|
value: "3"
|
|
# Longhorn's on-node replica directory: a SUBDIRECTORY of the claimed data partition, not
|
|
# the mount root. Prod's ~424 GB partition (LABEL=k3s-data) is mounted at the neutral /data
|
|
# and serves TWO tenants — /data/longhorn here, and /data/k3s for k3s's entire data dir
|
|
# (embedded etcd + the containerd image store), which had to move off the 11.8 GiB /var:
|
|
# measured on dev, containerd alone is 21-25 GB per node, so prod would have filled /var on
|
|
# its first image pulls and etcd could not have written its WAL. infra/PROD-PROVISIONING.md §2.
|
|
#
|
|
# [parity] MUST equal node_baseline_longhorn_data_path in
|
|
# infra/ansible/inventory/group_vars/k3s_prod/vars.yml — Ansible creates and mounts the
|
|
# directory, this tells Longhorn to use it. If they disagree, Longhorn writes replicas to a
|
|
# path on the OS filesystem instead of the data partition and /var fills up.
|
|
- op: replace
|
|
path: /data/LONGHORN_DATA_PATH
|
|
value: "/data/longhorn"
|
|
# longer retention / larger volumes on prod NVMe.
|
|
- op: replace
|
|
path: /data/MONITORING_PROMETHEUS_RETENTION
|
|
value: "30d"
|
|
- op: replace
|
|
path: /data/MONITORING_PROMETHEUS_PVC_SIZE
|
|
value: "50Gi"
|
|
# Size-cap below the 50Gi PVC (headroom for WAL + head + compaction scratch): prod may hit the
|
|
# size cap before 30d if series volume is high, keeping the disk from filling either way.
|
|
- op: replace
|
|
# LOWERED 45GiB -> 36GiB on 2026-09-13. 45 of 50Gi left only 5 GiB, which is not headroom for
|
|
# what the comment above already names: retentionSize counts ONLY persisted blocks, not the WAL,
|
|
# not the head's in-flight segments, and not compaction scratch, and a large compaction can need
|
|
# roughly the size of the blocks being merged. At 90% one bad compaction is ENOSPC, and ENOSPC on
|
|
# /prometheus wedges every WAL commit: scrapes keep succeeding, nothing is stored, and every
|
|
# absent()/==0 alert in the platform fires falsely at once. Costs NO history: the TSDB is 11.2 GiB
|
|
# at 30d today, so 36GiB is ~3x current and 30d stays the binding limit. To keep more history,
|
|
# grow the PVC rather than the fraction of it.
|
|
path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE
|
|
value: "36GiB"
|
|
- op: replace
|
|
path: /data/MONITORING_LOKI_PVC_SIZE
|
|
value: "50Gi"
|
|
# HA control-plane addons in prod.
|
|
- op: replace
|
|
path: /data/KEYCLOAK_REPLICAS
|
|
value: "2"
|
|
- op: replace
|
|
path: /data/OAUTH2_PROXY_REPLICAS
|
|
value: "2"
|
|
- op: replace
|
|
path: /data/VELERO_RETENTION
|
|
value: "720h"
|
|
# ==========================================================================================
|
|
# PROD LAN ADDRESSING (192.168.2.0/24). Nodes: k3s-prod-1/2/3 = .10 / .11 / .12.
|
|
# ==========================================================================================
|
|
# These knobs were HARDCODED to the dev LAN inside base manifests until 2026-08-31; prod would
|
|
# have come up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN. They are knobs now, and
|
|
# this overlay supplies every one of them. Anything the operator has NOT yet decided stays an
|
|
# INVALID `TBD-PROD-*` placeholder on purpose: base defaults are the DEV values, so an OMITTED
|
|
# op means prod silently inherits a dev address — far worse than a failed apply. `TBD-PROD-*`
|
|
# is not a valid IP/URL, so Cilium CRD validation / LB-IPAM / CNPG / Velero reject it LOUDLY.
|
|
#
|
|
# ------------------------------------------------------------------------------------------
|
|
# ADDRESS PLAN (operator, 2026-08-31) — see infra/PROD-PROVISIONING.md §4 for the full table.
|
|
# ------------------------------------------------------------------------------------------
|
|
# nodes 192.168.2.10 / .11 / .12 (Ansible inventory)
|
|
# control-plane VIP 192.168.2.9 (Ansible group_vars/k3s_prod — NOT here)
|
|
# ingress Gateway VIP 192.168.2.20 (GATEWAY_VIP, below)
|
|
# Cilium LB pool 192.168.2.20 - .30 (CILIUM_LB_POOL_*, below)
|
|
#
|
|
# CONTAINMENT INVARIANT SATISFIED: GATEWAY_VIP (.20) IS CILIUM_LB_POOL_START, mirroring dev
|
|
# (.53 of .53-.60). This matters because the `shared` Gateway (platform/gateway/gateway.yaml)
|
|
# pins NOTHING — no `lbipam.cilium.io/ips`, no spec.addresses (unlike platform/crafty and
|
|
# apps/vms/unifi, which DO pin) — so Cilium LB-IPAM assigns it the FIRST FREE address of a
|
|
# matching pool block. Had GATEWAY_VIP stayed outside every pool (the original .21-.30
|
|
# proposal), the Gateway would have come up on .21 while split-horizon coredns-custom kept
|
|
# answering *.home.keithjackson.dev with .20 — an address nothing announces — blackholing
|
|
# in-cluster hairpin traffic (OIDC clients -> Keycloak/Grafana) while looking healthy.
|
|
# KEEP GATEWAY_VIP == CILIUM_LB_POOL_START when changing either.
|
|
# The control-plane VIP (.9) is deliberately OUTSIDE this pool: kube-vip floats it, and
|
|
# LB-IPAM must never be able to hand it to a Service.
|
|
- op: replace
|
|
path: /data/GATEWAY_VIP
|
|
value: "192.168.2.20"
|
|
# Cilium LB IP pool, operator-supplied: 192.168.2.20-.30 (START == GATEWAY_VIP, see above).
|
|
# Mirror this range into the prod tofu reservation (cilium_l2_vip_range) if prod ever gets one.
|
|
- op: replace
|
|
path: /data/CILIUM_LB_POOL_START
|
|
value: "192.168.2.20"
|
|
- op: replace
|
|
path: /data/CILIUM_LB_POOL_STOP
|
|
value: "192.168.2.30"
|
|
# k3s API endpoint Cilium's kube-proxy replacement dials. This is the prod CONTROL-PLANE VIP
|
|
# (kube-vip), NOT .20 (the ingress VIP) and NOT the single node address .10.
|
|
#
|
|
# *** DECIDED 2026-09-06 by the operator: the VIP, as intended. *** (This block previously
|
|
# read "Not yet decided" while shipping .10 — the founder's own address — and that is what
|
|
# the LIVE prod cluster is running: all three cilium-agents carry
|
|
# KUBERNETES_SERVICE_HOST=192.168.2.10 while kube-vip holds .9, verified 2026-09-06. On a
|
|
# 3-node HA cluster that makes the CNI's control-plane path depend on k3s-prod-1: lose it and
|
|
# the datapath keeps running off Cilium's cache while no service or policy update lands, with
|
|
# nothing looking broken. The VIP is a virtual address that reaches the cluster wherever the
|
|
# control plane is, which is the whole reason kube-vip exists here.)
|
|
#
|
|
# It is owned Ansible-side by **k3s_apiserver_endpoint** (group_vars/all.yml, derived from
|
|
# this environment's kube_vip_address = 192.168.2.9 in group_vars/k3s_prod/vars.yml) and MUST
|
|
# equal that value — [parity] contract in platform/cilium/helmrelease.yaml. NOT k3s_api_host:
|
|
# that variable still exists and still holds the FOUNDER's .10, because the cold-bootstrap
|
|
# `--server` join needs a host that is up before any VIP is floated. Two addresses, two
|
|
# variables, deliberately.
|
|
#
|
|
# .9 sits OUTSIDE the Cilium LB pool (.20-.30) and is not a node address, so kube-vip and
|
|
# LB-IPAM can never be handed the same IP — asserted by
|
|
# conformance/tests/l0-static/ha-control-plane-as-code.
|
|
- op: replace
|
|
path: /data/K3S_API_HOST
|
|
value: "192.168.2.9"
|
|
# --- still UNDECIDED -> loud placeholders (must be real before `flux bootstrap`) ------------
|
|
# Pinned L4 VIPs, both carved out of the pool by lbipam.cilium.io/ips. Not yet assigned for
|
|
# prod. Leaving the dev literals in place would hand a prod LoadBalancer a dev-LAN address, so
|
|
# they stay placeholders; whatever is chosen MUST fall inside [POOL_START, POOL_STOP] and not
|
|
# collide with the Gateway VIP or the node IPs (.10/.11/.12).
|
|
- op: replace
|
|
path: /data/UNIFI_L4_VIP
|
|
value: "192.168.2.21"
|
|
- op: replace
|
|
path: /data/CRAFTY_L4_VIP
|
|
value: "192.168.2.22"
|
|
# PiHole's DNS listener VIP — udp/53 + tcp/53 for LAN clients (apps/wave-0/pihole-dns.yaml).
|
|
# 192.168.2.23 is the NEXT FREE address in the prod pool 192.168.2.20-.30, verified live
|
|
# 2026-09-14: the CiliumLoadBalancerIPPool `default-pool` reported IPsTotal 11 / IPsUsed 3 /
|
|
# IPsAvailable 8, and the only three allocations are .20 (cilium-gateway-shared), .21
|
|
# (unifi-l4-tcp/-udp) and .22 (crafty-l4-tcp/-udp). ARP-probed from a LAN host in the same
|
|
# broadcast domain (arping 192.168.2.23, 3 probes, 0 responses), so nothing off-cluster holds
|
|
# it either, and it is inside the pool block the operator already asserted sits outside the
|
|
# SOHO router's DHCP scope.
|
|
- op: replace
|
|
path: /data/PIHOLE_DNS_VIP
|
|
value: "192.168.2.23"
|
|
# The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns —
|
|
# NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path.
|
|
# It scopes the ingress allow for PiHole's DNS listener to LAN clients only.
|
|
- op: replace
|
|
path: /data/LAN_CIDR
|
|
value: "192.168.2.0/24"
|
|
# ------------------------------------------------------------------------------------------
|
|
# BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
|
|
# ------------------------------------------------------------------------------------------
|
|
# Backs every `class: bulk` volume: the 8.33 T media library (Storage/plex — plex, sonarr,
|
|
# radarr) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
|
|
#
|
|
# THE STORAGE HOST IS NOT REACHABLE FROM PROD AT ALL RIGHT NOW. It has one interface,
|
|
# vmbr0 192.168.1.9/24, on pfSense's LAN; prod's nodes are on 192.168.2.0/24, pfSense's WAN.
|
|
# 22 and 8006 answer from prod only because pfSense port-forwards them via 192.168.2.102;
|
|
# 2049 and 111 do not. Opening that path is an OPERATOR decision, and the two candidate
|
|
# outcomes give DIFFERENT values here:
|
|
# port-forward 192.168.2.102:2049 -> 192.168.1.9:2049 => "192.168.2.102"
|
|
# a WAN->LAN route to 192.168.1.0/24 => "192.168.1.9"
|
|
# Guessing either one authors a prod cluster that mounts nothing, or — worse, if the guess
|
|
# ever became reachable — mounts something unintended. So it stays a placeholder.
|
|
#
|
|
# WHICH PORTS THE DECISION MUST OPEN: exactly ONE, tcp/2049. The PVs mount nfsvers=4.1, and
|
|
# NFSv4 carries mount and lock in-protocol — no rpcbind/111, no mountd, no statd. Choosing v4
|
|
# over v3 was in part this decision's cost: v3 would have needed four ports and three of them
|
|
# on ephemeral numbers.
|
|
#
|
|
# WHAT THE EXPORT MUST ALLOW IS ALREADY SETTLED, and it is the same under BOTH outcomes.
|
|
# Measured 2026-09-07 from this workstation (192.168.2.132) through the existing port-forward:
|
|
# the storage host logged the connection as coming FROM 192.168.2.132, not from a pfSense LAN
|
|
# address (`ss -tn` peer 192.168.2.132, and the host's only route off-LAN is its default via
|
|
# 192.168.1.1). pfSense DNATs WITHOUT source NAT, so the NFS server will see prod's REAL node
|
|
# addresses either way. The additive `zfs set sharenfs=` values are in docs/BULK-NFS-STORAGE.md.
|
|
#
|
|
# *** BLAST RADIUS — READ BEFORE MERGING THIS TO A BRANCH PROD RECONCILES. ***
|
|
# A PersistentVolume's spec.nfs.server is an unvalidated string: the API server ACCEPTS this
|
|
# placeholder, the PV is created and the PVC BINDS to it (binding is pure control-plane). The
|
|
# failure surfaces only when a pod mounts — MountVolume.SetUp "Failed to resolve server
|
|
# TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr and nextcloud would sit in
|
|
# ContainerCreating on prod. Unlike the LAN knobs above, this placeholder does NOT fail at
|
|
# apply time and there is no CRD validation to lean on. Set this value BEFORE prod reconciles
|
|
# a revision containing the bulk volumes; the dev overlay renders a working value, which is
|
|
# where these manifests are meant to be proven first.
|
|
- op: replace
|
|
path: /data/BULK_NFS_SERVER
|
|
# 192.168.2.102 is pfSense's WAN address, which DNATs tcp/2049 through to the Proxmox
|
|
# host's REAL address 192.168.1.9 on pfSense's LAN. Prod sits on the WAN side and cannot
|
|
# reach 192.168.1.9 directly, so the cluster mounts the NAT address. Dev is on the LAN and
|
|
# keeps the base value 192.168.1.9.
|
|
#
|
|
# PROVEN from k3s-prod-1 on 2026-09-12, not inferred:
|
|
# - mount of 192.168.2.102:/mnt/Storage/plex negotiated nfs4 vers=4.1 proto=tcp
|
|
# - a write as uid 1000 succeeded. Tested as 1000 and NOT root deliberately: the export
|
|
# carries no_root_squash, so a root write would have passed and proved nothing, while
|
|
# LSIO images default to PUID 911 and would have gone Ready then failed every import
|
|
# with EACCES. The tree is 0755 1000:3000, so uid 1000 is the only writer.
|
|
# - ln() across downloads/ -> tv/ returned link count 2 (same inode), so the arrs get
|
|
# real atomic moves instead of silently degrading to copy-then-delete on EXDEV.
|
|
# pfSense DNATs WITHOUT source NAT (the host logs prod's own node addresses as the peer),
|
|
# so the ZFS export ACL names 192.168.2.10/.11/.12 directly rather than one translated IP.
|
|
value: "192.168.2.102"
|
|
# ==========================================================================================
|
|
# OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05; bucket + token 2026-09-07).
|
|
# ==========================================================================================
|
|
# Dev's versitygw/MinIO (192.168.1.9:9000) sits on the DEV side of the network and is NOT an
|
|
# option for prod, for two independent reasons:
|
|
# 1. UNREACHABLE. Prod is on the SOHO network 192.168.2.0/24 — pfSense's WAN — while that
|
|
# MinIO is inside pfSense's LAN. Measured from a pod INSIDE prod (the workstation proves
|
|
# nothing about pod egress): `curl http://192.168.1.9:9000/minio/health/live` → connection
|
|
# timed out after 15s. Reaching it would need a WAN→LAN rule that does not exist.
|
|
# 2. NOT OFF-CLUSTER DR. It runs on the same Proxmox host as dev's VMs, so it already shares
|
|
# a physical failure domain with dev. R2 is genuinely off-site, and its free egress means
|
|
# restore REHEARSALS cost nothing — R-STATE-03 has no financial disincentive to stay proven.
|
|
#
|
|
# THIS UNBLOCKS AN ACTIVE INCIDENT, not just a roadmap item. All three prod CNPG Clusters ship
|
|
# `archive_mode=on` with `endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`, so barman-cloud-wal-archive
|
|
# exits 4 on every segment and Postgres — correctly — REFUSES TO RECYCLE WAL. Verified live
|
|
# 2026-09-07: keycloak-db / nextcloud-db / semaphore-db all report
|
|
# ContinuousArchiving=False/ContinuousArchivingFailing, and keycloak-db's 5 GiB PVC is at
|
|
# 579 MiB used / 4360 MiB free and climbing monotonically. A disk-full PANIC there takes the
|
|
# cluster's SSO database down, and with it kubectl OIDC, Grafana, Headlamp, Gitea and
|
|
# forward-auth. Fixing the endpoint lets Postgres drain the whole backlog.
|
|
#
|
|
# ENDPOINT IS ACCOUNT-SCOPED, so it stays a prod-overlay value even though it is not a LAN
|
|
# address: this is the operator's Cloudflare account id, confirmed from inside a prod pod
|
|
# (`head-bucket` on k3s-prod-backup → BucketRegion WNAM; PUT and DELETE both succeed).
|
|
# It is NOT put in base, because base is dev's live MinIO and dev is deliberately STAYING
|
|
# there — moving dev to R2 would reset its PITR window and orphan the backups its PROVEN
|
|
# restore depends on. Two environments, two targets, isolated by URL *and* by bucket.
|
|
- op: replace
|
|
path: /data/S3_BACKUP_ENDPOINT
|
|
value: "https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com"
|
|
# R2 requires the region string to be exactly `auto`. (`head-bucket` answers
|
|
# `{"BucketRegion":"WNAM"}` — that is a PHYSICAL LOCATION HINT, not a signing region; using it
|
|
# here produces AuthorizationHeaderMalformed.) Dev keeps the base `us-east-1` its live MinIO
|
|
# has always been signed with.
|
|
- op: replace
|
|
path: /data/S3_BACKUP_REGION
|
|
value: "auto"
|
|
# ------------------------------------------------------------------------------------------
|
|
# OBJECT TENANCY INSIDE THE ONE BUCKET. The operator created a SINGLE R2 bucket, so velero and
|
|
# barman cannot be separated by bucket the way they are on dev's MinIO (`velero` and `cnpg`).
|
|
# They are separated by disjoint TOP-LEVEL PREFIXES instead:
|
|
#
|
|
# s3://k3s-prod-backup/velero/… Velero: kopia repo + backup metadata
|
|
# s3://k3s-prod-backup/barman/<clusterName>/… CNPG: WAL + base backups
|
|
#
|
|
# The bucket ROOT is left empty on purpose — a misrendered prefix then lands somewhere
|
|
# obviously wrong instead of silently colliding with a live tenant. Velero's retention TTL
|
|
# deletes objects under its own prefix only, so it can never reap a WAL segment; and barman's
|
|
# per-cluster serverName subdirectory keeps the three Postgres servers' chains apart, which
|
|
# matters because interleaving two servers' WAL into one prefix corrupts the chain outright.
|
|
# Dev cannot be reached from any of these: different endpoint AND different bucket names.
|
|
# ------------------------------------------------------------------------------------------
|
|
- op: replace
|
|
path: /data/VELERO_BACKUP_BUCKET
|
|
value: "k3s-prod-backup"
|
|
- op: replace
|
|
path: /data/VELERO_BACKUP_PREFIX
|
|
value: "velero"
|
|
- op: replace
|
|
path: /data/CNPG_BACKUP_PATH
|
|
value: "s3://k3s-prod-backup/barman"
|
|
# ------------------------------------------------------------------------------------------
|
|
# WHICH CREDENTIAL SIGNS THE REQUESTS TO THAT BUCKET (per-env credential split, 2026-09-07).
|
|
# ------------------------------------------------------------------------------------------
|
|
# The knobs above moved the ENDPOINT and the OBJECT LAYOUT per environment. The credential did
|
|
# not move with them, and it could not: the four backup-credential Secrets are SHARED BASE
|
|
# manifests, applied to both clusters by the same Flux tiers. Writing the R2 token into them
|
|
# gave prod a working key by TAKING DEV'S AWAY — dev would have signed MinIO requests with an
|
|
# R2 key id, and under archive_mode=on that is a filling pgdata PVC, not a missing backup.
|
|
#
|
|
# So each credential is now two objects, and this suffix selects prod's:
|
|
# velero-cloud-credentials-prod the R2 token in the AWS INI `cloud` shape
|
|
# <db>-backup-credentials-prod the same token as ACCESS_KEY_ID / ACCESS_SECRET_KEY
|
|
# for keycloak-db, nextcloud-db and semaphore-db. Both halves ship to both clusters (they are
|
|
# base resources); ONLY this value decides which one each consumer reads. It is a SUFFIX rather
|
|
# than four secret-name knobs deliberately — four could be updated three-quarters of the way,
|
|
# leaving one consumer on dev's MinIO key with nothing rendering wrong, and strict-mode
|
|
# substitution cannot catch a defined-but-stale variable.
|
|
#
|
|
# THE R2 TOKEN IS THE ONE PROVEN FROM INSIDE PROD (chore/r2-credentials 0a6a862): head-bucket on
|
|
# k3s-prod-backup returns BucketRegion WNAM, and PUT and DELETE both succeed. Rotating it now
|
|
# touches ONLY the four `-prod` files; dev's MinIO keys are untouched objects. That isolation is
|
|
# the structural reason for the split, independent of dev's eventual decommission.
|
|
- op: replace
|
|
path: /data/BACKUP_CREDENTIALS_SUFFIX
|
|
value: "-prod"
|
|
# TODO(WS-INFRA, Phase 7): pin prod node placement once the bare-metal role labels exist.
|