--- # clusters/prod — DESIGN-ONLY overlay. NOT a build target until roadmap Phase 7 opens prod # (spec roadmap/phases.md, AGENTS.md §6: "Never build directly on production"). # # When Phase 7 is opened, `flux bootstrap ... --path=clusters/prod` points the prod cluster's Flux # here. Like dev, this overlay adds NO resources — it ONLY patches cluster-vars with prod-scoped # values (R-ENV-02). Promotion to prod is a change to THESE variables, never a base edit. # # *** PHASE-7 BLOCKER — READ BEFORE BOOTSTRAPPING PROD *** # `clusters/base/platform.yaml` now carries `dependsOn: [flux-system]` (the 2026-09-01 substitution- # race fix). That names the ROOT Kustomization, which clusters/dev supplies as `dev/sync.yaml` # (path ./clusters/dev). THIS overlay has no sync.yaml yet, so as written a prod bootstrap would # leave `platform` stuck on "dependency 'flux-system/flux-system' is not ready" FOREVER and nothing # downstream would reconcile. Phase 7 MUST add `clusters/prod/sync.yaml` — a copy of dev/sync.yaml # with `path: ./clusters/prod` — as part of opening this overlay. (Adding it now is deliberately NOT # done here: prod is design-only and its root object is the prod bring-up owner's call.) apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../base - sync.yaml # self-managing root Kustomization -> ./clusters/prod (REQUIRED: see below) patches: # --- SOPS decryption (operator decision 2026-09-01: prod uses the SAME age key as dev) --- # Mirrors clusters/dev exactly, one stanza per Kustomization that carries encrypted Secrets. # The `sops-age` Secret is created OUT OF BAND on the prod cluster # (bootstrap/sops/install-age-key.sh) — referenced by NAME only, never inlined. # Add a stanza here whenever `render.py --emit-waves` introduces a new deepest wave. - patch: |- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: platform namespace: flux-system spec: decryption: provider: sops secretRef: name: sops-age - patch: |- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: apps-wave-0 namespace: flux-system spec: decryption: provider: sops secretRef: name: sops-age - patch: |- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: apps-wave-1 namespace: flux-system spec: decryption: provider: sops secretRef: name: sops-age - patch: |- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: apps-wave-2 namespace: flux-system spec: decryption: provider: sops secretRef: name: sops-age - patch: |- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: oauth2-proxy namespace: flux-system spec: decryption: provider: sops secretRef: name: sops-age - patch: |- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: headlamp namespace: flux-system spec: decryption: provider: sops secretRef: name: sops-age - patch: |- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: crafty namespace: flux-system spec: decryption: provider: sops secretRef: name: sops-age - patch: |- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: velero namespace: flux-system spec: decryption: provider: sops secretRef: name: sops-age # vpn-ingress (the WireGuard remote-access VPN, clusters/base/vpn-ingress.yaml) carries # platform/vpn-ingress/wg-server-key.sops.yaml — the WireGuard SERVER private key, the one # secret in that component. Without this stanza kustomize hands Flux the raw ciphertext and the # Secret applies full of ENC[...]: the APPLY SUCCEEDS and every Flux condition stays green, and # the only symptom is a VPN that no client can handshake with. The DaemonSet's init container # validates the value with `wg pubkey` specifically so that state is a named CrashLoop instead. # The `sops-age` Secret is out-of-band (name only). - patch: |- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: vpn-ingress namespace: flux-system spec: decryption: provider: sops secretRef: name: sops-age # cloudflared (the Cloudflare Tunnel connector, clusters/base/cloudflared.yaml) carries # platform/cloudflared/cloudflared-credentials.sops.yaml — the tunnel credentials JSON, the one # secret in that component. Without this stanza the Secret applies full of ENC[...]: the APPLY # SUCCEEDS, the Kustomization stays Ready, and the only symptom is a connector that cannot # authenticate to the edge (CloudflaredTunnelDown fires). The `sops-age` Secret is out-of-band # (name only). - patch: |- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: cloudflared namespace: flux-system spec: decryption: provider: sops secretRef: name: sops-age # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot # account and the Anthropic subscription token from `claude setup-token`. Without this # stanza kustomize hands Flux the raw SOPS document and the Secret applies full of # ciphertext: the APPLY SUCCEEDS, and the only symptom is a dispatcher that cannot # authenticate to Gitea — so this stanza is as load-bearing as the file it decrypts (same # silent-failure shape as the apps-vms note below/above). The `sops-age` Secret is created # out-of-band (name only). - patch: |- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: pm-agent namespace: flux-system spec: decryption: provider: sops secretRef: name: sops-age # apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret # on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL, # which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize # would hand Flux the raw SOPS document and the VM would mount a disk full of ciphertext — the # failure is silent from the cluster's point of view (the Secret applies fine), so the stanza is # as load-bearing as the file itself. Mirrors clusters/dev. `sops-age` is out-of-band (name only). - patch: |- apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: apps-vms namespace: flux-system spec: decryption: provider: sops secretRef: name: sops-age # --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) --- # # ############################################################################################ # ### ### # ### *** T E M P O R A R Y B O O T S T R A P S O U R C E — NOT PERMANENT *** ### # ### ### # ### The url below is a Gitea container running ON THE OPERATOR'S WORKSTATION, outside ### # ### any cluster. It exists ONLY to get prod's first reconcile off the ground. It MUST ### # ### be cut over to the in-cluster URL once Gitea runs on prod. If you are reading this ### # ### and prod has been serving traffic for months, THE CUTOVER WAS NEVER DONE — that is ### # ### a live single point of failure, not a settled design. See "CUTOVER" below. ### # ### ### # ############################################################################################ # # WHY THE IN-CLUSTER URL CANNOT BE USED YET (blocker found 2026-09-01, still true; the # unusable value was `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`). # Two independent reasons, both consequences of the split-horizon design: # 1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's # GitOps source lives inside the very cluster slated for decommission. Gitea must move to # prod (or off-cluster) BEFORE dev is retired, or prod loses its source of truth. # 2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml # answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP}. On prod that is # 192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve # gitea.home.keithjackson.dev to itself and never reach a repository. # # WHAT THE TEMPORARY SOURCE IS (stood up 2026-09-04 on explicit operator direction): # bootstrap/gitea/bootstrap-gitea.sh with HOST_NAT_IP=192.168.2.132 — a podman container on the # operator's workstation, Gitea 1.22.6, listening on 192.168.2.132:3000. All three prod nodes # reach it DIRECTLY on the LAN: no libvirt NAT hop (that network is retired) and no firewall # change was needed. This is the same "local bootstrap source" pattern dev itself used before # its Phase-6 cutover (docs/05 "Source-of-truth"). # # PLAIN HTTP, AND WHY THAT IS ACCEPTED HERE: the bootstrap Gitea has no TLS, so there is no cert # for Flux to trust and no caCertSecretRef to wire. Traffic is LAN-local between prod nodes and # the workstation, the repo carries NO plaintext secrets (everything sensitive is SOPS-encrypted, # .sops.yaml), and the window is the bring-up only. Do NOT extend this posture past cutover. # # NO secretRef, DELIBERATELY: the repo is PUBLIC on the bootstrap Gitea, so anonymous clone # works and Flux needs no git credentials. VERIFIED 2026-09-04 by unauthenticated request: # curl http://192.168.2.132:3000/api/v1/repos/Home-Network/home-network-k3s-impl -> "private":false # git ls-remote http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git -> refs/heads/main # If the GitRepository ever reports an auth failure that looks like a network problem, RE-CHECK # THAT FIRST: a private repo here fails exactly that way. # # *** CUTOVER — the exit condition, mirroring dev's Phase-6 cutover (docs/05) *** # WHEN: Gitea runs on prod (reconciled in-cluster) and answers gitea.home.keithjackson.dev # through prod's own Gateway VIP with a publicly-trusted cert. # DO: 1. push/mirror this repo into the prod in-cluster Gitea; # 2. change the url below to # https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git # and DELETE this whole TEMPORARY block (dev's equivalent stanza, ~15 lines above in # clusters/dev/kustomization.yaml, is what "done" looks like); # 3. confirm `flux get sources git -n flux-system` reconciles the new revision; # 4. only THEN stop the workstation container. Keep it stopped-not-deleted as recovery, # exactly as dev retained its bootstrap Gitea. # UNTIL THEN: prod's reconciliation depends on a container on a workstation that may be # powered off. Flux keeps the last-reconciled state running, but NO new revision lands while # it is down. Tracked as an open blocker in infra/PROD-PROVISIONING.md §7. # # The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev. # CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH # CLUSTERS. There is no promotion step and no soak window — dev and prod move together. # NOTE for bring-up: bootstrap-gitea.sh pushes the operator's CURRENT HEAD to the bootstrap # Gitea's `main`, whatever branch that is. Before bootstrapping prod, confirm that remote `main` # is the revision you intend prod to run (`git ls-remote` above, against local `main`). - patch: |- apiVersion: source.toolkit.fluxcd.io/v1 kind: GitRepository metadata: name: flux-system namespace: flux-system spec: # CUTOVER DONE 2026-09-06 (operator direction: "move the repo to prod"). Prod now reconciles # from PROD'S OWN in-cluster Gitea. The workstation bootstrap container # (http://192.168.2.132:3000) is retired as prod's source — it was a podman container on the # operator's workstation holding a DHCP lease on WIFI, i.e. production's source of truth was a # laptop. It is kept only as a recovery path (bootstrap/gitea/bootstrap-gitea.sh). # # THIS LINE IS SELF-REFERENTIAL AND THAT IS THE WHOLE DIFFICULTY. The root Kustomization # renders ./clusters/prod, which contains this patch — so a live `kubectl patch` of # GitRepository.spec.url is REVERTED on the next reconcile unless the authored value moves # too. The cutover therefore had to be: (1) push the repo to prod's Gitea, (2) fast-forward # its `main` to the exact commit prod was already running so the source swap changed nothing # else, (3) live-patch the url to open a window, (4) land THIS commit through that window so # the authored value agrees. Dev hit the identical trap on 2026-09-05 # (infra/PROD-PROVISIONING.md §10a "D0/D1 as executed"). # # Verified before the switch: a throwaway pod INSIDE prod resolved this host through # in-cluster CoreDNS to the Gateway VIP and cloned over the public LE cert with no CA # supplied. The workstation being able to reach it proves nothing about pod egress. # MOVED WITH THE APEX FLIP (2026-09-13) — and it had to move in the SAME fetch. # This URL is a hardcoded literal, NOT templated on ${BASE_DOMAIN}, so the flip did not # carry it. But the flip DOES rename prod's own HTTPRoute hostnames, so: # - external-dns (policy=sync) deletes gitea.prod.home.keithjackson.dev, and # - platform/coredns/coredns-custom.yaml answers the whole ${BASE_DOMAIN} zone with # ${GATEWAY_VIP}, so in-cluster resolution moves from prod.home. to home. # Leaving this on the old name would therefore have severed prod from its OWN GitOps # source moments after the flip reconciled — self-reference, the same class of trap as # patching this url live and having the root Kustomization revert it. The fix has to land # THROUGH THE OLD SOURCE, which is why it is in the same push as the flip rather than # after it. # FOLLOW-UP worth doing: point this at the in-cluster Service # (gitea-http.gitea.svc.cluster.local) so prod's GitOps stops depending on public DNS and # on its own Gateway at all. Deferred: it needs the TLS/credential path rethought. # IN-CLUSTER SERVICE, NOT A PUBLIC HOSTNAME (2026-09-13). This is the durable fix for a # deadlock the apex flip walked straight into, and it is worth stating precisely. # # The url used to be a public name. The flip renamed prod's HTTPRoute hostnames, so the # name this GitRepository pointed at stopped serving — but the HTTPRoute that WOULD have # served the new name lives in a DOWNSTREAM tier, which cannot reconcile while the source # is broken. Source down -> routes never rename -> source stays down. Observed live: # `srcReady=False`, "unable to list remote for https://gitea.home.keithjackson.dev". # # Pointing at the Service breaks the cycle permanently: prod's GitOps no longer depends on # public DNS, on Cloudflare, on its own Gateway, or on ${BASE_DOMAIN} at all. A future # domain change cannot sever prod from its own source again. # # Plain http is deliberate and safe here: it is cluster-internal traffic to a ClusterIP, # TLS terminates at the Gateway for external clients only, the GitRepository has no # secretRef (the repo is cloned anonymously), and every secret in the tree is SOPS-encrypted # at rest regardless of transport. url: http://gitea.gitea.svc.cluster.local:3000/Home-Network/home-network-k3s-impl.git ref: branch: main - target: kind: ConfigMap name: cluster-vars patch: |- - op: replace path: /data/CLUSTER_ENV value: "prod" # trusted, rate-limited real certs in prod — ADR-0006. - op: replace path: /data/ACME_CLUSTER_ISSUER value: "letsencrypt-prod" # ========================================================================================== # PROD DNS IDENTITY (R-DNS-01/02/03) — the fix for the 2026-09-04 incident. # ========================================================================================== # PROD SERVES ITS OWN SUBDOMAIN. Until 2026-09-04 every hostname was HARDCODED to # home.keithjackson.dev in the base manifests, so prod was authored to serve the EXACT SAME # public names as the live dev cluster out of the SAME Cloudflare zone. Two consequences, one # of which already happened: # 1. HAPPENED: prod's external-dns (policy: sync, owner id `default` — the same implicit # default dev had) concluded it owned dev's records and DELETED EVERY ONE of them. # 2. QUEUED BEHIND IT: platform-config would have requested a `*.home.keithjackson.dev` # wildcard from letsencrypt-prod for a cert dev already holds, and both clusters would # have raced each other's `_acme-challenge` TXT records on that one zone. # Moving prod to its own suffix dissolves both: the two clusters can no longer name the same # record. This is what makes ACME_CLUSTER_ISSUER: letsencrypt-prod above safe here. # # dev -> home.keithjackson.dev (UNCHANGED — dev is live; phase 2 moves it) # prod -> prod.home.keithjackson.dev (this knob) # # CONSEQUENCE — prod's hostnames are NOT the apex names. Prod's Gitea is # gitea.prod.home.keithjackson.dev, its Keycloak keycloak.prod.home.keithjackson.dev, and so # on; the wildcard cert becomes `*.prod.home.keithjackson.dev`. The apex CNAMEs that make # `.home.keithjackson.dev` point at prod are PHASE 2, deliberately not done here — they # would take traffic off the live dev cluster. infra/PROD-PROVISIONING.md §10. # # [parity] MUST equal k3s_base_domain in infra/ansible/inventory/group_vars/k3s_prod/vars.yml. # The kube-apiserver's oidc-issuer-url is an INSTALL-TIME k3s flag, not a manifest, so it is # set Ansible-side from that variable and cannot read this one. If they disagree, kubectl SSO # trusts a different Keycloak than the one prod actually runs. - op: replace path: /data/BASE_DOMAIN # APEX AS OF 2026-09-13. Prod now serves home.keithjackson.dev directly; dev has been # decommissioned and released the name. Prerequisites completed and VERIFIED before this # flip, in this order: # 1. dev's 18 A records released by its own external-dns (policy=sync, sources deleted) # 2. dev's 18 LEGACY UNPREFIXED TXT registry entries deleted from Cloudflare by hand — # external-dns v0.21.0 removed only the `a-` prefixed form and then reported "All # records are already up to date", so it would NEVER have cleared them. They carried # `owner=default`, and prod runs `owner=prod`, so prod could not have claimed these # names while they existed. Confirmed gone at the AUTHORITY (betty.ns.cloudflare.com); # 1.1.1.1 served stale cache for minutes afterwards and must not be trusted here. # 3. dev's `*.home.keithjackson.dev` + apex Certificate deleted, so it stops renewing. # It was issued ~18d ago, so prod's request is #1 this week against LE's duplicate # limit of 5 per identical name set per week. # NOTE the zone has wildcards `*.keithjackson.dev` AND `*.home.keithjackson.dev` pointing # at the public IP 136.51.115.252 (legacy non-k3s prod). Dev's specific records were # SHADOWING that wildcard; prod's new specific records shadow it again. The legacy stack # lives on `*.keithjackson.dev` and is untouched by any of this. value: "home.keithjackson.dev" # SAFETY NET #2 (independent of the owner id below): narrow ExternalDNS's name scope to prod's # OWN subtree, so prod cannot create, update or delete any record outside it no matter what the # TXT registry says. VERIFY ON FIRST RECONCILE — the Cloudflare provider uses this value for # ZONE selection too, and on the version in use in 2026-08 a strict-subdomain filter dropped # the parent zone ("no hosted zone matching record DNS Name"). If prod's external-dns logs no # matching zone, it will create NOTHING — fail-closed, and it can still never delete dev's # records. Fallback in that case: set this to "keithjackson.dev" (dev's value) and rely on # EXTERNALDNS_TXT_OWNER_ID below, which is sufficient on its own. - op: replace path: /data/EXTERNALDNS_DOMAIN_FILTER # MUST be the Cloudflare ZONE, not prod's subdomain. FINDING F-DNS1 (DECISIONS-LOG, # 2026-08-17) proved a subdomain filter DROPS THE PARENT ZONE: ExternalDNS generated the # record then skipped it with "no hosted zone matching record DNS Name", so prod would # publish NOTHING. Isolation from dev does NOT come from this filter — it comes from # (1) prod only requesting *.prod.home.keithjackson.dev hostnames, and # (2) txtOwnerId: prod, so prod's registry never claims dev's `default`-owned records. # That pairing is what the 2026-09-04 incident needed and this filter cannot provide. value: "keithjackson.dev" # SAFETY NET #1 — THE ACTUAL ROOT CAUSE OF THE INCIDENT. Both clusters left this unset and so # both inherited the chart default `default`; prod then treated dev's TXT-registry entries as # its own and, under policy: sync, deleted the records behind them. This value MUST differ from # dev's (`default`, pinned in clusters/dev — dev's live records are registered under it and # cannot be renamed without orphaning them). `prod` is distinct, which is the entire fix. - op: replace path: /data/EXTERNALDNS_TXT_OWNER_ID value: "prod" - op: replace path: /data/LONGHORN_DEFAULT_REPLICA_COUNT value: "3" # Longhorn's on-node replica directory: a SUBDIRECTORY of the claimed data partition, not # the mount root. Prod's ~424 GB partition (LABEL=k3s-data) is mounted at the neutral /data # and serves TWO tenants — /data/longhorn here, and /data/k3s for k3s's entire data dir # (embedded etcd + the containerd image store), which had to move off the 11.8 GiB /var: # measured on dev, containerd alone is 21-25 GB per node, so prod would have filled /var on # its first image pulls and etcd could not have written its WAL. infra/PROD-PROVISIONING.md §2. # # [parity] MUST equal node_baseline_longhorn_data_path in # infra/ansible/inventory/group_vars/k3s_prod/vars.yml — Ansible creates and mounts the # directory, this tells Longhorn to use it. If they disagree, Longhorn writes replicas to a # path on the OS filesystem instead of the data partition and /var fills up. - op: replace path: /data/LONGHORN_DATA_PATH value: "/data/longhorn" # longer retention / larger volumes on prod NVMe. - op: replace path: /data/MONITORING_PROMETHEUS_RETENTION value: "30d" - op: replace path: /data/MONITORING_PROMETHEUS_PVC_SIZE value: "50Gi" # Size-cap below the 50Gi PVC (headroom for WAL + head + compaction scratch): prod may hit the # size cap before 30d if series volume is high, keeping the disk from filling either way. - op: replace # LOWERED 45GiB -> 36GiB on 2026-09-13. 45 of 50Gi left only 5 GiB, which is not headroom for # what the comment above already names: retentionSize counts ONLY persisted blocks, not the WAL, # not the head's in-flight segments, and not compaction scratch, and a large compaction can need # roughly the size of the blocks being merged. At 90% one bad compaction is ENOSPC, and ENOSPC on # /prometheus wedges every WAL commit: scrapes keep succeeding, nothing is stored, and every # absent()/==0 alert in the platform fires falsely at once. Costs NO history: the TSDB is 11.2 GiB # at 30d today, so 36GiB is ~3x current and 30d stays the binding limit. To keep more history, # grow the PVC rather than the fraction of it. path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE value: "36GiB" - op: replace path: /data/MONITORING_LOKI_PVC_SIZE value: "50Gi" # HA control-plane addons in prod. - op: replace path: /data/KEYCLOAK_REPLICAS value: "2" - op: replace path: /data/OAUTH2_PROXY_REPLICAS value: "2" - op: replace path: /data/VELERO_RETENTION value: "720h" # ========================================================================================== # PROD LAN ADDRESSING (192.168.2.0/24). Nodes: k3s-prod-1/2/3 = .10 / .11 / .12. # ========================================================================================== # These knobs were HARDCODED to the dev LAN inside base manifests until 2026-08-31; prod would # have come up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN. They are knobs now, and # this overlay supplies every one of them. Anything the operator has NOT yet decided stays an # INVALID `TBD-PROD-*` placeholder on purpose: base defaults are the DEV values, so an OMITTED # op means prod silently inherits a dev address — far worse than a failed apply. `TBD-PROD-*` # is not a valid IP/URL, so Cilium CRD validation / LB-IPAM / CNPG / Velero reject it LOUDLY. # # ------------------------------------------------------------------------------------------ # ADDRESS PLAN (operator, 2026-08-31) — see infra/PROD-PROVISIONING.md §4 for the full table. # ------------------------------------------------------------------------------------------ # nodes 192.168.2.10 / .11 / .12 (Ansible inventory) # control-plane VIP 192.168.2.9 (Ansible group_vars/k3s_prod — NOT here) # ingress Gateway VIP 192.168.2.20 (GATEWAY_VIP, below) # Cilium LB pool 192.168.2.20 - .30 (CILIUM_LB_POOL_*, below) # # CONTAINMENT INVARIANT SATISFIED: GATEWAY_VIP (.20) IS CILIUM_LB_POOL_START, mirroring dev # (.53 of .53-.60). This matters because the `shared` Gateway (platform/gateway/gateway.yaml) # pins NOTHING — no `lbipam.cilium.io/ips`, no spec.addresses (unlike platform/crafty and # apps/vms/unifi, which DO pin) — so Cilium LB-IPAM assigns it the FIRST FREE address of a # matching pool block. Had GATEWAY_VIP stayed outside every pool (the original .21-.30 # proposal), the Gateway would have come up on .21 while split-horizon coredns-custom kept # answering *.home.keithjackson.dev with .20 — an address nothing announces — blackholing # in-cluster hairpin traffic (OIDC clients -> Keycloak/Grafana) while looking healthy. # KEEP GATEWAY_VIP == CILIUM_LB_POOL_START when changing either. # The control-plane VIP (.9) is deliberately OUTSIDE this pool: kube-vip floats it, and # LB-IPAM must never be able to hand it to a Service. - op: replace path: /data/GATEWAY_VIP value: "192.168.2.20" # Cilium LB IP pool, operator-supplied: 192.168.2.20-.30 (START == GATEWAY_VIP, see above). # Mirror this range into the prod tofu reservation (cilium_l2_vip_range) if prod ever gets one. - op: replace path: /data/CILIUM_LB_POOL_START value: "192.168.2.20" - op: replace path: /data/CILIUM_LB_POOL_STOP value: "192.168.2.30" # k3s API endpoint Cilium's kube-proxy replacement dials. This is the prod CONTROL-PLANE VIP # (kube-vip), NOT .20 (the ingress VIP) and NOT the single node address .10. # # *** DECIDED 2026-09-06 by the operator: the VIP, as intended. *** (This block previously # read "Not yet decided" while shipping .10 — the founder's own address — and that is what # the LIVE prod cluster is running: all three cilium-agents carry # KUBERNETES_SERVICE_HOST=192.168.2.10 while kube-vip holds .9, verified 2026-09-06. On a # 3-node HA cluster that makes the CNI's control-plane path depend on k3s-prod-1: lose it and # the datapath keeps running off Cilium's cache while no service or policy update lands, with # nothing looking broken. The VIP is a virtual address that reaches the cluster wherever the # control plane is, which is the whole reason kube-vip exists here.) # # It is owned Ansible-side by **k3s_apiserver_endpoint** (group_vars/all.yml, derived from # this environment's kube_vip_address = 192.168.2.9 in group_vars/k3s_prod/vars.yml) and MUST # equal that value — [parity] contract in platform/cilium/helmrelease.yaml. NOT k3s_api_host: # that variable still exists and still holds the FOUNDER's .10, because the cold-bootstrap # `--server` join needs a host that is up before any VIP is floated. Two addresses, two # variables, deliberately. # # .9 sits OUTSIDE the Cilium LB pool (.20-.30) and is not a node address, so kube-vip and # LB-IPAM can never be handed the same IP — asserted by # conformance/tests/l0-static/ha-control-plane-as-code. - op: replace path: /data/K3S_API_HOST value: "192.168.2.9" # --- still UNDECIDED -> loud placeholders (must be real before `flux bootstrap`) ------------ # Pinned L4 VIPs, both carved out of the pool by lbipam.cilium.io/ips. Not yet assigned for # prod. Leaving the dev literals in place would hand a prod LoadBalancer a dev-LAN address, so # they stay placeholders; whatever is chosen MUST fall inside [POOL_START, POOL_STOP] and not # collide with the Gateway VIP or the node IPs (.10/.11/.12). - op: replace path: /data/UNIFI_L4_VIP value: "192.168.2.21" - op: replace path: /data/CRAFTY_L4_VIP value: "192.168.2.22" # PiHole's DNS listener VIP — udp/53 + tcp/53 for LAN clients (apps/wave-0/pihole-dns.yaml). # 192.168.2.23 is the NEXT FREE address in the prod pool 192.168.2.20-.30, verified live # 2026-09-14: the CiliumLoadBalancerIPPool `default-pool` reported IPsTotal 11 / IPsUsed 3 / # IPsAvailable 8, and the only three allocations are .20 (cilium-gateway-shared), .21 # (unifi-l4-tcp/-udp) and .22 (crafty-l4-tcp/-udp). ARP-probed from a LAN host in the same # broadcast domain (arping 192.168.2.23, 3 probes, 0 responses), so nothing off-cluster holds # it either, and it is inside the pool block the operator already asserted sits outside the # SOHO router's DHCP scope. - op: replace path: /data/PIHOLE_DNS_VIP value: "192.168.2.23" # Plex's remote-access listener VIP — raw tcp/32400 for REMOTE Plex clients # (apps/wave-0/plex-remote.yaml). 192.168.2.24 is the NEXT FREE address in the prod pool # 192.168.2.20-.30, verified live 2026-10-03: the CiliumLoadBalancerIPPool `default-pool` # reported IPsTotal 11 / IPsUsed 4 / IPsAvailable 7, and the only four allocations are .20 # (cilium-gateway-shared), .21 (unifi-l4-tcp/-udp), .22 (crafty-l4-tcp/-udp) and .23 # (pihole-dns-tcp/-udp). It is inside the pool block the operator already asserted sits outside # the SOHO router's DHCP scope, and it is not a node address (.10/.11/.12) or the # control-plane VIP (.9). # # UNLIKE .23, THIS ONE IS ALSO A WAN PORT-FORWARD TARGET. The router forwards WAN tcp/32400 # here, by hand, off-repo — so changing this value is a two-place edit and the router is the # half that CI cannot check. runbooks/plex-remote-access.md is the procedure. # # NOT ARP-PROBED. .23 was verified free with `arping` from a LAN host before it was claimed; # for .24 only the in-cluster pool accounting above was checked, so an off-cluster squatter on # this address would not have been caught. Probe before trusting it if remote access comes up # with a conflict. - op: replace path: /data/PLEX_L4_VIP value: "192.168.2.24" # The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns — # NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path. # It scopes the ingress allow for PiHole's DNS listener to LAN clients only. - op: replace path: /data/LAN_CIDR value: "192.168.2.0/24" # ------------------------------------------------------------------------------------------ # BULK (EXTERNAL NFS) STORAGE HOST — the TrueNAS at 192.168.2.30. # ------------------------------------------------------------------------------------------ # Backs every `class: bulk` volume: the 8.33 T media library (Storage/plex — plex, sonarr, # radarr, qbittorrent) and the 270 G legacy Nextcloud archive (Storage/nextcloud). # # THIS WAS THE REPO'S ONE LOAD-BEARING PLACEHOLDER, AND THE REASON IT COULD BE IS NOW GONE. # The datasets lived on a Proxmox host with a single interface on pfSense's LAN # (192.168.1.9), which prod could reach only through a DNAT at 192.168.2.102 — so the correct # value here depended on an undecided operator choice, and guessing it would have authored a # cluster that mounts nothing. Both pools were physically moved onto a standalone TrueNAS at # 192.168.2.30, prod's OWN /24. No crossing to arrange, no candidates to choose between. # Measured 2026-10-07: 192.168.2.102:2049 no longer answers; 192.168.2.30:2049 and :111 do. # # ONE PORT IS STILL ALL IT NEEDS: tcp/2049. The PVs mount nfsvers=4.1 and NFSv4 carries mount # and lock in-protocol — no rpcbind/111, no mountd, no statd. The NAS advertises # `+3 +4 +4.1 +4.2` (/proc/fs/nfsd/versions, measured), so 4.1 negotiates. - op: replace path: /data/BULK_NFS_SERVER # VERIFIED ON THE NAS 2026-10-07, before any cluster touched it (`exportfs -v`): # /mnt/Storage/plex 192.168.2.10/.11/.12 rw,sec=sys,secure,root_squash,no_all_squash # /mnt/Storage/nextcloud 192.168.2.10/.11/.12 ro,sec=sys,secure,root_squash,no_all_squash # # no_all_squash is the flag the entire permission model rests on: client uids pass through # untouched, so the arrs' PUID=1000 arrives as the owner of a 0755 1000:3000 tree. sec=sys # keeps that numeric rather than principal-mapped. The export deliberately does NOT carry # no_root_squash, which the Proxmox one did: nothing here needs it, because every write to # the tree happens after the LSIO drop to uid 1000 and the images only chown their Longhorn # /config. Both subPath directories (plex/downloads, nextcloud/nextcloud) already exist at # 0755 1000:3000, so a squashed root can still traverse them for subPath resolution. # # The ro= posture on the archive is now SERVER-enforced, which the old export also managed. # The media tree's read-only half (plex) stays a CLIENT-side guarantee — all four media pods # can land on any node, so the server sees the same three addresses for reader and writers # alike and cannot tell them apart. Unchanged by the move; see docs/BULK-NFS-STORAGE.md §5. # # *** CHANGING THIS VALUE REQUIRES DELETING AND RECREATING THE PVs. *** A bound PV's # spec.nfs is immutable — measured by server dry-run 2026-10-07: # spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation # so Flux's apply FAILS rather than updating and the pods keep their old, dead mount. The # operator sequence is docs/BULK-NFS-STORAGE.md §10. Retain + the renderer's pre-set # claimRef make it safe: no data is touched and each recreated PV rebinds its own PVC. value: "192.168.2.30" # ========================================================================================== # OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05; bucket + token 2026-09-07). # ========================================================================================== # Dev's versitygw/MinIO (192.168.1.9:9000) sits on the DEV side of the network and is NOT an # option for prod, for two independent reasons: # 1. UNREACHABLE. Prod is on the SOHO network 192.168.2.0/24 — pfSense's WAN — while that # MinIO is inside pfSense's LAN. Measured from a pod INSIDE prod (the workstation proves # nothing about pod egress): `curl http://192.168.1.9:9000/minio/health/live` → connection # timed out after 15s. Reaching it would need a WAN→LAN rule that does not exist. # 2. NOT OFF-CLUSTER DR. It runs on the same Proxmox host as dev's VMs, so it already shares # a physical failure domain with dev. R2 is genuinely off-site, and its free egress means # restore REHEARSALS cost nothing — R-STATE-03 has no financial disincentive to stay proven. # # THIS UNBLOCKS AN ACTIVE INCIDENT, not just a roadmap item. All three prod CNPG Clusters ship # `archive_mode=on` with `endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`, so barman-cloud-wal-archive # exits 4 on every segment and Postgres — correctly — REFUSES TO RECYCLE WAL. Verified live # 2026-09-07: keycloak-db / nextcloud-db / semaphore-db all report # ContinuousArchiving=False/ContinuousArchivingFailing, and keycloak-db's 5 GiB PVC is at # 579 MiB used / 4360 MiB free and climbing monotonically. A disk-full PANIC there takes the # cluster's SSO database down, and with it kubectl OIDC, Grafana, Headlamp, Gitea and # forward-auth. Fixing the endpoint lets Postgres drain the whole backlog. # # ENDPOINT IS ACCOUNT-SCOPED, so it stays a prod-overlay value even though it is not a LAN # address: this is the operator's Cloudflare account id, confirmed from inside a prod pod # (`head-bucket` on k3s-prod-backup → BucketRegion WNAM; PUT and DELETE both succeed). # It is NOT put in base, because base is dev's live MinIO and dev is deliberately STAYING # there — moving dev to R2 would reset its PITR window and orphan the backups its PROVEN # restore depends on. Two environments, two targets, isolated by URL *and* by bucket. - op: replace path: /data/S3_BACKUP_ENDPOINT value: "https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com" # R2 requires the region string to be exactly `auto`. (`head-bucket` answers # `{"BucketRegion":"WNAM"}` — that is a PHYSICAL LOCATION HINT, not a signing region; using it # here produces AuthorizationHeaderMalformed.) Dev keeps the base `us-east-1` its live MinIO # has always been signed with. - op: replace path: /data/S3_BACKUP_REGION value: "auto" # ------------------------------------------------------------------------------------------ # OBJECT TENANCY INSIDE THE ONE BUCKET. The operator created a SINGLE R2 bucket, so velero and # barman cannot be separated by bucket the way they are on dev's MinIO (`velero` and `cnpg`). # They are separated by disjoint TOP-LEVEL PREFIXES instead: # # s3://k3s-prod-backup/velero/… Velero: kopia repo + backup metadata # s3://k3s-prod-backup/barman//… CNPG: WAL + base backups # # The bucket ROOT is left empty on purpose — a misrendered prefix then lands somewhere # obviously wrong instead of silently colliding with a live tenant. Velero's retention TTL # deletes objects under its own prefix only, so it can never reap a WAL segment; and barman's # per-cluster serverName subdirectory keeps the three Postgres servers' chains apart, which # matters because interleaving two servers' WAL into one prefix corrupts the chain outright. # Dev cannot be reached from any of these: different endpoint AND different bucket names. # ------------------------------------------------------------------------------------------ - op: replace path: /data/VELERO_BACKUP_BUCKET value: "k3s-prod-backup" - op: replace path: /data/VELERO_BACKUP_PREFIX value: "velero" - op: replace path: /data/CNPG_BACKUP_PATH value: "s3://k3s-prod-backup/barman" # ------------------------------------------------------------------------------------------ # WHICH CREDENTIAL SIGNS THE REQUESTS TO THAT BUCKET (per-env credential split, 2026-09-07). # ------------------------------------------------------------------------------------------ # The knobs above moved the ENDPOINT and the OBJECT LAYOUT per environment. The credential did # not move with them, and it could not: the four backup-credential Secrets are SHARED BASE # manifests, applied to both clusters by the same Flux tiers. Writing the R2 token into them # gave prod a working key by TAKING DEV'S AWAY — dev would have signed MinIO requests with an # R2 key id, and under archive_mode=on that is a filling pgdata PVC, not a missing backup. # # So each credential is now two objects, and this suffix selects prod's: # velero-cloud-credentials-prod the R2 token in the AWS INI `cloud` shape # -backup-credentials-prod the same token as ACCESS_KEY_ID / ACCESS_SECRET_KEY # for keycloak-db, nextcloud-db and semaphore-db. Both halves ship to both clusters (they are # base resources); ONLY this value decides which one each consumer reads. It is a SUFFIX rather # than four secret-name knobs deliberately — four could be updated three-quarters of the way, # leaving one consumer on dev's MinIO key with nothing rendering wrong, and strict-mode # substitution cannot catch a defined-but-stale variable. # # THE R2 TOKEN IS THE ONE PROVEN FROM INSIDE PROD (chore/r2-credentials 0a6a862): head-bucket on # k3s-prod-backup returns BucketRegion WNAM, and PUT and DELETE both succeed. Rotating it now # touches ONLY the four `-prod` files; dev's MinIO keys are untouched objects. That isolation is # the structural reason for the split, independent of dev's eventual decommission. - op: replace path: /data/BACKUP_CREDENTIALS_SUFFIX value: "-prod" # TODO(WS-INFRA, Phase 7): pin prod node placement once the bare-metal role labels exist.