jakman85andClaude Opus 5
422e1dc7e8
OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach
the cluster and provide fault tolerant HA. Without a change it will not work in HA."
THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live
before changing anything: all three prod cilium-agent pods carried
KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip
VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the
CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said
the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated
that stale comment as an open question when the evidence already settled it.
SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs:
k3s_api_host unchanged — the FOUNDER's IP: the cold-bootstrap join
target and a tls-san. infra/tofu/outputs.tf GENERATES this
name, so renaming would ripple into tofu and the runbooks.
k3s_apiserver_endpoint NEW = {{ kube_vip_address }} — the HA address; what Cilium
consumes. Derived, not restated, so an environment overrides
ONE line and the VIP, the SAN and the CNI endpoint move
together. Prod needed no new override.
THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at
the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran
node-baseline -> k3s -> cilium, so Cilium would install against an address nothing
announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value:
site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier
(kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there
because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing
neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so
a cold run stops THERE with a named cause instead of surfacing as unreachable agents.
Also fixed a pre-existing bug that only mattered once this play joined site.yml: the
role's read-only interface probe ran without `check_mode: false`, so under --check it
was skipped and the assert fired with a misleading "set kube_vip_interface explicitly".
VERIFIED BY ME, not just reported:
renders dev K3S_API_HOST=192.168.1.49 prod=192.168.2.9 (each environment's VIP)
cilium template consumes k3s_apiserver_endpoint, not k3s_api_host
cold join still `--server https://{{ k3s_api_host }}:6443` — the founder
order site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux
gate full CI gate PASS
Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing
k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders
['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved
— .9 already answers under full CA verification; and --check --diff of kube-vip.yml
against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the
kube-vip prod already runs.
ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine
checks: the Cilium template must use the endpoint var; the endpoint must derive from
kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal
that environment's VIP and be none of its node IPs (the [parity] contract, now
machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium.
Every new assertion mutation-tested and bites. One honest negative recorded: reverting
only the BASE default is not caught, because both overlays patch the key — the check
reads effective per-environment values, so the base default is unguarded.
APPLY ORDER, and this is the operator's step — nothing was applied:
PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let
Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet
ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01
incident), it will not proceed past an unready pod, existing datapath survives an
agent restart, and .9 is already live before the change lands.
DEV SEPARATELY, operator present. Same mechanics but live dev agents dial
192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that.
ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a
cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook.
NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live
KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that
test's gap list.
Refs R-PLAT-01, R-NET-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
287 lines
14 KiB
YAML
287 lines
14 KiB
YAML
---
|
|
# clusters/dev — the ACTIVE build target overlay.
|
|
#
|
|
# `flux bootstrap ... --path=clusters/dev` points the dev cluster's Flux here. It reuses the base
|
|
# reconciliation graph and patches the shared cluster-vars ConfigMap with dev-scoped values (R-ENV-02).
|
|
# It also adds a self-managing ROOT Kustomization (sync.yaml) so the whole graph reconciles from git
|
|
# on every push — no more manual `kubectl apply -k` for cluster-vars/overlay changes (F-OPS1).
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
resources:
|
|
- ../base
|
|
- sync.yaml # self-managing root Kustomization → ./clusters/dev
|
|
patches:
|
|
# --- SOPS decryption + bootstrap source (PM-inlined from bootstrap/flux/patches/, 2026-08-16) ---
|
|
# Inlined (not cross-dir file refs) to stay within kustomize's default load restrictions.
|
|
# Realises the commented decryption TODOs in clusters/base and sets the real source URL.
|
|
# The `sops-age` Secret is created out-of-band (bootstrap/sops/install-age-key.sh) — name only.
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: platform
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
# SOPS decryption on EACH app wave (the app tier is now apps-wave-0..N, not a single `apps`
|
|
# Kustomization). Same stanza the retired `apps` Kustomization received — one per wave, by
|
|
# name. Add a patch here whenever `render.py --emit-waves` introduces a new deepest wave
|
|
# (DESIGN-waves.md cutover note). The `sops-age` Secret is created out-of-band (name only).
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: apps-wave-0
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: apps-wave-1
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: apps-wave-2
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
# oauth2-proxy moved to its own post-platform Flux Kustomization (clusters/base/oauth2-proxy.yaml).
|
|
# It carries oauth2-proxy-credentials.sops.yaml, previously decrypted via the `platform`
|
|
# Kustomization; now that this tier owns the encrypted Secret it needs its own decryption stanza.
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: oauth2-proxy
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
# headlamp moved to its own post-platform Flux Kustomization (clusters/base/headlamp.yaml).
|
|
# It carries headlamp-oidc-credentials.sops.yaml, previously decrypted via the `platform`
|
|
# Kustomization; now that this tier owns the encrypted Secret it needs its own decryption stanza.
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: headlamp
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
# crafty has its own post-platform Flux Kustomization (clusters/base/crafty.yaml). It carries
|
|
# crafty-oauth2-proxy-credentials.sops.yaml, so it needs its own SOPS decryption stanza (same as
|
|
# headlamp/oauth2-proxy). The `sops-age` Secret is created out-of-band (name only).
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: crafty
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
# velero (backup controller) has its own post-platform Flux Kustomization (clusters/base/backup.yaml).
|
|
# It carries velero-cloud.sops.yaml (the scoped MinIO `cloud` credential), so it needs its own SOPS
|
|
# decryption stanza (same as headlamp/oauth2-proxy/crafty). The `sops-age` Secret is out-of-band (name only).
|
|
- patch: |-
|
|
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
|
kind: Kustomization
|
|
metadata:
|
|
name: velero
|
|
namespace: flux-system
|
|
spec:
|
|
decryption:
|
|
provider: sops
|
|
secretRef:
|
|
name: sops-age
|
|
- patch: |-
|
|
apiVersion: source.toolkit.fluxcd.io/v1
|
|
kind: GitRepository
|
|
metadata:
|
|
name: flux-system
|
|
namespace: flux-system
|
|
spec:
|
|
# Phase-6 cutover done: source is now the in-cluster Gitea; bootstrap host-Gitea retained
|
|
# stopped as recovery. PUBLIC repo => Flux needs no creds; Flux resolves the host in-cluster
|
|
# via coredns -> Gateway VIP, and the LE-prod wildcard cert is publicly trusted so NO
|
|
# secretRef/caCert is needed. Recovery: restart bootstrap Gitea (192.168.2.103:3000) and
|
|
# revert this url (docs/05).
|
|
url: https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
|
|
ref:
|
|
branch: main
|
|
# --- dev-scoped cluster-vars (R-ENV-02) ---
|
|
- target:
|
|
kind: ConfigMap
|
|
name: cluster-vars
|
|
patch: |-
|
|
- op: replace
|
|
path: /data/CLUSTER_ENV
|
|
value: "dev"
|
|
# DNS-01 via LE PRODUCTION: in-cluster OIDC clients (oauth2-proxy, Grafana) must trust the
|
|
# wildcard cert for SSO to work; the staging root isn't in any trust store. Prod issuer is
|
|
# registered + Ready; one wildcard cert is well within LE rate limits — ADR-0006, R-TLS-01/03.
|
|
- op: replace
|
|
path: /data/ACME_CLUSTER_ISSUER
|
|
value: "letsencrypt-prod"
|
|
# --- DNS identity (R-DNS-01/02/03) -----------------------------------------------------
|
|
# DEV STAYS ON THE APEX. Restated explicitly even though it equals the base default, exactly
|
|
# like the LAN block and LONGHORN_DATA_PATH below: this is what the LIVE cluster is running.
|
|
# Every one of dev's public DNS records, its `*.home.keithjackson.dev` wildcard cert, its
|
|
# Keycloak issuer, every OIDC client redirect URI, kubectl SSO, and gitea.home.keithjackson.dev
|
|
# (which IS this repo's GitOps source) are all on this exact suffix RIGHT NOW. Prod moved to
|
|
# prod.home.keithjackson.dev instead (2026-09-04) — that alone ended the collision, so dev did
|
|
# not have to move. Relocating dev to dev.home.keithjackson.dev is phase 2 and is a migration,
|
|
# not a variable change: infra/PROD-PROVISIONING.md §10.
|
|
- op: replace
|
|
path: /data/BASE_DOMAIN
|
|
value: "home.keithjackson.dev"
|
|
# ExternalDNS scope. MUST stay the zone apex: dev serves the home.keithjackson.dev APEX itself,
|
|
# and the Cloudflare provider selects zones with this value (a strict-subdomain filter dropped
|
|
# the parent zone — DECISIONS-LOG 2026-08). Same value the retired DNS_ZONE knob held.
|
|
- op: replace
|
|
path: /data/EXTERNALDNS_DOMAIN_FILTER
|
|
value: "keithjackson.dev"
|
|
# DO NOT CHANGE THIS VALUE. Dev's LIVE public records are registered in Cloudflare's TXT
|
|
# registry under owner `default` (the external-dns chart default, which is what both clusters
|
|
# silently shared until the 2026-09-04 incident). Dev's external-dns identifies its own records
|
|
# BY this string: any other value and it stops recognising every record it manages, then
|
|
# re-creates duplicates beside the orphaned originals. Prod is `prod` — distinct, which is the
|
|
# whole point. The base default is a non-colliding sentinel, so this line is load-bearing.
|
|
- op: replace
|
|
path: /data/EXTERNALDNS_TXT_OWNER_ID
|
|
value: "default"
|
|
# --- dev LAN addressing (192.168.1.0/24 on the Proxmox LAN) -----------------------------
|
|
# EVERY knob in cluster-vars' "LAN addressing" block is restated here EXPLICITLY, even where it
|
|
# equals the base default. These values are what the LIVE dev cluster is running right now, so
|
|
# the substituted output is byte-identical to the pre-parameterization hardcoded manifests
|
|
# (verified 2026-08-31). Never let this block drift from the live cluster.
|
|
#
|
|
# Split-horizon DNS target: the Cilium-L2 VIP of the `shared` Gateway on the Proxmox LAN
|
|
# (= CILIUM_LB_POOL_START, the pool's first address). coredns-custom answers
|
|
# *.home.keithjackson.dev with this so in-cluster OIDC clients reach Keycloak via the Gateway.
|
|
- op: replace
|
|
path: /data/GATEWAY_VIP
|
|
value: "192.168.1.53"
|
|
# k3s API endpoint Cilium's kube-proxy replacement dials — DEV'S CONTROL-PLANE VIP,
|
|
# the address kube-vip floats, NOT the primary node .50.
|
|
# [parity] equals the Ansible inventory's k3s_apiserver_endpoint for dev
|
|
# (= kube_vip_address, group_vars/all.yml). See clusters/base/cluster-vars.yaml for why.
|
|
#
|
|
# *** THE ONE EXCEPTION TO "NEVER LET THIS BLOCK DRIFT FROM THE LIVE CLUSTER" ***
|
|
# Every other value in this block is what dev is running right now. This one is not:
|
|
# the live dev cilium-agents dial 192.168.1.50 (k3s-1). Changing it here is a
|
|
# DELIBERATE, OPERATOR-APPROVED move (2026-09-06) and reconciling it ROLLS THE CILIUM
|
|
# AGENT DAEMONSET on dev — one node at a time (updateStrategy.maxUnavailable: 1, kept
|
|
# in parity for exactly this reason). Do not push this to dev-live casually, and not
|
|
# while dev's DNS migration is mid-flight; the L2 datapath survives an agent restart
|
|
# but a wrong value here is the 2026-09-01 incident. .49 is in dev's serving-cert SANs
|
|
# already (k3s_tls_sans) and dev's own kubeconfig validates against it via
|
|
# tls-server-name, so TLS is not the risk — the risk is reachability, and .49 is on the
|
|
# same LAN as the nodes and ARP-announced by kube-vip on every server.
|
|
- op: replace
|
|
path: /data/K3S_API_HOST
|
|
value: "192.168.1.49"
|
|
# Cilium LB IP pool = the tofu reservation cilium_l2_vip_range .53-.60, outside the node IPs
|
|
# (.50/.51/.52) and outside the DHCP scope. .53 Gateway, .54 unifi, .55 crafty; .56-.60 free.
|
|
- op: replace
|
|
path: /data/CILIUM_LB_POOL_START
|
|
value: "192.168.1.53"
|
|
- op: replace
|
|
path: /data/CILIUM_LB_POOL_STOP
|
|
value: "192.168.1.60"
|
|
# Pinned L4 VIPs out of that pool (lbipam.cilium.io/ips). Changing either MOVES a live device-
|
|
# facing endpoint: unifi is the inform/STUN target every adopted UniFi device holds.
|
|
- op: replace
|
|
path: /data/UNIFI_L4_VIP
|
|
value: "192.168.1.54"
|
|
- op: replace
|
|
path: /data/CRAFTY_L4_VIP
|
|
value: "192.168.1.55"
|
|
# Off-cluster backup target: the on-Proxmox MinIO (plain HTTP, path-style) that every CNPG
|
|
# barmanObjectStore and Velero's backupStorageLocation write to.
|
|
- op: replace
|
|
path: /data/S3_BACKUP_ENDPOINT
|
|
value: "http://192.168.1.9:9000"
|
|
# dev mirrors prod's 3-node topology, so the replicated tier keeps 3 replicas.
|
|
- op: replace
|
|
path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
|
|
value: "3"
|
|
# DO NOT CHANGE THIS VALUE. The LIVE dev cluster has 18 Longhorn volumes whose replicas are
|
|
# on disk under /var/lib/longhorn — the dedicated /dev/vdb ext4 disk roles/node_baseline
|
|
# mounts there. Longhorn addresses a replica by dataPath; pointing defaultDataPath anywhere
|
|
# else would orphan every one of them (they are not recoverable from the old path by
|
|
# rescheduling). Restated explicitly even though it equals the base default, exactly like the
|
|
# LAN block above: this is what the live cluster is running. Prod uses /data/longhorn because
|
|
# its data partition also carries k3s's data dir — see clusters/prod and PROD-PROVISIONING §2.
|
|
- op: replace
|
|
path: /data/LONGHORN_DATA_PATH
|
|
value: "/var/lib/longhorn"
|
|
# smaller retention / volumes to keep the dev workstation footprint light.
|
|
- op: replace
|
|
path: /data/MONITORING_PROMETHEUS_RETENTION
|
|
value: "3d"
|
|
# 5Gi was too tight: 3d of this cluster's series volume (~1.2Gi/day of blocks) plus a WAL that
|
|
# ballooned past 1Gi overran the disk, wedging WAL commits and firing every absent()/==0 alert
|
|
# falsely. Bumped to 8Gi (still lighter than base 10Gi) so the size-cap below has real headroom.
|
|
- op: replace
|
|
path: /data/MONITORING_PROMETHEUS_PVC_SIZE
|
|
value: "8Gi"
|
|
# Size-cap kept well below the 8Gi PVC (headroom for WAL + head + compaction scratch) so
|
|
# Prometheus prunes oldest blocks by size and never fills the disk on this young cluster.
|
|
- op: replace
|
|
path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE
|
|
value: "5GiB"
|
|
- op: replace
|
|
path: /data/MONITORING_LOKI_PVC_SIZE
|
|
value: "5Gi"
|
|
# single-replica control-plane addons in dev (no HA needed to iterate).
|
|
- op: replace
|
|
path: /data/KEYCLOAK_REPLICAS
|
|
value: "1"
|
|
- op: replace
|
|
path: /data/OAUTH2_PROXY_REPLICAS
|
|
value: "1"
|
|
- op: replace
|
|
path: /data/VELERO_RETENTION
|
|
value: "72h"
|
|
# dev-softened Kyverno for cold-start (base defaults Enforce/Fail/3 = prod posture).
|
|
# Not credential-driven — bootstrap-safety knobs so a cold cluster comes up before
|
|
# admission is proven/HA; flip dev to the strict posture once green.
|
|
- op: replace
|
|
path: /data/KYVERNO_VALIDATION_ACTION
|
|
value: "Audit"
|
|
- op: replace
|
|
path: /data/KYVERNO_FORCE_FAILURE_POLICY_IGNORE
|
|
value: "true"
|
|
# 2 (not base-default 1): the webhook is fail-closed, so a single-pod restart 502s every
|
|
# admission-gated write cluster-wide. Two replicas give a fail-closed webhook availability
|
|
# headroom during a restart/upgrade while staying dev-modest (prod runs 3).
|
|
- op: replace
|
|
path: /data/KYVERNO_ADMISSION_REPLICAS
|
|
value: "2"
|