R-ENV-02 promises dev and prod differ ONLY by scoped cluster-vars overrides. In fact
dev LAN addresses were hardcoded into base manifests, so clusters/prod would have come
up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN (192.168.2.x).
New cluster-vars knobs (base default = dev literal, per the file's existing convention):
K3S_API_HOST platform/cilium/helmrelease.yaml (+ the values.yaml mirror)
CILIUM_LB_POOL_START platform-config/cilium/l2-ip-pool.yaml
CILIUM_LB_POOL_STOP "
UNIFI_L4_VIP apps/vms/unifi/50-service-l4.yaml
CRAFTY_L4_VIP platform/crafty/service-l4.yaml
S3_BACKUP_ENDPOINT platform/keycloak/postgres-cluster.yaml, apps/wave-0/{nextcloud,
semaphore}-db.yaml, and platform/velero/helmrelease.yaml
Also fixes GATEWAY_VIP's base default, which was still the RETIRED libvirt address
10.73.0.240, and adds the GATEWAY_VIP patch that clusters/prod was missing entirely.
clusters/base/apps-vms.yaml gains postBuild.substituteFrom (it was the one consuming
Kustomization without it, so a ${TOKEN} there would have shipped as a literal string).
The envsubst contract is documented there: ${VAR} substitutes and escapes as $${VAR};
$VAR and $(cmd) are left alone; ${ARR[0]} hard-errors. The three UOS_URL expansions in
the unifi cloud-init are escaped accordingly.
PROD ADDRESS PLAN (operator, 2026-08-31):
nodes .10/.11/.12 · control-plane VIP .9 (Ansible-side) · GATEWAY_VIP .20
Cilium LB pool .20-.30
CONTAINMENT INVARIANT, now documented at the pool and in the prod overlay: GATEWAY_VIP
MUST equal CILIUM_LB_POOL_START. The shared Gateway pins no address, so Cilium LB-IPAM
gives it the first free address of a matching pool block. An initial prod proposal put
.20 outside a .21-.30 pool; the Gateway would have come up on .21 while split-horizon
coredns-custom kept answering *.home.keithjackson.dev with .20 — an address nothing
announces — blackholing in-cluster OIDC hairpin traffic while reporting healthy. The
pool was widened to .20-.30. The control-plane VIP (.9) stays outside the pool so
kube-vip and LB-IPAM can never contend.
Values prod still cannot know stay INVALID `TBD-PROD-*` placeholders on purpose: base
defaults are dev values, so an omitted patch means prod silently inherits a dev address
— far worse than a loud failed apply.
DEV IS BYTE-IDENTICAL. Verified by kustomize build over 13 paths before/after, then
flux envsubst with the dev cluster-vars values applied to both sides: zero drift. The
only deltas are the intended additive keys in the overlay ConfigMap itself. No
unresolved ${TOKEN} survives. render-golden green (10 goldens); yamllint clean.
Serves: R-ENV-02. Task: T-7.1 (prerequisite). Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Text-only cleanup after the upgrade campaign. Updates stale version
strings, prose, and comments to the verified current facts; no
Kubernetes behavior or version pins changed except the kubectl
toolchain bump (below).
Verified facts applied: k3s v1.36.4+k3s1, Debian 13 trixie,
Cilium 1.20.1 / Gateway API v1.6.1, Longhorn 1.12.1, node RAM 24 GiB,
dedicated virtio2 SSD etcd disk per node, Velero chart 12.x / v1.18,
Grafana 13.2.0.
- Version strings across infra/ and platform/ READMEs + comments
- kubectl toolchain 1.31.4 -> 1.36.4 (devcontainer/ci/conformance
versions.env; 1.31 vs a 1.36 cluster exceeded the +-1 skew)
- Retired-single-apps-Kustomization refs -> apps-wave-0/1/2 + apps-vms
with per-wave SOPS decryption (flux/clusters docs + bootstrap.sh)
- COMPLETED/historical banners on the three executed plan docs
- Resolved TODO(WS-SEC) markers -> Renovate-maintained ranges
- Appended dated DECISIONS-LOG entry (2026-08-30) superseding the
2026-08-26 LIVE STATE line
Left as revalidation items (not blind-edited): external-snapshotter
v8.2.0 vs 1.36, Cilium ext_authz/forward-auth relevance under 1.20.1,
conformance ImageVolume feature-gate assertions under 1.36.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Supersedes the previous kubectl-image repoint. The velero chart's pre-install
`velero-upgrade-crds` HOOK job runs docker.io/bitnami/kubectl, whose public
Docker Hub catalog Bitnami removed (Aug 2025); `bitnami/kubectl:1.31` 404s,
leaving the hook in Init:ImagePullBackOff so Velero never installs.
Repointing the hook to the exact-version rancher/kubectl:v1.31.5 (prior commit)
does not work: that image is distroless, and the hook's init runs
`/bin/sh -c "cp $(which sh)/... $(which kubectl) /tmp"` — with no /bin/sh it
crashes StartError (verified on-cluster).
The hook exists ONLY to UPDATE CRDs on chart upgrades — it is NOT how CRDs are
first installed. The chart ships all 13 Velero CRDs in its native `crds/`
directory, which Helm applies unconditionally on install (before hooks),
independent of this flag; the CRDs are already present on-cluster from Helm's
crds/ handling. Setting `upgradeCRDs: false` drops the fragile external
kubectl-image dependency entirely while CRDs still install. Trade-off: Helm
won't manage crds/ after install, so CRD schema upgrades become a
manual/Renovate step; acceptable given Velero's CRDs are stable.
All other Velero values (BSL -> versitygw s3://velero, kopia FSB node-agent, the
two Schedules, the velero-cloud-credentials secret) are unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The velero chart's pre-install `velero-upgrade-crds` hook defaults its kubectl
image to docker.io/bitnami/kubectl, whose public Docker Hub catalog Bitnami
removed (Aug 2025). `docker.io/bitnami/kubectl:1.31` now 404s, leaving the hook
job in Init:ImagePullBackOff so Velero never installs (no Deployment,
node-agent, or BackupStorageLocation).
Set kubectl.image to the Rancher-maintained mirror docker.io/rancher/kubectl,
pinned to the cluster's exact k8s minor v1.31.5 (multi-arch amd64/arm64,
verified pullable via the Docker Hub registry API). All other Velero values
(BSL → versitygw s3://velero, kopia FSB node-agent, the two Schedules, the
velero-cloud-credentials secret) are unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
jakman85andClaude Opus 5