Steady-state CPU PSI 15-23% + 15-min load 5.6-8.0 on 6 cores (run queue at/over core count) left no
headroom for bursty workloads. Proxmox host has 24 cores at ~4% load, so 3x8 = 24 vCPU is 1:1 with
zero oversubscription. In-place tofu update (cores change is pending until a per-node VM stop/start);
rolled one node at a time to keep etcd quorum. Comment also corrected 16->24 GiB (stale).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
All 3 k3s nodes re-provisioned to Debian 13 (kernel 6.12) via immutable tofu -replace,
one at a time, leader/primary last. variables.tf image defaults -> trixie generic.
k3s_server role: ensure /etc/rancher/k3s exists (fresh-node re-provision fix).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Interim bridge to stop Longhorn replica-rebuild I/O storms on the shared HDD
Storage pool from starving etcd WAL fsync (crash-looping k3s / flapping the
kube-vip VIP). Adds a small dedicated SSD disk per node for etcd only; Longhorn
stays on HDD. See docs/plans/ssd-etcd-isolation.md.
tofu (infra/tofu):
- New vars etcd_ssd_datastore (default "local-zfs" = rpool/data, Micron 5100
SATA SSD mirror w/ PLP) and etcd_disk_gib (default 16).
- New virtio2 disk block on the SSD datastore per VM: discard=on, cache=none
(honor guest fsync straight to the ZFS ZIL — safe on PLP SSD), iothread=false.
Kept last (ascending interface) so bpg does not reorder virtio0/virtio1.
- Revert the debian_cloud_image resource to the long type name
proxmox_virtual_environment_download_file to match the R2 state: bpg >= 0.111
dropped the cross-type moved{} linkage to the short proxmox_download_file, so
the stale moved block failed EVERY plan with "Resource type mismatch" (broken
on main too). Aligning code to state yields a clean plan with no image
re-download. Short-name migration is deferred to a deliberate `tofu state mv`.
ansible (node_baseline):
- Format /dev/vdc (label k3s-etcd) and mount it at
/var/lib/rancher/k3s/server/db (defaults,noatime; no nofail so a missing etcd
disk fails loudly rather than silently landing etcd on the root HDD) before
k3s starts, so a rebuilt node comes up with etcd isolated on SSD. Data-safe:
filesystem module never reformats an existing fs.
Phase 1 only: non-disruptive. No tofu apply, no node/VM/etcd/Longhorn changes.
tofu plan verified: 0 to add, 3 to change, 0 to destroy — every VM an in-place
disk ADD (virtio2), dev-3 also an in-place memory 16384->24576; no VM replace.
Merge gate green (render==golden included).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Raise per-node RAM from 16 GiB to 24 GiB for proper failover headroom.
Matches the live rolling resize of the three k3s-dev VMs on Proxmox
(qm set --memory 24576 per node, one at a time, KSM enabled first).
Single memory_mib var applies to all three nodes via for_each, so a
future `tofu apply` would resize all three at once and break etcd
quorum — the resize MUST be done live/rolling, not via apply.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The docs had drifted badly behind ~85 commits of delivery. The root README still
said "Phase 0 — Foundations. No cluster exists yet"; the infra docs still
described a libvirt/KVM cluster that has been retired for two days; and
DECISIONS-LOG.md — the PM's authoritative record — stopped at 2026-08-25 AM.
Anyone following these docs would have been working from a false map.
Verified against the live cluster (3/3 nodes, 7/7 Flux Kustomizations Ready,
10/10 HelmReleases Ready) rather than inferred from commit messages.
README.md
- Phase 0 -> a per-phase status table: 0-6 delivered and deployed, 7 not open.
- Layout now shows the wave split, platform-config, apps/vms, infra, bootstrap.
- Real conformance number (36/51, 70.6%) and how to regenerate it.
- Devcontainer workflow; note that the gate was red and why.
infra/PHASE1-STATUS.md — rewritten. Was a `virsh`/libvirt resume guide for a
cluster that no longer exists. Now: the Proxmox substrate, the pfSense NAT
paths, both kubectl access paths (Keycloak OIDC normal / cert break-glass),
the toolchain, off-repo state that must not be lost, and the fact that Flux's
git source is an app inside the cluster (`git push origin main` IS the deploy).
infra/PHASE1-SCOPE.md — historical banner. The exit criteria and sizing
rationale are still worth keeping; the libvirt execution steps are not.
infra/README.md — Proxmox, plus a map of the subtree and where the cluster
actually lives (it is not on this workstation).
infra/devcontainer/README.md — the "run tofu and ansible on the HOST" limitation
is OBSOLETE: it was a dmacvicar/libvirt polkit/PID problem, and bpg/proxmox
talks to an HTTP API. Everything runs in the container now. Documents the full
CI-gate toolchain and what each bind mount is load-bearing for (the age keyring
and the read-only spec repo especially — without the latter, spec-immutability
silently skips instead of verifying).
infra/tofu/README.md — was "Authoring only — nothing here has been applied". It
is applied and live. Disk sizes corrected 40/100 -> 64/128 GiB (confirmed
against the running nodes), pfSense NAT marked done, and a warning that
terraform.tfstate is absent on a fresh workstation — an apply with empty state
would try to recreate the three live nodes.
infra/tofu/variables.tf — section comment disagreed with its own defaults.
infra/cluster.sh — retired-with-instructions header. It silently drives libvirt
domains that do not exist; behaviour unchanged, but it no longer lies.
apps/vms/unifi/README.md — was "NOT WIRED IN", 60Gi, and listed a file that has
since been dropped. Now: live status, 40Gi, resolved open questions marked
resolved, and the two constraints the next person will otherwise rediscover —
KubeVirt's 2048B inline userData cap, and that apps-vms must never carry
postBuild.substituteFrom (envsubst corrupts the cloud-init).
DECISIONS-LOG.md — backfilled the gap as one structured entry: the Proxmox
migration, NAT + kube-vip, the twice-proven oauth2-proxy cold-start deadlock,
the Flux source cutover, CI going live in-cluster, ten app onboardings and the
renderer bugs they exposed, the wave split (verified: DBs adopted, not
recreated — original ages intact, prune restored), KubeVirt storage, the UniFi
VM, and kubectl SSO. Plus verified live state and the carried-forward open list.
Two corrections to the record while verifying:
- kubectl OIDC is LIVE (`kubectl auth whoami` -> oidc:<user>,
[oidc:platform-admins]), so the rolling restart 3472f1d deferred has
happened.
- UniFi OS Server is NOT installed in the guest. The VM is healthy (Ubuntu
24.04.4, guest agent connected, live-migratable) but :11443 answers nothing,
and the CNP explicitly permits cluster -> :11443, so it is genuinely not
listening rather than a policy artifact.
Docs and comments only; no manifest semantics change.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Replace the dmacvicar/libvirt provider + resources with the modern
bpg/proxmox provider. The dev cluster moves off the memory-constrained
local libvirt host onto the Proxmox VE box.
- versions.tf: pin bpg/proxmox ~> 0.66; provider reads auth from env
(PROXMOX_VE_*, token never a tofu var) and normalizes a schemeless
PROXMOX_VE_ENDPOINT (prepend https://) to avoid the :8006 HTTP 301.
- variables.tf: Proxmox placement vars (node=proxmox, datastore=Storage,
image_datastore=local, bridge=vmbr0); LAN 192.168.1.0/24 gw .1 DNS .18;
node_ips .50/.51/.52; sizing 6 vCPU / 16384 MiB / 40+100 GiB; cpu_type
host; Cilium VIP range .53-.60.
- main.tf: proxmox_virtual_environment_download_file (cloud image -> local),
per-node user-data snippet (proxmox_virtual_environment_file), and
proxmox_virtual_environment_vm per node via name-keyed for_each (imported
OS disk virtio0, blank data disk virtio1, native initialization ip_config
+ dns + user_data_file_id).
- remove moved.tf (libvirt-specific count->for_each state migration).
Authoring only; no apply. Traces R-ENV-01, R-PLAT-01.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
jakman85andClaude Opus 4.8