The docs had drifted badly behind ~85 commits of delivery. The root README still
said "Phase 0 — Foundations. No cluster exists yet"; the infra docs still
described a libvirt/KVM cluster that has been retired for two days; and
DECISIONS-LOG.md — the PM's authoritative record — stopped at 2026-08-25 AM.
Anyone following these docs would have been working from a false map.
Verified against the live cluster (3/3 nodes, 7/7 Flux Kustomizations Ready,
10/10 HelmReleases Ready) rather than inferred from commit messages.
README.md
- Phase 0 -> a per-phase status table: 0-6 delivered and deployed, 7 not open.
- Layout now shows the wave split, platform-config, apps/vms, infra, bootstrap.
- Real conformance number (36/51, 70.6%) and how to regenerate it.
- Devcontainer workflow; note that the gate was red and why.
infra/PHASE1-STATUS.md — rewritten. Was a `virsh`/libvirt resume guide for a
cluster that no longer exists. Now: the Proxmox substrate, the pfSense NAT
paths, both kubectl access paths (Keycloak OIDC normal / cert break-glass),
the toolchain, off-repo state that must not be lost, and the fact that Flux's
git source is an app inside the cluster (`git push origin main` IS the deploy).
infra/PHASE1-SCOPE.md — historical banner. The exit criteria and sizing
rationale are still worth keeping; the libvirt execution steps are not.
infra/README.md — Proxmox, plus a map of the subtree and where the cluster
actually lives (it is not on this workstation).
infra/devcontainer/README.md — the "run tofu and ansible on the HOST" limitation
is OBSOLETE: it was a dmacvicar/libvirt polkit/PID problem, and bpg/proxmox
talks to an HTTP API. Everything runs in the container now. Documents the full
CI-gate toolchain and what each bind mount is load-bearing for (the age keyring
and the read-only spec repo especially — without the latter, spec-immutability
silently skips instead of verifying).
infra/tofu/README.md — was "Authoring only — nothing here has been applied". It
is applied and live. Disk sizes corrected 40/100 -> 64/128 GiB (confirmed
against the running nodes), pfSense NAT marked done, and a warning that
terraform.tfstate is absent on a fresh workstation — an apply with empty state
would try to recreate the three live nodes.
infra/tofu/variables.tf — section comment disagreed with its own defaults.
infra/cluster.sh — retired-with-instructions header. It silently drives libvirt
domains that do not exist; behaviour unchanged, but it no longer lies.
apps/vms/unifi/README.md — was "NOT WIRED IN", 60Gi, and listed a file that has
since been dropped. Now: live status, 40Gi, resolved open questions marked
resolved, and the two constraints the next person will otherwise rediscover —
KubeVirt's 2048B inline userData cap, and that apps-vms must never carry
postBuild.substituteFrom (envsubst corrupts the cloud-init).
DECISIONS-LOG.md — backfilled the gap as one structured entry: the Proxmox
migration, NAT + kube-vip, the twice-proven oauth2-proxy cold-start deadlock,
the Flux source cutover, CI going live in-cluster, ten app onboardings and the
renderer bugs they exposed, the wave split (verified: DBs adopted, not
recreated — original ages intact, prune restored), KubeVirt storage, the UniFi
VM, and kubectl SSO. Plus verified live state and the carried-forward open list.
Two corrections to the record while verifying:
- kubectl OIDC is LIVE (`kubectl auth whoami` -> oidc:<user>,
[oidc:platform-admins]), so the rolling restart 3472f1d deferred has
happened.
- UniFi OS Server is NOT installed in the guest. The VM is healthy (Ubuntu
24.04.4, guest agent connected, live-migratable) but :11443 answers nothing,
and the CNP explicitly permits cluster -> :11443, so it is genuinely not
listening rather than a policy artifact.
Docs and comments only; no manifest semantics change.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The hand-authored UniFi VM manifests used column-aligned inline flow mappings
(`{ name: inform, port: 8080, protocol: TCP }`), which yamllint's `commas`
rule rejects as "too many spaces after comma". That made `make -C ci lint` FAIL,
so the merge gate has been BLOCKED on `main` since the UniFi commits landed —
every other static stage was green.
Collapse each run of spaces after a comma to one, and re-align the trailing
comments so the port tables stay readable. Whitespace only:
`kubectl kustomize apps/vms` renders byte-for-byte identically before and after.
The lint config is left untouched — relaxing the rule to make failing files pass
is moving the goalposts, not fixing the code.
Also refresh these files' stale `STAGING / DRAFT` headers: the tree is live,
reconciled by the non-gating `apps-vms` Flux Kustomization.
Gate after this change: spec-immutability / lint / l0 / l1 all green,
MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
760084e only committed the unifi-cloudinit.yaml deletion — the `git add` errored
on that already-staged pathspec and skipped the other two files, leaving the
kustomization still referencing the deleted file (broken build). Commit the
intended changes: 30-virtualmachine.yaml inline userData + kustomization.yaml
without the secretGenerator.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
userDataSecretRef tripped a Flux server-side-apply schema quirk on the KubeVirt
VirtualMachine CRD ("field not declared in schema"). Go back to inline userData,
trimmed to 1112 bytes (< the 2048 limit) by dropping the break-glass SSH key —
no longer needed for debugging now that the postBuild cloud-init corruption is
fixed. Keeps qemu-guest-agent (AgentConnected proves cloud-init ran) and the
public-API UOS resolver + non-interactive install. Removes the secretGenerator +
unifi-cloudinit.yaml. (Add SSH back via userDataSecretRef only if the SSA quirk is
worked around and guest access is needed.)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Adding the SSH key pushed the inline cloudInitNoCloud.userData over KubeVirt's
2048-byte limit ("userdata exceeds 2048 byte limit. Should use UserDataSecretRef").
Move the cloud-init into apps/vms/unifi/unifi-cloudinit.yaml and wrap it as the
`unifi-cloudinit` Secret via a Kustomize secretGenerator (disableNameSuffixHash
so the VM can reference it by fixed name); VM now uses userDataSecretRef. Content
is non-sensitive (public key + public installer script), so not SOPS-encrypted.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The first UOS bring-up left no way into the guest (cloud-init set no
credentials), so the in-guest UOS install couldn't be observed. Add:
- ssh_authorized_keys (public key, safe to commit) on the default ubuntu user
- qemu-guest-agent (KubeVirt AgentConnected + virtctl guest access)
- tee the install-uos output to the serial console + /var/log/uos-install.log,
and record the installer exit code, so the install is observable.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The UniFi Block import got past the device-permission fix but then failed
ReplicaSchedulingFailure / "insufficient storage": CDI sizes scratch ~= the disk
virtual size and, on the default 3-replica `longhorn` class, the ~60Gi scratch x3
(189G) on top of the disk's x3 (180G) overflowed the Longhorn tier (~125G/node).
- platform/longhorn/storageclasses.yaml: NEW `longhorn-scratch` StorageClass
(numberOfReplicas: 1) — CDI scratch is ephemeral/throwaway, replicating it 3x is
waste; if a node dies mid-import CDI just retries.
- platform/kubevirt/cdi-cr.yaml: spec.config.scratchSpaceStorageClass ->
longhorn-scratch.
- apps/vms/unifi/20-datavolume.yaml: boot disk 60Gi -> 40Gi (UOS min is 25Gi;
40Gi is comfortable for a home site and lighter on the replicated tier).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The UniFi VM boot-disk DataVolume (RWX + Block on longhorn-vm) failed to import:
CDI's non-root importer (UID 107, cap-drop ALL) hit
`blockdev: cannot open /dev/cdi-block-volume: Permission denied`.
Kubernetes only applies fsGroup ownership to *filesystem* volumes — it never
chowns a raw block device — and Longhorn's CSIDriver (fsGroupPolicy:
ReadWriteOnceWithFSType) skips fsGroup on RWX/Block volumes regardless, so the
device stays root:root. The CDI-documented fix is the container-runtime setting
`device_ownership_from_security_context`, exposed on k3s (>= v1.31.3+k3s1, this
cluster is v1.31.5) as `nonroot-devices: true`. It makes containerd set the
in-container device node's owner from the pod's securityContext.
Added to the Ansible-managed /etc/rancher/k3s/config.yaml alongside tls-san;
applied via the existing rolling `Restart k3s` handler (serial:1, kube-vip keeps
the API up, etcd holds 2/3). Node-scoped, security-neutral (pods stay non-root),
and preserves the longhorn-vm RWX/Block/migratable contract unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
First KubeVirt VM app + first real CDI http import. UniFi OS Server (UOS) is
Ubiquiti's current self-host product (the .deb is deprecated); it bundles UniFi
OS + Network app + its own datastore, so no MongoDB companion.
- apps/vms/unifi: CDI DataVolume (Ubuntu 24.04 cloud image -> longhorn-vm
RWX/Block, live-migratable) + VirtualMachine (evictionStrategy: LiveMigrate)
with cloud-init that installs podman/slirp4netns/jq and runs the UOS installer
NON-INTERACTIVELY. The installer URL is resolved at boot from Ubiquiti's public
firmware API (fw-update.ui.com/api/firmware-latest) — always-latest, no pinned
URL to maintain (currently UOS v5.1.37).
- L4 exposure: two LoadBalancer Services (TCP+UDP) sharing a pinned, L2-announced
Cilium VIP 192.168.1.54 (device inform/adoption + STUN can't ride the L7
Gateway); external-dns publishes unifi.home.keithjackson.dev. UI on :11443.
- default-deny + additive CNPs (controller ingress, egress, and CDI-importer
egress for the image pull).
- clusters/base/apps-vms.yaml: NON-GATING Flux Kustomization (wait:false,
dependsOn platform-config) — VMs are a separate lifecycle from the app waves,
so a slow VM import/install can't stall the app tier.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
jakman85andClaude Opus 5