infra/devcontainer/kubeconfig.sh was never staged into the delivery-A commit, so
the branch built on main carried main's single-cluster tooling and `--context
k3s-prod` failed with 'context does not exist' mid-incident. Not Flux-reconciled,
so it never affected the cluster — but it breaks prod diagnosis from a checkout of
this branch, which is when you need it most.
Carries the concurrency fix: the merged kubeconfig holds both contexts and NO
current-context, with each invocation getting a per-invocation selector file
prepended to KUBECONFIG. Two concurrent dev.sh runs with different CLUSTER values
previously overwrote each other's current-context mid-command — reproduced live,
a CLUSTER=dev reader returned k3s-prod nodes on 3 of 7 iterations.
Co-Authored-By: Claude Opus 5 <[email protected]>
`ci/touched-requirements.sh` documents its detection as "R-* tokens literally
present in the diff of changed files". It did not do that — it grepped the ENTIRE
CONTENT of every changed file. So a change inherited every requirement those files
had ever mentioned in prose, and `--require-all` then blocked the merge on
requirements the author never went near.
On this branch that meant editing the decision log and the tofu docs dragged in
five unrelated requirements, all on untouched lines, all legitimately uncovered:
[coverage] REQUIRE: FAILED — required requirement(s) uncovered: [5 IDs]
The practical effect was that the PM could not append to the decision log without
failing CI — which is a fair part of why the log went stale in the first place.
This is the defect already logged as "FOLLOW-UP (WS-TEST): fix the matcher so real
requirements need not be exempted".
Two scoping rules, both learned from the gate blocking merges it should not have:
1. Scan the ADDED/REMOVED lines of the diff, as the header always claimed.
Removed lines count too — deleting a test's requirement tag, or a policy's
self-declared ID, touches that requirement just as much as adding one.
2. Skip `*.md`. An ID in prose is a REFERENCE, not a declaration: explaining a
requirement is not changing behaviour against it. Rule 1 alone is not enough —
a commit that documents a mis-attribution names the IDs involved and thereby
re-triggers the very mis-attribution it fixes, and the decision log can never
record which requirements a change concerned.
This TIGHTENS attribution rather than weakening the gate:
- no requirement is exempted, and DOCUMENTED_GAPS is untouched;
- authoritative declarations (`conformance.home.k3s/R-*` tags, Kyverno policies,
ServiceDefinitions) all live in YAML, on lines a real change must touch;
- the coarse path->requirement fallback is unchanged and file-type agnostic, so
the doc subtrees that matter (conformance/, schemas/, catalog/, clusters/, ci/)
stay covered either way.
Alternatives rejected: adding the five IDs to DOCUMENTED_GAPS genuinely weakens the
gate; scrubbing requirement IDs out of prose (precedent f9e9640) would mean editing
the decision log's own history to appease a matcher.
Gate semantics change — flagged in DECISIONS-LOG.md for review.
Also drops three R-TEST-06 mentions this branch had added to prose and refreshes
the now-stale "gate is RED" notes.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The docs had drifted badly behind ~85 commits of delivery. The root README still
said "Phase 0 — Foundations. No cluster exists yet"; the infra docs still
described a libvirt/KVM cluster that has been retired for two days; and
DECISIONS-LOG.md — the PM's authoritative record — stopped at 2026-08-25 AM.
Anyone following these docs would have been working from a false map.
Verified against the live cluster (3/3 nodes, 7/7 Flux Kustomizations Ready,
10/10 HelmReleases Ready) rather than inferred from commit messages.
README.md
- Phase 0 -> a per-phase status table: 0-6 delivered and deployed, 7 not open.
- Layout now shows the wave split, platform-config, apps/vms, infra, bootstrap.
- Real conformance number (36/51, 70.6%) and how to regenerate it.
- Devcontainer workflow; note that the gate was red and why.
infra/PHASE1-STATUS.md — rewritten. Was a `virsh`/libvirt resume guide for a
cluster that no longer exists. Now: the Proxmox substrate, the pfSense NAT
paths, both kubectl access paths (Keycloak OIDC normal / cert break-glass),
the toolchain, off-repo state that must not be lost, and the fact that Flux's
git source is an app inside the cluster (`git push origin main` IS the deploy).
infra/PHASE1-SCOPE.md — historical banner. The exit criteria and sizing
rationale are still worth keeping; the libvirt execution steps are not.
infra/README.md — Proxmox, plus a map of the subtree and where the cluster
actually lives (it is not on this workstation).
infra/devcontainer/README.md — the "run tofu and ansible on the HOST" limitation
is OBSOLETE: it was a dmacvicar/libvirt polkit/PID problem, and bpg/proxmox
talks to an HTTP API. Everything runs in the container now. Documents the full
CI-gate toolchain and what each bind mount is load-bearing for (the age keyring
and the read-only spec repo especially — without the latter, spec-immutability
silently skips instead of verifying).
infra/tofu/README.md — was "Authoring only — nothing here has been applied". It
is applied and live. Disk sizes corrected 40/100 -> 64/128 GiB (confirmed
against the running nodes), pfSense NAT marked done, and a warning that
terraform.tfstate is absent on a fresh workstation — an apply with empty state
would try to recreate the three live nodes.
infra/tofu/variables.tf — section comment disagreed with its own defaults.
infra/cluster.sh — retired-with-instructions header. It silently drives libvirt
domains that do not exist; behaviour unchanged, but it no longer lies.
apps/vms/unifi/README.md — was "NOT WIRED IN", 60Gi, and listed a file that has
since been dropped. Now: live status, 40Gi, resolved open questions marked
resolved, and the two constraints the next person will otherwise rediscover —
KubeVirt's 2048B inline userData cap, and that apps-vms must never carry
postBuild.substituteFrom (envsubst corrupts the cloud-init).
DECISIONS-LOG.md — backfilled the gap as one structured entry: the Proxmox
migration, NAT + kube-vip, the twice-proven oauth2-proxy cold-start deadlock,
the Flux source cutover, CI going live in-cluster, ten app onboardings and the
renderer bugs they exposed, the wave split (verified: DBs adopted, not
recreated — original ages intact, prune restored), KubeVirt storage, the UniFi
VM, and kubectl SSO. Plus verified live state and the carried-forward open list.
Two corrections to the record while verifying:
- kubectl OIDC is LIVE (`kubectl auth whoami` -> oidc:<user>,
[oidc:platform-admins]), so the rolling restart 3472f1d deferred has
happened.
- UniFi OS Server is NOT installed in the guest. The VM is healthy (Ubuntu
24.04.4, guest agent connected, live-migratable) but :11443 answers nothing,
and the CNP explicitly permits cluster -> :11443, so it is genuinely not
listening rather than a policy artifact.
Docs and comments only; no manifest semantics change.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Cluster is UP (3/3 Ready, Cilium OK); Phase 1 paused before Flux bootstrap
(Option A: suspend platform/apps until Phase 2). Add infra/PHASE1-STATUS.md
(what's done, state file locations, reconnect steps, remaining T-1.4/1.5,
Phase-2 platform prereq), infra/cluster.sh (status/start/stop/kubeconfig), and
document that tofu+ansible run on the host (libvirt provider limitation).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Per operator rule: OpenTofu/age/sops/flux are Go binaries, so bake the toolchain
into a pinned Fedora-44 dev container (matches host libvirt 12) instead of a venv.
infra/devcontainer/: Containerfile + versions.env (tofu 1.9.0, age 1.2.1, sops
3.9.4, flux 2.4.0, kubectl 1.32.0, helm 3.16.0) + dev.sh (runs with host libvirt
socket + repo + ssh mounted) + README. Built & validated: all tools resolve;
virsh reaches host libvirt from inside the container.
Also record resolved Phase 1 decisions in PHASE1-SCOPE.md (6GiB/4vCPU x3;
k3s-dev NAT 10.73.0.0/24; container toolchain). Gate green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
jakman85andClaude Opus 5