Operator report: dev.sh only reached the dev cluster. Line 72 hardcoded
`KUBECONFIG=${DEV_KUBECONFIG:-…/dev.kubeconfig}`, and the only escape was an
undocumented override whose NAME misleads — you set `DEV_KUBECONFIG` to a *prod*
path. infra/PHASE1-STATUS.md had recorded this as a "Known gap" for months.
WHY THIS WAS NOT A ONE-LINER. Both kubeconfigs name their context, cluster AND
user `default`. kubectl's merge keeps the FIRST occurrence of a duplicated name,
so the obvious `KUBECONFIG=dev.kubeconfig:prod.kubeconfig` resolves `default` to
DEV and `--context default` can never select prod — an operator or agent would
believe they were on prod and be issuing commands against the LIVE dev cluster
that runs the operator's platform. Same collision class as ~/.kube/config an hour
earlier, and worse here because agents run in this container.
Interface: `CLUSTER=dev|prod` as an env prefix, default **dev** (back-compat: every
existing invocation, CI job and habit keeps meaning dev).
infra/devcontainer/dev.sh run kubectl get nodes # dev
CLUSTER=prod infra/devcontainer/dev.sh run kubectl get nodes # prod
Its one weakness — `CLUSTER` can be inherited from an exported shell var — is
covered by a banner printed on EVERY invocation before the container starts:
`==> dev.sh: cluster=prod (kube context k3s-prod)`.
New infra/devcontainer/kubeconfig.sh generates a merged config with DISTINCT names
(contexts k3s-dev/k3s-prod, users k3s-*-admin), renaming structurally with `yq`
on .clusters[].name / .users[].name / .contexts[].{name,context.cluster,
context.user} before letting `kubectl config view --flatten` merge them. NOT sed —
sed renaming corrupted a config earlier today because cluster and user entries are
both `- name: default` at identical indent. Output is 0600 in the gitignored
.cache/, regenerated every run, and post-conditions assert the current context is
`k3s-<target>` and that NO context named `default` survived.
FAILS LOUDLY AT BOTH LAYERS, and never falls back to the other cluster — that
fallback is exactly how a prod command ends up hitting dev. Host-side refusal if
the requested cluster's kubeconfig is missing; in-container refusal likewise;
unknown CLUSTER exits 2. A missing source for the cluster you did NOT ask for is
tolerated (merges the other one only).
VERIFIED BY ME, independently of the agent — the two BASE_DOMAIN reads are the
unambiguous proof of which cluster was reached, since a node list is not:
default (no selector) -> home.keithjackson.dev (DEV, unchanged)
CLUSTER=prod -> prod.home.keithjackson.dev (PROD)
contexts -> k3s-dev, k3s-prod; NO `default` context survives
exit codes -> invalid CLUSTER=2, valid dev/prod=0
standup.sh -> `--kubeconfig` still wins over the new default
(preflight against prod: target confirmed 192.168.2.9)
credentials -> dev.kubeconfig and prod.kubeconfig byte-intact and each
still reaching 3 nodes after the agent's move-aside test
git -> no kubeconfig committable; clusters.kubeconfig ignored
Full CI gate PASS.
NOT VERIFIED: `dev.sh build` and the interactive shell path (existing image reused,
no TTY available). The shell path shares run()'s code; only the TTY differs.
DEV_KUBECONFIG still wins verbatim, now with a deprecation warning. All existing
mounts are untouched: SOPS age keyring rw, ~/.ssh ro, ansible-vault-pass ro, spec
repo ro + SPEC_REPO, --network host, label=disable.
Closes the PHASE1-STATUS "Known gap".
Refs R-ENV-02, R-TEST-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
Text-only cleanup after the upgrade campaign. Updates stale version
strings, prose, and comments to the verified current facts; no
Kubernetes behavior or version pins changed except the kubectl
toolchain bump (below).
Verified facts applied: k3s v1.36.4+k3s1, Debian 13 trixie,
Cilium 1.20.1 / Gateway API v1.6.1, Longhorn 1.12.1, node RAM 24 GiB,
dedicated virtio2 SSD etcd disk per node, Velero chart 12.x / v1.18,
Grafana 13.2.0.
- Version strings across infra/ and platform/ READMEs + comments
- kubectl toolchain 1.31.4 -> 1.36.4 (devcontainer/ci/conformance
versions.env; 1.31 vs a 1.36 cluster exceeded the +-1 skew)
- Retired-single-apps-Kustomization refs -> apps-wave-0/1/2 + apps-vms
with per-wave SOPS decryption (flux/clusters docs + bootstrap.sh)
- COMPLETED/historical banners on the three executed plan docs
- Resolved TODO(WS-SEC) markers -> Renovate-maintained ranges
- Appended dated DECISIONS-LOG entry (2026-08-30) superseding the
2026-08-26 LIVE STATE line
Left as revalidation items (not blind-edited): external-snapshotter
v8.2.0 vs 1.36, Cilium ext_authz/forward-auth relevance under 1.20.1,
conformance ImageVolume feature-gate assertions under 1.36.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
`ci/touched-requirements.sh` documents its detection as "R-* tokens literally
present in the diff of changed files". It did not do that — it grepped the ENTIRE
CONTENT of every changed file. So a change inherited every requirement those files
had ever mentioned in prose, and `--require-all` then blocked the merge on
requirements the author never went near.
On this branch that meant editing the decision log and the tofu docs dragged in
five unrelated requirements, all on untouched lines, all legitimately uncovered:
[coverage] REQUIRE: FAILED — required requirement(s) uncovered: [5 IDs]
The practical effect was that the PM could not append to the decision log without
failing CI — which is a fair part of why the log went stale in the first place.
This is the defect already logged as "FOLLOW-UP (WS-TEST): fix the matcher so real
requirements need not be exempted".
Two scoping rules, both learned from the gate blocking merges it should not have:
1. Scan the ADDED/REMOVED lines of the diff, as the header always claimed.
Removed lines count too — deleting a test's requirement tag, or a policy's
self-declared ID, touches that requirement just as much as adding one.
2. Skip `*.md`. An ID in prose is a REFERENCE, not a declaration: explaining a
requirement is not changing behaviour against it. Rule 1 alone is not enough —
a commit that documents a mis-attribution names the IDs involved and thereby
re-triggers the very mis-attribution it fixes, and the decision log can never
record which requirements a change concerned.
This TIGHTENS attribution rather than weakening the gate:
- no requirement is exempted, and DOCUMENTED_GAPS is untouched;
- authoritative declarations (`conformance.home.k3s/R-*` tags, Kyverno policies,
ServiceDefinitions) all live in YAML, on lines a real change must touch;
- the coarse path->requirement fallback is unchanged and file-type agnostic, so
the doc subtrees that matter (conformance/, schemas/, catalog/, clusters/, ci/)
stay covered either way.
Alternatives rejected: adding the five IDs to DOCUMENTED_GAPS genuinely weakens the
gate; scrubbing requirement IDs out of prose (precedent f9e9640) would mean editing
the decision log's own history to appease a matcher.
Gate semantics change — flagged in DECISIONS-LOG.md for review.
Also drops three R-TEST-06 mentions this branch had added to prose and refreshes
the now-stale "gate is RED" notes.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The docs had drifted badly behind ~85 commits of delivery. The root README still
said "Phase 0 — Foundations. No cluster exists yet"; the infra docs still
described a libvirt/KVM cluster that has been retired for two days; and
DECISIONS-LOG.md — the PM's authoritative record — stopped at 2026-08-25 AM.
Anyone following these docs would have been working from a false map.
Verified against the live cluster (3/3 nodes, 7/7 Flux Kustomizations Ready,
10/10 HelmReleases Ready) rather than inferred from commit messages.
README.md
- Phase 0 -> a per-phase status table: 0-6 delivered and deployed, 7 not open.
- Layout now shows the wave split, platform-config, apps/vms, infra, bootstrap.
- Real conformance number (36/51, 70.6%) and how to regenerate it.
- Devcontainer workflow; note that the gate was red and why.
infra/PHASE1-STATUS.md — rewritten. Was a `virsh`/libvirt resume guide for a
cluster that no longer exists. Now: the Proxmox substrate, the pfSense NAT
paths, both kubectl access paths (Keycloak OIDC normal / cert break-glass),
the toolchain, off-repo state that must not be lost, and the fact that Flux's
git source is an app inside the cluster (`git push origin main` IS the deploy).
infra/PHASE1-SCOPE.md — historical banner. The exit criteria and sizing
rationale are still worth keeping; the libvirt execution steps are not.
infra/README.md — Proxmox, plus a map of the subtree and where the cluster
actually lives (it is not on this workstation).
infra/devcontainer/README.md — the "run tofu and ansible on the HOST" limitation
is OBSOLETE: it was a dmacvicar/libvirt polkit/PID problem, and bpg/proxmox
talks to an HTTP API. Everything runs in the container now. Documents the full
CI-gate toolchain and what each bind mount is load-bearing for (the age keyring
and the read-only spec repo especially — without the latter, spec-immutability
silently skips instead of verifying).
infra/tofu/README.md — was "Authoring only — nothing here has been applied". It
is applied and live. Disk sizes corrected 40/100 -> 64/128 GiB (confirmed
against the running nodes), pfSense NAT marked done, and a warning that
terraform.tfstate is absent on a fresh workstation — an apply with empty state
would try to recreate the three live nodes.
infra/tofu/variables.tf — section comment disagreed with its own defaults.
infra/cluster.sh — retired-with-instructions header. It silently drives libvirt
domains that do not exist; behaviour unchanged, but it no longer lies.
apps/vms/unifi/README.md — was "NOT WIRED IN", 60Gi, and listed a file that has
since been dropped. Now: live status, 40Gi, resolved open questions marked
resolved, and the two constraints the next person will otherwise rediscover —
KubeVirt's 2048B inline userData cap, and that apps-vms must never carry
postBuild.substituteFrom (envsubst corrupts the cloud-init).
DECISIONS-LOG.md — backfilled the gap as one structured entry: the Proxmox
migration, NAT + kube-vip, the twice-proven oauth2-proxy cold-start deadlock,
the Flux source cutover, CI going live in-cluster, ten app onboardings and the
renderer bugs they exposed, the wave split (verified: DBs adopted, not
recreated — original ages intact, prune restored), KubeVirt storage, the UniFi
VM, and kubectl SSO. Plus verified live state and the carried-forward open list.
Two corrections to the record while verifying:
- kubectl OIDC is LIVE (`kubectl auth whoami` -> oidc:<user>,
[oidc:platform-admins]), so the rolling restart 3472f1d deferred has
happened.
- UniFi OS Server is NOT installed in the guest. The VM is healthy (Ubuntu
24.04.4, guest agent connected, live-migratable) but :11443 answers nothing,
and the CNP explicitly permits cluster -> :11443, so it is genuinely not
listening rather than a policy artifact.
Docs and comments only; no manifest semantics change.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Controller now reaches the kube-API directly through the pfSense NAT
(192.168.2.50:6443 -> kube-vip VIP 192.168.1.49:6443), validating TLS
against the VIP via tls-server-name. Removes the dev.sh SSH-tunnel
scaffolding and updates the ansible source-of-truth so a rebuild
regenerates a direct-access kubeconfig.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Bake the conformance harness toolchain into infra/devcontainer so `make -C ci gate`
runs identically here and in CI:
- Containerfile: install kyverno CLI (1.13.4), kyverno-json, chainsaw (0.2.12),
yamllint (1.35.1), check-jsonschema (0.29.4), yq (4.44.6) via new pinned RUN block;
add diffutils + python3-pip; drop stale LIBVIRT_DEFAULT_URI; refresh header for the
Proxmox(bpg)+conformance toolbox; extend /etc/dev-toolchain-versions provenance.
- versions.env: add KYVERNO/KYVERNO_JSON/CHAINSAW/YAMLLINT/YQ/CHECK_JSONSCHEMA;
align KUBECTL 1.32.0 -> 1.31.4. dev.sh build(): pass the new --build-args.
- ci/versions.env: pin CHECK_JSONSCHEMA_VERSION=0.29.4 (single source of truth).
Fix harness seams so the gate can run:
- dev.sh run(): mount the sibling spec repo (home-network-k3s) read-only at
/home-network-k3s for ci/spec-immutability.sh (R-TEST-06); conditional so the
container still runs without it.
- conformance/lib/common.sh conf_dirs_for_layer(): accept a comma/space list of
layers and union the dirs, so ci/conformance.sh `run.sh --layer l2,l3` selects
both layers in one pass (backward compatible for single layers).
NOTE: ci/versions.env pins KYVERNO_JSON_VERSION=0.0.5, which has no upstream release
(max tag is v0.0.3; platform/kyverno/scripts/run-l0.sh already defaults to v0.0.3).
Baked v0.0.3 and flagged for PM to correct ci/versions.env. kyverno-json is not used
by any static gate stage.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
- drop stale libvirt socket mount + LIBVIRT_DEFAULT_URI (old KVM cluster)
- auto-open an SSH tunnel (local :16443 -> primary 127.0.0.1:6443 over the
pfSense SSH NAT) on each run so kubectl/flux/helm reach the cluster with no
manual setup; DEV_NO_TUNNEL=1 opts out, K3S_SSH_* env overrides the NAT front
- refresh header/usage comments (bpg/proxmox, not dmacvicar/libvirt)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
- inventory routes SSH via 192.168.2.102:2250/2251/2252; new k3s_node_ip carries
each node's real 192.168.1.x for --node-ip / k3s_api_host / Cilium parity
- dev.sh bind-mounts ~/.config/home-k3s/ansible-vault-pass (ro) so vaulted
playbooks can run in the pinned container (ansible.cfg vault_password_file)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Two dev.sh gaps surfaced while running bootstrap/sops/install-age-key.sh:
- kubectl defaulted to localhost:8080 (connection refused) because KUBECONFIG
was never set in the toolbox. Set KUBECONFIG=/work/infra/ansible/.cache/dev.kubeconfig
(override via DEV_KUBECONFIG); --network host already reaches 10.73.0.11:6443.
- age-keygen.sh writes the PRIVATE key to ~/.config/sops/age/keys.txt, but that
path was not mounted in the --rm container, so the key was lost on exit. Mount
${HOME}/.config/sops -> /root/.config/sops (created first) so the key persists
on the host and SOPS_AGE_KEY_FILE resolves to it.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Cluster is UP (3/3 Ready, Cilium OK); Phase 1 paused before Flux bootstrap
(Option A: suspend platform/apps until Phase 2). Add infra/PHASE1-STATUS.md
(what's done, state file locations, reconnect steps, remaining T-1.4/1.5,
Phase-2 platform prereq), infra/cluster.sh (status/start/stop/kubeconfig), and
document that tofu+ansible run on the host (libvirt provider limitation).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Per operator rule: OpenTofu/age/sops/flux are Go binaries, so bake the toolchain
into a pinned Fedora-44 dev container (matches host libvirt 12) instead of a venv.
infra/devcontainer/: Containerfile + versions.env (tofu 1.9.0, age 1.2.1, sops
3.9.4, flux 2.4.0, kubectl 1.32.0, helm 3.16.0) + dev.sh (runs with host libvirt
socket + repo + ssh mounted) + README. Built & validated: all tools resolve;
virsh reaches host libvirt from inside the container.
Also record resolved Phase 1 decisions in PHASE1-SCOPE.md (6GiB/4vCPU x3;
k3s-dev NAT 10.73.0.0/24; container toolchain). Gate green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
jakman85andClaude Opus 5