jakman85andClaude Opus 5
f7d3884ad2
`ci/touched-requirements.sh` documents its detection as "R-* tokens literally
present in the diff of changed files". It did not do that — it grepped the ENTIRE
CONTENT of every changed file. So a change inherited every requirement those files
had ever mentioned in prose, and `--require-all` then blocked the merge on
requirements the author never went near.
On this branch that meant editing the decision log and the tofu docs dragged in
five unrelated requirements, all on untouched lines, all legitimately uncovered:
[coverage] REQUIRE: FAILED — required requirement(s) uncovered: [5 IDs]
The practical effect was that the PM could not append to the decision log without
failing CI — which is a fair part of why the log went stale in the first place.
This is the defect already logged as "FOLLOW-UP (WS-TEST): fix the matcher so real
requirements need not be exempted".
Two scoping rules, both learned from the gate blocking merges it should not have:
1. Scan the ADDED/REMOVED lines of the diff, as the header always claimed.
Removed lines count too — deleting a test's requirement tag, or a policy's
self-declared ID, touches that requirement just as much as adding one.
2. Skip `*.md`. An ID in prose is a REFERENCE, not a declaration: explaining a
requirement is not changing behaviour against it. Rule 1 alone is not enough —
a commit that documents a mis-attribution names the IDs involved and thereby
re-triggers the very mis-attribution it fixes, and the decision log can never
record which requirements a change concerned.
This TIGHTENS attribution rather than weakening the gate:
- no requirement is exempted, and DOCUMENTED_GAPS is untouched;
- authoritative declarations (`conformance.home.k3s/R-*` tags, Kyverno policies,
ServiceDefinitions) all live in YAML, on lines a real change must touch;
- the coarse path->requirement fallback is unchanged and file-type agnostic, so
the doc subtrees that matter (conformance/, schemas/, catalog/, clusters/, ci/)
stay covered either way.
Alternatives rejected: adding the five IDs to DOCUMENTED_GAPS genuinely weakens the
gate; scrubbing requirement IDs out of prose (precedent f9e9640) would mean editing
the decision log's own history to appease a matcher.
Gate semantics change — flagged in DECISIONS-LOG.md for review.
Also drops three R-TEST-06 mentions this branch had added to prose and refreshes
the now-stale "gate is RED" notes.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
7.2 KiB
Dev cluster — current state & reconnect guide
Last verified: 2026-08-26. Phase 1 is complete and long since exited; the cluster has been running the full platform (phases 2–6) for some time. This doc is how a session — human or agent — reconnects to it and re-establishes the toolchain.
History note. The cluster was originally 3 libvirt/KVM VMs on the workstation (
10.73.0.0/24NAT). That was retired after the workstation ran out of RAM/disk; the cluster now lives on Proxmox VE. Anything mentioningvirsh,qemu:///system,tofu+dmacvicar/libvirt, or10.73.0.xfor the cluster is historical — includingcluster.sh(retired) andPHASE1-SCOPE.md(historical scoping doc).
The cluster
| Substrate | Proxmox VE host, node proxmox, disks on the Storage ZFS datastore, bridge vmbr0 |
| Nodes | k3s-dev-1/2/3 at 192.168.1.50 / .51 / .52 — 6 vCPU, 16 GiB, Debian 12 (bookworm) |
| k3s | v1.31.5+k3s1, 3 servers, embedded etcd (HA); no flannel / kube-proxy / Traefik |
| CNI | Cilium 1.16.5 (kube-proxy replacement, L2 announcements, egress gateway) |
| kube-API | kube-vip VIP 192.168.1.49:6443, tls-san'd on the serving cert |
| Ingress VIP | 192.168.1.53 (shared Gateway). Pool 192.168.1.53–.60 for LB VIPs; UniFi holds .54 |
| Storage | Longhorn 1.7.3 — classes longhorn (default), longhorn-static, longhorn-vm (RWX/Block, migratable), longhorn-scratch. bulk (NFS CSI) is a non-provisioning stub pending OQ-2 |
| Nested virt | cpu_type = host → real KVM in-guest; KubeVirt + CDI run on it |
Everything above is provisioned by IaC: tofu/ creates the VMs,
cloud-init/ baselines first boot, ansible/ installs k3s, Cilium,
kube-vip and the VPN egress gateway. Flux then reconciles the whole platform from clusters/dev.
Network path from the workstation (there is a NAT in the way)
The workstation is not on the cluster LAN. A pfSense NAT bridges the two:
| Purpose | Workstation-side target | Forwards to |
|---|---|---|
| kube-API | https://192.168.2.50:6443 |
kube-vip VIP 192.168.1.49:6443 |
| Node SSH (Ansible only) | 192.168.2.102:2250 / 2251 / 2252 |
192.168.1.50 / .51 / .52 port 22 |
| Proxmox API | https://192.168.2.102:8006/ |
the Proxmox host |
TLS through the NAT validates because the kubeconfig sets tls-server-name: 192.168.1.49 — the
VIP is in the serving cert's SAN list, so the NAT address does not have to be.
There is no SSH tunnel any more; the old tunnel was retired when kube-vip + the NAT landed.
Access
kubectl — Keycloak SSO (normal path). ~/.kube/config context k3s-oidc authenticates
against the homelab realm via kubelogin (auth-code + PKCE, public client, no secret in the
kubeconfig). Group membership maps to RBAC by platform/keycloak/k8s-oidc-rbac.yaml:
kubectl config use-context k3s-oidc
kubectl auth whoami # -> oidc:<you>, groups [oidc:platform-admins ...]
oidc:platform-admins → cluster-admin, oidc:viewers → view. First use opens a browser, so
this path needs an interactive desktop.
kubectl — cert kubeconfig (break-glass). The k3s admin cert kubeconfig from
/etc/rancher/k3s/k3s.yaml on k3s-dev-1 (rewrite the server to https://192.168.2.50:6443 and
add tls-server-name: 192.168.1.49). Fully independent of Keycloak — use this when OIDC, the
gateway, or Keycloak itself is down. Headless contexts (CI jobs, containers, agents) also need
this, since they cannot complete a browser login.
Node SSH. ssh -i ~/.ssh/home_network -p 2250 [email protected] (user k3s, NOPASSWD sudo).
Ports 2251/2252 for nodes 2/3.
Toolchain — use the dev container, not the host
Every pinned tool (tofu, ansible, sops, age, flux, kubectl, helm and the full CI gate toolchain: yamllint, check-jsonschema, kyverno, chainsaw, yq) lives in the image:
infra/devcontainer/dev.sh build # once
infra/devcontainer/dev.sh run make -C ci gate # exactly what CI runs
infra/devcontainer/dev.sh # interactive shell at /work
Unlike the libvirt era, tofu and ansible now run fine inside the container — the
bpg/proxmox provider talks to the Proxmox API over the network and needs no host libvirt socket,
which is what broke the old dmacvicar/libvirt provider in a container.
dev.sh mounts the repo at /work, ~/.ssh (read-only), ~/.config/sops (the age keyring —
persisted on the host, or you lose access to every SOPS secret), the ansible-vault passphrase
file, and the sibling spec repo read-only at /home-network-k3s so the spec-immutability
check runs instead of silently skipping.
Known gap.
dev.shdefaultsKUBECONFIGto/work/infra/ansible/.cache/dev.kubeconfig, a libvirt-era path that no longer exists, and does not mount~/.kube. In-containerkubectltherefore needs an explicitDEV_KUBECONFIG=<path to a break-glass cert kubeconfig>— the OIDC context cannot work there anyway (no browser). Not yet fixed.
Off-repo state you must not lose
| What | Where |
|---|---|
| SOPS age private key | ~/.config/sops/age/keys.txt — losing it locks you out of every encrypted secret |
| Ansible vault passphrase | ~/.config/home-k3s/ansible-vault-pass |
| Node SSH key | ~/.ssh/home_network |
| Proxmox API token | operator env only (PROXMOX_VE_API_TOKEN) — never a tofu var, never in state |
| tofu state | infra/tofu/terraform.tfstate (gitignored) — absent on a fresh workstation; a tofu apply without it would try to recreate the live nodes |
| Generated SSO secrets | ~/.config/home-k3s/keycloak-sso-secrets.txt |
Health check
kubectl get nodes # 3/3 Ready
flux get kustomizations # flux-system, platform, platform-config,
# apps-wave-0/1/2, apps-vms — all Ready
kubectl get helmreleases -A # all Ready
kubectl get cluster -A # CNPG: keycloak-db, nextcloud-db, semaphore-db
kubectl get vmi -A # unifi Running
Flux's git source is inside the cluster — mind the loop
GitRepository/flux-system points at https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git:
the in-cluster Gitea, which is itself an app Flux deploys in apps/wave-0. Resolution goes
through split-horizon CoreDNS → Gateway VIP 192.168.1.53, so a Flux reconcile depends on Gitea,
the Gateway, CoreDNS and the wildcard cert all being up.
In practice this is stable — Gitea's Deployment and PVC survive a reboot and come back without
Flux's help, so a Proxmox host reboot self-heals (VMs auto-start, cluster reconverges). The loop
only bites if Gitea is lost rather than restarted. The recovery path is the host bootstrap
Gitea (bash bootstrap/gitea/bootstrap-gitea.sh) plus repointing the GitRepository at it — that
is what the bootstrap/ tree exists for, and it is the reason the bootstrap Gitea is kept.
Push target. origin is that same in-cluster Gitea, so git push origin main is the deploy.
There is no separate "apply" step and no GitHub remote.