jakman85andClaude Opus 5
f7d3884ad2
`ci/touched-requirements.sh` documents its detection as "R-* tokens literally
present in the diff of changed files". It did not do that — it grepped the ENTIRE
CONTENT of every changed file. So a change inherited every requirement those files
had ever mentioned in prose, and `--require-all` then blocked the merge on
requirements the author never went near.
On this branch that meant editing the decision log and the tofu docs dragged in
five unrelated requirements, all on untouched lines, all legitimately uncovered:
[coverage] REQUIRE: FAILED — required requirement(s) uncovered: [5 IDs]
The practical effect was that the PM could not append to the decision log without
failing CI — which is a fair part of why the log went stale in the first place.
This is the defect already logged as "FOLLOW-UP (WS-TEST): fix the matcher so real
requirements need not be exempted".
Two scoping rules, both learned from the gate blocking merges it should not have:
1. Scan the ADDED/REMOVED lines of the diff, as the header always claimed.
Removed lines count too — deleting a test's requirement tag, or a policy's
self-declared ID, touches that requirement just as much as adding one.
2. Skip `*.md`. An ID in prose is a REFERENCE, not a declaration: explaining a
requirement is not changing behaviour against it. Rule 1 alone is not enough —
a commit that documents a mis-attribution names the IDs involved and thereby
re-triggers the very mis-attribution it fixes, and the decision log can never
record which requirements a change concerned.
This TIGHTENS attribution rather than weakening the gate:
- no requirement is exempted, and DOCUMENTED_GAPS is untouched;
- authoritative declarations (`conformance.home.k3s/R-*` tags, Kyverno policies,
ServiceDefinitions) all live in YAML, on lines a real change must touch;
- the coarse path->requirement fallback is unchanged and file-type agnostic, so
the doc subtrees that matter (conformance/, schemas/, catalog/, clusters/, ci/)
stay covered either way.
Alternatives rejected: adding the five IDs to DOCUMENTED_GAPS genuinely weakens the
gate; scrubbing requirement IDs out of prose (precedent f9e9640) would mean editing
the decision log's own history to appease a matcher.
Gate semantics change — flagged in DECISIONS-LOG.md for review.
Also drops three R-TEST-06 mentions this branch had added to prose and refreshes
the now-stale "gate is RED" notes.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
130 lines
7.2 KiB
Markdown
130 lines
7.2 KiB
Markdown
# Dev cluster — current state & reconnect guide
|
||
|
||
**Last verified: 2026-08-26.** Phase 1 is **complete and long since exited**; the cluster has been
|
||
running the full platform (phases 2–6) for some time. This doc is how a session — human or agent —
|
||
reconnects to it and re-establishes the toolchain.
|
||
|
||
> **History note.** The cluster was originally 3 libvirt/KVM VMs on the workstation
|
||
> (`10.73.0.0/24` NAT). That was **retired** after the workstation ran out of RAM/disk; the cluster
|
||
> now lives on **Proxmox VE**. Anything mentioning `virsh`, `qemu:///system`, `tofu`+`dmacvicar/libvirt`,
|
||
> or `10.73.0.x` for the cluster is historical — including [`cluster.sh`](cluster.sh) (retired) and
|
||
> [`PHASE1-SCOPE.md`](PHASE1-SCOPE.md) (historical scoping doc).
|
||
|
||
## The cluster
|
||
|
||
| | |
|
||
|---|---|
|
||
| Substrate | Proxmox VE host, node `proxmox`, disks on the `Storage` ZFS datastore, bridge `vmbr0` |
|
||
| Nodes | `k3s-dev-1/2/3` at `192.168.1.50 / .51 / .52` — 6 vCPU, 16 GiB, Debian 12 (bookworm) |
|
||
| k3s | **v1.31.5+k3s1**, 3 servers, embedded etcd (HA); no flannel / kube-proxy / Traefik |
|
||
| CNI | Cilium 1.16.5 (kube-proxy replacement, L2 announcements, egress gateway) |
|
||
| kube-API | kube-vip VIP **`192.168.1.49:6443`**, tls-san'd on the serving cert |
|
||
| Ingress VIP | **`192.168.1.53`** (shared Gateway). Pool `192.168.1.53–.60` for LB VIPs; UniFi holds `.54` |
|
||
| Storage | Longhorn 1.7.3 — classes `longhorn` (default), `longhorn-static`, `longhorn-vm` (RWX/Block, migratable), `longhorn-scratch`. `bulk` (NFS CSI) is a **non-provisioning stub** pending OQ-2 |
|
||
| Nested virt | `cpu_type = host` → real KVM in-guest; KubeVirt + CDI run on it |
|
||
|
||
Everything above is provisioned by IaC: [`tofu/`](tofu/) creates the VMs,
|
||
[`cloud-init/`](cloud-init/) baselines first boot, [`ansible/`](ansible/) installs k3s, Cilium,
|
||
kube-vip and the VPN egress gateway. Flux then reconciles the whole platform from `clusters/dev`.
|
||
|
||
## Network path from the workstation (there is a NAT in the way)
|
||
|
||
The workstation is **not** on the cluster LAN. A pfSense NAT bridges the two:
|
||
|
||
| Purpose | Workstation-side target | Forwards to |
|
||
|---|---|---|
|
||
| kube-API | `https://192.168.2.50:6443` | kube-vip VIP `192.168.1.49:6443` |
|
||
| Node SSH (Ansible only) | `192.168.2.102:2250 / 2251 / 2252` | `192.168.1.50 / .51 / .52` port 22 |
|
||
| Proxmox API | `https://192.168.2.102:8006/` | the Proxmox host |
|
||
|
||
TLS through the NAT validates because the kubeconfig sets `tls-server-name: 192.168.1.49` — the
|
||
VIP is in the serving cert's SAN list, so the NAT address does not have to be.
|
||
|
||
There is **no SSH tunnel** any more; the old tunnel was retired when kube-vip + the NAT landed.
|
||
|
||
## Access
|
||
|
||
**kubectl — Keycloak SSO (normal path).** `~/.kube/config` context **`k3s-oidc`** authenticates
|
||
against the `homelab` realm via `kubelogin` (auth-code + PKCE, public client, no secret in the
|
||
kubeconfig). Group membership maps to RBAC by `platform/keycloak/k8s-oidc-rbac.yaml`:
|
||
|
||
```
|
||
kubectl config use-context k3s-oidc
|
||
kubectl auth whoami # -> oidc:<you>, groups [oidc:platform-admins ...]
|
||
```
|
||
|
||
`oidc:platform-admins` → `cluster-admin`, `oidc:viewers` → `view`. First use opens a browser, so
|
||
this path needs an interactive desktop.
|
||
|
||
**kubectl — cert kubeconfig (break-glass).** The k3s admin cert kubeconfig from
|
||
`/etc/rancher/k3s/k3s.yaml` on `k3s-dev-1` (rewrite the server to `https://192.168.2.50:6443` and
|
||
add `tls-server-name: 192.168.1.49`). Fully independent of Keycloak — **use this when OIDC, the
|
||
gateway, or Keycloak itself is down.** Headless contexts (CI jobs, containers, agents) also need
|
||
this, since they cannot complete a browser login.
|
||
|
||
**Node SSH.** `ssh -i ~/.ssh/home_network -p 2250 [email protected]` (user `k3s`, NOPASSWD sudo).
|
||
Ports 2251/2252 for nodes 2/3.
|
||
|
||
## Toolchain — use the dev container, not the host
|
||
|
||
Every pinned tool (tofu, ansible, sops, age, flux, kubectl, helm **and** the full CI gate
|
||
toolchain: yamllint, check-jsonschema, kyverno, chainsaw, yq) lives in the image:
|
||
|
||
```bash
|
||
infra/devcontainer/dev.sh build # once
|
||
infra/devcontainer/dev.sh run make -C ci gate # exactly what CI runs
|
||
infra/devcontainer/dev.sh # interactive shell at /work
|
||
```
|
||
|
||
Unlike the libvirt era, **tofu and ansible now run fine inside the container** — the
|
||
`bpg/proxmox` provider talks to the Proxmox API over the network and needs no host libvirt socket,
|
||
which is what broke the old `dmacvicar/libvirt` provider in a container.
|
||
|
||
`dev.sh` mounts the repo at `/work`, `~/.ssh` (read-only), `~/.config/sops` (the age keyring —
|
||
**persisted on the host, or you lose access to every SOPS secret**), the ansible-vault passphrase
|
||
file, and the sibling spec repo read-only at `/home-network-k3s` so the spec-immutability
|
||
check runs instead of silently skipping.
|
||
|
||
> **Known gap.** `dev.sh` defaults `KUBECONFIG` to `/work/infra/ansible/.cache/dev.kubeconfig`,
|
||
> a libvirt-era path that no longer exists, and does not mount `~/.kube`. In-container `kubectl`
|
||
> therefore needs an explicit `DEV_KUBECONFIG=<path to a break-glass cert kubeconfig>` — the OIDC
|
||
> context cannot work there anyway (no browser). Not yet fixed.
|
||
|
||
## Off-repo state you must not lose
|
||
|
||
| What | Where |
|
||
|---|---|
|
||
| SOPS **age private key** | `~/.config/sops/age/keys.txt` — losing it locks you out of every encrypted secret |
|
||
| Ansible vault passphrase | `~/.config/home-k3s/ansible-vault-pass` |
|
||
| Node SSH key | `~/.ssh/home_network` |
|
||
| Proxmox API token | operator env only (`PROXMOX_VE_API_TOKEN`) — never a tofu var, never in state |
|
||
| tofu state | `infra/tofu/terraform.tfstate` (gitignored) — **absent on a fresh workstation**; a `tofu apply` without it would try to recreate the live nodes |
|
||
| Generated SSO secrets | `~/.config/home-k3s/keycloak-sso-secrets.txt` |
|
||
|
||
## Health check
|
||
|
||
```bash
|
||
kubectl get nodes # 3/3 Ready
|
||
flux get kustomizations # flux-system, platform, platform-config,
|
||
# apps-wave-0/1/2, apps-vms — all Ready
|
||
kubectl get helmreleases -A # all Ready
|
||
kubectl get cluster -A # CNPG: keycloak-db, nextcloud-db, semaphore-db
|
||
kubectl get vmi -A # unifi Running
|
||
```
|
||
|
||
## Flux's git source is inside the cluster — mind the loop
|
||
|
||
`GitRepository/flux-system` points at **`https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`**:
|
||
the **in-cluster** Gitea, which is itself an app Flux deploys in `apps/wave-0`. Resolution goes
|
||
through split-horizon CoreDNS → Gateway VIP `192.168.1.53`, so a Flux reconcile depends on Gitea,
|
||
the Gateway, CoreDNS and the wildcard cert all being up.
|
||
|
||
In practice this is stable — Gitea's Deployment and PVC survive a reboot and come back without
|
||
Flux's help, so a Proxmox host reboot self-heals (VMs auto-start, cluster reconverges). The loop
|
||
only bites if Gitea is **lost** rather than restarted. The recovery path is the host bootstrap
|
||
Gitea (`bash bootstrap/gitea/bootstrap-gitea.sh`) plus repointing the GitRepository at it — that
|
||
is what the `bootstrap/` tree exists for, and it is the reason the bootstrap Gitea is kept.
|
||
|
||
**Push target.** `origin` is that same in-cluster Gitea, so `git push origin main` *is* the deploy.
|
||
There is no separate "apply" step and no GitHub remote.
|