Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity
Files
docs/pm-agent-registry-pull-prereq
home-network-k3s-impl/infra/PHASE1-STATUS.md
T
jakman85andClaude Opus 5 9739e9be91 chore(backup): R2 credentials for the off-cluster DR target 
Written by platform/backup/set-r2-credentials.sh (operator, 2026-09-07T05:41:34Z)
into all four consumers, in the two shapes they need: velero wants an AWS INI
file, CNPG/barman wants discrete ACCESS_KEY_ID / ACCESS_SECRET_KEY. All four
verified to carry the SAME 32-character key id, so no consumer was missed.

Proven from a pod INSIDE prod against
https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com:
head-bucket on k3s-prod-backup returns BucketRegion WNAM, and PUT and DELETE both
succeed. Byte-integrity of GET is NOT yet proven (the aws-cli image has no cmp);
s3check is what will establish it.

Credentials only — no endpoint or bucket wiring. That is feat/r2-backup-prod.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 00:58:45 -05:00

156 lines
9.3 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Dev cluster — current state & reconnect guide
**Last verified: 2026-08-31.** Phase 1 is **complete and long since exited**; the cluster has been
running the full platform (phases 2–6) for some time. This doc is how a session — human or agent —
reconnects to it and re-establishes the toolchain.
> **New PM / agent taking over?** Read this doc (reconnect + toolchain), then the **latest
> `DECISIONS-LOG.md` entry (2026-08-31)** — it is the current-state + gotchas + prioritized **Next
> Steps** handoff — and the plans under `docs/plans/`. Start at Next Steps.
> **Phase 7 (prod) is OPEN as of 2026-08-31 — prep only.** The first prod node
> (`k3s-prod-1`, `192.168.2.10`) is on the bench. This document covers the **dev** cluster;
> for prod see [`PROD-PROVISIONING.md`](PROD-PROVISIONING.md) (T-7.1 checklist) and the
> Phase-7 entry in `DECISIONS-LOG.md`. Nothing is installed on prod hardware until the
> R-ENV-02 tokenization blockers close and the gate is green.
> **History note.** The cluster was originally 3 libvirt/KVM VMs on the workstation
> (`10.73.0.0/24` NAT). That was **retired** after the workstation ran out of RAM/disk; the cluster
> now lives on **Proxmox VE**. Anything mentioning `virsh`, `qemu:///system`, `tofu`+`dmacvicar/libvirt`,
> or `10.73.0.x` for the cluster is historical — including [`cluster.sh`](cluster.sh) (retired) and
> [`PHASE1-SCOPE.md`](PHASE1-SCOPE.md) (historical scoping doc).
## The cluster
| | |
|---|---|
| Substrate | Proxmox VE host, node `proxmox`, disks on the `Storage` ZFS datastore, bridge `vmbr0` |
| Nodes | `k3s-dev-1/2/3` at `192.168.1.50 / .51 / .52` — 8 vCPU, 24 GiB, Debian 13 (trixie) |
| k3s | **v1.36.4+k3s1**, 3 servers, embedded etcd (HA); no flannel / kube-proxy / Traefik |
| CNI | Cilium 1.20.1 (kube-proxy replacement, L2 announcements, egress gateway) |
| kube-API | kube-vip VIP **`192.168.1.49:6443`**, tls-san'd on the serving cert |
| Ingress VIP | **`192.168.1.53`** (shared Gateway). Pool `192.168.1.53–.60` for LB VIPs; UniFi holds `.54` |
| Storage | Longhorn 1.12.1 — classes `longhorn` (default), `longhorn-static`, `longhorn-vm` (RWX/Block, migratable), `longhorn-scratch`. `bulk` (NFS CSI) is a **non-provisioning stub** pending OQ-2 |
| Nested virt | `cpu_type = host` → real KVM in-guest; KubeVirt + CDI run on it |
Everything above is provisioned by IaC: [`tofu/`](tofu/) creates the VMs,
[`cloud-init/`](cloud-init/) baselines first boot, [`ansible/`](ansible/) installs k3s, Cilium,
kube-vip and the VPN egress gateway. Flux then reconciles the whole platform from `clusters/dev`.
## Network path from the workstation (there is a NAT in the way)
The workstation is **not** on the cluster LAN. A pfSense NAT bridges the two:
| Purpose | Workstation-side target | Forwards to |
|---|---|---|
| kube-API | `https://192.168.2.50:6443` | kube-vip VIP `192.168.1.49:6443` |
| Node SSH (Ansible only) | `192.168.2.102:2250 / 2251 / 2252` | `192.168.1.50 / .51 / .52` port 22 |
| Proxmox API | `https://192.168.2.102:8006/` | the Proxmox host |
TLS through the NAT validates because the kubeconfig sets `tls-server-name: 192.168.1.49` — the
VIP is in the serving cert's SAN list, so the NAT address does not have to be.
There is **no SSH tunnel** any more; the old tunnel was retired when kube-vip + the NAT landed.
## Access
**kubectl — Keycloak SSO (normal path).** `~/.kube/config` context **`k3s-oidc`** authenticates
against the `homelab` realm via `kubelogin` (auth-code + PKCE, public client, no secret in the
kubeconfig). Group membership maps to RBAC by `platform/keycloak/k8s-oidc-rbac.yaml`:
```
kubectl config use-context k3s-oidc
kubectl auth whoami # -> oidc:<you>, groups [oidc:platform-admins ...]
```
`oidc:platform-admins` → `cluster-admin`, `oidc:viewers` → `view`. First use opens a browser, so
this path needs an interactive desktop.
**kubectl — cert kubeconfig (break-glass).** The k3s admin cert kubeconfig from
`/etc/rancher/k3s/k3s.yaml` on `k3s-dev-1` (rewrite the server to `https://192.168.2.50:6443` and
add `tls-server-name: 192.168.1.49`). Fully independent of Keycloak — **use this when OIDC, the
gateway, or Keycloak itself is down.** Headless contexts (CI jobs, containers, agents) also need
this, since they cannot complete a browser login.
**Node SSH.** `ssh -i ~/.ssh/home_network -p 2250 [email protected]` (user `k3s`, NOPASSWD sudo).
Ports 2251/2252 for nodes 2/3.
## Toolchain — use the dev container, not the host
Every pinned tool (tofu, ansible, sops, age, flux, kubectl, helm **and** the full CI gate
toolchain: yamllint, check-jsonschema, kyverno, chainsaw, yq) lives in the image:
```bash
infra/devcontainer/dev.sh build # once
infra/devcontainer/dev.sh run make -C ci gate # exactly what CI runs
infra/devcontainer/dev.sh # interactive shell at /work
infra/devcontainer/dev.sh run kubectl get nodes # DEV (the default)
CLUSTER=prod infra/devcontainer/dev.sh run kubectl get nodes # PROD
```
Unlike the libvirt era, **tofu and ansible now run fine inside the container** — the
`bpg/proxmox` provider talks to the Proxmox API over the network and needs no host libvirt socket,
which is what broke the old `dmacvicar/libvirt` provider in a container.
`dev.sh` mounts the repo at `/work`, `~/.ssh` (read-only), `~/.config/sops` (the age keyring —
**persisted on the host, or you lose access to every SOPS secret**), the ansible-vault passphrase
file, and the sibling spec repo read-only at `/home-network-k3s` so the spec-immutability
check runs instead of silently skipping.
> **Closed (was a known gap): `dev.sh` can now target either cluster.** It used to hard-wire
> `KUBECONFIG` to `dev.kubeconfig`, leaving an undocumented — and misleadingly named —
> `DEV_KUBECONFIG=<prod path>` as the only route to prod. Select with **`CLUSTER=dev|prod`**;
> **dev remains the default**, so every existing invocation means what it always meant. Inside the
> container `KUBECONFIG` is a chain of a tiny per-invocation selector file over a merged config
> carrying both clusters as contexts **`k3s-dev`** and **`k3s-prod`** (never `default`), regenerated
> each run by [`infra/devcontainer/kubeconfig.sh`](devcontainer/kubeconfig.sh) into the gitignored
> `infra/ansible/.cache/clusters.kubeconfig`. The merged file deliberately carries **no**
> `current-context`: it is shared by every container, and stamping the current cluster into it made
> concurrent `CLUSTER=dev` and `CLUSTER=prod` runs silently swap clusters mid-command. Naming matters: k3s writes *both* source kubeconfigs
> with context, cluster and user all called `default`, and kubectl's merge keeps the first
> duplicate — so a plain `KUBECONFIG=dev:prod` would silently resolve `default` to **dev** while you
> believed you were on prod. Requesting a cluster whose kubeconfig is absent is a hard error, never
> a fallback. `DEV_KUBECONFIG` still works and still wins, but is **superseded** — it is opaque
> about its target and yields an ambiguous `default` context. `~/.kube` is still not mounted and the
> OIDC context still cannot work in a container (no browser); use the break-glass cert kubeconfigs
> above. Details: [`devcontainer/README.md`](devcontainer/README.md).
## Off-repo state you must not lose
| What | Where |
|---|---|
| SOPS **age private key** | `~/.config/sops/age/keys.txt` — losing it locks you out of every encrypted secret |
| Ansible vault passphrase | `~/.config/home-k3s/ansible-vault-pass` |
| Node SSH key | `~/.ssh/home_network` |
| Proxmox API token | operator env only (`PROXMOX_VE_API_TOKEN`) — never a tofu var, never in state |
| tofu state | `infra/tofu/terraform.tfstate` (gitignored) — **absent on a fresh workstation**; a `tofu apply` without it would try to recreate the live nodes |
| Generated SSO secrets | `~/.config/home-k3s/keycloak-sso-secrets.txt` |
## Health check
```bash
kubectl get nodes # 3/3 Ready
flux get kustomizations # flux-system, platform, platform-config,
# apps-wave-0/1/2, apps-vms — all Ready
kubectl get helmreleases -A # all Ready
kubectl get cluster -A # CNPG: keycloak-db, nextcloud-db, semaphore-db
kubectl get vmi -A # unifi Running
```
## Flux's git source is inside the cluster — mind the loop
`GitRepository/flux-system` points at **`https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`**:
the **in-cluster** Gitea, which is itself an app Flux deploys in `apps/wave-0`. Resolution goes
through split-horizon CoreDNS → Gateway VIP `192.168.1.53`, so a Flux reconcile depends on Gitea,
the Gateway, CoreDNS and the wildcard cert all being up.
In practice this is stable — Gitea's Deployment and PVC survive a reboot and come back without
Flux's help, so a Proxmox host reboot self-heals (VMs auto-start, cluster reconverges). The loop
only bites if Gitea is **lost** rather than restarted. The recovery path is the host bootstrap
Gitea (`bash bootstrap/gitea/bootstrap-gitea.sh`) plus repointing the GitRepository at it — that
is what the `bootstrap/` tree exists for, and it is the reason the bootstrap Gitea is kept.
**Push target.** `origin` is that same in-cluster Gitea, so `git push origin main` *is* the deploy.
There is no separate "apply" step and no GitHub remote.
Reference in new issue View Git Blame Copy Permalink