jakman85andClaude Opus 5
f92707f38c
Lets the operator's Android / Linux / Windows / macOS devices dial in from anywhere and behave like LAN machines: full reach to 192.168.2.0/24 plus internet egress out the house WAN. Opposite direction to platform/vpn-egress, which sends pods OUT through PIA; the two share nothing but the word "VPN". WireGuard because it is the only option with a first-party client on all four named platforms, and because its handshake MAC is keyed with the responder's own public key: an unauthenticated packet gets NO reply, so udp/51820 is indistinguishable from a closed port. That, not "it is encrypted", is what makes an unfiltered WAN port defensible, and why there is no source allowlist. hostNetwork: true is REQUIRED, not convenient. A pod-networked server must SNAT clients to its own pod IP, so every remote client would carry that pod's Cilium identity -- and a pod-sourced request to the shared Gateway VIP returns 403 from Envoy for ALL 16 hostnames (measured 2026-10-04), because Cilium's Gateway preserves the original source identity and enforces the destination's ingress policy. The obvious build would have shipped a healthy-looking tunnel on which every cluster web UI was 403 and *.home.keithjackson.dev did not even resolve. In the host netns traffic is SNATed to the node and arrives as host/remote-node, exactly like a LAN client. It also sidesteps the unsafe ip_forward sysctl, the L2-announcement lease lottery, and double encapsulation (hence MTU 1420). Permitted by restrict-host-namespaces' own carve-out for platform namespaces. The cost is stated in networkpolicy.yaml rather than papered over: no NetworkPolicy constrains a hostNetwork pod. A DaemonSet so the hand-made WAN forward cannot be invalidated by a reschedule; all three nodes answer the same keys, so failover is a one-line router change. Peer PUBLIC keys are plaintext in peers.yaml, which makes "who can get into my house" a reviewable PR diff. The only secret -- the server's private key -- is an out-of-band Secret: an agent must not mint a long-lived private key, and the two halves are joined only at pod start into a tmpfs that dies with the pod. Ships INERT-BUT-HEALTHY with zero peers. THREE off-repo manual steps remain (kernel module, server key, WAN forward) -- runbooks/vpn-remote-access.md. Touches clusters/** , so ci/approval-policy.sh will block pending the operator's /approve-override. Deliberately not posted by me (CLAUDE.md §1). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
113 lines
8.1 KiB
Markdown
113 lines
8.1 KiB
Markdown
# home-network-k3s-impl
|
||
|
||
Implementation of the home k3s + Longhorn platform. This repo holds the **reconciled artifacts**
|
||
(Flux config, Kustomize overlays, KubeVirt/Helm manifests, ServiceDefinitions, Kyverno policies,
|
||
Chainsaw conformance tests, CI). It is driven by — and subordinate to — the **read-only
|
||
specification** at [`../home-network-k3s`](../home-network-k3s).
|
||
|
||
## Rules (from the spec — non-negotiable)
|
||
- **The spec is read-only.** Never modify anything under `../home-network-k3s`. Implement against it;
|
||
escalate spec problems, don't edit them (spec `AGENTS.md` §11, R-TEST-06).
|
||
- **Done = a green, reproducible, requirement-tagged conformance test** — never "merged," never a
|
||
manual demo (spec `docs/14`, `agents/definition-of-done.md`).
|
||
- **No work on `main`.** Every change is a feature branch merged into `main` (spec `CONTRIBUTING.md`).
|
||
- **No plaintext secrets** — SOPS-encrypted only (spec `docs/07`).
|
||
- **Phase discipline** — only build what the active roadmap phase authorizes (spec `roadmap/phases.md`).
|
||
|
||
## Layout
|
||
```
|
||
infra/ node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible, and the
|
||
pinned dev container that carries the whole toolchain
|
||
bootstrap/ one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
|
||
platform/ platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
|
||
longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
|
||
vpn-egress, vpn-ingress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
|
||
(vpn-egress = pods OUT via PIA; vpn-ingress = remote clients IN via WireGuard)
|
||
platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
|
||
(ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
|
||
Split out because CRs and the CRDs they use cannot apply in one pass.
|
||
apps/ application tier, reconciled as dependency-ordered WAVES:
|
||
wave-0/ wave-1/ wave-2/ rendered container apps (see DESIGN-waves.md)
|
||
vms/ KubeVirt VirtualMachines (non-gating tier)
|
||
catalog/ ServiceDefinition entries (the onboarding surface) + render.py
|
||
schemas/ ServiceDefinition JSON Schema + renderer contract
|
||
clusters/ Flux entrypoints + dev/ and prod/ overlays
|
||
conformance/ Chainsaw behavior/conformance suite + harness + coverage report
|
||
ci/ the gate as scripts (lint · schema · Kyverno L0 · L1 render · decision)
|
||
.gitea/ Gitea Actions CI running that gate on an in-cluster runner
|
||
```
|
||
|
||
## Status
|
||
|
||
Roadmap phases **0–6 are delivered**, and **Phase 7 (Production Readiness) is OPEN and in
|
||
progress**. `clusters/prod` is no longer a design-only overlay: prod is the live cluster, serving the
|
||
apex `home.keithjackson.dev`. The dev cluster was **decommissioned on 2026-09-12** (full migration:
|
||
dev out, prod onto the apex — see the DECISIONS-LOG), so phases 0–6 are now exercised on prod
|
||
hardware rather than on Proxmox VMs.
|
||
|
||
**Phase 7 exit criteria** — *prod runs the same config via the prod overlay; DR proven; monitoring +
|
||
alerts live*:
|
||
|
||
| criterion | state |
|
||
|---|---|
|
||
| prod runs the prod overlay | **met** — 3× mini-PC bare metal, one Flux tree, prod overlay only |
|
||
| monitoring + alerts live | **met 2026-09-13** — see the note below; this was not true before that date |
|
||
| **DR proven** | **R-STATE-03 PROVEN on prod 2026-09-14** — off-cluster restore from Cloudflare R2. `R-STATE-02` (node-loss) is still dev-only |
|
||
|
||
**DR restore is proven on prod.** `R-STATE-03` reads `proven` in `--status` as of 2026-09-14 —
|
||
the first requirement in this repo to reach that state. Both of its verifiers are recorded:
|
||
`dr-restore` (L3, run `20260914T021505Z-24b8111`, sentinel `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK`)
|
||
and `placeholder-guard-coupling` (L0). The L3 drill seeds a canary on the replicated `longhorn`
|
||
class, backs the namespace and volume data up off-cluster, **destroys every in-cluster copy**
|
||
(namespace, PVC and PV), restores into a clean namespace, and asserts the restored PVC binds a
|
||
*different* PersistentVolume — so an in-cluster survival cannot pass as a restore.
|
||
|
||
**What remains for Phase 7:** `R-STATE-02` (`node-loss`) is still proven only on the decommissioned
|
||
dev cluster. That drill takes a real node down on a live cluster, so it is a deliberate, scheduled
|
||
exercise rather than something to run in passing.
|
||
|
||
Read `--coverage-only` and `--status` as two different questions; the earlier `PROVEN: 0` was not a
|
||
bug but the honest answer, because proofs are claims about a specific tree on a specific cluster.
|
||
|
||
| Phase | Delivered |
|
||
|---|---|
|
||
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
|
||
| 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; **now 3× mini-PC bare metal** (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13) |
|
||
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the authoritative LAN resolver** for `${BASE_DOMAIN}` on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
|
||
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
|
||
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled |
|
||
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |
|
||
| 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · **kubectl OIDC SSO with group→RBAC** · in-cluster Gitea Actions merge gate |
|
||
| **7 Production readiness** | **IN PROGRESS** (DR restore proven 2026-09-14; node-loss outstanding). Prod live on bare metal serving the apex; dev decommissioned; CNPG HA 3 instances with primaries spread one per node; NFS bulk media + legacy Nextcloud mounted; VPN egress on prod. Alerting made real 2026-09-13 (Longhorn, cert-manager, Flux and twelve app SLIs had all been unable to fire). **Open: `node-loss` (R-STATE-02) re-proven on prod.** |
|
||
|
||
**Conformance: 51 / 51 requirements covered (100%)**, 0 gaps, 0 safety-critical gaps (closed
|
||
2026-09-13; `R-C2-01`/`R-C2-02` were the last two). Regenerate with
|
||
`./conformance/run.sh --coverage-only`; the full report is
|
||
`conformance/_artifacts/coverage/coverage.md`.
|
||
|
||
Coverage is **not** proof. A requirement is *covered* when an executable test exists for it, and
|
||
*PROVEN* only when a recorded run of that test is in `conformance/runs/ledger.jsonl` at the current
|
||
SHA on the cluster it claims. Read both numbers: `--coverage-only` for the first, `--status` for the
|
||
second.
|
||
|
||
The running narrative — decisions, root causes, incidents, follow-ups — is
|
||
[`DECISIONS-LOG.md`](DECISIONS-LOG.md). It, not this table, is the authoritative record.
|
||
|
||
## Working on this repo
|
||
|
||
Everything the gate needs is baked into the pinned dev container — **do not install the
|
||
toolchain on the host**:
|
||
|
||
```bash
|
||
infra/devcontainer/dev.sh build # once
|
||
infra/devcontainer/dev.sh run make -C ci gate # the same gate CI runs
|
||
infra/devcontainer/dev.sh # interactive shell (sops, flux, kubectl, tofu, ansible)
|
||
```
|
||
|
||
The container mounts the repo at `/work`, your age keyring, and the sibling spec repo read-only
|
||
so the spec-immutability check can actually run. See
|
||
[`infra/devcontainer/README.md`](infra/devcontainer/README.md).
|
||
|
||
A green run prints `MERGE GATE: PASS`. The L2/L3 conformance layers stay `SKIPPED` unless
|
||
`CLUSTER_AVAILABLE=true` — the static gate never pretends to have run them.
|