Initial baseline: home platform (k3s + Longhorn) specification repository

Greenfield spec set for a GitOps-driven PaaS+IaaS home platform: vision &
requirements (R-* IDs), architecture, 14 domain specs, 11 ADRs, roadmap
(phases + workstreams), and AI-agent coordination (roles, protocol,
definition-of-done, task board).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

agents/coordination-protocol.md

@@ -0,0 +1,69 @@
+# Coordination Protocol
+
+How agents work together without stepping on each other. Read alongside [`roles.md`](roles.md),
+[`../roadmap/workstreams.md`](../roadmap/workstreams.md), and [`../AGENTS.md`](../AGENTS.md).
+
+## 1. Claim before you build
+- Take an item from [`task-board.md`](task-board.md), set its owner and status to `in-progress`.
+- Respect `blocked-by`. Do not start blocked work; help unblock it instead.
+- Keep items small and single-responsibility.
+
+## 2. Contracts are sacred
+Workstreams communicate through **interfaces** (listed in [`../roadmap/workstreams.md`](../roadmap/workstreams.md)):
+the ServiceDefinition schema, StorageClasses, the network policy/egress rendering, TLS issuer +
+Gateway, the observability contract, the update-policy contract.
+
+- **Consume** a contract freely.
+- **Change** a contract only by: (1) proposing the change on the task board with the affected
+  workstreams tagged, (2) getting the owning role's sign-off, (3) updating the spec that defines the
+  contract in the *same* change. Never silently break a downstream consumer.
+- The **ServiceDefinition schema** is the highest-traffic contract; treat schema changes as
+  platform-wide events.
+
+## 3. Decisions vs. discussion — and the no-spec-change rule
+- **You may not modify the specifications** (`docs/`, requirements, accepted ADRs, roadmap) — they
+  are human-owned and immutable to agents (R-TEST-06, [`../AGENTS.md` §11](../AGENTS.md)).
+- Anything that changes a locked choice ⇒ **draft** a superseding ADR and **escalate to the human**
+  ([`../decisions/README.md`](../decisions/README.md)); do not self-approve or merge it.
+- Genuine unknowns, or a spec that is wrong/blocking/ambiguous ⇒ **stop and escalate** on the task
+  board as an **Open Question (OQ-n)** for the human to resolve in the spec; don't bury ambiguity in
+  an implementation and don't patch the spec yourself.
+
+## 4. Definition of done
+Nothing is "done" until its **automated, reproducible behavior test passes green** in **dev/test**,
+committed with the change and tagged to its requirement IDs ([`definition-of-done.md`](definition-of-done.md),
+[`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md)). "Merged" ≠ "done";
+"demonstrated manually" ≠ "done". You write the test as well as the implementation.
+
+## 5. Phase discipline
+- Only work items in the **active phase(s)** ([`../roadmap/phases.md`](../roadmap/phases.md)) unless
+  explicitly parallelizable.
+- Respect entry gates; don't start a phase whose predecessor's exit criteria aren't demonstrated.
+- **Prod (Phase 7) is closed** until the operator opens it.
+
+## 6. Handoffs
+When your output feeds another workstream:
+- State the interface you're delivering and where it's documented.
+- Provide the demonstrable proof (the gate you passed) so the consumer can trust it.
+- If you changed a shared contract, announce it on the board with consumers tagged.
+
+## 7. Conflict resolution
+1. Re-read the source-of-truth hierarchy ([`../AGENTS.md` §2](../AGENTS.md)); the higher document
+   wins.
+2. If specs genuinely conflict or are silent, escalate to the **Platform Architect** to amend the
+   spec/ADR — fix the documents first, then build.
+3. Never resolve a conflict by silently picking an interpretation in code.
+
+## 8. Safety rails that never bend
+- Default-deny + VPN kill-switch are **fail-closed** — if unsure, deny/close.
+- Secrets stay encrypted; never printed.
+- Nothing that risks data ships without a tested backup/restore behind it.
+- No click-ops except ADR-sanctioned bootstrap steps.
+- **Specs are immutable to you**; a spec you can't meet is an escalation, never an edit.
+- **Never weaken a test or a requirement to get green** — that's a defect, not progress.
+
+## 9. Communication artifacts
+- **task-board.md** — the live who's-doing-what + status + blockers.
+- **specs/ADRs** — durable decisions and contracts.
+- **PR descriptions** — reference requirement IDs, specs, ADRs, and the gate demonstrated.
+Keep these three in sync; they are how independent agents stay coherent.

agents/definition-of-done.md

@@ -0,0 +1,84 @@
+# Definition of Done
+
+A change is **done** only when an **automated, reproducible, behavior-based test** for each relevant
+gate below **passes green in dev/test**, the test is committed with the change and tagged with the
+requirement IDs it verifies, and the change is traced to those IDs. A manual demonstration, a
+screenshot, or "it reconciles" is **not** done (R-TEST-01..04, [`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md)).
+
+> **Governance (R-TEST-06):** you may not change the specifications to reach done. If the spec can't
+> be met, that is an **escalation to the human**, not a spec/requirement/test-intent edit. Weakening
+> a test so it passes is a defect, not a completion.
+
+## Universal checklist (every change)
+- [ ] Traced to one or more requirement IDs (`R-*`) and the spec/ADR it implements.
+- [ ] An **automated behavior test** (Given/When/Then) exists, is committed with the change, tagged
+      with its `R-*` IDs, and **passes**; it appears in the coverage report.
+- [ ] The test is **reproducible**: single-command, hermetic (self-provisioning + self-cleaning),
+      deterministic, and yields identical results on re-run.
+- [ ] Expressed as GitOps (in git, reconciled by Flux) — no click-ops except ADR-sanctioned bootstrap.
+- [ ] No plaintext secrets anywhere; secrets are SOPS-encrypted (and no secret printed in output).
+- [ ] Authored once with environment-scoped overrides (no per-env base fork).
+- [ ] Observability wired (metrics/logs) and at least the relevant alert exists.
+- [ ] No specification/ADR/requirement was modified to achieve this (R-TEST-06); any spec problem was
+      escalated, not edited.
+- [ ] Result recorded on the task board / PR with a link to the passing test run.
+
+## Requirement-class gates (apply those relevant to the change)
+
+> Each checkbox below is the **intent of an automated behavior test** — implement it as a
+> reproducible, requirement-tagged test (see the behavior catalog in
+> [`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md)), not a manual
+> check. Safety-critical gates (marked *fault-injection*) must break the system and assert
+> fail-closed / no-data-loss (R-TEST-05).
+
+### State & rebuild (R-STATE-*)
+- [ ] Destroy the workload entirely and re-apply from git; **data survives**.
+- [ ] Stateful data is on `longhorn` (replicated) or the `bulk` tier — never ephemeral/hostPath.
+- [ ] DR-critical data restores from an **off-cluster** backup (drill for at least one service/phase).
+
+### Network isolation & egress (R-NET-*)
+- [ ] With no explicit allows, the workload can reach **only DNS** (default-deny proven via Hubble +
+      probe).
+- [ ] Declared `dependencies` work; an **undeclared** destination is blocked (negative test).
+- [ ] `egress: internet` leaves via the home/WAN IP; `egress: vpn` leaves via the **VPN IP**.
+- [ ] **VPN kill-switch:** tunnel forced down ⇒ the `vpn` workload leaks nothing; `internet`
+      workloads unaffected.
+- [ ] Changing `egress` class in git re-routes with no workload rewrite.
+
+### TLS (R-TLS-*)
+- [ ] Endpoint serves a **valid, non-self-signed, unexpired** cert (external check), including for
+      `internal` services.
+- [ ] Renewal is automatic (observed or verified via cert-manager) with no manual step.
+
+### DNS & ingress (R-DNS-*)
+- [ ] `<service>.home.keithjackson.dev` record appears automatically and routes correctly.
+- [ ] `internal` is unreachable publicly; `public` is reachable + TLS-valid.
+- [ ] Internal clients resolve to the internal Gateway (no public hairpin).
+
+### Automated updates (R-UPD-*)
+- [ ] Advances to a newer **stable** version automatically on schedule with **no human/AI edit**.
+- [ ] A deliberately bad update is health-gated: **auto-rollback** + alert.
+- [ ] A **pinned** service does not advance.
+
+### Observability & alerting (R-OBS-*)
+- [ ] Metrics + logs flow; a meaningful alert **fires** in a fault test and is delivered by **email**.
+- [ ] Dashboard renders and is linked from the C2 UI.
+
+### VM/IaaS (R-PLAT-03)
+- [ ] VM created/started/stopped/destroyed via `kubectl`/`virtctl` (same K8s surface).
+- [ ] VM disk is Longhorn-backed; rebuild preserves data; default-deny + egress class apply; guest
+      auto-updates.
+
+### Security & auditability (R-SEC-*)
+- [ ] Least-privilege RBAC (no broad cluster rights for workloads).
+- [ ] The auditability proofs (isolation / egress path / kill-switch / TLS / backup) are producible.
+
+### Per-service onboarding (WS-APPS)
+- [ ] The service is **fully wired from its ServiceDefinition alone** (compute/storage/policy/egress/
+      TLS/DNS/monitoring/update) — no manual steps beyond merge.
+
+## Phase exit
+A phase is done when the **full conformance suite for that phase's requirements passes green**
+(R-TEST-03) and the exit criteria in [`../roadmap/phases.md`](../roadmap/phases.md) are recorded with
+links to the passing runs. Prod (Phase 7) additionally requires a successful **automated** prod DR
+drill.

agents/roles.md

@@ -0,0 +1,56 @@
+# Agent Roles
+
+Each role owns a workstream ([`../roadmap/workstreams.md`](../roadmap/workstreams.md)) and is
+accountable for that workstream's requirement IDs and acceptance gates. Roles are responsibilities,
+not people — one agent may hold several early on; split as work scales.
+
+## Coordination roles
+
+### Platform Architect (steward of this spec repo)
+- Guards the source-of-truth hierarchy ([`../AGENTS.md` §2](../AGENTS.md)); approves/records ADRs.
+- Resolves cross-workstream interface conflicts; keeps requirement traceability honest.
+- Does **not** write implementation code; owns specs, ADRs, roadmap, and this coordination layer.
+
+### Integration/Release Lead
+- Owns the implementation repo structure, CI gates, and Flux bootstrap health.
+- Runs phase entry/exit reviews against the **automated conformance suite**; keeps the task board
+  truthful.
+
+### QA/Conformance Engineer (WS-TEST)
+- Owns the conformance testing capability ([14](../docs/14-testing-and-conformance.md),
+  [ADR-0010](../decisions/adr-0010-conformance-testing.md)): the harness, the reproducibility
+  contract, the requirement-coverage report, and the CI/phase gates.
+- Does **not** own writing every test in isolation — **each workstream authors the behavior tests
+  for its own requirements**; WS-TEST provides the framework, patterns, fault-injection harness, and
+  gate, and guards that tests assert real behavior and are never weakened to pass (R-TEST-05/06).
+- Accountable for: every `R-*` requirement having a passing, reproducible, tagged test.
+
+## Implementation roles (one per workstream)
+
+| Role | Workstream | Primary specs | Accountable for (gates) |
+|---|---|---|---|
+| **Infra/Cluster Engineer** | WS-INFRA | [02](../docs/02-environments-and-provisioning.md) | Reproducible node provisioning; k3s HA + Cilium; node rebuild w/o data loss |
+| **GitOps/Platform Engineer** | WS-GITOPS | [05](../docs/05-gitops-and-delivery.md), [13](../docs/13-service-catalog.md) | Reconcile-from-git; drift correction; ServiceDefinition renderer; CI |
+| **Network Engineer** | WS-NET | [03](../docs/03-networking-isolation-and-egress.md), [09](../docs/09-dns-and-ingress.md) | Default-deny; explicit allows; egress classes; VPN kill-switch; ingress/DNS |
+| **Storage Engineer** | WS-STORAGE | [04](../docs/04-storage-and-state.md) | Replicated Longhorn; bulk tier; backups; **tested restore/DR** |
+| **Security Engineer** | WS-SEC | [07](../docs/07-security-and-secrets.md), [08](../docs/08-tls-and-certificates.md) | SOPS secrets; trusted auto-TLS; least-privilege; auditability proofs |
+| **Observability Engineer** | WS-OBS | [10](../docs/10-observability-and-alerting.md) | Metrics/logs; email alerts; dashboards; platform-health alerts |
+| **Automation Engineer** | WS-UPDATE | [11](../docs/11-automated-updates.md) | Hands-off stable updates; health-gate + auto-rollback; pin |
+| **Virtualization Engineer** | WS-VM | [06](../docs/06-vm-workloads-kubevirt.md) | KubeVirt IaaS via same K8s surface; VM rebuild w/o data loss |
+| **Application Onboarding Engineer** | WS-APPS | [13](../docs/13-service-catalog.md) | Each service wired from its ServiceDefinition; deps correct; nothing undeclared |
+| **C2/UX Engineer** | WS-C2 | [12](../docs/12-command-center-ui.md) | Root C2 UI; live catalog + health; safe, audited actions |
+
+## Standing expectations for every role
+- Trace all work to requirement IDs; reference specs/ADRs by ID (stable across file moves).
+- **Author an automated, reproducible behavior test** for every requirement you implement; "done" is
+  a green, tagged test, not "merged" ([definition-of-done.md](definition-of-done.md),
+  [14](../docs/14-testing-and-conformance.md)).
+- **Never modify the specifications** (`docs/`, requirements, accepted ADRs, roadmap) — they are
+  human-owned (R-TEST-06). A spec you can't meet is an **escalation**, not an edit. Never weaken a
+  test/requirement to get green.
+- Never re-litigate an accepted ADR; **draft + escalate** a superseding ADR for human approval
+  ([`../decisions/README.md`](../decisions/README.md)).
+- Negotiate interface changes before breaking another workstream's contract
+  ([coordination-protocol.md](coordination-protocol.md)).
+- Never build on prod hardware before Phase 7 is opened.
+- Never emit plaintext secrets.

agents/task-board.md

@@ -0,0 +1,100 @@
+# Task Board
+
+The live who's-doing-what. Agents **claim** an item (set Owner + Status) before starting, respect
+`Blocked-by`, and mark `done` only after its **automated, reproducible behavior test passes green**
+([`definition-of-done.md`](definition-of-done.md), [`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md)).
+Seeded from [`../roadmap/phases.md`](../roadmap/phases.md) + [`../roadmap/workstreams.md`](../roadmap/workstreams.md).
+
+> **This board is the one working surface agents may edit** (claim items, update status, add
+> items/escalations). The specs, ADRs, and roadmap are **not** editable by agents (R-TEST-06) —
+> raise spec problems here as an `OQ-*` escalation for the human. Every "**Prove …**" task below is
+> an automated conformance test (co-owned with **WS-TEST**), not a manual demo.
+
+**Status values:** `todo` · `in-progress` · `blocked` · `review` · `done`
+**Add new items** at the bottom of the relevant phase; keep IDs stable.
+
+---
+
+## Phase 0 — Foundations
+| ID | Task | WS | Serves | Blocked-by | Owner | Status |
+|---|---|---|---|---|---|---|
+| T-0.1 | Create implementation repo structure (platform/apps/catalog/clusters) | WS-GITOPS | R-GITOPS-02 | — | — | todo |
+| T-0.2 | CI: lint + schema validation + build/dry-run gates | WS-GITOPS | R-GITOPS-01 | T-0.1 | — | todo |
+| T-0.3 | Author ServiceDefinition schema + renderer contract | WS-GITOPS | R-GITOPS-02 | T-0.1 | — | todo |
+| T-0.4 | Conformance harness: **Chainsaw** runner + reproducibility contract + JUnit + coverage report + CI merge gate | WS-TEST | R-TEST-01/02/03/04 | T-0.1 | — | todo |
+| T-0.5 | L0 **Kyverno** policy gate (default-deny present, no plaintext secrets, schema-valid ServiceDefinitions) | WS-TEST | R-TEST-01, R-NET-01, R-SEC-01 | T-0.4 | — | todo |
+
+## Phase 1 — dev/test Cluster Bring-up
+| ID | Task | WS | Serves | Blocked-by | Owner | Status |
+|---|---|---|---|---|---|---|
+| T-1.1 | OpenTofu libvirt module for 3 node VMs (+ Longhorn data disk) | WS-INFRA | R-ENV-01 | — | — | todo |
+| T-1.2 | cloud-init baseline (users, SSH, sysctls, virt/Longhorn prereqs) | WS-INFRA | R-ENV-01 | T-1.1 | — | todo |
+| T-1.3 | Ansible: k3s HA install, Cilium CNI, disable flannel/kube-proxy/Traefik | WS-INFRA | R-PLAT-01 | T-1.2 | — | todo |
+| T-1.4 | Bootstrap Flux + SOPS age key | WS-GITOPS/WS-SEC | R-GITOPS-01, R-SEC-01 | T-1.3 | — | todo |
+| T-1.5 | Prove node destroy/rebuild from IaC (no data loss) | WS-INFRA | R-ENV-01, R-STATE-01 | T-1.4 | — | todo |
+
+## Phase 2 — Platform Core (net/storage/TLS/DNS/ingress)
+| ID | Task | WS | Serves | Blocked-by | Owner | Status |
+|---|---|---|---|---|---|---|
+| T-2.1 | Longhorn (3-replica StorageClass) + bulk-tier plan (OQ-2) | WS-STORAGE | R-STATE-02/04 | T-1.4 | — | todo |
+| T-2.2 | Default-deny network baseline (per-namespace) + Hubble | WS-NET | R-NET-01/03 | T-1.3 | — | todo |
+| T-2.3 | Render `dependencies` → CiliumNetworkPolicy from ServiceDefinition | WS-NET/WS-GITOPS | R-NET-02 | T-0.3, T-2.2 | — | todo |
+| T-2.4 | cert-manager + Let's Encrypt DNS-01 wildcard `*.home.keithjackson.dev` | WS-SEC | R-TLS-01/02/03 | T-1.4 | — | todo |
+| T-2.5 | ExternalDNS + Gateway API listeners + split-horizon (PiHole) foundation | WS-NET | R-DNS-01/02/03 | T-2.4 | — | todo |
+| T-2.6 | Prove: default-deny (DNS-only probe), PVC survives rebuild, trusted cert, auto-DNS | WS-NET/STORAGE/SEC | R-NET/STATE/TLS/DNS | T-2.1..2.5 | — | todo |
+| T-2.7 | Deploy **Kyverno in-cluster as admission enforcer** (same policies as the CI L0 gate) | WS-TEST/WS-SEC | R-SEC-02, R-NET-01, R-SEC-01 | T-1.4, T-0.5 | — | todo |
+
+## Phase 3 — Egress, VPN & Multi-WAN
+| ID | Task | WS | Serves | Blocked-by | Owner | Status |
+|---|---|---|---|---|---|---|
+| T-3.1 | Egress classes; Cilium Egress Gateway wiring | WS-NET | R-NET-04/05 | T-2.2 | — | todo |
+| T-3.2 | VPN egress (Gluetun sidecar) + kill-switch (ADR-0008) | WS-NET | R-NET-05/06 | T-3.1, OQ-3 | — | todo |
+| T-3.3 | Edge-router multi-WAN split for `internet` class (OQ-1) | WS-NET | R-NET-04 | OQ-1 | — | todo |
+| T-3.0 | L3 fault-injection harness — **in-cluster Chainsaw Job** (CI-triggered, scoped test SA) using `script`/`try`/`finally` (force tunnel down, kill node, inject bad image, trigger restore) | WS-TEST | R-TEST-05, R-SEC-02 | T-0.4 | — | todo |
+| T-3.4 | Prove: VPN IP egress, kill-switch fail-closed, egress-class flip via git | WS-NET | R-NET-05/06/07 | T-3.0, T-3.2 | — | todo |
+
+## Phase 4 — Observability, Backup/DR & Updates
+| ID | Task | WS | Serves | Blocked-by | Owner | Status |
+|---|---|---|---|---|---|---|
+| T-4.1 | kube-prometheus-stack + Loki + Alloy + Alertmanager(email) | WS-OBS | R-OBS-01/02/03 | T-2.1 | — | todo |
+| T-4.2 | Platform-health alerts (node/Longhorn/Flux/cert/VPN/backup) | WS-OBS | R-OBS-04 | T-4.1 | — | todo |
+| T-4.3 | Velero + Longhorn off-cluster backup schedule | WS-STORAGE | R-STATE-03 | T-2.1, OQ-2 | — | todo |
+| T-4.4 | DR drill: restore namespace + volumes from off-cluster backup | WS-STORAGE | R-STATE-03 | T-4.3 | — | todo |
+| T-4.5 | Flux image-automation + Renovate + per-service update policy | WS-UPDATE | R-UPD-01/02/03 | T-0.3, T-4.1 | — | todo |
+| T-4.6 | Prove: auto stable update on schedule, bad-update auto-rollback+alert, pin holds | WS-UPDATE | R-UPD-* | T-4.5 | — | todo |
+
+## Phase 5 — IaaS Proof (KubeVirt)
+| ID | Task | WS | Serves | Blocked-by | Owner | Status |
+|---|---|---|---|---|---|---|
+| T-5.1 | Install KubeVirt + CDI; Longhorn-backed VM disks | WS-VM | R-PLAT-03 | T-2.1, T-2.2 | — | todo |
+| T-5.2 | `kind: vm` ServiceDefinition path; run one representative VM | WS-VM/WS-GITOPS | R-PLAT-03/04 | T-5.1, T-0.3 | — | todo |
+| T-5.3 | Prove: VM lifecycle via kubectl/virtctl, rebuild w/o data loss, default-deny+egress, guest auto-update | WS-VM | R-PLAT-03, R-NET-*, R-UPD-* | T-5.2 | — | todo |
+
+## Phase 6 — Service Onboarding + C2
+| ID | Task | WS | Serves | Blocked-by | Owner | Status |
+|---|---|---|---|---|---|---|
+| T-6.1 | Onboard **pihole** (DNS underpinning) | WS-APPS | R-DNS-03 | T-2.5 | — | todo |
+| T-6.2 | Onboard **plex** (bulk tier + public exposure) | WS-APPS | R-STATE-04 | T-2.6, T-6.1 | — | todo |
+| T-6.3 | Onboard **qbittorrent, jackett** (VPN + kill-switch) | WS-APPS | R-NET-05/06 | T-3.4 | — | todo |
+| T-6.4 | Onboard **sonarr, radarr** (VPN + arr↔downloader allows) | WS-APPS | R-NET-02/05 | T-6.3 | — | todo |
+| T-6.5 | Onboard **seerr** (allows to arr + plex) | WS-APPS | R-NET-02 | T-6.2, T-6.4 | — | todo |
+| T-6.6 | Onboard **nextcloud, gitea** (DR-critical + backups) | WS-APPS | R-STATE-03 | T-4.4 | — | todo |
+| T-6.7 | Onboard **minecraft, unifi, semaphore, kasm** | WS-APPS | app reqs | T-2.6 | — | todo |
+| T-6.8 | C2 UI: Homepage root + Grafana + Headlamp + Flux status + auth | WS-C2 | R-C2-01/02 | T-4.2, T-6.* | — | todo |
+
+## Phase 7 — Production Readiness (CLOSED until operator opens it)
+| ID | Task | WS | Serves | Blocked-by | Owner | Status |
+|---|---|---|---|---|---|---|
+| T-7.1 | Bare-metal provisioning checklist (design-only) | WS-INFRA | R-ENV-03 | — | — | todo |
+| T-7.2 | prod overlay (no base edits) + cutover plan | WS-INFRA/WS-GITOPS | R-ENV-02 | Phases 1–6 done | — | blocked |
+| T-7.3 | Full prod DR drill | WS-STORAGE | R-STATE-03 | T-7.2 | — | blocked |
+
+---
+
+## Open questions (resolve to unblock)
+| OQ | Question | Blocks | Owner |
+|---|---|---|---|
+| OQ-1 | WAN uplink count + edge-router product | T-3.3 | — |
+| OQ-2 | Bulk-media home + backup posture | T-2.1, T-4.3 | — |
+| OQ-3 | VPN provider/protocol + concurrent-tunnel limits | T-3.2 | — |
+| OQ-4 | DNS provider API (Cloudflare) token for DNS-01 + ExternalDNS | T-2.4, T-2.5 | — |