commit e8c555cfebdc6c059fb8667d09eecdcf73916657 Author: Keith Jackson Date: Sat Aug 15 22:37:56 2026 -0500 Initial baseline: home platform (k3s + Longhorn) specification repository Greenfield spec set for a GitOps-driven PaaS+IaaS home platform: vision & requirements (R-* IDs), architecture, 14 domain specs, 11 ADRs, roadmap (phases + workstreams), and AI-agent coordination (roles, protocol, definition-of-done, task board). Co-Authored-By: Claude Opus 4.8 (1M context) diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..2e2a514 --- /dev/null +++ b/.gitignore @@ -0,0 +1,30 @@ +# OS / editor cruft +.DS_Store +Thumbs.db +*.swp +*~ +.idea/ +.vscode/ + +# Secrets — never commit plaintext (secrets live SOPS-encrypted only; see docs/07) +*.dec.yaml +*.decrypted.* +*.key +*.pem +age.key +*.kubeconfig +kubeconfig + +# IaC state / local runtime (for the future implementation tree) +.terraform/ +*.tfstate +*.tfstate.* +.terraform.lock.hcl +*.tfvars +!*.example.tfvars + +# Test / build artifacts +/results/ +/reports/ +*.junit.xml +coverage/ diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..5c526e5 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,148 @@ +# AGENTS.md — Rules of Engagement + +This file governs how AI agents (and their human reviewers) work inside this repository. Read it +fully before making any change. These rules **override** any conflicting default behavior. + +> **Two governing constraints, above all others:** +> 1. **You may not change the specifications.** The `docs/` tree, the requirements, and accepted +> ADRs are human-owned and immutable to agents (R-TEST-06). You implement against them and prove +> conformance; you never edit them to fit your implementation. See §2 and §11. +> 2. **Conformance is proven only by automated, reproducible behavior tests.** "Done" means a green, +> requirement-tagged conformance test — never "merged," never a manual demo (R-TEST-01..05). +> See §7 and [`docs/14-testing-and-conformance.md`](docs/14-testing-and-conformance.md). + +## 1. This repository defines intent, not implementation (yet) + +Today this repo contains **specifications only**. Implementation artifacts (Helm values, Flux +Kustomizations, KubeVirt manifests, OpenTofu/Ansible IaC, etc.) will be added under an +`implementation/` tree **only after** the corresponding spec and ADRs are approved. Until then: + +- Do **not** invent implementation details that contradict the specs or ADRs. +- Do **not** create the `implementation/` tree until the roadmap phase that authorizes it is active + (see [`roadmap/phases.md`](roadmap/phases.md)). +- When you implement, the spec is the acceptance criteria. If the spec is wrong, blocking, + ambiguous, or missing, **you may not fix it yourself** — stop and escalate to the human operator/ + architect (§2, §11). You may draft a proposal; you may not merge a spec change. + +## 2. The source-of-truth hierarchy + +When two documents conflict, the higher item wins: + +1. [`docs/00-vision-and-scope.md`](docs/00-vision-and-scope.md) — requirements (`R-*` IDs) +2. Approved ADRs in [`decisions/`](decisions/) — locked technology & pattern choices +3. Domain specs in [`docs/`](docs/) +4. [`roadmap/`](roadmap/) and [`agents/`](agents/) — sequencing and coordination +5. Implementation code (once it exists) + +If you find a conflict, **stop and escalate it to the human operator/architect** — do not silently +pick one interpretation, and do not resolve it by editing the specs yourself (you may not change +them, §11). Wait for the human to amend the spec, then implement against the amended spec. + +## 3. Do not re-litigate decisions + +Every major choice (k3s, Cilium, Longhorn, KubeVirt, Flux, SOPS, cert-manager, etc.) is captured in +an ADR with rationale and rejected alternatives. If new information genuinely invalidates a +decision: + +1. **Draft** a proposed superseding ADR and **escalate to the human operator/architect** — you may + not merge it yourself (ADRs are specifications; §11 forbids agents changing them). +2. State what changed and why. +3. Only the human approves/accepts it. Implement against the new decision only after it is accepted. + +Never edit an accepted ADR's decision in place. ADRs are an append-only, **human-approved** ledger. + +## 4. Requirement traceability is mandatory + +- Every requirement has a stable ID like `R-NET-01`. +- Every ADR lists the requirement IDs it serves. +- Every task-board item lists the requirement IDs it satisfies. +- Every implementation PR references the requirement IDs and the spec sections it fulfills. + +If you cannot trace a piece of work to a requirement, either it is out of scope or a requirement is +missing — surface it, don't proceed silently. + +## 5. GitOps-first, click-ops-never + +- The desired state of the cluster lives in git and is reconciled automatically. +- Manual changes to the running cluster are prohibited **except** where an ADR explicitly carves out + an exception (e.g. a bootstrap step, an out-of-band edge-router config). Such exceptions must be + documented and, where possible, later folded back into IaC. +- Secrets live in git **encrypted** (SOPS + age). Never commit plaintext secrets. Never print a + decrypted secret into logs, PR descriptions, or agent output. + +## 6. Two environments, one configuration + +- Author configuration so dev/test and prod differ **only** by environment-scoped overrides + (replica counts, node selectors, storage sizes, WAN endpoints, DNS zones). +- Never fork the base configuration per environment. Use overlays/variables. +- **Never build directly on production.** Prod hardware is a design target, not a development + surface, until the roadmap declares it ready. + +## 7. Acceptance gates — proven by automated, reproducible behavior tests (non-negotiable) + +A change is not "done" until an **automated, reproducible, behavior-based test** (Given/When/Then, +tagged with the requirement IDs it verifies) **passes green in dev/test** and is committed alongside +the change. A manual demonstration, a screenshot, or "it reconciled" is **not** acceptance +(R-TEST-01..04). The approach — reproducibility rules, layers, tooling, and the conformance suite — +is specified in [`docs/14-testing-and-conformance.md`](docs/14-testing-and-conformance.md) and +[`decisions/adr-0010-conformance-testing.md`](decisions/adr-0010-conformance-testing.md). + +The behaviors each change must prove (each expressed as an automated test): + +- **No data loss on rebuild** — destroy and recreate the workload; state survives (R-STATE-*). +- **Default-deny holds** — the workload cannot reach services it wasn't explicitly allowed to + (R-NET-*). +- **Correct egress path** — internet-class services egress locally; VPN-class services egress only + through the VPN, with a verified kill-switch (R-NET-*). +- **Trusted TLS** — the endpoint serves a valid, non-self-signed, auto-renewing certificate + (R-TLS-*). +- **Observed** — metrics, logs, and at least one meaningful alert exist and fire correctly + (R-OBS-*). +- **Hands-off updates** — the service advances to new stable versions automatically per policy, + with no hand-authored change (R-UPD-*). + +See [`agents/definition-of-done.md`](agents/definition-of-done.md) for the full checklist. + +## 8. Coordination + +- Claim work on [`agents/task-board.md`](agents/task-board.md) before starting; respect declared + dependencies (`blocked-by`). +- Follow [`agents/coordination-protocol.md`](agents/coordination-protocol.md) for handoffs, + interface contracts, and conflict resolution between agents/workstreams. +- Prefer small, reviewable, single-responsibility changes. One workstream should not silently change + another workstream's interfaces — negotiate the contract change first. + +## 9. Style & conventions + +- Markdown for all specs. One concept per file. Link generously between documents. +- Use requirement IDs and ADR numbers in prose so references stay stable when files move. +- Be decisive in specs. Where a genuine open question remains, record it under an explicit + **Open Questions** heading rather than burying ambiguity in the body. +- Keep the [`GLOSSARY.md`](GLOSSARY.md) authoritative — if you introduce a term, define it there. + +## 10. What "you do not write code" means for the architect vs. what agents do + +The **architect** authored the specs, ADRs, and this coordination structure — no code. The +**implementation agents** who consume this repo *do* write code (implementation *and* the automated +conformance tests that prove it), but only in service of an approved spec, only under the active +roadmap phase, and only against the automated acceptance gates above. Agents write tests; agents do +not write specs. + +## 11. Specifications are immutable to agents (governance) + +This is a hard boundary (R-TEST-06): + +- **You MUST NOT modify** anything under [`docs/`](docs/), the requirements, accepted ADRs in + [`decisions/`](decisions/), the roadmap, or these coordination files. These are human-owned. +- **You MAY and MUST create/modify**: implementation code, **automated conformance tests**, the + implementation repo, and the *live coordination state* the specs explicitly delegate to you — + namely claiming/updating items on [`agents/task-board.md`](agents/task-board.md) and recording + proposals/escalations there. (The task board is a working surface; the *rules* around it are not.) +- **When the spec is wrong, blocking, ambiguous, or missing:** STOP. Do not proceed on a guess and + do not edit the spec. Raise an escalation on the task board (and draft a proposal if useful) for + the **human operator/architect** to decide. Resume only after the human amends the spec. +- **Never** make the implementation pass by weakening the spec, the requirement, or the test's + intent. If reality can't meet the spec, that is an escalation, not a spec edit. + +The point: the specification is the fixed target. Agents move the implementation and its tests +toward the target; they never move the target. diff --git a/GLOSSARY.md b/GLOSSARY.md new file mode 100644 index 0000000..8d87d8e --- /dev/null +++ b/GLOSSARY.md @@ -0,0 +1,90 @@ +# Glossary + +Shared vocabulary for this platform. If you introduce a term, define it here. When a word could mean +two things (e.g. "VM"), this file disambiguates. + +## Platform concepts +- **PaaS** — Platform-as-a-Service; here, containerized workloads onboarded via GitOps. +- **IaaS** — Infrastructure-as-a-Service; here, virtual machines run *inside* the cluster via + KubeVirt, managed through the same Kubernetes surface. +- **ServiceDefinition** — the operator-facing YAML object that declares a service's intent + (kind, source, storage, egress, exposure, dependencies, update policy, observability). The + platform renders concrete resources from it. See [`docs/13-service-catalog.md`](docs/13-service-catalog.md). +- **Golden path** — the standard flow a service follows from catalog entry to running/monitored/ + updated. See [`docs/01-architecture-overview.md`](docs/01-architecture-overview.md). +- **C2 / Command & Control** — the UI at the root `home.keithjackson.dev`. See + [`docs/12-command-center-ui.md`](docs/12-command-center-ui.md). + +## The two kinds of VM (do not conflate) +- **Infrastructure VM (Layer 1)** — a cluster *node*. In dev/test these are libvirt KVMs; in prod + they are bare-metal mini-PCs. Managed by OpenTofu/Ansible, **not** KubeVirt. +- **Application VM (Layer 2)** — an IaaS *workload* running inside the cluster via **KubeVirt**, + managed by Kubernetes/GitOps like a container. + +## Networking +- **Default-deny** — the baseline where no service may talk to another unless explicitly allowed at + the port level (R-NET-01). +- **Egress class** — a service's declared outbound path: `internet` (standard WAN) or `vpn` + (commercial VPN tunnel). See [`docs/03-networking-isolation-and-egress.md`](docs/03-networking-isolation-and-egress.md). +- **Exposure class** — how a service is reachable: `internal` (LAN-only) or `public` (edge + port-forwarded). +- **Kill-switch** — the guarantee that a `vpn`-class workload leaks nothing to the plain internet if + the tunnel drops (R-NET-06); fail-closed. +- **Multi-WAN** — multiple upstream internet uplinks with routing-rule-based selection; physical WAN + selection is largely an edge-router responsibility (OQ-1). +- **Egress Gateway** — Cilium feature steering selected pods' egress through a chosen node/identity. +- **Hubble** — Cilium's flow-observability tool; the audit surface for isolation/egress. +- **Gateway API** — the Kubernetes standard for north-south ingress, implemented here by Cilium. +- **Split-horizon DNS** — serving different answers to internal vs. external clients for the same + name. + +## Storage +- **Longhorn** — distributed replicated block storage across the 3 nodes (default 3 replicas). The + durable home for app/config state and VM disks. +- **Bulk tier** — a separate storage class for large media (Plex library, downloads) that is **not** + 3× replicated (R-STATE-04). See [`docs/04-storage-and-state.md`](docs/04-storage-and-state.md). +- **CDI** — Containerized Data Importer; provisions KubeVirt VM disks onto Longhorn PVCs. +- **Velero** — cluster backup/restore orchestration to an off-cluster target (DR). +- **DR drill** — a tested restore of resources + volumes proving R-STATE-03. + +## Delivery & lifecycle +- **GitOps** — desired state lives in git and is reconciled into the cluster automatically + (pull-based). Engine: **Flux CD**. +- **Drift correction** — Flux reverting live changes back to git-declared state (R-GITOPS-03). +- **Image automation** — Flux controllers advancing container image versions per declared policy + with no human/AI edit (R-UPD-02). +- **Renovate** — bot that opens (and auto-merges) chart/manifest version-bump PRs. +- **SOPS + age** — the secret-encryption scheme; secrets live encrypted in git, decrypted in-cluster. +- **cert-manager / ACME DNS-01 / Let's Encrypt** — the trusted, automated TLS pipeline (R-TLS-*). +- **ExternalDNS** — controller that publishes DNS records from declared service intent. + +## Environments +- **dev/test** — local KVM cluster; the active build target where everything is proven. +- **prod** — 3× mini-PC bare-metal cluster; design-only until roadmap Phase 7. +- **Overlay** — environment-scoped configuration (Kustomize/vars) layered on shared base config; the + only thing that differs between dev and prod (R-ENV-02). + +## Governance +- **Requirement ID (`R-*`)** — a stable, referenceable requirement from + [`docs/00-vision-and-scope.md`](docs/00-vision-and-scope.md). +- **ADR** — Architecture Decision Record; an append-only, superseding-only decision ledger under + [`decisions/`](decisions/). +- **Workstream (`WS-*`)** — a durable area of ownership; maps to an agent role. +- **Acceptance gate** — an **automated, reproducible behavior test** a change must pass to be "done" + ([`agents/definition-of-done.md`](agents/definition-of-done.md), [`docs/14-testing-and-conformance.md`](docs/14-testing-and-conformance.md)). +- **Open Question (`OQ-n`)** — a tracked unknown that may block specific tasks until answered. + +## Testing & conformance +- **Behavior-based test** — a test asserting observable system behavior (Given/When/Then), not + implementation internals; a policy object *existing* is not proof it *enforces* (R-TEST-01). +- **Reproducible test** — deterministic, hermetic, version-pinned, single-command, environment- + agnostic; identical pass/fail on any conforming cluster (R-TEST-02). +- **Conformance suite** — the aggregate, requirement-tagged test set that gates merges and phase + exits; green suite = progress (R-TEST-03). See [`docs/14-testing-and-conformance.md`](docs/14-testing-and-conformance.md). +- **Fault-injection test** — a test that *breaks* something (tunnel down, node killed, bad update) + and asserts a fail-closed / no-data-loss outcome; required for safety-critical behavior (R-TEST-05). +- **Coverage report** — the generated map of every `R-*` requirement → its verifying test(s); an + uncovered requirement is an unmet requirement (R-TEST-04). +- **Spec immutability (to agents)** — the rule that AI agents may not modify the specifications + (`docs/`, requirements, accepted ADRs, roadmap); they escalate spec problems to the human and never + weaken a spec/test to pass (R-TEST-06). Agents edit implementation, tests, and the task board only. diff --git a/README.md b/README.md new file mode 100644 index 0000000..47144bd --- /dev/null +++ b/README.md @@ -0,0 +1,105 @@ +# Home Platform — k3s + Longhorn Onboarding Platform + +> **This repository is a specification, not an implementation.** It defines *what* to build +> and *how the work is organized* so that teams of AI agents (and their human reviewers) can +> implement a self-service onboarding platform for home infrastructure at both the **PaaS** +> (containerized apps) and **IaaS** (virtual machines) level. + +## What we are building + +A single, GitOps-driven platform, running on a **k3s** Kubernetes cluster backed by **Longhorn** +distributed storage, that lets an operator onboard, run, update, and tear down services — both +**containers** and **virtual machines** — through one consistent Kubernetes API, CLI, and YAML +configuration surface. + +The platform must provide: + +- **PaaS**: containerized workloads onboarded via GitOps. +- **IaaS**: virtual machines managed through the *same* Kubernetes API/CLI/YAML (via KubeVirt), + for workloads that cannot be containerized. +- **Network isolation**: default-deny east-west traffic; explicit, port-level allow rules between + services (firewall-like). +- **Multi-WAN egress**: some services egress the normal internet; others are routed through a + custom VPN connection. Egress path is a declarative property of a service. +- **Stateful durability**: any service can be destroyed and rebuilt on a whim without losing data. +- **Hands-off updates**: services update to latest stable versions on a schedule — no human, + no AI, no hand-written GitOps change required. +- **Trusted TLS**: every service gets an auto-issued, auto-renewed, non-self-signed certificate. +- **Observability**: services are monitored; issues are reported by email and on dashboards. +- **A command & control center**: the root of `home.keithjackson.dev` is a UI that surfaces and + controls everything running in the cluster. +- **Provable conformance**: every requirement is verified by an automated, reproducible, behavior- + based test — and the specifications themselves are human-owned and immutable to the implementing + agents. + +## Two environments + +| Environment | Substrate | Purpose | Status | +|---|---|---|---| +| **dev/test** | Local KVM virtual machines on a workstation, provisioned by IaC | Prove the platform end-to-end before hardware exists | **Active development target** | +| **prod** | 3× mini-PC bare-metal k3s cluster, each with local storage | The real deployment | **Not ready — design for it, do not build on it yet** | + +The dev/test environment is the proving ground. Everything must be authored so that the *same* +GitOps configuration promotes to production with only environment-scoped overrides. + +## How to use this repository (start here) + +1. **Read [`AGENTS.md`](AGENTS.md)** — the rules of engagement for every AI agent working here. +2. **Read [`docs/00-vision-and-scope.md`](docs/00-vision-and-scope.md)** — the north star and the + non-negotiable requirements (with stable requirement IDs). +3. **Read [`docs/01-architecture-overview.md`](docs/01-architecture-overview.md)** — the chosen + stack and how the pieces fit. +4. **Consult [`decisions/`](decisions/)** — the Architecture Decision Records (ADRs) that lock in + every major technology choice and *why*. Do not re-litigate a decision; if you disagree, open a + superseding ADR per the process in [`decisions/README.md`](decisions/README.md). +5. **Pick up work from [`roadmap/`](roadmap/) and [`agents/task-board.md`](agents/task-board.md)** — + phased delivery plan, workstreams, dependencies, and the live task board. + +## Repository map + +``` +home-network-k3s/ +├── README.md ← you are here +├── AGENTS.md ← rules of engagement for AI agents +├── GLOSSARY.md ← shared vocabulary (read to avoid ambiguity) +├── docs/ ← the specification, by domain +│ ├── 00-vision-and-scope.md +│ ├── 01-architecture-overview.md +│ ├── 02-environments-and-provisioning.md +│ ├── 03-networking-isolation-and-egress.md +│ ├── 04-storage-and-state.md +│ ├── 05-gitops-and-delivery.md +│ ├── 06-vm-workloads-kubevirt.md +│ ├── 07-security-and-secrets.md +│ ├── 08-tls-and-certificates.md +│ ├── 09-dns-and-ingress.md +│ ├── 10-observability-and-alerting.md +│ ├── 11-automated-updates.md +│ ├── 12-command-center-ui.md +│ ├── 13-service-catalog.md +│ └── 14-testing-and-conformance.md +├── decisions/ ← ADRs (one decision per file) + index/template +├── roadmap/ ← phases.md, workstreams.md +└── agents/ ← roles, coordination protocol, DoD, task board +``` + +## Requirement traceability + +Every hard requirement has a stable ID (e.g. `R-NET-01`) defined in +[`docs/00-vision-and-scope.md`](docs/00-vision-and-scope.md). Implementation work, ADRs, and +task-board items must reference the requirement IDs they satisfy. This is how we prove the built +system meets the spec. + +## Ground rules (the short version — full detail in `AGENTS.md`) + +- **Specs are the source of truth — and are immutable to agents.** Code and tests serve the spec. + AI agents **may not change** the `docs/`, requirements, or accepted ADRs; a spec that is wrong or + can't be met is an **escalation to the human**, never an edit (R-TEST-06). +- **Done means a green, reproducible, behavior-based test.** Every requirement is verified by an + automated, requirement-tagged conformance test ([`docs/14-testing-and-conformance.md`](docs/14-testing-and-conformance.md), + [ADR-0010](decisions/adr-0010-conformance-testing.md)); a manual demo or "it merged" is not + acceptance. +- **Everything is GitOps.** No click-ops in the cluster except where an ADR explicitly allows it. +- **Nothing is self-signed, nothing is manual, nothing loses data.** These three properties are + acceptance gates, not aspirations — and each is proven by a test. +- **Design once, deploy to two environments.** dev/test and prod differ only by scoped overrides. diff --git a/agents/coordination-protocol.md b/agents/coordination-protocol.md new file mode 100644 index 0000000..5fa344e --- /dev/null +++ b/agents/coordination-protocol.md @@ -0,0 +1,69 @@ +# Coordination Protocol + +How agents work together without stepping on each other. Read alongside [`roles.md`](roles.md), +[`../roadmap/workstreams.md`](../roadmap/workstreams.md), and [`../AGENTS.md`](../AGENTS.md). + +## 1. Claim before you build +- Take an item from [`task-board.md`](task-board.md), set its owner and status to `in-progress`. +- Respect `blocked-by`. Do not start blocked work; help unblock it instead. +- Keep items small and single-responsibility. + +## 2. Contracts are sacred +Workstreams communicate through **interfaces** (listed in [`../roadmap/workstreams.md`](../roadmap/workstreams.md)): +the ServiceDefinition schema, StorageClasses, the network policy/egress rendering, TLS issuer + +Gateway, the observability contract, the update-policy contract. + +- **Consume** a contract freely. +- **Change** a contract only by: (1) proposing the change on the task board with the affected + workstreams tagged, (2) getting the owning role's sign-off, (3) updating the spec that defines the + contract in the *same* change. Never silently break a downstream consumer. +- The **ServiceDefinition schema** is the highest-traffic contract; treat schema changes as + platform-wide events. + +## 3. Decisions vs. discussion — and the no-spec-change rule +- **You may not modify the specifications** (`docs/`, requirements, accepted ADRs, roadmap) — they + are human-owned and immutable to agents (R-TEST-06, [`../AGENTS.md` §11](../AGENTS.md)). +- Anything that changes a locked choice ⇒ **draft** a superseding ADR and **escalate to the human** + ([`../decisions/README.md`](../decisions/README.md)); do not self-approve or merge it. +- Genuine unknowns, or a spec that is wrong/blocking/ambiguous ⇒ **stop and escalate** on the task + board as an **Open Question (OQ-n)** for the human to resolve in the spec; don't bury ambiguity in + an implementation and don't patch the spec yourself. + +## 4. Definition of done +Nothing is "done" until its **automated, reproducible behavior test passes green** in **dev/test**, +committed with the change and tagged to its requirement IDs ([`definition-of-done.md`](definition-of-done.md), +[`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md)). "Merged" ≠ "done"; +"demonstrated manually" ≠ "done". You write the test as well as the implementation. + +## 5. Phase discipline +- Only work items in the **active phase(s)** ([`../roadmap/phases.md`](../roadmap/phases.md)) unless + explicitly parallelizable. +- Respect entry gates; don't start a phase whose predecessor's exit criteria aren't demonstrated. +- **Prod (Phase 7) is closed** until the operator opens it. + +## 6. Handoffs +When your output feeds another workstream: +- State the interface you're delivering and where it's documented. +- Provide the demonstrable proof (the gate you passed) so the consumer can trust it. +- If you changed a shared contract, announce it on the board with consumers tagged. + +## 7. Conflict resolution +1. Re-read the source-of-truth hierarchy ([`../AGENTS.md` §2](../AGENTS.md)); the higher document + wins. +2. If specs genuinely conflict or are silent, escalate to the **Platform Architect** to amend the + spec/ADR — fix the documents first, then build. +3. Never resolve a conflict by silently picking an interpretation in code. + +## 8. Safety rails that never bend +- Default-deny + VPN kill-switch are **fail-closed** — if unsure, deny/close. +- Secrets stay encrypted; never printed. +- Nothing that risks data ships without a tested backup/restore behind it. +- No click-ops except ADR-sanctioned bootstrap steps. +- **Specs are immutable to you**; a spec you can't meet is an escalation, never an edit. +- **Never weaken a test or a requirement to get green** — that's a defect, not progress. + +## 9. Communication artifacts +- **task-board.md** — the live who's-doing-what + status + blockers. +- **specs/ADRs** — durable decisions and contracts. +- **PR descriptions** — reference requirement IDs, specs, ADRs, and the gate demonstrated. +Keep these three in sync; they are how independent agents stay coherent. diff --git a/agents/definition-of-done.md b/agents/definition-of-done.md new file mode 100644 index 0000000..8c05715 --- /dev/null +++ b/agents/definition-of-done.md @@ -0,0 +1,84 @@ +# Definition of Done + +A change is **done** only when an **automated, reproducible, behavior-based test** for each relevant +gate below **passes green in dev/test**, the test is committed with the change and tagged with the +requirement IDs it verifies, and the change is traced to those IDs. A manual demonstration, a +screenshot, or "it reconciles" is **not** done (R-TEST-01..04, [`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md)). + +> **Governance (R-TEST-06):** you may not change the specifications to reach done. If the spec can't +> be met, that is an **escalation to the human**, not a spec/requirement/test-intent edit. Weakening +> a test so it passes is a defect, not a completion. + +## Universal checklist (every change) +- [ ] Traced to one or more requirement IDs (`R-*`) and the spec/ADR it implements. +- [ ] An **automated behavior test** (Given/When/Then) exists, is committed with the change, tagged + with its `R-*` IDs, and **passes**; it appears in the coverage report. +- [ ] The test is **reproducible**: single-command, hermetic (self-provisioning + self-cleaning), + deterministic, and yields identical results on re-run. +- [ ] Expressed as GitOps (in git, reconciled by Flux) — no click-ops except ADR-sanctioned bootstrap. +- [ ] No plaintext secrets anywhere; secrets are SOPS-encrypted (and no secret printed in output). +- [ ] Authored once with environment-scoped overrides (no per-env base fork). +- [ ] Observability wired (metrics/logs) and at least the relevant alert exists. +- [ ] No specification/ADR/requirement was modified to achieve this (R-TEST-06); any spec problem was + escalated, not edited. +- [ ] Result recorded on the task board / PR with a link to the passing test run. + +## Requirement-class gates (apply those relevant to the change) + +> Each checkbox below is the **intent of an automated behavior test** — implement it as a +> reproducible, requirement-tagged test (see the behavior catalog in +> [`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md)), not a manual +> check. Safety-critical gates (marked *fault-injection*) must break the system and assert +> fail-closed / no-data-loss (R-TEST-05). + +### State & rebuild (R-STATE-*) +- [ ] Destroy the workload entirely and re-apply from git; **data survives**. +- [ ] Stateful data is on `longhorn` (replicated) or the `bulk` tier — never ephemeral/hostPath. +- [ ] DR-critical data restores from an **off-cluster** backup (drill for at least one service/phase). + +### Network isolation & egress (R-NET-*) +- [ ] With no explicit allows, the workload can reach **only DNS** (default-deny proven via Hubble + + probe). +- [ ] Declared `dependencies` work; an **undeclared** destination is blocked (negative test). +- [ ] `egress: internet` leaves via the home/WAN IP; `egress: vpn` leaves via the **VPN IP**. +- [ ] **VPN kill-switch:** tunnel forced down ⇒ the `vpn` workload leaks nothing; `internet` + workloads unaffected. +- [ ] Changing `egress` class in git re-routes with no workload rewrite. + +### TLS (R-TLS-*) +- [ ] Endpoint serves a **valid, non-self-signed, unexpired** cert (external check), including for + `internal` services. +- [ ] Renewal is automatic (observed or verified via cert-manager) with no manual step. + +### DNS & ingress (R-DNS-*) +- [ ] `.home.keithjackson.dev` record appears automatically and routes correctly. +- [ ] `internal` is unreachable publicly; `public` is reachable + TLS-valid. +- [ ] Internal clients resolve to the internal Gateway (no public hairpin). + +### Automated updates (R-UPD-*) +- [ ] Advances to a newer **stable** version automatically on schedule with **no human/AI edit**. +- [ ] A deliberately bad update is health-gated: **auto-rollback** + alert. +- [ ] A **pinned** service does not advance. + +### Observability & alerting (R-OBS-*) +- [ ] Metrics + logs flow; a meaningful alert **fires** in a fault test and is delivered by **email**. +- [ ] Dashboard renders and is linked from the C2 UI. + +### VM/IaaS (R-PLAT-03) +- [ ] VM created/started/stopped/destroyed via `kubectl`/`virtctl` (same K8s surface). +- [ ] VM disk is Longhorn-backed; rebuild preserves data; default-deny + egress class apply; guest + auto-updates. + +### Security & auditability (R-SEC-*) +- [ ] Least-privilege RBAC (no broad cluster rights for workloads). +- [ ] The auditability proofs (isolation / egress path / kill-switch / TLS / backup) are producible. + +### Per-service onboarding (WS-APPS) +- [ ] The service is **fully wired from its ServiceDefinition alone** (compute/storage/policy/egress/ + TLS/DNS/monitoring/update) — no manual steps beyond merge. + +## Phase exit +A phase is done when the **full conformance suite for that phase's requirements passes green** +(R-TEST-03) and the exit criteria in [`../roadmap/phases.md`](../roadmap/phases.md) are recorded with +links to the passing runs. Prod (Phase 7) additionally requires a successful **automated** prod DR +drill. diff --git a/agents/roles.md b/agents/roles.md new file mode 100644 index 0000000..ce2ea51 --- /dev/null +++ b/agents/roles.md @@ -0,0 +1,56 @@ +# Agent Roles + +Each role owns a workstream ([`../roadmap/workstreams.md`](../roadmap/workstreams.md)) and is +accountable for that workstream's requirement IDs and acceptance gates. Roles are responsibilities, +not people — one agent may hold several early on; split as work scales. + +## Coordination roles + +### Platform Architect (steward of this spec repo) +- Guards the source-of-truth hierarchy ([`../AGENTS.md` §2](../AGENTS.md)); approves/records ADRs. +- Resolves cross-workstream interface conflicts; keeps requirement traceability honest. +- Does **not** write implementation code; owns specs, ADRs, roadmap, and this coordination layer. + +### Integration/Release Lead +- Owns the implementation repo structure, CI gates, and Flux bootstrap health. +- Runs phase entry/exit reviews against the **automated conformance suite**; keeps the task board + truthful. + +### QA/Conformance Engineer (WS-TEST) +- Owns the conformance testing capability ([14](../docs/14-testing-and-conformance.md), + [ADR-0010](../decisions/adr-0010-conformance-testing.md)): the harness, the reproducibility + contract, the requirement-coverage report, and the CI/phase gates. +- Does **not** own writing every test in isolation — **each workstream authors the behavior tests + for its own requirements**; WS-TEST provides the framework, patterns, fault-injection harness, and + gate, and guards that tests assert real behavior and are never weakened to pass (R-TEST-05/06). +- Accountable for: every `R-*` requirement having a passing, reproducible, tagged test. + +## Implementation roles (one per workstream) + +| Role | Workstream | Primary specs | Accountable for (gates) | +|---|---|---|---| +| **Infra/Cluster Engineer** | WS-INFRA | [02](../docs/02-environments-and-provisioning.md) | Reproducible node provisioning; k3s HA + Cilium; node rebuild w/o data loss | +| **GitOps/Platform Engineer** | WS-GITOPS | [05](../docs/05-gitops-and-delivery.md), [13](../docs/13-service-catalog.md) | Reconcile-from-git; drift correction; ServiceDefinition renderer; CI | +| **Network Engineer** | WS-NET | [03](../docs/03-networking-isolation-and-egress.md), [09](../docs/09-dns-and-ingress.md) | Default-deny; explicit allows; egress classes; VPN kill-switch; ingress/DNS | +| **Storage Engineer** | WS-STORAGE | [04](../docs/04-storage-and-state.md) | Replicated Longhorn; bulk tier; backups; **tested restore/DR** | +| **Security Engineer** | WS-SEC | [07](../docs/07-security-and-secrets.md), [08](../docs/08-tls-and-certificates.md) | SOPS secrets; trusted auto-TLS; least-privilege; auditability proofs | +| **Observability Engineer** | WS-OBS | [10](../docs/10-observability-and-alerting.md) | Metrics/logs; email alerts; dashboards; platform-health alerts | +| **Automation Engineer** | WS-UPDATE | [11](../docs/11-automated-updates.md) | Hands-off stable updates; health-gate + auto-rollback; pin | +| **Virtualization Engineer** | WS-VM | [06](../docs/06-vm-workloads-kubevirt.md) | KubeVirt IaaS via same K8s surface; VM rebuild w/o data loss | +| **Application Onboarding Engineer** | WS-APPS | [13](../docs/13-service-catalog.md) | Each service wired from its ServiceDefinition; deps correct; nothing undeclared | +| **C2/UX Engineer** | WS-C2 | [12](../docs/12-command-center-ui.md) | Root C2 UI; live catalog + health; safe, audited actions | + +## Standing expectations for every role +- Trace all work to requirement IDs; reference specs/ADRs by ID (stable across file moves). +- **Author an automated, reproducible behavior test** for every requirement you implement; "done" is + a green, tagged test, not "merged" ([definition-of-done.md](definition-of-done.md), + [14](../docs/14-testing-and-conformance.md)). +- **Never modify the specifications** (`docs/`, requirements, accepted ADRs, roadmap) — they are + human-owned (R-TEST-06). A spec you can't meet is an **escalation**, not an edit. Never weaken a + test/requirement to get green. +- Never re-litigate an accepted ADR; **draft + escalate** a superseding ADR for human approval + ([`../decisions/README.md`](../decisions/README.md)). +- Negotiate interface changes before breaking another workstream's contract + ([coordination-protocol.md](coordination-protocol.md)). +- Never build on prod hardware before Phase 7 is opened. +- Never emit plaintext secrets. diff --git a/agents/task-board.md b/agents/task-board.md new file mode 100644 index 0000000..72552cd --- /dev/null +++ b/agents/task-board.md @@ -0,0 +1,100 @@ +# Task Board + +The live who's-doing-what. Agents **claim** an item (set Owner + Status) before starting, respect +`Blocked-by`, and mark `done` only after its **automated, reproducible behavior test passes green** +([`definition-of-done.md`](definition-of-done.md), [`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md)). +Seeded from [`../roadmap/phases.md`](../roadmap/phases.md) + [`../roadmap/workstreams.md`](../roadmap/workstreams.md). + +> **This board is the one working surface agents may edit** (claim items, update status, add +> items/escalations). The specs, ADRs, and roadmap are **not** editable by agents (R-TEST-06) — +> raise spec problems here as an `OQ-*` escalation for the human. Every "**Prove …**" task below is +> an automated conformance test (co-owned with **WS-TEST**), not a manual demo. + +**Status values:** `todo` · `in-progress` · `blocked` · `review` · `done` +**Add new items** at the bottom of the relevant phase; keep IDs stable. + +--- + +## Phase 0 — Foundations +| ID | Task | WS | Serves | Blocked-by | Owner | Status | +|---|---|---|---|---|---|---| +| T-0.1 | Create implementation repo structure (platform/apps/catalog/clusters) | WS-GITOPS | R-GITOPS-02 | — | — | todo | +| T-0.2 | CI: lint + schema validation + build/dry-run gates | WS-GITOPS | R-GITOPS-01 | T-0.1 | — | todo | +| T-0.3 | Author ServiceDefinition schema + renderer contract | WS-GITOPS | R-GITOPS-02 | T-0.1 | — | todo | +| T-0.4 | Conformance harness: **Chainsaw** runner + reproducibility contract + JUnit + coverage report + CI merge gate | WS-TEST | R-TEST-01/02/03/04 | T-0.1 | — | todo | +| T-0.5 | L0 **Kyverno** policy gate (default-deny present, no plaintext secrets, schema-valid ServiceDefinitions) | WS-TEST | R-TEST-01, R-NET-01, R-SEC-01 | T-0.4 | — | todo | + +## Phase 1 — dev/test Cluster Bring-up +| ID | Task | WS | Serves | Blocked-by | Owner | Status | +|---|---|---|---|---|---|---| +| T-1.1 | OpenTofu libvirt module for 3 node VMs (+ Longhorn data disk) | WS-INFRA | R-ENV-01 | — | — | todo | +| T-1.2 | cloud-init baseline (users, SSH, sysctls, virt/Longhorn prereqs) | WS-INFRA | R-ENV-01 | T-1.1 | — | todo | +| T-1.3 | Ansible: k3s HA install, Cilium CNI, disable flannel/kube-proxy/Traefik | WS-INFRA | R-PLAT-01 | T-1.2 | — | todo | +| T-1.4 | Bootstrap Flux + SOPS age key | WS-GITOPS/WS-SEC | R-GITOPS-01, R-SEC-01 | T-1.3 | — | todo | +| T-1.5 | Prove node destroy/rebuild from IaC (no data loss) | WS-INFRA | R-ENV-01, R-STATE-01 | T-1.4 | — | todo | + +## Phase 2 — Platform Core (net/storage/TLS/DNS/ingress) +| ID | Task | WS | Serves | Blocked-by | Owner | Status | +|---|---|---|---|---|---|---| +| T-2.1 | Longhorn (3-replica StorageClass) + bulk-tier plan (OQ-2) | WS-STORAGE | R-STATE-02/04 | T-1.4 | — | todo | +| T-2.2 | Default-deny network baseline (per-namespace) + Hubble | WS-NET | R-NET-01/03 | T-1.3 | — | todo | +| T-2.3 | Render `dependencies` → CiliumNetworkPolicy from ServiceDefinition | WS-NET/WS-GITOPS | R-NET-02 | T-0.3, T-2.2 | — | todo | +| T-2.4 | cert-manager + Let's Encrypt DNS-01 wildcard `*.home.keithjackson.dev` | WS-SEC | R-TLS-01/02/03 | T-1.4 | — | todo | +| T-2.5 | ExternalDNS + Gateway API listeners + split-horizon (PiHole) foundation | WS-NET | R-DNS-01/02/03 | T-2.4 | — | todo | +| T-2.6 | Prove: default-deny (DNS-only probe), PVC survives rebuild, trusted cert, auto-DNS | WS-NET/STORAGE/SEC | R-NET/STATE/TLS/DNS | T-2.1..2.5 | — | todo | +| T-2.7 | Deploy **Kyverno in-cluster as admission enforcer** (same policies as the CI L0 gate) | WS-TEST/WS-SEC | R-SEC-02, R-NET-01, R-SEC-01 | T-1.4, T-0.5 | — | todo | + +## Phase 3 — Egress, VPN & Multi-WAN +| ID | Task | WS | Serves | Blocked-by | Owner | Status | +|---|---|---|---|---|---|---| +| T-3.1 | Egress classes; Cilium Egress Gateway wiring | WS-NET | R-NET-04/05 | T-2.2 | — | todo | +| T-3.2 | VPN egress (Gluetun sidecar) + kill-switch (ADR-0008) | WS-NET | R-NET-05/06 | T-3.1, OQ-3 | — | todo | +| T-3.3 | Edge-router multi-WAN split for `internet` class (OQ-1) | WS-NET | R-NET-04 | OQ-1 | — | todo | +| T-3.0 | L3 fault-injection harness — **in-cluster Chainsaw Job** (CI-triggered, scoped test SA) using `script`/`try`/`finally` (force tunnel down, kill node, inject bad image, trigger restore) | WS-TEST | R-TEST-05, R-SEC-02 | T-0.4 | — | todo | +| T-3.4 | Prove: VPN IP egress, kill-switch fail-closed, egress-class flip via git | WS-NET | R-NET-05/06/07 | T-3.0, T-3.2 | — | todo | + +## Phase 4 — Observability, Backup/DR & Updates +| ID | Task | WS | Serves | Blocked-by | Owner | Status | +|---|---|---|---|---|---|---| +| T-4.1 | kube-prometheus-stack + Loki + Alloy + Alertmanager(email) | WS-OBS | R-OBS-01/02/03 | T-2.1 | — | todo | +| T-4.2 | Platform-health alerts (node/Longhorn/Flux/cert/VPN/backup) | WS-OBS | R-OBS-04 | T-4.1 | — | todo | +| T-4.3 | Velero + Longhorn off-cluster backup schedule | WS-STORAGE | R-STATE-03 | T-2.1, OQ-2 | — | todo | +| T-4.4 | DR drill: restore namespace + volumes from off-cluster backup | WS-STORAGE | R-STATE-03 | T-4.3 | — | todo | +| T-4.5 | Flux image-automation + Renovate + per-service update policy | WS-UPDATE | R-UPD-01/02/03 | T-0.3, T-4.1 | — | todo | +| T-4.6 | Prove: auto stable update on schedule, bad-update auto-rollback+alert, pin holds | WS-UPDATE | R-UPD-* | T-4.5 | — | todo | + +## Phase 5 — IaaS Proof (KubeVirt) +| ID | Task | WS | Serves | Blocked-by | Owner | Status | +|---|---|---|---|---|---|---| +| T-5.1 | Install KubeVirt + CDI; Longhorn-backed VM disks | WS-VM | R-PLAT-03 | T-2.1, T-2.2 | — | todo | +| T-5.2 | `kind: vm` ServiceDefinition path; run one representative VM | WS-VM/WS-GITOPS | R-PLAT-03/04 | T-5.1, T-0.3 | — | todo | +| T-5.3 | Prove: VM lifecycle via kubectl/virtctl, rebuild w/o data loss, default-deny+egress, guest auto-update | WS-VM | R-PLAT-03, R-NET-*, R-UPD-* | T-5.2 | — | todo | + +## Phase 6 — Service Onboarding + C2 +| ID | Task | WS | Serves | Blocked-by | Owner | Status | +|---|---|---|---|---|---|---| +| T-6.1 | Onboard **pihole** (DNS underpinning) | WS-APPS | R-DNS-03 | T-2.5 | — | todo | +| T-6.2 | Onboard **plex** (bulk tier + public exposure) | WS-APPS | R-STATE-04 | T-2.6, T-6.1 | — | todo | +| T-6.3 | Onboard **qbittorrent, jackett** (VPN + kill-switch) | WS-APPS | R-NET-05/06 | T-3.4 | — | todo | +| T-6.4 | Onboard **sonarr, radarr** (VPN + arr↔downloader allows) | WS-APPS | R-NET-02/05 | T-6.3 | — | todo | +| T-6.5 | Onboard **seerr** (allows to arr + plex) | WS-APPS | R-NET-02 | T-6.2, T-6.4 | — | todo | +| T-6.6 | Onboard **nextcloud, gitea** (DR-critical + backups) | WS-APPS | R-STATE-03 | T-4.4 | — | todo | +| T-6.7 | Onboard **minecraft, unifi, semaphore, kasm** | WS-APPS | app reqs | T-2.6 | — | todo | +| T-6.8 | C2 UI: Homepage root + Grafana + Headlamp + Flux status + auth | WS-C2 | R-C2-01/02 | T-4.2, T-6.* | — | todo | + +## Phase 7 — Production Readiness (CLOSED until operator opens it) +| ID | Task | WS | Serves | Blocked-by | Owner | Status | +|---|---|---|---|---|---|---| +| T-7.1 | Bare-metal provisioning checklist (design-only) | WS-INFRA | R-ENV-03 | — | — | todo | +| T-7.2 | prod overlay (no base edits) + cutover plan | WS-INFRA/WS-GITOPS | R-ENV-02 | Phases 1–6 done | — | blocked | +| T-7.3 | Full prod DR drill | WS-STORAGE | R-STATE-03 | T-7.2 | — | blocked | + +--- + +## Open questions (resolve to unblock) +| OQ | Question | Blocks | Owner | +|---|---|---|---| +| OQ-1 | WAN uplink count + edge-router product | T-3.3 | — | +| OQ-2 | Bulk-media home + backup posture | T-2.1, T-4.3 | — | +| OQ-3 | VPN provider/protocol + concurrent-tunnel limits | T-3.2 | — | +| OQ-4 | DNS provider API (Cloudflare) token for DNS-01 + ExternalDNS | T-2.4, T-2.5 | — | diff --git a/decisions/README.md b/decisions/README.md new file mode 100644 index 0000000..353206d --- /dev/null +++ b/decisions/README.md @@ -0,0 +1,55 @@ +# Architecture Decision Records (ADRs) + +ADRs are an **append-only ledger** of the significant technology and pattern choices for this +platform. Each records the context, the decision, the rejected alternatives, and the requirements it +serves. **Do not edit an accepted decision in place** — supersede it with a new ADR. + +> **ADRs are specifications and are immutable to AI agents** (R-TEST-06). Agents may *draft* a +> proposed ADR and escalate it, but only the **human operator/architect** approves/accepts or edits +> ADRs. See [`../AGENTS.md` §3 and §11](../AGENTS.md). + +## Process (read before proposing a change) + +1. To change a decision, create a **new** ADR with the next number and set `Status: Proposed`, + `Supersedes: ADR-XXXX`. (Agents draft + escalate; the human approves.) +2. State what new information invalidates the old decision. +3. On approval, set the new ADR `Accepted` and the old one `Superseded by: ADR-YYYY`. +4. Never silently re-interpret an accepted ADR (see [`../AGENTS.md` §3](../AGENTS.md)). + +## Template + +```markdown +# ADR-XXXX — +- Status: Proposed | Accepted | Superseded by ADR-YYYY +- Date: <YYYY-MM-DD> +- Serves: <requirement IDs, e.g. R-NET-01, R-NET-06> +- Supersedes: <ADR-ZZZZ or —> + +## Context +<forces, constraints, why a decision is needed> + +## Decision +<what we chose, stated decisively> + +## Alternatives considered +<options and why rejected> + +## Consequences +<positive, negative, follow-ups, risks> +``` + +## Index + +| ADR | Title | Status | Serves | +|---|---|---|---| +| [0001](adr-0001-kubernetes-distribution-k3s.md) | Kubernetes distribution: k3s | Accepted | R-PLAT-01 | +| [0002](adr-0002-cni-cilium.md) | CNI & network policy & ingress: Cilium + Gateway API | Accepted | R-NET-01..07, R-DNS-01 | +| [0003](adr-0003-storage-longhorn.md) | Distributed storage: Longhorn (+ bulk tier, Velero) | Accepted | R-STATE-01..04 | +| [0004](adr-0004-gitops-flux.md) | GitOps engine: Flux CD | Accepted | R-GITOPS-01..03 | +| [0005](adr-0005-vm-workloads-kubevirt.md) | VM workloads: KubeVirt + CDI | Accepted | R-PLAT-03 | +| [0006](adr-0006-tls-cert-manager-letsencrypt.md) | TLS: cert-manager + Let's Encrypt (DNS-01) | Accepted | R-TLS-01..03 | +| [0007](adr-0007-secrets-sops-age.md) | Secrets: SOPS + age | Accepted | R-SEC-01 | +| [0008](adr-0008-vpn-egress-model.md) | VPN egress model: Gluetun sidecar + Cilium Egress Gateway | Accepted | R-NET-05/06 | +| [0009](adr-0009-automated-image-updates.md) | Automated updates: Flux image automation + Renovate | Accepted | R-UPD-01..03 | +| [0010](adr-0010-conformance-testing.md) | Conformance testing: automated, reproducible, behavior-based | Accepted | R-TEST-01..06 | +| [0011](adr-0011-test-tooling-kyverno-chainsaw.md) | Conformance test tooling: Kyverno + Chainsaw | Accepted | R-TEST-01..05 | diff --git a/decisions/adr-0001-kubernetes-distribution-k3s.md b/decisions/adr-0001-kubernetes-distribution-k3s.md new file mode 100644 index 0000000..202a666 --- /dev/null +++ b/decisions/adr-0001-kubernetes-distribution-k3s.md @@ -0,0 +1,32 @@ +# ADR-0001 — Kubernetes distribution: k3s +- Status: Accepted +- Date: 2026-08-15 +- Serves: R-PLAT-01, R-PLAT-02, R-ENV-01/02/03 +- Supersedes: — + +## Context +The platform runs on 3 small nodes (dev: KVMs; prod: mini-PCs) and must offer a full Kubernetes API +for both containers and VMs. It must be lightweight enough for constrained hardware yet +production-capable, HA, and identical across dev/test and prod. + +## Decision +Use **k3s** as the Kubernetes distribution, deployed as a 3-node HA cluster with **embedded etcd** +(all three nodes are servers). Bundled components that conflict with our chosen stack are disabled at +install time: flannel (→ Cilium), kube-proxy (→ Cilium kube-proxy replacement), Traefik (→ Gateway +API via Cilium), and servicelb where a different LB strategy is selected. Nodes are provisioned by +OpenTofu (libvirt, dev) / bare-metal imaging (prod) + cloud-init, and joined/configured by Ansible. + +## Alternatives considered +- **Full kubeadm/vanilla Kubernetes** — heavier operationally for 3 small nodes; k3s gives the same + API with far less overhead. +- **k0s / MicroK8s / Talos** — all viable. Talos is attractive (immutable, API-driven) and may be + revisited via a superseding ADR for prod hardening; k3s chosen now for its ubiquity, small + footprint, single-binary simplicity, strong community, and the operator's existing familiarity. +- **Managed cloud Kubernetes** — out of scope; this is on-prem home hardware. + +## Consequences +- Lightweight, HA, uniform dev↔prod topology (R-ENV-02). +- Must explicitly disable bundled CNI/ingress/proxy to make room for Cilium + Gateway API (ADR-0002) + — captured in the Ansible bootstrap. +- Embedded etcd across 3 nodes tolerates single-node loss; do not reduce below 3 servers in prod. +- Revisit Talos for prod if immutability/hardening becomes a priority. diff --git a/decisions/adr-0002-cni-cilium.md b/decisions/adr-0002-cni-cilium.md new file mode 100644 index 0000000..8381aed --- /dev/null +++ b/decisions/adr-0002-cni-cilium.md @@ -0,0 +1,37 @@ +# ADR-0002 — CNI, network policy & ingress: Cilium + Gateway API +- Status: Accepted +- Date: 2026-08-15 +- Serves: R-NET-01, R-NET-02, R-NET-03, R-NET-04, R-NET-05, R-NET-06, R-NET-07, R-DNS-01, R-SEC-02 +- Supersedes: — + +## Context +The platform requires: default-deny east-west with explicit port-level allows (firewall-like), +network isolation/segmentation, multi-WAN/egress routing including per-service VPN egress, and +north-south ingress with TLS. These are the most safety-critical requirements. A single, coherent +networking layer that does policy, egress steering, observability, and ingress reduces moving parts. + +## Decision +Use **Cilium** as the CNI (replacing k3s' flannel), with **kube-proxy replacement**, and adopt: +- **CiliumNetworkPolicy / NetworkPolicy** for default-deny + explicit L3/L4 (optionally L7) allows. +- **Cilium Egress Gateway** to steer selected pods' egress through chosen nodes/identities/SNAT IPs + — the mechanism for egress-class routing and part of the VPN model (with ADR-0008). +- **Hubble** for flow observability — the audit/troubleshooting tool for isolation and egress. +- **Gateway API implemented by Cilium** for north-south ingress and TLS termination (no separate + ingress controller). + +## Alternatives considered +- **Calico** — strong policy engine; lacks Cilium's integrated egress gateway + Hubble + Gateway API + in one stack. +- **Flannel + separate ingress-nginx/Traefik + separate egress solution** — more components, no + identity-aware policy or built-in flow visibility. +- **Service mesh (Istio/Linkerd) for policy/mTLS** — heavier; not needed at launch. mTLS can be + added later via Cilium/mesh as a future ADR ([08](../docs/08-tls-and-certificates.md)). + +## Consequences +- One networking layer covers policy, egress steering, observability, and ingress — fewer components, + coherent model, VMs (KubeVirt) inherit the same policy fabric. +- Requires disabling k3s bundled flannel/kube-proxy/Traefik (see ADR-0001). +- Egress Gateway + VPN model must be validated end-to-end including kill-switch (ADR-0008, R-NET-06). +- Physical multi-WAN (dual ISP) remains partly an edge-router responsibility (OQ-1); the cluster + decides egress *class*, the edge router decides physical WAN for the internet class. +- Gateway LB/VIP mechanism (Cilium L2 announcement vs BGP) to be pinned during networking bring-up. diff --git a/decisions/adr-0003-storage-longhorn.md b/decisions/adr-0003-storage-longhorn.md new file mode 100644 index 0000000..f9414ee --- /dev/null +++ b/decisions/adr-0003-storage-longhorn.md @@ -0,0 +1,38 @@ +# ADR-0003 — Distributed storage: Longhorn (+ bulk tier + Velero) +- Status: Accepted +- Date: 2026-08-15 +- Serves: R-STATE-01, R-STATE-02, R-STATE-03, R-STATE-04 +- Supersedes: — + +## Context +Three nodes with **local disks** and no SAN must provide durable, replicated persistent storage so +any workload can be destroyed and rebuilt without data loss, tolerate a single node failure, and +support off-cluster backup/DR. Plex's multi-TB media library makes uniform 3× replication +impractical, requiring a distinct bulk tier. + +## Decision +- Use **Longhorn** for distributed replicated block storage across the 3 nodes; default StorageClass + is **3 replicas**. This is the durable home for app/config state and KubeVirt VM disks (via CDI). +- Define a separate **bulk media tier** for large, non-replicated data (Plex library, download + staging). Its concrete backing (NAS/NFS vs. dedicated large disk vs. mixed) is decided under OQ-2; + **Plex's metadata DB stays on 3-replica Longhorn** regardless. +- Use **Velero** (with Longhorn backups) to push volume + resource-state backups to an **off-cluster** + S3-compatible/NFS target on a schedule, with a **tested restore** path (DR drill). + +## Alternatives considered +- **Rook/Ceph** — more powerful but far heavier/operationally complex for 3 small nodes. +- **OpenEBS (Mayastor/other)** — viable; Longhorn chosen for simplicity, strong UI, snapshot/backup + ergonomics, and fit with small clusters. +- **Uniform 3× replication of everything including media** — rejected: capacity-prohibitive on + mini-PC local disks (R-STATE-04). +- **hostPath/local-only PVs for stateful apps** — rejected: node-bound, no replication, breaks the + rebuild guarantee. + +## Consequences +- Single-node loss keeps replicated data available; workloads rebuild by re-attaching PVCs + (R-STATE-01/02). +- A two-tier storage story must be clearly documented so implementers don't accidentally place bulk + media on the replicated tier or app DBs on the bulk tier ([04](../docs/04-storage-and-state.md)). +- Off-cluster backup target + its credentials/resilience are prerequisites (OQ-2); DR drill is part + of acceptance. +- Longhorn's own capacity/health must be monitored and alerted ([10](../docs/10-observability-and-alerting.md)). diff --git a/decisions/adr-0004-gitops-flux.md b/decisions/adr-0004-gitops-flux.md new file mode 100644 index 0000000..61fed40 --- /dev/null +++ b/decisions/adr-0004-gitops-flux.md @@ -0,0 +1,35 @@ +# ADR-0004 — GitOps engine: Flux CD +- Status: Accepted +- Date: 2026-08-15 +- Serves: R-GITOPS-01, R-GITOPS-02, R-GITOPS-03, R-UPD-02, R-SEC-01, R-ENV-02 +- Supersedes: — + +## Context +All desired state must be reconciled from git (pull-based), with drift correction, environment +overlays (dev↔prod), encrypted secrets, and — critically — **hands-off image updates** driven by +declared policy (R-UPD-02). The GitOps engine choice is tightly coupled to the update mechanism and +the secrets model. + +## Decision +Use **Flux CD** for reconciliation, chosen decisively because it *natively* provides the two +capabilities that matter most for this platform's hard requirements: +- **image-update-automation** (image-reflector + image-automation controllers) for policy-driven, + no-human image updates (R-UPD-02, ADR-0009). +- **native SOPS decryption** for encrypted-in-git secrets (R-SEC-01, ADR-0007). +Flux Kustomize/Helm controllers handle overlays (dev↔prod, R-ENV-02) and chart-based platform +components. + +## Alternatives considered +- **Argo CD** — excellent UI and ApplicationSets; but image auto-update (Argo Image Updater) and + secret decryption are add-ons rather than first-class, and the platform gets its UI from the C2 + composition ([12](../docs/12-command-center-ui.md)) rather than from the GitOps tool. Reasonable + alternative; Flux wins on native image-automation + SOPS fit. +- **Rancher Fleet / Kustomize-only + CI push** — weaker fit for the automated-update and drift + requirements. + +## Consequences +- Native, first-class path to R-UPD-02 (no bolt-on for automated updates) and R-SEC-01 (SOPS). +- GitOps status/health is surfaced in the C2 UI via Headlamp/Flux views ([12](../docs/12-command-center-ui.md)), + not via a bundled GitOps GUI. +- Reconciliation failures/drift are alerting conditions (R-OBS-04). +- If a rich built-in GitOps GUI later becomes a priority, revisit via superseding ADR (Argo). diff --git a/decisions/adr-0005-vm-workloads-kubevirt.md b/decisions/adr-0005-vm-workloads-kubevirt.md new file mode 100644 index 0000000..03de98e --- /dev/null +++ b/decisions/adr-0005-vm-workloads-kubevirt.md @@ -0,0 +1,36 @@ +# ADR-0005 — VM workloads: KubeVirt + CDI +- Status: Accepted +- Date: 2026-08-15 +- Serves: R-PLAT-03, R-PLAT-04, R-STATE-01 +- Supersedes: — + +## Context +Not everything fits in a container. The platform must run VMs, but the operator must manage them +through the **same** Kubernetes API/CLI/YAML/GitOps as containers, with the same network isolation, +egress classes, storage durability, and observability — no separate hypervisor management plane for +application workloads. + +## Decision +Use **KubeVirt** to run application VMs as Kubernetes custom resources (`VirtualMachine` / +`VirtualMachineInstance`), with **CDI (Containerized Data Importer)** to provision VM disks onto +**Longhorn** PVCs. VMs run on the same Cilium network (inheriting default-deny + egress classes) and +are managed by Flux like any other workload. `virtctl` provides console/lifecycle helpers alongside +`kubectl`. + +## Alternatives considered +- **Proxmox / standalone hypervisor + separate IaC** — a second management plane and tooling; breaks + the "same Kubernetes surface" requirement (R-PLAT-03). (The operator's prior environment used + Proxmox; this platform deliberately unifies on Kubernetes instead.) +- **Harvester (SUSE)** — HCI built on KubeVirt; heavier and opinionated about the whole node OS; k3s + + KubeVirt keeps one lightweight cluster for both containers and VMs. +- **Containers only** — rejected: the requirement explicitly needs a VM escape hatch. + +## Consequences +- One control plane, one GitOps flow, one network/storage/observability model for containers *and* + VMs. +- Nodes must expose virtualization extensions (nested virt in dev KVMs; native on prod) — a node + baseline prerequisite ([02](../docs/02-environments-and-provisioning.md)). +- VMs are heavier than pods; capacity planning on the 3-node cluster must account for them. +- At least one representative VM workload must be proven in dev/test so the IaaS path is real + ([06](../docs/06-vm-workloads-kubevirt.md)). +- VM guest updates use in-guest automation (R-UPD-*), distinct from container image tag bumps. diff --git a/decisions/adr-0006-tls-cert-manager-letsencrypt.md b/decisions/adr-0006-tls-cert-manager-letsencrypt.md new file mode 100644 index 0000000..1182f5b --- /dev/null +++ b/decisions/adr-0006-tls-cert-manager-letsencrypt.md @@ -0,0 +1,33 @@ +# ADR-0006 — TLS: cert-manager + Let's Encrypt (DNS-01) +- Status: Accepted +- Date: 2026-08-15 +- Serves: R-TLS-01, R-TLS-02, R-TLS-03, R-DNS-01 +- Supersedes: — + +## Context +Every service needs a **trusted, non-self-signed** certificate, auto issued/renewed/revoked, +including **internal-only** services (which must not be forced public just to validate a cert). +Wildcard coverage of `*.home.keithjackson.dev` would make onboarding certificate-free. + +## Decision +Use **cert-manager** with **Let's Encrypt** (a publicly trusted, free, automatable CA) via the +**ACME DNS-01** challenge against the `keithjackson.dev` DNS zone (Cloudflare or equivalent, OQ-4). +DNS-01 validates domain control without exposing the service and enables **wildcard** certs. +Default: a **wildcard `*.home.keithjackson.dev`** certificate terminated at the Gateway ([09](../docs/09-dns-and-ingress.md)), +with per-service certificates available where isolation warrants. Provide both `letsencrypt-staging` +(bring-up) and `letsencrypt-prod` ClusterIssuers. + +## Alternatives considered +- **HTTP-01 challenge** — rejected: requires public exposure of each service; fails R-TLS-03 for + internal-only services and can't do wildcards. +- **Self-signed / private CA** — rejected: explicitly disallowed (R-TLS-01, "not self-signed"). +- **Paid CA** — unnecessary; Let's Encrypt is trusted and free. +- **ZeroSSL/other ACME CA** — viable fallback; Let's Encrypt chosen as default. + +## Consequences +- Internal services get real, trusted certs; wildcard makes onboarding certificate-free (default). +- Requires a DNS-provider API token scoped for DNS-01, stored SOPS-encrypted (OQ-4, ADR-0007). +- Certificate expiry/issuance failures are alert conditions (R-OBS-04). +- Internal clients trust the certs with no custom-CA distribution; split-horizon DNS serves the same + cert internally and externally. +- Optional future mTLS between workloads is a separate concern (Cilium/mesh), not covered here. diff --git a/decisions/adr-0007-secrets-sops-age.md b/decisions/adr-0007-secrets-sops-age.md new file mode 100644 index 0000000..4fc403d --- /dev/null +++ b/decisions/adr-0007-secrets-sops-age.md @@ -0,0 +1,31 @@ +# ADR-0007 — Secrets: SOPS + age +- Status: Accepted +- Date: 2026-08-15 +- Serves: R-SEC-01, R-GITOPS-01 +- Supersedes: — + +## Context +Secrets (VPN keys, DNS API token, SMTP creds, service passwords, backup-target creds) must live in +git for GitOps but MUST NOT be plaintext. The engine is Flux (ADR-0004), which decrypts SOPS +natively. + +## Decision +Encrypt secrets in git with **SOPS** using **age** recipients; **Flux decrypts in-cluster** at +reconciliation via an age private key bootstrapped as a Kubernetes Secret. Plaintext secrets never +enter git or logs. Secret rotation = re-encrypt + commit; Flux rolls it out (restart triggered by +checksum annotation where needed). + +## Alternatives considered +- **Sealed Secrets (Bitnami)** — solid; one-way encryption to a cluster key. SOPS chosen for native + Flux integration, multi-recipient/age simplicity, and readable, reviewable encrypted diffs. +- **External Secrets Operator + external vault (Vault/cloud SM)** — stronger for large orgs but adds + an external dependency and operational weight not justified for a single-operator home platform. + May be revisited via superseding ADR if a real external secret store is introduced. +- **Plaintext + private repo** — rejected outright (R-SEC-01). + +## Consequences +- Secrets are versioned, encrypted, and reconciled with the rest of GitOps state. +- The **age private key bootstrap** is a sanctioned out-of-band step (documented; one of the few + non-GitOps actions, [07](../docs/07-security-and-secrets.md)). +- Key custody/rotation for the age key itself must be documented; losing it blocks decryption. +- No secret material may appear in agent output, PRs, or logs ([AGENTS.md §5](../AGENTS.md)). diff --git a/decisions/adr-0008-vpn-egress-model.md b/decisions/adr-0008-vpn-egress-model.md new file mode 100644 index 0000000..a37d215 --- /dev/null +++ b/decisions/adr-0008-vpn-egress-model.md @@ -0,0 +1,42 @@ +# ADR-0008 — VPN egress model: Gluetun sidecar (default) + Cilium Egress Gateway +- Status: Accepted +- Date: 2026-08-15 +- Serves: R-NET-05, R-NET-06, R-NET-07 +- Supersedes: — + +## Context +A subset of services (sonarr, radarr, jackett, qbittorrent) must egress **only** through a commercial +VPN, with a **kill-switch** guaranteeing no leak to the plain internet if the tunnel drops. Egress +class must be a declarative service property (`egress: vpn`) that doesn't force a workload rewrite. + +## Decision +Adopt a **two-pattern** model behind the single `egress: vpn` contract, with **Gluetun sidecar as the +default** for the initial catalog: +- **Default — Pattern A: Gluetun sidecar per pod.** A Gluetun container holds the WireGuard/OpenVPN + tunnel; the app shares its network namespace. Gluetun's built-in **firewall kill-switch** blocks + all non-tunnel egress by default — satisfying R-NET-06 intrinsically and per-service. +- **Available — Pattern B: central VPN egress gateway + Cilium Egress Gateway.** One (HA) gateway + holds the tunnel; Cilium Egress Gateway steers `vpn`-labeled pods through it, with policy denying + any fallback to the internet path. Use when tunnel count must be minimized. + +The service-facing contract is identical for both: declare `egress: vpn`; the platform wires the +chosen pattern. Kill-switch behavior is a hard acceptance gate for both. + +## Alternatives considered +- **Central gateway only** — fewer tunnels but a shared failure domain and more complex kill-switch + guarantees across the fabric; kept as Pattern B, not the default. +- **Node-level policy routing / per-node VPN** — couples egress to node identity, harder to reason + about per-service; rejected as the primary model. +- **App-native VPN config** — rejected: not declarative, not uniform, easy to misconfigure and leak. + +## Consequences +- Per-service isolation and an intrinsic kill-switch out of the box (Pattern A). +- More sidecars/tunnels at scale (Pattern A) — Pattern B is the escape hatch if the VPN provider + limits concurrent tunnels/egress IPs (OQ-3). +- Kill-switch must be **fault-tested** (tunnel forced down ⇒ zero internet egress) as a standing + acceptance gate, not a one-time check ([03](../docs/03-networking-isolation-and-egress.md), R-NET-06). +- VPN credentials are SOPS-encrypted (ADR-0007). WireGuard preferred over OpenVPN where supported. + +## Open questions +- **OQ-3** — VPN provider/protocol and concurrent-tunnel/egress-IP limits (drives Pattern A vs B and + gateway HA). diff --git a/decisions/adr-0009-automated-image-updates.md b/decisions/adr-0009-automated-image-updates.md new file mode 100644 index 0000000..09132b1 --- /dev/null +++ b/decisions/adr-0009-automated-image-updates.md @@ -0,0 +1,43 @@ +# ADR-0009 — Automated updates: Flux image automation + Renovate +- Status: Accepted +- Date: 2026-08-15 +- Serves: R-UPD-01, R-UPD-02, R-UPD-03 +- Supersedes: — + +## Context +Services must advance to the **latest stable** version **on a schedule** with **no human, no AI, and +no hand-authored GitOps change** (R-UPD-02). This must coexist with GitOps (git = source of truth) +and must be safe (stable-only, health-gated, reversible). + +## Decision +Split by artifact type: +- **Container image tags → Flux image-update-automation.** Per-service **ImagePolicy** declares the + acceptable range/channel (semver range or stable tag regex). Flux's image-reflector scans the + registry on an interval (the schedule) and image-automation advances the running version. The + policy is the human-authored intent; the version advance is machine-performed — no human/AI edits + a tag (R-UPD-02). +- **Helm charts / manifests → Renovate.** Renovate opens version-bump PRs; allowed update classes + **auto-merge** on green CI. Major/breaking bumps may require review per policy. +- **VM guests → in-guest auto-update** (unattended-upgrades/dnf-automatic) declared in the VM's + cloud-init ([06](../docs/06-vm-workloads-kubevirt.md)). + +**Default write-back behavior:** image-automation commits the resolved version back to git (machine- +authored) so history is auditable; a **per-service escape hatch** allows digest-tracking of a mutable +stable tag for zero-commit updates where the operator prefers it. Per-service `update` policy +(channel, range, cadence, pin, autoRollback) lives in the ServiceDefinition ([13](../docs/13-service-catalog.md)). + +## Alternatives considered +- **Argo CD Image Updater** — bolt-on to Argo; we chose Flux (ADR-0004) whose image automation is + first-class. +- **`latest`/`stable` mutable tag + scheduled rollout restart only** — simplest but loses version + visibility and semver guardrails; offered only as the per-service digest-tracking escape hatch. +- **Human/AI-authored bumps** — explicitly prohibited by R-UPD-02. + +## Consequences +- Hands-off updates satisfied via declared policy, not manual/AI edits (R-UPD-02). +- Safety rails required and are acceptance gates: **stable-only**, **health-gated rollout**, + **auto-rollback** on failed health, **pin** escape hatch, and **alerting** on update failure + ([10](../docs/10-observability-and-alerting.md), [11](../docs/11-automated-updates.md)). +- Registry rate-limits/credentials for cadence scanning must be handled. +- Machine write-back commits will appear in history — an intended, auditable artifact, not a + violation of "no hand-authored change." diff --git a/decisions/adr-0010-conformance-testing.md b/decisions/adr-0010-conformance-testing.md new file mode 100644 index 0000000..dedf84e --- /dev/null +++ b/decisions/adr-0010-conformance-testing.md @@ -0,0 +1,48 @@ +# ADR-0010 — Conformance testing: automated, reproducible, behavior-based +- Status: Accepted +- Date: 2026-08-15 +- Serves: R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04, R-TEST-05, R-TEST-06 +- Supersedes: — + +## Context +Acceptance must be objective and repeatable, not a human eyeballing a demo. Every requirement +(`R-*`) must be verified by an automated behavior test that any machine can run and reproduce, that +gates merges and phase exits, and that traces back to the requirement it proves. Safety-critical +behaviors (default-deny, VPN kill-switch, data survival, DR, auto-rollback) demand fault-injection, +not happy-path checks. Additionally, agents must not weaken the target: specs are immutable to +agents; tests encode spec intent. + +## Decision +Adopt an **automated, reproducible, behavior-based conformance suite** as the sole definition of +acceptance, structured in layers (L0 static/policy, L1 rendering, L2 behavioral, L3 fault-injection, +L4 aggregate suite), specified in [`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md). +The suite is **requirement-tagged**, **single-command**, **hermetic**, **deterministic**, and emits +**JUnit + a requirement-coverage report**. It gates: +- **merges** — the subset for the touched requirements must pass; +- **phase exits** — the phase's full suite must pass. + +Tooling: the *approach* is fixed; the concrete mix is **policy-as-code** for L0 (e.g. Kyverno / +Conftest-OPA), a **declarative Kubernetes test runner** for L1/L2 (e.g. Chainsaw or kuttl), and a +**small programmatic harness** (e.g. Ginkgo/Gomega or a scripted runner) for L3 fault-injection that +needs orchestration. The final tool selection is the one remaining open item and may be pinned by a +short follow-up (still human-approved), but no choice may relax the properties above. + +## Alternatives considered +- **Manual demonstrations / checklists** — rejected: not reproducible, not gating, not traceable + (violates R-TEST-01..04). +- **Unit tests of manifests only (L0/L1)** — necessary but insufficient: a policy object existing + does not prove enforcement; behavior (L2) and fault-injection (L3) are required (R-TEST-05). +- **A single heavyweight framework for everything** — rejected: L3 fault-injection needs + orchestration a declarative runner can't express, while L0/L1/L2 are cleaner declaratively. A + layered mix fits better. +- **Letting agents adjust tests/requirements to pass** — explicitly prohibited (R-TEST-06); a + failing spec is an escalation, not a test edit. + +## Consequences +- "Done" becomes objective and machine-checkable; progress = green, requirement-tagged suite. +- Every requirement needs a test; uncovered requirements are visible and blocking (R-TEST-04). +- Establishing L3 fault-injection (kill-switch, node-kill, bad-update, DR) is real work and is + first-class in the roadmap (WS-TEST), not an afterthought. +- CI must have cluster access (or spin an ephemeral dev/test cluster) to run L2/L3. +- The suite is versioned with the implementation and evolves only when the human-owned spec it + encodes changes. diff --git a/decisions/adr-0011-test-tooling-kyverno-chainsaw.md b/decisions/adr-0011-test-tooling-kyverno-chainsaw.md new file mode 100644 index 0000000..a7627b7 --- /dev/null +++ b/decisions/adr-0011-test-tooling-kyverno-chainsaw.md @@ -0,0 +1,67 @@ +# ADR-0011 — Conformance test tooling: Kyverno + Chainsaw +- Status: Accepted +- Date: 2026-08-15 +- Serves: R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04, R-TEST-05 +- Refines: [ADR-0010](adr-0010-conformance-testing.md) (which fixed the *approach* and left the tool mix as the one open item) +- Supersedes: — + +## Context +[ADR-0010](adr-0010-conformance-testing.md) fixed the conformance approach (automated, reproducible, +behavior-based, layered, requirement-tagged, gating) and left the concrete tool selection as the sole +open item. A choice is now made by the operator/architect. + +## Decision +Use **two tools from the Kyverno ecosystem**, covering all test layers: + +- **Kyverno** — the **L0 static/policy gate** (policy-as-code) **and in-cluster admission enforcer**. + The Kyverno CLI (`kyverno apply` / `kyverno test`, incl. `kyverno-json` for arbitrary YAML/JSON + like ServiceDefinitions) validates manifests and rendered resources in CI against platform + invariants: every namespace carries a default-deny baseline, no plaintext secrets, required + labels/egress-class set, ServiceDefinition schema conformance, exposure/egress sanity. **The same + policies also run in-cluster as an admission controller**, rejecting non-conforming resources at + reconcile time (defense in depth) — authored once, shared between CLI gate and admission. +- **Chainsaw** — the **L1 rendering, L2 behavioral, and L3 fault-injection** runner. Chainsaw's + declarative test steps (apply/assert/error, `script`/`command`, `try`/`catch`/`finally`), bounded + polling with explicit timeouts, per-test namespace isolation, and automatic cleanup directly + satisfy the reproducibility contract (hermetic, deterministic, self-cleaning). Fault-injection + (force the VPN tunnel down, cordon/kill a node, inject a known-bad image, trigger a restore) is + expressed via Chainsaw `script` steps, with the fail-closed / no-data-loss outcome asserted by + probe fixtures and `finally` steps restoring state. + +Both emit structured, requirement-tagged results feeding the coverage report and the CI merge / phase +exit gates ([ADR-0010](adr-0010-conformance-testing.md), [`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md)). +Tests declare their `R-*` IDs (e.g. via labels/annotations/naming) so coverage is generated +automatically. + +## Alternatives considered +- **kuttl** (for L1/L2) — Chainsaw is its spiritual successor with a richer assertion model, native + scripting (needed for L3), and try/catch/finally cleanup; chosen over kuttl. +- **Conftest/OPA (Rego)** for L0 — viable, but Kyverno keeps policy-as-code in the same + YAML/ecosystem as Chainsaw and can double as an in-cluster enforcer; chosen over introducing Rego. +- **Ginkgo/Gomega (Go)** for L3 — more power but more code and a second language/toolchain; Chainsaw + `script` steps cover the platform's fault-injection needs declaratively. Revisit via a superseding + ADR only if a scenario genuinely exceeds Chainsaw's orchestration. +- **Testkube** as an orchestrator — unnecessary layer for this scale; CI + Chainsaw + Kyverno CLI + suffice. + +## Consequences +- A single, coherent ecosystem (Kyverno project) spans policy enforcement and behavior/fault tests — + fewer tools, shared idioms. +- L3 fault-injection is expressed in Chainsaw `script` steps. **L2/L3 run as an in-cluster Chainsaw + Job** on the dev/test cluster, **triggered by CI** — so CI holds no cluster credentials; the Job's + own scoped ServiceAccount does. L0/L1 still run in CI without a cluster. (Resolves ADR-0010's open + harness-location item.) +- The test Job's ServiceAccount is a **privileged, scoped test identity** (create/delete test + namespaces + fixtures, patch NetworkPolicies, scale VPN sidecar/gateway, cordon/kill a node, inject + a bad image, trigger a Velero restore). It must be least-privilege for that set, named, documented, + never reused by workloads, and treated as its own audit item (R-SEC-02). +- Kyverno runs **both** as the CI policy gate **and** as an in-cluster admission enforcer (decided): + non-conforming resources are rejected at reconcile time as well as in CI. This must not *replace* + behavior tests — a policy admitting a manifest is not proof the runtime behaves (R-TEST-01), so + L2/L3 Chainsaw tests remain mandatory. Admission enforcement adds an operational failure mode + (a bad policy can block legitimate reconciliation); policies ship with their own tests and roll out + via GitOps. +- Reproducibility properties (hermetic, deterministic, self-cleaning, requirement-tagged, JUnit + + coverage) remain the acceptance bar; the tools serve them, never relax them. +- Pin tool versions (Kyverno, Chainsaw) for deterministic runs; upgrades flow through the normal + update path with the suite re-green as the gate. diff --git a/docs/00-vision-and-scope.md b/docs/00-vision-and-scope.md new file mode 100644 index 0000000..d317e7a --- /dev/null +++ b/docs/00-vision-and-scope.md @@ -0,0 +1,189 @@ +# 00 — Vision & Scope + +## Vision + +Build a home platform that behaves like a small private cloud. An operator declares a service — a +container app *or* a virtual machine — in git, and the platform provisions it, secures it, exposes +it under a trusted TLS URL, routes its traffic on the correct WAN, protects its data, updates it +automatically, and reports on its health. Onboarding a new service is a pull request against a +catalog, not a weekend of manual work. + +The platform is delivered in two stages: + +- **dev/test** on local KVMs — the active build target, used to prove viability. +- **prod** on 3 mini-PCs — the eventual home, designed for now but not built on yet. + +## Personas + +- **Operator (primary user)**: runs the home network; wants to onboard/offboard services with + minimal ceremony and zero data loss. +- **Platform (the system)**: reconciles declared state, enforces isolation and egress policy, + renews certificates, updates workloads, and raises alerts. +- **Implementation agents**: the AI teams building the platform from this spec. + +## In scope + +- A k3s-based platform with distributed storage (Longhorn) across 3 nodes. +- Container workloads (PaaS) and VM workloads (IaaS) through one Kubernetes surface. +- GitOps delivery, automated updates, trusted TLS, DNS, ingress. +- Network isolation (default-deny east-west) and multi-WAN egress including VPN routing. +- Observability with email + dashboard alerting. +- A command-and-control web UI at the root of `home.keithjackson.dev`. +- The initial service catalog (13 services, see [`13-service-catalog.md`](13-service-catalog.md)). + +## Out of scope (for now) + +- Building on production mini-PC hardware (design-only until the roadmap flips it to active). +- Multi-tenant / multi-operator RBAC beyond a single operator + platform automation. +- Public multi-user SaaS concerns (billing, per-tenant isolation beyond network policy). +- Non-`keithjackson.dev` domains. + +## The domain & DNS model + +- Public apex `*.keithjackson.dev` is available and its records resolve to the network's public IP. +- All platform services live under `*.home.keithjackson.dev`. +- Each service is reachable at `<service>.home.keithjackson.dev` (e.g. `plex.home.keithjackson.dev`). +- The **root** `home.keithjackson.dev` serves the **Command & Control (C2) center** UI. + +## Requirements (stable IDs) + +Requirement IDs are stable references used across ADRs, specs, the task board, and PRs. `MUST` is +mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate. + +### Platform & API surface — `R-PLAT-*` +- **R-PLAT-01** — The platform MUST run on k3s across 3 nodes with production-grade HA control plane + in prod; dev/test MAY run fewer nodes but MUST exercise the same multi-node topology. +- **R-PLAT-02** — Container workloads MUST be onboarded and managed declaratively via the Kubernetes + API, `kubectl`-class CLI, and YAML. +- **R-PLAT-03** — Virtual-machine workloads MUST be created, managed, and destroyed through the + *same* Kubernetes API/CLI/YAML surface as containers (no separate hypervisor management plane for + application VMs). +- **R-PLAT-04** — The platform MUST support create / destroy / rebuild of any workload on demand. + +### GitOps & delivery — `R-GITOPS-*` +- **R-GITOPS-01** — All application and platform desired state MUST be defined in git and reconciled + automatically (pull-based GitOps). +- **R-GITOPS-02** — Onboarding a new service MUST be expressible as a change to a versioned catalog + in git. +- **R-GITOPS-03** — The system MUST detect and correct drift between git and cluster state. + +### Networking, isolation & egress — `R-NET-*` +- **R-NET-01** — East-west traffic between services MUST be **denied by default**; communication is + allowed only by explicit, port-level configuration (firewall-like). +- **R-NET-02** — Allow rules MUST be declarative, reviewable in git, and scoped per source → + destination → port/protocol. +- **R-NET-03** — Networks MUST be configurable and isolatable (segments/namespaces with enforced + boundaries). +- **R-NET-04** — The platform MUST support **multiple WANs** with egress selection driven by routing + rules. +- **R-NET-05** — A service MUST be able to declare its **egress class**: `internet` (standard local + network/internet) or `vpn` (routed through a custom VPN connection). +- **R-NET-06** — VPN-class egress MUST enforce a **kill-switch**: if the VPN tunnel is down, the + service MUST NOT leak traffic to the standard internet. +- **R-NET-07** — Egress class MUST be a declarative property of the service, changeable via GitOps + without re-architecting the workload. + +### Storage & state — `R-STATE-*` +- **R-STATE-01** — Stateful data MUST survive destroy/rebuild of the workload (PaaS and IaaS). +- **R-STATE-02** — Persistent storage MUST be distributed/replicated across the 3 nodes (Longhorn) + so a single node loss does not lose data. +- **R-STATE-03** — The platform MUST provide scheduled backups to an **off-cluster** target and a + tested restore path (disaster recovery). +- **R-STATE-04** — Bulk media storage (Plex library) MUST have a capacity strategy distinct from + replicated app-config storage (replicating multi-TB media 3× is out of scope; see ADR-0003). + +### Automated updates — `R-UPD-*` +- **R-UPD-01** — Services MUST update to the latest **stable** versions on a schedule. +- **R-UPD-02** — Updates MUST require **no human, no AI, and no hand-authored GitOps change** to + occur. Automation controllers acting on declarative policy are the permitted mechanism. +- **R-UPD-03** — Update policy MUST be per-service configurable (cadence, stability channel, + pin/hold, auto-rollback on failed health). + +### TLS & certificates — `R-TLS-*` +- **R-TLS-01** — Every externally reachable service MUST present TLS from a **credible, trusted, + non-self-signed** authority. +- **R-TLS-02** — Certificates MUST be automatically **issued, renewed, and revoked** without manual + steps. +- **R-TLS-03** — Certificate issuance MUST work for internal-only services (no requirement to expose + a service publicly just to validate a certificate). + +### DNS & ingress — `R-DNS-*` +- **R-DNS-01** — Services MUST be reachable at `<service>.home.keithjackson.dev`. +- **R-DNS-02** — DNS records MUST be managed automatically from declared service intent. +- **R-DNS-03** — Internal clients MUST resolve service names correctly (split-horizon where public + and internal answers differ). + +### Observability & alerting — `R-OBS-*` +- **R-OBS-01** — Services MUST be monitored (metrics, logs, health). +- **R-OBS-02** — Issues MUST be reported by **email**. +- **R-OBS-03** — Issues and health MUST be visible on **dashboards**. +- **R-OBS-04** — The platform MUST alert on its own health (nodes, storage, GitOps reconciliation, + certificate expiry, VPN tunnel state, backup success). + +### Command & control — `R-C2-*` +- **R-C2-01** — The root `home.keithjackson.dev` MUST serve a UI that surfaces the services running + in the cluster and links to them. +- **R-C2-02** — The C2 UI MUST surface platform health (cluster, storage, GitOps, alerts) and + provide controlled operational actions where safe. + +### Security & secrets — `R-SEC-*` +- **R-SEC-01** — Secrets MUST NOT be stored in plaintext in git; they MUST be encrypted at rest in + the repo and decrypted only in-cluster. +- **R-SEC-02** — The platform MUST follow least-privilege for workload identities and network reach. +- **R-SEC-03** — All the above properties MUST be demonstrable/auditable (an operator can prove + isolation, egress path, cert validity, and backup integrity). + +### Environments — `R-ENV-*` +- **R-ENV-01** — dev/test MUST be provisioned as local KVMs via IaC: created, baselined, cluster- + joined, and managed as code. +- **R-ENV-02** — The same GitOps configuration MUST promote from dev/test to prod with only + environment-scoped overrides. +- **R-ENV-03** — prod is a 3× mini-PC bare-metal k3s cluster with local storage; it MUST NOT be an + active development surface until the roadmap declares it ready. + +### Conformance testing & governance — `R-TEST-*` +- **R-TEST-01** — Every requirement MUST be verified by **automated, behavior-based tests** + (Given/When/Then style) that assert observable system behavior, not implementation internals. +- **R-TEST-02** — Tests MUST be **reproducible**: deterministic, hermetic (create and tear down + their own fixtures), version-pinned, and runnable by a single command locally and in CI against + any conforming cluster, producing machine-readable, requirement-tagged results. +- **R-TEST-03** — A **conformance suite** MUST gate delivery: merges are blocked unless the tests + for the touched requirements pass, and phase exits are blocked unless the phase's full suite + passes. "Demonstrated" means "the automated test passes," never a manual one-off. +- **R-TEST-04** — Tests MUST be **traceable**: each test declares the requirement ID(s) it verifies, + and a coverage report maps every `R-*` requirement to its verifying test(s). Uncovered + requirements are a gap to close, not an acceptable state. +- **R-TEST-05** — Safety-critical behaviors (default-deny, VPN kill-switch, data-survival on + rebuild, DR restore, auto-rollback) MUST be verified by **fault-injection** behavior tests that + prove fail-closed / no-data-loss outcomes, not merely happy-path checks. +- **R-TEST-06 (governance)** — **AI agents MUST NOT modify the specifications** (this `docs/` tree, + the requirements, or accepted ADRs). Specs are changed only by the human operator/architect. + Agents that find a spec wrong, blocking, ambiguous, or missing MUST **stop and escalate** and MAY + draft a proposal, but MUST NOT self-approve or merge a spec/ADR change (see + [`AGENTS.md`](../AGENTS.md) §2 and §11). + +## Acceptance philosophy + +Each requirement class maps to an **automated, reproducible, behavior-based conformance test** in +dev/test (`R-TEST-*`). "Done" means *the conformance test for the touched requirements passes* — a +green, reproducible result traced to requirement IDs — never "manifest merged" and never a manual +demonstration. The testing approach is specified in +[`docs/14-testing-and-conformance.md`](14-testing-and-conformance.md) and +[ADR-0010](../decisions/adr-0010-conformance-testing.md); the gate checklist lives in +[`agents/definition-of-done.md`](../agents/definition-of-done.md). + +**Governance:** the specifications are human-owned and **immutable to agents** (R-TEST-06). Agents +implement against the spec and prove conformance with tests; they never edit the spec to fit the +implementation. + +## Open questions (tracked, not blocking the architecture) + +- **OQ-1 (WAN hardware)**: How many physical WAN uplinks exist in prod, and does the edge router do + the multi-WAN policy routing, or is a portion done in-cluster via egress gateways? See + [`03-networking-isolation-and-egress.md`](03-networking-isolation-and-egress.md). +- **OQ-2 (bulk media)**: Final home for the Plex media library — NAS/NFS vs. a dedicated large disk + vs. object storage. See [`04-storage-and-state.md`](04-storage-and-state.md) / ADR-0003. +- **OQ-3 (VPN provider)**: Commercial VPN provider + protocol (WireGuard preferred) and how many + concurrent tunnels/egress IPs are available. +- **OQ-4 (DNS provider API)**: Confirm Cloudflare (or equivalent) manages `keithjackson.dev` with an + API token scoped for ACME DNS-01 and ExternalDNS. diff --git a/docs/01-architecture-overview.md b/docs/01-architecture-overview.md new file mode 100644 index 0000000..e5a19e5 --- /dev/null +++ b/docs/01-architecture-overview.md @@ -0,0 +1,109 @@ +# 01 — Architecture Overview + +This document is the map. It names the chosen stack, shows how the layers fit, and points to the +ADR that locks in each choice and the domain spec that details it. **Do not change a technology +choice here without a superseding ADR.** + +## The stack at a glance + +| Layer | Choice | Serves | ADR | Spec | +|---|---|---|---|---| +| Kubernetes distro | **k3s** (HA, embedded etcd) | R-PLAT-01/02 | [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md) | [02](02-environments-and-provisioning.md) | +| CNI / network policy / egress | **Cilium** (+ Hubble, Gateway API, Egress Gateway) | R-NET-01..07 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [03](03-networking-isolation-and-egress.md) | +| Distributed storage | **Longhorn** | R-STATE-01..03 | [ADR-0003](../decisions/adr-0003-storage-longhorn.md) | [04](04-storage-and-state.md) | +| GitOps engine | **Flux CD** | R-GITOPS-01..03 | [ADR-0004](../decisions/adr-0004-gitops-flux.md) | [05](05-gitops-and-delivery.md) | +| VM workloads (IaaS) | **KubeVirt** (+ CDI) | R-PLAT-03 | [ADR-0005](../decisions/adr-0005-vm-workloads-kubevirt.md) | [06](06-vm-workloads-kubevirt.md) | +| TLS | **cert-manager** + Let's Encrypt (DNS-01) | R-TLS-01..03 | [ADR-0006](../decisions/adr-0006-tls-cert-manager-letsencrypt.md) | [08](08-tls-and-certificates.md) | +| Secrets | **SOPS + age** (Flux-native) | R-SEC-01 | [ADR-0007](../decisions/adr-0007-secrets-sops-age.md) | [07](07-security-and-secrets.md) | +| VPN egress | **Gluetun sidecar** + **Cilium Egress Gateway** | R-NET-05/06 | [ADR-0008](../decisions/adr-0008-vpn-egress-model.md) | [03](03-networking-isolation-and-egress.md) | +| Automated updates | **Flux image-update-automation** (images) + **Renovate** (charts) | R-UPD-01..03 | [ADR-0009](../decisions/adr-0009-automated-image-updates.md) | [11](11-automated-updates.md) | +| DNS records | **ExternalDNS** (public via Cloudflare) + **PiHole** (internal) | R-DNS-01..03 | — | [09](09-dns-and-ingress.md) | +| Ingress (north-south) | **Gateway API** implemented by Cilium | R-DNS-01, R-TLS-01 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [09](09-dns-and-ingress.md) | +| Observability | **kube-prometheus-stack** + **Loki** + **Grafana Alloy** + Alertmanager→SMTP | R-OBS-01..04 | — | [10](10-observability-and-alerting.md) | +| Backup/DR | **Velero** + Longhorn backups → off-cluster S3/NFS | R-STATE-03 | [ADR-0003](../decisions/adr-0003-storage-longhorn.md) | [04](04-storage-and-state.md) | +| Node/infra provisioning | **OpenTofu** (libvirt) + **cloud-init** + **Ansible** (k3s bootstrap) | R-ENV-01..03 | [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md) | [02](02-environments-and-provisioning.md) | +| Command center UI | **Homepage** + **Headlamp** + **Grafana** panels | R-C2-01/02 | — | [12](12-command-center-ui.md) | +| Conformance testing | **Kyverno** (L0 policy) + **Chainsaw** (L1/L2/L3 behavior + fault-injection) | R-TEST-01..06 | [ADR-0010](../decisions/adr-0010-conformance-testing.md), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md) | [14](14-testing-and-conformance.md) | + +## Layer cake + +``` +┌──────────────────────────────────────────────────────────────────────────┐ +│ Command & Control UI (home.keithjackson.dev root) │ +│ Homepage catalog · Headlamp k8s console · Grafana dashboards · Flux status │ +├──────────────────────────────────────────────────────────────────────────┤ +│ Workloads │ +│ PaaS: containers (Plex, PiHole, Nextcloud, Gitea, Seerr, …) │ +│ IaaS: KubeVirt VMs (anything that can't be a container) │ +├──────────────────────────────────────────────────────────────────────────┤ +│ Platform services │ +│ Flux (GitOps) · cert-manager (TLS) · ExternalDNS · Gateway API │ +│ kube-prometheus-stack + Loki + Alloy (observability) │ +│ Velero (backup) · Renovate (chart updates) · SOPS (secrets) │ +├──────────────────────────────────────────────────────────────────────────┤ +│ Storage: Longhorn (replicated block) + bulk media store (see ADR-0003) │ +├──────────────────────────────────────────────────────────────────────────┤ +│ Network: Cilium (default-deny NetworkPolicy · Egress Gateway · Hubble) │ +│ VPN egress (Gluetun sidecars / VPN gateway) · Multi-WAN routing │ +├──────────────────────────────────────────────────────────────────────────┤ +│ Kubernetes: k3s (3 nodes, HA control plane, embedded etcd) │ +├──────────────────────────────────────────────────────────────────────────┤ +│ Nodes │ +│ dev/test: libvirt KVMs (OpenTofu + cloud-init + Ansible) │ +│ prod: 3× mini-PC bare metal, local disks │ +├──────────────────────────────────────────────────────────────────────────┤ +│ Edge: router/firewall — multi-WAN uplinks, port-forward to Gateway, │ +│ VPN uplink(s), split-horizon DNS │ +└──────────────────────────────────────────────────────────────────────────┘ +``` + +## Two clearly-separated virtualization layers (do not conflate) + +This platform has **two** kinds of VM, and confusing them is the most common design error: + +1. **Infrastructure VMs = the cluster nodes themselves.** + In dev/test these are libvirt KVMs on the workstation, provisioned by OpenTofu + cloud-init and + configured/joined by Ansible. In prod these are the bare-metal mini-PCs. See + [`02-environments-and-provisioning.md`](02-environments-and-provisioning.md). + +2. **Application VMs = IaaS workloads that run *inside* the cluster via KubeVirt.** + These are managed by the same Kubernetes API/YAML/GitOps as containers. They satisfy R-PLAT-03 + ("not everything fits in a container"). See [`06-vm-workloads-kubevirt.md`](06-vm-workloads-kubevirt.md). + +> Layer 1 is how the platform exists. Layer 2 is a feature the platform offers. + +## How a service flows through the platform (the golden path) + +1. Operator adds/edits a **ServiceDefinition** in the git catalog (see + [`13-service-catalog.md`](13-service-catalog.md)) declaring: kind (container|vm), image/source, + storage, **egress class** (`internet`|`vpn`), exposure (`internal`|`public`), dependencies + (explicit allow rules), update policy, and observability expectations. +2. **Flux** reconciles the change into the cluster. +3. **Longhorn** provisions replicated persistent volumes; state is durable. +4. **Cilium** applies default-deny plus the explicitly declared allow rules; assigns the correct + **egress path** (local internet or VPN, with kill-switch). +5. **ExternalDNS** publishes `<service>.home.keithjackson.dev`; **Gateway API** routes north-south + traffic; **cert-manager** issues a trusted TLS cert via ACME DNS-01. +6. **kube-prometheus-stack + Loki** scrape metrics/logs; **Alertmanager** emails on issues; the + **C2 UI** lists and links the service. +7. **Flux image automation / Renovate** keep the service on the latest stable version per policy — + no human action. +8. **Velero + Longhorn backups** protect the data; destroy/rebuild restores state. + +Every numbered step maps to a domain spec and a workstream in [`roadmap/workstreams.md`](../roadmap/workstreams.md). + +## Design principles + +- **Declarative everything.** If it can't be expressed as YAML reconciled from git, question it. +- **One control plane.** Containers and VMs share the Kubernetes surface; operators learn one model. +- **Secure by default.** Deny-by-default network, least-privilege identity, encrypted secrets, + trusted TLS everywhere. +- **Durable by default.** No workload is "pet" hardware; state lives on replicated storage with + tested restore. +- **Hands-off operations.** Updates, cert renewal, DNS, and drift correction are automated. +- **Promotable.** dev/test and prod are the same config with scoped overrides. +- **Provable.** Every requirement is verified by an automated, reproducible, behavior-based + conformance test ([14](14-testing-and-conformance.md), [ADR-0010](../decisions/adr-0010-conformance-testing.md)); + "done" is a green, requirement-tagged suite, not a manual demo. +- **Spec-fixed.** The specification is human-owned and immutable to agents (R-TEST-06); agents move + the implementation and its tests toward the spec, never the spec toward the implementation. diff --git a/docs/02-environments-and-provisioning.md b/docs/02-environments-and-provisioning.md new file mode 100644 index 0000000..eb5b575 --- /dev/null +++ b/docs/02-environments-and-provisioning.md @@ -0,0 +1,93 @@ +# 02 — Environments & Provisioning + +Serves: **R-ENV-01/02/03**, **R-PLAT-01**. Related: [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md). + +## Two environments, one configuration + +| Aspect | dev/test | prod | +|---|---|---| +| Substrate | libvirt/KVM VMs on the operator workstation | 3× mini-PC bare metal | +| Node count | 3 VMs (mirror prod topology) | 3 physical nodes | +| Control plane | HA embedded etcd (k3s server on all 3) | HA embedded etcd (k3s server on all 3) | +| Storage | virtual disks presented to Longhorn | local SSD/NVMe presented to Longhorn | +| Status | **active build target** | **design-only until roadmap flips it** | + +**R-ENV-02 is the golden constraint:** the GitOps configuration is authored once. Environments +differ only through scoped overrides (Flux `Kustomization` overlays / cluster-scoped variables): +node counts, disk sizes, WAN endpoints, VPN credentials, DNS zone specifics, resource limits. +Never fork the base config per environment. + +## Infrastructure provisioning (Layer 1 — the cluster nodes) + +> These are the machines that *run* k3s. Distinct from KubeVirt application VMs (Layer 2). + +### Toolchain +- **OpenTofu** (Terraform-compatible) with the **libvirt** provider defines dev/test node VMs as + code: CPU, RAM, disks (a dedicated data disk for Longhorn), NICs, and a **cloud-init** seed. +- **cloud-init** performs first-boot baseline: hostname, users, SSH keys, package baseline, kernel + modules/sysctls required by k3s/Longhorn/KubeVirt, and marks the node ready for k3s. +- **Ansible** performs cluster bootstrap: installs k3s in HA mode, joins nodes, lays down cluster- + wide prerequisites (Cilium is installed as the CNI, not flannel), and hands off to Flux. +- For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation step differs + (bare-metal image/cloud-init via USB/PXE instead of libvirt). + +### Node lifecycle stages (R-ENV-01 — "created, baselined, cluster-joined, managed as code") +1. **Create** — OpenTofu instantiates the VM (dev) / operator images the mini-PC (prod). +2. **Baseline** — cloud-init applies OS baseline and prerequisites. +3. **Cluster-join** — Ansible installs/joins k3s with Cilium as CNI; disables k3s bundled + flannel/traefik/servicelb/kube-proxy as required by the Cilium ADR. +4. **Manage** — from here the cluster is GitOps-managed by Flux; node config changes are made via + IaC and re-applied, never hand-edited on the box. + +### k3s topology decisions +- HA control plane with **embedded etcd** across all 3 nodes (no external datastore). +- Bundled components that conflict with our stack are **disabled at install**: flannel (→ Cilium), + kube-proxy (→ Cilium kube-proxy replacement), Traefik (→ Gateway API via Cilium), servicelb where + a different LB strategy is chosen. Exact flags live in the Ansible bootstrap; the *intent* is + fixed here and in ADR-0001/0002. +- Nodes are both control-plane and workload (3-node converged cluster); with only 3 nodes we do not + isolate control-plane nodes. + +### Bare-metal specifics (prod, design-only for now) +- Each mini-PC contributes local NVMe/SSD to Longhorn; document the disk layout (OS vs. Longhorn + data device) before prod build. +- BIOS/firmware, virtualization extensions (for KubeVirt), and wake/boot behavior are captured as a + provisioning checklist under the prod-readiness workstream. +- Edge networking (multi-WAN, VPN uplink) is largely an **edge-router** concern — see + [`03-networking-isolation-and-egress.md`](03-networking-isolation-and-egress.md) and OQ-1. + +## Environment promotion model + +``` + git repo (single source of truth) + │ + base configuration ──────────────┐ + │ │ + overlay: dev/test overlay: prod + │ │ + Flux on dev cluster Flux on prod cluster (later) +``` + +- A change proves out in **dev/test** against the acceptance gates ([`AGENTS.md` §7](../AGENTS.md)). +- Promotion to prod is a change to the prod overlay only (no base edits), gated by the roadmap's + prod-readiness phase. +- **Never develop against prod.** Prod hardware is a target, not a sandbox (R-ENV-03). + +## Statelessness of the nodes themselves + +Nodes are cattle. A node can be destroyed and rebuilt from IaC + cloud-init + Ansible; cluster and +application state lives on **Longhorn** (replicated) and in **git** (desired state), so node loss is +a re-provision, not a data-loss event (supports R-STATE-01/02). + +## Deliverables for this spec (tracked on the task board) +- OpenTofu module for dev/test node VMs (libvirt) with a dedicated Longhorn data disk. +- cloud-init baseline template (users, SSH, sysctls, kernel modules for Longhorn/KubeVirt). +- Ansible role(s): k3s HA install with Cilium, component disabling, Flux bootstrap handoff. +- Documented prod bare-metal provisioning checklist (design-only; do not execute). +- Proof: destroy a dev node VM and rebuild it; cluster re-converges with no data loss. + +## Open questions +- **OQ-1** (edge router / WAN count) — blocks final multi-WAN design, not node provisioning. +- Do we want **Cluster API** later to unify dev VM and prod bare-metal lifecycle? Deferred; OpenTofu + + Ansible is sufficient and lower-complexity for 3 nodes now. Revisit via ADR if prod scaling + demands it. diff --git a/docs/03-networking-isolation-and-egress.md b/docs/03-networking-isolation-and-egress.md new file mode 100644 index 0000000..efc0859 --- /dev/null +++ b/docs/03-networking-isolation-and-egress.md @@ -0,0 +1,126 @@ +# 03 — Networking, Isolation & Egress + +Serves: **R-NET-01..07**, **R-SEC-02**. Related: [ADR-0002 (Cilium)](../decisions/adr-0002-cni-cilium.md), +[ADR-0008 (VPN egress)](../decisions/adr-0008-vpn-egress-model.md). + +This is the most safety-critical spec. Get default-deny and the VPN kill-switch right; everything +else is convenience. + +## Principles + +1. **Deny by default, allow by exception.** No service may talk to another unless a rule explicitly + permits it, at the port/protocol level (R-NET-01/02). +2. **Egress is a declared property.** Each service declares an **egress class**; the platform routes + it accordingly (R-NET-05/07). +3. **VPN means VPN-only.** A VPN-class service must never leak to the plain internet, including + during tunnel failure (R-NET-06, kill-switch). +4. **Isolation is auditable.** An operator can prove what a service can and cannot reach (R-SEC-03). + +## CNI: Cilium + +Cilium replaces the default k3s CNI and provides: +- **NetworkPolicy / CiliumNetworkPolicy** for L3/L4 (and optional L7) enforcement. +- **kube-proxy replacement** (eBPF) for efficient service routing. +- **Egress Gateway** to force selected pods' egress through a chosen node/identity/SNAT IP — the + building block for WAN/VPN path selection. +- **Hubble** for flow observability — the audit and troubleshooting tool for isolation. +- **Gateway API** implementation for north-south ingress (see [09](09-dns-and-ingress.md)). + +## Segmentation model + +- **Namespaces as trust zones.** Each service (or tightly-coupled service group) gets its own + namespace. Namespace = isolation boundary. +- **Egress classes as zones:** `internet` and `vpn` are first-class labels that drive both network + policy and egress routing. +- **Platform namespace(s)** (Flux, cert-manager, monitoring, storage) are isolated from application + namespaces; only the specific flows they need (e.g. metrics scraping) are opened. + +### Default-deny baseline (R-NET-01) +Every namespace gets a baseline policy set applied by GitOps at namespace creation: +- **Deny all ingress** and **deny all egress** by default. +- Then explicitly allow only: + - DNS to the in-cluster resolver (required for anything to function). + - Declared intra-cluster dependencies (source → destination → port). + - The declared egress class path (internet or VPN). + - Observability scraping from the monitoring namespace (metrics/logs). + +> A workload with no explicit allow rules can do *nothing* except resolve DNS. This is the intended +> starting posture for every onboarded service. + +### Explicit allow rules (R-NET-02) +Service dependencies from the catalog compile into allow rules. Example intent (illustrative, not +final manifest): + +``` +seerr → sonarr tcp/8989 (Seerr requests to Sonarr) +seerr → radarr tcp/7878 (Seerr requests to Radarr) +seerr → plex tcp/32400 (Seerr talks to Plex) +``` + +Rules are declared **once** in the service's catalog entry (its `dependencies`) and rendered into +CiliumNetworkPolicies by the platform. Reviewers see the intent, not hand-written policy YAML. + +## Multi-WAN & egress routing (R-NET-04/05) + +The platform recognizes **egress profiles**. Initial profiles: + +| Profile | Meaning | Mechanism | +|---|---|---| +| `internet` | Standard local network → default WAN(s) | Normal pod egress via node → edge router | +| `vpn` | Routed through a commercial VPN tunnel | Gluetun sidecar and/or Cilium Egress Gateway to a VPN egress node (ADR-0008) | + +**Where multi-WAN policy routing lives (OQ-1):** +- **Physical multi-WAN** (two ISP uplinks, failover/balancing by routing rules) is primarily an + **edge-router/firewall** responsibility (e.g. OPNsense/pfSense) sitting in front of the cluster. + Traffic leaving the cluster on the `internet` profile is policy-routed by the edge router across + available WANs. +- **In-cluster egress selection** (which pods use VPN vs. plain internet) is handled by Cilium + Egress Gateway + the VPN egress model. The cluster decides *class*; the edge router decides + *physical WAN* for the internet class. +- Final split between edge-router and in-cluster responsibility is pinned once OQ-1 (WAN hardware) + is answered. The **service-facing contract does not change**: a service still just declares + `egress: internet|vpn`. + +## VPN egress model (R-NET-05/06 — ADR-0008) + +VPN-class services: **sonarr, radarr, jackett, qbittorrent**. Requirements: all their egress leaves +via the VPN; if the tunnel drops, they lose connectivity rather than leaking. + +Two supported patterns (ADR-0008 selects the default; both preserve the `egress: vpn` contract): + +- **Pattern A — Gluetun sidecar per pod.** A Gluetun container in the pod holds the WireGuard/OpenVPN + tunnel; the app container shares its network namespace and routes all traffic through it. Gluetun's + **firewall kill-switch** blocks non-tunnel egress by default — satisfies R-NET-06 intrinsically. + Simple, self-contained, per-service isolation; more sidecars to run. +- **Pattern B — Central VPN egress gateway + Cilium Egress Gateway.** One (HA) VPN gateway + deployment holds the tunnel; Cilium Egress Gateway steers `vpn`-labeled pods' egress through it. + Fewer tunnels; a shared failure domain; kill-switch enforced at the gateway + policy denying + fallback to the internet path. + +**Kill-switch is mandatory in both patterns and is an acceptance gate:** with the tunnel forcibly +down, a `vpn` service must fail closed (no packets to the plain internet). This must be tested, not +assumed. + +## Ingress (north-south) — summary + +North-south exposure is handled by **Gateway API implemented by Cilium**, with TLS from +cert-manager and DNS from ExternalDNS. Detailed in [`09-dns-and-ingress.md`](09-dns-and-ingress.md). +Exposure is a declared property: `internal` (LAN-only) vs `public` (port-forwarded at the edge). + +## Observability of the network (supports R-SEC-03, R-OBS-04) +- **Hubble** provides flow visibility to verify default-deny and troubleshoot allow rules. +- Alert on: VPN tunnel down, egress-gateway unhealthy, unexpected denied flows spiking, policy + reconciliation errors. Wired into [`10-observability-and-alerting.md`](10-observability-and-alerting.md). + +## Acceptance gates for this spec +- A freshly onboarded service with no rules can reach **nothing** but DNS (default-deny proven via + Hubble + a connectivity probe). +- Declared dependencies work; **undeclared** ones are blocked. +- A `vpn` service's public egress IP is the VPN's, not the home IP (verified). +- Kill-switch: tunnel down ⇒ `vpn` service has no internet path; `internet` services unaffected. +- Changing a service's `egress` class in git flips its path with no workload rearchitecting. + +## Open questions +- **OQ-1** — WAN uplink count and edge-router product; determines edge-vs-cluster routing split. +- **OQ-3** — VPN provider/protocol (WireGuard preferred), number of concurrent tunnels / distinct + egress IPs (affects Pattern A vs B and HA of the gateway). diff --git a/docs/04-storage-and-state.md b/docs/04-storage-and-state.md new file mode 100644 index 0000000..1d5c55a --- /dev/null +++ b/docs/04-storage-and-state.md @@ -0,0 +1,91 @@ +# 04 — Storage & State + +Serves: **R-STATE-01..04**, **R-PLAT-04**. Related: [ADR-0003 (Longhorn)](../decisions/adr-0003-storage-longhorn.md). + +The core promise: **any service can be destroyed and rebuilt on a whim without losing data.** This +requires (a) durable replicated storage, (b) a clean separation of state from workload, and (c) a +tested backup/restore path. + +## Storage tiers + +| Tier | Backing | Use | Replication | +|---|---|---|---| +| **App/config state** | Longhorn block volumes | PVCs for service config, databases, small app data | 3-way replica across nodes | +| **Bulk media** | Dedicated large store (see below) | Plex library, download staging | Not 3× replicated (capacity strategy) | +| **VM disks** | Longhorn (via KubeVirt/CDI) | KubeVirt application-VM root/data disks | 3-way replica | +| **Backups** | Off-cluster S3/NFS target | Velero + Longhorn backups for DR | Off-cluster, separate failure domain | + +## Longhorn (primary persistent storage — R-STATE-02) + +- Provides replicated block storage from each node's local disk — exactly the model for a 3× mini-PC + cluster with local storage and no SAN. +- Default StorageClass: **3 replicas** so any single node loss keeps data available. +- Volumes are the durable home for stateful services. Destroying a pod/Deployment/VM leaves the PVC + and its data intact; rebuilding re-attaches it (R-STATE-01). +- Snapshots for point-in-time; **backups** (to off-cluster target) for DR. +- Guardrails: reserve disk headroom, set replica anti-affinity across nodes, monitor volume health + and degraded replicas (alerts in [10](10-observability-and-alerting.md)). + +## Separation of state from workload (the rebuild guarantee) + +Every service's catalog entry declares its **persistent volumes** explicitly. The rule: + +> If it must survive a rebuild, it lives on a named PVC (Longhorn) or the bulk store — **never** in +> the container's ephemeral filesystem or on a single node's hostPath. + +The rebuild test (an acceptance gate): delete the workload entirely, re-apply from git, and the +service comes back with its data. This is verified per service, not assumed. + +## Bulk media strategy (R-STATE-04 — the Plex problem) + +Plex is the largest service and its library can be multiple terabytes. **3× replicating multi-TB +media across three mini-PCs is impractical** (capacity and cost). Therefore bulk media is a +**distinct tier** from replicated app-config storage. Options to be pinned by ADR-0003 / OQ-2: + +- **NAS / external NFS** exporting the media library, mounted into Plex (and the arr stack) — keeps + bulk data off the replicated tier; simplest capacity story; introduces an external dependency. +- **Dedicated large disk** on one node with **single-replica Longhorn** or a local PV — cheaper, but + node-bound; requires its own backup plan. +- **Object storage** for some content — generally poor fit for Plex direct play. + +Regardless of choice: **Plex's config/metadata database is small and MUST live on 3-replica +Longhorn** (so Plex rebuilds cleanly); only the large media files use the bulk tier. Downloads +(qbittorrent) stage to the bulk tier and hand off to the arr stack. + +## Backup & disaster recovery (R-STATE-03) + +- **Velero** backs up Kubernetes resource state (namespaces, manifests, PVC metadata) and + orchestrates volume backups. +- **Longhorn backups** push volume data to an **off-cluster** S3-compatible or NFS target — a + separate failure domain from the cluster itself. +- **Schedule**: regular automated backups (cadence per data criticality) with retention policy. +- **Restore is a tested procedure**, not a theoretical one: a DR drill (restore a namespace + its + volumes to a clean state) is part of the platform acceptance suite. +- Bulk media has its own backup posture (e.g. it may be re-acquirable rather than backed up 1:1 — + an explicit operator decision recorded per OQ-2), but **all irreplaceable state** (configs, + databases, Nextcloud data, Gitea repos) MUST be backed up off-cluster. + +## KubeVirt VM disks + +Application-VM disks are Longhorn volumes provisioned via **CDI** (Containerized Data Importer). VMs +inherit the same durability and backup story as container PVCs — a KubeVirt VM can be destroyed and +rebuilt from its declarative definition + persistent disk. See [`06-vm-workloads-kubevirt.md`](06-vm-workloads-kubevirt.md). + +## GitOps + state = full rebuildability + +- **Desired state** lives in git (manifests, service definitions). +- **Data** lives on Longhorn/bulk store + off-cluster backups. +- Therefore a total cluster loss is recoverable: re-provision nodes ([02](02-environments-and-provisioning.md)), + re-bootstrap Flux, let it reconcile from git, and restore volumes from backup. This end-to-end + recovery is the ultimate expression of R-STATE-01/03 and SHOULD be drilled at least once in + dev/test before prod is declared ready. + +## Acceptance gates +- Delete and recreate each stateful service; data survives (per-service rebuild test). +- Kill a node (dev/test); replicated volumes stay available; cluster re-converges. +- Restore a namespace + volumes from off-cluster backup into a clean state (DR drill). +- Plex rebuilds with its metadata DB from replicated storage; media re-attaches from the bulk tier. + +## Open questions +- **OQ-2** — final bulk-media home (NAS/NFS vs dedicated disk vs mixed) and its backup posture. +- Off-cluster backup **target**: which S3-compatible endpoint or NFS host, and its own resilience. diff --git a/docs/05-gitops-and-delivery.md b/docs/05-gitops-and-delivery.md new file mode 100644 index 0000000..2f30849 --- /dev/null +++ b/docs/05-gitops-and-delivery.md @@ -0,0 +1,93 @@ +# 05 — GitOps & Delivery + +Serves: **R-GITOPS-01..03**, **R-PLAT-02/04**, **R-ENV-02**. Related: [ADR-0004 (Flux)](../decisions/adr-0004-gitops-flux.md). + +## Model + +Pull-based GitOps with **Flux CD**. The cluster continuously reconciles its state from git. Git is +the single source of truth; the cluster conforms to git, not the reverse (R-GITOPS-01). Drift is +detected and corrected automatically (R-GITOPS-03). + +## Repository layout (the implementation repo, created in a later phase) + +> This spec repo (`home-network-k3s`) holds intent. The **implementation** repo(s) hold the +> reconciled manifests. Proposed structure: + +``` +platform/ ← cluster-wide platform services (Flux-managed) + cilium/ ← CNI, egress gateway, network baseline + longhorn/ ← storage + cert-manager/ ← TLS issuers + ClusterIssuer + externaldns/ ← DNS automation + gateway/ ← Gateway API + listeners/TLS + monitoring/ ← kube-prometheus-stack, Loki, Alloy, Alertmanager + velero/ ← backup/DR + kubevirt/ ← KubeVirt + CDI + image-automation/ ← Flux image update automation config +apps/ ← application workloads, one dir per service + plex/ sonarr/ radarr/ jackett/ seerr/ pihole/ nextcloud/ + minecraft/ unifi/ semaphore/ kasm/ gitea/ qbittorrent/ +catalog/ ← ServiceDefinition entries (the onboarding surface) +clusters/ + dev/ ← Flux entrypoint + dev overlay/vars + prod/ ← Flux entrypoint + prod overlay/vars (design-only) +``` + +- **Kustomize overlays** (or equivalent) express environment differences (R-ENV-02). Base is shared; + `clusters/dev` and `clusters/prod` supply scoped values only. +- **HelmReleases** (Flux) wrap upstream charts (Cilium, Longhorn, cert-manager, kube-prometheus- + stack, etc.); app workloads use charts where good ones exist, otherwise plain manifests. + +## The onboarding surface: ServiceDefinition (R-GITOPS-02) + +Onboarding a service = adding one **ServiceDefinition** under `catalog/`. It is the operator-facing +abstraction that the platform compiles into the concrete resources (Deployment/VM, PVCs, +NetworkPolicies, HTTPRoute, Certificate, DNS record, ServiceMonitor, update policy). See +[`13-service-catalog.md`](13-service-catalog.md) for the schema. Intent lives in one place; the +platform renders the rest. + +## Reconciliation flow + +1. Operator opens a PR against the implementation repo (new/updated ServiceDefinition or platform + change). +2. CI validates (lint, policy checks, schema validation, dry-run/build). +3. Merge to the tracked branch. +4. Flux reconciles: applies manifests, reports status, corrects drift on an interval. +5. Health/rollout status surfaces in the C2 UI ([12](12-command-center-ui.md)) and alerts on failure + ([10](10-observability-and-alerting.md)). + +## Secrets in GitOps + +Secrets are committed **encrypted** with **SOPS + age**; Flux decrypts them in-cluster at +reconciliation. No plaintext secret ever lands in git. See [`07-security-and-secrets.md`](07-security-and-secrets.md) +and [ADR-0007](../decisions/adr-0007-secrets-sops-age.md). + +## Automated updates interplay (R-UPD-*) + +GitOps and hands-off updates coexist: +- **Container image tags** advance automatically via **Flux image-update-automation** driven by + per-service **image policies** (semver range / stable channel) — the controller resolves and + reconciles the new version with **no human-authored change** (R-UPD-02). +- **Helm chart / manifest version bumps** are proposed by **Renovate** with auto-merge for allowed + update classes. + +Full detail and the "is an automated commit a GitOps change?" nuance is in +[`11-automated-updates.md`](11-automated-updates.md) and [ADR-0009](../decisions/adr-0009-automated-image-updates.md). + +## Bootstrap & recovery + +- Flux is bootstrapped onto a fresh cluster by the Ansible handoff ([02](02-environments-and-provisioning.md)). +- Because desired state is in git and data is on Longhorn/backups, a rebuilt cluster reconverges by + re-bootstrapping Flux and restoring volumes ([04](04-storage-and-state.md)). + +## Guardrails +- No `kubectl apply` of app state by hand — everything flows through git/Flux (R-GITOPS-01, click- + ops-never per [AGENTS.md](../AGENTS.md)). +- Reconciliation errors and drift are **alerting conditions** (R-OBS-04). +- Base config is never forked per environment (R-ENV-02). + +## Acceptance gates +- Adding a ServiceDefinition PR results in a fully wired service (compute, storage, policy, TLS, DNS, + monitoring) with no manual steps beyond the merge. +- Manually mutating a live resource is auto-reverted by Flux (drift correction proven). +- The same base config renders correctly for both dev and prod overlays. diff --git a/docs/06-vm-workloads-kubevirt.md b/docs/06-vm-workloads-kubevirt.md new file mode 100644 index 0000000..a070f5d --- /dev/null +++ b/docs/06-vm-workloads-kubevirt.md @@ -0,0 +1,79 @@ +# 06 — VM Workloads (IaaS via KubeVirt) + +Serves: **R-PLAT-03**, **R-PLAT-04**, **R-STATE-01**. Related: [ADR-0005 (KubeVirt)](../decisions/adr-0005-vm-workloads-kubevirt.md). + +## Why this exists + +"Not everything fits in a container." Some workloads need a full OS, a specific kernel, non-container +software, or a lift-and-shift of an existing VM. The requirement (R-PLAT-03) is that these VMs are +managed through the **same Kubernetes API, CLI, and YAML** as containers — one control plane, one +GitOps flow, one network/storage/observability model. **KubeVirt** delivers exactly this: VMs as +Kubernetes custom resources. + +## Do not confuse the two VM layers + +- **Layer 1 — cluster nodes** are VMs only in dev/test (libvirt) and bare metal in prod. Managed by + OpenTofu/Ansible ([02](02-environments-and-provisioning.md)). **Not** KubeVirt. +- **Layer 2 — application VMs** run *inside* the cluster via KubeVirt and are the subject of this + spec. Managed by Kubernetes/GitOps like any other workload. + +## What KubeVirt provides + +- `VirtualMachine` / `VirtualMachineInstance` custom resources — declare a VM in YAML, reconcile via + Flux, manage via `kubectl` (and `virtctl` for console/start/stop). +- **CDI (Containerized Data Importer)** to import/clone disk images into **Longhorn** PVCs — VM disks + get the same replication and backup story as container volumes ([04](04-storage-and-state.md)). +- VMs sit on the **same Cilium network** — so **default-deny isolation and egress classes apply to + VMs too** (R-NET-*). A KubeVirt VM can be `egress: vpn` just like a container. +- VMs are exposed through the same **Gateway API / Service** model and monitored through the same + stack. + +## The unified contract + +A VM onboarded via a ServiceDefinition (`kind: vm`) gets, from the *same* declarative surface: +- Compute (vCPU/RAM), an OS image (via CDI), and Longhorn-backed disks. +- Default-deny networking + explicit allow rules + egress class (R-NET-*). +- TLS + DNS + ingress if it exposes an HTTP(S) service (R-TLS/R-DNS). +- Metrics/logs and alerting (R-OBS-*) — node-exporter-class metrics from inside the guest plus + KubeVirt/virt metrics from the platform. +- Backup/DR of its disks (R-STATE-03) and rebuildability (R-STATE-01): destroy the VM, keep the + disk PVC, recreate from the declaration. + +## Update policy for VMs (R-UPD-*) + +VM "updates" differ from container image tag bumps: +- **Guest OS package updates** happen *inside* the guest on a schedule (unattended-upgrades / dnf- + automatic style), configured via the VM's cloud-init/declaration — no human action. +- **VM image/template updates** (new base image) are handled as declarative changes to the VM + definition; where a golden-image pipeline exists, new images can be promoted automatically per + policy. +- The requirement "latest stable without human/AI/GitOps change" is met for guest packages via in- + guest automation; base-image rebuilds follow the same automated-promotion philosophy as containers + where feasible. Record any per-VM exceptions explicitly. + +## When to choose VM vs container + +Prefer containers (PaaS) by default. Choose a KubeVirt VM (IaaS) when: +- The software isn't reasonably containerizable (kernel modules, full desktop, licensing tied to a + VM, appliance images). +- Lifting an existing VM is faster/safer than re-platforming. +- Strong OS-level isolation is required beyond namespace/network policy. + +For the initial catalog, all 13 named services are container-first; KubeVirt is the escape hatch and +must be **proven** in dev/test with at least one representative VM workload so the IaaS path is real, +not theoretical. + +## Prerequisites & constraints +- Nodes must expose virtualization extensions (nested virt in dev/test KVMs; native on prod mini- + PCs) — captured in the node baseline ([02](02-environments-and-provisioning.md)). +- Live migration across 3 converged nodes is a nice-to-have, not a requirement; document whether it's + enabled. +- Resource pressure: VMs are heavier than pods — capacity planning must account for them, + particularly on the 3-node prod cluster. + +## Acceptance gates +- Create, start, stop, and destroy a VM entirely through Kubernetes YAML + `kubectl`/`virtctl`. +- The VM's disk is Longhorn-backed; destroy/rebuild preserves data (R-STATE-01). +- Default-deny applies: the VM can reach only what its allow rules permit. +- An `egress: vpn` VM egresses via the VPN with a working kill-switch (R-NET-06). +- Guest OS auto-updates on schedule with no manual step (R-UPD-*). diff --git a/docs/07-security-and-secrets.md b/docs/07-security-and-secrets.md new file mode 100644 index 0000000..22800d4 --- /dev/null +++ b/docs/07-security-and-secrets.md @@ -0,0 +1,83 @@ +# 07 — Security & Secrets + +Serves: **R-SEC-01/02/03**, supports all `R-NET-*` and `R-TLS-*`. Related: +[ADR-0007 (SOPS+age)](../decisions/adr-0007-secrets-sops-age.md). + +## Security posture + +Defense in depth with declarative, auditable controls: + +- **Network**: default-deny east-west, explicit port-level allow, egress-class routing, VPN kill- + switch ([03](03-networking-isolation-and-egress.md)). This is the primary isolation control. +- **Identity/least privilege**: each workload runs with a minimal ServiceAccount and only the RBAC + it needs; no workload gets cluster-admin. VMs and containers alike get least network reach + (R-SEC-02). +- **Secrets**: encrypted at rest in git; decrypted only in-cluster (R-SEC-01). +- **TLS**: trusted, auto-managed certificates end to end ([08](08-tls-and-certificates.md), R-TLS-*). +- **Supply chain**: pin images by digest where automation allows; prefer known-good upstreams; + updates flow through policy, not ad-hoc pulls ([11](11-automated-updates.md)). +- **Policy enforcement (admission)**: **Kyverno** runs in-cluster as an admission controller and + rejects non-conforming resources at reconcile time — e.g. a namespace missing its default-deny + baseline, a plaintext Secret, a workload without a required egress-class label. The *same* policies + gate CI ([14](14-testing-and-conformance.md), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md)), + so invariants are enforced both before merge and at admission (defense in depth). Enforcement + complements — never replaces — the runtime behavior tests. +- **Auditability**: isolation, egress path, cert validity, and backup integrity are all + demonstrable (R-SEC-03) via Hubble, cert status, and DR drills. + +## Secrets management (R-SEC-01) + +**SOPS + age**, native to Flux: +- Secrets are committed to git **encrypted** with `age` recipients; Flux decrypts them at + reconciliation using an in-cluster age key. Plaintext never enters git. +- The age private key is bootstrapped into the cluster out-of-band (documented bootstrap step, one + of the few sanctioned non-GitOps actions) and stored as a Kubernetes Secret Flux can read. +- **Never** print decrypted secret material into logs, PR text, or agent output (see + [AGENTS.md §5](../AGENTS.md)). + +### Secret categories to manage +- VPN credentials (WireGuard keys / OpenVPN creds) for the egress model. +- DNS provider API token (Cloudflare) for ACME DNS-01 + ExternalDNS. +- SMTP credentials for Alertmanager email. +- Service credentials (Nextcloud, Gitea admin, UniFi, Semaphore, Kasm, DB passwords). +- Off-cluster backup target credentials (S3/NFS) for Velero/Longhorn. +- Grafana admin, and any API tokens the C2 UI needs. + +### Rotation +- Rotating a secret = re-encrypting with SOPS and committing; Flux rolls it out. Where a secret + change requires a workload restart, that is triggered by the reconcile (e.g. checksum annotation). +- VPN keys, DNS tokens, and SMTP creds SHOULD have a documented rotation procedure. + +## Alternatives considered (see ADR-0007) +- **Sealed Secrets** — viable; SOPS chosen for Flux-native decryption and multi-recipient/age + simplicity and readable diffs. +- **External Secrets Operator + external vault** — heavier; deferred until a real external secret + store exists. May be revisited via ADR. + +## RBAC & tenancy +- Single operator + platform automation; no multi-tenant RBAC now (out of scope, [00](00-vision-and-scope.md)). +- Platform controllers (Flux, cert-manager, ExternalDNS, Velero, KubeVirt) get scoped permissions; + application workloads get near-zero cluster permissions. +- The C2 UI's operational actions run through a constrained, audited path — not a shared admin + kubeconfig handed to a browser. + +## Node & host security +- Nodes baselined via cloud-init/Ansible ([02](02-environments-and-provisioning.md)); SSH key-only, + minimal packages, required sysctls only. +- Host-level changes are IaC, not hand edits (drift + auditability). + +## Auditability requirements (R-SEC-03) — what an operator must be able to prove +- **Isolation**: "service X cannot reach service Y" — via Hubble flow logs + a denied-probe test. +- **Egress path**: "service X's public IP is the VPN's" — via an egress IP check. +- **Kill-switch**: "with VPN down, X leaks nothing" — via a fault-injection test. +- **TLS**: "every endpoint serves a valid, non-self-signed, unexpired cert" — via cert status + + external check. +- **Backups**: "irreplaceable data is restorable" — via a DR drill. + +These proofs are part of the platform acceptance suite, not one-time checks. + +## Acceptance gates +- No plaintext secret exists anywhere in git history (scanned). +- Flux decrypts SOPS secrets and workloads consume them without manual steps. +- Least-privilege RBAC verified (no workload has broad cluster rights). +- The five auditability proofs above can be produced on demand. diff --git a/docs/08-tls-and-certificates.md b/docs/08-tls-and-certificates.md new file mode 100644 index 0000000..cbc38cb --- /dev/null +++ b/docs/08-tls-and-certificates.md @@ -0,0 +1,61 @@ +# 08 — TLS & Certificates + +Serves: **R-TLS-01/02/03**, **R-DNS-01**. Related: [ADR-0006 (cert-manager + Let's Encrypt)](../decisions/adr-0006-tls-cert-manager-letsencrypt.md). + +## Requirement recap + +Every externally reachable service presents TLS from a **credible, trusted, non-self-signed** +authority, with certificates **auto issued, renewed, and revoked** — including for **internal-only** +services (R-TLS-01/02/03). + +## Solution: cert-manager + Let's Encrypt via DNS-01 + +- **cert-manager** runs in the cluster and manages the full certificate lifecycle. +- **Let's Encrypt** is the trusted, non-self-signed CA (publicly trusted; free; automatable) — + satisfies R-TLS-01. +- **ACME DNS-01 challenge** against the `keithjackson.dev` DNS zone (Cloudflare or equivalent, per + OQ-4). DNS-01 is the key choice because it validates domain control **without exposing the service + publicly** — so internal-only services get real, trusted certs (R-TLS-03), and **wildcard** + certificates (`*.home.keithjackson.dev`) are possible. + +## Issuance model + +- A `ClusterIssuer` for Let's Encrypt production (and one for staging, used during bring-up to avoid + rate limits). +- Certificates are requested declaratively — either a **wildcard** `*.home.keithjackson.dev` cert + shared by the Gateway, or **per-service** certs, decided per ADR-0006. Recommendation: a wildcard + cert for `*.home.keithjackson.dev` terminated at the Gateway simplifies onboarding (a new service + needs no new cert), with per-service certs available where isolation warrants. +- The DNS provider API token is a SOPS-encrypted secret ([07](07-security-and-secrets.md)). + +## Renewal & revocation (R-TLS-02) + +- cert-manager renews well before expiry automatically; no human action. +- Revocation: removing/rotating a certificate is a declarative change; cert-manager handles + issuance of replacements. Expiry monitoring is an alert condition ([10](10-observability-and-alerting.md), R-OBS-04). + +## Where TLS terminates + +- **North-south**: at the **Gateway API** listener (implemented by Cilium, [09](09-dns-and-ingress.md)), + using the cert-manager-issued cert. Public services are reached via the edge router forwarding to + the Gateway; internal services resolve to the Gateway on the LAN. +- Internal service-to-service traffic relies primarily on network policy for security; mTLS between + workloads is **optional** future hardening (could be added via Cilium/service mesh) and is not a + launch requirement — record as a possible future ADR. + +## Trust chain for internal clients + +Because certs come from Let's Encrypt (publicly trusted), internal clients trust them with no custom +CA distribution. Split-horizon DNS ([09](09-dns-and-ingress.md)) resolves +`<service>.home.keithjackson.dev` to the internal Gateway while the same trusted cert is served. + +## Acceptance gates +- `https://<service>.home.keithjackson.dev` serves a valid, non-self-signed, unexpired certificate + (verified by an external TLS checker), including for internal-only services. +- Certificates renew automatically ahead of expiry with no manual step (observed across a renewal). +- Certificate expiry and issuance failures raise alerts. +- Onboarding a new service requires no manual certificate step. + +## Open questions +- **OQ-4** — confirm DNS provider (Cloudflare assumed) and an API token scoped for DNS-01. +- Wildcard vs. per-service certs as the default (pinned in ADR-0006). diff --git a/docs/09-dns-and-ingress.md b/docs/09-dns-and-ingress.md new file mode 100644 index 0000000..e6a1b36 --- /dev/null +++ b/docs/09-dns-and-ingress.md @@ -0,0 +1,71 @@ +# 09 — DNS & Ingress + +Serves: **R-DNS-01/02/03**, **R-C2-01**, works with **R-TLS-***. Related: +[ADR-0002 (Cilium/Gateway API)](../decisions/adr-0002-cni-cilium.md). + +## Naming model + +- All services live at `<service>.home.keithjackson.dev` (e.g. `plex.home.keithjackson.dev`). +- The **root** `home.keithjackson.dev` serves the **Command & Control UI** ([12](12-command-center-ui.md)). +- Public apex `*.keithjackson.dev` resolves to the network's public IP; `*.home.keithjackson.dev` + is the platform's zone. + +## DNS automation (R-DNS-02) + +- **ExternalDNS** watches Gateway/HTTPRoute (and Service) resources and manages records in the + authoritative DNS provider (Cloudflare, per OQ-4) automatically — a new service's DNS appears from + its declared intent, no manual record editing. +- Records for `*.home.keithjackson.dev` are created/updated/removed as services come and go. + +## Split-horizon resolution (R-DNS-03) + +Internal and external answers differ: +- **External** (`*.home.keithjackson.dev` from the internet): resolves to the public IP; reaches + the edge router, which forwards `public`-class services to the in-cluster Gateway. +- **Internal** (LAN clients): resolves `<service>.home.keithjackson.dev` to the **internal Gateway + address** directly, avoiding hairpinning through the public IP. +- **PiHole** (itself an onboarded service) provides internal LAN DNS and the split-horizon answers + for `.home.keithjackson.dev`; the trusted Let's Encrypt cert is served either way ([08](08-tls-and-certificates.md)). + +> DNS is thus both a platform capability *and* a hosted service. Bootstrapping order matters: the +> platform needs working DNS before PiHole is fully managed — handled by the bring-up sequence in +> [`roadmap/phases.md`](../roadmap/phases.md). + +## Ingress (north-south) via Gateway API + +- **Gateway API implemented by Cilium** is the ingress layer (no separate ingress controller; + reuses the CNI, [ADR-0002](../decisions/adr-0002-cni-cilium.md)). +- A shared **Gateway** defines HTTPS listeners on `*.home.keithjackson.dev` terminating TLS with the + cert-manager cert ([08](08-tls-and-certificates.md)). +- Each service contributes an **HTTPRoute** mapping its hostname to its Service — rendered from the + ServiceDefinition, not hand-authored. +- **Exposure class** is declared per service: + - `internal` — reachable only on the LAN (no edge port-forward; internal DNS only). + - `public` — additionally reachable from the internet via the edge router forwarding 80/443 to the + Gateway. + +## Edge integration + +- The edge router/firewall forwards inbound 80/443 for `public` services to the Gateway's address + and applies multi-WAN inbound rules as needed (OQ-1). +- `internal` services are never port-forwarded; they rely on internal DNS + Gateway on the LAN. +- A stable ingress VIP for the Gateway is provided by the cluster's LB strategy (Cilium + L2 announcement / BGP / a chosen LB); pin the mechanism in the networking bring-up. + +## Interaction with default-deny + +Ingress opens **north-south** paths to the specific service; it does **not** relax **east-west** +default-deny ([03](03-networking-isolation-and-egress.md)). A service being publicly reachable does +not let other in-cluster services reach it unless an explicit allow rule exists. + +## Acceptance gates +- A new service's `<service>.home.keithjackson.dev` record appears automatically and routes to it. +- `internal` services are unreachable from the public internet; `public` services are reachable and + TLS-valid. +- Internal clients resolve to the internal Gateway (no public hairpin) and get a trusted cert. +- The root `home.keithjackson.dev` serves the C2 UI. + +## Open questions +- **OQ-1/OQ-4** — edge router product + DNS provider API (affects ExternalDNS provider config and + inbound WAN rules). +- Gateway LB/VIP mechanism (Cilium L2 vs BGP) — pin during networking bring-up. diff --git a/docs/10-observability-and-alerting.md b/docs/10-observability-and-alerting.md new file mode 100644 index 0000000..6bbc3a4 --- /dev/null +++ b/docs/10-observability-and-alerting.md @@ -0,0 +1,69 @@ +# 10 — Observability & Alerting + +Serves: **R-OBS-01/02/03/04**, supports **R-C2-02**, **R-SEC-03**. + +## Stack + +- **kube-prometheus-stack** — Prometheus (metrics), Alertmanager (routing/notify), Grafana + (dashboards), and the standard Kubernetes exporters/rules. +- **Loki** — log aggregation. +- **Grafana Alloy** (or Promtail-class agent) — ships node/pod/VM logs to Loki. +- **Hubble** — network flow visibility from Cilium ([03](03-networking-isolation-and-egress.md)). +- **Alertmanager → SMTP** — issues reported by **email** (R-OBS-02). +- **Grafana** — dashboards for services and platform health (R-OBS-03). + +## What is monitored (R-OBS-01/04) + +Application services: +- Up/health, resource use, request/error signals where exposed, and per-service SLIs declared in the + ServiceDefinition (a service can declare the alerts it cares about). + +Platform health (R-OBS-04) — first-class alerts: +- **Nodes**: down/unreachable, resource pressure, disk fill. +- **Storage (Longhorn)**: degraded/faulted volumes, replica rebuilds, capacity thresholds. +- **GitOps (Flux)**: reconciliation failures, drift, suspended resources. +- **Certificates (cert-manager)**: upcoming expiry, issuance failures. +- **Networking**: VPN tunnel down, egress-gateway unhealthy, spikes in denied flows (Hubble), + NetworkPolicy reconcile errors. +- **Backups (Velero/Longhorn)**: backup job failure, missed schedule, restore-test failures. +- **KubeVirt**: VM crash-loops / failed launches. + +## Alert delivery (R-OBS-02) + +- Alertmanager routes by severity to **email** (SMTP creds are SOPS-encrypted, [07](07-security-and-secrets.md)). +- Severity model: `critical` (page-worthy: data risk, outage, VPN leak potential, cert about to + expire) vs `warning` (degraded, capacity, needs attention). Both go to email; only routing/subject + differs. +- Alerts include enough context (runbook link, affected object) to act without spelunking. + +## Dashboards (R-OBS-03) + +Grafana dashboards, surfaced in the C2 UI ([12](12-command-center-ui.md)): +- **Platform overview**: nodes, cluster capacity, Flux status, cert expiry, backup status. +- **Storage**: Longhorn volume health, capacity, replica state. +- **Network**: Hubble flows, egress/VPN health, denied-flow trends. +- **Per-service**: one row/dashboard per catalog service with its declared SLIs. + +## Observability as onboarding output + +The ServiceDefinition ([13](13-service-catalog.md)) declares observability expectations; the +platform renders **ServiceMonitors/PodMonitors**, log scraping, dashboards, and alert rules from it. +A newly onboarded service is monitored **by default**, not as a follow-up task (part of the golden +path, [01](01-architecture-overview.md)). + +## Retention & sizing +- Metrics/log retention set per environment overlay (dev shorter, prod longer). Loki retention + requires a compactor with retention enabled — do not rely on a period setting alone. +- Monitoring storage lives on Longhorn; ensure its own capacity is alerted on (don't let the monitor + blind itself). + +## Auditability tie-in (R-SEC-03) +Hubble + Prometheus + cert status provide the evidence for the isolation/egress/TLS/backup proofs +required by [07](07-security-and-secrets.md). + +## Acceptance gates +- Every catalog service emits metrics + logs and has at least one meaningful alert that fires + correctly in a fault test. +- Platform-health alerts (node down, Longhorn degraded, Flux failure, cert expiry, VPN down, backup + failure) each fire and email correctly in an injected-fault test. +- Grafana dashboards render for platform and each service; the C2 UI links them. diff --git a/docs/11-automated-updates.md b/docs/11-automated-updates.md new file mode 100644 index 0000000..e576bad --- /dev/null +++ b/docs/11-automated-updates.md @@ -0,0 +1,76 @@ +# 11 — Automated Updates + +Serves: **R-UPD-01/02/03**. Related: [ADR-0009](../decisions/adr-0009-automated-image-updates.md). + +## The requirement, precisely + +> Services must update to latest **stable** versions **on a schedule**, with **no human, no AI, and +> no hand-authored GitOps change**. (R-UPD-01/02) + +This has a subtle tension with GitOps (where "the desired state is what's in git"). We resolve it +explicitly below. + +## Resolving "no GitOps change" vs. GitOps + +There are two interpretations; we honor the operator's intent (hands-off), and ADR-0009 records the +mechanism: + +- **A human/AI authoring a version bump = prohibited.** No person and no chatbot edits a tag. +- **An automation controller advancing a version under a policy declared once = permitted**, because + the *operator's* declared intent (the update policy / semver range) is the stable, human-authored + thing; the controller merely resolves it. Two mechanisms, chosen by artifact type: + +### 1. Container image tags → Flux image-update-automation +- Per service, an **ImagePolicy** declares the acceptable range/channel (e.g. semver `>=1.0.0` + minor/patch, or a `stable` tag pattern). Flux's image-reflector scans the registry; image- + automation advances the running version automatically **on its scan interval** (the schedule). +- Where automation writes the resolved version back to git, that commit is machine-authored under + the declared policy — no human/AI touches it (satisfies R-UPD-02 in spirit and practice). +- Alternatively, digest-tracking of a mutable stable tag avoids per-update commits entirely. ADR-0009 + picks the default and the per-service escape hatch. + +### 2. Helm charts / manifests → Renovate +- **Renovate** watches chart and dependency versions and opens PRs; allowed update classes + **auto-merge** on green CI. This keeps platform components current with no human authoring. +- Chart *major* bumps (potentially breaking) can require review — configurable per policy. + +### 3. VM guests → in-guest automation +- KubeVirt guests auto-update OS packages inside the guest on a schedule (unattended-upgrades / dnf- + automatic) declared in the VM's cloud-init ([06](06-vm-workloads-kubevirt.md)). Base-image + promotion follows the same automated philosophy where a golden-image pipeline exists. + +## Per-service policy (R-UPD-03) + +Each ServiceDefinition ([13](13-service-catalog.md)) declares an **update policy**: +- **channel/range**: which versions are "stable" and acceptable (semver range, tag regex). +- **cadence**: how often to check/apply (the schedule). +- **hold/pin**: temporarily freeze a service (e.g. a known-bad upstream release). +- **rollout & health**: how the new version rolls out and what health gates it. +- **auto-rollback**: revert on failed health so a bad upstream release self-heals. + +## Safety rails (so hands-off ≠ reckless) + +- **Stable only**: never track pre-release/edge tags unless explicitly opted in per service. +- **Health-gated rollout**: a failing update is caught by health checks and rolled back (R-UPD-03), + and raises an alert ([10](10-observability-and-alerting.md)). +- **Backups precede risk**: state is on Longhorn with backups, so a bad update never risks data + ([04](04-storage-and-state.md)). +- **Observability**: update events (image advanced, chart merged, rollback) are visible and, on + failure, alert by email. +- **Pin escape hatch**: an operator can pin a service via git if an upstream release misbehaves — + the one sanctioned human touch, and it *stops* automation rather than performing an update. + +## Interaction with default-deny & egress +Updates change image versions, not network posture. A VPN-class service stays VPN-class across +updates; policies are reconciled from the (unchanged) ServiceDefinition. + +## Acceptance gates +- A service advances to a newer stable version automatically, on schedule, with no human/AI edit. +- A deliberately bad update is caught by health checks, auto-rolled-back, and alerts by email. +- A pinned service does **not** advance. +- Platform components (charts) stay current via auto-merged Renovate PRs. + +## Open questions +- Exact default in ADR-0009: write-back commits vs. digest-tracking mutable tags (trade readable + history vs. zero-commit updates). +- Registry rate-limit / credential handling for image scanning at cadence. diff --git a/docs/12-command-center-ui.md b/docs/12-command-center-ui.md new file mode 100644 index 0000000..99e04f3 --- /dev/null +++ b/docs/12-command-center-ui.md @@ -0,0 +1,64 @@ +# 12 — Command & Control (C2) Center UI + +Serves: **R-C2-01/02**, surfaces **R-OBS-***, **R-GITOPS-***. + +## Purpose + +The root `home.keithjackson.dev` is the operator's front door: a single UI that shows what's running +in the cluster, links to every service, and surfaces platform health and safe operational actions +(R-C2-01/02). + +## Composition (assemble, don't reinvent) + +The C2 center is a curated composition of best-in-class UIs behind the root domain, not a bespoke +app to build from scratch: + +- **Homepage (gethomepage.dev)** — the landing dashboard at the root: a live catalog of services + with links, status pings, and per-service widgets (many of the catalog services have native + Homepage widgets: Plex, Sonarr, Radarr, qBittorrent, PiHole, Gitea, etc.). This is the primary + R-C2-01 surface. +- **Grafana** — embedded platform + per-service dashboards (R-OBS-03) for health at a glance. +- **Headlamp** (or a comparable Kubernetes web UI) — a controlled Kubernetes console for viewing + workloads/VMs and performing scoped operational actions (restart, scale, inspect) — the R-C2-02 + "controlled operational actions where safe" surface. +- **Flux status** — GitOps reconciliation state (via Headlamp's Flux view, the Flux UI, or a + Homepage widget) so the operator sees drift/errors. +- **Hubble UI** (optional) — network flow visualization for isolation troubleshooting. + +Each sub-UI is itself an onboarded, TLS-protected, access-controlled service; the root aggregates and +links them. ADR-worthy only if a custom portal is later preferred over Homepage. + +## Data the C2 surfaces (R-C2-02) + +- **Service catalog**: every running service, its URL, its health, its egress class, its exposure. +- **Platform health**: node status, Longhorn capacity/health, Flux reconciliation, cert expiry, + backup status, VPN tunnel state — sourced from the observability stack ([10](10-observability-and-alerting.md)). +- **Alerts**: current firing alerts (mirrors what email delivers). + +## Actions (safe, audited) + +Operational actions exposed through the C2 must be **scoped and audited**, never a raw cluster-admin +kubeconfig in a browser: +- Restart/scale a workload, start/stop a KubeVirt VM, trigger a Flux reconcile, acknowledge an + alert. +- Destructive actions (delete, offboard) require explicit confirmation and are logged. +- Onboarding itself remains a **git PR** (GitOps), not a C2 button — the C2 may *link* to the + catalog/PR flow but does not bypass GitOps ([05](05-gitops-and-delivery.md)). + +## Access control + +- The C2 root and its sub-UIs are `internal` exposure by default (LAN-only) unless the operator + explicitly makes a component `public`. +- Authentication in front of the C2 (and sensitive sub-UIs like Headlamp) — SSO/reverse-proxy auth — + is required; the mechanism (e.g. an identity-aware proxy) is pinned during the C2 workstream. +- Trusted TLS on the root and all sub-UIs ([08](08-tls-and-certificates.md)). + +## Acceptance gates +- `https://home.keithjackson.dev` serves the C2 landing UI over trusted TLS. +- Every running service appears with a working link and live status. +- Platform health (nodes, storage, Flux, certs, backups, VPN) is visible without leaving the C2. +- Safe operational actions work and are access-controlled + audited; destructive actions confirm. + +## Open questions +- Auth mechanism for the C2 (identity-aware proxy vs per-app auth) — pin in the C2 workstream. +- Custom portal vs. Homepage-as-root long term (ADR if we diverge from Homepage). diff --git a/docs/13-service-catalog.md b/docs/13-service-catalog.md new file mode 100644 index 0000000..2509540 --- /dev/null +++ b/docs/13-service-catalog.md @@ -0,0 +1,112 @@ +# 13 — Service Catalog & the ServiceDefinition + +Serves: **R-GITOPS-02**, ties together **R-NET-***, **R-STATE-***, **R-TLS-***, **R-DNS-***, +**R-OBS-***, **R-UPD-***. The catalog is the operator-facing onboarding surface. + +## The ServiceDefinition abstraction + +Onboarding a service is adding/editing one **ServiceDefinition** under `catalog/` in the +implementation repo. It captures *intent*; the platform compiles it into concrete resources +(Deployment or KubeVirt VM, PVCs, CiliumNetworkPolicies, egress routing, HTTPRoute, Certificate, DNS +record, ServiceMonitor + dashboards + alerts, and update policy). Operators declare **what**; +platform provides the **how**. This keeps every requirement wired from a single reviewable object. + +### Fields (canonical schema — implementers render manifests from these) + +```yaml +name: <string> # DNS label → <name>.home.keithjackson.dev +kind: container | vm # PaaS (default) or IaaS via KubeVirt (R-PLAT-02/03) +source: + image: <repo/image> # container: image ref + # or vmTemplate: <image/source> # vm: CDI source image +exposure: internal | public # LAN-only vs edge port-forwarded (R-DNS/R-NET) +egress: internet | vpn # egress class (R-NET-05); vpn ⇒ kill-switch (R-NET-06) +storage: + - name: config + size: <e.g. 5Gi> + class: longhorn # replicated app/config state (R-STATE-02) + - name: media # optional bulk tier (R-STATE-04) + class: bulk +dependencies: # explicit allow rules; everything else denied (R-NET-01/02) + - to: <service> + port: <n> + protocol: tcp|udp +ingressRoutes: # hostnames/paths to expose (rendered to HTTPRoute + TLS + DNS) + - host: <name>.home.keithjackson.dev + port: <servicePort> +update: + channel: stable # never pre-release unless opted in (R-UPD-01) + range: <semver range | tag regex># acceptable versions + cadence: <schedule> # how often to check/apply + autoRollbackOnFailedHealth: true # R-UPD-03 + pin: <optional version> # operator freeze escape hatch +observability: + metrics: true + slis: [ up, ... ] # per-service alert intents (R-OBS-01) +backup: + policy: <ref to backup schedule> # off-cluster (R-STATE-03); or 'reacquirable' for bulk media +``` + +> The schema is the **contract** between the operator surface and the implementation. Changing it is +> an interface change — coordinate per [`agents/coordination-protocol.md`](../agents/coordination-protocol.md). + +## Default posture for every service (unless overridden) +- `kind: container`, `exposure: internal`, `egress: internet`. +- **Default-deny**: no `dependencies` ⇒ can reach only DNS ([03](03-networking-isolation-and-egress.md)). +- Longhorn-backed config PVC; off-cluster backup; trusted TLS; auto-DNS; monitored; auto-updated on + `stable`. + +## The initial catalog (13 services) + +Egress and dependencies below are **normative intent**; implementers render the exact policies. +"Bulk" = large media tier ([04](04-storage-and-state.md)). All get trusted TLS + auto-DNS + monitoring ++ auto-update unless noted. + +| # | Service | kind | exposure | egress | Storage | Key dependencies (explicit allows) | Notes | +|---|---|---|---|---|---|---|---| +| 1 | **plex** | container | public | **internet** | config: longhorn (small, replicated); media: **bulk** | — (clients reach it; Seerr → Plex) | Largest service; media on bulk tier, metadata DB on Longhorn ([04](04-storage-and-state.md)) | +| 2 | **sonarr** | container | internal | **vpn** | config: longhorn; media/downloads: bulk | → qbittorrent, → jackett; ← seerr | VPN + kill-switch (R-NET-06) | +| 3 | **radarr** | container | internal | **vpn** | config: longhorn; media/downloads: bulk | → qbittorrent, → jackett; ← seerr | VPN + kill-switch | +| 4 | **jackett** | container | internal | **vpn** | config: longhorn | ← sonarr, ← radarr | VPN + kill-switch (indexer) | +| 5 | **seerr** | container | public | internet | config: longhorn | → sonarr:8989, → radarr:7878, → plex:32400 | Request UI; talks to arr + Plex | +| 6 | **pihole** | container | internal | internet | config: longhorn | (provides DNS to cluster/LAN) | Also a platform DNS dependency ([09](09-dns-and-ingress.md)) | +| 7 | **nextcloud** | container | public | internet | data: longhorn (irreplaceable → backed up) | → its DB | File sync; DR-critical data | +| 8 | **minecraft** | container | public | internet | world: longhorn (backed up) | — | Exposes game port via edge | +| 9 | **unifi** | container | internal | internet | config: longhorn | — | UniFi network controller | +| 10 | **semaphore** | container | internal | internet | config/db: longhorn | — | Ansible UI | +| 11 | **kasm** | container | internal | internet | config: longhorn | — | Kasm Workspaces (may need more resources; container-first, VM if required) | +| 12 | **gitea** | container | public | internet | repos/db: longhorn (irreplaceable → backed up) | — | Git hosting; DR-critical | +| 13 | **qbittorrent** | container | internal | **vpn** | config: longhorn; downloads: **bulk** | ← sonarr, ← radarr | VPN + kill-switch; download client | + +**VPN-class set (R-NET-05/06):** sonarr, radarr, jackett, qbittorrent — all egress via VPN with an +enforced kill-switch. Their inter-dependencies (arr → qbittorrent/jackett) are **east-west allows** +within the deny-by-default fabric; being VPN-class governs *egress*, not intra-cluster reachability. + +**Dependency graph (explicit allows only; all else denied):** +``` +seerr → sonarr, radarr, plex +sonarr → qbittorrent, jackett +radarr → qbittorrent, jackett +(everything → DNS only, by default) +``` + +**"…but not limited to":** the catalog is open-ended. New services follow the same ServiceDefinition +and inherit all guarantees automatically — that is the point of the abstraction. + +## Cross-cutting notes +- **Plex media** is the R-STATE-04 case: keep the metadata DB on replicated Longhorn; keep the large + library on the bulk tier; downloads stage on bulk and are handed off by the arr stack. +- **DR-critical data** (Nextcloud, Gitea, Minecraft worlds, PiHole config, service DBs) MUST have + off-cluster backups ([04](04-storage-and-state.md)); bulk media may be `reacquirable` per operator + choice (OQ-2). +- **Public exposure** (Plex, Seerr, Nextcloud, Minecraft, Gitea) requires edge port-forwarding + ([09](09-dns-and-ingress.md)); everything else stays `internal`. +- **Kasm** is resource-heavy and may be a candidate for `kind: vm` if container mode proves + insufficient — the ServiceDefinition makes that a one-field change. + +## Acceptance gates +- Each service is fully wired from its ServiceDefinition alone (compute, storage, policy, egress, + TLS, DNS, monitoring, updates) — no manual steps beyond merge. +- Default-deny holds; only listed dependencies work. +- VPN-class services egress via VPN with a proven kill-switch. +- Stateful services survive destroy/rebuild; DR-critical data restores from off-cluster backup. diff --git a/docs/14-testing-and-conformance.md b/docs/14-testing-and-conformance.md new file mode 100644 index 0000000..c1a4c57 --- /dev/null +++ b/docs/14-testing-and-conformance.md @@ -0,0 +1,168 @@ +# 14 — Testing & Conformance + +Serves: **R-TEST-01..06**, and is the verification mechanism for **every other requirement**. +Related: [ADR-0010](../decisions/adr-0010-conformance-testing.md). + +> **This spec changes what "done" means.** Nothing is accepted on a manual demo, a screenshot, or +> "it reconciled." Every requirement is proven by an **automated, reproducible, behavior-based test** +> that a machine can run on demand and that is tagged with the requirement ID(s) it verifies. + +## Principles + +1. **Behavior-based (R-TEST-01).** Tests assert *observable system behavior* — Given a state, When + an action, Then an outcome — not implementation internals. "A pod with no allow rules cannot reach + Plex" is a behavior; "a CiliumNetworkPolicy object exists" is not (it may exist and still not + enforce). Test the behavior. +2. **Reproducible (R-TEST-02).** Deterministic, hermetic, version-pinned, single-command, and + environment-agnostic. Any conforming cluster + a kubeconfig ⇒ the same pass/fail. No flakiness + budget; eventual-consistency is handled with bounded polling, not sleeps-and-hope. +3. **Conformance gates delivery (R-TEST-03).** The suite blocks merges (for touched requirements) + and blocks phase exits (full phase suite). Green suite is the definition of progress. +4. **Traceable (R-TEST-04).** Each test declares its `R-*` IDs; a generated coverage report maps + every requirement to its verifying test(s). Uncovered requirements are visible gaps. +5. **Fault-injection for safety-critical behavior (R-TEST-05).** Default-deny, VPN kill-switch, + data-survival, DR restore, and auto-rollback are proven by *breaking things* and asserting + fail-closed / no-data-loss — not by happy-path checks. +6. **Specs are the fixed target (R-TEST-06).** Tests encode the spec's intent. Agents never weaken a + test or a requirement to get green; a genuine inability to meet the spec is an escalation to the + human, not a spec/test edit. + +## Reproducibility contract (what "reproducible" concretely requires) + +- **Single command** runs the suite (or a tagged subset) locally and in CI. +- **Hermetic fixtures**: each test provisions its own namespace/workloads/probes and tears them down; + it never depends on pre-existing mutable state or on another test's side effects. +- **Deterministic**: fixed inputs, pinned tool/image versions, bounded retries with explicit + timeouts for eventual consistency; no reliance on wall-clock timing or ordering. +- **Isolated & parallel-safe**: unique namespaces/labels; tests can run concurrently without + cross-talk. +- **Machine-readable output**: JUnit (or equivalent) plus a requirement-coverage report; every + result carries the `R-*` tags it verifies. +- **Self-cleaning**: on pass or fail, fixtures are removed so a re-run starts clean (leaked fixtures + are themselves a test failure). +- **Versioned with the code**: tests live in the implementation repo next to what they verify and + evolve only when the human-owned spec they encode changes. + +## Test layers + +| Layer | Runs | Verifies | Example | +|---|---|---|---| +| **L0 Static/policy** | Pre-apply, in CI | Manifests/ServiceDefinitions obey invariants | Every namespace has a default-deny baseline; no plaintext secrets; schema-valid ServiceDefinition | +| **L1 Rendering** | CI, no cluster | ServiceDefinition → rendered resources match golden expectations | `egress: vpn` renders the kill-switched egress path | +| **L2 Behavioral/integration** | Against dev/test cluster | Real runtime behavior | Default-deny probe, dependency allow/deny, egress-IP, TLS validity, DNS, PVC rebuild | +| **L3 Fault-injection** | Against dev/test cluster | Fail-closed / no-data-loss under failure | VPN tunnel forced down ⇒ no leak; node killed ⇒ data survives; bad update ⇒ auto-rollback; DR restore | +| **L4 Conformance suite** | CI + on-demand | Aggregate, requirement-tagged gate | The union of the above, reported per `R-*` | + +## Behavior test catalog (normative — implementers author these as automated tests) + +Each maps to requirement IDs; phrased as Given/When/Then intent. + +- **Default-deny (R-NET-01):** *Given* a fresh workload with no allow rules, *When* it attempts any + in-cluster connection except DNS, *Then* the connection fails; DNS succeeds. +- **Explicit allow / deny-undeclared (R-NET-02):** *Given* declared `dependencies`, *When* the + source reaches a declared dest port, *Then* it succeeds; *When* it reaches an **undeclared** dest, + *Then* it fails. +- **Egress class (R-NET-05):** *Given* `egress: internet` vs `egress: vpn` workloads, *When* each + queries an external echo of its public IP, *Then* the internet one shows the home/WAN IP and the + vpn one shows the VPN IP. +- **VPN kill-switch (R-NET-06, L3):** *Given* a `vpn` workload with a healthy tunnel, *When* the + tunnel is forced down, *Then* the workload has **zero** egress to the plain internet and + `internet`-class workloads are unaffected; *When* the tunnel recovers, connectivity returns. +- **Egress flip (R-NET-07):** *Given* a running service, *When* its `egress` class is changed in + git and reconciled, *Then* its observed egress path changes with no workload rewrite. +- **Data survival (R-STATE-01, L3):** *Given* a stateful service with written data, *When* the + workload is destroyed and re-applied from git, *Then* the data is intact. +- **Node loss (R-STATE-02, L3):** *Given* a 3-replica volume, *When* a node is removed, *Then* the + volume stays available and the cluster reconverges. +- **DR restore (R-STATE-03, L3):** *Given* an off-cluster backup, *When* a namespace + volumes are + restored into a clean state, *Then* the service returns with its data. +- **Trusted TLS (R-TLS-01/02):** *Given* any exposed endpoint (incl. `internal`), *When* its cert is + inspected, *Then* it is valid, non-self-signed, unexpired; renewal occurs before expiry with no + manual step. +- **DNS/exposure (R-DNS-*):** *Given* a service, *When* resolved, *Then* the record exists and + internal clients hit the internal Gateway; `internal` is unreachable publicly, `public` is + reachable + TLS-valid. +- **Hands-off update (R-UPD-01/02, L3):** *Given* an out-of-date service under a stable policy, + *When* the scan interval elapses, *Then* it advances with no human/AI edit; *Given* a deliberately + bad update, *Then* health-gating auto-rolls-back and an email alert fires; *Given* a pinned + service, *Then* it does not advance. +- **VM lifecycle (R-PLAT-03, L2/L3):** *Given* a `kind: vm` ServiceDefinition, *When* create/start/ + stop/destroy via `kubectl`/`virtctl`, *Then* it behaves; its disk survives rebuild; default-deny + + egress class apply. +- **Alerting (R-OBS-02/04, L3):** *Given* a monitored condition, *When* a fault is injected, *Then* + an email alert is delivered and the dashboard reflects it. +- **GitOps drift (R-GITOPS-03, L3):** *Given* a reconciled resource, *When* it is mutated live, + *Then* Flux reverts it to the git-declared state. +- **Onboarding completeness (WS-APPS):** *Given* only a merged ServiceDefinition, *Then* the service + is fully wired (compute/storage/policy/egress/TLS/DNS/monitoring/update) with no manual step — + asserted by the union of the relevant tests above against that service. + +## Gating rules (R-TEST-03) + +- **Per-change (merge gate):** the conformance subset tagged with the requirement IDs the change + touches MUST pass before merge. +- **Per-phase (exit gate):** the full conformance suite for the phase's requirements MUST pass + before the phase's exit criteria are considered met ([`../roadmap/phases.md`](../roadmap/phases.md)). +- **Regression:** the growing suite runs continuously against dev/test; a previously-green + requirement going red blocks further delivery until fixed. +- **Coverage:** a requirement with no passing test is **not** satisfied, regardless of implementation + state. + +## Tooling (pinned — [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md)) + +The approach ([ADR-0010](../decisions/adr-0010-conformance-testing.md)) is realized with two tools +from the **Kyverno ecosystem**: + +- **Kyverno** — **L0 static/policy gate _and_ in-cluster admission enforcer.** In CI, the Kyverno CLI + (`kyverno apply` / `kyverno test`, incl. `kyverno-json` for ServiceDefinitions) validates manifests + and rendered resources against platform invariants (default-deny baseline present, no plaintext + secrets, required labels/egress-class, schema conformance). **The same policies also run in-cluster + as an admission controller**, rejecting any non-conforming resource at reconcile time (defense in + depth — a resource that somehow bypassed CI still cannot be admitted). Enforcement **never replaces + behavior tests**: a policy admitting a manifest is not proof the runtime behaves (R-TEST-01), so + L2/L3 Chainsaw tests remain mandatory. Policies are authored once and shared between the CLI gate + and the admission controller. +- **Chainsaw** — **L1 rendering, L2 behavioral, and L3 fault-injection.** Declarative steps + (apply/assert/error, `script`/`command`, `try`/`catch`/`finally`), bounded polling with explicit + timeouts, per-test namespace isolation, and automatic cleanup satisfy the reproducibility contract. + Fault-injection (force the tunnel down, cordon/kill a node, inject a bad image, trigger a restore) + is expressed as Chainsaw `script` steps, asserting fail-closed / no-data-loss via probe fixtures, + with `finally` restoring state. + +Probes are ephemeral job pods; egress-IP checks use an external echo; tests carry their `R-*` IDs +(labels/annotations/naming) so the coverage report is generated automatically; both tools' output +feeds JUnit + the requirement-coverage report and the CI merge / phase-exit gates. Tool versions are +pinned for determinism. + +### Where each layer runs +- **L0 (Kyverno CLI) and L1 (rendering assertions)** run **in CI without a cluster** — pure manifest/ + ServiceDefinition validation; fast merge-gate feedback. +- **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job** on the dev/test cluster. + CI **triggers** the Job (and waits for it); the Job — not CI — holds cluster access, so CI never + needs a broad kubeconfig. This keeps credentials in-cluster and least-privileged (R-SEC-02). + - The Job runs under a **dedicated, scoped test ServiceAccount** whose RBAC grants exactly what the + behavior/fault tests need (create/delete test namespaces + fixtures + probes, apply/patch + NetworkPolicies, scale the VPN sidecar/gateway, cordon/drain a node, inject a bad image tag, + trigger a Velero restore). This is a **privileged test identity** — it MUST be scoped, named, and + documented, MUST NOT be reused by workloads, and its own reach is itself an audit item. + - The Job writes **JUnit + coverage artifacts** to a known location (log stream and/or a results + PVC/object) that CI collects to enforce the merge / phase-exit gates. + - Fixtures are hermetic and self-cleaning (`finally` steps); a leaked fixture fails the run. + +## Acceptance gates (for the testing capability itself) +- A single command runs the suite and emits requirement-tagged JUnit + coverage. +- Every `R-*` requirement maps to at least one passing behavior test (coverage report proves it). +- Safety-critical requirements each have a **fault-injection** test that fails closed / loses no + data. +- Re-running the suite yields identical results (reproducibility) with no leaked fixtures. + +## Open questions +- Tool mix is **resolved**: Kyverno (L0) + Chainsaw (L1/L2/L3), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md). +- Harness location is **resolved**: L2/L3 run as an **in-cluster Chainsaw Job** triggered by CI; CI + holds no cluster credentials. +- Kyverno-as-admission-enforcer is **resolved**: yes — the L0 policies also enforce at admission + in-cluster (defense in depth), in addition to the CI gate. +- Delegated to WS-TEST at implementation time (not human decisions): the **exact RBAC scope** of the + test ServiceAccount (author it **least-privilege** for precisely the fault-injection actions it + performs, [07](07-security-and-secrets.md)) and the **results-artifact mechanism** (log stream vs + results PVC/object store). Both are implementer's discretion within the stated constraints. diff --git a/roadmap/phases.md b/roadmap/phases.md new file mode 100644 index 0000000..236012e --- /dev/null +++ b/roadmap/phases.md @@ -0,0 +1,108 @@ +# Roadmap — Delivery Phases + +Delivery is phased. Each phase has an **entry gate** (what must be true to start), **exit criteria** +(demonstrable outcomes tied to requirement IDs), and the **workstreams** ([workstreams.md](workstreams.md)) +that execute it. **Everything is proven in dev/test; prod is design-only until Phase 7.** + +A phase's exit criteria are **passing automated conformance tests** (per [`../AGENTS.md` §7](../AGENTS.md) +and [`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md)), not "manifests +merged" and not manual demos. Every "**Prove:**" / "**demonstrate**" item below is an automated, +reproducible, requirement-tagged behavior test committed with the work. A phase exits only when its +full conformance subset is green. **The conformance harness (WS-TEST) is stood up in Phase 0, so +every subsequent phase ships with tests, not after them.** + +--- + +## Phase 0 — Foundations & Repo Bootstrap +**Entry:** specs (this repo) approved. +**Do:** +- Create the implementation repo structure ([`../docs/05-gitops-and-delivery.md`](../docs/05-gitops-and-delivery.md)). +- Stand up CI (lint, schema validation, dry-run/build) for manifests and ServiceDefinitions. +- Author the ServiceDefinition schema + renderer contract ([`../docs/13-service-catalog.md`](../docs/13-service-catalog.md)). +- **Stand up the conformance harness (WS-TEST):** single-command runner, reproducibility contract, + requirement-tagged JUnit + coverage report, L0 policy-as-code gate, and the CI merge gate + ([`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md), [ADR-0010](../decisions/adr-0010-conformance-testing.md)). +**Exit:** implementation repo + CI exist; ServiceDefinition schema validated; **conformance harness +runs green with the coverage report wired**; no cluster yet. +**Serves:** R-GITOPS-02, R-ENV-02, R-TEST-01..04. + +## Phase 1 — dev/test Cluster Bring-up (IaC) +**Entry:** Phase 0 done. +**Do:** OpenTofu libvirt node VMs + cloud-init baseline + Ansible k3s HA install with **Cilium**; +disable bundled flannel/kube-proxy/Traefik; bootstrap **Flux**; bootstrap **SOPS age key**. +**Exit:** 3-node k3s HA cluster with Cilium up; Flux reconciling from git; a dev node VM can be +destroyed and rebuilt from IaC. **Prove:** node-loss re-provision. +**Serves:** R-PLAT-01, R-ENV-01, R-GITOPS-01, ADR-0001/0002/0004/0007. + +## Phase 2 — Platform Core: Network, Storage, TLS, DNS, Ingress +**Entry:** Phase 1 done. +**Do:** Longhorn (3-replica) + bulk-tier plan; default-deny network baseline + Hubble; Cilium +Egress Gateway; cert-manager + Let's Encrypt (DNS-01, wildcard); ExternalDNS; Gateway API listeners; +split-horizon DNS foundation. +**Exit (demonstrate):** +- Default-deny holds: a probe pod can reach only DNS (R-NET-01). +- A PVC survives pod destroy/rebuild (R-STATE-01). +- A test host serves a **trusted, non-self-signed** cert at `*.home.keithjackson.dev` (R-TLS-01/03). +- Auto DNS record appears for a test route (R-DNS-02). +**Serves:** R-NET-01/02/03, R-STATE-01/02, R-TLS-01/02/03, R-DNS-01/02/03, ADR-0002/0003/0006. + +## Phase 3 — Egress, VPN & Multi-WAN +**Entry:** Phase 2 done; OQ-1/OQ-3 answered enough to build. +**Do:** implement egress classes; VPN egress (Gluetun sidecar default) with **kill-switch**; +integrate edge-router multi-WAN split for the `internet` class. +**Exit (demonstrate):** +- `vpn` test workload egresses via VPN IP; `internet` workload via home/WAN IP (R-NET-05). +- **Kill-switch:** tunnel forced down ⇒ `vpn` workload leaks nothing; `internet` unaffected (R-NET-06). +- Flipping `egress` class in git re-routes without workload rewrite (R-NET-07). +**Serves:** R-NET-04/05/06/07, ADR-0008. + +## Phase 4 — Observability, Backup/DR & Automated Updates +**Entry:** Phase 2 done (can overlap Phase 3). +**Do:** kube-prometheus-stack + Loki + Alloy + Alertmanager(email); platform-health alerts; Velero + +Longhorn off-cluster backup; Flux image-automation + Renovate; per-service update policy plumbing. +**Exit (demonstrate):** +- Injected fault emails an alert; dashboards render (R-OBS-01/02/03/04). +- DR drill: restore a namespace + volumes from off-cluster backup (R-STATE-03). +- A test service auto-advances a stable version on schedule with no human/AI edit; a bad update + auto-rolls-back and alerts (R-UPD-01/02/03). +**Serves:** R-OBS-*, R-STATE-03, R-UPD-*, ADR-0009/0003. + +## Phase 5 — IaaS Proof (KubeVirt) +**Entry:** Phases 2–3 done. +**Do:** KubeVirt + CDI; run one representative application VM via ServiceDefinition `kind: vm`. +**Exit (demonstrate):** create/start/stop/destroy a VM via `kubectl`/`virtctl`; Longhorn-backed disk +survives rebuild; default-deny + `egress` apply to the VM; guest auto-updates. +**Serves:** R-PLAT-03/04, R-STATE-01, R-NET-*, R-UPD-*, ADR-0005. + +## Phase 6 — Service Catalog Onboarding + C2 UI +**Entry:** Phases 2–4 done. +**Do:** onboard the 13 services via ServiceDefinitions in dependency order (see workstream WS-APPS); +stand up the C2 UI (Homepage root + Grafana + Headlamp + Flux status). +**Exit (demonstrate):** each service fully wired from its ServiceDefinition (compute/storage/policy/ +egress/TLS/DNS/monitoring/updates) with default-deny + correct egress; the arr↔qbittorrent↔jackett↔ +seerr↔plex dependency graph works and nothing undeclared does; `home.keithjackson.dev` serves the C2 +with live links + health. +**Serves:** R-C2-01/02, all app-facing requirements, [`../docs/13-service-catalog.md`](../docs/13-service-catalog.md). + +## Phase 7 — Production Readiness (design → build) +**Entry:** Phases 1–6 proven in dev/test; operator declares prod hardware ready. +**Do:** finalize bare-metal provisioning checklist; apply the **prod overlay** (no base edits); +full DR drill on prod; cutover plan. +**Exit:** prod 3× mini-PC cluster runs the same config via the prod overlay; DR proven; monitoring + +alerts live. +**Serves:** R-ENV-02/03, R-PLAT-01. +**Note:** Do not begin building on prod hardware before this phase is explicitly opened +([`../AGENTS.md` §6](../AGENTS.md)). + +--- + +## Dependency flow + +``` +P0 → P1 → P2 ─┬→ P3 ─┐ + ├→ P4 ──┼→ P6 → P7 + └───────┘ ↑ + P5 ───┘ +``` + +P3, P4, P5 can proceed in parallel once P2 lands; P6 needs P2–P4; P7 needs everything proven. diff --git a/roadmap/workstreams.md b/roadmap/workstreams.md new file mode 100644 index 0000000..33a677c --- /dev/null +++ b/roadmap/workstreams.md @@ -0,0 +1,59 @@ +# Roadmap — Workstreams + +A **workstream** is a durable area of ownership. Each maps to an agent role +([`../agents/roles.md`](../agents/roles.md)), owns a set of specs, exposes **interfaces** other +workstreams depend on, and delivers against requirement IDs. Interfaces are contracts — change them +only via [`../agents/coordination-protocol.md`](../agents/coordination-protocol.md). + +| ID | Workstream | Owns (specs) | Serves | Depends on | +|---|---|---|---|---| +| **WS-INFRA** | Infra & cluster bring-up | [02](../docs/02-environments-and-provisioning.md) | R-PLAT-01, R-ENV-* | — | +| **WS-GITOPS** | GitOps, CI, catalog renderer | [05](../docs/05-gitops-and-delivery.md), [13](../docs/13-service-catalog.md) | R-GITOPS-*, R-UPD-02 | WS-INFRA | +| **WS-NET** | Networking, isolation, egress, ingress | [03](../docs/03-networking-isolation-and-egress.md), [09](../docs/09-dns-and-ingress.md) | R-NET-*, R-DNS-* | WS-INFRA | +| **WS-STORAGE** | Storage, state, backup/DR | [04](../docs/04-storage-and-state.md) | R-STATE-* | WS-INFRA | +| **WS-SEC** | Secrets, TLS, RBAC, auditability | [07](../docs/07-security-and-secrets.md), [08](../docs/08-tls-and-certificates.md) | R-SEC-*, R-TLS-* | WS-INFRA, WS-NET | +| **WS-OBS** | Observability & alerting | [10](../docs/10-observability-and-alerting.md) | R-OBS-* | WS-INFRA, WS-STORAGE | +| **WS-UPDATE** | Automated updates | [11](../docs/11-automated-updates.md) | R-UPD-* | WS-GITOPS, WS-OBS | +| **WS-VM** | KubeVirt IaaS | [06](../docs/06-vm-workloads-kubevirt.md) | R-PLAT-03 | WS-NET, WS-STORAGE | +| **WS-APPS** | Service onboarding (13 services) | [13](../docs/13-service-catalog.md) | app requirements | WS-GITOPS, WS-NET, WS-STORAGE, WS-SEC, WS-OBS | +| **WS-C2** | Command & Control UI | [12](../docs/12-command-center-ui.md) | R-C2-* | WS-OBS, WS-APPS, WS-SEC | +| **WS-TEST** | Conformance testing (cross-cutting) | [14](../docs/14-testing-and-conformance.md) | R-TEST-* | WS-INFRA, WS-GITOPS | + +## Key interfaces (contracts between workstreams) + +- **ServiceDefinition schema** (WS-GITOPS owns) — the operator surface every app/VM is expressed in. + WS-APPS, WS-VM consume it; WS-NET/STORAGE/SEC/OBS render pieces of it. **The most important + contract in the system.** +- **Network baseline & policy rendering** (WS-NET) — default-deny + how `dependencies` and `egress` + compile to CiliumNetworkPolicies/egress routing. Consumed by WS-APPS/WS-VM. +- **StorageClasses** (WS-STORAGE) — `longhorn` (replicated) and `bulk` (media). Consumed by everyone + with state. +- **Issuer + Gateway TLS** (WS-SEC + WS-NET) — wildcard cert + Gateway listeners that HTTPRoutes + attach to. +- **Observability contract** (WS-OBS) — how a service declares metrics/SLIs/alerts and gets + dashboards. Consumed by WS-APPS/WS-VM. +- **Update policy contract** (WS-UPDATE) — ImagePolicy/Renovate conventions a ServiceDefinition's + `update` block maps to. +- **Conformance harness + gate** (WS-TEST) — the reproducible test framework (**Kyverno** for L0 + policy, **Chainsaw** for L1/L2/L3 behavior + fault-injection, [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md)), + requirement-coverage report, and CI/phase gates every workstream's tests plug into. Each workstream + **authors its own behavior tests** against this harness ([14](../docs/14-testing-and-conformance.md)). + +## Parallelization guidance +- **WS-TEST stands up the conformance harness early (Phase 0–1)** so every other workstream lands + with passing, reproducible tests from day one — testing is not a trailing phase. +- After WS-INFRA delivers a cluster + Flux (Phase 1), WS-NET, WS-STORAGE, WS-SEC can proceed largely + in parallel (Phase 2). +- WS-OBS, WS-UPDATE, WS-VM layer on next (Phases 3–5), overlapping. +- WS-APPS is gated on the platform contracts being stable; onboard services in the dependency order + below. +- WS-C2 is last-ish (Phase 6) since it aggregates everything. + +## WS-APPS onboarding order (respect dependencies from [13](../docs/13-service-catalog.md)) +1. **pihole** (DNS underpinning) → 2. **plex** (largest; validates bulk tier + public exposure) → +3. **qbittorrent, jackett** (VPN + kill-switch proof) → 4. **sonarr, radarr** (VPN + arr↔downloader +allows) → 5. **seerr** (cross-service allows to arr + plex) → 6. **nextcloud, gitea** (DR-critical +data + backups) → 7. **minecraft, unifi, semaphore, kasm** (remaining). + +Each onboarding is a full acceptance-gate pass ([`../agents/definition-of-done.md`](../agents/definition-of-done.md)), +not just a merged manifest.