Initial baseline: home platform (k3s + Longhorn) specification repository
Greenfield spec set for a GitOps-driven PaaS+IaaS home platform: vision & requirements (R-* IDs), architecture, 14 domain specs, 11 ADRs, roadmap (phases + workstreams), and AI-agent coordination (roles, protocol, definition-of-done, task board). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
37 files changed
+2830
No files matched your search
+30
@@ -0,0 +1,30 @@
|
||||
# OS / editor cruft
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
*.swp
|
||||
*~
|
||||
.idea/
|
||||
.vscode/
|
||||
|
||||
# Secrets — never commit plaintext (secrets live SOPS-encrypted only; see docs/07)
|
||||
*.dec.yaml
|
||||
*.decrypted.*
|
||||
*.key
|
||||
*.pem
|
||||
age.key
|
||||
*.kubeconfig
|
||||
kubeconfig
|
||||
|
||||
# IaC state / local runtime (for the future implementation tree)
|
||||
.terraform/
|
||||
*.tfstate
|
||||
*.tfstate.*
|
||||
.terraform.lock.hcl
|
||||
*.tfvars
|
||||
!*.example.tfvars
|
||||
|
||||
# Test / build artifacts
|
||||
/results/
|
||||
/reports/
|
||||
*.junit.xml
|
||||
coverage/
|
||||
@@ -0,0 +1,148 @@
|
||||
# AGENTS.md — Rules of Engagement
|
||||
|
||||
This file governs how AI agents (and their human reviewers) work inside this repository. Read it
|
||||
fully before making any change. These rules **override** any conflicting default behavior.
|
||||
|
||||
> **Two governing constraints, above all others:**
|
||||
> 1. **You may not change the specifications.** The `docs/` tree, the requirements, and accepted
|
||||
> ADRs are human-owned and immutable to agents (R-TEST-06). You implement against them and prove
|
||||
> conformance; you never edit them to fit your implementation. See §2 and §11.
|
||||
> 2. **Conformance is proven only by automated, reproducible behavior tests.** "Done" means a green,
|
||||
> requirement-tagged conformance test — never "merged," never a manual demo (R-TEST-01..05).
|
||||
> See §7 and [`docs/14-testing-and-conformance.md`](docs/14-testing-and-conformance.md).
|
||||
|
||||
## 1. This repository defines intent, not implementation (yet)
|
||||
|
||||
Today this repo contains **specifications only**. Implementation artifacts (Helm values, Flux
|
||||
Kustomizations, KubeVirt manifests, OpenTofu/Ansible IaC, etc.) will be added under an
|
||||
`implementation/` tree **only after** the corresponding spec and ADRs are approved. Until then:
|
||||
|
||||
- Do **not** invent implementation details that contradict the specs or ADRs.
|
||||
- Do **not** create the `implementation/` tree until the roadmap phase that authorizes it is active
|
||||
(see [`roadmap/phases.md`](roadmap/phases.md)).
|
||||
- When you implement, the spec is the acceptance criteria. If the spec is wrong, blocking,
|
||||
ambiguous, or missing, **you may not fix it yourself** — stop and escalate to the human operator/
|
||||
architect (§2, §11). You may draft a proposal; you may not merge a spec change.
|
||||
|
||||
## 2. The source-of-truth hierarchy
|
||||
|
||||
When two documents conflict, the higher item wins:
|
||||
|
||||
1. [`docs/00-vision-and-scope.md`](docs/00-vision-and-scope.md) — requirements (`R-*` IDs)
|
||||
2. Approved ADRs in [`decisions/`](decisions/) — locked technology & pattern choices
|
||||
3. Domain specs in [`docs/`](docs/)
|
||||
4. [`roadmap/`](roadmap/) and [`agents/`](agents/) — sequencing and coordination
|
||||
5. Implementation code (once it exists)
|
||||
|
||||
If you find a conflict, **stop and escalate it to the human operator/architect** — do not silently
|
||||
pick one interpretation, and do not resolve it by editing the specs yourself (you may not change
|
||||
them, §11). Wait for the human to amend the spec, then implement against the amended spec.
|
||||
|
||||
## 3. Do not re-litigate decisions
|
||||
|
||||
Every major choice (k3s, Cilium, Longhorn, KubeVirt, Flux, SOPS, cert-manager, etc.) is captured in
|
||||
an ADR with rationale and rejected alternatives. If new information genuinely invalidates a
|
||||
decision:
|
||||
|
||||
1. **Draft** a proposed superseding ADR and **escalate to the human operator/architect** — you may
|
||||
not merge it yourself (ADRs are specifications; §11 forbids agents changing them).
|
||||
2. State what changed and why.
|
||||
3. Only the human approves/accepts it. Implement against the new decision only after it is accepted.
|
||||
|
||||
Never edit an accepted ADR's decision in place. ADRs are an append-only, **human-approved** ledger.
|
||||
|
||||
## 4. Requirement traceability is mandatory
|
||||
|
||||
- Every requirement has a stable ID like `R-NET-01`.
|
||||
- Every ADR lists the requirement IDs it serves.
|
||||
- Every task-board item lists the requirement IDs it satisfies.
|
||||
- Every implementation PR references the requirement IDs and the spec sections it fulfills.
|
||||
|
||||
If you cannot trace a piece of work to a requirement, either it is out of scope or a requirement is
|
||||
missing — surface it, don't proceed silently.
|
||||
|
||||
## 5. GitOps-first, click-ops-never
|
||||
|
||||
- The desired state of the cluster lives in git and is reconciled automatically.
|
||||
- Manual changes to the running cluster are prohibited **except** where an ADR explicitly carves out
|
||||
an exception (e.g. a bootstrap step, an out-of-band edge-router config). Such exceptions must be
|
||||
documented and, where possible, later folded back into IaC.
|
||||
- Secrets live in git **encrypted** (SOPS + age). Never commit plaintext secrets. Never print a
|
||||
decrypted secret into logs, PR descriptions, or agent output.
|
||||
|
||||
## 6. Two environments, one configuration
|
||||
|
||||
- Author configuration so dev/test and prod differ **only** by environment-scoped overrides
|
||||
(replica counts, node selectors, storage sizes, WAN endpoints, DNS zones).
|
||||
- Never fork the base configuration per environment. Use overlays/variables.
|
||||
- **Never build directly on production.** Prod hardware is a design target, not a development
|
||||
surface, until the roadmap declares it ready.
|
||||
|
||||
## 7. Acceptance gates — proven by automated, reproducible behavior tests (non-negotiable)
|
||||
|
||||
A change is not "done" until an **automated, reproducible, behavior-based test** (Given/When/Then,
|
||||
tagged with the requirement IDs it verifies) **passes green in dev/test** and is committed alongside
|
||||
the change. A manual demonstration, a screenshot, or "it reconciled" is **not** acceptance
|
||||
(R-TEST-01..04). The approach — reproducibility rules, layers, tooling, and the conformance suite —
|
||||
is specified in [`docs/14-testing-and-conformance.md`](docs/14-testing-and-conformance.md) and
|
||||
[`decisions/adr-0010-conformance-testing.md`](decisions/adr-0010-conformance-testing.md).
|
||||
|
||||
The behaviors each change must prove (each expressed as an automated test):
|
||||
|
||||
- **No data loss on rebuild** — destroy and recreate the workload; state survives (R-STATE-*).
|
||||
- **Default-deny holds** — the workload cannot reach services it wasn't explicitly allowed to
|
||||
(R-NET-*).
|
||||
- **Correct egress path** — internet-class services egress locally; VPN-class services egress only
|
||||
through the VPN, with a verified kill-switch (R-NET-*).
|
||||
- **Trusted TLS** — the endpoint serves a valid, non-self-signed, auto-renewing certificate
|
||||
(R-TLS-*).
|
||||
- **Observed** — metrics, logs, and at least one meaningful alert exist and fire correctly
|
||||
(R-OBS-*).
|
||||
- **Hands-off updates** — the service advances to new stable versions automatically per policy,
|
||||
with no hand-authored change (R-UPD-*).
|
||||
|
||||
See [`agents/definition-of-done.md`](agents/definition-of-done.md) for the full checklist.
|
||||
|
||||
## 8. Coordination
|
||||
|
||||
- Claim work on [`agents/task-board.md`](agents/task-board.md) before starting; respect declared
|
||||
dependencies (`blocked-by`).
|
||||
- Follow [`agents/coordination-protocol.md`](agents/coordination-protocol.md) for handoffs,
|
||||
interface contracts, and conflict resolution between agents/workstreams.
|
||||
- Prefer small, reviewable, single-responsibility changes. One workstream should not silently change
|
||||
another workstream's interfaces — negotiate the contract change first.
|
||||
|
||||
## 9. Style & conventions
|
||||
|
||||
- Markdown for all specs. One concept per file. Link generously between documents.
|
||||
- Use requirement IDs and ADR numbers in prose so references stay stable when files move.
|
||||
- Be decisive in specs. Where a genuine open question remains, record it under an explicit
|
||||
**Open Questions** heading rather than burying ambiguity in the body.
|
||||
- Keep the [`GLOSSARY.md`](GLOSSARY.md) authoritative — if you introduce a term, define it there.
|
||||
|
||||
## 10. What "you do not write code" means for the architect vs. what agents do
|
||||
|
||||
The **architect** authored the specs, ADRs, and this coordination structure — no code. The
|
||||
**implementation agents** who consume this repo *do* write code (implementation *and* the automated
|
||||
conformance tests that prove it), but only in service of an approved spec, only under the active
|
||||
roadmap phase, and only against the automated acceptance gates above. Agents write tests; agents do
|
||||
not write specs.
|
||||
|
||||
## 11. Specifications are immutable to agents (governance)
|
||||
|
||||
This is a hard boundary (R-TEST-06):
|
||||
|
||||
- **You MUST NOT modify** anything under [`docs/`](docs/), the requirements, accepted ADRs in
|
||||
[`decisions/`](decisions/), the roadmap, or these coordination files. These are human-owned.
|
||||
- **You MAY and MUST create/modify**: implementation code, **automated conformance tests**, the
|
||||
implementation repo, and the *live coordination state* the specs explicitly delegate to you —
|
||||
namely claiming/updating items on [`agents/task-board.md`](agents/task-board.md) and recording
|
||||
proposals/escalations there. (The task board is a working surface; the *rules* around it are not.)
|
||||
- **When the spec is wrong, blocking, ambiguous, or missing:** STOP. Do not proceed on a guess and
|
||||
do not edit the spec. Raise an escalation on the task board (and draft a proposal if useful) for
|
||||
the **human operator/architect** to decide. Resume only after the human amends the spec.
|
||||
- **Never** make the implementation pass by weakening the spec, the requirement, or the test's
|
||||
intent. If reality can't meet the spec, that is an escalation, not a spec edit.
|
||||
|
||||
The point: the specification is the fixed target. Agents move the implementation and its tests
|
||||
toward the target; they never move the target.
|
||||
+90
@@ -0,0 +1,90 @@
|
||||
# Glossary
|
||||
|
||||
Shared vocabulary for this platform. If you introduce a term, define it here. When a word could mean
|
||||
two things (e.g. "VM"), this file disambiguates.
|
||||
|
||||
## Platform concepts
|
||||
- **PaaS** — Platform-as-a-Service; here, containerized workloads onboarded via GitOps.
|
||||
- **IaaS** — Infrastructure-as-a-Service; here, virtual machines run *inside* the cluster via
|
||||
KubeVirt, managed through the same Kubernetes surface.
|
||||
- **ServiceDefinition** — the operator-facing YAML object that declares a service's intent
|
||||
(kind, source, storage, egress, exposure, dependencies, update policy, observability). The
|
||||
platform renders concrete resources from it. See [`docs/13-service-catalog.md`](docs/13-service-catalog.md).
|
||||
- **Golden path** — the standard flow a service follows from catalog entry to running/monitored/
|
||||
updated. See [`docs/01-architecture-overview.md`](docs/01-architecture-overview.md).
|
||||
- **C2 / Command & Control** — the UI at the root `home.keithjackson.dev`. See
|
||||
[`docs/12-command-center-ui.md`](docs/12-command-center-ui.md).
|
||||
|
||||
## The two kinds of VM (do not conflate)
|
||||
- **Infrastructure VM (Layer 1)** — a cluster *node*. In dev/test these are libvirt KVMs; in prod
|
||||
they are bare-metal mini-PCs. Managed by OpenTofu/Ansible, **not** KubeVirt.
|
||||
- **Application VM (Layer 2)** — an IaaS *workload* running inside the cluster via **KubeVirt**,
|
||||
managed by Kubernetes/GitOps like a container.
|
||||
|
||||
## Networking
|
||||
- **Default-deny** — the baseline where no service may talk to another unless explicitly allowed at
|
||||
the port level (R-NET-01).
|
||||
- **Egress class** — a service's declared outbound path: `internet` (standard WAN) or `vpn`
|
||||
(commercial VPN tunnel). See [`docs/03-networking-isolation-and-egress.md`](docs/03-networking-isolation-and-egress.md).
|
||||
- **Exposure class** — how a service is reachable: `internal` (LAN-only) or `public` (edge
|
||||
port-forwarded).
|
||||
- **Kill-switch** — the guarantee that a `vpn`-class workload leaks nothing to the plain internet if
|
||||
the tunnel drops (R-NET-06); fail-closed.
|
||||
- **Multi-WAN** — multiple upstream internet uplinks with routing-rule-based selection; physical WAN
|
||||
selection is largely an edge-router responsibility (OQ-1).
|
||||
- **Egress Gateway** — Cilium feature steering selected pods' egress through a chosen node/identity.
|
||||
- **Hubble** — Cilium's flow-observability tool; the audit surface for isolation/egress.
|
||||
- **Gateway API** — the Kubernetes standard for north-south ingress, implemented here by Cilium.
|
||||
- **Split-horizon DNS** — serving different answers to internal vs. external clients for the same
|
||||
name.
|
||||
|
||||
## Storage
|
||||
- **Longhorn** — distributed replicated block storage across the 3 nodes (default 3 replicas). The
|
||||
durable home for app/config state and VM disks.
|
||||
- **Bulk tier** — a separate storage class for large media (Plex library, downloads) that is **not**
|
||||
3× replicated (R-STATE-04). See [`docs/04-storage-and-state.md`](docs/04-storage-and-state.md).
|
||||
- **CDI** — Containerized Data Importer; provisions KubeVirt VM disks onto Longhorn PVCs.
|
||||
- **Velero** — cluster backup/restore orchestration to an off-cluster target (DR).
|
||||
- **DR drill** — a tested restore of resources + volumes proving R-STATE-03.
|
||||
|
||||
## Delivery & lifecycle
|
||||
- **GitOps** — desired state lives in git and is reconciled into the cluster automatically
|
||||
(pull-based). Engine: **Flux CD**.
|
||||
- **Drift correction** — Flux reverting live changes back to git-declared state (R-GITOPS-03).
|
||||
- **Image automation** — Flux controllers advancing container image versions per declared policy
|
||||
with no human/AI edit (R-UPD-02).
|
||||
- **Renovate** — bot that opens (and auto-merges) chart/manifest version-bump PRs.
|
||||
- **SOPS + age** — the secret-encryption scheme; secrets live encrypted in git, decrypted in-cluster.
|
||||
- **cert-manager / ACME DNS-01 / Let's Encrypt** — the trusted, automated TLS pipeline (R-TLS-*).
|
||||
- **ExternalDNS** — controller that publishes DNS records from declared service intent.
|
||||
|
||||
## Environments
|
||||
- **dev/test** — local KVM cluster; the active build target where everything is proven.
|
||||
- **prod** — 3× mini-PC bare-metal cluster; design-only until roadmap Phase 7.
|
||||
- **Overlay** — environment-scoped configuration (Kustomize/vars) layered on shared base config; the
|
||||
only thing that differs between dev and prod (R-ENV-02).
|
||||
|
||||
## Governance
|
||||
- **Requirement ID (`R-*`)** — a stable, referenceable requirement from
|
||||
[`docs/00-vision-and-scope.md`](docs/00-vision-and-scope.md).
|
||||
- **ADR** — Architecture Decision Record; an append-only, superseding-only decision ledger under
|
||||
[`decisions/`](decisions/).
|
||||
- **Workstream (`WS-*`)** — a durable area of ownership; maps to an agent role.
|
||||
- **Acceptance gate** — an **automated, reproducible behavior test** a change must pass to be "done"
|
||||
([`agents/definition-of-done.md`](agents/definition-of-done.md), [`docs/14-testing-and-conformance.md`](docs/14-testing-and-conformance.md)).
|
||||
- **Open Question (`OQ-n`)** — a tracked unknown that may block specific tasks until answered.
|
||||
|
||||
## Testing & conformance
|
||||
- **Behavior-based test** — a test asserting observable system behavior (Given/When/Then), not
|
||||
implementation internals; a policy object *existing* is not proof it *enforces* (R-TEST-01).
|
||||
- **Reproducible test** — deterministic, hermetic, version-pinned, single-command, environment-
|
||||
agnostic; identical pass/fail on any conforming cluster (R-TEST-02).
|
||||
- **Conformance suite** — the aggregate, requirement-tagged test set that gates merges and phase
|
||||
exits; green suite = progress (R-TEST-03). See [`docs/14-testing-and-conformance.md`](docs/14-testing-and-conformance.md).
|
||||
- **Fault-injection test** — a test that *breaks* something (tunnel down, node killed, bad update)
|
||||
and asserts a fail-closed / no-data-loss outcome; required for safety-critical behavior (R-TEST-05).
|
||||
- **Coverage report** — the generated map of every `R-*` requirement → its verifying test(s); an
|
||||
uncovered requirement is an unmet requirement (R-TEST-04).
|
||||
- **Spec immutability (to agents)** — the rule that AI agents may not modify the specifications
|
||||
(`docs/`, requirements, accepted ADRs, roadmap); they escalate spec problems to the human and never
|
||||
weaken a spec/test to pass (R-TEST-06). Agents edit implementation, tests, and the task board only.
|
||||
@@ -0,0 +1,105 @@
|
||||
# Home Platform — k3s + Longhorn Onboarding Platform
|
||||
|
||||
> **This repository is a specification, not an implementation.** It defines *what* to build
|
||||
> and *how the work is organized* so that teams of AI agents (and their human reviewers) can
|
||||
> implement a self-service onboarding platform for home infrastructure at both the **PaaS**
|
||||
> (containerized apps) and **IaaS** (virtual machines) level.
|
||||
|
||||
## What we are building
|
||||
|
||||
A single, GitOps-driven platform, running on a **k3s** Kubernetes cluster backed by **Longhorn**
|
||||
distributed storage, that lets an operator onboard, run, update, and tear down services — both
|
||||
**containers** and **virtual machines** — through one consistent Kubernetes API, CLI, and YAML
|
||||
configuration surface.
|
||||
|
||||
The platform must provide:
|
||||
|
||||
- **PaaS**: containerized workloads onboarded via GitOps.
|
||||
- **IaaS**: virtual machines managed through the *same* Kubernetes API/CLI/YAML (via KubeVirt),
|
||||
for workloads that cannot be containerized.
|
||||
- **Network isolation**: default-deny east-west traffic; explicit, port-level allow rules between
|
||||
services (firewall-like).
|
||||
- **Multi-WAN egress**: some services egress the normal internet; others are routed through a
|
||||
custom VPN connection. Egress path is a declarative property of a service.
|
||||
- **Stateful durability**: any service can be destroyed and rebuilt on a whim without losing data.
|
||||
- **Hands-off updates**: services update to latest stable versions on a schedule — no human,
|
||||
no AI, no hand-written GitOps change required.
|
||||
- **Trusted TLS**: every service gets an auto-issued, auto-renewed, non-self-signed certificate.
|
||||
- **Observability**: services are monitored; issues are reported by email and on dashboards.
|
||||
- **A command & control center**: the root of `home.keithjackson.dev` is a UI that surfaces and
|
||||
controls everything running in the cluster.
|
||||
- **Provable conformance**: every requirement is verified by an automated, reproducible, behavior-
|
||||
based test — and the specifications themselves are human-owned and immutable to the implementing
|
||||
agents.
|
||||
|
||||
## Two environments
|
||||
|
||||
| Environment | Substrate | Purpose | Status |
|
||||
|---|---|---|---|
|
||||
| **dev/test** | Local KVM virtual machines on a workstation, provisioned by IaC | Prove the platform end-to-end before hardware exists | **Active development target** |
|
||||
| **prod** | 3× mini-PC bare-metal k3s cluster, each with local storage | The real deployment | **Not ready — design for it, do not build on it yet** |
|
||||
|
||||
The dev/test environment is the proving ground. Everything must be authored so that the *same*
|
||||
GitOps configuration promotes to production with only environment-scoped overrides.
|
||||
|
||||
## How to use this repository (start here)
|
||||
|
||||
1. **Read [`AGENTS.md`](AGENTS.md)** — the rules of engagement for every AI agent working here.
|
||||
2. **Read [`docs/00-vision-and-scope.md`](docs/00-vision-and-scope.md)** — the north star and the
|
||||
non-negotiable requirements (with stable requirement IDs).
|
||||
3. **Read [`docs/01-architecture-overview.md`](docs/01-architecture-overview.md)** — the chosen
|
||||
stack and how the pieces fit.
|
||||
4. **Consult [`decisions/`](decisions/)** — the Architecture Decision Records (ADRs) that lock in
|
||||
every major technology choice and *why*. Do not re-litigate a decision; if you disagree, open a
|
||||
superseding ADR per the process in [`decisions/README.md`](decisions/README.md).
|
||||
5. **Pick up work from [`roadmap/`](roadmap/) and [`agents/task-board.md`](agents/task-board.md)** —
|
||||
phased delivery plan, workstreams, dependencies, and the live task board.
|
||||
|
||||
## Repository map
|
||||
|
||||
```
|
||||
home-network-k3s/
|
||||
├── README.md ← you are here
|
||||
├── AGENTS.md ← rules of engagement for AI agents
|
||||
├── GLOSSARY.md ← shared vocabulary (read to avoid ambiguity)
|
||||
├── docs/ ← the specification, by domain
|
||||
│ ├── 00-vision-and-scope.md
|
||||
│ ├── 01-architecture-overview.md
|
||||
│ ├── 02-environments-and-provisioning.md
|
||||
│ ├── 03-networking-isolation-and-egress.md
|
||||
│ ├── 04-storage-and-state.md
|
||||
│ ├── 05-gitops-and-delivery.md
|
||||
│ ├── 06-vm-workloads-kubevirt.md
|
||||
│ ├── 07-security-and-secrets.md
|
||||
│ ├── 08-tls-and-certificates.md
|
||||
│ ├── 09-dns-and-ingress.md
|
||||
│ ├── 10-observability-and-alerting.md
|
||||
│ ├── 11-automated-updates.md
|
||||
│ ├── 12-command-center-ui.md
|
||||
│ ├── 13-service-catalog.md
|
||||
│ └── 14-testing-and-conformance.md
|
||||
├── decisions/ ← ADRs (one decision per file) + index/template
|
||||
├── roadmap/ ← phases.md, workstreams.md
|
||||
└── agents/ ← roles, coordination protocol, DoD, task board
|
||||
```
|
||||
|
||||
## Requirement traceability
|
||||
|
||||
Every hard requirement has a stable ID (e.g. `R-NET-01`) defined in
|
||||
[`docs/00-vision-and-scope.md`](docs/00-vision-and-scope.md). Implementation work, ADRs, and
|
||||
task-board items must reference the requirement IDs they satisfy. This is how we prove the built
|
||||
system meets the spec.
|
||||
|
||||
## Ground rules (the short version — full detail in `AGENTS.md`)
|
||||
|
||||
- **Specs are the source of truth — and are immutable to agents.** Code and tests serve the spec.
|
||||
AI agents **may not change** the `docs/`, requirements, or accepted ADRs; a spec that is wrong or
|
||||
can't be met is an **escalation to the human**, never an edit (R-TEST-06).
|
||||
- **Done means a green, reproducible, behavior-based test.** Every requirement is verified by an
|
||||
automated, requirement-tagged conformance test ([`docs/14-testing-and-conformance.md`](docs/14-testing-and-conformance.md),
|
||||
[ADR-0010](decisions/adr-0010-conformance-testing.md)); a manual demo or "it merged" is not
|
||||
acceptance.
|
||||
- **Everything is GitOps.** No click-ops in the cluster except where an ADR explicitly allows it.
|
||||
- **Nothing is self-signed, nothing is manual, nothing loses data.** These three properties are
|
||||
acceptance gates, not aspirations — and each is proven by a test.
|
||||
- **Design once, deploy to two environments.** dev/test and prod differ only by scoped overrides.
|
||||
@@ -0,0 +1,69 @@
|
||||
# Coordination Protocol
|
||||
|
||||
How agents work together without stepping on each other. Read alongside [`roles.md`](roles.md),
|
||||
[`../roadmap/workstreams.md`](../roadmap/workstreams.md), and [`../AGENTS.md`](../AGENTS.md).
|
||||
|
||||
## 1. Claim before you build
|
||||
- Take an item from [`task-board.md`](task-board.md), set its owner and status to `in-progress`.
|
||||
- Respect `blocked-by`. Do not start blocked work; help unblock it instead.
|
||||
- Keep items small and single-responsibility.
|
||||
|
||||
## 2. Contracts are sacred
|
||||
Workstreams communicate through **interfaces** (listed in [`../roadmap/workstreams.md`](../roadmap/workstreams.md)):
|
||||
the ServiceDefinition schema, StorageClasses, the network policy/egress rendering, TLS issuer +
|
||||
Gateway, the observability contract, the update-policy contract.
|
||||
|
||||
- **Consume** a contract freely.
|
||||
- **Change** a contract only by: (1) proposing the change on the task board with the affected
|
||||
workstreams tagged, (2) getting the owning role's sign-off, (3) updating the spec that defines the
|
||||
contract in the *same* change. Never silently break a downstream consumer.
|
||||
- The **ServiceDefinition schema** is the highest-traffic contract; treat schema changes as
|
||||
platform-wide events.
|
||||
|
||||
## 3. Decisions vs. discussion — and the no-spec-change rule
|
||||
- **You may not modify the specifications** (`docs/`, requirements, accepted ADRs, roadmap) — they
|
||||
are human-owned and immutable to agents (R-TEST-06, [`../AGENTS.md` §11](../AGENTS.md)).
|
||||
- Anything that changes a locked choice ⇒ **draft** a superseding ADR and **escalate to the human**
|
||||
([`../decisions/README.md`](../decisions/README.md)); do not self-approve or merge it.
|
||||
- Genuine unknowns, or a spec that is wrong/blocking/ambiguous ⇒ **stop and escalate** on the task
|
||||
board as an **Open Question (OQ-n)** for the human to resolve in the spec; don't bury ambiguity in
|
||||
an implementation and don't patch the spec yourself.
|
||||
|
||||
## 4. Definition of done
|
||||
Nothing is "done" until its **automated, reproducible behavior test passes green** in **dev/test**,
|
||||
committed with the change and tagged to its requirement IDs ([`definition-of-done.md`](definition-of-done.md),
|
||||
[`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md)). "Merged" ≠ "done";
|
||||
"demonstrated manually" ≠ "done". You write the test as well as the implementation.
|
||||
|
||||
## 5. Phase discipline
|
||||
- Only work items in the **active phase(s)** ([`../roadmap/phases.md`](../roadmap/phases.md)) unless
|
||||
explicitly parallelizable.
|
||||
- Respect entry gates; don't start a phase whose predecessor's exit criteria aren't demonstrated.
|
||||
- **Prod (Phase 7) is closed** until the operator opens it.
|
||||
|
||||
## 6. Handoffs
|
||||
When your output feeds another workstream:
|
||||
- State the interface you're delivering and where it's documented.
|
||||
- Provide the demonstrable proof (the gate you passed) so the consumer can trust it.
|
||||
- If you changed a shared contract, announce it on the board with consumers tagged.
|
||||
|
||||
## 7. Conflict resolution
|
||||
1. Re-read the source-of-truth hierarchy ([`../AGENTS.md` §2](../AGENTS.md)); the higher document
|
||||
wins.
|
||||
2. If specs genuinely conflict or are silent, escalate to the **Platform Architect** to amend the
|
||||
spec/ADR — fix the documents first, then build.
|
||||
3. Never resolve a conflict by silently picking an interpretation in code.
|
||||
|
||||
## 8. Safety rails that never bend
|
||||
- Default-deny + VPN kill-switch are **fail-closed** — if unsure, deny/close.
|
||||
- Secrets stay encrypted; never printed.
|
||||
- Nothing that risks data ships without a tested backup/restore behind it.
|
||||
- No click-ops except ADR-sanctioned bootstrap steps.
|
||||
- **Specs are immutable to you**; a spec you can't meet is an escalation, never an edit.
|
||||
- **Never weaken a test or a requirement to get green** — that's a defect, not progress.
|
||||
|
||||
## 9. Communication artifacts
|
||||
- **task-board.md** — the live who's-doing-what + status + blockers.
|
||||
- **specs/ADRs** — durable decisions and contracts.
|
||||
- **PR descriptions** — reference requirement IDs, specs, ADRs, and the gate demonstrated.
|
||||
Keep these three in sync; they are how independent agents stay coherent.
|
||||
@@ -0,0 +1,84 @@
|
||||
# Definition of Done
|
||||
|
||||
A change is **done** only when an **automated, reproducible, behavior-based test** for each relevant
|
||||
gate below **passes green in dev/test**, the test is committed with the change and tagged with the
|
||||
requirement IDs it verifies, and the change is traced to those IDs. A manual demonstration, a
|
||||
screenshot, or "it reconciles" is **not** done (R-TEST-01..04, [`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md)).
|
||||
|
||||
> **Governance (R-TEST-06):** you may not change the specifications to reach done. If the spec can't
|
||||
> be met, that is an **escalation to the human**, not a spec/requirement/test-intent edit. Weakening
|
||||
> a test so it passes is a defect, not a completion.
|
||||
|
||||
## Universal checklist (every change)
|
||||
- [ ] Traced to one or more requirement IDs (`R-*`) and the spec/ADR it implements.
|
||||
- [ ] An **automated behavior test** (Given/When/Then) exists, is committed with the change, tagged
|
||||
with its `R-*` IDs, and **passes**; it appears in the coverage report.
|
||||
- [ ] The test is **reproducible**: single-command, hermetic (self-provisioning + self-cleaning),
|
||||
deterministic, and yields identical results on re-run.
|
||||
- [ ] Expressed as GitOps (in git, reconciled by Flux) — no click-ops except ADR-sanctioned bootstrap.
|
||||
- [ ] No plaintext secrets anywhere; secrets are SOPS-encrypted (and no secret printed in output).
|
||||
- [ ] Authored once with environment-scoped overrides (no per-env base fork).
|
||||
- [ ] Observability wired (metrics/logs) and at least the relevant alert exists.
|
||||
- [ ] No specification/ADR/requirement was modified to achieve this (R-TEST-06); any spec problem was
|
||||
escalated, not edited.
|
||||
- [ ] Result recorded on the task board / PR with a link to the passing test run.
|
||||
|
||||
## Requirement-class gates (apply those relevant to the change)
|
||||
|
||||
> Each checkbox below is the **intent of an automated behavior test** — implement it as a
|
||||
> reproducible, requirement-tagged test (see the behavior catalog in
|
||||
> [`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md)), not a manual
|
||||
> check. Safety-critical gates (marked *fault-injection*) must break the system and assert
|
||||
> fail-closed / no-data-loss (R-TEST-05).
|
||||
|
||||
### State & rebuild (R-STATE-*)
|
||||
- [ ] Destroy the workload entirely and re-apply from git; **data survives**.
|
||||
- [ ] Stateful data is on `longhorn` (replicated) or the `bulk` tier — never ephemeral/hostPath.
|
||||
- [ ] DR-critical data restores from an **off-cluster** backup (drill for at least one service/phase).
|
||||
|
||||
### Network isolation & egress (R-NET-*)
|
||||
- [ ] With no explicit allows, the workload can reach **only DNS** (default-deny proven via Hubble +
|
||||
probe).
|
||||
- [ ] Declared `dependencies` work; an **undeclared** destination is blocked (negative test).
|
||||
- [ ] `egress: internet` leaves via the home/WAN IP; `egress: vpn` leaves via the **VPN IP**.
|
||||
- [ ] **VPN kill-switch:** tunnel forced down ⇒ the `vpn` workload leaks nothing; `internet`
|
||||
workloads unaffected.
|
||||
- [ ] Changing `egress` class in git re-routes with no workload rewrite.
|
||||
|
||||
### TLS (R-TLS-*)
|
||||
- [ ] Endpoint serves a **valid, non-self-signed, unexpired** cert (external check), including for
|
||||
`internal` services.
|
||||
- [ ] Renewal is automatic (observed or verified via cert-manager) with no manual step.
|
||||
|
||||
### DNS & ingress (R-DNS-*)
|
||||
- [ ] `<service>.home.keithjackson.dev` record appears automatically and routes correctly.
|
||||
- [ ] `internal` is unreachable publicly; `public` is reachable + TLS-valid.
|
||||
- [ ] Internal clients resolve to the internal Gateway (no public hairpin).
|
||||
|
||||
### Automated updates (R-UPD-*)
|
||||
- [ ] Advances to a newer **stable** version automatically on schedule with **no human/AI edit**.
|
||||
- [ ] A deliberately bad update is health-gated: **auto-rollback** + alert.
|
||||
- [ ] A **pinned** service does not advance.
|
||||
|
||||
### Observability & alerting (R-OBS-*)
|
||||
- [ ] Metrics + logs flow; a meaningful alert **fires** in a fault test and is delivered by **email**.
|
||||
- [ ] Dashboard renders and is linked from the C2 UI.
|
||||
|
||||
### VM/IaaS (R-PLAT-03)
|
||||
- [ ] VM created/started/stopped/destroyed via `kubectl`/`virtctl` (same K8s surface).
|
||||
- [ ] VM disk is Longhorn-backed; rebuild preserves data; default-deny + egress class apply; guest
|
||||
auto-updates.
|
||||
|
||||
### Security & auditability (R-SEC-*)
|
||||
- [ ] Least-privilege RBAC (no broad cluster rights for workloads).
|
||||
- [ ] The auditability proofs (isolation / egress path / kill-switch / TLS / backup) are producible.
|
||||
|
||||
### Per-service onboarding (WS-APPS)
|
||||
- [ ] The service is **fully wired from its ServiceDefinition alone** (compute/storage/policy/egress/
|
||||
TLS/DNS/monitoring/update) — no manual steps beyond merge.
|
||||
|
||||
## Phase exit
|
||||
A phase is done when the **full conformance suite for that phase's requirements passes green**
|
||||
(R-TEST-03) and the exit criteria in [`../roadmap/phases.md`](../roadmap/phases.md) are recorded with
|
||||
links to the passing runs. Prod (Phase 7) additionally requires a successful **automated** prod DR
|
||||
drill.
|
||||
@@ -0,0 +1,56 @@
|
||||
# Agent Roles
|
||||
|
||||
Each role owns a workstream ([`../roadmap/workstreams.md`](../roadmap/workstreams.md)) and is
|
||||
accountable for that workstream's requirement IDs and acceptance gates. Roles are responsibilities,
|
||||
not people — one agent may hold several early on; split as work scales.
|
||||
|
||||
## Coordination roles
|
||||
|
||||
### Platform Architect (steward of this spec repo)
|
||||
- Guards the source-of-truth hierarchy ([`../AGENTS.md` §2](../AGENTS.md)); approves/records ADRs.
|
||||
- Resolves cross-workstream interface conflicts; keeps requirement traceability honest.
|
||||
- Does **not** write implementation code; owns specs, ADRs, roadmap, and this coordination layer.
|
||||
|
||||
### Integration/Release Lead
|
||||
- Owns the implementation repo structure, CI gates, and Flux bootstrap health.
|
||||
- Runs phase entry/exit reviews against the **automated conformance suite**; keeps the task board
|
||||
truthful.
|
||||
|
||||
### QA/Conformance Engineer (WS-TEST)
|
||||
- Owns the conformance testing capability ([14](../docs/14-testing-and-conformance.md),
|
||||
[ADR-0010](../decisions/adr-0010-conformance-testing.md)): the harness, the reproducibility
|
||||
contract, the requirement-coverage report, and the CI/phase gates.
|
||||
- Does **not** own writing every test in isolation — **each workstream authors the behavior tests
|
||||
for its own requirements**; WS-TEST provides the framework, patterns, fault-injection harness, and
|
||||
gate, and guards that tests assert real behavior and are never weakened to pass (R-TEST-05/06).
|
||||
- Accountable for: every `R-*` requirement having a passing, reproducible, tagged test.
|
||||
|
||||
## Implementation roles (one per workstream)
|
||||
|
||||
| Role | Workstream | Primary specs | Accountable for (gates) |
|
||||
|---|---|---|---|
|
||||
| **Infra/Cluster Engineer** | WS-INFRA | [02](../docs/02-environments-and-provisioning.md) | Reproducible node provisioning; k3s HA + Cilium; node rebuild w/o data loss |
|
||||
| **GitOps/Platform Engineer** | WS-GITOPS | [05](../docs/05-gitops-and-delivery.md), [13](../docs/13-service-catalog.md) | Reconcile-from-git; drift correction; ServiceDefinition renderer; CI |
|
||||
| **Network Engineer** | WS-NET | [03](../docs/03-networking-isolation-and-egress.md), [09](../docs/09-dns-and-ingress.md) | Default-deny; explicit allows; egress classes; VPN kill-switch; ingress/DNS |
|
||||
| **Storage Engineer** | WS-STORAGE | [04](../docs/04-storage-and-state.md) | Replicated Longhorn; bulk tier; backups; **tested restore/DR** |
|
||||
| **Security Engineer** | WS-SEC | [07](../docs/07-security-and-secrets.md), [08](../docs/08-tls-and-certificates.md) | SOPS secrets; trusted auto-TLS; least-privilege; auditability proofs |
|
||||
| **Observability Engineer** | WS-OBS | [10](../docs/10-observability-and-alerting.md) | Metrics/logs; email alerts; dashboards; platform-health alerts |
|
||||
| **Automation Engineer** | WS-UPDATE | [11](../docs/11-automated-updates.md) | Hands-off stable updates; health-gate + auto-rollback; pin |
|
||||
| **Virtualization Engineer** | WS-VM | [06](../docs/06-vm-workloads-kubevirt.md) | KubeVirt IaaS via same K8s surface; VM rebuild w/o data loss |
|
||||
| **Application Onboarding Engineer** | WS-APPS | [13](../docs/13-service-catalog.md) | Each service wired from its ServiceDefinition; deps correct; nothing undeclared |
|
||||
| **C2/UX Engineer** | WS-C2 | [12](../docs/12-command-center-ui.md) | Root C2 UI; live catalog + health; safe, audited actions |
|
||||
|
||||
## Standing expectations for every role
|
||||
- Trace all work to requirement IDs; reference specs/ADRs by ID (stable across file moves).
|
||||
- **Author an automated, reproducible behavior test** for every requirement you implement; "done" is
|
||||
a green, tagged test, not "merged" ([definition-of-done.md](definition-of-done.md),
|
||||
[14](../docs/14-testing-and-conformance.md)).
|
||||
- **Never modify the specifications** (`docs/`, requirements, accepted ADRs, roadmap) — they are
|
||||
human-owned (R-TEST-06). A spec you can't meet is an **escalation**, not an edit. Never weaken a
|
||||
test/requirement to get green.
|
||||
- Never re-litigate an accepted ADR; **draft + escalate** a superseding ADR for human approval
|
||||
([`../decisions/README.md`](../decisions/README.md)).
|
||||
- Negotiate interface changes before breaking another workstream's contract
|
||||
([coordination-protocol.md](coordination-protocol.md)).
|
||||
- Never build on prod hardware before Phase 7 is opened.
|
||||
- Never emit plaintext secrets.
|
||||
@@ -0,0 +1,100 @@
|
||||
# Task Board
|
||||
|
||||
The live who's-doing-what. Agents **claim** an item (set Owner + Status) before starting, respect
|
||||
`Blocked-by`, and mark `done` only after its **automated, reproducible behavior test passes green**
|
||||
([`definition-of-done.md`](definition-of-done.md), [`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md)).
|
||||
Seeded from [`../roadmap/phases.md`](../roadmap/phases.md) + [`../roadmap/workstreams.md`](../roadmap/workstreams.md).
|
||||
|
||||
> **This board is the one working surface agents may edit** (claim items, update status, add
|
||||
> items/escalations). The specs, ADRs, and roadmap are **not** editable by agents (R-TEST-06) —
|
||||
> raise spec problems here as an `OQ-*` escalation for the human. Every "**Prove …**" task below is
|
||||
> an automated conformance test (co-owned with **WS-TEST**), not a manual demo.
|
||||
|
||||
**Status values:** `todo` · `in-progress` · `blocked` · `review` · `done`
|
||||
**Add new items** at the bottom of the relevant phase; keep IDs stable.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 — Foundations
|
||||
| ID | Task | WS | Serves | Blocked-by | Owner | Status |
|
||||
|---|---|---|---|---|---|---|
|
||||
| T-0.1 | Create implementation repo structure (platform/apps/catalog/clusters) | WS-GITOPS | R-GITOPS-02 | — | — | todo |
|
||||
| T-0.2 | CI: lint + schema validation + build/dry-run gates | WS-GITOPS | R-GITOPS-01 | T-0.1 | — | todo |
|
||||
| T-0.3 | Author ServiceDefinition schema + renderer contract | WS-GITOPS | R-GITOPS-02 | T-0.1 | — | todo |
|
||||
| T-0.4 | Conformance harness: **Chainsaw** runner + reproducibility contract + JUnit + coverage report + CI merge gate | WS-TEST | R-TEST-01/02/03/04 | T-0.1 | — | todo |
|
||||
| T-0.5 | L0 **Kyverno** policy gate (default-deny present, no plaintext secrets, schema-valid ServiceDefinitions) | WS-TEST | R-TEST-01, R-NET-01, R-SEC-01 | T-0.4 | — | todo |
|
||||
|
||||
## Phase 1 — dev/test Cluster Bring-up
|
||||
| ID | Task | WS | Serves | Blocked-by | Owner | Status |
|
||||
|---|---|---|---|---|---|---|
|
||||
| T-1.1 | OpenTofu libvirt module for 3 node VMs (+ Longhorn data disk) | WS-INFRA | R-ENV-01 | — | — | todo |
|
||||
| T-1.2 | cloud-init baseline (users, SSH, sysctls, virt/Longhorn prereqs) | WS-INFRA | R-ENV-01 | T-1.1 | — | todo |
|
||||
| T-1.3 | Ansible: k3s HA install, Cilium CNI, disable flannel/kube-proxy/Traefik | WS-INFRA | R-PLAT-01 | T-1.2 | — | todo |
|
||||
| T-1.4 | Bootstrap Flux + SOPS age key | WS-GITOPS/WS-SEC | R-GITOPS-01, R-SEC-01 | T-1.3 | — | todo |
|
||||
| T-1.5 | Prove node destroy/rebuild from IaC (no data loss) | WS-INFRA | R-ENV-01, R-STATE-01 | T-1.4 | — | todo |
|
||||
|
||||
## Phase 2 — Platform Core (net/storage/TLS/DNS/ingress)
|
||||
| ID | Task | WS | Serves | Blocked-by | Owner | Status |
|
||||
|---|---|---|---|---|---|---|
|
||||
| T-2.1 | Longhorn (3-replica StorageClass) + bulk-tier plan (OQ-2) | WS-STORAGE | R-STATE-02/04 | T-1.4 | — | todo |
|
||||
| T-2.2 | Default-deny network baseline (per-namespace) + Hubble | WS-NET | R-NET-01/03 | T-1.3 | — | todo |
|
||||
| T-2.3 | Render `dependencies` → CiliumNetworkPolicy from ServiceDefinition | WS-NET/WS-GITOPS | R-NET-02 | T-0.3, T-2.2 | — | todo |
|
||||
| T-2.4 | cert-manager + Let's Encrypt DNS-01 wildcard `*.home.keithjackson.dev` | WS-SEC | R-TLS-01/02/03 | T-1.4 | — | todo |
|
||||
| T-2.5 | ExternalDNS + Gateway API listeners + split-horizon (PiHole) foundation | WS-NET | R-DNS-01/02/03 | T-2.4 | — | todo |
|
||||
| T-2.6 | Prove: default-deny (DNS-only probe), PVC survives rebuild, trusted cert, auto-DNS | WS-NET/STORAGE/SEC | R-NET/STATE/TLS/DNS | T-2.1..2.5 | — | todo |
|
||||
| T-2.7 | Deploy **Kyverno in-cluster as admission enforcer** (same policies as the CI L0 gate) | WS-TEST/WS-SEC | R-SEC-02, R-NET-01, R-SEC-01 | T-1.4, T-0.5 | — | todo |
|
||||
|
||||
## Phase 3 — Egress, VPN & Multi-WAN
|
||||
| ID | Task | WS | Serves | Blocked-by | Owner | Status |
|
||||
|---|---|---|---|---|---|---|
|
||||
| T-3.1 | Egress classes; Cilium Egress Gateway wiring | WS-NET | R-NET-04/05 | T-2.2 | — | todo |
|
||||
| T-3.2 | VPN egress (Gluetun sidecar) + kill-switch (ADR-0008) | WS-NET | R-NET-05/06 | T-3.1, OQ-3 | — | todo |
|
||||
| T-3.3 | Edge-router multi-WAN split for `internet` class (OQ-1) | WS-NET | R-NET-04 | OQ-1 | — | todo |
|
||||
| T-3.0 | L3 fault-injection harness — **in-cluster Chainsaw Job** (CI-triggered, scoped test SA) using `script`/`try`/`finally` (force tunnel down, kill node, inject bad image, trigger restore) | WS-TEST | R-TEST-05, R-SEC-02 | T-0.4 | — | todo |
|
||||
| T-3.4 | Prove: VPN IP egress, kill-switch fail-closed, egress-class flip via git | WS-NET | R-NET-05/06/07 | T-3.0, T-3.2 | — | todo |
|
||||
|
||||
## Phase 4 — Observability, Backup/DR & Updates
|
||||
| ID | Task | WS | Serves | Blocked-by | Owner | Status |
|
||||
|---|---|---|---|---|---|---|
|
||||
| T-4.1 | kube-prometheus-stack + Loki + Alloy + Alertmanager(email) | WS-OBS | R-OBS-01/02/03 | T-2.1 | — | todo |
|
||||
| T-4.2 | Platform-health alerts (node/Longhorn/Flux/cert/VPN/backup) | WS-OBS | R-OBS-04 | T-4.1 | — | todo |
|
||||
| T-4.3 | Velero + Longhorn off-cluster backup schedule | WS-STORAGE | R-STATE-03 | T-2.1, OQ-2 | — | todo |
|
||||
| T-4.4 | DR drill: restore namespace + volumes from off-cluster backup | WS-STORAGE | R-STATE-03 | T-4.3 | — | todo |
|
||||
| T-4.5 | Flux image-automation + Renovate + per-service update policy | WS-UPDATE | R-UPD-01/02/03 | T-0.3, T-4.1 | — | todo |
|
||||
| T-4.6 | Prove: auto stable update on schedule, bad-update auto-rollback+alert, pin holds | WS-UPDATE | R-UPD-* | T-4.5 | — | todo |
|
||||
|
||||
## Phase 5 — IaaS Proof (KubeVirt)
|
||||
| ID | Task | WS | Serves | Blocked-by | Owner | Status |
|
||||
|---|---|---|---|---|---|---|
|
||||
| T-5.1 | Install KubeVirt + CDI; Longhorn-backed VM disks | WS-VM | R-PLAT-03 | T-2.1, T-2.2 | — | todo |
|
||||
| T-5.2 | `kind: vm` ServiceDefinition path; run one representative VM | WS-VM/WS-GITOPS | R-PLAT-03/04 | T-5.1, T-0.3 | — | todo |
|
||||
| T-5.3 | Prove: VM lifecycle via kubectl/virtctl, rebuild w/o data loss, default-deny+egress, guest auto-update | WS-VM | R-PLAT-03, R-NET-*, R-UPD-* | T-5.2 | — | todo |
|
||||
|
||||
## Phase 6 — Service Onboarding + C2
|
||||
| ID | Task | WS | Serves | Blocked-by | Owner | Status |
|
||||
|---|---|---|---|---|---|---|
|
||||
| T-6.1 | Onboard **pihole** (DNS underpinning) | WS-APPS | R-DNS-03 | T-2.5 | — | todo |
|
||||
| T-6.2 | Onboard **plex** (bulk tier + public exposure) | WS-APPS | R-STATE-04 | T-2.6, T-6.1 | — | todo |
|
||||
| T-6.3 | Onboard **qbittorrent, jackett** (VPN + kill-switch) | WS-APPS | R-NET-05/06 | T-3.4 | — | todo |
|
||||
| T-6.4 | Onboard **sonarr, radarr** (VPN + arr↔downloader allows) | WS-APPS | R-NET-02/05 | T-6.3 | — | todo |
|
||||
| T-6.5 | Onboard **seerr** (allows to arr + plex) | WS-APPS | R-NET-02 | T-6.2, T-6.4 | — | todo |
|
||||
| T-6.6 | Onboard **nextcloud, gitea** (DR-critical + backups) | WS-APPS | R-STATE-03 | T-4.4 | — | todo |
|
||||
| T-6.7 | Onboard **minecraft, unifi, semaphore, kasm** | WS-APPS | app reqs | T-2.6 | — | todo |
|
||||
| T-6.8 | C2 UI: Homepage root + Grafana + Headlamp + Flux status + auth | WS-C2 | R-C2-01/02 | T-4.2, T-6.* | — | todo |
|
||||
|
||||
## Phase 7 — Production Readiness (CLOSED until operator opens it)
|
||||
| ID | Task | WS | Serves | Blocked-by | Owner | Status |
|
||||
|---|---|---|---|---|---|---|
|
||||
| T-7.1 | Bare-metal provisioning checklist (design-only) | WS-INFRA | R-ENV-03 | — | — | todo |
|
||||
| T-7.2 | prod overlay (no base edits) + cutover plan | WS-INFRA/WS-GITOPS | R-ENV-02 | Phases 1–6 done | — | blocked |
|
||||
| T-7.3 | Full prod DR drill | WS-STORAGE | R-STATE-03 | T-7.2 | — | blocked |
|
||||
|
||||
---
|
||||
|
||||
## Open questions (resolve to unblock)
|
||||
| OQ | Question | Blocks | Owner |
|
||||
|---|---|---|---|
|
||||
| OQ-1 | WAN uplink count + edge-router product | T-3.3 | — |
|
||||
| OQ-2 | Bulk-media home + backup posture | T-2.1, T-4.3 | — |
|
||||
| OQ-3 | VPN provider/protocol + concurrent-tunnel limits | T-3.2 | — |
|
||||
| OQ-4 | DNS provider API (Cloudflare) token for DNS-01 + ExternalDNS | T-2.4, T-2.5 | — |
|
||||
@@ -0,0 +1,55 @@
|
||||
# Architecture Decision Records (ADRs)
|
||||
|
||||
ADRs are an **append-only ledger** of the significant technology and pattern choices for this
|
||||
platform. Each records the context, the decision, the rejected alternatives, and the requirements it
|
||||
serves. **Do not edit an accepted decision in place** — supersede it with a new ADR.
|
||||
|
||||
> **ADRs are specifications and are immutable to AI agents** (R-TEST-06). Agents may *draft* a
|
||||
> proposed ADR and escalate it, but only the **human operator/architect** approves/accepts or edits
|
||||
> ADRs. See [`../AGENTS.md` §3 and §11](../AGENTS.md).
|
||||
|
||||
## Process (read before proposing a change)
|
||||
|
||||
1. To change a decision, create a **new** ADR with the next number and set `Status: Proposed`,
|
||||
`Supersedes: ADR-XXXX`. (Agents draft + escalate; the human approves.)
|
||||
2. State what new information invalidates the old decision.
|
||||
3. On approval, set the new ADR `Accepted` and the old one `Superseded by: ADR-YYYY`.
|
||||
4. Never silently re-interpret an accepted ADR (see [`../AGENTS.md` §3](../AGENTS.md)).
|
||||
|
||||
## Template
|
||||
|
||||
```markdown
|
||||
# ADR-XXXX — <title>
|
||||
- Status: Proposed | Accepted | Superseded by ADR-YYYY
|
||||
- Date: <YYYY-MM-DD>
|
||||
- Serves: <requirement IDs, e.g. R-NET-01, R-NET-06>
|
||||
- Supersedes: <ADR-ZZZZ or —>
|
||||
|
||||
## Context
|
||||
<forces, constraints, why a decision is needed>
|
||||
|
||||
## Decision
|
||||
<what we chose, stated decisively>
|
||||
|
||||
## Alternatives considered
|
||||
<options and why rejected>
|
||||
|
||||
## Consequences
|
||||
<positive, negative, follow-ups, risks>
|
||||
```
|
||||
|
||||
## Index
|
||||
|
||||
| ADR | Title | Status | Serves |
|
||||
|---|---|---|---|
|
||||
| [0001](adr-0001-kubernetes-distribution-k3s.md) | Kubernetes distribution: k3s | Accepted | R-PLAT-01 |
|
||||
| [0002](adr-0002-cni-cilium.md) | CNI & network policy & ingress: Cilium + Gateway API | Accepted | R-NET-01..07, R-DNS-01 |
|
||||
| [0003](adr-0003-storage-longhorn.md) | Distributed storage: Longhorn (+ bulk tier, Velero) | Accepted | R-STATE-01..04 |
|
||||
| [0004](adr-0004-gitops-flux.md) | GitOps engine: Flux CD | Accepted | R-GITOPS-01..03 |
|
||||
| [0005](adr-0005-vm-workloads-kubevirt.md) | VM workloads: KubeVirt + CDI | Accepted | R-PLAT-03 |
|
||||
| [0006](adr-0006-tls-cert-manager-letsencrypt.md) | TLS: cert-manager + Let's Encrypt (DNS-01) | Accepted | R-TLS-01..03 |
|
||||
| [0007](adr-0007-secrets-sops-age.md) | Secrets: SOPS + age | Accepted | R-SEC-01 |
|
||||
| [0008](adr-0008-vpn-egress-model.md) | VPN egress model: Gluetun sidecar + Cilium Egress Gateway | Accepted | R-NET-05/06 |
|
||||
| [0009](adr-0009-automated-image-updates.md) | Automated updates: Flux image automation + Renovate | Accepted | R-UPD-01..03 |
|
||||
| [0010](adr-0010-conformance-testing.md) | Conformance testing: automated, reproducible, behavior-based | Accepted | R-TEST-01..06 |
|
||||
| [0011](adr-0011-test-tooling-kyverno-chainsaw.md) | Conformance test tooling: Kyverno + Chainsaw | Accepted | R-TEST-01..05 |
|
||||
@@ -0,0 +1,32 @@
|
||||
# ADR-0001 — Kubernetes distribution: k3s
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-PLAT-01, R-PLAT-02, R-ENV-01/02/03
|
||||
- Supersedes: —
|
||||
|
||||
## Context
|
||||
The platform runs on 3 small nodes (dev: KVMs; prod: mini-PCs) and must offer a full Kubernetes API
|
||||
for both containers and VMs. It must be lightweight enough for constrained hardware yet
|
||||
production-capable, HA, and identical across dev/test and prod.
|
||||
|
||||
## Decision
|
||||
Use **k3s** as the Kubernetes distribution, deployed as a 3-node HA cluster with **embedded etcd**
|
||||
(all three nodes are servers). Bundled components that conflict with our chosen stack are disabled at
|
||||
install time: flannel (→ Cilium), kube-proxy (→ Cilium kube-proxy replacement), Traefik (→ Gateway
|
||||
API via Cilium), and servicelb where a different LB strategy is selected. Nodes are provisioned by
|
||||
OpenTofu (libvirt, dev) / bare-metal imaging (prod) + cloud-init, and joined/configured by Ansible.
|
||||
|
||||
## Alternatives considered
|
||||
- **Full kubeadm/vanilla Kubernetes** — heavier operationally for 3 small nodes; k3s gives the same
|
||||
API with far less overhead.
|
||||
- **k0s / MicroK8s / Talos** — all viable. Talos is attractive (immutable, API-driven) and may be
|
||||
revisited via a superseding ADR for prod hardening; k3s chosen now for its ubiquity, small
|
||||
footprint, single-binary simplicity, strong community, and the operator's existing familiarity.
|
||||
- **Managed cloud Kubernetes** — out of scope; this is on-prem home hardware.
|
||||
|
||||
## Consequences
|
||||
- Lightweight, HA, uniform dev↔prod topology (R-ENV-02).
|
||||
- Must explicitly disable bundled CNI/ingress/proxy to make room for Cilium + Gateway API (ADR-0002)
|
||||
— captured in the Ansible bootstrap.
|
||||
- Embedded etcd across 3 nodes tolerates single-node loss; do not reduce below 3 servers in prod.
|
||||
- Revisit Talos for prod if immutability/hardening becomes a priority.
|
||||
@@ -0,0 +1,37 @@
|
||||
# ADR-0002 — CNI, network policy & ingress: Cilium + Gateway API
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-NET-01, R-NET-02, R-NET-03, R-NET-04, R-NET-05, R-NET-06, R-NET-07, R-DNS-01, R-SEC-02
|
||||
- Supersedes: —
|
||||
|
||||
## Context
|
||||
The platform requires: default-deny east-west with explicit port-level allows (firewall-like),
|
||||
network isolation/segmentation, multi-WAN/egress routing including per-service VPN egress, and
|
||||
north-south ingress with TLS. These are the most safety-critical requirements. A single, coherent
|
||||
networking layer that does policy, egress steering, observability, and ingress reduces moving parts.
|
||||
|
||||
## Decision
|
||||
Use **Cilium** as the CNI (replacing k3s' flannel), with **kube-proxy replacement**, and adopt:
|
||||
- **CiliumNetworkPolicy / NetworkPolicy** for default-deny + explicit L3/L4 (optionally L7) allows.
|
||||
- **Cilium Egress Gateway** to steer selected pods' egress through chosen nodes/identities/SNAT IPs
|
||||
— the mechanism for egress-class routing and part of the VPN model (with ADR-0008).
|
||||
- **Hubble** for flow observability — the audit/troubleshooting tool for isolation and egress.
|
||||
- **Gateway API implemented by Cilium** for north-south ingress and TLS termination (no separate
|
||||
ingress controller).
|
||||
|
||||
## Alternatives considered
|
||||
- **Calico** — strong policy engine; lacks Cilium's integrated egress gateway + Hubble + Gateway API
|
||||
in one stack.
|
||||
- **Flannel + separate ingress-nginx/Traefik + separate egress solution** — more components, no
|
||||
identity-aware policy or built-in flow visibility.
|
||||
- **Service mesh (Istio/Linkerd) for policy/mTLS** — heavier; not needed at launch. mTLS can be
|
||||
added later via Cilium/mesh as a future ADR ([08](../docs/08-tls-and-certificates.md)).
|
||||
|
||||
## Consequences
|
||||
- One networking layer covers policy, egress steering, observability, and ingress — fewer components,
|
||||
coherent model, VMs (KubeVirt) inherit the same policy fabric.
|
||||
- Requires disabling k3s bundled flannel/kube-proxy/Traefik (see ADR-0001).
|
||||
- Egress Gateway + VPN model must be validated end-to-end including kill-switch (ADR-0008, R-NET-06).
|
||||
- Physical multi-WAN (dual ISP) remains partly an edge-router responsibility (OQ-1); the cluster
|
||||
decides egress *class*, the edge router decides physical WAN for the internet class.
|
||||
- Gateway LB/VIP mechanism (Cilium L2 announcement vs BGP) to be pinned during networking bring-up.
|
||||
@@ -0,0 +1,38 @@
|
||||
# ADR-0003 — Distributed storage: Longhorn (+ bulk tier + Velero)
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-STATE-01, R-STATE-02, R-STATE-03, R-STATE-04
|
||||
- Supersedes: —
|
||||
|
||||
## Context
|
||||
Three nodes with **local disks** and no SAN must provide durable, replicated persistent storage so
|
||||
any workload can be destroyed and rebuilt without data loss, tolerate a single node failure, and
|
||||
support off-cluster backup/DR. Plex's multi-TB media library makes uniform 3× replication
|
||||
impractical, requiring a distinct bulk tier.
|
||||
|
||||
## Decision
|
||||
- Use **Longhorn** for distributed replicated block storage across the 3 nodes; default StorageClass
|
||||
is **3 replicas**. This is the durable home for app/config state and KubeVirt VM disks (via CDI).
|
||||
- Define a separate **bulk media tier** for large, non-replicated data (Plex library, download
|
||||
staging). Its concrete backing (NAS/NFS vs. dedicated large disk vs. mixed) is decided under OQ-2;
|
||||
**Plex's metadata DB stays on 3-replica Longhorn** regardless.
|
||||
- Use **Velero** (with Longhorn backups) to push volume + resource-state backups to an **off-cluster**
|
||||
S3-compatible/NFS target on a schedule, with a **tested restore** path (DR drill).
|
||||
|
||||
## Alternatives considered
|
||||
- **Rook/Ceph** — more powerful but far heavier/operationally complex for 3 small nodes.
|
||||
- **OpenEBS (Mayastor/other)** — viable; Longhorn chosen for simplicity, strong UI, snapshot/backup
|
||||
ergonomics, and fit with small clusters.
|
||||
- **Uniform 3× replication of everything including media** — rejected: capacity-prohibitive on
|
||||
mini-PC local disks (R-STATE-04).
|
||||
- **hostPath/local-only PVs for stateful apps** — rejected: node-bound, no replication, breaks the
|
||||
rebuild guarantee.
|
||||
|
||||
## Consequences
|
||||
- Single-node loss keeps replicated data available; workloads rebuild by re-attaching PVCs
|
||||
(R-STATE-01/02).
|
||||
- A two-tier storage story must be clearly documented so implementers don't accidentally place bulk
|
||||
media on the replicated tier or app DBs on the bulk tier ([04](../docs/04-storage-and-state.md)).
|
||||
- Off-cluster backup target + its credentials/resilience are prerequisites (OQ-2); DR drill is part
|
||||
of acceptance.
|
||||
- Longhorn's own capacity/health must be monitored and alerted ([10](../docs/10-observability-and-alerting.md)).
|
||||
@@ -0,0 +1,35 @@
|
||||
# ADR-0004 — GitOps engine: Flux CD
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-GITOPS-01, R-GITOPS-02, R-GITOPS-03, R-UPD-02, R-SEC-01, R-ENV-02
|
||||
- Supersedes: —
|
||||
|
||||
## Context
|
||||
All desired state must be reconciled from git (pull-based), with drift correction, environment
|
||||
overlays (dev↔prod), encrypted secrets, and — critically — **hands-off image updates** driven by
|
||||
declared policy (R-UPD-02). The GitOps engine choice is tightly coupled to the update mechanism and
|
||||
the secrets model.
|
||||
|
||||
## Decision
|
||||
Use **Flux CD** for reconciliation, chosen decisively because it *natively* provides the two
|
||||
capabilities that matter most for this platform's hard requirements:
|
||||
- **image-update-automation** (image-reflector + image-automation controllers) for policy-driven,
|
||||
no-human image updates (R-UPD-02, ADR-0009).
|
||||
- **native SOPS decryption** for encrypted-in-git secrets (R-SEC-01, ADR-0007).
|
||||
Flux Kustomize/Helm controllers handle overlays (dev↔prod, R-ENV-02) and chart-based platform
|
||||
components.
|
||||
|
||||
## Alternatives considered
|
||||
- **Argo CD** — excellent UI and ApplicationSets; but image auto-update (Argo Image Updater) and
|
||||
secret decryption are add-ons rather than first-class, and the platform gets its UI from the C2
|
||||
composition ([12](../docs/12-command-center-ui.md)) rather than from the GitOps tool. Reasonable
|
||||
alternative; Flux wins on native image-automation + SOPS fit.
|
||||
- **Rancher Fleet / Kustomize-only + CI push** — weaker fit for the automated-update and drift
|
||||
requirements.
|
||||
|
||||
## Consequences
|
||||
- Native, first-class path to R-UPD-02 (no bolt-on for automated updates) and R-SEC-01 (SOPS).
|
||||
- GitOps status/health is surfaced in the C2 UI via Headlamp/Flux views ([12](../docs/12-command-center-ui.md)),
|
||||
not via a bundled GitOps GUI.
|
||||
- Reconciliation failures/drift are alerting conditions (R-OBS-04).
|
||||
- If a rich built-in GitOps GUI later becomes a priority, revisit via superseding ADR (Argo).
|
||||
@@ -0,0 +1,36 @@
|
||||
# ADR-0005 — VM workloads: KubeVirt + CDI
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-PLAT-03, R-PLAT-04, R-STATE-01
|
||||
- Supersedes: —
|
||||
|
||||
## Context
|
||||
Not everything fits in a container. The platform must run VMs, but the operator must manage them
|
||||
through the **same** Kubernetes API/CLI/YAML/GitOps as containers, with the same network isolation,
|
||||
egress classes, storage durability, and observability — no separate hypervisor management plane for
|
||||
application workloads.
|
||||
|
||||
## Decision
|
||||
Use **KubeVirt** to run application VMs as Kubernetes custom resources (`VirtualMachine` /
|
||||
`VirtualMachineInstance`), with **CDI (Containerized Data Importer)** to provision VM disks onto
|
||||
**Longhorn** PVCs. VMs run on the same Cilium network (inheriting default-deny + egress classes) and
|
||||
are managed by Flux like any other workload. `virtctl` provides console/lifecycle helpers alongside
|
||||
`kubectl`.
|
||||
|
||||
## Alternatives considered
|
||||
- **Proxmox / standalone hypervisor + separate IaC** — a second management plane and tooling; breaks
|
||||
the "same Kubernetes surface" requirement (R-PLAT-03). (The operator's prior environment used
|
||||
Proxmox; this platform deliberately unifies on Kubernetes instead.)
|
||||
- **Harvester (SUSE)** — HCI built on KubeVirt; heavier and opinionated about the whole node OS; k3s
|
||||
+ KubeVirt keeps one lightweight cluster for both containers and VMs.
|
||||
- **Containers only** — rejected: the requirement explicitly needs a VM escape hatch.
|
||||
|
||||
## Consequences
|
||||
- One control plane, one GitOps flow, one network/storage/observability model for containers *and*
|
||||
VMs.
|
||||
- Nodes must expose virtualization extensions (nested virt in dev KVMs; native on prod) — a node
|
||||
baseline prerequisite ([02](../docs/02-environments-and-provisioning.md)).
|
||||
- VMs are heavier than pods; capacity planning on the 3-node cluster must account for them.
|
||||
- At least one representative VM workload must be proven in dev/test so the IaaS path is real
|
||||
([06](../docs/06-vm-workloads-kubevirt.md)).
|
||||
- VM guest updates use in-guest automation (R-UPD-*), distinct from container image tag bumps.
|
||||
@@ -0,0 +1,33 @@
|
||||
# ADR-0006 — TLS: cert-manager + Let's Encrypt (DNS-01)
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-TLS-01, R-TLS-02, R-TLS-03, R-DNS-01
|
||||
- Supersedes: —
|
||||
|
||||
## Context
|
||||
Every service needs a **trusted, non-self-signed** certificate, auto issued/renewed/revoked,
|
||||
including **internal-only** services (which must not be forced public just to validate a cert).
|
||||
Wildcard coverage of `*.home.keithjackson.dev` would make onboarding certificate-free.
|
||||
|
||||
## Decision
|
||||
Use **cert-manager** with **Let's Encrypt** (a publicly trusted, free, automatable CA) via the
|
||||
**ACME DNS-01** challenge against the `keithjackson.dev` DNS zone (Cloudflare or equivalent, OQ-4).
|
||||
DNS-01 validates domain control without exposing the service and enables **wildcard** certs.
|
||||
Default: a **wildcard `*.home.keithjackson.dev`** certificate terminated at the Gateway ([09](../docs/09-dns-and-ingress.md)),
|
||||
with per-service certificates available where isolation warrants. Provide both `letsencrypt-staging`
|
||||
(bring-up) and `letsencrypt-prod` ClusterIssuers.
|
||||
|
||||
## Alternatives considered
|
||||
- **HTTP-01 challenge** — rejected: requires public exposure of each service; fails R-TLS-03 for
|
||||
internal-only services and can't do wildcards.
|
||||
- **Self-signed / private CA** — rejected: explicitly disallowed (R-TLS-01, "not self-signed").
|
||||
- **Paid CA** — unnecessary; Let's Encrypt is trusted and free.
|
||||
- **ZeroSSL/other ACME CA** — viable fallback; Let's Encrypt chosen as default.
|
||||
|
||||
## Consequences
|
||||
- Internal services get real, trusted certs; wildcard makes onboarding certificate-free (default).
|
||||
- Requires a DNS-provider API token scoped for DNS-01, stored SOPS-encrypted (OQ-4, ADR-0007).
|
||||
- Certificate expiry/issuance failures are alert conditions (R-OBS-04).
|
||||
- Internal clients trust the certs with no custom-CA distribution; split-horizon DNS serves the same
|
||||
cert internally and externally.
|
||||
- Optional future mTLS between workloads is a separate concern (Cilium/mesh), not covered here.
|
||||
@@ -0,0 +1,31 @@
|
||||
# ADR-0007 — Secrets: SOPS + age
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-SEC-01, R-GITOPS-01
|
||||
- Supersedes: —
|
||||
|
||||
## Context
|
||||
Secrets (VPN keys, DNS API token, SMTP creds, service passwords, backup-target creds) must live in
|
||||
git for GitOps but MUST NOT be plaintext. The engine is Flux (ADR-0004), which decrypts SOPS
|
||||
natively.
|
||||
|
||||
## Decision
|
||||
Encrypt secrets in git with **SOPS** using **age** recipients; **Flux decrypts in-cluster** at
|
||||
reconciliation via an age private key bootstrapped as a Kubernetes Secret. Plaintext secrets never
|
||||
enter git or logs. Secret rotation = re-encrypt + commit; Flux rolls it out (restart triggered by
|
||||
checksum annotation where needed).
|
||||
|
||||
## Alternatives considered
|
||||
- **Sealed Secrets (Bitnami)** — solid; one-way encryption to a cluster key. SOPS chosen for native
|
||||
Flux integration, multi-recipient/age simplicity, and readable, reviewable encrypted diffs.
|
||||
- **External Secrets Operator + external vault (Vault/cloud SM)** — stronger for large orgs but adds
|
||||
an external dependency and operational weight not justified for a single-operator home platform.
|
||||
May be revisited via superseding ADR if a real external secret store is introduced.
|
||||
- **Plaintext + private repo** — rejected outright (R-SEC-01).
|
||||
|
||||
## Consequences
|
||||
- Secrets are versioned, encrypted, and reconciled with the rest of GitOps state.
|
||||
- The **age private key bootstrap** is a sanctioned out-of-band step (documented; one of the few
|
||||
non-GitOps actions, [07](../docs/07-security-and-secrets.md)).
|
||||
- Key custody/rotation for the age key itself must be documented; losing it blocks decryption.
|
||||
- No secret material may appear in agent output, PRs, or logs ([AGENTS.md §5](../AGENTS.md)).
|
||||
@@ -0,0 +1,42 @@
|
||||
# ADR-0008 — VPN egress model: Gluetun sidecar (default) + Cilium Egress Gateway
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-NET-05, R-NET-06, R-NET-07
|
||||
- Supersedes: —
|
||||
|
||||
## Context
|
||||
A subset of services (sonarr, radarr, jackett, qbittorrent) must egress **only** through a commercial
|
||||
VPN, with a **kill-switch** guaranteeing no leak to the plain internet if the tunnel drops. Egress
|
||||
class must be a declarative service property (`egress: vpn`) that doesn't force a workload rewrite.
|
||||
|
||||
## Decision
|
||||
Adopt a **two-pattern** model behind the single `egress: vpn` contract, with **Gluetun sidecar as the
|
||||
default** for the initial catalog:
|
||||
- **Default — Pattern A: Gluetun sidecar per pod.** A Gluetun container holds the WireGuard/OpenVPN
|
||||
tunnel; the app shares its network namespace. Gluetun's built-in **firewall kill-switch** blocks
|
||||
all non-tunnel egress by default — satisfying R-NET-06 intrinsically and per-service.
|
||||
- **Available — Pattern B: central VPN egress gateway + Cilium Egress Gateway.** One (HA) gateway
|
||||
holds the tunnel; Cilium Egress Gateway steers `vpn`-labeled pods through it, with policy denying
|
||||
any fallback to the internet path. Use when tunnel count must be minimized.
|
||||
|
||||
The service-facing contract is identical for both: declare `egress: vpn`; the platform wires the
|
||||
chosen pattern. Kill-switch behavior is a hard acceptance gate for both.
|
||||
|
||||
## Alternatives considered
|
||||
- **Central gateway only** — fewer tunnels but a shared failure domain and more complex kill-switch
|
||||
guarantees across the fabric; kept as Pattern B, not the default.
|
||||
- **Node-level policy routing / per-node VPN** — couples egress to node identity, harder to reason
|
||||
about per-service; rejected as the primary model.
|
||||
- **App-native VPN config** — rejected: not declarative, not uniform, easy to misconfigure and leak.
|
||||
|
||||
## Consequences
|
||||
- Per-service isolation and an intrinsic kill-switch out of the box (Pattern A).
|
||||
- More sidecars/tunnels at scale (Pattern A) — Pattern B is the escape hatch if the VPN provider
|
||||
limits concurrent tunnels/egress IPs (OQ-3).
|
||||
- Kill-switch must be **fault-tested** (tunnel forced down ⇒ zero internet egress) as a standing
|
||||
acceptance gate, not a one-time check ([03](../docs/03-networking-isolation-and-egress.md), R-NET-06).
|
||||
- VPN credentials are SOPS-encrypted (ADR-0007). WireGuard preferred over OpenVPN where supported.
|
||||
|
||||
## Open questions
|
||||
- **OQ-3** — VPN provider/protocol and concurrent-tunnel/egress-IP limits (drives Pattern A vs B and
|
||||
gateway HA).
|
||||
@@ -0,0 +1,43 @@
|
||||
# ADR-0009 — Automated updates: Flux image automation + Renovate
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-UPD-01, R-UPD-02, R-UPD-03
|
||||
- Supersedes: —
|
||||
|
||||
## Context
|
||||
Services must advance to the **latest stable** version **on a schedule** with **no human, no AI, and
|
||||
no hand-authored GitOps change** (R-UPD-02). This must coexist with GitOps (git = source of truth)
|
||||
and must be safe (stable-only, health-gated, reversible).
|
||||
|
||||
## Decision
|
||||
Split by artifact type:
|
||||
- **Container image tags → Flux image-update-automation.** Per-service **ImagePolicy** declares the
|
||||
acceptable range/channel (semver range or stable tag regex). Flux's image-reflector scans the
|
||||
registry on an interval (the schedule) and image-automation advances the running version. The
|
||||
policy is the human-authored intent; the version advance is machine-performed — no human/AI edits
|
||||
a tag (R-UPD-02).
|
||||
- **Helm charts / manifests → Renovate.** Renovate opens version-bump PRs; allowed update classes
|
||||
**auto-merge** on green CI. Major/breaking bumps may require review per policy.
|
||||
- **VM guests → in-guest auto-update** (unattended-upgrades/dnf-automatic) declared in the VM's
|
||||
cloud-init ([06](../docs/06-vm-workloads-kubevirt.md)).
|
||||
|
||||
**Default write-back behavior:** image-automation commits the resolved version back to git (machine-
|
||||
authored) so history is auditable; a **per-service escape hatch** allows digest-tracking of a mutable
|
||||
stable tag for zero-commit updates where the operator prefers it. Per-service `update` policy
|
||||
(channel, range, cadence, pin, autoRollback) lives in the ServiceDefinition ([13](../docs/13-service-catalog.md)).
|
||||
|
||||
## Alternatives considered
|
||||
- **Argo CD Image Updater** — bolt-on to Argo; we chose Flux (ADR-0004) whose image automation is
|
||||
first-class.
|
||||
- **`latest`/`stable` mutable tag + scheduled rollout restart only** — simplest but loses version
|
||||
visibility and semver guardrails; offered only as the per-service digest-tracking escape hatch.
|
||||
- **Human/AI-authored bumps** — explicitly prohibited by R-UPD-02.
|
||||
|
||||
## Consequences
|
||||
- Hands-off updates satisfied via declared policy, not manual/AI edits (R-UPD-02).
|
||||
- Safety rails required and are acceptance gates: **stable-only**, **health-gated rollout**,
|
||||
**auto-rollback** on failed health, **pin** escape hatch, and **alerting** on update failure
|
||||
([10](../docs/10-observability-and-alerting.md), [11](../docs/11-automated-updates.md)).
|
||||
- Registry rate-limits/credentials for cadence scanning must be handled.
|
||||
- Machine write-back commits will appear in history — an intended, auditable artifact, not a
|
||||
violation of "no hand-authored change."
|
||||
@@ -0,0 +1,48 @@
|
||||
# ADR-0010 — Conformance testing: automated, reproducible, behavior-based
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04, R-TEST-05, R-TEST-06
|
||||
- Supersedes: —
|
||||
|
||||
## Context
|
||||
Acceptance must be objective and repeatable, not a human eyeballing a demo. Every requirement
|
||||
(`R-*`) must be verified by an automated behavior test that any machine can run and reproduce, that
|
||||
gates merges and phase exits, and that traces back to the requirement it proves. Safety-critical
|
||||
behaviors (default-deny, VPN kill-switch, data survival, DR, auto-rollback) demand fault-injection,
|
||||
not happy-path checks. Additionally, agents must not weaken the target: specs are immutable to
|
||||
agents; tests encode spec intent.
|
||||
|
||||
## Decision
|
||||
Adopt an **automated, reproducible, behavior-based conformance suite** as the sole definition of
|
||||
acceptance, structured in layers (L0 static/policy, L1 rendering, L2 behavioral, L3 fault-injection,
|
||||
L4 aggregate suite), specified in [`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md).
|
||||
The suite is **requirement-tagged**, **single-command**, **hermetic**, **deterministic**, and emits
|
||||
**JUnit + a requirement-coverage report**. It gates:
|
||||
- **merges** — the subset for the touched requirements must pass;
|
||||
- **phase exits** — the phase's full suite must pass.
|
||||
|
||||
Tooling: the *approach* is fixed; the concrete mix is **policy-as-code** for L0 (e.g. Kyverno /
|
||||
Conftest-OPA), a **declarative Kubernetes test runner** for L1/L2 (e.g. Chainsaw or kuttl), and a
|
||||
**small programmatic harness** (e.g. Ginkgo/Gomega or a scripted runner) for L3 fault-injection that
|
||||
needs orchestration. The final tool selection is the one remaining open item and may be pinned by a
|
||||
short follow-up (still human-approved), but no choice may relax the properties above.
|
||||
|
||||
## Alternatives considered
|
||||
- **Manual demonstrations / checklists** — rejected: not reproducible, not gating, not traceable
|
||||
(violates R-TEST-01..04).
|
||||
- **Unit tests of manifests only (L0/L1)** — necessary but insufficient: a policy object existing
|
||||
does not prove enforcement; behavior (L2) and fault-injection (L3) are required (R-TEST-05).
|
||||
- **A single heavyweight framework for everything** — rejected: L3 fault-injection needs
|
||||
orchestration a declarative runner can't express, while L0/L1/L2 are cleaner declaratively. A
|
||||
layered mix fits better.
|
||||
- **Letting agents adjust tests/requirements to pass** — explicitly prohibited (R-TEST-06); a
|
||||
failing spec is an escalation, not a test edit.
|
||||
|
||||
## Consequences
|
||||
- "Done" becomes objective and machine-checkable; progress = green, requirement-tagged suite.
|
||||
- Every requirement needs a test; uncovered requirements are visible and blocking (R-TEST-04).
|
||||
- Establishing L3 fault-injection (kill-switch, node-kill, bad-update, DR) is real work and is
|
||||
first-class in the roadmap (WS-TEST), not an afterthought.
|
||||
- CI must have cluster access (or spin an ephemeral dev/test cluster) to run L2/L3.
|
||||
- The suite is versioned with the implementation and evolves only when the human-owned spec it
|
||||
encodes changes.
|
||||
@@ -0,0 +1,67 @@
|
||||
# ADR-0011 — Conformance test tooling: Kyverno + Chainsaw
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04, R-TEST-05
|
||||
- Refines: [ADR-0010](adr-0010-conformance-testing.md) (which fixed the *approach* and left the tool mix as the one open item)
|
||||
- Supersedes: —
|
||||
|
||||
## Context
|
||||
[ADR-0010](adr-0010-conformance-testing.md) fixed the conformance approach (automated, reproducible,
|
||||
behavior-based, layered, requirement-tagged, gating) and left the concrete tool selection as the sole
|
||||
open item. A choice is now made by the operator/architect.
|
||||
|
||||
## Decision
|
||||
Use **two tools from the Kyverno ecosystem**, covering all test layers:
|
||||
|
||||
- **Kyverno** — the **L0 static/policy gate** (policy-as-code) **and in-cluster admission enforcer**.
|
||||
The Kyverno CLI (`kyverno apply` / `kyverno test`, incl. `kyverno-json` for arbitrary YAML/JSON
|
||||
like ServiceDefinitions) validates manifests and rendered resources in CI against platform
|
||||
invariants: every namespace carries a default-deny baseline, no plaintext secrets, required
|
||||
labels/egress-class set, ServiceDefinition schema conformance, exposure/egress sanity. **The same
|
||||
policies also run in-cluster as an admission controller**, rejecting non-conforming resources at
|
||||
reconcile time (defense in depth) — authored once, shared between CLI gate and admission.
|
||||
- **Chainsaw** — the **L1 rendering, L2 behavioral, and L3 fault-injection** runner. Chainsaw's
|
||||
declarative test steps (apply/assert/error, `script`/`command`, `try`/`catch`/`finally`), bounded
|
||||
polling with explicit timeouts, per-test namespace isolation, and automatic cleanup directly
|
||||
satisfy the reproducibility contract (hermetic, deterministic, self-cleaning). Fault-injection
|
||||
(force the VPN tunnel down, cordon/kill a node, inject a known-bad image, trigger a restore) is
|
||||
expressed via Chainsaw `script` steps, with the fail-closed / no-data-loss outcome asserted by
|
||||
probe fixtures and `finally` steps restoring state.
|
||||
|
||||
Both emit structured, requirement-tagged results feeding the coverage report and the CI merge / phase
|
||||
exit gates ([ADR-0010](adr-0010-conformance-testing.md), [`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md)).
|
||||
Tests declare their `R-*` IDs (e.g. via labels/annotations/naming) so coverage is generated
|
||||
automatically.
|
||||
|
||||
## Alternatives considered
|
||||
- **kuttl** (for L1/L2) — Chainsaw is its spiritual successor with a richer assertion model, native
|
||||
scripting (needed for L3), and try/catch/finally cleanup; chosen over kuttl.
|
||||
- **Conftest/OPA (Rego)** for L0 — viable, but Kyverno keeps policy-as-code in the same
|
||||
YAML/ecosystem as Chainsaw and can double as an in-cluster enforcer; chosen over introducing Rego.
|
||||
- **Ginkgo/Gomega (Go)** for L3 — more power but more code and a second language/toolchain; Chainsaw
|
||||
`script` steps cover the platform's fault-injection needs declaratively. Revisit via a superseding
|
||||
ADR only if a scenario genuinely exceeds Chainsaw's orchestration.
|
||||
- **Testkube** as an orchestrator — unnecessary layer for this scale; CI + Chainsaw + Kyverno CLI
|
||||
suffice.
|
||||
|
||||
## Consequences
|
||||
- A single, coherent ecosystem (Kyverno project) spans policy enforcement and behavior/fault tests —
|
||||
fewer tools, shared idioms.
|
||||
- L3 fault-injection is expressed in Chainsaw `script` steps. **L2/L3 run as an in-cluster Chainsaw
|
||||
Job** on the dev/test cluster, **triggered by CI** — so CI holds no cluster credentials; the Job's
|
||||
own scoped ServiceAccount does. L0/L1 still run in CI without a cluster. (Resolves ADR-0010's open
|
||||
harness-location item.)
|
||||
- The test Job's ServiceAccount is a **privileged, scoped test identity** (create/delete test
|
||||
namespaces + fixtures, patch NetworkPolicies, scale VPN sidecar/gateway, cordon/kill a node, inject
|
||||
a bad image, trigger a Velero restore). It must be least-privilege for that set, named, documented,
|
||||
never reused by workloads, and treated as its own audit item (R-SEC-02).
|
||||
- Kyverno runs **both** as the CI policy gate **and** as an in-cluster admission enforcer (decided):
|
||||
non-conforming resources are rejected at reconcile time as well as in CI. This must not *replace*
|
||||
behavior tests — a policy admitting a manifest is not proof the runtime behaves (R-TEST-01), so
|
||||
L2/L3 Chainsaw tests remain mandatory. Admission enforcement adds an operational failure mode
|
||||
(a bad policy can block legitimate reconciliation); policies ship with their own tests and roll out
|
||||
via GitOps.
|
||||
- Reproducibility properties (hermetic, deterministic, self-cleaning, requirement-tagged, JUnit +
|
||||
coverage) remain the acceptance bar; the tools serve them, never relax them.
|
||||
- Pin tool versions (Kyverno, Chainsaw) for deterministic runs; upgrades flow through the normal
|
||||
update path with the suite re-green as the gate.
|
||||
@@ -0,0 +1,189 @@
|
||||
# 00 — Vision & Scope
|
||||
|
||||
## Vision
|
||||
|
||||
Build a home platform that behaves like a small private cloud. An operator declares a service — a
|
||||
container app *or* a virtual machine — in git, and the platform provisions it, secures it, exposes
|
||||
it under a trusted TLS URL, routes its traffic on the correct WAN, protects its data, updates it
|
||||
automatically, and reports on its health. Onboarding a new service is a pull request against a
|
||||
catalog, not a weekend of manual work.
|
||||
|
||||
The platform is delivered in two stages:
|
||||
|
||||
- **dev/test** on local KVMs — the active build target, used to prove viability.
|
||||
- **prod** on 3 mini-PCs — the eventual home, designed for now but not built on yet.
|
||||
|
||||
## Personas
|
||||
|
||||
- **Operator (primary user)**: runs the home network; wants to onboard/offboard services with
|
||||
minimal ceremony and zero data loss.
|
||||
- **Platform (the system)**: reconciles declared state, enforces isolation and egress policy,
|
||||
renews certificates, updates workloads, and raises alerts.
|
||||
- **Implementation agents**: the AI teams building the platform from this spec.
|
||||
|
||||
## In scope
|
||||
|
||||
- A k3s-based platform with distributed storage (Longhorn) across 3 nodes.
|
||||
- Container workloads (PaaS) and VM workloads (IaaS) through one Kubernetes surface.
|
||||
- GitOps delivery, automated updates, trusted TLS, DNS, ingress.
|
||||
- Network isolation (default-deny east-west) and multi-WAN egress including VPN routing.
|
||||
- Observability with email + dashboard alerting.
|
||||
- A command-and-control web UI at the root of `home.keithjackson.dev`.
|
||||
- The initial service catalog (13 services, see [`13-service-catalog.md`](13-service-catalog.md)).
|
||||
|
||||
## Out of scope (for now)
|
||||
|
||||
- Building on production mini-PC hardware (design-only until the roadmap flips it to active).
|
||||
- Multi-tenant / multi-operator RBAC beyond a single operator + platform automation.
|
||||
- Public multi-user SaaS concerns (billing, per-tenant isolation beyond network policy).
|
||||
- Non-`keithjackson.dev` domains.
|
||||
|
||||
## The domain & DNS model
|
||||
|
||||
- Public apex `*.keithjackson.dev` is available and its records resolve to the network's public IP.
|
||||
- All platform services live under `*.home.keithjackson.dev`.
|
||||
- Each service is reachable at `<service>.home.keithjackson.dev` (e.g. `plex.home.keithjackson.dev`).
|
||||
- The **root** `home.keithjackson.dev` serves the **Command & Control (C2) center** UI.
|
||||
|
||||
## Requirements (stable IDs)
|
||||
|
||||
Requirement IDs are stable references used across ADRs, specs, the task board, and PRs. `MUST` is
|
||||
mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
|
||||
|
||||
### Platform & API surface — `R-PLAT-*`
|
||||
- **R-PLAT-01** — The platform MUST run on k3s across 3 nodes with production-grade HA control plane
|
||||
in prod; dev/test MAY run fewer nodes but MUST exercise the same multi-node topology.
|
||||
- **R-PLAT-02** — Container workloads MUST be onboarded and managed declaratively via the Kubernetes
|
||||
API, `kubectl`-class CLI, and YAML.
|
||||
- **R-PLAT-03** — Virtual-machine workloads MUST be created, managed, and destroyed through the
|
||||
*same* Kubernetes API/CLI/YAML surface as containers (no separate hypervisor management plane for
|
||||
application VMs).
|
||||
- **R-PLAT-04** — The platform MUST support create / destroy / rebuild of any workload on demand.
|
||||
|
||||
### GitOps & delivery — `R-GITOPS-*`
|
||||
- **R-GITOPS-01** — All application and platform desired state MUST be defined in git and reconciled
|
||||
automatically (pull-based GitOps).
|
||||
- **R-GITOPS-02** — Onboarding a new service MUST be expressible as a change to a versioned catalog
|
||||
in git.
|
||||
- **R-GITOPS-03** — The system MUST detect and correct drift between git and cluster state.
|
||||
|
||||
### Networking, isolation & egress — `R-NET-*`
|
||||
- **R-NET-01** — East-west traffic between services MUST be **denied by default**; communication is
|
||||
allowed only by explicit, port-level configuration (firewall-like).
|
||||
- **R-NET-02** — Allow rules MUST be declarative, reviewable in git, and scoped per source →
|
||||
destination → port/protocol.
|
||||
- **R-NET-03** — Networks MUST be configurable and isolatable (segments/namespaces with enforced
|
||||
boundaries).
|
||||
- **R-NET-04** — The platform MUST support **multiple WANs** with egress selection driven by routing
|
||||
rules.
|
||||
- **R-NET-05** — A service MUST be able to declare its **egress class**: `internet` (standard local
|
||||
network/internet) or `vpn` (routed through a custom VPN connection).
|
||||
- **R-NET-06** — VPN-class egress MUST enforce a **kill-switch**: if the VPN tunnel is down, the
|
||||
service MUST NOT leak traffic to the standard internet.
|
||||
- **R-NET-07** — Egress class MUST be a declarative property of the service, changeable via GitOps
|
||||
without re-architecting the workload.
|
||||
|
||||
### Storage & state — `R-STATE-*`
|
||||
- **R-STATE-01** — Stateful data MUST survive destroy/rebuild of the workload (PaaS and IaaS).
|
||||
- **R-STATE-02** — Persistent storage MUST be distributed/replicated across the 3 nodes (Longhorn)
|
||||
so a single node loss does not lose data.
|
||||
- **R-STATE-03** — The platform MUST provide scheduled backups to an **off-cluster** target and a
|
||||
tested restore path (disaster recovery).
|
||||
- **R-STATE-04** — Bulk media storage (Plex library) MUST have a capacity strategy distinct from
|
||||
replicated app-config storage (replicating multi-TB media 3× is out of scope; see ADR-0003).
|
||||
|
||||
### Automated updates — `R-UPD-*`
|
||||
- **R-UPD-01** — Services MUST update to the latest **stable** versions on a schedule.
|
||||
- **R-UPD-02** — Updates MUST require **no human, no AI, and no hand-authored GitOps change** to
|
||||
occur. Automation controllers acting on declarative policy are the permitted mechanism.
|
||||
- **R-UPD-03** — Update policy MUST be per-service configurable (cadence, stability channel,
|
||||
pin/hold, auto-rollback on failed health).
|
||||
|
||||
### TLS & certificates — `R-TLS-*`
|
||||
- **R-TLS-01** — Every externally reachable service MUST present TLS from a **credible, trusted,
|
||||
non-self-signed** authority.
|
||||
- **R-TLS-02** — Certificates MUST be automatically **issued, renewed, and revoked** without manual
|
||||
steps.
|
||||
- **R-TLS-03** — Certificate issuance MUST work for internal-only services (no requirement to expose
|
||||
a service publicly just to validate a certificate).
|
||||
|
||||
### DNS & ingress — `R-DNS-*`
|
||||
- **R-DNS-01** — Services MUST be reachable at `<service>.home.keithjackson.dev`.
|
||||
- **R-DNS-02** — DNS records MUST be managed automatically from declared service intent.
|
||||
- **R-DNS-03** — Internal clients MUST resolve service names correctly (split-horizon where public
|
||||
and internal answers differ).
|
||||
|
||||
### Observability & alerting — `R-OBS-*`
|
||||
- **R-OBS-01** — Services MUST be monitored (metrics, logs, health).
|
||||
- **R-OBS-02** — Issues MUST be reported by **email**.
|
||||
- **R-OBS-03** — Issues and health MUST be visible on **dashboards**.
|
||||
- **R-OBS-04** — The platform MUST alert on its own health (nodes, storage, GitOps reconciliation,
|
||||
certificate expiry, VPN tunnel state, backup success).
|
||||
|
||||
### Command & control — `R-C2-*`
|
||||
- **R-C2-01** — The root `home.keithjackson.dev` MUST serve a UI that surfaces the services running
|
||||
in the cluster and links to them.
|
||||
- **R-C2-02** — The C2 UI MUST surface platform health (cluster, storage, GitOps, alerts) and
|
||||
provide controlled operational actions where safe.
|
||||
|
||||
### Security & secrets — `R-SEC-*`
|
||||
- **R-SEC-01** — Secrets MUST NOT be stored in plaintext in git; they MUST be encrypted at rest in
|
||||
the repo and decrypted only in-cluster.
|
||||
- **R-SEC-02** — The platform MUST follow least-privilege for workload identities and network reach.
|
||||
- **R-SEC-03** — All the above properties MUST be demonstrable/auditable (an operator can prove
|
||||
isolation, egress path, cert validity, and backup integrity).
|
||||
|
||||
### Environments — `R-ENV-*`
|
||||
- **R-ENV-01** — dev/test MUST be provisioned as local KVMs via IaC: created, baselined, cluster-
|
||||
joined, and managed as code.
|
||||
- **R-ENV-02** — The same GitOps configuration MUST promote from dev/test to prod with only
|
||||
environment-scoped overrides.
|
||||
- **R-ENV-03** — prod is a 3× mini-PC bare-metal k3s cluster with local storage; it MUST NOT be an
|
||||
active development surface until the roadmap declares it ready.
|
||||
|
||||
### Conformance testing & governance — `R-TEST-*`
|
||||
- **R-TEST-01** — Every requirement MUST be verified by **automated, behavior-based tests**
|
||||
(Given/When/Then style) that assert observable system behavior, not implementation internals.
|
||||
- **R-TEST-02** — Tests MUST be **reproducible**: deterministic, hermetic (create and tear down
|
||||
their own fixtures), version-pinned, and runnable by a single command locally and in CI against
|
||||
any conforming cluster, producing machine-readable, requirement-tagged results.
|
||||
- **R-TEST-03** — A **conformance suite** MUST gate delivery: merges are blocked unless the tests
|
||||
for the touched requirements pass, and phase exits are blocked unless the phase's full suite
|
||||
passes. "Demonstrated" means "the automated test passes," never a manual one-off.
|
||||
- **R-TEST-04** — Tests MUST be **traceable**: each test declares the requirement ID(s) it verifies,
|
||||
and a coverage report maps every `R-*` requirement to its verifying test(s). Uncovered
|
||||
requirements are a gap to close, not an acceptable state.
|
||||
- **R-TEST-05** — Safety-critical behaviors (default-deny, VPN kill-switch, data-survival on
|
||||
rebuild, DR restore, auto-rollback) MUST be verified by **fault-injection** behavior tests that
|
||||
prove fail-closed / no-data-loss outcomes, not merely happy-path checks.
|
||||
- **R-TEST-06 (governance)** — **AI agents MUST NOT modify the specifications** (this `docs/` tree,
|
||||
the requirements, or accepted ADRs). Specs are changed only by the human operator/architect.
|
||||
Agents that find a spec wrong, blocking, ambiguous, or missing MUST **stop and escalate** and MAY
|
||||
draft a proposal, but MUST NOT self-approve or merge a spec/ADR change (see
|
||||
[`AGENTS.md`](../AGENTS.md) §2 and §11).
|
||||
|
||||
## Acceptance philosophy
|
||||
|
||||
Each requirement class maps to an **automated, reproducible, behavior-based conformance test** in
|
||||
dev/test (`R-TEST-*`). "Done" means *the conformance test for the touched requirements passes* — a
|
||||
green, reproducible result traced to requirement IDs — never "manifest merged" and never a manual
|
||||
demonstration. The testing approach is specified in
|
||||
[`docs/14-testing-and-conformance.md`](14-testing-and-conformance.md) and
|
||||
[ADR-0010](../decisions/adr-0010-conformance-testing.md); the gate checklist lives in
|
||||
[`agents/definition-of-done.md`](../agents/definition-of-done.md).
|
||||
|
||||
**Governance:** the specifications are human-owned and **immutable to agents** (R-TEST-06). Agents
|
||||
implement against the spec and prove conformance with tests; they never edit the spec to fit the
|
||||
implementation.
|
||||
|
||||
## Open questions (tracked, not blocking the architecture)
|
||||
|
||||
- **OQ-1 (WAN hardware)**: How many physical WAN uplinks exist in prod, and does the edge router do
|
||||
the multi-WAN policy routing, or is a portion done in-cluster via egress gateways? See
|
||||
[`03-networking-isolation-and-egress.md`](03-networking-isolation-and-egress.md).
|
||||
- **OQ-2 (bulk media)**: Final home for the Plex media library — NAS/NFS vs. a dedicated large disk
|
||||
vs. object storage. See [`04-storage-and-state.md`](04-storage-and-state.md) / ADR-0003.
|
||||
- **OQ-3 (VPN provider)**: Commercial VPN provider + protocol (WireGuard preferred) and how many
|
||||
concurrent tunnels/egress IPs are available.
|
||||
- **OQ-4 (DNS provider API)**: Confirm Cloudflare (or equivalent) manages `keithjackson.dev` with an
|
||||
API token scoped for ACME DNS-01 and ExternalDNS.
|
||||
@@ -0,0 +1,109 @@
|
||||
# 01 — Architecture Overview
|
||||
|
||||
This document is the map. It names the chosen stack, shows how the layers fit, and points to the
|
||||
ADR that locks in each choice and the domain spec that details it. **Do not change a technology
|
||||
choice here without a superseding ADR.**
|
||||
|
||||
## The stack at a glance
|
||||
|
||||
| Layer | Choice | Serves | ADR | Spec |
|
||||
|---|---|---|---|---|
|
||||
| Kubernetes distro | **k3s** (HA, embedded etcd) | R-PLAT-01/02 | [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md) | [02](02-environments-and-provisioning.md) |
|
||||
| CNI / network policy / egress | **Cilium** (+ Hubble, Gateway API, Egress Gateway) | R-NET-01..07 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [03](03-networking-isolation-and-egress.md) |
|
||||
| Distributed storage | **Longhorn** | R-STATE-01..03 | [ADR-0003](../decisions/adr-0003-storage-longhorn.md) | [04](04-storage-and-state.md) |
|
||||
| GitOps engine | **Flux CD** | R-GITOPS-01..03 | [ADR-0004](../decisions/adr-0004-gitops-flux.md) | [05](05-gitops-and-delivery.md) |
|
||||
| VM workloads (IaaS) | **KubeVirt** (+ CDI) | R-PLAT-03 | [ADR-0005](../decisions/adr-0005-vm-workloads-kubevirt.md) | [06](06-vm-workloads-kubevirt.md) |
|
||||
| TLS | **cert-manager** + Let's Encrypt (DNS-01) | R-TLS-01..03 | [ADR-0006](../decisions/adr-0006-tls-cert-manager-letsencrypt.md) | [08](08-tls-and-certificates.md) |
|
||||
| Secrets | **SOPS + age** (Flux-native) | R-SEC-01 | [ADR-0007](../decisions/adr-0007-secrets-sops-age.md) | [07](07-security-and-secrets.md) |
|
||||
| VPN egress | **Gluetun sidecar** + **Cilium Egress Gateway** | R-NET-05/06 | [ADR-0008](../decisions/adr-0008-vpn-egress-model.md) | [03](03-networking-isolation-and-egress.md) |
|
||||
| Automated updates | **Flux image-update-automation** (images) + **Renovate** (charts) | R-UPD-01..03 | [ADR-0009](../decisions/adr-0009-automated-image-updates.md) | [11](11-automated-updates.md) |
|
||||
| DNS records | **ExternalDNS** (public via Cloudflare) + **PiHole** (internal) | R-DNS-01..03 | — | [09](09-dns-and-ingress.md) |
|
||||
| Ingress (north-south) | **Gateway API** implemented by Cilium | R-DNS-01, R-TLS-01 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [09](09-dns-and-ingress.md) |
|
||||
| Observability | **kube-prometheus-stack** + **Loki** + **Grafana Alloy** + Alertmanager→SMTP | R-OBS-01..04 | — | [10](10-observability-and-alerting.md) |
|
||||
| Backup/DR | **Velero** + Longhorn backups → off-cluster S3/NFS | R-STATE-03 | [ADR-0003](../decisions/adr-0003-storage-longhorn.md) | [04](04-storage-and-state.md) |
|
||||
| Node/infra provisioning | **OpenTofu** (libvirt) + **cloud-init** + **Ansible** (k3s bootstrap) | R-ENV-01..03 | [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md) | [02](02-environments-and-provisioning.md) |
|
||||
| Command center UI | **Homepage** + **Headlamp** + **Grafana** panels | R-C2-01/02 | — | [12](12-command-center-ui.md) |
|
||||
| Conformance testing | **Kyverno** (L0 policy) + **Chainsaw** (L1/L2/L3 behavior + fault-injection) | R-TEST-01..06 | [ADR-0010](../decisions/adr-0010-conformance-testing.md), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md) | [14](14-testing-and-conformance.md) |
|
||||
|
||||
## Layer cake
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────────────┐
|
||||
│ Command & Control UI (home.keithjackson.dev root) │
|
||||
│ Homepage catalog · Headlamp k8s console · Grafana dashboards · Flux status │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Workloads │
|
||||
│ PaaS: containers (Plex, PiHole, Nextcloud, Gitea, Seerr, …) │
|
||||
│ IaaS: KubeVirt VMs (anything that can't be a container) │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Platform services │
|
||||
│ Flux (GitOps) · cert-manager (TLS) · ExternalDNS · Gateway API │
|
||||
│ kube-prometheus-stack + Loki + Alloy (observability) │
|
||||
│ Velero (backup) · Renovate (chart updates) · SOPS (secrets) │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Storage: Longhorn (replicated block) + bulk media store (see ADR-0003) │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Network: Cilium (default-deny NetworkPolicy · Egress Gateway · Hubble) │
|
||||
│ VPN egress (Gluetun sidecars / VPN gateway) · Multi-WAN routing │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Kubernetes: k3s (3 nodes, HA control plane, embedded etcd) │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Nodes │
|
||||
│ dev/test: libvirt KVMs (OpenTofu + cloud-init + Ansible) │
|
||||
│ prod: 3× mini-PC bare metal, local disks │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Edge: router/firewall — multi-WAN uplinks, port-forward to Gateway, │
|
||||
│ VPN uplink(s), split-horizon DNS │
|
||||
└──────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Two clearly-separated virtualization layers (do not conflate)
|
||||
|
||||
This platform has **two** kinds of VM, and confusing them is the most common design error:
|
||||
|
||||
1. **Infrastructure VMs = the cluster nodes themselves.**
|
||||
In dev/test these are libvirt KVMs on the workstation, provisioned by OpenTofu + cloud-init and
|
||||
configured/joined by Ansible. In prod these are the bare-metal mini-PCs. See
|
||||
[`02-environments-and-provisioning.md`](02-environments-and-provisioning.md).
|
||||
|
||||
2. **Application VMs = IaaS workloads that run *inside* the cluster via KubeVirt.**
|
||||
These are managed by the same Kubernetes API/YAML/GitOps as containers. They satisfy R-PLAT-03
|
||||
("not everything fits in a container"). See [`06-vm-workloads-kubevirt.md`](06-vm-workloads-kubevirt.md).
|
||||
|
||||
> Layer 1 is how the platform exists. Layer 2 is a feature the platform offers.
|
||||
|
||||
## How a service flows through the platform (the golden path)
|
||||
|
||||
1. Operator adds/edits a **ServiceDefinition** in the git catalog (see
|
||||
[`13-service-catalog.md`](13-service-catalog.md)) declaring: kind (container|vm), image/source,
|
||||
storage, **egress class** (`internet`|`vpn`), exposure (`internal`|`public`), dependencies
|
||||
(explicit allow rules), update policy, and observability expectations.
|
||||
2. **Flux** reconciles the change into the cluster.
|
||||
3. **Longhorn** provisions replicated persistent volumes; state is durable.
|
||||
4. **Cilium** applies default-deny plus the explicitly declared allow rules; assigns the correct
|
||||
**egress path** (local internet or VPN, with kill-switch).
|
||||
5. **ExternalDNS** publishes `<service>.home.keithjackson.dev`; **Gateway API** routes north-south
|
||||
traffic; **cert-manager** issues a trusted TLS cert via ACME DNS-01.
|
||||
6. **kube-prometheus-stack + Loki** scrape metrics/logs; **Alertmanager** emails on issues; the
|
||||
**C2 UI** lists and links the service.
|
||||
7. **Flux image automation / Renovate** keep the service on the latest stable version per policy —
|
||||
no human action.
|
||||
8. **Velero + Longhorn backups** protect the data; destroy/rebuild restores state.
|
||||
|
||||
Every numbered step maps to a domain spec and a workstream in [`roadmap/workstreams.md`](../roadmap/workstreams.md).
|
||||
|
||||
## Design principles
|
||||
|
||||
- **Declarative everything.** If it can't be expressed as YAML reconciled from git, question it.
|
||||
- **One control plane.** Containers and VMs share the Kubernetes surface; operators learn one model.
|
||||
- **Secure by default.** Deny-by-default network, least-privilege identity, encrypted secrets,
|
||||
trusted TLS everywhere.
|
||||
- **Durable by default.** No workload is "pet" hardware; state lives on replicated storage with
|
||||
tested restore.
|
||||
- **Hands-off operations.** Updates, cert renewal, DNS, and drift correction are automated.
|
||||
- **Promotable.** dev/test and prod are the same config with scoped overrides.
|
||||
- **Provable.** Every requirement is verified by an automated, reproducible, behavior-based
|
||||
conformance test ([14](14-testing-and-conformance.md), [ADR-0010](../decisions/adr-0010-conformance-testing.md));
|
||||
"done" is a green, requirement-tagged suite, not a manual demo.
|
||||
- **Spec-fixed.** The specification is human-owned and immutable to agents (R-TEST-06); agents move
|
||||
the implementation and its tests toward the spec, never the spec toward the implementation.
|
||||
@@ -0,0 +1,93 @@
|
||||
# 02 — Environments & Provisioning
|
||||
|
||||
Serves: **R-ENV-01/02/03**, **R-PLAT-01**. Related: [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md).
|
||||
|
||||
## Two environments, one configuration
|
||||
|
||||
| Aspect | dev/test | prod |
|
||||
|---|---|---|
|
||||
| Substrate | libvirt/KVM VMs on the operator workstation | 3× mini-PC bare metal |
|
||||
| Node count | 3 VMs (mirror prod topology) | 3 physical nodes |
|
||||
| Control plane | HA embedded etcd (k3s server on all 3) | HA embedded etcd (k3s server on all 3) |
|
||||
| Storage | virtual disks presented to Longhorn | local SSD/NVMe presented to Longhorn |
|
||||
| Status | **active build target** | **design-only until roadmap flips it** |
|
||||
|
||||
**R-ENV-02 is the golden constraint:** the GitOps configuration is authored once. Environments
|
||||
differ only through scoped overrides (Flux `Kustomization` overlays / cluster-scoped variables):
|
||||
node counts, disk sizes, WAN endpoints, VPN credentials, DNS zone specifics, resource limits.
|
||||
Never fork the base config per environment.
|
||||
|
||||
## Infrastructure provisioning (Layer 1 — the cluster nodes)
|
||||
|
||||
> These are the machines that *run* k3s. Distinct from KubeVirt application VMs (Layer 2).
|
||||
|
||||
### Toolchain
|
||||
- **OpenTofu** (Terraform-compatible) with the **libvirt** provider defines dev/test node VMs as
|
||||
code: CPU, RAM, disks (a dedicated data disk for Longhorn), NICs, and a **cloud-init** seed.
|
||||
- **cloud-init** performs first-boot baseline: hostname, users, SSH keys, package baseline, kernel
|
||||
modules/sysctls required by k3s/Longhorn/KubeVirt, and marks the node ready for k3s.
|
||||
- **Ansible** performs cluster bootstrap: installs k3s in HA mode, joins nodes, lays down cluster-
|
||||
wide prerequisites (Cilium is installed as the CNI, not flannel), and hands off to Flux.
|
||||
- For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation step differs
|
||||
(bare-metal image/cloud-init via USB/PXE instead of libvirt).
|
||||
|
||||
### Node lifecycle stages (R-ENV-01 — "created, baselined, cluster-joined, managed as code")
|
||||
1. **Create** — OpenTofu instantiates the VM (dev) / operator images the mini-PC (prod).
|
||||
2. **Baseline** — cloud-init applies OS baseline and prerequisites.
|
||||
3. **Cluster-join** — Ansible installs/joins k3s with Cilium as CNI; disables k3s bundled
|
||||
flannel/traefik/servicelb/kube-proxy as required by the Cilium ADR.
|
||||
4. **Manage** — from here the cluster is GitOps-managed by Flux; node config changes are made via
|
||||
IaC and re-applied, never hand-edited on the box.
|
||||
|
||||
### k3s topology decisions
|
||||
- HA control plane with **embedded etcd** across all 3 nodes (no external datastore).
|
||||
- Bundled components that conflict with our stack are **disabled at install**: flannel (→ Cilium),
|
||||
kube-proxy (→ Cilium kube-proxy replacement), Traefik (→ Gateway API via Cilium), servicelb where
|
||||
a different LB strategy is chosen. Exact flags live in the Ansible bootstrap; the *intent* is
|
||||
fixed here and in ADR-0001/0002.
|
||||
- Nodes are both control-plane and workload (3-node converged cluster); with only 3 nodes we do not
|
||||
isolate control-plane nodes.
|
||||
|
||||
### Bare-metal specifics (prod, design-only for now)
|
||||
- Each mini-PC contributes local NVMe/SSD to Longhorn; document the disk layout (OS vs. Longhorn
|
||||
data device) before prod build.
|
||||
- BIOS/firmware, virtualization extensions (for KubeVirt), and wake/boot behavior are captured as a
|
||||
provisioning checklist under the prod-readiness workstream.
|
||||
- Edge networking (multi-WAN, VPN uplink) is largely an **edge-router** concern — see
|
||||
[`03-networking-isolation-and-egress.md`](03-networking-isolation-and-egress.md) and OQ-1.
|
||||
|
||||
## Environment promotion model
|
||||
|
||||
```
|
||||
git repo (single source of truth)
|
||||
│
|
||||
base configuration ──────────────┐
|
||||
│ │
|
||||
overlay: dev/test overlay: prod
|
||||
│ │
|
||||
Flux on dev cluster Flux on prod cluster (later)
|
||||
```
|
||||
|
||||
- A change proves out in **dev/test** against the acceptance gates ([`AGENTS.md` §7](../AGENTS.md)).
|
||||
- Promotion to prod is a change to the prod overlay only (no base edits), gated by the roadmap's
|
||||
prod-readiness phase.
|
||||
- **Never develop against prod.** Prod hardware is a target, not a sandbox (R-ENV-03).
|
||||
|
||||
## Statelessness of the nodes themselves
|
||||
|
||||
Nodes are cattle. A node can be destroyed and rebuilt from IaC + cloud-init + Ansible; cluster and
|
||||
application state lives on **Longhorn** (replicated) and in **git** (desired state), so node loss is
|
||||
a re-provision, not a data-loss event (supports R-STATE-01/02).
|
||||
|
||||
## Deliverables for this spec (tracked on the task board)
|
||||
- OpenTofu module for dev/test node VMs (libvirt) with a dedicated Longhorn data disk.
|
||||
- cloud-init baseline template (users, SSH, sysctls, kernel modules for Longhorn/KubeVirt).
|
||||
- Ansible role(s): k3s HA install with Cilium, component disabling, Flux bootstrap handoff.
|
||||
- Documented prod bare-metal provisioning checklist (design-only; do not execute).
|
||||
- Proof: destroy a dev node VM and rebuild it; cluster re-converges with no data loss.
|
||||
|
||||
## Open questions
|
||||
- **OQ-1** (edge router / WAN count) — blocks final multi-WAN design, not node provisioning.
|
||||
- Do we want **Cluster API** later to unify dev VM and prod bare-metal lifecycle? Deferred; OpenTofu
|
||||
+ Ansible is sufficient and lower-complexity for 3 nodes now. Revisit via ADR if prod scaling
|
||||
demands it.
|
||||
@@ -0,0 +1,126 @@
|
||||
# 03 — Networking, Isolation & Egress
|
||||
|
||||
Serves: **R-NET-01..07**, **R-SEC-02**. Related: [ADR-0002 (Cilium)](../decisions/adr-0002-cni-cilium.md),
|
||||
[ADR-0008 (VPN egress)](../decisions/adr-0008-vpn-egress-model.md).
|
||||
|
||||
This is the most safety-critical spec. Get default-deny and the VPN kill-switch right; everything
|
||||
else is convenience.
|
||||
|
||||
## Principles
|
||||
|
||||
1. **Deny by default, allow by exception.** No service may talk to another unless a rule explicitly
|
||||
permits it, at the port/protocol level (R-NET-01/02).
|
||||
2. **Egress is a declared property.** Each service declares an **egress class**; the platform routes
|
||||
it accordingly (R-NET-05/07).
|
||||
3. **VPN means VPN-only.** A VPN-class service must never leak to the plain internet, including
|
||||
during tunnel failure (R-NET-06, kill-switch).
|
||||
4. **Isolation is auditable.** An operator can prove what a service can and cannot reach (R-SEC-03).
|
||||
|
||||
## CNI: Cilium
|
||||
|
||||
Cilium replaces the default k3s CNI and provides:
|
||||
- **NetworkPolicy / CiliumNetworkPolicy** for L3/L4 (and optional L7) enforcement.
|
||||
- **kube-proxy replacement** (eBPF) for efficient service routing.
|
||||
- **Egress Gateway** to force selected pods' egress through a chosen node/identity/SNAT IP — the
|
||||
building block for WAN/VPN path selection.
|
||||
- **Hubble** for flow observability — the audit and troubleshooting tool for isolation.
|
||||
- **Gateway API** implementation for north-south ingress (see [09](09-dns-and-ingress.md)).
|
||||
|
||||
## Segmentation model
|
||||
|
||||
- **Namespaces as trust zones.** Each service (or tightly-coupled service group) gets its own
|
||||
namespace. Namespace = isolation boundary.
|
||||
- **Egress classes as zones:** `internet` and `vpn` are first-class labels that drive both network
|
||||
policy and egress routing.
|
||||
- **Platform namespace(s)** (Flux, cert-manager, monitoring, storage) are isolated from application
|
||||
namespaces; only the specific flows they need (e.g. metrics scraping) are opened.
|
||||
|
||||
### Default-deny baseline (R-NET-01)
|
||||
Every namespace gets a baseline policy set applied by GitOps at namespace creation:
|
||||
- **Deny all ingress** and **deny all egress** by default.
|
||||
- Then explicitly allow only:
|
||||
- DNS to the in-cluster resolver (required for anything to function).
|
||||
- Declared intra-cluster dependencies (source → destination → port).
|
||||
- The declared egress class path (internet or VPN).
|
||||
- Observability scraping from the monitoring namespace (metrics/logs).
|
||||
|
||||
> A workload with no explicit allow rules can do *nothing* except resolve DNS. This is the intended
|
||||
> starting posture for every onboarded service.
|
||||
|
||||
### Explicit allow rules (R-NET-02)
|
||||
Service dependencies from the catalog compile into allow rules. Example intent (illustrative, not
|
||||
final manifest):
|
||||
|
||||
```
|
||||
seerr → sonarr tcp/8989 (Seerr requests to Sonarr)
|
||||
seerr → radarr tcp/7878 (Seerr requests to Radarr)
|
||||
seerr → plex tcp/32400 (Seerr talks to Plex)
|
||||
```
|
||||
|
||||
Rules are declared **once** in the service's catalog entry (its `dependencies`) and rendered into
|
||||
CiliumNetworkPolicies by the platform. Reviewers see the intent, not hand-written policy YAML.
|
||||
|
||||
## Multi-WAN & egress routing (R-NET-04/05)
|
||||
|
||||
The platform recognizes **egress profiles**. Initial profiles:
|
||||
|
||||
| Profile | Meaning | Mechanism |
|
||||
|---|---|---|
|
||||
| `internet` | Standard local network → default WAN(s) | Normal pod egress via node → edge router |
|
||||
| `vpn` | Routed through a commercial VPN tunnel | Gluetun sidecar and/or Cilium Egress Gateway to a VPN egress node (ADR-0008) |
|
||||
|
||||
**Where multi-WAN policy routing lives (OQ-1):**
|
||||
- **Physical multi-WAN** (two ISP uplinks, failover/balancing by routing rules) is primarily an
|
||||
**edge-router/firewall** responsibility (e.g. OPNsense/pfSense) sitting in front of the cluster.
|
||||
Traffic leaving the cluster on the `internet` profile is policy-routed by the edge router across
|
||||
available WANs.
|
||||
- **In-cluster egress selection** (which pods use VPN vs. plain internet) is handled by Cilium
|
||||
Egress Gateway + the VPN egress model. The cluster decides *class*; the edge router decides
|
||||
*physical WAN* for the internet class.
|
||||
- Final split between edge-router and in-cluster responsibility is pinned once OQ-1 (WAN hardware)
|
||||
is answered. The **service-facing contract does not change**: a service still just declares
|
||||
`egress: internet|vpn`.
|
||||
|
||||
## VPN egress model (R-NET-05/06 — ADR-0008)
|
||||
|
||||
VPN-class services: **sonarr, radarr, jackett, qbittorrent**. Requirements: all their egress leaves
|
||||
via the VPN; if the tunnel drops, they lose connectivity rather than leaking.
|
||||
|
||||
Two supported patterns (ADR-0008 selects the default; both preserve the `egress: vpn` contract):
|
||||
|
||||
- **Pattern A — Gluetun sidecar per pod.** A Gluetun container in the pod holds the WireGuard/OpenVPN
|
||||
tunnel; the app container shares its network namespace and routes all traffic through it. Gluetun's
|
||||
**firewall kill-switch** blocks non-tunnel egress by default — satisfies R-NET-06 intrinsically.
|
||||
Simple, self-contained, per-service isolation; more sidecars to run.
|
||||
- **Pattern B — Central VPN egress gateway + Cilium Egress Gateway.** One (HA) VPN gateway
|
||||
deployment holds the tunnel; Cilium Egress Gateway steers `vpn`-labeled pods' egress through it.
|
||||
Fewer tunnels; a shared failure domain; kill-switch enforced at the gateway + policy denying
|
||||
fallback to the internet path.
|
||||
|
||||
**Kill-switch is mandatory in both patterns and is an acceptance gate:** with the tunnel forcibly
|
||||
down, a `vpn` service must fail closed (no packets to the plain internet). This must be tested, not
|
||||
assumed.
|
||||
|
||||
## Ingress (north-south) — summary
|
||||
|
||||
North-south exposure is handled by **Gateway API implemented by Cilium**, with TLS from
|
||||
cert-manager and DNS from ExternalDNS. Detailed in [`09-dns-and-ingress.md`](09-dns-and-ingress.md).
|
||||
Exposure is a declared property: `internal` (LAN-only) vs `public` (port-forwarded at the edge).
|
||||
|
||||
## Observability of the network (supports R-SEC-03, R-OBS-04)
|
||||
- **Hubble** provides flow visibility to verify default-deny and troubleshoot allow rules.
|
||||
- Alert on: VPN tunnel down, egress-gateway unhealthy, unexpected denied flows spiking, policy
|
||||
reconciliation errors. Wired into [`10-observability-and-alerting.md`](10-observability-and-alerting.md).
|
||||
|
||||
## Acceptance gates for this spec
|
||||
- A freshly onboarded service with no rules can reach **nothing** but DNS (default-deny proven via
|
||||
Hubble + a connectivity probe).
|
||||
- Declared dependencies work; **undeclared** ones are blocked.
|
||||
- A `vpn` service's public egress IP is the VPN's, not the home IP (verified).
|
||||
- Kill-switch: tunnel down ⇒ `vpn` service has no internet path; `internet` services unaffected.
|
||||
- Changing a service's `egress` class in git flips its path with no workload rearchitecting.
|
||||
|
||||
## Open questions
|
||||
- **OQ-1** — WAN uplink count and edge-router product; determines edge-vs-cluster routing split.
|
||||
- **OQ-3** — VPN provider/protocol (WireGuard preferred), number of concurrent tunnels / distinct
|
||||
egress IPs (affects Pattern A vs B and HA of the gateway).
|
||||
@@ -0,0 +1,91 @@
|
||||
# 04 — Storage & State
|
||||
|
||||
Serves: **R-STATE-01..04**, **R-PLAT-04**. Related: [ADR-0003 (Longhorn)](../decisions/adr-0003-storage-longhorn.md).
|
||||
|
||||
The core promise: **any service can be destroyed and rebuilt on a whim without losing data.** This
|
||||
requires (a) durable replicated storage, (b) a clean separation of state from workload, and (c) a
|
||||
tested backup/restore path.
|
||||
|
||||
## Storage tiers
|
||||
|
||||
| Tier | Backing | Use | Replication |
|
||||
|---|---|---|---|
|
||||
| **App/config state** | Longhorn block volumes | PVCs for service config, databases, small app data | 3-way replica across nodes |
|
||||
| **Bulk media** | Dedicated large store (see below) | Plex library, download staging | Not 3× replicated (capacity strategy) |
|
||||
| **VM disks** | Longhorn (via KubeVirt/CDI) | KubeVirt application-VM root/data disks | 3-way replica |
|
||||
| **Backups** | Off-cluster S3/NFS target | Velero + Longhorn backups for DR | Off-cluster, separate failure domain |
|
||||
|
||||
## Longhorn (primary persistent storage — R-STATE-02)
|
||||
|
||||
- Provides replicated block storage from each node's local disk — exactly the model for a 3× mini-PC
|
||||
cluster with local storage and no SAN.
|
||||
- Default StorageClass: **3 replicas** so any single node loss keeps data available.
|
||||
- Volumes are the durable home for stateful services. Destroying a pod/Deployment/VM leaves the PVC
|
||||
and its data intact; rebuilding re-attaches it (R-STATE-01).
|
||||
- Snapshots for point-in-time; **backups** (to off-cluster target) for DR.
|
||||
- Guardrails: reserve disk headroom, set replica anti-affinity across nodes, monitor volume health
|
||||
and degraded replicas (alerts in [10](10-observability-and-alerting.md)).
|
||||
|
||||
## Separation of state from workload (the rebuild guarantee)
|
||||
|
||||
Every service's catalog entry declares its **persistent volumes** explicitly. The rule:
|
||||
|
||||
> If it must survive a rebuild, it lives on a named PVC (Longhorn) or the bulk store — **never** in
|
||||
> the container's ephemeral filesystem or on a single node's hostPath.
|
||||
|
||||
The rebuild test (an acceptance gate): delete the workload entirely, re-apply from git, and the
|
||||
service comes back with its data. This is verified per service, not assumed.
|
||||
|
||||
## Bulk media strategy (R-STATE-04 — the Plex problem)
|
||||
|
||||
Plex is the largest service and its library can be multiple terabytes. **3× replicating multi-TB
|
||||
media across three mini-PCs is impractical** (capacity and cost). Therefore bulk media is a
|
||||
**distinct tier** from replicated app-config storage. Options to be pinned by ADR-0003 / OQ-2:
|
||||
|
||||
- **NAS / external NFS** exporting the media library, mounted into Plex (and the arr stack) — keeps
|
||||
bulk data off the replicated tier; simplest capacity story; introduces an external dependency.
|
||||
- **Dedicated large disk** on one node with **single-replica Longhorn** or a local PV — cheaper, but
|
||||
node-bound; requires its own backup plan.
|
||||
- **Object storage** for some content — generally poor fit for Plex direct play.
|
||||
|
||||
Regardless of choice: **Plex's config/metadata database is small and MUST live on 3-replica
|
||||
Longhorn** (so Plex rebuilds cleanly); only the large media files use the bulk tier. Downloads
|
||||
(qbittorrent) stage to the bulk tier and hand off to the arr stack.
|
||||
|
||||
## Backup & disaster recovery (R-STATE-03)
|
||||
|
||||
- **Velero** backs up Kubernetes resource state (namespaces, manifests, PVC metadata) and
|
||||
orchestrates volume backups.
|
||||
- **Longhorn backups** push volume data to an **off-cluster** S3-compatible or NFS target — a
|
||||
separate failure domain from the cluster itself.
|
||||
- **Schedule**: regular automated backups (cadence per data criticality) with retention policy.
|
||||
- **Restore is a tested procedure**, not a theoretical one: a DR drill (restore a namespace + its
|
||||
volumes to a clean state) is part of the platform acceptance suite.
|
||||
- Bulk media has its own backup posture (e.g. it may be re-acquirable rather than backed up 1:1 —
|
||||
an explicit operator decision recorded per OQ-2), but **all irreplaceable state** (configs,
|
||||
databases, Nextcloud data, Gitea repos) MUST be backed up off-cluster.
|
||||
|
||||
## KubeVirt VM disks
|
||||
|
||||
Application-VM disks are Longhorn volumes provisioned via **CDI** (Containerized Data Importer). VMs
|
||||
inherit the same durability and backup story as container PVCs — a KubeVirt VM can be destroyed and
|
||||
rebuilt from its declarative definition + persistent disk. See [`06-vm-workloads-kubevirt.md`](06-vm-workloads-kubevirt.md).
|
||||
|
||||
## GitOps + state = full rebuildability
|
||||
|
||||
- **Desired state** lives in git (manifests, service definitions).
|
||||
- **Data** lives on Longhorn/bulk store + off-cluster backups.
|
||||
- Therefore a total cluster loss is recoverable: re-provision nodes ([02](02-environments-and-provisioning.md)),
|
||||
re-bootstrap Flux, let it reconcile from git, and restore volumes from backup. This end-to-end
|
||||
recovery is the ultimate expression of R-STATE-01/03 and SHOULD be drilled at least once in
|
||||
dev/test before prod is declared ready.
|
||||
|
||||
## Acceptance gates
|
||||
- Delete and recreate each stateful service; data survives (per-service rebuild test).
|
||||
- Kill a node (dev/test); replicated volumes stay available; cluster re-converges.
|
||||
- Restore a namespace + volumes from off-cluster backup into a clean state (DR drill).
|
||||
- Plex rebuilds with its metadata DB from replicated storage; media re-attaches from the bulk tier.
|
||||
|
||||
## Open questions
|
||||
- **OQ-2** — final bulk-media home (NAS/NFS vs dedicated disk vs mixed) and its backup posture.
|
||||
- Off-cluster backup **target**: which S3-compatible endpoint or NFS host, and its own resilience.
|
||||
@@ -0,0 +1,93 @@
|
||||
# 05 — GitOps & Delivery
|
||||
|
||||
Serves: **R-GITOPS-01..03**, **R-PLAT-02/04**, **R-ENV-02**. Related: [ADR-0004 (Flux)](../decisions/adr-0004-gitops-flux.md).
|
||||
|
||||
## Model
|
||||
|
||||
Pull-based GitOps with **Flux CD**. The cluster continuously reconciles its state from git. Git is
|
||||
the single source of truth; the cluster conforms to git, not the reverse (R-GITOPS-01). Drift is
|
||||
detected and corrected automatically (R-GITOPS-03).
|
||||
|
||||
## Repository layout (the implementation repo, created in a later phase)
|
||||
|
||||
> This spec repo (`home-network-k3s`) holds intent. The **implementation** repo(s) hold the
|
||||
> reconciled manifests. Proposed structure:
|
||||
|
||||
```
|
||||
platform/ ← cluster-wide platform services (Flux-managed)
|
||||
cilium/ ← CNI, egress gateway, network baseline
|
||||
longhorn/ ← storage
|
||||
cert-manager/ ← TLS issuers + ClusterIssuer
|
||||
externaldns/ ← DNS automation
|
||||
gateway/ ← Gateway API + listeners/TLS
|
||||
monitoring/ ← kube-prometheus-stack, Loki, Alloy, Alertmanager
|
||||
velero/ ← backup/DR
|
||||
kubevirt/ ← KubeVirt + CDI
|
||||
image-automation/ ← Flux image update automation config
|
||||
apps/ ← application workloads, one dir per service
|
||||
plex/ sonarr/ radarr/ jackett/ seerr/ pihole/ nextcloud/
|
||||
minecraft/ unifi/ semaphore/ kasm/ gitea/ qbittorrent/
|
||||
catalog/ ← ServiceDefinition entries (the onboarding surface)
|
||||
clusters/
|
||||
dev/ ← Flux entrypoint + dev overlay/vars
|
||||
prod/ ← Flux entrypoint + prod overlay/vars (design-only)
|
||||
```
|
||||
|
||||
- **Kustomize overlays** (or equivalent) express environment differences (R-ENV-02). Base is shared;
|
||||
`clusters/dev` and `clusters/prod` supply scoped values only.
|
||||
- **HelmReleases** (Flux) wrap upstream charts (Cilium, Longhorn, cert-manager, kube-prometheus-
|
||||
stack, etc.); app workloads use charts where good ones exist, otherwise plain manifests.
|
||||
|
||||
## The onboarding surface: ServiceDefinition (R-GITOPS-02)
|
||||
|
||||
Onboarding a service = adding one **ServiceDefinition** under `catalog/`. It is the operator-facing
|
||||
abstraction that the platform compiles into the concrete resources (Deployment/VM, PVCs,
|
||||
NetworkPolicies, HTTPRoute, Certificate, DNS record, ServiceMonitor, update policy). See
|
||||
[`13-service-catalog.md`](13-service-catalog.md) for the schema. Intent lives in one place; the
|
||||
platform renders the rest.
|
||||
|
||||
## Reconciliation flow
|
||||
|
||||
1. Operator opens a PR against the implementation repo (new/updated ServiceDefinition or platform
|
||||
change).
|
||||
2. CI validates (lint, policy checks, schema validation, dry-run/build).
|
||||
3. Merge to the tracked branch.
|
||||
4. Flux reconciles: applies manifests, reports status, corrects drift on an interval.
|
||||
5. Health/rollout status surfaces in the C2 UI ([12](12-command-center-ui.md)) and alerts on failure
|
||||
([10](10-observability-and-alerting.md)).
|
||||
|
||||
## Secrets in GitOps
|
||||
|
||||
Secrets are committed **encrypted** with **SOPS + age**; Flux decrypts them in-cluster at
|
||||
reconciliation. No plaintext secret ever lands in git. See [`07-security-and-secrets.md`](07-security-and-secrets.md)
|
||||
and [ADR-0007](../decisions/adr-0007-secrets-sops-age.md).
|
||||
|
||||
## Automated updates interplay (R-UPD-*)
|
||||
|
||||
GitOps and hands-off updates coexist:
|
||||
- **Container image tags** advance automatically via **Flux image-update-automation** driven by
|
||||
per-service **image policies** (semver range / stable channel) — the controller resolves and
|
||||
reconciles the new version with **no human-authored change** (R-UPD-02).
|
||||
- **Helm chart / manifest version bumps** are proposed by **Renovate** with auto-merge for allowed
|
||||
update classes.
|
||||
|
||||
Full detail and the "is an automated commit a GitOps change?" nuance is in
|
||||
[`11-automated-updates.md`](11-automated-updates.md) and [ADR-0009](../decisions/adr-0009-automated-image-updates.md).
|
||||
|
||||
## Bootstrap & recovery
|
||||
|
||||
- Flux is bootstrapped onto a fresh cluster by the Ansible handoff ([02](02-environments-and-provisioning.md)).
|
||||
- Because desired state is in git and data is on Longhorn/backups, a rebuilt cluster reconverges by
|
||||
re-bootstrapping Flux and restoring volumes ([04](04-storage-and-state.md)).
|
||||
|
||||
## Guardrails
|
||||
- No `kubectl apply` of app state by hand — everything flows through git/Flux (R-GITOPS-01, click-
|
||||
ops-never per [AGENTS.md](../AGENTS.md)).
|
||||
- Reconciliation errors and drift are **alerting conditions** (R-OBS-04).
|
||||
- Base config is never forked per environment (R-ENV-02).
|
||||
|
||||
## Acceptance gates
|
||||
- Adding a ServiceDefinition PR results in a fully wired service (compute, storage, policy, TLS, DNS,
|
||||
monitoring) with no manual steps beyond the merge.
|
||||
- Manually mutating a live resource is auto-reverted by Flux (drift correction proven).
|
||||
- The same base config renders correctly for both dev and prod overlays.
|
||||
@@ -0,0 +1,79 @@
|
||||
# 06 — VM Workloads (IaaS via KubeVirt)
|
||||
|
||||
Serves: **R-PLAT-03**, **R-PLAT-04**, **R-STATE-01**. Related: [ADR-0005 (KubeVirt)](../decisions/adr-0005-vm-workloads-kubevirt.md).
|
||||
|
||||
## Why this exists
|
||||
|
||||
"Not everything fits in a container." Some workloads need a full OS, a specific kernel, non-container
|
||||
software, or a lift-and-shift of an existing VM. The requirement (R-PLAT-03) is that these VMs are
|
||||
managed through the **same Kubernetes API, CLI, and YAML** as containers — one control plane, one
|
||||
GitOps flow, one network/storage/observability model. **KubeVirt** delivers exactly this: VMs as
|
||||
Kubernetes custom resources.
|
||||
|
||||
## Do not confuse the two VM layers
|
||||
|
||||
- **Layer 1 — cluster nodes** are VMs only in dev/test (libvirt) and bare metal in prod. Managed by
|
||||
OpenTofu/Ansible ([02](02-environments-and-provisioning.md)). **Not** KubeVirt.
|
||||
- **Layer 2 — application VMs** run *inside* the cluster via KubeVirt and are the subject of this
|
||||
spec. Managed by Kubernetes/GitOps like any other workload.
|
||||
|
||||
## What KubeVirt provides
|
||||
|
||||
- `VirtualMachine` / `VirtualMachineInstance` custom resources — declare a VM in YAML, reconcile via
|
||||
Flux, manage via `kubectl` (and `virtctl` for console/start/stop).
|
||||
- **CDI (Containerized Data Importer)** to import/clone disk images into **Longhorn** PVCs — VM disks
|
||||
get the same replication and backup story as container volumes ([04](04-storage-and-state.md)).
|
||||
- VMs sit on the **same Cilium network** — so **default-deny isolation and egress classes apply to
|
||||
VMs too** (R-NET-*). A KubeVirt VM can be `egress: vpn` just like a container.
|
||||
- VMs are exposed through the same **Gateway API / Service** model and monitored through the same
|
||||
stack.
|
||||
|
||||
## The unified contract
|
||||
|
||||
A VM onboarded via a ServiceDefinition (`kind: vm`) gets, from the *same* declarative surface:
|
||||
- Compute (vCPU/RAM), an OS image (via CDI), and Longhorn-backed disks.
|
||||
- Default-deny networking + explicit allow rules + egress class (R-NET-*).
|
||||
- TLS + DNS + ingress if it exposes an HTTP(S) service (R-TLS/R-DNS).
|
||||
- Metrics/logs and alerting (R-OBS-*) — node-exporter-class metrics from inside the guest plus
|
||||
KubeVirt/virt metrics from the platform.
|
||||
- Backup/DR of its disks (R-STATE-03) and rebuildability (R-STATE-01): destroy the VM, keep the
|
||||
disk PVC, recreate from the declaration.
|
||||
|
||||
## Update policy for VMs (R-UPD-*)
|
||||
|
||||
VM "updates" differ from container image tag bumps:
|
||||
- **Guest OS package updates** happen *inside* the guest on a schedule (unattended-upgrades / dnf-
|
||||
automatic style), configured via the VM's cloud-init/declaration — no human action.
|
||||
- **VM image/template updates** (new base image) are handled as declarative changes to the VM
|
||||
definition; where a golden-image pipeline exists, new images can be promoted automatically per
|
||||
policy.
|
||||
- The requirement "latest stable without human/AI/GitOps change" is met for guest packages via in-
|
||||
guest automation; base-image rebuilds follow the same automated-promotion philosophy as containers
|
||||
where feasible. Record any per-VM exceptions explicitly.
|
||||
|
||||
## When to choose VM vs container
|
||||
|
||||
Prefer containers (PaaS) by default. Choose a KubeVirt VM (IaaS) when:
|
||||
- The software isn't reasonably containerizable (kernel modules, full desktop, licensing tied to a
|
||||
VM, appliance images).
|
||||
- Lifting an existing VM is faster/safer than re-platforming.
|
||||
- Strong OS-level isolation is required beyond namespace/network policy.
|
||||
|
||||
For the initial catalog, all 13 named services are container-first; KubeVirt is the escape hatch and
|
||||
must be **proven** in dev/test with at least one representative VM workload so the IaaS path is real,
|
||||
not theoretical.
|
||||
|
||||
## Prerequisites & constraints
|
||||
- Nodes must expose virtualization extensions (nested virt in dev/test KVMs; native on prod mini-
|
||||
PCs) — captured in the node baseline ([02](02-environments-and-provisioning.md)).
|
||||
- Live migration across 3 converged nodes is a nice-to-have, not a requirement; document whether it's
|
||||
enabled.
|
||||
- Resource pressure: VMs are heavier than pods — capacity planning must account for them,
|
||||
particularly on the 3-node prod cluster.
|
||||
|
||||
## Acceptance gates
|
||||
- Create, start, stop, and destroy a VM entirely through Kubernetes YAML + `kubectl`/`virtctl`.
|
||||
- The VM's disk is Longhorn-backed; destroy/rebuild preserves data (R-STATE-01).
|
||||
- Default-deny applies: the VM can reach only what its allow rules permit.
|
||||
- An `egress: vpn` VM egresses via the VPN with a working kill-switch (R-NET-06).
|
||||
- Guest OS auto-updates on schedule with no manual step (R-UPD-*).
|
||||
@@ -0,0 +1,83 @@
|
||||
# 07 — Security & Secrets
|
||||
|
||||
Serves: **R-SEC-01/02/03**, supports all `R-NET-*` and `R-TLS-*`. Related:
|
||||
[ADR-0007 (SOPS+age)](../decisions/adr-0007-secrets-sops-age.md).
|
||||
|
||||
## Security posture
|
||||
|
||||
Defense in depth with declarative, auditable controls:
|
||||
|
||||
- **Network**: default-deny east-west, explicit port-level allow, egress-class routing, VPN kill-
|
||||
switch ([03](03-networking-isolation-and-egress.md)). This is the primary isolation control.
|
||||
- **Identity/least privilege**: each workload runs with a minimal ServiceAccount and only the RBAC
|
||||
it needs; no workload gets cluster-admin. VMs and containers alike get least network reach
|
||||
(R-SEC-02).
|
||||
- **Secrets**: encrypted at rest in git; decrypted only in-cluster (R-SEC-01).
|
||||
- **TLS**: trusted, auto-managed certificates end to end ([08](08-tls-and-certificates.md), R-TLS-*).
|
||||
- **Supply chain**: pin images by digest where automation allows; prefer known-good upstreams;
|
||||
updates flow through policy, not ad-hoc pulls ([11](11-automated-updates.md)).
|
||||
- **Policy enforcement (admission)**: **Kyverno** runs in-cluster as an admission controller and
|
||||
rejects non-conforming resources at reconcile time — e.g. a namespace missing its default-deny
|
||||
baseline, a plaintext Secret, a workload without a required egress-class label. The *same* policies
|
||||
gate CI ([14](14-testing-and-conformance.md), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md)),
|
||||
so invariants are enforced both before merge and at admission (defense in depth). Enforcement
|
||||
complements — never replaces — the runtime behavior tests.
|
||||
- **Auditability**: isolation, egress path, cert validity, and backup integrity are all
|
||||
demonstrable (R-SEC-03) via Hubble, cert status, and DR drills.
|
||||
|
||||
## Secrets management (R-SEC-01)
|
||||
|
||||
**SOPS + age**, native to Flux:
|
||||
- Secrets are committed to git **encrypted** with `age` recipients; Flux decrypts them at
|
||||
reconciliation using an in-cluster age key. Plaintext never enters git.
|
||||
- The age private key is bootstrapped into the cluster out-of-band (documented bootstrap step, one
|
||||
of the few sanctioned non-GitOps actions) and stored as a Kubernetes Secret Flux can read.
|
||||
- **Never** print decrypted secret material into logs, PR text, or agent output (see
|
||||
[AGENTS.md §5](../AGENTS.md)).
|
||||
|
||||
### Secret categories to manage
|
||||
- VPN credentials (WireGuard keys / OpenVPN creds) for the egress model.
|
||||
- DNS provider API token (Cloudflare) for ACME DNS-01 + ExternalDNS.
|
||||
- SMTP credentials for Alertmanager email.
|
||||
- Service credentials (Nextcloud, Gitea admin, UniFi, Semaphore, Kasm, DB passwords).
|
||||
- Off-cluster backup target credentials (S3/NFS) for Velero/Longhorn.
|
||||
- Grafana admin, and any API tokens the C2 UI needs.
|
||||
|
||||
### Rotation
|
||||
- Rotating a secret = re-encrypting with SOPS and committing; Flux rolls it out. Where a secret
|
||||
change requires a workload restart, that is triggered by the reconcile (e.g. checksum annotation).
|
||||
- VPN keys, DNS tokens, and SMTP creds SHOULD have a documented rotation procedure.
|
||||
|
||||
## Alternatives considered (see ADR-0007)
|
||||
- **Sealed Secrets** — viable; SOPS chosen for Flux-native decryption and multi-recipient/age
|
||||
simplicity and readable diffs.
|
||||
- **External Secrets Operator + external vault** — heavier; deferred until a real external secret
|
||||
store exists. May be revisited via ADR.
|
||||
|
||||
## RBAC & tenancy
|
||||
- Single operator + platform automation; no multi-tenant RBAC now (out of scope, [00](00-vision-and-scope.md)).
|
||||
- Platform controllers (Flux, cert-manager, ExternalDNS, Velero, KubeVirt) get scoped permissions;
|
||||
application workloads get near-zero cluster permissions.
|
||||
- The C2 UI's operational actions run through a constrained, audited path — not a shared admin
|
||||
kubeconfig handed to a browser.
|
||||
|
||||
## Node & host security
|
||||
- Nodes baselined via cloud-init/Ansible ([02](02-environments-and-provisioning.md)); SSH key-only,
|
||||
minimal packages, required sysctls only.
|
||||
- Host-level changes are IaC, not hand edits (drift + auditability).
|
||||
|
||||
## Auditability requirements (R-SEC-03) — what an operator must be able to prove
|
||||
- **Isolation**: "service X cannot reach service Y" — via Hubble flow logs + a denied-probe test.
|
||||
- **Egress path**: "service X's public IP is the VPN's" — via an egress IP check.
|
||||
- **Kill-switch**: "with VPN down, X leaks nothing" — via a fault-injection test.
|
||||
- **TLS**: "every endpoint serves a valid, non-self-signed, unexpired cert" — via cert status +
|
||||
external check.
|
||||
- **Backups**: "irreplaceable data is restorable" — via a DR drill.
|
||||
|
||||
These proofs are part of the platform acceptance suite, not one-time checks.
|
||||
|
||||
## Acceptance gates
|
||||
- No plaintext secret exists anywhere in git history (scanned).
|
||||
- Flux decrypts SOPS secrets and workloads consume them without manual steps.
|
||||
- Least-privilege RBAC verified (no workload has broad cluster rights).
|
||||
- The five auditability proofs above can be produced on demand.
|
||||
@@ -0,0 +1,61 @@
|
||||
# 08 — TLS & Certificates
|
||||
|
||||
Serves: **R-TLS-01/02/03**, **R-DNS-01**. Related: [ADR-0006 (cert-manager + Let's Encrypt)](../decisions/adr-0006-tls-cert-manager-letsencrypt.md).
|
||||
|
||||
## Requirement recap
|
||||
|
||||
Every externally reachable service presents TLS from a **credible, trusted, non-self-signed**
|
||||
authority, with certificates **auto issued, renewed, and revoked** — including for **internal-only**
|
||||
services (R-TLS-01/02/03).
|
||||
|
||||
## Solution: cert-manager + Let's Encrypt via DNS-01
|
||||
|
||||
- **cert-manager** runs in the cluster and manages the full certificate lifecycle.
|
||||
- **Let's Encrypt** is the trusted, non-self-signed CA (publicly trusted; free; automatable) —
|
||||
satisfies R-TLS-01.
|
||||
- **ACME DNS-01 challenge** against the `keithjackson.dev` DNS zone (Cloudflare or equivalent, per
|
||||
OQ-4). DNS-01 is the key choice because it validates domain control **without exposing the service
|
||||
publicly** — so internal-only services get real, trusted certs (R-TLS-03), and **wildcard**
|
||||
certificates (`*.home.keithjackson.dev`) are possible.
|
||||
|
||||
## Issuance model
|
||||
|
||||
- A `ClusterIssuer` for Let's Encrypt production (and one for staging, used during bring-up to avoid
|
||||
rate limits).
|
||||
- Certificates are requested declaratively — either a **wildcard** `*.home.keithjackson.dev` cert
|
||||
shared by the Gateway, or **per-service** certs, decided per ADR-0006. Recommendation: a wildcard
|
||||
cert for `*.home.keithjackson.dev` terminated at the Gateway simplifies onboarding (a new service
|
||||
needs no new cert), with per-service certs available where isolation warrants.
|
||||
- The DNS provider API token is a SOPS-encrypted secret ([07](07-security-and-secrets.md)).
|
||||
|
||||
## Renewal & revocation (R-TLS-02)
|
||||
|
||||
- cert-manager renews well before expiry automatically; no human action.
|
||||
- Revocation: removing/rotating a certificate is a declarative change; cert-manager handles
|
||||
issuance of replacements. Expiry monitoring is an alert condition ([10](10-observability-and-alerting.md), R-OBS-04).
|
||||
|
||||
## Where TLS terminates
|
||||
|
||||
- **North-south**: at the **Gateway API** listener (implemented by Cilium, [09](09-dns-and-ingress.md)),
|
||||
using the cert-manager-issued cert. Public services are reached via the edge router forwarding to
|
||||
the Gateway; internal services resolve to the Gateway on the LAN.
|
||||
- Internal service-to-service traffic relies primarily on network policy for security; mTLS between
|
||||
workloads is **optional** future hardening (could be added via Cilium/service mesh) and is not a
|
||||
launch requirement — record as a possible future ADR.
|
||||
|
||||
## Trust chain for internal clients
|
||||
|
||||
Because certs come from Let's Encrypt (publicly trusted), internal clients trust them with no custom
|
||||
CA distribution. Split-horizon DNS ([09](09-dns-and-ingress.md)) resolves
|
||||
`<service>.home.keithjackson.dev` to the internal Gateway while the same trusted cert is served.
|
||||
|
||||
## Acceptance gates
|
||||
- `https://<service>.home.keithjackson.dev` serves a valid, non-self-signed, unexpired certificate
|
||||
(verified by an external TLS checker), including for internal-only services.
|
||||
- Certificates renew automatically ahead of expiry with no manual step (observed across a renewal).
|
||||
- Certificate expiry and issuance failures raise alerts.
|
||||
- Onboarding a new service requires no manual certificate step.
|
||||
|
||||
## Open questions
|
||||
- **OQ-4** — confirm DNS provider (Cloudflare assumed) and an API token scoped for DNS-01.
|
||||
- Wildcard vs. per-service certs as the default (pinned in ADR-0006).
|
||||
@@ -0,0 +1,71 @@
|
||||
# 09 — DNS & Ingress
|
||||
|
||||
Serves: **R-DNS-01/02/03**, **R-C2-01**, works with **R-TLS-***. Related:
|
||||
[ADR-0002 (Cilium/Gateway API)](../decisions/adr-0002-cni-cilium.md).
|
||||
|
||||
## Naming model
|
||||
|
||||
- All services live at `<service>.home.keithjackson.dev` (e.g. `plex.home.keithjackson.dev`).
|
||||
- The **root** `home.keithjackson.dev` serves the **Command & Control UI** ([12](12-command-center-ui.md)).
|
||||
- Public apex `*.keithjackson.dev` resolves to the network's public IP; `*.home.keithjackson.dev`
|
||||
is the platform's zone.
|
||||
|
||||
## DNS automation (R-DNS-02)
|
||||
|
||||
- **ExternalDNS** watches Gateway/HTTPRoute (and Service) resources and manages records in the
|
||||
authoritative DNS provider (Cloudflare, per OQ-4) automatically — a new service's DNS appears from
|
||||
its declared intent, no manual record editing.
|
||||
- Records for `*.home.keithjackson.dev` are created/updated/removed as services come and go.
|
||||
|
||||
## Split-horizon resolution (R-DNS-03)
|
||||
|
||||
Internal and external answers differ:
|
||||
- **External** (`*.home.keithjackson.dev` from the internet): resolves to the public IP; reaches
|
||||
the edge router, which forwards `public`-class services to the in-cluster Gateway.
|
||||
- **Internal** (LAN clients): resolves `<service>.home.keithjackson.dev` to the **internal Gateway
|
||||
address** directly, avoiding hairpinning through the public IP.
|
||||
- **PiHole** (itself an onboarded service) provides internal LAN DNS and the split-horizon answers
|
||||
for `.home.keithjackson.dev`; the trusted Let's Encrypt cert is served either way ([08](08-tls-and-certificates.md)).
|
||||
|
||||
> DNS is thus both a platform capability *and* a hosted service. Bootstrapping order matters: the
|
||||
> platform needs working DNS before PiHole is fully managed — handled by the bring-up sequence in
|
||||
> [`roadmap/phases.md`](../roadmap/phases.md).
|
||||
|
||||
## Ingress (north-south) via Gateway API
|
||||
|
||||
- **Gateway API implemented by Cilium** is the ingress layer (no separate ingress controller;
|
||||
reuses the CNI, [ADR-0002](../decisions/adr-0002-cni-cilium.md)).
|
||||
- A shared **Gateway** defines HTTPS listeners on `*.home.keithjackson.dev` terminating TLS with the
|
||||
cert-manager cert ([08](08-tls-and-certificates.md)).
|
||||
- Each service contributes an **HTTPRoute** mapping its hostname to its Service — rendered from the
|
||||
ServiceDefinition, not hand-authored.
|
||||
- **Exposure class** is declared per service:
|
||||
- `internal` — reachable only on the LAN (no edge port-forward; internal DNS only).
|
||||
- `public` — additionally reachable from the internet via the edge router forwarding 80/443 to the
|
||||
Gateway.
|
||||
|
||||
## Edge integration
|
||||
|
||||
- The edge router/firewall forwards inbound 80/443 for `public` services to the Gateway's address
|
||||
and applies multi-WAN inbound rules as needed (OQ-1).
|
||||
- `internal` services are never port-forwarded; they rely on internal DNS + Gateway on the LAN.
|
||||
- A stable ingress VIP for the Gateway is provided by the cluster's LB strategy (Cilium
|
||||
L2 announcement / BGP / a chosen LB); pin the mechanism in the networking bring-up.
|
||||
|
||||
## Interaction with default-deny
|
||||
|
||||
Ingress opens **north-south** paths to the specific service; it does **not** relax **east-west**
|
||||
default-deny ([03](03-networking-isolation-and-egress.md)). A service being publicly reachable does
|
||||
not let other in-cluster services reach it unless an explicit allow rule exists.
|
||||
|
||||
## Acceptance gates
|
||||
- A new service's `<service>.home.keithjackson.dev` record appears automatically and routes to it.
|
||||
- `internal` services are unreachable from the public internet; `public` services are reachable and
|
||||
TLS-valid.
|
||||
- Internal clients resolve to the internal Gateway (no public hairpin) and get a trusted cert.
|
||||
- The root `home.keithjackson.dev` serves the C2 UI.
|
||||
|
||||
## Open questions
|
||||
- **OQ-1/OQ-4** — edge router product + DNS provider API (affects ExternalDNS provider config and
|
||||
inbound WAN rules).
|
||||
- Gateway LB/VIP mechanism (Cilium L2 vs BGP) — pin during networking bring-up.
|
||||
@@ -0,0 +1,69 @@
|
||||
# 10 — Observability & Alerting
|
||||
|
||||
Serves: **R-OBS-01/02/03/04**, supports **R-C2-02**, **R-SEC-03**.
|
||||
|
||||
## Stack
|
||||
|
||||
- **kube-prometheus-stack** — Prometheus (metrics), Alertmanager (routing/notify), Grafana
|
||||
(dashboards), and the standard Kubernetes exporters/rules.
|
||||
- **Loki** — log aggregation.
|
||||
- **Grafana Alloy** (or Promtail-class agent) — ships node/pod/VM logs to Loki.
|
||||
- **Hubble** — network flow visibility from Cilium ([03](03-networking-isolation-and-egress.md)).
|
||||
- **Alertmanager → SMTP** — issues reported by **email** (R-OBS-02).
|
||||
- **Grafana** — dashboards for services and platform health (R-OBS-03).
|
||||
|
||||
## What is monitored (R-OBS-01/04)
|
||||
|
||||
Application services:
|
||||
- Up/health, resource use, request/error signals where exposed, and per-service SLIs declared in the
|
||||
ServiceDefinition (a service can declare the alerts it cares about).
|
||||
|
||||
Platform health (R-OBS-04) — first-class alerts:
|
||||
- **Nodes**: down/unreachable, resource pressure, disk fill.
|
||||
- **Storage (Longhorn)**: degraded/faulted volumes, replica rebuilds, capacity thresholds.
|
||||
- **GitOps (Flux)**: reconciliation failures, drift, suspended resources.
|
||||
- **Certificates (cert-manager)**: upcoming expiry, issuance failures.
|
||||
- **Networking**: VPN tunnel down, egress-gateway unhealthy, spikes in denied flows (Hubble),
|
||||
NetworkPolicy reconcile errors.
|
||||
- **Backups (Velero/Longhorn)**: backup job failure, missed schedule, restore-test failures.
|
||||
- **KubeVirt**: VM crash-loops / failed launches.
|
||||
|
||||
## Alert delivery (R-OBS-02)
|
||||
|
||||
- Alertmanager routes by severity to **email** (SMTP creds are SOPS-encrypted, [07](07-security-and-secrets.md)).
|
||||
- Severity model: `critical` (page-worthy: data risk, outage, VPN leak potential, cert about to
|
||||
expire) vs `warning` (degraded, capacity, needs attention). Both go to email; only routing/subject
|
||||
differs.
|
||||
- Alerts include enough context (runbook link, affected object) to act without spelunking.
|
||||
|
||||
## Dashboards (R-OBS-03)
|
||||
|
||||
Grafana dashboards, surfaced in the C2 UI ([12](12-command-center-ui.md)):
|
||||
- **Platform overview**: nodes, cluster capacity, Flux status, cert expiry, backup status.
|
||||
- **Storage**: Longhorn volume health, capacity, replica state.
|
||||
- **Network**: Hubble flows, egress/VPN health, denied-flow trends.
|
||||
- **Per-service**: one row/dashboard per catalog service with its declared SLIs.
|
||||
|
||||
## Observability as onboarding output
|
||||
|
||||
The ServiceDefinition ([13](13-service-catalog.md)) declares observability expectations; the
|
||||
platform renders **ServiceMonitors/PodMonitors**, log scraping, dashboards, and alert rules from it.
|
||||
A newly onboarded service is monitored **by default**, not as a follow-up task (part of the golden
|
||||
path, [01](01-architecture-overview.md)).
|
||||
|
||||
## Retention & sizing
|
||||
- Metrics/log retention set per environment overlay (dev shorter, prod longer). Loki retention
|
||||
requires a compactor with retention enabled — do not rely on a period setting alone.
|
||||
- Monitoring storage lives on Longhorn; ensure its own capacity is alerted on (don't let the monitor
|
||||
blind itself).
|
||||
|
||||
## Auditability tie-in (R-SEC-03)
|
||||
Hubble + Prometheus + cert status provide the evidence for the isolation/egress/TLS/backup proofs
|
||||
required by [07](07-security-and-secrets.md).
|
||||
|
||||
## Acceptance gates
|
||||
- Every catalog service emits metrics + logs and has at least one meaningful alert that fires
|
||||
correctly in a fault test.
|
||||
- Platform-health alerts (node down, Longhorn degraded, Flux failure, cert expiry, VPN down, backup
|
||||
failure) each fire and email correctly in an injected-fault test.
|
||||
- Grafana dashboards render for platform and each service; the C2 UI links them.
|
||||
@@ -0,0 +1,76 @@
|
||||
# 11 — Automated Updates
|
||||
|
||||
Serves: **R-UPD-01/02/03**. Related: [ADR-0009](../decisions/adr-0009-automated-image-updates.md).
|
||||
|
||||
## The requirement, precisely
|
||||
|
||||
> Services must update to latest **stable** versions **on a schedule**, with **no human, no AI, and
|
||||
> no hand-authored GitOps change**. (R-UPD-01/02)
|
||||
|
||||
This has a subtle tension with GitOps (where "the desired state is what's in git"). We resolve it
|
||||
explicitly below.
|
||||
|
||||
## Resolving "no GitOps change" vs. GitOps
|
||||
|
||||
There are two interpretations; we honor the operator's intent (hands-off), and ADR-0009 records the
|
||||
mechanism:
|
||||
|
||||
- **A human/AI authoring a version bump = prohibited.** No person and no chatbot edits a tag.
|
||||
- **An automation controller advancing a version under a policy declared once = permitted**, because
|
||||
the *operator's* declared intent (the update policy / semver range) is the stable, human-authored
|
||||
thing; the controller merely resolves it. Two mechanisms, chosen by artifact type:
|
||||
|
||||
### 1. Container image tags → Flux image-update-automation
|
||||
- Per service, an **ImagePolicy** declares the acceptable range/channel (e.g. semver `>=1.0.0`
|
||||
minor/patch, or a `stable` tag pattern). Flux's image-reflector scans the registry; image-
|
||||
automation advances the running version automatically **on its scan interval** (the schedule).
|
||||
- Where automation writes the resolved version back to git, that commit is machine-authored under
|
||||
the declared policy — no human/AI touches it (satisfies R-UPD-02 in spirit and practice).
|
||||
- Alternatively, digest-tracking of a mutable stable tag avoids per-update commits entirely. ADR-0009
|
||||
picks the default and the per-service escape hatch.
|
||||
|
||||
### 2. Helm charts / manifests → Renovate
|
||||
- **Renovate** watches chart and dependency versions and opens PRs; allowed update classes
|
||||
**auto-merge** on green CI. This keeps platform components current with no human authoring.
|
||||
- Chart *major* bumps (potentially breaking) can require review — configurable per policy.
|
||||
|
||||
### 3. VM guests → in-guest automation
|
||||
- KubeVirt guests auto-update OS packages inside the guest on a schedule (unattended-upgrades / dnf-
|
||||
automatic) declared in the VM's cloud-init ([06](06-vm-workloads-kubevirt.md)). Base-image
|
||||
promotion follows the same automated philosophy where a golden-image pipeline exists.
|
||||
|
||||
## Per-service policy (R-UPD-03)
|
||||
|
||||
Each ServiceDefinition ([13](13-service-catalog.md)) declares an **update policy**:
|
||||
- **channel/range**: which versions are "stable" and acceptable (semver range, tag regex).
|
||||
- **cadence**: how often to check/apply (the schedule).
|
||||
- **hold/pin**: temporarily freeze a service (e.g. a known-bad upstream release).
|
||||
- **rollout & health**: how the new version rolls out and what health gates it.
|
||||
- **auto-rollback**: revert on failed health so a bad upstream release self-heals.
|
||||
|
||||
## Safety rails (so hands-off ≠ reckless)
|
||||
|
||||
- **Stable only**: never track pre-release/edge tags unless explicitly opted in per service.
|
||||
- **Health-gated rollout**: a failing update is caught by health checks and rolled back (R-UPD-03),
|
||||
and raises an alert ([10](10-observability-and-alerting.md)).
|
||||
- **Backups precede risk**: state is on Longhorn with backups, so a bad update never risks data
|
||||
([04](04-storage-and-state.md)).
|
||||
- **Observability**: update events (image advanced, chart merged, rollback) are visible and, on
|
||||
failure, alert by email.
|
||||
- **Pin escape hatch**: an operator can pin a service via git if an upstream release misbehaves —
|
||||
the one sanctioned human touch, and it *stops* automation rather than performing an update.
|
||||
|
||||
## Interaction with default-deny & egress
|
||||
Updates change image versions, not network posture. A VPN-class service stays VPN-class across
|
||||
updates; policies are reconciled from the (unchanged) ServiceDefinition.
|
||||
|
||||
## Acceptance gates
|
||||
- A service advances to a newer stable version automatically, on schedule, with no human/AI edit.
|
||||
- A deliberately bad update is caught by health checks, auto-rolled-back, and alerts by email.
|
||||
- A pinned service does **not** advance.
|
||||
- Platform components (charts) stay current via auto-merged Renovate PRs.
|
||||
|
||||
## Open questions
|
||||
- Exact default in ADR-0009: write-back commits vs. digest-tracking mutable tags (trade readable
|
||||
history vs. zero-commit updates).
|
||||
- Registry rate-limit / credential handling for image scanning at cadence.
|
||||
@@ -0,0 +1,64 @@
|
||||
# 12 — Command & Control (C2) Center UI
|
||||
|
||||
Serves: **R-C2-01/02**, surfaces **R-OBS-***, **R-GITOPS-***.
|
||||
|
||||
## Purpose
|
||||
|
||||
The root `home.keithjackson.dev` is the operator's front door: a single UI that shows what's running
|
||||
in the cluster, links to every service, and surfaces platform health and safe operational actions
|
||||
(R-C2-01/02).
|
||||
|
||||
## Composition (assemble, don't reinvent)
|
||||
|
||||
The C2 center is a curated composition of best-in-class UIs behind the root domain, not a bespoke
|
||||
app to build from scratch:
|
||||
|
||||
- **Homepage (gethomepage.dev)** — the landing dashboard at the root: a live catalog of services
|
||||
with links, status pings, and per-service widgets (many of the catalog services have native
|
||||
Homepage widgets: Plex, Sonarr, Radarr, qBittorrent, PiHole, Gitea, etc.). This is the primary
|
||||
R-C2-01 surface.
|
||||
- **Grafana** — embedded platform + per-service dashboards (R-OBS-03) for health at a glance.
|
||||
- **Headlamp** (or a comparable Kubernetes web UI) — a controlled Kubernetes console for viewing
|
||||
workloads/VMs and performing scoped operational actions (restart, scale, inspect) — the R-C2-02
|
||||
"controlled operational actions where safe" surface.
|
||||
- **Flux status** — GitOps reconciliation state (via Headlamp's Flux view, the Flux UI, or a
|
||||
Homepage widget) so the operator sees drift/errors.
|
||||
- **Hubble UI** (optional) — network flow visualization for isolation troubleshooting.
|
||||
|
||||
Each sub-UI is itself an onboarded, TLS-protected, access-controlled service; the root aggregates and
|
||||
links them. ADR-worthy only if a custom portal is later preferred over Homepage.
|
||||
|
||||
## Data the C2 surfaces (R-C2-02)
|
||||
|
||||
- **Service catalog**: every running service, its URL, its health, its egress class, its exposure.
|
||||
- **Platform health**: node status, Longhorn capacity/health, Flux reconciliation, cert expiry,
|
||||
backup status, VPN tunnel state — sourced from the observability stack ([10](10-observability-and-alerting.md)).
|
||||
- **Alerts**: current firing alerts (mirrors what email delivers).
|
||||
|
||||
## Actions (safe, audited)
|
||||
|
||||
Operational actions exposed through the C2 must be **scoped and audited**, never a raw cluster-admin
|
||||
kubeconfig in a browser:
|
||||
- Restart/scale a workload, start/stop a KubeVirt VM, trigger a Flux reconcile, acknowledge an
|
||||
alert.
|
||||
- Destructive actions (delete, offboard) require explicit confirmation and are logged.
|
||||
- Onboarding itself remains a **git PR** (GitOps), not a C2 button — the C2 may *link* to the
|
||||
catalog/PR flow but does not bypass GitOps ([05](05-gitops-and-delivery.md)).
|
||||
|
||||
## Access control
|
||||
|
||||
- The C2 root and its sub-UIs are `internal` exposure by default (LAN-only) unless the operator
|
||||
explicitly makes a component `public`.
|
||||
- Authentication in front of the C2 (and sensitive sub-UIs like Headlamp) — SSO/reverse-proxy auth —
|
||||
is required; the mechanism (e.g. an identity-aware proxy) is pinned during the C2 workstream.
|
||||
- Trusted TLS on the root and all sub-UIs ([08](08-tls-and-certificates.md)).
|
||||
|
||||
## Acceptance gates
|
||||
- `https://home.keithjackson.dev` serves the C2 landing UI over trusted TLS.
|
||||
- Every running service appears with a working link and live status.
|
||||
- Platform health (nodes, storage, Flux, certs, backups, VPN) is visible without leaving the C2.
|
||||
- Safe operational actions work and are access-controlled + audited; destructive actions confirm.
|
||||
|
||||
## Open questions
|
||||
- Auth mechanism for the C2 (identity-aware proxy vs per-app auth) — pin in the C2 workstream.
|
||||
- Custom portal vs. Homepage-as-root long term (ADR if we diverge from Homepage).
|
||||
@@ -0,0 +1,112 @@
|
||||
# 13 — Service Catalog & the ServiceDefinition
|
||||
|
||||
Serves: **R-GITOPS-02**, ties together **R-NET-***, **R-STATE-***, **R-TLS-***, **R-DNS-***,
|
||||
**R-OBS-***, **R-UPD-***. The catalog is the operator-facing onboarding surface.
|
||||
|
||||
## The ServiceDefinition abstraction
|
||||
|
||||
Onboarding a service is adding/editing one **ServiceDefinition** under `catalog/` in the
|
||||
implementation repo. It captures *intent*; the platform compiles it into concrete resources
|
||||
(Deployment or KubeVirt VM, PVCs, CiliumNetworkPolicies, egress routing, HTTPRoute, Certificate, DNS
|
||||
record, ServiceMonitor + dashboards + alerts, and update policy). Operators declare **what**;
|
||||
platform provides the **how**. This keeps every requirement wired from a single reviewable object.
|
||||
|
||||
### Fields (canonical schema — implementers render manifests from these)
|
||||
|
||||
```yaml
|
||||
name: <string> # DNS label → <name>.home.keithjackson.dev
|
||||
kind: container | vm # PaaS (default) or IaaS via KubeVirt (R-PLAT-02/03)
|
||||
source:
|
||||
image: <repo/image> # container: image ref
|
||||
# or vmTemplate: <image/source> # vm: CDI source image
|
||||
exposure: internal | public # LAN-only vs edge port-forwarded (R-DNS/R-NET)
|
||||
egress: internet | vpn # egress class (R-NET-05); vpn ⇒ kill-switch (R-NET-06)
|
||||
storage:
|
||||
- name: config
|
||||
size: <e.g. 5Gi>
|
||||
class: longhorn # replicated app/config state (R-STATE-02)
|
||||
- name: media # optional bulk tier (R-STATE-04)
|
||||
class: bulk
|
||||
dependencies: # explicit allow rules; everything else denied (R-NET-01/02)
|
||||
- to: <service>
|
||||
port: <n>
|
||||
protocol: tcp|udp
|
||||
ingressRoutes: # hostnames/paths to expose (rendered to HTTPRoute + TLS + DNS)
|
||||
- host: <name>.home.keithjackson.dev
|
||||
port: <servicePort>
|
||||
update:
|
||||
channel: stable # never pre-release unless opted in (R-UPD-01)
|
||||
range: <semver range | tag regex># acceptable versions
|
||||
cadence: <schedule> # how often to check/apply
|
||||
autoRollbackOnFailedHealth: true # R-UPD-03
|
||||
pin: <optional version> # operator freeze escape hatch
|
||||
observability:
|
||||
metrics: true
|
||||
slis: [ up, ... ] # per-service alert intents (R-OBS-01)
|
||||
backup:
|
||||
policy: <ref to backup schedule> # off-cluster (R-STATE-03); or 'reacquirable' for bulk media
|
||||
```
|
||||
|
||||
> The schema is the **contract** between the operator surface and the implementation. Changing it is
|
||||
> an interface change — coordinate per [`agents/coordination-protocol.md`](../agents/coordination-protocol.md).
|
||||
|
||||
## Default posture for every service (unless overridden)
|
||||
- `kind: container`, `exposure: internal`, `egress: internet`.
|
||||
- **Default-deny**: no `dependencies` ⇒ can reach only DNS ([03](03-networking-isolation-and-egress.md)).
|
||||
- Longhorn-backed config PVC; off-cluster backup; trusted TLS; auto-DNS; monitored; auto-updated on
|
||||
`stable`.
|
||||
|
||||
## The initial catalog (13 services)
|
||||
|
||||
Egress and dependencies below are **normative intent**; implementers render the exact policies.
|
||||
"Bulk" = large media tier ([04](04-storage-and-state.md)). All get trusted TLS + auto-DNS + monitoring
|
||||
+ auto-update unless noted.
|
||||
|
||||
| # | Service | kind | exposure | egress | Storage | Key dependencies (explicit allows) | Notes |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 1 | **plex** | container | public | **internet** | config: longhorn (small, replicated); media: **bulk** | — (clients reach it; Seerr → Plex) | Largest service; media on bulk tier, metadata DB on Longhorn ([04](04-storage-and-state.md)) |
|
||||
| 2 | **sonarr** | container | internal | **vpn** | config: longhorn; media/downloads: bulk | → qbittorrent, → jackett; ← seerr | VPN + kill-switch (R-NET-06) |
|
||||
| 3 | **radarr** | container | internal | **vpn** | config: longhorn; media/downloads: bulk | → qbittorrent, → jackett; ← seerr | VPN + kill-switch |
|
||||
| 4 | **jackett** | container | internal | **vpn** | config: longhorn | ← sonarr, ← radarr | VPN + kill-switch (indexer) |
|
||||
| 5 | **seerr** | container | public | internet | config: longhorn | → sonarr:8989, → radarr:7878, → plex:32400 | Request UI; talks to arr + Plex |
|
||||
| 6 | **pihole** | container | internal | internet | config: longhorn | (provides DNS to cluster/LAN) | Also a platform DNS dependency ([09](09-dns-and-ingress.md)) |
|
||||
| 7 | **nextcloud** | container | public | internet | data: longhorn (irreplaceable → backed up) | → its DB | File sync; DR-critical data |
|
||||
| 8 | **minecraft** | container | public | internet | world: longhorn (backed up) | — | Exposes game port via edge |
|
||||
| 9 | **unifi** | container | internal | internet | config: longhorn | — | UniFi network controller |
|
||||
| 10 | **semaphore** | container | internal | internet | config/db: longhorn | — | Ansible UI |
|
||||
| 11 | **kasm** | container | internal | internet | config: longhorn | — | Kasm Workspaces (may need more resources; container-first, VM if required) |
|
||||
| 12 | **gitea** | container | public | internet | repos/db: longhorn (irreplaceable → backed up) | — | Git hosting; DR-critical |
|
||||
| 13 | **qbittorrent** | container | internal | **vpn** | config: longhorn; downloads: **bulk** | ← sonarr, ← radarr | VPN + kill-switch; download client |
|
||||
|
||||
**VPN-class set (R-NET-05/06):** sonarr, radarr, jackett, qbittorrent — all egress via VPN with an
|
||||
enforced kill-switch. Their inter-dependencies (arr → qbittorrent/jackett) are **east-west allows**
|
||||
within the deny-by-default fabric; being VPN-class governs *egress*, not intra-cluster reachability.
|
||||
|
||||
**Dependency graph (explicit allows only; all else denied):**
|
||||
```
|
||||
seerr → sonarr, radarr, plex
|
||||
sonarr → qbittorrent, jackett
|
||||
radarr → qbittorrent, jackett
|
||||
(everything → DNS only, by default)
|
||||
```
|
||||
|
||||
**"…but not limited to":** the catalog is open-ended. New services follow the same ServiceDefinition
|
||||
and inherit all guarantees automatically — that is the point of the abstraction.
|
||||
|
||||
## Cross-cutting notes
|
||||
- **Plex media** is the R-STATE-04 case: keep the metadata DB on replicated Longhorn; keep the large
|
||||
library on the bulk tier; downloads stage on bulk and are handed off by the arr stack.
|
||||
- **DR-critical data** (Nextcloud, Gitea, Minecraft worlds, PiHole config, service DBs) MUST have
|
||||
off-cluster backups ([04](04-storage-and-state.md)); bulk media may be `reacquirable` per operator
|
||||
choice (OQ-2).
|
||||
- **Public exposure** (Plex, Seerr, Nextcloud, Minecraft, Gitea) requires edge port-forwarding
|
||||
([09](09-dns-and-ingress.md)); everything else stays `internal`.
|
||||
- **Kasm** is resource-heavy and may be a candidate for `kind: vm` if container mode proves
|
||||
insufficient — the ServiceDefinition makes that a one-field change.
|
||||
|
||||
## Acceptance gates
|
||||
- Each service is fully wired from its ServiceDefinition alone (compute, storage, policy, egress,
|
||||
TLS, DNS, monitoring, updates) — no manual steps beyond merge.
|
||||
- Default-deny holds; only listed dependencies work.
|
||||
- VPN-class services egress via VPN with a proven kill-switch.
|
||||
- Stateful services survive destroy/rebuild; DR-critical data restores from off-cluster backup.
|
||||
@@ -0,0 +1,168 @@
|
||||
# 14 — Testing & Conformance
|
||||
|
||||
Serves: **R-TEST-01..06**, and is the verification mechanism for **every other requirement**.
|
||||
Related: [ADR-0010](../decisions/adr-0010-conformance-testing.md).
|
||||
|
||||
> **This spec changes what "done" means.** Nothing is accepted on a manual demo, a screenshot, or
|
||||
> "it reconciled." Every requirement is proven by an **automated, reproducible, behavior-based test**
|
||||
> that a machine can run on demand and that is tagged with the requirement ID(s) it verifies.
|
||||
|
||||
## Principles
|
||||
|
||||
1. **Behavior-based (R-TEST-01).** Tests assert *observable system behavior* — Given a state, When
|
||||
an action, Then an outcome — not implementation internals. "A pod with no allow rules cannot reach
|
||||
Plex" is a behavior; "a CiliumNetworkPolicy object exists" is not (it may exist and still not
|
||||
enforce). Test the behavior.
|
||||
2. **Reproducible (R-TEST-02).** Deterministic, hermetic, version-pinned, single-command, and
|
||||
environment-agnostic. Any conforming cluster + a kubeconfig ⇒ the same pass/fail. No flakiness
|
||||
budget; eventual-consistency is handled with bounded polling, not sleeps-and-hope.
|
||||
3. **Conformance gates delivery (R-TEST-03).** The suite blocks merges (for touched requirements)
|
||||
and blocks phase exits (full phase suite). Green suite is the definition of progress.
|
||||
4. **Traceable (R-TEST-04).** Each test declares its `R-*` IDs; a generated coverage report maps
|
||||
every requirement to its verifying test(s). Uncovered requirements are visible gaps.
|
||||
5. **Fault-injection for safety-critical behavior (R-TEST-05).** Default-deny, VPN kill-switch,
|
||||
data-survival, DR restore, and auto-rollback are proven by *breaking things* and asserting
|
||||
fail-closed / no-data-loss — not by happy-path checks.
|
||||
6. **Specs are the fixed target (R-TEST-06).** Tests encode the spec's intent. Agents never weaken a
|
||||
test or a requirement to get green; a genuine inability to meet the spec is an escalation to the
|
||||
human, not a spec/test edit.
|
||||
|
||||
## Reproducibility contract (what "reproducible" concretely requires)
|
||||
|
||||
- **Single command** runs the suite (or a tagged subset) locally and in CI.
|
||||
- **Hermetic fixtures**: each test provisions its own namespace/workloads/probes and tears them down;
|
||||
it never depends on pre-existing mutable state or on another test's side effects.
|
||||
- **Deterministic**: fixed inputs, pinned tool/image versions, bounded retries with explicit
|
||||
timeouts for eventual consistency; no reliance on wall-clock timing or ordering.
|
||||
- **Isolated & parallel-safe**: unique namespaces/labels; tests can run concurrently without
|
||||
cross-talk.
|
||||
- **Machine-readable output**: JUnit (or equivalent) plus a requirement-coverage report; every
|
||||
result carries the `R-*` tags it verifies.
|
||||
- **Self-cleaning**: on pass or fail, fixtures are removed so a re-run starts clean (leaked fixtures
|
||||
are themselves a test failure).
|
||||
- **Versioned with the code**: tests live in the implementation repo next to what they verify and
|
||||
evolve only when the human-owned spec they encode changes.
|
||||
|
||||
## Test layers
|
||||
|
||||
| Layer | Runs | Verifies | Example |
|
||||
|---|---|---|---|
|
||||
| **L0 Static/policy** | Pre-apply, in CI | Manifests/ServiceDefinitions obey invariants | Every namespace has a default-deny baseline; no plaintext secrets; schema-valid ServiceDefinition |
|
||||
| **L1 Rendering** | CI, no cluster | ServiceDefinition → rendered resources match golden expectations | `egress: vpn` renders the kill-switched egress path |
|
||||
| **L2 Behavioral/integration** | Against dev/test cluster | Real runtime behavior | Default-deny probe, dependency allow/deny, egress-IP, TLS validity, DNS, PVC rebuild |
|
||||
| **L3 Fault-injection** | Against dev/test cluster | Fail-closed / no-data-loss under failure | VPN tunnel forced down ⇒ no leak; node killed ⇒ data survives; bad update ⇒ auto-rollback; DR restore |
|
||||
| **L4 Conformance suite** | CI + on-demand | Aggregate, requirement-tagged gate | The union of the above, reported per `R-*` |
|
||||
|
||||
## Behavior test catalog (normative — implementers author these as automated tests)
|
||||
|
||||
Each maps to requirement IDs; phrased as Given/When/Then intent.
|
||||
|
||||
- **Default-deny (R-NET-01):** *Given* a fresh workload with no allow rules, *When* it attempts any
|
||||
in-cluster connection except DNS, *Then* the connection fails; DNS succeeds.
|
||||
- **Explicit allow / deny-undeclared (R-NET-02):** *Given* declared `dependencies`, *When* the
|
||||
source reaches a declared dest port, *Then* it succeeds; *When* it reaches an **undeclared** dest,
|
||||
*Then* it fails.
|
||||
- **Egress class (R-NET-05):** *Given* `egress: internet` vs `egress: vpn` workloads, *When* each
|
||||
queries an external echo of its public IP, *Then* the internet one shows the home/WAN IP and the
|
||||
vpn one shows the VPN IP.
|
||||
- **VPN kill-switch (R-NET-06, L3):** *Given* a `vpn` workload with a healthy tunnel, *When* the
|
||||
tunnel is forced down, *Then* the workload has **zero** egress to the plain internet and
|
||||
`internet`-class workloads are unaffected; *When* the tunnel recovers, connectivity returns.
|
||||
- **Egress flip (R-NET-07):** *Given* a running service, *When* its `egress` class is changed in
|
||||
git and reconciled, *Then* its observed egress path changes with no workload rewrite.
|
||||
- **Data survival (R-STATE-01, L3):** *Given* a stateful service with written data, *When* the
|
||||
workload is destroyed and re-applied from git, *Then* the data is intact.
|
||||
- **Node loss (R-STATE-02, L3):** *Given* a 3-replica volume, *When* a node is removed, *Then* the
|
||||
volume stays available and the cluster reconverges.
|
||||
- **DR restore (R-STATE-03, L3):** *Given* an off-cluster backup, *When* a namespace + volumes are
|
||||
restored into a clean state, *Then* the service returns with its data.
|
||||
- **Trusted TLS (R-TLS-01/02):** *Given* any exposed endpoint (incl. `internal`), *When* its cert is
|
||||
inspected, *Then* it is valid, non-self-signed, unexpired; renewal occurs before expiry with no
|
||||
manual step.
|
||||
- **DNS/exposure (R-DNS-*):** *Given* a service, *When* resolved, *Then* the record exists and
|
||||
internal clients hit the internal Gateway; `internal` is unreachable publicly, `public` is
|
||||
reachable + TLS-valid.
|
||||
- **Hands-off update (R-UPD-01/02, L3):** *Given* an out-of-date service under a stable policy,
|
||||
*When* the scan interval elapses, *Then* it advances with no human/AI edit; *Given* a deliberately
|
||||
bad update, *Then* health-gating auto-rolls-back and an email alert fires; *Given* a pinned
|
||||
service, *Then* it does not advance.
|
||||
- **VM lifecycle (R-PLAT-03, L2/L3):** *Given* a `kind: vm` ServiceDefinition, *When* create/start/
|
||||
stop/destroy via `kubectl`/`virtctl`, *Then* it behaves; its disk survives rebuild; default-deny +
|
||||
egress class apply.
|
||||
- **Alerting (R-OBS-02/04, L3):** *Given* a monitored condition, *When* a fault is injected, *Then*
|
||||
an email alert is delivered and the dashboard reflects it.
|
||||
- **GitOps drift (R-GITOPS-03, L3):** *Given* a reconciled resource, *When* it is mutated live,
|
||||
*Then* Flux reverts it to the git-declared state.
|
||||
- **Onboarding completeness (WS-APPS):** *Given* only a merged ServiceDefinition, *Then* the service
|
||||
is fully wired (compute/storage/policy/egress/TLS/DNS/monitoring/update) with no manual step —
|
||||
asserted by the union of the relevant tests above against that service.
|
||||
|
||||
## Gating rules (R-TEST-03)
|
||||
|
||||
- **Per-change (merge gate):** the conformance subset tagged with the requirement IDs the change
|
||||
touches MUST pass before merge.
|
||||
- **Per-phase (exit gate):** the full conformance suite for the phase's requirements MUST pass
|
||||
before the phase's exit criteria are considered met ([`../roadmap/phases.md`](../roadmap/phases.md)).
|
||||
- **Regression:** the growing suite runs continuously against dev/test; a previously-green
|
||||
requirement going red blocks further delivery until fixed.
|
||||
- **Coverage:** a requirement with no passing test is **not** satisfied, regardless of implementation
|
||||
state.
|
||||
|
||||
## Tooling (pinned — [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md))
|
||||
|
||||
The approach ([ADR-0010](../decisions/adr-0010-conformance-testing.md)) is realized with two tools
|
||||
from the **Kyverno ecosystem**:
|
||||
|
||||
- **Kyverno** — **L0 static/policy gate _and_ in-cluster admission enforcer.** In CI, the Kyverno CLI
|
||||
(`kyverno apply` / `kyverno test`, incl. `kyverno-json` for ServiceDefinitions) validates manifests
|
||||
and rendered resources against platform invariants (default-deny baseline present, no plaintext
|
||||
secrets, required labels/egress-class, schema conformance). **The same policies also run in-cluster
|
||||
as an admission controller**, rejecting any non-conforming resource at reconcile time (defense in
|
||||
depth — a resource that somehow bypassed CI still cannot be admitted). Enforcement **never replaces
|
||||
behavior tests**: a policy admitting a manifest is not proof the runtime behaves (R-TEST-01), so
|
||||
L2/L3 Chainsaw tests remain mandatory. Policies are authored once and shared between the CLI gate
|
||||
and the admission controller.
|
||||
- **Chainsaw** — **L1 rendering, L2 behavioral, and L3 fault-injection.** Declarative steps
|
||||
(apply/assert/error, `script`/`command`, `try`/`catch`/`finally`), bounded polling with explicit
|
||||
timeouts, per-test namespace isolation, and automatic cleanup satisfy the reproducibility contract.
|
||||
Fault-injection (force the tunnel down, cordon/kill a node, inject a bad image, trigger a restore)
|
||||
is expressed as Chainsaw `script` steps, asserting fail-closed / no-data-loss via probe fixtures,
|
||||
with `finally` restoring state.
|
||||
|
||||
Probes are ephemeral job pods; egress-IP checks use an external echo; tests carry their `R-*` IDs
|
||||
(labels/annotations/naming) so the coverage report is generated automatically; both tools' output
|
||||
feeds JUnit + the requirement-coverage report and the CI merge / phase-exit gates. Tool versions are
|
||||
pinned for determinism.
|
||||
|
||||
### Where each layer runs
|
||||
- **L0 (Kyverno CLI) and L1 (rendering assertions)** run **in CI without a cluster** — pure manifest/
|
||||
ServiceDefinition validation; fast merge-gate feedback.
|
||||
- **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job** on the dev/test cluster.
|
||||
CI **triggers** the Job (and waits for it); the Job — not CI — holds cluster access, so CI never
|
||||
needs a broad kubeconfig. This keeps credentials in-cluster and least-privileged (R-SEC-02).
|
||||
- The Job runs under a **dedicated, scoped test ServiceAccount** whose RBAC grants exactly what the
|
||||
behavior/fault tests need (create/delete test namespaces + fixtures + probes, apply/patch
|
||||
NetworkPolicies, scale the VPN sidecar/gateway, cordon/drain a node, inject a bad image tag,
|
||||
trigger a Velero restore). This is a **privileged test identity** — it MUST be scoped, named, and
|
||||
documented, MUST NOT be reused by workloads, and its own reach is itself an audit item.
|
||||
- The Job writes **JUnit + coverage artifacts** to a known location (log stream and/or a results
|
||||
PVC/object) that CI collects to enforce the merge / phase-exit gates.
|
||||
- Fixtures are hermetic and self-cleaning (`finally` steps); a leaked fixture fails the run.
|
||||
|
||||
## Acceptance gates (for the testing capability itself)
|
||||
- A single command runs the suite and emits requirement-tagged JUnit + coverage.
|
||||
- Every `R-*` requirement maps to at least one passing behavior test (coverage report proves it).
|
||||
- Safety-critical requirements each have a **fault-injection** test that fails closed / loses no
|
||||
data.
|
||||
- Re-running the suite yields identical results (reproducibility) with no leaked fixtures.
|
||||
|
||||
## Open questions
|
||||
- Tool mix is **resolved**: Kyverno (L0) + Chainsaw (L1/L2/L3), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md).
|
||||
- Harness location is **resolved**: L2/L3 run as an **in-cluster Chainsaw Job** triggered by CI; CI
|
||||
holds no cluster credentials.
|
||||
- Kyverno-as-admission-enforcer is **resolved**: yes — the L0 policies also enforce at admission
|
||||
in-cluster (defense in depth), in addition to the CI gate.
|
||||
- Delegated to WS-TEST at implementation time (not human decisions): the **exact RBAC scope** of the
|
||||
test ServiceAccount (author it **least-privilege** for precisely the fault-injection actions it
|
||||
performs, [07](07-security-and-secrets.md)) and the **results-artifact mechanism** (log stream vs
|
||||
results PVC/object store). Both are implementer's discretion within the stated constraints.
|
||||
@@ -0,0 +1,108 @@
|
||||
# Roadmap — Delivery Phases
|
||||
|
||||
Delivery is phased. Each phase has an **entry gate** (what must be true to start), **exit criteria**
|
||||
(demonstrable outcomes tied to requirement IDs), and the **workstreams** ([workstreams.md](workstreams.md))
|
||||
that execute it. **Everything is proven in dev/test; prod is design-only until Phase 7.**
|
||||
|
||||
A phase's exit criteria are **passing automated conformance tests** (per [`../AGENTS.md` §7](../AGENTS.md)
|
||||
and [`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md)), not "manifests
|
||||
merged" and not manual demos. Every "**Prove:**" / "**demonstrate**" item below is an automated,
|
||||
reproducible, requirement-tagged behavior test committed with the work. A phase exits only when its
|
||||
full conformance subset is green. **The conformance harness (WS-TEST) is stood up in Phase 0, so
|
||||
every subsequent phase ships with tests, not after them.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 — Foundations & Repo Bootstrap
|
||||
**Entry:** specs (this repo) approved.
|
||||
**Do:**
|
||||
- Create the implementation repo structure ([`../docs/05-gitops-and-delivery.md`](../docs/05-gitops-and-delivery.md)).
|
||||
- Stand up CI (lint, schema validation, dry-run/build) for manifests and ServiceDefinitions.
|
||||
- Author the ServiceDefinition schema + renderer contract ([`../docs/13-service-catalog.md`](../docs/13-service-catalog.md)).
|
||||
- **Stand up the conformance harness (WS-TEST):** single-command runner, reproducibility contract,
|
||||
requirement-tagged JUnit + coverage report, L0 policy-as-code gate, and the CI merge gate
|
||||
([`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md), [ADR-0010](../decisions/adr-0010-conformance-testing.md)).
|
||||
**Exit:** implementation repo + CI exist; ServiceDefinition schema validated; **conformance harness
|
||||
runs green with the coverage report wired**; no cluster yet.
|
||||
**Serves:** R-GITOPS-02, R-ENV-02, R-TEST-01..04.
|
||||
|
||||
## Phase 1 — dev/test Cluster Bring-up (IaC)
|
||||
**Entry:** Phase 0 done.
|
||||
**Do:** OpenTofu libvirt node VMs + cloud-init baseline + Ansible k3s HA install with **Cilium**;
|
||||
disable bundled flannel/kube-proxy/Traefik; bootstrap **Flux**; bootstrap **SOPS age key**.
|
||||
**Exit:** 3-node k3s HA cluster with Cilium up; Flux reconciling from git; a dev node VM can be
|
||||
destroyed and rebuilt from IaC. **Prove:** node-loss re-provision.
|
||||
**Serves:** R-PLAT-01, R-ENV-01, R-GITOPS-01, ADR-0001/0002/0004/0007.
|
||||
|
||||
## Phase 2 — Platform Core: Network, Storage, TLS, DNS, Ingress
|
||||
**Entry:** Phase 1 done.
|
||||
**Do:** Longhorn (3-replica) + bulk-tier plan; default-deny network baseline + Hubble; Cilium
|
||||
Egress Gateway; cert-manager + Let's Encrypt (DNS-01, wildcard); ExternalDNS; Gateway API listeners;
|
||||
split-horizon DNS foundation.
|
||||
**Exit (demonstrate):**
|
||||
- Default-deny holds: a probe pod can reach only DNS (R-NET-01).
|
||||
- A PVC survives pod destroy/rebuild (R-STATE-01).
|
||||
- A test host serves a **trusted, non-self-signed** cert at `*.home.keithjackson.dev` (R-TLS-01/03).
|
||||
- Auto DNS record appears for a test route (R-DNS-02).
|
||||
**Serves:** R-NET-01/02/03, R-STATE-01/02, R-TLS-01/02/03, R-DNS-01/02/03, ADR-0002/0003/0006.
|
||||
|
||||
## Phase 3 — Egress, VPN & Multi-WAN
|
||||
**Entry:** Phase 2 done; OQ-1/OQ-3 answered enough to build.
|
||||
**Do:** implement egress classes; VPN egress (Gluetun sidecar default) with **kill-switch**;
|
||||
integrate edge-router multi-WAN split for the `internet` class.
|
||||
**Exit (demonstrate):**
|
||||
- `vpn` test workload egresses via VPN IP; `internet` workload via home/WAN IP (R-NET-05).
|
||||
- **Kill-switch:** tunnel forced down ⇒ `vpn` workload leaks nothing; `internet` unaffected (R-NET-06).
|
||||
- Flipping `egress` class in git re-routes without workload rewrite (R-NET-07).
|
||||
**Serves:** R-NET-04/05/06/07, ADR-0008.
|
||||
|
||||
## Phase 4 — Observability, Backup/DR & Automated Updates
|
||||
**Entry:** Phase 2 done (can overlap Phase 3).
|
||||
**Do:** kube-prometheus-stack + Loki + Alloy + Alertmanager(email); platform-health alerts; Velero +
|
||||
Longhorn off-cluster backup; Flux image-automation + Renovate; per-service update policy plumbing.
|
||||
**Exit (demonstrate):**
|
||||
- Injected fault emails an alert; dashboards render (R-OBS-01/02/03/04).
|
||||
- DR drill: restore a namespace + volumes from off-cluster backup (R-STATE-03).
|
||||
- A test service auto-advances a stable version on schedule with no human/AI edit; a bad update
|
||||
auto-rolls-back and alerts (R-UPD-01/02/03).
|
||||
**Serves:** R-OBS-*, R-STATE-03, R-UPD-*, ADR-0009/0003.
|
||||
|
||||
## Phase 5 — IaaS Proof (KubeVirt)
|
||||
**Entry:** Phases 2–3 done.
|
||||
**Do:** KubeVirt + CDI; run one representative application VM via ServiceDefinition `kind: vm`.
|
||||
**Exit (demonstrate):** create/start/stop/destroy a VM via `kubectl`/`virtctl`; Longhorn-backed disk
|
||||
survives rebuild; default-deny + `egress` apply to the VM; guest auto-updates.
|
||||
**Serves:** R-PLAT-03/04, R-STATE-01, R-NET-*, R-UPD-*, ADR-0005.
|
||||
|
||||
## Phase 6 — Service Catalog Onboarding + C2 UI
|
||||
**Entry:** Phases 2–4 done.
|
||||
**Do:** onboard the 13 services via ServiceDefinitions in dependency order (see workstream WS-APPS);
|
||||
stand up the C2 UI (Homepage root + Grafana + Headlamp + Flux status).
|
||||
**Exit (demonstrate):** each service fully wired from its ServiceDefinition (compute/storage/policy/
|
||||
egress/TLS/DNS/monitoring/updates) with default-deny + correct egress; the arr↔qbittorrent↔jackett↔
|
||||
seerr↔plex dependency graph works and nothing undeclared does; `home.keithjackson.dev` serves the C2
|
||||
with live links + health.
|
||||
**Serves:** R-C2-01/02, all app-facing requirements, [`../docs/13-service-catalog.md`](../docs/13-service-catalog.md).
|
||||
|
||||
## Phase 7 — Production Readiness (design → build)
|
||||
**Entry:** Phases 1–6 proven in dev/test; operator declares prod hardware ready.
|
||||
**Do:** finalize bare-metal provisioning checklist; apply the **prod overlay** (no base edits);
|
||||
full DR drill on prod; cutover plan.
|
||||
**Exit:** prod 3× mini-PC cluster runs the same config via the prod overlay; DR proven; monitoring +
|
||||
alerts live.
|
||||
**Serves:** R-ENV-02/03, R-PLAT-01.
|
||||
**Note:** Do not begin building on prod hardware before this phase is explicitly opened
|
||||
([`../AGENTS.md` §6](../AGENTS.md)).
|
||||
|
||||
---
|
||||
|
||||
## Dependency flow
|
||||
|
||||
```
|
||||
P0 → P1 → P2 ─┬→ P3 ─┐
|
||||
├→ P4 ──┼→ P6 → P7
|
||||
└───────┘ ↑
|
||||
P5 ───┘
|
||||
```
|
||||
|
||||
P3, P4, P5 can proceed in parallel once P2 lands; P6 needs P2–P4; P7 needs everything proven.
|
||||
@@ -0,0 +1,59 @@
|
||||
# Roadmap — Workstreams
|
||||
|
||||
A **workstream** is a durable area of ownership. Each maps to an agent role
|
||||
([`../agents/roles.md`](../agents/roles.md)), owns a set of specs, exposes **interfaces** other
|
||||
workstreams depend on, and delivers against requirement IDs. Interfaces are contracts — change them
|
||||
only via [`../agents/coordination-protocol.md`](../agents/coordination-protocol.md).
|
||||
|
||||
| ID | Workstream | Owns (specs) | Serves | Depends on |
|
||||
|---|---|---|---|---|
|
||||
| **WS-INFRA** | Infra & cluster bring-up | [02](../docs/02-environments-and-provisioning.md) | R-PLAT-01, R-ENV-* | — |
|
||||
| **WS-GITOPS** | GitOps, CI, catalog renderer | [05](../docs/05-gitops-and-delivery.md), [13](../docs/13-service-catalog.md) | R-GITOPS-*, R-UPD-02 | WS-INFRA |
|
||||
| **WS-NET** | Networking, isolation, egress, ingress | [03](../docs/03-networking-isolation-and-egress.md), [09](../docs/09-dns-and-ingress.md) | R-NET-*, R-DNS-* | WS-INFRA |
|
||||
| **WS-STORAGE** | Storage, state, backup/DR | [04](../docs/04-storage-and-state.md) | R-STATE-* | WS-INFRA |
|
||||
| **WS-SEC** | Secrets, TLS, RBAC, auditability | [07](../docs/07-security-and-secrets.md), [08](../docs/08-tls-and-certificates.md) | R-SEC-*, R-TLS-* | WS-INFRA, WS-NET |
|
||||
| **WS-OBS** | Observability & alerting | [10](../docs/10-observability-and-alerting.md) | R-OBS-* | WS-INFRA, WS-STORAGE |
|
||||
| **WS-UPDATE** | Automated updates | [11](../docs/11-automated-updates.md) | R-UPD-* | WS-GITOPS, WS-OBS |
|
||||
| **WS-VM** | KubeVirt IaaS | [06](../docs/06-vm-workloads-kubevirt.md) | R-PLAT-03 | WS-NET, WS-STORAGE |
|
||||
| **WS-APPS** | Service onboarding (13 services) | [13](../docs/13-service-catalog.md) | app requirements | WS-GITOPS, WS-NET, WS-STORAGE, WS-SEC, WS-OBS |
|
||||
| **WS-C2** | Command & Control UI | [12](../docs/12-command-center-ui.md) | R-C2-* | WS-OBS, WS-APPS, WS-SEC |
|
||||
| **WS-TEST** | Conformance testing (cross-cutting) | [14](../docs/14-testing-and-conformance.md) | R-TEST-* | WS-INFRA, WS-GITOPS |
|
||||
|
||||
## Key interfaces (contracts between workstreams)
|
||||
|
||||
- **ServiceDefinition schema** (WS-GITOPS owns) — the operator surface every app/VM is expressed in.
|
||||
WS-APPS, WS-VM consume it; WS-NET/STORAGE/SEC/OBS render pieces of it. **The most important
|
||||
contract in the system.**
|
||||
- **Network baseline & policy rendering** (WS-NET) — default-deny + how `dependencies` and `egress`
|
||||
compile to CiliumNetworkPolicies/egress routing. Consumed by WS-APPS/WS-VM.
|
||||
- **StorageClasses** (WS-STORAGE) — `longhorn` (replicated) and `bulk` (media). Consumed by everyone
|
||||
with state.
|
||||
- **Issuer + Gateway TLS** (WS-SEC + WS-NET) — wildcard cert + Gateway listeners that HTTPRoutes
|
||||
attach to.
|
||||
- **Observability contract** (WS-OBS) — how a service declares metrics/SLIs/alerts and gets
|
||||
dashboards. Consumed by WS-APPS/WS-VM.
|
||||
- **Update policy contract** (WS-UPDATE) — ImagePolicy/Renovate conventions a ServiceDefinition's
|
||||
`update` block maps to.
|
||||
- **Conformance harness + gate** (WS-TEST) — the reproducible test framework (**Kyverno** for L0
|
||||
policy, **Chainsaw** for L1/L2/L3 behavior + fault-injection, [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md)),
|
||||
requirement-coverage report, and CI/phase gates every workstream's tests plug into. Each workstream
|
||||
**authors its own behavior tests** against this harness ([14](../docs/14-testing-and-conformance.md)).
|
||||
|
||||
## Parallelization guidance
|
||||
- **WS-TEST stands up the conformance harness early (Phase 0–1)** so every other workstream lands
|
||||
with passing, reproducible tests from day one — testing is not a trailing phase.
|
||||
- After WS-INFRA delivers a cluster + Flux (Phase 1), WS-NET, WS-STORAGE, WS-SEC can proceed largely
|
||||
in parallel (Phase 2).
|
||||
- WS-OBS, WS-UPDATE, WS-VM layer on next (Phases 3–5), overlapping.
|
||||
- WS-APPS is gated on the platform contracts being stable; onboard services in the dependency order
|
||||
below.
|
||||
- WS-C2 is last-ish (Phase 6) since it aggregates everything.
|
||||
|
||||
## WS-APPS onboarding order (respect dependencies from [13](../docs/13-service-catalog.md))
|
||||
1. **pihole** (DNS underpinning) → 2. **plex** (largest; validates bulk tier + public exposure) →
|
||||
3. **qbittorrent, jackett** (VPN + kill-switch proof) → 4. **sonarr, radarr** (VPN + arr↔downloader
|
||||
allows) → 5. **seerr** (cross-service allows to arr + plex) → 6. **nextcloud, gitea** (DR-critical
|
||||
data + backups) → 7. **minecraft, unifi, semaphore, kasm** (remaining).
|
||||
|
||||
Each onboarding is a full acceptance-gate pass ([`../agents/definition-of-done.md`](../agents/definition-of-done.md)),
|
||||
not just a merged manifest.
|
||||
Reference in new issue
Block a user