Initial baseline: home platform (k3s + Longhorn) specification repository
Greenfield spec set for a GitOps-driven PaaS+IaaS home platform: vision & requirements (R-* IDs), architecture, 14 domain specs, 11 ADRs, roadmap (phases + workstreams), and AI-agent coordination (roles, protocol, definition-of-done, task board). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
37 files changed
+2830
No files matched your search
@@ -0,0 +1,189 @@
|
||||
# 00 — Vision & Scope
|
||||
|
||||
## Vision
|
||||
|
||||
Build a home platform that behaves like a small private cloud. An operator declares a service — a
|
||||
container app *or* a virtual machine — in git, and the platform provisions it, secures it, exposes
|
||||
it under a trusted TLS URL, routes its traffic on the correct WAN, protects its data, updates it
|
||||
automatically, and reports on its health. Onboarding a new service is a pull request against a
|
||||
catalog, not a weekend of manual work.
|
||||
|
||||
The platform is delivered in two stages:
|
||||
|
||||
- **dev/test** on local KVMs — the active build target, used to prove viability.
|
||||
- **prod** on 3 mini-PCs — the eventual home, designed for now but not built on yet.
|
||||
|
||||
## Personas
|
||||
|
||||
- **Operator (primary user)**: runs the home network; wants to onboard/offboard services with
|
||||
minimal ceremony and zero data loss.
|
||||
- **Platform (the system)**: reconciles declared state, enforces isolation and egress policy,
|
||||
renews certificates, updates workloads, and raises alerts.
|
||||
- **Implementation agents**: the AI teams building the platform from this spec.
|
||||
|
||||
## In scope
|
||||
|
||||
- A k3s-based platform with distributed storage (Longhorn) across 3 nodes.
|
||||
- Container workloads (PaaS) and VM workloads (IaaS) through one Kubernetes surface.
|
||||
- GitOps delivery, automated updates, trusted TLS, DNS, ingress.
|
||||
- Network isolation (default-deny east-west) and multi-WAN egress including VPN routing.
|
||||
- Observability with email + dashboard alerting.
|
||||
- A command-and-control web UI at the root of `home.keithjackson.dev`.
|
||||
- The initial service catalog (13 services, see [`13-service-catalog.md`](13-service-catalog.md)).
|
||||
|
||||
## Out of scope (for now)
|
||||
|
||||
- Building on production mini-PC hardware (design-only until the roadmap flips it to active).
|
||||
- Multi-tenant / multi-operator RBAC beyond a single operator + platform automation.
|
||||
- Public multi-user SaaS concerns (billing, per-tenant isolation beyond network policy).
|
||||
- Non-`keithjackson.dev` domains.
|
||||
|
||||
## The domain & DNS model
|
||||
|
||||
- Public apex `*.keithjackson.dev` is available and its records resolve to the network's public IP.
|
||||
- All platform services live under `*.home.keithjackson.dev`.
|
||||
- Each service is reachable at `<service>.home.keithjackson.dev` (e.g. `plex.home.keithjackson.dev`).
|
||||
- The **root** `home.keithjackson.dev` serves the **Command & Control (C2) center** UI.
|
||||
|
||||
## Requirements (stable IDs)
|
||||
|
||||
Requirement IDs are stable references used across ADRs, specs, the task board, and PRs. `MUST` is
|
||||
mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
|
||||
|
||||
### Platform & API surface — `R-PLAT-*`
|
||||
- **R-PLAT-01** — The platform MUST run on k3s across 3 nodes with production-grade HA control plane
|
||||
in prod; dev/test MAY run fewer nodes but MUST exercise the same multi-node topology.
|
||||
- **R-PLAT-02** — Container workloads MUST be onboarded and managed declaratively via the Kubernetes
|
||||
API, `kubectl`-class CLI, and YAML.
|
||||
- **R-PLAT-03** — Virtual-machine workloads MUST be created, managed, and destroyed through the
|
||||
*same* Kubernetes API/CLI/YAML surface as containers (no separate hypervisor management plane for
|
||||
application VMs).
|
||||
- **R-PLAT-04** — The platform MUST support create / destroy / rebuild of any workload on demand.
|
||||
|
||||
### GitOps & delivery — `R-GITOPS-*`
|
||||
- **R-GITOPS-01** — All application and platform desired state MUST be defined in git and reconciled
|
||||
automatically (pull-based GitOps).
|
||||
- **R-GITOPS-02** — Onboarding a new service MUST be expressible as a change to a versioned catalog
|
||||
in git.
|
||||
- **R-GITOPS-03** — The system MUST detect and correct drift between git and cluster state.
|
||||
|
||||
### Networking, isolation & egress — `R-NET-*`
|
||||
- **R-NET-01** — East-west traffic between services MUST be **denied by default**; communication is
|
||||
allowed only by explicit, port-level configuration (firewall-like).
|
||||
- **R-NET-02** — Allow rules MUST be declarative, reviewable in git, and scoped per source →
|
||||
destination → port/protocol.
|
||||
- **R-NET-03** — Networks MUST be configurable and isolatable (segments/namespaces with enforced
|
||||
boundaries).
|
||||
- **R-NET-04** — The platform MUST support **multiple WANs** with egress selection driven by routing
|
||||
rules.
|
||||
- **R-NET-05** — A service MUST be able to declare its **egress class**: `internet` (standard local
|
||||
network/internet) or `vpn` (routed through a custom VPN connection).
|
||||
- **R-NET-06** — VPN-class egress MUST enforce a **kill-switch**: if the VPN tunnel is down, the
|
||||
service MUST NOT leak traffic to the standard internet.
|
||||
- **R-NET-07** — Egress class MUST be a declarative property of the service, changeable via GitOps
|
||||
without re-architecting the workload.
|
||||
|
||||
### Storage & state — `R-STATE-*`
|
||||
- **R-STATE-01** — Stateful data MUST survive destroy/rebuild of the workload (PaaS and IaaS).
|
||||
- **R-STATE-02** — Persistent storage MUST be distributed/replicated across the 3 nodes (Longhorn)
|
||||
so a single node loss does not lose data.
|
||||
- **R-STATE-03** — The platform MUST provide scheduled backups to an **off-cluster** target and a
|
||||
tested restore path (disaster recovery).
|
||||
- **R-STATE-04** — Bulk media storage (Plex library) MUST have a capacity strategy distinct from
|
||||
replicated app-config storage (replicating multi-TB media 3× is out of scope; see ADR-0003).
|
||||
|
||||
### Automated updates — `R-UPD-*`
|
||||
- **R-UPD-01** — Services MUST update to the latest **stable** versions on a schedule.
|
||||
- **R-UPD-02** — Updates MUST require **no human, no AI, and no hand-authored GitOps change** to
|
||||
occur. Automation controllers acting on declarative policy are the permitted mechanism.
|
||||
- **R-UPD-03** — Update policy MUST be per-service configurable (cadence, stability channel,
|
||||
pin/hold, auto-rollback on failed health).
|
||||
|
||||
### TLS & certificates — `R-TLS-*`
|
||||
- **R-TLS-01** — Every externally reachable service MUST present TLS from a **credible, trusted,
|
||||
non-self-signed** authority.
|
||||
- **R-TLS-02** — Certificates MUST be automatically **issued, renewed, and revoked** without manual
|
||||
steps.
|
||||
- **R-TLS-03** — Certificate issuance MUST work for internal-only services (no requirement to expose
|
||||
a service publicly just to validate a certificate).
|
||||
|
||||
### DNS & ingress — `R-DNS-*`
|
||||
- **R-DNS-01** — Services MUST be reachable at `<service>.home.keithjackson.dev`.
|
||||
- **R-DNS-02** — DNS records MUST be managed automatically from declared service intent.
|
||||
- **R-DNS-03** — Internal clients MUST resolve service names correctly (split-horizon where public
|
||||
and internal answers differ).
|
||||
|
||||
### Observability & alerting — `R-OBS-*`
|
||||
- **R-OBS-01** — Services MUST be monitored (metrics, logs, health).
|
||||
- **R-OBS-02** — Issues MUST be reported by **email**.
|
||||
- **R-OBS-03** — Issues and health MUST be visible on **dashboards**.
|
||||
- **R-OBS-04** — The platform MUST alert on its own health (nodes, storage, GitOps reconciliation,
|
||||
certificate expiry, VPN tunnel state, backup success).
|
||||
|
||||
### Command & control — `R-C2-*`
|
||||
- **R-C2-01** — The root `home.keithjackson.dev` MUST serve a UI that surfaces the services running
|
||||
in the cluster and links to them.
|
||||
- **R-C2-02** — The C2 UI MUST surface platform health (cluster, storage, GitOps, alerts) and
|
||||
provide controlled operational actions where safe.
|
||||
|
||||
### Security & secrets — `R-SEC-*`
|
||||
- **R-SEC-01** — Secrets MUST NOT be stored in plaintext in git; they MUST be encrypted at rest in
|
||||
the repo and decrypted only in-cluster.
|
||||
- **R-SEC-02** — The platform MUST follow least-privilege for workload identities and network reach.
|
||||
- **R-SEC-03** — All the above properties MUST be demonstrable/auditable (an operator can prove
|
||||
isolation, egress path, cert validity, and backup integrity).
|
||||
|
||||
### Environments — `R-ENV-*`
|
||||
- **R-ENV-01** — dev/test MUST be provisioned as local KVMs via IaC: created, baselined, cluster-
|
||||
joined, and managed as code.
|
||||
- **R-ENV-02** — The same GitOps configuration MUST promote from dev/test to prod with only
|
||||
environment-scoped overrides.
|
||||
- **R-ENV-03** — prod is a 3× mini-PC bare-metal k3s cluster with local storage; it MUST NOT be an
|
||||
active development surface until the roadmap declares it ready.
|
||||
|
||||
### Conformance testing & governance — `R-TEST-*`
|
||||
- **R-TEST-01** — Every requirement MUST be verified by **automated, behavior-based tests**
|
||||
(Given/When/Then style) that assert observable system behavior, not implementation internals.
|
||||
- **R-TEST-02** — Tests MUST be **reproducible**: deterministic, hermetic (create and tear down
|
||||
their own fixtures), version-pinned, and runnable by a single command locally and in CI against
|
||||
any conforming cluster, producing machine-readable, requirement-tagged results.
|
||||
- **R-TEST-03** — A **conformance suite** MUST gate delivery: merges are blocked unless the tests
|
||||
for the touched requirements pass, and phase exits are blocked unless the phase's full suite
|
||||
passes. "Demonstrated" means "the automated test passes," never a manual one-off.
|
||||
- **R-TEST-04** — Tests MUST be **traceable**: each test declares the requirement ID(s) it verifies,
|
||||
and a coverage report maps every `R-*` requirement to its verifying test(s). Uncovered
|
||||
requirements are a gap to close, not an acceptable state.
|
||||
- **R-TEST-05** — Safety-critical behaviors (default-deny, VPN kill-switch, data-survival on
|
||||
rebuild, DR restore, auto-rollback) MUST be verified by **fault-injection** behavior tests that
|
||||
prove fail-closed / no-data-loss outcomes, not merely happy-path checks.
|
||||
- **R-TEST-06 (governance)** — **AI agents MUST NOT modify the specifications** (this `docs/` tree,
|
||||
the requirements, or accepted ADRs). Specs are changed only by the human operator/architect.
|
||||
Agents that find a spec wrong, blocking, ambiguous, or missing MUST **stop and escalate** and MAY
|
||||
draft a proposal, but MUST NOT self-approve or merge a spec/ADR change (see
|
||||
[`AGENTS.md`](../AGENTS.md) §2 and §11).
|
||||
|
||||
## Acceptance philosophy
|
||||
|
||||
Each requirement class maps to an **automated, reproducible, behavior-based conformance test** in
|
||||
dev/test (`R-TEST-*`). "Done" means *the conformance test for the touched requirements passes* — a
|
||||
green, reproducible result traced to requirement IDs — never "manifest merged" and never a manual
|
||||
demonstration. The testing approach is specified in
|
||||
[`docs/14-testing-and-conformance.md`](14-testing-and-conformance.md) and
|
||||
[ADR-0010](../decisions/adr-0010-conformance-testing.md); the gate checklist lives in
|
||||
[`agents/definition-of-done.md`](../agents/definition-of-done.md).
|
||||
|
||||
**Governance:** the specifications are human-owned and **immutable to agents** (R-TEST-06). Agents
|
||||
implement against the spec and prove conformance with tests; they never edit the spec to fit the
|
||||
implementation.
|
||||
|
||||
## Open questions (tracked, not blocking the architecture)
|
||||
|
||||
- **OQ-1 (WAN hardware)**: How many physical WAN uplinks exist in prod, and does the edge router do
|
||||
the multi-WAN policy routing, or is a portion done in-cluster via egress gateways? See
|
||||
[`03-networking-isolation-and-egress.md`](03-networking-isolation-and-egress.md).
|
||||
- **OQ-2 (bulk media)**: Final home for the Plex media library — NAS/NFS vs. a dedicated large disk
|
||||
vs. object storage. See [`04-storage-and-state.md`](04-storage-and-state.md) / ADR-0003.
|
||||
- **OQ-3 (VPN provider)**: Commercial VPN provider + protocol (WireGuard preferred) and how many
|
||||
concurrent tunnels/egress IPs are available.
|
||||
- **OQ-4 (DNS provider API)**: Confirm Cloudflare (or equivalent) manages `keithjackson.dev` with an
|
||||
API token scoped for ACME DNS-01 and ExternalDNS.
|
||||
@@ -0,0 +1,109 @@
|
||||
# 01 — Architecture Overview
|
||||
|
||||
This document is the map. It names the chosen stack, shows how the layers fit, and points to the
|
||||
ADR that locks in each choice and the domain spec that details it. **Do not change a technology
|
||||
choice here without a superseding ADR.**
|
||||
|
||||
## The stack at a glance
|
||||
|
||||
| Layer | Choice | Serves | ADR | Spec |
|
||||
|---|---|---|---|---|
|
||||
| Kubernetes distro | **k3s** (HA, embedded etcd) | R-PLAT-01/02 | [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md) | [02](02-environments-and-provisioning.md) |
|
||||
| CNI / network policy / egress | **Cilium** (+ Hubble, Gateway API, Egress Gateway) | R-NET-01..07 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [03](03-networking-isolation-and-egress.md) |
|
||||
| Distributed storage | **Longhorn** | R-STATE-01..03 | [ADR-0003](../decisions/adr-0003-storage-longhorn.md) | [04](04-storage-and-state.md) |
|
||||
| GitOps engine | **Flux CD** | R-GITOPS-01..03 | [ADR-0004](../decisions/adr-0004-gitops-flux.md) | [05](05-gitops-and-delivery.md) |
|
||||
| VM workloads (IaaS) | **KubeVirt** (+ CDI) | R-PLAT-03 | [ADR-0005](../decisions/adr-0005-vm-workloads-kubevirt.md) | [06](06-vm-workloads-kubevirt.md) |
|
||||
| TLS | **cert-manager** + Let's Encrypt (DNS-01) | R-TLS-01..03 | [ADR-0006](../decisions/adr-0006-tls-cert-manager-letsencrypt.md) | [08](08-tls-and-certificates.md) |
|
||||
| Secrets | **SOPS + age** (Flux-native) | R-SEC-01 | [ADR-0007](../decisions/adr-0007-secrets-sops-age.md) | [07](07-security-and-secrets.md) |
|
||||
| VPN egress | **Gluetun sidecar** + **Cilium Egress Gateway** | R-NET-05/06 | [ADR-0008](../decisions/adr-0008-vpn-egress-model.md) | [03](03-networking-isolation-and-egress.md) |
|
||||
| Automated updates | **Flux image-update-automation** (images) + **Renovate** (charts) | R-UPD-01..03 | [ADR-0009](../decisions/adr-0009-automated-image-updates.md) | [11](11-automated-updates.md) |
|
||||
| DNS records | **ExternalDNS** (public via Cloudflare) + **PiHole** (internal) | R-DNS-01..03 | — | [09](09-dns-and-ingress.md) |
|
||||
| Ingress (north-south) | **Gateway API** implemented by Cilium | R-DNS-01, R-TLS-01 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [09](09-dns-and-ingress.md) |
|
||||
| Observability | **kube-prometheus-stack** + **Loki** + **Grafana Alloy** + Alertmanager→SMTP | R-OBS-01..04 | — | [10](10-observability-and-alerting.md) |
|
||||
| Backup/DR | **Velero** + Longhorn backups → off-cluster S3/NFS | R-STATE-03 | [ADR-0003](../decisions/adr-0003-storage-longhorn.md) | [04](04-storage-and-state.md) |
|
||||
| Node/infra provisioning | **OpenTofu** (libvirt) + **cloud-init** + **Ansible** (k3s bootstrap) | R-ENV-01..03 | [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md) | [02](02-environments-and-provisioning.md) |
|
||||
| Command center UI | **Homepage** + **Headlamp** + **Grafana** panels | R-C2-01/02 | — | [12](12-command-center-ui.md) |
|
||||
| Conformance testing | **Kyverno** (L0 policy) + **Chainsaw** (L1/L2/L3 behavior + fault-injection) | R-TEST-01..06 | [ADR-0010](../decisions/adr-0010-conformance-testing.md), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md) | [14](14-testing-and-conformance.md) |
|
||||
|
||||
## Layer cake
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────────────┐
|
||||
│ Command & Control UI (home.keithjackson.dev root) │
|
||||
│ Homepage catalog · Headlamp k8s console · Grafana dashboards · Flux status │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Workloads │
|
||||
│ PaaS: containers (Plex, PiHole, Nextcloud, Gitea, Seerr, …) │
|
||||
│ IaaS: KubeVirt VMs (anything that can't be a container) │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Platform services │
|
||||
│ Flux (GitOps) · cert-manager (TLS) · ExternalDNS · Gateway API │
|
||||
│ kube-prometheus-stack + Loki + Alloy (observability) │
|
||||
│ Velero (backup) · Renovate (chart updates) · SOPS (secrets) │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Storage: Longhorn (replicated block) + bulk media store (see ADR-0003) │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Network: Cilium (default-deny NetworkPolicy · Egress Gateway · Hubble) │
|
||||
│ VPN egress (Gluetun sidecars / VPN gateway) · Multi-WAN routing │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Kubernetes: k3s (3 nodes, HA control plane, embedded etcd) │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Nodes │
|
||||
│ dev/test: libvirt KVMs (OpenTofu + cloud-init + Ansible) │
|
||||
│ prod: 3× mini-PC bare metal, local disks │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Edge: router/firewall — multi-WAN uplinks, port-forward to Gateway, │
|
||||
│ VPN uplink(s), split-horizon DNS │
|
||||
└──────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Two clearly-separated virtualization layers (do not conflate)
|
||||
|
||||
This platform has **two** kinds of VM, and confusing them is the most common design error:
|
||||
|
||||
1. **Infrastructure VMs = the cluster nodes themselves.**
|
||||
In dev/test these are libvirt KVMs on the workstation, provisioned by OpenTofu + cloud-init and
|
||||
configured/joined by Ansible. In prod these are the bare-metal mini-PCs. See
|
||||
[`02-environments-and-provisioning.md`](02-environments-and-provisioning.md).
|
||||
|
||||
2. **Application VMs = IaaS workloads that run *inside* the cluster via KubeVirt.**
|
||||
These are managed by the same Kubernetes API/YAML/GitOps as containers. They satisfy R-PLAT-03
|
||||
("not everything fits in a container"). See [`06-vm-workloads-kubevirt.md`](06-vm-workloads-kubevirt.md).
|
||||
|
||||
> Layer 1 is how the platform exists. Layer 2 is a feature the platform offers.
|
||||
|
||||
## How a service flows through the platform (the golden path)
|
||||
|
||||
1. Operator adds/edits a **ServiceDefinition** in the git catalog (see
|
||||
[`13-service-catalog.md`](13-service-catalog.md)) declaring: kind (container|vm), image/source,
|
||||
storage, **egress class** (`internet`|`vpn`), exposure (`internal`|`public`), dependencies
|
||||
(explicit allow rules), update policy, and observability expectations.
|
||||
2. **Flux** reconciles the change into the cluster.
|
||||
3. **Longhorn** provisions replicated persistent volumes; state is durable.
|
||||
4. **Cilium** applies default-deny plus the explicitly declared allow rules; assigns the correct
|
||||
**egress path** (local internet or VPN, with kill-switch).
|
||||
5. **ExternalDNS** publishes `<service>.home.keithjackson.dev`; **Gateway API** routes north-south
|
||||
traffic; **cert-manager** issues a trusted TLS cert via ACME DNS-01.
|
||||
6. **kube-prometheus-stack + Loki** scrape metrics/logs; **Alertmanager** emails on issues; the
|
||||
**C2 UI** lists and links the service.
|
||||
7. **Flux image automation / Renovate** keep the service on the latest stable version per policy —
|
||||
no human action.
|
||||
8. **Velero + Longhorn backups** protect the data; destroy/rebuild restores state.
|
||||
|
||||
Every numbered step maps to a domain spec and a workstream in [`roadmap/workstreams.md`](../roadmap/workstreams.md).
|
||||
|
||||
## Design principles
|
||||
|
||||
- **Declarative everything.** If it can't be expressed as YAML reconciled from git, question it.
|
||||
- **One control plane.** Containers and VMs share the Kubernetes surface; operators learn one model.
|
||||
- **Secure by default.** Deny-by-default network, least-privilege identity, encrypted secrets,
|
||||
trusted TLS everywhere.
|
||||
- **Durable by default.** No workload is "pet" hardware; state lives on replicated storage with
|
||||
tested restore.
|
||||
- **Hands-off operations.** Updates, cert renewal, DNS, and drift correction are automated.
|
||||
- **Promotable.** dev/test and prod are the same config with scoped overrides.
|
||||
- **Provable.** Every requirement is verified by an automated, reproducible, behavior-based
|
||||
conformance test ([14](14-testing-and-conformance.md), [ADR-0010](../decisions/adr-0010-conformance-testing.md));
|
||||
"done" is a green, requirement-tagged suite, not a manual demo.
|
||||
- **Spec-fixed.** The specification is human-owned and immutable to agents (R-TEST-06); agents move
|
||||
the implementation and its tests toward the spec, never the spec toward the implementation.
|
||||
@@ -0,0 +1,93 @@
|
||||
# 02 — Environments & Provisioning
|
||||
|
||||
Serves: **R-ENV-01/02/03**, **R-PLAT-01**. Related: [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md).
|
||||
|
||||
## Two environments, one configuration
|
||||
|
||||
| Aspect | dev/test | prod |
|
||||
|---|---|---|
|
||||
| Substrate | libvirt/KVM VMs on the operator workstation | 3× mini-PC bare metal |
|
||||
| Node count | 3 VMs (mirror prod topology) | 3 physical nodes |
|
||||
| Control plane | HA embedded etcd (k3s server on all 3) | HA embedded etcd (k3s server on all 3) |
|
||||
| Storage | virtual disks presented to Longhorn | local SSD/NVMe presented to Longhorn |
|
||||
| Status | **active build target** | **design-only until roadmap flips it** |
|
||||
|
||||
**R-ENV-02 is the golden constraint:** the GitOps configuration is authored once. Environments
|
||||
differ only through scoped overrides (Flux `Kustomization` overlays / cluster-scoped variables):
|
||||
node counts, disk sizes, WAN endpoints, VPN credentials, DNS zone specifics, resource limits.
|
||||
Never fork the base config per environment.
|
||||
|
||||
## Infrastructure provisioning (Layer 1 — the cluster nodes)
|
||||
|
||||
> These are the machines that *run* k3s. Distinct from KubeVirt application VMs (Layer 2).
|
||||
|
||||
### Toolchain
|
||||
- **OpenTofu** (Terraform-compatible) with the **libvirt** provider defines dev/test node VMs as
|
||||
code: CPU, RAM, disks (a dedicated data disk for Longhorn), NICs, and a **cloud-init** seed.
|
||||
- **cloud-init** performs first-boot baseline: hostname, users, SSH keys, package baseline, kernel
|
||||
modules/sysctls required by k3s/Longhorn/KubeVirt, and marks the node ready for k3s.
|
||||
- **Ansible** performs cluster bootstrap: installs k3s in HA mode, joins nodes, lays down cluster-
|
||||
wide prerequisites (Cilium is installed as the CNI, not flannel), and hands off to Flux.
|
||||
- For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation step differs
|
||||
(bare-metal image/cloud-init via USB/PXE instead of libvirt).
|
||||
|
||||
### Node lifecycle stages (R-ENV-01 — "created, baselined, cluster-joined, managed as code")
|
||||
1. **Create** — OpenTofu instantiates the VM (dev) / operator images the mini-PC (prod).
|
||||
2. **Baseline** — cloud-init applies OS baseline and prerequisites.
|
||||
3. **Cluster-join** — Ansible installs/joins k3s with Cilium as CNI; disables k3s bundled
|
||||
flannel/traefik/servicelb/kube-proxy as required by the Cilium ADR.
|
||||
4. **Manage** — from here the cluster is GitOps-managed by Flux; node config changes are made via
|
||||
IaC and re-applied, never hand-edited on the box.
|
||||
|
||||
### k3s topology decisions
|
||||
- HA control plane with **embedded etcd** across all 3 nodes (no external datastore).
|
||||
- Bundled components that conflict with our stack are **disabled at install**: flannel (→ Cilium),
|
||||
kube-proxy (→ Cilium kube-proxy replacement), Traefik (→ Gateway API via Cilium), servicelb where
|
||||
a different LB strategy is chosen. Exact flags live in the Ansible bootstrap; the *intent* is
|
||||
fixed here and in ADR-0001/0002.
|
||||
- Nodes are both control-plane and workload (3-node converged cluster); with only 3 nodes we do not
|
||||
isolate control-plane nodes.
|
||||
|
||||
### Bare-metal specifics (prod, design-only for now)
|
||||
- Each mini-PC contributes local NVMe/SSD to Longhorn; document the disk layout (OS vs. Longhorn
|
||||
data device) before prod build.
|
||||
- BIOS/firmware, virtualization extensions (for KubeVirt), and wake/boot behavior are captured as a
|
||||
provisioning checklist under the prod-readiness workstream.
|
||||
- Edge networking (multi-WAN, VPN uplink) is largely an **edge-router** concern — see
|
||||
[`03-networking-isolation-and-egress.md`](03-networking-isolation-and-egress.md) and OQ-1.
|
||||
|
||||
## Environment promotion model
|
||||
|
||||
```
|
||||
git repo (single source of truth)
|
||||
│
|
||||
base configuration ──────────────┐
|
||||
│ │
|
||||
overlay: dev/test overlay: prod
|
||||
│ │
|
||||
Flux on dev cluster Flux on prod cluster (later)
|
||||
```
|
||||
|
||||
- A change proves out in **dev/test** against the acceptance gates ([`AGENTS.md` §7](../AGENTS.md)).
|
||||
- Promotion to prod is a change to the prod overlay only (no base edits), gated by the roadmap's
|
||||
prod-readiness phase.
|
||||
- **Never develop against prod.** Prod hardware is a target, not a sandbox (R-ENV-03).
|
||||
|
||||
## Statelessness of the nodes themselves
|
||||
|
||||
Nodes are cattle. A node can be destroyed and rebuilt from IaC + cloud-init + Ansible; cluster and
|
||||
application state lives on **Longhorn** (replicated) and in **git** (desired state), so node loss is
|
||||
a re-provision, not a data-loss event (supports R-STATE-01/02).
|
||||
|
||||
## Deliverables for this spec (tracked on the task board)
|
||||
- OpenTofu module for dev/test node VMs (libvirt) with a dedicated Longhorn data disk.
|
||||
- cloud-init baseline template (users, SSH, sysctls, kernel modules for Longhorn/KubeVirt).
|
||||
- Ansible role(s): k3s HA install with Cilium, component disabling, Flux bootstrap handoff.
|
||||
- Documented prod bare-metal provisioning checklist (design-only; do not execute).
|
||||
- Proof: destroy a dev node VM and rebuild it; cluster re-converges with no data loss.
|
||||
|
||||
## Open questions
|
||||
- **OQ-1** (edge router / WAN count) — blocks final multi-WAN design, not node provisioning.
|
||||
- Do we want **Cluster API** later to unify dev VM and prod bare-metal lifecycle? Deferred; OpenTofu
|
||||
+ Ansible is sufficient and lower-complexity for 3 nodes now. Revisit via ADR if prod scaling
|
||||
demands it.
|
||||
@@ -0,0 +1,126 @@
|
||||
# 03 — Networking, Isolation & Egress
|
||||
|
||||
Serves: **R-NET-01..07**, **R-SEC-02**. Related: [ADR-0002 (Cilium)](../decisions/adr-0002-cni-cilium.md),
|
||||
[ADR-0008 (VPN egress)](../decisions/adr-0008-vpn-egress-model.md).
|
||||
|
||||
This is the most safety-critical spec. Get default-deny and the VPN kill-switch right; everything
|
||||
else is convenience.
|
||||
|
||||
## Principles
|
||||
|
||||
1. **Deny by default, allow by exception.** No service may talk to another unless a rule explicitly
|
||||
permits it, at the port/protocol level (R-NET-01/02).
|
||||
2. **Egress is a declared property.** Each service declares an **egress class**; the platform routes
|
||||
it accordingly (R-NET-05/07).
|
||||
3. **VPN means VPN-only.** A VPN-class service must never leak to the plain internet, including
|
||||
during tunnel failure (R-NET-06, kill-switch).
|
||||
4. **Isolation is auditable.** An operator can prove what a service can and cannot reach (R-SEC-03).
|
||||
|
||||
## CNI: Cilium
|
||||
|
||||
Cilium replaces the default k3s CNI and provides:
|
||||
- **NetworkPolicy / CiliumNetworkPolicy** for L3/L4 (and optional L7) enforcement.
|
||||
- **kube-proxy replacement** (eBPF) for efficient service routing.
|
||||
- **Egress Gateway** to force selected pods' egress through a chosen node/identity/SNAT IP — the
|
||||
building block for WAN/VPN path selection.
|
||||
- **Hubble** for flow observability — the audit and troubleshooting tool for isolation.
|
||||
- **Gateway API** implementation for north-south ingress (see [09](09-dns-and-ingress.md)).
|
||||
|
||||
## Segmentation model
|
||||
|
||||
- **Namespaces as trust zones.** Each service (or tightly-coupled service group) gets its own
|
||||
namespace. Namespace = isolation boundary.
|
||||
- **Egress classes as zones:** `internet` and `vpn` are first-class labels that drive both network
|
||||
policy and egress routing.
|
||||
- **Platform namespace(s)** (Flux, cert-manager, monitoring, storage) are isolated from application
|
||||
namespaces; only the specific flows they need (e.g. metrics scraping) are opened.
|
||||
|
||||
### Default-deny baseline (R-NET-01)
|
||||
Every namespace gets a baseline policy set applied by GitOps at namespace creation:
|
||||
- **Deny all ingress** and **deny all egress** by default.
|
||||
- Then explicitly allow only:
|
||||
- DNS to the in-cluster resolver (required for anything to function).
|
||||
- Declared intra-cluster dependencies (source → destination → port).
|
||||
- The declared egress class path (internet or VPN).
|
||||
- Observability scraping from the monitoring namespace (metrics/logs).
|
||||
|
||||
> A workload with no explicit allow rules can do *nothing* except resolve DNS. This is the intended
|
||||
> starting posture for every onboarded service.
|
||||
|
||||
### Explicit allow rules (R-NET-02)
|
||||
Service dependencies from the catalog compile into allow rules. Example intent (illustrative, not
|
||||
final manifest):
|
||||
|
||||
```
|
||||
seerr → sonarr tcp/8989 (Seerr requests to Sonarr)
|
||||
seerr → radarr tcp/7878 (Seerr requests to Radarr)
|
||||
seerr → plex tcp/32400 (Seerr talks to Plex)
|
||||
```
|
||||
|
||||
Rules are declared **once** in the service's catalog entry (its `dependencies`) and rendered into
|
||||
CiliumNetworkPolicies by the platform. Reviewers see the intent, not hand-written policy YAML.
|
||||
|
||||
## Multi-WAN & egress routing (R-NET-04/05)
|
||||
|
||||
The platform recognizes **egress profiles**. Initial profiles:
|
||||
|
||||
| Profile | Meaning | Mechanism |
|
||||
|---|---|---|
|
||||
| `internet` | Standard local network → default WAN(s) | Normal pod egress via node → edge router |
|
||||
| `vpn` | Routed through a commercial VPN tunnel | Gluetun sidecar and/or Cilium Egress Gateway to a VPN egress node (ADR-0008) |
|
||||
|
||||
**Where multi-WAN policy routing lives (OQ-1):**
|
||||
- **Physical multi-WAN** (two ISP uplinks, failover/balancing by routing rules) is primarily an
|
||||
**edge-router/firewall** responsibility (e.g. OPNsense/pfSense) sitting in front of the cluster.
|
||||
Traffic leaving the cluster on the `internet` profile is policy-routed by the edge router across
|
||||
available WANs.
|
||||
- **In-cluster egress selection** (which pods use VPN vs. plain internet) is handled by Cilium
|
||||
Egress Gateway + the VPN egress model. The cluster decides *class*; the edge router decides
|
||||
*physical WAN* for the internet class.
|
||||
- Final split between edge-router and in-cluster responsibility is pinned once OQ-1 (WAN hardware)
|
||||
is answered. The **service-facing contract does not change**: a service still just declares
|
||||
`egress: internet|vpn`.
|
||||
|
||||
## VPN egress model (R-NET-05/06 — ADR-0008)
|
||||
|
||||
VPN-class services: **sonarr, radarr, jackett, qbittorrent**. Requirements: all their egress leaves
|
||||
via the VPN; if the tunnel drops, they lose connectivity rather than leaking.
|
||||
|
||||
Two supported patterns (ADR-0008 selects the default; both preserve the `egress: vpn` contract):
|
||||
|
||||
- **Pattern A — Gluetun sidecar per pod.** A Gluetun container in the pod holds the WireGuard/OpenVPN
|
||||
tunnel; the app container shares its network namespace and routes all traffic through it. Gluetun's
|
||||
**firewall kill-switch** blocks non-tunnel egress by default — satisfies R-NET-06 intrinsically.
|
||||
Simple, self-contained, per-service isolation; more sidecars to run.
|
||||
- **Pattern B — Central VPN egress gateway + Cilium Egress Gateway.** One (HA) VPN gateway
|
||||
deployment holds the tunnel; Cilium Egress Gateway steers `vpn`-labeled pods' egress through it.
|
||||
Fewer tunnels; a shared failure domain; kill-switch enforced at the gateway + policy denying
|
||||
fallback to the internet path.
|
||||
|
||||
**Kill-switch is mandatory in both patterns and is an acceptance gate:** with the tunnel forcibly
|
||||
down, a `vpn` service must fail closed (no packets to the plain internet). This must be tested, not
|
||||
assumed.
|
||||
|
||||
## Ingress (north-south) — summary
|
||||
|
||||
North-south exposure is handled by **Gateway API implemented by Cilium**, with TLS from
|
||||
cert-manager and DNS from ExternalDNS. Detailed in [`09-dns-and-ingress.md`](09-dns-and-ingress.md).
|
||||
Exposure is a declared property: `internal` (LAN-only) vs `public` (port-forwarded at the edge).
|
||||
|
||||
## Observability of the network (supports R-SEC-03, R-OBS-04)
|
||||
- **Hubble** provides flow visibility to verify default-deny and troubleshoot allow rules.
|
||||
- Alert on: VPN tunnel down, egress-gateway unhealthy, unexpected denied flows spiking, policy
|
||||
reconciliation errors. Wired into [`10-observability-and-alerting.md`](10-observability-and-alerting.md).
|
||||
|
||||
## Acceptance gates for this spec
|
||||
- A freshly onboarded service with no rules can reach **nothing** but DNS (default-deny proven via
|
||||
Hubble + a connectivity probe).
|
||||
- Declared dependencies work; **undeclared** ones are blocked.
|
||||
- A `vpn` service's public egress IP is the VPN's, not the home IP (verified).
|
||||
- Kill-switch: tunnel down ⇒ `vpn` service has no internet path; `internet` services unaffected.
|
||||
- Changing a service's `egress` class in git flips its path with no workload rearchitecting.
|
||||
|
||||
## Open questions
|
||||
- **OQ-1** — WAN uplink count and edge-router product; determines edge-vs-cluster routing split.
|
||||
- **OQ-3** — VPN provider/protocol (WireGuard preferred), number of concurrent tunnels / distinct
|
||||
egress IPs (affects Pattern A vs B and HA of the gateway).
|
||||
@@ -0,0 +1,91 @@
|
||||
# 04 — Storage & State
|
||||
|
||||
Serves: **R-STATE-01..04**, **R-PLAT-04**. Related: [ADR-0003 (Longhorn)](../decisions/adr-0003-storage-longhorn.md).
|
||||
|
||||
The core promise: **any service can be destroyed and rebuilt on a whim without losing data.** This
|
||||
requires (a) durable replicated storage, (b) a clean separation of state from workload, and (c) a
|
||||
tested backup/restore path.
|
||||
|
||||
## Storage tiers
|
||||
|
||||
| Tier | Backing | Use | Replication |
|
||||
|---|---|---|---|
|
||||
| **App/config state** | Longhorn block volumes | PVCs for service config, databases, small app data | 3-way replica across nodes |
|
||||
| **Bulk media** | Dedicated large store (see below) | Plex library, download staging | Not 3× replicated (capacity strategy) |
|
||||
| **VM disks** | Longhorn (via KubeVirt/CDI) | KubeVirt application-VM root/data disks | 3-way replica |
|
||||
| **Backups** | Off-cluster S3/NFS target | Velero + Longhorn backups for DR | Off-cluster, separate failure domain |
|
||||
|
||||
## Longhorn (primary persistent storage — R-STATE-02)
|
||||
|
||||
- Provides replicated block storage from each node's local disk — exactly the model for a 3× mini-PC
|
||||
cluster with local storage and no SAN.
|
||||
- Default StorageClass: **3 replicas** so any single node loss keeps data available.
|
||||
- Volumes are the durable home for stateful services. Destroying a pod/Deployment/VM leaves the PVC
|
||||
and its data intact; rebuilding re-attaches it (R-STATE-01).
|
||||
- Snapshots for point-in-time; **backups** (to off-cluster target) for DR.
|
||||
- Guardrails: reserve disk headroom, set replica anti-affinity across nodes, monitor volume health
|
||||
and degraded replicas (alerts in [10](10-observability-and-alerting.md)).
|
||||
|
||||
## Separation of state from workload (the rebuild guarantee)
|
||||
|
||||
Every service's catalog entry declares its **persistent volumes** explicitly. The rule:
|
||||
|
||||
> If it must survive a rebuild, it lives on a named PVC (Longhorn) or the bulk store — **never** in
|
||||
> the container's ephemeral filesystem or on a single node's hostPath.
|
||||
|
||||
The rebuild test (an acceptance gate): delete the workload entirely, re-apply from git, and the
|
||||
service comes back with its data. This is verified per service, not assumed.
|
||||
|
||||
## Bulk media strategy (R-STATE-04 — the Plex problem)
|
||||
|
||||
Plex is the largest service and its library can be multiple terabytes. **3× replicating multi-TB
|
||||
media across three mini-PCs is impractical** (capacity and cost). Therefore bulk media is a
|
||||
**distinct tier** from replicated app-config storage. Options to be pinned by ADR-0003 / OQ-2:
|
||||
|
||||
- **NAS / external NFS** exporting the media library, mounted into Plex (and the arr stack) — keeps
|
||||
bulk data off the replicated tier; simplest capacity story; introduces an external dependency.
|
||||
- **Dedicated large disk** on one node with **single-replica Longhorn** or a local PV — cheaper, but
|
||||
node-bound; requires its own backup plan.
|
||||
- **Object storage** for some content — generally poor fit for Plex direct play.
|
||||
|
||||
Regardless of choice: **Plex's config/metadata database is small and MUST live on 3-replica
|
||||
Longhorn** (so Plex rebuilds cleanly); only the large media files use the bulk tier. Downloads
|
||||
(qbittorrent) stage to the bulk tier and hand off to the arr stack.
|
||||
|
||||
## Backup & disaster recovery (R-STATE-03)
|
||||
|
||||
- **Velero** backs up Kubernetes resource state (namespaces, manifests, PVC metadata) and
|
||||
orchestrates volume backups.
|
||||
- **Longhorn backups** push volume data to an **off-cluster** S3-compatible or NFS target — a
|
||||
separate failure domain from the cluster itself.
|
||||
- **Schedule**: regular automated backups (cadence per data criticality) with retention policy.
|
||||
- **Restore is a tested procedure**, not a theoretical one: a DR drill (restore a namespace + its
|
||||
volumes to a clean state) is part of the platform acceptance suite.
|
||||
- Bulk media has its own backup posture (e.g. it may be re-acquirable rather than backed up 1:1 —
|
||||
an explicit operator decision recorded per OQ-2), but **all irreplaceable state** (configs,
|
||||
databases, Nextcloud data, Gitea repos) MUST be backed up off-cluster.
|
||||
|
||||
## KubeVirt VM disks
|
||||
|
||||
Application-VM disks are Longhorn volumes provisioned via **CDI** (Containerized Data Importer). VMs
|
||||
inherit the same durability and backup story as container PVCs — a KubeVirt VM can be destroyed and
|
||||
rebuilt from its declarative definition + persistent disk. See [`06-vm-workloads-kubevirt.md`](06-vm-workloads-kubevirt.md).
|
||||
|
||||
## GitOps + state = full rebuildability
|
||||
|
||||
- **Desired state** lives in git (manifests, service definitions).
|
||||
- **Data** lives on Longhorn/bulk store + off-cluster backups.
|
||||
- Therefore a total cluster loss is recoverable: re-provision nodes ([02](02-environments-and-provisioning.md)),
|
||||
re-bootstrap Flux, let it reconcile from git, and restore volumes from backup. This end-to-end
|
||||
recovery is the ultimate expression of R-STATE-01/03 and SHOULD be drilled at least once in
|
||||
dev/test before prod is declared ready.
|
||||
|
||||
## Acceptance gates
|
||||
- Delete and recreate each stateful service; data survives (per-service rebuild test).
|
||||
- Kill a node (dev/test); replicated volumes stay available; cluster re-converges.
|
||||
- Restore a namespace + volumes from off-cluster backup into a clean state (DR drill).
|
||||
- Plex rebuilds with its metadata DB from replicated storage; media re-attaches from the bulk tier.
|
||||
|
||||
## Open questions
|
||||
- **OQ-2** — final bulk-media home (NAS/NFS vs dedicated disk vs mixed) and its backup posture.
|
||||
- Off-cluster backup **target**: which S3-compatible endpoint or NFS host, and its own resilience.
|
||||
@@ -0,0 +1,93 @@
|
||||
# 05 — GitOps & Delivery
|
||||
|
||||
Serves: **R-GITOPS-01..03**, **R-PLAT-02/04**, **R-ENV-02**. Related: [ADR-0004 (Flux)](../decisions/adr-0004-gitops-flux.md).
|
||||
|
||||
## Model
|
||||
|
||||
Pull-based GitOps with **Flux CD**. The cluster continuously reconciles its state from git. Git is
|
||||
the single source of truth; the cluster conforms to git, not the reverse (R-GITOPS-01). Drift is
|
||||
detected and corrected automatically (R-GITOPS-03).
|
||||
|
||||
## Repository layout (the implementation repo, created in a later phase)
|
||||
|
||||
> This spec repo (`home-network-k3s`) holds intent. The **implementation** repo(s) hold the
|
||||
> reconciled manifests. Proposed structure:
|
||||
|
||||
```
|
||||
platform/ ← cluster-wide platform services (Flux-managed)
|
||||
cilium/ ← CNI, egress gateway, network baseline
|
||||
longhorn/ ← storage
|
||||
cert-manager/ ← TLS issuers + ClusterIssuer
|
||||
externaldns/ ← DNS automation
|
||||
gateway/ ← Gateway API + listeners/TLS
|
||||
monitoring/ ← kube-prometheus-stack, Loki, Alloy, Alertmanager
|
||||
velero/ ← backup/DR
|
||||
kubevirt/ ← KubeVirt + CDI
|
||||
image-automation/ ← Flux image update automation config
|
||||
apps/ ← application workloads, one dir per service
|
||||
plex/ sonarr/ radarr/ jackett/ seerr/ pihole/ nextcloud/
|
||||
minecraft/ unifi/ semaphore/ kasm/ gitea/ qbittorrent/
|
||||
catalog/ ← ServiceDefinition entries (the onboarding surface)
|
||||
clusters/
|
||||
dev/ ← Flux entrypoint + dev overlay/vars
|
||||
prod/ ← Flux entrypoint + prod overlay/vars (design-only)
|
||||
```
|
||||
|
||||
- **Kustomize overlays** (or equivalent) express environment differences (R-ENV-02). Base is shared;
|
||||
`clusters/dev` and `clusters/prod` supply scoped values only.
|
||||
- **HelmReleases** (Flux) wrap upstream charts (Cilium, Longhorn, cert-manager, kube-prometheus-
|
||||
stack, etc.); app workloads use charts where good ones exist, otherwise plain manifests.
|
||||
|
||||
## The onboarding surface: ServiceDefinition (R-GITOPS-02)
|
||||
|
||||
Onboarding a service = adding one **ServiceDefinition** under `catalog/`. It is the operator-facing
|
||||
abstraction that the platform compiles into the concrete resources (Deployment/VM, PVCs,
|
||||
NetworkPolicies, HTTPRoute, Certificate, DNS record, ServiceMonitor, update policy). See
|
||||
[`13-service-catalog.md`](13-service-catalog.md) for the schema. Intent lives in one place; the
|
||||
platform renders the rest.
|
||||
|
||||
## Reconciliation flow
|
||||
|
||||
1. Operator opens a PR against the implementation repo (new/updated ServiceDefinition or platform
|
||||
change).
|
||||
2. CI validates (lint, policy checks, schema validation, dry-run/build).
|
||||
3. Merge to the tracked branch.
|
||||
4. Flux reconciles: applies manifests, reports status, corrects drift on an interval.
|
||||
5. Health/rollout status surfaces in the C2 UI ([12](12-command-center-ui.md)) and alerts on failure
|
||||
([10](10-observability-and-alerting.md)).
|
||||
|
||||
## Secrets in GitOps
|
||||
|
||||
Secrets are committed **encrypted** with **SOPS + age**; Flux decrypts them in-cluster at
|
||||
reconciliation. No plaintext secret ever lands in git. See [`07-security-and-secrets.md`](07-security-and-secrets.md)
|
||||
and [ADR-0007](../decisions/adr-0007-secrets-sops-age.md).
|
||||
|
||||
## Automated updates interplay (R-UPD-*)
|
||||
|
||||
GitOps and hands-off updates coexist:
|
||||
- **Container image tags** advance automatically via **Flux image-update-automation** driven by
|
||||
per-service **image policies** (semver range / stable channel) — the controller resolves and
|
||||
reconciles the new version with **no human-authored change** (R-UPD-02).
|
||||
- **Helm chart / manifest version bumps** are proposed by **Renovate** with auto-merge for allowed
|
||||
update classes.
|
||||
|
||||
Full detail and the "is an automated commit a GitOps change?" nuance is in
|
||||
[`11-automated-updates.md`](11-automated-updates.md) and [ADR-0009](../decisions/adr-0009-automated-image-updates.md).
|
||||
|
||||
## Bootstrap & recovery
|
||||
|
||||
- Flux is bootstrapped onto a fresh cluster by the Ansible handoff ([02](02-environments-and-provisioning.md)).
|
||||
- Because desired state is in git and data is on Longhorn/backups, a rebuilt cluster reconverges by
|
||||
re-bootstrapping Flux and restoring volumes ([04](04-storage-and-state.md)).
|
||||
|
||||
## Guardrails
|
||||
- No `kubectl apply` of app state by hand — everything flows through git/Flux (R-GITOPS-01, click-
|
||||
ops-never per [AGENTS.md](../AGENTS.md)).
|
||||
- Reconciliation errors and drift are **alerting conditions** (R-OBS-04).
|
||||
- Base config is never forked per environment (R-ENV-02).
|
||||
|
||||
## Acceptance gates
|
||||
- Adding a ServiceDefinition PR results in a fully wired service (compute, storage, policy, TLS, DNS,
|
||||
monitoring) with no manual steps beyond the merge.
|
||||
- Manually mutating a live resource is auto-reverted by Flux (drift correction proven).
|
||||
- The same base config renders correctly for both dev and prod overlays.
|
||||
@@ -0,0 +1,79 @@
|
||||
# 06 — VM Workloads (IaaS via KubeVirt)
|
||||
|
||||
Serves: **R-PLAT-03**, **R-PLAT-04**, **R-STATE-01**. Related: [ADR-0005 (KubeVirt)](../decisions/adr-0005-vm-workloads-kubevirt.md).
|
||||
|
||||
## Why this exists
|
||||
|
||||
"Not everything fits in a container." Some workloads need a full OS, a specific kernel, non-container
|
||||
software, or a lift-and-shift of an existing VM. The requirement (R-PLAT-03) is that these VMs are
|
||||
managed through the **same Kubernetes API, CLI, and YAML** as containers — one control plane, one
|
||||
GitOps flow, one network/storage/observability model. **KubeVirt** delivers exactly this: VMs as
|
||||
Kubernetes custom resources.
|
||||
|
||||
## Do not confuse the two VM layers
|
||||
|
||||
- **Layer 1 — cluster nodes** are VMs only in dev/test (libvirt) and bare metal in prod. Managed by
|
||||
OpenTofu/Ansible ([02](02-environments-and-provisioning.md)). **Not** KubeVirt.
|
||||
- **Layer 2 — application VMs** run *inside* the cluster via KubeVirt and are the subject of this
|
||||
spec. Managed by Kubernetes/GitOps like any other workload.
|
||||
|
||||
## What KubeVirt provides
|
||||
|
||||
- `VirtualMachine` / `VirtualMachineInstance` custom resources — declare a VM in YAML, reconcile via
|
||||
Flux, manage via `kubectl` (and `virtctl` for console/start/stop).
|
||||
- **CDI (Containerized Data Importer)** to import/clone disk images into **Longhorn** PVCs — VM disks
|
||||
get the same replication and backup story as container volumes ([04](04-storage-and-state.md)).
|
||||
- VMs sit on the **same Cilium network** — so **default-deny isolation and egress classes apply to
|
||||
VMs too** (R-NET-*). A KubeVirt VM can be `egress: vpn` just like a container.
|
||||
- VMs are exposed through the same **Gateway API / Service** model and monitored through the same
|
||||
stack.
|
||||
|
||||
## The unified contract
|
||||
|
||||
A VM onboarded via a ServiceDefinition (`kind: vm`) gets, from the *same* declarative surface:
|
||||
- Compute (vCPU/RAM), an OS image (via CDI), and Longhorn-backed disks.
|
||||
- Default-deny networking + explicit allow rules + egress class (R-NET-*).
|
||||
- TLS + DNS + ingress if it exposes an HTTP(S) service (R-TLS/R-DNS).
|
||||
- Metrics/logs and alerting (R-OBS-*) — node-exporter-class metrics from inside the guest plus
|
||||
KubeVirt/virt metrics from the platform.
|
||||
- Backup/DR of its disks (R-STATE-03) and rebuildability (R-STATE-01): destroy the VM, keep the
|
||||
disk PVC, recreate from the declaration.
|
||||
|
||||
## Update policy for VMs (R-UPD-*)
|
||||
|
||||
VM "updates" differ from container image tag bumps:
|
||||
- **Guest OS package updates** happen *inside* the guest on a schedule (unattended-upgrades / dnf-
|
||||
automatic style), configured via the VM's cloud-init/declaration — no human action.
|
||||
- **VM image/template updates** (new base image) are handled as declarative changes to the VM
|
||||
definition; where a golden-image pipeline exists, new images can be promoted automatically per
|
||||
policy.
|
||||
- The requirement "latest stable without human/AI/GitOps change" is met for guest packages via in-
|
||||
guest automation; base-image rebuilds follow the same automated-promotion philosophy as containers
|
||||
where feasible. Record any per-VM exceptions explicitly.
|
||||
|
||||
## When to choose VM vs container
|
||||
|
||||
Prefer containers (PaaS) by default. Choose a KubeVirt VM (IaaS) when:
|
||||
- The software isn't reasonably containerizable (kernel modules, full desktop, licensing tied to a
|
||||
VM, appliance images).
|
||||
- Lifting an existing VM is faster/safer than re-platforming.
|
||||
- Strong OS-level isolation is required beyond namespace/network policy.
|
||||
|
||||
For the initial catalog, all 13 named services are container-first; KubeVirt is the escape hatch and
|
||||
must be **proven** in dev/test with at least one representative VM workload so the IaaS path is real,
|
||||
not theoretical.
|
||||
|
||||
## Prerequisites & constraints
|
||||
- Nodes must expose virtualization extensions (nested virt in dev/test KVMs; native on prod mini-
|
||||
PCs) — captured in the node baseline ([02](02-environments-and-provisioning.md)).
|
||||
- Live migration across 3 converged nodes is a nice-to-have, not a requirement; document whether it's
|
||||
enabled.
|
||||
- Resource pressure: VMs are heavier than pods — capacity planning must account for them,
|
||||
particularly on the 3-node prod cluster.
|
||||
|
||||
## Acceptance gates
|
||||
- Create, start, stop, and destroy a VM entirely through Kubernetes YAML + `kubectl`/`virtctl`.
|
||||
- The VM's disk is Longhorn-backed; destroy/rebuild preserves data (R-STATE-01).
|
||||
- Default-deny applies: the VM can reach only what its allow rules permit.
|
||||
- An `egress: vpn` VM egresses via the VPN with a working kill-switch (R-NET-06).
|
||||
- Guest OS auto-updates on schedule with no manual step (R-UPD-*).
|
||||
@@ -0,0 +1,83 @@
|
||||
# 07 — Security & Secrets
|
||||
|
||||
Serves: **R-SEC-01/02/03**, supports all `R-NET-*` and `R-TLS-*`. Related:
|
||||
[ADR-0007 (SOPS+age)](../decisions/adr-0007-secrets-sops-age.md).
|
||||
|
||||
## Security posture
|
||||
|
||||
Defense in depth with declarative, auditable controls:
|
||||
|
||||
- **Network**: default-deny east-west, explicit port-level allow, egress-class routing, VPN kill-
|
||||
switch ([03](03-networking-isolation-and-egress.md)). This is the primary isolation control.
|
||||
- **Identity/least privilege**: each workload runs with a minimal ServiceAccount and only the RBAC
|
||||
it needs; no workload gets cluster-admin. VMs and containers alike get least network reach
|
||||
(R-SEC-02).
|
||||
- **Secrets**: encrypted at rest in git; decrypted only in-cluster (R-SEC-01).
|
||||
- **TLS**: trusted, auto-managed certificates end to end ([08](08-tls-and-certificates.md), R-TLS-*).
|
||||
- **Supply chain**: pin images by digest where automation allows; prefer known-good upstreams;
|
||||
updates flow through policy, not ad-hoc pulls ([11](11-automated-updates.md)).
|
||||
- **Policy enforcement (admission)**: **Kyverno** runs in-cluster as an admission controller and
|
||||
rejects non-conforming resources at reconcile time — e.g. a namespace missing its default-deny
|
||||
baseline, a plaintext Secret, a workload without a required egress-class label. The *same* policies
|
||||
gate CI ([14](14-testing-and-conformance.md), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md)),
|
||||
so invariants are enforced both before merge and at admission (defense in depth). Enforcement
|
||||
complements — never replaces — the runtime behavior tests.
|
||||
- **Auditability**: isolation, egress path, cert validity, and backup integrity are all
|
||||
demonstrable (R-SEC-03) via Hubble, cert status, and DR drills.
|
||||
|
||||
## Secrets management (R-SEC-01)
|
||||
|
||||
**SOPS + age**, native to Flux:
|
||||
- Secrets are committed to git **encrypted** with `age` recipients; Flux decrypts them at
|
||||
reconciliation using an in-cluster age key. Plaintext never enters git.
|
||||
- The age private key is bootstrapped into the cluster out-of-band (documented bootstrap step, one
|
||||
of the few sanctioned non-GitOps actions) and stored as a Kubernetes Secret Flux can read.
|
||||
- **Never** print decrypted secret material into logs, PR text, or agent output (see
|
||||
[AGENTS.md §5](../AGENTS.md)).
|
||||
|
||||
### Secret categories to manage
|
||||
- VPN credentials (WireGuard keys / OpenVPN creds) for the egress model.
|
||||
- DNS provider API token (Cloudflare) for ACME DNS-01 + ExternalDNS.
|
||||
- SMTP credentials for Alertmanager email.
|
||||
- Service credentials (Nextcloud, Gitea admin, UniFi, Semaphore, Kasm, DB passwords).
|
||||
- Off-cluster backup target credentials (S3/NFS) for Velero/Longhorn.
|
||||
- Grafana admin, and any API tokens the C2 UI needs.
|
||||
|
||||
### Rotation
|
||||
- Rotating a secret = re-encrypting with SOPS and committing; Flux rolls it out. Where a secret
|
||||
change requires a workload restart, that is triggered by the reconcile (e.g. checksum annotation).
|
||||
- VPN keys, DNS tokens, and SMTP creds SHOULD have a documented rotation procedure.
|
||||
|
||||
## Alternatives considered (see ADR-0007)
|
||||
- **Sealed Secrets** — viable; SOPS chosen for Flux-native decryption and multi-recipient/age
|
||||
simplicity and readable diffs.
|
||||
- **External Secrets Operator + external vault** — heavier; deferred until a real external secret
|
||||
store exists. May be revisited via ADR.
|
||||
|
||||
## RBAC & tenancy
|
||||
- Single operator + platform automation; no multi-tenant RBAC now (out of scope, [00](00-vision-and-scope.md)).
|
||||
- Platform controllers (Flux, cert-manager, ExternalDNS, Velero, KubeVirt) get scoped permissions;
|
||||
application workloads get near-zero cluster permissions.
|
||||
- The C2 UI's operational actions run through a constrained, audited path — not a shared admin
|
||||
kubeconfig handed to a browser.
|
||||
|
||||
## Node & host security
|
||||
- Nodes baselined via cloud-init/Ansible ([02](02-environments-and-provisioning.md)); SSH key-only,
|
||||
minimal packages, required sysctls only.
|
||||
- Host-level changes are IaC, not hand edits (drift + auditability).
|
||||
|
||||
## Auditability requirements (R-SEC-03) — what an operator must be able to prove
|
||||
- **Isolation**: "service X cannot reach service Y" — via Hubble flow logs + a denied-probe test.
|
||||
- **Egress path**: "service X's public IP is the VPN's" — via an egress IP check.
|
||||
- **Kill-switch**: "with VPN down, X leaks nothing" — via a fault-injection test.
|
||||
- **TLS**: "every endpoint serves a valid, non-self-signed, unexpired cert" — via cert status +
|
||||
external check.
|
||||
- **Backups**: "irreplaceable data is restorable" — via a DR drill.
|
||||
|
||||
These proofs are part of the platform acceptance suite, not one-time checks.
|
||||
|
||||
## Acceptance gates
|
||||
- No plaintext secret exists anywhere in git history (scanned).
|
||||
- Flux decrypts SOPS secrets and workloads consume them without manual steps.
|
||||
- Least-privilege RBAC verified (no workload has broad cluster rights).
|
||||
- The five auditability proofs above can be produced on demand.
|
||||
@@ -0,0 +1,61 @@
|
||||
# 08 — TLS & Certificates
|
||||
|
||||
Serves: **R-TLS-01/02/03**, **R-DNS-01**. Related: [ADR-0006 (cert-manager + Let's Encrypt)](../decisions/adr-0006-tls-cert-manager-letsencrypt.md).
|
||||
|
||||
## Requirement recap
|
||||
|
||||
Every externally reachable service presents TLS from a **credible, trusted, non-self-signed**
|
||||
authority, with certificates **auto issued, renewed, and revoked** — including for **internal-only**
|
||||
services (R-TLS-01/02/03).
|
||||
|
||||
## Solution: cert-manager + Let's Encrypt via DNS-01
|
||||
|
||||
- **cert-manager** runs in the cluster and manages the full certificate lifecycle.
|
||||
- **Let's Encrypt** is the trusted, non-self-signed CA (publicly trusted; free; automatable) —
|
||||
satisfies R-TLS-01.
|
||||
- **ACME DNS-01 challenge** against the `keithjackson.dev` DNS zone (Cloudflare or equivalent, per
|
||||
OQ-4). DNS-01 is the key choice because it validates domain control **without exposing the service
|
||||
publicly** — so internal-only services get real, trusted certs (R-TLS-03), and **wildcard**
|
||||
certificates (`*.home.keithjackson.dev`) are possible.
|
||||
|
||||
## Issuance model
|
||||
|
||||
- A `ClusterIssuer` for Let's Encrypt production (and one for staging, used during bring-up to avoid
|
||||
rate limits).
|
||||
- Certificates are requested declaratively — either a **wildcard** `*.home.keithjackson.dev` cert
|
||||
shared by the Gateway, or **per-service** certs, decided per ADR-0006. Recommendation: a wildcard
|
||||
cert for `*.home.keithjackson.dev` terminated at the Gateway simplifies onboarding (a new service
|
||||
needs no new cert), with per-service certs available where isolation warrants.
|
||||
- The DNS provider API token is a SOPS-encrypted secret ([07](07-security-and-secrets.md)).
|
||||
|
||||
## Renewal & revocation (R-TLS-02)
|
||||
|
||||
- cert-manager renews well before expiry automatically; no human action.
|
||||
- Revocation: removing/rotating a certificate is a declarative change; cert-manager handles
|
||||
issuance of replacements. Expiry monitoring is an alert condition ([10](10-observability-and-alerting.md), R-OBS-04).
|
||||
|
||||
## Where TLS terminates
|
||||
|
||||
- **North-south**: at the **Gateway API** listener (implemented by Cilium, [09](09-dns-and-ingress.md)),
|
||||
using the cert-manager-issued cert. Public services are reached via the edge router forwarding to
|
||||
the Gateway; internal services resolve to the Gateway on the LAN.
|
||||
- Internal service-to-service traffic relies primarily on network policy for security; mTLS between
|
||||
workloads is **optional** future hardening (could be added via Cilium/service mesh) and is not a
|
||||
launch requirement — record as a possible future ADR.
|
||||
|
||||
## Trust chain for internal clients
|
||||
|
||||
Because certs come from Let's Encrypt (publicly trusted), internal clients trust them with no custom
|
||||
CA distribution. Split-horizon DNS ([09](09-dns-and-ingress.md)) resolves
|
||||
`<service>.home.keithjackson.dev` to the internal Gateway while the same trusted cert is served.
|
||||
|
||||
## Acceptance gates
|
||||
- `https://<service>.home.keithjackson.dev` serves a valid, non-self-signed, unexpired certificate
|
||||
(verified by an external TLS checker), including for internal-only services.
|
||||
- Certificates renew automatically ahead of expiry with no manual step (observed across a renewal).
|
||||
- Certificate expiry and issuance failures raise alerts.
|
||||
- Onboarding a new service requires no manual certificate step.
|
||||
|
||||
## Open questions
|
||||
- **OQ-4** — confirm DNS provider (Cloudflare assumed) and an API token scoped for DNS-01.
|
||||
- Wildcard vs. per-service certs as the default (pinned in ADR-0006).
|
||||
@@ -0,0 +1,71 @@
|
||||
# 09 — DNS & Ingress
|
||||
|
||||
Serves: **R-DNS-01/02/03**, **R-C2-01**, works with **R-TLS-***. Related:
|
||||
[ADR-0002 (Cilium/Gateway API)](../decisions/adr-0002-cni-cilium.md).
|
||||
|
||||
## Naming model
|
||||
|
||||
- All services live at `<service>.home.keithjackson.dev` (e.g. `plex.home.keithjackson.dev`).
|
||||
- The **root** `home.keithjackson.dev` serves the **Command & Control UI** ([12](12-command-center-ui.md)).
|
||||
- Public apex `*.keithjackson.dev` resolves to the network's public IP; `*.home.keithjackson.dev`
|
||||
is the platform's zone.
|
||||
|
||||
## DNS automation (R-DNS-02)
|
||||
|
||||
- **ExternalDNS** watches Gateway/HTTPRoute (and Service) resources and manages records in the
|
||||
authoritative DNS provider (Cloudflare, per OQ-4) automatically — a new service's DNS appears from
|
||||
its declared intent, no manual record editing.
|
||||
- Records for `*.home.keithjackson.dev` are created/updated/removed as services come and go.
|
||||
|
||||
## Split-horizon resolution (R-DNS-03)
|
||||
|
||||
Internal and external answers differ:
|
||||
- **External** (`*.home.keithjackson.dev` from the internet): resolves to the public IP; reaches
|
||||
the edge router, which forwards `public`-class services to the in-cluster Gateway.
|
||||
- **Internal** (LAN clients): resolves `<service>.home.keithjackson.dev` to the **internal Gateway
|
||||
address** directly, avoiding hairpinning through the public IP.
|
||||
- **PiHole** (itself an onboarded service) provides internal LAN DNS and the split-horizon answers
|
||||
for `.home.keithjackson.dev`; the trusted Let's Encrypt cert is served either way ([08](08-tls-and-certificates.md)).
|
||||
|
||||
> DNS is thus both a platform capability *and* a hosted service. Bootstrapping order matters: the
|
||||
> platform needs working DNS before PiHole is fully managed — handled by the bring-up sequence in
|
||||
> [`roadmap/phases.md`](../roadmap/phases.md).
|
||||
|
||||
## Ingress (north-south) via Gateway API
|
||||
|
||||
- **Gateway API implemented by Cilium** is the ingress layer (no separate ingress controller;
|
||||
reuses the CNI, [ADR-0002](../decisions/adr-0002-cni-cilium.md)).
|
||||
- A shared **Gateway** defines HTTPS listeners on `*.home.keithjackson.dev` terminating TLS with the
|
||||
cert-manager cert ([08](08-tls-and-certificates.md)).
|
||||
- Each service contributes an **HTTPRoute** mapping its hostname to its Service — rendered from the
|
||||
ServiceDefinition, not hand-authored.
|
||||
- **Exposure class** is declared per service:
|
||||
- `internal` — reachable only on the LAN (no edge port-forward; internal DNS only).
|
||||
- `public` — additionally reachable from the internet via the edge router forwarding 80/443 to the
|
||||
Gateway.
|
||||
|
||||
## Edge integration
|
||||
|
||||
- The edge router/firewall forwards inbound 80/443 for `public` services to the Gateway's address
|
||||
and applies multi-WAN inbound rules as needed (OQ-1).
|
||||
- `internal` services are never port-forwarded; they rely on internal DNS + Gateway on the LAN.
|
||||
- A stable ingress VIP for the Gateway is provided by the cluster's LB strategy (Cilium
|
||||
L2 announcement / BGP / a chosen LB); pin the mechanism in the networking bring-up.
|
||||
|
||||
## Interaction with default-deny
|
||||
|
||||
Ingress opens **north-south** paths to the specific service; it does **not** relax **east-west**
|
||||
default-deny ([03](03-networking-isolation-and-egress.md)). A service being publicly reachable does
|
||||
not let other in-cluster services reach it unless an explicit allow rule exists.
|
||||
|
||||
## Acceptance gates
|
||||
- A new service's `<service>.home.keithjackson.dev` record appears automatically and routes to it.
|
||||
- `internal` services are unreachable from the public internet; `public` services are reachable and
|
||||
TLS-valid.
|
||||
- Internal clients resolve to the internal Gateway (no public hairpin) and get a trusted cert.
|
||||
- The root `home.keithjackson.dev` serves the C2 UI.
|
||||
|
||||
## Open questions
|
||||
- **OQ-1/OQ-4** — edge router product + DNS provider API (affects ExternalDNS provider config and
|
||||
inbound WAN rules).
|
||||
- Gateway LB/VIP mechanism (Cilium L2 vs BGP) — pin during networking bring-up.
|
||||
@@ -0,0 +1,69 @@
|
||||
# 10 — Observability & Alerting
|
||||
|
||||
Serves: **R-OBS-01/02/03/04**, supports **R-C2-02**, **R-SEC-03**.
|
||||
|
||||
## Stack
|
||||
|
||||
- **kube-prometheus-stack** — Prometheus (metrics), Alertmanager (routing/notify), Grafana
|
||||
(dashboards), and the standard Kubernetes exporters/rules.
|
||||
- **Loki** — log aggregation.
|
||||
- **Grafana Alloy** (or Promtail-class agent) — ships node/pod/VM logs to Loki.
|
||||
- **Hubble** — network flow visibility from Cilium ([03](03-networking-isolation-and-egress.md)).
|
||||
- **Alertmanager → SMTP** — issues reported by **email** (R-OBS-02).
|
||||
- **Grafana** — dashboards for services and platform health (R-OBS-03).
|
||||
|
||||
## What is monitored (R-OBS-01/04)
|
||||
|
||||
Application services:
|
||||
- Up/health, resource use, request/error signals where exposed, and per-service SLIs declared in the
|
||||
ServiceDefinition (a service can declare the alerts it cares about).
|
||||
|
||||
Platform health (R-OBS-04) — first-class alerts:
|
||||
- **Nodes**: down/unreachable, resource pressure, disk fill.
|
||||
- **Storage (Longhorn)**: degraded/faulted volumes, replica rebuilds, capacity thresholds.
|
||||
- **GitOps (Flux)**: reconciliation failures, drift, suspended resources.
|
||||
- **Certificates (cert-manager)**: upcoming expiry, issuance failures.
|
||||
- **Networking**: VPN tunnel down, egress-gateway unhealthy, spikes in denied flows (Hubble),
|
||||
NetworkPolicy reconcile errors.
|
||||
- **Backups (Velero/Longhorn)**: backup job failure, missed schedule, restore-test failures.
|
||||
- **KubeVirt**: VM crash-loops / failed launches.
|
||||
|
||||
## Alert delivery (R-OBS-02)
|
||||
|
||||
- Alertmanager routes by severity to **email** (SMTP creds are SOPS-encrypted, [07](07-security-and-secrets.md)).
|
||||
- Severity model: `critical` (page-worthy: data risk, outage, VPN leak potential, cert about to
|
||||
expire) vs `warning` (degraded, capacity, needs attention). Both go to email; only routing/subject
|
||||
differs.
|
||||
- Alerts include enough context (runbook link, affected object) to act without spelunking.
|
||||
|
||||
## Dashboards (R-OBS-03)
|
||||
|
||||
Grafana dashboards, surfaced in the C2 UI ([12](12-command-center-ui.md)):
|
||||
- **Platform overview**: nodes, cluster capacity, Flux status, cert expiry, backup status.
|
||||
- **Storage**: Longhorn volume health, capacity, replica state.
|
||||
- **Network**: Hubble flows, egress/VPN health, denied-flow trends.
|
||||
- **Per-service**: one row/dashboard per catalog service with its declared SLIs.
|
||||
|
||||
## Observability as onboarding output
|
||||
|
||||
The ServiceDefinition ([13](13-service-catalog.md)) declares observability expectations; the
|
||||
platform renders **ServiceMonitors/PodMonitors**, log scraping, dashboards, and alert rules from it.
|
||||
A newly onboarded service is monitored **by default**, not as a follow-up task (part of the golden
|
||||
path, [01](01-architecture-overview.md)).
|
||||
|
||||
## Retention & sizing
|
||||
- Metrics/log retention set per environment overlay (dev shorter, prod longer). Loki retention
|
||||
requires a compactor with retention enabled — do not rely on a period setting alone.
|
||||
- Monitoring storage lives on Longhorn; ensure its own capacity is alerted on (don't let the monitor
|
||||
blind itself).
|
||||
|
||||
## Auditability tie-in (R-SEC-03)
|
||||
Hubble + Prometheus + cert status provide the evidence for the isolation/egress/TLS/backup proofs
|
||||
required by [07](07-security-and-secrets.md).
|
||||
|
||||
## Acceptance gates
|
||||
- Every catalog service emits metrics + logs and has at least one meaningful alert that fires
|
||||
correctly in a fault test.
|
||||
- Platform-health alerts (node down, Longhorn degraded, Flux failure, cert expiry, VPN down, backup
|
||||
failure) each fire and email correctly in an injected-fault test.
|
||||
- Grafana dashboards render for platform and each service; the C2 UI links them.
|
||||
@@ -0,0 +1,76 @@
|
||||
# 11 — Automated Updates
|
||||
|
||||
Serves: **R-UPD-01/02/03**. Related: [ADR-0009](../decisions/adr-0009-automated-image-updates.md).
|
||||
|
||||
## The requirement, precisely
|
||||
|
||||
> Services must update to latest **stable** versions **on a schedule**, with **no human, no AI, and
|
||||
> no hand-authored GitOps change**. (R-UPD-01/02)
|
||||
|
||||
This has a subtle tension with GitOps (where "the desired state is what's in git"). We resolve it
|
||||
explicitly below.
|
||||
|
||||
## Resolving "no GitOps change" vs. GitOps
|
||||
|
||||
There are two interpretations; we honor the operator's intent (hands-off), and ADR-0009 records the
|
||||
mechanism:
|
||||
|
||||
- **A human/AI authoring a version bump = prohibited.** No person and no chatbot edits a tag.
|
||||
- **An automation controller advancing a version under a policy declared once = permitted**, because
|
||||
the *operator's* declared intent (the update policy / semver range) is the stable, human-authored
|
||||
thing; the controller merely resolves it. Two mechanisms, chosen by artifact type:
|
||||
|
||||
### 1. Container image tags → Flux image-update-automation
|
||||
- Per service, an **ImagePolicy** declares the acceptable range/channel (e.g. semver `>=1.0.0`
|
||||
minor/patch, or a `stable` tag pattern). Flux's image-reflector scans the registry; image-
|
||||
automation advances the running version automatically **on its scan interval** (the schedule).
|
||||
- Where automation writes the resolved version back to git, that commit is machine-authored under
|
||||
the declared policy — no human/AI touches it (satisfies R-UPD-02 in spirit and practice).
|
||||
- Alternatively, digest-tracking of a mutable stable tag avoids per-update commits entirely. ADR-0009
|
||||
picks the default and the per-service escape hatch.
|
||||
|
||||
### 2. Helm charts / manifests → Renovate
|
||||
- **Renovate** watches chart and dependency versions and opens PRs; allowed update classes
|
||||
**auto-merge** on green CI. This keeps platform components current with no human authoring.
|
||||
- Chart *major* bumps (potentially breaking) can require review — configurable per policy.
|
||||
|
||||
### 3. VM guests → in-guest automation
|
||||
- KubeVirt guests auto-update OS packages inside the guest on a schedule (unattended-upgrades / dnf-
|
||||
automatic) declared in the VM's cloud-init ([06](06-vm-workloads-kubevirt.md)). Base-image
|
||||
promotion follows the same automated philosophy where a golden-image pipeline exists.
|
||||
|
||||
## Per-service policy (R-UPD-03)
|
||||
|
||||
Each ServiceDefinition ([13](13-service-catalog.md)) declares an **update policy**:
|
||||
- **channel/range**: which versions are "stable" and acceptable (semver range, tag regex).
|
||||
- **cadence**: how often to check/apply (the schedule).
|
||||
- **hold/pin**: temporarily freeze a service (e.g. a known-bad upstream release).
|
||||
- **rollout & health**: how the new version rolls out and what health gates it.
|
||||
- **auto-rollback**: revert on failed health so a bad upstream release self-heals.
|
||||
|
||||
## Safety rails (so hands-off ≠ reckless)
|
||||
|
||||
- **Stable only**: never track pre-release/edge tags unless explicitly opted in per service.
|
||||
- **Health-gated rollout**: a failing update is caught by health checks and rolled back (R-UPD-03),
|
||||
and raises an alert ([10](10-observability-and-alerting.md)).
|
||||
- **Backups precede risk**: state is on Longhorn with backups, so a bad update never risks data
|
||||
([04](04-storage-and-state.md)).
|
||||
- **Observability**: update events (image advanced, chart merged, rollback) are visible and, on
|
||||
failure, alert by email.
|
||||
- **Pin escape hatch**: an operator can pin a service via git if an upstream release misbehaves —
|
||||
the one sanctioned human touch, and it *stops* automation rather than performing an update.
|
||||
|
||||
## Interaction with default-deny & egress
|
||||
Updates change image versions, not network posture. A VPN-class service stays VPN-class across
|
||||
updates; policies are reconciled from the (unchanged) ServiceDefinition.
|
||||
|
||||
## Acceptance gates
|
||||
- A service advances to a newer stable version automatically, on schedule, with no human/AI edit.
|
||||
- A deliberately bad update is caught by health checks, auto-rolled-back, and alerts by email.
|
||||
- A pinned service does **not** advance.
|
||||
- Platform components (charts) stay current via auto-merged Renovate PRs.
|
||||
|
||||
## Open questions
|
||||
- Exact default in ADR-0009: write-back commits vs. digest-tracking mutable tags (trade readable
|
||||
history vs. zero-commit updates).
|
||||
- Registry rate-limit / credential handling for image scanning at cadence.
|
||||
@@ -0,0 +1,64 @@
|
||||
# 12 — Command & Control (C2) Center UI
|
||||
|
||||
Serves: **R-C2-01/02**, surfaces **R-OBS-***, **R-GITOPS-***.
|
||||
|
||||
## Purpose
|
||||
|
||||
The root `home.keithjackson.dev` is the operator's front door: a single UI that shows what's running
|
||||
in the cluster, links to every service, and surfaces platform health and safe operational actions
|
||||
(R-C2-01/02).
|
||||
|
||||
## Composition (assemble, don't reinvent)
|
||||
|
||||
The C2 center is a curated composition of best-in-class UIs behind the root domain, not a bespoke
|
||||
app to build from scratch:
|
||||
|
||||
- **Homepage (gethomepage.dev)** — the landing dashboard at the root: a live catalog of services
|
||||
with links, status pings, and per-service widgets (many of the catalog services have native
|
||||
Homepage widgets: Plex, Sonarr, Radarr, qBittorrent, PiHole, Gitea, etc.). This is the primary
|
||||
R-C2-01 surface.
|
||||
- **Grafana** — embedded platform + per-service dashboards (R-OBS-03) for health at a glance.
|
||||
- **Headlamp** (or a comparable Kubernetes web UI) — a controlled Kubernetes console for viewing
|
||||
workloads/VMs and performing scoped operational actions (restart, scale, inspect) — the R-C2-02
|
||||
"controlled operational actions where safe" surface.
|
||||
- **Flux status** — GitOps reconciliation state (via Headlamp's Flux view, the Flux UI, or a
|
||||
Homepage widget) so the operator sees drift/errors.
|
||||
- **Hubble UI** (optional) — network flow visualization for isolation troubleshooting.
|
||||
|
||||
Each sub-UI is itself an onboarded, TLS-protected, access-controlled service; the root aggregates and
|
||||
links them. ADR-worthy only if a custom portal is later preferred over Homepage.
|
||||
|
||||
## Data the C2 surfaces (R-C2-02)
|
||||
|
||||
- **Service catalog**: every running service, its URL, its health, its egress class, its exposure.
|
||||
- **Platform health**: node status, Longhorn capacity/health, Flux reconciliation, cert expiry,
|
||||
backup status, VPN tunnel state — sourced from the observability stack ([10](10-observability-and-alerting.md)).
|
||||
- **Alerts**: current firing alerts (mirrors what email delivers).
|
||||
|
||||
## Actions (safe, audited)
|
||||
|
||||
Operational actions exposed through the C2 must be **scoped and audited**, never a raw cluster-admin
|
||||
kubeconfig in a browser:
|
||||
- Restart/scale a workload, start/stop a KubeVirt VM, trigger a Flux reconcile, acknowledge an
|
||||
alert.
|
||||
- Destructive actions (delete, offboard) require explicit confirmation and are logged.
|
||||
- Onboarding itself remains a **git PR** (GitOps), not a C2 button — the C2 may *link* to the
|
||||
catalog/PR flow but does not bypass GitOps ([05](05-gitops-and-delivery.md)).
|
||||
|
||||
## Access control
|
||||
|
||||
- The C2 root and its sub-UIs are `internal` exposure by default (LAN-only) unless the operator
|
||||
explicitly makes a component `public`.
|
||||
- Authentication in front of the C2 (and sensitive sub-UIs like Headlamp) — SSO/reverse-proxy auth —
|
||||
is required; the mechanism (e.g. an identity-aware proxy) is pinned during the C2 workstream.
|
||||
- Trusted TLS on the root and all sub-UIs ([08](08-tls-and-certificates.md)).
|
||||
|
||||
## Acceptance gates
|
||||
- `https://home.keithjackson.dev` serves the C2 landing UI over trusted TLS.
|
||||
- Every running service appears with a working link and live status.
|
||||
- Platform health (nodes, storage, Flux, certs, backups, VPN) is visible without leaving the C2.
|
||||
- Safe operational actions work and are access-controlled + audited; destructive actions confirm.
|
||||
|
||||
## Open questions
|
||||
- Auth mechanism for the C2 (identity-aware proxy vs per-app auth) — pin in the C2 workstream.
|
||||
- Custom portal vs. Homepage-as-root long term (ADR if we diverge from Homepage).
|
||||
@@ -0,0 +1,112 @@
|
||||
# 13 — Service Catalog & the ServiceDefinition
|
||||
|
||||
Serves: **R-GITOPS-02**, ties together **R-NET-***, **R-STATE-***, **R-TLS-***, **R-DNS-***,
|
||||
**R-OBS-***, **R-UPD-***. The catalog is the operator-facing onboarding surface.
|
||||
|
||||
## The ServiceDefinition abstraction
|
||||
|
||||
Onboarding a service is adding/editing one **ServiceDefinition** under `catalog/` in the
|
||||
implementation repo. It captures *intent*; the platform compiles it into concrete resources
|
||||
(Deployment or KubeVirt VM, PVCs, CiliumNetworkPolicies, egress routing, HTTPRoute, Certificate, DNS
|
||||
record, ServiceMonitor + dashboards + alerts, and update policy). Operators declare **what**;
|
||||
platform provides the **how**. This keeps every requirement wired from a single reviewable object.
|
||||
|
||||
### Fields (canonical schema — implementers render manifests from these)
|
||||
|
||||
```yaml
|
||||
name: <string> # DNS label → <name>.home.keithjackson.dev
|
||||
kind: container | vm # PaaS (default) or IaaS via KubeVirt (R-PLAT-02/03)
|
||||
source:
|
||||
image: <repo/image> # container: image ref
|
||||
# or vmTemplate: <image/source> # vm: CDI source image
|
||||
exposure: internal | public # LAN-only vs edge port-forwarded (R-DNS/R-NET)
|
||||
egress: internet | vpn # egress class (R-NET-05); vpn ⇒ kill-switch (R-NET-06)
|
||||
storage:
|
||||
- name: config
|
||||
size: <e.g. 5Gi>
|
||||
class: longhorn # replicated app/config state (R-STATE-02)
|
||||
- name: media # optional bulk tier (R-STATE-04)
|
||||
class: bulk
|
||||
dependencies: # explicit allow rules; everything else denied (R-NET-01/02)
|
||||
- to: <service>
|
||||
port: <n>
|
||||
protocol: tcp|udp
|
||||
ingressRoutes: # hostnames/paths to expose (rendered to HTTPRoute + TLS + DNS)
|
||||
- host: <name>.home.keithjackson.dev
|
||||
port: <servicePort>
|
||||
update:
|
||||
channel: stable # never pre-release unless opted in (R-UPD-01)
|
||||
range: <semver range | tag regex># acceptable versions
|
||||
cadence: <schedule> # how often to check/apply
|
||||
autoRollbackOnFailedHealth: true # R-UPD-03
|
||||
pin: <optional version> # operator freeze escape hatch
|
||||
observability:
|
||||
metrics: true
|
||||
slis: [ up, ... ] # per-service alert intents (R-OBS-01)
|
||||
backup:
|
||||
policy: <ref to backup schedule> # off-cluster (R-STATE-03); or 'reacquirable' for bulk media
|
||||
```
|
||||
|
||||
> The schema is the **contract** between the operator surface and the implementation. Changing it is
|
||||
> an interface change — coordinate per [`agents/coordination-protocol.md`](../agents/coordination-protocol.md).
|
||||
|
||||
## Default posture for every service (unless overridden)
|
||||
- `kind: container`, `exposure: internal`, `egress: internet`.
|
||||
- **Default-deny**: no `dependencies` ⇒ can reach only DNS ([03](03-networking-isolation-and-egress.md)).
|
||||
- Longhorn-backed config PVC; off-cluster backup; trusted TLS; auto-DNS; monitored; auto-updated on
|
||||
`stable`.
|
||||
|
||||
## The initial catalog (13 services)
|
||||
|
||||
Egress and dependencies below are **normative intent**; implementers render the exact policies.
|
||||
"Bulk" = large media tier ([04](04-storage-and-state.md)). All get trusted TLS + auto-DNS + monitoring
|
||||
+ auto-update unless noted.
|
||||
|
||||
| # | Service | kind | exposure | egress | Storage | Key dependencies (explicit allows) | Notes |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 1 | **plex** | container | public | **internet** | config: longhorn (small, replicated); media: **bulk** | — (clients reach it; Seerr → Plex) | Largest service; media on bulk tier, metadata DB on Longhorn ([04](04-storage-and-state.md)) |
|
||||
| 2 | **sonarr** | container | internal | **vpn** | config: longhorn; media/downloads: bulk | → qbittorrent, → jackett; ← seerr | VPN + kill-switch (R-NET-06) |
|
||||
| 3 | **radarr** | container | internal | **vpn** | config: longhorn; media/downloads: bulk | → qbittorrent, → jackett; ← seerr | VPN + kill-switch |
|
||||
| 4 | **jackett** | container | internal | **vpn** | config: longhorn | ← sonarr, ← radarr | VPN + kill-switch (indexer) |
|
||||
| 5 | **seerr** | container | public | internet | config: longhorn | → sonarr:8989, → radarr:7878, → plex:32400 | Request UI; talks to arr + Plex |
|
||||
| 6 | **pihole** | container | internal | internet | config: longhorn | (provides DNS to cluster/LAN) | Also a platform DNS dependency ([09](09-dns-and-ingress.md)) |
|
||||
| 7 | **nextcloud** | container | public | internet | data: longhorn (irreplaceable → backed up) | → its DB | File sync; DR-critical data |
|
||||
| 8 | **minecraft** | container | public | internet | world: longhorn (backed up) | — | Exposes game port via edge |
|
||||
| 9 | **unifi** | container | internal | internet | config: longhorn | — | UniFi network controller |
|
||||
| 10 | **semaphore** | container | internal | internet | config/db: longhorn | — | Ansible UI |
|
||||
| 11 | **kasm** | container | internal | internet | config: longhorn | — | Kasm Workspaces (may need more resources; container-first, VM if required) |
|
||||
| 12 | **gitea** | container | public | internet | repos/db: longhorn (irreplaceable → backed up) | — | Git hosting; DR-critical |
|
||||
| 13 | **qbittorrent** | container | internal | **vpn** | config: longhorn; downloads: **bulk** | ← sonarr, ← radarr | VPN + kill-switch; download client |
|
||||
|
||||
**VPN-class set (R-NET-05/06):** sonarr, radarr, jackett, qbittorrent — all egress via VPN with an
|
||||
enforced kill-switch. Their inter-dependencies (arr → qbittorrent/jackett) are **east-west allows**
|
||||
within the deny-by-default fabric; being VPN-class governs *egress*, not intra-cluster reachability.
|
||||
|
||||
**Dependency graph (explicit allows only; all else denied):**
|
||||
```
|
||||
seerr → sonarr, radarr, plex
|
||||
sonarr → qbittorrent, jackett
|
||||
radarr → qbittorrent, jackett
|
||||
(everything → DNS only, by default)
|
||||
```
|
||||
|
||||
**"…but not limited to":** the catalog is open-ended. New services follow the same ServiceDefinition
|
||||
and inherit all guarantees automatically — that is the point of the abstraction.
|
||||
|
||||
## Cross-cutting notes
|
||||
- **Plex media** is the R-STATE-04 case: keep the metadata DB on replicated Longhorn; keep the large
|
||||
library on the bulk tier; downloads stage on bulk and are handed off by the arr stack.
|
||||
- **DR-critical data** (Nextcloud, Gitea, Minecraft worlds, PiHole config, service DBs) MUST have
|
||||
off-cluster backups ([04](04-storage-and-state.md)); bulk media may be `reacquirable` per operator
|
||||
choice (OQ-2).
|
||||
- **Public exposure** (Plex, Seerr, Nextcloud, Minecraft, Gitea) requires edge port-forwarding
|
||||
([09](09-dns-and-ingress.md)); everything else stays `internal`.
|
||||
- **Kasm** is resource-heavy and may be a candidate for `kind: vm` if container mode proves
|
||||
insufficient — the ServiceDefinition makes that a one-field change.
|
||||
|
||||
## Acceptance gates
|
||||
- Each service is fully wired from its ServiceDefinition alone (compute, storage, policy, egress,
|
||||
TLS, DNS, monitoring, updates) — no manual steps beyond merge.
|
||||
- Default-deny holds; only listed dependencies work.
|
||||
- VPN-class services egress via VPN with a proven kill-switch.
|
||||
- Stateful services survive destroy/rebuild; DR-critical data restores from off-cluster backup.
|
||||
@@ -0,0 +1,168 @@
|
||||
# 14 — Testing & Conformance
|
||||
|
||||
Serves: **R-TEST-01..06**, and is the verification mechanism for **every other requirement**.
|
||||
Related: [ADR-0010](../decisions/adr-0010-conformance-testing.md).
|
||||
|
||||
> **This spec changes what "done" means.** Nothing is accepted on a manual demo, a screenshot, or
|
||||
> "it reconciled." Every requirement is proven by an **automated, reproducible, behavior-based test**
|
||||
> that a machine can run on demand and that is tagged with the requirement ID(s) it verifies.
|
||||
|
||||
## Principles
|
||||
|
||||
1. **Behavior-based (R-TEST-01).** Tests assert *observable system behavior* — Given a state, When
|
||||
an action, Then an outcome — not implementation internals. "A pod with no allow rules cannot reach
|
||||
Plex" is a behavior; "a CiliumNetworkPolicy object exists" is not (it may exist and still not
|
||||
enforce). Test the behavior.
|
||||
2. **Reproducible (R-TEST-02).** Deterministic, hermetic, version-pinned, single-command, and
|
||||
environment-agnostic. Any conforming cluster + a kubeconfig ⇒ the same pass/fail. No flakiness
|
||||
budget; eventual-consistency is handled with bounded polling, not sleeps-and-hope.
|
||||
3. **Conformance gates delivery (R-TEST-03).** The suite blocks merges (for touched requirements)
|
||||
and blocks phase exits (full phase suite). Green suite is the definition of progress.
|
||||
4. **Traceable (R-TEST-04).** Each test declares its `R-*` IDs; a generated coverage report maps
|
||||
every requirement to its verifying test(s). Uncovered requirements are visible gaps.
|
||||
5. **Fault-injection for safety-critical behavior (R-TEST-05).** Default-deny, VPN kill-switch,
|
||||
data-survival, DR restore, and auto-rollback are proven by *breaking things* and asserting
|
||||
fail-closed / no-data-loss — not by happy-path checks.
|
||||
6. **Specs are the fixed target (R-TEST-06).** Tests encode the spec's intent. Agents never weaken a
|
||||
test or a requirement to get green; a genuine inability to meet the spec is an escalation to the
|
||||
human, not a spec/test edit.
|
||||
|
||||
## Reproducibility contract (what "reproducible" concretely requires)
|
||||
|
||||
- **Single command** runs the suite (or a tagged subset) locally and in CI.
|
||||
- **Hermetic fixtures**: each test provisions its own namespace/workloads/probes and tears them down;
|
||||
it never depends on pre-existing mutable state or on another test's side effects.
|
||||
- **Deterministic**: fixed inputs, pinned tool/image versions, bounded retries with explicit
|
||||
timeouts for eventual consistency; no reliance on wall-clock timing or ordering.
|
||||
- **Isolated & parallel-safe**: unique namespaces/labels; tests can run concurrently without
|
||||
cross-talk.
|
||||
- **Machine-readable output**: JUnit (or equivalent) plus a requirement-coverage report; every
|
||||
result carries the `R-*` tags it verifies.
|
||||
- **Self-cleaning**: on pass or fail, fixtures are removed so a re-run starts clean (leaked fixtures
|
||||
are themselves a test failure).
|
||||
- **Versioned with the code**: tests live in the implementation repo next to what they verify and
|
||||
evolve only when the human-owned spec they encode changes.
|
||||
|
||||
## Test layers
|
||||
|
||||
| Layer | Runs | Verifies | Example |
|
||||
|---|---|---|---|
|
||||
| **L0 Static/policy** | Pre-apply, in CI | Manifests/ServiceDefinitions obey invariants | Every namespace has a default-deny baseline; no plaintext secrets; schema-valid ServiceDefinition |
|
||||
| **L1 Rendering** | CI, no cluster | ServiceDefinition → rendered resources match golden expectations | `egress: vpn` renders the kill-switched egress path |
|
||||
| **L2 Behavioral/integration** | Against dev/test cluster | Real runtime behavior | Default-deny probe, dependency allow/deny, egress-IP, TLS validity, DNS, PVC rebuild |
|
||||
| **L3 Fault-injection** | Against dev/test cluster | Fail-closed / no-data-loss under failure | VPN tunnel forced down ⇒ no leak; node killed ⇒ data survives; bad update ⇒ auto-rollback; DR restore |
|
||||
| **L4 Conformance suite** | CI + on-demand | Aggregate, requirement-tagged gate | The union of the above, reported per `R-*` |
|
||||
|
||||
## Behavior test catalog (normative — implementers author these as automated tests)
|
||||
|
||||
Each maps to requirement IDs; phrased as Given/When/Then intent.
|
||||
|
||||
- **Default-deny (R-NET-01):** *Given* a fresh workload with no allow rules, *When* it attempts any
|
||||
in-cluster connection except DNS, *Then* the connection fails; DNS succeeds.
|
||||
- **Explicit allow / deny-undeclared (R-NET-02):** *Given* declared `dependencies`, *When* the
|
||||
source reaches a declared dest port, *Then* it succeeds; *When* it reaches an **undeclared** dest,
|
||||
*Then* it fails.
|
||||
- **Egress class (R-NET-05):** *Given* `egress: internet` vs `egress: vpn` workloads, *When* each
|
||||
queries an external echo of its public IP, *Then* the internet one shows the home/WAN IP and the
|
||||
vpn one shows the VPN IP.
|
||||
- **VPN kill-switch (R-NET-06, L3):** *Given* a `vpn` workload with a healthy tunnel, *When* the
|
||||
tunnel is forced down, *Then* the workload has **zero** egress to the plain internet and
|
||||
`internet`-class workloads are unaffected; *When* the tunnel recovers, connectivity returns.
|
||||
- **Egress flip (R-NET-07):** *Given* a running service, *When* its `egress` class is changed in
|
||||
git and reconciled, *Then* its observed egress path changes with no workload rewrite.
|
||||
- **Data survival (R-STATE-01, L3):** *Given* a stateful service with written data, *When* the
|
||||
workload is destroyed and re-applied from git, *Then* the data is intact.
|
||||
- **Node loss (R-STATE-02, L3):** *Given* a 3-replica volume, *When* a node is removed, *Then* the
|
||||
volume stays available and the cluster reconverges.
|
||||
- **DR restore (R-STATE-03, L3):** *Given* an off-cluster backup, *When* a namespace + volumes are
|
||||
restored into a clean state, *Then* the service returns with its data.
|
||||
- **Trusted TLS (R-TLS-01/02):** *Given* any exposed endpoint (incl. `internal`), *When* its cert is
|
||||
inspected, *Then* it is valid, non-self-signed, unexpired; renewal occurs before expiry with no
|
||||
manual step.
|
||||
- **DNS/exposure (R-DNS-*):** *Given* a service, *When* resolved, *Then* the record exists and
|
||||
internal clients hit the internal Gateway; `internal` is unreachable publicly, `public` is
|
||||
reachable + TLS-valid.
|
||||
- **Hands-off update (R-UPD-01/02, L3):** *Given* an out-of-date service under a stable policy,
|
||||
*When* the scan interval elapses, *Then* it advances with no human/AI edit; *Given* a deliberately
|
||||
bad update, *Then* health-gating auto-rolls-back and an email alert fires; *Given* a pinned
|
||||
service, *Then* it does not advance.
|
||||
- **VM lifecycle (R-PLAT-03, L2/L3):** *Given* a `kind: vm` ServiceDefinition, *When* create/start/
|
||||
stop/destroy via `kubectl`/`virtctl`, *Then* it behaves; its disk survives rebuild; default-deny +
|
||||
egress class apply.
|
||||
- **Alerting (R-OBS-02/04, L3):** *Given* a monitored condition, *When* a fault is injected, *Then*
|
||||
an email alert is delivered and the dashboard reflects it.
|
||||
- **GitOps drift (R-GITOPS-03, L3):** *Given* a reconciled resource, *When* it is mutated live,
|
||||
*Then* Flux reverts it to the git-declared state.
|
||||
- **Onboarding completeness (WS-APPS):** *Given* only a merged ServiceDefinition, *Then* the service
|
||||
is fully wired (compute/storage/policy/egress/TLS/DNS/monitoring/update) with no manual step —
|
||||
asserted by the union of the relevant tests above against that service.
|
||||
|
||||
## Gating rules (R-TEST-03)
|
||||
|
||||
- **Per-change (merge gate):** the conformance subset tagged with the requirement IDs the change
|
||||
touches MUST pass before merge.
|
||||
- **Per-phase (exit gate):** the full conformance suite for the phase's requirements MUST pass
|
||||
before the phase's exit criteria are considered met ([`../roadmap/phases.md`](../roadmap/phases.md)).
|
||||
- **Regression:** the growing suite runs continuously against dev/test; a previously-green
|
||||
requirement going red blocks further delivery until fixed.
|
||||
- **Coverage:** a requirement with no passing test is **not** satisfied, regardless of implementation
|
||||
state.
|
||||
|
||||
## Tooling (pinned — [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md))
|
||||
|
||||
The approach ([ADR-0010](../decisions/adr-0010-conformance-testing.md)) is realized with two tools
|
||||
from the **Kyverno ecosystem**:
|
||||
|
||||
- **Kyverno** — **L0 static/policy gate _and_ in-cluster admission enforcer.** In CI, the Kyverno CLI
|
||||
(`kyverno apply` / `kyverno test`, incl. `kyverno-json` for ServiceDefinitions) validates manifests
|
||||
and rendered resources against platform invariants (default-deny baseline present, no plaintext
|
||||
secrets, required labels/egress-class, schema conformance). **The same policies also run in-cluster
|
||||
as an admission controller**, rejecting any non-conforming resource at reconcile time (defense in
|
||||
depth — a resource that somehow bypassed CI still cannot be admitted). Enforcement **never replaces
|
||||
behavior tests**: a policy admitting a manifest is not proof the runtime behaves (R-TEST-01), so
|
||||
L2/L3 Chainsaw tests remain mandatory. Policies are authored once and shared between the CLI gate
|
||||
and the admission controller.
|
||||
- **Chainsaw** — **L1 rendering, L2 behavioral, and L3 fault-injection.** Declarative steps
|
||||
(apply/assert/error, `script`/`command`, `try`/`catch`/`finally`), bounded polling with explicit
|
||||
timeouts, per-test namespace isolation, and automatic cleanup satisfy the reproducibility contract.
|
||||
Fault-injection (force the tunnel down, cordon/kill a node, inject a bad image, trigger a restore)
|
||||
is expressed as Chainsaw `script` steps, asserting fail-closed / no-data-loss via probe fixtures,
|
||||
with `finally` restoring state.
|
||||
|
||||
Probes are ephemeral job pods; egress-IP checks use an external echo; tests carry their `R-*` IDs
|
||||
(labels/annotations/naming) so the coverage report is generated automatically; both tools' output
|
||||
feeds JUnit + the requirement-coverage report and the CI merge / phase-exit gates. Tool versions are
|
||||
pinned for determinism.
|
||||
|
||||
### Where each layer runs
|
||||
- **L0 (Kyverno CLI) and L1 (rendering assertions)** run **in CI without a cluster** — pure manifest/
|
||||
ServiceDefinition validation; fast merge-gate feedback.
|
||||
- **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job** on the dev/test cluster.
|
||||
CI **triggers** the Job (and waits for it); the Job — not CI — holds cluster access, so CI never
|
||||
needs a broad kubeconfig. This keeps credentials in-cluster and least-privileged (R-SEC-02).
|
||||
- The Job runs under a **dedicated, scoped test ServiceAccount** whose RBAC grants exactly what the
|
||||
behavior/fault tests need (create/delete test namespaces + fixtures + probes, apply/patch
|
||||
NetworkPolicies, scale the VPN sidecar/gateway, cordon/drain a node, inject a bad image tag,
|
||||
trigger a Velero restore). This is a **privileged test identity** — it MUST be scoped, named, and
|
||||
documented, MUST NOT be reused by workloads, and its own reach is itself an audit item.
|
||||
- The Job writes **JUnit + coverage artifacts** to a known location (log stream and/or a results
|
||||
PVC/object) that CI collects to enforce the merge / phase-exit gates.
|
||||
- Fixtures are hermetic and self-cleaning (`finally` steps); a leaked fixture fails the run.
|
||||
|
||||
## Acceptance gates (for the testing capability itself)
|
||||
- A single command runs the suite and emits requirement-tagged JUnit + coverage.
|
||||
- Every `R-*` requirement maps to at least one passing behavior test (coverage report proves it).
|
||||
- Safety-critical requirements each have a **fault-injection** test that fails closed / loses no
|
||||
data.
|
||||
- Re-running the suite yields identical results (reproducibility) with no leaked fixtures.
|
||||
|
||||
## Open questions
|
||||
- Tool mix is **resolved**: Kyverno (L0) + Chainsaw (L1/L2/L3), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md).
|
||||
- Harness location is **resolved**: L2/L3 run as an **in-cluster Chainsaw Job** triggered by CI; CI
|
||||
holds no cluster credentials.
|
||||
- Kyverno-as-admission-enforcer is **resolved**: yes — the L0 policies also enforce at admission
|
||||
in-cluster (defense in depth), in addition to the CI gate.
|
||||
- Delegated to WS-TEST at implementation time (not human decisions): the **exact RBAC scope** of the
|
||||
test ServiceAccount (author it **least-privilege** for precisely the fault-injection actions it
|
||||
performs, [07](07-security-and-secrets.md)) and the **results-artifact mechanism** (log stream vs
|
||||
results PVC/object store). Both are implementer's discretion within the stated constraints.
|
||||
Reference in new issue
Block a user