Initial baseline: home platform (k3s + Longhorn) specification repository
Greenfield spec set for a GitOps-driven PaaS+IaaS home platform: vision & requirements (R-* IDs), architecture, 14 domain specs, 11 ADRs, roadmap (phases + workstreams), and AI-agent coordination (roles, protocol, definition-of-done, task board). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
37 files changed
+2830
No files matched your search
@@ -0,0 +1,55 @@
|
||||
# Architecture Decision Records (ADRs)
|
||||
|
||||
ADRs are an **append-only ledger** of the significant technology and pattern choices for this
|
||||
platform. Each records the context, the decision, the rejected alternatives, and the requirements it
|
||||
serves. **Do not edit an accepted decision in place** — supersede it with a new ADR.
|
||||
|
||||
> **ADRs are specifications and are immutable to AI agents** (R-TEST-06). Agents may *draft* a
|
||||
> proposed ADR and escalate it, but only the **human operator/architect** approves/accepts or edits
|
||||
> ADRs. See [`../AGENTS.md` §3 and §11](../AGENTS.md).
|
||||
|
||||
## Process (read before proposing a change)
|
||||
|
||||
1. To change a decision, create a **new** ADR with the next number and set `Status: Proposed`,
|
||||
`Supersedes: ADR-XXXX`. (Agents draft + escalate; the human approves.)
|
||||
2. State what new information invalidates the old decision.
|
||||
3. On approval, set the new ADR `Accepted` and the old one `Superseded by: ADR-YYYY`.
|
||||
4. Never silently re-interpret an accepted ADR (see [`../AGENTS.md` §3](../AGENTS.md)).
|
||||
|
||||
## Template
|
||||
|
||||
```markdown
|
||||
# ADR-XXXX — <title>
|
||||
- Status: Proposed | Accepted | Superseded by ADR-YYYY
|
||||
- Date: <YYYY-MM-DD>
|
||||
- Serves: <requirement IDs, e.g. R-NET-01, R-NET-06>
|
||||
- Supersedes: <ADR-ZZZZ or —>
|
||||
|
||||
## Context
|
||||
<forces, constraints, why a decision is needed>
|
||||
|
||||
## Decision
|
||||
<what we chose, stated decisively>
|
||||
|
||||
## Alternatives considered
|
||||
<options and why rejected>
|
||||
|
||||
## Consequences
|
||||
<positive, negative, follow-ups, risks>
|
||||
```
|
||||
|
||||
## Index
|
||||
|
||||
| ADR | Title | Status | Serves |
|
||||
|---|---|---|---|
|
||||
| [0001](adr-0001-kubernetes-distribution-k3s.md) | Kubernetes distribution: k3s | Accepted | R-PLAT-01 |
|
||||
| [0002](adr-0002-cni-cilium.md) | CNI & network policy & ingress: Cilium + Gateway API | Accepted | R-NET-01..07, R-DNS-01 |
|
||||
| [0003](adr-0003-storage-longhorn.md) | Distributed storage: Longhorn (+ bulk tier, Velero) | Accepted | R-STATE-01..04 |
|
||||
| [0004](adr-0004-gitops-flux.md) | GitOps engine: Flux CD | Accepted | R-GITOPS-01..03 |
|
||||
| [0005](adr-0005-vm-workloads-kubevirt.md) | VM workloads: KubeVirt + CDI | Accepted | R-PLAT-03 |
|
||||
| [0006](adr-0006-tls-cert-manager-letsencrypt.md) | TLS: cert-manager + Let's Encrypt (DNS-01) | Accepted | R-TLS-01..03 |
|
||||
| [0007](adr-0007-secrets-sops-age.md) | Secrets: SOPS + age | Accepted | R-SEC-01 |
|
||||
| [0008](adr-0008-vpn-egress-model.md) | VPN egress model: Gluetun sidecar + Cilium Egress Gateway | Accepted | R-NET-05/06 |
|
||||
| [0009](adr-0009-automated-image-updates.md) | Automated updates: Flux image automation + Renovate | Accepted | R-UPD-01..03 |
|
||||
| [0010](adr-0010-conformance-testing.md) | Conformance testing: automated, reproducible, behavior-based | Accepted | R-TEST-01..06 |
|
||||
| [0011](adr-0011-test-tooling-kyverno-chainsaw.md) | Conformance test tooling: Kyverno + Chainsaw | Accepted | R-TEST-01..05 |
|
||||
@@ -0,0 +1,32 @@
|
||||
# ADR-0001 — Kubernetes distribution: k3s
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-PLAT-01, R-PLAT-02, R-ENV-01/02/03
|
||||
- Supersedes: —
|
||||
|
||||
## Context
|
||||
The platform runs on 3 small nodes (dev: KVMs; prod: mini-PCs) and must offer a full Kubernetes API
|
||||
for both containers and VMs. It must be lightweight enough for constrained hardware yet
|
||||
production-capable, HA, and identical across dev/test and prod.
|
||||
|
||||
## Decision
|
||||
Use **k3s** as the Kubernetes distribution, deployed as a 3-node HA cluster with **embedded etcd**
|
||||
(all three nodes are servers). Bundled components that conflict with our chosen stack are disabled at
|
||||
install time: flannel (→ Cilium), kube-proxy (→ Cilium kube-proxy replacement), Traefik (→ Gateway
|
||||
API via Cilium), and servicelb where a different LB strategy is selected. Nodes are provisioned by
|
||||
OpenTofu (libvirt, dev) / bare-metal imaging (prod) + cloud-init, and joined/configured by Ansible.
|
||||
|
||||
## Alternatives considered
|
||||
- **Full kubeadm/vanilla Kubernetes** — heavier operationally for 3 small nodes; k3s gives the same
|
||||
API with far less overhead.
|
||||
- **k0s / MicroK8s / Talos** — all viable. Talos is attractive (immutable, API-driven) and may be
|
||||
revisited via a superseding ADR for prod hardening; k3s chosen now for its ubiquity, small
|
||||
footprint, single-binary simplicity, strong community, and the operator's existing familiarity.
|
||||
- **Managed cloud Kubernetes** — out of scope; this is on-prem home hardware.
|
||||
|
||||
## Consequences
|
||||
- Lightweight, HA, uniform dev↔prod topology (R-ENV-02).
|
||||
- Must explicitly disable bundled CNI/ingress/proxy to make room for Cilium + Gateway API (ADR-0002)
|
||||
— captured in the Ansible bootstrap.
|
||||
- Embedded etcd across 3 nodes tolerates single-node loss; do not reduce below 3 servers in prod.
|
||||
- Revisit Talos for prod if immutability/hardening becomes a priority.
|
||||
@@ -0,0 +1,37 @@
|
||||
# ADR-0002 — CNI, network policy & ingress: Cilium + Gateway API
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-NET-01, R-NET-02, R-NET-03, R-NET-04, R-NET-05, R-NET-06, R-NET-07, R-DNS-01, R-SEC-02
|
||||
- Supersedes: —
|
||||
|
||||
## Context
|
||||
The platform requires: default-deny east-west with explicit port-level allows (firewall-like),
|
||||
network isolation/segmentation, multi-WAN/egress routing including per-service VPN egress, and
|
||||
north-south ingress with TLS. These are the most safety-critical requirements. A single, coherent
|
||||
networking layer that does policy, egress steering, observability, and ingress reduces moving parts.
|
||||
|
||||
## Decision
|
||||
Use **Cilium** as the CNI (replacing k3s' flannel), with **kube-proxy replacement**, and adopt:
|
||||
- **CiliumNetworkPolicy / NetworkPolicy** for default-deny + explicit L3/L4 (optionally L7) allows.
|
||||
- **Cilium Egress Gateway** to steer selected pods' egress through chosen nodes/identities/SNAT IPs
|
||||
— the mechanism for egress-class routing and part of the VPN model (with ADR-0008).
|
||||
- **Hubble** for flow observability — the audit/troubleshooting tool for isolation and egress.
|
||||
- **Gateway API implemented by Cilium** for north-south ingress and TLS termination (no separate
|
||||
ingress controller).
|
||||
|
||||
## Alternatives considered
|
||||
- **Calico** — strong policy engine; lacks Cilium's integrated egress gateway + Hubble + Gateway API
|
||||
in one stack.
|
||||
- **Flannel + separate ingress-nginx/Traefik + separate egress solution** — more components, no
|
||||
identity-aware policy or built-in flow visibility.
|
||||
- **Service mesh (Istio/Linkerd) for policy/mTLS** — heavier; not needed at launch. mTLS can be
|
||||
added later via Cilium/mesh as a future ADR ([08](../docs/08-tls-and-certificates.md)).
|
||||
|
||||
## Consequences
|
||||
- One networking layer covers policy, egress steering, observability, and ingress — fewer components,
|
||||
coherent model, VMs (KubeVirt) inherit the same policy fabric.
|
||||
- Requires disabling k3s bundled flannel/kube-proxy/Traefik (see ADR-0001).
|
||||
- Egress Gateway + VPN model must be validated end-to-end including kill-switch (ADR-0008, R-NET-06).
|
||||
- Physical multi-WAN (dual ISP) remains partly an edge-router responsibility (OQ-1); the cluster
|
||||
decides egress *class*, the edge router decides physical WAN for the internet class.
|
||||
- Gateway LB/VIP mechanism (Cilium L2 announcement vs BGP) to be pinned during networking bring-up.
|
||||
@@ -0,0 +1,38 @@
|
||||
# ADR-0003 — Distributed storage: Longhorn (+ bulk tier + Velero)
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-STATE-01, R-STATE-02, R-STATE-03, R-STATE-04
|
||||
- Supersedes: —
|
||||
|
||||
## Context
|
||||
Three nodes with **local disks** and no SAN must provide durable, replicated persistent storage so
|
||||
any workload can be destroyed and rebuilt without data loss, tolerate a single node failure, and
|
||||
support off-cluster backup/DR. Plex's multi-TB media library makes uniform 3× replication
|
||||
impractical, requiring a distinct bulk tier.
|
||||
|
||||
## Decision
|
||||
- Use **Longhorn** for distributed replicated block storage across the 3 nodes; default StorageClass
|
||||
is **3 replicas**. This is the durable home for app/config state and KubeVirt VM disks (via CDI).
|
||||
- Define a separate **bulk media tier** for large, non-replicated data (Plex library, download
|
||||
staging). Its concrete backing (NAS/NFS vs. dedicated large disk vs. mixed) is decided under OQ-2;
|
||||
**Plex's metadata DB stays on 3-replica Longhorn** regardless.
|
||||
- Use **Velero** (with Longhorn backups) to push volume + resource-state backups to an **off-cluster**
|
||||
S3-compatible/NFS target on a schedule, with a **tested restore** path (DR drill).
|
||||
|
||||
## Alternatives considered
|
||||
- **Rook/Ceph** — more powerful but far heavier/operationally complex for 3 small nodes.
|
||||
- **OpenEBS (Mayastor/other)** — viable; Longhorn chosen for simplicity, strong UI, snapshot/backup
|
||||
ergonomics, and fit with small clusters.
|
||||
- **Uniform 3× replication of everything including media** — rejected: capacity-prohibitive on
|
||||
mini-PC local disks (R-STATE-04).
|
||||
- **hostPath/local-only PVs for stateful apps** — rejected: node-bound, no replication, breaks the
|
||||
rebuild guarantee.
|
||||
|
||||
## Consequences
|
||||
- Single-node loss keeps replicated data available; workloads rebuild by re-attaching PVCs
|
||||
(R-STATE-01/02).
|
||||
- A two-tier storage story must be clearly documented so implementers don't accidentally place bulk
|
||||
media on the replicated tier or app DBs on the bulk tier ([04](../docs/04-storage-and-state.md)).
|
||||
- Off-cluster backup target + its credentials/resilience are prerequisites (OQ-2); DR drill is part
|
||||
of acceptance.
|
||||
- Longhorn's own capacity/health must be monitored and alerted ([10](../docs/10-observability-and-alerting.md)).
|
||||
@@ -0,0 +1,35 @@
|
||||
# ADR-0004 — GitOps engine: Flux CD
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-GITOPS-01, R-GITOPS-02, R-GITOPS-03, R-UPD-02, R-SEC-01, R-ENV-02
|
||||
- Supersedes: —
|
||||
|
||||
## Context
|
||||
All desired state must be reconciled from git (pull-based), with drift correction, environment
|
||||
overlays (dev↔prod), encrypted secrets, and — critically — **hands-off image updates** driven by
|
||||
declared policy (R-UPD-02). The GitOps engine choice is tightly coupled to the update mechanism and
|
||||
the secrets model.
|
||||
|
||||
## Decision
|
||||
Use **Flux CD** for reconciliation, chosen decisively because it *natively* provides the two
|
||||
capabilities that matter most for this platform's hard requirements:
|
||||
- **image-update-automation** (image-reflector + image-automation controllers) for policy-driven,
|
||||
no-human image updates (R-UPD-02, ADR-0009).
|
||||
- **native SOPS decryption** for encrypted-in-git secrets (R-SEC-01, ADR-0007).
|
||||
Flux Kustomize/Helm controllers handle overlays (dev↔prod, R-ENV-02) and chart-based platform
|
||||
components.
|
||||
|
||||
## Alternatives considered
|
||||
- **Argo CD** — excellent UI and ApplicationSets; but image auto-update (Argo Image Updater) and
|
||||
secret decryption are add-ons rather than first-class, and the platform gets its UI from the C2
|
||||
composition ([12](../docs/12-command-center-ui.md)) rather than from the GitOps tool. Reasonable
|
||||
alternative; Flux wins on native image-automation + SOPS fit.
|
||||
- **Rancher Fleet / Kustomize-only + CI push** — weaker fit for the automated-update and drift
|
||||
requirements.
|
||||
|
||||
## Consequences
|
||||
- Native, first-class path to R-UPD-02 (no bolt-on for automated updates) and R-SEC-01 (SOPS).
|
||||
- GitOps status/health is surfaced in the C2 UI via Headlamp/Flux views ([12](../docs/12-command-center-ui.md)),
|
||||
not via a bundled GitOps GUI.
|
||||
- Reconciliation failures/drift are alerting conditions (R-OBS-04).
|
||||
- If a rich built-in GitOps GUI later becomes a priority, revisit via superseding ADR (Argo).
|
||||
@@ -0,0 +1,36 @@
|
||||
# ADR-0005 — VM workloads: KubeVirt + CDI
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-PLAT-03, R-PLAT-04, R-STATE-01
|
||||
- Supersedes: —
|
||||
|
||||
## Context
|
||||
Not everything fits in a container. The platform must run VMs, but the operator must manage them
|
||||
through the **same** Kubernetes API/CLI/YAML/GitOps as containers, with the same network isolation,
|
||||
egress classes, storage durability, and observability — no separate hypervisor management plane for
|
||||
application workloads.
|
||||
|
||||
## Decision
|
||||
Use **KubeVirt** to run application VMs as Kubernetes custom resources (`VirtualMachine` /
|
||||
`VirtualMachineInstance`), with **CDI (Containerized Data Importer)** to provision VM disks onto
|
||||
**Longhorn** PVCs. VMs run on the same Cilium network (inheriting default-deny + egress classes) and
|
||||
are managed by Flux like any other workload. `virtctl` provides console/lifecycle helpers alongside
|
||||
`kubectl`.
|
||||
|
||||
## Alternatives considered
|
||||
- **Proxmox / standalone hypervisor + separate IaC** — a second management plane and tooling; breaks
|
||||
the "same Kubernetes surface" requirement (R-PLAT-03). (The operator's prior environment used
|
||||
Proxmox; this platform deliberately unifies on Kubernetes instead.)
|
||||
- **Harvester (SUSE)** — HCI built on KubeVirt; heavier and opinionated about the whole node OS; k3s
|
||||
+ KubeVirt keeps one lightweight cluster for both containers and VMs.
|
||||
- **Containers only** — rejected: the requirement explicitly needs a VM escape hatch.
|
||||
|
||||
## Consequences
|
||||
- One control plane, one GitOps flow, one network/storage/observability model for containers *and*
|
||||
VMs.
|
||||
- Nodes must expose virtualization extensions (nested virt in dev KVMs; native on prod) — a node
|
||||
baseline prerequisite ([02](../docs/02-environments-and-provisioning.md)).
|
||||
- VMs are heavier than pods; capacity planning on the 3-node cluster must account for them.
|
||||
- At least one representative VM workload must be proven in dev/test so the IaaS path is real
|
||||
([06](../docs/06-vm-workloads-kubevirt.md)).
|
||||
- VM guest updates use in-guest automation (R-UPD-*), distinct from container image tag bumps.
|
||||
@@ -0,0 +1,33 @@
|
||||
# ADR-0006 — TLS: cert-manager + Let's Encrypt (DNS-01)
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-TLS-01, R-TLS-02, R-TLS-03, R-DNS-01
|
||||
- Supersedes: —
|
||||
|
||||
## Context
|
||||
Every service needs a **trusted, non-self-signed** certificate, auto issued/renewed/revoked,
|
||||
including **internal-only** services (which must not be forced public just to validate a cert).
|
||||
Wildcard coverage of `*.home.keithjackson.dev` would make onboarding certificate-free.
|
||||
|
||||
## Decision
|
||||
Use **cert-manager** with **Let's Encrypt** (a publicly trusted, free, automatable CA) via the
|
||||
**ACME DNS-01** challenge against the `keithjackson.dev` DNS zone (Cloudflare or equivalent, OQ-4).
|
||||
DNS-01 validates domain control without exposing the service and enables **wildcard** certs.
|
||||
Default: a **wildcard `*.home.keithjackson.dev`** certificate terminated at the Gateway ([09](../docs/09-dns-and-ingress.md)),
|
||||
with per-service certificates available where isolation warrants. Provide both `letsencrypt-staging`
|
||||
(bring-up) and `letsencrypt-prod` ClusterIssuers.
|
||||
|
||||
## Alternatives considered
|
||||
- **HTTP-01 challenge** — rejected: requires public exposure of each service; fails R-TLS-03 for
|
||||
internal-only services and can't do wildcards.
|
||||
- **Self-signed / private CA** — rejected: explicitly disallowed (R-TLS-01, "not self-signed").
|
||||
- **Paid CA** — unnecessary; Let's Encrypt is trusted and free.
|
||||
- **ZeroSSL/other ACME CA** — viable fallback; Let's Encrypt chosen as default.
|
||||
|
||||
## Consequences
|
||||
- Internal services get real, trusted certs; wildcard makes onboarding certificate-free (default).
|
||||
- Requires a DNS-provider API token scoped for DNS-01, stored SOPS-encrypted (OQ-4, ADR-0007).
|
||||
- Certificate expiry/issuance failures are alert conditions (R-OBS-04).
|
||||
- Internal clients trust the certs with no custom-CA distribution; split-horizon DNS serves the same
|
||||
cert internally and externally.
|
||||
- Optional future mTLS between workloads is a separate concern (Cilium/mesh), not covered here.
|
||||
@@ -0,0 +1,31 @@
|
||||
# ADR-0007 — Secrets: SOPS + age
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-SEC-01, R-GITOPS-01
|
||||
- Supersedes: —
|
||||
|
||||
## Context
|
||||
Secrets (VPN keys, DNS API token, SMTP creds, service passwords, backup-target creds) must live in
|
||||
git for GitOps but MUST NOT be plaintext. The engine is Flux (ADR-0004), which decrypts SOPS
|
||||
natively.
|
||||
|
||||
## Decision
|
||||
Encrypt secrets in git with **SOPS** using **age** recipients; **Flux decrypts in-cluster** at
|
||||
reconciliation via an age private key bootstrapped as a Kubernetes Secret. Plaintext secrets never
|
||||
enter git or logs. Secret rotation = re-encrypt + commit; Flux rolls it out (restart triggered by
|
||||
checksum annotation where needed).
|
||||
|
||||
## Alternatives considered
|
||||
- **Sealed Secrets (Bitnami)** — solid; one-way encryption to a cluster key. SOPS chosen for native
|
||||
Flux integration, multi-recipient/age simplicity, and readable, reviewable encrypted diffs.
|
||||
- **External Secrets Operator + external vault (Vault/cloud SM)** — stronger for large orgs but adds
|
||||
an external dependency and operational weight not justified for a single-operator home platform.
|
||||
May be revisited via superseding ADR if a real external secret store is introduced.
|
||||
- **Plaintext + private repo** — rejected outright (R-SEC-01).
|
||||
|
||||
## Consequences
|
||||
- Secrets are versioned, encrypted, and reconciled with the rest of GitOps state.
|
||||
- The **age private key bootstrap** is a sanctioned out-of-band step (documented; one of the few
|
||||
non-GitOps actions, [07](../docs/07-security-and-secrets.md)).
|
||||
- Key custody/rotation for the age key itself must be documented; losing it blocks decryption.
|
||||
- No secret material may appear in agent output, PRs, or logs ([AGENTS.md §5](../AGENTS.md)).
|
||||
@@ -0,0 +1,42 @@
|
||||
# ADR-0008 — VPN egress model: Gluetun sidecar (default) + Cilium Egress Gateway
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-NET-05, R-NET-06, R-NET-07
|
||||
- Supersedes: —
|
||||
|
||||
## Context
|
||||
A subset of services (sonarr, radarr, jackett, qbittorrent) must egress **only** through a commercial
|
||||
VPN, with a **kill-switch** guaranteeing no leak to the plain internet if the tunnel drops. Egress
|
||||
class must be a declarative service property (`egress: vpn`) that doesn't force a workload rewrite.
|
||||
|
||||
## Decision
|
||||
Adopt a **two-pattern** model behind the single `egress: vpn` contract, with **Gluetun sidecar as the
|
||||
default** for the initial catalog:
|
||||
- **Default — Pattern A: Gluetun sidecar per pod.** A Gluetun container holds the WireGuard/OpenVPN
|
||||
tunnel; the app shares its network namespace. Gluetun's built-in **firewall kill-switch** blocks
|
||||
all non-tunnel egress by default — satisfying R-NET-06 intrinsically and per-service.
|
||||
- **Available — Pattern B: central VPN egress gateway + Cilium Egress Gateway.** One (HA) gateway
|
||||
holds the tunnel; Cilium Egress Gateway steers `vpn`-labeled pods through it, with policy denying
|
||||
any fallback to the internet path. Use when tunnel count must be minimized.
|
||||
|
||||
The service-facing contract is identical for both: declare `egress: vpn`; the platform wires the
|
||||
chosen pattern. Kill-switch behavior is a hard acceptance gate for both.
|
||||
|
||||
## Alternatives considered
|
||||
- **Central gateway only** — fewer tunnels but a shared failure domain and more complex kill-switch
|
||||
guarantees across the fabric; kept as Pattern B, not the default.
|
||||
- **Node-level policy routing / per-node VPN** — couples egress to node identity, harder to reason
|
||||
about per-service; rejected as the primary model.
|
||||
- **App-native VPN config** — rejected: not declarative, not uniform, easy to misconfigure and leak.
|
||||
|
||||
## Consequences
|
||||
- Per-service isolation and an intrinsic kill-switch out of the box (Pattern A).
|
||||
- More sidecars/tunnels at scale (Pattern A) — Pattern B is the escape hatch if the VPN provider
|
||||
limits concurrent tunnels/egress IPs (OQ-3).
|
||||
- Kill-switch must be **fault-tested** (tunnel forced down ⇒ zero internet egress) as a standing
|
||||
acceptance gate, not a one-time check ([03](../docs/03-networking-isolation-and-egress.md), R-NET-06).
|
||||
- VPN credentials are SOPS-encrypted (ADR-0007). WireGuard preferred over OpenVPN where supported.
|
||||
|
||||
## Open questions
|
||||
- **OQ-3** — VPN provider/protocol and concurrent-tunnel/egress-IP limits (drives Pattern A vs B and
|
||||
gateway HA).
|
||||
@@ -0,0 +1,43 @@
|
||||
# ADR-0009 — Automated updates: Flux image automation + Renovate
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-UPD-01, R-UPD-02, R-UPD-03
|
||||
- Supersedes: —
|
||||
|
||||
## Context
|
||||
Services must advance to the **latest stable** version **on a schedule** with **no human, no AI, and
|
||||
no hand-authored GitOps change** (R-UPD-02). This must coexist with GitOps (git = source of truth)
|
||||
and must be safe (stable-only, health-gated, reversible).
|
||||
|
||||
## Decision
|
||||
Split by artifact type:
|
||||
- **Container image tags → Flux image-update-automation.** Per-service **ImagePolicy** declares the
|
||||
acceptable range/channel (semver range or stable tag regex). Flux's image-reflector scans the
|
||||
registry on an interval (the schedule) and image-automation advances the running version. The
|
||||
policy is the human-authored intent; the version advance is machine-performed — no human/AI edits
|
||||
a tag (R-UPD-02).
|
||||
- **Helm charts / manifests → Renovate.** Renovate opens version-bump PRs; allowed update classes
|
||||
**auto-merge** on green CI. Major/breaking bumps may require review per policy.
|
||||
- **VM guests → in-guest auto-update** (unattended-upgrades/dnf-automatic) declared in the VM's
|
||||
cloud-init ([06](../docs/06-vm-workloads-kubevirt.md)).
|
||||
|
||||
**Default write-back behavior:** image-automation commits the resolved version back to git (machine-
|
||||
authored) so history is auditable; a **per-service escape hatch** allows digest-tracking of a mutable
|
||||
stable tag for zero-commit updates where the operator prefers it. Per-service `update` policy
|
||||
(channel, range, cadence, pin, autoRollback) lives in the ServiceDefinition ([13](../docs/13-service-catalog.md)).
|
||||
|
||||
## Alternatives considered
|
||||
- **Argo CD Image Updater** — bolt-on to Argo; we chose Flux (ADR-0004) whose image automation is
|
||||
first-class.
|
||||
- **`latest`/`stable` mutable tag + scheduled rollout restart only** — simplest but loses version
|
||||
visibility and semver guardrails; offered only as the per-service digest-tracking escape hatch.
|
||||
- **Human/AI-authored bumps** — explicitly prohibited by R-UPD-02.
|
||||
|
||||
## Consequences
|
||||
- Hands-off updates satisfied via declared policy, not manual/AI edits (R-UPD-02).
|
||||
- Safety rails required and are acceptance gates: **stable-only**, **health-gated rollout**,
|
||||
**auto-rollback** on failed health, **pin** escape hatch, and **alerting** on update failure
|
||||
([10](../docs/10-observability-and-alerting.md), [11](../docs/11-automated-updates.md)).
|
||||
- Registry rate-limits/credentials for cadence scanning must be handled.
|
||||
- Machine write-back commits will appear in history — an intended, auditable artifact, not a
|
||||
violation of "no hand-authored change."
|
||||
@@ -0,0 +1,48 @@
|
||||
# ADR-0010 — Conformance testing: automated, reproducible, behavior-based
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04, R-TEST-05, R-TEST-06
|
||||
- Supersedes: —
|
||||
|
||||
## Context
|
||||
Acceptance must be objective and repeatable, not a human eyeballing a demo. Every requirement
|
||||
(`R-*`) must be verified by an automated behavior test that any machine can run and reproduce, that
|
||||
gates merges and phase exits, and that traces back to the requirement it proves. Safety-critical
|
||||
behaviors (default-deny, VPN kill-switch, data survival, DR, auto-rollback) demand fault-injection,
|
||||
not happy-path checks. Additionally, agents must not weaken the target: specs are immutable to
|
||||
agents; tests encode spec intent.
|
||||
|
||||
## Decision
|
||||
Adopt an **automated, reproducible, behavior-based conformance suite** as the sole definition of
|
||||
acceptance, structured in layers (L0 static/policy, L1 rendering, L2 behavioral, L3 fault-injection,
|
||||
L4 aggregate suite), specified in [`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md).
|
||||
The suite is **requirement-tagged**, **single-command**, **hermetic**, **deterministic**, and emits
|
||||
**JUnit + a requirement-coverage report**. It gates:
|
||||
- **merges** — the subset for the touched requirements must pass;
|
||||
- **phase exits** — the phase's full suite must pass.
|
||||
|
||||
Tooling: the *approach* is fixed; the concrete mix is **policy-as-code** for L0 (e.g. Kyverno /
|
||||
Conftest-OPA), a **declarative Kubernetes test runner** for L1/L2 (e.g. Chainsaw or kuttl), and a
|
||||
**small programmatic harness** (e.g. Ginkgo/Gomega or a scripted runner) for L3 fault-injection that
|
||||
needs orchestration. The final tool selection is the one remaining open item and may be pinned by a
|
||||
short follow-up (still human-approved), but no choice may relax the properties above.
|
||||
|
||||
## Alternatives considered
|
||||
- **Manual demonstrations / checklists** — rejected: not reproducible, not gating, not traceable
|
||||
(violates R-TEST-01..04).
|
||||
- **Unit tests of manifests only (L0/L1)** — necessary but insufficient: a policy object existing
|
||||
does not prove enforcement; behavior (L2) and fault-injection (L3) are required (R-TEST-05).
|
||||
- **A single heavyweight framework for everything** — rejected: L3 fault-injection needs
|
||||
orchestration a declarative runner can't express, while L0/L1/L2 are cleaner declaratively. A
|
||||
layered mix fits better.
|
||||
- **Letting agents adjust tests/requirements to pass** — explicitly prohibited (R-TEST-06); a
|
||||
failing spec is an escalation, not a test edit.
|
||||
|
||||
## Consequences
|
||||
- "Done" becomes objective and machine-checkable; progress = green, requirement-tagged suite.
|
||||
- Every requirement needs a test; uncovered requirements are visible and blocking (R-TEST-04).
|
||||
- Establishing L3 fault-injection (kill-switch, node-kill, bad-update, DR) is real work and is
|
||||
first-class in the roadmap (WS-TEST), not an afterthought.
|
||||
- CI must have cluster access (or spin an ephemeral dev/test cluster) to run L2/L3.
|
||||
- The suite is versioned with the implementation and evolves only when the human-owned spec it
|
||||
encodes changes.
|
||||
@@ -0,0 +1,67 @@
|
||||
# ADR-0011 — Conformance test tooling: Kyverno + Chainsaw
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04, R-TEST-05
|
||||
- Refines: [ADR-0010](adr-0010-conformance-testing.md) (which fixed the *approach* and left the tool mix as the one open item)
|
||||
- Supersedes: —
|
||||
|
||||
## Context
|
||||
[ADR-0010](adr-0010-conformance-testing.md) fixed the conformance approach (automated, reproducible,
|
||||
behavior-based, layered, requirement-tagged, gating) and left the concrete tool selection as the sole
|
||||
open item. A choice is now made by the operator/architect.
|
||||
|
||||
## Decision
|
||||
Use **two tools from the Kyverno ecosystem**, covering all test layers:
|
||||
|
||||
- **Kyverno** — the **L0 static/policy gate** (policy-as-code) **and in-cluster admission enforcer**.
|
||||
The Kyverno CLI (`kyverno apply` / `kyverno test`, incl. `kyverno-json` for arbitrary YAML/JSON
|
||||
like ServiceDefinitions) validates manifests and rendered resources in CI against platform
|
||||
invariants: every namespace carries a default-deny baseline, no plaintext secrets, required
|
||||
labels/egress-class set, ServiceDefinition schema conformance, exposure/egress sanity. **The same
|
||||
policies also run in-cluster as an admission controller**, rejecting non-conforming resources at
|
||||
reconcile time (defense in depth) — authored once, shared between CLI gate and admission.
|
||||
- **Chainsaw** — the **L1 rendering, L2 behavioral, and L3 fault-injection** runner. Chainsaw's
|
||||
declarative test steps (apply/assert/error, `script`/`command`, `try`/`catch`/`finally`), bounded
|
||||
polling with explicit timeouts, per-test namespace isolation, and automatic cleanup directly
|
||||
satisfy the reproducibility contract (hermetic, deterministic, self-cleaning). Fault-injection
|
||||
(force the VPN tunnel down, cordon/kill a node, inject a known-bad image, trigger a restore) is
|
||||
expressed via Chainsaw `script` steps, with the fail-closed / no-data-loss outcome asserted by
|
||||
probe fixtures and `finally` steps restoring state.
|
||||
|
||||
Both emit structured, requirement-tagged results feeding the coverage report and the CI merge / phase
|
||||
exit gates ([ADR-0010](adr-0010-conformance-testing.md), [`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md)).
|
||||
Tests declare their `R-*` IDs (e.g. via labels/annotations/naming) so coverage is generated
|
||||
automatically.
|
||||
|
||||
## Alternatives considered
|
||||
- **kuttl** (for L1/L2) — Chainsaw is its spiritual successor with a richer assertion model, native
|
||||
scripting (needed for L3), and try/catch/finally cleanup; chosen over kuttl.
|
||||
- **Conftest/OPA (Rego)** for L0 — viable, but Kyverno keeps policy-as-code in the same
|
||||
YAML/ecosystem as Chainsaw and can double as an in-cluster enforcer; chosen over introducing Rego.
|
||||
- **Ginkgo/Gomega (Go)** for L3 — more power but more code and a second language/toolchain; Chainsaw
|
||||
`script` steps cover the platform's fault-injection needs declaratively. Revisit via a superseding
|
||||
ADR only if a scenario genuinely exceeds Chainsaw's orchestration.
|
||||
- **Testkube** as an orchestrator — unnecessary layer for this scale; CI + Chainsaw + Kyverno CLI
|
||||
suffice.
|
||||
|
||||
## Consequences
|
||||
- A single, coherent ecosystem (Kyverno project) spans policy enforcement and behavior/fault tests —
|
||||
fewer tools, shared idioms.
|
||||
- L3 fault-injection is expressed in Chainsaw `script` steps. **L2/L3 run as an in-cluster Chainsaw
|
||||
Job** on the dev/test cluster, **triggered by CI** — so CI holds no cluster credentials; the Job's
|
||||
own scoped ServiceAccount does. L0/L1 still run in CI without a cluster. (Resolves ADR-0010's open
|
||||
harness-location item.)
|
||||
- The test Job's ServiceAccount is a **privileged, scoped test identity** (create/delete test
|
||||
namespaces + fixtures, patch NetworkPolicies, scale VPN sidecar/gateway, cordon/kill a node, inject
|
||||
a bad image, trigger a Velero restore). It must be least-privilege for that set, named, documented,
|
||||
never reused by workloads, and treated as its own audit item (R-SEC-02).
|
||||
- Kyverno runs **both** as the CI policy gate **and** as an in-cluster admission enforcer (decided):
|
||||
non-conforming resources are rejected at reconcile time as well as in CI. This must not *replace*
|
||||
behavior tests — a policy admitting a manifest is not proof the runtime behaves (R-TEST-01), so
|
||||
L2/L3 Chainsaw tests remain mandatory. Admission enforcement adds an operational failure mode
|
||||
(a bad policy can block legitimate reconciliation); policies ship with their own tests and roll out
|
||||
via GitOps.
|
||||
- Reproducibility properties (hermetic, deterministic, self-cleaning, requirement-tagged, JUnit +
|
||||
coverage) remain the acceptance bar; the tools serve them, never relax them.
|
||||
- Pin tool versions (Kyverno, Chainsaw) for deterministic runs; upgrades flow through the normal
|
||||
update path with the suite re-green as the gate.
|
||||
Reference in new issue
Block a user