docs: bring the specification in line with the accepted implementation

The operator ruled the implementation repo's main (50c8719) the latest
and accepted state. Sync the spec to it.

Accepted changes are reflected: one bare-metal environment, TrueNAS bulk
storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway
with Cilium fail-closed, inbound WireGuard, per-app auth sidecars,
Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025;
ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended.

Where the implementation has simply not met a requirement, the
requirement stands and the shortfall is recorded in docs/16-known-gaps.md.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5.5 committed 2026-11-08 22:46:00 -06:00
1 parent 1ddc14b918
commit 250fb54154
43 files changed
+1672 -575

No files matched your search

+42 -38
View File
@@ -8,10 +8,9 @@ it under a trusted TLS URL, routes its traffic on the correct WAN, protects its
automatically, and reports on its health. Onboarding a new service is a pull request against a
catalog, not a weekend of manual work.
The platform is delivered in two stages:
- **dev/test** on local KVMs — the active build target, used to prove viability.
- **prod** on 3 mini-PCs — the eventual home, designed for now but not built on yet.
The platform runs as one live environment: **prod**, on 3 bare-metal mini-PCs (ADR-0017). The dev/test
cluster that proved its viability was decommissioned in 2026-09; every change reaches prod through
the PR gate and Flux.
## Personas
@@ -26,23 +25,23 @@ The platform is delivered in two stages:
- A k3s-based platform with distributed storage (Longhorn) across 3 nodes.
- Container workloads (PaaS) and VM workloads (IaaS) through one Kubernetes surface.
- GitOps delivery, automated updates, trusted TLS, DNS, ingress.
- Network isolation (default-deny east-west) and multi-WAN egress including VPN routing.
- Network isolation (default-deny east-west) and declared egress classes, including VPN routing.
- Observability with email + dashboard alerting.
- A command-and-control web UI at the root of `home.keithjackson.dev`.
- The initial service catalog (13 services, see [`13-service-catalog.md`](13-service-catalog.md)).
- The service catalog (see [`13-service-catalog.md`](13-service-catalog.md)).
## Out of scope (for now)
- Building on production mini-PC hardware (design-only until the roadmap flips it to active).
- Multi-tenant / multi-operator RBAC beyond a single operator + platform automation.
- Public multi-user SaaS concerns (billing, per-tenant isolation beyond network policy).
- Non-`keithjackson.dev` domains.
## The domain & DNS model
- Public apex `*.keithjackson.dev` is available in Cloudflare DNS. `public` services under
`*.home.keithjackson.dev` are published through a **Cloudflare Tunnel** (proxied records, no inbound
80/443 at the edge — ADR-0016); the network's public IP stays in use for WireGuard and Plex only.
- Public apex `*.keithjackson.dev` is available in Cloudflare DNS. `public` services (catalog
`exposure: public` or a registered platform route) under `*.home.keithjackson.dev` are published
through a **Cloudflare Tunnel** (proxied records, no inbound 80/443 at the edge — ADR-0016); the
network's public IP stays in use for WireGuard and Plex only.
- All platform services live under `*.home.keithjackson.dev`.
- Each service is reachable at `<service>.home.keithjackson.dev` (e.g. `plex.home.keithjackson.dev`).
- The **root** `home.keithjackson.dev` serves the **Command & Control (C2) center** UI.
@@ -54,7 +53,7 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
### Platform & API surface — `R-PLAT-*`
- **R-PLAT-01** — The platform MUST run on k3s across 3 nodes with production-grade HA control plane
in prod; dev/test MAY run fewer nodes but MUST exercise the same multi-node topology.
in prod.
- **R-PLAT-02** — Container workloads MUST be onboarded and managed declaratively via the Kubernetes
API, `kubectl`-class CLI, and YAML.
- **R-PLAT-03** — Virtual-machine workloads MUST be created, managed, and destroyed through the
@@ -82,9 +81,9 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
routing. Physical multi-WAN (multiple ISP uplinks) is **out of scope** unless a future ADR reopens
it.
- **R-NET-05** — A service MUST be able to declare its **egress class**: `internet` (standard local
network/internet) or `vpn` (routed through a custom VPN connection). VPN routing MUST be enforced
**outside the service** (a single shared in-cluster gateway), so a service cannot leak by
misconfiguration (ADR-0013, OQ-3).
network/internet) or `vpn` (routed through a custom VPN connection). VPN routing MUST be enforced by
a single shared node-level gateway outside the service (ADR-0015, ADR-0020), so a service cannot
leak by misconfiguration (OQ-3).
- **R-NET-06** — VPN-class egress MUST enforce a **kill-switch**: if the VPN tunnel is down, the
service MUST NOT leak traffic to the standard internet.
- **R-NET-07** — Egress class MUST be a declarative property of the service, changeable via GitOps
@@ -101,8 +100,10 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
### Automated updates — `R-UPD-*`
- **R-UPD-01** — Services MUST update to the latest **stable** versions on a schedule.
- **R-UPD-02** — Updates MUST require **no human, no AI, and no hand-authored GitOps change** to
occur. Automation controllers acting on declarative policy are the permitted mechanism.
- **R-UPD-02** — Updates in the allowed classes MUST occur with **no human, no AI, and no
hand-authored GitOps change**; automation acting on declarative policy opens machine-authored
changes through the merge gate. Major versions and operator-designated blast-radius components MAY
wait for human review of the machine-authored change.
- **R-UPD-03** — Update policy MUST be per-service configurable (cadence, stability channel,
pin/hold, auto-rollback on failed health).
@@ -122,9 +123,10 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
- **R-DNS-04** — `public` HTTP(S) services MUST be reachable from the internet **without inbound
80/443 port-forwarding at the edge**, and LAN clients MUST reach them correctly **whichever resolver
answers** (public or internal) — no dependency on NAT hairpin (ADR-0016).
- **R-DNS-05** — Only services declared `exposure: public` MAY be reachable through the public
ingress path. A request for any other hostname at that path MUST be refused (HTTP 404), and
`internal` services MUST have no public DNS record (ADR-0016).
- **R-DNS-05** — Only services declared public (catalog `exposure: public` or a registered platform
route) MAY be reachable through the public ingress path. A request for a LAN/VPN-only hostname at
that path MUST be refused (HTTP 404). `internal` services MUST have no public record, except
non-HTTP L4 names, which MAY hold DNS-only records at LAN addresses (ADR-0016).
### Observability & alerting — `R-OBS-*`
- **R-OBS-01** — Services MUST be monitored (metrics, logs, health).
@@ -154,7 +156,8 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
- **R-IDP-02** — Services MUST authenticate against the IdP — natively via OIDC where supported, or
behind an identity-aware (forward-auth) proxy otherwise. No management UI may be exposed
unauthenticated on the network. (Documented exceptions: Plex/Seerr use Plex auth; Minecraft uses
Mojang/Microsoft.)
Mojang/Microsoft; UniFi uses its native login on its own L4 VIP; the read-only runbooks at
`/runbooks` carry no login; Crafty web uses forward-auth plus its own local login, no OIDC.)
- **R-IDP-03** — Identity configuration (realms, clients, groups, roles) MUST be declarative and
GitOps-managed, not console-only.
- **R-IDP-04** — IdP secrets (admin credential, OIDC client secrets) MUST be SOPS-encrypted; MFA MUST
@@ -166,12 +169,12 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
console, alerting) require an authenticated user in an authorized group, not merely "logged in."
### Environments — `R-ENV-*`
- **R-ENV-01** — dev/test MUST be provisioned as local KVMs via IaC: created, baselined, cluster-
joined, and managed as code.
- **R-ENV-02** — The same GitOps configuration MUST promote from dev/test to prod with only
environment-scoped overrides.
- **R-ENV-03** — prod is a 3× mini-PC bare-metal k3s cluster with local storage; it MUST NOT be an
active development surface until the roadmap declares it ready.
- **R-ENV-01** — Nodes MUST be baselined, cluster-joined and managed as code; machine creation on
bare metal is an imaged, checklist-driven step.
- **R-ENV-02** — GitOps configuration MUST be authored once as a base plus environment-scoped
overlay; any future environment MUST need overrides only.
- **R-ENV-03** — Production runs on 3× mini-PC bare metal. Changes MUST reach it only through the
PR gate and Flux.
### Conformance testing & governance — `R-TEST-*`
- **R-TEST-01** — Every requirement MUST be verified by **automated, behavior-based tests**
@@ -196,8 +199,8 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
## Acceptance philosophy
Each requirement class maps to an **automated, reproducible, behavior-based conformance test** in
dev/test (`R-TEST-*`). "Done" means *the conformance test for the touched requirements passes* — a
Each requirement class maps to an **automated, reproducible, behavior-based conformance test**
(`R-TEST-*`). "Done" means *the conformance test for the touched requirements passes* — a
green, reproducible result traced to requirement IDs — never "manifest merged" and never a manual
demonstration. The testing approach is specified in
[`docs/14-testing-and-conformance.md`](14-testing-and-conformance.md) and
@@ -210,19 +213,18 @@ implementation.
## Open questions
### Resolved (2026-08-15)
### Resolved
- **OQ-1 (WAN / edge)** — **RESOLVED**: one commercial WAN, no multi-ISP plans; dumb SOHO edge (no
VLAN/VPN/services). All segmentation + VPN in-cluster; edge does NAT + port-forward only. Physical
multi-WAN out of scope. See [`03`](03-networking-isolation-and-egress.md), [`09`](09-dns-and-ingress.md).
**Amended 2026-11-08 (ADR-0016):** public web ingress moved to an outbound-only **Cloudflare Tunnel**;
the edge no longer forwards 80/443. Its remaining forwards are WireGuard udp/51820 and Plex tcp/32400.
- **OQ-2 (bulk media)** — **RESOLVED (deferred backing)**: Plex bulk media lives **outside the
cluster**; the specific backing is **TBD later** and does not block the core platform. Plex's
metadata DB still lives on replicated Longhorn. The **off-cluster DR backup target** (for app
config, Nextcloud, Gitea, etc.) remains to be chosen at Phase 4. See [`04`](04-storage-and-state.md).
- **OQ-2 (bulk media)** — **RESOLVED (ADR-0018)**: Plex bulk media lives **outside the cluster** on
a standalone TrueNAS NFS server on the production LAN, and sits outside platform backup. Plex's
metadata DB still lives on replicated Longhorn. See [`04`](04-storage-and-state.md).
- **OQ-3 (VPN)** — **RESOLVED**: **PIA over OpenVPN-UDP** (non-TCP; ADR-0013). Egress is a **node-level
host tunnel steered by Cilium Egress Gateway** with a host nftables kill-switch (ADR-0015) — `vpn`-class
pods routed through it, rules enforced outside the service. See [`03`](03-networking-isolation-and-egress.md).
host tunnel steered by Cilium Egress Gateway**, failing closed under Cilium's own enforcement (ADR-0015,
ADR-0020) — `vpn`-class pods routed through it, rules enforced outside the service. See [`03`](03-networking-isolation-and-egress.md).
- **OQ-4 (TLS/DNS challenge access)** — **RESOLVED**: a **scoped Cloudflare API token**
(`Zone·DNS·Edit`, `keithjackson.dev`), SOPS-encrypted, drives cert-manager DNS-01 and ExternalDNS.
@@ -230,7 +232,9 @@ implementation.
handles issuance + record automation. Operator provides the token once. See
[`08`](08-tls-and-certificates.md).
- **OQ-BKP (off-cluster DR backup target)** — **RESOLVED (ADR-0018)**: Cloudflare R2 stores Velero
backups (kopia file-system volume data) and CNPG Barman archives of the irreplaceable tier; Longhorn
has no backup target of its own. See [`04`](04-storage-and-state.md).
### Still open
- **OQ-BKP (off-cluster DR backup target)**: which S3-compatible endpoint or NFS host stores
Velero/Longhorn backups of the irreplaceable tier. Needed by **Phase 4**, not blocking Phase 2.
See [`04`](04-storage-and-state.md).
None tracked here. Unmet requirements are recorded in [known gaps](16-known-gaps.md).