docs: bring the specification in line with the accepted implementation
The operator ruled the implementation repo's main (50c8719) the latest and accepted state. Sync the spec to it. Accepted changes are reflected: one bare-metal environment, TrueNAS bulk storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway with Cilium fail-closed, inbound WireGuard, per-app auth sidecars, Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025; ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended. Where the implementation has simply not met a requirement, the requirement stands and the shortfall is recorded in docs/16-known-gaps.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
1 parent
1ddc14b918
commit
250fb54154
43 files changed
+1672
-575
No files matched your search
+42
-38
@@ -8,10 +8,9 @@ it under a trusted TLS URL, routes its traffic on the correct WAN, protects its
|
||||
automatically, and reports on its health. Onboarding a new service is a pull request against a
|
||||
catalog, not a weekend of manual work.
|
||||
|
||||
The platform is delivered in two stages:
|
||||
|
||||
- **dev/test** on local KVMs — the active build target, used to prove viability.
|
||||
- **prod** on 3 mini-PCs — the eventual home, designed for now but not built on yet.
|
||||
The platform runs as one live environment: **prod**, on 3 bare-metal mini-PCs (ADR-0017). The dev/test
|
||||
cluster that proved its viability was decommissioned in 2026-09; every change reaches prod through
|
||||
the PR gate and Flux.
|
||||
|
||||
## Personas
|
||||
|
||||
@@ -26,23 +25,23 @@ The platform is delivered in two stages:
|
||||
- A k3s-based platform with distributed storage (Longhorn) across 3 nodes.
|
||||
- Container workloads (PaaS) and VM workloads (IaaS) through one Kubernetes surface.
|
||||
- GitOps delivery, automated updates, trusted TLS, DNS, ingress.
|
||||
- Network isolation (default-deny east-west) and multi-WAN egress including VPN routing.
|
||||
- Network isolation (default-deny east-west) and declared egress classes, including VPN routing.
|
||||
- Observability with email + dashboard alerting.
|
||||
- A command-and-control web UI at the root of `home.keithjackson.dev`.
|
||||
- The initial service catalog (13 services, see [`13-service-catalog.md`](13-service-catalog.md)).
|
||||
- The service catalog (see [`13-service-catalog.md`](13-service-catalog.md)).
|
||||
|
||||
## Out of scope (for now)
|
||||
|
||||
- Building on production mini-PC hardware (design-only until the roadmap flips it to active).
|
||||
- Multi-tenant / multi-operator RBAC beyond a single operator + platform automation.
|
||||
- Public multi-user SaaS concerns (billing, per-tenant isolation beyond network policy).
|
||||
- Non-`keithjackson.dev` domains.
|
||||
|
||||
## The domain & DNS model
|
||||
|
||||
- Public apex `*.keithjackson.dev` is available in Cloudflare DNS. `public` services under
|
||||
`*.home.keithjackson.dev` are published through a **Cloudflare Tunnel** (proxied records, no inbound
|
||||
80/443 at the edge — ADR-0016); the network's public IP stays in use for WireGuard and Plex only.
|
||||
- Public apex `*.keithjackson.dev` is available in Cloudflare DNS. `public` services (catalog
|
||||
`exposure: public` or a registered platform route) under `*.home.keithjackson.dev` are published
|
||||
through a **Cloudflare Tunnel** (proxied records, no inbound 80/443 at the edge — ADR-0016); the
|
||||
network's public IP stays in use for WireGuard and Plex only.
|
||||
- All platform services live under `*.home.keithjackson.dev`.
|
||||
- Each service is reachable at `<service>.home.keithjackson.dev` (e.g. `plex.home.keithjackson.dev`).
|
||||
- The **root** `home.keithjackson.dev` serves the **Command & Control (C2) center** UI.
|
||||
@@ -54,7 +53,7 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
|
||||
|
||||
### Platform & API surface — `R-PLAT-*`
|
||||
- **R-PLAT-01** — The platform MUST run on k3s across 3 nodes with production-grade HA control plane
|
||||
in prod; dev/test MAY run fewer nodes but MUST exercise the same multi-node topology.
|
||||
in prod.
|
||||
- **R-PLAT-02** — Container workloads MUST be onboarded and managed declaratively via the Kubernetes
|
||||
API, `kubectl`-class CLI, and YAML.
|
||||
- **R-PLAT-03** — Virtual-machine workloads MUST be created, managed, and destroyed through the
|
||||
@@ -82,9 +81,9 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
|
||||
routing. Physical multi-WAN (multiple ISP uplinks) is **out of scope** unless a future ADR reopens
|
||||
it.
|
||||
- **R-NET-05** — A service MUST be able to declare its **egress class**: `internet` (standard local
|
||||
network/internet) or `vpn` (routed through a custom VPN connection). VPN routing MUST be enforced
|
||||
**outside the service** (a single shared in-cluster gateway), so a service cannot leak by
|
||||
misconfiguration (ADR-0013, OQ-3).
|
||||
network/internet) or `vpn` (routed through a custom VPN connection). VPN routing MUST be enforced by
|
||||
a single shared node-level gateway outside the service (ADR-0015, ADR-0020), so a service cannot
|
||||
leak by misconfiguration (OQ-3).
|
||||
- **R-NET-06** — VPN-class egress MUST enforce a **kill-switch**: if the VPN tunnel is down, the
|
||||
service MUST NOT leak traffic to the standard internet.
|
||||
- **R-NET-07** — Egress class MUST be a declarative property of the service, changeable via GitOps
|
||||
@@ -101,8 +100,10 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
|
||||
|
||||
### Automated updates — `R-UPD-*`
|
||||
- **R-UPD-01** — Services MUST update to the latest **stable** versions on a schedule.
|
||||
- **R-UPD-02** — Updates MUST require **no human, no AI, and no hand-authored GitOps change** to
|
||||
occur. Automation controllers acting on declarative policy are the permitted mechanism.
|
||||
- **R-UPD-02** — Updates in the allowed classes MUST occur with **no human, no AI, and no
|
||||
hand-authored GitOps change**; automation acting on declarative policy opens machine-authored
|
||||
changes through the merge gate. Major versions and operator-designated blast-radius components MAY
|
||||
wait for human review of the machine-authored change.
|
||||
- **R-UPD-03** — Update policy MUST be per-service configurable (cadence, stability channel,
|
||||
pin/hold, auto-rollback on failed health).
|
||||
|
||||
@@ -122,9 +123,10 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
|
||||
- **R-DNS-04** — `public` HTTP(S) services MUST be reachable from the internet **without inbound
|
||||
80/443 port-forwarding at the edge**, and LAN clients MUST reach them correctly **whichever resolver
|
||||
answers** (public or internal) — no dependency on NAT hairpin (ADR-0016).
|
||||
- **R-DNS-05** — Only services declared `exposure: public` MAY be reachable through the public
|
||||
ingress path. A request for any other hostname at that path MUST be refused (HTTP 404), and
|
||||
`internal` services MUST have no public DNS record (ADR-0016).
|
||||
- **R-DNS-05** — Only services declared public (catalog `exposure: public` or a registered platform
|
||||
route) MAY be reachable through the public ingress path. A request for a LAN/VPN-only hostname at
|
||||
that path MUST be refused (HTTP 404). `internal` services MUST have no public record, except
|
||||
non-HTTP L4 names, which MAY hold DNS-only records at LAN addresses (ADR-0016).
|
||||
|
||||
### Observability & alerting — `R-OBS-*`
|
||||
- **R-OBS-01** — Services MUST be monitored (metrics, logs, health).
|
||||
@@ -154,7 +156,8 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
|
||||
- **R-IDP-02** — Services MUST authenticate against the IdP — natively via OIDC where supported, or
|
||||
behind an identity-aware (forward-auth) proxy otherwise. No management UI may be exposed
|
||||
unauthenticated on the network. (Documented exceptions: Plex/Seerr use Plex auth; Minecraft uses
|
||||
Mojang/Microsoft.)
|
||||
Mojang/Microsoft; UniFi uses its native login on its own L4 VIP; the read-only runbooks at
|
||||
`/runbooks` carry no login; Crafty web uses forward-auth plus its own local login, no OIDC.)
|
||||
- **R-IDP-03** — Identity configuration (realms, clients, groups, roles) MUST be declarative and
|
||||
GitOps-managed, not console-only.
|
||||
- **R-IDP-04** — IdP secrets (admin credential, OIDC client secrets) MUST be SOPS-encrypted; MFA MUST
|
||||
@@ -166,12 +169,12 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
|
||||
console, alerting) require an authenticated user in an authorized group, not merely "logged in."
|
||||
|
||||
### Environments — `R-ENV-*`
|
||||
- **R-ENV-01** — dev/test MUST be provisioned as local KVMs via IaC: created, baselined, cluster-
|
||||
joined, and managed as code.
|
||||
- **R-ENV-02** — The same GitOps configuration MUST promote from dev/test to prod with only
|
||||
environment-scoped overrides.
|
||||
- **R-ENV-03** — prod is a 3× mini-PC bare-metal k3s cluster with local storage; it MUST NOT be an
|
||||
active development surface until the roadmap declares it ready.
|
||||
- **R-ENV-01** — Nodes MUST be baselined, cluster-joined and managed as code; machine creation on
|
||||
bare metal is an imaged, checklist-driven step.
|
||||
- **R-ENV-02** — GitOps configuration MUST be authored once as a base plus environment-scoped
|
||||
overlay; any future environment MUST need overrides only.
|
||||
- **R-ENV-03** — Production runs on 3× mini-PC bare metal. Changes MUST reach it only through the
|
||||
PR gate and Flux.
|
||||
|
||||
### Conformance testing & governance — `R-TEST-*`
|
||||
- **R-TEST-01** — Every requirement MUST be verified by **automated, behavior-based tests**
|
||||
@@ -196,8 +199,8 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
|
||||
|
||||
## Acceptance philosophy
|
||||
|
||||
Each requirement class maps to an **automated, reproducible, behavior-based conformance test** in
|
||||
dev/test (`R-TEST-*`). "Done" means *the conformance test for the touched requirements passes* — a
|
||||
Each requirement class maps to an **automated, reproducible, behavior-based conformance test**
|
||||
(`R-TEST-*`). "Done" means *the conformance test for the touched requirements passes* — a
|
||||
green, reproducible result traced to requirement IDs — never "manifest merged" and never a manual
|
||||
demonstration. The testing approach is specified in
|
||||
[`docs/14-testing-and-conformance.md`](14-testing-and-conformance.md) and
|
||||
@@ -210,19 +213,18 @@ implementation.
|
||||
|
||||
## Open questions
|
||||
|
||||
### Resolved (2026-08-15)
|
||||
### Resolved
|
||||
- **OQ-1 (WAN / edge)** — **RESOLVED**: one commercial WAN, no multi-ISP plans; dumb SOHO edge (no
|
||||
VLAN/VPN/services). All segmentation + VPN in-cluster; edge does NAT + port-forward only. Physical
|
||||
multi-WAN out of scope. See [`03`](03-networking-isolation-and-egress.md), [`09`](09-dns-and-ingress.md).
|
||||
**Amended 2026-11-08 (ADR-0016):** public web ingress moved to an outbound-only **Cloudflare Tunnel**;
|
||||
the edge no longer forwards 80/443. Its remaining forwards are WireGuard udp/51820 and Plex tcp/32400.
|
||||
- **OQ-2 (bulk media)** — **RESOLVED (deferred backing)**: Plex bulk media lives **outside the
|
||||
cluster**; the specific backing is **TBD later** and does not block the core platform. Plex's
|
||||
metadata DB still lives on replicated Longhorn. The **off-cluster DR backup target** (for app
|
||||
config, Nextcloud, Gitea, etc.) remains to be chosen at Phase 4. See [`04`](04-storage-and-state.md).
|
||||
- **OQ-2 (bulk media)** — **RESOLVED (ADR-0018)**: Plex bulk media lives **outside the cluster** on
|
||||
a standalone TrueNAS NFS server on the production LAN, and sits outside platform backup. Plex's
|
||||
metadata DB still lives on replicated Longhorn. See [`04`](04-storage-and-state.md).
|
||||
- **OQ-3 (VPN)** — **RESOLVED**: **PIA over OpenVPN-UDP** (non-TCP; ADR-0013). Egress is a **node-level
|
||||
host tunnel steered by Cilium Egress Gateway** with a host nftables kill-switch (ADR-0015) — `vpn`-class
|
||||
pods routed through it, rules enforced outside the service. See [`03`](03-networking-isolation-and-egress.md).
|
||||
host tunnel steered by Cilium Egress Gateway**, failing closed under Cilium's own enforcement (ADR-0015,
|
||||
ADR-0020) — `vpn`-class pods routed through it, rules enforced outside the service. See [`03`](03-networking-isolation-and-egress.md).
|
||||
|
||||
- **OQ-4 (TLS/DNS challenge access)** — **RESOLVED**: a **scoped Cloudflare API token**
|
||||
(`Zone·DNS·Edit`, `keithjackson.dev`), SOPS-encrypted, drives cert-manager DNS-01 and ExternalDNS.
|
||||
@@ -230,7 +232,9 @@ implementation.
|
||||
handles issuance + record automation. Operator provides the token once. See
|
||||
[`08`](08-tls-and-certificates.md).
|
||||
|
||||
- **OQ-BKP (off-cluster DR backup target)** — **RESOLVED (ADR-0018)**: Cloudflare R2 stores Velero
|
||||
backups (kopia file-system volume data) and CNPG Barman archives of the irreplaceable tier; Longhorn
|
||||
has no backup target of its own. See [`04`](04-storage-and-state.md).
|
||||
|
||||
### Still open
|
||||
- **OQ-BKP (off-cluster DR backup target)**: which S3-compatible endpoint or NFS host stores
|
||||
Velero/Longhorn backups of the irreplaceable tier. Needed by **Phase 4**, not blocking Phase 2.
|
||||
See [`04`](04-storage-and-state.md).
|
||||
None tracked here. Unmet requirements are recorded in [known gaps](16-known-gaps.md).
|
||||
Reference in new issue
Block a user