docs: bring the specification in line with the accepted implementation
The operator ruled the implementation repo's main (50c8719) the latest and accepted state. Sync the spec to it. Accepted changes are reflected: one bare-metal environment, TrueNAS bulk storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway with Cilium fail-closed, inbound WireGuard, per-app auth sidecars, Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025; ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended. Where the implementation has simply not met a requirement, the requirement stands and the shortfall is recorded in docs/16-known-gaps.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5.5 committed
1 parent
1ddc14b918
commit
250fb54154
43 files changed
+1672
-575
No files matched your search
+42
-38
@@ -8,10 +8,9 @@ it under a trusted TLS URL, routes its traffic on the correct WAN, protects its
|
||||
automatically, and reports on its health. Onboarding a new service is a pull request against a
|
||||
catalog, not a weekend of manual work.
|
||||
|
||||
The platform is delivered in two stages:
|
||||
|
||||
- **dev/test** on local KVMs — the active build target, used to prove viability.
|
||||
- **prod** on 3 mini-PCs — the eventual home, designed for now but not built on yet.
|
||||
The platform runs as one live environment: **prod**, on 3 bare-metal mini-PCs (ADR-0017). The dev/test
|
||||
cluster that proved its viability was decommissioned in 2026-09; every change reaches prod through
|
||||
the PR gate and Flux.
|
||||
|
||||
## Personas
|
||||
|
||||
@@ -26,23 +25,23 @@ The platform is delivered in two stages:
|
||||
- A k3s-based platform with distributed storage (Longhorn) across 3 nodes.
|
||||
- Container workloads (PaaS) and VM workloads (IaaS) through one Kubernetes surface.
|
||||
- GitOps delivery, automated updates, trusted TLS, DNS, ingress.
|
||||
- Network isolation (default-deny east-west) and multi-WAN egress including VPN routing.
|
||||
- Network isolation (default-deny east-west) and declared egress classes, including VPN routing.
|
||||
- Observability with email + dashboard alerting.
|
||||
- A command-and-control web UI at the root of `home.keithjackson.dev`.
|
||||
- The initial service catalog (13 services, see [`13-service-catalog.md`](13-service-catalog.md)).
|
||||
- The service catalog (see [`13-service-catalog.md`](13-service-catalog.md)).
|
||||
|
||||
## Out of scope (for now)
|
||||
|
||||
- Building on production mini-PC hardware (design-only until the roadmap flips it to active).
|
||||
- Multi-tenant / multi-operator RBAC beyond a single operator + platform automation.
|
||||
- Public multi-user SaaS concerns (billing, per-tenant isolation beyond network policy).
|
||||
- Non-`keithjackson.dev` domains.
|
||||
|
||||
## The domain & DNS model
|
||||
|
||||
- Public apex `*.keithjackson.dev` is available in Cloudflare DNS. `public` services under
|
||||
`*.home.keithjackson.dev` are published through a **Cloudflare Tunnel** (proxied records, no inbound
|
||||
80/443 at the edge — ADR-0016); the network's public IP stays in use for WireGuard and Plex only.
|
||||
- Public apex `*.keithjackson.dev` is available in Cloudflare DNS. `public` services (catalog
|
||||
`exposure: public` or a registered platform route) under `*.home.keithjackson.dev` are published
|
||||
through a **Cloudflare Tunnel** (proxied records, no inbound 80/443 at the edge — ADR-0016); the
|
||||
network's public IP stays in use for WireGuard and Plex only.
|
||||
- All platform services live under `*.home.keithjackson.dev`.
|
||||
- Each service is reachable at `<service>.home.keithjackson.dev` (e.g. `plex.home.keithjackson.dev`).
|
||||
- The **root** `home.keithjackson.dev` serves the **Command & Control (C2) center** UI.
|
||||
@@ -54,7 +53,7 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
|
||||
|
||||
### Platform & API surface — `R-PLAT-*`
|
||||
- **R-PLAT-01** — The platform MUST run on k3s across 3 nodes with production-grade HA control plane
|
||||
in prod; dev/test MAY run fewer nodes but MUST exercise the same multi-node topology.
|
||||
in prod.
|
||||
- **R-PLAT-02** — Container workloads MUST be onboarded and managed declaratively via the Kubernetes
|
||||
API, `kubectl`-class CLI, and YAML.
|
||||
- **R-PLAT-03** — Virtual-machine workloads MUST be created, managed, and destroyed through the
|
||||
@@ -82,9 +81,9 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
|
||||
routing. Physical multi-WAN (multiple ISP uplinks) is **out of scope** unless a future ADR reopens
|
||||
it.
|
||||
- **R-NET-05** — A service MUST be able to declare its **egress class**: `internet` (standard local
|
||||
network/internet) or `vpn` (routed through a custom VPN connection). VPN routing MUST be enforced
|
||||
**outside the service** (a single shared in-cluster gateway), so a service cannot leak by
|
||||
misconfiguration (ADR-0013, OQ-3).
|
||||
network/internet) or `vpn` (routed through a custom VPN connection). VPN routing MUST be enforced by
|
||||
a single shared node-level gateway outside the service (ADR-0015, ADR-0020), so a service cannot
|
||||
leak by misconfiguration (OQ-3).
|
||||
- **R-NET-06** — VPN-class egress MUST enforce a **kill-switch**: if the VPN tunnel is down, the
|
||||
service MUST NOT leak traffic to the standard internet.
|
||||
- **R-NET-07** — Egress class MUST be a declarative property of the service, changeable via GitOps
|
||||
@@ -101,8 +100,10 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
|
||||
|
||||
### Automated updates — `R-UPD-*`
|
||||
- **R-UPD-01** — Services MUST update to the latest **stable** versions on a schedule.
|
||||
- **R-UPD-02** — Updates MUST require **no human, no AI, and no hand-authored GitOps change** to
|
||||
occur. Automation controllers acting on declarative policy are the permitted mechanism.
|
||||
- **R-UPD-02** — Updates in the allowed classes MUST occur with **no human, no AI, and no
|
||||
hand-authored GitOps change**; automation acting on declarative policy opens machine-authored
|
||||
changes through the merge gate. Major versions and operator-designated blast-radius components MAY
|
||||
wait for human review of the machine-authored change.
|
||||
- **R-UPD-03** — Update policy MUST be per-service configurable (cadence, stability channel,
|
||||
pin/hold, auto-rollback on failed health).
|
||||
|
||||
@@ -122,9 +123,10 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
|
||||
- **R-DNS-04** — `public` HTTP(S) services MUST be reachable from the internet **without inbound
|
||||
80/443 port-forwarding at the edge**, and LAN clients MUST reach them correctly **whichever resolver
|
||||
answers** (public or internal) — no dependency on NAT hairpin (ADR-0016).
|
||||
- **R-DNS-05** — Only services declared `exposure: public` MAY be reachable through the public
|
||||
ingress path. A request for any other hostname at that path MUST be refused (HTTP 404), and
|
||||
`internal` services MUST have no public DNS record (ADR-0016).
|
||||
- **R-DNS-05** — Only services declared public (catalog `exposure: public` or a registered platform
|
||||
route) MAY be reachable through the public ingress path. A request for a LAN/VPN-only hostname at
|
||||
that path MUST be refused (HTTP 404). `internal` services MUST have no public record, except
|
||||
non-HTTP L4 names, which MAY hold DNS-only records at LAN addresses (ADR-0016).
|
||||
|
||||
### Observability & alerting — `R-OBS-*`
|
||||
- **R-OBS-01** — Services MUST be monitored (metrics, logs, health).
|
||||
@@ -154,7 +156,8 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
|
||||
- **R-IDP-02** — Services MUST authenticate against the IdP — natively via OIDC where supported, or
|
||||
behind an identity-aware (forward-auth) proxy otherwise. No management UI may be exposed
|
||||
unauthenticated on the network. (Documented exceptions: Plex/Seerr use Plex auth; Minecraft uses
|
||||
Mojang/Microsoft.)
|
||||
Mojang/Microsoft; UniFi uses its native login on its own L4 VIP; the read-only runbooks at
|
||||
`/runbooks` carry no login; Crafty web uses forward-auth plus its own local login, no OIDC.)
|
||||
- **R-IDP-03** — Identity configuration (realms, clients, groups, roles) MUST be declarative and
|
||||
GitOps-managed, not console-only.
|
||||
- **R-IDP-04** — IdP secrets (admin credential, OIDC client secrets) MUST be SOPS-encrypted; MFA MUST
|
||||
@@ -166,12 +169,12 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
|
||||
console, alerting) require an authenticated user in an authorized group, not merely "logged in."
|
||||
|
||||
### Environments — `R-ENV-*`
|
||||
- **R-ENV-01** — dev/test MUST be provisioned as local KVMs via IaC: created, baselined, cluster-
|
||||
joined, and managed as code.
|
||||
- **R-ENV-02** — The same GitOps configuration MUST promote from dev/test to prod with only
|
||||
environment-scoped overrides.
|
||||
- **R-ENV-03** — prod is a 3× mini-PC bare-metal k3s cluster with local storage; it MUST NOT be an
|
||||
active development surface until the roadmap declares it ready.
|
||||
- **R-ENV-01** — Nodes MUST be baselined, cluster-joined and managed as code; machine creation on
|
||||
bare metal is an imaged, checklist-driven step.
|
||||
- **R-ENV-02** — GitOps configuration MUST be authored once as a base plus environment-scoped
|
||||
overlay; any future environment MUST need overrides only.
|
||||
- **R-ENV-03** — Production runs on 3× mini-PC bare metal. Changes MUST reach it only through the
|
||||
PR gate and Flux.
|
||||
|
||||
### Conformance testing & governance — `R-TEST-*`
|
||||
- **R-TEST-01** — Every requirement MUST be verified by **automated, behavior-based tests**
|
||||
@@ -196,8 +199,8 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
|
||||
|
||||
## Acceptance philosophy
|
||||
|
||||
Each requirement class maps to an **automated, reproducible, behavior-based conformance test** in
|
||||
dev/test (`R-TEST-*`). "Done" means *the conformance test for the touched requirements passes* — a
|
||||
Each requirement class maps to an **automated, reproducible, behavior-based conformance test**
|
||||
(`R-TEST-*`). "Done" means *the conformance test for the touched requirements passes* — a
|
||||
green, reproducible result traced to requirement IDs — never "manifest merged" and never a manual
|
||||
demonstration. The testing approach is specified in
|
||||
[`docs/14-testing-and-conformance.md`](14-testing-and-conformance.md) and
|
||||
@@ -210,19 +213,18 @@ implementation.
|
||||
|
||||
## Open questions
|
||||
|
||||
### Resolved (2026-08-15)
|
||||
### Resolved
|
||||
- **OQ-1 (WAN / edge)** — **RESOLVED**: one commercial WAN, no multi-ISP plans; dumb SOHO edge (no
|
||||
VLAN/VPN/services). All segmentation + VPN in-cluster; edge does NAT + port-forward only. Physical
|
||||
multi-WAN out of scope. See [`03`](03-networking-isolation-and-egress.md), [`09`](09-dns-and-ingress.md).
|
||||
**Amended 2026-11-08 (ADR-0016):** public web ingress moved to an outbound-only **Cloudflare Tunnel**;
|
||||
the edge no longer forwards 80/443. Its remaining forwards are WireGuard udp/51820 and Plex tcp/32400.
|
||||
- **OQ-2 (bulk media)** — **RESOLVED (deferred backing)**: Plex bulk media lives **outside the
|
||||
cluster**; the specific backing is **TBD later** and does not block the core platform. Plex's
|
||||
metadata DB still lives on replicated Longhorn. The **off-cluster DR backup target** (for app
|
||||
config, Nextcloud, Gitea, etc.) remains to be chosen at Phase 4. See [`04`](04-storage-and-state.md).
|
||||
- **OQ-2 (bulk media)** — **RESOLVED (ADR-0018)**: Plex bulk media lives **outside the cluster** on
|
||||
a standalone TrueNAS NFS server on the production LAN, and sits outside platform backup. Plex's
|
||||
metadata DB still lives on replicated Longhorn. See [`04`](04-storage-and-state.md).
|
||||
- **OQ-3 (VPN)** — **RESOLVED**: **PIA over OpenVPN-UDP** (non-TCP; ADR-0013). Egress is a **node-level
|
||||
host tunnel steered by Cilium Egress Gateway** with a host nftables kill-switch (ADR-0015) — `vpn`-class
|
||||
pods routed through it, rules enforced outside the service. See [`03`](03-networking-isolation-and-egress.md).
|
||||
host tunnel steered by Cilium Egress Gateway**, failing closed under Cilium's own enforcement (ADR-0015,
|
||||
ADR-0020) — `vpn`-class pods routed through it, rules enforced outside the service. See [`03`](03-networking-isolation-and-egress.md).
|
||||
|
||||
- **OQ-4 (TLS/DNS challenge access)** — **RESOLVED**: a **scoped Cloudflare API token**
|
||||
(`Zone·DNS·Edit`, `keithjackson.dev`), SOPS-encrypted, drives cert-manager DNS-01 and ExternalDNS.
|
||||
@@ -230,7 +232,9 @@ implementation.
|
||||
handles issuance + record automation. Operator provides the token once. See
|
||||
[`08`](08-tls-and-certificates.md).
|
||||
|
||||
- **OQ-BKP (off-cluster DR backup target)** — **RESOLVED (ADR-0018)**: Cloudflare R2 stores Velero
|
||||
backups (kopia file-system volume data) and CNPG Barman archives of the irreplaceable tier; Longhorn
|
||||
has no backup target of its own. See [`04`](04-storage-and-state.md).
|
||||
|
||||
### Still open
|
||||
- **OQ-BKP (off-cluster DR backup target)**: which S3-compatible endpoint or NFS host stores
|
||||
Velero/Longhorn backups of the irreplaceable tier. Needed by **Phase 4**, not blocking Phase 2.
|
||||
See [`04`](04-storage-and-state.md).
|
||||
None tracked here. Unmet requirements are recorded in [known gaps](16-known-gaps.md).
|
||||
@@ -9,22 +9,32 @@ choice here without a superseding ADR.**
|
||||
| Layer | Choice | Serves | ADR | Spec |
|
||||
|---|---|---|---|---|
|
||||
| Kubernetes distro | **k3s** (HA, embedded etcd) | R-PLAT-01/02 | [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md) | [02](02-environments-and-provisioning.md) |
|
||||
| CNI / network policy / egress | **Cilium** (+ Hubble, Gateway API, Egress Gateway) | R-NET-01..07 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [03](03-networking-isolation-and-egress.md) |
|
||||
| Distributed storage | **Longhorn** | R-STATE-01..03 | [ADR-0003](../decisions/adr-0003-storage-longhorn.md) | [04](04-storage-and-state.md) |
|
||||
| CNI / network policy / egress | **Cilium** (+ Hubble, Gateway API, Egress Gateway, L2 LB-IPAM) | R-NET-01..07 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [03](03-networking-isolation-and-egress.md) |
|
||||
| Distributed storage | **Longhorn** | R-STATE-01..02 | [ADR-0003](../decisions/adr-0003-storage-longhorn.md) | [04](04-storage-and-state.md) |
|
||||
| Bulk media | **NFS** from a standalone **TrueNAS** on the production LAN | R-STATE-04 | [ADR-0018](../decisions/adr-0018-storage-backup-truenas-r2.md) | [04](04-storage-and-state.md) |
|
||||
| PostgreSQL | **CloudNativePG (CNPG)** | R-STATE-01/03, R-IDP-05 | [ADR-0019](../decisions/adr-0019-postgresql-cloudnativepg.md) | [04](04-storage-and-state.md) |
|
||||
| GitOps engine | **Flux CD** | R-GITOPS-01..03 | [ADR-0004](../decisions/adr-0004-gitops-flux.md) | [05](05-gitops-and-delivery.md) |
|
||||
| VM workloads (IaaS) | **KubeVirt** (+ CDI) | R-PLAT-03 | [ADR-0005](../decisions/adr-0005-vm-workloads-kubevirt.md) | [06](06-vm-workloads-kubevirt.md) |
|
||||
| TLS | **cert-manager** + Let's Encrypt (DNS-01) | R-TLS-01..03 | [ADR-0006](../decisions/adr-0006-tls-cert-manager-letsencrypt.md) | [08](08-tls-and-certificates.md) |
|
||||
| Secrets | **SOPS + age** (Flux-native) | R-SEC-01 | [ADR-0007](../decisions/adr-0007-secrets-sops-age.md) | [07](07-security-and-secrets.md) |
|
||||
| VPN egress | **Node-level PIA/OpenVPN tunnel** steered by **Cilium Egress Gateway** (host nftables kill-switch) | R-NET-05/06/07 | [ADR-0015](../decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md) (provider [0013](../decisions/adr-0013-vpn-provider-pia-gluetun-wireguard.md)) | [03](03-networking-isolation-and-egress.md) |
|
||||
| Automated updates | **Flux image-update-automation** (images) + **Renovate** (charts) | R-UPD-01..03 | [ADR-0009](../decisions/adr-0009-automated-image-updates.md) | [11](11-automated-updates.md) |
|
||||
| DNS records | **ExternalDNS** (public via Cloudflare) + **PiHole** (internal) | R-DNS-01..03 | — | [09](09-dns-and-ingress.md) |
|
||||
| VPN egress | **Node-level PIA/OpenVPN tunnel** on a single gateway node, steered by **Cilium Egress Gateway** (Cilium-enforced fail-closed) | R-NET-05/06/07 | [ADR-0015](../decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md), [ADR-0020](../decisions/adr-0020-vpn-egress-single-gateway-cilium-fail-closed.md) (provider [0013](../decisions/adr-0013-vpn-provider-pia-gluetun-wireguard.md)) | [03](03-networking-isolation-and-egress.md) |
|
||||
| Automated updates | **Renovate** (images and charts), every change PR-gated | R-UPD-01..03 | [ADR-0023](../decisions/adr-0023-automated-updates-renovate.md) | [11](11-automated-updates.md) |
|
||||
| DNS records | **ExternalDNS** (public via Cloudflare) + **PiHole** (explicit LAN answers) | R-DNS-01..03 | — | [09](09-dns-and-ingress.md) |
|
||||
| Ingress (north-south) | **Gateway API** implemented by Cilium | R-DNS-01, R-TLS-01 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [09](09-dns-and-ingress.md) |
|
||||
| Public web ingress | **Cloudflare Tunnel** (outbound-only cloudflared) | R-DNS-04/05, R-SEC-04 | [ADR-0016](../decisions/adr-0016-public-ingress-cloudflare-tunnel.md) | [09](09-dns-and-ingress.md) |
|
||||
| Remote access | **WireGuard** on a hostNetwork DaemonSet | R-SEC-02/04 | [ADR-0021](../decisions/adr-0021-inbound-remote-access-wireguard.md) | [03](03-networking-isolation-and-egress.md) |
|
||||
| Observability | **kube-prometheus-stack** + **Loki** + **Grafana Alloy** + Alertmanager→SMTP | R-OBS-01..04 | — | [10](10-observability-and-alerting.md) |
|
||||
| Backup/DR | **Velero** + Longhorn backups → off-cluster S3/NFS | R-STATE-03 | [ADR-0003](../decisions/adr-0003-storage-longhorn.md) | [04](04-storage-and-state.md) |
|
||||
| Node/infra provisioning | **OpenTofu** (libvirt) + **cloud-init** + **Ansible** (k3s bootstrap) | R-ENV-01..03 | [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md) | [02](02-environments-and-provisioning.md) |
|
||||
| Identity / SSO | **Keycloak** (OIDC + SAML) + **oauth2-proxy** forward-auth | R-IDP-01..06 | [ADR-0014](../decisions/adr-0014-idp-keycloak.md) | [15](15-identity-and-access.md) |
|
||||
| Backup/DR | **Velero** (kopia) + **CNPG Barman** → Cloudflare R2; no Longhorn backup target | R-STATE-03 | [ADR-0018](../decisions/adr-0018-storage-backup-truenas-r2.md) | [04](04-storage-and-state.md) |
|
||||
| Node/infra provisioning | **Imaged Debian** on bare metal + **Ansible** (baseline, k3s bootstrap) | R-ENV-01..03 | [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md), [ADR-0017](../decisions/adr-0017-single-environment-bare-metal-production.md) | [02](02-environments-and-provisioning.md) |
|
||||
| Control-plane endpoint | **kube-vip** virtual IP | R-PLAT-01 | [ADR-0017](../decisions/adr-0017-single-environment-bare-metal-production.md) | [02](02-environments-and-provisioning.md) |
|
||||
| Node patching & reboots | security-only **unattended-upgrades** + **kured** | R-PLAT-01 | — | [02](02-environments-and-provisioning.md) |
|
||||
| Identity / SSO | **Keycloak** (OIDC + SAML) + per-app **oauth2-proxy** forward-auth sidecars | R-IDP-01..06 | [ADR-0014](../decisions/adr-0014-idp-keycloak.md), [ADR-0022](../decisions/adr-0022-forward-auth-per-app-oauth2-proxy-sidecars.md) | [15](15-identity-and-access.md) |
|
||||
| Command center UI | **Homepage** + **Headlamp** + **Grafana** panels (behind Keycloak SSO) | R-C2-01/02 | — | [12](12-command-center-ui.md) |
|
||||
| Conformance testing | **Kyverno** (L0 policy) + **Chainsaw** (L1/L2/L3 behavior + fault-injection) | R-TEST-01..06 | [ADR-0010](../decisions/adr-0010-conformance-testing.md), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md) | [14](14-testing-and-conformance.md) |
|
||||
| Delivery & merge gate | **Gitea** + **Gitea Actions**; the CI gate is the approver | R-GITOPS-01, R-TEST-03 | [ADR-0024](../decisions/adr-0024-delivery-authority-ci-gate.md) | [05](05-gitops-and-delivery.md) |
|
||||
| Image builds | CI-run rootless **BuildKit**, published to Gitea's registry | — | — | [05](05-gitops-and-delivery.md) |
|
||||
| AI Program Manager | **pm-agent** (issues to pull requests through the gate) | — | [ADR-0025](../decisions/adr-0025-in-cluster-ai-program-manager.md) | [05](05-gitops-and-delivery.md) |
|
||||
| Hardware transcode | **Intel GPU device plugin** (Plex Quick Sync) | — | — | [13](13-service-catalog.md) |
|
||||
|
||||
## Layer cake
|
||||
|
||||
@@ -41,18 +51,17 @@ choice here without a superseding ADR.**
|
||||
│ Flux (GitOps) · cert-manager (TLS) · ExternalDNS · Gateway API │
|
||||
│ Keycloak (SSO/IdP) + oauth2-proxy (forward-auth) │
|
||||
│ kube-prometheus-stack + Loki + Alloy (observability) │
|
||||
│ Velero (backup) · Renovate (chart updates) · SOPS (secrets) │
|
||||
│ Velero + CNPG Barman (backup) · Renovate (updates) · SOPS (secrets) │
|
||||
│ cloudflared (public ingress) · CNPG (PostgreSQL) · kured (node reboots) │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Storage: Longhorn (replicated block) + bulk media store (see ADR-0003) │
|
||||
│ Storage: Longhorn (replicated block) + TrueNAS NFS bulk tier (ADR-0018) │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Network: Cilium (default-deny NetworkPolicy · Egress Gateway · Hubble) │
|
||||
│ VPN egress (node tunnel + Cilium egress GW) · Multi-WAN routing │
|
||||
│ VPN egress (node tunnel + Cilium egress GW) · WireGuard inbound │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Kubernetes: k3s (3 nodes, HA control plane, embedded etcd) │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Nodes │
|
||||
│ dev/test: libvirt KVMs (OpenTofu + cloud-init + Ansible) │
|
||||
│ prod: 3× mini-PC bare metal, local disks │
|
||||
│ Nodes: 3× mini-PC bare metal, local disks (imaged Debian + Ansible) │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Edge: SOHO router — single WAN, NAT; forwards only WireGuard + Plex │
|
||||
│ Public web ingress: Cloudflare Tunnel (outbound-only cloudflared, ADR-0016)│
|
||||
@@ -64,8 +73,8 @@ choice here without a superseding ADR.**
|
||||
This platform has **two** kinds of VM, and confusing them is the most common design error:
|
||||
|
||||
1. **Infrastructure VMs = the cluster nodes themselves.**
|
||||
In dev/test these are libvirt KVMs on the workstation, provisioned by OpenTofu + cloud-init and
|
||||
configured/joined by Ansible. In prod these are the bare-metal mini-PCs. See
|
||||
These are the bare-metal mini-PCs, imaged and then baselined and joined by Ansible; "VM" is
|
||||
historical shorthand for this layer, and it remains the place a node would be virtualized. See
|
||||
[`02-environments-and-provisioning.md`](02-environments-and-provisioning.md).
|
||||
|
||||
2. **Application VMs = IaaS workloads that run *inside* the cluster via KubeVirt.**
|
||||
@@ -81,16 +90,20 @@ This platform has **two** kinds of VM, and confusing them is the most common des
|
||||
storage, **egress class** (`internet`|`vpn`), exposure (`internal`|`public`), dependencies
|
||||
(explicit allow rules), update policy, and observability expectations.
|
||||
2. **Flux** reconciles the change into the cluster.
|
||||
3. **Longhorn** provisions replicated persistent volumes; state is durable.
|
||||
3. **Longhorn** provisions replicated persistent volumes, so state is durable; large media attaches
|
||||
from the NFS **bulk tier**.
|
||||
4. **Cilium** applies default-deny plus the explicitly declared allow rules; assigns the correct
|
||||
**egress path** (local internet or VPN, with kill-switch).
|
||||
5. **ExternalDNS** publishes `<service>.home.keithjackson.dev`; **Gateway API** routes north-south
|
||||
traffic; **cert-manager** issues a trusted TLS cert via ACME DNS-01.
|
||||
5. A `public` service gets a proxied **ExternalDNS** record at `<service>.home.keithjackson.dev` and
|
||||
arrives through the **Cloudflare Tunnel**. An `internal` service has no public record and is
|
||||
answered on the LAN by **PiHole**. **Gateway API** routes north-south traffic for both;
|
||||
**cert-manager** issues a trusted TLS cert via ACME DNS-01.
|
||||
6. **kube-prometheus-stack + Loki** scrape metrics/logs; **Alertmanager** emails on issues; the
|
||||
**C2 UI** lists and links the service.
|
||||
7. **Flux image automation / Renovate** keep the service on the latest stable version per policy —
|
||||
no human action.
|
||||
8. **Velero + Longhorn backups** protect the data; destroy/rebuild restores state.
|
||||
7. **Renovate** keeps the service on the latest stable version per policy, through machine-authored
|
||||
changes that pass the merge gate.
|
||||
8. **Velero and CNPG Barman**, writing off-cluster, protect the data; destroy/rebuild restores
|
||||
state.
|
||||
|
||||
Every numbered step maps to a domain spec and a workstream in [`roadmap/workstreams.md`](../roadmap/workstreams.md).
|
||||
|
||||
@@ -103,7 +116,7 @@ Every numbered step maps to a domain spec and a workstream in [`roadmap/workstre
|
||||
- **Durable by default.** No workload is "pet" hardware; state lives on replicated storage with
|
||||
tested restore.
|
||||
- **Hands-off operations.** Updates, cert renewal, DNS, and drift correction are automated.
|
||||
- **Promotable.** dev/test and prod are the same config with scoped overrides.
|
||||
- **Promotable.** Configuration is one base plus an environment-scoped overlay.
|
||||
- **Provable.** Every requirement is verified by an automated, reproducible, behavior-based
|
||||
conformance test ([14](14-testing-and-conformance.md), [ADR-0010](../decisions/adr-0010-conformance-testing.md));
|
||||
"done" is a green, requirement-tagged suite, not a manual demo.
|
||||
|
||||
@@ -1,39 +1,41 @@
|
||||
# 02 — Environments & Provisioning
|
||||
|
||||
Serves: **R-ENV-01/02/03**, **R-PLAT-01**. Related: [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md).
|
||||
Serves: **R-ENV-01/02/03**, **R-PLAT-01**. Related: [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md),
|
||||
[ADR-0017](../decisions/adr-0017-single-environment-bare-metal-production.md).
|
||||
|
||||
## Two environments, one configuration
|
||||
## One live environment, one configuration
|
||||
|
||||
| Aspect | dev/test | prod |
|
||||
|---|---|---|
|
||||
| Substrate | libvirt/KVM VMs on the operator workstation | 3× mini-PC bare metal |
|
||||
| Node count | 3 VMs (mirror prod topology) | 3 physical nodes |
|
||||
| Control plane | HA embedded etcd (k3s server on all 3) | HA embedded etcd (k3s server on all 3) |
|
||||
| Storage | virtual disks presented to Longhorn | local SSD/NVMe presented to Longhorn |
|
||||
| Status | **active build target** | **design-only until roadmap flips it** |
|
||||
| Aspect | prod |
|
||||
|---|---|
|
||||
| Substrate | 3× mini-PC bare metal |
|
||||
| Node count | 3 physical nodes |
|
||||
| Control plane | HA embedded etcd (k3s server on all 3) |
|
||||
| Storage | local SSD/NVMe presented to Longhorn; bulk media on an off-cluster NAS |
|
||||
| Status | **live** |
|
||||
|
||||
**R-ENV-02 is the golden constraint:** the GitOps configuration is authored once. Environments
|
||||
differ only through scoped overrides (Flux `Kustomization` overlays / cluster-scoped variables):
|
||||
node counts, disk sizes, WAN endpoints, VPN credentials, DNS zone specifics, resource limits.
|
||||
Never fork the base config per environment.
|
||||
A dev/test cluster proved the platform and was decommissioned in 2026-09. Prod is the only
|
||||
environment.
|
||||
|
||||
**R-ENV-02 is the golden constraint:** the GitOps configuration is authored once, as a base plus an
|
||||
environment-scoped overlay (Flux `Kustomization` overlays / cluster-scoped variables): node counts,
|
||||
disk sizes, WAN endpoints, VPN credentials, DNS zone specifics, resource limits. Never fork the base
|
||||
config per environment. Any future environment needs overrides only.
|
||||
|
||||
## Infrastructure provisioning (Layer 1 — the cluster nodes)
|
||||
|
||||
> These are the machines that *run* k3s. Distinct from KubeVirt application VMs (Layer 2).
|
||||
|
||||
### Toolchain
|
||||
- **OpenTofu** (Terraform-compatible) with the **libvirt** provider defines dev/test node VMs as
|
||||
code: CPU, RAM, disks (a dedicated data disk for Longhorn), NICs, and a **cloud-init** seed.
|
||||
- **cloud-init** performs first-boot baseline: hostname, users, SSH keys, package baseline, kernel
|
||||
modules/sysctls required by k3s/Longhorn/KubeVirt, and marks the node ready for k3s.
|
||||
- **Bare-metal imaging** is an operator step driven by a checklist. There is no hypervisor API on
|
||||
bare metal to automate machine creation.
|
||||
- **Ansible** performs the node baseline: users, package floor, kernel, kernel modules/sysctls
|
||||
required by k3s/Longhorn/KubeVirt, and the data partition.
|
||||
- **Ansible** performs cluster bootstrap: installs k3s in HA mode, joins nodes, lays down cluster-
|
||||
wide prerequisites (Cilium is installed as the CNI, not flannel), and hands off to Flux.
|
||||
- For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation step differs
|
||||
(bare-metal image/cloud-init via USB/PXE instead of libvirt).
|
||||
|
||||
### Node lifecycle stages (R-ENV-01 — "created, baselined, cluster-joined, managed as code")
|
||||
1. **Create** — OpenTofu instantiates the VM (dev) / operator images the mini-PC (prod).
|
||||
2. **Baseline** — cloud-init applies OS baseline and prerequisites.
|
||||
### Node lifecycle stages (R-ENV-01 — "baselined, cluster-joined, managed as code")
|
||||
1. **Create** — the operator images the mini-PC from the provisioning checklist.
|
||||
2. **Baseline** — Ansible applies the OS baseline and prerequisites.
|
||||
3. **Cluster-join** — Ansible installs/joins k3s with Cilium as CNI; disables k3s bundled
|
||||
flannel/traefik/servicelb/kube-proxy as required by the Cilium ADR.
|
||||
4. **Manage** — from here the cluster is GitOps-managed by Flux; node config changes are made via
|
||||
@@ -41,10 +43,13 @@ Never fork the base config per environment.
|
||||
|
||||
### k3s topology decisions
|
||||
- HA control plane with **embedded etcd** across all 3 nodes (no external datastore).
|
||||
- The Kubernetes API has one HA endpoint: a **kube-vip** virtual IP. Steady-state consumers use the
|
||||
virtual IP. A node joining during a cold bootstrap uses the founding server's own address, because
|
||||
the virtual IP does not exist until a cluster does.
|
||||
- Bundled components that conflict with our stack are **disabled at install**: flannel (→ Cilium),
|
||||
kube-proxy (→ Cilium kube-proxy replacement), Traefik (→ Gateway API via Cilium), servicelb where
|
||||
a different LB strategy is chosen. Exact flags live in the Ansible bootstrap; the *intent* is
|
||||
fixed here and in ADR-0001/0002.
|
||||
kube-proxy (→ Cilium kube-proxy replacement), Traefik (→ Gateway API via Cilium), servicelb
|
||||
(→ Cilium L2 announcements and LB-IPAM), local-path storage (→ `longhorn` as the sole default).
|
||||
Exact flags live in the Ansible bootstrap; the *intent* is fixed here and in ADR-0001/0002.
|
||||
- Nodes are both control-plane and workload (3-node converged cluster); with only 3 nodes we do not
|
||||
isolate control-plane nodes.
|
||||
|
||||
@@ -57,46 +62,63 @@ resolves the apparent overlap between "Ansible installs the CNI" (here) and "Flu
|
||||
in **parity** for the settings that define cluster networking — notably `kubeProxyReplacement` and
|
||||
`k8sServiceHost`/`k8sServicePort` — or the two layers will fight over the CNI config.
|
||||
|
||||
### Bare-metal specifics (prod, design-only for now)
|
||||
- Each mini-PC contributes local NVMe/SSD to Longhorn; document the disk layout (OS vs. Longhorn
|
||||
data device) before prod build.
|
||||
- BIOS/firmware, virtualization extensions (for KubeVirt), and wake/boot behavior are captured as a
|
||||
provisioning checklist under the prod-readiness workstream.
|
||||
- Edge networking (multi-WAN, VPN uplink) is largely an **edge-router** concern — see
|
||||
### Bare-metal specifics
|
||||
- The nodes are 3 HP EliteDesk Mini desktops with VT-x. Their drives differ (SATA SSD and NVMe); the
|
||||
disk *layout* is identical across all 3. The node inventory, BIOS/firmware settings, and the
|
||||
provisioning checklist are `infra/PROD-PROVISIONING.md` in the
|
||||
[implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl).
|
||||
- Each node has **one physical disk**: an OS area and one data partition. The data partition holds
|
||||
both the Longhorn store and the k3s data directory (embedded etcd and the container image store).
|
||||
- Nodes run a newer kernel series than the distribution default, because the firmware publishes a
|
||||
defective EFI memory table that older kernels mishandle. This is a standing, deliberate deviation.
|
||||
- **Guardrail and accepted risk (etcd and Longhorn share a device).** With no second device in the
|
||||
chassis, etcd shares each node's disk with Longhorn replica I/O. Every node MUST be SSD or NVMe,
|
||||
and etcd takes I/O priority over other work on the device. The risk is accepted knowingly. Its
|
||||
exit condition is that etcd write latency stays within budget during a Longhorn rebuild, or each
|
||||
node gains a dedicated etcd device (see [known gaps](16-known-gaps.md)).
|
||||
- The edge is a single-WAN SOHO router — see
|
||||
[`03-networking-isolation-and-egress.md`](03-networking-isolation-and-egress.md) and OQ-1.
|
||||
|
||||
### Off-host storage
|
||||
Bulk media lives on a standalone TrueNAS NFS server on the production LAN, directly reachable by the
|
||||
nodes (ADR-0018, [04](04-storage-and-state.md)). It is storage only and runs no platform compute.
|
||||
|
||||
### Node maintenance
|
||||
OS patching is automatic and security-only, excluding the kernel. Kernel upgrades are deliberate
|
||||
operator actions, run one node at a time. A reboot daemon carries out the mechanics of a needed
|
||||
reboot and takes a single cluster-wide lock, so at most one node reboots at a time, and the next
|
||||
waits for Longhorn volumes to recover. A hung node holds the lock, stopping the cycle until a human
|
||||
intervenes: on 3 etcd members, losing a second node is the failure to avoid.
|
||||
|
||||
## Environment promotion model
|
||||
|
||||
```
|
||||
git repo (single source of truth)
|
||||
│
|
||||
base configuration ──────────────┐
|
||||
│ │
|
||||
overlay: dev/test overlay: prod
|
||||
│ │
|
||||
Flux on dev cluster Flux on prod cluster (later)
|
||||
base configuration
|
||||
│
|
||||
overlay: prod
|
||||
│
|
||||
Flux on the prod cluster
|
||||
```
|
||||
|
||||
- A change proves out in **dev/test** against the acceptance gates ([`AGENTS.md` §7](../AGENTS.md)).
|
||||
- Promotion to prod is a change to the prod overlay only (no base edits), gated by the roadmap's
|
||||
prod-readiness phase.
|
||||
- **Never develop against prod.** Prod hardware is a target, not a sandbox (R-ENV-03).
|
||||
- A change proves out against the acceptance gates ([`AGENTS.md` §7](../AGENTS.md)) before it merges.
|
||||
- A future environment is a new overlay only (no base edits).
|
||||
- **Changes reach prod only through the PR gate and Flux** (R-ENV-03). Fault-injecting tests against
|
||||
prod are deliberate, scheduled exercises.
|
||||
|
||||
## Statelessness of the nodes themselves
|
||||
|
||||
Nodes are cattle. A node can be destroyed and rebuilt from IaC + cloud-init + Ansible; cluster and
|
||||
Nodes are cattle. A node can be destroyed and rebuilt by imaging and Ansible; cluster and
|
||||
application state lives on **Longhorn** (replicated) and in **git** (desired state), so node loss is
|
||||
a re-provision, not a data-loss event (supports R-STATE-01/02).
|
||||
|
||||
## Deliverables for this spec (tracked on the task board)
|
||||
- OpenTofu module for dev/test node VMs (libvirt) with a dedicated Longhorn data disk.
|
||||
- cloud-init baseline template (users, SSH, sysctls, kernel modules for Longhorn/KubeVirt).
|
||||
- Ansible role(s): k3s HA install with Cilium, component disabling, Flux bootstrap handoff.
|
||||
- Documented prod bare-metal provisioning checklist (design-only; do not execute).
|
||||
- Proof: destroy a dev node VM and rebuild it; cluster re-converges with no data loss.
|
||||
## Deliverables for this spec
|
||||
- Ansible role(s): node baseline, k3s HA install with Cilium, component disabling, Flux bootstrap
|
||||
handoff.
|
||||
- Documented bare-metal provisioning checklist.
|
||||
- Proof: lose a node and rebuild it; the cluster re-converges with no data loss.
|
||||
|
||||
## Open questions
|
||||
- **OQ-1** (edge router / WAN count) — blocks final multi-WAN design, not node provisioning.
|
||||
- Do we want **Cluster API** later to unify dev VM and prod bare-metal lifecycle? Deferred; OpenTofu
|
||||
+ Ansible is sufficient and lower-complexity for 3 nodes now. Revisit via ADR if prod scaling
|
||||
demands it.
|
||||
- Do we want **Cluster API** later to automate bare-metal lifecycle? Deferred; imaging + Ansible is
|
||||
sufficient and lower-complexity for 3 nodes. Revisit via ADR if scaling demands it.
|
||||
@@ -1,7 +1,9 @@
|
||||
# 03 — Networking, Isolation & Egress
|
||||
|
||||
Serves: **R-NET-01..07**, **R-SEC-02**. Related: [ADR-0002 (Cilium)](../decisions/adr-0002-cni-cilium.md),
|
||||
[ADR-0008 (VPN egress)](../decisions/adr-0008-vpn-egress-model.md).
|
||||
[ADR-0015 (VPN egress steering)](../decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md),
|
||||
[ADR-0020 (single gateway, Cilium fail-closed)](../decisions/adr-0020-vpn-egress-single-gateway-cilium-fail-closed.md),
|
||||
[ADR-0021 (inbound remote access)](../decisions/adr-0021-inbound-remote-access-wireguard.md).
|
||||
|
||||
This is the most safety-critical spec. Get default-deny and the VPN kill-switch right; everything
|
||||
else is convenience.
|
||||
@@ -76,7 +78,7 @@ as multi-ISP routing but as **two egress paths selected by routing rules**:
|
||||
| Profile | Meaning | Mechanism |
|
||||
|---|---|---|
|
||||
| `internet` | Out the single commercial WAN | Normal pod egress via node → SOHO edge router → internet |
|
||||
| `vpn` | Routed through the in-cluster VPN tunnel | Steered to the shared VPN egress gateway (ADR-0013) |
|
||||
| `vpn` | Routed through the in-cluster VPN tunnel | Steered to the shared VPN egress gateway (ADR-0015, ADR-0020) |
|
||||
|
||||
- **Physical multi-WAN (multi-ISP) is out of scope.** With a single WAN there is nothing to policy-
|
||||
route across; the `internet` path is simply the default route out.
|
||||
@@ -88,7 +90,7 @@ as multi-ISP routing but as **two egress paths selected by routing rules**:
|
||||
not forward 80/443.
|
||||
- The **service-facing contract is unchanged**: a service still just declares `egress: internet|vpn`.
|
||||
|
||||
## VPN egress model: node-level tunnel + Cilium Egress Gateway (R-NET-05/06 — ADR-0015; provider ADR-0013)
|
||||
## VPN egress model: node-level tunnel + Cilium Egress Gateway (R-NET-05/06 — ADR-0015, ADR-0020; provider ADR-0013)
|
||||
|
||||
VPN-class services: **sonarr, radarr, jackett, qbittorrent**. Per OQ-3 the design is a **single,
|
||||
shared VPN connection** — not a per-service sidecar — so that **network rules are enforced outside
|
||||
@@ -105,7 +107,7 @@ the wrong class.
|
||||
|
||||
The model:
|
||||
|
||||
- **Tunnel on the node, not in a pod.** One or two labeled **egress-gateway nodes** run a **host OpenVPN
|
||||
- **Tunnel on the node, not in a pod.** Exactly one labeled **egress-gateway node** runs a **host OpenVPN
|
||||
systemd service** — **PIA over OpenVPN-UDP** (provider/protocol per ADR-0013; "non-TCP or better"),
|
||||
**Ansible-managed host infra** (like k3s itself), NOT a Flux workload. It is **split-tunnel**: it
|
||||
creates `tun0` but does **not** set the node's default route (`route-noexec` + policy routing), so the
|
||||
@@ -114,23 +116,52 @@ The model:
|
||||
- **`vpn`-class pods are steered by Cilium Egress Gateway.** A `CiliumEgressGatewayPolicy` selects pods
|
||||
labeled `egress-class=vpn` and egresses them via the gateway node's `tun0` (native eBPF SNAT to the
|
||||
tunnel IP). **No client-pod changes, no sidecar, no overlay.** They have **no independent egress path**;
|
||||
the service is just labeled `egress: vpn` and cannot leak on its own.
|
||||
- **Kill-switch (fail-closed) — host nftables.** An **nftables rule on the gateway node** DROPS steered
|
||||
(`egress-class=vpn`) traffic whenever `tun0` is down — a host backstop independent of any Cilium
|
||||
egress-gateway fail-open edge case. If the tunnel is down, `vpn` pods have **no route to the plain
|
||||
internet**; `internet`-class services are unaffected.
|
||||
the service is just labeled `egress: vpn` and cannot leak on its own. Only public destinations are
|
||||
tunnelled; private ranges (the cluster, the LAN, bulk storage) stay on the local datapath.
|
||||
- **Kill-switch (fail-closed) — Cilium's egress datapath (ADR-0020).** Steered traffic is forwarded in
|
||||
eBPF, and **Cilium drops it** whenever the gateway's tunnel interface cannot be used. If the tunnel is
|
||||
down, `vpn` pods have **no route to the plain internet**; `internet`-class services are unaffected.
|
||||
Steered pod traffic never reaches netfilter, so a **host nftables table** on the gateway is a narrow
|
||||
backstop only: it covers node-originated traffic sourced from a tunnel address.
|
||||
- **Re-steering across reconnects.** Cilium resolves the tunnel interface once and does not follow its
|
||||
re-creation, so every reconnect is signalled back to Cilium from the gateway node, without writing to
|
||||
GitOps state. A reconnect Cilium was not told about raises an alert
|
||||
([10](10-observability-and-alerting.md)).
|
||||
- **Port forwarding for qBittorrent:** PIA NAT-PMP, re-solved **host-side** (a small pf/natpmp helper on
|
||||
the gateway node that publishes the assigned forwarded port into qBittorrent) — kept hands-off. Alert
|
||||
if PF acquisition fails.
|
||||
- **Resilience:** two egress-gateway nodes + **Cilium Egress Gateway HA** (gateway failover) give the
|
||||
active/standby resilience; alert on tunnel state ([10](10-observability-and-alerting.md)).
|
||||
- **Prerequisite:** the **PIA username/password + `.ovpn`** on the gateway node(s) via **Ansible vault**
|
||||
if PF acquisition fails (see [known gaps](16-known-gaps.md)).
|
||||
- **Resilience: one gateway, no failover (ADR-0020).** Cilium OSS resolves the gateway selector to a
|
||||
single node, so a second labeled node is not a standby: it silently takes the role, and the public
|
||||
egress IP, by name order. **Label exactly one node.** Losing it is a `vpn`-class outage, fail-closed:
|
||||
no leak and no service. Alert on tunnel state ([10](10-observability-and-alerting.md)).
|
||||
- **Prerequisite:** the **PIA username/password + `.ovpn`** on the gateway node via **Ansible vault**
|
||||
([07](07-security-and-secrets.md)) — the tunnel is host infra, so its creds are host-side (not a cluster
|
||||
SOPS Secret). Validate end-to-end (tunnel + steering + kill-switch + PF) in dev/test.
|
||||
SOPS Secret). Validate end-to-end (tunnel + steering + kill-switch + PF) by conformance test
|
||||
([14](14-testing-and-conformance.md)).
|
||||
|
||||
**Kill-switch is an acceptance gate:** with the tunnel forcibly down, every `vpn` service must fail
|
||||
closed (no packets to the plain internet), while `internet`-class is unaffected. Tested, not assumed.
|
||||
|
||||
## Remote access (inbound): WireGuard (ADR-0021)
|
||||
|
||||
The operator's devices reach the home network through a **WireGuard** listener on udp/51820. This is
|
||||
the opposite direction to `vpn` egress and shares nothing with it: `vpn` egress sends pods *out*
|
||||
through the provider tunnel; remote access lets clients *in*. It is the "VPN" in "LAN/VPN-only".
|
||||
|
||||
- **Host network, every node.** A Flux-managed listener runs in the host network namespace on all
|
||||
nodes. A pod-networked server would give every client that pod's Cilium identity, and every Gateway
|
||||
hostname would refuse it. A listener on each node makes loss of the forwarded node a one-line router
|
||||
change, with no new client profiles.
|
||||
- **A deliberate exception to default-deny.** A host-network workload has no Cilium endpoint, so no
|
||||
network policy constrains it. The boundary around the port is **WireGuard's peer authentication**: a
|
||||
packet from an unknown key gets no reply. The exception applies to this platform component only;
|
||||
application namespaces may not use the host network.
|
||||
- **Authorization is reviewable.** Peers are public keys in plain text in git, so granting or revoking
|
||||
a device is a reviewed change. The server's private key is SOPS-encrypted.
|
||||
- **Reach equals a LAN machine.** An enrolled device can reach the whole LAN, including the nodes'
|
||||
control-plane ports, and egresses through the house WAN. A lost device must be revoked promptly.
|
||||
Clients stay `internet` class and are never steered to the provider tunnel.
|
||||
|
||||
## Ingress (north-south) — summary
|
||||
|
||||
North-south exposure is handled by **Gateway API implemented by Cilium**, with TLS from
|
||||
@@ -157,9 +188,8 @@ Cloudflare Tunnel, ADR-0016).
|
||||
Amended 2026-11-08 (ADR-0016): 80/443 are no longer forwarded; public web ingress is a Cloudflare
|
||||
Tunnel.
|
||||
- **OQ-3** — **PIA over OpenVPN-UDP** (ADR-0013); egress is a **node-level host tunnel** steered by **Cilium
|
||||
Egress Gateway** with a host nftables kill-switch (ADR-0015), rules enforced outside the service.
|
||||
Egress Gateway** and failing closed in Cilium's datapath (ADR-0015, ADR-0020), rules enforced outside
|
||||
the service.
|
||||
|
||||
## Open questions
|
||||
- Exact steering mechanism for `vpn` pods → gateway (Cilium Egress Gateway vs routing/SNAT) —
|
||||
WS-NET implementation detail; the kill-switch acceptance test is the gate.
|
||||
- Which **PF-enabled PIA region** to use, and the port-sync helper wiring for qBittorrent.
|
||||
@@ -1,6 +1,8 @@
|
||||
# 04 — Storage & State
|
||||
|
||||
Serves: **R-STATE-01..04**, **R-PLAT-04**. Related: [ADR-0003 (Longhorn)](../decisions/adr-0003-storage-longhorn.md).
|
||||
Serves: **R-STATE-01..04**, **R-PLAT-04**. Related: [ADR-0003 (Longhorn)](../decisions/adr-0003-storage-longhorn.md),
|
||||
[ADR-0018 (bulk tier + backup)](../decisions/adr-0018-storage-backup-truenas-r2.md),
|
||||
[ADR-0019 (PostgreSQL via CloudNativePG)](../decisions/adr-0019-postgresql-cloudnativepg.md).
|
||||
|
||||
The core promise: **any service can be destroyed and rebuilt on a whim without losing data.** This
|
||||
requires (a) durable replicated storage, (b) a clean separation of state from workload, and (c) a
|
||||
@@ -10,21 +12,41 @@ tested backup/restore path.
|
||||
|
||||
| Tier | Backing | Use | Replication |
|
||||
|---|---|---|---|
|
||||
| **App/config state** | Longhorn block volumes | PVCs for service config, databases, small app data | 3-way replica across nodes |
|
||||
| **Bulk media** | Dedicated large store (see below) | Plex library, download staging | Not 3× replicated (capacity strategy) |
|
||||
| **App/config state** | Longhorn block volumes | PVCs for service config, embedded databases, small app data | 3-way replica across nodes |
|
||||
| **Databases** | CloudNativePG clusters on Longhorn | PostgreSQL for Keycloak, Nextcloud, Semaphore | 3 instances, each on a 3-way replicated volume |
|
||||
| **Bulk media** | Off-cluster NAS, consumed over NFS | Plex library, download staging, read-only legacy Nextcloud archive | Not 3× replicated (capacity strategy) |
|
||||
| **VM disks** | Longhorn (via KubeVirt/CDI) | KubeVirt application-VM root/data disks | 3-way replica |
|
||||
| **Backups** | Off-cluster S3/NFS target | Velero + Longhorn backups for DR | Off-cluster, separate failure domain |
|
||||
| **Backups** | Cloudflare R2 (S3-compatible, off-site) | Velero + kopia volume data; CloudNativePG WAL and base backups | Off-cluster, separate failure domain |
|
||||
|
||||
## Longhorn (primary persistent storage — R-STATE-02)
|
||||
|
||||
- Provides replicated block storage from each node's local disk — exactly the model for a 3× mini-PC
|
||||
cluster with local storage and no SAN.
|
||||
- Default StorageClass: **3 replicas** so any single node loss keeps data available.
|
||||
- Two further classes serve KubeVirt: a live-migratable block class (3 replicas, the default for VM
|
||||
disks) and a single-replica class for CDI's throwaway import scratch, the one non-replicated use of
|
||||
Longhorn.
|
||||
- Volumes are the durable home for stateful services. Destroying a pod/Deployment/VM leaves the PVC
|
||||
and its data intact; rebuilding re-attaches it (R-STATE-01).
|
||||
- Snapshots for point-in-time; **backups** (to off-cluster target) for DR.
|
||||
- Snapshots are available on demand and are how VM disks clone; **backups** for DR go off-cluster
|
||||
through Velero (below), not through a Longhorn backup target.
|
||||
- Guardrails: reserve disk headroom, set replica anti-affinity across nodes, monitor volume health
|
||||
and degraded replicas (alerts in [10](10-observability-and-alerting.md)).
|
||||
and degraded replicas (alerts in [10](10-observability-and-alerting.md)), and cap Longhorn's write
|
||||
concurrency so replica I/O cannot starve etcd's WAL.
|
||||
- **Accepted risk:** each node has one data device, so etcd's WAL and every Longhorn replica share it.
|
||||
The concurrency caps reduce that contention and do not remove it. The exit condition is a dedicated
|
||||
etcd device per node (see [known gaps](16-known-gaps.md)).
|
||||
|
||||
## Database state (PostgreSQL)
|
||||
|
||||
Services that need PostgreSQL get a **CloudNativePG** cluster per database ([ADR-0019](../decisions/adr-0019-postgresql-cloudnativepg.md)):
|
||||
three instances on three different nodes, each on replicated Longhorn. Longhorn replication keeps the
|
||||
*data* through a node loss; a streaming standby keeps the *service*, because promotion takes seconds.
|
||||
Standbys replicate corruption faithfully, so neither is a backup: WAL is archived continuously and
|
||||
base backups run daily to the off-cluster target, giving point-in-time recovery.
|
||||
|
||||
Services whose datastore is embedded (Gitea's and Crafty's SQLite, for example) keep it on their
|
||||
Longhorn PVC and are covered by the ordinary volume backup.
|
||||
|
||||
## Separation of state from workload (the rebuild guarantee)
|
||||
|
||||
@@ -33,44 +55,65 @@ Every service's catalog entry declares its **persistent volumes** explicitly. Th
|
||||
> If it must survive a rebuild, it lives on a named PVC (Longhorn) or the bulk store — **never** in
|
||||
> the container's ephemeral filesystem or on a single node's hostPath.
|
||||
|
||||
A volume counts only when it is mounted at the path the application actually writes. A PVC mounted
|
||||
anywhere else binds, reports healthy and backs up successfully while the real state sits on the
|
||||
ephemeral layer, so a service's mount path is declared deliberately rather than left to a default.
|
||||
|
||||
The rebuild test (an acceptance gate): delete the workload entirely, re-apply from git, and the
|
||||
service comes back with its data. This is verified per service, not assumed.
|
||||
|
||||
## Bulk media strategy (R-STATE-04 — the Plex problem)
|
||||
|
||||
Plex is the largest service and its library can be multiple terabytes. **3× replicating multi-TB
|
||||
Plex is the largest service and its library is multiple terabytes. **3× replicating multi-TB
|
||||
media across three mini-PCs is impractical** (capacity and cost). Per **OQ-2 (resolved)**, **bulk
|
||||
media lives OUTSIDE the cluster**; the specific backing is **to be determined later** and does not
|
||||
block the core platform.
|
||||
media lives OUTSIDE the cluster** on a standalone TrueNAS appliance: storage only, ZFS-backed, and on
|
||||
the same LAN as the nodes ([ADR-0018](../decisions/adr-0018-storage-backup-truenas-r2.md)).
|
||||
|
||||
- The bulk store is consumed by Plex and the arr stack as an **external mount** (NFS/SMB share to be
|
||||
specified when chosen). The `bulk` StorageClass / mount is a thin indirection so the concrete
|
||||
backing can be filled in later without touching service definitions.
|
||||
- The bulk store is consumed by Plex, the arr stack and qbittorrent as an **external NFS mount**.
|
||||
Nextcloud mounts a separate legacy archive from it read-only.
|
||||
- A catalog entry's `class: bulk` plus a named export is a thin indirection: it attaches an existing
|
||||
dataset as a statically bound volume, and the NFS server is a per-environment value, so service
|
||||
definitions never name the storage host. Nothing is dynamically provisioned, because every bulk
|
||||
volume attaches data that already exists.
|
||||
- Downloads (qbittorrent) stage to the bulk store and hand off to the arr stack.
|
||||
- Layout, export settings and the uid model are in `docs/BULK-NFS-STORAGE.md` in the
|
||||
[implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl).
|
||||
|
||||
**Regardless of the bulk backing: Plex's config/metadata database is small and MUST live on 3-replica
|
||||
Longhorn** (so Plex rebuilds cleanly); only the large media files use the external bulk store. Bulk
|
||||
media is treated as **`reacquirable`** for backup purposes unless the operator specifies otherwise
|
||||
when the backing is chosen.
|
||||
Longhorn** (so Plex rebuilds cleanly); only the large media files use the external bulk store.
|
||||
|
||||
**The bulk tier is outside platform backup.** Velero skips NFS volumes by volume type, so the exclusion
|
||||
also covers any future export, while the same namespaces' Longhorn PVCs are still backed up. A
|
||||
service's `backup.policy` therefore speaks for its Longhorn volumes only, and the bulk data's
|
||||
durability is the NAS's own (see [known gaps](16-known-gaps.md)).
|
||||
|
||||
## Backup & disaster recovery (R-STATE-03)
|
||||
|
||||
- **Velero** backs up Kubernetes resource state (namespaces, manifests, PVC metadata) and
|
||||
orchestrates volume backups.
|
||||
- **Longhorn backups** push volume data to an **off-cluster** S3-compatible or NFS target — a
|
||||
separate failure domain from the cluster itself.
|
||||
- **Schedule**: regular automated backups (cadence per data criticality) with retention policy.
|
||||
- The **off-cluster target is Cloudflare R2**: S3-compatible, off-site and a separate failure domain
|
||||
from the cluster and from the NAS. Backup tools share one bucket under disjoint prefixes, so one
|
||||
tool's retention can never reap another's objects.
|
||||
- **Velero** backs up Kubernetes resource state (namespaces, manifests, PVC metadata) and, through
|
||||
**kopia file-system backup**, the data of every volume. Backup is opt-out, so a new volume is
|
||||
covered without anyone remembering to enrol it.
|
||||
- **Longhorn has no backup target** ([ADR-0018](../decisions/adr-0018-storage-backup-truenas-r2.md)).
|
||||
Velero and kopia are the single restore path for volume data, an accepted trade-off.
|
||||
- **CloudNativePG** archives WAL and base backups to the same target, independently of Velero.
|
||||
- **Schedule**: regular automated backups with retention policy. Databases archive continuously;
|
||||
everything else runs nightly.
|
||||
- **Exclusions are limited to reconstructable data**, whose Kubernetes resources are still captured:
|
||||
monitoring's metrics and log volumes, and the UniFi VM's disk, for which the design relies on
|
||||
UniFi's own configuration export ([06](06-vm-workloads-kubevirt.md)).
|
||||
- **Restore is a tested procedure**, not a theoretical one: a DR drill (restore a namespace + its
|
||||
volumes to a clean state) is part of the platform acceptance suite.
|
||||
- Bulk media has its own backup posture (e.g. it may be re-acquirable rather than backed up 1:1 —
|
||||
an explicit operator decision recorded per OQ-2), but **all irreplaceable state** (configs,
|
||||
- Bulk media is outside this posture (above), but **all irreplaceable state** (configs,
|
||||
databases, Nextcloud data, Gitea repos) MUST be backed up off-cluster.
|
||||
|
||||
## KubeVirt VM disks
|
||||
|
||||
Application-VM disks are Longhorn volumes provisioned via **CDI** (Containerized Data Importer). VMs
|
||||
inherit the same durability and backup story as container PVCs — a KubeVirt VM can be destroyed and
|
||||
rebuilt from its declarative definition + persistent disk. See [`06-vm-workloads-kubevirt.md`](06-vm-workloads-kubevirt.md).
|
||||
share the durability of container PVCs and are backed up by default, with per-VM exceptions recorded
|
||||
in [06](06-vm-workloads-kubevirt.md) — a KubeVirt VM can be destroyed and rebuilt from its declarative
|
||||
definition + persistent disk. See [`06-vm-workloads-kubevirt.md`](06-vm-workloads-kubevirt.md).
|
||||
|
||||
## GitOps + state = full rebuildability
|
||||
|
||||
@@ -78,21 +121,20 @@ rebuilt from its declarative definition + persistent disk. See [`06-vm-workloads
|
||||
- **Data** lives on Longhorn/bulk store + off-cluster backups.
|
||||
- Therefore a total cluster loss is recoverable: re-provision nodes ([02](02-environments-and-provisioning.md)),
|
||||
re-bootstrap Flux, let it reconcile from git, and restore volumes from backup. This end-to-end
|
||||
recovery is the ultimate expression of R-STATE-01/03 and SHOULD be drilled at least once in
|
||||
dev/test before prod is declared ready.
|
||||
recovery is the ultimate expression of R-STATE-01/03 and SHOULD be drilled at least once, in a
|
||||
planned maintenance window.
|
||||
|
||||
## Acceptance gates
|
||||
- Delete and recreate each stateful service; data survives (per-service rebuild test).
|
||||
- Kill a node (dev/test); replicated volumes stay available; cluster re-converges.
|
||||
- Kill a node in a planned maintenance window; replicated volumes stay available; cluster re-converges.
|
||||
- Restore a namespace + volumes from off-cluster backup into a clean state (DR drill).
|
||||
- Plex rebuilds with its metadata DB from replicated storage; media re-attaches from the bulk tier.
|
||||
|
||||
## Resolved
|
||||
- **OQ-2** — bulk media lives **outside the cluster**; concrete backing **TBD later** (consumed as an
|
||||
external NFS/SMB mount via the `bulk` indirection). Does not block the core platform.
|
||||
- **OQ-2** — bulk media lives **outside the cluster**, on a TrueNAS appliance consumed over NFS through
|
||||
the catalog's `class: bulk` indirection (ADR-0018).
|
||||
- **OQ-BKP** — the off-cluster DR target for the *irreplaceable* tier is **Cloudflare R2** (ADR-0018).
|
||||
|
||||
## Open questions
|
||||
- **Off-cluster DR backup target** (separate from bulk media): which S3-compatible endpoint or NFS
|
||||
host holds Velero/Longhorn backups of the *irreplaceable* tier (app config, Nextcloud, Gitea, DBs),
|
||||
and its own resilience. Needed by **Phase 4**, not Phase 2.
|
||||
- Concrete bulk-media backing + whether any of it is non-`reacquirable` (operator decides when chosen).
|
||||
- Whether any bulk data, notably the photo library, is non-`reacquirable` and needs its own backup
|
||||
(operator decides).
|
||||
+110
-50
@@ -1,6 +1,10 @@
|
||||
# 05 — GitOps & Delivery
|
||||
|
||||
Serves: **R-GITOPS-01..03**, **R-PLAT-02/04**, **R-ENV-02**. Related: [ADR-0004 (Flux)](../decisions/adr-0004-gitops-flux.md).
|
||||
Serves: **R-GITOPS-01..03**, **R-PLAT-02/04**, **R-ENV-02**, **R-TEST-03**. Related:
|
||||
[ADR-0004 (Flux)](../decisions/adr-0004-gitops-flux.md),
|
||||
[ADR-0023 (updates)](../decisions/adr-0023-automated-updates-renovate.md),
|
||||
[ADR-0024 (delivery authority)](../decisions/adr-0024-delivery-authority-ci-gate.md),
|
||||
[ADR-0025 (pm-agent)](../decisions/adr-0025-in-cluster-ai-program-manager.md).
|
||||
|
||||
## Model
|
||||
|
||||
@@ -8,35 +12,40 @@ Pull-based GitOps with **Flux CD**. The cluster continuously reconciles its stat
|
||||
the single source of truth; the cluster conforms to git, not the reverse (R-GITOPS-01). Drift is
|
||||
detected and corrected automatically (R-GITOPS-03).
|
||||
|
||||
## Repository layout (the implementation repo, created in a later phase)
|
||||
## Repository layout (the implementation repo)
|
||||
|
||||
> This spec repo (`home-network-k3s`) holds intent. The **implementation** repo(s) hold the
|
||||
> reconciled manifests. Proposed structure:
|
||||
> This spec repo (`home-network-k3s`) holds intent. The
|
||||
> [implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl)
|
||||
> holds the reconciled manifests, the gate and the tests. Its top-level areas:
|
||||
|
||||
```
|
||||
platform/ ← cluster-wide platform services (Flux-managed)
|
||||
cilium/ ← CNI, egress gateway, network baseline
|
||||
longhorn/ ← storage
|
||||
cert-manager/ ← TLS issuers + ClusterIssuer
|
||||
externaldns/ ← DNS automation
|
||||
gateway/ ← Gateway API + listeners/TLS
|
||||
monitoring/ ← kube-prometheus-stack, Loki, Alloy, Alertmanager
|
||||
velero/ ← backup/DR
|
||||
kubevirt/ ← KubeVirt + CDI
|
||||
image-automation/ ← Flux image update automation config
|
||||
apps/ ← application workloads, one dir per service
|
||||
plex/ sonarr/ radarr/ jackett/ seerr/ pihole/ nextcloud/
|
||||
minecraft/ unifi/ semaphore/ kasm/ gitea/ qbittorrent/
|
||||
catalog/ ← ServiceDefinition entries (the onboarding surface)
|
||||
catalog/ ← ServiceDefinition entries (the onboarding surface) + the renderer
|
||||
schemas/ ← ServiceDefinition JSON Schema
|
||||
apps/ ← application workloads by reconcile wave, plus VMs; rendered from
|
||||
catalog/ and committed (the render is checked against them)
|
||||
platform/ ← cluster-wide platform services (Flux-managed): CNI, storage, TLS, DNS,
|
||||
Gateway, public tunnel, observability, backup/DR, KubeVirt, identity,
|
||||
admission policy, image builder, delivery automation
|
||||
platform-config*/ ← platform configuration Flux applies after the platform tier, including
|
||||
the admission policies
|
||||
clusters/
|
||||
dev/ ← Flux entrypoint + dev overlay/vars
|
||||
prod/ ← Flux entrypoint + prod overlay/vars (design-only)
|
||||
base/ ← shared Flux entrypoints and reconcile ordering
|
||||
prod/ ← the live cluster's overlay and environment values
|
||||
bootstrap/ ← Flux, SOPS and Gitea bootstrap
|
||||
infra/ ← node provisioning as code; Gitea settings; dev toolbox; agent image
|
||||
ci/ .gitea/ ← the merge gate and the workflows that run it
|
||||
conformance/ ← requirement registry, behavior tests, harness, run ledger
|
||||
runbooks/ ← operator procedures (also served to the alert links)
|
||||
decisions/ ← dated implementation handoff log (distinct from this repo's ADRs)
|
||||
```
|
||||
|
||||
- **Kustomize overlays** (or equivalent) express environment differences (R-ENV-02). Base is shared;
|
||||
`clusters/dev` and `clusters/prod` supply scoped values only.
|
||||
- **Kustomize overlays** express environment differences (R-ENV-02). The base is shared; an
|
||||
environment overlay supplies scoped values only. `prod` is the one live environment; the dev
|
||||
environment was decommissioned in 2026-09 ([ADR-0017](../decisions/adr-0017-single-environment-bare-metal-production.md)).
|
||||
- **HelmReleases** (Flux) wrap upstream charts (Cilium, Longhorn, cert-manager, kube-prometheus-
|
||||
stack, etc.); app workloads use charts where good ones exist, otherwise plain manifests.
|
||||
- **Rendered output is committed.** `apps/` is generated from `catalog/`; the merge gate fails when
|
||||
the two disagree, so a ServiceDefinition is always the source of a rendered workload.
|
||||
|
||||
## The onboarding surface: ServiceDefinition (R-GITOPS-02)
|
||||
|
||||
@@ -48,14 +57,52 @@ platform renders the rest.
|
||||
|
||||
## Reconciliation flow
|
||||
|
||||
1. Operator opens a PR against the implementation repo (new/updated ServiceDefinition or platform
|
||||
change).
|
||||
2. CI validates (lint, policy checks, schema validation, dry-run/build).
|
||||
3. Merge to the tracked branch.
|
||||
4. Flux reconciles: applies manifests, reports status, corrects drift on an interval.
|
||||
1. A contributor opens a PR against the implementation repo (new/updated ServiceDefinition or
|
||||
platform change). Contributors are the operator, implementation agents, Renovate and pm-agent;
|
||||
all of them take the same path.
|
||||
2. The **merge gate** validates the change (below).
|
||||
3. Merge to `main`, the tracked branch.
|
||||
4. Flux reconciles from the platform's own Gitea: applies manifests, reports status, corrects drift
|
||||
on an interval.
|
||||
5. Health/rollout status surfaces in the C2 UI ([12](12-command-center-ui.md)) and alerts on failure
|
||||
([10](10-observability-and-alerting.md)).
|
||||
|
||||
Nothing reaches `main` by any other route: direct pushes are rejected for every account.
|
||||
|
||||
## The merge gate (R-TEST-03)
|
||||
|
||||
The implementation repo's `main` is protected by one required status check, the merge gate, which
|
||||
runs on every PR as a Gitea Actions job. **A red gate is unmergeable by everyone, administrators
|
||||
included.** The gate runs in the same pinned toolbox image locally and in CI (R-TEST-02) and
|
||||
establishes, in order: the adopted spec pin (R-TEST-06), the handoff log's format, lint, L0 schema +
|
||||
policy + repository invariants, render-equals-committed, L1 rendering, the sensitive-path approval
|
||||
policy, that the live Gitea settings match their declaration, and L2/L3 conformance
|
||||
([14](14-testing-and-conformance.md); see [known gaps](16-known-gaps.md)).
|
||||
|
||||
Its verdict is one of three: **pass**, **blocked**, or **undetermined**. A stage that did not run is
|
||||
never green, and no automation may read a non-answer as approval. The rule is that a capability the
|
||||
environment *declares* absent is named in the verdict, while a stage that *could not run where it
|
||||
must* blocks. Detail: `ci/README.md` in the implementation repo.
|
||||
|
||||
## Who approves ([ADR-0024](../decisions/adr-0024-delivery-authority-ci-gate.md))
|
||||
|
||||
**The CI gate is the approver.** A green gate is sufficient for an ordinary change; branch
|
||||
protection requires no approval count. A weak gate is a defect to fix, not a reason to add a human.
|
||||
|
||||
**One exception: a change to the gate cannot be gated by the gate.** A change touching the pipeline,
|
||||
the tests and harness, the Flux wiring, the admission policies, the SOPS recipients and secrets, the
|
||||
RBAC, or the delivery contract needs an approving review from someone other than its author. The
|
||||
path list is data, with a stated reason per entry (`infra/gitea/approval-policy.json` in the
|
||||
implementation repo), and is itself a sensitive path. This stage is fail-closed: if it cannot
|
||||
evaluate, it blocks.
|
||||
|
||||
**Attributed administrator override.** There is one human on this platform, and Gitea refuses
|
||||
self-approval, so an administrator may waive that review with an explicit, reasoned directive on the
|
||||
PR. Authority is re-checked live, a new commit invalidates it, and the overrider, time and reason
|
||||
are recorded in the gate output. For an administrator the exception above is therefore advisory, not
|
||||
structural; what it guarantees is an attributed record of each waiver. **Agents never post this
|
||||
directive to clear their own work**, whatever permission their account holds.
|
||||
|
||||
## Secrets in GitOps
|
||||
|
||||
Secrets are committed **encrypted** with **SOPS + age**; Flux decrypts them in-cluster at
|
||||
@@ -64,15 +111,32 @@ and [ADR-0007](../decisions/adr-0007-secrets-sops-age.md).
|
||||
|
||||
## Automated updates interplay (R-UPD-*)
|
||||
|
||||
GitOps and hands-off updates coexist:
|
||||
- **Container image tags** advance automatically via **Flux image-update-automation** driven by
|
||||
per-service **image policies** (semver range / stable channel) — the controller resolves and
|
||||
reconciles the new version with **no human-authored change** (R-UPD-02).
|
||||
- **Helm chart / manifest version bumps** are proposed by **Renovate** with auto-merge for allowed
|
||||
update classes.
|
||||
GitOps and hands-off updates coexist because the update bot is one more contributor to the same
|
||||
flow. **Renovate** runs on a schedule and proposes machine-authored PRs for container images, Helm
|
||||
charts and manifest versions; each passes the merge gate, and allowed update classes auto-merge on
|
||||
green (R-UPD-02). No in-cluster controller holds a credential to write to the repo that defines the
|
||||
cluster. Full detail is in [`11-automated-updates.md`](11-automated-updates.md) and
|
||||
[ADR-0023](../decisions/adr-0023-automated-updates-renovate.md).
|
||||
|
||||
Full detail and the "is an automated commit a GitOps change?" nuance is in
|
||||
[`11-automated-updates.md`](11-automated-updates.md) and [ADR-0009](../decisions/adr-0009-automated-image-updates.md).
|
||||
## Images are built by CI
|
||||
|
||||
Container images the platform itself produces (the toolbox, the agent runner, the conformance
|
||||
runner) are built, verified and published only by the pipeline, never from a workstation. A PR
|
||||
builds and verifies; a push to `main` publishes to the Gitea registry. References are pinned by
|
||||
digest, and Renovate carries each new digest into git as a PR. **No CI job may reach a privileged
|
||||
container daemon**: builds go to a rootless builder over mutually-authenticated TLS.
|
||||
|
||||
## Delivery automation: pm-agent ([ADR-0025](../decisions/adr-0025-in-cluster-ai-program-manager.md))
|
||||
|
||||
An in-cluster AI Program Manager reads labelled issues on the implementation repo, decides what
|
||||
should happen about each thread, and runs one worker agent per assignment to author a change, run
|
||||
the merge gate and open a PR. It is delivery tooling rather than a catalog service: it declares
|
||||
Kubernetes objects a ServiceDefinition cannot express, and it requires no `R-*` of its own.
|
||||
|
||||
It is contributor-shaped, never approver-shaped. It cannot push to `main`, it never merges, its
|
||||
account is not an administrator, and its workers hold no cluster credential. It is an implementation
|
||||
agent bound by [AGENTS.md](../AGENTS.md) and R-TEST-06: spec work is out of its reach and is
|
||||
declined and escalated.
|
||||
|
||||
## Bootstrap & recovery
|
||||
|
||||
@@ -80,25 +144,21 @@ Full detail and the "is an automated commit a GitOps change?" nuance is in
|
||||
- Because desired state is in git and data is on Longhorn/backups, a rebuilt cluster reconverges by
|
||||
re-bootstrapping Flux and restoring volumes ([04](04-storage-and-state.md)).
|
||||
|
||||
## Source-of-truth: local bootstrap → self-hosted Gitea
|
||||
## Source-of-truth: self-hosted Gitea
|
||||
|
||||
The platform hosts its own git, which creates a deliberate bootstrapping sequence
|
||||
([`../CONTRIBUTING.md`](../CONTRIBUTING.md)):
|
||||
The platform hosts its own git at `gitea.home.keithjackson.dev`
|
||||
([`../CONTRIBUTING.md`](../CONTRIBUTING.md)); Flux reconciles from it and Gitea Actions runs the
|
||||
merge gate. Standing the platform up from nothing follows a deliberate order:
|
||||
|
||||
1. **Bootstrap source.** Initially there is no remote git on this network (the prior `home-network`
|
||||
Gitea is gone from here). Development is local; the cluster is first reconciled from a
|
||||
**bootstrap source** (this local repo / an out-of-band source).
|
||||
2. **Gitea comes online early.** **Gitea is one of the first services onboarded** ([13](13-service-catalog.md))
|
||||
so the platform gains its own git host at `gitea.home.keithjackson.dev`.
|
||||
3. **Cut Flux over to Gitea.** Once Gitea is stable, Flux's `GitRepository` source is migrated from
|
||||
the bootstrap source to Gitea; the platform then reconciles from its own git.
|
||||
4. **CI/CD moves to Gitea Actions.** The Kyverno L0 policy gate and the trigger for the in-cluster
|
||||
Chainsaw conformance Job ([14](14-testing-and-conformance.md)) run as Gitea Actions pipelines on
|
||||
push/PR; PRs become the enforced merge gate into `main`.
|
||||
1. **Bootstrap source.** The cluster is first reconciled from an out-of-band copy of the repo.
|
||||
2. **Gitea comes online early.** Gitea is one of the first services onboarded
|
||||
([13](13-service-catalog.md)).
|
||||
3. **Cut Flux over to Gitea**, so the platform reconciles from its own git.
|
||||
4. **CI/CD runs in Gitea Actions**, with PRs as the enforced merge gate into `main`.
|
||||
|
||||
**Circularity & DR.** Gitea runs *on* the cluster Flux reconciles *from* Gitea. Mitigations:
|
||||
- Keep an **off-cluster mirror** of the platform repo so a source-of-truth exists that outlives the
|
||||
cluster.
|
||||
cluster (see [known gaps](16-known-gaps.md)).
|
||||
- Treat **Gitea as DR-critical**: its repos/DB are backed up off-cluster ([04](04-storage-and-state.md)),
|
||||
so a cluster loss never loses the git history needed to rebuild the platform.
|
||||
|
||||
@@ -112,4 +172,4 @@ The platform hosts its own git, which creates a deliberate bootstrapping sequenc
|
||||
- Adding a ServiceDefinition PR results in a fully wired service (compute, storage, policy, TLS, DNS,
|
||||
monitoring) with no manual steps beyond the merge.
|
||||
- Manually mutating a live resource is auto-reverted by Flux (drift correction proven).
|
||||
- The same base config renders correctly for both dev and prod overlays.
|
||||
- The base config renders correctly under the environment overlay, which supplies scoped values only.
|
||||
@@ -12,8 +12,8 @@ Kubernetes custom resources.
|
||||
|
||||
## Do not confuse the two VM layers
|
||||
|
||||
- **Layer 1 — cluster nodes** are VMs only in dev/test (libvirt) and bare metal in prod. Managed by
|
||||
OpenTofu/Ansible ([02](02-environments-and-provisioning.md)). **Not** KubeVirt.
|
||||
- **Layer 1 — cluster nodes** are bare-metal mini-PCs, baselined and managed as code
|
||||
([02](02-environments-and-provisioning.md)). **Not** KubeVirt.
|
||||
- **Layer 2 — application VMs** run *inside* the cluster via KubeVirt and are the subject of this
|
||||
spec. Managed by Kubernetes/GitOps like any other workload.
|
||||
|
||||
@@ -22,7 +22,8 @@ Kubernetes custom resources.
|
||||
- `VirtualMachine` / `VirtualMachineInstance` custom resources — declare a VM in YAML, reconcile via
|
||||
Flux, manage via `kubectl` (and `virtctl` for console/start/stop).
|
||||
- **CDI (Containerized Data Importer)** to import/clone disk images into **Longhorn** PVCs — VM disks
|
||||
get the same replication and backup story as container volumes ([04](04-storage-and-state.md)).
|
||||
get the same replication as container volumes and are backed up by default
|
||||
([04](04-storage-and-state.md)).
|
||||
- VMs sit on the **same Cilium network** — so **default-deny isolation and egress classes apply to
|
||||
VMs too** (R-NET-*). A KubeVirt VM can be `egress: vpn` just like a container.
|
||||
- VMs are exposed through the same **Gateway API / Service** model and monitored through the same
|
||||
@@ -39,6 +40,14 @@ A VM onboarded via a ServiceDefinition (`kind: vm`) gets, from the *same* declar
|
||||
- Backup/DR of its disks (R-STATE-03) and rebuildability (R-STATE-01): destroy the VM, keep the
|
||||
disk PVC, recreate from the declaration.
|
||||
|
||||
`kind: vm` is the catalog path. A VM with a custom install flow MAY instead be declared directly in
|
||||
git under the same controls: Flux reconciliation, default-deny networking, Longhorn disks and
|
||||
platform monitoring, in its own VM tier that never gates the application waves.
|
||||
|
||||
**Recorded per-VM exception:** UniFi's disk is excluded from volume backup; its Kubernetes resources
|
||||
are captured, and the design relies on UniFi's own configuration export for application state
|
||||
([04](04-storage-and-state.md)).
|
||||
|
||||
## Update policy for VMs (R-UPD-*)
|
||||
|
||||
VM "updates" differ from container image tag bumps:
|
||||
@@ -59,17 +68,22 @@ Prefer containers (PaaS) by default. Choose a KubeVirt VM (IaaS) when:
|
||||
- Lifting an existing VM is faster/safer than re-platforming.
|
||||
- Strong OS-level isolation is required beyond namespace/network policy.
|
||||
|
||||
For the initial catalog, all 13 named services are container-first; KubeVirt is the escape hatch and
|
||||
must be **proven** in dev/test with at least one representative VM workload so the IaaS path is real,
|
||||
not theoretical.
|
||||
The catalog is container-first and KubeVirt is the escape hatch. **UniFi is the production VM
|
||||
workload**: UniFi OS Server is an installer that expects a full Ubuntu host and runs its own container
|
||||
stack and embedded datastore, so it is an appliance image in all but name. It is hand-authored in git
|
||||
under the rule above. The catalog's `vm-demo` entry is the `kind: vm` reference and conformance
|
||||
fixture and is not deployed. The IaaS path must be **proven** with at least one representative VM
|
||||
workload so it is real, not theoretical.
|
||||
|
||||
## Prerequisites & constraints
|
||||
- Nodes must expose virtualization extensions (nested virt in dev/test KVMs; native on prod mini-
|
||||
PCs) — captured in the node baseline ([02](02-environments-and-provisioning.md)).
|
||||
- Live migration across 3 converged nodes is a nice-to-have, not a requirement; document whether it's
|
||||
enabled.
|
||||
- Nodes must expose virtualization extensions (native on the mini-PCs) — captured in the node baseline
|
||||
([02](02-environments-and-provisioning.md)). VMs never fall back silently to software emulation: a
|
||||
node without KVM fails loudly.
|
||||
- Live migration is **enabled and opt-in per VM**: the VM sets a live-migrate eviction strategy and
|
||||
keeps its disks on the migratable Longhorn class. There is no cluster-wide policy, because forcing
|
||||
it would wedge node drains for VMs on non-migratable disks. Other VMs restart on drain.
|
||||
- Resource pressure: VMs are heavier than pods — capacity planning must account for them,
|
||||
particularly on the 3-node prod cluster.
|
||||
particularly on the 3-node cluster.
|
||||
|
||||
## Acceptance gates
|
||||
- Create, start, stop, and destroy a VM entirely through Kubernetes YAML + `kubectl`/`virtctl`.
|
||||
|
||||
@@ -17,22 +17,32 @@ Defense in depth with declarative, auditable controls:
|
||||
- **Supply chain**: pin images by digest where automation allows; prefer known-good upstreams;
|
||||
updates flow through policy, not ad-hoc pulls ([11](11-automated-updates.md)).
|
||||
- **Identity / SSO**: a central IdP (**Keycloak**) provides single sign-on, one user directory,
|
||||
group-based authorization, and MFA; services authenticate via native OIDC or an oauth2-proxy
|
||||
forward-auth layer so no management UI is unauthenticated ([15](15-identity-and-access.md),
|
||||
[ADR-0014](../decisions/adr-0014-idp-keycloak.md), R-IDP-*). This is the primary *authentication*
|
||||
control, complementing network default-deny (the primary *isolation* control).
|
||||
- **Policy enforcement (admission)**: **Kyverno** runs in-cluster as an admission controller and
|
||||
rejects non-conforming resources at reconcile time — e.g. a namespace missing its default-deny
|
||||
baseline, or a workload without a required egress-class/auth-mode label. The *same* policies
|
||||
gate CI ([14](14-testing-and-conformance.md), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md)),
|
||||
so invariants are enforced both before merge and at admission (defense in depth). Enforcement
|
||||
complements — never replaces — the runtime behavior tests.
|
||||
group-based authorization, and MFA; services authenticate via native OIDC or a per-app
|
||||
oauth2-proxy forward-auth gate so no management UI is unauthenticated
|
||||
([15](15-identity-and-access.md), [ADR-0014](../decisions/adr-0014-idp-keycloak.md),
|
||||
[ADR-0022](../decisions/adr-0022-forward-auth-per-app-oauth2-proxy-sidecars.md), R-IDP-*). This is
|
||||
the primary *authentication* control, complementing network default-deny (the primary *isolation*
|
||||
control).
|
||||
- **Policy enforcement (admission)**: **Kyverno** runs in-cluster as an admission controller over
|
||||
**application namespaces**. It generates and backfills each namespace's default-deny baseline,
|
||||
rejects a malformed baseline, and rejects a workload that lacks a required egress-class/auth-mode
|
||||
label or uses host namespaces or privileged mode. Separate enforcing policies constrain the exact
|
||||
Job shapes that CI and agent credentials may create; these are security boundaries, so they fail
|
||||
closed and no environment overlay can downgrade them. No policy-exception mechanism is in use:
|
||||
controller-generated pods are mutated to carry truthful labels instead. Platform namespaces sit
|
||||
outside these policies and carry hand-authored equivalents. Policies are authored once and gate
|
||||
CI ([14](14-testing-and-conformance.md), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md)),
|
||||
so invariants are enforced both before merge and at admission (defense in depth); the job-shape
|
||||
policies act only at admission and are unit-tested in CI. Enforcement complements — never
|
||||
replaces — the runtime behavior tests.
|
||||
- **Plaintext-secret enforcement is split by layer (do not conflate):** the "no plaintext Secret"
|
||||
invariant is a **CI-time** check (the committed Secret must carry SOPS ciphertext). It **cannot**
|
||||
be an admission rule, because Flux **decrypts SOPS before** the object reaches the API server —
|
||||
an admitted Secret never carries ciphertext, so an admission "reject plaintext" rule would reject
|
||||
*every* real Secret. Admission instead enforces **SOPS provenance** (the Secret was reconciled
|
||||
through the Flux/SOPS path), not ciphertext presence.
|
||||
*every* real Secret. Admission instead enforces **SOPS provenance** on application-namespace
|
||||
Secrets (the Secret was reconciled through the Flux/SOPS path), not ciphertext presence.
|
||||
- Host-namespace and privileged-pod control is Kyverno-based for the application tier; Pod
|
||||
Security Admission labels are applied selectively, on namespaces with their own need for them.
|
||||
- **Auditability**: isolation, egress path, cert validity, and backup integrity are all
|
||||
demonstrable (R-SEC-03) via Hubble, cert status, and DR drills.
|
||||
|
||||
@@ -53,8 +63,13 @@ Defense in depth with declarative, auditable controls:
|
||||
- DNS provider API token (Cloudflare) for ACME DNS-01 + ExternalDNS.
|
||||
- Cloudflare Tunnel credential/token for in-cluster cloudflared (SOPS-encrypted; ADR-0016).
|
||||
- SMTP credentials for Alertmanager email.
|
||||
- Service credentials (Nextcloud, Gitea admin, UniFi, Semaphore, Kasm, DB passwords).
|
||||
- Off-cluster backup target credentials (S3/NFS) for Velero/Longhorn.
|
||||
- Service credentials (Nextcloud, Gitea admin, UniFi, Semaphore, DB passwords; Kasm is not onboarded).
|
||||
- Per-app forward-auth gate credentials (IdP client secret + cookie secret; [ADR-0022](../decisions/adr-0022-forward-auth-per-app-oauth2-proxy-sidecars.md)).
|
||||
- WireGuard ingress server key, held in SOPS because a rebuild must restore it without re-enrolling
|
||||
every client (ADR-0021).
|
||||
- Off-cluster backup target credentials (Cloudflare R2) for Velero and the database archives (ADR-0018).
|
||||
- In-cluster AI agent credentials (ADR-0025) and infrastructure-as-code runtime secrets (state
|
||||
encryption, remote-state keys).
|
||||
- Grafana admin, and any API tokens the C2 UI needs.
|
||||
|
||||
### Rotation
|
||||
@@ -73,10 +88,14 @@ Defense in depth with declarative, auditable controls:
|
||||
- Platform controllers (Flux, cert-manager, ExternalDNS, Velero, KubeVirt) get scoped permissions;
|
||||
application workloads get near-zero cluster permissions.
|
||||
- The C2 UI's operational actions run through a constrained, audited path — not a shared admin
|
||||
kubeconfig handed to a browser.
|
||||
kubeconfig handed to a browser. The Kubernetes console forwards each user's own IdP-issued token,
|
||||
so an action runs under that user's group-bound RBAC; the console's ServiceAccount holds no
|
||||
cluster rights (see [known gaps](16-known-gaps.md)).
|
||||
- Human access to the Kubernetes API is Keycloak OIDC, with roles bound to IdP groups
|
||||
([15](15-identity-and-access.md)).
|
||||
|
||||
## Node & host security
|
||||
- Nodes baselined via cloud-init/Ansible ([02](02-environments-and-provisioning.md)); SSH key-only,
|
||||
- Nodes baselined as code with Ansible ([02](02-environments-and-provisioning.md)); SSH key-only,
|
||||
minimal packages, required sysctls only.
|
||||
- Host-level changes are IaC, not hand edits (drift + auditability).
|
||||
|
||||
|
||||
@@ -26,14 +26,15 @@ services (R-TLS-01/02/03).
|
||||
- Certificates are requested declaratively — either a **wildcard** `*.home.keithjackson.dev` cert
|
||||
shared by the Gateway, or **per-service** certs, decided per ADR-0006. Recommendation: a wildcard
|
||||
cert for `*.home.keithjackson.dev` terminated at the Gateway simplifies onboarding (a new service
|
||||
needs no new cert), with per-service certs available where isolation warrants.
|
||||
needs no new cert), with per-service certs available where isolation warrants (Plex's L4 VIP name
|
||||
has its own).
|
||||
- The DNS provider API token is a SOPS-encrypted secret ([07](07-security-and-secrets.md)).
|
||||
|
||||
## DNS-01 challenge access (OQ-4 — RESOLVED: scoped Cloudflare API token)
|
||||
|
||||
Cloudflare already has a wildcard `*.keithjackson.dev` → public-IP record, which does not cover
|
||||
certificate issuance (ACME DNS-01 must **write `_acme-challenge` TXT records**,
|
||||
and that wildcard does not match two-label-deep names like `plex.home.keithjackson.dev`).
|
||||
Certificate issuance needs programmatic write access to the zone: ACME DNS-01 must **write
|
||||
`_acme-challenge` TXT records**, including for two-label-deep names like
|
||||
`plex.home.keithjackson.dev`.
|
||||
|
||||
**Decision (OQ-4):** use a **scoped Cloudflare API token** — `Zone·DNS·Edit` limited to the
|
||||
`keithjackson.dev` zone — stored **SOPS-encrypted** ([07](07-security-and-secrets.md)). With it:
|
||||
@@ -42,6 +43,7 @@ and that wildcard does not match two-label-deep names like `plex.home.keithjacks
|
||||
services (R-TLS-03).
|
||||
- **ExternalDNS** manages the `*.home.keithjackson.dev` records automatically ([09](09-dns-and-ingress.md),
|
||||
R-DNS-02) — no manual record editing for public services.
|
||||
- **cloudflare-ddns** keeps the DNS-only `wan.` record current for WireGuard ([09](09-dns-and-ingress.md)).
|
||||
|
||||
**Operator prerequisite:** create the scoped token in Cloudflare and hand it over once; it is then
|
||||
committed SOPS-encrypted and never touched manually again. The token is least-privilege (single zone,
|
||||
@@ -71,9 +73,10 @@ DNS edit only) so its blast radius is bounded (R-SEC-02). Rotation = re-encrypt
|
||||
|
||||
## Trust chain for internal clients
|
||||
|
||||
Because certs come from Let's Encrypt (publicly trusted), internal clients trust them with no custom
|
||||
CA distribution. Split-horizon DNS ([09](09-dns-and-ingress.md)) resolves
|
||||
`<service>.home.keithjackson.dev` to the internal Gateway while the same trusted cert is served.
|
||||
Both certificates a client can meet are publicly trusted, so internal clients need no custom CA
|
||||
distribution. A name Pi-hole holds a local record for ([09](09-dns-and-ingress.md)) resolves to the
|
||||
internal Gateway and serves the Let's Encrypt wildcard; any other `public` name resolves to
|
||||
Cloudflare's edge and serves its edge certificate.
|
||||
|
||||
## Acceptance gates
|
||||
- `https://<service>.home.keithjackson.dev` serves a valid, non-self-signed, unexpired certificate
|
||||
@@ -84,6 +87,5 @@ CA distribution. Split-horizon DNS ([09](09-dns-and-ingress.md)) resolves
|
||||
|
||||
## Resolved
|
||||
- **OQ-4** — DNS-01 via a **scoped Cloudflare API token** (`Zone·DNS·Edit`, `keithjackson.dev`),
|
||||
SOPS-encrypted; drives both cert-manager DNS-01 and ExternalDNS. Phase 2 TLS/DNS is unblocked once
|
||||
the operator provides the token.
|
||||
SOPS-encrypted; drives cert-manager DNS-01, ExternalDNS and the `wan.` record updater.
|
||||
- Wildcard vs. per-service certs default — **wildcard** `*.home.keithjackson.dev` (ADR-0006).
|
||||
+33
-14
@@ -2,7 +2,8 @@
|
||||
|
||||
Serves: **R-DNS-01/02/03/04/05**, **R-C2-01**, **R-SEC-04**, works with **R-TLS-***. Related:
|
||||
[ADR-0002 (Cilium/Gateway API)](../decisions/adr-0002-cni-cilium.md),
|
||||
[ADR-0016 (Cloudflare Tunnel public ingress)](../decisions/adr-0016-public-ingress-cloudflare-tunnel.md).
|
||||
[ADR-0016 (Cloudflare Tunnel public ingress)](../decisions/adr-0016-public-ingress-cloudflare-tunnel.md),
|
||||
[ADR-0021 (inbound remote access)](../decisions/adr-0021-inbound-remote-access-wireguard.md).
|
||||
|
||||
## Naming model
|
||||
|
||||
@@ -18,8 +19,10 @@ Serves: **R-DNS-01/02/03/04/05**, **R-C2-01**, **R-SEC-04**, works with **R-TLS-
|
||||
authoritative DNS provider (Cloudflare, per OQ-4) automatically — a new service's DNS appears from
|
||||
its declared intent, no manual record editing.
|
||||
- Records for `*.home.keithjackson.dev` are created/updated/removed as services come and go.
|
||||
- Public records are created **only for `exposure: public` services** (R-DNS-05). The DNS-only
|
||||
(unproxied) `wan.` record for WireGuard is kept current by cloudflare-ddns, outside this flow.
|
||||
- Public records are created **only for services declared public** (see the exposure class below,
|
||||
R-DNS-05). The non-HTTP L4 names **unifi** and **crafty-mc** are the one exception: they hold DNS-only
|
||||
records at their LAN addresses, and are never proxied or tunnelled. The DNS-only (unproxied) `wan.`
|
||||
record for WireGuard is kept current by cloudflare-ddns, outside this flow.
|
||||
|
||||
## Split-horizon resolution (R-DNS-03)
|
||||
|
||||
@@ -27,7 +30,7 @@ Internal and external answers differ:
|
||||
- **External** (`<service>.home.keithjackson.dev` from the internet): a `public` service resolves to
|
||||
**Cloudflare's edge** (proxied CNAME to the tunnel). Cloudflare forwards the request down the
|
||||
outbound-only tunnel to cloudflared in the cluster, which forwards it to the Gateway (ADR-0016).
|
||||
`internal` services have no public answer.
|
||||
`internal` services have no public answer, apart from the L4 exception above.
|
||||
- **Internal** (LAN clients): Pi-hole answers with the **internal Gateway address** (or the service's
|
||||
L4 VIP) for the names it holds local records for.
|
||||
- **Correctness does not depend on split horizon for `public` web apps** (R-DNS-04). The ISP router
|
||||
@@ -40,6 +43,11 @@ Internal and external answers differ:
|
||||
- the names that must avoid Cloudflare's 100 MB body / 100 s timeout limits or must work during an
|
||||
internet outage: **gitea**, **nextcloud**, **keycloak/auth**.
|
||||
|
||||
It holds no zone-wide entry. Every other name is forwarded upstream, so the remaining `public` names
|
||||
resolve to Cloudflare from the LAN too and depend on the internet and the tunnel. Its DNS address
|
||||
accepts queries only from the LAN and WireGuard clients. Pi-hole runs in the cluster, so LAN DNS
|
||||
depends on cluster health; an off-cluster resolver is an operator choice not taken.
|
||||
|
||||
A trusted certificate is served either way: Cloudflare's edge certificate on the tunnel path, the
|
||||
Let's Encrypt wildcard on the LAN path ([08](08-tls-and-certificates.md)).
|
||||
- **In-cluster** resolution keeps its own split horizon (CoreDNS) and the node `/etc/hosts` pins (OIDC
|
||||
@@ -58,11 +66,17 @@ Internal and external answers differ:
|
||||
- Each service contributes an **HTTPRoute** mapping its hostname to its Service — rendered from the
|
||||
ServiceDefinition, not hand-authored.
|
||||
- **Exposure class** is declared per service:
|
||||
- `internal` — reachable only on the LAN/VPN (no tunnel route, no public record; internal DNS only).
|
||||
- `internal` — reachable only on the LAN/VPN (refused at the tunnel, no public record; internal DNS only).
|
||||
- `public` — additionally reachable from the internet through the **Cloudflare Tunnel**:
|
||||
cloudflared → the Gateway over HTTPS (ADR-0016).
|
||||
cloudflared → the Gateway over HTTPS (ADR-0016). A service is public by catalog
|
||||
`exposure: public`, or as a **registered platform route**: `auth`, Grafana, Crafty's web UI, and
|
||||
the apex with its `/runbooks`. Platform routes have no catalog entry, so they are held in a
|
||||
shrink-only register, enforced by the static check
|
||||
`conformance/tests/l0-static/internal-services-publish-no-public-dns` in the
|
||||
[implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl).
|
||||
- **Never `public`:** jackett, pihole, semaphore, qbittorrent, radarr, sonarr, **headlamp** (a front
|
||||
end to the Kubernetes API), plex's web UI, and the non-HTTP L4 services unifi and crafty-mc.
|
||||
end to the Kubernetes API), plex's web UI, and the non-HTTP L4 services unifi and crafty-mc (which
|
||||
hold only the DNS-only LAN records above).
|
||||
- **The Kubernetes API is never published** — no tunnel route, no edge forward, no public record
|
||||
(R-SEC-04).
|
||||
|
||||
@@ -70,9 +84,12 @@ Internal and external answers differ:
|
||||
|
||||
- **cloudflared** runs in-cluster as a Flux-managed workload and holds an **outbound-only**
|
||||
connection to Cloudflare. No inbound 80/443 reaches the WAN IP.
|
||||
- Its ingress rules list one route per `public` hostname → the Gateway (HTTPS, origin certificate
|
||||
validated with the hostname as SNI). The rules end in a **catch-all `http_status:404`**, so an
|
||||
unlisted hostname is refused at the tunnel (R-DNS-05).
|
||||
- Its ingress rules are a **deny-list plus a zone wildcard**. Each LAN/VPN-only hostname maps to
|
||||
`http_status:404`; the apex and `*.home.keithjackson.dev` then go to the Gateway (HTTPS, origin
|
||||
certificate validated with the hostname as SNI); a **catch-all `http_status:404`** closes the list
|
||||
for hosts outside the zone (R-DNS-05). A static check keeps the deny-list and the set of unpublished
|
||||
routes identical in both directions, so a LAN/VPN-only name cannot be published or tunnelled by
|
||||
omission.
|
||||
- **Edge TLS** is Cloudflare ACM/Total TLS (Universal SSL does not cover two-label-deep names);
|
||||
**origin TLS** is the Let's Encrypt wildcard at the Gateway ([08](08-tls-and-certificates.md)).
|
||||
- **Limits and trade-offs:**
|
||||
@@ -86,7 +103,7 @@ Internal and external answers differ:
|
||||
|
||||
- The environment has **one SOHO edge router**: no VLANs, no VPN, no services, no API/IaC. It
|
||||
**does not forward 80/443** (public web ingress is the tunnel, ADR-0016). Its only remaining
|
||||
inbound forwards are **WireGuard udp/51820** (remote access, via the DNS-only `wan.` record) and
|
||||
inbound forwards are **WireGuard udp/51820** (remote access, ADR-0021, via the DNS-only `wan.` record) and
|
||||
**Plex tcp/32400** (remote streaming via plex.tv; Cloudflare's terms restrict proxied video). This
|
||||
is a documented manual edge step (the SOHO device has no automation), the analogue of the old
|
||||
pfSense manual-config exception. The WAN IP therefore stays public.
|
||||
@@ -111,7 +128,9 @@ but **not** `plex.home.keithjackson.dev`. Consequences:
|
||||
|
||||
Ingress opens **north-south** paths to the specific service; it does **not** relax **east-west**
|
||||
default-deny ([03](03-networking-isolation-and-egress.md)). A service being publicly reachable does
|
||||
not let other in-cluster services reach it unless an explicit allow rule exists.
|
||||
not let other in-cluster services reach it unless an explicit allow rule exists. Cilium's Gateway
|
||||
re-evaluates the calling pod's egress policy against the real backend, so any in-cluster client of the
|
||||
Gateway, cloudflared included, needs an explicit egress allow to the backends it reaches.
|
||||
|
||||
## Acceptance gates
|
||||
- A new service's `<service>.home.keithjackson.dev` record appears automatically and routes to it.
|
||||
@@ -121,8 +140,8 @@ not let other in-cluster services reach it unless an explicit allow rule exists.
|
||||
records for, and get a trusted cert.
|
||||
- A LAN client resolving a `public` name through a **public** resolver reaches it, TLS-valid
|
||||
(R-DNS-04). The WAN IP does not answer on 80/443.
|
||||
- A request at the tunnel for any hostname that is not a `public` service returns **404**, and no
|
||||
`internal` service has a public DNS record (R-DNS-05).
|
||||
- A request at the tunnel for a LAN/VPN-only hostname returns **404**, and no `internal` service has
|
||||
a public DNS record, except the DNS-only L4 names (R-DNS-05).
|
||||
- The Kubernetes API is unreachable from the internet: no tunnel route, no edge forward, no public
|
||||
record (R-SEC-04).
|
||||
- The root `home.keithjackson.dev` serves the C2 UI.
|
||||
|
||||
@@ -7,7 +7,7 @@ Serves: **R-OBS-01/02/03/04**, supports **R-C2-02**, **R-SEC-03**.
|
||||
- **kube-prometheus-stack** — Prometheus (metrics), Alertmanager (routing/notify), Grafana
|
||||
(dashboards), and the standard Kubernetes exporters/rules.
|
||||
- **Loki** — log aggregation.
|
||||
- **Grafana Alloy** (or Promtail-class agent) — ships node/pod/VM logs to Loki.
|
||||
- **Grafana Alloy** — ships node/pod/VM logs to Loki.
|
||||
- **Hubble** — network flow visibility from Cilium ([03](03-networking-isolation-and-egress.md)).
|
||||
- **Alertmanager → SMTP** — issues reported by **email** (R-OBS-02).
|
||||
- **Grafana** — dashboards for services and platform health (R-OBS-03).
|
||||
@@ -17,18 +17,29 @@ Serves: **R-OBS-01/02/03/04**, supports **R-C2-02**, **R-SEC-03**.
|
||||
Application services:
|
||||
- Up/health, resource use, request/error signals where exposed, and per-service SLIs declared in the
|
||||
ServiceDefinition (a service can declare the alerts it cares about).
|
||||
- Each forward-auth gate ([15](15-identity-and-access.md)) is scraped in its own right, and the
|
||||
platform alerts on a gate that nothing observes: an auth failure otherwise looks like a healthy
|
||||
response.
|
||||
|
||||
Platform health (R-OBS-04) — first-class alerts:
|
||||
- **Nodes**: down/unreachable, resource pressure, disk fill.
|
||||
- **Storage (Longhorn)**: degraded/faulted volumes, replica rebuilds, capacity thresholds.
|
||||
- **Nodes**: down/unreachable, CPU, memory and I/O pressure, disk fill, disk latency.
|
||||
- **Storage (Longhorn + NFS bulk tier)**: degraded/faulted volumes, replica rebuilds, capacity
|
||||
thresholds, PVC capacity, NFS server latency and client timeouts.
|
||||
- **Databases (CloudNativePG)**: archiving failures, missed backups, primary placement.
|
||||
- **GitOps (Flux)**: reconciliation failures, drift, suspended resources.
|
||||
- **Automated updates**: image automation failures, stalled image policies, failed health gates,
|
||||
node reboot automation.
|
||||
- **Certificates (cert-manager)**: upcoming expiry, issuance failures.
|
||||
- **Networking**: VPN tunnel down, egress-gateway unhealthy, spikes in denied flows (Hubble),
|
||||
NetworkPolicy reconcile errors.
|
||||
NetworkPolicy reconcile errors, public ingress tunnel down.
|
||||
- **Backups (Velero/Longhorn)**: backup job failure, missed schedule, restore-test failures.
|
||||
- **KubeVirt**: VM crash-loops / failed launches.
|
||||
- **Identity (Keycloak)**: IdP down/unhealthy or its DB unavailable (SSO is a critical dependency —
|
||||
[15](15-identity-and-access.md)); certificate/login-endpoint failures.
|
||||
[15](15-identity-and-access.md)); certificate/login-endpoint failures; a failing or stale
|
||||
identity-config reconciler.
|
||||
- **The monitor itself**: monitoring volume capacity, missing scrape targets.
|
||||
|
||||
Coverage against this list is tracked in [known gaps](16-known-gaps.md).
|
||||
|
||||
## Alert delivery (R-OBS-02)
|
||||
|
||||
@@ -36,7 +47,12 @@ Platform health (R-OBS-04) — first-class alerts:
|
||||
- Severity model: `critical` (page-worthy: data risk, outage, VPN leak potential, cert about to
|
||||
expire) vs `warning` (degraded, capacity, needs attention). Both go to email; only routing/subject
|
||||
differs.
|
||||
- Alerts include enough context (runbook link, affected object) to act without spelunking.
|
||||
- **Deploy notices** are a separate, non-severity notification class, also by email: one message
|
||||
when Flux starts applying a revision and one when every Kustomization has reached it, so a routine
|
||||
deploy never reads as a platform fault.
|
||||
- Alerts include enough context (runbook link, affected object) to act without spelunking. Every
|
||||
alert's runbook link resolves to a runbook served under `/runbooks` on the apex, and every rule has
|
||||
a unit test; the merge gate checks both.
|
||||
|
||||
## Dashboards (R-OBS-03)
|
||||
|
||||
@@ -49,13 +65,16 @@ Grafana dashboards, surfaced in the C2 UI ([12](12-command-center-ui.md)):
|
||||
## Observability as onboarding output
|
||||
|
||||
The ServiceDefinition ([13](13-service-catalog.md)) declares observability expectations; the
|
||||
platform renders **ServiceMonitors/PodMonitors**, log scraping, dashboards, and alert rules from it.
|
||||
A newly onboarded service is monitored **by default**, not as a follow-up task (part of the golden
|
||||
path, [01](01-architecture-overview.md)).
|
||||
platform renders **ServiceMonitors/PodMonitors** for services that declare a metrics endpoint, and
|
||||
dashboards and alert rules from its declared SLIs. Log collection needs no per-service rendering:
|
||||
the log agent discovers every pod. A newly onboarded service is monitored **by default**, not as a
|
||||
follow-up task (part of the golden path, [01](01-architecture-overview.md)).
|
||||
|
||||
## Retention & sizing
|
||||
- Metrics/log retention set per environment overlay (dev shorter, prod longer). Loki retention
|
||||
requires a compactor with retention enabled — do not rely on a period setting alone.
|
||||
- Metrics/log retention is set in the environment overlay. Metrics retention is capped by size as
|
||||
well as age, so a young or busy TSDB cannot fill its volume and blind the alerts that depend on
|
||||
it. Loki retention requires a compactor with retention enabled — do not rely on a period setting
|
||||
alone.
|
||||
- Monitoring storage lives on Longhorn; ensure its own capacity is alerted on (don't let the monitor
|
||||
blind itself).
|
||||
|
||||
|
||||
@@ -1,62 +1,80 @@
|
||||
# 11 — Automated Updates
|
||||
|
||||
Serves: **R-UPD-01/02/03**. Related: [ADR-0009](../decisions/adr-0009-automated-image-updates.md).
|
||||
Serves: **R-UPD-01/02/03**. Related: [ADR-0023](../decisions/adr-0023-automated-updates-renovate.md).
|
||||
|
||||
## The requirement, precisely
|
||||
|
||||
> Services must update to latest **stable** versions **on a schedule**, with **no human, no AI, and
|
||||
> no hand-authored GitOps change**. (R-UPD-01/02)
|
||||
> Services must update to latest **stable** versions **on a schedule**; updates in the allowed
|
||||
> classes occur with **no human, no AI, and no hand-authored GitOps change**. (R-UPD-01/02)
|
||||
|
||||
This has a subtle tension with GitOps (where "the desired state is what's in git"). We resolve it
|
||||
explicitly below.
|
||||
|
||||
## Resolving "no GitOps change" vs. GitOps
|
||||
|
||||
There are two interpretations; we honor the operator's intent (hands-off), and ADR-0009 records the
|
||||
There are two interpretations; we honor the operator's intent (hands-off), and ADR-0023 records the
|
||||
mechanism:
|
||||
|
||||
- **A human/AI authoring a version bump = prohibited.** No person and no chatbot edits a tag.
|
||||
- **An automation controller advancing a version under a policy declared once = permitted**, because
|
||||
the *operator's* declared intent (the update policy / semver range) is the stable, human-authored
|
||||
thing; the controller merely resolves it. Two mechanisms, chosen by artifact type:
|
||||
Reviewing a machine-authored change is not authoring one.
|
||||
- **An automation bot proposing a version advance under a policy declared once = permitted**,
|
||||
because the *operator's* declared intent (the update policy / version range) is the stable,
|
||||
human-authored thing; the bot merely resolves it, and its change reaches `main` through the same
|
||||
merge gate as every other change ([05](05-gitops-and-delivery.md)). Mechanisms by artifact type:
|
||||
|
||||
### 1. Container image tags → Flux image-update-automation
|
||||
- Per service, an **ImagePolicy** declares the acceptable range/channel (e.g. semver `>=1.0.0`
|
||||
minor/patch, or a `stable` tag pattern). Flux's image-reflector scans the registry; image-
|
||||
automation advances the running version automatically **on its scan interval** (the schedule).
|
||||
- Where automation writes the resolved version back to git, that commit is machine-authored under
|
||||
the declared policy — no human/AI touches it (satisfies R-UPD-02 in spirit and practice).
|
||||
- Alternatively, digest-tracking of a mutable stable tag avoids per-update commits entirely. ADR-0009
|
||||
picks the default and the per-service escape hatch.
|
||||
### 1. Container images, Helm charts and manifests → Renovate
|
||||
- **Renovate** runs on a schedule, watches image tags, chart and dependency versions, and opens PRs.
|
||||
Git history records every advance as an auditable, machine-authored commit.
|
||||
- **Minor, patch and digest** updates **auto-merge** on a green gate. This keeps services and
|
||||
platform components current with no human authoring.
|
||||
- **Major** updates never auto-merge; they wait for human review of the machine-authored PR.
|
||||
- **Operator-designated blast-radius components** never auto-merge, not even a patch: the storage and
|
||||
network planes, the reboot daemon, the database engine, the metrics stack, Gitea and its runner
|
||||
(they *are* the merge gate), the identity provider and its auth sidecars, and Renovate itself and
|
||||
the pinned CI toolchain. The list is data in the implementation repo's `renovate.json5`, each entry
|
||||
with its reason.
|
||||
- Because `apps/` is rendered from `catalog/`, a bump updates the source and its rendered output in
|
||||
one PR, so render-equals-committed holds.
|
||||
|
||||
### 2. Helm charts / manifests → Renovate
|
||||
- **Renovate** watches chart and dependency versions and opens PRs; allowed update classes
|
||||
**auto-merge** on green CI. This keeps platform components current with no human authoring.
|
||||
- Chart *major* bumps (potentially breaking) can require review — configurable per policy.
|
||||
### 2. The platform's own images → Renovate digest tracking
|
||||
- Images built by CI ([05](05-gitops-and-delivery.md)) are referenced by digest. Renovate follows
|
||||
each image's moving tag and proposes the digest bump as a PR, whose own gate run exercises it where
|
||||
the image is the toolbox.
|
||||
|
||||
### 3. VM guests → in-guest automation
|
||||
- KubeVirt guests auto-update OS packages inside the guest on a schedule (unattended-upgrades / dnf-
|
||||
automatic) declared in the VM's cloud-init ([06](06-vm-workloads-kubevirt.md)). Base-image
|
||||
promotion follows the same automated philosophy where a golden-image pipeline exists.
|
||||
|
||||
### 4. Node OS → security-only upgrades, coordinated reboots
|
||||
- Nodes apply **security-only** unattended upgrades, with kernels excluded. A reboot daemon performs
|
||||
only the mechanical half of a reboot, when a node requests one: one node at a time under a single
|
||||
cluster-wide lock, gated on storage health so etcd quorum and volume replicas are never put at
|
||||
risk. A hang stops the cycle and pages a human. Kernel upgrades stay a deliberate operator act.
|
||||
|
||||
## Per-service policy (R-UPD-03)
|
||||
|
||||
Each ServiceDefinition ([13](13-service-catalog.md)) declares an **update policy**:
|
||||
Each service's **update policy** is declared in git:
|
||||
- **channel/range**: which versions are "stable" and acceptable (semver range, tag regex).
|
||||
- **cadence**: how often to check/apply (the schedule).
|
||||
- **hold/pin**: temporarily freeze a service (e.g. a known-bad upstream release).
|
||||
- **rollout & health**: how the new version rolls out and what health gates it.
|
||||
- **auto-rollback**: revert on failed health so a bad upstream release self-heals.
|
||||
|
||||
The ServiceDefinition ([13](13-service-catalog.md)) records that intent. Renovate does not read it,
|
||||
so the policy takes effect through per-package rules in the Renovate configuration, which mirrors
|
||||
each declared ceiling and carries any per-service cadence.
|
||||
|
||||
## Safety rails (so hands-off ≠ reckless)
|
||||
|
||||
- **Stable only**: never track pre-release/edge tags unless explicitly opted in per service.
|
||||
- **Merge gate first**: every update is a PR that must pass the gate before it can merge.
|
||||
- **Health-gated rollout**: a failing update is caught by health checks and rolled back (R-UPD-03),
|
||||
and raises an alert ([10](10-observability-and-alerting.md)).
|
||||
- **Backups precede risk**: state is on Longhorn with backups, so a bad update never risks data
|
||||
([04](04-storage-and-state.md)).
|
||||
- **Observability**: update events (image advanced, chart merged, rollback) are visible and, on
|
||||
failure, alert by email.
|
||||
- **Observability**: update events (update proposed, merged, rollback) are visible and, on failure,
|
||||
alert by email.
|
||||
- **Pin escape hatch**: an operator can pin a service via git if an upstream release misbehaves —
|
||||
the one sanctioned human touch, and it *stops* automation rather than performing an update.
|
||||
|
||||
@@ -68,9 +86,5 @@ updates; policies are reconciled from the (unchanged) ServiceDefinition.
|
||||
- A service advances to a newer stable version automatically, on schedule, with no human/AI edit.
|
||||
- A deliberately bad update is caught by health checks, auto-rolled-back, and alerts by email.
|
||||
- A pinned service does **not** advance.
|
||||
- Platform components (charts) stay current via auto-merged Renovate PRs.
|
||||
|
||||
## Open questions
|
||||
- Exact default in ADR-0009: write-back commits vs. digest-tracking mutable tags (trade readable
|
||||
history vs. zero-commit updates).
|
||||
- Registry rate-limit / credential handling for image scanning at cadence.
|
||||
- Components in the allowed classes stay current via auto-merged Renovate PRs; majors and
|
||||
blast-radius components arrive as machine-authored PRs awaiting review.
|
||||
@@ -18,15 +18,18 @@ app to build from scratch:
|
||||
Homepage widgets: Plex, Sonarr, Radarr, qBittorrent, PiHole, Gitea, etc.). This is the primary
|
||||
R-C2-01 surface.
|
||||
- **Grafana** — embedded platform + per-service dashboards (R-OBS-03) for health at a glance.
|
||||
- **Headlamp** (or a comparable Kubernetes web UI) — a controlled Kubernetes console for viewing
|
||||
workloads/VMs and performing scoped operational actions (restart, scale, inspect) — the R-C2-02
|
||||
"controlled operational actions where safe" surface.
|
||||
- **Headlamp** — a controlled Kubernetes console for viewing workloads/VMs and performing scoped
|
||||
operational actions (restart, scale, inspect) — the R-C2-02 "controlled operational actions where
|
||||
safe" surface. Each user acts under their own IdP identity and group-bound RBAC, never a shared
|
||||
console identity.
|
||||
- **Flux status** — GitOps reconciliation state (via Headlamp's Flux view, the Flux UI, or a
|
||||
Homepage widget) so the operator sees drift/errors.
|
||||
- **Hubble UI** (optional) — network flow visualization for isolation troubleshooting.
|
||||
- **Hubble UI** (optional) — network flow visualization for isolation troubleshooting; deployed
|
||||
with no route, so it is not reachable from outside the cluster.
|
||||
|
||||
Each sub-UI is itself an onboarded, TLS-protected, access-controlled service; the root aggregates and
|
||||
links them. ADR-worthy only if a custom portal is later preferred over Homepage.
|
||||
Each sub-UI is itself a platform-managed or onboarded, TLS-protected, access-controlled service; the
|
||||
root aggregates and links them. Status of this composition is tracked in
|
||||
[known gaps](16-known-gaps.md).
|
||||
|
||||
## Data the C2 surfaces (R-C2-02)
|
||||
|
||||
@@ -47,12 +50,16 @@ kubeconfig in a browser:
|
||||
|
||||
## Access control
|
||||
|
||||
- The C2 root and its sub-UIs are `internal` exposure by default (LAN-only) unless the operator
|
||||
explicitly makes a component `public`.
|
||||
- C2 sub-UIs are `internal` exposure by default (LAN/VPN-only) unless the operator explicitly makes
|
||||
a component `public`. The apex and Grafana are `public` as registered platform routes; Headlamp is
|
||||
LAN/VPN-only because it holds a path to the Kubernetes API on the user's behalf.
|
||||
- Prometheus, Alertmanager and Hubble UI have **no route** and are not exposed; operators reach
|
||||
their data through Grafana. Any future exposure of them MUST be gated.
|
||||
- Authentication is via **Keycloak SSO** ([15](15-identity-and-access.md), [ADR-0014](../decisions/adr-0014-idp-keycloak.md)):
|
||||
Headlamp/Grafana use **native OIDC**; UIs without OIDC (Homepage, Prometheus, Alertmanager, Hubble)
|
||||
sit behind the **oauth2-proxy forward-auth** at the Gateway. Access to the C2 and sensitive
|
||||
surfaces requires an authenticated user **in an authorized group** with **MFA** (R-IDP-06).
|
||||
Headlamp/Grafana use **native OIDC**; a UI without OIDC, such as Homepage, requires a per-app
|
||||
**oauth2-proxy forward-auth gate** ([ADR-0022](../decisions/adr-0022-forward-auth-per-app-oauth2-proxy-sidecars.md)).
|
||||
Access to the C2 and sensitive surfaces requires an authenticated user **in an authorized group**
|
||||
with **MFA** (R-IDP-06).
|
||||
- Trusted TLS on the root and all sub-UIs ([08](08-tls-and-certificates.md)).
|
||||
|
||||
## Acceptance gates
|
||||
@@ -62,8 +69,7 @@ kubeconfig in a browser:
|
||||
- Safe operational actions work and are access-controlled + audited; destructive actions confirm.
|
||||
|
||||
## Resolved
|
||||
- Auth mechanism — **Keycloak SSO** (native OIDC + oauth2-proxy forward-auth), [15](15-identity-and-access.md) /
|
||||
[ADR-0014](../decisions/adr-0014-idp-keycloak.md).
|
||||
|
||||
## Open questions
|
||||
- Custom portal vs. Homepage-as-root long term (ADR if we diverge from Homepage).
|
||||
- Auth mechanism — **Keycloak SSO** (native OIDC + per-app oauth2-proxy forward-auth gates),
|
||||
[15](15-identity-and-access.md) / [ADR-0014](../decisions/adr-0014-idp-keycloak.md) /
|
||||
[ADR-0022](../decisions/adr-0022-forward-auth-per-app-oauth2-proxy-sidecars.md).
|
||||
- Root UI — **Homepage** at the apex. A custom portal would need an ADR.
|
||||
+72
-40
@@ -11,6 +11,11 @@ implementation repo. It captures *intent*; the platform compiles it into concret
|
||||
record, ServiceMonitor + dashboards + alerts, and update policy). Operators declare **what**;
|
||||
platform provides the **how**. This keeps every requirement wired from a single reviewable object.
|
||||
|
||||
**Platform-managed services** are the deliberate exception. A component with its own operator,
|
||||
lifecycle or install flow is declared directly in git, under the same controls (default-deny, trusted
|
||||
TLS, SSO, GitOps delivery through the merge gate): keycloak, Crafty, the command-center UIs, and VMs
|
||||
with a custom install flow such as UniFi. `kind: vm` remains the catalog path for VMs.
|
||||
|
||||
### Fields (canonical schema — implementers render manifests from these)
|
||||
|
||||
```yaml
|
||||
@@ -27,6 +32,7 @@ storage:
|
||||
class: longhorn # replicated app/config state (R-STATE-02)
|
||||
- name: media # optional bulk tier (R-STATE-04)
|
||||
class: bulk
|
||||
export: <path> # bulk only: the data's location on the bulk NFS server
|
||||
dependencies: # explicit allow rules; everything else denied (R-NET-01/02)
|
||||
- to: <service>
|
||||
port: <n>
|
||||
@@ -37,7 +43,7 @@ ingressRoutes: # hostnames/paths to expose (rendered to HTTP
|
||||
auth: # SSO integration (R-IDP-02; see 15-identity-and-access.md)
|
||||
mode: oidc | forward-auth | native | none
|
||||
# oidc → app configured as a Keycloak OIDC client (does its own login)
|
||||
# forward-auth → protected by oauth2-proxy → Keycloak at the Gateway (for UIs w/o OIDC)
|
||||
# forward-auth → per-app oauth2-proxy sidecar → Keycloak (UIs w/o OIDC; ADR-0022)
|
||||
# native → app manages its own auth (e.g. Plex/Seerr via Plex) — documented exception
|
||||
# none → no auth (internal-only, discouraged)
|
||||
groups: [ <keycloak-group>, ... ] # groups allowed access (R-IDP-06)
|
||||
@@ -57,6 +63,14 @@ backup:
|
||||
> The schema is the **contract** between the operator surface and the implementation. Changing it is
|
||||
> an interface change — coordinate per [`agents/coordination-protocol.md`](../agents/coordination-protocol.md).
|
||||
|
||||
The fields above are the core of the contract. The formal schema,
|
||||
`schemas/servicedefinition.schema.json` in the
|
||||
[implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl), is
|
||||
canonical and adds workload-shaping fields for what the core cannot express: `env`, `envFrom`,
|
||||
`lifecycle`, `podSecurityContext`, `availability`, `hardware` (a host capability class such as a
|
||||
GPU, never a device path), `inClusterClients`, `auth.apiRoutes`, `observability.livenessWorkload`
|
||||
and `storage[].{mountPath,subPath,readOnly}`.
|
||||
|
||||
## Default posture for every service (unless overridden)
|
||||
- `kind: container`, `exposure: internal`, `egress: internet`.
|
||||
- **Default-deny**: no `dependencies` ⇒ can reach only DNS ([03](03-networking-isolation-and-egress.md)).
|
||||
@@ -65,28 +79,29 @@ backup:
|
||||
- Longhorn-backed config PVC; off-cluster backup; trusted TLS; auto-DNS; monitored; auto-updated on
|
||||
`stable`.
|
||||
|
||||
## The initial catalog (14 services)
|
||||
## The catalog (14 services)
|
||||
|
||||
Egress and dependencies below are **normative intent**; implementers render the exact policies.
|
||||
"Bulk" = large media tier ([04](04-storage-and-state.md)). All get trusted TLS + auto-DNS + monitoring
|
||||
+ auto-update unless noted.
|
||||
+ auto-update unless noted. **Status** gives each service's delivery path: `catalog` (rendered from
|
||||
its ServiceDefinition), `platform-managed` (declared directly in git), or `not onboarded`.
|
||||
|
||||
| # | Service | kind | exposure | egress | Storage | Key dependencies (explicit allows) | Notes |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 0 | **keycloak** | container | public | internet | db (PostgreSQL): longhorn (irreplaceable → backed up) | → its DB; ← all SSO clients + oauth2-proxy | **IdP/SSO** ([15](15-identity-and-access.md)); **foundational**, onboarded **early**; DR-critical + break-glass path; hardened public exposure |
|
||||
| 1 | **plex** | container | public | **internet** | config: longhorn (small, replicated); media: **bulk** | — (clients reach it; Seerr → Plex) | Largest service; media on bulk tier, metadata DB on Longhorn ([04](04-storage-and-state.md)); auth: **native** (Plex) |
|
||||
| 2 | **sonarr** | container | internal | **vpn** | config: longhorn; media/downloads: bulk | → qbittorrent, → jackett; ← seerr | VPN + kill-switch (R-NET-06) |
|
||||
| 3 | **radarr** | container | internal | **vpn** | config: longhorn; media/downloads: bulk | → qbittorrent, → jackett; ← seerr | VPN + kill-switch |
|
||||
| 4 | **jackett** | container | internal | **vpn** | config: longhorn | ← sonarr, ← radarr | VPN + kill-switch (indexer) |
|
||||
| 5 | **seerr** | container | public | internet | config: longhorn | → sonarr:8989, → radarr:7878, → plex:32400 | Request UI; talks to arr + Plex |
|
||||
| 6 | **pihole** | container | internal | internet | config: longhorn | (provides DNS to cluster/LAN) | Also a platform DNS dependency ([09](09-dns-and-ingress.md)) |
|
||||
| 7 | **nextcloud** | container | public | internet | data: longhorn (irreplaceable → backed up) | → its DB | File sync; DR-critical data |
|
||||
| 8 | **minecraft** | container | public | internet | world: longhorn (backed up) | — | Exposes game port via edge |
|
||||
| 9 | **unifi** | container | internal | internet | config: longhorn | — | UniFi network controller |
|
||||
| 10 | **semaphore** | container | internal | internet | config/db: longhorn | — | Ansible UI |
|
||||
| 11 | **kasm** | container | internal | internet | config: longhorn | — | Kasm Workspaces (may need more resources; container-first, VM if required) |
|
||||
| 12 | **gitea** | container | public | internet | repos/db: longhorn (irreplaceable → backed up) | — | Git hosting; **bootstrapping-critical** — self-hosts the platform's GitOps source + CI/CD; onboarded **early** (see below); DR-critical (off-cluster backup + mirror) |
|
||||
| 13 | **qbittorrent** | container | internal | **vpn** | config: longhorn; downloads: **bulk** | ← sonarr, ← radarr | VPN + kill-switch; download client |
|
||||
| # | Service | Status | kind | exposure | egress | Storage | Key dependencies (explicit allows) | Notes |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| 0 | **keycloak** | platform-managed | container | public | internet | db (PostgreSQL, ADR-0019): longhorn (irreplaceable → backed up) | → its DB; ← all SSO clients + oauth2-proxy | **IdP/SSO** ([15](15-identity-and-access.md)); **foundational**, onboarded **early**; DR-critical + break-glass path; hardened public exposure. Run by its operator; its catalog entry is declaration only and deploys nothing |
|
||||
| 1 | **plex** | catalog | container | internal | **internet** | config: longhorn (small, replicated); media: **bulk** | ← seerr, ← sonarr, ← radarr; clients reach it | Largest service; media on bulk tier, metadata DB on Longhorn ([04](04-storage-and-state.md)); auth: **native** (Plex); web UI LAN/VPN-only, remote streaming per Cross-cutting notes |
|
||||
| 2 | **sonarr** | catalog | container | internal | **vpn** | config: longhorn; media/downloads: bulk | → qbittorrent, → jackett, → plex:32400; ← seerr | VPN + kill-switch (R-NET-06) |
|
||||
| 3 | **radarr** | catalog | container | internal | **vpn** | config: longhorn; media/downloads: bulk | → qbittorrent, → jackett, → plex:32400; ← seerr | VPN + kill-switch |
|
||||
| 4 | **jackett** | catalog | container | internal | **vpn** | config: longhorn | ← sonarr, ← radarr | VPN + kill-switch (indexer) |
|
||||
| 5 | **seerr** | catalog | container | public | internet | config: longhorn | → sonarr:8989, → radarr:7878, → plex:32400 | Request UI; talks to arr + Plex |
|
||||
| 6 | **pihole** | catalog | container | internal | internet | config: longhorn | (provides DNS to cluster/LAN) | Also a platform DNS dependency ([09](09-dns-and-ingress.md)) |
|
||||
| 7 | **nextcloud** | catalog | container | public | internet | data: longhorn (irreplaceable → backed up); legacy archive: bulk, read-only | → its DB | File sync; DR-critical data |
|
||||
| 8 | **minecraft** (Crafty) | platform-managed | container | internal (game ports); public (web UI) | internet | world: longhorn (backed up) | — | Delivered as the Crafty controller, which runs the game servers; game ports on a LAN/VPN L4 address |
|
||||
| 9 | **unifi** | platform-managed | vm | internal | internet | VM disk: longhorn (replicated, live-migratable) | — | UniFi OS Server in a KubeVirt VM, declared directly in git (custom install flow); browser and device traffic on a LAN/VPN L4 address, no Gateway route |
|
||||
| 10 | **semaphore** | catalog | container | internal | internet | config/db: longhorn | → its DB | Ansible UI |
|
||||
| 11 | **kasm** | not onboarded | container | internal | internet | config: longhorn | — | Kasm Workspaces (may need more resources; container-first, VM if required) |
|
||||
| 12 | **gitea** | catalog | container | public | internet | repos/db: longhorn (irreplaceable → backed up) | — | Git hosting; **bootstrapping-critical** — self-hosts the platform's GitOps source + CI/CD; onboarded **early** (see below); DR-critical (off-cluster backup + mirror) |
|
||||
| 13 | **qbittorrent** | catalog | container | internal | **vpn** | config: longhorn; downloads: **bulk** | ← sonarr, ← radarr | VPN + kill-switch; download client |
|
||||
|
||||
**VPN-class set (R-NET-05/06):** sonarr, radarr, jackett, qbittorrent — all egress via VPN with an
|
||||
enforced kill-switch. Their inter-dependencies (arr → qbittorrent/jackett) are **east-west allows**
|
||||
@@ -95,19 +110,32 @@ within the deny-by-default fabric; being VPN-class governs *egress*, not intra-c
|
||||
**Dependency graph (explicit allows only; all else denied):**
|
||||
```
|
||||
seerr → sonarr, radarr, plex
|
||||
sonarr → qbittorrent, jackett
|
||||
radarr → qbittorrent, jackett
|
||||
sonarr → qbittorrent, jackett, plex
|
||||
radarr → qbittorrent, jackett, plex
|
||||
(everything → DNS only, by default)
|
||||
```
|
||||
|
||||
The arr → plex allows exist because the bulk tier is NFS, which delivers no change notifications:
|
||||
the arr stack tells Plex when new media lands.
|
||||
|
||||
**Delivery order (waves).** Catalog services reconcile in dependency-ordered waves computed from the
|
||||
`dependencies` graph: a service's wave is its longest dependency path, so a database always lands
|
||||
below its consumer. Each wave health-gates only itself and what depends on it, so a crashloop cannot
|
||||
stall an earlier wave. VMs and leaf platform components sit on their own non-gating tiers: a slow
|
||||
image import or a broken UI never holds up the app tier.
|
||||
|
||||
**SSO integration (`auth.mode`, R-IDP-02; see [15](15-identity-and-access.md)):**
|
||||
- **oidc** (native Keycloak client): gitea, nextcloud, semaphore, kasm, grafana*, headlamp*
|
||||
(*platform UIs, not in the numbered app list).
|
||||
- **forward-auth** (oauth2-proxy → Keycloak): pihole, sonarr, radarr, jackett, qbittorrent, unifi,
|
||||
and platform UIs (Homepage, Prometheus, Alertmanager, Hubble).
|
||||
- **native** (own auth, documented exceptions): plex, seerr (Plex accounts); minecraft
|
||||
(Mojang/Microsoft).
|
||||
- **forward-auth** (per-app oauth2-proxy sidecar → Keycloak, ADR-0022): pihole, sonarr, radarr,
|
||||
jackett, qbittorrent, Homepage*, and the Crafty web UI* (which adds its own local login; it has no
|
||||
OIDC).
|
||||
- **native** (own auth, documented exceptions): plex, seerr (Plex accounts); minecraft game ports
|
||||
(Mojang/Microsoft); unifi (its own login, on its own L4 address).
|
||||
- **none** (documented exception, R-IDP-02): the read-only runbooks at `/runbooks`, because gating
|
||||
them on Keycloak would lock the operator out of the Keycloak recovery procedures.
|
||||
- **keycloak** itself is the IdP — its own admin login is local + MFA (the root of trust).
|
||||
- Prometheus, Alertmanager and Hubble UI are not exposed; any future exposure MUST be gated.
|
||||
|
||||
**"…but not limited to":** the catalog is open-ended. New services follow the same ServiceDefinition
|
||||
and inherit all guarantees automatically — that is the point of the abstraction.
|
||||
@@ -118,7 +146,8 @@ Points the schema left ambiguous, now pinned so the JSON Schema, Kyverno policie
|
||||
- **`source` is exactly-one-of** — `image` for `kind: container`, `vmTemplate` for `kind: vm`
|
||||
(schema-enforced conditional).
|
||||
- **`storage[].class: longhorn ⇒ size` is required** — enforced by an **L0 Kyverno policy** (a
|
||||
replicated volume with no size is a defect). `bulk` size may be omitted (external mount).
|
||||
replicated volume with no size is a defect). `storage[].class: bulk ⇒ export` is required: a bulk
|
||||
volume is statically bound to data that already exists, and its `size` is advertised capacity only.
|
||||
- **`backup` is a reference** — `backup: {policy: <ref>}` where `policy` is either the literal
|
||||
`reacquirable` (bulk/re-acquirable data) or a named off-cluster backup schedule. Cadence/retention
|
||||
live in the referenced schedule object, **not** inline in the ServiceDefinition.
|
||||
@@ -128,32 +157,35 @@ Points the schema left ambiguous, now pinned so the JSON Schema, Kyverno policie
|
||||
least one Keycloak group (R-IDP-06); optional for `native`/`none`.
|
||||
- **`exposure: public ⇒ auth.mode ≠ none`** — a public service may not be unauthenticated
|
||||
(docs/15, R-IDP-02); **schema-enforced** invariant.
|
||||
- **Irreplaceable ⇒ off-cluster backup** — DR-critical services MUST NOT be `reacquirable`. This
|
||||
depends on per-service criticality the schema can't see, so it is enforced by an **L0 Kyverno
|
||||
policy** keyed on the service's criticality signal, not by the JSON Schema.
|
||||
- **Irreplaceable ⇒ off-cluster backup** — DR-critical services MUST NOT be `reacquirable`. The JSON
|
||||
Schema can't see criticality, so an **L0 Kyverno policy** applies the conservative rule: a service
|
||||
with any `longhorn` volume MUST NOT declare `reacquirable`.
|
||||
- **`update` declares policy; it does not run updates.** Renovate acts on it through machine-authored
|
||||
changes at the merge gate ([11](11-automated-updates.md), ADR-0023).
|
||||
|
||||
## Cross-cutting notes
|
||||
- **Plex media** is the R-STATE-04 case: keep the metadata DB on replicated Longhorn; keep the large
|
||||
library on the bulk tier; downloads stage on bulk and are handed off by the arr stack.
|
||||
- **DR-critical data** (Nextcloud, Gitea, Minecraft worlds, PiHole config, service DBs) MUST have
|
||||
off-cluster backups ([04](04-storage-and-state.md)); bulk media may be `reacquirable` per operator
|
||||
choice (OQ-2).
|
||||
- **Public exposure** (Seerr, Nextcloud, Gitea, …) is through the **Cloudflare Tunnel**
|
||||
([09](09-dns-and-ingress.md), ADR-0016). Everything else stays `internal`. **Plex** remote
|
||||
off-cluster backups ([04](04-storage-and-state.md)). The bulk tier is outside platform backup.
|
||||
- **Public exposure** is through the **Cloudflare Tunnel** ([09](09-dns-and-ingress.md), ADR-0016).
|
||||
A service is public by catalog `exposure: public` (Keycloak, Seerr, Nextcloud, Gitea) or as a
|
||||
registered platform route (the Homepage apex, `/runbooks`, Grafana, the Crafty web UI, and the
|
||||
oauth2-proxy host). Everything else stays `internal`. **Plex** remote
|
||||
streaming uses plex.tv + an edge forward of tcp/32400 (not the tunnel), and its web UI is LAN-only.
|
||||
**Minecraft (crafty-mc)** and other L4 services are LAN/VPN-only.
|
||||
- **Kasm** is resource-heavy and may be a candidate for `kind: vm` if container mode proves
|
||||
insufficient — the ServiceDefinition makes that a one-field change.
|
||||
- **Gitea is bootstrapping-critical.** The platform self-hosts its git and CI/CD, so Gitea is brought
|
||||
online **early** (WS-APPS step 2, right after PiHole). Once stable, the GitOps source and CI move
|
||||
from the local bootstrap source to Gitea + Gitea Actions ([05](05-gitops-and-delivery.md),
|
||||
[`../CONTRIBUTING.md`](../CONTRIBUTING.md)). Because a cluster loss would take its own git host with
|
||||
it, Gitea's repos/DB are backed up **off-cluster** and mirrored, so the platform can always be
|
||||
rebuilt from a source that outlives the cluster.
|
||||
online **early** (the first wave, with PiHole). Gitea + Gitea Actions are the GitOps source and CI
|
||||
([05](05-gitops-and-delivery.md), [`../CONTRIBUTING.md`](../CONTRIBUTING.md)). Because a cluster
|
||||
loss would take its own git host with it, Gitea's repos/DB are backed up **off-cluster** and
|
||||
mirrored, so the platform can always be rebuilt from a source that outlives the cluster.
|
||||
|
||||
## Acceptance gates
|
||||
- Each service is fully wired from its ServiceDefinition alone (compute, storage, policy, egress,
|
||||
TLS, DNS, monitoring, updates) — no manual steps beyond merge.
|
||||
- Each `catalog` service is fully wired from its ServiceDefinition alone (compute, storage, policy,
|
||||
egress, TLS, DNS, monitoring, updates) — no manual steps beyond merge. Each `platform-managed`
|
||||
service meets the same guarantees from its declaration in git.
|
||||
- Default-deny holds; only listed dependencies work.
|
||||
- VPN-class services egress via VPN with a proven kill-switch.
|
||||
- Stateful services survive destroy/rebuild; DR-critical data restores from off-cluster backup.
|
||||
@@ -25,11 +25,13 @@ Related: [ADR-0010](../decisions/adr-0010-conformance-testing.md).
|
||||
fail-closed / no-data-loss — not by happy-path checks.
|
||||
6. **Specs are the fixed target (R-TEST-06).** Tests encode the spec's intent. Agents never weaken a
|
||||
test or a requirement to get green; a genuine inability to meet the spec is an escalation to the
|
||||
human, not a spec/test edit.
|
||||
human, not a spec/test edit. The implementation pins the spec commit it is built against, and each
|
||||
spec advance is adopted explicitly ([`../CONTRIBUTING.md`](../CONTRIBUTING.md)).
|
||||
|
||||
## Reproducibility contract (what "reproducible" concretely requires)
|
||||
|
||||
- **Single command** runs the suite (or a tagged subset) locally and in CI.
|
||||
- **Single command** runs the suite (or a tagged subset) locally and in CI, in the same pinned
|
||||
toolbox image in both places.
|
||||
- **Hermetic fixtures**: each test provisions its own namespace/workloads/probes and tears them down;
|
||||
it never depends on pre-existing mutable state or on another test's side effects.
|
||||
- **Deterministic**: fixed inputs, pinned tool/image versions, bounded retries with explicit
|
||||
@@ -49,8 +51,8 @@ Related: [ADR-0010](../decisions/adr-0010-conformance-testing.md).
|
||||
|---|---|---|---|
|
||||
| **L0 Static/policy** | Pre-apply, in CI | Manifests/ServiceDefinitions obey invariants | Every namespace has a default-deny baseline; no plaintext secrets; schema-valid ServiceDefinition |
|
||||
| **L1 Rendering** | CI, no cluster | ServiceDefinition → rendered resources match golden expectations | `egress: vpn` renders the kill-switched egress path |
|
||||
| **L2 Behavioral/integration** | Against dev/test cluster | Real runtime behavior | Default-deny probe, dependency allow/deny, egress-IP, TLS validity, DNS, PVC rebuild |
|
||||
| **L3 Fault-injection** | Against dev/test cluster | Fail-closed / no-data-loss under failure | VPN tunnel forced down ⇒ no leak; node killed ⇒ data survives; bad update ⇒ auto-rollback; DR restore |
|
||||
| **L2 Behavioral/integration** | Against the live cluster | Real runtime behavior | Default-deny probe, dependency allow/deny, egress-IP, TLS validity, DNS, PVC rebuild |
|
||||
| **L3 Fault-injection** | Against the live cluster | Fail-closed / no-data-loss under failure | VPN tunnel forced down ⇒ no leak; node killed ⇒ data survives; bad update ⇒ auto-rollback; DR restore |
|
||||
| **L4 Conformance suite** | CI + on-demand | Aggregate, requirement-tagged gate | The union of the above, reported per `R-*` |
|
||||
|
||||
## Behavior test catalog (normative — implementers author these as automated tests)
|
||||
@@ -82,13 +84,15 @@ Each maps to requirement IDs; phrased as Given/When/Then intent.
|
||||
- **DNS/exposure (R-DNS-*):** *Given* a service, *When* resolved, *Then* the record exists and
|
||||
internal clients hit the internal Gateway; `internal` is unreachable publicly, `public` is
|
||||
reachable + TLS-valid.
|
||||
- **Public ingress path (R-DNS-04/05, R-SEC-04):** *Given* the tunnel ingress config, *When* it is
|
||||
rendered, *Then* every route is a `public` service, the last rule is `http_status:404`, and no rule
|
||||
targets the Kubernetes API (L0). *When* a non-`public` hostname is requested at the tunnel, *Then*
|
||||
it gets 404. *When* a `public` name is resolved via a public resolver from the LAN, *Then* it is
|
||||
reachable + TLS-valid (L2).
|
||||
- **Public ingress path (R-DNS-04/05, R-SEC-04):** *Given* the tunnel ingress config (a deny-list of
|
||||
LAN/VPN-only hostnames ahead of a zone wildcard to the shared Gateway), *When* it is rendered,
|
||||
*Then* every rule targets only the shared Gateway or a 404, every LAN/VPN-only hostname 404s ahead
|
||||
of the wildcard, the last rule is `http_status:404`, and no rule targets the Kubernetes API (L0).
|
||||
*When* a LAN/VPN-only hostname is requested at the tunnel, *Then* it gets 404. *When* a `public`
|
||||
name is resolved via a public resolver from the LAN, *Then* it is reachable + TLS-valid (L2).
|
||||
- **Hands-off update (R-UPD-01/02, L3):** *Given* an out-of-date service under a stable policy,
|
||||
*When* the scan interval elapses, *Then* it advances with no human/AI edit; *Given* a deliberately
|
||||
*When* the update schedule elapses, *Then* a machine-authored change advances it through the merge
|
||||
gate with no human/AI edit; *Given* a deliberately
|
||||
bad update, *Then* health-gating auto-rolls-back and an email alert fires; *Given* a pinned
|
||||
service, *Then* it does not advance.
|
||||
- **VM lifecycle (R-PLAT-03, L2/L3):** *Given* a `kind: vm` ServiceDefinition, *When* create/start/
|
||||
@@ -108,10 +112,25 @@ Each maps to requirement IDs; phrased as Given/When/Then intent.
|
||||
touches MUST pass before merge.
|
||||
- **Per-phase (exit gate):** the full conformance suite for the phase's requirements MUST pass
|
||||
before the phase's exit criteria are considered met ([`../roadmap/phases.md`](../roadmap/phases.md)).
|
||||
- **Regression:** the growing suite runs continuously against dev/test; a previously-green
|
||||
- **Regression:** the growing suite runs continuously against the live cluster; a previously-green
|
||||
requirement going red blocks further delivery until fixed.
|
||||
- **Coverage:** a requirement with no passing test is **not** satisfied, regardless of implementation
|
||||
state.
|
||||
- **Non-answers are never green:** a stage that did not run is never reported as passing
|
||||
([05](05-gitops-and-delivery.md); see [known gaps](16-known-gaps.md)).
|
||||
|
||||
## Coverage, status and proof (R-TEST-04)
|
||||
|
||||
Three different questions, answered separately so that none is mistaken for another:
|
||||
|
||||
- **Coverage** — does a requirement-tagged test exist for each `R-*`?
|
||||
- **Status** — derived from the repository alone: is the requirement implemented, is that
|
||||
implementation actually reconciled by Flux, and is its test executable or a scaffold?
|
||||
- **Proof** — did a run actually pass? Proof is a committed, machine-readable run ledger. A test that
|
||||
skips itself is never proof, and a proof goes stale when the implementation or test moves on.
|
||||
|
||||
`./conformance/run.sh --status` in the implementation repo produces the live view.
|
||||
[`16-known-gaps.md`](16-known-gaps.md) is a dated snapshot.
|
||||
|
||||
## Tooling (pinned — [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md))
|
||||
|
||||
@@ -138,7 +157,11 @@ from the **Kyverno ecosystem**:
|
||||
timeouts, per-test namespace isolation, and automatic cleanup satisfy the reproducibility contract.
|
||||
Fault-injection (force the tunnel down, cordon/kill a node, inject a bad image, trigger a restore)
|
||||
is expressed as Chainsaw `script` steps, asserting fail-closed / no-data-loss via probe fixtures,
|
||||
with `finally` restoring state.
|
||||
with `finally` restoring state. Cluster-free L0 and L1 checks are declared as Chainsaw tests for
|
||||
tagging and execute through the harness's offline runner, so they need no cluster.
|
||||
- **Repository invariants** — L0 also carries whole-repo checks a per-resource policy cannot express
|
||||
(secret hygiene across every tracked file, least-privilege RBAC, the spec pin), which run on every
|
||||
gate regardless of the requirements a change touches.
|
||||
|
||||
Probes are ephemeral job pods; egress-IP checks use an external echo; tests carry their `R-*` IDs
|
||||
(labels/annotations/naming) so the coverage report is generated automatically; both tools' output
|
||||
@@ -148,16 +171,18 @@ pinned for determinism.
|
||||
### Where each layer runs
|
||||
- **L0 (Kyverno CLI) and L1 (rendering assertions)** run **in CI without a cluster** — pure manifest/
|
||||
ServiceDefinition validation; fast merge-gate feedback.
|
||||
- **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job** on the dev/test cluster.
|
||||
- **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job** on the live cluster.
|
||||
CI **triggers** the Job (and waits for it); the Job — not CI — holds cluster access, so CI never
|
||||
needs a broad kubeconfig. This keeps credentials in-cluster and least-privileged (R-SEC-02).
|
||||
needs a broad kubeconfig. CI holds only a narrow trigger credential, whose reach is bounded by a
|
||||
fail-closed admission policy on the Job's shape. This keeps credentials in-cluster and
|
||||
least-privileged (R-SEC-02).
|
||||
- The Job runs under a **dedicated, scoped test ServiceAccount** whose RBAC grants exactly what the
|
||||
behavior/fault tests need (create/delete test namespaces + fixtures + probes, apply/patch
|
||||
NetworkPolicies, scale the VPN sidecar/gateway, cordon/drain a node, inject a bad image tag,
|
||||
trigger a Velero restore). This is a **privileged test identity** — it MUST be scoped, named, and
|
||||
documented, MUST NOT be reused by workloads, and its own reach is itself an audit item.
|
||||
- The Job writes **JUnit + coverage artifacts** to a known location (log stream and/or a results
|
||||
PVC/object) that CI collects to enforce the merge / phase-exit gates.
|
||||
- The Job emits **JUnit + coverage artifacts** over its log stream, in an integrity-checked
|
||||
envelope that CI collects to enforce the merge / phase-exit gates.
|
||||
- Fixtures are hermetic and self-cleaning (`finally` steps); a leaked fixture fails the run.
|
||||
|
||||
## Acceptance gates (for the testing capability itself)
|
||||
@@ -170,10 +195,10 @@ pinned for determinism.
|
||||
## Open questions
|
||||
- Tool mix is **resolved**: Kyverno (L0) + Chainsaw (L1/L2/L3), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md).
|
||||
- Harness location is **resolved**: L2/L3 run as an **in-cluster Chainsaw Job** triggered by CI; CI
|
||||
holds no cluster credentials.
|
||||
holds no broad cluster credential, only the admission-bounded trigger.
|
||||
- Kyverno-as-admission-enforcer is **resolved**: yes — the L0 policies also enforce at admission
|
||||
in-cluster (defense in depth), in addition to the CI gate.
|
||||
- Delegated to WS-TEST at implementation time (not human decisions): the **exact RBAC scope** of the
|
||||
test ServiceAccount (author it **least-privilege** for precisely the fault-injection actions it
|
||||
performs, [07](07-security-and-secrets.md)) and the **results-artifact mechanism** (log stream vs
|
||||
results PVC/object store). Both are implementer's discretion within the stated constraints.
|
||||
- The test ServiceAccount's RBAC is **resolved**: scoped **least-privilege** to precisely the
|
||||
fault-injection actions it performs ([07](07-security-and-secrets.md)), with its grants justified
|
||||
per test in the implementation repo.
|
||||
- The results-artifact mechanism is **resolved**: the log stream, as above.
|
||||
@@ -20,22 +20,33 @@ self-hostable OpenID Connect (OIDC) + SAML provider with users, groups, roles, c
|
||||
Every service authenticates against Keycloak, via whichever pattern it supports:
|
||||
|
||||
1. **Native OIDC** — the app is configured as a Keycloak OIDC **client** and does its own login
|
||||
redirect. Preferred where available. Initial set: **Gitea, Nextcloud, Grafana, Kasm, Semaphore,
|
||||
Headlamp**.
|
||||
2. **Forward-auth proxy** — for apps with **no** usable OIDC, an identity-aware proxy
|
||||
(**oauth2-proxy**) sits in front at the Gateway, authenticates the user against Keycloak, and only
|
||||
then forwards the request. Used for: **PiHole admin, the *arr web UIs (Sonarr/Radarr/Jackett),
|
||||
qBittorrent UI, Homepage, Prometheus, Alertmanager, Hubble UI**, and any other bare UI.
|
||||
redirect. Preferred where available. Initial set: **Gitea, Nextcloud, Grafana, Semaphore,
|
||||
Headlamp** (Kasm, not onboarded, joins this set when it is).
|
||||
2. **Forward-auth gate** — for apps with **no** usable OIDC, a per-app identity-aware proxy
|
||||
(**oauth2-proxy**) runs as a sidecar in the app's pod, authenticates the user against Keycloak,
|
||||
and only then forwards the request ([ADR-0022](../decisions/adr-0022-forward-auth-per-app-oauth2-proxy-sidecars.md)).
|
||||
Used for: **PiHole admin, the *arr web UIs (Sonarr/Radarr/Jackett), qBittorrent UI, the Crafty
|
||||
web UI**, and any other bare UI; **Homepage** requires the same (see
|
||||
[known gaps](16-known-gaps.md)). Prometheus, Alertmanager and Hubble UI are not exposed; any future
|
||||
exposure MUST be gated.
|
||||
|
||||
> This resolves the C2 auth open question ([12](12-command-center-ui.md)): it is **both** — native
|
||||
> OIDC where supported, forward-auth everywhere else, all backed by Keycloak.
|
||||
|
||||
Each gate is an independent instance with its own IdP client and credentials, and enforces the
|
||||
service's `auth.groups`. Session cookies are per-gate: a unique name, scoped to the app's own host,
|
||||
because instances that share a cookie evict one another's sessions. A service MAY declare its API
|
||||
paths so that unauthenticated programmatic calls are refused rather than redirected to a login page.
|
||||
|
||||
### Exceptions (documented, not oversights)
|
||||
- **Plex** and **Seerr** use **Plex's own account system** (Plex OAuth) — they do not federate to
|
||||
Keycloak.
|
||||
- **Minecraft** authenticates players via Mojang/Microsoft — N/A.
|
||||
- **UniFi** keeps its native account model (limited SSO support); protect its UI via forward-auth if
|
||||
desired.
|
||||
- **Minecraft** players authenticate via Mojang/Microsoft — N/A. The **Crafty web UI** that manages
|
||||
it is behind a forward-auth gate and then presents **its own local login** (no OIDC); the second
|
||||
login is intended.
|
||||
- **UniFi** keeps its native account model and login on its own L4 VIP (limited SSO support).
|
||||
- **Runbooks** served read-only under `/runbooks` carry no login.
|
||||
- **Keycloak** itself is `native`: it is the root of trust.
|
||||
|
||||
## Authorization model (R-IDP-06)
|
||||
|
||||
@@ -45,47 +56,51 @@ Every service authenticates against Keycloak, via whichever pattern it supports:
|
||||
- **Least privilege**: the C2 console, Headlamp, Alertmanager, and other sensitive UIs require an
|
||||
authenticated user **in an authorized group** — not merely "logged in."
|
||||
- **MFA**: TOTP/WebAuthn available for all users and **required for privileged groups**
|
||||
(`platform-admins`).
|
||||
(`platform-admins`). Passkeys (WebAuthn passwordless) are an optional additive login path,
|
||||
bound into the login flow by an explicit switch in git.
|
||||
- **Kubernetes API**: humans authenticate with Keycloak OIDC, and cluster roles are bound to Keycloak
|
||||
groups, so `kubectl` and the Kubernetes console share this model.
|
||||
|
||||
## Identity config as code (R-IDP-03)
|
||||
|
||||
Realms, clients, groups, and roles are **declarative and GitOps-managed** — not clicked into the
|
||||
admin console and forgotten:
|
||||
- The Keycloak instance is deployed via the **Keycloak Operator** (or Helm), and **realm/client/group
|
||||
configuration is reconciled declaratively** (e.g. `keycloak-config-cli` or realm-import CRs). Exact
|
||||
tool is an implementation detail; the requirement is that identity config lives in git and
|
||||
reconciles like everything else.
|
||||
- The Keycloak instance is deployed via the **Keycloak Operator**, and a **realm-import CR** seeds
|
||||
realm/client/group configuration from git. A realm import is one-shot and fails open, so
|
||||
**reconcilers** continuously converge what git declares after the first import: client secrets,
|
||||
client URLs, the WebAuthn policy and the login flow.
|
||||
- **OIDC client secrets and the Keycloak admin credential are SOPS-encrypted** ([07](07-security-and-secrets.md)).
|
||||
|
||||
## State, exposure & resilience
|
||||
|
||||
- **Datastore**: Keycloak runs on **PostgreSQL**; the DB lives on **3-replica Longhorn** and is
|
||||
**DR-critical → backed up off-cluster** ([04](04-storage-and-state.md)). Keycloak is stateless
|
||||
itself and rebuilds from the DB + git-managed realm config.
|
||||
- **Datastore**: Keycloak runs on **PostgreSQL via CloudNativePG** (ADR-0019); the DB lives on
|
||||
**3-replica Longhorn** and is **DR-critical → backed up off-cluster** ([04](04-storage-and-state.md)).
|
||||
Keycloak is stateless itself and rebuilds from the DB + git-managed realm config.
|
||||
- **Exposure**: `keycloak.home.keithjackson.dev` is **`public`** so it can serve login/redirect flows
|
||||
for the internet-exposed services (Nextcloud, Gitea) — but it is **security-sensitive** and
|
||||
hardened: brute-force detection on, strong admin auth + MFA, trusted TLS ([08](08-tls-and-certificates.md)),
|
||||
minimal exposed endpoints. If the operator prefers internal-only, public services fall back to
|
||||
their native auth.
|
||||
- **HA**: a single replica with DB on Longhorn + backups is acceptable for the home scale; note
|
||||
clustered Keycloak as a future hardening. **Availability caveat:** SSO becomes a dependency for
|
||||
logging into federated services — document a break-glass path (e.g. a local admin on Gitea) so an
|
||||
IdP outage doesn't lock the operator out of recovery tools.
|
||||
minimal exposed endpoints.
|
||||
- **HA**: Keycloak runs multiple replicas against a database with one instance per node and a hot
|
||||
standby to promote, because a single node must not be able to take SSO down.
|
||||
**Availability caveat:** SSO is a dependency for logging into federated services and the Kubernetes
|
||||
API, so a **break-glass** path exists: local admin logins on Gitea, Nextcloud and Grafana, the
|
||||
Keycloak bootstrap admin, and a certificate kubeconfig for the cluster.
|
||||
|
||||
## Interaction with default-deny & the gateway
|
||||
|
||||
- Keycloak and oauth2-proxy are ordinary workloads under **default-deny** ([03](03-networking-isolation-and-egress.md));
|
||||
the specific flows they need (app → Keycloak OIDC, proxy → Keycloak, Keycloak → its DB) are
|
||||
the specific flows they need (app → Keycloak OIDC, gate → Keycloak, Keycloak → its DB) are
|
||||
explicit allow rules.
|
||||
- Forward-auth is enforced at the **Gateway** ([09](09-dns-and-ingress.md)) so protection happens
|
||||
before traffic reaches the app.
|
||||
- Forward-auth is enforced **in the data path**: the Gateway ([09](09-dns-and-ingress.md)) routes
|
||||
only to the gate's port and network policy closes the app's own port to it, so protection happens
|
||||
before traffic reaches the app and a failed gate fails closed.
|
||||
|
||||
## Bootstrapping & order
|
||||
|
||||
- Keycloak is **foundational** and onboarded **early** (with PiHole/Gitea, [13](13-service-catalog.md)) so
|
||||
later services can integrate SSO as they come up.
|
||||
- No hard circular dependency at bring-up: Gitea can start with a local admin and add OIDC once
|
||||
Keycloak is live; the C2 and sensitive UIs gain SSO as soon as the proxy + realm exist.
|
||||
Keycloak is live; the C2 and sensitive UIs gain SSO as soon as their gate + the realm exist.
|
||||
|
||||
## Acceptance gates
|
||||
- A user logs in once at Keycloak and reaches all OIDC-integrated services without re-entering
|
||||
@@ -97,8 +112,6 @@ admin console and forgotten:
|
||||
- Privileged access (C2/Headlamp) requires MFA and group membership; a non-member is denied.
|
||||
- Client secrets and admin credentials exist only SOPS-encrypted.
|
||||
|
||||
## Open questions
|
||||
- Final declarative-config tool (`keycloak-config-cli` vs operator realm-import CRs) — implementation
|
||||
detail for WS-SEC.
|
||||
- Confirm `public` exposure for Keycloak (recommended) vs internal-only with native-auth fallback for
|
||||
public services.
|
||||
## Resolved
|
||||
- Declarative-config tool — operator **realm-import CRs**, plus the reconcilers above.
|
||||
- Keycloak exposure — **`public`**, hardened.
|
||||
@@ -0,0 +1,108 @@
|
||||
# 16 — Known Gaps
|
||||
|
||||
**Snapshot of implementation `main` at `50c8719`, taken 2026-11-08.** This register records where the
|
||||
implementation does not yet meet a requirement this specification still holds. It exists so that bringing
|
||||
the specification in line with the implementation did not silently delete the intent the implementation
|
||||
has yet to reach.
|
||||
|
||||
- A gap is **not** an accepted change. Where the implementation deliberately changed direction, the
|
||||
requirement or an ADR was amended instead, and the item does not appear here.
|
||||
- This file is a dated snapshot, not a status surface. Every commit here costs the implementation a
|
||||
`ci/SPEC_PIN` bump, so it is refreshed in batches. **Live status comes from `conformance/run.sh --status`**
|
||||
in the [implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl).
|
||||
- Entries were derived by reading both repositories. **Nothing here was measured against the running
|
||||
platform.**
|
||||
|
||||
## Operator decisions needed
|
||||
|
||||
These cannot be settled by reading either repository.
|
||||
|
||||
| Topic | Question | Evidence (implementation repo) |
|
||||
|---|---|---|
|
||||
| Bulk-tier photos (R-STATE-03/04) | The bulk tier has no platform backup, and its media dataset includes a `photos/` tree. Is that tree irreplaceable, and if so, what protects it? | `docs/BULK-NFS-STORAGE.md` §2; `platform/velero/volume-policies.yaml` |
|
||||
| Homepage exposure (R-IDP-02) | The apex is routed publicly with no enforced authentication. Enforce the gate (the requirement stands), or rule on an exception? | `platform/homepage/httproute.yaml`; `platform/homepage/README.md` |
|
||||
| ACME revocation (R-TLS-02) | cert-manager has no revocation mechanism. Amend the requirement, or keep it as an open gap? | `conformance/tests/l0-static/tls-lifecycle-automated/chainsaw-test.yaml:36` |
|
||||
| L3 in CI (R-TEST-03, R-TEST-05) | Production is the only cluster. Is automated fault injection against it acceptable once CI is wired? | `docs/plans/l2-l3-conformance-ci-wiring.md` |
|
||||
| Kasm (docs/13) | Never onboarded, and no decision retires it. Onboard it, or retire it from the catalog? | `catalog/README.md:20` |
|
||||
|
||||
## Environment and provisioning
|
||||
|
||||
| Requirement or area | Requirement and shortfall | Evidence (implementation repo) |
|
||||
|---|---|---|
|
||||
| R-ENV-01, R-STATE-01 (Phase 1 exit) | A node must be destroyable and rebuildable from code with no data loss; a node was reimaged and rejoined by hand, but no automated destroy/rebuild proof exists on bare metal and the checklist item is unticked. | `infra/PROD-PROVISIONING.md` sections 0c and 9 |
|
||||
| docs/02 etcd guardrail (accepted risk) | Shared-device etcd/Longhorn contention is accepted only if etcd write latency stays within budget during a Longhorn rebuild; the measurement failed on node 1 on 2026-09-07 and no dedicated etcd device was added, so the exit condition is unmet. | `infra/PROD-PROVISIONING.md` section 2 ("The accepted risk, and its exit condition"), section 0c |
|
||||
| R-PLAT-03, Phase 5 exit | The VM tier must run a working application VM; at the last recorded check (2026-09-14) the UniFi guest had no IP address and its controller ports refused connections. | `apps/vms/unifi/README.md` |
|
||||
|
||||
## Storage, state and VMs
|
||||
|
||||
| Requirement or area | Requirement and shortfall | Evidence (implementation repo) |
|
||||
|---|---|---|
|
||||
| Off-cluster Gitea mirror (R-STATE-03; spec CONTRIBUTING, docs/05 "Circularity & DR") | An off-cluster mirror of the platform repo must outlive the cluster; Gitea has no mirrors and no mirror job exists, while prod's Flux source is in-cluster Gitea. | `catalog/gitea.yaml:52`; `decisions/2026-11-07-gitea-28.md` |
|
||||
| R-STATE-03 (Gitea) | Gitea is DR-critical and its repos and SQLite are backed up by file-level kopia with no quiesce hook, and no Gitea-specific restore has been proven (the DR test restores its own scratch workload). | `apps/wave-0/gitea.yaml`; `conformance/tests/l3-fault-injection/dr-restore/chainsaw-test.yaml` |
|
||||
| R-STATE-03 (UniFi) | Irreplaceable VM state must be recoverable off-cluster; the UniFi disk is excluded on the basis of a configuration export that nothing in the repo declares. | `platform/velero/helmrelease.yaml:182-188`; `apps/vms/unifi/` |
|
||||
| R-STATE-01 | The per-service rebuild test is an acceptance gate; `data-survival` and `vm-data-survival` exist but have no ledger row, and per-service rebuilds are not recorded. | `conformance/runs/ledger.jsonl`; `conformance/tests/l3-fault-injection/data-survival/` |
|
||||
| R-STATE-01/03 | End-to-end total-cluster recovery should be drilled at least once; bare-metal rebuild then restore has not been exercised. | `conformance/tests/l3-fault-injection/dr-restore/chainsaw-test.yaml` (KNOWN, DELIBERATE GAPS) |
|
||||
| R-STATE-01 (Loki) | Destroying or scaling a workload must leave its PVC; Loki's StatefulSet retention policy deleted its PVC on scale-down (log data is reconstructable). | `decisions/2026-10-17-graceful-cluster-shutdown.md` |
|
||||
| R-PLAT-03, R-PLAT-04 | VM create/start/stop/destroy must be proven; `vm-lifecycle` is logged green only on the decommissioned dev cluster and has no ledger row. | `DECISIONS-LOG.md:896`; `conformance/runs/ledger.jsonl` |
|
||||
| R-NET-06 (VMs) | An `egress: vpn` VM must egress through the VPN with a working kill-switch; no VM VPN conformance test exists and `vm-demo` calls it a follow-up. | `catalog/vm-demo.yaml`; `conformance/tests/l2-behavioral/vm-lifecycle/` |
|
||||
| R-UPD-* (UniFi guest) | Guest OS packages must auto-update on schedule with per-VM exceptions recorded; UniFi's cloud-init sets `package_upgrade: false` and installs no unattended-upgrades. | `apps/vms/unifi/30-virtualmachine.yaml:206-207` |
|
||||
| R-OBS-* (VM guests) | A VM must expose node-exporter-class metrics from inside the guest; neither the renderer nor UniFi provides any, only KubeVirt/virt platform metrics. | `catalog/render.py`; `platform/kubevirt/servicemonitor.yaml` |
|
||||
| R-STATE-03 (VM disks) | VM disks are meant to share the platform backup story; the backup path is Velero file-system backup, which upstream documents as not supporting block-mode volumes, and the default VM class is block-mode. Unmeasured here: whether any block-mode VM disk can be captured by this path. | `platform/velero/helmrelease.yaml:175`; `platform/longhorn/storageclasses.yaml` (`longhorn-vm`) |
|
||||
|
||||
## Networking, DNS and TLS
|
||||
|
||||
| Requirement or area | Requirement and shortfall | Evidence (implementation repo) |
|
||||
|---|---|---|
|
||||
| VPN port forwarding (docs/03, ADR-0015) | The spec requires a host-side PIA port-forward helper for qBittorrent and an alert on PF acquisition failure; neither exists. | `infra/ansible/roles/vpn_egress_gateway/README.md:170`, `platform/vpn-egress/README.md` |
|
||||
| R-NET-01 | Every namespace must carry a default-deny baseline; Kyverno generates it only for `app-namespace=true`, and no policy file exists for cert-manager, cnpg, external-dns, longhorn, velero, backup, image-automation, external-snapshotter or vpn-egress. | `platform/kyverno/policies/require-default-deny-baseline.yaml`, `decisions/2026-10-03-public-dns-wan-target.md` |
|
||||
| R-SEC-03, R-OBS-04 | docs/03 requires alerting on unexpected denied flows; `HubbleDeniedFlowSpike` cannot fire because no Cilium or Hubble metrics are scraped. | `platform/monitoring/alerts.yaml:1047-1050,1162-1173` |
|
||||
| R-TLS-02 | Certificates must be automatically revoked; renewal is met, but cert-manager performs no ACME revocation and no layer proves it. | `conformance/tests/l0-static/tls-lifecycle-automated/chainsaw-test.yaml:36`, `DECISIONS-LOG.md:3350,3786` |
|
||||
| Client IP (ADR-0016) | IP-based logging and policy must read `CF-Connecting-IP` / `X-Forwarded-For` and trust them only from cloudflared; no trusted-proxy or client-IP configuration exists in `catalog/`, `platform/` or `apps/`. | `platform/cloudflared/config.yaml` (no client-IP handling downstream of it) |
|
||||
| R-DNS-04 | The edge must not forward 80/443; closing them is a manual router step, last recorded as not done, so the state is unverified. | `decisions/2026-11-08-dns-via-cloudflare-tunnel.md`, `runbooks/cloudflare-tunnel.md:47-55` |
|
||||
| R-DNS-04, R-DNS-05, R-SEC-04 | docs/09 and docs/14 require behavioural proof at the tunnel and from the LAN through a public resolver; only L0 static checks exist, and no L2 test touches the tunnel. | `conformance/tests/l0-static/cloudflared-ingress-rules`, `conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml` |
|
||||
|
||||
## Security, identity, observability and the command center
|
||||
|
||||
| Requirement or area | Requirement and shortfall | Evidence (implementation repo) |
|
||||
|---|---|---|
|
||||
| R-IDP-02, R-IDP-06 | No management UI may be exposed unauthenticated and the C2 requires an authorized group, but Homepage at the apex is routed straight to its Service with no gate and its `forward-auth` annotation enforces nothing. | `platform/homepage/httproute.yaml`, `platform/homepage/README.md` |
|
||||
| R-SEC-03, R-C2-02 | Operational actions must be audited, but the k3s API server has only OIDC arguments and no audit log or audit policy. | `infra/ansible/roles/k3s_server/tasks/main.yml` |
|
||||
| R-SEC-01 | The acceptance gate requires no plaintext secret anywhere in git history (scanned), but the check inspects the tracked tree only and no history scanner runs in CI. | `conformance/tests/l0-static/no-plaintext-secrets/check.sh`, `ci/` |
|
||||
| R-SEC-01 | Rotation is meant to trigger a workload restart through reconcile (e.g. a checksum annotation), but no generated mechanism exists; the only annotations are hand-stamped on Homepage and Grafana. | `platform/homepage/deployment.yaml`, `platform/monitoring/kube-prometheus-stack.yaml`, `catalog/render.py` |
|
||||
| R-SEC-01 | VPN keys, DNS tokens and SMTP credentials should have a documented rotation procedure, but the only trace is a one-line note on the Cloudflare token and none exists for VPN keys or SMTP. | `runbooks/cert-issuance.md`, `runbooks/` |
|
||||
| R-OBS-01 | Alloy is specified to ship node, pod and VM logs, but it collects pod logs only, with no node or journal source. | `platform/monitoring/alloy.yaml` |
|
||||
| R-OBS-04 | NetworkPolicy reconcile errors are a specified platform alert, but no such rule exists. | `platform/monitoring/alerts.yaml` |
|
||||
| R-OBS-04 | Restore-test failures are a specified platform alert, but only backup and schedule alerts exist. | `platform/monitoring/alerts.yaml` |
|
||||
| R-OBS-04 | Keycloak login-endpoint failures are a specified alert, but the repo has no endpoint probe, only workload and database health rules. | `platform/monitoring/alerts.yaml` |
|
||||
| R-OBS-01 | Every forward-auth gate is meant to be observed, but Crafty's gate has no metrics listener, so `OAuth2ProxyGateUnobserved` fires for it by design. | `platform/crafty/deployment.yaml`, `platform/oauth2-proxy/README.md` |
|
||||
| R-OBS-03 | Per-service dashboards are specified as onboarding output, but the renderer emits no dashboards; the in-repo dashboards are node pressure, KubeVirt and Keycloak only. | `catalog/render.py`, `platform/monitoring/node-pressure-dashboard.yaml`, `platform/kubevirt/dashboard.yaml`, `platform/keycloak/grafana-dashboard.yaml` |
|
||||
| R-OBS-01, R-ENV-02 | Log retention is specified as an overlay setting, but Loki's retention is hardcoded with no overlay knob. | `platform/monitoring/loki.yaml`, `platform/monitoring/README.md` |
|
||||
| R-OBS-04 | Monitoring storage is specified on Longhorn, but Alertmanager and Grafana run without persistent volumes, so silences and notification state are lost on restart. | `platform/monitoring/kube-prometheus-stack.yaml` |
|
||||
| R-OBS-02, R-OBS-04 | Acceptance requires platform alerts to fire and email in injected-fault tests, but the L3 alerting test is a scaffold and L2/L3 do not run in CI. | `conformance/tests/l3-fault-injection/alerting/chainsaw-test.yaml`, `CLAUDE.md` |
|
||||
| R-C2-01 | Homepage is specified as a live catalog with status pings and per-service widgets, but it is a hand-maintained ConfigMap of link tiles with no status or widgets. | `platform/homepage/configmap.yaml`, `platform/homepage/README.md` |
|
||||
| R-C2-02 | The C2 must surface platform health and firing alerts, but none is shown on the C2 landing surface; health is reachable only by linking out to Grafana. | `platform/homepage/configmap.yaml` |
|
||||
| R-C2-02 | Grafana dashboards are specified as embedded in the C2, but Homepage only links to Grafana. | `platform/homepage/configmap.yaml` |
|
||||
| R-C2-02 | Flux status is specified via Headlamp's Flux view, the Flux UI or a Homepage widget, but Headlamp has no Flux plugin configured and Homepage has no widget. | `platform/headlamp/helmrelease.yaml`, `platform/homepage/configmap.yaml` |
|
||||
| R-C2-02 | C2 actions are specified to include VM start/stop, Flux reconcile and alert acknowledgement, but only native Headlamp workload actions exist, with no plugins for the others. | `platform/headlamp/helmrelease.yaml` |
|
||||
|
||||
## Service catalog
|
||||
|
||||
| Requirement or area | Requirement and shortfall | Evidence (implementation repo) |
|
||||
|---|---|---|
|
||||
| docs/13 catalog row 11 (kasm), R-IDP-02 | The catalog includes Kasm with native OIDC, and no ServiceDefinition, manifest or decision exists for it, so it is not onboarded. | `catalog/README.md:20` (only mention in the repo) |
|
||||
| R-STATE-03, docs/13 `backup.policy` | `backup.policy` must reference a named off-cluster schedule, and `daily-offcluster` resolves to no schedule object; it is an annotation only, while coverage comes from cluster-wide Velero schedules. | `platform/velero/README.md:33-34`, `schemas/renderer-contract.md:428`, `catalog/*.yaml` (`backup.policy`) |
|
||||
| R-UPD-01/02, docs/13 `update.cadence` | `update.cadence` should set how often a service checks or applies updates, and the renderer does not read it for containers or VMs. | `catalog/render.py` (no `cadence` reference), `schemas/renderer-contract.md:388-390` |
|
||||
|
||||
## Delivery, updates and conformance
|
||||
|
||||
| Requirement or area | Requirement and shortfall | Evidence (implementation repo) |
|
||||
|---|---|---|
|
||||
| R-TEST-03 | Merges require the tests for touched requirements to pass; the gate runs L0/L1 and tagged-coverage only, because L2/L3 do not run in CI (job disabled, trigger path unimplemented, cluster not declared), and a PASS names that declared gap. | `.gitea/workflows/gate.yml` (`conformance` job `if: false`), `ci/conformance.sh`, `docs/plans/l2-l3-conformance-ci-wiring.md` |
|
||||
| R-TEST-03 | Phase exits require the full suite and regression runs continuously; no pipeline runs the full-suite gate and no scheduled run exists, so L2/L3 evidence comes only from operator-recorded runs, last recorded 2026-09-19. | `conformance/run.sh` (`--require-all`, `--record`), `conformance/runs/ledger.jsonl` |
|
||||
| R-TEST-01, R-TEST-04 | A requirement with no passing test is not satisfied; all 51 registry requirements have a tagged test but PROVEN is 0 (3 stale: R-NET-06, R-STATE-02, R-STATE-03; 48 not derivable from the ledger). | `conformance/lib/status.py`, `conformance/runs/ledger.jsonl` |
|
||||
| R-OBS-02, R-GITOPS-02, R-GITOPS-03 | Each requirement needs an executable behavior test; the only verifying tests (`alerting`, `onboarding-completeness`, `gitops-drift`) are scaffolds. | `conformance/lib/status.py`, `conformance/tests/` |
|
||||
| R-TEST-06 | The spec must be verifiably immutable to agents; CI never checks the spec repo out, so spec-immutability always skips there (tolerated) and only local gates with a sibling checkout enforce it. | `.gitea/workflows/gate.yml:193,227`, `ci/spec-immutability.sh`, `ci/gate.sh` (`REQUIRE_SPEC_VERIFICATION`) |
|
||||
| R-UPD-01, R-UPD-02 | Hands-off updates must be proven by a behavior test; the only verifying test exercises Flux image-policy resolution, not the Renovate path that delivers updates. | `conformance/tests/l3-fault-injection/hands-off-update/chainsaw-test.yaml` |
|
||||
| R-UPD-03 | A pinned service must not advance; the ServiceDefinition `update.pin` only affects a disabled render path, so it would not stop Renovate (no catalog entry uses it; a Renovate rule is the working pin). | `catalog/render.py:1731`, `ci/render-golden.sh`, `renovate.json5` |
|
||||
| R-UPD-03 | A failed update must auto-roll-back; only HelmReleases have remediation, while plain Deployments stall on readiness and `autoRollbackOnFailedHealth` renders only onto the disabled image policy. | `catalog/render.py:1747`, `conformance/tests/l3-fault-injection/hands-off-update/chainsaw-test.yaml` |
|
||||
| R-UPD-02, R-OBS-04 | Update failures must alert; the update alerts watch Flux objects that do not exist and so cannot fire, and nothing alerts when Renovate's schedule stops. | `platform/monitoring/alerts.yaml:1538,1552`, `.gitea/workflows/renovate.yaml` header |
|
||||
Reference in new issue
Block a user