diff --git a/AGENTS.md b/AGENTS.md index 1bbbf63..b6e10b6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -11,18 +11,18 @@ fully before making any change. These rules **override** any conflicting default > requirement-tagged conformance test — never "merged," never a manual demo (R-TEST-01..05). > See §7 and [`docs/14-testing-and-conformance.md`](docs/14-testing-and-conformance.md). -## 1. This repository defines intent, not implementation (yet) +## 1. This repository defines intent; the implementation has its own repository -Today this repo contains **specifications only**. Implementation artifacts (Helm values, Flux -Kustomizations, KubeVirt manifests, OpenTofu/Ansible IaC, etc.) will be added under an -`implementation/` tree **only after** the corresponding spec and ADRs are approved. Until then: +This repo contains **specifications only**. Implementation artifacts (Helm values, Flux +Kustomizations, KubeVirt manifests, Ansible IaC, the merge gate, the conformance tests) live in the +[implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl), whose +`CLAUDE.md` governs how work lands there. - Do **not** invent implementation details that contradict the specs or ADRs. -- Do **not** create the `implementation/` tree until the roadmap phase that authorizes it is active - (see [`roadmap/phases.md`](roadmap/phases.md)). +- Do **not** add implementation artifacts to this repo. - When you implement, the spec is the acceptance criteria. If the spec is wrong, blocking, ambiguous, or missing, **you may not fix it yourself** — stop and escalate to the human operator/ - architect (§2, §11). You may draft a proposal; you may not merge a spec change. + System Architect (§2, §11). You may draft a proposal; you may not merge a spec change. ## 2. The source-of-truth hierarchy @@ -32,9 +32,9 @@ When two documents conflict, the higher item wins: 2. Approved ADRs in [`decisions/`](decisions/) — locked technology & pattern choices 3. Domain specs in [`docs/`](docs/) 4. [`roadmap/`](roadmap/) and [`agents/`](agents/) — sequencing and coordination -5. Implementation code (once it exists) +5. Implementation code -If you find a conflict, **stop and escalate it to the human operator/architect** — do not silently +If you find a conflict, **stop and escalate it to the human operator/System Architect** — do not silently pick one interpretation, and do not resolve it by editing the specs yourself (you may not change them, §11). Wait for the human to amend the spec, then implement against the amended spec. @@ -44,7 +44,7 @@ Every major choice (k3s, Cilium, Longhorn, KubeVirt, Flux, SOPS, cert-manager, e an ADR with rationale and rejected alternatives. If new information genuinely invalidates a decision: -1. **Draft** a proposed superseding ADR and **escalate to the human operator/architect** — you may +1. **Draft** a proposed superseding ADR and **escalate to the human operator/System Architect** — you may not merge it yourself (ADRs are specifications; §11 forbids agents changing them). 2. State what changed and why. 3. Only the human approves/accepts it. Implement against the new decision only after it is accepted. @@ -55,7 +55,6 @@ Never edit an accepted ADR's decision in place. ADRs are an append-only, **human - Every requirement has a stable ID like `R-NET-01`. - Every ADR lists the requirement IDs it serves. -- Every task-board item lists the requirement IDs it satisfies. - Every implementation PR references the requirement IDs and the spec sections it fulfills. If you cannot trace a piece of work to a requirement, either it is out of scope or a requirement is @@ -70,19 +69,20 @@ missing — surface it, don't proceed silently. - Secrets live in git **encrypted** (SOPS + age). Never commit plaintext secrets. Never print a decrypted secret into logs, PR descriptions, or agent output. -## 6. Two environments, one configuration +## 6. One environment, one configuration -- Author configuration so dev/test and prod differ **only** by environment-scoped overrides - (replica counts, node selectors, storage sizes, WAN endpoints, DNS zones). +- Production is the one live environment ([ADR-0017](decisions/adr-0017-single-environment-bare-metal-production.md)). +- Author configuration as a shared base plus an environment-scoped overlay (replica counts, node + selectors, storage sizes, WAN endpoints, DNS zones), so any future environment needs overrides only. - Never fork the base configuration per environment. Use overlays/variables. -- **Never build directly on production.** Prod hardware is a design target, not a development - surface, until the roadmap declares it ready. +- **Changes reach production only through the PR gate and Flux** (R-ENV-03). Production is not a + surface to experiment on by hand. ## 7. Acceptance gates — proven by automated, reproducible behavior tests (non-negotiable) A change is not "done" until an **automated, reproducible, behavior-based test** (Given/When/Then, -tagged with the requirement IDs it verifies) **passes green in dev/test** and is committed alongside -the change. A manual demonstration, a screenshot, or "it reconciled" is **not** acceptance +tagged with the requirement IDs it verifies) **passes green** (offline for L0/L1, against the live +cluster for L2/L3) and is committed alongside the change. A manual demonstration, a screenshot, or "it reconciled" is **not** acceptance (R-TEST-01..04). The approach — reproducibility rules, layers, tooling, and the conformance suite — is specified in [`docs/14-testing-and-conformance.md`](docs/14-testing-and-conformance.md) and [`decisions/adr-0010-conformance-testing.md`](decisions/adr-0010-conformance-testing.md). @@ -105,8 +105,11 @@ See [`agents/definition-of-done.md`](agents/definition-of-done.md) for the full ## 8. Coordination -- Claim work on [`agents/task-board.md`](agents/task-board.md) before starting; respect declared - dependencies (`blocked-by`). +- Implementation work is taken up through issues on the implementation repo; each unit of work + records a dated handoff entry in that repo's `decisions/`. +- Requirement status is derived from the repository (`./conformance/run.sh --status` in the + implementation repo), not from a hand-maintained board. [`agents/task-board.md`](agents/task-board.md) + is a frozen historical seed; nobody edits it. - Follow [`agents/coordination-protocol.md`](agents/coordination-protocol.md) for handoffs, interface contracts, and conflict resolution between agents/workstreams. - Prefer small, reviewable, single-responsibility changes. One workstream should not silently change @@ -120,9 +123,9 @@ See [`agents/definition-of-done.md`](agents/definition-of-done.md) for the full **Open Questions** heading rather than burying ambiguity in the body. - Keep the [`GLOSSARY.md`](GLOSSARY.md) authoritative — if you introduce a term, define it there. -## 10. What "you do not write code" means for the architect vs. what agents do +## 10. What "you do not write code" means for the System Architect vs. what agents do -The **architect** authored the specs, ADRs, and this coordination structure — no code. The +The **System Architect** authored the specs, ADRs, and this coordination structure — no code. The **implementation agents** who consume this repo *do* write code (implementation *and* the automated conformance tests that prove it), but only in service of an approved spec, only under the active roadmap phase, and only against the automated acceptance gates above. Agents write tests; agents do @@ -135,12 +138,14 @@ This is a hard boundary (R-TEST-06): - **You MUST NOT modify** anything under [`docs/`](docs/), the requirements, accepted ADRs in [`decisions/`](decisions/), the roadmap, or these coordination files. These are human-owned. - **You MAY and MUST create/modify**: implementation code, **automated conformance tests**, the - implementation repo, and the *live coordination state* the specs explicitly delegate to you — - namely claiming/updating items on [`agents/task-board.md`](agents/task-board.md) and recording - proposals/escalations there. (The task board is a working surface; the *rules* around it are not.) + and the implementation repo, including its handoff log. Nothing in this repo is a working surface + for agents: any commit moves the spec `HEAD` away from the commit the implementation has pinned. - **When the spec is wrong, blocking, ambiguous, or missing:** STOP. Do not proceed on a guess and - do not edit the spec. Raise an escalation on the task board (and draft a proposal if useful) for - the **human operator/architect** to decide. Resume only after the human amends the spec. + do not edit the spec. Raise an escalation as an issue on the implementation repo or directly to + the operator (and draft a proposal if useful) for the **human operator/System Architect** to + decide. Resume only after the human amends the spec. +- **Every spec advance is adopted explicitly** by the implementation + ([`CONTRIBUTING.md`](CONTRIBUTING.md), "Adopting a spec change in the implementation"). - **Never** make the implementation pass by weakening the spec, the requirement, or the test's intent. If reality can't meet the spec, that is an escalation, not a spec edit. @@ -151,11 +156,12 @@ toward the target; they never move the target. Full detail in [`CONTRIBUTING.md`](CONTRIBUTING.md). The rules that bind every change: -- **No work on `main`.** `main` holds only completed, reviewed work. -- **Every change is a feature branch** (`feature/…`, `fix/…`, `docs/…`, `chore/…`), **merged into - `main`** when complete (and, for implementation, when its conformance tests pass). -- **Local git for now; Gitea soon.** There is no remote host on this network yet — the prior - `home-network` Gitea is gone from this network. **Gitea is one of the first services brought online - on the cluster**, after which repos, the GitOps source, and CI/CD move to Gitea (Gitea Actions - running the Kyverno gate + Chainsaw conformance Job, with PRs as the enforced merge gate). +- **No work on `main`.** `main` holds only completed work. +- **Every change is a branch and a pull request.** A spec change is merged only by the human + operator. An implementation change merges on the green merge gate, the approver, with a non-author + review on sensitive paths ([ADR-0024](decisions/adr-0024-delivery-authority-ci-gate.md)). +- **Agents never push to `main`, never merge their own PR, and never post the administrator override + to clear their own work.** +- **Both repos are hosted on the platform's own Gitea**, which is also the GitOps source; Gitea + Actions runs the merge gate. - **Never commit plaintext secrets** (SOPS-encrypted only, [`docs/07-security-and-secrets.md`](docs/07-security-and-secrets.md)). diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2df3cc6..24328da 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,18 +4,15 @@ How work is version-controlled for this repository. This complements the governa [`AGENTS.md`](AGENTS.md) (what agents may/may not change) — this file covers *how changes land in git*. -## Interim state (now): local git, feature branches → main +## This repo (the spec) -There is **no remote git host on this network today**. (The previous `home-network` Gitea instance is -gone from this network; the infrastructure it built still runs but is not reachable here.) So for now: +The spec repo is hosted on the platform's own Gitea (`gitea.home.keithjackson.dev`). -- **Local git only.** The repository lives on the workstation; there is no push target yet. -- **No work on `main`.** `main` holds only reviewed, completed work. Do **not** commit work-in- - progress directly to `main`. -- **Every change is a feature branch.** Create `feature/` (or `fix/…`, `docs/…`) - for any unit of work, commit there, then **merge into `main`** when the work is complete and — for - implementation work — its conformance tests pass ([`agents/definition-of-done.md`](agents/definition-of-done.md)). -- **Merge to `main` = "done".** Keep `main` always in a coherent, mergeable state. +- **No work on `main`.** `main` holds only reviewed, completed work. +- **Every change is a feature branch**, merged into `main` by pull request. Only the human operator + merges a spec change (R-TEST-06). +- **Merge a coherent batch.** Every spec commit on `main` must be adopted by the implementation (see + below), so small drive-by commits each carry a real cost. ### Branch naming - `feature/` — new capability or spec area @@ -23,36 +20,67 @@ gone from this network; the infrastructure it built still runs but is not reacha - `docs/` — documentation-only changes - `chore/` — tooling/repo housekeeping +The implementation repo has its own convention (`feat/`, `fix/`, `docs/`, `chore/`). + ### Commit hygiene - Small, single-responsibility commits; imperative subject lines. -- Reference requirement IDs / ADR numbers / task-board IDs where relevant (they're stable across - file moves). +- Reference requirement IDs / ADR numbers where relevant (they're stable across file moves). - **Never commit plaintext secrets** — secrets are SOPS-encrypted only ([`docs/07-security-and-secrets.md`](docs/07-security-and-secrets.md)); the [`.gitignore`](.gitignore) guards common leak paths but is not a substitute for care. -## Target state (soon): Gitea-hosted, CI/CD-driven GitOps +## The implementation repo -The platform will **host its own git**. **Gitea is one of the first services brought online on the -cluster** ([`docs/13-service-catalog.md`](docs/13-service-catalog.md), roadmap Phase 6 is pulled -early — see [`roadmap/phases.md`](roadmap/phases.md)). Once Gitea is up and stable, the workflow -shifts: +The [implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl) is +CI/CD-driven GitOps. Its `CLAUDE.md` is the delivery contract for how work lands there; intent is in +[`docs/05-gitops-and-delivery.md`](docs/05-gitops-and-delivery.md) and +[ADR-0024](decisions/adr-0024-delivery-authority-ci-gate.md). -1. **Repos move to Gitea** (`gitea.home.keithjackson.dev`). This local repo is pushed up as the - origin of record. -2. **GitOps source migrates** from local/bootstrap to Gitea: Flux reconciles the cluster from Gitea - ([`docs/05-gitops-and-delivery.md`](docs/05-gitops-and-delivery.md)). -3. **CI/CD runs in Gitea** (Gitea Actions): the Kyverno L0 policy gate and the trigger for the - in-cluster Chainsaw conformance Job run as pipelines on push/PR - ([`docs/14-testing-and-conformance.md`](docs/14-testing-and-conformance.md)); Renovate opens - update PRs ([`docs/11-automated-updates.md`](docs/11-automated-updates.md)). -4. **Pull requests become the merge gate** into `main`, with CI (policy + conformance) required - green before merge — the same no-work-in-main rule, now enforced by the platform instead of by - convention. +- Every change is branch → PR → **merge gate green** → merge. Direct pushes to `main` are rejected. +- The CI gate is the approver; sensitive paths also need a non-author review. +- Flux reconciles the cluster from that repo's `main`. +- Each unit of work records a dated handoff entry under that repo's `decisions/`. Those are + implementation handoffs; this repo's `decisions/` holds ADRs. -### The bootstrapping caveat (self-hosted git is circular) +### Where status lives + +**`./conformance/run.sh --status` in the implementation repo is the live status surface**: per +requirement, whether it is implemented, reconciled, has an executable test, and has recorded proof. +[`docs/16-known-gaps.md`](docs/16-known-gaps.md) is a dated snapshot of the shortfalls at one point in +time. [`agents/task-board.md`](agents/task-board.md) is a frozen historical seed. + +## Adopting a spec change in the implementation + +The implementation pins the exact spec commit it is built against, in `ci/SPEC_PIN`. Its +spec-immutability check (R-TEST-06) fails when a spec checkout has uncommitted changes or its `HEAD` +differs from the pin, so every movement of spec `main` is an explicit, reviewed act. On **every** +spec change: + +1. **Merge to spec `main`.** The human operator does this. Until step 2 lands, a local gate against + a checkout of the new `main` reports blocked. +2. **Bump the pin** in an implementation PR, to the new spec `main` commit. `ci/` is a sensitive path, + so the PR needs the operator's review or the operator's own override; an agent never overrides + for itself. +3. **Reconcile requirement IDs in that same PR.** The registry + (`conformance/requirements/registry.yaml`) mirrors the `R-*` IDs in + [`docs/00-vision-and-scope.md`](docs/00-vision-and-scope.md): + - A **new ID** must arrive with a requirement-tagged test, or be listed as a documented gap in the + harness. Adding the registry entry attributes that ID to the change, so an uncovered ID blocks + the merge; a safety-critical ID also needs an L3 test. + - An **amended requirement** needs only its abridged registry summary updated, when the meaning + changed. + - A **retired ID** leaves the registry, every test label, and the gate's path-to-requirement map + together. IDs are never renumbered or reused. +4. **Write the handoff entry** for the adoption. + +Some implementation checks read spec text when a spec checkout is present: the auditability section +of [`docs/07`](docs/07-security-and-secrets.md), the Phase 7 entry in +[`roadmap/phases.md`](roadmap/phases.md), and the requirement definitions in `docs/00`. Change those +structures knowingly. + +## The bootstrapping caveat (self-hosted git is circular) Gitea runs *on* the cluster that Flux reconciles *from* Gitea — a chicken-and-egg. Handling: -- Bring the cluster up from a **bootstrap source** (this local repo / an out-of-band source), stand - up Gitea, then **cut Flux's source over to Gitea**. +- Bring the cluster up from a **bootstrap source** (an out-of-band copy of the repo), stand up Gitea, + then **cut Flux's source over to Gitea** ([`docs/05-gitops-and-delivery.md`](docs/05-gitops-and-delivery.md)). - Because a cluster loss would take its own git host with it, **Gitea is DR-critical**: its repos are backed up **off-cluster** ([`docs/04-storage-and-state.md`](docs/04-storage-and-state.md)), and an off-cluster mirror of the platform repo is kept so the platform can always be rebuilt from a source diff --git a/GLOSSARY.md b/GLOSSARY.md index 43866fb..732c441 100644 --- a/GLOSSARY.md +++ b/GLOSSARY.md @@ -10,14 +10,19 @@ two things (e.g. "VM"), this file disambiguates. - **ServiceDefinition** — the operator-facing YAML object that declares a service's intent (kind, source, storage, egress, exposure, dependencies, update policy, observability). The platform renders concrete resources from it. See [`docs/13-service-catalog.md`](docs/13-service-catalog.md). +- **Renderer** — the catalog tool that generates a service's concrete manifests from its + ServiceDefinition; generated output is not hand-edited. +- **Wave** — a dependency-ordered group of rendered apps reconciled and health-gated together, so a + failing service holds back only its own wave and those above it. - **Golden path** — the standard flow a service follows from catalog entry to running/monitored/ updated. See [`docs/01-architecture-overview.md`](docs/01-architecture-overview.md). - **C2 / Command & Control** — the UI at the root `home.keithjackson.dev`. See [`docs/12-command-center-ui.md`](docs/12-command-center-ui.md). +- **Apex** — the root name `home.keithjackson.dev`, which serves the C2 UI. ## The two kinds of VM (do not conflate) -- **Infrastructure VM (Layer 1)** — a cluster *node*. In dev/test these are libvirt KVMs; in prod - they are bare-metal mini-PCs. Managed by OpenTofu/Ansible, **not** KubeVirt. +- **Infrastructure VM (Layer 1)** — a cluster *node*: a bare-metal mini-PC, imaged and managed by + Ansible, **not** KubeVirt. The name is historical. - **Application VM (Layer 2)** — an IaaS *workload* running inside the cluster via **KubeVirt**, managed by Kubernetes/GitOps like a container. @@ -27,14 +32,22 @@ two things (e.g. "VM"), this file disambiguates. - **Egress class** — a service's declared outbound path: `internet` (standard WAN) or `vpn` (commercial VPN tunnel). See [`docs/03-networking-isolation-and-egress.md`](docs/03-networking-isolation-and-egress.md). - **Exposure class** — how a service is reachable: `internal` (LAN/VPN-only) or `public` (published - through the Cloudflare Tunnel, ADR-0016). + through the Cloudflare Tunnel, ADR-0016). A registered platform route may also be `public`. - **Cloudflare Tunnel** — the outbound-only connection from in-cluster cloudflared to Cloudflare that carries all public HTTP(S) ingress; public records are proxied CNAMEs to `.cfargotunnel.com` (ADR-0016). - **Kill-switch** — the guarantee that a `vpn`-class workload leaks nothing to the plain internet if the tunnel drops (R-NET-06); fail-closed. -- **Multi-WAN** — multiple upstream internet uplinks with routing-rule-based selection; physical WAN - selection is largely an edge-router responsibility (OQ-1). +- **Multi-WAN** — multiple upstream internet uplinks with routing-rule-based selection. Out of scope: + the site has one WAN (OQ-1). +- **Egress VPN** — the outbound path that carries `vpn`-class pods to the internet through the + commercial tunnel (ADR-0015, ADR-0020). +- **Remote-access VPN** — the inbound WireGuard path that lets the operator's own devices join the + LAN (ADR-0021). It shares nothing with the egress VPN but the word. +- **Hairpin** — a LAN client reaching a service through the site's own public address; the edge + cannot be relied on for it, so no path may depend on it (R-DNS-04). +- **L4 VIP** — a LAN virtual IP that exposes a non-HTTP service directly, outside the shared Gateway. +- **kube-vip** — the virtual IP that gives the Kubernetes API one HA endpoint across the 3 servers. - **Egress Gateway** — Cilium feature steering selected pods' egress through a chosen node/identity. - **Hubble** — Cilium's flow-observability tool; the audit surface for isolation/egress. - **Gateway API** — the Kubernetes standard for north-south ingress, implemented here by Cilium. @@ -45,7 +58,10 @@ two things (e.g. "VM"), this file disambiguates. - **Longhorn** — distributed replicated block storage across the 3 nodes (default 3 replicas). The durable home for app/config state and VM disks. - **Bulk tier** — a separate storage class for large media (Plex library, downloads) that is **not** - 3× replicated (R-STATE-04). See [`docs/04-storage-and-state.md`](docs/04-storage-and-state.md). + 3× replicated (R-STATE-04): NFS from a standalone TrueNAS, outside platform backup (ADR-0018). See + [`docs/04-storage-and-state.md`](docs/04-storage-and-state.md). +- **CNPG (CloudNativePG)** — the operator that runs the platform's replicated PostgreSQL clusters and + archives them off-cluster with Barman (ADR-0019). - **CDI** — Containerized Data Importer; provisions KubeVirt VM disks onto Longhorn PVCs. - **Velero** — cluster backup/restore orchestration to an off-cluster target (DR). - **DR drill** — a tested restore of resources + volumes proving R-STATE-03. @@ -54,9 +70,12 @@ two things (e.g. "VM"), this file disambiguates. - **GitOps** — desired state lives in git and is reconciled into the cluster automatically (pull-based). Engine: **Flux CD**. - **Drift correction** — Flux reverting live changes back to git-declared state (R-GITOPS-03). -- **Image automation** — Flux controllers advancing container image versions per declared policy - with no human/AI edit (R-UPD-02). -- **Renovate** — bot that opens (and auto-merges) chart/manifest version-bump PRs. +- **Renovate** — bot that opens machine-authored version-bump PRs for images and charts, and + auto-merges those in the allowed classes through the merge gate (R-UPD-02, ADR-0023). +- **Blast-radius component** — an operator-designated platform component whose machine-authored + update waits for human review (R-UPD-02). +- **PM agent** — the in-cluster AI Program Manager (pm-agent) that turns issues into pull requests; + it never pushes to `main` or merges (ADR-0025). - **SOPS + age** — the secret-encryption scheme; secrets live encrypted in git, decrypted in-cluster. - **cert-manager / ACME DNS-01 / Let's Encrypt** — the trusted, automated TLS pipeline (R-TLS-*). - **ExternalDNS** — controller that publishes DNS records from declared service intent. @@ -67,8 +86,10 @@ two things (e.g. "VM"), this file disambiguates. - **SSO (Single Sign-On)** — log in once, reach every integrated service without re-authenticating. - **OIDC (OpenID Connect)** — the token-based auth protocol services use to federate to Keycloak natively (a Keycloak **client**). -- **Forward-auth** — an identity-aware proxy (**oauth2-proxy**) at the Gateway that authenticates a - request against Keycloak before forwarding it — used for apps without native OIDC. +- **Forward-auth** — an identity-aware proxy (**oauth2-proxy**) that authenticates a request against + Keycloak before forwarding it — used for apps without native OIDC. +- **Forward-auth sidecar** — the per-app oauth2-proxy container that enforces forward-auth inside + the app's own pod, rather than at the Gateway (ADR-0022). - **Realm** — a Keycloak tenant/namespace holding the platform's users, groups, and clients. - **Group** — a Keycloak grouping that drives per-service authorization (`auth.groups` in a ServiceDefinition); the basis for least-privilege access (R-IDP-06). @@ -76,10 +97,10 @@ two things (e.g. "VM"), this file disambiguates. doesn't lock the operator out of recovery tools. ## Environments -- **dev/test** — local KVM cluster; the active build target where everything is proven. -- **prod** — 3× mini-PC bare-metal cluster; design-only until roadmap Phase 7. +- **dev/test** — the former cluster that proved the platform; decommissioned 2026-09 (ADR-0017). +- **prod** — 3× mini-PC bare-metal cluster; the one live environment. - **Overlay** — environment-scoped configuration (Kustomize/vars) layered on shared base config; the - only thing that differs between dev and prod (R-ENV-02). + only thing a future environment would change (R-ENV-02). ## Governance - **Requirement ID (`R-*`)** — a stable, referenceable requirement from @@ -100,6 +121,11 @@ two things (e.g. "VM"), this file disambiguates. exits; green suite = progress (R-TEST-03). See [`docs/14-testing-and-conformance.md`](docs/14-testing-and-conformance.md). - **Fault-injection test** — a test that *breaks* something (tunnel down, node killed, bad update) and asserts a fail-closed / no-data-loss outcome; required for safety-critical behavior (R-TEST-05). +- **Conformance layers (L0–L3)** — L0 static policy and repo checks; L1 offline render checks; L2 + behavior on a live cluster; L3 fault injection. See [`docs/14-testing-and-conformance.md`](docs/14-testing-and-conformance.md). +- **Covered vs. proven** — a requirement is *covered* when an executable test exists for it, and + *proven* only when a recorded run of that test passed at the current revision on the cluster it + claims. - **Coverage report** — the generated map of every `R-*` requirement → its verifying test(s); an uncovered requirement is an unmet requirement (R-TEST-04). - **Spec immutability (to agents)** — the rule that AI agents may not modify the specifications diff --git a/README.md b/README.md index 77aa728..bbe5890 100644 --- a/README.md +++ b/README.md @@ -19,8 +19,8 @@ The platform must provide: for workloads that cannot be containerized. - **Network isolation**: default-deny east-west traffic; explicit, port-level allow rules between services (firewall-like). -- **Multi-WAN egress**: some services egress the normal internet; others are routed through a - custom VPN connection. Egress path is a declarative property of a service. +- **Multiple egress paths**: some services egress the normal internet; others are routed through + a custom VPN connection. Egress path is a declarative property of a service. - **Stateful durability**: any service can be destroyed and rebuilt on a whim without losing data. - **Hands-off updates**: services update to latest stable versions on a schedule — no human, no AI, no hand-written GitOps change required. @@ -32,15 +32,16 @@ The platform must provide: based test — and the specifications themselves are human-owned and immutable to the implementing agents. -## Two environments +## One live environment | Environment | Substrate | Purpose | Status | |---|---|---|---| -| **dev/test** | Local KVM virtual machines on a workstation, provisioned by IaC | Prove the platform end-to-end before hardware exists | **Active development target** | -| **prod** | 3× mini-PC bare-metal k3s cluster, each with local storage | The real deployment | **Not ready — design for it, do not build on it yet** | +| **prod** | 3× mini-PC bare-metal k3s cluster, each with local storage | The real deployment | **Live; roadmap Phase 7 open** | -The dev/test environment is the proving ground. Everything must be authored so that the *same* -GitOps configuration promotes to production with only environment-scoped overrides. +The dev/test cluster that proved the platform was decommissioned in 2026-09 (ADR-0017). Configuration +is authored once as a base plus an environment-scoped overlay, so any future environment needs +overrides only. The platform is built in the +[implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl). ## How to use this repository (start here) @@ -61,7 +62,7 @@ GitOps configuration promotes to production with only environment-scoped overrid home-network-k3s/ ├── README.md ← you are here ├── AGENTS.md ← rules of engagement for AI agents -├── CONTRIBUTING.md ← git workflow (feature branches → main; Gitea/CI-CD to come) +├── CONTRIBUTING.md ← git workflow (feature branches → PRs → main) ├── GLOSSARY.md ← shared vocabulary (read to avoid ambiguity) ├── docs/ ← the specification, by domain │ ├── 00-vision-and-scope.md @@ -79,7 +80,8 @@ home-network-k3s/ │ ├── 12-command-center-ui.md │ ├── 13-service-catalog.md │ ├── 14-testing-and-conformance.md -│ └── 15-identity-and-access.md +│ ├── 15-identity-and-access.md +│ └── 16-known-gaps.md ├── decisions/ ← ADRs (one decision per file) + index/template ├── roadmap/ ← phases.md, workstreams.md └── agents/ ← roles, coordination protocol, DoD, task board @@ -104,4 +106,4 @@ system meets the spec. - **Everything is GitOps.** No click-ops in the cluster except where an ADR explicitly allows it. - **Nothing is self-signed, nothing is manual, nothing loses data.** These three properties are acceptance gates, not aspirations — and each is proven by a test. -- **Design once, deploy to two environments.** dev/test and prod differ only by scoped overrides. +- **Design once.** Configuration is one base plus scoped overlay overrides. diff --git a/agents/coordination-protocol.md b/agents/coordination-protocol.md index 5fa344e..56c0089 100644 --- a/agents/coordination-protocol.md +++ b/agents/coordination-protocol.md @@ -3,9 +3,9 @@ How agents work together without stepping on each other. Read alongside [`roles.md`](roles.md), [`../roadmap/workstreams.md`](../roadmap/workstreams.md), and [`../AGENTS.md`](../AGENTS.md). -## 1. Claim before you build -- Take an item from [`task-board.md`](task-board.md), set its owner and status to `in-progress`. -- Respect `blocked-by`. Do not start blocked work; help unblock it instead. +## 1. Take work from an issue, in your own workspace +- Implementation work is taken up through issues on the implementation repo, on a branch of its own. +- Do not start work that depends on something unfinished; help unblock it instead. - Keep items small and single-responsibility. ## 2. Contracts are sacred @@ -14,7 +14,7 @@ the ServiceDefinition schema, StorageClasses, the network policy/egress renderin Gateway, the observability contract, the update-policy contract. - **Consume** a contract freely. -- **Change** a contract only by: (1) proposing the change on the task board with the affected +- **Change** a contract only by: (1) proposing the change in an implementation-repo issue with the affected workstreams tagged, (2) getting the owning role's sign-off, (3) updating the spec that defines the contract in the *same* change. Never silently break a downstream consumer. - The **ServiceDefinition schema** is the highest-traffic contract; treat schema changes as @@ -25,12 +25,12 @@ Gateway, the observability contract, the update-policy contract. are human-owned and immutable to agents (R-TEST-06, [`../AGENTS.md` §11](../AGENTS.md)). - Anything that changes a locked choice ⇒ **draft** a superseding ADR and **escalate to the human** ([`../decisions/README.md`](../decisions/README.md)); do not self-approve or merge it. -- Genuine unknowns, or a spec that is wrong/blocking/ambiguous ⇒ **stop and escalate** on the task - board as an **Open Question (OQ-n)** for the human to resolve in the spec; don't bury ambiguity in - an implementation and don't patch the spec yourself. +- Genuine unknowns, or a spec that is wrong/blocking/ambiguous ⇒ **stop and escalate** in an + implementation-repo issue or directly to the operator, for the human to resolve in the spec; don't + bury ambiguity in an implementation and don't patch the spec yourself. ## 4. Definition of done -Nothing is "done" until its **automated, reproducible behavior test passes green** in **dev/test**, +Nothing is "done" until its **automated, reproducible behavior test passes green**, committed with the change and tagged to its requirement IDs ([`definition-of-done.md`](definition-of-done.md), [`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md)). "Merged" ≠ "done"; "demonstrated manually" ≠ "done". You write the test as well as the implementation. @@ -39,18 +39,19 @@ committed with the change and tagged to its requirement IDs ([`definition-of-don - Only work items in the **active phase(s)** ([`../roadmap/phases.md`](../roadmap/phases.md)) unless explicitly parallelizable. - Respect entry gates; don't start a phase whose predecessor's exit criteria aren't demonstrated. -- **Prod (Phase 7) is closed** until the operator opens it. +- **Prod (Phase 7)** was opened by the operator's declaration; production is the one live environment + and changes reach it only through the PR gate and Flux. ## 6. Handoffs When your output feeds another workstream: - State the interface you're delivering and where it's documented. - Provide the demonstrable proof (the gate you passed) so the consumer can trust it. -- If you changed a shared contract, announce it on the board with consumers tagged. +- If you changed a shared contract, say so in your handoff entry and PR, with consumers tagged. ## 7. Conflict resolution 1. Re-read the source-of-truth hierarchy ([`../AGENTS.md` §2](../AGENTS.md)); the higher document wins. -2. If specs genuinely conflict or are silent, escalate to the **Platform Architect** to amend the +2. If specs genuinely conflict or are silent, escalate to the **System Architect** to amend the spec/ADR — fix the documents first, then build. 3. Never resolve a conflict by silently picking an interpretation in code. @@ -63,7 +64,9 @@ When your output feeds another workstream: - **Never weaken a test or a requirement to get green** — that's a defect, not progress. ## 9. Communication artifacts -- **task-board.md** — the live who's-doing-what + status + blockers. +- **Implementation-repo issues** — work intake and escalations. +- **Implementation handoff log** (`decisions/` there) — what changed, what was verified, what was not. +- **`./conformance/run.sh --status`** (implementation repo) — the live requirement status. - **specs/ADRs** — durable decisions and contracts. - **PR descriptions** — reference requirement IDs, specs, ADRs, and the gate demonstrated. -Keep these three in sync; they are how independent agents stay coherent. +Keep these in sync; they are how independent agents stay coherent. diff --git a/agents/definition-of-done.md b/agents/definition-of-done.md index fb47753..67b7347 100644 --- a/agents/definition-of-done.md +++ b/agents/definition-of-done.md @@ -1,7 +1,7 @@ # Definition of Done A change is **done** only when an **automated, reproducible, behavior-based test** for each relevant -gate below **passes green in dev/test**, the test is committed with the change and tagged with the +gate below **passes green**, the test is committed with the change and tagged with the requirement IDs it verifies, and the change is traced to those IDs. A manual demonstration, a screenshot, or "it reconciles" is **not** done (R-TEST-01..04, [`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md)). @@ -21,7 +21,7 @@ screenshot, or "it reconciles" is **not** done (R-TEST-01..04, [`../docs/14-test - [ ] Observability wired (metrics/logs) and at least the relevant alert exists. - [ ] No specification/ADR/requirement was modified to achieve this (R-TEST-06); any spec problem was escalated, not edited. -- [ ] Result recorded on the task board / PR with a link to the passing test run. +- [ ] Result recorded in the PR and a handoff entry, with a link to or ledger row for the passing run. ## Requirement-class gates (apply those relevant to the change) @@ -55,11 +55,13 @@ screenshot, or "it reconciles" is **not** done (R-TEST-01..04, [`../docs/14-test - [ ] `internal` is unreachable publicly; `public` is reachable + TLS-valid. - [ ] Internal clients resolving via Pi-hole reach the internal Gateway; via a public resolver, a `public` service still works through the tunnel (R-DNS-04). -- [ ] Non-`public` hostnames get 404 at the tunnel and have no public record (R-DNS-05); the - Kubernetes API is not publicly reachable (R-SEC-04). +- [ ] LAN/VPN-only hostnames get 404 at the tunnel; `internal` services have no public record, except + non-HTTP L4 names, which may hold DNS-only records at LAN addresses (R-DNS-05); the Kubernetes + API is not publicly reachable (R-SEC-04). ### Automated updates (R-UPD-*) -- [ ] Advances to a newer **stable** version automatically on schedule with **no human/AI edit**. +- [ ] Advances to a newer **stable** version automatically on schedule with **no human/AI edit**, as + a machine-authored change through the merge gate (allowed update classes). - [ ] A deliberately bad update is health-gated: **auto-rollback** + alert. - [ ] A **pinned** service does not advance. diff --git a/agents/roles.md b/agents/roles.md index ce2ea51..7dc54f6 100644 --- a/agents/roles.md +++ b/agents/roles.md @@ -6,15 +6,22 @@ not people — one agent may hold several early on; split as work scales. ## Coordination roles -### Platform Architect (steward of this spec repo) +### System Architect (steward of this spec repo) - Guards the source-of-truth hierarchy ([`../AGENTS.md` §2](../AGENTS.md)); approves/records ADRs. - Resolves cross-workstream interface conflicts; keeps requirement traceability honest. - Does **not** write implementation code; owns specs, ADRs, roadmap, and this coordination layer. ### Integration/Release Lead - Owns the implementation repo structure, CI gates, and Flux bootstrap health. -- Runs phase entry/exit reviews against the **automated conformance suite**; keeps the task board - truthful. +- Runs phase entry/exit reviews against the **automated conformance suite**; keeps the handoff log + and the derived requirement status truthful. + +### Program Manager +- Decides what should happen about implementation work: starts, revises, or declines it, and briefs + workers. Lives in the implementation repo; delegates code and debugging rather than doing them. +- Does **not** hold the System Architect role: work that needs a spec change is escalated. The + in-cluster pm-agent ([ADR-0025](../decisions/adr-0025-in-cluster-ai-program-manager.md)) is an + automated instance of this role. ### QA/Conformance Engineer (WS-TEST) - Owns the conformance testing capability ([14](../docs/14-testing-and-conformance.md), @@ -52,5 +59,5 @@ not people — one agent may hold several early on; split as work scales. ([`../decisions/README.md`](../decisions/README.md)). - Negotiate interface changes before breaking another workstream's contract ([coordination-protocol.md](coordination-protocol.md)). -- Never build on prod hardware before Phase 7 is opened. +- Reach production only through the PR gate and Flux (R-ENV-03). - Never emit plaintext secrets. diff --git a/agents/task-board.md b/agents/task-board.md index 7e4ca04..d2b667f 100644 --- a/agents/task-board.md +++ b/agents/task-board.md @@ -1,14 +1,18 @@ # Task Board +> **FROZEN — historical seed.** By the operator's ruling of 2026-08-31 this board is not the status +> surface; the repository is. **Nobody edits it**, and its contents and the instructions below no +> longer describe live work. Its `T-*` IDs stay stable because the implementation repo cites them. +> Live status: `./conformance/run.sh --status` in the implementation repo; dated snapshot: +> [`../docs/16-known-gaps.md`](../docs/16-known-gaps.md). + The live who's-doing-what. Agents **claim** an item (set Owner + Status) before starting, respect `Blocked-by`, and mark `done` only after its **automated, reproducible behavior test passes green** ([`definition-of-done.md`](definition-of-done.md), [`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md)). Seeded from [`../roadmap/phases.md`](../roadmap/phases.md) + [`../roadmap/workstreams.md`](../roadmap/workstreams.md). -> **This board is the one working surface agents may edit** (claim items, update status, add -> items/escalations). The specs, ADRs, and roadmap are **not** editable by agents (R-TEST-06) — -> raise spec problems here as an `OQ-*` escalation for the human. Every "**Prove …**" task below is -> an automated conformance test (co-owned with **WS-TEST**), not a manual demo. +> The specs, ADRs, and roadmap are **not** editable by agents (R-TEST-06). Every "**Prove …**" task +> below is an automated conformance test (co-owned with **WS-TEST**), not a manual demo. **Status values:** `todo` · `in-progress` · `blocked` · `review` · `done` **Add new items** at the bottom of the relevant phase; keep IDs stable. diff --git a/decisions/README.md b/decisions/README.md index 596f5cc..20629a2 100644 --- a/decisions/README.md +++ b/decisions/README.md @@ -8,6 +8,14 @@ serves. **Do not edit an accepted decision in place** — supersede it with a ne > proposed ADR and escalate it, but only the **human operator/architect** approves/accepts or edits > ADRs. See [`../AGENTS.md` §3 and §11](../AGENTS.md). +**These ADRs are not the implementation's handoff log.** The dated files in the +[implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl)'s `decisions/` +directory record what each change did and how it was verified. ADRs here record locked technology and +pattern choices. + +**Partial supersession.** When a new ADR replaces only part of an accepted one, the older ADR stays +`Accepted` and its Status line names the clause superseded (the ADR-0013 precedent). + ## Process (read before proposing a change) 1. To change a decision, create a **new** ADR with the next number and set `Status: Proposed`, @@ -42,19 +50,28 @@ serves. **Do not edit an accepted decision in place** — supersede it with a ne | ADR | Title | Status | Serves | |---|---|---|---| -| [0001](adr-0001-kubernetes-distribution-k3s.md) | Kubernetes distribution: k3s | Accepted | R-PLAT-01 | +| [0001](adr-0001-kubernetes-distribution-k3s.md) | Kubernetes distribution: k3s | Accepted — dev provisioning clause superseded by 0017 | R-PLAT-01 | | [0002](adr-0002-cni-cilium.md) | CNI & network policy & ingress: Cilium + Gateway API | Accepted | R-NET-01..07, R-DNS-01 | -| [0003](adr-0003-storage-longhorn.md) | Distributed storage: Longhorn (+ bulk tier, Velero) | Accepted | R-STATE-01..04 | +| [0003](adr-0003-storage-longhorn.md) | Distributed storage: Longhorn (+ bulk tier, Velero) | Accepted — backup and bulk-backing clauses superseded by 0018 | R-STATE-01..04 | | [0004](adr-0004-gitops-flux.md) | GitOps engine: Flux CD | Accepted | R-GITOPS-01..03 | | [0005](adr-0005-vm-workloads-kubevirt.md) | VM workloads: KubeVirt + CDI | Accepted | R-PLAT-03 | | [0006](adr-0006-tls-cert-manager-letsencrypt.md) | TLS: cert-manager + Let's Encrypt (DNS-01) | Accepted | R-TLS-01..03 | | [0007](adr-0007-secrets-sops-age.md) | Secrets: SOPS + age | Accepted | R-SEC-01 | | [0008](adr-0008-vpn-egress-model.md) | VPN egress model: Gluetun sidecar + Cilium Egress Gateway | ~~Superseded by 0012~~ | R-NET-05/06 | -| [0009](adr-0009-automated-image-updates.md) | Automated updates: Flux image automation + Renovate | Accepted | R-UPD-01..03 | +| [0009](adr-0009-automated-image-updates.md) | Automated updates: Flux image automation + Renovate | ~~Superseded by 0023~~ | R-UPD-01..03 | | [0010](adr-0010-conformance-testing.md) | Conformance testing: automated, reproducible, behavior-based | Accepted | R-TEST-01..06 | | [0011](adr-0011-test-tooling-kyverno-chainsaw.md) | Conformance test tooling: Kyverno + Chainsaw | Accepted | R-TEST-01..05 | | [0012](adr-0012-vpn-egress-single-expressvpn-gateway.md) | VPN egress: single shared in-cluster ExpressVPN gateway | ~~Superseded by 0013~~ | R-NET-05/06/07 | | [0013](adr-0013-vpn-provider-pia-gluetun-wireguard.md) | VPN provider: PIA over OpenVPN-UDP + port forwarding | Accepted — provider/protocol only (client+steering superseded by 0015) | R-NET-05/06/07 | -| [0014](adr-0014-idp-keycloak.md) | Identity Provider: Keycloak (SSO via OIDC + forward-auth) | Accepted | R-IDP-01..06 | -| [0015](adr-0015-vpn-egress-steering-node-tunnel-cilium.md) | VPN egress steering: node-level tunnel + Cilium Egress Gateway (host kill-switch) | Accepted | R-NET-05/06/07 | -| [0016](adr-0016-public-ingress-cloudflare-tunnel.md) | Public ingress: Cloudflare Tunnel (outbound-only) instead of edge port-forwarding | Proposed | R-DNS-01..05, R-TLS-01/02, R-SEC-02/04 | +| [0014](adr-0014-idp-keycloak.md) | Identity Provider: Keycloak (SSO via OIDC + forward-auth) | Accepted — forward-auth enforcement point and app list superseded by 0022 | R-IDP-01..06 | +| [0015](adr-0015-vpn-egress-steering-node-tunnel-cilium.md) | VPN egress steering: node-level tunnel + Cilium Egress Gateway (host kill-switch) | Accepted — Kill-switch and Resilience bullets superseded by 0020 | R-NET-05/06/07 | +| [0016](adr-0016-public-ingress-cloudflare-tunnel.md) | Public ingress: Cloudflare Tunnel (outbound-only) instead of edge port-forwarding | Accepted | R-DNS-01..05, R-TLS-01/02, R-SEC-02/04 | +| [0017](adr-0017-single-environment-bare-metal-production.md) | Single environment: bare-metal production; dev retired | Accepted | R-PLAT-01, R-ENV-01/02/03, R-STATE-01/02 | +| [0018](adr-0018-storage-backup-truenas-r2.md) | Storage and backup: TrueNAS NFS bulk tier; Velero/kopia + Barman to Cloudflare R2 | Accepted | R-STATE-03, R-STATE-04 | +| [0019](adr-0019-postgresql-cloudnativepg.md) | PostgreSQL via CloudNativePG | Accepted | R-IDP-05, R-STATE-01, R-STATE-02, R-STATE-03 | +| [0020](adr-0020-vpn-egress-single-gateway-cilium-fail-closed.md) | VPN egress: single gateway node, Cilium-enforced fail-closed | Accepted | R-NET-05, R-NET-06 | +| [0021](adr-0021-inbound-remote-access-wireguard.md) | Inbound remote access: WireGuard on a hostNetwork DaemonSet | Accepted | remote operator access (no dedicated requirement ID); bounds R-NET-01, R-SEC-02, R-SEC-04 | +| [0022](adr-0022-forward-auth-per-app-oauth2-proxy-sidecars.md) | Forward-auth via per-app oauth2-proxy sidecars | Accepted | R-IDP-02, R-IDP-06 | +| [0023](adr-0023-automated-updates-renovate.md) | Automated updates: Renovate for images and charts, every change PR-gated | Accepted | R-UPD-01, R-UPD-02, R-UPD-03, R-TEST-03 | +| [0024](adr-0024-delivery-authority-ci-gate.md) | Delivery authority: the CI gate is the approver; sensitive-path review; attributed admin override | Accepted | R-TEST-03, R-TEST-06, R-GITOPS-01, R-SEC-02 | +| [0025](adr-0025-in-cluster-ai-program-manager.md) | In-cluster AI Program Manager (pm-agent) | Accepted | R-GITOPS-01, R-TEST-03, R-TEST-06, R-SEC-02 | diff --git a/decisions/adr-0001-kubernetes-distribution-k3s.md b/decisions/adr-0001-kubernetes-distribution-k3s.md index 202a666..a00adcb 100644 --- a/decisions/adr-0001-kubernetes-distribution-k3s.md +++ b/decisions/adr-0001-kubernetes-distribution-k3s.md @@ -1,5 +1,5 @@ # ADR-0001 — Kubernetes distribution: k3s -- Status: Accepted +- Status: Accepted — k3s, embedded etcd and its disables stand (dev provisioning clause superseded by [ADR-0017](adr-0017-single-environment-bare-metal-production.md)) - Date: 2026-08-15 - Serves: R-PLAT-01, R-PLAT-02, R-ENV-01/02/03 - Supersedes: — diff --git a/decisions/adr-0003-storage-longhorn.md b/decisions/adr-0003-storage-longhorn.md index f9414ee..f6b2bb8 100644 --- a/decisions/adr-0003-storage-longhorn.md +++ b/decisions/adr-0003-storage-longhorn.md @@ -1,5 +1,5 @@ # ADR-0003 — Distributed storage: Longhorn (+ bulk tier + Velero) -- Status: Accepted +- Status: Accepted — Longhorn stands (backup and bulk-backing clauses superseded by [ADR-0018](adr-0018-storage-backup-truenas-r2.md)) - Date: 2026-08-15 - Serves: R-STATE-01, R-STATE-02, R-STATE-03, R-STATE-04 - Supersedes: — diff --git a/decisions/adr-0009-automated-image-updates.md b/decisions/adr-0009-automated-image-updates.md index 09132b1..a72ed1d 100644 --- a/decisions/adr-0009-automated-image-updates.md +++ b/decisions/adr-0009-automated-image-updates.md @@ -1,5 +1,5 @@ # ADR-0009 — Automated updates: Flux image automation + Renovate -- Status: Accepted +- Status: **Superseded by [ADR-0023](adr-0023-automated-updates-renovate.md)** (2026-11-08) - Date: 2026-08-15 - Serves: R-UPD-01, R-UPD-02, R-UPD-03 - Supersedes: — diff --git a/decisions/adr-0014-idp-keycloak.md b/decisions/adr-0014-idp-keycloak.md index 3763dfe..9bdde11 100644 --- a/decisions/adr-0014-idp-keycloak.md +++ b/decisions/adr-0014-idp-keycloak.md @@ -1,5 +1,5 @@ # ADR-0014 — Identity Provider: Keycloak (SSO via OIDC + forward-auth) -- Status: Accepted +- Status: Accepted — Keycloak, native OIDC, groups and MFA stand (forward-auth enforcement point and app list superseded by [ADR-0022](adr-0022-forward-auth-per-app-oauth2-proxy-sidecars.md)) - Date: 2026-08-15 - Serves: R-IDP-01, R-IDP-02, R-IDP-03, R-IDP-04, R-IDP-05, R-IDP-06 - Supersedes: — diff --git a/decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md b/decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md index 42f6930..c1003b8 100644 --- a/decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md +++ b/decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md @@ -1,5 +1,5 @@ # ADR-0015 — VPN egress steering: node-level tunnel + Cilium Egress Gateway (host kill-switch) -- Status: Accepted +- Status: Accepted — tunnel placement, steering and the four invariants stand (Kill-switch and Resilience bullets, and the mechanism named in invariant 3, superseded by [ADR-0020](adr-0020-vpn-egress-single-gateway-cilium-fail-closed.md)) - Date: 2026-08-23 - Serves: R-NET-05, R-NET-06, R-NET-07 - Supersedes (egress-steering + tunnel placement of): [ADR-0013](adr-0013-vpn-provider-pia-gluetun-wireguard.md) diff --git a/decisions/adr-0016-public-ingress-cloudflare-tunnel.md b/decisions/adr-0016-public-ingress-cloudflare-tunnel.md index 5785582..08a66be 100644 --- a/decisions/adr-0016-public-ingress-cloudflare-tunnel.md +++ b/decisions/adr-0016-public-ingress-cloudflare-tunnel.md @@ -1,6 +1,7 @@ # ADR-0016 — Public ingress: Cloudflare Tunnel (outbound-only) instead of edge port-forwarding -- Status: Proposed (decided by the operator 2026-11-08; set Accepted on merge) +- Status: Accepted - Date: 2026-11-08 +- Accepted by: operator ruling of 2026-11-08 that the implementation is the accepted state; effective on merge. - Serves: R-DNS-01, R-DNS-02, R-DNS-03, R-DNS-04, R-DNS-05, R-TLS-01, R-TLS-02, R-SEC-02, R-SEC-04 - Supersedes: — (amends the **ingress** part of the OQ-1 resolution, "edge does NAT + port-forward only", in [00](../docs/00-vision-and-scope.md), [03](../docs/03-networking-isolation-and-egress.md) @@ -28,9 +29,10 @@ public web apps has to come from the public answer itself working from the LAN. - **cloudflared runs in-cluster** (a Flux-managed workload) and holds an **outbound-only** connection to Cloudflare. No inbound connection to the cluster is needed for public web traffic. -- **Public records** for `exposure: public` services are **proxied CNAMEs to - `.cfargotunnel.com`**, managed by **ExternalDNS** from declared service intent - (R-DNS-02). No record is hand-edited. +- **Public records** for services declared public (catalog `exposure: public`, or a registered + platform route: `auth`, Grafana, Crafty's web UI, the apex and its `/runbooks`) are **proxied + CNAMEs to `.cfargotunnel.com`**, managed by **ExternalDNS** from declared service + intent (R-DNS-02). No record is hand-edited. - **The edge router's 80/443 forwards are closed** after cutover. - **Hostnames are unchanged:** `.home.keithjackson.dev` and the apex `home.keithjackson.dev` (R-DNS-01, R-C2-01). @@ -40,19 +42,23 @@ public web apps has to come from the public answer itself working from the LAN. - **Origin TLS stays at the Gateway** with the cert-manager Let's Encrypt wildcard (ADR-0006). cloudflared connects to the Gateway over HTTPS and **validates the origin certificate**, using the service hostname as SNI. It does not skip verification (`noTLSVerify` is not used). -- **Only `exposure: public` services have a tunnel route.** The tunnel's ingress rules end in a - **catch-all `http_status:404`**, so any other hostname presented at the tunnel is refused - (R-DNS-05). +- **The tunnel's ingress rules are a deny-list plus a zone wildcard.** Each LAN/VPN-only hostname + maps to `http_status:404`; the apex and `*.home.keithjackson.dev` then go to the Gateway; a + **catch-all `http_status:404`** closes the list for hosts outside the zone (R-DNS-05). A static + conformance check keeps the deny-list and the set of unpublished routes identical in both + directions. - **The Kubernetes API is never reachable through the tunnel**, and is never published by any other path. It is LAN/VPN-only (R-SEC-04). **Never published (LAN/VPN-only, `exposure: internal`):** jackett, pihole, semaphore, qbittorrent, radarr, sonarr, **headlamp** (a front end to the Kubernetes API), and the non-HTTP L4 services -**unifi** and **crafty-mc** (Minecraft). They get no public DNS record and no tunnel route. +**unifi** and **crafty-mc** (Minecraft). The tunnel refuses all of them with a 404. None has a +public DNS record, except that unifi and crafty-mc hold **DNS-only records at their LAN addresses**, +which are never proxied or tunnelled. **Plex stays off the tunnel.** Cloudflare's terms restrict proxying video, so Plex remote streaming keeps using **plex.tv + WAN tcp/32400** (an edge forward that stays). The `plex.home` web UI becomes -**LAN-only**: no tunnel route and no proxied record. +**LAN-only**: refused at the tunnel, with no proxied record. **Unchanged:** - **WireGuard remote access**: udp/51820, reached through the DNS-only (unproxied) `wan.` record, @@ -88,7 +94,7 @@ keeps using **plex.tv + WAN tcp/32400** (an edge forward that stays). The `plex. ## Consequences - **Positive:** LAN clients reach public web apps correctly whichever resolver answers. No NAT hairpin is needed. Inbound 80/443 is no longer open on the WAN IP. Publishing is an explicit, - declared, per-hostname act (tunnel route + record), and anything else 404s at the edge. + declared, per-hostname act (a public record), and LAN/VPN-only names 404 at the tunnel. - **Dependence on Cloudflare.** A Cloudflare or tunnel outage takes public access down. The LAN keeps working through Pi-hole's local records for gitea, nextcloud and keycloak/auth and the internal-only names. Public apps without a local record are unreachable from the LAN during such an diff --git a/decisions/adr-0017-single-environment-bare-metal-production.md b/decisions/adr-0017-single-environment-bare-metal-production.md new file mode 100644 index 0000000..24968bf --- /dev/null +++ b/decisions/adr-0017-single-environment-bare-metal-production.md @@ -0,0 +1,59 @@ +# ADR-0017 — Single environment: bare-metal production; dev retired +- Status: Accepted +- Date: 2026-11-08 +- Serves: R-PLAT-01, R-ENV-01/02/03, R-STATE-01/02 +- Supersedes: ADR-0001 provisioning clause only ("Nodes are provisioned by OpenTofu (libvirt, dev) / + bare-metal imaging (prod) + cloud-init, and joined/configured by Ansible"). The rest of ADR-0001 + stands: k3s, 3-node HA on embedded etcd, and the bundled components disabled at install. +- Accepted by: operator ruling of 2026-11-08 that the implementation is the accepted state; effective on merge. + +## Context +ADR-0001 and the original specs assumed two environments: a dev/test cluster of local libvirt KVMs, +provisioned by OpenTofu and cloud-init, and a prod cluster of 3 mini-PCs, kept design-only until a +later phase. + +Delivery did not stay on that path. The workstation host ran out of RAM and disk, so dev moved to VMs +on a Proxmox host. The operator opened Phase 7 on 2026-08-31, and prod was built on 3 HP EliteDesk +Mini bare-metal nodes. On 2026-09-12/13 prod took over the apex `home.keithjackson.dev`, and the dev +VMs were stopped. The storage pools of the host those VMs ran on then moved to a standalone NAS, and +the migration plan retires that host. + +Keeping "dev/test" in the specification now describes something that does not exist. It also leaves +R-ENV-01 and R-ENV-03 unsatisfiable as written. + +## Decision +The platform has **one live environment: prod, 3× mini-PC bare metal**. + +- **Create** is an imaged, checklist-driven operator step: there is no hypervisor API to call on + bare metal. The checklist is `infra/PROD-PROVISIONING.md` in the + [implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl). +- **Baseline, cluster-join and manage** are code: Ansible owns all of them, including the first-boot + baseline that cloud-init used to carry. Cloud-init is not part of node provisioning. +- OpenTofu is not part of node provisioning. +- The Kubernetes API has one HA endpoint, a **kube-vip** virtual IP across the 3 servers. The address + a joining node uses during a cold bootstrap is deliberately a different one, because the virtual IP + does not exist until a cluster does. +- GitOps configuration stays **one base plus an environment-scoped overlay**. No second environment + exists, but the separation is kept, so a future environment needs overrides only. +- With no staging surface, the safeguard is the merge gate: every change reaches prod through the PR + gate and Flux. Destructive or fault-injecting tests against prod are deliberate, scheduled + exercises. + +## Alternatives considered +- **Keep a dev cluster on Proxmox.** Rejected: the migration plan retires that host, and dev does + not move to the NAS, which is storage only. +- **Rebuild dev as local KVMs on the workstation.** Rejected: that is the path already abandoned for + lack of RAM and disk. +- **Drive bare metal through Cluster API or PXE automation.** Deferred: imaging plus Ansible is + sufficient and lower-complexity for 3 nodes. + +## Consequences +- The spec stops gating work on a dev cluster; R-ENV-01/02/03 and R-PLAT-01 are reworded to match. +- Conformance runs against prod. There is no separate cluster on which to absorb a bad change, so the + static gate and the discipline around fault injection carry more weight. +- Node-rebuild is exercised by hand rather than by a destroy/rebuild pipeline; a node is reimaged and + rejoined through the checklist and Ansible. +- The `clusters/dev` overlay, `infra/tofu` and dev defaults remaining in the implementation repo are + inert. Removing them is implementation cleanup, and a conformance check that pins them must change + with them. +- ADR-0001's note that prod must not go below 3 servers stands, and it has no dev counterpart. diff --git a/decisions/adr-0018-storage-backup-truenas-r2.md b/decisions/adr-0018-storage-backup-truenas-r2.md new file mode 100644 index 0000000..800141e --- /dev/null +++ b/decisions/adr-0018-storage-backup-truenas-r2.md @@ -0,0 +1,72 @@ +# ADR-0018 — Storage and backup: TrueNAS NFS bulk tier; Velero/kopia + Barman to Cloudflare R2; no Longhorn backup target +- Status: Accepted +- Date: 2026-11-08 +- Accepted by: operator ruling of 2026-11-08 that the implementation is the accepted state; effective on merge. +- Serves: R-STATE-03, R-STATE-04 +- Supersedes: [ADR-0003](adr-0003-storage-longhorn.md) — the backup and bulk-backing clauses only. + **Longhorn as the replicated block store, 3 replicas by default, and Plex's metadata on Longhorn + all stand.** + +## Context +ADR-0003 left two things open or conditional. The bulk tier's backing was "decided under OQ-2", and +the DR path was "Velero (with Longhorn backups)" to an unnamed off-cluster target (OQ-BKP). + +Operating the platform settled both: +- The S3 endpoint on the old Proxmox host could not serve prod. It was unreachable across the network + boundary, measured from a pod inside prod, and that host shared a failure domain with the VMs it + ran. With WAL archiving on and no working target, Postgres refused to recycle WAL and database + volumes filled, so the missing target was an availability incident as well as a DR gap. +- A Longhorn backup target was escalated as a blocker for the DR drill. The owner ruled on 2026-09-13 + that Velero + kopia to R2 is the volume backup and that no Longhorn target will be configured. +- The Proxmox host was decommissioned and its ZFS pools physically moved to a storage-only TrueNAS + appliance on the nodes' own LAN. The multi-terabyte library was already there, so the tier attaches + existing data and creates none. + +Uniform file-system backup also collides with NFS. Opt-out backup would read the whole media library +over NFS every night and upload it, once per consuming namespace. + +## Decision +**Bulk tier.** Bulk media lives on a standalone **TrueNAS** appliance, storage only, consumed over +**NFS**. A catalog `class: bulk` entry with a named export renders a statically bound volume, and the +server address is a per-environment value. There is no dynamic provisioning and no NFS CSI driver. + +**Off-cluster target.** **Cloudflare R2**, one bucket, with Velero and CloudNativePG/Barman under +disjoint prefixes so that neither's retention can reach the other's objects. + +**Volume and resource backup.** **Velero with kopia file-system backup**, opt-out by default so no +volume is forgotten. No CSI snapshot data-mover and no Volume Snapshot Location. + +**No Longhorn backup target.** Longhorn snapshots remain an in-cluster mechanism only. + +**Database backup.** Continuous WAL and base backups through CloudNativePG/Barman ([ADR-0019](adr-0019-postgresql-cloudnativepg.md)). + +**Exclusions by property.** A volume policy skips NFS and emptyDir volumes by source type, so the rule +covers volumes that do not exist yet. The bulk tier is therefore **outside platform backup**. +Reconstructable data (monitoring's metrics and logs, the UniFi VM disk) is captured as Kubernetes +resources only. + +## Alternatives considered +- **MinIO/versitygw on the Proxmox host** — unreachable from prod and in the same failure domain as + what it protected; the host no longer exists. +- **MinIO on the TrueNAS** — planned during the host migration; not adopted. It would keep backups on + site beside the cluster, where R2 is off-site with free egress, so restore rehearsals carry no cost. +- **Longhorn backup target as well as, or instead of, Velero** — a second restore path, declined by + the owner because kopia to R2 already provides a genuine off-cluster copy. +- **Excluding media namespaces from Velero** — rejected: those namespaces also hold irreplaceable + Longhorn PVCs. +- **Per-pod exclusion annotations on NFS volumes** — rejected: fails open. The next NFS volume is + uploaded in full until someone remembers to annotate it. +- **Dynamic NFS provisioning (the `bulk` StorageClass ADR-0003 implied)** — unnecessary: every bulk + volume attaches a dataset that already holds data. + +## Consequences +- The restore path is single. A failure in Velero's path to R2 has no Longhorn-side fallback; revisit + if that path proves unreliable, not on principle. +- R2 is not a full S3 implementation and rejects some behaviours other stores tolerate. Compatibility + settings live in the implementation repo. +- A bound NFS volume's source is immutable, so moving the bulk server means recreating the volumes + rather than reconciling them. +- The bulk tier's durability is the NAS's own. Whether any of it is irreplaceable, notably photos, is + an open operator decision. +- Detail: `docs/BULK-NFS-STORAGE.md` and `platform/velero/` in the + [implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl). diff --git a/decisions/adr-0019-postgresql-cloudnativepg.md b/decisions/adr-0019-postgresql-cloudnativepg.md new file mode 100644 index 0000000..c47ea5e --- /dev/null +++ b/decisions/adr-0019-postgresql-cloudnativepg.md @@ -0,0 +1,50 @@ +# ADR-0019 — PostgreSQL via CloudNativePG +- Status: Accepted +- Date: 2026-11-08 +- Accepted by: operator ruling of 2026-11-08 that the implementation is the accepted state; effective on merge. +- Serves: R-IDP-05, R-STATE-01, R-STATE-02, R-STATE-03 +- Supersedes: — + +## Context +[ADR-0014](adr-0014-idp-keycloak.md) requires Keycloak's PostgreSQL to live on replicated Longhorn, +be DR-critical and be backed up off-cluster; it does not say how PostgreSQL is run. Nextcloud and +Semaphore need PostgreSQL too. + +The first design used the Bitnami Keycloak chart's bundled PostgreSQL. Bitnami's free images were +purged or relocated, so a deploy on stock tags fails to pull. + +Single-instance PostgreSQL also proved insufficient on its own terms. Longhorn replication preserves +the data through a node loss but not the service. With one instance each, all three primaries drifted +onto the same node, so one node failure would have taken Nextcloud, Semaphore and SSO down together. + +## Decision +Run every platform PostgreSQL database under the **CloudNativePG** operator, one cluster per database: +- **Three instances** with hard anti-affinity, one per node, each on the replicated `longhorn` class. +- **Off-cluster recovery** through Barman: continuous WAL archiving plus a daily base backup to the + object store ([ADR-0018](adr-0018-storage-backup-truenas-r2.md)), retained 30 days. +- Primary changes roll by **switchover** rather than an in-place restart. +- Standbys are added only behind a working archive, because they replicate corruption and are not a + backup. + +Scope: services that need PostgreSQL. An application with a small embedded datastore (Gitea's SQLite, +for example) keeps it on its Longhorn PVC. + +## Alternatives considered +- **Bitnami-bundled PostgreSQL** — images no longer dependably available from the free registry. +- **Single-instance PostgreSQL on Longhorn** — keeps the data through a node loss but loses the + service until the pod reschedules, and gives no point-in-time recovery by itself. +- **Relying on file-system volume backup alone** — no point-in-time recovery; a kopia copy of a live + data directory is the weaker path. It is retained as a secondary copy, not as the database backup. +- **PostgreSQL embedded in each application's chart** — one lifecycle per application, with backup + and failover solved separately each time. + +## Consequences +- Failover takes seconds, and every database has one declarative point-in-time recovery path. +- WAL archiving is an availability dependency. If archiving fails, PostgreSQL correctly refuses to + recycle WAL and the volume fills, so archive health is alerted on as an availability signal. +- Storage cost multiplies: three instances on three-way replicated volumes means nine copies of each + database. Accepted, because the databases are small. +- The operator's custom resources need CRDs before they apply, which fits the platform's + controllers-then-configs ordering. +- Detail: `platform/cnpg/` and the database manifests in the + [implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl). diff --git a/decisions/adr-0020-vpn-egress-single-gateway-cilium-fail-closed.md b/decisions/adr-0020-vpn-egress-single-gateway-cilium-fail-closed.md new file mode 100644 index 0000000..5417488 --- /dev/null +++ b/decisions/adr-0020-vpn-egress-single-gateway-cilium-fail-closed.md @@ -0,0 +1,61 @@ +# ADR-0020 — VPN egress: single gateway node, Cilium-enforced fail-closed +- Status: Accepted +- Date: 2026-11-08 +- Accepted by: operator ruling of 2026-11-08 that the implementation is the accepted state; effective on merge. +- Serves: R-NET-05, R-NET-06 +- Supersedes (the **Kill-switch** and **Resilience** bullets of): [ADR-0015](adr-0015-vpn-egress-steering-node-tunnel-cilium.md) + — whose node-level tunnel placement, Cilium Egress Gateway steering and four invariants stand. + +## Context +ADR-0015 made a host nftables rule the fail-closed backstop for `vpn`-class traffic, and relied on two +gateway nodes plus "Cilium Egress Gateway HA" for failover. Building and measuring it disproved both. + +- **nftables never sees the traffic.** Egress Gateway requires eBPF masquerade, and the datapath + redirects steered packets straight to the egress device without passing netfilter. A counters-only + table on the gateway counted zero steered packets on the forward and postrouting hooks while a + `vpn`-class pod egressed, with the same hooks counting WAN and node-originated traffic. +- **Cilium is what fails closed.** With the tunnel stopped and during a restart, packet captures on + both the WAN and the tunnel device showed nothing leaving either one. +- **Gateway HA does not exist in Cilium OSS.** The policy's `egressGateway` field resolves a matching + set to the first node in name order. The plural `egressGateways` field shards endpoints across nodes + rather than providing standby: losing a node blackholes its shard and re-shards the rest, and each + shard exits from a different public IP. Upstream: cilium/cilium#30157 and #18230. +- **Cilium caches the tunnel interface.** It resolves the interface once. A reconnect re-creates the + device under a new index, and `vpn`-class egress is then silently blackholed. + +## Decision +- **Exactly one egress-gateway node.** One node carries the gateway label. There is no failover, and + a second labeled node is never added as a "standby". +- **Fail-closed is enforced by Cilium's egress datapath.** R-NET-06 is unchanged: tunnel down means no + `vpn`-class packet reaches the plain internet. The guarantee is proven by the `vpn-killswitch` + conformance test, against Cilium's behaviour. +- **The host nftables table is a narrow backstop.** It stays, with counters, for what it can see: + node-originated traffic sourced from a tunnel address. +- **The gateway node signals every reconnect to Cilium**, through a node label that makes Cilium + re-resolve the interface. A node never writes to the Flux-managed policy. + +### What stands from ADR-0015 +Tunnel placement (host OpenVPN service, split-tunnel, Ansible-managed), steering (a +`CiliumEgressGatewayPolicy` selecting `egress-class=vpn`), host-side credentials, and all four +invariants. Invariant 3's scope stands: only `vpn`-class traffic is dropped when the tunnel is down. +The enforcing mechanism it names is the one this ADR replaces. + +## Alternatives considered +- **Label a second node as standby** — rejected: it carries nothing as a standby, and because selection + is by name order it can silently take the role, the tunnel and the public egress IP. +- **Plural `egressGateways`** — rejected: sharding, not failover; a node loss still blackholes a share, + breaks live connections, and splits the service across exit IPs. +- **Make the Cilium policy stamp a mark for a host firewall to match** — not implementable: Cilium OSS + has no supported way to mark this traffic, and it would not reach a host firewall hook anyway. +- **Keep ADR-0015's text and treat nftables as authoritative** — rejected: it would document a control + that measurably does nothing for the traffic it claims to protect. + +## Consequences +- The gateway node is a **single point of failure for `vpn`-class egress**. Its loss is an outage with + no leak, the direction R-NET-06 requires. +- **Labeling is a safety-relevant act.** Labeling another node moves the tunnel and the public egress + IP silently, with no event or alert. +- The re-steer path is alerted (a reconnect not reflected in the label; a gateway with no label). + Cilium's egress map is not exposed as a metric, so "the label changed and Cilium ignored it" is + not detected by these alerts. +- If Cilium OSS gains real egress-gateway failover, this ADR can be reopened by a superseding ADR. diff --git a/decisions/adr-0021-inbound-remote-access-wireguard.md b/decisions/adr-0021-inbound-remote-access-wireguard.md new file mode 100644 index 0000000..fb6ee4e --- /dev/null +++ b/decisions/adr-0021-inbound-remote-access-wireguard.md @@ -0,0 +1,67 @@ +# ADR-0021 — Inbound remote access: WireGuard on a hostNetwork DaemonSet +- Status: Accepted +- Date: 2026-11-08 +- Accepted by: operator ruling of 2026-11-08 that the implementation is the accepted state; effective on merge. +- Serves: remote operator access (no dedicated requirement ID); bounds R-NET-01, R-SEC-02, R-SEC-04 +- Supersedes: — + +## Context +The operator's Android, Linux, Windows and other devices need to reach the home network from anywhere +as though on the LAN. Several services are LAN/VPN-only by design (ADR-0016), and the Kubernetes API is +LAN/VPN-only (R-SEC-04), so the "VPN" in that phrase needs a defined mechanism. This is unrelated to +`vpn`-class egress (ADR-0015, ADR-0020), which sends pods out through a provider tunnel. + +Two facts about this cluster shape the design: +- Cilium decides what a packet may reach from the **security identity of its source**. Measured: a + request to the shared Gateway from inside a pod is refused for every Gateway hostname. A + pod-networked VPN server would give every client that pod's identity: a healthy tunnel on which + every web UI is refused. +- The SOHO edge has no automation and cannot hairpin NAT. An inbound forward is a manual step that names + one node address. + +## Decision +**Remote access is WireGuard on udp/51820, run as a Flux-managed DaemonSet in the host network +namespace on every node.** + +- **Host network** so that clients never carry a pod identity. +- **DaemonSet** so that every node holds an identical warm listener, and losing the forwarded node is a + one-line router change with no cluster work and no new client profiles. +- **Authorization is a reviewable git diff:** peer public keys are in plain text in git, so granting or + revoking a device is a reviewed change that rolls the listeners. The server's private key is + SOPS-encrypted (ADR-0007), and joined with the peer list only in memory at pod start. +- **Reach equals a LAN machine:** the whole LAN, the Gateway, Pi-hole, and the internet through the + house WAN. Clients are `internet` class, never steered to the provider tunnel. +- **Endpoint:** the DNS-only `wan.` record (ADR-0016). A client inside the house cannot use it, as there + is no hairpin, and uses a node's LAN address instead. + +**This is a deliberate exception to default-deny (R-NET-01).** A host-network workload has no Cilium +endpoint, so no network policy applies to it. The exception is scoped to this platform component; +application namespaces remain barred from the host network by admission policy. The boundary is +WireGuard's peer authentication: a packet from an unknown key gets no reply, so the port is +indistinguishable from a closed one. + +## Alternatives considered +- **OpenVPN** — rejected: a CA, per-device certificates and a revocation list are long-lived state to + protect and rotate. +- **IKEv2/IPsec** — rejected: painful to containerize and brittle behind double NAT. +- **Pod-networked WireGuard** — rejected: clients inherit the pod's identity and every Gateway hostname + refuses them, while the tunnel reports healthy. +- **A LoadBalancer VIP for the listener** — rejected: adds an L2-announcement lease, SNAT and a + node-to-node hop to the one path whose failure means the operator is locked out while away. +- **Single-replica Deployment** — rejected: a reschedule silently invalidates the router forward. +- **Carry it over the Cloudflare Tunnel** — not possible: the tunnel carries HTTP(S), and WireGuard is + UDP to a node. +- **Encrypt the peer list / add preshared keys** — rejected: hides the change a reviewer most needs to + see, and a preshared key would be a further secret for a threat not in the model. + +## Consequences +- **No network policy constrains this workload.** Peer authentication and the single forwarded port are + the controls. +- **An enrolled device is as privileged as a machine in the house**, including reach to the nodes' + control-plane ports. A stolen device is that device until revoked, so revocation must be prompt. + Narrowing is possible per profile (split tunnel) or by host firewall rules in the listener. +- The WAN IP stays public for this port (ADR-0016); the forward is a manual edge step. +- The tunnel carries IPv4 only. Client profiles default to a conservative MTU, because a client's own + path can be narrower than the server's. +- No per-peer metrics exist; a WireGuard exporter would need a second privileged container beside the + internet-facing tunnel. diff --git a/decisions/adr-0022-forward-auth-per-app-oauth2-proxy-sidecars.md b/decisions/adr-0022-forward-auth-per-app-oauth2-proxy-sidecars.md new file mode 100644 index 0000000..24a5e75 --- /dev/null +++ b/decisions/adr-0022-forward-auth-per-app-oauth2-proxy-sidecars.md @@ -0,0 +1,69 @@ +# ADR-0022 — Forward-auth via per-app oauth2-proxy sidecars +- Status: Accepted +- Date: 2026-11-08 +- Accepted by: operator ruling of 2026-11-08 that the implementation is the accepted state; effective on merge. +- Serves: R-IDP-02, R-IDP-06 +- Supersedes: [ADR-0014](adr-0014-idp-keycloak.md) — **enforcement point and app list only**: the + "at the Gateway" clause and the list of forward-auth apps in its Decision. Everything else in + ADR-0014 stands. + +## Context +ADR-0014 put a single oauth2-proxy at the Gateway as an external-authorization check in front of +every app without usable OIDC. That design needs a per-route authorization hook on the shared +Gateway, and the platform's Gateway implementation does not offer a usable one: + +- Cilium generates and owns the shared Gateway's Envoy configuration, regenerating it on every + route change, with no supported way to inject an authorization filter into it. +- Cilium's native Gateway API `ExternalAuth` filter needs the **experimental** Gateway API CRD + channel, a cluster-wide change, and shipped with a fail-**open** bug in the very property + R-IDP-02 forbids. It was evaluated and deferred. +- Without a gate, the bare UIs stayed reachable unauthenticated on the LAN. + +Crafty's web UI was already gated by a co-located oauth2-proxy, which proved the pattern on this +platform. Detail: `docs/plans/cilium-native-externalauth-evaluation.md` in the +[implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl). + +## Decision +Forward-auth is enforced by a **per-app oauth2-proxy running as a reverse-proxy sidecar in the +app's pod**, in the data path, rather than by a shared check at the Gateway. + +- The app's route targets the sidecar's port. Network policy opens **only** that port to the + Gateway, so the bare UI port is unreachable from it and a failed or absent gate fails closed. +- Each gate is its own instance: its own IdP client, its own SOPS-encrypted credentials, and the + authorized groups declared in the service's `auth` block. +- **Session cookies are per-instance**: a unique cookie name per gate, scoped to the app's own + host. Instances hold different cookie secrets, so a shared name on a zone-wide domain makes them + evict one another's sessions. +- An app MAY declare the path prefixes of its programmatic API, so unauthenticated calls there are + refused rather than redirected to an HTML login page. +- The catalog renderer generates the sidecar for every `forward-auth` service with an exposed UI. + +**App list (replaces ADR-0014's).** Gated this way: PiHole, Sonarr, Radarr, Jackett, qBittorrent +and the Crafty web UI. Homepage has no usable OIDC and requires a gate under R-IDP-02. +Prometheus, Alertmanager and Hubble UI are not exposed, so no gate applies; any future exposure +MUST be gated. + +## Alternatives considered +- **Self-authored Envoy listener with an authorization filter on its own VIP** — works, and was + proven against the live proxy, but needs hand-maintained TLS and routes per app, a second ingress + address and DNS repointing. Rejected; retained in the implementation repo as an unwired template. +- **Cilium native `ExternalAuth`** — the cleanest shape and the long-term target, one per-route + filter and no per-app proxies. Deferred: experimental CRDs, an immature security-critical filter, + thin documentation. A future ADR may adopt it and retire the sidecars. + +## Consequences +- Enforcement is Gateway-API-native (a backend swap) and does not depend on the Gateway's generated + configuration. +- More moving parts: one proxy, one IdP client and one credential set per gated app. Adding a gated + app needs all three. +- Sessions are per-host. A live Keycloak session makes the bounce silent, so SSO still holds, but a + browser performs one redirect per app. +- An expired session answers an unauthenticated XHR with a redirect that resolves as a successful + HTML response, which is why API paths are declarable and why each gate exposes metrics for + scraping ([10](../docs/10-observability-and-alerting.md)). +- Crafty is platform-managed outside the renderer, so its gate is hand-maintained and must follow + the same per-instance cookie rule. +- A UI whose pod the renderer does not generate, such as Homepage, gets a gate only by explicit + work ([known gaps](../docs/16-known-gaps.md)). +- [15](../docs/15-identity-and-access.md) carries the resulting design; ADR-0014's text is not + edited. diff --git a/decisions/adr-0023-automated-updates-renovate.md b/decisions/adr-0023-automated-updates-renovate.md new file mode 100644 index 0000000..6e39f75 --- /dev/null +++ b/decisions/adr-0023-automated-updates-renovate.md @@ -0,0 +1,64 @@ +# ADR-0023 — Automated updates: Renovate for images and charts, every change PR-gated +- Status: Accepted +- Date: 2026-11-08 +- Serves: R-UPD-01, R-UPD-02, R-UPD-03, R-TEST-03 +- Supersedes: [ADR-0009](adr-0009-automated-image-updates.md) (the VM guest mechanism stands and is + restated below) +- Accepted by: operator ruling of 2026-11-08 that the implementation is the accepted state; effective on merge. + +## Context +ADR-0009 split updates by artifact: Flux image-update-automation for container image tags, Renovate +for charts and manifests. The image half could not be delivered on this platform's terms: + +1. **It bypasses the merge gate.** Image automation commits and pushes rather than opening a pull + request. Aimed at `main` it writes past the PR flow; aimed at a side branch, nothing closes the + loop. +2. **It cannot write here.** `main` is protected against direct pushes for every account, and Flux's + source is anonymous and read-only. Enabling write-back means an in-cluster git **write** credential + for the repository that defines the cluster, a far larger blast radius than a CI-scoped token. +3. **Renovate already opens PRs**, so every change it proposes goes through the gate with no extra + machinery. + +The controller was authored but never enabled, and no image-policy objects ever existed in the +cluster. Meanwhile plain `image:` tags were maintained by nothing at all, and pins drifted many minor +versions behind. This ADR closes a path that was never taken and changes no running state. + +## Decision +**Renovate owns container images, Helm charts and manifest versions.** It runs on a schedule as a +Gitea Actions job and every change it proposes is a machine-authored pull request through the merge +gate. + +- **Minor, patch and digest** updates auto-merge on a green gate. +- **Majors never auto-merge**; they wait for human review of the machine-authored PR. +- **Operator-designated blast-radius components never auto-merge**, not even a patch, because a green + static gate says nothing about whether a bump breaks them, or because they are the merge gate + itself. The list and each reason are data in the Renovate configuration. +- **The platform's own CI-built images** are pinned by digest and follow a moving tag through a + marker comment; Renovate proposes the digest bump. +- **Per-service policy** is declared in git. Renovate does not read the ServiceDefinition, so + ceilings and cadences take effect through per-package rules in its configuration. +- **VM guests → in-guest auto-update**, declared in cloud-init, unchanged from ADR-0009. + +Flux image-update-automation stays disabled. Flux itself stands +([ADR-0004](adr-0004-gitops-flux.md)); its native image automation is no longer a load-bearing part +of that choice, and native SOPS decryption and reconciliation remain so. + +## Alternatives considered +- **Flux image automation writing to `main`** — bypasses the required gate; needs an in-cluster write + credential. Rejected. +- **Flux image automation writing to a side branch** — nothing opens or merges the PR; bumps accumulate + unseen. Rejected. +- **Auto-merge everything on green** — the gate is static for these changes and cannot see whether a + bump breaks the storage plane, the IdP or the gate itself. Rejected for the blast-radius set. +- **Digest-tracking a mutable stable tag** — loses version visibility and semver guardrails; ADR-0009 + already held this as an escape hatch only. Not adopted. + +## Consequences +- Every update is auditable as a PR and is held to the same gate as a human change (R-TEST-03). +- R-UPD-02 is amended to match: allowed classes are hands-off; majors and blast-radius components MAY + wait for human review of the machine-authored change. No human or AI authors a version bump. +- Update-failure signals and verification written around the Flux controller must move to Renovate's + path ([known gaps](../docs/16-known-gaps.md)). +- Renovate is a CI job with a scoped token; its own version never auto-merges, because nothing + re-validates its configuration against a new release before it takes effect. +- The update cadence is bounded by CI capacity, so PR concurrency and hourly rates are capped. diff --git a/decisions/adr-0024-delivery-authority-ci-gate.md b/decisions/adr-0024-delivery-authority-ci-gate.md new file mode 100644 index 0000000..f754012 --- /dev/null +++ b/decisions/adr-0024-delivery-authority-ci-gate.md @@ -0,0 +1,55 @@ +# ADR-0024 — Delivery authority: the CI gate is the approver; sensitive-path review; attributed admin override +- Status: Accepted +- Date: 2026-11-08 +- Serves: R-TEST-03, R-TEST-06, R-GITOPS-01, R-SEC-02 +- Supersedes: — +- Accepted by: operator ruling of 2026-11-08 that the implementation is the accepted state; effective on merge. + +## Context +[ADR-0010](adr-0010-conformance-testing.md) makes the conformance suite gate merges but does not say +who approves. The operator's stated principle (2026-09-29): the CI/CD pipeline is the approver, and +if it is not good enough to be trusted for that, it is not good enough to do its job and needs +improving. Manual approval belongs only on architecturally or security-relevant changes. + +That has one structural hole: **a change to the gate cannot be gated by the gate**, since a PR that +weakens a check would be approved by the weakened check. + +There is also one human on this platform, and Gitea refuses to let an author approve their own pull +request. Requiring one approval on the branch was a lock-out rather than a protection: a change with a +green gate and no possible reviewer could never merge, including the change that would fix that. + +## Decision +- **The merge gate is the approver.** It is the single required status check on `main`. Branch + protection requires zero approvals, direct pushes are rejected, and **a red gate is unmergeable by + everyone, administrators included**. +- **Sensitive paths need a non-author approving review.** This covers the pipeline, the tests and + harness, the Flux wiring, the admission policies, the SOPS recipients and secrets, the RBAC, and the + delivery contract. The path list is data with a stated reason per entry, and is itself sensitive. The + stage is **fail-closed**: if it cannot evaluate, it blocks, since a skip there would be a bypass. +- **An administrator may waive that review** with an explicit directive on the PR. It must start a line + and carry a reason. Permission is re-checked live when the gate runs. A newer commit invalidates it. + The overrider, source, time and reason are printed in the gate output. +- **Agents never use the override on their own work**, regardless of the permission their account + holds. This is a rule of the delivery contract; the check itself authorises by live permission alone + (operator decision, 2026-10-01). + +## Alternatives considered +- **Require one approval on the branch** — unsatisfiable with one human and author self-approval + refused. Rejected after a green PR proved unmergeable. +- **Allow administrators to merge past the gate** — reopens the route by which PRs had merged with zero + checks while settings read as enforced. Rejected; only the approval count was dropped. +- **Gate approves everything, including its own changes** — a weakening PR would approve itself. + Rejected. +- **No override** — the sole administrator could not change the pipeline at all. Rejected. +- **Restrict the override to an allowlist of human accounts** — stronger, and reachable later without + logic changes. Not adopted at this time, with the consequence below accepted knowingly. + +## Consequences +- Ordinary changes merge on green with no human step; a weak gate is a defect, not a reason for review. +- For an administrator, "the gate cannot gate itself" is **advisory, not structural**. What is + guaranteed is an attributed, reasoned, commit-scoped record of every waiver. +- An agent account with administrator permission *can* clear the review on its own PR. Only the + delivery contract prohibits it. Automated agents such as pm-agent + ([ADR-0025](adr-0025-in-cluster-ai-program-manager.md)) therefore use non-administrator accounts. +- Live Gitea settings sit outside git, so a gate stage compares them against a committed declaration. +- The approver is only as good as what the gate runs ([known gaps](../docs/16-known-gaps.md)). diff --git a/decisions/adr-0025-in-cluster-ai-program-manager.md b/decisions/adr-0025-in-cluster-ai-program-manager.md new file mode 100644 index 0000000..d3ea5b5 --- /dev/null +++ b/decisions/adr-0025-in-cluster-ai-program-manager.md @@ -0,0 +1,52 @@ +# ADR-0025 — In-cluster AI Program Manager (pm-agent) +- Status: Accepted +- Date: 2026-11-08 +- Serves: R-GITOPS-01, R-TEST-03, R-TEST-06, R-SEC-02 +- Supersedes: — +- Accepted by: operator ruling of 2026-11-08 that the implementation is the accepted state; effective on merge. + +## Context +The operator wants a Gitea issue to be able to become a pull request without a workstation session. That +puts an AI agent inside the cluster, holding a repository write token and acting on text written by +whoever can reach the repository. It is a new trust boundary: input is untrusted, the actor can author +code, and it needs egress to an external model API. "May create Jobs" is also dangerous in its own +right, since Kubernetes places no check on a Job's `serviceAccountName`. + +## Decision +Run **pm-agent**: a dispatcher in its own namespace that polls labelled issues on the implementation +repo and acts as the Program Manager on each thread; replying to an explicit mention on a PR is an +optional capability, off by default. It starts at most a small, capped number of worker Jobs, each +of which authors a change, runs the merge gate and opens a PR. The issue thread and the PR hold all of its state and correspondence. + +- **Contributor, never approver.** It cannot push to `main` and never merges. Its bot account is a + non-administrator, so it cannot waive review ([ADR-0024](adr-0024-delivery-authority-ci-gate.md)). +- **Three independent layers between a hostile comment and the cluster.** The deciding model holds no + tools; the dispatcher accepts only a validated vocabulary and refuses rather than repairs; and a + fail-closed admission policy admits only the committed Job shape. +- **Workers hold no cluster credential**; egress is limited to Gitea and the model API over TLS. +- **Delivery tooling, not a catalog service.** It declares objects a ServiceDefinition cannot express + and is hand-authored under `platform/` with its own non-gating Flux tier, so it can never stall the + app tier. It adds no `R-*` requirement. +- **Bound by the governance every agent is bound by.** It is an implementation agent under + [AGENTS.md](../AGENTS.md) and R-TEST-06. It does not hold the System Architect role, and it declines + and escalates work that needs a spec change. + +## Alternatives considered +- **Receive Gitea webhooks** — needs a security setting on the git host widened so it can call into + the cluster, and a missed delivery is lost silently. Rejected; polling reads current state. +- **One agent that both triages and implements** — puts tools in reach of raw issue text. Rejected. +- **A database for state** — a second source of truth beside the thread. Rejected. +- **Federated Keycloak machine identity** — Gitea's API cannot validate a third-party token. Rejected; + a local account with a scoped token is what Gitea can verify. +- **RBAC alone as the boundary** — cannot constrain what a created Job runs as. Rejected; admission is + the boundary and RBAC its necessary half. + +## Consequences +- Issues become an intake for implementation work, and every result lands as an ordinary gated PR. +- Accepted residual risks, documented in the implementation repo's threat model: a worker can read its + own credentials (bounded by one-repository, non-admin scope and by egress); TLS egress is wider than + one API host; and no Gitea token scope separates pushing from merging, so "never merges" rests on the + code making no merge call. +- An external model provider sees repository and issue content. +- Its capability can be stopped by configuration without removing the service. +- The worker cannot measure the live platform; anything needing a cluster stays unverified in its report. diff --git a/docs/00-vision-and-scope.md b/docs/00-vision-and-scope.md index a580ec4..0e842f5 100644 --- a/docs/00-vision-and-scope.md +++ b/docs/00-vision-and-scope.md @@ -8,10 +8,9 @@ it under a trusted TLS URL, routes its traffic on the correct WAN, protects its automatically, and reports on its health. Onboarding a new service is a pull request against a catalog, not a weekend of manual work. -The platform is delivered in two stages: - -- **dev/test** on local KVMs — the active build target, used to prove viability. -- **prod** on 3 mini-PCs — the eventual home, designed for now but not built on yet. +The platform runs as one live environment: **prod**, on 3 bare-metal mini-PCs (ADR-0017). The dev/test +cluster that proved its viability was decommissioned in 2026-09; every change reaches prod through +the PR gate and Flux. ## Personas @@ -26,23 +25,23 @@ The platform is delivered in two stages: - A k3s-based platform with distributed storage (Longhorn) across 3 nodes. - Container workloads (PaaS) and VM workloads (IaaS) through one Kubernetes surface. - GitOps delivery, automated updates, trusted TLS, DNS, ingress. -- Network isolation (default-deny east-west) and multi-WAN egress including VPN routing. +- Network isolation (default-deny east-west) and declared egress classes, including VPN routing. - Observability with email + dashboard alerting. - A command-and-control web UI at the root of `home.keithjackson.dev`. -- The initial service catalog (13 services, see [`13-service-catalog.md`](13-service-catalog.md)). +- The service catalog (see [`13-service-catalog.md`](13-service-catalog.md)). ## Out of scope (for now) -- Building on production mini-PC hardware (design-only until the roadmap flips it to active). - Multi-tenant / multi-operator RBAC beyond a single operator + platform automation. - Public multi-user SaaS concerns (billing, per-tenant isolation beyond network policy). - Non-`keithjackson.dev` domains. ## The domain & DNS model -- Public apex `*.keithjackson.dev` is available in Cloudflare DNS. `public` services under - `*.home.keithjackson.dev` are published through a **Cloudflare Tunnel** (proxied records, no inbound - 80/443 at the edge — ADR-0016); the network's public IP stays in use for WireGuard and Plex only. +- Public apex `*.keithjackson.dev` is available in Cloudflare DNS. `public` services (catalog + `exposure: public` or a registered platform route) under `*.home.keithjackson.dev` are published + through a **Cloudflare Tunnel** (proxied records, no inbound 80/443 at the edge — ADR-0016); the + network's public IP stays in use for WireGuard and Plex only. - All platform services live under `*.home.keithjackson.dev`. - Each service is reachable at `.home.keithjackson.dev` (e.g. `plex.home.keithjackson.dev`). - The **root** `home.keithjackson.dev` serves the **Command & Control (C2) center** UI. @@ -54,7 +53,7 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate. ### Platform & API surface — `R-PLAT-*` - **R-PLAT-01** — The platform MUST run on k3s across 3 nodes with production-grade HA control plane - in prod; dev/test MAY run fewer nodes but MUST exercise the same multi-node topology. + in prod. - **R-PLAT-02** — Container workloads MUST be onboarded and managed declaratively via the Kubernetes API, `kubectl`-class CLI, and YAML. - **R-PLAT-03** — Virtual-machine workloads MUST be created, managed, and destroyed through the @@ -82,9 +81,9 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate. routing. Physical multi-WAN (multiple ISP uplinks) is **out of scope** unless a future ADR reopens it. - **R-NET-05** — A service MUST be able to declare its **egress class**: `internet` (standard local - network/internet) or `vpn` (routed through a custom VPN connection). VPN routing MUST be enforced - **outside the service** (a single shared in-cluster gateway), so a service cannot leak by - misconfiguration (ADR-0013, OQ-3). + network/internet) or `vpn` (routed through a custom VPN connection). VPN routing MUST be enforced by + a single shared node-level gateway outside the service (ADR-0015, ADR-0020), so a service cannot + leak by misconfiguration (OQ-3). - **R-NET-06** — VPN-class egress MUST enforce a **kill-switch**: if the VPN tunnel is down, the service MUST NOT leak traffic to the standard internet. - **R-NET-07** — Egress class MUST be a declarative property of the service, changeable via GitOps @@ -101,8 +100,10 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate. ### Automated updates — `R-UPD-*` - **R-UPD-01** — Services MUST update to the latest **stable** versions on a schedule. -- **R-UPD-02** — Updates MUST require **no human, no AI, and no hand-authored GitOps change** to - occur. Automation controllers acting on declarative policy are the permitted mechanism. +- **R-UPD-02** — Updates in the allowed classes MUST occur with **no human, no AI, and no + hand-authored GitOps change**; automation acting on declarative policy opens machine-authored + changes through the merge gate. Major versions and operator-designated blast-radius components MAY + wait for human review of the machine-authored change. - **R-UPD-03** — Update policy MUST be per-service configurable (cadence, stability channel, pin/hold, auto-rollback on failed health). @@ -122,9 +123,10 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate. - **R-DNS-04** — `public` HTTP(S) services MUST be reachable from the internet **without inbound 80/443 port-forwarding at the edge**, and LAN clients MUST reach them correctly **whichever resolver answers** (public or internal) — no dependency on NAT hairpin (ADR-0016). -- **R-DNS-05** — Only services declared `exposure: public` MAY be reachable through the public - ingress path. A request for any other hostname at that path MUST be refused (HTTP 404), and - `internal` services MUST have no public DNS record (ADR-0016). +- **R-DNS-05** — Only services declared public (catalog `exposure: public` or a registered platform + route) MAY be reachable through the public ingress path. A request for a LAN/VPN-only hostname at + that path MUST be refused (HTTP 404). `internal` services MUST have no public record, except + non-HTTP L4 names, which MAY hold DNS-only records at LAN addresses (ADR-0016). ### Observability & alerting — `R-OBS-*` - **R-OBS-01** — Services MUST be monitored (metrics, logs, health). @@ -154,7 +156,8 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate. - **R-IDP-02** — Services MUST authenticate against the IdP — natively via OIDC where supported, or behind an identity-aware (forward-auth) proxy otherwise. No management UI may be exposed unauthenticated on the network. (Documented exceptions: Plex/Seerr use Plex auth; Minecraft uses - Mojang/Microsoft.) + Mojang/Microsoft; UniFi uses its native login on its own L4 VIP; the read-only runbooks at + `/runbooks` carry no login; Crafty web uses forward-auth plus its own local login, no OIDC.) - **R-IDP-03** — Identity configuration (realms, clients, groups, roles) MUST be declarative and GitOps-managed, not console-only. - **R-IDP-04** — IdP secrets (admin credential, OIDC client secrets) MUST be SOPS-encrypted; MFA MUST @@ -166,12 +169,12 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate. console, alerting) require an authenticated user in an authorized group, not merely "logged in." ### Environments — `R-ENV-*` -- **R-ENV-01** — dev/test MUST be provisioned as local KVMs via IaC: created, baselined, cluster- - joined, and managed as code. -- **R-ENV-02** — The same GitOps configuration MUST promote from dev/test to prod with only - environment-scoped overrides. -- **R-ENV-03** — prod is a 3× mini-PC bare-metal k3s cluster with local storage; it MUST NOT be an - active development surface until the roadmap declares it ready. +- **R-ENV-01** — Nodes MUST be baselined, cluster-joined and managed as code; machine creation on + bare metal is an imaged, checklist-driven step. +- **R-ENV-02** — GitOps configuration MUST be authored once as a base plus environment-scoped + overlay; any future environment MUST need overrides only. +- **R-ENV-03** — Production runs on 3× mini-PC bare metal. Changes MUST reach it only through the + PR gate and Flux. ### Conformance testing & governance — `R-TEST-*` - **R-TEST-01** — Every requirement MUST be verified by **automated, behavior-based tests** @@ -196,8 +199,8 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate. ## Acceptance philosophy -Each requirement class maps to an **automated, reproducible, behavior-based conformance test** in -dev/test (`R-TEST-*`). "Done" means *the conformance test for the touched requirements passes* — a +Each requirement class maps to an **automated, reproducible, behavior-based conformance test** +(`R-TEST-*`). "Done" means *the conformance test for the touched requirements passes* — a green, reproducible result traced to requirement IDs — never "manifest merged" and never a manual demonstration. The testing approach is specified in [`docs/14-testing-and-conformance.md`](14-testing-and-conformance.md) and @@ -210,19 +213,18 @@ implementation. ## Open questions -### Resolved (2026-08-15) +### Resolved - **OQ-1 (WAN / edge)** — **RESOLVED**: one commercial WAN, no multi-ISP plans; dumb SOHO edge (no VLAN/VPN/services). All segmentation + VPN in-cluster; edge does NAT + port-forward only. Physical multi-WAN out of scope. See [`03`](03-networking-isolation-and-egress.md), [`09`](09-dns-and-ingress.md). **Amended 2026-11-08 (ADR-0016):** public web ingress moved to an outbound-only **Cloudflare Tunnel**; the edge no longer forwards 80/443. Its remaining forwards are WireGuard udp/51820 and Plex tcp/32400. -- **OQ-2 (bulk media)** — **RESOLVED (deferred backing)**: Plex bulk media lives **outside the - cluster**; the specific backing is **TBD later** and does not block the core platform. Plex's - metadata DB still lives on replicated Longhorn. The **off-cluster DR backup target** (for app - config, Nextcloud, Gitea, etc.) remains to be chosen at Phase 4. See [`04`](04-storage-and-state.md). +- **OQ-2 (bulk media)** — **RESOLVED (ADR-0018)**: Plex bulk media lives **outside the cluster** on + a standalone TrueNAS NFS server on the production LAN, and sits outside platform backup. Plex's + metadata DB still lives on replicated Longhorn. See [`04`](04-storage-and-state.md). - **OQ-3 (VPN)** — **RESOLVED**: **PIA over OpenVPN-UDP** (non-TCP; ADR-0013). Egress is a **node-level - host tunnel steered by Cilium Egress Gateway** with a host nftables kill-switch (ADR-0015) — `vpn`-class - pods routed through it, rules enforced outside the service. See [`03`](03-networking-isolation-and-egress.md). + host tunnel steered by Cilium Egress Gateway**, failing closed under Cilium's own enforcement (ADR-0015, + ADR-0020) — `vpn`-class pods routed through it, rules enforced outside the service. See [`03`](03-networking-isolation-and-egress.md). - **OQ-4 (TLS/DNS challenge access)** — **RESOLVED**: a **scoped Cloudflare API token** (`Zone·DNS·Edit`, `keithjackson.dev`), SOPS-encrypted, drives cert-manager DNS-01 and ExternalDNS. @@ -230,7 +232,9 @@ implementation. handles issuance + record automation. Operator provides the token once. See [`08`](08-tls-and-certificates.md). +- **OQ-BKP (off-cluster DR backup target)** — **RESOLVED (ADR-0018)**: Cloudflare R2 stores Velero + backups (kopia file-system volume data) and CNPG Barman archives of the irreplaceable tier; Longhorn + has no backup target of its own. See [`04`](04-storage-and-state.md). + ### Still open -- **OQ-BKP (off-cluster DR backup target)**: which S3-compatible endpoint or NFS host stores - Velero/Longhorn backups of the irreplaceable tier. Needed by **Phase 4**, not blocking Phase 2. - See [`04`](04-storage-and-state.md). +None tracked here. Unmet requirements are recorded in [known gaps](16-known-gaps.md). diff --git a/docs/01-architecture-overview.md b/docs/01-architecture-overview.md index 818c61b..fec1ddb 100644 --- a/docs/01-architecture-overview.md +++ b/docs/01-architecture-overview.md @@ -9,22 +9,32 @@ choice here without a superseding ADR.** | Layer | Choice | Serves | ADR | Spec | |---|---|---|---|---| | Kubernetes distro | **k3s** (HA, embedded etcd) | R-PLAT-01/02 | [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md) | [02](02-environments-and-provisioning.md) | -| CNI / network policy / egress | **Cilium** (+ Hubble, Gateway API, Egress Gateway) | R-NET-01..07 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [03](03-networking-isolation-and-egress.md) | -| Distributed storage | **Longhorn** | R-STATE-01..03 | [ADR-0003](../decisions/adr-0003-storage-longhorn.md) | [04](04-storage-and-state.md) | +| CNI / network policy / egress | **Cilium** (+ Hubble, Gateway API, Egress Gateway, L2 LB-IPAM) | R-NET-01..07 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [03](03-networking-isolation-and-egress.md) | +| Distributed storage | **Longhorn** | R-STATE-01..02 | [ADR-0003](../decisions/adr-0003-storage-longhorn.md) | [04](04-storage-and-state.md) | +| Bulk media | **NFS** from a standalone **TrueNAS** on the production LAN | R-STATE-04 | [ADR-0018](../decisions/adr-0018-storage-backup-truenas-r2.md) | [04](04-storage-and-state.md) | +| PostgreSQL | **CloudNativePG (CNPG)** | R-STATE-01/03, R-IDP-05 | [ADR-0019](../decisions/adr-0019-postgresql-cloudnativepg.md) | [04](04-storage-and-state.md) | | GitOps engine | **Flux CD** | R-GITOPS-01..03 | [ADR-0004](../decisions/adr-0004-gitops-flux.md) | [05](05-gitops-and-delivery.md) | | VM workloads (IaaS) | **KubeVirt** (+ CDI) | R-PLAT-03 | [ADR-0005](../decisions/adr-0005-vm-workloads-kubevirt.md) | [06](06-vm-workloads-kubevirt.md) | | TLS | **cert-manager** + Let's Encrypt (DNS-01) | R-TLS-01..03 | [ADR-0006](../decisions/adr-0006-tls-cert-manager-letsencrypt.md) | [08](08-tls-and-certificates.md) | | Secrets | **SOPS + age** (Flux-native) | R-SEC-01 | [ADR-0007](../decisions/adr-0007-secrets-sops-age.md) | [07](07-security-and-secrets.md) | -| VPN egress | **Node-level PIA/OpenVPN tunnel** steered by **Cilium Egress Gateway** (host nftables kill-switch) | R-NET-05/06/07 | [ADR-0015](../decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md) (provider [0013](../decisions/adr-0013-vpn-provider-pia-gluetun-wireguard.md)) | [03](03-networking-isolation-and-egress.md) | -| Automated updates | **Flux image-update-automation** (images) + **Renovate** (charts) | R-UPD-01..03 | [ADR-0009](../decisions/adr-0009-automated-image-updates.md) | [11](11-automated-updates.md) | -| DNS records | **ExternalDNS** (public via Cloudflare) + **PiHole** (internal) | R-DNS-01..03 | — | [09](09-dns-and-ingress.md) | +| VPN egress | **Node-level PIA/OpenVPN tunnel** on a single gateway node, steered by **Cilium Egress Gateway** (Cilium-enforced fail-closed) | R-NET-05/06/07 | [ADR-0015](../decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md), [ADR-0020](../decisions/adr-0020-vpn-egress-single-gateway-cilium-fail-closed.md) (provider [0013](../decisions/adr-0013-vpn-provider-pia-gluetun-wireguard.md)) | [03](03-networking-isolation-and-egress.md) | +| Automated updates | **Renovate** (images and charts), every change PR-gated | R-UPD-01..03 | [ADR-0023](../decisions/adr-0023-automated-updates-renovate.md) | [11](11-automated-updates.md) | +| DNS records | **ExternalDNS** (public via Cloudflare) + **PiHole** (explicit LAN answers) | R-DNS-01..03 | — | [09](09-dns-and-ingress.md) | | Ingress (north-south) | **Gateway API** implemented by Cilium | R-DNS-01, R-TLS-01 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [09](09-dns-and-ingress.md) | +| Public web ingress | **Cloudflare Tunnel** (outbound-only cloudflared) | R-DNS-04/05, R-SEC-04 | [ADR-0016](../decisions/adr-0016-public-ingress-cloudflare-tunnel.md) | [09](09-dns-and-ingress.md) | +| Remote access | **WireGuard** on a hostNetwork DaemonSet | R-SEC-02/04 | [ADR-0021](../decisions/adr-0021-inbound-remote-access-wireguard.md) | [03](03-networking-isolation-and-egress.md) | | Observability | **kube-prometheus-stack** + **Loki** + **Grafana Alloy** + Alertmanager→SMTP | R-OBS-01..04 | — | [10](10-observability-and-alerting.md) | -| Backup/DR | **Velero** + Longhorn backups → off-cluster S3/NFS | R-STATE-03 | [ADR-0003](../decisions/adr-0003-storage-longhorn.md) | [04](04-storage-and-state.md) | -| Node/infra provisioning | **OpenTofu** (libvirt) + **cloud-init** + **Ansible** (k3s bootstrap) | R-ENV-01..03 | [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md) | [02](02-environments-and-provisioning.md) | -| Identity / SSO | **Keycloak** (OIDC + SAML) + **oauth2-proxy** forward-auth | R-IDP-01..06 | [ADR-0014](../decisions/adr-0014-idp-keycloak.md) | [15](15-identity-and-access.md) | +| Backup/DR | **Velero** (kopia) + **CNPG Barman** → Cloudflare R2; no Longhorn backup target | R-STATE-03 | [ADR-0018](../decisions/adr-0018-storage-backup-truenas-r2.md) | [04](04-storage-and-state.md) | +| Node/infra provisioning | **Imaged Debian** on bare metal + **Ansible** (baseline, k3s bootstrap) | R-ENV-01..03 | [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md), [ADR-0017](../decisions/adr-0017-single-environment-bare-metal-production.md) | [02](02-environments-and-provisioning.md) | +| Control-plane endpoint | **kube-vip** virtual IP | R-PLAT-01 | [ADR-0017](../decisions/adr-0017-single-environment-bare-metal-production.md) | [02](02-environments-and-provisioning.md) | +| Node patching & reboots | security-only **unattended-upgrades** + **kured** | R-PLAT-01 | — | [02](02-environments-and-provisioning.md) | +| Identity / SSO | **Keycloak** (OIDC + SAML) + per-app **oauth2-proxy** forward-auth sidecars | R-IDP-01..06 | [ADR-0014](../decisions/adr-0014-idp-keycloak.md), [ADR-0022](../decisions/adr-0022-forward-auth-per-app-oauth2-proxy-sidecars.md) | [15](15-identity-and-access.md) | | Command center UI | **Homepage** + **Headlamp** + **Grafana** panels (behind Keycloak SSO) | R-C2-01/02 | — | [12](12-command-center-ui.md) | | Conformance testing | **Kyverno** (L0 policy) + **Chainsaw** (L1/L2/L3 behavior + fault-injection) | R-TEST-01..06 | [ADR-0010](../decisions/adr-0010-conformance-testing.md), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md) | [14](14-testing-and-conformance.md) | +| Delivery & merge gate | **Gitea** + **Gitea Actions**; the CI gate is the approver | R-GITOPS-01, R-TEST-03 | [ADR-0024](../decisions/adr-0024-delivery-authority-ci-gate.md) | [05](05-gitops-and-delivery.md) | +| Image builds | CI-run rootless **BuildKit**, published to Gitea's registry | — | — | [05](05-gitops-and-delivery.md) | +| AI Program Manager | **pm-agent** (issues to pull requests through the gate) | — | [ADR-0025](../decisions/adr-0025-in-cluster-ai-program-manager.md) | [05](05-gitops-and-delivery.md) | +| Hardware transcode | **Intel GPU device plugin** (Plex Quick Sync) | — | — | [13](13-service-catalog.md) | ## Layer cake @@ -41,18 +51,17 @@ choice here without a superseding ADR.** │ Flux (GitOps) · cert-manager (TLS) · ExternalDNS · Gateway API │ │ Keycloak (SSO/IdP) + oauth2-proxy (forward-auth) │ │ kube-prometheus-stack + Loki + Alloy (observability) │ -│ Velero (backup) · Renovate (chart updates) · SOPS (secrets) │ +│ Velero + CNPG Barman (backup) · Renovate (updates) · SOPS (secrets) │ +│ cloudflared (public ingress) · CNPG (PostgreSQL) · kured (node reboots) │ ├──────────────────────────────────────────────────────────────────────────┤ -│ Storage: Longhorn (replicated block) + bulk media store (see ADR-0003) │ +│ Storage: Longhorn (replicated block) + TrueNAS NFS bulk tier (ADR-0018) │ ├──────────────────────────────────────────────────────────────────────────┤ │ Network: Cilium (default-deny NetworkPolicy · Egress Gateway · Hubble) │ -│ VPN egress (node tunnel + Cilium egress GW) · Multi-WAN routing │ +│ VPN egress (node tunnel + Cilium egress GW) · WireGuard inbound │ ├──────────────────────────────────────────────────────────────────────────┤ │ Kubernetes: k3s (3 nodes, HA control plane, embedded etcd) │ ├──────────────────────────────────────────────────────────────────────────┤ -│ Nodes │ -│ dev/test: libvirt KVMs (OpenTofu + cloud-init + Ansible) │ -│ prod: 3× mini-PC bare metal, local disks │ +│ Nodes: 3× mini-PC bare metal, local disks (imaged Debian + Ansible) │ ├──────────────────────────────────────────────────────────────────────────┤ │ Edge: SOHO router — single WAN, NAT; forwards only WireGuard + Plex │ │ Public web ingress: Cloudflare Tunnel (outbound-only cloudflared, ADR-0016)│ @@ -64,8 +73,8 @@ choice here without a superseding ADR.** This platform has **two** kinds of VM, and confusing them is the most common design error: 1. **Infrastructure VMs = the cluster nodes themselves.** - In dev/test these are libvirt KVMs on the workstation, provisioned by OpenTofu + cloud-init and - configured/joined by Ansible. In prod these are the bare-metal mini-PCs. See + These are the bare-metal mini-PCs, imaged and then baselined and joined by Ansible; "VM" is + historical shorthand for this layer, and it remains the place a node would be virtualized. See [`02-environments-and-provisioning.md`](02-environments-and-provisioning.md). 2. **Application VMs = IaaS workloads that run *inside* the cluster via KubeVirt.** @@ -81,16 +90,20 @@ This platform has **two** kinds of VM, and confusing them is the most common des storage, **egress class** (`internet`|`vpn`), exposure (`internal`|`public`), dependencies (explicit allow rules), update policy, and observability expectations. 2. **Flux** reconciles the change into the cluster. -3. **Longhorn** provisions replicated persistent volumes; state is durable. +3. **Longhorn** provisions replicated persistent volumes, so state is durable; large media attaches + from the NFS **bulk tier**. 4. **Cilium** applies default-deny plus the explicitly declared allow rules; assigns the correct **egress path** (local internet or VPN, with kill-switch). -5. **ExternalDNS** publishes `.home.keithjackson.dev`; **Gateway API** routes north-south - traffic; **cert-manager** issues a trusted TLS cert via ACME DNS-01. +5. A `public` service gets a proxied **ExternalDNS** record at `.home.keithjackson.dev` and + arrives through the **Cloudflare Tunnel**. An `internal` service has no public record and is + answered on the LAN by **PiHole**. **Gateway API** routes north-south traffic for both; + **cert-manager** issues a trusted TLS cert via ACME DNS-01. 6. **kube-prometheus-stack + Loki** scrape metrics/logs; **Alertmanager** emails on issues; the **C2 UI** lists and links the service. -7. **Flux image automation / Renovate** keep the service on the latest stable version per policy — - no human action. -8. **Velero + Longhorn backups** protect the data; destroy/rebuild restores state. +7. **Renovate** keeps the service on the latest stable version per policy, through machine-authored + changes that pass the merge gate. +8. **Velero and CNPG Barman**, writing off-cluster, protect the data; destroy/rebuild restores + state. Every numbered step maps to a domain spec and a workstream in [`roadmap/workstreams.md`](../roadmap/workstreams.md). @@ -103,7 +116,7 @@ Every numbered step maps to a domain spec and a workstream in [`roadmap/workstre - **Durable by default.** No workload is "pet" hardware; state lives on replicated storage with tested restore. - **Hands-off operations.** Updates, cert renewal, DNS, and drift correction are automated. -- **Promotable.** dev/test and prod are the same config with scoped overrides. +- **Promotable.** Configuration is one base plus an environment-scoped overlay. - **Provable.** Every requirement is verified by an automated, reproducible, behavior-based conformance test ([14](14-testing-and-conformance.md), [ADR-0010](../decisions/adr-0010-conformance-testing.md)); "done" is a green, requirement-tagged suite, not a manual demo. diff --git a/docs/02-environments-and-provisioning.md b/docs/02-environments-and-provisioning.md index cf1bb87..4c9f714 100644 --- a/docs/02-environments-and-provisioning.md +++ b/docs/02-environments-and-provisioning.md @@ -1,39 +1,41 @@ # 02 — Environments & Provisioning -Serves: **R-ENV-01/02/03**, **R-PLAT-01**. Related: [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md). +Serves: **R-ENV-01/02/03**, **R-PLAT-01**. Related: [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md), +[ADR-0017](../decisions/adr-0017-single-environment-bare-metal-production.md). -## Two environments, one configuration +## One live environment, one configuration -| Aspect | dev/test | prod | -|---|---|---| -| Substrate | libvirt/KVM VMs on the operator workstation | 3× mini-PC bare metal | -| Node count | 3 VMs (mirror prod topology) | 3 physical nodes | -| Control plane | HA embedded etcd (k3s server on all 3) | HA embedded etcd (k3s server on all 3) | -| Storage | virtual disks presented to Longhorn | local SSD/NVMe presented to Longhorn | -| Status | **active build target** | **design-only until roadmap flips it** | +| Aspect | prod | +|---|---| +| Substrate | 3× mini-PC bare metal | +| Node count | 3 physical nodes | +| Control plane | HA embedded etcd (k3s server on all 3) | +| Storage | local SSD/NVMe presented to Longhorn; bulk media on an off-cluster NAS | +| Status | **live** | -**R-ENV-02 is the golden constraint:** the GitOps configuration is authored once. Environments -differ only through scoped overrides (Flux `Kustomization` overlays / cluster-scoped variables): -node counts, disk sizes, WAN endpoints, VPN credentials, DNS zone specifics, resource limits. -Never fork the base config per environment. +A dev/test cluster proved the platform and was decommissioned in 2026-09. Prod is the only +environment. + +**R-ENV-02 is the golden constraint:** the GitOps configuration is authored once, as a base plus an +environment-scoped overlay (Flux `Kustomization` overlays / cluster-scoped variables): node counts, +disk sizes, WAN endpoints, VPN credentials, DNS zone specifics, resource limits. Never fork the base +config per environment. Any future environment needs overrides only. ## Infrastructure provisioning (Layer 1 — the cluster nodes) > These are the machines that *run* k3s. Distinct from KubeVirt application VMs (Layer 2). ### Toolchain -- **OpenTofu** (Terraform-compatible) with the **libvirt** provider defines dev/test node VMs as - code: CPU, RAM, disks (a dedicated data disk for Longhorn), NICs, and a **cloud-init** seed. -- **cloud-init** performs first-boot baseline: hostname, users, SSH keys, package baseline, kernel - modules/sysctls required by k3s/Longhorn/KubeVirt, and marks the node ready for k3s. +- **Bare-metal imaging** is an operator step driven by a checklist. There is no hypervisor API on + bare metal to automate machine creation. +- **Ansible** performs the node baseline: users, package floor, kernel, kernel modules/sysctls + required by k3s/Longhorn/KubeVirt, and the data partition. - **Ansible** performs cluster bootstrap: installs k3s in HA mode, joins nodes, lays down cluster- wide prerequisites (Cilium is installed as the CNI, not flannel), and hands off to Flux. -- For prod bare-metal, the *same* Ansible bootstrap runs; only the machine-creation step differs - (bare-metal image/cloud-init via USB/PXE instead of libvirt). -### Node lifecycle stages (R-ENV-01 — "created, baselined, cluster-joined, managed as code") -1. **Create** — OpenTofu instantiates the VM (dev) / operator images the mini-PC (prod). -2. **Baseline** — cloud-init applies OS baseline and prerequisites. +### Node lifecycle stages (R-ENV-01 — "baselined, cluster-joined, managed as code") +1. **Create** — the operator images the mini-PC from the provisioning checklist. +2. **Baseline** — Ansible applies the OS baseline and prerequisites. 3. **Cluster-join** — Ansible installs/joins k3s with Cilium as CNI; disables k3s bundled flannel/traefik/servicelb/kube-proxy as required by the Cilium ADR. 4. **Manage** — from here the cluster is GitOps-managed by Flux; node config changes are made via @@ -41,10 +43,13 @@ Never fork the base config per environment. ### k3s topology decisions - HA control plane with **embedded etcd** across all 3 nodes (no external datastore). +- The Kubernetes API has one HA endpoint: a **kube-vip** virtual IP. Steady-state consumers use the + virtual IP. A node joining during a cold bootstrap uses the founding server's own address, because + the virtual IP does not exist until a cluster does. - Bundled components that conflict with our stack are **disabled at install**: flannel (→ Cilium), - kube-proxy (→ Cilium kube-proxy replacement), Traefik (→ Gateway API via Cilium), servicelb where - a different LB strategy is chosen. Exact flags live in the Ansible bootstrap; the *intent* is - fixed here and in ADR-0001/0002. + kube-proxy (→ Cilium kube-proxy replacement), Traefik (→ Gateway API via Cilium), servicelb + (→ Cilium L2 announcements and LB-IPAM), local-path storage (→ `longhorn` as the sole default). + Exact flags live in the Ansible bootstrap; the *intent* is fixed here and in ADR-0001/0002. - Nodes are both control-plane and workload (3-node converged cluster); with only 3 nodes we do not isolate control-plane nodes. @@ -57,46 +62,63 @@ resolves the apparent overlap between "Ansible installs the CNI" (here) and "Flu in **parity** for the settings that define cluster networking — notably `kubeProxyReplacement` and `k8sServiceHost`/`k8sServicePort` — or the two layers will fight over the CNI config. -### Bare-metal specifics (prod, design-only for now) -- Each mini-PC contributes local NVMe/SSD to Longhorn; document the disk layout (OS vs. Longhorn - data device) before prod build. -- BIOS/firmware, virtualization extensions (for KubeVirt), and wake/boot behavior are captured as a - provisioning checklist under the prod-readiness workstream. -- Edge networking (multi-WAN, VPN uplink) is largely an **edge-router** concern — see +### Bare-metal specifics +- The nodes are 3 HP EliteDesk Mini desktops with VT-x. Their drives differ (SATA SSD and NVMe); the + disk *layout* is identical across all 3. The node inventory, BIOS/firmware settings, and the + provisioning checklist are `infra/PROD-PROVISIONING.md` in the + [implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl). +- Each node has **one physical disk**: an OS area and one data partition. The data partition holds + both the Longhorn store and the k3s data directory (embedded etcd and the container image store). +- Nodes run a newer kernel series than the distribution default, because the firmware publishes a + defective EFI memory table that older kernels mishandle. This is a standing, deliberate deviation. +- **Guardrail and accepted risk (etcd and Longhorn share a device).** With no second device in the + chassis, etcd shares each node's disk with Longhorn replica I/O. Every node MUST be SSD or NVMe, + and etcd takes I/O priority over other work on the device. The risk is accepted knowingly. Its + exit condition is that etcd write latency stays within budget during a Longhorn rebuild, or each + node gains a dedicated etcd device (see [known gaps](16-known-gaps.md)). +- The edge is a single-WAN SOHO router — see [`03-networking-isolation-and-egress.md`](03-networking-isolation-and-egress.md) and OQ-1. +### Off-host storage +Bulk media lives on a standalone TrueNAS NFS server on the production LAN, directly reachable by the +nodes (ADR-0018, [04](04-storage-and-state.md)). It is storage only and runs no platform compute. + +### Node maintenance +OS patching is automatic and security-only, excluding the kernel. Kernel upgrades are deliberate +operator actions, run one node at a time. A reboot daemon carries out the mechanics of a needed +reboot and takes a single cluster-wide lock, so at most one node reboots at a time, and the next +waits for Longhorn volumes to recover. A hung node holds the lock, stopping the cycle until a human +intervenes: on 3 etcd members, losing a second node is the failure to avoid. + ## Environment promotion model ``` git repo (single source of truth) │ - base configuration ──────────────┐ - │ │ - overlay: dev/test overlay: prod - │ │ - Flux on dev cluster Flux on prod cluster (later) + base configuration + │ + overlay: prod + │ + Flux on the prod cluster ``` -- A change proves out in **dev/test** against the acceptance gates ([`AGENTS.md` §7](../AGENTS.md)). -- Promotion to prod is a change to the prod overlay only (no base edits), gated by the roadmap's - prod-readiness phase. -- **Never develop against prod.** Prod hardware is a target, not a sandbox (R-ENV-03). +- A change proves out against the acceptance gates ([`AGENTS.md` §7](../AGENTS.md)) before it merges. +- A future environment is a new overlay only (no base edits). +- **Changes reach prod only through the PR gate and Flux** (R-ENV-03). Fault-injecting tests against + prod are deliberate, scheduled exercises. ## Statelessness of the nodes themselves -Nodes are cattle. A node can be destroyed and rebuilt from IaC + cloud-init + Ansible; cluster and +Nodes are cattle. A node can be destroyed and rebuilt by imaging and Ansible; cluster and application state lives on **Longhorn** (replicated) and in **git** (desired state), so node loss is a re-provision, not a data-loss event (supports R-STATE-01/02). -## Deliverables for this spec (tracked on the task board) -- OpenTofu module for dev/test node VMs (libvirt) with a dedicated Longhorn data disk. -- cloud-init baseline template (users, SSH, sysctls, kernel modules for Longhorn/KubeVirt). -- Ansible role(s): k3s HA install with Cilium, component disabling, Flux bootstrap handoff. -- Documented prod bare-metal provisioning checklist (design-only; do not execute). -- Proof: destroy a dev node VM and rebuild it; cluster re-converges with no data loss. +## Deliverables for this spec +- Ansible role(s): node baseline, k3s HA install with Cilium, component disabling, Flux bootstrap + handoff. +- Documented bare-metal provisioning checklist. +- Proof: lose a node and rebuild it; the cluster re-converges with no data loss. ## Open questions -- **OQ-1** (edge router / WAN count) — blocks final multi-WAN design, not node provisioning. -- Do we want **Cluster API** later to unify dev VM and prod bare-metal lifecycle? Deferred; OpenTofu - + Ansible is sufficient and lower-complexity for 3 nodes now. Revisit via ADR if prod scaling - demands it. +- Do we want **Cluster API** later to automate bare-metal lifecycle? Deferred; imaging + Ansible is + sufficient and lower-complexity for 3 nodes. Revisit via ADR if scaling demands it. diff --git a/docs/03-networking-isolation-and-egress.md b/docs/03-networking-isolation-and-egress.md index 4025b02..3914ec1 100644 --- a/docs/03-networking-isolation-and-egress.md +++ b/docs/03-networking-isolation-and-egress.md @@ -1,7 +1,9 @@ # 03 — Networking, Isolation & Egress Serves: **R-NET-01..07**, **R-SEC-02**. Related: [ADR-0002 (Cilium)](../decisions/adr-0002-cni-cilium.md), -[ADR-0008 (VPN egress)](../decisions/adr-0008-vpn-egress-model.md). +[ADR-0015 (VPN egress steering)](../decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md), +[ADR-0020 (single gateway, Cilium fail-closed)](../decisions/adr-0020-vpn-egress-single-gateway-cilium-fail-closed.md), +[ADR-0021 (inbound remote access)](../decisions/adr-0021-inbound-remote-access-wireguard.md). This is the most safety-critical spec. Get default-deny and the VPN kill-switch right; everything else is convenience. @@ -76,7 +78,7 @@ as multi-ISP routing but as **two egress paths selected by routing rules**: | Profile | Meaning | Mechanism | |---|---|---| | `internet` | Out the single commercial WAN | Normal pod egress via node → SOHO edge router → internet | -| `vpn` | Routed through the in-cluster VPN tunnel | Steered to the shared VPN egress gateway (ADR-0013) | +| `vpn` | Routed through the in-cluster VPN tunnel | Steered to the shared VPN egress gateway (ADR-0015, ADR-0020) | - **Physical multi-WAN (multi-ISP) is out of scope.** With a single WAN there is nothing to policy- route across; the `internet` path is simply the default route out. @@ -88,7 +90,7 @@ as multi-ISP routing but as **two egress paths selected by routing rules**: not forward 80/443. - The **service-facing contract is unchanged**: a service still just declares `egress: internet|vpn`. -## VPN egress model: node-level tunnel + Cilium Egress Gateway (R-NET-05/06 — ADR-0015; provider ADR-0013) +## VPN egress model: node-level tunnel + Cilium Egress Gateway (R-NET-05/06 — ADR-0015, ADR-0020; provider ADR-0013) VPN-class services: **sonarr, radarr, jackett, qbittorrent**. Per OQ-3 the design is a **single, shared VPN connection** — not a per-service sidecar — so that **network rules are enforced outside @@ -105,7 +107,7 @@ the wrong class. The model: -- **Tunnel on the node, not in a pod.** One or two labeled **egress-gateway nodes** run a **host OpenVPN +- **Tunnel on the node, not in a pod.** Exactly one labeled **egress-gateway node** runs a **host OpenVPN systemd service** — **PIA over OpenVPN-UDP** (provider/protocol per ADR-0013; "non-TCP or better"), **Ansible-managed host infra** (like k3s itself), NOT a Flux workload. It is **split-tunnel**: it creates `tun0` but does **not** set the node's default route (`route-noexec` + policy routing), so the @@ -114,23 +116,52 @@ The model: - **`vpn`-class pods are steered by Cilium Egress Gateway.** A `CiliumEgressGatewayPolicy` selects pods labeled `egress-class=vpn` and egresses them via the gateway node's `tun0` (native eBPF SNAT to the tunnel IP). **No client-pod changes, no sidecar, no overlay.** They have **no independent egress path**; - the service is just labeled `egress: vpn` and cannot leak on its own. -- **Kill-switch (fail-closed) — host nftables.** An **nftables rule on the gateway node** DROPS steered - (`egress-class=vpn`) traffic whenever `tun0` is down — a host backstop independent of any Cilium - egress-gateway fail-open edge case. If the tunnel is down, `vpn` pods have **no route to the plain - internet**; `internet`-class services are unaffected. + the service is just labeled `egress: vpn` and cannot leak on its own. Only public destinations are + tunnelled; private ranges (the cluster, the LAN, bulk storage) stay on the local datapath. +- **Kill-switch (fail-closed) — Cilium's egress datapath (ADR-0020).** Steered traffic is forwarded in + eBPF, and **Cilium drops it** whenever the gateway's tunnel interface cannot be used. If the tunnel is + down, `vpn` pods have **no route to the plain internet**; `internet`-class services are unaffected. + Steered pod traffic never reaches netfilter, so a **host nftables table** on the gateway is a narrow + backstop only: it covers node-originated traffic sourced from a tunnel address. +- **Re-steering across reconnects.** Cilium resolves the tunnel interface once and does not follow its + re-creation, so every reconnect is signalled back to Cilium from the gateway node, without writing to + GitOps state. A reconnect Cilium was not told about raises an alert + ([10](10-observability-and-alerting.md)). - **Port forwarding for qBittorrent:** PIA NAT-PMP, re-solved **host-side** (a small pf/natpmp helper on the gateway node that publishes the assigned forwarded port into qBittorrent) — kept hands-off. Alert - if PF acquisition fails. -- **Resilience:** two egress-gateway nodes + **Cilium Egress Gateway HA** (gateway failover) give the - active/standby resilience; alert on tunnel state ([10](10-observability-and-alerting.md)). -- **Prerequisite:** the **PIA username/password + `.ovpn`** on the gateway node(s) via **Ansible vault** + if PF acquisition fails (see [known gaps](16-known-gaps.md)). +- **Resilience: one gateway, no failover (ADR-0020).** Cilium OSS resolves the gateway selector to a + single node, so a second labeled node is not a standby: it silently takes the role, and the public + egress IP, by name order. **Label exactly one node.** Losing it is a `vpn`-class outage, fail-closed: + no leak and no service. Alert on tunnel state ([10](10-observability-and-alerting.md)). +- **Prerequisite:** the **PIA username/password + `.ovpn`** on the gateway node via **Ansible vault** ([07](07-security-and-secrets.md)) — the tunnel is host infra, so its creds are host-side (not a cluster - SOPS Secret). Validate end-to-end (tunnel + steering + kill-switch + PF) in dev/test. + SOPS Secret). Validate end-to-end (tunnel + steering + kill-switch + PF) by conformance test + ([14](14-testing-and-conformance.md)). **Kill-switch is an acceptance gate:** with the tunnel forcibly down, every `vpn` service must fail closed (no packets to the plain internet), while `internet`-class is unaffected. Tested, not assumed. +## Remote access (inbound): WireGuard (ADR-0021) + +The operator's devices reach the home network through a **WireGuard** listener on udp/51820. This is +the opposite direction to `vpn` egress and shares nothing with it: `vpn` egress sends pods *out* +through the provider tunnel; remote access lets clients *in*. It is the "VPN" in "LAN/VPN-only". + +- **Host network, every node.** A Flux-managed listener runs in the host network namespace on all + nodes. A pod-networked server would give every client that pod's Cilium identity, and every Gateway + hostname would refuse it. A listener on each node makes loss of the forwarded node a one-line router + change, with no new client profiles. +- **A deliberate exception to default-deny.** A host-network workload has no Cilium endpoint, so no + network policy constrains it. The boundary around the port is **WireGuard's peer authentication**: a + packet from an unknown key gets no reply. The exception applies to this platform component only; + application namespaces may not use the host network. +- **Authorization is reviewable.** Peers are public keys in plain text in git, so granting or revoking + a device is a reviewed change. The server's private key is SOPS-encrypted. +- **Reach equals a LAN machine.** An enrolled device can reach the whole LAN, including the nodes' + control-plane ports, and egresses through the house WAN. A lost device must be revoked promptly. + Clients stay `internet` class and are never steered to the provider tunnel. + ## Ingress (north-south) — summary North-south exposure is handled by **Gateway API implemented by Cilium**, with TLS from @@ -157,9 +188,8 @@ Cloudflare Tunnel, ADR-0016). Amended 2026-11-08 (ADR-0016): 80/443 are no longer forwarded; public web ingress is a Cloudflare Tunnel. - **OQ-3** — **PIA over OpenVPN-UDP** (ADR-0013); egress is a **node-level host tunnel** steered by **Cilium - Egress Gateway** with a host nftables kill-switch (ADR-0015), rules enforced outside the service. + Egress Gateway** and failing closed in Cilium's datapath (ADR-0015, ADR-0020), rules enforced outside + the service. ## Open questions -- Exact steering mechanism for `vpn` pods → gateway (Cilium Egress Gateway vs routing/SNAT) — - WS-NET implementation detail; the kill-switch acceptance test is the gate. - Which **PF-enabled PIA region** to use, and the port-sync helper wiring for qBittorrent. diff --git a/docs/04-storage-and-state.md b/docs/04-storage-and-state.md index 3d21259..9f250d8 100644 --- a/docs/04-storage-and-state.md +++ b/docs/04-storage-and-state.md @@ -1,6 +1,8 @@ # 04 — Storage & State -Serves: **R-STATE-01..04**, **R-PLAT-04**. Related: [ADR-0003 (Longhorn)](../decisions/adr-0003-storage-longhorn.md). +Serves: **R-STATE-01..04**, **R-PLAT-04**. Related: [ADR-0003 (Longhorn)](../decisions/adr-0003-storage-longhorn.md), +[ADR-0018 (bulk tier + backup)](../decisions/adr-0018-storage-backup-truenas-r2.md), +[ADR-0019 (PostgreSQL via CloudNativePG)](../decisions/adr-0019-postgresql-cloudnativepg.md). The core promise: **any service can be destroyed and rebuilt on a whim without losing data.** This requires (a) durable replicated storage, (b) a clean separation of state from workload, and (c) a @@ -10,21 +12,41 @@ tested backup/restore path. | Tier | Backing | Use | Replication | |---|---|---|---| -| **App/config state** | Longhorn block volumes | PVCs for service config, databases, small app data | 3-way replica across nodes | -| **Bulk media** | Dedicated large store (see below) | Plex library, download staging | Not 3× replicated (capacity strategy) | +| **App/config state** | Longhorn block volumes | PVCs for service config, embedded databases, small app data | 3-way replica across nodes | +| **Databases** | CloudNativePG clusters on Longhorn | PostgreSQL for Keycloak, Nextcloud, Semaphore | 3 instances, each on a 3-way replicated volume | +| **Bulk media** | Off-cluster NAS, consumed over NFS | Plex library, download staging, read-only legacy Nextcloud archive | Not 3× replicated (capacity strategy) | | **VM disks** | Longhorn (via KubeVirt/CDI) | KubeVirt application-VM root/data disks | 3-way replica | -| **Backups** | Off-cluster S3/NFS target | Velero + Longhorn backups for DR | Off-cluster, separate failure domain | +| **Backups** | Cloudflare R2 (S3-compatible, off-site) | Velero + kopia volume data; CloudNativePG WAL and base backups | Off-cluster, separate failure domain | ## Longhorn (primary persistent storage — R-STATE-02) - Provides replicated block storage from each node's local disk — exactly the model for a 3× mini-PC cluster with local storage and no SAN. - Default StorageClass: **3 replicas** so any single node loss keeps data available. +- Two further classes serve KubeVirt: a live-migratable block class (3 replicas, the default for VM + disks) and a single-replica class for CDI's throwaway import scratch, the one non-replicated use of + Longhorn. - Volumes are the durable home for stateful services. Destroying a pod/Deployment/VM leaves the PVC and its data intact; rebuilding re-attaches it (R-STATE-01). -- Snapshots for point-in-time; **backups** (to off-cluster target) for DR. +- Snapshots are available on demand and are how VM disks clone; **backups** for DR go off-cluster + through Velero (below), not through a Longhorn backup target. - Guardrails: reserve disk headroom, set replica anti-affinity across nodes, monitor volume health - and degraded replicas (alerts in [10](10-observability-and-alerting.md)). + and degraded replicas (alerts in [10](10-observability-and-alerting.md)), and cap Longhorn's write + concurrency so replica I/O cannot starve etcd's WAL. +- **Accepted risk:** each node has one data device, so etcd's WAL and every Longhorn replica share it. + The concurrency caps reduce that contention and do not remove it. The exit condition is a dedicated + etcd device per node (see [known gaps](16-known-gaps.md)). + +## Database state (PostgreSQL) + +Services that need PostgreSQL get a **CloudNativePG** cluster per database ([ADR-0019](../decisions/adr-0019-postgresql-cloudnativepg.md)): +three instances on three different nodes, each on replicated Longhorn. Longhorn replication keeps the +*data* through a node loss; a streaming standby keeps the *service*, because promotion takes seconds. +Standbys replicate corruption faithfully, so neither is a backup: WAL is archived continuously and +base backups run daily to the off-cluster target, giving point-in-time recovery. + +Services whose datastore is embedded (Gitea's and Crafty's SQLite, for example) keep it on their +Longhorn PVC and are covered by the ordinary volume backup. ## Separation of state from workload (the rebuild guarantee) @@ -33,44 +55,65 @@ Every service's catalog entry declares its **persistent volumes** explicitly. Th > If it must survive a rebuild, it lives on a named PVC (Longhorn) or the bulk store — **never** in > the container's ephemeral filesystem or on a single node's hostPath. +A volume counts only when it is mounted at the path the application actually writes. A PVC mounted +anywhere else binds, reports healthy and backs up successfully while the real state sits on the +ephemeral layer, so a service's mount path is declared deliberately rather than left to a default. + The rebuild test (an acceptance gate): delete the workload entirely, re-apply from git, and the service comes back with its data. This is verified per service, not assumed. ## Bulk media strategy (R-STATE-04 — the Plex problem) -Plex is the largest service and its library can be multiple terabytes. **3× replicating multi-TB +Plex is the largest service and its library is multiple terabytes. **3× replicating multi-TB media across three mini-PCs is impractical** (capacity and cost). Per **OQ-2 (resolved)**, **bulk -media lives OUTSIDE the cluster**; the specific backing is **to be determined later** and does not -block the core platform. +media lives OUTSIDE the cluster** on a standalone TrueNAS appliance: storage only, ZFS-backed, and on +the same LAN as the nodes ([ADR-0018](../decisions/adr-0018-storage-backup-truenas-r2.md)). -- The bulk store is consumed by Plex and the arr stack as an **external mount** (NFS/SMB share to be - specified when chosen). The `bulk` StorageClass / mount is a thin indirection so the concrete - backing can be filled in later without touching service definitions. +- The bulk store is consumed by Plex, the arr stack and qbittorrent as an **external NFS mount**. + Nextcloud mounts a separate legacy archive from it read-only. +- A catalog entry's `class: bulk` plus a named export is a thin indirection: it attaches an existing + dataset as a statically bound volume, and the NFS server is a per-environment value, so service + definitions never name the storage host. Nothing is dynamically provisioned, because every bulk + volume attaches data that already exists. - Downloads (qbittorrent) stage to the bulk store and hand off to the arr stack. +- Layout, export settings and the uid model are in `docs/BULK-NFS-STORAGE.md` in the + [implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl). **Regardless of the bulk backing: Plex's config/metadata database is small and MUST live on 3-replica -Longhorn** (so Plex rebuilds cleanly); only the large media files use the external bulk store. Bulk -media is treated as **`reacquirable`** for backup purposes unless the operator specifies otherwise -when the backing is chosen. +Longhorn** (so Plex rebuilds cleanly); only the large media files use the external bulk store. + +**The bulk tier is outside platform backup.** Velero skips NFS volumes by volume type, so the exclusion +also covers any future export, while the same namespaces' Longhorn PVCs are still backed up. A +service's `backup.policy` therefore speaks for its Longhorn volumes only, and the bulk data's +durability is the NAS's own (see [known gaps](16-known-gaps.md)). ## Backup & disaster recovery (R-STATE-03) -- **Velero** backs up Kubernetes resource state (namespaces, manifests, PVC metadata) and - orchestrates volume backups. -- **Longhorn backups** push volume data to an **off-cluster** S3-compatible or NFS target — a - separate failure domain from the cluster itself. -- **Schedule**: regular automated backups (cadence per data criticality) with retention policy. +- The **off-cluster target is Cloudflare R2**: S3-compatible, off-site and a separate failure domain + from the cluster and from the NAS. Backup tools share one bucket under disjoint prefixes, so one + tool's retention can never reap another's objects. +- **Velero** backs up Kubernetes resource state (namespaces, manifests, PVC metadata) and, through + **kopia file-system backup**, the data of every volume. Backup is opt-out, so a new volume is + covered without anyone remembering to enrol it. +- **Longhorn has no backup target** ([ADR-0018](../decisions/adr-0018-storage-backup-truenas-r2.md)). + Velero and kopia are the single restore path for volume data, an accepted trade-off. +- **CloudNativePG** archives WAL and base backups to the same target, independently of Velero. +- **Schedule**: regular automated backups with retention policy. Databases archive continuously; + everything else runs nightly. +- **Exclusions are limited to reconstructable data**, whose Kubernetes resources are still captured: + monitoring's metrics and log volumes, and the UniFi VM's disk, for which the design relies on + UniFi's own configuration export ([06](06-vm-workloads-kubevirt.md)). - **Restore is a tested procedure**, not a theoretical one: a DR drill (restore a namespace + its volumes to a clean state) is part of the platform acceptance suite. -- Bulk media has its own backup posture (e.g. it may be re-acquirable rather than backed up 1:1 — - an explicit operator decision recorded per OQ-2), but **all irreplaceable state** (configs, +- Bulk media is outside this posture (above), but **all irreplaceable state** (configs, databases, Nextcloud data, Gitea repos) MUST be backed up off-cluster. ## KubeVirt VM disks Application-VM disks are Longhorn volumes provisioned via **CDI** (Containerized Data Importer). VMs -inherit the same durability and backup story as container PVCs — a KubeVirt VM can be destroyed and -rebuilt from its declarative definition + persistent disk. See [`06-vm-workloads-kubevirt.md`](06-vm-workloads-kubevirt.md). +share the durability of container PVCs and are backed up by default, with per-VM exceptions recorded +in [06](06-vm-workloads-kubevirt.md) — a KubeVirt VM can be destroyed and rebuilt from its declarative +definition + persistent disk. See [`06-vm-workloads-kubevirt.md`](06-vm-workloads-kubevirt.md). ## GitOps + state = full rebuildability @@ -78,21 +121,20 @@ rebuilt from its declarative definition + persistent disk. See [`06-vm-workloads - **Data** lives on Longhorn/bulk store + off-cluster backups. - Therefore a total cluster loss is recoverable: re-provision nodes ([02](02-environments-and-provisioning.md)), re-bootstrap Flux, let it reconcile from git, and restore volumes from backup. This end-to-end - recovery is the ultimate expression of R-STATE-01/03 and SHOULD be drilled at least once in - dev/test before prod is declared ready. + recovery is the ultimate expression of R-STATE-01/03 and SHOULD be drilled at least once, in a + planned maintenance window. ## Acceptance gates - Delete and recreate each stateful service; data survives (per-service rebuild test). -- Kill a node (dev/test); replicated volumes stay available; cluster re-converges. +- Kill a node in a planned maintenance window; replicated volumes stay available; cluster re-converges. - Restore a namespace + volumes from off-cluster backup into a clean state (DR drill). - Plex rebuilds with its metadata DB from replicated storage; media re-attaches from the bulk tier. ## Resolved -- **OQ-2** — bulk media lives **outside the cluster**; concrete backing **TBD later** (consumed as an - external NFS/SMB mount via the `bulk` indirection). Does not block the core platform. +- **OQ-2** — bulk media lives **outside the cluster**, on a TrueNAS appliance consumed over NFS through + the catalog's `class: bulk` indirection (ADR-0018). +- **OQ-BKP** — the off-cluster DR target for the *irreplaceable* tier is **Cloudflare R2** (ADR-0018). ## Open questions -- **Off-cluster DR backup target** (separate from bulk media): which S3-compatible endpoint or NFS - host holds Velero/Longhorn backups of the *irreplaceable* tier (app config, Nextcloud, Gitea, DBs), - and its own resilience. Needed by **Phase 4**, not Phase 2. -- Concrete bulk-media backing + whether any of it is non-`reacquirable` (operator decides when chosen). +- Whether any bulk data, notably the photo library, is non-`reacquirable` and needs its own backup + (operator decides). diff --git a/docs/05-gitops-and-delivery.md b/docs/05-gitops-and-delivery.md index e035119..18774ae 100644 --- a/docs/05-gitops-and-delivery.md +++ b/docs/05-gitops-and-delivery.md @@ -1,6 +1,10 @@ # 05 — GitOps & Delivery -Serves: **R-GITOPS-01..03**, **R-PLAT-02/04**, **R-ENV-02**. Related: [ADR-0004 (Flux)](../decisions/adr-0004-gitops-flux.md). +Serves: **R-GITOPS-01..03**, **R-PLAT-02/04**, **R-ENV-02**, **R-TEST-03**. Related: +[ADR-0004 (Flux)](../decisions/adr-0004-gitops-flux.md), +[ADR-0023 (updates)](../decisions/adr-0023-automated-updates-renovate.md), +[ADR-0024 (delivery authority)](../decisions/adr-0024-delivery-authority-ci-gate.md), +[ADR-0025 (pm-agent)](../decisions/adr-0025-in-cluster-ai-program-manager.md). ## Model @@ -8,35 +12,40 @@ Pull-based GitOps with **Flux CD**. The cluster continuously reconciles its stat the single source of truth; the cluster conforms to git, not the reverse (R-GITOPS-01). Drift is detected and corrected automatically (R-GITOPS-03). -## Repository layout (the implementation repo, created in a later phase) +## Repository layout (the implementation repo) -> This spec repo (`home-network-k3s`) holds intent. The **implementation** repo(s) hold the -> reconciled manifests. Proposed structure: +> This spec repo (`home-network-k3s`) holds intent. The +> [implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl) +> holds the reconciled manifests, the gate and the tests. Its top-level areas: ``` -platform/ ← cluster-wide platform services (Flux-managed) - cilium/ ← CNI, egress gateway, network baseline - longhorn/ ← storage - cert-manager/ ← TLS issuers + ClusterIssuer - externaldns/ ← DNS automation - gateway/ ← Gateway API + listeners/TLS - monitoring/ ← kube-prometheus-stack, Loki, Alloy, Alertmanager - velero/ ← backup/DR - kubevirt/ ← KubeVirt + CDI - image-automation/ ← Flux image update automation config -apps/ ← application workloads, one dir per service - plex/ sonarr/ radarr/ jackett/ seerr/ pihole/ nextcloud/ - minecraft/ unifi/ semaphore/ kasm/ gitea/ qbittorrent/ -catalog/ ← ServiceDefinition entries (the onboarding surface) +catalog/ ← ServiceDefinition entries (the onboarding surface) + the renderer +schemas/ ← ServiceDefinition JSON Schema +apps/ ← application workloads by reconcile wave, plus VMs; rendered from + catalog/ and committed (the render is checked against them) +platform/ ← cluster-wide platform services (Flux-managed): CNI, storage, TLS, DNS, + Gateway, public tunnel, observability, backup/DR, KubeVirt, identity, + admission policy, image builder, delivery automation +platform-config*/ ← platform configuration Flux applies after the platform tier, including + the admission policies clusters/ - dev/ ← Flux entrypoint + dev overlay/vars - prod/ ← Flux entrypoint + prod overlay/vars (design-only) + base/ ← shared Flux entrypoints and reconcile ordering + prod/ ← the live cluster's overlay and environment values +bootstrap/ ← Flux, SOPS and Gitea bootstrap +infra/ ← node provisioning as code; Gitea settings; dev toolbox; agent image +ci/ .gitea/ ← the merge gate and the workflows that run it +conformance/ ← requirement registry, behavior tests, harness, run ledger +runbooks/ ← operator procedures (also served to the alert links) +decisions/ ← dated implementation handoff log (distinct from this repo's ADRs) ``` -- **Kustomize overlays** (or equivalent) express environment differences (R-ENV-02). Base is shared; - `clusters/dev` and `clusters/prod` supply scoped values only. +- **Kustomize overlays** express environment differences (R-ENV-02). The base is shared; an + environment overlay supplies scoped values only. `prod` is the one live environment; the dev + environment was decommissioned in 2026-09 ([ADR-0017](../decisions/adr-0017-single-environment-bare-metal-production.md)). - **HelmReleases** (Flux) wrap upstream charts (Cilium, Longhorn, cert-manager, kube-prometheus- stack, etc.); app workloads use charts where good ones exist, otherwise plain manifests. +- **Rendered output is committed.** `apps/` is generated from `catalog/`; the merge gate fails when + the two disagree, so a ServiceDefinition is always the source of a rendered workload. ## The onboarding surface: ServiceDefinition (R-GITOPS-02) @@ -48,14 +57,52 @@ platform renders the rest. ## Reconciliation flow -1. Operator opens a PR against the implementation repo (new/updated ServiceDefinition or platform - change). -2. CI validates (lint, policy checks, schema validation, dry-run/build). -3. Merge to the tracked branch. -4. Flux reconciles: applies manifests, reports status, corrects drift on an interval. +1. A contributor opens a PR against the implementation repo (new/updated ServiceDefinition or + platform change). Contributors are the operator, implementation agents, Renovate and pm-agent; + all of them take the same path. +2. The **merge gate** validates the change (below). +3. Merge to `main`, the tracked branch. +4. Flux reconciles from the platform's own Gitea: applies manifests, reports status, corrects drift + on an interval. 5. Health/rollout status surfaces in the C2 UI ([12](12-command-center-ui.md)) and alerts on failure ([10](10-observability-and-alerting.md)). +Nothing reaches `main` by any other route: direct pushes are rejected for every account. + +## The merge gate (R-TEST-03) + +The implementation repo's `main` is protected by one required status check, the merge gate, which +runs on every PR as a Gitea Actions job. **A red gate is unmergeable by everyone, administrators +included.** The gate runs in the same pinned toolbox image locally and in CI (R-TEST-02) and +establishes, in order: the adopted spec pin (R-TEST-06), the handoff log's format, lint, L0 schema + +policy + repository invariants, render-equals-committed, L1 rendering, the sensitive-path approval +policy, that the live Gitea settings match their declaration, and L2/L3 conformance +([14](14-testing-and-conformance.md); see [known gaps](16-known-gaps.md)). + +Its verdict is one of three: **pass**, **blocked**, or **undetermined**. A stage that did not run is +never green, and no automation may read a non-answer as approval. The rule is that a capability the +environment *declares* absent is named in the verdict, while a stage that *could not run where it +must* blocks. Detail: `ci/README.md` in the implementation repo. + +## Who approves ([ADR-0024](../decisions/adr-0024-delivery-authority-ci-gate.md)) + +**The CI gate is the approver.** A green gate is sufficient for an ordinary change; branch +protection requires no approval count. A weak gate is a defect to fix, not a reason to add a human. + +**One exception: a change to the gate cannot be gated by the gate.** A change touching the pipeline, +the tests and harness, the Flux wiring, the admission policies, the SOPS recipients and secrets, the +RBAC, or the delivery contract needs an approving review from someone other than its author. The +path list is data, with a stated reason per entry (`infra/gitea/approval-policy.json` in the +implementation repo), and is itself a sensitive path. This stage is fail-closed: if it cannot +evaluate, it blocks. + +**Attributed administrator override.** There is one human on this platform, and Gitea refuses +self-approval, so an administrator may waive that review with an explicit, reasoned directive on the +PR. Authority is re-checked live, a new commit invalidates it, and the overrider, time and reason +are recorded in the gate output. For an administrator the exception above is therefore advisory, not +structural; what it guarantees is an attributed record of each waiver. **Agents never post this +directive to clear their own work**, whatever permission their account holds. + ## Secrets in GitOps Secrets are committed **encrypted** with **SOPS + age**; Flux decrypts them in-cluster at @@ -64,15 +111,32 @@ and [ADR-0007](../decisions/adr-0007-secrets-sops-age.md). ## Automated updates interplay (R-UPD-*) -GitOps and hands-off updates coexist: -- **Container image tags** advance automatically via **Flux image-update-automation** driven by - per-service **image policies** (semver range / stable channel) — the controller resolves and - reconciles the new version with **no human-authored change** (R-UPD-02). -- **Helm chart / manifest version bumps** are proposed by **Renovate** with auto-merge for allowed - update classes. +GitOps and hands-off updates coexist because the update bot is one more contributor to the same +flow. **Renovate** runs on a schedule and proposes machine-authored PRs for container images, Helm +charts and manifest versions; each passes the merge gate, and allowed update classes auto-merge on +green (R-UPD-02). No in-cluster controller holds a credential to write to the repo that defines the +cluster. Full detail is in [`11-automated-updates.md`](11-automated-updates.md) and +[ADR-0023](../decisions/adr-0023-automated-updates-renovate.md). -Full detail and the "is an automated commit a GitOps change?" nuance is in -[`11-automated-updates.md`](11-automated-updates.md) and [ADR-0009](../decisions/adr-0009-automated-image-updates.md). +## Images are built by CI + +Container images the platform itself produces (the toolbox, the agent runner, the conformance +runner) are built, verified and published only by the pipeline, never from a workstation. A PR +builds and verifies; a push to `main` publishes to the Gitea registry. References are pinned by +digest, and Renovate carries each new digest into git as a PR. **No CI job may reach a privileged +container daemon**: builds go to a rootless builder over mutually-authenticated TLS. + +## Delivery automation: pm-agent ([ADR-0025](../decisions/adr-0025-in-cluster-ai-program-manager.md)) + +An in-cluster AI Program Manager reads labelled issues on the implementation repo, decides what +should happen about each thread, and runs one worker agent per assignment to author a change, run +the merge gate and open a PR. It is delivery tooling rather than a catalog service: it declares +Kubernetes objects a ServiceDefinition cannot express, and it requires no `R-*` of its own. + +It is contributor-shaped, never approver-shaped. It cannot push to `main`, it never merges, its +account is not an administrator, and its workers hold no cluster credential. It is an implementation +agent bound by [AGENTS.md](../AGENTS.md) and R-TEST-06: spec work is out of its reach and is +declined and escalated. ## Bootstrap & recovery @@ -80,25 +144,21 @@ Full detail and the "is an automated commit a GitOps change?" nuance is in - Because desired state is in git and data is on Longhorn/backups, a rebuilt cluster reconverges by re-bootstrapping Flux and restoring volumes ([04](04-storage-and-state.md)). -## Source-of-truth: local bootstrap → self-hosted Gitea +## Source-of-truth: self-hosted Gitea -The platform hosts its own git, which creates a deliberate bootstrapping sequence -([`../CONTRIBUTING.md`](../CONTRIBUTING.md)): +The platform hosts its own git at `gitea.home.keithjackson.dev` +([`../CONTRIBUTING.md`](../CONTRIBUTING.md)); Flux reconciles from it and Gitea Actions runs the +merge gate. Standing the platform up from nothing follows a deliberate order: -1. **Bootstrap source.** Initially there is no remote git on this network (the prior `home-network` - Gitea is gone from here). Development is local; the cluster is first reconciled from a - **bootstrap source** (this local repo / an out-of-band source). -2. **Gitea comes online early.** **Gitea is one of the first services onboarded** ([13](13-service-catalog.md)) - so the platform gains its own git host at `gitea.home.keithjackson.dev`. -3. **Cut Flux over to Gitea.** Once Gitea is stable, Flux's `GitRepository` source is migrated from - the bootstrap source to Gitea; the platform then reconciles from its own git. -4. **CI/CD moves to Gitea Actions.** The Kyverno L0 policy gate and the trigger for the in-cluster - Chainsaw conformance Job ([14](14-testing-and-conformance.md)) run as Gitea Actions pipelines on - push/PR; PRs become the enforced merge gate into `main`. +1. **Bootstrap source.** The cluster is first reconciled from an out-of-band copy of the repo. +2. **Gitea comes online early.** Gitea is one of the first services onboarded + ([13](13-service-catalog.md)). +3. **Cut Flux over to Gitea**, so the platform reconciles from its own git. +4. **CI/CD runs in Gitea Actions**, with PRs as the enforced merge gate into `main`. **Circularity & DR.** Gitea runs *on* the cluster Flux reconciles *from* Gitea. Mitigations: - Keep an **off-cluster mirror** of the platform repo so a source-of-truth exists that outlives the - cluster. + cluster (see [known gaps](16-known-gaps.md)). - Treat **Gitea as DR-critical**: its repos/DB are backed up off-cluster ([04](04-storage-and-state.md)), so a cluster loss never loses the git history needed to rebuild the platform. @@ -112,4 +172,4 @@ The platform hosts its own git, which creates a deliberate bootstrapping sequenc - Adding a ServiceDefinition PR results in a fully wired service (compute, storage, policy, TLS, DNS, monitoring) with no manual steps beyond the merge. - Manually mutating a live resource is auto-reverted by Flux (drift correction proven). -- The same base config renders correctly for both dev and prod overlays. +- The base config renders correctly under the environment overlay, which supplies scoped values only. diff --git a/docs/06-vm-workloads-kubevirt.md b/docs/06-vm-workloads-kubevirt.md index a070f5d..d38eaec 100644 --- a/docs/06-vm-workloads-kubevirt.md +++ b/docs/06-vm-workloads-kubevirt.md @@ -12,8 +12,8 @@ Kubernetes custom resources. ## Do not confuse the two VM layers -- **Layer 1 — cluster nodes** are VMs only in dev/test (libvirt) and bare metal in prod. Managed by - OpenTofu/Ansible ([02](02-environments-and-provisioning.md)). **Not** KubeVirt. +- **Layer 1 — cluster nodes** are bare-metal mini-PCs, baselined and managed as code + ([02](02-environments-and-provisioning.md)). **Not** KubeVirt. - **Layer 2 — application VMs** run *inside* the cluster via KubeVirt and are the subject of this spec. Managed by Kubernetes/GitOps like any other workload. @@ -22,7 +22,8 @@ Kubernetes custom resources. - `VirtualMachine` / `VirtualMachineInstance` custom resources — declare a VM in YAML, reconcile via Flux, manage via `kubectl` (and `virtctl` for console/start/stop). - **CDI (Containerized Data Importer)** to import/clone disk images into **Longhorn** PVCs — VM disks - get the same replication and backup story as container volumes ([04](04-storage-and-state.md)). + get the same replication as container volumes and are backed up by default + ([04](04-storage-and-state.md)). - VMs sit on the **same Cilium network** — so **default-deny isolation and egress classes apply to VMs too** (R-NET-*). A KubeVirt VM can be `egress: vpn` just like a container. - VMs are exposed through the same **Gateway API / Service** model and monitored through the same @@ -39,6 +40,14 @@ A VM onboarded via a ServiceDefinition (`kind: vm`) gets, from the *same* declar - Backup/DR of its disks (R-STATE-03) and rebuildability (R-STATE-01): destroy the VM, keep the disk PVC, recreate from the declaration. +`kind: vm` is the catalog path. A VM with a custom install flow MAY instead be declared directly in +git under the same controls: Flux reconciliation, default-deny networking, Longhorn disks and +platform monitoring, in its own VM tier that never gates the application waves. + +**Recorded per-VM exception:** UniFi's disk is excluded from volume backup; its Kubernetes resources +are captured, and the design relies on UniFi's own configuration export for application state +([04](04-storage-and-state.md)). + ## Update policy for VMs (R-UPD-*) VM "updates" differ from container image tag bumps: @@ -59,17 +68,22 @@ Prefer containers (PaaS) by default. Choose a KubeVirt VM (IaaS) when: - Lifting an existing VM is faster/safer than re-platforming. - Strong OS-level isolation is required beyond namespace/network policy. -For the initial catalog, all 13 named services are container-first; KubeVirt is the escape hatch and -must be **proven** in dev/test with at least one representative VM workload so the IaaS path is real, -not theoretical. +The catalog is container-first and KubeVirt is the escape hatch. **UniFi is the production VM +workload**: UniFi OS Server is an installer that expects a full Ubuntu host and runs its own container +stack and embedded datastore, so it is an appliance image in all but name. It is hand-authored in git +under the rule above. The catalog's `vm-demo` entry is the `kind: vm` reference and conformance +fixture and is not deployed. The IaaS path must be **proven** with at least one representative VM +workload so it is real, not theoretical. ## Prerequisites & constraints -- Nodes must expose virtualization extensions (nested virt in dev/test KVMs; native on prod mini- - PCs) — captured in the node baseline ([02](02-environments-and-provisioning.md)). -- Live migration across 3 converged nodes is a nice-to-have, not a requirement; document whether it's - enabled. +- Nodes must expose virtualization extensions (native on the mini-PCs) — captured in the node baseline + ([02](02-environments-and-provisioning.md)). VMs never fall back silently to software emulation: a + node without KVM fails loudly. +- Live migration is **enabled and opt-in per VM**: the VM sets a live-migrate eviction strategy and + keeps its disks on the migratable Longhorn class. There is no cluster-wide policy, because forcing + it would wedge node drains for VMs on non-migratable disks. Other VMs restart on drain. - Resource pressure: VMs are heavier than pods — capacity planning must account for them, - particularly on the 3-node prod cluster. + particularly on the 3-node cluster. ## Acceptance gates - Create, start, stop, and destroy a VM entirely through Kubernetes YAML + `kubectl`/`virtctl`. diff --git a/docs/07-security-and-secrets.md b/docs/07-security-and-secrets.md index 0ca36e1..45c25b7 100644 --- a/docs/07-security-and-secrets.md +++ b/docs/07-security-and-secrets.md @@ -17,22 +17,32 @@ Defense in depth with declarative, auditable controls: - **Supply chain**: pin images by digest where automation allows; prefer known-good upstreams; updates flow through policy, not ad-hoc pulls ([11](11-automated-updates.md)). - **Identity / SSO**: a central IdP (**Keycloak**) provides single sign-on, one user directory, - group-based authorization, and MFA; services authenticate via native OIDC or an oauth2-proxy - forward-auth layer so no management UI is unauthenticated ([15](15-identity-and-access.md), - [ADR-0014](../decisions/adr-0014-idp-keycloak.md), R-IDP-*). This is the primary *authentication* - control, complementing network default-deny (the primary *isolation* control). -- **Policy enforcement (admission)**: **Kyverno** runs in-cluster as an admission controller and - rejects non-conforming resources at reconcile time — e.g. a namespace missing its default-deny - baseline, or a workload without a required egress-class/auth-mode label. The *same* policies - gate CI ([14](14-testing-and-conformance.md), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md)), - so invariants are enforced both before merge and at admission (defense in depth). Enforcement - complements — never replaces — the runtime behavior tests. + group-based authorization, and MFA; services authenticate via native OIDC or a per-app + oauth2-proxy forward-auth gate so no management UI is unauthenticated + ([15](15-identity-and-access.md), [ADR-0014](../decisions/adr-0014-idp-keycloak.md), + [ADR-0022](../decisions/adr-0022-forward-auth-per-app-oauth2-proxy-sidecars.md), R-IDP-*). This is + the primary *authentication* control, complementing network default-deny (the primary *isolation* + control). +- **Policy enforcement (admission)**: **Kyverno** runs in-cluster as an admission controller over + **application namespaces**. It generates and backfills each namespace's default-deny baseline, + rejects a malformed baseline, and rejects a workload that lacks a required egress-class/auth-mode + label or uses host namespaces or privileged mode. Separate enforcing policies constrain the exact + Job shapes that CI and agent credentials may create; these are security boundaries, so they fail + closed and no environment overlay can downgrade them. No policy-exception mechanism is in use: + controller-generated pods are mutated to carry truthful labels instead. Platform namespaces sit + outside these policies and carry hand-authored equivalents. Policies are authored once and gate + CI ([14](14-testing-and-conformance.md), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md)), + so invariants are enforced both before merge and at admission (defense in depth); the job-shape + policies act only at admission and are unit-tested in CI. Enforcement complements — never + replaces — the runtime behavior tests. - **Plaintext-secret enforcement is split by layer (do not conflate):** the "no plaintext Secret" invariant is a **CI-time** check (the committed Secret must carry SOPS ciphertext). It **cannot** be an admission rule, because Flux **decrypts SOPS before** the object reaches the API server — an admitted Secret never carries ciphertext, so an admission "reject plaintext" rule would reject - *every* real Secret. Admission instead enforces **SOPS provenance** (the Secret was reconciled - through the Flux/SOPS path), not ciphertext presence. + *every* real Secret. Admission instead enforces **SOPS provenance** on application-namespace + Secrets (the Secret was reconciled through the Flux/SOPS path), not ciphertext presence. + - Host-namespace and privileged-pod control is Kyverno-based for the application tier; Pod + Security Admission labels are applied selectively, on namespaces with their own need for them. - **Auditability**: isolation, egress path, cert validity, and backup integrity are all demonstrable (R-SEC-03) via Hubble, cert status, and DR drills. @@ -53,8 +63,13 @@ Defense in depth with declarative, auditable controls: - DNS provider API token (Cloudflare) for ACME DNS-01 + ExternalDNS. - Cloudflare Tunnel credential/token for in-cluster cloudflared (SOPS-encrypted; ADR-0016). - SMTP credentials for Alertmanager email. -- Service credentials (Nextcloud, Gitea admin, UniFi, Semaphore, Kasm, DB passwords). -- Off-cluster backup target credentials (S3/NFS) for Velero/Longhorn. +- Service credentials (Nextcloud, Gitea admin, UniFi, Semaphore, DB passwords; Kasm is not onboarded). +- Per-app forward-auth gate credentials (IdP client secret + cookie secret; [ADR-0022](../decisions/adr-0022-forward-auth-per-app-oauth2-proxy-sidecars.md)). +- WireGuard ingress server key, held in SOPS because a rebuild must restore it without re-enrolling + every client (ADR-0021). +- Off-cluster backup target credentials (Cloudflare R2) for Velero and the database archives (ADR-0018). +- In-cluster AI agent credentials (ADR-0025) and infrastructure-as-code runtime secrets (state + encryption, remote-state keys). - Grafana admin, and any API tokens the C2 UI needs. ### Rotation @@ -73,10 +88,14 @@ Defense in depth with declarative, auditable controls: - Platform controllers (Flux, cert-manager, ExternalDNS, Velero, KubeVirt) get scoped permissions; application workloads get near-zero cluster permissions. - The C2 UI's operational actions run through a constrained, audited path — not a shared admin - kubeconfig handed to a browser. + kubeconfig handed to a browser. The Kubernetes console forwards each user's own IdP-issued token, + so an action runs under that user's group-bound RBAC; the console's ServiceAccount holds no + cluster rights (see [known gaps](16-known-gaps.md)). +- Human access to the Kubernetes API is Keycloak OIDC, with roles bound to IdP groups + ([15](15-identity-and-access.md)). ## Node & host security -- Nodes baselined via cloud-init/Ansible ([02](02-environments-and-provisioning.md)); SSH key-only, +- Nodes baselined as code with Ansible ([02](02-environments-and-provisioning.md)); SSH key-only, minimal packages, required sysctls only. - Host-level changes are IaC, not hand edits (drift + auditability). diff --git a/docs/08-tls-and-certificates.md b/docs/08-tls-and-certificates.md index 65a434c..5289eb4 100644 --- a/docs/08-tls-and-certificates.md +++ b/docs/08-tls-and-certificates.md @@ -26,14 +26,15 @@ services (R-TLS-01/02/03). - Certificates are requested declaratively — either a **wildcard** `*.home.keithjackson.dev` cert shared by the Gateway, or **per-service** certs, decided per ADR-0006. Recommendation: a wildcard cert for `*.home.keithjackson.dev` terminated at the Gateway simplifies onboarding (a new service - needs no new cert), with per-service certs available where isolation warrants. + needs no new cert), with per-service certs available where isolation warrants (Plex's L4 VIP name + has its own). - The DNS provider API token is a SOPS-encrypted secret ([07](07-security-and-secrets.md)). ## DNS-01 challenge access (OQ-4 — RESOLVED: scoped Cloudflare API token) -Cloudflare already has a wildcard `*.keithjackson.dev` → public-IP record, which does not cover -certificate issuance (ACME DNS-01 must **write `_acme-challenge` TXT records**, -and that wildcard does not match two-label-deep names like `plex.home.keithjackson.dev`). +Certificate issuance needs programmatic write access to the zone: ACME DNS-01 must **write +`_acme-challenge` TXT records**, including for two-label-deep names like +`plex.home.keithjackson.dev`. **Decision (OQ-4):** use a **scoped Cloudflare API token** — `Zone·DNS·Edit` limited to the `keithjackson.dev` zone — stored **SOPS-encrypted** ([07](07-security-and-secrets.md)). With it: @@ -42,6 +43,7 @@ and that wildcard does not match two-label-deep names like `plex.home.keithjacks services (R-TLS-03). - **ExternalDNS** manages the `*.home.keithjackson.dev` records automatically ([09](09-dns-and-ingress.md), R-DNS-02) — no manual record editing for public services. +- **cloudflare-ddns** keeps the DNS-only `wan.` record current for WireGuard ([09](09-dns-and-ingress.md)). **Operator prerequisite:** create the scoped token in Cloudflare and hand it over once; it is then committed SOPS-encrypted and never touched manually again. The token is least-privilege (single zone, @@ -71,9 +73,10 @@ DNS edit only) so its blast radius is bounded (R-SEC-02). Rotation = re-encrypt ## Trust chain for internal clients -Because certs come from Let's Encrypt (publicly trusted), internal clients trust them with no custom -CA distribution. Split-horizon DNS ([09](09-dns-and-ingress.md)) resolves -`.home.keithjackson.dev` to the internal Gateway while the same trusted cert is served. +Both certificates a client can meet are publicly trusted, so internal clients need no custom CA +distribution. A name Pi-hole holds a local record for ([09](09-dns-and-ingress.md)) resolves to the +internal Gateway and serves the Let's Encrypt wildcard; any other `public` name resolves to +Cloudflare's edge and serves its edge certificate. ## Acceptance gates - `https://.home.keithjackson.dev` serves a valid, non-self-signed, unexpired certificate @@ -84,6 +87,5 @@ CA distribution. Split-horizon DNS ([09](09-dns-and-ingress.md)) resolves ## Resolved - **OQ-4** — DNS-01 via a **scoped Cloudflare API token** (`Zone·DNS·Edit`, `keithjackson.dev`), - SOPS-encrypted; drives both cert-manager DNS-01 and ExternalDNS. Phase 2 TLS/DNS is unblocked once - the operator provides the token. + SOPS-encrypted; drives cert-manager DNS-01, ExternalDNS and the `wan.` record updater. - Wildcard vs. per-service certs default — **wildcard** `*.home.keithjackson.dev` (ADR-0006). diff --git a/docs/09-dns-and-ingress.md b/docs/09-dns-and-ingress.md index e44070b..71c15a2 100644 --- a/docs/09-dns-and-ingress.md +++ b/docs/09-dns-and-ingress.md @@ -2,7 +2,8 @@ Serves: **R-DNS-01/02/03/04/05**, **R-C2-01**, **R-SEC-04**, works with **R-TLS-***. Related: [ADR-0002 (Cilium/Gateway API)](../decisions/adr-0002-cni-cilium.md), -[ADR-0016 (Cloudflare Tunnel public ingress)](../decisions/adr-0016-public-ingress-cloudflare-tunnel.md). +[ADR-0016 (Cloudflare Tunnel public ingress)](../decisions/adr-0016-public-ingress-cloudflare-tunnel.md), +[ADR-0021 (inbound remote access)](../decisions/adr-0021-inbound-remote-access-wireguard.md). ## Naming model @@ -18,8 +19,10 @@ Serves: **R-DNS-01/02/03/04/05**, **R-C2-01**, **R-SEC-04**, works with **R-TLS- authoritative DNS provider (Cloudflare, per OQ-4) automatically — a new service's DNS appears from its declared intent, no manual record editing. - Records for `*.home.keithjackson.dev` are created/updated/removed as services come and go. -- Public records are created **only for `exposure: public` services** (R-DNS-05). The DNS-only - (unproxied) `wan.` record for WireGuard is kept current by cloudflare-ddns, outside this flow. +- Public records are created **only for services declared public** (see the exposure class below, + R-DNS-05). The non-HTTP L4 names **unifi** and **crafty-mc** are the one exception: they hold DNS-only + records at their LAN addresses, and are never proxied or tunnelled. The DNS-only (unproxied) `wan.` + record for WireGuard is kept current by cloudflare-ddns, outside this flow. ## Split-horizon resolution (R-DNS-03) @@ -27,7 +30,7 @@ Internal and external answers differ: - **External** (`.home.keithjackson.dev` from the internet): a `public` service resolves to **Cloudflare's edge** (proxied CNAME to the tunnel). Cloudflare forwards the request down the outbound-only tunnel to cloudflared in the cluster, which forwards it to the Gateway (ADR-0016). - `internal` services have no public answer. + `internal` services have no public answer, apart from the L4 exception above. - **Internal** (LAN clients): Pi-hole answers with the **internal Gateway address** (or the service's L4 VIP) for the names it holds local records for. - **Correctness does not depend on split horizon for `public` web apps** (R-DNS-04). The ISP router @@ -40,6 +43,11 @@ Internal and external answers differ: - the names that must avoid Cloudflare's 100 MB body / 100 s timeout limits or must work during an internet outage: **gitea**, **nextcloud**, **keycloak/auth**. + It holds no zone-wide entry. Every other name is forwarded upstream, so the remaining `public` names + resolve to Cloudflare from the LAN too and depend on the internet and the tunnel. Its DNS address + accepts queries only from the LAN and WireGuard clients. Pi-hole runs in the cluster, so LAN DNS + depends on cluster health; an off-cluster resolver is an operator choice not taken. + A trusted certificate is served either way: Cloudflare's edge certificate on the tunnel path, the Let's Encrypt wildcard on the LAN path ([08](08-tls-and-certificates.md)). - **In-cluster** resolution keeps its own split horizon (CoreDNS) and the node `/etc/hosts` pins (OIDC @@ -58,11 +66,17 @@ Internal and external answers differ: - Each service contributes an **HTTPRoute** mapping its hostname to its Service — rendered from the ServiceDefinition, not hand-authored. - **Exposure class** is declared per service: - - `internal` — reachable only on the LAN/VPN (no tunnel route, no public record; internal DNS only). + - `internal` — reachable only on the LAN/VPN (refused at the tunnel, no public record; internal DNS only). - `public` — additionally reachable from the internet through the **Cloudflare Tunnel**: - cloudflared → the Gateway over HTTPS (ADR-0016). + cloudflared → the Gateway over HTTPS (ADR-0016). A service is public by catalog + `exposure: public`, or as a **registered platform route**: `auth`, Grafana, Crafty's web UI, and + the apex with its `/runbooks`. Platform routes have no catalog entry, so they are held in a + shrink-only register, enforced by the static check + `conformance/tests/l0-static/internal-services-publish-no-public-dns` in the + [implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl). - **Never `public`:** jackett, pihole, semaphore, qbittorrent, radarr, sonarr, **headlamp** (a front - end to the Kubernetes API), plex's web UI, and the non-HTTP L4 services unifi and crafty-mc. + end to the Kubernetes API), plex's web UI, and the non-HTTP L4 services unifi and crafty-mc (which + hold only the DNS-only LAN records above). - **The Kubernetes API is never published** — no tunnel route, no edge forward, no public record (R-SEC-04). @@ -70,9 +84,12 @@ Internal and external answers differ: - **cloudflared** runs in-cluster as a Flux-managed workload and holds an **outbound-only** connection to Cloudflare. No inbound 80/443 reaches the WAN IP. -- Its ingress rules list one route per `public` hostname → the Gateway (HTTPS, origin certificate - validated with the hostname as SNI). The rules end in a **catch-all `http_status:404`**, so an - unlisted hostname is refused at the tunnel (R-DNS-05). +- Its ingress rules are a **deny-list plus a zone wildcard**. Each LAN/VPN-only hostname maps to + `http_status:404`; the apex and `*.home.keithjackson.dev` then go to the Gateway (HTTPS, origin + certificate validated with the hostname as SNI); a **catch-all `http_status:404`** closes the list + for hosts outside the zone (R-DNS-05). A static check keeps the deny-list and the set of unpublished + routes identical in both directions, so a LAN/VPN-only name cannot be published or tunnelled by + omission. - **Edge TLS** is Cloudflare ACM/Total TLS (Universal SSL does not cover two-label-deep names); **origin TLS** is the Let's Encrypt wildcard at the Gateway ([08](08-tls-and-certificates.md)). - **Limits and trade-offs:** @@ -86,7 +103,7 @@ Internal and external answers differ: - The environment has **one SOHO edge router**: no VLANs, no VPN, no services, no API/IaC. It **does not forward 80/443** (public web ingress is the tunnel, ADR-0016). Its only remaining - inbound forwards are **WireGuard udp/51820** (remote access, via the DNS-only `wan.` record) and + inbound forwards are **WireGuard udp/51820** (remote access, ADR-0021, via the DNS-only `wan.` record) and **Plex tcp/32400** (remote streaming via plex.tv; Cloudflare's terms restrict proxied video). This is a documented manual edge step (the SOHO device has no automation), the analogue of the old pfSense manual-config exception. The WAN IP therefore stays public. @@ -111,7 +128,9 @@ but **not** `plex.home.keithjackson.dev`. Consequences: Ingress opens **north-south** paths to the specific service; it does **not** relax **east-west** default-deny ([03](03-networking-isolation-and-egress.md)). A service being publicly reachable does -not let other in-cluster services reach it unless an explicit allow rule exists. +not let other in-cluster services reach it unless an explicit allow rule exists. Cilium's Gateway +re-evaluates the calling pod's egress policy against the real backend, so any in-cluster client of the +Gateway, cloudflared included, needs an explicit egress allow to the backends it reaches. ## Acceptance gates - A new service's `.home.keithjackson.dev` record appears automatically and routes to it. @@ -121,8 +140,8 @@ not let other in-cluster services reach it unless an explicit allow rule exists. records for, and get a trusted cert. - A LAN client resolving a `public` name through a **public** resolver reaches it, TLS-valid (R-DNS-04). The WAN IP does not answer on 80/443. -- A request at the tunnel for any hostname that is not a `public` service returns **404**, and no - `internal` service has a public DNS record (R-DNS-05). +- A request at the tunnel for a LAN/VPN-only hostname returns **404**, and no `internal` service has + a public DNS record, except the DNS-only L4 names (R-DNS-05). - The Kubernetes API is unreachable from the internet: no tunnel route, no edge forward, no public record (R-SEC-04). - The root `home.keithjackson.dev` serves the C2 UI. diff --git a/docs/10-observability-and-alerting.md b/docs/10-observability-and-alerting.md index cd391b2..e94ecb3 100644 --- a/docs/10-observability-and-alerting.md +++ b/docs/10-observability-and-alerting.md @@ -7,7 +7,7 @@ Serves: **R-OBS-01/02/03/04**, supports **R-C2-02**, **R-SEC-03**. - **kube-prometheus-stack** — Prometheus (metrics), Alertmanager (routing/notify), Grafana (dashboards), and the standard Kubernetes exporters/rules. - **Loki** — log aggregation. -- **Grafana Alloy** (or Promtail-class agent) — ships node/pod/VM logs to Loki. +- **Grafana Alloy** — ships node/pod/VM logs to Loki. - **Hubble** — network flow visibility from Cilium ([03](03-networking-isolation-and-egress.md)). - **Alertmanager → SMTP** — issues reported by **email** (R-OBS-02). - **Grafana** — dashboards for services and platform health (R-OBS-03). @@ -17,18 +17,29 @@ Serves: **R-OBS-01/02/03/04**, supports **R-C2-02**, **R-SEC-03**. Application services: - Up/health, resource use, request/error signals where exposed, and per-service SLIs declared in the ServiceDefinition (a service can declare the alerts it cares about). +- Each forward-auth gate ([15](15-identity-and-access.md)) is scraped in its own right, and the + platform alerts on a gate that nothing observes: an auth failure otherwise looks like a healthy + response. Platform health (R-OBS-04) — first-class alerts: -- **Nodes**: down/unreachable, resource pressure, disk fill. -- **Storage (Longhorn)**: degraded/faulted volumes, replica rebuilds, capacity thresholds. +- **Nodes**: down/unreachable, CPU, memory and I/O pressure, disk fill, disk latency. +- **Storage (Longhorn + NFS bulk tier)**: degraded/faulted volumes, replica rebuilds, capacity + thresholds, PVC capacity, NFS server latency and client timeouts. +- **Databases (CloudNativePG)**: archiving failures, missed backups, primary placement. - **GitOps (Flux)**: reconciliation failures, drift, suspended resources. +- **Automated updates**: image automation failures, stalled image policies, failed health gates, + node reboot automation. - **Certificates (cert-manager)**: upcoming expiry, issuance failures. - **Networking**: VPN tunnel down, egress-gateway unhealthy, spikes in denied flows (Hubble), - NetworkPolicy reconcile errors. + NetworkPolicy reconcile errors, public ingress tunnel down. - **Backups (Velero/Longhorn)**: backup job failure, missed schedule, restore-test failures. - **KubeVirt**: VM crash-loops / failed launches. - **Identity (Keycloak)**: IdP down/unhealthy or its DB unavailable (SSO is a critical dependency — - [15](15-identity-and-access.md)); certificate/login-endpoint failures. + [15](15-identity-and-access.md)); certificate/login-endpoint failures; a failing or stale + identity-config reconciler. +- **The monitor itself**: monitoring volume capacity, missing scrape targets. + +Coverage against this list is tracked in [known gaps](16-known-gaps.md). ## Alert delivery (R-OBS-02) @@ -36,7 +47,12 @@ Platform health (R-OBS-04) — first-class alerts: - Severity model: `critical` (page-worthy: data risk, outage, VPN leak potential, cert about to expire) vs `warning` (degraded, capacity, needs attention). Both go to email; only routing/subject differs. -- Alerts include enough context (runbook link, affected object) to act without spelunking. +- **Deploy notices** are a separate, non-severity notification class, also by email: one message + when Flux starts applying a revision and one when every Kustomization has reached it, so a routine + deploy never reads as a platform fault. +- Alerts include enough context (runbook link, affected object) to act without spelunking. Every + alert's runbook link resolves to a runbook served under `/runbooks` on the apex, and every rule has + a unit test; the merge gate checks both. ## Dashboards (R-OBS-03) @@ -49,13 +65,16 @@ Grafana dashboards, surfaced in the C2 UI ([12](12-command-center-ui.md)): ## Observability as onboarding output The ServiceDefinition ([13](13-service-catalog.md)) declares observability expectations; the -platform renders **ServiceMonitors/PodMonitors**, log scraping, dashboards, and alert rules from it. -A newly onboarded service is monitored **by default**, not as a follow-up task (part of the golden -path, [01](01-architecture-overview.md)). +platform renders **ServiceMonitors/PodMonitors** for services that declare a metrics endpoint, and +dashboards and alert rules from its declared SLIs. Log collection needs no per-service rendering: +the log agent discovers every pod. A newly onboarded service is monitored **by default**, not as a +follow-up task (part of the golden path, [01](01-architecture-overview.md)). ## Retention & sizing -- Metrics/log retention set per environment overlay (dev shorter, prod longer). Loki retention - requires a compactor with retention enabled — do not rely on a period setting alone. +- Metrics/log retention is set in the environment overlay. Metrics retention is capped by size as + well as age, so a young or busy TSDB cannot fill its volume and blind the alerts that depend on + it. Loki retention requires a compactor with retention enabled — do not rely on a period setting + alone. - Monitoring storage lives on Longhorn; ensure its own capacity is alerted on (don't let the monitor blind itself). diff --git a/docs/11-automated-updates.md b/docs/11-automated-updates.md index e576bad..66dd7aa 100644 --- a/docs/11-automated-updates.md +++ b/docs/11-automated-updates.md @@ -1,62 +1,80 @@ # 11 — Automated Updates -Serves: **R-UPD-01/02/03**. Related: [ADR-0009](../decisions/adr-0009-automated-image-updates.md). +Serves: **R-UPD-01/02/03**. Related: [ADR-0023](../decisions/adr-0023-automated-updates-renovate.md). ## The requirement, precisely -> Services must update to latest **stable** versions **on a schedule**, with **no human, no AI, and -> no hand-authored GitOps change**. (R-UPD-01/02) +> Services must update to latest **stable** versions **on a schedule**; updates in the allowed +> classes occur with **no human, no AI, and no hand-authored GitOps change**. (R-UPD-01/02) This has a subtle tension with GitOps (where "the desired state is what's in git"). We resolve it explicitly below. ## Resolving "no GitOps change" vs. GitOps -There are two interpretations; we honor the operator's intent (hands-off), and ADR-0009 records the +There are two interpretations; we honor the operator's intent (hands-off), and ADR-0023 records the mechanism: - **A human/AI authoring a version bump = prohibited.** No person and no chatbot edits a tag. -- **An automation controller advancing a version under a policy declared once = permitted**, because - the *operator's* declared intent (the update policy / semver range) is the stable, human-authored - thing; the controller merely resolves it. Two mechanisms, chosen by artifact type: + Reviewing a machine-authored change is not authoring one. +- **An automation bot proposing a version advance under a policy declared once = permitted**, + because the *operator's* declared intent (the update policy / version range) is the stable, + human-authored thing; the bot merely resolves it, and its change reaches `main` through the same + merge gate as every other change ([05](05-gitops-and-delivery.md)). Mechanisms by artifact type: -### 1. Container image tags → Flux image-update-automation -- Per service, an **ImagePolicy** declares the acceptable range/channel (e.g. semver `>=1.0.0` - minor/patch, or a `stable` tag pattern). Flux's image-reflector scans the registry; image- - automation advances the running version automatically **on its scan interval** (the schedule). -- Where automation writes the resolved version back to git, that commit is machine-authored under - the declared policy — no human/AI touches it (satisfies R-UPD-02 in spirit and practice). -- Alternatively, digest-tracking of a mutable stable tag avoids per-update commits entirely. ADR-0009 - picks the default and the per-service escape hatch. +### 1. Container images, Helm charts and manifests → Renovate +- **Renovate** runs on a schedule, watches image tags, chart and dependency versions, and opens PRs. + Git history records every advance as an auditable, machine-authored commit. +- **Minor, patch and digest** updates **auto-merge** on a green gate. This keeps services and + platform components current with no human authoring. +- **Major** updates never auto-merge; they wait for human review of the machine-authored PR. +- **Operator-designated blast-radius components** never auto-merge, not even a patch: the storage and + network planes, the reboot daemon, the database engine, the metrics stack, Gitea and its runner + (they *are* the merge gate), the identity provider and its auth sidecars, and Renovate itself and + the pinned CI toolchain. The list is data in the implementation repo's `renovate.json5`, each entry + with its reason. +- Because `apps/` is rendered from `catalog/`, a bump updates the source and its rendered output in + one PR, so render-equals-committed holds. -### 2. Helm charts / manifests → Renovate -- **Renovate** watches chart and dependency versions and opens PRs; allowed update classes - **auto-merge** on green CI. This keeps platform components current with no human authoring. -- Chart *major* bumps (potentially breaking) can require review — configurable per policy. +### 2. The platform's own images → Renovate digest tracking +- Images built by CI ([05](05-gitops-and-delivery.md)) are referenced by digest. Renovate follows + each image's moving tag and proposes the digest bump as a PR, whose own gate run exercises it where + the image is the toolbox. ### 3. VM guests → in-guest automation - KubeVirt guests auto-update OS packages inside the guest on a schedule (unattended-upgrades / dnf- automatic) declared in the VM's cloud-init ([06](06-vm-workloads-kubevirt.md)). Base-image promotion follows the same automated philosophy where a golden-image pipeline exists. +### 4. Node OS → security-only upgrades, coordinated reboots +- Nodes apply **security-only** unattended upgrades, with kernels excluded. A reboot daemon performs + only the mechanical half of a reboot, when a node requests one: one node at a time under a single + cluster-wide lock, gated on storage health so etcd quorum and volume replicas are never put at + risk. A hang stops the cycle and pages a human. Kernel upgrades stay a deliberate operator act. + ## Per-service policy (R-UPD-03) -Each ServiceDefinition ([13](13-service-catalog.md)) declares an **update policy**: +Each service's **update policy** is declared in git: - **channel/range**: which versions are "stable" and acceptable (semver range, tag regex). - **cadence**: how often to check/apply (the schedule). - **hold/pin**: temporarily freeze a service (e.g. a known-bad upstream release). - **rollout & health**: how the new version rolls out and what health gates it. - **auto-rollback**: revert on failed health so a bad upstream release self-heals. +The ServiceDefinition ([13](13-service-catalog.md)) records that intent. Renovate does not read it, +so the policy takes effect through per-package rules in the Renovate configuration, which mirrors +each declared ceiling and carries any per-service cadence. + ## Safety rails (so hands-off ≠ reckless) - **Stable only**: never track pre-release/edge tags unless explicitly opted in per service. +- **Merge gate first**: every update is a PR that must pass the gate before it can merge. - **Health-gated rollout**: a failing update is caught by health checks and rolled back (R-UPD-03), and raises an alert ([10](10-observability-and-alerting.md)). - **Backups precede risk**: state is on Longhorn with backups, so a bad update never risks data ([04](04-storage-and-state.md)). -- **Observability**: update events (image advanced, chart merged, rollback) are visible and, on - failure, alert by email. +- **Observability**: update events (update proposed, merged, rollback) are visible and, on failure, + alert by email. - **Pin escape hatch**: an operator can pin a service via git if an upstream release misbehaves — the one sanctioned human touch, and it *stops* automation rather than performing an update. @@ -68,9 +86,5 @@ updates; policies are reconciled from the (unchanged) ServiceDefinition. - A service advances to a newer stable version automatically, on schedule, with no human/AI edit. - A deliberately bad update is caught by health checks, auto-rolled-back, and alerts by email. - A pinned service does **not** advance. -- Platform components (charts) stay current via auto-merged Renovate PRs. - -## Open questions -- Exact default in ADR-0009: write-back commits vs. digest-tracking mutable tags (trade readable - history vs. zero-commit updates). -- Registry rate-limit / credential handling for image scanning at cadence. +- Components in the allowed classes stay current via auto-merged Renovate PRs; majors and + blast-radius components arrive as machine-authored PRs awaiting review. diff --git a/docs/12-command-center-ui.md b/docs/12-command-center-ui.md index 1688010..cba9eba 100644 --- a/docs/12-command-center-ui.md +++ b/docs/12-command-center-ui.md @@ -18,15 +18,18 @@ app to build from scratch: Homepage widgets: Plex, Sonarr, Radarr, qBittorrent, PiHole, Gitea, etc.). This is the primary R-C2-01 surface. - **Grafana** — embedded platform + per-service dashboards (R-OBS-03) for health at a glance. -- **Headlamp** (or a comparable Kubernetes web UI) — a controlled Kubernetes console for viewing - workloads/VMs and performing scoped operational actions (restart, scale, inspect) — the R-C2-02 - "controlled operational actions where safe" surface. +- **Headlamp** — a controlled Kubernetes console for viewing workloads/VMs and performing scoped + operational actions (restart, scale, inspect) — the R-C2-02 "controlled operational actions where + safe" surface. Each user acts under their own IdP identity and group-bound RBAC, never a shared + console identity. - **Flux status** — GitOps reconciliation state (via Headlamp's Flux view, the Flux UI, or a Homepage widget) so the operator sees drift/errors. -- **Hubble UI** (optional) — network flow visualization for isolation troubleshooting. +- **Hubble UI** (optional) — network flow visualization for isolation troubleshooting; deployed + with no route, so it is not reachable from outside the cluster. -Each sub-UI is itself an onboarded, TLS-protected, access-controlled service; the root aggregates and -links them. ADR-worthy only if a custom portal is later preferred over Homepage. +Each sub-UI is itself a platform-managed or onboarded, TLS-protected, access-controlled service; the +root aggregates and links them. Status of this composition is tracked in +[known gaps](16-known-gaps.md). ## Data the C2 surfaces (R-C2-02) @@ -47,12 +50,16 @@ kubeconfig in a browser: ## Access control -- The C2 root and its sub-UIs are `internal` exposure by default (LAN-only) unless the operator - explicitly makes a component `public`. +- C2 sub-UIs are `internal` exposure by default (LAN/VPN-only) unless the operator explicitly makes + a component `public`. The apex and Grafana are `public` as registered platform routes; Headlamp is + LAN/VPN-only because it holds a path to the Kubernetes API on the user's behalf. +- Prometheus, Alertmanager and Hubble UI have **no route** and are not exposed; operators reach + their data through Grafana. Any future exposure of them MUST be gated. - Authentication is via **Keycloak SSO** ([15](15-identity-and-access.md), [ADR-0014](../decisions/adr-0014-idp-keycloak.md)): - Headlamp/Grafana use **native OIDC**; UIs without OIDC (Homepage, Prometheus, Alertmanager, Hubble) - sit behind the **oauth2-proxy forward-auth** at the Gateway. Access to the C2 and sensitive - surfaces requires an authenticated user **in an authorized group** with **MFA** (R-IDP-06). + Headlamp/Grafana use **native OIDC**; a UI without OIDC, such as Homepage, requires a per-app + **oauth2-proxy forward-auth gate** ([ADR-0022](../decisions/adr-0022-forward-auth-per-app-oauth2-proxy-sidecars.md)). + Access to the C2 and sensitive surfaces requires an authenticated user **in an authorized group** + with **MFA** (R-IDP-06). - Trusted TLS on the root and all sub-UIs ([08](08-tls-and-certificates.md)). ## Acceptance gates @@ -62,8 +69,7 @@ kubeconfig in a browser: - Safe operational actions work and are access-controlled + audited; destructive actions confirm. ## Resolved -- Auth mechanism — **Keycloak SSO** (native OIDC + oauth2-proxy forward-auth), [15](15-identity-and-access.md) / - [ADR-0014](../decisions/adr-0014-idp-keycloak.md). - -## Open questions -- Custom portal vs. Homepage-as-root long term (ADR if we diverge from Homepage). +- Auth mechanism — **Keycloak SSO** (native OIDC + per-app oauth2-proxy forward-auth gates), + [15](15-identity-and-access.md) / [ADR-0014](../decisions/adr-0014-idp-keycloak.md) / + [ADR-0022](../decisions/adr-0022-forward-auth-per-app-oauth2-proxy-sidecars.md). +- Root UI — **Homepage** at the apex. A custom portal would need an ADR. diff --git a/docs/13-service-catalog.md b/docs/13-service-catalog.md index 050a509..8362440 100644 --- a/docs/13-service-catalog.md +++ b/docs/13-service-catalog.md @@ -11,6 +11,11 @@ implementation repo. It captures *intent*; the platform compiles it into concret record, ServiceMonitor + dashboards + alerts, and update policy). Operators declare **what**; platform provides the **how**. This keeps every requirement wired from a single reviewable object. +**Platform-managed services** are the deliberate exception. A component with its own operator, +lifecycle or install flow is declared directly in git, under the same controls (default-deny, trusted +TLS, SSO, GitOps delivery through the merge gate): keycloak, Crafty, the command-center UIs, and VMs +with a custom install flow such as UniFi. `kind: vm` remains the catalog path for VMs. + ### Fields (canonical schema — implementers render manifests from these) ```yaml @@ -27,6 +32,7 @@ storage: class: longhorn # replicated app/config state (R-STATE-02) - name: media # optional bulk tier (R-STATE-04) class: bulk + export: # bulk only: the data's location on the bulk NFS server dependencies: # explicit allow rules; everything else denied (R-NET-01/02) - to: port: @@ -37,7 +43,7 @@ ingressRoutes: # hostnames/paths to expose (rendered to HTTP auth: # SSO integration (R-IDP-02; see 15-identity-and-access.md) mode: oidc | forward-auth | native | none # oidc → app configured as a Keycloak OIDC client (does its own login) - # forward-auth → protected by oauth2-proxy → Keycloak at the Gateway (for UIs w/o OIDC) + # forward-auth → per-app oauth2-proxy sidecar → Keycloak (UIs w/o OIDC; ADR-0022) # native → app manages its own auth (e.g. Plex/Seerr via Plex) — documented exception # none → no auth (internal-only, discouraged) groups: [ , ... ] # groups allowed access (R-IDP-06) @@ -57,6 +63,14 @@ backup: > The schema is the **contract** between the operator surface and the implementation. Changing it is > an interface change — coordinate per [`agents/coordination-protocol.md`](../agents/coordination-protocol.md). +The fields above are the core of the contract. The formal schema, +`schemas/servicedefinition.schema.json` in the +[implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl), is +canonical and adds workload-shaping fields for what the core cannot express: `env`, `envFrom`, +`lifecycle`, `podSecurityContext`, `availability`, `hardware` (a host capability class such as a +GPU, never a device path), `inClusterClients`, `auth.apiRoutes`, `observability.livenessWorkload` +and `storage[].{mountPath,subPath,readOnly}`. + ## Default posture for every service (unless overridden) - `kind: container`, `exposure: internal`, `egress: internet`. - **Default-deny**: no `dependencies` ⇒ can reach only DNS ([03](03-networking-isolation-and-egress.md)). @@ -65,28 +79,29 @@ backup: - Longhorn-backed config PVC; off-cluster backup; trusted TLS; auto-DNS; monitored; auto-updated on `stable`. -## The initial catalog (14 services) +## The catalog (14 services) Egress and dependencies below are **normative intent**; implementers render the exact policies. "Bulk" = large media tier ([04](04-storage-and-state.md)). All get trusted TLS + auto-DNS + monitoring -+ auto-update unless noted. ++ auto-update unless noted. **Status** gives each service's delivery path: `catalog` (rendered from +its ServiceDefinition), `platform-managed` (declared directly in git), or `not onboarded`. -| # | Service | kind | exposure | egress | Storage | Key dependencies (explicit allows) | Notes | -|---|---|---|---|---|---|---|---| -| 0 | **keycloak** | container | public | internet | db (PostgreSQL): longhorn (irreplaceable → backed up) | → its DB; ← all SSO clients + oauth2-proxy | **IdP/SSO** ([15](15-identity-and-access.md)); **foundational**, onboarded **early**; DR-critical + break-glass path; hardened public exposure | -| 1 | **plex** | container | public | **internet** | config: longhorn (small, replicated); media: **bulk** | — (clients reach it; Seerr → Plex) | Largest service; media on bulk tier, metadata DB on Longhorn ([04](04-storage-and-state.md)); auth: **native** (Plex) | -| 2 | **sonarr** | container | internal | **vpn** | config: longhorn; media/downloads: bulk | → qbittorrent, → jackett; ← seerr | VPN + kill-switch (R-NET-06) | -| 3 | **radarr** | container | internal | **vpn** | config: longhorn; media/downloads: bulk | → qbittorrent, → jackett; ← seerr | VPN + kill-switch | -| 4 | **jackett** | container | internal | **vpn** | config: longhorn | ← sonarr, ← radarr | VPN + kill-switch (indexer) | -| 5 | **seerr** | container | public | internet | config: longhorn | → sonarr:8989, → radarr:7878, → plex:32400 | Request UI; talks to arr + Plex | -| 6 | **pihole** | container | internal | internet | config: longhorn | (provides DNS to cluster/LAN) | Also a platform DNS dependency ([09](09-dns-and-ingress.md)) | -| 7 | **nextcloud** | container | public | internet | data: longhorn (irreplaceable → backed up) | → its DB | File sync; DR-critical data | -| 8 | **minecraft** | container | public | internet | world: longhorn (backed up) | — | Exposes game port via edge | -| 9 | **unifi** | container | internal | internet | config: longhorn | — | UniFi network controller | -| 10 | **semaphore** | container | internal | internet | config/db: longhorn | — | Ansible UI | -| 11 | **kasm** | container | internal | internet | config: longhorn | — | Kasm Workspaces (may need more resources; container-first, VM if required) | -| 12 | **gitea** | container | public | internet | repos/db: longhorn (irreplaceable → backed up) | — | Git hosting; **bootstrapping-critical** — self-hosts the platform's GitOps source + CI/CD; onboarded **early** (see below); DR-critical (off-cluster backup + mirror) | -| 13 | **qbittorrent** | container | internal | **vpn** | config: longhorn; downloads: **bulk** | ← sonarr, ← radarr | VPN + kill-switch; download client | +| # | Service | Status | kind | exposure | egress | Storage | Key dependencies (explicit allows) | Notes | +|---|---|---|---|---|---|---|---|---| +| 0 | **keycloak** | platform-managed | container | public | internet | db (PostgreSQL, ADR-0019): longhorn (irreplaceable → backed up) | → its DB; ← all SSO clients + oauth2-proxy | **IdP/SSO** ([15](15-identity-and-access.md)); **foundational**, onboarded **early**; DR-critical + break-glass path; hardened public exposure. Run by its operator; its catalog entry is declaration only and deploys nothing | +| 1 | **plex** | catalog | container | internal | **internet** | config: longhorn (small, replicated); media: **bulk** | ← seerr, ← sonarr, ← radarr; clients reach it | Largest service; media on bulk tier, metadata DB on Longhorn ([04](04-storage-and-state.md)); auth: **native** (Plex); web UI LAN/VPN-only, remote streaming per Cross-cutting notes | +| 2 | **sonarr** | catalog | container | internal | **vpn** | config: longhorn; media/downloads: bulk | → qbittorrent, → jackett, → plex:32400; ← seerr | VPN + kill-switch (R-NET-06) | +| 3 | **radarr** | catalog | container | internal | **vpn** | config: longhorn; media/downloads: bulk | → qbittorrent, → jackett, → plex:32400; ← seerr | VPN + kill-switch | +| 4 | **jackett** | catalog | container | internal | **vpn** | config: longhorn | ← sonarr, ← radarr | VPN + kill-switch (indexer) | +| 5 | **seerr** | catalog | container | public | internet | config: longhorn | → sonarr:8989, → radarr:7878, → plex:32400 | Request UI; talks to arr + Plex | +| 6 | **pihole** | catalog | container | internal | internet | config: longhorn | (provides DNS to cluster/LAN) | Also a platform DNS dependency ([09](09-dns-and-ingress.md)) | +| 7 | **nextcloud** | catalog | container | public | internet | data: longhorn (irreplaceable → backed up); legacy archive: bulk, read-only | → its DB | File sync; DR-critical data | +| 8 | **minecraft** (Crafty) | platform-managed | container | internal (game ports); public (web UI) | internet | world: longhorn (backed up) | — | Delivered as the Crafty controller, which runs the game servers; game ports on a LAN/VPN L4 address | +| 9 | **unifi** | platform-managed | vm | internal | internet | VM disk: longhorn (replicated, live-migratable) | — | UniFi OS Server in a KubeVirt VM, declared directly in git (custom install flow); browser and device traffic on a LAN/VPN L4 address, no Gateway route | +| 10 | **semaphore** | catalog | container | internal | internet | config/db: longhorn | → its DB | Ansible UI | +| 11 | **kasm** | not onboarded | container | internal | internet | config: longhorn | — | Kasm Workspaces (may need more resources; container-first, VM if required) | +| 12 | **gitea** | catalog | container | public | internet | repos/db: longhorn (irreplaceable → backed up) | — | Git hosting; **bootstrapping-critical** — self-hosts the platform's GitOps source + CI/CD; onboarded **early** (see below); DR-critical (off-cluster backup + mirror) | +| 13 | **qbittorrent** | catalog | container | internal | **vpn** | config: longhorn; downloads: **bulk** | ← sonarr, ← radarr | VPN + kill-switch; download client | **VPN-class set (R-NET-05/06):** sonarr, radarr, jackett, qbittorrent — all egress via VPN with an enforced kill-switch. Their inter-dependencies (arr → qbittorrent/jackett) are **east-west allows** @@ -95,19 +110,32 @@ within the deny-by-default fabric; being VPN-class governs *egress*, not intra-c **Dependency graph (explicit allows only; all else denied):** ``` seerr → sonarr, radarr, plex -sonarr → qbittorrent, jackett -radarr → qbittorrent, jackett +sonarr → qbittorrent, jackett, plex +radarr → qbittorrent, jackett, plex (everything → DNS only, by default) ``` +The arr → plex allows exist because the bulk tier is NFS, which delivers no change notifications: +the arr stack tells Plex when new media lands. + +**Delivery order (waves).** Catalog services reconcile in dependency-ordered waves computed from the +`dependencies` graph: a service's wave is its longest dependency path, so a database always lands +below its consumer. Each wave health-gates only itself and what depends on it, so a crashloop cannot +stall an earlier wave. VMs and leaf platform components sit on their own non-gating tiers: a slow +image import or a broken UI never holds up the app tier. + **SSO integration (`auth.mode`, R-IDP-02; see [15](15-identity-and-access.md)):** - **oidc** (native Keycloak client): gitea, nextcloud, semaphore, kasm, grafana*, headlamp* (*platform UIs, not in the numbered app list). -- **forward-auth** (oauth2-proxy → Keycloak): pihole, sonarr, radarr, jackett, qbittorrent, unifi, - and platform UIs (Homepage, Prometheus, Alertmanager, Hubble). -- **native** (own auth, documented exceptions): plex, seerr (Plex accounts); minecraft - (Mojang/Microsoft). +- **forward-auth** (per-app oauth2-proxy sidecar → Keycloak, ADR-0022): pihole, sonarr, radarr, + jackett, qbittorrent, Homepage*, and the Crafty web UI* (which adds its own local login; it has no + OIDC). +- **native** (own auth, documented exceptions): plex, seerr (Plex accounts); minecraft game ports + (Mojang/Microsoft); unifi (its own login, on its own L4 address). +- **none** (documented exception, R-IDP-02): the read-only runbooks at `/runbooks`, because gating + them on Keycloak would lock the operator out of the Keycloak recovery procedures. - **keycloak** itself is the IdP — its own admin login is local + MFA (the root of trust). +- Prometheus, Alertmanager and Hubble UI are not exposed; any future exposure MUST be gated. **"…but not limited to":** the catalog is open-ended. New services follow the same ServiceDefinition and inherit all guarantees automatically — that is the point of the abstraction. @@ -118,7 +146,8 @@ Points the schema left ambiguous, now pinned so the JSON Schema, Kyverno policie - **`source` is exactly-one-of** — `image` for `kind: container`, `vmTemplate` for `kind: vm` (schema-enforced conditional). - **`storage[].class: longhorn ⇒ size` is required** — enforced by an **L0 Kyverno policy** (a - replicated volume with no size is a defect). `bulk` size may be omitted (external mount). + replicated volume with no size is a defect). `storage[].class: bulk ⇒ export` is required: a bulk + volume is statically bound to data that already exists, and its `size` is advertised capacity only. - **`backup` is a reference** — `backup: {policy: }` where `policy` is either the literal `reacquirable` (bulk/re-acquirable data) or a named off-cluster backup schedule. Cadence/retention live in the referenced schedule object, **not** inline in the ServiceDefinition. @@ -128,32 +157,35 @@ Points the schema left ambiguous, now pinned so the JSON Schema, Kyverno policie least one Keycloak group (R-IDP-06); optional for `native`/`none`. - **`exposure: public ⇒ auth.mode ≠ none`** — a public service may not be unauthenticated (docs/15, R-IDP-02); **schema-enforced** invariant. -- **Irreplaceable ⇒ off-cluster backup** — DR-critical services MUST NOT be `reacquirable`. This - depends on per-service criticality the schema can't see, so it is enforced by an **L0 Kyverno - policy** keyed on the service's criticality signal, not by the JSON Schema. +- **Irreplaceable ⇒ off-cluster backup** — DR-critical services MUST NOT be `reacquirable`. The JSON + Schema can't see criticality, so an **L0 Kyverno policy** applies the conservative rule: a service + with any `longhorn` volume MUST NOT declare `reacquirable`. +- **`update` declares policy; it does not run updates.** Renovate acts on it through machine-authored + changes at the merge gate ([11](11-automated-updates.md), ADR-0023). ## Cross-cutting notes - **Plex media** is the R-STATE-04 case: keep the metadata DB on replicated Longhorn; keep the large library on the bulk tier; downloads stage on bulk and are handed off by the arr stack. - **DR-critical data** (Nextcloud, Gitea, Minecraft worlds, PiHole config, service DBs) MUST have - off-cluster backups ([04](04-storage-and-state.md)); bulk media may be `reacquirable` per operator - choice (OQ-2). -- **Public exposure** (Seerr, Nextcloud, Gitea, …) is through the **Cloudflare Tunnel** - ([09](09-dns-and-ingress.md), ADR-0016). Everything else stays `internal`. **Plex** remote + off-cluster backups ([04](04-storage-and-state.md)). The bulk tier is outside platform backup. +- **Public exposure** is through the **Cloudflare Tunnel** ([09](09-dns-and-ingress.md), ADR-0016). + A service is public by catalog `exposure: public` (Keycloak, Seerr, Nextcloud, Gitea) or as a + registered platform route (the Homepage apex, `/runbooks`, Grafana, the Crafty web UI, and the + oauth2-proxy host). Everything else stays `internal`. **Plex** remote streaming uses plex.tv + an edge forward of tcp/32400 (not the tunnel), and its web UI is LAN-only. **Minecraft (crafty-mc)** and other L4 services are LAN/VPN-only. - **Kasm** is resource-heavy and may be a candidate for `kind: vm` if container mode proves insufficient — the ServiceDefinition makes that a one-field change. - **Gitea is bootstrapping-critical.** The platform self-hosts its git and CI/CD, so Gitea is brought - online **early** (WS-APPS step 2, right after PiHole). Once stable, the GitOps source and CI move - from the local bootstrap source to Gitea + Gitea Actions ([05](05-gitops-and-delivery.md), - [`../CONTRIBUTING.md`](../CONTRIBUTING.md)). Because a cluster loss would take its own git host with - it, Gitea's repos/DB are backed up **off-cluster** and mirrored, so the platform can always be - rebuilt from a source that outlives the cluster. + online **early** (the first wave, with PiHole). Gitea + Gitea Actions are the GitOps source and CI + ([05](05-gitops-and-delivery.md), [`../CONTRIBUTING.md`](../CONTRIBUTING.md)). Because a cluster + loss would take its own git host with it, Gitea's repos/DB are backed up **off-cluster** and + mirrored, so the platform can always be rebuilt from a source that outlives the cluster. ## Acceptance gates -- Each service is fully wired from its ServiceDefinition alone (compute, storage, policy, egress, - TLS, DNS, monitoring, updates) — no manual steps beyond merge. +- Each `catalog` service is fully wired from its ServiceDefinition alone (compute, storage, policy, + egress, TLS, DNS, monitoring, updates) — no manual steps beyond merge. Each `platform-managed` + service meets the same guarantees from its declaration in git. - Default-deny holds; only listed dependencies work. - VPN-class services egress via VPN with a proven kill-switch. - Stateful services survive destroy/rebuild; DR-critical data restores from off-cluster backup. diff --git a/docs/14-testing-and-conformance.md b/docs/14-testing-and-conformance.md index f8a1307..6bdbe52 100644 --- a/docs/14-testing-and-conformance.md +++ b/docs/14-testing-and-conformance.md @@ -25,11 +25,13 @@ Related: [ADR-0010](../decisions/adr-0010-conformance-testing.md). fail-closed / no-data-loss — not by happy-path checks. 6. **Specs are the fixed target (R-TEST-06).** Tests encode the spec's intent. Agents never weaken a test or a requirement to get green; a genuine inability to meet the spec is an escalation to the - human, not a spec/test edit. + human, not a spec/test edit. The implementation pins the spec commit it is built against, and each + spec advance is adopted explicitly ([`../CONTRIBUTING.md`](../CONTRIBUTING.md)). ## Reproducibility contract (what "reproducible" concretely requires) -- **Single command** runs the suite (or a tagged subset) locally and in CI. +- **Single command** runs the suite (or a tagged subset) locally and in CI, in the same pinned + toolbox image in both places. - **Hermetic fixtures**: each test provisions its own namespace/workloads/probes and tears them down; it never depends on pre-existing mutable state or on another test's side effects. - **Deterministic**: fixed inputs, pinned tool/image versions, bounded retries with explicit @@ -49,8 +51,8 @@ Related: [ADR-0010](../decisions/adr-0010-conformance-testing.md). |---|---|---|---| | **L0 Static/policy** | Pre-apply, in CI | Manifests/ServiceDefinitions obey invariants | Every namespace has a default-deny baseline; no plaintext secrets; schema-valid ServiceDefinition | | **L1 Rendering** | CI, no cluster | ServiceDefinition → rendered resources match golden expectations | `egress: vpn` renders the kill-switched egress path | -| **L2 Behavioral/integration** | Against dev/test cluster | Real runtime behavior | Default-deny probe, dependency allow/deny, egress-IP, TLS validity, DNS, PVC rebuild | -| **L3 Fault-injection** | Against dev/test cluster | Fail-closed / no-data-loss under failure | VPN tunnel forced down ⇒ no leak; node killed ⇒ data survives; bad update ⇒ auto-rollback; DR restore | +| **L2 Behavioral/integration** | Against the live cluster | Real runtime behavior | Default-deny probe, dependency allow/deny, egress-IP, TLS validity, DNS, PVC rebuild | +| **L3 Fault-injection** | Against the live cluster | Fail-closed / no-data-loss under failure | VPN tunnel forced down ⇒ no leak; node killed ⇒ data survives; bad update ⇒ auto-rollback; DR restore | | **L4 Conformance suite** | CI + on-demand | Aggregate, requirement-tagged gate | The union of the above, reported per `R-*` | ## Behavior test catalog (normative — implementers author these as automated tests) @@ -82,13 +84,15 @@ Each maps to requirement IDs; phrased as Given/When/Then intent. - **DNS/exposure (R-DNS-*):** *Given* a service, *When* resolved, *Then* the record exists and internal clients hit the internal Gateway; `internal` is unreachable publicly, `public` is reachable + TLS-valid. -- **Public ingress path (R-DNS-04/05, R-SEC-04):** *Given* the tunnel ingress config, *When* it is - rendered, *Then* every route is a `public` service, the last rule is `http_status:404`, and no rule - targets the Kubernetes API (L0). *When* a non-`public` hostname is requested at the tunnel, *Then* - it gets 404. *When* a `public` name is resolved via a public resolver from the LAN, *Then* it is - reachable + TLS-valid (L2). +- **Public ingress path (R-DNS-04/05, R-SEC-04):** *Given* the tunnel ingress config (a deny-list of + LAN/VPN-only hostnames ahead of a zone wildcard to the shared Gateway), *When* it is rendered, + *Then* every rule targets only the shared Gateway or a 404, every LAN/VPN-only hostname 404s ahead + of the wildcard, the last rule is `http_status:404`, and no rule targets the Kubernetes API (L0). + *When* a LAN/VPN-only hostname is requested at the tunnel, *Then* it gets 404. *When* a `public` + name is resolved via a public resolver from the LAN, *Then* it is reachable + TLS-valid (L2). - **Hands-off update (R-UPD-01/02, L3):** *Given* an out-of-date service under a stable policy, - *When* the scan interval elapses, *Then* it advances with no human/AI edit; *Given* a deliberately + *When* the update schedule elapses, *Then* a machine-authored change advances it through the merge + gate with no human/AI edit; *Given* a deliberately bad update, *Then* health-gating auto-rolls-back and an email alert fires; *Given* a pinned service, *Then* it does not advance. - **VM lifecycle (R-PLAT-03, L2/L3):** *Given* a `kind: vm` ServiceDefinition, *When* create/start/ @@ -108,10 +112,25 @@ Each maps to requirement IDs; phrased as Given/When/Then intent. touches MUST pass before merge. - **Per-phase (exit gate):** the full conformance suite for the phase's requirements MUST pass before the phase's exit criteria are considered met ([`../roadmap/phases.md`](../roadmap/phases.md)). -- **Regression:** the growing suite runs continuously against dev/test; a previously-green +- **Regression:** the growing suite runs continuously against the live cluster; a previously-green requirement going red blocks further delivery until fixed. - **Coverage:** a requirement with no passing test is **not** satisfied, regardless of implementation state. +- **Non-answers are never green:** a stage that did not run is never reported as passing + ([05](05-gitops-and-delivery.md); see [known gaps](16-known-gaps.md)). + +## Coverage, status and proof (R-TEST-04) + +Three different questions, answered separately so that none is mistaken for another: + +- **Coverage** — does a requirement-tagged test exist for each `R-*`? +- **Status** — derived from the repository alone: is the requirement implemented, is that + implementation actually reconciled by Flux, and is its test executable or a scaffold? +- **Proof** — did a run actually pass? Proof is a committed, machine-readable run ledger. A test that + skips itself is never proof, and a proof goes stale when the implementation or test moves on. + +`./conformance/run.sh --status` in the implementation repo produces the live view. +[`16-known-gaps.md`](16-known-gaps.md) is a dated snapshot. ## Tooling (pinned — [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md)) @@ -138,7 +157,11 @@ from the **Kyverno ecosystem**: timeouts, per-test namespace isolation, and automatic cleanup satisfy the reproducibility contract. Fault-injection (force the tunnel down, cordon/kill a node, inject a bad image, trigger a restore) is expressed as Chainsaw `script` steps, asserting fail-closed / no-data-loss via probe fixtures, - with `finally` restoring state. + with `finally` restoring state. Cluster-free L0 and L1 checks are declared as Chainsaw tests for + tagging and execute through the harness's offline runner, so they need no cluster. +- **Repository invariants** — L0 also carries whole-repo checks a per-resource policy cannot express + (secret hygiene across every tracked file, least-privilege RBAC, the spec pin), which run on every + gate regardless of the requirements a change touches. Probes are ephemeral job pods; egress-IP checks use an external echo; tests carry their `R-*` IDs (labels/annotations/naming) so the coverage report is generated automatically; both tools' output @@ -148,16 +171,18 @@ pinned for determinism. ### Where each layer runs - **L0 (Kyverno CLI) and L1 (rendering assertions)** run **in CI without a cluster** — pure manifest/ ServiceDefinition validation; fast merge-gate feedback. -- **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job** on the dev/test cluster. +- **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job** on the live cluster. CI **triggers** the Job (and waits for it); the Job — not CI — holds cluster access, so CI never - needs a broad kubeconfig. This keeps credentials in-cluster and least-privileged (R-SEC-02). + needs a broad kubeconfig. CI holds only a narrow trigger credential, whose reach is bounded by a + fail-closed admission policy on the Job's shape. This keeps credentials in-cluster and + least-privileged (R-SEC-02). - The Job runs under a **dedicated, scoped test ServiceAccount** whose RBAC grants exactly what the behavior/fault tests need (create/delete test namespaces + fixtures + probes, apply/patch NetworkPolicies, scale the VPN sidecar/gateway, cordon/drain a node, inject a bad image tag, trigger a Velero restore). This is a **privileged test identity** — it MUST be scoped, named, and documented, MUST NOT be reused by workloads, and its own reach is itself an audit item. - - The Job writes **JUnit + coverage artifacts** to a known location (log stream and/or a results - PVC/object) that CI collects to enforce the merge / phase-exit gates. + - The Job emits **JUnit + coverage artifacts** over its log stream, in an integrity-checked + envelope that CI collects to enforce the merge / phase-exit gates. - Fixtures are hermetic and self-cleaning (`finally` steps); a leaked fixture fails the run. ## Acceptance gates (for the testing capability itself) @@ -170,10 +195,10 @@ pinned for determinism. ## Open questions - Tool mix is **resolved**: Kyverno (L0) + Chainsaw (L1/L2/L3), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md). - Harness location is **resolved**: L2/L3 run as an **in-cluster Chainsaw Job** triggered by CI; CI - holds no cluster credentials. + holds no broad cluster credential, only the admission-bounded trigger. - Kyverno-as-admission-enforcer is **resolved**: yes — the L0 policies also enforce at admission in-cluster (defense in depth), in addition to the CI gate. -- Delegated to WS-TEST at implementation time (not human decisions): the **exact RBAC scope** of the - test ServiceAccount (author it **least-privilege** for precisely the fault-injection actions it - performs, [07](07-security-and-secrets.md)) and the **results-artifact mechanism** (log stream vs - results PVC/object store). Both are implementer's discretion within the stated constraints. +- The test ServiceAccount's RBAC is **resolved**: scoped **least-privilege** to precisely the + fault-injection actions it performs ([07](07-security-and-secrets.md)), with its grants justified + per test in the implementation repo. +- The results-artifact mechanism is **resolved**: the log stream, as above. diff --git a/docs/15-identity-and-access.md b/docs/15-identity-and-access.md index c2ea4b2..fca01a8 100644 --- a/docs/15-identity-and-access.md +++ b/docs/15-identity-and-access.md @@ -20,22 +20,33 @@ self-hostable OpenID Connect (OIDC) + SAML provider with users, groups, roles, c Every service authenticates against Keycloak, via whichever pattern it supports: 1. **Native OIDC** — the app is configured as a Keycloak OIDC **client** and does its own login - redirect. Preferred where available. Initial set: **Gitea, Nextcloud, Grafana, Kasm, Semaphore, - Headlamp**. -2. **Forward-auth proxy** — for apps with **no** usable OIDC, an identity-aware proxy - (**oauth2-proxy**) sits in front at the Gateway, authenticates the user against Keycloak, and only - then forwards the request. Used for: **PiHole admin, the *arr web UIs (Sonarr/Radarr/Jackett), - qBittorrent UI, Homepage, Prometheus, Alertmanager, Hubble UI**, and any other bare UI. + redirect. Preferred where available. Initial set: **Gitea, Nextcloud, Grafana, Semaphore, + Headlamp** (Kasm, not onboarded, joins this set when it is). +2. **Forward-auth gate** — for apps with **no** usable OIDC, a per-app identity-aware proxy + (**oauth2-proxy**) runs as a sidecar in the app's pod, authenticates the user against Keycloak, + and only then forwards the request ([ADR-0022](../decisions/adr-0022-forward-auth-per-app-oauth2-proxy-sidecars.md)). + Used for: **PiHole admin, the *arr web UIs (Sonarr/Radarr/Jackett), qBittorrent UI, the Crafty + web UI**, and any other bare UI; **Homepage** requires the same (see + [known gaps](16-known-gaps.md)). Prometheus, Alertmanager and Hubble UI are not exposed; any future + exposure MUST be gated. > This resolves the C2 auth open question ([12](12-command-center-ui.md)): it is **both** — native > OIDC where supported, forward-auth everywhere else, all backed by Keycloak. +Each gate is an independent instance with its own IdP client and credentials, and enforces the +service's `auth.groups`. Session cookies are per-gate: a unique name, scoped to the app's own host, +because instances that share a cookie evict one another's sessions. A service MAY declare its API +paths so that unauthenticated programmatic calls are refused rather than redirected to a login page. + ### Exceptions (documented, not oversights) - **Plex** and **Seerr** use **Plex's own account system** (Plex OAuth) — they do not federate to Keycloak. -- **Minecraft** authenticates players via Mojang/Microsoft — N/A. -- **UniFi** keeps its native account model (limited SSO support); protect its UI via forward-auth if - desired. +- **Minecraft** players authenticate via Mojang/Microsoft — N/A. The **Crafty web UI** that manages + it is behind a forward-auth gate and then presents **its own local login** (no OIDC); the second + login is intended. +- **UniFi** keeps its native account model and login on its own L4 VIP (limited SSO support). +- **Runbooks** served read-only under `/runbooks` carry no login. +- **Keycloak** itself is `native`: it is the root of trust. ## Authorization model (R-IDP-06) @@ -45,47 +56,51 @@ Every service authenticates against Keycloak, via whichever pattern it supports: - **Least privilege**: the C2 console, Headlamp, Alertmanager, and other sensitive UIs require an authenticated user **in an authorized group** — not merely "logged in." - **MFA**: TOTP/WebAuthn available for all users and **required for privileged groups** - (`platform-admins`). + (`platform-admins`). Passkeys (WebAuthn passwordless) are an optional additive login path, + bound into the login flow by an explicit switch in git. +- **Kubernetes API**: humans authenticate with Keycloak OIDC, and cluster roles are bound to Keycloak + groups, so `kubectl` and the Kubernetes console share this model. ## Identity config as code (R-IDP-03) Realms, clients, groups, and roles are **declarative and GitOps-managed** — not clicked into the admin console and forgotten: -- The Keycloak instance is deployed via the **Keycloak Operator** (or Helm), and **realm/client/group - configuration is reconciled declaratively** (e.g. `keycloak-config-cli` or realm-import CRs). Exact - tool is an implementation detail; the requirement is that identity config lives in git and - reconciles like everything else. +- The Keycloak instance is deployed via the **Keycloak Operator**, and a **realm-import CR** seeds + realm/client/group configuration from git. A realm import is one-shot and fails open, so + **reconcilers** continuously converge what git declares after the first import: client secrets, + client URLs, the WebAuthn policy and the login flow. - **OIDC client secrets and the Keycloak admin credential are SOPS-encrypted** ([07](07-security-and-secrets.md)). ## State, exposure & resilience -- **Datastore**: Keycloak runs on **PostgreSQL**; the DB lives on **3-replica Longhorn** and is - **DR-critical → backed up off-cluster** ([04](04-storage-and-state.md)). Keycloak is stateless - itself and rebuilds from the DB + git-managed realm config. +- **Datastore**: Keycloak runs on **PostgreSQL via CloudNativePG** (ADR-0019); the DB lives on + **3-replica Longhorn** and is **DR-critical → backed up off-cluster** ([04](04-storage-and-state.md)). + Keycloak is stateless itself and rebuilds from the DB + git-managed realm config. - **Exposure**: `keycloak.home.keithjackson.dev` is **`public`** so it can serve login/redirect flows for the internet-exposed services (Nextcloud, Gitea) — but it is **security-sensitive** and hardened: brute-force detection on, strong admin auth + MFA, trusted TLS ([08](08-tls-and-certificates.md)), - minimal exposed endpoints. If the operator prefers internal-only, public services fall back to - their native auth. -- **HA**: a single replica with DB on Longhorn + backups is acceptable for the home scale; note - clustered Keycloak as a future hardening. **Availability caveat:** SSO becomes a dependency for - logging into federated services — document a break-glass path (e.g. a local admin on Gitea) so an - IdP outage doesn't lock the operator out of recovery tools. + minimal exposed endpoints. +- **HA**: Keycloak runs multiple replicas against a database with one instance per node and a hot + standby to promote, because a single node must not be able to take SSO down. + **Availability caveat:** SSO is a dependency for logging into federated services and the Kubernetes + API, so a **break-glass** path exists: local admin logins on Gitea, Nextcloud and Grafana, the + Keycloak bootstrap admin, and a certificate kubeconfig for the cluster. ## Interaction with default-deny & the gateway - Keycloak and oauth2-proxy are ordinary workloads under **default-deny** ([03](03-networking-isolation-and-egress.md)); - the specific flows they need (app → Keycloak OIDC, proxy → Keycloak, Keycloak → its DB) are + the specific flows they need (app → Keycloak OIDC, gate → Keycloak, Keycloak → its DB) are explicit allow rules. -- Forward-auth is enforced at the **Gateway** ([09](09-dns-and-ingress.md)) so protection happens - before traffic reaches the app. +- Forward-auth is enforced **in the data path**: the Gateway ([09](09-dns-and-ingress.md)) routes + only to the gate's port and network policy closes the app's own port to it, so protection happens + before traffic reaches the app and a failed gate fails closed. ## Bootstrapping & order - Keycloak is **foundational** and onboarded **early** (with PiHole/Gitea, [13](13-service-catalog.md)) so later services can integrate SSO as they come up. - No hard circular dependency at bring-up: Gitea can start with a local admin and add OIDC once - Keycloak is live; the C2 and sensitive UIs gain SSO as soon as the proxy + realm exist. + Keycloak is live; the C2 and sensitive UIs gain SSO as soon as their gate + the realm exist. ## Acceptance gates - A user logs in once at Keycloak and reaches all OIDC-integrated services without re-entering @@ -97,8 +112,6 @@ admin console and forgotten: - Privileged access (C2/Headlamp) requires MFA and group membership; a non-member is denied. - Client secrets and admin credentials exist only SOPS-encrypted. -## Open questions -- Final declarative-config tool (`keycloak-config-cli` vs operator realm-import CRs) — implementation - detail for WS-SEC. -- Confirm `public` exposure for Keycloak (recommended) vs internal-only with native-auth fallback for - public services. +## Resolved +- Declarative-config tool — operator **realm-import CRs**, plus the reconcilers above. +- Keycloak exposure — **`public`**, hardened. diff --git a/docs/16-known-gaps.md b/docs/16-known-gaps.md new file mode 100644 index 0000000..b0441f5 --- /dev/null +++ b/docs/16-known-gaps.md @@ -0,0 +1,108 @@ +# 16 — Known Gaps + +**Snapshot of implementation `main` at `50c8719`, taken 2026-11-08.** This register records where the +implementation does not yet meet a requirement this specification still holds. It exists so that bringing +the specification in line with the implementation did not silently delete the intent the implementation +has yet to reach. + +- A gap is **not** an accepted change. Where the implementation deliberately changed direction, the + requirement or an ADR was amended instead, and the item does not appear here. +- This file is a dated snapshot, not a status surface. Every commit here costs the implementation a + `ci/SPEC_PIN` bump, so it is refreshed in batches. **Live status comes from `conformance/run.sh --status`** + in the [implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl). +- Entries were derived by reading both repositories. **Nothing here was measured against the running + platform.** + +## Operator decisions needed + +These cannot be settled by reading either repository. + +| Topic | Question | Evidence (implementation repo) | +|---|---|---| +| Bulk-tier photos (R-STATE-03/04) | The bulk tier has no platform backup, and its media dataset includes a `photos/` tree. Is that tree irreplaceable, and if so, what protects it? | `docs/BULK-NFS-STORAGE.md` §2; `platform/velero/volume-policies.yaml` | +| Homepage exposure (R-IDP-02) | The apex is routed publicly with no enforced authentication. Enforce the gate (the requirement stands), or rule on an exception? | `platform/homepage/httproute.yaml`; `platform/homepage/README.md` | +| ACME revocation (R-TLS-02) | cert-manager has no revocation mechanism. Amend the requirement, or keep it as an open gap? | `conformance/tests/l0-static/tls-lifecycle-automated/chainsaw-test.yaml:36` | +| L3 in CI (R-TEST-03, R-TEST-05) | Production is the only cluster. Is automated fault injection against it acceptable once CI is wired? | `docs/plans/l2-l3-conformance-ci-wiring.md` | +| Kasm (docs/13) | Never onboarded, and no decision retires it. Onboard it, or retire it from the catalog? | `catalog/README.md:20` | + +## Environment and provisioning + +| Requirement or area | Requirement and shortfall | Evidence (implementation repo) | +|---|---|---| +| R-ENV-01, R-STATE-01 (Phase 1 exit) | A node must be destroyable and rebuildable from code with no data loss; a node was reimaged and rejoined by hand, but no automated destroy/rebuild proof exists on bare metal and the checklist item is unticked. | `infra/PROD-PROVISIONING.md` sections 0c and 9 | +| docs/02 etcd guardrail (accepted risk) | Shared-device etcd/Longhorn contention is accepted only if etcd write latency stays within budget during a Longhorn rebuild; the measurement failed on node 1 on 2026-09-07 and no dedicated etcd device was added, so the exit condition is unmet. | `infra/PROD-PROVISIONING.md` section 2 ("The accepted risk, and its exit condition"), section 0c | +| R-PLAT-03, Phase 5 exit | The VM tier must run a working application VM; at the last recorded check (2026-09-14) the UniFi guest had no IP address and its controller ports refused connections. | `apps/vms/unifi/README.md` | + +## Storage, state and VMs + +| Requirement or area | Requirement and shortfall | Evidence (implementation repo) | +|---|---|---| +| Off-cluster Gitea mirror (R-STATE-03; spec CONTRIBUTING, docs/05 "Circularity & DR") | An off-cluster mirror of the platform repo must outlive the cluster; Gitea has no mirrors and no mirror job exists, while prod's Flux source is in-cluster Gitea. | `catalog/gitea.yaml:52`; `decisions/2026-11-07-gitea-28.md` | +| R-STATE-03 (Gitea) | Gitea is DR-critical and its repos and SQLite are backed up by file-level kopia with no quiesce hook, and no Gitea-specific restore has been proven (the DR test restores its own scratch workload). | `apps/wave-0/gitea.yaml`; `conformance/tests/l3-fault-injection/dr-restore/chainsaw-test.yaml` | +| R-STATE-03 (UniFi) | Irreplaceable VM state must be recoverable off-cluster; the UniFi disk is excluded on the basis of a configuration export that nothing in the repo declares. | `platform/velero/helmrelease.yaml:182-188`; `apps/vms/unifi/` | +| R-STATE-01 | The per-service rebuild test is an acceptance gate; `data-survival` and `vm-data-survival` exist but have no ledger row, and per-service rebuilds are not recorded. | `conformance/runs/ledger.jsonl`; `conformance/tests/l3-fault-injection/data-survival/` | +| R-STATE-01/03 | End-to-end total-cluster recovery should be drilled at least once; bare-metal rebuild then restore has not been exercised. | `conformance/tests/l3-fault-injection/dr-restore/chainsaw-test.yaml` (KNOWN, DELIBERATE GAPS) | +| R-STATE-01 (Loki) | Destroying or scaling a workload must leave its PVC; Loki's StatefulSet retention policy deleted its PVC on scale-down (log data is reconstructable). | `decisions/2026-10-17-graceful-cluster-shutdown.md` | +| R-PLAT-03, R-PLAT-04 | VM create/start/stop/destroy must be proven; `vm-lifecycle` is logged green only on the decommissioned dev cluster and has no ledger row. | `DECISIONS-LOG.md:896`; `conformance/runs/ledger.jsonl` | +| R-NET-06 (VMs) | An `egress: vpn` VM must egress through the VPN with a working kill-switch; no VM VPN conformance test exists and `vm-demo` calls it a follow-up. | `catalog/vm-demo.yaml`; `conformance/tests/l2-behavioral/vm-lifecycle/` | +| R-UPD-* (UniFi guest) | Guest OS packages must auto-update on schedule with per-VM exceptions recorded; UniFi's cloud-init sets `package_upgrade: false` and installs no unattended-upgrades. | `apps/vms/unifi/30-virtualmachine.yaml:206-207` | +| R-OBS-* (VM guests) | A VM must expose node-exporter-class metrics from inside the guest; neither the renderer nor UniFi provides any, only KubeVirt/virt platform metrics. | `catalog/render.py`; `platform/kubevirt/servicemonitor.yaml` | +| R-STATE-03 (VM disks) | VM disks are meant to share the platform backup story; the backup path is Velero file-system backup, which upstream documents as not supporting block-mode volumes, and the default VM class is block-mode. Unmeasured here: whether any block-mode VM disk can be captured by this path. | `platform/velero/helmrelease.yaml:175`; `platform/longhorn/storageclasses.yaml` (`longhorn-vm`) | + +## Networking, DNS and TLS + +| Requirement or area | Requirement and shortfall | Evidence (implementation repo) | +|---|---|---| +| VPN port forwarding (docs/03, ADR-0015) | The spec requires a host-side PIA port-forward helper for qBittorrent and an alert on PF acquisition failure; neither exists. | `infra/ansible/roles/vpn_egress_gateway/README.md:170`, `platform/vpn-egress/README.md` | +| R-NET-01 | Every namespace must carry a default-deny baseline; Kyverno generates it only for `app-namespace=true`, and no policy file exists for cert-manager, cnpg, external-dns, longhorn, velero, backup, image-automation, external-snapshotter or vpn-egress. | `platform/kyverno/policies/require-default-deny-baseline.yaml`, `decisions/2026-10-03-public-dns-wan-target.md` | +| R-SEC-03, R-OBS-04 | docs/03 requires alerting on unexpected denied flows; `HubbleDeniedFlowSpike` cannot fire because no Cilium or Hubble metrics are scraped. | `platform/monitoring/alerts.yaml:1047-1050,1162-1173` | +| R-TLS-02 | Certificates must be automatically revoked; renewal is met, but cert-manager performs no ACME revocation and no layer proves it. | `conformance/tests/l0-static/tls-lifecycle-automated/chainsaw-test.yaml:36`, `DECISIONS-LOG.md:3350,3786` | +| Client IP (ADR-0016) | IP-based logging and policy must read `CF-Connecting-IP` / `X-Forwarded-For` and trust them only from cloudflared; no trusted-proxy or client-IP configuration exists in `catalog/`, `platform/` or `apps/`. | `platform/cloudflared/config.yaml` (no client-IP handling downstream of it) | +| R-DNS-04 | The edge must not forward 80/443; closing them is a manual router step, last recorded as not done, so the state is unverified. | `decisions/2026-11-08-dns-via-cloudflare-tunnel.md`, `runbooks/cloudflare-tunnel.md:47-55` | +| R-DNS-04, R-DNS-05, R-SEC-04 | docs/09 and docs/14 require behavioural proof at the tunnel and from the LAN through a public resolver; only L0 static checks exist, and no L2 test touches the tunnel. | `conformance/tests/l0-static/cloudflared-ingress-rules`, `conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml` | + +## Security, identity, observability and the command center + +| Requirement or area | Requirement and shortfall | Evidence (implementation repo) | +|---|---|---| +| R-IDP-02, R-IDP-06 | No management UI may be exposed unauthenticated and the C2 requires an authorized group, but Homepage at the apex is routed straight to its Service with no gate and its `forward-auth` annotation enforces nothing. | `platform/homepage/httproute.yaml`, `platform/homepage/README.md` | +| R-SEC-03, R-C2-02 | Operational actions must be audited, but the k3s API server has only OIDC arguments and no audit log or audit policy. | `infra/ansible/roles/k3s_server/tasks/main.yml` | +| R-SEC-01 | The acceptance gate requires no plaintext secret anywhere in git history (scanned), but the check inspects the tracked tree only and no history scanner runs in CI. | `conformance/tests/l0-static/no-plaintext-secrets/check.sh`, `ci/` | +| R-SEC-01 | Rotation is meant to trigger a workload restart through reconcile (e.g. a checksum annotation), but no generated mechanism exists; the only annotations are hand-stamped on Homepage and Grafana. | `platform/homepage/deployment.yaml`, `platform/monitoring/kube-prometheus-stack.yaml`, `catalog/render.py` | +| R-SEC-01 | VPN keys, DNS tokens and SMTP credentials should have a documented rotation procedure, but the only trace is a one-line note on the Cloudflare token and none exists for VPN keys or SMTP. | `runbooks/cert-issuance.md`, `runbooks/` | +| R-OBS-01 | Alloy is specified to ship node, pod and VM logs, but it collects pod logs only, with no node or journal source. | `platform/monitoring/alloy.yaml` | +| R-OBS-04 | NetworkPolicy reconcile errors are a specified platform alert, but no such rule exists. | `platform/monitoring/alerts.yaml` | +| R-OBS-04 | Restore-test failures are a specified platform alert, but only backup and schedule alerts exist. | `platform/monitoring/alerts.yaml` | +| R-OBS-04 | Keycloak login-endpoint failures are a specified alert, but the repo has no endpoint probe, only workload and database health rules. | `platform/monitoring/alerts.yaml` | +| R-OBS-01 | Every forward-auth gate is meant to be observed, but Crafty's gate has no metrics listener, so `OAuth2ProxyGateUnobserved` fires for it by design. | `platform/crafty/deployment.yaml`, `platform/oauth2-proxy/README.md` | +| R-OBS-03 | Per-service dashboards are specified as onboarding output, but the renderer emits no dashboards; the in-repo dashboards are node pressure, KubeVirt and Keycloak only. | `catalog/render.py`, `platform/monitoring/node-pressure-dashboard.yaml`, `platform/kubevirt/dashboard.yaml`, `platform/keycloak/grafana-dashboard.yaml` | +| R-OBS-01, R-ENV-02 | Log retention is specified as an overlay setting, but Loki's retention is hardcoded with no overlay knob. | `platform/monitoring/loki.yaml`, `platform/monitoring/README.md` | +| R-OBS-04 | Monitoring storage is specified on Longhorn, but Alertmanager and Grafana run without persistent volumes, so silences and notification state are lost on restart. | `platform/monitoring/kube-prometheus-stack.yaml` | +| R-OBS-02, R-OBS-04 | Acceptance requires platform alerts to fire and email in injected-fault tests, but the L3 alerting test is a scaffold and L2/L3 do not run in CI. | `conformance/tests/l3-fault-injection/alerting/chainsaw-test.yaml`, `CLAUDE.md` | +| R-C2-01 | Homepage is specified as a live catalog with status pings and per-service widgets, but it is a hand-maintained ConfigMap of link tiles with no status or widgets. | `platform/homepage/configmap.yaml`, `platform/homepage/README.md` | +| R-C2-02 | The C2 must surface platform health and firing alerts, but none is shown on the C2 landing surface; health is reachable only by linking out to Grafana. | `platform/homepage/configmap.yaml` | +| R-C2-02 | Grafana dashboards are specified as embedded in the C2, but Homepage only links to Grafana. | `platform/homepage/configmap.yaml` | +| R-C2-02 | Flux status is specified via Headlamp's Flux view, the Flux UI or a Homepage widget, but Headlamp has no Flux plugin configured and Homepage has no widget. | `platform/headlamp/helmrelease.yaml`, `platform/homepage/configmap.yaml` | +| R-C2-02 | C2 actions are specified to include VM start/stop, Flux reconcile and alert acknowledgement, but only native Headlamp workload actions exist, with no plugins for the others. | `platform/headlamp/helmrelease.yaml` | + +## Service catalog + +| Requirement or area | Requirement and shortfall | Evidence (implementation repo) | +|---|---|---| +| docs/13 catalog row 11 (kasm), R-IDP-02 | The catalog includes Kasm with native OIDC, and no ServiceDefinition, manifest or decision exists for it, so it is not onboarded. | `catalog/README.md:20` (only mention in the repo) | +| R-STATE-03, docs/13 `backup.policy` | `backup.policy` must reference a named off-cluster schedule, and `daily-offcluster` resolves to no schedule object; it is an annotation only, while coverage comes from cluster-wide Velero schedules. | `platform/velero/README.md:33-34`, `schemas/renderer-contract.md:428`, `catalog/*.yaml` (`backup.policy`) | +| R-UPD-01/02, docs/13 `update.cadence` | `update.cadence` should set how often a service checks or applies updates, and the renderer does not read it for containers or VMs. | `catalog/render.py` (no `cadence` reference), `schemas/renderer-contract.md:388-390` | + +## Delivery, updates and conformance + +| Requirement or area | Requirement and shortfall | Evidence (implementation repo) | +|---|---|---| +| R-TEST-03 | Merges require the tests for touched requirements to pass; the gate runs L0/L1 and tagged-coverage only, because L2/L3 do not run in CI (job disabled, trigger path unimplemented, cluster not declared), and a PASS names that declared gap. | `.gitea/workflows/gate.yml` (`conformance` job `if: false`), `ci/conformance.sh`, `docs/plans/l2-l3-conformance-ci-wiring.md` | +| R-TEST-03 | Phase exits require the full suite and regression runs continuously; no pipeline runs the full-suite gate and no scheduled run exists, so L2/L3 evidence comes only from operator-recorded runs, last recorded 2026-09-19. | `conformance/run.sh` (`--require-all`, `--record`), `conformance/runs/ledger.jsonl` | +| R-TEST-01, R-TEST-04 | A requirement with no passing test is not satisfied; all 51 registry requirements have a tagged test but PROVEN is 0 (3 stale: R-NET-06, R-STATE-02, R-STATE-03; 48 not derivable from the ledger). | `conformance/lib/status.py`, `conformance/runs/ledger.jsonl` | +| R-OBS-02, R-GITOPS-02, R-GITOPS-03 | Each requirement needs an executable behavior test; the only verifying tests (`alerting`, `onboarding-completeness`, `gitops-drift`) are scaffolds. | `conformance/lib/status.py`, `conformance/tests/` | +| R-TEST-06 | The spec must be verifiably immutable to agents; CI never checks the spec repo out, so spec-immutability always skips there (tolerated) and only local gates with a sibling checkout enforce it. | `.gitea/workflows/gate.yml:193,227`, `ci/spec-immutability.sh`, `ci/gate.sh` (`REQUIRE_SPEC_VERIFICATION`) | +| R-UPD-01, R-UPD-02 | Hands-off updates must be proven by a behavior test; the only verifying test exercises Flux image-policy resolution, not the Renovate path that delivers updates. | `conformance/tests/l3-fault-injection/hands-off-update/chainsaw-test.yaml` | +| R-UPD-03 | A pinned service must not advance; the ServiceDefinition `update.pin` only affects a disabled render path, so it would not stop Renovate (no catalog entry uses it; a Renovate rule is the working pin). | `catalog/render.py:1731`, `ci/render-golden.sh`, `renovate.json5` | +| R-UPD-03 | A failed update must auto-roll-back; only HelmReleases have remediation, while plain Deployments stall on readiness and `autoRollbackOnFailedHealth` renders only onto the disabled image policy. | `catalog/render.py:1747`, `conformance/tests/l3-fault-injection/hands-off-update/chainsaw-test.yaml` | +| R-UPD-02, R-OBS-04 | Update failures must alert; the update alerts watch Flux objects that do not exist and so cannot fire, and nothing alerts when Renovate's schedule stops. | `platform/monitoring/alerts.yaml:1538,1552`, `.gitea/workflows/renovate.yaml` header | diff --git a/roadmap/phases.md b/roadmap/phases.md index 201237a..541bb19 100644 --- a/roadmap/phases.md +++ b/roadmap/phases.md @@ -2,7 +2,9 @@ Delivery is phased. Each phase has an **entry gate** (what must be true to start), **exit criteria** (demonstrable outcomes tied to requirement IDs), and the **workstreams** ([workstreams.md](workstreams.md)) -that execute it. **Everything is proven in dev/test; prod is design-only until Phase 7.** +that execute it. **Status: Phases 0–6 are delivered and Phase 7 is open** (see +[known gaps](../docs/16-known-gaps.md) for what remains unproven). Phases 1–6 were first built on a +dev/test cluster, decommissioned in 2026-09; prod is the only environment (ADR-0017). A phase's exit criteria are **passing automated conformance tests** (per [`../AGENTS.md` §7](../AGENTS.md) and [`../docs/14-testing-and-conformance.md`](../docs/14-testing-and-conformance.md)), not "manifests @@ -26,12 +28,12 @@ every subsequent phase ships with tests, not after them.** runs green with the coverage report wired**; no cluster yet. **Serves:** R-GITOPS-02, R-ENV-02, R-TEST-01..04. -## Phase 1 — dev/test Cluster Bring-up (IaC) +## Phase 1 — Cluster Bring-up (IaC) **Entry:** Phase 0 done. -**Do:** OpenTofu libvirt node VMs + cloud-init baseline + Ansible k3s HA install with **Cilium**; -disable bundled flannel/kube-proxy/Traefik; bootstrap **Flux**; bootstrap **SOPS age key**. -**Exit:** 3-node k3s HA cluster with Cilium up; Flux reconciling from git; a dev node VM can be -destroyed and rebuilt from IaC. **Prove:** node-loss re-provision. +**Do:** node baseline + Ansible k3s HA install with **Cilium**; disable bundled +flannel/kube-proxy/Traefik; bootstrap **Flux**; bootstrap **SOPS age key**. +**Exit:** 3-node k3s HA cluster with Cilium up; Flux reconciling from git; a node can be +destroyed and rebuilt from code. **Prove:** node-loss re-provision. **Serves:** R-PLAT-01, R-ENV-01, R-GITOPS-01, ADR-0001/0002/0004/0007. ## Phase 2 — Platform Core: Network, Storage, TLS, DNS, Ingress @@ -46,26 +48,27 @@ split-horizon DNS foundation. - Auto DNS record appears for a test route (R-DNS-02). **Serves:** R-NET-01/02/03, R-STATE-01/02, R-TLS-01/02/03, R-DNS-01/02/03, ADR-0002/0003/0006. -## Phase 3 — Egress, VPN & Multi-WAN +## Phase 3 — Egress & VPN **Entry:** Phase 2 done; OQ-1/OQ-3 answered enough to build. -**Do:** implement egress classes; VPN egress (Gluetun sidecar default) with **kill-switch**; -integrate edge-router multi-WAN split for the `internet` class. +**Do:** implement egress classes; node-level VPN egress steered by Cilium Egress Gateway with a +fail-closed **kill-switch** (ADR-0015, ADR-0020). **Exit (demonstrate):** - `vpn` test workload egresses via VPN IP; `internet` workload via home/WAN IP (R-NET-05). - **Kill-switch:** tunnel forced down ⇒ `vpn` workload leaks nothing; `internet` unaffected (R-NET-06). - Flipping `egress` class in git re-routes without workload rewrite (R-NET-07). -**Serves:** R-NET-04/05/06/07, ADR-0008. +**Serves:** R-NET-04/05/06/07, ADR-0013/0015/0020. ## Phase 4 — Observability, Backup/DR & Automated Updates **Entry:** Phase 2 done (can overlap Phase 3). -**Do:** kube-prometheus-stack + Loki + Alloy + Alertmanager(email); platform-health alerts; Velero + -Longhorn off-cluster backup; Flux image-automation + Renovate; per-service update policy plumbing. +**Do:** kube-prometheus-stack + Loki + Alloy + Alertmanager(email); platform-health alerts; Velero +and CNPG Barman off-cluster backup to Cloudflare R2; Renovate for images and charts; per-service +update policy plumbing. **Exit (demonstrate):** - Injected fault emails an alert; dashboards render (R-OBS-01/02/03/04). - DR drill: restore a namespace + volumes from off-cluster backup (R-STATE-03). - A test service auto-advances a stable version on schedule with no human/AI edit; a bad update auto-rolls-back and alerts (R-UPD-01/02/03). -**Serves:** R-OBS-*, R-STATE-03, R-UPD-*, ADR-0009/0003. +**Serves:** R-OBS-*, R-STATE-03, R-UPD-*, ADR-0003/0018/0019/0023. ## Phase 5 — IaaS Proof (KubeVirt) **Entry:** Phases 2–3 done. @@ -76,7 +79,7 @@ survives rebuild; default-deny + `egress` apply to the VM; guest auto-updates. ## Phase 6 — Service Catalog Onboarding + C2 UI **Entry:** Phases 2–4 done. -**Do:** onboard the 14 services via ServiceDefinitions in dependency order (see workstream WS-APPS); +**Do:** onboard the catalog services in dependency order (see workstream WS-APPS); **bring Gitea and Keycloak online early in this phase** — Keycloak is the foundational IdP so services adopt SSO (native OIDC / forward-auth) as they onboard ([`../docs/15-identity-and-access.md`](../docs/15-identity-and-access.md)); and cut the GitOps source + CI/CD over to Gitea (Gitea @@ -89,14 +92,15 @@ seerr↔plex dependency graph works and nothing undeclared does; `home.keithjack with live links + health. **Serves:** R-C2-01/02, all app-facing requirements, [`../docs/13-service-catalog.md`](../docs/13-service-catalog.md). -## Phase 7 — Production Readiness (design → build) -**Entry:** Phases 1–6 proven in dev/test; operator declares prod hardware ready. +## Phase 7 — Production Readiness +**Status:** open; opened 2026-08-31. +**Entry:** Phases 1–6 delivered; operator declares prod hardware ready. **Do:** finalize bare-metal provisioning checklist; apply the **prod overlay** (no base edits); -full DR drill on prod; cutover plan. +full DR drill on prod; decommission dev/test. **Exit:** prod 3× mini-PC cluster runs the same config via the prod overlay; DR proven; monitoring + alerts live. **Serves:** R-ENV-02/03, R-PLAT-01. -**Note:** Do not begin building on prod hardware before this phase is explicitly opened +**Note:** This phase was explicitly opened by the operator before any build on prod hardware ([`../AGENTS.md` §6](../AGENTS.md)). --- diff --git a/roadmap/workstreams.md b/roadmap/workstreams.md index 9e99300..889bbfb 100644 --- a/roadmap/workstreams.md +++ b/roadmap/workstreams.md @@ -8,14 +8,14 @@ only via [`../agents/coordination-protocol.md`](../agents/coordination-protocol. | ID | Workstream | Owns (specs) | Serves | Depends on | |---|---|---|---|---| | **WS-INFRA** | Infra & cluster bring-up | [02](../docs/02-environments-and-provisioning.md) | R-PLAT-01, R-ENV-* | — | -| **WS-GITOPS** | GitOps, CI, catalog renderer | [05](../docs/05-gitops-and-delivery.md), [13](../docs/13-service-catalog.md) | R-GITOPS-*, R-UPD-02 | WS-INFRA | +| **WS-GITOPS** | GitOps, CI, catalog renderer, image pipeline, PM agent | [05](../docs/05-gitops-and-delivery.md), [13](../docs/13-service-catalog.md) | R-GITOPS-*, R-UPD-02 | WS-INFRA | | **WS-NET** | Networking, isolation, egress, ingress | [03](../docs/03-networking-isolation-and-egress.md), [09](../docs/09-dns-and-ingress.md) | R-NET-*, R-DNS-* | WS-INFRA | | **WS-STORAGE** | Storage, state, backup/DR | [04](../docs/04-storage-and-state.md) | R-STATE-* | WS-INFRA | | **WS-SEC** | Secrets, TLS, RBAC, **identity/SSO**, auditability | [07](../docs/07-security-and-secrets.md), [08](../docs/08-tls-and-certificates.md), [15](../docs/15-identity-and-access.md) | R-SEC-*, R-TLS-*, R-IDP-* | WS-INFRA, WS-NET | | **WS-OBS** | Observability & alerting | [10](../docs/10-observability-and-alerting.md) | R-OBS-* | WS-INFRA, WS-STORAGE | | **WS-UPDATE** | Automated updates | [11](../docs/11-automated-updates.md) | R-UPD-* | WS-GITOPS, WS-OBS | | **WS-VM** | KubeVirt IaaS | [06](../docs/06-vm-workloads-kubevirt.md) | R-PLAT-03 | WS-NET, WS-STORAGE | -| **WS-APPS** | Service onboarding (13 services) | [13](../docs/13-service-catalog.md) | app requirements | WS-GITOPS, WS-NET, WS-STORAGE, WS-SEC, WS-OBS | +| **WS-APPS** | Service onboarding | [13](../docs/13-service-catalog.md) | app requirements | WS-GITOPS, WS-NET, WS-STORAGE, WS-SEC, WS-OBS | | **WS-C2** | Command & Control UI | [12](../docs/12-command-center-ui.md) | R-C2-* | WS-OBS, WS-APPS, WS-SEC | | **WS-TEST** | Conformance testing (cross-cutting) | [14](../docs/14-testing-and-conformance.md) | R-TEST-* | WS-INFRA, WS-GITOPS | @@ -26,14 +26,14 @@ only via [`../agents/coordination-protocol.md`](../agents/coordination-protocol. contract in the system.** - **Network baseline & policy rendering** (WS-NET) — default-deny + how `dependencies` and `egress` compile to CiliumNetworkPolicies/egress routing. Consumed by WS-APPS/WS-VM. -- **StorageClasses** (WS-STORAGE) — `longhorn` (replicated) and `bulk` (media). Consumed by everyone - with state. +- **StorageClasses** (WS-STORAGE) — the `longhorn` family (replicated, including static and VM + variants) and `bulk` (media, NFS). Consumed by everyone with state. - **Issuer + Gateway TLS** (WS-SEC + WS-NET) — wildcard cert + Gateway listeners that HTTPRoutes attach to. - **Observability contract** (WS-OBS) — how a service declares metrics/SLIs/alerts and gets dashboards. Consumed by WS-APPS/WS-VM. -- **Update policy contract** (WS-UPDATE) — ImagePolicy/Renovate conventions a ServiceDefinition's - `update` block maps to. +- **Update policy contract** (WS-UPDATE) — the Renovate conventions a ServiceDefinition's `update` + block maps to. - **SSO/auth contract** (WS-SEC) — how a ServiceDefinition's `auth` block (oidc / forward-auth / native / groups) maps to a Keycloak client or oauth2-proxy protection ([15](../docs/15-identity-and-access.md)). Consumed by WS-APPS/WS-C2. @@ -61,17 +61,18 @@ only via [`../agents/coordination-protocol.md`](../agents/coordination-protocol. 3. **keycloak (+ oauth2-proxy)** — **foundational IdP/SSO**, brought online early so later services integrate SSO as they onboard ([`../docs/15-identity-and-access.md`](../docs/15-identity-and-access.md)). DR-critical + break-glass path. → -4. **plex** (largest; validates bulk tier + public exposure) → +4. **plex** (largest; validates bulk tier + L4 remote access) → 5. **qbittorrent, jackett** (VPN + kill-switch proof) → 6. **sonarr, radarr** (VPN + arr↔downloader allows) → 7. **seerr** (cross-service allows to arr + plex) → 8. **nextcloud** (DR-critical data + -backups) → 9. **minecraft, unifi, semaphore, kasm** (remaining). +backups) → 9. **minecraft** (delivered as Crafty, platform-managed), **unifi** (a KubeVirt VM), +**semaphore** → **kasm** (not onboarded). As each service onboards, it adopts SSO per its `auth` mode (native OIDC or forward-auth) — [`../docs/15-identity-and-access.md`](../docs/15-identity-and-access.md). -**GitOps source migration** (WS-GITOPS, tied to step 2): local bootstrap source → self-hosted Gitea; -CI (Kyverno gate + Chainsaw conformance Job trigger) moves to Gitea Actions; PRs become the enforced -merge gate into `main`. +**GitOps source** (WS-GITOPS, tied to step 2): Flux follows the self-hosted, in-cluster Gitea; CI +(Kyverno gate + Chainsaw conformance Job trigger) runs on Gitea Actions; PRs are the enforced merge +gate into `main`. Each onboarding is a full acceptance-gate pass ([`../agents/definition-of-done.md`](../agents/definition-of-done.md)), not just a merged manifest.