Home-Network/home-network-k3s
2
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: bring the specification in line with the accepted implementation

Browse Source 
The operator ruled the implementation repo's main (50c8719) the latest
and accepted state. Sync the spec to it.

Accepted changes are reflected: one bare-metal environment, TrueNAS bulk
storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway
with Cilium fail-closed, inbound WireGuard, per-app auth sidecars,
Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025;
ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended.

Where the implementation has simply not met a requirement, the
requirement stands and the shortfall is recorded in docs/16-known-gaps.md.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5.5 committed 2026-11-08 22:46:00 -06:00
1 parent 1ddc14b918
commit 250fb54154
43 files changed
+1672 -575
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/05-gitops-and-delivery.md
+110 -50
View File
@@ -1,6 +1,10 @@
# 05 — GitOps & Delivery
Serves: **R-GITOPS-01..03**, **R-PLAT-02/04**, **R-ENV-02**. Related: [ADR-0004 (Flux)](../decisions/adr-0004-gitops-flux.md).
Serves: **R-GITOPS-01..03**, **R-PLAT-02/04**, **R-ENV-02**, **R-TEST-03**. Related:
[ADR-0004 (Flux)](../decisions/adr-0004-gitops-flux.md),
[ADR-0023 (updates)](../decisions/adr-0023-automated-updates-renovate.md),
[ADR-0024 (delivery authority)](../decisions/adr-0024-delivery-authority-ci-gate.md),
[ADR-0025 (pm-agent)](../decisions/adr-0025-in-cluster-ai-program-manager.md).
## Model
@@ -8,35 +12,40 @@ Pull-based GitOps with **Flux CD**. The cluster continuously reconciles its stat
the single source of truth; the cluster conforms to git, not the reverse (R-GITOPS-01). Drift is
detected and corrected automatically (R-GITOPS-03).
## Repository layout (the implementation repo, created in a later phase)
## Repository layout (the implementation repo)
> This spec repo (`home-network-k3s`) holds intent. The **implementation** repo(s) hold the
> reconciled manifests. Proposed structure:
> This spec repo (`home-network-k3s`) holds intent. The
> [implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl)
> holds the reconciled manifests, the gate and the tests. Its top-level areas:
```
platform/ ← cluster-wide platform services (Flux-managed)
cilium/ ← CNI, egress gateway, network baseline
longhorn/ ← storage
cert-manager/ ← TLS issuers + ClusterIssuer
externaldns/ ← DNS automation
gateway/ ← Gateway API + listeners/TLS
monitoring/ ← kube-prometheus-stack, Loki, Alloy, Alertmanager
velero/ ← backup/DR
kubevirt/ ← KubeVirt + CDI
image-automation/ ← Flux image update automation config
apps/ ← application workloads, one dir per service
plex/ sonarr/ radarr/ jackett/ seerr/ pihole/ nextcloud/
minecraft/ unifi/ semaphore/ kasm/ gitea/ qbittorrent/
catalog/ ← ServiceDefinition entries (the onboarding surface)
catalog/ ← ServiceDefinition entries (the onboarding surface) + the renderer
schemas/ ← ServiceDefinition JSON Schema
apps/ ← application workloads by reconcile wave, plus VMs; rendered from
catalog/ and committed (the render is checked against them)
platform/ ← cluster-wide platform services (Flux-managed): CNI, storage, TLS, DNS,
Gateway, public tunnel, observability, backup/DR, KubeVirt, identity,
admission policy, image builder, delivery automation
platform-config*/ ← platform configuration Flux applies after the platform tier, including
the admission policies
clusters/
dev/ ← Flux entrypoint + dev overlay/vars
prod/ ← Flux entrypoint + prod overlay/vars (design-only)
base/ ← shared Flux entrypoints and reconcile ordering
prod/ ← the live cluster's overlay and environment values
bootstrap/ ← Flux, SOPS and Gitea bootstrap
infra/ ← node provisioning as code; Gitea settings; dev toolbox; agent image
ci/ .gitea/ ← the merge gate and the workflows that run it
conformance/ ← requirement registry, behavior tests, harness, run ledger
runbooks/ ← operator procedures (also served to the alert links)
decisions/ ← dated implementation handoff log (distinct from this repo's ADRs)
```
- **Kustomize overlays** (or equivalent) express environment differences (R-ENV-02). Base is shared;
`clusters/dev` and `clusters/prod` supply scoped values only.
- **Kustomize overlays** express environment differences (R-ENV-02). The base is shared; an
environment overlay supplies scoped values only. `prod` is the one live environment; the dev
environment was decommissioned in 2026-09 ([ADR-0017](../decisions/adr-0017-single-environment-bare-metal-production.md)).
- **HelmReleases** (Flux) wrap upstream charts (Cilium, Longhorn, cert-manager, kube-prometheus-
stack, etc.); app workloads use charts where good ones exist, otherwise plain manifests.
- **Rendered output is committed.** `apps/` is generated from `catalog/`; the merge gate fails when
the two disagree, so a ServiceDefinition is always the source of a rendered workload.
## The onboarding surface: ServiceDefinition (R-GITOPS-02)
@@ -48,14 +57,52 @@ platform renders the rest.
## Reconciliation flow
1. Operator opens a PR against the implementation repo (new/updated ServiceDefinition or platform
change).
2. CI validates (lint, policy checks, schema validation, dry-run/build).
3. Merge to the tracked branch.
4. Flux reconciles: applies manifests, reports status, corrects drift on an interval.
1. A contributor opens a PR against the implementation repo (new/updated ServiceDefinition or
platform change). Contributors are the operator, implementation agents, Renovate and pm-agent;
all of them take the same path.
2. The **merge gate** validates the change (below).
3. Merge to `main`, the tracked branch.
4. Flux reconciles from the platform's own Gitea: applies manifests, reports status, corrects drift
on an interval.
5. Health/rollout status surfaces in the C2 UI ([12](12-command-center-ui.md)) and alerts on failure
([10](10-observability-and-alerting.md)).
Nothing reaches `main` by any other route: direct pushes are rejected for every account.
## The merge gate (R-TEST-03)
The implementation repo's `main` is protected by one required status check, the merge gate, which
runs on every PR as a Gitea Actions job. **A red gate is unmergeable by everyone, administrators
included.** The gate runs in the same pinned toolbox image locally and in CI (R-TEST-02) and
establishes, in order: the adopted spec pin (R-TEST-06), the handoff log's format, lint, L0 schema +
policy + repository invariants, render-equals-committed, L1 rendering, the sensitive-path approval
policy, that the live Gitea settings match their declaration, and L2/L3 conformance
([14](14-testing-and-conformance.md); see [known gaps](16-known-gaps.md)).
Its verdict is one of three: **pass**, **blocked**, or **undetermined**. A stage that did not run is
never green, and no automation may read a non-answer as approval. The rule is that a capability the
environment *declares* absent is named in the verdict, while a stage that *could not run where it
must* blocks. Detail: `ci/README.md` in the implementation repo.
## Who approves ([ADR-0024](../decisions/adr-0024-delivery-authority-ci-gate.md))
**The CI gate is the approver.** A green gate is sufficient for an ordinary change; branch
protection requires no approval count. A weak gate is a defect to fix, not a reason to add a human.
**One exception: a change to the gate cannot be gated by the gate.** A change touching the pipeline,
the tests and harness, the Flux wiring, the admission policies, the SOPS recipients and secrets, the
RBAC, or the delivery contract needs an approving review from someone other than its author. The
path list is data, with a stated reason per entry (`infra/gitea/approval-policy.json` in the
implementation repo), and is itself a sensitive path. This stage is fail-closed: if it cannot
evaluate, it blocks.
**Attributed administrator override.** There is one human on this platform, and Gitea refuses
self-approval, so an administrator may waive that review with an explicit, reasoned directive on the
PR. Authority is re-checked live, a new commit invalidates it, and the overrider, time and reason
are recorded in the gate output. For an administrator the exception above is therefore advisory, not
structural; what it guarantees is an attributed record of each waiver. **Agents never post this
directive to clear their own work**, whatever permission their account holds.
## Secrets in GitOps
Secrets are committed **encrypted** with **SOPS + age**; Flux decrypts them in-cluster at
@@ -64,15 +111,32 @@ and [ADR-0007](../decisions/adr-0007-secrets-sops-age.md).
## Automated updates interplay (R-UPD-*)
GitOps and hands-off updates coexist:
- **Container image tags** advance automatically via **Flux image-update-automation** driven by
per-service **image policies** (semver range / stable channel) — the controller resolves and
reconciles the new version with **no human-authored change** (R-UPD-02).
- **Helm chart / manifest version bumps** are proposed by **Renovate** with auto-merge for allowed
update classes.
GitOps and hands-off updates coexist because the update bot is one more contributor to the same
flow. **Renovate** runs on a schedule and proposes machine-authored PRs for container images, Helm
charts and manifest versions; each passes the merge gate, and allowed update classes auto-merge on
green (R-UPD-02). No in-cluster controller holds a credential to write to the repo that defines the
cluster. Full detail is in [`11-automated-updates.md`](11-automated-updates.md) and
[ADR-0023](../decisions/adr-0023-automated-updates-renovate.md).
Full detail and the "is an automated commit a GitOps change?" nuance is in
[`11-automated-updates.md`](11-automated-updates.md) and [ADR-0009](../decisions/adr-0009-automated-image-updates.md).
## Images are built by CI
Container images the platform itself produces (the toolbox, the agent runner, the conformance
runner) are built, verified and published only by the pipeline, never from a workstation. A PR
builds and verifies; a push to `main` publishes to the Gitea registry. References are pinned by
digest, and Renovate carries each new digest into git as a PR. **No CI job may reach a privileged
container daemon**: builds go to a rootless builder over mutually-authenticated TLS.
## Delivery automation: pm-agent ([ADR-0025](../decisions/adr-0025-in-cluster-ai-program-manager.md))
An in-cluster AI Program Manager reads labelled issues on the implementation repo, decides what
should happen about each thread, and runs one worker agent per assignment to author a change, run
the merge gate and open a PR. It is delivery tooling rather than a catalog service: it declares
Kubernetes objects a ServiceDefinition cannot express, and it requires no `R-*` of its own.
It is contributor-shaped, never approver-shaped. It cannot push to `main`, it never merges, its
account is not an administrator, and its workers hold no cluster credential. It is an implementation
agent bound by [AGENTS.md](../AGENTS.md) and R-TEST-06: spec work is out of its reach and is
declined and escalated.
## Bootstrap & recovery
@@ -80,25 +144,21 @@ Full detail and the "is an automated commit a GitOps change?" nuance is in
- Because desired state is in git and data is on Longhorn/backups, a rebuilt cluster reconverges by
re-bootstrapping Flux and restoring volumes ([04](04-storage-and-state.md)).
## Source-of-truth: local bootstrap → self-hosted Gitea
## Source-of-truth: self-hosted Gitea
The platform hosts its own git, which creates a deliberate bootstrapping sequence
([`../CONTRIBUTING.md`](../CONTRIBUTING.md)):
The platform hosts its own git at `gitea.home.keithjackson.dev`
([`../CONTRIBUTING.md`](../CONTRIBUTING.md)); Flux reconciles from it and Gitea Actions runs the
merge gate. Standing the platform up from nothing follows a deliberate order:
1. **Bootstrap source.** Initially there is no remote git on this network (the prior `home-network`
Gitea is gone from here). Development is local; the cluster is first reconciled from a
**bootstrap source** (this local repo / an out-of-band source).
2. **Gitea comes online early.** **Gitea is one of the first services onboarded** ([13](13-service-catalog.md))
so the platform gains its own git host at `gitea.home.keithjackson.dev`.
3. **Cut Flux over to Gitea.** Once Gitea is stable, Flux's `GitRepository` source is migrated from
the bootstrap source to Gitea; the platform then reconciles from its own git.
4. **CI/CD moves to Gitea Actions.** The Kyverno L0 policy gate and the trigger for the in-cluster
Chainsaw conformance Job ([14](14-testing-and-conformance.md)) run as Gitea Actions pipelines on
push/PR; PRs become the enforced merge gate into `main`.
1. **Bootstrap source.** The cluster is first reconciled from an out-of-band copy of the repo.
2. **Gitea comes online early.** Gitea is one of the first services onboarded
([13](13-service-catalog.md)).
3. **Cut Flux over to Gitea**, so the platform reconciles from its own git.
4. **CI/CD runs in Gitea Actions**, with PRs as the enforced merge gate into `main`.
**Circularity & DR.** Gitea runs *on* the cluster Flux reconciles *from* Gitea. Mitigations:
- Keep an **off-cluster mirror** of the platform repo so a source-of-truth exists that outlives the
cluster.
cluster (see [known gaps](16-known-gaps.md)).
- Treat **Gitea as DR-critical**: its repos/DB are backed up off-cluster ([04](04-storage-and-state.md)),
so a cluster loss never loses the git history needed to rebuild the platform.
@@ -112,4 +172,4 @@ The platform hosts its own git, which creates a deliberate bootstrapping sequenc
- Adding a ServiceDefinition PR results in a fully wired service (compute, storage, policy, TLS, DNS,
monitoring) with no manual steps beyond the merge.
- Manually mutating a live resource is auto-reverted by Flux (drift correction proven).
- The same base config renders correctly for both dev and prod overlays.
- The base config renders correctly under the environment overlay, which supplies scoped values only.