# 05 — GitOps & Delivery Serves: **R-GITOPS-01..03**, **R-PLAT-02/04**, **R-ENV-02**, **R-TEST-03**. Related: [ADR-0004 (Flux)](../decisions/adr-0004-gitops-flux.md), [ADR-0023 (updates)](../decisions/adr-0023-automated-updates-renovate.md), [ADR-0024 (delivery authority)](../decisions/adr-0024-delivery-authority-ci-gate.md), [ADR-0025 (pm-agent)](../decisions/adr-0025-in-cluster-ai-program-manager.md). ## Model Pull-based GitOps with **Flux CD**. The cluster continuously reconciles its state from git. Git is the single source of truth; the cluster conforms to git, not the reverse (R-GITOPS-01). Drift is detected and corrected automatically (R-GITOPS-03). ## Repository layout (the implementation repo) > This spec repo (`home-network-k3s`) holds intent. The > [implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl) > holds the reconciled manifests, the gate and the tests. Its top-level areas: ``` catalog/ ← ServiceDefinition entries (the onboarding surface) + the renderer schemas/ ← ServiceDefinition JSON Schema apps/ ← application workloads by reconcile wave, plus VMs; rendered from catalog/ and committed (the render is checked against them) platform/ ← cluster-wide platform services (Flux-managed): CNI, storage, TLS, DNS, Gateway, public tunnel, observability, backup/DR, KubeVirt, identity, admission policy, image builder, delivery automation platform-config*/ ← platform configuration Flux applies after the platform tier, including the admission policies clusters/ base/ ← shared Flux entrypoints and reconcile ordering prod/ ← the live cluster's overlay and environment values bootstrap/ ← Flux, SOPS and Gitea bootstrap infra/ ← node provisioning as code; Gitea settings; dev toolbox; agent image ci/ .gitea/ ← the merge gate and the workflows that run it conformance/ ← requirement registry, behavior tests, harness, run ledger runbooks/ ← operator procedures (also served to the alert links) decisions/ ← dated implementation handoff log (distinct from this repo's ADRs) ``` - **Kustomize overlays** express environment differences (R-ENV-02). The base is shared; an environment overlay supplies scoped values only. `prod` is the one live environment; the dev environment was decommissioned in 2026-09 ([ADR-0017](../decisions/adr-0017-single-environment-bare-metal-production.md)). - **HelmReleases** (Flux) wrap upstream charts (Cilium, Longhorn, cert-manager, kube-prometheus- stack, etc.); app workloads use charts where good ones exist, otherwise plain manifests. - **Rendered output is committed.** `apps/` is generated from `catalog/`; the merge gate fails when the two disagree, so a ServiceDefinition is always the source of a rendered workload. ## The onboarding surface: ServiceDefinition (R-GITOPS-02) Onboarding a service = adding one **ServiceDefinition** under `catalog/`. It is the operator-facing abstraction that the platform compiles into the concrete resources (Deployment/VM, PVCs, NetworkPolicies, HTTPRoute, Certificate, DNS record, ServiceMonitor, update policy). See [`13-service-catalog.md`](13-service-catalog.md) for the schema. Intent lives in one place; the platform renders the rest. ## Reconciliation flow 1. A contributor opens a PR against the implementation repo (new/updated ServiceDefinition or platform change). Contributors are the operator, implementation agents, Renovate and pm-agent; all of them take the same path. 2. The **merge gate** validates the change (below). 3. Merge to `main`, the tracked branch. 4. Flux reconciles from the platform's own Gitea: applies manifests, reports status, corrects drift on an interval. 5. Health/rollout status surfaces in the C2 UI ([12](12-command-center-ui.md)) and alerts on failure ([10](10-observability-and-alerting.md)). Nothing reaches `main` by any other route: direct pushes are rejected for every account. ## The merge gate (R-TEST-03) The implementation repo's `main` is protected by one required status check, the merge gate, which runs on every PR as a Gitea Actions job. **A red gate is unmergeable by everyone, administrators included.** The gate runs in the same pinned toolbox image locally and in CI (R-TEST-02) and establishes, in order: the adopted spec pin (R-TEST-06), the handoff log's format, lint, L0 schema + policy + repository invariants, render-equals-committed, L1 rendering, the sensitive-path approval policy, that the live Gitea settings match their declaration, and L2/L3 conformance ([14](14-testing-and-conformance.md); see [known gaps](16-known-gaps.md)). Its verdict is one of three: **pass**, **blocked**, or **undetermined**. A stage that did not run is never green, and no automation may read a non-answer as approval. The rule is that a capability the environment *declares* absent is named in the verdict, while a stage that *could not run where it must* blocks. Detail: `ci/README.md` in the implementation repo. ## Who approves ([ADR-0024](../decisions/adr-0024-delivery-authority-ci-gate.md)) **The CI gate is the approver.** A green gate is sufficient for an ordinary change; branch protection requires no approval count. A weak gate is a defect to fix, not a reason to add a human. **One exception: a change to the gate cannot be gated by the gate.** A change touching the pipeline, the tests and harness, the Flux wiring, the admission policies, the SOPS recipients and secrets, the RBAC, or the delivery contract needs an approving review from someone other than its author. The path list is data, with a stated reason per entry (`infra/gitea/approval-policy.json` in the implementation repo), and is itself a sensitive path. This stage is fail-closed: if it cannot evaluate, it blocks. **Attributed administrator override.** There is one human on this platform, and Gitea refuses self-approval, so an administrator may waive that review with an explicit, reasoned directive on the PR. Authority is re-checked live, a new commit invalidates it, and the overrider, time and reason are recorded in the gate output. For an administrator the exception above is therefore advisory, not structural; what it guarantees is an attributed record of each waiver. **Agents never post this directive to clear their own work**, whatever permission their account holds. ## Secrets in GitOps Secrets are committed **encrypted** with **SOPS + age**; Flux decrypts them in-cluster at reconciliation. No plaintext secret ever lands in git. See [`07-security-and-secrets.md`](07-security-and-secrets.md) and [ADR-0007](../decisions/adr-0007-secrets-sops-age.md). ## Automated updates interplay (R-UPD-*) GitOps and hands-off updates coexist because the update bot is one more contributor to the same flow. **Renovate** runs on a schedule and proposes machine-authored PRs for container images, Helm charts and manifest versions; each passes the merge gate, and allowed update classes auto-merge on green (R-UPD-02). No in-cluster controller holds a credential to write to the repo that defines the cluster. Full detail is in [`11-automated-updates.md`](11-automated-updates.md) and [ADR-0023](../decisions/adr-0023-automated-updates-renovate.md). ## Images are built by CI Container images the platform itself produces (the toolbox, the agent runner, the conformance runner) are built, verified and published only by the pipeline, never from a workstation. A PR builds and verifies; a push to `main` publishes to the Gitea registry. References are pinned by digest, and Renovate carries each new digest into git as a PR. **No CI job may reach a privileged container daemon**: builds go to a rootless builder over mutually-authenticated TLS. ## Delivery automation: pm-agent ([ADR-0025](../decisions/adr-0025-in-cluster-ai-program-manager.md)) An in-cluster AI Program Manager reads labelled issues on the implementation repo, decides what should happen about each thread, and runs one worker agent per assignment to author a change, run the merge gate and open a PR. It is delivery tooling rather than a catalog service: it declares Kubernetes objects a ServiceDefinition cannot express, and it requires no `R-*` of its own. It is contributor-shaped, never approver-shaped. It cannot push to `main`, it never merges, its account is not an administrator, and its workers hold no cluster credential. It is an implementation agent bound by [AGENTS.md](../AGENTS.md) and R-TEST-06: spec work is out of its reach and is declined and escalated. ## Bootstrap & recovery - Flux is bootstrapped onto a fresh cluster by the Ansible handoff ([02](02-environments-and-provisioning.md)). - Because desired state is in git and data is on Longhorn/backups, a rebuilt cluster reconverges by re-bootstrapping Flux and restoring volumes ([04](04-storage-and-state.md)). ## Source-of-truth: self-hosted Gitea The platform hosts its own git at `gitea.home.keithjackson.dev` ([`../CONTRIBUTING.md`](../CONTRIBUTING.md)); Flux reconciles from it and Gitea Actions runs the merge gate. Standing the platform up from nothing follows a deliberate order: 1. **Bootstrap source.** The cluster is first reconciled from an out-of-band copy of the repo. 2. **Gitea comes online early.** Gitea is one of the first services onboarded ([13](13-service-catalog.md)). 3. **Cut Flux over to Gitea**, so the platform reconciles from its own git. 4. **CI/CD runs in Gitea Actions**, with PRs as the enforced merge gate into `main`. **Circularity & DR.** Gitea runs *on* the cluster Flux reconciles *from* Gitea. Mitigations: - Keep an **off-cluster mirror** of the platform repo so a source-of-truth exists that outlives the cluster (see [known gaps](16-known-gaps.md)). - Treat **Gitea as DR-critical**: its repos/DB are backed up off-cluster ([04](04-storage-and-state.md)), so a cluster loss never loses the git history needed to rebuild the platform. ## Guardrails - No `kubectl apply` of app state by hand — everything flows through git/Flux (R-GITOPS-01, click- ops-never per [AGENTS.md](../AGENTS.md)). - Reconciliation errors and drift are **alerting conditions** (R-OBS-04). - Base config is never forked per environment (R-ENV-02). ## Acceptance gates - Adding a ServiceDefinition PR results in a fully wired service (compute, storage, policy, TLS, DNS, monitoring) with no manual steps beyond the merge. - Manually mutating a live resource is auto-reverted by Flux (drift correction proven). - The base config renders correctly under the environment overlay, which supplies scoped values only.