home-network-k3s/docs/05-gitops-and-delivery.md

05 — GitOps & Delivery

Serves: R-GITOPS-01..03, R-PLAT-02/04, R-ENV-02, R-TEST-03. Related: ADR-0004 (Flux), ADR-0023 (updates), ADR-0024 (delivery authority), ADR-0025 (pm-agent).

Model

Pull-based GitOps with Flux CD. The cluster continuously reconciles its state from git. Git is the single source of truth; the cluster conforms to git, not the reverse (R-GITOPS-01). Drift is detected and corrected automatically (R-GITOPS-03).

Repository layout (the implementation repo)

This spec repo (home-network-k3s) holds intent. The implementation repo holds the reconciled manifests, the gate and the tests. Its top-level areas:

catalog/                  ← ServiceDefinition entries (the onboarding surface) + the renderer
schemas/                  ← ServiceDefinition JSON Schema
apps/                     ← application workloads by reconcile wave, plus VMs; rendered from
                            catalog/ and committed (the render is checked against them)
platform/                 ← cluster-wide platform services (Flux-managed): CNI, storage, TLS, DNS,
                            Gateway, public tunnel, observability, backup/DR, KubeVirt, identity,
                            admission policy, image builder, delivery automation
platform-config*/         ← platform configuration Flux applies after the platform tier, including
                            the admission policies
clusters/
  base/                   ← shared Flux entrypoints and reconcile ordering
  prod/                   ← the live cluster's overlay and environment values
bootstrap/                ← Flux, SOPS and Gitea bootstrap
infra/                    ← node provisioning as code; Gitea settings; dev toolbox; agent image
ci/  .gitea/              ← the merge gate and the workflows that run it
conformance/              ← requirement registry, behavior tests, harness, run ledger
runbooks/                 ← operator procedures (also served to the alert links)
decisions/                ← dated implementation handoff log (distinct from this repo's ADRs)

• Kustomize overlays express environment differences (R-ENV-02). The base is shared; an environment overlay supplies scoped values only. prod is the one live environment; the dev environment was decommissioned in 2026-09 (ADR-0017).
• HelmReleases (Flux) wrap upstream charts (Cilium, Longhorn, cert-manager, kube-prometheus- stack, etc.); app workloads use charts where good ones exist, otherwise plain manifests.
• Rendered output is committed. apps/ is generated from catalog/; the merge gate fails when the two disagree, so a ServiceDefinition is always the source of a rendered workload.

The onboarding surface: ServiceDefinition (R-GITOPS-02)

Onboarding a service = adding one ServiceDefinition under catalog/. It is the operator-facing abstraction that the platform compiles into the concrete resources (Deployment/VM, PVCs, NetworkPolicies, HTTPRoute, Certificate, DNS record, ServiceMonitor, update policy). See 13-service-catalog.md for the schema. Intent lives in one place; the platform renders the rest.

Reconciliation flow

1. A contributor opens a PR against the implementation repo (new/updated ServiceDefinition or platform change). Contributors are the operator, implementation agents, Renovate and pm-agent; all of them take the same path.
2. The merge gate validates the change (below).
3. Merge to main, the tracked branch.
4. Flux reconciles from the platform's own Gitea: applies manifests, reports status, corrects drift on an interval.
5. Health/rollout status surfaces in the C2 UI (12) and alerts on failure (10).

Nothing reaches main by any other route: direct pushes are rejected for every account.

The merge gate (R-TEST-03)

The implementation repo's main is protected by one required status check, the merge gate, which runs on every PR as a Gitea Actions job. A red gate is unmergeable by everyone, administrators included. The gate runs in the same pinned toolbox image locally and in CI (R-TEST-02) and establishes, in order: the adopted spec pin (R-TEST-06), the handoff log's format, lint, L0 schema + policy + repository invariants, render-equals-committed, L1 rendering, the sensitive-path approval policy, that the live Gitea settings match their declaration, and L2 conformance (14; see known gaps).

Its verdict is one of three: pass, blocked, or undetermined. A stage that did not run is never green, and no automation may read a non-answer as approval. The rule is that a capability the environment declares absent is named in the verdict, while a stage that could not run where it must blocks. Detail: ci/README.md in the implementation repo.

Who approves (ADR-0024)

The CI gate is the approver. A green gate is sufficient for an ordinary change; branch protection requires no approval count. A weak gate is a defect to fix, not a reason to add a human.

One exception: a change to the gate cannot be gated by the gate. A change touching the pipeline, the tests and harness, the Flux wiring, the admission policies, the SOPS recipients and secrets, the RBAC, or the delivery contract needs an approving review from someone other than its author. The path list is data, with a stated reason per entry (infra/gitea/approval-policy.json in the implementation repo), and is itself a sensitive path. This stage is fail-closed: if it cannot evaluate, it blocks.

Attributed administrator override. There is one human on this platform, and Gitea refuses self-approval, so an administrator may waive that review with an explicit, reasoned directive on the PR. Authority is re-checked live, a new commit invalidates it, and the overrider, time and reason are recorded in the gate output. For an administrator the exception above is therefore advisory, not structural; what it guarantees is an attributed record of each waiver. Agents never post this directive to clear their own work, whatever permission their account holds.

Secrets in GitOps

Secrets are committed encrypted with SOPS + age; Flux decrypts them in-cluster at reconciliation. No plaintext secret ever lands in git. See 07-security-and-secrets.md and ADR-0007.

Automated updates interplay (R-UPD-*)

GitOps and hands-off updates coexist because the update bot is one more contributor to the same flow. Renovate runs on a schedule and proposes machine-authored PRs for container images, Helm charts and manifest versions; each passes the merge gate, and allowed update classes auto-merge on green (R-UPD-02). No in-cluster controller holds a credential to write to the repo that defines the cluster. Full detail is in 11-automated-updates.md and ADR-0023.

Images are built by CI

Container images the platform itself produces (the toolbox, the agent runner, the conformance runner) are built, verified and published only by the pipeline, never from a workstation. A PR builds and verifies; a push to main publishes to the Gitea registry. References are pinned by digest, and Renovate carries each new digest into git as a PR. No CI job may reach a privileged container daemon: builds go to a rootless builder over mutually-authenticated TLS.

Delivery automation: pm-agent (ADR-0025)

An in-cluster AI Program Manager reads labelled issues on the implementation repo, decides what should happen about each thread, and runs one worker agent per assignment to author a change, run the merge gate and open a PR. It is delivery tooling rather than a catalog service: it declares Kubernetes objects a ServiceDefinition cannot express, and it requires no R-* of its own.

It is contributor-shaped, never approver-shaped. It cannot push to main, it never merges, its account is not an administrator, and its workers hold no cluster credential. It is an implementation agent bound by AGENTS.md and R-TEST-06: spec work is out of its reach and is declined and escalated.

Bootstrap & recovery

• Flux is bootstrapped onto a fresh cluster by the Ansible handoff (02).
• Because desired state is in git and data is on Longhorn/backups, a rebuilt cluster reconverges by re-bootstrapping Flux and restoring volumes (04).

Source-of-truth: self-hosted Gitea

The platform hosts its own git at gitea.home.keithjackson.dev (../CONTRIBUTING.md); Flux reconciles from it and Gitea Actions runs the merge gate. Standing the platform up from nothing follows a deliberate order:

1. Bootstrap source. The cluster is first reconciled from an out-of-band copy of the repo.
2. Gitea comes online early. Gitea is one of the first services onboarded (13).
3. Cut Flux over to Gitea, so the platform reconciles from its own git.
4. CI/CD runs in Gitea Actions, with PRs as the enforced merge gate into main.

Circularity & DR. Gitea runs on the cluster Flux reconciles from Gitea. Mitigations:

• Keep an off-cluster mirror of the platform repo so a source-of-truth exists that outlives the cluster (see known gaps).
• Treat Gitea as DR-critical: its repos/DB are backed up off-cluster (04), so a cluster loss never loses the git history needed to rebuild the platform.

Guardrails

• No kubectl apply of app state by hand — everything flows through git/Flux (R-GITOPS-01, click- ops-never per AGENTS.md).
• Reconciliation errors and drift are alerting conditions (R-OBS-04).
• Base config is never forked per environment (R-ENV-02).

Acceptance gates

• Adding a ServiceDefinition PR results in a fully wired service (compute, storage, policy, TLS, DNS, monitoring) with no manual steps beyond the merge.
• Manually mutating a live resource is auto-reverted by Flux (drift correction proven).
• The base config renders correctly under the environment overlay, which supplies scoped values only.