home-network-k3s/docs/05-gitops-and-delivery.md

05 — GitOps & Delivery

Serves: R-GITOPS-01..03, R-PLAT-02/04, R-ENV-02. Related: ADR-0004 (Flux).

Model

Pull-based GitOps with Flux CD. The cluster continuously reconciles its state from git. Git is the single source of truth; the cluster conforms to git, not the reverse (R-GITOPS-01). Drift is detected and corrected automatically (R-GITOPS-03).

Repository layout (the implementation repo, created in a later phase)

This spec repo (home-network-k3s) holds intent. The implementation repo(s) hold the reconciled manifests. Proposed structure:

platform/                 ← cluster-wide platform services (Flux-managed)
  cilium/                 ← CNI, egress gateway, network baseline
  longhorn/               ← storage
  cert-manager/           ← TLS issuers + ClusterIssuer
  externaldns/            ← DNS automation
  gateway/                ← Gateway API + listeners/TLS
  monitoring/             ← kube-prometheus-stack, Loki, Alloy, Alertmanager
  velero/                 ← backup/DR
  kubevirt/               ← KubeVirt + CDI
  image-automation/       ← Flux image update automation config
apps/                     ← application workloads, one dir per service
  plex/  sonarr/  radarr/  jackett/  seerr/  pihole/  nextcloud/
  minecraft/  unifi/  semaphore/  kasm/  gitea/  qbittorrent/
catalog/                  ← ServiceDefinition entries (the onboarding surface)
clusters/
  dev/                    ← Flux entrypoint + dev overlay/vars
  prod/                   ← Flux entrypoint + prod overlay/vars (design-only)

• Kustomize overlays (or equivalent) express environment differences (R-ENV-02). Base is shared; clusters/dev and clusters/prod supply scoped values only.
• HelmReleases (Flux) wrap upstream charts (Cilium, Longhorn, cert-manager, kube-prometheus- stack, etc.); app workloads use charts where good ones exist, otherwise plain manifests.

The onboarding surface: ServiceDefinition (R-GITOPS-02)

Onboarding a service = adding one ServiceDefinition under catalog/. It is the operator-facing abstraction that the platform compiles into the concrete resources (Deployment/VM, PVCs, NetworkPolicies, HTTPRoute, Certificate, DNS record, ServiceMonitor, update policy). See 13-service-catalog.md for the schema. Intent lives in one place; the platform renders the rest.

Reconciliation flow

1. Operator opens a PR against the implementation repo (new/updated ServiceDefinition or platform change).
2. CI validates (lint, policy checks, schema validation, dry-run/build).
3. Merge to the tracked branch.
4. Flux reconciles: applies manifests, reports status, corrects drift on an interval.
5. Health/rollout status surfaces in the C2 UI (12) and alerts on failure (10).

Secrets in GitOps

Secrets are committed encrypted with SOPS + age; Flux decrypts them in-cluster at reconciliation. No plaintext secret ever lands in git. See 07-security-and-secrets.md and ADR-0007.

Automated updates interplay (R-UPD-*)

GitOps and hands-off updates coexist:

• Container image tags advance automatically via Flux image-update-automation driven by per-service image policies (semver range / stable channel) — the controller resolves and reconciles the new version with no human-authored change (R-UPD-02).
• Helm chart / manifest version bumps are proposed by Renovate with auto-merge for allowed update classes.

Full detail and the "is an automated commit a GitOps change?" nuance is in 11-automated-updates.md and ADR-0009.

Bootstrap & recovery

• Flux is bootstrapped onto a fresh cluster by the Ansible handoff (02).
• Because desired state is in git and data is on Longhorn/backups, a rebuilt cluster reconverges by re-bootstrapping Flux and restoring volumes (04).

Source-of-truth: local bootstrap → self-hosted Gitea

The platform hosts its own git, which creates a deliberate bootstrapping sequence (../CONTRIBUTING.md):

1. Bootstrap source. Initially there is no remote git on this network (the prior home-network Gitea is gone from here). Development is local; the cluster is first reconciled from a bootstrap source (this local repo / an out-of-band source).
2. Gitea comes online early. Gitea is one of the first services onboarded (13) so the platform gains its own git host at gitea.home.keithjackson.dev.
3. Cut Flux over to Gitea. Once Gitea is stable, Flux's GitRepository source is migrated from the bootstrap source to Gitea; the platform then reconciles from its own git.
4. CI/CD moves to Gitea Actions. The Kyverno L0 policy gate and the trigger for the in-cluster Chainsaw conformance Job (14) run as Gitea Actions pipelines on push/PR; PRs become the enforced merge gate into main.

Circularity & DR. Gitea runs on the cluster Flux reconciles from Gitea. Mitigations:

• Keep an off-cluster mirror of the platform repo so a source-of-truth exists that outlives the cluster.
• Treat Gitea as DR-critical: its repos/DB are backed up off-cluster (04), so a cluster loss never loses the git history needed to rebuild the platform.

Guardrails

• No kubectl apply of app state by hand — everything flows through git/Flux (R-GITOPS-01, click- ops-never per AGENTS.md).
• Reconciliation errors and drift are alerting conditions (R-OBS-04).
• Base config is never forked per environment (R-ENV-02).

Acceptance gates

• Adding a ServiceDefinition PR results in a fully wired service (compute, storage, policy, TLS, DNS, monitoring) with no manual steps beyond the merge.
• Manually mutating a live resource is auto-reverted by Flux (drift correction proven).
• The same base config renders correctly for both dev and prod overlays.