jakman85andClaude Opus 5.5
18f7ba8d0f
Retire Kasm from the catalog. Drop ACME revocation from R-TLS-02. Make L3 fault injection operator-run and exclude it from the merge gate (R-TEST-03, ADR-0026). Update the gap register to match; Homepage authentication is now ruled must-enforce and stays a gap until the implementation lands. Co-Authored-By: Claude Opus 5.5 <[email protected]>
176 lines
11 KiB
Markdown
176 lines
11 KiB
Markdown
# 05 — GitOps & Delivery
|
|
|
|
Serves: **R-GITOPS-01..03**, **R-PLAT-02/04**, **R-ENV-02**, **R-TEST-03**. Related:
|
|
[ADR-0004 (Flux)](../decisions/adr-0004-gitops-flux.md),
|
|
[ADR-0023 (updates)](../decisions/adr-0023-automated-updates-renovate.md),
|
|
[ADR-0024 (delivery authority)](../decisions/adr-0024-delivery-authority-ci-gate.md),
|
|
[ADR-0025 (pm-agent)](../decisions/adr-0025-in-cluster-ai-program-manager.md).
|
|
|
|
## Model
|
|
|
|
Pull-based GitOps with **Flux CD**. The cluster continuously reconciles its state from git. Git is
|
|
the single source of truth; the cluster conforms to git, not the reverse (R-GITOPS-01). Drift is
|
|
detected and corrected automatically (R-GITOPS-03).
|
|
|
|
## Repository layout (the implementation repo)
|
|
|
|
> This spec repo (`home-network-k3s`) holds intent. The
|
|
> [implementation repo](https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl)
|
|
> holds the reconciled manifests, the gate and the tests. Its top-level areas:
|
|
|
|
```
|
|
catalog/ ← ServiceDefinition entries (the onboarding surface) + the renderer
|
|
schemas/ ← ServiceDefinition JSON Schema
|
|
apps/ ← application workloads by reconcile wave, plus VMs; rendered from
|
|
catalog/ and committed (the render is checked against them)
|
|
platform/ ← cluster-wide platform services (Flux-managed): CNI, storage, TLS, DNS,
|
|
Gateway, public tunnel, observability, backup/DR, KubeVirt, identity,
|
|
admission policy, image builder, delivery automation
|
|
platform-config*/ ← platform configuration Flux applies after the platform tier, including
|
|
the admission policies
|
|
clusters/
|
|
base/ ← shared Flux entrypoints and reconcile ordering
|
|
prod/ ← the live cluster's overlay and environment values
|
|
bootstrap/ ← Flux, SOPS and Gitea bootstrap
|
|
infra/ ← node provisioning as code; Gitea settings; dev toolbox; agent image
|
|
ci/ .gitea/ ← the merge gate and the workflows that run it
|
|
conformance/ ← requirement registry, behavior tests, harness, run ledger
|
|
runbooks/ ← operator procedures (also served to the alert links)
|
|
decisions/ ← dated implementation handoff log (distinct from this repo's ADRs)
|
|
```
|
|
|
|
- **Kustomize overlays** express environment differences (R-ENV-02). The base is shared; an
|
|
environment overlay supplies scoped values only. `prod` is the one live environment; the dev
|
|
environment was decommissioned in 2026-09 ([ADR-0017](../decisions/adr-0017-single-environment-bare-metal-production.md)).
|
|
- **HelmReleases** (Flux) wrap upstream charts (Cilium, Longhorn, cert-manager, kube-prometheus-
|
|
stack, etc.); app workloads use charts where good ones exist, otherwise plain manifests.
|
|
- **Rendered output is committed.** `apps/` is generated from `catalog/`; the merge gate fails when
|
|
the two disagree, so a ServiceDefinition is always the source of a rendered workload.
|
|
|
|
## The onboarding surface: ServiceDefinition (R-GITOPS-02)
|
|
|
|
Onboarding a service = adding one **ServiceDefinition** under `catalog/`. It is the operator-facing
|
|
abstraction that the platform compiles into the concrete resources (Deployment/VM, PVCs,
|
|
NetworkPolicies, HTTPRoute, Certificate, DNS record, ServiceMonitor, update policy). See
|
|
[`13-service-catalog.md`](13-service-catalog.md) for the schema. Intent lives in one place; the
|
|
platform renders the rest.
|
|
|
|
## Reconciliation flow
|
|
|
|
1. A contributor opens a PR against the implementation repo (new/updated ServiceDefinition or
|
|
platform change). Contributors are the operator, implementation agents, Renovate and pm-agent;
|
|
all of them take the same path.
|
|
2. The **merge gate** validates the change (below).
|
|
3. Merge to `main`, the tracked branch.
|
|
4. Flux reconciles from the platform's own Gitea: applies manifests, reports status, corrects drift
|
|
on an interval.
|
|
5. Health/rollout status surfaces in the C2 UI ([12](12-command-center-ui.md)) and alerts on failure
|
|
([10](10-observability-and-alerting.md)).
|
|
|
|
Nothing reaches `main` by any other route: direct pushes are rejected for every account.
|
|
|
|
## The merge gate (R-TEST-03)
|
|
|
|
The implementation repo's `main` is protected by one required status check, the merge gate, which
|
|
runs on every PR as a Gitea Actions job. **A red gate is unmergeable by everyone, administrators
|
|
included.** The gate runs in the same pinned toolbox image locally and in CI (R-TEST-02) and
|
|
establishes, in order: the adopted spec pin (R-TEST-06), the handoff log's format, lint, L0 schema +
|
|
policy + repository invariants, render-equals-committed, L1 rendering, the sensitive-path approval
|
|
policy, that the live Gitea settings match their declaration, and L2 conformance
|
|
([14](14-testing-and-conformance.md); see [known gaps](16-known-gaps.md)).
|
|
|
|
Its verdict is one of three: **pass**, **blocked**, or **undetermined**. A stage that did not run is
|
|
never green, and no automation may read a non-answer as approval. The rule is that a capability the
|
|
environment *declares* absent is named in the verdict, while a stage that *could not run where it
|
|
must* blocks. Detail: `ci/README.md` in the implementation repo.
|
|
|
|
## Who approves ([ADR-0024](../decisions/adr-0024-delivery-authority-ci-gate.md))
|
|
|
|
**The CI gate is the approver.** A green gate is sufficient for an ordinary change; branch
|
|
protection requires no approval count. A weak gate is a defect to fix, not a reason to add a human.
|
|
|
|
**One exception: a change to the gate cannot be gated by the gate.** A change touching the pipeline,
|
|
the tests and harness, the Flux wiring, the admission policies, the SOPS recipients and secrets, the
|
|
RBAC, or the delivery contract needs an approving review from someone other than its author. The
|
|
path list is data, with a stated reason per entry (`infra/gitea/approval-policy.json` in the
|
|
implementation repo), and is itself a sensitive path. This stage is fail-closed: if it cannot
|
|
evaluate, it blocks.
|
|
|
|
**Attributed administrator override.** There is one human on this platform, and Gitea refuses
|
|
self-approval, so an administrator may waive that review with an explicit, reasoned directive on the
|
|
PR. Authority is re-checked live, a new commit invalidates it, and the overrider, time and reason
|
|
are recorded in the gate output. For an administrator the exception above is therefore advisory, not
|
|
structural; what it guarantees is an attributed record of each waiver. **Agents never post this
|
|
directive to clear their own work**, whatever permission their account holds.
|
|
|
|
## Secrets in GitOps
|
|
|
|
Secrets are committed **encrypted** with **SOPS + age**; Flux decrypts them in-cluster at
|
|
reconciliation. No plaintext secret ever lands in git. See [`07-security-and-secrets.md`](07-security-and-secrets.md)
|
|
and [ADR-0007](../decisions/adr-0007-secrets-sops-age.md).
|
|
|
|
## Automated updates interplay (R-UPD-*)
|
|
|
|
GitOps and hands-off updates coexist because the update bot is one more contributor to the same
|
|
flow. **Renovate** runs on a schedule and proposes machine-authored PRs for container images, Helm
|
|
charts and manifest versions; each passes the merge gate, and allowed update classes auto-merge on
|
|
green (R-UPD-02). No in-cluster controller holds a credential to write to the repo that defines the
|
|
cluster. Full detail is in [`11-automated-updates.md`](11-automated-updates.md) and
|
|
[ADR-0023](../decisions/adr-0023-automated-updates-renovate.md).
|
|
|
|
## Images are built by CI
|
|
|
|
Container images the platform itself produces (the toolbox, the agent runner, the conformance
|
|
runner) are built, verified and published only by the pipeline, never from a workstation. A PR
|
|
builds and verifies; a push to `main` publishes to the Gitea registry. References are pinned by
|
|
digest, and Renovate carries each new digest into git as a PR. **No CI job may reach a privileged
|
|
container daemon**: builds go to a rootless builder over mutually-authenticated TLS.
|
|
|
|
## Delivery automation: pm-agent ([ADR-0025](../decisions/adr-0025-in-cluster-ai-program-manager.md))
|
|
|
|
An in-cluster AI Program Manager reads labelled issues on the implementation repo, decides what
|
|
should happen about each thread, and runs one worker agent per assignment to author a change, run
|
|
the merge gate and open a PR. It is delivery tooling rather than a catalog service: it declares
|
|
Kubernetes objects a ServiceDefinition cannot express, and it requires no `R-*` of its own.
|
|
|
|
It is contributor-shaped, never approver-shaped. It cannot push to `main`, it never merges, its
|
|
account is not an administrator, and its workers hold no cluster credential. It is an implementation
|
|
agent bound by [AGENTS.md](../AGENTS.md) and R-TEST-06: spec work is out of its reach and is
|
|
declined and escalated.
|
|
|
|
## Bootstrap & recovery
|
|
|
|
- Flux is bootstrapped onto a fresh cluster by the Ansible handoff ([02](02-environments-and-provisioning.md)).
|
|
- Because desired state is in git and data is on Longhorn/backups, a rebuilt cluster reconverges by
|
|
re-bootstrapping Flux and restoring volumes ([04](04-storage-and-state.md)).
|
|
|
|
## Source-of-truth: self-hosted Gitea
|
|
|
|
The platform hosts its own git at `gitea.home.keithjackson.dev`
|
|
([`../CONTRIBUTING.md`](../CONTRIBUTING.md)); Flux reconciles from it and Gitea Actions runs the
|
|
merge gate. Standing the platform up from nothing follows a deliberate order:
|
|
|
|
1. **Bootstrap source.** The cluster is first reconciled from an out-of-band copy of the repo.
|
|
2. **Gitea comes online early.** Gitea is one of the first services onboarded
|
|
([13](13-service-catalog.md)).
|
|
3. **Cut Flux over to Gitea**, so the platform reconciles from its own git.
|
|
4. **CI/CD runs in Gitea Actions**, with PRs as the enforced merge gate into `main`.
|
|
|
|
**Circularity & DR.** Gitea runs *on* the cluster Flux reconciles *from* Gitea. Mitigations:
|
|
- Keep an **off-cluster mirror** of the platform repo so a source-of-truth exists that outlives the
|
|
cluster (see [known gaps](16-known-gaps.md)).
|
|
- Treat **Gitea as DR-critical**: its repos/DB are backed up off-cluster ([04](04-storage-and-state.md)),
|
|
so a cluster loss never loses the git history needed to rebuild the platform.
|
|
|
|
## Guardrails
|
|
- No `kubectl apply` of app state by hand — everything flows through git/Flux (R-GITOPS-01, click-
|
|
ops-never per [AGENTS.md](../AGENTS.md)).
|
|
- Reconciliation errors and drift are **alerting conditions** (R-OBS-04).
|
|
- Base config is never forked per environment (R-ENV-02).
|
|
|
|
## Acceptance gates
|
|
- Adding a ServiceDefinition PR results in a fully wired service (compute, storage, policy, TLS, DNS,
|
|
monitoring) with no manual steps beyond the merge.
|
|
- Manually mutating a live resource is auto-reverted by Flux (drift correction proven).
|
|
- The base config renders correctly under the environment overlay, which supplies scoped values only.
|