docs: bring the specification in line with the accepted implementation
The operator ruled the implementation repo's main (50c8719) the latest and accepted state. Sync the spec to it. Accepted changes are reflected: one bare-metal environment, TrueNAS bulk storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway with Cilium fail-closed, inbound WireGuard, per-app auth sidecars, Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025; ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended. Where the implementation has simply not met a requirement, the requirement stands and the shortfall is recorded in docs/16-known-gaps.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
1 parent
1ddc14b918
commit
250fb54154
43 files changed
+1672
-575
No files matched your search
@@ -1,62 +1,80 @@
|
||||
# 11 — Automated Updates
|
||||
|
||||
Serves: **R-UPD-01/02/03**. Related: [ADR-0009](../decisions/adr-0009-automated-image-updates.md).
|
||||
Serves: **R-UPD-01/02/03**. Related: [ADR-0023](../decisions/adr-0023-automated-updates-renovate.md).
|
||||
|
||||
## The requirement, precisely
|
||||
|
||||
> Services must update to latest **stable** versions **on a schedule**, with **no human, no AI, and
|
||||
> no hand-authored GitOps change**. (R-UPD-01/02)
|
||||
> Services must update to latest **stable** versions **on a schedule**; updates in the allowed
|
||||
> classes occur with **no human, no AI, and no hand-authored GitOps change**. (R-UPD-01/02)
|
||||
|
||||
This has a subtle tension with GitOps (where "the desired state is what's in git"). We resolve it
|
||||
explicitly below.
|
||||
|
||||
## Resolving "no GitOps change" vs. GitOps
|
||||
|
||||
There are two interpretations; we honor the operator's intent (hands-off), and ADR-0009 records the
|
||||
There are two interpretations; we honor the operator's intent (hands-off), and ADR-0023 records the
|
||||
mechanism:
|
||||
|
||||
- **A human/AI authoring a version bump = prohibited.** No person and no chatbot edits a tag.
|
||||
- **An automation controller advancing a version under a policy declared once = permitted**, because
|
||||
the *operator's* declared intent (the update policy / semver range) is the stable, human-authored
|
||||
thing; the controller merely resolves it. Two mechanisms, chosen by artifact type:
|
||||
Reviewing a machine-authored change is not authoring one.
|
||||
- **An automation bot proposing a version advance under a policy declared once = permitted**,
|
||||
because the *operator's* declared intent (the update policy / version range) is the stable,
|
||||
human-authored thing; the bot merely resolves it, and its change reaches `main` through the same
|
||||
merge gate as every other change ([05](05-gitops-and-delivery.md)). Mechanisms by artifact type:
|
||||
|
||||
### 1. Container image tags → Flux image-update-automation
|
||||
- Per service, an **ImagePolicy** declares the acceptable range/channel (e.g. semver `>=1.0.0`
|
||||
minor/patch, or a `stable` tag pattern). Flux's image-reflector scans the registry; image-
|
||||
automation advances the running version automatically **on its scan interval** (the schedule).
|
||||
- Where automation writes the resolved version back to git, that commit is machine-authored under
|
||||
the declared policy — no human/AI touches it (satisfies R-UPD-02 in spirit and practice).
|
||||
- Alternatively, digest-tracking of a mutable stable tag avoids per-update commits entirely. ADR-0009
|
||||
picks the default and the per-service escape hatch.
|
||||
### 1. Container images, Helm charts and manifests → Renovate
|
||||
- **Renovate** runs on a schedule, watches image tags, chart and dependency versions, and opens PRs.
|
||||
Git history records every advance as an auditable, machine-authored commit.
|
||||
- **Minor, patch and digest** updates **auto-merge** on a green gate. This keeps services and
|
||||
platform components current with no human authoring.
|
||||
- **Major** updates never auto-merge; they wait for human review of the machine-authored PR.
|
||||
- **Operator-designated blast-radius components** never auto-merge, not even a patch: the storage and
|
||||
network planes, the reboot daemon, the database engine, the metrics stack, Gitea and its runner
|
||||
(they *are* the merge gate), the identity provider and its auth sidecars, and Renovate itself and
|
||||
the pinned CI toolchain. The list is data in the implementation repo's `renovate.json5`, each entry
|
||||
with its reason.
|
||||
- Because `apps/` is rendered from `catalog/`, a bump updates the source and its rendered output in
|
||||
one PR, so render-equals-committed holds.
|
||||
|
||||
### 2. Helm charts / manifests → Renovate
|
||||
- **Renovate** watches chart and dependency versions and opens PRs; allowed update classes
|
||||
**auto-merge** on green CI. This keeps platform components current with no human authoring.
|
||||
- Chart *major* bumps (potentially breaking) can require review — configurable per policy.
|
||||
### 2. The platform's own images → Renovate digest tracking
|
||||
- Images built by CI ([05](05-gitops-and-delivery.md)) are referenced by digest. Renovate follows
|
||||
each image's moving tag and proposes the digest bump as a PR, whose own gate run exercises it where
|
||||
the image is the toolbox.
|
||||
|
||||
### 3. VM guests → in-guest automation
|
||||
- KubeVirt guests auto-update OS packages inside the guest on a schedule (unattended-upgrades / dnf-
|
||||
automatic) declared in the VM's cloud-init ([06](06-vm-workloads-kubevirt.md)). Base-image
|
||||
promotion follows the same automated philosophy where a golden-image pipeline exists.
|
||||
|
||||
### 4. Node OS → security-only upgrades, coordinated reboots
|
||||
- Nodes apply **security-only** unattended upgrades, with kernels excluded. A reboot daemon performs
|
||||
only the mechanical half of a reboot, when a node requests one: one node at a time under a single
|
||||
cluster-wide lock, gated on storage health so etcd quorum and volume replicas are never put at
|
||||
risk. A hang stops the cycle and pages a human. Kernel upgrades stay a deliberate operator act.
|
||||
|
||||
## Per-service policy (R-UPD-03)
|
||||
|
||||
Each ServiceDefinition ([13](13-service-catalog.md)) declares an **update policy**:
|
||||
Each service's **update policy** is declared in git:
|
||||
- **channel/range**: which versions are "stable" and acceptable (semver range, tag regex).
|
||||
- **cadence**: how often to check/apply (the schedule).
|
||||
- **hold/pin**: temporarily freeze a service (e.g. a known-bad upstream release).
|
||||
- **rollout & health**: how the new version rolls out and what health gates it.
|
||||
- **auto-rollback**: revert on failed health so a bad upstream release self-heals.
|
||||
|
||||
The ServiceDefinition ([13](13-service-catalog.md)) records that intent. Renovate does not read it,
|
||||
so the policy takes effect through per-package rules in the Renovate configuration, which mirrors
|
||||
each declared ceiling and carries any per-service cadence.
|
||||
|
||||
## Safety rails (so hands-off ≠ reckless)
|
||||
|
||||
- **Stable only**: never track pre-release/edge tags unless explicitly opted in per service.
|
||||
- **Merge gate first**: every update is a PR that must pass the gate before it can merge.
|
||||
- **Health-gated rollout**: a failing update is caught by health checks and rolled back (R-UPD-03),
|
||||
and raises an alert ([10](10-observability-and-alerting.md)).
|
||||
- **Backups precede risk**: state is on Longhorn with backups, so a bad update never risks data
|
||||
([04](04-storage-and-state.md)).
|
||||
- **Observability**: update events (image advanced, chart merged, rollback) are visible and, on
|
||||
failure, alert by email.
|
||||
- **Observability**: update events (update proposed, merged, rollback) are visible and, on failure,
|
||||
alert by email.
|
||||
- **Pin escape hatch**: an operator can pin a service via git if an upstream release misbehaves —
|
||||
the one sanctioned human touch, and it *stops* automation rather than performing an update.
|
||||
|
||||
@@ -68,9 +86,5 @@ updates; policies are reconciled from the (unchanged) ServiceDefinition.
|
||||
- A service advances to a newer stable version automatically, on schedule, with no human/AI edit.
|
||||
- A deliberately bad update is caught by health checks, auto-rolled-back, and alerts by email.
|
||||
- A pinned service does **not** advance.
|
||||
- Platform components (charts) stay current via auto-merged Renovate PRs.
|
||||
|
||||
## Open questions
|
||||
- Exact default in ADR-0009: write-back commits vs. digest-tracking mutable tags (trade readable
|
||||
history vs. zero-commit updates).
|
||||
- Registry rate-limit / credential handling for image scanning at cadence.
|
||||
- Components in the allowed classes stay current via auto-merged Renovate PRs; majors and
|
||||
blast-radius components arrive as machine-authored PRs awaiting review.
|
||||
Reference in new issue
Block a user