docs: bring the specification in line with the accepted implementation

The operator ruled the implementation repo's main (50c8719) the latest
and accepted state. Sync the spec to it.

Accepted changes are reflected: one bare-metal environment, TrueNAS bulk
storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway
with Cilium fail-closed, inbound WireGuard, per-app auth sidecars,
Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025;
ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended.

Where the implementation has simply not met a requirement, the
requirement stands and the shortfall is recorded in docs/16-known-gaps.md.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5.5 committed 2026-11-08 22:46:00 -06:00
1 parent 1ddc14b918
commit 250fb54154
43 files changed
+1672 -575

No files matched your search

+42 -28
View File
@@ -1,62 +1,80 @@
# 11 — Automated Updates
Serves: **R-UPD-01/02/03**. Related: [ADR-0009](../decisions/adr-0009-automated-image-updates.md).
Serves: **R-UPD-01/02/03**. Related: [ADR-0023](../decisions/adr-0023-automated-updates-renovate.md).
## The requirement, precisely
> Services must update to latest **stable** versions **on a schedule**, with **no human, no AI, and
> no hand-authored GitOps change**. (R-UPD-01/02)
> Services must update to latest **stable** versions **on a schedule**; updates in the allowed
> classes occur with **no human, no AI, and no hand-authored GitOps change**. (R-UPD-01/02)
This has a subtle tension with GitOps (where "the desired state is what's in git"). We resolve it
explicitly below.
## Resolving "no GitOps change" vs. GitOps
There are two interpretations; we honor the operator's intent (hands-off), and ADR-0009 records the
There are two interpretations; we honor the operator's intent (hands-off), and ADR-0023 records the
mechanism:
- **A human/AI authoring a version bump = prohibited.** No person and no chatbot edits a tag.
- **An automation controller advancing a version under a policy declared once = permitted**, because
the *operator's* declared intent (the update policy / semver range) is the stable, human-authored
thing; the controller merely resolves it. Two mechanisms, chosen by artifact type:
Reviewing a machine-authored change is not authoring one.
- **An automation bot proposing a version advance under a policy declared once = permitted**,
because the *operator's* declared intent (the update policy / version range) is the stable,
human-authored thing; the bot merely resolves it, and its change reaches `main` through the same
merge gate as every other change ([05](05-gitops-and-delivery.md)). Mechanisms by artifact type:
### 1. Container image tags → Flux image-update-automation
- Per service, an **ImagePolicy** declares the acceptable range/channel (e.g. semver `>=1.0.0`
minor/patch, or a `stable` tag pattern). Flux's image-reflector scans the registry; image-
automation advances the running version automatically **on its scan interval** (the schedule).
- Where automation writes the resolved version back to git, that commit is machine-authored under
the declared policy — no human/AI touches it (satisfies R-UPD-02 in spirit and practice).
- Alternatively, digest-tracking of a mutable stable tag avoids per-update commits entirely. ADR-0009
picks the default and the per-service escape hatch.
### 1. Container images, Helm charts and manifests → Renovate
- **Renovate** runs on a schedule, watches image tags, chart and dependency versions, and opens PRs.
Git history records every advance as an auditable, machine-authored commit.
- **Minor, patch and digest** updates **auto-merge** on a green gate. This keeps services and
platform components current with no human authoring.
- **Major** updates never auto-merge; they wait for human review of the machine-authored PR.
- **Operator-designated blast-radius components** never auto-merge, not even a patch: the storage and
network planes, the reboot daemon, the database engine, the metrics stack, Gitea and its runner
(they *are* the merge gate), the identity provider and its auth sidecars, and Renovate itself and
the pinned CI toolchain. The list is data in the implementation repo's `renovate.json5`, each entry
with its reason.
- Because `apps/` is rendered from `catalog/`, a bump updates the source and its rendered output in
one PR, so render-equals-committed holds.
### 2. Helm charts / manifests → Renovate
- **Renovate** watches chart and dependency versions and opens PRs; allowed update classes
**auto-merge** on green CI. This keeps platform components current with no human authoring.
- Chart *major* bumps (potentially breaking) can require review — configurable per policy.
### 2. The platform's own images → Renovate digest tracking
- Images built by CI ([05](05-gitops-and-delivery.md)) are referenced by digest. Renovate follows
each image's moving tag and proposes the digest bump as a PR, whose own gate run exercises it where
the image is the toolbox.
### 3. VM guests → in-guest automation
- KubeVirt guests auto-update OS packages inside the guest on a schedule (unattended-upgrades / dnf-
automatic) declared in the VM's cloud-init ([06](06-vm-workloads-kubevirt.md)). Base-image
promotion follows the same automated philosophy where a golden-image pipeline exists.
### 4. Node OS → security-only upgrades, coordinated reboots
- Nodes apply **security-only** unattended upgrades, with kernels excluded. A reboot daemon performs
only the mechanical half of a reboot, when a node requests one: one node at a time under a single
cluster-wide lock, gated on storage health so etcd quorum and volume replicas are never put at
risk. A hang stops the cycle and pages a human. Kernel upgrades stay a deliberate operator act.
## Per-service policy (R-UPD-03)
Each ServiceDefinition ([13](13-service-catalog.md)) declares an **update policy**:
Each service's **update policy** is declared in git:
- **channel/range**: which versions are "stable" and acceptable (semver range, tag regex).
- **cadence**: how often to check/apply (the schedule).
- **hold/pin**: temporarily freeze a service (e.g. a known-bad upstream release).
- **rollout & health**: how the new version rolls out and what health gates it.
- **auto-rollback**: revert on failed health so a bad upstream release self-heals.
The ServiceDefinition ([13](13-service-catalog.md)) records that intent. Renovate does not read it,
so the policy takes effect through per-package rules in the Renovate configuration, which mirrors
each declared ceiling and carries any per-service cadence.
## Safety rails (so hands-off ≠ reckless)
- **Stable only**: never track pre-release/edge tags unless explicitly opted in per service.
- **Merge gate first**: every update is a PR that must pass the gate before it can merge.
- **Health-gated rollout**: a failing update is caught by health checks and rolled back (R-UPD-03),
and raises an alert ([10](10-observability-and-alerting.md)).
- **Backups precede risk**: state is on Longhorn with backups, so a bad update never risks data
([04](04-storage-and-state.md)).
- **Observability**: update events (image advanced, chart merged, rollback) are visible and, on
failure, alert by email.
- **Observability**: update events (update proposed, merged, rollback) are visible and, on failure,
alert by email.
- **Pin escape hatch**: an operator can pin a service via git if an upstream release misbehaves —
the one sanctioned human touch, and it *stops* automation rather than performing an update.
@@ -68,9 +86,5 @@ updates; policies are reconciled from the (unchanged) ServiceDefinition.
- A service advances to a newer stable version automatically, on schedule, with no human/AI edit.
- A deliberately bad update is caught by health checks, auto-rolled-back, and alerts by email.
- A pinned service does **not** advance.
- Platform components (charts) stay current via auto-merged Renovate PRs.
## Open questions
- Exact default in ADR-0009: write-back commits vs. digest-tracking mutable tags (trade readable
history vs. zero-commit updates).
- Registry rate-limit / credential handling for image scanning at cadence.
- Components in the allowed classes stay current via auto-merged Renovate PRs; majors and
blast-radius components arrive as machine-authored PRs awaiting review.