jakman85andClaude Opus 5.5
250fb54154
The operator ruled the implementation repo's main (50c8719) the latest and accepted state. Sync the spec to it. Accepted changes are reflected: one bare-metal environment, TrueNAS bulk storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway with Cilium fail-closed, inbound WireGuard, per-app auth sidecars, Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025; ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended. Where the implementation has simply not met a requirement, the requirement stands and the shortfall is recorded in docs/16-known-gaps.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
5.4 KiB
5.4 KiB
11 — Automated Updates
Serves: R-UPD-01/02/03. Related: ADR-0023.
The requirement, precisely
Services must update to latest stable versions on a schedule; updates in the allowed classes occur with no human, no AI, and no hand-authored GitOps change. (R-UPD-01/02)
This has a subtle tension with GitOps (where "the desired state is what's in git"). We resolve it explicitly below.
Resolving "no GitOps change" vs. GitOps
There are two interpretations; we honor the operator's intent (hands-off), and ADR-0023 records the mechanism:
- A human/AI authoring a version bump = prohibited. No person and no chatbot edits a tag. Reviewing a machine-authored change is not authoring one.
- An automation bot proposing a version advance under a policy declared once = permitted,
because the operator's declared intent (the update policy / version range) is the stable,
human-authored thing; the bot merely resolves it, and its change reaches
mainthrough the same merge gate as every other change (05). Mechanisms by artifact type:
1. Container images, Helm charts and manifests → Renovate
- Renovate runs on a schedule, watches image tags, chart and dependency versions, and opens PRs. Git history records every advance as an auditable, machine-authored commit.
- Minor, patch and digest updates auto-merge on a green gate. This keeps services and platform components current with no human authoring.
- Major updates never auto-merge; they wait for human review of the machine-authored PR.
- Operator-designated blast-radius components never auto-merge, not even a patch: the storage and
network planes, the reboot daemon, the database engine, the metrics stack, Gitea and its runner
(they are the merge gate), the identity provider and its auth sidecars, and Renovate itself and
the pinned CI toolchain. The list is data in the implementation repo's
renovate.json5, each entry with its reason. - Because
apps/is rendered fromcatalog/, a bump updates the source and its rendered output in one PR, so render-equals-committed holds.
2. The platform's own images → Renovate digest tracking
- Images built by CI (05) are referenced by digest. Renovate follows each image's moving tag and proposes the digest bump as a PR, whose own gate run exercises it where the image is the toolbox.
3. VM guests → in-guest automation
- KubeVirt guests auto-update OS packages inside the guest on a schedule (unattended-upgrades / dnf- automatic) declared in the VM's cloud-init (06). Base-image promotion follows the same automated philosophy where a golden-image pipeline exists.
4. Node OS → security-only upgrades, coordinated reboots
- Nodes apply security-only unattended upgrades, with kernels excluded. A reboot daemon performs only the mechanical half of a reboot, when a node requests one: one node at a time under a single cluster-wide lock, gated on storage health so etcd quorum and volume replicas are never put at risk. A hang stops the cycle and pages a human. Kernel upgrades stay a deliberate operator act.
Per-service policy (R-UPD-03)
Each service's update policy is declared in git:
- channel/range: which versions are "stable" and acceptable (semver range, tag regex).
- cadence: how often to check/apply (the schedule).
- hold/pin: temporarily freeze a service (e.g. a known-bad upstream release).
- rollout & health: how the new version rolls out and what health gates it.
- auto-rollback: revert on failed health so a bad upstream release self-heals.
The ServiceDefinition (13) records that intent. Renovate does not read it, so the policy takes effect through per-package rules in the Renovate configuration, which mirrors each declared ceiling and carries any per-service cadence.
Safety rails (so hands-off ≠ reckless)
- Stable only: never track pre-release/edge tags unless explicitly opted in per service.
- Merge gate first: every update is a PR that must pass the gate before it can merge.
- Health-gated rollout: a failing update is caught by health checks and rolled back (R-UPD-03), and raises an alert (10).
- Backups precede risk: state is on Longhorn with backups, so a bad update never risks data (04).
- Observability: update events (update proposed, merged, rollback) are visible and, on failure, alert by email.
- Pin escape hatch: an operator can pin a service via git if an upstream release misbehaves — the one sanctioned human touch, and it stops automation rather than performing an update.
Interaction with default-deny & egress
Updates change image versions, not network posture. A VPN-class service stays VPN-class across updates; policies are reconciled from the (unchanged) ServiceDefinition.
Acceptance gates
- A service advances to a newer stable version automatically, on schedule, with no human/AI edit.
- A deliberately bad update is caught by health checks, auto-rolled-back, and alerts by email.
- A pinned service does not advance.
- Components in the allowed classes stay current via auto-merged Renovate PRs; majors and blast-radius components arrive as machine-authored PRs awaiting review.