home-network-k3s/docs/11-automated-updates.md

11 — Automated Updates

Serves: R-UPD-01/02/03. Related: ADR-0009.

The requirement, precisely

Services must update to latest stable versions on a schedule, with no human, no AI, and no hand-authored GitOps change. (R-UPD-01/02)

This has a subtle tension with GitOps (where "the desired state is what's in git"). We resolve it explicitly below.

Resolving "no GitOps change" vs. GitOps

There are two interpretations; we honor the operator's intent (hands-off), and ADR-0009 records the mechanism:

• A human/AI authoring a version bump = prohibited. No person and no chatbot edits a tag.
• An automation controller advancing a version under a policy declared once = permitted, because the operator's declared intent (the update policy / semver range) is the stable, human-authored thing; the controller merely resolves it. Two mechanisms, chosen by artifact type:

1. Container image tags → Flux image-update-automation

• Per service, an ImagePolicy declares the acceptable range/channel (e.g. semver >=1.0.0 minor/patch, or a stable tag pattern). Flux's image-reflector scans the registry; image- automation advances the running version automatically on its scan interval (the schedule).
• Where automation writes the resolved version back to git, that commit is machine-authored under the declared policy — no human/AI touches it (satisfies R-UPD-02 in spirit and practice).
• Alternatively, digest-tracking of a mutable stable tag avoids per-update commits entirely. ADR-0009 picks the default and the per-service escape hatch.

2. Helm charts / manifests → Renovate

• Renovate watches chart and dependency versions and opens PRs; allowed update classes auto-merge on green CI. This keeps platform components current with no human authoring.
• Chart major bumps (potentially breaking) can require review — configurable per policy.

3. VM guests → in-guest automation

• KubeVirt guests auto-update OS packages inside the guest on a schedule (unattended-upgrades / dnf- automatic) declared in the VM's cloud-init (06). Base-image promotion follows the same automated philosophy where a golden-image pipeline exists.

Per-service policy (R-UPD-03)

Each ServiceDefinition (13) declares an update policy:

• channel/range: which versions are "stable" and acceptable (semver range, tag regex).
• cadence: how often to check/apply (the schedule).
• hold/pin: temporarily freeze a service (e.g. a known-bad upstream release).
• rollout & health: how the new version rolls out and what health gates it.
• auto-rollback: revert on failed health so a bad upstream release self-heals.

Safety rails (so hands-off ≠ reckless)

• Stable only: never track pre-release/edge tags unless explicitly opted in per service.
• Health-gated rollout: a failing update is caught by health checks and rolled back (R-UPD-03), and raises an alert (10).
• Backups precede risk: state is on Longhorn with backups, so a bad update never risks data (04).
• Observability: update events (image advanced, chart merged, rollback) are visible and, on failure, alert by email.
• Pin escape hatch: an operator can pin a service via git if an upstream release misbehaves — the one sanctioned human touch, and it stops automation rather than performing an update.

Interaction with default-deny & egress

Updates change image versions, not network posture. A VPN-class service stays VPN-class across updates; policies are reconciled from the (unchanged) ServiceDefinition.

Acceptance gates

• A service advances to a newer stable version automatically, on schedule, with no human/AI edit.
• A deliberately bad update is caught by health checks, auto-rolled-back, and alerts by email.
• A pinned service does not advance.
• Platform components (charts) stay current via auto-merged Renovate PRs.

Open questions

• Exact default in ADR-0009: write-back commits vs. digest-tracking mutable tags (trade readable history vs. zero-commit updates).
• Registry rate-limit / credential handling for image scanning at cadence.