home-network-k3s/docs/11-automated-updates.md

# 11 — Automated Updates

Serves: **R-UPD-01/02/03**. Related: [ADR-0023](../decisions/adr-0023-automated-updates-renovate.md).

## The requirement, precisely

> Services must update to latest **stable** versions **on a schedule**; updates in the allowed
> classes occur with **no human, no AI, and no hand-authored GitOps change**. (R-UPD-01/02)

This has a subtle tension with GitOps (where "the desired state is what's in git"). We resolve it
explicitly below.

## Resolving "no GitOps change" vs. GitOps

There are two interpretations; we honor the operator's intent (hands-off), and ADR-0023 records the
mechanism:

- **A human/AI authoring a version bump = prohibited.** No person and no chatbot edits a tag.
  Reviewing a machine-authored change is not authoring one.
- **An automation bot proposing a version advance under a policy declared once = permitted**,
  because the *operator's* declared intent (the update policy / version range) is the stable,
  human-authored thing; the bot merely resolves it, and its change reaches `main` through the same
  merge gate as every other change ([05](05-gitops-and-delivery.md)). Mechanisms by artifact type:

### 1. Container images, Helm charts and manifests → Renovate
- **Renovate** runs on a schedule, watches image tags, chart and dependency versions, and opens PRs.
  Git history records every advance as an auditable, machine-authored commit.
- **Minor, patch and digest** updates **auto-merge** on a green gate. This keeps services and
  platform components current with no human authoring.
- **Major** updates never auto-merge; they wait for human review of the machine-authored PR.
- **Operator-designated blast-radius components** never auto-merge, not even a patch: the storage and
  network planes, the reboot daemon, the database engine, the metrics stack, Gitea and its runner
  (they *are* the merge gate), the identity provider and its auth sidecars, and Renovate itself and
  the pinned CI toolchain. The list is data in the implementation repo's `renovate.json5`, each entry
  with its reason.
- Because `apps/` is rendered from `catalog/`, a bump updates the source and its rendered output in
  one PR, so render-equals-committed holds.

### 2. The platform's own images → Renovate digest tracking
- Images built by CI ([05](05-gitops-and-delivery.md)) are referenced by digest. Renovate follows
  each image's moving tag and proposes the digest bump as a PR, whose own gate run exercises it where
  the image is the toolbox.

### 3. VM guests → in-guest automation
- KubeVirt guests auto-update OS packages inside the guest on a schedule (unattended-upgrades / dnf-
  automatic) declared in the VM's cloud-init ([06](06-vm-workloads-kubevirt.md)). Base-image
  promotion follows the same automated philosophy where a golden-image pipeline exists.

### 4. Node OS → security-only upgrades, coordinated reboots
- Nodes apply **security-only** unattended upgrades, with kernels excluded. A reboot daemon performs
  only the mechanical half of a reboot, when a node requests one: one node at a time under a single
  cluster-wide lock, gated on storage health so etcd quorum and volume replicas are never put at
  risk. A hang stops the cycle and pages a human. Kernel upgrades stay a deliberate operator act.

## Per-service policy (R-UPD-03)

Each service's **update policy** is declared in git:
- **channel/range**: which versions are "stable" and acceptable (semver range, tag regex).
- **cadence**: how often to check/apply (the schedule).
- **hold/pin**: temporarily freeze a service (e.g. a known-bad upstream release).
- **rollout & health**: how the new version rolls out and what health gates it.
- **auto-rollback**: revert on failed health so a bad upstream release self-heals.

The ServiceDefinition ([13](13-service-catalog.md)) records that intent. Renovate does not read it,
so the policy takes effect through per-package rules in the Renovate configuration, which mirrors
each declared ceiling and carries any per-service cadence.

## Safety rails (so hands-off ≠ reckless)

- **Stable only**: never track pre-release/edge tags unless explicitly opted in per service.
- **Merge gate first**: every update is a PR that must pass the gate before it can merge.
- **Health-gated rollout**: a failing update is caught by health checks and rolled back (R-UPD-03),
  and raises an alert ([10](10-observability-and-alerting.md)).
- **Backups precede risk**: state is on Longhorn with backups, so a bad update never risks data
  ([04](04-storage-and-state.md)).
- **Observability**: update events (update proposed, merged, rollback) are visible and, on failure,
  alert by email.
- **Pin escape hatch**: an operator can pin a service via git if an upstream release misbehaves —
  the one sanctioned human touch, and it *stops* automation rather than performing an update.

## Interaction with default-deny & egress
Updates change image versions, not network posture. A VPN-class service stays VPN-class across
updates; policies are reconciled from the (unchanged) ServiceDefinition.

## Acceptance gates
- A service advances to a newer stable version automatically, on schedule, with no human/AI edit.
- A deliberately bad update is caught by health checks, auto-rolled-back, and alerts by email.
- A pinned service does **not** advance.
- Components in the allowed classes stay current via auto-merged Renovate PRs; majors and
  blast-radius components arrive as machine-authored PRs awaiting review.