Merge feature/phase4-image-automation: Phase 4 hands-off updates (R-UPD-01/02/03)

platform/image-automation/README.md

@@ -8,22 +8,69 @@
 - **Serves:** R-UPD-02 (with R-UPD-01/03 safety rails)
 - **Owner:** WS-UPDATE (mechanism), WS-GITOPS (policy rendering from ServiceDefinition)
 
-## What this stub is
+## What lives here
 
-The **cluster-wide** `ImageUpdateAutomation` that scans registries and **commits the resolved version
-back to git** (machine-authored, auditable — ADR-0009). No human or AI authors a tag bump (R-UPD-02).
+- **`image-update-automation.yaml`** — the **cluster-wide** `ImageUpdateAutomation` that scans
+  registries per each service's `ImagePolicy` and **commits the resolved stable version back to git**
+  (machine-authored, auditable — ADR-0009). No human or AI authors a tag bump (R-UPD-02).
+- **`per-service/`** — the **render contract** (`update` block → `ImageRepository`/`ImagePolicy`/setter
+  marker) plus a worked example (`pihole.example.yaml`). **Reference only, not deployed** — WS-GITOPS
+  emits the equivalent next to each real app's Deployment/HelmRelease under `apps/`.
+
+## Controllers (how the CRDs get here)
+
+The two controllers — **image-reflector-controller** + **image-automation-controller** — are installed
+as part of the `flux-system` install by **`bootstrap/flux/bootstrap.sh`** via
+`flux install --components-extra=image-reflector-controller,image-automation-controller`, exactly like
+the other four Flux controllers. Consequences:
+
+- The `image.toolkit.fluxcd.io` CRDs (`ImageRepository`/`ImagePolicy`/`ImageUpdateAutomation`) **exist
+  before the first reconcile** — just like `GitRepository`/`Kustomization` do.
+- So the `ImageUpdateAutomation` CR here needs **no** F-MON1 controllers/configs split: F-MON1 bites
+  only when a CR is applied in the *same* Flux pass that installs its CRD (via a HelmRelease). Here the
+  CRD is a bootstrap artifact, so the CR applies cleanly from the `platform` Kustomization. (The
+  per-service CRs likewise ride with `apps/`, which `dependsOn platform`.)
 
 ## Scope boundary (contract)
 
-This dir holds **only** the cluster-wide automation + git write-back settings. **Per-service**
-`ImageRepository` / `ImagePolicy` (semver range / stable channel) are rendered from each
-ServiceDefinition's `update` block by WS-GITOPS/WS-UPDATE and live with `apps/` — the "Update policy
-contract" in `roadmap/workstreams.md`. **Helm chart / manifest bumps** are Renovate's job (separate),
-not this controller's.
+- **This dir:** the cluster-wide automation + git write-back settings.
+- **`apps/<svc>/`:** per-service `ImageRepository` / `ImagePolicy` (semver stable range) + the Deployment
+  setter marker — rendered from each ServiceDefinition's `update` block (see `per-service/README.md`).
+- **Renovate (`/renovate.json`):** Helm **chart** / dependency / operator-manifest bumps (opens PRs,
+  auto-merges allowed classes on green CI). Not this controller's job, and deliberately non-overlapping
+  (Renovate's docker/kubernetes image managers are disabled so it never fights Flux over a tag).
 
-## TODO (later phases — Phase 4)
+## Safety rails (acceptance gates — ADR-0009)
 
-- Confirm write-back strategy (direct commit to tracked branch vs. Renovate-style PRs).
-- Wire the safety rails that are **acceptance gates** (ADR-0009): stable-only, health-gated rollout,
-  auto-rollback on failed health, `pin` escape hatch, alert on update failure.
-- Requires the Flux **image-reflector** + **image-automation** controllers (Flux bootstrap component).
+| Rail | Where enforced |
+|---|---|
+| **Stable-only** (R-UPD-01) | `ImagePolicy` semver range + a `filterTags` pattern that admits only bare `X.Y.Z` (excludes `-rc`/`latest`). |
+| **No human/AI authoring** (R-UPD-02) | image-automation writes the tag at the `$imagepolicy` setter; the commit is machine-authored (`fluxcdbot`). |
+| **Health-gated auto-rollback** (R-UPD-03) | HelmRelease `upgrade.remediation` + `rollback` (preferred), or Deployment `progressDeadlineSeconds` + readiness probe. |
+| **Pin escape hatch** (R-UPD-03) | `update.pin` renders a fixed image with **no** setter marker and no/suspended `ImagePolicy` — automation is frozen. |
+| **Alert on failure** (R-UPD-02 + R-OBS) | `platform/monitoring/alerts.yaml` group `platform-health.automated-updates` (`ImageUpdateAutomationFailing`, `ImagePolicyStalled`, `UpdateHealthGateFailed`). |
+
+Proven by the L3 conformance test
+`conformance/tests/l3-fault-injection/hands-off-update/` (auto-advance stable, bad-update
+auto-rollback + alert-wired, pin holds).
+
+## Operator hand-off: git write credential (required before enabling)
+
+image-automation-controller reuses the **`flux-system` GitRepository's `.spec.secretRef`** for push auth
+(it has no independent secretRef). The Phase-1 source is anonymous/read-only, so before write-back works
+the operator must:
+
+1. Create a Gitea access token (scope `write:repository`) for the bot and put it in the SOPS secret
+   from the template `bootstrap/sops/secret-templates/gitea-image-automation-credentials.yaml.example`
+   → `platform/image-automation/gitea-image-automation-credentials.sops.yaml` (name
+   `gitea-image-automation-credentials`, keys `username`/`password`), then `sops --encrypt --in-place`.
+2. Add that file to this dir's `kustomization.yaml` `resources:`.
+3. Attach it to the source: add `spec.secretRef.name: gitea-image-automation-credentials` to the
+   `flux-system` GitRepository via `bootstrap/flux/patches/flux-system-source-url.yaml`.
+
+## Enabling (deferred — like velero/oauth2-proxy)
+
+`image-automation` is intentionally **commented out** in `platform/kustomization.yaml`. Enable it only
+after the write credential above is provisioned — otherwise the `ImageUpdateAutomation` reconcile errors
+on push and, under the unified graph's `platform` wait, could stall the downstream graph. Uncomment the
+`- image-automation` line once the hand-off is done.

platform/image-automation/image-update-automation.yaml

@@ -1,39 +1,57 @@
 ---
 # Flux image-update-automation — the CLUSTER-WIDE write-back engine for hands-off image tag advances
-# (ADR-0009, R-UPD-02). It scans registries per each service's ImagePolicy and commits the resolved
-# version back to git — no human/AI authors the bump. Phase 0 STUB.
+# (ADR-0009, R-UPD-02). image-reflector-controller scans each per-service ImageRepository on its
+# interval; image-automation-controller resolves each ImagePolicy and writes the advanced tag back to
+# git at the `# {"$imagepolicy": "..."}` setter markers. The commit is MACHINE-authored under the
+# operator's once-declared policy — no human/AI edits a tag (R-UPD-01/02).
 #
-# SCOPE: this dir holds ONLY the cluster-wide ImageUpdateAutomation + git write-back settings.
-# Per-service `ImageRepository`/`ImagePolicy` are rendered from each ServiceDefinition's `update`
-# block (WS-GITOPS/WS-UPDATE) and live with apps/ — NOT here. (Contract: roadmap/workstreams.md
-# "Update policy contract".)
+# SCOPE (contract, roadmap/workstreams.md "Update policy contract"): this dir holds ONLY the
+# cluster-wide ImageUpdateAutomation + git write-back settings. PER-SERVICE ImageRepository/ImagePolicy
+# are rendered from each ServiceDefinition's `update` block (renderer-contract §update) and live NEXT
+# TO each app's Deployment/HelmRelease under apps/ — see ./per-service/ for the render contract + a
+# worked example. Helm CHART / dependency bumps are Renovate's job (renovate.json), not this controller.
+#
+# CONTROLLERS: image-reflector-controller + image-automation-controller are installed as part of the
+# flux-system install (bootstrap/flux/bootstrap.sh `--components-extra`), so the image.toolkit CRDs
+# pre-exist before reconcile — this CR applies cleanly without the F-MON1 controllers/configs split
+# (the CRD is present at bootstrap, exactly like GitRepository/Kustomization are). See README.md.
 apiVersion: image.toolkit.fluxcd.io/v1beta2
 kind: ImageUpdateAutomation
 metadata:
   name: flux-image-automation
   namespace: flux-system
 spec:
-  interval: 30m                 # the "schedule" on which stable versions are picked up (R-UPD-02)
+  # The scan/apply schedule (R-UPD-02). image-reflector scans registries continuously per
+  # ImageRepository.spec.interval; this interval bounds how often resolved tags are written back.
+  interval: 30m
   sourceRef:
     kind: GitRepository
     name: flux-system
+    # ^ Write-back auth REUSES this GitRepository's `.spec.secretRef` (image-automation-controller has
+    #   no independent secretRef). The Phase-1 flux-system source is currently anonymous/read-only
+    #   (public host-Gitea clone). For write-back the operator must attach a WRITE credential to the
+    #   flux-system GitRepository — see README.md "Operator hand-off: git write credential" and the
+    #   SOPS template bootstrap/sops/secret-templates/gitea-image-automation-credentials.yaml.example.
   git:
     checkout:
       ref:
-        branch: main            # TODO(WS-UPDATE): confirm write-back branch (may open PRs instead)
+        branch: main            # reconciled branch (matches clusters/base/flux-system.yaml)
     commit:
       author:
-        name: fluxcdbot         # machine author — the auditable, intended write-back (ADR-0009)
+        # Machine author — the auditable, intended write-back (ADR-0009). No human/AI authors the bump.
+        name: fluxcdbot
         email: [email protected]
       messageTemplate: |
-        chore(images): automated update
+        chore(images): automated stable update [ci skip]
 
-        Machine-authored image tag advance per ImagePolicy (R-UPD-02, ADR-0009).
+        Machine-authored image tag advance under the declared per-service ImagePolicy
+        (R-UPD-01 stable-only, R-UPD-02 no human/AI authoring). ADR-0009.
+        {{ range .Changed.Changes -}}
+        - {{ .OldValue }} -> {{ .NewValue }} ({{ .Setter }})
+        {{ end -}}
     push:
-      branch: main
+      branch: main             # push straight to the tracked branch (default write-back strategy,
+                               # ADR-0009). Renovate handles PR-style bumps for charts/deps separately.
   update:
-    strategy: Setters           # image-automation setter markers in per-service manifests (apps/)
-    path: ./apps
-# TODO(WS-UPDATE, Phase 4): safety rails are acceptance gates (ADR-0009) — stable-only, health-gated
-#   rollout, auto-rollback on failed health, pin escape hatch, alert on update failure. Prove a test
-#   service auto-advances with no human/AI edit; a bad update auto-rolls-back + alerts (R-UPD-01/02/03).
+    strategy: Setters          # advance the `$imagepolicy` setter markers in per-service manifests
+    path: ./apps               # only apps/ carries per-service image markers (platform charts = Renovate)

platform/image-automation/per-service/README.md

@@ -0,0 +1,48 @@
+# Per-service image-automation — the render contract (`update` block → Flux CRs)
+
+**Owner:** WS-UPDATE (this contract) + WS-GITOPS (emits these next to each app's Deployment/HelmRelease
+under `apps/`). **Serves:** R-UPD-01 (stable-only), R-UPD-02 (no human/AI authoring), R-UPD-03
+(pin / auto-rollback). **Source of truth:** [`renderer-contract.md` §`update`](../../../schemas/renderer-contract.md),
+[ADR-0009](../../../../home-network-k3s/decisions/adr-0009-automated-image-updates.md),
+[docs/11](../../../../home-network-k3s/docs/11-automated-updates.md).
+
+> These files are **reference examples**, deliberately **not** wired into any `kustomization.yaml`, so
+> they never deploy. WS-GITOPS renders the equivalent per real container service into `apps/<svc>/`.
+> The cluster-wide write-back engine that consumes the setter markers is `../image-update-automation.yaml`.
+
+## What a container `kind` ServiceDefinition `update` block renders
+
+For every `kind: container` service whose `update.pin` is **unset**, render THREE things:
+
+| `update` field | Renders to | Notes |
+|---|---|---|
+| (implicit — `source.image`) | **`ImageRepository`** | scans the registry repo on `interval`. |
+| `channel: stable` + `range` | **`ImagePolicy`** `.spec.policy.semver.range` | **stable-only**: the range plus a filter that **excludes pre-release** tags (`-alpha/-rc/…`). |
+| `cadence` | `ImageRepository.spec.interval` (scan schedule) | the "on a schedule" of R-UPD-02. Cron `cadence` → nearest Flux duration (Flux scans on a duration, not cron). |
+| (image ref in the Deployment) | a **setter marker** comment on the `image:` field | `# {"$imagepolicy": "<ns>:<name>"}` — where image-automation writes the advanced tag. |
+| `autoRollbackOnFailedHealth: true` | delivery-side **health gate + rollback** | see below — HelmRelease remediation, or Deployment progressDeadline + revert. |
+| `pin: <version>` | **freeze** | render the image `image:` pinned to `<version>` with **NO** setter marker, and **suspend** (or omit) the ImagePolicy. Automation cannot advance a pinned service (R-UPD-03). |
+
+`kind: vm` services do **not** render these — they update in-guest (dnf-automatic/unattended-upgrades on
+`cadence`, cloud-init), per docs/11 §3. See `catalog/vm-demo.yaml`.
+
+## Stable-only (R-UPD-01) — the two guardrails
+1. **semver range** (`.spec.policy.semver.range`) — only versions matching the operator's `range`.
+2. **pre-release exclusion** — either a `filterTags` pattern that only admits `X.Y.Z` (no `-suffix`),
+   or rely on semver's rule that pre-releases (`1.2.3-rc1`) sort **below** their release and are only
+   selected if the range explicitly asks for them. We do BOTH (belt-and-braces) so an upstream `latest`
+   or `-rc` tag can never be picked.
+
+## Auto-rollback (R-UPD-03) — where the health gate lives
+The setter write only changes a tag in git; the **rollback** is enforced at delivery:
+- **HelmRelease-delivered services (preferred):** Flux `HelmRelease.spec.upgrade.remediation.retries`
+  + `.spec.rollback` — a failed, health-gated upgrade auto-rolls-back to the last-good Helm revision.
+  The image tag lives in the HelmRelease `values` with the setter marker, so a bad advance is caught
+  and reverted by Flux with no human step. This is the mechanism the conformance test exercises.
+- **Plain-Deployment services:** the Deployment `progressDeadlineSeconds` + readiness probe make a bad
+  image a stalled (never-Available) rollout; the old ReplicaSet keeps serving, and the
+  `UpdateHealthGateFailed` alert fires (see `platform/monitoring/alerts.yaml`). Recovery is the next
+  good advance or the operator `pin` escape hatch.
+
+Either way a failed/rolled-back update **alerts by email** (R-UPD-02 + R-OBS) via the
+`automated-updates` PrometheusRule group.

platform/image-automation/per-service/pihole.example.yaml

@@ -0,0 +1,75 @@
+---
+# WORKED EXAMPLE (reference only — NOT wired into a kustomization, never deployed).
+# Rendered from catalog/pihole.yaml's `update` block by WS-GITOSS into apps/pihole/. Shows the exact
+# ImageRepository + ImagePolicy + Deployment setter-marker shape for a container ServiceDefinition.
+#
+#   update:
+#     channel: stable
+#     range: ">=2024.0.0"
+#     cadence: "0 4 * * *"
+#     autoRollbackOnFailedHealth: true
+#
+# See ./README.md for the field-by-field contract.
+# ---------------------------------------------------------------------------------------------------
+apiVersion: image.toolkit.fluxcd.io/v1beta2
+kind: ImageRepository
+metadata:
+  name: pihole
+  namespace: pihole
+spec:
+  image: pihole/pihole            # from source.image
+  interval: 6h                    # scan schedule; nearest Flux duration to cadence "0 4 * * *" (daily-ish)
+  # provider: generic             # dockerhub anon; a pull secret goes here if registry rate-limits bite
+---
+apiVersion: image.toolkit.fluxcd.io/v1beta2
+kind: ImagePolicy
+metadata:
+  name: pihole
+  namespace: pihole
+spec:
+  imageRepositoryRef:
+    name: pihole
+  # STABLE-ONLY (R-UPD-01): (1) semver range from `update.range`; (2) a tag filter that admits only
+  # bare X.Y.Z releases so a pre-release / `latest` / date-nightly tag can never be selected.
+  filterTags:
+    pattern: '^v?(?P<v>[0-9]+\.[0-9]+\.[0-9]+)$'
+    extract: '$v'
+  policy:
+    semver:
+      range: '>=2024.0.0'
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: pihole
+  namespace: pihole
+spec:
+  replicas: 1
+  # progressDeadlineSeconds bounds a bad rollout so the health gate (readiness probe) can catch it and
+  # the old ReplicaSet keeps serving — the plain-Deployment auto-rollback path (R-UPD-03). A
+  # HelmRelease-delivered service would instead use spec.upgrade.remediation + spec.rollback.
+  progressDeadlineSeconds: 120
+  selector:
+    matchLabels: { app: pihole }
+  template:
+    metadata:
+      labels: { app: pihole }
+    spec:
+      containers:
+        - name: pihole
+          # THE SETTER MARKER: image-automation writes the ImagePolicy-resolved stable tag here.
+          # No human/AI ever edits this tag — the marker binds it to the pihole ImagePolicy above.
+          image: pihole/pihole:2024.7.0 # {"$imagepolicy": "pihole:pihole"}
+          readinessProbe:               # the health gate: a bad image never becomes Ready
+            httpGet: { path: /admin/, port: 80 }
+            initialDelaySeconds: 10
+            periodSeconds: 5
+---
+# PINNED VARIANT (R-UPD-03 escape hatch) — what the SAME service renders when `update.pin: "2024.7.0"`
+# is set: the image is fixed, the setter marker is ABSENT, and NO ImagePolicy/ImageRepository is
+# rendered (or it is suspended). Automation cannot advance it. This is the one sanctioned human touch,
+# and it STOPS automation rather than performing an update.
+#
+#   containers:
+#     - name: pihole
+#       image: pihole/pihole:2024.7.0   # pinned — no $imagepolicy marker; automation is frozen

platform/monitoring/alerts.yaml

@@ -369,3 +369,68 @@ spec:
             summary: "More than 10% of Prometheus targets are down"
             description: "Over 10% of scrape targets have been down for 15m — a broad monitoring blind spot."
             runbook_url: "https://home.keithjackson.dev/runbooks/targets-down"
+
+# =====================================================================================
+# Automated updates — Flux image-automation + hands-off update health (docs/11, ADR-0009)
+#   R-UPD-02 (a failed/rolled-back update alerts) + R-OBS. All Flux controllers (incl. the image
+#   reflector/automation + helm controllers) export the shared `gotk_reconcile_condition` /
+#   `gotk_suspend_status` series; these rules add UPDATE-SPECIFIC visibility on top of the generic
+#   platform-health.flux group so a silently-stalled or a rolled-back update is not missed.
+# =====================================================================================
+---
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: platform-health-automated-updates
+  namespace: monitoring
+  labels:
+    release: kube-prometheus-stack
+    platform-health: "true"
+spec:
+  groups:
+    - name: platform-health.automated-updates
+      rules:
+        # The write-back engine (ADR-0009) is broken → hands-off image updates SILENTLY stop advancing.
+        - alert: ImageUpdateAutomationFailing
+          expr: |
+            max by (namespace, name) (
+              gotk_reconcile_condition{kind="ImageUpdateAutomation", type="Ready", status="False"}
+            ) == 1
+          for: 30m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Flux image-update-automation failing: {{ $labels.name }}"
+            description: "ImageUpdateAutomation {{ $labels.namespace }}/{{ $labels.name }} cannot reconcile (git write-back) for 30m — stable image advances have stopped (R-UPD-02). Check the git write credential."
+            runbook_url: "https://home.keithjackson.dev/runbooks/image-automation"
+        # An ImagePolicy/ImageRepository can't resolve the latest stable tag (registry rate-limit /
+        # auth / bad range) → that service stops receiving updates even though automation is healthy.
+        - alert: ImagePolicyStalled
+          expr: |
+            max by (namespace, name, kind) (
+              gotk_reconcile_condition{kind=~"ImagePolicy|ImageRepository", type="Ready", status="False"}
+            ) == 1
+          for: 1h
+          labels:
+            severity: warning
+          annotations:
+            summary: "Image scan/policy stalled: {{ $labels.kind }}/{{ $labels.name }}"
+            description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} cannot resolve its stable tag for 1h (registry rate-limit / creds / range) — no longer receiving updates (R-UPD-01)."
+            runbook_url: "https://home.keithjackson.dev/runbooks/image-policy-stalled"
+        # A bad update was health-gated and AUTO-ROLLED-BACK (or is stuck mid-remediation). helm-controller
+        # holds Ready=False while a failed upgrade remediates/rolls back to the last-good revision
+        # (HelmRelease.spec.upgrade.remediation + .spec.rollback). This is the R-UPD-03 safety rail firing.
+        # TODO(bring-up): tighten with the `reason` label (e.g. UpgradeFailed / RolledBack) once confirmed
+        # against the deployed helm-controller metrics.
+        - alert: UpdateHealthGateFailed
+          expr: |
+            max by (namespace, name) (
+              gotk_reconcile_condition{kind="HelmRelease", type="Ready", status="False"}
+            ) == 1
+          for: 10m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Update health-gate failed / rolled back: {{ $labels.name }}"
+            description: "HelmRelease {{ $labels.namespace }}/{{ $labels.name }} failed its health-gated upgrade and is rolling back to the last-good revision (bad-update auto-rollback — R-UPD-03). Consider an `update.pin` escape hatch (docs/11)."
+            runbook_url: "https://home.keithjackson.dev/runbooks/update-rollback"