jakman85andClaude Opus 5.5
0fb44bb0a8
Adds R-VULN-01..09, docs/17 and ADR-0027: a scheduled digest scanner in the cluster, a versioned report, an SSO page listing each CVE against its images, containers and workloads, grouped email alerts, and a Renovate-shaped CI triage bot. Renovate stays the only author of version changes, and "not scanned" is always visible rather than reading as clean. Co-Authored-By: Claude Opus 5.5 <[email protected]>
4.5 KiB
4.5 KiB
ADR-0027 — Vulnerability visibility: scheduled digest scanning with a Renovate-shaped bot
- Status: Accepted
- Date: 2026-11-09
- Serves: R-VULN-01..09, R-OBS-02, R-OBS-03, R-C2-02, R-UPD-02, R-SEC-02, R-IDP-06
- Supersedes: —
- Accepted by: operator merge of this change; effective on merge.
Context
The operator has no view of the vulnerabilities in what production runs. Renovate moves versions forward on a schedule, but it knows nothing about CVEs, and nothing in the platform assesses a running image. Measured on 2026-11-09:
- 259 containers run from 98 unique digests, across eight or more registries, with no scanner present.
- Prometheus holds 427k head series, using about 2.75 GiB of its 3 GiB limit.
- etcd and Longhorn share a partition on every node (accepted risk, docs/02), and etcd's own metrics are not scraped, so the headroom for added etcd writes cannot be measured.
- CI holds no cluster credential by decision, and has two runner slots that also serve the merge gate.
- The implementation repository is readable without authentication.
- Trivy's release pipeline was compromised in March 2026 (CVE-2026-33634).
Decision
A small scheduled scanner runs in the cluster, built on the Trivy CLI, and a Renovate-shaped CI bot turns its results into action. The full contract is docs/17.
- Scanner. Takes its inventory from the Kubernetes API, scans each unique digest with Trivy, and stores one versioned report on a small dedicated volume, outside etcd and Prometheus.
- Page. Served from that report behind SSO and restricted to
platform-admins. - Metrics. Aggregates only.
- Bot. A scheduled Gitea Actions workflow using the existing CI-scoped Renovate credential. It maintains a counts-only dashboard issue and requests Renovate PRs through Renovate's own dashboard approval mechanism.
- Renovate stays the sole author of version changes. No second bot edits a tag.
- Delivery tooling, not a catalog service. Its cluster-wide read role is something a
ServiceDefinition cannot express, so it is hand-authored under
platform/with its own non-gating Flux tier, as in ADR-0025.
Alternatives considered
- Trivy Operator. The obvious choice, and rejected on measurement. It stores one report per workload as a CRD, adding write load to an etcd we cannot currently measure and which shares its disk with Longhorn; the largest images produce the largest reports, so size-limit failures would hide exactly the worst offenders. It needs broad cluster permissions and job-creation rights. It would also still need a bespoke aggregator, because the per-CVE page cannot be fed from Prometheus.
- Grafana table over per-CVE Prometheus metrics. Needs no new page, but adds tens of thousands of long-labelled series to a Prometheus at about 92% of its memory limit. Raising the limit treats the symptom; a CVE inventory is not time-series data.
- Scanning from Gitea Actions. Closest to Renovate literally, but CI has no view of what is running, and a long scan would occupy one of the two runner slots that gate merges.
- A third-party Trivy dashboard. Assumes the operator's CRDs, and adds a further small-maintainer supply-chain dependency to a sensitive surface.
- An in-cluster bot with a Gitea token. Puts a git-writing credential in the cluster, which ADR-0023 rejects.
- A CVE dashboard issue listing the findings. The most Renovate-like, and rejected because the repository is publicly readable and the list would be an attack map.
- A bot that authors bump PRs itself. A package's fixed version does not identify the image tag containing it, and two version authors would contend. Rejected in favour of driving Renovate.
Consequences
- Scanning and its output are hands-off, and "not scanned" is visible rather than silent.
- The bot reuses the Renovate token, so both act as
renovate-bot. This avoids a new credential; splitting them later changes one workflow secret. - The platform carries a small custom component that it must maintain and test. The report schema isolates the scanner, so Trivy can be replaced without touching the page or the bot.
- Findings for private-registry images, VMs and nodes remain unassessed, and are shown as such (known gaps).
- The critical-fix alert will fire on first deployment. That is the backlog becoming visible.
- A tick can only open a Renovate PR. Remediation of majors and blast-radius components still waits for human review.