docs: specify automated vulnerability visibility for production (R-VULN-*, ADR-0027)
Adds R-VULN-01..09, docs/17 and ADR-0027: a scheduled digest scanner in the cluster, a versioned report, an SSO page listing each CVE against its images, containers and workloads, grouped email alerts, and a Renovate-shaped CI triage bot. Renovate stays the only author of version changes, and "not scanned" is always visible rather than reading as clean. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5.5 committed
1 parent
c626349c7d
commit
0fb44bb0a8
12 files changed
+413
-1
No files matched your search
@@ -150,6 +150,28 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
|
||||
- **R-SEC-04** — The Kubernetes API MUST NOT be reachable from the public internet by any path (no
|
||||
tunnel route, no edge forward, no public DNS record); it is LAN/VPN-only (ADR-0016).
|
||||
|
||||
### Vulnerability visibility — `R-VULN-*`
|
||||
- **R-VULN-01** — The platform MUST maintain an inventory of every container image running in
|
||||
production, by digest, mapped to the namespaces, workloads and containers using it, taken from the
|
||||
running cluster and not from git.
|
||||
- **R-VULN-02** — Every running image MUST be evaluated against a current vulnerability database at
|
||||
least every 24 hours, and a newly running image within 2 hours.
|
||||
- **R-VULN-03** — Any target not successfully evaluated MUST be visibly reported as such with its
|
||||
reason. Absence of data MUST NOT be presented as absence of vulnerabilities.
|
||||
- **R-VULN-04** — An authenticated page MUST list every known CVE and, for each, the images,
|
||||
containers and workloads it affects, and MUST be searchable and filterable.
|
||||
- **R-VULN-05** — Inventory, scanning, reporting and triage MUST run on a schedule with **no human
|
||||
and no AI action**, and MUST fail loudly when they cannot run.
|
||||
- **R-VULN-06** — Scan failure, staleness, and unaccepted CRITICAL vulnerabilities with an available
|
||||
fix MUST be reported by **email**, as grouped alerts.
|
||||
- **R-VULN-07** — A scheduled bot MUST maintain a single dashboard issue and request remediation
|
||||
through Renovate. Renovate MUST remain the only author of version changes, and CVE detail MUST NOT
|
||||
be published to any surface readable without authentication.
|
||||
- **R-VULN-08** — Accepted risk MUST be declared in git, through the merge gate, with a reason and a
|
||||
mandatory expiry, and MUST revert to open automatically when it expires.
|
||||
- **R-VULN-09** — The scanner MUST be digest-pinned, MUST NOT auto-merge its own updates, MUST NOT
|
||||
read Secrets or hold Kubernetes write access, and MUST hold no git credential.
|
||||
|
||||
### Identity & access (SSO) — `R-IDP-*`
|
||||
- **R-IDP-01** — The platform MUST provide a central Identity Provider (SSO) as the single user
|
||||
directory and login for platform services.
|
||||
|
||||
@@ -62,6 +62,9 @@ Grafana dashboards, surfaced in the C2 UI ([12](12-command-center-ui.md)):
|
||||
- **Network**: Hubble flows, egress/VPN health, denied-flow trends.
|
||||
- **Per-service**: one row/dashboard per catalog service with its declared SLIs.
|
||||
|
||||
The CVE listing is deliberately **not** a Grafana dashboard: per-CVE data does not belong in
|
||||
Prometheus. See [17](17-vulnerability-visibility.md), which also defines the vulnerability alerts.
|
||||
|
||||
## Observability as onboarding output
|
||||
|
||||
The ServiceDefinition ([13](13-service-catalog.md)) declares observability expectations; the
|
||||
|
||||
@@ -78,6 +78,12 @@ each declared ceiling and carries any per-service cadence.
|
||||
- **Pin escape hatch**: an operator can pin a service via git if an upstream release misbehaves —
|
||||
the one sanctioned human touch, and it *stops* automation rather than performing an update.
|
||||
|
||||
## Vulnerability-driven requests
|
||||
|
||||
The CVE triage bot ([17](17-vulnerability-visibility.md)) can ask Renovate to open a pending update
|
||||
for an image with an unaccepted CRITICAL that has a fix, using Renovate's own dashboard approval.
|
||||
Renovate remains the only author of version changes, and every merge rule above still applies.
|
||||
|
||||
## Interaction with default-deny & egress
|
||||
Updates change image versions, not network posture. A VPN-class service stays VPN-class across
|
||||
updates; policies are reconciled from the (unchanged) ServiceDefinition.
|
||||
|
||||
@@ -37,6 +37,8 @@ root aggregates and links them. Status of this composition is tracked in
|
||||
- **Platform health**: node status, Longhorn capacity/health, Flux reconciliation, cert expiry,
|
||||
backup status, VPN tunnel state — sourced from the observability stack ([10](10-observability-and-alerting.md)).
|
||||
- **Alerts**: current firing alerts (mirrors what email delivers).
|
||||
- **Vulnerabilities**: a link to the CVE page, restricted to `platform-admins`
|
||||
([17](17-vulnerability-visibility.md)).
|
||||
|
||||
## Actions (safe, audited)
|
||||
|
||||
|
||||
@@ -102,6 +102,10 @@ Each maps to requirement IDs; phrased as Given/When/Then intent.
|
||||
an email alert is delivered and the dashboard reflects it.
|
||||
- **GitOps drift (R-GITOPS-03, L3):** *Given* a reconciled resource, *When* it is mutated live,
|
||||
*Then* Flux reverts it to the git-declared state.
|
||||
- **Vulnerability visibility (R-VULN-*, L0/L1/L2/L3):** *Given* the running inventory, *Then* every
|
||||
image is either evaluated within its window or visibly reported as not evaluated, and each CVE is
|
||||
listed against its images, containers and workloads. Full list in
|
||||
[17](17-vulnerability-visibility.md).
|
||||
- **Onboarding completeness (WS-APPS):** *Given* only a merged ServiceDefinition, *Then* the service
|
||||
is fully wired (compute/storage/policy/egress/TLS/DNS/monitoring/update) with no manual step —
|
||||
asserted by the union of the relevant tests above against that service.
|
||||
|
||||
@@ -84,6 +84,7 @@ These cannot be settled by reading either repository.
|
||||
| R-C2-02 | Grafana dashboards are specified as embedded in the C2, but Homepage only links to Grafana. | `platform/homepage/configmap.yaml` |
|
||||
| R-C2-02 | Flux status is specified via Headlamp's Flux view, the Flux UI or a Homepage widget, but Headlamp has no Flux plugin configured and Homepage has no widget. | `platform/headlamp/helmrelease.yaml`, `platform/homepage/configmap.yaml` |
|
||||
| R-C2-02 | C2 actions are specified to include VM start/stop, Flux reconcile and alert acknowledgement, but only native Headlamp workload actions exist, with no plugins for the others. | `platform/headlamp/helmrelease.yaml` |
|
||||
| R-VULN-01, R-VULN-02 | KubeVirt guests, node OS packages and the k3s binary are inventoried but not vulnerability-assessed; private-registry images need a read-only credential the operator has yet to mint. All report as not evaluated (docs/17). | not yet implemented |
|
||||
|
||||
## Service catalog
|
||||
|
||||
|
||||
@@ -0,0 +1,289 @@
|
||||
# 17 — Vulnerability Visibility
|
||||
|
||||
Serves: **R-VULN-01..09**, surfaces **R-OBS-02/03**, **R-C2-02**, **R-UPD-01/02**, **R-SEC-02**,
|
||||
**R-IDP-06**. Decision record: [ADR-0027](../decisions/adr-0027-vulnerability-visibility.md).
|
||||
|
||||
## Purpose
|
||||
|
||||
The operator must be able to answer three questions at any time, with no manual step:
|
||||
|
||||
1. **What is running in production?** Every container image, by digest, and the workloads using it.
|
||||
2. **What is it vulnerable to?** Every known CVE in those images, and where each one lands.
|
||||
3. **Can I trust the answer?** How fresh it is, and exactly what was *not* examined.
|
||||
|
||||
The third question is as important as the first two. A vulnerability list that silently omits an
|
||||
image reads the same as a clean bill of health. Everything below treats "not scanned" as a
|
||||
first-class state that is always visible, and never as zero findings.
|
||||
|
||||
## Shape of the solution
|
||||
|
||||
The workflow is deliberately the same shape as Renovate ([11](11-automated-updates.md)): scheduled,
|
||||
self-hosted, run to completion, machine-authored output, a single dashboard issue, and a loud failure
|
||||
when it cannot run.
|
||||
|
||||
```
|
||||
Kubernetes API (read-only)
|
||||
│ running pods → digests → workloads
|
||||
▼
|
||||
┌──────────────────────────── namespace: vuln-scanner ─────────────────────────┐
|
||||
│ Scanner (scheduled, one image at a time, Trivy) │
|
||||
│ └─► Report store (small PVC) ─► report.json (schema below) │
|
||||
│ ├─► CVE page cve.<domain>, SSO, platform-admins │
|
||||
│ ├─► /metrics aggregates only → Prometheus → email │
|
||||
│ └─► internal API ◄──────────────┐ │
|
||||
└─────────────────────────────────────────────┼────────────────────────────────┘
|
||||
│ read report; post heartbeat + remediation state
|
||||
┌────────────────┴───────────────┐
|
||||
│ Gitea Actions: CVE triage bot │ scheduled, before Renovate
|
||||
└────────────────┬───────────────┘
|
||||
│ scoped CI token
|
||||
▼
|
||||
CVE Dashboard issue (counts only) · approval ticks on Renovate's dashboard
|
||||
│
|
||||
▼
|
||||
Renovate opens the PR → merge gate → Flux (the existing update path)
|
||||
```
|
||||
|
||||
The split follows two standing boundaries: **CI holds no cluster credential**, and **the cluster
|
||||
holds no git credential** ([ADR-0023](../decisions/adr-0023-automated-updates-renovate.md)). The
|
||||
scanner sees the cluster and never writes to git. The bot writes to Gitea and never touches the
|
||||
Kubernetes API.
|
||||
|
||||
## Inventory (R-VULN-01)
|
||||
|
||||
The inventory is taken from the **Kubernetes API**, never from git. Git states intent; the question
|
||||
is what is actually running.
|
||||
|
||||
- It covers containers, init containers and ephemeral containers of every pod in every namespace.
|
||||
- The identity of a running image is the **digest reported in the container status**. A mutable tag
|
||||
is not an identity.
|
||||
- Each pod resolves through its owner chain to the **top-level workload** the operator recognises
|
||||
(Deployment, StatefulSet, DaemonSet, CronJob, or a bare Pod), not the intermediate ReplicaSet or Job.
|
||||
- **CronJobs with no pod currently running are still inventoried**, from their pod template. Their
|
||||
images are recorded by reference and marked `by-tag`, because no digest exists until they run.
|
||||
- KubeVirt VMs and nodes are inventoried as **non-container targets** and reported as not covered
|
||||
(see [scope](#scope)).
|
||||
|
||||
## Scanning (R-VULN-02, R-VULN-05)
|
||||
|
||||
| Property | Contract |
|
||||
|---|---|
|
||||
| Unit of work | One scan per **unique digest**, mapped back to all its workloads. Measured 2026-11-09: 259 containers resolve to 98 digests. |
|
||||
| Full re-evaluation | Every running digest is evaluated against a current database **at least every 24 hours**, so a CVE published against an unchanged image is found without any deployment. |
|
||||
| New digests | A digest not yet seen is scanned **within 2 hours** of first appearing. |
|
||||
| Database freshness | A scan uses a vulnerability database no older than **48 hours**. Older than that, the run is reported as degraded, not as a success. |
|
||||
| Severity scope | All severities are recorded, including those with no fix. Filtering is a page concern. |
|
||||
| Failure isolation | One image failing (pull error, timeout, out of memory) records that image as `failed` with its reason and the run continues. |
|
||||
| Load | Sequential, one image at a time, with CPU and memory limits. Scan caches MUST NOT live on a replicated Longhorn volume or in etcd, because etcd and Longhorn share a partition on every node ([02](02-environments-and-provisioning.md)). |
|
||||
| Trigger | Schedule only. No human or AI action starts, resumes or completes a cycle. |
|
||||
|
||||
Findings persist a **first-seen** timestamp across runs, keyed by CVE and digest, so age is
|
||||
measurable. A finding that disappears because an image was updated leaves the report at the next
|
||||
inventory; one that disappears because a scan *failed* does not.
|
||||
|
||||
## Coverage (R-VULN-03)
|
||||
|
||||
Every inventoried target has exactly one state:
|
||||
|
||||
| State | Meaning |
|
||||
|---|---|
|
||||
| `scanned` | Evaluated within the 24-hour window. Zero findings is meaningful only in this state. |
|
||||
| `pending` | Discovered and not yet scanned, within the 2-hour window. |
|
||||
| `stale` | Last successful scan is older than the 24-hour window. Prior findings are retained and labelled stale. |
|
||||
| `failed` | The most recent attempt failed. The reason is shown. Prior findings are retained. |
|
||||
| `credential-required` | The registry refused anonymous access. Shown with the registry. |
|
||||
| `by-tag` | Scanned by mutable reference, because no running digest exists. |
|
||||
| `not-covered` | A target class this scanner cannot examine: VMs and nodes. |
|
||||
|
||||
The page and the report MUST NOT present a total or a "clean" indication that silently excludes any
|
||||
target. The header of the page always states how many targets are in each state.
|
||||
|
||||
## The report contract
|
||||
|
||||
`report.json` is the one interface between the scanner, the page and the bot. It is versioned and
|
||||
the implementation validates it against a schema.
|
||||
|
||||
```yaml
|
||||
schemaVersion: 1
|
||||
generatedAt: <RFC 3339>
|
||||
database: { updatedAt: <RFC 3339>, version: <string> }
|
||||
run: { startedAt: <RFC 3339>, finishedAt: <RFC 3339>, status: complete | degraded }
|
||||
targets:
|
||||
- id: <stable identifier>
|
||||
type: image | vm | node
|
||||
image: <registry/repository>
|
||||
digest: <sha256:… or null>
|
||||
state: <one of the coverage states>
|
||||
reason: <string or null>
|
||||
lastScannedAt: <RFC 3339 or null>
|
||||
workloads:
|
||||
- { namespace: <ns>, kind: <kind>, name: <name>, containers: [<name>] }
|
||||
findings:
|
||||
- id: <CVE or advisory id>
|
||||
targetId: <targets[].id>
|
||||
severity: CRITICAL | HIGH | MEDIUM | LOW | UNKNOWN
|
||||
cvss: <number or null>
|
||||
package: <name>
|
||||
installedVersion: <string>
|
||||
fixedVersion: <string or null>
|
||||
title: <string>
|
||||
url: <advisory URL>
|
||||
firstSeenAt: <RFC 3339>
|
||||
status: open | fix-available | accepted
|
||||
acceptance: { reason: <string>, expires: <date> } # present only when accepted
|
||||
remediation: # written by the triage bot, may be empty
|
||||
- { image: <registry/repository>, state: <state>, url: <PR URL or null>, updatedAt: <RFC 3339> }
|
||||
```
|
||||
|
||||
Scanner-derived text (titles, package names, advisory URLs) originates upstream and is **untrusted**.
|
||||
The page MUST escape it and serve a restrictive Content-Security-Policy, and only `http(s)` URLs may
|
||||
become links.
|
||||
|
||||
## The CVE page (R-VULN-04)
|
||||
|
||||
A page at `cve.<domain>`, linked from the C2 landing page.
|
||||
|
||||
**Header, always visible:** report age, database age, per-state target counts, and open findings by
|
||||
severity. If the report is stale or degraded, the page says so prominently and does not show a
|
||||
normal state.
|
||||
|
||||
**CVEs view (default):** one row per CVE, with id (linking to the advisory), severity, CVSS,
|
||||
package, fixed version, first seen, status, and the counts of affected images and workloads. Each
|
||||
row expands to list **every image, container and workload** it touches, with namespace.
|
||||
|
||||
**Workloads view:** one row per workload, with its containers, images and per-severity counts,
|
||||
expanding to its CVEs.
|
||||
|
||||
**Inventory view:** every target and its coverage state, *including those with zero findings*. This
|
||||
is the answer to "what am I running".
|
||||
|
||||
**Interaction:** free-text search, and filters for severity, namespace, workload, status and "fix
|
||||
available", with sortable columns. Filter state is in the URL so a view can be linked. The
|
||||
default view excludes nothing.
|
||||
|
||||
**Remediation column:** for images with a fix available, one of `pr-open` (with link),
|
||||
`approval-requested`, `no-update-identified` or `not-tracked`, taken from the bot.
|
||||
|
||||
**Access:** internal exposure only, behind SSO, restricted to `platform-admins` (R-IDP-06). A list of
|
||||
what is exploitable where is sensitive, so this is not a "logged-in user" surface. The page is
|
||||
read-only and offers no actions.
|
||||
|
||||
## Alerting (R-VULN-06)
|
||||
|
||||
Alerts are delivered through the existing Alertmanager email route. Each alert is **one grouped
|
||||
message per condition**, never one per CVE.
|
||||
|
||||
| Alert | Fires when |
|
||||
|---|---|
|
||||
| Scan stale | No complete run within 26 hours, **or the metric is absent** |
|
||||
| Database stale | Vulnerability database older than 48 hours |
|
||||
| Coverage gap | Any target in `failed` for more than 24 hours |
|
||||
| Critical fix available | Any unaccepted CRITICAL with a fix, first seen more than 72 hours ago |
|
||||
| Triage bot silent | No bot heartbeat for 26 hours, **or the metric is absent** |
|
||||
|
||||
The 72-hour grace gives Renovate's daily cycle time to act, so email marks the cases automation has
|
||||
not resolved. The critical-fix alert is expected to fire on first deployment.
|
||||
|
||||
Prometheus receives **aggregates only**: counts by severity and fixability, per-state target counts,
|
||||
timestamps. It MUST NOT receive a per-CVE series. Measured 2026-11-09: Prometheus holds 427k head
|
||||
series at about 2.75 GiB of a 3 GiB limit.
|
||||
|
||||
## The triage bot (R-VULN-05, R-VULN-07)
|
||||
|
||||
A scheduled Gitea Actions workflow that follows the conventions of the Renovate workflow: it fails
|
||||
hard on a missing credential rather than skipping, has a `workflow_dispatch` dry run, never runs
|
||||
concurrently with itself, and records what it did in the job summary. It runs **before** the daily
|
||||
Renovate run so that approvals take effect the same morning.
|
||||
|
||||
Each run it:
|
||||
|
||||
1. Reads `report.json` over the internal API.
|
||||
2. Creates or updates a single **CVE Dashboard** issue on the implementation repository.
|
||||
3. Reads Renovate's Dependency Dashboard and open PRs, and derives each image's remediation state.
|
||||
4. Ticks the **approval** checkbox on Renovate's dashboard for a pending update whose package matches
|
||||
an image with an unaccepted CRITICAL that has a fix.
|
||||
5. Posts its heartbeat and the remediation states back to the scanner.
|
||||
|
||||
**Renovate remains the only author of version changes** (R-UPD-02, ADR-0023). The bot never edits a
|
||||
version, opens a bump PR or merges anything. A tick only asks Renovate to *open* a PR; majors and
|
||||
blast-radius components still wait for human review, exactly as today.
|
||||
|
||||
**The dashboard issue carries aggregates only:** total findings by severity, per-state coverage,
|
||||
timestamps and a link to the page. It MUST NOT name a CVE, package or image-to-workload relationship,
|
||||
because the repository is readable without authentication (measured 2026-11-09). It MUST NOT carry any
|
||||
label that the in-cluster Program Manager acts on ([ADR-0025](../decisions/adr-0025-in-cluster-ai-program-manager.md)).
|
||||
|
||||
The internal API accepts reads and a single bounded write (heartbeat plus remediation states, schema
|
||||
validated). It is reachable only from the CI runner namespace, by network policy.
|
||||
|
||||
## Risk acceptance (R-VULN-08)
|
||||
|
||||
A CVE the operator has decided to live with is declared **in git**, through the merge gate, with
|
||||
an expiry:
|
||||
|
||||
```yaml
|
||||
- id: CVE-2026-00000
|
||||
image: registry/repository # exact match; omit only for a platform-wide acceptance
|
||||
reason: not reachable — feature disabled
|
||||
expires: 2027-02-01 # mandatory, at most 180 days out
|
||||
```
|
||||
|
||||
An accepted finding stays in the listing, marked `accepted` with its reason and expiry, and stops
|
||||
counting toward the critical-fix alert. **When the expiry passes it reverts to open automatically.**
|
||||
An entry with no reason or expiry is rejected by the gate.
|
||||
|
||||
## Security of the scanner itself (R-VULN-09)
|
||||
|
||||
A scanner reads every image and is a supply-chain target. Trivy's release and CI tooling were
|
||||
compromised in March 2026 ([CVE-2026-33634](https://nvd.nist.gov/vuln/detail/CVE-2026-33634)), so:
|
||||
|
||||
- The scanner binary is pinned **by digest**. It is a Renovate blast-radius component and never
|
||||
auto-merges, and an update is proposed only after a **minimum release age of 7 days**.
|
||||
- Trivy 0.69.4, 0.69.5 and 0.69.6 MUST NOT be used.
|
||||
- Identity: `get/list/watch` on pods and their owner kinds, CronJobs, nodes and VM instances, and
|
||||
**nothing else**. It MUST NOT read Secrets or hold any write verb on the Kubernetes API.
|
||||
- It holds no git credential and no registry credential.
|
||||
- Network: ingress only from the Gateway for the page, from CI for the internal API and from
|
||||
Prometheus for metrics. Egress only to the Kubernetes API, DNS, and TCP/443 on the public internet,
|
||||
with cluster and LAN ranges excluded, through the direct path.
|
||||
- It runs unprivileged, with a read-only root filesystem and no host namespaces or mounts.
|
||||
- The scanner's own images appear in the report like any other.
|
||||
|
||||
## Scope
|
||||
|
||||
**In scope:** every container image running in a pod, on every node, in every namespace, including
|
||||
the platform's own components.
|
||||
|
||||
**Not in scope for this version, and reported as `not-covered` rather than omitted:** guest OS
|
||||
packages inside KubeVirt VMs, and node OS packages and the k3s binary. These are updated by the
|
||||
mechanisms in [11](11-automated-updates.md) but are not vulnerability-assessed here.
|
||||
|
||||
Also out of scope: application dependency scanning of source, misconfiguration and secret scanning,
|
||||
runtime exploit detection, and blocking deployment based on findings.
|
||||
|
||||
## Acceptance gates
|
||||
|
||||
Each is an automated, requirement-tagged test ([14](14-testing-and-conformance.md)).
|
||||
|
||||
- **L0:** digest-pinned scanner image; no banned version; Renovate rule keeps it out of auto-merge
|
||||
with the minimum release age; RBAC contains no Secret access and no write verb; no per-CVE metric
|
||||
label; alerts unit-tested, including the absent-metric cases; the bot workflow fails on a missing
|
||||
credential; acceptance entries without reason or expiry rejected.
|
||||
- **L1 (fixtures, no cluster):** inventory resolves ReplicaSet to Deployment and Job to CronJob;
|
||||
two workloads sharing a digest produce one scan and both mappings; one failed image leaves the run
|
||||
continuing with that target `failed` and its prior findings retained; an expired acceptance reverts
|
||||
to open; the rendered page contains every fixture CVE, container and workload; a hostile title is
|
||||
escaped; the page shows a warning for a stale report; a dashboard issue rendered from a fixture
|
||||
contains no CVE id or package name; a tick is applied only to an `approve` row for a matching image
|
||||
with a qualifying finding.
|
||||
- **L2:** a newly started workload's digest appears in the report within the window; the page requires
|
||||
a `platform-admins` login; the internal API is unreachable from an unrelated namespace.
|
||||
- **L3:** breaking registry access produces `failed` targets and fires the coverage alert.
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Private registry coverage.** The platform's own images need a registry credential, so they report
|
||||
`credential-required`. Closing this requires a read-only registry token, an operator action.
|
||||
- **Exploitability ranking.** Adding CISA KEV or EPSS data, and weighting by whether a workload is
|
||||
publicly exposed, would rank the list better than CVSS alone.
|
||||
- **VM and node assessment.** How to bring `not-covered` targets into scope.
|
||||
Reference in new issue
Block a user