home-network-k3s/docs/17-vulnerability-visibility.md

# 17 — Vulnerability Visibility

Serves: **R-VULN-01..09**, surfaces **R-OBS-02/03**, **R-C2-02**, **R-UPD-01/02**, **R-SEC-02**,
**R-IDP-06**. Decision record: [ADR-0027](../decisions/adr-0027-vulnerability-visibility.md).

## Purpose

The operator must be able to answer three questions at any time, with no manual step:

1. **What is running in production?** Every container image, by digest, and the workloads using it.
2. **What is it vulnerable to?** Every known CVE in those images, and where each one lands.
3. **Can I trust the answer?** How fresh it is, and exactly what was *not* examined.

The third question is as important as the first two. A vulnerability list that silently omits an
image reads the same as a clean bill of health. Everything below treats "not scanned" as a
first-class state that is always visible, and never as zero findings.

## Shape of the solution

The workflow is deliberately the same shape as Renovate ([11](11-automated-updates.md)): scheduled,
self-hosted, run to completion, machine-authored output, a single dashboard issue, and a loud failure
when it cannot run.

```
 Kubernetes API (read-only)
        │  running pods → digests → workloads
        ▼
 ┌──────────────────────────── namespace: vuln-scanner ─────────────────────────┐
 │  Scanner (scheduled, one image at a time, Trivy)                             │
 │     └─► Report store (small PVC) ─► report.json (schema below)               │
 │             ├─► CVE page            cve.<domain>, SSO, platform-admins       │
 │             ├─► /metrics            aggregates only → Prometheus → email     │
 │             └─► internal API ◄──────────────┐                                │
 └─────────────────────────────────────────────┼────────────────────────────────┘
                                               │ read report; post heartbeat + remediation state
                              ┌────────────────┴───────────────┐
                              │ Gitea Actions: CVE triage bot  │  scheduled, before Renovate
                              └────────────────┬───────────────┘
                                               │ scoped CI token
                                               ▼
                 CVE Dashboard issue (counts only) · approval ticks on Renovate's dashboard
                                               │
                                               ▼
                 Renovate opens the PR → merge gate → Flux (the existing update path)
```

The split follows two standing boundaries: **CI holds no cluster credential**, and **the cluster
holds no git credential** ([ADR-0023](../decisions/adr-0023-automated-updates-renovate.md)). The
scanner sees the cluster and never writes to git. The bot writes to Gitea and never touches the
Kubernetes API.

## Inventory (R-VULN-01)

The inventory is taken from the **Kubernetes API**, never from git. Git states intent; the question
is what is actually running.

- It covers containers, init containers and ephemeral containers of every pod in every namespace.
- The identity of a running image is the **digest reported in the container status**. A mutable tag
  is not an identity.
- Each pod resolves through its owner chain to the **top-level workload** the operator recognises
  (Deployment, StatefulSet, DaemonSet, CronJob, or a bare Pod), not the intermediate ReplicaSet or Job.
- **CronJobs with no pod currently running are still inventoried**, from their pod template. Their
  images are recorded by reference and marked `by-tag`, because no digest exists until they run.
- KubeVirt VMs and nodes are inventoried as **non-container targets** and reported as not covered
  (see [scope](#scope)).

## Scanning (R-VULN-02, R-VULN-05)

| Property | Contract |
|---|---|
| Unit of work | One scan per **unique digest**, mapped back to all its workloads. Measured 2026-11-09: 259 containers resolve to 98 digests. |
| Full re-evaluation | Every running digest is evaluated against a current database **at least every 24 hours**, so a CVE published against an unchanged image is found without any deployment. |
| New digests | A digest not yet seen is scanned **within 2 hours** of first appearing. |
| Database freshness | A scan uses a vulnerability database no older than **48 hours**. Older than that, the run is reported as degraded, not as a success. |
| Severity scope | All severities are recorded, including those with no fix. Filtering is a page concern. |
| Failure isolation | One image failing (pull error, timeout, out of memory) records that image as `failed` with its reason and the run continues. |
| Load | Sequential, one image at a time, with CPU and memory limits. Scan caches MUST NOT live on a replicated Longhorn volume or in etcd, because etcd and Longhorn share a partition on every node ([02](02-environments-and-provisioning.md)). |
| Trigger | Schedule only. No human or AI action starts, resumes or completes a cycle. |

Findings persist a **first-seen** timestamp across runs, keyed by CVE and digest, so age is
measurable. A finding that disappears because an image was updated leaves the report at the next
inventory; one that disappears because a scan *failed* does not.

## Coverage (R-VULN-03)

Every inventoried target has exactly one state:

| State | Meaning |
|---|---|
| `scanned` | Evaluated within the 24-hour window. Zero findings is meaningful only in this state. |
| `pending` | Discovered and not yet scanned, within the 2-hour window. |
| `stale` | Last successful scan is older than the 24-hour window. Prior findings are retained and labelled stale. |
| `failed` | The most recent attempt failed. The reason is shown. Prior findings are retained. |
| `credential-required` | The image is on a registry declared as needing a credential the scanner does not hold. Shown with the registry. A declared list is used because the private registry sits on the LAN, which the scanner's egress excludes, so it would otherwise surface as a network failure. |
| `by-tag` | Scanned by mutable reference, because no running digest exists. |
| `not-covered` | A target class this scanner cannot examine: VMs and nodes. |

The page and the report MUST NOT present a total or a "clean" indication that silently excludes any
target. The header of the page always states how many targets are in each state.

## The report contract

`report.json` is the one interface between the scanner, the page and the bot. It is versioned and
the implementation validates it against a schema.

```yaml
schemaVersion: 1
generatedAt: <RFC 3339>
database: { updatedAt: <RFC 3339>, version: <string> }
run: { startedAt: <RFC 3339>, finishedAt: <RFC 3339>, status: complete | degraded }
targets:
  - id: <stable identifier>
    type: image | vm | node
    image: <registry/repository>
    digest: <sha256:… or null>
    state: <one of the coverage states>
    reason: <string or null>
    lastScannedAt: <RFC 3339 or null>
    workloads:
      - { namespace: <ns>, kind: <kind>, name: <name>, containers: [<name>] }
findings:
  - id: <CVE or advisory id>
    targetId: <targets[].id>
    severity: CRITICAL | HIGH | MEDIUM | LOW | UNKNOWN
    cvss: <number or null>
    package: <name>
    installedVersion: <string>
    fixedVersion: <string or null>
    title: <string>
    url: <advisory URL>
    firstSeenAt: <RFC 3339>
    status: open | fix-available | accepted
    acceptance: { reason: <string>, expires: <date> }   # present only when accepted
remediation:                                            # written by the triage bot, may be empty
  - { image: <registry/repository>, state: <state>, url: <PR URL or null>, updatedAt: <RFC 3339> }
```

Scanner-derived text (titles, package names, advisory URLs) originates upstream and is **untrusted**.
The page MUST escape it and serve a restrictive Content-Security-Policy, and only `http(s)` URLs may
become links.

## The CVE page (R-VULN-04)

A page at `cve.<domain>`, linked from the C2 landing page.

**Header, always visible:** report age, database age, per-state target counts, and open findings by
severity. If the report is stale or degraded, the page says so prominently and does not show a
normal state.

**CVEs view (default):** one row per CVE, with id (linking to the advisory), severity, CVSS,
package, fixed version, first seen, status, and the counts of affected images and workloads. Each
row expands to list **every image, container and workload** it touches, with namespace.

**Workloads view:** one row per workload, with its containers, images and per-severity counts,
expanding to its CVEs.

**Inventory view:** every target and its coverage state, *including those with zero findings*. This
is the answer to "what am I running".

**Interaction:** free-text search, and filters for severity, namespace, workload, status and "fix
available", with sortable columns. Filter state is in the URL so a view can be linked. The
default view excludes nothing.

**Remediation column:** for images with a fix available, one of `pr-open` (with link),
`approval-requested`, `no-update-identified` or `not-tracked`, taken from the bot.

**Access:** internal exposure only, behind SSO, restricted to `platform-admins` (R-IDP-06). A list of
what is exploitable where is sensitive, so this is not a "logged-in user" surface. The page is
read-only and offers no actions.

## Alerting (R-VULN-06)

Alerts are delivered through the existing Alertmanager email route. Each alert is **one grouped
message per condition**, never one per CVE.

| Alert | Fires when |
|---|---|
| Scan stale | No complete run within 26 hours, **or the metric is absent** |
| Database stale | Vulnerability database older than 48 hours |
| Coverage gap | Any target in `failed` for more than 24 hours |
| Critical fix available | Any unaccepted CRITICAL with a fix, first seen more than 72 hours ago |
| Triage bot silent | No bot heartbeat for 26 hours, **or the metric is absent** |

The 72-hour grace gives Renovate's daily cycle time to act, so email marks the cases automation has
not resolved. The critical-fix alert is expected to fire on first deployment.

Prometheus receives **aggregates only**: counts by severity and fixability, per-state target counts,
timestamps. It MUST NOT receive a per-CVE series. Measured 2026-11-09: Prometheus holds 427k head
series at about 2.75 GiB of a 3 GiB limit.

## The triage bot (R-VULN-05, R-VULN-07)

A scheduled Gitea Actions workflow that follows the conventions of the Renovate workflow: it fails
hard on a missing credential rather than skipping, has a `workflow_dispatch` dry run, never runs
concurrently with itself, and records what it did in the job summary. It runs **before** the daily
Renovate run so that approvals take effect the same morning.

Each run it:

1. Reads `report.json` over the internal API.
2. Creates or updates a single **CVE Dashboard** issue on the implementation repository.
3. Reads Renovate's Dependency Dashboard and open PRs, and derives each image's remediation state.
4. Ticks the **approval** checkbox on Renovate's dashboard for a pending update whose package matches
   an image with an unaccepted CRITICAL that has a fix.
5. Posts its heartbeat and the remediation states back to the scanner.

**Renovate remains the only author of version changes** (R-UPD-02, ADR-0023). The bot never edits a
version, opens a bump PR or merges anything. A tick only asks Renovate to *open* a PR; majors and
blast-radius components still wait for human review, exactly as today.

**The dashboard issue carries aggregates only:** total findings by severity, per-state coverage,
timestamps and a link to the page. It MUST NOT name a CVE, package or image-to-workload relationship,
because the repository is readable without authentication (measured 2026-11-09). It MUST NOT carry any
label that the in-cluster Program Manager acts on ([ADR-0025](../decisions/adr-0025-in-cluster-ai-program-manager.md)).

The internal API accepts reads and a single bounded write (heartbeat plus remediation states, schema
validated). It is reachable only from the CI runner namespace, by network policy.

## Risk acceptance (R-VULN-08)

A CVE the operator has decided to live with is declared **in git**, through the merge gate, with
an expiry:

```yaml
- id: CVE-2026-00000
  image: registry/repository          # exact match; omit only for a platform-wide acceptance
  reason: not reachable — feature disabled
  expires: 2027-02-01                 # mandatory, at most 180 days out
```

An accepted finding stays in the listing, marked `accepted` with its reason and expiry, and stops
counting toward the critical-fix alert. **When the expiry passes it reverts to open automatically.**
An entry with no reason or expiry is rejected by the gate.

## Security of the scanner itself (R-VULN-09)

A scanner reads every image and is a supply-chain target. Trivy's release and CI tooling were
compromised in March 2026 ([CVE-2026-33634](https://nvd.nist.gov/vuln/detail/CVE-2026-33634)), so:

- The scanner binary is pinned **by digest**. It is a Renovate blast-radius component and never
  auto-merges, and an update is proposed only after a **minimum release age of 7 days**.
- Trivy 0.69.4, 0.69.5 and 0.69.6 MUST NOT be used.
- Identity: `get/list/watch` on pods and their owner kinds, CronJobs, nodes and VM instances, and
  **nothing else**. It MUST NOT read Secrets or hold any write verb on the Kubernetes API.
- It holds no git credential and no registry credential.
- Network: ingress only from the Gateway for the page, from CI for the internal API and from
  Prometheus for metrics. Egress only to the Kubernetes API, DNS, the platform Gateway and identity provider (required by the
  SSO sidecar), and TCP/443 on the public internet, with cluster and LAN ranges otherwise excluded,
  through the direct path.
- It runs unprivileged, with a read-only root filesystem and no host namespaces or mounts.
- The scanner's own images appear in the report like any other.

## Scope

**In scope:** every container image running in a pod, on every node, in every namespace, including
the platform's own components.

**Not in scope for this version, and reported as `not-covered` rather than omitted:** guest OS
packages inside KubeVirt VMs, and node OS packages and the k3s binary. These are updated by the
mechanisms in [11](11-automated-updates.md) but are not vulnerability-assessed here.

Also out of scope: application dependency scanning of source, misconfiguration and secret scanning,
runtime exploit detection, and blocking deployment based on findings.

## Acceptance gates

Each is an automated, requirement-tagged test ([14](14-testing-and-conformance.md)).

- **L0:** digest-pinned scanner image; no banned version; Renovate rule keeps it out of auto-merge
  with the minimum release age; RBAC contains no Secret access and no write verb; no per-CVE metric
  label; alerts unit-tested, including the absent-metric cases; the bot workflow fails on a missing
  credential; acceptance entries without reason or expiry rejected.
- **L1 (fixtures, no cluster):** inventory resolves ReplicaSet to Deployment and Job to CronJob;
  two workloads sharing a digest produce one scan and both mappings; one failed image leaves the run
  continuing with that target `failed` and its prior findings retained; an expired acceptance reverts
  to open; the rendered page contains every fixture CVE, container and workload; a hostile title is
  escaped; the page shows a warning for a stale report; a dashboard issue rendered from a fixture
  contains no CVE id or package name; a tick is applied only to an `approve` row for a matching image
  with a qualifying finding.
- **L2:** a newly started workload's digest appears in the report within the window; the page requires
  a `platform-admins` login; the internal API is unreachable from an unrelated namespace.
- **L3:** breaking registry access produces `failed` targets and fires the coverage alert.

## Open questions

- **Private registry coverage.** The platform's own images need a registry credential, so they report
  `credential-required`. Closing this requires a read-only registry token, an operator action.
- **Exploitability ranking.** Adding CISA KEV or EPSS data, and weighting by whether a workload is
  publicly exposed, would rank the list better than CVSS alone.
- **VM and node assessment.** How to bring `not-covered` targets into scope.