home-network-k3s/docs/17-vulnerability-visibility.md

17 — Vulnerability Visibility

Serves: R-VULN-01..09, surfaces R-OBS-02/03, R-C2-02, R-UPD-01/02, R-SEC-02, R-IDP-06. Decision record: ADR-0027.

Purpose

The operator must be able to answer three questions at any time, with no manual step:

1. What is running in production? Every container image, by digest, and the workloads using it.
2. What is it vulnerable to? Every known CVE in those images, and where each one lands.
3. Can I trust the answer? How fresh it is, and exactly what was not examined.

The third question is as important as the first two. A vulnerability list that silently omits an image reads the same as a clean bill of health. Everything below treats "not scanned" as a first-class state that is always visible, and never as zero findings.

Shape of the solution

The workflow is deliberately the same shape as Renovate (11): scheduled, self-hosted, run to completion, machine-authored output, a single dashboard issue, and a loud failure when it cannot run.

 Kubernetes API (read-only)
        │  running pods → digests → workloads
        ▼
 ┌──────────────────────────── namespace: vuln-scanner ─────────────────────────┐
 │  Scanner (scheduled, one image at a time, Trivy)                             │
 │     └─► Report store (small PVC) ─► report.json (schema below)               │
 │             ├─► CVE page            cve.<domain>, SSO, platform-admins       │
 │             ├─► /metrics            aggregates only → Prometheus → email     │
 │             └─► internal API ◄──────────────┐                                │
 └─────────────────────────────────────────────┼────────────────────────────────┘
                                               │ read report; post heartbeat + remediation state
                              ┌────────────────┴───────────────┐
                              │ Gitea Actions: CVE triage bot  │  scheduled, before Renovate
                              └────────────────┬───────────────┘
                                               │ scoped CI token
                                               ▼
                 CVE Dashboard issue (counts only) · approval ticks on Renovate's dashboard
                                               │
                                               ▼
                 Renovate opens the PR → merge gate → Flux (the existing update path)

The split follows two standing boundaries: CI holds no cluster credential, and the cluster holds no git credential (ADR-0023). The scanner sees the cluster and never writes to git. The bot writes to Gitea and never touches the Kubernetes API.

Inventory (R-VULN-01)

The inventory is taken from the Kubernetes API, never from git. Git states intent; the question is what is actually running.

• It covers containers, init containers and ephemeral containers of every pod in every namespace.
• The identity of a running image is the digest reported in the container status. A mutable tag is not an identity.
• Each pod resolves through its owner chain to the top-level workload the operator recognises (Deployment, StatefulSet, DaemonSet, CronJob, or a bare Pod), not the intermediate ReplicaSet or Job.
• CronJobs with no pod currently running are still inventoried, from their pod template. Their images are recorded by reference and marked by-tag, because no digest exists until they run.
• KubeVirt VMs and nodes are inventoried as non-container targets and reported as not covered (see scope).

Scanning (R-VULN-02, R-VULN-05)

Property	Contract
Unit of work	One scan per unique digest, mapped back to all its workloads. Measured 2026-11-09: 259 containers resolve to 98 digests.
Full re-evaluation	Every running digest is evaluated against a current database at least every 24 hours, so a CVE published against an unchanged image is found without any deployment.
New digests	A digest not yet seen is scanned within 2 hours of first appearing.
Database freshness	A scan uses a vulnerability database no older than 48 hours. Older than that, the run is reported as degraded, not as a success.
Severity scope	All severities are recorded, including those with no fix. Filtering is a page concern.
Failure isolation	One image failing (pull error, timeout, out of memory) records that image as failed with its reason and the run continues.
Load	Sequential, one image at a time, with CPU and memory limits. Scan caches MUST NOT live on a replicated Longhorn volume or in etcd, because etcd and Longhorn share a partition on every node (02).
Trigger	Schedule only. No human or AI action starts, resumes or completes a cycle.

Findings persist a first-seen timestamp across runs, keyed by CVE and digest, so age is measurable. A finding that disappears because an image was updated leaves the report at the next inventory; one that disappears because a scan failed does not.

Coverage (R-VULN-03)

Every inventoried target has exactly one state:

State	Meaning
scanned	Evaluated within the 24-hour window. Zero findings is meaningful only in this state.
pending	Discovered and not yet scanned, within the 2-hour window.
stale	Last successful scan is older than the 24-hour window. Prior findings are retained and labelled stale.
failed	The most recent attempt failed. The reason is shown. Prior findings are retained.
credential-required	The registry refused anonymous access. Shown with the registry.
by-tag	Scanned by mutable reference, because no running digest exists.
not-covered	A target class this scanner cannot examine: VMs and nodes.

The page and the report MUST NOT present a total or a "clean" indication that silently excludes any target. The header of the page always states how many targets are in each state.

The report contract

report.json is the one interface between the scanner, the page and the bot. It is versioned and the implementation validates it against a schema.

schemaVersion: 1
generatedAt: <RFC 3339>
database: { updatedAt: <RFC 3339>, version: <string> }
run: { startedAt: <RFC 3339>, finishedAt: <RFC 3339>, status: complete | degraded }
targets:
  - id: <stable identifier>
    type: image | vm | node
    image: <registry/repository>
    digest: <sha256:… or null>
    state: <one of the coverage states>
    reason: <string or null>
    lastScannedAt: <RFC 3339 or null>
    workloads:
      - { namespace: <ns>, kind: <kind>, name: <name>, containers: [<name>] }
findings:
  - id: <CVE or advisory id>
    targetId: <targets[].id>
    severity: CRITICAL | HIGH | MEDIUM | LOW | UNKNOWN
    cvss: <number or null>
    package: <name>
    installedVersion: <string>
    fixedVersion: <string or null>
    title: <string>
    url: <advisory URL>
    firstSeenAt: <RFC 3339>
    status: open | fix-available | accepted
    acceptance: { reason: <string>, expires: <date> }   # present only when accepted
remediation:                                            # written by the triage bot, may be empty
  - { image: <registry/repository>, state: <state>, url: <PR URL or null>, updatedAt: <RFC 3339> }

Scanner-derived text (titles, package names, advisory URLs) originates upstream and is untrusted. The page MUST escape it and serve a restrictive Content-Security-Policy, and only http(s) URLs may become links.

The CVE page (R-VULN-04)

A page at cve.<domain>, linked from the C2 landing page.

Header, always visible: report age, database age, per-state target counts, and open findings by severity. If the report is stale or degraded, the page says so prominently and does not show a normal state.

CVEs view (default): one row per CVE, with id (linking to the advisory), severity, CVSS, package, fixed version, first seen, status, and the counts of affected images and workloads. Each row expands to list every image, container and workload it touches, with namespace.

Workloads view: one row per workload, with its containers, images and per-severity counts, expanding to its CVEs.

Inventory view: every target and its coverage state, including those with zero findings. This is the answer to "what am I running".

Interaction: free-text search, and filters for severity, namespace, workload, status and "fix available", with sortable columns. Filter state is in the URL so a view can be linked. The default view excludes nothing.

Remediation column: for images with a fix available, one of pr-open (with link), approval-requested, no-update-identified or not-tracked, taken from the bot.

Access: internal exposure only, behind SSO, restricted to platform-admins (R-IDP-06). A list of what is exploitable where is sensitive, so this is not a "logged-in user" surface. The page is read-only and offers no actions.

Alerting (R-VULN-06)

Alerts are delivered through the existing Alertmanager email route. Each alert is one grouped message per condition, never one per CVE.

Alert	Fires when
Scan stale	No complete run within 26 hours, or the metric is absent
Database stale	Vulnerability database older than 48 hours
Coverage gap	Any target in failed for more than 24 hours
Critical fix available	Any unaccepted CRITICAL with a fix, first seen more than 72 hours ago
Triage bot silent	No bot heartbeat for 26 hours, or the metric is absent

The 72-hour grace gives Renovate's daily cycle time to act, so email marks the cases automation has not resolved. The critical-fix alert is expected to fire on first deployment.

Prometheus receives aggregates only: counts by severity and fixability, per-state target counts, timestamps. It MUST NOT receive a per-CVE series. Measured 2026-11-09: Prometheus holds 427k head series at about 2.75 GiB of a 3 GiB limit.

The triage bot (R-VULN-05, R-VULN-07)

A scheduled Gitea Actions workflow that follows the conventions of the Renovate workflow: it fails hard on a missing credential rather than skipping, has a workflow_dispatch dry run, never runs concurrently with itself, and records what it did in the job summary. It runs before the daily Renovate run so that approvals take effect the same morning.

Each run it:

1. Reads report.json over the internal API.
2. Creates or updates a single CVE Dashboard issue on the implementation repository.
3. Reads Renovate's Dependency Dashboard and open PRs, and derives each image's remediation state.
4. Ticks the approval checkbox on Renovate's dashboard for a pending update whose package matches an image with an unaccepted CRITICAL that has a fix.
5. Posts its heartbeat and the remediation states back to the scanner.

Renovate remains the only author of version changes (R-UPD-02, ADR-0023). The bot never edits a version, opens a bump PR or merges anything. A tick only asks Renovate to open a PR; majors and blast-radius components still wait for human review, exactly as today.

The dashboard issue carries aggregates only: total findings by severity, per-state coverage, timestamps and a link to the page. It MUST NOT name a CVE, package or image-to-workload relationship, because the repository is readable without authentication (measured 2026-11-09). It MUST NOT carry any label that the in-cluster Program Manager acts on (ADR-0025).

The internal API accepts reads and a single bounded write (heartbeat plus remediation states, schema validated). It is reachable only from the CI runner namespace, by network policy.

Risk acceptance (R-VULN-08)

A CVE the operator has decided to live with is declared in git, through the merge gate, with an expiry:

- id: CVE-2026-00000
  image: registry/repository          # exact match; omit only for a platform-wide acceptance
  reason: not reachable — feature disabled
  expires: 2027-02-01                 # mandatory, at most 180 days out

An accepted finding stays in the listing, marked accepted with its reason and expiry, and stops counting toward the critical-fix alert. When the expiry passes it reverts to open automatically. An entry with no reason or expiry is rejected by the gate.

Security of the scanner itself (R-VULN-09)

A scanner reads every image and is a supply-chain target. Trivy's release and CI tooling were compromised in March 2026 (CVE-2026-33634), so:

• The scanner binary is pinned by digest. It is a Renovate blast-radius component and never auto-merges, and an update is proposed only after a minimum release age of 7 days.
• Trivy 0.69.4, 0.69.5 and 0.69.6 MUST NOT be used.
• Identity: get/list/watch on pods and their owner kinds, CronJobs, nodes and VM instances, and nothing else. It MUST NOT read Secrets or hold any write verb on the Kubernetes API.
• It holds no git credential and no registry credential.
• Network: ingress only from the Gateway for the page, from CI for the internal API and from Prometheus for metrics. Egress only to the Kubernetes API, DNS, and TCP/443 on the public internet, with cluster and LAN ranges excluded, through the direct path.
• It runs unprivileged, with a read-only root filesystem and no host namespaces or mounts.
• The scanner's own images appear in the report like any other.

Scope

In scope: every container image running in a pod, on every node, in every namespace, including the platform's own components.

Not in scope for this version, and reported as not-covered rather than omitted: guest OS packages inside KubeVirt VMs, and node OS packages and the k3s binary. These are updated by the mechanisms in 11 but are not vulnerability-assessed here.

Also out of scope: application dependency scanning of source, misconfiguration and secret scanning, runtime exploit detection, and blocking deployment based on findings.

Acceptance gates

Each is an automated, requirement-tagged test (14).

• L0: digest-pinned scanner image; no banned version; Renovate rule keeps it out of auto-merge with the minimum release age; RBAC contains no Secret access and no write verb; no per-CVE metric label; alerts unit-tested, including the absent-metric cases; the bot workflow fails on a missing credential; acceptance entries without reason or expiry rejected.
• L1 (fixtures, no cluster): inventory resolves ReplicaSet to Deployment and Job to CronJob; two workloads sharing a digest produce one scan and both mappings; one failed image leaves the run continuing with that target failed and its prior findings retained; an expired acceptance reverts to open; the rendered page contains every fixture CVE, container and workload; a hostile title is escaped; the page shows a warning for a stale report; a dashboard issue rendered from a fixture contains no CVE id or package name; a tick is applied only to an approve row for a matching image with a qualifying finding.
• L2: a newly started workload's digest appears in the report within the window; the page requires a platform-admins login; the internal API is unreachable from an unrelated namespace.
• L3: breaking registry access produces failed targets and fires the coverage alert.

Open questions

• Private registry coverage. The platform's own images need a registry credential, so they report credential-required. Closing this requires a read-only registry token, an operator action.
• Exploitability ranking. Adding CISA KEV or EPSS data, and weighting by whether a workload is publicly exposed, would rank the list better than CVSS alone.
• VM and node assessment. How to bring not-covered targets into scope.