home-network-k3s-impl/runbooks/vuln-scanner.md

vuln-scanner — the CVE page and the five scan alerts

Alerts that link here: VulnScanStale, VulnDatabaseStale, VulnCoverageGap, VulnCriticalFixAvailable and VulnTriageBotSilent.

Component: platform/vuln-scanner/ (Flux Kustomization vuln-scanner, wait: false, nothing depends on it). Design record: spec docs/17-vulnerability-visibility.md, ADR-0027. The page is https://cve.<domain>, internal only, behind SSO, restricted to platform-admins.

Every mutating step below is an operator step (CLAUDE.md §5).

What this means here

The scanner runs on a schedule and needs nobody to start, resume or complete a cycle. The bot is a scheduled Gitea Actions workflow (.gitea/workflows/cve-triage.yaml). Each alert is one grouped email for one condition and never names a CVE, image or workload; the page has the detail.

Alert	Fires when	What it means
VulnScanStale	No complete run for 26 hours, or the metric is absent	Scanning has stopped, the scanner is down, or its scrape is broken. It fires from first deployment until the first complete run (a full run covers about 100 images, sequentially).
VulnDatabaseStale	Trivy DB older than 48 hours	trivy --download-db-only is failing; the run is degraded, not a success.
VulnCoverageGap	Any target failed for more than 24 hours	A registry unreachable or a scan keeps failing. credential-required targets do not count.
VulnCriticalFixAvailable	Unaccepted CRITICAL with a fix, first seen more than 72 hours ago	The backlog automation has not resolved. Expected on first deployment.
VulnTriageBotSilent	No heartbeat for 26 hours, or the metric is absent	The bot has not reported, so remediation states and Renovate approvals are stale.

A failing scanner is contained by design: its Kustomization is wait: false and nothing depends on it, so none of these alerts mean anything else on the platform is affected.

First commands

All cluster access is read-only and goes through the pinned toolbox, always naming the context:

./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n vuln-scanner get pods -o wide'
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n vuln-scanner logs deploy/vuln-scanner -c scanner --tail=100'
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n vuln-scanner exec deploy/vuln-scanner -c scanner -- python3 -c "import urllib.request as u; print(u.urlopen(\"http://127.0.0.1:9090/metrics\").read().decode())"'

The metrics are the quickest read: vuln_scanner_targets{state} gives the per-state counts, and vuln_scanner_last_complete_run_timestamp_seconds the last complete run. Then open https://cve.<domain>, Inventory view, and sort by state to see which targets, and why.

For the bot, read the most recent run of the CVE triage workflow in Gitea Actions and its job summary.

Likely causes, most likely first

VulnScanStale — (1) first deployment: wait for the first complete run; (2) pod not Running: see the Pod's events and the longhorn PVC attachment; (3) metrics not scraped: up{namespace="vuln-scanner"} and platform/vuln-scanner/networkpolicy.yaml's Prometheus rule; (4) the scanner logged a run failure.

VulnDatabaseStale — egress to ghcr.io:443 is blocked (the egress CIDR rule, or the registry is down) or GitHub's rate limit; the log has the trivy error.

VulnCoverageGap — a registry is down or a tag was deleted. The page shows each failed target's reason. A target whose registry refuses anonymous pulls is credential-required and does not fire this.

VulnCriticalFixAvailable — expected once on first deployment. Otherwise a Renovate update is not landing: see the bot's dashboard issue and Renovate's dashboard. To accept a risk instead, add an entry to platform/vuln-scanner/acceptances.yaml (reason plus an expiry at most 180 days out) through a normal PR.

VulnTriageBotSilent — (1) RENOVATE_TOKEN is missing or the workflow is disabled; (2) the bot cannot reach the scanner's internal API (gitea-actions namespace -> vuln-scanner:8081, networkpolicy.yaml; the bot falls back to the pinned ClusterIP when in-cluster DNS is unavailable to a DinD job container); (3) the scanner is down too: VulnScanStale will also be firing.

When to escalate or stop

• If the pod is crash-looping, read the scanner's logs before changing anything; a crash loop here is contained and is not urgent for the rest of the platform.
• Do not weaken networkpolicy.yaml or rbac.yaml to make a scan pass. The scanner must never read Secrets, hold a write verb, or reach the LAN. Those are the requirement (R-VULN-09), not an oversight.
• Do not edit platform/vuln-scanner/alerts.yaml thresholds to silence an alert; the spec fixes them.
• A persistent VulnCriticalFixAvailable for a component that cannot move is what the risk-acceptance file is for, not silencing.