feat(vuln): vuln-scanner platform manifests, Flux tier, alerts, L0/L2/L3 tests
Hand-authored platform/vuln-scanner (non-gating Flux tier, digest-pinned Trivy 0.76.1 and python 3.14.9-alpine, read-only RBAC, per-pod oauth2-proxy, internal-only cve.<domain>), five alerts with promtool tests, R-VULN-01..09 registry entries, and vuln-* L0 checks with mutation self-tests. src/ and the bot workflow are integrated separately. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
1 parent
4c3a284cad
commit
b85d1e3ed7
47 files changed
+2572
-6
No files matched your search
@@ -87,6 +87,8 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
|
||||
## The runbooks
|
||||
|
||||
### Nodes
|
||||
- **`vuln-scanner`** — the CVE page and the five scan alerts (`VulnScanStale`, `VulnDatabaseStale`,
|
||||
`VulnCoverageGap`, `VulnCriticalFixAvailable`, `VulnTriageBotSilent`), and what each one means here.
|
||||
- **`node-down`** — a node's node-exporter is unscrapable; the node may be gone, or its kubelet may have
|
||||
stopped posting status while the host stays up (the 2026-09-07 shape).
|
||||
- **`node-io`** — disk I/O pressure/starvation. The etcd WAL shares a partition with Longhorn replicas;
|
||||
|
||||
@@ -49,6 +49,7 @@ configMapGenerator:
|
||||
files:
|
||||
# The index, served at /runbooks/ (nginx `try_files /runbooks/README`).
|
||||
- README=README.md
|
||||
- vuln-scanner=vuln-scanner.md
|
||||
- backup-failure=backup-failure.md
|
||||
- backup-missed=backup-missed.md
|
||||
- backup-partial=backup-partial.md
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
# vuln-scanner — the CVE page and the five scan alerts
|
||||
|
||||
Alerts that link here: **`VulnScanStale`**, **`VulnDatabaseStale`**, **`VulnCoverageGap`**,
|
||||
**`VulnCriticalFixAvailable`** and **`VulnTriageBotSilent`**.
|
||||
|
||||
Component: `platform/vuln-scanner/` (Flux Kustomization `vuln-scanner`, `wait: false`, nothing depends on
|
||||
it). Design record: spec `docs/17-vulnerability-visibility.md`, ADR-0027. The page is
|
||||
`https://cve.<domain>`, internal only, behind SSO, restricted to `platform-admins`.
|
||||
|
||||
Every mutating step below is an **operator** step (CLAUDE.md §5).
|
||||
|
||||
## What this means here
|
||||
|
||||
The scanner runs on a schedule and needs nobody to start, resume or complete a cycle. The bot is a
|
||||
scheduled Gitea Actions workflow (`.gitea/workflows/cve-triage.yaml`). Each alert is one grouped email
|
||||
for one condition and never names a CVE, image or workload; the page has the detail.
|
||||
|
||||
| Alert | Fires when | What it means |
|
||||
|---|---|---|
|
||||
| `VulnScanStale` | No complete run for 26 hours, or the metric is absent | Scanning has stopped, the scanner is down, or its scrape is broken. **It fires from first deployment until the first complete run** (a full run covers about 100 images, sequentially). |
|
||||
| `VulnDatabaseStale` | Trivy DB older than 48 hours | `trivy --download-db-only` is failing; the run is degraded, not a success. |
|
||||
| `VulnCoverageGap` | Any target `failed` for more than 24 hours | A registry unreachable or a scan keeps failing. `credential-required` targets do not count. |
|
||||
| `VulnCriticalFixAvailable` | Unaccepted CRITICAL with a fix, first seen more than 72 hours ago | The backlog automation has not resolved. **Expected on first deployment.** |
|
||||
| `VulnTriageBotSilent` | No heartbeat for 26 hours, or the metric is absent | The bot has not reported, so remediation states and Renovate approvals are stale. |
|
||||
|
||||
A failing scanner is contained by design: its Kustomization is `wait: false` and nothing depends on it, so
|
||||
none of these alerts mean anything else on the platform is affected.
|
||||
|
||||
## First commands
|
||||
|
||||
All cluster access is read-only and goes through the pinned toolbox, always naming the context:
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n vuln-scanner get pods -o wide'
|
||||
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n vuln-scanner logs deploy/vuln-scanner -c scanner --tail=100'
|
||||
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n vuln-scanner exec deploy/vuln-scanner -c scanner -- python3 -c "import urllib.request as u; print(u.urlopen(\"http://127.0.0.1:9090/metrics\").read().decode())"'
|
||||
```
|
||||
|
||||
The metrics are the quickest read: `vuln_scanner_targets{state}` gives the per-state counts, and
|
||||
`vuln_scanner_last_complete_run_timestamp_seconds` the last complete run. Then open
|
||||
`https://cve.<domain>`, Inventory view, and sort by state to see which targets, and why.
|
||||
|
||||
For the bot, read the most recent run of the `CVE triage` workflow in Gitea Actions and its job
|
||||
summary.
|
||||
|
||||
## Likely causes, most likely first
|
||||
|
||||
**`VulnScanStale`** — (1) first deployment: wait for the first complete run; (2) pod not Running: see the
|
||||
Pod's events and the `longhorn` PVC attachment; (3) metrics not scraped: `up{namespace="vuln-scanner"}`
|
||||
and `platform/vuln-scanner/networkpolicy.yaml`'s Prometheus rule; (4) the scanner logged a run failure.
|
||||
|
||||
**`VulnDatabaseStale`** — egress to `ghcr.io:443` is blocked (the egress CIDR rule, or the registry is
|
||||
down) or GitHub's rate limit; the log has the `trivy` error.
|
||||
|
||||
**`VulnCoverageGap`** — a registry is down or a tag was deleted. The page shows each failed target's reason.
|
||||
A target whose registry refuses anonymous pulls is `credential-required` and does not fire this.
|
||||
|
||||
**`VulnCriticalFixAvailable`** — expected once on first deployment. Otherwise a Renovate update is not
|
||||
landing: see the bot's dashboard issue and Renovate's dashboard. To accept a risk instead, add an entry to
|
||||
`platform/vuln-scanner/acceptances.yaml` (reason plus an expiry at most 180 days out) through a normal PR.
|
||||
|
||||
**`VulnTriageBotSilent`** — (1) `RENOVATE_TOKEN` is missing or the workflow is disabled; (2) the bot
|
||||
cannot reach the scanner's internal API (`gitea-actions` namespace -> `vuln-scanner:8081`,
|
||||
`networkpolicy.yaml`; the bot falls back to the pinned ClusterIP when in-cluster DNS is unavailable to a
|
||||
DinD job container); (3) the scanner is down too: `VulnScanStale` will also be firing.
|
||||
|
||||
## When to escalate or stop
|
||||
|
||||
- If the pod is crash-looping, read the scanner's logs before changing anything; a crash loop here is
|
||||
contained and is not urgent for the rest of the platform.
|
||||
- Do **not** weaken `networkpolicy.yaml` or `rbac.yaml` to make a scan pass. The scanner must never read
|
||||
Secrets, hold a write verb, or reach the LAN. Those are the requirement (R-VULN-09), not an oversight.
|
||||
- Do not edit `platform/vuln-scanner/alerts.yaml` thresholds to silence an alert; the spec fixes them.
|
||||
- A persistent `VulnCriticalFixAvailable` for a component that cannot move is what the risk-acceptance file
|
||||
is for, not silencing.
|
||||
Reference in new issue
Block a user