Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
  jakman85andClaude Opus 5.5
|
18f7ba8d0f |
No files matched your search
@@ -67,11 +67,6 @@ two things (e.g. "VM"), this file disambiguates.
|
||||
- **DR drill** — a tested restore of resources + volumes proving R-STATE-03.
|
||||
|
||||
## Delivery & lifecycle
|
||||
- **Coverage state** — whether a target was actually evaluated for vulnerabilities (`scanned`,
|
||||
`pending`, `stale`, `failed`, `credential-required`, `by-tag`, `not-covered`). Zero findings means
|
||||
something only in `scanned`. See [`docs/17-vulnerability-visibility.md`](docs/17-vulnerability-visibility.md).
|
||||
- **CVE triage bot** — the scheduled CI workflow that summarises the vulnerability report and asks
|
||||
Renovate to open pending updates; it never authors a version change.
|
||||
- **GitOps** — desired state lives in git and is reconciled into the cluster automatically
|
||||
(pull-based). Engine: **Flux CD**.
|
||||
- **Drift correction** — Flux reverting live changes back to git-declared state (R-GITOPS-03).
|
||||
|
||||
@@ -81,8 +81,7 @@ home-network-k3s/
|
||||
│ ├── 13-service-catalog.md
|
||||
│ ├── 14-testing-and-conformance.md
|
||||
│ ├── 15-identity-and-access.md
|
||||
│ ├── 16-known-gaps.md
|
||||
│ └── 17-vulnerability-visibility.md
|
||||
│ └── 16-known-gaps.md
|
||||
├── decisions/ ← ADRs (one decision per file) + index/template
|
||||
├── roadmap/ ← phases.md, workstreams.md
|
||||
└── agents/ ← roles, coordination protocol, DoD, task board
|
||||
|
||||
+3
-3
@@ -59,8 +59,8 @@ pattern choices.
|
||||
| [0007](adr-0007-secrets-sops-age.md) | Secrets: SOPS + age | Accepted | R-SEC-01 |
|
||||
| [0008](adr-0008-vpn-egress-model.md) | VPN egress model: Gluetun sidecar + Cilium Egress Gateway | ~~Superseded by 0012~~ | R-NET-05/06 |
|
||||
| [0009](adr-0009-automated-image-updates.md) | Automated updates: Flux image automation + Renovate | ~~Superseded by 0023~~ | R-UPD-01..03 |
|
||||
| [0010](adr-0010-conformance-testing.md) | Conformance testing: automated, reproducible, behavior-based | Accepted | R-TEST-01..06 |
|
||||
| [0011](adr-0011-test-tooling-kyverno-chainsaw.md) | Conformance test tooling: Kyverno + Chainsaw | Accepted | R-TEST-01..05 |
|
||||
| [0010](adr-0010-conformance-testing.md) | Conformance testing: automated, reproducible, behavior-based | Accepted — L3 merge gating superseded by 0026 | R-TEST-01..06 |
|
||||
| [0011](adr-0011-test-tooling-kyverno-chainsaw.md) | Conformance test tooling: Kyverno + Chainsaw | Accepted — L3 CI trigger superseded by 0026 | R-TEST-01..05 |
|
||||
| [0012](adr-0012-vpn-egress-single-expressvpn-gateway.md) | VPN egress: single shared in-cluster ExpressVPN gateway | ~~Superseded by 0013~~ | R-NET-05/06/07 |
|
||||
| [0013](adr-0013-vpn-provider-pia-gluetun-wireguard.md) | VPN provider: PIA over OpenVPN-UDP + port forwarding | Accepted — provider/protocol only (client+steering superseded by 0015) | R-NET-05/06/07 |
|
||||
| [0014](adr-0014-idp-keycloak.md) | Identity Provider: Keycloak (SSO via OIDC + forward-auth) | Accepted — forward-auth enforcement point and app list superseded by 0022 | R-IDP-01..06 |
|
||||
@@ -75,4 +75,4 @@ pattern choices.
|
||||
| [0023](adr-0023-automated-updates-renovate.md) | Automated updates: Renovate for images and charts, every change PR-gated | Accepted | R-UPD-01, R-UPD-02, R-UPD-03, R-TEST-03 |
|
||||
| [0024](adr-0024-delivery-authority-ci-gate.md) | Delivery authority: the CI gate is the approver; sensitive-path review; attributed admin override | Accepted | R-TEST-03, R-TEST-06, R-GITOPS-01, R-SEC-02 |
|
||||
| [0025](adr-0025-in-cluster-ai-program-manager.md) | In-cluster AI Program Manager (pm-agent) | Accepted | R-GITOPS-01, R-TEST-03, R-TEST-06, R-SEC-02 |
|
||||
| [0027](adr-0027-vulnerability-visibility.md) | Vulnerability visibility: scheduled digest scanning with a Renovate-shaped bot | Accepted | R-VULN-01..09 |
|
||||
| [0026](adr-0026-fault-injection-operator-run.md) | Fault injection is operator-run, never automated | Accepted | R-TEST-03, R-TEST-05, R-ENV-03 |
|
||||
@@ -1,5 +1,5 @@
|
||||
# ADR-0010 — Conformance testing: automated, reproducible, behavior-based
|
||||
- Status: Accepted
|
||||
- Status: Accepted — stands, except that L3 fault injection does not gate merges ([ADR-0026](adr-0026-fault-injection-operator-run.md))
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04, R-TEST-05, R-TEST-06
|
||||
- Supersedes: —
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
# ADR-0011 — Conformance test tooling: Kyverno + Chainsaw
|
||||
- Status: Accepted
|
||||
- Status: Accepted — stands, except that L3 is operator-run rather than CI-triggered ([ADR-0026](adr-0026-fault-injection-operator-run.md))
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04, R-TEST-05
|
||||
- Refines: [ADR-0010](adr-0010-conformance-testing.md) (which fixed the *approach* and left the tool mix as the one open item)
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
# ADR-0026 — Fault injection is operator-run, never automated
|
||||
- Status: Accepted
|
||||
- Date: 2026-11-08
|
||||
- Serves: R-TEST-03, R-TEST-05, R-ENV-03
|
||||
- Supersedes: the L3 portion of ADR-0010's merge gating and of ADR-0011's CI-triggered harness
|
||||
- Accepted by: operator ruling of 2026-11-08; effective on merge.
|
||||
|
||||
## Context
|
||||
[ADR-0010](adr-0010-conformance-testing.md) has the conformance suite gate merges, and
|
||||
[ADR-0011](adr-0011-test-tooling-kyverno-chainsaw.md) has CI trigger L2 and L3 as an in-cluster Job.
|
||||
Both were written when a dev/test cluster existed to break.
|
||||
[ADR-0017](adr-0017-single-environment-bare-metal-production.md) retired that cluster, so production
|
||||
is now the only target. An L3 test takes the VPN tunnel down, kills a node, injects a bad update or
|
||||
restores a backup. Running that automatically would put a production outage on a pipeline's
|
||||
schedule, and the operator sees no reason to do so.
|
||||
|
||||
## Decision
|
||||
- **L3 fault-injection tests are never triggered by CI, a schedule or any other automation.** The
|
||||
operator starts them deliberately, at a chosen time.
|
||||
- **L3 does not gate individual merges.** The merge gate covers the automatable layers.
|
||||
- **Fault injection stays required.** R-TEST-05 is unchanged: safety-critical behaviors are still
|
||||
verified by fault-injection tests, using the same in-cluster Job and scoped test identity. Each
|
||||
operator run is recorded in the implementation's run ledger.
|
||||
- ADR-0011's decision stands for L2, and for L3's tooling and harness.
|
||||
|
||||
## Alternatives considered
|
||||
- **Automate L3 against production on a schedule** — rejected by the operator: a deliberate outage
|
||||
has no business being unattended.
|
||||
- **Rebuild a dev/test cluster to automate against** — cost with no stated need; ADR-0017 retired it.
|
||||
- **Retire fault-injection tests altogether** — not ruled. R-TEST-05 protects properties such as the
|
||||
VPN kill-switch and data survival, which happy-path checks cannot prove.
|
||||
|
||||
## Consequences
|
||||
- No credential or pipeline path can start a fault against production.
|
||||
- A merge can change a safety-critical behavior without its L3 test re-running, so an L3 proof can go
|
||||
stale. The implementation's status tooling reports that staleness; re-running is an operator choice.
|
||||
- The wiring that remains to be built for CI is the L2 trigger only.
|
||||
@@ -1,67 +0,0 @@
|
||||
# ADR-0027 — Vulnerability visibility: scheduled digest scanning with a Renovate-shaped bot
|
||||
- Status: Accepted
|
||||
- Date: 2026-11-09
|
||||
- Serves: R-VULN-01..09, R-OBS-02, R-OBS-03, R-C2-02, R-UPD-02, R-SEC-02, R-IDP-06
|
||||
- Supersedes: —
|
||||
- Accepted by: operator merge of this change; effective on merge.
|
||||
|
||||
## Context
|
||||
The operator has no view of the vulnerabilities in what production runs. Renovate moves versions
|
||||
forward on a schedule, but it knows nothing about CVEs, and nothing in the platform assesses a running
|
||||
image. Measured on 2026-11-09:
|
||||
|
||||
- 259 containers run from 98 unique digests, across eight or more registries, with no scanner present.
|
||||
- Prometheus holds 427k head series, using about 2.75 GiB of its 3 GiB limit.
|
||||
- etcd and Longhorn share a partition on every node (accepted risk, [docs/02](../docs/02-environments-and-provisioning.md)),
|
||||
and etcd's own metrics are not scraped, so the headroom for added etcd writes cannot be measured.
|
||||
- CI holds no cluster credential by decision, and has two runner slots that also serve the merge gate.
|
||||
- The implementation repository is readable without authentication.
|
||||
- Trivy's release pipeline was compromised in March 2026 (CVE-2026-33634).
|
||||
|
||||
## Decision
|
||||
**A small scheduled scanner runs in the cluster, built on the Trivy CLI, and a Renovate-shaped CI
|
||||
bot turns its results into action.** The full contract is [docs/17](../docs/17-vulnerability-visibility.md).
|
||||
|
||||
- **Scanner.** Takes its inventory from the Kubernetes API, scans each unique digest with Trivy, and
|
||||
stores one versioned report on a small dedicated volume, outside etcd and Prometheus.
|
||||
- **Page.** Served from that report behind SSO and restricted to `platform-admins`.
|
||||
- **Metrics.** Aggregates only.
|
||||
- **Bot.** A scheduled Gitea Actions workflow using the existing CI-scoped Renovate credential. It
|
||||
maintains a counts-only dashboard issue and requests Renovate PRs through Renovate's own
|
||||
dashboard approval mechanism.
|
||||
- **Renovate stays the sole author of version changes.** No second bot edits a tag.
|
||||
- **Delivery tooling, not a catalog service.** Its cluster-wide read role is something a
|
||||
ServiceDefinition cannot express, so it is hand-authored under `platform/` with its own non-gating
|
||||
Flux tier, as in ADR-0025.
|
||||
|
||||
## Alternatives considered
|
||||
- **Trivy Operator.** The obvious choice, and rejected on measurement. It stores one report per
|
||||
workload as a CRD, adding write load to an etcd we cannot currently measure and which shares its
|
||||
disk with Longhorn; the largest images produce the largest reports, so size-limit failures would
|
||||
hide exactly the worst offenders. It needs broad cluster permissions and job-creation rights. It
|
||||
would also still need a bespoke aggregator, because the per-CVE page cannot be fed from Prometheus.
|
||||
- **Grafana table over per-CVE Prometheus metrics.** Needs no new page, but adds tens of thousands of
|
||||
long-labelled series to a Prometheus at about 92% of its memory limit. Raising the limit treats the
|
||||
symptom; a CVE inventory is not time-series data.
|
||||
- **Scanning from Gitea Actions.** Closest to Renovate literally, but CI has no view of what is
|
||||
running, and a long scan would occupy one of the two runner slots that gate merges.
|
||||
- **A third-party Trivy dashboard.** Assumes the operator's CRDs, and adds a further small-maintainer
|
||||
supply-chain dependency to a sensitive surface.
|
||||
- **An in-cluster bot with a Gitea token.** Puts a git-writing credential in the cluster, which
|
||||
ADR-0023 rejects.
|
||||
- **A CVE dashboard issue listing the findings.** The most Renovate-like, and rejected because the
|
||||
repository is publicly readable and the list would be an attack map.
|
||||
- **A bot that authors bump PRs itself.** A package's fixed version does not identify the image tag
|
||||
containing it, and two version authors would contend. Rejected in favour of driving Renovate.
|
||||
|
||||
## Consequences
|
||||
- Scanning and its output are hands-off, and "not scanned" is visible rather than silent.
|
||||
- The bot reuses the Renovate token, so both act as `renovate-bot`. This avoids a new credential;
|
||||
splitting them later changes one workflow secret.
|
||||
- The platform carries a small custom component that it must maintain and test. The report schema
|
||||
isolates the scanner, so Trivy can be replaced without touching the page or the bot.
|
||||
- Findings for private-registry images, VMs and nodes remain unassessed, and are shown as such
|
||||
([known gaps](../docs/16-known-gaps.md)).
|
||||
- The critical-fix alert will fire on first deployment. That is the backlog becoming visible.
|
||||
- A tick can only open a Renovate PR. Remediation of majors and blast-radius components still waits
|
||||
for human review.
|
||||
@@ -110,8 +110,7 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
|
||||
### TLS & certificates — `R-TLS-*`
|
||||
- **R-TLS-01** — Every externally reachable service MUST present TLS from a **credible, trusted,
|
||||
non-self-signed** authority.
|
||||
- **R-TLS-02** — Certificates MUST be automatically **issued, renewed, and revoked** without manual
|
||||
steps.
|
||||
- **R-TLS-02** — Certificates MUST be automatically **issued and renewed** without manual steps.
|
||||
- **R-TLS-03** — Certificate issuance MUST work for internal-only services (no requirement to expose
|
||||
a service publicly just to validate a certificate).
|
||||
|
||||
@@ -150,28 +149,6 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
|
||||
- **R-SEC-04** — The Kubernetes API MUST NOT be reachable from the public internet by any path (no
|
||||
tunnel route, no edge forward, no public DNS record); it is LAN/VPN-only (ADR-0016).
|
||||
|
||||
### Vulnerability visibility — `R-VULN-*`
|
||||
- **R-VULN-01** — The platform MUST maintain an inventory of every container image running in
|
||||
production, by digest, mapped to the namespaces, workloads and containers using it, taken from the
|
||||
running cluster and not from git.
|
||||
- **R-VULN-02** — Every running image MUST be evaluated against a current vulnerability database at
|
||||
least every 24 hours, and a newly running image within 2 hours.
|
||||
- **R-VULN-03** — Any target not successfully evaluated MUST be visibly reported as such with its
|
||||
reason. Absence of data MUST NOT be presented as absence of vulnerabilities.
|
||||
- **R-VULN-04** — An authenticated page MUST list every known CVE and, for each, the images,
|
||||
containers and workloads it affects, and MUST be searchable and filterable.
|
||||
- **R-VULN-05** — Inventory, scanning, reporting and triage MUST run on a schedule with **no human
|
||||
and no AI action**, and MUST fail loudly when they cannot run.
|
||||
- **R-VULN-06** — Scan failure, staleness, and unaccepted CRITICAL vulnerabilities with an available
|
||||
fix MUST be reported by **email**, as grouped alerts.
|
||||
- **R-VULN-07** — A scheduled bot MUST maintain a single dashboard issue and request remediation
|
||||
through Renovate. Renovate MUST remain the only author of version changes, and CVE detail MUST NOT
|
||||
be published to any surface readable without authentication.
|
||||
- **R-VULN-08** — Accepted risk MUST be declared in git, through the merge gate, with a reason and a
|
||||
mandatory expiry, and MUST revert to open automatically when it expires.
|
||||
- **R-VULN-09** — The scanner MUST be digest-pinned, MUST NOT auto-merge its own updates, MUST NOT
|
||||
read Secrets or hold Kubernetes write access, and MUST hold no git credential.
|
||||
|
||||
### Identity & access (SSO) — `R-IDP-*`
|
||||
- **R-IDP-01** — The platform MUST provide a central Identity Provider (SSO) as the single user
|
||||
directory and login for platform services.
|
||||
@@ -206,7 +183,9 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
|
||||
any conforming cluster, producing machine-readable, requirement-tagged results.
|
||||
- **R-TEST-03** — A **conformance suite** MUST gate delivery: merges are blocked unless the tests
|
||||
for the touched requirements pass, and phase exits are blocked unless the phase's full suite
|
||||
passes. "Demonstrated" means "the automated test passes," never a manual one-off.
|
||||
passes. "Demonstrated" means "the automated test passes," never a manual one-off. Fault-injection
|
||||
tests (R-TEST-05) are not automated: the operator runs them deliberately and records the result,
|
||||
and they do not gate individual merges (ADR-0026).
|
||||
- **R-TEST-04** — Tests MUST be **traceable**: each test declares the requirement ID(s) it verifies,
|
||||
and a coverage report maps every `R-*` requirement to its verifying test(s). Uncovered
|
||||
requirements are a gap to close, not an acceptable state.
|
||||
|
||||
@@ -76,7 +76,7 @@ runs on every PR as a Gitea Actions job. **A red gate is unmergeable by everyone
|
||||
included.** The gate runs in the same pinned toolbox image locally and in CI (R-TEST-02) and
|
||||
establishes, in order: the adopted spec pin (R-TEST-06), the handoff log's format, lint, L0 schema +
|
||||
policy + repository invariants, render-equals-committed, L1 rendering, the sensitive-path approval
|
||||
policy, that the live Gitea settings match their declaration, and L2/L3 conformance
|
||||
policy, that the live Gitea settings match their declaration, and L2 conformance
|
||||
([14](14-testing-and-conformance.md); see [known gaps](16-known-gaps.md)).
|
||||
|
||||
Its verdict is one of three: **pass**, **blocked**, or **undetermined**. A stage that did not run is
|
||||
|
||||
@@ -63,7 +63,7 @@ Defense in depth with declarative, auditable controls:
|
||||
- DNS provider API token (Cloudflare) for ACME DNS-01 + ExternalDNS.
|
||||
- Cloudflare Tunnel credential/token for in-cluster cloudflared (SOPS-encrypted; ADR-0016).
|
||||
- SMTP credentials for Alertmanager email.
|
||||
- Service credentials (Nextcloud, Gitea admin, UniFi, Semaphore, DB passwords; Kasm is not onboarded).
|
||||
- Service credentials (Nextcloud, Gitea admin, UniFi, Semaphore, DB passwords).
|
||||
- Per-app forward-auth gate credentials (IdP client secret + cookie secret; [ADR-0022](../decisions/adr-0022-forward-auth-per-app-oauth2-proxy-sidecars.md)).
|
||||
- WireGuard ingress server key, held in SOPS because a rebuild must restore it without re-enrolling
|
||||
every client (ADR-0021).
|
||||
|
||||
@@ -6,7 +6,7 @@ Serves: **R-TLS-01/02/03**, **R-DNS-01**. Related: [ADR-0006 (cert-manager + Let
|
||||
## Requirement recap
|
||||
|
||||
Every externally reachable service presents TLS from a **credible, trusted, non-self-signed**
|
||||
authority, with certificates **auto issued, renewed, and revoked** — including for **internal-only**
|
||||
authority, with certificates **auto issued and renewed** — including for **internal-only**
|
||||
services (R-TLS-01/02/03).
|
||||
|
||||
## Solution: cert-manager + Let's Encrypt via DNS-01
|
||||
@@ -39,7 +39,7 @@ Certificate issuance needs programmatic write access to the zone: ACME DNS-01 mu
|
||||
**Decision (OQ-4):** use a **scoped Cloudflare API token** — `Zone·DNS·Edit` limited to the
|
||||
`keithjackson.dev` zone — stored **SOPS-encrypted** ([07](07-security-and-secrets.md)). With it:
|
||||
- **cert-manager** solves DNS-01 by writing/removing TXT challenge records automatically (issue,
|
||||
renew, revoke — R-TLS-02), including the wildcard `*.home.keithjackson.dev` cert and internal-only
|
||||
renew — R-TLS-02), including the wildcard `*.home.keithjackson.dev` cert and internal-only
|
||||
services (R-TLS-03).
|
||||
- **ExternalDNS** manages the `*.home.keithjackson.dev` records automatically ([09](09-dns-and-ingress.md),
|
||||
R-DNS-02) — no manual record editing for public services.
|
||||
@@ -49,11 +49,11 @@ Certificate issuance needs programmatic write access to the zone: ACME DNS-01 mu
|
||||
committed SOPS-encrypted and never touched manually again. The token is least-privilege (single zone,
|
||||
DNS edit only) so its blast radius is bounded (R-SEC-02). Rotation = re-encrypt + commit.
|
||||
|
||||
## Renewal & revocation (R-TLS-02)
|
||||
## Renewal (R-TLS-02)
|
||||
|
||||
- cert-manager renews well before expiry automatically; no human action.
|
||||
- Revocation: removing/rotating a certificate is a declarative change; cert-manager handles
|
||||
issuance of replacements. Expiry monitoring is an alert condition ([10](10-observability-and-alerting.md), R-OBS-04).
|
||||
- Removing or rotating a certificate is a declarative change; cert-manager issues the replacement.
|
||||
The platform does not perform ACME revocation: a compromised key is handled by rotation. Expiry monitoring is an alert condition ([10](10-observability-and-alerting.md), R-OBS-04).
|
||||
|
||||
## Where TLS terminates
|
||||
|
||||
|
||||
@@ -62,9 +62,6 @@ Grafana dashboards, surfaced in the C2 UI ([12](12-command-center-ui.md)):
|
||||
- **Network**: Hubble flows, egress/VPN health, denied-flow trends.
|
||||
- **Per-service**: one row/dashboard per catalog service with its declared SLIs.
|
||||
|
||||
The CVE listing is deliberately **not** a Grafana dashboard: per-CVE data does not belong in
|
||||
Prometheus. See [17](17-vulnerability-visibility.md), which also defines the vulnerability alerts.
|
||||
|
||||
## Observability as onboarding output
|
||||
|
||||
The ServiceDefinition ([13](13-service-catalog.md)) declares observability expectations; the
|
||||
|
||||
@@ -78,12 +78,6 @@ each declared ceiling and carries any per-service cadence.
|
||||
- **Pin escape hatch**: an operator can pin a service via git if an upstream release misbehaves —
|
||||
the one sanctioned human touch, and it *stops* automation rather than performing an update.
|
||||
|
||||
## Vulnerability-driven requests
|
||||
|
||||
The CVE triage bot ([17](17-vulnerability-visibility.md)) can ask Renovate to open a pending update
|
||||
for an image with an unaccepted CRITICAL that has a fix, using Renovate's own dashboard approval.
|
||||
Renovate remains the only author of version changes, and every merge rule above still applies.
|
||||
|
||||
## Interaction with default-deny & egress
|
||||
Updates change image versions, not network posture. A VPN-class service stays VPN-class across
|
||||
updates; policies are reconciled from the (unchanged) ServiceDefinition.
|
||||
|
||||
@@ -37,8 +37,6 @@ root aggregates and links them. Status of this composition is tracked in
|
||||
- **Platform health**: node status, Longhorn capacity/health, Flux reconciliation, cert expiry,
|
||||
backup status, VPN tunnel state — sourced from the observability stack ([10](10-observability-and-alerting.md)).
|
||||
- **Alerts**: current firing alerts (mirrors what email delivers).
|
||||
- **Vulnerabilities**: a link to the CVE page, restricted to `platform-admins`
|
||||
([17](17-vulnerability-visibility.md)).
|
||||
|
||||
## Actions (safe, audited)
|
||||
|
||||
|
||||
@@ -79,12 +79,12 @@ and `storage[].{mountPath,subPath,readOnly}`.
|
||||
- Longhorn-backed config PVC; off-cluster backup; trusted TLS; auto-DNS; monitored; auto-updated on
|
||||
`stable`.
|
||||
|
||||
## The catalog (14 services)
|
||||
## The catalog (13 services)
|
||||
|
||||
Egress and dependencies below are **normative intent**; implementers render the exact policies.
|
||||
"Bulk" = large media tier ([04](04-storage-and-state.md)). All get trusted TLS + auto-DNS + monitoring
|
||||
+ auto-update unless noted. **Status** gives each service's delivery path: `catalog` (rendered from
|
||||
its ServiceDefinition), `platform-managed` (declared directly in git), or `not onboarded`.
|
||||
its ServiceDefinition) or `platform-managed` (declared directly in git).
|
||||
|
||||
| # | Service | Status | kind | exposure | egress | Storage | Key dependencies (explicit allows) | Notes |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
@@ -99,9 +99,8 @@ its ServiceDefinition), `platform-managed` (declared directly in git), or `not o
|
||||
| 8 | **minecraft** (Crafty) | platform-managed | container | internal (game ports); public (web UI) | internet | world: longhorn (backed up) | — | Delivered as the Crafty controller, which runs the game servers; game ports on a LAN/VPN L4 address |
|
||||
| 9 | **unifi** | platform-managed | vm | internal | internet | VM disk: longhorn (replicated, live-migratable) | — | UniFi OS Server in a KubeVirt VM, declared directly in git (custom install flow); browser and device traffic on a LAN/VPN L4 address, no Gateway route |
|
||||
| 10 | **semaphore** | catalog | container | internal | internet | config/db: longhorn | → its DB | Ansible UI |
|
||||
| 11 | **kasm** | not onboarded | container | internal | internet | config: longhorn | — | Kasm Workspaces (may need more resources; container-first, VM if required) |
|
||||
| 12 | **gitea** | catalog | container | public | internet | repos/db: longhorn (irreplaceable → backed up) | — | Git hosting; **bootstrapping-critical** — self-hosts the platform's GitOps source + CI/CD; onboarded **early** (see below); DR-critical (off-cluster backup + mirror) |
|
||||
| 13 | **qbittorrent** | catalog | container | internal | **vpn** | config: longhorn; downloads: **bulk** | ← sonarr, ← radarr | VPN + kill-switch; download client |
|
||||
| 11 | **gitea** | catalog | container | public | internet | repos/db: longhorn (irreplaceable → backed up) | — | Git hosting; **bootstrapping-critical** — self-hosts the platform's GitOps source + CI/CD; onboarded **early** (see below); DR-critical (off-cluster backup + mirror) |
|
||||
| 12 | **qbittorrent** | catalog | container | internal | **vpn** | config: longhorn; downloads: **bulk** | ← sonarr, ← radarr | VPN + kill-switch; download client |
|
||||
|
||||
**VPN-class set (R-NET-05/06):** sonarr, radarr, jackett, qbittorrent — all egress via VPN with an
|
||||
enforced kill-switch. Their inter-dependencies (arr → qbittorrent/jackett) are **east-west allows**
|
||||
@@ -125,7 +124,7 @@ stall an earlier wave. VMs and leaf platform components sit on their own non-gat
|
||||
image import or a broken UI never holds up the app tier.
|
||||
|
||||
**SSO integration (`auth.mode`, R-IDP-02; see [15](15-identity-and-access.md)):**
|
||||
- **oidc** (native Keycloak client): gitea, nextcloud, semaphore, kasm, grafana*, headlamp*
|
||||
- **oidc** (native Keycloak client): gitea, nextcloud, semaphore, grafana*, headlamp*
|
||||
(*platform UIs, not in the numbered app list).
|
||||
- **forward-auth** (per-app oauth2-proxy sidecar → Keycloak, ADR-0022): pihole, sonarr, radarr,
|
||||
jackett, qbittorrent, Homepage*, and the Crafty web UI* (which adds its own local login; it has no
|
||||
@@ -174,8 +173,6 @@ Points the schema left ambiguous, now pinned so the JSON Schema, Kyverno policie
|
||||
oauth2-proxy host). Everything else stays `internal`. **Plex** remote
|
||||
streaming uses plex.tv + an edge forward of tcp/32400 (not the tunnel), and its web UI is LAN-only.
|
||||
**Minecraft (crafty-mc)** and other L4 services are LAN/VPN-only.
|
||||
- **Kasm** is resource-heavy and may be a candidate for `kind: vm` if container mode proves
|
||||
insufficient — the ServiceDefinition makes that a one-field change.
|
||||
- **Gitea is bootstrapping-critical.** The platform self-hosts its git and CI/CD, so Gitea is brought
|
||||
online **early** (the first wave, with PiHole). Gitea + Gitea Actions are the GitOps source and CI
|
||||
([05](05-gitops-and-delivery.md), [`../CONTRIBUTING.md`](../CONTRIBUTING.md)). Because a cluster
|
||||
|
||||
@@ -17,7 +17,8 @@ Related: [ADR-0010](../decisions/adr-0010-conformance-testing.md).
|
||||
environment-agnostic. Any conforming cluster + a kubeconfig ⇒ the same pass/fail. No flakiness
|
||||
budget; eventual-consistency is handled with bounded polling, not sleeps-and-hope.
|
||||
3. **Conformance gates delivery (R-TEST-03).** The suite blocks merges (for touched requirements)
|
||||
and blocks phase exits (full phase suite). Green suite is the definition of progress.
|
||||
and blocks phase exits (full phase suite). Green suite is the definition of progress. Fault
|
||||
injection is operator-run and does not gate individual merges (ADR-0026).
|
||||
4. **Traceable (R-TEST-04).** Each test declares its `R-*` IDs; a generated coverage report maps
|
||||
every requirement to its verifying test(s). Uncovered requirements are visible gaps.
|
||||
5. **Fault-injection for safety-critical behavior (R-TEST-05).** Default-deny, VPN kill-switch,
|
||||
@@ -52,7 +53,7 @@ Related: [ADR-0010](../decisions/adr-0010-conformance-testing.md).
|
||||
| **L0 Static/policy** | Pre-apply, in CI | Manifests/ServiceDefinitions obey invariants | Every namespace has a default-deny baseline; no plaintext secrets; schema-valid ServiceDefinition |
|
||||
| **L1 Rendering** | CI, no cluster | ServiceDefinition → rendered resources match golden expectations | `egress: vpn` renders the kill-switched egress path |
|
||||
| **L2 Behavioral/integration** | Against the live cluster | Real runtime behavior | Default-deny probe, dependency allow/deny, egress-IP, TLS validity, DNS, PVC rebuild |
|
||||
| **L3 Fault-injection** | Against the live cluster | Fail-closed / no-data-loss under failure | VPN tunnel forced down ⇒ no leak; node killed ⇒ data survives; bad update ⇒ auto-rollback; DR restore |
|
||||
| **L3 Fault-injection** | Against the live cluster, operator-run only | Fail-closed / no-data-loss under failure | VPN tunnel forced down ⇒ no leak; node killed ⇒ data survives; bad update ⇒ auto-rollback; DR restore |
|
||||
| **L4 Conformance suite** | CI + on-demand | Aggregate, requirement-tagged gate | The union of the above, reported per `R-*` |
|
||||
|
||||
## Behavior test catalog (normative — implementers author these as automated tests)
|
||||
@@ -102,10 +103,6 @@ Each maps to requirement IDs; phrased as Given/When/Then intent.
|
||||
an email alert is delivered and the dashboard reflects it.
|
||||
- **GitOps drift (R-GITOPS-03, L3):** *Given* a reconciled resource, *When* it is mutated live,
|
||||
*Then* Flux reverts it to the git-declared state.
|
||||
- **Vulnerability visibility (R-VULN-*, L0/L1/L2/L3):** *Given* the running inventory, *Then* every
|
||||
image is either evaluated within its window or visibly reported as not evaluated, and each CVE is
|
||||
listed against its images, containers and workloads. Full list in
|
||||
[17](17-vulnerability-visibility.md).
|
||||
- **Onboarding completeness (WS-APPS):** *Given* only a merged ServiceDefinition, *Then* the service
|
||||
is fully wired (compute/storage/policy/egress/TLS/DNS/monitoring/update) with no manual step —
|
||||
asserted by the union of the relevant tests above against that service.
|
||||
@@ -113,10 +110,13 @@ Each maps to requirement IDs; phrased as Given/When/Then intent.
|
||||
## Gating rules (R-TEST-03)
|
||||
|
||||
- **Per-change (merge gate):** the conformance subset tagged with the requirement IDs the change
|
||||
touches MUST pass before merge.
|
||||
touches MUST pass before merge, across the automatable layers (L0–L2).
|
||||
- **Per-phase (exit gate):** the full conformance suite for the phase's requirements MUST pass
|
||||
before the phase's exit criteria are considered met ([`../roadmap/phases.md`](../roadmap/phases.md)).
|
||||
- **Regression:** the growing suite runs continuously against the live cluster; a previously-green
|
||||
- **Fault injection (L3) is operator-run** ([ADR-0026](../decisions/adr-0026-fault-injection-operator-run.md)):
|
||||
no pipeline or schedule ever starts it. The operator runs it deliberately and records each result
|
||||
in the run ledger.
|
||||
- **Regression:** the automatable layers run continuously; a previously-green
|
||||
requirement going red blocks further delivery until fixed.
|
||||
- **Coverage:** a requirement with no passing test is **not** satisfied, regardless of implementation
|
||||
state.
|
||||
@@ -176,7 +176,8 @@ pinned for determinism.
|
||||
- **L0 (Kyverno CLI) and L1 (rendering assertions)** run **in CI without a cluster** — pure manifest/
|
||||
ServiceDefinition validation; fast merge-gate feedback.
|
||||
- **L2 behavioral and L3 fault-injection run as an in-cluster Chainsaw Job** on the live cluster.
|
||||
CI **triggers** the Job (and waits for it); the Job — not CI — holds cluster access, so CI never
|
||||
CI **triggers** the L2 Job (and waits for it), and only the operator starts an L3 run
|
||||
(ADR-0026). The Job — not CI — holds cluster access, so CI never
|
||||
needs a broad kubeconfig. CI holds only a narrow trigger credential, whose reach is bounded by a
|
||||
fail-closed admission policy on the Job's shape. This keeps credentials in-cluster and
|
||||
least-privileged (R-SEC-02).
|
||||
@@ -198,7 +199,7 @@ pinned for determinism.
|
||||
|
||||
## Open questions
|
||||
- Tool mix is **resolved**: Kyverno (L0) + Chainsaw (L1/L2/L3), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md).
|
||||
- Harness location is **resolved**: L2/L3 run as an **in-cluster Chainsaw Job** triggered by CI; CI
|
||||
- Harness location is **resolved**: L2/L3 run as an **in-cluster Chainsaw Job**, L2 triggered by CI and L3 by the operator; CI
|
||||
holds no broad cluster credential, only the admission-bounded trigger.
|
||||
- Kyverno-as-admission-enforcer is **resolved**: yes — the L0 policies also enforce at admission
|
||||
in-cluster (defense in depth), in addition to the CI gate.
|
||||
|
||||
@@ -21,7 +21,7 @@ Every service authenticates against Keycloak, via whichever pattern it supports:
|
||||
|
||||
1. **Native OIDC** — the app is configured as a Keycloak OIDC **client** and does its own login
|
||||
redirect. Preferred where available. Initial set: **Gitea, Nextcloud, Grafana, Semaphore,
|
||||
Headlamp** (Kasm, not onboarded, joins this set when it is).
|
||||
Headlamp**.
|
||||
2. **Forward-auth gate** — for apps with **no** usable OIDC, a per-app identity-aware proxy
|
||||
(**oauth2-proxy**) runs as a sidecar in the app's pod, authenticates the user against Keycloak,
|
||||
and only then forwards the request ([ADR-0022](../decisions/adr-0022-forward-auth-per-app-oauth2-proxy-sidecars.md)).
|
||||
|
||||
+5
-12
@@ -15,15 +15,11 @@ has yet to reach.
|
||||
|
||||
## Operator decisions needed
|
||||
|
||||
These cannot be settled by reading either repository.
|
||||
This cannot be settled by reading either repository.
|
||||
|
||||
| Topic | Question | Evidence (implementation repo) |
|
||||
|---|---|---|
|
||||
| Bulk-tier photos (R-STATE-03/04) | The bulk tier has no platform backup, and its media dataset includes a `photos/` tree. Is that tree irreplaceable, and if so, what protects it? | `docs/BULK-NFS-STORAGE.md` §2; `platform/velero/volume-policies.yaml` |
|
||||
| Homepage exposure (R-IDP-02) | The apex is routed publicly with no enforced authentication. Enforce the gate (the requirement stands), or rule on an exception? | `platform/homepage/httproute.yaml`; `platform/homepage/README.md` |
|
||||
| ACME revocation (R-TLS-02) | cert-manager has no revocation mechanism. Amend the requirement, or keep it as an open gap? | `conformance/tests/l0-static/tls-lifecycle-automated/chainsaw-test.yaml:36` |
|
||||
| L3 in CI (R-TEST-03, R-TEST-05) | Production is the only cluster. Is automated fault injection against it acceptable once CI is wired? | `docs/plans/l2-l3-conformance-ci-wiring.md` |
|
||||
| Kasm (docs/13) | Never onboarded, and no decision retires it. Onboard it, or retire it from the catalog? | `catalog/README.md:20` |
|
||||
|
||||
## Environment and provisioning
|
||||
|
||||
@@ -56,7 +52,6 @@ These cannot be settled by reading either repository.
|
||||
| VPN port forwarding (docs/03, ADR-0015) | The spec requires a host-side PIA port-forward helper for qBittorrent and an alert on PF acquisition failure; neither exists. | `infra/ansible/roles/vpn_egress_gateway/README.md:170`, `platform/vpn-egress/README.md` |
|
||||
| R-NET-01 | Every namespace must carry a default-deny baseline; Kyverno generates it only for `app-namespace=true`, and no policy file exists for cert-manager, cnpg, external-dns, longhorn, velero, backup, image-automation, external-snapshotter or vpn-egress. | `platform/kyverno/policies/require-default-deny-baseline.yaml`, `decisions/2026-10-03-public-dns-wan-target.md` |
|
||||
| R-SEC-03, R-OBS-04 | docs/03 requires alerting on unexpected denied flows; `HubbleDeniedFlowSpike` cannot fire because no Cilium or Hubble metrics are scraped. | `platform/monitoring/alerts.yaml:1047-1050,1162-1173` |
|
||||
| R-TLS-02 | Certificates must be automatically revoked; renewal is met, but cert-manager performs no ACME revocation and no layer proves it. | `conformance/tests/l0-static/tls-lifecycle-automated/chainsaw-test.yaml:36`, `DECISIONS-LOG.md:3350,3786` |
|
||||
| Client IP (ADR-0016) | IP-based logging and policy must read `CF-Connecting-IP` / `X-Forwarded-For` and trust them only from cloudflared; no trusted-proxy or client-IP configuration exists in `catalog/`, `platform/` or `apps/`. | `platform/cloudflared/config.yaml` (no client-IP handling downstream of it) |
|
||||
| R-DNS-04 | The edge must not forward 80/443; closing them is a manual router step, last recorded as not done, so the state is unverified. | `decisions/2026-11-08-dns-via-cloudflare-tunnel.md`, `runbooks/cloudflare-tunnel.md:47-55` |
|
||||
| R-DNS-04, R-DNS-05, R-SEC-04 | docs/09 and docs/14 require behavioural proof at the tunnel and from the LAN through a public resolver; only L0 static checks exist, and no L2 test touches the tunnel. | `conformance/tests/l0-static/cloudflared-ingress-rules`, `conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml` |
|
||||
@@ -65,7 +60,7 @@ These cannot be settled by reading either repository.
|
||||
|
||||
| Requirement or area | Requirement and shortfall | Evidence (implementation repo) |
|
||||
|---|---|---|
|
||||
| R-IDP-02, R-IDP-06 | No management UI may be exposed unauthenticated and the C2 requires an authorized group, but Homepage at the apex is routed straight to its Service with no gate and its `forward-auth` annotation enforces nothing. | `platform/homepage/httproute.yaml`, `platform/homepage/README.md` |
|
||||
| R-IDP-02, R-IDP-06 (operator ruling 2026-11-08: enforce it) | No management UI may be exposed unauthenticated and the C2 requires an authorized group, but Homepage at the apex is routed straight to its Service with no gate and its `forward-auth` annotation enforces nothing. | `platform/homepage/httproute.yaml`, `platform/homepage/README.md` |
|
||||
| R-SEC-03, R-C2-02 | Operational actions must be audited, but the k3s API server has only OIDC arguments and no audit log or audit policy. | `infra/ansible/roles/k3s_server/tasks/main.yml` |
|
||||
| R-SEC-01 | The acceptance gate requires no plaintext secret anywhere in git history (scanned), but the check inspects the tracked tree only and no history scanner runs in CI. | `conformance/tests/l0-static/no-plaintext-secrets/check.sh`, `ci/` |
|
||||
| R-SEC-01 | Rotation is meant to trigger a workload restart through reconcile (e.g. a checksum annotation), but no generated mechanism exists; the only annotations are hand-stamped on Homepage and Grafana. | `platform/homepage/deployment.yaml`, `platform/monitoring/kube-prometheus-stack.yaml`, `catalog/render.py` |
|
||||
@@ -78,19 +73,17 @@ These cannot be settled by reading either repository.
|
||||
| R-OBS-03 | Per-service dashboards are specified as onboarding output, but the renderer emits no dashboards; the in-repo dashboards are node pressure, KubeVirt and Keycloak only. | `catalog/render.py`, `platform/monitoring/node-pressure-dashboard.yaml`, `platform/kubevirt/dashboard.yaml`, `platform/keycloak/grafana-dashboard.yaml` |
|
||||
| R-OBS-01, R-ENV-02 | Log retention is specified as an overlay setting, but Loki's retention is hardcoded with no overlay knob. | `platform/monitoring/loki.yaml`, `platform/monitoring/README.md` |
|
||||
| R-OBS-04 | Monitoring storage is specified on Longhorn, but Alertmanager and Grafana run without persistent volumes, so silences and notification state are lost on restart. | `platform/monitoring/kube-prometheus-stack.yaml` |
|
||||
| R-OBS-02, R-OBS-04 | Acceptance requires platform alerts to fire and email in injected-fault tests, but the L3 alerting test is a scaffold and L2/L3 do not run in CI. | `conformance/tests/l3-fault-injection/alerting/chainsaw-test.yaml`, `CLAUDE.md` |
|
||||
| R-OBS-02, R-OBS-04 | Acceptance requires platform alerts to fire and email in injected-fault tests, but the L3 alerting test is a scaffold. | `conformance/tests/l3-fault-injection/alerting/chainsaw-test.yaml`, `CLAUDE.md` |
|
||||
| R-C2-01 | Homepage is specified as a live catalog with status pings and per-service widgets, but it is a hand-maintained ConfigMap of link tiles with no status or widgets. | `platform/homepage/configmap.yaml`, `platform/homepage/README.md` |
|
||||
| R-C2-02 | The C2 must surface platform health and firing alerts, but none is shown on the C2 landing surface; health is reachable only by linking out to Grafana. | `platform/homepage/configmap.yaml` |
|
||||
| R-C2-02 | Grafana dashboards are specified as embedded in the C2, but Homepage only links to Grafana. | `platform/homepage/configmap.yaml` |
|
||||
| R-C2-02 | Flux status is specified via Headlamp's Flux view, the Flux UI or a Homepage widget, but Headlamp has no Flux plugin configured and Homepage has no widget. | `platform/headlamp/helmrelease.yaml`, `platform/homepage/configmap.yaml` |
|
||||
| R-C2-02 | C2 actions are specified to include VM start/stop, Flux reconcile and alert acknowledgement, but only native Headlamp workload actions exist, with no plugins for the others. | `platform/headlamp/helmrelease.yaml` |
|
||||
| R-VULN-01, R-VULN-02 | KubeVirt guests, node OS packages and the k3s binary are inventoried but not vulnerability-assessed; private-registry images need a read-only credential the operator has yet to mint. All report as not evaluated (docs/17). | not yet implemented |
|
||||
|
||||
## Service catalog
|
||||
|
||||
| Requirement or area | Requirement and shortfall | Evidence (implementation repo) |
|
||||
|---|---|---|
|
||||
| docs/13 catalog row 11 (kasm), R-IDP-02 | The catalog includes Kasm with native OIDC, and no ServiceDefinition, manifest or decision exists for it, so it is not onboarded. | `catalog/README.md:20` (only mention in the repo) |
|
||||
| R-STATE-03, docs/13 `backup.policy` | `backup.policy` must reference a named off-cluster schedule, and `daily-offcluster` resolves to no schedule object; it is an annotation only, while coverage comes from cluster-wide Velero schedules. | `platform/velero/README.md:33-34`, `schemas/renderer-contract.md:428`, `catalog/*.yaml` (`backup.policy`) |
|
||||
| R-UPD-01/02, docs/13 `update.cadence` | `update.cadence` should set how often a service checks or applies updates, and the renderer does not read it for containers or VMs. | `catalog/render.py` (no `cadence` reference), `schemas/renderer-contract.md:388-390` |
|
||||
|
||||
@@ -98,8 +91,8 @@ These cannot be settled by reading either repository.
|
||||
|
||||
| Requirement or area | Requirement and shortfall | Evidence (implementation repo) |
|
||||
|---|---|---|
|
||||
| R-TEST-03 | Merges require the tests for touched requirements to pass; the gate runs L0/L1 and tagged-coverage only, because L2/L3 do not run in CI (job disabled, trigger path unimplemented, cluster not declared), and a PASS names that declared gap. | `.gitea/workflows/gate.yml` (`conformance` job `if: false`), `ci/conformance.sh`, `docs/plans/l2-l3-conformance-ci-wiring.md` |
|
||||
| R-TEST-03 | Phase exits require the full suite and regression runs continuously; no pipeline runs the full-suite gate and no scheduled run exists, so L2/L3 evidence comes only from operator-recorded runs, last recorded 2026-09-19. | `conformance/run.sh` (`--require-all`, `--record`), `conformance/runs/ledger.jsonl` |
|
||||
| R-TEST-03 | Merges require the tests for touched requirements to pass; the gate runs L0/L1 and tagged-coverage only, because L2 does not run in CI (job disabled, trigger path unimplemented, cluster not declared), and a PASS names that declared gap. | `.gitea/workflows/gate.yml` (`conformance` job `if: false`), `ci/conformance.sh`, `docs/plans/l2-l3-conformance-ci-wiring.md` |
|
||||
| R-TEST-03 | Phase exits require the full suite and regression runs continuously; no pipeline runs the full-suite gate and no scheduled run exists, so L2 evidence comes only from operator-recorded runs, last recorded 2026-09-19. L3 is operator-run by design (ADR-0026). | `conformance/run.sh` (`--require-all`, `--record`), `conformance/runs/ledger.jsonl` |
|
||||
| R-TEST-01, R-TEST-04 | A requirement with no passing test is not satisfied; all 51 registry requirements have a tagged test but PROVEN is 0 (3 stale: R-NET-06, R-STATE-02, R-STATE-03; 48 not derivable from the ledger). | `conformance/lib/status.py`, `conformance/runs/ledger.jsonl` |
|
||||
| R-OBS-02, R-GITOPS-02, R-GITOPS-03 | Each requirement needs an executable behavior test; the only verifying tests (`alerting`, `onboarding-completeness`, `gitops-drift`) are scaffolds. | `conformance/lib/status.py`, `conformance/tests/` |
|
||||
| R-TEST-06 | The spec must be verifiably immutable to agents; CI never checks the spec repo out, so spec-immutability always skips there (tolerated) and only local gates with a sibling checkout enforce it. | `.gitea/workflows/gate.yml:193,227`, `ci/spec-immutability.sh`, `ci/gate.sh` (`REQUIRE_SPEC_VERIFICATION`) |
|
||||
|
||||
@@ -1,289 +0,0 @@
|
||||
# 17 — Vulnerability Visibility
|
||||
|
||||
Serves: **R-VULN-01..09**, surfaces **R-OBS-02/03**, **R-C2-02**, **R-UPD-01/02**, **R-SEC-02**,
|
||||
**R-IDP-06**. Decision record: [ADR-0027](../decisions/adr-0027-vulnerability-visibility.md).
|
||||
|
||||
## Purpose
|
||||
|
||||
The operator must be able to answer three questions at any time, with no manual step:
|
||||
|
||||
1. **What is running in production?** Every container image, by digest, and the workloads using it.
|
||||
2. **What is it vulnerable to?** Every known CVE in those images, and where each one lands.
|
||||
3. **Can I trust the answer?** How fresh it is, and exactly what was *not* examined.
|
||||
|
||||
The third question is as important as the first two. A vulnerability list that silently omits an
|
||||
image reads the same as a clean bill of health. Everything below treats "not scanned" as a
|
||||
first-class state that is always visible, and never as zero findings.
|
||||
|
||||
## Shape of the solution
|
||||
|
||||
The workflow is deliberately the same shape as Renovate ([11](11-automated-updates.md)): scheduled,
|
||||
self-hosted, run to completion, machine-authored output, a single dashboard issue, and a loud failure
|
||||
when it cannot run.
|
||||
|
||||
```
|
||||
Kubernetes API (read-only)
|
||||
│ running pods → digests → workloads
|
||||
▼
|
||||
┌──────────────────────────── namespace: vuln-scanner ─────────────────────────┐
|
||||
│ Scanner (scheduled, one image at a time, Trivy) │
|
||||
│ └─► Report store (small PVC) ─► report.json (schema below) │
|
||||
│ ├─► CVE page cve.<domain>, SSO, platform-admins │
|
||||
│ ├─► /metrics aggregates only → Prometheus → email │
|
||||
│ └─► internal API ◄──────────────┐ │
|
||||
└─────────────────────────────────────────────┼────────────────────────────────┘
|
||||
│ read report; post heartbeat + remediation state
|
||||
┌────────────────┴───────────────┐
|
||||
│ Gitea Actions: CVE triage bot │ scheduled, before Renovate
|
||||
└────────────────┬───────────────┘
|
||||
│ scoped CI token
|
||||
▼
|
||||
CVE Dashboard issue (counts only) · approval ticks on Renovate's dashboard
|
||||
│
|
||||
▼
|
||||
Renovate opens the PR → merge gate → Flux (the existing update path)
|
||||
```
|
||||
|
||||
The split follows two standing boundaries: **CI holds no cluster credential**, and **the cluster
|
||||
holds no git credential** ([ADR-0023](../decisions/adr-0023-automated-updates-renovate.md)). The
|
||||
scanner sees the cluster and never writes to git. The bot writes to Gitea and never touches the
|
||||
Kubernetes API.
|
||||
|
||||
## Inventory (R-VULN-01)
|
||||
|
||||
The inventory is taken from the **Kubernetes API**, never from git. Git states intent; the question
|
||||
is what is actually running.
|
||||
|
||||
- It covers containers, init containers and ephemeral containers of every pod in every namespace.
|
||||
- The identity of a running image is the **digest reported in the container status**. A mutable tag
|
||||
is not an identity.
|
||||
- Each pod resolves through its owner chain to the **top-level workload** the operator recognises
|
||||
(Deployment, StatefulSet, DaemonSet, CronJob, or a bare Pod), not the intermediate ReplicaSet or Job.
|
||||
- **CronJobs with no pod currently running are still inventoried**, from their pod template. Their
|
||||
images are recorded by reference and marked `by-tag`, because no digest exists until they run.
|
||||
- KubeVirt VMs and nodes are inventoried as **non-container targets** and reported as not covered
|
||||
(see [scope](#scope)).
|
||||
|
||||
## Scanning (R-VULN-02, R-VULN-05)
|
||||
|
||||
| Property | Contract |
|
||||
|---|---|
|
||||
| Unit of work | One scan per **unique digest**, mapped back to all its workloads. Measured 2026-11-09: 259 containers resolve to 98 digests. |
|
||||
| Full re-evaluation | Every running digest is evaluated against a current database **at least every 24 hours**, so a CVE published against an unchanged image is found without any deployment. |
|
||||
| New digests | A digest not yet seen is scanned **within 2 hours** of first appearing. |
|
||||
| Database freshness | A scan uses a vulnerability database no older than **48 hours**. Older than that, the run is reported as degraded, not as a success. |
|
||||
| Severity scope | All severities are recorded, including those with no fix. Filtering is a page concern. |
|
||||
| Failure isolation | One image failing (pull error, timeout, out of memory) records that image as `failed` with its reason and the run continues. |
|
||||
| Load | Sequential, one image at a time, with CPU and memory limits. Scan caches MUST NOT live on a replicated Longhorn volume or in etcd, because etcd and Longhorn share a partition on every node ([02](02-environments-and-provisioning.md)). |
|
||||
| Trigger | Schedule only. No human or AI action starts, resumes or completes a cycle. |
|
||||
|
||||
Findings persist a **first-seen** timestamp across runs, keyed by CVE and digest, so age is
|
||||
measurable. A finding that disappears because an image was updated leaves the report at the next
|
||||
inventory; one that disappears because a scan *failed* does not.
|
||||
|
||||
## Coverage (R-VULN-03)
|
||||
|
||||
Every inventoried target has exactly one state:
|
||||
|
||||
| State | Meaning |
|
||||
|---|---|
|
||||
| `scanned` | Evaluated within the 24-hour window. Zero findings is meaningful only in this state. |
|
||||
| `pending` | Discovered and not yet scanned, within the 2-hour window. |
|
||||
| `stale` | Last successful scan is older than the 24-hour window. Prior findings are retained and labelled stale. |
|
||||
| `failed` | The most recent attempt failed. The reason is shown. Prior findings are retained. |
|
||||
| `credential-required` | The registry refused anonymous access. Shown with the registry. |
|
||||
| `by-tag` | Scanned by mutable reference, because no running digest exists. |
|
||||
| `not-covered` | A target class this scanner cannot examine: VMs and nodes. |
|
||||
|
||||
The page and the report MUST NOT present a total or a "clean" indication that silently excludes any
|
||||
target. The header of the page always states how many targets are in each state.
|
||||
|
||||
## The report contract
|
||||
|
||||
`report.json` is the one interface between the scanner, the page and the bot. It is versioned and
|
||||
the implementation validates it against a schema.
|
||||
|
||||
```yaml
|
||||
schemaVersion: 1
|
||||
generatedAt: <RFC 3339>
|
||||
database: { updatedAt: <RFC 3339>, version: <string> }
|
||||
run: { startedAt: <RFC 3339>, finishedAt: <RFC 3339>, status: complete | degraded }
|
||||
targets:
|
||||
- id: <stable identifier>
|
||||
type: image | vm | node
|
||||
image: <registry/repository>
|
||||
digest: <sha256:… or null>
|
||||
state: <one of the coverage states>
|
||||
reason: <string or null>
|
||||
lastScannedAt: <RFC 3339 or null>
|
||||
workloads:
|
||||
- { namespace: <ns>, kind: <kind>, name: <name>, containers: [<name>] }
|
||||
findings:
|
||||
- id: <CVE or advisory id>
|
||||
targetId: <targets[].id>
|
||||
severity: CRITICAL | HIGH | MEDIUM | LOW | UNKNOWN
|
||||
cvss: <number or null>
|
||||
package: <name>
|
||||
installedVersion: <string>
|
||||
fixedVersion: <string or null>
|
||||
title: <string>
|
||||
url: <advisory URL>
|
||||
firstSeenAt: <RFC 3339>
|
||||
status: open | fix-available | accepted
|
||||
acceptance: { reason: <string>, expires: <date> } # present only when accepted
|
||||
remediation: # written by the triage bot, may be empty
|
||||
- { image: <registry/repository>, state: <state>, url: <PR URL or null>, updatedAt: <RFC 3339> }
|
||||
```
|
||||
|
||||
Scanner-derived text (titles, package names, advisory URLs) originates upstream and is **untrusted**.
|
||||
The page MUST escape it and serve a restrictive Content-Security-Policy, and only `http(s)` URLs may
|
||||
become links.
|
||||
|
||||
## The CVE page (R-VULN-04)
|
||||
|
||||
A page at `cve.<domain>`, linked from the C2 landing page.
|
||||
|
||||
**Header, always visible:** report age, database age, per-state target counts, and open findings by
|
||||
severity. If the report is stale or degraded, the page says so prominently and does not show a
|
||||
normal state.
|
||||
|
||||
**CVEs view (default):** one row per CVE, with id (linking to the advisory), severity, CVSS,
|
||||
package, fixed version, first seen, status, and the counts of affected images and workloads. Each
|
||||
row expands to list **every image, container and workload** it touches, with namespace.
|
||||
|
||||
**Workloads view:** one row per workload, with its containers, images and per-severity counts,
|
||||
expanding to its CVEs.
|
||||
|
||||
**Inventory view:** every target and its coverage state, *including those with zero findings*. This
|
||||
is the answer to "what am I running".
|
||||
|
||||
**Interaction:** free-text search, and filters for severity, namespace, workload, status and "fix
|
||||
available", with sortable columns. Filter state is in the URL so a view can be linked. The
|
||||
default view excludes nothing.
|
||||
|
||||
**Remediation column:** for images with a fix available, one of `pr-open` (with link),
|
||||
`approval-requested`, `no-update-identified` or `not-tracked`, taken from the bot.
|
||||
|
||||
**Access:** internal exposure only, behind SSO, restricted to `platform-admins` (R-IDP-06). A list of
|
||||
what is exploitable where is sensitive, so this is not a "logged-in user" surface. The page is
|
||||
read-only and offers no actions.
|
||||
|
||||
## Alerting (R-VULN-06)
|
||||
|
||||
Alerts are delivered through the existing Alertmanager email route. Each alert is **one grouped
|
||||
message per condition**, never one per CVE.
|
||||
|
||||
| Alert | Fires when |
|
||||
|---|---|
|
||||
| Scan stale | No complete run within 26 hours, **or the metric is absent** |
|
||||
| Database stale | Vulnerability database older than 48 hours |
|
||||
| Coverage gap | Any target in `failed` for more than 24 hours |
|
||||
| Critical fix available | Any unaccepted CRITICAL with a fix, first seen more than 72 hours ago |
|
||||
| Triage bot silent | No bot heartbeat for 26 hours, **or the metric is absent** |
|
||||
|
||||
The 72-hour grace gives Renovate's daily cycle time to act, so email marks the cases automation has
|
||||
not resolved. The critical-fix alert is expected to fire on first deployment.
|
||||
|
||||
Prometheus receives **aggregates only**: counts by severity and fixability, per-state target counts,
|
||||
timestamps. It MUST NOT receive a per-CVE series. Measured 2026-11-09: Prometheus holds 427k head
|
||||
series at about 2.75 GiB of a 3 GiB limit.
|
||||
|
||||
## The triage bot (R-VULN-05, R-VULN-07)
|
||||
|
||||
A scheduled Gitea Actions workflow that follows the conventions of the Renovate workflow: it fails
|
||||
hard on a missing credential rather than skipping, has a `workflow_dispatch` dry run, never runs
|
||||
concurrently with itself, and records what it did in the job summary. It runs **before** the daily
|
||||
Renovate run so that approvals take effect the same morning.
|
||||
|
||||
Each run it:
|
||||
|
||||
1. Reads `report.json` over the internal API.
|
||||
2. Creates or updates a single **CVE Dashboard** issue on the implementation repository.
|
||||
3. Reads Renovate's Dependency Dashboard and open PRs, and derives each image's remediation state.
|
||||
4. Ticks the **approval** checkbox on Renovate's dashboard for a pending update whose package matches
|
||||
an image with an unaccepted CRITICAL that has a fix.
|
||||
5. Posts its heartbeat and the remediation states back to the scanner.
|
||||
|
||||
**Renovate remains the only author of version changes** (R-UPD-02, ADR-0023). The bot never edits a
|
||||
version, opens a bump PR or merges anything. A tick only asks Renovate to *open* a PR; majors and
|
||||
blast-radius components still wait for human review, exactly as today.
|
||||
|
||||
**The dashboard issue carries aggregates only:** total findings by severity, per-state coverage,
|
||||
timestamps and a link to the page. It MUST NOT name a CVE, package or image-to-workload relationship,
|
||||
because the repository is readable without authentication (measured 2026-11-09). It MUST NOT carry any
|
||||
label that the in-cluster Program Manager acts on ([ADR-0025](../decisions/adr-0025-in-cluster-ai-program-manager.md)).
|
||||
|
||||
The internal API accepts reads and a single bounded write (heartbeat plus remediation states, schema
|
||||
validated). It is reachable only from the CI runner namespace, by network policy.
|
||||
|
||||
## Risk acceptance (R-VULN-08)
|
||||
|
||||
A CVE the operator has decided to live with is declared **in git**, through the merge gate, with
|
||||
an expiry:
|
||||
|
||||
```yaml
|
||||
- id: CVE-2026-00000
|
||||
image: registry/repository # exact match; omit only for a platform-wide acceptance
|
||||
reason: not reachable — feature disabled
|
||||
expires: 2027-02-01 # mandatory, at most 180 days out
|
||||
```
|
||||
|
||||
An accepted finding stays in the listing, marked `accepted` with its reason and expiry, and stops
|
||||
counting toward the critical-fix alert. **When the expiry passes it reverts to open automatically.**
|
||||
An entry with no reason or expiry is rejected by the gate.
|
||||
|
||||
## Security of the scanner itself (R-VULN-09)
|
||||
|
||||
A scanner reads every image and is a supply-chain target. Trivy's release and CI tooling were
|
||||
compromised in March 2026 ([CVE-2026-33634](https://nvd.nist.gov/vuln/detail/CVE-2026-33634)), so:
|
||||
|
||||
- The scanner binary is pinned **by digest**. It is a Renovate blast-radius component and never
|
||||
auto-merges, and an update is proposed only after a **minimum release age of 7 days**.
|
||||
- Trivy 0.69.4, 0.69.5 and 0.69.6 MUST NOT be used.
|
||||
- Identity: `get/list/watch` on pods and their owner kinds, CronJobs, nodes and VM instances, and
|
||||
**nothing else**. It MUST NOT read Secrets or hold any write verb on the Kubernetes API.
|
||||
- It holds no git credential and no registry credential.
|
||||
- Network: ingress only from the Gateway for the page, from CI for the internal API and from
|
||||
Prometheus for metrics. Egress only to the Kubernetes API, DNS, and TCP/443 on the public internet,
|
||||
with cluster and LAN ranges excluded, through the direct path.
|
||||
- It runs unprivileged, with a read-only root filesystem and no host namespaces or mounts.
|
||||
- The scanner's own images appear in the report like any other.
|
||||
|
||||
## Scope
|
||||
|
||||
**In scope:** every container image running in a pod, on every node, in every namespace, including
|
||||
the platform's own components.
|
||||
|
||||
**Not in scope for this version, and reported as `not-covered` rather than omitted:** guest OS
|
||||
packages inside KubeVirt VMs, and node OS packages and the k3s binary. These are updated by the
|
||||
mechanisms in [11](11-automated-updates.md) but are not vulnerability-assessed here.
|
||||
|
||||
Also out of scope: application dependency scanning of source, misconfiguration and secret scanning,
|
||||
runtime exploit detection, and blocking deployment based on findings.
|
||||
|
||||
## Acceptance gates
|
||||
|
||||
Each is an automated, requirement-tagged test ([14](14-testing-and-conformance.md)).
|
||||
|
||||
- **L0:** digest-pinned scanner image; no banned version; Renovate rule keeps it out of auto-merge
|
||||
with the minimum release age; RBAC contains no Secret access and no write verb; no per-CVE metric
|
||||
label; alerts unit-tested, including the absent-metric cases; the bot workflow fails on a missing
|
||||
credential; acceptance entries without reason or expiry rejected.
|
||||
- **L1 (fixtures, no cluster):** inventory resolves ReplicaSet to Deployment and Job to CronJob;
|
||||
two workloads sharing a digest produce one scan and both mappings; one failed image leaves the run
|
||||
continuing with that target `failed` and its prior findings retained; an expired acceptance reverts
|
||||
to open; the rendered page contains every fixture CVE, container and workload; a hostile title is
|
||||
escaped; the page shows a warning for a stale report; a dashboard issue rendered from a fixture
|
||||
contains no CVE id or package name; a tick is applied only to an `approve` row for a matching image
|
||||
with a qualifying finding.
|
||||
- **L2:** a newly started workload's digest appears in the report within the window; the page requires
|
||||
a `platform-admins` login; the internal API is unreachable from an unrelated namespace.
|
||||
- **L3:** breaking registry access produces `failed` targets and fires the coverage alert.
|
||||
|
||||
## Open questions
|
||||
|
||||
- **Private registry coverage.** The platform's own images need a registry credential, so they report
|
||||
`credential-required`. Closing this requires a read-only registry token, an operator action.
|
||||
- **Exploitability ranking.** Adding CISA KEV or EPSS data, and weighting by whether a workload is
|
||||
publicly exposed, would rank the list better than CVSS alone.
|
||||
- **VM and node assessment.** How to bring `not-covered` targets into scope.
|
||||
@@ -105,17 +105,6 @@ alerts live.
|
||||
|
||||
---
|
||||
|
||||
## Phase 8 — Vulnerability Visibility
|
||||
**Status:** open; may proceed alongside Phase 7.
|
||||
**Entry:** ADR-0027 accepted.
|
||||
**Do:** deliver the scanner, report, CVE page, alerts and triage bot in
|
||||
[`../docs/17-vulnerability-visibility.md`](../docs/17-vulnerability-visibility.md).
|
||||
**Exit:** every running image is evaluated within its window or visibly reported as not evaluated;
|
||||
the acceptance gates in docs/17 pass.
|
||||
**Serves:** R-VULN-01..09.
|
||||
|
||||
---
|
||||
|
||||
## Dependency flow
|
||||
|
||||
```
|
||||
|
||||
@@ -65,7 +65,7 @@ only via [`../agents/coordination-protocol.md`](../agents/coordination-protocol.
|
||||
5. **qbittorrent, jackett** (VPN + kill-switch proof) → 6. **sonarr, radarr** (VPN + arr↔downloader
|
||||
allows) → 7. **seerr** (cross-service allows to arr + plex) → 8. **nextcloud** (DR-critical data +
|
||||
backups) → 9. **minecraft** (delivered as Crafty, platform-managed), **unifi** (a KubeVirt VM),
|
||||
**semaphore** → **kasm** (not onboarded).
|
||||
**semaphore**.
|
||||
|
||||
As each service onboards, it adopts SSO per its `auth` mode (native OIDC or forward-auth) —
|
||||
[`../docs/15-identity-and-access.md`](../docs/15-identity-and-access.md).
|
||||
|
||||
Reference in new issue
Block a user