Compare commits

...
Author SHA1 Message Date
jakman85 190cd6b81a Merge pull request 'feat(vuln): automated CVE visibility for production (R-VULN-*, ADR-0027)' (#165) from feat/vuln-visibility into main
gate / gate (push) Successful in 58s
gate / conformance (push) Skipped
Reviewed-on: #165
Reviewed-by: Keith Jackson <[email protected]>
2026-11-09 22:41:50 +00:00
jakman85 6680f361a1 Whitespace linting fix
gate / gate (pull_request) Successful in 58s
gate / conformance (pull_request) Skipped
2026-11-09 16:40:21 -06:00
jakman85 fd462a4ede Updated SPEC_PIN
gate / gate (pull_request) Failing after 59s
gate / conformance (pull_request) Skipped
2026-11-09 16:31:22 -06:00
jakman85 1ace03569c Added required keycloak config
gate / gate (pull_request) Failing after 1m0s
gate / conformance (pull_request) Skipped
2026-11-09 16:29:29 -06:00
jakman85andClaude Opus 5.5 e1adc3963c docs(vuln): handoff for automated CVE visibility
gate / gate (pull_request) Successful in 1m5s
gate / conformance (pull_request) Skipped
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-09 12:40:15 -06:00
jakman85 4ca0ebfac6 Merge branch 'wip/vuln-fixes' into feat/vuln-visibility 2026-11-09 12:38:41 -06:00
jakman85 1ff527f546 Merge branch 'wip/vuln-platform' into feat/vuln-visibility 2026-11-09 12:38:41 -06:00
jakman85andClaude Opus 5.5 d8b25a67fe fix(vuln): survive OOM kills, isolate bad upstream strings, label findings per line
Review findings on the integrated scanner:
- An OOM kill no longer wedges the loop: inFlight is persisted before each scan, and on restart
  that image is marked failed and sits out full runs for 20h.
- One bad upstream string can no longer freeze the report: safe_url requires an exact lowercase
  scheme, acceptance reasons are bounded, /triage urls use the same rule, and report.build
  repairs or drops any entry the schema would refuse. lastCompleteRunAt advances only after a
  report is actually published.
- The private registry reports credential-required, not failed: VULN_PRIVATE_REGISTRIES names
  it, and Trivy is not invoked.
- loadError clears after a full run against a good inventory.
- Every image under platform/vuln-scanner stays out of Renovate auto-merge.
- The page labels status and target state per line, and mixed acceptance reads as mixed.

The Keycloak client is still an operator action: its key must go in an existing encrypted file
that an agent must not modify. platform/vuln-scanner/README.md has the exact steps.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-09 12:28:29 -06:00
jakman85andClaude Opus 5.5 1763e27859 fix(vuln): metric label extractor handles escaped quotes in f-strings
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-09 12:28:28 -06:00
jakman85andClaude Opus 5.5 4c82c64b2f docs(vuln): list the CVE triage workflow in the workflows README
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-09 10:30:43 -06:00
jakman85andClaude Opus 5.5 d06dff9a19 fix(vuln): no-per-cve-label check also matches escaped-quote label forms
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-09 10:29:05 -06:00
jakman85 b1a6fe6503 Merge branch 'wip/vuln-platform' into feat/vuln-visibility 2026-11-09 10:29:05 -06:00
jakman85 b265d08278 Merge branch 'wip/vuln-bot' into feat/vuln-visibility 2026-11-09 10:29:05 -06:00
jakman85 2b9dcef6c7 Merge branch 'wip/vuln-scanner' into feat/vuln-visibility 2026-11-09 10:29:05 -06:00
jakman85andClaude Opus 5.5 d4bea8a78c feat(vuln): encrypted oauth2-proxy credential for the CVE page
Generated through pipes and encrypted to the repo's public age recipient; values never echoed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-09 09:43:12 -06:00
jakman85andClaude Opus 5.5 b85d1e3ed7 feat(vuln): vuln-scanner platform manifests, Flux tier, alerts, L0/L2/L3 tests
Hand-authored platform/vuln-scanner (non-gating Flux tier, digest-pinned Trivy 0.76.1 and
python 3.14.9-alpine, read-only RBAC, per-pod oauth2-proxy, internal-only cve.<domain>),
five alerts with promtool tests, R-VULN-01..09 registry entries, and vuln-* L0 checks with
mutation self-tests. src/ and the bot workflow are integrated separately.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-09 09:42:52 -06:00
jakman85andClaude Opus 5.5 50ec2297ee feat(vuln): scanner service and fixture-driven L1 tests
Stdlib-only scanner per docs/17: Kubernetes API inventory, Trivy runner,
persisted state, schema-validated report.json, strict acceptances parser,
server-rendered page, aggregate metrics, internal API, and a scan loop
that stays live while scanning.

L1 tests under conformance/tests/l1-rendering/vuln-scanner-* run the
actual modules against recorded fixtures with a fake Trivy runner.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-09 09:41:23 -06:00
jakman85andClaude Opus 5.5 ec6b39bb91 feat(vuln): point the triage bot's fallback at the pinned scanner ClusterIP
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-09 09:39:55 -06:00
jakman85andClaude Opus 5.5 c3c72c47bb feat(vuln): CVE triage bot workflow, script and L1 tests
Scheduled Gitea Actions bot in the Renovate shape: maintains a
counts-only CVE Dashboard issue, derives remediation states from
Renovate's dashboard and PRs, ticks approve rows for exact-matched
images with an unaccepted CRITICAL with a fix, and posts the heartbeat.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-09 09:39:24 -06:00
renovate-bot 4c3a284cad Merge pull request 'chore(deps): Update lscr.io/linuxserver/jackett Docker tag to v0.24.3030-ls30' (#162) from renovate/lscr.io-linuxserver-jackett-0.x into main
gate / gate (push) Successful in 1m3s
gate / conformance (push) Skipped
2026-11-09 08:20:07 +00:00
renovate-bot 4a89aacfbc chore(deps): Update lscr.io/linuxserver/jackett Docker tag to v0.24.3030-ls30
gate / gate (pull_request) Successful in 46s
gate / conformance (pull_request) Skipped
2026-11-09 08:16:20 +00:00
jakman85 50c8719b12 Merge pull request 'fix(vpn): default client profiles to MTU 1280; correct the VPN runbook' (#159) from fix/wg-client-mtu-docs into main
gate / gate (push) Successful in 57s
gate / conformance (push) Skipped
Reviewed-on: #159
2026-11-09 03:26:02 +00:00
jakman85andClaude Opus 5.5 8f60e6103a fix(vpn): default client profiles to MTU 1280; correct the VPN runbook
gate / gate (pull_request) Successful in 57s
gate / conformance (pull_request) Skipped
new-peer.sh generated MTU 1420, which stalled traffic on mobile data
despite a healthy handshake (one phone, one carrier, 2026-11-09). Default
to 1280, overridable through WG_MTU.

Correct stale claims in the runbook and the DaemonSet header against
live BPF conntrack: LoadBalancer VIPs see 10.90.0.x as world, not a
SNATed host identity. Add a troubleshooting section.

The DaemonSet render is unchanged, so WireGuard pods do not roll.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 21:15:20 -06:00
jakman85 da0515c8c8 Merge pull request 'fix(vpn): enrol keith-phone-home and admit the WireGuard subnet to PiHole DNS' (#158) from fix/wg-home-peer-pihole-sources into main
gate / gate (push) Successful in 55s
gate / conformance (push) Skipped
Reviewed-on: #158
2026-11-09 02:50:04 +00:00
jakman85 3a2ec5a12c Merge pull request 'feat(dns): PiHole answers an explicit LAN name list, not a zone wildcard' (#157) from feat/pihole-explicit-lan-names into main
gate / gate (push) Canceled after 25s
gate / conformance (push) Canceled after 0s
Reviewed-on: #157
Reviewed-by: Keith Jackson <[email protected]>
2026-11-09 02:49:39 +00:00
jakman85andClaude Opus 5.5 f13d336630 fix(vpn): enrol keith-phone-home and admit the WireGuard subnet to PiHole DNS
gate / gate (pull_request) Successful in 1m12s
gate / conformance (pull_request) Skipped
Add the at-home peer (10.90.0.3) and list ${WG_CLIENT_CIDR} in the PiHole
DNS frontends' loadBalancerSourceRanges. wg0 is a Cilium device, so tunnel
clients reach the source-range check untranslated; a LAN-only list is
predicted to drop their DNS. Adds an L0 guard that both ranges are present
and every range is RFC1918, and corrects the README's SNAT and IPv6 claims.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 20:45:48 -06:00
jakman85andClaude Opus 5.5 8fe8d5df8a feat(dns): PiHole answers an explicit LAN name list, not a zone wildcard
gate / gate (pull_request) Successful in 1m14s
gate / conformance (pull_request) Skipped
Public names are proxied CNAMEs to the Cloudflare Tunnel since PR #156 and
work from the LAN, so they should resolve publicly there too. Replace the
address=/${BASE_DOMAIN}/ wildcard with explicit entries: the LAN-only names
(derived from every controller:none HTTPRoute), gitea/nextcloud (off
Cloudflare's 100 MB / 100 s limits), keycloak/auth, and the L4 VIPs.

Add L0 check pihole-lan-names, mutation-tested. Correct docs that still
described public names as CNAME to wan. and the cloudflared entrypoint comment.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 19:51:49 -06:00
jakman85 020e736541 Merge pull request 'feat(dns): public names via the Cloudflare Tunnel (Phase 2) — proxied CNAMEs, Plex/Headlamp LAN-only' (#156) from feat/dns-via-cloudflare-tunnel into main
gate / gate (push) Successful in 40s
gate / conformance (push) Skipped
Reviewed-on: #156
Reviewed-by: Keith Jackson <[email protected]>
2026-11-09 01:27:06 +00:00
jakman85andClaude Opus 5.5 92944486d1 feat(dns): publish public names as proxied CNAMEs to the Cloudflare Tunnel
gate / gate (pull_request) Successful in 45s
gate / conformance (pull_request) Skipped
Gateway target -> fc81938b-...cfargotunnel.com, --cloudflare-proxied globally,
L4 service publishers (unifi, crafty-mc) pinned DNS-only, Plex web UI and
Headlamp disowned (LAN/VPN-only). L0 internal-services-publish-no-public-dns
gains assertion (9) tying the Gateway target to the tunnel UUID, requiring
cloudflare-proxied=false on every service-source publisher, and keeping the
tunnel's 404 list and the unpublished set consistent. L2 dns-exposure fixture
opts out of proxying.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 17:55:41 -06:00
jakman85 2e777e13ca Merge pull request 'chore(cloudflared): rotate tunnel credential (old secret exposed in a transcript)' (#155) from feature/cloudflare-tunnel-config into main
gate / gate (push) Successful in 41s
gate / conformance (push) Skipped
Reviewed-on: #155
2026-11-08 22:11:24 +00:00
jakman85 063c8acf77 chore(cloudflared): rotate tunnel credential (old secret exposed in a transcript)
gate / gate (pull_request) Successful in 41s
gate / conformance (pull_request) Skipped
2026-11-08 16:05:11 -06:00
jakman85 16222606ae Merge pull request 'fix(cloudflared): fsGroup so uid 65532 can read the 0440 credentials Secret' (#154) from fix/cloudflared-creds-permission into main
gate / gate (push) Successful in 40s
gate / conformance (push) Skipped
Reviewed-on: #154
Reviewed-by: Keith Jackson <[email protected]>
2026-11-08 21:57:46 +00:00
jakman85andClaude Opus 5.5 d9ab57f2d1 fix(cloudflared): fsGroup so uid 65532 can read the 0440 credentials Secret
gate / gate (pull_request) Successful in 42s
gate / conformance (pull_request) Skipped
The pod ran as 65532:65532 with no fsGroup, so the kubelet laid the Secret's
credentials.json down root:root 0440 — unreadable to the container. cloudflared
logged "permission denied" and then the red-herring "Error locating origin cert".

- deployment.yaml: securityContext.fsGroup 65532 (mode stays 0440)
- L0 cloudflared-ingress-rules (7): Secret volumes must be readable by the
  container's run-as uid/groups; mutation self-tested
- runbook: permission-denied troubleshooting entry, credential rotation section
- decisions/2026-11-08-cloudflared-creds-permission.md

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 15:49:56 -06:00
jakman85 9724445a08 Merge pull request 'Added cloudflare creds and tunnel config' (#153) from feature/cloudflare-tunnel-config into main
gate / gate (push) Successful in 40s
gate / conformance (push) Skipped
Reviewed-on: #153
2026-11-08 21:34:24 +00:00
jakman85 32b7d5bac1 Added cloudflare creds and tunnel config
gate / gate (pull_request) Successful in 42s
gate / conformance (pull_request) Skipped
2026-11-08 15:30:19 -06:00
jakman85 83e0ac194f Merge pull request 'feat(cloudflared): Cloudflare Tunnel connector, Phase 1 (isolated, no traffic change)' (#152) from feat/cloudflare-tunnel into main
gate / gate (push) Successful in 40s
gate / conformance (push) Skipped
Reviewed-on: #152
Reviewed-by: Keith Jackson <[email protected]>
2026-11-08 18:43:42 +00:00
jakman85andClaude Opus 5.5 09f219fe7f fix(cloudflared): auditable Service+ServiceMonitor scrape; placeholder-only template value
gate / gate (pull_request) Successful in 42s
gate / conformance (pull_request) Skipped
servicemonitor-ports-resolve could not audit a PodMonitor with no in-repo
Service, and no-plaintext-secrets requires a template's secret value to be a
bare placeholder.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 12:39:56 -06:00
jakman85andClaude Opus 5.5 aabc8c60b6 feat(cloudflared): Cloudflare Tunnel connector, Phase 1 (isolated, no traffic change)
Adds platform/cloudflared: a locally-managed tunnel whose ingress rules live in
git. LAN/VPN-only hosts (plex, plex-vip, jackett, pihole, pihole-dns, semaphore,
qbittorrent, radarr, sonarr, headlamp, unifi, crafty-mc) map to 404. The apex
and *.home go to cilium-gateway-shared.gateway over TLS with matchSNItoHost.
Last comes a catch-all 404. Nothing routes to the Kubernetes API.

- 2-replica Deployment, hard node spread, non-root, pinned digest, PDB
- default-deny plus an egress CNP: Cloudflare edge 7844/443, the Gateway, and
  Gateway backends by gateway-access=shared minus the LAN-only namespaces
  (needed because of the CVE-2025-30162 L7LB egress re-check)
- PodMonitor plus two alerts that fire on absent series, with promtool tests
- its own Flux Kustomization (dependsOn platform-config, wait:false, nothing
  depends on it), plus SOPS decryption patches in both overlays
- placeholder credentials (SOPS-encrypted) so it can merge before enrolment
- L0 cloudflared-ingress-rules, and a cloudflared_ scrape owner
- runbooks/cloudflare-tunnel.md and decisions/2026-11-08-cloudflare-tunnel.md

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 12:38:22 -06:00
renovate-bot eb62c13c6b Merge pull request 'chore(deps): Update gitea.home.keithjackson.dev/home-network/conformance-runner:main Docker digest to 63e1aa7' (#151) from renovate/gitea.home.keithjackson.dev-home-network-conformance-runner-main into main
gate / gate (push) Successful in 41s
gate / conformance (push) Skipped
2026-11-08 14:15:18 +00:00
renovate-bot 4226292dd6 Merge pull request 'chore(deps): Update gitea.home.keithjackson.dev/home-network/agent-runner:main Docker digest to 89ce7ee' (#150) from renovate/gitea.home.keithjackson.dev-home-network-agent-runner-main into main
gate / gate (push) Successful in 40s
gate / conformance (push) Skipped
2026-11-08 08:17:22 +00:00
renovate-bot 213429a41f chore(deps): Update gitea.home.keithjackson.dev/home-network/conformance-runner:main Docker digest to 63e1aa7
gate / gate (pull_request) Successful in 41s
gate / conformance (pull_request) Skipped
2026-11-08 08:16:28 +00:00
renovate-bot 0120bc10d4 chore(deps): Update gitea.home.keithjackson.dev/home-network/agent-runner:main Docker digest to 89ce7ee
gate / gate (pull_request) Successful in 52s
gate / conformance (pull_request) Skipped
2026-11-08 08:16:27 +00:00
jakman85 d1963bd000 Merge pull request 'chore(deps): Update Helm release longhorn to <1.14.0' (#131) from renovate/longhorn-1.x into main
gate / gate (push) Failing after 40s
gate / conformance (push) Skipped
Reviewed-on: #131
2026-11-08 06:55:55 +00:00
jakman85 0b0b5abc8e Merge pull request 'feat(ci): run L0/L1 offline checks, mutation suites and yamllint in bounded parallel pools' (#149) from feat/parallel-l0-checks into main
gate / gate (push) Successful in 39s
gate / conformance (push) Skipped
Reviewed-on: #149
Reviewed-by: Keith Jackson <[email protected]>
2026-11-08 06:53:42 +00:00
jakman85 6e2e353bcc Merge pull request 'feat(ci): run the merge gate inside the CI-built toolbox image, pinned by digest' (#147) from feat/gate-toolbox-container into main
gate / gate (push) Successful in 2m33s
gate / conformance (push) Skipped
Reviewed-on: #147
Reviewed-by: Keith Jackson <[email protected]>
2026-11-08 06:51:00 +00:00
jakman85 045d81c7a2 Merge pull request 'feat(gitea-actions): act_runner capacity 2, dind sized for two concurrent jobs' (#146) from feat/runner-capacity-2 into main
gate / gate (push) Canceled after 1m15s
gate / conformance (push) Canceled after 0s
Reviewed-on: #146
2026-11-08 06:47:05 +00:00
jakman85 7be693bf37 Merge pull request 'chore(deps): Update quay.io/oauth2-proxy/oauth2-proxy Docker tag to v7.15.6' (#136) from renovate/quay.io-oauth2-proxy-oauth2-proxy-7.x into main
gate / gate (push) Canceled after 1m32s
gate / conformance (push) Canceled after 0s
Reviewed-on: #136
Reviewed-by: Keith Jackson <[email protected]>
2026-11-08 06:45:31 +00:00
jakman85 9e9a3ea8cc Merge pull request 'fix(keycloak): one owner for the keycloak ServiceMonitor (26.8 operator 409 loop blocks Flux)' (#148) from fix/keycloak-servicemonitor-owner into main
gate / gate (push) Successful in 3m0s
gate / conformance (push) Skipped
Reviewed-on: #148
2026-11-08 06:37:13 +00:00
jakman85andClaude Opus 5.5 89f48e4a5f docs(decisions): handoff for parallel L0 checks; runner usage names --jobs
gate / gate (pull_request) Successful in 59s
gate / conformance (pull_request) Skipped
Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:34:48 -06:00
jakman85andClaude Opus 5.5 35ba875380 fix(keycloak): one owner for the keycloak ServiceMonitor (26.8 operator 409 loop blocks Flux)
gate / gate (pull_request) Successful in 3m2s
gate / conformance (pull_request) Skipped
The 26.x Keycloak operator creates its own ServiceMonitor named after the
CR (keycloak/keycloak) when metrics are enabled; spec.serviceMonitor.enabled
defaults true. That collides with the Flux-owned platform/keycloak/
servicemonitor.yaml: the operator's PATCH 409s, it exhausts retries, the CR
stays Ready=False/HasErrors=True, and platform-config (wait: true) stalls
every downstream Kustomization.

Keep the repo's ServiceMonitor (conformance L0/L2 and the dashboard are
written against it; the operator's scrapes the same port 9000 /metrics) and
set spec.serviceMonitor.enabled: false.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:25:45 -06:00
jakman85andClaude Opus 5.5 a775082ae1 feat(ci): run L0/L1 offline checks, mutation suites and yamllint in bounded parallel pools
- conformance/lib/run_l1_offline.py: checks run in a thread pool (--jobs / CONF_OFFLINE_JOBS,
  default min(8, usable CPUs)); results consumed in submission order, so log lines, JUnit,
  step summary and the 0/1/3 aggregation are unchanged. Exports a smaller MUTATION_JOBS to
  the children when parallel, so nested pools do not oversubscribe.
- keycloak-{passkey,client-secret}-reconciler/mutations.py: anchors checked first (serial),
  then the --selftest subprocesses run in a pool (MUTATION_JOBS, default min(8, CPUs));
  verdicts reported in MUTATIONS order. Output byte-identical to the serial loop.
- ci/lint.sh: yamllint over LINT_JOBS contiguous slices concurrently, per-slice output
  buffered and printed in order; stage rc = highest slice rc.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:18:57 -06:00
jakman85 1a33814522 Merge pull request 'chore(renovate): hand-migrate docker key, track gitea/runner, keep range lower bounds' (#145) from chore/renovate-config-cleanup into main
gate / gate (push) Successful in 2m59s
gate / conformance (push) Skipped
Reviewed-on: #145
2026-11-08 06:17:59 +00:00
jakman85 4a3f6dee59 Merge pull request 'fix(keycloak): install the 26.8.2 operator's new CRDs (operator CrashLoop)' (#144) from fix/keycloak-operator-crds into main
gate / gate (push) Canceled after 0s
gate / conformance (push) Canceled after 0s
Reviewed-on: #144
2026-11-08 06:16:56 +00:00
jakman85andClaude Opus 5.5 b548224db8 feat(gitea-actions): act_runner capacity 2, dind sized for two concurrent jobs
gate / gate (pull_request) Successful in 2m59s
gate / conformance (pull_request) Skipped
The in-cluster runner ran at the default capacity 1 with no config file, so
gate / images / renovate runs on different refs queued behind each other
(29-759s waits; run 472 waited 759s behind a renovate burst).

- ConfigMap act-runner-config with ONLY runner.capacity: 2, selected via
  CONFIG_FILE (honoured by the v0.6.1 image's run.sh). v0.6.1 LoadDefault()
  applies the same defaults to a partial file as to no file; the
  generate-config example is deliberately not used (its labels would override
  the registered ones, force_pull differs from the default).
- Pod-template config-revision annotation so ConfigMap-only edits can roll it.
- dind: cpu request 100m -> 2 (no limit), memory 256Mi/2Gi -> 1Gi/4Gi.
  Measured at capacity 1: peak 0.69 core, 447Mi working set (30d), 0 OOMs.
- runner: memory 128Mi/512Mi -> 256Mi/1Gi (peak 314Mi at capacity 1).

apps/wave-1/gitea-actions.yaml is a hand-owned EXTRA_UNITS file (the
renderer only places it in a wave), so it is the source, not generated output.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:09:08 -06:00
jakman85andClaude Opus 5.5 759a0e51c8 chore(renovate): hand-migrate docker key, track gitea/runner, keep range lower bounds
gate / gate (pull_request) Successful in 2m58s
gate / conformance (pull_request) Skipped
- Replace the deprecated top-level `docker` key with the packageRule Renovate's
  own migration produced (matchCategories docker, enabled true), keeping every
  comment. Supersedes #139, which dropped all comment lines.
- Add gitea/runner (act_runner's upstream rename) to the blast-radius rule.
- rangeStrategy bump for flux/helm `>=A <B` ranges: the default `auto` resolves
  to `replace`, which turned `>=1.12.0 <1.13.0` into `<1.14.0` (#131, #130,
  #115). In-range `bump` updates are disabled; Flux adopts those unaided.
- Restore headlamp's lower bound: `<0.47.0` -> `>=0.46.0 <0.47.0` (live 0.46.0).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:05:49 -06:00
jakman85andClaude Opus 5.5 ea30ba876c fix(keycloak): install the 26.8.2 operator's new CRDs (operator CrashLoop)
gate / gate (pull_request) Successful in 3m5s
gate / conformance (pull_request) Skipped
PR #121 bumped the three keycloak-k8s-resources URLs to 26.8.2, but 26.8.2
ships two more CRDs (keycloakoidcclients, keycloaksamlclients). The new
operator fails to start its keycloaksamlclients informer (Not Found) and
CrashLoops, holding Flux `platform` in health checks.

- add both CRD URLs with the same renovate marker/depName
- patch operator memory 450Mi -> limit 768Mi / request 512Mi (26.0.6 was
  OOMKilled at 421/450Mi, 154 restarts)
- README/renovate comments: Renovate cannot add CRDs a release introduces

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 00:00:01 -06:00
jakman85andClaude Opus 5.5 3d7ecbcfe9 fix(conformance): regenerate L1 render goldens for oauth2-proxy v7.15.6
gate / gate (pull_request) Successful in 3m0s
gate / conformance (pull_request) Skipped
PR #136 bumps the renderer's OAUTH2_PROXY_IMAGE; the container-workload
and egress-class (vpn + internet) goldens still pinned v7.8.1. Regenerated
with the check scripts' own invocation; each diff is the image tag only.
Also updates the crafty README's current-pin row.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-07 23:58:29 -06:00
renovate-bot 893c9dee4f chore(deps): Update quay.io/oauth2-proxy/oauth2-proxy Docker tag to v7.15.6
gate / gate (pull_request) Failing after 2m57s
gate / conformance (pull_request) Has been skipped
2026-11-07 08:18:51 +00:00
renovate-bot 619c2943da chore(deps): Update Helm release longhorn to <1.14.0
gate / gate (pull_request) Successful in 2m54s
gate / conformance (pull_request) Has been skipped
2026-11-07 08:17:40 +00:00
176 changed files with 13574 additions and 245 deletions

No files matched your search

+3
View File
@@ -164,11 +164,14 @@ Cost: the job shares the single act_runner slot with the gate, but the build its
.gitea/workflows/ .gitea/workflows/
merge-gate.yaml the pipeline (lint → l0 → l1 → conformance → decision) merge-gate.yaml the pipeline (lint → l0 → l1 → conformance → decision)
images.yaml build/verify/publish this repo's container images (see above) images.yaml build/verify/publish this repo's container images (see above)
cve-triage.yaml scheduled CVE triage bot (R-VULN-05/07, ADR-0027; runs before Renovate)
scripts/ scripts/
install-tools.sh installs the pinned tools per stage (from ci/versions.env) — used by the install-tools.sh installs the pinned tools per stage (from ci/versions.env) — used by the
dormant merge-gate.yaml blueprint only; gate.yml runs in the toolbox image dormant merge-gate.yaml blueprint only; gate.yml runs in the toolbox image
decide.sh the R-TEST-03 merge decision over job results decide.sh the R-TEST-03 merge decision over job results
images.sh the image pipeline's driver (what to build, base, push, digest) images.sh the image pipeline's driver (what to build, base, push, digest)
cve_triage.py the CVE triage bot: reads the scanner report, maintains the counts-only
CVE Dashboard issue, ticks Renovate approval rows (cve-triage.yaml)
README.md this file README.md this file
``` ```
+159
View File
@@ -0,0 +1,159 @@
# CVE triage bot (R-VULN-05, R-VULN-07; docs/17 "The triage bot"; ADR-0027). Same shape as
# renovate.yaml: scheduled, loud on a missing credential, never concurrent with itself, and it
# records what it did in the job summary. The logic and its reasoning live in
# scripts/cve_triage.py; this file only supplies the image, the code, the credential and the
# schedule.
#
# WHY IT CANNOT SUCCEED SILENTLY. A skipped Gitea job reports `success` (CLAUDE.md §2), so the
# token preflight exits 1 rather than skipping, and an unreachable scanner fails the script
# rather than skipping. The heartbeat POST is the script's last action, so VulnTriageBotSilent
# fires for every run that did not get all the way through, and when the schedule stops firing.
#
# THE ONE SECRET: RENOVATE_TOKEN, deliberately reused (ADR-0027). It acts as renovate-bot, which
# is also what authored the dashboard this edits. It needs read:repository and write:issue.
name: cve-triage
on:
schedule:
# 07:15 UTC, one hour before Renovate's 08:15 (renovate.yaml), so a tick is read the same
# morning, and clear of its run so the bot's edit cannot race Renovate's rewrite of the body.
- cron: "15 7 * * *"
workflow_dispatch:
inputs:
dry_run:
description: "Dry run: read everything, write nothing, print counts."
required: false
default: "true"
concurrency:
group: cve-triage
cancel-in-progress: false
jobs:
triage:
runs-on: ubuntu-latest
# THE TOOLBOX, the image gate.yml already runs, pinned by digest. A job here has no Docker
# daemon, so a job container is the only way to choose an image. The toolbox provides
# python3, git, curl and jq, pulls anonymously like the gate's, and is normally already in
# the DinD store. It has no Node, so actions/checkout cannot run; a public python image has
# neither Node nor git and would add a second image to keep pinned. The cost is a cold pull
# after a runner restart, which the gate pays anyway.
container:
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/toolbox tag=main
image: gitea.home.keithjackson.dev/home-network/toolbox@sha256:83a3501efe34fa0b039e3f0d360738686a0f59109866c49779768f5d5ed0ef81
# Without a bound, a wedged run would hold the concurrency group and block every later run.
timeout-minutes: 15
steps:
- name: Preflight - the token must exist, and its absence must be LOUD
shell: bash
env:
RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
run: |
set -euo pipefail
if [ -z "${RENOVATE_TOKEN:-}" ]; then
echo "::error::RENOVATE_TOKEN is not set on this repository." >&2
echo "A skipped job reports success, so this fails on purpose: without the token no dashboard is maintained and no approval is requested. Add the Actions secret RENOVATE_TOKEN (scopes read:repository, write:issue)." >&2
exit 1
fi
echo "RENOVATE_TOKEN is present (value not printed)."
- name: Preflight - Gitea is reachable and the token can do this job
shell: bash
env:
RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
# GETs only. The token reaches curl on stdin, never argv. A 401 is the token's value, a 403
# its scope; the script's own errors would only show a bare status later.
run: |
set -uo pipefail
GITEA=https://gitea.home.keithjackson.dev
REPO=Home-Network/home-network-k3s-impl
fail=0
body=$(mktemp)
fetch() {
printf 'header = "Authorization: Bearer %s"\n' "$RENOVATE_TOKEN" \
| curl -sS --max-time 30 --config - -o "$body" -w '%{http_code}' "$GITEA$1" || true
}
probe() { # <what it proves> <path>
local code
code=$(fetch "$2")
if [ "$code" = 200 ]; then
echo "ok $1"
else
fail=1
echo "::error::$1: GET ${2%%\?*} -> $code"
fi
}
probe "read:repository, Gitea reachable" "/api/v1/repos/$REPO"
if [ "$(jq -r '.permissions.push // false' "$body" 2>/dev/null)" != "true" ]; then
fail=1
echo "::error::renovate-bot has no write access to $REPO, so it cannot create labels or edit issues."
fi
probe "read:issue" "/api/v1/repos/$REPO/labels?limit=1"
probe "Renovate's dashboard readable" "/api/v1/repos/$REPO/issues/91"
# A GET cannot prove a write scope; Gitea's scope refusal lists the token's scopes
# (names only). Same technique as renovate.yaml, advisory if the message format changes.
fetch "/api/v1/admin/orgs?limit=1" > /dev/null
scopes=$(jq -r '.message // empty' "$body" 2>/dev/null | sed -n 's/.*token scope=//p')
if [ -n "$scopes" ]; then
case ",$scopes," in
*",write:issue,"*|*",all,"*) echo "ok write:issue" ;;
*) echo "::error::scope write:issue is missing (token scope=$scopes)"; fail=1 ;;
esac
else
echo "::warning::could not read the token's scope list; write:issue is UNVERIFIED"
fi
rm -f "$body"
[ "$fail" -eq 0 ]
# Plain git for the same reason as gate.yml: the toolbox has no Node. One commit is enough,
# and the job token goes to the single fetch only, never into .git/config.
- name: Checkout the script (git, one commit)
shell: bash
env:
GIT_TOKEN: ${{ github.token }}
SERVER_URL: ${{ github.server_url }}
REPOSITORY: ${{ github.repository }}
SHA: ${{ github.sha }}
run: |
set -euo pipefail
umask 0022
git config --global --add safe.directory "$PWD"
git init -q .
git remote add origin "${SERVER_URL}/${REPOSITORY}"
auth="$(printf 'x-access-token:%s' "$GIT_TOKEN" | base64 -w0)"
GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=http.extraHeader \
GIT_CONFIG_VALUE_0="AUTHORIZATION: basic ${auth}" \
git -c protocol.version=2 fetch --quiet --depth=1 --no-tags origin \
"+${SHA}:refs/remotes/checkout/head"
unset auth GIT_TOKEN
git checkout -q --force --detach "$SHA"
[ "$(git rev-parse HEAD)" = "$SHA" ]
- name: Triage
shell: bash
env:
RENOVATE_TOKEN: ${{ secrets.RENOVATE_TOKEN }}
GITEA_URL: https://gitea.home.keithjackson.dev
GITEA_REPOSITORY: Home-Network/home-network-k3s-impl
RENOVATE_DASHBOARD_ISSUE: "91"
VULN_API_URL: http://vuln-scanner.vuln-scanner.svc.cluster.local:8081
# The Service's pinned spec.clusterIP (platform/vuln-scanner/service.yaml), used only
# when the name above fails to resolve. Keep the two in step.
VULN_API_FALLBACK_URL: http://10.43.210.10:8081
VULN_PAGE_URL: https://cve.home.keithjackson.dev
# Anything other than an explicit false on a dispatch is a dry run; a schedule is live.
DRY_RUN: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run != 'false' && 'true' || 'false' }}
run: python3 .gitea/workflows/scripts/cve_triage.py
- name: Summary
if: always()
run: |
{
echo "## CVE triage"
echo
echo "- trigger: \`${{ github.event_name }}\`"
echo "- dry run requested: \`${{ github.event_name == 'workflow_dispatch' && inputs.dry_run != 'false' && 'true' || 'false' }}\`"
echo "- outcome: \`${{ job.status }}\`"
echo
echo "Counts only: which CVE lands where is on the CVE page, never here."
} >> "$GITHUB_STEP_SUMMARY"
+820
View File
@@ -0,0 +1,820 @@
#!/usr/bin/env python3
"""CVE triage bot (docs/17 "The triage bot", ADR-0027; R-VULN-05, R-VULN-07).
Runs in .gitea/workflows/cve-triage.yaml, once a day, before Renovate. Each run:
1. reads report.json from the scanner's internal API;
2. creates or updates the single `CVE Dashboard` issue (aggregates only);
3. reads Renovate's Dependency Dashboard and open PRs, derives each image's remediation state;
4. ticks `approve-branch` boxes on that dashboard, for packages matching an image with an
unaccepted CRITICAL that has a fix;
5. POSTs the heartbeat and the remediation states to the scanner.
WHAT IT MUST NEVER DO. Renovate is the only author of version changes (R-UPD-02): this script
never edits a version, opens a bump PR, merges anything, or posts `/approve-override`. It never
ticks a `rebase-branch` row. The dashboard issue carries no CVE id, package, image or workload,
because the repository is readable without authentication, and no label the in-cluster PM acts
on (platform/pm-agent/configmap.yaml PM_REQUIRE_LABEL = `agent`).
OUTPUT IS COUNTS ONLY. Job logs and the summary are as visible as the issue, so nothing here
prints a report string. Every report value is untrusted: strings are validated, and only counts
and timestamps that passed RFC 3339 validation ever reach markdown.
HOW A JOB CONTAINER REACHES THE SCANNER, AND WHAT IS NOT PROVEN
---------------------------------------------------------------
Jobs are containers on a per-job docker bridge network inside the DinD sidecar of the act-runner
pod (apps/wave-1/gitea-actions.yaml). The runner config sets no `container.network` and dockerd no
`--dns`, so name resolution is docker's embedded resolver forwarding to the resolvers of the
dockerd container, which are that pod's: cluster DNS. Egress is masqueraded to the pod IP
(measured 2026-10-29, platform/buildkit/networkpolicy.yaml header), so a ClusterIP is routed and
policy-matched as the act-runner pod in gitea-actions. images.yaml relies on the same path to
reach buildkitd.buildkit.svc.cluster.local, which is indirect evidence only.
UNVERIFIED: in-cluster DNS from a DinD job container.
UNVERIFIED: ClusterIP reachability from a DinD job container to vuln-scanner:8081.
Hence VULN_API_FALLBACK_URL (the Service's pinned ClusterIP), used only when the name fails to
resolve. Anything else, including refused and timed-out connections, fails the job: an
unreachable API is a red run, never a skip.
"""
from __future__ import annotations
import http.client
import json
import os
import re
import socket
import sys
import urllib.error
import urllib.parse
import urllib.request
from datetime import datetime, timedelta, timezone
from typing import Callable, Iterable, NamedTuple
DASHBOARD_TITLE = "CVE Dashboard"
DASHBOARD_LABEL = "cve-dashboard"
DASHBOARD_LABEL_COLOR = "b60205"
# Labels the in-cluster PM dispatcher selects work by. Never applied to anything here.
FORBIDDEN_LABELS = frozenset({"agent"})
RENOVATE_DASHBOARD_TITLE = "Dependency Dashboard"
SEVERITIES = ("CRITICAL", "HIGH", "MEDIUM", "LOW", "UNKNOWN")
FINDING_STATUSES = ("open", "fix-available", "accepted")
TARGET_STATES = ("scanned", "pending", "stale", "failed", "credential-required", "by-tag", "not-covered")
TARGET_TYPES = ("image", "vm", "node")
RUN_STATUSES = ("complete", "degraded")
REMEDIATION_STATES = ("pr-open", "approval-requested", "no-update-identified", "not-tracked")
MAX_REPORT_BYTES = 128 * 1024 * 1024
MAX_API_BYTES = 8 * 1024 * 1024
MAX_FIELD_LEN = 4096
MAX_IMAGE_LEN = 512
MAX_ISSUE_BODY = 2 * 1024 * 1024
MAX_DASHBOARD_ROWS = 2000
MAX_DETECTED_DEPS = 20000
MAX_REMEDIATION = 5000
MAX_PAGES = 200
PAGE_SIZE = 50
REPORT_STALE_HOURS = 26
DATABASE_STALE_HOURS = 48
HTTP_TIMEOUT = 60
RFC3339_RE = re.compile(r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(\.\d{1,9})?(Z|[+-]\d{2}:\d{2})")
IMAGE_RE = re.compile(r"[a-z0-9]([a-z0-9._:/-]*[a-z0-9])?")
HTTP_URL_RE = re.compile(r"https?://[A-Za-z0-9.-]+(:\d{1,5})?(/[A-Za-z0-9._~%/-]*)?")
REPO_RE = re.compile(r"[A-Za-z0-9._-]+/[A-Za-z0-9._-]+")
# Renovate's own row shape (lib/workers/repository/dependency-dashboard.ts, getListItem). The kind
# is captured whole so `approvePr-branch` can never read as `approve-branch`.
ROW_RE = re.compile(r" - \[(?P<mark>[ xX])\] <!-- (?P<kind>[A-Za-z]+)-branch=(?P<branch>\S+) -->(?P<rest>.*)")
LINKED_TITLE_RE = re.compile(r"\[(?P<title>.*)\]\((?P<target>[^()\s]*)\)(?P<tail>.*)")
PR_LINK_RE = re.compile(r"(?:^|/)pulls?/(\d{1,9})$")
PACKAGE_LIST_RE = re.compile(r"\s\(((?:`[^`\s]+`(?:, )?)+)\)\s*$")
TITLE_PACKAGE_RE = re.compile(r"(?:^|\s)(?:Update|Pin)\s+(\S+)\s+Docker\s+(?:tag|digest)\b")
SUMMARY_RE = re.compile(r"<details><summary>([^<]*?)(?: \(\d+\))?</summary>")
DETECTED_RE = re.compile(r" - `(?P<name>[^\s`]+)(?: [^`]*)?`(?P<updates>\s*→ \[Updates:.*\])?\s*")
# Detected-dependency managers whose names are charts or actions, never container images.
NON_IMAGE_MANAGERS = frozenset({"flux", "helmv3", "github-actions"})
class BotError(Exception):
"""A failure that ends the run. The message must be safe to print: no report text, no secret."""
class DnsFailure(Exception):
"""The hostname did not resolve. The only condition that selects the fallback address."""
# ---------------------------------------------------------------- pure: normalisation + report
def normalise_image(value: object) -> str:
"""Canonical form for exact comparison, or "" when the value is not an image reference.
Lowercase; drop digest and tag; drop the Docker Hub registry and the `library/` namespace.
`library/` is only Hub's implicit namespace, so ghcr.io/library/x keeps it. Comparison of the
result is always string equality, never a substring test.
"""
if not isinstance(value, str) or not value or len(value) > MAX_IMAGE_LEN:
return ""
name = value.strip().lower().split("@", 1)[0]
head, sep, last = name.rpartition("/")
name = f"{head}{sep}{last.split(':', 1)[0]}"
for hub in ("docker.io/", "index.docker.io/", "registry-1.docker.io/"):
if name.startswith(hub):
name = name[len(hub):]
break
if name.startswith("library/"):
name = name[len("library/"):]
return name if IMAGE_RE.fullmatch(name) else ""
def parse_timestamp(value: object, field: str, nullable: bool = False) -> datetime | None:
if value is None and nullable:
return None
if not isinstance(value, str) or not RFC3339_RE.fullmatch(value):
raise BotError(f"report: {field} is not an RFC 3339 timestamp")
text = re.sub(r"\.\d+", "", value).replace("Z", "+00:00")
try:
return datetime.fromisoformat(text)
except ValueError as exc:
raise BotError(f"report: {field} is not a valid timestamp") from exc
def _object(value: object, field: str) -> dict:
if not isinstance(value, dict):
raise BotError(f"report: {field} is not an object")
return value
def _array(value: object, field: str) -> list:
if not isinstance(value, list):
raise BotError(f"report: {field} is not an array")
return value
def _string(value: object, field: str, nullable: bool = False) -> str | None:
if value is None and nullable:
return None
if not isinstance(value, str) or len(value) > MAX_FIELD_LEN:
raise BotError(f"report: {field} is not a bounded string")
return value
def _member(value: object, allowed: Iterable[str], field: str) -> str:
if not isinstance(value, str) or value not in allowed:
raise BotError(f"report: {field} has a value outside its enumeration")
return value
def parse_report(raw: bytes) -> dict:
"""Decode and validate report.json into the only shape the rest of this file reads."""
try:
doc = json.loads(raw)
except (ValueError, RecursionError) as exc:
raise BotError("report: body is not valid JSON") from exc
doc = _object(doc, "document")
if doc.get("schemaVersion") != 1 or isinstance(doc.get("schemaVersion"), bool):
raise BotError("report: unsupported schemaVersion")
parse_timestamp(doc.get("generatedAt"), "generatedAt")
database = _object(doc.get("database"), "database")
run = _object(doc.get("run"), "run")
parse_timestamp(database.get("updatedAt"), "database.updatedAt", nullable=True)
parse_timestamp(run.get("startedAt"), "run.startedAt", nullable=True)
parse_timestamp(run.get("finishedAt"), "run.finishedAt", nullable=True)
if run.get("status") is not None:
_member(run.get("status"), RUN_STATUSES, "run.status")
targets = []
known_ids = {}
for entry in _array(doc.get("targets"), "targets"):
entry = _object(entry, "target")
target_id = _string(entry.get("id"), "target.id")
kind = _member(entry.get("type"), TARGET_TYPES, "target.type")
state = _member(entry.get("state"), TARGET_STATES, "target.state")
image = _string(entry.get("image"), "target.image", nullable=True)
if kind == "image" and not normalise_image(image):
raise BotError("report: an image target has an invalid image reference")
known_ids[target_id] = image if kind == "image" else None
targets.append({"id": target_id, "type": kind, "state": state, "image": image})
findings = []
for entry in _array(doc.get("findings"), "findings"):
entry = _object(entry, "finding")
target_id = _string(entry.get("targetId"), "finding.targetId")
if target_id not in known_ids:
raise BotError("report: a finding references an unknown target")
findings.append({
"targetId": target_id,
"image": known_ids[target_id],
"severity": _member(entry.get("severity"), SEVERITIES, "finding.severity"),
"status": _member(entry.get("status"), FINDING_STATUSES, "finding.status"),
"fixedVersion": _string(entry.get("fixedVersion"), "finding.fixedVersion", nullable=True),
})
return {
"generatedAt": doc["generatedAt"],
"database": {"updatedAt": database.get("updatedAt")},
"run": {"finishedAt": run.get("finishedAt"), "status": run.get("status")},
"targets": targets,
"findings": findings,
}
def qualifies(finding: dict) -> bool:
"""An unaccepted CRITICAL with a fix: the only thing that may cause a tick."""
return (
finding["severity"] == "CRITICAL"
and finding["status"] == "fix-available"
and bool(finding["fixedVersion"])
)
def qualifying_images(report: dict) -> set[str]:
return {
norm
for finding in report["findings"]
if qualifies(finding) and (norm := normalise_image(finding["image"]))
}
def aggregate(report: dict) -> dict:
severity = {sev: dict.fromkeys(FINDING_STATUSES, 0) for sev in SEVERITIES}
for finding in report["findings"]:
severity[finding["severity"]][finding["status"]] += 1
coverage = dict.fromkeys(TARGET_STATES, 0)
for target in report["targets"]:
coverage[target["state"]] += 1
return {
"severity": severity,
"coverage": coverage,
"targets_total": len(report["targets"]),
"critical_with_fix": sum(1 for f in report["findings"] if qualifies(f)),
"generatedAt": report["generatedAt"],
"databaseUpdatedAt": report["database"]["updatedAt"],
"runFinishedAt": report["run"]["finishedAt"],
"runStatus": report["run"]["status"],
}
def render_dashboard(agg: dict, page_url: str, now: datetime) -> str:
"""The CVE Dashboard issue body: a fixed template over counts and validated timestamps."""
if not HTTP_URL_RE.fullmatch(page_url):
raise BotError("config: VULN_PAGE_URL is not a plain http(s) URL")
warnings = []
generated = parse_timestamp(agg["generatedAt"], "generatedAt")
if now - generated > timedelta(hours=REPORT_STALE_HOURS):
warnings.append(f"The report is older than {REPORT_STALE_HOURS} hours.")
updated = parse_timestamp(agg["databaseUpdatedAt"], "database.updatedAt", nullable=True)
if updated is None or now - updated > timedelta(hours=DATABASE_STALE_HOURS):
warnings.append(f"The vulnerability database is older than {DATABASE_STALE_HOURS} hours or unknown.")
if agg["runStatus"] != "complete":
warnings.append("The latest scan run did not complete cleanly.")
lines = [
"Maintained by the CVE triage bot, once a day. **Counts only**: this repository is readable",
"without authentication, so which CVE lands where is only on the",
f"[CVE page]({page_url}) (SSO, platform-admins).",
"",
]
lines += [f"> **WARNING:** {text}" for text in warnings]
if warnings:
lines.append("")
lines += [
"## Freshness",
"",
"| | |",
"|---|---|",
f"| Report generated | {agg['generatedAt']} |",
f"| Database updated | {agg['databaseUpdatedAt'] or 'unknown'} |",
f"| Last run finished | {agg['runFinishedAt'] or 'none yet'} |",
f"| Last run status | {agg['runStatus'] or 'none yet'} |",
f"| Dashboard rendered | {now.strftime('%Y-%m-%dT%H:%M:%SZ')} |",
"",
"## Findings by severity",
"",
"| Severity | Open | Fix available | Accepted |",
"|---|---:|---:|---:|",
]
for sev in SEVERITIES:
row = agg["severity"][sev]
lines.append(f"| {sev} | {row['open']} | {row['fix-available']} | {row['accepted']} |")
lines += [
"",
f"Unaccepted CRITICAL with a fix: **{agg['critical_with_fix']}**",
"",
"## Coverage",
"",
f"{agg['targets_total']} targets. A count in any state other than `scanned` is something"
" that was not freshly examined, and is not a clean result.",
"",
"| State | Targets |",
"|---|---:|",
]
lines += [f"| {state} | {agg['coverage'][state]} |" for state in TARGET_STATES]
return "\n".join(lines) + "\n"
# ---------------------------------------------------------------- pure: Renovate's dashboard
class Row(NamedTuple):
kind: str
mark: str
branch: str
packages: tuple[str, ...]
pr_number: int | None
offset: int # offset in the body of the first character of the line
def _title_package(title: str) -> str:
match = TITLE_PACKAGE_RE.search(title)
return normalise_image(match.group(1)) if match else ""
def _row_details(rest: str) -> tuple[tuple[str, ...], int | None]:
"""Packages and PR number of a row. No recognisable package means no packages, so no tick."""
rest = rest.rstrip("\r")
number = None
title = rest
tail = rest
linked = LINKED_TITLE_RE.fullmatch(rest)
if linked:
title, tail = linked.group("title"), linked.group("tail")
link = PR_LINK_RE.search(linked.group("target"))
number = int(link.group(1)) if link else None
listed = PACKAGE_LIST_RE.search(tail)
if listed:
names = re.findall(r"`([^`]+)`", listed.group(1))
packages = tuple(n for n in map(normalise_image, names) if n)
else:
package = _title_package(title)
packages = (package,) if package else ()
return packages, number
def _is_detected_heading(line: str) -> bool:
return line.startswith("## ") and line[3:].strip().lower().startswith("detected dependencies")
def parse_rows(body: str) -> list[Row]:
"""Checkbox rows above `## Detected Dependencies`. Only a whole-line match counts."""
if len(body) > MAX_ISSUE_BODY:
raise BotError("renovate dashboard: body exceeds its bound")
rows: list[Row] = []
offset = 0
for line in body.split("\n"):
if _is_detected_heading(line):
break
match = ROW_RE.fullmatch(line.rstrip("\r"))
if match:
packages, number = _row_details(match.group("rest"))
rows.append(Row(match.group("kind"), match.group("mark"), match.group("branch"),
packages, number, offset))
if len(rows) > MAX_DASHBOARD_ROWS:
raise BotError("renovate dashboard: too many rows")
offset += len(line) + 1
return rows
def parse_detected(body: str) -> dict[str, bool]:
"""{normalised depName: has a pending update} from `## Detected Dependencies`."""
tracked: dict[str, bool] = {}
lines = body.split("\n")
in_section = False
manager = ""
count = 0
for index, raw in enumerate(lines):
line = raw.rstrip("\r")
if line.startswith("## "):
in_section = _is_detected_heading(line)
continue
if not in_section:
continue
summary = SUMMARY_RE.fullmatch(line)
if summary:
# A manager heading is the summary directly followed by <blockquote>; a file heading
# is not.
if index + 1 < len(lines) and lines[index + 1].strip() == "<blockquote>":
manager = summary.group(1)
continue
match = DETECTED_RE.fullmatch(line)
if not match or manager in NON_IMAGE_MANAGERS:
continue
count += 1
if count > MAX_DETECTED_DEPS:
raise BotError("renovate dashboard: too many detected dependencies")
norm = normalise_image(match.group("name"))
if norm:
tracked[norm] = tracked.get(norm, False) or bool(match.group("updates"))
return tracked
def row_matches(row: Row, qualifying: set[str]) -> bool:
return any(package in qualifying for package in row.packages)
def select_ticks(rows: list[Row], qualifying: set[str]) -> list[Row]:
return [
row for row in rows
if row.kind == "approve" and row.mark == " " and row_matches(row, qualifying)
]
def apply_ticks(body: str, ticks: list[Row]) -> str:
"""Replace only the `[ ]` of each row, re-verifying the row where it stands.
Every byte other than the three of each marker is carried over unchanged.
"""
pieces = []
cursor = 0
for row in sorted(ticks, key=lambda r: r.offset):
expected = f" - [ ] <!-- approve-branch={row.branch} -->"
if row.kind != "approve" or not body.startswith(expected, row.offset):
raise BotError("renovate dashboard: a row changed between parse and write")
marker = row.offset + len(" - ")
pieces += [body[cursor:marker], "[x]"]
cursor = marker + len("[ ]")
pieces.append(body[cursor:])
return "".join(pieces)
def derive_remediation(
images: Iterable[str],
rows: list[Row],
open_prs: list[dict],
tracked: dict[str, bool],
pr_url: Callable[[int], str],
now_text: str,
) -> list[dict]:
"""One entry per report image.
pr-open: an open-PR row or an open Renovate PR for that package. approval-requested: an
approve row. no-update-identified: Renovate tracks it and lists no update. not-tracked:
everything else, which deliberately includes "Renovate knows an update but no PR or approval
row exists" (awaiting schedule, rate limited): saying no update was identified would be
false, and the enumeration has no closer value, so it lands on the state that draws attention.
"""
by_branch = {pr["branch"]: pr["number"] for pr in open_prs}
result = []
for image in sorted(set(images)):
norm = normalise_image(image)
state = "not-tracked"
url = None
rebase = next((r for r in rows if r.kind == "rebase" and norm in r.packages), None)
pr = next((p for p in open_prs if p["package"] == norm), None)
if rebase is not None or pr is not None:
state = "pr-open"
number = None
if rebase is not None:
number = rebase.pr_number or by_branch.get(rebase.branch)
if number is None and pr is not None:
number = pr["number"]
url = pr_url(number) if number is not None else None
elif any(r.kind == "approve" and norm in r.packages for r in rows):
state = "approval-requested"
elif norm in tracked and not tracked[norm]:
state = "no-update-identified"
result.append({"image": image, "state": state, "url": url, "updatedAt": now_text})
return result
def count_states(remediation: list[dict]) -> dict[str, int]:
counts = dict.fromkeys(REMEDIATION_STATES, 0)
for entry in remediation:
counts[entry["state"]] += 1
return counts
# ---------------------------------------------------------------- I/O
class _NoRedirect(urllib.request.HTTPRedirectHandler):
"""A redirect would carry the Authorization header to wherever it points."""
def redirect_request(self, *args, **kwargs):
return None
class UrllibTransport:
"""(method, url, headers, body, max_bytes) -> (status, bytes); never uses a proxy."""
def __init__(self) -> None:
self._opener = urllib.request.build_opener(urllib.request.ProxyHandler({}), _NoRedirect())
def __call__(self, method: str, url: str, headers: dict, body: bytes | None, max_bytes: int):
request = urllib.request.Request(url, data=body, method=method, headers=headers)
try:
with self._opener.open(request, timeout=HTTP_TIMEOUT) as response:
data = response.read(max_bytes + 1)
return response.status, data
except urllib.error.HTTPError as exc:
return exc.code, b""
except urllib.error.URLError as exc:
if isinstance(exc.reason, socket.gaierror):
raise DnsFailure() from exc
raise BotError(f"{method} {urllib.parse.urlsplit(url).path}: {type(exc.reason).__name__}") from exc
except (OSError, http.client.HTTPException) as exc:
raise BotError(f"{method} {urllib.parse.urlsplit(url).path}: {type(exc).__name__}") from exc
class GiteaClient:
def __init__(self, base_url: str, repo: str, token: str, transport) -> None:
self._api = f"{base_url.rstrip('/')}/api/v1/repos/{repo}"
self._web = f"{base_url.rstrip('/')}/{repo}"
self._token = token
self._transport = transport
def _call(self, method: str, path: str, query: dict | None = None, payload=None, expect=(200,)):
url = self._api + path + (f"?{urllib.parse.urlencode(query)}" if query else "")
headers = {"Authorization": f"token {self._token}", "Accept": "application/json"}
data = None
if payload is not None:
data = json.dumps(payload).encode()
headers["Content-Type"] = "application/json"
try:
status, raw = self._transport(method, url, headers, data, MAX_API_BYTES)
except DnsFailure as exc:
raise BotError(f"gitea: {method} {path}: name resolution failed") from exc
if status not in expect:
raise BotError(f"gitea: {method} {path} -> HTTP {status}")
if len(raw) > MAX_API_BYTES:
raise BotError(f"gitea: {method} {path}: response exceeds its bound")
if not raw:
return None
try:
return json.loads(raw)
except ValueError as exc:
raise BotError(f"gitea: {method} {path}: response is not JSON") from exc
def _list(self, path: str, query: dict | None = None) -> list:
items: list = []
for page in range(1, MAX_PAGES + 1):
batch = self._call("GET", path, {**(query or {}), "limit": PAGE_SIZE, "page": page})
if not isinstance(batch, list):
raise BotError(f"gitea: GET {path}: expected a list")
items += batch
if len(batch) < PAGE_SIZE:
return items
raise BotError(f"gitea: GET {path}: more than {MAX_PAGES} pages")
def pr_url(self, number: int) -> str:
return f"{self._web}/pulls/{int(number)}"
def labels(self) -> list:
return self._list("/labels")
def create_label(self, name: str, color: str) -> dict:
return self._call("POST", "/labels", payload={"name": name, "color": color}, expect=(201,))
def issues_with_label(self, label: str) -> list:
return self._list("/issues", {"state": "all", "type": "issues", "labels": label})
def issues_matching(self, text: str) -> list:
return self._list("/issues", {"state": "all", "type": "issues", "q": text})
def create_issue(self, title: str, body: str, label_id: int) -> dict:
return self._call("POST", "/issues", payload={"title": title, "body": body, "labels": [label_id]}, expect=(201,))
def update_issue(self, number: int, fields: dict) -> None:
self._call("PATCH", f"/issues/{int(number)}", payload=fields, expect=(200, 201))
def add_issue_label(self, number: int, label_id: int) -> None:
self._call("POST", f"/issues/{int(number)}/labels", payload={"labels": [label_id]}, expect=(200,))
def get_issue(self, number: int) -> dict:
issue = self._call("GET", f"/issues/{int(number)}")
if not isinstance(issue, dict):
raise BotError("gitea: issue response is not an object")
return issue
def open_renovate_prs(self) -> list[dict]:
result = []
for pr in self._list("/pulls", {"state": "open"}):
head = pr.get("head") if isinstance(pr, dict) else None
ref = head.get("ref") if isinstance(head, dict) else None
number = pr.get("number") if isinstance(pr, dict) else None
title = pr.get("title") if isinstance(pr, dict) else None
if not isinstance(ref, str) or not isinstance(number, int) or isinstance(number, bool):
raise BotError("gitea: a pull request has an unexpected shape")
if ref.startswith("renovate/"):
package = _title_package(title) if isinstance(title, str) else ""
result.append({"number": number, "branch": ref, "package": package})
return result
class ScannerClient:
"""The scanner's internal API. Falls back to the pinned ClusterIP on a DNS failure only."""
def __init__(self, primary: str, fallback: str, transport) -> None:
self._bases = [base.rstrip("/") for base in (primary, fallback) if base]
self._transport = transport
self.used_fallback = False
def _call(self, method: str, path: str, body: bytes | None, headers: dict, max_bytes: int):
last = len(self._bases) - 1
for index, base in enumerate(self._bases):
try:
result = self._transport(method, base + path, headers, body, max_bytes)
except DnsFailure as exc:
if index == last:
raise BotError(f"scanner: {method} {path}: name resolution failed") from exc
continue
if index:
self.used_fallback = True
self._bases = self._bases[index:]
return result
raise BotError("scanner: no address configured")
def report(self) -> dict:
status, raw = self._call("GET", "/report.json", None, {"Accept": "application/json"}, MAX_REPORT_BYTES)
if status != 200:
raise BotError(f"scanner: GET /report.json -> HTTP {status}")
if len(raw) > MAX_REPORT_BYTES:
raise BotError("scanner: report exceeds its bound")
return parse_report(raw)
def triage(self, heartbeat: str, remediation: list[dict]) -> None:
if len(remediation) > MAX_REMEDIATION:
raise BotError("scanner: more remediation entries than POST /triage accepts")
body = json.dumps({"heartbeatAt": heartbeat, "remediation": remediation}).encode()
status, _ = self._call("POST", "/triage", body, {"Content-Type": "application/json"}, 4096)
if status != 204:
raise BotError(f"scanner: POST /triage -> HTTP {status}")
# ---------------------------------------------------------------- orchestration
def _label_names(issue: dict) -> set:
return {label.get("name") for label in (issue.get("labels") or []) if isinstance(label, dict)}
def find_dashboard_issues(gitea: GiteaClient) -> list[dict]:
"""Every issue that is the dashboard: by label, and by exact title in case a label was removed."""
found: dict[int, dict] = {}
for issue in gitea.issues_with_label(DASHBOARD_LABEL) + gitea.issues_matching(DASHBOARD_TITLE):
if (
isinstance(issue, dict)
and issue.get("title") == DASHBOARD_TITLE
and not issue.get("pull_request")
and isinstance(issue.get("number"), int)
):
found[issue["number"]] = issue
return list(found.values())
def sync_dashboard_issue(gitea: GiteaClient, body: str, dry_run: bool) -> dict:
"""Create or update the one dashboard issue. Returns counts of what it did or would do."""
if DASHBOARD_LABEL in FORBIDDEN_LABELS:
raise BotError("config: the dashboard label is one the PM agent acts on")
plan = {"label_created": 0, "issue_created": 0, "issue_updated": 0, "issue_reopened": 0}
existing = find_dashboard_issues(gitea)
if len(existing) > 1:
# Closing an issue on a guess is worse than a red run naming the numbers to look at.
numbers = sorted(issue["number"] for issue in existing)
raise BotError(f"dashboard: {len(numbers)} issues titled {DASHBOARD_TITLE!r}: {numbers}")
label = next((x for x in gitea.labels() if isinstance(x, dict) and x.get("name") == DASHBOARD_LABEL), None)
if label is None:
plan["label_created"] = 1
if not dry_run:
label = gitea.create_label(DASHBOARD_LABEL, DASHBOARD_LABEL_COLOR)
if not existing:
plan["issue_created"] = 1
if not dry_run:
created = gitea.create_issue(DASHBOARD_TITLE, body, label["id"])
if not isinstance(created, dict) or not isinstance(created.get("number"), int):
raise BotError("gitea: issue creation returned an unexpected shape")
return plan
issue = existing[0]
fields: dict = {}
if issue.get("body") != body:
fields["body"] = body
if issue.get("state") != "open":
fields["state"] = "open"
plan["issue_reopened"] = 1
if fields:
plan["issue_updated"] = 1
if not dry_run:
if DASHBOARD_LABEL not in _label_names(issue):
gitea.add_issue_label(issue["number"], label["id"])
if fields:
gitea.update_issue(issue["number"], fields)
return plan
def _renovate_issue(gitea: GiteaClient, number: int) -> dict:
issue = gitea.get_issue(number)
body = issue.get("body")
if issue.get("title") != RENOVATE_DASHBOARD_TITLE or issue.get("state") != "open" or not isinstance(body, str):
raise BotError(f"renovate dashboard: issue {number} is not the open Dependency Dashboard")
return issue
def report_images(report: dict) -> list[str]:
return [t["image"] for t in report["targets"] if t["type"] == "image"]
def run(cfg: dict, gitea: GiteaClient, scanner: ScannerClient, now: datetime, emit: Callable[[str], None]) -> dict:
"""One triage pass. Any failure raises, and the heartbeat is the last thing it writes."""
dry_run = cfg["dry_run"]
report = scanner.report()
agg = aggregate(report)
now_text = now.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
summary: dict = {"dry_run": dry_run, "targets": agg["targets_total"], "critical_with_fix": agg["critical_with_fix"]}
summary["dashboard"] = sync_dashboard_issue(gitea, render_dashboard(agg, cfg["page_url"], now), dry_run)
number = cfg["renovate_issue"]
body = _renovate_issue(gitea, number)["body"]
remediation = derive_remediation(
report_images(report), parse_rows(body), gitea.open_renovate_prs(), parse_detected(body),
gitea.pr_url, now_text,
)
summary["remediation"] = count_states(remediation)
# Re-read immediately before writing: Renovate rewrites this body, and a tick must apply to
# the body as it is now.
fresh = _renovate_issue(gitea, number)["body"]
ticks = select_ticks(parse_rows(fresh), qualifying_images(report))
summary["ticks"] = len(ticks)
if ticks and not dry_run:
gitea.update_issue(number, {"body": apply_ticks(fresh, ticks)})
if not dry_run:
scanner.triage(now_text, remediation)
summary["heartbeat"] = 0 if dry_run else 1
summary["fallback_used"] = scanner.used_fallback
emit(render_summary(summary))
return summary
def render_summary(summary: dict) -> str:
verb = "would be" if summary["dry_run"] else "were"
dash = summary["dashboard"]
states = ", ".join(f"{name} {count}" for name, count in summary["remediation"].items())
return "\n".join([
f"### CVE triage ({'DRY RUN, no writes' if summary['dry_run'] else 'live'})",
"",
f"- targets in report: {summary['targets']}",
f"- unaccepted CRITICAL with a fix: {summary['critical_with_fix']}",
f"- dashboard label created: {dash['label_created']}; issue created: {dash['issue_created']};"
f" updated: {dash['issue_updated']}; reopened: {dash['issue_reopened']} ({verb} applied)",
f"- remediation states: {states}",
f"- approval boxes that {verb} ticked: {summary['ticks']}",
f"- heartbeat posted: {summary['heartbeat']}",
f"- scanner reached through the ClusterIP fallback: {summary['fallback_used']}",
"",
])
def load_config(env: dict) -> dict:
token = env.get("RENOVATE_TOKEN", "")
if not token:
raise BotError("RENOVATE_TOKEN is not set")
repo = env.get("GITEA_REPOSITORY", "Home-Network/home-network-k3s-impl")
if not REPO_RE.fullmatch(repo):
raise BotError("config: GITEA_REPOSITORY is malformed")
urls = {
"gitea_url": env.get("GITEA_URL", "https://gitea.home.keithjackson.dev"),
"vuln_url": env.get("VULN_API_URL", "http://vuln-scanner.vuln-scanner.svc.cluster.local:8081"),
"page_url": env.get("VULN_PAGE_URL", "https://cve.home.keithjackson.dev"),
}
fallback = env.get("VULN_API_FALLBACK_URL", "")
for name, value in {**urls, **({"fallback": fallback} if fallback else {})}.items():
if not HTTP_URL_RE.fullmatch(value):
raise BotError(f"config: {name} is not a plain http(s) URL")
dry_run = env.get("DRY_RUN", "false")
if dry_run not in ("true", "false"):
raise BotError("config: DRY_RUN must be true or false")
issue = env.get("RENOVATE_DASHBOARD_ISSUE", "91")
if not issue.isdigit():
raise BotError("config: RENOVATE_DASHBOARD_ISSUE is not a number")
return {**urls, "token": token, "repo": repo, "fallback": fallback,
"dry_run": dry_run == "true", "renovate_issue": int(issue)}
def main() -> int:
def emit(text: str) -> None:
print(text)
path = os.environ.get("GITHUB_STEP_SUMMARY")
if path:
with open(path, "a", encoding="utf-8") as handle:
handle.write(text + "\n")
try:
cfg = load_config(dict(os.environ))
transport = UrllibTransport()
gitea = GiteaClient(cfg["gitea_url"], cfg["repo"], cfg["token"], transport)
scanner = ScannerClient(cfg["vuln_url"], cfg["fallback"], transport)
run(cfg, gitea, scanner, datetime.now(timezone.utc), emit)
except BotError as exc:
print(f"::error::CVE triage failed: {exc}", file=sys.stderr)
emit(f"### CVE triage FAILED\n\n{exc}\n")
return 1
return 0
if __name__ == "__main__":
sys.exit(main())
+1 -1
View File
@@ -76,7 +76,7 @@ bug but the honest answer, because proofs are claims about a specific tree on a
|---|---| |---|---|
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI | | 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
| 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; **now 3× mini-PC bare metal** (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13) | | 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; **now 3× mini-PC bare metal** (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13) |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the authoritative LAN resolver** for `${BASE_DOMAIN}` on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files | | 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the LAN resolver**, answering an explicit list of `${BASE_DOMAIN}` names (LAN-only apps, gitea/nextcloud/keycloak/auth, the L4 VIPs — everything else resolves publicly to Cloudflare, since 2026-11-09) on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch | | 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled | | 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled |
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** | | 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |
+5
View File
@@ -100,6 +100,11 @@ metadata:
# external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP. # external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP.
# This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443. # This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443.
external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}" external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
# DNS-ONLY, NEVER PROXIED (2026-11-08). external-dns runs with --cloudflare-proxied globally (the
# gateway names point at the Cloudflare Tunnel); Cloudflare's proxy carries HTTP(S) only, so an
# orange-clouded A for these L4 ports would be broken. Required by the L0 check
# internal-services-publish-no-public-dns (9b) on every `service`-source publisher.
external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
spec: spec:
type: LoadBalancer type: LoadBalancer
externalTrafficPolicy: Cluster externalTrafficPolicy: Cluster
+2 -2
View File
@@ -176,8 +176,8 @@ or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:
**UI access:** `https://unifi.home.keithjackson.dev:11443`. That name resolves to the VIP and not to **UI access:** `https://unifi.home.keithjackson.dev:11443`. That name resolves to the VIP and not to
the Gateway on purpose: both `platform/coredns/coredns-custom.yaml` (in-cluster) and the Gateway on purpose: both `platform/coredns/coredns-custom.yaml` (in-cluster) and
`apps/wave-0/pihole.yaml` (LAN split-horizon) carve `unifi.${BASE_DOMAIN}` out of the `apps/wave-0/pihole.yaml` (LAN split-horizon, an explicit name list since 2026-11-09) point
zone-wide `${GATEWAY_VIP}` answer and point it at `${UNIFI_L4_VIP}`. `${BASE_DOMAIN}` on prod is the `unifi.${BASE_DOMAIN}` at `${UNIFI_L4_VIP}` rather than `${GATEWAY_VIP}`. `${BASE_DOMAIN}` on prod is the
apex `home.keithjackson.dev` (confirmed live in the `cluster-vars` ConfigMap) after the 2026-09-13 apex `home.keithjackson.dev` (confirmed live in the `cluster-vars` ConfigMap) after the 2026-09-13
flip, so this hostname is current, not a dev leftover. flip, so this hostname is current, not a dev leftover.
+2 -2
View File
@@ -78,14 +78,14 @@ spec:
spec: spec:
containers: containers:
- name: jackett - name: jackett
image: lscr.io/linuxserver/jackett:v0.24.3025-ls29 image: lscr.io/linuxserver/jackett:v0.24.3030-ls30
ports: ports:
- containerPort: 9117 - containerPort: 9117
volumeMounts: volumeMounts:
- name: config - name: config
mountPath: /config mountPath: /config
- name: oauth2-proxy - name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
imagePullPolicy: IfNotPresent imagePullPolicy: IfNotPresent
args: args:
- --http-address=0.0.0.0:4180 - --http-address=0.0.0.0:4180
+19 -5
View File
@@ -83,15 +83,25 @@
# and would also fix the same latent defect in unifi and crafty; it belongs in its own change. # and would also fix the same latent defect in unifi and crafty; it belongs in its own change.
# #
# ------------------------------------------------------------------------------------------------- # -------------------------------------------------------------------------------------------------
# loadBalancerSourceRanges — THE LAN RESTRICTION, ENFORCED IN THE DATAPATH. # loadBalancerSourceRanges — THE LAN + VPN RESTRICTION, ENFORCED IN THE DATAPATH.
# ------------------------------------------------------------------------------------------------- # -------------------------------------------------------------------------------------------------
# Cilium enforces this list in eBPF on the ingress node, on the VIP, BEFORE load-balancing and # Cilium enforces this list in eBPF on the ingress node, on the VIP, BEFORE load-balancing and
# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} is dropped by the # before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} and ${WG_CLIENT_CIDR} is
# datapath — it never reaches a backend and never reaches a network policy. This is what makes # dropped by the datapath — it never reaches a backend and never reaches a network policy. This is what makes
# "LAN clients only" a real, tested restriction rather than a comment: VERIFIED by narrowing the # "LAN clients only" a real, tested restriction rather than a comment: VERIFIED by narrowing the
# range to a /24 that excludes the test client, after which the same query that had been answering # range to a /24 that excludes the test client, after which the same query that had been answering
# stopped being answered at all, and restoring it, after which it answered again. # stopped being answered at all, and restoring it, after which it answered again.
# #
# ${WG_CLIENT_CIDR} IS LISTED BECAUSE VPN CLIENTS REACH THIS CHECK UNTRANSLATED. wg0 is one of
# Cilium's kube-proxy-replacement devices (measured 2026-11-08: `cilium-dbg status` lists eno1, wg0,
# tun0), so a tunnel client's query is load-balanced as it enters from wg0, ahead of the iptables
# MASQUERADE in platform/vpn-ingress/daemonset.yaml, and the source tested here is still 10.90.0.x
# rather than the node's address. Without this entry a full-tunnel client's DNS is predicted to die
# with DROP_NOT_IN_SRC_RANGE (derived from Cilium's source; NOT observed live, because no peer has
# ever completed a handshake). The range is private and reachable only by a key in peers.conf, so
# PiHole stays off the internet; conformance/tests/l0-static/pihole-dns-source-ranges fails the gate
# if either entry goes missing or any range here stops being RFC1918.
#
# It is also the layer that survives the SNAT above. A CiliumNetworkPolicy cannot express this # It is also the layer that survives the SNAT above. A CiliumNetworkPolicy cannot express this
# restriction once the source has been rewritten: measured on this cluster, the forwarded query # restriction once the source has been rewritten: measured on this cluster, the forwarded query
# arrives from the ingress node's internal router address and Cilium resolves that source to the # arrives from the ingress node's internal router address and Cilium resolves that source to the
@@ -113,6 +123,7 @@ spec:
externalTrafficPolicy: Cluster externalTrafficPolicy: Cluster
loadBalancerSourceRanges: loadBalancerSourceRanges:
- ${LAN_CIDR} - ${LAN_CIDR}
- ${WG_CLIENT_CIDR}
selector: selector:
app.kubernetes.io/name: pihole app.kubernetes.io/name: pihole
ports: ports:
@@ -133,6 +144,7 @@ spec:
externalTrafficPolicy: Cluster externalTrafficPolicy: Cluster
loadBalancerSourceRanges: loadBalancerSourceRanges:
- ${LAN_CIDR} - ${LAN_CIDR}
- ${WG_CLIENT_CIDR}
selector: selector:
app.kubernetes.io/name: pihole app.kubernetes.io/name: pihole
ports: ports:
@@ -158,7 +170,9 @@ spec:
# "Policy denied ... identity world" drops in `cilium-dbg monitor`). So `world` is the only peer # "Policy denied ... identity world" drops in `cilium-dbg monitor`). So `world` is the only peer
# selector that can describe LAN traffic that has been through the LoadBalancer, and the set of # selector that can describe LAN traffic that has been through the LoadBalancer, and the set of
# addresses that can actually BE that traffic is fixed by loadBalancerSourceRanges before this # addresses that can actually BE that traffic is fixed by loadBalancerSourceRanges before this
# policy is consulted. Together: ${LAN_CIDR} in the datapath, udp/53 + tcp/53 here. # policy is consulted. Together: ${LAN_CIDR} + ${WG_CLIENT_CIDR} in the datapath, udp/53 + tcp/53
# here. A VPN client delivered to a backend on its own ingress node keeps its 10.90.0.x source,
# which Cilium also classifies as `world`, so this rule needs no new peer for it.
# #
# This is NOT wider than the precedent it follows. unifi-allow-l4-ingress and crafty-allow-ingress # This is NOT wider than the precedent it follows. unifi-allow-l4-ingress and crafty-allow-ingress
# admit `fromEntities: [world, cluster, host]` on their L4 ports with NO source-range filter at all; # admit `fromEntities: [world, cluster, host]` on their L4 ports with NO source-range filter at all;
@@ -187,7 +201,7 @@ spec:
app.kubernetes.io/name: pihole app.kubernetes.io/name: pihole
ingress: ingress:
- fromEntities: - fromEntities:
- world # LAN clients, post-LoadBalancer: see the block above for why not fromCIDR - world # LAN + VPN clients, post-LoadBalancer: see the block above for why not fromCIDR
- host # this node's kubelet probes - host # this node's kubelet probes
- remote-node # the node-to-node hop that forwards a query to wherever the pod runs - remote-node # the node-to-node hop that forwards a query to wherever the pod runs
toPorts: toPorts:
+2 -2
View File
@@ -73,7 +73,7 @@ spec:
- name: FTLCONF_dns_rateLimit_count - name: FTLCONF_dns_rateLimit_count
value: '0' value: '0'
- name: FTLCONF_misc_dnsmasq_lines - name: FTLCONF_misc_dnsmasq_lines
value: local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP} value: local-ttl=60;address=/gitea.${BASE_DOMAIN}/${GATEWAY_VIP};address=/nextcloud.${BASE_DOMAIN}/${GATEWAY_VIP};address=/keycloak.${BASE_DOMAIN}/${GATEWAY_VIP};address=/auth.${BASE_DOMAIN}/${GATEWAY_VIP};address=/headlamp.${BASE_DOMAIN}/${GATEWAY_VIP};address=/cve.${BASE_DOMAIN}/${GATEWAY_VIP};address=/jackett.${BASE_DOMAIN}/${GATEWAY_VIP};address=/pihole.${BASE_DOMAIN}/${GATEWAY_VIP};address=/plex.${BASE_DOMAIN}/${GATEWAY_VIP};address=/qbittorrent.${BASE_DOMAIN}/${GATEWAY_VIP};address=/radarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/semaphore.${BASE_DOMAIN}/${GATEWAY_VIP};address=/sonarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
- name: FTLCONF_webserver_api_password - name: FTLCONF_webserver_api_password
valueFrom: valueFrom:
secretKeyRef: secretKeyRef:
@@ -90,7 +90,7 @@ spec:
- -c - -c
- "# Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and\n# does NOT run a gravity rebuild.\nset -u\nGRAVITY=/etc/pihole/gravity.db\n\n# THE DECLARED SET \u2014 one URL per line, the source of truth for what PiHole blocks.\n# To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.\nDECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'\n\n# On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,\n# and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.\ni=0\nwhile [ ! -s \"$GRAVITY\" ] && [ \"$i\" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done\nif [ ! -s \"$GRAVITY\" ]; then\n echo \"pihole-adlists: gravity.db absent after 180s - leaving adlists untouched\" >&2\n exit 0\nfi\n\nsql() { pihole-FTL sqlite3 \"$GRAVITY\" \"$1\"; }\nCHANGED=0\n\n# 1) Every declared URL must be present AND enabled.\necho \"$DECLARED\" | while IFS= read -r url; do\n [ -n \"$url\" ] || continue\n printf '%s\\n' \"$url\"\ndone > /tmp/declared.txt\nwhile IFS= read -r url; do\n [ -n \"$url\" ] || continue\n # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.\n have=$(sql \"select count(*) from adlist where address='$url';\")\n if [ \"$have\" = \"0\" ]; then\n echo \"pihole-adlists: adding $url\"\n sql \"insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');\"\n CHANGED=1\n else\n en=$(sql \"select enabled from adlist where address='$url';\")\n if [ \"$en\" != \"1\" ]; then\n echo \"pihole-adlists: re-enabling $url\"\n sql \"update adlist set enabled=1 where address='$url';\"\n CHANGED=1\n fi\n fi\ndone < /tmp/declared.txt\n\n# 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).\nsql \"select address from adlist where enabled=1;\" | while IFS= read -r addr; do\n [ -n \"$addr\" ] || continue\n if ! grep -qxF \"$addr\" /tmp/declared.txt; then\n echo \"pihole-adlists: disabling undeclared $addr\"\n sql \"update adlist set enabled=0 where address='$addr';\"\n echo changed >> /tmp/adlist-changed\n fi\ndone\n[ -s /tmp/adlist-changed ] && CHANGED=1\nrm -f /tmp/adlist-changed /tmp/declared.txt\n\n# 3) Rebuild gravity ONLY if the declared set actually moved.\nif [ \"$CHANGED\" = \"1\" ]; then\n echo \"pihole-adlists: declared set changed - running gravity update\"\n pihole -g || echo \"pihole-adlists: gravity update failed; existing gravity.db retained\" >&2\nelse\n echo \"pihole-adlists: already matches the declared set - nothing to do\"\nfi\nexit 0\n" - "# Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and\n# does NOT run a gravity rebuild.\nset -u\nGRAVITY=/etc/pihole/gravity.db\n\n# THE DECLARED SET \u2014 one URL per line, the source of truth for what PiHole blocks.\n# To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.\nDECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'\n\n# On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,\n# and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.\ni=0\nwhile [ ! -s \"$GRAVITY\" ] && [ \"$i\" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done\nif [ ! -s \"$GRAVITY\" ]; then\n echo \"pihole-adlists: gravity.db absent after 180s - leaving adlists untouched\" >&2\n exit 0\nfi\n\nsql() { pihole-FTL sqlite3 \"$GRAVITY\" \"$1\"; }\nCHANGED=0\n\n# 1) Every declared URL must be present AND enabled.\necho \"$DECLARED\" | while IFS= read -r url; do\n [ -n \"$url\" ] || continue\n printf '%s\\n' \"$url\"\ndone > /tmp/declared.txt\nwhile IFS= read -r url; do\n [ -n \"$url\" ] || continue\n # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.\n have=$(sql \"select count(*) from adlist where address='$url';\")\n if [ \"$have\" = \"0\" ]; then\n echo \"pihole-adlists: adding $url\"\n sql \"insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');\"\n CHANGED=1\n else\n en=$(sql \"select enabled from adlist where address='$url';\")\n if [ \"$en\" != \"1\" ]; then\n echo \"pihole-adlists: re-enabling $url\"\n sql \"update adlist set enabled=1 where address='$url';\"\n CHANGED=1\n fi\n fi\ndone < /tmp/declared.txt\n\n# 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).\nsql \"select address from adlist where enabled=1;\" | while IFS= read -r addr; do\n [ -n \"$addr\" ] || continue\n if ! grep -qxF \"$addr\" /tmp/declared.txt; then\n echo \"pihole-adlists: disabling undeclared $addr\"\n sql \"update adlist set enabled=0 where address='$addr';\"\n echo changed >> /tmp/adlist-changed\n fi\ndone\n[ -s /tmp/adlist-changed ] && CHANGED=1\nrm -f /tmp/adlist-changed /tmp/declared.txt\n\n# 3) Rebuild gravity ONLY if the declared set actually moved.\nif [ \"$CHANGED\" = \"1\" ]; then\n echo \"pihole-adlists: declared set changed - running gravity update\"\n pihole -g || echo \"pihole-adlists: gravity update failed; existing gravity.db retained\" >&2\nelse\n echo \"pihole-adlists: already matches the declared set - nothing to do\"\nfi\nexit 0\n"
- name: oauth2-proxy - name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
imagePullPolicy: IfNotPresent imagePullPolicy: IfNotPresent
args: args:
- --http-address=0.0.0.0:4180 - --http-address=0.0.0.0:4180
+2
View File
@@ -302,6 +302,8 @@ metadata:
namespace: plex namespace: plex
labels: labels:
app.kubernetes.io/name: plex app.kubernetes.io/name: plex
annotations:
external-dns.alpha.kubernetes.io/controller: none
spec: spec:
parentRefs: parentRefs:
- name: shared - name: shared
+1 -1
View File
@@ -156,7 +156,7 @@ spec:
, \"[Preferences]\"\n\ndef matches(raw):\n \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n v = raw.strip().strip('\"')\n if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n v = v[len(\"@ByteArray(\"):-1]\n try:\n salt_b64, digest_b64 = v.split(\":\", 1)\n salt = base64.b64decode(salt_b64, validate=True)\n digest = base64.b64decode(digest_b64, validate=True)\n except Exception:\n return False\n if not salt or not digest:\n return False\n return hmac.compare_digest(\n hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n lines = []\n\ndef value_of(key):\n for i, line in enumerate(lines):\n if line.startswith(key + \"=\"):\n return i, line.split(\"=\", 1)[1]\n return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n raise SystemExit(0) # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n at = out.index(SECTION) + 1\nelse:\n if out and out[-1].strip():\n out.append(\"\")\n out.append(SECTION)\n at = len(out)\nout[at:at] = desired\n\ntry:\n os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n tmp = conf + \".webui-cred.new\"\n with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n fh.write(\"\\n\".join(out) + \"\\n\")\n fh.flush()\n os.fsync(fh.fileno())\n # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n # cost the app the ability to save its own settings, so carry the ownership over.\n try:\n st = os.stat(conf)\n os.chown(tmp, st.st_uid, st.st_gid)\n os.chmod(tmp, stat.S_IMODE(st.st_mode))\n except FileNotFoundError:\n os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n int(os.environ.get(\"PGID\", \"1000\")))\n os.chmod(tmp, 0o664)\n os.replace(tmp, conf) # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n raise SystemExit(1)\nraise SystemExit(10)\nPY\n rc=$?\n\n if [ \"$rc\" = 0 ]; then\n log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n elif [ \"$rc\" = 10 ]; then\n log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n # have read the old, credential-less conf and be serving a random temporary password.\n # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n # Nothing to do when the service has not started yet - it will read the new file.\n # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the\n # 2026-09-13 failure spent the entire verification budget before the first login and\n # then fired attempts 2 and 3 with no wait left at all.\n sv=0\n while [ \"$sv\" -lt 60 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n sv=$((sv + 1)); sleep 1\n done\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"qbittorrent-nox was already running - restarting it to load the credential\"\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n sleep 3\n fi\n else\n log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n log \"minting a random password\ , \"[Preferences]\"\n\ndef matches(raw):\n \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n v = raw.strip().strip('\"')\n if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n v = v[len(\"@ByteArray(\"):-1]\n try:\n salt_b64, digest_b64 = v.split(\":\", 1)\n salt = base64.b64decode(salt_b64, validate=True)\n digest = base64.b64decode(digest_b64, validate=True)\n except Exception:\n return False\n if not salt or not digest:\n return False\n return hmac.compare_digest(\n hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n lines = []\n\ndef value_of(key):\n for i, line in enumerate(lines):\n if line.startswith(key + \"=\"):\n return i, line.split(\"=\", 1)[1]\n return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n raise SystemExit(0) # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n at = out.index(SECTION) + 1\nelse:\n if out and out[-1].strip():\n out.append(\"\")\n out.append(SECTION)\n at = len(out)\nout[at:at] = desired\n\ntry:\n os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n tmp = conf + \".webui-cred.new\"\n with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n fh.write(\"\\n\".join(out) + \"\\n\")\n fh.flush()\n os.fsync(fh.fileno())\n # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n # cost the app the ability to save its own settings, so carry the ownership over.\n try:\n st = os.stat(conf)\n os.chown(tmp, st.st_uid, st.st_gid)\n os.chmod(tmp, stat.S_IMODE(st.st_mode))\n except FileNotFoundError:\n os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n int(os.environ.get(\"PGID\", \"1000\")))\n os.chmod(tmp, 0o664)\n os.replace(tmp, conf) # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n raise SystemExit(1)\nraise SystemExit(10)\nPY\n rc=$?\n\n if [ \"$rc\" = 0 ]; then\n log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n elif [ \"$rc\" = 10 ]; then\n log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n # have read the old, credential-less conf and be serving a random temporary password.\n # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n # Nothing to do when the service has not started yet - it will read the new file.\n # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the\n # 2026-09-13 failure spent the entire verification budget before the first login and\n # then fired attempts 2 and 3 with no wait left at all.\n sv=0\n while [ \"$sv\" -lt 60 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n sv=$((sv + 1)); sleep 1\n done\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"qbittorrent-nox was already running - restarting it to load the credential\"\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n sleep 3\n fi\n else\n log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n log \"minting a random password\
\ every start and BOTH arrs will break. Fix and restart.\" >&2\n exit 0\n fi\n\n # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL\n # (shared ~120s budget across attempts), then spend exactly ONE login per attempt -\n # at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n #\n # \"THE API NEVER ANSWERED\" IS A DIFFERENT FAULT FROM \"THE PASSWORD WAS REJECTED\", and\n # conflating them is what made this hook lie once already (see the PORT comment). An\n # unreachable WebUI is reported as unreachable, and the hook stops rather than spending\n # its remaining login attempts to produce two more identical wrong diagnoses.\n api_ok=0\n while [ \"$waited\" -lt 120 ]; do\n code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n if [ -n \"$code\" ] && [ \"$code\" != \"000\" ]; then api_ok=1; break; fi\n waited=$((waited + 1)); sleep 1\n done\n if [ \"$api_ok\" != 1 ]; then\n log \"the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the\" >&2\n log \"credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was\" >&2\n log \"never submitted. Do not rotate the credential on the strength of this line.\" >&2\n exit 0\n fi\n body=$(curl -sS -m 10 \\\n --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n if [ \"$body\" = \"Ok.\" ]; then\n log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n exit 0\n fi\n log \"WebUI rejected the durable credential (login said '$body')\" >&2\n # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean\n # the running qbittorrent-nox is serving something it read from an EARLIER conf, so\n # rewriting the same bytes on the next attempt would change nothing. The remedy is to\n # make the process RE-READ the file, so restart the supervised service and let the next\n # attempt re-verify. Without this the retry loop could never change its own outcome.\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"restarting qbittorrent-nox so it re-reads $CONF, then re-verifying\" >&2\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\" >&2\n sleep 3\n fi\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n" \ every start and BOTH arrs will break. Fix and restart.\" >&2\n exit 0\n fi\n\n # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL\n # (shared ~120s budget across attempts), then spend exactly ONE login per attempt -\n # at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n #\n # \"THE API NEVER ANSWERED\" IS A DIFFERENT FAULT FROM \"THE PASSWORD WAS REJECTED\", and\n # conflating them is what made this hook lie once already (see the PORT comment). An\n # unreachable WebUI is reported as unreachable, and the hook stops rather than spending\n # its remaining login attempts to produce two more identical wrong diagnoses.\n api_ok=0\n while [ \"$waited\" -lt 120 ]; do\n code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n if [ -n \"$code\" ] && [ \"$code\" != \"000\" ]; then api_ok=1; break; fi\n waited=$((waited + 1)); sleep 1\n done\n if [ \"$api_ok\" != 1 ]; then\n log \"the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the\" >&2\n log \"credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was\" >&2\n log \"never submitted. Do not rotate the credential on the strength of this line.\" >&2\n exit 0\n fi\n body=$(curl -sS -m 10 \\\n --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n if [ \"$body\" = \"Ok.\" ]; then\n log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n exit 0\n fi\n log \"WebUI rejected the durable credential (login said '$body')\" >&2\n # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean\n # the running qbittorrent-nox is serving something it read from an EARLIER conf, so\n # rewriting the same bytes on the next attempt would change nothing. The remedy is to\n # make the process RE-READ the file, so restart the supervised service and let the next\n # attempt re-verify. Without this the retry loop could never change its own outcome.\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"restarting qbittorrent-nox so it re-reads $CONF, then re-verifying\" >&2\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\" >&2\n sleep 3\n fi\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n"
- name: oauth2-proxy - name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
imagePullPolicy: IfNotPresent imagePullPolicy: IfNotPresent
args: args:
- --http-address=0.0.0.0:4180 - --http-address=0.0.0.0:4180
+69 -5
View File
@@ -77,6 +77,42 @@ spec:
requests: requests:
storage: 1Gi storage: 1Gi
--- ---
# act_runner config file — DELTAS ONLY. act_runner v0.6.1 LoadDefault()
# (internal/pkg/config/config.go) unmarshals this file onto a ZERO Config and then fills every
# unset field with the same default it uses when there is no file at all, so a file holding
# only `runner.capacity` changes exactly that and nothing else. Do NOT paste in the output of
# `act_runner generate-config`: that example is NOT the defaults. Its `runner.labels` would
# OVERRIDE the registered labels in /data/.runner (daemon.go: config labels win when non-empty)
# and repoint ubuntu-latest at a different job image; its `force_pull: true` differs from the
# no-file default (false); and its sample `envs` would be injected into every job.
#
# Using a file also makes act_runner IGNORE the legacy GITEA_RUNNER_CAPACITY / _FILE /
# _ENVIRON / _ENV_FILE / GITEA_DEBUG / GITEA_TRACE env vars (deprecated.go). None of them is
# set on this Deployment; the GITEA_INSTANCE_URL / _REGISTRATION_TOKEN / _NAME / _LABELS env
# below is consumed by the image's run.sh for registration, not by that legacy path.
#
# A ConfigMap edit alone does NOT restart the runner (it reads the file once at start). If a
# future change touches ONLY this ConfigMap, also bump the Deployment's pod template (e.g. the
# config-revision annotation below) or the new value stays inert until the next restart.
apiVersion: v1
kind: ConfigMap
metadata:
name: act-runner-config
namespace: gitea-actions
labels:
app.kubernetes.io/name: act-runner
data:
config.yaml: |
runner:
# Two concurrent jobs (default 1). Measured 2026-11-08: queue wait ~1s normally, but
# 29-759s whenever gate / images / renovate runs collide (run 472 waited 759s behind a
# renovate burst). Both jobs share the ONE dind daemon below; act_runner names each job's
# containers GITEA-ACTIONS-TASK-<task id> and creates a network per job, so they do not
# collide. Workflow `concurrency:` groups are enforced by Gitea server-side and are
# unaffected: gate.yml's gate-<ref> group still cancels a superseded run on the SAME ref,
# and renovate's group still serialises renovate. The dind resources are sized for TWO jobs.
capacity: 2
---
apiVersion: apps/v1 apiVersion: apps/v1
kind: Deployment kind: Deployment
metadata: metadata:
@@ -95,6 +131,10 @@ spec:
metadata: metadata:
labels: labels:
app.kubernetes.io/name: act-runner app.kubernetes.io/name: act-runner
annotations:
# Bump when act-runner-config changes without any other pod-template change, so the
# Deployment rolls and act_runner re-reads its config (see the ConfigMap header).
homelab.keithjackson.dev/act-runner-config-revision: "1" # capacity 2
spec: spec:
# CI holds no cluster credentials — do not mount a SA token into the runner or # CI holds no cluster credentials — do not mount a SA token into the runner or
# (via the shared netns) the job containers (R-SEC-02, docs/14). # (via the shared netns) the job containers (R-SEC-02, docs/14).
@@ -146,12 +186,23 @@ spec:
mountPath: /certs mountPath: /certs
- name: dind-storage - name: dind-storage
mountPath: /var/lib/docker mountPath: /var/lib/docker
# Sized for runner.capacity: 2. Every job's containers run UNDER this daemon, so this
# container's cgroup carries the whole CI workload. Measured 2026-11-08 (Prometheus,
# capacity 1): peak CPU 0.69 core (1m rate, 30d), peak working set 447Mi (30d), 0 OOMs.
# cpu request 2: two jobs at today's peak is ~1.4 core, and the gate is being
# parallelised, so per-job demand rises. The request is what the CFS share is
# weighted by under contention (100m was a 1/20 share); NO cpu limit, deliberately,
# so an idle node's spare cores are still usable. k3s-prod-2 (8 CPU) had 2945m
# requested; +1.9 core = ~4.85 (61%), and prod-1/prod-3 fit too if it reschedules.
# memory request 1Gi / limit 4Gi: two jobs at today's peak is ~0.9Gi. An OOM kill here
# takes down dockerd and with it BOTH running jobs, so the limit carries ~4x headroom
# over that for the parallelised gate. Nodes have ~31Gi allocatable.
resources: resources:
requests: requests:
cpu: 100m cpu: "2"
memory: 256Mi memory: 1Gi
limits: limits:
memory: 2Gi memory: 4Gi
startupProbe: startupProbe:
exec: exec:
command: command:
@@ -195,23 +246,36 @@ spec:
value: /certs/client value: /certs/client
- name: DOCKER_TLS_VERIFY - name: DOCKER_TLS_VERIFY
value: "1" value: "1"
# The image's run.sh passes `--config <this>` to BOTH `act_runner register` and
# `act_runner daemon` when CONFIG_FILE is set (act_runner v0.6.1 scripts/run.sh).
- name: CONFIG_FILE
value: /etc/act_runner/config.yaml
volumeMounts: volumeMounts:
- name: data - name: data
mountPath: /data mountPath: /data
- name: certs - name: certs
mountPath: /certs mountPath: /certs
readOnly: true readOnly: true
- name: config
mountPath: /etc/act_runner
readOnly: true
# Measured 2026-11-08 at capacity 1: peak CPU 0.24 core, peak working set 314Mi (30d)
# against the old 512Mi limit. It holds per-task state, log buffers and the embedded
# actions/cache server for every concurrent task, so the limit is doubled for capacity 2.
resources: resources:
requests: requests:
cpu: 100m cpu: 100m
memory: 128Mi memory: 256Mi
limits: limits:
memory: 512Mi memory: 1Gi
volumes: volumes:
- name: data - name: data
persistentVolumeClaim: persistentVolumeClaim:
claimName: act-runner-data claimName: act-runner-data
- name: certs - name: certs
emptyDir: {} emptyDir: {}
- name: config
configMap:
name: act-runner-config
- name: dind-storage - name: dind-storage
emptyDir: {} emptyDir: {}
+1 -1
View File
@@ -138,7 +138,7 @@ spec:
- name: media - name: media
mountPath: /media mountPath: /media
- name: oauth2-proxy - name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
imagePullPolicy: IfNotPresent imagePullPolicy: IfNotPresent
args: args:
- --http-address=0.0.0.0:4180 - --http-address=0.0.0.0:4180
+1 -1
View File
@@ -138,7 +138,7 @@ spec:
- name: media - name: media
mountPath: /media mountPath: /media
- name: oauth2-proxy - name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
imagePullPolicy: IfNotPresent imagePullPolicy: IfNotPresent
args: args:
- --http-address=0.0.0.0:4180 - --http-address=0.0.0.0:4180
+1 -1
View File
@@ -16,7 +16,7 @@
name: jackett name: jackett
kind: container kind: container
source: source:
image: lscr.io/linuxserver/jackett:v0.24.3025-ls29 image: lscr.io/linuxserver/jackett:v0.24.3030-ls30
exposure: internal # LAN-only web UI at jackett.${BASE_DOMAIN} (docs/13 row 4; R-DNS/R-NET) exposure: internal # LAN-only web UI at jackett.${BASE_DOMAIN} (docs/13 row 4; R-DNS/R-NET)
egress: vpn # indexer scraping via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06) egress: vpn # indexer scraping via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
storage: storage:
+58 -31
View File
@@ -85,7 +85,7 @@ env:
# Default is LOCAL, which answers only clients on the same subnet as the listening interface. The # Default is LOCAL, which answers only clients on the same subnet as the listening interface. The
# pod's interface is on the CLUSTER pod CIDR, so every LAN client would be refused. ALL is # pod's interface is on the CLUSTER pod CIDR, so every LAN client would be refused. ALL is
# required for a LoadBalancer-fronted resolver; what keeps PiHole from being an OPEN resolver is # required for a LoadBalancer-fronted resolver; what keeps PiHole from being an OPEN resolver is
# the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}]` on the DNS Services, # the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}, ${WG_CLIENT_CIDR}]` on the DNS Services,
# enforced by Cilium in the datapath before load-balancing, plus a CiliumNetworkPolicy that admits # enforced by Cilium in the datapath before load-balancing, plus a CiliumNetworkPolicy that admits
# nothing but udp/53 and tcp/53. See apps/wave-0/pihole-dns.yaml. # nothing but udp/53 and tcp/53. See apps/wave-0/pihole-dns.yaml.
- name: FTLCONF_dns_listeningMode - name: FTLCONF_dns_listeningMode
@@ -108,41 +108,61 @@ env:
# `loadBalancer.mode: dsr`, which preserves the client address. # `loadBalancer.mode: dsr`, which preserves the client address.
- name: FTLCONF_dns_rateLimit_count - name: FTLCONF_dns_rateLimit_count
value: "0" value: "0"
# THE SPLIT-HORIZON ANSWERS. A dnsmasq `address=/<domain>/<ip>` entry answers that domain AND # THE LAN ANSWERS — AN EXPLICIT LIST, NOT A ZONE WILDCARD (2026-11-09). A dnsmasq
# every subdomain of it, and dnsmasq resolves overlapping entries by LONGEST MATCHING DOMAIN — # `address=/<domain>/<ip>` entry answers that domain AND every subdomain of it (on LABEL
# so the first line is a wildcard for the whole zone and each later line is a specific override. # boundaries: `pihole.` does not match `pihole-dns.`), and dnsmasq resolves overlapping entries by
# PRECEDENCE IS THEREFORE STRUCTURAL, not order-dependent: `unifi.${BASE_DOMAIN}` matches both # LONGEST MATCHING DOMAIN, so the order below is irrelevant. Every name NOT listed here is
# line 1 and line 2, line 2 is the longer domain, line 2 wins. Rewriting these in any order # FORWARDED to the upstreams above and gets the PUBLIC answer.
# changes nothing.
# #
# A NEWLY ONBOARDED SERVICE NEEDS NO CHANGE HERE. Every catalog ingressRoute host is # WHY NO WILDCARD ANY MORE. Until 2026-11-09 the first entry was `address=/${BASE_DOMAIN}/
# `<label>.${BASE_DOMAIN}` served by the shared Gateway, so the wildcard already answers it with # ${GATEWAY_VIP}`, answering the whole zone. That made sense while every public record was a
# ${GATEWAY_VIP} — the same guarantee, and the same reasoning, as the `template IN A` block in # DNS-only CNAME to `wan.${BASE_DOMAIN}` (no NAT hairpin on this router, so the LAN NEEDED a
# platform/coredns/coredns-custom.yaml. Add a line here ONLY for a service that does not live # private answer). Since PR #156 every public name is a PROXIED CNAME to the Cloudflare Tunnel
# behind the Gateway: an L4 LoadBalancer on its own pinned VIP. Those are enumerated from the # (platform/externaldns/README.md), which works identically from the LAN — so a name that is
# repo, not from memory — grep `external-dns.alpha.kubernetes.io/hostname` on a Service, which # public now SHOULD resolve publicly from the LAN too, and the LAN no longer depends on PiHole for
# is the annotation that means "this Service publishes its own hostname at its own address": # it: a client that bypasses PiHole (DoH browser, phone private DNS, the router's fallback) gets
# apps/vms/unifi/50-service-l4.yaml (unifi -> ${UNIFI_L4_VIP}) and # the same working answer. The wildcard had also become actively misleading: it answered names
# platform/crafty/service-l4.yaml (crafty-mc -> ${CRAFTY_L4_VIP}). Those two, plus PiHole's own # that exist nowhere (`wan.` itself resolved to the Gateway VIP on the LAN).
# resolver address, are the complete set of exceptions today.
# #
# AAAA needs nothing: dnsmasq answers a AAAA query for an `address=`d name with NODATA # WHAT IS LISTED, and why each — enforced by conformance/tests/l0-static/pihole-lan-names:
# (NOERROR, no answer), so IPv6-capable clients fall back to the A record instead of getting # -> ${GATEWAY_VIP}, LAN-ONLY: every HTTPRoute disowned from public DNS
# NXDOMAIN — the same behaviour coredns-custom hand-writes a `template IN AAAA` block for. # (`external-dns.alpha.kubernetes.io/controller: none`, i.e. catalog `exposure: internal`,
# plus plex and headlamp) — jackett, pihole, plex, qbittorrent, radarr, semaphore, sonarr,
# headlamp. These have NO public record and the tunnel 404s them, so THIS LINE IS THEIR ONLY
# PATH: drop one and the app is unreachable everywhere. A NEWLY ONBOARDED `internal` SERVICE
# MUST BE ADDED HERE (the L0 check fails until it is). A new `public` service needs nothing.
# -> ${GATEWAY_VIP}, PUBLIC BUT KEPT ON THE LAN PATH: gitea (git push, LFS, container registry)
# and nextcloud (uploads) — Cloudflare's proxy caps a request body at 100 MB and a response
# at 100 s, which a large push or upload exceeds; keycloak and auth (oauth2-proxy) — so the
# OIDC login of those two LAN-path apps stays on the same path as the apps themselves.
# -> their own L4 VIPs: unifi (${UNIFI_L4_VIP}) and crafty-mc (${CRAFTY_L4_VIP}) are not behind
# the Gateway (grep `external-dns.alpha.kubernetes.io/hostname` on a Service:
# apps/vms/unifi/50-service-l4.yaml, platform/crafty/service-l4.yaml); their public records
# are DNS-only A records to the same RFC1918 VIPs, which this router's rebind protection
# strips, so the LAN answer is still required. pihole-dns (${PIHOLE_DNS_VIP}) is PiHole's own
# resolver address; plex-vip (${PLEX_L4_VIP}) is below.
# NOT listed, deliberately, so they resolve via Cloudflare from the LAN as from anywhere: the apex,
# grafana, crafty, seerr. In-cluster (platform/coredns/coredns-custom.yaml) and node
# (/etc/hosts pins, node_baseline_hosts_pins) resolution are separate and unchanged.
#
# AAAA: dnsmasq answers a AAAA query for an `address=`d name with NODATA (NOERROR, no answer),
# so IPv6-capable clients fall back to the A record instead of getting NXDOMAIN — the same
# behaviour coredns-custom hand-writes a `template IN AAAA` block for. Unlisted (proxied) names
# now get Cloudflare's real A and AAAA answers.
# #
# Array-valued FTLCONF variables are SEMICOLON-separated. A JSON-looking list is accepted and # Array-valued FTLCONF variables are SEMICOLON-separated. A JSON-looking list is accepted and
# then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal # then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal
# characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid" # characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid"
# while resolution for the zone does not work. Confirmed both ways against the live image. # while resolution for the zone does not work. Confirmed both ways against the live image.
# plex-vip -> ${PLEX_L4_VIP} (added 2026-10-04) is the LAN HALF of Plex's single advertised #
# hostname. Plex publishes ONE URL to plex.tv for both local and remote clients # plex-vip -> ${PLEX_L4_VIP} (added 2026-10-04) is the name of the tcp/32400 LoadBalancer
# (catalog/plex.yaml ADVERTISE_IP): on the LAN this answer points it at the tcp/32400 # (apps/wave-0/plex-remote.yaml) and the domain of the certificate Plex serves there
# LoadBalancer, while Cloudflare answers the same name publicly with the WAN address, NATed to # (catalog/plex.yaml CERTDOMAIN / customCertificateDomain, Secret `plex-vip-tls`). It is NOT what
# the same VIP. WITHOUT THIS LINE the entry above answers it with ${GATEWAY_VIP} via the zone # Plex advertises any more (ADVERTISE_IP is a resolver-independent plex.direct name since
# wildcard, where NOTHING listens on 32400 — measured 2026-10-04, plex-vip.${BASE_DOMAIN} # 2026-10-05) and it has NO public record, so this entry is its only resolution — kept for
# resolved to the Gateway VIP and a connect to :32400 there returned http=000. That is a silent # reaching the VIP by name with a matching certificate from a LAN shell. Measured 2026-10-04,
# failure: the Plex web UI on plex.${BASE_DOMAIN} keeps working through the Gateway on 443 while # before it existed the zone wildcard answered it with ${GATEWAY_VIP}, where nothing listens on
# every client that takes the ADVERTISED connection cannot reach the server at all. # 32400; without it now, the name is NXDOMAIN.
# #
# ── local-ttl=60 — THE SPLIT-HORIZON ANSWER MUST BE CACHEABLE ──────────────────────────────── # ── local-ttl=60 — THE SPLIT-HORIZON ANSWER MUST BE CACHEABLE ────────────────────────────────
# MEASURED 2026-10-18, diagnosing "DNS flapping between the cluster ingress and the public IP": # MEASURED 2026-10-18, diagnosing "DNS flapping between the cluster ingress and the public IP":
@@ -172,10 +192,17 @@ env:
# cache expires, the router falls back, and the public answer latches again. That is a real # cache expires, the router falls back, and the public answer latches again. That is a real
# outage, and the fix for it is PiHole availability, not DNS caching. # outage, and the fix for it is PiHole availability, not DNS caching.
# #
# SINCE 2026-11-09 THE LATCH IS MOSTLY HARMLESS. The measurement above predates the tunnel: the
# public answer was then `wan.`, unreachable from the LAN. Now the public answer for every
# proxied name is Cloudflare, which works from the LAN, and those names are not listed below at
# all. A latch still bites only the listed names: gitea/nextcloud/keycloak/auth fall back to
# Cloudflare (working, but under its 100 MB / 100 s limits); the LAN-only names and the L4 VIPs
# get NXDOMAIN or a stripped answer until the router re-queries PiHole.
#
# Applies to every `address=` reply below (dnsmasq: local-ttl governs replies made from # Applies to every `address=` reply below (dnsmasq: local-ttl governs replies made from
# configuration, /etc/hosts and DHCP leases), so one directive covers all five names. # configuration, /etc/hosts and DHCP leases), so one directive covers every listed name.
- name: FTLCONF_misc_dnsmasq_lines - name: FTLCONF_misc_dnsmasq_lines
value: "local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}" value: "local-ttl=60;address=/gitea.${BASE_DOMAIN}/${GATEWAY_VIP};address=/nextcloud.${BASE_DOMAIN}/${GATEWAY_VIP};address=/keycloak.${BASE_DOMAIN}/${GATEWAY_VIP};address=/auth.${BASE_DOMAIN}/${GATEWAY_VIP};address=/headlamp.${BASE_DOMAIN}/${GATEWAY_VIP};address=/cve.${BASE_DOMAIN}/${GATEWAY_VIP};address=/jackett.${BASE_DOMAIN}/${GATEWAY_VIP};address=/pihole.${BASE_DOMAIN}/${GATEWAY_VIP};address=/plex.${BASE_DOMAIN}/${GATEWAY_VIP};address=/qbittorrent.${BASE_DOMAIN}/${GATEWAY_VIP};address=/radarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/semaphore.${BASE_DOMAIN}/${GATEWAY_VIP};address=/sonarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
# THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from # THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
# apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which # apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
# carries the full rationale and the measurement that found it. # carries the full rationale and the measurement that found it.
+8 -1
View File
@@ -347,7 +347,14 @@ lifecycle:
# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took. # `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took.
hardware: hardware:
gpu: intel-quicksync gpu: intel-quicksync
exposure: public # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET) # EXPOSURE: internal (operator decision 2026-11-08, Cloudflare Tunnel cut-over). The plex.${BASE_DOMAIN}
# web UI is LAN/VPN-only: `internal` renders `external-dns ... controller: none` on the HTTPRoute, so
# external-dns DELETES the public plex record and its TXT, and platform/cloudflared/config.yaml maps the
# name to http_status:404 so it can never be served through the tunnel either. REMOTE STREAMING IS NOT
# AFFECTED: it never used this name. It rides plex.tv + the advertised plex.direct URL (ADVERTISE_IP
# above) to the raw tcp/32400 L4 VIP (${PLEX_L4_VIP}, apps/wave-0/plex-remote.yaml) via the WAN
# tcp/32400 forward — none of which this field touches. See runbooks/plex-remote-access.md.
exposure: internal # web UI LAN/VPN-only; remote streaming = plex.tv + WAN tcp/32400 (docs/09; R-DNS/R-NET)
egress: internet # normal WAN egress (docs/03; R-NET-05) egress: internet # normal WAN egress (docs/03; R-NET-05)
storage: storage:
- name: config - name: config
+1 -1
View File
@@ -129,7 +129,7 @@ AUTH_MODES = ("oidc", "forward-auth", "native", "none")
# the sidecar authenticates the browser against Keycloak (homelab realm, the app's auth.groups) # the sidecar authenticates the browser against Keycloak (homelab realm, the app's auth.groups)
# and only THEN proxies to the co-located app UI at http://127.0.0.1:<uiport>. The gate is IN # and only THEN proxies to the co-located app UI at http://127.0.0.1:<uiport>. The gate is IN
# the data path (fail-closed: no valid session in an authorized group => no upstream; R-IDP-02). # the data path (fail-closed: no valid session in an authorized group => no upstream; R-IDP-02).
OAUTH2_PROXY_IMAGE = "quay.io/oauth2-proxy/oauth2-proxy:v7.8.1" OAUTH2_PROXY_IMAGE = "quay.io/oauth2-proxy/oauth2-proxy:v7.15.6"
OAUTH2_PROXY_PORT = 4180 OAUTH2_PROXY_PORT = 4180
KEYCLOAK_ISSUER = f"https://keycloak.{ZONE}/realms/homelab" KEYCLOAK_ISSUER = f"https://keycloak.{ZONE}/realms/homelab"
+1 -1
View File
@@ -3,4 +3,4 @@
# System Architect; each legitimate spec advance is adopted here by bumping this pin # System Architect; each legitimate spec advance is adopted here by bumping this pin
# in a reviewed change. CI (ci/spec-immutability.sh) fails if the spec repo is dirty # in a reviewed change. CI (ci/spec-immutability.sh) fails if the spec repo is dirty
# or its HEAD != this pin — catching any unauthorized/unadopted spec modification. # or its HEAD != this pin — catching any unauthorized/unadopted spec modification.
b633b7fb089ecd9c51a0056a3082f8dd638cd8cf 2760d69bab038f5dde66bcad2632bc7cc5806a2a
+38 -2
View File
@@ -25,8 +25,44 @@ if [ "${#files[@]}" -eq 0 ]; then
exit 0 exit 0
fi fi
yamllint -c "${CI_DIR}/yamllint.yaml" -f "${YAMLLINT_FORMAT:-standard}" "${files[@]}" # yamllint is single-threaded, so the file list is split into LINT_JOBS CONTIGUOUS slices of the
ok "yaml/lint passed (${#files[@]} files)" # sorted list, one yamllint per slice, run concurrently. Each slice's output goes to its own file
# and the files are printed in slice order afterwards: the log is the same text, in the same
# file order, as one serial `yamllint "${files[@]}"`, and a finding is never interleaved with
# another process's output (yamllint's standard format puts the FILE NAME on its own line above
# that file's `line:col` findings, so interleaving would misattribute them). The stage's exit
# code is the highest any slice returned (yamllint: 1 = error, 2 = warning under strict), so a
# single failing slice fails the stage exactly as the serial call did. LINT_JOBS=1 is serial.
lint_jobs="${LINT_JOBS:-$(n="$(nproc 2>/dev/null || echo 1)"; [ "$n" -gt 8 ] && n=8; echo "$n")}"
case "$lint_jobs" in ''|*[!0-9]*|0) err "LINT_JOBS must be a positive integer, got '${lint_jobs}'"; exit 2 ;; esac
[ "$lint_jobs" -gt "${#files[@]}" ] && lint_jobs="${#files[@]}"
lint_tmp="$(mktemp -d)"
trap 'rm -rf "$lint_tmp"' EXIT
per=$(( (${#files[@]} + lint_jobs - 1) / lint_jobs ))
pids=()
for (( i = 0; i < lint_jobs; i++ )); do
slice=("${files[@]:$(( i * per )):${per}}")
[ "${#slice[@]}" -gt 0 ] || break
( rc=0
yamllint -c "${CI_DIR}/yamllint.yaml" -f "${YAMLLINT_FORMAT:-standard}" "${slice[@]}" \
>"${lint_tmp}/${i}.out" 2>&1 || rc=$?
echo "$rc" >"${lint_tmp}/${i}.rc" ) &
pids+=("$!")
done
wait "${pids[@]}" || true
lint_rc=0
for (( i = 0; i < ${#pids[@]}; i++ )); do
cat "${lint_tmp}/${i}.out"
# A slice that left no rc file died before yamllint returned: that is a failure, not a pass.
slice_rc="$(cat "${lint_tmp}/${i}.rc" 2>/dev/null || true)"
case "$slice_rc" in ''|*[!0-9]*) slice_rc=1 ;; esac
[ "$slice_rc" -gt "$lint_rc" ] && lint_rc="$slice_rc"
done
if [ "$lint_rc" -ne 0 ]; then
err "yaml/lint failed (yamllint exit ${lint_rc}; ${#files[@]} files in ${#pids[@]} parallel slice(s))"
exit "$lint_rc"
fi
ok "yaml/lint passed (${#files[@]} files, ${#pids[@]} parallel slice(s))"
# A workflow's `configurationFile:` must name a file that exists. Renovate's did not after #79 # A workflow's `configurationFile:` must name a file that exists. Renovate's did not after #79
# renamed renovate.json -> renovate.json5, and every scheduled run failed with ENOENT for 11 days # renamed renovate.json -> renovate.json5, and every scheduled run failed with ENOENT for 11 days
+61
View File
@@ -0,0 +1,61 @@
---
# Flux Kustomization: Cloudflare Tunnel connector — reconciles ./platform/cloudflared.
#
# ISOLATION IS THE POINT OF THIS FILE. The Deployment it applies CANNOT become Ready until the
# operator enrols the tunnel (runbooks/cloudflare-tunnel.md): it ships with REPLACE-ME credentials
# and a placeholder UUID, and a wrong or revoked credential later produces the same state. So:
#
# * ITS OWN Kustomization, not a line in platform/kustomization.yaml. `platform` is wait:true and
# gates platform-config, which gates every apps wave; an unenrolled tunnel there would be a
# platform-wide stall ON MERGE, with the cause (a missing manual step) invisible from any Flux
# condition. Same reasoning, and same failure shape, as vpn-ingress's unfilled server key.
# * wait: false. Its pods are legitimately not-Ready whenever the credentials are absent or the
# edge is unreachable. Health is reported by the CloudflaredTunnelDown / ReplicasDegraded
# alerts (platform/cloudflared/alerts.yaml), not by this Kustomization's Ready condition —
# which reaches Ready on APPLY success.
# * NOTHING dependsOn THIS, and nothing should. A missing or invalid credentials Secret can then
# only ever affect the tunnel itself. conformance/tests/l0-static/cloudflared-ingress-rules
# fails the gate if any Kustomization gains `dependsOn: cloudflared`, or if this one stops
# being wait:false.
# * No Certificate, no HelmRelease, no CRD: nothing in the path for Flux's health checks to wait
# on even if wait were flipped (the Certificate-stall lesson).
#
# dependsOn: [platform-config] — A REAL EDGE.
# 1. CRD ORDERING: PodMonitor and PrometheusRule are monitoring.coreos.com CRs whose CRDs are
# installed by kube-prometheus-stack in `platform`; platform-config dependsOn platform, so this
# edge carries that guarantee transitively (Flux aborts a whole pass on an unknown kind).
# 2. THE FAR SIDE OF THE TUNNEL: the Gateway (and so the cilium-gateway-shared Service the ingress
# rules point at) is applied by platform-config. A tunnel that comes up before its origin is a
# support call.
#
# ZERO EFFECT ON LIVE TRAFFIC UNTIL DNS MOVES. Cloudflare only sends a hostname through a tunnel
# when that hostname's DNS record is a CNAME to <uuid>.cfargotunnel.com. This change creates no
# such record and leaves external-dns, the Gateway annotations, Pi-hole and CoreDNS untouched.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: cloudflared
namespace: flux-system
spec:
interval: 10m
retryInterval: 2m
timeout: 5m
prune: true
wait: false
dependsOn:
- name: platform-config
sourceRef:
kind: GitRepository
name: flux-system
path: ./platform/cloudflared
# REQUIRED, not boilerplate: platform/cloudflared/config.yaml renders `${BASE_DOMAIN}` into every
# ingress hostname, and alerts.yaml into both runbook_url annotations. Without this block the
# token becomes the EMPTY STRING and the Kustomization still reports Ready (substitution fails
# OPEN), leaving ingress rules for `plex.` and `*.` — a tunnel that 404s everything at best.
postBuild:
substituteFrom:
- kind: ConfigMap
name: cluster-vars
# SOPS decryption is layered per-environment: clusters/dev and clusters/prod each patch THIS
# Kustomization by name with a `decryption:` stanza for cloudflared-credentials.sops.yaml.
# Without it the Secret applies cleanly full of ENC[...] and cloudflared cannot authenticate.
+11
View File
@@ -78,6 +78,17 @@ resources:
# `wg-server-key` Secret, so in the gating `platform` tier an unfinished manual step would have # `wg-server-key` Secret, so in the gating `platform` tier an unfinished manual step would have
# stalled every apps wave. Carries no SOPS secret, so neither overlay adds a decryption patch. # stalled every apps wave. Carries no SOPS secret, so neither overlay adds a decryption patch.
- vpn-ingress.yaml # Flux Kustomization -> ./platform/vpn-ingress (dependsOn platform-config) - vpn-ingress.yaml # Flux Kustomization -> ./platform/vpn-ingress (dependsOn platform-config)
# CVE visibility scanner (docs/17, ADR-0027) — POST-platform tier (dependsOn platform-config,
# wait:false): a failing scanner must never gate the platform, and nothing dependsOn it. Carries the
# oauth2-proxy credential SOPS file, so both overlays add a decryption patch by name.
- vuln-scanner.yaml # Flux Kustomization -> ./platform/vuln-scanner (dependsOn platform-config)
# Cloudflare Tunnel connector (north-south INGRESS from the Cloudflare edge) — POST-platform tier
# (dependsOn platform-config, wait:false), and NOTHING dependsOn it. Non-gating is load-bearing,
# as for vpn-ingress: the Deployment cannot become Ready until the operator enrols the tunnel
# (runbooks/cloudflare-tunnel.md), so in a gating tier an unfinished manual step — or a revoked
# credential later — would stall every apps wave. Carries cloudflared-credentials.sops.yaml, so
# both overlays add a SOPS decryption patch for it (by name). See clusters/base/cloudflared.yaml.
- cloudflared.yaml # Flux Kustomization -> ./platform/cloudflared (dependsOn platform-config)
# PM agent — reads Gitea issues and runs one worker agent per assignment that authors a # PM agent — reads Gitea issues and runs one worker agent per assignment that authors a
# change, gates it and opens a pull request. POST-APP tier (dependsOn apps-wave-0, # change, gates it and opens a pull request. POST-APP tier (dependsOn apps-wave-0,
# wait:false): what it needs is GITEA, not Keycloak or the Gateway, and it must never gate # wait:false): what it needs is GITEA, not Keycloak or the Gateway, and it must never gate
+41
View File
@@ -0,0 +1,41 @@
---
# Flux Kustomization: the CVE visibility scanner (docs/17, ADR-0027) — reconciles ./platform/vuln-scanner.
#
# ISOLATION IS THE POINT, as for clusters/base/cloudflared.yaml. A failing scanner, an unbound PVC or a
# pending Certificate must never become a stalled platform, so:
# * ITS OWN Kustomization, not a line in platform/kustomization.yaml;
# * wait: false — its health is reported by the VulnScanStale/VulnDatabaseStale/VulnCoverageGap/
# VulnCriticalFixAvailable/VulnTriageBotSilent alerts (platform/vuln-scanner/alerts.yaml), not by
# this object's Ready condition, which reaches Ready on apply success;
# * NOTHING dependsOn THIS. conformance/tests/l0-static/vuln-scanner-isolation fails the gate if any
# Kustomization gains `dependsOn: vuln-scanner` or this one stops being wait:false.
#
# dependsOn: [platform-config] — a real edge: PrometheusRule/ServiceMonitor/Gateway API CRDs, the
# Longhorn StorageClass, the Gateway, Keycloak's realm, and Longhorn itself come up through
# `platform`/`platform-config` first.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: vuln-scanner
namespace: flux-system
spec:
interval: 10m
retryInterval: 2m
timeout: 5m
prune: true
wait: false
dependsOn:
- name: platform-config
sourceRef:
kind: GitRepository
name: flux-system
path: ./platform/vuln-scanner
# REQUIRED: the HTTPRoute hostname, the oauth2-proxy issuer/redirect/cookie-domain args and every
# alert's runbook_url are written `${BASE_DOMAIN}`. Substitution fails OPEN, so the file
# conformance/tests/l0-static/flux-postbuild-substitution audits this tier.
postBuild:
substituteFrom:
- kind: ConfigMap
name: cluster-vars
# SOPS decryption is layered per-environment: clusters/dev and clusters/prod each patch THIS
# Kustomization by name with a `decryption:` stanza for the oauth2-proxy credentials.
+32
View File
@@ -140,6 +140,38 @@ patches:
provider: sops provider: sops
secretRef: secretRef:
name: sops-age name: sops-age
# vuln-scanner (the CVE visibility scanner, clusters/base/vuln-scanner.yaml) carries
# platform/vuln-scanner/vuln-scanner-oauth2-proxy-credentials.sops.yaml — the per-pod oauth2-proxy
# client and cookie secrets. Without this stanza the Secret applies full of ENC[...] and the
# sidecar cannot authenticate; the rest of the scanner runs regardless. `sops-age` is out-of-band.
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: vuln-scanner
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
# cloudflared (the Cloudflare Tunnel connector, clusters/base/cloudflared.yaml) carries
# platform/cloudflared/cloudflared-credentials.sops.yaml — the tunnel credentials JSON, the one
# secret in that component. Without this stanza the Secret applies full of ENC[...]: the APPLY
# SUCCEEDS, the Kustomization stays Ready, and the only symptom is a connector that cannot
# authenticate to the edge (CloudflaredTunnelDown fires). The `sops-age` Secret is out-of-band
# (name only).
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: cloudflared
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
# pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
# pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
# account and the Anthropic subscription token from `claude setup-token`. Without this # account and the Anthropic subscription token from `claude setup-token`. Without this
+32
View File
@@ -131,6 +131,38 @@ patches:
provider: sops provider: sops
secretRef: secretRef:
name: sops-age name: sops-age
# vuln-scanner (the CVE visibility scanner, clusters/base/vuln-scanner.yaml) carries
# platform/vuln-scanner/vuln-scanner-oauth2-proxy-credentials.sops.yaml — the per-pod oauth2-proxy
# client and cookie secrets. Without this stanza the Secret applies full of ENC[...] and the
# sidecar cannot authenticate; the rest of the scanner runs regardless. `sops-age` is out-of-band.
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: vuln-scanner
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
# cloudflared (the Cloudflare Tunnel connector, clusters/base/cloudflared.yaml) carries
# platform/cloudflared/cloudflared-credentials.sops.yaml — the tunnel credentials JSON, the one
# secret in that component. Without this stanza the Secret applies full of ENC[...]: the APPLY
# SUCCEEDS, the Kustomization stays Ready, and the only symptom is a connector that cannot
# authenticate to the edge (CloudflaredTunnelDown fires). The `sops-age` Secret is out-of-band
# (name only).
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: cloudflared
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
# pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
# pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
# account and the Anthropic subscription token from `claude setup-token`. Without this # account and the Anthropic subscription token from `claude setup-token`. Without this
+101 -10
View File
@@ -14,7 +14,7 @@ Pure standard library (matches the harness's no-pip-dependency posture); the che
themselves shell out (render checks to catalog/render.py, whose PyYAML dependency lives themselves shell out (render checks to catalog/render.py, whose PyYAML dependency lives
there; L0 checks to git + PyYAML). there; L0 checks to git + PyYAML).
Usage: run_l1_offline.py --junit <out.xml> [--label-prefix P] <test-dir> [<test-dir> ...] Usage: run_l1_offline.py --junit <out.xml> [--label-prefix P] [--jobs N] <test-dir> [<test-dir> ...]
Exit 0 iff every selected offline check passes. Exit 0 iff every selected offline check passes.
The per-dir check script is whichever of `render-check.sh` / `check.sh` the directory The per-dir check script is whichever of `render-check.sh` / `check.sh` the directory
@@ -43,13 +43,35 @@ Exit codes: 0 all selected checks passed / 1 at least one FAILED / 3 nothing fai
scope-declared check was UNWIRED. conformance/run.sh propagates 3 verbatim; ci/l0.sh turns it scope-declared check was UNWIRED. conformance/run.sh propagates 3 verbatim; ci/l0.sh turns it
into the gate's own rc 3. This is the ONE place the "unwired, not green" rule lives now, into the gate's own rc 3. This is the ONE place the "unwired, not green" rule lives now,
instead of once per hand-written stage. instead of once per hand-written stage.
PARALLELISM (--jobs, env CONF_OFFLINE_JOBS)
-------------------------------------------
The checks are independent processes, so they run in a bounded thread pool (one `bash` child
per worker). Default: min(8, usable CPUs); `--jobs 1` is the old serial runner exactly. Each
check's stdout+stderr is CAPTURED and only reported after every check has finished, in the
ORDER THE DIRS WERE GIVEN, so the log, the JUnit and the step summary are byte-for-byte
independent of which check finished first (only the trailing "[N.Ns]" wall-time on each
status line varies). The 0/1/3 aggregation runs over the same ordered case list as before.
Parallel-safety is a property of the CHECKS, not of this runner: every check must keep its
scratch state in a private mktemp/mkdtemp dir and must not write into the tree or a fixed
path. The audit that established that for the current set is recorded in
decisions/2026-11-08-parallel-l0-checks.md; a new check that needs shared state must say so.
Nested pools: the keycloak reconciler checks run their own mutation suites in a pool
(mutations.py, env MUTATION_JOBS). When this runner is itself parallel it exports a SMALLER
MUTATION_JOBS (unless the caller already set one), so the two suites share the CPUs the
other jobs-2 checks are not using instead of each taking min(8, CPUs). See inner_jobs().
""" """
from __future__ import annotations from __future__ import annotations
import argparse import argparse
import os
import re import re
import subprocess import subprocess
import sys import sys
import time
from concurrent.futures import ThreadPoolExecutor
import xml.etree.ElementTree as ET import xml.etree.ElementTree as ET
from pathlib import Path from pathlib import Path
from xml.dom import minidom from xml.dom import minidom
@@ -97,6 +119,46 @@ def collect_meta(test_yaml: Path, prefix: str):
return name, reqs return name, reqs
def usable_cpus() -> int:
"""CPUs this process may run on (the cgroup/affinity view, not the host's count)."""
try:
return max(1, len(os.sched_getaffinity(0)))
except (AttributeError, OSError):
return max(1, os.cpu_count() or 1)
def positive_int(text: str) -> int:
n = int(text)
if n < 1:
raise argparse.ArgumentTypeError(f"must be >= 1, got {n}")
return n
def inner_jobs(cpus: int, outer: int) -> int:
"""MUTATION_JOBS to hand the checks when this runner runs `outer` of them at once.
Two checks (the keycloak client-secret and passkey reconcilers) fan out internally; the
other outer-2 workers are single processes. Give the two suites an equal share of the CPUs
those others leave free, never fewer than 2 each (a 1-wide pool would make the 40-mutation
passkey suite the critical path of the whole stage), never more than 8. Peak process count
is then about (outer-2) + 2*inner, i.e. <= cpus on any machine with cpus >= outer + 2, and
at worst ~1.25-1.5x cpus on a small runner — CPU-bound oversubscription of that size costs
latency per process, not throughput.
"""
if outer <= 1:
return min(8, cpus)
return max(2, min(8, (cpus - outer + 2) // 2))
def run_check(check: Path, env: dict) -> tuple[int, str, float]:
t0 = time.monotonic()
proc = subprocess.run(
["bash", str(check)], capture_output=True, text=True,
stdin=subprocess.DEVNULL, env=env,
)
return proc.returncode, (proc.stdout + proc.stderr).strip(), time.monotonic() - t0
def main() -> int: def main() -> int:
ap = argparse.ArgumentParser(description=__doc__) ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--junit", type=Path, required=True) ap.add_argument("--junit", type=Path, required=True)
@@ -108,26 +170,54 @@ def main() -> int:
"(makes this runner exit 3); repeatable") "(makes this runner exit 3); repeatable")
ap.add_argument("--summary-out", type=Path, default=None, ap.add_argument("--summary-out", type=Path, default=None,
help="write one Markdown line per case here, for the CI step summary") help="write one Markdown line per case here, for the CI step summary")
ap.add_argument("--jobs", type=positive_int, default=None,
help="checks to run at once (default: env CONF_OFFLINE_JOBS, else "
"min(8, usable CPUs)); 1 = serial")
ap.add_argument("dirs", nargs="+", type=Path) ap.add_argument("dirs", nargs="+", type=Path)
args = ap.parse_args() args = ap.parse_args()
strict_scopes = set(args.unwired_scope) strict_scopes = set(args.unwired_scope)
cases = [] # (name, reqs, dir, rc, output, scope) cpus = usable_cpus()
if args.jobs is not None:
jobs = args.jobs
elif os.environ.get("CONF_OFFLINE_JOBS", "").strip():
try:
jobs = positive_int(os.environ["CONF_OFFLINE_JOBS"].strip())
except (ValueError, argparse.ArgumentTypeError) as e:
ap.error(f"CONF_OFFLINE_JOBS={os.environ['CONF_OFFLINE_JOBS']!r}: {e}")
else:
jobs = min(8, cpus)
child_env = dict(os.environ)
if not child_env.get("MUTATION_JOBS", "").strip():
child_env["MUTATION_JOBS"] = str(inner_jobs(cpus, jobs))
print(f"[offline] running {len(args.dirs)} check(s), jobs={jobs} "
f"(cpus={cpus}, MUTATION_JOBS={child_env['MUTATION_JOBS']})", file=sys.stderr)
# Metadata first, in the given order; then the checks themselves through the pool. The
# results are consumed in SUBMISSION order, so `cases` is identical to the serial runner's.
planned = [] # (name, reqs, dir, scope, check-or-None)
for d in args.dirs: for d in args.dirs:
check = next((d / s for s in CHECK_SCRIPTS if (d / s).exists()), None) check = next((d / s for s in CHECK_SCRIPTS if (d / s).exists()), None)
name, reqs = collect_meta(d / "chainsaw-test.yaml", args.label_prefix) name, reqs = collect_meta(d / "chainsaw-test.yaml", args.label_prefix)
scope = scope_of(d / "chainsaw-test.yaml", args.scope_annotation) scope = scope_of(d / "chainsaw-test.yaml", args.scope_annotation)
if check is None: planned.append((name, reqs, d, scope, check))
cases = [] # (name, reqs, dir, rc, output, scope)
elapsed = [] # wall seconds per case, same order (None: nothing ran)
with ThreadPoolExecutor(max_workers=jobs) as pool:
futures = [pool.submit(run_check, check, child_env) if check is not None else None
for (_n, _r, _d, _s, check) in planned]
for (name, reqs, d, scope, check), fut in zip(planned, futures):
if fut is None:
cases.append((name, reqs, d, 3, cases.append((name, reqs, d, 3,
f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}", f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}",
scope)) scope))
elapsed.append(None)
continue continue
proc = subprocess.run( rc, out, secs = fut.result()
["bash", str(check)], capture_output=True, text=True cases.append((name, reqs, d, rc, out, scope))
) elapsed.append(secs)
out = (proc.stdout + proc.stderr).strip()
cases.append((name, reqs, d, proc.returncode, out, scope))
failures = sum(1 for c in cases if c[3] not in (0, 3)) failures = sum(1 for c in cases if c[3] not in (0, 3))
skipped = sum(1 for c in cases if c[3] == 3) skipped = sum(1 for c in cases if c[3] == 3)
@@ -145,7 +235,7 @@ def main() -> int:
}, },
) )
summary_lines = [] summary_lines = []
for name, reqs, d, rc, out, scope in cases: for (name, reqs, d, rc, out, scope), secs in zip(cases, elapsed):
tc = ET.SubElement( tc = ET.SubElement(
suite, suite,
"testcase", "testcase",
@@ -164,8 +254,9 @@ def main() -> int:
fail.text = out fail.text = out
status = "PASS" if rc == 0 else ("SKIP" if rc == 3 else "FAIL") status = "PASS" if rc == 0 else ("SKIP" if rc == 3 else "FAIL")
mark = f" scope={scope}" if scope else "" mark = f" scope={scope}" if scope else ""
took = f" [{secs:.1f}s]" if secs is not None else ""
print(f"[offline] {layer_of(d):3s} {status:4s} {name} " print(f"[offline] {layer_of(d):3s} {status:4s} {name} "
f"({','.join(reqs) or 'untagged'}){mark}", f"({','.join(reqs) or 'untagged'}){mark}{took}",
file=sys.stderr) file=sys.stderr)
if rc not in (0, 3): if rc not in (0, 3):
print(out, file=sys.stderr) print(out, file=sys.stderr)
+38
View File
@@ -135,6 +135,44 @@ requirements:
safety_critical: false safety_critical: false
summary: "Update policy per-service configurable (cadence, channel, pinning)." summary: "Update policy per-service configurable (cadence, channel, pinning)."
# --- Vulnerability visibility ---
- id: R-VULN-01
doc: docs/17-vulnerability-visibility.md
safety_critical: false
summary: "Inventory every running container image by digest with its workloads, from the cluster."
- id: R-VULN-02
doc: docs/17-vulnerability-visibility.md
safety_critical: false
summary: "Every running image evaluated against a current database at least every 24h; new images within 2h."
- id: R-VULN-03
doc: docs/17-vulnerability-visibility.md
safety_critical: false
summary: "Any target not evaluated is visibly reported with its reason; absence is never shown as clean."
- id: R-VULN-04
doc: docs/17-vulnerability-visibility.md
safety_critical: false
summary: "Authenticated page lists every CVE with affected images, containers and workloads; searchable and filterable."
- id: R-VULN-05
doc: docs/17-vulnerability-visibility.md
safety_critical: false
summary: "Inventory, scanning, reporting and triage run on a schedule with no human or AI action; loud failure."
- id: R-VULN-06
doc: docs/17-vulnerability-visibility.md
safety_critical: false
summary: "Scan failure, staleness and unaccepted CRITICAL fixes reported by email as grouped alerts."
- id: R-VULN-07
doc: docs/17-vulnerability-visibility.md
safety_critical: false
summary: "Scheduled bot keeps one dashboard issue and requests remediation via Renovate; no CVE detail exposed."
- id: R-VULN-08
doc: docs/17-vulnerability-visibility.md
safety_critical: false
summary: "Accepted risk declared in git with reason and mandatory expiry; reverts to open on expiry."
- id: R-VULN-09
doc: docs/17-vulnerability-visibility.md
safety_critical: false
summary: "Scanner digest-pinned, no auto-merged updates, no Secret read or write access, no git credential."
# --- Observability & alerting --- # --- Observability & alerting ---
- id: R-OBS-01 - id: R-OBS-01
doc: docs/10-observability-and-alerting.md doc: docs/10-observability-and-alerting.md
@@ -411,6 +411,54 @@ SCRAPE_OWNERS: list[dict] = [
"Keeping the two in one file makes that pair inseparable", "Keeping the two in one file makes that pair inseparable",
}, },
}, },
# ── ADDED 2026-11-09 (CVE visibility, ADR-0027). A NEW OWNER, NOT A RELAXED ASSERTION. ──────
# Registers the vuln_scanner_* family and the Service+ServiceMonitor that scrapes it.
{
"owner": "vuln-scanner: Service + ServiceMonitor on port `metrics` (:9090)",
"prefixes": ("vuln_scanner_",),
"file": "platform/vuln-scanner/servicemonitor.yaml",
"require": [
r"^kind:\s*ServiceMonitor\s*$",
r"^\s*-\s*port:\s*metrics\s*$",
],
"forbid": [],
"why": "vuln_scanner_* powers the five VulnScan alerts, the only email path for scan "
"failure, staleness and overdue critical fixes (R-VULN-06). Two of those rules fire "
"on an absent metric, so a lost scrape is loud rather than inert",
"companion": {
"file": "platform/vuln-scanner/kustomization.yaml",
"require": [r"^\s*-\s*networkpolicy\.yaml\s*$"],
"why": "platform/vuln-scanner/networkpolicy.yaml ships the namespace's default-deny "
"and the ingress allow for Prometheus on :9090 in one file, so a baseline "
"without its allow cannot empty the TSDB",
},
},
# ── ADDED 2026-11-08 (feat/cloudflare-tunnel). A NEW OWNER, NOT A RELAXED ASSERTION. ───────
# Purely additive, the same shape as the kured entry: it registers the cloudflared_* family
# and the Service+ServiceMonitor that scrapes it, and makes the check STRICTER for that component.
{
"owner": "cloudflared: headless metrics Service + ServiceMonitor on port `metrics` (:2000)",
"prefixes": ("cloudflared_",),
"file": "platform/cloudflared/servicemonitor.yaml",
"require": [
r"^kind:\s*ServiceMonitor\s*$",
r"^kind:\s*Service\s*$",
r"^\s*-\s*port:\s*metrics\s*$",
],
"forbid": [],
"why": "cloudflared_tunnel_ha_connections powers CloudflaredTunnelDown, the only signal "
"that the tunnel has lost the Cloudflare edge. That rule is written to FIRE on an "
"absent series, so a lost scrape is loud rather than inert — but it would then be "
"permanently firing for a wiring reason, which teaches the operator to ignore it",
"companion": {
"file": "platform/cloudflared/kustomization.yaml",
"require": [r"^\s*-\s*networkpolicy\.yaml\s*$"],
"why": "platform/cloudflared/networkpolicy.yaml ships the namespace's default-deny "
"(a platform namespace, outside Kyverno's generator) together with the ingress "
"allow for Prometheus on :2000. Keeping them in one file, applied together, is "
"what stops a baseline-without-allow from emptying the TSDB",
},
},
] ]
# `up` is produced by EVERY scrape target, so no owner can meaningfully claim it and assertion (1) # `up` is produced by EVERY scrape target, so no owner can meaningfully claim it and assertion (1)
@@ -0,0 +1,47 @@
# =============================================================================
# L0 STATIC — the Cloudflare Tunnel exposes only what the operator decided, and cannot stall Flux
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# platform/cloudflared/config.yaml is the entire internet-facing surface of the tunnel. This test
# asserts the operator decisions of 2026-11-08 against it on every gate run: no rule may target
# anything but a 404 or the shared Gateway (so never the Kubernetes API); the LAN/VPN-only
# hostnames (plex, jackett, pihole, semaphore, qbittorrent, radarr, sonarr, headlamp,
# unifi, crafty-mc) 404 before the wildcard; the
# egress policy admits no API/node entity; and the Flux tier is isolated (wait:false, nothing
# dependsOn it, both overlays decrypt it). It proves itself non-vacuous by mutation. Rationale and
# the honest NOT-proven list: check.sh.
#
# WHY SCOPE, NOT REQUIREMENTS: the property spans platform/cloudflared, platform/gateway (the
# derived Service name), clusters/base and both overlays. A requirement-scoped selection would skip
# it on, e.g., a Gateway rename — precisely a change that breaks it.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: cloudflared-ingress-rules
labels:
conformance.home.k3s/R-NET-01: "true"
conformance.home.k3s/R-NET-02: "true"
annotations:
conformance.home.k3s/requirements: "R-NET-01, R-NET-02"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "true"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
The Cloudflare Tunnel routes only to the shared Gateway or a 404, never to the Kubernetes
API; the LAN/VPN-only hostnames 404 ahead of the wildcard; the connector's egress admits no
API or node entity; and its Flux tier cannot stall any other tier.
spec:
steps:
- name: assert-tunnel-ingress-rules-and-isolation
description: >-
Parse platform/cloudflared/config.yaml with BASE_DOMAIN substituted, assert every rule's
target against the Gateway Service derived from platform/gateway/gateway.yaml, assert the
excluded hostnames 404 first, check the egress CiliumNetworkPolicy and the Flux wiring, and
run the evaluator on mutated configs to prove it is not vacuous; and assert the
credentials Secret volume is readable by the container's run-as uid/groups (fsGroup +
mode), self-tested by mutation.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,321 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — cloudflared-ingress-rules
# Serves: R-NET-01 / R-NET-02 (default-deny; allow rules declarative and scoped), and the four
# operator decisions of 2026-11-08 recorded in decisions/2026-11-08-cloudflare-tunnel.md.
#
# WHY THIS EXISTS. platform/cloudflared/config.yaml is the whole of what the internet can reach
# through the Cloudflare Tunnel. A one-line edit there — a rule pointing at the Kubernetes API, a
# LAN-only hostname moved below the wildcard, a dropped catch-all — publishes something the operator
# decided must never be public, and nothing in the cluster turns red: the tunnel stays connected and
# the Kustomization stays Ready. So the decisions are asserted here, on every gate run.
#
# GIVEN the repository as authored,
# THEN (1) every ingress rule's `service` is `http_status:404` or EXACTLY the shared Gateway's
# in-cluster Service, derived from platform/gateway/gateway.yaml (so a Gateway rename
# fails here rather than in production) — which by construction excludes the
# Kubernetes API, and a named deny-list says so explicitly when it is violated;
# (2) each LAN/VPN-only hostname (plex, plex-vip, jackett, pihole, pihole-dns, semaphore,
# qbittorrent, radarr, sonarr, headlamp, unifi, crafty-mc) has a rule mapping it to http_status:404 that comes BEFORE any
# Gateway-bound rule (cloudflared matches first-rule-wins);
# (3) the apex and the wildcard are the Gateway rules, with matchSNItoHost, and nothing
# disables origin TLS verification; the last rule is a hostname-less 404 catch-all;
# (4) the egress CiliumNetworkPolicy admits none of kube-apiserver/host/remote-node/
# cluster/all and no CIDR rule, and its backend selector excludes the LAN-only
# namespaces (the second, independent layer behind (2));
# (5) the Flux tier is ISOLATED: its own Kustomization, wait:false, not listed in
# platform/kustomization.yaml, nothing anywhere dependsOn it, and both overlays carry its
# SOPS decryption patch — so a missing or invalid credential can stall nothing else;
# (6) the rule evaluator is not vacuous: it is run against mutated copies of the real config
# (an API rule added, plex moved below the wildcard, the catch-all removed) and must
# flag each one;
# AND (7) every Secret volume the container mounts — the tunnel credentials above all — is
# READABLE by the container's run-as identity, modelled the way the kubelet lays it out:
# files owned by uid 0, group-owned by fsGroup if set (else gid 0), with the item mode or
# defaultMode (API default 0644). The identity is the container securityContext over
# the pod's; the process's groups are runAsGroup + fsGroup + supplementalGroups. An
# unset runAsUser is a FAIL, not a pass: the image default cannot be read offline. The
# model is deliberately conservative — it does not credit the kubelet's habit of
# OR-ing group-read into fsGroup-owned volumes, so 0400+fsGroup fails here though it
# would work. Self-tested by mutation (fsGroup removed; mode 0400; runAsUser unset).
# This is the 2026-11-08 CrashLoop: runAsUser 65532, no fsGroup, 0440 root:root.
#
# NOT proven (static): that Cloudflare routes only the hostnames whose DNS points at the tunnel,
# that the Gateway's certificate validates from cloudflared, or that Envoy enforces (4) — those are
# runtime properties, verified by the operator per runbooks/cloudflare-tunnel.md.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
exec python3 - "$ROOT" <<'PY'
import copy
import re
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("CHECK FAIL (cloudflared-ingress-rules): PyYAML is required", file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
FAILURES, CHECKS = [], 0
def check(cond, desc):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(desc)
def docs(path):
return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)]
COMP = ROOT / "platform" / "cloudflared"
CONFIG = COMP / "config.yaml"
if not CONFIG.is_file():
print(f"CHECK FAIL (cloudflared-ingress-rules): {CONFIG.relative_to(ROOT)} is absent — the "
"component this check guards was moved or deleted; re-point or delete this check in the "
"SAME change", file=sys.stderr)
raise SystemExit(1)
# The hostnames the operator decided never go through the tunnel (decisions 2026-11-08).
# headlamp: operator decision 2026-11-08 (LAN/VPN-only). qbittorrent/radarr/sonarr: catalog
# `exposure: internal`, like jackett/pihole/semaphore.
EXCLUDED_HOSTS = ("plex", "plex-vip", "jackett", "pihole", "pihole-dns", "semaphore",
"qbittorrent", "radarr", "sonarr", "headlamp", "cve", "unifi", "crafty-mc")
# Their Gateway-backend namespaces (unifi / crafty-mc are L4 VIPs, not Gateway backends).
EXCLUDED_NAMESPACES = ("plex", "jackett", "pihole", "semaphore", "qbittorrent", "radarr",
"sonarr", "headlamp", "vuln-scanner")
FORBIDDEN_ENTITIES = {"kube-apiserver", "host", "remote-node", "cluster", "all"}
# Any of these in a `service` means a route toward the Kubernetes API (or a raw-TCP/SSH bastion).
API_MARKERS = ("kubernetes", "kube-apiserver", "apiserver", ":6443", "10.43.0.1", "tcp://",
"ssh://", "unix:", "bastion")
# ── substitution: ${BASE_DOMAIN} from cluster-vars, the ONLY token allowed ──────────────────────
cv = next(d for d in docs(ROOT / "clusters" / "base" / "cluster-vars.yaml")
if d.get("kind") == "ConfigMap")
base_domain = cv["data"]["BASE_DOMAIN"]
raw = CONFIG.read_text()
tokens = set(re.findall(r"\$\{([A-Za-z0-9_]+)\}", raw))
check(tokens == {"BASE_DOMAIN"},
f"(0) config.yaml uses exactly the ${{BASE_DOMAIN}} token (found {sorted(tokens)}); any other "
f"token is substituted from cluster-vars unseen by this check")
check(re.search(r"\$(?!\{BASE_DOMAIN\})", raw) is None,
"(0) config.yaml contains a bare `$` — Flux envsubst would rewrite or empty it")
cfg = yaml.safe_load(raw.replace("${BASE_DOMAIN}", base_domain))
# ── the Gateway Service, derived rather than hard-coded ─────────────────────────────────────────
gw = next(d for d in docs(ROOT / "platform" / "gateway" / "gateway.yaml") if d.get("kind") == "Gateway")
GATEWAY_SVC = (f"https://cilium-gateway-{gw['metadata']['name']}."
f"{gw['metadata']['namespace']}.svc.cluster.local:443")
def evaluate(cfg):
"""Return the list of rule violations for a parsed cloudflared config."""
bad = []
rules = cfg.get("ingress") or []
if len(rules) < len(EXCLUDED_HOSTS) + 3:
bad.append(f"only {len(rules)} ingress rules — the scan is vacuous")
return bad
for i, r in enumerate(rules):
svc = str(r.get("service", ""))
if any(m in svc.lower() for m in API_MARKERS):
bad.append(f"rule {i} ({r.get('hostname', '<catch-all>')}) targets {svc!r}, which "
f"is a route toward the Kubernetes API / a raw bastion — the API stays "
f"local-only (operator decision 2026-11-08)")
if svc not in ("http_status:404", GATEWAY_SVC):
bad.append(f"rule {i} ({r.get('hostname', '<catch-all>')}) targets {svc!r}; only "
f"http_status:404 or {GATEWAY_SVC} are permitted")
if (r.get("originRequest") or {}).get("noTLSVerify"):
bad.append(f"rule {i} disables origin TLS verification (noTLSVerify)")
if (cfg.get("originRequest") or {}).get("noTLSVerify"):
bad.append("top-level originRequest disables origin TLS verification (noTLSVerify)")
gw_idx = [i for i, r in enumerate(rules) if r.get("service") == GATEWAY_SVC]
first_gw = min(gw_idx) if gw_idx else len(rules)
for h in EXCLUDED_HOSTS:
fqdn = f"{h}.{base_domain}"
hits = [i for i, r in enumerate(rules) if r.get("hostname") == fqdn]
if not hits:
bad.append(f"no rule for LAN/VPN-only host {fqdn}; it would fall through to the "
f"wildcard and reach the Gateway")
continue
i = hits[0]
if rules[i].get("service") != "http_status:404":
bad.append(f"{fqdn} maps to {rules[i].get('service')!r}, not http_status:404")
if i > first_gw:
bad.append(f"{fqdn}'s 404 rule (#{i}) comes AFTER the first Gateway rule (#{first_gw}); "
f"cloudflared is first-match, so the Gateway rule wins")
gw_hosts = {rules[i].get("hostname") for i in gw_idx}
if gw_hosts != {base_domain, f"*.{base_domain}"}:
bad.append(f"the Gateway-bound hostnames are {sorted(map(str, gw_hosts))}; expected "
f"exactly the apex and the wildcard")
for i in gw_idx:
if (rules[i].get("originRequest") or {}).get("matchSNItoHost") is not True:
bad.append(f"Gateway rule #{i} lacks originRequest.matchSNItoHost: true — the Gateway "
f"would not see the SNI that selects its wildcard certificate")
last = rules[-1]
if "hostname" in last or last.get("service") != "http_status:404":
bad.append("the last rule is not a hostname-less http_status:404 catch-all")
return bad
# (1)-(3) the real config
for v in evaluate(cfg):
check(False, f"(1-3) {v}")
check(str(cfg.get("credentials-file", "")).startswith("/etc/cloudflared/creds/"),
"(3) credentials-file points into the mounted Secret volume")
# (6) mutation self-test — the evaluator must flag each broken variant
mutants = {}
m = copy.deepcopy(cfg)
m["ingress"].insert(0, {"hostname": f"k8s.{base_domain}",
"service": "https://kubernetes.default.svc:443"})
mutants["API rule added"] = m
m = copy.deepcopy(cfg)
plex = next(r for r in m["ingress"] if r.get("hostname") == f"plex.{base_domain}")
m["ingress"].remove(plex)
m["ingress"].insert(len(m["ingress"]) - 1, plex)
mutants["plex moved below the wildcard"] = m
m = copy.deepcopy(cfg)
m["ingress"] = m["ingress"][:-1]
mutants["catch-all removed"] = m
for name, mut in mutants.items():
check(evaluate(mut), f"(6) self-test: the evaluator did NOT flag the mutant '{name}' — the "
f"check is vacuous")
# (4) the egress policy
np_docs = docs(COMP / "networkpolicy.yaml")
cnps = [d for d in np_docs if d.get("kind") == "CiliumNetworkPolicy"]
check(len(cnps) == 1, f"(4) exactly one CiliumNetworkPolicy in networkpolicy.yaml (found {len(cnps)})")
for cnp in cnps:
for i, rule in enumerate(cnp["spec"].get("egress") or []):
ents = set(rule.get("toEntities") or [])
check(not (ents & FORBIDDEN_ENTITIES),
f"(4) egress rule {i} admits {sorted(ents & FORBIDDEN_ENTITIES)} — that reaches the "
f"Kubernetes API or the nodes; the API stays local-only")
check(not (rule.get("toCIDR") or rule.get("toCIDRSet")),
f"(4) egress rule {i} uses a CIDR — a CIDR can name the API VIP; use entities")
for ep in rule.get("toEndpoints") or []:
notin = [e for e in ep.get("matchExpressions") or []
if e.get("key") == "io.kubernetes.pod.namespace" and e.get("operator") == "NotIn"]
missing = set(EXCLUDED_NAMESPACES) - set(notin[0]["values"] if notin else [])
check(not missing,
f"(4) backend selector in egress rule {i} does not exclude {sorted(missing)}")
for d in np_docs:
if d.get("kind") == "NetworkPolicy" and d["metadata"]["name"] == "default-deny":
check(set(d["spec"].get("policyTypes", [])) == {"Ingress", "Egress"},
"(4) default-deny covers both directions")
break
else:
check(False, "(4) networkpolicy.yaml ships no default-deny baseline for the namespace")
# (5) Flux isolation
fk = next(d for d in docs(ROOT / "clusters" / "base" / "cloudflared.yaml")
if d.get("kind") == "Kustomization")
check(fk["spec"].get("wait") is False,
"(5) clusters/base/cloudflared.yaml must be wait: false — its pods are legitimately not-Ready "
"until the tunnel is enrolled")
check(fk["spec"].get("path") == "./platform/cloudflared", "(5) the Kustomization reconciles ./platform/cloudflared")
check("cloudflared.yaml" in (ROOT / "clusters" / "base" / "kustomization.yaml").read_text(),
"(5) clusters/base/kustomization.yaml lists cloudflared.yaml")
check("cloudflared" not in (ROOT / "platform" / "kustomization.yaml").read_text(),
"(5) platform/kustomization.yaml must not include cloudflared — that is the gating tier")
for f in sorted((ROOT / "clusters").rglob("*.yaml")):
for d in docs(f):
for dep in (d.get("spec") or {}).get("dependsOn") or []:
check(not (isinstance(dep, dict) and dep.get("name") == "cloudflared"),
f"(5) {f.relative_to(ROOT)} {d.get('metadata', {}).get('name')} dependsOn "
f"cloudflared — that hands an unenrolled tunnel the power to stall it")
for env in ("dev", "prod"):
text = (ROOT / "clusters" / env / "kustomization.yaml").read_text()
check(re.search(r"name:\s*cloudflared\s*\n\s*namespace:\s*flux-system\s*\n\s*spec:\s*\n"
r"\s*decryption:", text) is not None,
f"(5) clusters/{env}/kustomization.yaml carries the cloudflared SOPS decryption patch")
dep = next(d for d in docs(COMP / "deployment.yaml") if d.get("kind") == "Deployment")
check(dep["spec"].get("replicas", 1) >= 2, "(5) the Deployment runs at least 2 replicas")
check(dep["spec"]["template"]["metadata"]["labels"].get("homelab.keithjackson.dev/egress-class")
== "internet",
"(4) the pods are egress-class=internet — vpn would put the edge connections behind PIA")
check(dep["spec"]["template"]["spec"].get("automountServiceAccountToken") is False,
"(4) the pods mount no ServiceAccount token")
# (7) the credentials Secret is readable by the run-as identity
def unreadable_secret_volumes(pod_spec):
"""Return a description per Secret volume the containers mount but cannot read."""
bad = []
psc = pod_spec.get("securityContext") or {}
vols = {v["name"]: v for v in pod_spec.get("volumes") or [] if "secret" in v}
for c in (pod_spec.get("initContainers") or []) + (pod_spec.get("containers") or []):
csc = c.get("securityContext") or {}
uid = csc.get("runAsUser", psc.get("runAsUser"))
gid = csc.get("runAsGroup", psc.get("runAsGroup"))
fs_group = psc.get("fsGroup")
groups = {g for g in [gid, fs_group, *(psc.get("supplementalGroups") or [])]
if g is not None}
for vm in c.get("volumeMounts") or []:
v = vols.get(vm["name"])
if v is None:
continue
sec = v["secret"]
if uid is None:
bad.append(f"container {c['name']} mounts Secret volume {vm['name']} but sets no "
f"runAsUser (pod or container) — readability cannot be proven offline")
continue
default = sec.get("defaultMode", 0o644)
modes = [it.get("mode", default) for it in sec.get("items") or []] or [default]
owner_gid = fs_group if fs_group is not None else 0
for mode in modes:
readable = ((uid == 0 and mode & 0o400) or (owner_gid in groups and mode & 0o040)
or mode & 0o004)
if not readable:
bad.append(f"container {c['name']} (uid {uid}, groups {sorted(groups)}) cannot "
f"read Secret volume {vm['name']} ({sec.get('secretName')}): files "
f"are 0:{owner_gid} mode {mode:04o} — set pod fsGroup to a group "
f"the container runs with, keeping 0440")
return bad
pod_spec = dep["spec"]["template"]["spec"]
creds_vol = [v for v in pod_spec.get("volumes") or []
if (v.get("secret") or {}).get("secretName") == "cloudflared-credentials"]
check(len(creds_vol) == 1, "(7) the Deployment mounts Secret cloudflared-credentials as a volume")
for v in unreadable_secret_volumes(pod_spec):
check(False, f"(7) {v}")
sec_mutants = {}
m = copy.deepcopy(pod_spec)
m["securityContext"].pop("fsGroup", None)
m["securityContext"].pop("supplementalGroups", None)
sec_mutants["fsGroup removed"] = m
m = copy.deepcopy(pod_spec)
for v in m["volumes"]:
if "secret" in v:
v["secret"]["defaultMode"] = 0o400
for it in v["secret"].get("items") or []:
it["mode"] = 0o400
sec_mutants["mode 0400"] = m
m = copy.deepcopy(pod_spec)
m["securityContext"].pop("runAsUser", None)
for c in m["containers"]:
(c.get("securityContext") or {}).pop("runAsUser", None)
sec_mutants["runAsUser unset"] = m
for name, mut in sec_mutants.items():
check(unreadable_secret_volumes(mut),
f"(7) self-test: the readability model did NOT flag the mutant '{name}' — vacuous")
if CHECKS < 25:
FAILURES.append(f"only {CHECKS} assertions ran — the check degenerated")
if FAILURES:
print(f"CHECK FAIL (cloudflared-ingress-rules): {len(FAILURES)} of {CHECKS} assertions failed",
file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
print(f"PASS (cloudflared-ingress-rules): {CHECKS} assertions")
PY
@@ -92,9 +92,10 @@
# the NAME axis only. R-NET-04 / the edge port-forward posture owns the other. # the NAME axis only. R-NET-04 / the edge port-forward posture owns the other.
# - Anything about the seven non-catalog publishers' own correctness. The register records WHAT # - Anything about the seven non-catalog publishers' own correctness. The register records WHAT
# each publishes so the claim is reviewable, which is all a static check can offer. # each publishes so the claim is reviewable, which is all a static check can offer.
# - Anything about the five `public` services' TARGET. They still resolve to the Gateway VIP # - That the public names RESOLVE to Cloudflare at runtime. check.sh (9) pins the TREE: the
# rather than a routable address; that is an open operator decision recorded in # Gateway target is the tunnel UUID in platform/cloudflared/config.yaml, proxying is on, the L4
# platform/externaldns/helmrelease.yaml, and asserting a target here would hard-code a guess. # `service` publishers opt out, and every name the tunnel 404s is unpublished. Whether Cloudflare
# then answers anycast for them is a live read (`dig @1.1.1.1`), not a static one.
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06). # - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
# ============================================================================= # =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1 apiVersion: chainsaw.kyverno.io/v1alpha1
@@ -29,7 +29,17 @@
# ci/l0.sh stage (whole-repo-invariants-wired fails the gate if one appears); # ci/l0.sh stage (whole-repo-invariants-wired fails the gate if one appears);
# (8) the external-dns HelmRelease declares `annotationPrefix` explicitly and it matches the # (8) the external-dns HelmRelease declares `annotationPrefix` explicitly and it matches the
# prefix of EVERY external-dns annotation in the tree and in catalog/render.py — added after # prefix of EVERY external-dns annotation in the tree and in catalog/render.py — added after
# the 2026-11-07 v0.23.0 default-prefix change silently voided (1)-(3) at runtime. # the 2026-11-07 v0.23.0 default-prefix change silently voided (1)-(3) at runtime;
# (9) THE CLOUDFLARE TUNNEL WIRING (2026-11-08) — every public name is a PROXIED CNAME to the tunnel:
# (a) the shared Gateway's `target` is EXACTLY `<tunnel:>.cfargotunnel.com`, the UUID read from
# platform/cloudflared/config.yaml, and the HelmRelease sets `--cloudflare-proxied`;
# (b) every `service`-source publisher carries `cloudflare-proxied: "false"` (L4 A records stay
# DNS-only), and no HTTPRoute opts out of proxying (a DNS-only tunnel CNAME is dead);
# (c) no hostname the tunnel maps to `http_status:404` has a public record — no undisowned route
# names it, and the only `service`-source publishers of one are the declared L4 DNS-only
# names (unifi, crafty-mc); conversely every disowned route's hostname IS 404'd by the
# tunnel, so the "unpublished" and "refused at the tunnel" lists cannot drift apart;
# (d) the operator's LAN/VPN-only names (plex, headlamp) are disowned everywhere they appear.
# #
# WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC, offline, cluster-free): # WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC, offline, cluster-free):
# - That the record is gone from Cloudflare. Live read; belongs beside # - That the record is gone from Cloudflare. Live read; belongs beside
@@ -39,8 +49,11 @@
# both the A and its `a-` TXT within ~60s) — but a manual proof is not a gate. # both the A and its `a-` TXT within ~60s) — but a manual proof is not a gate.
# - That an internal name is UNREACHABLE. A port-forward plus a literal IP needs no DNS record. # - That an internal name is UNREACHABLE. A port-forward plus a literal IP needs no DNS record.
# This check owns the NAME axis only. # This check owns the NAME axis only.
# - Anything about the five public services' TARGET. They still answer with the Gateway VIP; the # - That the tunnel actually SERVES the public names, or that Cloudflare honoured `proxied`. (9)
# routable-target decision is open and recorded in platform/externaldns/helmrelease.yaml. # proves the tree asks for a proxied CNAME to the right tunnel; the live answer (`dig @1.1.1.1`
# returning Cloudflare anycast) is a runtime read. Nor does (8)/(9) prove the RUNNING controller
# honours the prefix — a future chart bump that drops it would publish PROXIED A records at the
# Gateway VIP; (8) only keeps the tree and the HelmRelease value consistent.
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06). # - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
# #
# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns # Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns
@@ -141,9 +154,12 @@ def docs_of(p: Path):
# unadjudicated set cannot grow without a review event. Same ratchet reasoning as # unadjudicated set cannot grow without a review event. Same ratchet reasoning as
# KNOWN_EXTERNAL_MONITORS in servicemonitor-ports-resolve. # KNOWN_EXTERNAL_MONITORS in servicemonitor-ports-resolve.
# #
# EVERY ENTRY HERE IS A NAME THAT STILL ANSWERS WITH AN RFC1918 ADDRESS IN A PUBLIC ZONE. That is # WHAT THE ENTRIES PUBLISH NOW (2026-11-08). Every HTTPRoute entry that is not disowned publishes a
# the honest state after this change, and it is deliberate: converting them is an operator # PROXIED CNAME to the Cloudflare Tunnel (the shared Gateway's `target`, assertion 9a), so no origin
# decision per name (is grafana meant to be publicly resolvable at all?), not a renderer change, # address of ours is in the public zone for it. Headlamp is disowned (operator: LAN/VPN-only, 9d).
# The two `service`-source entries still publish their RFC1918 LB VIP as a DNS-only A record —
# deliberate (L4 ports cannot ride Cloudflare's HTTP proxy), and pinned DNS-only by 9b. Whether each
# remaining name should be public at all stays an operator decision per name, not a renderer change,
# and a cluster-wide ExternalDNS opt-in filter — the available hardening, see # and a cluster-wide ExternalDNS opt-in filter — the available hardening, see
# platform/externaldns/helmrelease.yaml — would DELETE all of them at once the moment it is set. # platform/externaldns/helmrelease.yaml — would DELETE all of them at once the moment it is set.
# #
@@ -151,27 +167,32 @@ def docs_of(p: Path):
# ============================================================================================= # =============================================================================================
KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = { KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = {
"platform/oauth2-proxy/httproute.yaml:oauth2-proxy/oauth2-proxy": "platform/oauth2-proxy/httproute.yaml:oauth2-proxy/oauth2-proxy":
"publishes auth.${BASE_DOMAIN} -> the Gateway VIP. The oauth2-proxy login endpoint is a " "publishes auth.${BASE_DOMAIN} -> the tunnel (proxied CNAME). The oauth2-proxy login endpoint is a "
"platform component, not a catalog service; it must be reachable by every browser that " "platform component, not a catalog service; it must be reachable by every browser that "
"any forward-auth app redirects, so whether it is PUBLICLY resolvable is coupled to " "any forward-auth app redirects, so whether it is PUBLICLY resolvable is coupled to "
"whether the five public services are — an operator decision, not a renderer one.", "whether the five public services are — an operator decision, not a renderer one.",
"platform/keycloak/httproute.yaml:keycloak/keycloak": "platform/keycloak/httproute.yaml:keycloak/keycloak":
"publishes keycloak.${BASE_DOMAIN} -> the Gateway VIP, and it is the route that actually " "publishes keycloak.${BASE_DOMAIN} -> the tunnel (proxied CNAME), and it is the route that actually "
"OWNS that record (the TXT registry entry reads resource=httproute/keycloak/keycloak, not " "OWNS that record (the TXT registry entry reads resource=httproute/keycloak/keycloak, not "
"the catalog-rendered keycloak-8080 route, because external-dns merges duplicate " "the catalog-rendered keycloak-8080 route, because external-dns merges duplicate "
"hostnames and this one won). Keycloak is `exposure: public` in the catalog, so the name " "hostnames and this one won). Keycloak is `exposure: public` in the catalog, so the name "
"is meant to be public either way; it is listed because the PUBLISHING object is the " "is meant to be public either way; it is listed because the PUBLISHING object is the "
"hand-authored platform route, which carries no exposure field.", "hand-authored platform route, which carries no exposure field.",
"platform/monitoring/grafana-httproute.yaml:monitoring/grafana": "platform/monitoring/grafana-httproute.yaml:monitoring/grafana":
"publishes grafana.${BASE_DOMAIN} -> the Gateway VIP. An admin dashboard; almost " "publishes grafana.${BASE_DOMAIN} -> the tunnel (proxied CNAME). An admin dashboard "
"certainly should not be publicly resolvable, but it is not a catalog service and " "behind Keycloak; it is not a catalog service, so withdrawing it is an operator call per "
"withdrawing it is an operator call with the same LAN-resolution prerequisite as the six " "name (it stayed public in the 2026-11-08 tunnel decision).",
"internal services (Pi-hole must answer first).",
"platform/headlamp/httproute.yaml:headlamp/headlamp": "platform/headlamp/httproute.yaml:headlamp/headlamp":
"publishes headlamp.${BASE_DOMAIN} -> the Gateway VIP. A Kubernetes web UI with native " "publishes NOTHING since 2026-11-08: the route carries `controller: none` (operator "
"Keycloak OIDC; same adjudication as grafana.", "decision: Headlamp is LAN/VPN-only) and the tunnel 404s the name. A Kubernetes web UI "
"with native Keycloak OIDC; listed because it is a hand-authored route, and pinned "
"disowned by (9d).",
"platform/vuln-scanner/httproute.yaml:vuln-scanner/vuln-scanner":
"publishes NOTHING: the route carries `controller: none` (docs/17: the CVE page is internal "
"only) and the tunnel 404s the name. Hand-authored because the scanner is not a catalog "
"service (its cluster-wide read role cannot be expressed in a ServiceDefinition, ADR-0027).",
"platform/homepage/httproute.yaml:homepage/homepage": "platform/homepage/httproute.yaml:homepage/homepage":
"publishes the APEX ${BASE_DOMAIN} itself -> the Gateway VIP, via the https-apex " "publishes the APEX ${BASE_DOMAIN} itself -> the tunnel (proxied CNAME), via the https-apex "
"listener. The apex is not a per-service name and cannot be withdrawn by an exposure " "listener. The apex is not a per-service name and cannot be withdrawn by an exposure "
"declaration.", "declaration.",
"runbooks/serving/httproute.yaml:runbooks/runbooks": "runbooks/serving/httproute.yaml:runbooks/runbooks":
@@ -180,16 +201,16 @@ KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = {
"entry names the homepage route). Deliberately un-authenticated — see the manifest's own " "entry names the homepage route). Deliberately un-authenticated — see the manifest's own "
"header for why gating runbooks on Keycloak is a loop with no exit.", "header for why gating runbooks on Keycloak is a loop with no exit.",
"platform/crafty/httproute.yaml:crafty/crafty": "platform/crafty/httproute.yaml:crafty/crafty":
"publishes crafty.${BASE_DOMAIN} -> the Gateway VIP (the Minecraft admin UI, gated by an " "publishes crafty.${BASE_DOMAIN} -> the tunnel (proxied CNAME) (the Minecraft admin UI, gated by an "
"oauth2-proxy sidecar). Platform component, no exposure field.", "oauth2-proxy sidecar). Platform component, no exposure field.",
"platform/crafty/service-l4.yaml:crafty/crafty-l4-tcp": "platform/crafty/service-l4.yaml:crafty/crafty-l4-tcp":
"the `service` SOURCE, not a route: publishes crafty-mc.${BASE_DOMAIN} -> " "the `service` SOURCE, not a route: publishes crafty-mc.${BASE_DOMAIN} -> "
"${CRAFTY_L4_VIP} (192.168.2.22) for the raw Minecraft TCP/UDP game ports the HTTPS " "${CRAFTY_L4_VIP} (192.168.2.22) for the raw Minecraft TCP/UDP game ports the HTTPS "
"Gateway cannot express. A game server is the one case where a public name may be " "Gateway cannot express. A game server is the one case where a public name may be "
"genuinely wanted; the address it answers with is still RFC1918.", "genuinely wanted; the address it answers with is still RFC1918, DNS-only (9b).",
"apps/vms/unifi/50-service-l4.yaml:unifi/unifi-l4-tcp": "apps/vms/unifi/50-service-l4.yaml:unifi/unifi-l4-tcp":
"the `service` source: publishes unifi.${BASE_DOMAIN} -> ${UNIFI_L4_VIP} " "the `service` source: publishes unifi.${BASE_DOMAIN} -> ${UNIFI_L4_VIP} "
"(192.168.2.21) for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored " "(192.168.2.21), DNS-only (9b), for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored "
"under apps/vms (a VM, not a rendered container service); the catalog's only VM entry is " "under apps/vms (a VM, not a rendered container service); the catalog's only VM entry is "
"vm-demo, which declares no ingressRoutes and publishes nothing.", "vm-demo, which declares no ingressRoutes and publishes nothing.",
} }
@@ -515,6 +536,113 @@ check(not bad_render,
f"(8d) catalog/render.py emits external-dns keys under {bad_render}, but the controller " f"(8d) catalog/render.py emits external-dns keys under {bad_render}, but the controller "
f"reads {ANNOTATION_PREFIX!r}") f"reads {ANNOTATION_PREFIX!r}")
# =============================================================================================
# (9) THE CLOUDFLARE TUNNEL WIRING (2026-11-08). The public names are PROXIED CNAMEs to the tunnel.
# =============================================================================================
CF_CONFIG = ROOT / "platform/cloudflared/config.yaml"
GATEWAY_FILE = ROOT / "platform/gateway/gateway.yaml"
TARGET_KEY = ANNOTATION_PREFIX + "target"
PROXIED_KEY = ANNOTATION_PREFIX + "cloudflare-proxied"
UUID_RE = re.compile(r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$")
# The L4 names the tunnel 404s that are NONETHELESS published — as DNS-only A records from the
# `service` source, at their own LB VIP, because they are not HTTP and never touch the tunnel
# (operator decision 2026-11-08). Shrink-only, and must agree with the tunnel's 404 list (9c).
L4_DNS_ONLY_NAMES = {"unifi.${BASE_DOMAIN}", "crafty-mc.${BASE_DOMAIN}"}
# The operator's LAN/VPN-only HTTP names (2026-11-08). Pinned here as well as in the tunnel config, so
# deleting one from config.yaml cannot quietly also release it from (9c).
OPERATOR_LAN_ONLY_NAMES = {"plex.${BASE_DOMAIN}", "headlamp.${BASE_DOMAIN}"}
cf = next(iter(docs_of(CF_CONFIG)), None)
check(cf is not None and isinstance(cf.get("ingress"), list),
f"(9) {CF_CONFIG.relative_to(ROOT)} unreadable or has no `ingress` list — cannot prove what "
f"the tunnel refuses or which tunnel the Gateway must target")
cf = cf or {}
tunnel = str(cf.get("tunnel") or "")
check(bool(UUID_RE.match(tunnel)),
f"(9a) platform/cloudflared/config.yaml `tunnel:` is {tunnel!r}, not a literal tunnel UUID — "
f"the Gateway target is derived from it")
gw = next((d for d in docs_of(GATEWAY_FILE) if d.get("kind") == "Gateway"
and ident(d) == ("gateway", "shared")), None)
check(gw is not None, f"(9a) {GATEWAY_FILE.relative_to(ROOT)} holds no Gateway gateway/shared")
want_target = f"{tunnel}.cfargotunnel.com"
got_target = annotations_of(gw or {}).get(TARGET_KEY)
check(got_target == want_target,
f"(9a) Gateway gateway/shared `{TARGET_KEY}` is {got_target!r}; it must be EXACTLY "
f"{want_target!r} (the tunnel in platform/cloudflared/config.yaml). Any other value publishes "
f"every public name at the wrong place — a dead cfargotunnel.com name (Cloudflare 1016), or "
f"an origin address of ours.")
check("--cloudflare-proxied" in [str(a) for a in (extra if isinstance(extra, list) else [])]
or (isinstance(extra, dict) and str(extra.get("cloudflare-proxied", "")).lower() == "true"),
"(9a) external-dns HelmRelease extraArgs must include `--cloudflare-proxied`: a DNS-only "
"CNAME to <uuid>.cfargotunnel.com resolves to nothing routable, so every public name is dead")
for rel, d in dns_services:
if is_disowned(d):
continue
if HOSTNAME_KEY not in annotations_of(d):
continue
v = annotations_of(d).get(PROXIED_KEY)
check(str(v).lower() == "false",
f"(9b) {rel} Service {'/'.join(ident(d))} publishes via the `service` source but carries "
f"`{PROXIED_KEY}: {v!r}`, not \"false\". With --cloudflare-proxied global it would become "
f"an orange-clouded A record, which cannot carry L4 traffic and hides nothing (the LB VIP "
f"is RFC1918). Every `service`-source publisher must opt out explicitly.")
for rel, d in routes:
v = annotations_of(d).get(PROXIED_KEY)
check(v is None or str(v).lower() == "true",
f"(9b) {rel} HTTPRoute {'/'.join(ident(d))} sets `{PROXIED_KEY}: {v!r}`. A gateway-sourced "
f"record is a CNAME to the tunnel; DNS-only, it is dead. Disown the route instead "
f"(`{CONTROLLER_KEY}: none`) if the name should not be public.")
refused = {str(r.get("hostname")) for r in cf.get("ingress") or []
if isinstance(r, dict) and r.get("hostname") and r.get("service") == "http_status:404"}
check(len(refused) > 0, "(9c) the tunnel config 404s no hostname — parse contract broken (the "
"operator's LAN/VPN-only list is known to be non-empty)")
published_l4: set[str] = set()
for rel, d in routes:
hosts = {str(h) for h in (d.get("spec") or {}).get("hostnames") or []}
leaked = sorted(hosts & refused)
if is_disowned(d):
unrefused = sorted(hosts - refused)
check(not unrefused,
f"(9c) {rel} HTTPRoute {'/'.join(ident(d))} is disowned (no public record) but the "
f"tunnel does not 404 {unrefused}: add them to platform/cloudflared/config.yaml above "
f"the wildcard, or the two halves of \"LAN/VPN-only\" have drifted apart.")
continue
check(not leaked,
f"(9c) {rel} HTTPRoute {'/'.join(ident(d))} PUBLISHES {leaked}, which the tunnel maps to "
f"http_status:404 (LAN/VPN-only). The name would be a public proxied record that serves "
f"only 404s — disown the route (`{CONTROLLER_KEY}: none`) or adjudicate the name.")
for rel, d in dns_services:
if is_disowned(d):
continue
hosts = {h.strip() for h in str(annotations_of(d).get(HOSTNAME_KEY) or "").split(",")
if h.strip()}
for h in sorted(hosts & refused):
published_l4.add(h)
check(h in L4_DNS_ONLY_NAMES,
f"(9c) {rel} Service {'/'.join(ident(d))} publishes {h}, which the tunnel 404s, and it "
f"is not a declared L4 DNS-only name {sorted(L4_DNS_ONLY_NAMES)}")
check(L4_DNS_ONLY_NAMES <= refused,
f"(9c) L4 DNS-only names {sorted(L4_DNS_ONLY_NAMES - refused)} are not 404'd by the tunnel; a "
f"stray CNAME to the tunnel for them would hit the Gateway's catch-all instead of a 404")
check(published_l4 == L4_DNS_ONLY_NAMES,
f"(9c) L4_DNS_ONLY_NAMES is shrink-only: {sorted(L4_DNS_ONLY_NAMES - published_l4)} no longer "
f"published by any `service`-source publisher — delete them from the list")
check(OPERATOR_LAN_ONLY_NAMES <= refused,
f"(9d) operator LAN/VPN-only names {sorted(OPERATOR_LAN_ONLY_NAMES - refused)} are missing "
f"from the tunnel's http_status:404 rules (platform/cloudflared/config.yaml)")
for name in sorted(OPERATOR_LAN_ONLY_NAMES):
naming = [(rel, d) for rel, d in routes
if name in {str(h) for h in (d.get("spec") or {}).get("hostnames") or []}]
check(len(naming) > 0, f"(9d) no HTTPRoute names {name} — stale OPERATOR_LAN_ONLY_NAMES entry")
for rel, d in naming:
check(is_disowned(d),
f"(9d) {rel} HTTPRoute {'/'.join(ident(d))} names LAN/VPN-only {name} but is not "
f"disowned, so external-dns publishes it publicly (operator decision 2026-11-08)")
# ============================================================================================= # =============================================================================================
if FAILURES: if FAILURES:
for f in FAILURES: for f in FAILURES:
@@ -25,15 +25,19 @@
# #
# python3 mutations.py <path-to-extracted-reconciler.py> # python3 mutations.py <path-to-extracted-reconciler.py>
# #
# Runtime is ~20s (one subprocess per mutation, each running the full selftest # Runtime is ~11s serial (one subprocess per mutation, each running the full selftest
# against an in-process stub Keycloak; no network, no cluster). # against an in-process stub Keycloak; no network, no cluster). The selftests run in a
# bounded pool (env MUTATION_JOBS, default min(8, CPUs); 1 = serial); output order and
# verdicts are independent of the pool size.
# ============================================================================= # =============================================================================
from __future__ import annotations from __future__ import annotations
import os
import re import re
import subprocess import subprocess
import sys import sys
import tempfile import tempfile
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path from pathlib import Path
@@ -136,32 +140,84 @@ MUTATIONS: list[tuple[str, str, str, str]] = [
] ]
def mutation_jobs() -> int:
"""Selftests to run at once: env MUTATION_JOBS, else min(8, usable CPUs).
conformance/lib/run_l1_offline.py exports a smaller MUTATION_JOBS when it is itself
running several checks in parallel, so the nested pools do not oversubscribe the host.
MUTATION_JOBS=1 reproduces the old serial loop exactly.
"""
raw = os.environ.get("MUTATION_JOBS", "").strip()
if raw:
try:
n = int(raw)
except ValueError:
n = 0
if n < 1:
raise ValueError("MUTATION_JOBS must be a positive integer, got %r" % raw)
return n
try:
cpus = len(os.sched_getaffinity(0))
except (AttributeError, OSError):
cpus = os.cpu_count() or 1
return max(1, min(8, cpus))
def run_selftest(mutated_src: str) -> subprocess.CompletedProcess:
with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh:
fh.write(mutated_src)
path = fh.name
try:
return subprocess.run([sys.executable, path, "--selftest"],
capture_output=True, text=True, timeout=240,
stdin=subprocess.DEVNULL)
finally:
Path(path).unlink(missing_ok=True)
def main() -> int: def main() -> int:
if len(sys.argv) != 2: if len(sys.argv) != 2:
print("usage: mutations.py <extracted-reconciler.py>", file=sys.stderr) print("usage: mutations.py <extracted-reconciler.py>", file=sys.stderr)
return 2 return 2
src = Path(sys.argv[1]).read_text() src = Path(sys.argv[1]).read_text()
problems: list[str] = [] try:
killed = 0 jobs = mutation_jobs()
except ValueError as e:
print("mutations.py: %s" % e, file=sys.stderr)
return 2
# Pass 1, serial and cheap: every anchor is checked BEFORE anything runs, exactly as the
# serial loop did, and each mutation's source is built here. None = anchor failure.
planned: list[tuple[str, str, str | None, str | None]] = []
for name, why, old, new in MUTATIONS: for name, why, old, new in MUTATIONS:
old, new = dedent4(old), dedent4(new) old, new = dedent4(old), dedent4(new)
# An anchor that no longer matches is a FAILURE, not a skip: it means the code # An anchor that no longer matches is a FAILURE, not a skip: it means the code
# moved and this mutation silently stopped proving anything. # moved and this mutation silently stopped proving anything.
n = src.count(old) n = src.count(old)
if n != 1: if n != 1:
problems.append("%s: anchor matches %d times (expected exactly 1) -- the " planned.append((name, why,
"%s: anchor matches %d times (expected exactly 1) -- the "
"code moved and this mutation no longer proves anything. " "code moved and this mutation no longer proves anything. "
"Re-anchor it against the current source." % (name, n)) "Re-anchor it against the current source." % (name, n), None))
continue continue
with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh: planned.append((name, why, None, src.replace(old, new, 1)))
fh.write(src.replace(old, new, 1))
path = fh.name # Pass 2: the selftests, `jobs` at a time. Each child gets its own temp file, its own
# mkdtemp dirs and its own 127.0.0.1:0 stub port, so they share nothing. Results are
# consumed in MUTATIONS order, so every line below is printed in the same order, with the
# same verdict, as the serial loop produced -- whichever child finishes first.
problems: list[str] = []
killed = 0
pool = ThreadPoolExecutor(max_workers=jobs)
try: try:
proc = subprocess.run([sys.executable, path, "--selftest"], futures = [pool.submit(run_selftest, mutated) if mutated is not None else None
capture_output=True, text=True, timeout=240) for (_n, _w, _a, mutated) in planned]
finally: for (name, why, anchor_problem, _m), fut in zip(planned, futures):
Path(path).unlink(missing_ok=True) if anchor_problem is not None:
problems.append(anchor_problem)
continue
proc = fut.result()
if proc.returncode == 0: if proc.returncode == 0:
problems.append("%s SURVIVED -- --selftest still passed with this regression " problems.append("%s SURVIVED -- --selftest still passed with this regression "
"applied, so nothing in the suite actually checks it. The " "applied, so nothing in the suite actually checks it. The "
@@ -173,6 +229,10 @@ def main() -> int:
how = ("%s assertion(s) caught it" % m.group(1) if m how = ("%s assertion(s) caught it" % m.group(1) if m
else "the selftest aborted (caught, less cleanly)") else "the selftest aborted (caught, less cleanly)")
print(" killed %-42s %s" % (name, how)) print(" killed %-42s %s" % (name, how))
finally:
# A timeout (or any other exception) still propagates and fails the check, as it did
# serially; queued mutations are cancelled rather than run for nothing.
pool.shutdown(wait=True, cancel_futures=True)
print() print()
if problems: if problems:
@@ -40,15 +40,19 @@
# #
# python3 mutations.py <path-to-extracted-reconciler.py> # python3 mutations.py <path-to-extracted-reconciler.py>
# #
# Runtime is ~60s (one subprocess per mutation, each running the full selftest # Runtime is ~85s serial (one subprocess per mutation, each running the full selftest
# against an in-process stub Keycloak; no network, no cluster). # against an in-process stub Keycloak; no network, no cluster). The selftests run in a
# bounded pool (env MUTATION_JOBS, default min(8, CPUs); 1 = serial); output order and
# verdicts are independent of the pool size.
# ============================================================================= # =============================================================================
from __future__ import annotations from __future__ import annotations
import os
import re import re
import subprocess import subprocess
import sys import sys
import tempfile import tempfile
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path from pathlib import Path
# (name, the mistake being modelled, anchor, replacement) # (name, the mistake being modelled, anchor, replacement)
@@ -403,31 +407,83 @@ MUTATIONS: list[tuple[str, str, str, str]] = [
] ]
def mutation_jobs() -> int:
"""Selftests to run at once: env MUTATION_JOBS, else min(8, usable CPUs).
conformance/lib/run_l1_offline.py exports a smaller MUTATION_JOBS when it is itself
running several checks in parallel, so the nested pools do not oversubscribe the host.
MUTATION_JOBS=1 reproduces the old serial loop exactly.
"""
raw = os.environ.get("MUTATION_JOBS", "").strip()
if raw:
try:
n = int(raw)
except ValueError:
n = 0
if n < 1:
raise ValueError("MUTATION_JOBS must be a positive integer, got %r" % raw)
return n
try:
cpus = len(os.sched_getaffinity(0))
except (AttributeError, OSError):
cpus = os.cpu_count() or 1
return max(1, min(8, cpus))
def run_selftest(mutated_src: str) -> subprocess.CompletedProcess:
with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh:
fh.write(mutated_src)
path = fh.name
try:
return subprocess.run([sys.executable, path, "--selftest"],
capture_output=True, text=True, timeout=300,
stdin=subprocess.DEVNULL)
finally:
Path(path).unlink(missing_ok=True)
def main() -> int: def main() -> int:
if len(sys.argv) != 2: if len(sys.argv) != 2:
print("usage: mutations.py <extracted-reconciler.py>", file=sys.stderr) print("usage: mutations.py <extracted-reconciler.py>", file=sys.stderr)
return 2 return 2
src = Path(sys.argv[1]).read_text() src = Path(sys.argv[1]).read_text()
problems: list[str] = [] try:
killed = 0 jobs = mutation_jobs()
except ValueError as e:
print("mutations.py: %s" % e, file=sys.stderr)
return 2
# Pass 1, serial and cheap: every anchor is checked BEFORE anything runs, exactly as the
# serial loop did, and each mutation's source is built here. None = anchor failure.
planned: list[tuple[str, str, str | None, str | None]] = []
for name, why, old, new in MUTATIONS: for name, why, old, new in MUTATIONS:
# An anchor that no longer matches is a FAILURE, not a skip: it means the code # An anchor that no longer matches is a FAILURE, not a skip: it means the code
# moved and this mutation silently stopped proving anything. # moved and this mutation silently stopped proving anything.
n = src.count(old) n = src.count(old)
if n != 1: if n != 1:
problems.append("%s: anchor matches %d times (expected exactly 1) -- the " planned.append((name, why,
"%s: anchor matches %d times (expected exactly 1) -- the "
"code moved and this mutation no longer proves anything. " "code moved and this mutation no longer proves anything. "
"Re-anchor it against the current source." % (name, n)) "Re-anchor it against the current source." % (name, n), None))
continue continue
with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh: planned.append((name, why, None, src.replace(old, new, 1)))
fh.write(src.replace(old, new, 1))
path = fh.name # Pass 2: the selftests, `jobs` at a time. Each child gets its own temp file, its own
# mkdtemp dirs and its own 127.0.0.1:0 stub port, so they share nothing. Results are
# consumed in MUTATIONS order, so every line below is printed in the same order, with the
# same verdict, as the serial loop produced -- whichever child finishes first.
problems: list[str] = []
killed = 0
pool = ThreadPoolExecutor(max_workers=jobs)
try: try:
proc = subprocess.run([sys.executable, path, "--selftest"], futures = [pool.submit(run_selftest, mutated) if mutated is not None else None
capture_output=True, text=True, timeout=300) for (_n, _w, _a, mutated) in planned]
finally: for (name, why, anchor_problem, _m), fut in zip(planned, futures):
Path(path).unlink(missing_ok=True) if anchor_problem is not None:
problems.append(anchor_problem)
continue
proc = fut.result()
if proc.returncode == 0: if proc.returncode == 0:
problems.append("%s SURVIVED -- --selftest still passed with this regression " problems.append("%s SURVIVED -- --selftest still passed with this regression "
"applied, so nothing in the suite actually checks it. The " "applied, so nothing in the suite actually checks it. The "
@@ -439,6 +495,10 @@ def main() -> int:
how = ("%s assertion(s) caught it" % m.group(1) if m how = ("%s assertion(s) caught it" % m.group(1) if m
else "the selftest aborted (caught, less cleanly)") else "the selftest aborted (caught, less cleanly)")
print(" killed %-54s %s" % (name, how)) print(" killed %-54s %s" % (name, how))
finally:
# A timeout (or any other exception) still propagates and fails the check, as it did
# serially; queued mutations are cancelled rather than run for nothing.
pool.shutdown(wait=True, cancel_futures=True)
print() print()
if problems: if problems:
@@ -0,0 +1,39 @@
# =============================================================================
# L0 STATIC — PiHole's DNS frontends admit the LAN and the VPN, and nothing public
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# loadBalancerSourceRanges is the only thing keeping PiHole from being an open resolver, and it
# must also carry the WireGuard client subnet, because Cilium load-balances tunnel traffic on wg0
# before the iptables MASQUERADE (so the source it tests is still 10.90.0.x). check.sh's header
# states the assertions and what they do not prove.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library + PyYAML);
# conformance/run.sh --layer l0 runs it directly and this Chainsaw Test wraps the SAME script.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: pihole-dns-source-ranges
labels:
conformance.home.k3s/R-DNS-03: "true"
annotations:
conformance.home.k3s/requirements: "R-DNS-03"
conformance.home.k3s/layer: "L0"
# WHOLE-REPO SCOPE: ci/touched-requirements.sh maps apps/* to no R-DNS requirement, so a change
# to these Services would otherwise select nothing.
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Every PiHole LoadBalancer Service lists both the LAN and the WireGuard client subnet in
loadBalancerSourceRanges, and every range resolves to RFC1918, so VPN clients can resolve
and PiHole stays unreachable from the internet.
spec:
steps:
- name: assert-pihole-source-ranges
description: >-
Collect the PiHole LoadBalancer Services, resolve each source range through cluster-vars,
and assert both required ranges are present and every range is private.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,146 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — pihole-dns-source-ranges
# Serves: R-DNS-03 (LAN split-horizon DNS — spec docs/09).
#
# THE BUG CLASS THIS CLOSES. PiHole's LoadBalancer frontends are the only thing stopping it being
# an open resolver, and their `loadBalancerSourceRanges` is pulled in two opposite directions:
# * Too WIDE, and the resolver is open: a public range, or 0.0.0.0/0, in this list is the whole
# restriction gone, with nothing else in the path to catch it (the CiliumNetworkPolicy admits
# `world` on :53, because that is the only peer that can describe post-LoadBalancer traffic).
# * Too NARROW, and VPN DNS dies: wg0 is a Cilium kube-proxy-replacement device, so a tunnel
# client's query is load-balanced with its 10.90.0.x source intact, ahead of the iptables
# MASQUERADE. A LAN-only list is predicted to drop it as DROP_NOT_IN_SRC_RANGE, and the
# tunnel would still report a healthy handshake.
#
# GIVEN the repository as git tracks it,
# WHEN every type=LoadBalancer Service in namespace `pihole` is collected and each source range
# is resolved through cluster-vars (base value AND the prod overlay's replacement),
# THEN (1) the frontends exist and cover both udp/53 and tcp/53;
# (2) every frontend lists ${LAN_CIDR} and ${WG_CLIENT_CIDR};
# (3) every range resolves to a valid IPv4 CIDR inside RFC1918 — so no 0.0.0.0/0, no public
# range, and no unresolvable token that could hide one;
# AND (4) the check is NON-VACUOUS: a real checkout and a floor on the assertion count.
#
# WHAT THIS DOES NOT PROVE: that a query over the tunnel is actually answered. That needs a
# handshake from a real client; see platform/vpn-ingress/README.md.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
import ipaddress, re, subprocess, sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
FAILURES: list[str] = []
CHECKS = 0
REQUIRED_TOKENS = ("LAN_CIDR", "WG_CLIENT_CIDR")
RFC1918 = [ipaddress.ip_network(n) for n in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16")]
TOKEN = re.compile(r"^\$\{([A-Z0-9_]+)\}$")
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def docs_of(rel: str):
try:
return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace"))
if isinstance(d, dict)]
except Exception:
return []
FILES = [p for p in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
capture_output=True, text=True, check=True).stdout.split("\0")
if p]
YAML_FILES = [f for f in FILES if re.search(r"\.ya?ml$", f)]
check(len(YAML_FILES) > 50, f"only {len(YAML_FILES)} YAML files seen — not a real checkout")
# ------------------------------------------------- every value a ${TOKEN} can take on this tree
values: dict[str, set[str]] = {}
for d in docs_of("clusters/base/cluster-vars.yaml"):
if d.get("kind") == "ConfigMap":
for k, v in (d.get("data") or {}).items():
values.setdefault(k, set()).add(str(v))
for d in docs_of("clusters/prod/kustomization.yaml"):
for p in d.get("patches") or []:
ops = p.get("patch") if isinstance(p, dict) else None
ops = yaml.safe_load(ops) if isinstance(ops, str) else ops
for op in ops if isinstance(ops, list) else []:
m = re.fullmatch(r"/data/([A-Z0-9_]+)", str(op.get("path", "")))
if m and op.get("op") in ("add", "replace"):
values.setdefault(m.group(1), set()).add(str(op.get("value")))
for tok in REQUIRED_TOKENS:
check(values.get(tok), f"clusters/base/cluster-vars.yaml defines no {tok} (R-DNS-03)")
def assert_private(where: str, cidr: str):
try:
net = ipaddress.ip_network(cidr, strict=False)
except ValueError:
check(False, f"{where}: {cidr!r} is not a valid CIDR (R-DNS-03)")
return
check(net.version == 4 and any(net.subnet_of(r) for r in RFC1918),
f"{where}: {cidr} is not inside RFC1918, so it opens PiHole beyond the LAN and the VPN "
"(R-DNS-03)")
# ---------------------------------------------------------------------------- the frontends
frontends = []
for rel in YAML_FILES:
for d in docs_of(rel):
md, spec = d.get("metadata") or {}, d.get("spec") or {}
if d.get("kind") == "Service" and md.get("namespace") == "pihole" \
and spec.get("type") == "LoadBalancer":
frontends.append((rel, md.get("name"), spec))
protocols = {(p.get("protocol", "TCP"), p.get("port"))
for _, _, spec in frontends for p in spec.get("ports") or []}
for want in (("UDP", 53), ("TCP", 53)):
check(want in protocols, f"no PiHole LoadBalancer Service serves {want[0]}/{want[1]} — this "
"check's corpus has moved and must be re-derived (R-DNS-03)")
for rel, name, spec in frontends:
where = f"{rel}: Service/{name}"
ranges = spec.get("loadBalancerSourceRanges") or []
check(ranges, f"{where} has no loadBalancerSourceRanges, so it answers anyone who can reach "
"the VIP (R-DNS-03)")
tokens = {m.group(1) for m in (TOKEN.match(str(r)) for r in ranges) if m}
for tok in REQUIRED_TOKENS:
check(tok in tokens, f"{where} does not list ${{{tok}}} in loadBalancerSourceRanges "
"(R-DNS-03)")
for r in ranges:
m = TOKEN.match(str(r))
if not m:
assert_private(where, str(r))
continue
resolved = values.get(m.group(1))
check(resolved, f"{where}: {r} is not defined in cluster-vars, so its value cannot be "
"shown to be private (R-DNS-03)")
for v in sorted(resolved or []):
assert_private(f"{where} ({r})", v)
check(CHECKS >= 15, f"only {CHECKS} assertions ran — this check has gone vacuous (R-DNS-03)")
if FAILURES:
for f in FAILURES:
print(f"check FAIL: {f}", file=sys.stderr)
raise SystemExit(1)
print(f"check OK: pihole-dns-source-ranges — {CHECKS} assertions over "
f"{len(frontends)} PiHole LoadBalancer Service(s)")
PY
@@ -0,0 +1,45 @@
# =============================================================================
# L0 STATIC — PiHole answers an explicit LAN name list, and no LAN-only name is left out of it
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# Since PR #156 every public name is a proxied CNAME to the Cloudflare Tunnel and works from the
# LAN, so PiHole's zone wildcard was replaced by an explicit list (catalog/pihole.yaml). That makes
# the list load-bearing: a route disowned from public DNS and absent from it resolves nowhere. This
# test derives the required names from the tree (every `controller: none` HTTPRoute, every hostname
# the tunnel 404s, the L4 Services' own VIPs) and asserts the list covers them, carries no
# zone-wide entry, and pins gitea and nextcloud to the Gateway VIP. It proves itself non-vacuous by
# mutation. Rationale and the honest NOT-proven list: check.sh.
#
# WHY SCOPE, NOT REQUIREMENTS: the property spans catalog/, apps/, platform/ and runbooks/; a new
# internal service in any of them can break it, and ci/touched-requirements.sh maps apps/* to no
# requirement at all, so a requirement-scoped selection would skip it on precisely those changes.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: pihole-lan-names
labels:
conformance.home.k3s/R-DNS-03: "true"
annotations:
conformance.home.k3s/requirements: "R-DNS-03"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "true"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
PiHole answers an explicit list of LAN names rather than the whole zone; every HTTPRoute
disowned from public DNS has an entry to the Gateway VIP; every hostname the Cloudflare
Tunnel 404s has an entry, with the L4 names on their own VIPs; and gitea and nextcloud stay
on the LAN path, off Cloudflare's request-size and timeout limits.
spec:
steps:
- name: assert-pihole-explicit-lan-names
description: >-
Parse FTLCONF_misc_dnsmasq_lines from catalog/pihole.yaml and the rendered
apps/wave-0/pihole.yaml, derive the required names from every `controller: none` HTTPRoute,
the http_status:404 hostnames in platform/cloudflared/config.yaml and the L4 Services'
lbipam annotations, assert the list covers them and contains no zone-wide entry, and run
the evaluator on mutated lists to prove it is not vacuous.
try:
- script:
content: bash ./check.sh
+269
View File
@@ -0,0 +1,269 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — pihole-lan-names
# Serves: R-DNS-03 (LAN split-horizon answers), and decisions/2026-11-09-pihole-explicit-lan-names.md.
#
# WHY THIS EXISTS. Since PR #156 every PUBLIC name is a proxied CNAME to the Cloudflare Tunnel and
# works from the LAN, so PiHole no longer answers the whole zone with a wildcard: it answers an
# EXPLICIT list (catalog/pihole.yaml, FTLCONF_misc_dnsmasq_lines) and forwards everything else to
# the public resolvers. That inverts the old failure mode. With the wildcard, a forgotten name still
# resolved; now a forgotten LAN-ONLY name — a route disowned from public DNS, which the tunnel also
# 404s — resolves NOWHERE, and the app is silently unreachable from every client. Nothing in the
# cluster turns red when that happens. So the list is derived from the tree and asserted here.
#
# GIVEN the repository as authored,
# THEN (0) the dnsmasq value parses (only `local-ttl=` and `address=/<domain>/<${VAR}>` entries,
# every VAR is a cluster-var), the catalog value and the rendered apps/wave-0/pihole.yaml
# value are identical, and the inventories below are non-empty (anti-vacuous-pass);
# (a) NO entry covers the whole zone: no `address=/${BASE_DOMAIN}/`, no `address=/#/`, and no
# entry for any parent of the zone (dnsmasq matches every subdomain of an entry);
# (b) every HTTPRoute hostname disowned from public DNS
# (`external-dns.alpha.kubernetes.io/controller: none`, anywhere in apps/ platform/
# runbooks/) has an explicit entry to ${GATEWAY_VIP} — a disowned route with no
# `hostnames:` is itself a FAIL, because it cannot be checked;
# (c) every hostname platform/cloudflared/config.yaml maps to http_status:404 has an entry;
# the L4 DNS-only names (derived: Services carrying both
# `external-dns.alpha.kubernetes.io/hostname` and `lbipam.cilium.io/ips` — unifi,
# crafty-mc) must map to THAT Service's VIP, not the Gateway;
# (d) gitea and nextcloud are pinned to ${GATEWAY_VIP} (Cloudflare's 100 MB body / 100 s
# limits break git push/LFS and large uploads);
# AND (e) the evaluator is not vacuous: it is run against mutated copies of the real entry list
# (zone wildcard added; a disowned host dropped; a 404'd host dropped; gitea repointed;
# unifi repointed to the Gateway) and must flag each one.
#
# NOT proven (static): that the running PiHole pods carry this value (FTL applies FTLCONF_* at
# start; the env change rolls the StatefulSet), that public resolution of the unlisted names
# reaches Cloudflare, or the substituted IPs. Runtime check: `dig @${PIHOLE_DNS_VIP} <name>`.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
exec python3 - "$ROOT" <<'PY'
import copy
import re
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("CHECK FAIL (pihole-lan-names): PyYAML is required", file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
FAILURES, CHECKS = [], 0
NAME = "pihole-lan-names"
ZONE = "${BASE_DOMAIN}"
GW = "${GATEWAY_VIP}"
DISOWN = "external-dns.alpha.kubernetes.io/controller"
EDNS_HOST = "external-dns.alpha.kubernetes.io/hostname"
LB_IPS = "lbipam.cilium.io/ips"
# Public, but must bypass Cloudflare's 100 MB request body / 100 s response limits.
LAN_PINNED = ("gitea", "nextcloud")
# Trees that hold applied manifests. conformance/ fixtures and .claude/ worktrees are not authored
# surface (CLAUDE.md §0); decisions/ is prose.
SCAN = ("apps", "platform", "runbooks")
def check(cond, desc):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(desc)
def docs(path):
try:
return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)]
except yaml.YAMLError:
return []
def dnsmasq_value(objs):
"""The FTLCONF_misc_dnsmasq_lines value from a list of docs (a ServiceDefinition or manifests)."""
found = []
for d in objs:
envs = list(d.get("env") or [])
tmpl = (((d.get("spec") or {}).get("template") or {}).get("spec") or {})
for c in tmpl.get("containers") or []:
envs += c.get("env") or []
found += [e.get("value") for e in envs if e.get("name") == "FTLCONF_misc_dnsmasq_lines"]
return found
def die(msg):
print(f"CHECK FAIL ({NAME}): {msg}", file=sys.stderr)
raise SystemExit(1)
CATALOG = ROOT / "catalog" / "pihole.yaml"
RENDERED = ROOT / "apps" / "wave-0" / "pihole.yaml"
CF = ROOT / "platform" / "cloudflared" / "config.yaml"
for p in (CATALOG, RENDERED, CF):
if not p.is_file():
die(f"{p.relative_to(ROOT)} is absent — the surface this check guards moved; re-point or "
"delete this check in the SAME change")
cat_vals = dnsmasq_value(docs(CATALOG))
ren_vals = dnsmasq_value(docs(RENDERED))
check(len(cat_vals) == 1, f"(0) catalog/pihole.yaml declares FTLCONF_misc_dnsmasq_lines exactly "
f"once (found {len(cat_vals)})")
check(len(ren_vals) == 1, f"(0) apps/wave-0/pihole.yaml carries FTLCONF_misc_dnsmasq_lines exactly "
f"once (found {len(ren_vals)})")
if not cat_vals:
die("no dnsmasq value to check")
check(cat_vals[:1] == ren_vals[:1],
"(0) the rendered apps/wave-0/pihole.yaml dnsmasq value differs from catalog/pihole.yaml — "
"re-render pihole (catalog/render.py --no-image-automation catalog/pihole.yaml)")
cv = next(d for d in docs(ROOT / "clusters" / "base" / "cluster-vars.yaml")
if d.get("kind") == "ConfigMap")
CLUSTER_VARS = set(cv["data"])
BASE_DOMAIN_VALUE = cv["data"]["BASE_DOMAIN"]
ENTRY = re.compile(r"^address=/([^/]+)/(\$\{[A-Z0-9_]+\})$")
def parse(value):
"""Return (entries {domain: target}, problems)."""
entries, problems = {}, []
for raw in value.split(";"):
line = raw.strip()
if raw != line:
problems.append(f"entry {raw!r} has surrounding whitespace (FTL writes it verbatim)")
if not line:
problems.append("empty entry (doubled or trailing `;`)")
continue
if re.fullmatch(r"local-ttl=\d+", line):
continue
m = ENTRY.match(line)
if not m:
problems.append(f"entry {line!r} is not `local-ttl=N` or `address=/<domain>/${{VAR}}` — "
f"this check cannot reason about it")
continue
dom, tgt = m.groups()
if tgt[2:-1] not in CLUSTER_VARS:
problems.append(f"{dom} -> {tgt}: not a key of clusters/base/cluster-vars.yaml")
if dom in entries and entries[dom] != tgt:
problems.append(f"{dom} is listed twice with different targets")
entries[dom] = tgt
return entries, problems
entries, problems = parse(cat_vals[0])
for pr in problems:
check(False, f"(0) {pr}")
# ── inventories, derived from the tree ──────────────────────────────────────────────────────────
disowned, unnamed = set(), []
l4 = {}
for top in SCAN:
for f in sorted((ROOT / top).rglob("*.yaml")):
for d in docs(f):
meta = d.get("metadata") or {}
ann = meta.get("annotations") or {}
if d.get("kind") == "HTTPRoute" and str(ann.get(DISOWN, "")).strip() == "none":
hosts = (d.get("spec") or {}).get("hostnames") or []
if not hosts:
unnamed.append(f"{f.relative_to(ROOT)}:{meta.get('name')}")
disowned.update(hosts)
if d.get("kind") == "Service" and EDNS_HOST in ann and LB_IPS in ann:
for h in str(ann[EDNS_HOST]).split(","):
l4[h.strip()] = str(ann[LB_IPS]).strip()
for u in unnamed:
check(False, f"(b) disowned HTTPRoute {u} declares no `hostnames:` — it inherits a listener "
f"hostname this check cannot see; name it explicitly")
check(len(disowned) >= 5, f"(0) only {len(disowned)} disowned HTTPRoute hostnames found — the scan "
f"is vacuous (the disown annotation was renamed?)")
check(len(l4) >= 2, f"(0) only {len(l4)} L4 Services publishing their own hostname found (expected "
f"unifi and crafty-mc) — the scan is vacuous")
cf = yaml.safe_load(CF.read_text())
cf_404 = {r["hostname"] for r in cf.get("ingress") or []
if r.get("hostname") and r.get("service") == "http_status:404"}
check(len(cf_404) >= 5, f"(0) only {len(cf_404)} http_status:404 hostnames in the cloudflared "
f"config — the scan is vacuous")
def covers_zone(dom):
"""True if an entry for `dom` answers every name in the zone (dom is the zone or a parent)."""
if dom in ("#", "", "."):
return True
if dom == ZONE:
return True
# A literal parent (e.g. `keithjackson.dev`, `dev`) of the zone's substituted value.
real = dom.replace(ZONE, BASE_DOMAIN_VALUE)
return real == BASE_DOMAIN_VALUE or BASE_DOMAIN_VALUE.endswith("." + real)
def evaluate(entries):
bad = []
# (a)
for dom in entries:
if covers_zone(dom):
bad.append(f"(a) `address=/{dom}/` answers EVERY name in the zone — the wildcard is gone "
f"on purpose (public names must resolve to Cloudflare from the LAN)")
# (b)
for h in sorted(disowned):
if entries.get(h) != GW:
bad.append(f"(b) {h} is disowned from public DNS but PiHole maps it to "
f"{entries.get(h, 'NOTHING')} — expected {GW}; without it the app is "
f"unreachable everywhere")
# (c)
for h in sorted(cf_404):
if h in l4:
if entries.get(h) != l4[h]:
bad.append(f"(c) {h} is an L4 DNS-only name; PiHole maps it to "
f"{entries.get(h, 'NOTHING')}, expected its own VIP {l4[h]}")
elif h not in entries:
bad.append(f"(c) the tunnel 404s {h} but PiHole has no entry for it — it would resolve "
f"only publicly, and the tunnel refuses it")
# (d)
for n in LAN_PINNED:
h = f"{n}.{ZONE}"
if entries.get(h) != GW:
bad.append(f"(d) {h} maps to {entries.get(h, 'NOTHING (public/Cloudflare)')}; it must be "
f"pinned to {GW} — Cloudflare caps request bodies at 100 MB and responses at "
f"100 s")
return bad
for v in evaluate(entries):
check(False, v)
# (e) mutation self-test
mutants = {}
m = dict(entries); m[ZONE] = GW
mutants["zone wildcard added"] = m
m = dict(entries); m[BASE_DOMAIN_VALUE.split(".", 1)[1]] = GW
mutants["parent-of-zone wildcard added"] = m
m = dict(entries); m.pop(sorted(disowned)[0], None)
mutants[f"disowned host {sorted(disowned)[0]} dropped"] = m
pure_404 = sorted(h for h in cf_404 if h not in l4 and h not in disowned)
if pure_404:
m = dict(entries); m.pop(pure_404[0], None)
mutants[f"404'd host {pure_404[0]} dropped"] = m
m = dict(entries); m.pop(f"gitea.{ZONE}", None)
mutants["gitea unpinned"] = m
m = dict(entries); m[f"nextcloud.{ZONE}"] = "${UNIFI_L4_VIP}"
mutants["nextcloud repointed"] = m
if l4:
h = sorted(l4)[0]
m = dict(entries); m[h] = GW
mutants[f"{h} repointed to the Gateway"] = m
for name, mut in mutants.items():
check(evaluate(mut), f"(e) self-test: the evaluator did NOT flag the mutant '{name}' — vacuous")
check(len(mutants) >= 5, f"(e) only {len(mutants)} mutants built — the self-test degenerated")
_, pr = parse("local-ttl=60;address=/x.${BASE_DOMAIN}/${NOT_A_CLUSTER_VAR}; address=/y/${GATEWAY_VIP}")
check(len(pr) >= 2, "(e) self-test: the parser accepted an unknown VAR or a whitespace-padded entry")
if CHECKS < 15:
FAILURES.append(f"only {CHECKS} assertions ran — the check degenerated")
if FAILURES:
print(f"CHECK FAIL ({NAME}): {len(FAILURES)} of {CHECKS} assertions failed", file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
print(f"PASS ({NAME}): {CHECKS} assertions; {len(entries)} PiHole names, {len(disowned)} disowned "
f"routes, {len(cf_404)} tunnel-404 names, {len(l4)} L4 names")
PY
@@ -0,0 +1,28 @@
# =============================================================================
# L0 STATIC — accepted CVE risk has a reason and a bounded expiry (R-VULN-08, docs/17 "Risk acceptance").
# Offline, no cluster.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: vuln-acceptances-schema
labels:
conformance.home.k3s/R-VULN-08: "true"
annotations:
conformance.home.k3s/requirements: "R-VULN-08"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
platform/vuln-scanner/acceptances.yaml rejects any entry without a reason or expiry, with a
malformed expiry, or expiring more than 180 days out. Proven with mutated negative fixtures.
spec:
steps:
- name: assert-acceptance-schema
description: >-
Validate acceptances.yaml against the docs/17 shape, then run the validator on mutated
negative fixtures to prove it can fail.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,105 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — vuln-acceptances-schema
# Serves: R-VULN-08 (docs/17 "Risk acceptance": "An entry with no reason or expiry is rejected by the gate").
#
# GIVEN platform/vuln-scanner/acceptances.yaml,
# THEN (1) it is a YAML list;
# (2) every entry has a non-empty id, a non-empty reason, an expires of the form YYYY-MM-DD
# at most 180 days from today, and only the permitted keys (image is optional);
# AND (3) the validator rejects mutated negative fixtures for each rule and accepts a good entry.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)"
exec python3 - "$ROOT" <<'PY'
import datetime as dt, re, sys
from pathlib import Path
import yaml
ROOT = Path(sys.argv[1])
NAME = "vuln-acceptances-schema"
FAILURES, CHECKS = [], 0
TODAY = dt.datetime.now(dt.timezone.utc).date()
MAX_DAYS = 180
ALLOWED_KEYS = {"id", "image", "reason", "expires"}
DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$")
def check(cond, desc):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(desc)
def validate(entries):
errs = []
if not isinstance(entries, list):
return ["acceptances must be a YAML list"]
for i, e in enumerate(entries):
if not isinstance(e, dict):
errs.append(f"entry {i} is not a mapping")
continue
if set(e) - ALLOWED_KEYS:
errs.append(f"entry {i} has unknown keys {sorted(set(e) - ALLOWED_KEYS)}")
for field in ("id", "reason"):
if not isinstance(e.get(field), str) or not e[field].strip():
errs.append(f"entry {i} has no non-empty {field}")
if "image" in e and (not isinstance(e["image"], str) or not e["image"].strip()):
errs.append(f"entry {i} has an empty image")
exp = e.get("expires")
if isinstance(exp, dt.date) and not isinstance(exp, dt.datetime):
exp_s = exp.isoformat()
elif isinstance(exp, str):
exp_s = exp
else:
errs.append(f"entry {i} has no expires")
continue
if not DATE_RE.match(exp_s):
errs.append(f"entry {i} expires is malformed: {exp_s!r}")
continue
try:
d = dt.date.fromisoformat(exp_s)
except ValueError:
errs.append(f"entry {i} expires is not a real date: {exp_s!r}")
continue
if (d - TODAY).days > MAX_DAYS:
errs.append(f"entry {i} expires {exp_s}, more than {MAX_DAYS} days out")
return errs
path = ROOT / "platform" / "vuln-scanner" / "acceptances.yaml"
if not path.is_file():
print(f"CHECK FAIL ({NAME}): platform/vuln-scanner/acceptances.yaml is absent", file=sys.stderr)
raise SystemExit(1)
entries = yaml.safe_load(path.read_text())
check(entries == [] or (isinstance(entries, list) and entries), "(1) acceptances.yaml parsed to a list")
errs = validate(entries)
check(not errs, f"(2) acceptances.yaml is valid: {errs}")
soon = (TODAY + dt.timedelta(days=30)).isoformat()
far = (TODAY + dt.timedelta(days=200)).isoformat()
good = {"id": "CVE-2026-00000", "image": "docker.io/library/x", "reason": "not reachable", "expires": soon}
check(validate([good]) == [], "(3) a well-formed entry is accepted")
check(validate([{k: v for k, v in good.items() if k != "image"}]) == [], "(3) image is optional (platform-wide)")
for name, entry in {
"no reason": {k: v for k, v in good.items() if k != "reason"},
"empty reason": dict(good, reason=""),
"no expires": {k: v for k, v in good.items() if k != "expires"},
"no id": {k: v for k, v in good.items() if k != "id"},
"malformed expiry": dict(good, expires="next-tuesday"),
"impossible date": dict(good, expires="2027-13-45"),
"past 180 days": dict(good, expires=far),
"unknown key": dict(good, owner="me"),
}.items():
check(bool(validate([entry])), f"(3) the validator accepted a bad entry: {name}")
check(bool(validate({"not": "a list"})), "(3) the validator accepted a non-list")
if FAILURES:
print(f"CHECK FAIL ({NAME}): {len(FAILURES)} of {CHECKS} assertions failed", file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
print(f"PASS ({NAME}): {CHECKS} assertions")
PY
@@ -0,0 +1,29 @@
# =============================================================================
# L0 STATIC — the five scanner alerts exist and are unit-tested, including the absent-metric cases
# (R-VULN-06, docs/17 "Acceptance gates"). Offline, no cluster.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: vuln-alerts-unit-tested
labels:
conformance.home.k3s/R-VULN-06: "true"
annotations:
conformance.home.k3s/requirements: "R-VULN-06"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
VulnScanStale, VulnDatabaseStale, VulnCoverageGap, VulnCriticalFixAvailable and
VulnTriageBotSilent exist, each promtool-tested firing, and the two absent-metric rules fire
in a test with no such series. Proven by mutation.
spec:
steps:
- name: assert-alert-unit-tests
description: >-
Parse alerts.yaml and alerts.promtool-test.yaml, assert every alert and the two absent cases
are pinned, run the harness when promtool is present, and run the evaluator on mutated copies.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,153 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — vuln-alerts-unit-tested
# Serves: R-VULN-06 (docs/17 "Acceptance gates": alerts unit-tested, including the absent-metric cases).
#
# GIVEN platform/vuln-scanner/alerts.yaml and alerts.promtool-test.yaml,
# THEN (1) the rules are exactly the five contract alerts, with the two absent()-bearing rules;
# (2) each alert is asserted FIRING in a promtool test;
# (3) VulnScanStale and VulnTriageBotSilent each fire in a test with NO input series for their
# own metric (the absent case) and in one where it vanishes (re-fire);
# (4) each alert carries severity and a runbook_url;
# (5) when promtool is present (the toolbox), the actual harness passes on this file;
# AND (6) the evaluator is run on mutated copies (the absent arm deleted, a firing assertion
# removed, an alert dropped) and must flag each (non-vacuity).
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)"
exec python3 - "$ROOT" <<'PY'
import copy, re, shutil, subprocess, sys
from pathlib import Path
import yaml
ROOT = Path(sys.argv[1])
NAME = "vuln-alerts-unit-tested"
FAILURES, CHECKS = [], 0
ALERTS = {"VulnScanStale", "VulnDatabaseStale", "VulnCoverageGap", "VulnCriticalFixAvailable", "VulnTriageBotSilent"}
ABSENT_ALERTS = {
"VulnScanStale": "vuln_scanner_last_complete_run_timestamp_seconds",
"VulnTriageBotSilent": "vuln_scanner_triage_heartbeat_timestamp_seconds",
}
def check(cond, desc):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(desc)
COMP = ROOT / "platform" / "vuln-scanner"
for f in ("alerts.yaml", "alerts.promtool-test.yaml"):
if not (COMP / f).is_file():
print(f"CHECK FAIL ({NAME}): platform/vuln-scanner/{f} is absent", file=sys.stderr)
raise SystemExit(1)
rules = yaml.safe_load((COMP / "alerts.yaml").read_text())
tests = yaml.safe_load((COMP / "alerts.promtool-test.yaml").read_text())
def alert_rules(doc):
return {r["alert"]: r for g in doc["spec"]["groups"] for r in g["rules"]}
def eval_alerts(doc):
bad = []
got = alert_rules(doc)
if set(got) != ALERTS:
bad.append(f"alerts are {sorted(got)}, expected {sorted(ALERTS)}")
for name, metric in ABSENT_ALERTS.items():
if name in got and f"absent({metric})" not in got[name]["expr"].replace(" ", ""):
bad.append(f"{name} lacks the absent({metric}) arm")
for name, r in got.items():
if not (r.get("labels") or {}).get("severity"):
bad.append(f"{name} has no severity label")
if "/runbooks/" not in ((r.get("annotations") or {}).get("runbook_url") or ""):
bad.append(f"{name} has no runbook_url")
return bad
def firing_assertions(test):
"""(alertname -> assertion list) for the firing expectations of one promtool test."""
out = {}
for pe in test.get("promql_expr_test") or []:
expr, samples = pe.get("expr", ""), pe.get("exp_samples") or []
m = re.search(r'alertname="(Vuln\w+)"', expr)
if m and samples and 'alertstate="firing"' in (expr + " " + " ".join(s.get("labels", "") for s in samples)):
out.setdefault(m.group(1), []).append(pe)
return out
def series_for(test):
return [s.get("series", "") for s in test.get("input_series") or []]
def eval_tests(doc):
bad = []
all_tests = doc.get("tests") or []
fired = set()
for t in all_tests:
fired |= set(firing_assertions(t))
missing = ALERTS - fired
if missing:
bad.append(f"no firing promtool assertion for {sorted(missing)}")
for alert, metric in ABSENT_ALERTS.items():
absent_case = any(
alert in firing_assertions(t) and not any(s.startswith(metric) for s in series_for(t))
for t in all_tests)
if not absent_case:
bad.append(f"{alert} has no firing test with NO {metric} series (the absent case)")
vanish_case = any(
alert in firing_assertions(t) and any(s.startswith(metric) and "stale" in str(i.get("values", ""))
for i in t.get("input_series", []) for s in [i.get("series", "")])
for t in all_tests)
if not vanish_case:
bad.append(f"{alert} has no test where its metric goes stale (the re-fire case)")
return bad
for v in eval_alerts(rules):
check(False, f"(1,4) {v}")
for v in eval_tests(tests):
check(False, f"(2,3) {v}")
check(len(alert_rules(rules)) == 5, "(1) five alert rules parsed")
# (5) the real harness, only when the rule engine is present.
if shutil.which("promtool"):
rc = subprocess.run(["bash", str(ROOT / "platform/monitoring/tests/run.sh"),
"platform/vuln-scanner/alerts.promtool-test.yaml"],
cwd=str(ROOT), capture_output=True, text=True)
check(rc.returncode == 0, f"(5) the promtool harness failed: rc={rc.returncode}\n{rc.stdout[-800:]}{rc.stderr[-800:]}")
# (6) non-vacuity
m = copy.deepcopy(rules)
for g in m["spec"]["groups"]:
for r in g["rules"]:
if r["alert"] == "VulnTriageBotSilent":
r["expr"] = r["expr"].split(" or ")[0]
check(bool(eval_alerts(m)), "(6) evaluator missed: absent arm deleted")
m = copy.deepcopy(rules)
m["spec"]["groups"][0]["rules"].pop(1)
check(bool(eval_alerts(m)), "(6) evaluator missed: alert dropped")
m = copy.deepcopy(rules)
for g in m["spec"]["groups"]:
for r in g["rules"]:
r["annotations"].pop("runbook_url", None)
check(bool(eval_alerts(m)), "(6) evaluator missed: runbook_url removed")
m = copy.deepcopy(tests)
for t in m["tests"]:
t["promql_expr_test"] = [p for p in t["promql_expr_test"]
if 'VulnCoverageGap' not in p.get("expr", "")]
check(bool(eval_tests(m)), "(6) evaluator missed: firing assertion removed")
m = copy.deepcopy(tests)
m["tests"] = [t for t in m["tests"] if "ABSENT" not in t.get("name", "")]
check(bool(eval_tests(m)), "(6) evaluator missed: absent-case tests removed")
if FAILURES:
print(f"CHECK FAIL ({NAME}): {len(FAILURES)} of {CHECKS} assertions failed", file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
print(f"PASS ({NAME}): {CHECKS} assertions")
PY
@@ -0,0 +1,28 @@
# =============================================================================
# L0 STATIC — Prometheus receives aggregates only (R-VULN-06, docs/17 "Alerting").
# Offline, no cluster. Depends on platform/vuln-scanner/src/metrics.py (the SCANNER's file).
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: vuln-metrics-no-per-cve-label
labels:
conformance.home.k3s/R-VULN-06: "true"
annotations:
conformance.home.k3s/requirements: "R-VULN-06"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
No vuln_scanner_* metric or alert rule carries a label other than state, severity or fixable,
so Prometheus can never hold a per-CVE, per-image or per-workload series. Proven by mutation.
spec:
steps:
- name: assert-metric-labels-are-aggregates
description: >-
Extract every vuln_scanner_* metric label from metrics.py and alerts.yaml, assert the
allowlist, and run the extractor on mutated text to prove it can fail.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,105 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — vuln-metrics-no-per-cve-label
# Serves: R-VULN-06 ("Prometheus receives aggregates only ... MUST NOT receive a per-CVE series").
#
# GIVEN platform/vuln-scanner/src/metrics.py and platform/vuln-scanner/alerts.yaml,
# THEN (1) every label name attached to a vuln_scanner_* metric is in the allowlist
# (state, severity, fixable);
# (2) metrics.py exposes every metric name from the contract;
# (3) alerts.yaml selects vuln_scanner_* metrics with no label matcher outside the allowlist;
# AND (4) the extractor is proven on mutated text: a `cve=` label, a `{image=...}` selector and a
# per-workload label are each flagged, and label-free text is not.
# A missing metrics.py is a FAIL: this is a whole-repo-scoped check and the file is part of the same
# delivery (the check has no skip path).
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)"
exec python3 - "$ROOT" <<'PY'
import re, sys
from pathlib import Path
import yaml
ROOT = Path(sys.argv[1])
NAME = "vuln-metrics-no-per-cve-label"
FAILURES, CHECKS = [], 0
ALLOWED = {"state", "severity", "fixable"}
METRICS = (
"vuln_scanner_report_generated_timestamp_seconds",
"vuln_scanner_last_complete_run_timestamp_seconds",
"vuln_scanner_database_updated_timestamp_seconds",
"vuln_scanner_targets",
"vuln_scanner_findings",
"vuln_scanner_failed_targets_over_24h",
"vuln_scanner_critical_fix_overdue",
"vuln_scanner_triage_heartbeat_timestamp_seconds",
)
# Label keys that could only carry per-CVE or per-target identity, anywhere a metric could be
# written. Catches label construction the name{...} scan cannot see (e.g. "cve_id": cve_id).
IDENTITY_LABEL = re.compile(r'''["'](cve|cve_id|advisory|package|pkg|image|digest|workload|namespace|pod|container|target|target_id)["']\s*[:=]''')
def check(cond, desc):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(desc)
def label_names(text):
"""Every label key in `vuln_scanner_x{key="value",...}` occurrences (Python or PromQL text),
tolerating f-strings' doubled braces and multi-line calls."""
found = set()
for m in re.finditer(r"vuln_scanner_[a-z0-9_]+\{", text):
tail = text[m.end(): m.end() + 200]
close = tail.find("}\n")
chunk = tail[: close if close != -1 else None].split("\n", 1)[0]
for key in re.findall(r"([a-zA-Z_][a-zA-Z0-9_]*)\s*=\s*\\?[\"'{]", chunk):
found.add(key)
return found
METRICS_PY = ROOT / "platform" / "vuln-scanner" / "src" / "metrics.py"
ALERTS = ROOT / "platform" / "vuln-scanner" / "alerts.yaml"
if not METRICS_PY.is_file():
print(f"CHECK FAIL ({NAME}): platform/vuln-scanner/src/metrics.py is absent (the SCANNER's "
f"deliverable); the aggregate-only property cannot be asserted without it", file=sys.stderr)
raise SystemExit(1)
src = METRICS_PY.read_text()
check(all(m in src for m in METRICS), f"(2) metrics.py names every contract metric "
f"(missing: {[m for m in METRICS if m not in src]})")
labels = label_names(src)
extra = labels - ALLOWED
check(not extra, f"(1) metrics.py attaches labels outside the allowlist: {sorted(extra)}")
hits = IDENTITY_LABEL.findall(src)
check(not hits, f"(1) metrics.py constructs an identity-bearing metric label: {sorted(set(hits))}")
rules = yaml.safe_load(ALERTS.read_text())
exprs = [r["expr"] for g in rules["spec"]["groups"] for r in g["rules"]]
check(len(exprs) >= 5, "(3) alerts.yaml parsed five or more rules")
alert_labels = set()
for e in exprs:
alert_labels |= label_names(e)
check(not (alert_labels - ALLOWED),
f"(3) alerts.yaml selects labels outside the allowlist: {sorted(alert_labels - ALLOWED)}")
# (4) non-vacuity
negatives = {
"cve label": 'vuln_scanner_findings{severity="HIGH",cve="CVE-1"} 1\n',
"image selector": 'max(vuln_scanner_findings{image="docker.io/x"})\n',
"workload label": 'f"vuln_scanner_targets{{workload=\\"{w}\\",state=\\"x\\"}} 1"\n',
}
for name, text in negatives.items():
check(bool(label_names(text) - ALLOWED), f"(4) the extractor missed: {name}")
ok_text = 'vuln_scanner_targets{state="scanned"} 3\nvuln_scanner_findings{severity="LOW",fixable="true"} 1\n'
check(not (label_names(ok_text) - ALLOWED), "(4) the extractor flagged the allowed labels")
if FAILURES:
print(f"CHECK FAIL ({NAME}): {len(FAILURES)} of {CHECKS} assertions failed", file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
print(f"PASS ({NAME}): {CHECKS} assertions")
PY
@@ -0,0 +1,30 @@
# =============================================================================
# L0 STATIC — the scanner's Flux tier cannot stall the platform, and its network reach is exactly what
# docs/17 "Security of the scanner itself" allows (R-VULN-09). Offline, no cluster.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: vuln-scanner-isolation
labels:
conformance.home.k3s/R-VULN-09: "true"
annotations:
conformance.home.k3s/requirements: "R-VULN-09"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
The vuln-scanner Flux tier is wait:false, in neither gating tier, and nothing dependsOn it;
ingress is exactly the Gateway to 4180, the CI runner to 8081 and Prometheus to 9090;
egress reaches the API, DNS, the public internet on 443 only with cluster and LAN excluded,
and the Keycloak path the sidecar needs. Proven by mutation.
spec:
steps:
- name: assert-isolation-and-network-shape
description: >-
Parse the Flux wiring and networkpolicy.yaml, assert isolation and the exact network shape,
and run the evaluators on mutated copies to prove they can fail.
try:
- script:
content: bash ./check.sh
+210
View File
@@ -0,0 +1,210 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — vuln-scanner-isolation
# Serves: R-VULN-09 (docs/17 "Security of the scanner itself").
#
# GIVEN the repository as authored,
# THEN (1) the Flux tier is wait:false, dependsOn platform-config, in neither gating tier, nothing
# anywhere dependsOn it, and both overlays carry its SOPS decryption patch;
# (2) ingress is EXACTLY: reserved:ingress to 4180, act-runner to 8081, prometheus to 9090;
# (3) egress to the world is tcp/443 only through a CIDR rule excluding every private range,
# and the only other destinations are the API entity, the Gateway entity on 443 and Keycloak
# (the oauth2-proxy sidecar's OIDC path, the one deliberate addition to the spec's list);
# no `world`, `all`, `cluster`, `host` or `remote-node` entity;
# (4) the namespace ships a default-deny baseline and is not app-namespace=true (single owner);
# AND (5) each evaluator is run on a mutated copy and must flag it (non-vacuity).
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)"
exec python3 - "$ROOT" <<'PY'
import copy, re, sys
from pathlib import Path
import yaml
ROOT = Path(sys.argv[1])
NAME = "vuln-scanner-isolation"
FAILURES, CHECKS = [], 0
FORBIDDEN_ENTITIES = {"world", "all", "cluster", "host", "remote-node"}
PRIVATE_RANGES = {"10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"}
def check(cond, desc):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(desc)
def docs(path):
return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)]
fk_path = ROOT / "clusters" / "base" / "vuln-scanner.yaml"
np_path = ROOT / "platform" / "vuln-scanner" / "networkpolicy.yaml"
for p in (fk_path, np_path):
if not p.is_file():
print(f"CHECK FAIL ({NAME}): {p.relative_to(ROOT)} is absent", file=sys.stderr)
raise SystemExit(1)
# (1) Flux wiring
fk = next(d for d in docs(fk_path) if d.get("kind") == "Kustomization")
check(fk["spec"].get("wait") is False, "(1) clusters/base/vuln-scanner.yaml must be wait: false")
check([d.get("name") for d in fk["spec"].get("dependsOn") or []] == ["platform-config"],
"(1) the tier dependsOn exactly platform-config")
check(fk["spec"].get("path") == "./platform/vuln-scanner", "(1) the tier reconciles ./platform/vuln-scanner")
check((fk["spec"].get("postBuild") or {}).get("substituteFrom"),
"(1) postBuild.substituteFrom is present (the route/alerts use ${BASE_DOMAIN})")
check("vuln-scanner.yaml" in (ROOT / "clusters" / "base" / "kustomization.yaml").read_text(),
"(1) clusters/base/kustomization.yaml lists vuln-scanner.yaml")
check("vuln-scanner" not in (ROOT / "platform" / "kustomization.yaml").read_text(),
"(1) platform/kustomization.yaml must not include vuln-scanner (the gating tier)")
for f in sorted((ROOT / "clusters").rglob("*.yaml")):
for d in docs(f):
for dep in (d.get("spec") or {}).get("dependsOn") or []:
check(not (isinstance(dep, dict) and dep.get("name") == "vuln-scanner"),
f"(1) {f.relative_to(ROOT)} {d.get('metadata', {}).get('name')} dependsOn vuln-scanner — "
"a broken scanner must never stall anything")
for env in ("dev", "prod"):
text = (ROOT / "clusters" / env / "kustomization.yaml").read_text()
check(re.search(r"name:\s*vuln-scanner\s*\n\s*namespace:\s*flux-system\s*\n\s*spec:\s*\n\s*decryption:",
text) is not None,
f"(1) clusters/{env}/kustomization.yaml carries the vuln-scanner SOPS decryption patch")
# Network shape
np_docs = docs(np_path)
dd = [d for d in np_docs if d.get("kind") == "NetworkPolicy" and d["metadata"]["name"] == "default-deny"]
check(len(dd) == 1 and set(dd[0]["spec"]["policyTypes"]) == {"Ingress", "Egress"} and dd[0]["spec"].get("ingress") == [],
"(4) default-deny baseline denies both directions with no ingress")
ns = next(d for d in docs(ROOT / "platform" / "vuln-scanner" / "namespace.yaml") if d.get("kind") == "Namespace")
check("homelab.keithjackson.dev/app-namespace" not in (ns["metadata"].get("labels") or {}),
"(4) the namespace must not be app-namespace=true (the default-deny would be double-owned)")
def cnp(name):
return next((d for d in np_docs if d.get("kind") == "CiliumNetworkPolicy" and d["metadata"]["name"] == name), None)
def ports_of(rule):
return {(p.get("port"), p.get("protocol", "TCP")) for tp in rule.get("toPorts") or [] for p in tp.get("ports") or []}
def eval_ingress(policy):
bad = []
rules = policy["spec"].get("ingress") or []
if len(rules) != 3:
bad.append(f"expected exactly 3 ingress rules, found {len(rules)}")
seen = set()
for r in rules:
ports = ports_of(r)
if r.get("fromEntities") == ["ingress"] and ports == {("4180", "TCP")}:
seen.add("gateway")
elif len(r.get("fromEndpoints") or []) == 1:
sel = r["fromEndpoints"][0].get("matchLabels") or {}
if (sel.get("io.kubernetes.pod.namespace") == "gitea-actions"
and sel.get("app.kubernetes.io/name") == "act-runner" and ports == {("8081", "TCP")}):
seen.add("ci")
elif (sel.get("io.kubernetes.pod.namespace") == "monitoring"
and sel.get("app.kubernetes.io/name") == "prometheus" and ports == {("9090", "TCP")}):
seen.add("prometheus")
else:
bad.append(f"unexpected ingress rule {r}")
else:
bad.append(f"unexpected ingress rule {r}")
missing = {"gateway", "ci", "prometheus"} - seen
if missing:
bad.append(f"ingress paths missing: {sorted(missing)}")
return bad
def eval_egress(policy):
bad = []
rules = policy["spec"].get("egress") or []
if len(rules) != 4:
bad.append(f"expected exactly 4 egress rules, found {len(rules)}")
world = 0
for r in rules:
ents = set(r.get("toEntities") or [])
if ents & FORBIDDEN_ENTITIES:
bad.append(f"egress admits forbidden entity {sorted(ents & FORBIDDEN_ENTITIES)}")
if ents == {"kube-apiserver"} and not r.get("toPorts"):
continue
if ents == {"ingress"} and ports_of(r) == {("443", "TCP")}:
continue
if r.get("toEndpoints"):
sel = r["toEndpoints"][0].get("matchLabels") or {}
if not (len(r["toEndpoints"]) == 1 and sel.get("io.kubernetes.pod.namespace") == "keycloak"
and sel.get("app") == "keycloak"):
bad.append(f"egress to unexpected endpoints {r}")
continue
if r.get("toCIDRSet"):
world += 1
sets = r["toCIDRSet"]
if len(sets) != 1 or sets[0].get("cidr") != "0.0.0.0/0":
bad.append(f"unexpected CIDR rule {r}")
continue
if not PRIVATE_RANGES.issubset(set(sets[0].get("except") or [])):
bad.append(f"the world CIDR rule does not exclude {sorted(PRIVATE_RANGES - set(sets[0].get('except') or []))}")
if ports_of(r) != {("443", "TCP")}:
bad.append(f"the world CIDR rule is not tcp/443 only ({sorted(ports_of(r))})")
continue
if r.get("toCIDR"):
bad.append(f"egress names a bare CIDR rule {r}")
continue
if not ents:
bad.append(f"unexpected egress rule {r}")
if world != 1:
bad.append(f"expected exactly one world CIDR rule, found {world}")
return bad
ing, eg = cnp("vuln-scanner-allow-ingress"), cnp("vuln-scanner-allow-egress")
check(ing is not None, "(2) the ingress CiliumNetworkPolicy exists")
check(eg is not None, "(3) the egress CiliumNetworkPolicy exists")
if ing is not None:
for v in eval_ingress(ing):
check(False, f"(2) {v}")
if eg is not None:
for v in eval_egress(eg):
check(False, f"(3) {v}")
check(len([d for d in np_docs if d.get("kind") == "CiliumNetworkPolicy"]) == 2,
"(3) exactly two CiliumNetworkPolicies, so no other path is added silently")
# (5) non-vacuity
if ing is not None and eg is not None:
def m_ing_open(p): p["spec"]["ingress"].append({"fromEntities": ["world"], "toPorts": [{"ports": [{"port": "8081", "protocol": "TCP"}]}]})
def m_ing_ns(p): p["spec"]["ingress"][1]["fromEndpoints"][0]["matchLabels"]["io.kubernetes.pod.namespace"] = "default"
def m_ing_port(p): p["spec"]["ingress"][0]["toPorts"][0]["ports"][0]["port"] = "8080"
for name, fn in (("extra world ingress", m_ing_open), ("CI rule moved to another namespace", m_ing_ns),
("Gateway reaches the page port", m_ing_port)):
m = copy.deepcopy(ing)
fn(m)
check(bool(eval_ingress(m)), f"(5) ingress evaluator missed mutant: {name}")
def m_eg_world(p): p["spec"]["egress"].append({"toEntities": ["world"]})
def m_eg_lan(p):
for r in p["spec"]["egress"]:
if r.get("toCIDRSet"):
r["toCIDRSet"][0]["except"] = ["10.0.0.0/8"]
def m_eg_port(p):
for r in p["spec"]["egress"]:
if r.get("toCIDRSet"):
r["toPorts"][0]["ports"].append({"port": "80", "protocol": "TCP"})
def m_eg_endpoint(p):
for r in p["spec"]["egress"]:
if r.get("toEndpoints"):
r["toEndpoints"][0]["matchLabels"]["io.kubernetes.pod.namespace"] = "default"
for name, fn in (("world entity added", m_eg_world), ("LAN not excluded", m_eg_lan),
("port 80 added", m_eg_port), ("endpoints moved", m_eg_endpoint)):
m = copy.deepcopy(eg)
fn(m)
check(bool(eval_egress(m)), f"(5) egress evaluator missed mutant: {name}")
if CHECKS < 18:
FAILURES.append(f"only {CHECKS} assertions ran — the check degenerated")
if FAILURES:
print(f"CHECK FAIL ({NAME}): {len(FAILURES)} of {CHECKS} assertions failed", file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
print(f"PASS ({NAME}): {CHECKS} assertions")
PY
@@ -0,0 +1,34 @@
# =============================================================================
# L0 STATIC — the CVE scanner's supply chain and privileges are what docs/17 "Security of the scanner
# itself" says (R-VULN-09). Offline, no cluster.
#
# WHY SCOPE, NOT REQUIREMENTS: the property spans platform/vuln-scanner and renovate.json5. A
# requirement-scoped selection would miss a Renovate edit that quietly lets the scanner auto-merge.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: vuln-scanner-supply-chain
labels:
conformance.home.k3s/R-VULN-09: "true"
annotations:
conformance.home.k3s/requirements: "R-VULN-09"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
The scanner image is pinned by digest and is not a banned Trivy version; Renovate never
auto-merges it and proposes it only after 7 days; its RBAC holds get/list/watch only and no
Secret; the pod is unprivileged with bounded resources and a size-limited cache. Each is
proven by mutation.
spec:
steps:
- name: assert-supply-chain-and-privileges
description: >-
Parse platform/vuln-scanner/deployment.yaml, rbac.yaml and renovate.json5, assert the
digest/version/Renovate/RBAC/security-context contract, and run each evaluator on mutated
copies to prove it can fail.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,351 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — vuln-scanner-supply-chain
# Serves: R-VULN-09 (docs/17 "Security of the scanner itself").
#
# GIVEN the repository as authored,
# THEN (1) the Trivy image is `ghcr.io/aquasecurity/trivy:<ver>@sha256:<64hex>` — by digest, and a
# tag so Renovate has something to follow;
# (2) that version is not 0.69.4/0.69.5/0.69.6 (the compromised March 2026 releases);
# (3) every pod image is digest-pinned except the oauth2-proxy sidecar, which follows the
# per-app sidecar pattern so one Renovate branch moves them all;
# (4) renovate.json5 keeps Trivy in the blast-radius list, automerge:false, minimumReleaseAge
# of at least 7 days, and no later rule re-enables automerge for it;
# (5) rbac.yaml: only ClusterRole/ClusterRoleBinding to the scanner SA, only get/list/watch, no
# Secret, no wildcard, and only the resources the inventory needs;
# (6) the pod: non-root, read-only rootfs, drop ALL, seccomp RuntimeDefault, no host
# namespaces or hostPath, CPU and memory limits on every container, an ephemeral-storage
# limit on the scanner, and a size-limited /cache emptyDir.
# (8) a renovate.json5 rule covers every file under platform/vuln-scanner: automerge:false,
# minimumReleaseAge of at least 7 days, and no later rule turns auto-merge back on (the
# python image holds the cluster-read token);
# (9) the scanner names the private registry it cannot reach, so that registry reports
# credential-required rather than a permanent failed.
# AND (7) each evaluator is run on a mutated copy and must flag it (non-vacuity).
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)"
exec python3 - "$ROOT" <<'PY'
import copy, json, re, sys
from pathlib import Path
import yaml
ROOT = Path(sys.argv[1])
NAME = "vuln-scanner-supply-chain"
FAILURES, CHECKS = [], 0
COMP = ROOT / "platform" / "vuln-scanner"
BANNED = {"0.69.4", "0.69.5", "0.69.6"}
DIGEST = re.compile(r"^sha256:[0-9a-f]{64}$")
ALLOWED_VERBS = {"get", "list", "watch"}
ALLOWED_RESOURCES = {
"": {"pods", "nodes"},
"apps": {"replicasets", "deployments", "statefulsets", "daemonsets"},
"batch": {"jobs", "cronjobs"},
"kubevirt.io": {"virtualmachineinstances"},
}
def check(cond, desc):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(desc)
def docs(path):
return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)]
for f in ("deployment.yaml", "rbac.yaml"):
if not (COMP / f).is_file():
print(f"CHECK FAIL ({NAME}): platform/vuln-scanner/{f} is absent", file=sys.stderr)
raise SystemExit(1)
def split_ref(image):
# A digest suffix, then a tag after the last ':' (only when it holds no '/', so a registry port
# stays in the repository part).
ref, _, digest = image.partition("@")
head, colon, tail = ref.rpartition(":")
if colon and "/" not in tail:
return head, tail, digest or None
return ref, None, digest or None
def containers(pod):
return (pod.get("initContainers") or []) + (pod.get("containers") or [])
def eval_images(pod):
bad = []
trivy = [c for c in containers(pod) if split_ref(c["image"])[0] == "ghcr.io/aquasecurity/trivy"]
if len(trivy) != 1:
bad.append(f"expected one Trivy image, found {len(trivy)}")
for c in containers(pod):
repo, tag, digest = split_ref(c["image"])
if repo == "quay.io/oauth2-proxy/oauth2-proxy":
if not tag or tag == "latest":
bad.append("oauth2-proxy sidecar has no version tag")
continue
if not digest or not DIGEST.match(digest):
bad.append(f"{c['name']} image {c['image']} is not pinned by digest")
if not tag:
bad.append(f"{c['name']} image {c['image']} has no tag for Renovate to follow")
if repo == "ghcr.io/aquasecurity/trivy" and tag in BANNED:
bad.append(f"Trivy {tag} is a banned (compromised) version")
return bad
def strip_json5(text):
# Drop // comments outside strings, then trailing commas.
out, i, in_str, esc = [], 0, False, False
while i < len(text):
ch = text[i]
if in_str:
out.append(ch)
if esc:
esc = False
elif ch == "\\":
esc = True
elif ch == '"':
in_str = False
i += 1
continue
if ch == '"':
in_str = True
out.append(ch)
i += 1
continue
if text.startswith("//", i):
while i < len(text) and text[i] != "\n":
i += 1
continue
out.append(ch)
i += 1
return re.sub(r",(\s*[}\]])", r"\1", "".join(out))
TRIVY_PKG = "ghcr.io/aquasecurity/trivy"
VERSION_UPDATE_RULE_RE = re.compile(r"^(\d+)\s*(day|days)$")
def eval_renovate(rules):
bad = []
trivy_rules = [r for r in rules if TRIVY_PKG in (r.get("matchPackageNames") or [])]
if not trivy_rules:
bad.append("no packageRule matches ghcr.io/aquasecurity/trivy (blast-radius list)")
return bad
if not any(r.get("automerge") is False and "blast-radius" in (r.get("addLabels") or [])
for r in trivy_rules):
bad.append("Trivy is not in a blast-radius rule with automerge:false")
ages = [r["minimumReleaseAge"] for r in trivy_rules if "minimumReleaseAge" in r]
if not ages or not any(VERSION_UPDATE_RULE_RE.match(a) and int(VERSION_UPDATE_RULE_RE.match(a).group(1)) >= 7
for a in ages):
bad.append(f"Trivy has no minimumReleaseAge of at least 7 days (found {ages})")
# Renovate applies packageRules in order, later wins; the last rule that names automerge for
# Trivy (or matches it generically by update type) must be false.
last = None
for r in rules:
if "automerge" in r and (TRIVY_PKG in (r.get("matchPackageNames") or [])
or set(r) <= {"description", "matchUpdateTypes", "automerge", "automergeType", "platformAutomerge"}):
last = r
if last is None or last.get("automerge") is not False:
bad.append("the last packageRule deciding automerge for Trivy leaves it true")
return bad
SCANNER_FILES = "platform/vuln-scanner/**"
def eval_renovate_files(rules):
positions = [i for i, r in enumerate(rules) if SCANNER_FILES in (r.get("matchFileNames") or [])]
if not positions:
return [f"no packageRule has matchFileNames {SCANNER_FILES}"]
bad = []
rule = rules[positions[-1]]
if rule.get("automerge") is not False:
bad.append("the platform/vuln-scanner file rule is not automerge:false")
age = VERSION_UPDATE_RULE_RE.match(rule.get("minimumReleaseAge") or "")
if not age or int(age.group(1)) < 7:
bad.append("the platform/vuln-scanner file rule has no minimumReleaseAge of at least 7 days")
if any(r.get("automerge") is True for r in rules[positions[-1] + 1:]):
bad.append("a rule after the platform/vuln-scanner file rule re-enables auto-merge")
return bad
def eval_private_registries(pod):
scanner = next(c for c in pod["containers"] if c["name"] == "scanner")
env = {e["name"]: e.get("value") for e in scanner.get("env") or []}
if env.get("VULN_PRIVATE_REGISTRIES") != "gitea.${BASE_DOMAIN}":
return ["the scanner does not name the private registry gitea.${BASE_DOMAIN}"]
return []
def eval_rbac(rbac):
bad = []
croles = [d for d in rbac if d.get("kind") == "ClusterRole"]
if len(croles) != 1:
bad.append(f"expected one ClusterRole, found {len(croles)}")
return bad
for rule in croles[0].get("rules") or []:
verbs = set(rule.get("verbs") or [])
if verbs - ALLOWED_VERBS:
bad.append(f"rule grants non-read verbs {sorted(verbs - ALLOWED_VERBS)}")
if "*" in verbs:
bad.append("rule grants the verb wildcard")
for group in rule.get("apiGroups") or []:
if group == "*" or group not in ALLOWED_RESOURCES:
bad.append(f"rule names apiGroup {group!r}")
continue
for res in rule.get("resources") or []:
if res == "*" or res not in ALLOWED_RESOURCES[group]:
bad.append(f"rule grants resource {group}/{res}")
if "secrets" in (rule.get("resources") or []):
bad.append("rule grants Secret access")
kinds = {d.get("kind") for d in rbac}
if kinds - {"ServiceAccount", "ClusterRole", "ClusterRoleBinding"}:
bad.append(f"unexpected RBAC kinds {sorted(kinds)}")
for b in (d for d in rbac if d.get("kind") == "ClusterRoleBinding"):
if (b.get("roleRef") or {}).get("name") != croles[0]["metadata"]["name"]:
bad.append("the ClusterRoleBinding does not bind the scanner's ClusterRole")
return bad
def eval_pod(pod):
bad = []
psc = pod.get("securityContext") or {}
if psc.get("runAsNonRoot") is not True:
bad.append("pod runAsNonRoot is not true")
if (psc.get("seccompProfile") or {}).get("type") != "RuntimeDefault":
bad.append("pod seccompProfile is not RuntimeDefault")
for f in ("hostNetwork", "hostPID", "hostIPC"):
if pod.get(f):
bad.append(f"pod sets {f}")
if any("hostPath" in v for v in pod.get("volumes") or []):
bad.append("pod mounts a hostPath volume")
for c in containers(pod):
sc = c.get("securityContext") or {}
if sc.get("readOnlyRootFilesystem") is not True:
bad.append(f"{c['name']} root filesystem is writable")
if sc.get("allowPrivilegeEscalation") is not False:
bad.append(f"{c['name']} allows privilege escalation")
if "ALL" not in ((sc.get("capabilities") or {}).get("drop") or []):
bad.append(f"{c['name']} does not drop ALL capabilities")
limits = (c.get("resources") or {}).get("limits") or {}
for lim in ("cpu", "memory"):
if lim not in limits:
bad.append(f"{c['name']} has no {lim} limit")
if c["name"] == "scanner" and "ephemeral-storage" not in limits:
bad.append("the scanner has no ephemeral-storage limit")
vols = {v["name"]: v for v in pod.get("volumes") or []}
if not (vols.get("cache", {}).get("emptyDir") or {}).get("sizeLimit"):
bad.append("the /cache emptyDir has no sizeLimit (Longhorn or etcd must never hold it)")
return bad
dep = next(d for d in docs(COMP / "deployment.yaml") if d.get("kind") == "Deployment")
pod = dep["spec"]["template"]["spec"]
for v in eval_images(pod) + eval_pod(pod):
check(False, f"(1-3,6) {v}")
check(dep["spec"].get("replicas") == 1, "(6) one replica (one RWO report store)")
check((dep["spec"].get("strategy") or {}).get("type") == "Recreate", "(6) Recreate strategy")
renovate = json.loads(strip_json5((ROOT / "renovate.json5").read_text()))
rules = renovate.get("packageRules") or []
check(len(rules) > 5, "(4) parsed renovate.json5 packageRules (parser degenerated)")
for v in eval_renovate(rules):
check(False, f"(4) {v}")
for v in eval_renovate_files(rules):
check(False, f"(8) {v}")
for v in eval_private_registries(pod):
check(False, f"(9) {v}")
rbac = docs(COMP / "rbac.yaml")
for v in eval_rbac(rbac):
check(False, f"(5) {v}")
# (7) non-vacuity
def mutated(fn, base):
m = copy.deepcopy(base)
fn(m)
return m
# Image mutants
def img_digest(p): p["initContainers"][0]["image"] = "ghcr.io/aquasecurity/trivy:0.76.1"
def img_banned(p): p["initContainers"][0]["image"] = re.sub(r":[0-9.]+@", ":0.69.5@", p["initContainers"][0]["image"])
def img_nopin(p): p["containers"][0]["image"] = "docker.io/library/python:3.14.9-alpine"
for name, fn in (("digest stripped from Trivy", img_digest), ("banned Trivy version", img_banned),
("python image unpinned", img_nopin)):
check(bool(eval_images(mutated(fn, pod))), f"(7) image evaluator missed mutant: {name}")
# RBAC mutants
def rbac_secret(r):
role = next(d for d in r if d.get("kind") == "ClusterRole")
role["rules"][0]["resources"].append("secrets")
def rbac_write(r):
role = next(d for d in r if d.get("kind") == "ClusterRole")
role["rules"][0]["verbs"].append("delete")
def rbac_wild(r):
role = next(d for d in r if d.get("kind") == "ClusterRole")
role["rules"][1]["resources"] = ["*"]
for name, fn in (("Secret resource granted", rbac_secret), ("write verb granted", rbac_write),
("resource wildcard granted", rbac_wild)):
check(bool(eval_rbac(mutated(fn, rbac))), f"(7) RBAC evaluator missed mutant: {name}")
# Renovate mutants
def ren_auto(r):
for x in r:
if TRIVY_PKG in (x.get("matchPackageNames") or []):
x["automerge"] = True
def ren_age(r):
for x in r:
if TRIVY_PKG in (x.get("matchPackageNames") or []) and "minimumReleaseAge" in x:
x["minimumReleaseAge"] = "2 days"
def ren_drop(r):
r[:] = [x for x in r if TRIVY_PKG not in (x.get("matchPackageNames") or [])]
def ren_later(r):
r.append({"matchUpdateTypes": ["digest"], "automerge": True})
for name, fn in (("Trivy automerge flipped true", ren_auto), ("minimumReleaseAge shortened", ren_age),
("Trivy removed from Renovate rules", ren_drop), ("later auto-merge rule appended", ren_later)):
check(bool(eval_renovate(mutated(fn, rules))), f"(7) Renovate evaluator missed mutant: {name}")
# Renovate file-rule mutants
def files_rule(r):
return next(x for x in r if SCANNER_FILES in (x.get("matchFileNames") or []))
def files_auto(r): files_rule(r)["automerge"] = True
def files_age(r): files_rule(r)["minimumReleaseAge"] = "1 day"
def files_drop(r): r.remove(files_rule(r))
def files_later(r): r.append({"matchDatasources": ["docker"], "automerge": True})
for name, fn in (("file rule automerge true", files_auto), ("file rule age shortened", files_age),
("file rule removed", files_drop), ("later rule re-enables auto-merge", files_later)):
check(bool(eval_renovate_files(mutated(fn, rules))), f"(7) file-rule evaluator missed mutant: {name}")
def reg_drop(p):
next(c for c in p["containers"] if c["name"] == "scanner")["env"][:] = [
e for e in next(c for c in p["containers"] if c["name"] == "scanner")["env"]
if e["name"] != "VULN_PRIVATE_REGISTRIES"]
check(bool(eval_private_registries(mutated(reg_drop, pod))), "(7) private-registry evaluator missed the removal")
# Pod mutants
def pod_root(p): p["securityContext"]["runAsNonRoot"] = False
def pod_host(p): p["hostNetwork"] = True
def pod_rw(p): p["containers"][0]["securityContext"]["readOnlyRootFilesystem"] = False
def pod_nolim(p): del p["containers"][0]["resources"]["limits"]["memory"]
def pod_nocap(p):
for v in p["volumes"]:
if v["name"] == "cache":
v["emptyDir"].pop("sizeLimit", None)
for name, fn in (("runAsNonRoot off", pod_root), ("hostNetwork on", pod_host), ("rw rootfs", pod_rw),
("no memory limit", pod_nolim), ("no cache sizeLimit", pod_nocap)):
check(bool(eval_pod(mutated(fn, pod))), f"(7) pod evaluator missed mutant: {name}")
if CHECKS < 15:
FAILURES.append(f"only {CHECKS} assertions ran — the check degenerated")
if FAILURES:
print(f"CHECK FAIL ({NAME}): {len(FAILURES)} of {CHECKS} assertions failed", file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
print(f"PASS ({NAME}): {CHECKS} assertions")
PY
@@ -0,0 +1,31 @@
# =============================================================================
# L0 STATIC — the CVE triage bot workflow fails loudly and is shaped like Renovate's (R-VULN-05,
# R-VULN-07, docs/17 "The triage bot"). Offline, no cluster. Depends on .gitea/workflows/cve-triage.yaml
# (the BOT's file); fails until that file is integrated.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: vuln-triage-bot-workflow-guards
labels:
conformance.home.k3s/R-VULN-05: "true"
conformance.home.k3s/R-VULN-07: "true"
annotations:
conformance.home.k3s/requirements: "R-VULN-05, R-VULN-07"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
The bot workflow runs on the 07:15 cron, has a workflow_dispatch dry-run input, never runs
concurrently with itself, and a step fails the job hard when RENOVATE_TOKEN is empty.
Proven by mutation.
spec:
steps:
- name: assert-workflow-guards
description: >-
Parse .gitea/workflows/cve-triage.yaml and assert the trigger, concurrency and hard
credential failure, then run the evaluator on mutated copies to prove it can fail.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,102 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — vuln-triage-bot-workflow-guards
# Serves: R-VULN-05 (runs on a schedule, fails loudly), R-VULN-07 (bot conventions, docs/17).
#
# GIVEN .gitea/workflows/cve-triage.yaml,
# THEN (1) it runs on `15 7 * * *` and has a workflow_dispatch `dry_run` input;
# (2) concurrency is set (the workflow never runs against itself);
# (3) a step fails the job HARD when RENOVATE_TOKEN is empty (exit non-zero, not skip);
# AND (4) the evaluator is run on mutated copies and must flag each (non-vacuity).
# A missing workflow file is a FAIL: this is whole-repo scoped and has no skip path.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)"
exec python3 - "$ROOT" <<'PY'
import copy, re, sys
from pathlib import Path
import yaml
ROOT = Path(sys.argv[1])
NAME = "vuln-triage-bot-workflow-guards"
FAILURES, CHECKS = [], 0
WF = ROOT / ".gitea" / "workflows" / "cve-triage.yaml"
def check(cond, desc):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(desc)
if not WF.is_file():
print(f"CHECK FAIL ({NAME}): .gitea/workflows/cve-triage.yaml is absent (the BOT's deliverable); "
f"whether it fails on a missing credential cannot be asserted without it", file=sys.stderr)
raise SystemExit(1)
doc = yaml.safe_load(WF.read_text())
def evaluate(doc):
bad = []
triggers = doc.get(True) if True in doc else doc.get("on") # YAML reads `on:` as boolean True
if not isinstance(triggers, dict):
return ["no triggers parsed"]
if "15 7 * * *" not in [s.get("cron") for s in triggers.get("schedule") or [] if isinstance(s, dict)]:
bad.append("no schedule with cron 15 7 * * *")
inputs = ((triggers.get("workflow_dispatch") or {}).get("inputs") or {})
if "dry_run" not in inputs:
bad.append("workflow_dispatch has no dry_run input")
conc = doc.get("concurrency")
jobs = doc.get("jobs") or {}
if not conc and not any(isinstance(j, dict) and j.get("concurrency") for j in jobs.values()):
bad.append("no concurrency set (the bot must not run against itself)")
hard_fail = False
for j in jobs.values():
for step in (j.get("steps") or []) if isinstance(j, dict) else []:
run = str(step.get("run", ""))
if ("RENOVATE_TOKEN" in run and re.search(r"(-z|!\s*-n)[^\n]*RENOVATE_TOKEN|RENOVATE_TOKEN[^\n]*(-z)", run)
and re.search(r"exit\s+[1-9]", run)):
hard_fail = True
if not hard_fail:
bad.append("no step fails hard (exit non-zero) when RENOVATE_TOKEN is empty")
return bad
for v in evaluate(doc):
check(False, f"(1-3) {v}")
check(isinstance(doc.get("jobs"), dict) and doc["jobs"], "(1) the workflow declares jobs")
# (4) non-vacuity
def mutate(fn):
m = copy.deepcopy(doc)
fn(m)
return m
def trig(m):
return m.get(True) if True in m else m.get("on")
def no_cron(m): trig(m).pop("schedule", None)
def no_dry(m): trig(m)["workflow_dispatch"] = {}
def no_conc(m):
m.pop("concurrency", None)
for j in m.get("jobs", {}).values():
if isinstance(j, dict):
j.pop("concurrency", None)
def no_guard(m):
for j in m.get("jobs", {}).values():
for step in j.get("steps") or []:
if "RENOVATE_TOKEN" in str(step.get("run", "")):
step["run"] = 'echo "::warning::RENOVATE_TOKEN unset, skipping"'
for name, fn in (("cron removed", no_cron), ("dry_run input removed", no_dry),
("concurrency removed", no_conc), ("credential check softened", no_guard)):
check(bool(evaluate(mutate(fn))), f"(4) evaluator missed mutant: {name}")
if FAILURES:
print(f"CHECK FAIL ({NAME}): {len(FAILURES)} of {CHECKS} assertions failed", file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
print(f"PASS ({NAME}): {CHECKS} assertions")
PY
@@ -62,7 +62,7 @@ spec:
ports: ports:
- containerPort: 8080 - containerPort: 8080
- name: oauth2-proxy - name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
imagePullPolicy: IfNotPresent imagePullPolicy: IfNotPresent
args: args:
- --http-address=0.0.0.0:4180 - --http-address=0.0.0.0:4180
@@ -85,7 +85,7 @@ spec:
- name: config - name: config
mountPath: /data/config mountPath: /data/config
- name: oauth2-proxy - name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
imagePullPolicy: IfNotPresent imagePullPolicy: IfNotPresent
args: args:
- --http-address=0.0.0.0:4180 - --http-address=0.0.0.0:4180
@@ -85,7 +85,7 @@ spec:
- name: config - name: config
mountPath: /data/config mountPath: /data/config
- name: oauth2-proxy - name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1 image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
imagePullPolicy: IfNotPresent imagePullPolicy: IfNotPresent
args: args:
- --http-address=0.0.0.0:4180 - --http-address=0.0.0.0:4180
@@ -0,0 +1,25 @@
# L1 fixture test: runs check.py (python3 stdlib, source imported from platform/vuln-scanner/src,
# no cluster, no network beyond loopback). conformance/run.sh --layer l1 runs check.sh offline;
# this Chainsaw Test wraps the same script so an in-cluster run executes identical assertions.
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: vuln-scanner-acceptances
labels:
conformance.home.k3s/R-VULN-08: "true"
annotations:
conformance.home.k3s/requirements: "R-VULN-08"
conformance.home.k3s/layer: "L1"
conformance.home.k3s/safety-critical: "false"
# Whole-repo scope: the touched-path map keys nothing to platform/vuln-scanner, so without this
# a change to the scanner would select no vuln-scanner test.
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/intent: >-
Given acceptance files, When parsed and applied, Then expiry reverts to open and malformed entries never hide a finding.
spec:
steps:
- name: run-offline-check
description: Run check.sh, the same assertions the offline harness runs.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,120 @@
#!/usr/bin/env python3
"""R-VULN-08: acceptance parsing, expiry and fail-safe behaviour."""
import sys
from datetime import date, datetime, timedelta, timezone
from pathlib import Path
HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(HERE.parents[3] / "platform/vuln-scanner/src"))
import acceptances # noqa: E402
import report # noqa: E402
FAILED = []
TODAY = date(2026, 11, 9)
NOW = datetime(2026, 11, 9, 12, 0, tzinfo=timezone.utc)
A1 = "sha256:" + "a1" * 32
def check(name, condition):
print(("ok " if condition else "FAIL ") + name)
if not condition:
FAILED.append(name)
def parse(text):
return acceptances.parse_text(text, TODAY)
entries, errors = parse("""# header comment
- id: CVE-2026-1
image: docker.io/library/nginx # exact match
reason: not reachable - feature disabled
expires: 2027-02-01
- id: CVE-2026-2
reason: "platform wide"
expires: 2026-12-01
""")
check("valid flat entries parse with no errors", len(entries) == 2 and errors == [])
check("inline comment is stripped", entries[0]["image"] == "docker.io/library/nginx")
check("omitted image means platform-wide", entries[1]["image"] is None)
check("empty list form is accepted", parse("# none\n[]\n") == ([], []))
check("empty file is accepted", parse("") == ([], []))
for name, text in (
("missing reason", "- id: CVE-1\n expires: 2026-12-01\n"),
("missing expires", "- id: CVE-1\n reason: x\n"),
("expires beyond 180 days", "- id: CVE-1\n reason: x\n expires: 2027-06-01\n"),
("unknown key", "- id: CVE-1\n reason: x\n expires: 2026-12-01\n scope: all\n"),
("not a date", "- id: CVE-1\n reason: x\n expires: soon\n"),
("impossible date", "- id: CVE-1\n reason: x\n expires: 2026-13-45\n"),
("duplicate key", "- id: CVE-1\n reason: x\n reason: y\n expires: 2026-12-01\n"),
("flow value", "- id: CVE-1\n reason: [x]\n expires: 2026-12-01\n"),
):
found, problems = parse(text)
check("rejected: " + name, found == [] and len(problems) == 1)
found, problems = parse("- id: CVE-1\n reason: x\n expires: 2026-12-01\n- id: bad entry\n expires: 2026-12-01\n")
check("one bad entry is ignored, the valid one kept", len(found) == 1 and len(problems) == 1)
found, problems = parse("- id: CVE-1\n reason: x\n")
check("bad indentation invalidates the whole file", found == [] and problems)
found, problems = parse("garbage\n- id: CVE-1\n reason: x\n expires: 2026-12-01\n")
check("a stray line invalidates the whole file", found == [] and problems)
check("missing file is an error, not an empty allowlist", acceptances.load("/nonexistent/acceptances.yaml")[1])
entry = {"id": "CVE-1", "image": None, "reason": "r", "expires": date(2026, 11, 9)}
check("acceptance holds on its expiry day", acceptances.match([entry], "CVE-1", "any/image", TODAY) is entry)
check("acceptance gone the day after", acceptances.match([entry], "CVE-1", "any/image", TODAY + timedelta(days=1)) is None)
scoped = {"id": "CVE-1", "image": "docker.io/library/nginx", "reason": "r", "expires": date(2026, 12, 1)}
check("image match is exact", acceptances.match([scoped], "CVE-1", "docker.io/library/nginx-extra", TODAY) is None)
check("image match accepts the exact image", acceptances.match([scoped], "CVE-1", "docker.io/library/nginx", TODAY) is scoped)
def state_with(findings):
target = {"id": "docker.io/library/nginx@" + A1, "type": "image", "image": "docker.io/library/nginx", "digest": A1,
"ref": "", "reason": None, "workloads": []}
return {
"inventory": [target], "database": {"updatedAt": "2026-11-09T06:00:00Z", "version": "2"},
"run": {"startedAt": "2026-11-09T09:00:00Z", "finishedAt": "2026-11-09T09:30:00Z", "status": "complete"},
"remediation": [], "inventoryError": None, "loadError": None,
"scans": {A1: {"firstDiscoveredAt": "2026-11-01T00:00:00Z", "lastInventoriedAt": "2026-11-09T09:00:00Z",
"lastAttemptAt": "2026-11-09T09:00:00Z", "lastSuccessAt": "2026-11-09T09:00:00Z", "outcome": "ok",
"reason": None, "failedSince": None, "findings": findings,
"firstSeen": {f["id"]: "2026-11-01T00:00:00Z" for f in findings}}},
}
FINDING = {"id": "CVE-1", "severity": "CRITICAL", "cvss": 9.0, "package": "p", "installedVersion": "1", "fixedVersion": "2",
"title": "t", "url": ""}
ST = state_with([FINDING])
def status(when, accepted, errs=()):
built = report.build(ST, accepted, list(errs), when)
return built, built["findings"][0]
def reason_entry(length):
return acceptances.parse_text("- id: CVE-1\n reason: " + "r" * length + "\n expires: 2026-12-01\n", NOW.date())
found, problems = reason_entry(3000)
check("a reason over 2048 characters ignores the entry and reports it", found == [] and any("2048" in p for p in problems))
found, problems = reason_entry(2048)
check("mutation: a reason of exactly 2048 characters is valid", len(found) == 1 and problems == [])
entry = {"id": "CVE-1", "image": None, "reason": "not reachable", "expires": date(2026, 11, 9)}
built, finding = status(NOW, [entry])
check("unexpired acceptance marks the finding accepted", finding["status"] == "accepted"
and finding["acceptance"] == {"reason": "not reachable", "expires": "2026-11-09"})
check("accepted report is schema-valid", report.validate(built) == [])
_, finding = status(NOW + timedelta(days=1), [entry])
check("expired acceptance reverts to the finding's own status", finding["status"] == "fix-available" and "acceptance" not in finding)
_, finding = status(NOW, [])
check("mutation: no acceptance leaves it open/fix-available", finding["status"] == "fix-available")
built, finding = status(NOW, [], ["bad entry"])
check("an invalid entry degrades the run", built["run"]["status"] == "degraded")
check("an invalid entry never hides the finding", finding["status"] == "fix-available")
print("FAILED: " + str(len(FAILED)) if FAILED else "all checks passed")
sys.exit(1 if FAILED else 0)
@@ -0,0 +1,7 @@
#!/usr/bin/env bash
# Runs the fixture-driven assertions in check.py. See chainsaw-test.yaml for the requirement tags.
# What this does NOT prove: anything at runtime (real Trivy, real API, real browser). L2/L3 cover that.
set -euo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export PYTHONDONTWRITEBYTECODE=1
exec python3 "${HERE}/check.py"
@@ -0,0 +1,28 @@
# L1 fixture test: runs check.py (python3 stdlib, source imported from platform/vuln-scanner/src,
# no cluster, no network beyond loopback). conformance/run.sh --layer l1 runs check.sh offline;
# this Chainsaw Test wraps the same script so an in-cluster run executes identical assertions.
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: vuln-scanner-api-metrics
labels:
conformance.home.k3s/R-VULN-02: "true"
conformance.home.k3s/R-VULN-03: "true"
conformance.home.k3s/R-VULN-05: "true"
conformance.home.k3s/R-VULN-06: "true"
annotations:
conformance.home.k3s/requirements: "R-VULN-02, R-VULN-03, R-VULN-05, R-VULN-06"
conformance.home.k3s/layer: "L1"
conformance.home.k3s/safety-critical: "false"
# Whole-repo scope: the touched-path map keys nothing to platform/vuln-scanner, so without this
# a change to the scanner would select no vuln-scanner test.
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/intent: >-
Given a running engine, When the HTTP surfaces are exercised, Then the API validates, metrics are aggregates only, and listeners answer during a scan.
spec:
steps:
- name: run-offline-check
description: Run check.sh, the same assertions the offline harness runs.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,230 @@
#!/usr/bin/env python3
"""R-VULN-02/03/05/06: internal API, metrics, publish refusal, and HTTP during a scan."""
import copy
import http.client
import json
import logging
import os
import re
import sys
import tempfile
import threading
from datetime import datetime, timedelta, timezone
from pathlib import Path
HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(HERE.parents[3] / "platform/vuln-scanner/src"))
import main # noqa: E402
import metrics # noqa: E402
import report # noqa: E402
import server # noqa: E402
import state as st # noqa: E402
logging.disable(logging.CRITICAL)
FAILED = []
NOW = datetime(2026, 11, 9, 12, 0, tzinfo=timezone.utc)
A1 = "sha256:" + "a1" * 32
def check(name, condition):
print(("ok " if condition else "FAIL ") + name)
if not condition:
FAILED.append(name)
class Clock:
def __init__(self):
self.now = NOW
def __call__(self):
return self.now
class Client:
def list_all(self, path):
if path.endswith("pods"):
return [{"metadata": {"namespace": "apps", "name": "p"}, "spec": {"containers": [{"name": "c", "image": "nginx:1"}]},
"status": {"containerStatuses": [{"name": "c", "imageID": "docker.io/library/nginx@" + A1}]}}]
return []
class Runner:
"""Blocks on release while a scan is 'running', so a test can prove the HTTP path stays live."""
def __init__(self):
self.entered = threading.Event()
self.release = threading.Event()
self.release.set()
self.result = {"SchemaVersion": 2, "Results": [{"Vulnerabilities": [
{"VulnerabilityID": "CVE-1", "Severity": "CRITICAL", "PkgName": "openssl", "InstalledVersion": "1",
"FixedVersion": "2", "Title": "t", "PrimaryURL": "https://example.test/1"}]}]}
def download_db(self):
pass
def db_info(self):
return {"updatedAt": "2026-11-09T06:00:00Z", "version": "2"}
def scan(self, _reference):
self.entered.set()
self.release.wait(10)
return self.result
def clean_cache(self):
pass
def request(port, method, path, body=None, headers=None):
conn = http.client.HTTPConnection("127.0.0.1", port, timeout=10)
conn.request(method, path, body, headers or {})
resp = conn.getresponse()
data = resp.read()
conn.close()
return resp.status, data, resp
def post(port, document, content_type="application/json"):
return request(port, "POST", "/triage", json.dumps(document), {"Content-Type": content_type})
with tempfile.TemporaryDirectory() as tmp:
accept = os.path.join(tmp, "a.yaml")
Path(accept).write_text("[]\n")
clock = Clock()
runner = Runner()
eng = main.Engine(tmp, Client(), runner, accept, clock=clock, publish_seconds=0)
servers = server.start_all(eng, ("127.0.0.1", 0), ("127.0.0.1", 0), ("127.0.0.1", 0))
page_port, internal, metrics_port = (s.server_address[1] for s in servers)
status, body, _ = request(internal, "GET", "/report.json")
check("report.json is 503 before any report", status == 503)
status, body, resp = request(page_port, "GET", "/")
check("page is 503 before any report, with the CSP", status == 503 and resp.getheader("Content-Security-Policy"))
text = request(metrics_port, "GET", "/metrics")[1].decode()
check("last-complete-run metric is absent before the first run", "last_complete_run" not in text)
check("triage heartbeat metric is absent before the first POST", "triage_heartbeat" not in text)
# HTTP must stay live while a scan is blocked mid-flight.
runner.release.clear()
worker = threading.Thread(target=eng.tick, daemon=True)
worker.start()
check("the scan is genuinely in flight", runner.entered.wait(10))
status_h, _, _ = request(internal, "GET", "/healthz")
status_p, _, _ = request(page_port, "GET", "/healthz")
status_m, _, _ = request(metrics_port, "GET", "/metrics")
check("listeners answer during a scan", (status_h, status_p, status_m) == (200, 200, 200))
runner.release.set()
worker.join(10)
check("the tick then completes", not worker.is_alive())
status, body, _ = request(internal, "GET", "/report.json")
check("report.json serves a schema-valid report after a tick", status == 200 and report.validate(json.loads(body)) == [])
# Page and CSP over HTTP
status, body, resp = request(page_port, "GET", "/?view=inventory")
check("page served over HTTP carries the CSP and nosniff",
status == 200 and resp.getheader("Content-Security-Policy").startswith("default-src 'none'") and resp.getheader("X-Content-Type-Options") == "nosniff")
status, body, resp = request(page_port, "GET", "/missing")
check("a 404 from the page listener also carries the CSP", status == 404 and resp.getheader("Content-Security-Policy"))
status, body, resp = request(page_port, "POST", "/")
check("an unsupported method on the page listener also carries the CSP", status == 501 and resp.getheader("Content-Security-Policy"))
# Triage API
valid = {"heartbeatAt": "2026-11-09T11:00:00Z", "remediation": [
{"image": "docker.io/library/nginx", "state": "pr-open", "url": "https://gitea.test/pulls/9", "updatedAt": "2026-11-09T11:00:00Z"}]}
status, body, _ = post(internal, valid)
check("valid triage is 204", status == 204 and body == b"")
check("remediation surfaces in the report", eng.report["remediation"][0]["state"] == "pr-open")
text = request(metrics_port, "GET", "/metrics")[1].decode()
check("heartbeat metric appears and uses server receipt time, not the bot's", f"triage_heartbeat_timestamp_seconds {int(NOW.timestamp())}" in text)
for name, mutate in (
("unknown top-level key", lambda d: d.update(extra=1)),
("missing heartbeat", lambda d: d.pop("heartbeatAt")),
("unknown remediation key", lambda d: d["remediation"][0].update(x=1)),
("bad remediation state", lambda d: d["remediation"][0].update(state="merged")),
("javascript url", lambda d: d["remediation"][0].update(url="javascript:x")),
("bad timestamp", lambda d: d.update(heartbeatAt="yesterday")),
):
document = copy.deepcopy(valid)
mutate(document)
status, _, _ = post(internal, document)
check("triage rejects " + name, status == 400)
for name, url in (("an uppercase scheme", "HTTP://gitea.test/pulls/1"), ("an empty url", ""), ("a javascript url", "javascript:alert(1)")):
bad = json.loads(json.dumps(valid))
bad["remediation"][0]["url"] = url
check("triage rejects " + name, post(internal, bad)[0] == 400)
check("triage rejects a wrong content type", post(internal, valid, "text/plain")[0] == 415)
status, _, _ = request(internal, "POST", "/triage", b"x" * 300000, {"Content-Type": "application/json"})
check("triage rejects an oversized body", status == 413)
many = {"heartbeatAt": "2026-11-09T11:00:00Z", "remediation": [valid["remediation"][0]] * 5001}
check("triage rejects more than 5000 entries", post(internal, many)[0] in (400, 413))
check("rejected triage leaves state untouched", eng.state["remediation"][0]["url"] == "https://gitea.test/pulls/9")
# Metrics labels and values
text = request(metrics_port, "GET", "/metrics")[1].decode()
allowed = {"state", "severity", "fixable"}
labels = {k for line in text.splitlines() for k in re.findall(r"([A-Za-z_]+)=\"", line)}
check("metric labels are within the allowlist", labels and labels <= allowed)
check("no metric carries a CVE, package, image or workload", not re.search(r"CVE-|openssl|nginx|apps", text))
for state in report.STATES:
check("targets gauge present for " + state, 'vuln_scanner_targets{state="' + state + '"}' in text)
check("critical fix findings counted with fixable true", 'vuln_scanner_findings{severity="CRITICAL",fixable="true"} 1' in text)
check("every severity and fixability appears", text.count("vuln_scanner_findings{") == 10)
check("failed-over-24h gauge exists", "vuln_scanner_failed_targets_over_24h 0" in text)
check("critical fix overdue is 0 inside the 72h grace", "vuln_scanner_critical_fix_overdue 0" in text)
clock.now += timedelta(hours=73)
text = request(metrics_port, "GET", "/metrics")[1].decode()
check("critical fix overdue is 1 after 72h", "vuln_scanner_critical_fix_overdue 1" in text)
check("last-complete-run metric present after a complete run", "vuln_scanner_last_complete_run_timestamp_seconds" in text)
for srv in servers:
srv.shutdown()
# Invalid report: refuse and keep the old one.
with tempfile.TemporaryDirectory() as tmp:
good = {"schemaVersion": 1, "generatedAt": "2026-11-09T10:00:00Z", "database": {"updatedAt": "2026-11-09T06:00:00Z", "version": "2"},
"run": {"startedAt": "2026-11-09T09:00:00Z", "finishedAt": "2026-11-09T09:30:00Z", "status": "complete"},
"targets": [], "findings": [], "remediation": []}
check("a valid report publishes", report.publish(good, tmp) is True)
before = Path(tmp, "report.json").read_bytes()
for name, mutate in (
("wrong schemaVersion", lambda r: r.update(schemaVersion=2)),
("extra top-level key", lambda r: r.update(extra=1)),
("bad timestamp", lambda r: r.update(generatedAt="soon")),
("unknown state", lambda r: r.update(targets=[{"id": "x", "type": "image", "image": "x", "digest": None, "state": "fine",
"reason": None, "lastScannedAt": None, "workloads": []}])),
("dangling targetId", lambda r: r.update(findings=[{"id": "CVE-1", "targetId": "nope", "severity": "LOW", "cvss": None,
"package": "p", "installedVersion": "1", "fixedVersion": None, "title": "t",
"url": "", "firstSeenAt": "2026-11-01T00:00:00Z", "status": "open"}])),
):
broken = copy.deepcopy(good)
mutate(broken)
check("invalid report refused: " + name, report.publish(broken, tmp) is False)
check("an invalid report writes nothing", Path(tmp, "report.json").read_bytes() == before)
check("accepted without acceptance is invalid", report.validate({**good, "targets": [{"id": "x", "type": "image", "image": "x", "digest": None,
"state": "scanned", "reason": None, "lastScannedAt": None, "workloads": []}], "findings": [{"id": "CVE-1", "targetId": "x", "severity": "LOW",
"cvss": None, "package": "p", "installedVersion": "1", "fixedVersion": None, "title": "t", "url": "", "firstSeenAt": "2026-11-01T00:00:00Z",
"status": "accepted"}]}) != [])
Path(tmp, "report.json").write_text("{not json")
check("a corrupt published report is not served", report.load_published(tmp) is None)
# Crash-consistency of state: the temp+rename leaves no temp files behind and the file parses.
with tempfile.TemporaryDirectory() as tmp:
store = st.Store(tmp)
state = st.new_state()
state["triageHeartbeatAt"] = "2026-11-09T11:00:00Z"
store.save(state)
check("state saves with no temp file left behind", sorted(os.listdir(tmp)) == ["state.json"])
check("state survives a reload", st.Store(tmp).load()["triageHeartbeatAt"] == "2026-11-09T11:00:00Z")
Path(tmp, "state.json").write_text("not json")
loaded = st.Store(tmp).load()
check("corrupt state starts empty and marks degraded", loaded["loadError"] and loaded["scans"] == {})
check("corrupt state is preserved for inspection", Path(tmp, "state.json.corrupt").exists())
# Metrics module directly, without the report.
empty_state = st.new_state()
text = metrics.render(None, empty_state, NOW)
check("with no report only the state-derived gauge exists", text == "vuln_scanner_failed_targets_over_24h 0\n")
print("FAILED: " + str(len(FAILED)) if FAILED else "all checks passed")
sys.exit(1 if FAILED else 0)
@@ -0,0 +1,7 @@
#!/usr/bin/env bash
# Runs the fixture-driven assertions in check.py. See chainsaw-test.yaml for the requirement tags.
# What this does NOT prove: anything at runtime (real Trivy, real API, real browser). L2/L3 cover that.
set -euo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export PYTHONDONTWRITEBYTECODE=1
exec python3 "${HERE}/check.py"
@@ -0,0 +1,25 @@
# L1 fixture test: runs check.py (python3 stdlib, source imported from platform/vuln-scanner/src,
# no cluster, no network beyond loopback). conformance/run.sh --layer l1 runs check.sh offline;
# this Chainsaw Test wraps the same script so an in-cluster run executes identical assertions.
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: vuln-scanner-inventory
labels:
conformance.home.k3s/R-VULN-01: "true"
annotations:
conformance.home.k3s/requirements: "R-VULN-01"
conformance.home.k3s/layer: "L1"
conformance.home.k3s/safety-critical: "false"
# Whole-repo scope: the touched-path map keys nothing to platform/vuln-scanner, so without this
# a change to the scanner would select no vuln-scanner test.
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/intent: >-
Given recorded API lists, When the inventory is built, Then owner chains, digests, by-tag CronJobs and not-covered targets resolve per R-VULN-01.
spec:
steps:
- name: run-offline-check
description: Run check.sh, the same assertions the offline harness runs.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,155 @@
#!/usr/bin/env python3
"""R-VULN-01: inventory from recorded API lists."""
import copy
import http.server
import json
import os
import sys
import tempfile
import threading
from pathlib import Path
HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(HERE.parents[3] / "platform/vuln-scanner/src"))
import inventory # noqa: E402
FAILED = []
def check(name, condition):
print(("ok " if condition else "FAIL ") + name)
if not condition:
FAILED.append(name)
CLUSTER = json.loads((HERE / "fixtures/cluster.json").read_text())
A1 = "sha256:" + "a1" * 32
B2 = "sha256:" + "b2" * 32
C3 = "sha256:" + "c3" * 32
D4 = "sha256:" + "d4" * 32
E5 = "sha256:" + "e5" * 32
F6 = "sha256:" + "f6" * 32
def by_id(targets):
return {t["id"]: t for t in targets}
def workloads(target):
return {(w["namespace"], w["kind"], w["name"]): w["containers"] for w in target["workloads"]}
targets = inventory.build(CLUSTER)
ids = by_id(targets)
nginx = ids["docker.io/library/nginx@" + A1]
check("ReplicaSet resolves to Deployment, not the ReplicaSet", workloads(nginx) == {
("apps", "Deployment", "web"): ["app"], ("apps", "Deployment", "api"): ["server"]})
check("docker-pullable:// prefix stripped and digest taken from imageID", nginx["digest"] == A1)
check("init container inventoried", ("apps", "Deployment", "web") in workloads(ids["ghcr.io/org/tool@" + B2]))
check("ephemeral container inventoried", "debug" in workloads(ids["docker.io/library/busybox@" + C3])[("apps", "Deployment", "web")])
backup = ids["ghcr.io/org/backup@" + E5]
check("Job resolves to CronJob", workloads(backup) == {("ops", "CronJob", "backup"): ["backup"]})
check("a CronJob with a running pod is not also inventoried from its template",
not any(t["id"].startswith("ghcr.io/org/backup:") for t in targets))
nightly = ids["ghcr.io/org/nightly:2.1"]
check("idle CronJob inventoried from template with no digest", nightly["digest"] is None
and workloads(nightly) == {("ops", "CronJob", "nightly"): ["job"]})
solo = ids["docker.io/library/alpine:3.20"]
check("empty imageID gives a by-tag target from the spec image", solo["digest"] is None)
check("bare Pod is its own workload", ("ops", "Pod", "solo") in workloads(solo))
check("Succeeded pod's image is not inventoried", not any(F6 in t["id"] for t in targets))
check("virt-launcher stays owned by its VirtualMachineInstance",
("kubevirt-vms", "VirtualMachineInstance", "vm1") in workloads(ids["quay.io/kubevirt/virt-launcher@sha256:" + "ab" * 32]))
vm, node = ids["vm/kubevirt-vms/vm1"], ids["node/prod-1"]
check("VM inventoried as non-container target", vm["type"] == "vm" and vm["digest"] is None and vm["reason"])
check("node inventoried as non-container target", node["type"] == "node" and node["reason"])
# Non-vacuous: mutating the fixture changes the answer the assertions above rely on.
broken = copy.deepcopy(CLUSTER)
broken["replicasets"] = []
broken["jobs"] = []
broken_ids = by_id(inventory.build(broken))
check("mutation: losing the ReplicaSet record leaves the ReplicaSet as the workload",
("apps", "ReplicaSet", "web-5f") in workloads(broken_ids["docker.io/library/nginx@" + A1]))
check("mutation: losing the Job record leaves the Job as the workload",
("ops", "Job", "backup-123") in workloads(broken_ids["ghcr.io/org/backup@" + E5]))
check("mutation: a Running phase on the Succeeded pod does inventory its image", any(
F6 in t["id"] for t in inventory.build({**CLUSTER, "pods": [
{**p, "status": {**p["status"], "phase": "Running"}} if p["metadata"]["name"] == "done-1" else p
for p in CLUSTER["pods"]]})))
for reference, expected in (
("nginx:1.27", ("docker.io/library/nginx", "1.27", None)),
("nginx", ("docker.io/library/nginx", None, None)),
("bitnami/redis:7", ("docker.io/bitnami/redis", "7", None)),
("ghcr.io/org/app:v1", ("ghcr.io/org/app", "v1", None)),
("registry.local:5000/x/y:2", ("registry.local:5000/x/y", "2", None)),
("index.docker.io/library/nginx:1", ("docker.io/library/nginx", "1", None)),
("ghcr.io/org/app:v1@" + A1, ("ghcr.io/org/app", "v1", A1)),
):
check("normalise " + reference, inventory.normalise_reference(reference) == expected)
check("bare config id is not a digest", inventory.digest_from_image_id("sha256:" + "a1" * 32) is None)
check("empty imageID has no digest", inventory.digest_from_image_id("") is None)
class FakeListClient:
"""list_all replaced; proves collect() tolerates exactly a 404 on the VMI API."""
def __init__(self, vmi_error):
self.vmi_error = vmi_error
def list_all(self, path):
if "kubevirt" in path and self.vmi_error:
raise self.vmi_error
return []
check("VMI API 404 tolerated as no VMs", inventory.collect(FakeListClient(inventory.NotFound("x")))["vmis"] == [])
try:
inventory.collect(FakeListClient(inventory.InventoryError("403")))
check("a non-404 VMI error is not swallowed", False)
except inventory.InventoryError:
check("a non-404 VMI error is not swallowed", True)
class PagedAPI(http.server.BaseHTTPRequestHandler):
def log_message(self, *args):
return
def do_GET(self):
if self.headers.get("Authorization") != "Bearer fixture-token":
self.send_response(403)
self.end_headers()
return
if self.path.startswith("/apis/kubevirt"):
self.send_response(404)
self.end_headers()
return
if "continue=" in self.path:
body = {"items": [{"metadata": {"name": "b"}}], "metadata": {}}
else:
body = {"items": [{"metadata": {"name": "a"}}], "metadata": {"continue": "page2"}}
payload = json.dumps(body).encode()
self.send_response(200)
self.send_header("Content-Length", str(len(payload)))
self.end_headers()
self.wfile.write(payload)
with tempfile.TemporaryDirectory() as tmp:
token = os.path.join(tmp, "token")
Path(token).write_text("fixture-token\n")
server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), PagedAPI)
threading.Thread(target=server.serve_forever, daemon=True).start()
client = inventory.KubeClient("http://127.0.0.1:" + str(server.server_address[1]), token, os.devnull)
items = client.list_all("/api/v1/pods")
check("pagination follows the continue token with the bearer token", [i["metadata"]["name"] for i in items] == ["a", "b"])
check("collect over real HTTP tolerates VMI 404", inventory.collect(client)["vmis"] == [])
server.shutdown()
print("FAILED: " + str(len(FAILED)) if FAILED else "all checks passed")
sys.exit(1 if FAILED else 0)
@@ -0,0 +1,7 @@
#!/usr/bin/env bash
# Runs the fixture-driven assertions in check.py. See chainsaw-test.yaml for the requirement tags.
# What this does NOT prove: anything at runtime (real Trivy, real API, real browser). L2/L3 cover that.
set -euo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export PYTHONDONTWRITEBYTECODE=1
exec python3 "${HERE}/check.py"
@@ -0,0 +1,395 @@
{
"cronjobs": [
{
"metadata": {
"name": "backup",
"namespace": "ops"
},
"spec": {
"jobTemplate": {
"spec": {
"template": {
"spec": {
"containers": [
{
"image": "ghcr.io/org/backup:1.0",
"name": "job"
}
]
}
}
}
}
}
},
{
"metadata": {
"name": "nightly",
"namespace": "ops"
},
"spec": {
"jobTemplate": {
"spec": {
"template": {
"spec": {
"containers": [
{
"image": "ghcr.io/org/nightly:2.1",
"name": "job"
}
]
}
}
}
}
}
}
],
"jobs": [
{
"metadata": {
"name": "backup-123",
"namespace": "ops",
"ownerReferences": [
{
"controller": true,
"kind": "CronJob",
"name": "backup"
}
]
}
},
{
"metadata": {
"name": "backup-99",
"namespace": "ops",
"ownerReferences": [
{
"controller": true,
"kind": "CronJob",
"name": "backup"
}
]
}
}
],
"nodes": [
{
"metadata": {
"name": "prod-1"
}
},
{
"metadata": {
"name": "prod-2"
}
}
],
"pods": [
{
"metadata": {
"name": "web-1",
"namespace": "apps",
"ownerReferences": [
{
"controller": true,
"kind": "ReplicaSet",
"name": "web-5f"
}
]
},
"spec": {
"containers": [
{
"image": "nginx:1.27",
"name": "app"
}
],
"ephemeralContainers": [
{
"image": "busybox:1.37",
"name": "debug"
}
],
"initContainers": [
{
"image": "ghcr.io/org/tool:v2",
"name": "init-db"
}
]
},
"status": {
"containerStatuses": [
{
"imageID": "docker-pullable://docker.io/library/nginx@sha256:a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1",
"name": "app"
}
],
"ephemeralContainerStatuses": [
{
"imageID": "docker.io/library/busybox@sha256:c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3c3",
"name": "debug"
}
],
"initContainerStatuses": [
{
"imageID": "ghcr.io/org/tool@sha256:b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2",
"name": "init-db"
}
],
"phase": "Running"
}
},
{
"metadata": {
"name": "web-2",
"namespace": "apps",
"ownerReferences": [
{
"controller": true,
"kind": "ReplicaSet",
"name": "web-5f"
}
]
},
"spec": {
"containers": [
{
"image": "nginx:1.27",
"name": "app"
}
],
"initContainers": [
{
"image": "ghcr.io/org/tool:v2",
"name": "init-db"
}
]
},
"status": {
"containerStatuses": [
{
"imageID": "docker.io/library/nginx@sha256:a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1",
"name": "app"
}
],
"initContainerStatuses": [
{
"imageID": "ghcr.io/org/tool@sha256:b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2",
"name": "init-db"
}
],
"phase": "Running"
}
},
{
"metadata": {
"name": "api-1",
"namespace": "apps",
"ownerReferences": [
{
"controller": true,
"kind": "ReplicaSet",
"name": "api-77"
}
]
},
"spec": {
"containers": [
{
"image": "nginx:1.27",
"name": "server"
}
]
},
"status": {
"containerStatuses": [
{
"imageID": "docker.io/library/nginx@sha256:a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1",
"name": "server"
}
],
"phase": "Running"
}
},
{
"metadata": {
"name": "db-0",
"namespace": "data",
"ownerReferences": [
{
"controller": true,
"kind": "StatefulSet",
"name": "db"
}
]
},
"spec": {
"containers": [
{
"image": "postgres:16",
"name": "postgres"
}
]
},
"status": {
"containerStatuses": [
{
"imageID": "docker.io/library/postgres@sha256:d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4",
"name": "postgres"
}
],
"phase": "Running"
}
},
{
"metadata": {
"name": "backup-123-x",
"namespace": "ops",
"ownerReferences": [
{
"controller": true,
"kind": "Job",
"name": "backup-123"
}
]
},
"spec": {
"containers": [
{
"image": "ghcr.io/org/backup:1.0",
"name": "backup"
}
]
},
"status": {
"containerStatuses": [
{
"imageID": "ghcr.io/org/backup@sha256:e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5",
"name": "backup"
}
],
"phase": "Running"
}
},
{
"metadata": {
"name": "solo",
"namespace": "ops"
},
"spec": {
"containers": [
{
"image": "alpine:3.20",
"name": "toolbox"
}
]
},
"status": {
"containerStatuses": [
{
"imageID": "",
"name": "toolbox"
}
],
"phase": "Pending"
}
},
{
"metadata": {
"name": "done-1",
"namespace": "ops",
"ownerReferences": [
{
"controller": true,
"kind": "Job",
"name": "backup-99"
}
]
},
"spec": {
"containers": [
{
"image": "ghcr.io/org/old-backup:0.1",
"name": "backup"
}
]
},
"status": {
"containerStatuses": [
{
"imageID": "ghcr.io/org/old-backup@sha256:f6f6f6f6f6f6f6f6f6f6f6f6f6f6f6f6f6f6f6f6f6f6f6f6f6f6f6f6f6f6f6f6",
"name": "backup"
}
],
"phase": "Succeeded"
}
},
{
"metadata": {
"name": "virt-launcher-vm1-abc",
"namespace": "kubevirt-vms",
"ownerReferences": [
{
"controller": true,
"kind": "VirtualMachineInstance",
"name": "vm1"
}
]
},
"spec": {
"containers": [
{
"image": "quay.io/kubevirt/virt-launcher:v1.8",
"name": "compute"
}
]
},
"status": {
"containerStatuses": [
{
"imageID": "quay.io/kubevirt/virt-launcher@sha256:abababababababababababababababababababababababababababababababab",
"name": "compute"
}
],
"phase": "Running"
}
}
],
"replicasets": [
{
"metadata": {
"name": "web-5f",
"namespace": "apps",
"ownerReferences": [
{
"controller": true,
"kind": "Deployment",
"name": "web"
}
]
}
},
{
"metadata": {
"name": "api-77",
"namespace": "apps",
"ownerReferences": [
{
"controller": true,
"kind": "Deployment",
"name": "api"
}
]
}
}
],
"vmis": [
{
"metadata": {
"name": "vm1",
"namespace": "kubevirt-vms"
}
}
]
}
@@ -0,0 +1,26 @@
# L1 fixture test: runs check.py (python3 stdlib, source imported from platform/vuln-scanner/src,
# no cluster, no network beyond loopback). conformance/run.sh --layer l1 runs check.sh offline;
# this Chainsaw Test wraps the same script so an in-cluster run executes identical assertions.
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: vuln-scanner-page
labels:
conformance.home.k3s/R-VULN-03: "true"
conformance.home.k3s/R-VULN-04: "true"
annotations:
conformance.home.k3s/requirements: "R-VULN-03, R-VULN-04"
conformance.home.k3s/layer: "L1"
conformance.home.k3s/safety-critical: "false"
# Whole-repo scope: the touched-path map keys nothing to platform/vuln-scanner, so without this
# a change to the scanner would select no vuln-scanner test.
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/intent: >-
Given a report fixture, When the page renders, Then all CVEs, containers and workloads appear in all views and hostile input is inert.
spec:
steps:
- name: run-offline-check
description: Run check.sh, the same assertions the offline harness runs.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,214 @@
#!/usr/bin/env python3
"""R-VULN-03/04: the rendered page. Assertions run against the actual HTML output."""
import base64
import copy
import hashlib
import json
import re
import sys
from datetime import datetime, timedelta, timezone
from html.parser import HTMLParser
from pathlib import Path
HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(HERE.parents[3] / "platform/vuln-scanner/src"))
import page # noqa: E402
FAILED = []
NOW = datetime(2026, 11, 9, 12, 0, tzinfo=timezone.utc)
REPORT = json.loads((HERE / "fixtures/report.json").read_text())
CLEAN_RE = re.compile(r"(?i)\bclean\b|no vulnerabilit")
def check(name, condition):
print(("ok " if condition else "FAIL ") + name)
if not condition:
FAILED.append(name)
def render(report=REPORT, query="", now=NOW):
status, headers, body = page.render(report, query, now)
return status, headers, body.decode("utf-8")
class Tags(HTMLParser):
def __init__(self):
super().__init__()
self.tags = []
self.attrs = []
def handle_starttag(self, tag, attrs):
self.tags.append(tag)
self.attrs.append((tag, dict(attrs)))
def tags(body):
parser = Tags()
parser.feed(body)
return parser
def covers_everything(body, report):
"""Every fixture CVE id, workload name and container name appears in the document."""
missing = [f["id"] for f in report["findings"] if f["id"] not in body]
for target in report["targets"]:
for workload in target["workloads"]:
if workload["name"] not in body:
missing.append(workload["name"])
missing.extend(c for c in workload["containers"] if c not in body)
return missing
def workload_view_covers(body, report):
"""Every workload row with its containers, and every CVE in the expansions."""
missing = covers_everything(body, report)
return missing
for view in ("cves", "workloads", "inventory"):
status, headers, body = render(query="view=" + view)
check(view + " view returns 200", status == 200)
check(view + " view has the Content-Security-Policy with no script source", "script-src" not in headers["Content-Security-Policy"]
and headers["Content-Security-Policy"].startswith("default-src 'none'"))
check(view + " view sets nosniff", headers["X-Content-Type-Options"] == "nosniff")
check(view + " view has no script element", "script" not in tags(body).tags)
style_body = re.search(r"<style>(.*?)</style>", body, re.S).group(1)
expected = base64.b64encode(hashlib.sha256(style_body.encode()).digest()).decode()
check(view + " view's CSP hash matches its inline style", "'sha256-" + expected + "'" in headers["Content-Security-Policy"])
check(view + " view header shows every state count", all(state + ": <strong>" in body for state in page.report_mod.STATES))
check(view + " view shows a coverage warning", "INCOMPLETE COVERAGE" in body)
check(view + " view makes no clean claim with unscanned targets", not CLEAN_RE.search(body))
status, headers, cves = render(query="view=cves")
check("CVE view contains every CVE", all(f["id"] in cves for f in REPORT["findings"]))
affected = {f["targetId"] for f in REPORT["findings"]}
expected_in_cves = {(t["id"], w["name"]) for t in REPORT["targets"] if t["id"] in affected for w in t["workloads"]}
check("CVE view expansions name every workload of an affected target", all(name in cves for _, name in expected_in_cves))
check("CVE view expansions name every container of an affected target",
all(c in cves for t in REPORT["targets"] if t["id"] in affected for w in t["workloads"] for c in w["containers"]))
for name in ("web", "api", "app", "server"):
check("CVE view expansion names " + name, name in cves)
status, headers, workloads = render(query="view=workloads")
check("workloads view contains every workload and container, including zero-finding ones", covers_everything(workloads, REPORT) == [])
status, headers, inventory = render(query="view=inventory")
check("inventory view contains every target", all(t["id"] in inventory for t in REPORT["targets"]))
check("inventory view contains every workload and container", covers_everything(inventory, REPORT) == [])
mutated = copy.deepcopy(REPORT)
mutated["findings"] = [f for f in mutated["findings"] if f["id"] != "CVE-2026-1003"]
check("mutation: the coverage evaluator catches a dropped CVE", "CVE-2026-1003" in covers_everything(render(mutated, "view=workloads")[2], REPORT))
_, _, body = render(query="view=cves")
check("hostile title is escaped", "&lt;script&gt;alert(1)&lt;/script&gt;" in body)
check("hostile title is never live", "<script>" not in body)
hostile = copy.deepcopy(REPORT)
hostile["findings"][0]["url"] = "javascript:alert(1)"
hostile["findings"][0]["title"] = '"><img src=x onerror=alert(1)>'
hostile["targets"][0]["reason"] = "<b onmouseover=alert(1)>x</b>"
hostile["remediation"][0]["url"] = "javascript:evil()"
_, _, body = render(hostile, "view=cves")
parsed = tags(body)
check("a javascript: advisory URL never becomes a link", not any("javascript:" in (a.get("href") or "") for t, a in parsed.attrs if t == "a"))
check("no element carries a hostile inline handler", not any(k.startswith("on") for _, a in parsed.attrs for k in a))
check("no img element was injected", "img" not in parsed.tags)
_, _, body = render(hostile, "view=inventory")
check("hostile reason is escaped in inventory", "<b onmouseover" not in body and "&lt;b onmouseover" in body)
_, _, body = render(query="view=cves&q=" + "%3Cscript%3E")
check("a hostile search term is escaped", "<script>" not in body)
check("a linked CVE points at its advisory", 'href="https://example.test/CVE-2026-1001"' in render(query="view=cves")[2])
stale = copy.deepcopy(REPORT)
stale["generatedAt"] = (NOW - timedelta(hours=27)).strftime("%Y-%m-%dT%H:%M:%SZ")
_, _, body = render(stale)
check("stale report shows a warning", "STALE REPORT" in body)
check("stale report shows no normal state", "Status: ATTENTION NEEDED" in body and "Status: CURRENT" not in body)
check("mutation: a fresh report has no stale warning", "STALE REPORT" not in render(REPORT)[2])
degraded = copy.deepcopy(REPORT)
degraded["run"]["status"] = "degraded"
_, _, body = render(degraded)
check("degraded run shows a warning and no normal state", "DEGRADED RUN" in body and "Status: ATTENTION NEEDED" in body and "Status: CURRENT" not in body)
allgood = copy.deepcopy(REPORT)
allgood["targets"] = [t for t in allgood["targets"] if t["state"] == "scanned"]
allgood["findings"] = []
_, _, body = render(allgood)
check("the clean claim appears when every target is scanned with zero findings", "Clean: all 1 targets" in body)
check("mutation: an all-scanned report shows the current state", "Status: CURRENT" in body)
one_pending = copy.deepcopy(allgood)
one_pending["targets"].append({"id": "x@sha256:" + "0" * 64, "type": "image", "image": "x", "digest": "sha256:" + "0" * 64,
"state": "pending", "reason": None, "lastScannedAt": None, "workloads": []})
_, _, body = render(one_pending)
check("one pending target removes the clean claim", "Clean:" not in body and not CLEAN_RE.search(body))
for state in ("failed", "stale", "credential-required", "by-tag", "not-covered"):
variant = copy.deepcopy(allgood)
variant["targets"].append({**one_pending["targets"][-1], "state": state})
check("a " + state + " target removes the clean claim", not CLEAN_RE.search(render(variant)[2]))
empty = copy.deepcopy(allgood)
empty["targets"] = []
check("zero targets is never a clean claim", not CLEAN_RE.search(render(empty)[2]))
degraded_clean = copy.deepcopy(allgood)
degraded_clean["run"]["status"] = "degraded"
check("a degraded run never makes a clean claim", not CLEAN_RE.search(render(degraded_clean)[2]))
status, headers, body = render(None)
check("no report yet is a 503 with the CSP", status == 503 and "Content-Security-Policy" in headers)
check("no report yet makes no clean claim", not CLEAN_RE.search(body))
_, _, body = render(query="view=cves&severity=CRITICAL")
check("the severity filter narrows the list", "CVE-2026-1001" in body and "CVE-2026-1004" not in body)
check("a filter is carried in pagination and sort links", "severity=CRITICAL" in body)
_, _, body = render(query="view=cves&status=accepted")
check("accepted findings stay listed with their reason", "CVE-2026-1003" in body and "not reachable" in body)
_, _, body = render(query="view=cves&fix=no")
check("the fix filter excludes fixed CVEs", "CVE-2026-1001" not in body and "CVE-2026-1002" in body)
_, _, body = render(query="view=inventory&status=credential-required")
check("inventory filters on coverage state", "ghcr.io/org/private" in body and "postgres" not in body.split("<main>")[1])
_, _, body = render(query="view=workloads&namespace=ops")
check("the namespace filter narrows workloads", "nightly" in body and "<td>web</td>" not in body)
_, _, body = render(query="view=cves&q=openssl")
check("free text search matches packages", "CVE-2026-1001" in body and "CVE-2026-1004" not in body)
_, _, body = render()
check("default view excludes nothing", all(f["id"] in body for f in REPORT["findings"]))
check("remediation column shows pr-open with a link", 'href="https://gitea.test/pulls/1"' in body and "pr-open" in body)
# Status and state belong to the line they describe (R-VULN-04).
images = [t for t in REPORT["targets"] if t["type"] == "image" and t["workloads"]]
check("fixture has two image targets with workloads", len(images) >= 2)
mixed = copy.deepcopy(REPORT)
first, second = next(t for t in mixed["targets"] if t["id"] == images[0]["id"]), next(t for t in mixed["targets"] if t["id"] == images[1]["id"])
first["state"], first["reason"] = "scanned", None
second["state"], second["reason"] = "stale", "last success long ago"
base = dict(REPORT["findings"][0], id="CVE-MIXED")
base.pop("acceptance", None)
mixed["findings"] = [
dict(base, targetId=first["id"], status="accepted", acceptance={"reason": "only-on-the-first", "expires": "2026-12-01"}),
dict(base, targetId=second["id"], status="fix-available"),
]
_, _, body = render(mixed, "view=cves&q=CVE-MIXED")
check("a mixed CVE reads as mixed, with counts", "mixed: 1 fix-available, 1 accepted" in body)
check("a mixed CVE's row does not carry one image's acceptance", "<td>mixed: 1 fix-available, 1 accepted</td>" in body)
first_line = page.e(first["id"]) + " — target scanned — status accepted (only-on-the-first, until 2026-12-01)"
second_line = page.e(second["id"]) + " — target stale (last success long ago) — status fix-available"
check("the accepted image's line carries its own acceptance", first_line in body)
check("the other image's line carries its own status and state", second_line in body)
check("no line attaches the acceptance to the unaccepted status", "status fix-available (only-on-the-first" not in body)
check("workload lines inside the expansion carry state and status",
"target stale (last success long ago) — status fix-available</li>" in body
and "containers: " + page.e(", ".join(second["workloads"][0]["containers"]) or "none") + " — target stale" in body)
_, _, plain = render(query="view=cves&status=accepted")
check("mutation: a uniformly accepted CVE does not read as mixed", "mixed:" not in plain and "CVE-2026-1003" in plain)
_, _, body = render(mixed, "view=workloads")
check("the workloads images column labels each image's state", page.e(second["id"]) + " — target stale (last success long ago)" in body)
check("the workloads expansion labels each finding's status, image and state",
"status fix-available — " + page.e(second["id"]) + " — target stale (last success long ago)</li>" in body)
check("the workloads expansion keeps an acceptance on its own line",
"status accepted (only-on-the-first, until 2026-12-01) — " + page.e(first["id"]) + " — target scanned</li>" in body)
big = copy.deepcopy(REPORT)
big["findings"] = [dict(REPORT["findings"][0], id="CVE-2026-" + format(i, "05d")) for i in range(250)]
_, _, body = render(big, "view=cves&size=50&page=2")
check("pagination states the total", "of 250 CVEs (page 2 of 5)" in body)
check("pagination shows only its page", "CVE-2026-00049" not in body and "CVE-2026-00050" in body and "CVE-2026-00100" not in body)
check("every page's links include next and previous", ">next<" in body and ">previous<" in body)
print("FAILED: " + str(len(FAILED)) if FAILED else "all checks passed")
sys.exit(1 if FAILED else 0)
@@ -0,0 +1,7 @@
#!/usr/bin/env bash
# Runs the fixture-driven assertions in check.py. See chainsaw-test.yaml for the requirement tags.
# What this does NOT prove: anything at runtime (real Trivy, real API, real browser). L2/L3 cover that.
set -euo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export PYTHONDONTWRITEBYTECODE=1
exec python3 "${HERE}/check.py"
@@ -0,0 +1,199 @@
{
"database": {
"updatedAt": "2026-11-09T06:00:00Z",
"version": "2"
},
"findings": [
{
"cvss": 9.1,
"firstSeenAt": "2026-11-01T00:00:00Z",
"fixedVersion": "3.0.2",
"id": "CVE-2026-1001",
"installedVersion": "1.0",
"package": "openssl",
"severity": "CRITICAL",
"status": "fix-available",
"targetId": "docker.io/library/nginx@sha256:a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1",
"title": "OpenSSL flaw",
"url": "https://example.test/CVE-2026-1001"
},
{
"cvss": null,
"firstSeenAt": "2026-11-01T00:00:00Z",
"fixedVersion": null,
"id": "CVE-2026-1002",
"installedVersion": "1.0",
"package": "zlib",
"severity": "LOW",
"status": "open",
"targetId": "docker.io/library/nginx@sha256:a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1",
"title": "<script>alert(1)</script>",
"url": "https://example.test/CVE-2026-1002"
},
{
"acceptance": {
"expires": "2027-01-01",
"reason": "not reachable"
},
"cvss": null,
"firstSeenAt": "2026-11-01T00:00:00Z",
"fixedVersion": "16.2",
"id": "CVE-2026-1003",
"installedVersion": "1.0",
"package": "libpq",
"severity": "HIGH",
"status": "accepted",
"targetId": "docker.io/library/postgres@sha256:d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4",
"title": "libpq flaw (retained from a prior scan)",
"url": "https://example.test/CVE-2026-1003"
},
{
"cvss": null,
"firstSeenAt": "2026-11-01T00:00:00Z",
"fixedVersion": null,
"id": "CVE-2026-1004",
"installedVersion": "1.0",
"package": "curl",
"severity": "MEDIUM",
"status": "open",
"targetId": "ghcr.io/org/nightly:2.1",
"title": "curl flaw",
"url": "https://example.test/CVE-2026-1004"
}
],
"generatedAt": "2026-11-09T10:00:00Z",
"remediation": [
{
"image": "docker.io/library/nginx",
"state": "pr-open",
"updatedAt": "2026-11-09T10:00:00Z",
"url": "https://gitea.test/pulls/1"
}
],
"run": {
"finishedAt": "2026-11-09T09:30:00Z",
"startedAt": "2026-11-09T09:00:00Z",
"status": "complete"
},
"schemaVersion": 1,
"targets": [
{
"digest": "sha256:a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1",
"id": "docker.io/library/nginx@sha256:a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1",
"image": "docker.io/library/nginx",
"lastScannedAt": "2026-11-09T10:00:00Z",
"reason": null,
"state": "scanned",
"type": "image",
"workloads": [
{
"containers": [
"app"
],
"kind": "Deployment",
"name": "web",
"namespace": "apps"
},
{
"containers": [
"server"
],
"kind": "Deployment",
"name": "api",
"namespace": "apps"
}
]
},
{
"digest": "sha256:d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4",
"id": "docker.io/library/postgres@sha256:d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4",
"image": "docker.io/library/postgres",
"lastScannedAt": "2026-11-07T09:00:00Z",
"reason": "trivy timed out",
"state": "failed",
"type": "image",
"workloads": [
{
"containers": [
"postgres"
],
"kind": "StatefulSet",
"name": "db",
"namespace": "data"
}
]
},
{
"digest": "sha256:e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5",
"id": "ghcr.io/org/private@sha256:e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5",
"image": "ghcr.io/org/private",
"lastScannedAt": null,
"reason": "registry ghcr.io refused anonymous access",
"state": "credential-required",
"type": "image",
"workloads": [
{
"containers": [
"priv"
],
"kind": "Deployment",
"name": "priv",
"namespace": "apps"
}
]
},
{
"digest": null,
"id": "ghcr.io/org/nightly:2.1",
"image": "ghcr.io/org/nightly",
"lastScannedAt": "2026-11-09T10:00:00Z",
"reason": null,
"state": "by-tag",
"type": "image",
"workloads": [
{
"containers": [
"job"
],
"kind": "CronJob",
"name": "nightly",
"namespace": "ops"
}
]
},
{
"digest": null,
"id": "vm/kubevirt-vms/vm1",
"image": "",
"lastScannedAt": null,
"reason": "guest OS packages are not assessed by this scanner",
"state": "not-covered",
"type": "vm",
"workloads": [
{
"containers": [],
"kind": "VirtualMachineInstance",
"name": "vm1",
"namespace": "kubevirt-vms"
}
]
},
{
"digest": null,
"id": "node/prod-1",
"image": "",
"lastScannedAt": null,
"reason": "node OS packages and the k3s binary are not assessed by this scanner",
"state": "not-covered",
"type": "node",
"workloads": [
{
"containers": [],
"kind": "Node",
"name": "prod-1",
"namespace": ""
}
]
}
]
}
@@ -0,0 +1,27 @@
# L1 fixture test: runs check.py (python3 stdlib, source imported from platform/vuln-scanner/src,
# no cluster, no network beyond loopback). conformance/run.sh --layer l1 runs check.sh offline;
# this Chainsaw Test wraps the same script so an in-cluster run executes identical assertions.
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: vuln-scanner-scan-loop
labels:
conformance.home.k3s/R-VULN-02: "true"
conformance.home.k3s/R-VULN-03: "true"
conformance.home.k3s/R-VULN-05: "true"
annotations:
conformance.home.k3s/requirements: "R-VULN-02, R-VULN-03, R-VULN-05"
conformance.home.k3s/layer: "L1"
conformance.home.k3s/safety-critical: "false"
# Whole-repo scope: the touched-path map keys nothing to platform/vuln-scanner, so without this
# a change to the scanner would select no vuln-scanner test.
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/intent: >-
Given a fake Trivy runner and clock, When ticks run, Then shared digests scan once, failures isolate, and coverage states and first-seen persist.
spec:
steps:
- name: run-offline-check
description: Run check.sh, the same assertions the offline harness runs.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,444 @@
#!/usr/bin/env python3
"""R-VULN-02/03/05: scan loop semantics with a fake Trivy runner and a controllable clock."""
import json
import logging
import os
import sys
import tempfile
import threading
from datetime import datetime, timedelta, timezone
from pathlib import Path
HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(HERE.parents[3] / "platform/vuln-scanner/src"))
import main # noqa: E402
import report # noqa: E402
import trivy # noqa: E402
logging.disable(logging.CRITICAL)
FAILED = []
A1 = "sha256:" + "a1" * 32
D4 = "sha256:" + "d4" * 32
E5 = "sha256:" + "e5" * 32
NGINX = "docker.io/library/nginx@" + A1
POSTGRES = "docker.io/library/postgres@" + D4
PRIVATE = "ghcr.io/org/private@" + E5
NIGHTLY = "ghcr.io/org/nightly:2.1"
CLUSTER = json.loads((HERE / "fixtures/cluster.json").read_text())
def fixture(name):
return json.loads((HERE / "fixtures" / name).read_text())
def check(name, condition):
print(("ok " if condition else "FAIL ") + name)
if not condition:
FAILED.append(name)
class Clock:
def __init__(self):
self.now = datetime(2026, 11, 9, 12, 0, tzinfo=timezone.utc)
def __call__(self):
return self.now
def advance(self, **delta):
self.now += timedelta(**delta)
class FakeClient:
def __init__(self, cluster):
self.cluster = cluster
self.fail = False
def list_all(self, path):
if self.fail:
raise RuntimeError("api down")
for name, item in (("pods", "/api/v1/pods"), ("replicasets", "replicasets"), ("jobs", "/jobs"),
("cronjobs", "cronjobs"), ("nodes", "/nodes"), ("vmis", "virtualmachineinstances")):
if path.endswith(item):
return self.cluster[name]
return []
class FakeRunner:
"""Recorded results, no Trivy binary."""
def __init__(self):
self.scans = []
self.outcomes = {
NGINX: fixture("trivy-nginx.json"), POSTGRES: fixture("trivy-postgres.json"),
NIGHTLY: fixture("trivy-nightly.json"),
PRIVATE: trivy.classify_failure("ghcr.io", "FATAL: GET https://ghcr.io/: UNAUTHORIZED: authentication required"),
}
self.db_age_hours = 1
self.db_fails = False
self.clock = None
def download_db(self):
if self.db_fails:
raise trivy.ScanFailure("failed", "database update failed")
def db_info(self):
return {"updatedAt": (self.clock() - timedelta(hours=self.db_age_hours)).strftime("%Y-%m-%dT%H:%M:%SZ"), "version": "2"}
def scan(self, reference):
self.scans.append(reference)
outcome = self.outcomes[reference]
if isinstance(outcome, Exception):
raise outcome
return outcome
def clean_cache(self):
pass
def engine(tmp, runner=None, clock=None, acc="[]\n", cluster=CLUSTER, private=()):
path = os.path.join(tmp, "acceptances.yaml")
if not os.path.exists(path):
Path(path).write_text(acc)
clock = clock or Clock()
runner = runner or FakeRunner()
runner.clock = clock
return main.Engine(tmp, FakeClient(cluster), runner, path, clock=clock, publish_seconds=0, private_registries=private), runner, clock
def targets(eng):
return {t["id"]: t for t in eng.report["targets"]}
def findings(eng, target_id):
return [f for f in eng.report["findings"] if f["targetId"] == target_id]
def one_scan_per_digest(scans):
return len(scans) == len(set(scans))
with tempfile.TemporaryDirectory() as tmp:
eng, runner, clock = engine(tmp)
eng.tick()
t = targets(eng)
check("two workloads sharing a digest produce exactly one scan", runner.scans.count(NGINX) == 1 and one_scan_per_digest(runner.scans))
check("mutation: the digest-uniqueness evaluator rejects a double scan", not one_scan_per_digest(["a", "a"]))
check("the shared digest maps back to both workloads",
{(w["kind"], w["name"]) for w in t[NGINX]["workloads"]} == {("Deployment", "web"), ("Deployment", "api")})
check("shared digest findings appear once per target", len(findings(eng, NGINX)) == 2)
check("digest target is scanned", t[NGINX]["state"] == "scanned")
check("idle CronJob template is by-tag after scanning, not scanned", t[NIGHTLY]["state"] == "by-tag")
check("node is not-covered", t["node/prod-1"]["state"] == "not-covered")
check("report is schema-valid", report.validate(eng.report) == [])
check("credential-required is its own state, not failed", t[PRIVATE]["state"] == "credential-required")
check("credential-required reason names the registry", "ghcr.io" in t[PRIVATE]["reason"])
check("credential-required does not count toward failed_over_24h", report.failed_over_24h(eng.state, clock() + timedelta(hours=48)) == 0)
check("report is written to disk", json.loads(Path(tmp, "report.json").read_text())["generatedAt"])
first_seen = {f["id"]: f["firstSeenAt"] for f in findings(eng, NGINX)}
clock.advance(hours=21)
runner.outcomes[POSTGRES] = trivy.ScanFailure("failed", "pull error: manifest unknown")
eng.tick()
t = targets(eng)
check("a failed image leaves the run continuing", runner.scans.count(NGINX) == 2 and runner.scans.count(NIGHTLY) == 2)
check("the failed target is failed", t[POSTGRES]["state"] == "failed")
check("the failed target's reason is shown", "manifest unknown" in t[POSTGRES]["reason"])
check("the failed target keeps its prior findings", [f["id"] for f in findings(eng, POSTGRES)] == ["CVE-2026-0010"])
check("other targets stay scanned", t[NGINX]["state"] == "scanned")
check("first-seen survives a re-scan", {f["id"]: f["firstSeenAt"] for f in findings(eng, NGINX)} == first_seen)
check("run with one failed image is still complete (failure is per-target)", eng.report["run"]["status"] == "complete")
clock.advance(hours=25)
check("failed target is counted over 24h after 25h", report.failed_over_24h(eng.state, clock()) == 1)
eng.publish()
check("a target whose last success aged past 24h is stale", targets(eng)[NGINX]["state"] == "stale")
check("stale target keeps findings", len(findings(eng, NGINX)) == 2)
# Restart: a new Engine loads everything from disk.
eng2, runner2, clock2 = engine(tmp, runner=FakeRunner(), clock=clock)
check("report survives restart", eng2.report is not None and eng2.report_bytes() is not None)
check("last complete run survives restart", eng2.state["lastCompleteRunAt"] is not None)
eng2.publish()
check("first-seen survives restart", {f["id"]: f["firstSeenAt"] for f in findings(eng2, NGINX)} == first_seen)
with tempfile.TemporaryDirectory() as tmp:
# Not-scanned is never zero findings: the first tick has not scanned anything yet.
eng, runner, clock = engine(tmp)
eng.refresh_inventory()
eng.publish()
t = targets(eng)
check("before scanning, digest targets are pending", t[NGINX]["state"] == "pending" and t[POSTGRES]["state"] == "pending")
check("run object exists and is degraded before the first run", eng.report["run"]["status"] == "degraded")
check("no fabricated findings before scanning", eng.report["findings"] == [])
check("zero findings report still counts every target", len(eng.report["targets"]) == 5)
clock.advance(hours=3)
eng.publish()
t = targets(eng)
check("pending beyond 2h becomes failed, never silently absent", t[NGINX]["state"] == "failed")
check("failed reason is the contract's text", t[NGINX]["reason"] == report.NOT_SCANNED_REASON)
check("a never-scanned failed target has lastScannedAt null", t[NGINX]["lastScannedAt"] is None)
clock.advance(hours=24)
check("never-scanned target counts as failed over 24h", report.failed_over_24h(eng.state, clock()) >= 1)
with tempfile.TemporaryDirectory() as tmp:
# Database problems degrade the run but it still reports.
eng, runner, clock = engine(tmp)
runner.db_fails = True
eng.tick()
check("DB update failure degrades the run", eng.report["run"]["status"] == "degraded")
check("DB update failure still scans and reports", runner.scans and eng.report["findings"])
check("a degraded run is not recorded as complete", eng.state["lastCompleteRunAt"] is None)
with tempfile.TemporaryDirectory() as tmp:
eng, runner, clock = engine(tmp)
runner.db_age_hours = 49
eng.tick()
check("a DB older than 48h degrades the run", eng.report["run"]["status"] == "degraded")
runner.db_age_hours = 1
clock.advance(hours=21)
eng.tick()
check("mutation: a fresh DB gives a complete run", eng.report["run"]["status"] == "complete" and eng.state["lastCompleteRunAt"])
with tempfile.TemporaryDirectory() as tmp:
# A target leaving the inventory leaves the report, with its findings.
eng, runner, clock = engine(tmp)
eng.tick()
check("postgres is present first", POSTGRES in targets(eng))
gone = dict(CLUSTER)
gone["pods"] = [p for p in CLUSTER["pods"] if p["metadata"]["name"] != "db-0"]
eng.client.cluster = gone
clock.advance(minutes=15)
eng.tick()
check("a target leaving the inventory leaves the report", POSTGRES not in targets(eng))
check("its findings go with it", not findings(eng, POSTGRES))
with tempfile.TemporaryDirectory() as tmp:
# Unexpected exceptions are contained at each level.
eng, runner, clock = engine(tmp)
runner.outcomes[POSTGRES] = RuntimeError("boom")
eng.tick()
check("an unexpected scan exception fails only that target", targets(eng)[POSTGRES]["state"] == "failed")
check("the run continued past it", runner.scans.count(NIGHTLY) == 1)
eng.client.fail = True
clock.advance(minutes=15)
eng.tick()
check("an inventory failure keeps the previous inventory", NGINX in targets(eng))
check("an inventory failure degrades the report", eng.report["run"]["status"] == "degraded")
calls = []
def crashing_tick():
calls.append(1)
if len(calls) == 1:
raise RuntimeError("cycle crash")
eng.stop.set()
eng.tick = crashing_tick
eng.tick_seconds = 0
thread = threading.Thread(target=eng.run_forever, daemon=True)
thread.start()
thread.join(10)
check("a crash in one cycle is logged and the loop continues", len(calls) == 2 and not thread.is_alive())
class Killed(BaseException):
"""What an OOM kill looks like from inside the process: no `except Exception` runs."""
class KillingRunner(FakeRunner):
def __init__(self, victim, data_dir):
super().__init__()
self.victim = victim
self.data_dir = data_dir
self.on_disk = None
def scan(self, reference):
if reference == self.victim:
self.on_disk = json.loads(Path(self.data_dir, "state.json").read_text()).get("inFlight")
raise Killed()
return super().scan(reference)
with tempfile.TemporaryDirectory() as tmp:
# R-VULN-05 BLOCKER: a scan that kills the container must cost that image only.
eng, runner, clock = engine(tmp, runner=KillingRunner(POSTGRES, tmp))
killed = False
try:
eng.tick()
except Killed:
killed = True
check("kill: the simulated OOM really aborted the process path", killed)
check("kill: inFlight is on disk BEFORE the scan starts", runner.on_disk == D4)
eng2, runner2, clock2 = engine(tmp, runner=FakeRunner(), clock=clock)
check("kill: startup clears inFlight", eng2.state["inFlight"] is None)
check("kill: clearing is persisted", json.loads(Path(tmp, "state.json").read_text())["inFlight"] is None)
eng2.publish()
t = targets(eng2)
check("kill: the target is failed", t[POSTGRES]["state"] == "failed")
check("kill: the reason is the killed-scan text", t[POSTGRES]["reason"] == main.KILLED_REASON)
eng2.tick()
t = targets(eng2)
check("kill: images after the killed one get scanned", PRIVATE in runner2.scans and t[PRIVATE]["state"] == "credential-required")
check("kill: the killed image is not retried on restart (no wedge)", POSTGRES not in runner2.scans)
check("kill: the killed target stays failed with its reason", t[POSTGRES]["reason"] == main.KILLED_REASON)
check("kill: inFlight is clear after a normal cycle", eng2.state["inFlight"] is None)
clock.advance(hours=21)
eng2.tick()
check("kill: the killed image is retried after the sit-out", runner2.scans.count(POSTGRES) == 1)
check("kill: a successful retry clears the failure", targets(eng2)[POSTGRES]["state"] == "scanned")
check("kill: a successful retry clears killedAt", eng2.state["scans"][D4]["killedAt"] is None)
with tempfile.TemporaryDirectory() as tmp:
eng, runner, clock = engine(tmp)
eng.refresh_inventory()
eng2, runner2, clock2 = engine(tmp, runner=FakeRunner(), clock=clock)
eng2.publish()
check("mutation: a restart with no inFlight fails nothing", targets(eng2)[POSTGRES]["state"] == "pending")
with tempfile.TemporaryDirectory() as tmp:
# A registry the scanner cannot reach is credential-required, and Trivy is never invoked.
eng, runner, clock = engine(tmp, private=[" GHCR.io ", ""])
eng.tick()
t = targets(eng)
check("private registry: reported credential-required", t[PRIVATE]["state"] == "credential-required")
check("private registry: the reason names the registry", "ghcr.io" in t[PRIVATE]["reason"])
check("private registry: Trivy is never invoked for it", PRIVATE not in runner.scans and NIGHTLY not in runner.scans)
check("private registry: other registries still scan", NGINX in runner.scans)
check("private registry: not counted as failed", report.failed_over_24h(eng.state, clock() + timedelta(hours=48)) == 0)
with tempfile.TemporaryDirectory() as tmp:
eng, runner, clock = engine(tmp)
eng.tick()
check("mutation: with no private registry configured Trivy is invoked", PRIVATE in runner.scans)
with tempfile.TemporaryDirectory() as tmp:
# A corrupt state.json degrades the run that rebuilds from it, not every run thereafter.
Path(tmp, "state.json").write_text("{not json")
eng, runner, clock = engine(tmp)
check("loadError: set after a corrupt state.json", eng.state["loadError"])
eng.tick()
check("loadError: the rebuilding run is degraded", eng.report["run"]["status"] == "degraded")
check("loadError: cleared after a full run with a good inventory", eng.state["loadError"] is None)
clock.advance(hours=21)
eng.tick()
check("loadError: the next full run is complete", eng.report["run"]["status"] == "complete" and eng.state["lastCompleteRunAt"])
with tempfile.TemporaryDirectory() as tmp:
Path(tmp, "state.json").write_text("{not json")
eng, runner, clock = engine(tmp)
eng.client.fail = True
eng.tick()
check("mutation: loadError is kept when the inventory failed", eng.state["loadError"])
with tempfile.TemporaryDirectory() as tmp:
# lastCompleteRunAt means a report reached the store; a refused report must not advance it.
eng, runner, clock = engine(tmp)
real_publish = report.publish
report.publish = lambda *_: False
try:
eng.tick()
finally:
report.publish = real_publish
check("a refused publish does not advance lastCompleteRunAt", eng.state["lastCompleteRunAt"] is None)
clock.advance(hours=21)
eng.tick()
check("mutation: a successful publish does advance it", eng.state["lastCompleteRunAt"] is not None)
with tempfile.TemporaryDirectory() as tmp:
# One upstream string costs that string. Each of these used to freeze the whole report.
long_reason = "x" * 3000
eng, runner, clock = engine(tmp, acc="- id: CVE-2026-0010\n reason: " + long_reason + "\n expires: 2026-12-01\n")
runner.outcomes[NGINX] = {"SchemaVersion": 2, "Results": [{"Vulnerabilities": [
{"VulnerabilityID": "CVE-UP", "Severity": "HIGH", "PkgName": "p", "InstalledVersion": "1",
"PrimaryURL": "HTTPS://upper.example/x", "Title": "t\ud800"}]}]}
eng.tick()
check("hostile strings: the report still publishes", json.loads(Path(tmp, "report.json").read_text())["findings"] and report.validate(eng.report) == [])
up = [f for f in findings(eng, NGINX) if f["id"] == "CVE-UP"][0]
check("hostile strings: an uppercase scheme drops only the URL", up["url"] == "")
check("hostile strings: a lone surrogate is removed", "\ud800" not in up["title"])
check("hostile strings: an over-long acceptance is ignored, the finding stays open", [f for f in findings(eng, POSTGRES) if f["id"] == "CVE-2026-0010"][0]["status"] != "accepted")
check("hostile strings: an over-long acceptance degrades the run", eng.report["run"]["status"] == "degraded")
check("hostile strings: the degraded run is not recorded as complete", eng.state["lastCompleteRunAt"] is None)
with tempfile.TemporaryDirectory() as tmp:
# Isolation: values already in state, whatever put them there, cannot refuse the whole report.
eng, runner, clock = engine(tmp)
eng.tick()
clean = report.build(eng.state, [], [], clock())
check("mutation: a clean state is not reported as repaired", clean["run"]["status"] == "complete" and clean == report.build(eng.state, [], [], clock()))
record = eng.state["scans"][A1]
record["findings"][0].update(url="HTTP://x", title="t" * 900, severity="weird", cvss=99, package="p" * 900)
eng.state["database"] = {"updatedAt": "not-a-time", "version": "1"}
eng.state["remediation"] = [{"image": "a/b", "state": "pr-open", "url": "HTTP://x", "updatedAt": "2026-11-09T00:00:00Z"}]
target = next(i for i in eng.state["inventory"] if i["id"] == NIGHTLY)
target["id"] = "ghcr.io/" + "o" * 3000 + ":1"
target["image"] = "ghcr.io/" + "o" * 3000
eng.state["scans"][target["id"]] = eng.state["scans"].pop(NIGHTLY) # a by-tag unit is keyed by its id
built = report.build(eng.state, [], [], clock())
check("isolation: the repaired report is schema-valid", report.validate(built) == [])
check("isolation: a repair degrades the run", built["run"]["status"] == "degraded")
bad = [f for f in built["findings"] if f["targetId"] == NGINX and f["title"] == "t" * 512][0]
check("isolation: the bad finding survives with sanitised fields",
bad["url"] == "" and bad["severity"] == "UNKNOWN" and bad["cvss"] is None and len(bad["package"]) == 256)
check("isolation: every other finding is untouched", len(built["findings"]) == len(clean["findings"]))
check("isolation: a bad remediation loses only its url", built["remediation"][0]["url"] is None)
check("isolation: an unusable database is reported as unavailable", built["database"]["version"] == "unavailable")
long_ids = [t for t in built["targets"] if len(t["id"]) > 2000]
check("isolation: an over-long target stays listed", len(long_ids) == 1 and len(long_ids[0]["id"]) <= 2048)
check("isolation: findings follow the shortened target id",
any(f["targetId"] == long_ids[0]["id"] for f in built["findings"]))
class FakeProcess:
returncode = 0
def __init__(self, stdout):
self.stdout = stdout
class VersionRunner(trivy.Runner):
def __init__(self, stdout):
super().__init__("/nonexistent", "/nonexistent")
self.stdout = stdout
def _run(self, args, timeout):
return FakeProcess(self.stdout)
def db_info(updated, version):
return VersionRunner(json.dumps({"VulnerabilityDB": {"UpdatedAt": updated, "Version": version}})).db_info()
check("db_info rejects an unparseable timestamp", db_info("yesterday", "2") is None)
check("db_info rejects a non-string timestamp", db_info(5, "2") is None)
check("db_info bounds the version", len(db_info("2026-11-09T06:13:52.123456789Z", "v" * 500)["version"]) == 128)
check("mutation: db_info keeps a good database", db_info("2026-11-09T06:13:52Z", "2") == {"updatedAt": "2026-11-09T06:13:52Z", "version": "2"})
check("a credential failure's reason is bounded", len(trivy.classify_failure("r" * 3000, "401 unauthorized").reason) <= 1024)
for url, want in (("https://a.test/x", "https://a.test/x"), ("http://a.test/", "http://a.test/"), ("HTTPS://a.test/x", ""),
("HTTP://a.test/x", ""), ("Https://a.test/x", ""), ("javascript:alert(1)", ""), ("", "")):
check("safe_url " + repr(url), trivy.safe_url(url) == want)
check("every url safe_url keeps satisfies the report schema",
report.re.search(report.SCHEMA["properties"]["findings"]["items"]["properties"]["url"]["pattern"], trivy.safe_url("https://a.test/x")))
# Recorded Trivy output through the normaliser: untrusted text.
parsed = trivy.parse_results({"SchemaVersion": 2, "Results": [{"Vulnerabilities": [
{"VulnerabilityID": "CVE-1", "Severity": "weird", "PkgName": "p", "InstalledVersion": "1", "Title": "x\x07y",
"PrimaryURL": "javascript:alert(1)", "CVSS": {"nvd": {"V3Score": 11}}},
{"VulnerabilityID": "CVE-2", "Severity": "HIGH", "PkgName": "q", "InstalledVersion": "1", "FixedVersion": "2",
"PrimaryURL": "https://example.test/x", "CVSS": {"nvd": {"V3Score": 7.5}}},
]}]})
check("unknown severity becomes UNKNOWN", parsed[0]["severity"] == "UNKNOWN")
check("control characters are stripped from upstream text", "\x07" not in parsed[0]["title"])
check("non-http(s) advisory URL becomes empty", parsed[0]["url"] == "")
check("out-of-range CVSS is dropped", parsed[0]["cvss"] is None)
check("http(s) URL is kept", parsed[1]["url"] == "https://example.test/x")
check("a recorded report with no Results is zero findings", trivy.parse_results(fixture("trivy-empty.json")) == [])
try:
trivy.parse_results({"not": "a report"})
check("an unrecognised document is a failure, never zero findings", False)
except trivy.ScanFailure as exc:
check("an unrecognised document is a failure, never zero findings", exc.kind == "failed")
failure = trivy.classify_failure("ghcr.io", "GET failed: 401 Unauthorized")
check("401 is credential-required", failure.kind == "credential-required" and "ghcr.io" in failure.reason)
check("permission denied on local disk is a plain failure", trivy.classify_failure("r", "open /cache/x: permission denied").kind == "failed")
check("a pull timeout is failed, not credential-required", trivy.classify_failure("r", "context deadline exceeded").kind == "failed")
print("FAILED: " + str(len(FAILED)) if FAILED else "all checks passed")
sys.exit(1 if FAILED else 0)
@@ -0,0 +1,7 @@
#!/usr/bin/env bash
# Runs the fixture-driven assertions in check.py. See chainsaw-test.yaml for the requirement tags.
# What this does NOT prove: anything at runtime (real Trivy, real API, real browser). L2/L3 cover that.
set -euo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export PYTHONDONTWRITEBYTECODE=1
exec python3 "${HERE}/check.py"
@@ -0,0 +1,198 @@
{
"cronjobs": [
{
"metadata": {
"name": "nightly",
"namespace": "ops"
},
"spec": {
"jobTemplate": {
"spec": {
"template": {
"spec": {
"containers": [
{
"image": "ghcr.io/org/nightly:2.1",
"name": "job"
}
]
}
}
}
}
}
}
],
"jobs": [],
"nodes": [
{
"metadata": {
"name": "prod-1"
}
}
],
"pods": [
{
"metadata": {
"name": "web-1",
"namespace": "apps",
"ownerReferences": [
{
"controller": true,
"kind": "ReplicaSet",
"name": "web-5f"
}
]
},
"spec": {
"containers": [
{
"image": "nginx:1.27",
"name": "app"
}
]
},
"status": {
"containerStatuses": [
{
"imageID": "docker.io/library/nginx@sha256:a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1",
"name": "app"
}
],
"phase": "Running"
}
},
{
"metadata": {
"name": "api-1",
"namespace": "apps",
"ownerReferences": [
{
"controller": true,
"kind": "ReplicaSet",
"name": "api-77"
}
]
},
"spec": {
"containers": [
{
"image": "nginx:1.27",
"name": "server"
}
]
},
"status": {
"containerStatuses": [
{
"imageID": "docker.io/library/nginx@sha256:a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1",
"name": "server"
}
],
"phase": "Running"
}
},
{
"metadata": {
"name": "db-0",
"namespace": "data",
"ownerReferences": [
{
"controller": true,
"kind": "StatefulSet",
"name": "db"
}
]
},
"spec": {
"containers": [
{
"image": "postgres:16",
"name": "postgres"
}
]
},
"status": {
"containerStatuses": [
{
"imageID": "docker.io/library/postgres@sha256:d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4d4",
"name": "postgres"
}
],
"phase": "Running"
}
},
{
"metadata": {
"name": "priv-1",
"namespace": "apps",
"ownerReferences": [
{
"controller": true,
"kind": "ReplicaSet",
"name": "priv-1a"
}
]
},
"spec": {
"containers": [
{
"image": "ghcr.io/org/private:1",
"name": "priv"
}
]
},
"status": {
"containerStatuses": [
{
"imageID": "ghcr.io/org/private@sha256:e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5",
"name": "priv"
}
],
"phase": "Running"
}
}
],
"replicasets": [
{
"metadata": {
"name": "web-5f",
"namespace": "apps",
"ownerReferences": [
{
"controller": true,
"kind": "Deployment",
"name": "web"
}
]
}
},
{
"metadata": {
"name": "api-77",
"namespace": "apps",
"ownerReferences": [
{
"controller": true,
"kind": "Deployment",
"name": "api"
}
]
}
},
{
"metadata": {
"name": "priv-1a",
"namespace": "apps",
"ownerReferences": [
{
"controller": true,
"kind": "Deployment",
"name": "priv"
}
]
}
}
],
"vmis": []
}
@@ -0,0 +1,4 @@
{
"ArtifactName": "x",
"SchemaVersion": 2
}
@@ -0,0 +1,33 @@
{
"ArtifactName": "x",
"Results": [
{
"Target": "t",
"Vulnerabilities": [
{
"CVSS": {
"nvd": {
"V3Score": 9.8
}
},
"FixedVersion": "3.0.2",
"InstalledVersion": "1.0",
"PkgName": "openssl",
"PrimaryURL": "https://example.test/CVE-2026-0001",
"Severity": "CRITICAL",
"Title": "OpenSSL flaw",
"VulnerabilityID": "CVE-2026-0001"
},
{
"InstalledVersion": "1.0",
"PkgName": "zlib",
"PrimaryURL": "https://example.test/CVE-2026-0002",
"Severity": "LOW",
"Title": "t",
"VulnerabilityID": "CVE-2026-0002"
}
]
}
],
"SchemaVersion": 2
}
@@ -0,0 +1,19 @@
{
"ArtifactName": "x",
"Results": [
{
"Target": "t",
"Vulnerabilities": [
{
"InstalledVersion": "1.0",
"PkgName": "curl",
"PrimaryURL": "https://example.test/CVE-2026-0020",
"Severity": "MEDIUM",
"Title": "t",
"VulnerabilityID": "CVE-2026-0020"
}
]
}
],
"SchemaVersion": 2
}
@@ -0,0 +1,25 @@
{
"ArtifactName": "x",
"Results": [
{
"Target": "t",
"Vulnerabilities": [
{
"CVSS": {
"nvd": {
"V3Score": 7.5
}
},
"FixedVersion": "16.2",
"InstalledVersion": "1.0",
"PkgName": "libpq",
"PrimaryURL": "https://example.test/CVE-2026-0010",
"Severity": "HIGH",
"Title": "libpq flaw",
"VulnerabilityID": "CVE-2026-0010"
}
]
}
],
"SchemaVersion": 2
}
@@ -0,0 +1,32 @@
# =============================================================================
# L1 RENDERING — the bot ticks only qualifying approve rows (R-VULN-05, R-VULN-07)
# Spec: docs/17 "The triage bot" and its L1 acceptance gates. GIVEN a Renovate dashboard in the
# real issue #91 format plus a constructed Pending Approval section, WHEN the bot plans ticks,
# THEN only `approve` rows exact-matching an image with an unaccepted CRITICAL with a fix change,
# by exactly their marker, and every negative row stays byte-identical.
#
# render-check.sh is a pure offline check; this Chainsaw Test wraps the SAME script.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: vuln-triage-bot-approval-ticks
labels:
conformance.home.k3s/R-VULN-05: "true"
conformance.home.k3s/R-VULN-07: "true"
annotations:
conformance.home.k3s/requirements: "R-VULN-05,R-VULN-07"
conformance.home.k3s/layer: "L1"
conformance.home.k3s/safety-critical: "false"
# The script under test lives in .gitea/workflows/, which requirement-scoped selection does
# not attribute to R-VULN-*, so a change to it must still select this test.
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/intent: >-
A tick is applied only to an approve row for an image with a qualifying finding, and dry
runs, duplicate prevention and heartbeat ordering hold through the whole run.
spec:
steps:
- name: plan-and-assert-ticks
try:
- script:
content: bash ./render-check.sh
@@ -0,0 +1,363 @@
#!/usr/bin/env python3
"""L1: the bot ticks only qualifying approve rows, and a whole run behaves (R-VULN-05, R-VULN-07).
fixtures/renovate-dashboard.md follows issue #91's real format (`## Open` rebase rows, Detected
Dependencies) plus a constructed Pending Approval section, because the live dashboard has none.
Runs the real functions and the real run() over a fake Gitea and a fake scanner.
What this does NOT prove: that Renovate honours a tick; that the live dashboard keeps this format
across Renovate upgrades (a format change yields no tick rather than a wrong one); that Gitea's
API matches the fake; or anything about the runner's network path to the scanner.
"""
import copy
import importlib.util
import json
import sys
import urllib.parse
from datetime import datetime, timezone
from pathlib import Path
sys.dont_write_bytecode = True
HERE = Path(__file__).resolve().parent
ROOT = HERE.parents[3]
spec = importlib.util.spec_from_file_location("cve_triage", ROOT / ".gitea/workflows/scripts/cve_triage.py")
bot = importlib.util.module_from_spec(spec)
sys.modules["cve_triage"] = bot
spec.loader.exec_module(bot)
RAW = (HERE / "fixtures/report.json").read_bytes()
FIXTURE = json.loads(RAW)
BODY = (HERE / "fixtures/renovate-dashboard.md").read_text()
OPEN_PRS = json.loads((HERE / "fixtures/open-prs.json").read_text())
NOW = datetime(2026, 11, 9, 7, 15, tzinfo=timezone.utc)
GITEA = "https://gitea.example.test"
REPO = "o/r"
VULN = "http://vuln-scanner.vuln-scanner.svc.cluster.local:8081"
FALLBACK = "http://10.43.0.99:8081"
EXPECTED = {
"renovate/nginx-2.x",
"renovate/ghcr.io-example-app-2.x",
"renovate/example-group",
}
failures = []
def check(name, condition):
print(f"{'ok ' if condition else 'FAIL'} {name}")
if not condition:
failures.append(name)
def fails(function):
try:
function()
except AssertionError:
return True
return False
def raises(function):
try:
function()
except bot.BotError:
return True
return False
def planned(report_doc, body):
report = bot.parse_report(json.dumps(report_doc).encode())
rows = bot.parse_rows(body)
return bot.select_ticks(rows, bot.qualifying_images(report))
def assert_ticks(report_doc, body, expected):
"""The tick plan is exactly `expected`, and writing it changes exactly those markers."""
ticks = planned(report_doc, body)
assert {row.branch for row in ticks} == expected, "tick plan differs"
after = bot.apply_ticks(body, ticks)
want = body
for branch in expected:
line = f" - [ ] <!-- approve-branch={branch} -->"
assert want.count(line) == 1, "expected row not found exactly once"
want = want.replace(line, line.replace("[ ]", "[x]", 1))
assert after == want, "output differs from a marker-only edit"
check("exactly the three qualifying approve rows are ticked, and nothing else changes",
not fails(lambda: assert_ticks(FIXTURE, BODY, EXPECTED)))
after = bot.apply_ticks(BODY, planned(FIXTURE, BODY))
changed = [(a, b) for a, b in zip(BODY.split("\n"), after.split("\n")) if a != b]
check("three lines differ, same line count, same length", len(changed) == 3 and len(BODY) == len(after))
for label, needle in {
"rebase row of a qualifying image": "rebase-branch=renovate/ghcr.io-example-has-pr",
"approve row for a non-matching package": "approve-branch=renovate/ghcr.io-other-unrelated",
"approve row matching only by substring": "approve-branch=renovate/ghcr.io-foo-bar-extras",
"approve row for a HIGH-only image": "approve-branch=renovate/ghcr.io-example-high-only",
"approve row for an accepted-CRITICAL image": "approve-branch=renovate/ghcr.io-example-accepted-only",
"approve row for a CRITICAL with no fix": "approve-branch=renovate/ghcr.io-example-nofix",
"approve row with no container package": "approve-branch=renovate/mychart",
"approvePr row of a qualifying image": "approvePr-branch=renovate/ghcr.io-example-app-automerge",
"unschedule row of a qualifying image": "unschedule-branch=renovate/ghcr.io-example-pending-sched",
"decoy approve row below Detected Dependencies": "approve-branch=renovate/decoy",
}.items():
original = next(line for line in BODY.split("\n") if needle in line)
check(f"untouched: {label}", original in after.split("\n"))
check("untouched: the approve-all row", "- [ ] <!-- approve-all-pending-prs -->" in after)
check("untouched: the already ticked row keeps its mark",
" - [x] <!-- approve-branch=renovate/ghcr.io-example-already-ticked-2.x -->" in after)
# Non-vacuity: each mutation must make the exact assertion above fail.
def mutated_finding(image, **fields):
doc = copy.deepcopy(FIXTURE)
target = next(t for t in doc["targets"] if t["image"] == image)
finding = next(f for f in doc["findings"] if f["targetId"] == target["id"])
finding.update(fields)
return doc
for label, doc in {
"HIGH-only image raised to CRITICAL": mutated_finding("ghcr.io/example/high-only", severity="CRITICAL"),
"accepted CRITICAL un-accepted": mutated_finding("ghcr.io/example/accepted-only", status="fix-available"),
"qualifying finding accepted": mutated_finding("ghcr.io/example/app", status="accepted"),
"CRITICAL gains a fix": mutated_finding("ghcr.io/example/nofix", status="fix-available", fixedVersion="2"),
}.items():
check(f"non-vacuity: mutated fixture fails ({label})", fails(lambda: assert_ticks(doc, BODY, EXPECTED)))
mutated_body = BODY.replace("rebase-branch=renovate/ghcr.io-example-has-pr", "approve-branch=renovate/ghcr.io-example-has-pr")
check("non-vacuity: a rebase row rewritten as approve is then ticked",
fails(lambda: assert_ticks(FIXTURE, mutated_body, EXPECTED)))
real_match = bot.row_matches
bot.row_matches = lambda row, qualifying: any(q in p for p in row.packages for q in qualifying)
check("non-vacuity: a substring matcher fails the assertion", fails(lambda: assert_ticks(FIXTURE, BODY, EXPECTED)))
bot.row_matches = real_match
for raw, want in {
"docker.io/library/nginx": "nginx",
"library/nginx": "nginx",
"nginx:1.27": "nginx",
"Docker.IO/Pihole/Pihole": "pihole/pihole",
"ghcr.io/library/x": "ghcr.io/library/x",
"registry.example:5000/a/b:v1@sha256:" + "0" * 64: "registry.example:5000/a/b",
"bad image": "",
"": "",
}.items():
check(f"normalise_image({raw[:40]!r})", bot.normalise_image(raw) == want)
ticks = planned(FIXTURE, BODY)
moved = BODY.replace(" - [ ] <!-- approve-branch=renovate/nginx-2.x -->", " - [ ] <!-- rebase-branch=renovate/nginx-2.x -->")
check("apply_ticks refuses a row that stopped being approve", raises(lambda: bot.apply_ticks(moved, ticks)))
crlf = BODY.replace("\n", "\r\n")
crlf_after = bot.apply_ticks(crlf, planned(FIXTURE, crlf))
check("CRLF body keeps every line ending", crlf_after.count("\r\n") == crlf.count("\r\n") and crlf_after.count("[x]") == 4)
class FakeWorld:
"""A fake Gitea and scanner reached through the transport interface. Records every write."""
def __init__(self, renovate_body=BODY, report=RAW):
self.report = report
self.labels = []
self.issues = {91: {"number": 91, "title": "Dependency Dashboard", "state": "open", "labels": [], "body": renovate_body, "pull_request": None}}
self.next_issue = 200
self.writes = []
self.triage = None
self.fail = None
self.dns_down = set()
self.seen_hosts = []
def __call__(self, method, url, headers, body, max_bytes):
parts = urllib.parse.urlsplit(url)
self.seen_hosts.append(parts.netloc)
if parts.netloc in self.dns_down:
raise bot.DnsFailure()
if self.fail and self.fail[0] == method and self.fail[1] in parts.path:
return 500, b""
payload = json.loads(body) if body else None
if parts.netloc != "gitea.example.test":
if method == "GET" and parts.path == "/report.json":
return 200, self.report
if method == "POST" and parts.path == "/triage":
self.writes.append(("POST", "triage"))
self.triage = payload
return 204, b""
return 404, b""
path = parts.path.removeprefix(f"/api/v1/repos/{REPO}")
query = dict(urllib.parse.parse_qsl(parts.query))
page = int(query.get("page", "1"))
if method != "GET":
self.writes.append((method, path))
if method == "GET" and path == "/labels":
return 200, json.dumps(self.labels if page == 1 else []).encode()
if method == "POST" and path == "/labels":
label = {"id": 900 + len(self.labels), "name": payload["name"]}
self.labels.append(label)
return 201, json.dumps(label).encode()
if method == "GET" and path == "/pulls":
return 200, json.dumps(OPEN_PRS if page == 1 else []).encode()
if method == "GET" and path == "/issues":
items = list(self.issues.values())
if "labels" in query:
items = [i for i in items if query["labels"] in {x["name"] for x in i["labels"]}]
if "q" in query:
items = [i for i in items if query["q"] in i["title"]]
return 200, json.dumps(items if page == 1 else []).encode()
if method == "POST" and path == "/issues":
number = self.next_issue
self.next_issue += 1
names = [x for x in self.labels if x["id"] in payload["labels"]]
self.issues[number] = {"number": number, "title": payload["title"], "state": "open", "labels": names, "body": payload["body"], "pull_request": None}
return 201, json.dumps(self.issues[number]).encode()
if path.startswith("/issues/") and path.endswith("/labels") and method == "POST":
number = int(path.split("/")[2])
self.issues[number]["labels"] += [x for x in self.labels if x["id"] in payload["labels"]]
return 200, b"[]"
if path.startswith("/issues/"):
number = int(path.split("/")[2])
if method == "GET":
return 200, json.dumps(self.issues[number]).encode()
if method == "PATCH":
self.issues[number].update(payload)
return 201, json.dumps(self.issues[number]).encode()
return 404, b""
def execute(world, dry_run=False, fallback=FALLBACK):
cfg = {"dry_run": dry_run, "page_url": "https://cve.example.test", "renovate_issue": 91}
gitea = bot.GiteaClient(GITEA, REPO, "fixture-token", world)
scanner = bot.ScannerClient(VULN, fallback, world)
lines = []
summary = bot.run(cfg, gitea, scanner, NOW, lines.append)
return summary, "\n".join(lines)
def dashboards(world):
return [i for i in world.issues.values() if i["title"] == bot.DASHBOARD_TITLE]
# Dry run: reads everything, writes nothing, reports the plan as counts.
world = FakeWorld()
summary, text = execute(world, dry_run=True)
check("dry run performs no writes", world.writes == [] and world.triage is None and not dashboards(world))
check("dry run plans the three ticks and the issue", summary["ticks"] == 3 and summary["dashboard"]["issue_created"] == 1)
check("dry run leaves the dashboard body untouched", world.issues[91]["body"] == BODY)
NAMES = {"CVE-2026", "fixture-pkg", "fixture-ns-", "fixture-workload", "nginx", "ghcr.io"}
def leaks(text_value):
return [name for name in NAMES if name.lower() in text_value.lower()]
check("dry-run summary is counts only", not leaks(text))
# Live run.
world = FakeWorld()
summary, text = execute(world)
board = dashboards(world)
check("live run creates one label and one dashboard issue", len(world.labels) == 1 and len(board) == 1)
check("the dashboard issue carries only its own label", [x["name"] for x in board[0]["labels"]] == ["cve-dashboard"])
check("issue body and summary are counts only", not leaks(board[0]["body"]) and not leaks(text))
check("exactly one PATCH to issue #91", world.writes.count(("PATCH", "/issues/91")) == 1)
want_body = BODY
for branch in EXPECTED:
line = f" - [ ] <!-- approve-branch={branch} -->"
want_body = want_body.replace(line, line.replace("[ ]", "[x]", 1))
check("the PATCHed body is the marker-only edit", world.issues[91]["body"] == want_body)
check("the heartbeat is the last write", world.writes[-1] == ("POST", "triage"))
check("heartbeat shape is exact", set(world.triage) == {"heartbeatAt", "remediation"} and world.triage["heartbeatAt"] == "2026-11-09T07:15:00Z")
for entry in world.triage["remediation"]:
if set(entry) != {"image", "state", "url", "updatedAt"} or entry["state"] not in bot.REMEDIATION_STATES:
check("remediation entry shape", False)
states = {e["image"]: (e["state"], e["url"]) for e in world.triage["remediation"]}
for image, want in {
"docker.io/library/nginx": ("approval-requested", None),
"ghcr.io/example/app": ("approval-requested", None),
"ghcr.io/example/grouped": ("approval-requested", None),
"ghcr.io/example/already-ticked": ("approval-requested", None),
"ghcr.io/example/high-only": ("approval-requested", None),
"ghcr.io/example/has-pr": ("pr-open", f"{GITEA}/{REPO}/pulls/173"),
"ghcr.io/example/pr-by-title": ("pr-open", f"{GITEA}/{REPO}/pulls/174"),
"ghcr.io/example/tracked-clean": ("no-update-identified", None),
"ghcr.io/foo/bar": ("no-update-identified", None),
"ghcr.io/example/untracked": ("not-tracked", None),
"ghcr.io/example/pending-sched": ("not-tracked", None),
}.items():
check(f"remediation for {image}", states.get(image) == want)
check("one remediation entry per image", len(states) == len(FIXTURE["targets"]))
# A second run updates the same issue.
before = len(world.issues)
second = copy.deepcopy(FIXTURE)
second["targets"][-1]["state"] = "stale"
world.report = json.dumps(second).encode()
summary, _ = execute(world)
check("a second run creates no duplicate", len(dashboards(world)) == 1 and len(world.issues) == before)
check("a second run updates the issue", summary["dashboard"]["issue_updated"] == 1 and "| stale | 1 |" in dashboards(world)[0]["body"])
check("a second run has nothing left to tick", summary["ticks"] == 0)
# The dashboard issue is found even without its label, and a closed one is reopened.
world = FakeWorld()
world.labels.append({"id": 900, "name": "cve-dashboard"})
world.issues[150] = {"number": 150, "title": "CVE Dashboard", "state": "closed", "labels": [], "body": "old", "pull_request": None}
summary, _ = execute(world)
check("an unlabelled closed dashboard is reused, relabelled and reopened",
len(dashboards(world)) == 1 and world.issues[150]["state"] == "open"
and [x["name"] for x in world.issues[150]["labels"]] == ["cve-dashboard"] and summary["dashboard"]["issue_created"] == 0)
world = FakeWorld()
world.labels.append({"id": 900, "name": "cve-dashboard"})
for number in (150, 151):
world.issues[number] = {"number": number, "title": "CVE Dashboard", "state": "open", "labels": [world.labels[0]], "body": "x", "pull_request": None}
check("two dashboard issues fail the run", raises(lambda: execute(world)))
check("two dashboard issues write nothing", world.writes == [])
# Failures end the run before the heartbeat.
world = FakeWorld()
world.fail = ("PATCH", "/issues/91")
check("a failed tick PATCH fails the run", raises(lambda: execute(world)))
check("a failed run posts no heartbeat", world.triage is None)
world = FakeWorld(report=b"{not json")
check("an unreadable report fails the run", raises(lambda: execute(world)))
check("an unreadable report writes nothing", world.writes == [])
world = FakeWorld()
world.issues[91]["title"] = "Something else"
check("issue 91 being something else fails the run", raises(lambda: execute(world)))
world = FakeWorld()
world.issues[91]["state"] = "closed"
check("a closed Renovate dashboard fails the run", raises(lambda: execute(world)))
# DNS fallback, and nothing but DNS failure selects it.
world = FakeWorld()
world.dns_down.add("vuln-scanner.vuln-scanner.svc.cluster.local:8081")
summary, _ = execute(world)
check("DNS failure uses the fallback for report and heartbeat", summary["fallback_used"] and world.triage is not None)
check("the fallback was actually contacted", "10.43.0.99:8081" in world.seen_hosts)
world = FakeWorld()
world.dns_down.add("vuln-scanner.vuln-scanner.svc.cluster.local:8081")
check("DNS failure with no fallback fails the run", raises(lambda: execute(world, fallback="")))
def refused(method, url, headers, body, max_bytes):
raise bot.BotError("GET /report.json: ConnectionRefusedError")
check("a refused connection fails the run and does not fall back",
raises(lambda: bot.ScannerClient(VULN, FALLBACK, refused).report()))
for label, env, expected_ok in (
("missing token", {}, False),
("present token", {"RENOVATE_TOKEN": "x"}, True),
("bad DRY_RUN", {"RENOVATE_TOKEN": "x", "DRY_RUN": "maybe"}, False),
("non-http API URL", {"RENOVATE_TOKEN": "x", "VULN_API_URL": "file:///etc/passwd"}, False),
):
try:
bot.load_config(env)
ok = True
except bot.BotError:
ok = False
check(f"load_config: {label}", ok is expected_ok)
print(f"{len(failures)} failure(s)")
sys.exit(1 if failures else 0)
@@ -0,0 +1,5 @@
[
{"number": 173, "title": "chore(deps): Update ghcr.io/example/has-pr Docker tag to v2", "head": {"ref": "renovate/ghcr.io-example-has-pr-2.x"}, "html_url": "https://gitea.example.test/o/r/pulls/173"},
{"number": 174, "title": "chore(deps): Update ghcr.io/example/pr-by-title Docker tag to v2", "head": {"ref": "renovate/ghcr.io-example-pr-by-title-2.x"}, "html_url": "https://gitea.example.test/o/r/pulls/174"},
{"number": 160, "title": "WIP: chore(spec): not Renovate work", "head": {"ref": "chore/adopt-spec-sync"}, "html_url": "https://gitea.example.test/o/r/pulls/160"}
]
@@ -0,0 +1,71 @@
This issue lists Renovate updates and detected dependencies. Read the [Dependency Dashboard](https://docs.renovatebot.com/key-concepts/dashboard/) docs to learn more.
## Pending Approval
The following branches are pending approval. To create them, click on a checkbox below.
- [ ] <!-- approve-branch=renovate/nginx-2.x -->chore(deps): Update nginx Docker tag to v2
- [ ] <!-- approve-branch=renovate/ghcr.io-example-app-2.x -->chore(deps): Update ghcr.io/example/app Docker tag to v2
- [ ] <!-- approve-branch=renovate/example-group -->chore(deps): Update example group (`ghcr.io/example/grouped`, `ghcr.io/example/other`)
- [ ] <!-- approve-branch=renovate/ghcr.io-foo-bar-extras-2.x -->chore(deps): Update ghcr.io/foo/bar-extras Docker tag to v2
- [ ] <!-- approve-branch=renovate/ghcr.io-other-unrelated-2.x -->chore(deps): Update ghcr.io/other/unrelated Docker tag to v2
- [ ] <!-- approve-branch=renovate/ghcr.io-example-high-only-2.x -->chore(deps): Update ghcr.io/example/high-only Docker tag to v2
- [ ] <!-- approve-branch=renovate/ghcr.io-example-accepted-only-2.x -->chore(deps): Update ghcr.io/example/accepted-only Docker tag to v2
- [ ] <!-- approve-branch=renovate/ghcr.io-example-nofix-2.x -->chore(deps): Update ghcr.io/example/nofix Docker tag to v2
- [ ] <!-- approve-branch=renovate/mychart-2.x -->chore(deps): Update Helm release mychart to v2
- [x] <!-- approve-branch=renovate/ghcr.io-example-already-ticked-2.x -->chore(deps): Update ghcr.io/example/already-ticked Docker tag to v2
- [ ] <!-- approve-all-pending-prs -->🔐 **Create all pending approval PRs at once** 🔐
## Awaiting Schedule
The following updates are awaiting their schedule. To get an update now, click on a checkbox below.
- [ ] <!-- unschedule-branch=renovate/ghcr.io-example-pending-sched-2.x -->chore(deps): Update ghcr.io/example/pending-sched Docker tag to v2
## Pending Branch Automerge
The following branches have automerge enabled. To abort a branch automerge, click on a checkbox below.
- [ ] <!-- approvePr-branch=renovate/ghcr.io-example-app-automerge -->chore(deps): Update ghcr.io/example/app Docker tag to v2
## Open
The following updates have all been created. To force a retry/rebase of any, click on a checkbox below.
- [ ] <!-- rebase-branch=renovate/gitea.home.keithjackson.dev-home-network-toolbox-main -->[chore(deps): Update gitea.home.keithjackson.dev/home-network/toolbox:main Docker digest to 83a3501](pulls/161)
- [ ] <!-- rebase-branch=renovate/kube-prometheus-stack-93.x -->[chore(deps): Update Helm release kube-prometheus-stack to v93](pulls/164)
- [ ] <!-- rebase-branch=renovate/ghcr.io-example-has-pr-2.x -->[chore(deps): Update ghcr.io/example/has-pr Docker tag to v2](pulls/173)
- [ ] <!-- rebase-all-open-prs -->**Click on this checkbox to rebase all open PRs at once**
## Detected Dependencies
<details><summary>flux (13)</summary>
<blockquote>
<details><summary>platform/cert-manager/helmrelease.yaml (1)</summary>
- `cert-manager >=1.21.0 <1.22.0` → [Updates: `>=1.21.3 <1.22.0`]
</details>
</blockquote>
</details>
<details><summary>kubernetes (69)</summary>
<blockquote>
<details><summary>apps/wave-0/jackett.yaml (4)</summary>
- `lscr.io/linuxserver/jackett v0.24.3025-ls29` → [Updates: `v0.24.3030-ls30`]
- `quay.io/oauth2-proxy/oauth2-proxy v7.15.6`
- `NetworkPolicy networking.k8s.io/v1`
- `ghcr.io/example/tracked-clean 1.0`
- `ghcr.io/foo/bar 1.0`
- `ghcr.io/example/pending-sched 1.0` → [Updates: `2.0`]
- `ghcr.io/example/app 1.0` → [Updates: `2.0`]
- [ ] <!-- approve-branch=renovate/decoy-below-detected -->chore(deps): Update nginx Docker tag to v9
</details>
</blockquote>
</details>
@@ -0,0 +1,425 @@
{
"schemaVersion": 1,
"generatedAt": "2026-11-09T06:00:00Z",
"database": {
"updatedAt": "2026-11-09T01:00:00Z",
"version": "fixture-db"
},
"run": {
"startedAt": "2026-11-09T05:00:00Z",
"finishedAt": "2026-11-09T05:30:00Z",
"status": "complete"
},
"targets": [
{
"id": "docker.io/library/nginx@sha256:0000000000000000000000000000000000000000000000000000000000000000",
"type": "image",
"image": "docker.io/library/nginx",
"digest": "sha256:0000000000000000000000000000000000000000000000000000000000000000",
"state": "scanned",
"reason": null,
"lastScannedAt": "2026-11-09T05:00:00Z",
"workloads": [
{
"namespace": "fixture-ns-0",
"kind": "Deployment",
"name": "fixture-workload-0",
"containers": [
"main"
]
}
]
},
{
"id": "ghcr.io/example/app@sha256:0000000000000000000000000000000000000000000000000000000000000001",
"type": "image",
"image": "ghcr.io/example/app",
"digest": "sha256:0000000000000000000000000000000000000000000000000000000000000001",
"state": "scanned",
"reason": null,
"lastScannedAt": "2026-11-09T05:00:00Z",
"workloads": [
{
"namespace": "fixture-ns-1",
"kind": "Deployment",
"name": "fixture-workload-1",
"containers": [
"main"
]
}
]
},
{
"id": "ghcr.io/example/grouped@sha256:0000000000000000000000000000000000000000000000000000000000000002",
"type": "image",
"image": "ghcr.io/example/grouped",
"digest": "sha256:0000000000000000000000000000000000000000000000000000000000000002",
"state": "scanned",
"reason": null,
"lastScannedAt": "2026-11-09T05:00:00Z",
"workloads": [
{
"namespace": "fixture-ns-2",
"kind": "Deployment",
"name": "fixture-workload-2",
"containers": [
"main"
]
}
]
},
{
"id": "ghcr.io/foo/bar@sha256:0000000000000000000000000000000000000000000000000000000000000003",
"type": "image",
"image": "ghcr.io/foo/bar",
"digest": "sha256:0000000000000000000000000000000000000000000000000000000000000003",
"state": "scanned",
"reason": null,
"lastScannedAt": "2026-11-09T05:00:00Z",
"workloads": [
{
"namespace": "fixture-ns-3",
"kind": "Deployment",
"name": "fixture-workload-3",
"containers": [
"main"
]
}
]
},
{
"id": "ghcr.io/example/has-pr@sha256:0000000000000000000000000000000000000000000000000000000000000004",
"type": "image",
"image": "ghcr.io/example/has-pr",
"digest": "sha256:0000000000000000000000000000000000000000000000000000000000000004",
"state": "scanned",
"reason": null,
"lastScannedAt": "2026-11-09T05:00:00Z",
"workloads": [
{
"namespace": "fixture-ns-4",
"kind": "Deployment",
"name": "fixture-workload-4",
"containers": [
"main"
]
}
]
},
{
"id": "ghcr.io/example/pr-by-title@sha256:0000000000000000000000000000000000000000000000000000000000000005",
"type": "image",
"image": "ghcr.io/example/pr-by-title",
"digest": "sha256:0000000000000000000000000000000000000000000000000000000000000005",
"state": "scanned",
"reason": null,
"lastScannedAt": "2026-11-09T05:00:00Z",
"workloads": [
{
"namespace": "fixture-ns-5",
"kind": "Deployment",
"name": "fixture-workload-5",
"containers": [
"main"
]
}
]
},
{
"id": "ghcr.io/example/already-ticked@sha256:0000000000000000000000000000000000000000000000000000000000000006",
"type": "image",
"image": "ghcr.io/example/already-ticked",
"digest": "sha256:0000000000000000000000000000000000000000000000000000000000000006",
"state": "scanned",
"reason": null,
"lastScannedAt": "2026-11-09T05:00:00Z",
"workloads": [
{
"namespace": "fixture-ns-6",
"kind": "Deployment",
"name": "fixture-workload-6",
"containers": [
"main"
]
}
]
},
{
"id": "ghcr.io/example/untracked@sha256:0000000000000000000000000000000000000000000000000000000000000007",
"type": "image",
"image": "ghcr.io/example/untracked",
"digest": "sha256:0000000000000000000000000000000000000000000000000000000000000007",
"state": "scanned",
"reason": null,
"lastScannedAt": "2026-11-09T05:00:00Z",
"workloads": [
{
"namespace": "fixture-ns-7",
"kind": "Deployment",
"name": "fixture-workload-7",
"containers": [
"main"
]
}
]
},
{
"id": "ghcr.io/example/pending-sched@sha256:0000000000000000000000000000000000000000000000000000000000000008",
"type": "image",
"image": "ghcr.io/example/pending-sched",
"digest": "sha256:0000000000000000000000000000000000000000000000000000000000000008",
"state": "scanned",
"reason": null,
"lastScannedAt": "2026-11-09T05:00:00Z",
"workloads": [
{
"namespace": "fixture-ns-8",
"kind": "Deployment",
"name": "fixture-workload-8",
"containers": [
"main"
]
}
]
},
{
"id": "ghcr.io/example/high-only@sha256:0000000000000000000000000000000000000000000000000000000000000009",
"type": "image",
"image": "ghcr.io/example/high-only",
"digest": "sha256:0000000000000000000000000000000000000000000000000000000000000009",
"state": "scanned",
"reason": null,
"lastScannedAt": "2026-11-09T05:00:00Z",
"workloads": [
{
"namespace": "fixture-ns-9",
"kind": "Deployment",
"name": "fixture-workload-9",
"containers": [
"main"
]
}
]
},
{
"id": "ghcr.io/example/accepted-only@sha256:000000000000000000000000000000000000000000000000000000000000000a",
"type": "image",
"image": "ghcr.io/example/accepted-only",
"digest": "sha256:000000000000000000000000000000000000000000000000000000000000000a",
"state": "scanned",
"reason": null,
"lastScannedAt": "2026-11-09T05:00:00Z",
"workloads": [
{
"namespace": "fixture-ns-10",
"kind": "Deployment",
"name": "fixture-workload-10",
"containers": [
"main"
]
}
]
},
{
"id": "ghcr.io/example/nofix@sha256:000000000000000000000000000000000000000000000000000000000000000b",
"type": "image",
"image": "ghcr.io/example/nofix",
"digest": "sha256:000000000000000000000000000000000000000000000000000000000000000b",
"state": "scanned",
"reason": null,
"lastScannedAt": "2026-11-09T05:00:00Z",
"workloads": [
{
"namespace": "fixture-ns-11",
"kind": "Deployment",
"name": "fixture-workload-11",
"containers": [
"main"
]
}
]
},
{
"id": "ghcr.io/example/tracked-clean@sha256:000000000000000000000000000000000000000000000000000000000000000c",
"type": "image",
"image": "ghcr.io/example/tracked-clean",
"digest": "sha256:000000000000000000000000000000000000000000000000000000000000000c",
"state": "scanned",
"reason": null,
"lastScannedAt": "2026-11-09T05:00:00Z",
"workloads": [
{
"namespace": "fixture-ns-12",
"kind": "Deployment",
"name": "fixture-workload-12",
"containers": [
"main"
]
}
]
}
],
"findings": [
{
"id": "CVE-2026-1000",
"targetId": "docker.io/library/nginx@sha256:0000000000000000000000000000000000000000000000000000000000000000",
"severity": "CRITICAL",
"cvss": 9.0,
"package": "fixture-pkg-0",
"installedVersion": "1.0",
"fixedVersion": "9.9.9",
"title": "fixture",
"url": "https://advisory.example/x",
"firstSeenAt": "2026-11-01T00:00:00Z",
"status": "fix-available"
},
{
"id": "CVE-2026-1010",
"targetId": "ghcr.io/example/app@sha256:0000000000000000000000000000000000000000000000000000000000000001",
"severity": "CRITICAL",
"cvss": 9.0,
"package": "fixture-pkg-1",
"installedVersion": "1.0",
"fixedVersion": "9.9.9",
"title": "fixture",
"url": "https://advisory.example/x",
"firstSeenAt": "2026-11-01T00:00:00Z",
"status": "fix-available"
},
{
"id": "CVE-2026-1020",
"targetId": "ghcr.io/example/grouped@sha256:0000000000000000000000000000000000000000000000000000000000000002",
"severity": "CRITICAL",
"cvss": 9.0,
"package": "fixture-pkg-2",
"installedVersion": "1.0",
"fixedVersion": "9.9.9",
"title": "fixture",
"url": "https://advisory.example/x",
"firstSeenAt": "2026-11-01T00:00:00Z",
"status": "fix-available"
},
{
"id": "CVE-2026-1030",
"targetId": "ghcr.io/foo/bar@sha256:0000000000000000000000000000000000000000000000000000000000000003",
"severity": "CRITICAL",
"cvss": 9.0,
"package": "fixture-pkg-3",
"installedVersion": "1.0",
"fixedVersion": "9.9.9",
"title": "fixture",
"url": "https://advisory.example/x",
"firstSeenAt": "2026-11-01T00:00:00Z",
"status": "fix-available"
},
{
"id": "CVE-2026-1040",
"targetId": "ghcr.io/example/has-pr@sha256:0000000000000000000000000000000000000000000000000000000000000004",
"severity": "CRITICAL",
"cvss": 9.0,
"package": "fixture-pkg-4",
"installedVersion": "1.0",
"fixedVersion": "9.9.9",
"title": "fixture",
"url": "https://advisory.example/x",
"firstSeenAt": "2026-11-01T00:00:00Z",
"status": "fix-available"
},
{
"id": "CVE-2026-1050",
"targetId": "ghcr.io/example/pr-by-title@sha256:0000000000000000000000000000000000000000000000000000000000000005",
"severity": "CRITICAL",
"cvss": 9.0,
"package": "fixture-pkg-5",
"installedVersion": "1.0",
"fixedVersion": "9.9.9",
"title": "fixture",
"url": "https://advisory.example/x",
"firstSeenAt": "2026-11-01T00:00:00Z",
"status": "fix-available"
},
{
"id": "CVE-2026-1060",
"targetId": "ghcr.io/example/already-ticked@sha256:0000000000000000000000000000000000000000000000000000000000000006",
"severity": "CRITICAL",
"cvss": 9.0,
"package": "fixture-pkg-6",
"installedVersion": "1.0",
"fixedVersion": "9.9.9",
"title": "fixture",
"url": "https://advisory.example/x",
"firstSeenAt": "2026-11-01T00:00:00Z",
"status": "fix-available"
},
{
"id": "CVE-2026-1070",
"targetId": "ghcr.io/example/untracked@sha256:0000000000000000000000000000000000000000000000000000000000000007",
"severity": "CRITICAL",
"cvss": 9.0,
"package": "fixture-pkg-7",
"installedVersion": "1.0",
"fixedVersion": "9.9.9",
"title": "fixture",
"url": "https://advisory.example/x",
"firstSeenAt": "2026-11-01T00:00:00Z",
"status": "fix-available"
},
{
"id": "CVE-2026-1080",
"targetId": "ghcr.io/example/pending-sched@sha256:0000000000000000000000000000000000000000000000000000000000000008",
"severity": "CRITICAL",
"cvss": 9.0,
"package": "fixture-pkg-8",
"installedVersion": "1.0",
"fixedVersion": "9.9.9",
"title": "fixture",
"url": "https://advisory.example/x",
"firstSeenAt": "2026-11-01T00:00:00Z",
"status": "fix-available"
},
{
"id": "CVE-2026-1090",
"targetId": "ghcr.io/example/high-only@sha256:0000000000000000000000000000000000000000000000000000000000000009",
"severity": "HIGH",
"cvss": 9.0,
"package": "fixture-pkg-9",
"installedVersion": "1.0",
"fixedVersion": "9.9.9",
"title": "fixture",
"url": "https://advisory.example/x",
"firstSeenAt": "2026-11-01T00:00:00Z",
"status": "fix-available"
},
{
"id": "CVE-2026-1100",
"targetId": "ghcr.io/example/accepted-only@sha256:000000000000000000000000000000000000000000000000000000000000000a",
"severity": "CRITICAL",
"cvss": 9.0,
"package": "fixture-pkg-10",
"installedVersion": "1.0",
"fixedVersion": "9.9.9",
"title": "fixture",
"url": "https://advisory.example/x",
"firstSeenAt": "2026-11-01T00:00:00Z",
"status": "accepted",
"acceptance": {
"reason": "fixture",
"expires": "2027-01-01"
}
},
{
"id": "CVE-2026-1110",
"targetId": "ghcr.io/example/nofix@sha256:000000000000000000000000000000000000000000000000000000000000000b",
"severity": "CRITICAL",
"cvss": 9.0,
"package": "fixture-pkg-11",
"installedVersion": "1.0",
"fixedVersion": null,
"title": "fixture",
"url": "https://advisory.example/x",
"firstSeenAt": "2026-11-01T00:00:00Z",
"status": "open"
}
],
"remediation": []
}
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
# L1 offline check for the CVE triage bot: fixture-driven, no cluster, no network. Assertions and
# their limits are in check.py. rc 0 pass, 1 fail.
set -euo pipefail
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PYTHONDONTWRITEBYTECODE=1 exec python3 "${here}/check.py"
@@ -0,0 +1,31 @@
# =============================================================================
# L1 RENDERING — CVE Dashboard issue carries aggregates only (R-VULN-05, R-VULN-07)
# Spec: docs/17 "The triage bot". GIVEN a report fixture with hostile and distinctive strings,
# WHEN the bot renders the dashboard issue, THEN no CVE id, package, image, workload, namespace or
# other report string appears, the counts are exact, and a deliberately leaking body is caught.
#
# render-check.sh is a pure offline check; this Chainsaw Test wraps the SAME script.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: vuln-triage-bot-dashboard-aggregates
labels:
conformance.home.k3s/R-VULN-05: "true"
conformance.home.k3s/R-VULN-07: "true"
annotations:
conformance.home.k3s/requirements: "R-VULN-05,R-VULN-07"
conformance.home.k3s/layer: "L1"
conformance.home.k3s/safety-critical: "false"
# The script under test lives in .gitea/workflows/, which requirement-scoped selection does
# not attribute to R-VULN-*, so a change to it must still select this test.
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/intent: >-
A dashboard issue rendered from a fixture contains counts and timestamps only, never a CVE id,
package, image or workload name.
spec:
steps:
- name: render-and-assert-aggregates-only
try:
- script:
content: bash ./render-check.sh
@@ -0,0 +1,162 @@
#!/usr/bin/env python3
"""L1: the CVE Dashboard issue is aggregates only (R-VULN-05, R-VULN-07).
Exercises the real renderer in .gitea/workflows/scripts/cve_triage.py over fixtures/report.json,
whose every name is distinctive so a leak cannot be confused with template text.
What this does NOT prove: that Gitea accepts the issue, or that the issue is the one the live bot
wrote. Those need a real run and are not exercised by any L1 test.
"""
import copy
import importlib.util
import json
import re
import sys
from datetime import datetime, timedelta, timezone
from pathlib import Path
sys.dont_write_bytecode = True
HERE = Path(__file__).resolve().parent
ROOT = HERE.parents[3]
spec = importlib.util.spec_from_file_location("cve_triage", ROOT / ".gitea/workflows/scripts/cve_triage.py")
bot = importlib.util.module_from_spec(spec)
sys.modules["cve_triage"] = bot
spec.loader.exec_module(bot)
RAW = (HERE / "fixtures/report.json").read_bytes()
FIXTURE = json.loads(RAW)
NOW = datetime(2026, 11, 9, 7, 15, tzinfo=timezone.utc)
PAGE = "https://cve.example.test"
failures = []
def check(name, condition):
print(f"{'ok ' if condition else 'FAIL'} {name}")
if not condition:
failures.append(name)
def sensitive_names(doc):
"""Every string in the fixture that must never reach the issue."""
names = set()
for target in doc["targets"]:
for field in ("id", "image", "digest", "reason"):
if target.get(field):
names.add(target[field])
if target.get("image"):
names.add(bot.normalise_image(target["image"]))
names.add(target["image"].rsplit("/", 1)[-1])
for workload in target["workloads"]:
names.update((workload["namespace"], workload["name"]))
names.update(workload["containers"])
for finding in doc["findings"]:
for field in ("id", "package", "installedVersion", "fixedVersion", "title", "url"):
if finding.get(field):
names.add(finding[field])
if finding.get("acceptance"):
names.add(finding["acceptance"]["reason"])
names.add(doc["database"]["version"])
return {name for name in names if name}
def assert_no_names(body, names):
lowered = body.lower()
leaked = sorted(name for name in names if name.lower() in lowered)
assert not leaked, f"{len(leaked)} fixture names leaked into the issue body"
def rendered(raw=RAW, now=NOW):
return bot.render_dashboard(bot.aggregate(bot.parse_report(raw)), PAGE, now)
def fails(function):
try:
function()
except AssertionError:
return True
return False
def rejected(doc):
try:
bot.parse_report(json.dumps(doc).encode())
except bot.BotError:
return True
return False
body = rendered()
names = sensitive_names(FIXTURE)
check("fixture carries a non-trivial set of names", len(names) >= 25)
check("rendered body contains no fixture name", not fails(lambda: assert_no_names(body, names)))
check("body has no CVE or GHSA identifier at all", not re.search(r"CVE-\d|GHSA-", body, re.I))
check("hostile title's markup is absent", "onerror" not in body and "evil.example" not in body and "@keith" not in body)
# Non-vacuity: the assertion above must detect each class of leak.
for label, leak in {
"CVE id": "CVE-2026-11111",
"package": "fixture-libcrit",
"image": "ghcr.io/fixture-org/fixture-api",
"workload": "fixture-workload-db",
"namespace": "fixture-ns-gamma",
}.items():
check(f"non-vacuity: a leaked {label} is caught", fails(lambda: assert_no_names(body + leak, names)))
for row in (
"| CRITICAL | 1 | 1 | 1 |",
"| HIGH | 0 | 1 | 0 |",
"| MEDIUM | 0 | 0 | 0 |",
"| LOW | 1 | 0 | 0 |",
"| UNKNOWN | 0 | 0 | 0 |",
"| scanned | 1 |",
"| pending | 0 |",
"| stale | 1 |",
"| failed | 0 |",
"| credential-required | 1 |",
"| by-tag | 0 |",
"| not-covered | 1 |",
"Unaccepted CRITICAL with a fix: **1**",
"4 targets.",
"| Report generated | 2026-11-09T06:00:00Z |",
f"[CVE page]({PAGE})",
):
check(f"body states {row!r}", row in body)
check("a fresh report carries no warning", "WARNING" not in body)
mutated = copy.deepcopy(FIXTURE)
extra = copy.deepcopy(mutated["findings"][0])
extra["id"] = "CVE-2026-55555"
mutated["findings"].append(extra)
changed = rendered(json.dumps(mutated).encode())
check("non-vacuity: a mutated fixture changes the counts", "| CRITICAL | 1 | 2 | 1 |" in changed and changed != body)
check("stale report warns", "older than 26 hours" in rendered(now=NOW + timedelta(hours=30)))
degraded = copy.deepcopy(FIXTURE)
degraded["run"]["status"] = "degraded"
check("degraded run warns", "did not complete cleanly" in rendered(json.dumps(degraded).encode()))
old_db = copy.deepcopy(FIXTURE)
old_db["database"]["updatedAt"] = "2026-11-01T00:00:00Z"
check("stale database warns", "database is older" in rendered(json.dumps(old_db).encode()))
for label, mutate in {
"wrong schemaVersion": lambda d: d.update(schemaVersion=2),
"unknown severity": lambda d: d["findings"][0].update(severity="SEVERE"),
"unknown status": lambda d: d["findings"][0].update(status="ignored"),
"unknown target state": lambda d: d["targets"][0].update(state="clean"),
"finding on an unknown target": lambda d: d["findings"][0].update(targetId="nope"),
"bad timestamp": lambda d: d.update(generatedAt="yesterday <b>"),
"invalid image": lambda d: d["targets"][0].update(image="bad image <x>"),
}.items():
doc = copy.deepcopy(FIXTURE)
mutate(doc)
check(f"parse rejects {label}", rejected(doc))
check("dashboard label is not one the PM acts on", bot.DASHBOARD_LABEL not in bot.FORBIDDEN_LABELS)
configmap = (ROOT / "platform/pm-agent/configmap.yaml").read_text()
match = re.search(r'^\s*PM_REQUIRE_LABEL:\s*"([^"]*)"', configmap, re.M)
check("PM_REQUIRE_LABEL is readable", match is not None)
check("the PM's label is in FORBIDDEN_LABELS and is not ours",
match is not None and match.group(1) in bot.FORBIDDEN_LABELS and match.group(1) != bot.DASHBOARD_LABEL)
print(f"{len(failures)} failure(s)")
sys.exit(1 if failures else 0)
@@ -0,0 +1,20 @@
{
"schemaVersion": 1,
"generatedAt": "2026-11-09T06:00:00Z",
"database": {"updatedAt": "2026-11-09T01:00:00Z", "version": "fixture-db-v2-20261109"},
"run": {"startedAt": "2026-11-09T05:00:00Z", "finishedAt": "2026-11-09T05:30:00Z", "status": "complete"},
"targets": [
{"id": "ghcr.io/fixture-org/fixture-api@sha256:1111111111111111111111111111111111111111111111111111111111111111", "type": "image", "image": "ghcr.io/fixture-org/fixture-api", "digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", "state": "scanned", "reason": null, "lastScannedAt": "2026-11-09T05:10:00Z", "workloads": [{"namespace": "fixture-ns-alpha", "kind": "Deployment", "name": "fixture-workload-api", "containers": ["fixture-container-api"]}]},
{"id": "docker.io/library/fixturedb@sha256:2222222222222222222222222222222222222222222222222222222222222222", "type": "image", "image": "docker.io/library/fixturedb", "digest": "sha256:2222222222222222222222222222222222222222222222222222222222222222", "state": "stale", "reason": "fixture-reason-stale-text", "lastScannedAt": "2026-11-07T05:10:00Z", "workloads": [{"namespace": "fixture-ns-beta", "kind": "StatefulSet", "name": "fixture-workload-db", "containers": ["fixture-container-db"]}]},
{"id": "fixture.example:5000/priv/fixture-private:v1", "type": "image", "image": "fixture.example:5000/priv/fixture-private", "digest": null, "state": "credential-required", "reason": "fixture.example:5000 refused anonymous access", "lastScannedAt": null, "workloads": [{"namespace": "fixture-ns-gamma", "kind": "CronJob", "name": "fixture-workload-cron", "containers": ["fixture-container-cron"]}]},
{"id": "vm/fixture-ns-vm/fixture-vm-name", "type": "vm", "image": null, "digest": null, "state": "not-covered", "reason": null, "lastScannedAt": null, "workloads": [{"namespace": "fixture-ns-vm", "kind": "VirtualMachineInstance", "name": "fixture-vm-name", "containers": []}]}
],
"findings": [
{"id": "CVE-2026-11111", "targetId": "ghcr.io/fixture-org/fixture-api@sha256:1111111111111111111111111111111111111111111111111111111111111111", "severity": "CRITICAL", "cvss": 9.8, "package": "fixture-libcrit", "installedVersion": "1.0.1-installed", "fixedVersion": "1.0.2-fixed", "title": "<img src=x onerror=alert(1)> @keith [click](http://evil.example/hook) | injected |", "url": "https://advisory.example/CVE-2026-11111", "firstSeenAt": "2026-11-01T00:00:00Z", "status": "fix-available"},
{"id": "GHSA-fixt-ure0-0001", "targetId": "ghcr.io/fixture-org/fixture-api@sha256:1111111111111111111111111111111111111111111111111111111111111111", "severity": "CRITICAL", "cvss": 9.1, "package": "fixture-libopen", "installedVersion": "2.0.0-installed", "fixedVersion": null, "title": "fixture title without a fix", "url": "https://advisory.example/ghsa", "firstSeenAt": "2026-11-02T00:00:00Z", "status": "open"},
{"id": "CVE-2026-22222", "targetId": "docker.io/library/fixturedb@sha256:2222222222222222222222222222222222222222222222222222222222222222", "severity": "CRITICAL", "cvss": 9.0, "package": "fixture-libaccepted", "installedVersion": "3.0.0-installed", "fixedVersion": "3.0.1-fixed", "title": "fixture accepted title", "url": "https://advisory.example/accepted", "firstSeenAt": "2026-10-01T00:00:00Z", "status": "accepted", "acceptance": {"reason": "fixture-acceptance-reason", "expires": "2027-01-01"}},
{"id": "CVE-2026-33333", "targetId": "docker.io/library/fixturedb@sha256:2222222222222222222222222222222222222222222222222222222222222222", "severity": "HIGH", "cvss": 7.5, "package": "fixture-libhigh", "installedVersion": "4.0.0-installed", "fixedVersion": "4.0.1-fixed", "title": "fixture high title", "url": "https://advisory.example/high", "firstSeenAt": "2026-11-03T00:00:00Z", "status": "fix-available"},
{"id": "CVE-2026-44444", "targetId": "ghcr.io/fixture-org/fixture-api@sha256:1111111111111111111111111111111111111111111111111111111111111111", "severity": "LOW", "cvss": null, "package": "fixture-liblow", "installedVersion": "5.0.0-installed", "fixedVersion": null, "title": "fixture low title", "url": "", "firstSeenAt": "2026-11-04T00:00:00Z", "status": "open"}
],
"remediation": []
}
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
# L1 offline check for the CVE triage bot: fixture-driven, no cluster, no network. Assertions and
# their limits are in check.py. rc 0 pass, 1 fail.
set -euo pipefail
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PYTHONDONTWRITEBYTECODE=1 exec python3 "${here}/check.py"
@@ -64,6 +64,12 @@ spec:
annotations: annotations:
# conformance-dns.<this cluster's BASE_DOMAIN> — never a literal apex. # conformance-dns.<this cluster's BASE_DOMAIN> — never a literal apex.
external-dns.alpha.kubernetes.io/hostname: (join('.', ['conformance-dns', $baseDomain])) external-dns.alpha.kubernetes.io/hostname: (join('.', ['conformance-dns', $baseDomain]))
# DNS-ONLY (2026-11-08). external-dns runs with --cloudflare-proxied globally (the
# gateway names point at the Cloudflare Tunnel). Without this opt-out the probe
# record is orange-clouded, 1.1.1.1 answers Cloudflare anycast instead of the LB IP,
# and the exact-IP poll below can never match. Same opt-out every real
# `service`-source publisher carries (unifi, crafty-mc).
external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
spec: spec:
type: LoadBalancer type: LoadBalancer
# No real backend needed — Cilium LB-IPAM assigns a VIP regardless, and # No real backend needed — Cilium LB-IPAM assigns a VIP regardless, and
@@ -0,0 +1,35 @@
# =============================================================================
# L2 BEHAVIORAL SCAFFOLD — the internal API is unreachable from an unrelated namespace (R-VULN-09)
# Catalog: docs/17 "Acceptance gates" (L2).
#
# A pod in an unrelated namespace cannot reach vuln-scanner :8081 or :9090, while the CI runner identity can reach :8081.
#
# STATUS: scaffold only. It asserts nothing and DOES NOT EXECUTE IN CI: vars.CLUSTER_AVAILABLE is unset,
# so L2/L3 conformance does not run. Coverage from this file is a named gap, not evidence.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: vuln-internal-api-isolated
labels:
conformance.home.k3s/R-VULN-09: "true"
annotations:
conformance.home.k3s/requirements: "R-VULN-09"
conformance.home.k3s/layer: "L2"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/intent: >-
A pod in an unrelated namespace cannot reach vuln-scanner :8081 or :9090, while the CI runner identity can reach :8081.
spec:
steps:
- name: given-an-unrelated-namespace
try:
- script:
content: 'echo "TODO: create a throwaway namespace and probe pod"'
- name: when-connecting
try:
- script:
content: 'echo "TODO: attempt tcp to vuln-scanner:8081 and :9090"'
- name: then-denied
try:
- script:
content: 'echo "TODO: assert a timeout on both, and success from the act-runner identity on :8081"'
@@ -0,0 +1,36 @@
# =============================================================================
# L2 BEHAVIORAL SCAFFOLD — new workload digest appears in the report (R-VULN-01, R-VULN-02)
# Catalog: docs/17 "Acceptance gates" (L2).
#
# A newly started workload's digest appears in report.json, mapped to that workload, within the 2-hour window.
#
# STATUS: scaffold only. It asserts nothing and DOES NOT EXECUTE IN CI: vars.CLUSTER_AVAILABLE is unset,
# so L2/L3 conformance does not run. Coverage from this file is a named gap, not evidence.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: vuln-new-digest-reported
labels:
conformance.home.k3s/R-VULN-01: "true"
conformance.home.k3s/R-VULN-02: "true"
annotations:
conformance.home.k3s/requirements: "R-VULN-01, R-VULN-02"
conformance.home.k3s/layer: "L2"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/intent: >-
A newly started workload's digest appears in report.json, mapped to that workload, within the 2-hour window.
spec:
steps:
- name: given-a-new-workload
try:
- script:
content: 'echo "TODO: start a pod from an image digest not yet in the report"'
- name: when-the-scanner-ticks
try:
- script:
content: 'echo "TODO: wait, bounded by the 2h SLO, polling GET /report.json from the CI runner identity"'
- name: then-digest-is-reported
try:
- script:
content: 'echo "TODO: assert a target with that digest and the workload mapping, state not pending past 2h"'
@@ -0,0 +1,39 @@
# =============================================================================
# L2 BEHAVIORAL SCAFFOLD — the CVE page requires a platform-admins login (R-VULN-04)
# Catalog: docs/17 "Acceptance gates" (L2).
#
# An unauthenticated request to cve.<domain> is redirected to Keycloak, and a logged-in user outside platform-admins gets 403.
#
# STATUS: scaffold only. It asserts nothing and DOES NOT EXECUTE IN CI: vars.CLUSTER_AVAILABLE is unset,
# so L2/L3 conformance does not run. Coverage from this file is a named gap, not evidence.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: vuln-page-requires-admin
labels:
conformance.home.k3s/R-VULN-04: "true"
annotations:
conformance.home.k3s/requirements: "R-VULN-04"
conformance.home.k3s/layer: "L2"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/intent: >-
An unauthenticated request to cve.<domain> is redirected to Keycloak, and a logged-in user outside platform-admins gets 403.
spec:
steps:
- name: given-the-page-route
try:
- script:
content: 'echo "TODO: resolve cve.<domain> through the Gateway"'
- name: when-unauthenticated
try:
- script:
content: 'echo "TODO: request the page with no session"'
- name: then-redirected-to-keycloak
try:
- script:
content: 'echo "TODO: assert a 302 to the homelab realm"'
- name: then-non-admin-forbidden
try:
- script:
content: 'echo "TODO: assert 403 for a session outside platform-admins"'
@@ -0,0 +1,40 @@
# =============================================================================
# L3 FAULT-INJECTION SCAFFOLD — breaking registry access produces failed targets and the coverage alert (R-VULN-03, R-VULN-06)
# Catalog: docs/17 "Acceptance gates" (L3).
#
# With registry access broken for one image, that target becomes failed with its reason, prior findings are retained, and VulnCoverageGap fires after 24h.
#
# STATUS: scaffold only. It asserts nothing and DOES NOT EXECUTE IN CI: vars.CLUSTER_AVAILABLE is unset,
# so L2/L3 conformance does not run. Coverage from this file is a named gap, not evidence.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: vuln-registry-failure
labels:
conformance.home.k3s/R-VULN-03: "true"
conformance.home.k3s/R-VULN-06: "true"
annotations:
conformance.home.k3s/requirements: "R-VULN-03, R-VULN-06"
conformance.home.k3s/layer: "L3"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/intent: >-
With registry access broken for one image, that target becomes failed with its reason, prior findings are retained, and VulnCoverageGap fires after 24h.
spec:
steps:
- name: given-a-scanned-target
try:
- script:
content: 'echo "TODO: record a scanned target and its findings"'
- name: when-registry-access-is-broken
try:
- script:
content: 'echo "TODO: inject a fault blocking that registry"'
- name: then-target-failed
try:
- script:
content: 'echo "TODO: assert state failed, reason present, findings retained, run continues"'
- name: then-coverage-alert-fires
try:
- script:
content: 'echo "TODO: assert VulnCoverageGap fires once the failure passes 24h"'
+1 -1
View File
@@ -61,7 +61,7 @@ CONFORMANCE_RUNNER_IMAGE_TAG="chainsaw0.2.12-kubectl1.36.4-fc44"
# chainsaw-job.yaml's @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ takes the 64 hex digits WITHOUT the # chainsaw-job.yaml's @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ takes the 64 hex digits WITHOUT the
# `sha256:` prefix this value carries — the trigger must strip it. # `sha256:` prefix this value carries — the trigger must strip it.
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/conformance-runner tag=main # renovate-digest: image=gitea.home.keithjackson.dev/home-network/conformance-runner tag=main
CONFORMANCE_RUNNER_IMAGE_DIGEST="sha256:65cc7c1ab576fe8cdf45e7f68c7cf4eb5ce7e0db46e62191f6190c51610b20d9" CONFORMANCE_RUNNER_IMAGE_DIGEST="sha256:63e1aa74289a8b77ef5cd2ec4e4d01e7fef638f965f2057329bc1991d3074f0d"
CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}" CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}"
# #
# AFTERWARDS, TIGHTEN THE BOUNDARY: the header of conformance-trigger-job-shape.yaml asks for # AFTERWARDS, TIGHTEN THE BOUNDARY: the header of conformance-trigger-job-shape.yaml asks for
+139
View File
@@ -0,0 +1,139 @@
---
date: 2026-11-08
branch: feat/cloudflare-tunnel
title: Cloudflare Tunnel connector, Phase 1 — the component only, isolated, with zero change to live traffic
---
## What changed
Ingress is moving to a Cloudflare Tunnel because the GFiber router cannot hairpin and hands LAN
clients Google's IPv6 DNS, so split DNS keeps breaking. This is **Phase 1: the connector only.**
external-dns, the Gateway annotations, Pi-hole and CoreDNS are untouched. A later change, by a
different agent, moves DNS.
**Operator decisions (2026-11-08), recorded here as the source for the guard:**
1. Public hostnames stay `*.home.keithjackson.dev` and the apex `home.keithjackson.dev`. The
operator enables Cloudflare ACM (ordered edge cert or Total TLS) for edge certificates.
2. **Plex stays OFF the tunnel.** No plex hostname goes through cloudflared (`plex`, `plex-vip` → 404).
3. **jackett, pihole and semaphore stay LAN/VPN-only**, never via the tunnel. The same goes for the
non-HTTP L4 VIPs **unifi** and **crafty-mc**. After the PM's read-only audit, the other three
catalog `exposure: internal` apps, **qbittorrent, radarr and sonarr**, are on the same list. All
six carry the external-dns `controller: none` route annotation live.
4. **The Kubernetes API stays local-only.** It must never be reachable through the tunnel.
5. Pi-hole keeps serving the LAN. Split DNS gets trimmed later, by another agent.
6. **Headlamp is LAN/VPN-only** (operator decision, relayed by the PM later the same day). It is
permanently in the 404 list and excluded from the egress backends. Its catalog/route exposure is
deliberately **not** changed here; the Phase 2 DNS-flip PR does that.
**Files:**
- `platform/cloudflared/` (new):
- `config.yaml` is the locally-managed tunnel config, projected through a content-hashed
`configMapGenerator`, so edits roll the pods. It has explicit `http_status:404` rules for
plex, plex-vip, jackett, pihole, pihole-dns, semaphore, qbittorrent, radarr, sonarr, headlamp,
unifi and crafty-mc. Then the apex and `*.${BASE_DOMAIN}` go to
`https://cilium-gateway-shared.gateway.svc.cluster.local:443` with `matchSNItoHost: true`. Last
comes a catch-all 404. The tunnel UUID is a **literal** in this file (`REPLACE-ME-TUNNEL-UUID`
until enrolment), not a cluster-var, so the Phase 2 PR can match it without touching `clusters/**`.
- `cloudflared-credentials.sops.yaml` is SOPS-encrypted and holds `REPLACE-ME` placeholders, with
a `.example` hand-off sibling. I encrypted the placeholder (not a credential) in the toolbox
with the root `.sops.yaml` recipient.
- `deployment.yaml` runs 2 replicas with a hard `topologySpreadConstraints` across nodes. The image
is `cloudflare/cloudflared:2026.10.7@sha256:4955524a…` (tag + digest, Renovate `kubernetes`
manager). It runs as non-root 65532, with a read-only root fs, all capabilities dropped and no
SA token. Requests are 10m/64Mi and limits 500m/256Mi. Readiness uses `/ready`. Liveness uses
`/healthcheck`, so an edge outage cannot restart-loop the pods. The pods are labelled
`egress-class: internet`. A PDB sets `minAvailable: 1`.
- `networkpolicy.yaml`:
- A namespace `default-deny` with DNS only. It has a **single owner**: the namespace is not
`app-namespace=true`, so Kyverno's generate rule never writes it.
- A Prometheus scrape allow on :2000.
- The egress CNP allows world tcp/udp 7844 and tcp 443, plus the `ingress` entity on tcp 443.
It also allows **Gateway backends by `io.cilium.k8s.namespace.labels.gateway-access=shared`,
minus the LAN-only namespaces.**
- `servicemonitor.yaml` (headless metrics Service + ServiceMonitor) and `alerts.yaml` (`CloudflaredTunnelDown`, `CloudflaredReplicasDegraded`)
ship with `alerts.promtool-test.yaml`, and `README.md` documents the component.
- `clusters/base/cloudflared.yaml` (new) is its own Flux Kustomization: `dependsOn: platform-config`,
**`wait: false`**, prune, and postBuild `cluster-vars`. It is listed in `clusters/base/kustomization.yaml`.
- `clusters/{dev,prod}/kustomization.yaml` each gain a SOPS `decryption` patch for `cloudflared`.
- `conformance/tests/l0-static/cloudflared-ingress-rules/` (new, whole-repo scope) asserts several
things:
- Every rule targets either 404 or exactly the Gateway Service. The Service name is derived from
`platform/gateway/gateway.yaml`, and API markers are named explicitly.
- The excluded hosts 404 before the first Gateway rule, the Gateway rules use `matchSNItoHost`,
there is no `noTLSVerify`, and a catch-all closes the list.
- The CNP has no kube-apiserver, host, remote-node, cluster or `all` entity, no CIDR, and the full
NotIn list. The pods carry egress-class internet and have no SA token.
- The Flux tier is isolated: wait:false, not in `platform/kustomization.yaml`, nothing
`dependsOn: cloudflared`, and both overlays decrypt it.
- **Mutation self-test:** an added API rule, plex moved below the wildcard, and a dropped
catch-all must each be flagged.
- `conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh` gains an additive
`cloudflared_` scrape owner, with a networkpolicy companion, in the same shape as the kured entry.
- `runbooks/cloudflare-tunnel.md` (new) is registered in `runbooks/kustomization.yaml` and the
README index.
**Design choices:**
- **Isolation (the merge-before-credentials question).** The PR is safe to merge with placeholders.
The Deployment CrashLoops, but in its own `wait:false` Kustomization that nothing depends on. Its
Ready condition tracks the apply, not pod health, so neither `platform`, `platform-config` nor any
apps wave can stall. There is no Certificate or HelmRelease in the path for health checks to hang
on. Health is reported by the two alerts instead. `CloudflaredTunnelDown` **will fire from merge
until enrolment**. That is honest, and it is `warning` because nothing routes through the tunnel
yet. Raise it to critical in the DNS-flip change.
- **Egress to backends, not `{}`.** Under Cilium 1.20 (CVE-2025-30162), the Gateway's Envoy
re-checks the source pod's egress against the resolved backend. An egress rule that names only
the Gateway would therefore make every tunnelled request `403 Access denied`
(decisions/2026-10-30-gitea-admit-buildkit.md; catalog/render.py, `r_internet_egress`). The
gitea-actions `- {}` pattern would also work, but it admits kube-apiserver and host, which
decision 4 rules out. The namespace-label selector names every Gateway backend and nothing else.
**Backends need no ingress change.** Envoy reaches them as `reserved:ingress`, and the 2026-10-30
measurement showed the identity check sits on the source's egress: gitea-actions is in none of
Gitea's ingress allows and still reaches it. So no app's `inClusterClients` was touched.
- **The alerts cannot self-silence.** Both expressions use `or on() vector(0)`, so absent series
(a broken scrape, zero pods, KSM gone) fire. The promtool tests pin the absent case and the
working-then-stale re-fire case.
## What I verified
- **The Gateway Service, measured live (read-only, `k3s-prod`, 2026-11-08).** It is
`gateway/cilium-gateway-shared`: type LoadBalancer, ClusterIP `10.43.225.79`, EXTERNAL-IP
`192.168.2.20`, ports 80/443, no selector, labelled `gateway.networking.k8s.io/gateway-name=shared`.
It belongs to Gateway `gateway/shared` (class `cilium`).
- **The namespace selector, measured live.** Cilium identities carry the label
`k8s:io.cilium.k8s.namespace.labels.gateway-access` (read from a gitea identity). Every namespace
with an HTTPRoute (crafty, gitea, headlamp, homepage, jackett, keycloak, monitoring, nextcloud,
oauth2-proxy, pihole, plex, qbittorrent, radarr, runbooks, seerr, semaphore, sonarr) is
`gateway-access=shared`.
- **Nothing exists yet.** Namespace `cloudflared` does not exist on prod. **cloudflared is not in
the pinned toolbox**, so the runbook runs the pinned image under podman instead.
- The image tag and digest came from Docker Hub's API: `2026.10.7` = `latest` =
`sha256:4955524a…c9b9c5ba`, published 2026-10-30.
- The credentials Secret is encrypted: `stringData.credentials.json` is `ENC[…]`, with the age
recipient from the root `.sops.yaml`.
- The new L0 check passes on the host (47 assertions, mutation self-test included). The toolbox
gate result is reported in the PR description. I ran it after committing.
## What I did NOT verify
- **UNVERIFIED: nothing ran in a cluster.** No pod has started, and no tunnel exists. I have not
observed placeholder CrashLoop behaviour itself. I expect cloudflared to treat
`REPLACE-ME-TUNNEL-UUID` as a tunnel name, find no origin cert, and exit, but whatever it does,
the pods are not Ready and the tier is non-gating.
- **UNVERIFIED: the CVE-2025-30162 path for cloudflared specifically.** The selector form was
checked against live identity labels, but no tunnelled request has been made. The runbook's
step 6 `kubectl debug` probe exists to prove "real backend, no 403" from the pod's own identity.
- **UNVERIFIED: whether the `ingress` entity rule's `toPorts: 443` restriction holds at Envoy.**
buildkit's working rule has no port restriction. If the canary shows drops, removing the port
restriction there is the first thing to try.
- The metric name `cloudflared_tunnel_ha_connections` and the `/ready` / `/healthcheck` endpoints
come from my knowledge of cloudflared, **not measured** against 2026.10.7.
- The Cloudflare edge egress is `world` + ports, not Cloudflare CIDRs. I did not narrow it blind.
- Egress `world:443` "for the API" follows the brief. I have not confirmed that a credentials-file
tunnel needs it at all.
- The Cloudflare-side settings (ACM, Total TLS / ordered cert, Bot Fight Mode, BIC/WAF, WebSockets)
are documented for the operator but not inspected. I have no Cloudflare access.
- The headlamp, qbittorrent, radarr and sonarr catalog exposures are deliberately unchanged
(Phase 2).
@@ -0,0 +1,59 @@
---
date: 2026-11-08
branch: fix/cloudflared-creds-permission
title: cloudflared CrashLoop — credentials Secret unreadable by uid 65532; add fsGroup, pin it in L0
---
## What changed
- **`platform/cloudflared/deployment.yaml`**: pod `securityContext.fsGroup: 65532`. Root cause of
the CrashLoop that followed #152/#153: the pod runs as `65532:65532` with no `fsGroup`, and the
Secret volume is `defaultMode: 0440`. The kubelet therefore laid `credentials.json` down as
`root:root 0440`, which uid 65532 (not owner, not in gid 0) cannot read. cloudflared logged
`couldn't read tunnel credentials ... permission denied`, then fell back to origin-cert lookup
and ended on the misleading `error parsing tunnel ID: Error locating origin cert`. With
`fsGroup` the files are group-owned by 65532, readable by the container and no other uid; the
mode stays 0440 (not loosened to 0444). Same fix as `platform/keycloak/client-secret-reconciler.yaml`.
Still non-root, `runAsNonRoot: true`, unchanged otherwise.
- **`conformance/tests/l0-static/cloudflared-ingress-rules/check.sh`**: new assertion (7). For each
Secret volume a container mounts, it models kubelet ownership (uid 0, gid = fsGroup or 0, item
mode / defaultMode / API default 0644) against the container's effective runAsUser and its groups
(runAsGroup + fsGroup + supplementalGroups) and fails if the file is unreadable. An unset
runAsUser fails (image default is not knowable offline). Conservative: it does not credit the
kubelet OR-ing group-read into fsGroup volumes. Mutation self-test: fsGroup removed, mode 0400,
runAsUser unset must each be flagged. Chainsaw step description updated to match.
- **`runbooks/cloudflare-tunnel.md`**: new first "likely cause" (permission denied on
credentials.json → the origin-cert error is a red herring) plus a "read the first error line"
note; causes renumbered. New section **Rotating the tunnel credential**: create a new tunnel,
swap the UUID and the encrypted credentials, merge, verify (new tunnel has two connectors, old
has none), then `cloudflared tunnel cleanup`/`delete <old>`.
- NOT touched, on purpose: `tunnel:` in `config.yaml` and `cloudflared-credentials.sops.yaml`. The
operator is rotating the credential (it was exposed in a transcript) in a separate PR.
## What I verified
- Live (k3s-prod, read-only, 2026-11-08 ~21:38Z): Deployment pod securityContext is
`{"runAsGroup":65532,"runAsNonRoot":true,"runAsUser":65532,"seccompProfile":...}`, no
`fsGroup`; creds volume `defaultMode: 288` (= 0440). All three pods CrashLoopBackOff (38
restarts). Secret `cloudflared-credentials` has the one key `credentials.json` (key names only).
- Pinned image `cloudflare/cloudflared@sha256:4955524a...` config (`skopeo inspect --config`):
`User: 65532:65532`, `Entrypoint: [cloudflared]`, `Cmd: [version]`.
- The new assertion FAILS (rc 1) on the pre-fix deployment.yaml with
`container cloudflared (uid 65532, groups [65532]) cannot read Secret volume creds ... files are
0:0 mode 0440`, and PASSES (51 assertions) with the fix. All three mutants are flagged.
- Kyverno admission policies under `platform/kyverno/admission/` that mention securityContext
fields are scoped to the conformance-trigger and pm-agent Jobs only; none constrains fsGroup here.
- Toolbox gate: see the PR description for the rc and skipped stages.
## What I did NOT verify
- **That the pods actually become Ready after merge.** Not deployed by me; Flux applies it. The
file mode inside the running container was not observed (pods crash before an exec is possible,
and exec would be an operator action anyway). The readability claim rests on the kubelet's
documented fsGroup behaviour and the identical, working keycloak reconciler.
- Whether the CURRENT credential is otherwise valid (UUID/TunnelID match, tunnel exists). Moot:
the operator is rotating it.
- The rotation runbook section has not been executed; `cloudflared tunnel cleanup` / `delete`
syntax is from the CLI's documented commands, not run here.
- The deployment comment says the image ENTRYPOINT is `cloudflared --no-autoupdate`; the pinned
image's ENTRYPOINT is plain `cloudflared`. Left unchanged (out of scope); flagged for follow-up.
@@ -0,0 +1,77 @@
---
date: 2026-11-08
branch: feat/dns-via-cloudflare-tunnel
title: Public DNS via the Cloudflare Tunnel (Phase 2) — proxied CNAMEs, Plex/Headlamp LAN-only, L0 guard
---
## What changed
One atomic change, so the conformance guard lands with the DNS flip.
- `platform/gateway/gateway.yaml`: the shared Gateway's `external-dns.alpha.kubernetes.io/target`
is now the literal `fc81938b-5855-47f1-8a27-6286f56dda93.cfargotunnel.com` (was
`wan.${BASE_DOMAIN}`). Comment block rewritten.
- `platform/externaldns/helmrelease.yaml`: `extraArgs: [--cloudflare-proxied]` (global, because
v0.23.0 reads per-object `cloudflare-proxied` from the route/Service, never the Gateway). The
public-target comment block rewritten, including the prefix-regression risk.
- `apps/vms/unifi/50-service-l4.yaml`, `platform/crafty/service-l4.yaml`:
`external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"` (L4 A records stay DNS-only). They
are the only `service`-source publishers in the tree and live (measured).
- Plex web UI LAN/VPN-only: `catalog/plex.yaml` `exposure: public` → `internal`; `apps/wave-0/plex.yaml`
re-rendered with the canonical command into a scratch dir and ONLY plex.yaml copied back. The
render diff is exactly +2 lines (`annotations: external-dns.alpha.kubernetes.io/controller: none`
on HTTPRoute plex-32400). No other golden moved, and nothing was onboarded (vm-demo/keycloak files
were not touched). ADVERTISE_IP, the L4 VIP .24 and `plex-remote.yaml` are untouched.
- Headlamp LAN/VPN-only: `controller: none` on the hand-authored `platform/headlamp/httproute.yaml`.
- `conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh`: header and
register prose updated; new assertion (9):
(9a) Gateway target == `<tunnel: from platform/cloudflared/config.yaml>.cfargotunnel.com`, and
`--cloudflare-proxied` is set; (9b) every `service`-source publisher has `cloudflare-proxied:
"false"` and no HTTPRoute opts out of proxying; (9c) no undisowned route names a hostname the
tunnel 404s, the only `service` publishers of a 404'd name are the declared L4 set
{unifi, crafty-mc}, and every disowned route's hostname IS 404'd by the tunnel (lists kept
consistent both ways); (9d) plex and headlamp pinned disowned. Its `chainsaw-test.yaml` gap prose
updated.
- `conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml`: fixture Service gets
`cloudflare-proxied: "false"`, else the global flag proxies it and the exact-IP poll fails.
- Docs: `platform/externaldns/ddns.yaml` header (wan. is WireGuard-only), `platform/externaldns/README.md`,
`platform/gateway/README.md`, `runbooks/plex-remote-access.md`, `runbooks/cloudflare-tunnel.md`
(Phase 2 done; operator step to close WAN tcp/80+443 and keep udp/51820 + tcp/32400; rotation
now also swaps the Gateway UUID; rollback path).
## What I verified
- external-dns v0.23.0 source, read at the tag: `source/gateway.go` takes target overrides from
`gw.Annotations` and provider-specific (incl. `cloudflare-proxied`) from the ROUTE's annotations;
`annotations.SetAnnotationPrefix` re-derives `CloudflarePrefix`, so the alpha-prefixed key is the
one read here; `provider/cloudflare` `shouldBeProxied` starts from `--cloudflare-proxied` and a
parsed `"false"` annotation overrides it.
- (8b) only forbids `annotation-prefix` / `enable-legacy-annotation-prefix` in extraArgs; a mutation
confirmed `--cloudflare-proxied` passes and `--annotation-prefix=x/` fails.
- The L0 check passes (116 assertions) and each new assertion FAILS on its mutation: Gateway target
back to wan (9a), flag removed (9a), unifi proxied "true" (9b), headlamp undisowned (9c,9d), plex
golden undisowned (9c,9d,(1)), jackett removed from the tunnel's 404 list (9c).
`cloudflared-ingress-rules` still passes; `render.py --selftest` PASS.
- Live, read-only (kubectl k3s-prod, ~23:55Z): 17 HTTPRoutes; the six internal apps carry
`controller: none`; plex and headlamp do not (yet); the only Services with an external-dns
hostname are crafty/crafty-l4-tcp and unifi/unifi-l4-tcp; Gateway target is currently
`wan.home.keithjackson.dev`.
- Toolbox gate result: see the PR description (rc and skipped stages quoted there).
## What I did NOT verify
- Anything after merge: that external-dns actually UPDATEs the eight public names to proxied CNAMEs
to the tunnel, DELETEs plex + headlamp and their TXT, and leaves unifi/crafty-mc/wan untouched.
Expected effect is listed in the PR; the PM must read external-dns logs and `dig @1.1.1.1`.
- A rendered HelmRelease: I read the chart 1.23.0 `templates/deployment.yaml` (a list `extraArgs`
is emitted item-by-item via `tpl . $ | quote`), but did not `helm template` it.
- kubectl OIDC from nodes: nodes resolve publicly, so `keycloak.` discovery/JWKS now goes through
Cloudflare + the tunnel. No `oidc-ca-file` is set (system roots), so the edge cert should validate,
but Cloudflare bot/WAF settings on that path are untested.
- RISK: if the annotation prefix stops being honoured at runtime again (11-07 incident class),
`target` is ignored and every name becomes a PROXIED A record at 192.168.2.20, while internal
names publish. Assertion (8) only keeps the tree and `values.annotationPrefix` consistent; it
cannot catch a controller that ignores the flag. No runtime alert exists for it.
- The router WAN 80/443 close is an operator step, not done. The Pi-hole wildcard trim is a later PR.
- Follow-up: spec requirement IDs for this (proposed R-DNS-04/05, R-SEC-04) once the spec ADR merges;
`platform/pm-agent/README.md` and a comment in `catalog/pihole.yaml` still describe the wan. CNAME.
@@ -0,0 +1,74 @@
---
date: 2026-11-08
branch: fix/keycloak-operator-crds
title: "Keycloak 26.8.2 operator CrashLoop: install its two new CRDs, raise its memory"
---
## What changed
**Incident.** PR #121 (Renovate) bumped `platform/keycloak/kustomization.yaml`'s three
`keycloak-k8s-resources` URLs 26.0.6 -> 26.8.2. Upstream `kubernetes/` at 26.8.2 holds **four**
CRDs, not two (GitHub contents API, `?ref=26.8.2`):
| file | at 26.0.6 | at 26.8.2 |
|---|---|---|
| `keycloaks.k8s.keycloak.org-v1.yml` | yes | yes |
| `keycloakrealmimports.k8s.keycloak.org-v1.yml` | yes | yes |
| `keycloakoidcclients.k8s.keycloak.org-v1.yml` | — | **new** (v2alpha1) |
| `keycloaksamlclients.k8s.keycloak.org-v1.yml` | — | **new** (v2alpha1) |
| `kubernetes.yml` | yes | yes (now carries RBAC for both new kinds) |
(`cluster-wide/` is a separate variant; upstream's own `kubernetes/kustomization.yml` lists exactly
the five files above.) The 26.8.2 operator starts an informer for every controller it has and dies
on the first missing kind: `Couldn't start informer for keycloaksamlclients.k8s.keycloak.org/v2alpha1
... GET .../namespaces/keycloak/keycloaksamlclients ... Not Found`, exit 1, CrashLoopBackOff. Flux
`platform` (`wait: true`, 5m) sat in health checks and every dependent Kustomization reported
`dependency ... is not ready`. The old 26.0.6 ReplicaSet kept serving (RollingUpdate 25%/25%), so SSO
stayed up.
**Fix.**
- `platform/keycloak/kustomization.yaml`: added the two CRD URLs, each with the same
`# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources` marker (the repo's
custom.regex manager matches all five — checked by running its regex over the file). Header
comment updated to v26.8.2, plus a warning that Renovate cannot add a CRD a release introduces.
- Same file: a `patches:` JSON6902 patch on Deployment `keycloak-operator`: memory limit
450Mi -> **768Mi**, request 450Mi -> **512Mi**. 450Mi is upstream's default (unchanged at 26.8.2,
so not something the bump changed). Evidence it is too small here: the 26.0.6 pod had **154
restarts**, `lastState.terminated.reason: OOMKilled` (exit 137, 2026-11-07T23:09:54Z), and
`kubectl top` showed **421Mi / 450Mi**. 26.8.2 runs four controllers instead of two on Quarkus
3.40.1.1 (was 3.15.1). No variable in the operator to disable watching unused kinds was used —
installing the CRDs is upstream's supported shape.
- `platform/keycloak/README.md` "Upgrading Keycloak": the diff-the-CRD-list step; "three URLs" ->
"the release URLs". `renovate.json5`: same wording fix in a comment only.
## What I verified
- Upstream file lists at 26.0.6 and 26.8.2 (GitHub API) and `diff` of `kubernetes.yml` between the
two: the new kinds' ClusterRoles + RoleBindings ship in it, and the Deployment's resources are
still 450Mi/450Mi.
- Live, read-only (`kubectl --context k3s-prod`, 2026-11-08 ~05:57Z): only `keycloaks` and
`keycloakrealmimports` CRDs installed; Deployment strategy RollingUpdate maxSurge/maxUnavailable
25%; RS `keycloak-operator-58dff6c4` (26.8.2) 0/1 CrashLoopBackOff, RS `78dbb75f64` (26.0.6) 1/1;
operator log is the informer error above; Flux `platform` `Healthy=Unknown` "Running health
checks for revision main@sha1:e8d2666"; no NetworkPolicy and no CiliumNetworkPolicy in ns
`keycloak`.
- No sensitive path touched (`infra/gitea/approval-policy.json`), so this should merge on green.
- Gate: see the PR body for the exact rc and stages of the run against the final commit.
## What I did NOT verify
- **That the 26.8.2 operator becomes Ready after this merges.** Only the two missing kinds were
observed failing; another start-up failure behind them is possible.
- **That 768Mi is enough** — no 26.8.2 memory measurement exists yet; re-check `kubectl top` after
a week.
- **The Keycloak server upgrade 26.0.6 -> 26.8.2 itself.** Once the operator runs it will roll
keycloak-0/1 and Keycloak migrates the DB schema. The CR's `spec.update.strategy` defaults to
`RecreateOnImageChange` — expect a **full SSO outage** while both pods recreate, not a rolling
one. Back up / confirm the latest CNPG backup first if that matters.
- **The 26.8.2 Keycloak CRD's `spec.networkPolicy.enabled` defaults to `true`.** The operator will
create a NetworkPolicy selecting the Keycloak pods (first policy in that namespace). With no
`from` lists set it allows all sources on the HTTP/management ports, so Gateway, ServiceMonitor
and the reconciler Jobs should still reach it — but this was read from the CRD text, not observed.
- **The passkey reconciler's 26.0.6 Admin API assumptions against 26.8.2** (README "Upgrading
Keycloak" list) — not re-checked here; PR #121 merged without it.
- `catalog/keycloak.yaml`'s documentation-only image tag was not updated.
@@ -0,0 +1,60 @@
---
date: 2026-11-08
branch: fix/keycloak-servicemonitor-owner
title: One owner for the keycloak ServiceMonitor — disable the 26.8 operator's (409 loop stalled platform-config)
---
## What changed
- `platform/keycloak/keycloak-instance.yaml`: added `spec.serviceMonitor.enabled: false`.
The 26.x Keycloak operator creates its own ServiceMonitor, named after the CR (`keycloak/keycloak`),
whenever `metrics-enabled=true` and the ServiceMonitor CRD exists. `serviceMonitor.enabled` defaults
to `true` in the 26.8.2 CRD, both in v2alpha1 and in v2beta1 (v2beta1 is now the storage version).
That is the same name and namespace as the repo's Flux-owned `platform/keycloak/servicemonitor.yaml`.
After the 26.0.6 -> 26.8.2 bump (#121/#144) the operator's PATCH against it returned
`409 Conflict` and it ran out of retries. The CR then sat at `Ready=False`/`HasErrors=True`, and
`platform-config` (`wait: true`, which health-checks every object it applies, this CR included)
blocked everything that `dependsOn` it: velero, apps-vms, apps-wave-0 (and so waves 1/2 and
pm-agent), homepage, oauth2-proxy, headlamp, conformance, vpn-ingress, crafty, runbooks.
- `platform/keycloak/servicemonitor.yaml`: a comment saying it is the sole owner.
- **Why ours is kept:** these are written against it: the L0 `servicemonitor-ports-resolve`
allow-list entry `platform/keycloak/servicemonitor.yaml:keycloak`, the L2 `keycloak-monitored`
chainsaw test, the README's ownership table, and the dashboard's `up{namespace="keycloak"}`.
The operator's version (upstream `KeycloakServiceMonitorDependentResource` @26.8.2) scrapes the same
target: port name `management` (9000 on `keycloak-service`), path `/metrics`, interval 30s,
selector = the operator's instance labels. It offers nothing ours lacks, so having two owners
would add a fight and gain nothing.
## What I verified
- Live, read-only, `k3s-prod` ~06:25Z: the CRD schema field `spec.serviceMonitor.enabled`
(`default: true`) exists in both served versions. The live `keycloak` ServiceMonitor has exactly
**one** managedFields manager, `kustomize-controller` (Apply, 2026-09-05), **no ownerReferences**,
and labels `kustomize.toolkit.fluxcd.io/name=platform-config`, `release=kube-prometheus-stack`. So
the operator's apply never landed, and nothing on the object ties it to the Keycloak CR.
- The ServiceMonitor will not be deleted when the operator is switched off: the operator's dependent
informer is filtered on `Constants.DEFAULT_LABELS_AS_STRING`
(`app.kubernetes.io/managed-by=keycloak-operator`). Our object does not carry that label and has
no ownerReference, so the operator cannot see it as its secondary and cannot delete or GC it.
This conclusion comes from reading the code. It was not observed.
- The other operator dependents in ns `keycloak` do not collide with anything: `keycloak-service`,
`keycloak-discovery`, the StatefulSet `keycloak`, `keycloak-initial-admin`, and
`keycloak-network-policy` (owner Keycloak, no Flux label). The repo declares none of these names.
It only *references* `keycloak-service` by name (HTTPRoute, reconcilers). There are no PDBs in the
namespace, and the Ingress is already `enabled: false`. The only NetworkPolicy in the namespace is
the operator's. The repo has no Keycloak NetworkPolicy, and no Kyverno-generated default-deny is
present there. NetworkPolicies add up, so the operator's allow-policy cannot fight another owner.
Left at its default.
- Local gate: see the PR body. It records the rc and stages.
## What I did NOT verify
- That the CR reaches `Ready=True`/`HasErrors=False` after merge. The status also says
"Performing Keycloak upgrade, scaling down the deployment". I am inferring that this half clears
once the workflow stops erroring. It is not proven.
- That the operator stops issuing the PATCH once `enabled: false` is applied. Watch the operator
logs for 409.
- That `platform-config` and its downstream Kustomizations go Ready on the new revision, and that
Prometheus keeps exactly one `serviceMonitor/keycloak/keycloak/0` pool with targets up.
- The operator's actual label set for `DEFAULT_LABELS_AS_STRING`. I took it from the observed labels
on operator-created objects, not from `Constants.java`.
+116
View File
@@ -0,0 +1,116 @@
---
date: 2026-11-08
branch: feat/parallel-l0-checks
title: L0 offline checks, mutation suites and yamllint run in bounded parallel pools (gate 184s -> 32s locally)
---
## What changed
The gate's wall time was dominated by serial loops over independent subprocesses. Three of
them now run in bounded pools; none of them changes a verdict, an exit code, or the order of
anything printed.
- `conformance/lib/run_l1_offline.py` — the per-check loop (L0 and L1 offline checks) runs in
a `ThreadPoolExecutor` of `bash <check>` children. `--jobs N` / env `CONF_OFFLINE_JOBS`,
default `min(8, usable CPUs)` (`sched_getaffinity`, so a cgroup cpuset is respected);
`--jobs 1` is the old serial runner. Results are consumed in SUBMISSION order, so `cases`,
the JUnit, the step summary and the 0/1/3 aggregation are built from exactly the same
ordered list as before. Additive output only: one `[offline] running N check(s), jobs=…`
line, and a trailing `[N.Ns]` wall time on each per-check status line. Children get
`stdin=/dev/null` (parallel children must not share the runner's stdin).
- `conformance/tests/l0-static/keycloak-{passkey,client-secret}-reconciler/mutations.py` —
pass 1 (serial) checks every anchor and builds each mutated source, exactly as before;
pass 2 runs the `--selftest` children in a pool (env `MUTATION_JOBS`, default
`min(8, CPUs)`, `1` = serial). Verdicts are consumed in `MUTATIONS` order. A timeout
(`TimeoutExpired`) still propagates and fails the check; queued mutations are cancelled.
An invalid `MUTATION_JOBS` exits 2 (fails the check, never skips it).
- **Nested-pool budget.** When the outer runner is parallel it exports
`MUTATION_JOBS = max(2, min(8, (cpus - jobs + 2) // 2))` unless the caller already set
one. Reason: only those two checks fan out; the other `jobs-2` workers are single
processes, so the two suites split the CPUs those leave free. Peak processes ≈
`(jobs-2) + 2*inner` ≤ cpus whenever cpus ≥ jobs+2 (24 CPUs: 6 + 16 = 22), and ~1.25–1.5×
cpus on a small runner (4 CPUs: 2 + 4 = 6). The floor of 2 is deliberate: a 1-wide inner
pool would make the 40-mutation passkey suite (~85s serial) the critical path of the stage.
- `ci/lint.sh` — yamllint runs over `LINT_JOBS` (default `min(8, nproc)`) CONTIGUOUS slices
of the sorted file list concurrently; each slice's output is buffered to its own file and
printed in slice order, so the text is the same as one serial call and a `file` header is
never separated from its `line:col` findings. Stage rc = the highest slice rc; a slice that
leaves no rc file counts as 1. Invalid `LINT_JOBS` exits 2.
### Parallel-safety audit (all 25 `l0-static/*/check.sh`, 3 `l1-rendering/*/render-check.sh`)
Grepped every check for fixed `/tmp` paths, `_artifacts`/`ARTIFACTS`/`COVERAGE_OUT`,
`write_text`/`open(…,"w")`, `mktemp`/`mkdtemp`/`TemporaryDirectory`, sockets/ports, `git`
writes and `run.sh` invocations, and read what each writer writes to:
- **Scratch state: all private.** Every writer uses its own `mktemp -d` / `mkdtemp` /
`TemporaryDirectory` (placeholder-guard-coupling, spec-immutability-enforced,
whole-repo-invariants-wired, auditability-proofs-producible, alert-rules-unit-tested and its
`platform/monitoring/tests/run.sh`, internal-services-publish-no-public-dns,
oauth2-proxy-{api-routes-per-app,cookie-names-distinct}, both keycloak reconcilers, the three
L1 render checks). No fixed `/tmp` path anywhere.
- **Git writes:** only spec-immutability-enforced, and only `git init/add/commit` inside its
own `mktemp` repo. Everything else is `git ls-files`/`ls-tree`/`log`/`rev-parse` (read-only).
- **Ports:** the reconciler selftests bind `127.0.0.1:0` (kernel-assigned); the passkey
selftest also dials `127.0.0.1:1` as a deliberate refused connection. No fixed listener.
- **Ansible** (dev-nodes-provisioned-as-code): `--syntax-check`/`--list-hosts` only,
`retry_files_enabled = False`; its local tmp is per-process.
- **The one shared writer:** auditability-proofs-producible runs
`./run.sh --coverage-only --requirements …`, which REWRITES
`conformance/_artifacts/coverage/coverage.{json,md}` (it inherits `CONF_ARTIFACTS`). That was
already true serially. No other offline check reads or writes that directory (the only
reader is `tests/meta/harness-selfcheck`, not an offline check), and the outer runner
writes coverage BEFORE and the JUnit AFTER the pool, so the final state is the same as
serial. Not serialised; left as is and recorded here. `lib/status.py` (run by that check)
reads `_artifacts/report.junit.xml` as an advisory — written only after the pool, unchanged.
- whole-repo-invariants-wired calls `run.sh --print-selection` (writes nothing) and the
runner itself against synthetic dirs in its own `mktemp`.
Result: no check needed fixing or serialising. A new check that needs shared state must say
so; the runner's docstring now states that contract.
## What I verified
All in the pinned toolbox (`dev.sh run`, `DEV_KUBECONFIG=/dev/null`, ENGINE shim adding a ro
mount of the main checkout's `.git`; spec repo resolved as a sibling, spec-immutability
`pass`), host 24 CPUs, every `dev.sh` call under `flock`.
- **Gate, before** (origin/main `1a33814`, detached scratch worktree + spec mount):
`GATE_RC=3`, in-container wall **183.9s**; `l0` stage **168.3s**, `lint` **7.8s**.
"Stages that did not run: gitea-settings" (plus the expected on-base
touched-set-not-determinable note, since main has no branch commits).
- **Gate, after** (this branch, commit `a775082`): `GATE_RC=3`, in-container wall **32.1s**;
`l0` **21.1s**, `lint` **2.4s**, `l1` 2.3s. "Stages that did not run: gitea-settings" — the
only not-run stage. **rc 3, not green.** Caveat: the branch run selected 23 L0 checks (the
touched-requirement scoping dropped dev-nodes-provisioned-as-code and
placeholder-guard-coupling), the main run 25.
- **Same-tree L0 comparison** (all 25 l0 dirs, same tree, same container):
`CONF_OFFLINE_JOBS=1 MUTATION_JOBS=1` **164.0s / 165.3s** vs default (jobs=8,
MUTATION_JOBS=8) **19.2s / 18.7s**. JUnit `cmp`-identical, step summary `cmp`-identical,
stderr identical except the timing suffixes and the JUnit path in the last line.
- **Mutation suites, host:** passkey 83.2s → 10.5s (`MUTATION_JOBS=8`), client-secret
11.2s → 2.0s; output of `MUTATION_JOBS=1` and `=8` byte-identical to the pre-change script's,
rc 0 all round.
- **Falsification (not committed):** in a scratch worktree at `a775082`, made the passkey
mutation "the protected-subtree screen is never called" a no-op (replacement == anchor).
Host: `MUTATION_JOBS=1` and `=8` both rc 1, output byte-identical (with a second, broken
anchor also injected: both reported). Toolbox: runner rc 1 both serial and parallel,
`L0 FAIL keycloak-passkey-reconciler … 1 of 40 mutation(s) not proven … SURVIVED`; full
gate **rc 1**, `stage l0: FAIL`, `MERGE GATE: BLOCKED`. Reverted; scratch worktree removed.
- **lint.sh, host scratch repo** (31 YAML files, two broken): `LINT_JOBS=1/4/8` all rc 1,
stdout identical, findings named `y/f07.yaml` / `2:1 … key-duplicates` etc.; clean tree rc 0;
`LINT_JOBS=x` rc 2.
## What I did NOT verify
- **CI timings at the time of writing.** The PR's CI run is measured after push and reported
on the PR / handoff message, not here. The CI runner's CPU count is UNVERIFIED — the new
`[offline] running … jobs=… cpus=…` line in the CI log is what will show it. On a small
runner the speedup will be smaller than locally.
- **approval-policy** will be red on CI: `ci/**` and `conformance/**` are sensitive paths and
need the operator's review. I did not and must not override it.
- Flakiness under load: the selftests have no wall-clock assertions I found, and 4 toolbox
runs + several host runs were all stable, but that is a handful of runs, not a soak.
- The coverage files under `conformance/_artifacts/coverage/` end in the same state as before
only because a single check rewrites them; I did not change that pre-existing quirk.
- L2/L3 conformance: not run (no cluster in CI, as before).
@@ -0,0 +1,65 @@
---
date: 2026-11-08
branch: chore/renovate-config-cleanup
title: Renovate config — hand-migrate the docker key, track gitea/runner, keep HelmRelease range lower bounds
---
## What changed
- **`renovate.json5`: the deprecated top-level `"docker": {"enabled": true}` is hand-migrated** to
`packageRules[0] = {"matchCategories": ["docker"], "enabled": true}`, the exact shape Renovate's
own migration produced on `origin/renovate/migrate-config` (PR #139). Its comment moved with it.
This replaces #139, which was closed because its rewrite deleted every comment in the file. No
other comment was removed or reflowed (diff is additions plus the moved block).
- **`gitea/runner` added to the blast-radius rule** beside `gitea/act_runner`, with a comment:
act_runner was renamed upstream to gitea/runner after 0.6.1; Renovate does not follow renames,
so the actual image migration is a separate manual PR.
- **Two new packageRules for flux/helm `>=A <B` ranges:** `rangeStrategy: "bump"`, plus
`matchUpdateTypes: ["bump"] -> enabled: false`, so in-range releases (which Flux adopts on its
own) do not become PRs. Cause of the bug, from the 44.167.0 source: rangeStrategy is unset
(`auto`); the flux manager has no `getRangeStrategy`/`updateLockedDependency`, so
`lib/modules/manager/index.ts getRangeStrategy` resolves `auto` to `replace`; helm versioning is
the npm api verbatim, and npm `getNewValue` under `replace` turns `>=1.12.0 <1.13.0` into
`<1.14.0` (what PRs #131, #130 and #115 did).
- **`platform/headlamp/helmrelease.yaml`: `version: "<0.47.0"` -> `">=0.46.0 <0.47.0"`**, comment
refreshed. The live chart is 0.46.0, so the set of installable charts is unchanged.
## What I verified
- **Semantic diff of #139:** I parsed `renovate.json5` on `origin/renovate/migrate-config` and on its
parent `5b484f6` with json5 (the copy inside the renovate image). With `docker` removed from the
parent and `packageRules[0]` removed from the migrated file, the two are identical. So the only
change was `{"matchCategories":["docker"],"enabled":true}`, inserted at packageRules[0].
- **Validator, `ghcr.io/renovatebot/renovate:44.167.0`** (the tag that `.gitea/workflows/renovate.yaml`
pins as `container.image`): `renovate-config-validator` gives `Config validated successfully`
(rc 0), and so does `--strict` (rc 0), both as a file path and in repo mode. **`--strict` on
origin/main's file gives rc 1** and prints the `docker` migration diff. Nothing is left to migrate.
- **`getNewValue` called in the 44.167.0 image** (helm versioning), currentValue
`>=1.12.0 <1.13.0`, newVersion 1.13.0: replace `<1.14.0`, widen `>=1.12.0 <1.14.0`,
bump `>=1.13.0 <1.14.0`, pin/update-lockfile `<1.14.0`. On `<0.47.0`, bump returns the value
unchanged, so it would never propose anything for that range. That is why the rule is scoped to
the `>=A <B` shape.
- **Local dry run** (`--platform=local`, `RENOVATE_DRY_RUN=lookup`, 44.167.0, live chart repos)
against a fixture repo of four HelmReleases under `platform/`:
- With this branch's config: `No config migration necessary`. longhorn `>=1.9.0 <1.10.0` ->
`>=1.13.0 <1.14.0` (minor). kube-prometheus-stack `>=88.0.0 <89.0.0` -> `>=93.1.0 <94.0.0`
(major). The in-range bump `>=88.6.5 <89.0.0` was dropped (`Filtered out 1 disabled update(s)`).
headlamp `>=0.46.0 <0.47.0` and `<0.47.0`: no update.
- Control run with origin/main's config: `Config migration necessary`, longhorn -> `<1.14.0`,
kube-prometheus-stack -> `<94.0.0`. This reproduces the reported bug.
- Live headlamp HelmRelease (read-only kubectl, k3s-prod): spec `<0.47.0`, history chartVersion
`0.46.0`, Ready `True`.
## What I did NOT verify
- **I have not seen a real scheduled Renovate run against this repo with this config.** The claim that
Renovate stops reopening the migrate-config PR rests on the validator and on `No config migration
necessary` in the local dry run, not on the Gitea bot. After merge, watch the next run, and if
`renovate/migrate-config` is still on the remote, delete it.
- **Under bump, the new lower bound is the release Renovate proposes**, e.g. `>=1.13.2 <1.14.0`, not
`>=1.13.0`. Flux still installs the newest release in the window. Ranges with other shapes keep the
default `replace`: `3.9.x`, `~`, ceiling-only, and exact pins.
- **The open PRs #131, #115 and any others still carry unbounded `<X` values.** They were generated
under the old strategy, and Renovate should rebase or regenerate them under bump on its next run.
I have not checked that it does.
- The gitea/runner rename itself (a new image reference and any config changes) is not done here.
+48
View File
@@ -0,0 +1,48 @@
---
date: 2026-11-08
branch: rework/136
title: "PR #136 rework: regenerate L1 render goldens for the oauth2-proxy v7.15.6 sidecar bump"
---
## What changed
- Renovate PR #136 bumps `OAUTH2_PROXY_IMAGE` in `catalog/render.py` (and the generated
`apps/wave-*` / `platform/crafty` pins) from `v7.8.1` to `v7.15.6`. Its gate failed at L1
because three byte-for-byte render goldens still carried the old tag:
- `conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml`
- `conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml`
- `conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml`
(CI only reported the vpn one because `render-check.sh` stops at the first failing diff;
the internet golden had drifted too.)
- Regenerated all three in the pinned toolbox with the exact invocation the check scripts use:
`python3 catalog/render.py <fixture> > <golden>`. The diff in each is **one line**:
`image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1` -> `...:v7.15.6`. Nothing else moved.
- `platform/crafty/README.md`: the "Sidecar" row described the current pin; updated to v7.15.6.
- Deliberately NOT changed (they record history, not the current pin):
- `catalog/render.py` comments / assertion messages saying a flag was "confirmed present in the
PINNED v7.8.1 binary" or that `--api-routes` "does not exist in v7.8.1". Rewriting them to
v7.15.6 would assert a `--help` verification nobody has done against v7.15.6.
- `conformance/tests/l0-static/oauth2-proxy-api-routes-per-app/check.sh`, the schema
description, `runbooks/oauth2-proxy-session.md`, `docs/plans/...` — same reasoning.
- `platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml` — v7.8.1 appears in
synthetic kube-state-metrics series (test input data), not as a pin.
- `.gitea/workflows/renovate.yaml` comment — a dated measurement of the pre-Renovate state.
- Branch was 55 commits behind `origin/main`, but main touches neither `catalog/render.py` nor
the L1 goldens, and the merge is conflict-free; no merge of main was needed for the gate.
## What I verified
- The golden diff is exactly the image tag in each of the three files (`git diff -U0`).
- No remaining `oauth2-proxy:v7.8.1` in any rendered/generated/golden file (`git grep`).
- Local toolbox gate result: see the PR #136 comment posted with this push (rc and per-stage
results are quoted there and in the worker report).
## What I did NOT verify
- That the v7.15.6 sidecar actually starts and authenticates in the cluster with the rendered
args. Earlier research said no flag used here was removed/renamed between 7.8.1 and 7.15.6;
I did not re-run `oauth2-proxy --help` against the v7.15.6 image.
- L2/L3 conformance (does not run in CI: `CLUSTER_AVAILABLE` unset).
- `approval-policy`: `conformance/**` is a sensitive path, so this PR is CI-red until the
operator posts `/approve-override <reason>` as a PR review. The local gate cannot evaluate it.
- Whether Renovate keeps rebasing this branch now that it carries a non-Renovate commit.
+155
View File
@@ -0,0 +1,155 @@
---
date: 2026-11-08
branch: feat/runner-capacity-2
title: act_runner runs two jobs at once, and dind is sized for two
---
## What changed
`apps/wave-1/gitea-actions.yaml` only. That file is a **hand-owned** non-catalog unit
(`EXTRA_UNITS["gitea-actions"]` in `catalog/render.py`): the renderer only decides which wave
it lands in, it does not generate its bytes, and `ci/render-golden.sh` compares only files the
catalog renders. So it is edited in place, and no re-render was needed or run.
1. **New ConfigMap `act-runner-config`** holding a config file with ONE key,
`runner.capacity: 2`, mounted read-only at `/etc/act_runner/` and selected with
`CONFIG_FILE=/etc/act_runner/config.yaml`. The image's `run.sh` (act_runner v0.6.1) turns
that into `--config` for both `register` and `daemon`.
- Deltas only, on evidence: v0.6.1 `LoadDefault()` unmarshals the file onto a zero Config and
then applies the same defaults as the no-file path, so the other settings are unchanged.
- The `generate-config` example was deliberately NOT used. Its `runner.labels` would override
the registered labels in `/data/.runner` and repoint `ubuntu-latest` at
`docker.gitea.com/runner-images`. Its `force_pull: true` is not the default. Its sample `envs`
would be injected into every job.
- With a file present, act_runner ignores the legacy `GITEA_RUNNER_CAPACITY` /
`_FILE` / `_ENVIRON` / `_ENV_FILE` / `GITEA_DEBUG` / `GITEA_TRACE` env vars. None of
them is set on this Deployment.
2. **Pod-template annotation `homelab.keithjackson.dev/act-runner-config-revision: "1"`.**
act_runner reads its config once, at start, so a change to the ConfigMap alone would never
take effect. Bump this annotation whenever you change only the ConfigMap.
3. **dind resources:** requests `cpu 100m -> 2` and `memory 256Mi -> 1Gi`; limit
`memory 2Gi -> 4Gi`. There is still no CPU limit, on purpose.
4. **runner resources:** request `memory 128Mi -> 256Mi`, limit `memory 512Mi -> 1Gi`. CPU stays
at `100m`.
Why these sizes. Measured 2026-11-08 through Prometheus, at capacity 1:
| container | peak CPU (1m rate, 30d) | peak working set (30d) | OOM events (7d) |
|---|---|---|---|
| dind | 0.69 core | 447 MiB | 0 |
| runner | 0.24 core | 314 MiB | 0 |
At the moment of measurement the pod was idle (`top`: dind 1m/138Mi, runner 1m/83Mi), on
k3s-prod-2.
- **CPU request 2.** Two jobs at today's peak is about 1.4 cores. The gate is being parallelised
in other work, so each job will use more than that. The request sets dind's CFS weight under
contention; at 100m it got a 1/20 share. With no limit, dind can still use spare cores on an
idle node.
- **Memory 1Gi / 4Gi.** Two jobs at today's peak is about 0.9 GiB. If dind is OOM-killed,
dockerd goes down and takes both running jobs with it, so the limit is about 4x that.
- **Runner limit 512Mi -> 1Gi.** Its 30-day peak was already 61% of the old limit, and with
capacity 2 it holds state for two tasks.
Scheduling headroom: CPU requests per node were measured with `describe node`. The pod's total
CPU request goes from 200m to 2100m (+1.9 cores).
| node | CPU requested now | after the change | allocatable |
|---|---|---|---|
| prod-2 (current) | 2945m | ~4845m (61%) | 8 |
| prod-1 | 3175m | ~5075m | 8 |
| prod-3 | 3840m | ~5740m (72%) | 8 |
The pod fits on any of the three if the RWO PVC moves it.
Concurrency semantics. Workflow `concurrency:` groups are enforced by Gitea on the server, not by
the runner, so capacity does not change them:
- `gate.yml` `gate-${{ github.ref }}` with cancel-in-progress still cancels a superseded run on
the same ref.
- `renovate` (group `renovate`, no cancel) and `images-${{ github.ref }}` still serialise
themselves.
What capacity 2 does change: runs on *different* refs (a PR gate next to a renovate or images
run, or two PRs) now run side by side instead of queueing. The required check
`gate / gate (pull_request)` is still one job per PR head, and gate.yml's other job
(`conformance`) is `if: false`, so it never runs.
Both jobs share one dockerd. act_runner gives each task its own container prefix
(`GITEA-ACTIONS-TASK-<task id>`) and its own network, and no workflow uses fixed container names
or host ports.
## What I verified
- **Source of truth.** `catalog/render.py` `EXTRA_UNITS` places `gitea-actions.yaml` by
anchor/offset only. `ci/render-golden.sh` iterates over rendered files only, so this file is
outside render==golden. No `--emit-waves` was run.
- **act_runner v0.6.1 behaviour, read in its source** (tag v0.6.1):
- `config.LoadDefault()` treats a partial file as described above.
- `daemon.go`: config labels override `.runner` labels only when non-empty.
- `poller.go`: a semaphore of `cfg.Runner.Capacity` bounds concurrent tasks.
- `runner.go`: per-task container name prefix.
- The image's `run.sh`, read from `gitea/act_runner:0.6.1`, honours `CONFIG_FILE`.
- **The live pod has no config file today.** `/data` holds only `.runner` and `lost+found`.
`act_runner --version` reports v0.6.1. No LimitRange or ResourceQuota exists in
`gitea-actions`.
- **Resource numbers.** All figures above come from read-only `kubectl top`, `describe node` and
Prometheus queries against k3s-prod.
- **dind disk.** The emptyDir sits on the node's 11.8G ephemeral filesystem: 4.0G used, 7.2G
free. Image cache is 2.18GB, which is the one `catthehacker/ubuntu:act-22.04` image. Two
concurrent jobs share that image.
- **Embedded config.** It parses as YAML to `{'runner': {'capacity': 2}}`. The only `$` in the
manifest is the existing `${BASE_DOMAIN}`, so nothing new is exposed to Flux postBuild
envsubst.
- **Approval policy.** `apps/wave-1/` matches no `sensitive_paths` glob in
`infra/gitea/approval-policy.json`, so this PR merges on a green gate alone.
- **Local gate.** Pinned toolbox, run from the worktree on the committed tree. The rc and skipped
stages are given in the PR description.
## What I did NOT verify
- **UNVERIFIED: that the runner actually starts with the config and runs two jobs at once.**
Nothing is applied until Flux reconciles `main`.
- **Capacity cannot be seen through Gitea's API.** act_runner v0.6.1 does not report capacity to
Gitea: `Declare` sends only version and labels. `GET /api/v1/admin/actions/runners` therefore
cannot show "capacity 2". Concurrency has to be observed instead.
- **UNVERIFIED: whether the parallelised gate fits in these sizes.** That gate has not landed.
Re-measure after it does, with
`max_over_time(sum by (container)(rate(container_cpu_usage_seconds_total{namespace="gitea-actions"}[1m]))[7d:30s])`.
- **Rollout cancels in-flight jobs.** The rollout restarts the runner pod (strategy Recreate,
`shutdown_timeout: 0s`), and any job running at that moment is cancelled, possibly including
the `push`-to-main gate run triggered by this very merge. Re-run it if that happens.
- **dind disk is not bounded by a limit.** With two jobs it holds two jobs' workspaces and layers
on the same 11.8G node filesystem. I did not add an `ephemeral-storage` limit, because an
eviction would kill both jobs. Watch kubelet DiskPressure on the runner's node.
- **Stale comments in sensitive paths.** `.gitea/workflows/gate.yml` (line ~52) and
`.gitea/workflows/README.md` (line ~86) still say "`act_runner` runs at capacity 1". I left
them alone because `.gitea/**` is a sensitive path and editing them would put this PR behind
human approval. Fix them in a separate PR.
### Operator post-merge checklist
1. Flux applied it:
`kubectl --context k3s-prod -n flux-system get kustomization apps-wave-1 -o jsonpath='{.status.lastAppliedRevision}'`
equals the merge commit.
2. The pod restarted with the config:
- `kubectl --context k3s-prod -n gitea-actions get pod -l app.kubernetes.io/name=act-runner`
shows a new pod, 2/2 Running.
- `kubectl ... exec deploy/act-runner -c runner -- cat /etc/act_runner/config.yaml` shows
`capacity: 2`.
- `kubectl ... exec deploy/act-runner -c runner -- ps` shows `act_runner daemon --config /etc/act_runner/config.yaml`.
- The logs show "declare successfully" with the two expected labels (`ubuntu-latest`,
`homelab` -> `catthehacker/ubuntu:act-22.04`). That confirms the labels did not change.
3. The runner is registered and online:
- `GET /api/v1/admin/actions/runners` lists `incluster-dind`, online, with the same ID as
before (no duplicate row).
- Capacity is NOT shown there; see above.
4. Concurrency is real: trigger two runs on different refs (e.g. two PR gates, or a PR gate plus a
`workflow_dispatch` of renovate). Then either:
- in `GET /api/v1/repos/Home-Network/home-network-k3s-impl/actions/tasks`, confirm their
`run_started_at` .. `updated_at` windows overlap; or
- check
`docker --host=tcp://127.0.0.1:2376 --tlsverify ... ps` in the dind container lists two
`GITEA-ACTIONS-TASK-*` container sets at once.
5. Resources: `kubectl --context k3s-prod -n gitea-actions describe pod` shows dind requests
cpu 2 / memory 1Gi, limit memory 4Gi, and runner limit memory 1Gi.
Loaded 100 of 176 files, more files were not shown because too many files have changed in this diff. Show more