fix(ci): kill the false-green base resolution; cover R-SEC-01/02 + R-TEST-06 (L0)
Unblocks the enforced Gitea gate, which correctly BLOCKED this branch on ['R-SEC-01','R-SEC-02','R-TEST-06'] being touched but uncovered. Tests were authored; the coverage rule was NOT narrowed. === THE FALSE GREEN (root cause) === `make -C ci gate` locally reported PASS having checked NOTHING. resolve_base() validated nothing, and EVERY way of failing to find a base degraded silently to an empty touched set + exit 0: an `HEAD~1` last resort, a two-dot diff fallback masking a failing three-dot diff, and no check that HEAD is even AHEAD of the base. Run while sitting on `main` (main == origin/main), `origin/main...HEAD` is empty -> "no changed files vs base" -> the empty set means run.sh is invoked with NO --requirements -> coverage.py's --require gate never runs at all. Green by vacuum. Now: candidates are BASE_REF > GITHUB_BASE_REF > origin/main > main, each verified to resolve. HEAD~1 and the two-dot fallback are gone. Unresolvable base / no merge base / failed diff / HEAD-not-ahead -> loud banner, EMPTY stdout, exit 3, and a machine-readable reason (on-base | no-base | diff-failed). gate.sh turns exit 3 into an UNDECIDED decision: `on-base` -> "NOT APPLICABLE — no branch under gate" (exit 0 so push-to-main runs don't break, but it never reads as PASS); `no-base`/`diff-failed` -> BLOCKED. It also now prints the diff base it used. A stale local main warns (widens, never falsely greens). Also fixed `make -C ci test`, which was SWALLOWING failing suites. Regression: ci/tests/touched-requirements-base.test.sh — 31 assertions over 8 throwaway-repo scenarios including the exact false-green case, an already-merged branch, no base ref, a single-commit repo, bogus BASE_REF, a shallow GITHUB_BASE_REF, and origin/main-preferred-over-stale-main. === NEW L0 STATIC LAYER (conformance/tests/l0-static/) === Where docs/14 actually puts these invariants. Each check.sh really executes offline via `run.sh --layer l0` and is wired into ci/l0.sh, so it runs on every PR. No scaffolds. R-SEC-01 no-plaintext-secrets (165 assertions / 445 tracked files): .sops.yaml holds only PUBLIC age recipients; every committed Secret is SOPS-wrapped with a mac and ENC[...] values; ansible vault.yml files are $ANSIBLE_VAULT; no secret-valued ansible key carries a literal outside a vault; no PEM key / age secret key / kubeconfig client-key-data is tracked; Flux declares sops decryption; the disallow-plaintext-secrets policy Enforces and has a proving `result: fail` row. Exemptions are SELF-POLICING, not path allowlists — a .example must be referenced by no kustomization and hold only placeholders, and the Kyverno fixture is exempt ONLY while a test asserts the policy rejects it. Proven against 7 injected violations, including deleting the negative row and weakening Enforce->Audit. R-SEC-02 least-privilege-rbac (41 assertions): no wildcard apiGroup/resource/verb and no escalate|bind|impersonate in any in-repo Role/ClusterRole; no ServiceAccount bound to cluster-admin or a wildcard role; the conformance runner's own test identity is scoped and holds no rights over RBAC objects; the default-deny/host-namespace/egress-class policies Enforce and each carry a negative row. Proven against 6 injected violations. R-TEST-06 spec-immutability-enforced (16 assertions): drives the REAL ci/spec-immutability.sh against HERMETIC throwaway spec repos and asserts its exit code in every state — clean-at-pin 0, edits under docs/ or decisions/ 1, untracked addition 1, unadopted drift 1, explicit pin bump 0, ABSENT spec repo 3 (honest skip, never green), unpinned SPEC_PIN 1 — plus that gate.sh sets BLOCK=1 on its failure, so the mechanism is load-bearing. The real spec repo is never written to. Each check documents what it does NOT cover (runtime decryption, git history, runtime least-privilege, branch protection) rather than overclaiming. status.py now reads check.sh too, so a placeholder L0 body still outranks an `executable: true` annotation. Coverage 37/51 -> 40/51 (11 gaps); executable tests 32 -> 35; scaffold-only stays 5 — no sixth scaffold was added. Serves: R-SEC-01, R-SEC-02, R-TEST-01..06. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
9f59c36000
commit
6e77150ae1
21 files changed
+1450
-97
No files matched your search
@@ -0,0 +1,43 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — no plaintext secrets in git (R-SEC-01)
|
||||
# Catalog: docs/14 L0 row, "Manifests/ServiceDefinitions obey invariants ...
|
||||
# no plaintext secrets" — runs in CI WITHOUT a cluster.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every tracked file is inspected,
|
||||
# THEN every committed Secret is SOPS ciphertext, every ansible group_vars vault is
|
||||
# ansible-vault encrypted, no private key material is tracked, decryption is
|
||||
# delegated to an in-cluster SOPS key, and the L0 policy that rejects plaintext
|
||||
# Secrets exists, Enforces, and is proven by a negative test row to reject one.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git ls-files +
|
||||
# PyYAML, no cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it
|
||||
# directly, and this Chainsaw Test wraps the SAME script so an in-cluster run executes
|
||||
# the identical assertions. Encryption at rest is necessary but NOT sufficient: the
|
||||
# runtime half of "decrypted only in-cluster", and a scan of git HISTORY, are named as
|
||||
# explicit gaps in check.sh's header and are NOT claimed here.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: no-plaintext-secrets
|
||||
labels:
|
||||
conformance.home.k3s/R-SEC-01: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-SEC-01"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
No secret is stored in plaintext in git: committed Secrets are SOPS ciphertext,
|
||||
ansible vaults are encrypted, no private key material is tracked, and the
|
||||
no-plaintext-secrets policy gate is enforced and proven to reject.
|
||||
spec:
|
||||
steps:
|
||||
- name: scan-tracked-tree-for-plaintext-secrets
|
||||
description: >-
|
||||
Assert over the whole tracked tree that no secret material is stored in
|
||||
plaintext and that decryption is delegated to the in-cluster SOPS key.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
+287
@@ -0,0 +1,287 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — no-plaintext-secrets
|
||||
# Serves: R-SEC-01 ("Secrets MUST NOT be stored in plaintext in git; they MUST be
|
||||
# encrypted at rest in the repo and decrypted only in-cluster."
|
||||
# — spec docs/00-vision-and-scope.md; detail in docs/07.)
|
||||
#
|
||||
# docs/14 puts exactly this invariant at L0 ("Static/policy | Pre-apply, in CI |
|
||||
# ... no plaintext secrets"), so it is asserted here over the WHOLE tracked tree,
|
||||
# with no cluster. Run by `conformance/run.sh --layer l0` and by ci/l0.sh.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every tracked file is inspected,
|
||||
# THEN (1) SOPS is configured at the repo root with a PUBLIC age recipient;
|
||||
# (2) every committed Kubernetes Secret is SOPS-encrypted — a top-level
|
||||
# `sops:` block with a mac AND every data/stringData value is ENC[...];
|
||||
# (3) the only two exempt classes are narrow and self-policing:
|
||||
# a. `*.example` templates, which must NOT be referenced by any
|
||||
# kustomization and whose secret-named values must be placeholders;
|
||||
# b. the Kyverno L0 negative fixture, which is exempt ONLY while it is
|
||||
# actually wired into a kyverno-test.yaml that asserts
|
||||
# `disallow-plaintext-secrets` REJECTS it (result: fail);
|
||||
# (4) every ansible-vault group_vars file is `$ANSIBLE_VAULT;`-encrypted, and
|
||||
# no secret-valued ansible key carries a literal outside a vault;
|
||||
# (5) no private key material of any kind is tracked (PEM private keys, an
|
||||
# age SECRET key, kubeconfig client-key-data);
|
||||
# (6) decryption is delegated to the cluster: a Flux Kustomization declares
|
||||
# `decryption.provider: sops` with a secretRef, and the private half of
|
||||
# that key pair is nowhere in the repo (5).
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gap, cluster layer):
|
||||
# - that the decrypted Secret only ever materialises in-cluster (the runtime half
|
||||
# of "decrypted only in-cluster") — needs a live cluster (L2), untested today.
|
||||
# - that no plaintext secret exists in git HISTORY. This walks the WORKING TREE at
|
||||
# HEAD only; docs/07's history-scan acceptance gate needs a secret scanner over
|
||||
# all revisions and is NOT implemented here.
|
||||
# - value-level entropy: a placeholder-looking string that is in fact a real
|
||||
# credential is not detectable statically.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
import os, re, subprocess, sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def tracked() -> list[str]:
|
||||
out = subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
|
||||
capture_output=True, text=True, check=True).stdout
|
||||
return [p for p in out.split("\0") if p]
|
||||
|
||||
|
||||
FILES = tracked()
|
||||
check(len(FILES) > 50, f"git ls-files returned only {len(FILES)} paths — not a real checkout")
|
||||
|
||||
|
||||
def docs_of(rel: str):
|
||||
"""Parsed YAML documents of a tracked file, or [] when it is not parseable YAML."""
|
||||
p = ROOT / rel
|
||||
try:
|
||||
return [d for d in yaml.safe_load_all(p.read_text(errors="replace")) if isinstance(d, dict)]
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
|
||||
YAMLISH = re.compile(r"\.(ya?ml)(\.example)?$")
|
||||
YAML_FILES = [f for f in FILES if YAMLISH.search(f)]
|
||||
|
||||
# ---------------------------------------------------------------- (1) SOPS config
|
||||
sops_cfg = ROOT / ".sops.yaml"
|
||||
check(sops_cfg.is_file(), ".sops.yaml is missing from the repository root (R-SEC-01)")
|
||||
if sops_cfg.is_file():
|
||||
cfg = yaml.safe_load(sops_cfg.read_text()) or {}
|
||||
rules = cfg.get("creation_rules") or []
|
||||
check(len(rules) > 0, ".sops.yaml declares no creation_rules — new secrets would be committed unencrypted")
|
||||
for r in rules:
|
||||
pr = r.get("path_regex", "<none>")
|
||||
recips = str(r.get("age", ""))
|
||||
check(bool(recips), f".sops.yaml creation_rule {pr!r} has no age recipient")
|
||||
check(all(x.startswith("age1") for x in re.split(r"[,\s]+", recips) if x),
|
||||
f".sops.yaml creation_rule {pr!r} recipient is not a PUBLIC age1... key")
|
||||
check("AGE-SECRET-KEY" not in recips.upper(),
|
||||
f".sops.yaml creation_rule {pr!r} contains an age PRIVATE key")
|
||||
|
||||
# --------------------------------------------- Kyverno negative-fixture exemption
|
||||
# A plaintext Secret is tolerated ONLY where it is the negative case that proves the
|
||||
# L0 policy rejects plaintext. We do not trust the path: the file must be listed in a
|
||||
# kyverno-test.yaml whose results assert disallow-plaintext-secrets returns `fail`.
|
||||
POLICY_NAME = "disallow-plaintext-secrets"
|
||||
negative_fixtures: set[str] = set()
|
||||
kyverno_suites = [f for f in FILES if os.path.basename(f) == "kyverno-test.yaml"]
|
||||
check(len(kyverno_suites) > 0, "no kyverno-test.yaml found — the L0 policy gate is not exercised at all")
|
||||
for suite in kyverno_suites:
|
||||
docs = docs_of(suite)
|
||||
if not docs:
|
||||
continue
|
||||
d = docs[0]
|
||||
results = d.get("results") or []
|
||||
rejects_plaintext = any(r.get("policy") == POLICY_NAME and r.get("result") == "fail"
|
||||
for r in results if isinstance(r, dict))
|
||||
if not rejects_plaintext:
|
||||
continue
|
||||
base = os.path.dirname(suite)
|
||||
for res in d.get("resources") or []:
|
||||
negative_fixtures.add(os.path.normpath(os.path.join(base, str(res))))
|
||||
|
||||
check(len(negative_fixtures) > 0,
|
||||
f"no kyverno-test.yaml asserts that {POLICY_NAME} REJECTS a plaintext Secret "
|
||||
"(result: fail) — the plaintext-secret gate is unproven, so nothing may be exempt")
|
||||
|
||||
# --------------------------------------------------- (2)/(3) committed Secret scan
|
||||
SECRET_KEY_RE = re.compile(r"(pass|secret|token|key|credential|cookie|cert)", re.I)
|
||||
PLACEHOLDER_RE = re.compile(r"^\s*(|<[^>]*>|replace[-_ ]?me.*|change[-_ ]?me.*|placeholder.*|x{3,})\s*$", re.I)
|
||||
|
||||
secrets_seen = 0
|
||||
for rel in YAML_FILES:
|
||||
for d in docs_of(rel):
|
||||
if d.get("kind") != "Secret":
|
||||
continue
|
||||
secrets_seen += 1
|
||||
values = []
|
||||
for field in ("data", "stringData"):
|
||||
for k, v in (d.get(field) or {}).items():
|
||||
values.append((k, "" if v is None else str(v)))
|
||||
|
||||
is_example = rel.endswith(".example")
|
||||
is_negative_fixture = os.path.normpath(rel) in negative_fixtures
|
||||
|
||||
if is_example:
|
||||
# Template: never applied, and its secret-named values must be placeholders.
|
||||
refs = [k for k in YAML_FILES
|
||||
if os.path.basename(k) == "kustomization.yaml"
|
||||
and os.path.basename(rel) in (ROOT / k).read_text(errors="replace")]
|
||||
check(not refs,
|
||||
f"{rel}: an unencrypted .example template is referenced by a kustomization "
|
||||
f"({refs}) — it would be applied as a plaintext Secret")
|
||||
for k, v in values:
|
||||
if SECRET_KEY_RE.search(k):
|
||||
check(bool(PLACEHOLDER_RE.match(v)),
|
||||
f"{rel}: template key {k!r} holds {v[:24]!r}, not a placeholder — "
|
||||
"a real credential may have been committed (R-SEC-01)")
|
||||
continue
|
||||
|
||||
if is_negative_fixture:
|
||||
# Exempt, but only as the proven negative case; assert it really is plaintext
|
||||
# (an encrypted 'bad' fixture would silently stop testing anything).
|
||||
continue
|
||||
|
||||
# Everything else: must be SOPS-encrypted at rest.
|
||||
sops_block = d.get("sops")
|
||||
check(isinstance(sops_block, dict) and bool(sops_block.get("mac")),
|
||||
f"{rel}: committed Secret {d.get('metadata', {}).get('name')!r} has no top-level "
|
||||
"`sops:` block with a mac — it is stored in plaintext (R-SEC-01)")
|
||||
for k, v in values:
|
||||
check(v.startswith("ENC["),
|
||||
f"{rel}: Secret value {k!r} is not SOPS ciphertext (expected ENC[...]) — "
|
||||
"plaintext secret material in git (R-SEC-01)")
|
||||
|
||||
check(secrets_seen > 0, "no Kubernetes Secret manifests found at all — the scan is not looking at the repo")
|
||||
|
||||
# ------------------------------------------------------------ (4) ansible vaults
|
||||
VAULT_HEADER = "$ANSIBLE_VAULT;"
|
||||
vault_files = [f for f in FILES
|
||||
if os.path.basename(f) in ("vault.yml", "vault.yaml")]
|
||||
check(len(vault_files) > 0, "no ansible vault.yml files found — the vault scan is not looking at the repo")
|
||||
for rel in vault_files:
|
||||
head = (ROOT / rel).read_text(errors="replace")[:len(VAULT_HEADER)]
|
||||
check(head == VAULT_HEADER,
|
||||
f"{rel}: an ansible group_vars vault file is NOT ansible-vault encrypted "
|
||||
"(must start with the $ANSIBLE_VAULT header) — R-SEC-01")
|
||||
|
||||
SECRET_ANSIBLE_KEYS = ("ansible_become_password", "ansible_become_pass", "ansible_password",
|
||||
"ansible_ssh_pass", "ansible_sudo_pass", "ansible_ssh_private_key_file")
|
||||
ASSIGN_RE = re.compile(
|
||||
r"^\s*(" + "|".join(SECRET_ANSIBLE_KEYS) + r")\s*:\s*(.*?)\s*$")
|
||||
for rel in FILES:
|
||||
p = ROOT / rel
|
||||
if not p.is_file() or p.is_symlink():
|
||||
continue
|
||||
try:
|
||||
text = p.read_text(errors="replace")
|
||||
except Exception:
|
||||
continue
|
||||
if text.startswith(VAULT_HEADER):
|
||||
continue # encrypted at rest: opaque, by design
|
||||
for lineno, line in enumerate(text.splitlines(), 1):
|
||||
if line.lstrip().startswith("#"):
|
||||
continue
|
||||
m = ASSIGN_RE.match(line)
|
||||
if not m:
|
||||
continue
|
||||
key, val = m.group(1), m.group(2).strip().strip("'\"")
|
||||
ok = (val.startswith("{{") or val.startswith("$")
|
||||
or bool(PLACEHOLDER_RE.match(val)) or val == "")
|
||||
check(ok, f"{rel}:{lineno}: {key} is assigned the literal {val[:24]!r} outside an "
|
||||
"ansible-vault file — a plaintext secret in git (R-SEC-01)")
|
||||
|
||||
# ------------------------------------------------------ (5) no private key material
|
||||
# NOTE: each marker is assembled from fragments so that THIS file never contains the
|
||||
# literal string it hunts for — otherwise the scanner would flag itself the moment it
|
||||
# is tracked, and the only way to silence it would be to exempt the scanner (which is
|
||||
# exactly the kind of self-exemption this suite exists to prevent).
|
||||
_DASH5 = "-" * 5
|
||||
PRIVKEY_PATTERNS = [
|
||||
(re.compile(_DASH5 + r"BEGIN (?:[A-Z ]+ )?PRIVATE" + r" KEY" + _DASH5),
|
||||
"a PEM private key block"),
|
||||
(re.compile("AGE" + "-SECRET" + "-KEY-1"), "an age SECRET (decryption) key"),
|
||||
(re.compile(r"^\s*client" + r"-key-data\s*:\s*\S"), "kubeconfig client-key-data"),
|
||||
]
|
||||
for rel in FILES:
|
||||
p = ROOT / rel
|
||||
if not p.is_file() or p.is_symlink():
|
||||
continue
|
||||
try:
|
||||
text = p.read_text(errors="replace")
|
||||
except Exception:
|
||||
continue
|
||||
if text.startswith(VAULT_HEADER):
|
||||
continue
|
||||
for rx, what in PRIVKEY_PATTERNS:
|
||||
for lineno, line in enumerate(text.splitlines(), 1):
|
||||
if line.lstrip().startswith("#"):
|
||||
continue
|
||||
if rx.search(line):
|
||||
check(False, f"{rel}:{lineno}: tracked file contains {what} (R-SEC-01)")
|
||||
|
||||
# ----------------------------------------- (6) decryption is delegated to the cluster
|
||||
sops_decryption = []
|
||||
for rel in YAML_FILES:
|
||||
for d in docs_of(rel):
|
||||
if d.get("kind") != "Kustomization":
|
||||
continue
|
||||
dec = d.get("spec", {}).get("decryption") if isinstance(d.get("spec"), dict) else None
|
||||
if isinstance(dec, dict) and dec.get("provider") == "sops" and dec.get("secretRef"):
|
||||
sops_decryption.append(rel)
|
||||
# Flux overlay patches carry the same block inside a patch body.
|
||||
for rel in YAML_FILES:
|
||||
text = (ROOT / rel).read_text(errors="replace")
|
||||
if re.search(r"^\s*decryption:\s*$", text, re.M) and re.search(r"^\s*provider:\s*sops\s*$", text, re.M) \
|
||||
and re.search(r"^\s*secretRef:\s*$", text, re.M):
|
||||
sops_decryption.append(rel)
|
||||
check(len(sops_decryption) > 0,
|
||||
"no Flux Kustomization declares `decryption: {provider: sops, secretRef: ...}` — nothing "
|
||||
"proves decryption happens in-cluster rather than from committed key material (R-SEC-01)")
|
||||
|
||||
# ------------------------------------------- (7) the L0 policy gate is real and strict
|
||||
policy_files = [f for f in FILES if f.endswith(f"{POLICY_NAME}.yaml")]
|
||||
check(len(policy_files) == 1, f"expected exactly one {POLICY_NAME} ClusterPolicy, found {policy_files}")
|
||||
for rel in policy_files:
|
||||
d = (docs_of(rel) or [{}])[0]
|
||||
action = str(d.get("spec", {}).get("validationFailureAction", ""))
|
||||
check(action == "Enforce" or action.startswith("${"),
|
||||
f"{rel}: validationFailureAction is {action!r} — the no-plaintext-secrets gate must "
|
||||
"Enforce (or carry the Flux token that substitutes to Enforce)")
|
||||
|
||||
# --------------------------------------------------------------------------- verdict
|
||||
if FAILURES:
|
||||
print(f"CHECK FAIL (no-plaintext-secrets): {len(FAILURES)} of {CHECKS} assertions failed",
|
||||
file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(f"check OK: no-plaintext-secrets — {CHECKS} assertions over {len(FILES)} tracked files, "
|
||||
f"{secrets_seen} committed Secret document(s), {len(vault_files)} ansible vault file(s)")
|
||||
PY
|
||||
Reference in new issue
Block a user