Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 6 Actions Packages Projects Releases Wiki Activity
465 Commits 140 Branches 0 Tags
34849559032da8b2c31ae16ba14333647934e030
Commit Graph
7 Commits
Author SHA1 Message Date
jakman85andClaude Opus 5 6e77150ae1 fix(ci): kill the false-green base resolution; cover R-SEC-01/02 + R-TEST-06 (L0) 
Unblocks the enforced Gitea gate, which correctly BLOCKED this branch on
['R-SEC-01','R-SEC-02','R-TEST-06'] being touched but uncovered. Tests were authored;
the coverage rule was NOT narrowed.

=== THE FALSE GREEN (root cause) ===
`make -C ci gate` locally reported PASS having checked NOTHING. resolve_base() validated
nothing, and EVERY way of failing to find a base degraded silently to an empty touched
set + exit 0: an `HEAD~1` last resort, a two-dot diff fallback masking a failing
three-dot diff, and no check that HEAD is even AHEAD of the base. Run while sitting on
`main` (main == origin/main), `origin/main...HEAD` is empty -> "no changed files vs
base" -> the empty set means run.sh is invoked with NO --requirements -> coverage.py's
--require gate never runs at all. Green by vacuum.

Now: candidates are BASE_REF > GITHUB_BASE_REF > origin/main > main, each verified to
resolve. HEAD~1 and the two-dot fallback are gone. Unresolvable base / no merge base /
failed diff / HEAD-not-ahead -> loud banner, EMPTY stdout, exit 3, and a machine-readable
reason (on-base | no-base | diff-failed). gate.sh turns exit 3 into an UNDECIDED
decision: `on-base` -> "NOT APPLICABLE — no branch under gate" (exit 0 so push-to-main
runs don't break, but it never reads as PASS); `no-base`/`diff-failed` -> BLOCKED. It
also now prints the diff base it used. A stale local main warns (widens, never falsely
greens). Also fixed `make -C ci test`, which was SWALLOWING failing suites.

Regression: ci/tests/touched-requirements-base.test.sh — 31 assertions over 8
throwaway-repo scenarios including the exact false-green case, an already-merged branch,
no base ref, a single-commit repo, bogus BASE_REF, a shallow GITHUB_BASE_REF, and
origin/main-preferred-over-stale-main.

=== NEW L0 STATIC LAYER (conformance/tests/l0-static/) ===
Where docs/14 actually puts these invariants. Each check.sh really executes offline via
`run.sh --layer l0` and is wired into ci/l0.sh, so it runs on every PR. No scaffolds.

R-SEC-01 no-plaintext-secrets (165 assertions / 445 tracked files): .sops.yaml holds only
PUBLIC age recipients; every committed Secret is SOPS-wrapped with a mac and ENC[...]
values; ansible vault.yml files are $ANSIBLE_VAULT; no secret-valued ansible key carries
a literal outside a vault; no PEM key / age secret key / kubeconfig client-key-data is
tracked; Flux declares sops decryption; the disallow-plaintext-secrets policy Enforces
and has a proving `result: fail` row. Exemptions are SELF-POLICING, not path allowlists —
a .example must be referenced by no kustomization and hold only placeholders, and the
Kyverno fixture is exempt ONLY while a test asserts the policy rejects it. Proven against
7 injected violations, including deleting the negative row and weakening Enforce->Audit.

R-SEC-02 least-privilege-rbac (41 assertions): no wildcard apiGroup/resource/verb and no
escalate|bind|impersonate in any in-repo Role/ClusterRole; no ServiceAccount bound to
cluster-admin or a wildcard role; the conformance runner's own test identity is scoped
and holds no rights over RBAC objects; the default-deny/host-namespace/egress-class
policies Enforce and each carry a negative row. Proven against 6 injected violations.

R-TEST-06 spec-immutability-enforced (16 assertions): drives the REAL
ci/spec-immutability.sh against HERMETIC throwaway spec repos and asserts its exit code
in every state — clean-at-pin 0, edits under docs/ or decisions/ 1, untracked addition 1,
unadopted drift 1, explicit pin bump 0, ABSENT spec repo 3 (honest skip, never green),
unpinned SPEC_PIN 1 — plus that gate.sh sets BLOCK=1 on its failure, so the mechanism is
load-bearing. The real spec repo is never written to.

Each check documents what it does NOT cover (runtime decryption, git history, runtime
least-privilege, branch protection) rather than overclaiming. status.py now reads
check.sh too, so a placeholder L0 body still outranks an `executable: true` annotation.

Coverage 37/51 -> 40/51 (11 gaps); executable tests 32 -> 35; scaffold-only stays 5 —
no sixth scaffold was added.

Serves: R-SEC-01, R-SEC-02, R-TEST-01..06.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-08-31 23:26:49 -05:00
jakman85andClaude Opus 5 1252823d80 feat(conformance): repo-derived spec status report (conformance/run.sh --status) 
Operator ruling: "I care about the state of the repository and what the status of each
spec is based on that state" — not a hand-maintained task board. This is that report.

conformance/lib/status.py (stdlib-only, --selftest) sits beside lib/coverage.py and
IMPORTS it rather than forking the registry/test parsers. Driven by a new
`conformance/run.sh --status`; artifacts land in _artifacts/status/ alongside coverage
and are git-ignored by the existing convention. Exit codes follow ci/lib.sh (0/1/3);
--strict-coverage also applies the status hygiene gate.

Signals derived per requirement: is there an implementing artifact; is it RECONCILED
(walk resources: from clusters/dev, expand each Flux Kustomization's spec.path, recurse
— so a commented-out resource correctly does not count); is there a tagged test and is
its body executable or a TODO scaffold (TODO markers OUTRANK an executable: "true"
annotation — ambiguity never upgrades a row).

DELIBERATELY NOT DERIVED, and reported as such rather than defaulting to green:
proof-of-pass (nothing tracked in git records a run result — _artifacts/ and results/
are git-ignored) and cluster liveness. "Reconciled" means Flux WOULD apply it, not that
it is healthy. PROVEN is reported as "not derivable" for all 51.

WHAT IT SHOWS THAT coverage.md DOES NOT. Coverage reports 37/51 covered, 0
safety-critical gaps. Three of those claims do not survive repo state:

1. 5 of the 37 "covered" requirements have only a TODO-body scaffold (R-STATE-03,
   R-OBS-02, R-OBS-04, R-GITOPS-03, R-GITOPS-02). Coverage counts a tagged file; it
   never looks inside it.
2. R-STATE-03 is SAFETY-CRITICAL and its L3 fault test asserts nothing. Coverage reports
   "0 safety-critical gaps" only because dr-restore/chainsaw-test.yaml exists and carries
   an L3 label. --strict now fails on this.
3. R-UPD-01/02/03 are NOT reconciled: platform/image-automation/ is commented out of
   platform/kustomization.yaml. Coverage shows them covered with executable L3 tests,
   which reads as delivered; automated updates are actually 0/3 wired+testable.

Headline: 51 requirements | reconciled 20 (+12 partial) | executable test 32 |
wired+testable 22 (+8 partial) | scaffold-only 5 | PROVEN 0 (not derivable).

Also surfaces 18 requirements with no reconciled manifest citing their R-ID — listed as
a traceability gap (R-TEST-04) rather than guessed either way. Annotating those manifests
makes the report self-correct, so it acts as a forcing function instead of another
hand-maintained map.

Serves: R-TEST-01..06. Gate: MERGE GATE: PASS.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-08-31 22:44:10 -05:00
jakman85andClaude Opus 4.8 fdf5c69adc test(conformance): add L3 node-loss test for R-STATE-02 (Longhorn node-loss resilience) 
R-STATE-02 was the last safety-critical requirement with no verifying test,
exempted via DOCUMENTED_GAPS. Author the L3 fault-injection test that proves the
guarantee and remove the exemption.

tests/l3-fault-injection/node-loss/chainsaw-test.yaml:
  GIVEN a 3-replica Longhorn volume with written data attached on node N,
  WHEN  N is removed from service (cordon + evict the attached workload, forcing
        the volume to detach from N),
  THEN  a fresh workload on a SURVIVING node re-attaches the same volume and reads
        the canary intact — volume stays available, cluster reconverges.

Safe-by-construction for the 3-node embedded-etcd dev cluster:
  - Disruption scoped to EXACTLY ONE node, at the scheduling layer only (cordon +
    evict our own pod). kubelet/etcd stay up — etcd quorum is never touched.
  - Never `kubectl drain`; only our own writer pod is deleted, so no unrelated or
    system pod is disturbed.
  - Victim node labelled conformance.home.k3s/cordoned-by=<test-ns>; `finally`
    ALWAYS uncordons every node with THAT label and removes it — restored even on
    failure/timeout, and never touches an operator-cordoned node.
  - Self-skips green until a longhorn StorageClass AND >= 3 Ready nodes exist.

Bookkeeping:
  - run.sh: remove R-STATE-02 from DOCUMENTED_GAPS; add node-loss to CONF_HEAVY_TESTS
    (serial pass, matches the other Longhorn/KubeVirt L3 tests).
  - incluster/rbac.yaml: enable the scoped nodes get/list/patch rule the test needs
    (cordon/uncordon/label); no eviction rights (no drain).
  - README.md / coverage.py docstring: refresh the documented-gap example.

Coverage: 36/51 -> 37/51 (70.6% -> 72.5%); safety-critical gaps 1 -> 0. Only
R-STATE-02 changed ([] -> node-loss); no other requirement or golden touched.

NOTE: LIVE execution is DEFERRED to an operator-chosen maintenance window. This
change AUTHORS + statically validates the test (renders, chainsaw-lints, harness
selects it, coverage updates). The static gate runs L2/L3 as SKIP without a
cluster (expected). The node-disruption test was NOT run live.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
 2026-08-27 18:12:33 -05:00
jakman85andClaude Opus 4.8 948c390272 conformance: serialize heavy L2/L3 tests + author alerting SMTP sink 
TASK 1 — reliably single-pass green L2/L3:
Split run_chainsaw into a LIGHT pass (concurrent, config parallelism) and a
HEAVY pass (serial, parallel=1). The KubeVirt-CDI/Longhorn/DR tests
(vm-lifecycle, vm-data-survival, vm-default-deny, data-survival, dr-restore)
spike embedded-etcd on the small dev VMs; running them serially in their own
pass keeps their etcd spikes from ever coinciding with another test's setup —
the coincidence that let one heavy test's apiserver blip fail-close Kyverno
admission and abort every in-flight test at 0.01s. Heavy set is keyed by test
dir basename and overridable via CONF_HEAVY_TESTS. Each pass writes its own
JUnit; lib/merge_junit.py stitches them into the one stable report.junit.xml
CI collects, so the split is invisible downstream. Overall rc is the union of
both passes (no masking — real failures still surface).

Proof: full --layer l2,l3 run now completes single-pass green: 24/24 passed,
0 failures/errors/skipped. vpn-killswitch, which previously failed with an
apiserver EOF (etcd blip), passed; no mass environmental setup-aborts.

TASK 2 (T-4.1) — author conformance/tests/l3-fault-injection/alerting/
fixtures/smtp-sink.yaml: a hermetic Mailpit SMTP capture sink (Service
smtp-sink, SMTP 1025, HTTP API 8025) the alerting test applies as its mail
sink. restricted-PodSecurity compliant, in-memory (no PVC). Unblocks the
previously-aborting alerting scaffold; it now passes.

Touches conformance/ + ci-read content only — no apps/ or platform/ workload.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
 2026-08-26 02:56:56 -05:00
jakman85andClaude Opus 4.8 bb48f675ee ci: bake pinned test toolchain into devcontainer + fix conformance layer/spec-immutability seams 
Bake the conformance harness toolchain into infra/devcontainer so `make -C ci gate`
runs identically here and in CI:
- Containerfile: install kyverno CLI (1.13.4), kyverno-json, chainsaw (0.2.12),
  yamllint (1.35.1), check-jsonschema (0.29.4), yq (4.44.6) via new pinned RUN block;
  add diffutils + python3-pip; drop stale LIBVIRT_DEFAULT_URI; refresh header for the
  Proxmox(bpg)+conformance toolbox; extend /etc/dev-toolchain-versions provenance.
- versions.env: add KYVERNO/KYVERNO_JSON/CHAINSAW/YAMLLINT/YQ/CHECK_JSONSCHEMA;
  align KUBECTL 1.32.0 -> 1.31.4. dev.sh build(): pass the new --build-args.
- ci/versions.env: pin CHECK_JSONSCHEMA_VERSION=0.29.4 (single source of truth).

Fix harness seams so the gate can run:
- dev.sh run(): mount the sibling spec repo (home-network-k3s) read-only at
  /home-network-k3s for ci/spec-immutability.sh (R-TEST-06); conditional so the
  container still runs without it.
- conformance/lib/common.sh conf_dirs_for_layer(): accept a comma/space list of
  layers and union the dirs, so ci/conformance.sh `run.sh --layer l2,l3` selects
  both layers in one pass (backward compatible for single layers).

NOTE: ci/versions.env pins KYVERNO_JSON_VERSION=0.0.5, which has no upstream release
(max tag is v0.0.3; platform/kyverno/scripts/run-l0.sh already defaults to v0.0.3).
Baked v0.0.3 and flagged for PM to correct ci/versions.env. kyverno-json is not used
by any static gate stage.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
 2026-08-25 15:43:19 -05:00
jakman85andClaude Opus 4.8 4a017c590e Wire L1 render conformance tests to the renderer (R-NET-01, R-NET-05/06/07) 
Implements the two L1 rendering tests that were failing scaffolds:
- egress-class-render (R-NET-05/06/07): renders vpn + internet fixtures, diffs each
  against a committed golden, asserts the vpn steering label is present on vpn and
  absent on internet (fail-closed), and that flipping the class changes ONLY the
  egress path (R-NET-07). This fixes the --requirements R-NET-05 merge gate.
- default-deny-baseline-render (R-NET-01): asserts every render carries the
  namespaced default-deny baseline, shaped deny-all + both directions.

Each test carries a pure-offline render-check.sh (renderer + golden diff, no
cluster). run.sh runs L1 render checks offline and emits a REAL pass/fail JUnit
(docs/14: L1 rendering assertions run in CI without a cluster) instead of skipping
L1 with the cluster-gated layers; the chainsaw-test.yaml wrappers invoke the same
render-check.sh so an in-cluster run executes the identical assertion.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
 2026-08-24 22:19:08 -05:00
jakman85andClaude Opus 4.8 9d8e03892f T-0.4 Chainsaw conformance harness + coverage + behavior scaffolds (R-TEST-01..06) 
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
 2026-08-16 00:08:32 -05:00