jakman85andClaude Opus 5
59ffbaf11a
Six L2 behavioural tests embedded `home.keithjackson.dev`. They are test INPUTS,
not reconciled manifests, so postBuild substitution never touched them — correct
while dev held the apex, wrong the moment prod came up on
prod.home.keithjackson.dev, and blocking §9's "suite passes green against prod".
Each test now asks the cluster what it is, via a test-level chainsaw binding over
flux-system/cluster-vars — the SAME ConfigMap Flux renders the platform from, so
a test cannot disagree with the cluster it runs on:
bindings:
- name: baseDomain
value: (x_k8s_get($client,'v1','ConfigMap','flux-system','cluster-vars').data.BASE_DOMAIN)
Chosen over a flag or env var because there is nothing to forget at run time, and
test-level scope covers every step (step outputs/bindings are step-scoped).
THE APEX WAS NOT THE ONLY LEAK. dns-exposure and dns-internal-resolution also
hardcoded DEV'S Cilium LB range 192.168.1.53-.60; prod's is 192.168.2.20-.30, so
both were wrong for prod regardless of hostname. Replaced with assertions on the
OBSERVED value — the DNS answer must equal that cluster's GATEWAY_VIP, and the
public record must equal the LB IP the Service was actually assigned — which is
strictly STRONGER than the old range check as well as environment-independent.
sso-forward-auth held the literal only in prose (it talks over Service DNS).
PROVEN, NOT ASSERTED — and the positive control is the important half:
* expressions extracted verbatim from the tracked files resolve to the correct,
DIFFERENT values per cluster (dev home.keithjackson.dev / 192.168.1.53,
prod prod.home.keithjackson.dev / 192.168.2.20);
* the same assert with the HARDCODED apex passes on dev and FAILS on prod
("Invalid value: *.prod.home... Expected: *.home..."), so the harness is
provably not blind — that is exactly the drift removed;
* chainsaw lint VALID on all six; run.sh --lint clean over the whole tree.
END-TO-END L2 RUN AGAINST LIVE DEV: `dns-exposure` PASSED (42.6s) — the binding
produced conformance-dns.home.keithjackson.dev, external-dns published it, and the
probe Job asserted the record against the assigned LB IP. This is the first time
one of these tests has been executed since parameterization, and it exercises the
full path rather than just the binding.
FINDING FROM THAT RUN, worth recording: the test's teardown is ASYNCHRONOUS. It
reports PASS as soon as chainsaw deletes the namespace, but the Cloudflare record
is reaped later by external-dns's own --interval=1m sync. A zone query immediately
after PASS still showed conformance-dns.home.keithjackson.dev plus its owner=default
TXT registry entry. Harmless here, but it means a green dns-exposure does NOT prove
cleanup happened, and if external-dns were suspended the record would persist
silently. Verified separately that it clears.
conformance/incluster/rbac.yaml: its cluster-wide `get configmaps` is now
LOAD-BEARING for these bindings — narrowing it to the test namespace breaks five
tests. Commented so it is not "tidied" later.
l3-fault-injection/dr-restore: its KNOWN-GAPS block said off-host is unproven
because the MinIO shares a Proxmox host with dev's VMs. Still true as run, so the
text now says both — that the R2 move (32d6766) is what closes it, that this test's
pass is against MinIO, and that the ledger row proving R-STATE-03 PREDATES the move,
so off-host DR cannot be claimed until the drill is re-run against R2.
NOT VERIFIED, plainly: no full L2 suite run against prod, so "green against prod"
remains unproven — what is proven is that the tests now derive the CORRECT
expectation for whichever cluster they are pointed at. Whether prod SATISFIES them
(split-horizon, LE wildcard, Grafana's issuer) is untested.
VERIFIED: full CI gate PASS. flux-postbuild-substitution 260 assertions,
render-golden 10/10.
Refs R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04, R-ENV-02, R-DNS-01/02/03.
Co-Authored-By: Claude Opus 5 <[email protected]>
177 lines
9.2 KiB
YAML
177 lines
9.2 KiB
YAML
# =============================================================================
|
|
# Scoped test ServiceAccount + RBAC for the in-cluster Chainsaw conformance Job
|
|
# (ADR-0011; docs/14 "Where each layer runs"; R-SEC-02 least-privilege).
|
|
#
|
|
# SKELETON / PLACEHOLDER — RBAC is deliberately LEFT EMPTY here and authored
|
|
# least-privilege as each fault-injection capability actually lands (T-3.0+). The
|
|
# rule is: grant nothing until a committed test needs it, then grant exactly that
|
|
# verb/resource, with a comment naming the test that requires it. This SA is a
|
|
# PRIVILEGED TEST IDENTITY — it is its own audit item (R-SEC-02): it MUST NOT be
|
|
# reused by workloads, and every widening is reviewed.
|
|
#
|
|
# CI holds NO cluster credentials. CI only triggers the Job (T-6.1b via Gitea
|
|
# Actions); the Job runs under THIS ServiceAccount, so cluster access stays
|
|
# in-cluster and scoped (docs/14, R-SEC-02).
|
|
#
|
|
# Deployment: reconciled by Flux into the `conformance` namespace once the cluster
|
|
# exists (Phase 1+). Not applied in Phase 0 (no cluster).
|
|
# =============================================================================
|
|
apiVersion: v1
|
|
kind: Namespace
|
|
metadata:
|
|
name: conformance
|
|
labels:
|
|
app.kubernetes.io/part-of: conformance
|
|
# The test namespace itself is exempt from nothing: the platform default-deny
|
|
# baseline still applies; the Job opens only the egress it declares.
|
|
pod-security.kubernetes.io/enforce: restricted
|
|
---
|
|
apiVersion: v1
|
|
kind: ServiceAccount
|
|
metadata:
|
|
name: conformance-runner
|
|
namespace: conformance
|
|
labels:
|
|
app.kubernetes.io/part-of: conformance
|
|
annotations:
|
|
conformance.home.k3s/audit: >-
|
|
PRIVILEGED TEST IDENTITY. Scoped least-privilege for fault-injection only.
|
|
Never reuse for workloads. Widen only alongside the test that needs it.
|
|
---
|
|
# ClusterRole — the union of what BEHAVIOR + FAULT tests need. Every rule is
|
|
# commented with the test/requirement that justifies it. Start empty; uncomment /
|
|
# add rules as the corresponding L2/L3 tests are delivered (do NOT pre-grant).
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: ClusterRole
|
|
metadata:
|
|
name: conformance-runner
|
|
labels:
|
|
app.kubernetes.io/part-of: conformance
|
|
rules:
|
|
# ---- Hermetic fixtures (all L2/L3): create/delete throwaway test namespaces + fixtures
|
|
- apiGroups: [""]
|
|
resources: [namespaces]
|
|
# `patch` is required by R-NET-06 (vpn-killswitch): the test relabels its own
|
|
# ephemeral namespace to PodSecurity enforce=privileged so it can launch the
|
|
# node-pinned privileged fault Job. Scoped to the per-test namespace only.
|
|
verbs: [create, delete, get, list, watch, patch]
|
|
- apiGroups: [""]
|
|
resources: [pods, services, configmaps, secrets, persistentvolumeclaims]
|
|
# LOAD-BEARING `get` ON CONFIGMAPS, BEYOND FIXTURE CREATION (R-ENV-02): the environment-
|
|
# agnostic L2 tests (dns-exposure, dns-internal-resolution, gateway-ingress-reachable,
|
|
# oidc-sso-grafana, trusted-tls) resolve the cluster's own BASE_DOMAIN / GATEWAY_VIP at run
|
|
# time from `flux-system/cluster-vars` — the same ConfigMap Flux substitutes from — via
|
|
# chainsaw `x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars')` bindings.
|
|
# If this rule is ever narrowed to the test's own namespace, those bindings fail and the
|
|
# tests error out. Narrow it only alongside a namespace-scoped read of that ConfigMap.
|
|
verbs: [create, delete, get, list, watch, patch]
|
|
- apiGroups: [""]
|
|
resources: [pods/exec]
|
|
verbs: [create] # probe exec for behavior assertions
|
|
- apiGroups: [""]
|
|
resources: [pods/log]
|
|
verbs: [get] # read fault/probe Job logs on failure
|
|
# ---- Storage-tier precondition gate (READ-ONLY). Every storage test self-skips on the
|
|
# absence of the `longhorn` StorageClass rather than failing: data-survival (R-STATE-01),
|
|
# node-loss (R-STATE-02), dr-restore (R-STATE-03), vm-data-survival / vm-lifecycle /
|
|
# vm-default-deny (R-PLAT-03). Without this read the gate cannot tell "absent" from
|
|
# "forbidden" and would skip for the wrong reason.
|
|
- apiGroups: [storage.k8s.io]
|
|
resources: [storageclasses]
|
|
verbs: [get, list]
|
|
# ---- Controller-readiness preconditions (READ-ONLY). dr-restore (R-STATE-03) skips cleanly
|
|
# unless the Velero Deployment and its node-agent DaemonSet (kopia File System Backup, the
|
|
# volume-DATA path) are actually ready — a backup path that is not up is an absent
|
|
# precondition, not a failure. No write verbs: tests never scale or patch a platform
|
|
# controller.
|
|
- apiGroups: [apps]
|
|
resources: [deployments, daemonsets]
|
|
verbs: [get, list, watch]
|
|
# ---- Probe / fault Jobs (R-TLS-01/03 tls-probe; R-NET-06 tunnel stop/start)
|
|
# `create` cannot be constrained by resourceNames, and the fault Job names are
|
|
# fixed per-test (tunnel-stop/tunnel-start/tunnel-start-finally); scoped by the
|
|
# in-namespace blast radius of the ephemeral test namespace.
|
|
- apiGroups: [batch]
|
|
resources: [jobs]
|
|
verbs: [create, delete, get, list, watch, patch]
|
|
#
|
|
# ---- R-NET-01/02 (default-deny, deny-undeclared): apply/patch/delete allow policies
|
|
# - apiGroups: [cilium.io]
|
|
# resources: [ciliumnetworkpolicies]
|
|
# verbs: [create, delete, get, list, patch]
|
|
#
|
|
# ---- R-NET-06 (VPN kill-switch, ADR-0015): the fault is a HOST systemd action
|
|
# (stop/start `vpn-egress-tunnel` on the egress-gateway node), NOT scaling an
|
|
# in-cluster gluetun Deployment (superseded design). It is driven by a privileged,
|
|
# hostPID, node-pinned Job (nsenter -t 1 -- systemctl ...). That power comes from
|
|
# PodSecurity (the test namespace relabeled to enforce=privileged) + the pod's
|
|
# securityContext, NOT from RBAC — the fault Job pod needs NO Kubernetes API
|
|
# rights. The runner only needs `jobs` create/delete + `namespaces` patch above.
|
|
# PREREQUISITE: if a cluster-wide Kyverno disallow-privileged / disallow-host-
|
|
# namespaces policy is enforced, the `chainsaw-conf-*` test namespaces MUST be in
|
|
# its exclusion set (documented, audited exception — R-SEC-02).
|
|
#
|
|
# ---- R-STATE-02 (node loss, tests/l3-fault-injection/node-loss): the fault is
|
|
# `kubectl cordon` + label of ONE node, reverted by `kubectl uncordon` + label removal
|
|
# in the test's `finally`. All three (cordon/uncordon/label) are a `patch` on the node
|
|
# object; `get`/`list` locate the victim + the finally cleanup selector. The test evicts
|
|
# only its OWN writer pod via `delete pod` (already granted above) — it deliberately does
|
|
# NOT `kubectl drain`, so no eviction rights and no reach over unrelated pods are needed.
|
|
- apiGroups: [""]
|
|
resources: [nodes]
|
|
verbs: [get, list, patch] # cordon/uncordon/label = patch spec.unschedulable + metadata.labels
|
|
#
|
|
# ---- R-UPD-02 (bad-update auto-rollback): inject a bad image tag via automation
|
|
# - apiGroups: [image.toolkit.fluxcd.io]
|
|
# resources: [imagepolicies, imagerepositories]
|
|
# verbs: [get, patch]
|
|
#
|
|
# ---- R-STATE-03 (DR restore, tests/l3-fault-injection/dr-restore): the drill creates a
|
|
# Velero Backup of its OWN scratch namespace, destroys that namespace + its PV, creates a
|
|
# Restore into a fresh namespace, and (in `finally`) issues a DeleteBackupRequest so the
|
|
# objects it wrote are removed from the off-cluster bucket again. `create` cannot be
|
|
# constrained by resourceNames; the blast radius is bounded by the fact that every object
|
|
# the test creates or deletes is named `<its-own-chainsaw-namespace>-dr-*`.
|
|
- apiGroups: [velero.io]
|
|
resources: [backups, restores, deletebackuprequests]
|
|
verbs: [create, delete, get, list, watch]
|
|
# Read-only evidence: the BSL is the off-cluster target whose Available phase + endpoint the
|
|
# test asserts; schedules prove backups are SCHEDULED (not one-off); podvolumebackups /
|
|
# podvolumerestores prove the kopia snapshot of the volume DATA reached/returned from the
|
|
# remote repository, and are dumped on failure for diagnosis.
|
|
- apiGroups: [velero.io]
|
|
resources: [backupstoragelocations, schedules, podvolumebackups, podvolumerestores]
|
|
verbs: [get, list, watch]
|
|
# R-STATE-03 needs `delete` on the PV, not merely read: the proof that the restore came from
|
|
# OFF-cluster is that no in-cluster copy of the data survived. The test asserts its own PV is
|
|
# gone, and force-deletes it if a Retain policy or a stuck finalizer left it behind. It only
|
|
# ever names the PV bound to the PVC it created itself.
|
|
- apiGroups: [""]
|
|
resources: [persistentvolumes]
|
|
verbs: [get, list, delete]
|
|
#
|
|
# ---- R-GITOPS-03 (drift): mutate a reconciled resource + observe Flux revert
|
|
# - apiGroups: [kustomize.toolkit.fluxcd.io]
|
|
# resources: [kustomizations]
|
|
# verbs: [get, list, watch] # observe only; do NOT grant reconcile suspension
|
|
#
|
|
# ---- R-PLAT-03 (VM lifecycle): create/start/stop/destroy VMs
|
|
# - apiGroups: [kubevirt.io]
|
|
# resources: [virtualmachines, virtualmachineinstances]
|
|
# verbs: [create, delete, get, list, patch]
|
|
---
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: ClusterRoleBinding
|
|
metadata:
|
|
name: conformance-runner
|
|
labels:
|
|
app.kubernetes.io/part-of: conformance
|
|
roleRef:
|
|
apiGroup: rbac.authorization.k8s.io
|
|
kind: ClusterRole
|
|
name: conformance-runner
|
|
subjects:
|
|
- kind: ServiceAccount
|
|
name: conformance-runner
|
|
namespace: conformance
|