jakman85andClaude Opus 5
6a8953e288
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
177 lines
9.2 KiB
YAML
177 lines
9.2 KiB
YAML
# =============================================================================
|
|
# Scoped test ServiceAccount + RBAC for the in-cluster Chainsaw conformance Job
|
|
# (ADR-0011; docs/14 "Where each layer runs"; R-SEC-02 least-privilege).
|
|
#
|
|
# SKELETON / PLACEHOLDER — RBAC is deliberately LEFT EMPTY here and authored
|
|
# least-privilege as each fault-injection capability actually lands (T-3.0+). The
|
|
# rule is: grant nothing until a committed test needs it, then grant exactly that
|
|
# verb/resource, with a comment naming the test that requires it. This SA is a
|
|
# PRIVILEGED TEST IDENTITY — it is its own audit item (R-SEC-02): it MUST NOT be
|
|
# reused by workloads, and every widening is reviewed.
|
|
#
|
|
# CI holds NO cluster credentials. CI only triggers the Job (T-6.1b via Gitea
|
|
# Actions); the Job runs under THIS ServiceAccount, so cluster access stays
|
|
# in-cluster and scoped (docs/14, R-SEC-02).
|
|
#
|
|
# Deployment: reconciled by Flux into the `conformance` namespace once the cluster
|
|
# exists (Phase 1+). Not applied in Phase 0 (no cluster).
|
|
# =============================================================================
|
|
apiVersion: v1
|
|
kind: Namespace
|
|
metadata:
|
|
name: conformance
|
|
labels:
|
|
app.kubernetes.io/part-of: conformance
|
|
# The test namespace itself is exempt from nothing: the platform default-deny
|
|
# baseline still applies; the Job opens only the egress it declares.
|
|
pod-security.kubernetes.io/enforce: restricted
|
|
---
|
|
apiVersion: v1
|
|
kind: ServiceAccount
|
|
metadata:
|
|
name: conformance-runner
|
|
namespace: conformance
|
|
labels:
|
|
app.kubernetes.io/part-of: conformance
|
|
annotations:
|
|
conformance.home.k3s/audit: >-
|
|
PRIVILEGED TEST IDENTITY. Scoped least-privilege for fault-injection only.
|
|
Never reuse for workloads. Widen only alongside the test that needs it.
|
|
---
|
|
# ClusterRole — the union of what BEHAVIOR + FAULT tests need. Every rule is
|
|
# commented with the test/requirement that justifies it. Start empty; uncomment /
|
|
# add rules as the corresponding L2/L3 tests are delivered (do NOT pre-grant).
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: ClusterRole
|
|
metadata:
|
|
name: conformance-runner
|
|
labels:
|
|
app.kubernetes.io/part-of: conformance
|
|
rules:
|
|
# ---- Hermetic fixtures (all L2/L3): create/delete throwaway test namespaces + fixtures
|
|
- apiGroups: [""]
|
|
resources: [namespaces]
|
|
# `patch` is required by R-NET-06 (vpn-killswitch): the test relabels its own
|
|
# ephemeral namespace to PodSecurity enforce=privileged so it can launch the
|
|
# node-pinned privileged fault Job. Scoped to the per-test namespace only.
|
|
verbs: [create, delete, get, list, watch, patch]
|
|
- apiGroups: [""]
|
|
resources: [pods, services, configmaps, secrets, persistentvolumeclaims]
|
|
# LOAD-BEARING `get` ON CONFIGMAPS, BEYOND FIXTURE CREATION (R-ENV-02): the environment-
|
|
# agnostic L2 tests (dns-exposure, dns-internal-resolution, gateway-ingress-reachable,
|
|
# oidc-sso-grafana, trusted-tls) resolve the cluster's own BASE_DOMAIN / GATEWAY_VIP at run
|
|
# time from `flux-system/cluster-vars` — the same ConfigMap Flux substitutes from — via
|
|
# chainsaw `x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars')` bindings.
|
|
# If this rule is ever narrowed to the test's own namespace, those bindings fail and the
|
|
# tests error out. Narrow it only alongside a namespace-scoped read of that ConfigMap.
|
|
verbs: [create, delete, get, list, watch, patch]
|
|
- apiGroups: [""]
|
|
resources: [pods/exec]
|
|
verbs: [create] # probe exec for behavior assertions
|
|
- apiGroups: [""]
|
|
resources: [pods/log]
|
|
verbs: [get] # read fault/probe Job logs on failure
|
|
# ---- Storage-tier precondition gate (READ-ONLY). Every storage test self-skips on the
|
|
# absence of the `longhorn` StorageClass rather than failing: data-survival (R-STATE-01),
|
|
# node-loss (R-STATE-02), dr-restore (R-STATE-03), vm-data-survival / vm-lifecycle /
|
|
# vm-default-deny (R-PLAT-03). Without this read the gate cannot tell "absent" from
|
|
# "forbidden" and would skip for the wrong reason.
|
|
- apiGroups: [storage.k8s.io]
|
|
resources: [storageclasses]
|
|
verbs: [get, list]
|
|
# ---- Controller-readiness preconditions (READ-ONLY). dr-restore (R-STATE-03) skips cleanly
|
|
# unless the Velero Deployment and its node-agent DaemonSet (kopia File System Backup, the
|
|
# volume-DATA path) are actually ready — a backup path that is not up is an absent
|
|
# precondition, not a failure. No write verbs: tests never scale or patch a platform
|
|
# controller.
|
|
- apiGroups: [apps]
|
|
resources: [deployments, daemonsets]
|
|
verbs: [get, list, watch]
|
|
# ---- Probe / fault Jobs (R-TLS-01/03 tls-probe; R-NET-06 tunnel stop/start)
|
|
# `create` cannot be constrained by resourceNames, and the fault Job names are
|
|
# fixed per-test (tunnel-stop/tunnel-start/tunnel-start-finally); scoped by the
|
|
# in-namespace blast radius of the ephemeral test namespace.
|
|
- apiGroups: [batch]
|
|
resources: [jobs]
|
|
verbs: [create, delete, get, list, watch, patch]
|
|
#
|
|
# ---- R-NET-01/02 (default-deny, deny-undeclared): apply/patch/delete allow policies
|
|
# - apiGroups: [cilium.io]
|
|
# resources: [ciliumnetworkpolicies]
|
|
# verbs: [create, delete, get, list, patch]
|
|
#
|
|
# ---- R-NET-06 (VPN kill-switch, ADR-0015): the fault is a HOST systemd action
|
|
# (stop/start `vpn-egress-tunnel` on the egress-gateway node), NOT scaling an
|
|
# in-cluster gluetun Deployment (superseded design). It is driven by a privileged,
|
|
# hostPID, node-pinned Job (nsenter -t 1 -- systemctl ...). That power comes from
|
|
# PodSecurity (the test namespace relabeled to enforce=privileged) + the pod's
|
|
# securityContext, NOT from RBAC — the fault Job pod needs NO Kubernetes API
|
|
# rights. The runner only needs `jobs` create/delete + `namespaces` patch above.
|
|
# PREREQUISITE: if a cluster-wide Kyverno disallow-privileged / disallow-host-
|
|
# namespaces policy is enforced, the `chainsaw-conf-*` test namespaces MUST be in
|
|
# its exclusion set (documented, audited exception — R-SEC-02).
|
|
#
|
|
# ---- R-STATE-02 (node loss, tests/l3-fault-injection/node-loss): the fault is
|
|
# `kubectl cordon` + label of ONE node, reverted by `kubectl uncordon` + label removal
|
|
# in the test's `finally`. All three (cordon/uncordon/label) are a `patch` on the node
|
|
# object; `get`/`list` locate the victim + the finally cleanup selector. The test evicts
|
|
# only its OWN writer pod via `delete pod` (already granted above) — it deliberately does
|
|
# NOT `kubectl drain`, so no eviction rights and no reach over unrelated pods are needed.
|
|
- apiGroups: [""]
|
|
resources: [nodes]
|
|
verbs: [get, list, patch] # cordon/uncordon/label = patch spec.unschedulable + metadata.labels
|
|
#
|
|
# ---- R-UPD-02 (bad-update auto-rollback): inject a bad image tag via automation
|
|
# - apiGroups: [image.toolkit.fluxcd.io]
|
|
# resources: [imagepolicies, imagerepositories]
|
|
# verbs: [get, patch]
|
|
#
|
|
# ---- R-STATE-03 (DR restore, tests/l3-fault-injection/dr-restore): the drill creates a
|
|
# Velero Backup of its OWN scratch namespace, destroys that namespace + its PV, creates a
|
|
# Restore into a fresh namespace, and (in `finally`) issues a DeleteBackupRequest so the
|
|
# objects it wrote are removed from the off-cluster bucket again. `create` cannot be
|
|
# constrained by resourceNames; the blast radius is bounded by the fact that every object
|
|
# the test creates or deletes is named `<its-own-chainsaw-namespace>-dr-*`.
|
|
- apiGroups: [velero.io]
|
|
resources: [backups, restores, deletebackuprequests]
|
|
verbs: [create, delete, get, list, watch]
|
|
# Read-only evidence: the BSL is the off-cluster target whose Available phase + endpoint the
|
|
# test asserts; schedules prove backups are SCHEDULED (not one-off); podvolumebackups /
|
|
# podvolumerestores prove the kopia snapshot of the volume DATA reached/returned from the
|
|
# remote repository, and are dumped on failure for diagnosis.
|
|
- apiGroups: [velero.io]
|
|
resources: [backupstoragelocations, schedules, podvolumebackups, podvolumerestores]
|
|
verbs: [get, list, watch]
|
|
# R-STATE-03 needs `delete` on the PV, not merely read: the proof that the restore came from
|
|
# OFF-cluster is that no in-cluster copy of the data survived. The test asserts its own PV is
|
|
# gone, and force-deletes it if a Retain policy or a stuck finalizer left it behind. It only
|
|
# ever names the PV bound to the PVC it created itself.
|
|
- apiGroups: [""]
|
|
resources: [persistentvolumes]
|
|
verbs: [get, list, delete]
|
|
#
|
|
# ---- R-GITOPS-03 (drift): mutate a reconciled resource + observe Flux revert
|
|
# - apiGroups: [kustomize.toolkit.fluxcd.io]
|
|
# resources: [kustomizations]
|
|
# verbs: [get, list, watch] # observe only; do NOT grant reconcile suspension
|
|
#
|
|
# ---- R-PLAT-03 (VM lifecycle): create/start/stop/destroy VMs
|
|
# - apiGroups: [kubevirt.io]
|
|
# resources: [virtualmachines, virtualmachineinstances]
|
|
# verbs: [create, delete, get, list, patch]
|
|
---
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: ClusterRoleBinding
|
|
metadata:
|
|
name: conformance-runner
|
|
labels:
|
|
app.kubernetes.io/part-of: conformance
|
|
roleRef:
|
|
apiGroup: rbac.authorization.k8s.io
|
|
kind: ClusterRole
|
|
name: conformance-runner
|
|
subjects:
|
|
- kind: ServiceAccount
|
|
name: conformance-runner
|
|
namespace: conformance
|