Files
home-network-k3s-impl/conformance/tests/l2-behavioral/default-deny/chainsaw-test.yaml
T
jakman85andClaude Opus 5 6a8953e288 feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.

Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.

WHAT THIS FIXES

1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
   ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
   anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
   to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
   volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
   database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
   the hour, and evaluates to 3 firing series on prod today.
   Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
   nextcloud-db's exporter cannot log in to its own database (permission denied for
   database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
   for it — the obvious metric would have been blind to one database in three.

2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
   by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
   job="kubelet" (the apiserver copy of the same series reports namespace=default),
   and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
   available=0 forever and would otherwise have fired critical permanently.

3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
   firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
   a node carries the egress-gateway label, with dev's behaviour unchanged today.

4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
   LITERALLY and the import still reports success, then sits at Done=True and never
   re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
   $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
   green. The CronJob is the level-triggered correction; two alerts make its silence
   observable. Repaired live already — this stops it recurring.

Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 00:59:22 -05:00

173 lines
7.7 KiB
YAML

# =============================================================================
# L2 BEHAVIORAL — Default-deny east-west (R-NET-01) [RUNNABLE]
# Catalog: docs/14-testing-and-conformance.md "Default-deny (R-NET-01)"
# Spec: docs/03-networking-isolation-and-egress.md "Default-deny baseline"
#
# EXEMPLAR: this file is the reference for the tagging convention and step
# style. Other scaffolds follow the same shape (see tests/README.md).
#
# GIVEN a fresh workload in a namespace with the default-deny baseline applied,
# WHEN it attempts any in-cluster connection except DNS,
# THEN the connection FAILS (fail-closed); DNS still succeeds.
#
# This asserts observable BEHAVIOR (a probe Job cannot connect) via ephemeral
# probe pods, not the mere existence of a policy object (R-TEST-01).
#
# RUNNABLE ON THE BASE CLUSTER: the platform-wide default-deny baseline (T-2.2)
# is not yet applied cluster-wide, so this test applies an equivalent baseline
# as its OWN hermetic fixture in the per-test namespace. The fixture is a
# standard Kubernetes NetworkPolicy (Cilium enforces it) — portable and
# version-agnostic; the platform's baseline will be the CiliumNetworkPolicy
# equivalent, but the asserted BEHAVIOR (DNS-only egress) is identical, which
# is what R-TEST-01 requires us to prove. Chainsaw auto-creates/cleans the
# namespace (hermetic); the probe Jobs are ephemeral and self-cleaned with it.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: default-deny
labels:
# --- requirement tags (authoritative; one label per verified R-* ID) ---
conformance.home.k3s/R-NET-01: "true"
annotations:
conformance.home.k3s/requirements: "R-NET-01"
conformance.home.k3s/layer: "L2"
conformance.home.k3s/safety-critical: "false"
# executable => this test runs real behavior against a cluster (not a scaffold).
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Given a bare workload in a namespace with the default-deny baseline, only
DNS egress succeeds; every other in-cluster connection is denied.
spec:
steps:
# -- GIVEN: apply the default-deny baseline to the per-test namespace. Deny all
# ingress + egress for every pod; allow ONLY DNS egress to the in-cluster
# resolver. This is the hermetic stand-in for the platform baseline (T-2.2).
- name: given-default-deny-baseline
try:
- apply:
resource:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-baseline
spec:
podSelector: {}
policyTypes: [Ingress, Egress]
# no ingress rules => deny all ingress
ingress: []
# only DNS egress to kube-dns (CoreDNS) in kube-system
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# -- THEN(DNS allowed): DNS egress MUST succeed even under default-deny. An
# ephemeral Job resolves an in-cluster name; we assert the Job SUCCEEDS.
# (Chainsaw polls the Job status within the assert timeout — bounded, no sleeps.)
- name: then-dns-succeeds
try:
- apply:
resource:
apiVersion: batch/v1
kind: Job
metadata:
name: dns-probe
spec:
backoffLimit: 4
activeDeadlineSeconds: 120
template:
metadata:
labels: { app: conformance-probe }
spec:
restartPolicy: Never
containers:
- name: probe
image: busybox:1.36.1
command:
- sh
- -c
- |
set -eu
# DNS is the one egress the baseline permits.
nslookup kubernetes.default.svc.cluster.local
echo DNS_OK
- assert:
resource:
apiVersion: batch/v1
kind: Job
metadata: { name: dns-probe }
status: { succeeded: 1 }
# -- THEN(everything else denied): a non-DNS in-cluster connection MUST fail
# closed. The Job attempts to reach the kube-apiserver ClusterIP (always
# present) on 443; the probe EXITS 0 only when that connection is BLOCKED,
# so a green Job proves the deny. A leak (connection succeeds) fails the Job.
- name: then-nondns-denied
try:
- apply:
resource:
apiVersion: batch/v1
kind: Job
metadata:
name: block-probe
spec:
backoffLimit: 4
activeDeadlineSeconds: 120
template:
metadata:
labels: { app: conformance-probe }
spec:
restartPolicy: Never
containers:
- name: probe
image: busybox:1.36.1
command:
- sh
- -c
- |
set -u
# DNS resolves (allowed); the TCP connect must be dropped.
if nc -w 4 kubernetes.default.svc.cluster.local 443 </dev/null 2>/dev/null; then
echo "LEAK: reached kube-apiserver under default-deny" 1>&2
exit 1
fi
echo DENIED_OK
- assert:
resource:
apiVersion: batch/v1
kind: Job
metadata: { name: block-probe }
status: { succeeded: 1 }
# -- chainsaw deletes the namespace (policy + probe Jobs) automatically; this
# block is the explicit hook for any out-of-namespace fixtures (none here).
finally:
- script:
content: 'echo "cleanup handled by chainsaw namespace teardown"'
# -----------------------------------------------------------------------------------
# PROOF-OF-PASS SENTINEL (R-SEC-03). Every step above is an assert/Job-status gate, so
# chainsaw already fails the test before reaching this step if any of them fails — but a
# green Chainsaw JUnit carries NO stdout, and the L2/L3 tests in this suite can self-skip
# GREEN, so lib/record_run.py refuses to treat a bare `passed` as evidence and records
# `unknown-no-sentinel-declared` instead. Without a declared token this test could pass
# forever and NEVER become a recorded proof of R-NET-01 — the isolation half of docs/07's audit proof list
# — which is precisely what docs/07 asks for ("These proofs are part of the platform
# acceptance suite, not one-time checks"). This step exists only to emit that token, and it
# is reached only when everything above has passed. Asserted by
# tests/l0-static/auditability-proofs-producible.
- name: then-emit-the-proof-of-pass-sentinel
try:
- script:
content: echo DEFAULT_DENY_PROVEN_OK
check:
(contains($stdout, 'DEFAULT_DENY_PROVEN_OK')): true