jakman85andClaude Opus 5
6a8953e288
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
113 lines
6.1 KiB
YAML
113 lines
6.1 KiB
YAML
# =============================================================================
|
|
# L2 BEHAVIORAL — Trusted TLS (R-TLS-01, R-TLS-03) [RUNNABLE]
|
|
# Catalog: docs/14 "Trusted TLS (R-TLS-01/02)"
|
|
# Spec: docs/09 (Gateway API + DNS-01 wildcard), ADR-0006 (LE issuers)
|
|
#
|
|
# ENVIRONMENT-AGNOSTIC (R-ENV-02): the hostnames are built from the ${BASE_DOMAIN} of the CLUSTER
|
|
# UNDER TEST, read from flux-system/cluster-vars (the ConfigMap Flux substitutes from) by a
|
|
# test-level chainsaw binding. The same authored test therefore verifies dev's wildcard on dev and
|
|
# prod's on prod, with nothing to pass at run time.
|
|
#
|
|
# GIVEN the shared Gateway terminating TLS with the cert-manager wildcard cert,
|
|
# WHEN an in-cluster probe dials it by hostname (SNI *.${BASE_DOMAIN}) and
|
|
# verifies the chain against the SYSTEM trust store,
|
|
# THEN the served cert is non-self-signed, unexpired, covers the wildcard SAN, and
|
|
# chains to a PUBLICLY-TRUSTED CA root (R-TLS-01). The wildcard SAN also covers
|
|
# internal-only hostnames, so internal services get the same trusted cert (R-TLS-03).
|
|
#
|
|
# TRUST MODEL: the cluster issues from Let's Encrypt **PRODUCTION** (ACME_CLUSTER_ISSUER
|
|
# = letsencrypt-prod), whose root (ISRG Root X1) is in the system trust store — so the
|
|
# probe verifies against the container's DEFAULT CA bundle (no --cacert). A clean verify
|
|
# is exactly what real in-cluster OIDC clients (oauth2-proxy, Grafana) depend on. The
|
|
# probe resolves the hostname via in-cluster split-horizon DNS (coredns-custom → Gateway
|
|
# VIP, R-DNS-01) — the same path real clients use — so no --connect-to hack is needed.
|
|
# Replaced the earlier LE-staging + embedded-root check when dev switched to LE-prod so
|
|
# in-cluster SSO clients trust the cert (DECISIONS-LOG Session 7).
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: trusted-tls
|
|
labels:
|
|
conformance.home.k3s/R-TLS-01: "true"
|
|
conformance.home.k3s/R-TLS-03: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-TLS-01, R-TLS-03"
|
|
conformance.home.k3s/layer: "L2"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
The shared Gateway serves a valid, unexpired, non-self-signed wildcard cert
|
|
that chains to a publicly-trusted CA root (LE production); the wildcard SAN
|
|
covers internal-only hostnames too.
|
|
spec:
|
|
# -- R-ENV-02: the hostname suffix is READ FROM THE CLUSTER UNDER TEST (flux-system/cluster-vars
|
|
# — the ConfigMap Flux substitutes ${BASE_DOMAIN} from), never authored per environment.
|
|
bindings:
|
|
- name: baseDomain
|
|
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
|
|
steps:
|
|
- name: then-gateway-cert-chains-to-trusted-root
|
|
try:
|
|
- apply:
|
|
resource:
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: tls-probe
|
|
spec:
|
|
backoffLimit: 4
|
|
activeDeadlineSeconds: 180
|
|
template:
|
|
spec:
|
|
restartPolicy: Never
|
|
containers:
|
|
- name: probe
|
|
image: curlimages/curl:8.11.1
|
|
env:
|
|
- name: BASE_DOMAIN
|
|
value: ($baseDomain)
|
|
command:
|
|
- sh
|
|
- -c
|
|
- |
|
|
set -eu
|
|
# Probe TWO wildcard hostnames — one "public", one "internal-only". Both
|
|
# match *.${BASE_DOMAIN}, so a clean verify against the SYSTEM CA
|
|
# bundle for BOTH proves R-TLS-01 (chains to a publicly-trusted root) AND
|
|
# R-TLS-03 (the same trusted wildcard covers internal-only names). Names
|
|
# resolve via in-cluster split-horizon DNS to the Gateway VIP. curl (no
|
|
# --cacert, no -k) validates against the default CA bundle and exits 0 on
|
|
# the gateway's 404 once TLS+hostname verify; it exits 60 if the chain or
|
|
# hostname fails. TLS terminates at the gateway before HTTP routing, so no
|
|
# backend is needed.
|
|
for host in "conformance.${BASE_DOMAIN}" "internal-only.${BASE_DOMAIN}"; do
|
|
curl -sS --retry 5 --retry-connrefused --retry-delay 3 --max-time 20 \
|
|
-o /dev/null "https://${host}/"
|
|
echo "TLS_TRUSTED ${host}"
|
|
done
|
|
echo ALL_TRUSTED
|
|
- assert:
|
|
resource:
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata: { name: tls-probe }
|
|
status: { succeeded: 1 }
|
|
# -----------------------------------------------------------------------------------
|
|
# PROOF-OF-PASS SENTINEL (R-SEC-03). Every step above is an assert/Job-status gate, so
|
|
# chainsaw already fails the test before reaching this step if any of them fails — but a
|
|
# green Chainsaw JUnit carries NO stdout, and the L2/L3 tests in this suite can self-skip
|
|
# GREEN, so lib/record_run.py refuses to treat a bare `passed` as evidence and records
|
|
# `unknown-no-sentinel-declared` instead. Without a declared token this test could pass
|
|
# forever and NEVER become a recorded proof of R-TLS-01/03 — the TLS half of docs/07's audit proof list
|
|
# — which is precisely what docs/07 asks for ("These proofs are part of the platform
|
|
# acceptance suite, not one-time checks"). This step exists only to emit that token, and it
|
|
# is reached only when everything above has passed. Asserted by
|
|
# tests/l0-static/auditability-proofs-producible.
|
|
- name: then-emit-the-proof-of-pass-sentinel
|
|
try:
|
|
- script:
|
|
content: echo TLS_TRUSTED_PROVEN_OK
|
|
check:
|
|
(contains($stdout, 'TLS_TRUSTED_PROVEN_OK')): true
|