jakman85andClaude Opus 5
6a8953e288
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
104 lines
4.7 KiB
YAML
104 lines
4.7 KiB
YAML
# =============================================================================
|
|
# L2 BEHAVIORAL — native-OIDC SSO via the central IdP (R-IDP-01) [RUNNABLE]
|
|
# Spec: docs/15 identity-and-access; ADR-0014. Impl: Grafana as a Keycloak OIDC client
|
|
# (platform/monitoring/kube-prometheus-stack.yaml auth.generic_oauth) — the self-contained SSO proof.
|
|
#
|
|
# ENVIRONMENT-AGNOSTIC (R-ENV-02): the expected IdP host is built from the ${BASE_DOMAIN} of the
|
|
# CLUSTER UNDER TEST, read from flux-system/cluster-vars (the ConfigMap Flux substitutes from) by
|
|
# a test-level chainsaw binding — so this asserts each cluster redirects to ITS OWN Keycloak, which
|
|
# is exactly what would catch a cross-environment OIDC issuer left pointing at the other cluster.
|
|
#
|
|
# GIVEN Grafana is deployed and configured as an OIDC client of the Keycloak `homelab` realm,
|
|
# WHEN an unauthenticated browser starts the OIDC login (GET /login/generic_oauth)
|
|
# -> THEN Grafana redirects (302) to the CENTRAL IdP's authorize endpoint on
|
|
# keycloak.${BASE_DOMAIN}/realms/homelab — proving SSO is driven by the one IdP
|
|
# (R-IDP-01: central Identity Provider as the single user directory).
|
|
#
|
|
# Goes GREEN once the grafana-oidc client-secret hand-off is filled and Keycloak + Grafana are up.
|
|
# chainsaw v0.2.12: Job + status assert; no test-level finally.
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: oidc-sso-grafana
|
|
labels:
|
|
conformance.home.k3s/R-IDP-01: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-IDP-01"
|
|
conformance.home.k3s/layer: "L2"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Grafana's OIDC login redirects to the central Keycloak realm — SSO is driven by the one IdP.
|
|
spec:
|
|
timeouts:
|
|
apply: 60s
|
|
assert: 3m
|
|
exec: 3m
|
|
# -- R-ENV-02: the IdP hostname suffix is READ FROM THE CLUSTER UNDER TEST
|
|
# (flux-system/cluster-vars — the ConfigMap Flux substitutes ${BASE_DOMAIN} from).
|
|
bindings:
|
|
- name: baseDomain
|
|
value: (x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars').data.BASE_DOMAIN)
|
|
steps:
|
|
- name: given-grafana-is-an-oidc-client
|
|
try:
|
|
- assert:
|
|
resource:
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: kube-prometheus-stack-grafana
|
|
namespace: monitoring
|
|
status:
|
|
availableReplicas: 1
|
|
- name: then-login-redirects-to-keycloak-realm
|
|
try:
|
|
- apply:
|
|
resource:
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: grafana-oidc-probe
|
|
spec:
|
|
backoffLimit: 2
|
|
activeDeadlineSeconds: 150
|
|
template:
|
|
spec:
|
|
restartPolicy: Never
|
|
containers:
|
|
- name: probe
|
|
image: busybox:1.36.1
|
|
env:
|
|
- name: BASE_DOMAIN
|
|
value: ($baseDomain)
|
|
command:
|
|
- sh
|
|
- -c
|
|
- |
|
|
set -u
|
|
svc="http://kube-prometheus-stack-grafana.monitoring.svc.cluster.local"
|
|
# busybox wget has no --max-redirect; -S dumps every response header
|
|
# (incl. the initial 302 Location to the IdP) before following, so the
|
|
# grep below still matches the Keycloak authorize URL. (Grafana verified
|
|
# live: 302 -> keycloak.${BASE_DOMAIN}/realms/homelab/.../auth.)
|
|
idp="keycloak.${BASE_DOMAIN}/realms/homelab"
|
|
out="$(wget -S -O /dev/null "$svc/login/generic_oauth" 2>&1 || true)"
|
|
echo "$out"
|
|
if echo "$out" | grep -qiF "$idp"; then
|
|
echo "GRAFANA_OIDC_REDIRECT_OK -> ${idp}"
|
|
else
|
|
echo "FAIL: Grafana did not redirect to ${idp} (this cluster's Keycloak homelab realm)"; exit 1
|
|
fi
|
|
- assert:
|
|
resource:
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: grafana-oidc-probe
|
|
status:
|
|
succeeded: 1
|
|
finally:
|
|
- script:
|
|
content: 'echo "teardown: chainsaw deletes the per-test namespace (Job included)"'
|