feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
d1a6628a97
commit
6a8953e288
46 files changed
+7898
-71
No files matched your search
@@ -0,0 +1,212 @@
|
||||
# =============================================================================
|
||||
# L2 BEHAVIORAL — every OIDC client's STORED secret equals the one git declares
|
||||
# (R-IDP-02, R-IDP-04) [RUNNABLE]
|
||||
# Spec: docs/15 identity-and-access; ADR-0014.
|
||||
# Impl: platform/keycloak/client-secret-reconciler.yaml (the CronJob + its script).
|
||||
#
|
||||
# WHY THIS TEST EXISTS
|
||||
# --------------------
|
||||
# The KeycloakRealmImport resolves each confidential client's secret from a `(env:NAME)`
|
||||
# placeholder mapped to a key of the SOPS-encrypted keycloak-realm-clients Secret. When
|
||||
# the Keycloak Operator cannot resolve a placeholder it stores the LITERAL placeholder
|
||||
# text as the client secret AND THE IMPORT JOB STILL REPORTS SUCCESS; the CR then rests
|
||||
# at Done=True / Started=False and never re-runs. Measured 2026-09-06: prod had all 12
|
||||
# clients holding the literal placeholder — every OIDC login on the cluster failed with
|
||||
# "Invalid client or Invalid client credentials" — and dev had one (oauth2-proxy), whose
|
||||
# Secret key was added after dev's import had already completed.
|
||||
#
|
||||
# Every SIGNAL the cluster offered was green: the import Job succeeded, the CR read
|
||||
# Done=True, the Keycloak pods were healthy, and the existing keycloak-config-as-code
|
||||
# test (which asserts Done=True) PASSED throughout. The only observable was a human
|
||||
# failing to log in. This test is that observable, automated: it compares STORED against
|
||||
# DECLARED for every client and fails on any difference, so the class cannot sit
|
||||
# undetected for months again.
|
||||
#
|
||||
# GIVEN the `homelab` realm and the SOPS-encrypted keycloak-realm-clients Secret,
|
||||
# WHEN the reconciler's script is run in --audit mode (READ-ONLY: it cannot write, and
|
||||
# its endpoint allow-list makes users/groups/roles unreachable),
|
||||
# -> THEN every declared client's stored secret matches git, and none holds an
|
||||
# unresolved import placeholder. Emits KEYCLOAK_CLIENT_SECRETS_AUDIT_OK.
|
||||
#
|
||||
# WHY IT RUNS THE SHIPPED SCRIPT rather than re-implementing the comparison: a detector
|
||||
# that duplicates the healer's logic drifts from it. This audits with the same code, in
|
||||
# the same mode the L0 gate selftests, so a passing audit is evidence about the healer
|
||||
# too. It runs as a Job IN the keycloak namespace (not in the ephemeral chainsaw
|
||||
# namespace) because the desired state is the mounted keycloak-realm-clients Secret and
|
||||
# the admin credential is keycloak-admin-credentials — both namespace-local, which is
|
||||
# also why neither this Job nor the CronJob needs any Kubernetes API access.
|
||||
#
|
||||
# chainsaw v0.2.12: Job + status assert; the leftover-Job delete keeps a re-run from
|
||||
# colliding with an immutable Job left by a previous failure.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: keycloak-client-secrets-match-git
|
||||
labels:
|
||||
conformance.home.k3s/R-IDP-02: "true"
|
||||
conformance.home.k3s/R-IDP-04: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-IDP-02, R-IDP-04"
|
||||
conformance.home.k3s/layer: "L2"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Every OIDC client's stored secret equals the SOPS-declared one and none holds an
|
||||
unresolved import placeholder — the observable that was missing when all 12 prod
|
||||
clients were broken behind a green realm import.
|
||||
spec:
|
||||
timeouts:
|
||||
apply: 60s
|
||||
assert: 5m
|
||||
exec: 5m
|
||||
steps:
|
||||
- name: given-the-reconciler-is-reconciled-onto-the-cluster
|
||||
description: >-
|
||||
The audit Job runs the CronJob's own script from its ConfigMap; if Flux has not
|
||||
applied the reconciler, there is nothing to audit WITH and that is the failure.
|
||||
try:
|
||||
- assert:
|
||||
resource:
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: keycloak-client-secret-reconciler
|
||||
namespace: keycloak
|
||||
- assert:
|
||||
resource:
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: keycloak-client-secret-reconciler
|
||||
namespace: keycloak
|
||||
spec:
|
||||
concurrencyPolicy: Forbid
|
||||
- name: then-every-stored-client-secret-matches-git
|
||||
try:
|
||||
- script:
|
||||
content: |
|
||||
kubectl -n keycloak delete job keycloak-client-secret-audit \
|
||||
--ignore-not-found --wait=true
|
||||
- apply:
|
||||
resource:
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: keycloak-client-secret-audit
|
||||
namespace: keycloak
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak-client-secret-audit
|
||||
app.kubernetes.io/part-of: conformance
|
||||
spec:
|
||||
backoffLimit: 0
|
||||
activeDeadlineSeconds: 240
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak-client-secret-audit
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
automountServiceAccountToken: false
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 65532
|
||||
runAsGroup: 65532
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: audit
|
||||
image: python:3.13.15-alpine3.23
|
||||
command:
|
||||
- python3
|
||||
- /opt/reconciler/reconcile-client-secrets.py
|
||||
# READ-ONLY. --audit never issues a write on any path; the
|
||||
# script's --selftest proves that (L0 gate).
|
||||
- --audit
|
||||
env:
|
||||
- name: KEYCLOAK_BASE_URL
|
||||
value: http://keycloak-service.keycloak.svc.cluster.local:8080
|
||||
- name: KEYCLOAK_REALM
|
||||
value: homelab
|
||||
- name: PYTHONDONTWRITEBYTECODE
|
||||
value: "1"
|
||||
- name: PYTHONUNBUFFERED
|
||||
value: "1"
|
||||
- name: KEYCLOAK_ADMIN_USER
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: keycloak-admin-credentials
|
||||
key: admin-user
|
||||
- name: KEYCLOAK_ADMIN_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: keycloak-admin-credentials
|
||||
key: admin-password
|
||||
volumeMounts:
|
||||
- name: script
|
||||
mountPath: /opt/reconciler
|
||||
readOnly: true
|
||||
- name: client-secrets
|
||||
mountPath: /etc/keycloak-realm-clients
|
||||
readOnly: true
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 48Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
volumes:
|
||||
- name: script
|
||||
configMap:
|
||||
name: keycloak-client-secret-reconciler
|
||||
defaultMode: 0444
|
||||
- name: client-secrets
|
||||
secret:
|
||||
secretName: keycloak-realm-clients
|
||||
defaultMode: 0400
|
||||
- assert:
|
||||
resource:
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: keycloak-client-secret-audit
|
||||
namespace: keycloak
|
||||
status:
|
||||
succeeded: 1
|
||||
# The Job's exit status already encodes the verdict, but the sentinel is what the
|
||||
# run LEDGER records (conformance/runs/*/run.log): a JUnit alone cannot tell a real
|
||||
# audit from a Job that succeeded without auditing anything.
|
||||
- script:
|
||||
content: |
|
||||
set -u
|
||||
out="$(kubectl -n keycloak logs job/keycloak-client-secret-audit \
|
||||
--tail=-1 2>&1 || true)"
|
||||
echo "$out"
|
||||
echo "$out" | grep -q 'KEYCLOAK_CLIENT_SECRETS_AUDIT_OK' \
|
||||
|| { echo "FAIL: the audit did not report every client secret matching git"; exit 1; }
|
||||
echo "$out" | grep -q 'UNRESOLVED IMPORT PLACEHOLDER' \
|
||||
&& { echo "FAIL: a client still holds the literal import placeholder"; exit 1; }
|
||||
echo KEYCLOAK_CLIENT_SECRET_DRIFT_NONE_OK
|
||||
check:
|
||||
(contains($stdout, 'KEYCLOAK_CLIENT_SECRET_DRIFT_NONE_OK')): true
|
||||
catch:
|
||||
- describe:
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
namespace: keycloak
|
||||
name: keycloak-client-secret-audit
|
||||
- podLogs:
|
||||
namespace: keycloak
|
||||
selector: app.kubernetes.io/name=keycloak-client-secret-audit
|
||||
tail: 60
|
||||
finally:
|
||||
- script:
|
||||
content: |
|
||||
kubectl -n keycloak delete job keycloak-client-secret-audit \
|
||||
--ignore-not-found --wait=false
|
||||
Reference in new issue
Block a user