jakman85andClaude Opus 5
6a8953e288
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
83 lines
4.4 KiB
YAML
83 lines
4.4 KiB
YAML
# =============================================================================
|
|
# L2 BEHAVIORAL — identity config as code (R-IDP-03, R-IDP-04, R-IDP-06) [RUNNABLE]
|
|
# Spec: docs/15 identity-and-access; ADR-0014. Impl: platform/keycloak/keycloak-realm-import.yaml.
|
|
#
|
|
# GIVEN the `homelab` realm (groups platform-admins/media-users/viewers, OIDC clients, brute-force,
|
|
# and the conditional MFA browser flow) is declared as a KeycloakRealmImport in git,
|
|
# WHEN the Keycloak Operator reconciles it
|
|
# -> THEN the import reaches Done=True — realm/groups/clients/roles + the MFA flow are applied
|
|
# PURELY from git (R-IDP-03 config-as-code; R-IDP-04 MFA flow bound; R-IDP-06 groups/roles for
|
|
# least-privilege). No console clicking.
|
|
#
|
|
# Goes GREEN once Keycloak is up + the realm client-secret hand-off (keycloak-realm-clients) is filled
|
|
# so the placeholder resolves and the import completes. chainsaw v0.2.12: script + block contains check.
|
|
#
|
|
# NOT PROVEN HERE — READ THIS BEFORE TRUSTING A GREEN RUN (added 2026-09-06).
|
|
# Done=True means the import JOB EXITED 0. It does NOT mean the placeholders resolved: when the
|
|
# operator cannot resolve one it stores the LITERAL `(env:...)` text as the client secret and the
|
|
# import still reports success, then rests at Done=True / Started=False and never re-runs. THIS
|
|
# TEST PASSED THROUGHOUT the 2026-09-06 outage in which all 12 prod clients held the literal
|
|
# placeholder and every OIDC login on the cluster failed. Done=True is a necessary, badly
|
|
# insufficient condition.
|
|
# The sufficient check is tests/l2-behavioral/keycloak-client-secrets-match-git, which compares
|
|
# each client's STORED secret against the SOPS-declared one; the static half is
|
|
# tests/l0-static/keycloak-client-secret-reconciler. Do not "simplify" by folding those into this
|
|
# test: the whole lesson is that the import's own success signal is not evidence.
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: keycloak-config-as-code
|
|
labels:
|
|
conformance.home.k3s/R-IDP-03: "true"
|
|
conformance.home.k3s/R-IDP-04: "true"
|
|
conformance.home.k3s/R-IDP-06: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-IDP-03, R-IDP-04, R-IDP-06"
|
|
conformance.home.k3s/layer: "L2"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
The homelab realm (groups, clients, MFA flow) reconciles to Done purely from the git-managed
|
|
KeycloakRealmImport — no manual console configuration.
|
|
spec:
|
|
timeouts:
|
|
exec: 5m
|
|
steps:
|
|
- name: realm-import-reconciles-from-git
|
|
try:
|
|
- script:
|
|
content: |
|
|
set -u
|
|
# Poll the KeycloakRealmImport until the operator reports Done=True.
|
|
for i in $(seq 1 30); do
|
|
st="$(kubectl -n keycloak get keycloakrealmimport homelab \
|
|
-o jsonpath='{.status.conditions[?(@.type=="Done")].status}' 2>/dev/null || true)"
|
|
echo "poll $i: Done=$st"
|
|
if [ "$st" = "True" ]; then
|
|
echo "REALM_IMPORT_DONE_OK"
|
|
break
|
|
fi
|
|
sleep 10
|
|
done
|
|
check:
|
|
(contains($stdout, 'REALM_IMPORT_DONE_OK')): true
|
|
- name: privileged-group-and-mfa-flow-present
|
|
try:
|
|
- script:
|
|
content: |
|
|
set -u
|
|
# The realm import declares the browser-mfa flow + platform-admins group. Confirm the
|
|
# realm CR / import spec carries them (declared-in-git evidence; the live realm state is
|
|
# exercised by the login flows).
|
|
flow="$(kubectl -n keycloak get keycloakrealmimport homelab \
|
|
-o jsonpath='{.spec.realm.browserFlow}' 2>/dev/null || true)"
|
|
grp="$(kubectl -n keycloak get keycloakrealmimport homelab \
|
|
-o jsonpath='{.spec.realm.groups[*].name}' 2>/dev/null || true)"
|
|
echo "browserFlow=$flow groups=$grp"
|
|
echo "$flow" | grep -q 'browser-mfa' || { echo "FAIL: MFA browser flow not bound"; exit 1; }
|
|
echo "$grp" | grep -q 'platform-admins' || { echo "FAIL: platform-admins group missing"; exit 1; }
|
|
echo "MFA_AND_GROUPS_OK"
|
|
check:
|
|
(contains($stdout, 'MFA_AND_GROUPS_OK')): true
|