jakman85andClaude Opus 5
6a8953e288
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
64 lines
3.6 KiB
YAML
64 lines
3.6 KiB
YAML
# =============================================================================
|
|
# L0 STATIC — Keycloak client secrets are declared consistently, ordered by a REAL
|
|
# Flux edge, and healed by a reconciler whose behaviour is proven offline
|
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
|
#
|
|
# R-IDP-03 (identity config is declarative) and R-IDP-04 (IdP secrets SOPS-encrypted)
|
|
# are both implemented through ONE mechanism that FAILS OPEN: the KeycloakRealmImport
|
|
# resolves each client's secret from a `(env:NAME)` placeholder mapped to a key of the
|
|
# SOPS-encrypted keycloak-realm-clients Secret. When the operator cannot resolve a
|
|
# placeholder it stores the LITERAL placeholder text as the client secret AND THE IMPORT
|
|
# JOB STILL REPORTS SUCCESS — then rests at Done=True / Started=False and never re-runs.
|
|
#
|
|
# Measured 2026-09-06: prod had all 12 clients holding the literal placeholder (every
|
|
# OIDC login failed with "Invalid client or Invalid client credentials"); dev had exactly
|
|
# one (oauth2-proxy), because that Secret key was added AFTER dev's import completed.
|
|
# Declarative config that silently does not apply is not declarative config.
|
|
#
|
|
# GIVEN the repository as authored,
|
|
# THEN every confidential client declares a placeholder, every placeholder is defined,
|
|
# resolves to the keycloak-realm-clients Secret, and names a key that EXISTS in the
|
|
# SOPS file under the `<clientId>-client-secret` contract the reconciler derives
|
|
# from; public clients declare no secret; the Flux Kustomization applying the Secret
|
|
# is a transitive `dependsOn` ancestor (wait: true) of the ones applying the import
|
|
# and the reconciler — a real edge, not a comment; the reconciler is reconciled from
|
|
# BOTH cluster entrypoints, mounts only what it needs, carries no ServiceAccount
|
|
# token and contains no dollar sign; and its own `--selftest` (extracted from the
|
|
# shipped ConfigMap and executed) proves compare-then-write, idempotence,
|
|
# no-secret-logging, the endpoint allow-list and fail-closed behaviour.
|
|
#
|
|
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git + PyYAML +
|
|
# stdlib); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this Chainsaw
|
|
# Test wraps the SAME script.
|
|
#
|
|
# NOT proven here: that the LIVE realm's stored secrets match git (that is
|
|
# tests/l2-behavioral/keycloak-client-secrets-match-git), that the SOPS ciphertext
|
|
# decrypts to a usable value (key PRESENCE is offline-checkable; plaintext is not), or
|
|
# that the CronJob's schedule fires.
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: keycloak-client-secret-reconciler
|
|
labels:
|
|
conformance.home.k3s/R-IDP-03: "true"
|
|
conformance.home.k3s/R-IDP-04: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-IDP-03, R-IDP-04"
|
|
conformance.home.k3s/layer: "L0"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Every OIDC client's secret is declared once, resolvable, ordered behind a real Flux
|
|
dependency, and continuously reconciled — so an unresolved import placeholder can
|
|
neither be merged nor silently persist in the realm.
|
|
spec:
|
|
steps:
|
|
- name: audit-declaration-ordering-and-reconciler
|
|
description: >-
|
|
Assert the realm import, the SOPS Secret and the reconciler agree; that the Flux
|
|
ordering is a real dependsOn edge; and run the reconciler's own selftest.
|
|
try:
|
|
- script:
|
|
content: bash ./check.sh
|