Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 6 Actions Packages Projects Releases Wiki Activity
Files
04606d48b2bdab7996ecc0e605f75a0ec1704c34
home-network-k3s-impl/platform-config/kustomization.yaml
T
jakman85andClaude Opus 5 6a8953e288 feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing 
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.

Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.

WHAT THIS FIXES

1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
   ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
   anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
   to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
   volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
   database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
   the hour, and evaluates to 3 firing series on prod today.
   Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
   nextcloud-db's exporter cannot log in to its own database (permission denied for
   database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
   for it — the obvious metric would have been blind to one database in three.

2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
   by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
   job="kubelet" (the apiserver copy of the same series reports namespace=default),
   and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
   available=0 forever and would otherwise have fired critical permanently.

3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
   firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
   a node carries the egress-gateway label, with dev's behaviour unchanged today.

4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
   LITERALLY and the import still reports success, then sits at Done=True and never
   re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
   $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
   green. The CronJob is the level-triggered correction; two alerts make its silence
   observable. Repaired live already — this stops it recurring.

Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 00:59:22 -05:00

110 lines
7.9 KiB
YAML
Raw Blame History

---
# platform-config — the CONFIGS half of the controllers/configs split (flux2 idiomatic).
#
# WHY: the `platform` Flux Kustomization installs the CONTROLLERS (HelmReleases) that register CRDs
# (cert-manager.io, kyverno.io, …). The custom resources BELOW use those CRDs, so applying them in
# the SAME pass fails atomically ("no matches for kind ClusterIssuer") and blocks the whole platform
# apply. This Kustomization carries the CRs and is gated `dependsOn: platform` (wait) so the CRDs
# exist first. See clusters/base/platform-config.yaml + DECISIONS-LOG.
#
# Files are referenced IN PLACE via ../platform/... (Flux builds with LoadRestrictionsNone, and the
# Kyverno policies must stay under platform/kyverno/ for the CI L0 `kyverno test` gate). Build/verify
# locally with: kubectl kustomize --load-restrictor LoadRestrictionsNone platform-config
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
# cert-manager: DNS-01 issuers (need cert-manager.io CRDs from the cert-manager HelmRelease)
- ../platform/cert-manager/clusterissuers.yaml
# gateway: the Gateway + wildcard Certificate (Certificate needs cert-manager.io; Gateway needs the
# GatewayClass, which stays in platform/gateway as a controller resource)
- ../platform/gateway/gateway.yaml
- ../platform/gateway/wildcard-certificate.yaml
# kyverno: ClusterPolicies (need kyverno.io CRDs from the Kyverno HelmRelease)
- ../platform/kyverno/policies/require-default-deny-baseline.yaml
- ../platform/kyverno/policies/validate-default-deny-shape.yaml
- ../platform/kyverno/policies/restrict-host-namespaces.yaml
- ../platform/kyverno/policies/require-egress-class-label.yaml
- ../platform/kyverno/policies/require-auth-mode-label.yaml
# COLD-START FIX (2026-09-06, prod): a MUTATE policy that must travel WITH the two label
# policies above. CDI generates its own short-lived worker pods (importer-*, uploader-*,
# cloner-*) inside app namespaces; nothing in git can label them, so both policies DENIED them
# and `DataVolume unifi/unifi-rootdisk` sat in ImportScheduled with no disk. This policy adds
# the two labels to pods created by ServiceAccount cdi/cdi-sa, so the requirement is SATISFIED
# rather than bypassed. Applying it here (not later) matters: if it lands after the validating
# policies, the next import is blocked in the gap.
- ../platform/kyverno/policies/label-cdi-worker-pods.yaml
- ../platform/kyverno/admission/secret-provenance.yaml
# cilium: L2 LB IP pool + announcement policy (cilium.io CRDs already registered by Cilium)
- cilium/l2-ip-pool.yaml
- cilium/l2-announcement-policy.yaml
# longhorn/external-snapshotter: the driver.longhorn.io VolumeSnapshotClass (needs the
# snapshot.storage.k8s.io CRDs from platform/external-snapshotter). Enables CSI snapshots +
# CDI snapshot smart-clone.
- ../platform/longhorn/volumesnapshotclass.yaml
# monitoring: PrometheusRule alert rules (need monitoring.coreos.com CRDs from kube-prometheus-stack)
- ../platform/monitoring/alerts.yaml
# Cluster-wide PVC capacity rules. A SEPARATE FILE from alerts.yaml, whose only PVC-fill rule
# (MonitoringVolumeFillingUp) is scoped namespace="monitoring" — so before this, no database PVC
# was covered by any fill alert. Partitioned by namespace so the two never double-fire; see the
# file header. (2026-09-06, from the CNPG WAL/archiving incident.)
- ../platform/monitoring/pvc-capacity-alerts.yaml
# kubevirt/cdi (Phase 5): the KubeVirt + CDI CRs consume the kubevirt.io / cdi.kubevirt.io CRDs the
# operators register; the ServiceMonitors/alerts consume monitoring.coreos.com. All CRs of platform-
# installed CRDs -> applied here, gated dependsOn platform (F-MON1). The dashboard is a plain
# ConfigMap kept beside them for locality.
- ../platform/kubevirt/kubevirt-cr.yaml
- ../platform/kubevirt/cdi-cr.yaml
# *** DO NOT ADD ../platform/kubevirt/cdi-storageprofiles.yaml BACK HERE. IT DEADLOCKS THIS TIER. ***
# The CDI StorageProfiles were MOVED OUT on 2026-09-05 into their own Flux Kustomization
# (clusters/base/platform-config-cdi.yaml -> ./platform-config-cdi, dependsOn platform-config).
#
# `StorageProfile`'s CRD is NOT registered by the CDI operator INSTALL — cdi-operator.yaml ships only
# `cdis.cdi.kubevirt.io`. cdi-operator creates `storageprofiles` (and datavolumes, datasources, …)
# only AFTER it has reconciled the `CDI` CR two lines above. Flux server-side DRY-RUNS EVERY object
# in a Kustomization before applying ANY of them, so having both in this list meant: unknown kind
# `StorageProfile` -> the whole pass aborts -> the `CDI` CR is never applied -> the CRD is never
# created -> THIS TIER CAN NEVER CONVERGE, and every tier that dependsOn it is blocked forever.
# A true deadlock, not a race. Observed on prod 2026-09-05 (cdi-operator Available 12h, no CR):
# StorageProfile/longhorn dry-run failed: no matches for kind "StorageProfile" in version
# "cdi.kubevirt.io/v1beta1"
# Invisible on dev, whose CDI CRDs have existed for months. See DECISIONS-LOG 2026-09-05.
#
# THE RULE: a CR whose CRD is created by a controller AT RUNTIME (rather than by a manifest in git)
# is SECOND-ORDER and belongs in ./platform-config-cdi, not here.
- ../platform/kubevirt/servicemonitor.yaml
- ../platform/kubevirt/alerts.yaml
- ../platform/kubevirt/dashboard.yaml
# --- Phase 6 — Identity / SSO (ADR-0014) ---
# keycloak: the CNPG DB Cluster + Keycloak instance + realm import (CRs of the CNPG / Keycloak
# operator CRDs installed by platform/cnpg + platform/keycloak) + its ingress + ServiceMonitor.
# ${KEYCLOAK_REPLICAS} in keycloak-instance.yaml is resolved by this Kustomization's substituteFrom.
# cnpg: CloudNativePG platform-health alerts (WAL archiving / recoverability). PrometheusRule is a
# CR of monitoring.coreos.com, so it belongs to this pass, not the platform controllers pass —
# identical wiring to platform/kubevirt/alerts.yaml. Added 2026-09-06 after all three prod CNPG
# clusters ran 38h+ with ContinuousArchiving=False and firstRecoverabilityPoint=null, unalerted.
- ../platform/cnpg/alerts.yaml
- ../platform/keycloak/postgres-cluster.yaml
- ../platform/keycloak/keycloak-instance.yaml
- ../platform/keycloak/keycloak-realm-import.yaml
# The realm import is a ONE-SHOT that fails OPEN: an unresolved secret placeholder is
# stored LITERALLY and the import still reports success, then rests at Done=True and
# never re-runs. This CronJob is the level-triggered loop that corrects that drift
# (2026-09-06 incident; see the file header). It belongs in THIS tier, not ./platform,
# so it applies after the keycloak-realm-clients Secret (rendered by ./platform) and
# alongside the import whose failure mode it covers.
- ../platform/keycloak/client-secret-reconciler.yaml
- ../platform/keycloak/k8s-oidc-rbac.yaml # maps OIDC groups -> cluster roles (kubectl SSO RBAC)
- ../platform/keycloak/httproute.yaml
- ../platform/keycloak/servicemonitor.yaml
# oauth2-proxy: forward-auth httproute + ServiceMonitor MOVED OUT of platform-config. They now
# travel with the oauth2-proxy controller in its own post-platform Flux Kustomization
# (clusters/base/oauth2-proxy.yaml, dependsOn platform-config). Keeping them here — while the
# controller sat in the gating `platform` tier — was one half of the twice-proven cold-start
# deadlock (platform never Ready -> platform-config never reconciles -> these never apply).
# monitoring SSO proof: Grafana native-OIDC login ingress (Gateway API CR)
- ../platform/monitoring/grafana-httproute.yaml
# headlamp: httproute MOVED OUT of platform-config. It now travels with the headlamp component in
# its own post-platform Flux Kustomization (clusters/base/headlamp.yaml, dependsOn platform-config).
# Keeping it here — while the HelmRelease sat in the gating `platform` tier — meant a headlamp
# crashloop stalled platform-config (and everything below it). See DECISIONS-LOG "Headlamp gating stall".
Reference in new issue View Git Blame Copy Permalink