fix(platform): controllers/configs split so CRDs precede their CRs

Un-suspend deadlocked: the single platform Kustomization applies HelmReleases (which register CRDs) AND custom resources of those CRDs (ClusterIssuer, Certificate, Kyverno ClusterPolicies) in ONE pass. Flux's server-side apply fails atomically ('no matches for kind ClusterIssuer') → nothing installs, so the CRD-registering HelmReleases never run either. Classic ordering deadlock. Fix (flux2 controllers/configs pattern, applies to dev+prod equally — no R-ENV-02 divergence): add a platform-config Flux Kustomization (dependsOn platform, wait) carrying the CRs; platform keeps only the controllers. - New clusters/base/platform-config.yaml (path ./platform-config, substituteFrom cluster-vars, no decryption — no SOPS content); wired into base + apps now dependsOn platform-config (transitively waits for platform). - platform-config/ references the CR files IN PLACE via ../platform/... (Flux builds LoadRestrictionsNone; Kyverno policies must stay under platform/kyverno/ for the CI L0 gate). CR resources dropped from the platform component kustomizations (cert-manager, gateway, kyverno). - Authored the missing Cilium L2 resources (were only a TODO): CiliumLoadBalancerIPPool (dev NAT 10.73.0.240-.250) + CiliumL2AnnouncementPolicy (ens*), so Gateway LoadBalancer listeners get a VIP. Prod range must be parameterized later. Verified: platform build has zero CRs; platform-config build has 2 ClusterIssuers, 6 ClusterPolicies, Certificate, Gateway, both Cilium L2 CRs; dev/prod build; gate PASS. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-17 01:06:23 -05:00
1 parent c04aea85aa
commit ca982864fc
@@ -12,9 +12,10 @@ spec:
  timeout: 5m
  prune: true
  wait: true
-  # Ordering: platform (CNI, storage, TLS, DNS, gateway, identity) must be healthy first.
+  # Ordering: the full platform (controllers + their configs) must be healthy first.
+  # platform-config dependsOn platform, so depending on platform-config transitively waits for both.
  dependsOn:
-    - name: platform
+    - name: platform-config
  sourceRef:
    kind: GitRepository
    name: flux-system
@@ -5,6 +5,7 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
  - flux-system.yaml     # GitRepository (reconciliation source)
-  - platform.yaml        # Flux Kustomization -> ./platform
-  - apps.yaml            # Flux Kustomization -> ./apps (dependsOn platform)
+  - platform.yaml        # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
+  - platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
+  - apps.yaml            # Flux Kustomization -> ./apps (dependsOn platform-config)
  - cluster-vars.yaml    # overlay knob surface (base defaults)
@@ -0,0 +1,29 @@
+---
+# Flux Kustomization: PLATFORM CONFIGS — the custom resources (cert-manager ClusterIssuers, the
+# wildcard Certificate + Gateway, Kyverno ClusterPolicies, Cilium L2 IP-pool/announcement) that
+# depend on CRDs registered by the `platform` CONTROLLERS. Split out per the flux2 controllers/
+# configs pattern so the CRDs exist BEFORE these apply — otherwise the single platform apply fails
+# atomically with "no matches for kind ClusterIssuer" and nothing installs (see DECISIONS-LOG).
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: platform-config
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: true
+  dependsOn:
+    - name: platform          # controllers (which register the CRDs) must be healthy first
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform-config
+  # No decryption: platform-config carries no SOPS-encrypted content (the encrypted Secrets live in
+  # ./platform and are applied by the `platform` Kustomization).
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars     # ACME_CLUSTER_ISSUER, KYVERNO_VALIDATION_ACTION, …
@@ -0,0 +1,17 @@
+---
+# CiliumL2AnnouncementPolicy — ARP-announce LoadBalancer VIPs on the node LAN so hosts on
+# 10.73.0.0/24 can reach Gateway listeners at the pool IP (l2announcements.enabled=true in the
+# Cilium HelmRelease turns on the controller; this policy scopes WHAT/WHERE it announces).
+# Announces on the primary ethernet interface (cloud-init names it ens3; `^en` matches ens*/enp*).
+apiVersion: cilium.io/v2alpha1
+kind: CiliumL2AnnouncementPolicy
+metadata:
+  name: default-l2
+spec:
+  loadBalancerIPs: true       # announce LoadBalancer Service VIPs (Gateway listeners)
+  externalIPs: false
+  interfaces:
+    - ^en
+  nodeSelector:
+    matchLabels:
+      kubernetes.io/os: linux
@@ -0,0 +1,16 @@
+---
+# CiliumLoadBalancerIPPool — reserved VIP range for LoadBalancer Services. Gateway API listeners
+# (gatewayClassName: cilium) get their address from here. Range matches the tofu reservation
+# (cilium_l2_vip_range 10.73.0.240-.250) on the k3s-dev NAT 10.73.0.0/24; DHCP is OFF on that
+# network so the range can never be leased elsewhere.
+#
+# NOTE (dev): the range is hardcoded to the dev NAT reservation. A prod overlay must supply the
+# prod LAN range (parameterize via cluster-vars when prod hardware is built). See DECISIONS-LOG.
+apiVersion: cilium.io/v2alpha1
+kind: CiliumLoadBalancerIPPool
+metadata:
+  name: default-pool
+spec:
+  blocks:
+    - start: "10.73.0.240"
+      stop: "10.73.0.250"
@@ -0,0 +1,31 @@
+---
+# platform-config — the CONFIGS half of the controllers/configs split (flux2 idiomatic).
+#
+# WHY: the `platform` Flux Kustomization installs the CONTROLLERS (HelmReleases) that register CRDs
+# (cert-manager.io, kyverno.io, …). The custom resources BELOW use those CRDs, so applying them in
+# the SAME pass fails atomically ("no matches for kind ClusterIssuer") and blocks the whole platform
+# apply. This Kustomization carries the CRs and is gated `dependsOn: platform` (wait) so the CRDs
+# exist first. See clusters/base/platform-config.yaml + DECISIONS-LOG.
+#
+# Files are referenced IN PLACE via ../platform/... (Flux builds with LoadRestrictionsNone, and the
+# Kyverno policies must stay under platform/kyverno/ for the CI L0 `kyverno test` gate). Build/verify
+# locally with: kubectl kustomize --load-restrictor LoadRestrictionsNone platform-config
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  # cert-manager: DNS-01 issuers (need cert-manager.io CRDs from the cert-manager HelmRelease)
+  - ../platform/cert-manager/clusterissuers.yaml
+  # gateway: the Gateway + wildcard Certificate (Certificate needs cert-manager.io; Gateway needs the
+  # GatewayClass, which stays in platform/gateway as a controller resource)
+  - ../platform/gateway/gateway.yaml
+  - ../platform/gateway/wildcard-certificate.yaml
+  # kyverno: ClusterPolicies (need kyverno.io CRDs from the Kyverno HelmRelease)
+  - ../platform/kyverno/policies/require-default-deny-baseline.yaml
+  - ../platform/kyverno/policies/validate-default-deny-shape.yaml
+  - ../platform/kyverno/policies/restrict-host-namespaces.yaml
+  - ../platform/kyverno/policies/require-egress-class-label.yaml
+  - ../platform/kyverno/policies/require-auth-mode-label.yaml
+  - ../platform/kyverno/admission/secret-provenance.yaml
+  # cilium: L2 LB IP pool + announcement policy (cilium.io CRDs already registered by Cilium)
+  - cilium/l2-ip-pool.yaml
+  - cilium/l2-announcement-policy.yaml
@@ -5,7 +5,7 @@ resources:
  - namespace.yaml
  - helmrepository.yaml
  - helmrelease.yaml
-  - clusterissuers.yaml
-  # SOPS-encrypted Cloudflare DNS-01 token (ClusterIssuer apiTokenSecretRef). Operator flow:
  # SOPS-encrypted Cloudflare DNS-01 token (ClusterIssuer apiTokenSecretRef). Flux decrypts at reconcile.
  - cloudflare-api-token.sops.yaml
+  # NOTE: clusterissuers.yaml is a cert-manager.io CR → applied by the platform-config Kustomization
+  # (dependsOn platform) so the cert-manager CRDs exist first. File stays here for locality.
@@ -4,5 +4,6 @@ kind: Kustomization
 resources:
  - namespace.yaml
  - gatewayclass.yaml
-  - wildcard-certificate.yaml
-  - gateway.yaml
+  # NOTE: gateway.yaml (Gateway) + wildcard-certificate.yaml (cert-manager.io Certificate) are CRs
+  # → applied by the platform-config Kustomization (dependsOn platform) so the GatewayClass and
+  # cert-manager CRDs exist first. Files stay here for locality.
@@ -16,9 +16,6 @@ namespace: kyverno
 resources:
  - admission/namespace.yaml
  - admission/helmrelease.yaml
-  - policies/require-default-deny-baseline.yaml
-  - policies/validate-default-deny-shape.yaml
-  - policies/restrict-host-namespaces.yaml
-  - policies/require-egress-class-label.yaml
-  - policies/require-auth-mode-label.yaml
-  - admission/secret-provenance.yaml
+  # NOTE: the ClusterPolicies (policies/*.yaml) + admission/secret-provenance.yaml are kyverno.io CRs
+  # → applied by the platform-config Kustomization (dependsOn platform) so the Kyverno CRDs exist
+  # first. Files STAY here so the CI L0 gate (`kyverno test platform/kyverno/`) still discovers them.