kube-prometheus-stack does not scrape flux-system: the controllers are
bootstrapped, expose no Service, and no chart value covers them. So every
gotk_* alert rule in platform/monitoring/alerts.yaml could never fire —
verified on prod, count({__name__=~"gotk.*"}) returned nothing. A repo whose
entire delivery model is Flux had no alerting on reconciliation failing, a
Kustomization erroring, or a resource left suspended.
Proven live before landing: with this PodMonitor applied, gotk_suspend_status
returns 39 series and gotk_reconcile_condition 117, and FluxResourceSuspended
actually fired on a deliberate test suspension — evidence the rule is live,
not merely present.
Extracted from the wider inert-alert audit on
fix/scrape-targets-for-inert-alerts, which stalled unfinished. This is the
self-contained, verified half; Longhorn and cert-manager metrics are still
ABSENT and their alerts remain inert. It also adopts the object into git: it
existed on prod only as kubectl-client-side-apply, so a rebuild would have
lost it.
Co-Authored-By: Claude Opus 5 <[email protected]>
The realm import is a one-shot: once Done=True it never re-runs, so a redirect URI
changed in git never reaches the realm. Healing that needs a DESIRED STATE the
reconciler can read -- and the reconciler pod has no Kubernetes API access at all
(automountServiceAccountToken: false), so it cannot read the KeycloakRealmImport CR.
Its desired state must be mounted, hence a ConfigMap.
A hand-authored ConfigMap would be a second source of truth for every public
hostname in the realm, free to disagree with the import -- the same shape as the bug
being closed. So it is GENERATED from the import and the render is gated, the same
render==golden idiom catalog/render.py already uses for apps/.
URL strings are copied VERBATIM: a ${BASE_DOMAIN} token in the source appears
unchanged here, and both files are rendered by the SAME Flux Kustomization
(./platform-config), so both are substituted in one pass with one value. The
renderer never parses a hostname and never reads cluster-vars, so there is no
second place the domain can be decided.
Co-Authored-By: Claude Opus 5 <[email protected]>
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
Prod's UniFi VM has no disk. `DataVolume unifi/unifi-rootdisk` stuck in
ImportScheduled for hours, because Kyverno denied CDI's own generated pod:
validate.kyverno.svc-fail denied Pod/unifi/importer-prime-<uid>
require-auth-mode-label (R-IDP-02)
require-egress-class-label (R-NET-05)
Both policies match any Pod in an `app-namespace: "true"` namespace with no
exclusion, and CDI generates `importer-*` itself — it cannot carry repo labels.
A COLD-START defect, not a prod quirk: dev's identical import Succeeded 9 days
ago, before the policies enforced, so dev has never exercised it and would hit it
on any rebuild. Fourth bug of this shape in two days.
MECHANISM: a MUTATE ClusterPolicy that ADDS
`egress-class: internet` + `auth-mode: none` (add-if-absent anchor) to Pods on
CREATE in app namespaces WHEN the requesting subject is ServiceAccount
`cdi/cdi-sa`. Both validate policies are untouched — no exclude, no exception,
`background: true` preserved on them.
Why mutate beats exempting: the requirement stays literally true ("every pod in an
app namespace declares both labels") and the values are HONEST — the importer does
pull over the default WAN path, and `internet` is the least-reach class (it does
not grant the VPN path); a transient worker pod with no UI genuinely is
`auth-mode: none`.
THE MATCH IS IDENTITY, NOT A NAME. `subjects: ServiceAccount cdi/cdi-sa` is the
AdmissionReview userInfo, taken from the real prod denial
(`system:serviceaccount:cdi:cdi-sa`, name importer-prime-…, op CREATE). A name
prefix is attacker-choosable; a requesting ServiceAccount is not — and the test
proves it, with a fixture `importer-prime-forged-by-an-app` that copies both the
name prefix AND CDI's labels and is still DENIED. Covers every CDI worker kind
(importer/uploader/cloner) without enumerating names that change per release.
Confirmed on prod that cdi-sa is the only cdi-namespace SA that can create pods.
REJECTED, and the first one is a trap worth recording:
* PolicyException — the CRD EXISTS (v2/v2beta1, from the pre-applied bundle) but
all controllers run `--enablePolicyException=false` (verified live on prod:
admission, background and reports controllers all carry the flag). An exception
would apply cleanly through Flux and be SILENTLY IGNORED. Enabling the gate is a
cluster-wide capability change for no gain over the mutate.
* `exclude` scoped to cdi-sa — sound identity, worse consequences: any userInfo
match forces `background: false`, which would drop background scanning and
policy-report coverage of R-NET-05/R-IDP-02 for ALL EXISTING PODS to fix one pod
class. It also leaves the importer with no labels at all — the unlabelled
fall-through that require-egress-class-label's own header calls a fail-open leak.
* name-prefix / CDI-label matching — attacker-choosable, and proven walkable.
TWO-WAY PROOF, non-vacuous by construction — same resource, different requester,
opposite outcomes (verified by me, not just reported):
userinfo cdi-sa -> mutate pass, require-egress-class pass, require-auth-mode pass
userinfo ordinary -> mutate SKIP, require-egress-class FAIL, require-auth-mode FAIL
Kyverno L0 went 17 -> 29 tests passed, 0 failed. The mutate policy is deliberately
kept OUT of the existing tests/kyverno-test.yaml: that suite declares no userinfo,
and without userinfo the offline engine does not discriminate on subjects — it
mutates everything, which would have silently turned its negative rows into passes.
platform-config/kustomization.yaml is the live apply path for these ClusterPolicies;
without it the policy never reaches a cluster.
RESIDUAL TRADE-OFF, reversible by deleting one file: anything created by cdi-sa in
an app namespace now has these labels supplied rather than being rejected for not
declaring them — including a pod that is not a CDI worker. That identity already
holds `pods create/delete` cluster-wide via ClusterRole `cdi`, so it is not a new
capability; what changes is that for that one identity the declaration is assumed
rather than demanded. restrict-host-namespaces, the namespace default-deny baseline
and the CiliumNetworkPolicies all still apply.
FIRST resource-mutating policy on this cluster: kyverno-resource-mutating-webhook-cfg
holds 0 webhooks today; with autoUpdateWebhooks=true Kyverno registers
mutate.kyverno.svc-fail for Pods within seconds of this applying. Prod's
forceFailurePolicyIgnore=false makes it failurePolicy: Fail — same posture as the
validating webhook already in that path.
CDI retries importer creation on a loop, so the stuck import should proceed once
this lands. The detached prime-be3f9d1c-… 40 GiB scratch PVC is separate operator
cleanup; untouched, as is the DataVolume.
VERIFIED: full CI gate PASS. Nothing applied to any cluster.
Refs R-IDP-02, R-NET-05, R-PLAT-01.
Co-Authored-By: Claude Opus 5 <[email protected]>
`platform-config` applied the CDI CR *and* cdi-storageprofiles.yaml in ONE pass.
StorageProfile's CRD does not exist until cdi-operator reconciles that CR, and
Flux server-side dry-runs every object before applying any — so the unknown kind
aborted the tier, the CR was never applied, the CRD was never created, and the
tier could NEVER converge. Observed on prod: cdi-operator Available 12h, no CDI
CR, no storageprofiles CRD, every downstream tier blocked.
The profiles now live in their own tier ./platform-config-cdi with
dependsOn: platform-config and wait: true. Because dependsOn is REVISION-gated
and platform-config runs wait: true, the CDI CR must be applied AND healthy —
exactly the event that creates the CRD — before the profiles are applied. Cold
clusters order themselves. `standup.sh crds` is retained as a fallback for
clusters already stuck, and because it is idempotent.
THE GENERALISABLE RULE (recorded in clusters/README.md): a CR whose CRD is
created by a CONTROLLER AT RUNTIME is second-order and needs its own pass.
cdi-operator.yaml registers exactly ONE CRD (cdis.cdi.kubevirt.io); storageprofiles,
datavolumes, datasources and dataimportcrons appear only after it reconciles the
CDI CR. An audit found StorageProfile is the only second-order kind in
platform-config — KubeVirt, CDI, KeycloakRealmImport, CNPG Cluster,
VolumeSnapshotClass, ClusterIssuer, PrometheusRule, ServiceMonitor and the Cilium
L2 CRs all have their CRD in git or in a chart.
Moving the CDI CR up into `platform` was rejected: that would put CDI's health on
the tier the CNI gates and everything waits on — the mistake the oauth2-proxy and
headlamp splits exist to undo.
apps-vms gains dependsOn: platform-config-cdi. LOAD-BEARING, not cosmetic:
apps/vms/unifi/20-datavolume.yaml provisions on storageClassName: longhorn-vm and
its comment claimed platform-config guaranteed the profiles — which this move
would have silently falsified.
THE PRUNE TRANSIENT, MEASURED RATHER THAN ASSUMED. platform-config has
prune: true, so it deletes both profiles once, CDI recreates them with empty spec,
and the new tier re-applies the overrides ~30-60s later. Compared our override
against CDI's unaided inference for the same Longhorn provisioner on dev:
longhorn-vm (we override) spec RWX/Block -> status ReadWriteMany/Block
longhorn-static (no override) spec unset -> status ReadWriteOnce/Block
Both infer cloneStrategy: snapshot. So cloneStrategy is NOT at risk; accessModes
IS — unaided CDI gives RWO. A DataVolume provisioned on longhorn-vm inside the
window would be stamped RWO/Block and be PERMANENTLY non-live-migratable. Flux can
no longer cause it (apps-vms gates on the new tier); a human must not do it by hand.
Corroborating evidence that the recreate happens at all: dev has 5 StorageClasses
and 5 StorageProfiles, three with fluxOwner=NONE, spec unset and status populated
— CDI creating and inferring them with no Flux involvement.
MERGE PRECONDITION: `flux get kustomization platform-config` must be Ready on the
target first. If it is not, the profiles are pruned and sit at CDI defaults until
it recovers — and prod was in that state hours earlier. Fallback is the two-step
the Cilium move used (merge with prune: false, verify the ownership label moved,
restore prune: true).
substituteFrom: cluster-vars is declared even though the manifest has no ${TOKEN},
deliberately: conformance/tests/l0-static/flux-postbuild-substitution only audits
paths whose Kustomization declares it, so omitting it would move the file OUT of
the audited set — a blind spot for a future literal-token ship. Gate confirms it
stayed in (path=./platform-config-cdi files=2 defined=0 hazards=0). No SOPS
decryption stanza: the moved file is two plain CRs, so no overlay change was needed.
VERIFIED INDEPENDENTLY OF THE AGENT: platform-config renders 0 StorageProfile
while keeping the CDI and KubeVirt CRs; platform-config-cdi renders exactly the
two profiles; the graph is byte-identical across dev and prod with apps-vms
carrying both edges and no cycle. Full CI gate PASS.
KNOWN FOLLOW-UP: bootstrap/standup.sh TIER_ORDER does not yet include
platform-config-cdi, so `resume` will not wait on it. Another agent holds that
file; the PM will add it.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
Headlamp's HelmRelease sat in the wait:true `platform` Flux tier. Its pod
crashlooped (`flag provided but not defined: -session-ttl` — chart 0.40.1
renders a container arg the appVersion v0.40.1 backend lacks; upstream
kubernetes-sigs/headlamp#4883), and because it gated `platform`, the whole
chain stalled: platform Ready=Unknown -> platform-config, oauth2-proxy,
apps-wave-0/1/2 and apps-vms all Ready=False "dependency not ready".
Part A (unblock + durable lesson): a leaf UI must never gate the chain. Move
all Headlamp resources into a new non-gating Flux Kustomization
clusters/base/headlamp.yaml (dependsOn platform-config, wait:false, SOPS
decryption patched per-env in clusters/dev) — mirroring oauth2-proxy exactly.
The new Kustomization owns the whole platform/headlamp component; its HTTPRoute
moves from platform-config into platform/headlamp/kustomization.yaml. Removed
headlamp from platform/kustomization.yaml and its httproute from
platform-config/kustomization.yaml. Now a crashlooping Headlamp can never stall
platform/platform-config/apps.
Part B (make it start): pin the chart to 0.41.0 (`>=0.41.0 <0.42.0`). The
backend added `f.Int("session-ttl", …)` in v0.41.0 and chart 0.41.0 pins that
same image (appVersion == chart version), so chart+image realign; every arg the
chart renders for our values is supported. OIDC config + per-user RBAC
(clusterRoleBinding.create:false) unchanged.
Gate: `make -C ci gate` PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add Headlamp (kubernetes-sigs/headlamp Helm chart) at
headlamp.home.keithjackson.dev with native Keycloak OIDC login and PER-USER
RBAC: Headlamp forwards each user's id_token to the k3s apiserver, so the
existing oidc:* group bindings (platform-admins->cluster-admin, viewers->view)
apply per user. The ServiceAccount is deliberately NOT bound to cluster-admin
(clusterRoleBinding.create=false), since user requests ride the forwarded token.
Key integration nuance: the apiserver is configured oidc-client-id=kubernetes,
so it only accepts tokens whose aud contains `kubernetes`. The new `headlamp`
Keycloak client carries an oidc-audience-mapper (included.client.audience=
kubernetes) so tokens carry aud=[headlamp, kubernetes] -- accepted by both
Headlamp's own verifier and the apiserver, leaving the apiserver config
untouched. A groups mapper emits the `groups` claim the apiserver maps to RBAC.
- platform/headlamp: namespace, HelmRepository/HelmRelease, SOPS OIDC secret
(externalSecret envFrom), default-deny NetworkPolicy + Cilium ingress
(gateway)/egress (apiserver+Keycloak) policies, README. Wired into
platform/kustomization.yaml (platform tier).
- platform/headlamp/httproute.yaml wired into platform-config (Gateway API CR,
dependsOn platform), modeled on grafana-httproute.
- keycloak-realm-import: add `headlamp` confidential client + groups/audience
mappers + HEADLAMP_CLIENT_SECRET placeholder. Client also created LIVE via
kcadm (realm import no-ops on the existing realm).
- keycloak-realm-clients.sops: add headlamp-client-secret (mirrored into
headlamp-oidc-credentials clientSecret). Both encrypted to the repo age key.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
oauth2-proxy is a leaf forward-auth service that depends on Keycloak +
trusted wildcard TLS + split-horizon DNS. It lived in the gating `platform`
Flux Kustomization (wait:true), so on a cold start it Failed until those
dependencies existed, keeping the whole `platform` tier not-Ready. That in
turn blocked `platform-config` — which brings up the very Keycloak instance,
wildcard Certificate and L2 VIP oauth2-proxy needs (and previously carried
oauth2-proxy's own httproute/servicemonitor) — from ever reconciling. A
self-gating cold-start deadlock, twice-proven (DECISIONS-LOG sessions 7 & 9).
Break the cycle by moving the whole oauth2-proxy component out of the
`platform` tier into its own non-gating post-platform Flux Kustomization:
- clusters/base/oauth2-proxy.yaml: new Kustomization -> ./platform/oauth2-proxy,
dependsOn platform-config, wait:false, substituteFrom cluster-vars.
- clusters/base/kustomization.yaml: reference the new Kustomization.
- platform/kustomization.yaml: drop the oauth2-proxy component (no longer in
the gating tier).
- platform/oauth2-proxy/kustomization.yaml: pull in httproute + servicemonitor
so the full component travels together.
- platform-config/kustomization.yaml: drop the in-place oauth2-proxy
httproute/servicemonitor refs (moved with the controller).
- clusters/dev/kustomization.yaml: add the SOPS decryption patch for the new
oauth2-proxy Kustomization (it now owns the encrypted credentials Secret that
the `platform` Kustomization used to decrypt).
Now `platform` reaches Ready without oauth2-proxy; platform-config brings up
Keycloak/TLS/DNS; then oauth2-proxy reconciles against dependencies that
already exist. wait:false guarantees its health can never gate anything above
it. No config of oauth2-proxy itself changed — only its reconciliation tier.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Lets users kubectl into the cluster via Keycloak SSO, mapping realm groups to
cluster RBAC. The cert-based admin kubeconfig is untouched and remains a fully
independent break-glass path (works even if Keycloak/OIDC is down).
- keycloak-realm-import.yaml: new `kubernetes` PUBLIC client + PKCE (S256) — no
client secret in any kubeconfig; kubelogin runs auth-code+PKCE to localhost.
groups mapper + email/profile/roles scopes.
- k8s-oidc-rbac.yaml (NEW, platform-config): ClusterRoleBindings
oidc:platform-admins -> cluster-admin, oidc:viewers -> view.
- k3s_server role: kube-apiserver-arg OIDC flags (issuer=homelab realm,
client-id=kubernetes, username-claim=email + oidc: prefix, groups-claim=groups
+ oidc: prefix). No oidc-ca-file (nodes reach the issuer via the gateway VIP
and trust the public LE cert). Applied via the rolling Restart k3s handler.
Keycloak/RBAC land via Flux now; the API-server flags take effect on the next
`ansible-playbook playbooks/k3s.yml` rolling restart (deferred until cluster load
from the VM-import incident settles).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Adds the VM/CDI storage tier approved by the PM (RWX for live-migration):
- platform/external-snapshotter (NEW, controllers pass): the cluster-singleton
snapshot.storage.k8s.io control plane (CRDs + snapshot-controller in
kube-system), pinned v8.2.0 (Kubernetes deps aligned to 1.31 since v8.1.0;
this cluster is k3s v1.31.5). These CRDs were absent — only snapshots.longhorn.io
and kubevirt.io snapshot CRDs existed.
- platform/longhorn/volumesnapshotclass.yaml (NEW, platform-config): the
driver.longhorn.io VolumeSnapshotClass (type: snap, in-cluster).
- platform/kubevirt/cdi-storageprofiles.yaml (NEW, platform-config): StorageProfile
longhorn -> cloneStrategy: snapshot; StorageProfile longhorn-vm -> snapshot clone
+ claimPropertySets RWX/Block. Non-degraded default virt class clears
CDIDefaultStorageClassDegraded.
- platform/longhorn/storageclasses.yaml: NEW longhorn-vm StorageClass —
migratable: "true" (Longhorn's KubeVirt-live-migration knob; RWX/Block-only on
1.7.3), is-default-virt-class (NOT the k8s default; longhorn stays RWO default).
- platform/kubevirt/kubevirt-cr.yaml: comment only — LiveMigration is GA on v1.9
(no feature gate); no cluster-wide evictionStrategy (would wedge RWO-disk drains);
live-migration stays per-VM opt-in on a longhorn-vm disk.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The 3x 6GB nodes are OOM (k3s-dev-2's kubelet stopped; dev-1/dev-3 <600MB free). KubeVirt runs
a full control plane (~1.5-2GB) for ZERO running VMs — its IaaS conformance (R-PLAT-03/04,
R-NET-01, R-STATE-01) is already certified, so disabling the running pods loses no delivered
capability. Re-enable (uncomment kubevirt in platform + platform-config) once the VMs' RAM is
grown. Manifests retained; reversible.
Generate the 3 OIDC client secrets (SOPS-encrypted, minimal manifests) so the realm
import can inject the grafana + oauth2-proxy client secrets and both consumers match:
- platform/keycloak/keycloak-realm-clients.sops.yaml (grafana + oauth2-proxy client secrets)
- platform/oauth2-proxy/oauth2-proxy-credentials.sops.yaml (client-id/secret/cookie)
- platform/monitoring/grafana-oidc-credentials.sops.yaml (grafana client secret)
Wire each into its component kustomization and re-enable oauth2-proxy in platform +
platform-config (reverses the temp deferral now that its secret exists).
Values generated by PM per operator instruction (random; recorded out-of-repo at
~/.config/home-k3s/keycloak-sso-secrets.txt for later rotation). Encrypted via sops in the
dev container to the age recipient in .sops.yaml; plaintext never committed.
Serves: R-IDP-01, R-IDP-02, R-IDP-04, R-SEC-01.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
oauth2-proxy's HelmRelease mounts existingSecret oauth2-proxy-credentials, an operator
hand-off not yet filled. Under the unified graph (platform wait:true), the not-ready HR
blocks platform -> platform-config -> the whole downstream graph including KubeVirt.
Deferring it (comment its line in platform/kustomization.yaml + its two entries in
platform-config) lets the no-secret components (KubeVirt/CDI, CNPG, Keycloak operator +
instance) reconcile and be certified now. Re-enable both together once the operator fills
platform/oauth2-proxy/oauth2-proxy-credentials.sops.yaml.
Serves: R-IDP-02 (deferred). PM sequencing correction — enable per phase AND readiness.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Enabling monitoring deadlocked the platform Kustomization: alerts.yaml ships
PrometheusRule CRs whose monitoring.coreos.com CRDs are installed by the
kube-prometheus-stack HelmRelease in the SAME pass → 'no matches for kind
PrometheusRule' → atomic apply failure → NO monitoring HelmReleases created.
Same controllers/configs fix as cert-manager/kyverno: drop alerts.yaml from
platform/monitoring and apply it via platform-config (dependsOn platform), so the
monitoring CRDs exist first. Verified: platform has 0 PrometheusRule (keeps the 3
monitoring HRs); platform-config has the 8 alert rules; gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Un-suspend deadlocked: the single platform Kustomization applies HelmReleases
(which register CRDs) AND custom resources of those CRDs (ClusterIssuer,
Certificate, Kyverno ClusterPolicies) in ONE pass. Flux's server-side apply
fails atomically ('no matches for kind ClusterIssuer') → nothing installs, so
the CRD-registering HelmReleases never run either. Classic ordering deadlock.
Fix (flux2 controllers/configs pattern, applies to dev+prod equally — no
R-ENV-02 divergence): add a platform-config Flux Kustomization (dependsOn
platform, wait) carrying the CRs; platform keeps only the controllers.
- New clusters/base/platform-config.yaml (path ./platform-config, substituteFrom
cluster-vars, no decryption — no SOPS content); wired into base + apps now
dependsOn platform-config (transitively waits for platform).
- platform-config/ references the CR files IN PLACE via ../platform/... (Flux
builds LoadRestrictionsNone; Kyverno policies must stay under platform/kyverno/
for the CI L0 gate). CR resources dropped from the platform
component kustomizations (cert-manager, gateway, kyverno).
- Authored the missing Cilium L2 resources (were only a TODO): CiliumLoadBalancerIPPool
(dev NAT 10.73.0.240-.250) + CiliumL2AnnouncementPolicy (ens*), so Gateway
LoadBalancer listeners get a VIP. Prod range must be parameterized later.
Verified: platform build has zero CRs; platform-config build has 2 ClusterIssuers,
6 ClusterPolicies, Certificate, Gateway, both Cilium L2 CRs; dev/prod build; gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
jakman85andClaude Opus 5