jakman85andClaude Opus 5
55b232e2ce
The operator got no mail when Flux began applying a git change and none when it finished. Measured cause: notification-controller v1.4.0 is running and its CRDs serve v1beta3, but there are ZERO Provider and ZERO Alert objects in the cluster — nothing in git ever created any. The obvious fix does not work. Neither Flux event can be filtered into a deploy signal: ReconciliationSucceeded is a heartbeat (101 events in one hour across 16 Kustomizations, message byte-identical whether the apply changed anything), and Progressing is worse (five Kustomizations re-apply the same object on every tick — event counts 2505/2504/2501/2495/2490). gotk_* carries no revision label at all (gotk_reconcile_condition 120 series, gotk_suspend_status 40, none with one). The applied revision is the only signal that moves when and only when git moves, and it lives in object status alone. So export it via the kube-state-metrics custom-resource-state path that already serves the Velero Schedule metrics — kube_flux_kustomization_info/_suspended and kube_flux_gitrepository_info — and ask the question in PromQL: FluxRolloutInProgress (severity: info, no `for:`) fires the moment a non-suspended Kustomization's applied revision is not the revision the GitRepository fetched from main, and CLEARS when all have caught up. Fired mail = a deploy started; RESOLVED mail = it completed. The completion half is send_resolved, not a second rule. FluxRolloutStuck (severity: warning, for: 30m) is the same expression as the escalation for a rollout that started and never converged. Two guards, each with a test whose only job is to fail if the clause is removed: `and on () (count(gitrepository_info) > 0)` — without it a missing exporter makes `unless on (revision) <empty>` remove nothing and all 16 Kustomizations alert forever, the permanently-firing mode this repo shipped once already — and the suspended-Kustomization exclusion, without which a suspended tier holds the rollout open and the "applied" mail never arrives. A dedicated Alertmanager severity=info route + email-deploy receiver keeps deploy notices off the [WARNING] catch-all. podAnnotations carries a roll-counter because the chart sets no checksum over the custom-resource-state ConfigMap, so a config-only edit would leave KSM on the old collector set with the new alerts silently inert. Verified: promtool SUCCESS on 6 scenarios (both guards, plus the convergence test that proves the alert empties itself); lint/l0/render-golden/l1/decisions-log/ spec-immutability green in the pinned toolbox. NOT verified: no mail has been sent — the metrics do not exist until this merges. Full gap list, and the deliberately-cut notification-controller half, in the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
platform/
Cluster-wide platform services, Flux-managed (R-PLAT-01, R-GITOPS-01/02/03). Reconciled
by the platform Flux Kustomization defined in clusters/base/platform.yaml,
before apps/.
Governing spec: docs/05-gitops-and-delivery.md,
docs/01-architecture-overview.md.
Phase 0 scaffolding only. Every component here is a structurally-correct stub — a Flux HelmRelease and/or Kustomization with a values placeholder, a README citing the governing ADR/spec, and TODOs for the phase that fills it in. No cluster exists; nothing reconciles yet. Do not add working config, secrets, or provisioning under Phase 0 (spec
AGENTS.md§1/§5/§6,roadmap/phases.md).
Reconciliation flow (how a change lands)
git (source of truth)
│ operator PR → CI (lint · schema · Kyverno L0 · Chainsaw) → merge to tracked branch
▼
Flux GitRepository (clusters/base/flux-system.yaml)
▼
Flux Kustomization "platform" → ./platform (this dir) ── substituteFrom cluster-vars ──┐
│ prune=true (drift correction, R-GITOPS-03) · wait=true │ R-ENV-02
▼ │
Flux Kustomization "apps" → ./apps (dependsOn platform) ─ substituteFrom cluster-vars ──┘
Git is authoritative; the cluster conforms to git and drift is auto-corrected (R-GITOPS-01/03).
Manual kubectl apply of platform state is prohibited (spec AGENTS.md §5).
The dev ↔ prod overlay model (R-ENV-02)
Platform manifests are authored once. Every value that legitimately differs between environments
is a ${TOKEN} resolved from the cluster-vars ConfigMap via Flux postBuild.substituteFrom. The
clusters/dev and clusters/prod overlays differ only by the values they patch into
cluster-vars (issuer, replica counts, storage sizes, retention). See
clusters/README.md. Base manifests are never forked per environment.
Components (one directory each)
| Dir | Component | Governing ADR / spec | Serves |
|---|---|---|---|
cilium/ |
CNI + kube-proxy replacement + Gateway API + Egress Gateway + Hubble | ADR-0002, docs/03 | R-NET-* |
longhorn/ |
Replicated block storage; 3-replica default StorageClass + bulk tier | ADR-0003, docs/04 | R-STATE-* |
cert-manager/ |
TLS via Let's Encrypt DNS-01 (Cloudflare); staging + prod ClusterIssuers | ADR-0006, docs/08 | R-TLS-* |
externaldns/ |
Automatic DNS records for public services (Cloudflare) | docs/09 | R-DNS-* |
gateway/ |
Gateway API GatewayClass + shared HTTPS Gateway + wildcard cert | ADR-0002/ADR-0006, docs/09 | R-DNS-01, R-TLS-01 |
monitoring/ |
kube-prometheus-stack + Loki + Grafana Alloy + Alertmanager | docs/10 | R-OBS-* |
velero/ |
Off-cluster backup/DR of volumes + resource state | ADR-0003, docs/04 | R-STATE-03 |
kubevirt/ |
KubeVirt + CDI (IaaS: application VMs on the same K8s surface) | ADR-0005, docs/06 | R-PLAT-03 |
image-automation/ |
Flux image-update-automation (hands-off image tag advance) | ADR-0009, docs/11 | R-UPD-02 |
keycloak/ |
Platform IdP (OIDC + SAML), config-as-code, PostgreSQL on Longhorn | ADR-0014, docs/15 | R-IDP-* |
oauth2-proxy/ |
Forward-auth at the Gateway for non-OIDC apps → Keycloak | ADR-0014, docs/15 | R-IDP-* |
kyverno/ |
L0 policy-as-code gate — OWNED BY THE POLICY ENGINEER (WS-TEST). | ADR-0011, docs/14 | R-TEST-* |
⚠️ INTEGRATION TODO(PM): wire kyverno into the platform kustomization
platform/kyverno/ is authored by a different agent (the policy engineer, WS-TEST). This
workstream did not create any files under platform/kyverno/ and must not.
platform/kustomization.yaml intentionally omits - kyverno. Once the
policy engineer's files land, the PM must add - kyverno to that list (recommended first, so
policies are present before other components reconcile). The line is present as a commented TODO in
the kustomization.
Secrets
Every secret referenced below (Cloudflare API token, Velero backup-target creds, Keycloak admin/DB, oauth2-proxy cookie/client secrets) is referenced by name only and will be committed SOPS-encrypted (WS-SEC, ADR-0007). No plaintext secret appears anywhere in this tree.
Cross-workstream contracts surfaced by these stubs
- cilium (WS-NET ↔ WS-INFRA): Cilium's initial CNI install is done by Ansible at cluster bootstrap (docs/02, Phase 1). The HelmRelease here manages Cilium config thereafter; its Helm values MUST match the Ansible bootstrap values (kubeProxyReplacement, k8sServiceHost/Port, API server host) or reconciliation will fight the bootstrap. Contract owner: WS-NET + WS-INFRA.
- gateway + cert-manager + externaldns (WS-NET + WS-SEC): the shared Gateway listener, the
wildcard
*.home.keithjackson.devcert, and the ExternalDNS source all reference each other; they form the "Issuer + Gateway TLS" interface inroadmap/workstreams.md. - image-automation (WS-UPDATE): this dir holds only the cluster-wide ImageUpdateAutomation +
git write-back. Per-service
ImageRepository/ImagePolicyare rendered from each ServiceDefinition'supdateblock (WS-GITOPS/WS-UPDATE), living withapps/. - keycloak/oauth2-proxy (WS-SEC): the SSO/auth contract (
authblock → Keycloak client / oauth2-proxy protection) is owned by WS-SEC; these are placeholders pending that contract.