jakman85andClaude Opus 5
4687521d65
LonghornVolumeCapacityHigh went pending within minutes of the Longhorn scrape
landing today. Re-measured on prod: it read 0.9731 on the Prometheus TSDB volume
(actual_size 52243681280 / capacity 53687091200) while the kubelet reported the
SAME filesystem 20.8% used. actual_size is head + pinned snapshots, and one
user_created=false system snapshot from prod-1's replica rebuild holds 40510484480
bytes (37.7 GiB) of it; Longhorn returns no freed blocks without discard/TRIM, and
retentionSize 45GiB on a 50Gi volume targets 90% by design. The ratio is pinned
near 1.0 and cannot fall back under 0.85 — a permanently-firing alert, the same
shape as CNPGNoRecoverabilityPoint.
DELETED, not re-thresholded. Raising the threshold would tune away a true
statement about allocation. Redundancy proven, not asserted: of 24 Longhorn
volumes on prod, 23 appear in kubelet_volume_stats_inodes > 0 and so fall inside
MonitoringVolumeFillingUp (namespace="monitoring") or PersistentVolumeFillingUp /
PersistentVolumeAlmostFull. The 24th is unifi/unifi-rootdisk, volumeMode Block —
no filesystem for anyone to measure, and the deleted rule was not covering it
either.
AND A SECOND INERTNESS the scrape did not fix. longhorn_volume_robustness on
longhorn 1.12.1 is one-hot over a `state` label with values 0/1, not the 0-3 enum
the rules assumed:
count(longhorn_volume_robustness) => 96 (24 volumes x 4 states)
count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
count(longhorn_volume_robustness == 2) => (empty)
count(longhorn_volume_robustness == 3) => (empty)
So a FAULTED volume would still have alerted nobody. Re-keyed to the label; the
"44 series / 72 unknown(0, detached)" note in platform/longhorn/helmrelease.yaml
was the same misreading and is corrected in place.
The allocation question moved to where allocation is a budget. LonghornNodeStorageHigh
fired at 0.85, but storage-minimal-available-percentage is 25 on this cluster, so a
disk leaves the scheduling pool at 75% used — the old threshold could only fire
after scheduling had already been refused. Replaced by:
LonghornDiskAllocationHigh (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70, 30m
LonghornDiskUnschedulable longhorn_disk_status{condition="schedulable"} == 0, 15m
Per DISK, not per node: longhorn_node_storage_usage_bytes sums a node's disks, so
one full disk could hide behind a half-empty sibling. Measured prod usage
0.2254 / 0.2306 / 0.2369 on prod-1/2/3.
PROVEN NEITHER PERMANENTLY FIRING NOR INERT. platform/monitoring/tests/longhorn.test.yaml
adds 6 cases pinning every rule quiet on today's measured prod state and firing on
a constructed bad state, with every `for:` asserted, plus regression assertions that
the old `== 2`/`== 3` forms match nothing and that per-disk 0.72 is visible where the
node-summed 0.475 is not. Mutation-checked 11 ways; each turns the suite RED. Live on
prod: all four new expressions return empty, while the sum of their selector
cardinalities is 54. The L0 UNTESTED_ALERTS register shrinks by seven (7/56 -> 14/58
alert rules unit-tested), which also brings MonitoringVolumeFillingUp,
PersistentVolumeFillingUp and PersistentVolumeAlmostFull under test since the
redundancy claim rests on them.
REPORTED, NOT CHANGED (DECISIONS-LOG, owner decisions): retentionSize 45GiB on a
50Gi volume is too tight — recommend 36GiB, with reasoning; and discard/TRIM on the
longhorn StorageClass is feasible but not advisable now — remove-snapshots-during-
filesystem-trim reads false on prod, so trim would not have reclaimed the 37.7 GiB
anyway, and continuous discard on nodes where the etcd WAL shares the spindle points
at this cluster's known failure mode.
Co-Authored-By: Claude Opus 5 <[email protected]>
platform/
Cluster-wide platform services, Flux-managed (R-PLAT-01, R-GITOPS-01/02/03). Reconciled
by the platform Flux Kustomization defined in clusters/base/platform.yaml,
before apps/.
Governing spec: docs/05-gitops-and-delivery.md,
docs/01-architecture-overview.md.
Phase 0 scaffolding only. Every component here is a structurally-correct stub — a Flux HelmRelease and/or Kustomization with a values placeholder, a README citing the governing ADR/spec, and TODOs for the phase that fills it in. No cluster exists; nothing reconciles yet. Do not add working config, secrets, or provisioning under Phase 0 (spec
AGENTS.md§1/§5/§6,roadmap/phases.md).
Reconciliation flow (how a change lands)
git (source of truth)
│ operator PR → CI (lint · schema · Kyverno L0 · Chainsaw) → merge to tracked branch
▼
Flux GitRepository (clusters/base/flux-system.yaml)
▼
Flux Kustomization "platform" → ./platform (this dir) ── substituteFrom cluster-vars ──┐
│ prune=true (drift correction, R-GITOPS-03) · wait=true │ R-ENV-02
▼ │
Flux Kustomization "apps" → ./apps (dependsOn platform) ─ substituteFrom cluster-vars ──┘
Git is authoritative; the cluster conforms to git and drift is auto-corrected (R-GITOPS-01/03).
Manual kubectl apply of platform state is prohibited (spec AGENTS.md §5).
The dev ↔ prod overlay model (R-ENV-02)
Platform manifests are authored once. Every value that legitimately differs between environments
is a ${TOKEN} resolved from the cluster-vars ConfigMap via Flux postBuild.substituteFrom. The
clusters/dev and clusters/prod overlays differ only by the values they patch into
cluster-vars (issuer, replica counts, storage sizes, retention). See
clusters/README.md. Base manifests are never forked per environment.
Components (one directory each)
| Dir | Component | Governing ADR / spec | Serves |
|---|---|---|---|
cilium/ |
CNI + kube-proxy replacement + Gateway API + Egress Gateway + Hubble | ADR-0002, docs/03 | R-NET-* |
longhorn/ |
Replicated block storage; 3-replica default StorageClass + bulk tier | ADR-0003, docs/04 | R-STATE-* |
cert-manager/ |
TLS via Let's Encrypt DNS-01 (Cloudflare); staging + prod ClusterIssuers | ADR-0006, docs/08 | R-TLS-* |
externaldns/ |
Automatic DNS records for public services (Cloudflare) | docs/09 | R-DNS-* |
gateway/ |
Gateway API GatewayClass + shared HTTPS Gateway + wildcard cert | ADR-0002/ADR-0006, docs/09 | R-DNS-01, R-TLS-01 |
monitoring/ |
kube-prometheus-stack + Loki + Grafana Alloy + Alertmanager | docs/10 | R-OBS-* |
velero/ |
Off-cluster backup/DR of volumes + resource state | ADR-0003, docs/04 | R-STATE-03 |
kubevirt/ |
KubeVirt + CDI (IaaS: application VMs on the same K8s surface) | ADR-0005, docs/06 | R-PLAT-03 |
image-automation/ |
Flux image-update-automation (hands-off image tag advance) | ADR-0009, docs/11 | R-UPD-02 |
keycloak/ |
Platform IdP (OIDC + SAML), config-as-code, PostgreSQL on Longhorn | ADR-0014, docs/15 | R-IDP-* |
oauth2-proxy/ |
Forward-auth at the Gateway for non-OIDC apps → Keycloak | ADR-0014, docs/15 | R-IDP-* |
kyverno/ |
L0 policy-as-code gate — OWNED BY THE POLICY ENGINEER (WS-TEST). | ADR-0011, docs/14 | R-TEST-* |
⚠️ INTEGRATION TODO(PM): wire kyverno into the platform kustomization
platform/kyverno/ is authored by a different agent (the policy engineer, WS-TEST). This
workstream did not create any files under platform/kyverno/ and must not.
platform/kustomization.yaml intentionally omits - kyverno. Once the
policy engineer's files land, the PM must add - kyverno to that list (recommended first, so
policies are present before other components reconcile). The line is present as a commented TODO in
the kustomization.
Secrets
Every secret referenced below (Cloudflare API token, Velero backup-target creds, Keycloak admin/DB, oauth2-proxy cookie/client secrets) is referenced by name only and will be committed SOPS-encrypted (WS-SEC, ADR-0007). No plaintext secret appears anywhere in this tree.
Cross-workstream contracts surfaced by these stubs
- cilium (WS-NET ↔ WS-INFRA): Cilium's initial CNI install is done by Ansible at cluster bootstrap (docs/02, Phase 1). The HelmRelease here manages Cilium config thereafter; its Helm values MUST match the Ansible bootstrap values (kubeProxyReplacement, k8sServiceHost/Port, API server host) or reconciliation will fight the bootstrap. Contract owner: WS-NET + WS-INFRA.
- gateway + cert-manager + externaldns (WS-NET + WS-SEC): the shared Gateway listener, the
wildcard
*.home.keithjackson.devcert, and the ExternalDNS source all reference each other; they form the "Issuer + Gateway TLS" interface inroadmap/workstreams.md. - image-automation (WS-UPDATE): this dir holds only the cluster-wide ImageUpdateAutomation +
git write-back. Per-service
ImageRepository/ImagePolicyare rendered from each ServiceDefinition'supdateblock (WS-GITOPS/WS-UPDATE), living withapps/. - keycloak/oauth2-proxy (WS-SEC): the SSO/auth contract (
authblock → Keycloak client / oauth2-proxy protection) is owned by WS-SEC; these are placeholders pending that contract.