jakman85andClaude Opus 5
f663e7c801
`runbooks/longhorn-degraded.md` told operators that, after the 2026-09-07 etcd
I/O-starvation incident, `concurrentVolumeBackupRestorePerNodeLimit` and
`snapshotHeavyTaskConcurrentLimit` had been tightened 5 -> 1. Measured live on
prod today: both read **5**, and neither name appeared anywhere in this
repository — so nothing declared them and nothing would have restored them.
Fourth instance of config that looked applied and was not.
Landed in `platform/longhorn/helmrelease.yaml` `defaultSettings` (the path Flux
reconciles; `values.yaml` is the reference mirror), NOT by `kubectl patch` —
a hand-applied Setting is exactly how this gap arose and Helm would silently
rewrite it on its next upgrade.
concurrentVolumeBackupRestorePerNodeLimit 5 -> 1
snapshotHeavyTaskConcurrentLimit 5 -> 1
replicaReplenishmentWaitInterval 600 PINNED, unchanged
backupConcurrentLimit / restoreConcurrentLimit 2 / 2 PINNED, unchanged
concurrentAutomaticEngineUpgradePerNodeLimit 0 PINNED, unchanged
nodeDrainPolicy block-if-contains-last-replica PINNED — it was in
force only as Longhorn's OWN DEFAULT, declared in no manifest and in no
rendered ConfigMap, so an upstream default change would have moved it
silently while two runbooks reasoned about drain safety from it.
EVERY KEY PROVEN TO LAND, not assumed. `helm template longhorn/longhorn
--version 1.12.1` (the version this chart range resolves to, per the
HelmRelease's own lastAttemptedRevision) on this branch's spec.values,
substituted as clusters/prod substitutes them, emits all eight into the rendered
`longhorn-default-setting` ConfigMap in kebab-case. Diffed against the LIVE
ConfigMap the change is purely ADDITIVE: 8 keys added, none altered, none lost.
The chain to the Settings is verified read-only on prod — all 115
settings.longhorn.io carry `longhorn.io/configmap-resource-version: 7917`
(= that ConfigMap's resourceVersion) with `status.applied: true`, and every key
in the ConfigMap equals its Setting's value. The two targets sit at 5 with the
key absent, i.e. Longhorn defaults, as measured.
`orphanAutoDeletion: true` REMOVED, verified inert two independent ways against
the deployed 1.12.1: the render emits no orphan key at all (Longhorn 1.9 renamed
it `orphanResourceAutoDeletion`, a list, and the chart drops unknown
defaultSettings keys with no error), and live, `setting/orphan-auto-deletion` is
NotFound while `orphan-resource-auto-deletion` reads "". Deliberately NOT
replaced — orphan deletion is extra writes on the constrained partition and buys
only space. Orphan cleanup stays manual by decision; the two runbooks that cite
the key by name are corrected rather than left to go stale.
BOTH NEW CAPS ARE 1, AND 1 COSTS SOMETHING — stated, not waved at.
`concurrentVolumeBackupRestorePerNodeLimit: 1` costs nothing TODAY: Longhorn has
no backup target (the `backup-target` Setting does not exist; still the
WS-STORAGE Phase-4 TODO), and velero's nightly run is not a Longhorn backup —
`snapshotVolumes: false` + `defaultVolumesToFsBackup: true` is kopia
file-system backup through velero's node-agent, which this Setting does not
gate; CNPG base backups are barman-level and ungated too. Its only consumer is
an R-STATE-03 restore, where serialising ~24 volumes beats losing etcd
mid-restore. Revisit when a Longhorn backup target lands.
`snapshotHeavyTaskConcurrentLimit: 1` rather than 2 because purge work is
generated by rebuild cleanup (auto-cleanup-system-generated-snapshot true, NO
RecurringJobs, 24 volumes / 24 snapshots) and rebuild is already serialised to
1 per node — so 1 keeps pace with its own generator, where 2 buys backlog drain
at the price of a second heavy writer during the exact rebuild that starved
etcd. The cost: space reclaims serially after a rebuild storm — real here (a
37.7 GiB snapshot once pinned a volume at 97.3% allocated), but instrumented by
LonghornDiskAllocationHigh and recoverable by hand, where a wedged etcd is not.
Headroom on the record: I/O PSI full is 0.30/0.62/0.37 % now, and 3.79 under a
full 18-replica rebuild after node 1's drive replacement, against a 5% warning
threshold. These caps are a BOUND, not a fix — etcd and Longhorn still share one
ext4 filesystem and nothing arbitrates the jbd2 journal fdatasync waits on.
NOT PORTED, deliberately: the same unmerged branch's node-level I/O arbitration
(`roles/etcd_io_priority` — BFQ on the shared device, nr_requests 64 -> 32,
systemd IOWeight drop-ins). The 2011 C400 it was written for was replaced on
2026-09-07 and the same load class measured 11.36 -> 3.79 I/O PSI with the node
Ready throughout; the role was never run against any cluster, its author
recorded it as unproven with a node-3-first sequenced rollout, and its own
README concedes it does not separate the two tenants. Node-wide blast radius,
low ceiling, stale measurements (node 1 reinstalled, founder moved to node 2).
It stays an operator option if I/O PSI climbs back toward 5% — which, unlike in
September, we would now be told about. Nothing under infra/ansible/ is touched.
ALSO ALREADY DONE, so skipped: that branch's NodeIOPressureHigh/Critical and the
dashboard I/O row are on main already (efc6db6) with runbooks/node-io.md.
Verified live, not by reading the file: both rules loaded in
platform-health.nodes, health ok, state inactive, and
ALERTS{alertname=~"NodeIOPressure.*"} returns no samples.
Runbooks corrected in both directions: longhorn-degraded.md's prominent "DO NOT
RELY ON THAT: it was never applied" block becomes an accurate statement of what
is now declared, what to verify before trusting it, what it cannot do, and what
it costs; longhorn-capacity.md / node-disk.md stop citing a key that no longer
exists; node-io.md gains the one line an operator needs mid-incident.
No new ${...} substitution variables and not one dollar sign added anywhere in
the diff, so neither StrictPostBuildSubstitutions nor the L0 doubled-dollar guard
is affected.
Co-Authored-By: Claude Opus 5 <[email protected]>
platform/
Cluster-wide platform services, Flux-managed (R-PLAT-01, R-GITOPS-01/02/03). Reconciled
by the platform Flux Kustomization defined in clusters/base/platform.yaml,
before apps/.
Governing spec: docs/05-gitops-and-delivery.md,
docs/01-architecture-overview.md.
Phase 0 scaffolding only. Every component here is a structurally-correct stub — a Flux HelmRelease and/or Kustomization with a values placeholder, a README citing the governing ADR/spec, and TODOs for the phase that fills it in. No cluster exists; nothing reconciles yet. Do not add working config, secrets, or provisioning under Phase 0 (spec
AGENTS.md§1/§5/§6,roadmap/phases.md).
Reconciliation flow (how a change lands)
git (source of truth)
│ operator PR → CI (lint · schema · Kyverno L0 · Chainsaw) → merge to tracked branch
▼
Flux GitRepository (clusters/base/flux-system.yaml)
▼
Flux Kustomization "platform" → ./platform (this dir) ── substituteFrom cluster-vars ──┐
│ prune=true (drift correction, R-GITOPS-03) · wait=true │ R-ENV-02
▼ │
Flux Kustomization "apps" → ./apps (dependsOn platform) ─ substituteFrom cluster-vars ──┘
Git is authoritative; the cluster conforms to git and drift is auto-corrected (R-GITOPS-01/03).
Manual kubectl apply of platform state is prohibited (spec AGENTS.md §5).
The dev ↔ prod overlay model (R-ENV-02)
Platform manifests are authored once. Every value that legitimately differs between environments
is a ${TOKEN} resolved from the cluster-vars ConfigMap via Flux postBuild.substituteFrom. The
clusters/dev and clusters/prod overlays differ only by the values they patch into
cluster-vars (issuer, replica counts, storage sizes, retention). See
clusters/README.md. Base manifests are never forked per environment.
Components (one directory each)
| Dir | Component | Governing ADR / spec | Serves |
|---|---|---|---|
cilium/ |
CNI + kube-proxy replacement + Gateway API + Egress Gateway + Hubble | ADR-0002, docs/03 | R-NET-* |
longhorn/ |
Replicated block storage; 3-replica default StorageClass + bulk tier | ADR-0003, docs/04 | R-STATE-* |
cert-manager/ |
TLS via Let's Encrypt DNS-01 (Cloudflare); staging + prod ClusterIssuers | ADR-0006, docs/08 | R-TLS-* |
externaldns/ |
Automatic DNS records for public services (Cloudflare) | docs/09 | R-DNS-* |
gateway/ |
Gateway API GatewayClass + shared HTTPS Gateway + wildcard cert | ADR-0002/ADR-0006, docs/09 | R-DNS-01, R-TLS-01 |
monitoring/ |
kube-prometheus-stack + Loki + Grafana Alloy + Alertmanager | docs/10 | R-OBS-* |
velero/ |
Off-cluster backup/DR of volumes + resource state | ADR-0003, docs/04 | R-STATE-03 |
kubevirt/ |
KubeVirt + CDI (IaaS: application VMs on the same K8s surface) | ADR-0005, docs/06 | R-PLAT-03 |
image-automation/ |
Flux image-update-automation (hands-off image tag advance) | ADR-0009, docs/11 | R-UPD-02 |
keycloak/ |
Platform IdP (OIDC + SAML), config-as-code, PostgreSQL on Longhorn | ADR-0014, docs/15 | R-IDP-* |
oauth2-proxy/ |
Forward-auth at the Gateway for non-OIDC apps → Keycloak | ADR-0014, docs/15 | R-IDP-* |
kyverno/ |
L0 policy-as-code gate — OWNED BY THE POLICY ENGINEER (WS-TEST). | ADR-0011, docs/14 | R-TEST-* |
⚠️ INTEGRATION TODO(PM): wire kyverno into the platform kustomization
platform/kyverno/ is authored by a different agent (the policy engineer, WS-TEST). This
workstream did not create any files under platform/kyverno/ and must not.
platform/kustomization.yaml intentionally omits - kyverno. Once the
policy engineer's files land, the PM must add - kyverno to that list (recommended first, so
policies are present before other components reconcile). The line is present as a commented TODO in
the kustomization.
Secrets
Every secret referenced below (Cloudflare API token, Velero backup-target creds, Keycloak admin/DB, oauth2-proxy cookie/client secrets) is referenced by name only and will be committed SOPS-encrypted (WS-SEC, ADR-0007). No plaintext secret appears anywhere in this tree.
Cross-workstream contracts surfaced by these stubs
- cilium (WS-NET ↔ WS-INFRA): Cilium's initial CNI install is done by Ansible at cluster bootstrap (docs/02, Phase 1). The HelmRelease here manages Cilium config thereafter; its Helm values MUST match the Ansible bootstrap values (kubeProxyReplacement, k8sServiceHost/Port, API server host) or reconciliation will fight the bootstrap. Contract owner: WS-NET + WS-INFRA.
- gateway + cert-manager + externaldns (WS-NET + WS-SEC): the shared Gateway listener, the
wildcard
*.home.keithjackson.devcert, and the ExternalDNS source all reference each other; they form the "Issuer + Gateway TLS" interface inroadmap/workstreams.md. - image-automation (WS-UPDATE): this dir holds only the cluster-wide ImageUpdateAutomation +
git write-back. Per-service
ImageRepository/ImagePolicyare rendered from each ServiceDefinition'supdateblock (WS-GITOPS/WS-UPDATE), living withapps/. - keycloak/oauth2-proxy (WS-SEC): the SSO/auth contract (
authblock → Keycloak client / oauth2-proxy protection) is owned by WS-SEC; these are placeholders pending that contract.