jakman85andClaude Opus 4.8
cc89b4b830
Author-only Phase-2 components (platform suspended; cluster untouched): - platform/longhorn: 3-replica SC + bulk external-mount indirection (R-STATE) - platform/monitoring: kube-prometheus-stack + Loki + Alloy (dev-trimmed) + 8 alert rules (R-OBS) - platform/cilium: parity fix (k8sServiceHost=10.73.0.11 + masquerade) safe to adopt Phase 2 - conformance: partial L2 tests (worker hit session limit; incomplete — see DECISIONS-LOG K2) - bootstrap/gitea: fix rootless-podman perms (host bind-mount -> named volume) Flux installed on the cluster; host Gitea up + repo pushed; GitRepository BLOCKED by host firewalld (VM->10.73.0.1:3000) — needs one operator sudo command (DECISIONS-LOG). Known issue K1: cluster-vars substitution-var case mismatch (fix before un-suspend). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
75 lines
4.7 KiB
Markdown
75 lines
4.7 KiB
Markdown
# platform/longhorn
|
|
|
|
**Longhorn** — distributed replicated block storage across the 3 nodes.
|
|
|
|
- **Governing ADR:** [ADR-0003 — Distributed storage: Longhorn (+ bulk tier + Velero)](../../../home-network-k3s/decisions/adr-0003-storage-longhorn.md)
|
|
- **Spec:** [docs/04 — Storage & State](../../../home-network-k3s/docs/04-storage-and-state.md)
|
|
- **Serves:** R-STATE-01, R-STATE-02, R-STATE-04 (R-STATE-03 is Velero's, Phase 4)
|
|
- **Owner:** WS-STORAGE
|
|
|
|
## What this is
|
|
|
|
A real, dev-appropriate Flux `HelmRelease` for Longhorn plus two `StorageClass`es:
|
|
|
|
- **`longhorn`** — the cluster's **only default** StorageClass, **3 replicas** with **hard node
|
|
anti-affinity** (one replica per node). Durable home for app/config state, databases, and
|
|
KubeVirt VM disks (via CDI). Replica count is the `${LONGHORN_DEFAULT_REPLICA_COUNT}` overlay
|
|
knob (cluster-vars, =3 in dev).
|
|
- **`bulk`** — thin indirection for the **external** (outside-cluster) media tier: Plex library,
|
|
download staging, arr handoff. **STUB placeholder** modelled as an external NFS mount; its backing
|
|
is **OQ-2 (unresolved)** and will not provision until answered. Plex's metadata DB stays on the
|
|
3-replica `longhorn` class regardless (docs/04, ADR-0003).
|
|
|
|
## Requirement mapping
|
|
|
|
| Requirement | Summary | How this component serves it |
|
|
|---|---|---|
|
|
| **R-STATE-01** | Stateful data survives destroy/rebuild of the workload | Volumes are Longhorn PVCs, independent of pod/VM lifecycle; `reclaimPolicy: Delete` on the class ties data to the PVC, not the workload. Destroy the pod/Deployment/VM → PVC + data persist → rebuild re-attaches. (Per-service rebuild test is the Phase-2 exit gate; T-1.5 proves node-level rebuild.) |
|
|
| **R-STATE-02** | Persistent storage replicated across the 3 nodes | `numberOfReplicas=3` on the default class + `replicaSoftAntiAffinity: false` (hard) → exactly one replica per node, so any single node loss keeps volumes available. Replicas live on the dedicated `/var/lib/longhorn` data disk. `replicaAutoBalance: best-effort` re-spreads on node return. |
|
|
| **R-STATE-04** | Bulk media has a capacity strategy distinct from the replicated tier | Separate `bulk` StorageClass (NON-replicated, external NFS mount) keeps multi-TB media off the 3x tier; only Plex's small metadata DB uses `longhorn`. |
|
|
|
|
## Dev-scoped choices (change for prod)
|
|
|
|
- **`reclaimPolicy: Delete`** on `longhorn` — convenient in dev; prod may prefer `Retain` for
|
|
accidental-delete protection (backups still cover DR — R-STATE-03).
|
|
- **`storageOverProvisioningPercentage: 100`** — no thin over-provisioning in dev (small disks);
|
|
prod may raise it.
|
|
- **`storageReservedPercentageForDefaultDisk: 20` / `storageMinimalAvailablePercentage: 25`** —
|
|
headroom guardrails tuned for small dev disks; revisit for prod disk sizes.
|
|
- **Replica count = 3** comes from `cluster-vars.longhorn_default_replica_count`; both dev and prod
|
|
are 3 today (3 nodes), so the value is identical — the knob exists so it is never forked per env.
|
|
|
|
## Guardrails baked in (docs/04)
|
|
|
|
- Reserved disk headroom (never fill a node to 100%).
|
|
- Hard replica anti-affinity across nodes (no two replicas of a volume on one node).
|
|
- Single-zone anti-affinity relaxed (3 mini-PCs = one failure zone).
|
|
- Volume/disk health + degraded-replica alerting is WS-OBS's job (docs/10) — not configured here.
|
|
|
|
## Node prerequisites (already handled, informational)
|
|
|
|
Each node presents a dedicated Longhorn data disk (`/dev/vdb` → ext4) mounted at
|
|
`/var/lib/longhorn`, and installs `open-iscsi` + `nfs-common` + kernel modules — all via
|
|
`infra/cloud-init/user-data.yaml.tftpl`. This component assumes that floor.
|
|
|
|
## Contract (consumed by everyone with state)
|
|
|
|
Per `roadmap/workstreams.md`, WS-STORAGE exposes the `longhorn` (replicated) and `bulk` (media)
|
|
StorageClasses; WS-APPS/WS-VM/WS-OBS reference them from ServiceDefinitions. **Do not place bulk
|
|
media on the replicated tier, or app DBs on the bulk tier.**
|
|
|
|
## Open decisions / TODO (later phases)
|
|
|
|
- **OQ-2 (human):** decide the `bulk`-tier backing (NAS/NFS host + share, vs dedicated large disk).
|
|
Until answered, the `bulk` class is a non-functional placeholder (`server`/`share` = REPLACE-ME).
|
|
Blocks `bulk` finalize; does **not** block the core platform (docs/04).
|
|
- **Version pin (WS-STORAGE):** `>=1.7.0 <1.8.0` is a range; pin exactly at bring-up, Renovate
|
|
maintains.
|
|
- **Phase 4 (WS-STORAGE):** off-cluster backup target via `defaultSettings.backupTarget` +
|
|
SOPS-encrypted credential secret, wired alongside `platform/velero/` (R-STATE-03).
|
|
|
|
## Build check
|
|
|
|
`kubectl kustomize platform/longhorn` renders the namespace, HelmRepository, HelmRelease, and both
|
|
StorageClasses. (Structural only — not applied to a cluster here; the PM validates + deploys.)
|