Autonomous session: Phase-2 platform authoring + Flux bootstrap progress
Author-only Phase-2 components (platform suspended; cluster untouched): - platform/longhorn: 3-replica SC + bulk external-mount indirection (R-STATE) - platform/monitoring: kube-prometheus-stack + Loki + Alloy (dev-trimmed) + 8 alert rules (R-OBS) - platform/cilium: parity fix (k8sServiceHost=10.73.0.11 + masquerade) safe to adopt Phase 2 - conformance: partial L2 tests (worker hit session limit; incomplete — see DECISIONS-LOG K2) - bootstrap/gitea: fix rootless-podman perms (host bind-mount -> named volume) Flux installed on the cluster; host Gitea up + repo pushed; GitRepository BLOCKED by host firewalld (VM->10.73.0.1:3000) — needs one operator sudo command (DECISIONS-LOG). Known issue K1: cluster-vars substitution-var case mismatch (fix before un-suspend). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 4.8 committed
1 parent
397f29e99a
commit
cc89b4b830
19 files changed
+1567
-221
No files matched your search
+59
-16
@@ -4,28 +4,71 @@
|
||||
|
||||
- **Governing ADR:** [ADR-0003 — Distributed storage: Longhorn (+ bulk tier + Velero)](../../../home-network-k3s/decisions/adr-0003-storage-longhorn.md)
|
||||
- **Spec:** [docs/04 — Storage & State](../../../home-network-k3s/docs/04-storage-and-state.md)
|
||||
- **Serves:** R-STATE-01..04
|
||||
- **Serves:** R-STATE-01, R-STATE-02, R-STATE-04 (R-STATE-03 is Velero's, Phase 4)
|
||||
- **Owner:** WS-STORAGE
|
||||
|
||||
## What this stub is
|
||||
## What this is
|
||||
|
||||
Flux `HelmRelease` + two `StorageClass`es:
|
||||
- **`longhorn`** — the **default**, **3-replica** replicated class (replica count is the
|
||||
`${LONGHORN_DEFAULT_REPLICA_COUNT}` overlay knob). Durable home for app/config state and KubeVirt
|
||||
VM disks (via CDI).
|
||||
- **`bulk`** — non-replicated tier for large media (Plex library, download staging). **STUB**: its
|
||||
backing store is **OQ-2 (unresolved)** — escalate before finalizing. Plex's metadata DB stays on
|
||||
the 3-replica class regardless (ADR-0003).
|
||||
A real, dev-appropriate Flux `HelmRelease` for Longhorn plus two `StorageClass`es:
|
||||
|
||||
- **`longhorn`** — the cluster's **only default** StorageClass, **3 replicas** with **hard node
|
||||
anti-affinity** (one replica per node). Durable home for app/config state, databases, and
|
||||
KubeVirt VM disks (via CDI). Replica count is the `${LONGHORN_DEFAULT_REPLICA_COUNT}` overlay
|
||||
knob (cluster-vars, =3 in dev).
|
||||
- **`bulk`** — thin indirection for the **external** (outside-cluster) media tier: Plex library,
|
||||
download staging, arr handoff. **STUB placeholder** modelled as an external NFS mount; its backing
|
||||
is **OQ-2 (unresolved)** and will not provision until answered. Plex's metadata DB stays on the
|
||||
3-replica `longhorn` class regardless (docs/04, ADR-0003).
|
||||
|
||||
## Requirement mapping
|
||||
|
||||
| Requirement | Summary | How this component serves it |
|
||||
|---|---|---|
|
||||
| **R-STATE-01** | Stateful data survives destroy/rebuild of the workload | Volumes are Longhorn PVCs, independent of pod/VM lifecycle; `reclaimPolicy: Delete` on the class ties data to the PVC, not the workload. Destroy the pod/Deployment/VM → PVC + data persist → rebuild re-attaches. (Per-service rebuild test is the Phase-2 exit gate; T-1.5 proves node-level rebuild.) |
|
||||
| **R-STATE-02** | Persistent storage replicated across the 3 nodes | `numberOfReplicas=3` on the default class + `replicaSoftAntiAffinity: false` (hard) → exactly one replica per node, so any single node loss keeps volumes available. Replicas live on the dedicated `/var/lib/longhorn` data disk. `replicaAutoBalance: best-effort` re-spreads on node return. |
|
||||
| **R-STATE-04** | Bulk media has a capacity strategy distinct from the replicated tier | Separate `bulk` StorageClass (NON-replicated, external NFS mount) keeps multi-TB media off the 3x tier; only Plex's small metadata DB uses `longhorn`. |
|
||||
|
||||
## Dev-scoped choices (change for prod)
|
||||
|
||||
- **`reclaimPolicy: Delete`** on `longhorn` — convenient in dev; prod may prefer `Retain` for
|
||||
accidental-delete protection (backups still cover DR — R-STATE-03).
|
||||
- **`storageOverProvisioningPercentage: 100`** — no thin over-provisioning in dev (small disks);
|
||||
prod may raise it.
|
||||
- **`storageReservedPercentageForDefaultDisk: 20` / `storageMinimalAvailablePercentage: 25`** —
|
||||
headroom guardrails tuned for small dev disks; revisit for prod disk sizes.
|
||||
- **Replica count = 3** comes from `cluster-vars.longhorn_default_replica_count`; both dev and prod
|
||||
are 3 today (3 nodes), so the value is identical — the knob exists so it is never forked per env.
|
||||
|
||||
## Guardrails baked in (docs/04)
|
||||
|
||||
- Reserved disk headroom (never fill a node to 100%).
|
||||
- Hard replica anti-affinity across nodes (no two replicas of a volume on one node).
|
||||
- Single-zone anti-affinity relaxed (3 mini-PCs = one failure zone).
|
||||
- Volume/disk health + degraded-replica alerting is WS-OBS's job (docs/10) — not configured here.
|
||||
|
||||
## Node prerequisites (already handled, informational)
|
||||
|
||||
Each node presents a dedicated Longhorn data disk (`/dev/vdb` → ext4) mounted at
|
||||
`/var/lib/longhorn`, and installs `open-iscsi` + `nfs-common` + kernel modules — all via
|
||||
`infra/cloud-init/user-data.yaml.tftpl`. This component assumes that floor.
|
||||
|
||||
## Contract (consumed by everyone with state)
|
||||
|
||||
Per `roadmap/workstreams.md`, WS-STORAGE exposes the `longhorn` (replicated) and `bulk` (media)
|
||||
StorageClasses; WS-APPS/WS-VM/WS-OBS reference them from ServiceDefinitions. Do not place bulk media
|
||||
on the replicated tier or app DBs on the bulk tier.
|
||||
StorageClasses; WS-APPS/WS-VM/WS-OBS reference them from ServiceDefinitions. **Do not place bulk
|
||||
media on the replicated tier, or app DBs on the bulk tier.**
|
||||
|
||||
## TODO (later phases)
|
||||
## Open decisions / TODO (later phases)
|
||||
|
||||
- **OQ-2 (human):** decide the bulk-tier backing (NAS/NFS vs dedicated disk). Blocks `bulk` finalize.
|
||||
- **Phase 2 (WS-STORAGE):** dedicated Longhorn data disk path (docs/02); prove a PVC survives
|
||||
pod destroy/rebuild (R-STATE-01, Phase-2 exit gate).
|
||||
- **Phase 4 (WS-STORAGE):** wire off-cluster backup target with Velero (R-STATE-03).
|
||||
- **OQ-2 (human):** decide the `bulk`-tier backing (NAS/NFS host + share, vs dedicated large disk).
|
||||
Until answered, the `bulk` class is a non-functional placeholder (`server`/`share` = REPLACE-ME).
|
||||
Blocks `bulk` finalize; does **not** block the core platform (docs/04).
|
||||
- **Version pin (WS-STORAGE):** `>=1.7.0 <1.8.0` is a range; pin exactly at bring-up, Renovate
|
||||
maintains.
|
||||
- **Phase 4 (WS-STORAGE):** off-cluster backup target via `defaultSettings.backupTarget` +
|
||||
SOPS-encrypted credential secret, wired alongside `platform/velero/` (R-STATE-03).
|
||||
|
||||
## Build check
|
||||
|
||||
`kubectl kustomize platform/longhorn` renders the namespace, HelmRepository, HelmRelease, and both
|
||||
StorageClasses. (Structural only — not applied to a cluster here; the PM validates + deploys.)
|
||||
@@ -1,5 +1,9 @@
|
||||
---
|
||||
# Longhorn — distributed replicated block storage (ADR-0003). Phase 0 STUB.
|
||||
# Longhorn — distributed replicated block storage across the 3 nodes (ADR-0003).
|
||||
# Serves R-STATE-01 (state survives destroy/rebuild) and R-STATE-02 (3-way replication).
|
||||
#
|
||||
# Dev-appropriate values. Environment differences enter ONLY via ${...} substitution from
|
||||
# cluster-vars (R-ENV-02); everything else is identical base config for dev and prod.
|
||||
apiVersion: helm.toolkit.fluxcd.io/v2
|
||||
kind: HelmRelease
|
||||
metadata:
|
||||
@@ -16,21 +20,60 @@ spec:
|
||||
name: longhorn
|
||||
namespace: longhorn-system
|
||||
install:
|
||||
createNamespace: false
|
||||
createNamespace: false # namespace.yaml owns it (GitOps-managed, labelled)
|
||||
remediation:
|
||||
retries: 3
|
||||
upgrade:
|
||||
remediation:
|
||||
retries: 3
|
||||
values:
|
||||
# We define StorageClasses ourselves (storageclasses.yaml) so the default 3-replica SC and the
|
||||
# bulk tier are explicit and GitOps-managed. Disable the chart's built-in default SC.
|
||||
# We own the StorageClasses (storageclasses.yaml) so the default 3-replica class and the bulk
|
||||
# tier are explicit and GitOps-managed. Disable the chart's built-in default SC so there is
|
||||
# exactly one default-class annotation in the cluster.
|
||||
persistence:
|
||||
defaultClass: false
|
||||
# ${LONGHORN_DEFAULT_REPLICA_COUNT} resolved from cluster-vars (R-ENV-02).
|
||||
defaultClassReplicaCount: ${LONGHORN_DEFAULT_REPLICA_COUNT}
|
||||
|
||||
defaultSettings:
|
||||
# TODO(WS-STORAGE, Phase 2): set defaultDataPath to the dedicated Longhorn data disk
|
||||
# (docs/02 — a dedicated data disk is presented to Longhorn on each node).
|
||||
# --- data path (R-STATE-02) ---
|
||||
# Dedicated Longhorn data disk (/dev/vdb → ext4) is mounted here by cloud-init on every node
|
||||
# (infra/cloud-init/user-data.yaml.tftpl). Longhorn stores replicas on this disk only.
|
||||
defaultDataPath: "/var/lib/longhorn"
|
||||
# TODO(WS-STORAGE, Phase 4): configure off-cluster backup target here (see velero/ + ADR-0003).
|
||||
|
||||
# --- replication (R-STATE-02) ---
|
||||
# Global fallback replica count. The `longhorn` StorageClass sets numberOfReplicas explicitly
|
||||
# (authoritative); this keeps any volume created without an explicit count consistent.
|
||||
# Substituted from cluster-vars.longhorn_default_replica_count (=3 in dev).
|
||||
defaultReplicaCount: ${LONGHORN_DEFAULT_REPLICA_COUNT}
|
||||
|
||||
# --- replica anti-affinity across nodes (R-STATE-02 durability guarantee) ---
|
||||
# HARD anti-affinity: never place two replicas of the same volume on the same node, so a
|
||||
# single node loss cannot take down more than one replica. On a 3-node / 3-replica cluster
|
||||
# this yields exactly one replica per node.
|
||||
replicaSoftAntiAffinity: false
|
||||
# Single zone (3 mini-PCs, no failure zones): allow replicas within the one zone.
|
||||
replicaZoneSoftAntiAffinity: true
|
||||
# When a downed node returns, rebalance replicas back toward one-per-node (best-effort so it
|
||||
# never blocks scheduling).
|
||||
replicaAutoBalance: "best-effort"
|
||||
|
||||
# --- capacity guardrails (docs/04 "reserve disk headroom") ---
|
||||
# Reserve 20% of the dedicated data disk so a node never fills to 100% (which wedges the
|
||||
# kubelet and Longhorn alike). Dev disks are small, so headroom matters.
|
||||
storageReservedPercentageForDefaultDisk: "20"
|
||||
# Stop scheduling new replicas onto a disk once free space drops below 25%.
|
||||
storageMinimalAvailablePercentage: "25"
|
||||
# Dev: no thin over-provisioning — requested capacity must physically fit (1:1).
|
||||
storageOverProvisioningPercentage: "100"
|
||||
|
||||
# --- misc dev-appropriate defaults ---
|
||||
# Keep replicas spread for durability rather than co-locating with the pod.
|
||||
defaultDataLocality: "disabled"
|
||||
# Reclaim replica space of deleted/failed volumes promptly in dev.
|
||||
orphanAutoDeletion: true
|
||||
|
||||
# Longhorn's own manager/UI stay in-cluster; UI exposure (if any) is gated behind
|
||||
# oauth2-proxy forward-auth (ADR-0014) by WS-SEC — not exposed here.
|
||||
|
||||
# TODO(WS-STORAGE, Phase 4): off-cluster backup target (defaultSettings.backupTarget +
|
||||
# backupTargetCredentialSecret) is configured alongside Velero (ADR-0003, R-STATE-03).
|
||||
# Credentials referenced BY NAME only, SOPS-encrypted (WS-SEC) — never inline here.
|
||||
@@ -1,6 +1,10 @@
|
||||
---
|
||||
# Default replicated StorageClass — 3 replicas (ADR-0003). The durable home for app/config state
|
||||
# and KubeVirt VM disks. Replica count is an overlay knob (R-ENV-02).
|
||||
# Default replicated StorageClass — 3 replicas (ADR-0003, R-STATE-02). The durable home for
|
||||
# app/config state, databases, and KubeVirt VM disks (via CDI). Destroying a pod/VM leaves the PVC
|
||||
# and its data intact; rebuilding re-attaches it (R-STATE-01).
|
||||
#
|
||||
# This is the cluster's ONLY default StorageClass (the chart's built-in default is disabled in
|
||||
# helmrelease.yaml), so PVCs without an explicit storageClassName land here.
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
@@ -9,26 +13,40 @@ metadata:
|
||||
storageclass.kubernetes.io/is-default-class: "true"
|
||||
provisioner: driver.longhorn.io
|
||||
allowVolumeExpansion: true
|
||||
reclaimPolicy: Delete
|
||||
reclaimPolicy: Delete # dev: reclaim space when the PVC is deleted
|
||||
volumeBindingMode: Immediate
|
||||
parameters:
|
||||
numberOfReplicas: "${LONGHORN_DEFAULT_REPLICA_COUNT}" # from cluster-vars (default 3)
|
||||
staleReplicaTimeout: "30"
|
||||
# 3 replicas across 3 nodes — substituted from cluster-vars.longhorn_default_replica_count.
|
||||
# HARD node anti-affinity in helmrelease.yaml (replicaSoftAntiAffinity=false) guarantees these
|
||||
# replicas land on distinct nodes, so any single node loss keeps the volume available.
|
||||
numberOfReplicas: "${LONGHORN_DEFAULT_REPLICA_COUNT}"
|
||||
staleReplicaTimeout: "30" # minutes before a down replica is considered stale and rebuilt
|
||||
dataLocality: "disabled" # keep replicas spread for durability, not co-located with the pod
|
||||
fsType: "ext4"
|
||||
---
|
||||
# Bulk media tier — large, NON-replicated data (Plex library, download staging). ADR-0003 / docs/04.
|
||||
# NOTE: backing store (NAS/NFS vs dedicated large disk) is OQ-2, unresolved — this is a STUB.
|
||||
# Plex's metadata DB stays on the 3-replica `longhorn` class regardless (ADR-0003).
|
||||
# Bulk media tier — large, NON-replicated data (Plex library, download staging, arr handoff).
|
||||
# docs/04 / R-STATE-04: 3x replicating multi-TB media across 3 mini-PCs is impractical, so bulk
|
||||
# media lives OUTSIDE the cluster and is consumed as an external NFS/SMB mount via this thin
|
||||
# indirection. Services reference storageClassName: bulk; the concrete backing is filled in later
|
||||
# WITHOUT touching service definitions.
|
||||
#
|
||||
# ⚠️ STUB — OQ-2 UNRESOLVED (human operator decision). The provisioner + parameters below are
|
||||
# PLACEHOLDERS: they name the standard external NFS CSI driver but the server/share are REPLACE-ME.
|
||||
# This class will NOT provision until OQ-2 is answered and the NFS CSI driver is installed. That is
|
||||
# intentional — per docs/04 the bulk backing does not block the core platform.
|
||||
#
|
||||
# Plex's config/metadata DB is small and MUST stay on the 3-replica `longhorn` class regardless
|
||||
# (docs/04, ADR-0003) — only the large media files use `bulk`.
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: bulk
|
||||
provisioner: driver.longhorn.io # TODO(WS-STORAGE): may become an NFS provisioner per OQ-2
|
||||
provisioner: nfs.csi.k8s.io # TODO(OQ-2): external NFS/SMB backing — driver + endpoint TBD
|
||||
allowVolumeExpansion: true
|
||||
reclaimPolicy: Retain
|
||||
reclaimPolicy: Retain # bulk media is precious/reacquirable — never auto-delete on PVC removal
|
||||
volumeBindingMode: Immediate
|
||||
parameters:
|
||||
numberOfReplicas: "1" # non-replicated bulk tier (capacity, not durability)
|
||||
staleReplicaTimeout: "30"
|
||||
fsType: "ext4"
|
||||
# TODO(WS-STORAGE): finalize once OQ-2 (bulk backing) is answered by the human operator.
|
||||
server: REPLACE-ME # TODO(OQ-2): NAS/NFS host holding the bulk media share
|
||||
share: REPLACE-ME # TODO(OQ-2): exported path (e.g. /export/media)
|
||||
mountOptions:
|
||||
- nfsvers=4.1
|
||||
@@ -1,7 +1,21 @@
|
||||
# Longhorn Helm values — PLACEHOLDER (Phase 0). Authoritative values inline in helmrelease.yaml.
|
||||
# Longhorn Helm values — REFERENCE ONLY (not consumed by kustomization.yaml).
|
||||
#
|
||||
# persistence.defaultClass: false # we own StorageClasses (storageclasses.yaml)
|
||||
# persistence.defaultClassReplicaCount: ${LONGHORN_DEFAULT_REPLICA_COUNT} # overlay knob
|
||||
# defaultSettings.defaultDataPath: <data disk> # TODO(WS-STORAGE): dedicated Longhorn disk (docs/02)
|
||||
# defaultSettings.backupTarget: <off-cluster> # TODO(WS-STORAGE, Phase 4): DR target (ADR-0003)
|
||||
# Longhorn UI exposure gated behind oauth2-proxy forward-auth (ADR-0014).
|
||||
# The authoritative values are inline in helmrelease.yaml (matching the sibling pattern across
|
||||
# platform/ — cilium, cert-manager, velero, etc.). This file documents the knobs that matter so a
|
||||
# reviewer can see intent without reading the HelmRelease. Keep it in sync with helmrelease.yaml.
|
||||
#
|
||||
# persistence:
|
||||
# defaultClass: false # we own StorageClasses (storageclasses.yaml)
|
||||
#
|
||||
# defaultSettings:
|
||||
# defaultDataPath: /var/lib/longhorn # dedicated data disk (infra/cloud-init)
|
||||
# defaultReplicaCount: ${LONGHORN_DEFAULT_REPLICA_COUNT} # overlay knob (=3 in dev)
|
||||
# replicaSoftAntiAffinity: false # HARD anti-affinity: one replica per node
|
||||
# replicaZoneSoftAntiAffinity: true # single zone (3 mini-PCs, no failure zones)
|
||||
# replicaAutoBalance: best-effort # rebalance toward one-per-node when a node returns
|
||||
# storageReservedPercentageForDefaultDisk: "20" # reserve disk headroom (docs/04 guardrail)
|
||||
# storageMinimalAvailablePercentage: "25" # stop scheduling below 25% free
|
||||
# storageOverProvisioningPercentage: "100" # dev: no thin over-provisioning (1:1)
|
||||
# defaultDataLocality: disabled # keep replicas spread for durability
|
||||
# orphanAutoDeletion: true # reclaim orphaned replica space promptly
|
||||
# backupTarget: <off-cluster> # TODO(WS-STORAGE, Phase 4): DR target (ADR-0003)
|
||||
Reference in new issue
Block a user