Autonomous session: Phase-2 platform authoring + Flux bootstrap progress

Author-only Phase-2 components (platform suspended; cluster untouched):
- platform/longhorn: 3-replica SC + bulk external-mount indirection (R-STATE)
- platform/monitoring: kube-prometheus-stack + Loki + Alloy (dev-trimmed) + 8 alert rules (R-OBS)
- platform/cilium: parity fix (k8sServiceHost=10.73.0.11 + masquerade) safe to adopt Phase 2
- conformance: partial L2 tests (worker hit session limit; incomplete — see DECISIONS-LOG K2)
- bootstrap/gitea: fix rootless-podman perms (host bind-mount -> named volume)

Flux installed on the cluster; host Gitea up + repo pushed; GitRepository BLOCKED by
host firewalld (VM->10.73.0.1:3000) — needs one operator sudo command (DECISIONS-LOG).
Known issue K1: cluster-vars substitution-var case mismatch (fix before un-suspend).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

platform/longhorn/README.md

@@ -4,28 +4,71 @@
 
 - **Governing ADR:** [ADR-0003 — Distributed storage: Longhorn (+ bulk tier + Velero)](../../../home-network-k3s/decisions/adr-0003-storage-longhorn.md)
 - **Spec:** [docs/04 — Storage & State](../../../home-network-k3s/docs/04-storage-and-state.md)
-- **Serves:** R-STATE-01..04
+- **Serves:** R-STATE-01, R-STATE-02, R-STATE-04 (R-STATE-03 is Velero's, Phase 4)
 - **Owner:** WS-STORAGE
 
-## What this stub is
+## What this is
 
-Flux `HelmRelease` + two `StorageClass`es:
-- **`longhorn`** — the **default**, **3-replica** replicated class (replica count is the
-  `${LONGHORN_DEFAULT_REPLICA_COUNT}` overlay knob). Durable home for app/config state and KubeVirt
-  VM disks (via CDI).
-- **`bulk`** — non-replicated tier for large media (Plex library, download staging). **STUB**: its
-  backing store is **OQ-2 (unresolved)** — escalate before finalizing. Plex's metadata DB stays on
-  the 3-replica class regardless (ADR-0003).
+A real, dev-appropriate Flux `HelmRelease` for Longhorn plus two `StorageClass`es:
+
+- **`longhorn`** — the cluster's **only default** StorageClass, **3 replicas** with **hard node
+  anti-affinity** (one replica per node). Durable home for app/config state, databases, and
+  KubeVirt VM disks (via CDI). Replica count is the `${LONGHORN_DEFAULT_REPLICA_COUNT}` overlay
+  knob (cluster-vars, =3 in dev).
+- **`bulk`** — thin indirection for the **external** (outside-cluster) media tier: Plex library,
+  download staging, arr handoff. **STUB placeholder** modelled as an external NFS mount; its backing
+  is **OQ-2 (unresolved)** and will not provision until answered. Plex's metadata DB stays on the
+  3-replica `longhorn` class regardless (docs/04, ADR-0003).
+
+## Requirement mapping
+
+| Requirement | Summary | How this component serves it |
+|---|---|---|
+| **R-STATE-01** | Stateful data survives destroy/rebuild of the workload | Volumes are Longhorn PVCs, independent of pod/VM lifecycle; `reclaimPolicy: Delete` on the class ties data to the PVC, not the workload. Destroy the pod/Deployment/VM → PVC + data persist → rebuild re-attaches. (Per-service rebuild test is the Phase-2 exit gate; T-1.5 proves node-level rebuild.) |
+| **R-STATE-02** | Persistent storage replicated across the 3 nodes | `numberOfReplicas=3` on the default class + `replicaSoftAntiAffinity: false` (hard) → exactly one replica per node, so any single node loss keeps volumes available. Replicas live on the dedicated `/var/lib/longhorn` data disk. `replicaAutoBalance: best-effort` re-spreads on node return. |
+| **R-STATE-04** | Bulk media has a capacity strategy distinct from the replicated tier | Separate `bulk` StorageClass (NON-replicated, external NFS mount) keeps multi-TB media off the 3x tier; only Plex's small metadata DB uses `longhorn`. |
+
+## Dev-scoped choices (change for prod)
+
+- **`reclaimPolicy: Delete`** on `longhorn` — convenient in dev; prod may prefer `Retain` for
+  accidental-delete protection (backups still cover DR — R-STATE-03).
+- **`storageOverProvisioningPercentage: 100`** — no thin over-provisioning in dev (small disks);
+  prod may raise it.
+- **`storageReservedPercentageForDefaultDisk: 20` / `storageMinimalAvailablePercentage: 25`** —
+  headroom guardrails tuned for small dev disks; revisit for prod disk sizes.
+- **Replica count = 3** comes from `cluster-vars.longhorn_default_replica_count`; both dev and prod
+  are 3 today (3 nodes), so the value is identical — the knob exists so it is never forked per env.
+
+## Guardrails baked in (docs/04)
+
+- Reserved disk headroom (never fill a node to 100%).
+- Hard replica anti-affinity across nodes (no two replicas of a volume on one node).
+- Single-zone anti-affinity relaxed (3 mini-PCs = one failure zone).
+- Volume/disk health + degraded-replica alerting is WS-OBS's job (docs/10) — not configured here.
+
+## Node prerequisites (already handled, informational)
+
+Each node presents a dedicated Longhorn data disk (`/dev/vdb` → ext4) mounted at
+`/var/lib/longhorn`, and installs `open-iscsi` + `nfs-common` + kernel modules — all via
+`infra/cloud-init/user-data.yaml.tftpl`. This component assumes that floor.
 
 ## Contract (consumed by everyone with state)
 
 Per `roadmap/workstreams.md`, WS-STORAGE exposes the `longhorn` (replicated) and `bulk` (media)
-StorageClasses; WS-APPS/WS-VM/WS-OBS reference them from ServiceDefinitions. Do not place bulk media
-on the replicated tier or app DBs on the bulk tier.
+StorageClasses; WS-APPS/WS-VM/WS-OBS reference them from ServiceDefinitions. **Do not place bulk
+media on the replicated tier, or app DBs on the bulk tier.**
 
-## TODO (later phases)
+## Open decisions / TODO (later phases)
 
-- **OQ-2 (human):** decide the bulk-tier backing (NAS/NFS vs dedicated disk). Blocks `bulk` finalize.
-- **Phase 2 (WS-STORAGE):** dedicated Longhorn data disk path (docs/02); prove a PVC survives
-  pod destroy/rebuild (R-STATE-01, Phase-2 exit gate).
-- **Phase 4 (WS-STORAGE):** wire off-cluster backup target with Velero (R-STATE-03).
+- **OQ-2 (human):** decide the `bulk`-tier backing (NAS/NFS host + share, vs dedicated large disk).
+  Until answered, the `bulk` class is a non-functional placeholder (`server`/`share` = REPLACE-ME).
+  Blocks `bulk` finalize; does **not** block the core platform (docs/04).
+- **Version pin (WS-STORAGE):** `>=1.7.0 <1.8.0` is a range; pin exactly at bring-up, Renovate
+  maintains.
+- **Phase 4 (WS-STORAGE):** off-cluster backup target via `defaultSettings.backupTarget` +
+  SOPS-encrypted credential secret, wired alongside `platform/velero/` (R-STATE-03).
+
+## Build check
+
+`kubectl kustomize platform/longhorn` renders the namespace, HelmRepository, HelmRelease, and both
+StorageClasses. (Structural only — not applied to a cluster here; the PM validates + deploys.)