jakman85andClaude Opus 5
8a2be824f6
Adds conformance/tests/l0-static/flux-postbuild-substitution/ — an offline, stdlib-only
L0 check tagged R-ENV-02 + R-GITOPS-01 (deliberately NOT R-TEST-01, which would
overclaim). It reuses lib/status.py's `resources:` walk rather than forking it, so "what a
path renders" has exactly one definition. It asserts: every ${...} on a substituted path
is defined-or-escaped (comments included); no default-operator form on an undefined name;
no unparseable expansion; the rendering path actually substitutes cluster-vars; every
overlay-patched /data/KEY has a base default; and that the gate itself is load-bearing.
WIRED AS AN UNCONDITIONAL STAGE (3c) in ci/l0.sh, and that matters: stage 3b scopes to
touched requirements, and ci/touched-requirements.sh maps only clusters/* to R-ENV-02 —
so an edit to a substituted manifest under apps/ or platform/ would have filtered this
check out of exactly the change that introduces the bug. l0 fail -> BLOCK=1 in gate.sh.
Mutation-tested all five assertion classes (undefined token, indexed expansion, :- form,
missing substituteFrom, un-wired gate); each fails as intended, baseline green.
STRICT MODE IS PROVEN SAFE TO ENABLE — measured, not inspected. For all 11 substituted
paths, `kubectl kustomize --load-restrictor LoadRestrictionsNone <path> | flux envsubst
--strict` exits 0 against cluster-vars base defaults. Negative control confirms the
incident: with K3S_API_HOST unset, non-strict renders k8sServiceHost empty while strict
exits 1 with `variable not set (strict mode)`. The feature gate itself is NOT flipped
here — that is a live-cluster change and the operator's call.
Escapes the six comment hazards that stood in the way (apps/vms/unifi x4,
platform/longhorn x2). All were prose ABOUT the escaping convention; since the repo's own
contract is that $${...} emits a literal ${...}, escaping them is both precise and
self-consistent. kustomize strips these particular comments today, but comment survival is
an implementation detail — a body comment carrying $ WAS observed surviving into built
apps/wave-1 — and a survivor would kill the whole Kustomization.
DOCTRINE CORRECTED (measured against flux 2.4.0): clusters/base/apps-vms.yaml claimed an
indexed expansion is fatal "escaped or not". Escaping is TOTAL — $${ARR[0]} emits the
literal text safely; only the UNESCAPED indexed form is fatal. Corrected in place.
Also empties DOCUMENTED_GAPS in conformance/run.sh with rationale: R-ENV-02 and
R-GITOPS-01 now have a real verifying test, which run.sh's own comment named as the
trigger for removing them.
Advisory output (not failures): 6 dead cluster-vars knobs — BASE_DOMAIN, CLUSTER_ENV,
CLUSTER_POD_CIDR, PIA_REGION, PLATFORM_NODE_SELECTOR_{KEY,VALUE}. The last four are
documented reference-only; BASE_DOMAIN and CLUSTER_ENV look like genuine rot (hostnames
are hardcoded elsewhere) and are worth a follow-up.
Gate: MERGE GATE: PASS. 102 assertions, 0 hazards.
Serves: R-ENV-02, R-GITOPS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
platform/longhorn
Longhorn — distributed replicated block storage across the 3 nodes.
- Governing ADR: ADR-0003 — Distributed storage: Longhorn (+ bulk tier + Velero)
- Spec: docs/04 — Storage & State
- Serves: R-STATE-01, R-STATE-02, R-STATE-04 (R-STATE-03 is Velero's, Phase 4)
- Owner: WS-STORAGE
What this is
A real, dev-appropriate Flux HelmRelease for Longhorn plus two StorageClasses:
longhorn— the cluster's only default StorageClass, 3 replicas with hard node anti-affinity (one replica per node). Durable home for app/config state, databases, and KubeVirt VM disks (via CDI). Replica count is the${LONGHORN_DEFAULT_REPLICA_COUNT}overlay knob (cluster-vars, =3 in dev).bulk— thin indirection for the external (outside-cluster) media tier: Plex library, download staging, arr handoff. STUB placeholder modelled as an external NFS mount; its backing is OQ-2 (unresolved) and will not provision until answered. Plex's metadata DB stays on the 3-replicalonghornclass regardless (docs/04, ADR-0003).
Requirement mapping
| Requirement | Summary | How this component serves it |
|---|---|---|
| R-STATE-01 | Stateful data survives destroy/rebuild of the workload | Volumes are Longhorn PVCs, independent of pod/VM lifecycle; reclaimPolicy: Delete on the class ties data to the PVC, not the workload. Destroy the pod/Deployment/VM → PVC + data persist → rebuild re-attaches. (Per-service rebuild test is the Phase-2 exit gate; T-1.5 proves node-level rebuild.) |
| R-STATE-02 | Persistent storage replicated across the 3 nodes | numberOfReplicas=3 on the default class + replicaSoftAntiAffinity: false (hard) → exactly one replica per node, so any single node loss keeps volumes available. Replicas live on the dedicated /var/lib/longhorn data disk. replicaAutoBalance: best-effort re-spreads on node return. |
| R-STATE-04 | Bulk media has a capacity strategy distinct from the replicated tier | Separate bulk StorageClass (NON-replicated, external NFS mount) keeps multi-TB media off the 3x tier; only Plex's small metadata DB uses longhorn. |
Dev-scoped choices (change for prod)
reclaimPolicy: Deleteonlonghorn— convenient in dev; prod may preferRetainfor accidental-delete protection (backups still cover DR — R-STATE-03).storageOverProvisioningPercentage: 100— no thin over-provisioning in dev (small disks); prod may raise it.storageReservedPercentageForDefaultDisk: 20/storageMinimalAvailablePercentage: 25— headroom guardrails tuned for small dev disks; revisit for prod disk sizes.- Replica count = 3 comes from
cluster-vars.longhorn_default_replica_count; both dev and prod are 3 today (3 nodes), so the value is identical — the knob exists so it is never forked per env.
Guardrails baked in (docs/04)
- Reserved disk headroom (never fill a node to 100%).
- Hard replica anti-affinity across nodes (no two replicas of a volume on one node).
- Single-zone anti-affinity relaxed (3 mini-PCs = one failure zone).
- Volume/disk health + degraded-replica alerting is WS-OBS's job (docs/10) — not configured here.
Node prerequisites (already handled, informational)
Each node presents a dedicated Longhorn data disk (/dev/vdb → ext4) mounted at
/var/lib/longhorn, and installs open-iscsi + nfs-common + kernel modules — all via
infra/cloud-init/user-data.yaml.tftpl. This component assumes that floor.
Contract (consumed by everyone with state)
Per roadmap/workstreams.md, WS-STORAGE exposes the longhorn (replicated) and bulk (media)
StorageClasses; WS-APPS/WS-VM/WS-OBS reference them from ServiceDefinitions. Do not place bulk
media on the replicated tier, or app DBs on the bulk tier.
Open decisions / TODO (later phases)
- OQ-2 (human): decide the
bulk-tier backing (NAS/NFS host + share, vs dedicated large disk). Until answered, thebulkclass is a non-functional placeholder (server/share= REPLACE-ME). Blocksbulkfinalize; does not block the core platform (docs/04). - Version pin (WS-STORAGE):
>=1.12.0 <1.13.0is a range; pin exactly at bring-up, Renovate maintains. - Phase 4 (WS-STORAGE): off-cluster backup target via
defaultSettings.backupTarget+ SOPS-encrypted credential secret, wired alongsideplatform/velero/(R-STATE-03).
Build check
kubectl kustomize platform/longhorn renders the namespace, HelmRepository, HelmRelease, and both
StorageClasses. (Structural only — not applied to a cluster here; the PM validates + deploys.)