jakman85andClaude Opus 5
8a2be824f6
Adds conformance/tests/l0-static/flux-postbuild-substitution/ — an offline, stdlib-only
L0 check tagged R-ENV-02 + R-GITOPS-01 (deliberately NOT R-TEST-01, which would
overclaim). It reuses lib/status.py's `resources:` walk rather than forking it, so "what a
path renders" has exactly one definition. It asserts: every ${...} on a substituted path
is defined-or-escaped (comments included); no default-operator form on an undefined name;
no unparseable expansion; the rendering path actually substitutes cluster-vars; every
overlay-patched /data/KEY has a base default; and that the gate itself is load-bearing.
WIRED AS AN UNCONDITIONAL STAGE (3c) in ci/l0.sh, and that matters: stage 3b scopes to
touched requirements, and ci/touched-requirements.sh maps only clusters/* to R-ENV-02 —
so an edit to a substituted manifest under apps/ or platform/ would have filtered this
check out of exactly the change that introduces the bug. l0 fail -> BLOCK=1 in gate.sh.
Mutation-tested all five assertion classes (undefined token, indexed expansion, :- form,
missing substituteFrom, un-wired gate); each fails as intended, baseline green.
STRICT MODE IS PROVEN SAFE TO ENABLE — measured, not inspected. For all 11 substituted
paths, `kubectl kustomize --load-restrictor LoadRestrictionsNone <path> | flux envsubst
--strict` exits 0 against cluster-vars base defaults. Negative control confirms the
incident: with K3S_API_HOST unset, non-strict renders k8sServiceHost empty while strict
exits 1 with `variable not set (strict mode)`. The feature gate itself is NOT flipped
here — that is a live-cluster change and the operator's call.
Escapes the six comment hazards that stood in the way (apps/vms/unifi x4,
platform/longhorn x2). All were prose ABOUT the escaping convention; since the repo's own
contract is that $${...} emits a literal ${...}, escaping them is both precise and
self-consistent. kustomize strips these particular comments today, but comment survival is
an implementation detail — a body comment carrying $ WAS observed surviving into built
apps/wave-1 — and a survivor would kill the whole Kustomization.
DOCTRINE CORRECTED (measured against flux 2.4.0): clusters/base/apps-vms.yaml claimed an
indexed expansion is fatal "escaped or not". Escaping is TOTAL — $${ARR[0]} emits the
literal text safely; only the UNESCAPED indexed form is fatal. Corrected in place.
Also empties DOCUMENTED_GAPS in conformance/run.sh with rationale: R-ENV-02 and
R-GITOPS-01 now have a real verifying test, which run.sh's own comment named as the
trigger for removing them.
Advisory output (not failures): 6 dead cluster-vars knobs — BASE_DOMAIN, CLUSTER_ENV,
CLUSTER_POD_CIDR, PIA_REGION, PLATFORM_NODE_SELECTOR_{KEY,VALUE}. The last four are
documented reference-only; BASE_DOMAIN and CLUSTER_ENV look like genuine rot (hostnames
are hardcoded elsewhere) and are worth a follow-up.
Gate: MERGE GATE: PASS. 102 assertions, 0 hazards.
Serves: R-ENV-02, R-GITOPS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
125 lines
7.2 KiB
YAML
125 lines
7.2 KiB
YAML
---
|
|
# Default replicated StorageClass — 3 replicas (ADR-0003, R-STATE-02). The durable home for
|
|
# app/config state, databases, and KubeVirt VM disks (via CDI). Destroying a pod/VM leaves the PVC
|
|
# and its data intact; rebuilding re-attaches it (R-STATE-01).
|
|
#
|
|
# This is the cluster's ONLY default StorageClass (the chart's built-in default is disabled in
|
|
# helmrelease.yaml), so PVCs without an explicit storageClassName land here.
|
|
apiVersion: storage.k8s.io/v1
|
|
kind: StorageClass
|
|
metadata:
|
|
name: longhorn
|
|
annotations:
|
|
storageclass.kubernetes.io/is-default-class: "true"
|
|
provisioner: driver.longhorn.io
|
|
allowVolumeExpansion: true
|
|
reclaimPolicy: Delete # dev: reclaim space when the PVC is deleted
|
|
volumeBindingMode: Immediate
|
|
parameters:
|
|
# 3 replicas across 3 nodes (topology-LOCKED). HARD node anti-affinity in helmrelease.yaml
|
|
# (replicaSoftAntiAffinity=false) guarantees these land on distinct nodes, so any single node
|
|
# loss keeps the volume available.
|
|
# HARDCODED, not ${LONGHORN_DEFAULT_REPLICA_COUNT}: StorageClass.parameters is map[string]string,
|
|
# but kustomize strips the quotes around a bare "$${VAR}" placeholder, so Flux postBuild substitution
|
|
# yields an unquoted numeric (3) that server-side apply rejects (int, not string). A quoted numeric
|
|
# literal like "3"/"30" IS preserved by kustomize. The replica knob still drives the GLOBAL default
|
|
# via defaultSettings.defaultReplicaCount in helmrelease.yaml (a Helm int value, unaffected).
|
|
numberOfReplicas: "3"
|
|
staleReplicaTimeout: "30" # minutes before a down replica is considered stale and rebuilt
|
|
dataLocality: "disabled" # keep replicas spread for durability, not co-located with the pod
|
|
fsType: "ext4"
|
|
---
|
|
# KubeVirt VM-disk tier — RWX + migratable, for LIVE-MIGRATABLE VM disks (ADR-0005, docs/06).
|
|
#
|
|
# WHY a second Longhorn class: the default `longhorn` class is ReadWriteOnce (a Longhorn volume
|
|
# attaches to ONE node). KubeVirt live migration needs the SOURCE and TARGET virt-launcher pods to
|
|
# hold the SAME disk on TWO nodes for the migration window, which requires ReadWriteMany. Longhorn
|
|
# provides this via the StorageClass parameter `migratable: "true"`.
|
|
#
|
|
# migratable RWX on Longhorn 1.7.3 is BLOCK-mode only (verified against Longhorn 1.7 docs + the live
|
|
# cluster: longhornio/longhorn-manager:v1.7.3). Longhorn's ORDINARY RWX (migratable=false) is an
|
|
# NFS/share-manager-backed *Filesystem* export; the MIGRATABLE RWX path is different — it is a raw
|
|
# block device that Longhorn allows attached to two nodes ONLY during a live migration (concurrent
|
|
# writers are not permitted, so a mounted filesystem could be corrupted). Therefore:
|
|
# * StorageClass parameter migratable: "true"
|
|
# * PVC/DataVolume accessModes ReadWriteMany
|
|
# * PVC/DataVolume volumeMode Block (Filesystem is NOT supported with migratable)
|
|
# The volumeMode/accessMode live on the PVC/DataVolume; CDI stamps them automatically for disks on
|
|
# this class via the matching StorageProfile claimPropertySets (../kubevirt/cdi-storageprofiles.yaml).
|
|
#
|
|
# is-default-virt-class: CDI treats THIS as the default *virtualization* StorageClass (precedence over
|
|
# the k8s default `longhorn` for VM disks when storageClassName is omitted). Because it supports smart
|
|
# clone (snapshot) AND ReadWriteMany, its StorageProfile is non-degraded — which is what clears the
|
|
# CDIDefaultStorageClassDegraded alert. It is NOT the cluster (k8s) default: `longhorn` keeps the
|
|
# storageclass.kubernetes.io/is-default-class annotation, so ordinary PVCs still land on RWO longhorn.
|
|
# Exactly one SC may carry is-default-virt-class (else CDIMultipleDefaultVirtStorageClasses fires).
|
|
apiVersion: storage.k8s.io/v1
|
|
kind: StorageClass
|
|
metadata:
|
|
name: longhorn-vm
|
|
annotations:
|
|
storageclass.kubevirt.io/is-default-virt-class: "true"
|
|
provisioner: driver.longhorn.io
|
|
allowVolumeExpansion: true
|
|
reclaimPolicy: Delete # dev: reclaim space when the PVC is deleted
|
|
volumeBindingMode: Immediate
|
|
parameters:
|
|
# Same 3-replica durability as the default class (HARD node anti-affinity in helmrelease.yaml).
|
|
# Quoted string literals — see the numberOfReplicas note on the `longhorn` class above.
|
|
numberOfReplicas: "3"
|
|
staleReplicaTimeout: "30"
|
|
dataLocality: "disabled"
|
|
# THE knob that makes this class usable for KubeVirt live migration: mark volumes migratable so a
|
|
# RWX Longhorn volume can be attached to two nodes during a live migration. (Block-mode only.)
|
|
migratable: "true"
|
|
# No fsType: migratable RWX volumes are Block-mode (raw device); a filesystem is never laid down
|
|
# by Longhorn here — the guest owns the block device.
|
|
---
|
|
# CDI scratch tier — SINGLE-replica, for CDI's EPHEMERAL import scratch space ONLY (ADR-0003).
|
|
#
|
|
# WHY a dedicated class: CDI carves transient scratch space (~= the imported disk's virtual size)
|
|
# while converting/decompressing an image, then throws it away when the import finishes. Replicating
|
|
# that 3x is pure waste — and for a large VM disk the 3x scratch (ON TOP of the 3x final disk) can
|
|
# exhaust the Longhorn tier and fail the import with ReplicaSchedulingFailure / "insufficient
|
|
# storage" (observed with the UniFi 60Gi import). One replica is correct for throwaway data: if the
|
|
# node dies mid-import CDI just retries. Referenced by the CDI CR (spec.config.scratchSpaceStorageClass).
|
|
apiVersion: storage.k8s.io/v1
|
|
kind: StorageClass
|
|
metadata:
|
|
name: longhorn-scratch
|
|
provisioner: driver.longhorn.io
|
|
allowVolumeExpansion: true
|
|
reclaimPolicy: Delete # scratch is ephemeral — always reclaim on PVC removal
|
|
volumeBindingMode: Immediate
|
|
parameters:
|
|
numberOfReplicas: "1" # ephemeral throwaway data — no HA replication
|
|
staleReplicaTimeout: "30"
|
|
dataLocality: "best-effort" # keep the lone replica local to the importer pod when possible
|
|
---
|
|
# Bulk media tier — large, NON-replicated data (Plex library, download staging, arr handoff).
|
|
# docs/04 / R-STATE-04: 3x replicating multi-TB media across 3 mini-PCs is impractical, so bulk
|
|
# media lives OUTSIDE the cluster and is consumed as an external NFS/SMB mount via this thin
|
|
# indirection. Services reference storageClassName: bulk; the concrete backing is filled in later
|
|
# WITHOUT touching service definitions.
|
|
#
|
|
# ⚠️ STUB — OQ-2 UNRESOLVED (human operator decision). The provisioner + parameters below are
|
|
# PLACEHOLDERS: they name the standard external NFS CSI driver but the server/share are REPLACE-ME.
|
|
# This class will NOT provision until OQ-2 is answered and the NFS CSI driver is installed. That is
|
|
# intentional — per docs/04 the bulk backing does not block the core platform.
|
|
#
|
|
# Plex's config/metadata DB is small and MUST stay on the 3-replica `longhorn` class regardless
|
|
# (docs/04, ADR-0003) — only the large media files use `bulk`.
|
|
apiVersion: storage.k8s.io/v1
|
|
kind: StorageClass
|
|
metadata:
|
|
name: bulk
|
|
provisioner: nfs.csi.k8s.io # TODO(OQ-2): external NFS/SMB backing — driver + endpoint TBD
|
|
allowVolumeExpansion: true
|
|
reclaimPolicy: Retain # bulk media is precious/reacquirable — never auto-delete on PVC removal
|
|
volumeBindingMode: Immediate
|
|
parameters:
|
|
server: REPLACE-ME # TODO(OQ-2): NAS/NFS host holding the bulk media share
|
|
share: REPLACE-ME # TODO(OQ-2): exported path (e.g. /export/media)
|
|
mountOptions:
|
|
- nfsvers=4.1
|