jakman85andClaude Opus 5
05baae90ec
Prod is being stood up now, without waiting for node 2's RAM upgrade. Three coupled
changes, all applied and verified on all three nodes.
=== 1. KERNEL >= 6.13 — now a PREREQUISITE, not a nice-to-have ===
A k3s SERVER that intermittently boots with 2.6 GiB is not a degraded node; it fails under
etcd plus its workload share, unpredictably, on any reboot. Fixed before k3s goes on.
roles/node_baseline gains a `kernel`-tagged block: asserts the suite matches the node's
Debian release, writes a deb822 trixie-backports source, installs, reads back the exact
version, writes a provenance/rollback note to the node. Scoped by
node_baseline_backports_kernel (default FALSE, true only in group_vars/k3s_prod) — chosen
over a hostname test so a fourth prod node inherits it from group membership. Verified:
--check against dev skips all 8 tasks on all 3 hosts. Two deliberate calls: `state: latest`
not `present` (linux-image-amd64 was already installed from main, so `present` would have
silently fixed nothing), and NO preferences.d pin (backports' NotAutomatic +
ButAutomaticUpgrades already gives 100/500; an explicit Pin-Priority: 100 would switch OFF
automatic security upgrades for the very kernel being installed).
Installed 7.1.8+deb13-amd64 on all three; 6.12.107 kept bootable.
THE DECISIVE PROOF: node 2 booted at MEMATTR=0xbeeb0018 — the EXACT address that gave it
2.6 GiB twice — and now reserves 634,904K instead of 13,958,980K. MemTotal 15.4 GiB. Same
address, handled correctly; not a lucky draw.
§0b's claim that a fixed kernel prints 'FW_BUG Corrupted...' is CONDITIONAL and was
corrected: >=6.13 has three rejection gates and only the third prints that line; all three
nodes hit gate 1 (version > 2). Proof instead is boot-independent — decompressing
/boot/vmlinuz on all three yields the FW_BUG and descriptor-size strings verbatim; neither
exists in 6.12.
TWO MORE BUGS IN MY OWN HEALTH CHECK, found because the fix couldn't otherwise be shown:
- MEMATTR_VERSION_RE matched (\d+) but the kernel prints that field with %d — SIGNED. Node
2's real value was -1569980416, which did not match, so state fell through to `clean` and
the node reported PASS "the kernel accepted it without complaint". A FALSE PASS on the
exact bug the check exists to find. Now (-?\d+).
- bad_version was WARN unconditionally, so a FIXED node would have stayed WARN forever.
Now keys on memattr_kernel_fixed: same line, WARN below 6.13, PASS at/above.
=== 2. DISK LAYOUT — Longhorn claimed, per-node paths DISCOVERED not hardcoded ===
Operator decision: reuse the ~424 GB partition; accept the shared-device I/O risk (dev's
etcd/Longhorn contention was on a spinning HDD, prod is SSD/NVMe).
Device paths differ per node (sda5 / nvme0n1p5 / sda5), so the role DISCOVERS the partition
via `findmnt --mountpoint` on a declared claim mountpoint rather than trusting inventory.
Per-host host_vars would have frozen today's hardware into inventory, where it goes stale on
the first re-image and the stale value's failure mode is mkfs on the WRONG partition.
Eleven fail-closed asserts before any write: real block device, not backing / /boot /usr
/var /home /tmp or active swap (built from the node's OWN gathered facts, never a hardcoded
list), already ext4, no foreign label, >= 100 GiB, and empty. Relabel, never reformat.
/srv verified empty first on every node (ls -A -> lost+found only; du 20K; the "5.1% used"
was ext4's default 5% root reserve).
=== 3. k3s DATA-DIR MOVED — /var was 11.8 GiB and would have killed etcd ===
Measured on the fully-deployed dev cluster: containerd 21-25 GB/node, /var total 58-70 GB.
Prod's /var is 11.8 GiB and held BOTH etcd and the image store, so prod would have filled
/var during its first image pulls — and a full /var means etcd cannot write its WAL. Cluster
death, not degradation, arriving during bring-up. Invisible when the disk decision was made,
because "etcd on root" sounded like the 30 GiB / rather than a separate 12 GiB /var.
Now: p5 -> /data (LABEL=k3s-data), with /data/longhorn and /data/k3s. Neutral names chosen
deliberately — the partition now has two tenants, and reusing /srv would make "claimed" and
"unclaimed" indistinguishable to the idempotency pivot, while a bare `data` label would make
a stranger's `data` volume look already-claimed and weaken an assert.
The transition off the existing LABEL=longhorn mount is DECLARED, not inferred
(node_baseline_data_retired_fs_label/_retired_mount, defaults ""), because changing the label
makes a claimed node answer "no" to the pivot and walk into the greenfield path. Self-retiring
once blkid -L k3s-data succeeds. THREE independent gates keep live dev out: retired vars
default ""; gated on longhorn_source == claim-mountpoint (dev is `device`); and the emptiness
assert fails closed on dev's 18 volumes even if both were defeated.
k3s_data_dir is one variable read by two roles (k3s_server --data-dir; node_baseline derives
the etcd mount). Role default is k3s's own default, so dev is unchanged and keeps its
dedicated etcd device. node_baseline_etcd_source becomes `shared-data-partition` and is
LOAD-BEARING: the role asserts per run that the filesystem backing the etcd dir really IS the
data partition — had the mount silently failed, etcd plus 21-25 GB of images would land back
on the OS filesystem invisibly until the disk filled.
LONGHORN_DATA_PATH is a new cluster-vars knob (R-ENV-02): base and DEV /var/lib/longhorn,
prod /data/longhorn.
DEV PROVEN UNCHANGED, which is the property that mattered — the live cluster has 18 Longhorn
volumes on that path. All 12 substituting Flux paths rendered through
`kubectl kustomize | flux envsubst --strict` for both trees: 12/12 per-file SHA-256 identical
and an identical aggregate SHA over every rendered dev byte. With a POSITIVE CONTROL showing
the same harness detects prod changing on exactly one line — an all-identical result is
worthless if the harness cannot see change. Live check after: dev's default-data-path setting
still /var/lib/longhorn, 18 volumes attached, 0 Kustomizations unready.
Idempotent: run 2 is changed=0 on all three nodes. Also fixed a real tag bug — include_tasks
does NOT propagate its own tags to included tasks (unlike import_tasks), so `--tags disks`
selected the include and ran nothing inside it while reporting success.
OPEN DEFECTS, tracked not forgotten: node 2's warm-reboot hang (one clean reboot since, NOT a
fix); and node 3 runs dhcpcd for its wifi interface, which rewrites /etc/resolv.conf on lease
renewal — the role's `contended` detection checks resolvconf and NetworkManager only, so node
3's decided resolvers will silently revert.
Gate: MERGE GATE: PASS. L0 substitution check 106 assertions.
Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-STATE-01, R-STATE-02, R-PLAT-01. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
clusters/
Flux entrypoints and the environment-overlay model (R-ENV-02, R-GITOPS-01/02/03).
Governing spec: docs/05-gitops-and-delivery.md,
docs/02-environments-and-provisioning.md,
ADR-0004 (Flux).
Phase 0 scaffolding only. These are structurally-correct stubs. No cluster exists — nothing here reconciles until WS-INFRA delivers a cluster and bootstraps Flux (roadmap Phase 1).
Layout — base + overlay (never fork the base)
clusters/
base/ shared Flux objects (one authoring, both environments)
flux-system.yaml GitRepository — the reconciliation source (bootstrap → Gitea)
platform.yaml Flux Kustomization → ./platform (reconciles first)
apps-wave-0.yaml Flux Kustomization → ./apps/wave-0 (dependsOn platform-config)
apps-wave-1.yaml Flux Kustomization → ./apps/wave-1 (dependsOn apps-wave-0)
apps-wave-2.yaml Flux Kustomization → ./apps/wave-N (dependsOn apps-wave-N-1)
cluster-vars.yaml ConfigMap of overlay knobs (BASE defaults)
kustomization.yaml
dev/ ACTIVE build target — patches cluster-vars with dev values only
kustomization.yaml
prod/ DESIGN-ONLY until roadmap Phase 7 — patches cluster-vars with prod values only
kustomization.yaml
How the overlay model satisfies R-ENV-02
The base authors the reconciliation graph once. platform.yaml and the app-wave Kustomizations
(apps-wave-0/1/2.yaml — the single apps.yaml was retired; apps-vms.yaml deliberately omits it,
see below) declare postBuild.substituteFrom the cluster-vars ConfigMap. Platform and
app manifests use ${TOKEN} placeholders for every value that legitimately differs between
environments (issuer, replica counts, storage sizes, retention, node placement).
clusters/dev and clusters/prod differ ONLY by the values they patch into cluster-vars —
they add no resources of their own and never copy/fork a base manifest. Promotion to prod
(docs/02) is a change to the
prod overlay's variables, nothing else.
LAN addressing is a knob, not a literal (fixed 2026-08-31)
Until 2026-08-31 the base manifests hardcoded the dev LAN: the k3s API host Cilium dials, the
Cilium LB IP pool range, the pinned crafty/unifi L4 VIPs and the off-cluster S3 backup endpoint. That
broke R-ENV-02 outright — clusters/prod would have rendered dev addresses (192.168.1.x) onto the
prod LAN (192.168.2.x). They are now the cluster-vars "LAN addressing" block
(K3S_API_HOST, CILIUM_LB_POOL_START/_STOP, UNIFI_L4_VIP, CRAFTY_L4_VIP,
S3_BACKUP_ENDPOINT, alongside the pre-existing GATEWAY_VIP), set explicitly by BOTH overlays.
Two rules when touching that block:
- Substitution coverage. A
${TOKEN}only resolves in a path whose Flux Kustomization declarespostBuild.substituteFrom: cluster-vars; anywhere else the literal string${TOKEN}ships to the cluster.apps-vmswas the one path without it and now has it — which means every shell${...}in a VM manifest's cloud-init must be escaped$${...}(seeclusters/base/apps-vms.yamlfor the full envsubst escape contract). GATEWAY_VIPmust be insideCILIUM_LB_POOL_START..STOP, and in practice equal to START: thesharedGateway pins no address, so Cilium LB-IPAM gives it the pool's first free IP while split-horizon CoreDNS answers with${GATEWAY_VIP}. A VIP outside the pool blackholes in-cluster hairpin traffic while the Gateway still looks healthy.
Values the operator has not decided are TBD-PROD-* placeholders, not guesses: prod inherits the
DEV base defaults if an overlay op is simply omitted, so an invalid value that fails loudly is
strictly safer than a plausible one that quietly comes up on the wrong network.
How Flux is pointed at an environment
flux bootstrap (run by WS-INFRA in Phase 1) targets one environment's path per cluster:
# dev cluster
flux bootstrap ... --path=clusters/dev
# prod cluster (Phase 7 only — design-only until then)
flux bootstrap ... --path=clusters/prod
Bootstrap materializes the flux-system Kustomization that renders the chosen overlay, which pulls
in base/ and applies the env-scoped cluster-vars.
Reconciliation order
GitRepository (flux-system)
│
▼
Kustomization: platform (./platform, substituteFrom cluster-vars)
│ healthy
▼
Kustomization: apps-wave-0 (./apps/wave-0, dependsOn platform-config, substituteFrom cluster-vars)
│ healthy
▼
Kustomization: apps-wave-1 (./apps/wave-1, dependsOn apps-wave-0)
│ healthy
▼
Kustomization: apps-wave-N (./apps/wave-N, dependsOn apps-wave-N-1)
Wave 0 dependsOn platform-config (transitively platform) so CNI/storage/TLS/DNS/gateway
exist before any workload reconciles. The app tier is split into dependency-ordered WAVES
(catalog/render.py --emit-waves): each apps-wave-<N> health-gates (wait:true) ONLY its own
wave, so a crashloop in a later wave can no longer stall an earlier wave or block new revisions.
See DESIGN-waves.md for the wave table + cutover runbook.
TODOs (later phases — do not action in Phase 0)
- Source cutover (WS-GITOPS, Phase 6):
base/flux-system.yamlcurrently points at the bootstrap source. After Gitea is onboarded, repointspec.urltogitea.home.keithjackson.devand keep an off-cluster mirror (specdocs/05"Source-of-truth",CONTRIBUTING.md). - SOPS decryption (WS-SEC, Phase 1): add
spec.decryption.provider: sops(age secret name only, never plaintext) to the platform/apps Kustomizations once the age key is bootstrapped (ADR-0007). - Split ordered Flux Kustomizations if intra-platform ordering needs to be explicit (e.g. a
separate
cilium/longhornKustomization othersdependsOn).