feat(tofu,ansible): SSD-backed etcd disk per k3s node (Phase 1: code + plan)

Interim bridge to stop Longhorn replica-rebuild I/O storms on the shared HDD
Storage pool from starving etcd WAL fsync (crash-looping k3s / flapping the
kube-vip VIP). Adds a small dedicated SSD disk per node for etcd only; Longhorn
stays on HDD. See docs/plans/ssd-etcd-isolation.md.

tofu (infra/tofu):
- New vars etcd_ssd_datastore (default "local-zfs" = rpool/data, Micron 5100
  SATA SSD mirror w/ PLP) and etcd_disk_gib (default 16).
- New virtio2 disk block on the SSD datastore per VM: discard=on, cache=none
  (honor guest fsync straight to the ZFS ZIL — safe on PLP SSD), iothread=false.
  Kept last (ascending interface) so bpg does not reorder virtio0/virtio1.
- Revert the debian_cloud_image resource to the long type name
  proxmox_virtual_environment_download_file to match the R2 state: bpg >= 0.111
  dropped the cross-type moved{} linkage to the short proxmox_download_file, so
  the stale moved block failed EVERY plan with "Resource type mismatch" (broken
  on main too). Aligning code to state yields a clean plan with no image
  re-download. Short-name migration is deferred to a deliberate `tofu state mv`.

ansible (node_baseline):
- Format /dev/vdc (label k3s-etcd) and mount it at
  /var/lib/rancher/k3s/server/db (defaults,noatime; no nofail so a missing etcd
  disk fails loudly rather than silently landing etcd on the root HDD) before
  k3s starts, so a rebuilt node comes up with etcd isolated on SSD. Data-safe:
  filesystem module never reformats an existing fs.

Phase 1 only: non-disruptive. No tofu apply, no node/VM/etcd/Longhorn changes.
tofu plan verified: 0 to add, 3 to change, 0 to destroy — every VM an in-place
disk ADD (virtio2), dev-3 also an in-place memory 16384->24576; no VM replace.
Merge gate green (render==golden included).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 4.8 committed 2026-08-29 17:21:06 -05:00
1 parent 7910128675
commit af5fc5b25f
5 files changed
+348 -17

No files matched your search

+270
View File
@@ -0,0 +1,270 @@
# SSD-backed etcd isolation for the k3s dev cluster
**Status:** PROPOSAL — awaiting user approval. No changes have been made.
**Author:** investigation agent (read-only), 2026-08-29
**Problem:** Longhorn replica-rebuild I/O storms on the HDD-backed VM disks starve
etcd's WAL fsync, crash-looping k3s and flapping the kube-vip VIP
(`192.168.2.50:6443`). Even `concurrent-replica-rebuild-per-node-limit=1` tips etcd.
**Question:** Can the Proxmox host's ~1 TB system SSD (`rpool`) host dedicated
SSD-backed storage for etcd (and/or Longhorn) to end these incidents?
**Verdict: YES for etcd (strongly recommended). NO for Longhorn.**
---
## 1. Measured facts (all collected read-only on 2026-08-29)
### 1.1 `rpool` — the system/boot SSD pool
| Property | Value |
|---|---|
| Pool | `rpool`, **mirror** (`mirror-0`) of 2 devices |
| Devices | `ata-MTFDDAK960TDS_...` × 2 — **Micron 5100/52x0 960 GB, 2.5" SATA** |
| Class | **Enterprise SATA SSD with power-loss protection (PLP)** — safe for sync/fsync |
| Health | `Percent_Lifetime_Remain` = **97%** (3% used), 13 904 power-on hours, 0 reallocated sectors, ~96 TB written. Excellent. |
| Pool size / alloc / **free** | 888 G / 538 G / **350 G free** (`zpool list`, 60% cap) |
| ZFS `AVAIL` | **~323 GiB** available to any new dataset/zvol |
| System root | `rpool/ROOT/pve-1` = **430 G used** (Proxmox host `/`) |
| Proxmox storage on it | `local-zfs` → pool `rpool/data`, `content images,rootdir`, **`sparse 1`** (thin), ~338 GiB available per `pvesm status` |
| Pool settings | `compression on`, **`sync standard`** (inherited), `dedup off`, recordsize 128K |
`local-zfs` already advertises `images,rootdir` content, so **Proxmox can place VM
disks on the SSD today with no storage-config change.**
### 1.2 Current VM disk backing — confirms the HDD assumption
VMs (bpg/OpenTofu-provisioned): **dev-1 = vmid 105, dev-2 = 102, dev-3 = 101**.
Each has two disks, **both on the `Storage` datastore**:
- `Storage` = ZFS pool on **2× WDC WUH721818ALE6L4 18 TB HDD** (`sdc`/`sdd`, `ROTA=1`), mirror, 16.4 T.
- Per node: `virtio0 = vm-<id>-disk-0` (64 G, OS/root) and `virtio1 = vm-<id>-disk-1` (128 G, Longhorn data). `volblocksize 16K`.
So **etcd, the OS, and Longhorn all currently sit on spinning rust.** That is the
root cause: etcd WAL fsync competes with Longhorn rebuild I/O on the same HDD spindles.
(`Storage2` = 2× Seagate ST8000VN004 8 TB HDD is a separate HDD pool, not involved.)
### 1.3 Size of the hot data
- **etcd DB: ~82.6 MB.** Measured via apiserver metric
`apiserver_storage_size_bytes{storage_cluster_id="etcd-0"} = 8.264e7`.
A 4-day-old cluster running ~18 services. Even allowing generous WAL + growth
headroom, **etcd fits in single-digit GB with room to spare — trivially ×3.**
- **Longhorn: ~22 GB actual data per node right now**, but the data disk is
**128 GiB** and Longhorn reports `storageScheduled ≈ 152 GiB` (thin-overcommitted)
vs `storageMaximum ≈ 125 GiB` per node. 18 PVCs, `numberOfReplicas=3`.
Provisioned total ≈ 152 GiB. Longhorn is the **churn source we want to isolate
etcd from**, and it can grow toward 128 GiB/node.
### 1.4 Sizing conclusion
| Candidate | Per-node footprint | ×3 on rpool | Fits in 323 GiB avail? |
|---|---|---|---|
| **etcd** | ~83 MB now; size a 16 GiB disk | ~48 GiB reserved (thin) | **Yes, easily** |
| **Longhorn** | ~22 GB now, up to 128 GiB provisioned | up to **~384 GiB** | **No — would fill/threaten the boot disk** |
---
## 2. Viability verdict
**etcd on SSD: viable and high-leverage.** etcd is ~83 MB; three 16 GiB SSD-backed
disks cost ~48 GiB of thin allocation (actual ~250 MB) against 323 GiB free on an
enterprise, PLP-equipped, 97%-healthy SSD mirror. This directly removes the WAL
fsync path from the HDD contention domain — the smallest-footprint fix for the exact
failure mode. **This is the recommendation.**
**Longhorn on SSD: not viable here.** Its provisioned footprint (~384 GiB across 3
replicas at full size) does not fit alongside the 430 GiB Proxmox root on a 888 GiB
pool, and co-locating heavy Longhorn rebuild I/O on the boot SSD reintroduces
contention with the host itself. Keep Longhorn on HDD (`Storage`). Rejected.
---
## 3. Options evaluated
- **Option A — dedicated small SSD disk per node, etcd-only (RECOMMENDED).**
Add a ~16 GiB SSD-backed virtual disk (zvol on `rpool/data` via `local-zfs`) to
each of the 3 VMs, mount it at the k3s etcd directory
(`/var/lib/rancher/k3s/server/db`), migrate etcd data onto it one node at a time.
Isolates WAL fsync from HDD Longhorn churn. Minimal space, minimal blast radius.
- **Option B — Longhorn replicas/disk on SSD.** Rejected (§2): does not fit and
re-couples host I/O.
- **Option C — do nothing; wait for the SSD mini-PCs.** Zero risk, zero work, but
leaves the cluster fragile to every node bounce until the mini-PC migration lands
(deferred, no committed date). Option A is a cheap, reversible bridge that Option C
eventually supersedes (§7).
**Recommendation: Option A now, as an interim fix, carried by tofu.**
---
## 4. Recommended implementation — Option A, step-by-step
### 4.1 Model it in tofu (preferred over manual)
The VMs are bpg/OpenTofu-managed (`infra/tofu/`). Disks are declared as `disk`
blocks in `main.tf`; both current disks (`virtio0`, `virtio1`) point at
`var.datastore` (`Storage`). Add a **third disk on the SSD datastore**:
1. **New variables** in `infra/tofu/variables.tf`:
```hcl
variable "etcd_ssd_datastore" {
description = "SSD-backed datastore for the dedicated etcd disk (VERIFIED: local-zfs = rpool/data, ZFS mirror on Micron 5100 SATA SSDs w/ PLP)."
type = string
default = "local-zfs"
}
variable "etcd_disk_gib" {
description = "Dedicated SSD etcd disk per node, GiB (etcd DB is ~83 MB; 16 GiB is generous headroom)."
type = number
default = 16
}
```
2. **New disk block** in the VM resource in `main.tf`, appended AFTER the existing
`virtio1` block (keep interface order ascending so bpg does not reorder):
```hcl
# Dedicated SSD etcd disk (virtio2 -> /dev/vdc). Isolates etcd WAL fsync from
# the HDD Storage pool where the OS + Longhorn disks live.
disk {
datastore_id = var.etcd_ssd_datastore
interface = "virtio2"
size = var.etcd_disk_gib
discard = "on"
cache = "none" # honor guest fsync through to ZFS ZIL (PLP SSD = safe+fast)
iothread = false
}
```
3. **Plan and inspect before applying.** `tofu plan` MUST show an **in-place update
(disk add)** for the targeted VM — NOT a replacement. If it shows replace, stop and
reassess (do not destroy a running etcd node).
4. **Apply one node at a time** with `-target` (see §4.3 for quorum ordering), e.g.
`-target='proxmox_virtual_environment_vm.node["k3s-dev-3"]'` (adjust to the actual
resource address in this repo). A disk add is applied online by bpg, but treat the
node as going down for the etcd migration regardless.
> **Coupling caveat — dev-3 memory drift.** dev-1/dev-2 are at `memory_mib = 24576`
> but **dev-3 is live at 16384** (drift from tofu default 24576). A tofu apply that
> touches dev-3 will also try to reset its memory to 24576. Per the Proxmox rule,
> memory changes need a **full stop/start**, not a reboot. Decide deliberately:
> either (a) let the same maintenance window fix the memory drift on dev-3 (stop →
> set → start), or (b) `-target` narrowly and accept that dev-3's plan will include
> the memory change. Do dev-3 in its own window with eyes on it.
**Manual alternative (if not using tofu):** on the Proxmox host,
`qm set <vmid> -virtio2 local-zfs:16,discard=on,cache=none` then stop/start the VM.
This drifts from tofu state and is discouraged — if done, import/reconcile into tofu
afterward. Prefer the tofu path.
### 4.2 Optional space guarantee
`local-zfs` is `sparse` (thin), so the etcd zvols consume only real usage. To
guarantee etcd can never be squeezed if the system disk fills, set a modest
`refreservation` on each etcd zvol (e.g. `zfs set refreservation=16G
rpool/data/vm-<id>-disk-2`). 3×16 GiB = 48 GiB reserved out of 323 GiB avail — safe.
This is a host-side ZFS tweak, not expressible in bpg; apply manually if desired.
### 4.3 Migrate etcd onto the new disk — quorum-safe, ONE node at a time
k3s has no flag to relocate only the etcd directory (`--data-dir` moves everything),
so the clean approach is to **mount the SSD disk at
`/var/lib/rancher/k3s/server/db`** (this directory holds `etcd/` — member data, WAL,
snapshots — and nothing else that matters).
**Pre-flight:** confirm all 3 etcd members healthy and the cluster is quiet (no active
Longhorn rebuild). `kubectl get nodes` all Ready; check etcd member health.
**Per node, repeat for dev-3, then dev-2, then dev-1 (leader last):**
1. Verify 3/3 etcd members healthy BEFORE touching this node (losing one of three
keeps quorum at 2/3; never take a second node until the previous is fully back).
2. Stop k3s on the node: `systemctl stop k3s`. (Node leaves quorum; 2/3 remain.)
3. Identify the new disk (`/dev/vdc` — the third virtio disk) and format it:
`mkfs.ext4 -L k3s-etcd /dev/vdc` (ext4 or xfs; ext4 is fine and simple).
4. Preserve and move existing data:
```
mv /var/lib/rancher/k3s/server/db /var/lib/rancher/k3s/server/db.bak
mkdir /var/lib/rancher/k3s/server/db
mount /dev/vdc /var/lib/rancher/k3s/server/db
rsync -aHAX --numeric-ids /var/lib/rancher/k3s/server/db.bak/ /var/lib/rancher/k3s/server/db/
```
5. Persist the mount in `/etc/fstab` **by UUID** (`blkid /dev/vdc`), e.g.
`UUID=<uuid> /var/lib/rancher/k3s/server/db ext4 defaults,noatime 0 2`, then
`systemctl daemon-reload`.
6. Start k3s: `systemctl start k3s`. Wait until the node is `Ready` AND its etcd
member is healthy and has re-joined (member list shows 3 started, no learner
stuck). Confirm the VIP `192.168.2.50:6443` is stable.
7. Only after full recovery, remove the backup: `rm -rf
/var/lib/rancher/k3s/server/db.bak`. Then proceed to the next node.
**Reproducibility for rebuilds:** the SSD disk + mount should also be baked into the
node **cloud-init first-boot baseline** (the same template that already provisions the
Longhorn `virtio1` disk and swap-off), so a `tofu`-rebuilt node comes up with
`/dev/vdc` formatted and mounted at the etcd path automatically. Update that template
in the same change set as the tofu disk block.
### 4.4 Validation
- `kubectl get nodes` — all 3 Ready; no flapping over a 30–60 min soak.
- etcd: 3 members, all started/healthy, one leader; DB size sane (~tens–hundreds MB).
- On each node, `df -h /var/lib/rancher/k3s/server/db` shows the new `/dev/vdc` mount.
- **Regression test the original failure:** with the cluster healthy, trigger a
controlled Longhorn replica rebuild (e.g. bounce ONE node after approval) and
confirm etcd stays healthy and the VIP does not flap — the whole point of the change.
- Host-side: `zpool status rpool` clean; `zfs list rpool/data` shows the 3 new
`vm-*-disk-2` zvols with small real usage.
### 4.5 Rollback
Per node, fully reversible while `db.bak` still exists (before step 7):
`systemctl stop k3s` → `umount .../server/db` → `rmdir` the empty mountpoint →
`mv db.bak db` → remove the fstab line → `systemctl start k3s`. To fully revert the
disk, detach `virtio2` (`qm set <vmid> -delete virtio2` after stop) or `tofu` revert
the disk block, then stop/start. Because etcd is a replicated 3-member cluster, a
botched single node can also simply be reset and re-joined from the other two.
---
## 5. Risks
1. **System-disk contention / exhaustion (top risk).** `rpool` is also the Proxmox
boot/root disk (430 GiB used of 888 GiB). Mitigations: keep the etcd disks tiny
(16 GiB, thin), optional `refreservation` (§4.2), and **never** put Longhorn here
(§2). etcd's steady-state real usage is ~hundreds of MB; the marginal I/O it adds
to the SSD is negligible next to what it removes from the HDD.
2. **Consumer-SSD fsync/PLP caveat — N/A here, but assert it.** etcd correctness under
power loss depends on honest fsync. These are **Micron 5100/52x0 enterprise SATA
SSDs with power-loss protection**, so `sync=standard` + `cache=none` is both fast
and durable. (If the disks were consumer SSDs without PLP this concern would be
real — it is not, for this hardware.)
3. **Quorum during migration.** Mitigated by strict one-node-at-a-time with full
recovery gates (§4.3) and leader-last ordering. Never take a second node down
before the previous rejoins.
4. **tofu coupling with dev-3 memory drift + apply semantics (§4.1).** An apply
touching dev-3 also resets memory 16384→24576 (needs stop/start). Verify `tofu
plan` shows disk *add*, not VM *replace*, before applying. Use `-target`.
5. **Backup/restore assumptions.** k3s etcd snapshots (`/var/lib/rancher/k3s/server/db/snapshots`)
now live on the SSD too — fine, but confirm any snapshot/backup job still points at
the (unchanged) path.
---
## 6. Recommendation summary
Implement **Option A**: a 16 GiB SSD-backed `virtio2` disk per k3s VM (zvol on
`rpool/data` via `local-zfs`), etcd-only, migrated one node at a time with the
quorum-safe procedure in §4.3, modeled in tofu (+ cloud-init baseline) for
reproducibility. Do **not** move Longhorn to SSD. This is a small, reversible,
high-leverage fix targeting the exact WAL-starvation failure.
---
## 7. Relationship to the deferred SSD-mini-PC migration
This is explicitly an **interim bridge** on the current Proxmox dev host
(`192.168.2.102`), whose bulk storage is HDD. Once the k3s cluster migrates to the
planned **all-SSD mini-PCs**, every I/O path (OS, etcd, Longhorn) is already on SSD
and the HDD-contention failure mode disappears — the dedicated etcd disk becomes
unnecessary. At that point the isolation can be either dropped or carried forward
harmlessly (a dedicated etcd disk is still good hygiene). Because Option A is captured
in tofu + cloud-init, it travels with the cluster definition and can be removed in one
reviewed change when the mini-PCs land. Until then it stops the recurring incidents
at minimal cost.
@@ -15,3 +15,10 @@ node_baseline_kernel_modules:
node_baseline_longhorn_device: /dev/vdb
node_baseline_longhorn_mount: /var/lib/longhorn
node_baseline_longhorn_fs_label: longhorn
# Dedicated SSD etcd disk (tofu virtio2 -> /dev/vdc), mounted at the k3s etcd data dir
# so etcd WAL fsync is isolated from Longhorn rebuild I/O on the HDD data disk.
# See docs/plans/ssd-etcd-isolation.md. FS label keeps the fstab entry stable across
# device renames (matches the Longhorn LABEL= approach).
node_baseline_etcd_device: /dev/vdc
node_baseline_etcd_mount: /var/lib/rancher/k3s/server/db
node_baseline_etcd_fs_label: k3s-etcd
@@ -72,6 +72,31 @@
opts: "defaults,nofail,x-systemd.device-timeout=5s"
state: mounted
# Dedicated SSD etcd disk (docs/plans/ssd-etcd-isolation.md). Formats /dev/vdc and
# mounts it at the k3s etcd data dir BEFORE k3s starts, so a rebuilt node comes up with
# etcd already isolated on SSD. Data-safe: community.general.filesystem never reformats
# a disk that already carries a filesystem, so re-running against an already-migrated
# node leaves the etcd data intact. On EXISTING nodes the one-time data migration
# (stop k3s -> rsync db -> mount) is done manually per the plan's §4.3 before this
# role would mount over a populated directory.
- name: Create ext4 filesystem on the dedicated etcd SSD disk (data-safe; skips if present)
community.general.filesystem:
fstype: ext4
dev: "{{ node_baseline_etcd_device }}"
opts: "-L {{ node_baseline_etcd_fs_label }}"
- name: Mount the dedicated etcd SSD disk at the k3s etcd data dir
ansible.posix.mount:
path: "{{ node_baseline_etcd_mount }}"
src: "LABEL={{ node_baseline_etcd_fs_label }}"
fstype: ext4
# noatime: fewer metadata writes under etcd's fsync-heavy load. No `nofail`: a
# missing etcd disk MUST fail loudly rather than let k3s start etcd on the root HDD.
opts: "defaults,noatime"
dump: "0"
passno: "2"
state: mounted
- name: Disable swap now
ansible.builtin.command: swapoff -a
changed_when: false
+28 -17
View File
@@ -41,13 +41,17 @@ locals {
# -> Storage -> local -> Content -> check "Import"), else the disk import fails with
# "wrong type 'iso' - needs to be 'images' or 'import'".
#
# Resource type is the SHORT name `proxmox_download_file` (bpg/proxmox ADR-007). The old
# `proxmox_virtual_environment_download_file` is deprecated ("Use `proxmox_download_file`
# instead. This resource will be removed in v1.0.") — both are the same dual-registered
# Framework resource, so the schema is identical. The `moved` block below migrates existing
# state in place (provider implements MoveState / migration.PrefixMoveState), so no
# re-download of the image and no change to the VMs.
resource "proxmox_download_file" "debian_cloud_image" {
# The Debian cloud image is a bpg `download_file` resource. This intentionally uses the
# LONG (deprecated) type name `proxmox_virtual_environment_download_file`: the R2 backend
# state still holds this resource under the long name, and bpg >= 0.111 dropped the
# cross-type `moved{}` linkage to the short `proxmox_download_file` (ADR-007) — a `moved`
# block between the two now fails plan with "Resource type mismatch" (this broke plan on
# main entirely). Keeping the long name makes config match state, so the plan is clean and
# the ~450 MB image is NOT re-downloaded. Migrating to the short name is a SEPARATE,
# deliberate step (`tofu state mv proxmox_virtual_environment_download_file.debian_cloud_image
# proxmox_download_file.debian_cloud_image`, then rename here) — not bundled with the
# etcd-disk change so this bounce stays focused and low-risk.
resource "proxmox_virtual_environment_download_file" "debian_cloud_image" {
content_type = "import"
datastore_id = var.image_datastore
node_name = var.proxmox_node
@@ -56,15 +60,6 @@ resource "proxmox_download_file" "debian_cloud_image" {
overwrite = false
}
# State migration for the deprecated -> short resource-type rename (ADR-007). OpenTofu
# 1.9 (>= 1.8 required) drives the provider's MoveState RPC: the download_file state is
# transferred in place, keeping the same id (datastore_id:content_type/file_name), so the
# already-downloaded Debian image is NOT re-downloaded and the VMs' import_from is unchanged.
moved {
from = proxmox_virtual_environment_download_file.debian_cloud_image
to = proxmox_download_file.debian_cloud_image
}
# --- cloud-init: NATIVE Proxmox cloud-init only (API-driven) ------------------
# The rich first-boot baseline (packages, kernel modules, Longhorn disk, swap-off)
# is NOT delivered as a custom snippet any more — custom snippets require SSH/SFTP
@@ -118,7 +113,7 @@ resource "proxmox_virtual_environment_vm" "node" {
# fs_setup/mount for /var/lib/longhorn already expects.
disk {
datastore_id = var.datastore
import_from = proxmox_download_file.debian_cloud_image.id
import_from = proxmox_virtual_environment_download_file.debian_cloud_image.id
interface = "virtio0"
size = var.os_disk_gib
discard = "on"
@@ -131,6 +126,22 @@ resource "proxmox_virtual_environment_vm" "node" {
discard = "on"
}
# Dedicated SSD etcd disk (virtio2 -> /dev/vdc). Isolates etcd WAL fsync from the HDD
# `Storage` pool where the OS + Longhorn disks live — the node_baseline role formats
# it and mounts it at /var/lib/rancher/k3s/server/db before k3s starts. On the SSD
# `local-zfs` (rpool/data) datastore: an enterprise Micron SATA SSD mirror WITH
# power-loss protection, so cache=none (honor guest fsync straight through to the ZFS
# ZIL) is both durable and fast. Kept last (ascending interface) so bpg won't reorder
# the existing virtio0/virtio1 disks. See docs/plans/ssd-etcd-isolation.md.
disk {
datastore_id = var.etcd_ssd_datastore
interface = "virtio2"
size = var.etcd_disk_gib
discard = "on"
cache = "none"
iothread = false
}
network_device {
bridge = var.bridge
}
+18
View File
@@ -100,6 +100,24 @@ variable "data_disk_gib" {
default = 128
}
# --- dedicated SSD etcd disk (interim etcd-on-SSD isolation, docs/plans/ssd-etcd-isolation.md) --
# etcd WAL fsync was being starved by Longhorn replica-rebuild I/O on the shared HDD
# `Storage` pool (OS + Longhorn disks). A tiny dedicated SSD disk per node moves the
# etcd data dir (/var/lib/rancher/k3s/server/db, mounted by node_baseline) off the HDD.
# Interim bridge until the all-SSD mini-PCs land; Longhorn stays on HDD (does not fit).
variable "etcd_ssd_datastore" {
description = "SSD-backed datastore for the dedicated etcd disk (VERIFIED: local-zfs = rpool/data, ZFS mirror on Micron 5100 SATA SSDs w/ power-loss protection)."
type = string
default = "local-zfs"
}
variable "etcd_disk_gib" {
description = "Dedicated SSD etcd disk per node, GiB (etcd DB is ~83 MB; 16 GiB is generous headroom)."
type = number
default = 16
}
variable "cpu_type" {
description = "Proxmox vCPU type. `host` passes through the physical CPU (nested-virt for KubeVirt, Phase 5)."
type = string