diff --git a/infra/PROD-PROVISIONING.md b/infra/PROD-PROVISIONING.md index 8a9be4e..295a536 100644 --- a/infra/PROD-PROVISIONING.md +++ b/infra/PROD-PROVISIONING.md @@ -64,8 +64,15 @@ is per-machine state. **BIOS settings compared programmatically (2026-09-01).** Both nodes load the `hp-bioscfg` kernel driver, which exposes **215 BIOS attributes** read/write at `/sys/class/firmware-attributes/hp-bioscfg/attributes//current_value`. No BIOS passwords -are set (`authentication/*/is_enabled` = 0), so these are also **writable from the OS** — see -§1, this changes how the firmware checklist can be enforced. +are set (`authentication/*/is_enabled` = 0). + +**They are READ-ONLY in practice, despite `current_value` having mode `rw`.** A write attempt was +made on 2026-09-01 and the FIRMWARE refused it: +`hp_bioscfg: Returned error 0x4, "Invalid command type"`, surfacing as `printf: I/O error`. +HP requires BIOS-admin authentication for programmatic changes, and both credentials are absent +— `Setup Password is_enabled=0` (role `bios-admin`) and `SPM key_mechanism=not provisioned`. +Secure Boot is also enabled. So BIOS changes on these nodes are an **F10 exercise** unless a BIOS +Setup Password is deliberately provisioned (see §1). What the comparison ruled OUT as the cause: - **Firmware inventory is correct.** Node 2's BIOS itself reports `Memory Size=16384 MB` and @@ -97,7 +104,7 @@ and a reboot, which needs operator sanction. To investigate, cheapest first: - [ ] **Set `Fast Boot=Disable` and reboot** (tests the hypothesis directly, and matches node 1). - Can be done from the OS — see §1 — or in the F10 setup menu. + **Must be done in F10** — the firmware refuses programmatic writes (above). - [ ] **Set `After Power Loss=Power On`** regardless of the memory question — §1 requires it and node 2 currently violates it. - [ ] Clear CMOS / load setup defaults on node 2 to force a full memory retrain, then re-check. @@ -115,13 +122,21 @@ To investigate, cheapest first: ## 1. Firmware / BIOS (per node, before OS install) -> **These are now MACHINE-CHECKABLE, not just eyeball-at-F10 items.** The `hp-bioscfg` driver -> exposes 215 BIOS attributes at `/sys/class/firmware-attributes/hp-bioscfg/attributes/`, and -> with no BIOS password set they are **writable from the OS**. That means the checklist below can -> be *asserted* by `roles/node_health` and *enforced* by a role, instead of trusted to whoever -> was last in the setup menu — which is exactly how node 2 ended up with `After Power Loss=Power -> Off` and `Fast Boot=Enable` while node 1 has the opposite. **Follow-up worth doing: extend -> node_health to assert the firmware checklist, and add a role to converge it.** +> **These are MACHINE-CHECKABLE — but NOT machine-enforceable.** The `hp-bioscfg` driver exposes +> 215 BIOS attributes at `/sys/class/firmware-attributes/hp-bioscfg/attributes/`, and **reads work +> fine**: node 1 and node 2 were diffed programmatically, which is how the drift in §0b was found +> at all. Nobody could have spotted `After Power Loss=Power Off` on node 2 by eye. +> +> **Writes are refused by the firmware** — verified 2026-09-01, `hp_bioscfg: Returned error 0x4, +> "Invalid command type"`. HP requires BIOS-admin authentication and neither credential is +> provisioned (`Setup Password is_enabled=0`, `SPM not provisioned`). Setting these therefore means +> F10 on the box. +> +> **Follow-up worth doing: extend `roles/node_health` to ASSERT this checklist** — that is +> achievable today with reads alone, and would have caught node 2's drift on its first health run. +> Converging it automatically additionally requires provisioning a BIOS Setup Password on every +> node, which is a real security decision (a stored credential that can rewrite firmware) and is +> the operator's call, not a default. - [ ] **Virtualization enabled** — VT-x *and* VT-d. Required for KubeVirt (R-PLAT-03); verify after