161430d9154b6c120a375aaa39d651e69e78f310
87
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
9739e9be91 |
chore(backup): R2 credentials for the off-cluster DR target
Written by platform/backup/set-r2-credentials.sh (operator, 2026-09-07T05:41:34Z) into all four consumers, in the two shapes they need: velero wants an AWS INI file, CNPG/barman wants discrete ACCESS_KEY_ID / ACCESS_SECRET_KEY. All four verified to carry the SAME 32-character key id, so no consumer was missed. Proven from a pod INSIDE prod against https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com: head-bucket on k3s-prod-backup returns BucketRegion WNAM, and PUT and DELETE both succeed. Byte-integrity of GET is NOT yet proven (the aws-cli image has no cmp); s3check is what will establish it. Credentials only — no endpoint or bucket wiring. That is feat/r2-backup-prod. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
d1a6628a97 |
feat(ha): the CNI dials the control-plane VIP — kube-vip becomes a CNI prerequisite
OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach
the cluster and provide fault tolerant HA. Without a change it will not work in HA."
THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live
before changing anything: all three prod cilium-agent pods carried
KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip
VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the
CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said
the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated
that stale comment as an open question when the evidence already settled it.
SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs:
k3s_api_host unchanged — the FOUNDER's IP: the cold-bootstrap join
target and a tls-san. infra/tofu/outputs.tf GENERATES this
name, so renaming would ripple into tofu and the runbooks.
k3s_apiserver_endpoint NEW = {{ kube_vip_address }} — the HA address; what Cilium
consumes. Derived, not restated, so an environment overrides
ONE line and the VIP, the SAN and the CNI endpoint move
together. Prod needed no new override.
THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at
the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran
node-baseline -> k3s -> cilium, so Cilium would install against an address nothing
announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value:
site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier
(kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there
because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing
neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so
a cold run stops THERE with a named cause instead of surfacing as unreachable agents.
Also fixed a pre-existing bug that only mattered once this play joined site.yml: the
role's read-only interface probe ran without `check_mode: false`, so under --check it
was skipped and the assert fired with a misleading "set kube_vip_interface explicitly".
VERIFIED BY ME, not just reported:
renders dev K3S_API_HOST=192.168.1.49 prod=192.168.2.9 (each environment's VIP)
cilium template consumes k3s_apiserver_endpoint, not k3s_api_host
cold join still `--server https://{{ k3s_api_host }}:6443` — the founder
order site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux
gate full CI gate PASS
Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing
k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders
['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved
— .9 already answers under full CA verification; and --check --diff of kube-vip.yml
against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the
kube-vip prod already runs.
ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine
checks: the Cilium template must use the endpoint var; the endpoint must derive from
kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal
that environment's VIP and be none of its node IPs (the [parity] contract, now
machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium.
Every new assertion mutation-tested and bites. One honest negative recorded: reverting
only the BASE default is not caught, because both overlays patch the key — the check
reads effective per-environment values, so the base default is unguarded.
APPLY ORDER, and this is the operator's step — nothing was applied:
PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let
Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet
ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01
incident), it will not proceed past an unready pod, existing datapath survives an
agent restart, and .9 is already live before the change lands.
DEV SEPARATELY, operator present. Same mechanics but live dev agents dial
192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that.
ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a
cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook.
NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live
KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that
test's gap list.
Refs R-PLAT-01, R-NET-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
c8c1d2935d |
feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).
THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.
Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.
PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.
BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.
TWO THAT CANNOT BE TOKENIZED, flagged not forced:
- headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
anyway — client secrets are realm-scoped.
- kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
prod.home.
EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.
I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.
DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.
L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.
PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.
Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
9b8979a427 |
feat(prod): temp Gitea source + wireless disabled + flux bootstrap path
Autonomous prod bring-up, operator away. Three changes that together clear the last blockers before `flux bootstrap`. === 1. TEMPORARY BOOTSTRAP GITEA — the last hard blocker, cleared === clusters/prod's source URL could not work: Gitea runs ON the dev cluster (the one slated for decommission) and split-horizon DNS resolves gitea.home.keithjackson.dev to PROD's OWN gateway. Stood up the existing bootstrap/gitea/bootstrap-gitea.sh on the operator's workstation with HOST_NAT_IP=192.168.2.132; all three prod nodes verified reachable (Gitea 1.22.6 via the API), no firewall change needed — prod is on the same /24, no NAT. clusters/prod now points at http://192.168.2.132:3000/... under a boxed TEMPORARY banner with a 5-step cutover checklist ending 'delete this whole block — dev's stanza is what done looks like'. Repo verified PUBLIC with anonymous read two independent ways (API "private":false; git ls-remote unauthenticated with GIT_TERMINAL_PROMPT=0), so NO secretRef is needed — documented so a future auth failure is not misread as a network fault. bootstrap-gitea.sh's header corrected: it documented HOST_NAT_IP=10.73.0.1, the RETIRED libvirt NAT, and forbade agent use; the operator has now directed this for prod. === 2. WIRELESS DISABLED ON PROD (operator-reported) === k3s-prod-3 was on ethernet AND wireless, and the wifi DHCP client was overwriting /etc/resolv.conf. Root cause of the silence: node_baseline's DNS `contended` detection checked resolvconf and NetworkManager but NOT dhcpcd, so the role wrote resolvers, reported success, and dhcpcd reverted them on lease renewal — the exact 'green run that fixed nothing' the contended branch exists to prevent. Now IaC, not a one-off: survey wireless hardware, **REFUSE to disable it if the path to the node runs over the wireless interface** (the guard that meant an agent dying mid-apply cost nothing), blacklist the driver so it never reloads, stop dhcpcd only where it holds a wireless interface and nothing else, take links down and unload modules with NO reboot, then report the resulting state. dhcpcd is now in the contended set without weakening fail-closed. Applied and verified: node 3's wlp1s0 is GONE from /sys/class/net, dhcpcd inactive and its unit not-found, ONE default route per node, all three on 8.8.8.8/1.1.1.1 resolving public names. Safe and idempotent on nodes 1 and 2, whose wifi hardware was already down. === 3. flux_bootstrap PATH PARAMETERISED === bootstrap/flux/bootstrap.sh already read FLUX_ENV_PATH but the role never exported it, so it ALWAYS bootstrapped clusters/dev — prod was impossible. Two behaviour-bearing lines: role default clusters/dev + `environment: FLUX_ENV_PATH`, overridden to clusters/prod in group_vars/k3s_prod. Proven per host with a read-only Ansible probe: prod hosts resolve clusters/prod, localhost/hypervisor resolves clusters/dev. Only two definitions repo-wide. DEV PROVEN BYTE-IDENTICAL for the source change: all 12 substituting Flux paths rendered for both trees, 27/27 SHA-256 artefacts identical, with TWO POSITIVE CONTROLS (an overlay-level canary and a deep-path content canary) confirming the harness detects change — and both canaries proven restored. Honest scope note recorded: kustomize strips comments, so the harness proves the built stream, not source comments. L0 substitution check: 106 assertions, 0 hazards. ci/l0.sh and ci/lint.sh exit 0. === STILL BLOCKING A FULL PROD RECONCILE (operator decisions) === - S3_BACKUP_ENDPOINT remains TBD-PROD-*: prod CANNOT reach dev's versitygw at 192.168.1.9:9000 (verified UNREACHABLE from all three — the pfSense boundary), and a prod backup target is a deferred operator decision. Deliberately NOT invented: a DR target on a workstation container would create false confidence. Velero/CNPG backups will fail loudly. - Prod's public resolvers mean the API server's oidc-issuer-url resolves PUBLICLY, so kubectl SSO will not work on prod until split-horizon is solved for it (§4). - The temp Gitea is a single container on the operator's workstation — a single point of failure for prod reconciliation, acceptable for bring-up only. Serves: R-GITOPS-01, R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
05baae90ec |
feat(prod): kernel fix + disk layout — prod nodes ready for k3s
Prod is being stood up now, without waiting for node 2's RAM upgrade. Three coupled
changes, all applied and verified on all three nodes.
=== 1. KERNEL >= 6.13 — now a PREREQUISITE, not a nice-to-have ===
A k3s SERVER that intermittently boots with 2.6 GiB is not a degraded node; it fails under
etcd plus its workload share, unpredictably, on any reboot. Fixed before k3s goes on.
roles/node_baseline gains a `kernel`-tagged block: asserts the suite matches the node's
Debian release, writes a deb822 trixie-backports source, installs, reads back the exact
version, writes a provenance/rollback note to the node. Scoped by
node_baseline_backports_kernel (default FALSE, true only in group_vars/k3s_prod) — chosen
over a hostname test so a fourth prod node inherits it from group membership. Verified:
--check against dev skips all 8 tasks on all 3 hosts. Two deliberate calls: `state: latest`
not `present` (linux-image-amd64 was already installed from main, so `present` would have
silently fixed nothing), and NO preferences.d pin (backports' NotAutomatic +
ButAutomaticUpgrades already gives 100/500; an explicit Pin-Priority: 100 would switch OFF
automatic security upgrades for the very kernel being installed).
Installed 7.1.8+deb13-amd64 on all three; 6.12.107 kept bootable.
THE DECISIVE PROOF: node 2 booted at MEMATTR=0xbeeb0018 — the EXACT address that gave it
2.6 GiB twice — and now reserves 634,904K instead of 13,958,980K. MemTotal 15.4 GiB. Same
address, handled correctly; not a lucky draw.
§0b's claim that a fixed kernel prints 'FW_BUG Corrupted...' is CONDITIONAL and was
corrected: >=6.13 has three rejection gates and only the third prints that line; all three
nodes hit gate 1 (version > 2). Proof instead is boot-independent — decompressing
/boot/vmlinuz on all three yields the FW_BUG and descriptor-size strings verbatim; neither
exists in 6.12.
TWO MORE BUGS IN MY OWN HEALTH CHECK, found because the fix couldn't otherwise be shown:
- MEMATTR_VERSION_RE matched (\d+) but the kernel prints that field with %d — SIGNED. Node
2's real value was -1569980416, which did not match, so state fell through to `clean` and
the node reported PASS "the kernel accepted it without complaint". A FALSE PASS on the
exact bug the check exists to find. Now (-?\d+).
- bad_version was WARN unconditionally, so a FIXED node would have stayed WARN forever.
Now keys on memattr_kernel_fixed: same line, WARN below 6.13, PASS at/above.
=== 2. DISK LAYOUT — Longhorn claimed, per-node paths DISCOVERED not hardcoded ===
Operator decision: reuse the ~424 GB partition; accept the shared-device I/O risk (dev's
etcd/Longhorn contention was on a spinning HDD, prod is SSD/NVMe).
Device paths differ per node (sda5 / nvme0n1p5 / sda5), so the role DISCOVERS the partition
via `findmnt --mountpoint` on a declared claim mountpoint rather than trusting inventory.
Per-host host_vars would have frozen today's hardware into inventory, where it goes stale on
the first re-image and the stale value's failure mode is mkfs on the WRONG partition.
Eleven fail-closed asserts before any write: real block device, not backing / /boot /usr
/var /home /tmp or active swap (built from the node's OWN gathered facts, never a hardcoded
list), already ext4, no foreign label, >= 100 GiB, and empty. Relabel, never reformat.
/srv verified empty first on every node (ls -A -> lost+found only; du 20K; the "5.1% used"
was ext4's default 5% root reserve).
=== 3. k3s DATA-DIR MOVED — /var was 11.8 GiB and would have killed etcd ===
Measured on the fully-deployed dev cluster: containerd 21-25 GB/node, /var total 58-70 GB.
Prod's /var is 11.8 GiB and held BOTH etcd and the image store, so prod would have filled
/var during its first image pulls — and a full /var means etcd cannot write its WAL. Cluster
death, not degradation, arriving during bring-up. Invisible when the disk decision was made,
because "etcd on root" sounded like the 30 GiB / rather than a separate 12 GiB /var.
Now: p5 -> /data (LABEL=k3s-data), with /data/longhorn and /data/k3s. Neutral names chosen
deliberately — the partition now has two tenants, and reusing /srv would make "claimed" and
"unclaimed" indistinguishable to the idempotency pivot, while a bare `data` label would make
a stranger's `data` volume look already-claimed and weaken an assert.
The transition off the existing LABEL=longhorn mount is DECLARED, not inferred
(node_baseline_data_retired_fs_label/_retired_mount, defaults ""), because changing the label
makes a claimed node answer "no" to the pivot and walk into the greenfield path. Self-retiring
once blkid -L k3s-data succeeds. THREE independent gates keep live dev out: retired vars
default ""; gated on longhorn_source == claim-mountpoint (dev is `device`); and the emptiness
assert fails closed on dev's 18 volumes even if both were defeated.
k3s_data_dir is one variable read by two roles (k3s_server --data-dir; node_baseline derives
the etcd mount). Role default is k3s's own default, so dev is unchanged and keeps its
dedicated etcd device. node_baseline_etcd_source becomes `shared-data-partition` and is
LOAD-BEARING: the role asserts per run that the filesystem backing the etcd dir really IS the
data partition — had the mount silently failed, etcd plus 21-25 GB of images would land back
on the OS filesystem invisibly until the disk filled.
LONGHORN_DATA_PATH is a new cluster-vars knob (R-ENV-02): base and DEV /var/lib/longhorn,
prod /data/longhorn.
DEV PROVEN UNCHANGED, which is the property that mattered — the live cluster has 18 Longhorn
volumes on that path. All 12 substituting Flux paths rendered through
`kubectl kustomize | flux envsubst --strict` for both trees: 12/12 per-file SHA-256 identical
and an identical aggregate SHA over every rendered dev byte. With a POSITIVE CONTROL showing
the same harness detects prod changing on exactly one line — an all-identical result is
worthless if the harness cannot see change. Live check after: dev's default-data-path setting
still /var/lib/longhorn, 18 volumes attached, 0 Kustomizations unready.
Idempotent: run 2 is changed=0 on all three nodes. Also fixed a real tag bug — include_tasks
does NOT propagate its own tags to included tasks (unlike import_tasks), so `--tags disks`
selected the include and ran nothing inside it while reporting success.
OPEN DEFECTS, tracked not forgotten: node 2's warm-reboot hang (one clean reboot since, NOT a
fix); and node 3 runs dhcpcd for its wifi interface, which rewrites /etc/resolv.conf on lease
renewal — the role's `contended` detection checks resolvconf and NetworkManager only, so node
3's decided resolvers will silently revert.
Gate: MERGE GATE: PASS. L0 substitution check 106 assertions.
Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-STATE-01, R-STATE-02, R-PLAT-01. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
8e913fdd7b |
fix(node_health): "Failed to map" does NOT mean memory lost — plus survey results
THE MODEL WAS WRONG AND MY CHECK WITH IT. A live counter-example: k3s-prod-2 after its power
cycle shows MEMATTR=0xbeeb0018 and 'Failed to map ... @ 0xbeeb0018' — the SAME address and SAME
line as both of its 2.6 GiB boots — with MemTotal 15857732 kB, i.e. 15.1 GiB of 16 GB. FULL
MEMORY. The check reported FAIL 'THIS NODE HAS LOST MEMORY TO THE BUG' on a healthy node while
memory.physical_total correctly PASSed at 4.2%. The two checks contradicted each other.
REFINED MECHANISM — two kernel functions were conflated:
efi_memattr_init() does the memblock_reserve(). Catastrophic loss needs tbl_size NEGATIVE
(signed int -> sign-extended -> clamped to PHYS_ADDR_MAX). A garbage-but-POSITIVE tbl_size
reserves a modest wrong amount and costs nothing noticeable.
efi_memattr_apply_permissions() does the later memremap(), and THAT is what prints
'Failed to map'. It fails for ANY sufficiently wrong size, negative or merely large-positive.
So the line proves the table was corrupt AND TRUSTED, not that memory was lost. The cost is set
by the garbage VALUE, which is why one address gave 2.6 GiB twice and 15.1 GiB once.
THIS ALSO REFUTES THE ADDRESS->OUTCOME HYPOTHESIS OUTRIGHT — not untested, disproven.
STRUCTURAL FIX so the contradiction cannot recur: the shortfall is computed ONCE into a shared
fact (node_health_mem_shortfall) with worst-wins sub-verdicts; memory.physical_total IS that
verdict and firmware.efi_memattr READS it, measuring nothing itself. FAIL is gated on the shared
verdict. 'Failed to map' + normal memory is now WARN ('the bug fired and this boot survived it').
No honesty guard was weakened; one was strengthened (the measurement clause falls back to the DIMM
shortfall when zoneinfo is unreadable, and only claims 'could NOT be measured' when neither was
read). All five paths incl. the two unproducible-live SKIPs exercised against synthetic facts.
Verdicts: prod-1 WARN, prod-2 WARN (was FAIL), prod-3 PASS, dev unaffected.
SECOND FAULT, arguably worse for prod: NODE 2 HANGS ON A WARM REBOOT, STAYING POWERED. Its
journal ends cleanly mid-shutdown ('eno1: NIC Link is Down') then stops; the box sat powered but
off-LAN for ~38 min until the operator held the power button. A node that cannot survive an
unattended warm reboot cannot be rolled in an HA cluster, and node_health cannot see a machine
that is off the network. Leading candidate: node 2 is the ONLY node with 'Recover OS after Boot
Failure = Enable' — promoted from 'coincidence' in the BIOS diff to a live availability defect.
The reboot survey is NOT viable unattended on node 2 until this is understood.
SURVEY RESULTS (40 boots): prod-1 0/20, prod-3 0/20, prod-2 0 ROUNDS (hung on its first reboot),
pooled 0/40 -> p(bad) 95% CI 0.000-0.088. NOT elimination: 0/20 still permits 17% per node, and
node 2's prior 2-of-3 is unchanged. What it did establish: addresses are node-specific and highly
concentrated (node 1 used 3 distinct addresses in 20 boots, 15 at one; no address ever shared
between nodes); MEMATTR -> parse was a PERFECT function across all 7 repeated addresses over 37
boots; and the corrupting bytes are stable PER MACHINE (node 1 always 1030, node 3 always
1463964749), so the 'shared OS boot chain' inference needs refining.
Adds playbooks/memattr-survey.yml + roles/memattr_survey — reboots opt-in, resumable, artifacts
gitignored. Documents an Ansible defect worth knowing repo-wide: a looped include_tasks under the
default 'linear' strategy is flattened per host, so hosts run SERIALLY; the first attempt rebooted
node 1 twenty times while the others idled. Fixed with strategy: free + a forks preflight assert.
Serves: R-ENV-01, R-ENV-03, R-OBS-04, R-TEST-02. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
b19c5db9ae |
feat(prod): node 3 onboarded — and it is NOT immune; BIOS-setting hypothesis disproven
k3s-prod-3 online at the planned 192.168.2.12 (serial 8CC015307N, same chassis/BIOS/CPU),
enabled in inventory, vaulted become confirmed, package floor installed. Health: 0 FAIL /
4 WARN / 14 PASS / 1 SKIP — the only node with zero FAILs. WD Blue WDS500G2B0A SATA SSD at
100% life, 3213 h, 0 grown bad blocks, 0 erase fails. Matched Crucial 2x16 GB pair — the only
node with symmetric memory.
*** NODE 3 IS NOT IMMUNE TO THE §0b BUG. *** Its firmware DOES publish a MAT
(MEMATTR=0xbeece018); the ABSENCE of a memattr diagnostic line means the kernel read a VALID
table, which is the opposite of immunity. There are THREE outcomes, not two: bail-out on a
garbage version >2, a valid parse with a sane size, and the catastrophic path. Seven boots across
three machines show the MAT address MOVES BOOT-TO-BOOT on every node with settings constant, and
each node has been seen on more than one path. Blast radius widens from two nodes to ALL THREE;
all three need the >= 6.13 kernel.
THE CORRUPTING BYTES ARE THE OS BOOT CHAIN, not firmware and not DRAM: garbage version
1463964749 = 0x5742544D appeared on node 2 AND node 3 — two different machines, identical bytes.
The overwriter is the shared shim/GRUB/EFI-stub/initrd placement writing into the page the
firmware pointed at. So initrd size, module set and boot device are part of the causal chain.
BIOS-SETTING HYPOTHESIS: DISPROVEN, and the three-way diff cannot support it even in principle.
All 240 attributes read on all three; firmware identical fleet-wide. The discriminating set (both
working nodes agree, broken differs) is seven attributes, of which:
- Fast Boot is the ONLY live candidate, and only as a probability-shifter — HP documents it
'skips some preboot initialization steps', which changes the heap top and hence where the MAT
lands. Different mechanism from the previously-killed MRC-caching theory; that stays dead.
- Legacy Boot Options is AFFIRMATIVELY INERT: 'Configure Legacy Support and Secure Boot' =
'Legacy Support Disable and Secure Boot Enable' on ALL THREE, Option ROM Launch Policy =
'All UEFI' on all three. No CSM is built on node 2 either. A naive diff gets this wrong.
- the rest are coincidence. After Power Loss is ELIMINATED by node 3, which works and shares
node 2's Power Off — though it is now a §1 checklist violation on TWO nodes.
FATAL CONFOUND: the 1=3!=2 partition is perfectly collinear with node 2 being the only
single-DIMM AND the only NVMe-booting node. Adding node 3 did not break the confound, it DOUBLED
it. No setting can be the cause anyway: node 2 booted good once and bad twice with settings held
constant.
EXPERIMENT DESIGN RECORDED because the obvious approach is statistically worthless: p(bad) is
known only to n=3, 95% CI ~0.09-0.99, so even 8 clean boots prove nothing. Elimination needs ~20
boots/arm; an odds shift needs ~50. Zero-touch first step: 20 reboots of node 2 unchanged, logging
MEMATTR address vs outcome — both failures were at exactly 0xbeeb0018 and all five successes
elsewhere, so if outcome maps onto a narrow address range the mechanism is settled without
touching anything.
ROLE BUG FIXED: node 3's WD drive reports a SMART threshold for only 1 of 25 attributes, and
smartctl OMITS the key rather than emitting 0 — summary.txt.j2 dereferenced a.thresh bare in five
places and the report task died, so no artifact was written. Added a thr() macro rendering 'n/a'
(deliberately not 0, which would read as a real ATA cannot-trip threshold). Would have hit any
drive of this class.
Also: three nodes, three different disk models, only one NVMe, and every node has a single disk
with no unpartitioned spare — so §2's identical-layout requirement is unmet on all three.
Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85
|
6e4d311545 |
docs(prod): will 6.12.y get the memattr fix? Researched — not on its own
Operator asked whether waiting is viable or whether a backports kernel is permanent. Answer: it will NOT arrive on its own, but it CAN be made to, and the ask is unusually strong. VERIFIED: - NOT in linux-6.12.y. Newest commit touching memattr.c on that branch is 2024-07-12. In mainline since v6.13 (2025-01-19) — ~19 months, ~97 point releases. - NO stable tag and NO Fixes: tag on either commit. Structural, not an oversight: the unvalidated arithmetic dates to the file's creation in 2016, so it is not a regression and there is no bad commit to point Fixes: at. BOTH automatic feeds — Fixes:-scanning and AUTOSEL — had nothing to grab. AUTOSEL never surfaced it for any branch. - NOBODY has requested it: nothing on [email protected], nothing in all 8,027 open+archived Debian bugs against src:linux. - 6.12 LTS EOL is December 2028 — outlives Debian 13's support window, so not time-barred. - Debian REBASES: trixie carries 6.12.94-1 in trixie-security ('New upstream stable update'), so an upstream pickup would reach us via a normal apt upgrade. THE DECISIVE CARD: the cherry-pick is BYTE-CLEAN. Debian trixie 6.12.94-1's memattr.c hashes to 164203429fa7a..., identical to the upstream patch's pre-image blob — the file has not diverged between v6.12 and the commit's parent. cherry-pick applies with zero conflicts and zero adjustment, which removes the usual reason such requests get declined. The commit also meets every stable-kernel-rules criterion: mainline, 17 insertions, 'hardware quirk', 'a real bug that bothers people'. PLAN: backports now as explicitly provisional; email [email protected] (rules Option 2) CC linux-efi and Ard Biesheuvel who authored AND committed it as EFI maintainer, leading with the clean-apply proof and the three-boot reproduction with byte-identical e820; Debian BTS as secondary under handbook §9.1.7; make the deviation permanent only if there is no movement in 4-6 weeks. Only 8fbe4c49c0cc is needed by hand — 5d0faa8e8369 loosens a heuristic that was rejecting LEGITIMATE tables and is not required for this symptom. Serves: R-ENV-01, R-ENV-03. Task: T-7.1. |
||
|
jakman85
|
0a733294ff |
docs(prod): node 1's DIMMs are not a matched pair
Surfaced by the new node_health DIMM reporting, unrelated to the memattr bug and NOT a fault: DIMM1 is a Micron 8ATF2G64HZ-3G2E1 (DDR4-3200), DIMM3 a Micron 16ATF2G64HZ-2G6E1 (DDR4-2667). Both 16 GB so capacity is right, but different rank organisation and rated speed mean the pair runs at the slower 2667. Recorded in §0 where the symmetry decision gets made, because §2 requires the three nodes to end up with identical layouts and the previous '2x16 GB Micron' note hid the mismatch. Serves: R-ENV-03. Task: T-7.1. |
||
|
jakman85andClaude Opus 5
|
cf33338448 |
feat(node_health): detect the EFI memattr bug on every run, two independent ways
The fault is BOOT-TO-BOOT NON-DETERMINISTIC — the same machine booted correctly two boots
earlier — so it has to be detected on every health run, not diagnosed once. And node 1 is
exposed to the identical lottery on every reboot while currently looking healthy.
firmware.efi_memattr — reads the kernel ring buffer and distinguishes six states:
Failed to map EFI Memory Attributes table @ 0x.. -> FAIL (memory lost NOW)
Unexpected EFI Memory Attributes table version <N> -> WARN (corrupt table still published;
intact only by luck)
[Firmware Bug]: Corrupted EFI ... detected! -> PASS (the >=6.13 fix working)
no memattr line, log reaches early boot -> PASS (no table / accepted / not UEFI)
no memattr line, log does NOT reach early boot -> SKIP
dmesg unreadable -> SKIP
memory.physical_total — deliberately a SEPARATE check, not folded in: it measures the SYMPTOM
with no kernel log at all, so an unreadable-dmesg node is still not unmeasured. Two sub-verdicts,
worst wins: /proc/zoneinfo present-vs-managed (world-readable, works on metal and VMs — sum of
managed IS MemTotal) and MemTotal vs the DIMM total from dmidecode. Thresholds 10% WARN / 25%
FAIL, set from measured baselines: 2.0% prod-1, 2.1% workstation, 2.2% dev VM, vs 83.4% prod-2.
THREE GUARDS AGAINST A FALSE PASS, because "no memattr line" must never silently mean "fine":
1. dmesg failure yields available:false with the exact reason (rc, uid, kernel.dmesg_restrict) -> SKIP
2. the ring buffer WRAPS and every memattr line prints in the first fraction of a second, so the
collector records a boot_covered anchor (Linux version / BIOS-provided physical RAM map); no
memattr line WITHOUT that anchor is a SKIP, not a PASS
3. memory.physical_total needs no kernel log at all
Verified by forcing the unprivileged path: SKIP with the EPERM reason while zoneinfo still PASSed.
VERDICTS, confirmed live:
k3s-prod-1 WARN "EXPOSED, not immune ... survived only because its garbage version field read
1030, which is > 2 ... one unlucky boot from the failure state" + PASS 2.0%
k3s-prod-2 FAIL "THIS NODE HAS LOST MEMORY TO THE BUG ... at 0xbeeb0018" + FAIL 83.4%/83.6%
k3s-dev-* PASS both, NO false finding: "no UEFI config-table line ... did not boot via UEFI.
Log reaches back to early boot, so that absence is measured, not assumed."
Both failure texts carry the remediation verbatim, including that there is NO kernel cmdline
workaround because efi_memattr_init() runs before any efi= bail-out.
Root-gating follows the role's existing pattern: --privileged re-runs THE SAME collector under
become, only when the unprivileged pass actually failed to read something. Reimplementing the log
grep and SMBIOS parse in Jinja would have been a second source of truth for the same fact.
LATENT BUG FIXED in passing: the collector's run() used bare shutil.which(), so dmidecode (in
/usr/sbin) was invisible under the unprivileged k3s PATH and produced a FALSE "not installed"
skip reason. Now shares the sbin-aware which() collect_tools() already had.
INCIDENTAL FINDING recorded in §0b: node 1's DIMMs are NOT a matched pair — DIMM1 Micron
8ATF2G64HZ-3G2E1 (DDR4-3200), DIMM3 Micron 16ATF2G64HZ-2G6E1 (DDR4-2667). Both 16 GB so capacity
is right, but different rank organisation and rated speed, so the pair runs at 2667. Not a fault;
recorded because §2 requires the three nodes to end up symmetric and "2×16 GB Micron" hid it.
Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85
|
47f02c6e06 |
docs(prod): second independent confirmation + no cmdline workaround exists
A second research pass reached the same root cause by a different route and pinned the arithmetic. Recording it because convergence from two independent lines is what turns this from a good hypothesis into a settled diagnosis. MECHANISM TO THE BYTE: - the garbage 'version' node 2 read was 1463964749 = 0x5742544D = ASCII 'MTBW'. The kernel is reading unrelated data. HP publishes the MEMATTR pointer into memory /sys/firmware/memmap calls plain System RAM, so the pre-Linux boot chain overwrites it. - tbl_size is a SIGNED int; num_entries * desc_size wraps mod 2^32 and goes NEGATIVE when bit 31 lands. memblock_reserve sign-extends it and memblock_cap_size clamps to PHYS_ADDR_MAX - base, so EVERYTHING above 0xbeeb0018 is reserved. - memblock_free_all then hands the allocator nothing above that line while present_pages (from memblock.memory) is untouched -> present 3340288 / managed 0. - 'Failed to map EFI Memory Attributes table' is emitted LATER by efi_memattr_apply_permissions(); it is positive PROOF the reservation already happened, a consequence rather than the cause. CLEANEST A/B TELL, one dmesg line: NODE_DATA(0) on prod-2 ends at 0xbeeb0000 — pgdat forced immediately BELOW the reservation — while on prod-1 it ends at 0x82f800000, the top of RAM. Identical 0x2a980 allocation, opposite placement. THERE IS NO KERNEL CMDLINE WORKAROUND. efi_memattr_init() is called from efi_config_parse_tables(), and x86's efi_init() calls efi_config_init() BEFORE the efi_runtime_disabled() bail-out — so noefi / efi=noruntime / efi=novamap do NOT prevent the reservation. A fixed kernel is the only real fix. EFI_MEMORY_SP / soft-reserved was the strongest competing theory and is EXCLUDED BY MEASUREMENT: /sys/firmware/memmap/10/type reads 'System RAM'. Soft-reserved would print as 'Soft Reserved' and is never memblock_add()ed, which would have left present at ~0 rather than 3340288. Also records that the hardening landed exactly one release late for Debian 13 (bug present since 2016; 8fbe4c49c0cc first in v6.13; follow-up 5d0faa8e8369) and that a Debian BTS report against src:linux is warranted — 6.12 is Debian 13's kernel for the release lifetime. Serves: R-ENV-01, R-ENV-03. Task: T-7.1. |
||
|
jakman85
|
49d82491dd |
docs(prod): node 2 memory SOLVED — Linux 6.12 EFI memattr bug, NOT a hardware fault
I framed this as an OPEN HARDWARE FAULT and blamed, in order: the DIMM, BIOS memory remapping,
and stale Fast Boot memory-training data. ALL THREE WERE WRONG. The hardware is fine.
ROOT CAUSE: HP's R21 firmware publishes a CORRUPT EFI Memory Attributes Table, and Linux 6.12
fails to validate it before reserving memory from its contents. drivers/firmware/efi/memattr.c
never validates num_entries or desc_size:
if (tbl->version > 2) { pr_warn(...); goto unmap; }
tbl_size = sizeof(*tbl) + tbl->num_entries * tbl->desc_size;
memblock_reserve(efi_mem_attr_table, tbl_size);
Garbage version > 2 -> kernel bails, memory intact. Garbage version <= 2 -> kernel multiplies two
garbage fields and reserves GIGABYTES from the table address to the top of RAM.
THE EVIDENCE THAT SETTLES IT: the same machine — same DIMM, same BIOS, same kernel, same cmdline —
booted with the FULL 15.9 GiB two boots earlier, with a byte-identical BIOS-e820 map. Boot -2:
15,917,744K available, 615,208K reserved, 'Unexpected EFI Memory Attributes table version
1463964749'. Boots -1/0: 2.6 GiB, 13.9 GiB reserved, 'Failed to map ... @ 0xbeeb0018'. No hardware
fault is intermittent in that shape or fixed by a reboot. MemTotal = DMA(3840) + DMA32(684481)
pages = 2,753,284 kB exactly, Normal contributing zero.
BOTH PROD NODES ARE AFFECTED. k3s-prod-1 prints 'Unexpected EFI Memory Attributes table version
1030' — also garbage, but >2, so the kernel bails and it keeps its 32 GiB BY LUCK, NOT IMMUNITY.
It is one unlucky boot from the identical fault. Dev nodes publish no memattr table at all
(Proxmox OVMF) and are unaffected.
FIX: kernel >= 6.13 — upstream 8fbe4c49c0cc 'efi/memattr: Ignore table if the size is clearly
bogus', whose commit message describes this verbatim. Verified absent from v6.12 and from current
linux-6.12.y. On Debian 13: linux-image-amd64 from trixie-backports.
DO NOT buy RAM, move the DIMM, RMA the board, or run memtest expecting failures. Rebooting,
disabling Fast Boot, clearing CMOS and unplugging USB are all the same lottery — they move where
the table lands and may LOOK like fixes.
Records every killed hypothesis with its disproof so none of them get re-run.
Serves: R-ENV-01, R-ENV-03. Task: T-7.1.
|
||
|
jakman85
|
5ea9c93f58 |
docs(prod): BIOS password answer + CORRECT the 0x4 'auth failure' reading
Operator asked for the default BIOS password. ANSWER: there is none, and no factory default exists to look up. HP business desktops ship with no Setup and no Power-On password; the is_enabled=0 we measured IS the factory state. Recovery for a forgotten password is the PSWD jumper — and 'stringent security' must NOT be enabled, since HP documents that it removes the physical bypass and turns a 5-minute fix into a signed per-serial HP service case. CORRECTION, and the kernel driver misled me: I recorded that the firmware refused our write for lack of BIOS-admin authentication. Wrong. hp-bioscfg passes the return of HP's WMI SetBIOSSetting through hp_wmi_error_and_message(), which is THE ERROR TABLE FOR A DIFFERENT INTERFACE (the "SECU" commands under HP_WMI_BIOS_GUID). Per HP's Client Management Interface white paper, SetBIOSSetting 4 = 'Failed' (generic) and 6 = 'Access Denied'. We got 4, not 6 — and the driver has a separate BIOS_ADMIN_NOT_SET = 0x38 -> -EPERM for the no-password case, which we also did not get. 'Invalid command type' is the driver looking 4 up in the wrong table: a genuine upstream bug worth reporting to platform-driver-x86. Most likely real cause (inferred, testable): with an empty password the driver emits an UNTAGGED empty UTF-16 string, while every HP-documented caller passes Password="<utf-16/>" even when blank — the driver only prepends that prefix for a non-empty string. Setting a password may fix writes, but by flipping onto the tagged code path, NOT because 0x4 meant unauthorised. Also verified from source: hp-bioscfg CANNOT set a password. current_password and new_password both funnel into store_password_instance(), which only strscpy's into driver RAM; new_password is write-only into a buffer nothing ever reads. Count corrected: 241 attributes exposed, 239 with a current_value, 215 non-empty. I had quoted 215 as the total; it was the non-empty subset. Recommendation recorded: land read-only drift detection now (zero risk, no new credential, and it already caught real drift); treat a Setup password as a deliberate experiment on node 3 only; hp-repsetup as fallback but it needs Secure Boot off or a self-signed module; skip Sure Admin/BCU at this scale. Serves: R-ENV-03, R-SEC-01. Task: T-7.1. |
||
|
jakman85
|
3a88abb958 |
docs(prod): iGPU ruled out as the cause; CORRECT my last_pfn misreading
Operator asked whether node 2's missing memory could be reserved for integrated graphics. TESTED, and no: /proc/iomem shows 'c8800000-cc7fffff : Graphics Stolen Memory' = EXACTLY 64 MiB, matching the BIOS Video Memory Size=64 MB setting, and it sits BELOW the 4 GiB line. The UHD 630 (i915 bound) takes precisely what it is configured to take. It cannot account for 13.3 GiB and is not even in the affected address range. CORRECTION to my own analysis, prompted by checking that hypothesis properly: I cited the 'last_pfn revised DOWN from 0x42f800 to 0xc3c10' line as evidence of a memory cap. That was a MISREADING. x86 boot prints last_pfn twice — first max_pfn from the full e820, then max_low_pfn, the end of RAM below 4 GiB. The e820 low usable region ends at 0xc3c0ffff, so the next PFN is exactly 0xc3c10, which is the second value observed. It matches to the byte and is normal on any machine with more than 4 GiB. I presented a routine boot message as a smoking gun, and the 'memory remapping above 4 GB not being applied' signature claim rested partly on it. The real anomaly is narrower than first stated: the Normal zone is 'present' but 'managed 0', 13.3 GiB is reserved, and nothing in the boot log explains by what. Ruled out so far: DIMM health, firmware inventory, memory-remap settings (none exist on this platform), MTRR trim, iomem registration, and now integrated graphics. Leading remaining hypothesis is stale memory-training data cached by Fast Boot — testable with one F10 toggle. Serves: R-ENV-03. Task: T-7.1. |
||
|
jakman85
|
9f7052ab2b |
docs(prod): fill in node-2 inventory; drop stale investigate bullets
Records node 2's as-found hardware alongside node 1 and removes two bullets that my earlier edit left duplicated/obsolete (the BIOS comparison is done; clear-CMOS was listed twice). Makes the hardware ASYMMETRY explicit in the table rather than buried in prose: node 1 is a ~2011 Micron C400 SATA SSD at 69% life with 15 grown bad blocks and 187 erase failures; node 2 is a Samsung PM9A1 NVMe at 96% life, PASSED. Same chassis, same BIOS, same CPU, same partition layout — very different storage. §2 requires an identical disk layout across all three nodes because Longhorn replica placement assumes symmetry, so this belongs in the inventory where the disk decision will be made. Serves: R-ENV-03. Task: T-7.1. |
||
|
jakman85
|
3c8f809aaf |
docs(prod): CORRECTION — BIOS attributes are read-only in practice, not writable
I recorded that the hp-bioscfg attributes were 'writable from the OS' because current_value has mode rw and no BIOS passwords are set. That was WRONG, and the firmware said so when the write was actually attempted: hp_bioscfg: Returned error 0x4, "Invalid command type" (surfaces as printf: I/O error) HP requires BIOS-admin authentication for programmatic changes and neither credential is provisioned on these nodes — Setup Password is_enabled=0 (role bios-admin), SPM key_mechanism=not provisioned. Secure Boot is also enabled. File mode is not authority; the firmware is. What survives the correction, and it is the useful half: READS WORK. Diffing 215 attributes between node 1 and node 2 is exactly how the drift in §0b was found — nobody spots 'After Power Loss=Power Off' by eye. So the §1 firmware checklist IS machine-CHECKABLE today with reads alone, and node_health should assert it. What does NOT survive: machine-ENFORCEABLE. Converging BIOS settings automatically would require provisioning a BIOS Setup Password on every node — a stored credential that can rewrite firmware. That is a real security decision and the operator's call, not something to default to. Consequence for the node-2 memory fault: setting Fast Boot=Disable and After Power Loss=Power On must be done in F10 on the box. Updated §0b and §1 accordingly. Serves: R-ENV-03. Task: T-7.1. |
||
|
jakman85
|
11b21f7347 |
docs(prod): BIOS settings compared programmatically via hp-bioscfg
Both nodes load the hp-bioscfg driver, exposing 215 BIOS attributes read/write at /sys/class/firmware-attributes/. No BIOS passwords are set, so they are writable FROM THE OS. RULED OUT as the cause of node 2's 2.6 GiB limit: firmware inventory is correct (BIOS itself reports Memory Size=16384 MB); no memory-remap/above-4G attribute exists on this platform at all; MTRRs are effectively identical with no trim warning; /proc/iomem registers the full 100000000-42f7fffff as System RAM and 127 memory blocks report online. REAL DIFFERENCES FOUND (node1 works / node2 broken): After Power Loss Power On / Power Off <- VIOLATES this document's own §1 Fast Boot Disable / Enable <- leading hypothesis Legacy Boot Options Disable / Enable TPM Activation Policy F1 to Boot / Allow user to reject MS+Windows UEFI CA 2023 Enable / Disable Leading hypothesis: HP Fast Boot reuses CACHED MEMORY-TRAINING DATA rather than retraining each boot. A DIMM fitted while Fast Boot was on is a plausible route to memory being present, inventoried, handed to the kernel — and then reserved. It is also the only memory-relevant setting differing from the working node. NOT TESTED: changing it is a firmware write plus a reboot and needs operator sanction. BROADER FINDING for T-7.1: the §1 firmware checklist is now MACHINE-CHECKABLE rather than an eyeball-at-F10 exercise, and machine-ENFORCEABLE too. That is precisely how node 2 came to have After Power Loss=Power Off while node 1 has Power On — nobody could see the drift. Follow-up recorded: extend node_health to assert the firmware checklist, add a role to converge it. Raw BIOS dumps deliberately NOT committed — they carry serials, UUIDs and MAC addresses, same reason node-health artifacts are gitignored. Only the findings are recorded. Serves: R-ENV-03. Task: T-7.1. |
||
|
jakman85andClaude Opus 5
|
cb69a53825 |
feat(prod): enable node 2 + HARDWARE FAULT — it addresses 2.6 GiB of its 16 GB
k3s-prod-2 (192.168.2.11) came online 2026-09-01. Enabled in the inventory, baseline package
floor installed, first health run completed. Vaulted become works on it unchanged.
*** OPEN FAULT: the node can only address ~2.6 GiB of a healthy, correctly-installed 16 GB
DIMM. *** Recorded in PROD-PROVISIONING §0b.
dmidecode 16 GB Samsung M471A2K43DB1-CTD in DIMM1 @ 2667 MT/s (DIMM3 empty)
BIOS e820 reports ~16 GiB usable, incl. 0x100000000-0x42f7fffff
kernel Memory: 2568544K/16546088K available (13958980K RESERVED)
last_pfn revised DOWN 0x42f800 (~16.7 GiB) -> 0xc3c10 (~3.06 GiB) in early boot
zones Normal present 3340288 pages (~12.7 GiB) but MANAGED 0
MemTotal 2,753,284 kB = 2.6 GiB
The DIMM is NOT the problem: it is present, correctly identified, and at rated speed. The
kernel sees all 16 GB and then reserves 13.3 GiB. Everything below ~3 GiB usable, everything
above 4 GiB present-but-unmanaged — the signature of memory remapping above 4 GB not being
applied, not of failing memory. Node 1 is the control: identical chassis, identical BIOS R21
Ver. 02.26.00, identical CPU, and it addresses its full 31 GiB. Per-machine state, not a
firmware-version defect.
WARNING RECORDED: do NOT assume the incoming 32 GB fixes this. If the fault is the slot, the
board or a BIOS setting rather than the DIMM, 32 GB presents identically and the money is
wasted. Verify MemTotal after any RAM change.
ALSO FOUND — the prod nodes are NOT hardware-symmetric:
node 1 Micron C400 512GB SATA SSD (~2011) — 69% life, 15 grown bad blocks, 187 erase
failures, 32 unexpected power losses
node 2 Samsung PM9A1 512GB NVMe — 96% life, 2142 power-on hours, 34 unexpected
power losses, overall-health PASSED
Same chassis, same BIOS, same CPU, same partition layout — very different storage. §2 requires
an identical disk layout across all three nodes because Longhorn replica placement assumes
symmetry, so this asymmetry is now a live input to the disk decision rather than a footnote.
Node 2's NVMe is the drive node 1 should have had.
Node 2 otherwise: 0 FAIL / 3 WARN / 12 PASS / 2 SKIP. Same three WARNs as node 1 (swap still
active pending full baseline, single IPv4 resolver, and the NVMe's 34 unexpected power losses),
plus the same unknowable-ECC SKIP.
Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
6b4da27f84 |
feat(prod): SOPS decryption (same age key as dev) + GitOps source on main
Operator decisions, 2026-09-01:
1. Prod uses the SAME SOPS age key as dev.
2. Prod tracks the SAME BRANCH as dev (`main`). Dev is a temporary instance and will be
decommissioned once prod is deemed stable.
SOPS: eight `decryption: {provider: sops, secretRef: sops-age}` patches mirroring dev
exactly — platform, apps-wave-0/1/2, oauth2-proxy, headlamp, crafty, velero. The sops-age
Secret is still created OUT OF BAND on prod (bootstrap/sops/install-age-key.sh); the
overlay references it by name only, never inlines key material.
RECORDED CONSEQUENCE of the shared branch: once prod is live, EVERY MERGE TO main
RECONCILES BOTH CLUSTERS. There is no promotion step and no soak window — dev and prod
move together. That is a deliberate consequence of dev being temporary, not an oversight,
but it removes the "prove it on dev first" safety net that the last few days have leaned
on heavily. Worth revisiting if dev outlives its expected life.
*** BOOTSTRAP BLOCKER recorded, NOT papered over *** — the source URL cannot work on a
cold prod cluster, for two independent reasons found while wiring this:
1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER — the one slated for
decommission. Prod syncing from it means prod's source of truth lives inside the
cluster being retired. Gitea must move to prod (or off-cluster) BEFORE dev is torn
down. This makes Gitea migration the FIRST task of the decommission plan, not the last.
2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml
answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP} via a template
plugin. On prod that is 192.168.2.20 — PROD's own gateway, where no Gitea listens. Prod
Flux would resolve gitea.home.keithjackson.dev to itself and never reach a repository.
This is the same split-horizon mechanism that makes dev work, inverted against us.
Bring-up therefore needs a TEMPORARY source (the bootstrap host-Gitea, or an off-cluster
mirror) with a cutover to the in-cluster URL once Gitea actually runs on prod — precisely
the sequence docs/05 describes for dev's own Phase-6 cutover. The branch is settled and
correct; only the host is blocked. Documented at the patch site and in
PROD-PROVISIONING.md §7.
Gate: MERGE GATE: PASS.
Serves: R-SEC-01, R-GITOPS-01, R-ENV-02, R-ENV-03. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
666a60776a |
fix(flux): close the substitution race — platform dependsOn root + strict substitution (P0/P1)
P0 — clusters/base/platform.yaml gains `dependsOn: [flux-system]`. This is the actual fix for the incident, not the feature gate. Verified against kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies` (our version): dependsOn is REVISION- gated, not merely Ready-gated — when the dependency shares a GitRepository, the dependent cannot reconcile revision N until the dependency has APPLIED revision N. Applying revision N is exactly what writes a newly-added cluster-vars knob. Graph verified mechanically, not by eye: built clusters/dev and clusters/prod and walked the resulting objects. `platform` was the ONLY Kustomization without a dependsOn, and all 11 substituting Kustomizations chain transitively through it (platform-config -> platform; the waves, apps-vms, oauth2-proxy, headlamp, homepage, crafty, velero -> platform-config). The "not chaining through platform" set is EMPTY, so this one edge protects the whole graph. DFS over the full edge set after the edit: no cycles. The root (clusters/dev/ sync.yaml) runs wait: false, which is what makes the edge deadlock-free — it reaches Ready on apply success without health-gating its children. Accepted trade-off, documented in-file: the root becomes a gate. A malformed manifest anywhere under clusters/<env> — including clusters/base — now stalls platform and everything downstream, where today it would not. Fail-closed, per doctrine. P1 — StrictPostBuildSubstitutions. MY PREMISE WAS WRONG about where this belongs and it matters: this repo deliberately does NOT use `flux bootstrap` (see bootstrap/flux/README). There is no gotk-components.yaml in git, and the live kustomize-controller Deployment carries NO Flux ownership labels — it was installed imperatively by `flux install` from bootstrap.sh and is not in Flux's inventory. `flux-system` self-manages the CR graph, NOT the controllers. Nothing under clusters/** can reach a controller flag. So it uses Flux's documented mechanism for hand-installed clusters: a new bootstrap/flux/components/kustomization.yaml wrapping `gotk-components.yaml` with a JSON-6902 patch appending --feature-gates=StrictPostBuildSubstitutions=true to kustomize-controller. JSON-6902 `add /args/-` deliberately, NOT strategic merge — a container's args is atomic and a merge patch would replace all five install-generated args. bootstrap.sh step 1 now renders (`flux install --export` into a temp dir, apply the patch, ASSERT the gate string is present so a future Flux bump that breaks the patch target fails loudly) then applies. FLUX_INSTALL_PLAIN=1 is a loud escape hatch. Nothing is vendored, so components can never drift from the pinned FLUX_VERSION. Rejected: vendoring gotk-components.yaml under the root's path (the root's prune: true would put Flux's own controllers and CRDs in a prune inventory), and shipping a partial Deployment as a Flux-managed resource (a future prune would delete kustomize-controller). ROLLOUT ORDER MATTERS. P0 arrives automatically on the next reconcile. P1 does NOT arrive on a git push — the controller Deployment is outside Flux's inventory, so an operator must re-run bootstrap.sh step 1, which restarts kustomize-controller (~10-20s; Flux is level-triggered so an interrupted reconcile is simply retried). P0 must be reconciled BEFORE P1 is rolled out: with the gate on but the ordering edge absent, the same race turns a transient empty render into a transient hard Kustomization failure. ALSO FIXES A PHASE-7 BLOCKER THIS CHANGE WOULD OTHERWISE HAVE CREATED. clusters/prod had no root Kustomization — dev/sync.yaml even carried the note "a prod overlay needs its own sync.yaml -> ./clusters/prod". With the new dependsOn, a prod bootstrap would have left platform blocked forever on a flux-system Kustomization that does not exist, and nothing downstream would ever reconcile. clusters/prod/sync.yaml is now authored, mirroring dev. Two further prod-overlay gaps found and DOCUMENTED, not guessed (PROD-PROVISIONING §7): prod has no SOPS `decryption` patch (every encrypted Secret would fail to decrypt — needs the prod age-key decision, a security choice not a copy), and no GitRepository source URL patch (needs a decision on whether prod tracks the same branch as dev, which would make a dev-facing merge reconcile prod). Substitution gate re-run: check OK, 102 assertions, 0 hazards. Gate: MERGE GATE: PASS. Serves: R-ENV-02, R-GITOPS-01, R-PLAT-01, R-ENV-03. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
fdc4a7d9f4 |
fix(cilium): maxUnavailable 2 -> 1 — bound CNI rollout blast radius (P2)
During the substitution incident a bad render (k8sServiceHost: "") took down TWO of three
Cilium agents simultaneously. The agents' `config` init container falls back to the API
ClusterIP, which only the Cilium agent itself can translate — and that agent is the one
being replaced — so it deadlocks on `dial tcp 10.43.0.1:443: i/o timeout`.
Root cause of the 2-at-once: `updateStrategy.rollingUpdate.maxUnavailable: 2` is the
UPSTREAM CHART DEFAULT, confirmed by downloading cilium-1.20.1.tgz and reading
values.yaml:437-444 — not a local choice, and not overridden anywhere in this repo.
maxUnavailable: 1 makes a rollout replace ONE agent at a time and stop at the first pod
that fails to become Ready. The whole value of the setting is that the rollout halts on
the first bad node instead of taking the majority of the dataplane with it, and that
guarantee only exists at 1. It matters far more on a cold PROD bootstrap, where there is
no healthy peer to recover from — dev only self-healed because k3s-dev-3 kept a working
agent throughout.
Set as a [parity] field in ALL THREE places, per this repo's documented contract that the
Ansible bootstrap values and the Flux HelmRelease values must agree on settings that
define cluster networking (Ansible installs Cilium at bootstrap; the HelmRelease adopts
it; the two layers fight otherwise):
platform/cilium/helmrelease.yaml (authoritative for Flux)
infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 (authoritative for bootstrap)
platform/cilium/values.yaml (reference mirror, kept in sync)
Verified by rendering the .j2 with roles/cilium/defaults and diffing full value sets: every
[parity] field is byte-equal across all three; only k8sServiceHost differs, as designed
(${K3S_API_HOST} vs {{ k3s_api_host }}).
MERGING THIS IS NON-DISRUPTIVE. updateStrategy lives at DaemonSet.spec.updateStrategy — a
SIBLING of spec.template (chart daemonset.yaml:49-53). The DaemonSet controller decides a
pod is out of date by hashing spec.template alone, so the pod-template hash does not move,
no new ControllerRevision is created, and zero pods are recreated. Helm patches the field
in place. (If it were disruptive it would be self-defeating: rolling agents under the old
maxUnavailable: 2 in order to install maxUnavailable: 1.) Verify after reconcile with
`kubectl -n kube-system get ds cilium -o jsonpath='{.spec.updateStrategy.rollingUpdate.maxUnavailable}'`
plus unchanged pod ages — that is the cleanest proof nothing rolled.
Trade-off recorded in the comments: upgrades serialise, roughly two waves to three at 3
nodes (a minute or two). On a large cluster this becomes painful and a percentage such as
"10%" would be right — noted so a future larger cluster revisits it deliberately.
SCOPE: this bounds blast radius, it does NOT close the substitution race. A bad render
still ships; it now breaks one node and stalls rather than taking the majority.
platform.dependsOn: [flux-system] (P0) is the fix for the cause.
Serves: R-PLAT-01, R-NET-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
ff0533106b |
feat(prod): operator decisions — node DNS, service VIPs, become vault, collections path
1. PROD NODE DNS = 8.8.8.8 + 1.1.1.1 (operator, 2026-09-01). Two INDEPENDENT public
upstreams, so the single-upstream SPOF that broke containerd pulls and CoreDNS
forwarding on dev cannot recur. Prod deliberately differs from dev, which is
PiHole-first because dev needs the local jackson.lan zone.
SCOPE CORRECTED IN THE COMMENTS. Three resolvers answer *.home.keithjackson.dev and
only one is configured here:
pods -> CoreDNS -> platform/coredns/coredns-custom.yaml, whose `template` plugin
answers the WHOLE zone with ${GATEWAY_VIP}. Self-contained in-cluster;
PiHole is NOT in this path. Prod is already correct (.20).
LAN -> PiHole. Not a cluster concern.
nodes -> THIS FILE.
The one thing public node DNS breaks: the k3s API SERVER is a node process carrying
oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab and reaches the
issuer through the NODE resolver, so prod would get the PUBLIC record instead of the
prod Gateway VIP — breaking kubectl OIDC SSO. Dev avoids this only because its node
resolver IS PiHole. Not blocking (prod has no cluster or Keycloak yet); options are
recorded in PROD-PROVISIONING.md §4 for the operator to pick.
2. SERVICE VIPs DECIDED: UNIFI_L4_VIP 192.168.2.21, CRAFTY_L4_VIP 192.168.2.22 — the
first two addresses after GATEWAY_VIP (.20), mirroring dev's .53/.54/.55 ordering
within the .20-.30 pool. S3_BACKUP_ENDPOINT stays TBD-PROD-* : no prod backup target
exists, and dev's versitygw is across the pfSense boundary on the dev host, so it is
not DR for prod.
3. PROD BECOME VAULT committed, ansible-vault encrypted, matching the hypervisor and
proxmox_hosts pattern. Verified working: `ansible k3s_prod -m command -a 'id -un'
--become` returns root. The L0 no-plaintext-secrets check asserts every ansible
vault.yml starts with $ANSIBLE_VAULT, so this stays honest by gate.
4. TOOLCHAIN GAP FIXED: the devcontainer image does NOT ship the Galaxy collections, so
EVERY ansible play died at parse time on community.general.modprobe. ansible.cfg now
sets collections_path = collections, and the collections are installed there from the
existing collections/requirements.yml (gitignored — pinned by requirements, fetched at
setup, vendored bytes are not source).
5. DRIVE HEALTH MEASURED and recorded in §2 with the raw attribute table so the numbers
are auditable. Verdict: usable, but not for production etcd/Longhorn state — 31% of
rated life consumed, 15 grown bad blocks, 187 erase failures, and 32 unexpected power
losses on a consumer drive with NO power-loss protection, in a single failure domain.
etcd is the worst possible tenant: it fsyncs constantly and unclean power loss without
PLP is exactly how a member corrupts.
6. NON-ECC RAM: RISK ACCEPTED by the operator, recorded as a CLOSED decision, not an open
item — do not re-litigate. Rationale: 3-node k3s homelab, not enterprise; corruption on
one node is outweighed by the other two and causes node ejection. That holds for the
dominant mode (etcd raft CRC + quorum; Longhorn 3 replicas + hard anti-affinity +
rebuild). Recorded residual: corruption introduced UPSTREAM of replication replicates
faithfully to all three replicas, so quorum sees no disagreement — low probability, and
the only mitigation is different hardware. Follow-up noted: Longhorn's
snapshotDataIntegrity is left at the chart default rather than set explicitly, and it
is the mechanism that operationalises the operator's own argument.
Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03, R-SEC-01. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
0b3db6e778 |
fix(node_health): SMART verdict was wrong by 4096x — judge on portable signals
The first real SMART run FAILED k3s-prod-1's drive on reallocated=61440. That verdict
was WRONG, and the failure mode is instructive: the check trusted a VENDOR-DEFINED raw
value as if it were an ATA-standard sector count.
Micron TN-FD-20 ("SMART Attributes for the C400 SSD") names attribute 5 "Reallocated
BLOCK Count" and defines its raw as grown-bad-blocks x Block_Sector_Count, where
Block_Sector_Count is 256 x 16 = 4096 for the 128Gb-NAND parts (256GB/512GB). This is
the 512GB C400-MTFDDAC512MAM: 61440 / 4096 = 15. Triply corroborated on the drive itself
— attr 170 Grown_Failing_Block_Ct = 15, attr 196 Reallocated_Event_Count = 15, attr 5
normalised value 100 against thresh 10, and `when_failed` empty on every attribute.
FIFTEEN grown bad blocks, not sixty-one thousand bad sectors.
smartctl(8) states it outright: since ATA/ATAPI-4 rev 4 the meaning of raw Attribute
fields is "entirely vendor-specific". So a raw value can never be a portable verdict.
FAIL now comes only from signals that mean the same thing on every drive: the drive's own
smart_status.passed; NORMALISED value vs the MANUFACTURER's own threshold via smartctl's
when_failed ("now"/"past"/""), split Pre-fail -> FAIL and Old_age-or-past -> WARN; the
ATA-standard counters 197/198/187 (199 CRC is WARN-only — it is link, not media); and for
NVMe critical_warning / media_errors / percentage_used. Wear reads from the NORMALISED
lifetime indicator (202/231/233, or NVMe percentage_used) — a percentage by definition.
Vendor raw counts are still collected and PRINTED WITH THEIR NUMBERS, and may raise a
WARN, but can never drive a FAIL. Attribute 5's raw is now shown beside the normalised
reading that contradicts it, labelled vendor-encoded, with a mechanically-derived
"(= 15 x 4096)". Endurance attributes are matched by THE NAME smartctl's drivedb resolved
for that model, not by bare ID — drivedb is the portability layer; on an unrecognised
drive the name is Unknown_Attribute, nothing matches, and the field stays null with no
claim made. The STORAGE section now prints 170/172/173/174/189/206, whose previous
omission is exactly what let the misread stand unchallenged.
Also documented from TN-FD-20: on this family attr 197 "will always be 0, as reallocation
will be done on-the-fly" — so pending=0 is NOT evidence of health here. The role reports
the number and infers nothing from it.
Branches real hardware cannot exercise were verified with a harness rendering the actual
role expressions against synthetic smartctl payloads: virtio/no-stdout -> SKIP; NVMe
healthy -> PASS; NVMe degraded -> FAIL; a genuinely dying ATA drive (attr 5 normalised 8
<= thresh 10, pending 48) -> FAIL driven by the threshold trip and standard counters, not
the raw; unknown-vendor attrs -> PASS with the unnamed counter ignored; failed-in-the-past
-> WARN.
NEW VERDICTS. k3s-prod-1: 0 FAIL / 4 WARN / 12 PASS / 1 SKIP — SMART is a WARN naming the
real findings (31% of rated life consumed, 15 grown bad blocks, 187 erase failures, 32
unexpected power losses on a drive with no power-loss protection). k3s-dev-1: unchanged,
all three virtio disks still SKIP with their stated reason.
BREAKING (JSON artifact): the fact `reallocated_sectors` is renamed and gone — that name
asserted a unit ATA does not guarantee. Replaced by realloc_attr (the whole attribute,
normalised + raw), realloc_events, and grown_bad_blocks. Nothing in the repo reads it.
Serves: R-ENV-03, R-OBS-04.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
c1f24e4a6d |
fix(dns): resilient node resolvers — mechanism-detecting, per-environment (R-DNS-01/03)
Re-authors the fix stranded on origin/fix/node-dns-resilient-upstream (
|
||
|
jakman85andClaude Opus 5
|
e623d27e2d |
feat(ansible): parameterise target groups so prod reuses the roles unchanged (R-ENV-02)
roles/k3s_server, roles/cilium and every node-host playbook hardcoded the dev group
names k3s_primary/k3s_secondary/k3s_servers, so the prod inventory groups could not be
selected at all. That broke the R-ENV-02 promise that prod reuses the SAME roles, and
blocked PROD-PROVISIONING.md §6 outright.
TWO KNOBS, deliberately separate — Ansible templates a play's `hosts:` BEFORE any host
context exists, so group_vars is unreadable there, while roles run WITH host context:
* which hosts a play RUNS ON — k3s_hosts, k3s_primary_hosts/k3s_secondary_hosts
(k3s.yml's split), vpn_egress_gateway_hosts; each `| default(<dev group>)`, set by -e
only. Follows node-health.yml's established `<name>_hosts | default(dev)` pattern.
* which group a ROLE means by "my primary"/"my servers" — k3s_primary_group,
k3s_servers_group in group_vars/all.yml (dev), overridden in group_vars/k3s_prod.
Follows the k3s_api_host convention already in this repo. No per-run flag.
DEV IS UNCHANGED — the key evidence. --list-hosts/--list-tasks/--syntax-check captured
for all 9 playbooks at HEAD and after: sorted host lists IDENTICAL for all 9, same
primary/secondary split, same 6-play site.yml. (Raw --list-hosts PRINT ORDER varies run
to run; proven to be Ansible's own set ordering by running UNMODIFIED HEAD cilium.yml
four times -> 3,1,2 / 1,3,2 / 1,2,3 / 1,2,3. Hence sorted comparison.) --list-tasks
differs in exactly two files, only by added TAGS annotations and one new assert; no task
reordered, renamed or removed. Var resolution verified per environment with -c local:
dev -> k3s_primary/192.168.1.50/sans .50,.49,127.0.0.1,.51,.52; prod ->
k3s_prod_primary/192.168.2.10/vip .9/sans .10,.9,127.0.0.1,.11,.12.
BREAKS THE SMARTMONTOOLS CIRCULAR DEPENDENCY. node_baseline tasks are now tagged
packages/system/disks, so the package install can run alone:
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
Skipping is the guard — no disks-tagged task is selected, so the
BLOCKED-pending-disk-decision paths are never templated, let alone passed to mkfs.
Belt-and-braces, a new disks-tagged assert fires first in a FULL run and names §2
explicitly instead of failing obscurely at mkfs. `tags: never` was deliberately NOT used
(it would change the default run).
FOOTGUN FOUND AND FIXED: roles/cilium, cluster_verify and flux_bootstrap all hardcoded
.cache/dev.kubeconfig — the exact file infra/devcontainer/dev.sh exports as $KUBECONFIG.
Once prod became targetable, a prod cilium/verify run would have SILENTLY OVERWRITTEN it,
leaving the operator's kubectl quietly answering from PROD. Now cluster_kubeconfig_name
(dev.kubeconfig -> prod.kubeconfig). Dev path byte-identical.
KNOWN, DOCUMENTED, NOT FIXED HERE: flux-bootstrap.yml retargets but roles/flux_bootstrap
shells out to bootstrap/flux/bootstrap.sh with a hardcoded --path=clusters/dev — a
bootstrap/ subtree change, not a target-group change (§6, cross-referenced to §7).
vpn.egress.gateway.yml is parameterised but prod has no egress group yet and would need
its own vaulted PIA credentials. --check against prod is still impossible until the prod
vault exists — even the packages-only run needs it.
Nothing was run against the dev cluster: --list-hosts/--list-tasks/--syntax-check, plus
debug/assert with -c local, only.
Serves: R-ENV-01, R-ENV-02, R-ENV-03. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
9f59c36000 |
feat(ansible): node_health role — read-only hardware health for prod + dev nodes
Operator ask: implement the roles needed to check prod node hardware health, and run them to find out our state. This is that role; the run's findings are below. roles/node_health is a read-only sibling to cluster_verify (cluster health vs hardware health), styled after node_baseline. One unprivileged collector script does the work in a single SSH round trip — reading /sys/class/dmi, thermal + hwmon, EDAC, cpufreq + thermal_throttle, /proc/pressure, /sys/block incl. SCSI ioerr/iotmo counters, lsblk, lscpu, mounts + statvfs, per-NIC stats, resolv.conf. THE DESIGN POINT: an unreadable field is emitted as null WITH A RECORDED REASON, never as zero. The summary ends with a "NOT READ (and why) — these are UNKNOWNS, not passes" section. A fake zero on a reallocated-sector count is exactly the kind of false green this project refuses. 22 checks report PASS/WARN/FAIL/SKIP/INFO; every SKIP says why. Report-only by default; thresholds live in defaults/main.yml. Portable across bare metal AND VMs — validated against k3s-prod-1 (bare metal, no root) and k3s-dev-1 (VM, root). VM-absent things (SMART on virtio, coretemp, EDAC) skip cleanly, and iSCSI/Longhorn volumes are split out of the physical-disk set so a live replica is never mistaken for a spare device. playbooks/node-health.yml takes its target group as a variable (-e node_health_hosts= k3s_prod), deliberately unlike the other playbooks that hardcode k3s_servers — see the group-name gap already flagged in hosts.ini. Two become facts found the hard way and documented in the role: a group_vars `ansible_become: true` OUTRANKS `become: false` as a play/task keyword, so unprivileged tasks must override the variable; and `failed_when: false` does NOT catch "Missing sudo password" — only `ignore_errors` does. FIRST RUN, k3s-prod-1, 2026-08-31: 0 FAIL · 3 WARN · 12 PASS · 3 SKIP · 4 INFO. Box is healthy, cool and unstressed: i7-9700T 8C/8T, VT-x + /dev/kvm (KubeVirt OK), zero throttle events, hottest sensor 33C, PSI 0.00 across the board, 31.1 GiB RAM, eno1 up at 1000 Mb/s, BIOS R21 02.26.00, no reboot pending. DRIVE HEALTH IS STILL UNKNOWN — the headline. SMART did not run, blocked TWICE over: no vaulted ansible_become_password for prod, AND smartmontools is not on the node at all (node_baseline has never run there). Power-on hours, wear, reallocated/pending sectors and drive temperature all remain unmeasured on a ~2011 consumer SSD. WARNs: kernel ioerr_cnt=6 on sda (read cautiously — four freshly attached iSCSI disks on dev each showed a flat 7, suggesting a fixed per-device discovery cost, not media trouble; attribution needs dmesg/root); 0.95 GiB swap still active (node_baseline does the swapoff); one IPv4 resolver only (the other two upstreams are IPv6 Google), which refines rather than resolves the DNS-SPOF finding. ESCALATION — ECC status is UNKNOWABLE on this hardware: /sys/devices/system/edac/mc exists but no controller is bound. Non-ECC consumer RAM. Silent memory corruption on a prod node would be undetectable. Recorded as a SKIP with reason, not a pass. Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a1fb31a7ef |
docs(prod): record the decided prod address plan; packages-in-role correction
PROD-PROVISIONING.md §4 now carries the full operator-decided address table (nodes .10/.11/.12, control-plane VIP .9, ingress Gateway VIP .20, Cilium LB pool .20-.30) with the two invariants that make it safe: the control-plane VIP sits outside the LB pool so kube-vip and LB-IPAM cannot contend, and GATEWAY_VIP is the FIRST address of the pool because the shared Gateway pins nothing and LB-IPAM hands it the first free address. §5 no longer tells the operator to hand-install smartmontools — package installation is the node_baseline role's job (operator ruling). §0 gains the node 2/3 addresses. Task: T-7.1. Gate: MERGE GATE: PASS. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
666cc0d809 |
feat(ansible): prod node inventory + vaulted become; smartmontools into node_baseline
Prod inventory scaffolding for the three bare-metal nodes, and the become posture the operator ruled for production. BECOME: prod does NOT get dev's NOPASSWD sudo. Prod keeps a real sudo boundary and supplies the password to Ansible as an ansible-vault-encrypted `ansible_become_password` in inventory/group_vars/k3s_prod/vault.yml, mirroring the existing hypervisor and proxmox_hosts groups. The decrypting passphrase stays outside the repo (~/.config/home-k3s/ansible-vault-pass), so plays still run unattended with no plaintext secret in git. Committed here: README.md + vault.yml.example only — the operator creates the encrypted vault (`ansible-vault create`), as with every other vaulted group. PACKAGES BELONG IN THE ROLE (operator ruling): smartmontools is now declared in node_baseline_packages, not hand-installed on a node. Drive health tooling is a permanent part of a bare-metal node's baseline — dev nodes are VMs and never needed it. Same click-ops-never principle as the rest of the repo. ADDRESSING: nodes .10/.11/.12, reached DIRECTLY on the LAN (no pfSense NAT hop, unlike dev's 192.168.2.102:225x mapping), so ansible_host is the real node IP. kube-vip control-plane VIP is 192.168.2.9 — deliberately BELOW the nodes and OUTSIDE the Cilium LB pool (.20-.30) so kube-vip and LB-IPAM can never contend for an address. The ingress Gateway VIP (.20) and the LB pool are Flux/cluster-vars concerns and deliberately do not appear in Ansible. BLOCKED-BY-DESIGN: node_baseline_longhorn_device / node_baseline_etcd_device are present but set to an obviously invalid /dev/BLOCKED-... path pending the operator's disk-layout decision (node 1 is a single SATA SSD; the role's /dev/vdb,/dev/vdc defaults are virtio names that do not exist on bare metal). Leaving them unset would silently fall back to those equally-nonexistent defaults — a confusing failure that looks like a wiring bug. This fails just as hard and says exactly why. KNOWN FOLLOW-UP: roles/k3s_server and every node-host playbook hardcode the literal group names k3s_primary/k3s_secondary/k3s_servers, so the k3s_prod_* groups are not selected by them as-is. Harmless now (nothing can accidentally run against prod) but the "same roles, unchanged" reuse R-ENV-02 promises needs the target group parameterized before §6 of PROD-PROVISIONING.md can execute. Documented in hosts.ini and both READMEs. Serves: R-ENV-01, R-ENV-03, R-SEC-01. Task: T-7.1. Gate: MERGE GATE: PASS. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
e3c44368c8 |
docs(prod): correct sudo posture to vaulted become; record status-surface ruling
Two operator corrections: 1. PROD DOES NOT GET NOPASSWD SUDO. My initial checklist recommended mirroring dev's passwordless sudo onto prod — rejected, correctly. Dev's NOPASSWD is a convenience that must not propagate to production. Prod instead keeps a real sudo boundary and drives Ansible `become` with an ansible-vault-encrypted `ansible_become_password` in inventory/group_vars/k3s_prod/vault.yml, mirroring the EXISTING hypervisor and proxmox_hosts groups. The decrypting passphrase stays outside the repo at ~/.config/home-k3s/ansible-vault-pass (ansible.cfg :: vault_password_file), so plays still run unattended with no plaintext secret in git. 2. THE TASK BOARD IS NOT THE STATUS SURFACE — THE REPOSITORY IS. Operator ruling closes the escalation raised in the previous commit. Requirement/spec status is to be DERIVED from repo state (what is authored, reconciled, and covered by a green requirement-tagged test), not from a hand-maintained board. The spec board is left as a historical seed; no agent edits it and no SPEC_PIN churn is spent on it. ci/spec-immutability.sh keeps its full strictness — no path exception was added, since weakening an R-TEST-06 gate for a bookkeeping file would be a defect, not a fix. A repo-derived spec status report becomes the artifact of record alongside the coverage report and this log. Serves: R-ENV-03 (prod provisioning posture), R-SEC-01 (no plaintext secrets), R-TEST-06. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
574088dfeb |
docs(prod): open Phase 7 (prep only) + T-7.1 bare-metal provisioning checklist
The operator added the first production node (k3s-prod-1, 192.168.2.10 — HP EliteDesk 800 G5 Mini, i7-9700T/31GiB, Debian 13, VT-x + /dev/kvm) and opened roadmap Phase 7 for PREP WORK ONLY: no k3s install until the R-ENV-02 blockers close and the gate is green. Records the Phase-7 entry-gate position honestly: the operator has declared hardware ready, but "Phases 1-6 proven in dev/test" is NOT met (37/51 conformance; R-ENV-01/02/03 themselves uncovered). Proceeding is an explicit, accepted risk — not an oversight. Adds infra/PROD-PROVISIONING.md (T-7.1, design-only): firmware/BIOS, disk layout, OS, network, access, Ansible reuse, repo blockers, build sequence, Phase-7 exit criteria. Key findings recorded in DECISIONS-LOG: - R-ENV-02 is violated in the base — dev LAN addresses hardcoded into base manifests (cilium k8sServiceHost, the Cilium LB pool, crafty/UniFi LB IPs, 3 S3 backup endpoints). Prod is on 192.168.2.x, so prod would build WRONG today. - clusters/prod never sets GATEWAY_VIP -> would inherit the base default 10.73.0.240, an address on the RETIRED libvirt network. - No NOPASSWD sudo on the prod node -> all Ansible fails at its first become. - node_baseline device defaults are virtio (/dev/vdb,vdc) and absent on bare metal. - ESCALATION: agents/task-board.md is structurally unmaintainable — AGENTS.md designates it the one agent-editable surface, but ci/spec-immutability.sh fails the gate on ANY spec-repo dirt with no path exception. Explains the all-todo board after six phases. NOT worked around: exempting it would weaken an R-TEST-06 gate. Operator decision. Operator decisions: Phase 7 open (prep only); wait for all 3 nodes before cluster-init; check drive health before choosing the single-SSD disk layout. Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-PLAT-01/03. Task: T-7.1. Gate: MERGE GATE: PASS. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
c182888133 |
docs: PM handoff — 2026-08-31 current-state, gotchas + next steps; PHASE1-STATUS 8 vCPU
Appends the definitive current-state/handoff entry to DECISIONS-LOG (SSO fixed via CVE-2025-30162 egress allow, 5 forward-auth UIs gated, 8 vCPU, Gitea Actions CI live, external-snapshotter vendored; operational gotchas; prioritized Next Steps → Longhorn-SSD, L2/L3-CI, forward-auth long-term, Phase-6 onboarding). Updates the reconnect guide (6→8 vCPU) and points new agents at the handoff. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
676c37a247 |
tofu: node vCPU 6 -> 8 (CPU spike headroom)
Steady-state CPU PSI 15-23% + 15-min load 5.6-8.0 on 6 cores (run queue at/over core count) left no headroom for bursty workloads. Proxmox host has 24 cores at ~4% load, so 3x8 = 24 vCPU is 1:1 with zero oversubscription. In-place tofu update (cores change is pending until a per-node VM stop/start); rolled one node at a time to keep etcd quorum. Comment also corrected 16->24 GiB (stale). Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
1f921fbfed |
docs(ansible): add re-provision runbook + node-host playbook list
A node rebuild (tofu -replace / fresh OS) wipes host-level config Flux/Helm don't manage. site.yml omits kube-vip.yml and vpn.egress.gateway.yml, so a re-provision missed the node-host OpenVPN on k3s-dev-3 and dropped all egress-class=vpn pod traffic. Document the node-host layers that must be re-run after any re-provision, and correct the stale author-only banner (these plays are live). Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
22db5fdb63 |
kyverno: 3.3/v1.13 -> 3.9/v1.19 (stay-current) + CLI lockstep
- HR chart 3.3.x -> 3.9.x (Kyverno v1.13.6 -> v1.19.0). - crds.install:false + CRDs pre-applied server-side (5.7 MB bundle exceeds kubectl's 256 KB annotation limit; Helm client-side apply would stall the platform Kustomization, kps-style). - CLI lockstep (ci/conformance/devcontainer versions.env + run-l0.sh default) -> 1.19.0. Pre-verified: v1.19.0 kyverno CLI passes the L0 suite 17/17 with the existing test format. - Policies are kyverno.io/v1 ClusterPolicy (stable API, unchanged in v1.19). Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
cbc2301b30 |
docs: reconcile docs/comments with post-upgrade cluster state
Text-only cleanup after the upgrade campaign. Updates stale version strings, prose, and comments to the verified current facts; no Kubernetes behavior or version pins changed except the kubectl toolchain bump (below). Verified facts applied: k3s v1.36.4+k3s1, Debian 13 trixie, Cilium 1.20.1 / Gateway API v1.6.1, Longhorn 1.12.1, node RAM 24 GiB, dedicated virtio2 SSD etcd disk per node, Velero chart 12.x / v1.18, Grafana 13.2.0. - Version strings across infra/ and platform/ READMEs + comments - kubectl toolchain 1.31.4 -> 1.36.4 (devcontainer/ci/conformance versions.env; 1.31 vs a 1.36 cluster exceeded the +-1 skew) - Retired-single-apps-Kustomization refs -> apps-wave-0/1/2 + apps-vms with per-wave SOPS decryption (flux/clusters docs + bootstrap.sh) - COMPLETED/historical banners on the three executed plan docs - Resolved TODO(WS-SEC) markers -> Renovate-maintained ranges - Appended dated DECISIONS-LOG entry (2026-08-30) superseding the 2026-08-26 LIVE STATE line Left as revalidation items (not blind-edited): external-snapshotter v8.2.0 vs 1.36, Cilium ext_authz/forward-auth relevance under 1.20.1, conformance ImageVolume feature-gate assertions under 1.36. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
dbd73f1f88 |
kube-vip: v0.8.9 -> v1.2.3 (k8s 1.36 catch-up)
Manifest env-var config (manager/cp_enable/svc_enable/vip_arp/leaderelection) is unchanged across 0.8->1.2, so the bump is image + re-applied auto-deploy manifest. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
d84feac736 |
OS upgrade complete: Debian 12->13 (trixie) baseline + re-provision role fixes
All 3 k3s nodes re-provisioned to Debian 13 (kernel 6.12) via immutable tofu -replace, one at a time, leader/primary last. variables.tf image defaults -> trixie generic. k3s_server role: ensure /etc/rancher/k3s exists (fresh-node re-provision fix). Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
ef0a735145 |
node_baseline: add cryptsetup (Longhorn 1.12 RequiredPackages) for Debian 13 re-provision
Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
7a59ef07bb |
Upgrade complete: pin k3s_version v1.31.5 -> v1.36.4+k3s1
Cluster upgraded 1.31.5 -> 1.36.4 over 5 sequential minor hops (binary-swap per node, leader-last, Longhorn rebuild-freeze). Components: Cilium 1.16.5->1.20.1, Longhorn 1.7.3->1.12.1, GW-API CRDs ->v1.6.1, etcd 3.5->3.6. All validated; data intact. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
b4b3bb0922 |
W5 (FINAL): Cilium 1.19.7->1.20.1 (+GW-API CRDs v1.6.1), Longhorn 1.11->1.12.1 (pre-k8s-1.36)
Cilium 1.20 is mandatory for k8s 1.36. GW-API CRDs v1.4.1->v1.6.1 (direct, never v1.5.x). Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
36173b9529 |
W3: Cilium 1.18.13->1.19.7 (+GW-API CRDs v1.4.1), Longhorn 1.9->1.10.2 (pre-k8s-1.34)
Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
cd8cf6220b |
W2: Cilium 1.17.18->1.18.13 (+GW-API CRDs v1.3.0), Longhorn 1.8->1.9 (pre-k8s-1.33)
Gateway API CRDs bumped v1.2.1->v1.3.0 (required by Cilium 1.18; applied server-side). Land on current k8s 1.32 before the 1.33 hop. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
5efd983239 |
W1: Cilium 1.16.5->1.17.18, Longhorn 1.7->1.8 (pre-k8s-1.32 component landing)
Land the two pacing components on current k8s 1.31 before the 1.32 k3s hop. Cilium 1.17 supports k8s 1.29-1.32; Longhorn 1.8 supports 1.25+; Gateway API CRDs stay v1.2.1 (Cilium 1.17 targets v1.2.x). Also commits the full upgrade plan doc. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
af5fc5b25f |
feat(tofu,ansible): SSD-backed etcd disk per k3s node (Phase 1: code + plan)
Interim bridge to stop Longhorn replica-rebuild I/O storms on the shared HDD
Storage pool from starving etcd WAL fsync (crash-looping k3s / flapping the
kube-vip VIP). Adds a small dedicated SSD disk per node for etcd only; Longhorn
stays on HDD. See docs/plans/ssd-etcd-isolation.md.
tofu (infra/tofu):
- New vars etcd_ssd_datastore (default "local-zfs" = rpool/data, Micron 5100
SATA SSD mirror w/ PLP) and etcd_disk_gib (default 16).
- New virtio2 disk block on the SSD datastore per VM: discard=on, cache=none
(honor guest fsync straight to the ZFS ZIL — safe on PLP SSD), iothread=false.
Kept last (ascending interface) so bpg does not reorder virtio0/virtio1.
- Revert the debian_cloud_image resource to the long type name
proxmox_virtual_environment_download_file to match the R2 state: bpg >= 0.111
dropped the cross-type moved{} linkage to the short proxmox_download_file, so
the stale moved block failed EVERY plan with "Resource type mismatch" (broken
on main too). Aligning code to state yields a clean plan with no image
re-download. Short-name migration is deferred to a deliberate `tofu state mv`.
ansible (node_baseline):
- Format /dev/vdc (label k3s-etcd) and mount it at
/var/lib/rancher/k3s/server/db (defaults,noatime; no nofail so a missing etcd
disk fails loudly rather than silently landing etcd on the root HDD) before
k3s starts, so a rebuilt node comes up with etcd isolated on SSD. Data-safe:
filesystem module never reformats an existing fs.
Phase 1 only: non-disruptive. No tofu apply, no node/VM/etcd/Longhorn changes.
tofu plan verified: 0 to add, 3 to change, 0 to destroy — every VM an in-place
disk ADD (virtio2), dev-3 also an in-place memory 16384->24576; no VM replace.
Merge gate green (render==golden included).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
a60d34a4c3 |
tofu: bump k3s node memory_mib default 16384 -> 24576 (24 GiB)
Raise per-node RAM from 16 GiB to 24 GiB for proper failover headroom. Matches the live rolling resize of the three k3s-dev VMs on Proxmox (qm set --memory 24576 per node, one at a time, KSM enabled first). Single memory_mib var applies to all three nodes via for_each, so a future `tofu apply` would resize all three at once and break etcd quorum — the resize MUST be done live/rolling, not via apply. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
6739339f53 |
Replace MinIO backup target with versitygw (s3_backup_target role)
MinIO's open-source edition is de-prioritised (console removed, commercial
pivot), so swap the off-cluster S3 backup target on the Proxmox host to
versitygw v1.7.0 (github.com/versity/versitygw) — a maintained Go S3 gateway
serving a POSIX filesystem as S3.
Rename role minio_backup_target -> s3_backup_target and playbook backup-minio
-> backup-s3. The role now:
- decommissions any pre-existing MinIO (service/binary/env/mc-client/user/
object-layout), KEEPING the ZFS dataset Storage/k3s-backups and its mount
- installs the pinned, checksummed versitygw binary (fetched+extracted on the
controller, host has no internet), a dedicated versitygw system user, and a
Type=simple hardened systemd unit running the posix backend rooted at
/mnt/Storage/k3s-backups/data, bound to 192.168.1.9:9000 (same endpoint),
plain HTTP / path-style
- ensures the dataset has xattr=sa (versitygw stores object metadata in xattrs)
- pre-creates the velero + cnpg buckets (bucket == top-level subdir)
Credentials are REUSED: versitygw's ROOT key is set to the same scoped
access_key/secret the cluster already uses, so endpoint + key + secret + bucket
names are byte-identical to the old MinIO target. The GitOps SOPS secret
(platform/backup/minio-credentials.sops.yaml) and all cluster-side Velero/CNPG
config need ZERO change. The vaulted group_vars now expose
s3_backup_target_access_key/_secret_key (same values, MinIO root creds retired).
Verified live on the Proxmox host: versitygw active+enabled on 192.168.1.9:9000,
MinIO fully removed, dataset intact; SigV4 ListBuckets with the reused key
returns velero+cnpg; a throwaway in-cluster pod reaches http://192.168.1.9:9000
(versitygw S3 response). Re-run is idempotent (no host changes).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
ac32f01f7b |
feat(backup): MinIO S3 backup target on Proxmox host for Velero + CNPG
Stand up a single-node/single-drive MinIO S3 endpoint on the Proxmox host as the off-cluster backup target for Velero (R-STATE-03) and CloudNativePG (R-IDP-05). Ansible (author + executed): - roles/minio_backup_target + playbooks/backup-minio.yml — idempotent, additive only: ZFS dataset Storage/k3s-backups (zstd, quota=100G, /mnt/Storage/k3s-backups, no sharenfs); minio system user; pinned+checksummed MinIO/mc binaries (fetched on the controller since the host has no WAN, copied to host); hardened systemd unit bound to the host's cluster-reachable LAN IP (auto-discovered) on :9000; velero + cnpg buckets; a non-root access key scoped by IAM policy to only those two buckets. - inventory: new [proxmox_hosts] group ([email protected].2.102); overrides the cluster-wide ansible_user. Does not disturb existing k3s node entries. - group_vars/proxmox_hosts/vault.yml (ansible-vault): root + scoped MinIO creds. GitOps hand-off: - platform/backup/minio-credentials.sops.yaml (+ .example) — SOPS/age-encrypted endpoint + scoped key/secret for the Velero/CNPG wiring (WS-STORAGE/WS-SEC). Not yet wired into a kustomization (left for the consuming agents). Verified live: dataset props correct; minio active+enabled bound to 192.168.1.9:9000; S3 endpoint http://192.168.1.9:9000/minio/health/ready returns 200 from an in-cluster pod; buckets velero+cnpg present; scoped key created+policy attached; re-run is idempotent. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
0cf0c669f4 |
tofu: migrate to non-deprecated proxmox_download_file (ADR-007)
bpg/proxmox deprecates proxmox_virtual_environment_download_file ("Use
proxmox_download_file instead. This resource will be removed in v1.0."),
the short-name rename from ADR-007. Both are the same dual-registered
Framework resource, so rename the resource type + import_from reference
and add a `moved` block. OpenTofu 1.9 drives the provider's MoveState
(migration.PrefixMoveState) to transfer state in place: same id, no
re-download of the Debian image, no change to the VMs.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85
|
7d93decd61 | tofu: fill encrypted secrets | ||
|
jakman85andClaude Opus 4.8
|
216bea2a16 |
tofu: move secrets into committed SOPS/age file + tf.sh wrapper
Replace the gitignored plaintext infra/tofu/.r2.env and backend.hcl with a single committed, SOPS/age-encrypted dotenv (secrets.sops.env) plus a thin wrapper (tf.sh), so tofu runs with no manual secret entry and secrets live encrypted in git like the repo's other secrets. - .sops.yaml: creation rule encrypting infra/tofu/secrets.sops.env (all values) to the existing age recipient. - secrets.sops.env: encrypted, PLACEHOLDER values only. Holds R2 S3 keys, the state-encryption passphrase (TF_STATE_PASSPHRASE scalar), and the Proxmox API token/endpoint. Operator fills real values via `sops`. - tf.sh: sops exec-env decrypts the dotenv, exports PROXMOX_VE_*/AWS_*, builds the multi-line TF_ENCRYPTION HCL from TF_STATE_PASSPHRASE, and runs tofu (injecting -backend-config=backend.r2.hcl on init). Proxmox token stays env-only, never a tofu variable/state. - backend.r2.hcl: committed non-secret R2 bucket/key/endpoint (account id is not a credential), passed to `tofu init`. - Retire .r2.env/backend.hcl(.example): drop the committed templates; keep the gitignore lines as a safety net for leftover local plaintext. - Update backend.tf/README/REMOTE-STATE for the new day-2 flow and DR story (repo + age key is sufficient). Verified: placeholder file round-trips sops -e/-d with the repo age key; tf.sh exports the expected VAR NAMES and builds valid TF_ENCRYPTION HCL (stub tofu, no real plan/secrets). Co-Authored-By: Claude Opus 4.8 <[email protected]> |