b93594b7c85aeec4d28e008b239d61d181e1e6de
118
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
ae5c7385a5 |
docs(gitea): the README still named the ILLEGAL secret GITEA_SETTINGS_TOKEN
PR #15 renamed the Actions secret to SETTINGS_API_TOKEN because Gitea RESERVES the GITEA_ prefix and rejects such a name with HTTP 400 "invalid secret name" -- the defect that left the branch-protection detector inert and unnoticed. The README in this very directory, which documents that story, went on naming the illegal name in its behaviour table and in its closing "remaining step" line. Both now say SETTINGS_API_TOKEN, plus a note recording what ci/gitea-settings.sh ACTUALLY does, quoted from the file rather than assumed: TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}" GITEA_SETTINGS_TOKEN is still honoured, but ONLY as an environment variable for a local run -- it can never be the CI wiring, because the secret that would feed it cannot be created at all. The closing line also predated PR #15 storing the secret; it now says so. Documentation only. No behaviour change. NOT fixed here, because it is outside this PR's diff: ci/gitea-settings.sh:45 still instructs the reader to "create a Gitea repo secret GITEA_SETTINGS_TOKEN", which is impossible. Same defect, different file -- worth its own one-line change. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
19f9a8ae0c |
feat(sec): provision the narrow conformance CI trigger credential -- and refuse to arm a fake green
docs/plans/l2-l3-conformance-ci-wiring.md §3 item 6, as an operator script:
infra/gitea/conformance-trigger-kubeconfig.sh. UNEXECUTED here -- minting a
cluster credential and changing Gitea variables are operator actions.
It mints a token for conformance/conformance-trigger (TokenRequest), wraps ONLY
that token with the cluster CA and server into a kubeconfig, and stores it as the
Actions secret CONFORMANCE_TRIGGER_KUBECONFIG. Never cluster-admin, never a copy
of the operator's own kubeconfig. The token is never printed and never reaches
argv: it goes into a 0600 file and into the PUT body FILE (--data @file), and the
script refuses to run under `set -x` because xtrace would print it.
Two traps this repo already paid for are designed around:
* GITEA_ is a RESERVED secret-name prefix (HTTP 400 "invalid secret name"), so
the name is CONFORMANCE_TRIGGER_KUBECONFIG -- which is also what gate.yml and
merge-gate.yaml already reference.
* `curl -sS` exits 0 on an HTTP error. Every write checks %{http_code} AND reads
the object back. The secret read-back proves presence by NAME only (Gitea
never returns secret values) and the script says so rather than implying its
content was checked; the variable read-back asserts the value is exactly
"true", which is what gate.yml compares against.
PRECONDITIONS ARE SPLIT IN TWO, and the split is the security argument:
mint-blocking (cluster) -- SA/Role/RoleBinding present, and ClusterPolicy
conformance-trigger-job-shape present + Enforce + failurePolicy Fail + Ready.
Any failure and NOTHING IS MINTED. Kubernetes places no `use` check on a Job's
serviceAccountName, so a `create jobs` token in that namespace IS the privileged
conformance-runner identity unless admission bounds its shape. RBAC cannot close
that (plan §4); minting before the boundary is live hands CI the escalation the
boundary exists to close.
arm-blocking (repo) -- digest-pinned CONFORMANCE_RUNNER_IMAGE, an initContainer
plus the admitted image in chainsaw-job.yaml, and CONFORMANCE_TRIGGER_KUBECONFIG
referenced in ci/conformance.sh CODE rather than a comment. These block only
vars.CLUSTER_AVAILABLE, because storing the secret early is harmless (the job is
gated off) and is the plan's own order.
All three arm-blocking checks FAIL in the tree today, so a live run would store
the secret and then REFUSE to arm CLUSTER_AVAILABLE, naming each missing item.
That is the intended behaviour: measured 2026-09-29, CLUSTER_AVAILABLE=true with
no trigger path scaffold-skipped all 54 tests at rc 0 and the stage printed "L2/L3
conformance passed". ci/conformance.sh now blocks that case, so arming early does
not enable L2/L3 -- it only breaks every PR. Arming needs the explicit flag, every
precondition green, and a typed confirmation on a TTY.
Before storing anything, the credential is PROVEN narrow: auth whoami must be
system:serviceaccount:conformance:conformance-trigger, a real `get jobs` round-trip
must succeed, and an auth can-i matrix asserts both halves -- granted (create/get
jobs, get pods, get pods/log) and DENIED (delete jobs, create pods/exec, get
secrets, create jobs in default, get nodes, '*' '*' --all-namespaces). A mismatch
aborts before any store, and says explicitly not to widen
conformance/incluster/rbac.yaml to make it pass.
Verified only by `bash -n` (clean) and reading. The Gitea variables API shape
(POST creates, PUT updates, body field `value` not `data`) is from documentation,
not measurement: the script tries POST then PUT, prints the response body on
failure, and treats the read-back as the authority. Kyverno readiness is accepted
from either .status.ready or a Ready condition, and "neither present" blocks
rather than assuming ready.
Gate (pinned toolbox): MERGE GATE: UNDETERMINED (rc 3) -- spec-immutability and
gitea-settings unrun, the expected local result.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
50a48e6f07 |
fix(ci): the settings token could NEVER have been stored -- Gitea reserves the GITEA_ prefix
The drift detector merged in #11 was inert and would have stayed inert silently. WHAT HAPPENED. infra/gitea/set-ci-token.sh minted a token and PUT it to the repo Actions secret GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for secret names -- exactly as GitHub reserves `GITHUB_` -- and rejects that PUT with: HTTP 400 {"message":"invalid secret name"} The script sent the PUT to /dev/null and printed "==> stored." unconditionally, and `curl -sS` exits 0 on an HTTP error, so it reported success for a store that never happened. Measured afterwards: the token existed in the user's token list, and the repo's secret list was EMPTY. ci/gitea-settings.sh would have skipped on every CI run forever, and with REQUIRE_GITEA_SETTINGS=true it would instead have BLOCKED EVERY PR -- including the one fixing it. Three changes: * the secret is SETTINGS_API_TOKEN; ci/gitea-settings.sh reads it, keeping GITEA_SETTINGS_TOKEN as a local-run fallback; * set-ci-token.sh checks the PUT's HTTP status AND reads the secret back, refusing to claim success otherwise. That missing verification is the actual defect -- the wrong name was just what it hid; * gate.yml maps the secret and ENABLES REQUIRE_GITEA_SETTINGS. WHY ENABLING IT HERE IS SAFE: this PR's own CI run is the proof. If the stage were still skipping, REQUIRE turns that into a failure and this PR goes RED rather than merging. Green means the token resolved and live Gitea matched infra/gitea/expected-branch-protection.json. Verified locally before pushing: REQUIRE=true, token present -> MERGE GATE: PASS REQUIRE=true, no token -> gitea-settings SKIPPED -> BLOCKED (not a silent skip) The secret is already stored and read back as present, so CI has what it needs. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
e6f6e4febb |
feat(ci): detect Gitea settings drift -- the blind spot that hid a month of outage
Every check in this repo reads the repository, the rendered manifests, or the cluster.
NONE could see Gitea's own database, and that is where branch protection and runner
registration live. Three failures came out of that blind spot on 2026-09-29, all the same
shape -- a setting that READ as enforced while not being, with nothing able to notice:
* main had NO branch protection for ~a month while DECISIONS-LOG recorded it as a
required status check;
* ZERO Actions runners were registered, so PR #1 -- the first PR this repo ever had --
sat pending for 30.6 minutes with nothing to execute it;
* after enable_status_check and required_approvals were turned on they STILL did not
bite, because block_admin_merge_override was false: PRs #5-#9 merged with zero
approvals recorded against required_approvals: 1.
infra/gitea/*.sh ASSERT the desired state. Nothing DETECTED drift from it. This is the
detector.
ci/gitea-settings.sh runs as a merge-gate stage: it diffs the live rule at
/branch_protections/main against infra/gitea/expected-branch-protection.json and asserts at
least one runner is online. It reads the RULE object, not /branches/main -- the branch view
does not expose block_admin_merge_override and showed all-green while the rule was
overridable by the only account that merges.
Verified by test, not assumed:
no token -> exit 3, skips loudly, does NOT block
live == declaration -> exit 0
injected drift -> exit 1, blocks, naming both the drifted key and the runner count
The skip is DELIBERATELY not in gate.sh's strict-skips loop. Making an absent credential
block every PR is precisely the lock-yourself-out failure this repo keeps walking into, so
a skip denies a PASS (exit 3) and says loudly what is unverified. REQUIRE_GITEA_SETTINGS=true
upgrades it to a block once the token is wired.
gate.yml's `|| [ "$?" -eq 3 ]` tolerance now covers a second stage. That widening is
declared in the file rather than left to be discovered, and it narrows back the moment
REQUIRE_GITEA_SETTINGS is set.
infra/gitea/set-ci-token.sh mints the token and stores it as the repo Actions secret without
ever printing the value.
UNTIL THAT TOKEN EXISTS THIS STAGE SKIPS, AND A SKIPPED DETECTOR DETECTS NOTHING.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
fcc66c7858 |
fix(gitea): admins could override the gate -- the enforcement was not enforcing
enable_status_check and required_approvals were set earlier today and recorded as done. They did not bite. block_admin_merge_override was false, and Gitea lets repo admins override merge requirements without it. jakman85 holds owner, so both requirements were bypassable by exactly the account that performs every merge. The evidence was already on the record: PRs #5-#9 all merged while required_approvals: 1 was in force, with ZERO approvals recorded on any of them. Two process failures, both the same shape as the defect this whole session started with: 1. Verified with the wrong view. /branches/main reports protected, enable_status_check and required_approvals but NOT block_admin_merge_override -- it showed everything green while the rule was overridable. /branch_protections/main is authoritative. 2. Probed for a field name assumed to exist. This Gitea (1.24.3) has no `apply_to_admins`; asking jq for it returns null, which reads like "set to false" but means "key absent", so a PATCH that silently did nothing produced a convincing before/after. jq returning null is not a value -- it may be an absence. Enumerate with to_entries. require-gate.sh now sets block_admin_merge_override: true; block-admin-override.sh fixes it in isolation. Both carry the field-name warning. This PR is the test. If it merges WITHOUT an approval, the fix did not work. Also records the governance question: with only jakman85 (human) and gitea-admin (agent), a human-authored PR now needs the automation account to approve it. That is ceremony, not review, and the choice between keeping it, dropping approvals in favour of the status check, or adding a second human is the operator's to make. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
b2d9883958 |
docs(ci): close out the hardening -- land the scripts, correct a claim that shipped wrong
The branch-protection + PR-gate + sidecar sequence is complete and verified. This records the finished state and fixes what the sequence itself proved wrong. CORRECTS A FALSE CLAIM I SHIPPED. PR #7 said the sidecar conversion would remove the one-per-start dockerd line "http: TLS handshake error from 127.0.0.1: EOF". It does NOT. Exactly one still appears, immediately after "API listen on [::]:2376" -- it is now the startupProbe's FIRST attempt, landing microseconds after dockerd begins listening and cut off by timeoutSeconds: 1; the next poll succeeds. Cosmetic, and deliberately not chased: raising the timeout to hide one benign startup line would slow every subsequent poll. The manifest comment now states this instead of the wrong prediction. Lands infra/gitea/require-gate.sh and verify-then-enforce.sh, which drove the rollout from a session temp dir. verify-then-enforce.sh merges nothing -- merging is a human's reviewed act -- but forces the reconcile and ABORTS unless the pod is healthy, a runner is online, and a real workflow goes green on it, before enabling enforcement and probing the push. Re-run it after any future act-runner change. Trims the now-obsolete feat/ rationale from claim-ship.sh: per-branch push triggers were deleted, so no prefix produces a duplicate run any more. CLAUDE.md's enforcement table flips from "weaker than this rule" to machine-enforced, with all four rows marked verified and the push rejection noted as PROVOKED rather than assumed. It also now states what a green PR does NOT prove: CLUSTER_AVAILABLE is unset, so L2/L3 never runs in CI and only the L0/L1 offline subset is covered. infra/gitea/README.md marks the sequence done and names the root cause that remains unfixed: Gitea repo settings live in its own database, invisible to Flux and unreadable by the conformance gate, so NOTHING detects drift from what this directory declares. These scripts assert desired state; they are not a detector. The check that reads the live API and fails on divergence is the highest-value work left, and it is the difference between this being fixed and this being unable to recur. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8e017fad91 |
docs(runbooks): the PR flow, not git push origin main -- that instruction is now blocked
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
234cc6a71a |
docs(gitea): a skipped job reports SUCCESS, not pending -- correct the deadlock claim
The runner came up on 2026-09-29 and the first CI jobs this repo has ever run went green, which falsified a claim merged in #2. MEASURED: gate / gate (pull_request) success 1m58s (PR 1), 1m34s (PR 2) gate / gate (push) success 1m32s (PR 2 only) gate / conformance (pull_request) success "Has been skipped" #2 said listing the conformance context as a required check would deadlock every PR, because the job is `if: false` and never runs. That is wrong. Gitea reports a skipped job as SUCCESS. Requiring it would not deadlock anything -- it would pass VACUOUSLY, forever, while nothing ran. The conclusion is unchanged (never require it) but the reason inverts, and the corrected reason matters more: under HARD RULE 4 a permanently-green check that verifies nothing is worse than a deadlock. A deadlock is loud. This is the exact fail-open shape the 2026-09-18 audit and this one both keep finding. Also records that the (push) contexts only appear for branches matching gate.yml's push trigger -- feature/**, fix/**, docs/**, chore/** -- and NOT feat/**, which is why PR 1's feat/ branch produced only pull_request contexts. Require the pull_request context alone, or PRs from non-matching prefixes never satisfy the rule. CLAUDE.md's enforcement table is updated: the runner half is genuinely fixed and observed green, the branch-protection half is not. Adds the warning that a green tick is not by itself evidence that anything ran. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
fe862ab527 |
infra(gitea): recover the Actions runner before anything requires the gate
No runner is registered at repo, org or site level, so gate.yml has never executed. PR #1 sat pending for 30.6 minutes across 122 polls and never reported. Requiring the gate as a status check in that state would lock the repo permanently. Adds runner-token-rotate.sh: mints a fresh instance-level registration token from the Gitea API and writes it into the SOPS secret via sops set, without ever printing the value. Records the likely root cause, which a new token alone does NOT fix. act_runner registers once and persists its identity in /data/.runner on the act-runner-data PVC. If Gitea no longer holds a matching runner row, the pod starts, sees .runner, SKIPS registration entirely, and polls with an identity Gitea does not recognise. Symptom: zero runners, pod running, nothing obviously broken. Clearing the PVC is what forces a genuine re-registration. The README now sequences runner-first, enforcement-second, and requires CONFIRMING the runner reports online rather than assuming it. That assumption is the whole defect: DECISIONS-LOG recorded the runner as online on 2026-08-31 and nobody checked since. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a2665c0511 |
infra(gitea): script the branch protection that was never applied
Gitea repo settings live in its own database, not in a manifest Flux reconciles, so they are drift-prone and invisible to the conformance gate -- which is how main came to accept direct pushes while the log recorded it as protected. Adds infra/gitea/branch-protection.sh (blocks direct pushes to main) and a README carrying the measured before-state and the ORDER the remaining steps must happen in. The script deliberately leaves enable_status_check:false and required_approvals:0. Both would deadlock today: there is no runner to satisfy a required context, and the only human account is not yet a repo collaborator, so no one could supply an approval. The README sequences both. It prints a push probe to run afterwards. The API response is not proof -- Gitea may let site admins bypass protection, and that has to be observed, not assumed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
78d0f1b0f6 |
feat(ci): promtool in the pinned toolchain, one alert-rule harness, and the gaps declared
The CNPG alert unit tests (324 lines, mutation-checked) were run by NOTHING. promtool was
not in the baked toolchain, no gate stage evaluated any *.promtool-test.yaml, and the only
runner `curl | tar -xz`'d a promtool release from GitHub at TEST-RUN time if one was
missing. Hours later a second, independent harness landed for the Velero rules
(platform/monitoring/tests/) with its own fallback — a `podman run` of
quay.io/prometheus/prometheus. So the repo had two harnesses, two acquisition paths, and
coverage that looked present and was enforced nowhere: the exact failure mode that had just
produced a permanently-firing CNPG alert and a structurally inert Velero alert.
PROMTOOL IS NOW PINNED LIKE EVERY OTHER TOOL. PROMTOOL_VERSION=3.14.0 in
infra/devcontainer/versions.env and ci/versions.env (the two stay in lockstep, per the
repo's rule), baked by the Containerfile from the prometheus release tarball (promtool ships
inside it), recorded in /etc/dev-toolchain-versions, passed by dev.sh, and installed for the
l0 stage by .gitea/workflows/scripts/install-tools.sh so the CI runner is not permanently
SKIPPED. The pin is not free-floating: promtool IS the rule engine, so it tracks the
Prometheus prod actually runs — verified against the live cluster, not assumed:
kubectl --context k3s-prod get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'
-> v3.14.0-distroless (image quay.io/prometheus/prometheus:v3.14.0-distroless)
ONE HARNESS. platform/cnpg/run-alert-tests.sh is folded into
platform/monitoring/tests/run.sh, which now covers every PrometheusRule in the repo that has
tests, via two conventions it discovers automatically: `<manifest>.promtool-test.yaml`
beside its manifest (cnpg), and `<component>/tests/*.test.yaml` over that component's
manifests (monitoring). It writes the flattened rules under whatever name each test file's
own `rule_files:` declares, so no test file needs to know the harness exists, and it
`promtool check rules` first — a rules file that does not parse would make every unit test
pass vacuously by finding no rules at all. The cnpg script's reasoning is preserved: why a
CR must be flattened (`.spec.groups`, never re-typed), and why generated files go to a temp
dir. Its `${BASE_DOMAIN}` substitution is dropped in favour of extract_rules.py's verbatim
placeholders, with the trade-off recorded there — the hazard it guarded against is addressed
by the temp dir, and verbatim leaves one transformation fewer between the shipped rule and
the rule under test.
NO ENGINE IS ACQUIRED AT RUN TIME any more — not the GitHub release, not the container
image. Not because a pinned download is unusual here (the Containerfile downloads every tool
it bakes) but because those two decided the version, the network and the machine SILENTLY,
at test time, on whatever host ran the tests; and each quietly turned "the toolchain has no
promtool" into a green run, which is how the toolchain stayed without one. Missing promtool
is exit 3 (UNWIRED, never green); a PROMTOOL= that does not resolve is exit 2.
ONE WIRING: conformance/tests/l0-static/alert-rules-unit-tested, NOT a gate stage. The
`prom-rules` stage (ci/prom-rules.sh + its Makefile target + ci/gate.sh entry) is deleted:
it is the bespoke-per-check pattern
|
||
|
jakman85andClaude Opus 5
|
9b4744d450 |
fix(devcontainer): dev.sh defaults to prod — dev is decommissioned
Every unprefixed './dev.sh run kubectl ...' pointed at the dev cluster, which was decommissioned on 2026-09-12. The banner cheerfully named dev and the command then hung or errored against a context that no longer answers, with nothing saying the cluster is gone. I hit this repeatedly tonight. CLUSTER=dev is still accepted rather than removed, because the dev VM disks were deliberately kept so the environment is reversible — but it now warns. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
9161cab9cd |
feat(dns): prod takes the apex — home.keithjackson.dev
Dev is decommissioned and has released the name. Prerequisites completed and
verified BEFORE this flip:
1. dev's 18 A records released by its own external-dns (policy=sync; sources
deleted after suspending dev's Flux so they could not be recreated).
2. dev's 18 LEGACY UNPREFIXED TXT registry entries deleted from Cloudflare by
hand. external-dns v0.21.0 deleted only the `a-` prefixed form and then logged
"All records are already up to date" — it would NEVER have cleared the older
unprefixed format it no longer writes but still READS for compatibility. They
carried `owner=default`; prod runs `owner=prod`, so prod could not have
claimed these names while they existed. This was the real blocker, and it was
invisible until measured.
3. dev's `*.home.keithjackson.dev` + apex Certificate deleted so it stops
renewing. Issued ~18d ago, so prod's request is #1 this week against LE's
duplicate limit (5 per identical name set per week).
Verified at the AUTHORITY (betty.ns.cloudflare.com), not via a resolver: 1.1.1.1
served stale TXT for minutes after deletion and would have shown a false failure.
UNCHANGED, deliberately: EXTERNALDNS_TXT_OWNER_ID stays `prod` — giving prod
`default` would let it adopt and sync-delete records it does not own, which is the
2026-09-04 incident. EXTERNALDNS_DOMAIN_FILTER stays the ZONE `keithjackson.dev`:
finding F-DNS1 proved a strict subdomain filter drops the hosted zone and external-dns
then publishes NOTHING.
The zone has wildcards `*.keithjackson.dev` and `*.home.keithjackson.dev` pointing at
the public IP (legacy non-k3s prod). Dev's specific records were shadowing the latter;
prod's new specific records shadow it again. The legacy stack lives on
`*.keithjackson.dev` and is untouched.
k3s_base_domain moves with it to hold the [parity] contract, but that value renders
the apiserver's install-time oidc-issuer-url: it does NOT take effect until the
k3s_server role re-renders config.yaml and k3s is restarted serial: 1. Until then the
apiserver trusts the OLD issuer while Keycloak serves the NEW one, so kubectl OIDC
fails in that window. Cert-based kubeconfigs are unaffected.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
37a6c94168 |
fix(vpn): two gateways on one pinned PIA server evict each other every 2 minutes
Standing up prod broke dev, and the mechanism is worth spelling out because
"same PIA account, well inside its connection limit" is true and still not enough.
The vaulted .ovpn pins ONE server. With both gateways using it, dev and prod
landed on the SAME server AND were handed the SAME tunnel address — both journals
read `[Server-10779-2a] Peer Connection Initiated with [AF_INET]158.173.241.112:1198`
and `ifconfig 10.72.192.7 255.255.192.0`. Each reconnect steals that address on
that server; the loser then receives nothing, trips `--ping-restart` after 60s,
reconnects and steals it back. Measured cycle: dev restarting at 22:57:05,
22:59:03, 23:01:05 UTC with prod interleaved about a minute off, each ~2 minutes
apart, indefinitely. Neither environment had reliable vpn-class egress — prod's
probes looked green only because they happened to land inside an up-window.
Fix: `vpn_egress_gateway_pia_remote` ("<host> <port>", empty = honour the .ovpn)
rewrites just the `remote` line of the deployed .ovpn, and prod's group_vars sets
it to 158.173.241.135. That is another member of the same PIA Stockholm LB pool —
`sweden.privacy.network` resolved from k3s-prod-3 to
158.173.241.{34,55,112,135,239} — and every server in the pool takes the same
credentials and the same inline CA, so only the address differs. Dev keeps the
.ovpn's .112 untouched, which is why the override is per-group rather than in the
role or a second vaulted .ovpn.
Two details that matter for whoever touches this next:
* The regex is anchored to a line STARTING with `remote` plus whitespace, so
`remote-cert-tls` and `remote-random` are not collateral damage. The task keeps
`no_log: true` — the transformation never widens what is logged.
* A PIA server address is NOT a credential, so it lives in
group_vars/vpn_egress_gateway_prod/vars.yml in the clear, next to the vault
rather than inside it. Keeping it out of the vault is what makes it reviewable.
If prod ever gets its own PIA account, this override can be deleted again.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
04606d48b2 |
docs(vpn): stop crediting the nftables kill-switch for a drop it never makes
Four places in the tree told the reader that the host nftables table is the authoritative fail-closed backstop for egress-class=vpn pods: platform/vpn-egress README, egress-steering.yaml, the role README, and the vpn-killswitch test's own GIVEN/THEN prose. It is not, and cannot be — Cilium's eBPF egress-gateway datapath bpf_redirect()s those skbs to the egress device without invoking netfilter POSTROUTING. Corrected with the measurement in each place rather than softened, because a guard the repo trusts and that does not fire is worse than no guard. What replaces the claim, everywhere: vpn-class fails closed because CILIUM drops the traffic when it cannot use its cached egress interface, verified by tcpdump on both the WAN and tun0 during a tunnel stop and a tunnel restart (0 packets out either). The nft table keeps a narrower, now-proven job: node-originated traffic sourced from a tunnel address (killswitch_drop moved 0 -> 3 packets on k3s-prod-3 when such a packet was routed at the WAN). Also recorded: * egress-steering.yaml — `interface: tun0` is resolved ONCE and cached, and the gateway node is what tells Cilium it changed (the egress-uplink node label). A future reader's instinct will be to have the node write to that object; the comment says not to, because it is Flux-managed, and points at the label and `cilium-dbg bpf egress list` as the debugging entry points instead. * egress-steering.yaml — why the new NFS media mounts are not affected by pod-scoped steering: an NFS PV is mounted by the node kernel in the host netns, so podSelector never matches it. The 192.168.0.0/16 exclusion covers the server too, but that is the second reason, not the first. * vpn-killswitch test — its recovery step depends on the re-steer path, and could not have passed before it existed. Its no-leak assertion was always right; only the explanation was wrong. Also states the prerequisite plainly: the test cannot run at all without a labelled gateway node. * platform README — prod is live on k3s-prod-3 with the before/after egress IPs, and the note that prod's vpn-class pods had NO internet beforehand (CEGP with no matching node -> `Gateway IP: Not Found` -> dropped), so arming the gateway restored egress rather than moving it off the WAN. * role README — the standing "add the shared fwmark to the interface contract" escalation is closed as not implementable on Cilium OSS, rather than left open for someone to attempt. DECISIONS-LOG carries the full session entry, including the drafted ADR-0015 / docs-03 amendment for the architect (specs are immutable to agents) and the five things left for the operator — chief among them that a silent gateway outage still has no alert, which is what let the dev gateway sit dead for 80 minutes. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
60e3f09c67 |
feat(vpn): prod egress-gateway group — k3s-prod-3, singleton, opt-in only
`[vpn_egress_gateway_prod]` is a SEPARATE inventory group with its own vaulted credentials, not a second member of `[vpn_egress_gateway]`. That group is the default target of playbooks/vpn.egress.gateway.yml, so adding a prod host to it would mean a flagless `ansible-playbook playbooks/vpn.egress.gateway.yml` silently reconfigures a prod node's kill-switch datapath. Prod is reached only by asking for it, with `--list-hosts` first: -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3 Node 3 because node 1's SSD was replaced yesterday (least operational history) and node 2 carries the §0b memattr history. Exactly one node, forever: Cilium OSS picks "the first node in lexical ordering" among nodeSelector matches, so a labelled k3s-prod-1 would silently steal the gateway role — and the public egress IP — from k3s-prod-3, with no event and no alert. vault.yml is a verbatim copy of the dev group's CIPHERTEXT: same PIA account per the operator, and copying the encrypted blob means the plaintext was never decrypted to disk, through a shell, or into a transcript. The two files are now independent — rotating means re-encrypting both, which is the right behaviour for two environments. Also corrects a stale inventory note that claimed k3s-prod-3 has wlp1s0 up on 192.168.2.195. Re-verified before choosing it as the gateway, because a second live uplink in a kill-switch datapath is not a footnote: there is no wl* netdev on the node at all, no phy80211 device, the /etc/network/interfaces wlp1s0 stanza is commented out, 192.168.2.195 is assigned nowhere, and the only IPv4 addresses are lo and eno1 192.168.2.12. Nothing needed bringing down. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
6f698f59af |
fix(vpn): Cilium never re-resolves tun0 — the gateway dies on every reconnect
Measured on k3s-dev-3, not reasoned about. Three findings, one of which had the dev gateway silently dead for ~80 minutes before this change: 1. Cilium caches (egress IP, egress ifindex) when it first resolves the CEGP's `interface: tun0` and does NOT re-resolve when tun0 changes. Every tunnel restart re-creates tun0 with a NEW ifindex, so vpn-class egress is blackholed from then on. Observed: node had tun0 at ifindex 1987 while `cilium-dbg bpf egress list` still said 1986; pod curl timed out. Forcing a re-apply moved the map to 1987 and egress recovered instantly. `egress-gateway-reconciliation- trigger-interval: 1s` does NOT cover netlink link events — dev has that same 1s setting and still did not self-heal, 20s+ after recovery. Fix: vpn-egress-resteer.sh publishes "<tun0 addr>-<ifindex>" as a node label. A node update makes the gateway re-process the policy (verified: cached ifindex 1986 -> 1987 within 10s). A node label, not an annotation on the policy, because the policy is Flux-managed GitOps and the node's labels are already this role's territory. Fired by the OpenVPN up-script on every connect, plus a 2-minute convergence timer for the boot case where the kube-API is not up yet. Proof: `systemctl stop` then `start` of the tunnel now recovers on its own (map 1989 -> 1990, PIA IP returns); before this commit the identical sequence stayed broken indefinitely. 2. The nft kill-switch CANNOT see the traffic it claims to guard. A counters-only diagnostic table on the gateway node recorded, while a vpn-class pod egressed: 0 packets in forward and 0 in postrouting for both the pod CIDR and the PIA range, against 45166 packets on the same postrouting hook for WAN traffic and 16 packets for a node-originated curl out tun0. Cilium's eBPF datapath bpf_redirect()s these skbs to the egress device without invoking netfilter. So `ip saddr @set oifname != tun0 drop` can never fire for vpn-class pods. What actually fails closed is Cilium's own drop when the cached egress interface is unusable — tcpdump during a tunnel stop AND during a restart showed 0 packets leaving either the WAN or tun0. Every rule now carries a `counter` so the claim stays falsifiable, and every comment that called this table the authoritative vpn-class backstop is corrected rather than left to mislead the next reader. Also removed the `meta mark 0x1c8 oifname != tun0 drop` rule and its variable: nothing stamps that mark (the only Cilium mark on the node is 0x200/0xf00, the L7-proxy rule to table 2004), the traffic never reaches netfilter anyway, and a live rule matching an arbitrary mark is a hazard to whatever picks it next. 3. The up-script flushed the nft source set on every connect, so a packet still sourced from the previous tunnel address matched neither the ip-rule nor the drop rule. The set is now additive (`add element` is idempotent; the down-script already retained it): an element can only ever tighten the guard. Finding 2 means that window could not have leaked POD traffic, but node-originated tunnel traffic does traverse netfilter, and additive is free. Kill-switch reloads no longer disarm the guard either: the .nft file uses `table` + `flush table`, which clears rules but leaves the named set's elements intact, so an Ansible run cannot empty the set mid-flight. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
a624c638ef | merge: docs/node-1-drive-replaced | ||
|
jakman85
|
90b39de527 | merge: fix/swap-off-persists-across-reboot | ||
|
jakman85andClaude Opus 5
|
df1d2ab44d |
docs(prod): node 1's drive was replaced — and §2's exit condition still is not met
Records the 2026-09-07 I/O-starvation incident and the drive swap: the C400 was replaced with a Samsung 850 EVO, and under the same load class (an active 18-replica Longhorn rebuild, with node 1 as the write target) I/O pressure fell from full avg300 11.36 to 3.79 — from ~4x worse than its peers into node 3's band. Updates the three §0 cells that had become false (disk model, drive health, kernel — all three nodes now run 7.1.8, not 6.12) and adds §0c rather than rewriting the 'NOT hardware-symmetric' analysis, which is still the correct reasoning about the state it described. Two things the rebuild taught, recorded so the next one is cheaper: Longhorn refuses a replaced disk with DiskFilesystemChanged and needs disable -> remove -> re-add in that order, and the inventory's [k3s_prod_primary] decides BOTH --cluster-init and k3s_api_host, so rebuilding the founder without moving that designation would have founded a second cluster. The replacement does NOT close §2: etcd and Longhorn still share one SATA spindle. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
69a25cff82 |
fix(node_health): say 'active via <unit>', not a bare parenthetical
The unit name in the swap-active detail read as an unlabelled aside. Naming it as the unit that activated the swap is what makes the line actionable. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
c1e36d9353 |
docs(node_baseline): say why the Longhorn claim guard's swap list is now always empty
The claim guard builds node_baseline_lh_protected_devices from ansible_mounts plus `swapon --show`. With swap.target masked, a baselined node has no active swap, so that half of the list contributes nothing — which reads like a guard that quietly stopped guarding. It is not: the claim candidate comes from whatever is mounted at node_baseline_longhorn_claim_mountpoint, and a swap partition is neither mounted nor ext4, so it could never be the candidate. Say so where the next reader will ask. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
9d00a3fcd3 |
fix(node_health): memory.swap blamed the wrong thing, and could not see the defect
The WARN read "roles/node_baseline runs swapoff and neutralises the fstab entry; this node has not been baselined yet". That was a MIS-DIAGNOSIS. All three prod nodes HAD been baselined, had their fstab swap line commented out by the role, and were still swapping 977 MiB each after a reboot. Anyone reading that WARN would have re-run the role that had already run, and watched swap come back next boot. The detail now names the actual mechanism — systemd-gpt-auto-generator re-creating a .swap unit from the partition TYPE GUID 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f at every boot, without reading /etc/fstab — says in as many words that this is not "the node has not been baselined", and points at `node-baseline.yml --tags system`. The check was also blind in the other direction: SwapTotal == 0 is not evidence that swap is disabled, only that nothing has been swapped on yet this boot, so a node freshly swapoff-ed PASSed while still set to swap again on reboot — exactly the state the old role left behind. The collector now reports memory.swap_persistence (swap.target masked?, .swap units the generators emitted this boot, which of them are masked, leftover active fstab swap lines — all world-readable, no root, no systemctl parsing), and the check WARNs on "clean now, swapping after the next reboot". Rendered against every fact shape (swap active + unmasked, swap off + unmasked, after the fix, a dev VM with no swap partition at all, per-unit mask only, persistence unreadable, an older collector's output with the key absent, and /proc/meminfo unreadable) and verified live on k3s-prod-1 post-reboot: [PASS] memory.swap — no swap active and none can be activated: swap.target is masked Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
0e92d258dd |
fix(node_baseline): swap-off now survives a reboot — mask swap.target
The role's `swapoff -a` + commented fstab line held only until the next boot,
on every prod node. Measured 2026-09-07: all three came back from a reboot with
977 MiB of swap ACTIVE (k3s-prod-1 /dev/sda4, k3s-prod-3 /dev/sda4, k3s-prod-2
/dev/nvme0n1p4) while their fstab swap entry was commented out by this role. The
role reported success and the setting silently expired.
CAUSE. systemd-gpt-auto-generator claims any GPT partition typed
0657fd6d-a4ab-43c4-84e5-0933c84b4f4f ("Linux swap") from the PARTITION TABLE,
synthesises a .swap unit into /run/systemd/generator.late/ and links it into
swap.target.wants/. It never reads /etc/fstab, so neutralising fstab cannot stop
it. Re-running the generator by hand on k3s-prod-1 says so verbatim:
Probed fstype 'swap' on partition /dev/disk/by-diskseq/1-part4.
Adding swap: /dev/disk/by-diskseq/1-part4
MECHANISM. Mask swap.target. `systemctl show` on the generated unit says its only
inbound dependency is `WantedBy=swap.target` — Requires/RequiredBy/BoundBy empty,
and the matching .device unit has an empty Wants=, so a device appearing does not
pull its swap unit in by itself. swap.target is the single choke point for every
activation path: gpt-auto, systemd-fstab-generator, systemd-cryptsetup-generator.
It is device-shape independent (no device name appears in the change, so one task
covers sda4, nvme0n1p4 and dev's vda VMs), boot-safe (swap.target is Wants=/After=
from sysinit.target and After= from tmp.mount; RequiredBy= is EMPTY on all three
prod and all three dev nodes, and a masked unit reached only via Wants= is skipped,
not failed), and inert if the device is absent or renamed. The generated units are
masked too, as a second independent barrier.
Rejected: `systemd.gpt_auto=0` (disables GPT auto-discovery wholesale, needs a GRUB
edit + reboot to become true, and on DEV would switch off root auto-discovery — dev's
vda1 carries the discoverable GUID 4f68bce3-e8cd-4db1-96e7-fbcaf984b709); clearing
the partition's GPT type GUID (a partition-table write to the live boot disk of a node
serving etcd and Longhorn replicas, and an irreversible decision about that 977 MiB
that belongs to the operator); `systemd.swap=0` (systemd-fstab-generator only — this
swap does not come from fstab).
`swapoff -a` is now conditional on `swapon --show` returning something, so the task
reports honestly instead of hiding a command that ran anyway behind changed_when: false.
PROVEN on k3s-prod-1 (all 18 Longhorn volumes healthy before the reboot):
run 1 changed=2, run 2 changed=0
after `reboot`: /proc/swaps empty, `free` Swap 0B, swap.target masked, the
generated unit still emitted by the generator and still masked, `systemctl
--failed` empty, `systemctl is-system-running` = running.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
08fbe85638 |
fix(ansible): move the founder designation to node 2
Node 1's SSD failed and was replaced; Debian was reinstalled, so node 1 is now a
JOINING node with no cluster state. It was still listed in [k3s_prod_primary],
and that group decides two things at once:
1. which play gets k3s_server_mode: primary -> --cluster-init, which FOUNDS A
NEW CLUSTER, and
2. what k3s_api_host resolves to -- the --server join target.
So re-provisioning node 1 unchanged would have pointed a rebuilt, empty node at
ITSELF and founded a second cluster beside the live one. The role's
`creates: /etc/rancher/k3s/k3s.yaml` guard does not help: on a fresh install that
file is absent, so cluster-init WOULD have run.
Node 2 is the senior surviving server (up since 2026-09-01) and takes the founder
slot. Its own cluster-init task is inert precisely because k3s.yaml already exists
there. Verified after the change: k3s_api_host renders 192.168.2.11, and
groups[k3s_primary_group][0] is k3s-prod-2.
This is a latent gap the role documented but did not handle -- its comment assumes
the founder is always alive ('On a WARM single-node rejoin the founder is up and
answering too'). Rebuilding the founder itself was not covered.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85
|
863af01062 |
docs: the §2 shared-device measurement came back, and it does not hold
infra/PROD-PROVISIONING.md §2 recorded the etcd/Longhorn shared-partition risk as "accepted to be MEASURED, not assumed", with an exit condition. The measurement arrived on 2026-09-07 in production, 2 d 7 h after bring-up, as an outage. §2 now carries "THE MEASUREMENT CAME BACK": the journal evidence, the three-node IO PSI table (11.36 / 0.75 / 1.29), and the honest verdict that the section's own premise — "every prod node is SSD or NVMe, where rebuild I/O costs latency rather than collapsing it" — was right about node 2 and wrong about node 1. A 2011 C400 with 61,440 reallocated sectors and no power-loss protection is not the "SSD" that reasoning assumed, and §0 had already named the three-way drive asymmetry as the fault line. The exit condition is OPEN and the second bullet (dedicated device) applies; that remains the operator's. Also records what the three software mitigations in this branch do and do not buy, including the point that no cgroup or scheduler control arbitrates the ext4 journal both tenants share — and the gap that etcd's own fsync histograms, which the exit condition literally asks for, are not collected because kubeEtcd is disabled for k3s. PSI is the signal that exists. DECISIONS-LOG gains the full entry: the four I/O mechanisms found INERT on these nodes (IOSchedulingClass, IOSchedulingPriority, io.latency, bare IOWeight), the finding that concurrentReplicaRebuildPerNodeLimit was already set and already held during the outage, the third instance of config that looked applied and was not (orphanAutoDeletion), and the three things still requiring the operator. infra/ansible/README.md: CORRECTED a stale claim that k3s-prod-2/-3 "stay commented out" of the inventory. They have been live in [k3s_prod_secondary] since 2026-09-03. That sentence is exactly the kind used to reason about blast radius — `-e k3s_hosts=k3s_prod` targets THREE hosts, not one — so it is replaced with the --list-hosts command that settles it. The new etcd-io-priority play is added to the run order, the layer table and the prod command list. Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit a00a27a2567ea6a805c8c196b75df2c92a7c63a4) |
||
|
jakman85
|
f7a9229d81 |
fix(ansible): the NVMe node would have died on an undefined variable
Self-review of the role I had just written. On k3s-prod-2 the scheduler path is
deliberately skipped, so the `stat` on /sys/fs/cgroup/system.slice/k3s.service is
skipped too — and a SKIPPED registration has no `.stat` key at all. Ansible
templates `loop:` BEFORE it evaluates `when:`, so
loop: "{{ ['system.slice'] + (['system.slice/k3s.service']
if _eio_k3s_cgroup.stat.exists else []) }}"
raises on that host even though the task would never have run. The role would have
failed on the one node it is designed to leave alone — and it would have failed at
the very end, after doing nothing, which reads like a broken mitigation rather than
an intentional skip.
`| default(false)` on both uses. Exercised offline against a simulated skipped
registration; `--syntax-check` re-run.
Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit 5ccf5111a54857abb934a64c41332b7aa5046323)
|
||
|
jakman85
|
a819c50ffe |
feat(ansible): arbitrate the shared etcd/Longhorn device — and prove it landed
roles/etcd_io_priority + playbooks/etcd-io-priority.yml, enabled for [k3s_prod]
only. Mitigation for the 2026-09-07 k3s-prod-1 outage: etcd's WAL fdatasync took
3.4-6.7 s (expects 1 s), the node went NotReady while the host stayed up, and
/proc/pressure/io full avg300 read 11.36 there against 0.75 (node 2, NVMe) and
1.29 (node 3).
WHAT IT DOES, on the SATA nodes only:
1. k3s-etcd-io-priority.service (oneshot, Before=k3s.service) switches the
shared device to BFQ and drops nr_requests 64 -> 32.
2. IOWeight=10000 drop-ins on system.slice AND k3s.service, which BFQ turns
into io.bfq.weight=1000 — a 10:1 share against the default-weighted
kubepods.slice where Longhorn's instance-managers live.
3. Reads all of it back out of the kernel and ASSERTS. If the scheduler did
not switch or the weight did not land, the play fails.
THE THREE OBVIOUS ANSWERS ARE INERT ON THESE NODES, ALL CHECKED LIVE, ALL
DELIBERATELY ABSENT:
* IOSchedulingClass=realtime on k3s.service. mq-deadline DOES honour the
priority class, so this looks right — but ioprio is inherited across fork
and kept across exec (block/blk-ioc.c __copy_io copies ioc->ioprio) and
CONTAINERD IS A CHILD OF THE K3S SERVER PROCESS. Verified on k3s-prod-1:
containerd pid 84553 has ppid 84333 = k3s-server, and
ionice -c2 -n0 -- sh -c 'ionice -p $$; sh -c "ionice -p \$\$"'
prints best-effort prio 0 at both levels. Realtime for k3s is realtime for
every container on the node, Longhorn included. No relative gain.
* IOSchedulingPriority (the level inside a class): mq-deadline buckets by
CLASS only, so BE-0 vs BE-4 is a no-op.
* io.latency, which would be the ideal scheduler-independent protection:
CONFIG_BLK_CGROUP_IOLATENCY is not built in Debian trixie's 7.1.8 kernel
(only IOCOST is) and no io.latency file exists under /sys/fs/cgroup.
AND ONE NON-OBVIOUS TRAP: bare IOWeight= is DECORATION here. systemd writes it
to io.weight (blk-iocost — compiled in but disabled, io.cost.qos is empty) and
io.bfq.weight (BFQ — not loaded). Neither reader exists on the stock nodes, so
the scheduler change and the weights have to be one role with one verification.
The value is 10000 rather than a "reasonable" 1000 for the same reason: systemd
rescales 1..10000 onto io.bfq.weight's 1..1000, so 1000 maps to ~101 against
BFQ's unweighted default of 100. The role asserts the result is >= 500.
k3s-prod-2 IS SKIPPED ON PURPOSE. The role discovers the device under
k3s_data_dir per node (verified: prod-1/3 -> /dev/sda5 -> sda, prod-2 ->
/dev/nvme0n1p5 -> nvme0n1) and writes NOTHING on NVMe rather than leaving inert
files behind — BFQ is a poor fit for NVMe and node 2 measured 0.75 IO PSI.
DEV IS UNTOUCHED. Master switch defaults false and is set true only in
group_vars/k3s_prod; dev has a dedicated etcd device (docs/plans/ssd-etcd-
isolation.md) and nothing to arbitrate. Verified with --list-hosts: no extra
vars -> the three dev hosts (where the role is a no-op); -e k3s_hosts=k3s_prod
-> the three prod hosts; --limit k3s-prod-3 -> one.
NOTHING IS RESTARTED. The oneshot writes two sysfs files; the weights go in
through daemon-reload. If daemon-reload turns out not to re-realize a running
k3s.service's cgroup, the verification FAILS and names the rolling restart as
the fix rather than quietly leaving the weight unapplied.
WHAT IT CANNOT DO, stated in the role README and worth repeating: it does not
separate the two tenants. Both still write one ext4 filesystem, and no cgroup
control arbitrates the shared jbd2 journal that etcd's fdatasync waits on. It is
also UNPROVEN on this hardware — it fails loudly if the settings do not land, it
cannot tell you they helped. Apply to k3s-prod-3 first and watch
NodeIOPressureHigh. Replacing the C400 / giving etcd its own device remains the
real fix and the operator's (PROD-PROVISIONING.md §2 exit condition).
Syntax-checked, and the fact-derivation and assert expressions were exercised
offline against all three node shapes plus negative controls (a node still on
mq-deadline fails the gate; IOWeight=1000 -> 101 fails the floor).
Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit de8c1f832724c011b03466842e18296de07a6776)
|
||
|
jakman85
|
3ca4ad6011 | merge: R2 off-cluster backups + per-environment backup credentials | ||
|
jakman85andClaude Opus 5
|
708a05dfc1 |
fix(devcontainer): restore multi-cluster kubeconfig selection
infra/devcontainer/kubeconfig.sh was never staged into the delivery-A commit, so the branch built on main carried main's single-cluster tooling and `--context k3s-prod` failed with 'context does not exist' mid-incident. Not Flux-reconciled, so it never affected the cluster — but it breaks prod diagnosis from a checkout of this branch, which is when you need it most. Carries the concurrency fix: the merged kubeconfig holds both contexts and NO current-context, with each invocation getting a per-invocation selector file prepended to KUBECONFIG. Two concurrent dev.sh runs with different CLUSTER values previously overwrote each other's current-context mid-command — reproduced live, a CLUSTER=dev reader returned k3s-prod nodes on 3 of 7 iterations. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
6a8953e288 |
feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing
Minimal prod delivery, authored directly on main (= prod's live revision
|
||
|
jakman85andClaude Opus 5
|
9739e9be91 |
chore(backup): R2 credentials for the off-cluster DR target
Written by platform/backup/set-r2-credentials.sh (operator, 2026-09-07T05:41:34Z) into all four consumers, in the two shapes they need: velero wants an AWS INI file, CNPG/barman wants discrete ACCESS_KEY_ID / ACCESS_SECRET_KEY. All four verified to carry the SAME 32-character key id, so no consumer was missed. Proven from a pod INSIDE prod against https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com: head-bucket on k3s-prod-backup returns BucketRegion WNAM, and PUT and DELETE both succeed. Byte-integrity of GET is NOT yet proven (the aws-cli image has no cmp); s3check is what will establish it. Credentials only — no endpoint or bucket wiring. That is feat/r2-backup-prod. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
d1a6628a97 |
feat(ha): the CNI dials the control-plane VIP — kube-vip becomes a CNI prerequisite
OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach
the cluster and provide fault tolerant HA. Without a change it will not work in HA."
THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live
before changing anything: all three prod cilium-agent pods carried
KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip
VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the
CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said
the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated
that stale comment as an open question when the evidence already settled it.
SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs:
k3s_api_host unchanged — the FOUNDER's IP: the cold-bootstrap join
target and a tls-san. infra/tofu/outputs.tf GENERATES this
name, so renaming would ripple into tofu and the runbooks.
k3s_apiserver_endpoint NEW = {{ kube_vip_address }} — the HA address; what Cilium
consumes. Derived, not restated, so an environment overrides
ONE line and the VIP, the SAN and the CNI endpoint move
together. Prod needed no new override.
THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at
the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran
node-baseline -> k3s -> cilium, so Cilium would install against an address nothing
announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value:
site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier
(kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there
because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing
neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so
a cold run stops THERE with a named cause instead of surfacing as unreachable agents.
Also fixed a pre-existing bug that only mattered once this play joined site.yml: the
role's read-only interface probe ran without `check_mode: false`, so under --check it
was skipped and the assert fired with a misleading "set kube_vip_interface explicitly".
VERIFIED BY ME, not just reported:
renders dev K3S_API_HOST=192.168.1.49 prod=192.168.2.9 (each environment's VIP)
cilium template consumes k3s_apiserver_endpoint, not k3s_api_host
cold join still `--server https://{{ k3s_api_host }}:6443` — the founder
order site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux
gate full CI gate PASS
Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing
k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders
['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved
— .9 already answers under full CA verification; and --check --diff of kube-vip.yml
against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the
kube-vip prod already runs.
ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine
checks: the Cilium template must use the endpoint var; the endpoint must derive from
kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal
that environment's VIP and be none of its node IPs (the [parity] contract, now
machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium.
Every new assertion mutation-tested and bites. One honest negative recorded: reverting
only the BASE default is not caught, because both overlays patch the key — the check
reads effective per-environment values, so the base default is unguarded.
APPLY ORDER, and this is the operator's step — nothing was applied:
PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let
Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet
ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01
incident), it will not proceed past an unready pod, existing datapath survives an
agent restart, and .9 is already live before the change lands.
DEV SEPARATELY, operator present. Same mechanics but live dev agents dial
192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that.
ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a
cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook.
NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live
KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that
test's gap list.
Refs R-PLAT-01, R-NET-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
c8c1d2935d |
feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).
THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.
Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.
PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.
BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.
TWO THAT CANNOT BE TOKENIZED, flagged not forced:
- headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
anyway — client secrets are realm-scoped.
- kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
prod.home.
EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.
I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.
DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.
L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.
PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.
Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
9b8979a427 |
feat(prod): temp Gitea source + wireless disabled + flux bootstrap path
Autonomous prod bring-up, operator away. Three changes that together clear the last blockers before `flux bootstrap`. === 1. TEMPORARY BOOTSTRAP GITEA — the last hard blocker, cleared === clusters/prod's source URL could not work: Gitea runs ON the dev cluster (the one slated for decommission) and split-horizon DNS resolves gitea.home.keithjackson.dev to PROD's OWN gateway. Stood up the existing bootstrap/gitea/bootstrap-gitea.sh on the operator's workstation with HOST_NAT_IP=192.168.2.132; all three prod nodes verified reachable (Gitea 1.22.6 via the API), no firewall change needed — prod is on the same /24, no NAT. clusters/prod now points at http://192.168.2.132:3000/... under a boxed TEMPORARY banner with a 5-step cutover checklist ending 'delete this whole block — dev's stanza is what done looks like'. Repo verified PUBLIC with anonymous read two independent ways (API "private":false; git ls-remote unauthenticated with GIT_TERMINAL_PROMPT=0), so NO secretRef is needed — documented so a future auth failure is not misread as a network fault. bootstrap-gitea.sh's header corrected: it documented HOST_NAT_IP=10.73.0.1, the RETIRED libvirt NAT, and forbade agent use; the operator has now directed this for prod. === 2. WIRELESS DISABLED ON PROD (operator-reported) === k3s-prod-3 was on ethernet AND wireless, and the wifi DHCP client was overwriting /etc/resolv.conf. Root cause of the silence: node_baseline's DNS `contended` detection checked resolvconf and NetworkManager but NOT dhcpcd, so the role wrote resolvers, reported success, and dhcpcd reverted them on lease renewal — the exact 'green run that fixed nothing' the contended branch exists to prevent. Now IaC, not a one-off: survey wireless hardware, **REFUSE to disable it if the path to the node runs over the wireless interface** (the guard that meant an agent dying mid-apply cost nothing), blacklist the driver so it never reloads, stop dhcpcd only where it holds a wireless interface and nothing else, take links down and unload modules with NO reboot, then report the resulting state. dhcpcd is now in the contended set without weakening fail-closed. Applied and verified: node 3's wlp1s0 is GONE from /sys/class/net, dhcpcd inactive and its unit not-found, ONE default route per node, all three on 8.8.8.8/1.1.1.1 resolving public names. Safe and idempotent on nodes 1 and 2, whose wifi hardware was already down. === 3. flux_bootstrap PATH PARAMETERISED === bootstrap/flux/bootstrap.sh already read FLUX_ENV_PATH but the role never exported it, so it ALWAYS bootstrapped clusters/dev — prod was impossible. Two behaviour-bearing lines: role default clusters/dev + `environment: FLUX_ENV_PATH`, overridden to clusters/prod in group_vars/k3s_prod. Proven per host with a read-only Ansible probe: prod hosts resolve clusters/prod, localhost/hypervisor resolves clusters/dev. Only two definitions repo-wide. DEV PROVEN BYTE-IDENTICAL for the source change: all 12 substituting Flux paths rendered for both trees, 27/27 SHA-256 artefacts identical, with TWO POSITIVE CONTROLS (an overlay-level canary and a deep-path content canary) confirming the harness detects change — and both canaries proven restored. Honest scope note recorded: kustomize strips comments, so the harness proves the built stream, not source comments. L0 substitution check: 106 assertions, 0 hazards. ci/l0.sh and ci/lint.sh exit 0. === STILL BLOCKING A FULL PROD RECONCILE (operator decisions) === - S3_BACKUP_ENDPOINT remains TBD-PROD-*: prod CANNOT reach dev's versitygw at 192.168.1.9:9000 (verified UNREACHABLE from all three — the pfSense boundary), and a prod backup target is a deferred operator decision. Deliberately NOT invented: a DR target on a workstation container would create false confidence. Velero/CNPG backups will fail loudly. - Prod's public resolvers mean the API server's oidc-issuer-url resolves PUBLICLY, so kubectl SSO will not work on prod until split-horizon is solved for it (§4). - The temp Gitea is a single container on the operator's workstation — a single point of failure for prod reconciliation, acceptable for bring-up only. Serves: R-GITOPS-01, R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
05baae90ec |
feat(prod): kernel fix + disk layout — prod nodes ready for k3s
Prod is being stood up now, without waiting for node 2's RAM upgrade. Three coupled
changes, all applied and verified on all three nodes.
=== 1. KERNEL >= 6.13 — now a PREREQUISITE, not a nice-to-have ===
A k3s SERVER that intermittently boots with 2.6 GiB is not a degraded node; it fails under
etcd plus its workload share, unpredictably, on any reboot. Fixed before k3s goes on.
roles/node_baseline gains a `kernel`-tagged block: asserts the suite matches the node's
Debian release, writes a deb822 trixie-backports source, installs, reads back the exact
version, writes a provenance/rollback note to the node. Scoped by
node_baseline_backports_kernel (default FALSE, true only in group_vars/k3s_prod) — chosen
over a hostname test so a fourth prod node inherits it from group membership. Verified:
--check against dev skips all 8 tasks on all 3 hosts. Two deliberate calls: `state: latest`
not `present` (linux-image-amd64 was already installed from main, so `present` would have
silently fixed nothing), and NO preferences.d pin (backports' NotAutomatic +
ButAutomaticUpgrades already gives 100/500; an explicit Pin-Priority: 100 would switch OFF
automatic security upgrades for the very kernel being installed).
Installed 7.1.8+deb13-amd64 on all three; 6.12.107 kept bootable.
THE DECISIVE PROOF: node 2 booted at MEMATTR=0xbeeb0018 — the EXACT address that gave it
2.6 GiB twice — and now reserves 634,904K instead of 13,958,980K. MemTotal 15.4 GiB. Same
address, handled correctly; not a lucky draw.
§0b's claim that a fixed kernel prints 'FW_BUG Corrupted...' is CONDITIONAL and was
corrected: >=6.13 has three rejection gates and only the third prints that line; all three
nodes hit gate 1 (version > 2). Proof instead is boot-independent — decompressing
/boot/vmlinuz on all three yields the FW_BUG and descriptor-size strings verbatim; neither
exists in 6.12.
TWO MORE BUGS IN MY OWN HEALTH CHECK, found because the fix couldn't otherwise be shown:
- MEMATTR_VERSION_RE matched (\d+) but the kernel prints that field with %d — SIGNED. Node
2's real value was -1569980416, which did not match, so state fell through to `clean` and
the node reported PASS "the kernel accepted it without complaint". A FALSE PASS on the
exact bug the check exists to find. Now (-?\d+).
- bad_version was WARN unconditionally, so a FIXED node would have stayed WARN forever.
Now keys on memattr_kernel_fixed: same line, WARN below 6.13, PASS at/above.
=== 2. DISK LAYOUT — Longhorn claimed, per-node paths DISCOVERED not hardcoded ===
Operator decision: reuse the ~424 GB partition; accept the shared-device I/O risk (dev's
etcd/Longhorn contention was on a spinning HDD, prod is SSD/NVMe).
Device paths differ per node (sda5 / nvme0n1p5 / sda5), so the role DISCOVERS the partition
via `findmnt --mountpoint` on a declared claim mountpoint rather than trusting inventory.
Per-host host_vars would have frozen today's hardware into inventory, where it goes stale on
the first re-image and the stale value's failure mode is mkfs on the WRONG partition.
Eleven fail-closed asserts before any write: real block device, not backing / /boot /usr
/var /home /tmp or active swap (built from the node's OWN gathered facts, never a hardcoded
list), already ext4, no foreign label, >= 100 GiB, and empty. Relabel, never reformat.
/srv verified empty first on every node (ls -A -> lost+found only; du 20K; the "5.1% used"
was ext4's default 5% root reserve).
=== 3. k3s DATA-DIR MOVED — /var was 11.8 GiB and would have killed etcd ===
Measured on the fully-deployed dev cluster: containerd 21-25 GB/node, /var total 58-70 GB.
Prod's /var is 11.8 GiB and held BOTH etcd and the image store, so prod would have filled
/var during its first image pulls — and a full /var means etcd cannot write its WAL. Cluster
death, not degradation, arriving during bring-up. Invisible when the disk decision was made,
because "etcd on root" sounded like the 30 GiB / rather than a separate 12 GiB /var.
Now: p5 -> /data (LABEL=k3s-data), with /data/longhorn and /data/k3s. Neutral names chosen
deliberately — the partition now has two tenants, and reusing /srv would make "claimed" and
"unclaimed" indistinguishable to the idempotency pivot, while a bare `data` label would make
a stranger's `data` volume look already-claimed and weaken an assert.
The transition off the existing LABEL=longhorn mount is DECLARED, not inferred
(node_baseline_data_retired_fs_label/_retired_mount, defaults ""), because changing the label
makes a claimed node answer "no" to the pivot and walk into the greenfield path. Self-retiring
once blkid -L k3s-data succeeds. THREE independent gates keep live dev out: retired vars
default ""; gated on longhorn_source == claim-mountpoint (dev is `device`); and the emptiness
assert fails closed on dev's 18 volumes even if both were defeated.
k3s_data_dir is one variable read by two roles (k3s_server --data-dir; node_baseline derives
the etcd mount). Role default is k3s's own default, so dev is unchanged and keeps its
dedicated etcd device. node_baseline_etcd_source becomes `shared-data-partition` and is
LOAD-BEARING: the role asserts per run that the filesystem backing the etcd dir really IS the
data partition — had the mount silently failed, etcd plus 21-25 GB of images would land back
on the OS filesystem invisibly until the disk filled.
LONGHORN_DATA_PATH is a new cluster-vars knob (R-ENV-02): base and DEV /var/lib/longhorn,
prod /data/longhorn.
DEV PROVEN UNCHANGED, which is the property that mattered — the live cluster has 18 Longhorn
volumes on that path. All 12 substituting Flux paths rendered through
`kubectl kustomize | flux envsubst --strict` for both trees: 12/12 per-file SHA-256 identical
and an identical aggregate SHA over every rendered dev byte. With a POSITIVE CONTROL showing
the same harness detects prod changing on exactly one line — an all-identical result is
worthless if the harness cannot see change. Live check after: dev's default-data-path setting
still /var/lib/longhorn, 18 volumes attached, 0 Kustomizations unready.
Idempotent: run 2 is changed=0 on all three nodes. Also fixed a real tag bug — include_tasks
does NOT propagate its own tags to included tasks (unlike import_tasks), so `--tags disks`
selected the include and ran nothing inside it while reporting success.
OPEN DEFECTS, tracked not forgotten: node 2's warm-reboot hang (one clean reboot since, NOT a
fix); and node 3 runs dhcpcd for its wifi interface, which rewrites /etc/resolv.conf on lease
renewal — the role's `contended` detection checks resolvconf and NetworkManager only, so node
3's decided resolvers will silently revert.
Gate: MERGE GATE: PASS. L0 substitution check 106 assertions.
Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-STATE-01, R-STATE-02, R-PLAT-01. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
8e913fdd7b |
fix(node_health): "Failed to map" does NOT mean memory lost — plus survey results
THE MODEL WAS WRONG AND MY CHECK WITH IT. A live counter-example: k3s-prod-2 after its power
cycle shows MEMATTR=0xbeeb0018 and 'Failed to map ... @ 0xbeeb0018' — the SAME address and SAME
line as both of its 2.6 GiB boots — with MemTotal 15857732 kB, i.e. 15.1 GiB of 16 GB. FULL
MEMORY. The check reported FAIL 'THIS NODE HAS LOST MEMORY TO THE BUG' on a healthy node while
memory.physical_total correctly PASSed at 4.2%. The two checks contradicted each other.
REFINED MECHANISM — two kernel functions were conflated:
efi_memattr_init() does the memblock_reserve(). Catastrophic loss needs tbl_size NEGATIVE
(signed int -> sign-extended -> clamped to PHYS_ADDR_MAX). A garbage-but-POSITIVE tbl_size
reserves a modest wrong amount and costs nothing noticeable.
efi_memattr_apply_permissions() does the later memremap(), and THAT is what prints
'Failed to map'. It fails for ANY sufficiently wrong size, negative or merely large-positive.
So the line proves the table was corrupt AND TRUSTED, not that memory was lost. The cost is set
by the garbage VALUE, which is why one address gave 2.6 GiB twice and 15.1 GiB once.
THIS ALSO REFUTES THE ADDRESS->OUTCOME HYPOTHESIS OUTRIGHT — not untested, disproven.
STRUCTURAL FIX so the contradiction cannot recur: the shortfall is computed ONCE into a shared
fact (node_health_mem_shortfall) with worst-wins sub-verdicts; memory.physical_total IS that
verdict and firmware.efi_memattr READS it, measuring nothing itself. FAIL is gated on the shared
verdict. 'Failed to map' + normal memory is now WARN ('the bug fired and this boot survived it').
No honesty guard was weakened; one was strengthened (the measurement clause falls back to the DIMM
shortfall when zoneinfo is unreadable, and only claims 'could NOT be measured' when neither was
read). All five paths incl. the two unproducible-live SKIPs exercised against synthetic facts.
Verdicts: prod-1 WARN, prod-2 WARN (was FAIL), prod-3 PASS, dev unaffected.
SECOND FAULT, arguably worse for prod: NODE 2 HANGS ON A WARM REBOOT, STAYING POWERED. Its
journal ends cleanly mid-shutdown ('eno1: NIC Link is Down') then stops; the box sat powered but
off-LAN for ~38 min until the operator held the power button. A node that cannot survive an
unattended warm reboot cannot be rolled in an HA cluster, and node_health cannot see a machine
that is off the network. Leading candidate: node 2 is the ONLY node with 'Recover OS after Boot
Failure = Enable' — promoted from 'coincidence' in the BIOS diff to a live availability defect.
The reboot survey is NOT viable unattended on node 2 until this is understood.
SURVEY RESULTS (40 boots): prod-1 0/20, prod-3 0/20, prod-2 0 ROUNDS (hung on its first reboot),
pooled 0/40 -> p(bad) 95% CI 0.000-0.088. NOT elimination: 0/20 still permits 17% per node, and
node 2's prior 2-of-3 is unchanged. What it did establish: addresses are node-specific and highly
concentrated (node 1 used 3 distinct addresses in 20 boots, 15 at one; no address ever shared
between nodes); MEMATTR -> parse was a PERFECT function across all 7 repeated addresses over 37
boots; and the corrupting bytes are stable PER MACHINE (node 1 always 1030, node 3 always
1463964749), so the 'shared OS boot chain' inference needs refining.
Adds playbooks/memattr-survey.yml + roles/memattr_survey — reboots opt-in, resumable, artifacts
gitignored. Documents an Ansible defect worth knowing repo-wide: a looped include_tasks under the
default 'linear' strategy is flattened per host, so hosts run SERIALLY; the first attempt rebooted
node 1 twenty times while the others idled. Fixed with strategy: free + a forks preflight assert.
Serves: R-ENV-01, R-ENV-03, R-OBS-04, R-TEST-02. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
b19c5db9ae |
feat(prod): node 3 onboarded — and it is NOT immune; BIOS-setting hypothesis disproven
k3s-prod-3 online at the planned 192.168.2.12 (serial 8CC015307N, same chassis/BIOS/CPU),
enabled in inventory, vaulted become confirmed, package floor installed. Health: 0 FAIL /
4 WARN / 14 PASS / 1 SKIP — the only node with zero FAILs. WD Blue WDS500G2B0A SATA SSD at
100% life, 3213 h, 0 grown bad blocks, 0 erase fails. Matched Crucial 2x16 GB pair — the only
node with symmetric memory.
*** NODE 3 IS NOT IMMUNE TO THE §0b BUG. *** Its firmware DOES publish a MAT
(MEMATTR=0xbeece018); the ABSENCE of a memattr diagnostic line means the kernel read a VALID
table, which is the opposite of immunity. There are THREE outcomes, not two: bail-out on a
garbage version >2, a valid parse with a sane size, and the catastrophic path. Seven boots across
three machines show the MAT address MOVES BOOT-TO-BOOT on every node with settings constant, and
each node has been seen on more than one path. Blast radius widens from two nodes to ALL THREE;
all three need the >= 6.13 kernel.
THE CORRUPTING BYTES ARE THE OS BOOT CHAIN, not firmware and not DRAM: garbage version
1463964749 = 0x5742544D appeared on node 2 AND node 3 — two different machines, identical bytes.
The overwriter is the shared shim/GRUB/EFI-stub/initrd placement writing into the page the
firmware pointed at. So initrd size, module set and boot device are part of the causal chain.
BIOS-SETTING HYPOTHESIS: DISPROVEN, and the three-way diff cannot support it even in principle.
All 240 attributes read on all three; firmware identical fleet-wide. The discriminating set (both
working nodes agree, broken differs) is seven attributes, of which:
- Fast Boot is the ONLY live candidate, and only as a probability-shifter — HP documents it
'skips some preboot initialization steps', which changes the heap top and hence where the MAT
lands. Different mechanism from the previously-killed MRC-caching theory; that stays dead.
- Legacy Boot Options is AFFIRMATIVELY INERT: 'Configure Legacy Support and Secure Boot' =
'Legacy Support Disable and Secure Boot Enable' on ALL THREE, Option ROM Launch Policy =
'All UEFI' on all three. No CSM is built on node 2 either. A naive diff gets this wrong.
- the rest are coincidence. After Power Loss is ELIMINATED by node 3, which works and shares
node 2's Power Off — though it is now a §1 checklist violation on TWO nodes.
FATAL CONFOUND: the 1=3!=2 partition is perfectly collinear with node 2 being the only
single-DIMM AND the only NVMe-booting node. Adding node 3 did not break the confound, it DOUBLED
it. No setting can be the cause anyway: node 2 booted good once and bad twice with settings held
constant.
EXPERIMENT DESIGN RECORDED because the obvious approach is statistically worthless: p(bad) is
known only to n=3, 95% CI ~0.09-0.99, so even 8 clean boots prove nothing. Elimination needs ~20
boots/arm; an odds shift needs ~50. Zero-touch first step: 20 reboots of node 2 unchanged, logging
MEMATTR address vs outcome — both failures were at exactly 0xbeeb0018 and all five successes
elsewhere, so if outcome maps onto a narrow address range the mechanism is settled without
touching anything.
ROLE BUG FIXED: node 3's WD drive reports a SMART threshold for only 1 of 25 attributes, and
smartctl OMITS the key rather than emitting 0 — summary.txt.j2 dereferenced a.thresh bare in five
places and the report task died, so no artifact was written. Added a thr() macro rendering 'n/a'
(deliberately not 0, which would read as a real ATA cannot-trip threshold). Would have hit any
drive of this class.
Also: three nodes, three different disk models, only one NVMe, and every node has a single disk
with no unpartitioned spare — so §2's identical-layout requirement is unmet on all three.
Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85
|
6e4d311545 |
docs(prod): will 6.12.y get the memattr fix? Researched — not on its own
Operator asked whether waiting is viable or whether a backports kernel is permanent. Answer: it will NOT arrive on its own, but it CAN be made to, and the ask is unusually strong. VERIFIED: - NOT in linux-6.12.y. Newest commit touching memattr.c on that branch is 2024-07-12. In mainline since v6.13 (2025-01-19) — ~19 months, ~97 point releases. - NO stable tag and NO Fixes: tag on either commit. Structural, not an oversight: the unvalidated arithmetic dates to the file's creation in 2016, so it is not a regression and there is no bad commit to point Fixes: at. BOTH automatic feeds — Fixes:-scanning and AUTOSEL — had nothing to grab. AUTOSEL never surfaced it for any branch. - NOBODY has requested it: nothing on [email protected], nothing in all 8,027 open+archived Debian bugs against src:linux. - 6.12 LTS EOL is December 2028 — outlives Debian 13's support window, so not time-barred. - Debian REBASES: trixie carries 6.12.94-1 in trixie-security ('New upstream stable update'), so an upstream pickup would reach us via a normal apt upgrade. THE DECISIVE CARD: the cherry-pick is BYTE-CLEAN. Debian trixie 6.12.94-1's memattr.c hashes to 164203429fa7a..., identical to the upstream patch's pre-image blob — the file has not diverged between v6.12 and the commit's parent. cherry-pick applies with zero conflicts and zero adjustment, which removes the usual reason such requests get declined. The commit also meets every stable-kernel-rules criterion: mainline, 17 insertions, 'hardware quirk', 'a real bug that bothers people'. PLAN: backports now as explicitly provisional; email [email protected] (rules Option 2) CC linux-efi and Ard Biesheuvel who authored AND committed it as EFI maintainer, leading with the clean-apply proof and the three-boot reproduction with byte-identical e820; Debian BTS as secondary under handbook §9.1.7; make the deviation permanent only if there is no movement in 4-6 weeks. Only 8fbe4c49c0cc is needed by hand — 5d0faa8e8369 loosens a heuristic that was rejecting LEGITIMATE tables and is not required for this symptom. Serves: R-ENV-01, R-ENV-03. Task: T-7.1. |
||
|
jakman85
|
0a733294ff |
docs(prod): node 1's DIMMs are not a matched pair
Surfaced by the new node_health DIMM reporting, unrelated to the memattr bug and NOT a fault: DIMM1 is a Micron 8ATF2G64HZ-3G2E1 (DDR4-3200), DIMM3 a Micron 16ATF2G64HZ-2G6E1 (DDR4-2667). Both 16 GB so capacity is right, but different rank organisation and rated speed mean the pair runs at the slower 2667. Recorded in §0 where the symmetry decision gets made, because §2 requires the three nodes to end up with identical layouts and the previous '2x16 GB Micron' note hid the mismatch. Serves: R-ENV-03. Task: T-7.1. |
||
|
jakman85andClaude Opus 5
|
cf33338448 |
feat(node_health): detect the EFI memattr bug on every run, two independent ways
The fault is BOOT-TO-BOOT NON-DETERMINISTIC — the same machine booted correctly two boots
earlier — so it has to be detected on every health run, not diagnosed once. And node 1 is
exposed to the identical lottery on every reboot while currently looking healthy.
firmware.efi_memattr — reads the kernel ring buffer and distinguishes six states:
Failed to map EFI Memory Attributes table @ 0x.. -> FAIL (memory lost NOW)
Unexpected EFI Memory Attributes table version <N> -> WARN (corrupt table still published;
intact only by luck)
[Firmware Bug]: Corrupted EFI ... detected! -> PASS (the >=6.13 fix working)
no memattr line, log reaches early boot -> PASS (no table / accepted / not UEFI)
no memattr line, log does NOT reach early boot -> SKIP
dmesg unreadable -> SKIP
memory.physical_total — deliberately a SEPARATE check, not folded in: it measures the SYMPTOM
with no kernel log at all, so an unreadable-dmesg node is still not unmeasured. Two sub-verdicts,
worst wins: /proc/zoneinfo present-vs-managed (world-readable, works on metal and VMs — sum of
managed IS MemTotal) and MemTotal vs the DIMM total from dmidecode. Thresholds 10% WARN / 25%
FAIL, set from measured baselines: 2.0% prod-1, 2.1% workstation, 2.2% dev VM, vs 83.4% prod-2.
THREE GUARDS AGAINST A FALSE PASS, because "no memattr line" must never silently mean "fine":
1. dmesg failure yields available:false with the exact reason (rc, uid, kernel.dmesg_restrict) -> SKIP
2. the ring buffer WRAPS and every memattr line prints in the first fraction of a second, so the
collector records a boot_covered anchor (Linux version / BIOS-provided physical RAM map); no
memattr line WITHOUT that anchor is a SKIP, not a PASS
3. memory.physical_total needs no kernel log at all
Verified by forcing the unprivileged path: SKIP with the EPERM reason while zoneinfo still PASSed.
VERDICTS, confirmed live:
k3s-prod-1 WARN "EXPOSED, not immune ... survived only because its garbage version field read
1030, which is > 2 ... one unlucky boot from the failure state" + PASS 2.0%
k3s-prod-2 FAIL "THIS NODE HAS LOST MEMORY TO THE BUG ... at 0xbeeb0018" + FAIL 83.4%/83.6%
k3s-dev-* PASS both, NO false finding: "no UEFI config-table line ... did not boot via UEFI.
Log reaches back to early boot, so that absence is measured, not assumed."
Both failure texts carry the remediation verbatim, including that there is NO kernel cmdline
workaround because efi_memattr_init() runs before any efi= bail-out.
Root-gating follows the role's existing pattern: --privileged re-runs THE SAME collector under
become, only when the unprivileged pass actually failed to read something. Reimplementing the log
grep and SMBIOS parse in Jinja would have been a second source of truth for the same fact.
LATENT BUG FIXED in passing: the collector's run() used bare shutil.which(), so dmidecode (in
/usr/sbin) was invisible under the unprivileged k3s PATH and produced a FALSE "not installed"
skip reason. Now shares the sbin-aware which() collect_tools() already had.
INCIDENTAL FINDING recorded in §0b: node 1's DIMMs are NOT a matched pair — DIMM1 Micron
8ATF2G64HZ-3G2E1 (DDR4-3200), DIMM3 Micron 16ATF2G64HZ-2G6E1 (DDR4-2667). Both 16 GB so capacity
is right, but different rank organisation and rated speed, so the pair runs at 2667. Not a fault;
recorded because §2 requires the three nodes to end up symmetric and "2×16 GB Micron" hid it.
Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85
|
47f02c6e06 |
docs(prod): second independent confirmation + no cmdline workaround exists
A second research pass reached the same root cause by a different route and pinned the arithmetic. Recording it because convergence from two independent lines is what turns this from a good hypothesis into a settled diagnosis. MECHANISM TO THE BYTE: - the garbage 'version' node 2 read was 1463964749 = 0x5742544D = ASCII 'MTBW'. The kernel is reading unrelated data. HP publishes the MEMATTR pointer into memory /sys/firmware/memmap calls plain System RAM, so the pre-Linux boot chain overwrites it. - tbl_size is a SIGNED int; num_entries * desc_size wraps mod 2^32 and goes NEGATIVE when bit 31 lands. memblock_reserve sign-extends it and memblock_cap_size clamps to PHYS_ADDR_MAX - base, so EVERYTHING above 0xbeeb0018 is reserved. - memblock_free_all then hands the allocator nothing above that line while present_pages (from memblock.memory) is untouched -> present 3340288 / managed 0. - 'Failed to map EFI Memory Attributes table' is emitted LATER by efi_memattr_apply_permissions(); it is positive PROOF the reservation already happened, a consequence rather than the cause. CLEANEST A/B TELL, one dmesg line: NODE_DATA(0) on prod-2 ends at 0xbeeb0000 — pgdat forced immediately BELOW the reservation — while on prod-1 it ends at 0x82f800000, the top of RAM. Identical 0x2a980 allocation, opposite placement. THERE IS NO KERNEL CMDLINE WORKAROUND. efi_memattr_init() is called from efi_config_parse_tables(), and x86's efi_init() calls efi_config_init() BEFORE the efi_runtime_disabled() bail-out — so noefi / efi=noruntime / efi=novamap do NOT prevent the reservation. A fixed kernel is the only real fix. EFI_MEMORY_SP / soft-reserved was the strongest competing theory and is EXCLUDED BY MEASUREMENT: /sys/firmware/memmap/10/type reads 'System RAM'. Soft-reserved would print as 'Soft Reserved' and is never memblock_add()ed, which would have left present at ~0 rather than 3340288. Also records that the hardening landed exactly one release late for Debian 13 (bug present since 2016; 8fbe4c49c0cc first in v6.13; follow-up 5d0faa8e8369) and that a Debian BTS report against src:linux is warranted — 6.12 is Debian 13's kernel for the release lifetime. Serves: R-ENV-01, R-ENV-03. Task: T-7.1. |
||
|
jakman85
|
49d82491dd |
docs(prod): node 2 memory SOLVED — Linux 6.12 EFI memattr bug, NOT a hardware fault
I framed this as an OPEN HARDWARE FAULT and blamed, in order: the DIMM, BIOS memory remapping,
and stale Fast Boot memory-training data. ALL THREE WERE WRONG. The hardware is fine.
ROOT CAUSE: HP's R21 firmware publishes a CORRUPT EFI Memory Attributes Table, and Linux 6.12
fails to validate it before reserving memory from its contents. drivers/firmware/efi/memattr.c
never validates num_entries or desc_size:
if (tbl->version > 2) { pr_warn(...); goto unmap; }
tbl_size = sizeof(*tbl) + tbl->num_entries * tbl->desc_size;
memblock_reserve(efi_mem_attr_table, tbl_size);
Garbage version > 2 -> kernel bails, memory intact. Garbage version <= 2 -> kernel multiplies two
garbage fields and reserves GIGABYTES from the table address to the top of RAM.
THE EVIDENCE THAT SETTLES IT: the same machine — same DIMM, same BIOS, same kernel, same cmdline —
booted with the FULL 15.9 GiB two boots earlier, with a byte-identical BIOS-e820 map. Boot -2:
15,917,744K available, 615,208K reserved, 'Unexpected EFI Memory Attributes table version
1463964749'. Boots -1/0: 2.6 GiB, 13.9 GiB reserved, 'Failed to map ... @ 0xbeeb0018'. No hardware
fault is intermittent in that shape or fixed by a reboot. MemTotal = DMA(3840) + DMA32(684481)
pages = 2,753,284 kB exactly, Normal contributing zero.
BOTH PROD NODES ARE AFFECTED. k3s-prod-1 prints 'Unexpected EFI Memory Attributes table version
1030' — also garbage, but >2, so the kernel bails and it keeps its 32 GiB BY LUCK, NOT IMMUNITY.
It is one unlucky boot from the identical fault. Dev nodes publish no memattr table at all
(Proxmox OVMF) and are unaffected.
FIX: kernel >= 6.13 — upstream 8fbe4c49c0cc 'efi/memattr: Ignore table if the size is clearly
bogus', whose commit message describes this verbatim. Verified absent from v6.12 and from current
linux-6.12.y. On Debian 13: linux-image-amd64 from trixie-backports.
DO NOT buy RAM, move the DIMM, RMA the board, or run memtest expecting failures. Rebooting,
disabling Fast Boot, clearing CMOS and unplugging USB are all the same lottery — they move where
the table lands and may LOOK like fixes.
Records every killed hypothesis with its disproof so none of them get re-run.
Serves: R-ENV-01, R-ENV-03. Task: T-7.1.
|
||
|
jakman85
|
5ea9c93f58 |
docs(prod): BIOS password answer + CORRECT the 0x4 'auth failure' reading
Operator asked for the default BIOS password. ANSWER: there is none, and no factory default exists to look up. HP business desktops ship with no Setup and no Power-On password; the is_enabled=0 we measured IS the factory state. Recovery for a forgotten password is the PSWD jumper — and 'stringent security' must NOT be enabled, since HP documents that it removes the physical bypass and turns a 5-minute fix into a signed per-serial HP service case. CORRECTION, and the kernel driver misled me: I recorded that the firmware refused our write for lack of BIOS-admin authentication. Wrong. hp-bioscfg passes the return of HP's WMI SetBIOSSetting through hp_wmi_error_and_message(), which is THE ERROR TABLE FOR A DIFFERENT INTERFACE (the "SECU" commands under HP_WMI_BIOS_GUID). Per HP's Client Management Interface white paper, SetBIOSSetting 4 = 'Failed' (generic) and 6 = 'Access Denied'. We got 4, not 6 — and the driver has a separate BIOS_ADMIN_NOT_SET = 0x38 -> -EPERM for the no-password case, which we also did not get. 'Invalid command type' is the driver looking 4 up in the wrong table: a genuine upstream bug worth reporting to platform-driver-x86. Most likely real cause (inferred, testable): with an empty password the driver emits an UNTAGGED empty UTF-16 string, while every HP-documented caller passes Password="<utf-16/>" even when blank — the driver only prepends that prefix for a non-empty string. Setting a password may fix writes, but by flipping onto the tagged code path, NOT because 0x4 meant unauthorised. Also verified from source: hp-bioscfg CANNOT set a password. current_password and new_password both funnel into store_password_instance(), which only strscpy's into driver RAM; new_password is write-only into a buffer nothing ever reads. Count corrected: 241 attributes exposed, 239 with a current_value, 215 non-empty. I had quoted 215 as the total; it was the non-empty subset. Recommendation recorded: land read-only drift detection now (zero risk, no new credential, and it already caught real drift); treat a Setup password as a deliberate experiment on node 3 only; hp-repsetup as fallback but it needs Secure Boot off or a self-signed module; skip Sure Admin/BCU at this scale. Serves: R-ENV-03, R-SEC-01. Task: T-7.1. |
||
|
jakman85
|
3a88abb958 |
docs(prod): iGPU ruled out as the cause; CORRECT my last_pfn misreading
Operator asked whether node 2's missing memory could be reserved for integrated graphics. TESTED, and no: /proc/iomem shows 'c8800000-cc7fffff : Graphics Stolen Memory' = EXACTLY 64 MiB, matching the BIOS Video Memory Size=64 MB setting, and it sits BELOW the 4 GiB line. The UHD 630 (i915 bound) takes precisely what it is configured to take. It cannot account for 13.3 GiB and is not even in the affected address range. CORRECTION to my own analysis, prompted by checking that hypothesis properly: I cited the 'last_pfn revised DOWN from 0x42f800 to 0xc3c10' line as evidence of a memory cap. That was a MISREADING. x86 boot prints last_pfn twice — first max_pfn from the full e820, then max_low_pfn, the end of RAM below 4 GiB. The e820 low usable region ends at 0xc3c0ffff, so the next PFN is exactly 0xc3c10, which is the second value observed. It matches to the byte and is normal on any machine with more than 4 GiB. I presented a routine boot message as a smoking gun, and the 'memory remapping above 4 GB not being applied' signature claim rested partly on it. The real anomaly is narrower than first stated: the Normal zone is 'present' but 'managed 0', 13.3 GiB is reserved, and nothing in the boot log explains by what. Ruled out so far: DIMM health, firmware inventory, memory-remap settings (none exist on this platform), MTRR trim, iomem registration, and now integrated graphics. Leading remaining hypothesis is stale memory-training data cached by Fast Boot — testable with one F10 toggle. Serves: R-ENV-03. Task: T-7.1. |
||
|
jakman85
|
9f7052ab2b |
docs(prod): fill in node-2 inventory; drop stale investigate bullets
Records node 2's as-found hardware alongside node 1 and removes two bullets that my earlier edit left duplicated/obsolete (the BIOS comparison is done; clear-CMOS was listed twice). Makes the hardware ASYMMETRY explicit in the table rather than buried in prose: node 1 is a ~2011 Micron C400 SATA SSD at 69% life with 15 grown bad blocks and 187 erase failures; node 2 is a Samsung PM9A1 NVMe at 96% life, PASSED. Same chassis, same BIOS, same CPU, same partition layout — very different storage. §2 requires an identical disk layout across all three nodes because Longhorn replica placement assumes symmetry, so this belongs in the inventory where the disk decision will be made. Serves: R-ENV-03. Task: T-7.1. |
||
|
jakman85
|
3c8f809aaf |
docs(prod): CORRECTION — BIOS attributes are read-only in practice, not writable
I recorded that the hp-bioscfg attributes were 'writable from the OS' because current_value has mode rw and no BIOS passwords are set. That was WRONG, and the firmware said so when the write was actually attempted: hp_bioscfg: Returned error 0x4, "Invalid command type" (surfaces as printf: I/O error) HP requires BIOS-admin authentication for programmatic changes and neither credential is provisioned on these nodes — Setup Password is_enabled=0 (role bios-admin), SPM key_mechanism=not provisioned. Secure Boot is also enabled. File mode is not authority; the firmware is. What survives the correction, and it is the useful half: READS WORK. Diffing 215 attributes between node 1 and node 2 is exactly how the drift in §0b was found — nobody spots 'After Power Loss=Power Off' by eye. So the §1 firmware checklist IS machine-CHECKABLE today with reads alone, and node_health should assert it. What does NOT survive: machine-ENFORCEABLE. Converging BIOS settings automatically would require provisioning a BIOS Setup Password on every node — a stored credential that can rewrite firmware. That is a real security decision and the operator's call, not something to default to. Consequence for the node-2 memory fault: setting Fast Boot=Disable and After Power Loss=Power On must be done in F10 on the box. Updated §0b and §1 accordingly. Serves: R-ENV-03. Task: T-7.1. |
||
|
jakman85
|
11b21f7347 |
docs(prod): BIOS settings compared programmatically via hp-bioscfg
Both nodes load the hp-bioscfg driver, exposing 215 BIOS attributes read/write at /sys/class/firmware-attributes/. No BIOS passwords are set, so they are writable FROM THE OS. RULED OUT as the cause of node 2's 2.6 GiB limit: firmware inventory is correct (BIOS itself reports Memory Size=16384 MB); no memory-remap/above-4G attribute exists on this platform at all; MTRRs are effectively identical with no trim warning; /proc/iomem registers the full 100000000-42f7fffff as System RAM and 127 memory blocks report online. REAL DIFFERENCES FOUND (node1 works / node2 broken): After Power Loss Power On / Power Off <- VIOLATES this document's own §1 Fast Boot Disable / Enable <- leading hypothesis Legacy Boot Options Disable / Enable TPM Activation Policy F1 to Boot / Allow user to reject MS+Windows UEFI CA 2023 Enable / Disable Leading hypothesis: HP Fast Boot reuses CACHED MEMORY-TRAINING DATA rather than retraining each boot. A DIMM fitted while Fast Boot was on is a plausible route to memory being present, inventoried, handed to the kernel — and then reserved. It is also the only memory-relevant setting differing from the working node. NOT TESTED: changing it is a firmware write plus a reboot and needs operator sanction. BROADER FINDING for T-7.1: the §1 firmware checklist is now MACHINE-CHECKABLE rather than an eyeball-at-F10 exercise, and machine-ENFORCEABLE too. That is precisely how node 2 came to have After Power Loss=Power Off while node 1 has Power On — nobody could see the drift. Follow-up recorded: extend node_health to assert the firmware checklist, add a role to converge it. Raw BIOS dumps deliberately NOT committed — they carry serials, UUIDs and MAC addresses, same reason node-health artifacts are gitignored. Only the findings are recorded. Serves: R-ENV-03. Task: T-7.1. |
||
|
jakman85andClaude Opus 5
|
cb69a53825 |
feat(prod): enable node 2 + HARDWARE FAULT — it addresses 2.6 GiB of its 16 GB
k3s-prod-2 (192.168.2.11) came online 2026-09-01. Enabled in the inventory, baseline package
floor installed, first health run completed. Vaulted become works on it unchanged.
*** OPEN FAULT: the node can only address ~2.6 GiB of a healthy, correctly-installed 16 GB
DIMM. *** Recorded in PROD-PROVISIONING §0b.
dmidecode 16 GB Samsung M471A2K43DB1-CTD in DIMM1 @ 2667 MT/s (DIMM3 empty)
BIOS e820 reports ~16 GiB usable, incl. 0x100000000-0x42f7fffff
kernel Memory: 2568544K/16546088K available (13958980K RESERVED)
last_pfn revised DOWN 0x42f800 (~16.7 GiB) -> 0xc3c10 (~3.06 GiB) in early boot
zones Normal present 3340288 pages (~12.7 GiB) but MANAGED 0
MemTotal 2,753,284 kB = 2.6 GiB
The DIMM is NOT the problem: it is present, correctly identified, and at rated speed. The
kernel sees all 16 GB and then reserves 13.3 GiB. Everything below ~3 GiB usable, everything
above 4 GiB present-but-unmanaged — the signature of memory remapping above 4 GB not being
applied, not of failing memory. Node 1 is the control: identical chassis, identical BIOS R21
Ver. 02.26.00, identical CPU, and it addresses its full 31 GiB. Per-machine state, not a
firmware-version defect.
WARNING RECORDED: do NOT assume the incoming 32 GB fixes this. If the fault is the slot, the
board or a BIOS setting rather than the DIMM, 32 GB presents identically and the money is
wasted. Verify MemTotal after any RAM change.
ALSO FOUND — the prod nodes are NOT hardware-symmetric:
node 1 Micron C400 512GB SATA SSD (~2011) — 69% life, 15 grown bad blocks, 187 erase
failures, 32 unexpected power losses
node 2 Samsung PM9A1 512GB NVMe — 96% life, 2142 power-on hours, 34 unexpected
power losses, overall-health PASSED
Same chassis, same BIOS, same CPU, same partition layout — very different storage. §2 requires
an identical disk layout across all three nodes because Longhorn replica placement assumes
symmetry, so this asymmetry is now a live input to the disk decision rather than a footnote.
Node 2's NVMe is the drive node 1 should have had.
Node 2 otherwise: 0 FAIL / 3 WARN / 12 PASS / 2 SKIP. Same three WARNs as node 1 (swap still
active pending full baseline, single IPv4 resolver, and the NVMe's 34 unexpected power losses),
plus the same unknowable-ECC SKIP.
Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
6b4da27f84 |
feat(prod): SOPS decryption (same age key as dev) + GitOps source on main
Operator decisions, 2026-09-01:
1. Prod uses the SAME SOPS age key as dev.
2. Prod tracks the SAME BRANCH as dev (`main`). Dev is a temporary instance and will be
decommissioned once prod is deemed stable.
SOPS: eight `decryption: {provider: sops, secretRef: sops-age}` patches mirroring dev
exactly — platform, apps-wave-0/1/2, oauth2-proxy, headlamp, crafty, velero. The sops-age
Secret is still created OUT OF BAND on prod (bootstrap/sops/install-age-key.sh); the
overlay references it by name only, never inlines key material.
RECORDED CONSEQUENCE of the shared branch: once prod is live, EVERY MERGE TO main
RECONCILES BOTH CLUSTERS. There is no promotion step and no soak window — dev and prod
move together. That is a deliberate consequence of dev being temporary, not an oversight,
but it removes the "prove it on dev first" safety net that the last few days have leaned
on heavily. Worth revisiting if dev outlives its expected life.
*** BOOTSTRAP BLOCKER recorded, NOT papered over *** — the source URL cannot work on a
cold prod cluster, for two independent reasons found while wiring this:
1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER — the one slated for
decommission. Prod syncing from it means prod's source of truth lives inside the
cluster being retired. Gitea must move to prod (or off-cluster) BEFORE dev is torn
down. This makes Gitea migration the FIRST task of the decommission plan, not the last.
2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml
answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP} via a template
plugin. On prod that is 192.168.2.20 — PROD's own gateway, where no Gitea listens. Prod
Flux would resolve gitea.home.keithjackson.dev to itself and never reach a repository.
This is the same split-horizon mechanism that makes dev work, inverted against us.
Bring-up therefore needs a TEMPORARY source (the bootstrap host-Gitea, or an off-cluster
mirror) with a cutover to the in-cluster URL once Gitea actually runs on prod — precisely
the sequence docs/05 describes for dev's own Phase-6 cutover. The branch is settled and
correct; only the host is blocked. Documented at the patch site and in
PROD-PROVISIONING.md §7.
Gate: MERGE GATE: PASS.
Serves: R-SEC-01, R-GITOPS-01, R-ENV-02, R-ENV-03. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|