Compare commits

...
13 Commits
Author SHA1 Message Date
jakman85 a624c638ef merge: docs/node-1-drive-replaced
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
2026-09-07 23:05:59 -05:00
jakman85 90b39de527 merge: fix/swap-off-persists-across-reboot 2026-09-07 23:05:59 -05:00
jakman85 0247d796a3 merge: fix/founder-moved-to-node-2 2026-09-07 23:05:59 -05:00
jakman85 c9e9dcf960 merge: fix/r-env-03-overlay-selected-credentials 2026-09-07 23:05:59 -05:00
jakman85 89cf59ffb7 docs(conformance): the GIVEN/WHEN/THEN header states the exception it now allows 2026-09-07 23:04:39 -05:00
jakman85 e319777849 docs(pm): draft the R-ENV-03 amendment, and name the gate hole that hid it
The requirement's literal reading ("no prod-specific artifact in a shared path")
forbids the environment-selected PAIR that R-ENV-02 prescribes for a value that
must differ per environment while the manifests stay identical — and a Secret's
difference cannot live in cluster-vars, because it is ciphertext in a SOPS file.
Drafted wording that keeps the guard's teeth (an artifact only prod APPLIES
still fails) and removes the false positive, for the System Architect: the spec
repo is read-only here and CI-gated by R-TEST-06.

Also recorded, checked rather than assumed: the merge that turned R-ENV-03 red
was gated with a touched-requirement subset that never selected R-ENV-03, so
ci/l0.sh Stage 3b filtered the already-red check out of the run that printed
PASS. Replaying that delivery's twelve paths through
fix/touched-requirements-fail-closed's requirement-paths.map still yields no
R-ENV-03, so landing that branch would NOT have caught this. The two changes
that would (unconditional whole-repo L0 stages; a changed conformance test
attributing its own requirement labels) are ci/ work and are left to that
branch, not smuggled in here.
2026-09-07 23:02:56 -05:00
jakman85 730d12ddc4 fix(conformance): R-ENV-03 shared-path rule tests the harm, not the filename
The blanket regex around RENDERED_DIRS failed any tracked file whose name
carries a delimited `prod` token. That caught the four per-environment backup
credential halves (BACKUP_CREDENTIALS_SUFFIX) even though BOTH halves are
applied to BOTH clusters as base resources and the overlay selects only which
one the shared consumer REFERENCES — i.e. exactly the R-ENV-02 contract the
regex exists to protect. Renaming the files to dodge the regex was rejected as
gaming the guard, so the rule now encodes the property instead.

The fix is not an allowlist of those four paths. A prod-token file in a shared
path is excused only when the tree proves all of:

  (A) it declares nothing but Secrets whose names end in the environment suffix
  (1) its own directory's kustomization applies it, and EVERY environment value
      of the selector resolves to a Secret that some resource of that same
      kustomization declares, in the same namespace
  (2) nothing names the suffixed Secret — or the suffixed file — literally;
      every reference is `<base-name>${VAR}`
  (3) that VAR is defined for the base and for every overlay, and prod's value
      is the one that selects this file

References are read from parsed YAML values, never raw text: the mechanism is
spelled out by name in comments beside the resources, and a comment references
nothing.

No other assertion is weakened; the check still runs 96 assertions, and the
pass output now notes each excused file and why.
2026-09-07 22:55:58 -05:00
jakman85andClaude Opus 5 df1d2ab44d docs(prod): node 1's drive was replaced — and §2's exit condition still is not met
Records the 2026-09-07 I/O-starvation incident and the drive swap: the C400 was
replaced with a Samsung 850 EVO, and under the same load class (an active
18-replica Longhorn rebuild, with node 1 as the write target) I/O pressure fell
from full avg300 11.36 to 3.79 — from ~4x worse than its peers into node 3's band.

Updates the three §0 cells that had become false (disk model, drive health, kernel
— all three nodes now run 7.1.8, not 6.12) and adds §0c rather than rewriting the
'NOT hardware-symmetric' analysis, which is still the correct reasoning about the
state it described.

Two things the rebuild taught, recorded so the next one is cheaper: Longhorn
refuses a replaced disk with DiskFilesystemChanged and needs disable -> remove ->
re-add in that order, and the inventory's [k3s_prod_primary] decides BOTH
--cluster-init and k3s_api_host, so rebuilding the founder without moving that
designation would have founded a second cluster.

The replacement does NOT close §2: etcd and Longhorn still share one SATA spindle.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 22:50:45 -05:00
jakman85andClaude Opus 5 69a25cff82 fix(node_health): say 'active via <unit>', not a bare parenthetical
The unit name in the swap-active detail read as an unlabelled aside. Naming it as
the unit that activated the swap is what makes the line actionable.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 22:42:38 -05:00
jakman85andClaude Opus 5 c1e36d9353 docs(node_baseline): say why the Longhorn claim guard's swap list is now always empty
The claim guard builds node_baseline_lh_protected_devices from ansible_mounts plus
`swapon --show`. With swap.target masked, a baselined node has no active swap, so
that half of the list contributes nothing — which reads like a guard that quietly
stopped guarding. It is not: the claim candidate comes from whatever is mounted at
node_baseline_longhorn_claim_mountpoint, and a swap partition is neither mounted nor
ext4, so it could never be the candidate. Say so where the next reader will ask.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 22:39:39 -05:00
jakman85andClaude Opus 5 9d00a3fcd3 fix(node_health): memory.swap blamed the wrong thing, and could not see the defect
The WARN read "roles/node_baseline runs swapoff and neutralises the fstab entry;
this node has not been baselined yet". That was a MIS-DIAGNOSIS. All three prod
nodes HAD been baselined, had their fstab swap line commented out by the role, and
were still swapping 977 MiB each after a reboot. Anyone reading that WARN would
have re-run the role that had already run, and watched swap come back next boot.

The detail now names the actual mechanism — systemd-gpt-auto-generator re-creating
a .swap unit from the partition TYPE GUID 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f at
every boot, without reading /etc/fstab — says in as many words that this is not
"the node has not been baselined", and points at `node-baseline.yml --tags system`.

The check was also blind in the other direction: SwapTotal == 0 is not evidence
that swap is disabled, only that nothing has been swapped on yet this boot, so a
node freshly swapoff-ed PASSed while still set to swap again on reboot — exactly
the state the old role left behind. The collector now reports memory.swap_persistence
(swap.target masked?, .swap units the generators emitted this boot, which of them
are masked, leftover active fstab swap lines — all world-readable, no root, no
systemctl parsing), and the check WARNs on "clean now, swapping after the next
reboot".

Rendered against every fact shape (swap active + unmasked, swap off + unmasked,
after the fix, a dev VM with no swap partition at all, per-unit mask only,
persistence unreadable, an older collector's output with the key absent, and
/proc/meminfo unreadable) and verified live on k3s-prod-1 post-reboot:
  [PASS] memory.swap — no swap active and none can be activated: swap.target is masked

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 22:35:31 -05:00
jakman85andClaude Opus 5 0e92d258dd fix(node_baseline): swap-off now survives a reboot — mask swap.target
The role's `swapoff -a` + commented fstab line held only until the next boot,
on every prod node. Measured 2026-09-07: all three came back from a reboot with
977 MiB of swap ACTIVE (k3s-prod-1 /dev/sda4, k3s-prod-3 /dev/sda4, k3s-prod-2
/dev/nvme0n1p4) while their fstab swap entry was commented out by this role. The
role reported success and the setting silently expired.

CAUSE. systemd-gpt-auto-generator claims any GPT partition typed
0657fd6d-a4ab-43c4-84e5-0933c84b4f4f ("Linux swap") from the PARTITION TABLE,
synthesises a .swap unit into /run/systemd/generator.late/ and links it into
swap.target.wants/. It never reads /etc/fstab, so neutralising fstab cannot stop
it. Re-running the generator by hand on k3s-prod-1 says so verbatim:
  Probed fstype 'swap' on partition /dev/disk/by-diskseq/1-part4.
  Adding swap: /dev/disk/by-diskseq/1-part4

MECHANISM. Mask swap.target. `systemctl show` on the generated unit says its only
inbound dependency is `WantedBy=swap.target` — Requires/RequiredBy/BoundBy empty,
and the matching .device unit has an empty Wants=, so a device appearing does not
pull its swap unit in by itself. swap.target is the single choke point for every
activation path: gpt-auto, systemd-fstab-generator, systemd-cryptsetup-generator.
It is device-shape independent (no device name appears in the change, so one task
covers sda4, nvme0n1p4 and dev's vda VMs), boot-safe (swap.target is Wants=/After=
from sysinit.target and After= from tmp.mount; RequiredBy= is EMPTY on all three
prod and all three dev nodes, and a masked unit reached only via Wants= is skipped,
not failed), and inert if the device is absent or renamed. The generated units are
masked too, as a second independent barrier.

Rejected: `systemd.gpt_auto=0` (disables GPT auto-discovery wholesale, needs a GRUB
edit + reboot to become true, and on DEV would switch off root auto-discovery — dev's
vda1 carries the discoverable GUID 4f68bce3-e8cd-4db1-96e7-fbcaf984b709); clearing
the partition's GPT type GUID (a partition-table write to the live boot disk of a node
serving etcd and Longhorn replicas, and an irreversible decision about that 977 MiB
that belongs to the operator); `systemd.swap=0` (systemd-fstab-generator only — this
swap does not come from fstab).

`swapoff -a` is now conditional on `swapon --show` returning something, so the task
reports honestly instead of hiding a command that ran anyway behind changed_when: false.

PROVEN on k3s-prod-1 (all 18 Longhorn volumes healthy before the reboot):
  run 1 changed=2, run 2 changed=0
  after `reboot`: /proc/swaps empty, `free` Swap 0B, swap.target masked, the
  generated unit still emitted by the generator and still masked, `systemctl
  --failed` empty, `systemctl is-system-running` = running.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 22:35:16 -05:00
jakman85andClaude Opus 5 08fbe85638 fix(ansible): move the founder designation to node 2
Node 1's SSD failed and was replaced; Debian was reinstalled, so node 1 is now a
JOINING node with no cluster state. It was still listed in [k3s_prod_primary],
and that group decides two things at once:

  1. which play gets k3s_server_mode: primary -> --cluster-init, which FOUNDS A
     NEW CLUSTER, and
  2. what k3s_api_host resolves to -- the --server join target.

So re-provisioning node 1 unchanged would have pointed a rebuilt, empty node at
ITSELF and founded a second cluster beside the live one. The role's
`creates: /etc/rancher/k3s/k3s.yaml` guard does not help: on a fresh install that
file is absent, so cluster-init WOULD have run.

Node 2 is the senior surviving server (up since 2026-09-01) and takes the founder
slot. Its own cluster-init task is inert precisely because k3s.yaml already exists
there. Verified after the change: k3s_api_host renders 192.168.2.11, and
groups[k3s_primary_group][0] is k3s-prod-2.

This is a latent gap the role documented but did not handle -- its comment assumes
the founder is always alive ('On a WARM single-node rejoin the founder is up and
answering too'). Rebuilding the founder itself was not covered.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 21:41:54 -05:00
10 changed files with 671 additions and 22 deletions

No files matched your search

+135
View File
@@ -1954,3 +1954,138 @@ not one.** Confirmed by `--list-hosts` (dev default → 3 dev hosts, where the n
Nothing was run against either cluster for this change.
3. **Decide on `kubeEtcd` metrics**, so §2's exit condition can be measured with etcd's own fsync
histograms rather than inferred from node PSI.
## 2026-09-07 — R-ENV-03's L0 check was RED on `main`, and the requirement's literal wording is why (branch `fix/r-env-03-overlay-selected-credentials`)
`main`'s `conformance/tests/l0-static/prod-baremetal-not-dev-surface` failed 4 of 96 assertions.
All four are the per-environment backup credentials merged in `e7c4a95`:
```
apps/wave-0/nextcloud-db-backup-credentials-prod.sops.yaml
apps/wave-0/semaphore-db-backup-credentials-prod.sops.yaml
platform/keycloak/keycloak-db-backup-credentials-prod.sops.yaml
platform/velero/velero-cloud-prod.sops.yaml
... is a prod-specific file inside the shared configuration
```
**The guard was a filename regex.** Any tracked file under `platform/ platform-config apps/
catalog/ clusters/base/` whose name contained a delimited `prod` token failed, with no allowlist
and no exception mechanism. Nothing about the four files' CONTENT was inspected.
**Renaming them to dodge the regex was rejected** (operator, explicitly): the guard exists for a
reason and a rename would leave the reason unserved while making the check lie. So the check was
refined to test the harm instead of the spelling, and the requirement wording is escalated below.
### Why these four files are compliant with R-ENV-02, and why the regex could not see it
Backup credentials became per-environment because dev backs up to MinIO and prod to Cloudflare
R2, and ONE shared Secret meant either environment's key rotation broke the other (worse for
CNPG: `archive_mode=on` turns a 403 on WAL archive into a monotonically filling pgdata PVC).
The mechanism is a single cluster-vars knob, `BACKUP_CREDENTIALS_SUFFIX` (base `""`, dev `""`,
prod `"-prod"`). **Both Secret objects are applied to BOTH clusters** as base resources; the
overlay changes only which one the shared consumer REFERENCES
(`<name>${BACKUP_CREDENTIALS_SUFFIX}`). Every environment renders the identical manifests and the
identical object set. That is exactly the "same manifests, overlay-scoped values" contract of
R-ENV-02 — nothing is forked, and there is no prod-only content for anyone to develop against.
The `-prod` in a filename is not a prod-only manifest; it is one half of an environment-selected
pair, present everywhere.
### What the check now asserts (committed on the branch; not a path allowlist)
A file whose name carries the environment token is excused ONLY when the tree proves all of:
- **(A)** it declares nothing but Secrets whose names end in the environment suffix — an overlay
can select which SECRET a shared manifest references and nothing else, so any other
prod-suffixed object in a shared path is prod-only content and still fails;
- **(1)** its own directory's `kustomization.yaml` lists it in `resources` (so EVERY environment
applies it), and EVERY environment's value of the selector resolves to a Secret that some
resource of that same kustomization declares, in the same namespace — no environment is left
selecting an object that does not exist;
- **(2)** nothing names the suffixed Secret — or the suffixed FILE — literally: every reference is
`<base-name>${VAR}`, and only the applying kustomization may name the file;
- **(3)** that `VAR` is defined for `clusters/base` AND for every `clusters/<env>` overlay
(StrictPostBuildSubstitutions fails on an UNDEFINED variable, so an overlay that forgot it
would stop that cluster reconciling), and prod's value is the one that selects THIS file.
References are read from PARSED YAML VALUES, never raw text — the mechanism is documented by name
in comments beside the resources, and a name in a comment references nothing.
Mutation-tested: eleven mutations, each restored afterwards, each of which must FAIL and does — a
prod-only ConfigMap in a shared path, a prod-only Secret with no counterpart, a consumer
hardcoding prod's Secret name, a kustomization naming prod's file from another directory, the
knob missing from dev, the knob missing from base, prod's value drifted to a suffix nothing
declares, the prod half dropped from `resources`, the base half deleted (with and without its
`resources` entry), the halves split across namespaces, and one of two consumer references moved
to a second knob. The legitimate four pass, with a `note:` per file saying why they were excused.
### PROPOSED R-ENV-03 AMENDMENT — for the System Architect (spec repo is READ-ONLY here)
**What the requirement says today** (`docs/00-vision-and-scope.md`, detail in `docs/02`): prod is
a 3x mini-PC bare-metal k3s cluster with local storage and **"MUST NOT be an active development
surface until the roadmap declares it ready"**; `docs/02` adds **"Never develop against prod.
Prod hardware is a target, not a sandbox (R-ENV-03)."**
**The problem with the literal reading.** Taken literally against the repository, "no prod-only
content in the shared configuration" reads as "no artifact in a shared path may be specific to
prod". That forbids the environment-selected PAIR, which is the very pattern R-ENV-02 prescribes
for a value that must differ per environment while the manifests stay identical. The two
requirements contradict each other on this case: R-ENV-02 says put the difference in an
overlay-scoped value, and a Secret's difference cannot live in cluster-vars (it is ciphertext, per
SOPS file, R-SEC-01), so it must live in a per-environment FILE that every environment applies.
The requirement should forbid what is actually harmful — configuration only prod runs — not the
appearance of an environment name in a path.
**Proposed wording** (add as a clarifying clause to R-ENV-03 in `docs/02`, and mirror the phrase
in `docs/00`):
> Prod MUST NOT be an active development surface. Concretely: no artifact in the shared
> configuration may be reachable by prod alone. An artifact whose NAME carries an environment
> token is not by itself a violation — provided every environment APPLIES it, no shared manifest
> references it by its environment-qualified name, and the selection between environment
> counterparts is made solely by an overlay-scoped cluster-vars value that is DEFINED for every
> environment. An artifact that only prod applies, or that a shared manifest names literally, IS
> a violation, whatever it is called.
That wording keeps the guard's teeth (an artifact only prod applies still fails), removes the
false positive, and states the property the check now enforces so the test and the spec agree.
**Related spec ask, same clause.** R-ENV-02 should say out loud that a per-environment SECRET is
expressed as counterpart SOPS files applied everywhere plus a cluster-vars selector, because that
is the one value class that cannot be expressed as a cluster-vars value itself. Without it, the
next person hits the same contradiction.
### How this reached `main` green — the gate hole, and whether the fix in flight closes it
`ci/l0.sh` Stage 3b scopes the L0 conformance run to the TOUCHED requirement subset
(`load_touched_requirements` -> `run.sh --requirements ...`). The delivery `e7c4a95` touched
`apps/wave-0/`, `catalog/render.py`, `platform/keycloak/`, `platform/velero/` and declared no
R-ID on a non-comment line, so the path map attributed R-GITOPS-02 / R-IDP-01..06 / R-STATE-03 —
**and never R-ENV-03**. `prod-baremetal-not-dev-surface` was therefore FILTERED OUT of the run
that gated the merge. The gate printed PASS having never executed the check that was already red.
**Landing `fix/touched-requirements-fail-closed` would NOT have caught this.** Checked, not
assumed: that branch's `ci/requirement-paths.map` classifies every path (so nothing is silently
unmapped any more), but it attributes R-ENV-03 only to `infra/ansible/*` and `infra/tofu/*`.
Replaying `e7c4a95`'s twelve paths through that map yields
`R-GITOPS-01 R-GITOPS-02 R-IDP-01..06 R-PLAT-02 R-STATE-03` — still no R-ENV-03. Its `ci/l0.sh`
change also leaves Stage 3b scoped (only the substitution stage stays unconditional). The
fail-closed branch fixes the "unmapped path contributes nothing" hole; it does not fix this one.
**Two things that would have caught it, for the operator/architect to choose between:**
1. **Whole-repo L0 invariants should be unconditional.** `flux-postbuild-substitution`,
`keycloak-client-secret-reconciler` and `spec-immutability` already run as their own
unconditional stages for exactly this reason. `prod-baremetal-not-dev-surface`,
`no-plaintext-secrets` and `least-privilege-rbac` are the same shape — they assert over the
WHOLE tracked tree, so scoping them to a changed path is meaningless. Cheapest correct fix.
2. **A changed test should attribute the requirements it declares.** `conformance/*` maps to
R-TEST-01..04 only, so editing `prod-baremetal-not-dev-surface/check.sh` does not select
R-ENV-03 — this very branch's gate run reports `touched requirements: R-ENV-02 R-TEST-01
R-TEST-02 R-TEST-03 R-TEST-04` and ran 2 of 12 L0 checks, NOT including the one it changed.
A changed `conformance/tests/<test>/` should contribute that test's own
`conformance.home.k3s/requirements` label.
Both belong to `ci/`, so they are left to that branch rather than smuggled in here. The full L0
layer was run unscoped on this branch to compensate: **12 offline checks, 11 PASS, 1 SKIP**
(`placeholder-guard-coupling`, pre-existing), 0 FAIL.
@@ -48,7 +48,10 @@
# sync.yaml, every patch it applies targets an ENVIRONMENT-SCOPED object
# (cluster-vars, a Flux Kustomization's sops decryption, or the Git source),
# prod's Git ref is the SAME ref dev tracks (no prod-only branch to develop on),
# and no prod-only manifest exists anywhere in the rendered configuration;
# and no prod-only manifest exists anywhere in the rendered configuration — a file
# whose NAME carries the environment token is excused ONLY when the tree proves it
# is an OVERLAY-SELECTED half that every environment applies (see the four
# conditions at "THE ONE STRUCTURAL EXCEPTION" below); anything else still fails;
# (6) prod artifacts exist only WITH a declaration of readiness: the spec roadmap
# carries the phase that opens prod build (checked when the spec repo is mounted),
# and the repository records that the operator opened it.
@@ -437,12 +440,172 @@ tracked = [f for f in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
capture_output=True, text=True, check=True
).stdout.split("\0") if f]
RENDERED_DIRS = ("platform/", "platform-config", "apps/", "catalog/", "clusters/base/")
for f in tracked:
if f.startswith(RENDERED_DIRS) and re.search(rf"(^|[/_.-]){re.escape(prod_token)}([/_.-]|$)",
f, re.I):
check(False,
f"{f} is a prod-specific file inside the shared configuration — every environment "
"renders these paths, so prod-only content here is development against prod")
# THE ONE STRUCTURAL EXCEPTION: AN OVERLAY-SELECTED ENVIRONMENT HALF.
# ------------------------------------------------------------------
# A filename carrying an environment token is a HEURISTIC for the harm R-ENV-03 names
# (content only prod runs, in a path every environment renders). It is not the harm
# itself. There is exactly one shape where a `-prod` name is the OPPOSITE of a prod-only
# fork: a credential that is TWO Secret objects, BOTH applied to BOTH clusters as base
# resources, where a cluster-vars knob decides only which one the shared consumer
# REFERENCES (clusters/base/cluster-vars.yaml, BACKUP_CREDENTIALS_SUFFIX: dev backs up to
# MinIO, prod to Cloudflare R2, and one shared Secret meant either environment's rotation
# broke the other). Nothing is forked: every environment renders the identical manifests
# and the identical object set, which is precisely the R-ENV-02 contract.
#
# So the exception is not a path allowlist — it is the PROPERTY that makes that shape
# safe, asserted from the tree, and a file is excused ONLY when all of it holds:
#
# (A) it declares nothing but Secrets whose names end in the environment suffix — the
# only kind of object this repo has a selection mechanism for;
# (1) it is listed in its own directory's kustomization.yaml, so EVERY environment
# applies it, and every environment's value of the selector resolves to an object
# that some resource of that same kustomization declares (same namespace) — no
# environment is left selecting a Secret that does not exist;
# (2) nothing references the suffixed name literally: every reference is a substituted
# expression `<base-name>${VAR}`, and the suffixed FILE is named only by the
# kustomization that applies it — so no shared manifest hardcodes prod's half;
# (3) that VAR is defined for the base AND for every overlay (StrictPostBuildSubstitutions
# fails on an UNDEFINED variable, so an overlay that forgot it would break that
# cluster's reconciliation), and prod's value is the one that selects THIS file.
#
# References are read from PARSED YAML VALUES, never from raw text: the mechanism is
# documented by name in comments beside the resources (cluster-vars says so explicitly),
# and a name in a comment references nothing. Anything that fails any condition still
# fails, naming the condition it broke.
ENV_SUFFIX = f"-{prod_token}"
# Every environment that renders the shared configuration.
ENVS = ["base"] + sorted(p.parent.name
for p in (ROOT / "clusters").glob("*/kustomization.yaml")
if p.parent.name != "base")
def _docs(rel: str):
p = ROOT / rel
if not p.is_file():
return None
try:
return [d for d in yaml.safe_load_all(p.read_text(errors="replace"))
if isinstance(d, dict)]
except Exception:
return None
def _strings(node):
if isinstance(node, str):
yield node
elif isinstance(node, dict):
for k, v in node.items():
yield from _strings(k)
yield from _strings(v)
elif isinstance(node, (list, tuple)):
for v in node:
yield from _strings(v)
RENDERED_YAML = [f for f in tracked
if f.startswith(RENDERED_DIRS) and f.endswith((".yaml", ".yml"))]
VALUE_STRINGS = {f: list(_strings(_docs(f) or [])) for f in RENDERED_YAML}
def overlay_selected(f: str) -> list[str]:
"""[] if f is an overlay-SELECTED environment half; else the conditions it breaks."""
docs = _docs(f)
objs = [((d.get("kind")), (d.get("metadata") or {}).get("name"),
(d.get("metadata") or {}).get("namespace")) for d in (docs or [])]
if not objs or any(k != "Secret" or not isinstance(n, str) or not n.endswith(ENV_SUFFIX)
for k, n, _ in objs):
return [f"(A) it does not consist solely of Secret objects whose names end in "
f"{ENV_SUFFIX!r} — an overlay can select which SECRET a shared manifest "
f"references, and nothing else, so any other prod-suffixed object in a "
f"shared path is prod-only content"]
reasons: list[str] = []
d = str(Path(f).parent)
kfile = f"{d}/kustomization.yaml"
kdocs = _docs(kfile) or []
res = [str(r) for r in ((kdocs[0].get("resources") if kdocs else None) or [])]
if Path(f).name not in res:
reasons.append(f"(1) {kfile} does not list it in `resources`, so it is not applied "
f"in every environment — one environment would carry an object the "
f"other does not")
# Every Secret the applying kustomization declares: name -> namespace.
declared: dict[str, object] = {}
for r in res:
for dd in (_docs(f"{d}/{r}") or []):
if dd.get("kind") == "Secret":
md = dd.get("metadata") or {}
declared[str(md.get("name"))] = md.get("namespace")
for _, name, ns in objs:
base_name = name[:-len(ENV_SUFFIX)]
pat = re.escape(base_name) + r"\$\{([A-Za-z_][A-Za-z0-9_]*)\}"
sel = sorted({m for ss in VALUE_STRINGS.values() for s in ss
for m in re.findall(pat, s)})
# (2) literal references to the suffixed OBJECT — anywhere but this file, except
# the applying kustomization naming this FILE (which is how it gets applied).
lit = sorted({f2 for f2, ss in VALUE_STRINGS.items() for s in ss
if name in s and f2 != f
and not (f2 == kfile and s == Path(f).name)})
if lit:
reasons.append(f"(2) {', '.join(lit)} names the Secret {name!r} literally — a "
f"shared manifest that hardcodes prod's half makes every "
f"environment use prod's credential; reference "
f"{base_name}${{...}} instead")
# (2) literal references to the suffixed FILE from anywhere but its kustomization.
stray = sorted({f2 for f2, ss in VALUE_STRINGS.items() for s in ss
if Path(f).name in s and f2 not in (f, kfile)})
if stray:
reasons.append(f"(2) {', '.join(stray)} names the file {Path(f).name!r} "
f"literally — only {kfile} may apply it")
if not sel:
reasons.append(f"(2) no manifest references {base_name}${{...}}: nothing selects "
f"between the halves, so this file is prod-only content that "
f"only prod could ever consume")
continue
if len(sel) > 1:
reasons.append(f"(2) {base_name} is selected by more than one variable {sel} — "
f"they can drift apart and leave one consumer on the other "
f"environment's credential")
var = sel[0]
vals = env_values(var)
missing = [e for e in ENVS if e not in vals]
if missing:
reasons.append(f"(3) ${{{var}}} is not defined for {missing} (defined for "
f"{sorted(vals)}) — StrictPostBuildSubstitutions fails on an "
f"undefined variable, so that environment would stop "
f"reconciling, or silently inherit the other's credential")
for e, v in sorted(vals.items()):
want = base_name + v
if want not in declared:
reasons.append(f"(1) environment {e!r} selects Secret {want!r} via "
f"${{{var}}}={v!r}, which no resource of {kfile} declares — "
f"that environment renders a dangling reference")
elif declared[want] != ns:
reasons.append(f"(1) environment {e!r} selects Secret {want!r}, declared in "
f"namespace {declared[want]!r}, but this half is in {ns!r} — "
f"the halves are not interchangeable")
if prod_overlay and prod_overlay in vals and base_name + vals[prod_overlay] != name:
reasons.append(f"(3) the prod overlay's ${{{var}}}={vals[prod_overlay]!r} selects "
f"{base_name + vals[prod_overlay]!r}, not this file's {name!r} — "
f"prod's half is applied but no environment selects it")
return reasons
flagged = [f for f in tracked
if f.startswith(RENDERED_DIRS)
and re.search(rf"(^|[/_.-]){re.escape(prod_token)}([/_.-]|$)", f, re.I)]
for f in flagged:
why = overlay_selected(f)
check(not why,
f"{f} is a prod-specific file inside the shared configuration — every environment "
f"renders these paths, so prod-only content here is development against prod. It is "
f"not an overlay-selected environment half either: " + " AND ".join(why))
if not why:
note(f"{f} carries the {prod_token!r} token but is an OVERLAY-SELECTED half: both "
f"halves are applied in every environment and only the cluster-vars selector "
f"differs (R-ENV-02), so it is not prod-only content")
check(len(tracked) > 50, f"git ls-files returned only {len(tracked)} paths — not a real checkout")
# --------------------------- (6) prod work exists only with a declaration of readiness
+44 -3
View File
@@ -28,9 +28,9 @@ Fill a row per node as it lands. Node 1 is as-found on 2026-08-31.
| Model | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini |
| CPU | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x |
| RAM | 31 GiB (2×16 GB Micron) | **2.6 GiB usable of 16 GB — see §0b** | 31.1 GiB usable of 32 GB (2×16 GB Crucial `CT16G4SFD8266.C16FE`, **a matched pair**) |
| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`, ~2011) | 1× 477 GB **NVMe** (`PM9A1 Samsung`) | 1× 466 GB SATA SSD (`WDC WDS500G2B0A-00SM50`) |
| Drive health | 69% life, 15 grown bad blocks, 187 erase fails | **96% life**, 2142 h, PASSED | **100% life**, 3213 h, PASSED, 0 grown bad blocks / 0 erase / 0 program fails, 0 CRC |
| OS | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12.107+deb13-amd64 |
| Disks | 1× 466 GB SATA SSD (`Samsung SSD 850 EVO 500GB`) — **replaced 2026-09-07, was `C400-MTFDDAC512M` (~2011); see §0c** | 1× 477 GB **NVMe** (`PM9A1 Samsung`) | 1× 466 GB SATA SSD (`WDC WDS500G2B0A-00SM50`) |
| Drive health | PASSED, **0 reallocated / 0 grown bad blocks**, `Wear_Leveling_Count` raw 106, but **77,047 power-on hours** (~8.8 y) over only 219 power cycles | **96% life**, 2142 h, PASSED | **100% life**, 3213 h, PASSED, 0 grown bad blocks / 0 erase / 0 program fails, 0 CRC |
| OS | Debian 13 (trixie), kernel **7.1.8+deb13-amd64** | Debian 13 (trixie), kernel **7.1.8+deb13-amd64** | Debian 13 (trixie), kernel **7.1.8+deb13-amd64** |
| Firmware | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 |
| `/dev/kvm` | present | present | present |
| Serial | MXL1052NGN | MXL0073949 | 8CC015307N |
@@ -48,6 +48,47 @@ modern NVMe; node 3 a third distinct model, a WD Blue 3D SATA SSD in excellent h
an identical disk layout across all three because Longhorn replica placement assumes symmetry — so
this is a live input to the disk decision, not a footnote. Node 2's NVMe is the drive node 1 should
have had; node 3 does not have it either. Node 3 is also the only node with **two matched DIMMs**
### §0c — node 1's drive was replaced (2026-09-07)
**The paragraph above is history, not current state.** On 2026-09-07 node 1's kubelet stopped
posting status while the host stayed up and SSH-able. Root cause was **etcd starved of disk I/O**,
not memory: `slow fdatasync` at 3.5–6.7 s against an expected 1 s, `apply request took too long` at
8.3 s against 100 ms, then `etcdserver: too many requests`. No OOM, MCE or hung task in the kernel
log. All three nodes place etcd's WAL (`/data/k3s`) and every Longhorn replica (`/data/longhorn`) on
the SAME partition; node 1's was the 2011 C400 with 61,440 reallocated sectors and ~31% of its
endurance consumed, and its `/proc/pressure/io` `full avg300` ran **11.36** against ~1% steady state
on the other two. The failure is self-reinforcing: the node dropping out degrades Longhorn volumes,
whose rebuilds add write load to the very disk etcd needs.
The operator replaced the drive with a **Samsung SSD 850 EVO 500GB** and reinstalled Debian.
Measured under the same load class — an active 18-replica Longhorn rebuild, with node 1 as the
*write target*, i.e. the heavier side:
| node | drive | `full avg300` during rebuild |
|---|---|---|
| `k3s-prod-1` | Samsung 850 EVO (new) | **3.79** |
| `k3s-prod-2` | Samsung PM9A1 NVMe | 0.33 |
| `k3s-prod-3` | WDC WDS500G2B0A | 2.47 |
| *`k3s-prod-1` before* | *C400-MTFDDAC512MAM* | ***11.36*** |
Roughly a **3× improvement**, moving node 1 from ~4× worse than its peers into the same band as
node 3, and it stayed `Ready` throughout. **The 850 EVO is not new**: 77,047 power-on hours over 219
power cycles — it evidently idled in an always-on system — but its wear tells the opposite story to
the C400's (0 reallocated vs 61,440; wear-leveling 106 vs 949).
**§2's exit condition is NOT met by this replacement.** etcd and Longhorn still share one SATA
spindle. The contention is reduced, not eliminated; giving etcd its own device remains the fix, and
node 1 still has exactly one disk. The three nodes also remain asymmetric — now 850 EVO / PM9A1
NVMe / WD Blue.
The rebuild itself surfaced two things worth carrying forward. Longhorn refused the new disk with
`DiskFilesystemChanged` — "record diskUUID doesn't match the one on the disk" — because its node CR
still held the old drive's `diskUUID`; the remedy is disable → remove the disk entry → re-add, in
that order (the validating webhook enforces it). And the Ansible inventory still listed node 1 in
`[k3s_prod_primary]`, which decides both which play gets `--cluster-init` *and* what `k3s_api_host`
resolves to — so re-provisioning it unchanged would have pointed a rebuilt, empty node at itself and
founded a second cluster beside the live one. The founder now sits on node 2.
(2× Crucial 2Rx8 DDR4-2666) and the only one with its **full** RAM complement of 32 GB actually
usable. **That is luck, not immunity:** node 3's firmware publishes an EFI Memory Attributes Table
too (`MEMATTR=0xbeece018`), the kernel simply parsed it cleanly on this boot, so node 3 is exposed to
+9 -2
View File
@@ -92,9 +92,17 @@ proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh
# =============================================================================
[k3s_prod_primary]
k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10
# FOUNDER MOVED TO NODE 2 (2026-09-07). Node 1 founded this cluster with --cluster-init,
# then its SSD was replaced and Debian reinstalled, so it is now a JOINING node like any
# other. This group decides two things at once: which play sets k3s_server_mode: primary
# (-> --cluster-init, which FOUNDS A NEW CLUSTER) and what k3s_api_host resolves to (the
# --server join target). Leaving node 1 here would have pointed a rebuilt, empty node at
# ITSELF and founded a second cluster beside the live one. Node 2 is the senior surviving
# server; its own --cluster-init is inert because /etc/rancher/k3s/k3s.yaml already exists.
k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11
[k3s_prod_secondary]
k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10
# Uncomment a node only once it is actually on the bench and answering SSH, so no play can
# target a host that isn't there. k3s-prod-2 came online 2026-09-01 (imaged identically to
# node 1: same HP EliteDesk 800 G5 Mini, same BIOS R21 02.26.00, same i7-9700T, same
@@ -110,7 +118,6 @@ k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10
# three-way, not a node-1 outlier.
# NOTE: node 3 also has wlp1s0 up on 192.168.2.195 (wifi). `ansible_host` is deliberately
# the WIRED eno1 address — the same directly-routable 192.168.2.x LAN as nodes 1 and 2.
k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11
k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12
# Umbrella group group_vars/k3s_prod/ attaches to (mirrors dev's [k3s_servers:children]
@@ -11,3 +11,13 @@
ansible.builtin.systemd:
name: systemd-resolved
state: restarted
# Masking a unit writes a symlink to /dev/null into /etc/systemd/system; the RUNNING
# systemd keeps its old view of the unit until it re-reads unit files. The mask is read
# fresh at the next boot either way, so this only exists so the running manager does not
# disagree with on-disk state in between. `daemon_reload` re-parses unit files and
# re-runs generators — it starts and stops nothing, so it cannot disturb k3s or Longhorn.
# Fires only when a mask actually changed, so a re-run of the role is silent.
- name: Reload systemd unit files
ansible.builtin.systemd_service:
daemon_reload: true
@@ -331,6 +331,15 @@
# The devices this role must never, under any circumstances, hand to e2label. Built from
# gathered facts (`gather_facts: true` in the play) plus the live swap list, so it is the
# NODE's account of what is load-bearing, not a hardcoded list of partition names.
#
# EXPECT THIS LIST TO BE EMPTY. Since the swap tasks in tasks/main.yml mask swap.target,
# a baselined node has no active swap for `swapon` to report, so the swap half of
# node_baseline_lh_protected_devices below contributes nothing. That is not a weakened
# guard: the claim CANDIDATE is derived from whatever is mounted at
# node_baseline_longhorn_claim_mountpoint, and a swap partition is neither mounted nor an
# ext4 filesystem, so it can never be the candidate in the first place. The task stays
# because the list must be the node's own account of what is load-bearing — including on
# a node reached before baseline has run, or one an operator has deliberately swapped on.
- name: "Longhorn (claim): list the devices currently backing active swap"
ansible.builtin.command: swapon --noheadings --show=NAME
register: node_baseline_lh_swapdevs
@@ -646,9 +646,109 @@
tags: [disks]
tags: [disks]
- name: Disable swap now
ansible.builtin.command: swapoff -a
# =====================================================================================
# SWAP — off NOW, and still off AFTER A REBOOT — tag `system`
# =====================================================================================
# `swapoff -a` plus commenting the fstab line is NOT durable on these nodes, and never
# was. Measured 2026-09-07: all three prod nodes were running with 977 MiB of swap
# ACTIVE (k3s-prod-1 /dev/sda4, k3s-prod-3 /dev/sda4, k3s-prod-2 /dev/nvme0n1p4) while
# their fstab swap line was commented out by this very role. The role reported success
# and the setting silently expired at the next boot, on every node.
#
# CAUSE. systemd-gpt-auto-generator (systemd 257, Debian 13) claims any GPT partition
# whose partition TYPE GUID is 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f ("Linux swap"),
# synthesises a .swap unit for it into /run/systemd/generator.late/ and drops a symlink
# to it into swap.target.wants/. It decides that from the PARTITION TABLE and never
# consults /etc/fstab, so neutralising fstab cannot stop it. Re-running the generator by
# hand on k3s-prod-1 says so in as many words:
# Dissecting swap partition with label n/a and UUID acd3876c-fd8e-4389-b856-...
# Probed fstype 'swap' on partition /dev/disk/by-diskseq/1-part4.
# Adding swap: /dev/disk/by-diskseq/1-part4
#
# MECHANISM — mask swap.target.
# `systemctl show` on the generated unit says its ONLY inbound dependency is
# `WantedBy=swap.target`: Requires/RequiredBy/BoundBy are empty, and the matching
# .device unit has an empty `Wants=`, so a device appearing does NOT pull its swap unit
# in by itself. swap.target is therefore the single choke point that every swap
# activation path goes through — the gpt-auto units, the units systemd-fstab-generator
# builds from /etc/fstab, and the ones systemd-cryptsetup-generator builds for encrypted
# swap. Masking it is:
# * DEVICE-SHAPE INDEPENDENT. No device name appears anywhere in the change, so one
# task is correct on prod's sda4 AND nvme0n1p4 and on dev's vda VMs, and there is
# nothing to keep in sync when a disk is replaced. (The generated unit is named after
# the kernel's DISKSEQ, not the disk — it is `dev-disk-by\x2ddiskseq-1\x2dpart4.swap`
# on all three prod nodes, NVMe and SATA alike — but diskseq is assigned per boot in
# device-probe order, so it is not a name worth depending on either.)
# * BOOT-SAFE. swap.target is reached by `Wants=`/`After=` from sysinit.target and by
# `After=` from tmp.mount, and `RequiredBy=` is EMPTY (verified on all three prod
# nodes and all three dev nodes). A masked unit reached only through Wants= is
# skipped with a log line; it cannot fail the boot transaction. Ordering deps on a
# unit that never starts are no-ops.
# * INERT WHEN THE DEVICE IS ABSENT OR RENAMED. The mask names no device, so there is
# nothing to go stale: a disk swap, a new partition layout or a restore onto
# different hardware all stay swap-free without touching this role.
# The per-unit masks that follow are a SECOND, independent barrier over the units that
# exist right now, so `systemctl status <unit>` says "masked" to whoever looks, and an
# operator who unmasks swap.target does not silently get swap back. Their failure mode is
# benign in the other direction too: a mask whose unit no longer exists is an unreferenced
# symlink to /dev/null, and the only thing it could ever shadow is a .swap unit.
#
# ALTERNATIVES REJECTED.
# * `systemd.gpt_auto=0` on the kernel cmdline — disables GPT auto-discovery WHOLESALE,
# not just swap, and needs a GRUB edit + update-grub + a reboot before it is even
# true. On prod that happens to be swap-only today (root= is already on the cmdline
# so root auto-detection is off, and the /boot/efi fstab entry already suppresses the
# ESP/XBOOTLDR mounts — the generator says so in its debug output), but on DEV the
# root partition carries the discoverable GUID 4f68bce3-e8cd-4db1-96e7-fbcaf984b709
# ("Linux root (x86-64)"), so the same flag would switch off root auto-discovery
# there. Editing a bootloader to fix a swap unit is the wrong blast radius, and the
# change would not be verifiable without the reboot it depends on.
# * Clearing or retyping the swap partition's GPT type GUID — a partition-table write
# to the live boot disk of a node currently serving etcd and Longhorn replicas, to
# fix something a symlink fixes. It would also decide, irreversibly and without
# asking, that those 977 MiB are never swap again and are simply unused; that is the
# operator's call to make, not this role's, and it needs a maintenance window.
# * `systemd.swap=0` — read by systemd-fstab-generator ONLY. The swap here does not
# come from fstab, so it would change nothing at all.
#
# DEV. Dev's nodes have no swap partition, no swap unit and no fstab swap line at all
# (checked on k3s-dev-1/2/3), so the mask is inert there: it flips to masked once and
# every later run is changed=0. It is correct rather than merely harmless — if a future
# dev image ever ships a swap partition, the same mask already covers it.
- name: Mask swap.target so no generator can pull a swap unit in at boot
ansible.builtin.systemd_service:
name: swap.target
masked: true
notify: Reload systemd unit files
tags: [system]
- name: Find the .swap units systemd's generators produced for this boot
ansible.builtin.find:
# gpt-auto writes to generator.late; systemd-fstab-generator and
# systemd-cryptsetup-generator (the other two that can emit a .swap unit) write to
# generator. /run/systemd/generator.early is deliberately NOT listed: nothing that
# makes swap units runs at early priority, and `find` warns on a path that does not
# exist. Anything a future early generator invented is still caught by the
# swap.target mask above, which is the mechanism this task only reinforces.
paths:
- /run/systemd/generator
- /run/systemd/generator.late
patterns: "*.swap"
file_type: file
register: node_baseline_generated_swap
changed_when: false
check_mode: false
tags: [system]
- name: Mask each generated .swap unit (second barrier, independent of swap.target)
ansible.builtin.systemd_service:
name: "{{ item }}"
masked: true
loop: "{{ node_baseline_generated_swap.files | map(attribute='path') | map('basename') | unique | sort }}"
loop_control:
label: "{{ item }}"
notify: Reload systemd unit files
tags: [system]
- name: Neutralize swap entries in fstab (kubelet requires swap off)
@@ -658,6 +758,23 @@
replace: '# \1 # disabled for k3s/kubelet'
tags: [system]
# The masks above only decide what the NEXT boot does. Anything swapped on already stays
# on until it is told otherwise, so read the live state and act only when there is
# something to act on — that is what makes a second run honestly changed=0 instead of
# `changed_when: false` hiding a command that ran anyway.
- name: Read the swap devices that are active right now
ansible.builtin.command: swapon --noheadings --show=NAME
register: node_baseline_swap_active
changed_when: false
check_mode: false
tags: [system]
- name: Disable swap now
ansible.builtin.command: swapoff -a
when: node_baseline_swap_active.stdout | trim | length > 0
changed_when: true
tags: [system]
# =====================================================================================
# DNS RESILIENCE — tag `dns` (R-DNS-01 / R-DNS-03)
# =====================================================================================
+32 -1
View File
@@ -47,7 +47,7 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
| `storage.io_errors` | kernel SCSI/ATA `ioerr_cnt` / `iotmo_cnt` per disk (no root needed) |
| `storage.capacity`, `storage.inodes` | worst mounted filesystem, space and inodes |
| `storage.layout` | physical disks, and whether any device is genuinely free for `node_baseline_longhorn_device` / `node_baseline_etcd_device` |
| `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite, EDAC correctable/uncorrectable counters |
| `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite **and whether it survives a reboot** (see [Swap off is not the same as swap disabled](#swap-off-is-not-the-same-as-swap-disabled)), EDAC correctable/uncorrectable counters |
| `memory.physical_total` | **is this machine using the RAM it has?** `/proc/zoneinfo` present-vs-managed pages (no root, metal or VM) and `MemTotal` vs the DIMM capacity SMBIOS reports (`dmidecode -t 17`, root, bare metal). Cause-independent by design — see [The EFI memattr check](#the-efi-memattr-check) |
| `cpu.throttle`, `cpu.inventory`, `cpu.virtualisation` | thermal throttle events since boot, topology + frequency scaling, VT-x/AMD-V + `/dev/kvm` (KubeVirt, R-PLAT-03) |
| `thermal.max` | hottest sensor across thermal zones and hwmon chips |
@@ -59,6 +59,37 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
Thresholds live in `defaults/main.yml` (`node_health_thresholds`); no verdict is hardcoded
in task logic.
### Swap off is not the same as swap disabled
`SwapTotal == 0` only says nothing has been swapped on *yet this boot*. It is not evidence
that swap is disabled, and on these nodes it was not: measured 2026-09-07, all three prod
nodes came back from a reboot with 977 MiB of swap active (`k3s-prod-1` `/dev/sda4`,
`k3s-prod-3` `/dev/sda4`, `k3s-prod-2` `/dev/nvme0n1p4`) even though `roles/node_baseline`
had run `swapoff -a` and commented their `/etc/fstab` swap line.
The cause is `systemd-gpt-auto-generator`. It claims any GPT partition whose partition
**type GUID** is `0657fd6d-a4ab-43c4-84e5-0933c84b4f4f` ("Linux swap") straight from the
partition table, synthesises a `.swap` unit into `/run/systemd/generator.late/` and links
it into `swap.target.wants/`. `/etc/fstab` is never consulted, so neutralising fstab
cannot stop it. `roles/node_baseline` masks `swap.target` — the one unit every generated
`.swap` unit is `WantedBy` — and masks the generated units as a second barrier.
So `memory.swap` reports on both halves, from `memory.swap_persistence` in the collector
(all world-readable, no root):
| fact | meaning |
|---|---|
| `swap_target_masked` | is `/etc/systemd/system/swap.target` a symlink to `/dev/null`? |
| `generated_units` | `.swap` units systemd's generators built for this boot |
| `masked_units` | `.swap` units masked in `/etc/systemd/system` |
| `unmasked_generated_units` | `generated_units` minus `masked_units` — what a reboot would bring back |
| `fstab_active_swap_lines` | uncommented `swap` lines left in `/etc/fstab` |
A node with no swap active but an unmasked generated unit and an unmasked `swap.target`
**WARNs**: it is clean now and will be swapping again after the next reboot. The check
used to blame that state on "this node has not been baselined yet", which sent the reader
back to re-run the role that had already run.
## The EFI memattr check
@@ -301,9 +301,94 @@ def collect_memory():
# Firmware's own inventory of what is physically installed. Root-only; attempted
# here anyway so the unprivileged pass records the exact reason it could not read.
mem["devices"] = collect_memory_devices()
# "Is swap off right now?" and "will it still be off after a reboot?" are DIFFERENT
# questions, and only the second one is kubelet's actual prerequisite. See below.
mem["swap_persistence"] = collect_swap_persistence()
return mem
# --- Will swap still be off after the next reboot? -----------------------------------
# SwapTotal=0 does not mean swap is DISABLED — it means nothing has been swapped on yet
# this boot. Measured 2026-09-07 on all three prod nodes: swap was active again after a
# reboot, 977 MiB each, with /etc/fstab's swap line commented out by node_baseline. The
# cause is systemd-gpt-auto-generator, which claims any GPT partition typed
# 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f ("Linux swap") straight from the PARTITION TABLE,
# synthesises a .swap unit into /run/systemd/generator.late/ and links it into
# swap.target.wants/. It never reads fstab, so neutralising fstab cannot stop it.
#
# roles/node_baseline masks swap.target — the single unit every generated .swap unit is
# `WantedBy` — and masks the generated units themselves as a second barrier. This
# collects the facts that let the report tell "off, and staying off" apart from "off
# until the next reboot", which a /proc/meminfo reading alone can never do.
#
# All of it is world-readable (the generator dirs are 0755 root:root), so it works in the
# UNPRIVILEGED pass: no root, no systemctl, no parsing of systemctl output.
def _is_systemd_mask(path):
"""True when `path` is a systemd mask, i.e. a symlink to /dev/null."""
try:
return os.path.islink(path) and os.path.realpath(path) == "/dev/null"
except OSError:
return False
def collect_swap_persistence():
out = {
"readable": False,
"swap_target_masked": None,
# .swap units systemd's generators built for THIS boot (gpt-auto, fstab,
# cryptsetup). These are the ones that would be activated again next boot.
"generated_units": [],
# .swap units masked in /etc/systemd/system — what node_baseline wrote.
"masked_units": [],
# generated MINUS masked: the units that would still come back. Empty is good.
"unmasked_generated_units": [],
"fstab_active_swap_lines": None,
}
generated = set()
for directory in (
"/run/systemd/generator.early",
"/run/systemd/generator",
"/run/systemd/generator.late",
):
try:
names = os.listdir(directory)
except FileNotFoundError:
# A generator produced nothing. That is a real answer, not a read failure:
# dev's VMs have no swap partition, so gpt-auto emits no unit at all.
continue
except OSError as exc:
UNREADABLE.append({"path": directory, "reason": "read error: %s" % exc})
return out
for name in names:
if name.endswith(".swap") and os.path.isfile(os.path.join(directory, name)):
generated.add(name)
masked = set()
try:
for name in os.listdir("/etc/systemd/system"):
if name.endswith(".swap") and _is_systemd_mask("/etc/systemd/system/" + name):
masked.add(name)
except OSError as exc:
UNREADABLE.append({"path": "/etc/systemd/system", "reason": "read error: %s" % exc})
return out
out["swap_target_masked"] = _is_systemd_mask("/etc/systemd/system/swap.target")
out["generated_units"] = sorted(generated)
out["masked_units"] = sorted(masked)
out["unmasked_generated_units"] = sorted(generated - masked)
out["readable"] = True
fstab = read_text("/etc/fstab")
if fstab is not None:
out["fstab_active_swap_lines"] = sum(
1
for line in fstab.splitlines()
if not line.lstrip().startswith("#") and re.search(r"\sswap\s", line)
)
return out
# --- Memory the kernel has but never hands out --------------------------------------
# /proc/zoneinfo is WORLD-READABLE on every Linux, VM or metal — no root, no dmidecode,
# no DMI. Per zone it reports:
@@ -499,25 +499,76 @@
if phys_status in ['WARN', 'FAIL'] else '')
~ (phys_cause if phys_status in ['WARN', 'FAIL'] else '') }}
- name: Assess — swap (kubelet requires it off)
# WHY THIS CHECK IS NOT JUST `SwapTotal == 0`. It used to be, and it read the answer
# wrong in both directions. It called active swap "this node has not been baselined yet",
# which was a MIS-DIAGNOSIS — measured 2026-09-07, all three prod nodes had been
# baselined, had their fstab swap line commented out by node_baseline, and were STILL
# swapping 977 MiB each after a reboot. Whoever read that WARN would have re-run the very
# role that had already run, and watched swap come back at the next boot again.
#
# The real mechanism is systemd-gpt-auto-generator: it claims any GPT partition typed
# 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f ("Linux swap") from the PARTITION TABLE, builds a
# .swap unit into /run/systemd/generator.late/ and links it into swap.target.wants/. It
# never reads /etc/fstab, so `swapoff -a` + a commented fstab line only holds until the
# next boot. roles/node_baseline now masks swap.target (the single unit those generated
# units are `WantedBy`) plus the generated units themselves.
#
# The other direction matters just as much: SwapTotal == 0 is NOT proof that swap is
# disabled — a node freshly `swapoff`-ed reads zero and swaps again on reboot. So this
# check now asks both questions, and WARNs on a node that is clean right now but has an
# unmasked generated swap unit waiting for the next boot.
- name: Assess — swap (kubelet requires it off, and off ACROSS REBOOTS)
ansible.builtin.set_fact:
node_health_checks: "{{ node_health_checks + [{
'id': 'memory.swap',
'title': 'Swap disabled (kubelet prerequisite)',
'title': 'Swap disabled, and disabled across reboots (kubelet prerequisite)',
'status': swap_status,
'detail': swap_detail }] }}"
vars:
ansible_become: false
m: "{{ node_health_facts.memory }}"
p: "{{ node_health_facts.memory.swap_persistence | default({}, true) }}"
# Durability was actually measurable on this node (the collector could read
# /etc/systemd/system and the generator dirs). When false, say so rather than
# implying a clean bill from a failed read.
swap_durable_known: "{{ (p.readable | default(false)) | bool }}"
# Units a reboot would bring back: produced by a generator, not masked, and with
# swap.target itself unmasked so something can still pull them in.
swap_will_return: >-
{{ (swap_durable_known | bool)
and not (p.swap_target_masked | default(false) | bool)
and (p.unmasked_generated_units | default([], true) | length) > 0 }}
swap_units_returning: "{{ p.unmasked_generated_units | default([], true) | join(', ') }}"
swap_active_gib: "{{ (m.swap_total_kb | int / 1048576) | round(2) | string }}"
swap_status: >-
{{ 'SKIP' if m.swap_total_kb is not number
else ('PASS' if m.swap_total_kb | int == 0 else 'WARN') }}
else ('WARN' if (m.swap_total_kb | int) > 0
else ('WARN' if (swap_will_return | bool) else 'PASS')) }}
swap_cause: >-
systemd-gpt-auto-generator re-creates a .swap unit for the GPT partition typed
0657fd6d-a4ab-43c4-84e5-0933c84b4f4f ("Linux swap") at EVERY boot, straight from
the partition table and without reading /etc/fstab. This is NOT "the node has not
been baselined" — `swapoff -a` and a commented fstab line simply do not survive a
reboot. The fix is roles/node_baseline masking swap.target and the generated units:
re-run `node-baseline.yml --tags system` against this node.
swap_detail: >-
{{ 'swap state unreadable' if m.swap_total_kb is not number
else ('no swap active' if m.swap_total_kb | int == 0
else ((m.swap_total_kb | int / 1048576) | round(2) | string
~ ' GiB swap active — roles/node_baseline runs swapoff and neutralises
the fstab entry; this node has not been baselined yet')) }}
else ((swap_active_gib ~ ' GiB swap active'
~ (' via ' ~ swap_units_returning if swap_units_returning else '')
~ ' — ' ~ swap_cause)
if (m.swap_total_kb | int) > 0
else ('no swap active, but it would come back at the next boot via '
~ swap_units_returning ~ ' — ' ~ swap_cause
if (swap_will_return | bool)
else ('no swap active'
~ (' and none can be activated: swap.target is masked'
if (p.swap_target_masked | default(false) | bool)
else (' and no swap unit exists on this node'
if (p.generated_units | default([], true) | length) == 0
else ' and every generated swap unit is masked'))
if (swap_durable_known | bool)
else 'no swap active (durability NOT verified — the collector
could not read /etc/systemd/system or /run/systemd)'))) }}
- name: Assess — ECC / EDAC correctable errors
ansible.builtin.set_fact: