Compare commits
13
Commits
863af01062
...
a624c638ef
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a624c638ef | ||
|
|
90b39de527 | ||
|
|
0247d796a3 | ||
|
|
c9e9dcf960 | ||
|
|
89cf59ffb7 | ||
|
|
e319777849 | ||
|
|
730d12ddc4 | ||
|
|
df1d2ab44d | ||
|
|
69a25cff82 | ||
|
|
c1e36d9353 | ||
|
|
9d00a3fcd3 | ||
|
|
0e92d258dd | ||
|
|
08fbe85638 |
No files matched your search
@@ -1954,3 +1954,138 @@ not one.** Confirmed by `--list-hosts` (dev default → 3 dev hosts, where the n
|
||||
Nothing was run against either cluster for this change.
|
||||
3. **Decide on `kubeEtcd` metrics**, so §2's exit condition can be measured with etcd's own fsync
|
||||
histograms rather than inferred from node PSI.
|
||||
|
||||
## 2026-09-07 — R-ENV-03's L0 check was RED on `main`, and the requirement's literal wording is why (branch `fix/r-env-03-overlay-selected-credentials`)
|
||||
|
||||
`main`'s `conformance/tests/l0-static/prod-baremetal-not-dev-surface` failed 4 of 96 assertions.
|
||||
All four are the per-environment backup credentials merged in `e7c4a95`:
|
||||
|
||||
```
|
||||
apps/wave-0/nextcloud-db-backup-credentials-prod.sops.yaml
|
||||
apps/wave-0/semaphore-db-backup-credentials-prod.sops.yaml
|
||||
platform/keycloak/keycloak-db-backup-credentials-prod.sops.yaml
|
||||
platform/velero/velero-cloud-prod.sops.yaml
|
||||
... is a prod-specific file inside the shared configuration
|
||||
```
|
||||
|
||||
**The guard was a filename regex.** Any tracked file under `platform/ platform-config apps/
|
||||
catalog/ clusters/base/` whose name contained a delimited `prod` token failed, with no allowlist
|
||||
and no exception mechanism. Nothing about the four files' CONTENT was inspected.
|
||||
|
||||
**Renaming them to dodge the regex was rejected** (operator, explicitly): the guard exists for a
|
||||
reason and a rename would leave the reason unserved while making the check lie. So the check was
|
||||
refined to test the harm instead of the spelling, and the requirement wording is escalated below.
|
||||
|
||||
### Why these four files are compliant with R-ENV-02, and why the regex could not see it
|
||||
|
||||
Backup credentials became per-environment because dev backs up to MinIO and prod to Cloudflare
|
||||
R2, and ONE shared Secret meant either environment's key rotation broke the other (worse for
|
||||
CNPG: `archive_mode=on` turns a 403 on WAL archive into a monotonically filling pgdata PVC).
|
||||
|
||||
The mechanism is a single cluster-vars knob, `BACKUP_CREDENTIALS_SUFFIX` (base `""`, dev `""`,
|
||||
prod `"-prod"`). **Both Secret objects are applied to BOTH clusters** as base resources; the
|
||||
overlay changes only which one the shared consumer REFERENCES
|
||||
(`<name>${BACKUP_CREDENTIALS_SUFFIX}`). Every environment renders the identical manifests and the
|
||||
identical object set. That is exactly the "same manifests, overlay-scoped values" contract of
|
||||
R-ENV-02 — nothing is forked, and there is no prod-only content for anyone to develop against.
|
||||
The `-prod` in a filename is not a prod-only manifest; it is one half of an environment-selected
|
||||
pair, present everywhere.
|
||||
|
||||
### What the check now asserts (committed on the branch; not a path allowlist)
|
||||
|
||||
A file whose name carries the environment token is excused ONLY when the tree proves all of:
|
||||
|
||||
- **(A)** it declares nothing but Secrets whose names end in the environment suffix — an overlay
|
||||
can select which SECRET a shared manifest references and nothing else, so any other
|
||||
prod-suffixed object in a shared path is prod-only content and still fails;
|
||||
- **(1)** its own directory's `kustomization.yaml` lists it in `resources` (so EVERY environment
|
||||
applies it), and EVERY environment's value of the selector resolves to a Secret that some
|
||||
resource of that same kustomization declares, in the same namespace — no environment is left
|
||||
selecting an object that does not exist;
|
||||
- **(2)** nothing names the suffixed Secret — or the suffixed FILE — literally: every reference is
|
||||
`<base-name>${VAR}`, and only the applying kustomization may name the file;
|
||||
- **(3)** that `VAR` is defined for `clusters/base` AND for every `clusters/<env>` overlay
|
||||
(StrictPostBuildSubstitutions fails on an UNDEFINED variable, so an overlay that forgot it
|
||||
would stop that cluster reconciling), and prod's value is the one that selects THIS file.
|
||||
|
||||
References are read from PARSED YAML VALUES, never raw text — the mechanism is documented by name
|
||||
in comments beside the resources, and a name in a comment references nothing.
|
||||
|
||||
Mutation-tested: eleven mutations, each restored afterwards, each of which must FAIL and does — a
|
||||
prod-only ConfigMap in a shared path, a prod-only Secret with no counterpart, a consumer
|
||||
hardcoding prod's Secret name, a kustomization naming prod's file from another directory, the
|
||||
knob missing from dev, the knob missing from base, prod's value drifted to a suffix nothing
|
||||
declares, the prod half dropped from `resources`, the base half deleted (with and without its
|
||||
`resources` entry), the halves split across namespaces, and one of two consumer references moved
|
||||
to a second knob. The legitimate four pass, with a `note:` per file saying why they were excused.
|
||||
|
||||
### PROPOSED R-ENV-03 AMENDMENT — for the System Architect (spec repo is READ-ONLY here)
|
||||
|
||||
**What the requirement says today** (`docs/00-vision-and-scope.md`, detail in `docs/02`): prod is
|
||||
a 3x mini-PC bare-metal k3s cluster with local storage and **"MUST NOT be an active development
|
||||
surface until the roadmap declares it ready"**; `docs/02` adds **"Never develop against prod.
|
||||
Prod hardware is a target, not a sandbox (R-ENV-03)."**
|
||||
|
||||
**The problem with the literal reading.** Taken literally against the repository, "no prod-only
|
||||
content in the shared configuration" reads as "no artifact in a shared path may be specific to
|
||||
prod". That forbids the environment-selected PAIR, which is the very pattern R-ENV-02 prescribes
|
||||
for a value that must differ per environment while the manifests stay identical. The two
|
||||
requirements contradict each other on this case: R-ENV-02 says put the difference in an
|
||||
overlay-scoped value, and a Secret's difference cannot live in cluster-vars (it is ciphertext, per
|
||||
SOPS file, R-SEC-01), so it must live in a per-environment FILE that every environment applies.
|
||||
The requirement should forbid what is actually harmful — configuration only prod runs — not the
|
||||
appearance of an environment name in a path.
|
||||
|
||||
**Proposed wording** (add as a clarifying clause to R-ENV-03 in `docs/02`, and mirror the phrase
|
||||
in `docs/00`):
|
||||
|
||||
> Prod MUST NOT be an active development surface. Concretely: no artifact in the shared
|
||||
> configuration may be reachable by prod alone. An artifact whose NAME carries an environment
|
||||
> token is not by itself a violation — provided every environment APPLIES it, no shared manifest
|
||||
> references it by its environment-qualified name, and the selection between environment
|
||||
> counterparts is made solely by an overlay-scoped cluster-vars value that is DEFINED for every
|
||||
> environment. An artifact that only prod applies, or that a shared manifest names literally, IS
|
||||
> a violation, whatever it is called.
|
||||
|
||||
That wording keeps the guard's teeth (an artifact only prod applies still fails), removes the
|
||||
false positive, and states the property the check now enforces so the test and the spec agree.
|
||||
|
||||
**Related spec ask, same clause.** R-ENV-02 should say out loud that a per-environment SECRET is
|
||||
expressed as counterpart SOPS files applied everywhere plus a cluster-vars selector, because that
|
||||
is the one value class that cannot be expressed as a cluster-vars value itself. Without it, the
|
||||
next person hits the same contradiction.
|
||||
|
||||
### How this reached `main` green — the gate hole, and whether the fix in flight closes it
|
||||
|
||||
`ci/l0.sh` Stage 3b scopes the L0 conformance run to the TOUCHED requirement subset
|
||||
(`load_touched_requirements` -> `run.sh --requirements ...`). The delivery `e7c4a95` touched
|
||||
`apps/wave-0/`, `catalog/render.py`, `platform/keycloak/`, `platform/velero/` and declared no
|
||||
R-ID on a non-comment line, so the path map attributed R-GITOPS-02 / R-IDP-01..06 / R-STATE-03 —
|
||||
**and never R-ENV-03**. `prod-baremetal-not-dev-surface` was therefore FILTERED OUT of the run
|
||||
that gated the merge. The gate printed PASS having never executed the check that was already red.
|
||||
|
||||
**Landing `fix/touched-requirements-fail-closed` would NOT have caught this.** Checked, not
|
||||
assumed: that branch's `ci/requirement-paths.map` classifies every path (so nothing is silently
|
||||
unmapped any more), but it attributes R-ENV-03 only to `infra/ansible/*` and `infra/tofu/*`.
|
||||
Replaying `e7c4a95`'s twelve paths through that map yields
|
||||
`R-GITOPS-01 R-GITOPS-02 R-IDP-01..06 R-PLAT-02 R-STATE-03` — still no R-ENV-03. Its `ci/l0.sh`
|
||||
change also leaves Stage 3b scoped (only the substitution stage stays unconditional). The
|
||||
fail-closed branch fixes the "unmapped path contributes nothing" hole; it does not fix this one.
|
||||
|
||||
**Two things that would have caught it, for the operator/architect to choose between:**
|
||||
|
||||
1. **Whole-repo L0 invariants should be unconditional.** `flux-postbuild-substitution`,
|
||||
`keycloak-client-secret-reconciler` and `spec-immutability` already run as their own
|
||||
unconditional stages for exactly this reason. `prod-baremetal-not-dev-surface`,
|
||||
`no-plaintext-secrets` and `least-privilege-rbac` are the same shape — they assert over the
|
||||
WHOLE tracked tree, so scoping them to a changed path is meaningless. Cheapest correct fix.
|
||||
2. **A changed test should attribute the requirements it declares.** `conformance/*` maps to
|
||||
R-TEST-01..04 only, so editing `prod-baremetal-not-dev-surface/check.sh` does not select
|
||||
R-ENV-03 — this very branch's gate run reports `touched requirements: R-ENV-02 R-TEST-01
|
||||
R-TEST-02 R-TEST-03 R-TEST-04` and ran 2 of 12 L0 checks, NOT including the one it changed.
|
||||
A changed `conformance/tests/<test>/` should contribute that test's own
|
||||
`conformance.home.k3s/requirements` label.
|
||||
|
||||
Both belong to `ci/`, so they are left to that branch rather than smuggled in here. The full L0
|
||||
layer was run unscoped on this branch to compensate: **12 offline checks, 11 PASS, 1 SKIP**
|
||||
(`placeholder-guard-coupling`, pre-existing), 0 FAIL.
|
||||
@@ -48,7 +48,10 @@
|
||||
# sync.yaml, every patch it applies targets an ENVIRONMENT-SCOPED object
|
||||
# (cluster-vars, a Flux Kustomization's sops decryption, or the Git source),
|
||||
# prod's Git ref is the SAME ref dev tracks (no prod-only branch to develop on),
|
||||
# and no prod-only manifest exists anywhere in the rendered configuration;
|
||||
# and no prod-only manifest exists anywhere in the rendered configuration — a file
|
||||
# whose NAME carries the environment token is excused ONLY when the tree proves it
|
||||
# is an OVERLAY-SELECTED half that every environment applies (see the four
|
||||
# conditions at "THE ONE STRUCTURAL EXCEPTION" below); anything else still fails;
|
||||
# (6) prod artifacts exist only WITH a declaration of readiness: the spec roadmap
|
||||
# carries the phase that opens prod build (checked when the spec repo is mounted),
|
||||
# and the repository records that the operator opened it.
|
||||
@@ -437,12 +440,172 @@ tracked = [f for f in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
|
||||
capture_output=True, text=True, check=True
|
||||
).stdout.split("\0") if f]
|
||||
RENDERED_DIRS = ("platform/", "platform-config", "apps/", "catalog/", "clusters/base/")
|
||||
for f in tracked:
|
||||
if f.startswith(RENDERED_DIRS) and re.search(rf"(^|[/_.-]){re.escape(prod_token)}([/_.-]|$)",
|
||||
f, re.I):
|
||||
check(False,
|
||||
f"{f} is a prod-specific file inside the shared configuration — every environment "
|
||||
"renders these paths, so prod-only content here is development against prod")
|
||||
|
||||
# THE ONE STRUCTURAL EXCEPTION: AN OVERLAY-SELECTED ENVIRONMENT HALF.
|
||||
# ------------------------------------------------------------------
|
||||
# A filename carrying an environment token is a HEURISTIC for the harm R-ENV-03 names
|
||||
# (content only prod runs, in a path every environment renders). It is not the harm
|
||||
# itself. There is exactly one shape where a `-prod` name is the OPPOSITE of a prod-only
|
||||
# fork: a credential that is TWO Secret objects, BOTH applied to BOTH clusters as base
|
||||
# resources, where a cluster-vars knob decides only which one the shared consumer
|
||||
# REFERENCES (clusters/base/cluster-vars.yaml, BACKUP_CREDENTIALS_SUFFIX: dev backs up to
|
||||
# MinIO, prod to Cloudflare R2, and one shared Secret meant either environment's rotation
|
||||
# broke the other). Nothing is forked: every environment renders the identical manifests
|
||||
# and the identical object set, which is precisely the R-ENV-02 contract.
|
||||
#
|
||||
# So the exception is not a path allowlist — it is the PROPERTY that makes that shape
|
||||
# safe, asserted from the tree, and a file is excused ONLY when all of it holds:
|
||||
#
|
||||
# (A) it declares nothing but Secrets whose names end in the environment suffix — the
|
||||
# only kind of object this repo has a selection mechanism for;
|
||||
# (1) it is listed in its own directory's kustomization.yaml, so EVERY environment
|
||||
# applies it, and every environment's value of the selector resolves to an object
|
||||
# that some resource of that same kustomization declares (same namespace) — no
|
||||
# environment is left selecting a Secret that does not exist;
|
||||
# (2) nothing references the suffixed name literally: every reference is a substituted
|
||||
# expression `<base-name>${VAR}`, and the suffixed FILE is named only by the
|
||||
# kustomization that applies it — so no shared manifest hardcodes prod's half;
|
||||
# (3) that VAR is defined for the base AND for every overlay (StrictPostBuildSubstitutions
|
||||
# fails on an UNDEFINED variable, so an overlay that forgot it would break that
|
||||
# cluster's reconciliation), and prod's value is the one that selects THIS file.
|
||||
#
|
||||
# References are read from PARSED YAML VALUES, never from raw text: the mechanism is
|
||||
# documented by name in comments beside the resources (cluster-vars says so explicitly),
|
||||
# and a name in a comment references nothing. Anything that fails any condition still
|
||||
# fails, naming the condition it broke.
|
||||
ENV_SUFFIX = f"-{prod_token}"
|
||||
# Every environment that renders the shared configuration.
|
||||
ENVS = ["base"] + sorted(p.parent.name
|
||||
for p in (ROOT / "clusters").glob("*/kustomization.yaml")
|
||||
if p.parent.name != "base")
|
||||
|
||||
|
||||
def _docs(rel: str):
|
||||
p = ROOT / rel
|
||||
if not p.is_file():
|
||||
return None
|
||||
try:
|
||||
return [d for d in yaml.safe_load_all(p.read_text(errors="replace"))
|
||||
if isinstance(d, dict)]
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _strings(node):
|
||||
if isinstance(node, str):
|
||||
yield node
|
||||
elif isinstance(node, dict):
|
||||
for k, v in node.items():
|
||||
yield from _strings(k)
|
||||
yield from _strings(v)
|
||||
elif isinstance(node, (list, tuple)):
|
||||
for v in node:
|
||||
yield from _strings(v)
|
||||
|
||||
|
||||
RENDERED_YAML = [f for f in tracked
|
||||
if f.startswith(RENDERED_DIRS) and f.endswith((".yaml", ".yml"))]
|
||||
VALUE_STRINGS = {f: list(_strings(_docs(f) or [])) for f in RENDERED_YAML}
|
||||
|
||||
|
||||
def overlay_selected(f: str) -> list[str]:
|
||||
"""[] if f is an overlay-SELECTED environment half; else the conditions it breaks."""
|
||||
docs = _docs(f)
|
||||
objs = [((d.get("kind")), (d.get("metadata") or {}).get("name"),
|
||||
(d.get("metadata") or {}).get("namespace")) for d in (docs or [])]
|
||||
if not objs or any(k != "Secret" or not isinstance(n, str) or not n.endswith(ENV_SUFFIX)
|
||||
for k, n, _ in objs):
|
||||
return [f"(A) it does not consist solely of Secret objects whose names end in "
|
||||
f"{ENV_SUFFIX!r} — an overlay can select which SECRET a shared manifest "
|
||||
f"references, and nothing else, so any other prod-suffixed object in a "
|
||||
f"shared path is prod-only content"]
|
||||
|
||||
reasons: list[str] = []
|
||||
d = str(Path(f).parent)
|
||||
kfile = f"{d}/kustomization.yaml"
|
||||
kdocs = _docs(kfile) or []
|
||||
res = [str(r) for r in ((kdocs[0].get("resources") if kdocs else None) or [])]
|
||||
if Path(f).name not in res:
|
||||
reasons.append(f"(1) {kfile} does not list it in `resources`, so it is not applied "
|
||||
f"in every environment — one environment would carry an object the "
|
||||
f"other does not")
|
||||
|
||||
# Every Secret the applying kustomization declares: name -> namespace.
|
||||
declared: dict[str, object] = {}
|
||||
for r in res:
|
||||
for dd in (_docs(f"{d}/{r}") or []):
|
||||
if dd.get("kind") == "Secret":
|
||||
md = dd.get("metadata") or {}
|
||||
declared[str(md.get("name"))] = md.get("namespace")
|
||||
|
||||
for _, name, ns in objs:
|
||||
base_name = name[:-len(ENV_SUFFIX)]
|
||||
pat = re.escape(base_name) + r"\$\{([A-Za-z_][A-Za-z0-9_]*)\}"
|
||||
sel = sorted({m for ss in VALUE_STRINGS.values() for s in ss
|
||||
for m in re.findall(pat, s)})
|
||||
# (2) literal references to the suffixed OBJECT — anywhere but this file, except
|
||||
# the applying kustomization naming this FILE (which is how it gets applied).
|
||||
lit = sorted({f2 for f2, ss in VALUE_STRINGS.items() for s in ss
|
||||
if name in s and f2 != f
|
||||
and not (f2 == kfile and s == Path(f).name)})
|
||||
if lit:
|
||||
reasons.append(f"(2) {', '.join(lit)} names the Secret {name!r} literally — a "
|
||||
f"shared manifest that hardcodes prod's half makes every "
|
||||
f"environment use prod's credential; reference "
|
||||
f"{base_name}${{...}} instead")
|
||||
# (2) literal references to the suffixed FILE from anywhere but its kustomization.
|
||||
stray = sorted({f2 for f2, ss in VALUE_STRINGS.items() for s in ss
|
||||
if Path(f).name in s and f2 not in (f, kfile)})
|
||||
if stray:
|
||||
reasons.append(f"(2) {', '.join(stray)} names the file {Path(f).name!r} "
|
||||
f"literally — only {kfile} may apply it")
|
||||
if not sel:
|
||||
reasons.append(f"(2) no manifest references {base_name}${{...}}: nothing selects "
|
||||
f"between the halves, so this file is prod-only content that "
|
||||
f"only prod could ever consume")
|
||||
continue
|
||||
if len(sel) > 1:
|
||||
reasons.append(f"(2) {base_name} is selected by more than one variable {sel} — "
|
||||
f"they can drift apart and leave one consumer on the other "
|
||||
f"environment's credential")
|
||||
var = sel[0]
|
||||
vals = env_values(var)
|
||||
missing = [e for e in ENVS if e not in vals]
|
||||
if missing:
|
||||
reasons.append(f"(3) ${{{var}}} is not defined for {missing} (defined for "
|
||||
f"{sorted(vals)}) — StrictPostBuildSubstitutions fails on an "
|
||||
f"undefined variable, so that environment would stop "
|
||||
f"reconciling, or silently inherit the other's credential")
|
||||
for e, v in sorted(vals.items()):
|
||||
want = base_name + v
|
||||
if want not in declared:
|
||||
reasons.append(f"(1) environment {e!r} selects Secret {want!r} via "
|
||||
f"${{{var}}}={v!r}, which no resource of {kfile} declares — "
|
||||
f"that environment renders a dangling reference")
|
||||
elif declared[want] != ns:
|
||||
reasons.append(f"(1) environment {e!r} selects Secret {want!r}, declared in "
|
||||
f"namespace {declared[want]!r}, but this half is in {ns!r} — "
|
||||
f"the halves are not interchangeable")
|
||||
if prod_overlay and prod_overlay in vals and base_name + vals[prod_overlay] != name:
|
||||
reasons.append(f"(3) the prod overlay's ${{{var}}}={vals[prod_overlay]!r} selects "
|
||||
f"{base_name + vals[prod_overlay]!r}, not this file's {name!r} — "
|
||||
f"prod's half is applied but no environment selects it")
|
||||
return reasons
|
||||
|
||||
|
||||
flagged = [f for f in tracked
|
||||
if f.startswith(RENDERED_DIRS)
|
||||
and re.search(rf"(^|[/_.-]){re.escape(prod_token)}([/_.-]|$)", f, re.I)]
|
||||
for f in flagged:
|
||||
why = overlay_selected(f)
|
||||
check(not why,
|
||||
f"{f} is a prod-specific file inside the shared configuration — every environment "
|
||||
f"renders these paths, so prod-only content here is development against prod. It is "
|
||||
f"not an overlay-selected environment half either: " + " AND ".join(why))
|
||||
if not why:
|
||||
note(f"{f} carries the {prod_token!r} token but is an OVERLAY-SELECTED half: both "
|
||||
f"halves are applied in every environment and only the cluster-vars selector "
|
||||
f"differs (R-ENV-02), so it is not prod-only content")
|
||||
check(len(tracked) > 50, f"git ls-files returned only {len(tracked)} paths — not a real checkout")
|
||||
|
||||
# --------------------------- (6) prod work exists only with a declaration of readiness
|
||||
|
||||
@@ -28,9 +28,9 @@ Fill a row per node as it lands. Node 1 is as-found on 2026-08-31.
|
||||
| Model | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini | HP EliteDesk 800 G5 Desktop Mini |
|
||||
| CPU | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x | i7-9700T, 8c/8t, VT-x |
|
||||
| RAM | 31 GiB (2×16 GB Micron) | **2.6 GiB usable of 16 GB — see §0b** | 31.1 GiB usable of 32 GB (2×16 GB Crucial `CT16G4SFD8266.C16FE`, **a matched pair**) |
|
||||
| Disks | 1× 477 GB SATA SSD (`C400-MTFDDAC512M`, ~2011) | 1× 477 GB **NVMe** (`PM9A1 Samsung`) | 1× 466 GB SATA SSD (`WDC WDS500G2B0A-00SM50`) |
|
||||
| Drive health | 69% life, 15 grown bad blocks, 187 erase fails | **96% life**, 2142 h, PASSED | **100% life**, 3213 h, PASSED, 0 grown bad blocks / 0 erase / 0 program fails, 0 CRC |
|
||||
| OS | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12 | Debian 13 (trixie), kernel 6.12.107+deb13-amd64 |
|
||||
| Disks | 1× 466 GB SATA SSD (`Samsung SSD 850 EVO 500GB`) — **replaced 2026-09-07, was `C400-MTFDDAC512M` (~2011); see §0c** | 1× 477 GB **NVMe** (`PM9A1 Samsung`) | 1× 466 GB SATA SSD (`WDC WDS500G2B0A-00SM50`) |
|
||||
| Drive health | PASSED, **0 reallocated / 0 grown bad blocks**, `Wear_Leveling_Count` raw 106, but **77,047 power-on hours** (~8.8 y) over only 219 power cycles | **96% life**, 2142 h, PASSED | **100% life**, 3213 h, PASSED, 0 grown bad blocks / 0 erase / 0 program fails, 0 CRC |
|
||||
| OS | Debian 13 (trixie), kernel **7.1.8+deb13-amd64** | Debian 13 (trixie), kernel **7.1.8+deb13-amd64** | Debian 13 (trixie), kernel **7.1.8+deb13-amd64** |
|
||||
| Firmware | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 | HP R21 Ver. 02.26.00 |
|
||||
| `/dev/kvm` | present | present | present |
|
||||
| Serial | MXL1052NGN | MXL0073949 | 8CC015307N |
|
||||
@@ -48,6 +48,47 @@ modern NVMe; node 3 a third distinct model, a WD Blue 3D SATA SSD in excellent h
|
||||
an identical disk layout across all three because Longhorn replica placement assumes symmetry — so
|
||||
this is a live input to the disk decision, not a footnote. Node 2's NVMe is the drive node 1 should
|
||||
have had; node 3 does not have it either. Node 3 is also the only node with **two matched DIMMs**
|
||||
|
||||
### §0c — node 1's drive was replaced (2026-09-07)
|
||||
|
||||
**The paragraph above is history, not current state.** On 2026-09-07 node 1's kubelet stopped
|
||||
posting status while the host stayed up and SSH-able. Root cause was **etcd starved of disk I/O**,
|
||||
not memory: `slow fdatasync` at 3.5–6.7 s against an expected 1 s, `apply request took too long` at
|
||||
8.3 s against 100 ms, then `etcdserver: too many requests`. No OOM, MCE or hung task in the kernel
|
||||
log. All three nodes place etcd's WAL (`/data/k3s`) and every Longhorn replica (`/data/longhorn`) on
|
||||
the SAME partition; node 1's was the 2011 C400 with 61,440 reallocated sectors and ~31% of its
|
||||
endurance consumed, and its `/proc/pressure/io` `full avg300` ran **11.36** against ~1% steady state
|
||||
on the other two. The failure is self-reinforcing: the node dropping out degrades Longhorn volumes,
|
||||
whose rebuilds add write load to the very disk etcd needs.
|
||||
|
||||
The operator replaced the drive with a **Samsung SSD 850 EVO 500GB** and reinstalled Debian.
|
||||
Measured under the same load class — an active 18-replica Longhorn rebuild, with node 1 as the
|
||||
*write target*, i.e. the heavier side:
|
||||
|
||||
| node | drive | `full avg300` during rebuild |
|
||||
|---|---|---|
|
||||
| `k3s-prod-1` | Samsung 850 EVO (new) | **3.79** |
|
||||
| `k3s-prod-2` | Samsung PM9A1 NVMe | 0.33 |
|
||||
| `k3s-prod-3` | WDC WDS500G2B0A | 2.47 |
|
||||
| *`k3s-prod-1` before* | *C400-MTFDDAC512MAM* | ***11.36*** |
|
||||
|
||||
Roughly a **3× improvement**, moving node 1 from ~4× worse than its peers into the same band as
|
||||
node 3, and it stayed `Ready` throughout. **The 850 EVO is not new**: 77,047 power-on hours over 219
|
||||
power cycles — it evidently idled in an always-on system — but its wear tells the opposite story to
|
||||
the C400's (0 reallocated vs 61,440; wear-leveling 106 vs 949).
|
||||
|
||||
**§2's exit condition is NOT met by this replacement.** etcd and Longhorn still share one SATA
|
||||
spindle. The contention is reduced, not eliminated; giving etcd its own device remains the fix, and
|
||||
node 1 still has exactly one disk. The three nodes also remain asymmetric — now 850 EVO / PM9A1
|
||||
NVMe / WD Blue.
|
||||
|
||||
The rebuild itself surfaced two things worth carrying forward. Longhorn refused the new disk with
|
||||
`DiskFilesystemChanged` — "record diskUUID doesn't match the one on the disk" — because its node CR
|
||||
still held the old drive's `diskUUID`; the remedy is disable → remove the disk entry → re-add, in
|
||||
that order (the validating webhook enforces it). And the Ansible inventory still listed node 1 in
|
||||
`[k3s_prod_primary]`, which decides both which play gets `--cluster-init` *and* what `k3s_api_host`
|
||||
resolves to — so re-provisioning it unchanged would have pointed a rebuilt, empty node at itself and
|
||||
founded a second cluster beside the live one. The founder now sits on node 2.
|
||||
(2× Crucial 2Rx8 DDR4-2666) and the only one with its **full** RAM complement of 32 GB actually
|
||||
usable. **That is luck, not immunity:** node 3's firmware publishes an EFI Memory Attributes Table
|
||||
too (`MEMATTR=0xbeece018`), the kernel simply parsed it cleanly on this boot, so node 3 is exposed to
|
||||
|
||||
@@ -92,9 +92,17 @@ proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh
|
||||
# =============================================================================
|
||||
|
||||
[k3s_prod_primary]
|
||||
k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10
|
||||
# FOUNDER MOVED TO NODE 2 (2026-09-07). Node 1 founded this cluster with --cluster-init,
|
||||
# then its SSD was replaced and Debian reinstalled, so it is now a JOINING node like any
|
||||
# other. This group decides two things at once: which play sets k3s_server_mode: primary
|
||||
# (-> --cluster-init, which FOUNDS A NEW CLUSTER) and what k3s_api_host resolves to (the
|
||||
# --server join target). Leaving node 1 here would have pointed a rebuilt, empty node at
|
||||
# ITSELF and founded a second cluster beside the live one. Node 2 is the senior surviving
|
||||
# server; its own --cluster-init is inert because /etc/rancher/k3s/k3s.yaml already exists.
|
||||
k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11
|
||||
|
||||
[k3s_prod_secondary]
|
||||
k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10
|
||||
# Uncomment a node only once it is actually on the bench and answering SSH, so no play can
|
||||
# target a host that isn't there. k3s-prod-2 came online 2026-09-01 (imaged identically to
|
||||
# node 1: same HP EliteDesk 800 G5 Mini, same BIOS R21 02.26.00, same i7-9700T, same
|
||||
@@ -110,7 +118,6 @@ k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10
|
||||
# three-way, not a node-1 outlier.
|
||||
# NOTE: node 3 also has wlp1s0 up on 192.168.2.195 (wifi). `ansible_host` is deliberately
|
||||
# the WIRED eno1 address — the same directly-routable 192.168.2.x LAN as nodes 1 and 2.
|
||||
k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11
|
||||
k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12
|
||||
|
||||
# Umbrella group group_vars/k3s_prod/ attaches to (mirrors dev's [k3s_servers:children]
|
||||
|
||||
@@ -11,3 +11,13 @@
|
||||
ansible.builtin.systemd:
|
||||
name: systemd-resolved
|
||||
state: restarted
|
||||
|
||||
# Masking a unit writes a symlink to /dev/null into /etc/systemd/system; the RUNNING
|
||||
# systemd keeps its old view of the unit until it re-reads unit files. The mask is read
|
||||
# fresh at the next boot either way, so this only exists so the running manager does not
|
||||
# disagree with on-disk state in between. `daemon_reload` re-parses unit files and
|
||||
# re-runs generators — it starts and stops nothing, so it cannot disturb k3s or Longhorn.
|
||||
# Fires only when a mask actually changed, so a re-run of the role is silent.
|
||||
- name: Reload systemd unit files
|
||||
ansible.builtin.systemd_service:
|
||||
daemon_reload: true
|
||||
@@ -331,6 +331,15 @@
|
||||
# The devices this role must never, under any circumstances, hand to e2label. Built from
|
||||
# gathered facts (`gather_facts: true` in the play) plus the live swap list, so it is the
|
||||
# NODE's account of what is load-bearing, not a hardcoded list of partition names.
|
||||
#
|
||||
# EXPECT THIS LIST TO BE EMPTY. Since the swap tasks in tasks/main.yml mask swap.target,
|
||||
# a baselined node has no active swap for `swapon` to report, so the swap half of
|
||||
# node_baseline_lh_protected_devices below contributes nothing. That is not a weakened
|
||||
# guard: the claim CANDIDATE is derived from whatever is mounted at
|
||||
# node_baseline_longhorn_claim_mountpoint, and a swap partition is neither mounted nor an
|
||||
# ext4 filesystem, so it can never be the candidate in the first place. The task stays
|
||||
# because the list must be the node's own account of what is load-bearing — including on
|
||||
# a node reached before baseline has run, or one an operator has deliberately swapped on.
|
||||
- name: "Longhorn (claim): list the devices currently backing active swap"
|
||||
ansible.builtin.command: swapon --noheadings --show=NAME
|
||||
register: node_baseline_lh_swapdevs
|
||||
|
||||
@@ -646,9 +646,109 @@
|
||||
tags: [disks]
|
||||
tags: [disks]
|
||||
|
||||
- name: Disable swap now
|
||||
ansible.builtin.command: swapoff -a
|
||||
# =====================================================================================
|
||||
# SWAP — off NOW, and still off AFTER A REBOOT — tag `system`
|
||||
# =====================================================================================
|
||||
# `swapoff -a` plus commenting the fstab line is NOT durable on these nodes, and never
|
||||
# was. Measured 2026-09-07: all three prod nodes were running with 977 MiB of swap
|
||||
# ACTIVE (k3s-prod-1 /dev/sda4, k3s-prod-3 /dev/sda4, k3s-prod-2 /dev/nvme0n1p4) while
|
||||
# their fstab swap line was commented out by this very role. The role reported success
|
||||
# and the setting silently expired at the next boot, on every node.
|
||||
#
|
||||
# CAUSE. systemd-gpt-auto-generator (systemd 257, Debian 13) claims any GPT partition
|
||||
# whose partition TYPE GUID is 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f ("Linux swap"),
|
||||
# synthesises a .swap unit for it into /run/systemd/generator.late/ and drops a symlink
|
||||
# to it into swap.target.wants/. It decides that from the PARTITION TABLE and never
|
||||
# consults /etc/fstab, so neutralising fstab cannot stop it. Re-running the generator by
|
||||
# hand on k3s-prod-1 says so in as many words:
|
||||
# Dissecting swap partition with label n/a and UUID acd3876c-fd8e-4389-b856-...
|
||||
# Probed fstype 'swap' on partition /dev/disk/by-diskseq/1-part4.
|
||||
# Adding swap: /dev/disk/by-diskseq/1-part4
|
||||
#
|
||||
# MECHANISM — mask swap.target.
|
||||
# `systemctl show` on the generated unit says its ONLY inbound dependency is
|
||||
# `WantedBy=swap.target`: Requires/RequiredBy/BoundBy are empty, and the matching
|
||||
# .device unit has an empty `Wants=`, so a device appearing does NOT pull its swap unit
|
||||
# in by itself. swap.target is therefore the single choke point that every swap
|
||||
# activation path goes through — the gpt-auto units, the units systemd-fstab-generator
|
||||
# builds from /etc/fstab, and the ones systemd-cryptsetup-generator builds for encrypted
|
||||
# swap. Masking it is:
|
||||
# * DEVICE-SHAPE INDEPENDENT. No device name appears anywhere in the change, so one
|
||||
# task is correct on prod's sda4 AND nvme0n1p4 and on dev's vda VMs, and there is
|
||||
# nothing to keep in sync when a disk is replaced. (The generated unit is named after
|
||||
# the kernel's DISKSEQ, not the disk — it is `dev-disk-by\x2ddiskseq-1\x2dpart4.swap`
|
||||
# on all three prod nodes, NVMe and SATA alike — but diskseq is assigned per boot in
|
||||
# device-probe order, so it is not a name worth depending on either.)
|
||||
# * BOOT-SAFE. swap.target is reached by `Wants=`/`After=` from sysinit.target and by
|
||||
# `After=` from tmp.mount, and `RequiredBy=` is EMPTY (verified on all three prod
|
||||
# nodes and all three dev nodes). A masked unit reached only through Wants= is
|
||||
# skipped with a log line; it cannot fail the boot transaction. Ordering deps on a
|
||||
# unit that never starts are no-ops.
|
||||
# * INERT WHEN THE DEVICE IS ABSENT OR RENAMED. The mask names no device, so there is
|
||||
# nothing to go stale: a disk swap, a new partition layout or a restore onto
|
||||
# different hardware all stay swap-free without touching this role.
|
||||
# The per-unit masks that follow are a SECOND, independent barrier over the units that
|
||||
# exist right now, so `systemctl status <unit>` says "masked" to whoever looks, and an
|
||||
# operator who unmasks swap.target does not silently get swap back. Their failure mode is
|
||||
# benign in the other direction too: a mask whose unit no longer exists is an unreferenced
|
||||
# symlink to /dev/null, and the only thing it could ever shadow is a .swap unit.
|
||||
#
|
||||
# ALTERNATIVES REJECTED.
|
||||
# * `systemd.gpt_auto=0` on the kernel cmdline — disables GPT auto-discovery WHOLESALE,
|
||||
# not just swap, and needs a GRUB edit + update-grub + a reboot before it is even
|
||||
# true. On prod that happens to be swap-only today (root= is already on the cmdline
|
||||
# so root auto-detection is off, and the /boot/efi fstab entry already suppresses the
|
||||
# ESP/XBOOTLDR mounts — the generator says so in its debug output), but on DEV the
|
||||
# root partition carries the discoverable GUID 4f68bce3-e8cd-4db1-96e7-fbcaf984b709
|
||||
# ("Linux root (x86-64)"), so the same flag would switch off root auto-discovery
|
||||
# there. Editing a bootloader to fix a swap unit is the wrong blast radius, and the
|
||||
# change would not be verifiable without the reboot it depends on.
|
||||
# * Clearing or retyping the swap partition's GPT type GUID — a partition-table write
|
||||
# to the live boot disk of a node currently serving etcd and Longhorn replicas, to
|
||||
# fix something a symlink fixes. It would also decide, irreversibly and without
|
||||
# asking, that those 977 MiB are never swap again and are simply unused; that is the
|
||||
# operator's call to make, not this role's, and it needs a maintenance window.
|
||||
# * `systemd.swap=0` — read by systemd-fstab-generator ONLY. The swap here does not
|
||||
# come from fstab, so it would change nothing at all.
|
||||
#
|
||||
# DEV. Dev's nodes have no swap partition, no swap unit and no fstab swap line at all
|
||||
# (checked on k3s-dev-1/2/3), so the mask is inert there: it flips to masked once and
|
||||
# every later run is changed=0. It is correct rather than merely harmless — if a future
|
||||
# dev image ever ships a swap partition, the same mask already covers it.
|
||||
|
||||
- name: Mask swap.target so no generator can pull a swap unit in at boot
|
||||
ansible.builtin.systemd_service:
|
||||
name: swap.target
|
||||
masked: true
|
||||
notify: Reload systemd unit files
|
||||
tags: [system]
|
||||
|
||||
- name: Find the .swap units systemd's generators produced for this boot
|
||||
ansible.builtin.find:
|
||||
# gpt-auto writes to generator.late; systemd-fstab-generator and
|
||||
# systemd-cryptsetup-generator (the other two that can emit a .swap unit) write to
|
||||
# generator. /run/systemd/generator.early is deliberately NOT listed: nothing that
|
||||
# makes swap units runs at early priority, and `find` warns on a path that does not
|
||||
# exist. Anything a future early generator invented is still caught by the
|
||||
# swap.target mask above, which is the mechanism this task only reinforces.
|
||||
paths:
|
||||
- /run/systemd/generator
|
||||
- /run/systemd/generator.late
|
||||
patterns: "*.swap"
|
||||
file_type: file
|
||||
register: node_baseline_generated_swap
|
||||
changed_when: false
|
||||
check_mode: false
|
||||
tags: [system]
|
||||
|
||||
- name: Mask each generated .swap unit (second barrier, independent of swap.target)
|
||||
ansible.builtin.systemd_service:
|
||||
name: "{{ item }}"
|
||||
masked: true
|
||||
loop: "{{ node_baseline_generated_swap.files | map(attribute='path') | map('basename') | unique | sort }}"
|
||||
loop_control:
|
||||
label: "{{ item }}"
|
||||
notify: Reload systemd unit files
|
||||
tags: [system]
|
||||
|
||||
- name: Neutralize swap entries in fstab (kubelet requires swap off)
|
||||
@@ -658,6 +758,23 @@
|
||||
replace: '# \1 # disabled for k3s/kubelet'
|
||||
tags: [system]
|
||||
|
||||
# The masks above only decide what the NEXT boot does. Anything swapped on already stays
|
||||
# on until it is told otherwise, so read the live state and act only when there is
|
||||
# something to act on — that is what makes a second run honestly changed=0 instead of
|
||||
# `changed_when: false` hiding a command that ran anyway.
|
||||
- name: Read the swap devices that are active right now
|
||||
ansible.builtin.command: swapon --noheadings --show=NAME
|
||||
register: node_baseline_swap_active
|
||||
changed_when: false
|
||||
check_mode: false
|
||||
tags: [system]
|
||||
|
||||
- name: Disable swap now
|
||||
ansible.builtin.command: swapoff -a
|
||||
when: node_baseline_swap_active.stdout | trim | length > 0
|
||||
changed_when: true
|
||||
tags: [system]
|
||||
|
||||
# =====================================================================================
|
||||
# DNS RESILIENCE — tag `dns` (R-DNS-01 / R-DNS-03)
|
||||
# =====================================================================================
|
||||
|
||||
@@ -47,7 +47,7 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
|
||||
| `storage.io_errors` | kernel SCSI/ATA `ioerr_cnt` / `iotmo_cnt` per disk (no root needed) |
|
||||
| `storage.capacity`, `storage.inodes` | worst mounted filesystem, space and inodes |
|
||||
| `storage.layout` | physical disks, and whether any device is genuinely free for `node_baseline_longhorn_device` / `node_baseline_etcd_device` |
|
||||
| `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite, EDAC correctable/uncorrectable counters |
|
||||
| `memory.available`, `memory.swap`, `memory.ecc` | headroom, kubelet's swap-off prerequisite **and whether it survives a reboot** (see [Swap off is not the same as swap disabled](#swap-off-is-not-the-same-as-swap-disabled)), EDAC correctable/uncorrectable counters |
|
||||
| `memory.physical_total` | **is this machine using the RAM it has?** `/proc/zoneinfo` present-vs-managed pages (no root, metal or VM) and `MemTotal` vs the DIMM capacity SMBIOS reports (`dmidecode -t 17`, root, bare metal). Cause-independent by design — see [The EFI memattr check](#the-efi-memattr-check) |
|
||||
| `cpu.throttle`, `cpu.inventory`, `cpu.virtualisation` | thermal throttle events since boot, topology + frequency scaling, VT-x/AMD-V + `/dev/kvm` (KubeVirt, R-PLAT-03) |
|
||||
| `thermal.max` | hottest sensor across thermal zones and hwmon chips |
|
||||
@@ -59,6 +59,37 @@ Serves **R-ENV-01** (you cannot certify a node you have not measured), **R-ENV-0
|
||||
Thresholds live in `defaults/main.yml` (`node_health_thresholds`); no verdict is hardcoded
|
||||
in task logic.
|
||||
|
||||
### Swap off is not the same as swap disabled
|
||||
|
||||
`SwapTotal == 0` only says nothing has been swapped on *yet this boot*. It is not evidence
|
||||
that swap is disabled, and on these nodes it was not: measured 2026-09-07, all three prod
|
||||
nodes came back from a reboot with 977 MiB of swap active (`k3s-prod-1` `/dev/sda4`,
|
||||
`k3s-prod-3` `/dev/sda4`, `k3s-prod-2` `/dev/nvme0n1p4`) even though `roles/node_baseline`
|
||||
had run `swapoff -a` and commented their `/etc/fstab` swap line.
|
||||
|
||||
The cause is `systemd-gpt-auto-generator`. It claims any GPT partition whose partition
|
||||
**type GUID** is `0657fd6d-a4ab-43c4-84e5-0933c84b4f4f` ("Linux swap") straight from the
|
||||
partition table, synthesises a `.swap` unit into `/run/systemd/generator.late/` and links
|
||||
it into `swap.target.wants/`. `/etc/fstab` is never consulted, so neutralising fstab
|
||||
cannot stop it. `roles/node_baseline` masks `swap.target` — the one unit every generated
|
||||
`.swap` unit is `WantedBy` — and masks the generated units as a second barrier.
|
||||
|
||||
So `memory.swap` reports on both halves, from `memory.swap_persistence` in the collector
|
||||
(all world-readable, no root):
|
||||
|
||||
| fact | meaning |
|
||||
|---|---|
|
||||
| `swap_target_masked` | is `/etc/systemd/system/swap.target` a symlink to `/dev/null`? |
|
||||
| `generated_units` | `.swap` units systemd's generators built for this boot |
|
||||
| `masked_units` | `.swap` units masked in `/etc/systemd/system` |
|
||||
| `unmasked_generated_units` | `generated_units` minus `masked_units` — what a reboot would bring back |
|
||||
| `fstab_active_swap_lines` | uncommented `swap` lines left in `/etc/fstab` |
|
||||
|
||||
A node with no swap active but an unmasked generated unit and an unmasked `swap.target`
|
||||
**WARNs**: it is clean now and will be swapping again after the next reboot. The check
|
||||
used to blame that state on "this node has not been baselined yet", which sent the reader
|
||||
back to re-run the role that had already run.
|
||||
|
||||
|
||||
## The EFI memattr check
|
||||
|
||||
|
||||
@@ -301,9 +301,94 @@ def collect_memory():
|
||||
# Firmware's own inventory of what is physically installed. Root-only; attempted
|
||||
# here anyway so the unprivileged pass records the exact reason it could not read.
|
||||
mem["devices"] = collect_memory_devices()
|
||||
# "Is swap off right now?" and "will it still be off after a reboot?" are DIFFERENT
|
||||
# questions, and only the second one is kubelet's actual prerequisite. See below.
|
||||
mem["swap_persistence"] = collect_swap_persistence()
|
||||
return mem
|
||||
|
||||
|
||||
# --- Will swap still be off after the next reboot? -----------------------------------
|
||||
# SwapTotal=0 does not mean swap is DISABLED — it means nothing has been swapped on yet
|
||||
# this boot. Measured 2026-09-07 on all three prod nodes: swap was active again after a
|
||||
# reboot, 977 MiB each, with /etc/fstab's swap line commented out by node_baseline. The
|
||||
# cause is systemd-gpt-auto-generator, which claims any GPT partition typed
|
||||
# 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f ("Linux swap") straight from the PARTITION TABLE,
|
||||
# synthesises a .swap unit into /run/systemd/generator.late/ and links it into
|
||||
# swap.target.wants/. It never reads fstab, so neutralising fstab cannot stop it.
|
||||
#
|
||||
# roles/node_baseline masks swap.target — the single unit every generated .swap unit is
|
||||
# `WantedBy` — and masks the generated units themselves as a second barrier. This
|
||||
# collects the facts that let the report tell "off, and staying off" apart from "off
|
||||
# until the next reboot", which a /proc/meminfo reading alone can never do.
|
||||
#
|
||||
# All of it is world-readable (the generator dirs are 0755 root:root), so it works in the
|
||||
# UNPRIVILEGED pass: no root, no systemctl, no parsing of systemctl output.
|
||||
def _is_systemd_mask(path):
|
||||
"""True when `path` is a systemd mask, i.e. a symlink to /dev/null."""
|
||||
try:
|
||||
return os.path.islink(path) and os.path.realpath(path) == "/dev/null"
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
def collect_swap_persistence():
|
||||
out = {
|
||||
"readable": False,
|
||||
"swap_target_masked": None,
|
||||
# .swap units systemd's generators built for THIS boot (gpt-auto, fstab,
|
||||
# cryptsetup). These are the ones that would be activated again next boot.
|
||||
"generated_units": [],
|
||||
# .swap units masked in /etc/systemd/system — what node_baseline wrote.
|
||||
"masked_units": [],
|
||||
# generated MINUS masked: the units that would still come back. Empty is good.
|
||||
"unmasked_generated_units": [],
|
||||
"fstab_active_swap_lines": None,
|
||||
}
|
||||
|
||||
generated = set()
|
||||
for directory in (
|
||||
"/run/systemd/generator.early",
|
||||
"/run/systemd/generator",
|
||||
"/run/systemd/generator.late",
|
||||
):
|
||||
try:
|
||||
names = os.listdir(directory)
|
||||
except FileNotFoundError:
|
||||
# A generator produced nothing. That is a real answer, not a read failure:
|
||||
# dev's VMs have no swap partition, so gpt-auto emits no unit at all.
|
||||
continue
|
||||
except OSError as exc:
|
||||
UNREADABLE.append({"path": directory, "reason": "read error: %s" % exc})
|
||||
return out
|
||||
for name in names:
|
||||
if name.endswith(".swap") and os.path.isfile(os.path.join(directory, name)):
|
||||
generated.add(name)
|
||||
|
||||
masked = set()
|
||||
try:
|
||||
for name in os.listdir("/etc/systemd/system"):
|
||||
if name.endswith(".swap") and _is_systemd_mask("/etc/systemd/system/" + name):
|
||||
masked.add(name)
|
||||
except OSError as exc:
|
||||
UNREADABLE.append({"path": "/etc/systemd/system", "reason": "read error: %s" % exc})
|
||||
return out
|
||||
|
||||
out["swap_target_masked"] = _is_systemd_mask("/etc/systemd/system/swap.target")
|
||||
out["generated_units"] = sorted(generated)
|
||||
out["masked_units"] = sorted(masked)
|
||||
out["unmasked_generated_units"] = sorted(generated - masked)
|
||||
out["readable"] = True
|
||||
|
||||
fstab = read_text("/etc/fstab")
|
||||
if fstab is not None:
|
||||
out["fstab_active_swap_lines"] = sum(
|
||||
1
|
||||
for line in fstab.splitlines()
|
||||
if not line.lstrip().startswith("#") and re.search(r"\sswap\s", line)
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
# --- Memory the kernel has but never hands out --------------------------------------
|
||||
# /proc/zoneinfo is WORLD-READABLE on every Linux, VM or metal — no root, no dmidecode,
|
||||
# no DMI. Per zone it reports:
|
||||
|
||||
@@ -499,25 +499,76 @@
|
||||
if phys_status in ['WARN', 'FAIL'] else '')
|
||||
~ (phys_cause if phys_status in ['WARN', 'FAIL'] else '') }}
|
||||
|
||||
- name: Assess — swap (kubelet requires it off)
|
||||
# WHY THIS CHECK IS NOT JUST `SwapTotal == 0`. It used to be, and it read the answer
|
||||
# wrong in both directions. It called active swap "this node has not been baselined yet",
|
||||
# which was a MIS-DIAGNOSIS — measured 2026-09-07, all three prod nodes had been
|
||||
# baselined, had their fstab swap line commented out by node_baseline, and were STILL
|
||||
# swapping 977 MiB each after a reboot. Whoever read that WARN would have re-run the very
|
||||
# role that had already run, and watched swap come back at the next boot again.
|
||||
#
|
||||
# The real mechanism is systemd-gpt-auto-generator: it claims any GPT partition typed
|
||||
# 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f ("Linux swap") from the PARTITION TABLE, builds a
|
||||
# .swap unit into /run/systemd/generator.late/ and links it into swap.target.wants/. It
|
||||
# never reads /etc/fstab, so `swapoff -a` + a commented fstab line only holds until the
|
||||
# next boot. roles/node_baseline now masks swap.target (the single unit those generated
|
||||
# units are `WantedBy`) plus the generated units themselves.
|
||||
#
|
||||
# The other direction matters just as much: SwapTotal == 0 is NOT proof that swap is
|
||||
# disabled — a node freshly `swapoff`-ed reads zero and swaps again on reboot. So this
|
||||
# check now asks both questions, and WARNs on a node that is clean right now but has an
|
||||
# unmasked generated swap unit waiting for the next boot.
|
||||
- name: Assess — swap (kubelet requires it off, and off ACROSS REBOOTS)
|
||||
ansible.builtin.set_fact:
|
||||
node_health_checks: "{{ node_health_checks + [{
|
||||
'id': 'memory.swap',
|
||||
'title': 'Swap disabled (kubelet prerequisite)',
|
||||
'title': 'Swap disabled, and disabled across reboots (kubelet prerequisite)',
|
||||
'status': swap_status,
|
||||
'detail': swap_detail }] }}"
|
||||
vars:
|
||||
ansible_become: false
|
||||
m: "{{ node_health_facts.memory }}"
|
||||
p: "{{ node_health_facts.memory.swap_persistence | default({}, true) }}"
|
||||
# Durability was actually measurable on this node (the collector could read
|
||||
# /etc/systemd/system and the generator dirs). When false, say so rather than
|
||||
# implying a clean bill from a failed read.
|
||||
swap_durable_known: "{{ (p.readable | default(false)) | bool }}"
|
||||
# Units a reboot would bring back: produced by a generator, not masked, and with
|
||||
# swap.target itself unmasked so something can still pull them in.
|
||||
swap_will_return: >-
|
||||
{{ (swap_durable_known | bool)
|
||||
and not (p.swap_target_masked | default(false) | bool)
|
||||
and (p.unmasked_generated_units | default([], true) | length) > 0 }}
|
||||
swap_units_returning: "{{ p.unmasked_generated_units | default([], true) | join(', ') }}"
|
||||
swap_active_gib: "{{ (m.swap_total_kb | int / 1048576) | round(2) | string }}"
|
||||
swap_status: >-
|
||||
{{ 'SKIP' if m.swap_total_kb is not number
|
||||
else ('PASS' if m.swap_total_kb | int == 0 else 'WARN') }}
|
||||
else ('WARN' if (m.swap_total_kb | int) > 0
|
||||
else ('WARN' if (swap_will_return | bool) else 'PASS')) }}
|
||||
swap_cause: >-
|
||||
systemd-gpt-auto-generator re-creates a .swap unit for the GPT partition typed
|
||||
0657fd6d-a4ab-43c4-84e5-0933c84b4f4f ("Linux swap") at EVERY boot, straight from
|
||||
the partition table and without reading /etc/fstab. This is NOT "the node has not
|
||||
been baselined" — `swapoff -a` and a commented fstab line simply do not survive a
|
||||
reboot. The fix is roles/node_baseline masking swap.target and the generated units:
|
||||
re-run `node-baseline.yml --tags system` against this node.
|
||||
swap_detail: >-
|
||||
{{ 'swap state unreadable' if m.swap_total_kb is not number
|
||||
else ('no swap active' if m.swap_total_kb | int == 0
|
||||
else ((m.swap_total_kb | int / 1048576) | round(2) | string
|
||||
~ ' GiB swap active — roles/node_baseline runs swapoff and neutralises
|
||||
the fstab entry; this node has not been baselined yet')) }}
|
||||
else ((swap_active_gib ~ ' GiB swap active'
|
||||
~ (' via ' ~ swap_units_returning if swap_units_returning else '')
|
||||
~ ' — ' ~ swap_cause)
|
||||
if (m.swap_total_kb | int) > 0
|
||||
else ('no swap active, but it would come back at the next boot via '
|
||||
~ swap_units_returning ~ ' — ' ~ swap_cause
|
||||
if (swap_will_return | bool)
|
||||
else ('no swap active'
|
||||
~ (' and none can be activated: swap.target is masked'
|
||||
if (p.swap_target_masked | default(false) | bool)
|
||||
else (' and no swap unit exists on this node'
|
||||
if (p.generated_units | default([], true) | length) == 0
|
||||
else ' and every generated swap unit is masked'))
|
||||
if (swap_durable_known | bool)
|
||||
else 'no swap active (durability NOT verified — the collector
|
||||
could not read /etc/systemd/system or /run/systemd)'))) }}
|
||||
|
||||
- name: Assess — ECC / EDAC correctable errors
|
||||
ansible.builtin.set_fact:
|
||||
|
||||
Reference in new issue
Block a user