712f4c93367bc72d2e2d27eda90c52433d4bf230
695
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
712f4c9336 |
fix(conformance): "unit-tested" counted an alert MENTIONED in a comment as tested
alert-rules-unit-tested decided whether an alert had a unit test with a bare word
match over the whole test file. Any mention counted — including prose in a `#`
comment.
Hit for real while adding platform/monitoring/tests/vpn-egress.test.yaml. Its header
explains that VPNTunnelDown stays quiet during an egress-steering failure, and that
sentence alone made this check declare VPNTunnelDown "asserted on" and demand its
removal from UNTESTED_ALERTS. Removing the entry would have recorded coverage that did
not exist — the precise "looks fixed" outcome this check exists to prevent, produced
by the check itself. (In that instance I made the coverage real instead, which also
exposed that the fixtures were missing tun0's receive counter entirely.)
Now an alert counts only when a test file names it in an assertion position, with
comment lines stripped first so a commented-out assertion does not count either.
BOTH promtool idioms are accepted, because both are real assertions:
alert_rule_test: - alertname: CNPGPrimariesCoLocated
promql_expr_test: - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
Matching only the first is not a stricter check, it is a WRONG one: it reported
CNPGPrimariesCoLocated as untested when platform/cnpg/alerts.promtool-test.yaml pins
it across four scenarios entirely through the second form. Believing that would have
produced a redundant duplicate test for an alert already covered better than most —
so the first version of this very fix had to be corrected before it landed.
MEASURED BEFORE LANDING, because the registers are shrink-only and a prose-only alert
would have required an entry this check forbids adding: of 61 alerts, 32 are asserted
in one of the two idioms, 29 are not mentioned at all (already declared), and ZERO were
prose-only. So this tightening failed nothing on the day it landed.
My first measurement said 31/30/0 and was wrong in a way worth recording: it scanned
only tests/*.test.yaml and missed the <manifest>.promtool-test.yaml companion
convention, which is exactly what hid the CNPGPrimariesCoLocated false negative.
Verified in both directions: 134 assertions pass with the real coverage intact, and a
temporary prose-only mention of NodeExporterDown (a registered untested alert) leaves
it correctly listed as untested instead of triggering the stale-entry failure.
Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
e3e56af461 |
docs(decisions-log): overnight handoff — backup alert crying wolf, 49 unmapped app manifests, egress branch finished
Records the three fixes and, in each case, the thing that was reporting green or red over something other than what it claimed to measure. Includes corrections to my own numbers (the '192 unmapped files' figure was my audit script's regex matching across a newline, not the repo) and the two check defects the egress branch exposed. PM HANDOFF lists open items in priority order. R-STATE-02 remains the Phase 7 exit item and is blocked on the permission classifier, not on readiness. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a3f4ddb971 |
merge: feat/egress-gateway-staleness-alert — finished so it can actually pass the gate
Brings the two alerts for the silent egress-gateway failure: VPNEgressSteeringStale
(critical) and VPNEgressUplinkLabelMissing (warning), plus the kube-state-metrics
allowlist entry that makes them non-inert.
WHY THE BRANCH SAT IN THE BACKLOG: it could not pass the merge gate. Three
whole-repo checks failed on it, and each one was a real gap rather than a nuisance:
* alert-runbooks-resolvable — both alerts link to /runbooks/vpn-egress-stale,
which did not exist. An alert firing now would hand the operator a 404.
* alert-rules-unit-tested — alerts.yaml is already a covered file, so new rules
in it must be tested or a "tested" file quietly grows untested rules.
* alert-metrics-have-scrape-targets — see the guard bug below.
WHAT THE ALERTS CATCH. egress-class=vpn pods leave through one gateway node's tun0;
Cilium resolves it to an ifindex and caches it. An OpenVPN reconnect destroys and
re-creates tun0 with a NEW ifindex, and the node label
homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex> is what forces Cilium to
re-resolve. If the label does not follow the tunnel, the cache points at a destroyed
interface and every vpn-class pod is silently dropped — while tun0 stays up and
VPNTunnelDown says nothing. That ran ~80 minutes undetected on the dev gateway.
Verified this is live on prod, not inert: k3s-prod-3 carries egress-gateway=true and
egress-uplink=10.150.192.86-139, and vpn-egress-steering has been in place 14 days.
The label was exported on 0 series before this change, so the allowlist entry is
what switches the alerts on.
runbooks/vpn-egress-stale.md — written and wired into the configMapGenerator, with
the four required operator sections. It leads with proving egress from a pod the
policy selects, because the symptom is "outbound traffic silently disappears", and
it says plainly not to "fix" this by deleting the policy: that does not restore
egress, it sends vpn-class traffic out of the real ISP connection, which is the leak
the class exists to prevent.
platform/monitoring/tests/vpn-egress.test.yaml — four promtool fixtures. 1 and 2 are
the SAME tunnel reconnect with and without the label following it, which isolates the
mechanism the rule actually keys on: a label VALUE change is not a changing sample,
it retires one series and starts another. 3 is the inert case (gateway node, no uplink
label) where the critical rule CANNOT fire, which is why the warning exists.
Two fixes the branch needed that were defects in the checks themselves:
1. alert-metrics-have-scrape-targets asserted the KSM allowlist matched
`nodes=[...egress-gateway]` with a literal closing bracket — i.e. that the list
contained EXACTLY that one label. Adding a second label correctly (egress-uplink,
which these alerts need) therefore FAILED the check while the property it guards
was perfectly satisfied. Loosened to `[,\]]`: in the list, not the whole list. A
guard that breaks when you correctly extend the thing it guards teaches people to
edit the guard, which is how it stops guarding.
2. UNTESTED_ALERTS listed VPNTunnelDown, and the check flagged it as stale because
"asserted on" is a bare word match over the test file — my header COMMENT
mentioning VPNTunnelDown was enough to count as coverage. Rather than delete the
register entry and claim coverage that did not exist, the fixtures now assert
VPNTunnelDown QUIET in all four cases. That required making them faithful: they
had no node_network_receive_bytes_total{device="tun0"} at all, so VPNTunnelDown
would have been firing throughout unnoticed, because promtool only checks the
alertnames you name. A healthy tun0 passes traffic, and that silence IS the claim
these rules exist for.
NOTE FOR LATER, not fixed here: that word-match means any alert named in prose counts
as tested. Tightening it to `alertname:` positions could reveal other prose-only
"coverage", and the register is shrink-only, so the cascade wants doing deliberately
rather than at 01:00 unattended.
Conflicts resolved: kept HEAD's Velero custom-resource-state block and took only the
branch's one-line allowlist change (14 lines down to 1); inserted the branch's
2026-09-12 DECISIONS-LOG entry in date order rather than appending it out of sequence.
Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired. The new unit tests pass under the pinned promtool 3.14.0.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85
|
5f3ea63b7f | merge: fix/touched-requirements-maps-apps-and-c2 | ||
|
jakman85andClaude Opus 5
|
23c335257b |
fix(ci): apps/* and the C2 UI attributed no requirement, so no behaviour test gated them
ci/touched-requirements.sh's coarse path map had no arm for apps/*. A change adding
or editing ANY workload on the cluster — 49 tracked manifests across apps/wave-0,
-1, -2 and apps/vms — attributed zero requirements, so the merge gate selected only
the whole-repo-scoped checks and none of the requirement-scoped L2 behaviour tests.
The tests that enforce default-deny, egress class and auth mode did not run on
exactly the changes that can violate them. Kyverno admission and the L0 egress-path
check are real backstops, but admission is post-merge and the gate exists to catch
it first.
platform/homepage and platform/headlamp were unmapped too — the C2 UI. That gap was
already written down in changed_diff()'s rule-3 comment ("the ONLY route to
R-C2-01/02 was rule 3", i.e. editing the C2 test itself) and had sat there since.
Twelve arms added: apps/vms/* (ordered BEFORE apps/*, because `case` takes the first
match), apps/*, platform/homepage|headlamp, platform/kubevirt, platform/gateway,
platform/coredns, platform/crafty, platform/external-snapshotter|backup, runbooks/*,
platform-config/*, platform/kustomization.yaml and .sops.yaml.
platform/kustomization.yaml earns its own arm: it is the assembly list, so one
deleted line disables an entire platform component while every file it points at
stays perfectly valid on disk. That is the most consequential single-line edit in
the repo and it attributed nothing.
Measured effect for an apps/ change, via --print-selection:
before: 0 requirement-scoped tests
after: 13 (3 L1, 8 L2, 2 L3)
Coverage stays 51/51, so the gate's "attributed IDs must be covered" assertion does
not false-block.
Ten new assertions in ci/tests/touched-requirements.test.sh (26 total, all passing),
including the ORDER property — apps/vms/* must win over apps/* — because arm order
is a real failure mode of `case`, not merely a question of both arms existing. Each
fixture cites no R-ID of its own, so any ID in the output came from the PATH and not
the content scan.
A correction on numbers I reported earlier in this session: I claimed "192 unmapped
files", then "182". Both were inflated by my own audit script rather than the repo.
Its regex used `[^)]*`, which matches across a newline, so the first arm parsed as
`case "$1" in\n catalog/*` and never matched anything; I also mis-parsed
alternation arms like `.gitea/*|ci/*`, which made ci/ look unmapped when it is
mapped. True remainder after this change is 18 files, all prose or captured
artifacts (docs/plans/, root READMEs, .standup-dns/*.txt).
Gated: offline l0,l1,meta --include-scope whole-repo, exit 0.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85
|
9339d5a79e | merge: fix/velero-skip-emptydir-volumes | ||
|
jakman85andClaude Opus 5
|
a1cc0a5556 |
fix(velero): skip emptyDir volumes by type; revert the narrower namespace exclusion
Supersedes
|
||
|
jakman85
|
6eaaaae334 | merge: fix/node-memory-swap-cnpg-placement | ||
|
jakman85andClaude Opus 5
|
cde02d19d2 |
fix(runbooks): node-memory-swap claimed the CNPG primaries sit on node 1
They do not, and the correction changes the operator's plan rather than just a detail. Verified live: all three CNPG clusters run 3 instances spread one per node by required anti-affinity, and k3s-prod-1 holds a REPLICA of each (keycloak-db-2, nextcloud-db-2, semaphore-db-2). No primary is on node 1, so draining it forces no failover at all — the best state this maintenance could ask for. The original sentence was carried over from earlier session state and was simply stale; the runbook now prints the command to re-derive it, because primaries move on their own after any restart or failover. Adds the warning that matters more. CNPGPrimariesCoLocated is currently firing for k3s-prod-2, which holds both the keycloak-db and semaphore-db primaries. The instinct is to tidy that before a maintenance window. Doing so would be actively harmful here: rebalancing means promoting a replica, the only node with spare primary capacity is node 1, and so "fixing" the alert would put a primary on the very node about to be powered off — converting a no-failover drain into a two-database failover. Rebalance after node 1 is back and caught up. Gated: L0 whole-repo 21 cases, 0 failures, 0 skips, 0 unwired. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
05a8094785 | merge: fix/velero-partialfail-on-own-maintenance-jobs | ||
|
jakman85andClaude Opus 5
|
d99d47494c |
fix(velero): nightly backup PartiallyFailed on Velero's own maintenance Jobs
VeleroNoRecentBackup has been firing CRITICAL on prod over a backup that was actually protecting everything. velero-daily-20260919030041: PartiallyFailed, 3079/3079 items backed up, exactly 2 errors — both PodVolumeBackups for the `plugins` and `scratch` volumes of homepage-default-kopia-maintain-job-1789786837269-qhsvh. Four such PVB failures had accumulated. Velero runs transient kopia repository-maintenance Jobs in its own namespace, one per BackupRepository, each with emptyDir `plugins` and `scratch` volumes. Schedule (1) runs defaultVolumesToFsBackup:true — the opt-out model — so it attempts fs-backup on those volumes too. The Job finishes and its pod disappears mid-backup, and the PodVolumeBackup fails. Nothing real was ever missed. But PartiallyFailed does not advance velero_backup_last_successful_timestamp, so the alert that exists to catch a genuine backup outage fires every night over Velero cleaning up after itself. An alert that cries wolf nightly is how a real failure gets ignored — and this one guards R-STATE-03. Fix uses the pattern already established for monitoring+unifi: exclude the namespace's volume DATA from schedule (1), keep its RESOURCES in schedule (2). Safe by inspection — ns velero holds no PVCs, so there is no volume data to lose, while the BSL, credentials and Backup CRs that DR actually needs are still captured. Deliberately narrower than the general rule. The general fix is a volume policy skipping emptyDir by type, which would also cover a transient Job in any other namespace, but it needs Velero's `volumeTypes` policy condition — unproven in this repo on 1.18.1, and a policies.yaml Velero cannot parse breaks EVERY backup. That is a much worse failure than the one being fixed, and all four observed failures were kopia-maintain pods in ns velero. Rationale recorded inline so the next person can widen it on purpose rather than by accident. Gated: L0 whole-repo 21 cases, 0 failures, 0 skips, 0 unwired. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
6f231a4311 | merge: docs/decisions-log-gate-fail-open-holes | ||
|
jakman85andClaude Opus 5
|
623fa14256 |
docs(decisions-log): PM handoff — two fail-open holes in the gate itself
Records the session's three pieces of work and, more importantly, the theme joining them: the gate was reporting green over things it had not checked. - runbooks/node-memory-swap.md, with the three claims I wrote and then removed because they were false (EtcdIsVoter does not exist here; node-drain-policy is not pinned so the runbook reads it live; the 5-vs-1 rebuild-limit confusion). - The nested-worktree scan: six discovery sites auditing 18 copies of the repo. - L1 rendering never running on a requirement-scoped gate, with the external-dns incident reproduced via --print-selection (0 tests before, 3 after). - The two defects that surfaced in the meta-check guarding all of it. PM HANDOFF lists the open items in priority order. The first is a judgement call worth flagging rather than burying: the planned memory swap on k3s-prod-1 IS a supervised single-node loss on prod, which is exactly the event R-STATE-02 describes and which is still proven only on the decommissioned dev cluster. Capturing evidence during the swap costs nothing; manufacturing the event later costs a second maintenance window. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
16cbd4a79e | merge: fix/l1-rendering-declares-whole-repo-scope | ||
|
jakman85andClaude Opus 5
|
755c08ab40 |
fix(conformance): the meta-check was layer-agnostic and reported failures silently
Declaring whole-repo scope on the L1 tests made whole-repo-invariants-wired (R-TEST-03) fail. It was right to change behaviour and wrong about why, and it could not tell anyone either way. Two independent defects. 1. LAYER-AGNOSTIC WHERE THE DESIGN IS LAYER-BOUNDED The "every scope-declared test is selected despite a requirement filter matching none" assertion compared EVERY scope-declaring dir against an `--layer l0` selection. That silently assumed only L0 tests would ever declare the scope — while ci/l1.sh was written from the opposite premise, that a whole-repo invariant could appear at any layer. So the first L1 declaration made the check blame the union for correctly excluding another layer. Now partitioned by layer, and the L1 half of the property is asserted too: every scope-declared L1 test must be selected by `--layer l1 --include-scope`, and that selection must not leak non-L1 tests. That is what makes the L1 declaration load-bearing rather than decorative — previously nothing proved the union reached L1 at all, because nothing at L1 declared the scope. 2. EVERY ASSERTION FAILED SILENTLY The idiom throughout was a bare test on one line and `chk $?` on the next, under `set -euo pipefail`. A false test aborts the script BEFORE chk can print, so the diagnostic — which is the entire value of this check — was never reached. The real failure above emitted 26 `ok` lines, rc=1, and no message at all; it read as a tooling glitch rather than a finding. All 12 assertion sites now capture the status (`chk $arc`) so the check reports and continues. Proven by forcing MIN_DECLARED=999 against the real check: the failure prints its full explanation and the run proceeds to the remaining assertions, where before it died mute. A meta-check that cannot say why it failed is worse than no meta-check. Verified: 35 assertions, rc=0; full offline l0,l1,meta 25 tests / 0 failures. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
b389566641 |
fix(conformance): L1 rendering never ran on a requirement-scoped gate
All three L1 tests now declare `conformance.home.k3s/scope: "whole-repo"`.
ci/l1.sh has passed --include-scope whole-repo since it was written, with a
comment saying no L1 test declared the scope yet and it was wired ahead of time
so the next whole-repo invariant "at any layer" would be picked up by annotation
alone. Nothing ever declared it. The consequence is that the entire L1 rendering
layer only ran when a touched path happened to attribute R-PLAT-02, R-NET-01 or
R-NET-05/06/07 — a fail-open hole for every other change in the repo.
The coupling requirement-scoped selection cannot see is that all three tests
exercise ONE code path: catalog/render.py against
schemas/servicedefinition.schema.json. Anything that changes rendered output
invalidates these goldens no matter which requirement the touched path maps to.
The 2026-09-04 external-dns merge is the proof, measured rather than asserted.
ci/touched-requirements.sh attributes platform/externaldns/helmrelease.yaml to
R-ENV-02, R-OBS-01/02/04, R-STATE-03 and R-TEST-01/02/03/04 — none of the
rendering requirements — yet that change added
external-dns.alpha.kubernetes.io/controller: none
to the rendered output of all three goldens. Via --print-selection:
requirement-scoped only (the 2026-09-04 behaviour): 0 L1 tests selected
with --include-scope whole-repo (how ci/l1.sh runs): 3 L1 tests selected
So the merge gated GREEN while leaving every rendering golden stale, and the
staleness surfaced only on a later full run.
Also corrects the now-false comment in ci/l1.sh that said no L1 test declares the
scope, and records why the annotation is load-bearing there.
Verified: offline l0,l1,meta --include-scope whole-repo in the pinned toolbox —
25 tests, 0 failures, 0 errors, exit 0.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85
|
7887ecbbdc | merge: docs/node-memory-swap-runbook | ||
|
jakman85
|
d717d8e979 | merge: fix/conformance-skips-nested-worktrees | ||
|
jakman85andClaude Opus 5
|
bc106caac1 |
docs(runbooks): node-memory-swap — replacing k3s-prod-1's mismatched DIMM pair
The operator has a matched 32 GB pair to install. k3s-prod-1 is the node that needs it: DIMM1 is a Micron 8ATF2G64HZ-3G2E1 (16 GB, DDR4-3200, 1Rx8) and DIMM3 a 16ATF2G64HZ-2G6E1 (16 GB, DDR4-2667, 2Rx8), so capacity is correct at 32 GB but the pair runs at the slower 2667. Nothing is faulty. k3s-prod-3 is a matched Crucial pair. The procedure exists because this node is the awkward one to take down: it held 24 Longhorn replicas, it is one of three etcd members (so the drain leaves quorum at 2 of 3 with no further margin), and its rebuilds are the load that starved etcd once already. So the runbook front-loads the checks that decide whether to start at all, and treats a BLOCKED drain as the safety property working rather than an obstacle to force past. Two things it deliberately does not assert: - node-drain-policy is not pinned in platform/longhorn/helmrelease.yaml, so the runbook has the operator READ the live Setting and set it before draining if it is not block-if-contains-last-replica, instead of assuming the default. - Configured Memory Speed, not Speed, is named as the field that proves the swap worked — a matched pair being clocked down still reports the rated Speed. It also flags the capacity decision as a decision: 2x16 GB keeps the three nodes symmetric as PROD-PROVISIONING.md §2 assumes, while 2x32 GB makes node 1 the asymmetric one on memory as well as disk. Recorded caveats: the node-health artifacts de-duplicate DIMM part numbers, so they can prove "two different modules" but cannot distinguish one module from two identical ones — verification therefore collects dmidecode live. And node 2's inventory is stale and self-contradictory (documented as the only single-DIMM node at 16 GB, yet reporting ~31.1 GiB), so it needs re-inventory before anyone claims three-way symmetry. This is NOT the §0b "do not buy RAM" case, which was an EFI memattr kernel bug on node 2. Wired into the configMapGenerator so it is served at /runbooks/node-memory-swap rather than existing only in git. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
79c703bc79 |
fix(conformance): whole-repo checks audited nested agent worktrees as source
Six discovery sites walked ROOT without pruning dot-directories, so they descended into .claude/worktrees/agent-*/ — up to 18 full copies of the repo, created for isolated agent runs and gitignored (.gitignore:43). The visible half was false FAILs: four ServiceMonitors reported as selecting no Service, and nine PrometheusRule manifests reported as "under test but not in the repo", all of them worktree paths. The whole-repo L0 gate was red for reasons that had nothing to do with authored surface. The dangerous half is quieter. A check that asks "does anything in the repo declare X" can be satisfied by a worktree copy instead of real source, so the same bug that invents failures can also manufacture a pass. That is the reason to fix all six sites rather than only the ones that happened to be failing. egress-paths-selected-by-rules already had this fix and documents it (it once saw 5 CiliumEgressGatewayPolicy objects instead of the 1 authored); this applies the same dot-directory predicate everywhere else, which skips dot-DIRECTORIES while still matching dot-FILES such as .sops.yaml. platform/monitoring/tests/run.sh is the other side of the same failure: its `--list` reported each manifest once per worktree, and the L0 check compares that listing against the tree. Both find calls now prune .claude alongside .git. `--list` returns 21 real pairs where it previously returned them ~19x over. Verified in the pinned toolbox: L0 --include-scope whole-repo is 21 cases, 0 failures, 0 skips, 0 unwired (was 4 failures, then 1). L1 3/3. The 18 stale worktrees are all on already-merged branches and remain on disk; cleaning them up is a separate change and is no longer load-bearing for the gate. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
3523617671 | Rolled unifi console password | ||
|
jakman85andClaude Opus 5
|
6a4dc0ada7 |
docs: INCIDENT — I printed the console password to the transcript
A kubectl jsonpath with {range .data}{@}{end} dumped the unifi Secret's data
map, putting the base64 console-password in the session transcript. Exactly the
constraint I put in every agent brief, broken by my own careless verification
command.
Exposure bounded: 0 commits contain it (git log -S), the capturing scratch file
was shredded, and a repo-wide grep finds nothing on disk. Critically it is not
yet a live credential — the Secret was created two days after the running VMI,
whose volume list is rootdisk and cloudinit only, so the UOSDATA disk was never
attached, install-uos never ran, and the ubuntu account is still locked.
Action recorded with the ordering that matters: rotate console-password BEFORE
recreating the VMI, so the leaked value never becomes valid.
Lesson: checking a Secret's existence never needs its values. Listing .data's
keys or .metadata.name answers the question; dumping .data answers one nobody
asked.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
14d5ccbadd |
fix(unifi): pin the MAC — an unpinned MAC is why the guest keeps losing its IP
Operator set console-password in the SOPS secret (installer-url deliberately left as REPLACE-ME, which install-uos handles by printing 'not set yet' and exiting 0; the uosserver guard exits before it is even read). Root cause of the lost address, from evidence rather than inference. KubeVirt generates a fresh MAC per VMI when none is pinned. cloud-init writes /etc/netplan/50-cloud-init.yaml matching the MAC it saw, and will not rewrite it while the NoCloud instance-id is unchanged — and that instance-id is domain.firmware.uuid, stable across restarts by design. So every VMI recreation handed the guest a MAC its retained netplan did not match, leaving enp1s0 administratively DOWN, never even attempting DHCP. What the 2026-09-12 boot log shows, and why it fits: cloud-init's own device table reads 'enp1s0 | Up: False | Address: .'; there are no apt failures because the per-instance modules were skipped; yet UOS is installed (6.3 GB of 40 GB) and qemu-guest-agent answers — both of which required working network on an EARLIER VMI. The MAC is identical between the current VMI status and that boot log, so it did not change on this boot; the mismatch is against a netplan written under a previous incarnation. Pinned to the MAC the current VMI already holds, so nothing changes for this guest. Combined with the instance-id bump already merged, the next restart re-runs cloud-init, rewrites netplan against a MAC that is now permanent, and the interface should come up — and the console password is in place as the fallback if it does not. Still unverified from outside: the netplan file's actual contents. That needs the console the operator can now reach. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
05881d5a6a |
fix(conformance): regenerate the three L1 goldens the external-dns change drifted
|
||
|
jakman85andClaude Opus 5
|
cf0d6c6beb |
docs: verified — internal services are gone from public DNS, split-horizon works
Confirmed at the authority, not a cache: all six internal names absent publicly, all five public names untouched, and Pi-hole answering for the six internally. The R-DNS-03 disclosure is closed for that tier. Records a correction to my own sequencing: I insisted this could not land before the DHCP repoint, but the router returns nothing for those six names, so the records served nobody and the dependency I asserted did not exist. One query would have shown it. Also records the operator's clarifications — Pi-hole is DNS only with the router still doing DHCP, reverse proxy by hostname rather than a second IP, and Minecraft staying open — plus the unencrypted-secret fix. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
20a28d0993 |
fix(unifi): the installer secret was committed unencrypted
Every other *.sops.yaml in this repo carries sops metadata and ENC[] values; this one shipped with neither and both fields as literal REPLACE-ME. Two consequences: 'sops <file>' fails with 'sops metadata not found', so the documented operator step could not work; and Flux would pass the file through undecrypted and create the Secret with the literal placeholder strings. Encrypted in place with the repo's age recipient, placeholders intact. Round-trip verified: sops -d returns installer-url and console-password as REPLACE-ME, so the operator's workflow is now the normal one — sops the file, set both values, commit. Both fields need setting, not just the password: installer-url is also REPLACE-ME, which I had omitted from the operator instructions. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
b184a42729 |
merge: internal services stop advertising themselves in public DNS
Six exposure:internal services published A records into the public Cloudflare zone answering 192.168.2.20, disclosing the internal subnet, the Gateway VIP and a full service inventory, and forming a DNS-rebind primitive that is why the SOHO router strips these answers in the first place. The renderer now stamps external-dns.alpha.kubernetes.io/controller: none on every HTTPRoute it renders for an internal service, derived from the exposure field alone — so a service that omits exposure defaults to internal and is likewise unpublishable. Established against the deployed v0.21.0's own source, where the controller check runs before hostname resolution in both configured sources, then proven live on a throwaway hostname including that adding the annotation to an existing record triggers the DELETE. Deletes exactly 12 records: the A and TXT pair for jackett, pihole, qbittorrent, radarr, semaphore and sonarr. A 39-record recovery baseline is committed alongside. Safe to land before the DHCP change, contrary to my earlier caution: the router returns NOTHING for all six names today, so these records serve nobody on the LAN, while Pi-hole at 192.168.2.23 already answers for anyone pointed at it. The five exposure:public services are untouched. They remain unreachable publicly, which is a separate problem: the legacy edge at 136.51.115.252 returns HTTP 403 for our hostnames and serves a placeholder certificate, so nothing is reverse-proxying them today. |
||
|
jakman85andClaude Opus 5
|
3173254cc4 |
docs: UniFi OS was never the blocker — it is installed and the guest lost its IP
Third confident claim of mine corrected in two days, and this one I took from a stale README rather than the machine. uosserver.service starts on the current boot and the root filesystem holds 6.3 of 40 GB, while enp1s0 is up with no address and does not answer ARP — so the controller runs and nothing can reach it. Records the near-miss: proceeding on my premise would have reinstalled over a working controller, which the new install-uos guard now prevents. Also records that there is no way into the guest at all, which is why the deliverable became console access, and the cloud-init instance-id discovery that explains why an earlier unattended install could never have taken effect. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
7cb209b9a0 |
merge: UniFi OS is already installed — the guest lost its IP; deliver console access
The premise I briefed was wrong and the agent checked before building on it. UniFi OS Server IS installed: the current boot starts uosserver.service and uosserver-updater.service, and the root filesystem holds 6.3 GB of 40 GB where bare Ubuntu plus our packages is about 2 GB. Verified independently. The actual fault is that the guest has no address — .status.interfaces shows enp1s0 link up with no ipAddress, and ip neigh inside virt-launcher reports 10.0.2.2 FAILED, so the guest does not answer ARP on its masquerade address. Probes from virt-launcher straight to the pod IP and to 10.0.2.2 RST on 11443/8443/8080/22, which exonerates the VIP, the traffic policy and the network policy. The cause cannot be determined from outside the guest. That made one guard non-negotiable: install-uos now bails the moment systemd knows a uosserver unit. Without it this merge would have reinstalled over a working controller. Second finding: there is no way into the guest at all — no SSH key, no password, and SSH cannot work by construction because masquerade forwards only declared ports and 22 is not one. So the deliverable is console access: a SOPS secret carrying installer-url AND console-password, mounted as a NoCloud disk and re-read every boot. The discovery that makes it work: NoCloud instance-id is spec.template.spec.domain.firmware.uuid, stamped on the VM object. Cloud-init compares it to the DMI product UUID, so editing userData alone changes nothing on a second boot — the cached datasource is restored and write_files/runcmd are skipped. That is why a previously shipped unattended install could never take effect. The UUID is now pinned in git and bumped once. Deliberately did NOT put the whole cloud-config behind secretRef: that would hide the shell in ciphertext, out of reach of the L0 dollar-collapse guard, while Flux's envsubst still runs over the decrypted resource — the nextcloud-db shape from 2026-09-13. Script in plaintext, operator values in SOPS. Unattended install is reported honestly as third-party practice, not verified here, and the README's stale dev-era addressing is corrected throughout since it doubles as the rebuild runbook. |
||
|
jakman85andClaude Opus 5
|
12d4e0ae40 |
feat(unifi): SOPS-delivered UOS installer URL + per-boot install unit; correct the README
The operator's UOS bring-up becomes one field. apps/vms/unifi/unifi-uos-installer.sops.yaml
(SOPS/age, with a .example sibling per the crafty/headlamp convention) carries `installer-url`
and `console-password`. KubeVirt mounts it as a read-only iso9660 disk labelled UOSDATA and an
in-guest `uos-install.service` reads it on EVERY boot, so a later URL change needs only a commit
and a VM restart.
Mechanism verified against the DEPLOYED KubeVirt v1.9.0, not from memory:
* inline userData cap is exactly 2048 bytes — server-side dry-run apply accepted 2048 and
rejected 2049; the payload here is 1454.
* `cloudInitNoCloud.secretRef` and `secret.volumeLabel` both exist in the live CRD; the secret
disk + third disk were accepted by a server-side dry-run of the real manifest.
* the NoCloud seed's instance-id IS spec.template.spec.domain.firmware.uuid (read out of
noCloud.iso inside virt-launcher), stamped on the VM OBJECT, so it survives restarts. Editing
userData alone therefore changes NOTHING — cloud-init restores its cached datasource and skips
write_files/runcmd. The UUID is now pinned in git and bumped once so the new payload lands.
secretRef was deliberately NOT used for the whole cloud-config: it would put the shell in
ciphertext, out of reach of the L0 dollar-collapse guard, while Flux's envsubst still runs over
decrypted resources — the 2026-09-13 nextcloud-db shape. Script in plaintext, operator values in
SOPS. apps-vms had no `decryption` stanza (first encrypted content under apps/vms/), so one was
added to both overlays.
CORRECTION, and it is the headline: UNIFI OS SERVER IS ALREADY INSTALLED on the live guest. The
serial console log starts uosserver.service and uosserver-updater.service, and / is 6.3 GB of
40 GB used. The controller is unreachable because THE GUEST HAS NO IP: .status.interfaces shows
enp1s0 up with no address, and `ip neigh` in virt-launcher reports 10.0.2.2 FAILED. 11443/8443/
8080 and 22 all RST, from the pod IP and from 10.0.2.2 directly, so the VIP and the CNP are
exonerated. install-uos therefore bails out the moment systemd knows a uosserver.service —
without that guard this change would have reinstalled over a working controller.
There was also no way into the guest at all: no key, no password, and 22 is not a declared
masquerade port so SSH cannot work by construction. Hence console-password.
README corrections (it doubles as the rebuild runbook and was misdirecting):
* VIP 192.168.1.54 -> 192.168.2.21; pool 192.168.1.53-.60/.53 -> 192.168.2.20-.30 with the
Gateway on .20; device subnet 192.168.1.0/24 -> 192.168.2.0/24; node k3s-dev-1 -> k3s-prod-1.
All re-read live (pool blocks, cluster-vars, the Service EXTERNAL-IPs, LAN_CIDR).
* "UOS is NOT installed" -> it is; the real fault is the lost IP, with the evidence table.
* "apps-vms must NOT use postBuild.substituteFrom" -> it has since 2026-08-31 and the escape
contract is in force.
* "install-uos is no longer baked into the guest" -> it was baked in, and it was broken twice
over (an unverified firmware API, and no `install` subcommand).
* "SSH in, cloud-init installs the operator's key" -> it installs none, and 22 is not forwarded.
* externalTrafficPolicy Local -> Cluster (the 8-day outage fix already in 50-service-l4.yaml).
* 60 GB -> 40Gi (the DataVolume and the Bound PVC); "10 GB minimum" -> ~25 GB.
* userDataSecretRef -> the YAML key is `secretRef`.
* 90-httproute-optional.yaml referenced as if it existed; it was dropped.
Unattended install, honestly: Ubiquiti documents no silent flag, no answer file, no env vars.
`yes y | <installer> install` is the convergent community practice and is what ships here, with a
bare-invocation fallback. It has NOT been run end-to-end on our guest. The installer URL also
does not look signed or time-limited — it is account-derived and per-release; it is in SOPS
anyway, but nothing should depend on expiry.
The running VM was not touched: no restart, no migration, no re-import. Every cluster call was a
read or an explicit --dry-run=server; the live VM still carries firmware uuid ...c07 and the
34h-old virt-launcher pod is untouched. The change takes effect on the operator's next VM restart.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
989781b352 |
docs: Minecraft server live, EULA accepted on the operator's behalf, world backed up
Records the EULA acceptance prominently with revocation instructions, the Server List Ping proof, and Crafty's autostart trap where the create API accepts 'autostart' but stores 'auto_start'. Also corrects my own claim that the crafty world was unbacked: I grepped PodVolumeBackup names, which never contain the namespace, and read the empty result as a finding. Crafty had 2 PVBs in the last run. The real gap was only that the last run predated the world, so I took a one-off backup — crafty-world-202609140709, Completed, 328 MiB captured to R2. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
97257515e8 |
merge: a Minecraft server exists and 192.168.2.22:25565 is playable
Crafty was healthy but had never had a server created — /crafty/servers was empty, which is why the port refused. Paper 26.2 (Mojang's latest.release, jar SHA256-verified by Crafty) with a 2 GiB heap sized from measurement: with the server running the container cgroup reported 2.86Gi against a 4Gi limit, so a 3 GiB heap would have risked OOMKill. EULA ACCEPTED ON THE OPERATOR'S BEHALF — https://aka.ms/MinecraftEULA — via Crafty's agree_to_eula, writing eula=true. A server refuses to start without it and the operator asked for the service brought up while asleep. No account, key or payment involved. To revoke: stop the server and set eula=false, or delete the server. Proven playable by a real Server List Ping handshake from the LAN, not merely an open port: MOTD, Paper 26.2, protocol 776, 0/20 players, 12.3 ms RTT. Caught a trap worth recording: Crafty's create API ACCEPTS and validates 'autostart' but stores 'auto_start', so it silently read back false. Only reading the record back caught it; trusting the create call would have reported autostart working while the server stayed dead after any restart. Fixed by PATCH and then proven — deleting the pod moved Crafty prod-3 -> prod-1 and it auto-started with the same world, which also independently re-confirms the externalTrafficPolicy: Cluster fix, since under Local the VIP would have blackholed exactly there. Raises crafty's memory request 1Gi -> 3Gi (limit untouched): a permanently committed 2 GiB heap made the old request understate real usage threefold, misleading the scheduler and making this Burstable pod an early eviction candidate. |
||
|
jakman85andClaude Opus 5
|
b4bc8e1cb9 |
feat(crafty): first Minecraft server — Paper 26.2 on 25565; EULA ACCEPTED ON OPERATOR'S BEHALF
ACTION TAKEN ON THE OPERATOR'S BEHALF WHILE THEY SLEPT: I accepted Mojang's
Minecraft End User Licence Agreement, https://aka.ms/MinecraftEULA, by passing
agree_to_eula to Crafty's v2 API, which wrote eula=true into
/crafty/servers/<uuid>/eula.txt. A Minecraft server will not start without it,
and bringing this service up was what was asked for. Please review. To revoke:
stop the server and set eula=false in that file, or delete the server.
Crafty was healthy but had never had a server created, so 192.168.2.22:25565
refused connections. Created "homelab" via the API (not UI automation):
Paper 26.2, TCP 25565, heap -Xms2000M -Xmx2000M, autostart on
uuid 13c7b85d-00bb-4a58-8c86-2433ca47891c
Paper over Vanilla for tick performance and plugins, with no cost to players
(vanilla clients connect unchanged). 26.2 is Mojang's latest.release, out
2026-06-16; the 26.3-rc-2 snapshot and 26.2-rc-2 were deliberately avoided.
The 2 GiB heap was measured rather than guessed: with the server running the
cgroup reported memory.current 2.86Gi against the 4Gi limit, so a 3 GiB heap
would have risked an OOMKill.
Proven playable, all real output: Crafty reports running with 'Done (12.737s)!';
Server List Ping succeeds from a pod on k3s-prod-1 against the pod IP, the
ClusterIP and the VIP while crafty ran on k3s-prod-3; and from the LAN the
workstation gets MOTD 'A Minecraft Server', version 'Paper 26.2', protocol 776,
players 0/20 over a full handshake plus ping/pong. Deleting the pod moved Crafty
to another node; it auto-started with no manual action and the LAN ping still
succeeded, re-confirming the externalTrafficPolicy: Cluster fix from
|
||
|
jakman85andClaude Opus 5
|
83f13f4b0a |
docs: correct my 'live outage' claim — unifi and crafty were never up
I reported the UniFi controller and Minecraft server as unreachable for up to 8 days. Probing the backend pods directly, bypassing VIP and policy, returns REFUSED on all four ports while crafty's own UI and pihole's DNS answer as controls. /crafty/servers is empty and the unifi README still lists installing UOS in the guest as the open blocker. Nothing regressed. The externalTrafficPolicy defect is real and is fixed, but it was not the cause. I observed unreachable ports, found a mechanism that produces unreachable ports, and concluded it was the cause without running the one control probe that separates them — the same error shape as nearly blaming the drive for the I/O pressure spike when the local disks were idle. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
adda6dd779 |
merge: L2-announced VIPs must serve from any node, not only the backend's
Cilium's L2 announcer elects a lease holder without regard to which nodes hold a backend, so externalTrafficPolicy: Local leaves the frontend absent from the announcing node's service table entirely and the VIP answers ICMP unreachable. Proven causally with cilium-dbg service list: under Local the frontend is missing from prod-2; flipped to Cluster it appears as an active backend entry; flipped back it vanishes. Durability proven too — cordon prod-1, crafty reschedules to prod-3, and prod-1's UDP lease re-points to the new backend with no intervention, which under Local is exactly the failure mode. IMPORTANT CORRECTION to my own framing: this is NOT why unifi and crafty do not answer, and fixing it does not make them answer. There is no application behind any of those ports. /crafty/servers is empty — no Minecraft server has ever been created — and apps/vms/unifi/README.md step 5 still reads 'NEXT: install UOS in the guest', with the unattended install flagged as the open blocker. Probing the pods directly, bypassing VIP and policy, returns REFUSED on all four ports while crafty's own UI and pihole's DNS answer as controls. Nothing regressed; the 8-day-old leases are just 8-day-old Services. I called it a live outage and it was not. What this commit does is remove a landmine that would have blackholed both VIPs the moment those applications exist. Client-IP cost assessed per service rather than assumed: zero for UniFi, whose KubeVirt masquerade binding already NATs everything into the guest, so Local was protecting an address the controller could never observe; real but acceptable for Crafty, where player identity is a Mojang UUID so bans and whitelists are unaffected, with /ban-ip documented as do-not-use. loadBalancerSourceRanges deliberately NOT copied from the Pi-hole pattern: those Services never had a LAN restriction, and adding one during a fix could cut off off-LAN players. Cilium DSR mode is recommended separately, not bundled — it carries an Ansible parity contract and rolls the CNI across all three nodes. |
||
|
jakman85andClaude Opus 5
|
4664b41db3 |
fix(l4): externalTrafficPolicy Cluster on unifi + crafty L4 VIPs
Cilium's L2 announcer elects one node per Service to answer ARP for its LoadBalancer VIP, and that election is NOT backend-aware. Combined with externalTrafficPolicy: Local -- which lets a node deliver only to a backend on itself -- an elected announcer holding no backend answers ARP for the address and then has nowhere to send the packet, so the client gets ICMP port unreachable. It never self-heals: the lease holder keeps renewing after the backend moves away. Measured on prod 2026-09-14: unifi's VM ran on k3s-prod-1 with both leases on k3s-prod-2, and crafty's pod ran on k3s-prod-1 with the -tcp lease on k3s-prod-2 while its identical -udp twin happened to draw k3s-prod-1 and worked. Confirmed causally on the announcing node with `cilium-dbg service list`: under Local the frontend 192.168.2.22:25565 is absent from that node's service table entirely; under Cluster it appears as `1 => 10.0.0.11:25565 (active)`; flipped back, it vanishes again. Durability confirmed by cordoning k3s-prod-1 and moving the crafty pod to k3s-prod-3, after which the announcing node re-pointed to the new backend with no intervention. Both files also carried a comment claiming Local "pins the VIP to the node running the pod", which is the precise thing that is false here; both are rewritten. apps/wave-0/pihole-dns.yaml is the reference implementation. The client-IP cost is stated per service rather than waved away. For unifi it is ZERO: the VM's KubeVirt `masquerade` binding already SNATs everything into the guest, so Local was never preserving a client address the controller could see. For crafty it is real and accepted -- player identity is a Mojang/Xbox UUID so /ban and the whitelist are unaffected, but /ban-ip would hit every player at once and must not be used; no IP ban or per-IP rate limit is configured today. No loadBalancerSourceRanges here, deliberately departing from pihole-dns.yaml: these Services never had a source restriction (their CNPs admit `world`), so Cluster removes no protection, and adding a LAN-only filter during an outage fix could cut off off-LAN players or a remote site's device informing. No CiliumNetworkPolicy change is needed, and that was verified rather than assumed: Cilium's `cluster` entity subsumes `remote-node`, confirmed by dumping the crafty endpoint's datapath policy map with the CNP unmodified (`reserved:remote-node` was already an Allow on the game ports). Unlike pihole, whose policy omits `cluster` and must name remote-node explicitly. Adds an L0 whole-repo guard for the class. It fails on the pre-fix tree with exactly these four Services named and passes after. whole-repo scope is necessary: ci/touched-requirements.sh has no arm for apps/* or platform/crafty/*, so the change most likely to reintroduce this would attribute no requirement and be gated by nothing. HONEST LIMIT -- this does NOT end the reported outage. All four ports also have no listening backend: UniFi OS was never installed in the VM (its README flags this as the open blocker) and /crafty/servers is empty, so no Minecraft server exists. Probed directly from the node, bypassing the VIP entirely, 10.0.0.74:8443/:8080/:11443 and 10.0.0.11:25565 all refuse, while the two controls answer. This commit fixes a real latent defect that would have blackholed those VIPs once the applications exist; it cannot make an absent application answer. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
4545611f4d |
merge: Pi-hole becomes the internal LAN DNS authority
The spec and README both claim a LAN split-horizon via Pi-hole; it never existed. Pi-hole's Service was ClusterIP with ports 80 and 4180 only, so LAN clients depended on hand-maintained per-client dnsmasq files — one of which went stale after the dev decommission and cost access to sonarr. Now DNS on udp/53 + tcp/53 on a pinned pool VIP 192.168.2.23, with the zone answered as a wildcard to the Gateway VIP plus longest-match overrides for the two hostname-publishing L4 Services. Declared via Pi-hole v6 FTLCONF env vars, which FTL treats as env-forced and applies at cold start — a real declarative entry point, unlike the Keycloak secret and CNPG grant whose reconcilers exist precisely because no such knob is available. Two defects found on the way. The live pod carried hand-set upstreams marked '### CHANGED' that no manifest knew about, so any restart would have left Pi-hole with upstreams = [] and no recursion at all; FTLCONF_dns_upstreams now declares them. And /etc/pihole is not on the config PVC, which mounts at /data/config, so that volume has been holding nothing. Deviates from the unifi/crafty pattern deliberately: externalTrafficPolicy Cluster, not Local. Cilium's L2 announcer does not filter by which nodes hold a backend, so with Local an announcing node without the pod answers ICMP port unreachable — measured 1-in-3 per protocol, and it does NOT self-heal: a cordoned node kept renewing its lease for over three minutes with LAN DNS dead. The LAN restriction moves to loadBalancerSourceRanges, enforced in eBPF on the VIP, and rate limiting is disabled because SNAT makes the whole LAN look like one client. Proven from the workstation in the worst case — pod and both protocol leases on three different nodes: wildcard, both overrides, recursion over udp and tcp, NXDOMAIN outside the zone, NODATA for AAAA, source-range drop when excluded, and survival of a cross-node reschedule with no intervention. |
||
|
jakman85andClaude Opus 5
|
fd43f332d6 |
feat(pihole): authoritative LAN DNS for the zone on udp/53 + tcp/53 at a pinned VIP
platform/coredns/coredns-custom.yaml has always said its split-horizon is for
PODS and deferred LAN clients to "PiHole split-horizon". PiHole served no DNS:
its Service was ClusterIP with ports 80 and 4180 only. The LAN half of R-DNS-03
that docs/09 and the README's Phase 2 row both claim did not exist, so LAN
clients ran on hand-maintained per-client dnsmasq files - one went stale after
the dev decommission and cost the operator sonarr.home.keithjackson.dev. The
SOHO router that owns 192.168.2.0/24 cannot cover for it either: it strips
RFC1918 answers for public domains as rebind protection and returns NOERROR
with no answer.
EXPOSURE (apps/wave-0/pihole-dns.yaml, hand-written, placed into wave 0 by
render.py's EXTRA_UNITS - the schema has no field for a LoadBalancer Service, a
pinned lbipam VIP, a sharing key, source ranges or UDP, the same reason
apps/vms/unifi and platform/crafty hand-write theirs):
- two Services, udp/53 and tcp/53, sharing one pinned VIP
${PIHOLE_DNS_VIP} = 192.168.2.23 on prod, the next free address in the
.20-.30 pool (pool status IPsUsed 3 of 11; .20 Gateway, .21 unifi, .22
crafty; arping from a LAN host got no reply).
- no external-dns hostname annotation: nothing public is created, and
platform/externaldns/ is untouched.
ANSWERS (catalog/pihole.yaml env - declarative, no reconciler): PiHole v6 takes
every FTL setting as an FTLCONF_ environment variable and treats it as
env-forced, applied ahead of pihole.toml on every cold start and read-only in
the UI. That is a real declarative entry point, which the Keycloak and CNPG
CronJob reconcilers exist precisely because their state lacks, so a reconciler
here would be ceremony. misc.dnsmasq_lines carries a wildcard
address=/<zone>/<GATEWAY_VIP> plus one longer-domain override per L4 service
(unifi, crafty-mc, pihole-dns); dnsmasq resolves overlaps by longest matching
domain, so precedence is structural and a newly onboarded Gateway-fronted
service needs no DNS change. dns.upstreams is declared too - the live pod
carried hand-set upstreams that no manifest knew about and that every restart
discarded.
externalTrafficPolicy Cluster, NOT Local, and loadBalancerSourceRanges for the
LAN restriction. Measured on prod, Cilium 1.20.1: the L2 announcer does not
filter lease candidates by which nodes hold a backend - the two Services landed
on k3s-prod-3 and k3s-prod-2 while the pod ran on k3s-prod-1 - and it keeps
renewing after the pod moves away, so with Local a reschedule leaves LAN DNS
dead indefinitely. Cluster serves the VIP from any node; the LAN restriction
moves to loadBalancerSourceRanges, which Cilium enforces in eBPF on the VIP
before load-balancing (verified by narrowing the range to exclude the test
client and back). The cost is SNAT: forwarded queries reach PiHole from the
ingress node, so PiHole's per-client rate limit is disabled in the catalog or
the whole LAN would count as one client against its 1000-per-60s default.
Proven from a LAN host (192.168.2.132) against 192.168.2.23 with the pod and
both leases on three DIFFERENT nodes: sonarr/grafana/nextcloud/keycloak/apex
-> 192.168.2.20, unifi -> .21, crafty-mc -> .22, pihole-dns -> .23, recursion
(cloudflare.com, github.com, debian.org) intact over both udp and tcp, AAAA
NODATA, NXDOMAIN outside the zone, and still answering after a forced
prod-1 -> prod-2 reschedule.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
4080065aa7 |
feat(dns): exposure: internal can no longer publish a public record
docs/09 R-DNS-03 and schemas/renderer-contract.md both said an `internal` service renders no public DNS record. Nothing enforced either sentence. ExternalDNS's `gateway-httproute` source admitted EVERY HTTPRoute on the shared Gateway, so six `exposure: internal` services — jackett, pihole, qbittorrent, radarr, semaphore, sonarr — published `<name>.home.keithjackson.dev` into the public authoritative Cloudflare zone answering with the Gateway VIP 192.168.2.20. That published the internal subnet, the Gateway VIP and a full service inventory to anyone who could guess a label, and made each of those names a DNS-rebind primitive — which is why the SOHO router strips RFC1918 answers and LAN clients need per-client dnsmasq overrides to reach their own services. THE MECHANISM, established against the DEPLOYED version rather than from memory. The live prod Deployment runs registry.k8s.io/external-dns/external-dns:v0.21.0. In v0.21.0, source/annotations/processors.go IsControllerMismatch() skips a source object whose `external-dns.alpha.kubernetes.io/controller` annotation is present and != its own `dns-controller`, and source/gateway.go applies that check inside Endpoints() BEFORE any hostname is resolved; source/service.go applies the identical check, covering the second configured source. catalog/render.py now emits that annotation on every HTTPRoute it renders for an `internal` service, derived from `exposure` alone — no per-service list — so a service that omits `exposure` inherits the internal default and is unpublishable too. Verified LIVE on prod on a throwaway hostname with no existing record at risk: the annotated route published nothing, and adding the annotation to a route that already had a record produced action=DELETE for both its A record and its `a-` TXT registry entry one ~60s reconcile later, moving nothing else in the zone. WHY PER-RESOURCE AND NOT A CLUSTER-WIDE OPT-IN FILTER. `labelFilter` is the stronger default posture and v0.21.0 supports it for both configured sources, but nine publishing objects here are hand-authored platform/apps manifests with no `exposure` field, seven of which hold live records (auth, crafty, crafty-mc, grafana, headlamp, the apex, unifi). An opt-in filter deletes all seven at once, and whether each should be publicly resolvable is an operator decision per name. Per-resource exclusion has exactly the blast radius of the defect; the seven are enumerated in a shrink-only register so the unadjudicated set cannot grow silently, and the filter stays available as hardening afterwards. `policy: sync`, `txtOwnerId` and `domainFilters` are UNTOUCHED. Per-resource exclusion only ever removes an object from this instance's view, so nothing here makes prod capable of deleting a record it does not own. WHAT THIS DELETES, and it is not reversible by a revert alone: 12 records — the A record and the `a-`-prefixed TXT registry entry for each of jackett, pihole, qbittorrent, radarr, semaphore and sonarr. The zone has no `*.home.keithjackson.dev` wildcard and `*.keithjackson.dev` does not match two labels, so each becomes a public NXDOMAIN with no fallback. The complete pre-state is captured at the authority and committed as a recovery baseline (.standup-dns/dns-20260914-pre-externaldns-scope.txt), with the restore procedure for both halves of a record. MERGE ORDER IS NOT OPTIONAL: feat/pihole-lan-dns-authority must land and be verified first. Removing these records while the LAN still depends on them, with the router stripping RFC1918 and no internal authority yet, breaks name resolution outright. The five `exposure: public` services are LEFT UNCHANGED and still answer with the Gateway VIP, so they remain publicly unreachable. That is two open operator decisions, both written up in platform/externaldns/helmrelease.yaml: what address they should answer with (136.51.115.252 is the legacy non-k3s stack serving a placeholder cert, not established as correct for this cluster), and how to separate them from the non-public names that share the Gateway — because in v0.21.0 the target annotation is read only from the GATEWAY and a ListenerSet, never from an HTTPRoute, so a per-service override is not expressible while they all attach to `shared`. No knob is added for it yet: one that cannot take effect invites someone to set it and expect one. Guard: conformance/tests/l0-static/internal-services-publish-no-public-dns, a whole-repo-scoped L0 static check (no ci/l0.sh stage — the whole-repo-invariants-wired meta-test forbids one). It asserts both directions (present on internal, absent on public), the `service`-source half, that the renderer DERIVES the annotation from `exposure` and cannot emit external-dns's own claiming value, proves the shipped renderer end-to-end on internal/public/omitted-field fixtures, and ratchets the non-catalog register. Proven to fail on the pre-fix tree (9 of 53 assertions, naming all six services) and pass after (56/56). Schema 1.6.0 -> 1.7.0: no field changes, a mapping change to `exposure`. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
ce7819a7da |
docs: FINDING — public DNS publishes internal RFC1918 addresses, contradicting R-DNS-03
Operator-reported, verified. external-dns runs provider cloudflare with gateway-httproute + service sources and publishes the Cilium Gateway VIP 192.168.2.20, so ~16 hostnames in a public authoritative zone answer with RFC1918 space. The spec says the opposite: external resolves to the public IP, Pi-hole serves the internal split-horizon answers, internal services are unreachable from the internet. Provenance stated plainly: the pattern predates me — external-dns always published the Gateway address, and dev did the same under owner=default. My apex migration changed which names carry it, propagating an existing flaw onto the apex rather than inventing it. Four consequences recorded: topology and service-inventory disclosure; a DNS-rebind primitive, which is why the SOHO router strips these answers and LAN clients need per-client overrides; the five services declaring exposure public are not actually publicly reachable; and seven internal services are publicly advertised. Also records why deleting the records is not a safe first step: the *.home.keithjackson.dev wildcard is gone, so removal yields NXDOMAIN rather than a fallback. Remediation sequenced, nothing started. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
3de79f81a5 |
docs: correct three mountstats figures I supplied from bad measurements
Handed an agent a baseline of 4.0-4.8 ms/op response, 0 ms/op queue, 0 major timeouts over 24h. Re-measurement found queue time is 0.013-0.067 ms/op (I read rounded output as zero); the 24h claim was really ten minutes, because the DaemonSet had just rolled and the range query returned 18 samples rather than ~2,880; and the baseline was idle, so 4.4 ms is a cached STATFS rather than real media I/O and thresholds could not be multiples of it. The 24h error is the same class as the label-values mistake recorded above: trusting a query whose window was never valid. Records two real discoveries: mounts sharing an NFS superblock share kernel RPC statistics, so prod-1's two plex mounts are one signal reported twice; and maximum_rpc_slots high-water is 586 on prod-1 against 2 on prod-2, an independent indication that prod-1 carries far more concurrent NFS work. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
5c96f4f3a7 |
merge: NFS per-export latency — separating a slow server from a backed-up client
Closes the gap the previous agent correctly left open: it could not write a rule against node_mountstats_* because the series did not exist until the collector rolled out, and the L0 guard rightly rejects a rule on an unscraped metric. 1,842 series are now live, so five rules land. The discriminator is the point. response_time is how long the SERVER held the request; queue_time is how long it waited on our node for an RPC slot. Both server rules carry 'unless queue_time/op > 5ms', so a queueing client can never be reported as a slow server, and the two server rules then partition the same slow-server state by our own op rate at 20/s — exactly one fires, no gap and no overlap. Plus major timeouts as critical, since that is hanging rather than slow. Corrects three figures I had supplied from bad measurements: client queue time is 0.013-0.067 ms/op, not zero; 'zero major timeouts over 24h' was really zero over about ten minutes, because the DaemonSet had just rolled and the range query returned 18 samples rather than ~2,880; and the baseline was an IDLE one whose 4.4 ms is a cached STATFS, so thresholds are argued from the link and the array rather than as a multiple of it. Also found that mounts sharing an NFS superblock share the kernel's RPC statistics — prod-1's plex and plex/downloads report identical rates to every digit — so one slow export raises two alerts. Documented rather than aggregated away, since aggregating would hide which path is affected. 14 mutations, all killed. Runbook triage table 10 rows -> 15, with 'not collected' stated plainly for both past incidents because the collector postdates them and that cannot be backfilled. |
||
|
jakman85andClaude Opus 5
|
79d9ab2c8c |
feat(monitoring): per-export NFS latency alerts — server-slow vs we-are-backed-up
Closes the gap `feat/io-pressure-diagnostic-context` deliberately left open. That
change enabled node-exporter's `--collector.mountstats` and recorded its own honest
limit: "an NFS export that is merely SLOW cannot be distinguished today", because
node_mountstats_* did not exist on prod until the HelmRelease rolled and the L0 guard
alert-metrics-have-scrape-targets rightly refuses a rule on an unscraped metric.
The rollout has landed. Verified on prod 2026-09-15: 1,842 node_mountstats_* series
(~0.4% of a 451,650-series head), six mounts, 426 per-operation series in each of the
five operation families and 6 in each transport family. Every metric referenced below
returns series under `promtool query instant`.
FIVE RULES in platform-health.node-io-cause, each naming its cause in `summary`:
NFSExportServerSlow response_time > 50 ms/op, our op rate > 20/s,
queue_time < 5 ms/op -> the server is slow BECAUSE OF US
NFSExportServerSlowNotOurLoad same, but our op rate 1-20/s -> the server is slow and
the load is NOT OURS. This is the operator's Proxmox
hypothesis made testable.
NFSClientQueueTimeHigh queue_time > 5 ms/op -> WE are saturating the transport
NFSTransportBacklogDeep backlog depth > 8 per send -> even the grown slot table is full
NFSExportMajorTimeouts any major timeout in 15m -> the export is HANGING (critical)
THE DISCRIMINATOR. mountstats splits a slow NFS op into the half the server owns
(response_time) and the half we own (queue_time). High response with low queue means the
SERVER; high queue means US. The two server rules then partition that state by our own op
rate at 20/s, so exactly one of them can fire and it names which reading applies. The
`unless queue_time > 5ms` clause is what licenses reading response_time as a statement
about 192.168.2.102 rather than about us, and it is pinned by fixture 5 (a client queueing
30 ms/op while the server appears to take 60 ms/op: both server rules must stay quiet).
MEASURED BASELINE, prod 2026-09-15, re-measured for this change:
server response time 4.40-4.81 ms/op (30m window), uniform across all six mounts
client queue time 0.013-0.067 ms/op; snapshot maximum 0.333 ms/op
our own op rate 0.019-0.163 ops/s — STATFS keepalive and nothing else
major timeouts 0 on every mount
backlog depth 0 on every mount
slot-table high-water 586 (prod-1), 162 (prod-3), 2 (prod-2)
THREE LIMITS ON THAT BASELINE, recorded in the rules and the runbook rather than papered
over:
(a) the history is MINUTES, not days — the series begin when the DaemonSet rolled
(count_over_time(...[24h]) returned 18, and time() - process_start_time_seconds read
558-561 s on all three nodes), so no rule uses a range longer than 15m;
(b) it is an IDLE baseline — STATFS keepalive only, no READ/WRITE traffic, so the 50 ms
threshold is argued from the link and the array (one outstanding 1 MB read at 50 ms is
20 MB/s) and must be re-cut after the first real media event. The runbook carries the
query;
(c) prod-1's /mnt/Storage/plex and /mnt/Storage/plex/downloads report byte-identical
per-operation counters (0.12443893333333335 on both, to every digit), as do prod-3's two
nextcloud mounts — mounts sharing one NFS superblock share the kernel's RPC statistics.
One slow export on prod-1 therefore raises two alerts. Documented, not aggregated away.
PROOF, not assertion. platform/monitoring/tests/nfs-export-latency.test.yaml: seven fixtures
(measured healthy; healthy under 60 ops/s of real load; server-slow-ours; server-slow-not-ours;
client-backed-up; backlog-deep; export-failing), and every fixture asserts all five alertnames
in the fires or the quiet direction. The measured baseline is pinned as a band, the
shared-superblock duplication is pinned, and both hazards the sibling recorded are asserted:
the PromQL precedence trap (`(ratio) and rate(ops) > 1 > 0.050` returns a series on a healthy
60 ops/s mount; the shipped form returns none) and the op-rate guards (a mount doing half an
operation a second at 1 s/op response, 1 s/op queue and 20 deep in the backlog — 20x, 200x and
2.5x the three thresholds — must stay silent).
MUTATION-CHECKED, 14 mutations, every one of them making the test file FAIL: each of the five
thresholds moved in both directions where the fixtures allow it, both `unless` clauses deleted,
and all four rate guards loosened. The two clause deletions fail with an exp/got alert diff,
not a parse error. No threshold here is pinned only by a promql_expr_test restating it.
Live on prod, all five shipped expressions return NO series while all four of their
sub-expressions return six each — so the silence is a verdict, not missing data.
runbooks/node-io.md: the triage table grows from 10 rows to 15 (per-export server response
time, client queue time, our own demand, major timeouts, backlog depth), with real measured
HEALTHY values and an explicit "not collected" in both incident columns, because the collector
postdates both events and that can never be backfilled. A new four-way verdict table reads rows
11-13 together and maps each combination to the alert that says it.
HONEST REMAINING GAP: these rules rule US out; they cannot rule Proxmox IN. The per-client view
of an export exists only on the server. What would settle it is one read on 192.168.2.102 while
NFSExportServerSlowNotOurLoad is firing — `nfsstat -s` and /proc/fs/nfsd/clients/*/info — which
is out of scope from this repo. Recorded in the runbook.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
de861b8672 |
docs: correct my own claim — node_nfs_* metrics DO exist; mountstats is the gap
I recorded that NFS was 'entirely unmonitored' with 'zero NFS observability',
based on a __name__ label-values query that came back empty. The query was at
fault: count({__name__=~"node_nfs.*"}) returns six live families on all three
nodes, and the nfs collector is on by default.
The real gap is narrower — node_mountstats_* is absent, and that is the one
carrying per-mount RPC round-trip time, so NFS latency specifically is
invisible while volume and retransmissions are not.
Two further signals already existed and had simply never been queried:
container_pressure_io_stalled_seconds_total, which names the stalling workload
and for tonight names sonarr; and etcd_request_duration_seconds via the
apiserver job, which the node-io runbook had explicitly claimed did not exist.
Also tempers the Proxmox hypothesis I floated: prod-1 was doing 114 NFS reads/s
of its own during the spike, so our traffic is substantial and the stall is not
obviously unexplained by us.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85
|
207c939f89 |
merge: I/O pressure alerts now name the cause, not just the stall
NodeIOPressureCritical fired on prod-1 and said nothing about why; establishing 'not a failing drive' took eight ad-hoc queries after the fact. Eight new rules each name a class in the summary — device saturation, device latency, NFS transport, etcd, not-local-disk, and which container is stalled — with the companion PromQL in a diagnostic_query annotation, a WHY row on the existing dashboard, and a 10-row triage table in runbooks/node-io.md carrying real HEALTHY / 2026-09-14 / 2026-09-07 columns. The discriminator is NodeIOPressureNotLocalDisk, written with 'unless' rather than 'and ... < T' on purpose: an empty right-hand side removes nothing, so it still fires when devices are too idle to have a measurable per-op latency — 'and <' would go silent in exactly the state it exists to detect. Acceptance is checked against both real profiles. Tonight: NotLocalDisk fires and ContainerIOStallHigh names sonarr. 2026-09-07 re-queried live at offset 164h: 1,039 ms per write on prod-1 sda, so the latency rules fire and name the device while NotLocalDisk correctly stays silent. Healthy state: all quiet. 10 mutations applied, 10 killed — one initially survived and exposed a threshold pinned only by a test that restated it. Corrects two claims: the runbook said no etcd fsync metric exists, but etcd_request_duration_seconds is present via the apiserver job and attributes per node; and container_pressure_io_stalled_seconds_total was already scraped with ~400 series and unqueried. Also fixes two bugs found in the work itself: a PromQL precedence error where 'ratio and rate(ops) > 10 > T' compares T against the op rate rather than the ratio, and node-exporter extraArgs replacing rather than appending, which would have silently dropped the chart's filesystem excludes. |
||
|
jakman85andClaude Opus 5
|
aae63c97a3 |
fix(monitoring): node-exporter extraArgs REPLACES the chart default, so restate it
Verified against the chart (helm show values prometheus-community/prometheus-node-exporter) and against the live DaemonSet's argv on prod: the subchart's extraArgs default is NOT empty — it ships the two filesystem excludes (mount-points-exclude for /dev,/proc,/sys,containerd,docker,kubelet and the fs-types-exclude list). Setting extraArgs to just --collector.mountstats would have silently dropped both, so the filesystem collector would have begun reporting every container overlay and every kubelet bind mount: a large cardinality jump, and a behaviour change in NodeDiskFillingUp, which alerts per mountpoint. Both defaults are now restated verbatim alongside the new collector, with a note to re-reconcile them on a chart upgrade. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
3a72b07cda |
chore(conformance): shrink UNTESTED_ALERTS by the two PSI detectors
node-io-cause.test.yaml's healthy fixture asserts NodeIOPressureHigh and NodeIOPressureCritical quiet on the same measured prod state the eight new cause rules are pinned against — the point of that fixture being that the detectors and the cause rules must agree about what "healthy" is. The shrink-only register entries are therefore stale, and the gate correctly refused them. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
b9f3143f42 |
feat(monitoring): name the CAUSE of node I/O pressure, not just the stall
NodeIOPressureCritical has now fired twice on k3s-prod-1 for opposite causes and
said the same thing both times. On 2026-09-14 02:40-02:45Z it fired at PSI(full)
20.98% — twice the threshold, four times the 2026-09-07 level — while every local
device was idle (0.09-0.27 utilisation) and fast (1.4-4.8 ms per write), NFS
retransmissions were zero and no Longhorn volume was degraded. Establishing that
took eight ad-hoc PromQL queries after the event had already recovered.
Adds the group platform-health.node-io-cause: eight rules, one per cause class,
each naming its cause in its own summary.
NodeDiskDeviceSaturated device saturation >0.85 busy, 10m
NodeDiskWriteLatencyHigh device latency >20ms/write, 10m
NodeDiskWriteLatencyCritical device latency >50ms/write, 5m
NodeDiskReadLatencyHigh device latency >250ms/read, 15m
NodeNFSRetransmissionRateHigh NFS transport >1% retransmitted, 10m
NodeIOPressureNotLocalDisk not local disk PSI>10% with all of the above absent
NodeEtcdRequestsSlow etcd >0.5% of etcd requests over 1s
ContainerIOStallHigh which workload per-container PSI >15%
Every threshold is measured on prod over 24-48h, not guessed, and each rule
records its headroom. The PSI thresholds themselves are deliberately unchanged:
the 21% stall was real, it fired twice in 24h for two real events on the one node
that has a problem, and raising the bar would also have raised it above the 11.36%
at which prod-1 was already losing etcd on 2026-09-07.
Context reaches the operator at page time three ways: the summary states the cause
class; every rule carries a diagnostic_query annotation with the exact companion
PromQL; and node-pressure-dashboard.yaml gains a WHY row (seven panels) plotting
saturation, write/read service time, NFS retransmissions, our own NFS op rate,
the etcd slow-request fraction and per-container PSI side by side. runbooks/node-io.md
gains a 10-row triage table with the real healthy / 2026-09-14 / 2026-09-07 columns,
which turns those eight queries into a two-minute read.
etcd and NFS metric availability, verified on prod rather than assumed:
* etcd SERVER disk metrics do NOT exist. etcd_disk_wal_fsync_duration_seconds and
etcd_disk_backend_commit_duration_seconds are absent (kubeEtcd disabled for k3s);
a query for etcd_disk_*/etcd_server_*/etcd_mvcc_* returns nothing at all.
NodeEtcdRequestsSlow uses the apiserver's client-side view of its own local
member instead, and says in its own description that it is a proxy.
* node_nfs_* DOES exist — the nfs collector is on by default, host-wide via
/proc/net/rpc/nfs, confirmed on all three nodes. It gives op counts and
retransmissions and NO latency, so it can see a dropping export but not a slow
one. The mountstats collector is enabled here to close that gap; NO rule
references node_mountstats_* yet, because the series do not exist until the
HelmRelease rolls out and the collector's visibility through the node-exporter
mount namespace must be verified on prod first. The runbook records the check
and says to revert if it comes back empty.
Proof, per the repo's two shipped alerting failure modes:
* platform/monitoring/tests/node-io-cause.test.yaml — 7 fixtures, all from
measured profiles: healthy (all 8 quiet), 2026-09-14 (NotLocalDisk fires and
names sonarr), 2026-09-07 re-queried at offset 164h (write-latency and etcd
fire, NotLocalDisk correctly silent), rebuild storm, NFS transport, read
latency, and the `unless` semantics.
* 10 mutations of the shipped rules, all killed. The mutation run found a real
gap: the etcd threshold could be changed without failing any test, because it
was only pinned by a promql_expr_test that restated the threshold in the test
file. An alert_rule_test now pins it.
* All 11 referenced metrics verified to return series on prod with
`promtool query instant`, and all 8 expressions verified quiet on today's state.
Also extends the alert-metrics-have-scrape-targets owner table for container_*
(the kubelet ServiceMonitor's cAdvisor path, which this entry already claimed in
its title) and for etcd_request_duration_seconds_* by EXACT NAME, deliberately not
by an etcd_ prefix, so a future rule on the unscraped etcd_disk_* names still fails
the gate.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
7369495b67 |
docs(pm): owner decisions — files stay, workloads stay, instrument NFS instead
Records three owner decisions and the finding that makes the third tractable. The /downloads malware stays: the Proxmox-hosted stack still has operational control of those files so deletion just triggers a re-download, nothing but the arr suite touches the directory, and there are no Windows machines anywhere — which removes the exposure I was concerned about. Written down so nobody 'discovers' and deletes them later. Anti-affinity for the co-located NFS pods declined; the concentration is accepted rather than overlooked. And the enabling finding: there are NO NFS metrics in the TSDB at all — node-exporter's nfs and mountstats collectors are off by default. A cluster with four NFS mounts carrying the media library and legacy Nextcloud data has zero NFS visibility, while PSI counts NFS stalls as I/O stalls. That is precisely why a 20.98% stall alongside idle local disks could not be explained. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
dafa9cd241 |
docs: prod-1 I/O critical was not a failing disk; confirmed malware in /downloads
NodeIOPressureCritical fired on prod-1 (peak 20.98% PSI, recovered
unattended) — the first time this alert has fired on prod, and only because
the scrape gap was closed yesterday.
Evidence rules out the 2026-09-07 drive repeat: 3.5 MB/s of writes, device
utilisation 0.09-0.21, write latency 1.4-4.8 ms, zero NFS retransmissions,
21.4 MB/s NFS throughput, no degraded Longhorn volumes, and KSM never observed
NotReady. A 21% stall on trivial volume with healthy latency is transient
blocking I/O, not hardware — and PSI counts NFS too.
Structural finding: sonarr, qbittorrent and jackett all run on prod-1, the
node with the I/O history. Same concentration class as the CNPG primaries.
Separately, the /downloads files are confirmed malware: two PE executables
('MZ') sharing one MD5, padded to ~926 MB and named as two different shows,
plus a ZIP masquerading as video. Not executed, cluster not compromised; the
risk is a Windows client opening them.
Co-Authored-By: Claude Opus 5 <[email protected]>
|