3173254cc4e2c04f22b32a4804e38e7e272482b9
666
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
3173254cc4 |
docs: UniFi OS was never the blocker — it is installed and the guest lost its IP
Third confident claim of mine corrected in two days, and this one I took from a stale README rather than the machine. uosserver.service starts on the current boot and the root filesystem holds 6.3 of 40 GB, while enp1s0 is up with no address and does not answer ARP — so the controller runs and nothing can reach it. Records the near-miss: proceeding on my premise would have reinstalled over a working controller, which the new install-uos guard now prevents. Also records that there is no way into the guest at all, which is why the deliverable became console access, and the cloud-init instance-id discovery that explains why an earlier unattended install could never have taken effect. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
7cb209b9a0 |
merge: UniFi OS is already installed — the guest lost its IP; deliver console access
The premise I briefed was wrong and the agent checked before building on it. UniFi OS Server IS installed: the current boot starts uosserver.service and uosserver-updater.service, and the root filesystem holds 6.3 GB of 40 GB where bare Ubuntu plus our packages is about 2 GB. Verified independently. The actual fault is that the guest has no address — .status.interfaces shows enp1s0 link up with no ipAddress, and ip neigh inside virt-launcher reports 10.0.2.2 FAILED, so the guest does not answer ARP on its masquerade address. Probes from virt-launcher straight to the pod IP and to 10.0.2.2 RST on 11443/8443/8080/22, which exonerates the VIP, the traffic policy and the network policy. The cause cannot be determined from outside the guest. That made one guard non-negotiable: install-uos now bails the moment systemd knows a uosserver unit. Without it this merge would have reinstalled over a working controller. Second finding: there is no way into the guest at all — no SSH key, no password, and SSH cannot work by construction because masquerade forwards only declared ports and 22 is not one. So the deliverable is console access: a SOPS secret carrying installer-url AND console-password, mounted as a NoCloud disk and re-read every boot. The discovery that makes it work: NoCloud instance-id is spec.template.spec.domain.firmware.uuid, stamped on the VM object. Cloud-init compares it to the DMI product UUID, so editing userData alone changes nothing on a second boot — the cached datasource is restored and write_files/runcmd are skipped. That is why a previously shipped unattended install could never take effect. The UUID is now pinned in git and bumped once. Deliberately did NOT put the whole cloud-config behind secretRef: that would hide the shell in ciphertext, out of reach of the L0 dollar-collapse guard, while Flux's envsubst still runs over the decrypted resource — the nextcloud-db shape from 2026-09-13. Script in plaintext, operator values in SOPS. Unattended install is reported honestly as third-party practice, not verified here, and the README's stale dev-era addressing is corrected throughout since it doubles as the rebuild runbook. |
||
|
jakman85andClaude Opus 5
|
12d4e0ae40 |
feat(unifi): SOPS-delivered UOS installer URL + per-boot install unit; correct the README
The operator's UOS bring-up becomes one field. apps/vms/unifi/unifi-uos-installer.sops.yaml
(SOPS/age, with a .example sibling per the crafty/headlamp convention) carries `installer-url`
and `console-password`. KubeVirt mounts it as a read-only iso9660 disk labelled UOSDATA and an
in-guest `uos-install.service` reads it on EVERY boot, so a later URL change needs only a commit
and a VM restart.
Mechanism verified against the DEPLOYED KubeVirt v1.9.0, not from memory:
* inline userData cap is exactly 2048 bytes — server-side dry-run apply accepted 2048 and
rejected 2049; the payload here is 1454.
* `cloudInitNoCloud.secretRef` and `secret.volumeLabel` both exist in the live CRD; the secret
disk + third disk were accepted by a server-side dry-run of the real manifest.
* the NoCloud seed's instance-id IS spec.template.spec.domain.firmware.uuid (read out of
noCloud.iso inside virt-launcher), stamped on the VM OBJECT, so it survives restarts. Editing
userData alone therefore changes NOTHING — cloud-init restores its cached datasource and skips
write_files/runcmd. The UUID is now pinned in git and bumped once so the new payload lands.
secretRef was deliberately NOT used for the whole cloud-config: it would put the shell in
ciphertext, out of reach of the L0 dollar-collapse guard, while Flux's envsubst still runs over
decrypted resources — the 2026-09-13 nextcloud-db shape. Script in plaintext, operator values in
SOPS. apps-vms had no `decryption` stanza (first encrypted content under apps/vms/), so one was
added to both overlays.
CORRECTION, and it is the headline: UNIFI OS SERVER IS ALREADY INSTALLED on the live guest. The
serial console log starts uosserver.service and uosserver-updater.service, and / is 6.3 GB of
40 GB used. The controller is unreachable because THE GUEST HAS NO IP: .status.interfaces shows
enp1s0 up with no address, and `ip neigh` in virt-launcher reports 10.0.2.2 FAILED. 11443/8443/
8080 and 22 all RST, from the pod IP and from 10.0.2.2 directly, so the VIP and the CNP are
exonerated. install-uos therefore bails out the moment systemd knows a uosserver.service —
without that guard this change would have reinstalled over a working controller.
There was also no way into the guest at all: no key, no password, and 22 is not a declared
masquerade port so SSH cannot work by construction. Hence console-password.
README corrections (it doubles as the rebuild runbook and was misdirecting):
* VIP 192.168.1.54 -> 192.168.2.21; pool 192.168.1.53-.60/.53 -> 192.168.2.20-.30 with the
Gateway on .20; device subnet 192.168.1.0/24 -> 192.168.2.0/24; node k3s-dev-1 -> k3s-prod-1.
All re-read live (pool blocks, cluster-vars, the Service EXTERNAL-IPs, LAN_CIDR).
* "UOS is NOT installed" -> it is; the real fault is the lost IP, with the evidence table.
* "apps-vms must NOT use postBuild.substituteFrom" -> it has since 2026-08-31 and the escape
contract is in force.
* "install-uos is no longer baked into the guest" -> it was baked in, and it was broken twice
over (an unverified firmware API, and no `install` subcommand).
* "SSH in, cloud-init installs the operator's key" -> it installs none, and 22 is not forwarded.
* externalTrafficPolicy Local -> Cluster (the 8-day outage fix already in 50-service-l4.yaml).
* 60 GB -> 40Gi (the DataVolume and the Bound PVC); "10 GB minimum" -> ~25 GB.
* userDataSecretRef -> the YAML key is `secretRef`.
* 90-httproute-optional.yaml referenced as if it existed; it was dropped.
Unattended install, honestly: Ubiquiti documents no silent flag, no answer file, no env vars.
`yes y | <installer> install` is the convergent community practice and is what ships here, with a
bare-invocation fallback. It has NOT been run end-to-end on our guest. The installer URL also
does not look signed or time-limited — it is account-derived and per-release; it is in SOPS
anyway, but nothing should depend on expiry.
The running VM was not touched: no restart, no migration, no re-import. Every cluster call was a
read or an explicit --dry-run=server; the live VM still carries firmware uuid ...c07 and the
34h-old virt-launcher pod is untouched. The change takes effect on the operator's next VM restart.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
989781b352 |
docs: Minecraft server live, EULA accepted on the operator's behalf, world backed up
Records the EULA acceptance prominently with revocation instructions, the Server List Ping proof, and Crafty's autostart trap where the create API accepts 'autostart' but stores 'auto_start'. Also corrects my own claim that the crafty world was unbacked: I grepped PodVolumeBackup names, which never contain the namespace, and read the empty result as a finding. Crafty had 2 PVBs in the last run. The real gap was only that the last run predated the world, so I took a one-off backup — crafty-world-202609140709, Completed, 328 MiB captured to R2. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
97257515e8 |
merge: a Minecraft server exists and 192.168.2.22:25565 is playable
Crafty was healthy but had never had a server created — /crafty/servers was empty, which is why the port refused. Paper 26.2 (Mojang's latest.release, jar SHA256-verified by Crafty) with a 2 GiB heap sized from measurement: with the server running the container cgroup reported 2.86Gi against a 4Gi limit, so a 3 GiB heap would have risked OOMKill. EULA ACCEPTED ON THE OPERATOR'S BEHALF — https://aka.ms/MinecraftEULA — via Crafty's agree_to_eula, writing eula=true. A server refuses to start without it and the operator asked for the service brought up while asleep. No account, key or payment involved. To revoke: stop the server and set eula=false, or delete the server. Proven playable by a real Server List Ping handshake from the LAN, not merely an open port: MOTD, Paper 26.2, protocol 776, 0/20 players, 12.3 ms RTT. Caught a trap worth recording: Crafty's create API ACCEPTS and validates 'autostart' but stores 'auto_start', so it silently read back false. Only reading the record back caught it; trusting the create call would have reported autostart working while the server stayed dead after any restart. Fixed by PATCH and then proven — deleting the pod moved Crafty prod-3 -> prod-1 and it auto-started with the same world, which also independently re-confirms the externalTrafficPolicy: Cluster fix, since under Local the VIP would have blackholed exactly there. Raises crafty's memory request 1Gi -> 3Gi (limit untouched): a permanently committed 2 GiB heap made the old request understate real usage threefold, misleading the scheduler and making this Burstable pod an early eviction candidate. |
||
|
jakman85andClaude Opus 5
|
b4bc8e1cb9 |
feat(crafty): first Minecraft server — Paper 26.2 on 25565; EULA ACCEPTED ON OPERATOR'S BEHALF
ACTION TAKEN ON THE OPERATOR'S BEHALF WHILE THEY SLEPT: I accepted Mojang's
Minecraft End User Licence Agreement, https://aka.ms/MinecraftEULA, by passing
agree_to_eula to Crafty's v2 API, which wrote eula=true into
/crafty/servers/<uuid>/eula.txt. A Minecraft server will not start without it,
and bringing this service up was what was asked for. Please review. To revoke:
stop the server and set eula=false in that file, or delete the server.
Crafty was healthy but had never had a server created, so 192.168.2.22:25565
refused connections. Created "homelab" via the API (not UI automation):
Paper 26.2, TCP 25565, heap -Xms2000M -Xmx2000M, autostart on
uuid 13c7b85d-00bb-4a58-8c86-2433ca47891c
Paper over Vanilla for tick performance and plugins, with no cost to players
(vanilla clients connect unchanged). 26.2 is Mojang's latest.release, out
2026-06-16; the 26.3-rc-2 snapshot and 26.2-rc-2 were deliberately avoided.
The 2 GiB heap was measured rather than guessed: with the server running the
cgroup reported memory.current 2.86Gi against the 4Gi limit, so a 3 GiB heap
would have risked an OOMKill.
Proven playable, all real output: Crafty reports running with 'Done (12.737s)!';
Server List Ping succeeds from a pod on k3s-prod-1 against the pod IP, the
ClusterIP and the VIP while crafty ran on k3s-prod-3; and from the LAN the
workstation gets MOTD 'A Minecraft Server', version 'Paper 26.2', protocol 776,
players 0/20 over a full handshake plus ping/pong. Deleting the pod moved Crafty
to another node; it auto-started with no manual action and the LAN ping still
succeeded, re-confirming the externalTrafficPolicy: Cluster fix from
|
||
|
jakman85andClaude Opus 5
|
83f13f4b0a |
docs: correct my 'live outage' claim — unifi and crafty were never up
I reported the UniFi controller and Minecraft server as unreachable for up to 8 days. Probing the backend pods directly, bypassing VIP and policy, returns REFUSED on all four ports while crafty's own UI and pihole's DNS answer as controls. /crafty/servers is empty and the unifi README still lists installing UOS in the guest as the open blocker. Nothing regressed. The externalTrafficPolicy defect is real and is fixed, but it was not the cause. I observed unreachable ports, found a mechanism that produces unreachable ports, and concluded it was the cause without running the one control probe that separates them — the same error shape as nearly blaming the drive for the I/O pressure spike when the local disks were idle. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
adda6dd779 |
merge: L2-announced VIPs must serve from any node, not only the backend's
Cilium's L2 announcer elects a lease holder without regard to which nodes hold a backend, so externalTrafficPolicy: Local leaves the frontend absent from the announcing node's service table entirely and the VIP answers ICMP unreachable. Proven causally with cilium-dbg service list: under Local the frontend is missing from prod-2; flipped to Cluster it appears as an active backend entry; flipped back it vanishes. Durability proven too — cordon prod-1, crafty reschedules to prod-3, and prod-1's UDP lease re-points to the new backend with no intervention, which under Local is exactly the failure mode. IMPORTANT CORRECTION to my own framing: this is NOT why unifi and crafty do not answer, and fixing it does not make them answer. There is no application behind any of those ports. /crafty/servers is empty — no Minecraft server has ever been created — and apps/vms/unifi/README.md step 5 still reads 'NEXT: install UOS in the guest', with the unattended install flagged as the open blocker. Probing the pods directly, bypassing VIP and policy, returns REFUSED on all four ports while crafty's own UI and pihole's DNS answer as controls. Nothing regressed; the 8-day-old leases are just 8-day-old Services. I called it a live outage and it was not. What this commit does is remove a landmine that would have blackholed both VIPs the moment those applications exist. Client-IP cost assessed per service rather than assumed: zero for UniFi, whose KubeVirt masquerade binding already NATs everything into the guest, so Local was protecting an address the controller could never observe; real but acceptable for Crafty, where player identity is a Mojang UUID so bans and whitelists are unaffected, with /ban-ip documented as do-not-use. loadBalancerSourceRanges deliberately NOT copied from the Pi-hole pattern: those Services never had a LAN restriction, and adding one during a fix could cut off off-LAN players. Cilium DSR mode is recommended separately, not bundled — it carries an Ansible parity contract and rolls the CNI across all three nodes. |
||
|
jakman85andClaude Opus 5
|
4664b41db3 |
fix(l4): externalTrafficPolicy Cluster on unifi + crafty L4 VIPs
Cilium's L2 announcer elects one node per Service to answer ARP for its LoadBalancer VIP, and that election is NOT backend-aware. Combined with externalTrafficPolicy: Local -- which lets a node deliver only to a backend on itself -- an elected announcer holding no backend answers ARP for the address and then has nowhere to send the packet, so the client gets ICMP port unreachable. It never self-heals: the lease holder keeps renewing after the backend moves away. Measured on prod 2026-09-14: unifi's VM ran on k3s-prod-1 with both leases on k3s-prod-2, and crafty's pod ran on k3s-prod-1 with the -tcp lease on k3s-prod-2 while its identical -udp twin happened to draw k3s-prod-1 and worked. Confirmed causally on the announcing node with `cilium-dbg service list`: under Local the frontend 192.168.2.22:25565 is absent from that node's service table entirely; under Cluster it appears as `1 => 10.0.0.11:25565 (active)`; flipped back, it vanishes again. Durability confirmed by cordoning k3s-prod-1 and moving the crafty pod to k3s-prod-3, after which the announcing node re-pointed to the new backend with no intervention. Both files also carried a comment claiming Local "pins the VIP to the node running the pod", which is the precise thing that is false here; both are rewritten. apps/wave-0/pihole-dns.yaml is the reference implementation. The client-IP cost is stated per service rather than waved away. For unifi it is ZERO: the VM's KubeVirt `masquerade` binding already SNATs everything into the guest, so Local was never preserving a client address the controller could see. For crafty it is real and accepted -- player identity is a Mojang/Xbox UUID so /ban and the whitelist are unaffected, but /ban-ip would hit every player at once and must not be used; no IP ban or per-IP rate limit is configured today. No loadBalancerSourceRanges here, deliberately departing from pihole-dns.yaml: these Services never had a source restriction (their CNPs admit `world`), so Cluster removes no protection, and adding a LAN-only filter during an outage fix could cut off off-LAN players or a remote site's device informing. No CiliumNetworkPolicy change is needed, and that was verified rather than assumed: Cilium's `cluster` entity subsumes `remote-node`, confirmed by dumping the crafty endpoint's datapath policy map with the CNP unmodified (`reserved:remote-node` was already an Allow on the game ports). Unlike pihole, whose policy omits `cluster` and must name remote-node explicitly. Adds an L0 whole-repo guard for the class. It fails on the pre-fix tree with exactly these four Services named and passes after. whole-repo scope is necessary: ci/touched-requirements.sh has no arm for apps/* or platform/crafty/*, so the change most likely to reintroduce this would attribute no requirement and be gated by nothing. HONEST LIMIT -- this does NOT end the reported outage. All four ports also have no listening backend: UniFi OS was never installed in the VM (its README flags this as the open blocker) and /crafty/servers is empty, so no Minecraft server exists. Probed directly from the node, bypassing the VIP entirely, 10.0.0.74:8443/:8080/:11443 and 10.0.0.11:25565 all refuse, while the two controls answer. This commit fixes a real latent defect that would have blackholed those VIPs once the applications exist; it cannot make an absent application answer. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
4545611f4d |
merge: Pi-hole becomes the internal LAN DNS authority
The spec and README both claim a LAN split-horizon via Pi-hole; it never existed. Pi-hole's Service was ClusterIP with ports 80 and 4180 only, so LAN clients depended on hand-maintained per-client dnsmasq files — one of which went stale after the dev decommission and cost access to sonarr. Now DNS on udp/53 + tcp/53 on a pinned pool VIP 192.168.2.23, with the zone answered as a wildcard to the Gateway VIP plus longest-match overrides for the two hostname-publishing L4 Services. Declared via Pi-hole v6 FTLCONF env vars, which FTL treats as env-forced and applies at cold start — a real declarative entry point, unlike the Keycloak secret and CNPG grant whose reconcilers exist precisely because no such knob is available. Two defects found on the way. The live pod carried hand-set upstreams marked '### CHANGED' that no manifest knew about, so any restart would have left Pi-hole with upstreams = [] and no recursion at all; FTLCONF_dns_upstreams now declares them. And /etc/pihole is not on the config PVC, which mounts at /data/config, so that volume has been holding nothing. Deviates from the unifi/crafty pattern deliberately: externalTrafficPolicy Cluster, not Local. Cilium's L2 announcer does not filter by which nodes hold a backend, so with Local an announcing node without the pod answers ICMP port unreachable — measured 1-in-3 per protocol, and it does NOT self-heal: a cordoned node kept renewing its lease for over three minutes with LAN DNS dead. The LAN restriction moves to loadBalancerSourceRanges, enforced in eBPF on the VIP, and rate limiting is disabled because SNAT makes the whole LAN look like one client. Proven from the workstation in the worst case — pod and both protocol leases on three different nodes: wildcard, both overrides, recursion over udp and tcp, NXDOMAIN outside the zone, NODATA for AAAA, source-range drop when excluded, and survival of a cross-node reschedule with no intervention. |
||
|
jakman85andClaude Opus 5
|
fd43f332d6 |
feat(pihole): authoritative LAN DNS for the zone on udp/53 + tcp/53 at a pinned VIP
platform/coredns/coredns-custom.yaml has always said its split-horizon is for
PODS and deferred LAN clients to "PiHole split-horizon". PiHole served no DNS:
its Service was ClusterIP with ports 80 and 4180 only. The LAN half of R-DNS-03
that docs/09 and the README's Phase 2 row both claim did not exist, so LAN
clients ran on hand-maintained per-client dnsmasq files - one went stale after
the dev decommission and cost the operator sonarr.home.keithjackson.dev. The
SOHO router that owns 192.168.2.0/24 cannot cover for it either: it strips
RFC1918 answers for public domains as rebind protection and returns NOERROR
with no answer.
EXPOSURE (apps/wave-0/pihole-dns.yaml, hand-written, placed into wave 0 by
render.py's EXTRA_UNITS - the schema has no field for a LoadBalancer Service, a
pinned lbipam VIP, a sharing key, source ranges or UDP, the same reason
apps/vms/unifi and platform/crafty hand-write theirs):
- two Services, udp/53 and tcp/53, sharing one pinned VIP
${PIHOLE_DNS_VIP} = 192.168.2.23 on prod, the next free address in the
.20-.30 pool (pool status IPsUsed 3 of 11; .20 Gateway, .21 unifi, .22
crafty; arping from a LAN host got no reply).
- no external-dns hostname annotation: nothing public is created, and
platform/externaldns/ is untouched.
ANSWERS (catalog/pihole.yaml env - declarative, no reconciler): PiHole v6 takes
every FTL setting as an FTLCONF_ environment variable and treats it as
env-forced, applied ahead of pihole.toml on every cold start and read-only in
the UI. That is a real declarative entry point, which the Keycloak and CNPG
CronJob reconcilers exist precisely because their state lacks, so a reconciler
here would be ceremony. misc.dnsmasq_lines carries a wildcard
address=/<zone>/<GATEWAY_VIP> plus one longer-domain override per L4 service
(unifi, crafty-mc, pihole-dns); dnsmasq resolves overlaps by longest matching
domain, so precedence is structural and a newly onboarded Gateway-fronted
service needs no DNS change. dns.upstreams is declared too - the live pod
carried hand-set upstreams that no manifest knew about and that every restart
discarded.
externalTrafficPolicy Cluster, NOT Local, and loadBalancerSourceRanges for the
LAN restriction. Measured on prod, Cilium 1.20.1: the L2 announcer does not
filter lease candidates by which nodes hold a backend - the two Services landed
on k3s-prod-3 and k3s-prod-2 while the pod ran on k3s-prod-1 - and it keeps
renewing after the pod moves away, so with Local a reschedule leaves LAN DNS
dead indefinitely. Cluster serves the VIP from any node; the LAN restriction
moves to loadBalancerSourceRanges, which Cilium enforces in eBPF on the VIP
before load-balancing (verified by narrowing the range to exclude the test
client and back). The cost is SNAT: forwarded queries reach PiHole from the
ingress node, so PiHole's per-client rate limit is disabled in the catalog or
the whole LAN would count as one client against its 1000-per-60s default.
Proven from a LAN host (192.168.2.132) against 192.168.2.23 with the pod and
both leases on three DIFFERENT nodes: sonarr/grafana/nextcloud/keycloak/apex
-> 192.168.2.20, unifi -> .21, crafty-mc -> .22, pihole-dns -> .23, recursion
(cloudflare.com, github.com, debian.org) intact over both udp and tcp, AAAA
NODATA, NXDOMAIN outside the zone, and still answering after a forced
prod-1 -> prod-2 reschedule.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
ce7819a7da |
docs: FINDING — public DNS publishes internal RFC1918 addresses, contradicting R-DNS-03
Operator-reported, verified. external-dns runs provider cloudflare with gateway-httproute + service sources and publishes the Cilium Gateway VIP 192.168.2.20, so ~16 hostnames in a public authoritative zone answer with RFC1918 space. The spec says the opposite: external resolves to the public IP, Pi-hole serves the internal split-horizon answers, internal services are unreachable from the internet. Provenance stated plainly: the pattern predates me — external-dns always published the Gateway address, and dev did the same under owner=default. My apex migration changed which names carry it, propagating an existing flaw onto the apex rather than inventing it. Four consequences recorded: topology and service-inventory disclosure; a DNS-rebind primitive, which is why the SOHO router strips these answers and LAN clients need per-client overrides; the five services declaring exposure public are not actually publicly reachable; and seven internal services are publicly advertised. Also records why deleting the records is not a safe first step: the *.home.keithjackson.dev wildcard is gone, so removal yields NXDOMAIN rather than a fallback. Remediation sequenced, nothing started. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
3de79f81a5 |
docs: correct three mountstats figures I supplied from bad measurements
Handed an agent a baseline of 4.0-4.8 ms/op response, 0 ms/op queue, 0 major timeouts over 24h. Re-measurement found queue time is 0.013-0.067 ms/op (I read rounded output as zero); the 24h claim was really ten minutes, because the DaemonSet had just rolled and the range query returned 18 samples rather than ~2,880; and the baseline was idle, so 4.4 ms is a cached STATFS rather than real media I/O and thresholds could not be multiples of it. The 24h error is the same class as the label-values mistake recorded above: trusting a query whose window was never valid. Records two real discoveries: mounts sharing an NFS superblock share kernel RPC statistics, so prod-1's two plex mounts are one signal reported twice; and maximum_rpc_slots high-water is 586 on prod-1 against 2 on prod-2, an independent indication that prod-1 carries far more concurrent NFS work. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
5c96f4f3a7 |
merge: NFS per-export latency — separating a slow server from a backed-up client
Closes the gap the previous agent correctly left open: it could not write a rule against node_mountstats_* because the series did not exist until the collector rolled out, and the L0 guard rightly rejects a rule on an unscraped metric. 1,842 series are now live, so five rules land. The discriminator is the point. response_time is how long the SERVER held the request; queue_time is how long it waited on our node for an RPC slot. Both server rules carry 'unless queue_time/op > 5ms', so a queueing client can never be reported as a slow server, and the two server rules then partition the same slow-server state by our own op rate at 20/s — exactly one fires, no gap and no overlap. Plus major timeouts as critical, since that is hanging rather than slow. Corrects three figures I had supplied from bad measurements: client queue time is 0.013-0.067 ms/op, not zero; 'zero major timeouts over 24h' was really zero over about ten minutes, because the DaemonSet had just rolled and the range query returned 18 samples rather than ~2,880; and the baseline was an IDLE one whose 4.4 ms is a cached STATFS, so thresholds are argued from the link and the array rather than as a multiple of it. Also found that mounts sharing an NFS superblock share the kernel's RPC statistics — prod-1's plex and plex/downloads report identical rates to every digit — so one slow export raises two alerts. Documented rather than aggregated away, since aggregating would hide which path is affected. 14 mutations, all killed. Runbook triage table 10 rows -> 15, with 'not collected' stated plainly for both past incidents because the collector postdates them and that cannot be backfilled. |
||
|
jakman85andClaude Opus 5
|
79d9ab2c8c |
feat(monitoring): per-export NFS latency alerts — server-slow vs we-are-backed-up
Closes the gap `feat/io-pressure-diagnostic-context` deliberately left open. That
change enabled node-exporter's `--collector.mountstats` and recorded its own honest
limit: "an NFS export that is merely SLOW cannot be distinguished today", because
node_mountstats_* did not exist on prod until the HelmRelease rolled and the L0 guard
alert-metrics-have-scrape-targets rightly refuses a rule on an unscraped metric.
The rollout has landed. Verified on prod 2026-09-15: 1,842 node_mountstats_* series
(~0.4% of a 451,650-series head), six mounts, 426 per-operation series in each of the
five operation families and 6 in each transport family. Every metric referenced below
returns series under `promtool query instant`.
FIVE RULES in platform-health.node-io-cause, each naming its cause in `summary`:
NFSExportServerSlow response_time > 50 ms/op, our op rate > 20/s,
queue_time < 5 ms/op -> the server is slow BECAUSE OF US
NFSExportServerSlowNotOurLoad same, but our op rate 1-20/s -> the server is slow and
the load is NOT OURS. This is the operator's Proxmox
hypothesis made testable.
NFSClientQueueTimeHigh queue_time > 5 ms/op -> WE are saturating the transport
NFSTransportBacklogDeep backlog depth > 8 per send -> even the grown slot table is full
NFSExportMajorTimeouts any major timeout in 15m -> the export is HANGING (critical)
THE DISCRIMINATOR. mountstats splits a slow NFS op into the half the server owns
(response_time) and the half we own (queue_time). High response with low queue means the
SERVER; high queue means US. The two server rules then partition that state by our own op
rate at 20/s, so exactly one of them can fire and it names which reading applies. The
`unless queue_time > 5ms` clause is what licenses reading response_time as a statement
about 192.168.2.102 rather than about us, and it is pinned by fixture 5 (a client queueing
30 ms/op while the server appears to take 60 ms/op: both server rules must stay quiet).
MEASURED BASELINE, prod 2026-09-15, re-measured for this change:
server response time 4.40-4.81 ms/op (30m window), uniform across all six mounts
client queue time 0.013-0.067 ms/op; snapshot maximum 0.333 ms/op
our own op rate 0.019-0.163 ops/s — STATFS keepalive and nothing else
major timeouts 0 on every mount
backlog depth 0 on every mount
slot-table high-water 586 (prod-1), 162 (prod-3), 2 (prod-2)
THREE LIMITS ON THAT BASELINE, recorded in the rules and the runbook rather than papered
over:
(a) the history is MINUTES, not days — the series begin when the DaemonSet rolled
(count_over_time(...[24h]) returned 18, and time() - process_start_time_seconds read
558-561 s on all three nodes), so no rule uses a range longer than 15m;
(b) it is an IDLE baseline — STATFS keepalive only, no READ/WRITE traffic, so the 50 ms
threshold is argued from the link and the array (one outstanding 1 MB read at 50 ms is
20 MB/s) and must be re-cut after the first real media event. The runbook carries the
query;
(c) prod-1's /mnt/Storage/plex and /mnt/Storage/plex/downloads report byte-identical
per-operation counters (0.12443893333333335 on both, to every digit), as do prod-3's two
nextcloud mounts — mounts sharing one NFS superblock share the kernel's RPC statistics.
One slow export on prod-1 therefore raises two alerts. Documented, not aggregated away.
PROOF, not assertion. platform/monitoring/tests/nfs-export-latency.test.yaml: seven fixtures
(measured healthy; healthy under 60 ops/s of real load; server-slow-ours; server-slow-not-ours;
client-backed-up; backlog-deep; export-failing), and every fixture asserts all five alertnames
in the fires or the quiet direction. The measured baseline is pinned as a band, the
shared-superblock duplication is pinned, and both hazards the sibling recorded are asserted:
the PromQL precedence trap (`(ratio) and rate(ops) > 1 > 0.050` returns a series on a healthy
60 ops/s mount; the shipped form returns none) and the op-rate guards (a mount doing half an
operation a second at 1 s/op response, 1 s/op queue and 20 deep in the backlog — 20x, 200x and
2.5x the three thresholds — must stay silent).
MUTATION-CHECKED, 14 mutations, every one of them making the test file FAIL: each of the five
thresholds moved in both directions where the fixtures allow it, both `unless` clauses deleted,
and all four rate guards loosened. The two clause deletions fail with an exp/got alert diff,
not a parse error. No threshold here is pinned only by a promql_expr_test restating it.
Live on prod, all five shipped expressions return NO series while all four of their
sub-expressions return six each — so the silence is a verdict, not missing data.
runbooks/node-io.md: the triage table grows from 10 rows to 15 (per-export server response
time, client queue time, our own demand, major timeouts, backlog depth), with real measured
HEALTHY values and an explicit "not collected" in both incident columns, because the collector
postdates both events and that can never be backfilled. A new four-way verdict table reads rows
11-13 together and maps each combination to the alert that says it.
HONEST REMAINING GAP: these rules rule US out; they cannot rule Proxmox IN. The per-client view
of an export exists only on the server. What would settle it is one read on 192.168.2.102 while
NFSExportServerSlowNotOurLoad is firing — `nfsstat -s` and /proc/fs/nfsd/clients/*/info — which
is out of scope from this repo. Recorded in the runbook.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
de861b8672 |
docs: correct my own claim — node_nfs_* metrics DO exist; mountstats is the gap
I recorded that NFS was 'entirely unmonitored' with 'zero NFS observability',
based on a __name__ label-values query that came back empty. The query was at
fault: count({__name__=~"node_nfs.*"}) returns six live families on all three
nodes, and the nfs collector is on by default.
The real gap is narrower — node_mountstats_* is absent, and that is the one
carrying per-mount RPC round-trip time, so NFS latency specifically is
invisible while volume and retransmissions are not.
Two further signals already existed and had simply never been queried:
container_pressure_io_stalled_seconds_total, which names the stalling workload
and for tonight names sonarr; and etcd_request_duration_seconds via the
apiserver job, which the node-io runbook had explicitly claimed did not exist.
Also tempers the Proxmox hypothesis I floated: prod-1 was doing 114 NFS reads/s
of its own during the spike, so our traffic is substantial and the stall is not
obviously unexplained by us.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85
|
207c939f89 |
merge: I/O pressure alerts now name the cause, not just the stall
NodeIOPressureCritical fired on prod-1 and said nothing about why; establishing 'not a failing drive' took eight ad-hoc queries after the fact. Eight new rules each name a class in the summary — device saturation, device latency, NFS transport, etcd, not-local-disk, and which container is stalled — with the companion PromQL in a diagnostic_query annotation, a WHY row on the existing dashboard, and a 10-row triage table in runbooks/node-io.md carrying real HEALTHY / 2026-09-14 / 2026-09-07 columns. The discriminator is NodeIOPressureNotLocalDisk, written with 'unless' rather than 'and ... < T' on purpose: an empty right-hand side removes nothing, so it still fires when devices are too idle to have a measurable per-op latency — 'and <' would go silent in exactly the state it exists to detect. Acceptance is checked against both real profiles. Tonight: NotLocalDisk fires and ContainerIOStallHigh names sonarr. 2026-09-07 re-queried live at offset 164h: 1,039 ms per write on prod-1 sda, so the latency rules fire and name the device while NotLocalDisk correctly stays silent. Healthy state: all quiet. 10 mutations applied, 10 killed — one initially survived and exposed a threshold pinned only by a test that restated it. Corrects two claims: the runbook said no etcd fsync metric exists, but etcd_request_duration_seconds is present via the apiserver job and attributes per node; and container_pressure_io_stalled_seconds_total was already scraped with ~400 series and unqueried. Also fixes two bugs found in the work itself: a PromQL precedence error where 'ratio and rate(ops) > 10 > T' compares T against the op rate rather than the ratio, and node-exporter extraArgs replacing rather than appending, which would have silently dropped the chart's filesystem excludes. |
||
|
jakman85andClaude Opus 5
|
aae63c97a3 |
fix(monitoring): node-exporter extraArgs REPLACES the chart default, so restate it
Verified against the chart (helm show values prometheus-community/prometheus-node-exporter) and against the live DaemonSet's argv on prod: the subchart's extraArgs default is NOT empty — it ships the two filesystem excludes (mount-points-exclude for /dev,/proc,/sys,containerd,docker,kubelet and the fs-types-exclude list). Setting extraArgs to just --collector.mountstats would have silently dropped both, so the filesystem collector would have begun reporting every container overlay and every kubelet bind mount: a large cardinality jump, and a behaviour change in NodeDiskFillingUp, which alerts per mountpoint. Both defaults are now restated verbatim alongside the new collector, with a note to re-reconcile them on a chart upgrade. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
3a72b07cda |
chore(conformance): shrink UNTESTED_ALERTS by the two PSI detectors
node-io-cause.test.yaml's healthy fixture asserts NodeIOPressureHigh and NodeIOPressureCritical quiet on the same measured prod state the eight new cause rules are pinned against — the point of that fixture being that the detectors and the cause rules must agree about what "healthy" is. The shrink-only register entries are therefore stale, and the gate correctly refused them. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
b9f3143f42 |
feat(monitoring): name the CAUSE of node I/O pressure, not just the stall
NodeIOPressureCritical has now fired twice on k3s-prod-1 for opposite causes and
said the same thing both times. On 2026-09-14 02:40-02:45Z it fired at PSI(full)
20.98% — twice the threshold, four times the 2026-09-07 level — while every local
device was idle (0.09-0.27 utilisation) and fast (1.4-4.8 ms per write), NFS
retransmissions were zero and no Longhorn volume was degraded. Establishing that
took eight ad-hoc PromQL queries after the event had already recovered.
Adds the group platform-health.node-io-cause: eight rules, one per cause class,
each naming its cause in its own summary.
NodeDiskDeviceSaturated device saturation >0.85 busy, 10m
NodeDiskWriteLatencyHigh device latency >20ms/write, 10m
NodeDiskWriteLatencyCritical device latency >50ms/write, 5m
NodeDiskReadLatencyHigh device latency >250ms/read, 15m
NodeNFSRetransmissionRateHigh NFS transport >1% retransmitted, 10m
NodeIOPressureNotLocalDisk not local disk PSI>10% with all of the above absent
NodeEtcdRequestsSlow etcd >0.5% of etcd requests over 1s
ContainerIOStallHigh which workload per-container PSI >15%
Every threshold is measured on prod over 24-48h, not guessed, and each rule
records its headroom. The PSI thresholds themselves are deliberately unchanged:
the 21% stall was real, it fired twice in 24h for two real events on the one node
that has a problem, and raising the bar would also have raised it above the 11.36%
at which prod-1 was already losing etcd on 2026-09-07.
Context reaches the operator at page time three ways: the summary states the cause
class; every rule carries a diagnostic_query annotation with the exact companion
PromQL; and node-pressure-dashboard.yaml gains a WHY row (seven panels) plotting
saturation, write/read service time, NFS retransmissions, our own NFS op rate,
the etcd slow-request fraction and per-container PSI side by side. runbooks/node-io.md
gains a 10-row triage table with the real healthy / 2026-09-14 / 2026-09-07 columns,
which turns those eight queries into a two-minute read.
etcd and NFS metric availability, verified on prod rather than assumed:
* etcd SERVER disk metrics do NOT exist. etcd_disk_wal_fsync_duration_seconds and
etcd_disk_backend_commit_duration_seconds are absent (kubeEtcd disabled for k3s);
a query for etcd_disk_*/etcd_server_*/etcd_mvcc_* returns nothing at all.
NodeEtcdRequestsSlow uses the apiserver's client-side view of its own local
member instead, and says in its own description that it is a proxy.
* node_nfs_* DOES exist — the nfs collector is on by default, host-wide via
/proc/net/rpc/nfs, confirmed on all three nodes. It gives op counts and
retransmissions and NO latency, so it can see a dropping export but not a slow
one. The mountstats collector is enabled here to close that gap; NO rule
references node_mountstats_* yet, because the series do not exist until the
HelmRelease rolls out and the collector's visibility through the node-exporter
mount namespace must be verified on prod first. The runbook records the check
and says to revert if it comes back empty.
Proof, per the repo's two shipped alerting failure modes:
* platform/monitoring/tests/node-io-cause.test.yaml — 7 fixtures, all from
measured profiles: healthy (all 8 quiet), 2026-09-14 (NotLocalDisk fires and
names sonarr), 2026-09-07 re-queried at offset 164h (write-latency and etcd
fire, NotLocalDisk correctly silent), rebuild storm, NFS transport, read
latency, and the `unless` semantics.
* 10 mutations of the shipped rules, all killed. The mutation run found a real
gap: the etcd threshold could be changed without failing any test, because it
was only pinned by a promql_expr_test that restated the threshold in the test
file. An alert_rule_test now pins it.
* All 11 referenced metrics verified to return series on prod with
`promtool query instant`, and all 8 expressions verified quiet on today's state.
Also extends the alert-metrics-have-scrape-targets owner table for container_*
(the kubelet ServiceMonitor's cAdvisor path, which this entry already claimed in
its title) and for etcd_request_duration_seconds_* by EXACT NAME, deliberately not
by an etcd_ prefix, so a future rule on the unscraped etcd_disk_* names still fails
the gate.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
7369495b67 |
docs(pm): owner decisions — files stay, workloads stay, instrument NFS instead
Records three owner decisions and the finding that makes the third tractable. The /downloads malware stays: the Proxmox-hosted stack still has operational control of those files so deletion just triggers a re-download, nothing but the arr suite touches the directory, and there are no Windows machines anywhere — which removes the exposure I was concerned about. Written down so nobody 'discovers' and deletes them later. Anti-affinity for the co-located NFS pods declined; the concentration is accepted rather than overlooked. And the enabling finding: there are NO NFS metrics in the TSDB at all — node-exporter's nfs and mountstats collectors are off by default. A cluster with four NFS mounts carrying the media library and legacy Nextcloud data has zero NFS visibility, while PSI counts NFS stalls as I/O stalls. That is precisely why a 20.98% stall alongside idle local disks could not be explained. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
dafa9cd241 |
docs: prod-1 I/O critical was not a failing disk; confirmed malware in /downloads
NodeIOPressureCritical fired on prod-1 (peak 20.98% PSI, recovered
unattended) — the first time this alert has fired on prod, and only because
the scrape gap was closed yesterday.
Evidence rules out the 2026-09-07 drive repeat: 3.5 MB/s of writes, device
utilisation 0.09-0.21, write latency 1.4-4.8 ms, zero NFS retransmissions,
21.4 MB/s NFS throughput, no degraded Longhorn volumes, and KSM never observed
NotReady. A 21% stall on trivial volume with healthy latency is transient
blocking I/O, not hardware — and PSI counts NFS too.
Structural finding: sonarr, qbittorrent and jackett all run on prod-1, the
node with the I/O history. Same concentration class as the CNPG primaries.
Separately, the /downloads files are confirmed malware: two PE executables
('MZ') sharing one MD5, padded to ~926 MB and named as two different shows,
plus a ZIP masquerading as video. Not executed, cluster not compromised; the
risk is a Windows client opening them.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
0841e2461c |
docs: correct a misattribution — 192.168.2.1 is the SOHO router, not pfSense
I wrote that pfSense was stripping the RFC1918 answers. The operator corrected it, and this repo already documented the topology I failed to read: 192.168.2.0/24 is pfSense's WAN (clusters/prod/kustomization.yaml:458, cluster-vars.yaml:190, DECISIONS-LOG 1772), pfSense lives on 192.168.1.0/24, and 192.168.2.1 was already recorded here as 'both gateway and only IPv4 resolver' and flagged as a SPOF (entries 1318, 1468). The measurement is unchanged — that resolver returns NOERROR/ANSWER:0 for this zone while resolving cloudflare.com — but the attribution invalidated my preferred fix: configuring pfSense cannot help, because prod's resolver path never reaches it. Fixes reordered accordingly. Exposing Pi-hole's DNS to the LAN is now first: it is the architecture the repo already claims, it does not depend on a consumer router's feature set, and it retires the single-resolver SPOF this log flagged months ago. Configuring the SOHO router drops to second, since many such devices cannot express a domain override and it is outside GitOps regardless. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8891df07b0 |
docs: LAN split-horizon DNS is documented but unimplemented; pfSense strips the answers
The workstation could not reach sonarr.home.keithjackson.dev. The stale dnsmasq override pinning the zone to dev's dead 192.168.2.50 was the obvious cause, but removing it — which is what I was about to recommend — would not have fixed it: pfSense returns NOERROR/ANSWER:0 for this zone while resolving cloudflare.com fine, the signature of private-address rebind filtering, since the zone legitimately points at RFC1918 space. The underlying gap: coredns-custom.yaml defers LAN clients to 'PiHole split-horizon', but Pi-hole's Service is ClusterIP with ports 80/4180 — it serves no DNS at all. So the LAN half of split-horizon does not exist and each client silently depends on a hand-maintained file. Records the corrected override (staged for the operator's sudo) and the three proper fixes, none applied. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
268b0a26b4 |
docs(readme): DR restore is proven on prod — Phase 7's remaining item is node-loss
R-STATE-03 reads proven in --status as of 2026-09-14, the first requirement in this repo to reach that state, with both verifiers recorded on prod. Also states why the earlier PROVEN: 0 was the honest answer rather than a bug: a proof is a claim about a specific tree on a specific cluster, so coverage (51/51) and proof are two different questions. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
4f2ca8c1a3 |
test(conformance): record the L0 verifier for R-STATE-03 / R-ENV-02
A requirement reads PROVEN only when EVERY verifying test has a fresh ledger row — the state is the minimum across them, so a test with no row contributes not-derivable and one recorded verifier cannot manufacture green. R-STATE-03 carries two executable verifiers (L0 + L3). The L3 dr-restore proof landed in bd17b65; this records the L0 side, placeholder-guard-coupling, which also carries R-ENV-02. Worth noting it is the test that was permanently SKIPPED until earlier today — it was hiding a red, and now it runs and is recorded. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
bd17b65db9 |
test(conformance): R-STATE-03 PROVEN on prod — DR restore from R2, off-cluster
Phase 7's outstanding exit criterion. The only previous dr-restore proof was
recorded on the dev cluster (decommissioned 2026-09-12) against MinIO, so it
could not speak for prod or for today's Cloudflare R2 target.
Run 20260914T021505Z-24b8111, env=prod, kube v1.36.4+k3s1, chainsaw v0.2.12,
150.70s, 1 passed / 0 failed / 0 skipped. Sentinel
DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK.
What makes it a real proof rather than a green check: the test seeds a canary
on the replicated longhorn class, backs the namespace AND volume data up to
the off-cluster store, then DESTROYS every in-cluster copy — namespace, PVC
and PV — before restoring into a clean, previously-nonexistent namespace. It
asserts the restored PVC binds a DIFFERENT PersistentVolume
(pvc-f0c061e1 != pvc-4107d59d) and fails explicitly if it rebinds the
original, because that would be an in-cluster survival masquerading as a
restore. The canary came back byte-identical.
It touches no real service's namespace, PVC or PV — the scratch namespaces are
named after its own Chainsaw namespace and are destroyed by design.
This is the run whose first attempt was discarded for recording env=dev; the
recorder fix is
|
||
|
jakman85andClaude Opus 5
|
24b8111d0c |
fix(conformance): the ledger stamped every proof env=dev regardless of cluster
record_run.py defaulted --env to the literal 'dev'
(os.environ.get("CONF_ENV", "dev")) and run.sh hardcoded the same fallback
at both call sites, while CONF_ENV is set nowhere in this repo. So every
recorded proof claimed dev no matter which cluster it ran against.
Harmless while dev was the only cluster. Dev was decommissioned 2026-09-12,
and today a genuine DR-restore run against k3s-prod was recorded as
env=dev — caught before it was committed, and the record discarded rather
than hand-edited. A proof that lies about WHERE it was proven is worse than
no proof, because locating the demonstration is the ledger's whole job; it
would also have looked like a sixth dev-only row while actually being the
prod proof Phase 7 is waiting on.
env is now derived from k3s-prod (k3s-prod -> prod),
which cannot drift from the cluster the run actually talked to. An
undeterminable context records 'unknown', never a cluster name, and an
unexpected context is recorded verbatim so it is visible rather than coerced.
CONF_ENV still overrides. Recorder selftest passes.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
229d7f2d9c |
docs: README status was stale by a whole phase; record the R-STATE-03 decision
The Status section still said phases 0-6 were deployed on the DEV cluster and that Phase 7 was not open with clusters/prod 'a design-only overlay'. Dev was decommissioned on 2026-09-12 and prod has been the live cluster serving the apex since. It also claimed 36/51 requirements covered with a safety-critical gap, and that Velero was blocked on OQ-BKP. Now states Phase 7 as open with its three exit criteria and which one is actually outstanding; corrects the node inventory from the Proxmox dev IPs to the bare-metal prod nodes; records Velero as unblocked to R2; and adds the distinction the old text elided — coverage (51/51) is not proof (PROVEN: 0), because the only dr-restore and node-loss proofs were recorded on a cluster that no longer exists, against MinIO rather than R2. Also records the owner's decision that Velero+kopia to R2 is the accepted volume backup, with no Longhorn backup target, and that Longhorn config belongs in source. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a62242bd78 |
docs(pm): session close — three corrections to my own claims, and the remaining backlog
Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
bfb4922de1 |
merge: the Longhorn I/O caps the runbook promised are now actually declared
runbooks/longhorn-degraded.md asserted concurrentVolumeBackupRestorePerNode- Limit and snapshotHeavyTaskConcurrentLimit were tightened 5 -> 1 after the 2026-09-07 etcd I/O starvation. Both read 5 on prod and neither name existed anywhere in the repo. Now declared in the helmrelease defaultSettings, the path Flux actually reconciles. Both go to 1 rather than 2, and the cost is smaller than I assumed: velero's schedules run defaultVolumesToFsBackup, i.e. kopia through the node-agent, so they are not Longhorn volume backups and this setting does not bind them; CNPG's base backups are barman-level; and Longhorn has no backup target configured at all. Purge work comes from rebuild cleanup, which is already serialised to 1/node, so 1 keeps pace with its own generator while 2 would add a second heavy writer during exactly the rebuild that starved etcd. Also declares nodeDrainPolicy, which was only ever Longhorn's own default — present in no manifest and no rendered ConfigMap, so an upstream default change would have moved it silently while two runbooks reason about drain safety from it. Value unchanged. The Ansible etcd_io_priority role is deliberately NOT ported. The pathological drive it was written for has been replaced (I/O PSI 11.36 -> 3.79 under the same load class), the role was never run against any cluster, and its own README concedes it cannot separate the tenants because they share one ext4 filesystem. Swapping the elevator and halving nr_requests on the live device holding etcd's WAL is a node-wide blast radius for a low ceiling. Kept as a named operator option; the real fix remains giving etcd its own device. Removes orphanAutoDeletion, re-verified present-and-inert two ways: helm template emits no orphan key for it, and the live setting is NotFound. |
||
|
jakman85andClaude Opus 5
|
f663e7c801 |
fix(longhorn): the I/O caps a runbook promised were never declared — declare them
`runbooks/longhorn-degraded.md` told operators that, after the 2026-09-07 etcd
I/O-starvation incident, `concurrentVolumeBackupRestorePerNodeLimit` and
`snapshotHeavyTaskConcurrentLimit` had been tightened 5 -> 1. Measured live on
prod today: both read **5**, and neither name appeared anywhere in this
repository — so nothing declared them and nothing would have restored them.
Fourth instance of config that looked applied and was not.
Landed in `platform/longhorn/helmrelease.yaml` `defaultSettings` (the path Flux
reconciles; `values.yaml` is the reference mirror), NOT by `kubectl patch` —
a hand-applied Setting is exactly how this gap arose and Helm would silently
rewrite it on its next upgrade.
concurrentVolumeBackupRestorePerNodeLimit 5 -> 1
snapshotHeavyTaskConcurrentLimit 5 -> 1
replicaReplenishmentWaitInterval 600 PINNED, unchanged
backupConcurrentLimit / restoreConcurrentLimit 2 / 2 PINNED, unchanged
concurrentAutomaticEngineUpgradePerNodeLimit 0 PINNED, unchanged
nodeDrainPolicy block-if-contains-last-replica PINNED — it was in
force only as Longhorn's OWN DEFAULT, declared in no manifest and in no
rendered ConfigMap, so an upstream default change would have moved it
silently while two runbooks reasoned about drain safety from it.
EVERY KEY PROVEN TO LAND, not assumed. `helm template longhorn/longhorn
--version 1.12.1` (the version this chart range resolves to, per the
HelmRelease's own lastAttemptedRevision) on this branch's spec.values,
substituted as clusters/prod substitutes them, emits all eight into the rendered
`longhorn-default-setting` ConfigMap in kebab-case. Diffed against the LIVE
ConfigMap the change is purely ADDITIVE: 8 keys added, none altered, none lost.
The chain to the Settings is verified read-only on prod — all 115
settings.longhorn.io carry `longhorn.io/configmap-resource-version: 7917`
(= that ConfigMap's resourceVersion) with `status.applied: true`, and every key
in the ConfigMap equals its Setting's value. The two targets sit at 5 with the
key absent, i.e. Longhorn defaults, as measured.
`orphanAutoDeletion: true` REMOVED, verified inert two independent ways against
the deployed 1.12.1: the render emits no orphan key at all (Longhorn 1.9 renamed
it `orphanResourceAutoDeletion`, a list, and the chart drops unknown
defaultSettings keys with no error), and live, `setting/orphan-auto-deletion` is
NotFound while `orphan-resource-auto-deletion` reads "". Deliberately NOT
replaced — orphan deletion is extra writes on the constrained partition and buys
only space. Orphan cleanup stays manual by decision; the two runbooks that cite
the key by name are corrected rather than left to go stale.
BOTH NEW CAPS ARE 1, AND 1 COSTS SOMETHING — stated, not waved at.
`concurrentVolumeBackupRestorePerNodeLimit: 1` costs nothing TODAY: Longhorn has
no backup target (the `backup-target` Setting does not exist; still the
WS-STORAGE Phase-4 TODO), and velero's nightly run is not a Longhorn backup —
`snapshotVolumes: false` + `defaultVolumesToFsBackup: true` is kopia
file-system backup through velero's node-agent, which this Setting does not
gate; CNPG base backups are barman-level and ungated too. Its only consumer is
an R-STATE-03 restore, where serialising ~24 volumes beats losing etcd
mid-restore. Revisit when a Longhorn backup target lands.
`snapshotHeavyTaskConcurrentLimit: 1` rather than 2 because purge work is
generated by rebuild cleanup (auto-cleanup-system-generated-snapshot true, NO
RecurringJobs, 24 volumes / 24 snapshots) and rebuild is already serialised to
1 per node — so 1 keeps pace with its own generator, where 2 buys backlog drain
at the price of a second heavy writer during the exact rebuild that starved
etcd. The cost: space reclaims serially after a rebuild storm — real here (a
37.7 GiB snapshot once pinned a volume at 97.3% allocated), but instrumented by
LonghornDiskAllocationHigh and recoverable by hand, where a wedged etcd is not.
Headroom on the record: I/O PSI full is 0.30/0.62/0.37 % now, and 3.79 under a
full 18-replica rebuild after node 1's drive replacement, against a 5% warning
threshold. These caps are a BOUND, not a fix — etcd and Longhorn still share one
ext4 filesystem and nothing arbitrates the jbd2 journal fdatasync waits on.
NOT PORTED, deliberately: the same unmerged branch's node-level I/O arbitration
(`roles/etcd_io_priority` — BFQ on the shared device, nr_requests 64 -> 32,
systemd IOWeight drop-ins). The 2011 C400 it was written for was replaced on
2026-09-07 and the same load class measured 11.36 -> 3.79 I/O PSI with the node
Ready throughout; the role was never run against any cluster, its author
recorded it as unproven with a node-3-first sequenced rollout, and its own
README concedes it does not separate the two tenants. Node-wide blast radius,
low ceiling, stale measurements (node 1 reinstalled, founder moved to node 2).
It stays an operator option if I/O PSI climbs back toward 5% — which, unlike in
September, we would now be told about. Nothing under infra/ansible/ is touched.
ALSO ALREADY DONE, so skipped: that branch's NodeIOPressureHigh/Critical and the
dashboard I/O row are on main already (
|
||
|
jakman85andClaude Opus 5
|
d96165e8fb |
fix(monitoring,runbook): 5 GiB is not compaction headroom, and a runbook claimed an I/O cap that was never applied
Two things found while reviewing the Longhorn alert work. Prometheus retentionSize was 45GiB of a 50Gi PVC, under a comment already naming the right intent — 'headroom for WAL + head + compaction scratch'. 5 GiB is not that: retentionSize counts only persisted blocks, and a large compaction can need roughly the size of the blocks being merged. ENOSPC on /prometheus wedges every WAL commit, so scrapes keep succeeding while nothing is stored and every absent()/==0 alert fires falsely at once. Lowered to 36GiB, which costs no history: the TSDB is 11.2 GiB at 30d, so 30d remains the binding limit. runbooks/longhorn-degraded.md asserted that concurrentVolumeBackupRestore- PerNodeLimit and snapshotHeavyTaskConcurrentLimit 'was actually tightened 5 -> 1'. Both read 5 on prod and neither name appears anywhere in this repo — the work is on the unmerged fix/etcd-io-priority branch. The page is read during etcd I/O starvation on a node whose WAL shares a device with Longhorn, so it was telling an operator mid-incident that the cap they depend on is in place when it is not. Corrected, with the two mitigations that ARE real named and a command to verify any of them. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
52398d4426 |
merge: Longhorn alerting measured the wrong things, three ways
(1) LonghornVolumeCapacityHigh alarmed on ALLOCATION while calling itself a capacity alarm. Prometheus's TSDB volume read 0.973 while the kubelet saw the same filesystem at 0.208, because 37.73 GiB of it is a Longhorn SYSTEM snapshot — the kind taken for the full replica rebuild that prod-1's drive replacement forced — growing +152 MB/h while the filesystem stayed flat. The rule was therefore unclearable by design, i.e. permanently firing. Deleted as redundant, shown rather than asserted: 23 of 24 volumes fall under the kubelet-based filling rules, and the 24th is volumeMode Block, which the deleted rule was not covering either. (2) LonghornVolumeDegraded and LonghornVolumeFaulted could not fire even with the scrape working. longhorn_volume_robustness on 1.12.1 is ONE-HOT over a state label with values 0/1, not the 0-3 enum the rules and their comment assumed, so a faulted volume would have alerted nobody. This survived the scrape fix landing hours earlier. (3) LonghornNodeStorageHigh was wrong twice: its 0.85 threshold sat above the 75% at which storage-minimal-available-percentage=25 already removes a disk from the scheduling pool, and longhorn_node_storage_usage_bytes SUMS a node's disks, so one full disk hid behind a half-empty sibling. Replaced with per-disk LonghornDiskAllocationHigh (>0.70) and LonghornDiskUnschedulable, the latter reading Longhorn's own verdict instead of re-deriving a formula from three settings no metric exports. Baseline green with 11/11 mutations red. retentionSize 45GiB of 50Gi and discard/TRIM are both REPORTED with recommendations, not changed: TRIM would not have helped here, since remove-snapshots-during-filesystem-trim is false and the 37.7 GiB is a snapshot. # Conflicts: # DECISIONS-LOG.md |
||
|
jakman85andClaude Opus 5
|
4687521d65 |
fix(monitoring): delete the Longhorn rule that measured allocation and called it capacity
LonghornVolumeCapacityHigh went pending within minutes of the Longhorn scrape
landing today. Re-measured on prod: it read 0.9731 on the Prometheus TSDB volume
(actual_size 52243681280 / capacity 53687091200) while the kubelet reported the
SAME filesystem 20.8% used. actual_size is head + pinned snapshots, and one
user_created=false system snapshot from prod-1's replica rebuild holds 40510484480
bytes (37.7 GiB) of it; Longhorn returns no freed blocks without discard/TRIM, and
retentionSize 45GiB on a 50Gi volume targets 90% by design. The ratio is pinned
near 1.0 and cannot fall back under 0.85 — a permanently-firing alert, the same
shape as CNPGNoRecoverabilityPoint.
DELETED, not re-thresholded. Raising the threshold would tune away a true
statement about allocation. Redundancy proven, not asserted: of 24 Longhorn
volumes on prod, 23 appear in kubelet_volume_stats_inodes > 0 and so fall inside
MonitoringVolumeFillingUp (namespace="monitoring") or PersistentVolumeFillingUp /
PersistentVolumeAlmostFull. The 24th is unifi/unifi-rootdisk, volumeMode Block —
no filesystem for anyone to measure, and the deleted rule was not covering it
either.
AND A SECOND INERTNESS the scrape did not fix. longhorn_volume_robustness on
longhorn 1.12.1 is one-hot over a `state` label with values 0/1, not the 0-3 enum
the rules assumed:
count(longhorn_volume_robustness) => 96 (24 volumes x 4 states)
count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
count(longhorn_volume_robustness == 2) => (empty)
count(longhorn_volume_robustness == 3) => (empty)
So a FAULTED volume would still have alerted nobody. Re-keyed to the label; the
"44 series / 72 unknown(0, detached)" note in platform/longhorn/helmrelease.yaml
was the same misreading and is corrected in place.
The allocation question moved to where allocation is a budget. LonghornNodeStorageHigh
fired at 0.85, but storage-minimal-available-percentage is 25 on this cluster, so a
disk leaves the scheduling pool at 75% used — the old threshold could only fire
after scheduling had already been refused. Replaced by:
LonghornDiskAllocationHigh (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70, 30m
LonghornDiskUnschedulable longhorn_disk_status{condition="schedulable"} == 0, 15m
Per DISK, not per node: longhorn_node_storage_usage_bytes sums a node's disks, so
one full disk could hide behind a half-empty sibling. Measured prod usage
0.2254 / 0.2306 / 0.2369 on prod-1/2/3.
PROVEN NEITHER PERMANENTLY FIRING NOR INERT. platform/monitoring/tests/longhorn.test.yaml
adds 6 cases pinning every rule quiet on today's measured prod state and firing on
a constructed bad state, with every `for:` asserted, plus regression assertions that
the old `== 2`/`== 3` forms match nothing and that per-disk 0.72 is visible where the
node-summed 0.475 is not. Mutation-checked 11 ways; each turns the suite RED. Live on
prod: all four new expressions return empty, while the sum of their selector
cardinalities is 54. The L0 UNTESTED_ALERTS register shrinks by seven (7/56 -> 14/58
alert rules unit-tested), which also brings MonitoringVolumeFillingUp,
PersistentVolumeFillingUp and PersistentVolumeAlmostFull under test since the
redundancy claim rests on them.
REPORTED, NOT CHANGED (DECISIONS-LOG, owner decisions): retentionSize 45GiB on a
50Gi volume is too tight — recommend 36GiB, with reasoning; and discard/TRIM on the
longhorn StorageClass is feasible but not advisable now — remove-snapshots-during-
filesystem-trim reads false on prod, so trim would not have reclaimed the 37.7 GiB
anyway, and continuous discard on nodes where the etcd WAL shares the spindle points
at this cluster's known failure mode.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
69a99e6ca6 |
docs(pm): two rendered manifests nothing applies — the keycloak one is a trap
Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
eb2510837c |
merge: twelve app SLI rules could never fire — fixed in the generator
Each <app>-down rule read up{job=...} from a ServiceMonitor selecting a
Service port named 'metrics' that no app Service defined, so every one
discovered zero targets. PrometheusTargetsMissing could not see it either: a
target never discovered has no up series to be 0.
The port name was only the first of THREE layers. None of the twelve apps
serves Prometheus metrics natively at all — Plex, qBittorrent, the arrs,
Jackett, Pi-hole, Seerr, Semaphore and Nextcloud would each need an exporter,
and Gitea needs an app.ini flag. And every app namespace ships default-deny
ingress with no monitoring allow, so a correct port would still have been
dropped by policy — while renderer-contract.md claimed a baseline allow that
never existed.
Fixed in catalog/render.py (schema 1.6.0) rather than in twelve rendered
files: slis: [up] now renders <app>-down against kube-state-metrics
readiness, with an 'or absent(...)' half so the signal VANISHING fires
instead of silencing. The twelve dead ServiceMonitors are gone; a new
observability.metricsEndpoint renders the Service port, the monitor endpoint
and the scrape allow together, so the three cannot disagree again.
Deletion was rejected with evidence: kube-prometheus-stack's generic
kubernetes-apps rules are live, but they are namespace-keyed warnings, not
the per-service critical SLI R-OBS-01 asks for.
Two incidental finds: keycloak's rendered monitor was a duplicate pointing at
the wrong port, and platform/monitoring/tests/run.sh discovered test suites
only inside its own directory while documenting the convention as general —
so a suite added anywhere else was never run and still looked like coverage.
|
||
|
jakman85andClaude Opus 5
|
cf7d97e546 |
fix(obs): twelve app SLI rules read a signal that exists; kill the dead ServiceMonitors
Twelve `<app>-down` SLI rules were STRUCTURALLY INERT. Each read `up{job="<app>"} == 0`,
and the ServiceMonitor rendered beside it named `endpoints[].port: metrics` — a port NO
rendered app Service defined. Every monitor discovered ZERO targets, so no `up{job="<app>"}`
series existed and the rule could not fire. Verified on prod before this change:
`count by (job) (up)` listed 24 jobs and not one of the twelve appeared; `up{job="plex"}`
and `up{job="gitea"}` both returned nothing. `PrometheusTargetsMissing` could not see it
either — a target that was never discovered has no `up` series to be 0.
THE PORT NAME WAS NEVER THE MISSING PIECE. Ground truth per app: Plex, Pi-hole, Jackett,
qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export NO Prometheus metrics
without a separate exporter sidecar; Gitea can but only with app.ini metrics enabled;
Keycloak does, on the operator's management port, already scraped by
platform/keycloak/servicemonitor.yaml; vm-demo is a KubeVirt VM with no Service at all, so
its monitor selected nothing whatsoever. `metrics: true` was asserting a fact none of them
had. There was also a THIRD layer of inertness nobody had hit yet: every app namespace ships
default-deny with `ingress: []` and the renderer emitted no monitoring allow, so even a
correct port would have been dropped by policy — while renderer-contract.md claimed the
baseline carried such an allow.
FIXED IN THE GENERATOR, not in twelve rendered files:
* `slis: [up]` now renders a `<app>-down` alert against kube-state-metrics readiness of the
service's own workload object —
`kube_deployment_status_replicas_available{namespace,deployment} == 0 or absent(...)`,
`kube_statefulset_status_replicas_ready` for a StatefulSet, and
`absent(kubevirt_vmi_info{namespace,name,phase="running"})` for a VM. The same idiom
KeycloakDown has used since it was fixed. kube-state-metrics IS scraped, so this needs no
exporter and no new target. The `or absent(...)` half is load-bearing: the signal
DISAPPEARING now fires the alert instead of silencing it.
* schema 1.6.0 adds `observability.metricsEndpoint` — WHERE metrics are served. A
ServiceMonitor is rendered ONLY from it, and it renders all three halves of a working
scrape together: the named Service port, the monitor endpoint, and the
monitoring-namespace CiliumNetworkPolicy allow. No catalog entry declares one, so twelve
dead ServiceMonitors are gone. `metrics: true` keeps meaning logs-to-Loki plus
kube-state-metrics liveness.
* schema 1.6.0 also adds `observability.livenessWorkload`, and catalog/keycloak.yaml uses
it. Keycloak's server is a StatefulSet run by the operator; the default Deployment shape
would have made its `absent()` half fire FOREVER — the other failure mode this repo has
shipped.
* renderer selftest gains cases 9/9a/9b/9c: no metricsEndpoint => no monitor; the up SLI
must not read `up{job=...}`; endpoint port must exist on the Service; the metrics port
must NOT reach the Gateway allow; an unknown livenessWorkload.kind is REFUSED.
PROOF, both directions, for all twelve (apps/wave-{0,1,2}/tests/app-slis.test.yaml, run by
platform/monitoring/tests/run.sh over the RENDERED manifests):
* quiet on the healthy state; firing on no-ready-replica; firing when the series is ABSENT;
the VM asserted both gone and in a non-running phase; per-app isolation asserted, and
keycloak asserted un-maskable by its neighbours (keycloak-operator Deployment,
keycloak-db StatefulSet ready in the same namespace).
* mutation-checked: six mutations, each turning its suite red — reverting plex to
`up{job=...} == 0`; dropping seerr's `absent()` half; pointing keycloak at a Deployment;
capitalising the VMI phase to `Running`; widening keycloak's matcher to the namespace;
inverting plex's `== 0` to `!= 0`. Control run green.
* live on prod with `promtool query instant`: all ten deployed app Deployments export
`kube_deployment_status_replicas_available`,
`kube_statefulset_status_replicas_ready{namespace="keycloak",statefulset="keycloak"}` = 2,
`kubevirt_vmi_info` present with a lowercase `phase="running"`; and evaluating the twelve
SHIPPED expressions returns no series for eleven (quiet, not permanently firing). The
twelfth is vm-demo, which is not in apps/wave-0/kustomization.yaml and so is not deployed
— its rule is not applied either.
DELETION WAS NOT THE ANSWER, and the evidence is recorded rather than assumed:
kube-prometheus-stack's `kubernetes-apps` group IS live on prod (17 rules, KubePodNotReady /
KubeDeploymentReplicasMismatch among them), so app pods are not unmonitored — but that is a
generic warning keyed to a namespace, not the per-service CRITICAL SLI R-OBS-01 declares.
platform/monitoring/alerts.yaml carries no generic app-readiness rule at all.
GUARD SO IT CANNOT RECUR SILENTLY: new whole-repo L0 invariant
conformance/tests/l0-static/servicemonitor-ports-resolve. It walks the MONITORS, not the
alerts, so it needs no `up{job=...}` rule to reach one, and resolves each endpoint by SELECTOR
across the whole tree rather than by same-file adjacency. Fixing the alerts DELETED the only
thing that made the broken monitors visible to the sibling check, which is exactly why this
axis needed its own home. It declares `conformance.home.k3s/scope: whole-repo` and adds no
ci/l0.sh stage. Proven to FAIL on the pre-fix tree: 15 failures — eleven port disagreements,
vm-demo's monitor matching no Service at all, and the three generator/contract assertions —
and to pass after. Its resolver is additionally proven on in-check fixtures (agreeing pair
accepted, disagreeing pair rejected, target-less endpoint rejected), because after the fix
there are zero in-repo monitors left for it to judge and a guard whose only evidence is
"nothing failed" cannot be told apart from one that stopped working. It also asserts
r_servicemonitor does not HARDCODE a port name — the mechanism by which one edit gave twelve
services the same wrong one.
Also fixed, found on the way:
* platform/monitoring/tests/run.sh discovered the SUITE convention only inside
platform/monitoring/tests/ while documenting it as general — a suite added anywhere else
was never run and still looked like coverage. Now discovered tree-wide; a misplaced
*.test.yaml fails loudly instead of vanishing.
* KNOWN_UNWIRED_JOBS in alert-metrics-have-scrape-targets is now EMPTY, and the twelve
apps/* entries in alert-rules-unit-tested's UNTESTED_FILES are gone — both because the
gaps were CLOSED, not waived. Both registers are shrink-only and a stale entry fails.
* renderer-contract.md claimed the default-deny baseline carried a standing monitoring
scrape allow. It never did; the claim is corrected and the allow is now real, per declared
metrics endpoint and scoped to that one port.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
cd454886c7 |
docs(pm): handoff resolved — all five WIP branches merged, two self-inflicted regressions recorded
Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
72f0ba4e1f |
fix(qbittorrent): the comment warning about doubled dollars contained one
The new flux-postbuild-substitution guard failed the merge on
apps/wave-0/qbittorrent.yaml:155 — a run of two dollars in a substituted
path that nothing justifies. It was inside the postStart hook's own comment
EXPLAINING the hazard: prose about what envsubst does to a doubled dollar,
written with a literal doubled dollar.
Functionally harmless — it is a bash comment, and collapsing it changes no
behaviour — but the guard cannot distinguish a comment from content that
reaches the workload, and it is right not to try: that is precisely how the
nextcloud-db break slipped through. Reworded in words instead.
Worth recording that this is the second time today the same trap caught an
explanation of itself: my own fix for the nextcloud-db incident (
|
||
|
jakman85
|
088ff13d15 |
merge: coverage debt paid — 51/51, and the root really did hide a service
R-C2-01 and R-C2-02 were the last two uncovered requirements (49/51). Now
100% with 0 gaps, via an offline L0 check (49 assertions) that compares two
repository facts: every home-zone hostname the repo routes over HTTP must be
linked from the apex dashboard, and every link the dashboard offers must
resolve to something the repo actually serves at L7 or L4.
'A service the root hid' was still a live defect: crafty.${BASE_DOMAIN} was
served and reconciled while nothing on the root linked to it. Committed RED
first so the history shows the check catching it, then fixed.
Also fixes a regex that could never have worked: ci/touched-requirements.sh
matched R-[A-Z]+-[0-9]+, where [A-Z]+ eats the C of R-C2-01 and then demands
a dash where the 2 is — so a manifest declaring either ID attributed nothing.
Every other consumer already used [A-Z0-9]+.
16 mutations proven caught. R-TLS-02's drafted amendment is recovered into
DECISIONS-LOG: revocation is not implementable with cert-manager, and the
choice between amending the requirement and a reviewed DOCUMENTED_GAPS entry
is the owner's.
# Conflicts:
# DECISIONS-LOG.md
|
||
|
jakman85
|
cc648ba02c |
merge: qBittorrent has a durable credential, and the arrs work again
qBittorrent 5.x minted a new temporary admin password on every process start, so the Sonarr/Radarr download clients broke on the next restart — and had already broken: both reported isValid: false, Authentication Failure. A SOPS secret is now the single source of truth and a postStart hook derives qBittorrent's stored PBKDF2 form on every container start, then logs in to prove it. The hook rather than a CronJob because the failure IS a container start: qbittorrent-nox reads the credential once at start from an RWO PVC the app holds, so a 15-minute cron leaves a window after every restart where both arrs are broken. An initContainer cannot do the ground-truth login. Proven: testall flipped false -> true on both arrs, and survived a real pod deletion and recreation with the conf still holding both keys. Two bugs found in the inherited work, both the envsubst trap again: a doubled-dollar port expansion that bash read as its own PID, which made the hook report 'the WebUI rejected the credential' three times without ever asking it anything; and one shared wait budget that let the first attempt burn all 90s. Also caught that the SOPS file was never listed in apps/wave-0/kustomization.yaml, so Flux would never have applied it — the live Secret was not going to be adopted at all. |
||
|
jakman85andClaude Opus 5
|
8b36242643 |
fix(monitoring): kube-state-metrics was OOMKilled 91 times — 13 rules went intermittent
Working set measured at 111 MiB with a 6h peak of 123.3 MiB against a 128Mi limit, so it died on every spike — exit 137, roughly every 11 minutes, 91 restarts in 16h. The consequence is worse than the restarts. Thirteen alert rules read kube_* series, so each kill made them intermittently unable to fire, which is the quiet version of the inert-alert class found elsewhere today; KubePodNotReady and KubeDeploymentReplicasMismatch were both sitting pending on it. Self-inflicted: enabling customResourceState for the Velero Schedule metrics earlier today added a CRD informer plus the mandatory CRD list/watch, and that marginal cost crossed a limit already sitting about 4 MiB above the working set. Raised to 384Mi, roughly 3x the observed peak, which costs nothing on nodes reporting 18.7-22.3 GiB available. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
e4bd76070e |
fix(qbittorrent): a durable WebUI credential, and the two bugs the WIP hid
Finishes the inherited WIP. qBittorrent 5.x had no WebUI\Username or
WebUI\Password_PBKDF2 in its conf, so it minted a NEW random admin password on
every process start. Both consequences had ALREADY fired: measured before this
change, `POST /api/v3/downloadclient/testall` returned isValid:false with
"Authentication Failure" on BOTH Sonarr and Radarr, so Completed Download
Handling was silently broken, and no human had a stable login either.
Mechanism: the SOPS/age qbittorrent-bootstrap Secret is the single source of
truth; a lifecycle.postStart hook derives qBittorrent's own stored form
(PBKDF2-HMAC-SHA512, 100000 iterations, 16-byte salt, 64-byte digest) and
asserts it into the conf on EVERY container start, then logs in to the WebUI to
prove it. Level-triggered like the repo's CronJob reconcilers, but triggered by
the container start because the failure it closes IS a container start: a cron
would leave a window after every restart in which the app serves a random
password, and would have to edit an RWO PVC from outside, under a running
process. AuthSubnetWhitelist was rejected — authentication is not weakened.
Two real bugs found in the WIP, both the postBuild-substitution trap:
* PORT came from a doubled-dollar WEBUI_PORT expansion. Correct on the cluster
(envsubst collapses it) and the renderer accepts it, but run any other way
bash reads the doubled dollar as its PID, so every curl got an unparseable
URL and the hook logged "the WebUI rejected the durable credential" three
times having never asked it anything — see the marker log at 05:35Z. Now a
literal PORT=8080; the script contains no braced expansion at all and
behaves identically however it is run.
* One shared wait budget covered both the s6 wait and the API wait, so the
first consumed all 90s and attempts 2-3 fired instantly. Separate counters
now; "the API never answered" is reported as a distinct fault from "the
password was rejected"; and a rejection restarts qbittorrent-nox so a retry
can actually change its outcome.
apps/wave-0/kustomization.yaml did not list the SOPS secret, so Flux would never
have applied it and the adoption of the live-only, kubectl-client-side-apply
qbittorrent-bootstrap Secret would silently not have happened. Listed now; the
decrypted values are SHA-256-identical to the live Secret's, so adopting it
cannot change the password.
Proven on prod: testall flipped isValid false -> true on both apps; after
deleting and recreating the pod the conf still held both keys, the WebUI answered
Ok. to a login with the credential decrypted from git, and both arrs still
reported isValid:true. The post-envsubst manifest was audited as `flux build
kustomization` emits it (no doubled dollar, no braced expansion, PORT=8080,
bash -n clean) and the hook's write path was exercised in-container against a
scratch conf: exit 10 when the keys are absent, exit 0 and a byte-identical file
on a second run, digest verified by an independent PBKDF2.
NOT proven, and recorded as such in DECISIONS-LOG: the hook has not yet run as a
real postStart on prod — the Deployment is Flux-owned from main and both routes
to the live object were refused by the session's permission classifier. Verify
on merge that the marker log gains a VERIFIED line on the next pod start.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
aff80d1d7a |
docs(pm): R-TLS-02 needs the owner — the drafted spec amendment, recovered and restated
The R-TLS-02 analysis and its drafted spec amendment were written on
feat/coverage-debt-paydown (b37defe, 2026-09-07) and lived ONLY in that branch's copy of
DECISIONS-LOG.md. That branch's tree is ~107 commits behind main and must never be merged,
so the amendment went out of view with it — the second time this decision has been raised
and lost. Recovered, re-checked against today's main, and written where it cannot vanish
with a branch.
THE SUBSTANCE IS UNCHANGED. R-TLS-02 says certificates MUST be automatically "issued,
renewed, and revoked without manual steps". Issuance and renewal hold. Revocation is not
implemented and is not implementable with the stack ADR-0006 selects: cert-manager has no
revoke-on-delete, no Certificate field requesting revocation, and no controller that calls
ACME revokeCert, and neither ADR-0006 nor docs/08 specifies a mechanism.
enableCertificateOwnerRef is NOT the remedy — it garbage-collects the Secret, which is
cleanup, not revocation — and it is a global controller flag that would drop the Gateway's
wildcard-home-tls Secret on any future Certificate rename/prune, so it needs its own risk
assessment rather than a side effect of a test task. Still not set.
WHAT CHANGED SINCE THE FIRST ESCALATION, and why it now matters more: R-TLS-02 is no
longer a COVERAGE gap (l0-static/tls-lifecycle-automated + l2-behavioral/
cert-lifecycle-automated both carry the label and main counts it covered), and the L0 check
is honest about the boundary in its own verdict line ("'revoked' is NOT proven at any
layer"). But the L2 test deletes a Certificate and requires its Secret to disappear, and
its failure message says "set enableCertificateOwnerRef: true ... (owner decision — WS-TEST
does not weaken the assertion instead)". That assertion is written to FAIL on a live run
until the owner decides. The decision is now arriving as a red test rather than as a
question, which is exactly how it gets weakened by whoever meets it next.
Two options are recorded verbatim for the architect, in the drafted preference order:
amend the requirement so revocation is an explicitly manual, documented break-glass action
(which also makes enableCertificateOwnerRef a CONSEQUENCE of the requirement), or keep the
text and record a reviewed DOCUMENTED_GAPS entry naming the clause, the reason and the
removal condition.
NOT APPLIED, and not appliable by an agent: the spec repo is read-only under R-TEST-06.
This is a request for a human decision, stated plainly as one.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
a76ae3bdfa |
fix(ci): the content scan could not match R-C2-01/02 — a digit in the workstream segment
Checked as part of closing the C2 coverage gaps, and it was live on main. The content
scan's pattern was `R-[A-Z]+-[0-9]+`, which CANNOT match an ID whose workstream segment
carries a digit: `[A-Z]+` consumes the `C` of `R-C2-01` and then demands a `-` where the
`2` is. `R-C2-01` and `R-C2-02` are the only two such IDs in
conformance/requirements/registry.yaml, and the scan was blind to BOTH.
REPRODUCED before the fix, on a throwaway repo whose only change is a manifest declaring
both IDs on real (non-comment) lines, in a path the map keys to nothing:
$ REPO_ROOT=$T BASE_REF=main ci/touched-requirements.sh
R-NET-06 # the all-letter control ID was the ONLY output
It failed by UNDER-attributing, which is the fail-OPEN direction: a change that retyped a
`conformance.home.k3s/R-C2-01` tag, or a manifest self-declaring R-C2-02, contributed
those IDs to NOBODY, so the C2 tests were filtered out of the merge gate. Same defect
class as the map's `*) : ;;` arm — a miss that reads as "nothing touched".
Every other consumer in the repo already allowed the digit, which is why this was the lone
dissenter: test_requirements_for_path() twenty lines above in this same script,
conformance/lib/coverage.py, conformance/lib/status.py and conformance/lib/
run_l1_offline.py all use `R-[A-Z0-9]+-[0-9]+`. Aligned to that form, with the reason
recorded at the line.
SCOPE OF THE HOLE. path_requirements() has no arm for platform/homepage/* or
platform/headlamp/* either, so before this the ONLY route to R-C2-01/02 was detection rule
3 (editing the C2 conformance test itself). No path-map arm is added here: c2-root-surface
declares `scope: whole-repo`, so ci/l0.sh Stage 3b's --include-scope union selects it on
EVERY gate invocation regardless of the touched set — which is the right mechanism for an
invariant whose input surface is the whole tree, and strictly stronger than a two-component
path arm would be.
REGRESSION-TESTED, in ci/tests/touched-requirements.test.sh (the suite `make -C ci test`
runs): a fixture declaring R-C2-01 / R-C2-02 / R-OBS-04 from `notes/` — a path outside
path_requirements(), so the IDs can only come from the content scan — asserts both digit
IDs ARE attributed, that the all-letter form still is (control), and that a purely numeric
workstream is not invented (the pattern must not widen the gate on noise).
PROVEN TO DETECT BREAKAGE: with the pattern reverted to `R-[A-Z]+-[0-9]+` the suite reports
`FAIL (2 failed, 14 passed)` naming both C2 IDs as missing; restored, all three ci/ suites
pass (16 assertions in this one).
ALSO: conformance/tests/l0-static/whole-repo-invariants-wired/check.sh cites this pattern
in the comment explaining why its BOGUS_REQ fixture is lowercase. Citation updated so the
reasoning still reads true; the fixture is unaffected (grep -E is case-sensitive, and the
check still passes 26/26).
NOTE: another branch may be editing this same script concurrently. This is a one-line
change to one statement plus its comment — expect a trivial conflict and keep the
`[A-Z0-9]` form.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
02c58aae15 |
fix(homepage): the C2 root hid a running service — Crafty had no tile
The R-C2-01 check added in the previous commit is RED on main for a real reason.
platform/crafty/httproute.yaml serves crafty.${BASE_DOMAIN} on the shared Gateway,
clusters/base/crafty.yaml reconciles it (dependsOn platform-config, wait:false), the web
UI is Keycloak-gated by a per-pod oauth2-proxy sidecar — and nothing on the operator's
front door linked to it. The service was running and the root did not admit it existed.
R-C2-01: "The root `home.keithjackson.dev` MUST serve a UI that surfaces the services
running in the cluster and links to them." An unsurfaced route is a requirement gap, not
a missing nicety, and it is the exact failure mode nothing else in the gate notices: every
cluster test stays green, the apex answers 200, and the estate is one service larger than
the dashboard says.
This is the SAME defect the unmerged 49c160d found on 2026-09-07 (feat/coverage-debt-
paydown, ~107 commits behind main). That branch was never merged, so the gap survived —
re-fixed here against today's tree, in token form (`crafty.${BASE_DOMAIN}`, not the
literal hostname that branch predated).
Grouped under Media because Crafty manages game servers; Platform would be equally
defensible and purely cosmetic. The tile also does duty for the L4 game ports
(crafty-mc.${BASE_DOMAIN}, TCP 25565-25567 / UDP 19132-19134), which have no browsable URL
of their own — the check reports those as an advisory rather than requiring a link.
ALSO: clusters/base/homepage.yaml's postBuild comment claimed "Homepage references no
${VARS} today; the substitution is retained for parity". That is no longer true and the
claim is dangerous, because it invites someone to delete the substitution as dead weight.
This component now references ${BASE_DOMAIN} in httproute.yaml's apex hostname AND in every
tile href; without substitution the token is applied literally, the apex route binds a host
matching no listener, every link on the root is dead, and Flux still reports Ready. Comment
corrected, and the property is asserted by c2-root-surface (5) so it cannot rot back.
Verified: conformance/tests/l0-static/c2-root-surface -> check OK, 49 assertions.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
f57c411953 |
test(conformance): L0 covers R-C2-01 / R-C2-02 — 49 assertions, and it is RED
The Test DECLARATION landed in the preceding WIP commit with no check.sh, so the test
could not run at all. This is the check: 49 offline assertions over the repository as
git tracks it, no cluster, no network, no spec repo (R-TEST-06).
THE RULE. Every home-zone hostname the repository routes over HTTP must be linked from
the apex dashboard's configuration; every home-zone link the dashboard offers must
resolve to something the repository actually serves (an HTTPRoute at L7 or a
hostname-publishing LoadBalancer Service at L4); and the two R-C2-02 surfaces must be
linked, reconciled and backed by real alert rules, with operational rights arriving
per-user through OIDC groups rather than through a shared cluster-admin identity.
IT IS RED ON MAIN, on the forward direction:
FAIL (8) every service the repo ROUTES over HTTP is surfaced on the C2 root.
Unsurfaced: crafty.${BASE_DOMAIN} (platform/crafty)
platform/crafty/httproute.yaml serves crafty.${BASE_DOMAIN} through the shared Gateway,
clusters/base/crafty.yaml reconciles it, and nothing on the C2 root links to it. R-C2-01
says the root MUST surface "the services running in the cluster and links to them", so
an unsurfaced route is a requirement gap, not a styling preference. Fixed in the next
commit, deliberately separate so the RED is in the history.
WHAT IS ASSERTED BEYOND THE PRIOR ATTEMPT (49c160d on feat/coverage-debt-paydown, whose
tree is ~107 commits behind and was NOT merged):
(0) THE INPUT SURFACE ITSELF. The forward direction is only as good as the set of
manifests it reads, so three assertions guard the guard: the scanned roots are
present, every manifest in them PARSES (an unparseable file is a service the check
cannot see), and EVERY Flux-reconciled path lies inside a scanned root — so a
future tier reconciling, say, ./services fails loudly instead of letting the
dashboard hide all of it. conformance/ is deliberately excluded from the scan: its
fixtures and goldens declare synthetic HTTPRoutes (paas-demo, gw-declared-
conformance) that are test data, not services the cluster serves. The prior
attempt excluded them only by accident, through a filename prefilter.
(5) postBuild.substituteFrom ON THE DASHBOARD'S TIER. Hostnames are ${BASE_DOMAIN}
tokens now, not the literals the prior attempt compared. Without substitution the
apex route binds the literal host `${BASE_DOMAIN}` (matching no listener) and
every tile links to a host that does not exist — while Flux still reports the
Kustomization Ready. It fails OPEN, which is this repo's most-repeated incident
shape.
(12) TOKENS, NEVER LITERALS. No href may name a literal `home.keithjackson.dev` host,
so prod's front door cannot link dev's services — the 2026-09-04 class (two
clusters authored to describe the same names, prod's external-dns then deleting
dev's records), applied to the C2.
(2) THE SECTIONNAME IS PROVEN LOAD-BEARING rather than asserted from prose: the
wildcard listeners are shown to exist AND to be distinct from the apex listener,
which is why `*.${BASE_DOMAIN}` cannot serve the bare apex.
(10) THE REVERSE DIRECTION NOW ACCEPTS L4. The UniFi tile points at :11443 on a
hostname published by apps/vms/unifi/50-service-l4.yaml (a KubeVirt VM behind a
pinned Cilium LoadBalancer VIP). That is something this repo serves, so it
resolves — no off-cluster declaration is needed for it any more. The declaration
hatch is kept and constrained (specific home-zone hostnames, may not shadow a
hostname the repo does serve) but is EMPTY today, which is the honest state.
WHOLE-REPO SCOPE, as the declaration says. The invariant quantifies over every tracked
manifest that can declare an HTTPRoute; the change that BREAKS it is a route added
somewhere else entirely (`apps/wave-1/newthing.yaml` attributes R-GITOPS-02 or nothing,
never R-C2-01), and ci/touched-requirements.sh has no path-map arm for platform/homepage
or platform/headlamp at all. Requirement-scoped selection would filter this check out of
exactly the change that introduces the bug — the R-TEST-03 blind spot. No ci/l0.sh stage
is added: Stage 3b's --include-scope union selects it, and whole-repo-invariants-wired
fails the gate if anyone hand-adds one.
HONEST GAPS, in the check's own header: it does not fetch the apex (L2, covered for the
Gateway/TLS half under R-DNS-01 / R-TLS-01); (19) reads a DECLARED auth posture because
no Gateway ext-authz attachment exists anywhere in this repo, and says so in the
assertion text; it does not judge whether a tile is useful; and an L4-ONLY endpoint
(crafty-mc, a Minecraft game port) is reported as an ADVISORY rather than required,
because a raw TCP/UDP endpoint has no URL a browser can follow.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
0fe12b8332 |
WIP: L0 test declaration for R-C2-01 / R-C2-02 (no check.sh yet)
INCOMPLETE — preserved by the PM after the authoring agent died to a server error mid-task (its third attempt; the first two died to a session rate limit). Do not merge. R-C2-01 and R-C2-02 are the only 2 remaining coverage gaps on main (49/51). This is the chainsaw Test declaration only: it quotes both requirements verbatim from the spec, cites docs/12-command-center-ui.md for which surface serves which half (Homepage at the root for R-C2-01, Headlamp for R-C2-02's 'controlled operational actions where safe', Grafana for dashboards/alerts), and argues why the deciding half is L0 rather than L2 — reachability is already covered by l2-behavioral/gateway-ingress-reachable, whereas whether the root surfaces THE SERVICES THE CLUSTER RUNS is a comparison of two sets that are both repository facts. WHAT REMAINS: check.sh does not exist, so this test cannot run at all. Also still open: whether 'a service the root hid' (from the prior art commit 49c160d on feat/coverage-debt-paydown) is still a live defect, whether ci/touched-requirements.sh can attribute R-C2-01/02 at all (a digit in the workstream segment reportedly broke the content scan), and the R-TLS-02 decision summary. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
37eea51f24 |
merge: the substitution check now catches a collapsed doubled dollar
The existing check audited only the inverse — a ${...} undefined or
unescaped — so it passed the manifest that broke prod: Postgres
dollar-quoting under a postBuild.substituteFrom path, which envsubst halves,
delivering 'DO $' and a syntax error on every reconciled run (fixed
|