ce7819a7da3fe03163970584a13d6bdb2349f0ca
655
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
ce7819a7da |
docs: FINDING — public DNS publishes internal RFC1918 addresses, contradicting R-DNS-03
Operator-reported, verified. external-dns runs provider cloudflare with gateway-httproute + service sources and publishes the Cilium Gateway VIP 192.168.2.20, so ~16 hostnames in a public authoritative zone answer with RFC1918 space. The spec says the opposite: external resolves to the public IP, Pi-hole serves the internal split-horizon answers, internal services are unreachable from the internet. Provenance stated plainly: the pattern predates me — external-dns always published the Gateway address, and dev did the same under owner=default. My apex migration changed which names carry it, propagating an existing flaw onto the apex rather than inventing it. Four consequences recorded: topology and service-inventory disclosure; a DNS-rebind primitive, which is why the SOHO router strips these answers and LAN clients need per-client overrides; the five services declaring exposure public are not actually publicly reachable; and seven internal services are publicly advertised. Also records why deleting the records is not a safe first step: the *.home.keithjackson.dev wildcard is gone, so removal yields NXDOMAIN rather than a fallback. Remediation sequenced, nothing started. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
3de79f81a5 |
docs: correct three mountstats figures I supplied from bad measurements
Handed an agent a baseline of 4.0-4.8 ms/op response, 0 ms/op queue, 0 major timeouts over 24h. Re-measurement found queue time is 0.013-0.067 ms/op (I read rounded output as zero); the 24h claim was really ten minutes, because the DaemonSet had just rolled and the range query returned 18 samples rather than ~2,880; and the baseline was idle, so 4.4 ms is a cached STATFS rather than real media I/O and thresholds could not be multiples of it. The 24h error is the same class as the label-values mistake recorded above: trusting a query whose window was never valid. Records two real discoveries: mounts sharing an NFS superblock share kernel RPC statistics, so prod-1's two plex mounts are one signal reported twice; and maximum_rpc_slots high-water is 586 on prod-1 against 2 on prod-2, an independent indication that prod-1 carries far more concurrent NFS work. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
5c96f4f3a7 |
merge: NFS per-export latency — separating a slow server from a backed-up client
Closes the gap the previous agent correctly left open: it could not write a rule against node_mountstats_* because the series did not exist until the collector rolled out, and the L0 guard rightly rejects a rule on an unscraped metric. 1,842 series are now live, so five rules land. The discriminator is the point. response_time is how long the SERVER held the request; queue_time is how long it waited on our node for an RPC slot. Both server rules carry 'unless queue_time/op > 5ms', so a queueing client can never be reported as a slow server, and the two server rules then partition the same slow-server state by our own op rate at 20/s — exactly one fires, no gap and no overlap. Plus major timeouts as critical, since that is hanging rather than slow. Corrects three figures I had supplied from bad measurements: client queue time is 0.013-0.067 ms/op, not zero; 'zero major timeouts over 24h' was really zero over about ten minutes, because the DaemonSet had just rolled and the range query returned 18 samples rather than ~2,880; and the baseline was an IDLE one whose 4.4 ms is a cached STATFS, so thresholds are argued from the link and the array rather than as a multiple of it. Also found that mounts sharing an NFS superblock share the kernel's RPC statistics — prod-1's plex and plex/downloads report identical rates to every digit — so one slow export raises two alerts. Documented rather than aggregated away, since aggregating would hide which path is affected. 14 mutations, all killed. Runbook triage table 10 rows -> 15, with 'not collected' stated plainly for both past incidents because the collector postdates them and that cannot be backfilled. |
||
|
jakman85andClaude Opus 5
|
79d9ab2c8c |
feat(monitoring): per-export NFS latency alerts — server-slow vs we-are-backed-up
Closes the gap `feat/io-pressure-diagnostic-context` deliberately left open. That
change enabled node-exporter's `--collector.mountstats` and recorded its own honest
limit: "an NFS export that is merely SLOW cannot be distinguished today", because
node_mountstats_* did not exist on prod until the HelmRelease rolled and the L0 guard
alert-metrics-have-scrape-targets rightly refuses a rule on an unscraped metric.
The rollout has landed. Verified on prod 2026-09-15: 1,842 node_mountstats_* series
(~0.4% of a 451,650-series head), six mounts, 426 per-operation series in each of the
five operation families and 6 in each transport family. Every metric referenced below
returns series under `promtool query instant`.
FIVE RULES in platform-health.node-io-cause, each naming its cause in `summary`:
NFSExportServerSlow response_time > 50 ms/op, our op rate > 20/s,
queue_time < 5 ms/op -> the server is slow BECAUSE OF US
NFSExportServerSlowNotOurLoad same, but our op rate 1-20/s -> the server is slow and
the load is NOT OURS. This is the operator's Proxmox
hypothesis made testable.
NFSClientQueueTimeHigh queue_time > 5 ms/op -> WE are saturating the transport
NFSTransportBacklogDeep backlog depth > 8 per send -> even the grown slot table is full
NFSExportMajorTimeouts any major timeout in 15m -> the export is HANGING (critical)
THE DISCRIMINATOR. mountstats splits a slow NFS op into the half the server owns
(response_time) and the half we own (queue_time). High response with low queue means the
SERVER; high queue means US. The two server rules then partition that state by our own op
rate at 20/s, so exactly one of them can fire and it names which reading applies. The
`unless queue_time > 5ms` clause is what licenses reading response_time as a statement
about 192.168.2.102 rather than about us, and it is pinned by fixture 5 (a client queueing
30 ms/op while the server appears to take 60 ms/op: both server rules must stay quiet).
MEASURED BASELINE, prod 2026-09-15, re-measured for this change:
server response time 4.40-4.81 ms/op (30m window), uniform across all six mounts
client queue time 0.013-0.067 ms/op; snapshot maximum 0.333 ms/op
our own op rate 0.019-0.163 ops/s — STATFS keepalive and nothing else
major timeouts 0 on every mount
backlog depth 0 on every mount
slot-table high-water 586 (prod-1), 162 (prod-3), 2 (prod-2)
THREE LIMITS ON THAT BASELINE, recorded in the rules and the runbook rather than papered
over:
(a) the history is MINUTES, not days — the series begin when the DaemonSet rolled
(count_over_time(...[24h]) returned 18, and time() - process_start_time_seconds read
558-561 s on all three nodes), so no rule uses a range longer than 15m;
(b) it is an IDLE baseline — STATFS keepalive only, no READ/WRITE traffic, so the 50 ms
threshold is argued from the link and the array (one outstanding 1 MB read at 50 ms is
20 MB/s) and must be re-cut after the first real media event. The runbook carries the
query;
(c) prod-1's /mnt/Storage/plex and /mnt/Storage/plex/downloads report byte-identical
per-operation counters (0.12443893333333335 on both, to every digit), as do prod-3's two
nextcloud mounts — mounts sharing one NFS superblock share the kernel's RPC statistics.
One slow export on prod-1 therefore raises two alerts. Documented, not aggregated away.
PROOF, not assertion. platform/monitoring/tests/nfs-export-latency.test.yaml: seven fixtures
(measured healthy; healthy under 60 ops/s of real load; server-slow-ours; server-slow-not-ours;
client-backed-up; backlog-deep; export-failing), and every fixture asserts all five alertnames
in the fires or the quiet direction. The measured baseline is pinned as a band, the
shared-superblock duplication is pinned, and both hazards the sibling recorded are asserted:
the PromQL precedence trap (`(ratio) and rate(ops) > 1 > 0.050` returns a series on a healthy
60 ops/s mount; the shipped form returns none) and the op-rate guards (a mount doing half an
operation a second at 1 s/op response, 1 s/op queue and 20 deep in the backlog — 20x, 200x and
2.5x the three thresholds — must stay silent).
MUTATION-CHECKED, 14 mutations, every one of them making the test file FAIL: each of the five
thresholds moved in both directions where the fixtures allow it, both `unless` clauses deleted,
and all four rate guards loosened. The two clause deletions fail with an exp/got alert diff,
not a parse error. No threshold here is pinned only by a promql_expr_test restating it.
Live on prod, all five shipped expressions return NO series while all four of their
sub-expressions return six each — so the silence is a verdict, not missing data.
runbooks/node-io.md: the triage table grows from 10 rows to 15 (per-export server response
time, client queue time, our own demand, major timeouts, backlog depth), with real measured
HEALTHY values and an explicit "not collected" in both incident columns, because the collector
postdates both events and that can never be backfilled. A new four-way verdict table reads rows
11-13 together and maps each combination to the alert that says it.
HONEST REMAINING GAP: these rules rule US out; they cannot rule Proxmox IN. The per-client view
of an export exists only on the server. What would settle it is one read on 192.168.2.102 while
NFSExportServerSlowNotOurLoad is firing — `nfsstat -s` and /proc/fs/nfsd/clients/*/info — which
is out of scope from this repo. Recorded in the runbook.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
de861b8672 |
docs: correct my own claim — node_nfs_* metrics DO exist; mountstats is the gap
I recorded that NFS was 'entirely unmonitored' with 'zero NFS observability',
based on a __name__ label-values query that came back empty. The query was at
fault: count({__name__=~"node_nfs.*"}) returns six live families on all three
nodes, and the nfs collector is on by default.
The real gap is narrower — node_mountstats_* is absent, and that is the one
carrying per-mount RPC round-trip time, so NFS latency specifically is
invisible while volume and retransmissions are not.
Two further signals already existed and had simply never been queried:
container_pressure_io_stalled_seconds_total, which names the stalling workload
and for tonight names sonarr; and etcd_request_duration_seconds via the
apiserver job, which the node-io runbook had explicitly claimed did not exist.
Also tempers the Proxmox hypothesis I floated: prod-1 was doing 114 NFS reads/s
of its own during the spike, so our traffic is substantial and the stall is not
obviously unexplained by us.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85
|
207c939f89 |
merge: I/O pressure alerts now name the cause, not just the stall
NodeIOPressureCritical fired on prod-1 and said nothing about why; establishing 'not a failing drive' took eight ad-hoc queries after the fact. Eight new rules each name a class in the summary — device saturation, device latency, NFS transport, etcd, not-local-disk, and which container is stalled — with the companion PromQL in a diagnostic_query annotation, a WHY row on the existing dashboard, and a 10-row triage table in runbooks/node-io.md carrying real HEALTHY / 2026-09-14 / 2026-09-07 columns. The discriminator is NodeIOPressureNotLocalDisk, written with 'unless' rather than 'and ... < T' on purpose: an empty right-hand side removes nothing, so it still fires when devices are too idle to have a measurable per-op latency — 'and <' would go silent in exactly the state it exists to detect. Acceptance is checked against both real profiles. Tonight: NotLocalDisk fires and ContainerIOStallHigh names sonarr. 2026-09-07 re-queried live at offset 164h: 1,039 ms per write on prod-1 sda, so the latency rules fire and name the device while NotLocalDisk correctly stays silent. Healthy state: all quiet. 10 mutations applied, 10 killed — one initially survived and exposed a threshold pinned only by a test that restated it. Corrects two claims: the runbook said no etcd fsync metric exists, but etcd_request_duration_seconds is present via the apiserver job and attributes per node; and container_pressure_io_stalled_seconds_total was already scraped with ~400 series and unqueried. Also fixes two bugs found in the work itself: a PromQL precedence error where 'ratio and rate(ops) > 10 > T' compares T against the op rate rather than the ratio, and node-exporter extraArgs replacing rather than appending, which would have silently dropped the chart's filesystem excludes. |
||
|
jakman85andClaude Opus 5
|
aae63c97a3 |
fix(monitoring): node-exporter extraArgs REPLACES the chart default, so restate it
Verified against the chart (helm show values prometheus-community/prometheus-node-exporter) and against the live DaemonSet's argv on prod: the subchart's extraArgs default is NOT empty — it ships the two filesystem excludes (mount-points-exclude for /dev,/proc,/sys,containerd,docker,kubelet and the fs-types-exclude list). Setting extraArgs to just --collector.mountstats would have silently dropped both, so the filesystem collector would have begun reporting every container overlay and every kubelet bind mount: a large cardinality jump, and a behaviour change in NodeDiskFillingUp, which alerts per mountpoint. Both defaults are now restated verbatim alongside the new collector, with a note to re-reconcile them on a chart upgrade. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
3a72b07cda |
chore(conformance): shrink UNTESTED_ALERTS by the two PSI detectors
node-io-cause.test.yaml's healthy fixture asserts NodeIOPressureHigh and NodeIOPressureCritical quiet on the same measured prod state the eight new cause rules are pinned against — the point of that fixture being that the detectors and the cause rules must agree about what "healthy" is. The shrink-only register entries are therefore stale, and the gate correctly refused them. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
b9f3143f42 |
feat(monitoring): name the CAUSE of node I/O pressure, not just the stall
NodeIOPressureCritical has now fired twice on k3s-prod-1 for opposite causes and
said the same thing both times. On 2026-09-14 02:40-02:45Z it fired at PSI(full)
20.98% — twice the threshold, four times the 2026-09-07 level — while every local
device was idle (0.09-0.27 utilisation) and fast (1.4-4.8 ms per write), NFS
retransmissions were zero and no Longhorn volume was degraded. Establishing that
took eight ad-hoc PromQL queries after the event had already recovered.
Adds the group platform-health.node-io-cause: eight rules, one per cause class,
each naming its cause in its own summary.
NodeDiskDeviceSaturated device saturation >0.85 busy, 10m
NodeDiskWriteLatencyHigh device latency >20ms/write, 10m
NodeDiskWriteLatencyCritical device latency >50ms/write, 5m
NodeDiskReadLatencyHigh device latency >250ms/read, 15m
NodeNFSRetransmissionRateHigh NFS transport >1% retransmitted, 10m
NodeIOPressureNotLocalDisk not local disk PSI>10% with all of the above absent
NodeEtcdRequestsSlow etcd >0.5% of etcd requests over 1s
ContainerIOStallHigh which workload per-container PSI >15%
Every threshold is measured on prod over 24-48h, not guessed, and each rule
records its headroom. The PSI thresholds themselves are deliberately unchanged:
the 21% stall was real, it fired twice in 24h for two real events on the one node
that has a problem, and raising the bar would also have raised it above the 11.36%
at which prod-1 was already losing etcd on 2026-09-07.
Context reaches the operator at page time three ways: the summary states the cause
class; every rule carries a diagnostic_query annotation with the exact companion
PromQL; and node-pressure-dashboard.yaml gains a WHY row (seven panels) plotting
saturation, write/read service time, NFS retransmissions, our own NFS op rate,
the etcd slow-request fraction and per-container PSI side by side. runbooks/node-io.md
gains a 10-row triage table with the real healthy / 2026-09-14 / 2026-09-07 columns,
which turns those eight queries into a two-minute read.
etcd and NFS metric availability, verified on prod rather than assumed:
* etcd SERVER disk metrics do NOT exist. etcd_disk_wal_fsync_duration_seconds and
etcd_disk_backend_commit_duration_seconds are absent (kubeEtcd disabled for k3s);
a query for etcd_disk_*/etcd_server_*/etcd_mvcc_* returns nothing at all.
NodeEtcdRequestsSlow uses the apiserver's client-side view of its own local
member instead, and says in its own description that it is a proxy.
* node_nfs_* DOES exist — the nfs collector is on by default, host-wide via
/proc/net/rpc/nfs, confirmed on all three nodes. It gives op counts and
retransmissions and NO latency, so it can see a dropping export but not a slow
one. The mountstats collector is enabled here to close that gap; NO rule
references node_mountstats_* yet, because the series do not exist until the
HelmRelease rolls out and the collector's visibility through the node-exporter
mount namespace must be verified on prod first. The runbook records the check
and says to revert if it comes back empty.
Proof, per the repo's two shipped alerting failure modes:
* platform/monitoring/tests/node-io-cause.test.yaml — 7 fixtures, all from
measured profiles: healthy (all 8 quiet), 2026-09-14 (NotLocalDisk fires and
names sonarr), 2026-09-07 re-queried at offset 164h (write-latency and etcd
fire, NotLocalDisk correctly silent), rebuild storm, NFS transport, read
latency, and the `unless` semantics.
* 10 mutations of the shipped rules, all killed. The mutation run found a real
gap: the etcd threshold could be changed without failing any test, because it
was only pinned by a promql_expr_test that restated the threshold in the test
file. An alert_rule_test now pins it.
* All 11 referenced metrics verified to return series on prod with
`promtool query instant`, and all 8 expressions verified quiet on today's state.
Also extends the alert-metrics-have-scrape-targets owner table for container_*
(the kubelet ServiceMonitor's cAdvisor path, which this entry already claimed in
its title) and for etcd_request_duration_seconds_* by EXACT NAME, deliberately not
by an etcd_ prefix, so a future rule on the unscraped etcd_disk_* names still fails
the gate.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
7369495b67 |
docs(pm): owner decisions — files stay, workloads stay, instrument NFS instead
Records three owner decisions and the finding that makes the third tractable. The /downloads malware stays: the Proxmox-hosted stack still has operational control of those files so deletion just triggers a re-download, nothing but the arr suite touches the directory, and there are no Windows machines anywhere — which removes the exposure I was concerned about. Written down so nobody 'discovers' and deletes them later. Anti-affinity for the co-located NFS pods declined; the concentration is accepted rather than overlooked. And the enabling finding: there are NO NFS metrics in the TSDB at all — node-exporter's nfs and mountstats collectors are off by default. A cluster with four NFS mounts carrying the media library and legacy Nextcloud data has zero NFS visibility, while PSI counts NFS stalls as I/O stalls. That is precisely why a 20.98% stall alongside idle local disks could not be explained. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
dafa9cd241 |
docs: prod-1 I/O critical was not a failing disk; confirmed malware in /downloads
NodeIOPressureCritical fired on prod-1 (peak 20.98% PSI, recovered
unattended) — the first time this alert has fired on prod, and only because
the scrape gap was closed yesterday.
Evidence rules out the 2026-09-07 drive repeat: 3.5 MB/s of writes, device
utilisation 0.09-0.21, write latency 1.4-4.8 ms, zero NFS retransmissions,
21.4 MB/s NFS throughput, no degraded Longhorn volumes, and KSM never observed
NotReady. A 21% stall on trivial volume with healthy latency is transient
blocking I/O, not hardware — and PSI counts NFS too.
Structural finding: sonarr, qbittorrent and jackett all run on prod-1, the
node with the I/O history. Same concentration class as the CNPG primaries.
Separately, the /downloads files are confirmed malware: two PE executables
('MZ') sharing one MD5, padded to ~926 MB and named as two different shows,
plus a ZIP masquerading as video. Not executed, cluster not compromised; the
risk is a Windows client opening them.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
0841e2461c |
docs: correct a misattribution — 192.168.2.1 is the SOHO router, not pfSense
I wrote that pfSense was stripping the RFC1918 answers. The operator corrected it, and this repo already documented the topology I failed to read: 192.168.2.0/24 is pfSense's WAN (clusters/prod/kustomization.yaml:458, cluster-vars.yaml:190, DECISIONS-LOG 1772), pfSense lives on 192.168.1.0/24, and 192.168.2.1 was already recorded here as 'both gateway and only IPv4 resolver' and flagged as a SPOF (entries 1318, 1468). The measurement is unchanged — that resolver returns NOERROR/ANSWER:0 for this zone while resolving cloudflare.com — but the attribution invalidated my preferred fix: configuring pfSense cannot help, because prod's resolver path never reaches it. Fixes reordered accordingly. Exposing Pi-hole's DNS to the LAN is now first: it is the architecture the repo already claims, it does not depend on a consumer router's feature set, and it retires the single-resolver SPOF this log flagged months ago. Configuring the SOHO router drops to second, since many such devices cannot express a domain override and it is outside GitOps regardless. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8891df07b0 |
docs: LAN split-horizon DNS is documented but unimplemented; pfSense strips the answers
The workstation could not reach sonarr.home.keithjackson.dev. The stale dnsmasq override pinning the zone to dev's dead 192.168.2.50 was the obvious cause, but removing it — which is what I was about to recommend — would not have fixed it: pfSense returns NOERROR/ANSWER:0 for this zone while resolving cloudflare.com fine, the signature of private-address rebind filtering, since the zone legitimately points at RFC1918 space. The underlying gap: coredns-custom.yaml defers LAN clients to 'PiHole split-horizon', but Pi-hole's Service is ClusterIP with ports 80/4180 — it serves no DNS at all. So the LAN half of split-horizon does not exist and each client silently depends on a hand-maintained file. Records the corrected override (staged for the operator's sudo) and the three proper fixes, none applied. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
268b0a26b4 |
docs(readme): DR restore is proven on prod — Phase 7's remaining item is node-loss
R-STATE-03 reads proven in --status as of 2026-09-14, the first requirement in this repo to reach that state, with both verifiers recorded on prod. Also states why the earlier PROVEN: 0 was the honest answer rather than a bug: a proof is a claim about a specific tree on a specific cluster, so coverage (51/51) and proof are two different questions. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
4f2ca8c1a3 |
test(conformance): record the L0 verifier for R-STATE-03 / R-ENV-02
A requirement reads PROVEN only when EVERY verifying test has a fresh ledger row — the state is the minimum across them, so a test with no row contributes not-derivable and one recorded verifier cannot manufacture green. R-STATE-03 carries two executable verifiers (L0 + L3). The L3 dr-restore proof landed in bd17b65; this records the L0 side, placeholder-guard-coupling, which also carries R-ENV-02. Worth noting it is the test that was permanently SKIPPED until earlier today — it was hiding a red, and now it runs and is recorded. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
bd17b65db9 |
test(conformance): R-STATE-03 PROVEN on prod — DR restore from R2, off-cluster
Phase 7's outstanding exit criterion. The only previous dr-restore proof was
recorded on the dev cluster (decommissioned 2026-09-12) against MinIO, so it
could not speak for prod or for today's Cloudflare R2 target.
Run 20260914T021505Z-24b8111, env=prod, kube v1.36.4+k3s1, chainsaw v0.2.12,
150.70s, 1 passed / 0 failed / 0 skipped. Sentinel
DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK.
What makes it a real proof rather than a green check: the test seeds a canary
on the replicated longhorn class, backs the namespace AND volume data up to
the off-cluster store, then DESTROYS every in-cluster copy — namespace, PVC
and PV — before restoring into a clean, previously-nonexistent namespace. It
asserts the restored PVC binds a DIFFERENT PersistentVolume
(pvc-f0c061e1 != pvc-4107d59d) and fails explicitly if it rebinds the
original, because that would be an in-cluster survival masquerading as a
restore. The canary came back byte-identical.
It touches no real service's namespace, PVC or PV — the scratch namespaces are
named after its own Chainsaw namespace and are destroyed by design.
This is the run whose first attempt was discarded for recording env=dev; the
recorder fix is
|
||
|
jakman85andClaude Opus 5
|
24b8111d0c |
fix(conformance): the ledger stamped every proof env=dev regardless of cluster
record_run.py defaulted --env to the literal 'dev'
(os.environ.get("CONF_ENV", "dev")) and run.sh hardcoded the same fallback
at both call sites, while CONF_ENV is set nowhere in this repo. So every
recorded proof claimed dev no matter which cluster it ran against.
Harmless while dev was the only cluster. Dev was decommissioned 2026-09-12,
and today a genuine DR-restore run against k3s-prod was recorded as
env=dev — caught before it was committed, and the record discarded rather
than hand-edited. A proof that lies about WHERE it was proven is worse than
no proof, because locating the demonstration is the ledger's whole job; it
would also have looked like a sixth dev-only row while actually being the
prod proof Phase 7 is waiting on.
env is now derived from k3s-prod (k3s-prod -> prod),
which cannot drift from the cluster the run actually talked to. An
undeterminable context records 'unknown', never a cluster name, and an
unexpected context is recorded verbatim so it is visible rather than coerced.
CONF_ENV still overrides. Recorder selftest passes.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
229d7f2d9c |
docs: README status was stale by a whole phase; record the R-STATE-03 decision
The Status section still said phases 0-6 were deployed on the DEV cluster and that Phase 7 was not open with clusters/prod 'a design-only overlay'. Dev was decommissioned on 2026-09-12 and prod has been the live cluster serving the apex since. It also claimed 36/51 requirements covered with a safety-critical gap, and that Velero was blocked on OQ-BKP. Now states Phase 7 as open with its three exit criteria and which one is actually outstanding; corrects the node inventory from the Proxmox dev IPs to the bare-metal prod nodes; records Velero as unblocked to R2; and adds the distinction the old text elided — coverage (51/51) is not proof (PROVEN: 0), because the only dr-restore and node-loss proofs were recorded on a cluster that no longer exists, against MinIO rather than R2. Also records the owner's decision that Velero+kopia to R2 is the accepted volume backup, with no Longhorn backup target, and that Longhorn config belongs in source. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a62242bd78 |
docs(pm): session close — three corrections to my own claims, and the remaining backlog
Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
bfb4922de1 |
merge: the Longhorn I/O caps the runbook promised are now actually declared
runbooks/longhorn-degraded.md asserted concurrentVolumeBackupRestorePerNode- Limit and snapshotHeavyTaskConcurrentLimit were tightened 5 -> 1 after the 2026-09-07 etcd I/O starvation. Both read 5 on prod and neither name existed anywhere in the repo. Now declared in the helmrelease defaultSettings, the path Flux actually reconciles. Both go to 1 rather than 2, and the cost is smaller than I assumed: velero's schedules run defaultVolumesToFsBackup, i.e. kopia through the node-agent, so they are not Longhorn volume backups and this setting does not bind them; CNPG's base backups are barman-level; and Longhorn has no backup target configured at all. Purge work comes from rebuild cleanup, which is already serialised to 1/node, so 1 keeps pace with its own generator while 2 would add a second heavy writer during exactly the rebuild that starved etcd. Also declares nodeDrainPolicy, which was only ever Longhorn's own default — present in no manifest and no rendered ConfigMap, so an upstream default change would have moved it silently while two runbooks reason about drain safety from it. Value unchanged. The Ansible etcd_io_priority role is deliberately NOT ported. The pathological drive it was written for has been replaced (I/O PSI 11.36 -> 3.79 under the same load class), the role was never run against any cluster, and its own README concedes it cannot separate the tenants because they share one ext4 filesystem. Swapping the elevator and halving nr_requests on the live device holding etcd's WAL is a node-wide blast radius for a low ceiling. Kept as a named operator option; the real fix remains giving etcd its own device. Removes orphanAutoDeletion, re-verified present-and-inert two ways: helm template emits no orphan key for it, and the live setting is NotFound. |
||
|
jakman85andClaude Opus 5
|
f663e7c801 |
fix(longhorn): the I/O caps a runbook promised were never declared — declare them
`runbooks/longhorn-degraded.md` told operators that, after the 2026-09-07 etcd
I/O-starvation incident, `concurrentVolumeBackupRestorePerNodeLimit` and
`snapshotHeavyTaskConcurrentLimit` had been tightened 5 -> 1. Measured live on
prod today: both read **5**, and neither name appeared anywhere in this
repository — so nothing declared them and nothing would have restored them.
Fourth instance of config that looked applied and was not.
Landed in `platform/longhorn/helmrelease.yaml` `defaultSettings` (the path Flux
reconciles; `values.yaml` is the reference mirror), NOT by `kubectl patch` —
a hand-applied Setting is exactly how this gap arose and Helm would silently
rewrite it on its next upgrade.
concurrentVolumeBackupRestorePerNodeLimit 5 -> 1
snapshotHeavyTaskConcurrentLimit 5 -> 1
replicaReplenishmentWaitInterval 600 PINNED, unchanged
backupConcurrentLimit / restoreConcurrentLimit 2 / 2 PINNED, unchanged
concurrentAutomaticEngineUpgradePerNodeLimit 0 PINNED, unchanged
nodeDrainPolicy block-if-contains-last-replica PINNED — it was in
force only as Longhorn's OWN DEFAULT, declared in no manifest and in no
rendered ConfigMap, so an upstream default change would have moved it
silently while two runbooks reasoned about drain safety from it.
EVERY KEY PROVEN TO LAND, not assumed. `helm template longhorn/longhorn
--version 1.12.1` (the version this chart range resolves to, per the
HelmRelease's own lastAttemptedRevision) on this branch's spec.values,
substituted as clusters/prod substitutes them, emits all eight into the rendered
`longhorn-default-setting` ConfigMap in kebab-case. Diffed against the LIVE
ConfigMap the change is purely ADDITIVE: 8 keys added, none altered, none lost.
The chain to the Settings is verified read-only on prod — all 115
settings.longhorn.io carry `longhorn.io/configmap-resource-version: 7917`
(= that ConfigMap's resourceVersion) with `status.applied: true`, and every key
in the ConfigMap equals its Setting's value. The two targets sit at 5 with the
key absent, i.e. Longhorn defaults, as measured.
`orphanAutoDeletion: true` REMOVED, verified inert two independent ways against
the deployed 1.12.1: the render emits no orphan key at all (Longhorn 1.9 renamed
it `orphanResourceAutoDeletion`, a list, and the chart drops unknown
defaultSettings keys with no error), and live, `setting/orphan-auto-deletion` is
NotFound while `orphan-resource-auto-deletion` reads "". Deliberately NOT
replaced — orphan deletion is extra writes on the constrained partition and buys
only space. Orphan cleanup stays manual by decision; the two runbooks that cite
the key by name are corrected rather than left to go stale.
BOTH NEW CAPS ARE 1, AND 1 COSTS SOMETHING — stated, not waved at.
`concurrentVolumeBackupRestorePerNodeLimit: 1` costs nothing TODAY: Longhorn has
no backup target (the `backup-target` Setting does not exist; still the
WS-STORAGE Phase-4 TODO), and velero's nightly run is not a Longhorn backup —
`snapshotVolumes: false` + `defaultVolumesToFsBackup: true` is kopia
file-system backup through velero's node-agent, which this Setting does not
gate; CNPG base backups are barman-level and ungated too. Its only consumer is
an R-STATE-03 restore, where serialising ~24 volumes beats losing etcd
mid-restore. Revisit when a Longhorn backup target lands.
`snapshotHeavyTaskConcurrentLimit: 1` rather than 2 because purge work is
generated by rebuild cleanup (auto-cleanup-system-generated-snapshot true, NO
RecurringJobs, 24 volumes / 24 snapshots) and rebuild is already serialised to
1 per node — so 1 keeps pace with its own generator, where 2 buys backlog drain
at the price of a second heavy writer during the exact rebuild that starved
etcd. The cost: space reclaims serially after a rebuild storm — real here (a
37.7 GiB snapshot once pinned a volume at 97.3% allocated), but instrumented by
LonghornDiskAllocationHigh and recoverable by hand, where a wedged etcd is not.
Headroom on the record: I/O PSI full is 0.30/0.62/0.37 % now, and 3.79 under a
full 18-replica rebuild after node 1's drive replacement, against a 5% warning
threshold. These caps are a BOUND, not a fix — etcd and Longhorn still share one
ext4 filesystem and nothing arbitrates the jbd2 journal fdatasync waits on.
NOT PORTED, deliberately: the same unmerged branch's node-level I/O arbitration
(`roles/etcd_io_priority` — BFQ on the shared device, nr_requests 64 -> 32,
systemd IOWeight drop-ins). The 2011 C400 it was written for was replaced on
2026-09-07 and the same load class measured 11.36 -> 3.79 I/O PSI with the node
Ready throughout; the role was never run against any cluster, its author
recorded it as unproven with a node-3-first sequenced rollout, and its own
README concedes it does not separate the two tenants. Node-wide blast radius,
low ceiling, stale measurements (node 1 reinstalled, founder moved to node 2).
It stays an operator option if I/O PSI climbs back toward 5% — which, unlike in
September, we would now be told about. Nothing under infra/ansible/ is touched.
ALSO ALREADY DONE, so skipped: that branch's NodeIOPressureHigh/Critical and the
dashboard I/O row are on main already (
|
||
|
jakman85andClaude Opus 5
|
d96165e8fb |
fix(monitoring,runbook): 5 GiB is not compaction headroom, and a runbook claimed an I/O cap that was never applied
Two things found while reviewing the Longhorn alert work. Prometheus retentionSize was 45GiB of a 50Gi PVC, under a comment already naming the right intent — 'headroom for WAL + head + compaction scratch'. 5 GiB is not that: retentionSize counts only persisted blocks, and a large compaction can need roughly the size of the blocks being merged. ENOSPC on /prometheus wedges every WAL commit, so scrapes keep succeeding while nothing is stored and every absent()/==0 alert fires falsely at once. Lowered to 36GiB, which costs no history: the TSDB is 11.2 GiB at 30d, so 30d remains the binding limit. runbooks/longhorn-degraded.md asserted that concurrentVolumeBackupRestore- PerNodeLimit and snapshotHeavyTaskConcurrentLimit 'was actually tightened 5 -> 1'. Both read 5 on prod and neither name appears anywhere in this repo — the work is on the unmerged fix/etcd-io-priority branch. The page is read during etcd I/O starvation on a node whose WAL shares a device with Longhorn, so it was telling an operator mid-incident that the cap they depend on is in place when it is not. Corrected, with the two mitigations that ARE real named and a command to verify any of them. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
52398d4426 |
merge: Longhorn alerting measured the wrong things, three ways
(1) LonghornVolumeCapacityHigh alarmed on ALLOCATION while calling itself a capacity alarm. Prometheus's TSDB volume read 0.973 while the kubelet saw the same filesystem at 0.208, because 37.73 GiB of it is a Longhorn SYSTEM snapshot — the kind taken for the full replica rebuild that prod-1's drive replacement forced — growing +152 MB/h while the filesystem stayed flat. The rule was therefore unclearable by design, i.e. permanently firing. Deleted as redundant, shown rather than asserted: 23 of 24 volumes fall under the kubelet-based filling rules, and the 24th is volumeMode Block, which the deleted rule was not covering either. (2) LonghornVolumeDegraded and LonghornVolumeFaulted could not fire even with the scrape working. longhorn_volume_robustness on 1.12.1 is ONE-HOT over a state label with values 0/1, not the 0-3 enum the rules and their comment assumed, so a faulted volume would have alerted nobody. This survived the scrape fix landing hours earlier. (3) LonghornNodeStorageHigh was wrong twice: its 0.85 threshold sat above the 75% at which storage-minimal-available-percentage=25 already removes a disk from the scheduling pool, and longhorn_node_storage_usage_bytes SUMS a node's disks, so one full disk hid behind a half-empty sibling. Replaced with per-disk LonghornDiskAllocationHigh (>0.70) and LonghornDiskUnschedulable, the latter reading Longhorn's own verdict instead of re-deriving a formula from three settings no metric exports. Baseline green with 11/11 mutations red. retentionSize 45GiB of 50Gi and discard/TRIM are both REPORTED with recommendations, not changed: TRIM would not have helped here, since remove-snapshots-during-filesystem-trim is false and the 37.7 GiB is a snapshot. # Conflicts: # DECISIONS-LOG.md |
||
|
jakman85andClaude Opus 5
|
4687521d65 |
fix(monitoring): delete the Longhorn rule that measured allocation and called it capacity
LonghornVolumeCapacityHigh went pending within minutes of the Longhorn scrape
landing today. Re-measured on prod: it read 0.9731 on the Prometheus TSDB volume
(actual_size 52243681280 / capacity 53687091200) while the kubelet reported the
SAME filesystem 20.8% used. actual_size is head + pinned snapshots, and one
user_created=false system snapshot from prod-1's replica rebuild holds 40510484480
bytes (37.7 GiB) of it; Longhorn returns no freed blocks without discard/TRIM, and
retentionSize 45GiB on a 50Gi volume targets 90% by design. The ratio is pinned
near 1.0 and cannot fall back under 0.85 — a permanently-firing alert, the same
shape as CNPGNoRecoverabilityPoint.
DELETED, not re-thresholded. Raising the threshold would tune away a true
statement about allocation. Redundancy proven, not asserted: of 24 Longhorn
volumes on prod, 23 appear in kubelet_volume_stats_inodes > 0 and so fall inside
MonitoringVolumeFillingUp (namespace="monitoring") or PersistentVolumeFillingUp /
PersistentVolumeAlmostFull. The 24th is unifi/unifi-rootdisk, volumeMode Block —
no filesystem for anyone to measure, and the deleted rule was not covering it
either.
AND A SECOND INERTNESS the scrape did not fix. longhorn_volume_robustness on
longhorn 1.12.1 is one-hot over a `state` label with values 0/1, not the 0-3 enum
the rules assumed:
count(longhorn_volume_robustness) => 96 (24 volumes x 4 states)
count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
count(longhorn_volume_robustness == 2) => (empty)
count(longhorn_volume_robustness == 3) => (empty)
So a FAULTED volume would still have alerted nobody. Re-keyed to the label; the
"44 series / 72 unknown(0, detached)" note in platform/longhorn/helmrelease.yaml
was the same misreading and is corrected in place.
The allocation question moved to where allocation is a budget. LonghornNodeStorageHigh
fired at 0.85, but storage-minimal-available-percentage is 25 on this cluster, so a
disk leaves the scheduling pool at 75% used — the old threshold could only fire
after scheduling had already been refused. Replaced by:
LonghornDiskAllocationHigh (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70, 30m
LonghornDiskUnschedulable longhorn_disk_status{condition="schedulable"} == 0, 15m
Per DISK, not per node: longhorn_node_storage_usage_bytes sums a node's disks, so
one full disk could hide behind a half-empty sibling. Measured prod usage
0.2254 / 0.2306 / 0.2369 on prod-1/2/3.
PROVEN NEITHER PERMANENTLY FIRING NOR INERT. platform/monitoring/tests/longhorn.test.yaml
adds 6 cases pinning every rule quiet on today's measured prod state and firing on
a constructed bad state, with every `for:` asserted, plus regression assertions that
the old `== 2`/`== 3` forms match nothing and that per-disk 0.72 is visible where the
node-summed 0.475 is not. Mutation-checked 11 ways; each turns the suite RED. Live on
prod: all four new expressions return empty, while the sum of their selector
cardinalities is 54. The L0 UNTESTED_ALERTS register shrinks by seven (7/56 -> 14/58
alert rules unit-tested), which also brings MonitoringVolumeFillingUp,
PersistentVolumeFillingUp and PersistentVolumeAlmostFull under test since the
redundancy claim rests on them.
REPORTED, NOT CHANGED (DECISIONS-LOG, owner decisions): retentionSize 45GiB on a
50Gi volume is too tight — recommend 36GiB, with reasoning; and discard/TRIM on the
longhorn StorageClass is feasible but not advisable now — remove-snapshots-during-
filesystem-trim reads false on prod, so trim would not have reclaimed the 37.7 GiB
anyway, and continuous discard on nodes where the etcd WAL shares the spindle points
at this cluster's known failure mode.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
69a99e6ca6 |
docs(pm): two rendered manifests nothing applies — the keycloak one is a trap
Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
eb2510837c |
merge: twelve app SLI rules could never fire — fixed in the generator
Each <app>-down rule read up{job=...} from a ServiceMonitor selecting a
Service port named 'metrics' that no app Service defined, so every one
discovered zero targets. PrometheusTargetsMissing could not see it either: a
target never discovered has no up series to be 0.
The port name was only the first of THREE layers. None of the twelve apps
serves Prometheus metrics natively at all — Plex, qBittorrent, the arrs,
Jackett, Pi-hole, Seerr, Semaphore and Nextcloud would each need an exporter,
and Gitea needs an app.ini flag. And every app namespace ships default-deny
ingress with no monitoring allow, so a correct port would still have been
dropped by policy — while renderer-contract.md claimed a baseline allow that
never existed.
Fixed in catalog/render.py (schema 1.6.0) rather than in twelve rendered
files: slis: [up] now renders <app>-down against kube-state-metrics
readiness, with an 'or absent(...)' half so the signal VANISHING fires
instead of silencing. The twelve dead ServiceMonitors are gone; a new
observability.metricsEndpoint renders the Service port, the monitor endpoint
and the scrape allow together, so the three cannot disagree again.
Deletion was rejected with evidence: kube-prometheus-stack's generic
kubernetes-apps rules are live, but they are namespace-keyed warnings, not
the per-service critical SLI R-OBS-01 asks for.
Two incidental finds: keycloak's rendered monitor was a duplicate pointing at
the wrong port, and platform/monitoring/tests/run.sh discovered test suites
only inside its own directory while documenting the convention as general —
so a suite added anywhere else was never run and still looked like coverage.
|
||
|
jakman85andClaude Opus 5
|
cf7d97e546 |
fix(obs): twelve app SLI rules read a signal that exists; kill the dead ServiceMonitors
Twelve `<app>-down` SLI rules were STRUCTURALLY INERT. Each read `up{job="<app>"} == 0`,
and the ServiceMonitor rendered beside it named `endpoints[].port: metrics` — a port NO
rendered app Service defined. Every monitor discovered ZERO targets, so no `up{job="<app>"}`
series existed and the rule could not fire. Verified on prod before this change:
`count by (job) (up)` listed 24 jobs and not one of the twelve appeared; `up{job="plex"}`
and `up{job="gitea"}` both returned nothing. `PrometheusTargetsMissing` could not see it
either — a target that was never discovered has no `up` series to be 0.
THE PORT NAME WAS NEVER THE MISSING PIECE. Ground truth per app: Plex, Pi-hole, Jackett,
qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export NO Prometheus metrics
without a separate exporter sidecar; Gitea can but only with app.ini metrics enabled;
Keycloak does, on the operator's management port, already scraped by
platform/keycloak/servicemonitor.yaml; vm-demo is a KubeVirt VM with no Service at all, so
its monitor selected nothing whatsoever. `metrics: true` was asserting a fact none of them
had. There was also a THIRD layer of inertness nobody had hit yet: every app namespace ships
default-deny with `ingress: []` and the renderer emitted no monitoring allow, so even a
correct port would have been dropped by policy — while renderer-contract.md claimed the
baseline carried such an allow.
FIXED IN THE GENERATOR, not in twelve rendered files:
* `slis: [up]` now renders a `<app>-down` alert against kube-state-metrics readiness of the
service's own workload object —
`kube_deployment_status_replicas_available{namespace,deployment} == 0 or absent(...)`,
`kube_statefulset_status_replicas_ready` for a StatefulSet, and
`absent(kubevirt_vmi_info{namespace,name,phase="running"})` for a VM. The same idiom
KeycloakDown has used since it was fixed. kube-state-metrics IS scraped, so this needs no
exporter and no new target. The `or absent(...)` half is load-bearing: the signal
DISAPPEARING now fires the alert instead of silencing it.
* schema 1.6.0 adds `observability.metricsEndpoint` — WHERE metrics are served. A
ServiceMonitor is rendered ONLY from it, and it renders all three halves of a working
scrape together: the named Service port, the monitor endpoint, and the
monitoring-namespace CiliumNetworkPolicy allow. No catalog entry declares one, so twelve
dead ServiceMonitors are gone. `metrics: true` keeps meaning logs-to-Loki plus
kube-state-metrics liveness.
* schema 1.6.0 also adds `observability.livenessWorkload`, and catalog/keycloak.yaml uses
it. Keycloak's server is a StatefulSet run by the operator; the default Deployment shape
would have made its `absent()` half fire FOREVER — the other failure mode this repo has
shipped.
* renderer selftest gains cases 9/9a/9b/9c: no metricsEndpoint => no monitor; the up SLI
must not read `up{job=...}`; endpoint port must exist on the Service; the metrics port
must NOT reach the Gateway allow; an unknown livenessWorkload.kind is REFUSED.
PROOF, both directions, for all twelve (apps/wave-{0,1,2}/tests/app-slis.test.yaml, run by
platform/monitoring/tests/run.sh over the RENDERED manifests):
* quiet on the healthy state; firing on no-ready-replica; firing when the series is ABSENT;
the VM asserted both gone and in a non-running phase; per-app isolation asserted, and
keycloak asserted un-maskable by its neighbours (keycloak-operator Deployment,
keycloak-db StatefulSet ready in the same namespace).
* mutation-checked: six mutations, each turning its suite red — reverting plex to
`up{job=...} == 0`; dropping seerr's `absent()` half; pointing keycloak at a Deployment;
capitalising the VMI phase to `Running`; widening keycloak's matcher to the namespace;
inverting plex's `== 0` to `!= 0`. Control run green.
* live on prod with `promtool query instant`: all ten deployed app Deployments export
`kube_deployment_status_replicas_available`,
`kube_statefulset_status_replicas_ready{namespace="keycloak",statefulset="keycloak"}` = 2,
`kubevirt_vmi_info` present with a lowercase `phase="running"`; and evaluating the twelve
SHIPPED expressions returns no series for eleven (quiet, not permanently firing). The
twelfth is vm-demo, which is not in apps/wave-0/kustomization.yaml and so is not deployed
— its rule is not applied either.
DELETION WAS NOT THE ANSWER, and the evidence is recorded rather than assumed:
kube-prometheus-stack's `kubernetes-apps` group IS live on prod (17 rules, KubePodNotReady /
KubeDeploymentReplicasMismatch among them), so app pods are not unmonitored — but that is a
generic warning keyed to a namespace, not the per-service CRITICAL SLI R-OBS-01 declares.
platform/monitoring/alerts.yaml carries no generic app-readiness rule at all.
GUARD SO IT CANNOT RECUR SILENTLY: new whole-repo L0 invariant
conformance/tests/l0-static/servicemonitor-ports-resolve. It walks the MONITORS, not the
alerts, so it needs no `up{job=...}` rule to reach one, and resolves each endpoint by SELECTOR
across the whole tree rather than by same-file adjacency. Fixing the alerts DELETED the only
thing that made the broken monitors visible to the sibling check, which is exactly why this
axis needed its own home. It declares `conformance.home.k3s/scope: whole-repo` and adds no
ci/l0.sh stage. Proven to FAIL on the pre-fix tree: 15 failures — eleven port disagreements,
vm-demo's monitor matching no Service at all, and the three generator/contract assertions —
and to pass after. Its resolver is additionally proven on in-check fixtures (agreeing pair
accepted, disagreeing pair rejected, target-less endpoint rejected), because after the fix
there are zero in-repo monitors left for it to judge and a guard whose only evidence is
"nothing failed" cannot be told apart from one that stopped working. It also asserts
r_servicemonitor does not HARDCODE a port name — the mechanism by which one edit gave twelve
services the same wrong one.
Also fixed, found on the way:
* platform/monitoring/tests/run.sh discovered the SUITE convention only inside
platform/monitoring/tests/ while documenting it as general — a suite added anywhere else
was never run and still looked like coverage. Now discovered tree-wide; a misplaced
*.test.yaml fails loudly instead of vanishing.
* KNOWN_UNWIRED_JOBS in alert-metrics-have-scrape-targets is now EMPTY, and the twelve
apps/* entries in alert-rules-unit-tested's UNTESTED_FILES are gone — both because the
gaps were CLOSED, not waived. Both registers are shrink-only and a stale entry fails.
* renderer-contract.md claimed the default-deny baseline carried a standing monitoring
scrape allow. It never did; the claim is corrected and the allow is now real, per declared
metrics endpoint and scoped to that one port.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
cd454886c7 |
docs(pm): handoff resolved — all five WIP branches merged, two self-inflicted regressions recorded
Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
72f0ba4e1f |
fix(qbittorrent): the comment warning about doubled dollars contained one
The new flux-postbuild-substitution guard failed the merge on
apps/wave-0/qbittorrent.yaml:155 — a run of two dollars in a substituted
path that nothing justifies. It was inside the postStart hook's own comment
EXPLAINING the hazard: prose about what envsubst does to a doubled dollar,
written with a literal doubled dollar.
Functionally harmless — it is a bash comment, and collapsing it changes no
behaviour — but the guard cannot distinguish a comment from content that
reaches the workload, and it is right not to try: that is precisely how the
nextcloud-db break slipped through. Reworded in words instead.
Worth recording that this is the second time today the same trap caught an
explanation of itself: my own fix for the nextcloud-db incident (
|
||
|
jakman85
|
088ff13d15 |
merge: coverage debt paid — 51/51, and the root really did hide a service
R-C2-01 and R-C2-02 were the last two uncovered requirements (49/51). Now
100% with 0 gaps, via an offline L0 check (49 assertions) that compares two
repository facts: every home-zone hostname the repo routes over HTTP must be
linked from the apex dashboard, and every link the dashboard offers must
resolve to something the repo actually serves at L7 or L4.
'A service the root hid' was still a live defect: crafty.${BASE_DOMAIN} was
served and reconciled while nothing on the root linked to it. Committed RED
first so the history shows the check catching it, then fixed.
Also fixes a regex that could never have worked: ci/touched-requirements.sh
matched R-[A-Z]+-[0-9]+, where [A-Z]+ eats the C of R-C2-01 and then demands
a dash where the 2 is — so a manifest declaring either ID attributed nothing.
Every other consumer already used [A-Z0-9]+.
16 mutations proven caught. R-TLS-02's drafted amendment is recovered into
DECISIONS-LOG: revocation is not implementable with cert-manager, and the
choice between amending the requirement and a reviewed DOCUMENTED_GAPS entry
is the owner's.
# Conflicts:
# DECISIONS-LOG.md
|
||
|
jakman85
|
cc648ba02c |
merge: qBittorrent has a durable credential, and the arrs work again
qBittorrent 5.x minted a new temporary admin password on every process start, so the Sonarr/Radarr download clients broke on the next restart — and had already broken: both reported isValid: false, Authentication Failure. A SOPS secret is now the single source of truth and a postStart hook derives qBittorrent's stored PBKDF2 form on every container start, then logs in to prove it. The hook rather than a CronJob because the failure IS a container start: qbittorrent-nox reads the credential once at start from an RWO PVC the app holds, so a 15-minute cron leaves a window after every restart where both arrs are broken. An initContainer cannot do the ground-truth login. Proven: testall flipped false -> true on both arrs, and survived a real pod deletion and recreation with the conf still holding both keys. Two bugs found in the inherited work, both the envsubst trap again: a doubled-dollar port expansion that bash read as its own PID, which made the hook report 'the WebUI rejected the credential' three times without ever asking it anything; and one shared wait budget that let the first attempt burn all 90s. Also caught that the SOPS file was never listed in apps/wave-0/kustomization.yaml, so Flux would never have applied it — the live Secret was not going to be adopted at all. |
||
|
jakman85andClaude Opus 5
|
8b36242643 |
fix(monitoring): kube-state-metrics was OOMKilled 91 times — 13 rules went intermittent
Working set measured at 111 MiB with a 6h peak of 123.3 MiB against a 128Mi limit, so it died on every spike — exit 137, roughly every 11 minutes, 91 restarts in 16h. The consequence is worse than the restarts. Thirteen alert rules read kube_* series, so each kill made them intermittently unable to fire, which is the quiet version of the inert-alert class found elsewhere today; KubePodNotReady and KubeDeploymentReplicasMismatch were both sitting pending on it. Self-inflicted: enabling customResourceState for the Velero Schedule metrics earlier today added a CRD informer plus the mandatory CRD list/watch, and that marginal cost crossed a limit already sitting about 4 MiB above the working set. Raised to 384Mi, roughly 3x the observed peak, which costs nothing on nodes reporting 18.7-22.3 GiB available. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
e4bd76070e |
fix(qbittorrent): a durable WebUI credential, and the two bugs the WIP hid
Finishes the inherited WIP. qBittorrent 5.x had no WebUI\Username or
WebUI\Password_PBKDF2 in its conf, so it minted a NEW random admin password on
every process start. Both consequences had ALREADY fired: measured before this
change, `POST /api/v3/downloadclient/testall` returned isValid:false with
"Authentication Failure" on BOTH Sonarr and Radarr, so Completed Download
Handling was silently broken, and no human had a stable login either.
Mechanism: the SOPS/age qbittorrent-bootstrap Secret is the single source of
truth; a lifecycle.postStart hook derives qBittorrent's own stored form
(PBKDF2-HMAC-SHA512, 100000 iterations, 16-byte salt, 64-byte digest) and
asserts it into the conf on EVERY container start, then logs in to the WebUI to
prove it. Level-triggered like the repo's CronJob reconcilers, but triggered by
the container start because the failure it closes IS a container start: a cron
would leave a window after every restart in which the app serves a random
password, and would have to edit an RWO PVC from outside, under a running
process. AuthSubnetWhitelist was rejected — authentication is not weakened.
Two real bugs found in the WIP, both the postBuild-substitution trap:
* PORT came from a doubled-dollar WEBUI_PORT expansion. Correct on the cluster
(envsubst collapses it) and the renderer accepts it, but run any other way
bash reads the doubled dollar as its PID, so every curl got an unparseable
URL and the hook logged "the WebUI rejected the durable credential" three
times having never asked it anything — see the marker log at 05:35Z. Now a
literal PORT=8080; the script contains no braced expansion at all and
behaves identically however it is run.
* One shared wait budget covered both the s6 wait and the API wait, so the
first consumed all 90s and attempts 2-3 fired instantly. Separate counters
now; "the API never answered" is reported as a distinct fault from "the
password was rejected"; and a rejection restarts qbittorrent-nox so a retry
can actually change its outcome.
apps/wave-0/kustomization.yaml did not list the SOPS secret, so Flux would never
have applied it and the adoption of the live-only, kubectl-client-side-apply
qbittorrent-bootstrap Secret would silently not have happened. Listed now; the
decrypted values are SHA-256-identical to the live Secret's, so adopting it
cannot change the password.
Proven on prod: testall flipped isValid false -> true on both apps; after
deleting and recreating the pod the conf still held both keys, the WebUI answered
Ok. to a login with the credential decrypted from git, and both arrs still
reported isValid:true. The post-envsubst manifest was audited as `flux build
kustomization` emits it (no doubled dollar, no braced expansion, PORT=8080,
bash -n clean) and the hook's write path was exercised in-container against a
scratch conf: exit 10 when the keys are absent, exit 0 and a byte-identical file
on a second run, digest verified by an independent PBKDF2.
NOT proven, and recorded as such in DECISIONS-LOG: the hook has not yet run as a
real postStart on prod — the Deployment is Flux-owned from main and both routes
to the live object were refused by the session's permission classifier. Verify
on merge that the marker log gains a VERIFIED line on the next pod start.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
aff80d1d7a |
docs(pm): R-TLS-02 needs the owner — the drafted spec amendment, recovered and restated
The R-TLS-02 analysis and its drafted spec amendment were written on
feat/coverage-debt-paydown (b37defe, 2026-09-07) and lived ONLY in that branch's copy of
DECISIONS-LOG.md. That branch's tree is ~107 commits behind main and must never be merged,
so the amendment went out of view with it — the second time this decision has been raised
and lost. Recovered, re-checked against today's main, and written where it cannot vanish
with a branch.
THE SUBSTANCE IS UNCHANGED. R-TLS-02 says certificates MUST be automatically "issued,
renewed, and revoked without manual steps". Issuance and renewal hold. Revocation is not
implemented and is not implementable with the stack ADR-0006 selects: cert-manager has no
revoke-on-delete, no Certificate field requesting revocation, and no controller that calls
ACME revokeCert, and neither ADR-0006 nor docs/08 specifies a mechanism.
enableCertificateOwnerRef is NOT the remedy — it garbage-collects the Secret, which is
cleanup, not revocation — and it is a global controller flag that would drop the Gateway's
wildcard-home-tls Secret on any future Certificate rename/prune, so it needs its own risk
assessment rather than a side effect of a test task. Still not set.
WHAT CHANGED SINCE THE FIRST ESCALATION, and why it now matters more: R-TLS-02 is no
longer a COVERAGE gap (l0-static/tls-lifecycle-automated + l2-behavioral/
cert-lifecycle-automated both carry the label and main counts it covered), and the L0 check
is honest about the boundary in its own verdict line ("'revoked' is NOT proven at any
layer"). But the L2 test deletes a Certificate and requires its Secret to disappear, and
its failure message says "set enableCertificateOwnerRef: true ... (owner decision — WS-TEST
does not weaken the assertion instead)". That assertion is written to FAIL on a live run
until the owner decides. The decision is now arriving as a red test rather than as a
question, which is exactly how it gets weakened by whoever meets it next.
Two options are recorded verbatim for the architect, in the drafted preference order:
amend the requirement so revocation is an explicitly manual, documented break-glass action
(which also makes enableCertificateOwnerRef a CONSEQUENCE of the requirement), or keep the
text and record a reviewed DOCUMENTED_GAPS entry naming the clause, the reason and the
removal condition.
NOT APPLIED, and not appliable by an agent: the spec repo is read-only under R-TEST-06.
This is a request for a human decision, stated plainly as one.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
a76ae3bdfa |
fix(ci): the content scan could not match R-C2-01/02 — a digit in the workstream segment
Checked as part of closing the C2 coverage gaps, and it was live on main. The content
scan's pattern was `R-[A-Z]+-[0-9]+`, which CANNOT match an ID whose workstream segment
carries a digit: `[A-Z]+` consumes the `C` of `R-C2-01` and then demands a `-` where the
`2` is. `R-C2-01` and `R-C2-02` are the only two such IDs in
conformance/requirements/registry.yaml, and the scan was blind to BOTH.
REPRODUCED before the fix, on a throwaway repo whose only change is a manifest declaring
both IDs on real (non-comment) lines, in a path the map keys to nothing:
$ REPO_ROOT=$T BASE_REF=main ci/touched-requirements.sh
R-NET-06 # the all-letter control ID was the ONLY output
It failed by UNDER-attributing, which is the fail-OPEN direction: a change that retyped a
`conformance.home.k3s/R-C2-01` tag, or a manifest self-declaring R-C2-02, contributed
those IDs to NOBODY, so the C2 tests were filtered out of the merge gate. Same defect
class as the map's `*) : ;;` arm — a miss that reads as "nothing touched".
Every other consumer in the repo already allowed the digit, which is why this was the lone
dissenter: test_requirements_for_path() twenty lines above in this same script,
conformance/lib/coverage.py, conformance/lib/status.py and conformance/lib/
run_l1_offline.py all use `R-[A-Z0-9]+-[0-9]+`. Aligned to that form, with the reason
recorded at the line.
SCOPE OF THE HOLE. path_requirements() has no arm for platform/homepage/* or
platform/headlamp/* either, so before this the ONLY route to R-C2-01/02 was detection rule
3 (editing the C2 conformance test itself). No path-map arm is added here: c2-root-surface
declares `scope: whole-repo`, so ci/l0.sh Stage 3b's --include-scope union selects it on
EVERY gate invocation regardless of the touched set — which is the right mechanism for an
invariant whose input surface is the whole tree, and strictly stronger than a two-component
path arm would be.
REGRESSION-TESTED, in ci/tests/touched-requirements.test.sh (the suite `make -C ci test`
runs): a fixture declaring R-C2-01 / R-C2-02 / R-OBS-04 from `notes/` — a path outside
path_requirements(), so the IDs can only come from the content scan — asserts both digit
IDs ARE attributed, that the all-letter form still is (control), and that a purely numeric
workstream is not invented (the pattern must not widen the gate on noise).
PROVEN TO DETECT BREAKAGE: with the pattern reverted to `R-[A-Z]+-[0-9]+` the suite reports
`FAIL (2 failed, 14 passed)` naming both C2 IDs as missing; restored, all three ci/ suites
pass (16 assertions in this one).
ALSO: conformance/tests/l0-static/whole-repo-invariants-wired/check.sh cites this pattern
in the comment explaining why its BOGUS_REQ fixture is lowercase. Citation updated so the
reasoning still reads true; the fixture is unaffected (grep -E is case-sensitive, and the
check still passes 26/26).
NOTE: another branch may be editing this same script concurrently. This is a one-line
change to one statement plus its comment — expect a trivial conflict and keep the
`[A-Z0-9]` form.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
02c58aae15 |
fix(homepage): the C2 root hid a running service — Crafty had no tile
The R-C2-01 check added in the previous commit is RED on main for a real reason.
platform/crafty/httproute.yaml serves crafty.${BASE_DOMAIN} on the shared Gateway,
clusters/base/crafty.yaml reconciles it (dependsOn platform-config, wait:false), the web
UI is Keycloak-gated by a per-pod oauth2-proxy sidecar — and nothing on the operator's
front door linked to it. The service was running and the root did not admit it existed.
R-C2-01: "The root `home.keithjackson.dev` MUST serve a UI that surfaces the services
running in the cluster and links to them." An unsurfaced route is a requirement gap, not
a missing nicety, and it is the exact failure mode nothing else in the gate notices: every
cluster test stays green, the apex answers 200, and the estate is one service larger than
the dashboard says.
This is the SAME defect the unmerged 49c160d found on 2026-09-07 (feat/coverage-debt-
paydown, ~107 commits behind main). That branch was never merged, so the gap survived —
re-fixed here against today's tree, in token form (`crafty.${BASE_DOMAIN}`, not the
literal hostname that branch predated).
Grouped under Media because Crafty manages game servers; Platform would be equally
defensible and purely cosmetic. The tile also does duty for the L4 game ports
(crafty-mc.${BASE_DOMAIN}, TCP 25565-25567 / UDP 19132-19134), which have no browsable URL
of their own — the check reports those as an advisory rather than requiring a link.
ALSO: clusters/base/homepage.yaml's postBuild comment claimed "Homepage references no
${VARS} today; the substitution is retained for parity". That is no longer true and the
claim is dangerous, because it invites someone to delete the substitution as dead weight.
This component now references ${BASE_DOMAIN} in httproute.yaml's apex hostname AND in every
tile href; without substitution the token is applied literally, the apex route binds a host
matching no listener, every link on the root is dead, and Flux still reports Ready. Comment
corrected, and the property is asserted by c2-root-surface (5) so it cannot rot back.
Verified: conformance/tests/l0-static/c2-root-surface -> check OK, 49 assertions.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
f57c411953 |
test(conformance): L0 covers R-C2-01 / R-C2-02 — 49 assertions, and it is RED
The Test DECLARATION landed in the preceding WIP commit with no check.sh, so the test
could not run at all. This is the check: 49 offline assertions over the repository as
git tracks it, no cluster, no network, no spec repo (R-TEST-06).
THE RULE. Every home-zone hostname the repository routes over HTTP must be linked from
the apex dashboard's configuration; every home-zone link the dashboard offers must
resolve to something the repository actually serves (an HTTPRoute at L7 or a
hostname-publishing LoadBalancer Service at L4); and the two R-C2-02 surfaces must be
linked, reconciled and backed by real alert rules, with operational rights arriving
per-user through OIDC groups rather than through a shared cluster-admin identity.
IT IS RED ON MAIN, on the forward direction:
FAIL (8) every service the repo ROUTES over HTTP is surfaced on the C2 root.
Unsurfaced: crafty.${BASE_DOMAIN} (platform/crafty)
platform/crafty/httproute.yaml serves crafty.${BASE_DOMAIN} through the shared Gateway,
clusters/base/crafty.yaml reconciles it, and nothing on the C2 root links to it. R-C2-01
says the root MUST surface "the services running in the cluster and links to them", so
an unsurfaced route is a requirement gap, not a styling preference. Fixed in the next
commit, deliberately separate so the RED is in the history.
WHAT IS ASSERTED BEYOND THE PRIOR ATTEMPT (49c160d on feat/coverage-debt-paydown, whose
tree is ~107 commits behind and was NOT merged):
(0) THE INPUT SURFACE ITSELF. The forward direction is only as good as the set of
manifests it reads, so three assertions guard the guard: the scanned roots are
present, every manifest in them PARSES (an unparseable file is a service the check
cannot see), and EVERY Flux-reconciled path lies inside a scanned root — so a
future tier reconciling, say, ./services fails loudly instead of letting the
dashboard hide all of it. conformance/ is deliberately excluded from the scan: its
fixtures and goldens declare synthetic HTTPRoutes (paas-demo, gw-declared-
conformance) that are test data, not services the cluster serves. The prior
attempt excluded them only by accident, through a filename prefilter.
(5) postBuild.substituteFrom ON THE DASHBOARD'S TIER. Hostnames are ${BASE_DOMAIN}
tokens now, not the literals the prior attempt compared. Without substitution the
apex route binds the literal host `${BASE_DOMAIN}` (matching no listener) and
every tile links to a host that does not exist — while Flux still reports the
Kustomization Ready. It fails OPEN, which is this repo's most-repeated incident
shape.
(12) TOKENS, NEVER LITERALS. No href may name a literal `home.keithjackson.dev` host,
so prod's front door cannot link dev's services — the 2026-09-04 class (two
clusters authored to describe the same names, prod's external-dns then deleting
dev's records), applied to the C2.
(2) THE SECTIONNAME IS PROVEN LOAD-BEARING rather than asserted from prose: the
wildcard listeners are shown to exist AND to be distinct from the apex listener,
which is why `*.${BASE_DOMAIN}` cannot serve the bare apex.
(10) THE REVERSE DIRECTION NOW ACCEPTS L4. The UniFi tile points at :11443 on a
hostname published by apps/vms/unifi/50-service-l4.yaml (a KubeVirt VM behind a
pinned Cilium LoadBalancer VIP). That is something this repo serves, so it
resolves — no off-cluster declaration is needed for it any more. The declaration
hatch is kept and constrained (specific home-zone hostnames, may not shadow a
hostname the repo does serve) but is EMPTY today, which is the honest state.
WHOLE-REPO SCOPE, as the declaration says. The invariant quantifies over every tracked
manifest that can declare an HTTPRoute; the change that BREAKS it is a route added
somewhere else entirely (`apps/wave-1/newthing.yaml` attributes R-GITOPS-02 or nothing,
never R-C2-01), and ci/touched-requirements.sh has no path-map arm for platform/homepage
or platform/headlamp at all. Requirement-scoped selection would filter this check out of
exactly the change that introduces the bug — the R-TEST-03 blind spot. No ci/l0.sh stage
is added: Stage 3b's --include-scope union selects it, and whole-repo-invariants-wired
fails the gate if anyone hand-adds one.
HONEST GAPS, in the check's own header: it does not fetch the apex (L2, covered for the
Gateway/TLS half under R-DNS-01 / R-TLS-01); (19) reads a DECLARED auth posture because
no Gateway ext-authz attachment exists anywhere in this repo, and says so in the
assertion text; it does not judge whether a tile is useful; and an L4-ONLY endpoint
(crafty-mc, a Minecraft game port) is reported as an ADVISORY rather than required,
because a raw TCP/UDP endpoint has no URL a browser can follow.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
0fe12b8332 |
WIP: L0 test declaration for R-C2-01 / R-C2-02 (no check.sh yet)
INCOMPLETE — preserved by the PM after the authoring agent died to a server error mid-task (its third attempt; the first two died to a session rate limit). Do not merge. R-C2-01 and R-C2-02 are the only 2 remaining coverage gaps on main (49/51). This is the chainsaw Test declaration only: it quotes both requirements verbatim from the spec, cites docs/12-command-center-ui.md for which surface serves which half (Homepage at the root for R-C2-01, Headlamp for R-C2-02's 'controlled operational actions where safe', Grafana for dashboards/alerts), and argues why the deciding half is L0 rather than L2 — reachability is already covered by l2-behavioral/gateway-ingress-reachable, whereas whether the root surfaces THE SERVICES THE CLUSTER RUNS is a comparison of two sets that are both repository facts. WHAT REMAINS: check.sh does not exist, so this test cannot run at all. Also still open: whether 'a service the root hid' (from the prior art commit 49c160d on feat/coverage-debt-paydown) is still a live defect, whether ci/touched-requirements.sh can attribute R-C2-01/02 at all (a digit in the workstream segment reportedly broke the content scan), and the R-TLS-02 decision summary. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
37eea51f24 |
merge: the substitution check now catches a collapsed doubled dollar
The existing check audited only the inverse — a ${...} undefined or
unescaped — so it passed the manifest that broke prod: Postgres
dollar-quoting under a postBuild.substituteFrom path, which envsubst halves,
delivering 'DO $' and a syntax error on every reconciled run (fixed
|
||
|
jakman85
|
2f579f103b |
merge: Longhorn and cert-manager metrics exist now — 13 alerts were inert
Neither family was scraped at all, so every Longhorn alert (degraded, faulted, capacity) and every cert-manager alert (expiry, issuance) was structurally incapable of firing: storage health unmonitored on a cluster whose node just had a failing drive, TLS expiry unmonitored with the live wildcard cert 89 days out. Now proven on prod: longhorn_* 8717 series with longhorn-backend up=3, certmanager_* 104 series across 3 targets. Adds an L0 guard (118 assertions) that crosses from each alert rule to the scrape config and re-proves each enabling declaration against the working tree, including the negatives — cert-manager's monitor mutual-exclusion, the Longhorn metrics NetworkPolicy, and kube_node_labels needing metricLabelsAllowlist. Proven able to fail five ways. Hubble is deliberately NOT enabled: it opens 9965 on every cilium-agent and rolls the CNI DaemonSet across all three nodes, and platform/cilium carries an explicit parity contract with the Ansible template that a HelmRelease-only change would silently break. Registered as a declared gap instead. Two findings this surfaced, neither fixed here and neither absorbed silently: twelve apps/* <app>-down SLI rules discover zero targets because their ServiceMonitors name port 'metrics' and no app Service defines that port; and kube-state-metrics is OOMKilled roughly every ten minutes, making thirteen kube_* rules intermittently inert. |
||
|
jakman85andClaude Opus 5
|
830983b791 |
test(l0): prove the dollar-collapse guard fires — and name the real envsubst
The implementation landed in the preceding WIP commit had never been executed. This finishes it: the arithmetic is re-measured, the guard and its allowlist are attacked, and the header's one factual error is corrected. WHAT ENVSUBST ACTUALLY IS. The header credited `github.com/drone/envsubst`. The pinned flux 2.4.0 binary in the dev image embeds `github.com/fluxcd/pkg/[email protected]` — Flux's own fork of drone/envsubst ("This package is a fork of drone/envsubst", its README) — read straight out of the binary with `grep -ao`. Same behaviour, different module; the header now says so and says how to check, because a claim about a dependency that nobody can re-derive is how the last wrong assumption survived. RE-MEASURED, NOT ASSERTED. Every row of the DOLLAR-RUN ARITHMETIC table was re-run through `flux envsubst` (flux 2.4.0, DEFINED=VALUE) and every row holds: `$$`->`$`, `$$$`->`$$`, `$${D}`->`${D}`, `$$${D}`->`$VALUE` (odd run: the brace is LIVE), `$$$${D}`->`$${D}`, `DO $$ x $$;`->`DO $ x $;`, and `$$${ARR[0]}` dies with "missing closing brace" while `$${ARR[0]}` is fine. `--strict` output was byte-identical to non-strict, exit 0 both ways (`cmp`) — strict mode is blind to this whole class, which is why a source assertion is the only guard available. The header now carries the 30-second reproduction, flagged to be re-measured on any FLUX_VERSION bump. END TO END ON THE REAL PATH. With the pre-16b7fcd manifest restored, `kubectl kustomize apps/wave-0` emits `DO $$` — valid SQL, and exactly what the `kubectl apply` verification saw — while the same stream through `flux envsubst --strict` emits `DO $` and `$;` and exits 0. That two-command contrast IS the incident. THE GUARD IS NOT TRIVIALLY SATISFIABLE — 16 adversarial cases, all as intended, run against a throwaway copy of the tree so nothing could be left behind: * an unmarked `$$` under a substituted path FAILS (DOLLAR-COLLAPSE); so do an odd `$$${...}` (DOLLAR-HALF-ESCAPED) and an even `$$$${...}` (OVER-ESCAPED); * a stale entry FAILS three ways — context no longer in the file, context that only covers an already-accepted `$${` escape and so excuses nothing, and a path that is not a file in the repo; * the allowlist cannot blanket-exempt a path: a 2-char context is rejected (<24), a `.*` wildcard is matched VERBATIM and therefore rejected as stale, a 24-char context carrying no dollar run is rejected, a 4-char reason is rejected (<40) — and, the central case, a genuine entry covers only the span its context matches, so a SECOND `$$` elsewhere in the same file still fails. Two genuine entries for two real occurrences do pass, so the mechanism is usable rather than merely unsatisfiable; * the CLASS, not the instance: the pre-fix `DO $$ ... $$;` reconstructed in a NEW ConfigMap under apps/wave-2 fails at both dollar runs, and restoring the real pre-16b7fcd apps/wave-0/nextcloud-db.yaml verbatim fails the check at lines 384 and 391. The guard rejects the exact commit that broke production. The scratch fixtures lived only in the throwaway copy; `git status` on this branch stayed clean throughout, which is the other half of the last attempt's unfinished business. LEGITIMATE USES EXIST AND STAY GREEN. 18 `$${...}` escapes (unifi cloud-init, the gitea and nextcloud OIDC bootstrap scripts, two longhorn comments) pass unremarked, and the 2 non-braced `$$` occurrences are the two allowlist entries. Those two are shell comments inside block scalars, and `kubectl kustomize` confirms the comment text really does survive into the stream envsubst reads — so they are genuine, reaching content whose collapse is cosmetic, not a hypothetical the allowlist was invented to excuse. STILL NOT PROVEN, AND STILL NOT PROXIED: that a change was verified through a substituting path. Which command a human ran is a fact about a shell session; git stores the manifest, never the invocation, so it is not statically decidable here and no honest formulation exists. Grepping commit prose for `kubectl apply` would be satisfiable by rewording and would fire on the break-glass runbooks whose whole content is that command. The answer is to move the proof off the verification path and onto the source, which is what assertion (7) does. No ci/l0.sh stage: the `conformance.home.k3s/scope: "whole-repo"` annotation is what wires it, and whole-repo-invariants-wired fails the gate on a hand-written stage. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
6d13822941 |
merge: the gate no longer prints PASS when a stage did not run
ci/gate.sh set BLOCK=1 on a FAILED stage but only warned on a SKIPPED one, so it printed the words 'not green' and then 'MERGE GATE: PASS', exit 0 — fail open in the one component whose job is to fail closed. Observed live with prom-rules, and independently with l0 when the pinned kyverno CLI was absent. New contract: PASS/0 means every stage that was supposed to run ran and was green; BLOCKED/1 a failure or a must-run stage that did not; UNDETERMINED/3 nothing failed but greenness was not established; NOT APPLICABLE/0 unchanged. No knob inside gate.sh can turn UNDETERMINED into 0 — tolerating a non-answer is the caller's decision to make and to document. The distinction that makes it usable: a capability the environment DECLARES absent may still pass (no cluster, named in the headline); a stage that COULD NOT RUN may not. CI declares GATE_STRICT_SKIPS in the workflow rather than sniffing for CI, so a missing pinned tool there is exit 1. Carries over 47068ee's unusable-subset blocking from the unmerged fix/touched-requirements-fail-closed, now with tests. Measured its companion claim honestly: 159 of 613 paths (26%) are unmapped today, not 255 of 476 (54%) — 56 prose, 103 non-prose — which remains the larger fail-open and is recommended as its own change rather than smuggled in here. 47 assertions, 32 of which fail against the old gate.sh. |
||
|
jakman85andClaude Opus 5
|
a92fa11701 |
feat(monitoring): scrape Longhorn + cert-manager — 8 alerts were structurally inert
An alert rule whose metric is never scraped is not a broken alert, it is an ABSENT
alert that reads as a present one. `longhorn_volume_robustness == 2` over a TSDB
with no longhorn_* series returns no vector: permanently quiet, permanently
"healthy", indistinguishable from a correct rule. It survives review (the PromQL is
right), it survives `promtool test rules` (the unit test supplies the series
itself), and it appears on no dashboard.
VERIFIED ABSENT ON PROD before this change (re-verified 2026-09-13 21:0xZ, ~15h
after the first audit, because a stale reading is not evidence):
count(longhorn_volume_robustness) => (empty) 4 rules inert
count({__name__=~"certmanager.*"}) => (empty) 3 rules inert
count(hubble_drop_total) / {__name__=~"cilium.*"} => (empty) 1 rule inert
count by (job) (up) => 20 jobs, none of them
Longhorn, cert-manager,
Cilium or Hubble
Consequence: storage health was unmonitored on the cluster whose prod-1 had just
had a failing SSD, and TLS expiry was unmonitored with the live `wildcard-home`
wildcard 89 days from expiry on an ACME DNS-01 renewal that can fail silently.
WHAT THIS CHANGES
* platform/longhorn: `metrics.serviceMonitor.enabled: true` (the supported chart
way — rendering longhorn 1.12.1 emits exactly 1 ServiceMonitor with the flag and
0 without), plus networkpolicy-metrics.yaml. The NetworkPolicy is LOAD-BEARING,
not belt-and-braces: measured with the monitor applied and no allowance, all 3
targets were DISCOVERED and every scrape failed `context deadline exceeded`
after 10.0s — a drop, not a refusal — while curling the endpoint from inside
the manager pod returned 200 in 0.031s. Longhorn 1.12.1 ships 6 NetworkPolicies
of its own, guarded on `networkPolicies.restrictInternalTraffic` (default TRUE)
and NOT on `networkPolicies.enabled`, so they are effectively unconditional and
the manager policy admits only same-namespace Longhorn components. The one-line
alternative (restrictInternalTraffic: false) was rejected: it deletes the
internal-traffic restriction across all six components to open one port to one
namespace, trading an R-SEC-02 control for an observability one when the two do
not conflict. This policy is Ingress-only, ns monitoring -> app=longhorn-manager,
TCP 9500 only; 9502/9503 (webhook, recovery backend) stay closed.
* platform/cert-manager: `prometheus.servicemonitor.enabled: true`. Chart default
already serves 9402; only the monitor was missing. No NetworkPolicy needed —
neither ns cert-manager nor ns monitoring carries one. NOT paired with
prometheus.podmonitor.enabled: the chart calls `fail` if both are set.
* platform/kubevirt: KubeVirtVMLaunchFailing queried
`kubevirt_vmi_phase_count{phase="Failed"}`, which DOES NOT EXIST on the deployed
KubeVirt — a different flavour of the same defect, with the scrape target up and
healthy. Prod has only `kubevirt_vmi_info` and `kubevirt_vmi_number_of_outdated`;
the only phase-bearing names are the phase_transition_time histograms. Corrected
to `kubevirt_vmi_info{phase="failed"}` — LOWERCASE, because KubeVirt lowercases
the value (observed `phase="running"` for the unifi VMI) and a capital-F matcher
would select nothing and be exactly as inert.
PROVEN, NOT ASSERTED. The rendered ServiceMonitors were applied to prod by hand and
the metrics observed to land, then REMOVED again so the merge's Helm upgrade creates
and owns them rather than colliding with an unowned object:
count(up{job="longhorn-backend"}) => 3 (all three managers, up==1)
count({__name__=~"longhorn.*"}) => 8717 (was empty)
count(longhorn_volume_robustness) => 44
up{namespace="cert-manager"} => 3 targets, all 1
count({__name__=~"certmanager.*"}) => 104 (was empty)
wildcard-home days to expiry => 89.2, ready_status{condition="True"}=1
All 58 alert rules in the repo were then evaluated against prod one by one: every
one returns a result without error, and exactly ONE fires.
TRIAGE OF THE ONE NEWLY-FIRING ALERT — A TRUE POSITIVE, NOT TUNED AWAY.
LonghornVolumeCapacityHigh on the Prometheus TSDB volume:
`longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes` = 0.971
(48.6 GiB allocated of 50 GiB) while the kubelet's view of the same filesystem is
only 20.4% used. Longhorn is not reclaiming freed blocks, so ~48.6 GiB x 3 replicas
is committed for ~10 GiB of real data. On a cluster whose failure mode is disk
pressure and whose etcd WAL shares a spindle with Longhorn, that is worth knowing
and was invisible until now. The threshold is left alone; the follow-up is
filesystem trim, not a bigger number.
THE GUARD (conformance/tests/l0-static/alert-metrics-have-scrape-targets)
Whole-repo L0 invariant, 118 assertions. It crosses from each alert to the scrape
config that makes its metric exist: every metric name is claimed by exactly one
declared owner, and each owner's enabling declaration is re-proven against the
working tree on every gate run, including the negative conditions that defeat an
enablement without deleting it. `up{job="X"}` is audited on its own axis, because
`up` always exists: where this repo authors a Service and a ServiceMonitor in the
same file, the monitor's endpoint port must be a port that Service defines.
It DECLARES conformance.home.k3s/scope: "whole-repo" and adds NO ci/l0.sh stage —
the whole-repo-invariants-wired meta-test fails the gate if one reappears.
PROVEN ABLE TO FAIL, five ways, each reverted:
1. an alert on a fabricated metric -> FAIL "claimed by NO entry"
2. longhorn metrics.serviceMonitor removed -> FAIL "no longer matches"
3. networkpolicy-metrics.yaml unwired -> FAIL "the monitor alone is NOT enough"
4. cert-manager podmonitor also enabled -> FAIL "FORBIDDEN pattern"
5. a register entry made stale / removed -> FAIL in both directions
TWO SHRINK-ONLY REGISTERS, because a blanket fail would make the gate red until
work that is not this change lands, and a disabled gate protects nothing:
* KNOWN_UNWIRED_METRICS (1): hubble_drop_total. See below.
* KNOWN_UNWIRED_JOBS (12): every apps/* `<app>-down` SLI rule is ALSO inert, and
this was found by the re-audit, not previously known. The rendered
ServiceMonitors name `port: metrics` and NOT ONE of the twelve app Services
defines a port of that name (zero occurrences of `name: metrics` across all
twelve manifests); prod confirms none of the twelve jobs appears in `up`. This
is the nastiest variant of the class: `up` exists in abundance so nothing looks
wrong, and PrometheusTargetsMissing cannot see it either, because a target that
was never discovered has no up series to be 0. Closing it needs an exporter per
app (Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr speak no
Prometheus) — except keycloak, where the live jobs are keycloak-service /
keycloak-discovery and the rule is a rename away from being live.
NOT DONE HERE, DELIBERATELY: Hubble. hubble.metrics.enabled RECONFIGURES THE CNI —
it opens 9965 on every cilium-agent and rolls the DaemonSet across all three nodes,
serialised by this repo's own maxUnavailable: 1 (set after the 2026-09-01
k8sServiceHost outage, where a CNI roll deadlocked 2 of 3 agents). It also needs a
matching edit to roles/cilium/templates/cilium-values.yaml.j2 in the same commit,
because platform/cilium/helmrelease.yaml carries a [parity] contract against it and
a rebuilt node would otherwise bootstrap a CNI config that disagrees with the
reconciled one. Recommended as its own change on the cilium Kustomization, with the
Ansible parity edit, a narrow labelsContext, and a one-node canary — not smuggled in
here.
ALSO FOUND, NOT FIXED HERE: kube-state-metrics is OOMKilled every ~10 minutes on
prod (90 restarts in 16h against a 128Mi limit) while its logs show the
custom-resource-state factory reloading every 3 seconds. Thirteen alert rules read
kube_* metrics, so those rules are INTERMITTENTLY inert — the same defect class,
arriving through a crashloop rather than a missing monitor. Needs its own change.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
cd64b4b672 |
fix(ci): SKIPPED is not green — the gate can no longer PASS on a stage that did not run
Completes the WIP preserved in the previous commit. `MERGE GATE: PASS` + exit 0 now means
what it says: every stage that was SUPPOSED to run ran, and was green.
THE DEFECT. In gate.sh's decision section a FAILED stage set BLOCK=1 but a SKIPPED stage only
warned, so the gate printed `l0: skipped (unwired seam / missing tool) — not green` and then
`MERGE GATE: PASS`, exit 0. Seen twice for real: `prom-rules` skipped because promtool was
absent, and `l0` skipped because the pinned kyverno CLI was. Exit 0 is what CI and every
`||`-chained script read as merge-approved, so the one component whose entire job is to fail
closed was failing OPEN — in a repo whose stated principle is that SKIPPED IS NOT GREEN, and
whose history already contains the 2026-08-31 "local false green" that UNDECIDED was added to
prevent. The offline stages simply never used that vocabulary.
A SKIP IS NOT ONE THING — the per-stage rule, with the justification in the file header:
* lint / l0 / render-golden / l1 — COULD NOT RUN (a pinned tool absent, a seam unwired).
BLOCKED under strict skips (a CI runner has the toolchain installed for it by
install-tools.sh, so a skip there is a defect); UNDETERMINED (exit 3) otherwise — a
workstation missing a tool is a real local condition, still non-zero, still never a pass.
* spec-immutability — the spec repo is not checked out, so R-TEST-06 COULD NOT BE VERIFIED:
UNDETERMINED. Deliberately NOT upgraded to a block by strictness, because the live runner
does not check the spec repo out at all and blocking would leave the only enforced status
check permanently red for a reason no PR can clear. It is the ONE skip a caller may
knowingly tolerate; REQUIRE_SPEC_VERIFICATION=true makes it hard once wired.
* conformance L2/L3 — CLUSTER_AVAILABLE!=true is the environment DECLARING the capability
absent, not a stage failing to run where it could have. A declared-inapplicable stage was
never "supposed to run", so it does not deny a PASS — but it is now NAMED in the PASS
headline and the step summary, so PASS never implies more than was verified. A skip
DESPITE CLUSTER_AVAILABLE=true still blocks.
declared-not-applicable may pass; could-not-run may not.
STRICTNESS IS DECLARED, NOT SNIFFED. .gitea/workflows/gate.yml sets GATE_STRICT_SKIPS=true on
the gate step — the workflow is the thing that knows it is CI — with GITHUB_ACTIONS/CI (set by
the runner itself) only as the `auto` fallback. No hostname/path/tool-availability heuristic.
A wrong answer in either direction cannot fail open: both branches are non-zero and neither is
a PASS. That step now calls ./ci/gate.sh directly and tolerates exit 3 in one visible line,
because under strict skips exit 3 can ONLY mean the unverifiable spec repo — a missing tool is
exit 1 and untolerated. Tolerating a non-answer is a reviewed choice in the CALLER; there is
deliberately no knob inside gate.sh that turns UNDETERMINED into exit 0.
ALSO, from the unlanded fix/touched-requirements-fail-closed (47068ee), whose reasoning is
carried over rather than its tree: gate.sh special-cased only exit 3 from
touched-requirements.sh, so ANY other non-zero exit left the touched set empty and printed PASS
having applied no coverage gate at all. An unusable subset is not an empty subset: it BLOCKS.
Latent today (the script exits only 0 or 3) and wired now so the hole is already closed when a
fail-closed path map makes exit 1 reachable — the measured state of that map is in the PM report.
TESTED — ci/tests/gate-decision.test.sh, 47 assertions over the whole decision table: the real
gate.sh and lib.sh run in a throwaway CI_DIR against STUB stage scripts whose exit codes the
suite dictates, so it asserts how the gate REASONS, with no tools, no cluster, no network. All
stages green -> PASS/0; each stage FAILED -> BLOCKED/1; each stage SKIPPED -> never PASS and
non-zero; strict skips -> BLOCKED/1; the spec exception; the declared-gap PASS; the
undetermined-touched-subset paths unchanged (on-base -> NOT APPLICABLE/0, no-base and
diff-failed -> BLOCKED/1); UNDETERMINED outranking NOT APPLICABLE; an unusable subset ->
BLOCKED/1; truthy()'s misspelling direction. 32 of the 47 FAIL against the pre-change gate,
including "l0 SKIPPED: printed 'MERGE GATE: PASS'" — the suite is falsifiable, not decorative.
lib.sh: truthy() is now the one spelling of "is this switch on" and cluster_available() uses it
(the comment claimed that; now it is true), and load_touched_requirements' rc!=0 branch says
what actually happened instead of "proceeding with the full layer".
Refs R-TEST-02, R-TEST-03, R-TEST-06.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
377fa205ea |
WIP: catch envsubst collapsing a doubled dollar in substituted paths
INCOMPLETE — preserved by the PM after the authoring agent hit the session
rate limit mid-task. Do not merge without finishing and gating it.
Closes the inverse of what this check already covered. It asserted that every
${...} in a substituted path is defined or escaped; it did NOT catch content
containing a literal doubled dollar that the author needs delivered verbatim
and that envsubst silently halves. That shipped a broken CronJob (fixed in
|
||
|
jakman85andClaude Opus 5
|
63bc890261 |
WIP: the gate must not print PASS when a stage was skipped
INCOMPLETE — preserved by the PM after the authoring agent hit the session
rate limit mid-task. Do not merge without finishing and gating it.
Problem: in ci/gate.sh a FAILED stage sets BLOCK=1 but a SKIPPED stage only
warns, so the gate prints the words 'not green' and then concludes
'MERGE GATE: PASS' and exits 0. Observed live tonight with prom-rules, and
independently with l0 when the pinned kyverno CLI was absent. That is
fail-open in the one component whose whole job is to fail closed.
The agent was mid-way through adding truthy/strict_skips helpers to ci/lib.sh.
WHAT REMAINS:
- Per-stage justification for which skips must BLOCK and which may degrade
to a non-zero UNDECIDED (spec-immutability and L2/L3 skip for legitimate
environmental reasons and must stay usable locally).
- Whether CI and local can be distinguished reliably here.
- New ci/tests/ suite for the decision logic, and a before/after proof.
- Reconciling with the unmerged branch fix/touched-requirements-fail-closed,
whose 47068ee already makes an unusable touched subset block, and whose
56c242d claims 255 of 476 paths were unmapped in the path->requirement
map — a far larger fail-open that still needs measuring on current main.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
204f0ec28e |
WIP: durable qBittorrent WebUI credential
INCOMPLETE — preserved by the PM after the authoring agent hit the session
rate limit mid-task. Do not merge without finishing and gating it.
Problem: qBittorrent 5.1.2 has no WebUI\Username or WebUI\Password_PBKDF2 in
its config, so it mints a NEW temporary admin password on every process start
and announces it only on stdout. Two consequences, both real: the Sonarr and
Radarr download clients configured earlier will silently stop working at the
next pod restart, and a human operator has no stable login either.
Adds a SOPS-encrypted qbittorrent-bootstrap Secret (verified encrypted: both
values are ENC[AES256_GCM]) plus the catalog/rendered wiring to consume it.
WHAT REMAINS:
- The agent's last note: the postStart hook works, but its output never
reaches kubectl logs, so it was switching to the marker-file logging
pattern gitea's hook already uses. That change is NOT here.
- The Secret exists on prod as kubectl-client-side-apply, i.e. live-only;
merging adopts it into git.
- Sonarr's and Radarr's stored download-client passwords were to be updated
to the durable value — NOT done, so the chain may break on restart until
it is.
- The required proof is unfinished: restart the pod, then show the WebUI
accepts the credential AND both arrs still report isValid: true.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
f4f6e5f7a3 |
docs(pm): handoff — five WIP branches preserved after a session rate limit
Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a79693d3bd |
feat(monitoring): scrape the Flux controllers — five gotk_* alerts were inert
kube-prometheus-stack does not scrape flux-system: the controllers are
bootstrapped, expose no Service, and no chart value covers them. So every
gotk_* alert rule in platform/monitoring/alerts.yaml could never fire —
verified on prod, count({__name__=~"gotk.*"}) returned nothing. A repo whose
entire delivery model is Flux had no alerting on reconciliation failing, a
Kustomization erroring, or a resource left suspended.
Proven live before landing: with this PodMonitor applied, gotk_suspend_status
returns 39 series and gotk_reconcile_condition 117, and FluxResourceSuspended
actually fired on a deliberate test suspension — evidence the rule is live,
not merely present.
Extracted from the wider inert-alert audit on
fix/scrape-targets-for-inert-alerts, which stalled unfinished. This is the
self-contained, verified half; Longhorn and cert-manager metrics are still
ABSENT and their alerts remain inert. It also adopts the object into git: it
existed on prod only as kubectl-client-side-apply, so a rebuild would have
lost it.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
16b7fcd677 |
fix(cnpg): the grant reconciler died on every Flux-managed run
The SQL asserted convergence in a dollar-quoted PL/pgSQL DO block. That cannot survive this delivery path: the ConfigMap is rendered through Flux's postBuild substitution, and envsubst collapses a doubled dollar into one, so Postgres received a half-quoted DO and failed with a syntax error. Three scheduled runs had left nine Error pods in nextcloud-db. It passed verification because it was hand-applied with kubectl, which runs no envsubst — the defect only existed once Flux owned the object. The GRANT on the first line still succeeded, so the privilege held and the blind spot stayed closed; what was broken was the assertion meant to prove it, which is the part that makes a silent regression visible. Rewritten in plain SQL with no dollar sign anywhere in the file, comments included. Escaping by doubling again would also work and would be the next person's trap. has_database_privilege()::int is 1 or 0, so the division raises division_by_zero and ON_ERROR_STOP=1 fails the Job; the divisor is not constant so it cannot be folded away at plan time. Co-Authored-By: Claude Opus 5 <[email protected]> |