97257515e8
merge: a Minecraft server exists and 192.168.2.22:25565 is playable
jakman852026-09-14 02:06:54 -05:00
b4bc8e1cb9
feat(crafty): first Minecraft server — Paper 26.2 on 25565; EULA ACCEPTED ON OPERATOR'S BEHALF
jakman85andClaude Opus 52026-09-14 02:03:20 -05:00
83f13f4b0a
docs: correct my 'live outage' claim — unifi and crafty were never up
jakman85andClaude Opus 52026-09-14 01:40:26 -05:00
adda6dd779
merge: L2-announced VIPs must serve from any node, not only the backend's
jakman852026-09-14 01:38:50 -05:00
4664b41db3
fix(l4): externalTrafficPolicy Cluster on unifi + crafty L4 VIPs
jakman85andClaude Opus 52026-09-14 01:32:47 -05:00
4545611f4d
merge: Pi-hole becomes the internal LAN DNS authority
jakman852026-09-14 01:02:15 -05:00
fd43f332d6
feat(pihole): authoritative LAN DNS for the zone on udp/53 + tcp/53 at a pinned VIP
jakman85andClaude Opus 52026-09-14 00:57:30 -05:00
4080065aa7
feat(dns): exposure: internal can no longer publish a public record
jakman85andClaude Opus 52026-09-14 00:42:46 -05:00
ce7819a7da
docs: FINDING — public DNS publishes internal RFC1918 addresses, contradicting R-DNS-03
jakman85andClaude Opus 52026-09-14 00:11:30 -05:00
3de79f81a5
docs: correct three mountstats figures I supplied from bad measurements
jakman85andClaude Opus 52026-09-13 23:22:04 -05:00
5c96f4f3a7
merge: NFS per-export latency — separating a slow server from a backed-up client
jakman852026-09-13 23:21:41 -05:00
79d9ab2c8c
feat(monitoring): per-export NFS latency alerts — server-slow vs we-are-backed-up
jakman85andClaude Opus 52026-09-13 23:15:54 -05:00
de861b8672
docs: correct my own claim — node_nfs_* metrics DO exist; mountstats is the gap
jakman85andClaude Opus 52026-09-13 22:45:32 -05:00
207c939f89
merge: I/O pressure alerts now name the cause, not just the stall
jakman852026-09-13 22:45:03 -05:00
aae63c97a3
fix(monitoring): node-exporter extraArgs REPLACES the chart default, so restate it
jakman85andClaude Opus 52026-09-13 22:42:08 -05:00
3a72b07cda
chore(conformance): shrink UNTESTED_ALERTS by the two PSI detectors
jakman85andClaude Opus 52026-09-13 22:38:09 -05:00
b9f3143f42
feat(monitoring): name the CAUSE of node I/O pressure, not just the stall
jakman85andClaude Opus 52026-09-13 22:35:19 -05:00
dafa9cd241
docs: prod-1 I/O critical was not a failing disk; confirmed malware in /downloads
jakman85andClaude Opus 52026-09-13 21:55:24 -05:00
0841e2461c
docs: correct a misattribution — 192.168.2.1 is the SOHO router, not pfSense
jakman85andClaude Opus 52026-09-13 21:40:13 -05:00
8891df07b0
docs: LAN split-horizon DNS is documented but unimplemented; pfSense strips the answers
jakman85andClaude Opus 52026-09-13 21:31:03 -05:00
268b0a26b4
docs(readme): DR restore is proven on prod — Phase 7's remaining item is node-loss
jakman85andClaude Opus 52026-09-13 21:20:13 -05:00
4f2ca8c1a3
test(conformance): record the L0 verifier for R-STATE-03 / R-ENV-02
jakman85andClaude Opus 52026-09-13 21:19:19 -05:00
bd17b65db9
test(conformance): R-STATE-03 PROVEN on prod — DR restore from R2, off-cluster
jakman85andClaude Opus 52026-09-13 21:18:11 -05:00
24b8111d0c
fix(conformance): the ledger stamped every proof env=dev regardless of cluster
jakman85andClaude Opus 52026-09-13 21:13:51 -05:00
229d7f2d9c
docs: README status was stale by a whole phase; record the R-STATE-03 decision
jakman85andClaude Opus 52026-09-13 21:10:31 -05:00
a62242bd78
docs(pm): session close — three corrections to my own claims, and the remaining backlog
jakman85andClaude Opus 52026-09-13 17:54:59 -05:00
bfb4922de1
merge: the Longhorn I/O caps the runbook promised are now actually declared
jakman852026-09-13 17:53:13 -05:00
f663e7c801
fix(longhorn): the I/O caps a runbook promised were never declared — declare them
jakman85andClaude Opus 52026-09-13 17:48:42 -05:00
d96165e8fb
fix(monitoring,runbook): 5 GiB is not compaction headroom, and a runbook claimed an I/O cap that was never applied
jakman85andClaude Opus 52026-09-13 17:27:20 -05:00
52398d4426
merge: Longhorn alerting measured the wrong things, three ways
jakman852026-09-13 17:23:37 -05:00
4687521d65
fix(monitoring): delete the Longhorn rule that measured allocation and called it capacity
jakman85andClaude Opus 52026-09-13 17:19:03 -05:00
69a99e6ca6
docs(pm): two rendered manifests nothing applies — the keycloak one is a trap
jakman85andClaude Opus 52026-09-13 17:18:28 -05:00
eb2510837c
merge: twelve app SLI rules could never fire — fixed in the generator
jakman852026-09-13 17:16:16 -05:00
cf7d97e546
fix(obs): twelve app SLI rules read a signal that exists; kill the dead ServiceMonitors
jakman85andClaude Opus 52026-09-13 17:11:30 -05:00
cd454886c7
docs(pm): handoff resolved — all five WIP branches merged, two self-inflicted regressions recorded
jakman85andClaude Opus 52026-09-13 16:46:40 -05:00
72f0ba4e1f
fix(qbittorrent): the comment warning about doubled dollars contained one
jakman85andClaude Opus 52026-09-13 16:38:14 -05:00
088ff13d15
merge: coverage debt paid — 51/51, and the root really did hide a service
jakman852026-09-13 16:34:54 -05:00
cc648ba02c
merge: qBittorrent has a durable credential, and the arrs work again
jakman852026-09-13 16:34:35 -05:00
8b36242643
fix(monitoring): kube-state-metrics was OOMKilled 91 times — 13 rules went intermittent
jakman85andClaude Opus 52026-09-13 16:32:28 -05:00
e4bd76070e
fix(qbittorrent): a durable WebUI credential, and the two bugs the WIP hid
jakman85andClaude Opus 52026-09-13 16:26:27 -05:00
aff80d1d7a
docs(pm): R-TLS-02 needs the owner — the drafted spec amendment, recovered and restated
jakman85andClaude Opus 52026-09-13 16:26:54 -05:00
a76ae3bdfa
fix(ci): the content scan could not match R-C2-01/02 — a digit in the workstream segment
jakman85andClaude Opus 52026-09-13 16:25:42 -05:00
02c58aae15
fix(homepage): the C2 root hid a running service — Crafty had no tile
jakman85andClaude Opus 52026-09-13 16:21:44 -05:00
f57c411953
test(conformance): L0 covers R-C2-01 / R-C2-02 — 49 assertions, and it is RED
jakman85andClaude Opus 52026-09-13 16:20:53 -05:00
0fe12b8332
WIP: L0 test declaration for R-C2-01 / R-C2-02 (no check.sh yet)
jakman85andClaude Opus 52026-09-13 16:08:23 -05:00
37eea51f24
merge: the substitution check now catches a collapsed doubled dollar
jakman852026-09-13 16:27:51 -05:00
2f579f103b
merge: Longhorn and cert-manager metrics exist now — 13 alerts were inert
jakman852026-09-13 16:26:12 -05:00
830983b791
test(l0): prove the dollar-collapse guard fires — and name the real envsubst
jakman85andClaude Opus 52026-09-13 16:17:51 -05:00
6d13822941
merge: the gate no longer prints PASS when a stage did not run
jakman852026-09-13 16:23:37 -05:00
a92fa11701
feat(monitoring): scrape Longhorn + cert-manager — 8 alerts were structurally inert
jakman85andClaude Opus 52026-09-13 08:51:41 -05:00
cd64b4b672
fix(ci): SKIPPED is not green — the gate can no longer PASS on a stage that did not run
jakman85andClaude Opus 52026-09-13 16:15:17 -05:00
377fa205ea
WIP: catch envsubst collapsing a doubled dollar in substituted paths
jakman85andClaude Opus 52026-09-13 08:51:58 -05:00
f4f6e5f7a3
docs(pm): handoff — five WIP branches preserved after a session rate limit
jakman85andClaude Opus 52026-09-13 08:55:05 -05:00
a79693d3bd
feat(monitoring): scrape the Flux controllers — five gotk_* alerts were inert
jakman85andClaude Opus 52026-09-13 08:53:21 -05:00
16b7fcd677
fix(cnpg): the grant reconciler died on every Flux-managed run
jakman85andClaude Opus 52026-09-13 00:19:49 -05:00
5fa893095b
merge: the placeholder guard test was hiding a RED, not a pass
jakman852026-09-13 00:14:17 -05:00
e49b86eb41
merge: 35 runbooks, served, with a guard that they resolve
jakman852026-09-13 00:11:03 -05:00
d616fd4025
merge: promtool is in the toolchain, so the alert unit tests actually run
jakman852026-09-13 00:11:03 -05:00
87c3d83dff
fix(conformance): placeholder-guard-coupling actually RUNS — it was skipping forever
jakman85andClaude Opus 52026-09-13 00:10:07 -05:00
bce8778096
docs(runbooks): forward-auth IS enforced — correct the posture note
jakman85andClaude Opus 52026-09-13 00:10:24 -05:00
78d0f1b0f6
feat(ci): promtool in the pinned toolchain, one alert-rule harness, and the gaps declared
jakman85andClaude Opus 52026-09-12 23:50:32 -05:00
55b90849d1
merge: the arrs had no download client at all, and now agree on paths
jakman852026-09-13 00:01:55 -05:00
b7341b329b
fix(runbooks): declare whole-repo scope, drop the bespoke l0 stage, +cnpg-primary-spread
jakman85andClaude Opus 52026-09-13 00:00:24 -05:00
2f38b8156b
fix(arr): wire qBittorrent as a download client + Remote Path Mappings
jakman85andClaude Opus 52026-09-12 23:57:56 -05:00
44983095e1
feat(runbooks): make the 42 alert runbook_url annotations resolve
jakman85andClaude Opus 52026-09-12 23:49:19 -05:00
4865734c5f
merge: alert when a backup schedule has NEVER run, and when one is paused
jakman852026-09-12 23:49:26 -05:00
b9f8807aa6
merge: whole-repo L0 invariants are DECLARED, not hand-wired per test
jakman852026-09-12 23:49:26 -05:00
7057bbb711
fix(monitoring): alert when a Velero schedule has NEVER backed up, or is paused
jakman85andClaude Opus 52026-09-12 23:45:15 -05:00
5fa278f61c
fix(ci): declare whole-repo scope instead of hand-writing a gate stage per invariant
jakman85andClaude Opus 52026-09-12 23:39:35 -05:00
78d76fa089
merge: qBittorrent can finally write where the arrs can see it
jakman852026-09-12 23:36:48 -05:00
9e7d919d12
merge: CNPG primaries spread one per node, and the condition is now reported
jakman852026-09-12 23:36:48 -05:00
9b4744d450
fix(devcontainer): dev.sh defaults to prod — dev is decommissioned
jakman85andClaude Opus 52026-09-12 23:33:12 -05:00
78bc53afc7
feat(qbittorrent): attach the downloads area of the bulk export — the chain was broken end to end
jakman85andClaude Opus 52026-09-12 23:32:03 -05:00
505083fded
feat(cnpg): alert when >1 CNPG primary shares a node — the drift CNPG cannot prevent
jakman85andClaude Opus 52026-09-12 23:28:54 -05:00
df77affc23
merge: the nextcloud database has never been monitored
jakman852026-09-12 23:28:23 -05:00
501d587541
merge: CNPG DR alerts scoped to the cluster, and the hint that lied
jakman852026-09-12 23:28:23 -05:00
bacb4c61ee
fix(cnpg): the alert told the on-call to run the command that lies
jakman85andClaude Opus 52026-09-12 23:21:49 -05:00
7880d84bce
fix(cnpg): grant cnpg_metrics_exporter CONNECT on the nextcloud database
jakman85andClaude Opus 52026-09-12 23:22:27 -05:00
5e7ab65c65
docs: kubectl get backup lies on this cluster — ambiguous Backup short name
jakman85andClaude Opus 52026-09-12 23:15:06 -05:00
21491f901b
fix(cnpg): scope DR alerts to the CLUSTER, not the pod — stop a permanent false positive
jakman85andClaude Opus 52026-09-12 23:09:53 -05:00
ebce6dfabb
docs: the Flux->Gitea allowance is rendered and Flux-adopted — debt closed
jakman85andClaude Opus 52026-09-12 22:58:39 -05:00
eda4a9f674
merge: the Flux->Gitea allowance is rendered, not hand-applied
jakman852026-09-12 22:51:35 -05:00
d617983e83
feat(gitea): render gitea-allow-flux-ingress — prod's GitOps source, in git at last
jakman85andClaude Opus 52026-09-12 22:46:14 -05:00
1f20506829
feat(renderer): inClusterClients — authorable direct east-west ingress (schema 1.5.0)
jakman85andClaude Opus 52026-09-12 22:45:57 -05:00
a863542bf7
docs: record the apex migration, the GitOps deadlock, and the live-only policy
jakman85andClaude Opus 52026-09-12 22:37:58 -05:00
f252ecec25
fix(gitops): prod's source is the in-cluster Service, not a public name
jakman85andClaude Opus 52026-09-12 22:14:27 -05:00
jakman85
