045d81c7a272d1a6f9f149771fdb210a3d90799a
109
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
72a46ad82d |
chore(log): the handoff log was serialising every parallel branch
DECISIONS-LOG.md was append-only in ONE file and every agent appended to its
tail, so any two concurrent branches collided on the same last hunk. Measured
against the live Gitea today: four of the five open PRs were mergeable=false,
and `git merge-tree` named DECISIONS-LOG.md as the ONLY conflicting path in
all four.
#22 fix/crafty-oauth2-proxy-signin CONFLICT: DECISIONS-LOG.md
#20 feat/conformance-job-template CONFLICT: DECISIONS-LOG.md
#19 feat/conformance-trigger-kubeconfig CONFLICT: DECISIONS-LOG.md
#18 feat/conformance-runner-image CONFLICT: DECISIONS-LOG.md
#17 feat/ci-is-the-approver clean — it wrote no handoff
Nothing was wrong with any of those changes. The log's storage format
serialised work that was otherwise parallel, and the cost grew with every
concurrent branch (N branches -> N-1 rebases, each invalidating the gate run
before it). The agents work in isolated worktrees precisely so they do not
block each other; this one file undid that.
So: stop sharing a file. Handoffs are now one-per-file under decisions/,
named YYYY-MM-DD-<slug>.md. Disjoint paths have nothing to merge. There is
deliberately no index file (a shared mutable file is the thing that just
failed) and no counter (two agents would race for the same number) — the
directory listing is the index.
DECISIONS-LOG.md is frozen as the pre-2026-09-29 archive, its SHA-256 pinned
in decisions/ARCHIVE.sha256, and a new gate stage rejects any change to it.
That is a guard and not a paragraph because the old habit is written into the
runbooks and into every agent that has read the log, and getting it wrong does
not fail: it produces a clean-looking commit that re-creates the conflict on
OTHER branches, later, misattributed.
ci/decisions-log.sh new stage: archive pinned; every entry named with
a real date, carrying date/branch/title frontmatter
that agrees with its filename, and all three of
CLAUDE.md §7's sections with a real body under each.
Exit 0 or 1 only — no skip path by construction, so
no fail-open surface on a guard that closes one.
ci/gate.sh runs it; fail and (impossible) skip both BLOCK.
No REQUIRE_DECISIONS_LOG knob: a guard with an off
switch is not a guard.
ci/tests/decisions-log.test.sh 22 assertions, incl. the one that matters —
an append to the archive is rejected, and the
message names the file to write instead.
ci/tests/gate-decision.test.sh +3 assertions for the new wiring, AND a
pre-existing bug: the stub list never included
gitea-settings, so run_stage saw exit 127 and
recorded a FAILED stage in every case. 15 of 47
assertions failed on main for that reason alone —
the suite was measuring its own missing stub.
Now 50/50.
CLAUDE.md §7 rewritten; §2b widened to the new path.
ci/Makefile, ci/README.md, .gitea/workflows/README.md — stage named where
the other stages are named.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
c59a4b76bb |
docs(crafty): a player has joined -- the allowlist chain is closed, recommendation now PROVEN
Operator added one gamertag and the Bedrock client connected and joined. The three-pass investigation closes with a working server, no manifest change, no transport change, no network change. Moved out of UNVERIFIED: * A successful join -- OPERATOR-REPORTED (I did not watch it; the distinction is kept deliberately). Corroborated directly on disk, which is stronger than the report alone: allowlist.json went [] -> one entry, 75 bytes, and that entry's xuid is POPULATED (mtime 22:03:17). audit.log shows the operator wrote only the name (Edited file .../allowlist.json, 22:02:13). Vendor doc: xuid "will be populated when someone with a matching name connects". So the server wrote the XUID on connect, ~64s after the edit -- server-authored evidence of an XBL-authenticated join. * The reasoned mapping of "You are not invited" to the empty allowlist is CONFIRMED: one gamertag added, one successful join. allow-list=true with allowlist.json == [] was the wall, and nothing else was. * transport=raknet survives a restart: the fix involved one (pid 47006 -> 47875); afterwards server.properties mtime is still 21:33:31 with transport=raknet, UDP 4ABC bound, still NO TCP 4ABC, and a fresh pong from 192.168.2.22:19132 at 22:06. * The operator's actual route was file-edit + restart (route 2), not the console `allowlist add` I listed first. Both proven; console still preferred (no restart). README: join-gate section marked RESOLVED; decision heading and option 1 promoted from provisional to PROVEN; Bedrock human-test 4 rewritten as a 5-step verified connect procedure, each step with its distinct failure signature, stating plainly that the TRANSPORT TYPE ERROR banner and the "too few fields" warning BOTH appear on a fully working server -- that pair misled two consecutive sessions. NOT weakened, and sharpened where possible: * auto_start=0 / crash_detection=0 STILL 0, re-measured at 22:06 after the join and after the operator's 22:02:28 "modified the server" action, which did not change them. The failure mode is now worse, not better: "worked, then vanished". Merging fix/crafty-oauth2-proxy-signin is such a restart. * Silent-reset now precisely characterised: a RESTART preserves both files (observed) but leaves the server down (auto_start=0); a server RECREATE or reinstall silently resets server.properties to transport=nethernet AND allowlist.json to []. * L2/L3 conformance still uncertified; multi-player SNAT still untested (exactly one player has joined -- that is precisely the evidence that does not settle it); BDS console output still never seen (log_path is ''). No manifest touched, in any of the three passes. Read-only throughout. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
cd2617e5b3 |
docs(crafty): "You are not invited" is an empty allowlist -- and it proves RakNet works
New operator observation: a Bedrock client got "Unable to connect / You are not invited to play on this server". That was put to me as evidence that NetherNet signalling is working, that the refusal may have come from Microsoft rather than our server, and that BDS players therefore never traverse crafty-l4-udp. Measured read-only at 21:56, all three are wrong. allow-list=true (server.properties line 37) allowlist.json == [] (3 bytes, mtime 20:51, never edited since create) online-mode=true transport=raknet (line 71; file mtime STILL 21:33:31) UDP 19132 bound (/proc/net/udp 00000000:4ABC, pid 47006) TCP 19132 ABSENT (no 4ABC in /proc/net/tcp or tcp6) bedrock_server pid 47006, same process since 21:34, never restarted 1. transport is still raknet -- nobody reverted it, the binary did not rewrite it (mtime unchanged, audit.log records no later edit, same pid). NetherNet is NOT in use and cannot be what served this client. 2. NetherNet could not have served it in principle: it signals over TCP on server-port, and there is no TCP 19132 listener in the pod at all. 3. So the refusal came from OUR server, over RakNet, over 19132/udp, through the VIP. The 19132-19134/UDP exposure is the opposite of inert -- it carried both the earlier pong and this refusal. 4. Cause is certain and trivial: allow-list=true with an empty allowlist.json refuses every player. The shipped how_to says so, and it keys on Xbox Gamertag. The requested log correlation cannot be done from disk: K3s-Bedrock's log_path in crafty.sqlite is the empty string and Dedicated_Server.txt is 0 bytes -- Crafty persists no BDS stdout at all, the console is panel-UI-only. commander.log 21:39-21:56 holds only homelab crash-detection and raknet_ping lines. That silence is evidence about Crafty's logging, not about the refusal's origin. Adds a README section on the join authorization model: the measured table, the proof the refusal is ours, the "console is not on disk" caveat, gamertag/XUID identity, and three operator routes (console `allowlist add`, edit-then-`allowlist reload`, or allow-list=false -- discouraged, since the game port is deliberately un-gated so the allowlist is the only join control). Notes allowlist.json is PVC state that resets to [] on a server recreate. Options table and human test 4 now check the allowlist before blaming the network. Ranking unchanged at the top: keep transport=raknet, add gamertags. Still zero manifest change. NetherNet would ADD work (TCP 19132 in crafty-l4-tcp and in crafty-allow-ingress, plus server-udp-ports) and admit not one extra player. Read-only throughout: allowlist.json, server.properties and the DB were read; no console command issued, no kubectl mutation, no flux reconcile. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8697702d51 |
docs(decisions-log): record the gate result -- rc 3, and why CI's green is not rc 0
The handoff entry asserted conclusions but did not carry its own gate numbers. Adds them, per CLAUDE.md section 7: lint / l0 / render-golden / l1 / spec-immutability : green gitea-settings : SKIPPED (SETTINGS_API_TOKEN unset) conformance L2/L3: SKIPPED (CLUSTER_AVAILABLE != true) => exit 3, UNDETERMINED. Not a PASS. Also records two things future sessions will otherwise re-learn the hard way: 1. spec-immutability skipped SPURIOUSLY from the worktree -- dev.sh looks for the spec repo at $REPO_ROOT/../home-network-k3s, which from a worktree is the wrong directory; it exists beside the MAIN checkout. Mounting it RO through the ENGINE shim turned the one tolerated skip into a real green, so R-TEST-06 is genuinely verified. The CLAUDE.md section 3 shim should mount two paths, not one. 2. CI's green tick does NOT mean rc 0. gate/conformance is `if: false` and reports success with "Has been skipped"; gate/gate runs `./ci/gate.sh || [ "$?" -eq 3 ]`, so CI tolerates exit 3 by declared design and would report the same green for the rc 3 above. Branch protection and the runner were measured against the API, not assumed: protected=true, user_can_push=false, required_approvals=1, context 'gate / gate (pull_request)', runner incluster-dind online. PR #23 left unmerged. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
d29a73cad8 |
docs(crafty): 19132/udp IS bound and answering -- the ERROR banner was advisory
The prior entry's operator fix (transport=raknet) was labelled REASONED, NOT
OBSERVED. The operator applied it, BDS 1.26.52 printed
"================ TRANSPORT TYPE ERROR ===================
NetherNet is the only supported transport type", and that was read as a refusal.
It was not a refusal. The process kept running and bound the RakNet sockets:
/proc/net/udp 00000000:4ABC and /proc/net/udp6 ...:4ABD, both owned by
bedrock_server pid 47006 (fd/25, fd/26). The earlier TCP bind on 4ABC is gone.
Crafty's raknet_ping warning changed kind at 21:34 -- last "Got timeout"
21:34:04,306, first "too few fields" 21:34:09,854 -- and per raknet_ping.py the
latter is reachable only after a 0x1C Unconnected Pong arrives with valid magic.
Proven end-to-end from off-cluster: a RakNet Unconnected Ping to the prod VIP
192.168.2.22:19132 returns a 139-byte 0x1c pong in ~12ms:
MCPE;Dedicated Server;2193;1.26.52;0;10;...;Bedrock level;Survival;1;19132;19133;0;0;0;
The vendor's own shipped bedrock_server_how_to.html documents raknet as a
supported value ("the server listens directly on server-port"), and RakNet is
still compiled in (RakNetConnector, createUniqueRakPeer, RakNetSocket2.cpp).
"Only supported transport type" is a support statement, not a capability one.
Also explains why nethernet could never have worked here: it signals over TCP on
server-port, and this repo exposes no TCP 19132 -- crafty-l4-tcp is 25565-25567
and crafty-allow-ingress allows 19132 UDP only. packet-statistics.txt for that
run: Total Received: 0, 0B over 2293s.
Decision: keep transport=raknet, NO manifest change. The 19132-19134/UDP
exposure is correct and live, not inert. Geyser+Floodgate documented as the
reserve migration if RakNet is ever really removed (also zero manifest change).
Other corrections: the "too few fields" warning is the success signature, not a
failure; enable-lan-visibility upgraded to OBSERVED (19133 answers with the same
server_guid, so slot 1 is NOT free -- previous text said it was unreachable);
Game VIP corrected from 192.168.1.55 (the base/dev value) to the live prod
192.168.2.22; noted transport=raknet is PVC state that silently regresses to
nethernet on a server recreate. K3s-Bedrock auto_start=0/crash_detection=0
re-confirmed from crafty.sqlite and still outstanding.
Read-only throughout: no kubectl mutation, no flux reconcile, no write inside
the container. server.properties was read, never touched.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
07f03536b3 |
docs(crafty): Bedrock 19132 refuses nothing -- BDS ships transport=nethernet
Reported as "users cannot connect to Bedrock on UDP 19132". Measured read-only on prod; the platform delivered the packets and the server never opened the socket. Server `K3s-Bedrock` (c5837301-..., type=minecraft-bedrock, server_port=19132) exists and `./bedrock_server` is alive, but /proc/net/udp and /proc/net/udp6 in the crafty container hold NO socket on 19132 (0x4ABC) in either family -- port 19132 is bound as TCP (/proc/net/tcp6 ...:4ABC state 0A). Cause, from the server's own server.properties: transport=nethernet, the value Microsoft's BDS archive ships. Its inline docs: `Allowed values: "raknet" or "nethernet"`, and against server-portv6, `This property is ignored when transport=nethernet`. NetherNet is the WebRTC/Xbox-signalled transport and opens no RakNet UDP listener. Crafty does not manage the property (no `transport` anywhere under /crafty/app/classes/) and its only Bedrock probe is raknet_ping.py, hence commander.log spamming "Got timeout while issuing bedrock ping to 127.0.0.1:19132" -- over loopback, which indicts the server, not the network. The path below it is sound and was measured as such: crafty-l4-udp live as externalTrafficPolicy: Cluster with the VIP in status.loadBalancer, a non-empty EndpointSlice, `cilium-dbg service list` showing 192.168.2.22:19132/UDP => 10.0.0.130:19132/UDP (active) on BOTH announcing nodes, the datapath policy map carrying `Allow Ingress reserved:world 19132/UDP` with 26 packets / 27660 bytes already counted through it, pihole answering crafty-mc -> 192.168.2.22, and Flux at main@sha1:20ea70d == the GitRepository artifact. So this is not a platform or manifest defect. No manifest is touched: adding ports or widening policy for this would be manufacturing work. The repo-side defect is documentation -- the README told the operator that creating a server on 19132 was sufficient. README: add "Bedrock on UDP 19132 -- the transport trap" (the measurement chain, and the raknet_ping warning as its signature) and "Bedrock realities that are NOT platform defects" (no LAN/Friends discovery behind a VIP, consoles cannot enter an address, enable-lan-visibility collides with slots 1-2, server-portv6 is not exposed). Correct three stale claims: "the operator has not created any servers yet" (two exist), the human test implying a started Bedrock server is reachable, and the single-server Runtime state section (now a two-row table recording K3s-Bedrock's auto_start=0 / crash_detection=0 -- the create-API trap, live again, so it will not survive a pod restart). Also measured, reported, and deliberately NOT changed: the shared VIP's two leases sit on different nodes (crafty-l4-tcp on k3s-prod-3, crafty-l4-udp on k3s-prod-1), so two nodes ARP-announce 192.168.2.22. Benign under `Cluster` -- both nodes carry the full active frontend set -- and the existing L0 check l2-announced-vips-serve-from-any-node already forbids the `Local` that would make it fatal. A "shared VIPs must co-locate" check would assert runtime lease placement a static check cannot see. Read-only throughout: no apply/edit/patch, no flux reconcile, no write inside the container. Operator actions (set transport=raknet and verify the UDP bind; PATCH auto_start; tell players to Add Server by hand) are in the DECISIONS-LOG entry, along with what is UNVERIFIED -- chiefly that `raknet` fixes it is reasoned from the shipped docs, not observed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
4b31f97486 |
docs(decisions-log): addendum — detector armed, and why an agent cannot merge here
Records two findings from the unattended session. The drift detector was INERT: set-ci-token.sh stored the secret under a name Gitea reserves (GITEA_ prefix, HTTP 400 "invalid secret name"), sent the PUT to /dev/null, and printed success anyway. A token existed; no secret did. Enabling REQUIRE_GITEA_SETTINGS on top of that would have blocked every PR including its own fix. PR #15 fixes it and its CI is green WITH require active, which is the only proof that counts. And: merging as gitea-admin returns HTTP 405 "Does not have enough approvals". required_approvals:1 + block_admin_merge_override:true + an author who cannot self-approve means "merge on green CI" is not executable by an agent. Recorded as the control working rather than a fault, and deliberately not circumvented -- self-approval would hollow out the boundary just built. The choice of what to do about it is the operator s, and whatever is chosen must also change the declaration in expected-branch-protection.json or the detector will correctly fail on the drift. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
911ead5fbd |
docs(decisions-log): end-of-session PM handoff — six PRs open, and what a new machine needs
Written for a different session on a different computer. Lists the three open gated PRs with their risk, the one command that must follow #11 (the drift detector skips until its token exists, and a skipped detector detects nothing), and the three credentials that are deliberately NOT in git and without which the toolbox and cluster are unreachable: the age private key, prod.kubeconfig, and a Gitea credential. Records the measured platform state while telling the reader to re-measure it rather than trust this log -- which is the whole point of CLAUDE.md section 2b, and which this log earned by being confidently wrong for a month. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
fcc66c7858 |
fix(gitea): admins could override the gate -- the enforcement was not enforcing
enable_status_check and required_approvals were set earlier today and recorded as done. They did not bite. block_admin_merge_override was false, and Gitea lets repo admins override merge requirements without it. jakman85 holds owner, so both requirements were bypassable by exactly the account that performs every merge. The evidence was already on the record: PRs #5-#9 all merged while required_approvals: 1 was in force, with ZERO approvals recorded on any of them. Two process failures, both the same shape as the defect this whole session started with: 1. Verified with the wrong view. /branches/main reports protected, enable_status_check and required_approvals but NOT block_admin_merge_override -- it showed everything green while the rule was overridable. /branch_protections/main is authoritative. 2. Probed for a field name assumed to exist. This Gitea (1.24.3) has no `apply_to_admins`; asking jq for it returns null, which reads like "set to false" but means "key absent", so a PATCH that silently did nothing produced a convincing before/after. jq returning null is not a value -- it may be an absence. Enumerate with to_entries. require-gate.sh now sets block_admin_merge_override: true; block-admin-override.sh fixes it in isolation. Both carry the field-name warning. This PR is the test. If it merges WITHOUT an approval, the fix did not work. Also records the governance question: with only jakman85 (human) and gitea-admin (agent), a human-authored PR now needs the automation account to approve it. That is ceremony, not review, and the choice between keeping it, dropping approvals in favour of the status check, or adding a second human is the operator's to make. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
b2d9883958 |
docs(ci): close out the hardening -- land the scripts, correct a claim that shipped wrong
The branch-protection + PR-gate + sidecar sequence is complete and verified. This records the finished state and fixes what the sequence itself proved wrong. CORRECTS A FALSE CLAIM I SHIPPED. PR #7 said the sidecar conversion would remove the one-per-start dockerd line "http: TLS handshake error from 127.0.0.1: EOF". It does NOT. Exactly one still appears, immediately after "API listen on [::]:2376" -- it is now the startupProbe's FIRST attempt, landing microseconds after dockerd begins listening and cut off by timeoutSeconds: 1; the next poll succeeds. Cosmetic, and deliberately not chased: raising the timeout to hide one benign startup line would slow every subsequent poll. The manifest comment now states this instead of the wrong prediction. Lands infra/gitea/require-gate.sh and verify-then-enforce.sh, which drove the rollout from a session temp dir. verify-then-enforce.sh merges nothing -- merging is a human's reviewed act -- but forces the reconcile and ABORTS unless the pod is healthy, a runner is online, and a real workflow goes green on it, before enabling enforcement and probing the push. Re-run it after any future act-runner change. Trims the now-obsolete feat/ rationale from claim-ship.sh: per-branch push triggers were deleted, so no prefix produces a duplicate run any more. CLAUDE.md's enforcement table flips from "weaker than this rule" to machine-enforced, with all four rows marked verified and the push rejection noted as PROVOKED rather than assumed. It also now states what a green PR does NOT prove: CLUSTER_AVAILABLE is unset, so L2/L3 never runs in CI and only the L0/L1 offline subset is covered. infra/gitea/README.md marks the sequence done and names the root cause that remains unfixed: Gitea repo settings live in its own database, invisible to Flux and unreadable by the conformance gate, so NOTHING detects drift from what this directory declares. These scripts assert desired state; they are not a detector. The check that reads the live API and fails on divergence is the highest-value work left, and it is the difference between this being fixed and this being unable to recur. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
db1a623073 |
Merge pull request 'docs(runbooks): the PR flow, not git push origin main' (#5) from docs/runbooks-pr-flow into main
Reviewed-on: #5 |
||
|
jakman85andClaude Opus 5
|
e72050c690 |
docs: correct CI/platform documentation that was stale, wrong, or load-bearing
The 2026-09-29 session found the merge gate did not exist -- no branch protection, no runner -- while the tree asserted both. This corrects the documents that carried, and in several cases CAUSED, that belief. CLAUDE.md gains section 2b: never infer platform state from this repository. It carries the specific API and kubectl checks for protection, runners, CI status, workloads and Flux revisions, and the rule that a green tick is not proof anything ran (a skipped Gitea job reports success). This is the rule whose absence produced every other finding. .gitea/workflows/README.md and merge-gate.yaml both said "there is no remote git host, this file is dormant". Gitea had been live for weeks. Both were written as plans and never revisited once the plan completed. The Transition section is now marked COMPLETE and says what is genuinely still outstanding (CLUSTER_AVAILABLE unset, so L2/L3 never runs in CI). DECISIONS-LOG 2026-08-31 is annotated IN PLACE. Anyone reading chronologically previously hit "required status check; runner incluster-dind online" with nothing marking it false until a month later. The correction now sits on the bullet itself. ci/README.md, clusters/README.md and both conformance READMEs carry a status banner: they speak as though no cluster exists, which makes a live L2/L3 coverage gap read as an expected phase. That framing is why the gap went unexamined. apps/wave-1/gitea-actions-runner-secret.sops.yaml: fixed a wrong path, replaced advice to `sops -d` into a terminal (which CLAUDE.md forbids) with the rotation script, and recorded that a fresh token alone does not re-register a runner holding a stale /data/.runner. apps/wave-0/plex-claim-secret.sops.yaml: the header claimed the file ships the placeholder. It has not since the claim -- it holds the spent token, which is inert and deliberately left in place. catalog/plex.yaml and the plex secret both asserted a "~4-minute TTL" as fact. It was uncited, upstream documents no expiry, and it was the basis for a since-disproven claim that the pipeline was too slow to fit inside it. Measured: ~2m45s. Both .sops.yaml files were verified to still decrypt after the comment edits (`sops -d > /dev/null`), and render==golden confirms the catalog comment does not propagate into the rendered manifest. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8e017fad91 |
docs(runbooks): the PR flow, not git push origin main -- that instruction is now blocked
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
5a9e455149 |
docs: delivery contract for agents + record that the merge gate does not exist
Going to land the plex claim wiring through the documented PR flow surfaced that the flow has never run. PR #1 is the first pull request this repo has ever had. Verified against the live Gitea, not inferred from the docs: * branch_protections is [] -- main is protected:false, enable_status_check:false, status_check_contexts:[], required_approvals:0, user_can_push:true * ZERO Actions runners registered at repo, org and site level DECISIONS-LOG 2026-08-31 asserts a "required status check" and a runner "incluster-dind online". Both are false of the live system. The gate is neither enforced nor able to run, and that went unnoticed because the log claimed otherwise -- the same fail-open class as the two holes found on 2026-09-18. Adds CLAUDE.md, the first agent-facing delivery doc here: the branch -> PR -> gate -> approval -> merge rule, stated alongside the honest fact that enforcement is currently weaker than the rule, so agents follow the rule rather than what they can get away with. Carries the toolbox gate invocation, the two git-worktree workarounds, the rc 0/1/3 contract, and secret handling. Records the ordering trap: enabling a required status check before a runner exists deadlocks every PR permanently, including the one that would fix it. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
e3e56af461 |
docs(decisions-log): overnight handoff — backup alert crying wolf, 49 unmapped app manifests, egress branch finished
Records the three fixes and, in each case, the thing that was reporting green or red over something other than what it claimed to measure. Includes corrections to my own numbers (the '192 unmapped files' figure was my audit script's regex matching across a newline, not the repo) and the two check defects the egress branch exposed. PM HANDOFF lists open items in priority order. R-STATE-02 remains the Phase 7 exit item and is blocked on the permission classifier, not on readiness. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a3f4ddb971 |
merge: feat/egress-gateway-staleness-alert — finished so it can actually pass the gate
Brings the two alerts for the silent egress-gateway failure: VPNEgressSteeringStale
(critical) and VPNEgressUplinkLabelMissing (warning), plus the kube-state-metrics
allowlist entry that makes them non-inert.
WHY THE BRANCH SAT IN THE BACKLOG: it could not pass the merge gate. Three
whole-repo checks failed on it, and each one was a real gap rather than a nuisance:
* alert-runbooks-resolvable — both alerts link to /runbooks/vpn-egress-stale,
which did not exist. An alert firing now would hand the operator a 404.
* alert-rules-unit-tested — alerts.yaml is already a covered file, so new rules
in it must be tested or a "tested" file quietly grows untested rules.
* alert-metrics-have-scrape-targets — see the guard bug below.
WHAT THE ALERTS CATCH. egress-class=vpn pods leave through one gateway node's tun0;
Cilium resolves it to an ifindex and caches it. An OpenVPN reconnect destroys and
re-creates tun0 with a NEW ifindex, and the node label
homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex> is what forces Cilium to
re-resolve. If the label does not follow the tunnel, the cache points at a destroyed
interface and every vpn-class pod is silently dropped — while tun0 stays up and
VPNTunnelDown says nothing. That ran ~80 minutes undetected on the dev gateway.
Verified this is live on prod, not inert: k3s-prod-3 carries egress-gateway=true and
egress-uplink=10.150.192.86-139, and vpn-egress-steering has been in place 14 days.
The label was exported on 0 series before this change, so the allowlist entry is
what switches the alerts on.
runbooks/vpn-egress-stale.md — written and wired into the configMapGenerator, with
the four required operator sections. It leads with proving egress from a pod the
policy selects, because the symptom is "outbound traffic silently disappears", and
it says plainly not to "fix" this by deleting the policy: that does not restore
egress, it sends vpn-class traffic out of the real ISP connection, which is the leak
the class exists to prevent.
platform/monitoring/tests/vpn-egress.test.yaml — four promtool fixtures. 1 and 2 are
the SAME tunnel reconnect with and without the label following it, which isolates the
mechanism the rule actually keys on: a label VALUE change is not a changing sample,
it retires one series and starts another. 3 is the inert case (gateway node, no uplink
label) where the critical rule CANNOT fire, which is why the warning exists.
Two fixes the branch needed that were defects in the checks themselves:
1. alert-metrics-have-scrape-targets asserted the KSM allowlist matched
`nodes=[...egress-gateway]` with a literal closing bracket — i.e. that the list
contained EXACTLY that one label. Adding a second label correctly (egress-uplink,
which these alerts need) therefore FAILED the check while the property it guards
was perfectly satisfied. Loosened to `[,\]]`: in the list, not the whole list. A
guard that breaks when you correctly extend the thing it guards teaches people to
edit the guard, which is how it stops guarding.
2. UNTESTED_ALERTS listed VPNTunnelDown, and the check flagged it as stale because
"asserted on" is a bare word match over the test file — my header COMMENT
mentioning VPNTunnelDown was enough to count as coverage. Rather than delete the
register entry and claim coverage that did not exist, the fixtures now assert
VPNTunnelDown QUIET in all four cases. That required making them faithful: they
had no node_network_receive_bytes_total{device="tun0"} at all, so VPNTunnelDown
would have been firing throughout unnoticed, because promtool only checks the
alertnames you name. A healthy tun0 passes traffic, and that silence IS the claim
these rules exist for.
NOTE FOR LATER, not fixed here: that word-match means any alert named in prose counts
as tested. Tightening it to `alertname:` positions could reveal other prose-only
"coverage", and the register is shrink-only, so the cascade wants doing deliberately
rather than at 01:00 unattended.
Conflicts resolved: kept HEAD's Velero custom-resource-state block and took only the
branch's one-line allowlist change (14 lines down to 1); inserted the branch's
2026-09-12 DECISIONS-LOG entry in date order rather than appending it out of sequence.
Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired. The new unit tests pass under the pinned promtool 3.14.0.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
623fa14256 |
docs(decisions-log): PM handoff — two fail-open holes in the gate itself
Records the session's three pieces of work and, more importantly, the theme joining them: the gate was reporting green over things it had not checked. - runbooks/node-memory-swap.md, with the three claims I wrote and then removed because they were false (EtcdIsVoter does not exist here; node-drain-policy is not pinned so the runbook reads it live; the 5-vs-1 rebuild-limit confusion). - The nested-worktree scan: six discovery sites auditing 18 copies of the repo. - L1 rendering never running on a requirement-scoped gate, with the external-dns incident reproduced via --print-selection (0 tests before, 3 after). - The two defects that surfaced in the meta-check guarding all of it. PM HANDOFF lists the open items in priority order. The first is a judgement call worth flagging rather than burying: the planned memory swap on k3s-prod-1 IS a supervised single-node loss on prod, which is exactly the event R-STATE-02 describes and which is still proven only on the decommissioned dev cluster. Capturing evidence during the swap costs nothing; manufacturing the event later costs a second maintenance window. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
6a4dc0ada7 |
docs: INCIDENT — I printed the console password to the transcript
A kubectl jsonpath with {range .data}{@}{end} dumped the unifi Secret's data
map, putting the base64 console-password in the session transcript. Exactly the
constraint I put in every agent brief, broken by my own careless verification
command.
Exposure bounded: 0 commits contain it (git log -S), the capturing scratch file
was shredded, and a repo-wide grep finds nothing on disk. Critically it is not
yet a live credential — the Secret was created two days after the running VMI,
whose volume list is rootdisk and cloudinit only, so the UOSDATA disk was never
attached, install-uos never ran, and the ubuntu account is still locked.
Action recorded with the ordering that matters: rotate console-password BEFORE
recreating the VMI, so the leaked value never becomes valid.
Lesson: checking a Secret's existence never needs its values. Listing .data's
keys or .metadata.name answers the question; dumping .data answers one nobody
asked.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
cf0d6c6beb |
docs: verified — internal services are gone from public DNS, split-horizon works
Confirmed at the authority, not a cache: all six internal names absent publicly, all five public names untouched, and Pi-hole answering for the six internally. The R-DNS-03 disclosure is closed for that tier. Records a correction to my own sequencing: I insisted this could not land before the DHCP repoint, but the router returns nothing for those six names, so the records served nobody and the dependency I asserted did not exist. One query would have shown it. Also records the operator's clarifications — Pi-hole is DNS only with the router still doing DHCP, reverse proxy by hostname rather than a second IP, and Minecraft staying open — plus the unencrypted-secret fix. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
3173254cc4 |
docs: UniFi OS was never the blocker — it is installed and the guest lost its IP
Third confident claim of mine corrected in two days, and this one I took from a stale README rather than the machine. uosserver.service starts on the current boot and the root filesystem holds 6.3 of 40 GB, while enp1s0 is up with no address and does not answer ARP — so the controller runs and nothing can reach it. Records the near-miss: proceeding on my premise would have reinstalled over a working controller, which the new install-uos guard now prevents. Also records that there is no way into the guest at all, which is why the deliverable became console access, and the cloud-init instance-id discovery that explains why an earlier unattended install could never have taken effect. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
989781b352 |
docs: Minecraft server live, EULA accepted on the operator's behalf, world backed up
Records the EULA acceptance prominently with revocation instructions, the Server List Ping proof, and Crafty's autostart trap where the create API accepts 'autostart' but stores 'auto_start'. Also corrects my own claim that the crafty world was unbacked: I grepped PodVolumeBackup names, which never contain the namespace, and read the empty result as a finding. Crafty had 2 PVBs in the last run. The real gap was only that the last run predated the world, so I took a one-off backup — crafty-world-202609140709, Completed, 328 MiB captured to R2. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
b4bc8e1cb9 |
feat(crafty): first Minecraft server — Paper 26.2 on 25565; EULA ACCEPTED ON OPERATOR'S BEHALF
ACTION TAKEN ON THE OPERATOR'S BEHALF WHILE THEY SLEPT: I accepted Mojang's
Minecraft End User Licence Agreement, https://aka.ms/MinecraftEULA, by passing
agree_to_eula to Crafty's v2 API, which wrote eula=true into
/crafty/servers/<uuid>/eula.txt. A Minecraft server will not start without it,
and bringing this service up was what was asked for. Please review. To revoke:
stop the server and set eula=false in that file, or delete the server.
Crafty was healthy but had never had a server created, so 192.168.2.22:25565
refused connections. Created "homelab" via the API (not UI automation):
Paper 26.2, TCP 25565, heap -Xms2000M -Xmx2000M, autostart on
uuid 13c7b85d-00bb-4a58-8c86-2433ca47891c
Paper over Vanilla for tick performance and plugins, with no cost to players
(vanilla clients connect unchanged). 26.2 is Mojang's latest.release, out
2026-06-16; the 26.3-rc-2 snapshot and 26.2-rc-2 were deliberately avoided.
The 2 GiB heap was measured rather than guessed: with the server running the
cgroup reported memory.current 2.86Gi against the 4Gi limit, so a 3 GiB heap
would have risked an OOMKill.
Proven playable, all real output: Crafty reports running with 'Done (12.737s)!';
Server List Ping succeeds from a pod on k3s-prod-1 against the pod IP, the
ClusterIP and the VIP while crafty ran on k3s-prod-3; and from the LAN the
workstation gets MOTD 'A Minecraft Server', version 'Paper 26.2', protocol 776,
players 0/20 over a full handshake plus ping/pong. Deleting the pod moved Crafty
to another node; it auto-started with no manual action and the LAN ping still
succeeded, re-confirming the externalTrafficPolicy: Cluster fix from
|
||
|
jakman85andClaude Opus 5
|
83f13f4b0a |
docs: correct my 'live outage' claim — unifi and crafty were never up
I reported the UniFi controller and Minecraft server as unreachable for up to 8 days. Probing the backend pods directly, bypassing VIP and policy, returns REFUSED on all four ports while crafty's own UI and pihole's DNS answer as controls. /crafty/servers is empty and the unifi README still lists installing UOS in the guest as the open blocker. Nothing regressed. The externalTrafficPolicy defect is real and is fixed, but it was not the cause. I observed unreachable ports, found a mechanism that produces unreachable ports, and concluded it was the cause without running the one control probe that separates them — the same error shape as nearly blaming the drive for the I/O pressure spike when the local disks were idle. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
ce7819a7da |
docs: FINDING — public DNS publishes internal RFC1918 addresses, contradicting R-DNS-03
Operator-reported, verified. external-dns runs provider cloudflare with gateway-httproute + service sources and publishes the Cilium Gateway VIP 192.168.2.20, so ~16 hostnames in a public authoritative zone answer with RFC1918 space. The spec says the opposite: external resolves to the public IP, Pi-hole serves the internal split-horizon answers, internal services are unreachable from the internet. Provenance stated plainly: the pattern predates me — external-dns always published the Gateway address, and dev did the same under owner=default. My apex migration changed which names carry it, propagating an existing flaw onto the apex rather than inventing it. Four consequences recorded: topology and service-inventory disclosure; a DNS-rebind primitive, which is why the SOHO router strips these answers and LAN clients need per-client overrides; the five services declaring exposure public are not actually publicly reachable; and seven internal services are publicly advertised. Also records why deleting the records is not a safe first step: the *.home.keithjackson.dev wildcard is gone, so removal yields NXDOMAIN rather than a fallback. Remediation sequenced, nothing started. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
3de79f81a5 |
docs: correct three mountstats figures I supplied from bad measurements
Handed an agent a baseline of 4.0-4.8 ms/op response, 0 ms/op queue, 0 major timeouts over 24h. Re-measurement found queue time is 0.013-0.067 ms/op (I read rounded output as zero); the 24h claim was really ten minutes, because the DaemonSet had just rolled and the range query returned 18 samples rather than ~2,880; and the baseline was idle, so 4.4 ms is a cached STATFS rather than real media I/O and thresholds could not be multiples of it. The 24h error is the same class as the label-values mistake recorded above: trusting a query whose window was never valid. Records two real discoveries: mounts sharing an NFS superblock share kernel RPC statistics, so prod-1's two plex mounts are one signal reported twice; and maximum_rpc_slots high-water is 586 on prod-1 against 2 on prod-2, an independent indication that prod-1 carries far more concurrent NFS work. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
de861b8672 |
docs: correct my own claim — node_nfs_* metrics DO exist; mountstats is the gap
I recorded that NFS was 'entirely unmonitored' with 'zero NFS observability',
based on a __name__ label-values query that came back empty. The query was at
fault: count({__name__=~"node_nfs.*"}) returns six live families on all three
nodes, and the nfs collector is on by default.
The real gap is narrower — node_mountstats_* is absent, and that is the one
carrying per-mount RPC round-trip time, so NFS latency specifically is
invisible while volume and retransmissions are not.
Two further signals already existed and had simply never been queried:
container_pressure_io_stalled_seconds_total, which names the stalling workload
and for tonight names sonarr; and etcd_request_duration_seconds via the
apiserver job, which the node-io runbook had explicitly claimed did not exist.
Also tempers the Proxmox hypothesis I floated: prod-1 was doing 114 NFS reads/s
of its own during the spike, so our traffic is substantial and the stall is not
obviously unexplained by us.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
7369495b67 |
docs(pm): owner decisions — files stay, workloads stay, instrument NFS instead
Records three owner decisions and the finding that makes the third tractable. The /downloads malware stays: the Proxmox-hosted stack still has operational control of those files so deletion just triggers a re-download, nothing but the arr suite touches the directory, and there are no Windows machines anywhere — which removes the exposure I was concerned about. Written down so nobody 'discovers' and deletes them later. Anti-affinity for the co-located NFS pods declined; the concentration is accepted rather than overlooked. And the enabling finding: there are NO NFS metrics in the TSDB at all — node-exporter's nfs and mountstats collectors are off by default. A cluster with four NFS mounts carrying the media library and legacy Nextcloud data has zero NFS visibility, while PSI counts NFS stalls as I/O stalls. That is precisely why a 20.98% stall alongside idle local disks could not be explained. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
dafa9cd241 |
docs: prod-1 I/O critical was not a failing disk; confirmed malware in /downloads
NodeIOPressureCritical fired on prod-1 (peak 20.98% PSI, recovered
unattended) — the first time this alert has fired on prod, and only because
the scrape gap was closed yesterday.
Evidence rules out the 2026-09-07 drive repeat: 3.5 MB/s of writes, device
utilisation 0.09-0.21, write latency 1.4-4.8 ms, zero NFS retransmissions,
21.4 MB/s NFS throughput, no degraded Longhorn volumes, and KSM never observed
NotReady. A 21% stall on trivial volume with healthy latency is transient
blocking I/O, not hardware — and PSI counts NFS too.
Structural finding: sonarr, qbittorrent and jackett all run on prod-1, the
node with the I/O history. Same concentration class as the CNPG primaries.
Separately, the /downloads files are confirmed malware: two PE executables
('MZ') sharing one MD5, padded to ~926 MB and named as two different shows,
plus a ZIP masquerading as video. Not executed, cluster not compromised; the
risk is a Windows client opening them.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
0841e2461c |
docs: correct a misattribution — 192.168.2.1 is the SOHO router, not pfSense
I wrote that pfSense was stripping the RFC1918 answers. The operator corrected it, and this repo already documented the topology I failed to read: 192.168.2.0/24 is pfSense's WAN (clusters/prod/kustomization.yaml:458, cluster-vars.yaml:190, DECISIONS-LOG 1772), pfSense lives on 192.168.1.0/24, and 192.168.2.1 was already recorded here as 'both gateway and only IPv4 resolver' and flagged as a SPOF (entries 1318, 1468). The measurement is unchanged — that resolver returns NOERROR/ANSWER:0 for this zone while resolving cloudflare.com — but the attribution invalidated my preferred fix: configuring pfSense cannot help, because prod's resolver path never reaches it. Fixes reordered accordingly. Exposing Pi-hole's DNS to the LAN is now first: it is the architecture the repo already claims, it does not depend on a consumer router's feature set, and it retires the single-resolver SPOF this log flagged months ago. Configuring the SOHO router drops to second, since many such devices cannot express a domain override and it is outside GitOps regardless. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8891df07b0 |
docs: LAN split-horizon DNS is documented but unimplemented; pfSense strips the answers
The workstation could not reach sonarr.home.keithjackson.dev. The stale dnsmasq override pinning the zone to dev's dead 192.168.2.50 was the obvious cause, but removing it — which is what I was about to recommend — would not have fixed it: pfSense returns NOERROR/ANSWER:0 for this zone while resolving cloudflare.com fine, the signature of private-address rebind filtering, since the zone legitimately points at RFC1918 space. The underlying gap: coredns-custom.yaml defers LAN clients to 'PiHole split-horizon', but Pi-hole's Service is ClusterIP with ports 80/4180 — it serves no DNS at all. So the LAN half of split-horizon does not exist and each client silently depends on a hand-maintained file. Records the corrected override (staged for the operator's sudo) and the three proper fixes, none applied. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
229d7f2d9c |
docs: README status was stale by a whole phase; record the R-STATE-03 decision
The Status section still said phases 0-6 were deployed on the DEV cluster and that Phase 7 was not open with clusters/prod 'a design-only overlay'. Dev was decommissioned on 2026-09-12 and prod has been the live cluster serving the apex since. It also claimed 36/51 requirements covered with a safety-critical gap, and that Velero was blocked on OQ-BKP. Now states Phase 7 as open with its three exit criteria and which one is actually outstanding; corrects the node inventory from the Proxmox dev IPs to the bare-metal prod nodes; records Velero as unblocked to R2; and adds the distinction the old text elided — coverage (51/51) is not proof (PROVEN: 0), because the only dr-restore and node-loss proofs were recorded on a cluster that no longer exists, against MinIO rather than R2. Also records the owner's decision that Velero+kopia to R2 is the accepted volume backup, with no Longhorn backup target, and that Longhorn config belongs in source. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a62242bd78 |
docs(pm): session close — three corrections to my own claims, and the remaining backlog
Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
f663e7c801 |
fix(longhorn): the I/O caps a runbook promised were never declared — declare them
`runbooks/longhorn-degraded.md` told operators that, after the 2026-09-07 etcd
I/O-starvation incident, `concurrentVolumeBackupRestorePerNodeLimit` and
`snapshotHeavyTaskConcurrentLimit` had been tightened 5 -> 1. Measured live on
prod today: both read **5**, and neither name appeared anywhere in this
repository — so nothing declared them and nothing would have restored them.
Fourth instance of config that looked applied and was not.
Landed in `platform/longhorn/helmrelease.yaml` `defaultSettings` (the path Flux
reconciles; `values.yaml` is the reference mirror), NOT by `kubectl patch` —
a hand-applied Setting is exactly how this gap arose and Helm would silently
rewrite it on its next upgrade.
concurrentVolumeBackupRestorePerNodeLimit 5 -> 1
snapshotHeavyTaskConcurrentLimit 5 -> 1
replicaReplenishmentWaitInterval 600 PINNED, unchanged
backupConcurrentLimit / restoreConcurrentLimit 2 / 2 PINNED, unchanged
concurrentAutomaticEngineUpgradePerNodeLimit 0 PINNED, unchanged
nodeDrainPolicy block-if-contains-last-replica PINNED — it was in
force only as Longhorn's OWN DEFAULT, declared in no manifest and in no
rendered ConfigMap, so an upstream default change would have moved it
silently while two runbooks reasoned about drain safety from it.
EVERY KEY PROVEN TO LAND, not assumed. `helm template longhorn/longhorn
--version 1.12.1` (the version this chart range resolves to, per the
HelmRelease's own lastAttemptedRevision) on this branch's spec.values,
substituted as clusters/prod substitutes them, emits all eight into the rendered
`longhorn-default-setting` ConfigMap in kebab-case. Diffed against the LIVE
ConfigMap the change is purely ADDITIVE: 8 keys added, none altered, none lost.
The chain to the Settings is verified read-only on prod — all 115
settings.longhorn.io carry `longhorn.io/configmap-resource-version: 7917`
(= that ConfigMap's resourceVersion) with `status.applied: true`, and every key
in the ConfigMap equals its Setting's value. The two targets sit at 5 with the
key absent, i.e. Longhorn defaults, as measured.
`orphanAutoDeletion: true` REMOVED, verified inert two independent ways against
the deployed 1.12.1: the render emits no orphan key at all (Longhorn 1.9 renamed
it `orphanResourceAutoDeletion`, a list, and the chart drops unknown
defaultSettings keys with no error), and live, `setting/orphan-auto-deletion` is
NotFound while `orphan-resource-auto-deletion` reads "". Deliberately NOT
replaced — orphan deletion is extra writes on the constrained partition and buys
only space. Orphan cleanup stays manual by decision; the two runbooks that cite
the key by name are corrected rather than left to go stale.
BOTH NEW CAPS ARE 1, AND 1 COSTS SOMETHING — stated, not waved at.
`concurrentVolumeBackupRestorePerNodeLimit: 1` costs nothing TODAY: Longhorn has
no backup target (the `backup-target` Setting does not exist; still the
WS-STORAGE Phase-4 TODO), and velero's nightly run is not a Longhorn backup —
`snapshotVolumes: false` + `defaultVolumesToFsBackup: true` is kopia
file-system backup through velero's node-agent, which this Setting does not
gate; CNPG base backups are barman-level and ungated too. Its only consumer is
an R-STATE-03 restore, where serialising ~24 volumes beats losing etcd
mid-restore. Revisit when a Longhorn backup target lands.
`snapshotHeavyTaskConcurrentLimit: 1` rather than 2 because purge work is
generated by rebuild cleanup (auto-cleanup-system-generated-snapshot true, NO
RecurringJobs, 24 volumes / 24 snapshots) and rebuild is already serialised to
1 per node — so 1 keeps pace with its own generator, where 2 buys backlog drain
at the price of a second heavy writer during the exact rebuild that starved
etcd. The cost: space reclaims serially after a rebuild storm — real here (a
37.7 GiB snapshot once pinned a volume at 97.3% allocated), but instrumented by
LonghornDiskAllocationHigh and recoverable by hand, where a wedged etcd is not.
Headroom on the record: I/O PSI full is 0.30/0.62/0.37 % now, and 3.79 under a
full 18-replica rebuild after node 1's drive replacement, against a 5% warning
threshold. These caps are a BOUND, not a fix — etcd and Longhorn still share one
ext4 filesystem and nothing arbitrates the jbd2 journal fdatasync waits on.
NOT PORTED, deliberately: the same unmerged branch's node-level I/O arbitration
(`roles/etcd_io_priority` — BFQ on the shared device, nr_requests 64 -> 32,
systemd IOWeight drop-ins). The 2011 C400 it was written for was replaced on
2026-09-07 and the same load class measured 11.36 -> 3.79 I/O PSI with the node
Ready throughout; the role was never run against any cluster, its author
recorded it as unproven with a node-3-first sequenced rollout, and its own
README concedes it does not separate the two tenants. Node-wide blast radius,
low ceiling, stale measurements (node 1 reinstalled, founder moved to node 2).
It stays an operator option if I/O PSI climbs back toward 5% — which, unlike in
September, we would now be told about. Nothing under infra/ansible/ is touched.
ALSO ALREADY DONE, so skipped: that branch's NodeIOPressureHigh/Critical and the
dashboard I/O row are on main already (
|
||
|
jakman85
|
52398d4426 |
merge: Longhorn alerting measured the wrong things, three ways
(1) LonghornVolumeCapacityHigh alarmed on ALLOCATION while calling itself a capacity alarm. Prometheus's TSDB volume read 0.973 while the kubelet saw the same filesystem at 0.208, because 37.73 GiB of it is a Longhorn SYSTEM snapshot — the kind taken for the full replica rebuild that prod-1's drive replacement forced — growing +152 MB/h while the filesystem stayed flat. The rule was therefore unclearable by design, i.e. permanently firing. Deleted as redundant, shown rather than asserted: 23 of 24 volumes fall under the kubelet-based filling rules, and the 24th is volumeMode Block, which the deleted rule was not covering either. (2) LonghornVolumeDegraded and LonghornVolumeFaulted could not fire even with the scrape working. longhorn_volume_robustness on 1.12.1 is ONE-HOT over a state label with values 0/1, not the 0-3 enum the rules and their comment assumed, so a faulted volume would have alerted nobody. This survived the scrape fix landing hours earlier. (3) LonghornNodeStorageHigh was wrong twice: its 0.85 threshold sat above the 75% at which storage-minimal-available-percentage=25 already removes a disk from the scheduling pool, and longhorn_node_storage_usage_bytes SUMS a node's disks, so one full disk hid behind a half-empty sibling. Replaced with per-disk LonghornDiskAllocationHigh (>0.70) and LonghornDiskUnschedulable, the latter reading Longhorn's own verdict instead of re-deriving a formula from three settings no metric exports. Baseline green with 11/11 mutations red. retentionSize 45GiB of 50Gi and discard/TRIM are both REPORTED with recommendations, not changed: TRIM would not have helped here, since remove-snapshots-during-filesystem-trim is false and the 37.7 GiB is a snapshot. # Conflicts: # DECISIONS-LOG.md |
||
|
jakman85andClaude Opus 5
|
4687521d65 |
fix(monitoring): delete the Longhorn rule that measured allocation and called it capacity
LonghornVolumeCapacityHigh went pending within minutes of the Longhorn scrape
landing today. Re-measured on prod: it read 0.9731 on the Prometheus TSDB volume
(actual_size 52243681280 / capacity 53687091200) while the kubelet reported the
SAME filesystem 20.8% used. actual_size is head + pinned snapshots, and one
user_created=false system snapshot from prod-1's replica rebuild holds 40510484480
bytes (37.7 GiB) of it; Longhorn returns no freed blocks without discard/TRIM, and
retentionSize 45GiB on a 50Gi volume targets 90% by design. The ratio is pinned
near 1.0 and cannot fall back under 0.85 — a permanently-firing alert, the same
shape as CNPGNoRecoverabilityPoint.
DELETED, not re-thresholded. Raising the threshold would tune away a true
statement about allocation. Redundancy proven, not asserted: of 24 Longhorn
volumes on prod, 23 appear in kubelet_volume_stats_inodes > 0 and so fall inside
MonitoringVolumeFillingUp (namespace="monitoring") or PersistentVolumeFillingUp /
PersistentVolumeAlmostFull. The 24th is unifi/unifi-rootdisk, volumeMode Block —
no filesystem for anyone to measure, and the deleted rule was not covering it
either.
AND A SECOND INERTNESS the scrape did not fix. longhorn_volume_robustness on
longhorn 1.12.1 is one-hot over a `state` label with values 0/1, not the 0-3 enum
the rules assumed:
count(longhorn_volume_robustness) => 96 (24 volumes x 4 states)
count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
count(longhorn_volume_robustness == 2) => (empty)
count(longhorn_volume_robustness == 3) => (empty)
So a FAULTED volume would still have alerted nobody. Re-keyed to the label; the
"44 series / 72 unknown(0, detached)" note in platform/longhorn/helmrelease.yaml
was the same misreading and is corrected in place.
The allocation question moved to where allocation is a budget. LonghornNodeStorageHigh
fired at 0.85, but storage-minimal-available-percentage is 25 on this cluster, so a
disk leaves the scheduling pool at 75% used — the old threshold could only fire
after scheduling had already been refused. Replaced by:
LonghornDiskAllocationHigh (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70, 30m
LonghornDiskUnschedulable longhorn_disk_status{condition="schedulable"} == 0, 15m
Per DISK, not per node: longhorn_node_storage_usage_bytes sums a node's disks, so
one full disk could hide behind a half-empty sibling. Measured prod usage
0.2254 / 0.2306 / 0.2369 on prod-1/2/3.
PROVEN NEITHER PERMANENTLY FIRING NOR INERT. platform/monitoring/tests/longhorn.test.yaml
adds 6 cases pinning every rule quiet on today's measured prod state and firing on
a constructed bad state, with every `for:` asserted, plus regression assertions that
the old `== 2`/`== 3` forms match nothing and that per-disk 0.72 is visible where the
node-summed 0.475 is not. Mutation-checked 11 ways; each turns the suite RED. Live on
prod: all four new expressions return empty, while the sum of their selector
cardinalities is 54. The L0 UNTESTED_ALERTS register shrinks by seven (7/56 -> 14/58
alert rules unit-tested), which also brings MonitoringVolumeFillingUp,
PersistentVolumeFillingUp and PersistentVolumeAlmostFull under test since the
redundancy claim rests on them.
REPORTED, NOT CHANGED (DECISIONS-LOG, owner decisions): retentionSize 45GiB on a
50Gi volume is too tight — recommend 36GiB, with reasoning; and discard/TRIM on the
longhorn StorageClass is feasible but not advisable now — remove-snapshots-during-
filesystem-trim reads false on prod, so trim would not have reclaimed the 37.7 GiB
anyway, and continuous discard on nodes where the etcd WAL shares the spindle points
at this cluster's known failure mode.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
69a99e6ca6 |
docs(pm): two rendered manifests nothing applies — the keycloak one is a trap
Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
cd454886c7 |
docs(pm): handoff resolved — all five WIP branches merged, two self-inflicted regressions recorded
Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
088ff13d15 |
merge: coverage debt paid — 51/51, and the root really did hide a service
R-C2-01 and R-C2-02 were the last two uncovered requirements (49/51). Now
100% with 0 gaps, via an offline L0 check (49 assertions) that compares two
repository facts: every home-zone hostname the repo routes over HTTP must be
linked from the apex dashboard, and every link the dashboard offers must
resolve to something the repo actually serves at L7 or L4.
'A service the root hid' was still a live defect: crafty.${BASE_DOMAIN} was
served and reconciled while nothing on the root linked to it. Committed RED
first so the history shows the check catching it, then fixed.
Also fixes a regex that could never have worked: ci/touched-requirements.sh
matched R-[A-Z]+-[0-9]+, where [A-Z]+ eats the C of R-C2-01 and then demands
a dash where the 2 is — so a manifest declaring either ID attributed nothing.
Every other consumer already used [A-Z0-9]+.
16 mutations proven caught. R-TLS-02's drafted amendment is recovered into
DECISIONS-LOG: revocation is not implementable with cert-manager, and the
choice between amending the requirement and a reviewed DOCUMENTED_GAPS entry
is the owner's.
# Conflicts:
# DECISIONS-LOG.md
|
||
|
jakman85andClaude Opus 5
|
e4bd76070e |
fix(qbittorrent): a durable WebUI credential, and the two bugs the WIP hid
Finishes the inherited WIP. qBittorrent 5.x had no WebUI\Username or
WebUI\Password_PBKDF2 in its conf, so it minted a NEW random admin password on
every process start. Both consequences had ALREADY fired: measured before this
change, `POST /api/v3/downloadclient/testall` returned isValid:false with
"Authentication Failure" on BOTH Sonarr and Radarr, so Completed Download
Handling was silently broken, and no human had a stable login either.
Mechanism: the SOPS/age qbittorrent-bootstrap Secret is the single source of
truth; a lifecycle.postStart hook derives qBittorrent's own stored form
(PBKDF2-HMAC-SHA512, 100000 iterations, 16-byte salt, 64-byte digest) and
asserts it into the conf on EVERY container start, then logs in to the WebUI to
prove it. Level-triggered like the repo's CronJob reconcilers, but triggered by
the container start because the failure it closes IS a container start: a cron
would leave a window after every restart in which the app serves a random
password, and would have to edit an RWO PVC from outside, under a running
process. AuthSubnetWhitelist was rejected — authentication is not weakened.
Two real bugs found in the WIP, both the postBuild-substitution trap:
* PORT came from a doubled-dollar WEBUI_PORT expansion. Correct on the cluster
(envsubst collapses it) and the renderer accepts it, but run any other way
bash reads the doubled dollar as its PID, so every curl got an unparseable
URL and the hook logged "the WebUI rejected the durable credential" three
times having never asked it anything — see the marker log at 05:35Z. Now a
literal PORT=8080; the script contains no braced expansion at all and
behaves identically however it is run.
* One shared wait budget covered both the s6 wait and the API wait, so the
first consumed all 90s and attempts 2-3 fired instantly. Separate counters
now; "the API never answered" is reported as a distinct fault from "the
password was rejected"; and a rejection restarts qbittorrent-nox so a retry
can actually change its outcome.
apps/wave-0/kustomization.yaml did not list the SOPS secret, so Flux would never
have applied it and the adoption of the live-only, kubectl-client-side-apply
qbittorrent-bootstrap Secret would silently not have happened. Listed now; the
decrypted values are SHA-256-identical to the live Secret's, so adopting it
cannot change the password.
Proven on prod: testall flipped isValid false -> true on both apps; after
deleting and recreating the pod the conf still held both keys, the WebUI answered
Ok. to a login with the credential decrypted from git, and both arrs still
reported isValid:true. The post-envsubst manifest was audited as `flux build
kustomization` emits it (no doubled dollar, no braced expansion, PORT=8080,
bash -n clean) and the hook's write path was exercised in-container against a
scratch conf: exit 10 when the keys are absent, exit 0 and a byte-identical file
on a second run, digest verified by an independent PBKDF2.
NOT proven, and recorded as such in DECISIONS-LOG: the hook has not yet run as a
real postStart on prod — the Deployment is Flux-owned from main and both routes
to the live object were refused by the session's permission classifier. Verify
on merge that the marker log gains a VERIFIED line on the next pod start.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
aff80d1d7a |
docs(pm): R-TLS-02 needs the owner — the drafted spec amendment, recovered and restated
The R-TLS-02 analysis and its drafted spec amendment were written on
feat/coverage-debt-paydown (b37defe, 2026-09-07) and lived ONLY in that branch's copy of
DECISIONS-LOG.md. That branch's tree is ~107 commits behind main and must never be merged,
so the amendment went out of view with it — the second time this decision has been raised
and lost. Recovered, re-checked against today's main, and written where it cannot vanish
with a branch.
THE SUBSTANCE IS UNCHANGED. R-TLS-02 says certificates MUST be automatically "issued,
renewed, and revoked without manual steps". Issuance and renewal hold. Revocation is not
implemented and is not implementable with the stack ADR-0006 selects: cert-manager has no
revoke-on-delete, no Certificate field requesting revocation, and no controller that calls
ACME revokeCert, and neither ADR-0006 nor docs/08 specifies a mechanism.
enableCertificateOwnerRef is NOT the remedy — it garbage-collects the Secret, which is
cleanup, not revocation — and it is a global controller flag that would drop the Gateway's
wildcard-home-tls Secret on any future Certificate rename/prune, so it needs its own risk
assessment rather than a side effect of a test task. Still not set.
WHAT CHANGED SINCE THE FIRST ESCALATION, and why it now matters more: R-TLS-02 is no
longer a COVERAGE gap (l0-static/tls-lifecycle-automated + l2-behavioral/
cert-lifecycle-automated both carry the label and main counts it covered), and the L0 check
is honest about the boundary in its own verdict line ("'revoked' is NOT proven at any
layer"). But the L2 test deletes a Certificate and requires its Secret to disappear, and
its failure message says "set enableCertificateOwnerRef: true ... (owner decision — WS-TEST
does not weaken the assertion instead)". That assertion is written to FAIL on a live run
until the owner decides. The decision is now arriving as a red test rather than as a
question, which is exactly how it gets weakened by whoever meets it next.
Two options are recorded verbatim for the architect, in the drafted preference order:
amend the requirement so revocation is an explicitly manual, documented break-glass action
(which also makes enableCertificateOwnerRef a CONSEQUENCE of the requirement), or keep the
text and record a reviewed DOCUMENTED_GAPS entry naming the clause, the reason and the
removal condition.
NOT APPLIED, and not appliable by an agent: the spec repo is read-only under R-TEST-06.
This is a request for a human decision, stated plainly as one.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
f4f6e5f7a3 |
docs(pm): handoff — five WIP branches preserved after a session rate limit
Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
87c3d83dff |
fix(conformance): placeholder-guard-coupling actually RUNS — it was skipping forever
`conformance/tests/l0-static/placeholder-guard-coupling` (R-ENV-02, R-STATE-03) exited 3 SKIPPED on every gate run since it landed, because its check keyed its whole existence on `bootstrap/standup.sh`. Measured, not assumed: that file has NEVER been on `main` — there is no deletion commit for it anywhere; it landed only on unmerged prod-standup branches (`work/prod-standup-campaign`, `prod/*`, `feat/prod-hardening`). It was planned and never merged, not removed. SKIPPED is not green, and a permanently-skipped incident-derived test is the quiet version of the failure it was written to prevent. The skip was also hiding a RED. Run against the real `standup.sh` from `work/prod-standup-campaign`, the old check FAILED 2 of 34 assertions — false positives from treating the empty string as placeholder-shaped for knobs cluster-vars authors empty on purpose (`BACKUP_CREDENTIALS_SUFFIX`, `VELERO_BACKUP_PREFIX`, both `""` on dev and both documented there as load-bearing). It would have false-blocked the merge landing its own input. NOT deleted — the hazard is not gone. Split into two halves, one of which needs no guard: HALF A (always runs): no environment overlay ships a placeholder-shaped EFFECTIVE cluster-vars value; an empty effective value is STATED by the overlay rather than inherited in silence; a LOUD base sentinel is explicitly overridden by every overlay. This is the 2026-09-05 outcome asserted at the source instead of at the guard — if prod's S3_BACKUP_ENDPOINT is never a placeholder, velero cannot be pointed at a hostname that does not exist. Input surface is `clusters/`, all of it on `main`. HALF B (self-activating): the real predicate extracted and executed against those same values, the consumer trace, and the mid-string anchoring property. The check NEVER exits 3 again, and the guard is DISCOVERED by content rather than hardcoded: any tracked `*.sh` defining `<name>_is_placeholder()` is found, so a guard landing at `bootstrap/standup.sh` — or moved into a sourceable library, the remedy the check already recommends — activates HALF B with no edit to the test. Two files defining it is a FAIL. When no guard exists the gap is printed loudly every run instead of swallowed as a skip. LOUD vs EMPTY is the correction that makes it honest in both modes: a loud placeholder is always a defect in an overlay, while `""` is held to the explicitness rule instead — and still counts as a placeholder for a knob a guard covers, because a `""` case-arm is the guard author's declaration that empty is not a usable target. Proof, all reverted: 84 assertions and exit 0 on `main`; prod S3_BACKUP_ENDPOINT set to the literal `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` FAILS (A1); deleting dev's explicit VELERO_BACKUP_PREFIX op so it inherits `""` FAILS (A2); a LOUD base BULK_NFS_SERVER with prod's override deleted FAILS 2/86 (A1+A3); and with the real `standup.sh` present, reverting `s3_is_placeholder` to the `TBD-*` PREFIX glob FAILS 12/110 (B3 anchoring). With the pristine `standup.sh` the check PASSES at 110 assertions where the old one failed 2 of 34. L0 layer: 14 checks, 14 PASS, 0 SKIP, 0 FAIL (was 13 PASS / 1 SKIP). Coverage unchanged — R-ENV-02 and R-STATE-03 both still list this test, 49/51 with the same 2 gaps. Also: * `ci/touched-requirements.sh` — `bootstrap/*` -> R-STATE-03 R-GITOPS-01 is KEPT, its stale comment corrected. It claimed standup.sh "carries the placeholder guards", describing a file that is not here; R-STATE-03 there is ANTICIPATORY. Deleting the entry would leave the change that lands a placeholder guard selected by nothing. * `ci/tests/touched-requirements.test.sh` — new path-map section (12 assertions, was 8) locking `bootstrap/*` and `clusters/*`, so removing either entry is a test failure rather than a silent loss of selection. All three ci/tests pass (12 + 31 + 15). * `docs/BULK-NFS-STORAGE.md` §7 — its "nothing in the gate currently refuses a placeholder-shaped overlay value" is no longer true; corrected. * Scope re-audited: still deliberately requirement-scoped, not whole-repo. Every path that can break it attributes a requirement it declares. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
2f38b8156b |
fix(arr): wire qBittorrent as a download client + Remote Path Mappings
qBittorrent reports finished releases at /downloads/<name>; sonarr and radarr
see the same inodes at /media/downloads/<name>, so Completed Download Handling
could never resolve what the client handed it. Configured, on prod, via each
app's own HTTP API:
* download client qBittorrent -> qbittorrent.qbittorrent.svc.cluster.local:8080
(it did not exist in EITHER arr: both /api/v3/downloadclient returned [])
* Remote Path Mapping /downloads/ -> /media/downloads/ for that host
Both built from each app's /downloadclient/schema, matched on identity before
writing, and re-running creates nothing. downloadclient/testall reports
isValid=true in both. Proven with a negative control: Sonarr's queue showed
outputPath=/downloads/... with the mapping deleted and /media/downloads/...
with it present, for the same completed torrent.
This state lives in sonarr.db / radarr.db on the config PVCs, NOT in git — a
rebuild from git alone does not restore it. Recorded as a wart, with path
agreement (repointing SavePath to /media/downloads) recommended as the durable
fix and put to the operator rather than taken unilaterally.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
505083fded |
feat(cnpg): alert when >1 CNPG primary shares a node — the drift CNPG cannot prevent
`instances: 3` + `podAntiAffinityType: required` fixed the INSTANCE spread but moved
the PRIMARY concentration from k3s-prod-2 to k3s-prod-1: all three primaries
(keycloak-db-2, nextcloud-db-2, semaphore-db-2) ended up on one node, so losing it
meant three simultaneous failovers.
There is NO declarative fix, established from the CRD installed on prod (CNPG 1.30.0,
clusters.postgresql.cnpg.io/v1) rather than from memory. Every spec.affinity field and
spec.topologySpreadConstraints applies to "all the cluster's pods" and is not role-aware;
spec.replica.primary names a Cluster object in a distributed topology, not a node;
primaryUpdateMethod/switchoverDelay/primaryLease govern how a promotion happens, never
where. `targetPrimary` exists only under STATUS, never spec — so `cnpg promote` is an
imperative status patch by construction and there is nothing to put in git. The obvious
workaround is worse than nothing: pod anti-affinity is only ever
...IgnoredDuringExecution while CNPG promotes by relabelling a RUNNING pod, and with
3 instances on 3 nodes it would leave the third instance unschedulable.
So this commits the part that IS durable — visibility — and rebalances live.
CNPGPrimariesCoLocated joins CNPG's own answer to "who is primary" (the <cluster>-rw
Service selects only the primary) against the node: kube_endpointslice_endpoints carries
endpoint_nodename + targetref_name, which two label_replaces lift into node/pod. Neither
family works alone — cnpg_* metrics have no node label, and kube_pod_labels is EMPTY here
(no kube-state-metrics label allowlist), so cnpg.io/instanceRole is not in Prometheus at
all. `* on (namespace, pod) group_left () cnpg_collector_up` keeps an unrelated `*-rw`
Service from triggering it, and uses the one metric present for EVERY instance rather than
the cnpg_pg_stat_* family that is missing for nextcloud-db.
A Ready-node gate stops this becoming another permanently-firing alert: with three
clusters on three nodes a node loss NECESSARILY co-locates two primaries, so the rule
only speaks when the spread is actually achievable, and re-arms when the node returns.
Proven both ways. Live: the expr returned {node="k3s-prod-1"} => 3 before, empty after,
with the inner join still resolving all three primaries (quiet because false, not blind);
loaded into prod's Prometheus as a throwaway `for: 1m` twin it went pending at +70s and
firing at +130s. alerts.promtool-test.yaml pins four cases under promtool 3.14.0 (the
exact version prod runs) and is mutation-checked — dropping the join, dropping the gate
or shortening `for:` each makes a test FAIL.
Rebalanced live, one cluster at a time: keycloak-db -> k3s-prod-2, nextcloud-db ->
k3s-prod-3, semaphore-db deliberately LEFT on k3s-prod-1 (lightest writer, and that node
has the known etcd/Longhorn I/O contention) — two switchovers, not three. Write outage
measured with a `select txid_current()` probe, not estimated: 9.4s and 9.43s, one
contiguous burst each. All three back to readyInstances: 3 with
synchronous_standby_names = ANY 1 (...) and both standbys streaming/quorum.
The placement is CURRENT, NOT DURABLE, and the alert is the point: the next failover or
rolling update can re-concentrate the primaries, and now that gets reported instead of
discovered by hand.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
5e7ab65c65 |
docs: kubectl get backup lies on this cluster — ambiguous Backup short name
Three API groups register kind Backup (longhorn, cnpg, velero). The bare short name resolves to Longhorn's empty set, so kubectl reports absence while 27 CNPG backups and live Velero backups exist. A false negative on 'do I have backups' is the worst possible time to be wrong, so record the rule: fully-qualified resource names only. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
ebce6dfabb |
docs: the Flux->Gitea allowance is rendered and Flux-adopted — debt closed
Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a863542bf7 |
docs: record the apex migration, the GitOps deadlock, and the live-only policy
Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
f7d4648851 |
docs(keycloak): record the URL-reconciliation decisions
Records K1-K9: extend-vs-sibling and why, the generated+gated desired state, verbatim token pass-through, per-pass fail-closed, order-insensitivity, unmanaged-means-untouched, public clients' URLs managed, one alert pair not two, and the mutation harness. Also records a process note: my first pass reported three false findings (reconciler absent from main, import not templated, BASE_DOMAIN dead) because I diffed against origin/main -- origin is dev's Gitea, stale at |
||
|
jakman85andClaude Opus 5
|
16c6234696 |
feat(monitoring): alert when egress steering goes stale, not just when tun0 dies
`VPNTunnelDown` goes green the instant a tun0 exists, which is not the same as vpn-class pods having egress: Cilium caches (egress IP, egress ifindex) at first resolution, so after a reconnect the BPF egress map points at a destroyed interface and the packets are dropped while tun0 looks perfectly healthy. Dev sat in exactly that state for ~80 minutes with nothing alerting. Two rules close it: * VPNEgressSteeringStale (critical, for: 15m) — fires when the live tun0 was re-created more times in the last hour than the gateway node's `egress-uplink` label changed, i.e. Cilium was never told about a reconnect. * VPNEgressUplinkLabelMissing (warning, for: 30m) — the watchdog for the watchdog: a gateway node whose uplink label never reaches kube-state-metrics is a cluster where the rule above is inert. Both gated on a gateway node existing, reusing `VPNTunnelDown`'s `kube_node_labels` guard — a cluster with no gateway cannot be made to fire them. The direct label-vs-map comparison is NOT available and nothing was invented to pretend otherwise: no `cilium_*`/`hubble_*` series exist on either cluster (the cilium-agent exposes no metrics port, there is no ServiceMonitor), and Cilium OSS publishes no series carrying egress-map contents at all — the map is readable only via `cilium-dbg bpf egress list`. PromQL also cannot compare a metric value (the live ifindex) against a number inside a label value, so the level-triggered equality test is not expressible; the same invariant is counted over a window instead. Both limits, and the textfile-collector/black-box-probe routes that would lift them, are written down in the rule comments and the DECISIONS-LOG. `egress-uplink` added to kube-state-metrics.metricLabelsAllowlist in the same kustomization, because without it the first rule has nothing to compare. Verified read-only against LIVE prod (port-forward + /api/v1/query), rendered expr extracted from `kubectl kustomize platform-config`: empty when healthy; empty when evaluated at a timestamp before prod had a gateway node; and FIRING when pointed at a label that genuinely failed to follow prod's real 128 -> 129 tun0 flap earlier today. Nothing on prod was disturbed to produce that. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
53e09a2610 |
docs(vpn): dev recovered the minute prod left the shared PIA server
Closes the loop on the collision entry with the recovery evidence rather than leaving it as an open question for the operator: dev's last Inactivity timeout is 23:05:06 UTC — the minute prod switched to .135 — and at 23:12:03 that same session was still the live one, seven minutes unbroken against a two-minute cycle, 4/4 ping replies at 146 ms, 0% loss, and vpn-class pod egress back to 158.173.241.36 on three consecutive probes. Co-Authored-By: Claude Opus 5 <[email protected]> |