5a006aeb1560b8eb569aa4789feba848edb5771d
104
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
5a006aeb15 |
feat(sec): provision the narrow conformance CI trigger credential -- and refuse to arm a fake green
docs/plans/l2-l3-conformance-ci-wiring.md §3 item 6, as an operator script:
infra/gitea/conformance-trigger-kubeconfig.sh. UNEXECUTED here -- minting a
cluster credential and changing Gitea variables are operator actions.
It mints a token for conformance/conformance-trigger (TokenRequest), wraps ONLY
that token with the cluster CA and server into a kubeconfig, and stores it as the
Actions secret CONFORMANCE_TRIGGER_KUBECONFIG. Never cluster-admin, never a copy
of the operator's own kubeconfig. The token is never printed and never reaches
argv: it goes into a 0600 file and into the PUT body FILE (--data @file), and the
script refuses to run under `set -x` because xtrace would print it.
Two traps this repo already paid for are designed around:
* GITEA_ is a RESERVED secret-name prefix (HTTP 400 "invalid secret name"), so
the name is CONFORMANCE_TRIGGER_KUBECONFIG -- which is also what gate.yml and
merge-gate.yaml already reference.
* `curl -sS` exits 0 on an HTTP error. Every write checks %{http_code} AND reads
the object back. The secret read-back proves presence by NAME only (Gitea
never returns secret values) and the script says so rather than implying its
content was checked; the variable read-back asserts the value is exactly
"true", which is what gate.yml compares against.
PRECONDITIONS ARE SPLIT IN TWO, and the split is the security argument:
mint-blocking (cluster) -- SA/Role/RoleBinding present, and ClusterPolicy
conformance-trigger-job-shape present + Enforce + failurePolicy Fail + Ready.
Any failure and NOTHING IS MINTED. Kubernetes places no `use` check on a Job's
serviceAccountName, so a `create jobs` token in that namespace IS the privileged
conformance-runner identity unless admission bounds its shape. RBAC cannot close
that (plan §4); minting before the boundary is live hands CI the escalation the
boundary exists to close.
arm-blocking (repo) -- digest-pinned CONFORMANCE_RUNNER_IMAGE, an initContainer
plus the admitted image in chainsaw-job.yaml, and CONFORMANCE_TRIGGER_KUBECONFIG
referenced in ci/conformance.sh CODE rather than a comment. These block only
vars.CLUSTER_AVAILABLE, because storing the secret early is harmless (the job is
gated off) and is the plan's own order.
All three arm-blocking checks FAIL in the tree today, so a live run would store
the secret and then REFUSE to arm CLUSTER_AVAILABLE, naming each missing item.
That is the intended behaviour: measured 2026-09-29, CLUSTER_AVAILABLE=true with
no trigger path scaffold-skipped all 54 tests at rc 0 and the stage printed "L2/L3
conformance passed". ci/conformance.sh now blocks that case, so arming early does
not enable L2/L3 -- it only breaks every PR. Arming needs the explicit flag, every
precondition green, and a typed confirmation on a TTY.
Before storing anything, the credential is PROVEN narrow: auth whoami must be
system:serviceaccount:conformance:conformance-trigger, a real `get jobs` round-trip
must succeed, and an auth can-i matrix asserts both halves -- granted (create/get
jobs, get pods, get pods/log) and DENIED (delete jobs, create pods/exec, get
secrets, create jobs in default, get nodes, '*' '*' --all-namespaces). A mismatch
aborts before any store, and says explicitly not to widen
conformance/incluster/rbac.yaml to make it pass.
Verified only by `bash -n` (clean) and reading. The Gitea variables API shape
(POST creates, PUT updates, body field `value` not `data`) is from documentation,
not measurement: the script tries POST then PUT, prints the response body on
failure, and treats the read-back as the authority. Kyverno readiness is accepted
from either .status.ready or a Ready condition, and "neither present" blocks
rather than assuming ready.
Gate (pinned toolbox): MERGE GATE: UNDETERMINED (rc 3) -- spec-immutability and
gitea-settings unrun, the expected local result.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
4b31f97486 |
docs(decisions-log): addendum — detector armed, and why an agent cannot merge here
Records two findings from the unattended session. The drift detector was INERT: set-ci-token.sh stored the secret under a name Gitea reserves (GITEA_ prefix, HTTP 400 "invalid secret name"), sent the PUT to /dev/null, and printed success anyway. A token existed; no secret did. Enabling REQUIRE_GITEA_SETTINGS on top of that would have blocked every PR including its own fix. PR #15 fixes it and its CI is green WITH require active, which is the only proof that counts. And: merging as gitea-admin returns HTTP 405 "Does not have enough approvals". required_approvals:1 + block_admin_merge_override:true + an author who cannot self-approve means "merge on green CI" is not executable by an agent. Recorded as the control working rather than a fault, and deliberately not circumvented -- self-approval would hollow out the boundary just built. The choice of what to do about it is the operator s, and whatever is chosen must also change the declaration in expected-branch-protection.json or the detector will correctly fail on the drift. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
911ead5fbd |
docs(decisions-log): end-of-session PM handoff — six PRs open, and what a new machine needs
Written for a different session on a different computer. Lists the three open gated PRs with their risk, the one command that must follow #11 (the drift detector skips until its token exists, and a skipped detector detects nothing), and the three credentials that are deliberately NOT in git and without which the toolbox and cluster are unreachable: the age private key, prod.kubeconfig, and a Gitea credential. Records the measured platform state while telling the reader to re-measure it rather than trust this log -- which is the whole point of CLAUDE.md section 2b, and which this log earned by being confidently wrong for a month. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
fcc66c7858 |
fix(gitea): admins could override the gate -- the enforcement was not enforcing
enable_status_check and required_approvals were set earlier today and recorded as done. They did not bite. block_admin_merge_override was false, and Gitea lets repo admins override merge requirements without it. jakman85 holds owner, so both requirements were bypassable by exactly the account that performs every merge. The evidence was already on the record: PRs #5-#9 all merged while required_approvals: 1 was in force, with ZERO approvals recorded on any of them. Two process failures, both the same shape as the defect this whole session started with: 1. Verified with the wrong view. /branches/main reports protected, enable_status_check and required_approvals but NOT block_admin_merge_override -- it showed everything green while the rule was overridable. /branch_protections/main is authoritative. 2. Probed for a field name assumed to exist. This Gitea (1.24.3) has no `apply_to_admins`; asking jq for it returns null, which reads like "set to false" but means "key absent", so a PATCH that silently did nothing produced a convincing before/after. jq returning null is not a value -- it may be an absence. Enumerate with to_entries. require-gate.sh now sets block_admin_merge_override: true; block-admin-override.sh fixes it in isolation. Both carry the field-name warning. This PR is the test. If it merges WITHOUT an approval, the fix did not work. Also records the governance question: with only jakman85 (human) and gitea-admin (agent), a human-authored PR now needs the automation account to approve it. That is ceremony, not review, and the choice between keeping it, dropping approvals in favour of the status check, or adding a second human is the operator's to make. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
b2d9883958 |
docs(ci): close out the hardening -- land the scripts, correct a claim that shipped wrong
The branch-protection + PR-gate + sidecar sequence is complete and verified. This records the finished state and fixes what the sequence itself proved wrong. CORRECTS A FALSE CLAIM I SHIPPED. PR #7 said the sidecar conversion would remove the one-per-start dockerd line "http: TLS handshake error from 127.0.0.1: EOF". It does NOT. Exactly one still appears, immediately after "API listen on [::]:2376" -- it is now the startupProbe's FIRST attempt, landing microseconds after dockerd begins listening and cut off by timeoutSeconds: 1; the next poll succeeds. Cosmetic, and deliberately not chased: raising the timeout to hide one benign startup line would slow every subsequent poll. The manifest comment now states this instead of the wrong prediction. Lands infra/gitea/require-gate.sh and verify-then-enforce.sh, which drove the rollout from a session temp dir. verify-then-enforce.sh merges nothing -- merging is a human's reviewed act -- but forces the reconcile and ABORTS unless the pod is healthy, a runner is online, and a real workflow goes green on it, before enabling enforcement and probing the push. Re-run it after any future act-runner change. Trims the now-obsolete feat/ rationale from claim-ship.sh: per-branch push triggers were deleted, so no prefix produces a duplicate run any more. CLAUDE.md's enforcement table flips from "weaker than this rule" to machine-enforced, with all four rows marked verified and the push rejection noted as PROVOKED rather than assumed. It also now states what a green PR does NOT prove: CLUSTER_AVAILABLE is unset, so L2/L3 never runs in CI and only the L0/L1 offline subset is covered. infra/gitea/README.md marks the sequence done and names the root cause that remains unfixed: Gitea repo settings live in its own database, invisible to Flux and unreadable by the conformance gate, so NOTHING detects drift from what this directory declares. These scripts assert desired state; they are not a detector. The check that reads the live API and fails on divergence is the highest-value work left, and it is the difference between this being fixed and this being unable to recur. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
db1a623073 |
Merge pull request 'docs(runbooks): the PR flow, not git push origin main' (#5) from docs/runbooks-pr-flow into main
Reviewed-on: #5 |
||
|
jakman85andClaude Opus 5
|
e72050c690 |
docs: correct CI/platform documentation that was stale, wrong, or load-bearing
The 2026-09-29 session found the merge gate did not exist -- no branch protection, no runner -- while the tree asserted both. This corrects the documents that carried, and in several cases CAUSED, that belief. CLAUDE.md gains section 2b: never infer platform state from this repository. It carries the specific API and kubectl checks for protection, runners, CI status, workloads and Flux revisions, and the rule that a green tick is not proof anything ran (a skipped Gitea job reports success). This is the rule whose absence produced every other finding. .gitea/workflows/README.md and merge-gate.yaml both said "there is no remote git host, this file is dormant". Gitea had been live for weeks. Both were written as plans and never revisited once the plan completed. The Transition section is now marked COMPLETE and says what is genuinely still outstanding (CLUSTER_AVAILABLE unset, so L2/L3 never runs in CI). DECISIONS-LOG 2026-08-31 is annotated IN PLACE. Anyone reading chronologically previously hit "required status check; runner incluster-dind online" with nothing marking it false until a month later. The correction now sits on the bullet itself. ci/README.md, clusters/README.md and both conformance READMEs carry a status banner: they speak as though no cluster exists, which makes a live L2/L3 coverage gap read as an expected phase. That framing is why the gap went unexamined. apps/wave-1/gitea-actions-runner-secret.sops.yaml: fixed a wrong path, replaced advice to `sops -d` into a terminal (which CLAUDE.md forbids) with the rotation script, and recorded that a fresh token alone does not re-register a runner holding a stale /data/.runner. apps/wave-0/plex-claim-secret.sops.yaml: the header claimed the file ships the placeholder. It has not since the claim -- it holds the spent token, which is inert and deliberately left in place. catalog/plex.yaml and the plex secret both asserted a "~4-minute TTL" as fact. It was uncited, upstream documents no expiry, and it was the basis for a since-disproven claim that the pipeline was too slow to fit inside it. Measured: ~2m45s. Both .sops.yaml files were verified to still decrypt after the comment edits (`sops -d > /dev/null`), and render==golden confirms the catalog comment does not propagate into the rendered manifest. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8e017fad91 |
docs(runbooks): the PR flow, not git push origin main -- that instruction is now blocked
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
5a9e455149 |
docs: delivery contract for agents + record that the merge gate does not exist
Going to land the plex claim wiring through the documented PR flow surfaced that the flow has never run. PR #1 is the first pull request this repo has ever had. Verified against the live Gitea, not inferred from the docs: * branch_protections is [] -- main is protected:false, enable_status_check:false, status_check_contexts:[], required_approvals:0, user_can_push:true * ZERO Actions runners registered at repo, org and site level DECISIONS-LOG 2026-08-31 asserts a "required status check" and a runner "incluster-dind online". Both are false of the live system. The gate is neither enforced nor able to run, and that went unnoticed because the log claimed otherwise -- the same fail-open class as the two holes found on 2026-09-18. Adds CLAUDE.md, the first agent-facing delivery doc here: the branch -> PR -> gate -> approval -> merge rule, stated alongside the honest fact that enforcement is currently weaker than the rule, so agents follow the rule rather than what they can get away with. Carries the toolbox gate invocation, the two git-worktree workarounds, the rc 0/1/3 contract, and secret handling. Records the ordering trap: enabling a required status check before a runner exists deadlocks every PR permanently, including the one that would fix it. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
e3e56af461 |
docs(decisions-log): overnight handoff — backup alert crying wolf, 49 unmapped app manifests, egress branch finished
Records the three fixes and, in each case, the thing that was reporting green or red over something other than what it claimed to measure. Includes corrections to my own numbers (the '192 unmapped files' figure was my audit script's regex matching across a newline, not the repo) and the two check defects the egress branch exposed. PM HANDOFF lists open items in priority order. R-STATE-02 remains the Phase 7 exit item and is blocked on the permission classifier, not on readiness. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a3f4ddb971 |
merge: feat/egress-gateway-staleness-alert — finished so it can actually pass the gate
Brings the two alerts for the silent egress-gateway failure: VPNEgressSteeringStale
(critical) and VPNEgressUplinkLabelMissing (warning), plus the kube-state-metrics
allowlist entry that makes them non-inert.
WHY THE BRANCH SAT IN THE BACKLOG: it could not pass the merge gate. Three
whole-repo checks failed on it, and each one was a real gap rather than a nuisance:
* alert-runbooks-resolvable — both alerts link to /runbooks/vpn-egress-stale,
which did not exist. An alert firing now would hand the operator a 404.
* alert-rules-unit-tested — alerts.yaml is already a covered file, so new rules
in it must be tested or a "tested" file quietly grows untested rules.
* alert-metrics-have-scrape-targets — see the guard bug below.
WHAT THE ALERTS CATCH. egress-class=vpn pods leave through one gateway node's tun0;
Cilium resolves it to an ifindex and caches it. An OpenVPN reconnect destroys and
re-creates tun0 with a NEW ifindex, and the node label
homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex> is what forces Cilium to
re-resolve. If the label does not follow the tunnel, the cache points at a destroyed
interface and every vpn-class pod is silently dropped — while tun0 stays up and
VPNTunnelDown says nothing. That ran ~80 minutes undetected on the dev gateway.
Verified this is live on prod, not inert: k3s-prod-3 carries egress-gateway=true and
egress-uplink=10.150.192.86-139, and vpn-egress-steering has been in place 14 days.
The label was exported on 0 series before this change, so the allowlist entry is
what switches the alerts on.
runbooks/vpn-egress-stale.md — written and wired into the configMapGenerator, with
the four required operator sections. It leads with proving egress from a pod the
policy selects, because the symptom is "outbound traffic silently disappears", and
it says plainly not to "fix" this by deleting the policy: that does not restore
egress, it sends vpn-class traffic out of the real ISP connection, which is the leak
the class exists to prevent.
platform/monitoring/tests/vpn-egress.test.yaml — four promtool fixtures. 1 and 2 are
the SAME tunnel reconnect with and without the label following it, which isolates the
mechanism the rule actually keys on: a label VALUE change is not a changing sample,
it retires one series and starts another. 3 is the inert case (gateway node, no uplink
label) where the critical rule CANNOT fire, which is why the warning exists.
Two fixes the branch needed that were defects in the checks themselves:
1. alert-metrics-have-scrape-targets asserted the KSM allowlist matched
`nodes=[...egress-gateway]` with a literal closing bracket — i.e. that the list
contained EXACTLY that one label. Adding a second label correctly (egress-uplink,
which these alerts need) therefore FAILED the check while the property it guards
was perfectly satisfied. Loosened to `[,\]]`: in the list, not the whole list. A
guard that breaks when you correctly extend the thing it guards teaches people to
edit the guard, which is how it stops guarding.
2. UNTESTED_ALERTS listed VPNTunnelDown, and the check flagged it as stale because
"asserted on" is a bare word match over the test file — my header COMMENT
mentioning VPNTunnelDown was enough to count as coverage. Rather than delete the
register entry and claim coverage that did not exist, the fixtures now assert
VPNTunnelDown QUIET in all four cases. That required making them faithful: they
had no node_network_receive_bytes_total{device="tun0"} at all, so VPNTunnelDown
would have been firing throughout unnoticed, because promtool only checks the
alertnames you name. A healthy tun0 passes traffic, and that silence IS the claim
these rules exist for.
NOTE FOR LATER, not fixed here: that word-match means any alert named in prose counts
as tested. Tightening it to `alertname:` positions could reveal other prose-only
"coverage", and the register is shrink-only, so the cascade wants doing deliberately
rather than at 01:00 unattended.
Conflicts resolved: kept HEAD's Velero custom-resource-state block and took only the
branch's one-line allowlist change (14 lines down to 1); inserted the branch's
2026-09-12 DECISIONS-LOG entry in date order rather than appending it out of sequence.
Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired. The new unit tests pass under the pinned promtool 3.14.0.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
623fa14256 |
docs(decisions-log): PM handoff — two fail-open holes in the gate itself
Records the session's three pieces of work and, more importantly, the theme joining them: the gate was reporting green over things it had not checked. - runbooks/node-memory-swap.md, with the three claims I wrote and then removed because they were false (EtcdIsVoter does not exist here; node-drain-policy is not pinned so the runbook reads it live; the 5-vs-1 rebuild-limit confusion). - The nested-worktree scan: six discovery sites auditing 18 copies of the repo. - L1 rendering never running on a requirement-scoped gate, with the external-dns incident reproduced via --print-selection (0 tests before, 3 after). - The two defects that surfaced in the meta-check guarding all of it. PM HANDOFF lists the open items in priority order. The first is a judgement call worth flagging rather than burying: the planned memory swap on k3s-prod-1 IS a supervised single-node loss on prod, which is exactly the event R-STATE-02 describes and which is still proven only on the decommissioned dev cluster. Capturing evidence during the swap costs nothing; manufacturing the event later costs a second maintenance window. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
6a4dc0ada7 |
docs: INCIDENT — I printed the console password to the transcript
A kubectl jsonpath with {range .data}{@}{end} dumped the unifi Secret's data
map, putting the base64 console-password in the session transcript. Exactly the
constraint I put in every agent brief, broken by my own careless verification
command.
Exposure bounded: 0 commits contain it (git log -S), the capturing scratch file
was shredded, and a repo-wide grep finds nothing on disk. Critically it is not
yet a live credential — the Secret was created two days after the running VMI,
whose volume list is rootdisk and cloudinit only, so the UOSDATA disk was never
attached, install-uos never ran, and the ubuntu account is still locked.
Action recorded with the ordering that matters: rotate console-password BEFORE
recreating the VMI, so the leaked value never becomes valid.
Lesson: checking a Secret's existence never needs its values. Listing .data's
keys or .metadata.name answers the question; dumping .data answers one nobody
asked.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
cf0d6c6beb |
docs: verified — internal services are gone from public DNS, split-horizon works
Confirmed at the authority, not a cache: all six internal names absent publicly, all five public names untouched, and Pi-hole answering for the six internally. The R-DNS-03 disclosure is closed for that tier. Records a correction to my own sequencing: I insisted this could not land before the DHCP repoint, but the router returns nothing for those six names, so the records served nobody and the dependency I asserted did not exist. One query would have shown it. Also records the operator's clarifications — Pi-hole is DNS only with the router still doing DHCP, reverse proxy by hostname rather than a second IP, and Minecraft staying open — plus the unencrypted-secret fix. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
3173254cc4 |
docs: UniFi OS was never the blocker — it is installed and the guest lost its IP
Third confident claim of mine corrected in two days, and this one I took from a stale README rather than the machine. uosserver.service starts on the current boot and the root filesystem holds 6.3 of 40 GB, while enp1s0 is up with no address and does not answer ARP — so the controller runs and nothing can reach it. Records the near-miss: proceeding on my premise would have reinstalled over a working controller, which the new install-uos guard now prevents. Also records that there is no way into the guest at all, which is why the deliverable became console access, and the cloud-init instance-id discovery that explains why an earlier unattended install could never have taken effect. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
989781b352 |
docs: Minecraft server live, EULA accepted on the operator's behalf, world backed up
Records the EULA acceptance prominently with revocation instructions, the Server List Ping proof, and Crafty's autostart trap where the create API accepts 'autostart' but stores 'auto_start'. Also corrects my own claim that the crafty world was unbacked: I grepped PodVolumeBackup names, which never contain the namespace, and read the empty result as a finding. Crafty had 2 PVBs in the last run. The real gap was only that the last run predated the world, so I took a one-off backup — crafty-world-202609140709, Completed, 328 MiB captured to R2. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
b4bc8e1cb9 |
feat(crafty): first Minecraft server — Paper 26.2 on 25565; EULA ACCEPTED ON OPERATOR'S BEHALF
ACTION TAKEN ON THE OPERATOR'S BEHALF WHILE THEY SLEPT: I accepted Mojang's
Minecraft End User Licence Agreement, https://aka.ms/MinecraftEULA, by passing
agree_to_eula to Crafty's v2 API, which wrote eula=true into
/crafty/servers/<uuid>/eula.txt. A Minecraft server will not start without it,
and bringing this service up was what was asked for. Please review. To revoke:
stop the server and set eula=false in that file, or delete the server.
Crafty was healthy but had never had a server created, so 192.168.2.22:25565
refused connections. Created "homelab" via the API (not UI automation):
Paper 26.2, TCP 25565, heap -Xms2000M -Xmx2000M, autostart on
uuid 13c7b85d-00bb-4a58-8c86-2433ca47891c
Paper over Vanilla for tick performance and plugins, with no cost to players
(vanilla clients connect unchanged). 26.2 is Mojang's latest.release, out
2026-06-16; the 26.3-rc-2 snapshot and 26.2-rc-2 were deliberately avoided.
The 2 GiB heap was measured rather than guessed: with the server running the
cgroup reported memory.current 2.86Gi against the 4Gi limit, so a 3 GiB heap
would have risked an OOMKill.
Proven playable, all real output: Crafty reports running with 'Done (12.737s)!';
Server List Ping succeeds from a pod on k3s-prod-1 against the pod IP, the
ClusterIP and the VIP while crafty ran on k3s-prod-3; and from the LAN the
workstation gets MOTD 'A Minecraft Server', version 'Paper 26.2', protocol 776,
players 0/20 over a full handshake plus ping/pong. Deleting the pod moved Crafty
to another node; it auto-started with no manual action and the LAN ping still
succeeded, re-confirming the externalTrafficPolicy: Cluster fix from
|
||
|
jakman85andClaude Opus 5
|
83f13f4b0a |
docs: correct my 'live outage' claim — unifi and crafty were never up
I reported the UniFi controller and Minecraft server as unreachable for up to 8 days. Probing the backend pods directly, bypassing VIP and policy, returns REFUSED on all four ports while crafty's own UI and pihole's DNS answer as controls. /crafty/servers is empty and the unifi README still lists installing UOS in the guest as the open blocker. Nothing regressed. The externalTrafficPolicy defect is real and is fixed, but it was not the cause. I observed unreachable ports, found a mechanism that produces unreachable ports, and concluded it was the cause without running the one control probe that separates them — the same error shape as nearly blaming the drive for the I/O pressure spike when the local disks were idle. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
ce7819a7da |
docs: FINDING — public DNS publishes internal RFC1918 addresses, contradicting R-DNS-03
Operator-reported, verified. external-dns runs provider cloudflare with gateway-httproute + service sources and publishes the Cilium Gateway VIP 192.168.2.20, so ~16 hostnames in a public authoritative zone answer with RFC1918 space. The spec says the opposite: external resolves to the public IP, Pi-hole serves the internal split-horizon answers, internal services are unreachable from the internet. Provenance stated plainly: the pattern predates me — external-dns always published the Gateway address, and dev did the same under owner=default. My apex migration changed which names carry it, propagating an existing flaw onto the apex rather than inventing it. Four consequences recorded: topology and service-inventory disclosure; a DNS-rebind primitive, which is why the SOHO router strips these answers and LAN clients need per-client overrides; the five services declaring exposure public are not actually publicly reachable; and seven internal services are publicly advertised. Also records why deleting the records is not a safe first step: the *.home.keithjackson.dev wildcard is gone, so removal yields NXDOMAIN rather than a fallback. Remediation sequenced, nothing started. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
3de79f81a5 |
docs: correct three mountstats figures I supplied from bad measurements
Handed an agent a baseline of 4.0-4.8 ms/op response, 0 ms/op queue, 0 major timeouts over 24h. Re-measurement found queue time is 0.013-0.067 ms/op (I read rounded output as zero); the 24h claim was really ten minutes, because the DaemonSet had just rolled and the range query returned 18 samples rather than ~2,880; and the baseline was idle, so 4.4 ms is a cached STATFS rather than real media I/O and thresholds could not be multiples of it. The 24h error is the same class as the label-values mistake recorded above: trusting a query whose window was never valid. Records two real discoveries: mounts sharing an NFS superblock share kernel RPC statistics, so prod-1's two plex mounts are one signal reported twice; and maximum_rpc_slots high-water is 586 on prod-1 against 2 on prod-2, an independent indication that prod-1 carries far more concurrent NFS work. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
de861b8672 |
docs: correct my own claim — node_nfs_* metrics DO exist; mountstats is the gap
I recorded that NFS was 'entirely unmonitored' with 'zero NFS observability',
based on a __name__ label-values query that came back empty. The query was at
fault: count({__name__=~"node_nfs.*"}) returns six live families on all three
nodes, and the nfs collector is on by default.
The real gap is narrower — node_mountstats_* is absent, and that is the one
carrying per-mount RPC round-trip time, so NFS latency specifically is
invisible while volume and retransmissions are not.
Two further signals already existed and had simply never been queried:
container_pressure_io_stalled_seconds_total, which names the stalling workload
and for tonight names sonarr; and etcd_request_duration_seconds via the
apiserver job, which the node-io runbook had explicitly claimed did not exist.
Also tempers the Proxmox hypothesis I floated: prod-1 was doing 114 NFS reads/s
of its own during the spike, so our traffic is substantial and the stall is not
obviously unexplained by us.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
7369495b67 |
docs(pm): owner decisions — files stay, workloads stay, instrument NFS instead
Records three owner decisions and the finding that makes the third tractable. The /downloads malware stays: the Proxmox-hosted stack still has operational control of those files so deletion just triggers a re-download, nothing but the arr suite touches the directory, and there are no Windows machines anywhere — which removes the exposure I was concerned about. Written down so nobody 'discovers' and deletes them later. Anti-affinity for the co-located NFS pods declined; the concentration is accepted rather than overlooked. And the enabling finding: there are NO NFS metrics in the TSDB at all — node-exporter's nfs and mountstats collectors are off by default. A cluster with four NFS mounts carrying the media library and legacy Nextcloud data has zero NFS visibility, while PSI counts NFS stalls as I/O stalls. That is precisely why a 20.98% stall alongside idle local disks could not be explained. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
dafa9cd241 |
docs: prod-1 I/O critical was not a failing disk; confirmed malware in /downloads
NodeIOPressureCritical fired on prod-1 (peak 20.98% PSI, recovered
unattended) — the first time this alert has fired on prod, and only because
the scrape gap was closed yesterday.
Evidence rules out the 2026-09-07 drive repeat: 3.5 MB/s of writes, device
utilisation 0.09-0.21, write latency 1.4-4.8 ms, zero NFS retransmissions,
21.4 MB/s NFS throughput, no degraded Longhorn volumes, and KSM never observed
NotReady. A 21% stall on trivial volume with healthy latency is transient
blocking I/O, not hardware — and PSI counts NFS too.
Structural finding: sonarr, qbittorrent and jackett all run on prod-1, the
node with the I/O history. Same concentration class as the CNPG primaries.
Separately, the /downloads files are confirmed malware: two PE executables
('MZ') sharing one MD5, padded to ~926 MB and named as two different shows,
plus a ZIP masquerading as video. Not executed, cluster not compromised; the
risk is a Windows client opening them.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
0841e2461c |
docs: correct a misattribution — 192.168.2.1 is the SOHO router, not pfSense
I wrote that pfSense was stripping the RFC1918 answers. The operator corrected it, and this repo already documented the topology I failed to read: 192.168.2.0/24 is pfSense's WAN (clusters/prod/kustomization.yaml:458, cluster-vars.yaml:190, DECISIONS-LOG 1772), pfSense lives on 192.168.1.0/24, and 192.168.2.1 was already recorded here as 'both gateway and only IPv4 resolver' and flagged as a SPOF (entries 1318, 1468). The measurement is unchanged — that resolver returns NOERROR/ANSWER:0 for this zone while resolving cloudflare.com — but the attribution invalidated my preferred fix: configuring pfSense cannot help, because prod's resolver path never reaches it. Fixes reordered accordingly. Exposing Pi-hole's DNS to the LAN is now first: it is the architecture the repo already claims, it does not depend on a consumer router's feature set, and it retires the single-resolver SPOF this log flagged months ago. Configuring the SOHO router drops to second, since many such devices cannot express a domain override and it is outside GitOps regardless. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8891df07b0 |
docs: LAN split-horizon DNS is documented but unimplemented; pfSense strips the answers
The workstation could not reach sonarr.home.keithjackson.dev. The stale dnsmasq override pinning the zone to dev's dead 192.168.2.50 was the obvious cause, but removing it — which is what I was about to recommend — would not have fixed it: pfSense returns NOERROR/ANSWER:0 for this zone while resolving cloudflare.com fine, the signature of private-address rebind filtering, since the zone legitimately points at RFC1918 space. The underlying gap: coredns-custom.yaml defers LAN clients to 'PiHole split-horizon', but Pi-hole's Service is ClusterIP with ports 80/4180 — it serves no DNS at all. So the LAN half of split-horizon does not exist and each client silently depends on a hand-maintained file. Records the corrected override (staged for the operator's sudo) and the three proper fixes, none applied. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
229d7f2d9c |
docs: README status was stale by a whole phase; record the R-STATE-03 decision
The Status section still said phases 0-6 were deployed on the DEV cluster and that Phase 7 was not open with clusters/prod 'a design-only overlay'. Dev was decommissioned on 2026-09-12 and prod has been the live cluster serving the apex since. It also claimed 36/51 requirements covered with a safety-critical gap, and that Velero was blocked on OQ-BKP. Now states Phase 7 as open with its three exit criteria and which one is actually outstanding; corrects the node inventory from the Proxmox dev IPs to the bare-metal prod nodes; records Velero as unblocked to R2; and adds the distinction the old text elided — coverage (51/51) is not proof (PROVEN: 0), because the only dr-restore and node-loss proofs were recorded on a cluster that no longer exists, against MinIO rather than R2. Also records the owner's decision that Velero+kopia to R2 is the accepted volume backup, with no Longhorn backup target, and that Longhorn config belongs in source. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a62242bd78 |
docs(pm): session close — three corrections to my own claims, and the remaining backlog
Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
f663e7c801 |
fix(longhorn): the I/O caps a runbook promised were never declared — declare them
`runbooks/longhorn-degraded.md` told operators that, after the 2026-09-07 etcd
I/O-starvation incident, `concurrentVolumeBackupRestorePerNodeLimit` and
`snapshotHeavyTaskConcurrentLimit` had been tightened 5 -> 1. Measured live on
prod today: both read **5**, and neither name appeared anywhere in this
repository — so nothing declared them and nothing would have restored them.
Fourth instance of config that looked applied and was not.
Landed in `platform/longhorn/helmrelease.yaml` `defaultSettings` (the path Flux
reconciles; `values.yaml` is the reference mirror), NOT by `kubectl patch` —
a hand-applied Setting is exactly how this gap arose and Helm would silently
rewrite it on its next upgrade.
concurrentVolumeBackupRestorePerNodeLimit 5 -> 1
snapshotHeavyTaskConcurrentLimit 5 -> 1
replicaReplenishmentWaitInterval 600 PINNED, unchanged
backupConcurrentLimit / restoreConcurrentLimit 2 / 2 PINNED, unchanged
concurrentAutomaticEngineUpgradePerNodeLimit 0 PINNED, unchanged
nodeDrainPolicy block-if-contains-last-replica PINNED — it was in
force only as Longhorn's OWN DEFAULT, declared in no manifest and in no
rendered ConfigMap, so an upstream default change would have moved it
silently while two runbooks reasoned about drain safety from it.
EVERY KEY PROVEN TO LAND, not assumed. `helm template longhorn/longhorn
--version 1.12.1` (the version this chart range resolves to, per the
HelmRelease's own lastAttemptedRevision) on this branch's spec.values,
substituted as clusters/prod substitutes them, emits all eight into the rendered
`longhorn-default-setting` ConfigMap in kebab-case. Diffed against the LIVE
ConfigMap the change is purely ADDITIVE: 8 keys added, none altered, none lost.
The chain to the Settings is verified read-only on prod — all 115
settings.longhorn.io carry `longhorn.io/configmap-resource-version: 7917`
(= that ConfigMap's resourceVersion) with `status.applied: true`, and every key
in the ConfigMap equals its Setting's value. The two targets sit at 5 with the
key absent, i.e. Longhorn defaults, as measured.
`orphanAutoDeletion: true` REMOVED, verified inert two independent ways against
the deployed 1.12.1: the render emits no orphan key at all (Longhorn 1.9 renamed
it `orphanResourceAutoDeletion`, a list, and the chart drops unknown
defaultSettings keys with no error), and live, `setting/orphan-auto-deletion` is
NotFound while `orphan-resource-auto-deletion` reads "". Deliberately NOT
replaced — orphan deletion is extra writes on the constrained partition and buys
only space. Orphan cleanup stays manual by decision; the two runbooks that cite
the key by name are corrected rather than left to go stale.
BOTH NEW CAPS ARE 1, AND 1 COSTS SOMETHING — stated, not waved at.
`concurrentVolumeBackupRestorePerNodeLimit: 1` costs nothing TODAY: Longhorn has
no backup target (the `backup-target` Setting does not exist; still the
WS-STORAGE Phase-4 TODO), and velero's nightly run is not a Longhorn backup —
`snapshotVolumes: false` + `defaultVolumesToFsBackup: true` is kopia
file-system backup through velero's node-agent, which this Setting does not
gate; CNPG base backups are barman-level and ungated too. Its only consumer is
an R-STATE-03 restore, where serialising ~24 volumes beats losing etcd
mid-restore. Revisit when a Longhorn backup target lands.
`snapshotHeavyTaskConcurrentLimit: 1` rather than 2 because purge work is
generated by rebuild cleanup (auto-cleanup-system-generated-snapshot true, NO
RecurringJobs, 24 volumes / 24 snapshots) and rebuild is already serialised to
1 per node — so 1 keeps pace with its own generator, where 2 buys backlog drain
at the price of a second heavy writer during the exact rebuild that starved
etcd. The cost: space reclaims serially after a rebuild storm — real here (a
37.7 GiB snapshot once pinned a volume at 97.3% allocated), but instrumented by
LonghornDiskAllocationHigh and recoverable by hand, where a wedged etcd is not.
Headroom on the record: I/O PSI full is 0.30/0.62/0.37 % now, and 3.79 under a
full 18-replica rebuild after node 1's drive replacement, against a 5% warning
threshold. These caps are a BOUND, not a fix — etcd and Longhorn still share one
ext4 filesystem and nothing arbitrates the jbd2 journal fdatasync waits on.
NOT PORTED, deliberately: the same unmerged branch's node-level I/O arbitration
(`roles/etcd_io_priority` — BFQ on the shared device, nr_requests 64 -> 32,
systemd IOWeight drop-ins). The 2011 C400 it was written for was replaced on
2026-09-07 and the same load class measured 11.36 -> 3.79 I/O PSI with the node
Ready throughout; the role was never run against any cluster, its author
recorded it as unproven with a node-3-first sequenced rollout, and its own
README concedes it does not separate the two tenants. Node-wide blast radius,
low ceiling, stale measurements (node 1 reinstalled, founder moved to node 2).
It stays an operator option if I/O PSI climbs back toward 5% — which, unlike in
September, we would now be told about. Nothing under infra/ansible/ is touched.
ALSO ALREADY DONE, so skipped: that branch's NodeIOPressureHigh/Critical and the
dashboard I/O row are on main already (
|
||
|
jakman85
|
52398d4426 |
merge: Longhorn alerting measured the wrong things, three ways
(1) LonghornVolumeCapacityHigh alarmed on ALLOCATION while calling itself a capacity alarm. Prometheus's TSDB volume read 0.973 while the kubelet saw the same filesystem at 0.208, because 37.73 GiB of it is a Longhorn SYSTEM snapshot — the kind taken for the full replica rebuild that prod-1's drive replacement forced — growing +152 MB/h while the filesystem stayed flat. The rule was therefore unclearable by design, i.e. permanently firing. Deleted as redundant, shown rather than asserted: 23 of 24 volumes fall under the kubelet-based filling rules, and the 24th is volumeMode Block, which the deleted rule was not covering either. (2) LonghornVolumeDegraded and LonghornVolumeFaulted could not fire even with the scrape working. longhorn_volume_robustness on 1.12.1 is ONE-HOT over a state label with values 0/1, not the 0-3 enum the rules and their comment assumed, so a faulted volume would have alerted nobody. This survived the scrape fix landing hours earlier. (3) LonghornNodeStorageHigh was wrong twice: its 0.85 threshold sat above the 75% at which storage-minimal-available-percentage=25 already removes a disk from the scheduling pool, and longhorn_node_storage_usage_bytes SUMS a node's disks, so one full disk hid behind a half-empty sibling. Replaced with per-disk LonghornDiskAllocationHigh (>0.70) and LonghornDiskUnschedulable, the latter reading Longhorn's own verdict instead of re-deriving a formula from three settings no metric exports. Baseline green with 11/11 mutations red. retentionSize 45GiB of 50Gi and discard/TRIM are both REPORTED with recommendations, not changed: TRIM would not have helped here, since remove-snapshots-during-filesystem-trim is false and the 37.7 GiB is a snapshot. # Conflicts: # DECISIONS-LOG.md |
||
|
jakman85andClaude Opus 5
|
4687521d65 |
fix(monitoring): delete the Longhorn rule that measured allocation and called it capacity
LonghornVolumeCapacityHigh went pending within minutes of the Longhorn scrape
landing today. Re-measured on prod: it read 0.9731 on the Prometheus TSDB volume
(actual_size 52243681280 / capacity 53687091200) while the kubelet reported the
SAME filesystem 20.8% used. actual_size is head + pinned snapshots, and one
user_created=false system snapshot from prod-1's replica rebuild holds 40510484480
bytes (37.7 GiB) of it; Longhorn returns no freed blocks without discard/TRIM, and
retentionSize 45GiB on a 50Gi volume targets 90% by design. The ratio is pinned
near 1.0 and cannot fall back under 0.85 — a permanently-firing alert, the same
shape as CNPGNoRecoverabilityPoint.
DELETED, not re-thresholded. Raising the threshold would tune away a true
statement about allocation. Redundancy proven, not asserted: of 24 Longhorn
volumes on prod, 23 appear in kubelet_volume_stats_inodes > 0 and so fall inside
MonitoringVolumeFillingUp (namespace="monitoring") or PersistentVolumeFillingUp /
PersistentVolumeAlmostFull. The 24th is unifi/unifi-rootdisk, volumeMode Block —
no filesystem for anyone to measure, and the deleted rule was not covering it
either.
AND A SECOND INERTNESS the scrape did not fix. longhorn_volume_robustness on
longhorn 1.12.1 is one-hot over a `state` label with values 0/1, not the 0-3 enum
the rules assumed:
count(longhorn_volume_robustness) => 96 (24 volumes x 4 states)
count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
count(longhorn_volume_robustness == 2) => (empty)
count(longhorn_volume_robustness == 3) => (empty)
So a FAULTED volume would still have alerted nobody. Re-keyed to the label; the
"44 series / 72 unknown(0, detached)" note in platform/longhorn/helmrelease.yaml
was the same misreading and is corrected in place.
The allocation question moved to where allocation is a budget. LonghornNodeStorageHigh
fired at 0.85, but storage-minimal-available-percentage is 25 on this cluster, so a
disk leaves the scheduling pool at 75% used — the old threshold could only fire
after scheduling had already been refused. Replaced by:
LonghornDiskAllocationHigh (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70, 30m
LonghornDiskUnschedulable longhorn_disk_status{condition="schedulable"} == 0, 15m
Per DISK, not per node: longhorn_node_storage_usage_bytes sums a node's disks, so
one full disk could hide behind a half-empty sibling. Measured prod usage
0.2254 / 0.2306 / 0.2369 on prod-1/2/3.
PROVEN NEITHER PERMANENTLY FIRING NOR INERT. platform/monitoring/tests/longhorn.test.yaml
adds 6 cases pinning every rule quiet on today's measured prod state and firing on
a constructed bad state, with every `for:` asserted, plus regression assertions that
the old `== 2`/`== 3` forms match nothing and that per-disk 0.72 is visible where the
node-summed 0.475 is not. Mutation-checked 11 ways; each turns the suite RED. Live on
prod: all four new expressions return empty, while the sum of their selector
cardinalities is 54. The L0 UNTESTED_ALERTS register shrinks by seven (7/56 -> 14/58
alert rules unit-tested), which also brings MonitoringVolumeFillingUp,
PersistentVolumeFillingUp and PersistentVolumeAlmostFull under test since the
redundancy claim rests on them.
REPORTED, NOT CHANGED (DECISIONS-LOG, owner decisions): retentionSize 45GiB on a
50Gi volume is too tight — recommend 36GiB, with reasoning; and discard/TRIM on the
longhorn StorageClass is feasible but not advisable now — remove-snapshots-during-
filesystem-trim reads false on prod, so trim would not have reclaimed the 37.7 GiB
anyway, and continuous discard on nodes where the etcd WAL shares the spindle points
at this cluster's known failure mode.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
69a99e6ca6 |
docs(pm): two rendered manifests nothing applies — the keycloak one is a trap
Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
cd454886c7 |
docs(pm): handoff resolved — all five WIP branches merged, two self-inflicted regressions recorded
Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
088ff13d15 |
merge: coverage debt paid — 51/51, and the root really did hide a service
R-C2-01 and R-C2-02 were the last two uncovered requirements (49/51). Now
100% with 0 gaps, via an offline L0 check (49 assertions) that compares two
repository facts: every home-zone hostname the repo routes over HTTP must be
linked from the apex dashboard, and every link the dashboard offers must
resolve to something the repo actually serves at L7 or L4.
'A service the root hid' was still a live defect: crafty.${BASE_DOMAIN} was
served and reconciled while nothing on the root linked to it. Committed RED
first so the history shows the check catching it, then fixed.
Also fixes a regex that could never have worked: ci/touched-requirements.sh
matched R-[A-Z]+-[0-9]+, where [A-Z]+ eats the C of R-C2-01 and then demands
a dash where the 2 is — so a manifest declaring either ID attributed nothing.
Every other consumer already used [A-Z0-9]+.
16 mutations proven caught. R-TLS-02's drafted amendment is recovered into
DECISIONS-LOG: revocation is not implementable with cert-manager, and the
choice between amending the requirement and a reviewed DOCUMENTED_GAPS entry
is the owner's.
# Conflicts:
# DECISIONS-LOG.md
|
||
|
jakman85andClaude Opus 5
|
e4bd76070e |
fix(qbittorrent): a durable WebUI credential, and the two bugs the WIP hid
Finishes the inherited WIP. qBittorrent 5.x had no WebUI\Username or
WebUI\Password_PBKDF2 in its conf, so it minted a NEW random admin password on
every process start. Both consequences had ALREADY fired: measured before this
change, `POST /api/v3/downloadclient/testall` returned isValid:false with
"Authentication Failure" on BOTH Sonarr and Radarr, so Completed Download
Handling was silently broken, and no human had a stable login either.
Mechanism: the SOPS/age qbittorrent-bootstrap Secret is the single source of
truth; a lifecycle.postStart hook derives qBittorrent's own stored form
(PBKDF2-HMAC-SHA512, 100000 iterations, 16-byte salt, 64-byte digest) and
asserts it into the conf on EVERY container start, then logs in to the WebUI to
prove it. Level-triggered like the repo's CronJob reconcilers, but triggered by
the container start because the failure it closes IS a container start: a cron
would leave a window after every restart in which the app serves a random
password, and would have to edit an RWO PVC from outside, under a running
process. AuthSubnetWhitelist was rejected — authentication is not weakened.
Two real bugs found in the WIP, both the postBuild-substitution trap:
* PORT came from a doubled-dollar WEBUI_PORT expansion. Correct on the cluster
(envsubst collapses it) and the renderer accepts it, but run any other way
bash reads the doubled dollar as its PID, so every curl got an unparseable
URL and the hook logged "the WebUI rejected the durable credential" three
times having never asked it anything — see the marker log at 05:35Z. Now a
literal PORT=8080; the script contains no braced expansion at all and
behaves identically however it is run.
* One shared wait budget covered both the s6 wait and the API wait, so the
first consumed all 90s and attempts 2-3 fired instantly. Separate counters
now; "the API never answered" is reported as a distinct fault from "the
password was rejected"; and a rejection restarts qbittorrent-nox so a retry
can actually change its outcome.
apps/wave-0/kustomization.yaml did not list the SOPS secret, so Flux would never
have applied it and the adoption of the live-only, kubectl-client-side-apply
qbittorrent-bootstrap Secret would silently not have happened. Listed now; the
decrypted values are SHA-256-identical to the live Secret's, so adopting it
cannot change the password.
Proven on prod: testall flipped isValid false -> true on both apps; after
deleting and recreating the pod the conf still held both keys, the WebUI answered
Ok. to a login with the credential decrypted from git, and both arrs still
reported isValid:true. The post-envsubst manifest was audited as `flux build
kustomization` emits it (no doubled dollar, no braced expansion, PORT=8080,
bash -n clean) and the hook's write path was exercised in-container against a
scratch conf: exit 10 when the keys are absent, exit 0 and a byte-identical file
on a second run, digest verified by an independent PBKDF2.
NOT proven, and recorded as such in DECISIONS-LOG: the hook has not yet run as a
real postStart on prod — the Deployment is Flux-owned from main and both routes
to the live object were refused by the session's permission classifier. Verify
on merge that the marker log gains a VERIFIED line on the next pod start.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
aff80d1d7a |
docs(pm): R-TLS-02 needs the owner — the drafted spec amendment, recovered and restated
The R-TLS-02 analysis and its drafted spec amendment were written on
feat/coverage-debt-paydown (b37defe, 2026-09-07) and lived ONLY in that branch's copy of
DECISIONS-LOG.md. That branch's tree is ~107 commits behind main and must never be merged,
so the amendment went out of view with it — the second time this decision has been raised
and lost. Recovered, re-checked against today's main, and written where it cannot vanish
with a branch.
THE SUBSTANCE IS UNCHANGED. R-TLS-02 says certificates MUST be automatically "issued,
renewed, and revoked without manual steps". Issuance and renewal hold. Revocation is not
implemented and is not implementable with the stack ADR-0006 selects: cert-manager has no
revoke-on-delete, no Certificate field requesting revocation, and no controller that calls
ACME revokeCert, and neither ADR-0006 nor docs/08 specifies a mechanism.
enableCertificateOwnerRef is NOT the remedy — it garbage-collects the Secret, which is
cleanup, not revocation — and it is a global controller flag that would drop the Gateway's
wildcard-home-tls Secret on any future Certificate rename/prune, so it needs its own risk
assessment rather than a side effect of a test task. Still not set.
WHAT CHANGED SINCE THE FIRST ESCALATION, and why it now matters more: R-TLS-02 is no
longer a COVERAGE gap (l0-static/tls-lifecycle-automated + l2-behavioral/
cert-lifecycle-automated both carry the label and main counts it covered), and the L0 check
is honest about the boundary in its own verdict line ("'revoked' is NOT proven at any
layer"). But the L2 test deletes a Certificate and requires its Secret to disappear, and
its failure message says "set enableCertificateOwnerRef: true ... (owner decision — WS-TEST
does not weaken the assertion instead)". That assertion is written to FAIL on a live run
until the owner decides. The decision is now arriving as a red test rather than as a
question, which is exactly how it gets weakened by whoever meets it next.
Two options are recorded verbatim for the architect, in the drafted preference order:
amend the requirement so revocation is an explicitly manual, documented break-glass action
(which also makes enableCertificateOwnerRef a CONSEQUENCE of the requirement), or keep the
text and record a reviewed DOCUMENTED_GAPS entry naming the clause, the reason and the
removal condition.
NOT APPLIED, and not appliable by an agent: the spec repo is read-only under R-TEST-06.
This is a request for a human decision, stated plainly as one.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
f4f6e5f7a3 |
docs(pm): handoff — five WIP branches preserved after a session rate limit
Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
87c3d83dff |
fix(conformance): placeholder-guard-coupling actually RUNS — it was skipping forever
`conformance/tests/l0-static/placeholder-guard-coupling` (R-ENV-02, R-STATE-03) exited 3 SKIPPED on every gate run since it landed, because its check keyed its whole existence on `bootstrap/standup.sh`. Measured, not assumed: that file has NEVER been on `main` — there is no deletion commit for it anywhere; it landed only on unmerged prod-standup branches (`work/prod-standup-campaign`, `prod/*`, `feat/prod-hardening`). It was planned and never merged, not removed. SKIPPED is not green, and a permanently-skipped incident-derived test is the quiet version of the failure it was written to prevent. The skip was also hiding a RED. Run against the real `standup.sh` from `work/prod-standup-campaign`, the old check FAILED 2 of 34 assertions — false positives from treating the empty string as placeholder-shaped for knobs cluster-vars authors empty on purpose (`BACKUP_CREDENTIALS_SUFFIX`, `VELERO_BACKUP_PREFIX`, both `""` on dev and both documented there as load-bearing). It would have false-blocked the merge landing its own input. NOT deleted — the hazard is not gone. Split into two halves, one of which needs no guard: HALF A (always runs): no environment overlay ships a placeholder-shaped EFFECTIVE cluster-vars value; an empty effective value is STATED by the overlay rather than inherited in silence; a LOUD base sentinel is explicitly overridden by every overlay. This is the 2026-09-05 outcome asserted at the source instead of at the guard — if prod's S3_BACKUP_ENDPOINT is never a placeholder, velero cannot be pointed at a hostname that does not exist. Input surface is `clusters/`, all of it on `main`. HALF B (self-activating): the real predicate extracted and executed against those same values, the consumer trace, and the mid-string anchoring property. The check NEVER exits 3 again, and the guard is DISCOVERED by content rather than hardcoded: any tracked `*.sh` defining `<name>_is_placeholder()` is found, so a guard landing at `bootstrap/standup.sh` — or moved into a sourceable library, the remedy the check already recommends — activates HALF B with no edit to the test. Two files defining it is a FAIL. When no guard exists the gap is printed loudly every run instead of swallowed as a skip. LOUD vs EMPTY is the correction that makes it honest in both modes: a loud placeholder is always a defect in an overlay, while `""` is held to the explicitness rule instead — and still counts as a placeholder for a knob a guard covers, because a `""` case-arm is the guard author's declaration that empty is not a usable target. Proof, all reverted: 84 assertions and exit 0 on `main`; prod S3_BACKUP_ENDPOINT set to the literal `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` FAILS (A1); deleting dev's explicit VELERO_BACKUP_PREFIX op so it inherits `""` FAILS (A2); a LOUD base BULK_NFS_SERVER with prod's override deleted FAILS 2/86 (A1+A3); and with the real `standup.sh` present, reverting `s3_is_placeholder` to the `TBD-*` PREFIX glob FAILS 12/110 (B3 anchoring). With the pristine `standup.sh` the check PASSES at 110 assertions where the old one failed 2 of 34. L0 layer: 14 checks, 14 PASS, 0 SKIP, 0 FAIL (was 13 PASS / 1 SKIP). Coverage unchanged — R-ENV-02 and R-STATE-03 both still list this test, 49/51 with the same 2 gaps. Also: * `ci/touched-requirements.sh` — `bootstrap/*` -> R-STATE-03 R-GITOPS-01 is KEPT, its stale comment corrected. It claimed standup.sh "carries the placeholder guards", describing a file that is not here; R-STATE-03 there is ANTICIPATORY. Deleting the entry would leave the change that lands a placeholder guard selected by nothing. * `ci/tests/touched-requirements.test.sh` — new path-map section (12 assertions, was 8) locking `bootstrap/*` and `clusters/*`, so removing either entry is a test failure rather than a silent loss of selection. All three ci/tests pass (12 + 31 + 15). * `docs/BULK-NFS-STORAGE.md` §7 — its "nothing in the gate currently refuses a placeholder-shaped overlay value" is no longer true; corrected. * Scope re-audited: still deliberately requirement-scoped, not whole-repo. Every path that can break it attributes a requirement it declares. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
2f38b8156b |
fix(arr): wire qBittorrent as a download client + Remote Path Mappings
qBittorrent reports finished releases at /downloads/<name>; sonarr and radarr
see the same inodes at /media/downloads/<name>, so Completed Download Handling
could never resolve what the client handed it. Configured, on prod, via each
app's own HTTP API:
* download client qBittorrent -> qbittorrent.qbittorrent.svc.cluster.local:8080
(it did not exist in EITHER arr: both /api/v3/downloadclient returned [])
* Remote Path Mapping /downloads/ -> /media/downloads/ for that host
Both built from each app's /downloadclient/schema, matched on identity before
writing, and re-running creates nothing. downloadclient/testall reports
isValid=true in both. Proven with a negative control: Sonarr's queue showed
outputPath=/downloads/... with the mapping deleted and /media/downloads/...
with it present, for the same completed torrent.
This state lives in sonarr.db / radarr.db on the config PVCs, NOT in git — a
rebuild from git alone does not restore it. Recorded as a wart, with path
agreement (repointing SavePath to /media/downloads) recommended as the durable
fix and put to the operator rather than taken unilaterally.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
505083fded |
feat(cnpg): alert when >1 CNPG primary shares a node — the drift CNPG cannot prevent
`instances: 3` + `podAntiAffinityType: required` fixed the INSTANCE spread but moved
the PRIMARY concentration from k3s-prod-2 to k3s-prod-1: all three primaries
(keycloak-db-2, nextcloud-db-2, semaphore-db-2) ended up on one node, so losing it
meant three simultaneous failovers.
There is NO declarative fix, established from the CRD installed on prod (CNPG 1.30.0,
clusters.postgresql.cnpg.io/v1) rather than from memory. Every spec.affinity field and
spec.topologySpreadConstraints applies to "all the cluster's pods" and is not role-aware;
spec.replica.primary names a Cluster object in a distributed topology, not a node;
primaryUpdateMethod/switchoverDelay/primaryLease govern how a promotion happens, never
where. `targetPrimary` exists only under STATUS, never spec — so `cnpg promote` is an
imperative status patch by construction and there is nothing to put in git. The obvious
workaround is worse than nothing: pod anti-affinity is only ever
...IgnoredDuringExecution while CNPG promotes by relabelling a RUNNING pod, and with
3 instances on 3 nodes it would leave the third instance unschedulable.
So this commits the part that IS durable — visibility — and rebalances live.
CNPGPrimariesCoLocated joins CNPG's own answer to "who is primary" (the <cluster>-rw
Service selects only the primary) against the node: kube_endpointslice_endpoints carries
endpoint_nodename + targetref_name, which two label_replaces lift into node/pod. Neither
family works alone — cnpg_* metrics have no node label, and kube_pod_labels is EMPTY here
(no kube-state-metrics label allowlist), so cnpg.io/instanceRole is not in Prometheus at
all. `* on (namespace, pod) group_left () cnpg_collector_up` keeps an unrelated `*-rw`
Service from triggering it, and uses the one metric present for EVERY instance rather than
the cnpg_pg_stat_* family that is missing for nextcloud-db.
A Ready-node gate stops this becoming another permanently-firing alert: with three
clusters on three nodes a node loss NECESSARILY co-locates two primaries, so the rule
only speaks when the spread is actually achievable, and re-arms when the node returns.
Proven both ways. Live: the expr returned {node="k3s-prod-1"} => 3 before, empty after,
with the inner join still resolving all three primaries (quiet because false, not blind);
loaded into prod's Prometheus as a throwaway `for: 1m` twin it went pending at +70s and
firing at +130s. alerts.promtool-test.yaml pins four cases under promtool 3.14.0 (the
exact version prod runs) and is mutation-checked — dropping the join, dropping the gate
or shortening `for:` each makes a test FAIL.
Rebalanced live, one cluster at a time: keycloak-db -> k3s-prod-2, nextcloud-db ->
k3s-prod-3, semaphore-db deliberately LEFT on k3s-prod-1 (lightest writer, and that node
has the known etcd/Longhorn I/O contention) — two switchovers, not three. Write outage
measured with a `select txid_current()` probe, not estimated: 9.4s and 9.43s, one
contiguous burst each. All three back to readyInstances: 3 with
synchronous_standby_names = ANY 1 (...) and both standbys streaming/quorum.
The placement is CURRENT, NOT DURABLE, and the alert is the point: the next failover or
rolling update can re-concentrate the primaries, and now that gets reported instead of
discovered by hand.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
5e7ab65c65 |
docs: kubectl get backup lies on this cluster — ambiguous Backup short name
Three API groups register kind Backup (longhorn, cnpg, velero). The bare short name resolves to Longhorn's empty set, so kubectl reports absence while 27 CNPG backups and live Velero backups exist. A false negative on 'do I have backups' is the worst possible time to be wrong, so record the rule: fully-qualified resource names only. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
ebce6dfabb |
docs: the Flux->Gitea allowance is rendered and Flux-adopted — debt closed
Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a863542bf7 |
docs: record the apex migration, the GitOps deadlock, and the live-only policy
Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
f7d4648851 |
docs(keycloak): record the URL-reconciliation decisions
Records K1-K9: extend-vs-sibling and why, the generated+gated desired state, verbatim token pass-through, per-pass fail-closed, order-insensitivity, unmanaged-means-untouched, public clients' URLs managed, one alert pair not two, and the mutation harness. Also records a process note: my first pass reported three false findings (reconciler absent from main, import not templated, BASE_DOMAIN dead) because I diffed against origin/main -- origin is dev's Gitea, stale at |
||
|
jakman85andClaude Opus 5
|
16c6234696 |
feat(monitoring): alert when egress steering goes stale, not just when tun0 dies
`VPNTunnelDown` goes green the instant a tun0 exists, which is not the same as vpn-class pods having egress: Cilium caches (egress IP, egress ifindex) at first resolution, so after a reconnect the BPF egress map points at a destroyed interface and the packets are dropped while tun0 looks perfectly healthy. Dev sat in exactly that state for ~80 minutes with nothing alerting. Two rules close it: * VPNEgressSteeringStale (critical, for: 15m) — fires when the live tun0 was re-created more times in the last hour than the gateway node's `egress-uplink` label changed, i.e. Cilium was never told about a reconnect. * VPNEgressUplinkLabelMissing (warning, for: 30m) — the watchdog for the watchdog: a gateway node whose uplink label never reaches kube-state-metrics is a cluster where the rule above is inert. Both gated on a gateway node existing, reusing `VPNTunnelDown`'s `kube_node_labels` guard — a cluster with no gateway cannot be made to fire them. The direct label-vs-map comparison is NOT available and nothing was invented to pretend otherwise: no `cilium_*`/`hubble_*` series exist on either cluster (the cilium-agent exposes no metrics port, there is no ServiceMonitor), and Cilium OSS publishes no series carrying egress-map contents at all — the map is readable only via `cilium-dbg bpf egress list`. PromQL also cannot compare a metric value (the live ifindex) against a number inside a label value, so the level-triggered equality test is not expressible; the same invariant is counted over a window instead. Both limits, and the textfile-collector/black-box-probe routes that would lift them, are written down in the rule comments and the DECISIONS-LOG. `egress-uplink` added to kube-state-metrics.metricLabelsAllowlist in the same kustomization, because without it the first rule has nothing to compare. Verified read-only against LIVE prod (port-forward + /api/v1/query), rendered expr extracted from `kubectl kustomize platform-config`: empty when healthy; empty when evaluated at a timestamp before prod had a gateway node; and FIRING when pointed at a label that genuinely failed to follow prod's real 128 -> 129 tun0 flap earlier today. Nothing on prod was disturbed to produce that. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
53e09a2610 |
docs(vpn): dev recovered the minute prod left the shared PIA server
Closes the loop on the collision entry with the recovery evidence rather than leaving it as an open question for the operator: dev's last Inactivity timeout is 23:05:06 UTC — the minute prod switched to .135 — and at 23:12:03 that same session was still the live one, seven minutes unbroken against a two-minute cycle, 4/4 ping replies at 146 ms, 0% loss, and vpn-class pod egress back to 158.173.241.36 on three consecutive probes. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
38e4bf1459 |
docs(vpn): record the PIA server collision, and the accidental proof it handed us
The stand-up broke dev, so the "one gateway per PIA server" rule and the measurement behind it are now written down where the next person will look: platform/vpn-egress/README.md and the session's DECISIONS-LOG entry. Both carry the verbatim journal lines — same server, same pushed `ifconfig 10.72.192.7` on both gateways — and the flap timestamps, because "same account, inside the connection limit" is true and is exactly what makes this non-obvious. Also corrected: the dev/prod table now shows the two environments on different servers with different exit IPs (dev 158.173.241.36, prod 158.173.241.11), and the open item about them sharing one public IP is replaced with the real remaining decision — whether prod gets its own PIA account, which would remove the hand-maintained per-environment server value entirely. The unplanned bonus, recorded in both places: moving prod to .135 gave it a genuinely NEW tunnel address on a real reconnect (10.72.192.7 ifindex 128 -> 10.150.192.86 ifindex 129) — the precise case Cilium does not follow and the one defect 1 was about. The egress-uplink label moved on its own, Cilium re-resolved, and all four vpn-class workloads kept egressing with no manual step. Better evidence than the synthetic address swap used to characterise it, and it cost nothing to capture. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
04606d48b2 |
docs(vpn): stop crediting the nftables kill-switch for a drop it never makes
Four places in the tree told the reader that the host nftables table is the authoritative fail-closed backstop for egress-class=vpn pods: platform/vpn-egress README, egress-steering.yaml, the role README, and the vpn-killswitch test's own GIVEN/THEN prose. It is not, and cannot be — Cilium's eBPF egress-gateway datapath bpf_redirect()s those skbs to the egress device without invoking netfilter POSTROUTING. Corrected with the measurement in each place rather than softened, because a guard the repo trusts and that does not fire is worse than no guard. What replaces the claim, everywhere: vpn-class fails closed because CILIUM drops the traffic when it cannot use its cached egress interface, verified by tcpdump on both the WAN and tun0 during a tunnel stop and a tunnel restart (0 packets out either). The nft table keeps a narrower, now-proven job: node-originated traffic sourced from a tunnel address (killswitch_drop moved 0 -> 3 packets on k3s-prod-3 when such a packet was routed at the WAN). Also recorded: * egress-steering.yaml — `interface: tun0` is resolved ONCE and cached, and the gateway node is what tells Cilium it changed (the egress-uplink node label). A future reader's instinct will be to have the node write to that object; the comment says not to, because it is Flux-managed, and points at the label and `cilium-dbg bpf egress list` as the debugging entry points instead. * egress-steering.yaml — why the new NFS media mounts are not affected by pod-scoped steering: an NFS PV is mounted by the node kernel in the host netns, so podSelector never matches it. The 192.168.0.0/16 exclusion covers the server too, but that is the second reason, not the first. * vpn-killswitch test — its recovery step depends on the re-steer path, and could not have passed before it existed. Its no-leak assertion was always right; only the explanation was wrong. Also states the prerequisite plainly: the test cannot run at all without a labelled gateway node. * platform README — prod is live on k3s-prod-3 with the before/after egress IPs, and the note that prod's vpn-class pods had NO internet beforehand (CEGP with no matching node -> `Gateway IP: Not Found` -> dropped), so arming the gateway restored egress rather than moving it off the WAN. * role README — the standing "add the shared fwmark to the interface contract" escalation is closed as not implementable on Cilium OSS, rather than left open for someone to attempt. DECISIONS-LOG carries the full session entry, including the drafted ADR-0015 / docs-03 amendment for the architect (specs are immutable to agents) and the five things left for the operator — chief among them that a silent gateway outage still has no alert, which is what let the dev gateway sit dead for 80 minutes. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
e319777849 |
docs(pm): draft the R-ENV-03 amendment, and name the gate hole that hid it
The requirement's literal reading ("no prod-specific artifact in a shared path")
forbids the environment-selected PAIR that R-ENV-02 prescribes for a value that
must differ per environment while the manifests stay identical — and a Secret's
difference cannot live in cluster-vars, because it is ciphertext in a SOPS file.
Drafted wording that keeps the guard's teeth (an artifact only prod APPLIES
still fails) and removes the false positive, for the System Architect: the spec
repo is read-only here and CI-gated by R-TEST-06.
Also recorded, checked rather than assumed: the merge that turned R-ENV-03 red
was gated with a touched-requirement subset that never selected R-ENV-03, so
ci/l0.sh Stage 3b filtered the already-red check out of the run that printed
PASS. Replaying that delivery's twelve paths through
fix/touched-requirements-fail-closed's requirement-paths.map still yields no
R-ENV-03, so landing that branch would NOT have caught this. The two changes
that would (unconditional whole-repo L0 stages; a changed conformance test
attributing its own requirement labels) are ci/ work and are left to that
branch, not smuggled in here.
|
||
|
jakman85
|
863af01062 |
docs: the §2 shared-device measurement came back, and it does not hold
infra/PROD-PROVISIONING.md §2 recorded the etcd/Longhorn shared-partition risk as "accepted to be MEASURED, not assumed", with an exit condition. The measurement arrived on 2026-09-07 in production, 2 d 7 h after bring-up, as an outage. §2 now carries "THE MEASUREMENT CAME BACK": the journal evidence, the three-node IO PSI table (11.36 / 0.75 / 1.29), and the honest verdict that the section's own premise — "every prod node is SSD or NVMe, where rebuild I/O costs latency rather than collapsing it" — was right about node 2 and wrong about node 1. A 2011 C400 with 61,440 reallocated sectors and no power-loss protection is not the "SSD" that reasoning assumed, and §0 had already named the three-way drive asymmetry as the fault line. The exit condition is OPEN and the second bullet (dedicated device) applies; that remains the operator's. Also records what the three software mitigations in this branch do and do not buy, including the point that no cgroup or scheduler control arbitrates the ext4 journal both tenants share — and the gap that etcd's own fsync histograms, which the exit condition literally asks for, are not collected because kubeEtcd is disabled for k3s. PSI is the signal that exists. DECISIONS-LOG gains the full entry: the four I/O mechanisms found INERT on these nodes (IOSchedulingClass, IOSchedulingPriority, io.latency, bare IOWeight), the finding that concurrentReplicaRebuildPerNodeLimit was already set and already held during the outage, the third instance of config that looked applied and was not (orphanAutoDeletion), and the three things still requiring the operator. infra/ansible/README.md: CORRECTED a stale claim that k3s-prod-2/-3 "stay commented out" of the inventory. They have been live in [k3s_prod_secondary] since 2026-09-03. That sentence is exactly the kind used to reason about blast radius — `-e k3s_hosts=k3s_prod` targets THREE hosts, not one — so it is replaced with the --list-hosts command that settles it. The new etcd-io-priority play is added to the run order, the layer table and the prod command list. Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit a00a27a2567ea6a805c8c196b75df2c92a7c63a4) |
||
|
jakman85andClaude Opus 5
|
e65a76dc51 |
docs(log): record the prod OIDC-URL outage and the rule it establishes
Non-secret, environment-specific values must not live inside an encrypted
Secret: Flux's ${BASE_DOMAIN} substitution rewrites rendered manifest text and
cannot reach inside ciphertext, so such a value silently freezes at whatever
one environment happened to need. Dev was correct only by coincidence (dev IS
the apex); prod was wrong from its first reconcile, for both Headlamp and
Semaphore. StrictPostBuildSubstitutions cannot catch this — there is no
unresolved ${...} to complain about.
Co-Authored-By: Claude Opus 5 <[email protected]>
|