diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 81c9cd4..f0a1a00 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -3920,3 +3920,51 @@ names either). 3. **Then decide what `public` means for the five.** If they are genuinely meant to be internet-reachable, their records must target the public IP with edge port-forwards and real certificates; if not, they should be re-declared `internal`. Today they are neither. + +--- + +## 2026-09-14 — CORRECTION: unifi/crafty were never up; the ETP defect is real but was not an outage + +I told the operator their UniFi controller and Minecraft server had been "unreachable from the LAN for +up to 8 days" and treated it as a live outage. **That was wrong.** An agent checked the thing I had +not: it probed the backend pods DIRECTLY, bypassing the VIP, the traffic policy and L2 entirely. + + OPEN 10.0.0.11:8443 crafty's own UI <- control: the method works + REFUSED 10.0.0.11:25565 Minecraft Java + REFUSED 10.0.0.74:8443 UniFi + REFUSED 10.0.0.74:8080 UniFi inform + REFUSED 10.0.0.74:11443 UniFi UOS UI + OPEN 10.0.0.114:53 pihole <- control + +**There is no application behind any of those four ports.** `/crafty/servers` is empty — no Minecraft +server has ever been created. `apps/vms/unifi/README.md` step 5 still reads *"→ NEXT: install UOS in +the guest"*, with the unattended install flagged at line 108 as *"STILL OPEN. This is the blocker."* +The VM is a bare Ubuntu 24.04 with a guest agent. Nothing regressed; the 8-day-old leases are simply +8-day-old Services. I verified all of this independently before accepting the correction. + +**What I got right, and what I got wrong.** The `externalTrafficPolicy: Local` defect is genuine: +Cilium's L2 announcer elects a lease holder without regard to which nodes hold a backend, so under +`Local` the frontend is absent from the announcing node's service table and the VIP answers ICMP +unreachable — demonstrated with `cilium-dbg service list` by flipping the policy and watching the +entry appear and vanish. What I got wrong was the inference: I observed unreachable ports, found a +mechanism that would produce unreachable ports, and concluded it was the cause. A control probe of the +backend would have separated them in one command, and I did not run one before reporting. **A +plausible mechanism is not a diagnosis.** Same shape as the I/O-pressure spike, where the local disks +were innocent and I nearly blamed the drive. + +The fix merged anyway (`adda6dd`) because it removes a landmine that would have blackholed both VIPs +the moment those applications exist — and the guard +`conformance/tests/l0-static/l2-announced-vips-serve-from-any-node` fails on the pre-fix tree naming +exactly those four Services. + +Two further things the agent got right by checking rather than assuming: +- It nearly added `remote-node` to both CiliumNetworkPolicies, then dumped the endpoint's datapath + policy map and found `reserved:remote-node` was ALREADY allowed — Cilium's `cluster` entity subsumes + it, and both CNPs already admit `cluster`. Reverted. Pi-hole needs it explicitly only because its + policy deliberately omits `cluster`. +- It did NOT copy Pi-hole's `loadBalancerSourceRanges`. Pi-hole was compensating for a restriction it + was giving up; these Services never had one, and adding a LAN-only filter could silently cut off + off-LAN players or a remote site informing over WAN. + +**The operator's actual next step here is not networking:** install UniFi OS in the guest, and create a +Minecraft server in Crafty.