docs: correct my 'live outage' claim — unifi and crafty were never up
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled

I reported the UniFi controller and Minecraft server as unreachable for up to 8
days. Probing the backend pods directly, bypassing VIP and policy, returns
REFUSED on all four ports while crafty's own UI and pihole's DNS answer as
controls. /crafty/servers is empty and the unifi README still lists installing
UOS in the guest as the open blocker. Nothing regressed.

The externalTrafficPolicy defect is real and is fixed, but it was not the cause.
I observed unreachable ports, found a mechanism that produces unreachable ports,
and concluded it was the cause without running the one control probe that
separates them — the same error shape as nearly blaming the drive for the I/O
pressure spike when the local disks were idle.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-14 01:40:26 -05:00
1 parent adda6dd779
commit 83f13f4b0a
1 file changed
+48
+48
View File
@@ -3920,3 +3920,51 @@ names either).
3. **Then decide what `public` means for the five.** If they are genuinely meant to be
internet-reachable, their records must target the public IP with edge port-forwards and real
certificates; if not, they should be re-declared `internal`. Today they are neither.
---
## 2026-09-14 — CORRECTION: unifi/crafty were never up; the ETP defect is real but was not an outage
I told the operator their UniFi controller and Minecraft server had been "unreachable from the LAN for
up to 8 days" and treated it as a live outage. **That was wrong.** An agent checked the thing I had
not: it probed the backend pods DIRECTLY, bypassing the VIP, the traffic policy and L2 entirely.
OPEN 10.0.0.11:8443 crafty's own UI <- control: the method works
REFUSED 10.0.0.11:25565 Minecraft Java
REFUSED 10.0.0.74:8443 UniFi
REFUSED 10.0.0.74:8080 UniFi inform
REFUSED 10.0.0.74:11443 UniFi UOS UI
OPEN 10.0.0.114:53 pihole <- control
**There is no application behind any of those four ports.** `/crafty/servers` is empty — no Minecraft
server has ever been created. `apps/vms/unifi/README.md` step 5 still reads *"→ NEXT: install UOS in
the guest"*, with the unattended install flagged at line 108 as *"STILL OPEN. This is the blocker."*
The VM is a bare Ubuntu 24.04 with a guest agent. Nothing regressed; the 8-day-old leases are simply
8-day-old Services. I verified all of this independently before accepting the correction.
**What I got right, and what I got wrong.** The `externalTrafficPolicy: Local` defect is genuine:
Cilium's L2 announcer elects a lease holder without regard to which nodes hold a backend, so under
`Local` the frontend is absent from the announcing node's service table and the VIP answers ICMP
unreachable — demonstrated with `cilium-dbg service list` by flipping the policy and watching the
entry appear and vanish. What I got wrong was the inference: I observed unreachable ports, found a
mechanism that would produce unreachable ports, and concluded it was the cause. A control probe of the
backend would have separated them in one command, and I did not run one before reporting. **A
plausible mechanism is not a diagnosis.** Same shape as the I/O-pressure spike, where the local disks
were innocent and I nearly blamed the drive.
The fix merged anyway (`adda6dd`) because it removes a landmine that would have blackholed both VIPs
the moment those applications exist — and the guard
`conformance/tests/l0-static/l2-announced-vips-serve-from-any-node` fails on the pre-fix tree naming
exactly those four Services.
Two further things the agent got right by checking rather than assuming:
- It nearly added `remote-node` to both CiliumNetworkPolicies, then dumped the endpoint's datapath
policy map and found `reserved:remote-node` was ALREADY allowed — Cilium's `cluster` entity subsumes
it, and both CNPs already admit `cluster`. Reverted. Pi-hole needs it explicitly only because its
policy deliberately omits `cluster`.
- It did NOT copy Pi-hole's `loadBalancerSourceRanges`. Pi-hole was compensating for a restriction it
was giving up; these Services never had one, and adding a LAN-only filter could silently cut off
off-LAN players or a remote site informing over WAN.
**The operator's actual next step here is not networking:** install UniFi OS in the guest, and create a
Minecraft server in Crafty.