feat(cilium): export Plex's pre-SNAT client address via Hubble, scoped to tcp/32400 #50
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: Home-Network/home-network-k3s-impl#50
Reference in new issue
Block a user
Log visibility only — which address connected to Plex. Not metering.
Why Plex's own log can't show it
Every LoadBalancer Service in this cluster is
externalTrafficPolicy: Cluster, so the node holdingthe VIP's L2 lease SNATs the client before forwarding to the backend pod.
192.168.2.24k3s-prod-3(also holds.20Gateway,.23pihole-dns)k3s-prod-1(10.0.0.135)10.0.1.x, 105 ×10.0.2.x, 120 × loopback192.168.xsources, everX-Forwarded-For10.0.1.78identityidentity=1→reserved:host= prod-3 itselfNothing on the router does this — there's no hairpin NAT. It's Cilium's BPF load-balancer on the
lease holder, because the backend pod is on a different node.
The address isn't lost, only unrecorded
Cilium observes the flow at ingress, before its own translation, so Hubble already has it:
What was missing is retention:
4095/4095 flows at ~303 flows/s≈ thirteen seconds ofhistory,
Metrics: Disabled, and nohubble-export-*key incilium-configat all.The change
hubble.export.staticenabled, port-scoped, to the agent's stdout — Alloy already tails everypod into Loki (
platform/monitoring/alloy.yaml), so this needs no new hostPath mount, norotation handling, and no Alloy change:
Applied to all three value files so the
[parity]contract holds: the HelmRelease (authoritative),values.yaml(mirror), and the Ansible bootstrap template.Scoped by PORT, not namespace — on evidence. A
--to-namespace plexquery returned only thepost-SNAT flows; the record carrying the real client address did not match it, because at that
observation point the destination identity isn't yet resolved to the plex pod. A namespace filter
would have dropped exactly what this exists to capture.
Alternatives rejected, with reasons
externalTrafficPolicy: Localwould put the real address in Plex's own log, but cilium 1.20'sL2 announcer is explicitly incompatible — upstream docs for this version: "may cause service IPs
to be announced on nodes without pods causing traffic drops", "L2 announcements isn't currently
aware of this setting". Not theoretical: the VIP is on prod-3 and the pod on prod-1, so the flip
would black-hole 32400 immediately. Making it safe costs Plex its ability to reschedule.
routing-modeis vxlan, not native/geneve, and DSRreturns bypass the ingress node — breaking the router's stateful NAT on the remote path.
Cluster-wide blast radius for one service's logging.
Corrects the repo's own record
decisions/2026-10-03-plex-remote-access-l4.mdsays the SNAT lands in192.168.0.0/16(the node'sLAN address), and I repeated that. It's
10.0.1.x— thecilium_hostinside the pod CIDR. Thatmatters:
LanNetworksBandwidthcan never separate local from remote while eTP isCluster, because10.0.0.0/8must be in the LAN set for Seerr and every external client arrives in that same range.#48 therefore fixes the Remote badge by making everything local — fine here only because metering
is explicitly not a goal.
Gate
rc 3 —
spec-immutability,decisions-log,lint,l0,render-golden,l1,approval-policyall pass; rc 3 is the two known environmental skips (gitea-settings,conformanceL2/L3). Not green — reported as rc 3. No sensitive paths, so no override needed.⚠️ Read before merging
established traffic should survive, but prod-3 currently announces the Gateway VIP, PiHole's DNS
VIP and Plex's VIP — a lease move during the rollout could briefly interrupt LAN DNS for the
whole house. Worth doing at a quiet moment.
filePath: stdoutis the largest risk. Taken from a verbatim upstream sentence ("stdoutcanbe defined as
hubble-export-file-path") but not confirmed against 1.20.1 source, and the chartdefault is a real path. If the literal isn't special-cased, the agent may create a file named
stdoutand nothing reaches Loki — a silent no-op. Fallback documented: keep the default path andadd a
loki.source.filetarget to Alloy with a hostPath mount.{"destination_port":["32400"]}matches the protoand the documented shape, but a malformed filter could export nothing silently.
app="cilium"depends on the cilium pods carryingapp.kubernetes.io/name; may needpod=~"cilium-.*".🤖 Generated with Claude Code
The requirement is log visibility only -- which address connected to Plex. Plex's own log cannot show it. Every LoadBalancer Service here is externalTrafficPolicy: Cluster, so the node holding the VIP's L2 lease SNATs the client before forwarding to the backend pod. Measured: Plex had logged 3,214 requests from 10.0.1.x (lease holder k3s-prod-3's own address, Cilium identity 1 = reserved:host) and ZERO from any 192.168.x address, ever. No X-Forwarded-For either -- it serves raw tcp/32400, not HTTP via the Gateway. Nothing on the router does this; there is no hairpin NAT. It is Cilium. The address is not lost, only unrecorded. Cilium observes the flow at ingress, BEFORE its own translation, so Hubble already has it: 192.168.2.133 -> 10.0.0.135 real LAN client -> the Plex pod 10.0.1.78 -> 10.0.0.135 same traffic post-SNAT, what Plex sees What was missing is RETENTION: 4095/4095 flows at ~303/s is about thirteen seconds of history, metrics disabled, no exporter configured. So enable the static exporter, port-scoped, to the agent's stdout -- Alloy already tails every pod into Loki, so this needs no new mount, no rotation handling and no Alloy change. {namespace="kube-system", app="cilium"} |= "32400" Scoped BY PORT, not namespace, on evidence: --to-namespace plex returned only the post-SNAT flows; the record carrying the real client address did not match it. A namespace filter would drop exactly what this exists to capture. Rejected, with reasons: eTP:Local is incompatible with cilium 1.20's L2 announcer per upstream docs for this version ("may cause service IPs to be announced on nodes without pods causing traffic drops") -- and the VIP is on prod-3 while the pod is on prod-1, so it would black-hole 32400 immediately; making it safe costs Plex its ability to reschedule. DSR: routing-mode is vxlan, not native/geneve, and DSR returns bypass the ingress node, breaking the router's stateful NAT on the remote path. Also corrects the repo's own record: 2026-10-03 says the SNAT lands in 192.168.0.0/16 (the node's LAN address). It is 10.0.1.x, the cilium_host inside the pod CIDR -- which is why LanNetworksBandwidth can never separate local from remote while eTP is Cluster. UNVERIFIED and stated in the handoff: nothing is reconciled; `filePath: stdout` is taken from a verbatim upstream sentence but not confirmed against 1.20.1 source (a wrong notation is a silent no-op, with the file+Alloy fallback documented); the port FlowFilter has no upstream example; and this change ROLLS CILIUM -- prod-3 announces the Gateway, PiHole DNS and Plex VIPs, so a lease move during the rollout could briefly interrupt LAN DNS. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>