home-network-k3s-impl/decisions/2026-10-04-hubble-flow-export-plex.md

date, branch, title

date	branch	title
2026-10-04	feat/hubble-flow-export-plex	Record the pre-SNAT client address for Plex via Hubble static flow export, instead of breaking externalTrafficPolicy to get it

What changed

Operator requirement, narrowed explicitly: this is 100% about log visibility — knowing which address connected to Plex. Not bandwidth metering, which does not matter here.

The problem, measured. Plex's own log cannot show it. Every LoadBalancer Service in this cluster is externalTrafficPolicy: Cluster, so the node holding the L2 lease for the VIP SNATs the client before forwarding to the backend pod:

Measured 2026-10-04	Value
Node announcing 192.168.2.24	k3s-prod-3 (also holds .20 Gateway, .23 pihole-dns)
Node running the Plex pod	k3s-prod-1 (10.0.0.135)
Sources in Plex's log	3,214 × 10.0.1.x, 105 × 10.0.2.x, 120 × loopback
192.168.x sources in Plex's log, ever	0
X-Forwarded-For in Plex's log	0 (it serves raw tcp/32400, not HTTP via the Gateway)
10.0.1.78 identity (cilium-dbg bpf ipcache)	identity=1 → reserved:host = prod-3 itself

Nothing on the router is involved — the operator correctly pointed out there is no hairpin NAT here. The rewrite is entirely in-cluster, done by Cilium's BPF load-balancer on the lease holder, because the backend pod is on a different node.

The address is not lost, only unrecorded. Cilium observes the flow at ingress, before its own translation, so Hubble already holds the true source:

192.168.2.133 -> 10.0.0.135    (real LAN client -> the Plex pod)
10.0.1.78     -> 10.0.0.135    (the same traffic post-SNAT — what Plex sees)

What was missing is retention: the ring buffer was 4095/4095 flows at ~303 flows/s — about thirteen seconds of history — with hubble metrics disabled and no exporter configured.

So: enable the static flow exporter, scoped to Plex's port, writing to the agent's stdout.

• platform/cilium/helmrelease.yaml (authoritative): hubble.export.static.enabled: true, filePath: stdout, allowList of {"destination_port":["32400"]} and {"source_port":["32400"]}.
• platform/cilium/values.yaml (reference mirror) and infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 (the [parity] source of truth): the same block, so a bootstrap render does not differ from what Flux reconciles — a mismatch would be a permanent Helm diff and an avoidable agent rollout.

filePath: stdout, not a file, because Grafana Alloy already runs as a DaemonSet tailing every pod's logs into Loki (platform/monitoring/alloy.yaml). stdout therefore reaches Loki with no new hostPath mount, no file-rotation handling, and no Alloy change at all. Upstream documents the value: "To get Hubble flows directly exported to the logs instead of written to a rotated file, stdout can be defined as hubble-export-file-path." Query:

{namespace="kube-system", app="cilium"} |= "32400"

SCOPED BY PORT, NOT BY NAMESPACE — a measured decision, not a preference. A --to-namespace plex query returned only the post-SNAT flows; the flow carrying the real client address did not match it, because at the point of observation the destination identity is not yet resolved to the plex pod. A namespace filter would have dropped exactly the record this change exists to capture.

WHY NOT FIX THE SNAT INSTEAD — both alternatives rejected on evidence.

• externalTrafficPolicy: Local would put the real address in Plex's own log, but cilium 1.20's L2 announcer is explicitly incompatible with it. Upstream docs for this exact version: "incompatible with the externalTrafficPolicy: Local on services as it may cause service IPs to be announced on nodes without pods causing traffic drops" and "L2 announcements isn't currently aware of this setting". Live proof it is not theoretical: the VIP is announced by prod-3 while the pod runs on prod-1, so the flip would black-hole tcp/32400 immediately. Making it safe means pinning both the pod and the announcement to one node and surrendering Plex's ability to reschedule — a real availability loss to buy a log field. Same trap platform/crafty/service-l4.yaml documents from an 8-day Minecraft outage.
• DSR also preserves the client address, but routing-mode here is vxlan, not native/geneve, and DSR returns bypass the ingress node, which breaks the SOHO router's stateful NAT on the remote path. Cluster-wide blast radius across all eight LB Services for one service's logging.

A correction to the repo's own record. decisions/2026-10-03-plex-remote-access-l4.md states the SNAT puts the client inside 192.168.0.0/16 (the ingress node's LAN address), and I repeated that in the previous entry. It is wrong: measured, the SNAT source is 10.0.1.x — the node's cilium_host inside the pod CIDR. This matters, because it means LanNetworksBandwidth can never discriminate local from remote while eTP is Cluster: 10.0.0.0/8 must be in the LAN set for the in-cluster consumers (Seerr) to count as local, and every external client arrives in that same range. PR #48's LAN definition therefore fixes the Remote badge by making everything local — acceptable only because metering is explicitly not a goal here.

What I verified

Live prod, 2026-10-04, read-only.

• Every measurement in the table above, including the VIP-lease-to-node map (cilium-dbg shell -- db/show l2-announce on all three agents) and the pod's node.
• Hubble already carries the pre-SNAT source — the single finding this whole approach rests on: hubble observe --port 32400 on the lease holder returned 192.168.2.133 -> 10.0.0.135 and its return flow, alongside the post-SNAT 10.0.1.78 -> 10.0.0.135.
• The retention gap is real: cilium-dbg status reported Hubble: Ok Current/Max Flows: 4095/4095 (100.00%), Flows/s: 303.62 Metrics: Disabled, and the cilium-config ConfigMap contains no hubble-export-* key at all.
• The namespace-filter trap, directly: --to-namespace plex returned only 10.0.1.78 and 10.0.1.71; the 192.168.2.133 flow was absent. This is why the allowList is port-scoped.
• All eight LoadBalancer Services are eTP=Cluster (crafty ×2, gateway, pihole-dns ×2, plex, unifi ×2) — so this is a cluster-wide pattern, not a Plex-specific choice.
• The eTP:Local incompatibility is upstream-documented for v1.20 specifically, quoted above from the docs for the running version — not inferred from the crafty incident.
• routing-mode: tunnel and bpf-lb-mode-annotation: false in the live cilium-config, which is what rules DSR out.
• The chart's value paths and defaults, read from cilium v1.20.1's own values.yaml: hubble.export.static.{enabled,filePath,fieldMask,fieldAggregate,aggregationInterval,allowList,denyList,fileMaxSizeMb,fileMaxBackups,fileCompress}, with filePath defaulting to /var/run/cilium/hubble/events.log. The keys used here exist.
• Alloy's pipeline shape, read from platform/monitoring/alloy.yaml: a DaemonSet with loki.source.kubernetes "pods" → loki.process → loki.write at http://loki.monitoring.svc.cluster.local:3100. It already tails kube-system pods, which is why stdout needs no Alloy change.
• All three value files agree (one destination_port occurrence each) and both YAML files parse, with export.static resolving to the intended object.
• Toolbox gate: rc 3, with spec-immutability, decisions-log, lint, l0, render-golden, l1 and approval-policy all pass; rc 3 is the two known environmental skips (gitea-settings — no local SETTINGS_API_TOKEN; conformance L2/L3 — CLUSTER_AVAILABLE unset). Not green. Reported as rc 3.
• No sensitive paths touched — platform/cilium/** and infra/ansible/** are not in infra/gitea/approval-policy.json, so this needs no approval override, unlike PR #48.

What I did NOT verify

• THE OUTCOME. Nothing has been reconciled and no flow has ever been exported. cilium-config still has no hubble-export-* key. Every claim is about inputs.
• Whether filePath: stdout is accepted as written. This is the single largest risk. The value comes from a verbatim upstream doc sentence ("stdout can be defined as hubble-export-file-path"), but I did not confirm the exact notation in cilium 1.20.1's source, and the chart default is a real filesystem path. If the literal string is not special-cased, the agent may create a FILE named stdout in its working directory and nothing reaches Loki — a silent no-op, not a crash. Fallback if so: leave filePath at the chart default and add a local.file_match + loki.source.file target to Alloy with a hostPath mount for /var/run/cilium/hubble.
• Whether the FlowFilter JSON is accepted. {"destination_port":["32400"]} follows the proto (destination_port is a repeated string) and the documented shape ('{"verdict":["DROPPED"]}'), but upstream gives no port example and I did not validate it against the running agent. A malformed filter could be rejected — or, worse, be silently treated as "no match" and export nothing.
• THIS CHANGE ROLLS CILIUM. Editing the HelmRelease restarts the agent DaemonSet across all three nodes. BPF maps are preserved so established traffic should survive, but prod-3 currently announces the Gateway VIP, PiHole's DNS VIP and Plex's VIP, so a lease change during the rollout could briefly interrupt LAN DNS for the whole house. Not measured, and worth doing at a quiet moment.
• Log volume in practice. Estimated "a few hundred to a few thousand lines a day" from the observed request counts; never measured as flows. Both directions are exported and no fieldMask is set, so each record is a full flow object. If it is noisy, narrow with fieldMask rather than by dropping a filter.
• Loki's retention for this data, and whether the {namespace="kube-system", app="cilium"} label set actually matches (it depends on the cilium pods carrying app.kubernetes.io/name, which Alloy maps to app — I did not confirm that label exists on the cilium DaemonSet pods). The query in the comment may need adjusting to pod=~"cilium-.*".
• That the exported flow actually contains the pre-SNAT address. Hubble's live stream does (measured). I am assuming the exporter sees the same flow objects; I did not confirm the exporter is not fed a post-translation view.
• The parity convention has no automated check. I kept all three files in sync by hand because the comments demand it; nothing in ci/ or conformance/ verifies cilium Helm↔Ansible parity, so a future edit can still drift them silently.