@@ -0,0 +1,157 @@
---
date : 2026-10-04
branch : feat/hubble-flow-export-plex
title : Record the pre-SNAT client address for Plex via Hubble static flow export, instead of breaking externalTrafficPolicy to get it
---
## What changed
Operator requirement, narrowed explicitly: **this is 100% about log visibility ** — knowing which
address connected to Plex. Not bandwidth metering, which does not matter here.
**The problem, measured. ** Plex's own log cannot show it. Every LoadBalancer Service in this cluster
is `externalTrafficPolicy: Cluster` , so the node holding the L2 lease for the VIP SNATs the client
before forwarding to the backend pod:
| Measured 2026-10-04 | Value |
|---|---|
| Node announcing `192.168.2.24` | `k3s-prod-3` (also holds `.20` Gateway, `.23` pihole-dns) |
| Node running the Plex pod | `k3s-prod-1` (`10.0.0.135` ) |
| Sources in Plex's log | **3,214 × ** , 105 × `10.0.2.x` , 120 ×
| `192.168.x` sources in Plex's log, ever | **0 ** |
| `X-Forwarded-For` in Plex's log | **0 ** (it serves raw tcp/32400, not HTTP via the Gateway) |
| `10.0.1.78` identity (`cilium-dbg bpf ipcache` ) | `identity=1` → `reserved:host` = prod-3 itself |
**Nothing on the router is involved ** — the operator correctly pointed out there is no hairpin NAT
here. The rewrite is entirely in-cluster, done by Cilium's BPF load-balancer on the lease holder,
because the backend pod is on a different node.
**The address is not lost, only unrecorded. ** Cilium observes the flow at ingress, * before * its own
translation, so Hubble already holds the true source:
192.168.2.133 -> 10.0.0.135 (real LAN client -> the Plex pod)
10.0.1.78 -> 10.0.0.135 (the same traffic post-SNAT — what Plex sees)
What was missing is **retention ** : the ring buffer was **4095/4095 flows at ~303 flows/s — about
thirteen seconds of history** — with hubble metrics disabled and no exporter configured.
So: enable the static flow exporter, scoped to Plex's port, writing to the agent's stdout.
- **`platform/cilium/helmrelease.yaml` ** (authoritative): `hubble.export.static.enabled: true` ,
`filePath: stdout` , `allowList` of `{"destination_port":["32400"]}` and
`{"source_port":["32400"]}` .
- **`platform/cilium/values.yaml` ** (reference mirror) and
* * `infra/ansible/roles/cilium/templates/cilium-values.yaml.j2` ** (the `[parity]` source of truth):
the same block, so a bootstrap render does not differ from what Flux reconciles — a mismatch would
be a permanent Helm diff and an avoidable agent rollout.
* * `filePath: stdout` , not a file,** because Grafana Alloy already runs as a DaemonSet tailing every
pod's logs into Loki (`platform/monitoring/alloy.yaml` ). stdout therefore reaches Loki with **no new
hostPath mount, no file-rotation handling, and no Alloy change at all**. Upstream documents the
value: *"To get Hubble flows directly exported to the logs instead of written to a rotated file,
`stdout` can be defined as `hubble-export-file-path` ."* Query:
{namespace="kube-system", app="cilium"} |= "32400"
**SCOPED BY PORT, NOT BY NAMESPACE — a measured decision, not a preference. ** A
`--to-namespace plex` query returned **only the post-SNAT flows ** ; the flow carrying the real client
address did not match it, because at the point of observation the destination identity is not yet
resolved to the plex pod. A namespace filter would have dropped exactly the record this change
exists to capture.
**WHY NOT FIX THE SNAT INSTEAD — both alternatives rejected on evidence. **
- `externalTrafficPolicy: Local` would put the real address in Plex's own log, but cilium 1.20's L2
announcer is **explicitly incompatible ** with it. Upstream docs for this exact version:
*"incompatible with the `externalTrafficPolicy: Local` on services as it may cause service IPs to
be announced on nodes without pods causing traffic drops"* and *"L2 announcements isn't currently
aware of this setting"*. Live proof it is not theoretical: the VIP is announced by prod-3 while the
pod runs on prod-1, so the flip would black-hole tcp/32400 **immediately ** . Making it safe means
pinning both the pod and the announcement to one node and surrendering Plex's ability to
reschedule — a real availability loss to buy a log field. Same trap
`platform/crafty/service-l4.yaml` documents from an 8-day Minecraft outage.
- **DSR** also preserves the client address, but `routing-mode` here is **vxlan ** , not
native/geneve, and DSR returns bypass the ingress node, which breaks the SOHO router's stateful
NAT on the remote path. Cluster-wide blast radius across all eight LB Services for one service's
logging.
**A correction to the repo's own record. ** `decisions/2026-10-03-plex-remote-access-l4.md` states the
SNAT puts the client inside `192.168.0.0/16` (the ingress node's LAN address), and I repeated that in
the previous entry. **It is wrong ** : measured, the SNAT source is `10.0.1.x` — the node's
`cilium_host` inside the pod CIDR. This matters, because it means `LanNetworksBandwidth` can never
discriminate local from remote while eTP is `Cluster` : `10.0.0.0/8` must be in the LAN set for the
in-cluster consumers (Seerr) to count as local, and every external client arrives in that same range.
PR #48 's LAN definition therefore fixes the Remote badge by making * everything * local — acceptable
only because metering is explicitly not a goal here.
## What I verified
Live prod, 2026-10-04, read-only.
- **Every measurement in the table above**, including the VIP-lease-to-node map
(`cilium-dbg shell -- db/show l2-announce` on all three agents) and the pod's node.
- **Hubble already carries the pre-SNAT source** — the single finding this whole approach rests on:
`hubble observe --port 32400` on the lease holder returned `192.168.2.133 -> 10.0.0.135` **and **
its return flow, alongside the post-SNAT `10.0.1.78 -> 10.0.0.135` .
- **The retention gap is real:** `cilium-dbg status` reported `Hubble: Ok Current/Max Flows:
4095/4095 (100.00%), Flows/s: 303.62 Metrics: Disabled`, and the ` cilium-config` ConfigMap
contains no ` hubble-export-*` key at all.
- **The namespace-filter trap**, directly: ` --to-namespace plex` returned only ` 10.0.1.78` and
` 10.0.1.71`; the ` 192.168.2.133` flow was absent. This is why the allowList is port-scoped.
- **All eight LoadBalancer Services are ` eTP=Cluster`** (crafty × ×
unifi ×
- **The eTP:Local incompatibility is upstream-documented for v1.20 specifically**, quoted above from
the docs for the running version — not inferred from the crafty incident.
- **` routing-mode: tunnel`** and **` bpf-lb-mode-annotation: false`** in the live ` cilium-config`,
which is what rules DSR out.
- **The chart's value paths and defaults**, read from cilium v1.20.1's own ` values.yaml`:
` hubble.export.static.{enabled,filePath,fieldMask,fieldAggregate,aggregationInterval,allowList,denyList,fileMaxSizeMb,fileMaxBackups,fileCompress}`,
with ` filePath` defaulting to ` /var/run/cilium/hubble/events.log`. The keys used here exist.
- **Alloy's pipeline shape**, read from ` platform/monitoring/alloy.yaml`: a DaemonSet with
` loki.source.kubernetes "pods"` → ` loki.process` → ` loki.write` at
` http://loki.monitoring.svc.cluster.local:3100`. It already tails kube-system pods, which is why
stdout needs no Alloy change.
- **All three value files agree** (one ` destination_port` occurrence each) and both YAML files parse,
with ` export.static` resolving to the intended object.
- **Toolbox gate**: **rc 3**, with ` spec-immutability`, ` decisions-log`, ` lint`, ` l0`,
` render-golden`, ` l1` and ` approval-policy` all **pass**; rc 3 is the two known environmental skips
(` gitea-settings` — no local ` SETTINGS_API_TOKEN`; ` conformance` L2/L3 — ` CLUSTER_AVAILABLE`
unset). **Not green. Reported as rc 3.**
- **No sensitive paths touched** — ` platform/cilium/**` and ` infra/ansible/**` are not in
` infra/gitea/approval-policy.json`, so this needs no approval override, unlike PR #48.
## What I did NOT verify
- **THE OUTCOME. Nothing has been reconciled and no flow has ever been exported.** ` cilium-config`
still has no ` hubble-export-*` key. Every claim is about inputs.
- **Whether ` filePath: stdout` is accepted as written.** This is the single largest risk. The value
comes from a verbatim upstream doc sentence (*"` stdout` can be defined as
` hubble-export-file-path`"*), but I did not confirm the exact notation in cilium 1.20.1's source,
and the chart default is a real filesystem path. If the literal string is not special-cased, the
agent may create a FILE named ` stdout` in its working directory and nothing reaches Loki — a
silent no-op, not a crash. **Fallback if so:** leave ` filePath` at the chart default and add a
` local.file_match` + ` loki.source.file` target to Alloy with a hostPath mount for
` /var/run/cilium/hubble`.
- **Whether the FlowFilter JSON is accepted.** ` {"destination_port":["32400"]}` follows the proto
(` destination_port` is a repeated string) and the documented shape (` '{"verdict":["DROPPED"]}'`),
but upstream gives **no port example** and I did not validate it against the running agent. A
malformed filter could be rejected — or, worse, be silently treated as "no match" and export
nothing.
- **THIS CHANGE ROLLS CILIUM.** Editing the HelmRelease restarts the agent DaemonSet across all
three nodes. BPF maps are preserved so established traffic should survive, but prod-3 currently
announces the Gateway VIP, **PiHole's DNS VIP** and Plex's VIP, so a lease change during the
rollout could briefly interrupt LAN DNS for the whole house. Not measured, and worth doing at a
quiet moment.
- **Log volume in practice.** Estimated "a few hundred to a few thousand lines a day" from the
observed request counts; never measured as flows. Both directions are exported and no ` fieldMask`
is set, so each record is a full flow object. If it is noisy, narrow with ` fieldMask` rather than
by dropping a filter.
- **Loki's retention for this data**, and whether the ` {namespace="kube-system", app="cilium"}` label
set actually matches (it depends on the cilium pods carrying
` app.kubernetes.io/name`, which Alloy maps to ` app` — I did not confirm that label exists on the
cilium DaemonSet pods). The query in the comment may need adjusting to ` pod=~"cilium-.*"`.
- **That the exported flow actually contains the pre-SNAT address.** Hubble's *live* stream does
(measured). I am assuming the exporter sees the same flow objects; I did not confirm the exporter
is not fed a post-translation view.
- **The parity convention has no automated check.** I kept all three files in sync by hand because
the comments demand it; nothing in ` ci/` or ` conformance/` verifies cilium Helm↔Ansible parity, so
a future edit can still drift them silently.