From 989649e64a6761195ed7b22984c0f30ed3fd5767 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 4 Oct 2026 16:36:12 -0500 Subject: [PATCH] feat(cilium): export Plex's pre-SNAT client address via Hubble, scoped to tcp/32400 The requirement is log visibility only -- which address connected to Plex. Plex's own log cannot show it. Every LoadBalancer Service here is externalTrafficPolicy: Cluster, so the node holding the VIP's L2 lease SNATs the client before forwarding to the backend pod. Measured: Plex had logged 3,214 requests from 10.0.1.x (lease holder k3s-prod-3's own address, Cilium identity 1 = reserved:host) and ZERO from any 192.168.x address, ever. No X-Forwarded-For either -- it serves raw tcp/32400, not HTTP via the Gateway. Nothing on the router does this; there is no hairpin NAT. It is Cilium. The address is not lost, only unrecorded. Cilium observes the flow at ingress, BEFORE its own translation, so Hubble already has it: 192.168.2.133 -> 10.0.0.135 real LAN client -> the Plex pod 10.0.1.78 -> 10.0.0.135 same traffic post-SNAT, what Plex sees What was missing is RETENTION: 4095/4095 flows at ~303/s is about thirteen seconds of history, metrics disabled, no exporter configured. So enable the static exporter, port-scoped, to the agent's stdout -- Alloy already tails every pod into Loki, so this needs no new mount, no rotation handling and no Alloy change. {namespace="kube-system", app="cilium"} |= "32400" Scoped BY PORT, not namespace, on evidence: --to-namespace plex returned only the post-SNAT flows; the record carrying the real client address did not match it. A namespace filter would drop exactly what this exists to capture. Rejected, with reasons: eTP:Local is incompatible with cilium 1.20's L2 announcer per upstream docs for this version ("may cause service IPs to be announced on nodes without pods causing traffic drops") -- and the VIP is on prod-3 while the pod is on prod-1, so it would black-hole 32400 immediately; making it safe costs Plex its ability to reschedule. DSR: routing-mode is vxlan, not native/geneve, and DSR returns bypass the ingress node, breaking the router's stateful NAT on the remote path. Also corrects the repo's own record: 2026-10-03 says the SNAT lands in 192.168.0.0/16 (the node's LAN address). It is 10.0.1.x, the cilium_host inside the pod CIDR -- which is why LanNetworksBandwidth can never separate local from remote while eTP is Cluster. UNVERIFIED and stated in the handoff: nothing is reconciled; `filePath: stdout` is taken from a verbatim upstream sentence but not confirmed against 1.20.1 source (a wrong notation is a silent no-op, with the file+Alloy fallback documented); the port FlowFilter has no upstream example; and this change ROLLS CILIUM -- prod-3 announces the Gateway, PiHole DNS and Plex VIPs, so a lease move during the rollout could briefly interrupt LAN DNS. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-10-04-hubble-flow-export-plex.md | 157 ++++++++++++++++++ .../cilium/templates/cilium-values.yaml.j2 | 14 ++ platform/cilium/helmrelease.yaml | 54 ++++++ platform/cilium/values.yaml | 12 ++ 4 files changed, 237 insertions(+) create mode 100644 decisions/2026-10-04-hubble-flow-export-plex.md diff --git a/decisions/2026-10-04-hubble-flow-export-plex.md b/decisions/2026-10-04-hubble-flow-export-plex.md new file mode 100644 index 0000000..ac2b3ae --- /dev/null +++ b/decisions/2026-10-04-hubble-flow-export-plex.md @@ -0,0 +1,157 @@ +--- +date: 2026-10-04 +branch: feat/hubble-flow-export-plex +title: Record the pre-SNAT client address for Plex via Hubble static flow export, instead of breaking externalTrafficPolicy to get it +--- + +## What changed + +Operator requirement, narrowed explicitly: **this is 100% about log visibility** — knowing which +address connected to Plex. Not bandwidth metering, which does not matter here. + +**The problem, measured.** Plex's own log cannot show it. Every LoadBalancer Service in this cluster +is `externalTrafficPolicy: Cluster`, so the node holding the L2 lease for the VIP SNATs the client +before forwarding to the backend pod: + +| Measured 2026-10-04 | Value | +|---|---| +| Node announcing `192.168.2.24` | `k3s-prod-3` (also holds `.20` Gateway, `.23` pihole-dns) | +| Node running the Plex pod | `k3s-prod-1` (`10.0.0.135`) | +| Sources in Plex's log | **3,214 × `10.0.1.x`**, 105 × `10.0.2.x`, 120 × loopback | +| `192.168.x` sources in Plex's log, ever | **0** | +| `X-Forwarded-For` in Plex's log | **0** (it serves raw tcp/32400, not HTTP via the Gateway) | +| `10.0.1.78` identity (`cilium-dbg bpf ipcache`) | `identity=1` → `reserved:host` = prod-3 itself | + +**Nothing on the router is involved** — the operator correctly pointed out there is no hairpin NAT +here. The rewrite is entirely in-cluster, done by Cilium's BPF load-balancer on the lease holder, +because the backend pod is on a different node. + +**The address is not lost, only unrecorded.** Cilium observes the flow at ingress, *before* its own +translation, so Hubble already holds the true source: + + 192.168.2.133 -> 10.0.0.135 (real LAN client -> the Plex pod) + 10.0.1.78 -> 10.0.0.135 (the same traffic post-SNAT — what Plex sees) + +What was missing is **retention**: the ring buffer was **4095/4095 flows at ~303 flows/s — about +thirteen seconds of history** — with hubble metrics disabled and no exporter configured. + +So: enable the static flow exporter, scoped to Plex's port, writing to the agent's stdout. + +- **`platform/cilium/helmrelease.yaml`** (authoritative): `hubble.export.static.enabled: true`, + `filePath: stdout`, `allowList` of `{"destination_port":["32400"]}` and + `{"source_port":["32400"]}`. +- **`platform/cilium/values.yaml`** (reference mirror) and + **`infra/ansible/roles/cilium/templates/cilium-values.yaml.j2`** (the `[parity]` source of truth): + the same block, so a bootstrap render does not differ from what Flux reconciles — a mismatch would + be a permanent Helm diff and an avoidable agent rollout. + +**`filePath: stdout`, not a file,** because Grafana Alloy already runs as a DaemonSet tailing every +pod's logs into Loki (`platform/monitoring/alloy.yaml`). stdout therefore reaches Loki with **no new +hostPath mount, no file-rotation handling, and no Alloy change at all**. Upstream documents the +value: *"To get Hubble flows directly exported to the logs instead of written to a rotated file, +`stdout` can be defined as `hubble-export-file-path`."* Query: + + {namespace="kube-system", app="cilium"} |= "32400" + +**SCOPED BY PORT, NOT BY NAMESPACE — a measured decision, not a preference.** A +`--to-namespace plex` query returned **only the post-SNAT flows**; the flow carrying the real client +address did not match it, because at the point of observation the destination identity is not yet +resolved to the plex pod. A namespace filter would have dropped exactly the record this change +exists to capture. + +**WHY NOT FIX THE SNAT INSTEAD — both alternatives rejected on evidence.** +- `externalTrafficPolicy: Local` would put the real address in Plex's own log, but cilium 1.20's L2 + announcer is **explicitly incompatible** with it. Upstream docs for this exact version: + *"incompatible with the `externalTrafficPolicy: Local` on services as it may cause service IPs to + be announced on nodes without pods causing traffic drops"* and *"L2 announcements isn't currently + aware of this setting"*. Live proof it is not theoretical: the VIP is announced by prod-3 while the + pod runs on prod-1, so the flip would black-hole tcp/32400 **immediately**. Making it safe means + pinning both the pod and the announcement to one node and surrendering Plex's ability to + reschedule — a real availability loss to buy a log field. Same trap + `platform/crafty/service-l4.yaml` documents from an 8-day Minecraft outage. +- **DSR** also preserves the client address, but `routing-mode` here is **vxlan**, not + native/geneve, and DSR returns bypass the ingress node, which breaks the SOHO router's stateful + NAT on the remote path. Cluster-wide blast radius across all eight LB Services for one service's + logging. + +**A correction to the repo's own record.** `decisions/2026-10-03-plex-remote-access-l4.md` states the +SNAT puts the client inside `192.168.0.0/16` (the ingress node's LAN address), and I repeated that in +the previous entry. **It is wrong**: measured, the SNAT source is `10.0.1.x` — the node's +`cilium_host` inside the pod CIDR. This matters, because it means `LanNetworksBandwidth` can never +discriminate local from remote while eTP is `Cluster`: `10.0.0.0/8` must be in the LAN set for the +in-cluster consumers (Seerr) to count as local, and every external client arrives in that same range. +PR #48's LAN definition therefore fixes the Remote badge by making *everything* local — acceptable +only because metering is explicitly not a goal here. + +## What I verified + +Live prod, 2026-10-04, read-only. + +- **Every measurement in the table above**, including the VIP-lease-to-node map + (`cilium-dbg shell -- db/show l2-announce` on all three agents) and the pod's node. +- **Hubble already carries the pre-SNAT source** — the single finding this whole approach rests on: + `hubble observe --port 32400` on the lease holder returned `192.168.2.133 -> 10.0.0.135` **and** + its return flow, alongside the post-SNAT `10.0.1.78 -> 10.0.0.135`. +- **The retention gap is real:** `cilium-dbg status` reported `Hubble: Ok Current/Max Flows: + 4095/4095 (100.00%), Flows/s: 303.62 Metrics: Disabled`, and the `cilium-config` ConfigMap + contains no `hubble-export-*` key at all. +- **The namespace-filter trap**, directly: `--to-namespace plex` returned only `10.0.1.78` and + `10.0.1.71`; the `192.168.2.133` flow was absent. This is why the allowList is port-scoped. +- **All eight LoadBalancer Services are `eTP=Cluster`** (crafty ×2, gateway, pihole-dns ×2, plex, + unifi ×2) — so this is a cluster-wide pattern, not a Plex-specific choice. +- **The eTP:Local incompatibility is upstream-documented for v1.20 specifically**, quoted above from + the docs for the running version — not inferred from the crafty incident. +- **`routing-mode: tunnel`** and **`bpf-lb-mode-annotation: false`** in the live `cilium-config`, + which is what rules DSR out. +- **The chart's value paths and defaults**, read from cilium v1.20.1's own `values.yaml`: + `hubble.export.static.{enabled,filePath,fieldMask,fieldAggregate,aggregationInterval,allowList,denyList,fileMaxSizeMb,fileMaxBackups,fileCompress}`, + with `filePath` defaulting to `/var/run/cilium/hubble/events.log`. The keys used here exist. +- **Alloy's pipeline shape**, read from `platform/monitoring/alloy.yaml`: a DaemonSet with + `loki.source.kubernetes "pods"` → `loki.process` → `loki.write` at + `http://loki.monitoring.svc.cluster.local:3100`. It already tails kube-system pods, which is why + stdout needs no Alloy change. +- **All three value files agree** (one `destination_port` occurrence each) and both YAML files parse, + with `export.static` resolving to the intended object. +- **Toolbox gate**: **rc 3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`, + `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two known environmental skips + (`gitea-settings` — no local `SETTINGS_API_TOKEN`; `conformance` L2/L3 — `CLUSTER_AVAILABLE` + unset). **Not green. Reported as rc 3.** +- **No sensitive paths touched** — `platform/cilium/**` and `infra/ansible/**` are not in + `infra/gitea/approval-policy.json`, so this needs no approval override, unlike PR #48. + +## What I did NOT verify + +- **THE OUTCOME. Nothing has been reconciled and no flow has ever been exported.** `cilium-config` + still has no `hubble-export-*` key. Every claim is about inputs. +- **Whether `filePath: stdout` is accepted as written.** This is the single largest risk. The value + comes from a verbatim upstream doc sentence (*"`stdout` can be defined as + `hubble-export-file-path`"*), but I did not confirm the exact notation in cilium 1.20.1's source, + and the chart default is a real filesystem path. If the literal string is not special-cased, the + agent may create a FILE named `stdout` in its working directory and nothing reaches Loki — a + silent no-op, not a crash. **Fallback if so:** leave `filePath` at the chart default and add a + `local.file_match` + `loki.source.file` target to Alloy with a hostPath mount for + `/var/run/cilium/hubble`. +- **Whether the FlowFilter JSON is accepted.** `{"destination_port":["32400"]}` follows the proto + (`destination_port` is a repeated string) and the documented shape (`'{"verdict":["DROPPED"]}'`), + but upstream gives **no port example** and I did not validate it against the running agent. A + malformed filter could be rejected — or, worse, be silently treated as "no match" and export + nothing. +- **THIS CHANGE ROLLS CILIUM.** Editing the HelmRelease restarts the agent DaemonSet across all + three nodes. BPF maps are preserved so established traffic should survive, but prod-3 currently + announces the Gateway VIP, **PiHole's DNS VIP** and Plex's VIP, so a lease change during the + rollout could briefly interrupt LAN DNS for the whole house. Not measured, and worth doing at a + quiet moment. +- **Log volume in practice.** Estimated "a few hundred to a few thousand lines a day" from the + observed request counts; never measured as flows. Both directions are exported and no `fieldMask` + is set, so each record is a full flow object. If it is noisy, narrow with `fieldMask` rather than + by dropping a filter. +- **Loki's retention for this data**, and whether the `{namespace="kube-system", app="cilium"}` label + set actually matches (it depends on the cilium pods carrying + `app.kubernetes.io/name`, which Alloy maps to `app` — I did not confirm that label exists on the + cilium DaemonSet pods). The query in the comment may need adjusting to `pod=~"cilium-.*"`. +- **That the exported flow actually contains the pre-SNAT address.** Hubble's *live* stream does + (measured). I am assuming the exporter sees the same flow objects; I did not confirm the exporter + is not fed a post-translation view. +- **The parity convention has no automated check.** I kept all three files in sync by hand because + the comments demand it; nothing in `ci/` or `conformance/` verifies cilium Helm↔Ansible parity, so + a future edit can still drift them silently. diff --git a/infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 b/infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 index fd15eb6..01f7f17 100644 --- a/infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 +++ b/infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 @@ -45,6 +45,20 @@ hubble: enabled: true ui: enabled: true + # Static flow export -> cilium-agent stdout. Kept EQUAL to the Flux values + # (platform/cilium/helmrelease.yaml), which carry the full rationale: it records the PRE-SNAT + # client address for Plex's raw tcp/32400, which Plex itself cannot log because every + # LoadBalancer Service here is externalTrafficPolicy: Cluster and the VIP lease holder SNATs the + # client. Present here so a BOOTSTRAP render does not differ from what Flux reconciles -- a + # mismatch would show up as a permanent Helm diff and an avoidable agent rollout. Scoped by PORT, + # not namespace: the flow carrying the real client address does not match a namespace filter. + export: + static: + enabled: true + filePath: stdout + allowList: + - '{"destination_port":["32400"]}' + - '{"source_port":["32400"]}' # Ingress VIP announcement on the NAT LAN. IP pool / CiliumL2AnnouncementPolicy # (from the reserved 10.73.0.240-.250 range) is authored by WS-NET in Phase 2. diff --git a/platform/cilium/helmrelease.yaml b/platform/cilium/helmrelease.yaml index 831c756..c1f2b65 100644 --- a/platform/cilium/helmrelease.yaml +++ b/platform/cilium/helmrelease.yaml @@ -101,6 +101,60 @@ spec: enabled: true ui: enabled: true # exposure gated behind oauth2-proxy forward-auth later (ADR-0014) + # --- STATIC FLOW EXPORT: persist the PRE-SNAT client address for Plex (2026-10-04) --- + # [parity] MUST equal Ansible: hubble.export.static.{enabled,filePath,allowList} + # + # WHY THIS EXISTS. Plex's own log cannot show who connected. Every LoadBalancer Service in + # this cluster is `externalTrafficPolicy: Cluster`, so the node holding the L2 lease for the + # VIP SNATs the client before forwarding to the backend pod. Measured 2026-10-04: Plex had + # logged 3,214 requests from `10.0.1.x` (the lease holder k3s-prod-3's own address, Cilium + # identity 1 = reserved:host) and ZERO from any 192.168.x address, ever. No X-Forwarded-For + # either -- it serves raw tcp/32400, not HTTP through the Gateway. + # + # THE ADDRESS IS NOT LOST, ONLY UNRECORDED. Cilium observes the flow at ingress, BEFORE its + # own translation, so Hubble already has the true source. Measured on the lease holder: + # 192.168.2.133 -> 10.0.0.135 (real LAN client -> the Plex pod) + # 10.0.1.78 -> 10.0.0.135 (the same traffic, post-SNAT -- what Plex sees) + # What was missing is RETENTION: the ring buffer was 4095/4095 flows at ~303 flows/s, i.e. + # about THIRTEEN SECONDS of history, with hubble metrics disabled and no exporter configured. + # + # WHY NOT FIX THE SNAT INSTEAD. `externalTrafficPolicy: Local` would preserve the address in + # Plex's own log, but cilium 1.20's L2 announcer is explicitly incompatible with it -- upstream + # docs for this exact version: "incompatible with the externalTrafficPolicy: Local on services + # as it may cause service IPs to be announced on nodes without pods causing traffic drops", + # and "L2 announcements isn't currently aware of this setting". Live proof that this is not + # theoretical: the VIP is announced by k3s-prod-3 while the pod runs on k3s-prod-1, so the + # flip would black-hole tcp/32400 immediately. Making it safe would mean pinning BOTH the pod + # and the announcement to one node and giving up Plex's ability to reschedule -- a real + # availability loss to buy a log field. (It is the same trap platform/crafty/service-l4.yaml + # documents from an 8-day Minecraft outage.) DSR was also rejected: routing-mode here is + # vxlan, not native/geneve, and DSR returns bypass the ingress node, which breaks the SOHO + # router's stateful NAT on the remote path. + # + # `filePath: stdout` RATHER THAN A FILE, deliberately. Grafana Alloy already runs as a + # DaemonSet tailing every pod's logs into Loki (platform/monitoring/alloy.yaml), so stdout + # reaches Loki with NO new hostPath mount, no file-rotation handling and no Alloy change. + # Upstream documents it: "To get Hubble flows directly exported to the logs instead of written + # to a rotated file, `stdout` can be defined as `hubble-export-file-path`." fileMaxSizeMb / + # fileMaxBackups are therefore left at chart defaults and do not apply. + # Query: {namespace="kube-system", app="cilium"} |= "32400" + # + # SCOPED BY PORT, NOT BY NAMESPACE -- and that is a measured decision, not a preference. A + # `--to-namespace plex` query returned ONLY the post-SNAT flows; the flow carrying the real + # client address did NOT match it, because at the point of observation the destination + # identity is not yet resolved to the plex pod. Filtering by namespace would therefore drop + # exactly the record this change exists to capture. Port it is. + # + # Both directions are listed so the return path is captured too. Volume is small (a few + # hundred to a few thousand lines a day); if it ever becomes noisy, narrow it with + # `fieldMask` rather than by removing a filter. + export: + static: + enabled: true + filePath: stdout + allowList: + - '{"destination_port":["32400"]}' + - '{"source_port":["32400"]}' # --- L2 announcement for the ingress VIP on the flat LAN (docs/09, OQ-1 resolved) --- # [parity] MUST equal Ansible: l2announcements.enabled: true. IP pool + # CiliumL2AnnouncementPolicy (reserved 10.73.0.240-.250) authored by WS-NET in Phase 2. diff --git a/platform/cilium/values.yaml b/platform/cilium/values.yaml index 4f0bf41..b37d11e 100644 --- a/platform/cilium/values.yaml +++ b/platform/cilium/values.yaml @@ -38,6 +38,18 @@ hubble: enabled: true ui: enabled: true + # [parity] static flow export -> cilium-agent stdout -> Alloy -> Loki. Records the PRE-SNAT + # client address for Plex's raw tcp/32400, which Plex's own log cannot show (every LB Service here + # is externalTrafficPolicy: Cluster, so the VIP lease holder SNATs the client). Scoped BY PORT, + # not namespace: the flow carrying the real client IP does not match a namespace filter. Full + # rationale, measurements and the rejected alternatives are in helmrelease.yaml. + export: + static: + enabled: true + filePath: stdout + allowList: + - '{"destination_port":["32400"]}' + - '{"source_port":["32400"]}' # [parity] ingress VIP announcement on the flat LAN (docs/09, OQ-1) l2announcements: enabled: true