jakman85andClaude Opus 5
989649e64a
The requirement is log visibility only -- which address connected to Plex.
Plex's own log cannot show it. Every LoadBalancer Service here is
externalTrafficPolicy: Cluster, so the node holding the VIP's L2 lease SNATs
the client before forwarding to the backend pod. Measured: Plex had logged
3,214 requests from 10.0.1.x (lease holder k3s-prod-3's own address, Cilium
identity 1 = reserved:host) and ZERO from any 192.168.x address, ever. No
X-Forwarded-For either -- it serves raw tcp/32400, not HTTP via the Gateway.
Nothing on the router does this; there is no hairpin NAT. It is Cilium.
The address is not lost, only unrecorded. Cilium observes the flow at ingress,
BEFORE its own translation, so Hubble already has it:
192.168.2.133 -> 10.0.0.135 real LAN client -> the Plex pod
10.0.1.78 -> 10.0.0.135 same traffic post-SNAT, what Plex sees
What was missing is RETENTION: 4095/4095 flows at ~303/s is about thirteen
seconds of history, metrics disabled, no exporter configured.
So enable the static exporter, port-scoped, to the agent's stdout -- Alloy
already tails every pod into Loki, so this needs no new mount, no rotation
handling and no Alloy change.
{namespace="kube-system", app="cilium"} |= "32400"
Scoped BY PORT, not namespace, on evidence: --to-namespace plex returned only
the post-SNAT flows; the record carrying the real client address did not match
it. A namespace filter would drop exactly what this exists to capture.
Rejected, with reasons: eTP:Local is incompatible with cilium 1.20's L2
announcer per upstream docs for this version ("may cause service IPs to be
announced on nodes without pods causing traffic drops") -- and the VIP is on
prod-3 while the pod is on prod-1, so it would black-hole 32400 immediately;
making it safe costs Plex its ability to reschedule. DSR: routing-mode is
vxlan, not native/geneve, and DSR returns bypass the ingress node, breaking
the router's stateful NAT on the remote path.
Also corrects the repo's own record: 2026-10-03 says the SNAT lands in
192.168.0.0/16 (the node's LAN address). It is 10.0.1.x, the cilium_host
inside the pod CIDR -- which is why LanNetworksBandwidth can never separate
local from remote while eTP is Cluster.
UNVERIFIED and stated in the handoff: nothing is reconciled; `filePath: stdout`
is taken from a verbatim upstream sentence but not confirmed against 1.20.1
source (a wrong notation is a silent no-op, with the file+Alloy fallback
documented); the port FlowFilter has no upstream example; and this change
ROLLS CILIUM -- prod-3 announces the Gateway, PiHole DNS and Plex VIPs, so a
lease move during the rollout could briefly interrupt LAN DNS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
78 lines
3.9 KiB
YAML
78 lines
3.9 KiB
YAML
# Cilium Helm values — REFERENCE COPY of the parity set (docs/02 Cilium-ownership contract).
|
|
#
|
|
# The AUTHORITATIVE values that Flux reconciles are inlined in helmrelease.yaml. This file is a
|
|
# readable, un-nested mirror of that value set so WS-NET can diff it against the live cluster and
|
|
# the Ansible bootstrap render at a glance. Keep the two in sync.
|
|
#
|
|
# SOURCE OF TRUTH (all [parity] fields MUST equal these):
|
|
# infra/ansible/roles/cilium/templates/cilium-values.yaml.j2
|
|
# infra/ansible/roles/cilium/defaults/main.yml (cilium_version: 1.20.1)
|
|
# infra/ansible/inventory/group_vars/all.yml (k3s_apiserver_endpoint = the kube-vip VIP,
|
|
# k3s_api_port) — NOT k3s_api_host, which is the
|
|
# founder's own address and bootstrap-only
|
|
#
|
|
# chart version (helmrelease.yaml chart.spec.version): 1.20.1 # [parity] = cilium_version
|
|
|
|
# [parity] k3s kube-proxy disabled → Cilium replaces it (ADR-0001/0002)
|
|
kubeProxyReplacement: true
|
|
# [parity] = the k3s API endpoint + port. THE CONTROL-PLANE VIP (kube-vip), not the primary node —
|
|
# changed 2026-09-06; the Ansible half is k3s_apiserver_endpoint, NOT k3s_api_host.
|
|
# Mirrors the HelmRelease's ${K3S_API_HOST} cluster-vars token (R-ENV-02). This file is NOT a
|
|
# kustomize resource, so nothing substitutes it — the token is shown with its dev value beside it.
|
|
k8sServiceHost: "${K3S_API_HOST}" # the CONTROL-PLANE VIP — dev: 192.168.1.49 | prod: 192.168.2.9
|
|
k8sServicePort: 6443
|
|
# [parity] ingress via Gateway API, no Traefik (docs/09)
|
|
gatewayAPI:
|
|
enabled: true
|
|
# [parity] egress-class steering (ADR-0002; VPN model ADR-0013)
|
|
egressGateway:
|
|
enabled: true
|
|
# [parity] Egress Gateway REQUIRES masquerade; ipv4 + eBPF is the Phase-1 crash fix — keep BOTH
|
|
enableIPv4Masquerade: true
|
|
bpf:
|
|
masquerade: true
|
|
# [parity] flow observability / isolation audit
|
|
hubble:
|
|
enabled: true
|
|
relay:
|
|
enabled: true
|
|
ui:
|
|
enabled: true
|
|
# [parity] static flow export -> cilium-agent stdout -> Alloy -> Loki. Records the PRE-SNAT
|
|
# client address for Plex's raw tcp/32400, which Plex's own log cannot show (every LB Service here
|
|
# is externalTrafficPolicy: Cluster, so the VIP lease holder SNATs the client). Scoped BY PORT,
|
|
# not namespace: the flow carrying the real client IP does not match a namespace filter. Full
|
|
# rationale, measurements and the rejected alternatives are in helmrelease.yaml.
|
|
export:
|
|
static:
|
|
enabled: true
|
|
filePath: stdout
|
|
allowList:
|
|
- '{"destination_port":["32400"]}'
|
|
- '{"source_port":["32400"]}'
|
|
# [parity] ingress VIP announcement on the flat LAN (docs/09, OQ-1)
|
|
l2announcements:
|
|
enabled: true
|
|
# [parity] the host's cgroup2 root is /sys/fs/cgroup (systemd mounts it); do not auto-mount.
|
|
# NOT /run/cilium/cgroupv2: nothing mounts cgroup2 there, so socket-LB attached to the agent's own
|
|
# cgroup and host processes lost Service/LB translation (2026-10-03 — see helmrelease.yaml).
|
|
cgroup:
|
|
autoMount:
|
|
enabled: false
|
|
hostRoot: /sys/fs/cgroup
|
|
# [parity] roll the agent DaemonSet ONE node at a time. Overrides the cilium 1.20.1 chart
|
|
# default of 2, which on 3 nodes lets a single bad render break 2/3 agents at once — as the
|
|
# 2026-09-01 incident did (k8sServiceHost: "" → the `config` init container fell back to the
|
|
# ClusterIP 10.43.0.1:443, translatable only by the very agent being replaced). A CNI upgrade
|
|
# has no healthy-peer fallback, least of all on a cold prod bootstrap; maxUnavailable: 1 stops
|
|
# the rollout at the first unready pod. Cost: serialised, slower upgrades (fine at 3 nodes).
|
|
updateStrategy:
|
|
type: RollingUpdate
|
|
rollingUpdate:
|
|
maxUnavailable: 1
|
|
# NON-parity (Flux/prod may differ; dev sets 1 to save RAM) — omitted from the HelmRelease:
|
|
# operator.replicas
|
|
#
|
|
# TODO (Phase 2, WS-NET): ipam / ipv4NativeRoutingCIDR pinned to cluster CIDRs; east-west
|
|
# encryption (wireguard/ipsec) decision (WS-SEC/WS-NET).
|