Files
home-network-k3s-impl/infra/ansible/roles/cilium/templates/cilium-values.yaml.j2
T
jakman85andClaude Opus 5 989649e64a
gate / gate (pull_request) Successful in 1m49s
gate / conformance (pull_request) Has been skipped
feat(cilium): export Plex's pre-SNAT client address via Hubble, scoped to tcp/32400
The requirement is log visibility only -- which address connected to Plex.

Plex's own log cannot show it. Every LoadBalancer Service here is
externalTrafficPolicy: Cluster, so the node holding the VIP's L2 lease SNATs
the client before forwarding to the backend pod. Measured: Plex had logged
3,214 requests from 10.0.1.x (lease holder k3s-prod-3's own address, Cilium
identity 1 = reserved:host) and ZERO from any 192.168.x address, ever. No
X-Forwarded-For either -- it serves raw tcp/32400, not HTTP via the Gateway.
Nothing on the router does this; there is no hairpin NAT. It is Cilium.

The address is not lost, only unrecorded. Cilium observes the flow at ingress,
BEFORE its own translation, so Hubble already has it:

  192.168.2.133 -> 10.0.0.135   real LAN client -> the Plex pod
  10.0.1.78     -> 10.0.0.135   same traffic post-SNAT, what Plex sees

What was missing is RETENTION: 4095/4095 flows at ~303/s is about thirteen
seconds of history, metrics disabled, no exporter configured.

So enable the static exporter, port-scoped, to the agent's stdout -- Alloy
already tails every pod into Loki, so this needs no new mount, no rotation
handling and no Alloy change.

  {namespace="kube-system", app="cilium"} |= "32400"

Scoped BY PORT, not namespace, on evidence: --to-namespace plex returned only
the post-SNAT flows; the record carrying the real client address did not match
it. A namespace filter would drop exactly what this exists to capture.

Rejected, with reasons: eTP:Local is incompatible with cilium 1.20's L2
announcer per upstream docs for this version ("may cause service IPs to be
announced on nodes without pods causing traffic drops") -- and the VIP is on
prod-3 while the pod is on prod-1, so it would black-hole 32400 immediately;
making it safe costs Plex its ability to reschedule. DSR: routing-mode is
vxlan, not native/geneve, and DSR returns bypass the ingress node, breaking
the router's stateful NAT on the remote path.

Also corrects the repo's own record: 2026-10-03 says the SNAT lands in
192.168.0.0/16 (the node's LAN address). It is 10.0.1.x, the cilium_host
inside the pod CIDR -- which is why LanNetworksBandwidth can never separate
local from remote while eTP is Cluster.

UNVERIFIED and stated in the handoff: nothing is reconciled; `filePath: stdout`
is taken from a verbatim upstream sentence but not confirmed against 1.20.1
source (a wrong notation is a silent no-op, with the file+Alloy fallback
documented); the port FlowFilter has no upstream example; and this change
ROLLS CILIUM -- prod-3 announces the Gateway, PiHole DNS and Plex VIPs, so a
lease move during the rollout could briefly interrupt LAN DNS.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-04 16:36:12 -05:00

96 lines
4.7 KiB
Django/Jinja

# Cilium Helm values — rendered by the Ansible bootstrap (T-1.3).
# PARITY CONTRACT (docs/02): the fields marked [parity] MUST equal the
# platform/cilium HelmRelease values, or Flux and the bootstrap fight over the CNI.
---
# [parity] k3s kube-proxy is disabled → Cilium replaces it (ADR-0001/0002).
kubeProxyReplacement: true
# [parity] = k3s_apiserver_endpoint, THE HA CONTROL-PLANE VIP (kube_vip_address) — dev
# 192.168.1.49, prod 192.168.2.9. NOT k3s_api_host, which is the FOUNDER'S OWN node IP:
# pointing the CNI's kube-proxy replacement at one machine on a 3-server HA cluster means
# losing that machine stops every service/policy update reaching the datapath, silently
# (the existing datapath keeps working from Cilium's cache, so nothing looks broken). That
# was the live state of prod until 2026-09-06 — all three cilium-agents carried
# KUBERNETES_SERVICE_HOST=192.168.2.10 while kube-vip held .9. See group_vars/all.yml.
#
# [parity] The Flux side is platform/cilium/helmrelease.yaml's k8sServiceHost:
# "${K3S_API_HOST}", whose per-environment value comes from clusters/<env>/kustomization.yaml.
# THAT VALUE MUST EQUAL k3s_apiserver_endpoint for the SAME environment. If it does not,
# Flux and this bootstrap render different control-plane paths and fight over the CNI.
#
# THE VIP MUST BE UP BEFORE THIS ROLE RUNS. playbooks/site.yml imports kube-vip.yml between
# k3s.yml and cilium.yml for exactly this reason, and roles/kube_vip blocks until the VIP
# answers. Rendering this against a VIP that nothing announces yet reproduces the
# 2026-09-01 shape: agents that cannot reach the API server on a cold cluster.
k8sServiceHost: "{{ k3s_apiserver_endpoint }}"
k8sServicePort: {{ k3s_api_port }}
# Ingress via Gateway API — no Traefik (docs/09, ADR-0002).
gatewayAPI:
enabled: true
# Egress-class steering (ADR-0002; VPN model ADR-0013, Phase 2/3).
egressGateway:
enabled: true
# Egress Gateway REQUIRES masquerade (ipv4 + eBPF). eBPF masquerade is the correct
# pairing with kubeProxyReplacement. [parity] must match platform/cilium HelmRelease.
enableIPv4Masquerade: true
bpf:
masquerade: true
# Flow observability / isolation audit (ADR-0002).
hubble:
enabled: true
relay:
enabled: true
ui:
enabled: true
# Static flow export -> cilium-agent stdout. Kept EQUAL to the Flux values
# (platform/cilium/helmrelease.yaml), which carry the full rationale: it records the PRE-SNAT
# client address for Plex's raw tcp/32400, which Plex itself cannot log because every
# LoadBalancer Service here is externalTrafficPolicy: Cluster and the VIP lease holder SNATs the
# client. Present here so a BOOTSTRAP render does not differ from what Flux reconciles -- a
# mismatch would show up as a permanent Helm diff and an avoidable agent rollout. Scoped by PORT,
# not namespace: the flow carrying the real client address does not match a namespace filter.
export:
static:
enabled: true
filePath: stdout
allowList:
- '{"destination_port":["32400"]}'
- '{"source_port":["32400"]}'
# Ingress VIP announcement on the NAT LAN. IP pool / CiliumL2AnnouncementPolicy
# (from the reserved 10.73.0.240-.250 range) is authored by WS-NET in Phase 2.
l2announcements:
enabled: true
# Point Cilium at the host's real cgroup2 root (/sys/fs/cgroup, mounted by systemd) instead of
# auto-mounting. NOT /run/cilium/cgroupv2: nothing mounts cgroup2 there, so the agent mounted its
# OWN namespaced cgroup and socket-LB covered only itself — host processes (the apiserver) lost
# Service/LB translation. [parity] platform/cilium/helmrelease.yaml explains (2026-10-03).
cgroup:
autoMount:
enabled: false
hostRoot: /sys/fs/cgroup
# Roll the cilium agent DaemonSet ONE node at a time. [parity] must match the
# platform/cilium HelmRelease. The cilium 1.20.1 chart defaults this to 2, which on a
# 3-node cluster lets one bad render take down two of three agents simultaneously —
# exactly the 2026-09-01 incident, where k8sServiceHost: "" made the agent's `config`
# init container fall back to the in-cluster ClusterIP 10.43.0.1:443. Only the Cilium
# agent can translate that ClusterIP, and the agent in question was the one being
# replaced: `dial tcp 10.43.0.1:443: i/o timeout`, deadlocked.
# A CNI rollout has no healthy-peer fallback — the component being upgraded is the one
# the upgrade needs to reach the API server. This bites hardest on a COLD BOOTSTRAP
# (this role's job), where nothing healthy exists to recover from. With maxUnavailable: 1
# the DaemonSet will not proceed past an unready pod, bounding damage to a single node.
# Cost: Cilium upgrades serialise and take longer; at 3 nodes that is an acceptable trade.
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
operator:
replicas: {{ cilium_operator_replicas }}