jakman85andClaude Opus 5
989649e64a
The requirement is log visibility only -- which address connected to Plex.
Plex's own log cannot show it. Every LoadBalancer Service here is
externalTrafficPolicy: Cluster, so the node holding the VIP's L2 lease SNATs
the client before forwarding to the backend pod. Measured: Plex had logged
3,214 requests from 10.0.1.x (lease holder k3s-prod-3's own address, Cilium
identity 1 = reserved:host) and ZERO from any 192.168.x address, ever. No
X-Forwarded-For either -- it serves raw tcp/32400, not HTTP via the Gateway.
Nothing on the router does this; there is no hairpin NAT. It is Cilium.
The address is not lost, only unrecorded. Cilium observes the flow at ingress,
BEFORE its own translation, so Hubble already has it:
192.168.2.133 -> 10.0.0.135 real LAN client -> the Plex pod
10.0.1.78 -> 10.0.0.135 same traffic post-SNAT, what Plex sees
What was missing is RETENTION: 4095/4095 flows at ~303/s is about thirteen
seconds of history, metrics disabled, no exporter configured.
So enable the static exporter, port-scoped, to the agent's stdout -- Alloy
already tails every pod into Loki, so this needs no new mount, no rotation
handling and no Alloy change.
{namespace="kube-system", app="cilium"} |= "32400"
Scoped BY PORT, not namespace, on evidence: --to-namespace plex returned only
the post-SNAT flows; the record carrying the real client address did not match
it. A namespace filter would drop exactly what this exists to capture.
Rejected, with reasons: eTP:Local is incompatible with cilium 1.20's L2
announcer per upstream docs for this version ("may cause service IPs to be
announced on nodes without pods causing traffic drops") -- and the VIP is on
prod-3 while the pod is on prod-1, so it would black-hole 32400 immediately;
making it safe costs Plex its ability to reschedule. DSR: routing-mode is
vxlan, not native/geneve, and DSR returns bypass the ingress node, breaking
the router's stateful NAT on the remote path.
Also corrects the repo's own record: 2026-10-03 says the SNAT lands in
192.168.0.0/16 (the node's LAN address). It is 10.0.1.x, the cilium_host
inside the pod CIDR -- which is why LanNetworksBandwidth can never separate
local from remote while eTP is Cluster.
UNVERIFIED and stated in the handoff: nothing is reconciled; `filePath: stdout`
is taken from a verbatim upstream sentence but not confirmed against 1.20.1
source (a wrong notation is a silent no-op, with the file+Alloy fallback
documented); the port FlowFilter has no upstream example; and this change
ROLLS CILIUM -- prod-3 announces the Gateway, PiHole DNS and Plex VIPs, so a
lease move during the rollout could briefly interrupt LAN DNS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
217 lines
14 KiB
YAML
217 lines
14 KiB
YAML
---
|
|
# Cilium — CNI, kube-proxy replacement, Gateway API, Egress Gateway, Hubble (ADR-0002).
|
|
# Serves: R-NET-01..07, R-DNS-01, R-SEC-02. Owner: WS-NET (bootstrap shared with WS-INFRA).
|
|
#
|
|
# ============================ PARITY CONTRACT ============================
|
|
# Cilium's INITIAL install is performed by the Ansible k3s bootstrap (spec docs/02, Phase 1) —
|
|
# a cluster needs a CNI before Flux can run. This HelmRelease ADOPTS and manages Cilium's config
|
|
# thereafter (docs/05). The values below MUST EQUAL the Ansible bootstrap values, or Flux and the
|
|
# bootstrap will fight over the running CNI and break cluster networking.
|
|
#
|
|
# SOURCE OF TRUTH (do not diverge without updating both):
|
|
# infra/ansible/roles/cilium/templates/cilium-values.yaml.j2
|
|
# infra/ansible/roles/cilium/defaults/main.yml (cilium_version)
|
|
# infra/ansible/inventory/group_vars/all.yml (k3s_apiserver_endpoint, k3s_api_port)
|
|
#
|
|
# Every value tagged [parity] below is asserted identical to the Ansible render. If you change one
|
|
# here, change it in the Ansible role in the same PR. (docs/02 Cilium-ownership contract.)
|
|
# ========================================================================
|
|
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
kind: HelmRelease
|
|
metadata:
|
|
name: cilium
|
|
namespace: kube-system
|
|
spec:
|
|
interval: 30m
|
|
chart:
|
|
spec:
|
|
chart: cilium
|
|
# [parity] Pinned to the exact version the Ansible bootstrap installs
|
|
# (cilium_version in roles/cilium/defaults/main.yml). Renovate bumps both together.
|
|
version: "1.20.1"
|
|
sourceRef:
|
|
kind: HelmRepository
|
|
name: cilium
|
|
namespace: kube-system
|
|
install:
|
|
remediation:
|
|
retries: 3
|
|
upgrade:
|
|
remediation:
|
|
retries: 3
|
|
# Values are inlined so the intent (Cilium REPLACES flannel + kube-proxy + Traefik) is visible at
|
|
# the release. MUST equal the Ansible bootstrap render (see PARITY CONTRACT header).
|
|
values:
|
|
# --- kube-proxy replacement (ADR-0001/0002: k3s kube-proxy disabled) ---
|
|
# [parity] MUST equal Ansible: kubeProxyReplacement: true
|
|
kubeProxyReplacement: true
|
|
# [parity] MUST equal Ansible: k8sServiceHost/k8sServicePort = the k3s API endpoint
|
|
# (**k3s_apiserver_endpoint** / k3s_api_port, infra/ansible/inventory/group_vars/all.yml).
|
|
# Setting these wrong breaks Cilium's connection to the API server on adopt.
|
|
#
|
|
# THIS IS THE CONTROL-PLANE VIP (kube-vip), NOT A NODE ADDRESS — operator decision 2026-09-06.
|
|
# ${K3S_API_HOST} (cluster-vars, R-ENV-02): dev 192.168.1.49 | prod 192.168.2.9, each equal to
|
|
# that environment's kube_vip_address. The knob's NAME still says "host"; read it as "the API
|
|
# host Cilium dials", and see clusters/base/cluster-vars.yaml for why it was not renamed.
|
|
#
|
|
# WHY, in one line: this value used to be the PRIMARY NODE's IP, so on a 3-server HA cluster the
|
|
# CNI's control-plane path depended on one machine. Verified on live prod 2026-09-06 — every
|
|
# cilium-agent carried KUBERNETES_SERVICE_HOST=192.168.2.10 (k3s-prod-1) while kube-vip held .9.
|
|
# Losing that node does not break the datapath (Cilium keeps serving from cache); it silently
|
|
# stops every service and policy update from ever landing, which is worse to detect.
|
|
#
|
|
# [parity] IS NOW AGAINST k3s_apiserver_endpoint AND NOT k3s_api_host. Those were one Ansible
|
|
# variable until 2026-09-06 and the conflation is the defect. k3s_api_host still exists and
|
|
# still holds the FOUNDER's own address, because roles/k3s_server's secondary
|
|
# `--server https://<host>:6443` join must target a host that is already up — on a COLD build
|
|
# no VIP exists yet. So: founder for the join, VIP for the CNI. If you change this line, change
|
|
# cilium-values.yaml.j2 in the same PR, and point it at the SAME variable.
|
|
#
|
|
# The `platform` Flux Kustomization substitutes it; the Ansible side is parameterized from
|
|
# inventory (k3s_apiserver_endpoint derives from kube_vip_address), so the parity contract holds
|
|
# per-environment, not per-literal.
|
|
#
|
|
# NEEDS NO CERT WORK: k3s_tls_sans has always carried kube_vip_address alongside every node IP,
|
|
# so both addresses are already SANs on the serving cert (verified live 2026-09-06 — prod
|
|
# answers through https://192.168.2.9:6443 under full CA verification, no insecure skip). This
|
|
# switch reissues nothing and restarts no apiserver.
|
|
k8sServiceHost: "${K3S_API_HOST}"
|
|
k8sServicePort: 6443
|
|
# --- Gateway API (ingress; replaces Traefik) — docs/09, ADR-0002 ---
|
|
# [parity] MUST equal Ansible: gatewayAPI.enabled: true
|
|
gatewayAPI:
|
|
enabled: true
|
|
# --- Egress Gateway (egress-class steering; part of the VPN model, ADR-0013) ---
|
|
# [parity] MUST equal Ansible: egressGateway.enabled: true
|
|
egressGateway:
|
|
enabled: true
|
|
# --- Masquerade (REQUIRED by Egress Gateway) — Phase-1 crash fix ---
|
|
# [parity] MUST equal Ansible: enableIPv4Masquerade: true + bpf.masquerade: true.
|
|
# This exact pairing (ipv4 masquerade via eBPF, the correct companion to kubeProxyReplacement)
|
|
# was the fix for the Phase-1 agent crash. Enabling one without the other regresses it —
|
|
# keep both, and keep them equal to the Ansible values.
|
|
enableIPv4Masquerade: true
|
|
bpf:
|
|
masquerade: true
|
|
# --- Hubble (flow observability / isolation audit) ---
|
|
# [parity] MUST equal Ansible: hubble.{enabled,relay.enabled,ui.enabled}: true
|
|
hubble:
|
|
enabled: true
|
|
relay:
|
|
enabled: true
|
|
ui:
|
|
enabled: true # exposure gated behind oauth2-proxy forward-auth later (ADR-0014)
|
|
# --- STATIC FLOW EXPORT: persist the PRE-SNAT client address for Plex (2026-10-04) ---
|
|
# [parity] MUST equal Ansible: hubble.export.static.{enabled,filePath,allowList}
|
|
#
|
|
# WHY THIS EXISTS. Plex's own log cannot show who connected. Every LoadBalancer Service in
|
|
# this cluster is `externalTrafficPolicy: Cluster`, so the node holding the L2 lease for the
|
|
# VIP SNATs the client before forwarding to the backend pod. Measured 2026-10-04: Plex had
|
|
# logged 3,214 requests from `10.0.1.x` (the lease holder k3s-prod-3's own address, Cilium
|
|
# identity 1 = reserved:host) and ZERO from any 192.168.x address, ever. No X-Forwarded-For
|
|
# either -- it serves raw tcp/32400, not HTTP through the Gateway.
|
|
#
|
|
# THE ADDRESS IS NOT LOST, ONLY UNRECORDED. Cilium observes the flow at ingress, BEFORE its
|
|
# own translation, so Hubble already has the true source. Measured on the lease holder:
|
|
# 192.168.2.133 -> 10.0.0.135 (real LAN client -> the Plex pod)
|
|
# 10.0.1.78 -> 10.0.0.135 (the same traffic, post-SNAT -- what Plex sees)
|
|
# What was missing is RETENTION: the ring buffer was 4095/4095 flows at ~303 flows/s, i.e.
|
|
# about THIRTEEN SECONDS of history, with hubble metrics disabled and no exporter configured.
|
|
#
|
|
# WHY NOT FIX THE SNAT INSTEAD. `externalTrafficPolicy: Local` would preserve the address in
|
|
# Plex's own log, but cilium 1.20's L2 announcer is explicitly incompatible with it -- upstream
|
|
# docs for this exact version: "incompatible with the externalTrafficPolicy: Local on services
|
|
# as it may cause service IPs to be announced on nodes without pods causing traffic drops",
|
|
# and "L2 announcements isn't currently aware of this setting". Live proof that this is not
|
|
# theoretical: the VIP is announced by k3s-prod-3 while the pod runs on k3s-prod-1, so the
|
|
# flip would black-hole tcp/32400 immediately. Making it safe would mean pinning BOTH the pod
|
|
# and the announcement to one node and giving up Plex's ability to reschedule -- a real
|
|
# availability loss to buy a log field. (It is the same trap platform/crafty/service-l4.yaml
|
|
# documents from an 8-day Minecraft outage.) DSR was also rejected: routing-mode here is
|
|
# vxlan, not native/geneve, and DSR returns bypass the ingress node, which breaks the SOHO
|
|
# router's stateful NAT on the remote path.
|
|
#
|
|
# `filePath: stdout` RATHER THAN A FILE, deliberately. Grafana Alloy already runs as a
|
|
# DaemonSet tailing every pod's logs into Loki (platform/monitoring/alloy.yaml), so stdout
|
|
# reaches Loki with NO new hostPath mount, no file-rotation handling and no Alloy change.
|
|
# Upstream documents it: "To get Hubble flows directly exported to the logs instead of written
|
|
# to a rotated file, `stdout` can be defined as `hubble-export-file-path`." fileMaxSizeMb /
|
|
# fileMaxBackups are therefore left at chart defaults and do not apply.
|
|
# Query: {namespace="kube-system", app="cilium"} |= "32400"
|
|
#
|
|
# SCOPED BY PORT, NOT BY NAMESPACE -- and that is a measured decision, not a preference. A
|
|
# `--to-namespace plex` query returned ONLY the post-SNAT flows; the flow carrying the real
|
|
# client address did NOT match it, because at the point of observation the destination
|
|
# identity is not yet resolved to the plex pod. Filtering by namespace would therefore drop
|
|
# exactly the record this change exists to capture. Port it is.
|
|
#
|
|
# Both directions are listed so the return path is captured too. Volume is small (a few
|
|
# hundred to a few thousand lines a day); if it ever becomes noisy, narrow it with
|
|
# `fieldMask` rather than by removing a filter.
|
|
export:
|
|
static:
|
|
enabled: true
|
|
filePath: stdout
|
|
allowList:
|
|
- '{"destination_port":["32400"]}'
|
|
- '{"source_port":["32400"]}'
|
|
# --- L2 announcement for the ingress VIP on the flat LAN (docs/09, OQ-1 resolved) ---
|
|
# [parity] MUST equal Ansible: l2announcements.enabled: true. IP pool +
|
|
# CiliumL2AnnouncementPolicy (reserved 10.73.0.240-.250) authored by WS-NET in Phase 2.
|
|
l2announcements:
|
|
enabled: true
|
|
# --- cgroup: point Cilium at the HOST's cgroup2 root, which systemd mounts at /sys/fs/cgroup ---
|
|
# [parity] MUST equal Ansible cgroup block (roles/cilium/templates/cilium-values.yaml.j2).
|
|
#
|
|
# FIXED 2026-10-03 — hostRoot WAS /run/cilium/cgroupv2, AND THAT SILENTLY BROKE SOCKET-LB FOR
|
|
# EVERY HOST PROCESS. With autoMount disabled, the chart only bind-mounts whatever already exists
|
|
# at hostRoot ("it's expected that users have mounted cgroup2 filesystem at the specified
|
|
# cgroup.hostRoot" — cilium 1.20.1 values.yaml). Nothing ever mounted cgroup2 at
|
|
# /run/cilium/cgroupv2 on these nodes (it is an empty DirectoryOrCreate hostPath), so the agent
|
|
# mounted one ITSELF from inside its own private cgroup namespace. That mount's root is the
|
|
# agent container's OWN leaf cgroup, so cil_sock4_connect & co. were attached there and covered
|
|
# only the agent. Every other host-netns process (the k3s apiserver in
|
|
# /system.slice/k3s.service, curl over SSH) was untranslated: Service ClusterIPs timed out from
|
|
# every node, and the node holding the Gateway's L2 lease could not reach the Gateway VIP at all
|
|
# ("no route to host"). That is why k3s-prod-1's apiserver logged `oidc: authenticator not
|
|
# initialized` from 2026-09-12 on — kubectl SSO through the kube-vip VIP failed outright.
|
|
# Measured on k3s-prod-1 before this change: TCP connect to 192.168.2.20:443, 10.43.225.79:443
|
|
# and 10.43.59.141:8080 SUCCEEDED from inside the agent container (the covered cgroup) and
|
|
# FAILED from the host, same node, same netns. Full write-up:
|
|
# decisions/2026-10-03-cilium-cgroup-hostroot.md.
|
|
#
|
|
# /sys/fs/cgroup IS a cgroup2 mount in the host's root cgroup namespace on every node (verified
|
|
# with findmnt on all three), so bind-mounting it gives the agent the real host root. autoMount
|
|
# stays off because the mount it would create already exists.
|
|
cgroup:
|
|
autoMount:
|
|
enabled: false
|
|
hostRoot: /sys/fs/cgroup
|
|
# --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident ---
|
|
# [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1.
|
|
# The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate.
|
|
# maxUnavailable: 2) — NOT a local choice, and on a 3-node cluster that lets a single bad
|
|
# Helm render take down two of three agents simultaneously. That is exactly what happened
|
|
# when a fails-open post-build substitution shipped k8sServiceHost: "": the agent's `config`
|
|
# init container fell back to the in-cluster ClusterIP 10.43.0.1:443, which only the Cilium
|
|
# agent itself can translate — and the agent translating it was the one being replaced.
|
|
# `dial tcp 10.43.0.1:443: i/o timeout`, deadlocked, on 2 of 3 nodes at once.
|
|
# A CNI rollout has no healthy-peer fallback: the thing being upgraded is the thing the
|
|
# upgrade needs in order to reach the API server. With maxUnavailable: 1 the DaemonSet
|
|
# will not proceed past an unready pod, so a bad render costs one node, not the cluster.
|
|
# This matters most on a COLD PROD BOOTSTRAP, where every knob is new at once and there is
|
|
# no already-healthy agent to recover from (see DECISIONS-LOG 2026-09-01 INCIDENT/RESEARCH).
|
|
# Cost, recorded: Cilium upgrades serialise and get slower. On 3 nodes that is minutes, and
|
|
# it buys a hard stop on the first bad node — the right trade here. Revisit if the cluster
|
|
# grows enough that serialised CNI upgrades become an operational problem.
|
|
updateStrategy:
|
|
type: RollingUpdate
|
|
rollingUpdate:
|
|
maxUnavailable: 1
|
|
# NOTE (NON-parity): operator.replicas is deliberately omitted here. The Ansible dev render sets
|
|
# replicas: 1 to save RAM, but defaults/main.yml declares it explicitly NOT a parity field
|
|
# (Flux/prod may differ). Only the [parity] values above must match the bootstrap.
|
|
#
|
|
# TODO(WS-NET, Phase 2): default-deny baseline, CiliumL2AnnouncementPolicy + IP pool,
|
|
# Egress Gateway policies, Hubble metrics export to kube-prometheus-stack.
|