--- # Cilium — CNI, kube-proxy replacement, Gateway API, Egress Gateway, Hubble (ADR-0002). # Serves: R-NET-01..07, R-DNS-01, R-SEC-02. Owner: WS-NET (bootstrap shared with WS-INFRA). # # ============================ PARITY CONTRACT ============================ # Cilium's INITIAL install is performed by the Ansible k3s bootstrap (spec docs/02, Phase 1) — # a cluster needs a CNI before Flux can run. This HelmRelease ADOPTS and manages Cilium's config # thereafter (docs/05). The values below MUST EQUAL the Ansible bootstrap values, or Flux and the # bootstrap will fight over the running CNI and break cluster networking. # # SOURCE OF TRUTH (do not diverge without updating both): # infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 # infra/ansible/roles/cilium/defaults/main.yml (cilium_version) # infra/ansible/inventory/group_vars/all.yml (k3s_apiserver_endpoint, k3s_api_port) # # Every value tagged [parity] below is asserted identical to the Ansible render. If you change one # here, change it in the Ansible role in the same PR. (docs/02 Cilium-ownership contract.) # ======================================================================== apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: cilium namespace: kube-system spec: interval: 30m chart: spec: chart: cilium # [parity] Pinned to the exact version the Ansible bootstrap installs # (cilium_version in roles/cilium/defaults/main.yml). Renovate bumps both together. version: "1.20.1" sourceRef: kind: HelmRepository name: cilium namespace: kube-system install: remediation: retries: 3 upgrade: remediation: retries: 3 # Values are inlined so the intent (Cilium REPLACES flannel + kube-proxy + Traefik) is visible at # the release. MUST equal the Ansible bootstrap render (see PARITY CONTRACT header). values: # --- kube-proxy replacement (ADR-0001/0002: k3s kube-proxy disabled) --- # [parity] MUST equal Ansible: kubeProxyReplacement: true kubeProxyReplacement: true # [parity] MUST equal Ansible: k8sServiceHost/k8sServicePort = the k3s API endpoint # (**k3s_apiserver_endpoint** / k3s_api_port, infra/ansible/inventory/group_vars/all.yml). # Setting these wrong breaks Cilium's connection to the API server on adopt. # # THIS IS THE CONTROL-PLANE VIP (kube-vip), NOT A NODE ADDRESS — operator decision 2026-09-06. # ${K3S_API_HOST} (cluster-vars, R-ENV-02): dev 192.168.1.49 | prod 192.168.2.9, each equal to # that environment's kube_vip_address. The knob's NAME still says "host"; read it as "the API # host Cilium dials", and see clusters/base/cluster-vars.yaml for why it was not renamed. # # WHY, in one line: this value used to be the PRIMARY NODE's IP, so on a 3-server HA cluster the # CNI's control-plane path depended on one machine. Verified on live prod 2026-09-06 — every # cilium-agent carried KUBERNETES_SERVICE_HOST=192.168.2.10 (k3s-prod-1) while kube-vip held .9. # Losing that node does not break the datapath (Cilium keeps serving from cache); it silently # stops every service and policy update from ever landing, which is worse to detect. # # [parity] IS NOW AGAINST k3s_apiserver_endpoint AND NOT k3s_api_host. Those were one Ansible # variable until 2026-09-06 and the conflation is the defect. k3s_api_host still exists and # still holds the FOUNDER's own address, because roles/k3s_server's secondary # `--server https://:6443` join must target a host that is already up — on a COLD build # no VIP exists yet. So: founder for the join, VIP for the CNI. If you change this line, change # cilium-values.yaml.j2 in the same PR, and point it at the SAME variable. # # The `platform` Flux Kustomization substitutes it; the Ansible side is parameterized from # inventory (k3s_apiserver_endpoint derives from kube_vip_address), so the parity contract holds # per-environment, not per-literal. # # NEEDS NO CERT WORK: k3s_tls_sans has always carried kube_vip_address alongside every node IP, # so both addresses are already SANs on the serving cert (verified live 2026-09-06 — prod # answers through https://192.168.2.9:6443 under full CA verification, no insecure skip). This # switch reissues nothing and restarts no apiserver. k8sServiceHost: "${K3S_API_HOST}" k8sServicePort: 6443 # --- Gateway API (ingress; replaces Traefik) — docs/09, ADR-0002 --- # [parity] MUST equal Ansible: gatewayAPI.enabled: true gatewayAPI: enabled: true # --- Egress Gateway (egress-class steering; part of the VPN model, ADR-0013) --- # [parity] MUST equal Ansible: egressGateway.enabled: true egressGateway: enabled: true # --- Masquerade (REQUIRED by Egress Gateway) — Phase-1 crash fix --- # [parity] MUST equal Ansible: enableIPv4Masquerade: true + bpf.masquerade: true. # This exact pairing (ipv4 masquerade via eBPF, the correct companion to kubeProxyReplacement) # was the fix for the Phase-1 agent crash. Enabling one without the other regresses it — # keep both, and keep them equal to the Ansible values. enableIPv4Masquerade: true bpf: masquerade: true # --- Hubble (flow observability / isolation audit) --- # [parity] MUST equal Ansible: hubble.{enabled,relay.enabled,ui.enabled}: true hubble: enabled: true relay: enabled: true ui: enabled: true # exposure gated behind oauth2-proxy forward-auth later (ADR-0014) # --- STATIC FLOW EXPORT: persist the PRE-SNAT client address for Plex (2026-10-04) --- # [parity] MUST equal Ansible: hubble.export.static.{enabled,filePath,allowList} # # WHY THIS EXISTS. Plex's own log cannot show who connected. Every LoadBalancer Service in # this cluster is `externalTrafficPolicy: Cluster`, so the node holding the L2 lease for the # VIP SNATs the client before forwarding to the backend pod. Measured 2026-10-04: Plex had # logged 3,214 requests from `10.0.1.x` (the lease holder k3s-prod-3's own address, Cilium # identity 1 = reserved:host) and ZERO from any 192.168.x address, ever. No X-Forwarded-For # either -- it serves raw tcp/32400, not HTTP through the Gateway. # # THE ADDRESS IS NOT LOST, ONLY UNRECORDED. Cilium observes the flow at ingress, BEFORE its # own translation, so Hubble already has the true source. Measured on the lease holder: # 192.168.2.133 -> 10.0.0.135 (real LAN client -> the Plex pod) # 10.0.1.78 -> 10.0.0.135 (the same traffic, post-SNAT -- what Plex sees) # What was missing is RETENTION: the ring buffer was 4095/4095 flows at ~303 flows/s, i.e. # about THIRTEEN SECONDS of history, with hubble metrics disabled and no exporter configured. # # WHY NOT FIX THE SNAT INSTEAD. `externalTrafficPolicy: Local` would preserve the address in # Plex's own log, but cilium 1.20's L2 announcer is explicitly incompatible with it -- upstream # docs for this exact version: "incompatible with the externalTrafficPolicy: Local on services # as it may cause service IPs to be announced on nodes without pods causing traffic drops", # and "L2 announcements isn't currently aware of this setting". Live proof that this is not # theoretical: the VIP is announced by k3s-prod-3 while the pod runs on k3s-prod-1, so the # flip would black-hole tcp/32400 immediately. Making it safe would mean pinning BOTH the pod # and the announcement to one node and giving up Plex's ability to reschedule -- a real # availability loss to buy a log field. (It is the same trap platform/crafty/service-l4.yaml # documents from an 8-day Minecraft outage.) DSR was also rejected: routing-mode here is # vxlan, not native/geneve, and DSR returns bypass the ingress node, which breaks the SOHO # router's stateful NAT on the remote path. # # `filePath: stdout` RATHER THAN A FILE, deliberately. Grafana Alloy already runs as a # DaemonSet tailing every pod's logs into Loki (platform/monitoring/alloy.yaml), so stdout # reaches Loki with NO new hostPath mount, no file-rotation handling and no Alloy change. # Upstream documents it: "To get Hubble flows directly exported to the logs instead of written # to a rotated file, `stdout` can be defined as `hubble-export-file-path`." fileMaxSizeMb / # fileMaxBackups are therefore left at chart defaults and do not apply. # Query: {namespace="kube-system", app="cilium"} |= "32400" # # SCOPED BY PORT, NOT BY NAMESPACE -- and that is a measured decision, not a preference. A # `--to-namespace plex` query returned ONLY the post-SNAT flows; the flow carrying the real # client address did NOT match it, because at the point of observation the destination # identity is not yet resolved to the plex pod. Filtering by namespace would therefore drop # exactly the record this change exists to capture. Port it is. # # Both directions are listed so the return path is captured too. Volume is small (a few # hundred to a few thousand lines a day); if it ever becomes noisy, narrow it with # `fieldMask` rather than by removing a filter. export: static: enabled: true filePath: stdout allowList: - '{"destination_port":["32400"]}' - '{"source_port":["32400"]}' # --- L2 announcement for the ingress VIP on the flat LAN (docs/09, OQ-1 resolved) --- # [parity] MUST equal Ansible: l2announcements.enabled: true. IP pool + # CiliumL2AnnouncementPolicy (reserved 10.73.0.240-.250) authored by WS-NET in Phase 2. l2announcements: enabled: true # --- cgroup: point Cilium at the HOST's cgroup2 root, which systemd mounts at /sys/fs/cgroup --- # [parity] MUST equal Ansible cgroup block (roles/cilium/templates/cilium-values.yaml.j2). # # FIXED 2026-10-03 — hostRoot WAS /run/cilium/cgroupv2, AND THAT SILENTLY BROKE SOCKET-LB FOR # EVERY HOST PROCESS. With autoMount disabled, the chart only bind-mounts whatever already exists # at hostRoot ("it's expected that users have mounted cgroup2 filesystem at the specified # cgroup.hostRoot" — cilium 1.20.1 values.yaml). Nothing ever mounted cgroup2 at # /run/cilium/cgroupv2 on these nodes (it is an empty DirectoryOrCreate hostPath), so the agent # mounted one ITSELF from inside its own private cgroup namespace. That mount's root is the # agent container's OWN leaf cgroup, so cil_sock4_connect & co. were attached there and covered # only the agent. Every other host-netns process (the k3s apiserver in # /system.slice/k3s.service, curl over SSH) was untranslated: Service ClusterIPs timed out from # every node, and the node holding the Gateway's L2 lease could not reach the Gateway VIP at all # ("no route to host"). That is why k3s-prod-1's apiserver logged `oidc: authenticator not # initialized` from 2026-09-12 on — kubectl SSO through the kube-vip VIP failed outright. # Measured on k3s-prod-1 before this change: TCP connect to 192.168.2.20:443, 10.43.225.79:443 # and 10.43.59.141:8080 SUCCEEDED from inside the agent container (the covered cgroup) and # FAILED from the host, same node, same netns. Full write-up: # decisions/2026-10-03-cilium-cgroup-hostroot.md. # # /sys/fs/cgroup IS a cgroup2 mount in the host's root cgroup namespace on every node (verified # with findmnt on all three), so bind-mounting it gives the agent the real host root. autoMount # stays off because the mount it would create already exists. cgroup: autoMount: enabled: false hostRoot: /sys/fs/cgroup # --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident --- # --- Roll the agents when their ConfigMap changes (2026-10-05) --- # [parity] MUST equal Ansible: rollOutCiliumPods: true # # WHY: a merged Cilium config change DID NOTHING. The chart default is `false` # (cilium 1.20.1 values.yaml: "Roll out cilium agent pods automatically when configmap is # updated. rollOutCiliumPods: false"), so Helm rewrites `cilium-config` and leaves the # DaemonSet's pod template untouched — and cilium-agent reads that ConfigMap only at STARTUP. # The result is a silently inert change: Flux green, HelmRelease `UpgradeSucceeded`, the new # keys visibly present in the ConfigMap, and the running agents still on the old settings # until a human remembers to restart them. # # MEASURED, and it is why this exists: the Hubble flow exporter merged 2026-10-04 wrote # `hubble-export-file-path: stdout` and its allowlist into `cilium-config`, the HelmRelease # reported `Helm upgrade succeeded ... cilium.v6`, and over a DAY later all three agents were # still the pre-merge pods (startTime 2026-10-04T07:11Z) with ZERO exported flow lines. The # feature was merged, believed delivered, and doing nothing at all. That is the same class of # failure this repo already documents elsewhere — a green signal over something that never ran. # # THE COST IS REAL AND IS THE POINT: every future Cilium config change now triggers a # cluster-wide agent rollout instead of waiting for a manual restart. That is strictly more # honest — the change applies when it is merged, or it fails visibly — but it means config # edits are disruptive by default, so they want a quiet moment. One node at a time: # `updateStrategy.rollingUpdate.maxUnavailable: 1` below means the DaemonSet will not proceed # past an unready agent, so a bad render costs one node rather than the cluster. Note that the # node holding the L2 lease for the Gateway, PiHole-DNS and Plex VIPs loses them briefly as it # cycles, so a rollout can blip LAN DNS. # # ENABLING THIS IS ITSELF ONE ROLLOUT: adding the ConfigMap checksum annotation changes the pod # template, so merging this rolls the agents once — which is what finally activates the Hubble # exporter that has been inert since 2026-10-04. rollOutCiliumPods: true # [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1. # The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate. # maxUnavailable: 2) — NOT a local choice, and on a 3-node cluster that lets a single bad # Helm render take down two of three agents simultaneously. That is exactly what happened # when a fails-open post-build substitution shipped k8sServiceHost: "": the agent's `config` # init container fell back to the in-cluster ClusterIP 10.43.0.1:443, which only the Cilium # agent itself can translate — and the agent translating it was the one being replaced. # `dial tcp 10.43.0.1:443: i/o timeout`, deadlocked, on 2 of 3 nodes at once. # A CNI rollout has no healthy-peer fallback: the thing being upgraded is the thing the # upgrade needs in order to reach the API server. With maxUnavailable: 1 the DaemonSet # will not proceed past an unready pod, so a bad render costs one node, not the cluster. # This matters most on a COLD PROD BOOTSTRAP, where every knob is new at once and there is # no already-healthy agent to recover from (see DECISIONS-LOG 2026-09-01 INCIDENT/RESEARCH). # Cost, recorded: Cilium upgrades serialise and get slower. On 3 nodes that is minutes, and # it buys a hard stop on the first bad node — the right trade here. Revisit if the cluster # grows enough that serialised CNI upgrades become an operational problem. updateStrategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 # NOTE (NON-parity): operator.replicas is deliberately omitted here. The Ansible dev render sets # replicas: 1 to save RAM, but defaults/main.yml declares it explicitly NOT a parity field # (Flux/prod may differ). Only the [parity] values above must match the bootstrap. # # TODO(WS-NET, Phase 2): default-deny baseline, CiliumL2AnnouncementPolicy + IP pool, # Egress Gateway policies, Hubble metrics export to kube-prometheus-stack.