home-network-k3s-impl/platform/cilium/helmrelease.yaml

---
# Cilium — CNI, kube-proxy replacement, Gateway API, Egress Gateway, Hubble (ADR-0002).
# Serves: R-NET-01..07, R-DNS-01, R-SEC-02. Owner: WS-NET (bootstrap shared with WS-INFRA).
#
# ============================ PARITY CONTRACT ============================
# Cilium's INITIAL install is performed by the Ansible k3s bootstrap (spec docs/02, Phase 1) —
# a cluster needs a CNI before Flux can run. This HelmRelease ADOPTS and manages Cilium's config
# thereafter (docs/05). The values below MUST EQUAL the Ansible bootstrap values, or Flux and the
# bootstrap will fight over the running CNI and break cluster networking.
#
# SOURCE OF TRUTH (do not diverge without updating both):
#   infra/ansible/roles/cilium/templates/cilium-values.yaml.j2
#   infra/ansible/roles/cilium/defaults/main.yml   (cilium_version)
#   infra/ansible/inventory/group_vars/all.yml     (k3s_apiserver_endpoint, k3s_api_port)
#
# Every value tagged [parity] below is asserted identical to the Ansible render. If you change one
# here, change it in the Ansible role in the same PR. (docs/02 Cilium-ownership contract.)
# ========================================================================
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: cilium
  namespace: kube-system
spec:
  interval: 30m
  chart:
    spec:
      chart: cilium
      # [parity] Pinned to the exact version the Ansible bootstrap installs
      # (cilium_version in roles/cilium/defaults/main.yml). Renovate bumps both together.
      version: "1.20.1"
      sourceRef:
        kind: HelmRepository
        name: cilium
        namespace: kube-system
  install:
    remediation:
      retries: 3
  upgrade:
    remediation:
      retries: 3
  # Values are inlined so the intent (Cilium REPLACES flannel + kube-proxy + Traefik) is visible at
  # the release. MUST equal the Ansible bootstrap render (see PARITY CONTRACT header).
  values:
    # --- kube-proxy replacement (ADR-0001/0002: k3s kube-proxy disabled) ---
    # [parity] MUST equal Ansible: kubeProxyReplacement: true
    kubeProxyReplacement: true
    # [parity] MUST equal Ansible: k8sServiceHost/k8sServicePort = the k3s API endpoint
    # (**k3s_apiserver_endpoint** / k3s_api_port, infra/ansible/inventory/group_vars/all.yml).
    # Setting these wrong breaks Cilium's connection to the API server on adopt.
    #
    # THIS IS THE CONTROL-PLANE VIP (kube-vip), NOT A NODE ADDRESS — operator decision 2026-09-06.
    # ${K3S_API_HOST} (cluster-vars, R-ENV-02): dev 192.168.1.49 | prod 192.168.2.9, each equal to
    # that environment's kube_vip_address. The knob's NAME still says "host"; read it as "the API
    # host Cilium dials", and see clusters/base/cluster-vars.yaml for why it was not renamed.
    #
    # WHY, in one line: this value used to be the PRIMARY NODE's IP, so on a 3-server HA cluster the
    # CNI's control-plane path depended on one machine. Verified on live prod 2026-09-06 — every
    # cilium-agent carried KUBERNETES_SERVICE_HOST=192.168.2.10 (k3s-prod-1) while kube-vip held .9.
    # Losing that node does not break the datapath (Cilium keeps serving from cache); it silently
    # stops every service and policy update from ever landing, which is worse to detect.
    #
    # [parity] IS NOW AGAINST k3s_apiserver_endpoint AND NOT k3s_api_host. Those were one Ansible
    # variable until 2026-09-06 and the conflation is the defect. k3s_api_host still exists and
    # still holds the FOUNDER's own address, because roles/k3s_server's secondary
    # `--server https://<host>:6443` join must target a host that is already up — on a COLD build
    # no VIP exists yet. So: founder for the join, VIP for the CNI. If you change this line, change
    # cilium-values.yaml.j2 in the same PR, and point it at the SAME variable.
    #
    # The `platform` Flux Kustomization substitutes it; the Ansible side is parameterized from
    # inventory (k3s_apiserver_endpoint derives from kube_vip_address), so the parity contract holds
    # per-environment, not per-literal.
    #
    # NEEDS NO CERT WORK: k3s_tls_sans has always carried kube_vip_address alongside every node IP,
    # so both addresses are already SANs on the serving cert (verified live 2026-09-06 — prod
    # answers through https://192.168.2.9:6443 under full CA verification, no insecure skip). This
    # switch reissues nothing and restarts no apiserver.
    k8sServiceHost: "${K3S_API_HOST}"
    k8sServicePort: 6443
    # --- Gateway API (ingress; replaces Traefik) — docs/09, ADR-0002 ---
    # [parity] MUST equal Ansible: gatewayAPI.enabled: true
    gatewayAPI:
      enabled: true
    # --- Egress Gateway (egress-class steering; part of the VPN model, ADR-0013) ---
    # [parity] MUST equal Ansible: egressGateway.enabled: true
    egressGateway:
      enabled: true
    # --- Masquerade (REQUIRED by Egress Gateway) — Phase-1 crash fix ---
    # [parity] MUST equal Ansible: enableIPv4Masquerade: true + bpf.masquerade: true.
    # This exact pairing (ipv4 masquerade via eBPF, the correct companion to kubeProxyReplacement)
    # was the fix for the Phase-1 agent crash. Enabling one without the other regresses it —
    # keep both, and keep them equal to the Ansible values.
    enableIPv4Masquerade: true
    bpf:
      masquerade: true
    # --- Hubble (flow observability / isolation audit) ---
    # [parity] MUST equal Ansible: hubble.{enabled,relay.enabled,ui.enabled}: true
    hubble:
      enabled: true
      relay:
        enabled: true
      ui:
        enabled: true   # exposure gated behind oauth2-proxy forward-auth later (ADR-0014)
    # --- L2 announcement for the ingress VIP on the flat LAN (docs/09, OQ-1 resolved) ---
    # [parity] MUST equal Ansible: l2announcements.enabled: true. IP pool +
    # CiliumL2AnnouncementPolicy (reserved 10.73.0.240-.250) authored by WS-NET in Phase 2.
    l2announcements:
      enabled: true
    # --- cgroup: point Cilium at the HOST's cgroup2 root, which systemd mounts at /sys/fs/cgroup ---
    # [parity] MUST equal Ansible cgroup block (roles/cilium/templates/cilium-values.yaml.j2).
    #
    # FIXED 2026-10-03 — hostRoot WAS /run/cilium/cgroupv2, AND THAT SILENTLY BROKE SOCKET-LB FOR
    # EVERY HOST PROCESS. With autoMount disabled, the chart only bind-mounts whatever already exists
    # at hostRoot ("it's expected that users have mounted cgroup2 filesystem at the specified
    # cgroup.hostRoot" — cilium 1.20.1 values.yaml). Nothing ever mounted cgroup2 at
    # /run/cilium/cgroupv2 on these nodes (it is an empty DirectoryOrCreate hostPath), so the agent
    # mounted one ITSELF from inside its own private cgroup namespace. That mount's root is the
    # agent container's OWN leaf cgroup, so cil_sock4_connect & co. were attached there and covered
    # only the agent. Every other host-netns process (the k3s apiserver in
    # /system.slice/k3s.service, curl over SSH) was untranslated: Service ClusterIPs timed out from
    # every node, and the node holding the Gateway's L2 lease could not reach the Gateway VIP at all
    # ("no route to host"). That is why k3s-prod-1's apiserver logged `oidc: authenticator not
    # initialized` from 2026-09-12 on — kubectl SSO through the kube-vip VIP failed outright.
    # Measured on k3s-prod-1 before this change: TCP connect to 192.168.2.20:443, 10.43.225.79:443
    # and 10.43.59.141:8080 SUCCEEDED from inside the agent container (the covered cgroup) and
    # FAILED from the host, same node, same netns. Full write-up:
    # decisions/2026-10-03-cilium-cgroup-hostroot.md.
    #
    # /sys/fs/cgroup IS a cgroup2 mount in the host's root cgroup namespace on every node (verified
    # with findmnt on all three), so bind-mounting it gives the agent the real host root. autoMount
    # stays off because the mount it would create already exists.
    cgroup:
      autoMount:
        enabled: false
      hostRoot: /sys/fs/cgroup
    # --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident ---
    # [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1.
    # The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate.
    # maxUnavailable: 2) — NOT a local choice, and on a 3-node cluster that lets a single bad
    # Helm render take down two of three agents simultaneously. That is exactly what happened
    # when a fails-open post-build substitution shipped k8sServiceHost: "": the agent's `config`
    # init container fell back to the in-cluster ClusterIP 10.43.0.1:443, which only the Cilium
    # agent itself can translate — and the agent translating it was the one being replaced.
    # `dial tcp 10.43.0.1:443: i/o timeout`, deadlocked, on 2 of 3 nodes at once.
    # A CNI rollout has no healthy-peer fallback: the thing being upgraded is the thing the
    # upgrade needs in order to reach the API server. With maxUnavailable: 1 the DaemonSet
    # will not proceed past an unready pod, so a bad render costs one node, not the cluster.
    # This matters most on a COLD PROD BOOTSTRAP, where every knob is new at once and there is
    # no already-healthy agent to recover from (see DECISIONS-LOG 2026-09-01 INCIDENT/RESEARCH).
    # Cost, recorded: Cilium upgrades serialise and get slower. On 3 nodes that is minutes, and
    # it buys a hard stop on the first bad node — the right trade here. Revisit if the cluster
    # grows enough that serialised CNI upgrades become an operational problem.
    updateStrategy:
      type: RollingUpdate
      rollingUpdate:
        maxUnavailable: 1
    # NOTE (NON-parity): operator.replicas is deliberately omitted here. The Ansible dev render sets
    # replicas: 1 to save RAM, but defaults/main.yml declares it explicitly NOT a parity field
    # (Flux/prod may differ). Only the [parity] values above must match the bootstrap.
    #
    # TODO(WS-NET, Phase 2): default-deny baseline, CiliumL2AnnouncementPolicy + IP pool,
    #   Egress Gateway policies, Hubble metrics export to kube-prometheus-stack.