home-network-k3s-impl/platform/cilium/helmrelease.yaml

---
# Cilium — CNI, kube-proxy replacement, Gateway API, Egress Gateway, Hubble (ADR-0002).
# Serves: R-NET-01..07, R-DNS-01, R-SEC-02. Owner: WS-NET (bootstrap shared with WS-INFRA).
#
# ============================ PARITY CONTRACT ============================
# Cilium's INITIAL install is performed by the Ansible k3s bootstrap (spec docs/02, Phase 1) —
# a cluster needs a CNI before Flux can run. This HelmRelease ADOPTS and manages Cilium's config
# thereafter (docs/05). The values below MUST EQUAL the Ansible bootstrap values, or Flux and the
# bootstrap will fight over the running CNI and break cluster networking.
#
# SOURCE OF TRUTH (do not diverge without updating both):
#   infra/ansible/roles/cilium/templates/cilium-values.yaml.j2
#   infra/ansible/roles/cilium/defaults/main.yml   (cilium_version)
#   infra/ansible/inventory/group_vars/all.yml     (k3s_apiserver_endpoint, k3s_api_port)
#
# Every value tagged [parity] below is asserted identical to the Ansible render. If you change one
# here, change it in the Ansible role in the same PR. (docs/02 Cilium-ownership contract.)
# ========================================================================
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: cilium
  namespace: kube-system
spec:
  interval: 30m
  chart:
    spec:
      chart: cilium
      # [parity] Pinned to the exact version the Ansible bootstrap installs
      # (cilium_version in roles/cilium/defaults/main.yml). Renovate bumps both together.
      version: "1.20.1"
      sourceRef:
        kind: HelmRepository
        name: cilium
        namespace: kube-system
  install:
    remediation:
      retries: 3
  upgrade:
    remediation:
      retries: 3
  # Values are inlined so the intent (Cilium REPLACES flannel + kube-proxy + Traefik) is visible at
  # the release. MUST equal the Ansible bootstrap render (see PARITY CONTRACT header).
  values:
    # --- kube-proxy replacement (ADR-0001/0002: k3s kube-proxy disabled) ---
    # [parity] MUST equal Ansible: kubeProxyReplacement: true
    kubeProxyReplacement: true
    # [parity] MUST equal Ansible: k8sServiceHost/k8sServicePort = the k3s API endpoint
    # (**k3s_apiserver_endpoint** / k3s_api_port, infra/ansible/inventory/group_vars/all.yml).
    # Setting these wrong breaks Cilium's connection to the API server on adopt.
    #
    # THIS IS THE CONTROL-PLANE VIP (kube-vip), NOT A NODE ADDRESS — operator decision 2026-09-06.
    # ${K3S_API_HOST} (cluster-vars, R-ENV-02): dev 192.168.1.49 | prod 192.168.2.9, each equal to
    # that environment's kube_vip_address. The knob's NAME still says "host"; read it as "the API
    # host Cilium dials", and see clusters/base/cluster-vars.yaml for why it was not renamed.
    #
    # WHY, in one line: this value used to be the PRIMARY NODE's IP, so on a 3-server HA cluster the
    # CNI's control-plane path depended on one machine. Verified on live prod 2026-09-06 — every
    # cilium-agent carried KUBERNETES_SERVICE_HOST=192.168.2.10 (k3s-prod-1) while kube-vip held .9.
    # Losing that node does not break the datapath (Cilium keeps serving from cache); it silently
    # stops every service and policy update from ever landing, which is worse to detect.
    #
    # [parity] IS NOW AGAINST k3s_apiserver_endpoint AND NOT k3s_api_host. Those were one Ansible
    # variable until 2026-09-06 and the conflation is the defect. k3s_api_host still exists and
    # still holds the FOUNDER's own address, because roles/k3s_server's secondary
    # `--server https://<host>:6443` join must target a host that is already up — on a COLD build
    # no VIP exists yet. So: founder for the join, VIP for the CNI. If you change this line, change
    # cilium-values.yaml.j2 in the same PR, and point it at the SAME variable.
    #
    # The `platform` Flux Kustomization substitutes it; the Ansible side is parameterized from
    # inventory (k3s_apiserver_endpoint derives from kube_vip_address), so the parity contract holds
    # per-environment, not per-literal.
    #
    # NEEDS NO CERT WORK: k3s_tls_sans has always carried kube_vip_address alongside every node IP,
    # so both addresses are already SANs on the serving cert (verified live 2026-09-06 — prod
    # answers through https://192.168.2.9:6443 under full CA verification, no insecure skip). This
    # switch reissues nothing and restarts no apiserver.
    k8sServiceHost: "${K3S_API_HOST}"
    k8sServicePort: 6443
    # --- Gateway API (ingress; replaces Traefik) — docs/09, ADR-0002 ---
    # [parity] MUST equal Ansible: gatewayAPI.enabled: true
    gatewayAPI:
      enabled: true
    # --- Egress Gateway (egress-class steering; part of the VPN model, ADR-0013) ---
    # [parity] MUST equal Ansible: egressGateway.enabled: true
    egressGateway:
      enabled: true
    # --- Masquerade (REQUIRED by Egress Gateway) — Phase-1 crash fix ---
    # [parity] MUST equal Ansible: enableIPv4Masquerade: true + bpf.masquerade: true.
    # This exact pairing (ipv4 masquerade via eBPF, the correct companion to kubeProxyReplacement)
    # was the fix for the Phase-1 agent crash. Enabling one without the other regresses it —
    # keep both, and keep them equal to the Ansible values.
    enableIPv4Masquerade: true
    bpf:
      masquerade: true
    # --- Hubble (flow observability / isolation audit) ---
    # [parity] MUST equal Ansible: hubble.{enabled,relay.enabled,ui.enabled}: true
    hubble:
      enabled: true
      relay:
        enabled: true
      ui:
        enabled: true   # exposure gated behind oauth2-proxy forward-auth later (ADR-0014)
      # --- STATIC FLOW EXPORT: persist the PRE-SNAT client address for Plex (2026-10-04) ---
      # [parity] MUST equal Ansible: hubble.export.static.{enabled,filePath,allowList}
      #
      # WHY THIS EXISTS. Plex's own log cannot show who connected. Every LoadBalancer Service in
      # this cluster is `externalTrafficPolicy: Cluster`, so the node holding the L2 lease for the
      # VIP SNATs the client before forwarding to the backend pod. Measured 2026-10-04: Plex had
      # logged 3,214 requests from `10.0.1.x` (the lease holder k3s-prod-3's own address, Cilium
      # identity 1 = reserved:host) and ZERO from any 192.168.x address, ever. No X-Forwarded-For
      # either -- it serves raw tcp/32400, not HTTP through the Gateway.
      #
      # THE ADDRESS IS NOT LOST, ONLY UNRECORDED. Cilium observes the flow at ingress, BEFORE its
      # own translation, so Hubble already has the true source. Measured on the lease holder:
      #     192.168.2.133 -> 10.0.0.135   (real LAN client -> the Plex pod)
      #     10.0.1.78     -> 10.0.0.135   (the same traffic, post-SNAT -- what Plex sees)
      # What was missing is RETENTION: the ring buffer was 4095/4095 flows at ~303 flows/s, i.e.
      # about THIRTEEN SECONDS of history, with hubble metrics disabled and no exporter configured.
      #
      # WHY NOT FIX THE SNAT INSTEAD. `externalTrafficPolicy: Local` would preserve the address in
      # Plex's own log, but cilium 1.20's L2 announcer is explicitly incompatible with it -- upstream
      # docs for this exact version: "incompatible with the externalTrafficPolicy: Local on services
      # as it may cause service IPs to be announced on nodes without pods causing traffic drops",
      # and "L2 announcements isn't currently aware of this setting". Live proof that this is not
      # theoretical: the VIP is announced by k3s-prod-3 while the pod runs on k3s-prod-1, so the
      # flip would black-hole tcp/32400 immediately. Making it safe would mean pinning BOTH the pod
      # and the announcement to one node and giving up Plex's ability to reschedule -- a real
      # availability loss to buy a log field. (It is the same trap platform/crafty/service-l4.yaml
      # documents from an 8-day Minecraft outage.) DSR was also rejected: routing-mode here is
      # vxlan, not native/geneve, and DSR returns bypass the ingress node, which breaks the SOHO
      # router's stateful NAT on the remote path.
      #
      # `filePath: stdout` RATHER THAN A FILE, deliberately. Grafana Alloy already runs as a
      # DaemonSet tailing every pod's logs into Loki (platform/monitoring/alloy.yaml), so stdout
      # reaches Loki with NO new hostPath mount, no file-rotation handling and no Alloy change.
      # Upstream documents it: "To get Hubble flows directly exported to the logs instead of written
      # to a rotated file, `stdout` can be defined as `hubble-export-file-path`." fileMaxSizeMb /
      # fileMaxBackups are therefore left at chart defaults and do not apply.
      #   Query:  {namespace="kube-system", app="cilium"} |= "32400"
      #
      # SCOPED BY PORT, NOT BY NAMESPACE -- and that is a measured decision, not a preference. A
      # `--to-namespace plex` query returned ONLY the post-SNAT flows; the flow carrying the real
      # client address did NOT match it, because at the point of observation the destination
      # identity is not yet resolved to the plex pod. Filtering by namespace would therefore drop
      # exactly the record this change exists to capture. Port it is.
      #
      # Both directions are listed so the return path is captured too. Volume is small (a few
      # hundred to a few thousand lines a day); if it ever becomes noisy, narrow it with
      # `fieldMask` rather than by removing a filter.
      export:
        static:
          enabled: true
          filePath: stdout
          allowList:
            - '{"destination_port":["32400"]}'
            - '{"source_port":["32400"]}'
    # --- L2 announcement for the ingress VIP on the flat LAN (docs/09, OQ-1 resolved) ---
    # [parity] MUST equal Ansible: l2announcements.enabled: true. IP pool +
    # CiliumL2AnnouncementPolicy (reserved 10.73.0.240-.250) authored by WS-NET in Phase 2.
    l2announcements:
      enabled: true
    # --- cgroup: point Cilium at the HOST's cgroup2 root, which systemd mounts at /sys/fs/cgroup ---
    # [parity] MUST equal Ansible cgroup block (roles/cilium/templates/cilium-values.yaml.j2).
    #
    # FIXED 2026-10-03 — hostRoot WAS /run/cilium/cgroupv2, AND THAT SILENTLY BROKE SOCKET-LB FOR
    # EVERY HOST PROCESS. With autoMount disabled, the chart only bind-mounts whatever already exists
    # at hostRoot ("it's expected that users have mounted cgroup2 filesystem at the specified
    # cgroup.hostRoot" — cilium 1.20.1 values.yaml). Nothing ever mounted cgroup2 at
    # /run/cilium/cgroupv2 on these nodes (it is an empty DirectoryOrCreate hostPath), so the agent
    # mounted one ITSELF from inside its own private cgroup namespace. That mount's root is the
    # agent container's OWN leaf cgroup, so cil_sock4_connect & co. were attached there and covered
    # only the agent. Every other host-netns process (the k3s apiserver in
    # /system.slice/k3s.service, curl over SSH) was untranslated: Service ClusterIPs timed out from
    # every node, and the node holding the Gateway's L2 lease could not reach the Gateway VIP at all
    # ("no route to host"). That is why k3s-prod-1's apiserver logged `oidc: authenticator not
    # initialized` from 2026-09-12 on — kubectl SSO through the kube-vip VIP failed outright.
    # Measured on k3s-prod-1 before this change: TCP connect to 192.168.2.20:443, 10.43.225.79:443
    # and 10.43.59.141:8080 SUCCEEDED from inside the agent container (the covered cgroup) and
    # FAILED from the host, same node, same netns. Full write-up:
    # decisions/2026-10-03-cilium-cgroup-hostroot.md.
    #
    # /sys/fs/cgroup IS a cgroup2 mount in the host's root cgroup namespace on every node (verified
    # with findmnt on all three), so bind-mounting it gives the agent the real host root. autoMount
    # stays off because the mount it would create already exists.
    cgroup:
      autoMount:
        enabled: false
      hostRoot: /sys/fs/cgroup
    # --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident ---
    # --- Roll the agents when their ConfigMap changes (2026-10-05) ---
    # [parity] MUST equal Ansible: rollOutCiliumPods: true
    #
    # WHY: a merged Cilium config change DID NOTHING. The chart default is `false`
    # (cilium 1.20.1 values.yaml: "Roll out cilium agent pods automatically when configmap is
    # updated. rollOutCiliumPods: false"), so Helm rewrites `cilium-config` and leaves the
    # DaemonSet's pod template untouched — and cilium-agent reads that ConfigMap only at STARTUP.
    # The result is a silently inert change: Flux green, HelmRelease `UpgradeSucceeded`, the new
    # keys visibly present in the ConfigMap, and the running agents still on the old settings
    # until a human remembers to restart them.
    #
    # MEASURED, and it is why this exists: the Hubble flow exporter merged 2026-10-04 wrote
    # `hubble-export-file-path: stdout` and its allowlist into `cilium-config`, the HelmRelease
    # reported `Helm upgrade succeeded ... cilium.v6`, and over a DAY later all three agents were
    # still the pre-merge pods (startTime 2026-10-04T07:11Z) with ZERO exported flow lines. The
    # feature was merged, believed delivered, and doing nothing at all. That is the same class of
    # failure this repo already documents elsewhere — a green signal over something that never ran.
    #
    # THE COST IS REAL AND IS THE POINT: every future Cilium config change now triggers a
    # cluster-wide agent rollout instead of waiting for a manual restart. That is strictly more
    # honest — the change applies when it is merged, or it fails visibly — but it means config
    # edits are disruptive by default, so they want a quiet moment. One node at a time:
    # `updateStrategy.rollingUpdate.maxUnavailable: 1` below means the DaemonSet will not proceed
    # past an unready agent, so a bad render costs one node rather than the cluster. Note that the
    # node holding the L2 lease for the Gateway, PiHole-DNS and Plex VIPs loses them briefly as it
    # cycles, so a rollout can blip LAN DNS.
    #
    # ENABLING THIS IS ITSELF ONE ROLLOUT: adding the ConfigMap checksum annotation changes the pod
    # template, so merging this rolls the agents once — which is what finally activates the Hubble
    # exporter that has been inert since 2026-10-04.
    rollOutCiliumPods: true
    # [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1.
    # The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate.
    # maxUnavailable: 2) — NOT a local choice, and on a 3-node cluster that lets a single bad
    # Helm render take down two of three agents simultaneously. That is exactly what happened
    # when a fails-open post-build substitution shipped k8sServiceHost: "": the agent's `config`
    # init container fell back to the in-cluster ClusterIP 10.43.0.1:443, which only the Cilium
    # agent itself can translate — and the agent translating it was the one being replaced.
    # `dial tcp 10.43.0.1:443: i/o timeout`, deadlocked, on 2 of 3 nodes at once.
    # A CNI rollout has no healthy-peer fallback: the thing being upgraded is the thing the
    # upgrade needs in order to reach the API server. With maxUnavailable: 1 the DaemonSet
    # will not proceed past an unready pod, so a bad render costs one node, not the cluster.
    # This matters most on a COLD PROD BOOTSTRAP, where every knob is new at once and there is
    # no already-healthy agent to recover from (see DECISIONS-LOG 2026-09-01 INCIDENT/RESEARCH).
    # Cost, recorded: Cilium upgrades serialise and get slower. On 3 nodes that is minutes, and
    # it buys a hard stop on the first bad node — the right trade here. Revisit if the cluster
    # grows enough that serialised CNI upgrades become an operational problem.
    updateStrategy:
      type: RollingUpdate
      rollingUpdate:
        maxUnavailable: 1
    # NOTE (NON-parity): operator.replicas is deliberately omitted here. The Ansible dev render sets
    # replicas: 1 to save RAM, but defaults/main.yml declares it explicitly NOT a parity field
    # (Flux/prod may differ). Only the [parity] values above must match the bootstrap.
    #
    # TODO(WS-NET, Phase 2): default-deny baseline, CiliumL2AnnouncementPolicy + IP pool,
    #   Egress Gateway policies, Hubble metrics export to kube-prometheus-stack.