jakman85andClaude Opus 5
fdc4a7d9f4
During the substitution incident a bad render (k8sServiceHost: "") took down TWO of three
Cilium agents simultaneously. The agents' `config` init container falls back to the API
ClusterIP, which only the Cilium agent itself can translate — and that agent is the one
being replaced — so it deadlocks on `dial tcp 10.43.0.1:443: i/o timeout`.
Root cause of the 2-at-once: `updateStrategy.rollingUpdate.maxUnavailable: 2` is the
UPSTREAM CHART DEFAULT, confirmed by downloading cilium-1.20.1.tgz and reading
values.yaml:437-444 — not a local choice, and not overridden anywhere in this repo.
maxUnavailable: 1 makes a rollout replace ONE agent at a time and stop at the first pod
that fails to become Ready. The whole value of the setting is that the rollout halts on
the first bad node instead of taking the majority of the dataplane with it, and that
guarantee only exists at 1. It matters far more on a cold PROD bootstrap, where there is
no healthy peer to recover from — dev only self-healed because k3s-dev-3 kept a working
agent throughout.
Set as a [parity] field in ALL THREE places, per this repo's documented contract that the
Ansible bootstrap values and the Flux HelmRelease values must agree on settings that
define cluster networking (Ansible installs Cilium at bootstrap; the HelmRelease adopts
it; the two layers fight otherwise):
platform/cilium/helmrelease.yaml (authoritative for Flux)
infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 (authoritative for bootstrap)
platform/cilium/values.yaml (reference mirror, kept in sync)
Verified by rendering the .j2 with roles/cilium/defaults and diffing full value sets: every
[parity] field is byte-equal across all three; only k8sServiceHost differs, as designed
(${K3S_API_HOST} vs {{ k3s_api_host }}).
MERGING THIS IS NON-DISRUPTIVE. updateStrategy lives at DaemonSet.spec.updateStrategy — a
SIBLING of spec.template (chart daemonset.yaml:49-53). The DaemonSet controller decides a
pod is out of date by hashing spec.template alone, so the pod-template hash does not move,
no new ControllerRevision is created, and zero pods are recreated. Helm patches the field
in place. (If it were disruptive it would be self-defeating: rolling agents under the old
maxUnavailable: 2 in order to install maxUnavailable: 1.) Verify after reconcile with
`kubectl -n kube-system get ds cilium -o jsonpath='{.spec.updateStrategy.rollingUpdate.maxUnavailable}'`
plus unchanged pod ages — that is the cleanest proof nothing rolled.
Trade-off recorded in the comments: upgrades serialise, roughly two waves to three at 3
nodes (a minute or two). On a large cluster this becomes painful and a percentage such as
"10%" would be right — noted so a future larger cluster revisits it deliberately.
SCOPE: this bounds blast radius, it does NOT close the substitution race. A bad render
still ships; it now breaks one node and stalls rather than taking the majority.
platform.dependsOn: [flux-system] (P0) is the fix for the cause.
Serves: R-PLAT-01, R-NET-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
119 lines
6.5 KiB
YAML
119 lines
6.5 KiB
YAML
---
|
|
# Cilium — CNI, kube-proxy replacement, Gateway API, Egress Gateway, Hubble (ADR-0002).
|
|
# Serves: R-NET-01..07, R-DNS-01, R-SEC-02. Owner: WS-NET (bootstrap shared with WS-INFRA).
|
|
#
|
|
# ============================ PARITY CONTRACT ============================
|
|
# Cilium's INITIAL install is performed by the Ansible k3s bootstrap (spec docs/02, Phase 1) —
|
|
# a cluster needs a CNI before Flux can run. This HelmRelease ADOPTS and manages Cilium's config
|
|
# thereafter (docs/05). The values below MUST EQUAL the Ansible bootstrap values, or Flux and the
|
|
# bootstrap will fight over the running CNI and break cluster networking.
|
|
#
|
|
# SOURCE OF TRUTH (do not diverge without updating both):
|
|
# infra/ansible/roles/cilium/templates/cilium-values.yaml.j2
|
|
# infra/ansible/roles/cilium/defaults/main.yml (cilium_version, k3s_api_host/port)
|
|
#
|
|
# Every value tagged [parity] below is asserted identical to the Ansible render. If you change one
|
|
# here, change it in the Ansible role in the same PR. (docs/02 Cilium-ownership contract.)
|
|
# ========================================================================
|
|
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
kind: HelmRelease
|
|
metadata:
|
|
name: cilium
|
|
namespace: kube-system
|
|
spec:
|
|
interval: 30m
|
|
chart:
|
|
spec:
|
|
chart: cilium
|
|
# [parity] Pinned to the exact version the Ansible bootstrap installs
|
|
# (cilium_version in roles/cilium/defaults/main.yml). Renovate bumps both together.
|
|
version: "1.20.1"
|
|
sourceRef:
|
|
kind: HelmRepository
|
|
name: cilium
|
|
namespace: kube-system
|
|
install:
|
|
remediation:
|
|
retries: 3
|
|
upgrade:
|
|
remediation:
|
|
retries: 3
|
|
# Values are inlined so the intent (Cilium REPLACES flannel + kube-proxy + Traefik) is visible at
|
|
# the release. MUST equal the Ansible bootstrap render (see PARITY CONTRACT header).
|
|
values:
|
|
# --- kube-proxy replacement (ADR-0001/0002: k3s kube-proxy disabled) ---
|
|
# [parity] MUST equal Ansible: kubeProxyReplacement: true
|
|
kubeProxyReplacement: true
|
|
# [parity] MUST equal Ansible: k8sServiceHost/k8sServicePort = the k3s API endpoint
|
|
# (k3s_api_host/k3s_api_port). This is the primary control-plane node IP.
|
|
# Setting these wrong breaks Cilium's connection to the API server on adopt.
|
|
# ${K3S_API_HOST} (cluster-vars, R-ENV-02): dev 192.168.1.50 | prod set in clusters/prod.
|
|
# The `platform` Flux Kustomization substitutes it; the Ansible side is already parameterized
|
|
# from inventory (k3s_api_host), so the parity contract holds per-environment, not per-literal.
|
|
k8sServiceHost: "${K3S_API_HOST}"
|
|
k8sServicePort: 6443
|
|
# --- Gateway API (ingress; replaces Traefik) — docs/09, ADR-0002 ---
|
|
# [parity] MUST equal Ansible: gatewayAPI.enabled: true
|
|
gatewayAPI:
|
|
enabled: true
|
|
# --- Egress Gateway (egress-class steering; part of the VPN model, ADR-0013) ---
|
|
# [parity] MUST equal Ansible: egressGateway.enabled: true
|
|
egressGateway:
|
|
enabled: true
|
|
# --- Masquerade (REQUIRED by Egress Gateway) — Phase-1 crash fix ---
|
|
# [parity] MUST equal Ansible: enableIPv4Masquerade: true + bpf.masquerade: true.
|
|
# This exact pairing (ipv4 masquerade via eBPF, the correct companion to kubeProxyReplacement)
|
|
# was the fix for the Phase-1 agent crash. Enabling one without the other regresses it —
|
|
# keep both, and keep them equal to the Ansible values.
|
|
enableIPv4Masquerade: true
|
|
bpf:
|
|
masquerade: true
|
|
# --- Hubble (flow observability / isolation audit) ---
|
|
# [parity] MUST equal Ansible: hubble.{enabled,relay.enabled,ui.enabled}: true
|
|
hubble:
|
|
enabled: true
|
|
relay:
|
|
enabled: true
|
|
ui:
|
|
enabled: true # exposure gated behind oauth2-proxy forward-auth later (ADR-0014)
|
|
# --- L2 announcement for the ingress VIP on the flat LAN (docs/09, OQ-1 resolved) ---
|
|
# [parity] MUST equal Ansible: l2announcements.enabled: true. IP pool +
|
|
# CiliumL2AnnouncementPolicy (reserved 10.73.0.240-.250) authored by WS-NET in Phase 2.
|
|
l2announcements:
|
|
enabled: true
|
|
# --- cgroup: k3s ships its own cgroup mount; point Cilium at it, don't auto-mount ---
|
|
# [parity] MUST equal Ansible cgroup block. Not in the deliverable's headline list, but it IS
|
|
# part of the working CNI config on k3s: reconciling without it lets Cilium try to auto-mount
|
|
# cgroupv2 and break on adopt. Kept in parity for a safe Phase-2 adoption.
|
|
cgroup:
|
|
autoMount:
|
|
enabled: false
|
|
hostRoot: /run/cilium/cgroupv2
|
|
# --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident ---
|
|
# [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1.
|
|
# The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate.
|
|
# maxUnavailable: 2) — NOT a local choice, and on a 3-node cluster that lets a single bad
|
|
# Helm render take down two of three agents simultaneously. That is exactly what happened
|
|
# when a fails-open post-build substitution shipped k8sServiceHost: "": the agent's `config`
|
|
# init container fell back to the in-cluster ClusterIP 10.43.0.1:443, which only the Cilium
|
|
# agent itself can translate — and the agent translating it was the one being replaced.
|
|
# `dial tcp 10.43.0.1:443: i/o timeout`, deadlocked, on 2 of 3 nodes at once.
|
|
# A CNI rollout has no healthy-peer fallback: the thing being upgraded is the thing the
|
|
# upgrade needs in order to reach the API server. With maxUnavailable: 1 the DaemonSet
|
|
# will not proceed past an unready pod, so a bad render costs one node, not the cluster.
|
|
# This matters most on a COLD PROD BOOTSTRAP, where every knob is new at once and there is
|
|
# no already-healthy agent to recover from (see DECISIONS-LOG 2026-09-01 INCIDENT/RESEARCH).
|
|
# Cost, recorded: Cilium upgrades serialise and get slower. On 3 nodes that is minutes, and
|
|
# it buys a hard stop on the first bad node — the right trade here. Revisit if the cluster
|
|
# grows enough that serialised CNI upgrades become an operational problem.
|
|
updateStrategy:
|
|
type: RollingUpdate
|
|
rollingUpdate:
|
|
maxUnavailable: 1
|
|
# NOTE (NON-parity): operator.replicas is deliberately omitted here. The Ansible dev render sets
|
|
# replicas: 1 to save RAM, but defaults/main.yml declares it explicitly NOT a parity field
|
|
# (Flux/prod may differ). Only the [parity] values above must match the bootstrap.
|
|
#
|
|
# TODO(WS-NET, Phase 2): default-deny baseline, CiliumL2AnnouncementPolicy + IP pool,
|
|
# Egress Gateway policies, Hubble metrics export to kube-prometheus-stack.
|