jakman85andClaude Opus 5
fdc4a7d9f4
During the substitution incident a bad render (k8sServiceHost: "") took down TWO of three
Cilium agents simultaneously. The agents' `config` init container falls back to the API
ClusterIP, which only the Cilium agent itself can translate — and that agent is the one
being replaced — so it deadlocks on `dial tcp 10.43.0.1:443: i/o timeout`.
Root cause of the 2-at-once: `updateStrategy.rollingUpdate.maxUnavailable: 2` is the
UPSTREAM CHART DEFAULT, confirmed by downloading cilium-1.20.1.tgz and reading
values.yaml:437-444 — not a local choice, and not overridden anywhere in this repo.
maxUnavailable: 1 makes a rollout replace ONE agent at a time and stop at the first pod
that fails to become Ready. The whole value of the setting is that the rollout halts on
the first bad node instead of taking the majority of the dataplane with it, and that
guarantee only exists at 1. It matters far more on a cold PROD bootstrap, where there is
no healthy peer to recover from — dev only self-healed because k3s-dev-3 kept a working
agent throughout.
Set as a [parity] field in ALL THREE places, per this repo's documented contract that the
Ansible bootstrap values and the Flux HelmRelease values must agree on settings that
define cluster networking (Ansible installs Cilium at bootstrap; the HelmRelease adopts
it; the two layers fight otherwise):
platform/cilium/helmrelease.yaml (authoritative for Flux)
infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 (authoritative for bootstrap)
platform/cilium/values.yaml (reference mirror, kept in sync)
Verified by rendering the .j2 with roles/cilium/defaults and diffing full value sets: every
[parity] field is byte-equal across all three; only k8sServiceHost differs, as designed
(${K3S_API_HOST} vs {{ k3s_api_host }}).
MERGING THIS IS NON-DISRUPTIVE. updateStrategy lives at DaemonSet.spec.updateStrategy — a
SIBLING of spec.template (chart daemonset.yaml:49-53). The DaemonSet controller decides a
pod is out of date by hashing spec.template alone, so the pod-template hash does not move,
no new ControllerRevision is created, and zero pods are recreated. Helm patches the field
in place. (If it were disruptive it would be self-defeating: rolling agents under the old
maxUnavailable: 2 in order to install maxUnavailable: 1.) Verify after reconcile with
`kubectl -n kube-system get ds cilium -o jsonpath='{.spec.updateStrategy.rollingUpdate.maxUnavailable}'`
plus unchanged pod ages — that is the cleanest proof nothing rolled.
Trade-off recorded in the comments: upgrades serialise, roughly two waves to three at 3
nodes (a minute or two). On a large cluster this becomes painful and a percentage such as
"10%" would be right — noted so a future larger cluster revisits it deliberately.
SCOPE: this bounds blast radius, it does NOT close the substitution race. A bad render
still ships; it now breaks one node and stalls rather than taking the majority.
platform.dependsOn: [flux-system] (P0) is the fix for the cause.
Serves: R-PLAT-01, R-NET-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
60 lines
2.6 KiB
YAML
60 lines
2.6 KiB
YAML
# Cilium Helm values — REFERENCE COPY of the parity set (docs/02 Cilium-ownership contract).
|
|
#
|
|
# The AUTHORITATIVE values that Flux reconciles are inlined in helmrelease.yaml. This file is a
|
|
# readable, un-nested mirror of that value set so WS-NET can diff it against the live cluster and
|
|
# the Ansible bootstrap render at a glance. Keep the two in sync.
|
|
#
|
|
# SOURCE OF TRUTH (all [parity] fields MUST equal these):
|
|
# infra/ansible/roles/cilium/templates/cilium-values.yaml.j2
|
|
# infra/ansible/roles/cilium/defaults/main.yml (cilium_version: 1.20.1, k3s_api_host/port)
|
|
#
|
|
# chart version (helmrelease.yaml chart.spec.version): 1.20.1 # [parity] = cilium_version
|
|
|
|
# [parity] k3s kube-proxy disabled → Cilium replaces it (ADR-0001/0002)
|
|
kubeProxyReplacement: true
|
|
# [parity] = the k3s API endpoint (primary control-plane node IP) + port.
|
|
# Mirrors the HelmRelease's ${K3S_API_HOST} cluster-vars token (R-ENV-02). This file is NOT a
|
|
# kustomize resource, so nothing substitutes it — the token is shown with its dev value beside it.
|
|
k8sServiceHost: "${K3S_API_HOST}" # dev: 192.168.1.50
|
|
k8sServicePort: 6443
|
|
# [parity] ingress via Gateway API, no Traefik (docs/09)
|
|
gatewayAPI:
|
|
enabled: true
|
|
# [parity] egress-class steering (ADR-0002; VPN model ADR-0013)
|
|
egressGateway:
|
|
enabled: true
|
|
# [parity] Egress Gateway REQUIRES masquerade; ipv4 + eBPF is the Phase-1 crash fix — keep BOTH
|
|
enableIPv4Masquerade: true
|
|
bpf:
|
|
masquerade: true
|
|
# [parity] flow observability / isolation audit
|
|
hubble:
|
|
enabled: true
|
|
relay:
|
|
enabled: true
|
|
ui:
|
|
enabled: true
|
|
# [parity] ingress VIP announcement on the flat LAN (docs/09, OQ-1)
|
|
l2announcements:
|
|
enabled: true
|
|
# [parity] k3s provides the cgroupv2 mount; do not auto-mount
|
|
cgroup:
|
|
autoMount:
|
|
enabled: false
|
|
hostRoot: /run/cilium/cgroupv2
|
|
# [parity] roll the agent DaemonSet ONE node at a time. Overrides the cilium 1.20.1 chart
|
|
# default of 2, which on 3 nodes lets a single bad render break 2/3 agents at once — as the
|
|
# 2026-09-01 incident did (k8sServiceHost: "" → the `config` init container fell back to the
|
|
# ClusterIP 10.43.0.1:443, translatable only by the very agent being replaced). A CNI upgrade
|
|
# has no healthy-peer fallback, least of all on a cold prod bootstrap; maxUnavailable: 1 stops
|
|
# the rollout at the first unready pod. Cost: serialised, slower upgrades (fine at 3 nodes).
|
|
updateStrategy:
|
|
type: RollingUpdate
|
|
rollingUpdate:
|
|
maxUnavailable: 1
|
|
# NON-parity (Flux/prod may differ; dev sets 1 to save RAM) — omitted from the HelmRelease:
|
|
# operator.replicas
|
|
#
|
|
# TODO (Phase 2, WS-NET): ipam / ipv4NativeRoutingCIDR pinned to cluster CIDRs; east-west
|
|
# encryption (wireguard/ipsec) decision (WS-SEC/WS-NET).
|