jakman85andClaude Opus 5
d1a6628a97
OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach
the cluster and provide fault tolerant HA. Without a change it will not work in HA."
THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live
before changing anything: all three prod cilium-agent pods carried
KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip
VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the
CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said
the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated
that stale comment as an open question when the evidence already settled it.
SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs:
k3s_api_host unchanged — the FOUNDER's IP: the cold-bootstrap join
target and a tls-san. infra/tofu/outputs.tf GENERATES this
name, so renaming would ripple into tofu and the runbooks.
k3s_apiserver_endpoint NEW = {{ kube_vip_address }} — the HA address; what Cilium
consumes. Derived, not restated, so an environment overrides
ONE line and the VIP, the SAN and the CNI endpoint move
together. Prod needed no new override.
THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at
the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran
node-baseline -> k3s -> cilium, so Cilium would install against an address nothing
announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value:
site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier
(kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there
because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing
neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so
a cold run stops THERE with a named cause instead of surfacing as unreachable agents.
Also fixed a pre-existing bug that only mattered once this play joined site.yml: the
role's read-only interface probe ran without `check_mode: false`, so under --check it
was skipped and the assert fired with a misleading "set kube_vip_interface explicitly".
VERIFIED BY ME, not just reported:
renders dev K3S_API_HOST=192.168.1.49 prod=192.168.2.9 (each environment's VIP)
cilium template consumes k3s_apiserver_endpoint, not k3s_api_host
cold join still `--server https://{{ k3s_api_host }}:6443` — the founder
order site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux
gate full CI gate PASS
Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing
k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders
['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved
— .9 already answers under full CA verification; and --check --diff of kube-vip.yml
against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the
kube-vip prod already runs.
ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine
checks: the Cilium template must use the endpoint var; the endpoint must derive from
kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal
that environment's VIP and be none of its node IPs (the [parity] contract, now
machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium.
Every new assertion mutation-tested and bites. One honest negative recorded: reverting
only the BASE default is not caught, because both overlays patch the key — the check
reads effective per-environment values, so the base default is unguarded.
APPLY ORDER, and this is the operator's step — nothing was applied:
PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let
Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet
ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01
incident), it will not proceed past an unready pod, existing datapath survives an
agent restart, and .9 is already live before the change lands.
DEV SEPARATELY, operator present. Same mechanics but live dev agents dial
192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that.
ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a
cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook.
NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live
KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that
test's gap list.
Refs R-PLAT-01, R-NET-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
144 lines
8.3 KiB
YAML
144 lines
8.3 KiB
YAML
---
|
|
# Cilium — CNI, kube-proxy replacement, Gateway API, Egress Gateway, Hubble (ADR-0002).
|
|
# Serves: R-NET-01..07, R-DNS-01, R-SEC-02. Owner: WS-NET (bootstrap shared with WS-INFRA).
|
|
#
|
|
# ============================ PARITY CONTRACT ============================
|
|
# Cilium's INITIAL install is performed by the Ansible k3s bootstrap (spec docs/02, Phase 1) —
|
|
# a cluster needs a CNI before Flux can run. This HelmRelease ADOPTS and manages Cilium's config
|
|
# thereafter (docs/05). The values below MUST EQUAL the Ansible bootstrap values, or Flux and the
|
|
# bootstrap will fight over the running CNI and break cluster networking.
|
|
#
|
|
# SOURCE OF TRUTH (do not diverge without updating both):
|
|
# infra/ansible/roles/cilium/templates/cilium-values.yaml.j2
|
|
# infra/ansible/roles/cilium/defaults/main.yml (cilium_version)
|
|
# infra/ansible/inventory/group_vars/all.yml (k3s_apiserver_endpoint, k3s_api_port)
|
|
#
|
|
# Every value tagged [parity] below is asserted identical to the Ansible render. If you change one
|
|
# here, change it in the Ansible role in the same PR. (docs/02 Cilium-ownership contract.)
|
|
# ========================================================================
|
|
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
kind: HelmRelease
|
|
metadata:
|
|
name: cilium
|
|
namespace: kube-system
|
|
spec:
|
|
interval: 30m
|
|
chart:
|
|
spec:
|
|
chart: cilium
|
|
# [parity] Pinned to the exact version the Ansible bootstrap installs
|
|
# (cilium_version in roles/cilium/defaults/main.yml). Renovate bumps both together.
|
|
version: "1.20.1"
|
|
sourceRef:
|
|
kind: HelmRepository
|
|
name: cilium
|
|
namespace: kube-system
|
|
install:
|
|
remediation:
|
|
retries: 3
|
|
upgrade:
|
|
remediation:
|
|
retries: 3
|
|
# Values are inlined so the intent (Cilium REPLACES flannel + kube-proxy + Traefik) is visible at
|
|
# the release. MUST equal the Ansible bootstrap render (see PARITY CONTRACT header).
|
|
values:
|
|
# --- kube-proxy replacement (ADR-0001/0002: k3s kube-proxy disabled) ---
|
|
# [parity] MUST equal Ansible: kubeProxyReplacement: true
|
|
kubeProxyReplacement: true
|
|
# [parity] MUST equal Ansible: k8sServiceHost/k8sServicePort = the k3s API endpoint
|
|
# (**k3s_apiserver_endpoint** / k3s_api_port, infra/ansible/inventory/group_vars/all.yml).
|
|
# Setting these wrong breaks Cilium's connection to the API server on adopt.
|
|
#
|
|
# THIS IS THE CONTROL-PLANE VIP (kube-vip), NOT A NODE ADDRESS — operator decision 2026-09-06.
|
|
# ${K3S_API_HOST} (cluster-vars, R-ENV-02): dev 192.168.1.49 | prod 192.168.2.9, each equal to
|
|
# that environment's kube_vip_address. The knob's NAME still says "host"; read it as "the API
|
|
# host Cilium dials", and see clusters/base/cluster-vars.yaml for why it was not renamed.
|
|
#
|
|
# WHY, in one line: this value used to be the PRIMARY NODE's IP, so on a 3-server HA cluster the
|
|
# CNI's control-plane path depended on one machine. Verified on live prod 2026-09-06 — every
|
|
# cilium-agent carried KUBERNETES_SERVICE_HOST=192.168.2.10 (k3s-prod-1) while kube-vip held .9.
|
|
# Losing that node does not break the datapath (Cilium keeps serving from cache); it silently
|
|
# stops every service and policy update from ever landing, which is worse to detect.
|
|
#
|
|
# [parity] IS NOW AGAINST k3s_apiserver_endpoint AND NOT k3s_api_host. Those were one Ansible
|
|
# variable until 2026-09-06 and the conflation is the defect. k3s_api_host still exists and
|
|
# still holds the FOUNDER's own address, because roles/k3s_server's secondary
|
|
# `--server https://<host>:6443` join must target a host that is already up — on a COLD build
|
|
# no VIP exists yet. So: founder for the join, VIP for the CNI. If you change this line, change
|
|
# cilium-values.yaml.j2 in the same PR, and point it at the SAME variable.
|
|
#
|
|
# The `platform` Flux Kustomization substitutes it; the Ansible side is parameterized from
|
|
# inventory (k3s_apiserver_endpoint derives from kube_vip_address), so the parity contract holds
|
|
# per-environment, not per-literal.
|
|
#
|
|
# NEEDS NO CERT WORK: k3s_tls_sans has always carried kube_vip_address alongside every node IP,
|
|
# so both addresses are already SANs on the serving cert (verified live 2026-09-06 — prod
|
|
# answers through https://192.168.2.9:6443 under full CA verification, no insecure skip). This
|
|
# switch reissues nothing and restarts no apiserver.
|
|
k8sServiceHost: "${K3S_API_HOST}"
|
|
k8sServicePort: 6443
|
|
# --- Gateway API (ingress; replaces Traefik) — docs/09, ADR-0002 ---
|
|
# [parity] MUST equal Ansible: gatewayAPI.enabled: true
|
|
gatewayAPI:
|
|
enabled: true
|
|
# --- Egress Gateway (egress-class steering; part of the VPN model, ADR-0013) ---
|
|
# [parity] MUST equal Ansible: egressGateway.enabled: true
|
|
egressGateway:
|
|
enabled: true
|
|
# --- Masquerade (REQUIRED by Egress Gateway) — Phase-1 crash fix ---
|
|
# [parity] MUST equal Ansible: enableIPv4Masquerade: true + bpf.masquerade: true.
|
|
# This exact pairing (ipv4 masquerade via eBPF, the correct companion to kubeProxyReplacement)
|
|
# was the fix for the Phase-1 agent crash. Enabling one without the other regresses it —
|
|
# keep both, and keep them equal to the Ansible values.
|
|
enableIPv4Masquerade: true
|
|
bpf:
|
|
masquerade: true
|
|
# --- Hubble (flow observability / isolation audit) ---
|
|
# [parity] MUST equal Ansible: hubble.{enabled,relay.enabled,ui.enabled}: true
|
|
hubble:
|
|
enabled: true
|
|
relay:
|
|
enabled: true
|
|
ui:
|
|
enabled: true # exposure gated behind oauth2-proxy forward-auth later (ADR-0014)
|
|
# --- L2 announcement for the ingress VIP on the flat LAN (docs/09, OQ-1 resolved) ---
|
|
# [parity] MUST equal Ansible: l2announcements.enabled: true. IP pool +
|
|
# CiliumL2AnnouncementPolicy (reserved 10.73.0.240-.250) authored by WS-NET in Phase 2.
|
|
l2announcements:
|
|
enabled: true
|
|
# --- cgroup: k3s ships its own cgroup mount; point Cilium at it, don't auto-mount ---
|
|
# [parity] MUST equal Ansible cgroup block. Not in the deliverable's headline list, but it IS
|
|
# part of the working CNI config on k3s: reconciling without it lets Cilium try to auto-mount
|
|
# cgroupv2 and break on adopt. Kept in parity for a safe Phase-2 adoption.
|
|
cgroup:
|
|
autoMount:
|
|
enabled: false
|
|
hostRoot: /run/cilium/cgroupv2
|
|
# --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident ---
|
|
# [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1.
|
|
# The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate.
|
|
# maxUnavailable: 2) — NOT a local choice, and on a 3-node cluster that lets a single bad
|
|
# Helm render take down two of three agents simultaneously. That is exactly what happened
|
|
# when a fails-open post-build substitution shipped k8sServiceHost: "": the agent's `config`
|
|
# init container fell back to the in-cluster ClusterIP 10.43.0.1:443, which only the Cilium
|
|
# agent itself can translate — and the agent translating it was the one being replaced.
|
|
# `dial tcp 10.43.0.1:443: i/o timeout`, deadlocked, on 2 of 3 nodes at once.
|
|
# A CNI rollout has no healthy-peer fallback: the thing being upgraded is the thing the
|
|
# upgrade needs in order to reach the API server. With maxUnavailable: 1 the DaemonSet
|
|
# will not proceed past an unready pod, so a bad render costs one node, not the cluster.
|
|
# This matters most on a COLD PROD BOOTSTRAP, where every knob is new at once and there is
|
|
# no already-healthy agent to recover from (see DECISIONS-LOG 2026-09-01 INCIDENT/RESEARCH).
|
|
# Cost, recorded: Cilium upgrades serialise and get slower. On 3 nodes that is minutes, and
|
|
# it buys a hard stop on the first bad node — the right trade here. Revisit if the cluster
|
|
# grows enough that serialised CNI upgrades become an operational problem.
|
|
updateStrategy:
|
|
type: RollingUpdate
|
|
rollingUpdate:
|
|
maxUnavailable: 1
|
|
# NOTE (NON-parity): operator.replicas is deliberately omitted here. The Ansible dev render sets
|
|
# replicas: 1 to save RAM, but defaults/main.yml declares it explicitly NOT a parity field
|
|
# (Flux/prod may differ). Only the [parity] values above must match the bootstrap.
|
|
#
|
|
# TODO(WS-NET, Phase 2): default-deny baseline, CiliumL2AnnouncementPolicy + IP pool,
|
|
# Egress Gateway policies, Hubble metrics export to kube-prometheus-stack.
|