jakman85andClaude Opus 5
d0db63911a
A merged Cilium config change did nothing, and everything reported success. The Hubble flow exporter (#50, merged 2026-10-04) never took effect: HelmRelease cilium Ready=True, "Helm upgrade succeeded ... cilium.v6" cilium-config hubble-export-file-path: stdout + the allowlist, PRESENT cilium agent pods all three still PRE-MERGE, startTime 2026-10-04T07:11Z exported flow lines 0, 0, 0 Flux every Kustomization Ready=True Cause: the chart's rollOutCiliumPods defaults to false, so Helm rewrites the ConfigMap and leaves the DaemonSet's pod template untouched -- and cilium-agent reads that ConfigMap only at STARTUP. Merged, believed delivered, green everywhere, doing nothing. The same class of failure this repo already guards against elsewhere: a green signal over something that never ran. Confirmed by contrast rather than reasoned: the cgroup fix (#42) DID apply -- both the DaemonSet and the running pod mount /sys/fs/cgroup -- because it changed the POD TEMPLATE, which rolls the DaemonSet on its own. That also explains why the agents are dated 2026-10-04T07:11Z: that was #42's rollout, and nothing has rolled them since. The inertness is specific to ConfigMap-only changes. Set in all three value files to hold the [parity] contract. Cost, and it is the point: every future Cilium config change now rolls the agents instead of waiting for a manual restart. Strictly more honest -- it applies when merged or fails visibly -- but config edits become disruptive by default. maxUnavailable: 1 bounds it to one node at a time. The lease-holding node briefly loses the Gateway, PiHole-DNS and Plex VIPs as it cycles, so merge at a quiet moment. Merging this IS one rollout (the ConfigMap checksum annotation changes the pod template) -- which is what finally activates the Hubble exporter. UNVERIFIED: unreconciled; `filePath: stdout` still unproven and only answerable after the agents restart (check for flow lines immediately); and the rollout's actual impact on LAN DNS is unmeasured. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
102 lines
5.1 KiB
Django/Jinja
102 lines
5.1 KiB
Django/Jinja
# Cilium Helm values — rendered by the Ansible bootstrap (T-1.3).
|
|
# PARITY CONTRACT (docs/02): the fields marked [parity] MUST equal the
|
|
# platform/cilium HelmRelease values, or Flux and the bootstrap fight over the CNI.
|
|
---
|
|
# [parity] k3s kube-proxy is disabled → Cilium replaces it (ADR-0001/0002).
|
|
kubeProxyReplacement: true
|
|
# [parity] = k3s_apiserver_endpoint, THE HA CONTROL-PLANE VIP (kube_vip_address) — dev
|
|
# 192.168.1.49, prod 192.168.2.9. NOT k3s_api_host, which is the FOUNDER'S OWN node IP:
|
|
# pointing the CNI's kube-proxy replacement at one machine on a 3-server HA cluster means
|
|
# losing that machine stops every service/policy update reaching the datapath, silently
|
|
# (the existing datapath keeps working from Cilium's cache, so nothing looks broken). That
|
|
# was the live state of prod until 2026-09-06 — all three cilium-agents carried
|
|
# KUBERNETES_SERVICE_HOST=192.168.2.10 while kube-vip held .9. See group_vars/all.yml.
|
|
#
|
|
# [parity] The Flux side is platform/cilium/helmrelease.yaml's k8sServiceHost:
|
|
# "${K3S_API_HOST}", whose per-environment value comes from clusters/<env>/kustomization.yaml.
|
|
# THAT VALUE MUST EQUAL k3s_apiserver_endpoint for the SAME environment. If it does not,
|
|
# Flux and this bootstrap render different control-plane paths and fight over the CNI.
|
|
#
|
|
# THE VIP MUST BE UP BEFORE THIS ROLE RUNS. playbooks/site.yml imports kube-vip.yml between
|
|
# k3s.yml and cilium.yml for exactly this reason, and roles/kube_vip blocks until the VIP
|
|
# answers. Rendering this against a VIP that nothing announces yet reproduces the
|
|
# 2026-09-01 shape: agents that cannot reach the API server on a cold cluster.
|
|
k8sServiceHost: "{{ k3s_apiserver_endpoint }}"
|
|
k8sServicePort: {{ k3s_api_port }}
|
|
|
|
# Ingress via Gateway API — no Traefik (docs/09, ADR-0002).
|
|
gatewayAPI:
|
|
enabled: true
|
|
|
|
# Egress-class steering (ADR-0002; VPN model ADR-0013, Phase 2/3).
|
|
egressGateway:
|
|
enabled: true
|
|
|
|
# Egress Gateway REQUIRES masquerade (ipv4 + eBPF). eBPF masquerade is the correct
|
|
# pairing with kubeProxyReplacement. [parity] must match platform/cilium HelmRelease.
|
|
enableIPv4Masquerade: true
|
|
bpf:
|
|
masquerade: true
|
|
|
|
# Flow observability / isolation audit (ADR-0002).
|
|
# Roll the agents when cilium-config changes. Kept EQUAL to the Flux values
|
|
# (platform/cilium/helmrelease.yaml), which carry the full rationale: the chart default of false
|
|
# leaves the pod template untouched on a config change, and cilium-agent reads that ConfigMap only
|
|
# at startup, so the change is silently inert until someone restarts the DaemonSet by hand.
|
|
rollOutCiliumPods: true
|
|
|
|
hubble:
|
|
enabled: true
|
|
relay:
|
|
enabled: true
|
|
ui:
|
|
enabled: true
|
|
# Static flow export -> cilium-agent stdout. Kept EQUAL to the Flux values
|
|
# (platform/cilium/helmrelease.yaml), which carry the full rationale: it records the PRE-SNAT
|
|
# client address for Plex's raw tcp/32400, which Plex itself cannot log because every
|
|
# LoadBalancer Service here is externalTrafficPolicy: Cluster and the VIP lease holder SNATs the
|
|
# client. Present here so a BOOTSTRAP render does not differ from what Flux reconciles -- a
|
|
# mismatch would show up as a permanent Helm diff and an avoidable agent rollout. Scoped by PORT,
|
|
# not namespace: the flow carrying the real client address does not match a namespace filter.
|
|
export:
|
|
static:
|
|
enabled: true
|
|
filePath: stdout
|
|
allowList:
|
|
- '{"destination_port":["32400"]}'
|
|
- '{"source_port":["32400"]}'
|
|
|
|
# Ingress VIP announcement on the NAT LAN. IP pool / CiliumL2AnnouncementPolicy
|
|
# (from the reserved 10.73.0.240-.250 range) is authored by WS-NET in Phase 2.
|
|
l2announcements:
|
|
enabled: true
|
|
|
|
# Point Cilium at the host's real cgroup2 root (/sys/fs/cgroup, mounted by systemd) instead of
|
|
# auto-mounting. NOT /run/cilium/cgroupv2: nothing mounts cgroup2 there, so the agent mounted its
|
|
# OWN namespaced cgroup and socket-LB covered only itself — host processes (the apiserver) lost
|
|
# Service/LB translation. [parity] platform/cilium/helmrelease.yaml explains (2026-10-03).
|
|
cgroup:
|
|
autoMount:
|
|
enabled: false
|
|
hostRoot: /sys/fs/cgroup
|
|
|
|
# Roll the cilium agent DaemonSet ONE node at a time. [parity] must match the
|
|
# platform/cilium HelmRelease. The cilium 1.20.1 chart defaults this to 2, which on a
|
|
# 3-node cluster lets one bad render take down two of three agents simultaneously —
|
|
# exactly the 2026-09-01 incident, where k8sServiceHost: "" made the agent's `config`
|
|
# init container fall back to the in-cluster ClusterIP 10.43.0.1:443. Only the Cilium
|
|
# agent can translate that ClusterIP, and the agent in question was the one being
|
|
# replaced: `dial tcp 10.43.0.1:443: i/o timeout`, deadlocked.
|
|
# A CNI rollout has no healthy-peer fallback — the component being upgraded is the one
|
|
# the upgrade needs to reach the API server. This bites hardest on a COLD BOOTSTRAP
|
|
# (this role's job), where nothing healthy exists to recover from. With maxUnavailable: 1
|
|
# the DaemonSet will not proceed past an unready pod, bounding damage to a single node.
|
|
# Cost: Cilium upgrades serialise and take longer; at 3 nodes that is an acceptable trade.
|
|
updateStrategy:
|
|
type: RollingUpdate
|
|
rollingUpdate:
|
|
maxUnavailable: 1
|
|
|
|
operator:
|
|
replicas: {{ cilium_operator_replicas }}
|