jakman85andClaude Opus 5
6f61c9622d
OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach
the cluster and provide fault tolerant HA. Without a change it will not work in HA."
THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live
before changing anything: all three prod cilium-agent pods carried
KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip
VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the
CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said
the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated
that stale comment as an open question when the evidence already settled it.
SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs:
k3s_api_host unchanged — the FOUNDER's IP: the cold-bootstrap join
target and a tls-san. infra/tofu/outputs.tf GENERATES this
name, so renaming would ripple into tofu and the runbooks.
k3s_apiserver_endpoint NEW = {{ kube_vip_address }} — the HA address; what Cilium
consumes. Derived, not restated, so an environment overrides
ONE line and the VIP, the SAN and the CNI endpoint move
together. Prod needed no new override.
THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at
the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran
node-baseline -> k3s -> cilium, so Cilium would install against an address nothing
announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value:
site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier
(kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there
because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing
neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so
a cold run stops THERE with a named cause instead of surfacing as unreachable agents.
Also fixed a pre-existing bug that only mattered once this play joined site.yml: the
role's read-only interface probe ran without `check_mode: false`, so under --check it
was skipped and the assert fired with a misleading "set kube_vip_interface explicitly".
VERIFIED BY ME, not just reported:
renders dev K3S_API_HOST=192.168.1.49 prod=192.168.2.9 (each environment's VIP)
cilium template consumes k3s_apiserver_endpoint, not k3s_api_host
cold join still `--server https://{{ k3s_api_host }}:6443` — the founder
order site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux
gate full CI gate PASS
Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing
k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders
['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved
— .9 already answers under full CA verification; and --check --diff of kube-vip.yml
against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the
kube-vip prod already runs.
ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine
checks: the Cilium template must use the endpoint var; the endpoint must derive from
kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal
that environment's VIP and be none of its node IPs (the [parity] contract, now
machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium.
Every new assertion mutation-tested and bites. One honest negative recorded: reverting
only the BASE default is not caught, because both overlays patch the key — the check
reads effective per-environment values, so the base default is unguarded.
APPLY ORDER, and this is the operator's step — nothing was applied:
PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let
Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet
ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01
incident), it will not proceed past an unready pod, existing datapath survives an
agent restart, and .9 is already live before the change lands.
DEV SEPARATELY, operator present. Same mechanics but live dev agents dial
192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that.
ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a
cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook.
NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live
KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that
test's gap list.
Refs R-PLAT-01, R-NET-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
121 lines
7.7 KiB
YAML
121 lines
7.7 KiB
YAML
---
|
|
# k3s_server role defaults (ADR-0001).
|
|
|
|
# Best-effort version pin (kept near the ci/versions kubectl 1.36.x line). Bump
|
|
# deliberately — a version change is a code change. Empty string => latest stable.
|
|
k3s_version: "v1.36.4+k3s1"
|
|
|
|
k3s_install_url: "https://get.k3s.io"
|
|
k3s_install_script: "/usr/local/bin/k3s-install.sh"
|
|
|
|
# Components disabled at install so Cilium + Gateway API own networking (ADR-0001/0002).
|
|
# flannel + kube-proxy + network-policy are disabled via dedicated flags below;
|
|
# traefik + servicelb via --disable.
|
|
# local-storage: k3s ships local-path as a DEFAULT StorageClass, which collides with Longhorn's
|
|
# default (two default SCs = ambiguous PVC binding). Longhorn (ADR-0003) is the cluster's storage,
|
|
# so disable k3s local-path entirely → `longhorn` is the sole default. (Effective on fresh installs;
|
|
# the k3s install task is idempotent, so this does not restart an existing cluster — see DECISIONS-LOG.)
|
|
k3s_disable: "traefik,servicelb,local-storage"
|
|
|
|
# --- k3s --data-dir (R-ENV-02) ------------------------------------------------------
|
|
# EVERYTHING k3s persists lives under this one directory: the embedded etcd database and
|
|
# WAL (server/db), the node token, the TLS CA and every issued cert, AND the containerd
|
|
# image store (agent/containerd) — which is by far the largest tenant.
|
|
#
|
|
# THE DEFAULT IS k3s'S OWN DEFAULT, restated explicitly rather than left implicit, so the
|
|
# LIVE dev cluster's install command is byte-for-byte what it already was: dev keeps
|
|
# /var/lib/rancher/k3s, where it also has a DEDICATED etcd device mounted at
|
|
# server/db (docs/plans/ssd-etcd-isolation.md). Changing this value on dev would move a
|
|
# running cluster's entire state directory out from under it and orphan that etcd mount.
|
|
#
|
|
# PROD OVERRIDES IT, in inventory/group_vars/k3s_prod/vars.yml — the same seam
|
|
# kube_vip_address / k3s_api_host / node_baseline_backports_kernel use, so a FOURTH prod
|
|
# node inherits it from group membership rather than from anyone remembering. It is set
|
|
# there as the CLUSTER-WIDE `k3s_data_dir` because TWO roles must agree on it: this role
|
|
# turns it into --data-dir, and roles/node_baseline creates the directory on the claimed
|
|
# data partition and derives etcd's location from it (node_baseline_k3s_data_dir defers to
|
|
# this same variable). One line moves both.
|
|
#
|
|
# WHY PROD MOVES IT (infra/PROD-PROVISIONING.md §2, revised 2026-09-04). Prod's /var is a
|
|
# separate partition of 11.8 GiB. Measured on the fully-deployed dev cluster, containerd
|
|
# alone is 21-25 GB per node and /var totals 58-70 GB — so prod would fill /var during its
|
|
# FIRST image pulls, and a full /var means etcd cannot write its WAL. That is cluster death,
|
|
# not degradation. Prod points --data-dir at /data/k3s on the claimed ~424 GB partition.
|
|
#
|
|
# NOT ALSO WRITTEN TO /etc/rancher/k3s/config.yaml, deliberately. --data-dir is consumed at
|
|
# INSTALL time (it is baked into the systemd unit's ExecStart); adding a `data-dir:` line to
|
|
# config.yaml would make that file `changed` on the LIVE dev cluster and fire the rolling
|
|
# Restart handler for a value that is not changing. The install flags are the correct and
|
|
# sufficient place, and prod has no k3s installed yet.
|
|
k3s_data_dir: "/var/lib/rancher/k3s"
|
|
|
|
# Set per-play: 'primary' (runs --cluster-init) or 'secondary' (joins the primary).
|
|
k3s_server_mode: "secondary"
|
|
|
|
# Subject Alternative Names baked into the kube-apiserver serving cert. Must include
|
|
# the kube-vip control-plane VIP (kube_vip_address) so TLS via the VIP validates,
|
|
# every server's real node IP (so the cert is valid whichever node answers behind
|
|
# the VIP), and 127.0.0.1 (local kubeconfig / health probes). k3s regenerates the
|
|
# serving cert on restart when this set changes. Deduplicated (the primary IP is
|
|
# also k3s_api_host).
|
|
#
|
|
# THIS LIST COVERS BOTH API ADDRESSES, WHICH IS WHY THE 2026-09-06 SPLIT NEEDED NO CERT
|
|
# CHANGE AND NO k3s RESTART. group_vars/all.yml now distinguishes k3s_api_host (the
|
|
# founder, for the cold join) from k3s_apiserver_endpoint (= kube_vip_address, what Cilium
|
|
# and every steady-state client dial). Both are already in this set, so repointing a
|
|
# consumer from one to the other cannot produce a TLS failure. VERIFIED LIVE 2026-09-06,
|
|
# not assumed: prod's apiserver answered through https://192.168.2.9:6443 with full CA
|
|
# verification and tls-server-name=192.168.2.9. Do not remove either entry — dropping
|
|
# kube_vip_address breaks every client of the VIP; dropping k3s_api_host breaks nothing
|
|
# today only because it is also in the k3s_servers_group node-IP list below, and that
|
|
# coincidence is not a contract.
|
|
#
|
|
# k3s_servers_group — NOT a literal group name — is what makes this role environment-
|
|
# agnostic (R-ENV-02): group_vars/all.yml sets it to `k3s_servers` (dev) and
|
|
# group_vars/k3s_prod/vars.yml to `k3s_prod`, so each environment SANs its own servers.
|
|
k3s_tls_sans: >-
|
|
{{ ([k3s_api_host, kube_vip_address, '127.0.0.1']
|
|
+ (groups[k3s_servers_group] | map('extract', hostvars, 'k3s_node_ip') | list))
|
|
| unique }}
|
|
|
|
# --- Keycloak OIDC issuer for kubectl SSO (R-IDP-03, R-ENV-02) -----------------------
|
|
# THE ANSIBLE-SIDE HALF OF ${BASE_DOMAIN}, and it cannot be the Flux-side half.
|
|
#
|
|
# The kube-apiserver's `oidc-issuer-url` is a k3s SERVER FLAG written into
|
|
# /etc/rancher/k3s/config.yaml by this role — it is NOT a Kubernetes manifest, so Flux never
|
|
# sees it and `postBuild.substituteFrom: cluster-vars` cannot reach it. The platform hostname
|
|
# suffix therefore exists as ONE FACT IN TWO PLACES, and they must agree:
|
|
#
|
|
# Flux side clusters/base/cluster-vars.yaml BASE_DOMAIN (per-overlay)
|
|
# Ansible side THIS variable k3s_base_domain (per-inventory-group)
|
|
#
|
|
# [parity] k3s_base_domain MUST equal the BASE_DOMAIN the SAME cluster's Flux overlay sets.
|
|
# If they disagree, the apiserver validates ID tokens against a DIFFERENT Keycloak than the
|
|
# one the cluster actually runs: every `kubectl` SSO login fails with an issuer mismatch,
|
|
# while the cert-based admin kubeconfig keeps working — so it looks like "SSO is broken",
|
|
# not like a config skew. The break-glass cert kubeconfig is why this is recoverable.
|
|
#
|
|
# THE DEFAULT IS THE DEV VALUE — the same convention as k3s_data_dir above and cluster-vars'
|
|
# GATEWAY_VIP / LONGHORN_DATA_PATH: the role default is dev's value and prod overrides it in
|
|
# inventory/group_vars/k3s_prod/vars.yml (the same inventory seam k3s_data_dir / k3s_api_host /
|
|
# kube_vip_address use, so a FOURTH prod server inherits it from group membership rather than
|
|
# from anyone remembering).
|
|
#
|
|
# *** WHICH DEV VALUE — UPDATED 2026-09-04, AND IT IS AHEAD OF THE LIVE DEV CLUSTER. ***
|
|
# The operator inverted the suffix split: PROD is now the apex `home.keithjackson.dev` and DEV
|
|
# carries the label `dev.home.keithjackson.dev`. This default therefore moved from the apex to
|
|
# dev's subdomain, mirroring BASE_DOMAIN in clusters/dev/kustomization.yaml.
|
|
#
|
|
# CONSEQUENCE — this default no longer matches what the live dev nodes are running, so unlike
|
|
# every previous edit to this file A DEV RE-RUN IS NOT A NO-OP. Rendering config.yaml with the
|
|
# new issuer CHANGES the file, FIRES the Restart handler, and rolls k3s on all three dev
|
|
# servers, which also invalidates every user's OIDC kubeconfig (SSO must be re-issued). That is
|
|
# a deliberate step of the dev DNS migration, NOT a side effect to discover during unrelated
|
|
# work: DO NOT run the k3s_server role against dev until step D6 of
|
|
# infra/PROD-PROVISIONING.md §10, and never while the operator is away. The break-glass
|
|
# cert-based admin kubeconfig keeps working throughout, which is what makes the step safe.
|
|
k3s_base_domain: "dev.home.keithjackson.dev"
|
|
|
|
# Derived, so the realm path is stated once. Override k3s_base_domain, not this.
|
|
k3s_oidc_issuer_url: "https://keycloak.{{ k3s_base_domain }}/realms/homelab"
|