Add an HA control-plane API VIP for the k3s cluster with kube-vip, and add the VIP + all node IPs to the kube-apiserver serving-cert SANs so TLS via the VIP validates. Prepares the stable failover endpoint the operator fronts with a pfSense port-forward to 192.168.1.49:6443. roles/kube_vip + playbooks/kube-vip.yml: - Templates a single manifest (RBAC + DaemonSet kube-vip-ds) into each server's k3s auto-deploy dir (/var/lib/rancher/k3s/server/manifests) so it is reproducible and independent of Flux; k3s auto-applies it. - Control-plane ONLY (cp_enable=true, svc_enable=false): Cilium keeps ownership of Service LoadBalancing (L2 pool .53-.60) — no ARP conflict. - Runs on control-plane nodes (nodeSelector node-role.kubernetes.io/control-plane), hostNetwork, NET_ADMIN/NET_RAW, image pinned ghcr.io/kube-vip/kube-vip:v0.8.9. - vip_interface auto-detected per node (the iface holding 192.168.1.5x = eth0). - Leader election (plndr-cp-lock) with 15/10/2s lease timings. - This cluster runs without kube-proxy (Cilium replacement), so a hostNetwork pod cannot reach the kubernetes ClusterIP; kube-vip is pointed at the node-local apiserver (KUBERNETES_SERVICE_HOST=127.0.0.1:6443) so leader election works. roles/k3s_server tls-san: - New k3s_tls_sans (VIP + all three node IPs + 127.0.0.1, deduped); install args now emit one --tls-san per entry. - Persist the SAN list to /etc/rancher/k3s/config.yaml (read by a running k3s) and Restart k3s handler so an existing cluster regenerates its serving cert. - k3s.yml secondaries run serial:1 so a SAN change rolls one node at a time, keeping etcd quorum (kube-vip keeps the API reachable during each restart). Verified live: DS 3/3 ready, VIP answers on 192.168.1.49:6443, all 3 certs and an s_client against the VIP carry 192.168.1.49 + .50/.51/.52 + 127.0.0.1, all nodes Ready throughout the rolling restart. Co-Authored-By: Claude Opus 4.8 <[email protected]>
112 lines
4.3 KiB
YAML
112 lines
4.3 KiB
YAML
---
|
|
# Install k3s as an HA server (embedded etcd) with Cilium-required components
|
|
# disabled (ADR-0001, docs/02 stage 3 "Cluster-join"). The primary runs
|
|
# --cluster-init; secondaries join it with the shared node-token.
|
|
|
|
- name: Download the k3s install script
|
|
ansible.builtin.get_url:
|
|
url: "{{ k3s_install_url }}"
|
|
dest: "{{ k3s_install_script }}"
|
|
mode: "0755"
|
|
|
|
# --node-ip uses k3s_node_ip (the real 192.168.1.x), NOT ansible_host — the latter is
|
|
# now the pfSense NAT SSH target (192.168.2.102:225x) and must not be advertised to k3s.
|
|
- name: Compose the common k3s server flags
|
|
ansible.builtin.set_fact:
|
|
k3s_common_args: >-
|
|
server
|
|
--flannel-backend=none
|
|
--disable-kube-proxy
|
|
--disable-network-policy
|
|
--disable={{ k3s_disable }}
|
|
--write-kubeconfig-mode=644
|
|
{% for san in k3s_tls_sans %}--tls-san={{ san }} {% endfor %}
|
|
--node-ip={{ k3s_node_ip }}
|
|
|
|
# On an ALREADY-installed cluster the install task above is idempotent (creates:
|
|
# k3s.yaml) and won't re-apply new --tls-san flags. Persist the SAN list in
|
|
# /etc/rancher/k3s/config.yaml, which a running k3s reads on the next restart —
|
|
# it then regenerates the serving cert when the SAN set changes. Apply the
|
|
# restart ROLLING, one node at a time (kube-vip keeps the API reachable; etcd
|
|
# quorum holds at 2/3). See playbooks note: run with `serial: 1`.
|
|
- name: Write /etc/rancher/k3s/config.yaml with the tls-san list
|
|
ansible.builtin.copy:
|
|
dest: /etc/rancher/k3s/config.yaml
|
|
owner: root
|
|
group: root
|
|
mode: "0600"
|
|
content: |
|
|
# Managed by Ansible (roles/k3s_server). Read by k3s in addition to the
|
|
# systemd unit flags; tls-san entries are merged into the serving cert.
|
|
tls-san:
|
|
{% for san in k3s_tls_sans %}
|
|
- "{{ san }}"
|
|
{% endfor %}
|
|
notify: Restart k3s
|
|
|
|
# --- primary: found the cluster --------------------------------------------
|
|
- name: Install k3s primary (cluster-init, embedded etcd)
|
|
ansible.builtin.command:
|
|
cmd: "{{ k3s_install_script }} {{ k3s_common_args }} --cluster-init"
|
|
creates: /etc/rancher/k3s/k3s.yaml
|
|
environment:
|
|
INSTALL_K3S_VERSION: "{{ k3s_version }}"
|
|
when: k3s_server_mode == 'primary'
|
|
|
|
- name: Wait for the k3s node-token to exist (primary)
|
|
ansible.builtin.wait_for:
|
|
path: /var/lib/rancher/k3s/server/token
|
|
timeout: 180
|
|
when: k3s_server_mode == 'primary'
|
|
|
|
- name: Read the k3s node-token (primary)
|
|
ansible.builtin.slurp:
|
|
src: /var/lib/rancher/k3s/server/token
|
|
register: k3s_token_raw
|
|
when: k3s_server_mode == 'primary'
|
|
|
|
- name: Publish the node-token as a fact for secondaries (primary)
|
|
ansible.builtin.set_fact:
|
|
k3s_node_token: "{{ k3s_token_raw.content | b64decode | trim }}"
|
|
when: k3s_server_mode == 'primary'
|
|
|
|
# --- secondaries: obtain the join token, then join the existing cluster -----
|
|
# Fetch the token straight from the primary via delegate_to, falling back to it
|
|
# only when the primary play didn't publish the fact this run. This makes a
|
|
# single-node rejoin (`--limit k3s-dev-N`, primary NOT in the batch) self-
|
|
# contained — delegate_to still reaches the primary even when it isn't a play
|
|
# target, so the join no longer depends on a fact an earlier play had to set.
|
|
# T-1.5 finding F2 (see DECISIONS-LOG.md).
|
|
- name: Fetch the k3s node-token directly from the primary (secondary)
|
|
ansible.builtin.slurp:
|
|
src: /var/lib/rancher/k3s/server/token
|
|
delegate_to: "{{ groups['k3s_primary'][0] }}"
|
|
register: k3s_primary_token_raw
|
|
when:
|
|
- k3s_server_mode == 'secondary'
|
|
- hostvars[groups['k3s_primary'][0]].k3s_node_token is not defined
|
|
|
|
- name: Resolve the join token (secondary)
|
|
ansible.builtin.set_fact:
|
|
k3s_join_token: >-
|
|
{{ hostvars[groups['k3s_primary'][0]].k3s_node_token
|
|
| default(k3s_primary_token_raw.content | b64decode | trim, true) }}
|
|
when: k3s_server_mode == 'secondary'
|
|
|
|
- name: Install k3s secondary (join as server)
|
|
ansible.builtin.command:
|
|
cmd: >-
|
|
{{ k3s_install_script }} {{ k3s_common_args }}
|
|
--server https://{{ k3s_api_host }}:{{ k3s_api_port }}
|
|
creates: /etc/rancher/k3s/k3s.yaml
|
|
environment:
|
|
INSTALL_K3S_VERSION: "{{ k3s_version }}"
|
|
K3S_TOKEN: "{{ k3s_join_token }}"
|
|
when: k3s_server_mode == 'secondary'
|
|
|
|
- name: Wait for the k3s API to be serving locally
|
|
ansible.builtin.wait_for:
|
|
host: "127.0.0.1"
|
|
port: "{{ k3s_api_port }}"
|
|
timeout: 180
|