Prod is being stood up now, without waiting for node 2's RAM upgrade. Three coupled
changes, all applied and verified on all three nodes.
=== 1. KERNEL >= 6.13 — now a PREREQUISITE, not a nice-to-have ===
A k3s SERVER that intermittently boots with 2.6 GiB is not a degraded node; it fails under
etcd plus its workload share, unpredictably, on any reboot. Fixed before k3s goes on.
roles/node_baseline gains a `kernel`-tagged block: asserts the suite matches the node's
Debian release, writes a deb822 trixie-backports source, installs, reads back the exact
version, writes a provenance/rollback note to the node. Scoped by
node_baseline_backports_kernel (default FALSE, true only in group_vars/k3s_prod) — chosen
over a hostname test so a fourth prod node inherits it from group membership. Verified:
--check against dev skips all 8 tasks on all 3 hosts. Two deliberate calls: `state: latest`
not `present` (linux-image-amd64 was already installed from main, so `present` would have
silently fixed nothing), and NO preferences.d pin (backports' NotAutomatic +
ButAutomaticUpgrades already gives 100/500; an explicit Pin-Priority: 100 would switch OFF
automatic security upgrades for the very kernel being installed).
Installed 7.1.8+deb13-amd64 on all three; 6.12.107 kept bootable.
THE DECISIVE PROOF: node 2 booted at MEMATTR=0xbeeb0018 — the EXACT address that gave it
2.6 GiB twice — and now reserves 634,904K instead of 13,958,980K. MemTotal 15.4 GiB. Same
address, handled correctly; not a lucky draw.
§0b's claim that a fixed kernel prints 'FW_BUG Corrupted...' is CONDITIONAL and was
corrected: >=6.13 has three rejection gates and only the third prints that line; all three
nodes hit gate 1 (version > 2). Proof instead is boot-independent — decompressing
/boot/vmlinuz on all three yields the FW_BUG and descriptor-size strings verbatim; neither
exists in 6.12.
TWO MORE BUGS IN MY OWN HEALTH CHECK, found because the fix couldn't otherwise be shown:
- MEMATTR_VERSION_RE matched (\d+) but the kernel prints that field with %d — SIGNED. Node
2's real value was -1569980416, which did not match, so state fell through to `clean` and
the node reported PASS "the kernel accepted it without complaint". A FALSE PASS on the
exact bug the check exists to find. Now (-?\d+).
- bad_version was WARN unconditionally, so a FIXED node would have stayed WARN forever.
Now keys on memattr_kernel_fixed: same line, WARN below 6.13, PASS at/above.
=== 2. DISK LAYOUT — Longhorn claimed, per-node paths DISCOVERED not hardcoded ===
Operator decision: reuse the ~424 GB partition; accept the shared-device I/O risk (dev's
etcd/Longhorn contention was on a spinning HDD, prod is SSD/NVMe).
Device paths differ per node (sda5 / nvme0n1p5 / sda5), so the role DISCOVERS the partition
via `findmnt --mountpoint` on a declared claim mountpoint rather than trusting inventory.
Per-host host_vars would have frozen today's hardware into inventory, where it goes stale on
the first re-image and the stale value's failure mode is mkfs on the WRONG partition.
Eleven fail-closed asserts before any write: real block device, not backing / /boot /usr
/var /home /tmp or active swap (built from the node's OWN gathered facts, never a hardcoded
list), already ext4, no foreign label, >= 100 GiB, and empty. Relabel, never reformat.
/srv verified empty first on every node (ls -A -> lost+found only; du 20K; the "5.1% used"
was ext4's default 5% root reserve).
=== 3. k3s DATA-DIR MOVED — /var was 11.8 GiB and would have killed etcd ===
Measured on the fully-deployed dev cluster: containerd 21-25 GB/node, /var total 58-70 GB.
Prod's /var is 11.8 GiB and held BOTH etcd and the image store, so prod would have filled
/var during its first image pulls — and a full /var means etcd cannot write its WAL. Cluster
death, not degradation, arriving during bring-up. Invisible when the disk decision was made,
because "etcd on root" sounded like the 30 GiB / rather than a separate 12 GiB /var.
Now: p5 -> /data (LABEL=k3s-data), with /data/longhorn and /data/k3s. Neutral names chosen
deliberately — the partition now has two tenants, and reusing /srv would make "claimed" and
"unclaimed" indistinguishable to the idempotency pivot, while a bare `data` label would make
a stranger's `data` volume look already-claimed and weaken an assert.
The transition off the existing LABEL=longhorn mount is DECLARED, not inferred
(node_baseline_data_retired_fs_label/_retired_mount, defaults ""), because changing the label
makes a claimed node answer "no" to the pivot and walk into the greenfield path. Self-retiring
once blkid -L k3s-data succeeds. THREE independent gates keep live dev out: retired vars
default ""; gated on longhorn_source == claim-mountpoint (dev is `device`); and the emptiness
assert fails closed on dev's 18 volumes even if both were defeated.
k3s_data_dir is one variable read by two roles (k3s_server --data-dir; node_baseline derives
the etcd mount). Role default is k3s's own default, so dev is unchanged and keeps its
dedicated etcd device. node_baseline_etcd_source becomes `shared-data-partition` and is
LOAD-BEARING: the role asserts per run that the filesystem backing the etcd dir really IS the
data partition — had the mount silently failed, etcd plus 21-25 GB of images would land back
on the OS filesystem invisibly until the disk filled.
LONGHORN_DATA_PATH is a new cluster-vars knob (R-ENV-02): base and DEV /var/lib/longhorn,
prod /data/longhorn.
DEV PROVEN UNCHANGED, which is the property that mattered — the live cluster has 18 Longhorn
volumes on that path. All 12 substituting Flux paths rendered through
`kubectl kustomize | flux envsubst --strict` for both trees: 12/12 per-file SHA-256 identical
and an identical aggregate SHA over every rendered dev byte. With a POSITIVE CONTROL showing
the same harness detects prod changing on exactly one line — an all-identical result is
worthless if the harness cannot see change. Live check after: dev's default-data-path setting
still /var/lib/longhorn, 18 volumes attached, 0 Kustomizations unready.
Idempotent: run 2 is changed=0 on all three nodes. Also fixed a real tag bug — include_tasks
does NOT propagate its own tags to included tasks (unlike import_tasks), so `--tags disks`
selected the include and ran nothing inside it while reporting success.
OPEN DEFECTS, tracked not forgotten: node 2's warm-reboot hang (one clean reboot since, NOT a
fix); and node 3 runs dhcpcd for its wifi interface, which rewrites /etc/resolv.conf on lease
renewal — the role's `contended` detection checks resolvconf and NetworkManager only, so node
3's decided resolvers will silently revert.
Gate: MERGE GATE: PASS. L0 substitution check 106 assertions.
Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-STATE-01, R-STATE-02, R-PLAT-01. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
161 lines
7.3 KiB
YAML
161 lines
7.3 KiB
YAML
---
|
|
# Install k3s as an HA server (embedded etcd) with Cilium-required components
|
|
# disabled (ADR-0001, docs/02 stage 3 "Cluster-join"). The primary runs
|
|
# --cluster-init; secondaries join it with the shared node-token.
|
|
|
|
- name: Download the k3s install script
|
|
ansible.builtin.get_url:
|
|
url: "{{ k3s_install_url }}"
|
|
dest: "{{ k3s_install_script }}"
|
|
mode: "0755"
|
|
|
|
# --node-ip uses k3s_node_ip (the real 192.168.1.x), NOT ansible_host — the latter is
|
|
# now the pfSense NAT SSH target (192.168.2.102:225x) and must not be advertised to k3s.
|
|
- name: Compose the common k3s server flags
|
|
ansible.builtin.set_fact:
|
|
k3s_common_args: >-
|
|
server
|
|
--flannel-backend=none
|
|
--disable-kube-proxy
|
|
--disable-network-policy
|
|
--disable={{ k3s_disable }}
|
|
--write-kubeconfig-mode=644
|
|
{% for san in k3s_tls_sans %}--tls-san={{ san }} {% endfor %}
|
|
--node-ip={{ k3s_node_ip }}
|
|
--data-dir={{ k3s_data_dir }}
|
|
|
|
# On an ALREADY-installed cluster the install task above is idempotent (creates:
|
|
# k3s.yaml) and won't re-apply new --tls-san flags. Persist the SAN list in
|
|
# /etc/rancher/k3s/config.yaml, which a running k3s reads on the next restart —
|
|
# it then regenerates the serving cert when the SAN set changes, and regenerates
|
|
# its containerd config.toml (so nonroot-devices takes effect). Apply the restart
|
|
# ROLLING, one node at a time (kube-vip keeps the API reachable; etcd quorum holds
|
|
# at 2/3). See playbooks note: run with `serial: 1`.
|
|
- name: Ensure /etc/rancher/k3s exists (a freshly re-provisioned node has no k3s dir until install)
|
|
ansible.builtin.file:
|
|
path: /etc/rancher/k3s
|
|
state: directory
|
|
owner: root
|
|
group: root
|
|
mode: "0755"
|
|
|
|
- name: Write /etc/rancher/k3s/config.yaml (tls-san + nonroot-devices + oidc)
|
|
ansible.builtin.copy:
|
|
dest: /etc/rancher/k3s/config.yaml
|
|
owner: root
|
|
group: root
|
|
mode: "0600"
|
|
content: |
|
|
# Managed by Ansible (roles/k3s_server). Read by k3s in addition to the
|
|
# systemd unit flags; tls-san entries are merged into the serving cert.
|
|
tls-san:
|
|
{% for san in k3s_tls_sans %}
|
|
- "{{ san }}"
|
|
{% endfor %}
|
|
|
|
# Let non-root pods own the raw block devices bind-mounted into them —
|
|
# sets containerd's `device_ownership_from_security_context = true`
|
|
# (k3s >= v1.31.3+k3s1 exposes this as the --nonroot-devices flag).
|
|
#
|
|
# REQUIRED for KubeVirt/CDI on Block-mode PVCs (ADR-0005): CDI's importer,
|
|
# uploader and virt-launcher pods run non-root (UID 107, runAsNonRoot,
|
|
# cap-drop ALL) and open the volume's raw block device (/dev/cdi-block-volume,
|
|
# then the guest disk). Kubernetes only applies fsGroup ownership to
|
|
# *filesystem* volumes — it never chowns a raw block device — and Longhorn's
|
|
# CSIDriver (fsGroupPolicy: ReadWriteOnceWithFSType) skips fsGroup on RWX/Block
|
|
# volumes anyway, so the device stays root:root and the pod fails with
|
|
# `blockdev: cannot open /dev/cdi-block-volume: Permission denied`. This flag
|
|
# makes containerd set the in-container device node's owner from the pod's
|
|
# securityContext, which is the fix the CDI project documents for this case
|
|
# (doc/block_cri_ownership_config.md). Node-scoped and security-neutral: the
|
|
# pod stays non-root, only the device node it already mounts follows its UID/GID.
|
|
# Preserves the longhorn-vm RWX + Block + migratable disk contract unchanged.
|
|
nonroot-devices: true
|
|
|
|
# OIDC (Keycloak SSO for kubectl). The API server trusts ID tokens from the homelab realm;
|
|
# the `kubernetes` PUBLIC client (PKCE) + kubelogin mint them, and RBAC binds the `groups`
|
|
# claim (platform/keycloak/k8s-oidc-rbac.yaml: oidc:platform-admins->cluster-admin,
|
|
# oidc:viewers->view). Nodes resolve keycloak.home.keithjackson.dev to the Cilium gateway VIP
|
|
# (192.168.1.53) and trust its public LE-prod cert, so NO oidc-ca-file is needed. The `oidc:`
|
|
# prefixes namespace federated identities away from cert / service-account subjects.
|
|
# ADDITIVE + break-glass-safe: the cert-based admin kubeconfig keeps full access with or
|
|
# without Keycloak, so a bad realm/OIDC state can never lock you out of the cluster.
|
|
kube-apiserver-arg:
|
|
- "oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab"
|
|
- "oidc-client-id=kubernetes"
|
|
- "oidc-username-claim=email"
|
|
- "oidc-username-prefix=oidc:"
|
|
- "oidc-groups-claim=groups"
|
|
- "oidc-groups-prefix=oidc:"
|
|
notify: Restart k3s
|
|
|
|
# --- primary: found the cluster --------------------------------------------
|
|
- name: Install k3s primary (cluster-init, embedded etcd)
|
|
ansible.builtin.command:
|
|
cmd: "{{ k3s_install_script }} {{ k3s_common_args }} --cluster-init"
|
|
creates: /etc/rancher/k3s/k3s.yaml
|
|
environment:
|
|
INSTALL_K3S_VERSION: "{{ k3s_version }}"
|
|
when: k3s_server_mode == 'primary'
|
|
|
|
- name: Wait for the k3s node-token to exist (primary)
|
|
ansible.builtin.wait_for:
|
|
path: /var/lib/rancher/k3s/server/token
|
|
timeout: 180
|
|
when: k3s_server_mode == 'primary'
|
|
|
|
- name: Read the k3s node-token (primary)
|
|
ansible.builtin.slurp:
|
|
src: /var/lib/rancher/k3s/server/token
|
|
register: k3s_token_raw
|
|
when: k3s_server_mode == 'primary'
|
|
|
|
- name: Publish the node-token as a fact for secondaries (primary)
|
|
ansible.builtin.set_fact:
|
|
k3s_node_token: "{{ k3s_token_raw.content | b64decode | trim }}"
|
|
when: k3s_server_mode == 'primary'
|
|
|
|
# --- secondaries: obtain the join token, then join the existing cluster -----
|
|
# Fetch the token straight from the primary via delegate_to, falling back to it
|
|
# only when the primary play didn't publish the fact this run. This makes a
|
|
# single-node rejoin (`--limit k3s-dev-N`, primary NOT in the batch) self-
|
|
# contained — delegate_to still reaches the primary even when it isn't a play
|
|
# target, so the join no longer depends on a fact an earlier play had to set.
|
|
# T-1.5 finding F2 (see DECISIONS-LOG.md).
|
|
#
|
|
# "The primary" is resolved through k3s_primary_group (group_vars: `k3s_primary` for
|
|
# dev, `k3s_prod_primary` for prod), never a literal group name — so a prod secondary
|
|
# delegates to the PROD primary using this same role, unchanged (R-ENV-02).
|
|
- name: Fetch the k3s node-token directly from the primary (secondary)
|
|
ansible.builtin.slurp:
|
|
src: /var/lib/rancher/k3s/server/token
|
|
delegate_to: "{{ groups[k3s_primary_group][0] }}"
|
|
register: k3s_primary_token_raw
|
|
when:
|
|
- k3s_server_mode == 'secondary'
|
|
- hostvars[groups[k3s_primary_group][0]].k3s_node_token is not defined
|
|
|
|
- name: Resolve the join token (secondary)
|
|
ansible.builtin.set_fact:
|
|
k3s_join_token: >-
|
|
{{ hostvars[groups[k3s_primary_group][0]].k3s_node_token
|
|
| default(k3s_primary_token_raw.content | b64decode | trim, true) }}
|
|
when: k3s_server_mode == 'secondary'
|
|
|
|
- name: Install k3s secondary (join as server)
|
|
ansible.builtin.command:
|
|
cmd: >-
|
|
{{ k3s_install_script }} {{ k3s_common_args }}
|
|
--server https://{{ k3s_api_host }}:{{ k3s_api_port }}
|
|
creates: /etc/rancher/k3s/k3s.yaml
|
|
environment:
|
|
INSTALL_K3S_VERSION: "{{ k3s_version }}"
|
|
K3S_TOKEN: "{{ k3s_join_token }}"
|
|
when: k3s_server_mode == 'secondary'
|
|
|
|
- name: Wait for the k3s API to be serving locally
|
|
ansible.builtin.wait_for:
|
|
host: "127.0.0.1"
|
|
port: "{{ k3s_api_port }}"
|
|
timeout: 180
|