jakman85andClaude Opus 5
60e3f09c67
`[vpn_egress_gateway_prod]` is a SEPARATE inventory group with its own vaulted credentials, not a second member of `[vpn_egress_gateway]`. That group is the default target of playbooks/vpn.egress.gateway.yml, so adding a prod host to it would mean a flagless `ansible-playbook playbooks/vpn.egress.gateway.yml` silently reconfigures a prod node's kill-switch datapath. Prod is reached only by asking for it, with `--list-hosts` first: -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3 Node 3 because node 1's SSD was replaced yesterday (least operational history) and node 2 carries the §0b memattr history. Exactly one node, forever: Cilium OSS picks "the first node in lexical ordering" among nodeSelector matches, so a labelled k3s-prod-1 would silently steal the gateway role — and the public egress IP — from k3s-prod-3, with no event and no alert. vault.yml is a verbatim copy of the dev group's CIPHERTEXT: same PIA account per the operator, and copying the encrypted blob means the plaintext was never decrypted to disk, through a shell, or into a transcript. The two files are now independent — rotating means re-encrypting both, which is the right behaviour for two environments. Also corrects a stale inventory note that claimed k3s-prod-3 has wlp1s0 up on 192.168.2.195. Re-verified before choosing it as the gateway, because a second live uplink in a kill-switch datapath is not a footnote: there is no wl* netdev on the node at all, no phy80211 device, the /etc/network/interfaces wlp1s0 stanza is commented out, 192.168.2.195 is assigned nowhere, and the only IPv4 addresses are lo and eno1 192.168.2.12. Nothing needed bringing down. Co-Authored-By: Claude Opus 5 <[email protected]>
161 lines
10 KiB
INI
161 lines
10 KiB
INI
# Static inventory — k3s node IPs ALIGNED with infra/tofu defaults (Proxmox migration IPs),
|
|
# but the SSH target is the pfSense NAT (see NAT HAND-OFF below) — this file is
|
|
# HAND-MAINTAINED and must NOT be regenerated from `tofu output -raw ansible_inventory`.
|
|
#
|
|
# k3s_primary = the --cluster-init server (embedded-etcd founder + Cilium k8sServiceHost).
|
|
# k3s_secondary = the two servers that join it. All three are servers (HA embedded etcd).
|
|
#
|
|
# NAT HAND-OFF (LIVE): the nodes live on the Proxmox LAN at 192.168.1.50/.51/.52, and
|
|
# Ansible reaches them over the pfSense NAT at 192.168.2.102, port-forwarded as
|
|
# 2250 -> 192.168.1.50:22, 2251 -> 192.168.1.51:22, 2252 -> 192.168.1.52:22.
|
|
# Therefore `ansible_host`/`ansible_port` below are the NAT SSH target (192.168.2.102:225x),
|
|
# and each node's REAL 192.168.1.x lives in the `k3s_node_ip` var — consumed for the k3s
|
|
# --node-ip, the API/tls-san endpoint (k3s_api_host), and general in-cluster parity.
|
|
# This file is HAND-MAINTAINED for the NAT: do NOT blindly regenerate it from
|
|
# tofu output -raw ansible_inventory
|
|
# because tofu has no knowledge of the pfSense NAT and would overwrite the mapping below.
|
|
|
|
[k3s_primary]
|
|
k3s-dev-1 ansible_host=192.168.2.102 ansible_port=2250 k3s_node_ip=192.168.1.50
|
|
|
|
[k3s_secondary]
|
|
k3s-dev-2 ansible_host=192.168.2.102 ansible_port=2251 k3s_node_ip=192.168.1.51
|
|
k3s-dev-3 ansible_host=192.168.2.102 ansible_port=2252 k3s_node_ip=192.168.1.52
|
|
|
|
[k3s_servers:children]
|
|
k3s_primary
|
|
k3s_secondary
|
|
|
|
# The hypervisor/workstation host itself (target of host-prereqs.yml). Its become
|
|
# (sudo) password comes from the vaulted group_vars/hypervisor/vault.yml.
|
|
[hypervisor]
|
|
localhost ansible_connection=local
|
|
|
|
# VPN egress-gateway node — runs the node-level split-tunnel OpenVPN + kill-switch
|
|
# (ADR-0015, playbooks/vpn.egress.gateway.yml). EXACTLY ONE host belongs here.
|
|
#
|
|
# THIS GROUP IS SINGLETON BY DESIGN — a second host does NOT buy failover. Cilium OSS has no
|
|
# egress-gateway failover: the CRD's own `egressGateway` description says "In case multiple nodes
|
|
# are a match for the given set of labels, the first node in lexical ordering based on their name
|
|
# will be selected". Two labelled nodes therefore means one silently wins by NAME ORDER (a labelled
|
|
# `k3s-prod-1` beats a labelled `k3s-prod-3`) and the other carries no traffic. See
|
|
# platform/vpn-egress/README.md "Resilience — there is no gateway failover" for the full statement
|
|
# and the ADR-0016 escalation. Members are already defined above with their ansible_host under
|
|
# k3s_secondary; this group only tags the egress role target.
|
|
[vpn_egress_gateway]
|
|
k3s-dev-3
|
|
|
|
# PROD's egress-gateway node — a SEPARATE group on purpose (2026-09-12).
|
|
#
|
|
# It is NOT a member of [vpn_egress_gateway] above, and k3s-prod-3 must never be added
|
|
# there: that group is the DEFAULT target of playbooks/vpn.egress.gateway.yml, so a
|
|
# flagless `ansible-playbook playbooks/vpn.egress.gateway.yml` would then silently
|
|
# reconfigure a prod node's kill-switch datapath. Prod is reached only by asking for it:
|
|
#
|
|
# ansible-playbook playbooks/vpn.egress.gateway.yml \
|
|
# -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3
|
|
#
|
|
# (`--list-hosts` first, every time. The play's target var is
|
|
# `vpn_egress_gateway_hosts`, NOT the cluster-wide `k3s_hosts`.)
|
|
#
|
|
# WHY NODE 3, AND WHY EXACTLY ONE:
|
|
# * node 1's SSD was replaced 2026-09-11 and it has the least operational history;
|
|
# * node 2 carries the §0b memattr history;
|
|
# * node 3 has a single uplink (eno1 192.168.2.12 — its wireless is gone: no wl* netdev
|
|
# exists, the /etc/network/interfaces wlp1s0 stanza is commented out, and the
|
|
# 192.168.2.195 address the earlier note below claimed is NOT present. Verified
|
|
# 2026-09-12) which is what a kill-switch datapath wants.
|
|
# * LEXICAL ORDERING HAZARD: Cilium OSS picks "the first node in lexical ordering" when
|
|
# several match the policy's nodeSelector, so labelling k3s-prod-1 would SILENTLY take
|
|
# the gateway role from k3s-prod-3. Exactly one host here, forever.
|
|
# Its PIA credentials are its own vaulted group_vars/vpn_egress_gateway_prod/vault.yml —
|
|
# the dev group's vault does not apply to a prod host.
|
|
[vpn_egress_gateway_prod]
|
|
k3s-prod-3
|
|
|
|
# The Proxmox HYPERVISOR host itself (NOT a k3s node) — target of playbooks/backup-s3.yml,
|
|
# which stands up the off-cluster versitygw S3 backup target for Velero + CNPG. Reached DIRECTLY
|
|
# over SSH as root (its own 192.168.2.102:22, the pfSense NAT to the host's 192.168.1.9), so
|
|
# it overrides the cluster-wide `ansible_user: k3s` from group_vars/all.yml. Distinct from the
|
|
# `hypervisor` group above (that is `localhost` for host-prereqs.yml, a different machine).
|
|
[proxmox_hosts]
|
|
proxmox ansible_host=192.168.2.102 ansible_port=22 ansible_user=root ansible_ssh_private_key_file=~/.ssh/home_network
|
|
|
|
# =============================================================================
|
|
# PROD (k3s-prod-*) — Phase 7 PREP ONLY. See infra/PROD-PROVISIONING.md (T-7.1,
|
|
# especially §5/§6) and DECISIONS-LOG.md "PHASE 7 OPENED" (2026-08-31). ALL THREE
|
|
# nodes are now on the bench and SSH-reachable (`k3s-prod-1` 2026-08-31, `k3s-prod-2`
|
|
# 2026-09-01, `k3s-prod-3` 2026-09-03) — every HP EliteDesk 800 G5 Mini. This section
|
|
# is still inventory/group_vars scaffolding only: nothing beyond the read-only health
|
|
# check and the node-baseline PACKAGE floor has been run, and the operator has decided
|
|
# the CLUSTER stays uninitialized (no k3s install, no --cluster-init) until the §2 disk
|
|
# decision is closed (decision #2 in the DECISIONS-LOG entry above) — no
|
|
# single-server-etcd window, no later member-add dance.
|
|
#
|
|
# KEY DIFFERENCE FROM DEV: prod sits DIRECTLY on the 192.168.2.x LAN — no pfSense NAT
|
|
# hop in front of it. Dev's `ansible_host` above is a NAT target (192.168.2.102:225x)
|
|
# that port-forwards to the node's real 192.168.1.x address, which is why dev also
|
|
# carries a separate `k3s_node_ip` per host. Prod needs no such split: `ansible_host`
|
|
# below IS already the node's real, routable address, so `k3s_node_ip` is set to the
|
|
# SAME value (kept explicit, not omitted, only so the k3s_server role's hostvar lookup
|
|
# — see the GAP note below — has something correct to find once that gap is closed).
|
|
# `ansible_port` is the node's own real sshd (22). See group_vars/k3s_prod/vars.yml for
|
|
# the k3s_api_host override that reflects this directly-reachable topology.
|
|
#
|
|
# GROUP-NAME GAP: CLOSED 2026-08-31. The roles and playbooks used to hardcode the
|
|
# LITERAL dev groups (`hosts: k3s_servers` / `k3s_primary` / `k3s_secondary`, and
|
|
# `groups['k3s_primary']` inside roles/k3s_server + roles/cilium), so nothing could
|
|
# select the k3s_prod_* groups below. Both halves are now variables, deliberately kept
|
|
# separate because Ansible resolves them at different times:
|
|
# * WHICH HOSTS A PLAY RUNS ON — `k3s_hosts` (whole-cluster plays), `k3s_primary_hosts`
|
|
# / `k3s_secondary_hosts` (k3s.yml's split), each defaulting to the dev group.
|
|
# EXTRA VARS ONLY: a play's `hosts:` is templated before any host exists, so
|
|
# group_vars cannot be read there. Prod: `-e k3s_hosts=k3s_prod` etc.
|
|
# * WHICH GROUP A ROLE MEANS BY "MY PRIMARY"/"MY SERVERS" — `k3s_primary_group` /
|
|
# `k3s_servers_group`, set in group_vars/all.yml (dev) and group_vars/k3s_prod/vars.yml
|
|
# (prod), so roles resolve their own environment with NO per-run flag — the same
|
|
# mechanism k3s_api_host already used.
|
|
# Running with no extra vars is byte-for-byte the dev behaviour it always had. See
|
|
# infra/ansible/README.md "Targeting prod" and PROD-PROVISIONING.md §6.
|
|
# =============================================================================
|
|
|
|
[k3s_prod_primary]
|
|
# FOUNDER MOVED TO NODE 2 (2026-09-07). Node 1 founded this cluster with --cluster-init,
|
|
# then its SSD was replaced and Debian reinstalled, so it is now a JOINING node like any
|
|
# other. This group decides two things at once: which play sets k3s_server_mode: primary
|
|
# (-> --cluster-init, which FOUNDS A NEW CLUSTER) and what k3s_api_host resolves to (the
|
|
# --server join target). Leaving node 1 here would have pointed a rebuilt, empty node at
|
|
# ITSELF and founded a second cluster beside the live one. Node 2 is the senior surviving
|
|
# server; its own --cluster-init is inert because /etc/rancher/k3s/k3s.yaml already exists.
|
|
k3s-prod-2 ansible_host=192.168.2.11 ansible_port=22 k3s_node_ip=192.168.2.11
|
|
|
|
[k3s_prod_secondary]
|
|
k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10
|
|
# Uncomment a node only once it is actually on the bench and answering SSH, so no play can
|
|
# target a host that isn't there. k3s-prod-2 came online 2026-09-01 (imaged identically to
|
|
# node 1: same HP EliteDesk 800 G5 Mini, same BIOS R21 02.26.00, same i7-9700T, same
|
|
# partition layout) but with a SAMSUNG PM9A1 NVMe instead of node 1's 2011 SATA C400 — the
|
|
# nodes are NOT hardware-symmetric, which matters for the §2 Longhorn/etcd disk decision.
|
|
# k3s-prod-3 came online 2026-09-03 at the planned 192.168.2.12 — same HP EliteDesk 800 G5
|
|
# Mini, same BIOS R21 02.26.00, same i7-9700T, 2x16 GB (a MATCHED Crucial pair, unlike node
|
|
# 1's mismatched Microns), full 31.1 GiB usable. It is NOT structurally immune to the §0b
|
|
# memattr bug: its firmware DOES publish a MAT (MEMATTR=0xbeece018) and the kernel merely
|
|
# happened to parse it cleanly this boot, so node 3 is on the same lottery as node 1 and
|
|
# needs the same >= 6.13 kernel. Its disk is a THIRD distinct model (WDC WDS500G2B0A SATA
|
|
# SSD) — all three nodes differ on disk, so the §2 Longhorn/etcd symmetry gap is now
|
|
# three-way, not a node-1 outlier.
|
|
# NOTE (CORRECTED 2026-09-12): this used to read "node 3 also has wlp1s0 up on
|
|
# 192.168.2.195 (wifi)". It does not — re-verified before making node 3 the VPN egress
|
|
# gateway, because a second live uplink in a kill-switch datapath is not a detail: there is
|
|
# NO wl* netdev on the node at all, no phy80211 device, the wlp1s0 stanza in
|
|
# /etc/network/interfaces is commented out, 192.168.2.195 is assigned nowhere, and the only
|
|
# IPv4 addresses are lo and eno1 192.168.2.12. `ansible_host` is that WIRED eno1 address —
|
|
# the same directly-routable 192.168.2.x LAN as nodes 1 and 2.
|
|
k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12
|
|
|
|
# Umbrella group group_vars/k3s_prod/ attaches to (mirrors dev's [k3s_servers:children]
|
|
# pattern; named to match the group_vars directory so Ansible auto-loads it).
|
|
[k3s_prod:children]
|
|
k3s_prod_primary
|
|
k3s_prod_secondary
|