From 788eaeae62728736fc0e791c4742c724b6ecccc9 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 3 Oct 2026 17:38:16 -0500 Subject: [PATCH] fix(node-baseline): pin the OIDC issuer to the Gateway VIP on prod nodes After public DNS moved to the WAN IP (#40), prod nodes resolve keycloak.home.keithjackson.dev to a WAN address the router does not hairpin, so the apiserver could not reach its OIDC issuer. Adds an opt-in node_baseline_hosts_pins /etc/hosts block; prod pins keycloak to GATEWAY_VIP. Co-Authored-By: Claude Opus 5.5 --- .../2026-10-03-node-keycloak-hosts-pin.md | 45 +++++++++++++++++++ .../inventory/group_vars/k3s_prod/vars.yml | 30 ++++++++----- .../roles/node_baseline/defaults/main.yml | 6 +++ .../roles/node_baseline/tasks/main.yml | 19 ++++++++ 4 files changed, 89 insertions(+), 11 deletions(-) create mode 100644 decisions/2026-10-03-node-keycloak-hosts-pin.md diff --git a/decisions/2026-10-03-node-keycloak-hosts-pin.md b/decisions/2026-10-03-node-keycloak-hosts-pin.md new file mode 100644 index 0000000..1a28c2c --- /dev/null +++ b/decisions/2026-10-03-node-keycloak-hosts-pin.md @@ -0,0 +1,45 @@ +--- +date: 2026-10-03 +branch: fix/node-keycloak-hosts-pin +title: Pin the OIDC issuer to the Gateway VIP on prod nodes — the WAN answer does not hairpin +--- + +## What changed + +Follow-up to PR #40 (`decisions/2026-10-03-public-dns-wan-target.md`). Public DNS for +`*.home.keithjackson.dev` now CNAMEs to the WAN IP. The prod nodes resolve through public DNS, +and the router does not hairpin, so the kube-apiserver could no longer reach its OIDC issuer +`keycloak.home.keithjackson.dev`. + +- `roles/node_baseline`: a new `node_baseline_hosts_pins` list (default `[]`) and one + `blockinfile` task (tags `dns`, `hosts`). It writes the pins into `/etc/hosts` as a single + marked block. An empty list removes the block. +- `group_vars/k3s_prod/vars.yml`: pins `keycloak.{{ k3s_base_domain }}` → `192.168.2.20`, which + must stay equal to `GATEWAY_VIP` (parity comment added). It also replaces the stale "no + /etc/hosts pin is needed" paragraph with the measured reason the pin is now required. + +Dev is unaffected (empty default). + +## What I verified + +- After PR #40 reconciled (`platform` lastAppliedRevision `576e1ab`, Healthy=True), on + k3s-prod-2 and k3s-prod-3: `getent hosts keycloak.home.keithjackson.dev` → + `136.51.115.252 wan.home.keithjackson.dev`. `curl https://keycloak…/realms/homelab/.well-known/openid-configuration` + → exit 60, `ssl_verify_result=18` (the router's self-signed cert, not the Gateway). +- `journalctl -u k3s` (last 20 min) showed no OIDC/JWKS errors yet. The apiserver still holds + the keys it cached before the flip, so the failure is latent until a k3s restart or key rotation. +- Neither node has cloud-init (`/etc/cloud` absent), so nothing rewrites `/etc/hosts` at boot. +- `--check --diff --tags hosts --limit k3s-prod-2,k3s-prod-3`: each node gets exactly the + three-line marked block `192.168.2.20 keycloak.home.keithjackson.dev`. +- Outside-in, PR #40 works: check-host.net probes from 3 external nodes got `200 OK` from + `https://plex.home.keithjackson.dev/identity` at `136.51.115.252`. + +## What I did NOT verify + +- **k3s-prod-1**: Ansible cannot reach it. SSH host key verification fails (`known_hosts:94`, + ED25519 `SHA256:zgUS8OeWznzCX9hlN7XdZ8iLNkU1G2loXjtJFfEUYpE`), which is consistent with + the 2026-09-11 SSD replacement and reinstall. The operator must confirm the fingerprint + before the pin can be applied there. +- The playbook has not been applied for real. It is an operator run after merge: + `ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. +- An end-to-end kubectl OIDC login after the pin (needs a browser). diff --git a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml index 41883e1..4813196 100644 --- a/infra/ansible/inventory/group_vars/k3s_prod/vars.yml +++ b/infra/ansible/inventory/group_vars/k3s_prod/vars.yml @@ -306,19 +306,27 @@ node_baseline_dns_servers: # k3s_base_domain, below). It reaches the issuer through the NODE resolver, i.e. through # these PUBLIC upstreams — not through CoreDNS's split-horizon answer. # - # THIS USED TO BE BROKEN, AND THE 2026-09-04 SUBDOMAIN SPLIT FIXED IT. While prod was - # authored to serve dev's hostnames, a prod node resolving keycloak.home.keithjackson.dev - # publicly got DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN, so - # kubectl OIDC SSO on prod could not work at all. Now prod's issuer is - # keycloak.prod.home.keithjackson.dev, and the public record for that name is published by - # PROD's own external-dns pointing at PROD's Gateway VIP (192.168.2.20). The public answer - # and the split-horizon answer therefore AGREE, and no /etc/hosts pin is needed. + # THE PUBLIC ANSWER IS NOW THE WAN, SO THE ISSUER IS PINNED (2026-10-03). Since + # platform/gateway/gateway.yaml set an external-dns `target`, every public name is a CNAME + # to wan.home.keithjackson.dev = the home WAN IP. The SOHO router does NOT hairpin: from the + # LAN, WAN:443 answers with the router's own self-signed placeholder cert (measured from + # k3s-prod-2/-3 2026-10-03: `curl` exit 60, ssl_verify_result=18). The apiserver would then + # be unable to fetch its issuer's discovery/JWKS, so kubectl OIDC SSO would fail on the + # next k3s restart or Keycloak key rotation. node_baseline_hosts_pins (below) pins the + # issuer host to the Gateway VIP in /etc/hosts, which glibc consults before these + # resolvers. (History: before the 2026-09-04 subdomain split the public answer was DEV's + # VIP; after it, the public answer was prod's own VIP and no pin was needed.) # - # STILL TO VERIFY on prod bring-up (PROD-PROVISIONING.md §10): this holds only once prod's - # external-dns has actually published the record. Until then the issuer does not resolve - # from the nodes and kubectl SSO fails — the cert-based admin kubeconfig is the break-glass - # path, and it always works. Ordering, not a design gap. + # The cert-based admin kubeconfig never depends on any of this. It is the break-glass path. node_baseline_dns_search: [] +# /etc/hosts pins: node-process names that must reach the in-cluster Gateway directly rather +# than via the public (WAN, non-hairpinned) answer. See "THE PUBLIC ANSWER IS NOW THE WAN" above. +# [parity] the IP MUST equal GATEWAY_VIP in clusters/prod/kustomization.yaml (192.168.2.20), +# and the name MUST follow k3s_oidc_issuer_url's host (roles/k3s_server/defaults/main.yml). +node_baseline_hosts_pins: + - ip: "192.168.2.20" + names: + - "keycloak.{{ k3s_base_domain }}" # --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) -------------------- # PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/ diff --git a/infra/ansible/roles/node_baseline/defaults/main.yml b/infra/ansible/roles/node_baseline/defaults/main.yml index c2a438f..d03016c 100644 --- a/infra/ansible/roles/node_baseline/defaults/main.yml +++ b/infra/ansible/roles/node_baseline/defaults/main.yml @@ -242,3 +242,9 @@ node_baseline_dns_search: node_baseline_dns_options: - timeout:2 - attempts:2 +# Static /etc/hosts entries (tags: dns, hosts), as a list of {ip, names}. EMPTY by default: +# dev needs none. Prod pins its OIDC issuer host to the Gateway VIP because the public +# answer for it is the non-hairpinned WAN IP (inventory/group_vars/k3s_prod/vars.yml). +# Written as ONE marked block, so removing an entry here removes it from the node, and +# an empty list removes the block entirely. +node_baseline_hosts_pins: [] diff --git a/infra/ansible/roles/node_baseline/tasks/main.yml b/infra/ansible/roles/node_baseline/tasks/main.yml index 11c914f..1422573 100644 --- a/infra/ansible/roles/node_baseline/tasks/main.yml +++ b/infra/ansible/roles/node_baseline/tasks/main.yml @@ -1090,3 +1090,22 @@ {% endif %} when: node_baseline_dns_mechanism == 'resolv-conf-file' tags: [dns] + +# /etc/hosts pins — tag `dns` (and `hosts` on its own). Some node processes must reach a +# public hostname at the in-cluster Gateway, not at its public answer. On prod, the +# kube-apiserver's OIDC issuer resolves publicly to the WAN IP, which the router does not +# hairpin. glibc reads /etc/hosts before any nameserver (nsswitch `files dns`), so a pin wins +# regardless of which resolver mechanism above is in effect. One marked block: entries +# dropped from node_baseline_hosts_pins disappear from the node, and an empty list +# removes the block. +- name: Pin node-process hostnames in /etc/hosts + ansible.builtin.blockinfile: + path: /etc/hosts + marker: "# {mark} node_baseline hosts pins (Ansible, tags: hosts)" + block: | + {% for pin in node_baseline_hosts_pins %} + {{ pin.ip }} {{ pin.names | join(' ') }} + {% endfor %} + state: "{{ 'present' if node_baseline_hosts_pins | length > 0 else 'absent' }}" + backup: true + tags: [dns, hosts]