jakman85andClaude Opus 5.5
788eaeae62
After public DNS moved to the WAN IP (#40), prod nodes resolve keycloak.home.keithjackson.dev to a WAN address the router does not hairpin, so the apiserver could not reach its OIDC issuer. Adds an opt-in node_baseline_hosts_pins /etc/hosts block; prod pins keycloak to GATEWAY_VIP. Co-Authored-By: Claude Opus 5.5 <[email protected]>
2.5 KiB
2.5 KiB
date, branch, title
| date | branch | title |
|---|---|---|
| 2026-10-03 | fix/node-keycloak-hosts-pin | Pin the OIDC issuer to the Gateway VIP on prod nodes — the WAN answer does not hairpin |
What changed
Follow-up to PR #40 (decisions/2026-10-03-public-dns-wan-target.md). Public DNS for
*.home.keithjackson.dev now CNAMEs to the WAN IP. The prod nodes resolve through public DNS,
and the router does not hairpin, so the kube-apiserver could no longer reach its OIDC issuer
keycloak.home.keithjackson.dev.
roles/node_baseline: a newnode_baseline_hosts_pinslist (default[]) and oneblockinfiletask (tagsdns,hosts). It writes the pins into/etc/hostsas a single marked block. An empty list removes the block.group_vars/k3s_prod/vars.yml: pinskeycloak.{{ k3s_base_domain }}→192.168.2.20, which must stay equal toGATEWAY_VIP(parity comment added). It also replaces the stale "no /etc/hosts pin is needed" paragraph with the measured reason the pin is now required.
Dev is unaffected (empty default).
What I verified
- After PR #40 reconciled (
platformlastAppliedRevision576e1ab, Healthy=True), on k3s-prod-2 and k3s-prod-3:getent hosts keycloak.home.keithjackson.dev→136.51.115.252 wan.home.keithjackson.dev.curl https://keycloak…/realms/homelab/.well-known/openid-configuration→ exit 60,ssl_verify_result=18(the router's self-signed cert, not the Gateway). journalctl -u k3s(last 20 min) showed no OIDC/JWKS errors yet. The apiserver still holds the keys it cached before the flip, so the failure is latent until a k3s restart or key rotation.- Neither node has cloud-init (
/etc/cloudabsent), so nothing rewrites/etc/hostsat boot. --check --diff --tags hosts --limit k3s-prod-2,k3s-prod-3: each node gets exactly the three-line marked block192.168.2.20 keycloak.home.keithjackson.dev.- Outside-in, PR #40 works: check-host.net probes from 3 external nodes got
200 OKfromhttps://plex.home.keithjackson.dev/identityat136.51.115.252.
What I did NOT verify
- k3s-prod-1: Ansible cannot reach it. SSH host key verification fails (
known_hosts:94, ED25519SHA256:zgUS8OeWznzCX9hlN7XdZ8iLNkU1G2loXjtJFfEUYpE), which is consistent with the 2026-09-11 SSD replacement and reinstall. The operator must confirm the fingerprint before the pin can be applied there. - The playbook has not been applied for real. It is an operator run after merge:
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts. - An end-to-end kubectl OIDC login after the pin (needs a browser).