jakman85andClaude Opus 5.5
788eaeae62
After public DNS moved to the WAN IP (#40), prod nodes resolve keycloak.home.keithjackson.dev to a WAN address the router does not hairpin, so the apiserver could not reach its OIDC issuer. Adds an opt-in node_baseline_hosts_pins /etc/hosts block; prod pins keycloak to GATEWAY_VIP. Co-Authored-By: Claude Opus 5.5 <[email protected]>
46 lines
2.5 KiB
Markdown
46 lines
2.5 KiB
Markdown
---
|
|
date: 2026-10-03
|
|
branch: fix/node-keycloak-hosts-pin
|
|
title: Pin the OIDC issuer to the Gateway VIP on prod nodes — the WAN answer does not hairpin
|
|
---
|
|
|
|
## What changed
|
|
|
|
Follow-up to PR #40 (`decisions/2026-10-03-public-dns-wan-target.md`). Public DNS for
|
|
`*.home.keithjackson.dev` now CNAMEs to the WAN IP. The prod nodes resolve through public DNS,
|
|
and the router does not hairpin, so the kube-apiserver could no longer reach its OIDC issuer
|
|
`keycloak.home.keithjackson.dev`.
|
|
|
|
- `roles/node_baseline`: a new `node_baseline_hosts_pins` list (default `[]`) and one
|
|
`blockinfile` task (tags `dns`, `hosts`). It writes the pins into `/etc/hosts` as a single
|
|
marked block. An empty list removes the block.
|
|
- `group_vars/k3s_prod/vars.yml`: pins `keycloak.{{ k3s_base_domain }}` → `192.168.2.20`, which
|
|
must stay equal to `GATEWAY_VIP` (parity comment added). It also replaces the stale "no
|
|
/etc/hosts pin is needed" paragraph with the measured reason the pin is now required.
|
|
|
|
Dev is unaffected (empty default).
|
|
|
|
## What I verified
|
|
|
|
- After PR #40 reconciled (`platform` lastAppliedRevision `576e1ab`, Healthy=True), on
|
|
k3s-prod-2 and k3s-prod-3: `getent hosts keycloak.home.keithjackson.dev` →
|
|
`136.51.115.252 wan.home.keithjackson.dev`. `curl https://keycloak…/realms/homelab/.well-known/openid-configuration`
|
|
→ exit 60, `ssl_verify_result=18` (the router's self-signed cert, not the Gateway).
|
|
- `journalctl -u k3s` (last 20 min) showed no OIDC/JWKS errors yet. The apiserver still holds
|
|
the keys it cached before the flip, so the failure is latent until a k3s restart or key rotation.
|
|
- Neither node has cloud-init (`/etc/cloud` absent), so nothing rewrites `/etc/hosts` at boot.
|
|
- `--check --diff --tags hosts --limit k3s-prod-2,k3s-prod-3`: each node gets exactly the
|
|
three-line marked block `192.168.2.20 keycloak.home.keithjackson.dev`.
|
|
- Outside-in, PR #40 works: check-host.net probes from 3 external nodes got `200 OK` from
|
|
`https://plex.home.keithjackson.dev/identity` at `136.51.115.252`.
|
|
|
|
## What I did NOT verify
|
|
|
|
- **k3s-prod-1**: Ansible cannot reach it. SSH host key verification fails (`known_hosts:94`,
|
|
ED25519 `SHA256:zgUS8OeWznzCX9hlN7XdZ8iLNkU1G2loXjtJFfEUYpE`), which is consistent with
|
|
the 2026-09-11 SSD replacement and reinstall. The operator must confirm the fingerprint
|
|
before the pin can be applied there.
|
|
- The playbook has not been applied for real. It is an operator run after merge:
|
|
`ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`.
|
|
- An end-to-end kubectl OIDC login after the pin (needs a browser).
|