From 6d62aca0d283542f01ed4023f5d76961bbf912ff Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 5 Sep 2026 11:08:54 -0500 Subject: [PATCH] =?UTF-8?q?feat(workstation):=20prod=20dnsmasq=20profile?= =?UTF-8?q?=20=E2=80=94=20and=20CORRECT=20the=20topology=20I=20had=20wrong?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit OPERATOR CORRECTION (2026-09-05). I had been calling the resolver that shadows prod "a pfSense override" throughout this repo. Both halves of that were wrong: - It is the WORKSTATION'S OWN dnsmasq, not pfSense. The live resolver is the one NetworkManager spawns (dns=dnsmasq, --conf-dir=/etc/NetworkManager/ dnsmasq.d); the standalone dnsmasq.service is INACTIVE, so /etc/dnsmasq.d is dormant. The dev profile's `address=/.home.keithjackson.dev/192.168.2.50` matches the domain AND EVERY SUBDOMAIN, so it also swallowed the whole new prod.home.* tree and answered it with dev's address. - The k3s prod cluster is NOT behind pfSense. It sits on the SOHO router network 192.168.2.0/24, which is pfSense's WAN; pfSense CONTAINS the dev LAN 192.168.1.0/24. This workstation is on 192.168.2.0/24 too, which is why the rules below use prod's real VIPs and need no NAT hop, unlike dev's .50. New infra/workstation/dnsmasq/ — authored, not click-ops: 20-prod.home.keithjackson.dev.conf prod.home.* -> 192.168.2.20 (Gateway), unifi -> .21, crafty-mc -> .22 (L4 VIPs) install.sh installs + reloads; refuses non-root dnsmasq resolves `address=` by LONGEST MATCH, so the prod rules win WITHOUT editing or reordering the dev profile. PROVEN, not assumed: ran a throwaway dnsmasq on 127.0.0.1:5354 loaded with both profiles — prod.home / gitea.prod / keycloak.prod -> 192.168.2.20 unifi.prod -> 192.168.2.21 crafty-mc.prod -> 192.168.2.22 gitea.home / home.keithjackson.dev -> 192.168.2.50 (DEV, UNCHANGED) `dnsmasq --test` on the new file: syntax OK. install.sh pins the destination because getting it wrong is SILENT: a file in /etc/dnsmasq.d exists, looks installed, and changes nothing. It also uses `systemctl reload NetworkManager` rather than SIGHUP — dnsmasq does not re-read its config on SIGHUP, so a HUP would look like it worked and not apply anything. WHY THE MISLABEL MATTERED: the trap PARTIALLY works. Dev serves a *.home.keithjackson.dev wildcard cert, which matches prod.home.keithjackson.dev (one label) so the prod apex returns 200 over VALID TLS, while keycloak.prod.home.keithjackson.dev (two labels) fails with a certificate-name error. It reads as a broken prod cert; it is the wrong server answering. I made exactly that misreading during verification. Also recorded, because it changes what the apex flip IS: the SOHO router NATs inbound 443 to the pfSense WAN (192.168.2.102) to serve the EXISTING NON-K3S production environment, which must stay live until the switch is deliberately made. So §10a sequence P is not just a k3s-side DNS/TLS change — it is the legacy-prod cutover, and P as written does NOT cover repointing that NAT. The two must be planned together or the apex resolves to k3s while inbound 443 still lands on the legacy box. Phase C touches none of it: prod's Gateway is 192.168.2.20:443 (a different address) and its cert is DNS-01, needing no inbound 443. And the MinIO timeout is now correctly described as expected topology — crossing INTO pfSense from its WAN — not a misconfiguration. VERIFIED: full CI gate PASS (spec-immutability, lint, l0, render-golden, l1). Refs R-DNS-02, R-ENV-02. Co-Authored-By: Claude Opus 5 --- bootstrap/standup.sh | 6 +- clusters/prod/kustomization.yaml | 6 +- infra/PROD-STANDUP.md | 77 +++++++++++++++---- .../20-prod.home.keithjackson.dev.conf | 49 ++++++++++++ infra/workstation/dnsmasq/install.sh | 48 ++++++++++++ 5 files changed, 168 insertions(+), 18 deletions(-) create mode 100644 infra/workstation/dnsmasq/20-prod.home.keithjackson.dev.conf create mode 100755 infra/workstation/dnsmasq/install.sh diff --git a/bootstrap/standup.sh b/bootstrap/standup.sh index 66f3eff..b7f9e3e 100755 --- a/bootstrap/standup.sh +++ b/bootstrap/standup.sh @@ -109,7 +109,8 @@ wait_ready() { # } # Snapshot the Cloudflare zone. This is how damage to the OTHER cluster's records becomes detectable. -# Queried against the Cloudflare API, never a local resolver: a pfSense override produced a false +# Queried against the Cloudflare API, never a local resolver: the workstation's own dnsmasq wildcard +# (address=/.home.keithjackson.dev/) answers the whole tree locally and produced a false # healthy reading during the 2026-09-04 incident and hid the damage for a while. dns_snapshot() { # local out="$1" tok zid page tp @@ -395,7 +396,8 @@ step_externaldns() { step_s3check() { say "S3 BACKUP ENDPOINT — reachability FROM INSIDE THE CLUSTER" # Velero runs in-cluster, so the only reachability that counts is the pods' own. Probing from the - # workstation proves nothing either way: it sits on a different side of pfSense than the pod + # workstation proves nothing either way: prod sits on the SOHO network (pfSense's WAN) while the + # dev LAN's MinIO sits inside pfSense, so host and pod egress differ; a healthy reading there # network, and a healthy result there has already misled this project once. local ep live; live="$(cvar S3_BACKUP_ENDPOINT)" info "live S3_BACKUP_ENDPOINT = ${live:-}" diff --git a/clusters/prod/kustomization.yaml b/clusters/prod/kustomization.yaml index ccae77d..f93420d 100644 --- a/clusters/prod/kustomization.yaml +++ b/clusters/prod/kustomization.yaml @@ -239,7 +239,7 @@ patches: # tiers are SUSPENDED for exactly this reason. The gate is not "review is done", it is: # (a) dev is reconciling on *.dev.home.keithjackson.dev with a Ready wildcard cert, AND # (b) dev's old apex A/CNAME records and their TXT registry entries are GONE from Cloudflare - # (verified against Cloudflare directly — a pfSense override is NOT evidence about public + # (verified against Cloudflare directly — the workstation's dnsmasq is NOT evidence about public # DNS; that mistake was made on 2026-09-04), AND # (c) dev's old `*.home.keithjackson.dev` cert/Certificate is released, because prod ordering # that same name set while dev's still exists counts against LE's duplicate-certificate @@ -393,7 +393,9 @@ patches: # *** NOT YET REACHABLE — velero and the CNPG backups CANNOT be enabled on prod today. *** # Measured 2026-09-05 from a throwaway pod INSIDE the prod cluster (not from the workstation, # which proves nothing about pod egress): `curl http://192.168.1.9:9000/minio/health/live` - # → connection timed out after 15s. pfSense does not route prod's pod network to the dev LAN's + # → connection timed out after 15s. EXPECTED TOPOLOGY: prod is NOT behind pfSense — it is on the + # SOHO network 192.168.2.0/24, which is pfSense's WAN, while MinIO sits inside pfSense's LAN + # 192.168.1.0/24. Reaching it means crossing INTO pfSense from its WAN, which needs a rule # MinIO. Until that routing exists, Velero and CNPG would install cleanly and fail every # backup at RUNTIME — a tier that reports healthy while no backup exists. # Re-test with: bootstrap/standup.sh ... --s3-endpoint http://192.168.1.9:9000 s3check diff --git a/infra/PROD-STANDUP.md b/infra/PROD-STANDUP.md index ba76121..61c4208 100644 --- a/infra/PROD-STANDUP.md +++ b/infra/PROD-STANDUP.md @@ -238,27 +238,54 @@ prod.home.keithjackson.dev 200 keycloak.… 302 (login redirect) gitea.prod.home.keithjackson.dev 200 grafana.… 302 (SSO redirect) ``` -### ACTION REQUIRED — the pfSense override shadows prod from the LAN +### Workstation DNS — install the prod dnsmasq profile -`getent hosts` on the workstation returns **192.168.2.50** (dev) for **every** `*.home.keithjackson.dev` -name — and the new `prod.home.…` subtree falls inside that wildcard. Cloudflare correctly returns -`192.168.2.20`. So **prod is reachable from anywhere using public DNS, but from this workstation and -anything else behind pfSense, prod hostnames resolve to dev.** +**CORRECTION (operator, 2026-09-05): this is the workstation's own dnsmasq, NOT pfSense.** Earlier +notes in this repo called it a "pfSense override"; that was wrong and the topology below is why. -Narrow the override so it does not cover `prod.home.keithjackson.dev`, or add a matching override -pointing that subtree at `192.168.2.20`. Until then, test prod with an explicit resolve: +`/etc/NetworkManager/dnsmasq.d/10-home.keithjackson.dev.conf` (the dev profile) contains: + +``` +address=/.home.keithjackson.dev/192.168.2.50 +``` + +dnsmasq's `address=` matches the named domain **and every subdomain**, so that one line also captures +the whole `prod.home.keithjackson.dev` tree and answers it with **dev's** address, while Cloudflare +correctly returns prod's `192.168.2.20`. + +**Fix — one command, already authored in this repo:** + +```bash +sudo infra/workstation/dnsmasq/install.sh +``` + +It installs `20-prod.home.keithjackson.dev.conf` and reloads the live resolver. dnsmasq resolves +`address=` by **longest match**, so the prod rules win without editing or reordering the dev profile. +Verified against a throwaway dnsmasq loaded with both profiles: + +| query | answer | +|---|---| +| `prod.home.keithjackson.dev`, `gitea.prod…`, `keycloak.prod…` | `192.168.2.20` (Gateway) | +| `unifi.prod.home.keithjackson.dev` | `192.168.2.21` (L4) | +| `crafty-mc.prod.home.keithjackson.dev` | `192.168.2.22` (L4) | +| `gitea.home.keithjackson.dev`, `home.keithjackson.dev` | `192.168.2.50` — **dev, unchanged** | + +> **Install to `/etc/NetworkManager/dnsmasq.d/`, never `/etc/dnsmasq.d/`.** The standalone +> `dnsmasq.service` is **inactive**; the live resolver is the one NetworkManager spawns +> (`dns=dnsmasq`) with `--conf-dir=/etc/NetworkManager/dnsmasq.d`. A file in the other directory +> changes nothing and looks like it should. `install.sh` pins the destination for exactly this reason. + +**Until it is installed, verify prod only with an explicit resolve** — and note *why* this trap is +convincing: dev serves a `*.home.keithjackson.dev` wildcard cert, which matches +`prod.home.keithjackson.dev` (one label), so the prod **apex returns 200 over valid TLS** while +`keycloak.prod.home.keithjackson.dev` (two labels) fails with a certificate-name error. It reads like +a broken prod certificate. It is the wrong server answering. ```bash curl -sS --resolve gitea.prod.home.keithjackson.dev:443:192.168.2.20 \ https://gitea.prod.home.keithjackson.dev/ ``` -**This is also the trap that produced a false healthy reading during the 2026-09-04 incident.** It is -subtle because the *apex* appears to work: dev's `*.home.keithjackson.dev` wildcard matches -`prod.home.keithjackson.dev` (one label) so TLS succeeds, while `keycloak.prod.home.keithjackson.dev` -(two labels) fails with a certificate-name error. A partial success is what makes it convincing. -**Never verify DNS or TLS through a local resolver here.** - ## Blockers ### 1. Backups: endpoint DECIDED, but prod cannot reach it — **prod has no backup coverage** @@ -273,9 +300,20 @@ from the workstation, which proves nothing about pod egress: curl http://192.168.1.9:9000/minio/health/live → connection timed out after 15s ``` -pfSense does not route prod's pod network to the dev LAN's MinIO. **Operator action required.** +**This is expected topology, not a misconfiguration** (operator, 2026-09-05): the k3s prod cluster is +**not behind pfSense**. It sits on the SOHO router network `192.168.2.0/24`, which is pfSense's +**WAN** side; pfSense *contains* `192.168.1.0/24`, the dev LAN where MinIO lives. Prod reaching +`192.168.1.9` therefore means crossing **into** pfSense from its WAN — which requires a deliberate +pfSense rule or port-forward that does not exist. **Operator decision required**, and worth weighing: +a prod backup that lands inside the dev LAN is arguably not off-cluster DR at all. + Re-test with `standup.sh --s3-endpoint http://192.168.1.9:9000 s3check` — it fails closed and says so. +> **Do not confuse this with the SOHO router's inbound NAT.** That router forwards inbound **443** to +> the pfSense WAN address `192.168.2.102` to serve the **existing, non-k3s production environment**, +> which must stay live on 443 until the k3s cutover is deliberately made. Nothing in this runbook +> touches that NAT, and prod's Gateway on `192.168.2.20:443` does not contend with it. + Until that routing exists, `velero` stays suspended and **`apps-wave-0` still applies GREEN** — the token *is* defined, so strict substitution succeeds — while every write to that endpoint fails at **runtime**. A green `apps-wave-0` does not mean backups work. **Treat prod as having no backup @@ -320,6 +358,17 @@ nothing, and standing it up first is strictly better: it proves the whole stack the cutover becomes a DNS/TLS change on a known-good cluster instead of a bring-up and a cutover at once. The apex move is `infra/PROD-PROVISIONING.md` §10a, sequence **D** (dev vacates) then **P**. +> **THE APEX FLIP IS ALSO THE LEGACY-PROD CUTOVER — they are the same event.** The SOHO router NATs +> inbound **443** to the pfSense WAN address `192.168.2.102`, serving the **existing, non-k3s +> production environment**, which the operator requires to stay live until the switch is deliberately +> made. So §10a sequence **P** is not merely a DNS/TLS change on the k3s side: taking +> `home.keithjackson.dev` means public traffic must also stop resolving to / being forwarded to the +> legacy stack. Sequence **P** as written does **not** cover repointing that NAT, and the two must be +> planned together or the apex will resolve to k3s while inbound 443 still lands on the legacy box. +> +> Nothing in phase C touches it: prod's Gateway is `192.168.2.20:443`, a different address from the +> NAT target, and prod's wildcard cert is issued by **DNS-01**, which needs no inbound 443 at all. + --- ## Not covered here diff --git a/infra/workstation/dnsmasq/20-prod.home.keithjackson.dev.conf b/infra/workstation/dnsmasq/20-prod.home.keithjackson.dev.conf new file mode 100644 index 0000000..df3ed8d --- /dev/null +++ b/infra/workstation/dnsmasq/20-prod.home.keithjackson.dev.conf @@ -0,0 +1,49 @@ +# Workstation split-horizon DNS for the k3s PROD cluster. +# +# INSTALL TO: /etc/NetworkManager/dnsmasq.d/20-prod.home.keithjackson.dev.conf +# NOT /etc/dnsmasq.d/ — the standalone dnsmasq.service is INACTIVE on this workstation. The live +# resolver is the one NetworkManager spawns (NetworkManager.conf `dns=dnsmasq`), and it reads +# ONLY --conf-dir=/etc/NetworkManager/dnsmasq.d. A file dropped in /etc/dnsmasq.d/ changes nothing +# and looks like it should — verify with: pgrep -a dnsmasq +# +# WHY THIS FILE EXISTS +# The dev profile (10-home.keithjackson.dev.conf) contains: +# address=/.home.keithjackson.dev/192.168.2.50 +# dnsmasq's `address=` matches the named domain AND EVERY SUBDOMAIN, so that one line also +# captures the entire `prod.home.keithjackson.dev` tree and answers it with DEV's address. Every +# prod hostname therefore resolved to dev from this workstation, while Cloudflare correctly +# returned prod's gateway. +# +# That failure is deceptive because it PARTIALLY works: dev serves a `*.home.keithjackson.dev` +# wildcard certificate, which matches `prod.home.keithjackson.dev` (one label) — so the prod apex +# returns 200 over valid TLS — but NOT `keycloak.prod.home.keithjackson.dev` (two labels), which +# fails with a certificate-name error. It reads like a broken prod certificate. It is not; it is +# the wrong server answering. NEVER verify prod's DNS or TLS through this workstation's resolver +# without confirming these rules are installed. +# +# dnsmasq resolves `address=` by LONGEST MATCH, so the more specific rules below win over the dev +# line without editing or reordering it. The dev profile stays untouched. +# +# TOPOLOGY (corrected 2026-09-05 by the operator — the earlier "pfSense override" framing was wrong) +# The k3s prod cluster is NOT behind pfSense. It sits on the SOHO router network, +# 192.168.2.0/24, which is pfSense's WAN side; pfSense contains 192.168.1.0/24 (the dev LAN). +# This workstation is on 192.168.2.0/24 too, so it reaches prod's VIPs DIRECTLY — no NAT hop, +# which is why these addresses are prod's real VIPs and not a forwarded address like dev's +# 192.168.2.50. +# +# The SOHO router NATs inbound 443 to the pfSense WAN address (192.168.2.102) to serve the +# EXISTING, NON-K3S production environment. That NAT is untouched by this file: these rules are +# local to this workstation's resolver only and change nothing about inbound traffic. The legacy +# prod stays live on 443 until the k3s cutover is deliberately made. + +# The prod cluster's shared ingress Gateway (Cilium LB-IPAM; == CILIUM_LB_POOL_START). +# Covers the apex and every service hostname not overridden below. +address=/prod.home.keithjackson.dev/192.168.2.20 + +# L4 (non-HTTP) services get their OWN LoadBalancer VIPs, so they must NOT be sent to the Gateway. +# These are longer matches than the line above and therefore take precedence. +address=/unifi.prod.home.keithjackson.dev/192.168.2.21 +address=/crafty-mc.prod.home.keithjackson.dev/192.168.2.22 + +# NOTE: no `local=` line is needed. The dev profile's `local=/home.keithjackson.dev/` already stops +# the whole tree — including prod.home.* — from being forwarded upstream. diff --git a/infra/workstation/dnsmasq/install.sh b/infra/workstation/dnsmasq/install.sh new file mode 100755 index 0000000..9cb91db --- /dev/null +++ b/infra/workstation/dnsmasq/install.sh @@ -0,0 +1,48 @@ +#!/usr/bin/env bash +# +# install.sh — install this repo's workstation dnsmasq profiles and reload the live resolver. +# +# Needs root (writes /etc/NetworkManager/dnsmasq.d). Run it as: +# sudo infra/workstation/dnsmasq/install.sh +# +# WHY A SCRIPT AND NOT A DOC LINE: these rules decide which CLUSTER this workstation talks to. Getting +# them into the wrong directory is silent — the file exists, looks installed, and changes nothing — +# so the destination and the reload are pinned here rather than retyped. Idempotent; safe to re-run. +# +set -euo pipefail + +src_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +# THE LIVE RESOLVER IS NETWORKMANAGER'S dnsmasq, not the standalone service. NetworkManager.conf sets +# `dns=dnsmasq`, and the process it spawns runs with `--conf-file=/dev/null` plus +# `--conf-dir=/etc/NetworkManager/dnsmasq.d` — so /etc/dnsmasq.d is read ONLY by the standalone +# dnsmasq.service, which is inactive here. Installing to the wrong one is the failure this pins down. +dest_dir=/etc/NetworkManager/dnsmasq.d + +[ "$(id -u)" = "0" ] || { echo "ERROR: must run as root — try: sudo $0" >&2; exit 1; } + +if ! pgrep -af 'dnsmasq.*NetworkManager' >/dev/null 2>&1; then + echo "WARN: no NetworkManager-spawned dnsmasq is running." >&2 + echo " Check 'dns=dnsmasq' in /etc/NetworkManager/NetworkManager.conf before trusting this." >&2 +fi + +install -d -m 0755 "${dest_dir}" +shopt -s nullglob +count=0 +for f in "${src_dir}"/*.conf; do + install -m 0644 "$f" "${dest_dir}/$(basename "$f")" + echo "installed: ${dest_dir}/$(basename "$f")" + count=$((count+1)) +done +[ "${count}" -gt 0 ] || { echo "ERROR: no .conf files found in ${src_dir}" >&2; exit 1; } + +# dnsmasq does NOT re-read its config on SIGHUP (SIGHUP only reloads /etc/hosts and the caches), so a +# reload of NetworkManager — which respawns dnsmasq — is what actually applies these files. +echo "reloading NetworkManager (respawns dnsmasq with the new config)" +systemctl reload NetworkManager || systemctl restart NetworkManager + +echo +echo "verify (expect prod's own VIPs, NOT dev's 192.168.2.50):" +echo " getent hosts gitea.prod.home.keithjackson.dev # -> 192.168.2.20" +echo " getent hosts unifi.prod.home.keithjackson.dev # -> 192.168.2.21" +echo " getent hosts gitea.home.keithjackson.dev # -> 192.168.2.50 (dev, unchanged)"