feat(workstation): prod dnsmasq profile — and CORRECT the topology I had wrong

OPERATOR CORRECTION (2026-09-05). I had been calling the resolver that shadows prod "a pfSense override" throughout this repo. Both halves of that were wrong: - It is the WORKSTATION'S OWN dnsmasq, not pfSense. The live resolver is the one NetworkManager spawns (dns=dnsmasq, --conf-dir=/etc/NetworkManager/ dnsmasq.d); the standalone dnsmasq.service is INACTIVE, so /etc/dnsmasq.d is dormant. The dev profile's `address=/.home.keithjackson.dev/192.168.2.50` matches the domain AND EVERY SUBDOMAIN, so it also swallowed the whole new prod.home.* tree and answered it with dev's address. - The k3s prod cluster is NOT behind pfSense. It sits on the SOHO router network 192.168.2.0/24, which is pfSense's WAN; pfSense CONTAINS the dev LAN 192.168.1.0/24. This workstation is on 192.168.2.0/24 too, which is why the rules below use prod's real VIPs and need no NAT hop, unlike dev's .50. New infra/workstation/dnsmasq/ — authored, not click-ops: 20-prod.home.keithjackson.dev.conf prod.home.* -> 192.168.2.20 (Gateway), unifi -> .21, crafty-mc -> .22 (L4 VIPs) install.sh installs + reloads; refuses non-root dnsmasq resolves `address=` by LONGEST MATCH, so the prod rules win WITHOUT editing or reordering the dev profile. PROVEN, not assumed: ran a throwaway dnsmasq on 127.0.0.1:5354 loaded with both profiles — prod.home / gitea.prod / keycloak.prod -> 192.168.2.20 unifi.prod -> 192.168.2.21 crafty-mc.prod -> 192.168.2.22 gitea.home / home.keithjackson.dev -> 192.168.2.50 (DEV, UNCHANGED) `dnsmasq --test` on the new file: syntax OK. install.sh pins the destination because getting it wrong is SILENT: a file in /etc/dnsmasq.d exists, looks installed, and changes nothing. It also uses `systemctl reload NetworkManager` rather than SIGHUP — dnsmasq does not re-read its config on SIGHUP, so a HUP would look like it worked and not apply anything. WHY THE MISLABEL MATTERED: the trap PARTIALLY works. Dev serves a *.home.keithjackson.dev wildcard cert, which matches prod.home.keithjackson.dev (one label) so the prod apex returns 200 over VALID TLS, while keycloak.prod.home.keithjackson.dev (two labels) fails with a certificate-name error. It reads as a broken prod cert; it is the wrong server answering. I made exactly that misreading during verification. Also recorded, because it changes what the apex flip IS: the SOHO router NATs inbound 443 to the pfSense WAN (192.168.2.102) to serve the EXISTING NON-K3S production environment, which must stay live until the switch is deliberately made. So §10a sequence P is not just a k3s-side DNS/TLS change — it is the legacy-prod cutover, and P as written does NOT cover repointing that NAT. The two must be planned together or the apex resolves to k3s while inbound 443 still lands on the legacy box. Phase C touches none of it: prod's Gateway is 192.168.2.20:443 (a different address) and its cert is DNS-01, needing no inbound 443. And the MinIO timeout is now correctly described as expected topology — crossing INTO pfSense from its WAN — not a misconfiguration. VERIFIED: full CI gate PASS (spec-immutability, lint, l0, render-golden, l1). Refs R-DNS-02, R-ENV-02. Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-05 11:08:54 -05:00
1 parent b80e8eef82
commit 6d62aca0d2
@@ -238,27 +238,54 @@ prod.home.keithjackson.dev            200     keycloak.…  302 (login redirect)
 gitea.prod.home.keithjackson.dev      200     grafana.…   302 (SSO redirect)
 ```

-### ACTION REQUIRED — the pfSense override shadows prod from the LAN
+### Workstation DNS — install the prod dnsmasq profile

-`getent hosts` on the workstation returns **192.168.2.50** (dev) for **every** `*.home.keithjackson.dev`
-name — and the new `prod.home.…` subtree falls inside that wildcard. Cloudflare correctly returns
-`192.168.2.20`. So **prod is reachable from anywhere using public DNS, but from this workstation and
-anything else behind pfSense, prod hostnames resolve to dev.**
+**CORRECTION (operator, 2026-09-05): this is the workstation's own dnsmasq, NOT pfSense.** Earlier
+notes in this repo called it a "pfSense override"; that was wrong and the topology below is why.

-Narrow the override so it does not cover `prod.home.keithjackson.dev`, or add a matching override
-pointing that subtree at `192.168.2.20`. Until then, test prod with an explicit resolve:
+`/etc/NetworkManager/dnsmasq.d/10-home.keithjackson.dev.conf` (the dev profile) contains:
+
+```
+address=/.home.keithjackson.dev/192.168.2.50
+```
+
+dnsmasq's `address=` matches the named domain **and every subdomain**, so that one line also captures
+the whole `prod.home.keithjackson.dev` tree and answers it with **dev's** address, while Cloudflare
+correctly returns prod's `192.168.2.20`.
+
+**Fix — one command, already authored in this repo:**
+
+```bash
+sudo infra/workstation/dnsmasq/install.sh
+```
+
+It installs `20-prod.home.keithjackson.dev.conf` and reloads the live resolver. dnsmasq resolves
+`address=` by **longest match**, so the prod rules win without editing or reordering the dev profile.
+Verified against a throwaway dnsmasq loaded with both profiles:
+
+| query | answer |
+|---|---|
+| `prod.home.keithjackson.dev`, `gitea.prod…`, `keycloak.prod…` | `192.168.2.20` (Gateway) |
+| `unifi.prod.home.keithjackson.dev` | `192.168.2.21` (L4) |
+| `crafty-mc.prod.home.keithjackson.dev` | `192.168.2.22` (L4) |
+| `gitea.home.keithjackson.dev`, `home.keithjackson.dev` | `192.168.2.50` — **dev, unchanged** |
+
+> **Install to `/etc/NetworkManager/dnsmasq.d/`, never `/etc/dnsmasq.d/`.** The standalone
+> `dnsmasq.service` is **inactive**; the live resolver is the one NetworkManager spawns
+> (`dns=dnsmasq`) with `--conf-dir=/etc/NetworkManager/dnsmasq.d`. A file in the other directory
+> changes nothing and looks like it should. `install.sh` pins the destination for exactly this reason.
+
+**Until it is installed, verify prod only with an explicit resolve** — and note *why* this trap is
+convincing: dev serves a `*.home.keithjackson.dev` wildcard cert, which matches
+`prod.home.keithjackson.dev` (one label), so the prod **apex returns 200 over valid TLS** while
+`keycloak.prod.home.keithjackson.dev` (two labels) fails with a certificate-name error. It reads like
+a broken prod certificate. It is the wrong server answering.

 ```bash
 curl -sS --resolve gitea.prod.home.keithjackson.dev:443:192.168.2.20 \
  https://gitea.prod.home.keithjackson.dev/
 ```

-**This is also the trap that produced a false healthy reading during the 2026-09-04 incident.** It is
-subtle because the *apex* appears to work: dev's `*.home.keithjackson.dev` wildcard matches
-`prod.home.keithjackson.dev` (one label) so TLS succeeds, while `keycloak.prod.home.keithjackson.dev`
-(two labels) fails with a certificate-name error. A partial success is what makes it convincing.
-**Never verify DNS or TLS through a local resolver here.**
-
 ## Blockers

 ### 1. Backups: endpoint DECIDED, but prod cannot reach it — **prod has no backup coverage**
@@ -273,9 +300,20 @@ from the workstation, which proves nothing about pod egress:
 curl http://192.168.1.9:9000/minio/health/live   →  connection timed out after 15s
 ```

-pfSense does not route prod's pod network to the dev LAN's MinIO. **Operator action required.**
+**This is expected topology, not a misconfiguration** (operator, 2026-09-05): the k3s prod cluster is
+**not behind pfSense**. It sits on the SOHO router network `192.168.2.0/24`, which is pfSense's
+**WAN** side; pfSense *contains* `192.168.1.0/24`, the dev LAN where MinIO lives. Prod reaching
+`192.168.1.9` therefore means crossing **into** pfSense from its WAN — which requires a deliberate
+pfSense rule or port-forward that does not exist. **Operator decision required**, and worth weighing:
+a prod backup that lands inside the dev LAN is arguably not off-cluster DR at all.
+
 Re-test with `standup.sh --s3-endpoint http://192.168.1.9:9000 s3check` — it fails closed and says so.

+> **Do not confuse this with the SOHO router's inbound NAT.** That router forwards inbound **443** to
+> the pfSense WAN address `192.168.2.102` to serve the **existing, non-k3s production environment**,
+> which must stay live on 443 until the k3s cutover is deliberately made. Nothing in this runbook
+> touches that NAT, and prod's Gateway on `192.168.2.20:443` does not contend with it.
+
 Until that routing exists, `velero` stays suspended and **`apps-wave-0` still applies GREEN** — the
 token *is* defined, so strict substitution succeeds — while every write to that endpoint fails at
 **runtime**. A green `apps-wave-0` does not mean backups work. **Treat prod as having no backup
@@ -320,6 +358,17 @@ nothing, and standing it up first is strictly better: it proves the whole stack
 the cutover becomes a DNS/TLS change on a known-good cluster instead of a bring-up and a cutover at
 once. The apex move is `infra/PROD-PROVISIONING.md` §10a, sequence **D** (dev vacates) then **P**.

+> **THE APEX FLIP IS ALSO THE LEGACY-PROD CUTOVER — they are the same event.** The SOHO router NATs
+> inbound **443** to the pfSense WAN address `192.168.2.102`, serving the **existing, non-k3s
+> production environment**, which the operator requires to stay live until the switch is deliberately
+> made. So §10a sequence **P** is not merely a DNS/TLS change on the k3s side: taking
+> `home.keithjackson.dev` means public traffic must also stop resolving to / being forwarded to the
+> legacy stack. Sequence **P** as written does **not** cover repointing that NAT, and the two must be
+> planned together or the apex will resolve to k3s while inbound 443 still lands on the legacy box.
+>
+> Nothing in phase C touches it: prod's Gateway is `192.168.2.20:443`, a different address from the
+> NAT target, and prod's wildcard cert is issued by **DNS-01**, which needs no inbound 443 at all.
+
 ---

 ## Not covered here
@@ -0,0 +1,49 @@
+# Workstation split-horizon DNS for the k3s PROD cluster.
+#
+# INSTALL TO: /etc/NetworkManager/dnsmasq.d/20-prod.home.keithjackson.dev.conf
+#   NOT /etc/dnsmasq.d/ — the standalone dnsmasq.service is INACTIVE on this workstation. The live
+#   resolver is the one NetworkManager spawns (NetworkManager.conf `dns=dnsmasq`), and it reads
+#   ONLY --conf-dir=/etc/NetworkManager/dnsmasq.d. A file dropped in /etc/dnsmasq.d/ changes nothing
+#   and looks like it should — verify with: pgrep -a dnsmasq
+#
+# WHY THIS FILE EXISTS
+#   The dev profile (10-home.keithjackson.dev.conf) contains:
+#       address=/.home.keithjackson.dev/192.168.2.50
+#   dnsmasq's `address=` matches the named domain AND EVERY SUBDOMAIN, so that one line also
+#   captures the entire `prod.home.keithjackson.dev` tree and answers it with DEV's address. Every
+#   prod hostname therefore resolved to dev from this workstation, while Cloudflare correctly
+#   returned prod's gateway.
+#
+#   That failure is deceptive because it PARTIALLY works: dev serves a `*.home.keithjackson.dev`
+#   wildcard certificate, which matches `prod.home.keithjackson.dev` (one label) — so the prod apex
+#   returns 200 over valid TLS — but NOT `keycloak.prod.home.keithjackson.dev` (two labels), which
+#   fails with a certificate-name error. It reads like a broken prod certificate. It is not; it is
+#   the wrong server answering. NEVER verify prod's DNS or TLS through this workstation's resolver
+#   without confirming these rules are installed.
+#
+#   dnsmasq resolves `address=` by LONGEST MATCH, so the more specific rules below win over the dev
+#   line without editing or reordering it. The dev profile stays untouched.
+#
+# TOPOLOGY (corrected 2026-09-05 by the operator — the earlier "pfSense override" framing was wrong)
+#   The k3s prod cluster is NOT behind pfSense. It sits on the SOHO router network,
+#   192.168.2.0/24, which is pfSense's WAN side; pfSense contains 192.168.1.0/24 (the dev LAN).
+#   This workstation is on 192.168.2.0/24 too, so it reaches prod's VIPs DIRECTLY — no NAT hop,
+#   which is why these addresses are prod's real VIPs and not a forwarded address like dev's
+#   192.168.2.50.
+#
+#   The SOHO router NATs inbound 443 to the pfSense WAN address (192.168.2.102) to serve the
+#   EXISTING, NON-K3S production environment. That NAT is untouched by this file: these rules are
+#   local to this workstation's resolver only and change nothing about inbound traffic. The legacy
+#   prod stays live on 443 until the k3s cutover is deliberately made.
+
+# The prod cluster's shared ingress Gateway (Cilium LB-IPAM; == CILIUM_LB_POOL_START).
+# Covers the apex and every service hostname not overridden below.
+address=/prod.home.keithjackson.dev/192.168.2.20
+
+# L4 (non-HTTP) services get their OWN LoadBalancer VIPs, so they must NOT be sent to the Gateway.
+# These are longer matches than the line above and therefore take precedence.
+address=/unifi.prod.home.keithjackson.dev/192.168.2.21
+address=/crafty-mc.prod.home.keithjackson.dev/192.168.2.22
+
+# NOTE: no `local=` line is needed. The dev profile's `local=/home.keithjackson.dev/` already stops
+# the whole tree — including prod.home.* — from being forwarded upstream.
@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+#
+# install.sh — install this repo's workstation dnsmasq profiles and reload the live resolver.
+#
+# Needs root (writes /etc/NetworkManager/dnsmasq.d). Run it as:
+#     sudo infra/workstation/dnsmasq/install.sh
+#
+# WHY A SCRIPT AND NOT A DOC LINE: these rules decide which CLUSTER this workstation talks to. Getting
+# them into the wrong directory is silent — the file exists, looks installed, and changes nothing —
+# so the destination and the reload are pinned here rather than retyped. Idempotent; safe to re-run.
+#
+set -euo pipefail
+
+src_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+# THE LIVE RESOLVER IS NETWORKMANAGER'S dnsmasq, not the standalone service. NetworkManager.conf sets
+# `dns=dnsmasq`, and the process it spawns runs with `--conf-file=/dev/null` plus
+# `--conf-dir=/etc/NetworkManager/dnsmasq.d` — so /etc/dnsmasq.d is read ONLY by the standalone
+# dnsmasq.service, which is inactive here. Installing to the wrong one is the failure this pins down.
+dest_dir=/etc/NetworkManager/dnsmasq.d
+
+[ "$(id -u)" = "0" ] || { echo "ERROR: must run as root — try: sudo $0" >&2; exit 1; }
+
+if ! pgrep -af 'dnsmasq.*NetworkManager' >/dev/null 2>&1; then
+  echo "WARN: no NetworkManager-spawned dnsmasq is running." >&2
+  echo "      Check 'dns=dnsmasq' in /etc/NetworkManager/NetworkManager.conf before trusting this." >&2
+fi
+
+install -d -m 0755 "${dest_dir}"
+shopt -s nullglob
+count=0
+for f in "${src_dir}"/*.conf; do
+  install -m 0644 "$f" "${dest_dir}/$(basename "$f")"
+  echo "installed: ${dest_dir}/$(basename "$f")"
+  count=$((count+1))
+done
+[ "${count}" -gt 0 ] || { echo "ERROR: no .conf files found in ${src_dir}" >&2; exit 1; }
+
+# dnsmasq does NOT re-read its config on SIGHUP (SIGHUP only reloads /etc/hosts and the caches), so a
+# reload of NetworkManager — which respawns dnsmasq — is what actually applies these files.
+echo "reloading NetworkManager (respawns dnsmasq with the new config)"
+systemctl reload NetworkManager || systemctl restart NetworkManager
+
+echo
+echo "verify (expect prod's own VIPs, NOT dev's 192.168.2.50):"
+echo "  getent hosts gitea.prod.home.keithjackson.dev     # -> 192.168.2.20"
+echo "  getent hosts unifi.prod.home.keithjackson.dev     # -> 192.168.2.21"
+echo "  getent hosts gitea.home.keithjackson.dev          # -> 192.168.2.50  (dev, unchanged)"