feat(workstation): prod dnsmasq profile — and CORRECT the topology I had wrong
OPERATOR CORRECTION (2026-09-05). I had been calling the resolver that shadows
prod "a pfSense override" throughout this repo. Both halves of that were wrong:
- It is the WORKSTATION'S OWN dnsmasq, not pfSense. The live resolver is the
one NetworkManager spawns (dns=dnsmasq, --conf-dir=/etc/NetworkManager/
dnsmasq.d); the standalone dnsmasq.service is INACTIVE, so /etc/dnsmasq.d is
dormant. The dev profile's `address=/.home.keithjackson.dev/192.168.2.50`
matches the domain AND EVERY SUBDOMAIN, so it also swallowed the whole new
prod.home.* tree and answered it with dev's address.
- The k3s prod cluster is NOT behind pfSense. It sits on the SOHO router
network 192.168.2.0/24, which is pfSense's WAN; pfSense CONTAINS the dev LAN
192.168.1.0/24. This workstation is on 192.168.2.0/24 too, which is why the
rules below use prod's real VIPs and need no NAT hop, unlike dev's .50.
New infra/workstation/dnsmasq/ — authored, not click-ops:
20-prod.home.keithjackson.dev.conf prod.home.* -> 192.168.2.20 (Gateway),
unifi -> .21, crafty-mc -> .22 (L4 VIPs)
install.sh installs + reloads; refuses non-root
dnsmasq resolves `address=` by LONGEST MATCH, so the prod rules win WITHOUT
editing or reordering the dev profile. PROVEN, not assumed: ran a throwaway
dnsmasq on 127.0.0.1:5354 loaded with both profiles —
prod.home / gitea.prod / keycloak.prod -> 192.168.2.20
unifi.prod -> 192.168.2.21 crafty-mc.prod -> 192.168.2.22
gitea.home / home.keithjackson.dev -> 192.168.2.50 (DEV, UNCHANGED)
`dnsmasq --test` on the new file: syntax OK.
install.sh pins the destination because getting it wrong is SILENT: a file in
/etc/dnsmasq.d exists, looks installed, and changes nothing. It also uses
`systemctl reload NetworkManager` rather than SIGHUP — dnsmasq does not re-read
its config on SIGHUP, so a HUP would look like it worked and not apply anything.
WHY THE MISLABEL MATTERED: the trap PARTIALLY works. Dev serves a
*.home.keithjackson.dev wildcard cert, which matches prod.home.keithjackson.dev
(one label) so the prod apex returns 200 over VALID TLS, while
keycloak.prod.home.keithjackson.dev (two labels) fails with a certificate-name
error. It reads as a broken prod cert; it is the wrong server answering. I made
exactly that misreading during verification.
Also recorded, because it changes what the apex flip IS: the SOHO router NATs
inbound 443 to the pfSense WAN (192.168.2.102) to serve the EXISTING NON-K3S
production environment, which must stay live until the switch is deliberately
made. So §10a sequence P is not just a k3s-side DNS/TLS change — it is the
legacy-prod cutover, and P as written does NOT cover repointing that NAT. The
two must be planned together or the apex resolves to k3s while inbound 443 still
lands on the legacy box. Phase C touches none of it: prod's Gateway is
192.168.2.20:443 (a different address) and its cert is DNS-01, needing no
inbound 443. And the MinIO timeout is now correctly described as expected
topology — crossing INTO pfSense from its WAN — not a misconfiguration.
VERIFIED: full CI gate PASS (spec-immutability, lint, l0, render-golden, l1).
Refs R-DNS-02, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
b80e8eef82
commit
6d62aca0d2
5 files changed
+168
-18
No files matched your search
@@ -109,7 +109,8 @@ wait_ready() { # <kustomization> <timeout-seconds>
|
||||
}
|
||||
|
||||
# Snapshot the Cloudflare zone. This is how damage to the OTHER cluster's records becomes detectable.
|
||||
# Queried against the Cloudflare API, never a local resolver: a pfSense override produced a false
|
||||
# Queried against the Cloudflare API, never a local resolver: the workstation's own dnsmasq wildcard
|
||||
# (address=/.home.keithjackson.dev/) answers the whole tree locally and produced a false
|
||||
# healthy reading during the 2026-09-04 incident and hid the damage for a while.
|
||||
dns_snapshot() { # <output-file>
|
||||
local out="$1" tok zid page tp
|
||||
@@ -395,7 +396,8 @@ step_externaldns() {
|
||||
step_s3check() {
|
||||
say "S3 BACKUP ENDPOINT — reachability FROM INSIDE THE CLUSTER"
|
||||
# Velero runs in-cluster, so the only reachability that counts is the pods' own. Probing from the
|
||||
# workstation proves nothing either way: it sits on a different side of pfSense than the pod
|
||||
# workstation proves nothing either way: prod sits on the SOHO network (pfSense's WAN) while the
|
||||
# dev LAN's MinIO sits inside pfSense, so host and pod egress differ; a healthy reading there
|
||||
# network, and a healthy result there has already misled this project once.
|
||||
local ep live; live="$(cvar S3_BACKUP_ENDPOINT)"
|
||||
info "live S3_BACKUP_ENDPOINT = ${live:-<unset>}"
|
||||
|
||||
@@ -239,7 +239,7 @@ patches:
|
||||
# tiers are SUSPENDED for exactly this reason. The gate is not "review is done", it is:
|
||||
# (a) dev is reconciling on *.dev.home.keithjackson.dev with a Ready wildcard cert, AND
|
||||
# (b) dev's old apex A/CNAME records and their TXT registry entries are GONE from Cloudflare
|
||||
# (verified against Cloudflare directly — a pfSense override is NOT evidence about public
|
||||
# (verified against Cloudflare directly — the workstation's dnsmasq is NOT evidence about public
|
||||
# DNS; that mistake was made on 2026-09-04), AND
|
||||
# (c) dev's old `*.home.keithjackson.dev` cert/Certificate is released, because prod ordering
|
||||
# that same name set while dev's still exists counts against LE's duplicate-certificate
|
||||
@@ -393,7 +393,9 @@ patches:
|
||||
# *** NOT YET REACHABLE — velero and the CNPG backups CANNOT be enabled on prod today. ***
|
||||
# Measured 2026-09-05 from a throwaway pod INSIDE the prod cluster (not from the workstation,
|
||||
# which proves nothing about pod egress): `curl http://192.168.1.9:9000/minio/health/live`
|
||||
# → connection timed out after 15s. pfSense does not route prod's pod network to the dev LAN's
|
||||
# → connection timed out after 15s. EXPECTED TOPOLOGY: prod is NOT behind pfSense — it is on the
|
||||
# SOHO network 192.168.2.0/24, which is pfSense's WAN, while MinIO sits inside pfSense's LAN
|
||||
# 192.168.1.0/24. Reaching it means crossing INTO pfSense from its WAN, which needs a rule
|
||||
# MinIO. Until that routing exists, Velero and CNPG would install cleanly and fail every
|
||||
# backup at RUNTIME — a tier that reports healthy while no backup exists.
|
||||
# Re-test with: bootstrap/standup.sh ... --s3-endpoint http://192.168.1.9:9000 s3check
|
||||
|
||||
+63
-14
@@ -238,27 +238,54 @@ prod.home.keithjackson.dev 200 keycloak.… 302 (login redirect)
|
||||
gitea.prod.home.keithjackson.dev 200 grafana.… 302 (SSO redirect)
|
||||
```
|
||||
|
||||
### ACTION REQUIRED — the pfSense override shadows prod from the LAN
|
||||
### Workstation DNS — install the prod dnsmasq profile
|
||||
|
||||
`getent hosts` on the workstation returns **192.168.2.50** (dev) for **every** `*.home.keithjackson.dev`
|
||||
name — and the new `prod.home.…` subtree falls inside that wildcard. Cloudflare correctly returns
|
||||
`192.168.2.20`. So **prod is reachable from anywhere using public DNS, but from this workstation and
|
||||
anything else behind pfSense, prod hostnames resolve to dev.**
|
||||
**CORRECTION (operator, 2026-09-05): this is the workstation's own dnsmasq, NOT pfSense.** Earlier
|
||||
notes in this repo called it a "pfSense override"; that was wrong and the topology below is why.
|
||||
|
||||
Narrow the override so it does not cover `prod.home.keithjackson.dev`, or add a matching override
|
||||
pointing that subtree at `192.168.2.20`. Until then, test prod with an explicit resolve:
|
||||
`/etc/NetworkManager/dnsmasq.d/10-home.keithjackson.dev.conf` (the dev profile) contains:
|
||||
|
||||
```
|
||||
address=/.home.keithjackson.dev/192.168.2.50
|
||||
```
|
||||
|
||||
dnsmasq's `address=` matches the named domain **and every subdomain**, so that one line also captures
|
||||
the whole `prod.home.keithjackson.dev` tree and answers it with **dev's** address, while Cloudflare
|
||||
correctly returns prod's `192.168.2.20`.
|
||||
|
||||
**Fix — one command, already authored in this repo:**
|
||||
|
||||
```bash
|
||||
sudo infra/workstation/dnsmasq/install.sh
|
||||
```
|
||||
|
||||
It installs `20-prod.home.keithjackson.dev.conf` and reloads the live resolver. dnsmasq resolves
|
||||
`address=` by **longest match**, so the prod rules win without editing or reordering the dev profile.
|
||||
Verified against a throwaway dnsmasq loaded with both profiles:
|
||||
|
||||
| query | answer |
|
||||
|---|---|
|
||||
| `prod.home.keithjackson.dev`, `gitea.prod…`, `keycloak.prod…` | `192.168.2.20` (Gateway) |
|
||||
| `unifi.prod.home.keithjackson.dev` | `192.168.2.21` (L4) |
|
||||
| `crafty-mc.prod.home.keithjackson.dev` | `192.168.2.22` (L4) |
|
||||
| `gitea.home.keithjackson.dev`, `home.keithjackson.dev` | `192.168.2.50` — **dev, unchanged** |
|
||||
|
||||
> **Install to `/etc/NetworkManager/dnsmasq.d/`, never `/etc/dnsmasq.d/`.** The standalone
|
||||
> `dnsmasq.service` is **inactive**; the live resolver is the one NetworkManager spawns
|
||||
> (`dns=dnsmasq`) with `--conf-dir=/etc/NetworkManager/dnsmasq.d`. A file in the other directory
|
||||
> changes nothing and looks like it should. `install.sh` pins the destination for exactly this reason.
|
||||
|
||||
**Until it is installed, verify prod only with an explicit resolve** — and note *why* this trap is
|
||||
convincing: dev serves a `*.home.keithjackson.dev` wildcard cert, which matches
|
||||
`prod.home.keithjackson.dev` (one label), so the prod **apex returns 200 over valid TLS** while
|
||||
`keycloak.prod.home.keithjackson.dev` (two labels) fails with a certificate-name error. It reads like
|
||||
a broken prod certificate. It is the wrong server answering.
|
||||
|
||||
```bash
|
||||
curl -sS --resolve gitea.prod.home.keithjackson.dev:443:192.168.2.20 \
|
||||
https://gitea.prod.home.keithjackson.dev/
|
||||
```
|
||||
|
||||
**This is also the trap that produced a false healthy reading during the 2026-09-04 incident.** It is
|
||||
subtle because the *apex* appears to work: dev's `*.home.keithjackson.dev` wildcard matches
|
||||
`prod.home.keithjackson.dev` (one label) so TLS succeeds, while `keycloak.prod.home.keithjackson.dev`
|
||||
(two labels) fails with a certificate-name error. A partial success is what makes it convincing.
|
||||
**Never verify DNS or TLS through a local resolver here.**
|
||||
|
||||
## Blockers
|
||||
|
||||
### 1. Backups: endpoint DECIDED, but prod cannot reach it — **prod has no backup coverage**
|
||||
@@ -273,9 +300,20 @@ from the workstation, which proves nothing about pod egress:
|
||||
curl http://192.168.1.9:9000/minio/health/live → connection timed out after 15s
|
||||
```
|
||||
|
||||
pfSense does not route prod's pod network to the dev LAN's MinIO. **Operator action required.**
|
||||
**This is expected topology, not a misconfiguration** (operator, 2026-09-05): the k3s prod cluster is
|
||||
**not behind pfSense**. It sits on the SOHO router network `192.168.2.0/24`, which is pfSense's
|
||||
**WAN** side; pfSense *contains* `192.168.1.0/24`, the dev LAN where MinIO lives. Prod reaching
|
||||
`192.168.1.9` therefore means crossing **into** pfSense from its WAN — which requires a deliberate
|
||||
pfSense rule or port-forward that does not exist. **Operator decision required**, and worth weighing:
|
||||
a prod backup that lands inside the dev LAN is arguably not off-cluster DR at all.
|
||||
|
||||
Re-test with `standup.sh --s3-endpoint http://192.168.1.9:9000 s3check` — it fails closed and says so.
|
||||
|
||||
> **Do not confuse this with the SOHO router's inbound NAT.** That router forwards inbound **443** to
|
||||
> the pfSense WAN address `192.168.2.102` to serve the **existing, non-k3s production environment**,
|
||||
> which must stay live on 443 until the k3s cutover is deliberately made. Nothing in this runbook
|
||||
> touches that NAT, and prod's Gateway on `192.168.2.20:443` does not contend with it.
|
||||
|
||||
Until that routing exists, `velero` stays suspended and **`apps-wave-0` still applies GREEN** — the
|
||||
token *is* defined, so strict substitution succeeds — while every write to that endpoint fails at
|
||||
**runtime**. A green `apps-wave-0` does not mean backups work. **Treat prod as having no backup
|
||||
@@ -320,6 +358,17 @@ nothing, and standing it up first is strictly better: it proves the whole stack
|
||||
the cutover becomes a DNS/TLS change on a known-good cluster instead of a bring-up and a cutover at
|
||||
once. The apex move is `infra/PROD-PROVISIONING.md` §10a, sequence **D** (dev vacates) then **P**.
|
||||
|
||||
> **THE APEX FLIP IS ALSO THE LEGACY-PROD CUTOVER — they are the same event.** The SOHO router NATs
|
||||
> inbound **443** to the pfSense WAN address `192.168.2.102`, serving the **existing, non-k3s
|
||||
> production environment**, which the operator requires to stay live until the switch is deliberately
|
||||
> made. So §10a sequence **P** is not merely a DNS/TLS change on the k3s side: taking
|
||||
> `home.keithjackson.dev` means public traffic must also stop resolving to / being forwarded to the
|
||||
> legacy stack. Sequence **P** as written does **not** cover repointing that NAT, and the two must be
|
||||
> planned together or the apex will resolve to k3s while inbound 443 still lands on the legacy box.
|
||||
>
|
||||
> Nothing in phase C touches it: prod's Gateway is `192.168.2.20:443`, a different address from the
|
||||
> NAT target, and prod's wildcard cert is issued by **DNS-01**, which needs no inbound 443 at all.
|
||||
|
||||
---
|
||||
|
||||
## Not covered here
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
# Workstation split-horizon DNS for the k3s PROD cluster.
|
||||
#
|
||||
# INSTALL TO: /etc/NetworkManager/dnsmasq.d/20-prod.home.keithjackson.dev.conf
|
||||
# NOT /etc/dnsmasq.d/ — the standalone dnsmasq.service is INACTIVE on this workstation. The live
|
||||
# resolver is the one NetworkManager spawns (NetworkManager.conf `dns=dnsmasq`), and it reads
|
||||
# ONLY --conf-dir=/etc/NetworkManager/dnsmasq.d. A file dropped in /etc/dnsmasq.d/ changes nothing
|
||||
# and looks like it should — verify with: pgrep -a dnsmasq
|
||||
#
|
||||
# WHY THIS FILE EXISTS
|
||||
# The dev profile (10-home.keithjackson.dev.conf) contains:
|
||||
# address=/.home.keithjackson.dev/192.168.2.50
|
||||
# dnsmasq's `address=` matches the named domain AND EVERY SUBDOMAIN, so that one line also
|
||||
# captures the entire `prod.home.keithjackson.dev` tree and answers it with DEV's address. Every
|
||||
# prod hostname therefore resolved to dev from this workstation, while Cloudflare correctly
|
||||
# returned prod's gateway.
|
||||
#
|
||||
# That failure is deceptive because it PARTIALLY works: dev serves a `*.home.keithjackson.dev`
|
||||
# wildcard certificate, which matches `prod.home.keithjackson.dev` (one label) — so the prod apex
|
||||
# returns 200 over valid TLS — but NOT `keycloak.prod.home.keithjackson.dev` (two labels), which
|
||||
# fails with a certificate-name error. It reads like a broken prod certificate. It is not; it is
|
||||
# the wrong server answering. NEVER verify prod's DNS or TLS through this workstation's resolver
|
||||
# without confirming these rules are installed.
|
||||
#
|
||||
# dnsmasq resolves `address=` by LONGEST MATCH, so the more specific rules below win over the dev
|
||||
# line without editing or reordering it. The dev profile stays untouched.
|
||||
#
|
||||
# TOPOLOGY (corrected 2026-09-05 by the operator — the earlier "pfSense override" framing was wrong)
|
||||
# The k3s prod cluster is NOT behind pfSense. It sits on the SOHO router network,
|
||||
# 192.168.2.0/24, which is pfSense's WAN side; pfSense contains 192.168.1.0/24 (the dev LAN).
|
||||
# This workstation is on 192.168.2.0/24 too, so it reaches prod's VIPs DIRECTLY — no NAT hop,
|
||||
# which is why these addresses are prod's real VIPs and not a forwarded address like dev's
|
||||
# 192.168.2.50.
|
||||
#
|
||||
# The SOHO router NATs inbound 443 to the pfSense WAN address (192.168.2.102) to serve the
|
||||
# EXISTING, NON-K3S production environment. That NAT is untouched by this file: these rules are
|
||||
# local to this workstation's resolver only and change nothing about inbound traffic. The legacy
|
||||
# prod stays live on 443 until the k3s cutover is deliberately made.
|
||||
|
||||
# The prod cluster's shared ingress Gateway (Cilium LB-IPAM; == CILIUM_LB_POOL_START).
|
||||
# Covers the apex and every service hostname not overridden below.
|
||||
address=/prod.home.keithjackson.dev/192.168.2.20
|
||||
|
||||
# L4 (non-HTTP) services get their OWN LoadBalancer VIPs, so they must NOT be sent to the Gateway.
|
||||
# These are longer matches than the line above and therefore take precedence.
|
||||
address=/unifi.prod.home.keithjackson.dev/192.168.2.21
|
||||
address=/crafty-mc.prod.home.keithjackson.dev/192.168.2.22
|
||||
|
||||
# NOTE: no `local=` line is needed. The dev profile's `local=/home.keithjackson.dev/` already stops
|
||||
# the whole tree — including prod.home.* — from being forwarded upstream.
|
||||
Executable
+48
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# install.sh — install this repo's workstation dnsmasq profiles and reload the live resolver.
|
||||
#
|
||||
# Needs root (writes /etc/NetworkManager/dnsmasq.d). Run it as:
|
||||
# sudo infra/workstation/dnsmasq/install.sh
|
||||
#
|
||||
# WHY A SCRIPT AND NOT A DOC LINE: these rules decide which CLUSTER this workstation talks to. Getting
|
||||
# them into the wrong directory is silent — the file exists, looks installed, and changes nothing —
|
||||
# so the destination and the reload are pinned here rather than retyped. Idempotent; safe to re-run.
|
||||
#
|
||||
set -euo pipefail
|
||||
|
||||
src_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
# THE LIVE RESOLVER IS NETWORKMANAGER'S dnsmasq, not the standalone service. NetworkManager.conf sets
|
||||
# `dns=dnsmasq`, and the process it spawns runs with `--conf-file=/dev/null` plus
|
||||
# `--conf-dir=/etc/NetworkManager/dnsmasq.d` — so /etc/dnsmasq.d is read ONLY by the standalone
|
||||
# dnsmasq.service, which is inactive here. Installing to the wrong one is the failure this pins down.
|
||||
dest_dir=/etc/NetworkManager/dnsmasq.d
|
||||
|
||||
[ "$(id -u)" = "0" ] || { echo "ERROR: must run as root — try: sudo $0" >&2; exit 1; }
|
||||
|
||||
if ! pgrep -af 'dnsmasq.*NetworkManager' >/dev/null 2>&1; then
|
||||
echo "WARN: no NetworkManager-spawned dnsmasq is running." >&2
|
||||
echo " Check 'dns=dnsmasq' in /etc/NetworkManager/NetworkManager.conf before trusting this." >&2
|
||||
fi
|
||||
|
||||
install -d -m 0755 "${dest_dir}"
|
||||
shopt -s nullglob
|
||||
count=0
|
||||
for f in "${src_dir}"/*.conf; do
|
||||
install -m 0644 "$f" "${dest_dir}/$(basename "$f")"
|
||||
echo "installed: ${dest_dir}/$(basename "$f")"
|
||||
count=$((count+1))
|
||||
done
|
||||
[ "${count}" -gt 0 ] || { echo "ERROR: no .conf files found in ${src_dir}" >&2; exit 1; }
|
||||
|
||||
# dnsmasq does NOT re-read its config on SIGHUP (SIGHUP only reloads /etc/hosts and the caches), so a
|
||||
# reload of NetworkManager — which respawns dnsmasq — is what actually applies these files.
|
||||
echo "reloading NetworkManager (respawns dnsmasq with the new config)"
|
||||
systemctl reload NetworkManager || systemctl restart NetworkManager
|
||||
|
||||
echo
|
||||
echo "verify (expect prod's own VIPs, NOT dev's 192.168.2.50):"
|
||||
echo " getent hosts gitea.prod.home.keithjackson.dev # -> 192.168.2.20"
|
||||
echo " getent hosts unifi.prod.home.keithjackson.dev # -> 192.168.2.21"
|
||||
echo " getent hosts gitea.home.keithjackson.dev # -> 192.168.2.50 (dev, unchanged)"
|
||||
Reference in new issue
Block a user