Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

feat(workstation): prod dnsmasq profile — and CORRECT the topology I had wrong

Browse Source 
OPERATOR CORRECTION (2026-09-05). I had been calling the resolver that shadows
prod "a pfSense override" throughout this repo. Both halves of that were wrong:

  - It is the WORKSTATION'S OWN dnsmasq, not pfSense. The live resolver is the
    one NetworkManager spawns (dns=dnsmasq, --conf-dir=/etc/NetworkManager/
    dnsmasq.d); the standalone dnsmasq.service is INACTIVE, so /etc/dnsmasq.d is
    dormant. The dev profile's `address=/.home.keithjackson.dev/192.168.2.50`
    matches the domain AND EVERY SUBDOMAIN, so it also swallowed the whole new
    prod.home.* tree and answered it with dev's address.
  - The k3s prod cluster is NOT behind pfSense. It sits on the SOHO router
    network 192.168.2.0/24, which is pfSense's WAN; pfSense CONTAINS the dev LAN
    192.168.1.0/24. This workstation is on 192.168.2.0/24 too, which is why the
    rules below use prod's real VIPs and need no NAT hop, unlike dev's .50.

New infra/workstation/dnsmasq/ — authored, not click-ops:
  20-prod.home.keithjackson.dev.conf   prod.home.* -> 192.168.2.20 (Gateway),
                                       unifi -> .21, crafty-mc -> .22 (L4 VIPs)
  install.sh                           installs + reloads; refuses non-root

dnsmasq resolves `address=` by LONGEST MATCH, so the prod rules win WITHOUT
editing or reordering the dev profile. PROVEN, not assumed: ran a throwaway
dnsmasq on 127.0.0.1:5354 loaded with both profiles —
  prod.home / gitea.prod / keycloak.prod -> 192.168.2.20
  unifi.prod -> 192.168.2.21   crafty-mc.prod -> 192.168.2.22
  gitea.home / home.keithjackson.dev     -> 192.168.2.50  (DEV, UNCHANGED)
`dnsmasq --test` on the new file: syntax OK.

install.sh pins the destination because getting it wrong is SILENT: a file in
/etc/dnsmasq.d exists, looks installed, and changes nothing. It also uses
`systemctl reload NetworkManager` rather than SIGHUP — dnsmasq does not re-read
its config on SIGHUP, so a HUP would look like it worked and not apply anything.

WHY THE MISLABEL MATTERED: the trap PARTIALLY works. Dev serves a
*.home.keithjackson.dev wildcard cert, which matches prod.home.keithjackson.dev
(one label) so the prod apex returns 200 over VALID TLS, while
keycloak.prod.home.keithjackson.dev (two labels) fails with a certificate-name
error. It reads as a broken prod cert; it is the wrong server answering. I made
exactly that misreading during verification.

Also recorded, because it changes what the apex flip IS: the SOHO router NATs
inbound 443 to the pfSense WAN (192.168.2.102) to serve the EXISTING NON-K3S
production environment, which must stay live until the switch is deliberately
made. So §10a sequence P is not just a k3s-side DNS/TLS change — it is the
legacy-prod cutover, and P as written does NOT cover repointing that NAT. The
two must be planned together or the apex resolves to k3s while inbound 443 still
lands on the legacy box. Phase C touches none of it: prod's Gateway is
192.168.2.20:443 (a different address) and its cert is DNS-01, needing no
inbound 443. And the MinIO timeout is now correctly described as expected
topology — crossing INTO pfSense from its WAN — not a misconfiguration.

VERIFIED: full CI gate PASS (spec-immutability, lint, l0, render-golden, l1).

Refs R-DNS-02, R-ENV-02.

Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-05 11:08:54 -05:00
1 parent b80e8eef82
commit 6d62aca0d2
5 files changed
+168 -18
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

infra/PROD-STANDUP.md
+63 -14
View File
@@ -238,27 +238,54 @@ prod.home.keithjackson.dev 200 keycloak.… 302 (login redirect)
gitea.prod.home.keithjackson.dev 200 grafana.… 302 (SSO redirect)
```
### ACTION REQUIRED — the pfSense override shadows prod from the LAN
### Workstation DNS — install the prod dnsmasq profile
`getent hosts` on the workstation returns **192.168.2.50** (dev) for **every** `*.home.keithjackson.dev`
name — and the new `prod.home.…` subtree falls inside that wildcard. Cloudflare correctly returns
`192.168.2.20`. So **prod is reachable from anywhere using public DNS, but from this workstation and
anything else behind pfSense, prod hostnames resolve to dev.**
**CORRECTION (operator, 2026-09-05): this is the workstation's own dnsmasq, NOT pfSense.** Earlier
notes in this repo called it a "pfSense override"; that was wrong and the topology below is why.
Narrow the override so it does not cover `prod.home.keithjackson.dev`, or add a matching override
pointing that subtree at `192.168.2.20`. Until then, test prod with an explicit resolve:
`/etc/NetworkManager/dnsmasq.d/10-home.keithjackson.dev.conf` (the dev profile) contains:
```
address=/.home.keithjackson.dev/192.168.2.50
```
dnsmasq's `address=` matches the named domain **and every subdomain**, so that one line also captures
the whole `prod.home.keithjackson.dev` tree and answers it with **dev's** address, while Cloudflare
correctly returns prod's `192.168.2.20`.
**Fix — one command, already authored in this repo:**
```bash
sudo infra/workstation/dnsmasq/install.sh
```
It installs `20-prod.home.keithjackson.dev.conf` and reloads the live resolver. dnsmasq resolves
`address=` by **longest match**, so the prod rules win without editing or reordering the dev profile.
Verified against a throwaway dnsmasq loaded with both profiles:
| query | answer |
|---|---|
| `prod.home.keithjackson.dev`, `gitea.prod…`, `keycloak.prod…` | `192.168.2.20` (Gateway) |
| `unifi.prod.home.keithjackson.dev` | `192.168.2.21` (L4) |
| `crafty-mc.prod.home.keithjackson.dev` | `192.168.2.22` (L4) |
| `gitea.home.keithjackson.dev`, `home.keithjackson.dev` | `192.168.2.50` — **dev, unchanged** |
> **Install to `/etc/NetworkManager/dnsmasq.d/`, never `/etc/dnsmasq.d/`.** The standalone
> `dnsmasq.service` is **inactive**; the live resolver is the one NetworkManager spawns
> (`dns=dnsmasq`) with `--conf-dir=/etc/NetworkManager/dnsmasq.d`. A file in the other directory
> changes nothing and looks like it should. `install.sh` pins the destination for exactly this reason.
**Until it is installed, verify prod only with an explicit resolve** — and note *why* this trap is
convincing: dev serves a `*.home.keithjackson.dev` wildcard cert, which matches
`prod.home.keithjackson.dev` (one label), so the prod **apex returns 200 over valid TLS** while
`keycloak.prod.home.keithjackson.dev` (two labels) fails with a certificate-name error. It reads like
a broken prod certificate. It is the wrong server answering.
```bash
curl -sS --resolve gitea.prod.home.keithjackson.dev:443:192.168.2.20 \
https://gitea.prod.home.keithjackson.dev/
```
**This is also the trap that produced a false healthy reading during the 2026-09-04 incident.** It is
subtle because the *apex* appears to work: dev's `*.home.keithjackson.dev` wildcard matches
`prod.home.keithjackson.dev` (one label) so TLS succeeds, while `keycloak.prod.home.keithjackson.dev`
(two labels) fails with a certificate-name error. A partial success is what makes it convincing.
**Never verify DNS or TLS through a local resolver here.**
## Blockers
### 1. Backups: endpoint DECIDED, but prod cannot reach it — **prod has no backup coverage**
@@ -273,9 +300,20 @@ from the workstation, which proves nothing about pod egress:
curl http://192.168.1.9:9000/minio/health/live → connection timed out after 15s
```
pfSense does not route prod's pod network to the dev LAN's MinIO. **Operator action required.**
**This is expected topology, not a misconfiguration** (operator, 2026-09-05): the k3s prod cluster is
**not behind pfSense**. It sits on the SOHO router network `192.168.2.0/24`, which is pfSense's
**WAN** side; pfSense *contains* `192.168.1.0/24`, the dev LAN where MinIO lives. Prod reaching
`192.168.1.9` therefore means crossing **into** pfSense from its WAN — which requires a deliberate
pfSense rule or port-forward that does not exist. **Operator decision required**, and worth weighing:
a prod backup that lands inside the dev LAN is arguably not off-cluster DR at all.
Re-test with `standup.sh --s3-endpoint http://192.168.1.9:9000 s3check` — it fails closed and says so.
> **Do not confuse this with the SOHO router's inbound NAT.** That router forwards inbound **443** to
> the pfSense WAN address `192.168.2.102` to serve the **existing, non-k3s production environment**,
> which must stay live on 443 until the k3s cutover is deliberately made. Nothing in this runbook
> touches that NAT, and prod's Gateway on `192.168.2.20:443` does not contend with it.
Until that routing exists, `velero` stays suspended and **`apps-wave-0` still applies GREEN** — the
token *is* defined, so strict substitution succeeds — while every write to that endpoint fails at
**runtime**. A green `apps-wave-0` does not mean backups work. **Treat prod as having no backup
@@ -320,6 +358,17 @@ nothing, and standing it up first is strictly better: it proves the whole stack
the cutover becomes a DNS/TLS change on a known-good cluster instead of a bring-up and a cutover at
once. The apex move is `infra/PROD-PROVISIONING.md` §10a, sequence **D** (dev vacates) then **P**.
> **THE APEX FLIP IS ALSO THE LEGACY-PROD CUTOVER — they are the same event.** The SOHO router NATs
> inbound **443** to the pfSense WAN address `192.168.2.102`, serving the **existing, non-k3s
> production environment**, which the operator requires to stay live until the switch is deliberately
> made. So §10a sequence **P** is not merely a DNS/TLS change on the k3s side: taking
> `home.keithjackson.dev` means public traffic must also stop resolving to / being forwarded to the
> legacy stack. Sequence **P** as written does **not** cover repointing that NAT, and the two must be
> planned together or the apex will resolve to k3s while inbound 443 still lands on the legacy box.
>
> Nothing in phase C touches it: prod's Gateway is `192.168.2.20:443`, a different address from the
> NAT target, and prod's wildcard cert is issued by **DNS-01**, which needs no inbound 443 at all.
---
## Not covered here