Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 3 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

feat(dns): INVERT to operator's simpler layout — dev to subdomain, prod to apex

Browse Source 
*** DO NOT MERGE TO main WHILE THE OPERATOR IS AWAY. MERGING IS THE MIGRATION. ***

Operator simplified the architecture, inverting c8c1d29:
  dev  -> *.dev.home.keithjackson.dev        (was the apex)
  prod -> *.home.keithjackson.dev            (the apex; no prod. subdomain, no CNAME layer)

WHY MERGING IS DANGEROUS, and this is the finding that matters most here:
clusters/dev/sync.yaml self-reconciles ./clusters/dev from `main` every 10 minutes. Merging
these files EXECUTES the cutover on the live dev cluster, unattended. And because the same
commit moves dev's GitRepository url to gitea.dev.home.keithjackson.dev, dev STRANDS — it
stops reconciling and cannot be fixed by pushing, because the broken thing IS the path Flux
reads pushes through. Banners added at the top of clusters/dev/kustomization.yaml and beside
the url itself. That url cannot be tokenized: the root Kustomization declares no
substituteFrom, so `gitea.${BASE_DOMAIN}` would ship the literal token.

Verified dev is currently safe: this work is NOT on origin/main, dev is reconciling
main@4809199, and its live BASE_DOMAIN is still home.keithjackson.dev. Deliberately NOT
pushed to the bootstrap Gitea either — prod reconciles from there, and prod on `prod.home.`
cannot collide with dev on the apex, so that is the safer interim for prod.

VALUES
                BASE_DOMAIN                  TXT_OWNER_ID   DOMAIN_FILTER
  base          dev.home.keithjackson.dev    sentinel       keithjackson.dev
  dev           dev.home.keithjackson.dev    dev            keithjackson.dev
  prod          home.keithjackson.dev        prod           keithjackson.dev
Ansible seam inverted too: role default k3s_base_domain = dev's subdomain, group_vars/k3s_prod
overrides to the apex.

dev's txtOwnerId moves `default` -> `dev`. The old value was FORCED (its live apex records
were already registered under `default`, so renaming would have orphaned them). On a new
suffix dev publishes a brand-new record set with no prior registry, so the id is chosen on
merits — and `default` has none: it is what you get by FORGETTING the flag, which is exactly
how both clusters came to share it and delete each other's records. Operationally it also
means dev's old apex records stay owned by a now-unclaimed id, so nothing deletes them
implicitly and vacating the apex becomes an explicit sequenceable step rather than a race
between two reconcile loops. PROD MUST NEVER BE GIVEN `default`: prod inherits the apex
NAMES, so `default` is the one value that would let it adopt and policy:sync-delete dev's
still-live apex records — the same incident with the roles swapped.

DOMAIN_FILTER stays the ZONE for both, with F-DNS1 written into all three files as a
do-not-retry: a subdomain filter drops the parent zone and publishes NOTHING.

VERIFICATION — the property changed, and so did the method. "Dev renders byte-identically" no
longer applies because dev is deliberately migrating. Instead: apply the INVERSE substitution
to the new render and demand byte-identity with the baseline. 145 changed line pairs, every
one classified: 143 hostname-move, 2 declared owner-id, ZERO unexplained, ZERO bare-apex
remnants. cilium and velero show 0 changes and provably contain no hostname, so they are not
blind spots. Prod: 0 occurrences of prod.home., 0 of dev.home., 143 apex names. Certs: dev
*.dev.home.keithjackson.dev, prod *.home.keithjackson.dev.

THREE positive controls, all fire — including one that proves the two checks are not
redundant: injecting a single un-moved hostname SURVIVES the inverse-substitution check and is
caught only by the independent bare-apex scan. That is why both exist.

Ansible probe (connection: local, zero SSH): dev nodes -> keycloak.dev.home..., prod nodes ->
keycloak.home.... Used roles: + --tags probe rather than include_vars, because include_vars
outranks group_vars and would have FALSELY shown prod inheriting dev's value.

Gates all PASS: l0, l1, render-golden (10 goldens), lint, substitution (254 assertions),
render.py --selftest, spec-immutability. Catalog ServiceDefinitions edited, never rendered apps/.

RUNBOOK rewritten in §10 + §10a with the ordering that avoids stranding: D0 snapshot escape
routes -> D1 break the automatic trigger -> D2 publish and PROVE gitea.dev.home + the new
wildcard cert WHILE dev still serves the apex -> D3 merge -> D4 release reconciliation
deliberately -> D5 Keycloak issuer + ~12 clients (headlamp's is SOPS, by hand) -> D6
one-server-at-a-time k3s restart + SSO re-issue (service restart only, NO reboots) -> D7
retire old apex records -> D8 verify the apex is empty AGAINST CLOUDFLARE, not pfSense —
the gate on everything after. Then P1 unsuspend prod.

ACME: prod's wildcard is EXACTLY the name set dev holds today, so prod's orders share dev's
5-per-week duplicate-certificate bucket. Order after D8, delete dev's old Certificate rather
than let it renew, rehearse on staging.

The 6 L2 chainsaw tests hardcoding the apex are enumerated but deliberately NOT edited — while
dev is live on the apex they are correct; they need per-environment parameterisation.

Serves: R-ENV-02, R-DNS-01/02/03, R-TLS-01.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-04 23:08:50 -05:00
1 parent c8c1d2935d
commit 9137bbd0c3
8 files changed
+556 -234
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

infra/PROD-PROVISIONING.md
+264 -92
View File
@@ -1172,7 +1172,7 @@ no extra vars is byte-for-byte the dev behaviour it always had (verified by a be
| `192.168.1.55` (crafty) | `platform/crafty/service-l4.yaml` | prod-LAN address |
| `192.168.1.54` (UniFi VM) | `apps/vms/unifi/50-service-l4.yaml` | prod-LAN address |
| `192.168.1.9:9000` (S3 backup) | `platform/keycloak/postgres-cluster.yaml`, `apps/wave-0/{nextcloud,semaphore}-db.yaml` | reachable prod backup target |
| ~~`home.keithjackson.dev` (every hostname)~~ | ~~HTTPRoutes, Gateway, wildcard cert, coredns-custom, Keycloak realm, oauth2-proxy, Homepage, `catalog/`~~ | **CLOSED 2026-09-04 — see §10.** This one was not merely "would build wrong": it **already destroyed dev's public DNS**. `BASE_DOMAIN` was a declared-but-DEAD knob; it is now real, and prod serves `prod.home.keithjackson.dev`. |
| ~~`home.keithjackson.dev` (every hostname)~~ | ~~HTTPRoutes, Gateway, wildcard cert, coredns-custom, Keycloak realm, oauth2-proxy, Homepage, `catalog/`~~ | **CLOSED 2026-09-04 — see §10.** This one was not merely "would build wrong": it **already destroyed dev's public DNS**. `BASE_DOMAIN` was a declared-but-DEAD knob; it is now real. Prod's value was inverted the same day to the APEX `home.keithjackson.dev`, with dev moving to `dev.home.keithjackson.dev` — see §10a, which is authored but NOT yet applied. |
- [ ] **Tokenize each into a `cluster-vars` knob** with a base default, then set the value in *both*
overlays. Adding a knob before referencing `${KNOB}` is the sanctioned mechanism (see the
@@ -1283,12 +1283,20 @@ single-server-etcd window, and avoids a later member-add dance.
specifically.
- [ ] Monitoring and alerts live on prod.
- [ ] The conformance suite passes green **against the prod cluster**, not just dev (R-TEST-03).
Blocked on parameterizing the L2 tests that hardcode `home.keithjackson.dev` — see §10 phase 2.
- [ ] **Prod serves its own DNS/TLS identity** and dev's records are provably untouched (§10):
distinct `EXTERNALDNS_TXT_OWNER_ID`, prod wildcard cert `*.prod.home.keithjackson.dev`, and
prod's own Keycloak realm + per-service client secrets.
Blocked on parameterizing the L2 tests that hardcode `home.keithjackson.dev` — see §10a
"The L2 tests hardcode the apex".
- [ ] **Prod serves the apex DNS/TLS identity** and dev's records are provably untouched (§10/§10a):
distinct `EXTERNALDNS_TXT_OWNER_ID` (`prod` vs `dev`), prod wildcard cert
`*.home.keithjackson.dev`, dev migrated to `*.dev.home.keithjackson.dev` with its old apex
records retired (§10a step D8), and prod's own Keycloak realm + per-service client secrets.
## 10. DNS/TLS environment split — **the 2026-09-04 incident, and phase 1 of the fix**
## 10. DNS/TLS environment split — **the 2026-09-04 incident, and the target architecture**
> **Current state (2026-09-04):** the target below is **AUTHORED, NOT APPLIED.** Prod's
> `platform` tier and its 10 downstream tiers are SUSPENDED. Dev is **live on the apex**. The
> ordered cutover is **§10a**, and merging the authored change to `main` is step **D3** of it,
> not step one — dev self-reconciles `main`, so a merge outside the runbook window executes
> the migration unattended.
### What happened
@@ -1320,43 +1328,82 @@ values; for DNS and TLS that promise was false, so `clusters/prod` was *authored
public identity. Two clusters that describe the same DNS name will fight over it no matter how the
DNS controller is configured.
### The architecture decision (operator)
### The architecture decision (operator) — **INVERTED 2026-09-04, this is the current target**
```
dev -> *.dev.home.keithjackson.dev gitea.dev.home.keithjackson.dev
prod -> *.prod.home.keithjackson.dev gitea.prod.home.keithjackson.dev
apex -> CNAME <svc>.home.keithjackson.dev -> <svc>.prod.home.keithjackson.dev
dev -> *.dev.home.keithjackson.dev gitea.dev.home.keithjackson.dev
prod -> *.home.keithjackson.dev gitea.home.keithjackson.dev (THE APEX)
```
### PHASE 1 — **prod only. Dev did not move.** (this change)
**Prod *is* the apex. There is no `prod.` label and no CNAME layer at all** —
`<svc>.home.keithjackson.dev` is prod's real name, not an alias for one. This **supersedes** the
interim split committed earlier the same day (`c8c1d29`), which put prod on
`*.prod.home.keithjackson.dev` behind apex CNAMEs pointing at it. That layer is deleted: fewer
moving parts, no CNAME chain to keep in step, and prod ends up on the names users already type.
`BASE_DOMAIN` is now real: every hostname in every Flux-reconciled manifest derives from it.
The cost of the inversion is that **dev must move**, and dev is the live cluster. The interim split
was chosen precisely because it moved only the unbuilt environment; this one does not have that
property, so the whole of §10's remaining risk now sits on dev's side, and it is sequenced below.
| | `BASE_DOMAIN` | `EXTERNALDNS_TXT_OWNER_ID` | `EXTERNALDNS_DOMAIN_FILTER` |
|---|---|---|---|
| base default | `home.keithjackson.dev` | `unset-see-cluster-vars` (non-colliding sentinel) | `keithjackson.dev` |
| **dev** | `home.keithjackson.dev` (**unchanged**) | `default` (**must not change**) | `keithjackson.dev` |
| **prod** | `prod.home.keithjackson.dev` | `prod` | `prod.home.keithjackson.dev` |
### The values, per environment
**Dev's effective hostname is deliberately still the apex `home.keithjackson.dev`.** Moving prod
alone ends the collision and unblocks prod, and it is the whole of the risk that had to be taken
while the operator is away. Dev's rendered output is **byte-identical** except for one added line,
`txtOwnerId: default`, which is the value dev's external-dns was already using implicitly — verified
by rendering all 12 `substituteFrom` paths through `kubectl kustomize | flux envsubst --strict` for
both overlays and diffing.
| | `BASE_DOMAIN` | `EXTERNALDNS_TXT_OWNER_ID` | `EXTERNALDNS_DOMAIN_FILTER` | Ansible `k3s_base_domain` |
|---|---|---|---|---|
| base default | `dev.home.keithjackson.dev` | `unset-see-cluster-vars` (non-colliding sentinel) | `keithjackson.dev` | `dev.home.keithjackson.dev` (role default) |
| **dev** | `dev.home.keithjackson.dev` | `dev` | `keithjackson.dev` | (inherits the role default) |
| **prod** | `home.keithjackson.dev` | `prod` | `keithjackson.dev` | `home.keithjackson.dev` (`group_vars/k3s_prod`) |
**Two independent guards now, either sufficient alone:**
`[parity]`: `k3s_base_domain` must equal the same cluster's `BASE_DOMAIN`. The kube-apiserver's
`oidc-issuer-url` is a k3s **server flag** in `/etc/rancher/k3s/config.yaml`, not a manifest, so Flux
post-build substitution can never reach it — hence one fact in two places. If they disagree the
apiserver validates tokens against a different Keycloak than the cluster runs: kubectl SSO fails
while the cert-based admin kubeconfig keeps working, so it presents as "SSO is broken", not as skew.
1. **`txtOwnerId`** — per-environment identity. A cluster only ever touches records it owns. This is
the direct fix for the incident. Dev **must** render exactly `default`: its live records are
already registered under that owner, and any other value would make dev's own external-dns stop
recognising them and re-create duplicates beside the orphans.
2. **`domainFilters`** — per-environment name scope. Prod cannot see a name outside its own subtree.
Defence in depth, added because the first guard is a single string in a single file.
**`EXTERNALDNS_DOMAIN_FILTER` is the Cloudflare ZONE for BOTH, and must stay that way.** FINDING
F-DNS1 (DECISIONS-LOG 2026-08-17): the Cloudflare provider uses this value for **zone selection** as
well as record matching, so a filter that is a strict subdomain of the hosted zone matches **no
zone** — external-dns generates the record and then skips it with *"no hosted zone matching record
DNS Name"*, publishing **nothing**. This has been "hardened" and reverted more than once. It is not
defence in depth; it is a total outage of DNS publication. Isolation comes from the two guards below,
which a domain filter cannot provide anyway: both environments live in the one zone regardless.
The base default for the owner id is a **non-colliding sentinel, deliberately not `default`**: a
future environment that forgets the knob must not silently inherit dev's identity — that is exactly
what happened here. With the sentinel such a cluster owns nothing, so it can delete nothing.
**The two guards that actually work:**
1. **`txtOwnerId`, per-environment identity** — a cluster only ever reconciles records its own owner
id claims in the TXT registry. This is the direct fix for the incident.
2. **Disjoint hostname sets** — after the migration the two clusters never describe the same record,
so there is nothing to fight over. *This guard does not exist during the migration window*, which
is the entire reason the ordering below is mandatory.
#### Why dev's owner id changed from `default` to `dev`
Until this migration dev **had** to render `default`: its live apex records were already registered
in Cloudflare's TXT registry under that owner, and renaming would have orphaned every one of them.
**The migration dissolves that constraint** — on `*.dev.home.keithjackson.dev` dev publishes an
entirely new record set with no prior registry entries to recognise, so the id is free and is chosen
on its merits. `default` has none: it is the value external-dns uses when you **forget** the flag,
which is exactly how both clusters came to share it, and an environment whose identity is the
forget-value can never be told apart from a misconfigured newcomer. Consequences, both deliberate:
- Dev's **old apex records stay owned by `default`**, which no environment now claims, so nothing
deletes them implicitly on some future reconcile. Retiring them is an explicit step (**D7**) and
prod's first publish is gated on that retirement being *confirmed* — not on two reconcile loops
racing each other.
- **Prod stays `prod` and must never be given `default`.** Prod is taking over the apex *names*, and
dev's existing apex records are registered under `default` — so `default` is the one value that
would let prod adopt them, and under `policy: sync` delete the ones no prod source claims, on a
cluster dev is still serving. That is the 2026-09-04 incident re-armed with the roles swapped.
Prod inherits the apex **names**; it must never inherit the apex **owner id**.
All three values differ, and the base default remains a sentinel so an environment that forgets the
knob owns nothing and can therefore delete nothing.
### PHASE 1 — prod tokenized and moved off dev's names (`c8c1d29`, done; suffix since inverted)
`BASE_DOMAIN` was a **dead knob** — declared in `clusters/base/cluster-vars.yaml` and referenced by
no reconciled manifest, because every hostname was a hardcoded literal. `c8c1d29` made it real (141
references across the reconciled set) and gave prod its own suffix. That work stands; only the
*value* prod holds has changed, from `prod.home.keithjackson.dev` to the apex.
### What could NOT be tokenized, and why
@@ -1365,70 +1412,195 @@ what happened here. With the sentinel such a cluster owns nothing, so it can del
Re-encrypting needs the operator's age key, rewrites the MAC over every value in the file, and
**cannot be verified by the offline render harness** (`kustomize` does not decrypt), so a mistake
would break Headlamp SSO on the live dev cluster with no pre-flight proof. Left as-is deliberately.
**Prod action required:** prod's Headlamp will otherwise authenticate against **dev's** Keycloak.
Prod needs its own copy of this Secret with `keycloak.prod.home.keithjackson.dev` and a client
secret from prod's own realm — which it needs regardless, since every `*.sops.yaml` client secret
is per-realm.
**Both environments now need action:** dev's copy must move to `keycloak.dev.home.keithjackson.dev`
as part of step **D5**, and prod needs its own copy with `keycloak.home.keithjackson.dev` and a
client secret from prod's own realm — which it needs regardless, since every `*.sops.yaml` client
secret is realm-scoped.
- **`oidc-issuer-url` in the kube-apiserver flags** (`roles/k3s_server/tasks/main.yml`) — a k3s
**server flag** in `/etc/rancher/k3s/config.yaml`, not a manifest, so Flux post-build substitution
can never reach it. Correctly tokenized on the **Ansible** side instead: new role default
`k3s_base_domain` (dev value) + derived `k3s_oidc_issuer_url`, overridden for prod in
`inventory/group_vars/k3s_prod/vars.yml` — the same inventory seam as `k3s_data_dir`/`k3s_api_host`.
`[parity]`: `k3s_base_domain` **must** equal the same cluster's `BASE_DOMAIN`, or the apiserver
validates tokens against a different Keycloak than the cluster runs and kubectl SSO fails while the
cert-based admin kubeconfig keeps working (so it looks like "SSO is broken", not like config skew).
server flag, not a manifest. Handled on the **Ansible** seam instead: role default
`k3s_base_domain` (dev's value) + derived `k3s_oidc_issuer_url`, overridden for prod in
`inventory/group_vars/k3s_prod/vars.yml` — the same inventory seam as `k3s_data_dir`/`k3s_api_host`,
so a fourth prod server inherits it from group membership. Verified per host by a read-only probe
(`connection: local`, zero SSH, `changed=0`): all three dev servers resolve
`https://keycloak.dev.home.keithjackson.dev/realms/homelab`, all three prod servers resolve
`https://keycloak.home.keithjackson.dev/realms/homelab`.
- **`spec.url` of dev's `GitRepository`** (`clusters/dev/kustomization.yaml`) — see the strand below.
It *cannot* be tokenized: the `flux-system` root Kustomization that renders `./clusters/dev`
declares no `postBuild.substituteFrom`, so `gitea.${BASE_DOMAIN}` there would ship the **literal
token** and break the source outright (the L0 `flux-postbuild-substitution` check asserts exactly
this). It is a hand-maintained literal that must move in lockstep with `BASE_DOMAIN`.
### A pre-existing prod problem this incidentally FIXED
---
§4 recorded that prod nodes use **public** resolvers, so the kube-apiserver would resolve its OIDC
issuer publicly and get **dev's** Gateway VIP (192.168.1.53) — the wrong cluster on the wrong LAN,
making prod kubectl SSO impossible, and requiring an `/etc/hosts` pin as a workaround. With prod on
its own subdomain the public record for `keycloak.prod.home.keithjackson.dev` is published by
**prod's own** external-dns pointing at **prod's** Gateway VIP (192.168.2.20), so the public answer
and the split-horizon answer agree. **No `/etc/hosts` pin is needed.** This holds only once prod's
external-dns has actually published the record; until then use the break-glass cert kubeconfig.
## 10a. THE MIGRATION RUNBOOK — ordering is load-bearing
### Verify on prod's first reconcile
> **Prod's `platform` tier and its 10 downstream tiers are SUSPENDED and must stay suspended until
> step P1.** Nothing below is safe to run out of order, and nothing below should be run while the
> operator is away.
- [ ] **`domainFilters` zone selection.** The Cloudflare provider uses this value for *zone* lookup
as well as record matching, and in 2026-08 a strict-subdomain filter dropped the parent zone
("no hosted zone matching record DNS Name"). If prod's external-dns logs no matching zone it
will create **nothing** — fail-closed, never destructive. Fallback: set prod's
`EXTERNALDNS_DOMAIN_FILTER` back to `keithjackson.dev` and rely on `txtOwnerId`, which is
sufficient on its own.
- [ ] **Prod's wildcard cert is `*.prod.home.keithjackson.dev`** and solves DNS-01 without touching
dev's `_acme-challenge` records.
- [ ] **Dev's records are untouched** after prod's tiers are resumed. Confirm dev's TXT registry
entries still read owner `default` and that prod's read owner `prod`.
- [ ] **Prod's own Keycloak realm + per-service client secrets** exist (every `*.sops.yaml` client
secret is realm-scoped; prod cannot reuse dev's).
### The two ways this goes wrong
### PHASE 2 — dev migration + apex CNAMEs (**deferred, not done**)
**(1) BOTH CLUSTERS ON THE APEX — the 2026-09-04 incident, recreated.** If prod publishes
`*.home.keithjackson.dev` before dev has vacated it, two clusters describe the same records in the
same zone again. Distinct `txtOwnerId`s stop prod from *deleting* dev's records — that guard holds —
but they do not stop the second failure mode: both clusters requesting a `letsencrypt-prod` wildcard
for the **same name set** and racing each other's `_acme-challenge` TXT records, so neither reliably
solves DNS-01. **Rule: dev vacates the apex first, and the vacate is verified against Cloudflare
before prod is unsuspended.**
Deliberately not attempted here: dev is **live** (18 Longhorn volumes, working SSO) and it hosts the
Gitea that is currently this repo's GitOps source. Moving it is a migration, not a variable change.
Required, roughly in order:
**(2) THE STRAND — dev's GitOps source is the thing that moves.** `spec.url` of dev's
`GitRepository` is `https://gitea.home.keithjackson.dev/...`, which is **both** dev's GitOps source
**and** the PM's push target. Flipping dev's `BASE_DOMAIN` moves Gitea's HTTPRoute, its public DNS
record and the wildcard cert covering it to `gitea.dev.home.keithjackson.dev` all at once. If that
host does not already resolve and serve a valid, Ready cert at that instant, **dev stops reconciling
and cannot be fixed by pushing** — the broken thing *is* the path Flux uses to read pushes. Recovery
is then hands-on and needs LAN access to the cluster.
1. **`BASE_DOMAIN` → `dev.home.keithjackson.dev`** in `clusters/dev`, plus `k3s_base_domain` for the
dev inventory group.
2. **A new `*.dev.home.keithjackson.dev` wildcard cert** issued and Ready *before* cutover.
3. **Keycloak issuer change.** The realm's issuer URL is baked into every token; every OIDC client
(`grafana`, `gitea`, `nextcloud`, `semaphore`, `headlamp`, `crafty`, `pihole`, `jackett`,
`qbittorrent`, `radarr`, `sonarr`, and the `kubernetes` public client) needs its redirect URIs
updated in the same change, and the k3s apiserver `oidc-issuer-url` with it — which means a
**rolling k3s restart** on the dev servers.
4. **kubectl SSO re-issue** for every user (issuer changed).
5. **`gitea.dev.home.keithjackson.dev`** — this is the GitOps source URL in
`clusters/dev/kustomization.yaml`. Flux must be able to resolve and trust the new host *before*
the URL is switched, or dev stops reconciling and the change cannot be pushed to fix itself.
Sequence this last, and keep the bootstrap Gitea available as the recovery path (docs/05).
6. **`EXTERNALDNS_TXT_OWNER_ID` for dev.** Once dev's names move, its existing `default`-owned
records are for names dev no longer serves; retire them deliberately rather than letting a
renamed owner orphan them.
7. **Apex CNAMEs** `<svc>.home.keithjackson.dev` → `<svc>.prod.home.keithjackson.dev`. This is what
actually **moves user traffic to prod**, and it must not land before prod serves those services.
8. **The L2 behavioural conformance tests hardcode `home.keithjackson.dev`** (`dns-exposure`,
`dns-internal-resolution`, `gateway-ingress-reachable`, `oidc-sso-grafana`, `sso-forward-auth`,
`trusted-tls`). They are test inputs, not reconciled manifests, so they were left alone in phase 1
— but they will fail against prod and must be parameterized before §9's "suite passes green
against the prod cluster" can be claimed.
**(2b) AND THE MERGE ITSELF IS THE TRIGGER.** `clusters/dev/sync.yaml` is a self-managing root
Kustomization reconciling `./clusters/dev` from **`main`** every 10 minutes, and prod tracks `main`
too. **A merge to `main` therefore *executes* this migration on the live dev cluster, unattended.**
The authored change is a *target*, not a deployment. **Do not merge these files to `main` outside the
runbook window.** Park them on a branch until the operator is at the keyboard and step **D1** has
been done.
### D — dev vacates the apex (operator present, in this order)
- [ ] **D0. Snapshot the escape routes before touching anything.** Confirm a working break-glass
**cert-based** admin kubeconfig for dev (not the OIDC one — OIDC breaks in D6). Confirm the
retained bootstrap Gitea can be started (`bootstrap/gitea/bootstrap-gitea.sh`; dev's historical
recovery host was `192.168.2.103:3000`, prod's current bootstrap is `192.168.2.132:3000`).
Record dev's current Cloudflare record set (names + TXT registry owners) so D7 has a checklist
and so an accidental deletion is detectable.
- [ ] **D1. Break the automatic trigger.** Either `flux suspend kustomization flux-system -n
flux-system` on dev (stops the root from self-applying the merge), **or** re-point dev's
`GitRepository` url at the bootstrap Gitea for the window. Do this **before** the merge, not
after. This is what converts "merging strands dev" into "merging is inert until you say go".
- [ ] **D2. Publish and prove the new host set *while dev still serves the apex*.** The new names
must exist before anything depends on them:
- a `*.dev.home.keithjackson.dev` + `dev.home.keithjackson.dev` **wildcard Certificate issued
and `Ready`** (LE prod, DNS-01) — see the ACME note below before ordering it;
- public DNS for `gitea.dev.home.keithjackson.dev` resolving to dev's Gateway VIP
(192.168.1.53), **checked against Cloudflare / a public resolver, never through the pfSense
override** — that mistake was made on 2026-09-04 and produced a false healthy reading;
- `curl -sS https://gitea.dev.home.keithjackson.dev/api/v1/version` succeeding with a
publicly-trusted cert from a host on the dev LAN.
Until D2 is green, D3 is not safe.
- [ ] **D3. Merge the authored change to `main`** (`clusters/base/cluster-vars.yaml`,
`clusters/dev/`, `clusters/prod/`, the two Ansible files, `platform/externaldns`,
`catalog/render.py`). With D1 in place nothing moves yet.
- [ ] **D4. Release dev's reconciliation deliberately.** Resume the root Kustomization (or restore
the in-cluster url, now pointing at `gitea.dev.home.keithjackson.dev`) and watch
`flux get kustomizations -A` converge. Dev's Gateway listeners, HTTPRoutes, coredns-custom zone
and Homepage links all move here. Dev's external-dns publishes the new record set under owner
`dev`.
- [ ] **D5. Keycloak: issuer + every client.** Dev's realm issuer becomes
`https://keycloak.dev.home.keithjackson.dev/realms/homelab`. The issuer is baked into every
token, and **~12 OIDC clients** need their redirect URIs moved in the same change: `grafana`,
`gitea`, `nextcloud`, `semaphore`, `headlamp`, `crafty`, `pihole`, `jackett`, `qbittorrent`,
`radarr`, `sonarr`, plus the `kubernetes` public client. Most are reconciled from
`platform/keycloak/keycloak-realm-import.yaml` and move with D4; **headlamp's is the exception**
— its issuer/callback live in a SOPS-encrypted Secret and must be re-encrypted by hand.
- [ ] **D6. Rolling k3s restart + kubectl SSO re-issue.** `k3s_base_domain` now renders a different
`oidc-issuer-url` into `/etc/rancher/k3s/config.yaml`, so the `k3s_server` role is **no longer
a no-op on dev**: it rewrites the file and fires the Restart handler on all three servers. Run
it **one server at a time**, confirming `kubectl get nodes` between each. Every user's OIDC
kubeconfig is invalidated and must be re-issued; the cert-based admin kubeconfig keeps working
throughout, which is what makes this step recoverable. **Do not reboot nodes** — restart the
`k3s` service only.
- [ ] **D7. Retire dev's old apex records.** They are owned by `default`, which no environment now
claims, so nothing removes them automatically. Delete the `<svc>.home.keithjackson.dev` A
records, the apex record, **and their TXT registry entries** (`external-dns-*` TXT), plus any
stale `_acme-challenge.home.keithjackson.dev`. *Alternative, if you prefer external-dns to do
its own cleanup:* set dev's `EXTERNALDNS_TXT_OWNER_ID` back to `default` for exactly one
reconcile after D4 — `policy: sync` will then garbage-collect the apex set it no longer has
sources for — and only then switch it to `dev`. That is two commits and needs watching; the
direct deletion is preferred because it is verifiable in one place.
- [ ] **D8. Verify the apex is EMPTY, against Cloudflare.** `dig +short <svc>.home.keithjackson.dev
@1.1.1.1` returns nothing for every service name, and no TXT registry entry for those names
remains. **This is the gate on the whole P sequence.** A pfSense/PiHole override is not
evidence about public DNS.
### P — prod takes the apex
- [ ] **P1. Only now unsuspend prod**, in dependency order: `flux resume kustomization platform`,
then the 10 downstream tiers. Confirm D8 first, in Cloudflare, not from a local resolver.
- [ ] **P2. Prod's external-dns publishes under owner `prod`.** Check the logs for a matching hosted
zone (F-DNS1 fail-closed mode: no zone selected ⇒ nothing created — which is safe, but means
the filter was narrowed against instruction). Confirm the new TXT registry entries read owner
`prod` and that no record owned by `dev` was touched.
- [ ] **P3. Prod's wildcard cert `*.home.keithjackson.dev` + `home.keithjackson.dev` is `Ready`** and
solved DNS-01 without colliding with any `_acme-challenge` dev still holds.
- [ ] **P4. Prod's kubectl SSO works without an `/etc/hosts` pin.** Prod nodes resolve
`keycloak.home.keithjackson.dev` through **public** upstreams (§4), so this only becomes true
once P2 has published that name at prod's Gateway VIP (192.168.2.20). Before then a prod node
resolving its own issuer reaches **dev**. Use the break-glass cert kubeconfig until P2 is done.
- [ ] **P5. Cut prod's GitOps source over** from the workstation bootstrap Gitea
(`http://192.168.2.132:3000/...`) to `https://gitea.home.keithjackson.dev/...` once prod runs
its own Gitea — the block in `clusters/prod/kustomization.yaml` owns that procedure. Note the
inversion **simplified** this: prod's apex Gitea name and dev's `gitea.dev.home...` are now
different hosts, so prod pointing at the apex means "prod's own Gitea", never dev's.
### ACME: the duplicate-certificate limit
Let's Encrypt rate-limits **duplicate certificates — 5 per week for an identical set of names.**
Prod's wildcard is `*.home.keithjackson.dev` + `home.keithjackson.dev`, which is **exactly the name
set dev holds today**. Every prod order therefore counts against the same bucket as dev's historical
renewals of that pair, and a retry loop during P3 can exhaust it and leave prod with **no trusted
cert for up to a week**. Mitigations:
- Order prod's cert **after** D8, so dev is no longer renewing the same pair.
- Let dev's old `*.home.keithjackson.dev` `Certificate`/Secret be **deleted** in D4/D7 rather than
left to renew in the background.
- Rehearse against **letsencrypt-staging** first (`ACME_CLUSTER_ISSUER`) — staging has its own,
far looser limits — and only switch prod to `letsencrypt-prod` when the DNS-01 solve is proven.
- Do **not** delete-and-recreate prod's `Certificate` to "retry"; each recreation is a fresh order.
### The L2 tests hardcode the apex
Six behavioural (L2) chainsaw tests embed `home.keithjackson.dev` as a literal:
| test | file |
|---|---|
| `dns-exposure` | `conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml` |
| `dns-internal-resolution` | `conformance/tests/l2-behavioral/dns-internal-resolution/chainsaw-test.yaml` |
| `gateway-ingress-reachable` | `conformance/tests/l2-behavioral/gateway-ingress-reachable/chainsaw-test.yaml` (+ `fixtures/app-and-probe.yaml`) |
| `oidc-sso-grafana` | `conformance/tests/l2-behavioral/oidc-sso-grafana/chainsaw-test.yaml` |
| `sso-forward-auth` | `conformance/tests/l2-behavioral/sso-forward-auth/chainsaw-test.yaml` |
| `trusted-tls` | `conformance/tests/l2-behavioral/trusted-tls/chainsaw-test.yaml` |
They are **test inputs, not reconciled manifests**, so they are deliberately left alone by the
authoring change: they are the only thing still asserting the apex, and while dev is live on the apex
they are *correct*. Under the new direction the literal happens to become right for **prod** and
wrong for **dev**, so they must be **parameterized per environment** (not merely edited) before
either §9's *"suite passes green against the prod cluster"* or a green L2 run against migrated dev
can be claimed. Until then, expect the dev L2 suite to fail between D4 and that parameterization —
a known, scheduled failure, not a regression.
The L1 *rendering* tests need no change: their goldens carry `${BASE_DOMAIN}` as a token, and the
literal hostnames in them come from each test's own fixture `ingressRoutes[].host`, which is
environment-independent by construction.
### Offline proof carried out for the authoring change (no cluster touched)
All 12 `substituteFrom` paths were rendered for **both** overlays through the real toolchain
(`kubectl kustomize --load-restrictor LoadRestrictionsNone <spec.path> | flux envsubst --strict`,
with each overlay's effective `cluster-vars` taken from `kubectl kustomize clusters/<env>`), before
and after the change.
- **Dev's diff is *only* the hostname move.** Proven by inverse substitution rather than by reading
the diff: applying `s/dev.home.keithjackson.dev/home.keithjackson.dev/` to the new render restores
the baseline **byte-for-byte** in all 12 paths, plus 2 declared `txtOwnerId: default → dev` lines.
145 changed line pairs, all classified: 143 hostname-move, 2 declared owner-id, **0 unexplained**,
and **0 bare-apex remnants**. The 2 paths with no change (`cilium`, `velero`) provably contain no
hostname at all (`grep -c keithjackson.dev` = 0).
- **Prod lands on the bare apex:** **0** occurrences of `prod.home.keithjackson.dev` and **0** of
`dev.home.keithjackson.dev` anywhere in prod's rendered output; 143 apex names.
- **Certs:** dev `*.dev.home.keithjackson.dev` + `dev.home.keithjackson.dev`; prod
`*.home.keithjackson.dev` + `home.keithjackson.dev`.
- **Positive controls** (the harness is provably not blind): perturbing an unrelated knob
(`GATEWAY_VIP`) surfaces 1 unexplained line; perturbing `BASE_DOMAIN` to a wrong value fails 10 of
12 paths; injecting a single un-moved hostname is caught by the independent bare-apex scan (which
the byte-identity check alone cannot see — which is why both exist).
- Gates: `ci/l0.sh`, `ci/l1.sh`, `ci/render-golden.sh`, `ci/lint.sh`,
`flux-postbuild-substitution/check.sh` (254 assertions), `catalog/render.py --selftest`,
`ci/spec-immutability.sh` — all **PASS**.
infra/ansible/inventory/group_vars/k3s_prod/vars.yml
+37 -25
View File
@@ -293,35 +293,47 @@ node_baseline_dns_servers:
# k3s_base_domain, below). It reaches the issuer through the NODE resolver, i.e. through
# these PUBLIC upstreams — not through CoreDNS's split-horizon answer.
#
# THIS USED TO BE BROKEN, AND THE 2026-09-04 SUBDOMAIN SPLIT FIXED IT. While prod was
# authored to serve dev's hostnames, a prod node resolving keycloak.home.keithjackson.dev
# publicly got DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN, so
# kubectl OIDC SSO on prod could not work at all. Now prod's issuer is
# keycloak.prod.home.keithjackson.dev, and the public record for that name is published by
# PROD's own external-dns pointing at PROD's Gateway VIP (192.168.2.20). The public answer
# and the split-horizon answer therefore AGREE, and no /etc/hosts pin is needed.
#
# STILL TO VERIFY on prod bring-up (PROD-PROVISIONING.md §10): this holds only once prod's
# external-dns has actually published the record. Until then the issuer does not resolve
# from the nodes and kubectl SSO fails — the cert-based admin kubeconfig is the break-glass
# path, and it always works. Ordering, not a design gap.
# ORDERING HAZARD, NOT YET CLEARED. Prod's issuer is keycloak.home.keithjackson.dev — an APEX
# name, because prod IS the apex under the 2026-09-04 inversion (k3s_base_domain below).
# That is the name the LIVE DEV CLUSTER answers today, and its public A record currently
# points at DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN. A prod
# node resolving its own issuer through these public upstreams therefore reaches dev until:
# (1) dev has migrated to *.dev.home.keithjackson.dev and RETIRED its apex records, and
# (2) PROD's own external-dns has published keycloak.home.keithjackson.dev at PROD's
# Gateway VIP (192.168.2.20).
# Only then do the public answer and the in-cluster split-horizon answer AGREE, and no
# /etc/hosts pin is needed. Until then kubectl OIDC SSO on prod cannot work; the cert-based
# admin kubeconfig is the break-glass path and always works. Ordering, not a design gap —
# the ordered steps are infra/PROD-PROVISIONING.md §10.
node_baseline_dns_search: []
# --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
# PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/
# kustomization.yaml — the same one-fact-in-two-places contract as k3s_data_dir and
# k3s_api_host. The Flux side cannot supply this: the kube-apiserver's oidc-issuer-url is a
# k3s server FLAG written to /etc/rancher/k3s/config.yaml, not a manifest, so
# postBuild.substituteFrom never sees it. Full rationale in
# roles/k3s_server/defaults/main.yml, where k3s_oidc_issuer_url is derived from this.
# PROD **IS** THE APEX. [parity] MUST equal BASE_DOMAIN in clusters/prod/kustomization.yaml —
# the same one-fact-in-two-places contract as k3s_data_dir and k3s_api_host. The Flux side
# cannot supply this: the kube-apiserver's oidc-issuer-url is a k3s server FLAG written to
# /etc/rancher/k3s/config.yaml, not a manifest, so postBuild.substituteFrom never sees it.
# Full rationale in roles/k3s_server/defaults/main.yml, where k3s_oidc_issuer_url derives.
#
# WHY PROD MOVED (infra/PROD-PROVISIONING.md §10): both clusters were authored to serve the
# SAME hostnames from the SAME Cloudflare zone, and on 2026-09-04 prod's external-dns
# deleted every one of dev's public records. Dev stays on home.keithjackson.dev (it is LIVE);
# prod takes prod.home.keithjackson.dev. Consequence here: prod's Keycloak issuer is
# https://keycloak.prod.home.keithjackson.dev/realms/homelab, so the `kubernetes` OIDC client
# in prod's OWN Keycloak realm is what prod's kubectl SSO authenticates against.
k3s_base_domain: "prod.home.keithjackson.dev"
# WHY THE SUFFIXES SPLIT (infra/PROD-PROVISIONING.md §10): both clusters were authored to serve
# the SAME hostnames from the SAME Cloudflare zone, and on 2026-09-04 prod's external-dns
# deleted every one of dev's public records.
#
# WHICH WAY ROUND — INVERTED 2026-09-04 by operator decision, superseding the interim split
# committed earlier the same day (prod on `prod.home.` behind apex CNAMEs). The simpler target:
# prod -> home.keithjackson.dev (THE APEX; this variable — no `prod.` label, no CNAMEs)
# dev -> dev.home.keithjackson.dev (roles/k3s_server/defaults/main.yml)
# So prod's Keycloak issuer is https://keycloak.home.keithjackson.dev/realms/homelab, and the
# `kubernetes` OIDC client in PROD's OWN Keycloak realm is what prod's kubectl SSO authenticates
# against.
#
# ORDERING HAZARD SPECIFIC TO THIS VARIABLE. keycloak.home.keithjackson.dev is the name the LIVE
# dev cluster answers today, and prod's nodes resolve the issuer through PUBLIC upstreams (see
# node_baseline_dns_search above). So until dev has vacated the apex and PROD's external-dns has
# published that name at PROD's Gateway VIP (192.168.2.20), a prod node resolving its own issuer
# gets DEV's VIP (192.168.1.53) — the wrong cluster, on the wrong LAN. kubectl OIDC SSO on prod
# therefore does not work until §10's dev-vacate steps are complete; the cert-based admin
# kubeconfig is the break-glass path and always works. Ordering, not a design gap.
k3s_base_domain: "home.keithjackson.dev"
# --- Flux bootstrap: WHICH overlay this environment reconciles from (R-ENV-02) -----
# roles/flux_bootstrap defaults flux_bootstrap_env_path to `clusters/dev` (dev is LIVE
infra/ansible/roles/k3s_server/defaults/main.yml
+19 -8
View File
@@ -84,15 +84,26 @@ k3s_tls_sans: >-
# while the cert-based admin kubeconfig keeps working — so it looks like "SSO is broken",
# not like a config skew. The break-glass cert kubeconfig is why this is recoverable.
#
# THE DEFAULT IS THE DEV VALUE, restated explicitly rather than left implicit — the same
# convention as k3s_data_dir above and cluster-vars' GATEWAY_VIP / LONGHORN_DATA_PATH: the
# base default is the known-good LIVE value, so dev's rendered config.yaml is byte-for-byte
# what it already is and the Restart handler does NOT fire on a re-run.
# THE DEFAULT IS THE DEV VALUE — the same convention as k3s_data_dir above and cluster-vars'
# GATEWAY_VIP / LONGHORN_DATA_PATH: the role default is dev's value and prod overrides it in
# inventory/group_vars/k3s_prod/vars.yml (the same inventory seam k3s_data_dir / k3s_api_host /
# kube_vip_address use, so a FOURTH prod server inherits it from group membership rather than
# from anyone remembering).
#
# PROD OVERRIDES IT in inventory/group_vars/k3s_prod/vars.yml — the same inventory seam
# k3s_data_dir / k3s_api_host / kube_vip_address use, so a FOURTH prod server inherits it
# from group membership rather than from anyone remembering.
k3s_base_domain: "home.keithjackson.dev"
# *** WHICH DEV VALUE — UPDATED 2026-09-04, AND IT IS AHEAD OF THE LIVE DEV CLUSTER. ***
# The operator inverted the suffix split: PROD is now the apex `home.keithjackson.dev` and DEV
# carries the label `dev.home.keithjackson.dev`. This default therefore moved from the apex to
# dev's subdomain, mirroring BASE_DOMAIN in clusters/dev/kustomization.yaml.
#
# CONSEQUENCE — this default no longer matches what the live dev nodes are running, so unlike
# every previous edit to this file A DEV RE-RUN IS NOT A NO-OP. Rendering config.yaml with the
# new issuer CHANGES the file, FIRES the Restart handler, and rolls k3s on all three dev
# servers, which also invalidates every user's OIDC kubeconfig (SSO must be re-issued). That is
# a deliberate step of the dev DNS migration, NOT a side effect to discover during unrelated
# work: DO NOT run the k3s_server role against dev until step D6 of
# infra/PROD-PROVISIONING.md §10, and never while the operator is away. The break-glass
# cert-based admin kubeconfig keeps working throughout, which is what makes the step safe.
k3s_base_domain: "dev.home.keithjackson.dev"
# Derived, so the realm path is stated once. Override k3s_base_domain, not this.
k3s_oidc_issuer_url: "https://keycloak.{{ k3s_base_domain }}/realms/homelab"