Revert "feat(dns): INVERT to operator's simpler layout — dev to subdomain, prod to apex"
This reverts commit 9137bbd0c3.
This commit is contained in:
jakman85 committed
1 parent
6f61c9622d
commit
f341a9b744
8 files changed
+229
-401
No files matched your search
+80
-109
@@ -1177,7 +1177,7 @@ no extra vars is byte-for-byte the dev behaviour it always had (verified by a be
|
||||
| `192.168.1.55` (crafty) | `platform/crafty/service-l4.yaml` | prod-LAN address |
|
||||
| `192.168.1.54` (UniFi VM) | `apps/vms/unifi/50-service-l4.yaml` | prod-LAN address |
|
||||
| `192.168.1.9:9000` (S3 backup) | `platform/keycloak/postgres-cluster.yaml`, `apps/wave-0/{nextcloud,semaphore}-db.yaml` | reachable prod backup target |
|
||||
| ~~`home.keithjackson.dev` (every hostname)~~ | ~~HTTPRoutes, Gateway, wildcard cert, coredns-custom, Keycloak realm, oauth2-proxy, Homepage, `catalog/`~~ | **CLOSED 2026-09-04 — see §10.** This one was not merely "would build wrong": it **already destroyed dev's public DNS**. `BASE_DOMAIN` was a declared-but-DEAD knob; it is now real. Prod's value was inverted the same day to the APEX `home.keithjackson.dev`, with dev moving to `dev.home.keithjackson.dev` — see §10a, which is authored but NOT yet applied. |
|
||||
| ~~`home.keithjackson.dev` (every hostname)~~ | ~~HTTPRoutes, Gateway, wildcard cert, coredns-custom, Keycloak realm, oauth2-proxy, Homepage, `catalog/`~~ | **CLOSED 2026-09-04 — see §10.** This one was not merely "would build wrong": it **already destroyed dev's public DNS**. `BASE_DOMAIN` was a declared-but-DEAD knob; it is now real, and prod serves `prod.home.keithjackson.dev`. |
|
||||
|
||||
- [ ] **Tokenize each into a `cluster-vars` knob** with a base default, then set the value in *both*
|
||||
overlays. Adding a knob before referencing `${KNOB}` is the sanctioned mechanism (see the
|
||||
@@ -1331,12 +1331,10 @@ single-server-etcd window, and avoids a later member-add dance.
|
||||
specifically.
|
||||
- [ ] Monitoring and alerts live on prod.
|
||||
- [ ] The conformance suite passes green **against the prod cluster**, not just dev (R-TEST-03).
|
||||
Blocked on parameterizing the L2 tests that hardcode `home.keithjackson.dev` — see §10a
|
||||
"The L2 tests hardcode the apex".
|
||||
- [ ] **Prod serves the apex DNS/TLS identity** and dev's records are provably untouched (§10/§10a):
|
||||
distinct `EXTERNALDNS_TXT_OWNER_ID` (`prod` vs `dev`), prod wildcard cert
|
||||
`*.home.keithjackson.dev`, dev migrated to `*.dev.home.keithjackson.dev` with its old apex
|
||||
records retired (§10a step D8), and prod's own Keycloak realm + per-service client secrets.
|
||||
Blocked on parameterizing the L2 tests that hardcode `home.keithjackson.dev` — see §10 phase 2.
|
||||
- [ ] **Prod serves its own DNS/TLS identity** and dev's records are provably untouched (§10):
|
||||
distinct `EXTERNALDNS_TXT_OWNER_ID`, prod wildcard cert `*.prod.home.keithjackson.dev`, and
|
||||
prod's own Keycloak realm + per-service client secrets.
|
||||
|
||||
## 9a. PROD BRING-UP STATE (2026-09-04) — what came up, and the TWO blockers that need the operator
|
||||
|
||||
@@ -1435,6 +1433,18 @@ unsuspending `platform` and re-queried after — **57 records, byte-for-byte ide
|
||||
entries still `owner=default`.** Snapshots are taken against the **Cloudflare API**, never a local
|
||||
resolver (a pfSense override produced a false healthy reading during the incident).
|
||||
|
||||
> ### THE APEX INVERSION IS NOT ON THIS BRANCH
|
||||
>
|
||||
> Sections 10 and 10a below describe the **target** DNS layout — prod on the bare apex, dev on
|
||||
> `dev.home.` — and the ordered runbook for getting there. **That change is deliberately not authored
|
||||
> on this branch.** It is one commit, `9137bbd`, kept on branch **`migration/dns-apex-inversion`**,
|
||||
> because merging it *is* dev's migration: `clusters/dev/sync.yaml` reconciles `./clusters/dev` from
|
||||
> `main` every 10 minutes, so a merge executes the cutover on the live dev cluster unattended.
|
||||
>
|
||||
> On this branch **prod is authored on `prod.home.keithjackson.dev`**, which is what prod actually
|
||||
> runs. Read what follows as the plan, not as a description of the current authored state. The
|
||||
> inversion is applied as step **D3** of the §10a runbook, operator present.
|
||||
|
||||
## 10. DNS/TLS environment split — **the 2026-09-04 incident, and the target architecture**
|
||||
|
||||
> **Current state (2026-09-04):** the target below is **AUTHORED, NOT APPLIED.** Prod's
|
||||
@@ -1473,82 +1483,43 @@ values; for DNS and TLS that promise was false, so `clusters/prod` was *authored
|
||||
public identity. Two clusters that describe the same DNS name will fight over it no matter how the
|
||||
DNS controller is configured.
|
||||
|
||||
### The architecture decision (operator) — **INVERTED 2026-09-04, this is the current target**
|
||||
### The architecture decision (operator)
|
||||
|
||||
```
|
||||
dev -> *.dev.home.keithjackson.dev gitea.dev.home.keithjackson.dev
|
||||
prod -> *.home.keithjackson.dev gitea.home.keithjackson.dev (THE APEX)
|
||||
dev -> *.dev.home.keithjackson.dev gitea.dev.home.keithjackson.dev
|
||||
prod -> *.prod.home.keithjackson.dev gitea.prod.home.keithjackson.dev
|
||||
apex -> CNAME <svc>.home.keithjackson.dev -> <svc>.prod.home.keithjackson.dev
|
||||
```
|
||||
|
||||
**Prod *is* the apex. There is no `prod.` label and no CNAME layer at all** —
|
||||
`<svc>.home.keithjackson.dev` is prod's real name, not an alias for one. This **supersedes** the
|
||||
interim split committed earlier the same day (`c8c1d29`), which put prod on
|
||||
`*.prod.home.keithjackson.dev` behind apex CNAMEs pointing at it. That layer is deleted: fewer
|
||||
moving parts, no CNAME chain to keep in step, and prod ends up on the names users already type.
|
||||
### PHASE 1 — **prod only. Dev did not move.** (this change)
|
||||
|
||||
The cost of the inversion is that **dev must move**, and dev is the live cluster. The interim split
|
||||
was chosen precisely because it moved only the unbuilt environment; this one does not have that
|
||||
property, so the whole of §10's remaining risk now sits on dev's side, and it is sequenced below.
|
||||
`BASE_DOMAIN` is now real: every hostname in every Flux-reconciled manifest derives from it.
|
||||
|
||||
### The values, per environment
|
||||
| | `BASE_DOMAIN` | `EXTERNALDNS_TXT_OWNER_ID` | `EXTERNALDNS_DOMAIN_FILTER` |
|
||||
|---|---|---|---|
|
||||
| base default | `home.keithjackson.dev` | `unset-see-cluster-vars` (non-colliding sentinel) | `keithjackson.dev` |
|
||||
| **dev** | `home.keithjackson.dev` (**unchanged**) | `default` (**must not change**) | `keithjackson.dev` |
|
||||
| **prod** | `prod.home.keithjackson.dev` | `prod` | `prod.home.keithjackson.dev` |
|
||||
|
||||
| | `BASE_DOMAIN` | `EXTERNALDNS_TXT_OWNER_ID` | `EXTERNALDNS_DOMAIN_FILTER` | Ansible `k3s_base_domain` |
|
||||
|---|---|---|---|---|
|
||||
| base default | `dev.home.keithjackson.dev` | `unset-see-cluster-vars` (non-colliding sentinel) | `keithjackson.dev` | `dev.home.keithjackson.dev` (role default) |
|
||||
| **dev** | `dev.home.keithjackson.dev` | `dev` | `keithjackson.dev` | (inherits the role default) |
|
||||
| **prod** | `home.keithjackson.dev` | `prod` | `keithjackson.dev` | `home.keithjackson.dev` (`group_vars/k3s_prod`) |
|
||||
**Dev's effective hostname is deliberately still the apex `home.keithjackson.dev`.** Moving prod
|
||||
alone ends the collision and unblocks prod, and it is the whole of the risk that had to be taken
|
||||
while the operator is away. Dev's rendered output is **byte-identical** except for one added line,
|
||||
`txtOwnerId: default`, which is the value dev's external-dns was already using implicitly — verified
|
||||
by rendering all 12 `substituteFrom` paths through `kubectl kustomize | flux envsubst --strict` for
|
||||
both overlays and diffing.
|
||||
|
||||
`[parity]`: `k3s_base_domain` must equal the same cluster's `BASE_DOMAIN`. The kube-apiserver's
|
||||
`oidc-issuer-url` is a k3s **server flag** in `/etc/rancher/k3s/config.yaml`, not a manifest, so Flux
|
||||
post-build substitution can never reach it — hence one fact in two places. If they disagree the
|
||||
apiserver validates tokens against a different Keycloak than the cluster runs: kubectl SSO fails
|
||||
while the cert-based admin kubeconfig keeps working, so it presents as "SSO is broken", not as skew.
|
||||
**Two independent guards now, either sufficient alone:**
|
||||
|
||||
**`EXTERNALDNS_DOMAIN_FILTER` is the Cloudflare ZONE for BOTH, and must stay that way.** FINDING
|
||||
F-DNS1 (DECISIONS-LOG 2026-08-17): the Cloudflare provider uses this value for **zone selection** as
|
||||
well as record matching, so a filter that is a strict subdomain of the hosted zone matches **no
|
||||
zone** — external-dns generates the record and then skips it with *"no hosted zone matching record
|
||||
DNS Name"*, publishing **nothing**. This has been "hardened" and reverted more than once. It is not
|
||||
defence in depth; it is a total outage of DNS publication. Isolation comes from the two guards below,
|
||||
which a domain filter cannot provide anyway: both environments live in the one zone regardless.
|
||||
1. **`txtOwnerId`** — per-environment identity. A cluster only ever touches records it owns. This is
|
||||
the direct fix for the incident. Dev **must** render exactly `default`: its live records are
|
||||
already registered under that owner, and any other value would make dev's own external-dns stop
|
||||
recognising them and re-create duplicates beside the orphans.
|
||||
2. **`domainFilters`** — per-environment name scope. Prod cannot see a name outside its own subtree.
|
||||
Defence in depth, added because the first guard is a single string in a single file.
|
||||
|
||||
**The two guards that actually work:**
|
||||
|
||||
1. **`txtOwnerId`, per-environment identity** — a cluster only ever reconciles records its own owner
|
||||
id claims in the TXT registry. This is the direct fix for the incident.
|
||||
2. **Disjoint hostname sets** — after the migration the two clusters never describe the same record,
|
||||
so there is nothing to fight over. *This guard does not exist during the migration window*, which
|
||||
is the entire reason the ordering below is mandatory.
|
||||
|
||||
#### Why dev's owner id changed from `default` to `dev`
|
||||
|
||||
Until this migration dev **had** to render `default`: its live apex records were already registered
|
||||
in Cloudflare's TXT registry under that owner, and renaming would have orphaned every one of them.
|
||||
**The migration dissolves that constraint** — on `*.dev.home.keithjackson.dev` dev publishes an
|
||||
entirely new record set with no prior registry entries to recognise, so the id is free and is chosen
|
||||
on its merits. `default` has none: it is the value external-dns uses when you **forget** the flag,
|
||||
which is exactly how both clusters came to share it, and an environment whose identity is the
|
||||
forget-value can never be told apart from a misconfigured newcomer. Consequences, both deliberate:
|
||||
|
||||
- Dev's **old apex records stay owned by `default`**, which no environment now claims, so nothing
|
||||
deletes them implicitly on some future reconcile. Retiring them is an explicit step (**D7**) and
|
||||
prod's first publish is gated on that retirement being *confirmed* — not on two reconcile loops
|
||||
racing each other.
|
||||
- **Prod stays `prod` and must never be given `default`.** Prod is taking over the apex *names*, and
|
||||
dev's existing apex records are registered under `default` — so `default` is the one value that
|
||||
would let prod adopt them, and under `policy: sync` delete the ones no prod source claims, on a
|
||||
cluster dev is still serving. That is the 2026-09-04 incident re-armed with the roles swapped.
|
||||
Prod inherits the apex **names**; it must never inherit the apex **owner id**.
|
||||
|
||||
All three values differ, and the base default remains a sentinel so an environment that forgets the
|
||||
knob owns nothing and can therefore delete nothing.
|
||||
|
||||
### PHASE 1 — prod tokenized and moved off dev's names (`c8c1d29`, done; suffix since inverted)
|
||||
|
||||
`BASE_DOMAIN` was a **dead knob** — declared in `clusters/base/cluster-vars.yaml` and referenced by
|
||||
no reconciled manifest, because every hostname was a hardcoded literal. `c8c1d29` made it real (141
|
||||
references across the reconciled set) and gave prod its own suffix. That work stands; only the
|
||||
*value* prod holds has changed, from `prod.home.keithjackson.dev` to the apex.
|
||||
The base default for the owner id is a **non-colliding sentinel, deliberately not `default`**: a
|
||||
future environment that forgets the knob must not silently inherit dev's identity — that is exactly
|
||||
what happened here. With the sentinel such a cluster owns nothing, so it can delete nothing.
|
||||
|
||||
### What could NOT be tokenized, and why
|
||||
|
||||
@@ -1557,49 +1528,49 @@ references across the reconciled set) and gave prod its own suffix. That work st
|
||||
Re-encrypting needs the operator's age key, rewrites the MAC over every value in the file, and
|
||||
**cannot be verified by the offline render harness** (`kustomize` does not decrypt), so a mistake
|
||||
would break Headlamp SSO on the live dev cluster with no pre-flight proof. Left as-is deliberately.
|
||||
**Both environments now need action:** dev's copy must move to `keycloak.dev.home.keithjackson.dev`
|
||||
as part of step **D5**, and prod needs its own copy with `keycloak.home.keithjackson.dev` and a
|
||||
client secret from prod's own realm — which it needs regardless, since every `*.sops.yaml` client
|
||||
secret is realm-scoped.
|
||||
**Prod action required:** prod's Headlamp will otherwise authenticate against **dev's** Keycloak.
|
||||
Prod needs its own copy of this Secret with `keycloak.prod.home.keithjackson.dev` and a client
|
||||
secret from prod's own realm — which it needs regardless, since every `*.sops.yaml` client secret
|
||||
is per-realm.
|
||||
- **`oidc-issuer-url` in the kube-apiserver flags** (`roles/k3s_server/tasks/main.yml`) — a k3s
|
||||
server flag, not a manifest. Handled on the **Ansible** seam instead: role default
|
||||
`k3s_base_domain` (dev's value) + derived `k3s_oidc_issuer_url`, overridden for prod in
|
||||
`inventory/group_vars/k3s_prod/vars.yml` — the same inventory seam as `k3s_data_dir`/`k3s_api_host`,
|
||||
so a fourth prod server inherits it from group membership. Verified per host by a read-only probe
|
||||
(`connection: local`, zero SSH, `changed=0`): all three dev servers resolve
|
||||
`https://keycloak.dev.home.keithjackson.dev/realms/homelab`, all three prod servers resolve
|
||||
`https://keycloak.home.keithjackson.dev/realms/homelab`.
|
||||
- **`spec.url` of dev's `GitRepository`** (`clusters/dev/kustomization.yaml`) — see the strand below.
|
||||
It *cannot* be tokenized: the `flux-system` root Kustomization that renders `./clusters/dev`
|
||||
declares no `postBuild.substituteFrom`, so `gitea.${BASE_DOMAIN}` there would ship the **literal
|
||||
token** and break the source outright (the L0 `flux-postbuild-substitution` check asserts exactly
|
||||
this). It is a hand-maintained literal that must move in lockstep with `BASE_DOMAIN`.
|
||||
**server flag** in `/etc/rancher/k3s/config.yaml`, not a manifest, so Flux post-build substitution
|
||||
can never reach it. Correctly tokenized on the **Ansible** side instead: new role default
|
||||
`k3s_base_domain` (dev value) + derived `k3s_oidc_issuer_url`, overridden for prod in
|
||||
`inventory/group_vars/k3s_prod/vars.yml` — the same inventory seam as `k3s_data_dir`/`k3s_api_host`.
|
||||
`[parity]`: `k3s_base_domain` **must** equal the same cluster's `BASE_DOMAIN`, or the apiserver
|
||||
validates tokens against a different Keycloak than the cluster runs and kubectl SSO fails while the
|
||||
cert-based admin kubeconfig keeps working (so it looks like "SSO is broken", not like config skew).
|
||||
|
||||
---
|
||||
### A pre-existing prod problem this incidentally FIXED
|
||||
|
||||
## 10a. THE MIGRATION RUNBOOK — ordering is load-bearing
|
||||
§4 recorded that prod nodes use **public** resolvers, so the kube-apiserver would resolve its OIDC
|
||||
issuer publicly and get **dev's** Gateway VIP (192.168.1.53) — the wrong cluster on the wrong LAN,
|
||||
making prod kubectl SSO impossible, and requiring an `/etc/hosts` pin as a workaround. With prod on
|
||||
its own subdomain the public record for `keycloak.prod.home.keithjackson.dev` is published by
|
||||
**prod's own** external-dns pointing at **prod's** Gateway VIP (192.168.2.20), so the public answer
|
||||
and the split-horizon answer agree. **No `/etc/hosts` pin is needed.** This holds only once prod's
|
||||
external-dns has actually published the record; until then use the break-glass cert kubeconfig.
|
||||
|
||||
> **Prod's `platform` tier and its 10 downstream tiers are SUSPENDED and must stay suspended until
|
||||
> step P1.** Nothing below is safe to run out of order, and nothing below should be run while the
|
||||
> operator is away.
|
||||
### Verify on prod's first reconcile
|
||||
|
||||
### The two ways this goes wrong
|
||||
- [ ] **`domainFilters` zone selection.** The Cloudflare provider uses this value for *zone* lookup
|
||||
as well as record matching, and in 2026-08 a strict-subdomain filter dropped the parent zone
|
||||
("no hosted zone matching record DNS Name"). If prod's external-dns logs no matching zone it
|
||||
will create **nothing** — fail-closed, never destructive. Fallback: set prod's
|
||||
`EXTERNALDNS_DOMAIN_FILTER` back to `keithjackson.dev` and rely on `txtOwnerId`, which is
|
||||
sufficient on its own.
|
||||
- [ ] **Prod's wildcard cert is `*.prod.home.keithjackson.dev`** and solves DNS-01 without touching
|
||||
dev's `_acme-challenge` records.
|
||||
- [ ] **Dev's records are untouched** after prod's tiers are resumed. Confirm dev's TXT registry
|
||||
entries still read owner `default` and that prod's read owner `prod`.
|
||||
- [ ] **Prod's own Keycloak realm + per-service client secrets** exist (every `*.sops.yaml` client
|
||||
secret is realm-scoped; prod cannot reuse dev's).
|
||||
|
||||
**(1) BOTH CLUSTERS ON THE APEX — the 2026-09-04 incident, recreated.** If prod publishes
|
||||
`*.home.keithjackson.dev` before dev has vacated it, two clusters describe the same records in the
|
||||
same zone again. Distinct `txtOwnerId`s stop prod from *deleting* dev's records — that guard holds —
|
||||
but they do not stop the second failure mode: both clusters requesting a `letsencrypt-prod` wildcard
|
||||
for the **same name set** and racing each other's `_acme-challenge` TXT records, so neither reliably
|
||||
solves DNS-01. **Rule: dev vacates the apex first, and the vacate is verified against Cloudflare
|
||||
before prod is unsuspended.**
|
||||
### PHASE 2 — dev migration + apex CNAMEs (**deferred, not done**)
|
||||
|
||||
**(2) THE STRAND — dev's GitOps source is the thing that moves.** `spec.url` of dev's
|
||||
`GitRepository` is `https://gitea.home.keithjackson.dev/...`, which is **both** dev's GitOps source
|
||||
**and** the PM's push target. Flipping dev's `BASE_DOMAIN` moves Gitea's HTTPRoute, its public DNS
|
||||
record and the wildcard cert covering it to `gitea.dev.home.keithjackson.dev` all at once. If that
|
||||
host does not already resolve and serve a valid, Ready cert at that instant, **dev stops reconciling
|
||||
and cannot be fixed by pushing** — the broken thing *is* the path Flux uses to read pushes. Recovery
|
||||
is then hands-on and needs LAN access to the cluster.
|
||||
Deliberately not attempted here: dev is **live** (18 Longhorn volumes, working SSO) and it hosts the
|
||||
Gitea that is currently this repo's GitOps source. Moving it is a migration, not a variable change.
|
||||
Required, roughly in order:
|
||||
|
||||
**(2b) AND THE MERGE ITSELF IS THE TRIGGER.** `clusters/dev/sync.yaml` is a self-managing root
|
||||
Kustomization reconciling `./clusters/dev` from **`main`** every 10 minutes, and prod tracks `main`
|
||||
|
||||
@@ -306,47 +306,35 @@ node_baseline_dns_servers:
|
||||
# k3s_base_domain, below). It reaches the issuer through the NODE resolver, i.e. through
|
||||
# these PUBLIC upstreams — not through CoreDNS's split-horizon answer.
|
||||
#
|
||||
# ORDERING HAZARD, NOT YET CLEARED. Prod's issuer is keycloak.home.keithjackson.dev — an APEX
|
||||
# name, because prod IS the apex under the 2026-09-04 inversion (k3s_base_domain below).
|
||||
# That is the name the LIVE DEV CLUSTER answers today, and its public A record currently
|
||||
# points at DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN. A prod
|
||||
# node resolving its own issuer through these public upstreams therefore reaches dev until:
|
||||
# (1) dev has migrated to *.dev.home.keithjackson.dev and RETIRED its apex records, and
|
||||
# (2) PROD's own external-dns has published keycloak.home.keithjackson.dev at PROD's
|
||||
# Gateway VIP (192.168.2.20).
|
||||
# Only then do the public answer and the in-cluster split-horizon answer AGREE, and no
|
||||
# /etc/hosts pin is needed. Until then kubectl OIDC SSO on prod cannot work; the cert-based
|
||||
# admin kubeconfig is the break-glass path and always works. Ordering, not a design gap —
|
||||
# the ordered steps are infra/PROD-PROVISIONING.md §10.
|
||||
# THIS USED TO BE BROKEN, AND THE 2026-09-04 SUBDOMAIN SPLIT FIXED IT. While prod was
|
||||
# authored to serve dev's hostnames, a prod node resolving keycloak.home.keithjackson.dev
|
||||
# publicly got DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN, so
|
||||
# kubectl OIDC SSO on prod could not work at all. Now prod's issuer is
|
||||
# keycloak.prod.home.keithjackson.dev, and the public record for that name is published by
|
||||
# PROD's own external-dns pointing at PROD's Gateway VIP (192.168.2.20). The public answer
|
||||
# and the split-horizon answer therefore AGREE, and no /etc/hosts pin is needed.
|
||||
#
|
||||
# STILL TO VERIFY on prod bring-up (PROD-PROVISIONING.md §10): this holds only once prod's
|
||||
# external-dns has actually published the record. Until then the issuer does not resolve
|
||||
# from the nodes and kubectl SSO fails — the cert-based admin kubeconfig is the break-glass
|
||||
# path, and it always works. Ordering, not a design gap.
|
||||
node_baseline_dns_search: []
|
||||
|
||||
# --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
|
||||
# PROD **IS** THE APEX. [parity] MUST equal BASE_DOMAIN in clusters/prod/kustomization.yaml —
|
||||
# the same one-fact-in-two-places contract as k3s_data_dir and k3s_api_host. The Flux side
|
||||
# cannot supply this: the kube-apiserver's oidc-issuer-url is a k3s server FLAG written to
|
||||
# /etc/rancher/k3s/config.yaml, not a manifest, so postBuild.substituteFrom never sees it.
|
||||
# Full rationale in roles/k3s_server/defaults/main.yml, where k3s_oidc_issuer_url derives.
|
||||
# PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/
|
||||
# kustomization.yaml — the same one-fact-in-two-places contract as k3s_data_dir and
|
||||
# k3s_api_host. The Flux side cannot supply this: the kube-apiserver's oidc-issuer-url is a
|
||||
# k3s server FLAG written to /etc/rancher/k3s/config.yaml, not a manifest, so
|
||||
# postBuild.substituteFrom never sees it. Full rationale in
|
||||
# roles/k3s_server/defaults/main.yml, where k3s_oidc_issuer_url is derived from this.
|
||||
#
|
||||
# WHY THE SUFFIXES SPLIT (infra/PROD-PROVISIONING.md §10): both clusters were authored to serve
|
||||
# the SAME hostnames from the SAME Cloudflare zone, and on 2026-09-04 prod's external-dns
|
||||
# deleted every one of dev's public records.
|
||||
#
|
||||
# WHICH WAY ROUND — INVERTED 2026-09-04 by operator decision, superseding the interim split
|
||||
# committed earlier the same day (prod on `prod.home.` behind apex CNAMEs). The simpler target:
|
||||
# prod -> home.keithjackson.dev (THE APEX; this variable — no `prod.` label, no CNAMEs)
|
||||
# dev -> dev.home.keithjackson.dev (roles/k3s_server/defaults/main.yml)
|
||||
# So prod's Keycloak issuer is https://keycloak.home.keithjackson.dev/realms/homelab, and the
|
||||
# `kubernetes` OIDC client in PROD's OWN Keycloak realm is what prod's kubectl SSO authenticates
|
||||
# against.
|
||||
#
|
||||
# ORDERING HAZARD SPECIFIC TO THIS VARIABLE. keycloak.home.keithjackson.dev is the name the LIVE
|
||||
# dev cluster answers today, and prod's nodes resolve the issuer through PUBLIC upstreams (see
|
||||
# node_baseline_dns_search above). So until dev has vacated the apex and PROD's external-dns has
|
||||
# published that name at PROD's Gateway VIP (192.168.2.20), a prod node resolving its own issuer
|
||||
# gets DEV's VIP (192.168.1.53) — the wrong cluster, on the wrong LAN. kubectl OIDC SSO on prod
|
||||
# therefore does not work until §10's dev-vacate steps are complete; the cert-based admin
|
||||
# kubeconfig is the break-glass path and always works. Ordering, not a design gap.
|
||||
k3s_base_domain: "home.keithjackson.dev"
|
||||
# WHY PROD MOVED (infra/PROD-PROVISIONING.md §10): both clusters were authored to serve the
|
||||
# SAME hostnames from the SAME Cloudflare zone, and on 2026-09-04 prod's external-dns
|
||||
# deleted every one of dev's public records. Dev stays on home.keithjackson.dev (it is LIVE);
|
||||
# prod takes prod.home.keithjackson.dev. Consequence here: prod's Keycloak issuer is
|
||||
# https://keycloak.prod.home.keithjackson.dev/realms/homelab, so the `kubernetes` OIDC client
|
||||
# in prod's OWN Keycloak realm is what prod's kubectl SSO authenticates against.
|
||||
k3s_base_domain: "prod.home.keithjackson.dev"
|
||||
|
||||
# --- Flux bootstrap: WHICH overlay this environment reconciles from (R-ENV-02) -----
|
||||
# roles/flux_bootstrap defaults flux_bootstrap_env_path to `clusters/dev` (dev is LIVE
|
||||
|
||||
@@ -95,26 +95,15 @@ k3s_tls_sans: >-
|
||||
# while the cert-based admin kubeconfig keeps working — so it looks like "SSO is broken",
|
||||
# not like a config skew. The break-glass cert kubeconfig is why this is recoverable.
|
||||
#
|
||||
# THE DEFAULT IS THE DEV VALUE — the same convention as k3s_data_dir above and cluster-vars'
|
||||
# GATEWAY_VIP / LONGHORN_DATA_PATH: the role default is dev's value and prod overrides it in
|
||||
# inventory/group_vars/k3s_prod/vars.yml (the same inventory seam k3s_data_dir / k3s_api_host /
|
||||
# kube_vip_address use, so a FOURTH prod server inherits it from group membership rather than
|
||||
# from anyone remembering).
|
||||
# THE DEFAULT IS THE DEV VALUE, restated explicitly rather than left implicit — the same
|
||||
# convention as k3s_data_dir above and cluster-vars' GATEWAY_VIP / LONGHORN_DATA_PATH: the
|
||||
# base default is the known-good LIVE value, so dev's rendered config.yaml is byte-for-byte
|
||||
# what it already is and the Restart handler does NOT fire on a re-run.
|
||||
#
|
||||
# *** WHICH DEV VALUE — UPDATED 2026-09-04, AND IT IS AHEAD OF THE LIVE DEV CLUSTER. ***
|
||||
# The operator inverted the suffix split: PROD is now the apex `home.keithjackson.dev` and DEV
|
||||
# carries the label `dev.home.keithjackson.dev`. This default therefore moved from the apex to
|
||||
# dev's subdomain, mirroring BASE_DOMAIN in clusters/dev/kustomization.yaml.
|
||||
#
|
||||
# CONSEQUENCE — this default no longer matches what the live dev nodes are running, so unlike
|
||||
# every previous edit to this file A DEV RE-RUN IS NOT A NO-OP. Rendering config.yaml with the
|
||||
# new issuer CHANGES the file, FIRES the Restart handler, and rolls k3s on all three dev
|
||||
# servers, which also invalidates every user's OIDC kubeconfig (SSO must be re-issued). That is
|
||||
# a deliberate step of the dev DNS migration, NOT a side effect to discover during unrelated
|
||||
# work: DO NOT run the k3s_server role against dev until step D6 of
|
||||
# infra/PROD-PROVISIONING.md §10, and never while the operator is away. The break-glass
|
||||
# cert-based admin kubeconfig keeps working throughout, which is what makes the step safe.
|
||||
k3s_base_domain: "dev.home.keithjackson.dev"
|
||||
# PROD OVERRIDES IT in inventory/group_vars/k3s_prod/vars.yml — the same inventory seam
|
||||
# k3s_data_dir / k3s_api_host / kube_vip_address use, so a FOURTH prod server inherits it
|
||||
# from group membership rather than from anyone remembering.
|
||||
k3s_base_domain: "home.keithjackson.dev"
|
||||
|
||||
# Derived, so the realm path is stated once. Override k3s_base_domain, not this.
|
||||
k3s_oidc_issuer_url: "https://keycloak.{{ k3s_base_domain }}/realms/homelab"
|
||||
Reference in new issue
Block a user