From e65a76dc5118ba2fac6419cfcc2e9d37a5b894e8 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 7 Sep 2026 02:10:10 -0500 Subject: [PATCH] docs(log): record the prod OIDC-URL outage and the rule it establishes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Non-secret, environment-specific values must not live inside an encrypted Secret: Flux's ${BASE_DOMAIN} substitution rewrites rendered manifest text and cannot reach inside ciphertext, so such a value silently freezes at whatever one environment happened to need. Dev was correct only by coincidence (dev IS the apex); prod was wrong from its first reconcile, for both Headlamp and Semaphore. StrictPostBuildSubstitutions cannot catch this — there is no unresolved ${...} to complain about. Co-Authored-By: Claude Opus 5 --- DECISIONS-LOG.md | 89 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index cf00eac..f9aedc0 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -1758,3 +1758,92 @@ into a transient hard Kustomization failure. - **Prod's source URL cannot work on a cold prod cluster** — split-horizon answers the whole zone with `${GATEWAY_VIP}`, so prod would resolve `gitea.home.keithjackson.dev` to its own gateway. Bring-up needs a temporary source and a cutover, as `docs/05` describes for dev's own Phase-6. + +## 2026-09-07 — LIVE BUG: prod's OIDC clients were pointed at DEV's Keycloak. Secrets are a substitution blind spot. + +The operator could not log in to Headlamp on prod: + +``` +Get "https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration": +dial tcp 192.168.1.53:443: i/o timeout +``` + +`keycloak.home.keithjackson.dev` is **dev's** Keycloak. It resolves to `192.168.1.53`, on the dev +LAN behind pfSense; prod lives on `192.168.2.0/24` (pfSense's WAN) and cannot route there, so the +discovery fetch simply hung until it timed out. + +### Root cause — the class, not the instance + +**Flux's `${BASE_DOMAIN}` postBuild substitution rewrites the RENDERED MANIFEST TEXT. It can never +reach inside a SOPS-encrypted Secret.** Environment-specific URLs had been stored *inside* two +encrypted Secrets purely because it was convenient to keep one blob per app: + +| File | Buried, un-substitutable values | +|---|---| +| `platform/headlamp/headlamp-oidc-credentials.sops.yaml` | `OIDC_ISSUER_URL`, `OIDC_CALLBACK_URL` | +| `apps/wave-1/semaphore-bootstrap-secret.sops.yaml` (`oidc-providers-json`) | the realm issuer + the OIDC redirect URI | + +Both were frozen at the apex `home.keithjackson.dev`. **Dev IS the apex, so dev was correct by +coincidence** — the values were never per-environment, they just happened to match one environment. +Prod was wrong from the first reconcile. Semaphore on prod was broken the same way and nobody had +noticed, because nobody had tried prod SSO yet. + +This is the same failure shape as the 2026-09-01 nextcloud guard: **a value that silently resolves +to something plausible is worse than one that fails.** `StrictPostBuildSubstitutions` cannot catch +it either — there is no `${...}` left unresolved to complain about; the ciphertext never contained +one. + +### The rule this establishes + +> **Non-secret, environment-specific values MUST NOT live inside an encrypted Secret.** Issuer URLs, +> callback URLs, redirect URIs and hostnames are public. Encrypt the secret half only, and template +> the rest where `${BASE_DOMAIN}` can reach it. + +### The fix + +- **Headlamp** — `OIDC_ISSUER_URL` / `OIDC_CALLBACK_URL` moved to the HelmRelease's + `spec.values.env`, templated `${BASE_DOMAIN}`. The Secret keeps `OIDC_CLIENT_SECRET` + + the environment-invariant `OIDC_CLIENT_ID` / `OIDC_SCOPES`, still `envFrom`'d. + **Exactly one source per variable** — an explicit `env:` entry silently outranks the same name + arriving via `envFrom`, so the two sets deliberately do not overlap. + Side effect, intended: naming `OIDC_CALLBACK_URL` in `env:` makes chart 0.41.0 finally emit + `-oidc-callback-url=$(OIDC_CALLBACK_URL)`; it gates that arg on `env:`/`config.oidc`, which a + Secret key alone never tripped — so the Secret's callback URL had been dead weight all along. +- **Semaphore** — the `SEMAPHORE_OIDC_PROVIDERS` JSON is now a plaintext literal in + `catalog/semaphore.yaml` (re-rendered to the `apps/wave-1` golden). Only the client secret stays + encrypted, in its own key `oidc-client-secret`, and the **kubelet** splices it back via + dependent-env-var expansion: `$(SEMAPHORE_OIDC_CLIENT_SECRET)` resolves against the `secretKeyRef` + env var declared immediately above it. **Order is load-bearing** — the kubelet resolves `env` in + list order and leaves a forward reference as literal text. `flux envsubst` ignores `$(...)`, so + the two expansion layers do not collide (the `$${VAR}` escape in `apps/wave-1/nextcloud.yaml` + documents the one form that DOES need escaping). + +### Proof + +Rendered per overlay (`kubectl kustomize --load-restrictor LoadRestrictionsNone`, then +`flux envsubst --strict` with that overlay's `cluster-vars`), plus `helm template` of headlamp +0.41.0 against the substituted values: + +| | dev | prod | +|---|---|---| +| Headlamp `OIDC_ISSUER_URL` | `https://keycloak.home.keithjackson.dev/realms/homelab` | `https://keycloak.prod.home.keithjackson.dev/realms/homelab` | +| Headlamp `OIDC_CALLBACK_URL` | `https://headlamp.home.keithjackson.dev/oidc-callback` | `https://headlamp.prod.home.keithjackson.dev/oidc-callback` | +| Semaphore `provider_url` | `https://keycloak.home.keithjackson.dev/realms/homelab` | `https://keycloak.prod.home.keithjackson.dev/realms/homelab` | +| Semaphore `redirect_url` | `https://semaphore.home.keithjackson.dev/api/...` | `https://semaphore.prod.home.keithjackson.dev/api/...` | + +Dev's values are byte-identical to what it runs today — this change moves prod only. + +### [parity] the apiserver invariant HOLDS + +Prod's kube-apiserver `oidc-issuer-url` is an **install-time k3s flag** derived from +`k3s_base_domain` (`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` = `prod.home.keithjackson.dev`), +not from `cluster-vars` — it cannot read the Flux variable. Headlamp forwards the user's `id_token` +to the apiserver, so the two must name the same issuer. After this change they do, in both +environments, because both derive from the same per-environment domain. Unchanged risk, now written +down in `platform/headlamp/README.md`: moving one without the other 401s every request. + +### Also fixed for the next person + +`headlamp-oidc-credentials.sops.yaml.example` used to *teach* the broken pattern — it listed +`OIDC_ISSUER_URL` and `OIDC_CALLBACK_URL` as keys to fill in. It now says, in the file, that adding +either key back re-breaks prod.