docs(log): record the prod OIDC-URL outage and the rule it establishes
Non-secret, environment-specific values must not live inside an encrypted
Secret: Flux's ${BASE_DOMAIN} substitution rewrites rendered manifest text and
cannot reach inside ciphertext, so such a value silently freezes at whatever
one environment happened to need. Dev was correct only by coincidence (dev IS
the apex); prod was wrong from its first reconcile, for both Headlamp and
Semaphore. StrictPostBuildSubstitutions cannot catch this — there is no
unresolved ${...} to complain about.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
6548f9a89c
commit
e65a76dc51
1 file changed
+89
@@ -1758,3 +1758,92 @@ into a transient hard Kustomization failure.
|
||||
- **Prod's source URL cannot work on a cold prod cluster** — split-horizon answers the whole zone
|
||||
with `${GATEWAY_VIP}`, so prod would resolve `gitea.home.keithjackson.dev` to its own gateway.
|
||||
Bring-up needs a temporary source and a cutover, as `docs/05` describes for dev's own Phase-6.
|
||||
|
||||
## 2026-09-07 — LIVE BUG: prod's OIDC clients were pointed at DEV's Keycloak. Secrets are a substitution blind spot.
|
||||
|
||||
The operator could not log in to Headlamp on prod:
|
||||
|
||||
```
|
||||
Get "https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration":
|
||||
dial tcp 192.168.1.53:443: i/o timeout
|
||||
```
|
||||
|
||||
`keycloak.home.keithjackson.dev` is **dev's** Keycloak. It resolves to `192.168.1.53`, on the dev
|
||||
LAN behind pfSense; prod lives on `192.168.2.0/24` (pfSense's WAN) and cannot route there, so the
|
||||
discovery fetch simply hung until it timed out.
|
||||
|
||||
### Root cause — the class, not the instance
|
||||
|
||||
**Flux's `${BASE_DOMAIN}` postBuild substitution rewrites the RENDERED MANIFEST TEXT. It can never
|
||||
reach inside a SOPS-encrypted Secret.** Environment-specific URLs had been stored *inside* two
|
||||
encrypted Secrets purely because it was convenient to keep one blob per app:
|
||||
|
||||
| File | Buried, un-substitutable values |
|
||||
|---|---|
|
||||
| `platform/headlamp/headlamp-oidc-credentials.sops.yaml` | `OIDC_ISSUER_URL`, `OIDC_CALLBACK_URL` |
|
||||
| `apps/wave-1/semaphore-bootstrap-secret.sops.yaml` (`oidc-providers-json`) | the realm issuer + the OIDC redirect URI |
|
||||
|
||||
Both were frozen at the apex `home.keithjackson.dev`. **Dev IS the apex, so dev was correct by
|
||||
coincidence** — the values were never per-environment, they just happened to match one environment.
|
||||
Prod was wrong from the first reconcile. Semaphore on prod was broken the same way and nobody had
|
||||
noticed, because nobody had tried prod SSO yet.
|
||||
|
||||
This is the same failure shape as the 2026-09-01 nextcloud guard: **a value that silently resolves
|
||||
to something plausible is worse than one that fails.** `StrictPostBuildSubstitutions` cannot catch
|
||||
it either — there is no `${...}` left unresolved to complain about; the ciphertext never contained
|
||||
one.
|
||||
|
||||
### The rule this establishes
|
||||
|
||||
> **Non-secret, environment-specific values MUST NOT live inside an encrypted Secret.** Issuer URLs,
|
||||
> callback URLs, redirect URIs and hostnames are public. Encrypt the secret half only, and template
|
||||
> the rest where `${BASE_DOMAIN}` can reach it.
|
||||
|
||||
### The fix
|
||||
|
||||
- **Headlamp** — `OIDC_ISSUER_URL` / `OIDC_CALLBACK_URL` moved to the HelmRelease's
|
||||
`spec.values.env`, templated `${BASE_DOMAIN}`. The Secret keeps `OIDC_CLIENT_SECRET` +
|
||||
the environment-invariant `OIDC_CLIENT_ID` / `OIDC_SCOPES`, still `envFrom`'d.
|
||||
**Exactly one source per variable** — an explicit `env:` entry silently outranks the same name
|
||||
arriving via `envFrom`, so the two sets deliberately do not overlap.
|
||||
Side effect, intended: naming `OIDC_CALLBACK_URL` in `env:` makes chart 0.41.0 finally emit
|
||||
`-oidc-callback-url=$(OIDC_CALLBACK_URL)`; it gates that arg on `env:`/`config.oidc`, which a
|
||||
Secret key alone never tripped — so the Secret's callback URL had been dead weight all along.
|
||||
- **Semaphore** — the `SEMAPHORE_OIDC_PROVIDERS` JSON is now a plaintext literal in
|
||||
`catalog/semaphore.yaml` (re-rendered to the `apps/wave-1` golden). Only the client secret stays
|
||||
encrypted, in its own key `oidc-client-secret`, and the **kubelet** splices it back via
|
||||
dependent-env-var expansion: `$(SEMAPHORE_OIDC_CLIENT_SECRET)` resolves against the `secretKeyRef`
|
||||
env var declared immediately above it. **Order is load-bearing** — the kubelet resolves `env` in
|
||||
list order and leaves a forward reference as literal text. `flux envsubst` ignores `$(...)`, so
|
||||
the two expansion layers do not collide (the `$${VAR}` escape in `apps/wave-1/nextcloud.yaml`
|
||||
documents the one form that DOES need escaping).
|
||||
|
||||
### Proof
|
||||
|
||||
Rendered per overlay (`kubectl kustomize --load-restrictor LoadRestrictionsNone`, then
|
||||
`flux envsubst --strict` with that overlay's `cluster-vars`), plus `helm template` of headlamp
|
||||
0.41.0 against the substituted values:
|
||||
|
||||
| | dev | prod |
|
||||
|---|---|---|
|
||||
| Headlamp `OIDC_ISSUER_URL` | `https://keycloak.home.keithjackson.dev/realms/homelab` | `https://keycloak.prod.home.keithjackson.dev/realms/homelab` |
|
||||
| Headlamp `OIDC_CALLBACK_URL` | `https://headlamp.home.keithjackson.dev/oidc-callback` | `https://headlamp.prod.home.keithjackson.dev/oidc-callback` |
|
||||
| Semaphore `provider_url` | `https://keycloak.home.keithjackson.dev/realms/homelab` | `https://keycloak.prod.home.keithjackson.dev/realms/homelab` |
|
||||
| Semaphore `redirect_url` | `https://semaphore.home.keithjackson.dev/api/...` | `https://semaphore.prod.home.keithjackson.dev/api/...` |
|
||||
|
||||
Dev's values are byte-identical to what it runs today — this change moves prod only.
|
||||
|
||||
### [parity] the apiserver invariant HOLDS
|
||||
|
||||
Prod's kube-apiserver `oidc-issuer-url` is an **install-time k3s flag** derived from
|
||||
`k3s_base_domain` (`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` = `prod.home.keithjackson.dev`),
|
||||
not from `cluster-vars` — it cannot read the Flux variable. Headlamp forwards the user's `id_token`
|
||||
to the apiserver, so the two must name the same issuer. After this change they do, in both
|
||||
environments, because both derive from the same per-environment domain. Unchanged risk, now written
|
||||
down in `platform/headlamp/README.md`: moving one without the other 401s every request.
|
||||
|
||||
### Also fixed for the next person
|
||||
|
||||
`headlamp-oidc-credentials.sops.yaml.example` used to *teach* the broken pattern — it listed
|
||||
`OIDC_ISSUER_URL` and `OIDC_CALLBACK_URL` as keys to fill in. It now says, in the file, that adding
|
||||
either key back re-breaks prod.
|
||||
Reference in new issue
Block a user