docs(log): record the prod OIDC-URL outage and the rule it establishes
Non-secret, environment-specific values must not live inside an encrypted
Secret: Flux's ${BASE_DOMAIN} substitution rewrites rendered manifest text and
cannot reach inside ciphertext, so such a value silently freezes at whatever
one environment happened to need. Dev was correct only by coincidence (dev IS
the apex); prod was wrong from its first reconcile, for both Headlamp and
Semaphore. StrictPostBuildSubstitutions cannot catch this — there is no
unresolved ${...} to complain about.
Co-Authored-By: Claude Opus 5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed
1 parent
6548f9a89c
commit
e65a76dc51
1 file changed
+89
@@ -1758,3 +1758,92 @@ into a transient hard Kustomization failure.
|
|||||||
- **Prod's source URL cannot work on a cold prod cluster** — split-horizon answers the whole zone
|
- **Prod's source URL cannot work on a cold prod cluster** — split-horizon answers the whole zone
|
||||||
with `${GATEWAY_VIP}`, so prod would resolve `gitea.home.keithjackson.dev` to its own gateway.
|
with `${GATEWAY_VIP}`, so prod would resolve `gitea.home.keithjackson.dev` to its own gateway.
|
||||||
Bring-up needs a temporary source and a cutover, as `docs/05` describes for dev's own Phase-6.
|
Bring-up needs a temporary source and a cutover, as `docs/05` describes for dev's own Phase-6.
|
||||||
|
|
||||||
|
## 2026-09-07 — LIVE BUG: prod's OIDC clients were pointed at DEV's Keycloak. Secrets are a substitution blind spot.
|
||||||
|
|
||||||
|
The operator could not log in to Headlamp on prod:
|
||||||
|
|
||||||
|
```
|
||||||
|
Get "https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration":
|
||||||
|
dial tcp 192.168.1.53:443: i/o timeout
|
||||||
|
```
|
||||||
|
|
||||||
|
`keycloak.home.keithjackson.dev` is **dev's** Keycloak. It resolves to `192.168.1.53`, on the dev
|
||||||
|
LAN behind pfSense; prod lives on `192.168.2.0/24` (pfSense's WAN) and cannot route there, so the
|
||||||
|
discovery fetch simply hung until it timed out.
|
||||||
|
|
||||||
|
### Root cause — the class, not the instance
|
||||||
|
|
||||||
|
**Flux's `${BASE_DOMAIN}` postBuild substitution rewrites the RENDERED MANIFEST TEXT. It can never
|
||||||
|
reach inside a SOPS-encrypted Secret.** Environment-specific URLs had been stored *inside* two
|
||||||
|
encrypted Secrets purely because it was convenient to keep one blob per app:
|
||||||
|
|
||||||
|
| File | Buried, un-substitutable values |
|
||||||
|
|---|---|
|
||||||
|
| `platform/headlamp/headlamp-oidc-credentials.sops.yaml` | `OIDC_ISSUER_URL`, `OIDC_CALLBACK_URL` |
|
||||||
|
| `apps/wave-1/semaphore-bootstrap-secret.sops.yaml` (`oidc-providers-json`) | the realm issuer + the OIDC redirect URI |
|
||||||
|
|
||||||
|
Both were frozen at the apex `home.keithjackson.dev`. **Dev IS the apex, so dev was correct by
|
||||||
|
coincidence** — the values were never per-environment, they just happened to match one environment.
|
||||||
|
Prod was wrong from the first reconcile. Semaphore on prod was broken the same way and nobody had
|
||||||
|
noticed, because nobody had tried prod SSO yet.
|
||||||
|
|
||||||
|
This is the same failure shape as the 2026-09-01 nextcloud guard: **a value that silently resolves
|
||||||
|
to something plausible is worse than one that fails.** `StrictPostBuildSubstitutions` cannot catch
|
||||||
|
it either — there is no `${...}` left unresolved to complain about; the ciphertext never contained
|
||||||
|
one.
|
||||||
|
|
||||||
|
### The rule this establishes
|
||||||
|
|
||||||
|
> **Non-secret, environment-specific values MUST NOT live inside an encrypted Secret.** Issuer URLs,
|
||||||
|
> callback URLs, redirect URIs and hostnames are public. Encrypt the secret half only, and template
|
||||||
|
> the rest where `${BASE_DOMAIN}` can reach it.
|
||||||
|
|
||||||
|
### The fix
|
||||||
|
|
||||||
|
- **Headlamp** — `OIDC_ISSUER_URL` / `OIDC_CALLBACK_URL` moved to the HelmRelease's
|
||||||
|
`spec.values.env`, templated `${BASE_DOMAIN}`. The Secret keeps `OIDC_CLIENT_SECRET` +
|
||||||
|
the environment-invariant `OIDC_CLIENT_ID` / `OIDC_SCOPES`, still `envFrom`'d.
|
||||||
|
**Exactly one source per variable** — an explicit `env:` entry silently outranks the same name
|
||||||
|
arriving via `envFrom`, so the two sets deliberately do not overlap.
|
||||||
|
Side effect, intended: naming `OIDC_CALLBACK_URL` in `env:` makes chart 0.41.0 finally emit
|
||||||
|
`-oidc-callback-url=$(OIDC_CALLBACK_URL)`; it gates that arg on `env:`/`config.oidc`, which a
|
||||||
|
Secret key alone never tripped — so the Secret's callback URL had been dead weight all along.
|
||||||
|
- **Semaphore** — the `SEMAPHORE_OIDC_PROVIDERS` JSON is now a plaintext literal in
|
||||||
|
`catalog/semaphore.yaml` (re-rendered to the `apps/wave-1` golden). Only the client secret stays
|
||||||
|
encrypted, in its own key `oidc-client-secret`, and the **kubelet** splices it back via
|
||||||
|
dependent-env-var expansion: `$(SEMAPHORE_OIDC_CLIENT_SECRET)` resolves against the `secretKeyRef`
|
||||||
|
env var declared immediately above it. **Order is load-bearing** — the kubelet resolves `env` in
|
||||||
|
list order and leaves a forward reference as literal text. `flux envsubst` ignores `$(...)`, so
|
||||||
|
the two expansion layers do not collide (the `$${VAR}` escape in `apps/wave-1/nextcloud.yaml`
|
||||||
|
documents the one form that DOES need escaping).
|
||||||
|
|
||||||
|
### Proof
|
||||||
|
|
||||||
|
Rendered per overlay (`kubectl kustomize --load-restrictor LoadRestrictionsNone`, then
|
||||||
|
`flux envsubst --strict` with that overlay's `cluster-vars`), plus `helm template` of headlamp
|
||||||
|
0.41.0 against the substituted values:
|
||||||
|
|
||||||
|
| | dev | prod |
|
||||||
|
|---|---|---|
|
||||||
|
| Headlamp `OIDC_ISSUER_URL` | `https://keycloak.home.keithjackson.dev/realms/homelab` | `https://keycloak.prod.home.keithjackson.dev/realms/homelab` |
|
||||||
|
| Headlamp `OIDC_CALLBACK_URL` | `https://headlamp.home.keithjackson.dev/oidc-callback` | `https://headlamp.prod.home.keithjackson.dev/oidc-callback` |
|
||||||
|
| Semaphore `provider_url` | `https://keycloak.home.keithjackson.dev/realms/homelab` | `https://keycloak.prod.home.keithjackson.dev/realms/homelab` |
|
||||||
|
| Semaphore `redirect_url` | `https://semaphore.home.keithjackson.dev/api/...` | `https://semaphore.prod.home.keithjackson.dev/api/...` |
|
||||||
|
|
||||||
|
Dev's values are byte-identical to what it runs today — this change moves prod only.
|
||||||
|
|
||||||
|
### [parity] the apiserver invariant HOLDS
|
||||||
|
|
||||||
|
Prod's kube-apiserver `oidc-issuer-url` is an **install-time k3s flag** derived from
|
||||||
|
`k3s_base_domain` (`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` = `prod.home.keithjackson.dev`),
|
||||||
|
not from `cluster-vars` — it cannot read the Flux variable. Headlamp forwards the user's `id_token`
|
||||||
|
to the apiserver, so the two must name the same issuer. After this change they do, in both
|
||||||
|
environments, because both derive from the same per-environment domain. Unchanged risk, now written
|
||||||
|
down in `platform/headlamp/README.md`: moving one without the other 401s every request.
|
||||||
|
|
||||||
|
### Also fixed for the next person
|
||||||
|
|
||||||
|
`headlamp-oidc-credentials.sops.yaml.example` used to *teach* the broken pattern — it listed
|
||||||
|
`OIDC_ISSUER_URL` and `OIDC_CALLBACK_URL` as keys to fill in. It now says, in the file, that adding
|
||||||
|
either key back re-breaks prod.
|
||||||
Reference in new issue
Block a user