Task 1 — browser SSO "Could not reach OpenID Connect provider" (confirmed root
cause, not guessed). keycloak.home.keithjackson.dev resolves split-horizon to the
PRIVATE Cilium gateway VIP (192.168.1.53). Nextcloud's SSRF guard refuses its own
server-side HTTP requests to RFC1918 addresses, so user_oidc's discovery/JWKS/token
fetch AT LOGIN throws OCP\Http\Client\LocalServerException "violates local access
rules" -> surfaced to the user as "Could not reach the provider". Proven from the
live pod: nextcloud.log carries the exact LocalServerException from
DiscoveryService::obtainDiscovery with Guzzle option allow_local_address:false; a
Nextcloud-context IClientService->get() of the discovery URL returns
LocalServerException with the flag false and HTTP 200 with it true (A/B toggled).
The postStart (b) curl check passed regardless because it uses system libcurl, not
Nextcloud's guard — which is why the provider registered yet login still failed.
Fix (reproducible, survives pod rolls): the postStart hook now runs
`occ config:system:set allow_local_remote_servers --value=true --type=boolean`
right after the install/maintenance wait, on every start.
Task 2 — promote Keycloak `platform-admins` members to Nextcloud admins (#5).
user_oidc 8.11 has NO native group->admin mapping (occ user_oidc:provider offers
group PROVISIONING and a login WHITELIST only), so the hook mirrors the provisioned
`platform-admins` membership into the built-in `admin` group idempotently (occ
group:list --output=json parsed via php argv; occ group:adduser admin per member).
A user is promotable after their first OIDC login provisions them + the next pod
start (or one manual `occ group:adduser admin <uid>`). The LOCAL admin stays as
break-glass (docs/15).
apps/wave-1/nextcloud.yaml re-rendered from catalog (single-file render; byte-diff
confined to the embedded hook). Rendered hook passes `sh -n`; parse step validated
in-pod. `make -C ci gate` green (lint/l0/l1; conformance + spec-immutability skip:
no cluster / spec repo unmounted).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The Keycloak OIDC postStart registration for gitea and nextcloud ran but
never persisted, leaving `gitea admin auth list` empty and nextcloud's
user_oidc with "No providers configured" despite the apps installing/
enabling and the client secret being present.
Two distinct root causes, confirmed by running the registration manually
against the live warm pods:
- nextcloud: a QUOTING BUG, not a race. The OCC() wrapper runs
`su -c "php occ $*"`, which word-split `--scope="openid email profile"`
back into three tokens, so occ saw `email profile` as extra positional
args -> "Too many arguments to user_oidc:provider" -> exit 1, swallowed
by `|| echo failed`. Failed on EVERY boot. Fixed by running the provider
call as ONE `su -c` with the scope single-quoted so www-data's shell
keeps it a single argument; keep the OCC() wrapper only for single-token
subcommands.
- gitea: a first-boot RACE. The direct `admin auth add-oauth` persists once
the app/network is warm (verified live), but fired too early at postStart
and its failure was swallowed (`|| echo ... >&2`, exit 0).
Hardening applied to both catalog sources (rendered into apps/wave-*):
- wait (bounded ~90s) for the Keycloak discovery doc to return HTTP 200;
- nextcloud also waits for `installed: true` AND `maintenance: false`;
- register idempotently (create-or-update) then VERIFY the source/provider
actually persisted, retrying the whole cycle (guards exit-0-but-nothing-
stuck);
- on give-up, log LOUDLY to stderr AND a marker file on the persistent
volume (/data, /var/www/html) instead of swallowing — next failure is
visible;
- never crashloop: missing-secret guard retained, break-glass local admin
stays usable.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Extends the SSO work to the two remaining `auth: oidc` apps still on local-admin
logins, each as a native confidential client of the Keycloak `homelab` realm with
its local admin kept as break-glass (docs/15 availability caveat; R-IDP-02/06).
Realm (source of truth): add the `nextcloud` and `semaphore` confidential clients
to platform/keycloak/keycloak-realm-import.yaml (modeled on grafana/gitea) + their
NEXTCLOUD_CLIENT_SECRET / SEMAPHORE_CLIENT_SECRET placeholders. As before, a
KeycloakRealmImport no-ops on an already-existing realm, so both live clients were
ALSO created via kcadm (nextcloud e95049ed-ca10-4085-97d9-4bbef495fe61, semaphore
1835bfe9-b125-4026-8f69-33557519cf4a); the YAML keeps git authoritative.
Secrets (SOPS/age): add nextcloud-client-secret + semaphore-client-secret to
keycloak-realm-clients.sops.yaml (keycloak ns). Mirror the nextcloud secret as
oidc-client-secret into nextcloud-bootstrap (nextcloud ns). For semaphore, the
whole SEMAPHORE_OIDC_PROVIDERS JSON (which embeds the client secret) is held
encrypted as oidc-providers-json in semaphore-bootstrap (semaphore ns). All values
verified == the live clients; existing keys untouched; no plaintext in git.
App config (catalog source + re-render):
- nextcloud: add NEXTCLOUD_OIDC_CLIENT_SECRET env + an idempotent postStart hook
that installs/enables user_oidc and runs `occ user_oidc:provider keycloak`
(create-or-update; secret via --clientsecret-env so it never hits argv; occ as
www-data; group provisioning on the `groups` claim). Syntax verified against the
running user_oidc 8.11.0 on Nextcloud 31.0.14; su-to-www-data env passthrough
verified. Missing-secret/not-installed guards prevent crashloops.
- semaphore: add SEMAPHORE_OIDC_PROVIDERS env sourced from the encrypted
oidc-providers-json (provider_url=realm issuer/auto-discovery, redirect
/api/auth/oidc/keycloak/redirect). Mechanism verified against semaphore v2.19.10
source (util ConfigType oidc_providers env tag + OidcProvider struct + router
redirect path); no kubelet service-link env collision.
apps/wave-1/{nextcloud,semaphore}.yaml re-rendered via
`render.py --no-image-automation` (diffs are env/lifecycle-only).
Verified: realm import Done=True/HasErrors=False and platform-config
Ready=True/Healthy=True (unchanged); edited realm YAML passes server dry-run; live
homelab clients now include gitea/nextcloud/semaphore; static gate (lint/l0/l1)
green. Browser SSO login for each still needs a human check.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Replaces the single `apps` Flux Kustomization with dependency-ordered waves
(apps-wave-0..2) so a crashlooping workload isolates to its wave + downstream
instead of stalling the entire app tier. Parallel within a wave, ordered across.
- catalog/render.py: compute waves by longest dependency-path depth over the
ServiceDefinition `dependencies:` edges (a `<svc>-db` is a leaf, so its
consumer lands strictly higher — the "DB below consumer" rule falls out of
longest-path, no special case). `--emit-waves` generates the per-wave
kustomizations + Flux CRs; `--wave-layout` places renders into the right dir.
wave 0: pihole gitea jackett plex qbittorrent nextcloud-db semaphore-db
wave 1: sonarr radarr nextcloud semaphore gitea-actions
wave 2: seerr
- apps/*.yaml -> apps/wave-{0,1,2}/ (byte-identical moves; SOPS files intact).
- clusters/base: apps.yaml removed; apps-wave-{0,1,2}.yaml added (dependsOn
chain: platform-config -> wave-0 -> wave-1 -> wave-2, prune+wait each).
- clusters/dev: one SOPS decryption patch per wave (load-bearing).
- DESIGN-waves.md: wave table, topology, cutover runbook.
Cutover: preceded by disabling prune on `apps` (1bfb0db) so removing that
object here does not GC its resources; the wave Kustomizations SSA-adopt the
byte-identical manifests in place (no restart, no PVC churn).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
jakman85andClaude Opus 4.8