Commit Graph
6 Commits
Author SHA1 Message Date
jakman85andClaude Opus 5 3484955903 fix(sso): gitea + nextcloud OIDC bootstrap was DEAD CODE — envsubst ate the guard
Both ServiceDefinitions guarded their Keycloak OIDC registration with an UNESCAPED
braced expansion:

    if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then      # gitea
    if [ -z "${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then  # nextcloud, then exit 0

apps-wave-0/1 both declare postBuild.substituteFrom: cluster-vars, and neither name is a
cluster-vars key — so Flux's envsubst substituted them away AT RECONCILE TIME, long
before the container ever ran. Verified in the LIVE cluster: the deployed Deployments
literally contain

    if [ -z "" ]; then

which is always true. The entire OIDC registration block has been skipped on EVERY boot
of both apps.

WHY NOBODY NOTICED: both apps DO have the provider registered — `gitea admin auth list`
shows keycloak/OAuth2/true and `occ user_oidc:provider` shows the keycloak provider.
Those live in each app's own database, persisted from an earlier successful registration.
SSO works today and DECISIONS-LOG is not wrong. But the idempotent re-registration is
dead, so any PVC rebuild, DB restore, or fresh cluster build comes up with NO OIDC and
only the break-glass local admin. A PROD BUILD FROM THIS REPO WOULD SILENTLY HAVE NO SSO
on gitea or nextcloud.

The irony: the surrounding code was hardened twice — see its own comments about a "silent
first-boot RACE", the verify-then-retry loops, the marker files. All of that hardening was
unreachable, because the guard above it always evaluated true.

Not a regression from the recent R-ENV-02 tokenization — introduced 2026-08-27 (8738871).

FIXED AT THE RIGHT SEAM: apps/wave-0/gitea.yaml and apps/wave-1/nextcloud.yaml are
RENDER-GENERATED, so the fix is in catalog/*.yaml and the goldens are regenerated.
Escaped to $${...}, the convention documented in clusters/base/apps-vms.yaml. Post-envsubst
the container now receives the intended `${VAR:-}` instead of `""`.

STRICT MODE WOULD NEVER HAVE CAUGHT THIS. Flux deliberately exempts default-providing
operators (-, :-, +, :+, =, :=, ?, :?) from StrictPostBuildSubstitutions, because they
"intentionally handle the unset case". So a targeted check was required.

MADE UN-REPEATABLE IN THE RENDERER: catalog/render.py gains a substitution-safety section
wired into emit(), so every render path refuses to emit a manifest containing a braced
expansion that is not exactly ${KNOWN_KEY} for a key read LIVE from cluster-vars (no
hardcoded copy to drift; unreadable => empty set => fail-closed). Dollar-runs are counted
properly ($$ is a literal $ to envsubst, so odd runs are flagged and even runs are not),
and indexed forms get their own hard-error message. Nine new --selftest assertions,
including the exact regression. Proven to bite: rendering the pre-fix catalog files from
HEAD now exits 1 with the precise diagnosis.

Sweep: the complete substituted surface was enumerated from every Kustomization's path.
catalog/ contains exactly TWO braced expansions and both were these bugs; every other
braced expansion on a substituted path is a declared cluster-vars key (zero typos, zero
undeclared names); existing $${UOS_URL} escapes verified intact.

Verified: --selftest PASS; canonical render is byte-identical to the committed apps/ for
all 10 onboarded services; the three L1 render checks pass.

NEXT RECONCILE: both pod templates change, so gitea (wave 0) and nextcloud (wave 1) roll —
strategy: Recreate on RWO Longhorn PVCs, so expect a brief single-pod outage each. The
OIDC block then executes for the first time in these apps' deployed lifetime. Both paths
are idempotent (gitea takes auth update-oauth on the existing id; nextcloud's
user_oidc:provider is create-or-update), so the expected result is a no-op re-write plus
the verify loop logging success. If the SOPS client secret ever diverged from Keycloak's,
this is when it gets corrected instead of silently left stale. Tail both pods after merge.

Serves: R-IDP-02, R-IDP-06, R-ENV-02, R-GITOPS-02.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-01 01:29:40 -05:00
jakman85andClaude Opus 4.8 14521d37d3 fix(nextcloud-sso): apply step-(d) retry hardening at SOURCE (catalog)
apps/wave-1/nextcloud.yaml is GENERATED from catalog/nextcloud.yaml by
catalog/render.py, so the previous commit hardened only the rendered golden —
the next render would have reverted step (d) to the old single-pass version.

Apply the same verify-and-retry hardening to the postStart hook in
catalog/nextcloud.yaml (the source of truth). Re-rendered with
`render.py --all catalog -o apps --wave-layout`; apps/wave-1/nextcloud.yaml is
byte-identical to the golden committed in the previous commit (render round-trip
clean), so no apps/ change is needed here.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-27 20:18:51 -05:00
jakman85andClaude Opus 4.8 7f92adfac6 fix(nextcloud): allow SSRF to private IdP + promote platform-admins to admin
Task 1 — browser SSO "Could not reach OpenID Connect provider" (confirmed root
cause, not guessed). keycloak.home.keithjackson.dev resolves split-horizon to the
PRIVATE Cilium gateway VIP (192.168.1.53). Nextcloud's SSRF guard refuses its own
server-side HTTP requests to RFC1918 addresses, so user_oidc's discovery/JWKS/token
fetch AT LOGIN throws OCP\Http\Client\LocalServerException "violates local access
rules" -> surfaced to the user as "Could not reach the provider". Proven from the
live pod: nextcloud.log carries the exact LocalServerException from
DiscoveryService::obtainDiscovery with Guzzle option allow_local_address:false; a
Nextcloud-context IClientService->get() of the discovery URL returns
LocalServerException with the flag false and HTTP 200 with it true (A/B toggled).
The postStart (b) curl check passed regardless because it uses system libcurl, not
Nextcloud's guard — which is why the provider registered yet login still failed.

Fix (reproducible, survives pod rolls): the postStart hook now runs
`occ config:system:set allow_local_remote_servers --value=true --type=boolean`
right after the install/maintenance wait, on every start.

Task 2 — promote Keycloak `platform-admins` members to Nextcloud admins (#5).
user_oidc 8.11 has NO native group->admin mapping (occ user_oidc:provider offers
group PROVISIONING and a login WHITELIST only), so the hook mirrors the provisioned
`platform-admins` membership into the built-in `admin` group idempotently (occ
group:list --output=json parsed via php argv; occ group:adduser admin per member).
A user is promotable after their first OIDC login provisions them + the next pod
start (or one manual `occ group:adduser admin <uid>`). The LOCAL admin stays as
break-glass (docs/15).

apps/wave-1/nextcloud.yaml re-rendered from catalog (single-file render; byte-diff
confined to the embedded hook). Rendered hook passes `sh -n`; parse step validated
in-pod. `make -C ci gate` green (lint/l0/l1; conformance + spec-immutability skip:
no cluster / spec repo unmounted).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-27 19:00:09 -05:00
jakman85andClaude Opus 4.8 d34c639269 fix(oidc): make gitea + nextcloud OIDC login-source registration persist
The Keycloak OIDC postStart registration for gitea and nextcloud ran but
never persisted, leaving `gitea admin auth list` empty and nextcloud's
user_oidc with "No providers configured" despite the apps installing/
enabling and the client secret being present.

Two distinct root causes, confirmed by running the registration manually
against the live warm pods:

- nextcloud: a QUOTING BUG, not a race. The OCC() wrapper runs
  `su -c "php occ $*"`, which word-split `--scope="openid email profile"`
  back into three tokens, so occ saw `email profile` as extra positional
  args -> "Too many arguments to user_oidc:provider" -> exit 1, swallowed
  by `|| echo failed`. Failed on EVERY boot. Fixed by running the provider
  call as ONE `su -c` with the scope single-quoted so www-data's shell
  keeps it a single argument; keep the OCC() wrapper only for single-token
  subcommands.
- gitea: a first-boot RACE. The direct `admin auth add-oauth` persists once
  the app/network is warm (verified live), but fired too early at postStart
  and its failure was swallowed (`|| echo ... >&2`, exit 0).

Hardening applied to both catalog sources (rendered into apps/wave-*):
- wait (bounded ~90s) for the Keycloak discovery doc to return HTTP 200;
- nextcloud also waits for `installed: true` AND `maintenance: false`;
- register idempotently (create-or-update) then VERIFY the source/provider
  actually persisted, retrying the whole cycle (guards exit-0-but-nothing-
  stuck);
- on give-up, log LOUDLY to stderr AND a marker file on the persistent
  volume (/data, /var/www/html) instead of swallowing — next failure is
  visible;
- never crashloop: missing-secret guard retained, break-glass local admin
  stays usable.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-27 17:41:54 -05:00
jakman85andClaude Opus 4.8 f163026954 feat(nextcloud,semaphore): wire native Keycloak OIDC SSO (break-glass admins retained)
Extends the SSO work to the two remaining `auth: oidc` apps still on local-admin
logins, each as a native confidential client of the Keycloak `homelab` realm with
its local admin kept as break-glass (docs/15 availability caveat; R-IDP-02/06).

Realm (source of truth): add the `nextcloud` and `semaphore` confidential clients
to platform/keycloak/keycloak-realm-import.yaml (modeled on grafana/gitea) + their
NEXTCLOUD_CLIENT_SECRET / SEMAPHORE_CLIENT_SECRET placeholders. As before, a
KeycloakRealmImport no-ops on an already-existing realm, so both live clients were
ALSO created via kcadm (nextcloud e95049ed-ca10-4085-97d9-4bbef495fe61, semaphore
1835bfe9-b125-4026-8f69-33557519cf4a); the YAML keeps git authoritative.

Secrets (SOPS/age): add nextcloud-client-secret + semaphore-client-secret to
keycloak-realm-clients.sops.yaml (keycloak ns). Mirror the nextcloud secret as
oidc-client-secret into nextcloud-bootstrap (nextcloud ns). For semaphore, the
whole SEMAPHORE_OIDC_PROVIDERS JSON (which embeds the client secret) is held
encrypted as oidc-providers-json in semaphore-bootstrap (semaphore ns). All values
verified == the live clients; existing keys untouched; no plaintext in git.

App config (catalog source + re-render):
- nextcloud: add NEXTCLOUD_OIDC_CLIENT_SECRET env + an idempotent postStart hook
  that installs/enables user_oidc and runs `occ user_oidc:provider keycloak`
  (create-or-update; secret via --clientsecret-env so it never hits argv; occ as
  www-data; group provisioning on the `groups` claim). Syntax verified against the
  running user_oidc 8.11.0 on Nextcloud 31.0.14; su-to-www-data env passthrough
  verified. Missing-secret/not-installed guards prevent crashloops.
- semaphore: add SEMAPHORE_OIDC_PROVIDERS env sourced from the encrypted
  oidc-providers-json (provider_url=realm issuer/auto-discovery, redirect
  /api/auth/oidc/keycloak/redirect). Mechanism verified against semaphore v2.19.10
  source (util ConfigType oidc_providers env tag + OidcProvider struct + router
  redirect path); no kubelet service-link env collision.
apps/wave-1/{nextcloud,semaphore}.yaml re-rendered via
`render.py --no-image-automation` (diffs are env/lifecycle-only).

Verified: realm import Done=True/HasErrors=False and platform-config
Ready=True/Healthy=True (unchanged); edited realm YAML passes server dry-run; live
homelab clients now include gitea/nextcloud/semaphore; static gate (lint/l0/l1)
green. Browser SSO login for each still needs a human check.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-27 17:17:13 -05:00
jakman85andClaude Opus 4.8 4be6752c71 feat(apps): onboard nextcloud (Phase 6) — first CNPG-DB + OIDC-target patterns
Onboard nextcloud (docs/13 row 7): public, DR-critical file sync backed by an
external PostgreSQL. Establishes the reusable CNPG-DB pattern for later DB-backed
services.

- catalog/nextcloud.yaml: ServiceDefinition (image nextcloud:31-apache :80, public,
  internet, data 20Gi longhorn @/var/www/html, backup daily-offcluster, auth oidc
  +platform-admins, ingress nextcloud.home.keithjackson.dev, dependency
  nextcloud-db:5432). Uses schema 1.2.0 env for the Postgres connection, break-glass
  admin, trusted-domain + OVERWRITE* proxy settings.
- apps/nextcloud-db.yaml: CloudNativePG Cluster in a dedicated nextcloud-db namespace
  (modeled on keycloak-db; no default-deny, mirroring the verified keycloak posture),
  inheritedMetadata stamps app.kubernetes.io/name=nextcloud-db so the rendered egress
  CNP matches. initdb consumes a pre-provisioned owner secret.
- apps/nextcloud-db-app.sops.yaml + apps/nextcloud-bootstrap-secret.sops.yaml: SOPS
  (age) — shared DB password mirrored across the two namespaces (Pods cannot read a
  Secret cross-namespace), plus the break-glass local admin.
- apps/nextcloud.yaml: rendered manifests (catalog/render.py --no-image-automation).
- apps/kustomization.yaml: wire in the four new resources.

OIDC realm-client wiring DEFERRED (WS-APPS step-3 escape hatch): editing the reconciled
realm import in platform-config risks the running Keycloak realm / whole platform-config
apply, and nextcloud OIDC needs post-install occ steps with no first-class env — nextcloud
comes up INSTALLED on the break-glass local admin (as Gitea did); the Keycloak client is a
follow-up. Off-cluster backup awaits Velero/OQ-BKP (intent declared).

Gates: render --selftest PASS; other apps byte-identical; ci l0 17/17; l1 3/3; lint clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 01:32:00 -05:00