fix(oidc): make gitea + nextcloud OIDC login-source registration persist

The Keycloak OIDC postStart registration for gitea and nextcloud ran but
never persisted, leaving `gitea admin auth list` empty and nextcloud's
user_oidc with "No providers configured" despite the apps installing/
enabling and the client secret being present.

Two distinct root causes, confirmed by running the registration manually
against the live warm pods:

- nextcloud: a QUOTING BUG, not a race. The OCC() wrapper runs
  `su -c "php occ $*"`, which word-split `--scope="openid email profile"`
  back into three tokens, so occ saw `email profile` as extra positional
  args -> "Too many arguments to user_oidc:provider" -> exit 1, swallowed
  by `|| echo failed`. Failed on EVERY boot. Fixed by running the provider
  call as ONE `su -c` with the scope single-quoted so www-data's shell
  keeps it a single argument; keep the OCC() wrapper only for single-token
  subcommands.
- gitea: a first-boot RACE. The direct `admin auth add-oauth` persists once
  the app/network is warm (verified live), but fired too early at postStart
  and its failure was swallowed (`|| echo ... >&2`, exit 0).

Hardening applied to both catalog sources (rendered into apps/wave-*):
- wait (bounded ~90s) for the Keycloak discovery doc to return HTTP 200;
- nextcloud also waits for `installed: true` AND `maintenance: false`;
- register idempotently (create-or-update) then VERIFY the source/provider
  actually persisted, retrying the whole cycle (guards exit-0-but-nothing-
  stuck);
- on give-up, log LOUDLY to stderr AND a marker file on the persistent
  volume (/data, /var/www/html) instead of swallowing — next failure is
  visible;
- never crashloop: missing-secret guard retained, break-glass local admin
  stays usable.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

catalog/nextcloud.yaml

@@ -83,11 +83,23 @@ lifecycle:
           # secret or not-yet-installed instance is skipped so the pod never crashloops (the local
           # break-glass admin still works).
           set -u
+          # OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,
+          # so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The
+          # provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as
+          # ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.
           OCC() { su -s /bin/sh www-data -c "php /var/www/html/occ $*"; }
+          NEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
+          nextcloud_oidc_fail() {
+            echo "[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1" >&2
+            { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true
+          }
+          # (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds
+          #     maintenance mode during first-boot install/upgrade; occ writes made in that window do not
+          #     stick — a second race on top of the quoting bug fixed at (c).
           i=0
-          until OCC status 2>/dev/null | grep -q "installed: true"; do
+          until OCC status 2>/dev/null | grep -q "installed: true" && ! OCC status 2>/dev/null | grep -q "maintenance: true"; do
             i=$((i + 1))
-            [ "$i" -ge 300 ] && { echo "[nextcloud-oidc] occ not ready after wait; skipping" >&2; exit 0; }
+            [ "$i" -ge 300 ] && { echo "[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping" >&2; exit 0; }
             sleep 2
           done
           if [ -z "${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then
@@ -98,20 +110,43 @@ lifecycle:
             OCC app:install user_oidc || echo "[nextcloud-oidc] app:install user_oidc failed" >&2
           fi
           OCC app:enable user_oidc >/dev/null 2>&1 || true
-          if OCC user_oidc:provider keycloak \
-            --clientid=nextcloud \
-            --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \
-            --discoveryuri=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration \
-            --scope="openid email profile" \
-            --unique-uid=0 \
-            --mapping-uid=preferred_username \
-            --mapping-email=email \
-            --mapping-display-name=name \
-            --group-provisioning=1 \
-            --mapping-groups=groups; then
-            echo "[nextcloud-oidc] provider 'keycloak' configured"
+          # (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.
+          d=0; disc_ok=0
+          while [ "$d" -lt 45 ]; do
+            if [ "$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 "$NEXTCLOUD_DISCOVERY_URL" 2>/dev/null)" = "200" ]; then
+              disc_ok=1; break
+            fi
+            d=$((d + 1)); sleep 2
+          done
+          [ "$disc_ok" = 1 ] || nextcloud_oidc_fail "Keycloak discovery endpoint not reachable (HTTP 200) within timeout"
+          # (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.
+          #     HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a
+          #     single argument. The previous `OCC user_oidc:provider ... --scope="openid email profile"`
+          #     form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->
+          #     occ "Too many arguments" -> silent failure on EVERY boot. Secret via --clientsecret-env
+          #     (never on argv). The verify-then-retry loop guarantees the provider actually persisted.
+          r=0; reg_ok=0
+          while [ "$r" -lt 10 ]; do
+            su -s /bin/sh www-data -c "php /var/www/html/occ user_oidc:provider keycloak \
+              --clientid=nextcloud \
+              --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \
+              --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \
+              --scope='openid email profile' \
+              --unique-uid=0 \
+              --mapping-uid=preferred_username \
+              --mapping-email=email \
+              --mapping-display-name=name \
+              --group-provisioning=1 \
+              --mapping-groups=groups" >/dev/null 2>&1 || true
+            if OCC user_oidc:provider 2>/dev/null | grep -q "keycloak"; then
+              reg_ok=1; break
+            fi
+            r=$((r + 1)); sleep 3
+          done
+          if [ "$reg_ok" = 1 ]; then
+            echo "[nextcloud-oidc] provider 'keycloak' configured and verified"
           else
-            echo "[nextcloud-oidc] provider config failed" >&2
+            nextcloud_oidc_fail "user_oidc:provider ran but the provider did not persist after retries"
           fi
 exposure: public                # edge port-forward + public DNS (docs/09; R-DNS/R-NET)
 egress: internet                # normal WAN egress (docs/03; R-NET-05) — app store, integrity checks