fix(oidc): make gitea + nextcloud OIDC login-source registration persist

The Keycloak OIDC postStart registration for gitea and nextcloud ran but
never persisted, leaving `gitea admin auth list` empty and nextcloud's
user_oidc with "No providers configured" despite the apps installing/
enabling and the client secret being present.

Two distinct root causes, confirmed by running the registration manually
against the live warm pods:

- nextcloud: a QUOTING BUG, not a race. The OCC() wrapper runs
  `su -c "php occ $*"`, which word-split `--scope="openid email profile"`
  back into three tokens, so occ saw `email profile` as extra positional
  args -> "Too many arguments to user_oidc:provider" -> exit 1, swallowed
  by `|| echo failed`. Failed on EVERY boot. Fixed by running the provider
  call as ONE `su -c` with the scope single-quoted so www-data's shell
  keeps it a single argument; keep the OCC() wrapper only for single-token
  subcommands.
- gitea: a first-boot RACE. The direct `admin auth add-oauth` persists once
  the app/network is warm (verified live), but fired too early at postStart
  and its failure was swallowed (`|| echo ... >&2`, exit 0).

Hardening applied to both catalog sources (rendered into apps/wave-*):
- wait (bounded ~90s) for the Keycloak discovery doc to return HTTP 200;
- nextcloud also waits for `installed: true` AND `maintenance: false`;
- register idempotently (create-or-update) then VERIFY the source/provider
  actually persisted, retrying the whole cycle (guards exit-0-but-nothing-
  stuck);
- on give-up, log LOUDLY to stderr AND a marker file on the persistent
  volume (/data, /var/www/html) instead of swallowing — next failure is
  visible;
- never crashloop: missing-secret guard retained, break-glass local admin
  stays usable.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

apps/wave-0/gitea.yaml

@@ -127,7 +127,8 @@ spec:
               command:
               - /bin/sh
               - -c
-              - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n  i=$((i + 1))\n  [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n  echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n  n=0\n  until su-exec git gitea admin user create \\\n    --username \"$GITEA_ADMIN_USERNAME\" \\\n    --password \"$GITEA_ADMIN_PASSWORD\" \\\n    --email \"[email protected]\" \\\n    --admin --must-change-password=false; do\n    n=$((n + 1))\n    [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n    sleep 3\n  done\n  echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps the platform-admins group to Gitea\n# admin (--admin-group). Auto-discovery pulls auth/token/userinfo/jwks endpoints. Runs on every\n# start: updates the existing source if present, else creates it. Missing secret => skip (the\n# local break-glass admin above still works), so an unfilled secret never crashes the pod.\nif [ -n \"${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n  oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n  if [ -n \"$oidc_id\" ]; then\n    su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n      --name keycloak --provider openidConnect --key gitea \\\n      --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n      --auto-discover-url https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration \\\n      --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins \\\n      && echo \"[gitea-oidc] updated login source 'keycloak' (id $oidc_id)\" \\\n      || echo \"[gitea-oidc] update-oauth failed\" >&2\n  else\n    su-exec git gitea admin auth add-oauth \\\n      --name keycloak --provider openidConnect --key gitea \\\n      --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n      --auto-discover-url https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration \\\n      --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins \\\n      && echo \"[gitea-oidc] added login source 'keycloak'\" \\\n      || echo \"[gitea-oidc] add-oauth failed\" >&2\n  fi\nelse\n  echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nfi\n"
+              - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n  i=$((i + 1))\n  [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n  echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n  n=0\n  until su-exec git gitea admin user create \\\n    --username \"$GITEA_ADMIN_USERNAME\" \\\n    --password \"$GITEA_ADMIN_PASSWORD\" \\\n    --email \"[email protected]\" \\\n    --admin --must-change-password=false; do\n    n=$((n + 1))\n    [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n    sleep 3\n  done\n  echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n  echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\nif [ -z \"${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n  # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n  d=0; disc_ok=0\n  while [ \"$d\" -lt 45 ]; do\n    if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n      disc_ok=1; break\n    fi\n    d=$((d + 1)); sleep 2\n  done\n  [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n  # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n  r=0; reg_ok=0\n  while [ \"$r\" -lt 10 ]; do\n    oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n    if [ -n \"$oidc_id\" ]; then\n      su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    else\n      su-exec git gitea admin auth add-oauth \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name\
+                \ groups --admin-group platform-admins >/dev/null 2>&1 || true\n    fi\n    # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n    if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n      reg_ok=1; break\n    fi\n    r=$((r + 1)); sleep 3\n  done\n  if [ \"$reg_ok\" = 1 ]; then\n    echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n  else\n    gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n  fi\nfi\n"
       volumes:
       - name: data
         persistentVolumeClaim:
@@ -254,3 +255,29 @@ spec:
       annotations:
         summary: gitea is down
         description: 'gitea has been unreachable for 5m (SLI: up).'
+---
+# Flux ImageRepository (R-UPD)
+apiVersion: image.toolkit.fluxcd.io/v1beta2
+kind: ImageRepository
+metadata:
+  name: gitea
+  namespace: gitea
+spec:
+  image: gitea/gitea
+  interval: 1h0m0s
+---
+# Flux ImagePolicy (R-UPD)
+apiVersion: image.toolkit.fluxcd.io/v1beta2
+kind: ImagePolicy
+metadata:
+  name: gitea
+  namespace: gitea
+  annotations:
+    homelab.keithjackson.dev/update-channel: stable
+    homelab.keithjackson.dev/auto-rollback: 'true'
+spec:
+  imageRepositoryRef:
+    name: gitea
+  policy:
+    semver:
+      range: '>=1.22.0 <2.0.0'

apps/wave-1/nextcloud.yaml

@@ -125,7 +125,8 @@ spec:
               command:
               - /bin/sh
               - -c
-              - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\"; do\n  i=$((i + 1))\n  [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif [ -z \"${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n  exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n  OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\nif OCC user_oidc:provider keycloak \\\n  --clientid=nextcloud \\\n  --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n  --discoveryuri=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration \\\n  --scope=\"openid email profile\" \\\n  --unique-uid=0 \\\n  --mapping-uid=preferred_username \\\n  --mapping-email=email \\\n  --mapping-display-name=name \\\n  --group-provisioning=1 \\\n  --mapping-groups=groups; then\n  echo \"[nextcloud-oidc] provider 'keycloak' configured\"\nelse\n  echo \"[nextcloud-oidc] provider config failed\" >&2\nfi\n"
+              - "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n  echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n#     maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n#     stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n  i=$((i + 1))\n  [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif [ -z \"${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n  exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n  OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n  if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n    disc_ok=1; break\n  fi\n  d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n#     HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n#     single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n#     form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n#     occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n#     (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n  su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n    --clientid=nextcloud \\\n    --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n    --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n    --scope='openid email profile' \\\n    --unique-uid=0 \\\n    --mapping-uid=preferred_username \\\n    --mapping-email=email \\\n    --mapping-display-name=name \\\n    --group-provisioning=1 \\\n    --mapping-groups=groups\" >/dev/null 2>&1 || true\n  if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n    reg_ok=1; break\n  fi\n  r=$((r + 1)); sleep\
+                \ 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n  echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n  nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n"
       volumes:
       - name: data
         persistentVolumeClaim:
@@ -274,3 +275,29 @@ spec:
       annotations:
         summary: nextcloud is down
         description: 'nextcloud has been unreachable for 5m (SLI: up).'
+---
+# Flux ImageRepository (R-UPD)
+apiVersion: image.toolkit.fluxcd.io/v1beta2
+kind: ImageRepository
+metadata:
+  name: nextcloud
+  namespace: nextcloud
+spec:
+  image: nextcloud
+  interval: 1h0m0s
+---
+# Flux ImagePolicy (R-UPD)
+apiVersion: image.toolkit.fluxcd.io/v1beta2
+kind: ImagePolicy
+metadata:
+  name: nextcloud
+  namespace: nextcloud
+  annotations:
+    homelab.keithjackson.dev/update-channel: stable
+    homelab.keithjackson.dev/auto-rollback: 'true'
+spec:
+  imageRepositoryRef:
+    name: nextcloud
+  policy:
+    semver:
+      range: '>=31.0.0 <32.0.0'

catalog/gitea.yaml

@@ -93,31 +93,62 @@ lifecycle:
           fi
           # --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---
           # Registers the confidential `gitea` client as the OAuth2 login source "keycloak" against the
-          # homelab realm's discovery doc. The `groups` claim maps the platform-admins group to Gitea
+          # homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin
-          # admin (--admin-group). Auto-discovery pulls auth/token/userinfo/jwks endpoints. Runs on every
+          # (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled
-          # start: updates the existing source if present, else creates it. Missing secret => skip (the
+          # secret never crashes the pod.
-          # local break-glass admin above still works), so an unfilled secret never crashes the pod.
+          #
-          if [ -n "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then
+          # HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was
-            oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2=="keycloak"{print $1; exit}')
+          # ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.
-            if [ -n "$oidc_id" ]; then
+          # Now it (1) waits — bounded, ~90s — for the Keycloak discovery doc to serve HTTP 200,
-              su-exec git gitea admin auth update-oauth --id "$oidc_id" \
+          # (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually
-                --name keycloak --provider openidConnect --key gitea \
+          # persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker
-                --secret "$GITEA_OIDC_CLIENT_SECRET" \
+          # file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays
-                --auto-discover-url https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration \
+          # usable). The verify-then-retry loop is the real guarantee against "exit 0 but nothing stuck".
-                --scopes "openid email profile" --group-claim-name groups --admin-group platform-admins \
+          GITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
-                && echo "[gitea-oidc] updated login source 'keycloak' (id $oidc_id)" \
+          gitea_oidc_fail() {
-                || echo "[gitea-oidc] update-oauth failed" >&2
+            echo "[gitea-oidc] FAILED to register 'keycloak' login source: $1" >&2
-            else
+            { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true
-              su-exec git gitea admin auth add-oauth \
+          }
-                --name keycloak --provider openidConnect --key gitea \
+          if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then
-                --secret "$GITEA_OIDC_CLIENT_SECRET" \
-                --auto-discover-url https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration \
-                --scopes "openid email profile" --group-claim-name groups --admin-group platform-admins \
-                && echo "[gitea-oidc] added login source 'keycloak'" \
-                || echo "[gitea-oidc] add-oauth failed" >&2
-            fi
-          else
             echo "[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)"
+          else
+            # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).
+            d=0; disc_ok=0
+            while [ "$d" -lt 45 ]; do
+              if [ "$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 "$GITEA_DISCOVERY_URL" 2>/dev/null)" = "200" ]; then
+                disc_ok=1; break
+              fi
+              d=$((d + 1)); sleep 2
+            done
+            [ "$disc_ok" = 1 ] || gitea_oidc_fail "Keycloak discovery endpoint not reachable (HTTP 200) within timeout"
+            # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.
+            r=0; reg_ok=0
+            while [ "$r" -lt 10 ]; do
+              oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2=="keycloak"{print $1; exit}')
+              if [ -n "$oidc_id" ]; then
+                su-exec git gitea admin auth update-oauth --id "$oidc_id" \
+                  --name keycloak --provider openidConnect --key gitea \
+                  --secret "$GITEA_OIDC_CLIENT_SECRET" \
+                  --auto-discover-url "$GITEA_DISCOVERY_URL" \
+                  --scopes "openid email profile" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true
+              else
+                su-exec git gitea admin auth add-oauth \
+                  --name keycloak --provider openidConnect --key gitea \
+                  --secret "$GITEA_OIDC_CLIENT_SECRET" \
+                  --auto-discover-url "$GITEA_DISCOVERY_URL" \
+                  --scopes "openid email profile" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true
+              fi
+              # VERIFY: the "keycloak" source must actually be listed now (guards the exit-0-but-nothing-persisted race).
+              if su-exec git gitea admin auth list 2>/dev/null | awk '$2=="keycloak"{f=1} END{exit !f}'; then
+                reg_ok=1; break
+              fi
+              r=$((r + 1)); sleep 3
+            done
+            if [ "$reg_ok" = 1 ]; then
+              echo "[gitea-oidc] login source 'keycloak' registered and verified"
+            else
+              gitea_oidc_fail "add/update-oauth ran but the 'keycloak' source did not persist after retries"
+            fi
           fi
 exposure: public                # edge-forwarded git host at gitea.home.keithjackson.dev (docs/09; R-DNS/R-NET)
 egress: internet                # normal WAN egress (docs/03; R-NET-05)

catalog/nextcloud.yaml

@@ -83,11 +83,23 @@ lifecycle:
           # secret or not-yet-installed instance is skipped so the pod never crashloops (the local
           # break-glass admin still works).
           set -u
+          # OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,
+          # so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The
+          # provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as
+          # ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.
           OCC() { su -s /bin/sh www-data -c "php /var/www/html/occ $*"; }
+          NEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
+          nextcloud_oidc_fail() {
+            echo "[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1" >&2
+            { echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true
+          }
+          # (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds
+          #     maintenance mode during first-boot install/upgrade; occ writes made in that window do not
+          #     stick — a second race on top of the quoting bug fixed at (c).
           i=0
-          until OCC status 2>/dev/null | grep -q "installed: true"; do
+          until OCC status 2>/dev/null | grep -q "installed: true" && ! OCC status 2>/dev/null | grep -q "maintenance: true"; do
             i=$((i + 1))
-            [ "$i" -ge 300 ] && { echo "[nextcloud-oidc] occ not ready after wait; skipping" >&2; exit 0; }
+            [ "$i" -ge 300 ] && { echo "[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping" >&2; exit 0; }
             sleep 2
           done
           if [ -z "${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then
@@ -98,20 +110,43 @@ lifecycle:
             OCC app:install user_oidc || echo "[nextcloud-oidc] app:install user_oidc failed" >&2
           fi
           OCC app:enable user_oidc >/dev/null 2>&1 || true
-          if OCC user_oidc:provider keycloak \
+          # (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.
-            --clientid=nextcloud \
+          d=0; disc_ok=0
-            --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \
+          while [ "$d" -lt 45 ]; do
-            --discoveryuri=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration \
+            if [ "$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 "$NEXTCLOUD_DISCOVERY_URL" 2>/dev/null)" = "200" ]; then
-            --scope="openid email profile" \
+              disc_ok=1; break
-            --unique-uid=0 \
+            fi
-            --mapping-uid=preferred_username \
+            d=$((d + 1)); sleep 2
-            --mapping-email=email \
+          done
-            --mapping-display-name=name \
+          [ "$disc_ok" = 1 ] || nextcloud_oidc_fail "Keycloak discovery endpoint not reachable (HTTP 200) within timeout"
-            --group-provisioning=1 \
+          # (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.
-            --mapping-groups=groups; then
+          #     HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a
-            echo "[nextcloud-oidc] provider 'keycloak' configured"
+          #     single argument. The previous `OCC user_oidc:provider ... --scope="openid email profile"`
+          #     form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->
+          #     occ "Too many arguments" -> silent failure on EVERY boot. Secret via --clientsecret-env
+          #     (never on argv). The verify-then-retry loop guarantees the provider actually persisted.
+          r=0; reg_ok=0
+          while [ "$r" -lt 10 ]; do
+            su -s /bin/sh www-data -c "php /var/www/html/occ user_oidc:provider keycloak \
+              --clientid=nextcloud \
+              --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \
+              --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \
+              --scope='openid email profile' \
+              --unique-uid=0 \
+              --mapping-uid=preferred_username \
+              --mapping-email=email \
+              --mapping-display-name=name \
+              --group-provisioning=1 \
+              --mapping-groups=groups" >/dev/null 2>&1 || true
+            if OCC user_oidc:provider 2>/dev/null | grep -q "keycloak"; then
+              reg_ok=1; break
+            fi
+            r=$((r + 1)); sleep 3
+          done
+          if [ "$reg_ok" = 1 ]; then
+            echo "[nextcloud-oidc] provider 'keycloak' configured and verified"
           else
-            echo "[nextcloud-oidc] provider config failed" >&2
+            nextcloud_oidc_fail "user_oidc:provider ran but the provider did not persist after retries"
           fi
 exposure: public                # edge port-forward + public DNS (docs/09; R-DNS/R-NET)
 egress: internet                # normal WAN egress (docs/03; R-NET-05) — app store, integrity checks