fix(oidc): make gitea + nextcloud OIDC login-source registration persist

The Keycloak OIDC postStart registration for gitea and nextcloud ran but
never persisted, leaving `gitea admin auth list` empty and nextcloud's
user_oidc with "No providers configured" despite the apps installing/
enabling and the client secret being present.

Two distinct root causes, confirmed by running the registration manually
against the live warm pods:

- nextcloud: a QUOTING BUG, not a race. The OCC() wrapper runs
  `su -c "php occ $*"`, which word-split `--scope="openid email profile"`
  back into three tokens, so occ saw `email profile` as extra positional
  args -> "Too many arguments to user_oidc:provider" -> exit 1, swallowed
  by `|| echo failed`. Failed on EVERY boot. Fixed by running the provider
  call as ONE `su -c` with the scope single-quoted so www-data's shell
  keeps it a single argument; keep the OCC() wrapper only for single-token
  subcommands.
- gitea: a first-boot RACE. The direct `admin auth add-oauth` persists once
  the app/network is warm (verified live), but fired too early at postStart
  and its failure was swallowed (`|| echo ... >&2`, exit 0).

Hardening applied to both catalog sources (rendered into apps/wave-*):
- wait (bounded ~90s) for the Keycloak discovery doc to return HTTP 200;
- nextcloud also waits for `installed: true` AND `maintenance: false`;
- register idempotently (create-or-update) then VERIFY the source/provider
  actually persisted, retrying the whole cycle (guards exit-0-but-nothing-
  stuck);
- on give-up, log LOUDLY to stderr AND a marker file on the persistent
  volume (/data, /var/www/html) instead of swallowing — next failure is
  visible;
- never crashloop: missing-secret guard retained, break-glass local admin
  stays usable.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

apps/wave-0/gitea.yaml

@@ -127,7 +127,8 @@ spec:
               command:
               - /bin/sh
               - -c
-              - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n  i=$((i + 1))\n  [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n  echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n  n=0\n  until su-exec git gitea admin user create \\\n    --username \"$GITEA_ADMIN_USERNAME\" \\\n    --password \"$GITEA_ADMIN_PASSWORD\" \\\n    --email \"[email protected]\" \\\n    --admin --must-change-password=false; do\n    n=$((n + 1))\n    [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n    sleep 3\n  done\n  echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps the platform-admins group to Gitea\n# admin (--admin-group). Auto-discovery pulls auth/token/userinfo/jwks endpoints. Runs on every\n# start: updates the existing source if present, else creates it. Missing secret => skip (the\n# local break-glass admin above still works), so an unfilled secret never crashes the pod.\nif [ -n \"${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n  oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n  if [ -n \"$oidc_id\" ]; then\n    su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n      --name keycloak --provider openidConnect --key gitea \\\n      --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n      --auto-discover-url https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration \\\n      --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins \\\n      && echo \"[gitea-oidc] updated login source 'keycloak' (id $oidc_id)\" \\\n      || echo \"[gitea-oidc] update-oauth failed\" >&2\n  else\n    su-exec git gitea admin auth add-oauth \\\n      --name keycloak --provider openidConnect --key gitea \\\n      --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n      --auto-discover-url https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration \\\n      --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins \\\n      && echo \"[gitea-oidc] added login source 'keycloak'\" \\\n      || echo \"[gitea-oidc] add-oauth failed\" >&2\n  fi\nelse\n  echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nfi\n"
+              - "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n  i=$((i + 1))\n  [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n  sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n  echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n  n=0\n  until su-exec git gitea admin user create \\\n    --username \"$GITEA_ADMIN_USERNAME\" \\\n    --password \"$GITEA_ADMIN_PASSWORD\" \\\n    --email \"[email protected]\" \\\n    --admin --must-change-password=false; do\n    n=$((n + 1))\n    [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n    sleep 3\n  done\n  echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n  echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n  { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\nif [ -z \"${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n  echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n  # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n  d=0; disc_ok=0\n  while [ \"$d\" -lt 45 ]; do\n    if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n      disc_ok=1; break\n    fi\n    d=$((d + 1)); sleep 2\n  done\n  [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n  # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n  r=0; reg_ok=0\n  while [ \"$r\" -lt 10 ]; do\n    oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n    if [ -n \"$oidc_id\" ]; then\n      su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n    else\n      su-exec git gitea admin auth add-oauth \\\n        --name keycloak --provider openidConnect --key gitea \\\n        --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n        --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n        --scopes \"openid email profile\" --group-claim-name\
+                \ groups --admin-group platform-admins >/dev/null 2>&1 || true\n    fi\n    # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n    if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n      reg_ok=1; break\n    fi\n    r=$((r + 1)); sleep 3\n  done\n  if [ \"$reg_ok\" = 1 ]; then\n    echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n  else\n    gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n  fi\nfi\n"
       volumes:
       - name: data
         persistentVolumeClaim:
@@ -254,3 +255,29 @@ spec:
       annotations:
         summary: gitea is down
         description: 'gitea has been unreachable for 5m (SLI: up).'
+---
+# Flux ImageRepository (R-UPD)
+apiVersion: image.toolkit.fluxcd.io/v1beta2
+kind: ImageRepository
+metadata:
+  name: gitea
+  namespace: gitea
+spec:
+  image: gitea/gitea
+  interval: 1h0m0s
+---
+# Flux ImagePolicy (R-UPD)
+apiVersion: image.toolkit.fluxcd.io/v1beta2
+kind: ImagePolicy
+metadata:
+  name: gitea
+  namespace: gitea
+  annotations:
+    homelab.keithjackson.dev/update-channel: stable
+    homelab.keithjackson.dev/auto-rollback: 'true'
+spec:
+  imageRepositoryRef:
+    name: gitea
+  policy:
+    semver:
+      range: '>=1.22.0 <2.0.0'