jakman85andClaude Opus 4.8
d34c639269
The Keycloak OIDC postStart registration for gitea and nextcloud ran but never persisted, leaving `gitea admin auth list` empty and nextcloud's user_oidc with "No providers configured" despite the apps installing/ enabling and the client secret being present. Two distinct root causes, confirmed by running the registration manually against the live warm pods: - nextcloud: a QUOTING BUG, not a race. The OCC() wrapper runs `su -c "php occ $*"`, which word-split `--scope="openid email profile"` back into three tokens, so occ saw `email profile` as extra positional args -> "Too many arguments to user_oidc:provider" -> exit 1, swallowed by `|| echo failed`. Failed on EVERY boot. Fixed by running the provider call as ONE `su -c` with the scope single-quoted so www-data's shell keeps it a single argument; keep the OCC() wrapper only for single-token subcommands. - gitea: a first-boot RACE. The direct `admin auth add-oauth` persists once the app/network is warm (verified live), but fired too early at postStart and its failure was swallowed (`|| echo ... >&2`, exit 0). Hardening applied to both catalog sources (rendered into apps/wave-*): - wait (bounded ~90s) for the Keycloak discovery doc to return HTTP 200; - nextcloud also waits for `installed: true` AND `maintenance: false`; - register idempotently (create-or-update) then VERIFY the source/provider actually persisted, retrying the whole cycle (guards exit-0-but-nothing- stuck); - on give-up, log LOUDLY to stderr AND a marker file on the persistent volume (/data, /var/www/html) instead of swallowing — next failure is visible; - never crashloop: missing-secret guard retained, break-glass local admin stays usable. Co-Authored-By: Claude Opus 4.8 <[email protected]>
188 lines
11 KiB
YAML
188 lines
11 KiB
YAML
# ServiceDefinition: nextcloud
|
|
# Source of truth: spec docs/13-service-catalog.md row 7; docs/15-identity-and-access.md; docs/04.
|
|
# public file-sync | egress internet | data on Longhorn (irreplaceable, DR-critical) | auth oidc.
|
|
#
|
|
# DR-critical (docs/13 "DR-critical data"): the user data lives on the 3-replica `longhorn` tier and
|
|
# is backed up off-cluster (backup.policy daily-offcluster; R-STATE-03). Nextcloud is stateless code
|
|
# over (a) its /var/www/html Longhorn PVC and (b) an external PostgreSQL — the first CNPG-DB app,
|
|
# establishing the reusable pattern (its DB is a CloudNativePG Cluster, apps/nextcloud-db.yaml).
|
|
#
|
|
# DB WIRING (schema >=1.2.0 env/envFrom): the app reaches its PostgreSQL cross-namespace at
|
|
# nextcloud-db-rw.nextcloud-db:5432 (the CNPG read-write Service). Its owner credentials are the
|
|
# SAME password held in two SOPS secrets — nextcloud-db-app (basic-auth, ns nextcloud-db; consumed by
|
|
# the CNPG Cluster's initdb) and nextcloud-bootstrap (ns nextcloud; consumed here as db-password).
|
|
# A Pod can only read Secrets in its OWN namespace, so the shared password is duplicated across the
|
|
# two encrypted files rather than replicated cross-namespace (documented in apps/nextcloud-db.yaml).
|
|
# Validates against schemas/servicedefinition.schema.json.
|
|
name: nextcloud
|
|
kind: container
|
|
source:
|
|
image: nextcloud:31-apache # apache variant serves on :80 (docs/13 row 7); pinned recent stable
|
|
# major (R-UPD-01); update.range governs advancement within the line
|
|
env:
|
|
# --- PostgreSQL connection (the CNPG-DB pattern). HOST/DB/USER are literals; PASSWORD is sourced
|
|
# from the in-namespace SOPS secret (the CNPG -app secret lives in ns nextcloud-db, unreachable
|
|
# by a Pod secretKeyRef, so the same value is mirrored into nextcloud-bootstrap here). ---
|
|
- name: POSTGRES_HOST
|
|
value: nextcloud-db-rw.nextcloud-db.svc.cluster.local
|
|
- name: POSTGRES_DB
|
|
value: nextcloud
|
|
- name: POSTGRES_USER
|
|
value: nextcloud
|
|
- name: POSTGRES_PASSWORD
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: nextcloud-bootstrap
|
|
key: db-password
|
|
# --- Break-glass local admin (docs/15 availability caveat): created on first-boot install so the
|
|
# operator is never locked out by an IdP outage. Sourced from the SOPS nextcloud-bootstrap secret. ---
|
|
- name: NEXTCLOUD_ADMIN_USER
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: nextcloud-bootstrap
|
|
key: admin-username
|
|
- name: NEXTCLOUD_ADMIN_PASSWORD
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: nextcloud-bootstrap
|
|
key: admin-password
|
|
# --- Public exposure behind the TLS-terminating shared Gateway (docs/08/09). TRUSTED_DOMAINS lets
|
|
# the vhost answer for the public FQDN; OVERWRITE* make Nextcloud emit https:// self-URLs even
|
|
# though it speaks plain HTTP behind the proxy. ---
|
|
- name: NEXTCLOUD_TRUSTED_DOMAINS
|
|
value: nextcloud.home.keithjackson.dev
|
|
- name: OVERWRITEHOST
|
|
value: nextcloud.home.keithjackson.dev
|
|
- name: OVERWRITEPROTOCOL
|
|
value: https
|
|
- name: OVERWRITECLIURL
|
|
value: https://nextcloud.home.keithjackson.dev
|
|
# --- Keycloak native OIDC SSO (auth.mode oidc; docs/15 "Native OIDC"; R-IDP-02/06). The confidential
|
|
# `nextcloud` realm client's secret is mirrored into this namespace's nextcloud-bootstrap Secret
|
|
# (oidc-client-secret; same value SOPS-held in keycloak-realm-clients). The postStart hook reads it
|
|
# via user_oidc's --clientsecret-env (never on argv) to register the "keycloak" provider. The LOCAL
|
|
# admin above stays as the break-glass path (docs/15). ---
|
|
- name: NEXTCLOUD_OIDC_CLIENT_SECRET
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: nextcloud-bootstrap
|
|
key: oidc-client-secret
|
|
lifecycle:
|
|
postStart:
|
|
exec:
|
|
command:
|
|
- /bin/sh
|
|
- -c
|
|
- |
|
|
# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint
|
|
# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR
|
|
# updates the "keycloak" login provider (occ user_oidc:provider is create-or-update by
|
|
# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud
|
|
# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via
|
|
# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing
|
|
# secret or not-yet-installed instance is skipped so the pod never crashloops (the local
|
|
# break-glass admin still works).
|
|
set -u
|
|
# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,
|
|
# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The
|
|
# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as
|
|
# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.
|
|
OCC() { su -s /bin/sh www-data -c "php /var/www/html/occ $*"; }
|
|
NEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
|
|
nextcloud_oidc_fail() {
|
|
echo "[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1" >&2
|
|
{ echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true
|
|
}
|
|
# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds
|
|
# maintenance mode during first-boot install/upgrade; occ writes made in that window do not
|
|
# stick — a second race on top of the quoting bug fixed at (c).
|
|
i=0
|
|
until OCC status 2>/dev/null | grep -q "installed: true" && ! OCC status 2>/dev/null | grep -q "maintenance: true"; do
|
|
i=$((i + 1))
|
|
[ "$i" -ge 300 ] && { echo "[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping" >&2; exit 0; }
|
|
sleep 2
|
|
done
|
|
if [ -z "${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then
|
|
echo "[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)"
|
|
exit 0
|
|
fi
|
|
if ! OCC app:list 2>/dev/null | grep -q "user_oidc"; then
|
|
OCC app:install user_oidc || echo "[nextcloud-oidc] app:install user_oidc failed" >&2
|
|
fi
|
|
OCC app:enable user_oidc >/dev/null 2>&1 || true
|
|
# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.
|
|
d=0; disc_ok=0
|
|
while [ "$d" -lt 45 ]; do
|
|
if [ "$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 "$NEXTCLOUD_DISCOVERY_URL" 2>/dev/null)" = "200" ]; then
|
|
disc_ok=1; break
|
|
fi
|
|
d=$((d + 1)); sleep 2
|
|
done
|
|
[ "$disc_ok" = 1 ] || nextcloud_oidc_fail "Keycloak discovery endpoint not reachable (HTTP 200) within timeout"
|
|
# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.
|
|
# HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a
|
|
# single argument. The previous `OCC user_oidc:provider ... --scope="openid email profile"`
|
|
# form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->
|
|
# occ "Too many arguments" -> silent failure on EVERY boot. Secret via --clientsecret-env
|
|
# (never on argv). The verify-then-retry loop guarantees the provider actually persisted.
|
|
r=0; reg_ok=0
|
|
while [ "$r" -lt 10 ]; do
|
|
su -s /bin/sh www-data -c "php /var/www/html/occ user_oidc:provider keycloak \
|
|
--clientid=nextcloud \
|
|
--clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \
|
|
--discoveryuri=$NEXTCLOUD_DISCOVERY_URL \
|
|
--scope='openid email profile' \
|
|
--unique-uid=0 \
|
|
--mapping-uid=preferred_username \
|
|
--mapping-email=email \
|
|
--mapping-display-name=name \
|
|
--group-provisioning=1 \
|
|
--mapping-groups=groups" >/dev/null 2>&1 || true
|
|
if OCC user_oidc:provider 2>/dev/null | grep -q "keycloak"; then
|
|
reg_ok=1; break
|
|
fi
|
|
r=$((r + 1)); sleep 3
|
|
done
|
|
if [ "$reg_ok" = 1 ]; then
|
|
echo "[nextcloud-oidc] provider 'keycloak' configured and verified"
|
|
else
|
|
nextcloud_oidc_fail "user_oidc:provider ran but the provider did not persist after retries"
|
|
fi
|
|
exposure: public # edge port-forward + public DNS (docs/09; R-DNS/R-NET)
|
|
egress: internet # normal WAN egress (docs/03; R-NET-05) — app store, integrity checks
|
|
storage:
|
|
- name: data
|
|
size: 20Gi
|
|
class: longhorn # ALL Nextcloud state (config/, data/, apps/, custom_apps/) on the
|
|
mountPath: /var/www/html # 3-replica tier; irreplaceable, DR-critical (docs/04; R-STATE-02).
|
|
# The whole html dir is persisted so config.php + user data survive a
|
|
# rebuild (the default /data/<name> would strand them on ephemeral rootfs).
|
|
dependencies:
|
|
- to: nextcloud-db # explicit allow: Nextcloud -> its PostgreSQL (docs/13 graph; R-NET-02).
|
|
port: 5432 # Renders the egress half; the DB namespace mirrors keycloak-db (no
|
|
protocol: tcp # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
|
|
ingressRoutes:
|
|
- host: nextcloud.home.keithjackson.dev
|
|
port: 80 # apache HTTP; wildcard TLS terminates at the shared Gateway (docs/08/09)
|
|
auth:
|
|
mode: oidc # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)
|
|
groups:
|
|
- platform-admins # access scoped to an authorized group (R-IDP-06)
|
|
# WIRED (native OIDC): the Keycloak `nextcloud` confidential client (platform/keycloak/keycloak-realm-
|
|
# import.yaml + live kcadm) is registered as a user_oidc login provider ("keycloak") by the postStart
|
|
# hook (occ user_oidc:provider), using the client secret mirrored into nextcloud-bootstrap
|
|
# (oidc-client-secret; same value SOPS-held in keycloak-realm-clients). The `groups` claim drives
|
|
# user_oidc group provisioning (R-IDP-06). Nextcloud KEEPS its LOCAL admin as the break-glass path so
|
|
# an IdP outage never locks the operator out (docs/15 "Availability caveat"). No secret is embedded here.
|
|
update:
|
|
channel: stable # R-UPD-01
|
|
range: ">=31.0.0 <32.0.0" # stay on the 31.x stable line
|
|
cadence: "0 4 * * *"
|
|
autoRollbackOnFailedHealth: true # R-UPD-03
|
|
observability:
|
|
metrics: true
|
|
slis:
|
|
- up # file-sync down is a critical alert (docs/10; R-OBS-01)
|
|
backup:
|
|
policy: daily-offcluster # DR-critical: data backed up off-cluster (docs/13, docs/04; R-STATE-03)
|