jakman85andClaude Opus 4.8
d34c639269
The Keycloak OIDC postStart registration for gitea and nextcloud ran but never persisted, leaving `gitea admin auth list` empty and nextcloud's user_oidc with "No providers configured" despite the apps installing/ enabling and the client secret being present. Two distinct root causes, confirmed by running the registration manually against the live warm pods: - nextcloud: a QUOTING BUG, not a race. The OCC() wrapper runs `su -c "php occ $*"`, which word-split `--scope="openid email profile"` back into three tokens, so occ saw `email profile` as extra positional args -> "Too many arguments to user_oidc:provider" -> exit 1, swallowed by `|| echo failed`. Failed on EVERY boot. Fixed by running the provider call as ONE `su -c` with the scope single-quoted so www-data's shell keeps it a single argument; keep the OCC() wrapper only for single-token subcommands. - gitea: a first-boot RACE. The direct `admin auth add-oauth` persists once the app/network is warm (verified live), but fired too early at postStart and its failure was swallowed (`|| echo ... >&2`, exit 0). Hardening applied to both catalog sources (rendered into apps/wave-*): - wait (bounded ~90s) for the Keycloak discovery doc to return HTTP 200; - nextcloud also waits for `installed: true` AND `maintenance: false`; - register idempotently (create-or-update) then VERIFY the source/provider actually persisted, retrying the whole cycle (guards exit-0-but-nothing- stuck); - on give-up, log LOUDLY to stderr AND a marker file on the persistent volume (/data, /var/www/html) instead of swallowing — next failure is visible; - never crashloop: missing-secret guard retained, break-glass local admin stays usable. Co-Authored-By: Claude Opus 4.8 <[email protected]>
186 lines
11 KiB
YAML
186 lines
11 KiB
YAML
# ServiceDefinition: gitea
|
|
# Source of truth: spec docs/13-service-catalog.md row 12; docs/15-identity-and-access.md; docs/05.
|
|
# public git hosting | egress internet | repos+DB on Longhorn (irreplaceable, DR-critical) | auth oidc.
|
|
# Bootstrapping-critical: self-hosts the platform's GitOps source + CI/CD, onboarded EARLY (docs/13
|
|
# "Gitea is bootstrapping-critical", WS-APPS step 2, right after PiHole). A cluster loss would take its
|
|
# own git host with it, so repos/DB are backed up off-cluster + mirrored (R-STATE-03).
|
|
# Single-container Gitea with the built-in SQLite datastore is sufficient at home scale; the repos and
|
|
# DB live on one Longhorn PVC (/data), so no external PostgreSQL is required for this onboarding.
|
|
# Validates against schemas/servicedefinition.schema.json.
|
|
name: gitea
|
|
kind: container
|
|
source:
|
|
image: gitea/gitea:1.24.3 # pinned recent stable tag (R-UPD-01); update.range governs advancement
|
|
# Headless-install bootstrap (formerly apps/gitea-bootstrap-patch.yaml; now first-class via the
|
|
# ServiceDefinition env/envFrom/lifecycle fields, schema >=1.2.0). Gitea must come up INSTALLED
|
|
# without the interactive web wizard and with a FIXED, persistent SECRET_KEY, so all of this is
|
|
# declared here and rendered verbatim onto the app container by catalog/render.py.
|
|
env:
|
|
- name: GITEA__security__INSTALL_LOCK # skip the first-run web installer (headless install)
|
|
value: "true"
|
|
- name: GITEA__security__SECRET_KEY # FIXED persistent key; MUST NOT regenerate per restart or
|
|
valueFrom: # previously-encrypted data breaks. Pinned via the SOPS Secret.
|
|
secretKeyRef:
|
|
name: gitea-bootstrap
|
|
key: secret-key
|
|
- name: GITEA__server__DOMAIN # correct public clone/redirect URLs for the TLS-terminated host
|
|
value: gitea.home.keithjackson.dev
|
|
- name: GITEA__server__ROOT_URL
|
|
value: https://gitea.home.keithjackson.dev/
|
|
- name: GITEA__actions__ENABLED # Phase 6 CI/CD: declared (not implicit upstream default) so the
|
|
value: "true" # in-cluster act_runner (ns gitea-actions) registers against it
|
|
- name: GITEA__actions__DEFAULT_ACTIONS_URL # resolve `uses: <org>/<action>@ref` from github.com (cluster
|
|
value: github # has internet egress; not air-gapped)
|
|
- name: GITEA_ADMIN_USERNAME # break-glass local admin creds (consumed by the postStart hook)
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: gitea-bootstrap
|
|
key: admin-username
|
|
- name: GITEA_ADMIN_PASSWORD
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: gitea-bootstrap
|
|
key: admin-password
|
|
# --- Keycloak native OIDC SSO (auth.mode oidc; docs/15 "Native OIDC"; R-IDP-02/06). The confidential
|
|
# `gitea` realm client's secret is mirrored into this namespace's gitea-bootstrap Secret (the same
|
|
# value SOPS-held in keycloak-realm-clients); the postStart hook registers it as the "keycloak"
|
|
# OAuth2 login source. The LOCAL admin above stays as the break-glass path (docs/15). ---
|
|
- name: GITEA_OIDC_CLIENT_SECRET
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: gitea-bootstrap
|
|
key: oidc-client-secret
|
|
# app.ini [oauth2_client]: auto-provision a Gitea account on first Keycloak login and link it to an
|
|
# existing local account by verified email, so SSO users don't need a pre-created local user.
|
|
- name: GITEA__oauth2_client__ENABLE_AUTO_REGISTRATION
|
|
value: "true"
|
|
- name: GITEA__oauth2_client__ACCOUNT_LINKING
|
|
value: auto
|
|
- name: GITEA__oauth2_client__USERNAME
|
|
value: preferred_username
|
|
lifecycle:
|
|
postStart:
|
|
exec:
|
|
command:
|
|
- /bin/sh
|
|
- -c
|
|
- |
|
|
# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint
|
|
# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first
|
|
# boot). Waits for the DB, then creates the admin only if it does not exist.
|
|
set -u
|
|
export HOME=/data/git
|
|
i=0
|
|
until su-exec git gitea admin user list >/dev/null 2>&1; do
|
|
i=$((i + 1))
|
|
[ "$i" -ge 150 ] && { echo "[gitea-bootstrap] DB not ready after wait; skipping" >&2; exit 0; }
|
|
sleep 2
|
|
done
|
|
if su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx "$GITEA_ADMIN_USERNAME"; then
|
|
echo "[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present"
|
|
else
|
|
n=0
|
|
until su-exec git gitea admin user create \
|
|
--username "$GITEA_ADMIN_USERNAME" \
|
|
--password "$GITEA_ADMIN_PASSWORD" \
|
|
--email "[email protected]" \
|
|
--admin --must-change-password=false; do
|
|
n=$((n + 1))
|
|
[ "$n" -ge 10 ] && { echo "[gitea-bootstrap] admin create failed after retries" >&2; break; }
|
|
sleep 3
|
|
done
|
|
echo "[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'"
|
|
fi
|
|
# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---
|
|
# Registers the confidential `gitea` client as the OAuth2 login source "keycloak" against the
|
|
# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin
|
|
# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled
|
|
# secret never crashes the pod.
|
|
#
|
|
# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was
|
|
# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.
|
|
# Now it (1) waits — bounded, ~90s — for the Keycloak discovery doc to serve HTTP 200,
|
|
# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually
|
|
# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker
|
|
# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays
|
|
# usable). The verify-then-retry loop is the real guarantee against "exit 0 but nothing stuck".
|
|
GITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
|
|
gitea_oidc_fail() {
|
|
echo "[gitea-oidc] FAILED to register 'keycloak' login source: $1" >&2
|
|
{ echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true
|
|
}
|
|
if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then
|
|
echo "[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)"
|
|
else
|
|
# (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).
|
|
d=0; disc_ok=0
|
|
while [ "$d" -lt 45 ]; do
|
|
if [ "$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 "$GITEA_DISCOVERY_URL" 2>/dev/null)" = "200" ]; then
|
|
disc_ok=1; break
|
|
fi
|
|
d=$((d + 1)); sleep 2
|
|
done
|
|
[ "$disc_ok" = 1 ] || gitea_oidc_fail "Keycloak discovery endpoint not reachable (HTTP 200) within timeout"
|
|
# (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.
|
|
r=0; reg_ok=0
|
|
while [ "$r" -lt 10 ]; do
|
|
oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2=="keycloak"{print $1; exit}')
|
|
if [ -n "$oidc_id" ]; then
|
|
su-exec git gitea admin auth update-oauth --id "$oidc_id" \
|
|
--name keycloak --provider openidConnect --key gitea \
|
|
--secret "$GITEA_OIDC_CLIENT_SECRET" \
|
|
--auto-discover-url "$GITEA_DISCOVERY_URL" \
|
|
--scopes "openid email profile" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true
|
|
else
|
|
su-exec git gitea admin auth add-oauth \
|
|
--name keycloak --provider openidConnect --key gitea \
|
|
--secret "$GITEA_OIDC_CLIENT_SECRET" \
|
|
--auto-discover-url "$GITEA_DISCOVERY_URL" \
|
|
--scopes "openid email profile" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true
|
|
fi
|
|
# VERIFY: the "keycloak" source must actually be listed now (guards the exit-0-but-nothing-persisted race).
|
|
if su-exec git gitea admin auth list 2>/dev/null | awk '$2=="keycloak"{f=1} END{exit !f}'; then
|
|
reg_ok=1; break
|
|
fi
|
|
r=$((r + 1)); sleep 3
|
|
done
|
|
if [ "$reg_ok" = 1 ]; then
|
|
echo "[gitea-oidc] login source 'keycloak' registered and verified"
|
|
else
|
|
gitea_oidc_fail "add/update-oauth ran but the 'keycloak' source did not persist after retries"
|
|
fi
|
|
fi
|
|
exposure: public # edge-forwarded git host at gitea.home.keithjackson.dev (docs/09; R-DNS/R-NET)
|
|
egress: internet # normal WAN egress (docs/03; R-NET-05)
|
|
storage:
|
|
- name: data
|
|
size: 10Gi
|
|
class: longhorn # repos + SQLite DB on 3-replica Longhorn; irreplaceable (docs/04; R-STATE-02)
|
|
mountPath: /data # Gitea keeps ALL state (repos, DB, app.ini, INTERNAL_TOKEN) directly under /data;
|
|
# the default /data/<name> would strand real state on the ephemeral rootfs (R-STATE-01)
|
|
dependencies: [] # default-deny: reaches only DNS; git clients reach it north-south via the Gateway (R-NET-01/02)
|
|
ingressRoutes:
|
|
- host: gitea.home.keithjackson.dev
|
|
port: 3000 # Gitea HTTP web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
|
|
auth:
|
|
mode: oidc # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC" set; R-IDP-02)
|
|
groups:
|
|
- platform-admins # access scoped to an authorized group (R-IDP-06)
|
|
# WIRED (native OIDC): the Keycloak `gitea` confidential client (platform/keycloak/keycloak-realm-
|
|
# import.yaml + live kcadm) is registered as Gitea's "keycloak" OAuth2 login source by the postStart
|
|
# hook, using the client secret mirrored into the gitea-bootstrap Secret (oidc-client-secret; same
|
|
# value SOPS-held in keycloak-realm-clients). platform-admins (groups claim) -> Gitea admin. Gitea
|
|
# KEEPS a LOCAL admin as the break-glass path so an IdP outage never locks the operator out of
|
|
# recovery tools (docs/15 "Availability caveat"). No secret is embedded here.
|
|
update:
|
|
channel: stable # R-UPD-01
|
|
range: ">=1.22.0 <2.0.0" # stay on the 1.x stable line
|
|
cadence: "0 4 * * *"
|
|
autoRollbackOnFailedHealth: true # R-UPD-03
|
|
observability:
|
|
metrics: true
|
|
slis:
|
|
- up # git host down is a critical alert (docs/10; R-OBS-01)
|
|
backup:
|
|
policy: daily-offcluster # DR-critical: repos/DB backed up off-cluster + mirrored (docs/13, docs/04; R-STATE-03)
|