jakman85andClaude Opus 5
4c4f5355d9
LIVE REGRESSION, caught on the reconcile after merging #5. nextcloud went
CrashLoopBackOff with FailedPostStartHook, 7 restarts, and apps-wave-1/2 stalled.
ROOT CAUSE — a latent bug that could never fire until now. The final statement of
nextcloud's postStart script was:
[ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote ... after retries"
On the SUCCESS path (p < 15) the test returns 1, and as the script's LAST command that
becomes the hook's exit status. A non-zero postStart exit makes the kubelet KILL the
container. This was harmless for as long as the OIDC guard above it was dead code that
exited 0 first (see 3484955) — fixing that guard made the block execute for the first
time in these apps' deployed lifetime, and the bug fired immediately.
That is the honest sequence: the previous commit did not introduce this defect, it
UNCOVERED it. Two bugs stacked, and the outer one hid the inner one.
FIX: convert to a plain `if`, then `exit 0` explicitly, with a comment recording why.
Also made gitea's hook explicit. Its last statement already returned 0 on both paths, so
it was correct by accident of ordering; leaving one hook implicitly-safe and one
explicitly-safe invites exactly this class back. Both now end with an unconditional
`exit 0`, and both say why in-file.
Every failure path in both hooks still logs loudly to stderr AND writes a marker file on
the persistent volume, and deliberately does NOT crash the pod — the break-glass local
admin must stay usable. That contract is now actually honoured; before, the success path
was the one that crashed.
Fixed in catalog/ (the ServiceDefinitions are the source of truth) and goldens
regenerated; render==golden green, renderer selftest PASS.
Serves: R-IDP-02, R-IDP-06, R-GITOPS-02.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
204 lines
13 KiB
YAML
204 lines
13 KiB
YAML
# ServiceDefinition: gitea
|
|
# Source of truth: spec docs/13-service-catalog.md row 12; docs/15-identity-and-access.md; docs/05.
|
|
# public git hosting | egress internet | repos+DB on Longhorn (irreplaceable, DR-critical) | auth oidc.
|
|
# Bootstrapping-critical: self-hosts the platform's GitOps source + CI/CD, onboarded EARLY (docs/13
|
|
# "Gitea is bootstrapping-critical", WS-APPS step 2, right after PiHole). A cluster loss would take its
|
|
# own git host with it, so repos/DB are backed up off-cluster + mirrored (R-STATE-03).
|
|
# Single-container Gitea with the built-in SQLite datastore is sufficient at home scale; the repos and
|
|
# DB live on one Longhorn PVC (/data), so no external PostgreSQL is required for this onboarding.
|
|
# Validates against schemas/servicedefinition.schema.json.
|
|
name: gitea
|
|
kind: container
|
|
source:
|
|
image: gitea/gitea:1.24.3 # pinned recent stable tag (R-UPD-01); update.range governs advancement
|
|
# Headless-install bootstrap (formerly apps/gitea-bootstrap-patch.yaml; now first-class via the
|
|
# ServiceDefinition env/envFrom/lifecycle fields, schema >=1.2.0). Gitea must come up INSTALLED
|
|
# without the interactive web wizard and with a FIXED, persistent SECRET_KEY, so all of this is
|
|
# declared here and rendered verbatim onto the app container by catalog/render.py.
|
|
env:
|
|
- name: GITEA__security__INSTALL_LOCK # skip the first-run web installer (headless install)
|
|
value: "true"
|
|
- name: GITEA__security__SECRET_KEY # FIXED persistent key; MUST NOT regenerate per restart or
|
|
valueFrom: # previously-encrypted data breaks. Pinned via the SOPS Secret.
|
|
secretKeyRef:
|
|
name: gitea-bootstrap
|
|
key: secret-key
|
|
- name: GITEA__server__DOMAIN # correct public clone/redirect URLs for the TLS-terminated host
|
|
value: gitea.home.keithjackson.dev
|
|
- name: GITEA__server__ROOT_URL
|
|
value: https://gitea.home.keithjackson.dev/
|
|
- name: GITEA__actions__ENABLED # Phase 6 CI/CD: declared (not implicit upstream default) so the
|
|
value: "true" # in-cluster act_runner (ns gitea-actions) registers against it
|
|
- name: GITEA__actions__DEFAULT_ACTIONS_URL # resolve `uses: <org>/<action>@ref` from github.com (cluster
|
|
value: github # has internet egress; not air-gapped)
|
|
- name: GITEA_ADMIN_USERNAME # break-glass local admin creds (consumed by the postStart hook)
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: gitea-bootstrap
|
|
key: admin-username
|
|
- name: GITEA_ADMIN_PASSWORD
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: gitea-bootstrap
|
|
key: admin-password
|
|
# --- Keycloak native OIDC SSO (auth.mode oidc; docs/15 "Native OIDC"; R-IDP-02/06). The confidential
|
|
# `gitea` realm client's secret is mirrored into this namespace's gitea-bootstrap Secret (the same
|
|
# value SOPS-held in keycloak-realm-clients); the postStart hook registers it as the "keycloak"
|
|
# OAuth2 login source. The LOCAL admin above stays as the break-glass path (docs/15). ---
|
|
- name: GITEA_OIDC_CLIENT_SECRET
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: gitea-bootstrap
|
|
key: oidc-client-secret
|
|
# app.ini [oauth2_client]: auto-provision a Gitea account on first Keycloak login and link it to an
|
|
# existing local account by verified email, so SSO users don't need a pre-created local user.
|
|
- name: GITEA__oauth2_client__ENABLE_AUTO_REGISTRATION
|
|
value: "true"
|
|
- name: GITEA__oauth2_client__ACCOUNT_LINKING
|
|
value: auto
|
|
- name: GITEA__oauth2_client__USERNAME
|
|
value: preferred_username
|
|
lifecycle:
|
|
postStart:
|
|
exec:
|
|
command:
|
|
- /bin/sh
|
|
- -c
|
|
- |
|
|
# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint
|
|
# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first
|
|
# boot). Waits for the DB, then creates the admin only if it does not exist.
|
|
set -u
|
|
export HOME=/data/git
|
|
i=0
|
|
until su-exec git gitea admin user list >/dev/null 2>&1; do
|
|
i=$((i + 1))
|
|
[ "$i" -ge 150 ] && { echo "[gitea-bootstrap] DB not ready after wait; skipping" >&2; exit 0; }
|
|
sleep 2
|
|
done
|
|
if su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx "$GITEA_ADMIN_USERNAME"; then
|
|
echo "[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present"
|
|
else
|
|
n=0
|
|
until su-exec git gitea admin user create \
|
|
--username "$GITEA_ADMIN_USERNAME" \
|
|
--password "$GITEA_ADMIN_PASSWORD" \
|
|
--email "[email protected]" \
|
|
--admin --must-change-password=false; do
|
|
n=$((n + 1))
|
|
[ "$n" -ge 10 ] && { echo "[gitea-bootstrap] admin create failed after retries" >&2; break; }
|
|
sleep 3
|
|
done
|
|
echo "[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'"
|
|
fi
|
|
# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---
|
|
# Registers the confidential `gitea` client as the OAuth2 login source "keycloak" against the
|
|
# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin
|
|
# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled
|
|
# secret never crashes the pod.
|
|
#
|
|
# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was
|
|
# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.
|
|
# Now it (1) waits — bounded, ~90s — for the Keycloak discovery doc to serve HTTP 200,
|
|
# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually
|
|
# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker
|
|
# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays
|
|
# usable). The verify-then-retry loop is the real guarantee against "exit 0 but nothing stuck".
|
|
GITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
|
|
gitea_oidc_fail() {
|
|
echo "[gitea-oidc] FAILED to register 'keycloak' login source: $1" >&2
|
|
{ echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true
|
|
}
|
|
# NOTE the `$$` escape below — it is load-bearing. apps-wave-0 declares postBuild.
|
|
# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE
|
|
# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not
|
|
# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z "" ]` — ALWAYS TRUE
|
|
# — making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on
|
|
# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was
|
|
# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.
|
|
# Flux STRICT substitution cannot catch this — it EXEMPTS default-operator forms by design.
|
|
# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.
|
|
# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping — which is why the
|
|
# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.
|
|
# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.
|
|
if [ -z "$${GITEA_OIDC_CLIENT_SECRET:-}" ]; then
|
|
echo "[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)"
|
|
else
|
|
# (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).
|
|
d=0; disc_ok=0
|
|
while [ "$d" -lt 45 ]; do
|
|
if [ "$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 "$GITEA_DISCOVERY_URL" 2>/dev/null)" = "200" ]; then
|
|
disc_ok=1; break
|
|
fi
|
|
d=$((d + 1)); sleep 2
|
|
done
|
|
[ "$disc_ok" = 1 ] || gitea_oidc_fail "Keycloak discovery endpoint not reachable (HTTP 200) within timeout"
|
|
# (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.
|
|
r=0; reg_ok=0
|
|
while [ "$r" -lt 10 ]; do
|
|
oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2=="keycloak"{print $1; exit}')
|
|
if [ -n "$oidc_id" ]; then
|
|
su-exec git gitea admin auth update-oauth --id "$oidc_id" \
|
|
--name keycloak --provider openidConnect --key gitea \
|
|
--secret "$GITEA_OIDC_CLIENT_SECRET" \
|
|
--auto-discover-url "$GITEA_DISCOVERY_URL" \
|
|
--scopes "openid email profile" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true
|
|
else
|
|
su-exec git gitea admin auth add-oauth \
|
|
--name keycloak --provider openidConnect --key gitea \
|
|
--secret "$GITEA_OIDC_CLIENT_SECRET" \
|
|
--auto-discover-url "$GITEA_DISCOVERY_URL" \
|
|
--scopes "openid email profile" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true
|
|
fi
|
|
# VERIFY: the "keycloak" source must actually be listed now (guards the exit-0-but-nothing-persisted race).
|
|
if su-exec git gitea admin auth list 2>/dev/null | awk '$2=="keycloak"{f=1} END{exit !f}'; then
|
|
reg_ok=1; break
|
|
fi
|
|
r=$((r + 1)); sleep 3
|
|
done
|
|
if [ "$reg_ok" = 1 ]; then
|
|
echo "[gitea-oidc] login source 'keycloak' registered and verified"
|
|
else
|
|
gitea_oidc_fail "add/update-oauth ran but the 'keycloak' source did not persist after retries"
|
|
fi
|
|
fi
|
|
# A postStart hook MUST exit 0 — a non-zero exit makes the kubelet KILL the container
|
|
# (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but
|
|
# it is made EXPLICIT so the guarantee does not depend on whatever statement happens to
|
|
# be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,
|
|
# the moment its OIDC guard was fixed and the block actually ran for the first time.
|
|
exit 0
|
|
exposure: public # edge-forwarded git host at gitea.home.keithjackson.dev (docs/09; R-DNS/R-NET)
|
|
egress: internet # normal WAN egress (docs/03; R-NET-05)
|
|
storage:
|
|
- name: data
|
|
size: 10Gi
|
|
class: longhorn # repos + SQLite DB on 3-replica Longhorn; irreplaceable (docs/04; R-STATE-02)
|
|
mountPath: /data # Gitea keeps ALL state (repos, DB, app.ini, INTERNAL_TOKEN) directly under /data;
|
|
# the default /data/<name> would strand real state on the ephemeral rootfs (R-STATE-01)
|
|
dependencies: [] # default-deny: reaches only DNS; git clients reach it north-south via the Gateway (R-NET-01/02)
|
|
ingressRoutes:
|
|
- host: gitea.home.keithjackson.dev
|
|
port: 3000 # Gitea HTTP web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
|
|
auth:
|
|
mode: oidc # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC" set; R-IDP-02)
|
|
groups:
|
|
- platform-admins # access scoped to an authorized group (R-IDP-06)
|
|
# WIRED (native OIDC): the Keycloak `gitea` confidential client (platform/keycloak/keycloak-realm-
|
|
# import.yaml + live kcadm) is registered as Gitea's "keycloak" OAuth2 login source by the postStart
|
|
# hook, using the client secret mirrored into the gitea-bootstrap Secret (oidc-client-secret; same
|
|
# value SOPS-held in keycloak-realm-clients). platform-admins (groups claim) -> Gitea admin. Gitea
|
|
# KEEPS a LOCAL admin as the break-glass path so an IdP outage never locks the operator out of
|
|
# recovery tools (docs/15 "Availability caveat"). No secret is embedded here.
|
|
update:
|
|
channel: stable # R-UPD-01
|
|
range: ">=1.22.0 <2.0.0" # stay on the 1.x stable line
|
|
cadence: "0 4 * * *"
|
|
autoRollbackOnFailedHealth: true # R-UPD-03
|
|
observability:
|
|
metrics: true
|
|
slis:
|
|
- up # git host down is a critical alert (docs/10; R-OBS-01)
|
|
backup:
|
|
policy: daily-offcluster # DR-critical: repos/DB backed up off-cluster + mirrored (docs/13, docs/04; R-STATE-03)
|