jakman85andClaude Opus 5
3484955903
Both ServiceDefinitions guarded their Keycloak OIDC registration with an UNESCAPED
braced expansion:
if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then # gitea
if [ -z "${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then # nextcloud, then exit 0
apps-wave-0/1 both declare postBuild.substituteFrom: cluster-vars, and neither name is a
cluster-vars key — so Flux's envsubst substituted them away AT RECONCILE TIME, long
before the container ever ran. Verified in the LIVE cluster: the deployed Deployments
literally contain
if [ -z "" ]; then
which is always true. The entire OIDC registration block has been skipped on EVERY boot
of both apps.
WHY NOBODY NOTICED: both apps DO have the provider registered — `gitea admin auth list`
shows keycloak/OAuth2/true and `occ user_oidc:provider` shows the keycloak provider.
Those live in each app's own database, persisted from an earlier successful registration.
SSO works today and DECISIONS-LOG is not wrong. But the idempotent re-registration is
dead, so any PVC rebuild, DB restore, or fresh cluster build comes up with NO OIDC and
only the break-glass local admin. A PROD BUILD FROM THIS REPO WOULD SILENTLY HAVE NO SSO
on gitea or nextcloud.
The irony: the surrounding code was hardened twice — see its own comments about a "silent
first-boot RACE", the verify-then-retry loops, the marker files. All of that hardening was
unreachable, because the guard above it always evaluated true.
Not a regression from the recent R-ENV-02 tokenization — introduced 2026-08-27 (8738871).
FIXED AT THE RIGHT SEAM: apps/wave-0/gitea.yaml and apps/wave-1/nextcloud.yaml are
RENDER-GENERATED, so the fix is in catalog/*.yaml and the goldens are regenerated.
Escaped to $${...}, the convention documented in clusters/base/apps-vms.yaml. Post-envsubst
the container now receives the intended `${VAR:-}` instead of `""`.
STRICT MODE WOULD NEVER HAVE CAUGHT THIS. Flux deliberately exempts default-providing
operators (-, :-, +, :+, =, :=, ?, :?) from StrictPostBuildSubstitutions, because they
"intentionally handle the unset case". So a targeted check was required.
MADE UN-REPEATABLE IN THE RENDERER: catalog/render.py gains a substitution-safety section
wired into emit(), so every render path refuses to emit a manifest containing a braced
expansion that is not exactly ${KNOWN_KEY} for a key read LIVE from cluster-vars (no
hardcoded copy to drift; unreadable => empty set => fail-closed). Dollar-runs are counted
properly ($$ is a literal $ to envsubst, so odd runs are flagged and even runs are not),
and indexed forms get their own hard-error message. Nine new --selftest assertions,
including the exact regression. Proven to bite: rendering the pre-fix catalog files from
HEAD now exits 1 with the precise diagnosis.
Sweep: the complete substituted surface was enumerated from every Kustomization's path.
catalog/ contains exactly TWO braced expansions and both were these bugs; every other
braced expansion on a substituted path is a declared cluster-vars key (zero typos, zero
undeclared names); existing $${UOS_URL} escapes verified intact.
Verified: --selftest PASS; canonical render is byte-identical to the committed apps/ for
all 10 onboarded services; the three L1 render checks pass.
NEXT RECONCILE: both pod templates change, so gitea (wave 0) and nextcloud (wave 1) roll —
strategy: Recreate on RWO Longhorn PVCs, so expect a brief single-pod outage each. The
OIDC block then executes for the first time in these apps' deployed lifetime. Both paths
are idempotent (gitea takes auth update-oauth on the existing id; nextcloud's
user_oidc:provider is create-or-update), so the expected result is a no-op re-write plus
the verify loop logging success. If the SOPS client secret ever diverged from Keycloak's,
this is when it gets corrected instead of silently left stale. Tail both pods after merge.
Serves: R-IDP-02, R-IDP-06, R-ENV-02, R-GITOPS-02.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
198 lines
12 KiB
YAML
198 lines
12 KiB
YAML
# ServiceDefinition: gitea
|
|
# Source of truth: spec docs/13-service-catalog.md row 12; docs/15-identity-and-access.md; docs/05.
|
|
# public git hosting | egress internet | repos+DB on Longhorn (irreplaceable, DR-critical) | auth oidc.
|
|
# Bootstrapping-critical: self-hosts the platform's GitOps source + CI/CD, onboarded EARLY (docs/13
|
|
# "Gitea is bootstrapping-critical", WS-APPS step 2, right after PiHole). A cluster loss would take its
|
|
# own git host with it, so repos/DB are backed up off-cluster + mirrored (R-STATE-03).
|
|
# Single-container Gitea with the built-in SQLite datastore is sufficient at home scale; the repos and
|
|
# DB live on one Longhorn PVC (/data), so no external PostgreSQL is required for this onboarding.
|
|
# Validates against schemas/servicedefinition.schema.json.
|
|
name: gitea
|
|
kind: container
|
|
source:
|
|
image: gitea/gitea:1.24.3 # pinned recent stable tag (R-UPD-01); update.range governs advancement
|
|
# Headless-install bootstrap (formerly apps/gitea-bootstrap-patch.yaml; now first-class via the
|
|
# ServiceDefinition env/envFrom/lifecycle fields, schema >=1.2.0). Gitea must come up INSTALLED
|
|
# without the interactive web wizard and with a FIXED, persistent SECRET_KEY, so all of this is
|
|
# declared here and rendered verbatim onto the app container by catalog/render.py.
|
|
env:
|
|
- name: GITEA__security__INSTALL_LOCK # skip the first-run web installer (headless install)
|
|
value: "true"
|
|
- name: GITEA__security__SECRET_KEY # FIXED persistent key; MUST NOT regenerate per restart or
|
|
valueFrom: # previously-encrypted data breaks. Pinned via the SOPS Secret.
|
|
secretKeyRef:
|
|
name: gitea-bootstrap
|
|
key: secret-key
|
|
- name: GITEA__server__DOMAIN # correct public clone/redirect URLs for the TLS-terminated host
|
|
value: gitea.home.keithjackson.dev
|
|
- name: GITEA__server__ROOT_URL
|
|
value: https://gitea.home.keithjackson.dev/
|
|
- name: GITEA__actions__ENABLED # Phase 6 CI/CD: declared (not implicit upstream default) so the
|
|
value: "true" # in-cluster act_runner (ns gitea-actions) registers against it
|
|
- name: GITEA__actions__DEFAULT_ACTIONS_URL # resolve `uses: <org>/<action>@ref` from github.com (cluster
|
|
value: github # has internet egress; not air-gapped)
|
|
- name: GITEA_ADMIN_USERNAME # break-glass local admin creds (consumed by the postStart hook)
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: gitea-bootstrap
|
|
key: admin-username
|
|
- name: GITEA_ADMIN_PASSWORD
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: gitea-bootstrap
|
|
key: admin-password
|
|
# --- Keycloak native OIDC SSO (auth.mode oidc; docs/15 "Native OIDC"; R-IDP-02/06). The confidential
|
|
# `gitea` realm client's secret is mirrored into this namespace's gitea-bootstrap Secret (the same
|
|
# value SOPS-held in keycloak-realm-clients); the postStart hook registers it as the "keycloak"
|
|
# OAuth2 login source. The LOCAL admin above stays as the break-glass path (docs/15). ---
|
|
- name: GITEA_OIDC_CLIENT_SECRET
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: gitea-bootstrap
|
|
key: oidc-client-secret
|
|
# app.ini [oauth2_client]: auto-provision a Gitea account on first Keycloak login and link it to an
|
|
# existing local account by verified email, so SSO users don't need a pre-created local user.
|
|
- name: GITEA__oauth2_client__ENABLE_AUTO_REGISTRATION
|
|
value: "true"
|
|
- name: GITEA__oauth2_client__ACCOUNT_LINKING
|
|
value: auto
|
|
- name: GITEA__oauth2_client__USERNAME
|
|
value: preferred_username
|
|
lifecycle:
|
|
postStart:
|
|
exec:
|
|
command:
|
|
- /bin/sh
|
|
- -c
|
|
- |
|
|
# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint
|
|
# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first
|
|
# boot). Waits for the DB, then creates the admin only if it does not exist.
|
|
set -u
|
|
export HOME=/data/git
|
|
i=0
|
|
until su-exec git gitea admin user list >/dev/null 2>&1; do
|
|
i=$((i + 1))
|
|
[ "$i" -ge 150 ] && { echo "[gitea-bootstrap] DB not ready after wait; skipping" >&2; exit 0; }
|
|
sleep 2
|
|
done
|
|
if su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx "$GITEA_ADMIN_USERNAME"; then
|
|
echo "[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present"
|
|
else
|
|
n=0
|
|
until su-exec git gitea admin user create \
|
|
--username "$GITEA_ADMIN_USERNAME" \
|
|
--password "$GITEA_ADMIN_PASSWORD" \
|
|
--email "[email protected]" \
|
|
--admin --must-change-password=false; do
|
|
n=$((n + 1))
|
|
[ "$n" -ge 10 ] && { echo "[gitea-bootstrap] admin create failed after retries" >&2; break; }
|
|
sleep 3
|
|
done
|
|
echo "[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'"
|
|
fi
|
|
# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---
|
|
# Registers the confidential `gitea` client as the OAuth2 login source "keycloak" against the
|
|
# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin
|
|
# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled
|
|
# secret never crashes the pod.
|
|
#
|
|
# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was
|
|
# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.
|
|
# Now it (1) waits — bounded, ~90s — for the Keycloak discovery doc to serve HTTP 200,
|
|
# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually
|
|
# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker
|
|
# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays
|
|
# usable). The verify-then-retry loop is the real guarantee against "exit 0 but nothing stuck".
|
|
GITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
|
|
gitea_oidc_fail() {
|
|
echo "[gitea-oidc] FAILED to register 'keycloak' login source: $1" >&2
|
|
{ echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true
|
|
}
|
|
# NOTE the `$$` escape below — it is load-bearing. apps-wave-0 declares postBuild.
|
|
# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE
|
|
# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not
|
|
# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z "" ]` — ALWAYS TRUE
|
|
# — making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on
|
|
# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was
|
|
# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.
|
|
# Flux STRICT substitution cannot catch this — it EXEMPTS default-operator forms by design.
|
|
# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.
|
|
# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping — which is why the
|
|
# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.
|
|
# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.
|
|
if [ -z "$${GITEA_OIDC_CLIENT_SECRET:-}" ]; then
|
|
echo "[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)"
|
|
else
|
|
# (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).
|
|
d=0; disc_ok=0
|
|
while [ "$d" -lt 45 ]; do
|
|
if [ "$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 "$GITEA_DISCOVERY_URL" 2>/dev/null)" = "200" ]; then
|
|
disc_ok=1; break
|
|
fi
|
|
d=$((d + 1)); sleep 2
|
|
done
|
|
[ "$disc_ok" = 1 ] || gitea_oidc_fail "Keycloak discovery endpoint not reachable (HTTP 200) within timeout"
|
|
# (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.
|
|
r=0; reg_ok=0
|
|
while [ "$r" -lt 10 ]; do
|
|
oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2=="keycloak"{print $1; exit}')
|
|
if [ -n "$oidc_id" ]; then
|
|
su-exec git gitea admin auth update-oauth --id "$oidc_id" \
|
|
--name keycloak --provider openidConnect --key gitea \
|
|
--secret "$GITEA_OIDC_CLIENT_SECRET" \
|
|
--auto-discover-url "$GITEA_DISCOVERY_URL" \
|
|
--scopes "openid email profile" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true
|
|
else
|
|
su-exec git gitea admin auth add-oauth \
|
|
--name keycloak --provider openidConnect --key gitea \
|
|
--secret "$GITEA_OIDC_CLIENT_SECRET" \
|
|
--auto-discover-url "$GITEA_DISCOVERY_URL" \
|
|
--scopes "openid email profile" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true
|
|
fi
|
|
# VERIFY: the "keycloak" source must actually be listed now (guards the exit-0-but-nothing-persisted race).
|
|
if su-exec git gitea admin auth list 2>/dev/null | awk '$2=="keycloak"{f=1} END{exit !f}'; then
|
|
reg_ok=1; break
|
|
fi
|
|
r=$((r + 1)); sleep 3
|
|
done
|
|
if [ "$reg_ok" = 1 ]; then
|
|
echo "[gitea-oidc] login source 'keycloak' registered and verified"
|
|
else
|
|
gitea_oidc_fail "add/update-oauth ran but the 'keycloak' source did not persist after retries"
|
|
fi
|
|
fi
|
|
exposure: public # edge-forwarded git host at gitea.home.keithjackson.dev (docs/09; R-DNS/R-NET)
|
|
egress: internet # normal WAN egress (docs/03; R-NET-05)
|
|
storage:
|
|
- name: data
|
|
size: 10Gi
|
|
class: longhorn # repos + SQLite DB on 3-replica Longhorn; irreplaceable (docs/04; R-STATE-02)
|
|
mountPath: /data # Gitea keeps ALL state (repos, DB, app.ini, INTERNAL_TOKEN) directly under /data;
|
|
# the default /data/<name> would strand real state on the ephemeral rootfs (R-STATE-01)
|
|
dependencies: [] # default-deny: reaches only DNS; git clients reach it north-south via the Gateway (R-NET-01/02)
|
|
ingressRoutes:
|
|
- host: gitea.home.keithjackson.dev
|
|
port: 3000 # Gitea HTTP web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
|
|
auth:
|
|
mode: oidc # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC" set; R-IDP-02)
|
|
groups:
|
|
- platform-admins # access scoped to an authorized group (R-IDP-06)
|
|
# WIRED (native OIDC): the Keycloak `gitea` confidential client (platform/keycloak/keycloak-realm-
|
|
# import.yaml + live kcadm) is registered as Gitea's "keycloak" OAuth2 login source by the postStart
|
|
# hook, using the client secret mirrored into the gitea-bootstrap Secret (oidc-client-secret; same
|
|
# value SOPS-held in keycloak-realm-clients). platform-admins (groups claim) -> Gitea admin. Gitea
|
|
# KEEPS a LOCAL admin as the break-glass path so an IdP outage never locks the operator out of
|
|
# recovery tools (docs/15 "Availability caveat"). No secret is embedded here.
|
|
update:
|
|
channel: stable # R-UPD-01
|
|
range: ">=1.22.0 <2.0.0" # stay on the 1.x stable line
|
|
cadence: "0 4 * * *"
|
|
autoRollbackOnFailedHealth: true # R-UPD-03
|
|
observability:
|
|
metrics: true
|
|
slis:
|
|
- up # git host down is a critical alert (docs/10; R-OBS-01)
|
|
backup:
|
|
policy: daily-offcluster # DR-critical: repos/DB backed up off-cluster + mirrored (docs/13, docs/04; R-STATE-03)
|