Files
jakman85andClaude Opus 5.5 8a9baa92bc
gate / gate (pull_request) Successful in 3m13s
gate / conformance (pull_request) Skipped
feat(gitea): upgrade 1.27.3 -> 28.3.0
Gitea dropped the "1." prefix; 28.x is the successor of 1.27.
- image 28.3.0, update.range >=28.0.0 <29.0.0, Renovate ceiling <29.0.0
- drop GITEA__server__DOMAIN (no longer read in 28, ROOT_URL is the source)
- declare RUN_RETENTION_DAYS=400, EGRESS_MODE=lax (no allow list: the OIDC
  login source is not gated by the egress policy in 28.3.0), and
  DISABLE_REGISTRATION=true (28's new default)
- document autoRollbackOnFailedHealth as inert (no ImagePolicy is rendered);
  rollback is a PVC restore because the DB migration is one-way

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-07 22:55:06 -06:00

279 lines
19 KiB
YAML

# ServiceDefinition: gitea
# Source of truth: spec docs/13-service-catalog.md row 12; docs/15-identity-and-access.md; docs/05.
# public git hosting | egress internet | repos+DB on Longhorn (irreplaceable, DR-critical) | auth oidc.
# Bootstrapping-critical: self-hosts the platform's GitOps source + CI/CD, onboarded EARLY (docs/13
# "Gitea is bootstrapping-critical", WS-APPS step 2, right after PiHole). A cluster loss would take its
# own git host with it, so repos/DB are backed up off-cluster + mirrored (R-STATE-03).
# Single-container Gitea with the built-in SQLite datastore is sufficient at home scale; the repos and
# DB live on one Longhorn PVC (/data), so no external PostgreSQL is required for this onboarding.
# Validates against schemas/servicedefinition.schema.json.
name: gitea
kind: container
source:
image: gitea/gitea:28.3.0 # pinned recent stable tag (R-UPD-01); update.range governs advancement
# Headless-install bootstrap (formerly apps/gitea-bootstrap-patch.yaml; now first-class via the
# ServiceDefinition env/envFrom/lifecycle fields, schema >=1.2.0). Gitea must come up INSTALLED
# without the interactive web wizard and with a FIXED, persistent SECRET_KEY, so all of this is
# declared here and rendered verbatim onto the app container by catalog/render.py.
env:
- name: GITEA__security__INSTALL_LOCK # skip the first-run web installer (headless install)
value: "true"
- name: GITEA__security__SECRET_KEY # FIXED persistent key; MUST NOT regenerate per restart or
valueFrom: # previously-encrypted data breaks. Pinned via the SOPS Secret.
secretKeyRef:
name: gitea-bootstrap
key: secret-key
# ROOT_URL is the ONLY source of the instance domain since Gitea 28 (go-gitea/gitea#39400): [server]
# DOMAIN is no longer read, and the SSH domain defaults from ROOT_URL too. GITEA__server__DOMAIN was
# removed in the 28 upgrade; the stale DOMAIN line environment-to-ini already wrote into app.ini is
# inert. Correct public clone/redirect URLs for the TLS-terminated host come from this one value.
- name: GITEA__server__ROOT_URL
value: https://gitea.${BASE_DOMAIN}/
- name: GITEA__actions__ENABLED # Phase 6 CI/CD: declared (not implicit upstream default) so the
value: "true" # in-cluster act_runner (ns gitea-actions) registers against it
- name: GITEA__actions__DEFAULT_ACTIONS_URL # resolve `uses: <org>/<action>@ref` from github.com (cluster
value: github # has internet egress; not air-gapped)
# Gitea 28 added run retention (go-gitea/gitea#38855): completed runs, with their jobs, logs and
# artifacts, are deleted by the cleanup_action_runs cron after RUN_RETENTION_DAYS. Upstream's default
# (400) is kept but DECLARED, so the deletion horizon is visible here rather than implicit. The spec
# repo states no CI-history retention requirement; 0 would mean keep forever.
- name: GITEA__actions__RUN_RETENTION_DAYS
value: "400"
# Gitea 28 (go-gitea/gitea#39426) puts webhooks, the OAuth2 avatar fetch and OpenID 2.0 behind an
# egress policy: in the default lax mode, PUBLIC targets are allowed and private/loopback/CGNAT
# targets need an IP or CIDR entry in [security] ALLOWED_HOST_LIST (a hostname entry is NOT enough
# for a private address). DECLARED lax with an EMPTY allow list, deliberately:
# * the Keycloak OIDC login source is NOT gated by it. In 28.3.0 the openidConnect provider is
# goth's, built with its default HTTP client (services/auth/source/oauth2/providers_openid.go);
# egress.NewSecurityPolicy is used only by routers/web/auth/oauth.go (avatar fetch, and
# [oauth2_client] UPDATE_AVATAR is off here) and modules/auth/openid (OpenID 2.0, unused).
# So keycloak.<domain> resolving to the private Gateway address in-cluster does not matter.
# * this instance has no webhooks and no mirrors (measured 2026-11-07), and 1.27 already refused
# private webhook targets by default, so nothing that worked before is newly denied.
# Adding an entry here would only WIDEN the webhook SSRF surface. Add the narrowest CIDR (e.g. the
# Gateway address /32) only if a real private webhook target appears.
- name: GITEA__security__EGRESS_MODE
value: lax
# Gitea 28 (go-gitea/gitea#39400) disables LOCAL self-registration unless DISABLE_REGISTRATION is
# explicitly false; 1.27 defaulted it open. Declared true: accounts arrive via Keycloak only.
# OAuth2 auto-registration below is unaffected — routers/web/auth/oauth.go gates it on
# ALLOW_ONLY_INTERNAL_REGISTRATION and ENABLE_AUTO_REGISTRATION, not on DISABLE_REGISTRATION.
- name: GITEA__service__DISABLE_REGISTRATION
value: "true"
- name: GITEA_ADMIN_USERNAME # break-glass local admin creds (consumed by the postStart hook)
valueFrom:
secretKeyRef:
name: gitea-bootstrap
key: admin-username
- name: GITEA_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
name: gitea-bootstrap
key: admin-password
# --- Keycloak native OIDC SSO (auth.mode oidc; docs/15 "Native OIDC"; R-IDP-02/06). The confidential
# `gitea` realm client's secret is mirrored into this namespace's gitea-bootstrap Secret (the same
# value SOPS-held in keycloak-realm-clients); the postStart hook registers it as the "keycloak"
# OAuth2 login source. The LOCAL admin above stays as the break-glass path (docs/15). ---
- name: GITEA_OIDC_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: gitea-bootstrap
key: oidc-client-secret
# app.ini [oauth2_client]: auto-provision a Gitea account on first Keycloak login and link it to an
# existing local account by verified email, so SSO users don't need a pre-created local user.
- name: GITEA__oauth2_client__ENABLE_AUTO_REGISTRATION
value: "true"
- name: GITEA__oauth2_client__ACCOUNT_LINKING
value: auto
- name: GITEA__oauth2_client__USERNAME
value: preferred_username
lifecycle:
postStart:
exec:
command:
- /bin/sh
- -c
- |
# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint
# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first
# boot). Waits for the DB, then creates the admin only if it does not exist.
set -u
export HOME=/data/git
i=0
until su-exec git gitea admin user list >/dev/null 2>&1; do
i=$((i + 1))
[ "$i" -ge 150 ] && { echo "[gitea-bootstrap] DB not ready after wait; skipping" >&2; exit 0; }
sleep 2
done
if su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx "$GITEA_ADMIN_USERNAME"; then
echo "[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present"
else
n=0
until su-exec git gitea admin user create \
--username "$GITEA_ADMIN_USERNAME" \
--password "$GITEA_ADMIN_PASSWORD" \
--email "[email protected]" \
--admin --must-change-password=false; do
n=$((n + 1))
[ "$n" -ge 10 ] && { echo "[gitea-bootstrap] admin create failed after retries" >&2; break; }
sleep 3
done
echo "[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'"
fi
# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---
# Registers the confidential `gitea` client as the OAuth2 login source "keycloak" against the
# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin
# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled
# secret never crashes the pod.
#
# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was
# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.
# Now it (1) waits — bounded, ~90s — for the Keycloak discovery doc to serve HTTP 200,
# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually
# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker
# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays
# usable). The verify-then-retry loop is the real guarantee against "exit 0 but nothing stuck".
GITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
gitea_oidc_fail() {
echo "[gitea-oidc] FAILED to register 'keycloak' login source: $1" >&2
{ echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true
}
# NOTE the `$$` escape below — it is load-bearing. apps-wave-0 declares postBuild.
# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE
# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not
# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z "" ]` — ALWAYS TRUE
# — making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on
# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was
# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.
# Flux STRICT substitution cannot catch this — it EXEMPTS default-operator forms by design.
# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.
# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping — which is why the
# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.
# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.
if [ -z "$${GITEA_OIDC_CLIENT_SECRET:-}" ]; then
echo "[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)"
else
# (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).
d=0; disc_ok=0
while [ "$d" -lt 45 ]; do
if [ "$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 "$GITEA_DISCOVERY_URL" 2>/dev/null)" = "200" ]; then
disc_ok=1; break
fi
d=$((d + 1)); sleep 2
done
[ "$disc_ok" = 1 ] || gitea_oidc_fail "Keycloak discovery endpoint not reachable (HTTP 200) within timeout"
# (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.
r=0; reg_ok=0
while [ "$r" -lt 10 ]; do
oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2=="keycloak"{print $1; exit}')
if [ -n "$oidc_id" ]; then
su-exec git gitea admin auth update-oauth --id "$oidc_id" \
--name keycloak --provider openidConnect --key gitea \
--secret "$GITEA_OIDC_CLIENT_SECRET" \
--auto-discover-url "$GITEA_DISCOVERY_URL" \
--scopes "openid email profile" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true
else
su-exec git gitea admin auth add-oauth \
--name keycloak --provider openidConnect --key gitea \
--secret "$GITEA_OIDC_CLIENT_SECRET" \
--auto-discover-url "$GITEA_DISCOVERY_URL" \
--scopes "openid email profile" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true
fi
# VERIFY: the "keycloak" source must actually be listed now (guards the exit-0-but-nothing-persisted race).
if su-exec git gitea admin auth list 2>/dev/null | awk '$2=="keycloak"{f=1} END{exit !f}'; then
reg_ok=1; break
fi
r=$((r + 1)); sleep 3
done
if [ "$reg_ok" = 1 ]; then
echo "[gitea-oidc] login source 'keycloak' registered and verified"
else
gitea_oidc_fail "add/update-oauth ran but the 'keycloak' source did not persist after retries"
fi
fi
# A postStart hook MUST exit 0 — a non-zero exit makes the kubelet KILL the container
# (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but
# it is made EXPLICIT so the guarantee does not depend on whatever statement happens to
# be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,
# the moment its OIDC guard was fixed and the block actually ran for the first time.
exit 0
exposure: public # edge-forwarded git host at gitea.${BASE_DOMAIN} (docs/09; R-DNS/R-NET)
egress: internet # normal WAN egress (docs/03; R-NET-05)
storage:
- name: data
size: 10Gi
class: longhorn # repos + SQLite DB on 3-replica Longhorn; irreplaceable (docs/04; R-STATE-02)
mountPath: /data # Gitea keeps ALL state (repos, DB, app.ini, INTERNAL_TOKEN) directly under /data;
# the default /data/<name> would strand real state on the ephemeral rootfs (R-STATE-01)
dependencies: [] # default-deny: reaches only DNS; git clients reach it north-south via the Gateway (R-NET-01/02)
ingressRoutes:
- host: gitea.${BASE_DOMAIN}
port: 3000 # Gitea HTTP web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
# THIS IS WHAT KEEPS PROD'S GITOPS OUT OF A DEADLOCK — it is not an incidental allowance. Remove it
# and prod cannot reconcile AT ALL, in a way that is very hard to diagnose from cold.
#
# Prod's Flux source is THIS SERVICE, not a public hostname:
# clusters/prod/kustomization.yaml -> GitRepository.spec.url =
# http://gitea.gitea.svc.cluster.local:3000/Home-Network/home-network-k3s-impl.git
# chosen deliberately so prod's GitOps depends on NEITHER public DNS, Cloudflare, prod's own
# Gateway, NOR ${BASE_DOMAIN} — a future domain change cannot sever prod from its own source. That
# file documents the other half of this pair at length; the two must be read together.
#
# The only other ingress this entry renders is gitea-allow-gateway-ingress, which admits
# `fromEntities: [ingress]` — the shared Gateway's Envoy proxy — and nothing else. Flux's
# source-controller dials the ClusterIP DIRECTLY, so under the namespace default-deny (R-NET-01) its
# clone was DROPPED (`wget` from source-controller timed out; srcReady=False, "unable to list
# remote"). The deadlock that made this urgent: the HTTPRoute that would have served the new public
# name after the 2026-09-13 apex flip lives in a DOWNSTREAM tier, which cannot apply while the
# source is down — source down -> routes never rename -> source stays down. It was broken on
# 2026-09-13 with a live `kubectl apply` of a hand-written gitea-allow-flux-ingress, i.e. with a
# cluster object that existed OUTSIDE git. This declaration renders that object, so one
# `kubectl delete` or one cluster rebuild can no longer put prod back in the deadlock.
inClusterClients:
# `name` is pinned to `flux` (rather than defaulting to the namespace) so the rendered policy name
# equals the live object's, `gitea-allow-flux-ingress`, and Flux ADOPTS it in place: no
# near-duplicate policy beside it, and no window in which prod's own source has no allow.
- name: flux
namespace: flux-system # matched by NAMESPACE alone — the flux controllers' pod labels belong
# to upstream's manifests, not to this platform's contract
port: 3000 # ONLY the Gitea HTTP port — the same single port the Gateway allow opens
# The PM agent service (platform/pm-agent/). Its dispatcher reads issues and posts comments
# over this port, and every worker Job clones, pushes and opens pull requests over it. Like
# Flux's source-controller above, it dials the ClusterIP DIRECTLY — deliberately, so the
# agent's path to the repository depends on neither public DNS, nor Cloudflare, nor the
# shared Gateway, nor ${BASE_DOMAIN}. And exactly as for Flux, that means this namespace's
# own default-deny (R-NET-01) DROPS the connection unless it is declared here: without this
# entry the dispatcher's egress allow is satisfied and the packet is still discarded at the
# Gitea end, which presents as an authentication timeout rather than as a policy denial.
# That is the 2026-09-13 lesson documented at length above, applied before it is relearned.
- name: pm-agent
namespace: pm-agent # matched by NAMESPACE alone, so the dispatcher Deployment and every
# ephemeral worker Job are covered by the one declaration
port: 3000 # ONLY the Gitea HTTP port
auth:
mode: oidc # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC" set; R-IDP-02)
groups:
- platform-admins # access scoped to an authorized group (R-IDP-06)
# WIRED (native OIDC): the Keycloak `gitea` confidential client (platform/keycloak/keycloak-realm-
# import.yaml + live kcadm) is registered as Gitea's "keycloak" OAuth2 login source by the postStart
# hook, using the client secret mirrored into the gitea-bootstrap Secret (oidc-client-secret; same
# value SOPS-held in keycloak-realm-clients). platform-admins (groups claim) -> Gitea admin. Gitea
# KEEPS a LOCAL admin as the break-glass path so an IdP outage never locks the operator out of
# recovery tools (docs/15 "Availability caveat"). No secret is embedded here.
update:
channel: stable # R-UPD-01
range: ">=28.0.0 <29.0.0" # Gitea dropped the "1." prefix at 28.0.0 (successor of 1.27); stay on 28.x
cadence: "0 4 * * *"
# R-UPD-03. INERT FOR THIS SERVICE, and it must stay that way: this flag renders only an annotation
# on a Flux ImagePolicy, and apps/ is rendered --no-image-automation (ci/render-golden.sh), so no
# ImagePolicy exists and the Deployment (strategy Recreate) is never auto-reverted. That matters
# because a Gitea upgrade (1.27 -> 28 included) MIGRATES THE SQLITE DB ONE-WAY: an image revert onto
# a migrated DB is not a rollback. Roll Gitea back by RESTORING THE DATA PVC from its pre-upgrade backup.
autoRollbackOnFailedHealth: true
observability:
metrics: true
slis:
- up # git host down is a critical alert (docs/10; R-OBS-01)
backup:
policy: daily-offcluster # DR-critical: repos/DB backed up off-cluster + mirrored (docs/13, docs/04; R-STATE-03)