docs/node2-memory-refine
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
4c4f5355d9 |
fix(sso): postStart hooks must exit 0 — nextcloud crashlooped the moment OIDC ran
LIVE REGRESSION, caught on the reconcile after merging #5. nextcloud went
CrashLoopBackOff with FailedPostStartHook, 7 restarts, and apps-wave-1/2 stalled.
ROOT CAUSE — a latent bug that could never fire until now. The final statement of
nextcloud's postStart script was:
[ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote ... after retries"
On the SUCCESS path (p < 15) the test returns 1, and as the script's LAST command that
becomes the hook's exit status. A non-zero postStart exit makes the kubelet KILL the
container. This was harmless for as long as the OIDC guard above it was dead code that
exited 0 first (see
|
||
|
jakman85andClaude Opus 5
|
3484955903 |
fix(sso): gitea + nextcloud OIDC bootstrap was DEAD CODE — envsubst ate the guard
Both ServiceDefinitions guarded their Keycloak OIDC registration with an UNESCAPED
braced expansion:
if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then # gitea
if [ -z "${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then # nextcloud, then exit 0
apps-wave-0/1 both declare postBuild.substituteFrom: cluster-vars, and neither name is a
cluster-vars key — so Flux's envsubst substituted them away AT RECONCILE TIME, long
before the container ever ran. Verified in the LIVE cluster: the deployed Deployments
literally contain
if [ -z "" ]; then
which is always true. The entire OIDC registration block has been skipped on EVERY boot
of both apps.
WHY NOBODY NOTICED: both apps DO have the provider registered — `gitea admin auth list`
shows keycloak/OAuth2/true and `occ user_oidc:provider` shows the keycloak provider.
Those live in each app's own database, persisted from an earlier successful registration.
SSO works today and DECISIONS-LOG is not wrong. But the idempotent re-registration is
dead, so any PVC rebuild, DB restore, or fresh cluster build comes up with NO OIDC and
only the break-glass local admin. A PROD BUILD FROM THIS REPO WOULD SILENTLY HAVE NO SSO
on gitea or nextcloud.
The irony: the surrounding code was hardened twice — see its own comments about a "silent
first-boot RACE", the verify-then-retry loops, the marker files. All of that hardening was
unreachable, because the guard above it always evaluated true.
Not a regression from the recent R-ENV-02 tokenization — introduced 2026-08-27 (
|
||
|
jakman85andClaude Opus 4.8
|
d34c639269 |
fix(oidc): make gitea + nextcloud OIDC login-source registration persist
The Keycloak OIDC postStart registration for gitea and nextcloud ran but never persisted, leaving `gitea admin auth list` empty and nextcloud's user_oidc with "No providers configured" despite the apps installing/ enabling and the client secret being present. Two distinct root causes, confirmed by running the registration manually against the live warm pods: - nextcloud: a QUOTING BUG, not a race. The OCC() wrapper runs `su -c "php occ $*"`, which word-split `--scope="openid email profile"` back into three tokens, so occ saw `email profile` as extra positional args -> "Too many arguments to user_oidc:provider" -> exit 1, swallowed by `|| echo failed`. Failed on EVERY boot. Fixed by running the provider call as ONE `su -c` with the scope single-quoted so www-data's shell keeps it a single argument; keep the OCC() wrapper only for single-token subcommands. - gitea: a first-boot RACE. The direct `admin auth add-oauth` persists once the app/network is warm (verified live), but fired too early at postStart and its failure was swallowed (`|| echo ... >&2`, exit 0). Hardening applied to both catalog sources (rendered into apps/wave-*): - wait (bounded ~90s) for the Keycloak discovery doc to return HTTP 200; - nextcloud also waits for `installed: true` AND `maintenance: false`; - register idempotently (create-or-update) then VERIFY the source/provider actually persisted, retrying the whole cycle (guards exit-0-but-nothing- stuck); - on give-up, log LOUDLY to stderr AND a marker file on the persistent volume (/data, /var/www/html) instead of swallowing — next failure is visible; - never crashloop: missing-secret guard retained, break-glass local admin stays usable. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
87388712f0 |
feat(gitea): wire native Keycloak OIDC SSO (break-glass admin retained)
Wires the bootstrapping-critical Gitea git host to the Keycloak `homelab` realm as a native confidential OIDC client, moving it off the local-admin-only login while keeping the local admin as the break-glass path (docs/15 availability caveat; R-IDP-02/06). Realm (source of truth): add the `gitea` confidential client to platform/keycloak/keycloak-realm-import.yaml (modeled on grafana) + its GITEA_CLIENT_SECRET placeholder sourced from the keycloak-realm-clients Secret. NOTE: a KeycloakRealmImport no-ops on an already-existing realm, so the live client was ALSO created via kcadm (id 70a6ca60-317a-42ce-8331-a60e380838ee); the YAML keeps git authoritative for a rebuild. Secrets (SOPS/age, never plaintext): add gitea-client-secret to platform/keycloak/keycloak-realm-clients.sops.yaml (keycloak ns) and mirror the same value as oidc-client-secret into apps/wave-0/gitea-bootstrap-secret.sops.yaml (gitea ns) so the app can read it in-namespace. Both == the live client secret. App config (catalog source + re-render): catalog/gitea.yaml gains the GITEA_OIDC_CLIENT_SECRET env (from the mirrored Secret), [oauth2_client] auto-registration/account-linking app.ini env, and an idempotent postStart step that registers the "keycloak" OAuth2 login source (auto-discovery; groups claim; platform-admins -> Gitea admin). apps/wave-0/gitea.yaml re-rendered via `render.py --no-image-automation`. Verified: realm import Done=True/HasErrors=False and platform-config Ready=True/Healthy=True (unchanged); edited realm YAML passes server dry-run; static gate (lint/l0/l1) green. Browser SSO login still needs a human check. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
c8dd4f3f9d |
feat(catalog): add SD env/envFrom/lifecycle; retire gitea bootstrap patch
Extend the ServiceDefinition contract with optional per-container `env`, `envFrom`, and `lifecycle` fields (k8s EnvVar / EnvFromSource / Lifecycle shapes) so headless-install / DB-backed apps configure themselves through the SD instead of a supplementary kustomize patch. - schema: bump 1.1.0 -> 1.2.0; add optional env/envFrom/lifecycle to the container kind (literal + secretKeyRef/configMapKeyRef; secretRef/ configMapRef; postStart/preStop exec via $defs/execHandler). Existing fields unchanged; all fields optional -> absent renders byte-identical. - render.py: emit env/envFrom/lifecycle verbatim onto the Deployment app container when present; --selftest coverage for the new fields. - catalog/gitea.yaml: fold the headless-install env (INSTALL_LOCK, fixed SECRET_KEY via secretKeyRef, DOMAIN/ROOT_URL, actions, break-glass admin creds) + the idempotent postStart break-glass admin bootstrap inline. - apps/gitea.yaml: re-rendered with env/lifecycle inline (effective pod spec provably equivalent to base+patch). - DELETE apps/gitea-bootstrap-patch.yaml + drop it from apps/kustomization patches. gitea-bootstrap-secret.sops.yaml kept (still sourced by the env). Proves the renderer feature by retiring the known TODO; unblocks env + DB onboarding for nextcloud/semaphore/unifi/minecraft. All other rendered apps byte-identical. Gates: selftest PASS, l0 17/17 + schema, l1 goldens, lint. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
73c4475d34 |
fix(gitea): mount PVC at /data + headless-install bootstrap (Phase 6)
Two defects blocked in-cluster Gitea from becoming the Flux source: 1. Storage mountPath: the renderer hard-coded /data/<name>, so Gitea's PVC mounted at /data/data while Gitea writes all state to /data — real repos/DB landed on the ephemeral rootfs. Add an OPTIONAL `mountPath` to the storage schema + render.py (default stays /data/<name>, back-compat). catalog/gitea.yaml sets data.mountPath: /data. Only gitea.yaml's rendered mount changes; every other app renders byte-identical. 2. Headless install: the instance was un-installed (INSTALL_LOCK=false, ephemeral SECRET_KEY, no admin, API 404). A Gitea-only supplementary manifest (apps/gitea-bootstrap-patch.yaml) patches the rendered Deployment with GITEA__security__INSTALL_LOCK=true, a FIXED SECRET_KEY from a persistent SOPS secret (apps/gitea-bootstrap-secret.sops.yaml), ROOT_URL/DOMAIN, and a postStart hook that idempotently creates a break-glass local admin. State persists on the now-correct /data Longhorn PVC. The env/lifecycle wiring is a supplementary kustomize patch because the ServiceDefinition schema/renderer does not yet express env/envFrom/lifecycle — noted as a renderer follow-up (TODO in the patch). Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
b76fb7bea7 |
feat(apps): onboard Gitea via ServiceDefinition renderer
Author catalog/gitea.yaml (kind: container, public, DR-critical git host per docs/13 row 12), render to apps/gitea.yaml (--no-image-automation), and wire it into apps/kustomization.yaml. - image gitea/gitea:1.24.3, web port 3000 at gitea.home.keithjackson.dev - 10Gi Longhorn PVC for /data (repos + SQLite DB), daily-offcluster backup - exposure public, egress internet, auth oidc (platform-admins), metrics on - Traces: R-STATE-02/03, R-NET-01/02, R-DNS/R-TLS, R-IDP-02/06, R-OBS-01, R-UPD-01/03 Co-Authored-By: Claude Opus 4.8 <[email protected]> |