docs/poststart-incident
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
4c4f5355d9 |
fix(sso): postStart hooks must exit 0 — nextcloud crashlooped the moment OIDC ran
LIVE REGRESSION, caught on the reconcile after merging #5. nextcloud went
CrashLoopBackOff with FailedPostStartHook, 7 restarts, and apps-wave-1/2 stalled.
ROOT CAUSE — a latent bug that could never fire until now. The final statement of
nextcloud's postStart script was:
[ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote ... after retries"
On the SUCCESS path (p < 15) the test returns 1, and as the script's LAST command that
becomes the hook's exit status. A non-zero postStart exit makes the kubelet KILL the
container. This was harmless for as long as the OIDC guard above it was dead code that
exited 0 first (see
|
||
|
jakman85andClaude Opus 5
|
3484955903 |
fix(sso): gitea + nextcloud OIDC bootstrap was DEAD CODE — envsubst ate the guard
Both ServiceDefinitions guarded their Keycloak OIDC registration with an UNESCAPED
braced expansion:
if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then # gitea
if [ -z "${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then # nextcloud, then exit 0
apps-wave-0/1 both declare postBuild.substituteFrom: cluster-vars, and neither name is a
cluster-vars key — so Flux's envsubst substituted them away AT RECONCILE TIME, long
before the container ever ran. Verified in the LIVE cluster: the deployed Deployments
literally contain
if [ -z "" ]; then
which is always true. The entire OIDC registration block has been skipped on EVERY boot
of both apps.
WHY NOBODY NOTICED: both apps DO have the provider registered — `gitea admin auth list`
shows keycloak/OAuth2/true and `occ user_oidc:provider` shows the keycloak provider.
Those live in each app's own database, persisted from an earlier successful registration.
SSO works today and DECISIONS-LOG is not wrong. But the idempotent re-registration is
dead, so any PVC rebuild, DB restore, or fresh cluster build comes up with NO OIDC and
only the break-glass local admin. A PROD BUILD FROM THIS REPO WOULD SILENTLY HAVE NO SSO
on gitea or nextcloud.
The irony: the surrounding code was hardened twice — see its own comments about a "silent
first-boot RACE", the verify-then-retry loops, the marker files. All of that hardening was
unreachable, because the guard above it always evaluated true.
Not a regression from the recent R-ENV-02 tokenization — introduced 2026-08-27 (
|
||
|
jakman85andClaude Opus 5
|
1157c1b071 |
fix(gitea-actions): wait for dind to ACCEPT connections, not just write certs
Re-authors the fix stranded on origin/fix/act-runner-dind-readiness (
|
||
|
jakman85andClaude Opus 4.8
|
b2752aec48 |
feat(auth): gate jackett/qbittorrent/radarr/sonarr UIs with oauth2-proxy sidecar (rollout)
Roll the verified Option-B per-app oauth2-proxy sidecar out to the remaining four forward-auth apps after the pihole canary was confirmed fail-closed (unauthenticated -> 302 to Keycloak, pod 2/2, clean sidecar OIDC discovery). Each app's Gateway route now terminates at its oauth2-proxy sidecar (:4180, --allowed-group=media-users, --upstream=http://127.0.0.1:<uiport>); the gateway-ingress CiliumNetworkPolicy allows the Gateway to reach ONLY :4180, so the bare UI is never served un-authenticated (fail-closed; R-IDP-02/06). The apps' own app/API ports stay in-Service for EAST-WEST callers (arr -> qbittorrent:8080 / jackett:9117) via the existing reciprocal-ingress allows. - keycloak: confidential jackett/qbittorrent/radarr/sonarr clients (realm-import + realm-clients mirror), provisioned live via kcadm (groups membership mapper). - secrets: four SOPS/age <app>-oauth2-proxy-credentials, wired into apps/wave-0 and apps/wave-1 by the renderer's sidecar-creds wave rule. - ci: remove the temporary canary deferral (all five goldens now reproduce). Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
68690e5019 |
fix(net): allow SSO apps egress to keycloak backend (Cilium 1.20 CVE-2025-30162)
App SSO broke cluster-wide after the Cilium 1.16.5->1.20.1 upgrade: the CVE-2025-30162 fix (in
>=1.16.8/1.17.2) makes the shared Gateway's L7 proxy re-evaluate a pod's egress policy against
Keycloak's REAL backend cluster identity (enforce_policy_on_l7lb) when it hairpins to keycloak.home.
'toEntities: world' matches the Gateway VIP at L3/L4 but not the resolved keycloak identity at L7, so
OIDC discovery/token 403'd ('Access denied') for every app with a NetworkPolicy (gitea/nextcloud/
semaphore/headlamp/crafty...); Grafana survived only because monitoring has no NetworkPolicy.
Fix: explicit toEndpoints egress allow to the keycloak backend for SSO apps.
- catalog/render.py: emit it for auth.mode in {oidc,forward-auth}; re-rendered 8 app goldens.
- platform/headlamp + platform/crafty: hand-written policies get the same allow.
Split-horizon design unchanged. Verified live: headlamp-ns -> keycloak discovery 403 -> 200.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
fe0ee698c4 |
fix(apps): reconcile goldens to one canonical render + enforce render==golden gate
The committed apps/ goldens were a MIX of two render flag sets: 8 services were rendered with `--no-image-automation` (no image CRs) while gitea and nextcloud carried stale ImageRepository/ImagePolicy CRs from a render WITHOUT the flag. Neither single command reproduced all goldens, and the gate did not catch it. Part A — canonical render: python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation `--no-image-automation` is canonical: image automation is authored-but-DISABLED (DECISIONS-LOG; image-reflector/-automation controllers not installed), and every app was onboarded via this flag. Re-render drops the ImageRepository + ImagePolicy objects from gitea + nextcloud only (52 line deletions, purely the trailing image-automation blocks). No functional content changed — the nextcloud postStart hardening and gitea OIDC hook are untouched. Part B — enforcement: ci/render-golden.sh renders every onboarded catalog service with the canonical command into a temp dir and FAILS on ANY diff vs committed apps/ (missing golden or content drift), across all 10 services (not just the 3 L1 spot-checks). Not-onboarded catalog entries (keycloak, vm-demo) are excluded per DESIGN-waves. Wired as a required no-cluster stage in ci/gate.sh, ci/Makefile, and both the live (.gitea/workflows/gate.yml) and dormant (merge-gate.yaml) pipelines. Proof: tampering a golden makes the stage exit 1 (blocks merge); reverting restores exit 0. `make -C ci gate` green in the devcontainer. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
7f86e1099b |
fix(nextcloud-sso): register post-logout redirect URI + harden admin promotion
Two SSO follow-ups the operator hit after Nextcloud OIDC login worked. 1. Sign-out "redirect failed". On logout user_oidc redirects the browser to Keycloak's end_session_endpoint with post_logout_redirect_uri=<Nextcloud base '/'> and client_id=nextcloud but NO id_token_hint (provider sendIdTokenHint=0). Keycloak then validates that redirect against the client's post.logout.redirect.uris attribute, which was empty -> "Invalid redirect uri" -> the operator's "redirect failed". Register https://nextcloud.home.keithjackson.dev/* on the nextcloud client (attributes.post.logout.redirect.uris). Applied live via kcadm and captured here so git stays authoritative (a RealmImport no-ops on the existing realm). 2. postStart step (d) mirrors platform-admins -> built-in admin group, but the previous single best-effort pass ran occ group:adduser once with output swallowed and no retry. At early boot (container Started ~9s after create) that call can transiently fail (DB / maintenance race), silently dropping the promotion until the next restart. Harden step (d) with a verify-and-retry loop (like step (c)): re-read group:list and re-add any platform-admins member still missing from admin until verified (bounded ~30s), no-op for existing members. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
7f92adfac6 |
fix(nextcloud): allow SSRF to private IdP + promote platform-admins to admin
Task 1 — browser SSO "Could not reach OpenID Connect provider" (confirmed root cause, not guessed). keycloak.home.keithjackson.dev resolves split-horizon to the PRIVATE Cilium gateway VIP (192.168.1.53). Nextcloud's SSRF guard refuses its own server-side HTTP requests to RFC1918 addresses, so user_oidc's discovery/JWKS/token fetch AT LOGIN throws OCP\Http\Client\LocalServerException "violates local access rules" -> surfaced to the user as "Could not reach the provider". Proven from the live pod: nextcloud.log carries the exact LocalServerException from DiscoveryService::obtainDiscovery with Guzzle option allow_local_address:false; a Nextcloud-context IClientService->get() of the discovery URL returns LocalServerException with the flag false and HTTP 200 with it true (A/B toggled). The postStart (b) curl check passed regardless because it uses system libcurl, not Nextcloud's guard — which is why the provider registered yet login still failed. Fix (reproducible, survives pod rolls): the postStart hook now runs `occ config:system:set allow_local_remote_servers --value=true --type=boolean` right after the install/maintenance wait, on every start. Task 2 — promote Keycloak `platform-admins` members to Nextcloud admins (#5). user_oidc 8.11 has NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a login WHITELIST only), so the hook mirrors the provisioned `platform-admins` membership into the built-in `admin` group idempotently (occ group:list --output=json parsed via php argv; occ group:adduser admin per member). A user is promotable after their first OIDC login provisions them + the next pod start (or one manual `occ group:adduser admin <uid>`). The LOCAL admin stays as break-glass (docs/15). apps/wave-1/nextcloud.yaml re-rendered from catalog (single-file render; byte-diff confined to the embedded hook). Rendered hook passes `sh -n`; parse step validated in-pod. `make -C ci gate` green (lint/l0/l1; conformance + spec-immutability skip: no cluster / spec repo unmounted). Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
d34c639269 |
fix(oidc): make gitea + nextcloud OIDC login-source registration persist
The Keycloak OIDC postStart registration for gitea and nextcloud ran but never persisted, leaving `gitea admin auth list` empty and nextcloud's user_oidc with "No providers configured" despite the apps installing/ enabling and the client secret being present. Two distinct root causes, confirmed by running the registration manually against the live warm pods: - nextcloud: a QUOTING BUG, not a race. The OCC() wrapper runs `su -c "php occ $*"`, which word-split `--scope="openid email profile"` back into three tokens, so occ saw `email profile` as extra positional args -> "Too many arguments to user_oidc:provider" -> exit 1, swallowed by `|| echo failed`. Failed on EVERY boot. Fixed by running the provider call as ONE `su -c` with the scope single-quoted so www-data's shell keeps it a single argument; keep the OCC() wrapper only for single-token subcommands. - gitea: a first-boot RACE. The direct `admin auth add-oauth` persists once the app/network is warm (verified live), but fired too early at postStart and its failure was swallowed (`|| echo ... >&2`, exit 0). Hardening applied to both catalog sources (rendered into apps/wave-*): - wait (bounded ~90s) for the Keycloak discovery doc to return HTTP 200; - nextcloud also waits for `installed: true` AND `maintenance: false`; - register idempotently (create-or-update) then VERIFY the source/provider actually persisted, retrying the whole cycle (guards exit-0-but-nothing- stuck); - on give-up, log LOUDLY to stderr AND a marker file on the persistent volume (/data, /var/www/html) instead of swallowing — next failure is visible; - never crashloop: missing-secret guard retained, break-glass local admin stays usable. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
f163026954 |
feat(nextcloud,semaphore): wire native Keycloak OIDC SSO (break-glass admins retained)
Extends the SSO work to the two remaining `auth: oidc` apps still on local-admin
logins, each as a native confidential client of the Keycloak `homelab` realm with
its local admin kept as break-glass (docs/15 availability caveat; R-IDP-02/06).
Realm (source of truth): add the `nextcloud` and `semaphore` confidential clients
to platform/keycloak/keycloak-realm-import.yaml (modeled on grafana/gitea) + their
NEXTCLOUD_CLIENT_SECRET / SEMAPHORE_CLIENT_SECRET placeholders. As before, a
KeycloakRealmImport no-ops on an already-existing realm, so both live clients were
ALSO created via kcadm (nextcloud e95049ed-ca10-4085-97d9-4bbef495fe61, semaphore
1835bfe9-b125-4026-8f69-33557519cf4a); the YAML keeps git authoritative.
Secrets (SOPS/age): add nextcloud-client-secret + semaphore-client-secret to
keycloak-realm-clients.sops.yaml (keycloak ns). Mirror the nextcloud secret as
oidc-client-secret into nextcloud-bootstrap (nextcloud ns). For semaphore, the
whole SEMAPHORE_OIDC_PROVIDERS JSON (which embeds the client secret) is held
encrypted as oidc-providers-json in semaphore-bootstrap (semaphore ns). All values
verified == the live clients; existing keys untouched; no plaintext in git.
App config (catalog source + re-render):
- nextcloud: add NEXTCLOUD_OIDC_CLIENT_SECRET env + an idempotent postStart hook
that installs/enables user_oidc and runs `occ user_oidc:provider keycloak`
(create-or-update; secret via --clientsecret-env so it never hits argv; occ as
www-data; group provisioning on the `groups` claim). Syntax verified against the
running user_oidc 8.11.0 on Nextcloud 31.0.14; su-to-www-data env passthrough
verified. Missing-secret/not-installed guards prevent crashloops.
- semaphore: add SEMAPHORE_OIDC_PROVIDERS env sourced from the encrypted
oidc-providers-json (provider_url=realm issuer/auto-discovery, redirect
/api/auth/oidc/keycloak/redirect). Mechanism verified against semaphore v2.19.10
source (util ConfigType oidc_providers env tag + OidcProvider struct + router
redirect path); no kubelet service-link env collision.
apps/wave-1/{nextcloud,semaphore}.yaml re-rendered via
`render.py --no-image-automation` (diffs are env/lifecycle-only).
Verified: realm import Done=True/HasErrors=False and platform-config
Ready=True/Healthy=True (unchanged); edited realm YAML passes server dry-run; live
homelab clients now include gitea/nextcloud/semaphore; static gate (lint/l0/l1)
green. Browser SSO login for each still needs a human check.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
36a1a4e1f9 |
refactor(apps): split monolithic apps Kustomization into dependency waves
Replaces the single `apps` Flux Kustomization with dependency-ordered waves
(apps-wave-0..2) so a crashlooping workload isolates to its wave + downstream
instead of stalling the entire app tier. Parallel within a wave, ordered across.
- catalog/render.py: compute waves by longest dependency-path depth over the
ServiceDefinition `dependencies:` edges (a `<svc>-db` is a leaf, so its
consumer lands strictly higher — the "DB below consumer" rule falls out of
longest-path, no special case). `--emit-waves` generates the per-wave
kustomizations + Flux CRs; `--wave-layout` places renders into the right dir.
wave 0: pihole gitea jackett plex qbittorrent nextcloud-db semaphore-db
wave 1: sonarr radarr nextcloud semaphore gitea-actions
wave 2: seerr
- apps/*.yaml -> apps/wave-{0,1,2}/ (byte-identical moves; SOPS files intact).
- clusters/base: apps.yaml removed; apps-wave-{0,1,2}.yaml added (dependsOn
chain: platform-config -> wave-0 -> wave-1 -> wave-2, prune+wait each).
- clusters/dev: one SOPS decryption patch per wave (load-bearing).
- DESIGN-waves.md: wave table, topology, cutover runbook.
Cutover: preceded by disabling prune on `apps` (
|