0 Fork 0

Files

T

jakman85andClaude Opus 5 3484955903 fix(sso): gitea + nextcloud OIDC bootstrap was DEAD CODE — envsubst ate the guard

Both ServiceDefinitions guarded their Keycloak OIDC registration with an UNESCAPED
braced expansion:

    if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then      # gitea
    if [ -z "${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then  # nextcloud, then exit 0

apps-wave-0/1 both declare postBuild.substituteFrom: cluster-vars, and neither name is a
cluster-vars key — so Flux's envsubst substituted them away AT RECONCILE TIME, long
before the container ever ran. Verified in the LIVE cluster: the deployed Deployments
literally contain

    if [ -z "" ]; then

which is always true. The entire OIDC registration block has been skipped on EVERY boot
of both apps.

WHY NOBODY NOTICED: both apps DO have the provider registered — `gitea admin auth list`
shows keycloak/OAuth2/true and `occ user_oidc:provider` shows the keycloak provider.
Those live in each app's own database, persisted from an earlier successful registration.
SSO works today and DECISIONS-LOG is not wrong. But the idempotent re-registration is
dead, so any PVC rebuild, DB restore, or fresh cluster build comes up with NO OIDC and
only the break-glass local admin. A PROD BUILD FROM THIS REPO WOULD SILENTLY HAVE NO SSO
on gitea or nextcloud.

The irony: the surrounding code was hardened twice — see its own comments about a "silent
first-boot RACE", the verify-then-retry loops, the marker files. All of that hardening was
unreachable, because the guard above it always evaluated true.

Not a regression from the recent R-ENV-02 tokenization — introduced 2026-08-27 (8738871).

FIXED AT THE RIGHT SEAM: apps/wave-0/gitea.yaml and apps/wave-1/nextcloud.yaml are
RENDER-GENERATED, so the fix is in catalog/*.yaml and the goldens are regenerated.
Escaped to $${...}, the convention documented in clusters/base/apps-vms.yaml. Post-envsubst
the container now receives the intended `${VAR:-}` instead of `""`.

STRICT MODE WOULD NEVER HAVE CAUGHT THIS. Flux deliberately exempts default-providing
operators (-, :-, +, :+, =, :=, ?, :?) from StrictPostBuildSubstitutions, because they
"intentionally handle the unset case". So a targeted check was required.

MADE UN-REPEATABLE IN THE RENDERER: catalog/render.py gains a substitution-safety section
wired into emit(), so every render path refuses to emit a manifest containing a braced
expansion that is not exactly ${KNOWN_KEY} for a key read LIVE from cluster-vars (no
hardcoded copy to drift; unreadable => empty set => fail-closed). Dollar-runs are counted
properly ($$ is a literal $ to envsubst, so odd runs are flagged and even runs are not),
and indexed forms get their own hard-error message. Nine new --selftest assertions,
including the exact regression. Proven to bite: rendering the pre-fix catalog files from
HEAD now exits 1 with the precise diagnosis.

Sweep: the complete substituted surface was enumerated from every Kustomization's path.
catalog/ contains exactly TWO braced expansions and both were these bugs; every other
braced expansion on a substituted path is a declared cluster-vars key (zero typos, zero
undeclared names); existing $${UOS_URL} escapes verified intact.

Verified: --selftest PASS; canonical render is byte-identical to the committed apps/ for
all 10 onboarded services; the three L1 render checks pass.

NEXT RECONCILE: both pod templates change, so gitea (wave 0) and nextcloud (wave 1) roll —
strategy: Recreate on RWO Longhorn PVCs, so expect a brief single-pod outage each. The
OIDC block then executes for the first time in these apps' deployed lifetime. Both paths
are idempotent (gitea takes auth update-oauth on the existing id; nextcloud's
user_oidc:provider is create-or-update), so the expected result is a no-op re-write plus
the verify loop logging success. If the SOPS client secret ever diverged from Keycloak's,
this is when it gets corrected instead of silently left stale. Tail both pods after merge.

Serves: R-IDP-02, R-IDP-06, R-ENV-02, R-GITOPS-02.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

2026-09-01 01:29:40 -05:00

5.0 KiB

Raw Blame History

catalog — ServiceDefinition entries (the onboarding surface)

One YAML file per service. Each file is a ServiceDefinition conforming to ../schemas/servicedefinition.schema.json. Adding or editing a service here is the only onboarding step — the platform renders compute, storage, network policy, egress, TLS, DNS, monitoring, and update policy from it (see ../schemas/renderer-contract.md). Serves R-GITOPS-02.

Source of truth for the initial catalog: spec docs/13-service-catalog.md (14-service table + SSO/egress/dependency intent).

Current examples

File	docs/13 row	exposure	egress	auth	Notes
`pihole.yaml`	6	internal	internet	forward-auth	admin UI behind oauth2-proxy; config on Longhorn (DR-critical); also platform DNS
`keycloak.yaml`	0	public	internet	native	the IdP itself (own admin login + MFA); PostgreSQL on Longhorn; DR-critical

The remaining catalog services (plex, sonarr, radarr, jackett, seerr, nextcloud, minecraft, unifi, semaphore, kasm, gitea, qbittorrent) are onboarded in later phases; they follow the same schema and inherit every guarantee automatically.

Authoring rules

Only name and source are mandatory; every other field defaults to docs/13's default posture (kind: container, exposure: internal, egress: internet, auth.mode: forward-auth, stable auto-update, DNS-only default-deny). Set a field only to override the default.
Any exposed UI must have an auth block: forward-auth (default) or oidc, each with a non-empty auth.groups; native/none are documented exceptions only.
Irreplaceable state (class: longhorn configs/DBs) must set backup.policy to an off-cluster schedule — never reacquirable (R-STATE-03). Only bulk media may be reacquirable.
Shell braces must be escaped $${...}. Anything rendered verbatim onto the workload — a lifecycle postStart script, an env value — lands under apps/wave-<N>/, and every apps-wave-<N> Flux Kustomization declares postBuild.substituteFrom: cluster-vars. Flux runs envsubst over the WHOLE manifest at reconcile time, so an unescaped ${VAR:-} meant for the shell is replaced with the empty string before the container ever starts (this silently killed gitea's and nextcloud's OIDC registration for days). Bare $VAR and $(cmd) are left alone and need no escaping; a genuine environment knob is declared in clusters/base/cluster-vars.yaml and written bare ${KNOB}. Full contract: ../clusters/base/apps-vms.yaml. render.py REFUSES to emit a manifest that breaks this rule (Flux's strict substitution mode cannot: it deliberately exempts the ${V:-} / ${V:=x} / ${V:?} default-operator forms).

Validate before commit — single command, no cluster (R-TEST-02)

check-jsonschema --schemafile schemas/servicedefinition.schema.json catalog/*.yaml

See ../schemas/README.md for install/pinning and a pure-Python fallback. This is the L0/L1 schema gate (spec docs/14); CI (.gitea/) runs it on every change.

Rendering — `render.py` (the ServiceDefinition renderer, T-0.3)

render.py compiles a ServiceDefinition into the concrete manifest set defined by ../schemas/renderer-contract.md: a Namespace (with the app-namespace/gateway-access labels), the default-deny baseline (R-NET-01), a Deployment+Service or a KubeVirt VirtualMachine + CDI DataVolume (R-PLAT-02/03), PVCs (R-STATE), one CiliumNetworkPolicy egress-allow per dependency (R-NET-02), the egress-class steering label (R-NET-05), HTTPRoute(s) on the shared Gateway (R-TLS/R-DNS), ServiceMonitor + SLI alerts (R-OBS-01), the Flux ImageRepository/ImagePolicy update plumbing (R-UPD), and the auth-mode / backup wiring hints (R-IDP/R-STATE-03).

python3 catalog/render.py catalog/pihole.yaml        # render one SD to stdout
python3 catalog/render.py --all catalog -o out/      # render every SD to out/<name>.yaml
python3 catalog/render.py --selftest                 # deterministic self-test (no cluster)

Deterministic: identical input → byte-identical output (golden-diffable). The L1 render conformance tests (conformance/tests/l1-rendering/*) diff the output against committed goldens — run offline by conformance/run.sh --layer l1 and ci/l1.sh (no cluster).
How CI/Flux consume it: CI invokes render.py in the L1 stage; a GitOps flow renders --all catalog -o <rendered-dir> and Flux reconciles the rendered directory. Design choice (renderer-contract is silent on mechanism): a render-then-consume script is the simplest reproducible option — see the header of render.py.
Dependency: PyYAML (the renderer's single dependency; the conformance harness stays stdlib-only and only shells out to this script + diffs text).

5.0 KiB Raw Blame History