The four backup-credential SOPS Secrets are SHARED BASE manifests: platform/ and apps/ are
rendered by the same Flux tiers on both clusters, and the ONLY per-environment difference in
this repo is cluster-vars substitution (R-ENV-02). So writing the R2 token into all four
(0a6a862) did not "give prod credentials" — it REPLACED DEV'S. Merging that as-is would have
handed dev's Velero and all three dev CNPG Clusters an R2 key id to sign MinIO requests with:
403 on every WAL segment, and archive_mode=on turns that into a disk-full clock on the pgdata
PVC, not a degraded backup. Dev's PROVEN DR restore (R-STATE-03) is the thing that would have
broken.
This commit splits the material. Each credential is now TWO Secret objects, both applied to
BOTH clusters, differing only in name suffix:
dev -> velero-cloud-credentials keycloak/nextcloud/semaphore-db-backup-credentials
prod -> velero-cloud-credentials-prod ...-db-backup-credentials-prod
The consumer picks one by name; that wiring is the NEXT commit, so this one changes no
rendered reference and is a no-op for both live clusters.
WHY DEV KEEPS THE UNSUFFIXED NAME. Dev is live. Renaming its Secret would change
`s3Credentials.accessKeyId.name` on three running CNPG Clusters — which projects into the
instance pods' environment and therefore ROLLS keycloak-db, nextcloud-db and semaphore-db —
and would roll the velero Deployment, all to rename an object whose contents do not change.
This repo already makes exactly this call for dev's live identifiers (EXTERNALDNS_TXT_OWNER_ID
"default", VELERO_BACKUP_PREFIX ""): the live cluster keeps the name it is running, and the
NEW environment takes the new one. The asymmetry is deliberate and documented at each
kustomization.
WHERE THE MATERIAL CAME FROM. Neither side is invented:
- the four unsuffixed files are restored byte-for-byte from `main` (the MinIO material as
it stood before 0a6a862), so dev's Secrets are IDENTICAL to what dev is running;
- the four `-prod` files carry the R2 token from 0a6a862 (already on this branch as
9739e9b), re-encrypted only because metadata.name changed.
RE-ENCRYPTION, AND WHY A COPY+SED IS NOT ENOUGH. `.sops.yaml` sets
`encrypted_regex: ^(data|stringData)$`, so metadata.name stays cleartext — but SOPS MACs the
WHOLE document (there is no `mac_only_encrypted` here), so editing that cleartext line
invalidates the MAC and `sops -d` then refuses the file. The `-prod` files were therefore
produced inside the dev toolbox (infra/devcontainer, which has sops 3.9.4 and the age key
mounted) by decrypt -> rename -> re-encrypt, with the plaintext never leaving a container
tmpfile. Note for the next person: `sops -e --output NEW OLD` matches creation rules against
the INPUT path, so encrypting a tmpfile into a repo path needs `--filename-override NEW` or
it dies with "no matching creation rules found".
Verified: all eight files `sops -d` cleanly; the four dev files hash to dev's material and the
four prod files to R2's, and the two sets differ. `.sops.yaml` needed no new rule — the
existing `(platform|apps|catalog)/.*(secret|sops)\.ya?ml$` rule already matches
`*-prod.sops.yaml`, so the new files encrypt to the same age recipient.
apps/wave-0/kustomization.yaml is GENERATED, so render.py learns the `-prod` sidecar rather
than the golden being hand-edited: `<db>-backup-credentials-prod.sops.yaml` now places into
the DB's wave alongside its unsuffixed twin (both must exist before the Cluster that names
one), and the selftest asserts it.
Co-Authored-By: Claude Opus 5 <[email protected]>
Roll the verified Option-B per-app oauth2-proxy sidecar out to the remaining
four forward-auth apps after the pihole canary was confirmed fail-closed
(unauthenticated -> 302 to Keycloak, pod 2/2, clean sidecar OIDC discovery).
Each app's Gateway route now terminates at its oauth2-proxy sidecar (:4180,
--allowed-group=media-users, --upstream=http://127.0.0.1:<uiport>); the
gateway-ingress CiliumNetworkPolicy allows the Gateway to reach ONLY :4180, so
the bare UI is never served un-authenticated (fail-closed; R-IDP-02/06). The
apps' own app/API ports stay in-Service for EAST-WEST callers (arr ->
qbittorrent:8080 / jackett:9117) via the existing reciprocal-ingress allows.
- keycloak: confidential jackett/qbittorrent/radarr/sonarr clients (realm-import
+ realm-clients mirror), provisioned live via kcadm (groups membership mapper).
- secrets: four SOPS/age <app>-oauth2-proxy-credentials, wired into apps/wave-0
and apps/wave-1 by the renderer's sidecar-creds wave rule.
- ci: remove the temporary canary deferral (all five goldens now reproduce).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Close the un-gated-UI security finding for forward-auth apps (R-IDP-02) by
rendering the Option-B per-app oauth2-proxy reverse-proxy sidecar that crafty
proves working. catalog/render.py now emits, for any forward-auth app with an
exposed UI: the oauth2-proxy sidecar (upstream http://127.0.0.1:<uiport>,
:4180, --allowed-group per auth.group), an authproxy :4180 Service port, an
HTTPRoute backendRef -> :4180, and a gateway-ingress CiliumNetworkPolicy that
allows the Gateway to reach ONLY :4180 (fail-closed: the bare UI is never
served un-authenticated). oidc/native/none apps render byte-identical.
This is the CANARY: only pihole's golden carries the sidecar and is verified
fail-closed in-cluster first. jackett/qbittorrent/radarr/sonarr are deferred to
the rollout commit (temporarily listed in ci/render-golden.sh) so their UIs
keep serving exactly as today until their clients+secrets land.
- keycloak: confidential pihole client (realm-import + realm-clients mirror),
provisioned live via kcadm (groups membership mapper, bare names).
- secret: SOPS/age pihole-oauth2-proxy-credentials (client-id/secret/cookie),
wired into apps/wave-0 via the renderer's new sidecar-creds wave rule.
- L1 render goldens regenerated for the sidecar (forward-auth fixtures).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Replaces the single `apps` Flux Kustomization with dependency-ordered waves
(apps-wave-0..2) so a crashlooping workload isolates to its wave + downstream
instead of stalling the entire app tier. Parallel within a wave, ordered across.
- catalog/render.py: compute waves by longest dependency-path depth over the
ServiceDefinition `dependencies:` edges (a `<svc>-db` is a leaf, so its
consumer lands strictly higher — the "DB below consumer" rule falls out of
longest-path, no special case). `--emit-waves` generates the per-wave
kustomizations + Flux CRs; `--wave-layout` places renders into the right dir.
wave 0: pihole gitea jackett plex qbittorrent nextcloud-db semaphore-db
wave 1: sonarr radarr nextcloud semaphore gitea-actions
wave 2: seerr
- apps/*.yaml -> apps/wave-{0,1,2}/ (byte-identical moves; SOPS files intact).
- clusters/base: apps.yaml removed; apps-wave-{0,1,2}.yaml added (dependsOn
chain: platform-config -> wave-0 -> wave-1 -> wave-2, prune+wait each).
- clusters/dev: one SOPS decryption patch per wave (load-bearing).
- DESIGN-waves.md: wave table, topology, cutover runbook.
Cutover: preceded by disabling prune on `apps` (1bfb0db) so removing that
object here does not GC its resources; the wave Kustomizations SSA-adopt the
byte-identical manifests in place (no restart, no PVC churn).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
jakman85andClaude Opus 5