a819c50ffe74caf542e4a9b021c5697e1655712e
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
c8c1d2935d |
feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).
THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.
Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.
PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.
BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.
TWO THAT CANNOT BE TOKENIZED, flagged not forced:
- headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
anyway — client secrets are realm-scoped.
- kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
prod.home.
EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.
I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.
DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.
L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.
PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.
Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
4c4f5355d9 |
fix(sso): postStart hooks must exit 0 — nextcloud crashlooped the moment OIDC ran
LIVE REGRESSION, caught on the reconcile after merging #5. nextcloud went
CrashLoopBackOff with FailedPostStartHook, 7 restarts, and apps-wave-1/2 stalled.
ROOT CAUSE — a latent bug that could never fire until now. The final statement of
nextcloud's postStart script was:
[ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote ... after retries"
On the SUCCESS path (p < 15) the test returns 1, and as the script's LAST command that
becomes the hook's exit status. A non-zero postStart exit makes the kubelet KILL the
container. This was harmless for as long as the OIDC guard above it was dead code that
exited 0 first (see
|
||
|
jakman85andClaude Opus 5
|
3484955903 |
fix(sso): gitea + nextcloud OIDC bootstrap was DEAD CODE — envsubst ate the guard
Both ServiceDefinitions guarded their Keycloak OIDC registration with an UNESCAPED
braced expansion:
if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then # gitea
if [ -z "${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then # nextcloud, then exit 0
apps-wave-0/1 both declare postBuild.substituteFrom: cluster-vars, and neither name is a
cluster-vars key — so Flux's envsubst substituted them away AT RECONCILE TIME, long
before the container ever ran. Verified in the LIVE cluster: the deployed Deployments
literally contain
if [ -z "" ]; then
which is always true. The entire OIDC registration block has been skipped on EVERY boot
of both apps.
WHY NOBODY NOTICED: both apps DO have the provider registered — `gitea admin auth list`
shows keycloak/OAuth2/true and `occ user_oidc:provider` shows the keycloak provider.
Those live in each app's own database, persisted from an earlier successful registration.
SSO works today and DECISIONS-LOG is not wrong. But the idempotent re-registration is
dead, so any PVC rebuild, DB restore, or fresh cluster build comes up with NO OIDC and
only the break-glass local admin. A PROD BUILD FROM THIS REPO WOULD SILENTLY HAVE NO SSO
on gitea or nextcloud.
The irony: the surrounding code was hardened twice — see its own comments about a "silent
first-boot RACE", the verify-then-retry loops, the marker files. All of that hardening was
unreachable, because the guard above it always evaluated true.
Not a regression from the recent R-ENV-02 tokenization — introduced 2026-08-27 (
|
||
|
jakman85andClaude Opus 4.8
|
14521d37d3 |
fix(nextcloud-sso): apply step-(d) retry hardening at SOURCE (catalog)
apps/wave-1/nextcloud.yaml is GENERATED from catalog/nextcloud.yaml by catalog/render.py, so the previous commit hardened only the rendered golden — the next render would have reverted step (d) to the old single-pass version. Apply the same verify-and-retry hardening to the postStart hook in catalog/nextcloud.yaml (the source of truth). Re-rendered with `render.py --all catalog -o apps --wave-layout`; apps/wave-1/nextcloud.yaml is byte-identical to the golden committed in the previous commit (render round-trip clean), so no apps/ change is needed here. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
7f92adfac6 |
fix(nextcloud): allow SSRF to private IdP + promote platform-admins to admin
Task 1 — browser SSO "Could not reach OpenID Connect provider" (confirmed root cause, not guessed). keycloak.home.keithjackson.dev resolves split-horizon to the PRIVATE Cilium gateway VIP (192.168.1.53). Nextcloud's SSRF guard refuses its own server-side HTTP requests to RFC1918 addresses, so user_oidc's discovery/JWKS/token fetch AT LOGIN throws OCP\Http\Client\LocalServerException "violates local access rules" -> surfaced to the user as "Could not reach the provider". Proven from the live pod: nextcloud.log carries the exact LocalServerException from DiscoveryService::obtainDiscovery with Guzzle option allow_local_address:false; a Nextcloud-context IClientService->get() of the discovery URL returns LocalServerException with the flag false and HTTP 200 with it true (A/B toggled). The postStart (b) curl check passed regardless because it uses system libcurl, not Nextcloud's guard — which is why the provider registered yet login still failed. Fix (reproducible, survives pod rolls): the postStart hook now runs `occ config:system:set allow_local_remote_servers --value=true --type=boolean` right after the install/maintenance wait, on every start. Task 2 — promote Keycloak `platform-admins` members to Nextcloud admins (#5). user_oidc 8.11 has NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a login WHITELIST only), so the hook mirrors the provisioned `platform-admins` membership into the built-in `admin` group idempotently (occ group:list --output=json parsed via php argv; occ group:adduser admin per member). A user is promotable after their first OIDC login provisions them + the next pod start (or one manual `occ group:adduser admin <uid>`). The LOCAL admin stays as break-glass (docs/15). apps/wave-1/nextcloud.yaml re-rendered from catalog (single-file render; byte-diff confined to the embedded hook). Rendered hook passes `sh -n`; parse step validated in-pod. `make -C ci gate` green (lint/l0/l1; conformance + spec-immutability skip: no cluster / spec repo unmounted). Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
d34c639269 |
fix(oidc): make gitea + nextcloud OIDC login-source registration persist
The Keycloak OIDC postStart registration for gitea and nextcloud ran but never persisted, leaving `gitea admin auth list` empty and nextcloud's user_oidc with "No providers configured" despite the apps installing/ enabling and the client secret being present. Two distinct root causes, confirmed by running the registration manually against the live warm pods: - nextcloud: a QUOTING BUG, not a race. The OCC() wrapper runs `su -c "php occ $*"`, which word-split `--scope="openid email profile"` back into three tokens, so occ saw `email profile` as extra positional args -> "Too many arguments to user_oidc:provider" -> exit 1, swallowed by `|| echo failed`. Failed on EVERY boot. Fixed by running the provider call as ONE `su -c` with the scope single-quoted so www-data's shell keeps it a single argument; keep the OCC() wrapper only for single-token subcommands. - gitea: a first-boot RACE. The direct `admin auth add-oauth` persists once the app/network is warm (verified live), but fired too early at postStart and its failure was swallowed (`|| echo ... >&2`, exit 0). Hardening applied to both catalog sources (rendered into apps/wave-*): - wait (bounded ~90s) for the Keycloak discovery doc to return HTTP 200; - nextcloud also waits for `installed: true` AND `maintenance: false`; - register idempotently (create-or-update) then VERIFY the source/provider actually persisted, retrying the whole cycle (guards exit-0-but-nothing- stuck); - on give-up, log LOUDLY to stderr AND a marker file on the persistent volume (/data, /var/www/html) instead of swallowing — next failure is visible; - never crashloop: missing-secret guard retained, break-glass local admin stays usable. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
f163026954 |
feat(nextcloud,semaphore): wire native Keycloak OIDC SSO (break-glass admins retained)
Extends the SSO work to the two remaining `auth: oidc` apps still on local-admin
logins, each as a native confidential client of the Keycloak `homelab` realm with
its local admin kept as break-glass (docs/15 availability caveat; R-IDP-02/06).
Realm (source of truth): add the `nextcloud` and `semaphore` confidential clients
to platform/keycloak/keycloak-realm-import.yaml (modeled on grafana/gitea) + their
NEXTCLOUD_CLIENT_SECRET / SEMAPHORE_CLIENT_SECRET placeholders. As before, a
KeycloakRealmImport no-ops on an already-existing realm, so both live clients were
ALSO created via kcadm (nextcloud e95049ed-ca10-4085-97d9-4bbef495fe61, semaphore
1835bfe9-b125-4026-8f69-33557519cf4a); the YAML keeps git authoritative.
Secrets (SOPS/age): add nextcloud-client-secret + semaphore-client-secret to
keycloak-realm-clients.sops.yaml (keycloak ns). Mirror the nextcloud secret as
oidc-client-secret into nextcloud-bootstrap (nextcloud ns). For semaphore, the
whole SEMAPHORE_OIDC_PROVIDERS JSON (which embeds the client secret) is held
encrypted as oidc-providers-json in semaphore-bootstrap (semaphore ns). All values
verified == the live clients; existing keys untouched; no plaintext in git.
App config (catalog source + re-render):
- nextcloud: add NEXTCLOUD_OIDC_CLIENT_SECRET env + an idempotent postStart hook
that installs/enables user_oidc and runs `occ user_oidc:provider keycloak`
(create-or-update; secret via --clientsecret-env so it never hits argv; occ as
www-data; group provisioning on the `groups` claim). Syntax verified against the
running user_oidc 8.11.0 on Nextcloud 31.0.14; su-to-www-data env passthrough
verified. Missing-secret/not-installed guards prevent crashloops.
- semaphore: add SEMAPHORE_OIDC_PROVIDERS env sourced from the encrypted
oidc-providers-json (provider_url=realm issuer/auto-discovery, redirect
/api/auth/oidc/keycloak/redirect). Mechanism verified against semaphore v2.19.10
source (util ConfigType oidc_providers env tag + OidcProvider struct + router
redirect path); no kubelet service-link env collision.
apps/wave-1/{nextcloud,semaphore}.yaml re-rendered via
`render.py --no-image-automation` (diffs are env/lifecycle-only).
Verified: realm import Done=True/HasErrors=False and platform-config
Ready=True/Healthy=True (unchanged); edited realm YAML passes server dry-run; live
homelab clients now include gitea/nextcloud/semaphore; static gate (lint/l0/l1)
green. Browser SSO login for each still needs a human check.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
4be6752c71 |
feat(apps): onboard nextcloud (Phase 6) — first CNPG-DB + OIDC-target patterns
Onboard nextcloud (docs/13 row 7): public, DR-critical file sync backed by an external PostgreSQL. Establishes the reusable CNPG-DB pattern for later DB-backed services. - catalog/nextcloud.yaml: ServiceDefinition (image nextcloud:31-apache :80, public, internet, data 20Gi longhorn @/var/www/html, backup daily-offcluster, auth oidc +platform-admins, ingress nextcloud.home.keithjackson.dev, dependency nextcloud-db:5432). Uses schema 1.2.0 env for the Postgres connection, break-glass admin, trusted-domain + OVERWRITE* proxy settings. - apps/nextcloud-db.yaml: CloudNativePG Cluster in a dedicated nextcloud-db namespace (modeled on keycloak-db; no default-deny, mirroring the verified keycloak posture), inheritedMetadata stamps app.kubernetes.io/name=nextcloud-db so the rendered egress CNP matches. initdb consumes a pre-provisioned owner secret. - apps/nextcloud-db-app.sops.yaml + apps/nextcloud-bootstrap-secret.sops.yaml: SOPS (age) — shared DB password mirrored across the two namespaces (Pods cannot read a Secret cross-namespace), plus the break-glass local admin. - apps/nextcloud.yaml: rendered manifests (catalog/render.py --no-image-automation). - apps/kustomization.yaml: wire in the four new resources. OIDC realm-client wiring DEFERRED (WS-APPS step-3 escape hatch): editing the reconciled realm import in platform-config risks the running Keycloak realm / whole platform-config apply, and nextcloud OIDC needs post-install occ steps with no first-class env — nextcloud comes up INSTALLED on the break-glass local admin (as Gitea did); the Keycloak client is a follow-up. Off-cluster backup awaits Velero/OQ-BKP (intent declared). Gates: render --selftest PASS; other apps byte-identical; ci l0 17/17; l1 3/3; lint clean. Co-Authored-By: Claude Opus 4.8 <[email protected]> |