Files
home-network-k3s-impl/apps/wave-1/nextcloud.yaml
T
jakman85andClaude Opus 4.8 d34c639269 fix(oidc): make gitea + nextcloud OIDC login-source registration persist
The Keycloak OIDC postStart registration for gitea and nextcloud ran but
never persisted, leaving `gitea admin auth list` empty and nextcloud's
user_oidc with "No providers configured" despite the apps installing/
enabling and the client secret being present.

Two distinct root causes, confirmed by running the registration manually
against the live warm pods:

- nextcloud: a QUOTING BUG, not a race. The OCC() wrapper runs
  `su -c "php occ $*"`, which word-split `--scope="openid email profile"`
  back into three tokens, so occ saw `email profile` as extra positional
  args -> "Too many arguments to user_oidc:provider" -> exit 1, swallowed
  by `|| echo failed`. Failed on EVERY boot. Fixed by running the provider
  call as ONE `su -c` with the scope single-quoted so www-data's shell
  keeps it a single argument; keep the OCC() wrapper only for single-token
  subcommands.
- gitea: a first-boot RACE. The direct `admin auth add-oauth` persists once
  the app/network is warm (verified live), but fired too early at postStart
  and its failure was swallowed (`|| echo ... >&2`, exit 0).

Hardening applied to both catalog sources (rendered into apps/wave-*):
- wait (bounded ~90s) for the Keycloak discovery doc to return HTTP 200;
- nextcloud also waits for `installed: true` AND `maintenance: false`;
- register idempotently (create-or-update) then VERIFY the source/provider
  actually persisted, retrying the whole cycle (guards exit-0-but-nothing-
  stuck);
- on give-up, log LOUDLY to stderr AND a marker file on the persistent
  volume (/data, /var/www/html) instead of swallowing — next failure is
  visible;
- never crashloop: missing-secret guard retained, break-glass local admin
  stays usable.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-27 17:41:54 -05:00

304 lines
11 KiB
YAML

---
# Namespace (trust zone; default-deny + policies apply)
apiVersion: v1
kind: Namespace
metadata:
name: nextcloud
labels:
homelab.keithjackson.dev/app-namespace: 'true'
gateway-access: shared
annotations:
homelab.keithjackson.dev/backup-policy: daily-offcluster
---
# Default-deny baseline (R-NET-01): deny all, allow only DNS
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: nextcloud
labels:
homelab.keithjackson.dev/baseline: default-deny
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress: []
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
---
# PVC nextcloud-data (R-STATE)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nextcloud-data
namespace: nextcloud
labels:
app.kubernetes.io/name: nextcloud
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 20Gi
---
# Deployment (R-PLAT-02)
apiVersion: apps/v1
kind: Deployment
metadata:
name: nextcloud
namespace: nextcloud
labels:
app.kubernetes.io/name: nextcloud
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: nextcloud
template:
metadata:
labels:
app.kubernetes.io/name: nextcloud
homelab.keithjackson.dev/egress-class: internet
homelab.keithjackson.dev/auth-mode: oidc
annotations:
homelab.keithjackson.dev/auth-mode: oidc
homelab.keithjackson.dev/auth-groups: platform-admins
spec:
containers:
- name: nextcloud
image: nextcloud:31-apache
ports:
- containerPort: 80
env:
- name: POSTGRES_HOST
value: nextcloud-db-rw.nextcloud-db.svc.cluster.local
- name: POSTGRES_DB
value: nextcloud
- name: POSTGRES_USER
value: nextcloud
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: nextcloud-bootstrap
key: db-password
- name: NEXTCLOUD_ADMIN_USER
valueFrom:
secretKeyRef:
name: nextcloud-bootstrap
key: admin-username
- name: NEXTCLOUD_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
name: nextcloud-bootstrap
key: admin-password
- name: NEXTCLOUD_TRUSTED_DOMAINS
value: nextcloud.home.keithjackson.dev
- name: OVERWRITEHOST
value: nextcloud.home.keithjackson.dev
- name: OVERWRITEPROTOCOL
value: https
- name: OVERWRITECLIURL
value: https://nextcloud.home.keithjackson.dev
- name: NEXTCLOUD_OIDC_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: nextcloud-bootstrap
key: oidc-client-secret
volumeMounts:
- name: data
mountPath: /var/www/html
lifecycle:
postStart:
exec:
command:
- /bin/sh
- -c
- "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n# maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n# stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n i=$((i + 1))\n [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\nif [ -z \"${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1; break\n fi\n d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n# HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n# single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n# form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n# occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n# (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n --clientid=nextcloud \\\n --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n --scope='openid email profile' \\\n --unique-uid=0 \\\n --mapping-uid=preferred_username \\\n --mapping-email=email \\\n --mapping-display-name=name \\\n --group-provisioning=1 \\\n --mapping-groups=groups\" >/dev/null 2>&1 || true\n if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep\
\ 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n"
volumes:
- name: data
persistentVolumeClaim:
claimName: nextcloud-data
---
# Service
apiVersion: v1
kind: Service
metadata:
name: nextcloud
namespace: nextcloud
labels:
app.kubernetes.io/name: nextcloud
spec:
selector:
app.kubernetes.io/name: nextcloud
ports:
- name: port-80
port: 80
targetPort: 80
---
# CiliumNetworkPolicy egress allow (R-NET-02): nextcloud-allow-nextcloud-db-5432
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: nextcloud-allow-nextcloud-db-5432
namespace: nextcloud
labels:
app.kubernetes.io/name: nextcloud
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: nextcloud
egress:
- toEndpoints:
- matchLabels:
app.kubernetes.io/name: nextcloud-db
k8s:io.kubernetes.pod.namespace: nextcloud-db
toPorts:
- ports:
- port: '5432'
protocol: TCP
---
# CiliumNetworkPolicy gateway-ingress allow (R-NET-01/02): nextcloud-allow-gateway-ingress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: nextcloud-allow-gateway-ingress
namespace: nextcloud
labels:
app.kubernetes.io/name: nextcloud
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: nextcloud
ingress:
- fromEntities:
- ingress
toPorts:
- ports:
- port: '80'
protocol: TCP
---
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): nextcloud-allow-internet-egress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: nextcloud-allow-internet-egress
namespace: nextcloud
labels:
app.kubernetes.io/name: nextcloud
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: nextcloud
egress:
- toEntities:
- world
---
# HTTPRoute nextcloud-80 (R-TLS/R-DNS)
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: nextcloud-80
namespace: nextcloud
labels:
app.kubernetes.io/name: nextcloud
annotations:
homelab.keithjackson.dev/auth-mode: oidc
homelab.keithjackson.dev/auth-groups: platform-admins
spec:
parentRefs:
- name: shared
namespace: gateway
sectionName: https
hostnames:
- nextcloud.home.keithjackson.dev
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: nextcloud
port: 80
---
# ServiceMonitor (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: nextcloud
namespace: nextcloud
labels:
release: kube-prometheus-stack
app.kubernetes.io/name: nextcloud
spec:
selector:
matchLabels:
app.kubernetes.io/name: nextcloud
namespaceSelector:
matchNames:
- nextcloud
endpoints:
- port: metrics
interval: 30s
---
# PrometheusRule SLI alerts (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: nextcloud-slis
namespace: nextcloud
labels:
release: kube-prometheus-stack
app.kubernetes.io/name: nextcloud
spec:
groups:
- name: nextcloud.slis
rules:
- alert: nextcloud-down
expr: up{job="nextcloud"} == 0
for: 5m
labels:
severity: critical
service: nextcloud
annotations:
summary: nextcloud is down
description: 'nextcloud has been unreachable for 5m (SLI: up).'
---
# Flux ImageRepository (R-UPD)
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
name: nextcloud
namespace: nextcloud
spec:
image: nextcloud
interval: 1h0m0s
---
# Flux ImagePolicy (R-UPD)
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
name: nextcloud
namespace: nextcloud
annotations:
homelab.keithjackson.dev/update-channel: stable
homelab.keithjackson.dev/auto-rollback: 'true'
spec:
imageRepositoryRef:
name: nextcloud
policy:
semver:
range: '>=31.0.0 <32.0.0'