Restores the four semaphore resources a peer temp-disabled in 0abadd3 to unblock
the apps Kustomization while the crashloop was diagnosed. Both root causes are now
fixed and verified against the live image:
1. permission-denied writing /etc/semaphore/config.json (non-root UID 1001 vs
root-owned Longhorn PVC) -> podSecurityContext.fsGroup=1001 (schema 1.3.0);
verified the pod got past config write and successfully pinged its PostgreSQL.
2. config panic on SEMAPHORE_PORT=tcp://<svcIP>:3000 (kubelet Docker-link
injection for the same-named Service colliding with Semaphore's listen port)
-> explicit SEMAPHORE_PORT=":3000" env shadowing the service-link var.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The semaphoreui/semaphore image falls through to its interactive setup wizard
and panics on /etc/semaphore/config.json permission-denied; the env-only pattern
doesn't yield a non-interactive config. Backed out to unblock the apps
Kustomization; files retained for a proper config-model follow-up.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Semaphore (Ansible UI, docs/13 row 10): internal exposure, internet egress,
config on Longhorn + external PostgreSQL, auth oidc. Reuses the CNPG-DB pattern
nextcloud established (apps/nextcloud-db.yaml, commit 4be6752).
- catalog/semaphore.yaml: ServiceDefinition (schema >=1.2.0 env) wiring the
SEMAPHORE_DB_* connection to semaphore-db-rw.semaphore-db:5432, break-glass
SEMAPHORE_ADMIN* local admin, and the FIXED base64 SEMAPHORE_ACCESS_KEY_ENCRYPTION.
- apps/semaphore.yaml: rendered (render.py --no-image-automation) — Deployment,
config PVC, Service, default-deny + egress/gateway/internet CNPs, HTTPRoute
semaphore.home.keithjackson.dev:3000, ServiceMonitor, SLI alert.
- apps/semaphore-db.yaml: CNPG Cluster in dedicated ns semaphore-db, initdb from
the owner secret, inheritedMetadata app.kubernetes.io/name=semaphore-db so the
dep-edge egress CNP matches.
- apps/semaphore-db-app.sops.yaml + apps/semaphore-bootstrap-secret.sops.yaml:
SOPS(age) owner creds (basic-auth) + mirrored db-password, break-glass admin,
and the persistent access-key encryption key.
OIDC (Keycloak realm client) is DEFERRED like gitea/nextcloud — comes up on the
break-glass local admin; realm-client wiring is the batched follow-up.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Onboard nextcloud (docs/13 row 7): public, DR-critical file sync backed by an
external PostgreSQL. Establishes the reusable CNPG-DB pattern for later DB-backed
services.
- catalog/nextcloud.yaml: ServiceDefinition (image nextcloud:31-apache :80, public,
internet, data 20Gi longhorn @/var/www/html, backup daily-offcluster, auth oidc
+platform-admins, ingress nextcloud.home.keithjackson.dev, dependency
nextcloud-db:5432). Uses schema 1.2.0 env for the Postgres connection, break-glass
admin, trusted-domain + OVERWRITE* proxy settings.
- apps/nextcloud-db.yaml: CloudNativePG Cluster in a dedicated nextcloud-db namespace
(modeled on keycloak-db; no default-deny, mirroring the verified keycloak posture),
inheritedMetadata stamps app.kubernetes.io/name=nextcloud-db so the rendered egress
CNP matches. initdb consumes a pre-provisioned owner secret.
- apps/nextcloud-db-app.sops.yaml + apps/nextcloud-bootstrap-secret.sops.yaml: SOPS
(age) — shared DB password mirrored across the two namespaces (Pods cannot read a
Secret cross-namespace), plus the break-glass local admin.
- apps/nextcloud.yaml: rendered manifests (catalog/render.py --no-image-automation).
- apps/kustomization.yaml: wire in the four new resources.
OIDC realm-client wiring DEFERRED (WS-APPS step-3 escape hatch): editing the reconciled
realm import in platform-config risks the running Keycloak realm / whole platform-config
apply, and nextcloud OIDC needs post-install occ steps with no first-class env — nextcloud
comes up INSTALLED on the break-glass local admin (as Gitea did); the Keycloak client is a
follow-up. Off-cluster backup awaits Velero/OQ-BKP (intent declared).
Gates: render --selftest PASS; other apps byte-identical; ci l0 17/17; l1 3/3; lint clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Extend the ServiceDefinition contract with optional per-container `env`,
`envFrom`, and `lifecycle` fields (k8s EnvVar / EnvFromSource / Lifecycle
shapes) so headless-install / DB-backed apps configure themselves through
the SD instead of a supplementary kustomize patch.
- schema: bump 1.1.0 -> 1.2.0; add optional env/envFrom/lifecycle to the
container kind (literal + secretKeyRef/configMapKeyRef; secretRef/
configMapRef; postStart/preStop exec via $defs/execHandler). Existing
fields unchanged; all fields optional -> absent renders byte-identical.
- render.py: emit env/envFrom/lifecycle verbatim onto the Deployment app
container when present; --selftest coverage for the new fields.
- catalog/gitea.yaml: fold the headless-install env (INSTALL_LOCK, fixed
SECRET_KEY via secretKeyRef, DOMAIN/ROOT_URL, actions, break-glass admin
creds) + the idempotent postStart break-glass admin bootstrap inline.
- apps/gitea.yaml: re-rendered with env/lifecycle inline (effective pod
spec provably equivalent to base+patch).
- DELETE apps/gitea-bootstrap-patch.yaml + drop it from apps/kustomization
patches. gitea-bootstrap-secret.sops.yaml kept (still sourced by the env).
Proves the renderer feature by retiring the known TODO; unblocks env + DB
onboarding for nextcloud/semaphore/unifi/minecraft. All other rendered apps
byte-identical. Gates: selftest PASS, l0 17/17 + schema, l1 goldens, lint.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
STEP 1 — enable Gitea Actions: pin GITEA__actions__ENABLED=true and
DEFAULT_ACTIONS_URL=github in the gitea bootstrap patch (was implicit upstream
default; make it a declared property).
STEP 2 — deploy + register an act_runner in a dedicated non-app namespace
(gitea-actions) so the privileged DinD executor sidecar is admitted without a
Kyverno policy exception (restrict-host-namespaces scopes to app namespaces only).
Docker executor + DinD sidecar over a shared TLS-cert emptyDir; runner state on a
Longhorn PVC (registers once). Registration token delivered as a SOPS-encrypted
Secret decrypted by Flux. Runner mounts no ServiceAccount token (R-SEC-02).
STEP 3 — author .gitea/workflows/gate.yml: the live enforced merge gate running
the STATIC layers (spec-immutability best-effort, lint, Kyverno L0, L1 render) via
`make -C ci lint l0 l1` with the pinned toolchain installed at job start. L2/L3
conformance is an if:false placeholder (needs a toolchain image + scoped cluster
creds — a design decision, not wired). merge-gate.yaml reduced to workflow_dispatch
(dormant full-pipeline blueprint). install-tools.sh gains check-jsonschema so the
schema stage validates in CI instead of skipping.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Two defects blocked in-cluster Gitea from becoming the Flux source:
1. Storage mountPath: the renderer hard-coded /data/<name>, so Gitea's PVC
mounted at /data/data while Gitea writes all state to /data — real repos/DB
landed on the ephemeral rootfs. Add an OPTIONAL `mountPath` to the storage
schema + render.py (default stays /data/<name>, back-compat). catalog/gitea.yaml
sets data.mountPath: /data. Only gitea.yaml's rendered mount changes; every
other app renders byte-identical.
2. Headless install: the instance was un-installed (INSTALL_LOCK=false, ephemeral
SECRET_KEY, no admin, API 404). A Gitea-only supplementary manifest
(apps/gitea-bootstrap-patch.yaml) patches the rendered Deployment with
GITEA__security__INSTALL_LOCK=true, a FIXED SECRET_KEY from a persistent SOPS
secret (apps/gitea-bootstrap-secret.sops.yaml), ROOT_URL/DOMAIN, and a postStart
hook that idempotently creates a break-glass local admin. State persists on the
now-correct /data Longhorn PVC.
The env/lifecycle wiring is a supplementary kustomize patch because the
ServiceDefinition schema/renderer does not yet express env/envFrom/lifecycle —
noted as a renderer follow-up (TODO in the patch).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add catalog/ ServiceDefinitions for qbittorrent and jackett and their
rendered apps/ manifests, wired into apps/kustomization.yaml.
Both are VPN-class (egress: vpn -> egress-class=vpn pod label steered by the
CiliumEgressGatewayPolicy + host kill-switch, ADR-0015) and internal
forward-auth UIs (oauth2-proxy -> Keycloak, group media-users). Config on
3-replica Longhorn. qBittorrent's bulk downloads volume is DEFERRED pending
OQ-2 (non-provisioning bulk StorageClass stub), mirroring plex/arr.
Traces: R-NET-05/06 (VPN egress + kill-switch), R-IDP-02/06 (forward-auth),
R-STATE-02/03, R-OBS-01, R-UPD-01/03, R-GITOPS-02.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add catalog/plex.yaml (kind: container) and its rendered apps/plex.yaml,
and wire apps/kustomization.yaml. Plex = docs/13 row 1, the catalog's
largest service; validates the public-exposure + config-on-Longhorn path.
- image plexinc/pms-docker (pinned recent stable tag), port 32400
- exposure: public at plex.home.keithjackson.dev; egress: internet
- auth: native (Plex accounts, not Keycloak — docs/13/15 exception)
- config PVC 20Gi on longhorn; off-cluster backup (daily-offcluster) as the
longhorn tier forbids reacquirable under L0 (R-STATE-03)
- observability up-SLI; update channel stable
Bulk media (R-STATE-04) deferred: the `bulk` StorageClass is a documented
stub (OQ-2 unresolved, NFS CSI not installed) so a bulk PVC would leave the
pod Pending. Media mount omitted; attach once bulk provisions.
Traces: R-GITOPS-02, R-NET-01/02/05, R-STATE-01/02/03, R-TLS/R-DNS,
R-OBS-01, R-UPD-01/03, R-IDP-02, R-PLAT-02.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
First app onboarded through catalog/render.py -> apps/. Added --no-image-automation so apps
deploy before the Flux image controllers exist (L1 goldens keep the default, unchanged).
KNOWN GAP surfaced: exposed apps get default-deny (ingress: []) with no gateway-ingress allow,
so the HTTPRoute path is blocked — renderer/WS-NET follow-up to allow Cilium gateway ingress
to services with an exposure/HTTPRoute.
jakman85andClaude Opus 4.8