The committed apps/ goldens were a MIX of two render flag sets: 8 services
were rendered with `--no-image-automation` (no image CRs) while gitea and
nextcloud carried stale ImageRepository/ImagePolicy CRs from a render WITHOUT
the flag. Neither single command reproduced all goldens, and the gate did not
catch it.
Part A — canonical render:
python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation
`--no-image-automation` is canonical: image automation is authored-but-DISABLED
(DECISIONS-LOG; image-reflector/-automation controllers not installed), and
every app was onboarded via this flag. Re-render drops the ImageRepository +
ImagePolicy objects from gitea + nextcloud only (52 line deletions, purely the
trailing image-automation blocks). No functional content changed — the
nextcloud postStart hardening and gitea OIDC hook are untouched.
Part B — enforcement:
ci/render-golden.sh renders every onboarded catalog service with the canonical
command into a temp dir and FAILS on ANY diff vs committed apps/ (missing
golden or content drift), across all 10 services (not just the 3 L1 spot-checks).
Not-onboarded catalog entries (keycloak, vm-demo) are excluded per DESIGN-waves.
Wired as a required no-cluster stage in ci/gate.sh, ci/Makefile, and both the
live (.gitea/workflows/gate.yml) and dormant (merge-gate.yaml) pipelines.
Proof: tampering a golden makes the stage exit 1 (blocks merge); reverting
restores exit 0. `make -C ci gate` green in the devcontainer.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Two SSO follow-ups the operator hit after Nextcloud OIDC login worked.
1. Sign-out "redirect failed". On logout user_oidc redirects the browser to
Keycloak's end_session_endpoint with post_logout_redirect_uri=<Nextcloud
base '/'> and client_id=nextcloud but NO id_token_hint (provider
sendIdTokenHint=0). Keycloak then validates that redirect against the
client's post.logout.redirect.uris attribute, which was empty -> "Invalid
redirect uri" -> the operator's "redirect failed". Register
https://nextcloud.home.keithjackson.dev/* on the nextcloud client
(attributes.post.logout.redirect.uris). Applied live via kcadm and captured
here so git stays authoritative (a RealmImport no-ops on the existing realm).
2. postStart step (d) mirrors platform-admins -> built-in admin group, but the
previous single best-effort pass ran occ group:adduser once with output
swallowed and no retry. At early boot (container Started ~9s after create)
that call can transiently fail (DB / maintenance race), silently dropping the
promotion until the next restart. Harden step (d) with a verify-and-retry
loop (like step (c)): re-read group:list and re-add any platform-admins
member still missing from admin until verified (bounded ~30s), no-op for
existing members.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Task 1 — browser SSO "Could not reach OpenID Connect provider" (confirmed root
cause, not guessed). keycloak.home.keithjackson.dev resolves split-horizon to the
PRIVATE Cilium gateway VIP (192.168.1.53). Nextcloud's SSRF guard refuses its own
server-side HTTP requests to RFC1918 addresses, so user_oidc's discovery/JWKS/token
fetch AT LOGIN throws OCP\Http\Client\LocalServerException "violates local access
rules" -> surfaced to the user as "Could not reach the provider". Proven from the
live pod: nextcloud.log carries the exact LocalServerException from
DiscoveryService::obtainDiscovery with Guzzle option allow_local_address:false; a
Nextcloud-context IClientService->get() of the discovery URL returns
LocalServerException with the flag false and HTTP 200 with it true (A/B toggled).
The postStart (b) curl check passed regardless because it uses system libcurl, not
Nextcloud's guard — which is why the provider registered yet login still failed.
Fix (reproducible, survives pod rolls): the postStart hook now runs
`occ config:system:set allow_local_remote_servers --value=true --type=boolean`
right after the install/maintenance wait, on every start.
Task 2 — promote Keycloak `platform-admins` members to Nextcloud admins (#5).
user_oidc 8.11 has NO native group->admin mapping (occ user_oidc:provider offers
group PROVISIONING and a login WHITELIST only), so the hook mirrors the provisioned
`platform-admins` membership into the built-in `admin` group idempotently (occ
group:list --output=json parsed via php argv; occ group:adduser admin per member).
A user is promotable after their first OIDC login provisions them + the next pod
start (or one manual `occ group:adduser admin <uid>`). The LOCAL admin stays as
break-glass (docs/15).
apps/wave-1/nextcloud.yaml re-rendered from catalog (single-file render; byte-diff
confined to the embedded hook). Rendered hook passes `sh -n`; parse step validated
in-pod. `make -C ci gate` green (lint/l0/l1; conformance + spec-immutability skip:
no cluster / spec repo unmounted).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The plexinc/pms-docker image writes its identity/claim token and the
Library metadata DB under /config, but the config PVC was mounted at the
renderer default /data/config. Plex therefore wrote all state to the
container's EPHEMERAL layer: the server stayed UNCLAIMED (servers.plex.tv
returns 401 with an empty X-Plex-Token) and lost everything on restart,
which is why seerr's "add Plex" failed even with SSL off.
Set the config volume mountPath to /config in catalog/plex.yaml and
re-render apps/wave-0/plex.yaml (only the mountPath line changes; all
other services byte-identical). No fsGroup/securityContext needed:
verified in-cluster the image inits as root (s6-svscan PID 1), chowns
/config to the plex user, then drops privileges (the "Plex Media Server"
process runs as plex) — root can chown the mounted PVC regardless of
fsGroup.
PLEX_CLAIM auto-claim is intentionally NOT wired: the env secretKeyRef
schema has no `optional`, so a placeholder/absent claim Secret would
crashloop the pod. The claim is an operator step (UI sign-in after the
roll, or add a PLEX_CLAIM env from a fresh https://plex.tv/claim token).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The Keycloak OIDC postStart registration for gitea and nextcloud ran but
never persisted, leaving `gitea admin auth list` empty and nextcloud's
user_oidc with "No providers configured" despite the apps installing/
enabling and the client secret being present.
Two distinct root causes, confirmed by running the registration manually
against the live warm pods:
- nextcloud: a QUOTING BUG, not a race. The OCC() wrapper runs
`su -c "php occ $*"`, which word-split `--scope="openid email profile"`
back into three tokens, so occ saw `email profile` as extra positional
args -> "Too many arguments to user_oidc:provider" -> exit 1, swallowed
by `|| echo failed`. Failed on EVERY boot. Fixed by running the provider
call as ONE `su -c` with the scope single-quoted so www-data's shell
keeps it a single argument; keep the OCC() wrapper only for single-token
subcommands.
- gitea: a first-boot RACE. The direct `admin auth add-oauth` persists once
the app/network is warm (verified live), but fired too early at postStart
and its failure was swallowed (`|| echo ... >&2`, exit 0).
Hardening applied to both catalog sources (rendered into apps/wave-*):
- wait (bounded ~90s) for the Keycloak discovery doc to return HTTP 200;
- nextcloud also waits for `installed: true` AND `maintenance: false`;
- register idempotently (create-or-update) then VERIFY the source/provider
actually persisted, retrying the whole cycle (guards exit-0-but-nothing-
stuck);
- on give-up, log LOUDLY to stderr AND a marker file on the persistent
volume (/data, /var/www/html) instead of swallowing — next failure is
visible;
- never crashloop: missing-secret guard retained, break-glass local admin
stays usable.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Extends the SSO work to the two remaining `auth: oidc` apps still on local-admin
logins, each as a native confidential client of the Keycloak `homelab` realm with
its local admin kept as break-glass (docs/15 availability caveat; R-IDP-02/06).
Realm (source of truth): add the `nextcloud` and `semaphore` confidential clients
to platform/keycloak/keycloak-realm-import.yaml (modeled on grafana/gitea) + their
NEXTCLOUD_CLIENT_SECRET / SEMAPHORE_CLIENT_SECRET placeholders. As before, a
KeycloakRealmImport no-ops on an already-existing realm, so both live clients were
ALSO created via kcadm (nextcloud e95049ed-ca10-4085-97d9-4bbef495fe61, semaphore
1835bfe9-b125-4026-8f69-33557519cf4a); the YAML keeps git authoritative.
Secrets (SOPS/age): add nextcloud-client-secret + semaphore-client-secret to
keycloak-realm-clients.sops.yaml (keycloak ns). Mirror the nextcloud secret as
oidc-client-secret into nextcloud-bootstrap (nextcloud ns). For semaphore, the
whole SEMAPHORE_OIDC_PROVIDERS JSON (which embeds the client secret) is held
encrypted as oidc-providers-json in semaphore-bootstrap (semaphore ns). All values
verified == the live clients; existing keys untouched; no plaintext in git.
App config (catalog source + re-render):
- nextcloud: add NEXTCLOUD_OIDC_CLIENT_SECRET env + an idempotent postStart hook
that installs/enables user_oidc and runs `occ user_oidc:provider keycloak`
(create-or-update; secret via --clientsecret-env so it never hits argv; occ as
www-data; group provisioning on the `groups` claim). Syntax verified against the
running user_oidc 8.11.0 on Nextcloud 31.0.14; su-to-www-data env passthrough
verified. Missing-secret/not-installed guards prevent crashloops.
- semaphore: add SEMAPHORE_OIDC_PROVIDERS env sourced from the encrypted
oidc-providers-json (provider_url=realm issuer/auto-discovery, redirect
/api/auth/oidc/keycloak/redirect). Mechanism verified against semaphore v2.19.10
source (util ConfigType oidc_providers env tag + OidcProvider struct + router
redirect path); no kubelet service-link env collision.
apps/wave-1/{nextcloud,semaphore}.yaml re-rendered via
`render.py --no-image-automation` (diffs are env/lifecycle-only).
Verified: realm import Done=True/HasErrors=False and platform-config
Ready=True/Healthy=True (unchanged); edited realm YAML passes server dry-run; live
homelab clients now include gitea/nextcloud/semaphore; static gate (lint/l0/l1)
green. Browser SSO login for each still needs a human check.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Wires the bootstrapping-critical Gitea git host to the Keycloak `homelab` realm
as a native confidential OIDC client, moving it off the local-admin-only login
while keeping the local admin as the break-glass path (docs/15 availability
caveat; R-IDP-02/06).
Realm (source of truth): add the `gitea` confidential client to
platform/keycloak/keycloak-realm-import.yaml (modeled on grafana) + its
GITEA_CLIENT_SECRET placeholder sourced from the keycloak-realm-clients Secret.
NOTE: a KeycloakRealmImport no-ops on an already-existing realm, so the live
client was ALSO created via kcadm (id 70a6ca60-317a-42ce-8331-a60e380838ee);
the YAML keeps git authoritative for a rebuild.
Secrets (SOPS/age, never plaintext): add gitea-client-secret to
platform/keycloak/keycloak-realm-clients.sops.yaml (keycloak ns) and mirror the
same value as oidc-client-secret into apps/wave-0/gitea-bootstrap-secret.sops.yaml
(gitea ns) so the app can read it in-namespace. Both == the live client secret.
App config (catalog source + re-render): catalog/gitea.yaml gains the
GITEA_OIDC_CLIENT_SECRET env (from the mirrored Secret), [oauth2_client]
auto-registration/account-linking app.ini env, and an idempotent postStart step
that registers the "keycloak" OAuth2 login source (auto-discovery; groups claim;
platform-admins -> Gitea admin). apps/wave-0/gitea.yaml re-rendered via
`render.py --no-image-automation`.
Verified: realm import Done=True/HasErrors=False and platform-config
Ready=True/Healthy=True (unchanged); edited realm YAML passes server dry-run;
static gate (lint/l0/l1) green. Browser SSO login still needs a human check.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Seerr v3.4.1 (ghcr.io/seerr-team/seerr) runs as the non-root user node:node
(UID/GID 1000), unlike the old root-running sctx/overseerr image. Its root-owned
Longhorn RWO config PVC at /app/config was not writable by UID 1000, so the pod
CrashLoopBackOff'd with:
Error: EACCES: permission denied, mkdir '/app/config/logs/'
Add a podSecurityContext with fsGroup=1000 (plus runAsUser/runAsGroup/runAsNonRoot)
to the seerr ServiceDefinition — the same schema 1.3.0 mechanism semaphore uses.
Longhorn's CSIDriver fsGroupPolicy: ReadWriteOnceWithFSType applies fsGroup to this
RWO filesystem volume, group-owning the PVC to GID 1000 so the process can write.
Re-rendered apps/wave-2/seerr.yaml from the catalog source.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Two operator-flagged defects in the seerr ServiceDefinition, fixed at source
(catalog/seerr.yaml) and re-rendered into apps/wave-2/seerr.yaml:
- Image: sctx/overseerr:latest (original Overseerr, unmaintained) ->
ghcr.io/seerr-team/seerr:v3.4.1. Overseerr and Jellyseerr have merged into
the actively maintained Seerr project (seerr-team/seerr, docs.seerr.dev);
ghcr.io/seerr-team/seerr is the officially recommended image. Pinned to the
current stable tag v3.4.1 per the arr-stack convention (sonarr/radarr/
qbittorrent/gitea pin explicit tags; R-UPD-01). update.range bumped
">=1.33.0" -> ">=3.4.0" to track the Seerr 3.x line. Plex-native (native
auth) support is retained.
- Config mountPath: /data/config -> /app/config. Overseerr/Jellyseerr/Seerr
store settings + request/cache DB at /app/config; the previous /data/config
left the config PVC effectively unused (config not persisted across
restarts). Set via the schema-supported storage[].mountPath field.
Dependencies (sonarr:8989, radarr:7878, plex:32400), NetworkPolicies,
public exposure, and native auth are unchanged. Render == golden verified;
lint, schema, and L1 render checks green.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The docs had drifted badly behind ~85 commits of delivery. The root README still
said "Phase 0 — Foundations. No cluster exists yet"; the infra docs still
described a libvirt/KVM cluster that has been retired for two days; and
DECISIONS-LOG.md — the PM's authoritative record — stopped at 2026-08-25 AM.
Anyone following these docs would have been working from a false map.
Verified against the live cluster (3/3 nodes, 7/7 Flux Kustomizations Ready,
10/10 HelmReleases Ready) rather than inferred from commit messages.
README.md
- Phase 0 -> a per-phase status table: 0-6 delivered and deployed, 7 not open.
- Layout now shows the wave split, platform-config, apps/vms, infra, bootstrap.
- Real conformance number (36/51, 70.6%) and how to regenerate it.
- Devcontainer workflow; note that the gate was red and why.
infra/PHASE1-STATUS.md — rewritten. Was a `virsh`/libvirt resume guide for a
cluster that no longer exists. Now: the Proxmox substrate, the pfSense NAT
paths, both kubectl access paths (Keycloak OIDC normal / cert break-glass),
the toolchain, off-repo state that must not be lost, and the fact that Flux's
git source is an app inside the cluster (`git push origin main` IS the deploy).
infra/PHASE1-SCOPE.md — historical banner. The exit criteria and sizing
rationale are still worth keeping; the libvirt execution steps are not.
infra/README.md — Proxmox, plus a map of the subtree and where the cluster
actually lives (it is not on this workstation).
infra/devcontainer/README.md — the "run tofu and ansible on the HOST" limitation
is OBSOLETE: it was a dmacvicar/libvirt polkit/PID problem, and bpg/proxmox
talks to an HTTP API. Everything runs in the container now. Documents the full
CI-gate toolchain and what each bind mount is load-bearing for (the age keyring
and the read-only spec repo especially — without the latter, spec-immutability
silently skips instead of verifying).
infra/tofu/README.md — was "Authoring only — nothing here has been applied". It
is applied and live. Disk sizes corrected 40/100 -> 64/128 GiB (confirmed
against the running nodes), pfSense NAT marked done, and a warning that
terraform.tfstate is absent on a fresh workstation — an apply with empty state
would try to recreate the three live nodes.
infra/tofu/variables.tf — section comment disagreed with its own defaults.
infra/cluster.sh — retired-with-instructions header. It silently drives libvirt
domains that do not exist; behaviour unchanged, but it no longer lies.
apps/vms/unifi/README.md — was "NOT WIRED IN", 60Gi, and listed a file that has
since been dropped. Now: live status, 40Gi, resolved open questions marked
resolved, and the two constraints the next person will otherwise rediscover —
KubeVirt's 2048B inline userData cap, and that apps-vms must never carry
postBuild.substituteFrom (envsubst corrupts the cloud-init).
DECISIONS-LOG.md — backfilled the gap as one structured entry: the Proxmox
migration, NAT + kube-vip, the twice-proven oauth2-proxy cold-start deadlock,
the Flux source cutover, CI going live in-cluster, ten app onboardings and the
renderer bugs they exposed, the wave split (verified: DBs adopted, not
recreated — original ages intact, prune restored), KubeVirt storage, the UniFi
VM, and kubectl SSO. Plus verified live state and the carried-forward open list.
Two corrections to the record while verifying:
- kubectl OIDC is LIVE (`kubectl auth whoami` -> oidc:<user>,
[oidc:platform-admins]), so the rolling restart 3472f1d deferred has
happened.
- UniFi OS Server is NOT installed in the guest. The VM is healthy (Ubuntu
24.04.4, guest agent connected, live-migratable) but :11443 answers nothing,
and the CNP explicitly permits cluster -> :11443, so it is genuinely not
listening rather than a policy artifact.
Docs and comments only; no manifest semantics change.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The hand-authored UniFi VM manifests used column-aligned inline flow mappings
(`{ name: inform, port: 8080, protocol: TCP }`), which yamllint's `commas`
rule rejects as "too many spaces after comma". That made `make -C ci lint` FAIL,
so the merge gate has been BLOCKED on `main` since the UniFi commits landed —
every other static stage was green.
Collapse each run of spaces after a comma to one, and re-align the trailing
comments so the port tables stay readable. Whitespace only:
`kubectl kustomize apps/vms` renders byte-for-byte identically before and after.
The lint config is left untouched — relaxing the rule to make failing files pass
is moving the goalposts, not fixing the code.
Also refresh these files' stale `STAGING / DRAFT` headers: the tree is live,
reconciled by the non-gating `apps-vms` Flux Kustomization.
Gate after this change: spec-immutability / lint / l0 / l1 all green,
MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
760084e only committed the unifi-cloudinit.yaml deletion — the `git add` errored
on that already-staged pathspec and skipped the other two files, leaving the
kustomization still referencing the deleted file (broken build). Commit the
intended changes: 30-virtualmachine.yaml inline userData + kustomization.yaml
without the secretGenerator.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
userDataSecretRef tripped a Flux server-side-apply schema quirk on the KubeVirt
VirtualMachine CRD ("field not declared in schema"). Go back to inline userData,
trimmed to 1112 bytes (< the 2048 limit) by dropping the break-glass SSH key —
no longer needed for debugging now that the postBuild cloud-init corruption is
fixed. Keeps qemu-guest-agent (AgentConnected proves cloud-init ran) and the
public-API UOS resolver + non-interactive install. Removes the secretGenerator +
unifi-cloudinit.yaml. (Add SSH back via userDataSecretRef only if the SSA quirk is
worked around and guest access is needed.)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Adding the SSH key pushed the inline cloudInitNoCloud.userData over KubeVirt's
2048-byte limit ("userdata exceeds 2048 byte limit. Should use UserDataSecretRef").
Move the cloud-init into apps/vms/unifi/unifi-cloudinit.yaml and wrap it as the
`unifi-cloudinit` Secret via a Kustomize secretGenerator (disableNameSuffixHash
so the VM can reference it by fixed name); VM now uses userDataSecretRef. Content
is non-sensitive (public key + public installer script), so not SOPS-encrypted.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The first UOS bring-up left no way into the guest (cloud-init set no
credentials), so the in-guest UOS install couldn't be observed. Add:
- ssh_authorized_keys (public key, safe to commit) on the default ubuntu user
- qemu-guest-agent (KubeVirt AgentConnected + virtctl guest access)
- tee the install-uos output to the serial console + /var/log/uos-install.log,
and record the installer exit code, so the install is observable.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The UniFi Block import got past the device-permission fix but then failed
ReplicaSchedulingFailure / "insufficient storage": CDI sizes scratch ~= the disk
virtual size and, on the default 3-replica `longhorn` class, the ~60Gi scratch x3
(189G) on top of the disk's x3 (180G) overflowed the Longhorn tier (~125G/node).
- platform/longhorn/storageclasses.yaml: NEW `longhorn-scratch` StorageClass
(numberOfReplicas: 1) — CDI scratch is ephemeral/throwaway, replicating it 3x is
waste; if a node dies mid-import CDI just retries.
- platform/kubevirt/cdi-cr.yaml: spec.config.scratchSpaceStorageClass ->
longhorn-scratch.
- apps/vms/unifi/20-datavolume.yaml: boot disk 60Gi -> 40Gi (UOS min is 25Gi;
40Gi is comfortable for a home site and lighter on the replicated tier).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The UniFi VM boot-disk DataVolume (RWX + Block on longhorn-vm) failed to import:
CDI's non-root importer (UID 107, cap-drop ALL) hit
`blockdev: cannot open /dev/cdi-block-volume: Permission denied`.
Kubernetes only applies fsGroup ownership to *filesystem* volumes — it never
chowns a raw block device — and Longhorn's CSIDriver (fsGroupPolicy:
ReadWriteOnceWithFSType) skips fsGroup on RWX/Block volumes regardless, so the
device stays root:root. The CDI-documented fix is the container-runtime setting
`device_ownership_from_security_context`, exposed on k3s (>= v1.31.3+k3s1, this
cluster is v1.31.5) as `nonroot-devices: true`. It makes containerd set the
in-container device node's owner from the pod's securityContext.
Added to the Ansible-managed /etc/rancher/k3s/config.yaml alongside tls-san;
applied via the existing rolling `Restart k3s` handler (serial:1, kube-vip keeps
the API up, etcd holds 2/3). Node-scoped, security-neutral (pods stay non-root),
and preserves the longhorn-vm RWX/Block/migratable contract unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
First KubeVirt VM app + first real CDI http import. UniFi OS Server (UOS) is
Ubiquiti's current self-host product (the .deb is deprecated); it bundles UniFi
OS + Network app + its own datastore, so no MongoDB companion.
- apps/vms/unifi: CDI DataVolume (Ubuntu 24.04 cloud image -> longhorn-vm
RWX/Block, live-migratable) + VirtualMachine (evictionStrategy: LiveMigrate)
with cloud-init that installs podman/slirp4netns/jq and runs the UOS installer
NON-INTERACTIVELY. The installer URL is resolved at boot from Ubiquiti's public
firmware API (fw-update.ui.com/api/firmware-latest) — always-latest, no pinned
URL to maintain (currently UOS v5.1.37).
- L4 exposure: two LoadBalancer Services (TCP+UDP) sharing a pinned, L2-announced
Cilium VIP 192.168.1.54 (device inform/adoption + STUN can't ride the L7
Gateway); external-dns publishes unifi.home.keithjackson.dev. UI on :11443.
- default-deny + additive CNPs (controller ingress, egress, and CDI-importer
egress for the image pull).
- clusters/base/apps-vms.yaml: NON-GATING Flux Kustomization (wait:false,
dependsOn platform-config) — VMs are a separate lifecycle from the app waves,
so a slow VM import/install can't stall the app tier.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Replaces the single `apps` Flux Kustomization with dependency-ordered waves
(apps-wave-0..2) so a crashlooping workload isolates to its wave + downstream
instead of stalling the entire app tier. Parallel within a wave, ordered across.
- catalog/render.py: compute waves by longest dependency-path depth over the
ServiceDefinition `dependencies:` edges (a `<svc>-db` is a leaf, so its
consumer lands strictly higher — the "DB below consumer" rule falls out of
longest-path, no special case). `--emit-waves` generates the per-wave
kustomizations + Flux CRs; `--wave-layout` places renders into the right dir.
wave 0: pihole gitea jackett plex qbittorrent nextcloud-db semaphore-db
wave 1: sonarr radarr nextcloud semaphore gitea-actions
wave 2: seerr
- apps/*.yaml -> apps/wave-{0,1,2}/ (byte-identical moves; SOPS files intact).
- clusters/base: apps.yaml removed; apps-wave-{0,1,2}.yaml added (dependsOn
chain: platform-config -> wave-0 -> wave-1 -> wave-2, prune+wait each).
- clusters/dev: one SOPS decryption patch per wave (load-bearing).
- DESIGN-waves.md: wave table, topology, cutover runbook.
Cutover: preceded by disabling prune on `apps` (1bfb0db) so removing that
object here does not GC its resources; the wave Kustomizations SSA-adopt the
byte-identical manifests in place (no restart, no PVC churn).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Restores the four semaphore resources a peer temp-disabled in 0abadd3 to unblock
the apps Kustomization while the crashloop was diagnosed. Both root causes are now
fixed and verified against the live image:
1. permission-denied writing /etc/semaphore/config.json (non-root UID 1001 vs
root-owned Longhorn PVC) -> podSecurityContext.fsGroup=1001 (schema 1.3.0);
verified the pod got past config write and successfully pinged its PostgreSQL.
2. config panic on SEMAPHORE_PORT=tcp://<svcIP>:3000 (kubelet Docker-link
injection for the same-named Service colliding with Semaphore's listen port)
-> explicit SEMAPHORE_PORT=":3000" env shadowing the service-link var.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
With the config PVC now writable (fsGroup), semaphore got past config write and
pinged its PostgreSQL successfully, then crashed on:
panic: value of field 'Port' is not valid: tcp://<svcIP>:3000
Because the Service is named `semaphore`, the kubelet injects the Docker-link-style
SEMAPHORE_PORT=tcp://<clusterIP>:3000, which collides with Semaphore's OWN
SEMAPHORE_PORT (its HTTP listen port). Add an explicit SEMAPHORE_PORT=":3000" env
entry — user-defined env takes precedence over service-link vars — pinning the
listen port back to :3000 (matches containerPort/Service). Uses the existing env
field; only apps/semaphore.yaml changes, all others byte-identical.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The semaphoreui/semaphore image falls through to its interactive setup wizard
and panics on /etc/semaphore/config.json permission-denied; the env-only pattern
doesn't yield a non-interactive config. Backed out to unblock the apps
Kustomization; files retained for a proper config-model follow-up.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The semaphoreui/semaphore image runs as non-root UID 1001 / GID 0 and writes its
generated config.json into the config dir on boot. A mounted Longhorn PVC is
root:root, so UID 1001 got "permission denied" and the pod crash-looped.
First-class the fix into the render contract (same mechanism the retired
gitea-bootstrap-patch was replaced by): schema 1.3.0 adds an optional pod-level
`podSecurityContext`, emitted verbatim onto spec.template.spec.securityContext by
render.py. Absent => every other app renders byte-identical (verified). semaphore
declares fsGroup=1001 (kubelet group-owns the PVC + adds the supplemental group so
UID 1001 can write) plus runAsUser/runAsGroup/runAsNonRoot pinning the image identity.
render.py selftest extended (verbatim render + absence). L0 (schema+kyverno 17/17),
L1 3/3, lint all green.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Semaphore (Ansible UI, docs/13 row 10): internal exposure, internet egress,
config on Longhorn + external PostgreSQL, auth oidc. Reuses the CNPG-DB pattern
nextcloud established (apps/nextcloud-db.yaml, commit 4be6752).
- catalog/semaphore.yaml: ServiceDefinition (schema >=1.2.0 env) wiring the
SEMAPHORE_DB_* connection to semaphore-db-rw.semaphore-db:5432, break-glass
SEMAPHORE_ADMIN* local admin, and the FIXED base64 SEMAPHORE_ACCESS_KEY_ENCRYPTION.
- apps/semaphore.yaml: rendered (render.py --no-image-automation) — Deployment,
config PVC, Service, default-deny + egress/gateway/internet CNPs, HTTPRoute
semaphore.home.keithjackson.dev:3000, ServiceMonitor, SLI alert.
- apps/semaphore-db.yaml: CNPG Cluster in dedicated ns semaphore-db, initdb from
the owner secret, inheritedMetadata app.kubernetes.io/name=semaphore-db so the
dep-edge egress CNP matches.
- apps/semaphore-db-app.sops.yaml + apps/semaphore-bootstrap-secret.sops.yaml:
SOPS(age) owner creds (basic-auth) + mirrored db-password, break-glass admin,
and the persistent access-key encryption key.
OIDC (Keycloak realm client) is DEFERRED like gitea/nextcloud — comes up on the
break-glass local admin; realm-client wiring is the batched follow-up.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Onboard nextcloud (docs/13 row 7): public, DR-critical file sync backed by an
external PostgreSQL. Establishes the reusable CNPG-DB pattern for later DB-backed
services.
- catalog/nextcloud.yaml: ServiceDefinition (image nextcloud:31-apache :80, public,
internet, data 20Gi longhorn @/var/www/html, backup daily-offcluster, auth oidc
+platform-admins, ingress nextcloud.home.keithjackson.dev, dependency
nextcloud-db:5432). Uses schema 1.2.0 env for the Postgres connection, break-glass
admin, trusted-domain + OVERWRITE* proxy settings.
- apps/nextcloud-db.yaml: CloudNativePG Cluster in a dedicated nextcloud-db namespace
(modeled on keycloak-db; no default-deny, mirroring the verified keycloak posture),
inheritedMetadata stamps app.kubernetes.io/name=nextcloud-db so the rendered egress
CNP matches. initdb consumes a pre-provisioned owner secret.
- apps/nextcloud-db-app.sops.yaml + apps/nextcloud-bootstrap-secret.sops.yaml: SOPS
(age) — shared DB password mirrored across the two namespaces (Pods cannot read a
Secret cross-namespace), plus the break-glass local admin.
- apps/nextcloud.yaml: rendered manifests (catalog/render.py --no-image-automation).
- apps/kustomization.yaml: wire in the four new resources.
OIDC realm-client wiring DEFERRED (WS-APPS step-3 escape hatch): editing the reconciled
realm import in platform-config risks the running Keycloak realm / whole platform-config
apply, and nextcloud OIDC needs post-install occ steps with no first-class env — nextcloud
comes up INSTALLED on the break-glass local admin (as Gitea did); the Keycloak client is a
follow-up. Off-cluster backup awaits Velero/OQ-BKP (intent declared).
Gates: render --selftest PASS; other apps byte-identical; ci l0 17/17; l1 3/3; lint clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Extend the ServiceDefinition contract with optional per-container `env`,
`envFrom`, and `lifecycle` fields (k8s EnvVar / EnvFromSource / Lifecycle
shapes) so headless-install / DB-backed apps configure themselves through
the SD instead of a supplementary kustomize patch.
- schema: bump 1.1.0 -> 1.2.0; add optional env/envFrom/lifecycle to the
container kind (literal + secretKeyRef/configMapKeyRef; secretRef/
configMapRef; postStart/preStop exec via $defs/execHandler). Existing
fields unchanged; all fields optional -> absent renders byte-identical.
- render.py: emit env/envFrom/lifecycle verbatim onto the Deployment app
container when present; --selftest coverage for the new fields.
- catalog/gitea.yaml: fold the headless-install env (INSTALL_LOCK, fixed
SECRET_KEY via secretKeyRef, DOMAIN/ROOT_URL, actions, break-glass admin
creds) + the idempotent postStart break-glass admin bootstrap inline.
- apps/gitea.yaml: re-rendered with env/lifecycle inline (effective pod
spec provably equivalent to base+patch).
- DELETE apps/gitea-bootstrap-patch.yaml + drop it from apps/kustomization
patches. gitea-bootstrap-secret.sops.yaml kept (still sourced by the env).
Proves the renderer feature by retiring the known TODO; unblocks env + DB
onboarding for nextcloud/semaphore/unifi/minecraft. All other rendered apps
byte-identical. Gates: selftest PASS, l0 17/17 + schema, l1 goldens, lint.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Follow-up to the reciprocal-ingress fix: in-cluster verification (Hubble) revealed
the east-west path was STILL fully blocked — and that the pre-existing EGRESS half
was also broken, contrary to the original diagnosis. Root cause: a CiliumNetworkPolicy
`toEndpoints`/`fromEndpoints` selector is scoped to the POLICY's OWN namespace unless
it carries the reserved `k8s:io.kubernetes.pod.namespace` label. Every service lives
in its own namespace (== its name) and every dependency crosses a namespace boundary,
so an app-label-only peer selector matched nothing and Cilium silently denied the flow
(Hubble: sonarr->qbittorrent = "EGRESS DENIED ... Policy denied DROPPED").
Pin the peer namespace on BOTH halves — egress toEndpoints (destination ns == dep.to)
and the reciprocal ingress fromEndpoints (source ns == source name). This is a strict
TIGHTENING (more precise, still L3/L4), not a weakening. Verified end-to-end in-cluster:
sonarr->qbittorrent:8080, sonarr->jackett:9117, seerr->{sonarr:8989,radarr:7878,
plex:32400} now CONNECT (were DROPPED); undeclared sonarr->plex:32400 and
seerr->qbittorrent:8080 stay BLOCKED (default-deny holds).
Selftest + L0 (17/17) + L1 (3/3, goldens byte-identical) + lint all pass. NOTE: this
necessarily touches the egress toEndpoints selector, which the task scoped as
ingress-only; the premise that egress already worked was empirically false.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A `dependencies` edge is declared ONCE on the SOURCE, but with the default-deny-
BOTH-directions baseline (docs/03 R-NET-01/02) it needs TWO matching halves: the
source egress-allow AND the destination ingress-allow. render.py emitted only the
egress half, so every declared east-west dependency was silently dropped by the
destination's default-deny ingress — verified in-cluster: sonarr->qbittorrent:8080,
sonarr->jackett:9117 and seerr->sonarr:8989 were all BLOCKED.
Add a cross-service reverse-dependency pass (build_reverse_map) and render the
reciprocal ingress half (r_reverse_dependency_policies): for each source that
declares a dependency on X, X gains an additive ingress-allow CiliumNetworkPolicy
`X-allow-<source>-<port>` whose fromEndpoints matches the source identity, mirroring
the egress half's L3/L4 granularity. Deterministic (sorted), additive (default-deny
ingress stays []), egress side unchanged. A lone render with no cross-catalog context
renders no reciprocal ingress, so the L1 render goldens are byte-identical.
New ingress CNPs: qbittorrent<-{sonarr,radarr}:8080, jackett<-{sonarr,radarr}:9117,
sonarr<-seerr:8989, radarr<-seerr:7878, plex<-seerr:32400. Selftest + L0 (17/17) +
L1 (3/3) + lint all pass.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
- install-tools.sh: the kyverno CLI release tarball is named _linux_x86_64, not
_linux_amd64 (mirrors infra/devcontainer/Containerfile). The gate job's kyverno
install 404'd on amd64; map amd64 -> x86_64 for kyverno only.
- act-runner: wait for /certs/client/ca.pem (published by the DinD sidecar) before
exec-ing the entrypoint, eliminating the startup crash-loop race.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
STEP 1 — enable Gitea Actions: pin GITEA__actions__ENABLED=true and
DEFAULT_ACTIONS_URL=github in the gitea bootstrap patch (was implicit upstream
default; make it a declared property).
STEP 2 — deploy + register an act_runner in a dedicated non-app namespace
(gitea-actions) so the privileged DinD executor sidecar is admitted without a
Kyverno policy exception (restrict-host-namespaces scopes to app namespaces only).
Docker executor + DinD sidecar over a shared TLS-cert emptyDir; runner state on a
Longhorn PVC (registers once). Registration token delivered as a SOPS-encrypted
Secret decrypted by Flux. Runner mounts no ServiceAccount token (R-SEC-02).
STEP 3 — author .gitea/workflows/gate.yml: the live enforced merge gate running
the STATIC layers (spec-immutability best-effort, lint, Kyverno L0, L1 render) via
`make -C ci lint l0 l1` with the pinned toolchain installed at job start. L2/L3
conformance is an if:false placeholder (needs a toolchain image + scoped cluster
creds — a design decision, not wired). merge-gate.yaml reduced to workflow_dispatch
(dormant full-pipeline blueprint). install-tools.sh gains check-jsonschema so the
schema stage validates in CI instead of skipping.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Any single-replica app with a `storage:` entry mounts a ReadWriteOnce Longhorn
PVC. The default RollingUpdate strategy starts the new pod before terminating the
old one, but the old pod still holds the RWO volume -> the new pod hangs in
ContainerCreating and the rollout deadlocks (observed on gitea's update). Emit
spec.strategy.type=Recreate whenever storage is present so the old pod releases
the PVC first. Non-storage apps (and VMs) keep the default (no strategy emitted).
Re-rendered apps: gitea, pihole, sonarr, radarr, plex, qbittorrent, jackett now
carry strategy: Recreate. Updated the egress-class L1 render goldens (storage
fixture) accordingly; all L1 render checks pass.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Two defects blocked in-cluster Gitea from becoming the Flux source:
1. Storage mountPath: the renderer hard-coded /data/<name>, so Gitea's PVC
mounted at /data/data while Gitea writes all state to /data — real repos/DB
landed on the ephemeral rootfs. Add an OPTIONAL `mountPath` to the storage
schema + render.py (default stays /data/<name>, back-compat). catalog/gitea.yaml
sets data.mountPath: /data. Only gitea.yaml's rendered mount changes; every
other app renders byte-identical.
2. Headless install: the instance was un-installed (INSTALL_LOCK=false, ephemeral
SECRET_KEY, no admin, API 404). A Gitea-only supplementary manifest
(apps/gitea-bootstrap-patch.yaml) patches the rendered Deployment with
GITEA__security__INSTALL_LOCK=true, a FIXED SECRET_KEY from a persistent SOPS
secret (apps/gitea-bootstrap-secret.sops.yaml), ROOT_URL/DOMAIN, and a postStart
hook that idempotently creates a break-glass local admin. State persists on the
now-correct /data Longhorn PVC.
The env/lifecycle wiring is a supplementary kustomize patch because the
ServiceDefinition schema/renderer does not yet express env/envFrom/lifecycle —
noted as a renderer follow-up (TODO in the patch).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Root cause the media stack surfaced: the default-deny baseline is DNS-only egress (correct for
R-NET-01 east-west), but the egress CLASS (internet/vpn) had no policy ALLOWING north-south
egress — so qbittorrent/sonarr/radarr couldn't reach the internet at all (VPN downloaders
broken). Add an additive per-app CiliumNetworkPolicy '<app>-allow-internet-egress' with
toEntities:[world] (non-cluster); the egress-class steering + kill-switch (ADR-0015) govern the
PATH (internet=WAN, vpn=tun0). East-west stays denied except declared dependencies; R-NET-01
baseline unchanged. Re-rendered all apps + regenerated L1 goldens. Also fixed jackett + plex
image tags (bad pins -> :latest; Renovate/update policy pins later).
The 39Gi/node Longhorn data disks at 1:1 over-provisioning can't fit the media stack
(plex 20Gi replica failed: 'insufficient storage'). Volumes are thin-provisioned config-only,
so 200% lets the full catalog fit without a disk resize; physical fill risk is low. Also
right-sized plex config (20Gi was excessive for metadata). Prod-grade path: grow the data
disks (WS-STORAGE follow-up).
Add catalog/ ServiceDefinitions for qbittorrent and jackett and their
rendered apps/ manifests, wired into apps/kustomization.yaml.
Both are VPN-class (egress: vpn -> egress-class=vpn pod label steered by the
CiliumEgressGatewayPolicy + host kill-switch, ADR-0015) and internal
forward-auth UIs (oauth2-proxy -> Keycloak, group media-users). Config on
3-replica Longhorn. qBittorrent's bulk downloads volume is DEFERRED pending
OQ-2 (non-provisioning bulk StorageClass stub), mirroring plex/arr.
Traces: R-NET-05/06 (VPN egress + kill-switch), R-IDP-02/06 (forward-auth),
R-STATE-02/03, R-OBS-01, R-UPD-01/03, R-GITOPS-02.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add catalog/plex.yaml (kind: container) and its rendered apps/plex.yaml,
and wire apps/kustomization.yaml. Plex = docs/13 row 1, the catalog's
largest service; validates the public-exposure + config-on-Longhorn path.
- image plexinc/pms-docker (pinned recent stable tag), port 32400
- exposure: public at plex.home.keithjackson.dev; egress: internet
- auth: native (Plex accounts, not Keycloak — docs/13/15 exception)
- config PVC 20Gi on longhorn; off-cluster backup (daily-offcluster) as the
longhorn tier forbids reacquirable under L0 (R-STATE-03)
- observability up-SLI; update channel stable
Bulk media (R-STATE-04) deferred: the `bulk` StorageClass is a documented
stub (OQ-2 unresolved, NFS CSI not installed) so a bulk PVC would leave the
pod Pending. Media mount omitted; attach once bulk provisions.
Traces: R-GITOPS-02, R-NET-01/02/05, R-STATE-01/02/03, R-TLS/R-DNS,
R-OBS-01, R-UPD-01/03, R-IDP-02, R-PLAT-02.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Exposed apps got a default-deny NetworkPolicy with ingress: [] (deny all), so the
Cilium Gateway (Envoy, reserved identity `ingress`) could not reach the backend — an
Accepted HTTPRoute was still unreachable. render.py now emits an ADDITIVE, least-
privilege CiliumNetworkPolicy for any ServiceDefinition that renders an HTTPRoute:
ingress:
- fromEntities: [ingress] # the shared Cilium Gateway/Envoy proxy identity
toPorts: [<exposed service port(s)> TCP]
Verified against the live cluster (Cilium 1.16.5): with default-deny in place, Hubble
shows the Gateway proxy reaching pihole as source identity `(ingress)` and the flow
`Policy denied DROPPED` on :80 — so `fromEntities: [ingress]` is exactly the source the
Gateway uses, scoped to the service targetPort. Default-deny stays intact (ingress: []
unchanged); the allow only opens the north-south path (docs/09), east-west stays denied.
- render.py: add r_gateway_ingress_policy(); emit it after dependency policies; selftest
asserts the allow is additive (default-deny ingress stays []), least-privilege (only the
exposed port, only the ingress identity), and absent for non-exposed services.
- L1 goldens updated for the two exposed fixtures (container-workload, egress-class);
default-deny-baseline golden unchanged (no exposure). --selftest + run.sh --layer l1 green.
- apps/pihole.yaml re-rendered (--no-image-automation) to carry the allow.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
First app onboarded through catalog/render.py -> apps/. Added --no-image-automation so apps
deploy before the Flux image controllers exist (L1 goldens keep the default, unchanged).
KNOWN GAP surfaced: exposed apps get default-deny (ingress: []) with no gateway-ingress allow,
so the HTTPRoute path is blocked — renderer/WS-NET follow-up to allow Cilium gateway ingress
to services with an exposure/HTTPRoute.
Phase 0 foundation shell (read-only spec at ../home-network-k3s). Top-level
tree + rules; workstream agents fill their subtrees on feature branches.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
jakman85andClaude Opus 4.8