plex, sonarr, radarr and nextcloud each gain one bulk volume. Authored in the catalog and
re-rendered, never hand-edited into apps/ (render-golden: 10/10). Layout, the export change the
operator must apply, the uid analysis and the validation probes are in docs/BULK-NFS-STORAGE.md;
everything there was measured on the host over read-only SSH, not assumed.
WHAT THE LAYOUT TURNED OUT TO BE, AND WHY IT DECIDED THE DESIGN
/mnt/Storage/plex holds movies/ tv/ photos/ AND downloads/ as SIBLINGS in one dataset — so the
arrs' import source and their libraries are one filesystem on the server, and the download client's
completed folders are already named radarr/ and tv-sonarr/. The k3s arrs are joining a tree an arr
stack has been driving for a long time.
That sibling fact is the whole argument for the arrs mounting the export ROOT rather than a
subPath per directory. Sonarr and Radarr import by hardlinking or atomically renaming, both
same-filesystem operations, and to the kernel "same filesystem" means SAME MOUNT. Two subPath
mounts of one export are two mounts: link() across them returns EXDEV regardless of what the
server knows, and the arrs do not fail loudly — they degrade to copy-then-delete, rewriting every
file over NFS and destroying the seeding copy. The smallest subtree containing both tv/ and
downloads/ IS the export root, so the root mount is the MINIMAL mount that preserves hardlinks.
Least privilege bounded below by a hard requirement. The cost is stated, not hidden: sonarr can
also write /media/movies.
plex /mnt/Storage/plex -> /media ro ReadOnlyMany
sonarr /mnt/Storage/plex -> /media rw ReadWriteMany
radarr /mnt/Storage/plex -> /media rw ReadWriteMany
nextcloud /mnt/Storage/nextcloud -> /mnt/legacy-nextcloud ro ReadOnlyMany (subPath: nextcloud)
Plex is read-only because it indexes and streams; every write on that tree belongs to the arrs, and
saying so removes Plex from the set of things that can damage 8.33 T. All three media consumers use
the SAME mountPath deliberately: when qbittorrent is eventually given this export, the arrs and the
download client will already agree on absolute paths, which is the other precondition for hardlinked
imports.
*** /media AND NOT /data — A COLLISION CAUGHT WHILE RENDERING. *** The arrs' Longhorn config PVC
renders at the renderer default /data/config. A media mount at /data is mounted OVER it and MASKS
the config volume: the pod comes up with its app database shadowed by an 8 T media tree. That
discovery also exposed a SEPARATE PRE-EXISTING BUG, reported and deliberately NOT fixed here:
LinuxServer arr images run with -data=/config, so /data/config is very likely the wrong home for
that PVC and the app has been writing its EPHEMERAL layer — the exact failure catalog/plex.yaml's
mountPath comment records already having been bitten by. Plex was fixed; sonarr, radarr, qbittorrent
and jackett were not. Fixing it starts those apps from an empty config volume, so it needs its own
change with the operator present. This change does not make it worse.
UID/GID — THE FAILURE THIS TIER ALWAYS HAS
The export is sec=sys + no_root_squash, so the client's numeric uid passes through verbatim and the
server's ownership is the entire access story. The tree is 0755 owned 1000:3000, so ONLY uid 1000
can write it. LinuxServer images default to 911. Without intervention sonarr and radarr would have
mounted cleanly, gone Ready, and failed EVERY import with EACCES.
And fsGroup cannot fix that — the in-tree NFS plugin does NO ownership management, so
podSecurityContext.fsGroup is a no-op on an NFS volume. Reaching for it is how this gets
misdiagnosed. So PUID=1000/PGID=3000 are added to both arrs (PGID 3000 keeps what they write
group-readable to the live non-k3s consumers of the same export), and fsGroup: 3000 is added for
the LONGHORN config PVC only, so changing PUID does not depend on the image re-chowning /config.
plex already runs as 1000 and reads a 0755 tree; nextcloud reads as www-data. Neither needs work.
NEXTCLOUD IS MOUNTED READ-ONLY, WHICH IS A DEVIATION FROM THE BRIEF — ON EVIDENCE
The task said nextcloud writes. The dataset says otherwise: of 270 G, 267 G is config_backup/, the
rest is config/ (103 M) and config_old/ (3.4 G). There is NO data/ directory. This is the OLD
non-k3s instance's configuration plus a backup blob — an archive — and that instance is STILL LIVE
on it (exported rw to 192.168.1.6 and .5, which the operator has said must stay active). Two
Nextclouds writing one instance's config directory, each with its own database and therefore its own
oc_filecache, is unrecoverable corruption. "Do not grant more than needed" lands on ro, and ro is
the trivially reversible choice. If the intent is in fact a MIGRATION, that is a larger and more
deliberate change than a mount; the one-line flip is documented.
Read-only for nextcloud can additionally be enforced SERVER-side (only nextcloud consumes that
dataset from k3s), and the sharenfs value in the doc uses ro= for the k3s clients. The media tree
cannot: plex and the arrs may land on any node, so the server sees the same addresses for reader
and writers. Plex's ro is therefore a client-side guarantee — stated as a limitation, not left
implied.
Co-Authored-By: Claude Opus 5 <[email protected]>
The `bulk` class has existed since Phase 0 as a deliberate stub: the StorageClass named an NFS
CSI driver that is not installed and a server/share of REPLACE-ME, so plex, sonarr, radarr and
qbittorrent all carry a comment saying "attach the library once OQ-2 is resolved". This is that
resolution — for the case that actually exists here, which is NOT the case the stub anticipated.
The stub assumed DYNAMIC provisioning: ask a CSI driver for a bulk volume and get a fresh empty
subdirectory. That is the wrong shape for this platform's bulk data. The media library is 8.33 TiB
that already exists, on ZFS datasets on the Proxmox host, with live non-k3s consumers. Nothing
needs provisioning; existing datasets need ATTACHING. So a bulk storage entry now renders a
STATIC, pre-bound NFS PersistentVolume plus its PVC, and needs no CSI driver at all (the in-tree
NFS volume source is not deprecated and has no CSI migration).
Schema 1.4.0 adds, on a storage entry:
export absolute NFS export path; REQUIRED when class is bulk, FORBIDDEN otherwise (an `if`
in the schema, so a bulk entry cannot be authored without saying what it attaches).
subPath mount a subtree instead of the volume root — any class.
readOnly any class; on bulk it also sets the PV's nfs.readOnly, so the mount really is `ro`.
The SERVER is deliberately NOT a schema field. It is the `${BULK_NFS_SERVER}` cluster-vars knob,
because dev and prod reach the SAME storage host by different addresses and R-ENV-02 says the
service definition is authored once. A ServiceDefinition says WHICH dataset; the overlay says
where the host is.
Three decisions worth the reader's time, argued in full in the r_bulk_pvs docstring:
* PV/PVC, not an inline `nfs:` pod volume. Inline is fewer objects and would work. It cannot
express `persistentVolumeReclaimPolicy: Retain`, which is the one property most worth having
in git for 8.33 TiB of data with consumers outside this cluster; it puts read-only enforcement
one Deployment edit away from silent removal; and it is invisible to `kubectl get pv,pvc`,
which is where an operator looks for attached external storage.
* storageClassName: "" on both ends, NOT the `bulk` StorageClass. Explicitly empty is the
canonical marker for a statically provisioned volume. It also sidesteps a trap: a
StorageClass's `provisioner` is IMMUTABLE, so re-pointing the deployed `bulk` stub would make
Flux fail on that object forever.
* Access mode is always `…Many` and follows the read/write posture — ReadOnlyMany for a reader,
ReadWriteMany for a writer. One export is consumed by several namespaces AND several nodes at
once, which is exactly what ReadWriteOnce cannot express; which `Many` it is then makes the
intent legible in `kubectl get pv`.
mountOptions are a platform decision held in the renderer rather than a per-service field:
nfsvers=4.1,hard,noatime,timeo=600,retrans=2. NFSv4 carries mount and lock in-protocol, so the
only port that must be reachable is tcp/2049 — no rpcbind/111, mountd or statd. That is a
firewall decision as much as a storage one. `hard` over `soft` because a soft mount converts a
server hiccup into a short read handed to the application.
The contract doc records the trap that will otherwise be discovered in production: fsGroup is a
NO-OP on an NFS volume (the in-tree plugin does no ownership management), so a writable bulk mount
requires the container's own uid/gid to suit the export's ownership. A mount that succeeds and
then cannot write is this tier's characteristic failure.
Every pre-existing golden renders byte-identical: the new keys are emitted only when declared, and
`readOnly` defaults to False. render-golden: 10/10. selftest: PASS.
Co-Authored-By: Claude Opus 5 <[email protected]>
Semaphore's whole SEMAPHORE_OIDC_PROVIDERS JSON blob lived inside the SOPS-encrypted
semaphore-bootstrap Secret (key `oidc-providers-json`) because it CONTAINS the confidential
client secret. That buried two environment-specific, NON-SECRET URLs in ciphertext —
provider_url (the realm issuer) and redirect_url — where Flux's ${BASE_DOMAIN} postBuild
substitution cannot reach. Both were frozen at `home.keithjackson.dev`, so prod's Semaphore
was pointed at dev's Keycloak and dev's own redirect URI: identical root cause to the
Headlamp login timeout, not yet reported only because nobody had tried prod SSO.
FIX: split the blob. The URLs are public and now live as a plaintext literal in
catalog/semaphore.yaml (rendered to the apps/wave-1 golden), ${BASE_DOMAIN}-substituted like
every other hostname. Only the client secret stays encrypted, in its own key
`oidc-client-secret`, and the KUBELET splices it back via dependent-env-var expansion:
$(SEMAPHORE_OIDC_CLIENT_SECRET) resolves against the secretKeyRef env var declared
immediately above it (order is load-bearing — the kubelet resolves env in list order and
leaves a forward reference as literal text).
Verified: the new key holds the same value as keycloak-realm-clients/semaphore-client-secret
(the mirror invariant), Semaphore v2.19.10's util.OidcProvider consumes client_secret as
before, and `flux envsubst --strict` leaves $(...) untouched while resolving ${BASE_DOMAIN} —
dev renders keycloak.home.keithjackson.dev, prod keycloak.prod.home.keithjackson.dev.
apps/wave-1/semaphore.yaml is a render golden; regenerated with the canonical
`catalog/render.py --all catalog -o apps --wave-layout --no-image-automation`.
Co-Authored-By: Claude Opus 5 <[email protected]>
The four backup-credential SOPS Secrets are SHARED BASE manifests: platform/ and apps/ are
rendered by the same Flux tiers on both clusters, and the ONLY per-environment difference in
this repo is cluster-vars substitution (R-ENV-02). So writing the R2 token into all four
(0a6a862) did not "give prod credentials" — it REPLACED DEV'S. Merging that as-is would have
handed dev's Velero and all three dev CNPG Clusters an R2 key id to sign MinIO requests with:
403 on every WAL segment, and archive_mode=on turns that into a disk-full clock on the pgdata
PVC, not a degraded backup. Dev's PROVEN DR restore (R-STATE-03) is the thing that would have
broken.
This commit splits the material. Each credential is now TWO Secret objects, both applied to
BOTH clusters, differing only in name suffix:
dev -> velero-cloud-credentials keycloak/nextcloud/semaphore-db-backup-credentials
prod -> velero-cloud-credentials-prod ...-db-backup-credentials-prod
The consumer picks one by name; that wiring is the NEXT commit, so this one changes no
rendered reference and is a no-op for both live clusters.
WHY DEV KEEPS THE UNSUFFIXED NAME. Dev is live. Renaming its Secret would change
`s3Credentials.accessKeyId.name` on three running CNPG Clusters — which projects into the
instance pods' environment and therefore ROLLS keycloak-db, nextcloud-db and semaphore-db —
and would roll the velero Deployment, all to rename an object whose contents do not change.
This repo already makes exactly this call for dev's live identifiers (EXTERNALDNS_TXT_OWNER_ID
"default", VELERO_BACKUP_PREFIX ""): the live cluster keeps the name it is running, and the
NEW environment takes the new one. The asymmetry is deliberate and documented at each
kustomization.
WHERE THE MATERIAL CAME FROM. Neither side is invented:
- the four unsuffixed files are restored byte-for-byte from `main` (the MinIO material as
it stood before 0a6a862), so dev's Secrets are IDENTICAL to what dev is running;
- the four `-prod` files carry the R2 token from 0a6a862 (already on this branch as
9739e9b), re-encrypted only because metadata.name changed.
RE-ENCRYPTION, AND WHY A COPY+SED IS NOT ENOUGH. `.sops.yaml` sets
`encrypted_regex: ^(data|stringData)$`, so metadata.name stays cleartext — but SOPS MACs the
WHOLE document (there is no `mac_only_encrypted` here), so editing that cleartext line
invalidates the MAC and `sops -d` then refuses the file. The `-prod` files were therefore
produced inside the dev toolbox (infra/devcontainer, which has sops 3.9.4 and the age key
mounted) by decrypt -> rename -> re-encrypt, with the plaintext never leaving a container
tmpfile. Note for the next person: `sops -e --output NEW OLD` matches creation rules against
the INPUT path, so encrypting a tmpfile into a repo path needs `--filename-override NEW` or
it dies with "no matching creation rules found".
Verified: all eight files `sops -d` cleanly; the four dev files hash to dev's material and the
four prod files to R2's, and the two sets differ. `.sops.yaml` needed no new rule — the
existing `(platform|apps|catalog)/.*(secret|sops)\.ya?ml$` rule already matches
`*-prod.sops.yaml`, so the new files encrypt to the same age recipient.
apps/wave-0/kustomization.yaml is GENERATED, so render.py learns the `-prod` sidecar rather
than the golden being hand-edited: `<db>-backup-credentials-prod.sops.yaml` now places into
the DB's wave alongside its unsuffixed twin (both must exist before the Cluster that names
one), and the selftest asserts it.
Co-Authored-By: Claude Opus 5 <[email protected]>
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).
THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.
Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.
PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.
BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.
TWO THAT CANNOT BE TOKENIZED, flagged not forced:
- headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
anyway — client secrets are realm-scoped.
- kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
prod.home.
EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.
I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.
DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.
L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.
PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.
Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
LIVE REGRESSION, caught on the reconcile after merging #5. nextcloud went
CrashLoopBackOff with FailedPostStartHook, 7 restarts, and apps-wave-1/2 stalled.
ROOT CAUSE — a latent bug that could never fire until now. The final statement of
nextcloud's postStart script was:
[ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote ... after retries"
On the SUCCESS path (p < 15) the test returns 1, and as the script's LAST command that
becomes the hook's exit status. A non-zero postStart exit makes the kubelet KILL the
container. This was harmless for as long as the OIDC guard above it was dead code that
exited 0 first (see 3484955) — fixing that guard made the block execute for the first
time in these apps' deployed lifetime, and the bug fired immediately.
That is the honest sequence: the previous commit did not introduce this defect, it
UNCOVERED it. Two bugs stacked, and the outer one hid the inner one.
FIX: convert to a plain `if`, then `exit 0` explicitly, with a comment recording why.
Also made gitea's hook explicit. Its last statement already returned 0 on both paths, so
it was correct by accident of ordering; leaving one hook implicitly-safe and one
explicitly-safe invites exactly this class back. Both now end with an unconditional
`exit 0`, and both say why in-file.
Every failure path in both hooks still logs loudly to stderr AND writes a marker file on
the persistent volume, and deliberately does NOT crash the pod — the break-glass local
admin must stay usable. That contract is now actually honoured; before, the success path
was the one that crashed.
Fixed in catalog/ (the ServiceDefinitions are the source of truth) and goldens
regenerated; render==golden green, renderer selftest PASS.
Serves: R-IDP-02, R-IDP-06, R-GITOPS-02.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Both ServiceDefinitions guarded their Keycloak OIDC registration with an UNESCAPED
braced expansion:
if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then # gitea
if [ -z "${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then # nextcloud, then exit 0
apps-wave-0/1 both declare postBuild.substituteFrom: cluster-vars, and neither name is a
cluster-vars key — so Flux's envsubst substituted them away AT RECONCILE TIME, long
before the container ever ran. Verified in the LIVE cluster: the deployed Deployments
literally contain
if [ -z "" ]; then
which is always true. The entire OIDC registration block has been skipped on EVERY boot
of both apps.
WHY NOBODY NOTICED: both apps DO have the provider registered — `gitea admin auth list`
shows keycloak/OAuth2/true and `occ user_oidc:provider` shows the keycloak provider.
Those live in each app's own database, persisted from an earlier successful registration.
SSO works today and DECISIONS-LOG is not wrong. But the idempotent re-registration is
dead, so any PVC rebuild, DB restore, or fresh cluster build comes up with NO OIDC and
only the break-glass local admin. A PROD BUILD FROM THIS REPO WOULD SILENTLY HAVE NO SSO
on gitea or nextcloud.
The irony: the surrounding code was hardened twice — see its own comments about a "silent
first-boot RACE", the verify-then-retry loops, the marker files. All of that hardening was
unreachable, because the guard above it always evaluated true.
Not a regression from the recent R-ENV-02 tokenization — introduced 2026-08-27 (8738871).
FIXED AT THE RIGHT SEAM: apps/wave-0/gitea.yaml and apps/wave-1/nextcloud.yaml are
RENDER-GENERATED, so the fix is in catalog/*.yaml and the goldens are regenerated.
Escaped to $${...}, the convention documented in clusters/base/apps-vms.yaml. Post-envsubst
the container now receives the intended `${VAR:-}` instead of `""`.
STRICT MODE WOULD NEVER HAVE CAUGHT THIS. Flux deliberately exempts default-providing
operators (-, :-, +, :+, =, :=, ?, :?) from StrictPostBuildSubstitutions, because they
"intentionally handle the unset case". So a targeted check was required.
MADE UN-REPEATABLE IN THE RENDERER: catalog/render.py gains a substitution-safety section
wired into emit(), so every render path refuses to emit a manifest containing a braced
expansion that is not exactly ${KNOWN_KEY} for a key read LIVE from cluster-vars (no
hardcoded copy to drift; unreadable => empty set => fail-closed). Dollar-runs are counted
properly ($$ is a literal $ to envsubst, so odd runs are flagged and even runs are not),
and indexed forms get their own hard-error message. Nine new --selftest assertions,
including the exact regression. Proven to bite: rendering the pre-fix catalog files from
HEAD now exits 1 with the precise diagnosis.
Sweep: the complete substituted surface was enumerated from every Kustomization's path.
catalog/ contains exactly TWO braced expansions and both were these bugs; every other
braced expansion on a substituted path is a declared cluster-vars key (zero typos, zero
undeclared names); existing $${UOS_URL} escapes verified intact.
Verified: --selftest PASS; canonical render is byte-identical to the committed apps/ for
all 10 onboarded services; the three L1 render checks pass.
NEXT RECONCILE: both pod templates change, so gitea (wave 0) and nextcloud (wave 1) roll —
strategy: Recreate on RWO Longhorn PVCs, so expect a brief single-pod outage each. The
OIDC block then executes for the first time in these apps' deployed lifetime. Both paths
are idempotent (gitea takes auth update-oauth on the existing id; nextcloud's
user_oidc:provider is create-or-update), so the expected result is a no-op re-write plus
the verify loop logging success. If the SOPS client secret ever diverged from Keycloak's,
this is when it gets corrected instead of silently left stale. Tail both pods after merge.
Serves: R-IDP-02, R-IDP-06, R-ENV-02, R-GITOPS-02.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Close the un-gated-UI security finding for forward-auth apps (R-IDP-02) by
rendering the Option-B per-app oauth2-proxy reverse-proxy sidecar that crafty
proves working. catalog/render.py now emits, for any forward-auth app with an
exposed UI: the oauth2-proxy sidecar (upstream http://127.0.0.1:<uiport>,
:4180, --allowed-group per auth.group), an authproxy :4180 Service port, an
HTTPRoute backendRef -> :4180, and a gateway-ingress CiliumNetworkPolicy that
allows the Gateway to reach ONLY :4180 (fail-closed: the bare UI is never
served un-authenticated). oidc/native/none apps render byte-identical.
This is the CANARY: only pihole's golden carries the sidecar and is verified
fail-closed in-cluster first. jackett/qbittorrent/radarr/sonarr are deferred to
the rollout commit (temporarily listed in ci/render-golden.sh) so their UIs
keep serving exactly as today until their clients+secrets land.
- keycloak: confidential pihole client (realm-import + realm-clients mirror),
provisioned live via kcadm (groups membership mapper, bare names).
- secret: SOPS/age pihole-oauth2-proxy-credentials (client-id/secret/cookie),
wired into apps/wave-0 via the renderer's new sidecar-creds wave rule.
- L1 render goldens regenerated for the sidecar (forward-auth fixtures).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
App SSO broke cluster-wide after the Cilium 1.16.5->1.20.1 upgrade: the CVE-2025-30162 fix (in
>=1.16.8/1.17.2) makes the shared Gateway's L7 proxy re-evaluate a pod's egress policy against
Keycloak's REAL backend cluster identity (enforce_policy_on_l7lb) when it hairpins to keycloak.home.
'toEntities: world' matches the Gateway VIP at L3/L4 but not the resolved keycloak identity at L7, so
OIDC discovery/token 403'd ('Access denied') for every app with a NetworkPolicy (gitea/nextcloud/
semaphore/headlamp/crafty...); Grafana survived only because monitoring has no NetworkPolicy.
Fix: explicit toEndpoints egress allow to the keycloak backend for SSO apps.
- catalog/render.py: emit it for auth.mode in {oidc,forward-auth}; re-rendered 8 app goldens.
- platform/headlamp + platform/crafty: hand-written policies get the same allow.
Split-horizon design unchanged. Verified live: headlamp-ns -> keycloak discovery 403 -> 200.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Text-only cleanup after the upgrade campaign. Updates stale version
strings, prose, and comments to the verified current facts; no
Kubernetes behavior or version pins changed except the kubectl
toolchain bump (below).
Verified facts applied: k3s v1.36.4+k3s1, Debian 13 trixie,
Cilium 1.20.1 / Gateway API v1.6.1, Longhorn 1.12.1, node RAM 24 GiB,
dedicated virtio2 SSD etcd disk per node, Velero chart 12.x / v1.18,
Grafana 13.2.0.
- Version strings across infra/ and platform/ READMEs + comments
- kubectl toolchain 1.31.4 -> 1.36.4 (devcontainer/ci/conformance
versions.env; 1.31 vs a 1.36 cluster exceeded the +-1 skew)
- Retired-single-apps-Kustomization refs -> apps-wave-0/1/2 + apps-vms
with per-wave SOPS decryption (flux/clusters docs + bootstrap.sh)
- COMPLETED/historical banners on the three executed plan docs
- Resolved TODO(WS-SEC) markers -> Renovate-maintained ranges
- Appended dated DECISIONS-LOG entry (2026-08-30) superseding the
2026-08-26 LIVE STATE line
Left as revalidation items (not blind-edited): external-snapshotter
v8.2.0 vs 1.36, Cilium ext_authz/forward-auth relevance under 1.20.1,
conformance ImageVolume feature-gate assertions under 1.36.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Migrate image.toolkit.fluxcd.io/v1beta2 -> /v1 for ImageRepository,
ImagePolicy, and ImageUpdateAutomation ahead of the Flux v2.7+ upgrade
(cluster-upgrade-plan §3.2). Flux v2.4 (current) already serves /v1, so
migrating the manifests now is a safe no-op at runtime.
- catalog/render.py: r_image_automation emits v1 for ImageRepository/ImagePolicy
- platform/image-automation/{image-update-automation,per-service/pihole.example}
- conformance L3 hands-off-update chainsaw applies v1 CRs
- conformance L1 goldens re-rendered to match (apiVersion-only diff)
No spec-field changes required: all spec fields used (image, interval,
imageRepositoryRef, filterTags.pattern/extract, policy.semver.range,
sourceRef, git.*, update.strategy/path) are unchanged in v1; the v1
messageTemplate already uses .Changed (not the removed .Updated).
apps/ golden unaffected (canonical render uses --no-image-automation).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Plex Deployment had no Plex env vars. Add via the ServiceDefinition env list
(rendered verbatim by catalog/render.py):
- PLEX_UID/PLEX_GID=1000 (image chowns /config to this and drops; matches the
Longhorn ownership convention seerr/semaphore use)
- TZ=Etc/UTC (no other service sets TZ repo-wide)
- ALLOWED_NETWORKS=10.0.0.0/8,192.168.0.0/16 so in-cluster clients (Seerr ->
plex:32400) are trusted as LAN (pod CIDR 10.0.0.0/16 + svc 10.43.0.0/16 + LAN)
PLEX_CLAIM intentionally not set (4-min TTL is unusable via git; one-time live
claim persists on /config). ADVERTISE_IP deferred: the only candidate host
resolves via split-horizon DNS to the internal Gateway VIP and its remote
reachability is unverified — a wrong value breaks remote access, so omitted
per prefer-safety, documented as a follow-up in catalog/plex.yaml.
Homepage still labeled seerr as "Overseerr"; renamed the tile to "Seerr"
(icon mdi-play-box-multiple keeps — not overseerr-specific).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
apps/wave-1/nextcloud.yaml is GENERATED from catalog/nextcloud.yaml by
catalog/render.py, so the previous commit hardened only the rendered golden —
the next render would have reverted step (d) to the old single-pass version.
Apply the same verify-and-retry hardening to the postStart hook in
catalog/nextcloud.yaml (the source of truth). Re-rendered with
`render.py --all catalog -o apps --wave-layout`; apps/wave-1/nextcloud.yaml is
byte-identical to the golden committed in the previous commit (render round-trip
clean), so no apps/ change is needed here.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Task 1 — browser SSO "Could not reach OpenID Connect provider" (confirmed root
cause, not guessed). keycloak.home.keithjackson.dev resolves split-horizon to the
PRIVATE Cilium gateway VIP (192.168.1.53). Nextcloud's SSRF guard refuses its own
server-side HTTP requests to RFC1918 addresses, so user_oidc's discovery/JWKS/token
fetch AT LOGIN throws OCP\Http\Client\LocalServerException "violates local access
rules" -> surfaced to the user as "Could not reach the provider". Proven from the
live pod: nextcloud.log carries the exact LocalServerException from
DiscoveryService::obtainDiscovery with Guzzle option allow_local_address:false; a
Nextcloud-context IClientService->get() of the discovery URL returns
LocalServerException with the flag false and HTTP 200 with it true (A/B toggled).
The postStart (b) curl check passed regardless because it uses system libcurl, not
Nextcloud's guard — which is why the provider registered yet login still failed.
Fix (reproducible, survives pod rolls): the postStart hook now runs
`occ config:system:set allow_local_remote_servers --value=true --type=boolean`
right after the install/maintenance wait, on every start.
Task 2 — promote Keycloak `platform-admins` members to Nextcloud admins (#5).
user_oidc 8.11 has NO native group->admin mapping (occ user_oidc:provider offers
group PROVISIONING and a login WHITELIST only), so the hook mirrors the provisioned
`platform-admins` membership into the built-in `admin` group idempotently (occ
group:list --output=json parsed via php argv; occ group:adduser admin per member).
A user is promotable after their first OIDC login provisions them + the next pod
start (or one manual `occ group:adduser admin <uid>`). The LOCAL admin stays as
break-glass (docs/15).
apps/wave-1/nextcloud.yaml re-rendered from catalog (single-file render; byte-diff
confined to the embedded hook). Rendered hook passes `sh -n`; parse step validated
in-pod. `make -C ci gate` green (lint/l0/l1; conformance + spec-immutability skip:
no cluster / spec repo unmounted).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The plexinc/pms-docker image writes its identity/claim token and the
Library metadata DB under /config, but the config PVC was mounted at the
renderer default /data/config. Plex therefore wrote all state to the
container's EPHEMERAL layer: the server stayed UNCLAIMED (servers.plex.tv
returns 401 with an empty X-Plex-Token) and lost everything on restart,
which is why seerr's "add Plex" failed even with SSL off.
Set the config volume mountPath to /config in catalog/plex.yaml and
re-render apps/wave-0/plex.yaml (only the mountPath line changes; all
other services byte-identical). No fsGroup/securityContext needed:
verified in-cluster the image inits as root (s6-svscan PID 1), chowns
/config to the plex user, then drops privileges (the "Plex Media Server"
process runs as plex) — root can chown the mounted PVC regardless of
fsGroup.
PLEX_CLAIM auto-claim is intentionally NOT wired: the env secretKeyRef
schema has no `optional`, so a placeholder/absent claim Secret would
crashloop the pod. The claim is an operator step (UI sign-in after the
roll, or add a PLEX_CLAIM env from a fresh https://plex.tv/claim token).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The Keycloak OIDC postStart registration for gitea and nextcloud ran but
never persisted, leaving `gitea admin auth list` empty and nextcloud's
user_oidc with "No providers configured" despite the apps installing/
enabling and the client secret being present.
Two distinct root causes, confirmed by running the registration manually
against the live warm pods:
- nextcloud: a QUOTING BUG, not a race. The OCC() wrapper runs
`su -c "php occ $*"`, which word-split `--scope="openid email profile"`
back into three tokens, so occ saw `email profile` as extra positional
args -> "Too many arguments to user_oidc:provider" -> exit 1, swallowed
by `|| echo failed`. Failed on EVERY boot. Fixed by running the provider
call as ONE `su -c` with the scope single-quoted so www-data's shell
keeps it a single argument; keep the OCC() wrapper only for single-token
subcommands.
- gitea: a first-boot RACE. The direct `admin auth add-oauth` persists once
the app/network is warm (verified live), but fired too early at postStart
and its failure was swallowed (`|| echo ... >&2`, exit 0).
Hardening applied to both catalog sources (rendered into apps/wave-*):
- wait (bounded ~90s) for the Keycloak discovery doc to return HTTP 200;
- nextcloud also waits for `installed: true` AND `maintenance: false`;
- register idempotently (create-or-update) then VERIFY the source/provider
actually persisted, retrying the whole cycle (guards exit-0-but-nothing-
stuck);
- on give-up, log LOUDLY to stderr AND a marker file on the persistent
volume (/data, /var/www/html) instead of swallowing — next failure is
visible;
- never crashloop: missing-secret guard retained, break-glass local admin
stays usable.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Extends the SSO work to the two remaining `auth: oidc` apps still on local-admin
logins, each as a native confidential client of the Keycloak `homelab` realm with
its local admin kept as break-glass (docs/15 availability caveat; R-IDP-02/06).
Realm (source of truth): add the `nextcloud` and `semaphore` confidential clients
to platform/keycloak/keycloak-realm-import.yaml (modeled on grafana/gitea) + their
NEXTCLOUD_CLIENT_SECRET / SEMAPHORE_CLIENT_SECRET placeholders. As before, a
KeycloakRealmImport no-ops on an already-existing realm, so both live clients were
ALSO created via kcadm (nextcloud e95049ed-ca10-4085-97d9-4bbef495fe61, semaphore
1835bfe9-b125-4026-8f69-33557519cf4a); the YAML keeps git authoritative.
Secrets (SOPS/age): add nextcloud-client-secret + semaphore-client-secret to
keycloak-realm-clients.sops.yaml (keycloak ns). Mirror the nextcloud secret as
oidc-client-secret into nextcloud-bootstrap (nextcloud ns). For semaphore, the
whole SEMAPHORE_OIDC_PROVIDERS JSON (which embeds the client secret) is held
encrypted as oidc-providers-json in semaphore-bootstrap (semaphore ns). All values
verified == the live clients; existing keys untouched; no plaintext in git.
App config (catalog source + re-render):
- nextcloud: add NEXTCLOUD_OIDC_CLIENT_SECRET env + an idempotent postStart hook
that installs/enables user_oidc and runs `occ user_oidc:provider keycloak`
(create-or-update; secret via --clientsecret-env so it never hits argv; occ as
www-data; group provisioning on the `groups` claim). Syntax verified against the
running user_oidc 8.11.0 on Nextcloud 31.0.14; su-to-www-data env passthrough
verified. Missing-secret/not-installed guards prevent crashloops.
- semaphore: add SEMAPHORE_OIDC_PROVIDERS env sourced from the encrypted
oidc-providers-json (provider_url=realm issuer/auto-discovery, redirect
/api/auth/oidc/keycloak/redirect). Mechanism verified against semaphore v2.19.10
source (util ConfigType oidc_providers env tag + OidcProvider struct + router
redirect path); no kubelet service-link env collision.
apps/wave-1/{nextcloud,semaphore}.yaml re-rendered via
`render.py --no-image-automation` (diffs are env/lifecycle-only).
Verified: realm import Done=True/HasErrors=False and platform-config
Ready=True/Healthy=True (unchanged); edited realm YAML passes server dry-run; live
homelab clients now include gitea/nextcloud/semaphore; static gate (lint/l0/l1)
green. Browser SSO login for each still needs a human check.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Wires the bootstrapping-critical Gitea git host to the Keycloak `homelab` realm
as a native confidential OIDC client, moving it off the local-admin-only login
while keeping the local admin as the break-glass path (docs/15 availability
caveat; R-IDP-02/06).
Realm (source of truth): add the `gitea` confidential client to
platform/keycloak/keycloak-realm-import.yaml (modeled on grafana) + its
GITEA_CLIENT_SECRET placeholder sourced from the keycloak-realm-clients Secret.
NOTE: a KeycloakRealmImport no-ops on an already-existing realm, so the live
client was ALSO created via kcadm (id 70a6ca60-317a-42ce-8331-a60e380838ee);
the YAML keeps git authoritative for a rebuild.
Secrets (SOPS/age, never plaintext): add gitea-client-secret to
platform/keycloak/keycloak-realm-clients.sops.yaml (keycloak ns) and mirror the
same value as oidc-client-secret into apps/wave-0/gitea-bootstrap-secret.sops.yaml
(gitea ns) so the app can read it in-namespace. Both == the live client secret.
App config (catalog source + re-render): catalog/gitea.yaml gains the
GITEA_OIDC_CLIENT_SECRET env (from the mirrored Secret), [oauth2_client]
auto-registration/account-linking app.ini env, and an idempotent postStart step
that registers the "keycloak" OAuth2 login source (auto-discovery; groups claim;
platform-admins -> Gitea admin). apps/wave-0/gitea.yaml re-rendered via
`render.py --no-image-automation`.
Verified: realm import Done=True/HasErrors=False and platform-config
Ready=True/Healthy=True (unchanged); edited realm YAML passes server dry-run;
static gate (lint/l0/l1) green. Browser SSO login still needs a human check.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Seerr v3.4.1 (ghcr.io/seerr-team/seerr) runs as the non-root user node:node
(UID/GID 1000), unlike the old root-running sctx/overseerr image. Its root-owned
Longhorn RWO config PVC at /app/config was not writable by UID 1000, so the pod
CrashLoopBackOff'd with:
Error: EACCES: permission denied, mkdir '/app/config/logs/'
Add a podSecurityContext with fsGroup=1000 (plus runAsUser/runAsGroup/runAsNonRoot)
to the seerr ServiceDefinition — the same schema 1.3.0 mechanism semaphore uses.
Longhorn's CSIDriver fsGroupPolicy: ReadWriteOnceWithFSType applies fsGroup to this
RWO filesystem volume, group-owning the PVC to GID 1000 so the process can write.
Re-rendered apps/wave-2/seerr.yaml from the catalog source.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Two operator-flagged defects in the seerr ServiceDefinition, fixed at source
(catalog/seerr.yaml) and re-rendered into apps/wave-2/seerr.yaml:
- Image: sctx/overseerr:latest (original Overseerr, unmaintained) ->
ghcr.io/seerr-team/seerr:v3.4.1. Overseerr and Jellyseerr have merged into
the actively maintained Seerr project (seerr-team/seerr, docs.seerr.dev);
ghcr.io/seerr-team/seerr is the officially recommended image. Pinned to the
current stable tag v3.4.1 per the arr-stack convention (sonarr/radarr/
qbittorrent/gitea pin explicit tags; R-UPD-01). update.range bumped
">=1.33.0" -> ">=3.4.0" to track the Seerr 3.x line. Plex-native (native
auth) support is retained.
- Config mountPath: /data/config -> /app/config. Overseerr/Jellyseerr/Seerr
store settings + request/cache DB at /app/config; the previous /data/config
left the config PVC effectively unused (config not persisted across
restarts). Set via the schema-supported storage[].mountPath field.
Dependencies (sonarr:8989, radarr:7878, plex:32400), NetworkPolicies,
public exposure, and native auth are unchanged. Render == golden verified;
lint, schema, and L1 render checks green.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Replaces the single `apps` Flux Kustomization with dependency-ordered waves
(apps-wave-0..2) so a crashlooping workload isolates to its wave + downstream
instead of stalling the entire app tier. Parallel within a wave, ordered across.
- catalog/render.py: compute waves by longest dependency-path depth over the
ServiceDefinition `dependencies:` edges (a `<svc>-db` is a leaf, so its
consumer lands strictly higher — the "DB below consumer" rule falls out of
longest-path, no special case). `--emit-waves` generates the per-wave
kustomizations + Flux CRs; `--wave-layout` places renders into the right dir.
wave 0: pihole gitea jackett plex qbittorrent nextcloud-db semaphore-db
wave 1: sonarr radarr nextcloud semaphore gitea-actions
wave 2: seerr
- apps/*.yaml -> apps/wave-{0,1,2}/ (byte-identical moves; SOPS files intact).
- clusters/base: apps.yaml removed; apps-wave-{0,1,2}.yaml added (dependsOn
chain: platform-config -> wave-0 -> wave-1 -> wave-2, prune+wait each).
- clusters/dev: one SOPS decryption patch per wave (load-bearing).
- DESIGN-waves.md: wave table, topology, cutover runbook.
Cutover: preceded by disabling prune on `apps` (1bfb0db) so removing that
object here does not GC its resources; the wave Kustomizations SSA-adopt the
byte-identical manifests in place (no restart, no PVC churn).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
With the config PVC now writable (fsGroup), semaphore got past config write and
pinged its PostgreSQL successfully, then crashed on:
panic: value of field 'Port' is not valid: tcp://<svcIP>:3000
Because the Service is named `semaphore`, the kubelet injects the Docker-link-style
SEMAPHORE_PORT=tcp://<clusterIP>:3000, which collides with Semaphore's OWN
SEMAPHORE_PORT (its HTTP listen port). Add an explicit SEMAPHORE_PORT=":3000" env
entry — user-defined env takes precedence over service-link vars — pinning the
listen port back to :3000 (matches containerPort/Service). Uses the existing env
field; only apps/semaphore.yaml changes, all others byte-identical.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The semaphoreui/semaphore image runs as non-root UID 1001 / GID 0 and writes its
generated config.json into the config dir on boot. A mounted Longhorn PVC is
root:root, so UID 1001 got "permission denied" and the pod crash-looped.
First-class the fix into the render contract (same mechanism the retired
gitea-bootstrap-patch was replaced by): schema 1.3.0 adds an optional pod-level
`podSecurityContext`, emitted verbatim onto spec.template.spec.securityContext by
render.py. Absent => every other app renders byte-identical (verified). semaphore
declares fsGroup=1001 (kubelet group-owns the PVC + adds the supplemental group so
UID 1001 can write) plus runAsUser/runAsGroup/runAsNonRoot pinning the image identity.
render.py selftest extended (verbatim render + absence). L0 (schema+kyverno 17/17),
L1 3/3, lint all green.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Semaphore (Ansible UI, docs/13 row 10): internal exposure, internet egress,
config on Longhorn + external PostgreSQL, auth oidc. Reuses the CNPG-DB pattern
nextcloud established (apps/nextcloud-db.yaml, commit 4be6752).
- catalog/semaphore.yaml: ServiceDefinition (schema >=1.2.0 env) wiring the
SEMAPHORE_DB_* connection to semaphore-db-rw.semaphore-db:5432, break-glass
SEMAPHORE_ADMIN* local admin, and the FIXED base64 SEMAPHORE_ACCESS_KEY_ENCRYPTION.
- apps/semaphore.yaml: rendered (render.py --no-image-automation) — Deployment,
config PVC, Service, default-deny + egress/gateway/internet CNPs, HTTPRoute
semaphore.home.keithjackson.dev:3000, ServiceMonitor, SLI alert.
- apps/semaphore-db.yaml: CNPG Cluster in dedicated ns semaphore-db, initdb from
the owner secret, inheritedMetadata app.kubernetes.io/name=semaphore-db so the
dep-edge egress CNP matches.
- apps/semaphore-db-app.sops.yaml + apps/semaphore-bootstrap-secret.sops.yaml:
SOPS(age) owner creds (basic-auth) + mirrored db-password, break-glass admin,
and the persistent access-key encryption key.
OIDC (Keycloak realm client) is DEFERRED like gitea/nextcloud — comes up on the
break-glass local admin; realm-client wiring is the batched follow-up.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Onboard nextcloud (docs/13 row 7): public, DR-critical file sync backed by an
external PostgreSQL. Establishes the reusable CNPG-DB pattern for later DB-backed
services.
- catalog/nextcloud.yaml: ServiceDefinition (image nextcloud:31-apache :80, public,
internet, data 20Gi longhorn @/var/www/html, backup daily-offcluster, auth oidc
+platform-admins, ingress nextcloud.home.keithjackson.dev, dependency
nextcloud-db:5432). Uses schema 1.2.0 env for the Postgres connection, break-glass
admin, trusted-domain + OVERWRITE* proxy settings.
- apps/nextcloud-db.yaml: CloudNativePG Cluster in a dedicated nextcloud-db namespace
(modeled on keycloak-db; no default-deny, mirroring the verified keycloak posture),
inheritedMetadata stamps app.kubernetes.io/name=nextcloud-db so the rendered egress
CNP matches. initdb consumes a pre-provisioned owner secret.
- apps/nextcloud-db-app.sops.yaml + apps/nextcloud-bootstrap-secret.sops.yaml: SOPS
(age) — shared DB password mirrored across the two namespaces (Pods cannot read a
Secret cross-namespace), plus the break-glass local admin.
- apps/nextcloud.yaml: rendered manifests (catalog/render.py --no-image-automation).
- apps/kustomization.yaml: wire in the four new resources.
OIDC realm-client wiring DEFERRED (WS-APPS step-3 escape hatch): editing the reconciled
realm import in platform-config risks the running Keycloak realm / whole platform-config
apply, and nextcloud OIDC needs post-install occ steps with no first-class env — nextcloud
comes up INSTALLED on the break-glass local admin (as Gitea did); the Keycloak client is a
follow-up. Off-cluster backup awaits Velero/OQ-BKP (intent declared).
Gates: render --selftest PASS; other apps byte-identical; ci l0 17/17; l1 3/3; lint clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Extend the ServiceDefinition contract with optional per-container `env`,
`envFrom`, and `lifecycle` fields (k8s EnvVar / EnvFromSource / Lifecycle
shapes) so headless-install / DB-backed apps configure themselves through
the SD instead of a supplementary kustomize patch.
- schema: bump 1.1.0 -> 1.2.0; add optional env/envFrom/lifecycle to the
container kind (literal + secretKeyRef/configMapKeyRef; secretRef/
configMapRef; postStart/preStop exec via $defs/execHandler). Existing
fields unchanged; all fields optional -> absent renders byte-identical.
- render.py: emit env/envFrom/lifecycle verbatim onto the Deployment app
container when present; --selftest coverage for the new fields.
- catalog/gitea.yaml: fold the headless-install env (INSTALL_LOCK, fixed
SECRET_KEY via secretKeyRef, DOMAIN/ROOT_URL, actions, break-glass admin
creds) + the idempotent postStart break-glass admin bootstrap inline.
- apps/gitea.yaml: re-rendered with env/lifecycle inline (effective pod
spec provably equivalent to base+patch).
- DELETE apps/gitea-bootstrap-patch.yaml + drop it from apps/kustomization
patches. gitea-bootstrap-secret.sops.yaml kept (still sourced by the env).
Proves the renderer feature by retiring the known TODO; unblocks env + DB
onboarding for nextcloud/semaphore/unifi/minecraft. All other rendered apps
byte-identical. Gates: selftest PASS, l0 17/17 + schema, l1 goldens, lint.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Follow-up to the reciprocal-ingress fix: in-cluster verification (Hubble) revealed
the east-west path was STILL fully blocked — and that the pre-existing EGRESS half
was also broken, contrary to the original diagnosis. Root cause: a CiliumNetworkPolicy
`toEndpoints`/`fromEndpoints` selector is scoped to the POLICY's OWN namespace unless
it carries the reserved `k8s:io.kubernetes.pod.namespace` label. Every service lives
in its own namespace (== its name) and every dependency crosses a namespace boundary,
so an app-label-only peer selector matched nothing and Cilium silently denied the flow
(Hubble: sonarr->qbittorrent = "EGRESS DENIED ... Policy denied DROPPED").
Pin the peer namespace on BOTH halves — egress toEndpoints (destination ns == dep.to)
and the reciprocal ingress fromEndpoints (source ns == source name). This is a strict
TIGHTENING (more precise, still L3/L4), not a weakening. Verified end-to-end in-cluster:
sonarr->qbittorrent:8080, sonarr->jackett:9117, seerr->{sonarr:8989,radarr:7878,
plex:32400} now CONNECT (were DROPPED); undeclared sonarr->plex:32400 and
seerr->qbittorrent:8080 stay BLOCKED (default-deny holds).
Selftest + L0 (17/17) + L1 (3/3, goldens byte-identical) + lint all pass. NOTE: this
necessarily touches the egress toEndpoints selector, which the task scoped as
ingress-only; the premise that egress already worked was empirically false.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A `dependencies` edge is declared ONCE on the SOURCE, but with the default-deny-
BOTH-directions baseline (docs/03 R-NET-01/02) it needs TWO matching halves: the
source egress-allow AND the destination ingress-allow. render.py emitted only the
egress half, so every declared east-west dependency was silently dropped by the
destination's default-deny ingress — verified in-cluster: sonarr->qbittorrent:8080,
sonarr->jackett:9117 and seerr->sonarr:8989 were all BLOCKED.
Add a cross-service reverse-dependency pass (build_reverse_map) and render the
reciprocal ingress half (r_reverse_dependency_policies): for each source that
declares a dependency on X, X gains an additive ingress-allow CiliumNetworkPolicy
`X-allow-<source>-<port>` whose fromEndpoints matches the source identity, mirroring
the egress half's L3/L4 granularity. Deterministic (sorted), additive (default-deny
ingress stays []), egress side unchanged. A lone render with no cross-catalog context
renders no reciprocal ingress, so the L1 render goldens are byte-identical.
New ingress CNPs: qbittorrent<-{sonarr,radarr}:8080, jackett<-{sonarr,radarr}:9117,
sonarr<-seerr:8989, radarr<-seerr:7878, plex<-seerr:32400. Selftest + L0 (17/17) +
L1 (3/3) + lint all pass.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Any single-replica app with a `storage:` entry mounts a ReadWriteOnce Longhorn
PVC. The default RollingUpdate strategy starts the new pod before terminating the
old one, but the old pod still holds the RWO volume -> the new pod hangs in
ContainerCreating and the rollout deadlocks (observed on gitea's update). Emit
spec.strategy.type=Recreate whenever storage is present so the old pod releases
the PVC first. Non-storage apps (and VMs) keep the default (no strategy emitted).
Re-rendered apps: gitea, pihole, sonarr, radarr, plex, qbittorrent, jackett now
carry strategy: Recreate. Updated the egress-class L1 render goldens (storage
fixture) accordingly; all L1 render checks pass.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Two defects blocked in-cluster Gitea from becoming the Flux source:
1. Storage mountPath: the renderer hard-coded /data/<name>, so Gitea's PVC
mounted at /data/data while Gitea writes all state to /data — real repos/DB
landed on the ephemeral rootfs. Add an OPTIONAL `mountPath` to the storage
schema + render.py (default stays /data/<name>, back-compat). catalog/gitea.yaml
sets data.mountPath: /data. Only gitea.yaml's rendered mount changes; every
other app renders byte-identical.
2. Headless install: the instance was un-installed (INSTALL_LOCK=false, ephemeral
SECRET_KEY, no admin, API 404). A Gitea-only supplementary manifest
(apps/gitea-bootstrap-patch.yaml) patches the rendered Deployment with
GITEA__security__INSTALL_LOCK=true, a FIXED SECRET_KEY from a persistent SOPS
secret (apps/gitea-bootstrap-secret.sops.yaml), ROOT_URL/DOMAIN, and a postStart
hook that idempotently creates a break-glass local admin. State persists on the
now-correct /data Longhorn PVC.
The env/lifecycle wiring is a supplementary kustomize patch because the
ServiceDefinition schema/renderer does not yet express env/envFrom/lifecycle —
noted as a renderer follow-up (TODO in the patch).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Root cause the media stack surfaced: the default-deny baseline is DNS-only egress (correct for
R-NET-01 east-west), but the egress CLASS (internet/vpn) had no policy ALLOWING north-south
egress — so qbittorrent/sonarr/radarr couldn't reach the internet at all (VPN downloaders
broken). Add an additive per-app CiliumNetworkPolicy '<app>-allow-internet-egress' with
toEntities:[world] (non-cluster); the egress-class steering + kill-switch (ADR-0015) govern the
PATH (internet=WAN, vpn=tun0). East-west stays denied except declared dependencies; R-NET-01
baseline unchanged. Re-rendered all apps + regenerated L1 goldens. Also fixed jackett + plex
image tags (bad pins -> :latest; Renovate/update policy pins later).
The 39Gi/node Longhorn data disks at 1:1 over-provisioning can't fit the media stack
(plex 20Gi replica failed: 'insufficient storage'). Volumes are thin-provisioned config-only,
so 200% lets the full catalog fit without a disk resize; physical fill risk is low. Also
right-sized plex config (20Gi was excessive for metadata). Prod-grade path: grow the data
disks (WS-STORAGE follow-up).
Add catalog/ ServiceDefinitions for qbittorrent and jackett and their
rendered apps/ manifests, wired into apps/kustomization.yaml.
Both are VPN-class (egress: vpn -> egress-class=vpn pod label steered by the
CiliumEgressGatewayPolicy + host kill-switch, ADR-0015) and internal
forward-auth UIs (oauth2-proxy -> Keycloak, group media-users). Config on
3-replica Longhorn. qBittorrent's bulk downloads volume is DEFERRED pending
OQ-2 (non-provisioning bulk StorageClass stub), mirroring plex/arr.
Traces: R-NET-05/06 (VPN egress + kill-switch), R-IDP-02/06 (forward-auth),
R-STATE-02/03, R-OBS-01, R-UPD-01/03, R-GITOPS-02.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add catalog/plex.yaml (kind: container) and its rendered apps/plex.yaml,
and wire apps/kustomization.yaml. Plex = docs/13 row 1, the catalog's
largest service; validates the public-exposure + config-on-Longhorn path.
- image plexinc/pms-docker (pinned recent stable tag), port 32400
- exposure: public at plex.home.keithjackson.dev; egress: internet
- auth: native (Plex accounts, not Keycloak — docs/13/15 exception)
- config PVC 20Gi on longhorn; off-cluster backup (daily-offcluster) as the
longhorn tier forbids reacquirable under L0 (R-STATE-03)
- observability up-SLI; update channel stable
Bulk media (R-STATE-04) deferred: the `bulk` StorageClass is a documented
stub (OQ-2 unresolved, NFS CSI not installed) so a bulk PVC would leave the
pod Pending. Media mount omitted; attach once bulk provisions.
Traces: R-GITOPS-02, R-NET-01/02/05, R-STATE-01/02/03, R-TLS/R-DNS,
R-OBS-01, R-UPD-01/03, R-IDP-02, R-PLAT-02.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Exposed apps got a default-deny NetworkPolicy with ingress: [] (deny all), so the
Cilium Gateway (Envoy, reserved identity `ingress`) could not reach the backend — an
Accepted HTTPRoute was still unreachable. render.py now emits an ADDITIVE, least-
privilege CiliumNetworkPolicy for any ServiceDefinition that renders an HTTPRoute:
ingress:
- fromEntities: [ingress] # the shared Cilium Gateway/Envoy proxy identity
toPorts: [<exposed service port(s)> TCP]
Verified against the live cluster (Cilium 1.16.5): with default-deny in place, Hubble
shows the Gateway proxy reaching pihole as source identity `(ingress)` and the flow
`Policy denied DROPPED` on :80 — so `fromEntities: [ingress]` is exactly the source the
Gateway uses, scoped to the service targetPort. Default-deny stays intact (ingress: []
unchanged); the allow only opens the north-south path (docs/09), east-west stays denied.
- render.py: add r_gateway_ingress_policy(); emit it after dependency policies; selftest
asserts the allow is additive (default-deny ingress stays []), least-privilege (only the
exposed port, only the ingress identity), and absent for non-exposed services.
- L1 goldens updated for the two exposed fixtures (container-workload, egress-class);
default-deny-baseline golden unchanged (no exposure). --selftest + run.sh --layer l1 green.
- apps/pihole.yaml re-rendered (--no-image-automation) to carry the allow.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
First app onboarded through catalog/render.py -> apps/. Added --no-image-automation so apps
deploy before the Flux image controllers exist (L1 goldens keep the default, unchanged).
KNOWN GAP surfaced: exposed apps get default-deny (ingress: []) with no gateway-ingress allow,
so the HTTPRoute path is blocked — renderer/WS-NET follow-up to allow Cilium gateway ingress
to services with an exposure/HTTPRoute.
catalog/render.py compiles a ServiceDefinition into the manifest set defined by
schemas/renderer-contract.md: Namespace + default-deny baseline (R-NET-01),
Deployment/Service or KubeVirt VM + CDI DataVolume (R-PLAT-02/03), PVCs (R-STATE),
per-dependency CiliumNetworkPolicy allows (R-NET-02), egress-class steering label
(R-NET-05), HTTPRoutes on the shared Gateway (R-TLS/R-DNS), ServiceMonitor + SLI
alerts (R-OBS-01), Flux image-automation (R-UPD), auth-mode/backup hints (R-IDP).
Deterministic (golden-diffable) with a built-in --selftest; single dependency
(PyYAML) so the conformance harness stays stdlib-only. Documents the mechanism
choice in catalog/README.md and the script header.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
jakman85andClaude Opus 5