Any single-replica app with a `storage:` entry mounts a ReadWriteOnce Longhorn
PVC. The default RollingUpdate strategy starts the new pod before terminating the
old one, but the old pod still holds the RWO volume -> the new pod hangs in
ContainerCreating and the rollout deadlocks (observed on gitea's update). Emit
spec.strategy.type=Recreate whenever storage is present so the old pod releases
the PVC first. Non-storage apps (and VMs) keep the default (no strategy emitted).
Re-rendered apps: gitea, pihole, sonarr, radarr, plex, qbittorrent, jackett now
carry strategy: Recreate. Updated the egress-class L1 render goldens (storage
fixture) accordingly; all L1 render checks pass.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Two defects blocked in-cluster Gitea from becoming the Flux source:
1. Storage mountPath: the renderer hard-coded /data/<name>, so Gitea's PVC
mounted at /data/data while Gitea writes all state to /data — real repos/DB
landed on the ephemeral rootfs. Add an OPTIONAL `mountPath` to the storage
schema + render.py (default stays /data/<name>, back-compat). catalog/gitea.yaml
sets data.mountPath: /data. Only gitea.yaml's rendered mount changes; every
other app renders byte-identical.
2. Headless install: the instance was un-installed (INSTALL_LOCK=false, ephemeral
SECRET_KEY, no admin, API 404). A Gitea-only supplementary manifest
(apps/gitea-bootstrap-patch.yaml) patches the rendered Deployment with
GITEA__security__INSTALL_LOCK=true, a FIXED SECRET_KEY from a persistent SOPS
secret (apps/gitea-bootstrap-secret.sops.yaml), ROOT_URL/DOMAIN, and a postStart
hook that idempotently creates a break-glass local admin. State persists on the
now-correct /data Longhorn PVC.
The env/lifecycle wiring is a supplementary kustomize patch because the
ServiceDefinition schema/renderer does not yet express env/envFrom/lifecycle —
noted as a renderer follow-up (TODO in the patch).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Root cause the media stack surfaced: the default-deny baseline is DNS-only egress (correct for
R-NET-01 east-west), but the egress CLASS (internet/vpn) had no policy ALLOWING north-south
egress — so qbittorrent/sonarr/radarr couldn't reach the internet at all (VPN downloaders
broken). Add an additive per-app CiliumNetworkPolicy '<app>-allow-internet-egress' with
toEntities:[world] (non-cluster); the egress-class steering + kill-switch (ADR-0015) govern the
PATH (internet=WAN, vpn=tun0). East-west stays denied except declared dependencies; R-NET-01
baseline unchanged. Re-rendered all apps + regenerated L1 goldens. Also fixed jackett + plex
image tags (bad pins -> :latest; Renovate/update policy pins later).
The 39Gi/node Longhorn data disks at 1:1 over-provisioning can't fit the media stack
(plex 20Gi replica failed: 'insufficient storage'). Volumes are thin-provisioned config-only,
so 200% lets the full catalog fit without a disk resize; physical fill risk is low. Also
right-sized plex config (20Gi was excessive for metadata). Prod-grade path: grow the data
disks (WS-STORAGE follow-up).
Add catalog/ ServiceDefinitions for qbittorrent and jackett and their
rendered apps/ manifests, wired into apps/kustomization.yaml.
Both are VPN-class (egress: vpn -> egress-class=vpn pod label steered by the
CiliumEgressGatewayPolicy + host kill-switch, ADR-0015) and internal
forward-auth UIs (oauth2-proxy -> Keycloak, group media-users). Config on
3-replica Longhorn. qBittorrent's bulk downloads volume is DEFERRED pending
OQ-2 (non-provisioning bulk StorageClass stub), mirroring plex/arr.
Traces: R-NET-05/06 (VPN egress + kill-switch), R-IDP-02/06 (forward-auth),
R-STATE-02/03, R-OBS-01, R-UPD-01/03, R-GITOPS-02.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add catalog/plex.yaml (kind: container) and its rendered apps/plex.yaml,
and wire apps/kustomization.yaml. Plex = docs/13 row 1, the catalog's
largest service; validates the public-exposure + config-on-Longhorn path.
- image plexinc/pms-docker (pinned recent stable tag), port 32400
- exposure: public at plex.home.keithjackson.dev; egress: internet
- auth: native (Plex accounts, not Keycloak — docs/13/15 exception)
- config PVC 20Gi on longhorn; off-cluster backup (daily-offcluster) as the
longhorn tier forbids reacquirable under L0 (R-STATE-03)
- observability up-SLI; update channel stable
Bulk media (R-STATE-04) deferred: the `bulk` StorageClass is a documented
stub (OQ-2 unresolved, NFS CSI not installed) so a bulk PVC would leave the
pod Pending. Media mount omitted; attach once bulk provisions.
Traces: R-GITOPS-02, R-NET-01/02/05, R-STATE-01/02/03, R-TLS/R-DNS,
R-OBS-01, R-UPD-01/03, R-IDP-02, R-PLAT-02.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Exposed apps got a default-deny NetworkPolicy with ingress: [] (deny all), so the
Cilium Gateway (Envoy, reserved identity `ingress`) could not reach the backend — an
Accepted HTTPRoute was still unreachable. render.py now emits an ADDITIVE, least-
privilege CiliumNetworkPolicy for any ServiceDefinition that renders an HTTPRoute:
ingress:
- fromEntities: [ingress] # the shared Cilium Gateway/Envoy proxy identity
toPorts: [<exposed service port(s)> TCP]
Verified against the live cluster (Cilium 1.16.5): with default-deny in place, Hubble
shows the Gateway proxy reaching pihole as source identity `(ingress)` and the flow
`Policy denied DROPPED` on :80 — so `fromEntities: [ingress]` is exactly the source the
Gateway uses, scoped to the service targetPort. Default-deny stays intact (ingress: []
unchanged); the allow only opens the north-south path (docs/09), east-west stays denied.
- render.py: add r_gateway_ingress_policy(); emit it after dependency policies; selftest
asserts the allow is additive (default-deny ingress stays []), least-privilege (only the
exposed port, only the ingress identity), and absent for non-exposed services.
- L1 goldens updated for the two exposed fixtures (container-workload, egress-class);
default-deny-baseline golden unchanged (no exposure). --selftest + run.sh --layer l1 green.
- apps/pihole.yaml re-rendered (--no-image-automation) to carry the allow.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
First app onboarded through catalog/render.py -> apps/. Added --no-image-automation so apps
deploy before the Flux image controllers exist (L1 goldens keep the default, unchanged).
KNOWN GAP surfaced: exposed apps get default-deny (ingress: []) with no gateway-ingress allow,
so the HTTPRoute path is blocked — renderer/WS-NET follow-up to allow Cilium gateway ingress
to services with an exposure/HTTPRoute.
catalog/render.py compiles a ServiceDefinition into the manifest set defined by
schemas/renderer-contract.md: Namespace + default-deny baseline (R-NET-01),
Deployment/Service or KubeVirt VM + CDI DataVolume (R-PLAT-02/03), PVCs (R-STATE),
per-dependency CiliumNetworkPolicy allows (R-NET-02), egress-class steering label
(R-NET-05), HTTPRoutes on the shared Gateway (R-TLS/R-DNS), ServiceMonitor + SLI
alerts (R-OBS-01), Flux image-automation (R-UPD), auth-mode/backup hints (R-IDP).
Deterministic (golden-diffable) with a built-in --selftest; single dependency
(PyYAML) so the conformance harness stays stdlib-only. Documents the mechanism
choice in catalog/README.md and the script header.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Phase 0 foundation shell (read-only spec at ../home-network-k3s). Top-level
tree + rules; workstream agents fill their subtrees on feature branches.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
jakman85andClaude Opus 4.8