jakman85andClaude Opus 5
cec550045b
The `bulk` class has existed since Phase 0 as a deliberate stub: the StorageClass named an NFS
CSI driver that is not installed and a server/share of REPLACE-ME, so plex, sonarr, radarr and
qbittorrent all carry a comment saying "attach the library once OQ-2 is resolved". This is that
resolution — for the case that actually exists here, which is NOT the case the stub anticipated.
The stub assumed DYNAMIC provisioning: ask a CSI driver for a bulk volume and get a fresh empty
subdirectory. That is the wrong shape for this platform's bulk data. The media library is 8.33 TiB
that already exists, on ZFS datasets on the Proxmox host, with live non-k3s consumers. Nothing
needs provisioning; existing datasets need ATTACHING. So a bulk storage entry now renders a
STATIC, pre-bound NFS PersistentVolume plus its PVC, and needs no CSI driver at all (the in-tree
NFS volume source is not deprecated and has no CSI migration).
Schema 1.4.0 adds, on a storage entry:
export absolute NFS export path; REQUIRED when class is bulk, FORBIDDEN otherwise (an `if`
in the schema, so a bulk entry cannot be authored without saying what it attaches).
subPath mount a subtree instead of the volume root — any class.
readOnly any class; on bulk it also sets the PV's nfs.readOnly, so the mount really is `ro`.
The SERVER is deliberately NOT a schema field. It is the `${BULK_NFS_SERVER}` cluster-vars knob,
because dev and prod reach the SAME storage host by different addresses and R-ENV-02 says the
service definition is authored once. A ServiceDefinition says WHICH dataset; the overlay says
where the host is.
Three decisions worth the reader's time, argued in full in the r_bulk_pvs docstring:
* PV/PVC, not an inline `nfs:` pod volume. Inline is fewer objects and would work. It cannot
express `persistentVolumeReclaimPolicy: Retain`, which is the one property most worth having
in git for 8.33 TiB of data with consumers outside this cluster; it puts read-only enforcement
one Deployment edit away from silent removal; and it is invisible to `kubectl get pv,pvc`,
which is where an operator looks for attached external storage.
* storageClassName: "" on both ends, NOT the `bulk` StorageClass. Explicitly empty is the
canonical marker for a statically provisioned volume. It also sidesteps a trap: a
StorageClass's `provisioner` is IMMUTABLE, so re-pointing the deployed `bulk` stub would make
Flux fail on that object forever.
* Access mode is always `…Many` and follows the read/write posture — ReadOnlyMany for a reader,
ReadWriteMany for a writer. One export is consumed by several namespaces AND several nodes at
once, which is exactly what ReadWriteOnce cannot express; which `Many` it is then makes the
intent legible in `kubectl get pv`.
mountOptions are a platform decision held in the renderer rather than a per-service field:
nfsvers=4.1,hard,noatime,timeo=600,retrans=2. NFSv4 carries mount and lock in-protocol, so the
only port that must be reachable is tcp/2049 — no rpcbind/111, mountd or statd. That is a
firewall decision as much as a storage one. `hard` over `soft` because a soft mount converts a
server hiccup into a short read handed to the application.
The contract doc records the trap that will otherwise be discovered in production: fsGroup is a
NO-OP on an NFS volume (the in-tree plugin does no ownership management), so a writable bulk mount
requires the container's own uid/gid to suit the export's ownership. A mount that succeeds and
then cannot write is this tier's characteristic failure.
Every pre-existing golden renders byte-identical: the new keys are emitted only when declared, and
`readOnly` defaults to False. render-golden: 10/10. selftest: PASS.
Co-Authored-By: Claude Opus 5 <[email protected]>
193 lines
13 KiB
Markdown
193 lines
13 KiB
Markdown
# Renderer Contract — ServiceDefinition → rendered resources
|
||
|
||
**Status:** authoritative Phase-0 interface contract. **Consumed by** the manifest renderer
|
||
(WS-GITOPS), the network/egress renderer (WS-NET), storage (WS-STATE), TLS/DNS (WS-EDGE), identity
|
||
(WS-SEC), observability (WS-OBS), updates (WS-UPD), and the conformance/policy/CI agents (WS-TEST).
|
||
|
||
**Source of truth:** spec [`docs/13-service-catalog.md`](../../home-network-k3s/docs/13-service-catalog.md)
|
||
(the ServiceDefinition schema and default posture). This document maps each field of
|
||
[`servicedefinition.schema.json`](./servicedefinition.schema.json) to the concrete resources the
|
||
platform must render, and cites the requirement each mapping satisfies. Changing this contract is an
|
||
**interface change** — coordinate per the spec's `agents/coordination-protocol.md` (§2).
|
||
|
||
> The renderer MUST first materialize the schema **defaults** (below), then apply this mapping. A
|
||
> field absent from a ServiceDefinition renders from its default posture, not "not at all."
|
||
|
||
## Default posture (applied before mapping) — docs/13 §"Default posture"
|
||
| Field | Default | Requirement |
|
||
|---|---|---|
|
||
| `kind` | `container` | R-PLAT-02/03 |
|
||
| `exposure` | `internal` | R-DNS/R-NET |
|
||
| `egress` | `internet` | R-NET-05 |
|
||
| `storage` | `[]` (config PVC still expected for stateful svc) | R-STATE-02 |
|
||
| `dependencies` | `[]` → reaches **only DNS** (default-deny) | R-NET-01 |
|
||
| `auth.mode` | `forward-auth` for any exposed UI | R-IDP-02 |
|
||
| `update.channel` | `stable` | R-UPD-01 |
|
||
| `update.autoRollbackOnFailedHealth` | `true` | R-UPD-03 |
|
||
| `observability.metrics` | `true` | R-OBS-01 |
|
||
| backup of irreplaceable state | off-cluster (never `reacquirable`) | R-STATE-03 |
|
||
|
||
---
|
||
|
||
## Field → resource mapping
|
||
|
||
### `name`
|
||
- **Renders:** the service Namespace (trust zone), Deployment/VM name, Service name, and the DNS label
|
||
`<name>.home.keithjackson.dev`.
|
||
- **Requirements:** R-DNS-01 (naming model, docs/09), R-NET (namespace = isolation boundary, docs/03).
|
||
|
||
### `kind` + `source`
|
||
- `container` + `source.image` → **Deployment** (+ Service) using the image ref.
|
||
- `vm` + `source.vmTemplate` → **KubeVirt VirtualMachine** with a **CDI**-imported root disk.
|
||
- The schema enforces exactly-one-of (`image` for container, `vmTemplate` for vm).
|
||
- **Requirements:** R-PLAT-02/03 (PaaS/IaaS, docs/13); VM disks inherit Longhorn durability (docs/04).
|
||
|
||
### `exposure` (`internal` | `public`)
|
||
- Both classes render an **HTTPRoute** on the shared Gateway and are reachable on the LAN via internal
|
||
split-horizon DNS.
|
||
- `public` **additionally** signals the documented **manual edge port-forward** of 80/443 (and any
|
||
service-specific port, e.g. Plex 32400, Minecraft 25565) to the ingress VIP, and a **public**
|
||
ExternalDNS record for `<name>.home.keithjackson.dev`.
|
||
- `internal` renders **no public DNS record** and is never port-forwarded (PiHole resolves it to the
|
||
internal Gateway VIP).
|
||
- Exposure does **not** relax east-west default-deny (docs/09 §"Interaction with default-deny").
|
||
- **Requirements:** R-DNS-02/03 (docs/09), R-NET-04 (docs/03).
|
||
|
||
### `egress` (`internet` | `vpn`) — egress steering
|
||
- `internet` → pod egress allow to the default WAN route (node → SOHO edge → internet).
|
||
- `vpn` → the pod is **steered through the shared in-cluster Gluetun/PIA (WireGuard) egress gateway**
|
||
(Cilium Egress Gateway / routing). The pod gets **no independent egress path**; the
|
||
**kill-switch (fail-closed)** and allow rules are enforced at the gateway + namespace/policy level,
|
||
**outside** the workload.
|
||
- **Requirements:** **R-NET-05** (egress class), **R-NET-06** (VPN kill-switch, mandatory & central).
|
||
VPN-class set is normative: sonarr, radarr, jackett, qbittorrent (docs/03/13).
|
||
|
||
### `dependencies[]` (`to` / `port` / `protocol`) — east-west policy
|
||
- Each namespace first gets the **default-deny baseline** (deny all ingress + egress) applied by
|
||
GitOps at namespace creation, with a standing allow only for **DNS to the in-cluster resolver** and
|
||
for **observability scraping** from the monitoring namespace.
|
||
- Each `dependencies[]` entry renders one **CiliumNetworkPolicy** egress allow: this service → `to`
|
||
service on `port`/`protocol`. Everything undeclared stays denied.
|
||
- Empty `dependencies` ⇒ the service can reach **only DNS**.
|
||
- **Requirements:** **R-NET-01** (default-deny baseline), **R-NET-02** (explicit allow / deny
|
||
undeclared). Auditable via Hubble (R-SEC-03, R-OBS-04).
|
||
|
||
### `storage[]` (`name` / `size` / `class` / `mountPath` / `export` / `subPath` / `readOnly`) — PVs + PVCs
|
||
- `class: longhorn` → a **PVC** on the 3-replica Longhorn StorageClass sized by `size` (replicated
|
||
config/DB/app state), `ReadWriteOnce`, dynamically provisioned. For a `vm`, the disk is a Longhorn
|
||
volume via CDI.
|
||
- `class: bulk` → a **static, pre-bound external NFS PersistentVolume _and_ its PVC**. The entry's
|
||
`export` is the absolute path on the bulk server; the SERVER is never named in a ServiceDefinition
|
||
— it is the `${BULK_NFS_SERVER}` cluster-vars knob, so one authored definition promotes across
|
||
environments that reach the same storage by different addresses (**R-ENV-02**). Not 3× replicated:
|
||
durability is the storage host's ZFS pool, not Longhorn.
|
||
- `storageClassName: ""` on **both** PV and PVC (explicitly empty ⇒ statically provisioned; no
|
||
provisioner is ever asked to create anything, and the default-StorageClass annotation is
|
||
suppressed). Deliberately NOT the `bulk` StorageClass: a StorageClass `provisioner` is
|
||
**immutable**, so re-pointing that stub would make Flux fail permanently on it.
|
||
- Binding is pinned from both ends — `claimRef` on the PV, `volumeName` on the PVC — so no other
|
||
claim of matching size/mode can win a race for the volume.
|
||
- `persistentVolumeReclaimPolicy: Retain`, always. Bulk data pre-exists the cluster, is far larger
|
||
than the cluster, and may have consumers outside it; nothing here may reclaim it.
|
||
- Access mode follows the read/write posture and is always `…Many`, because one export is
|
||
consumed by several namespaces and several nodes at once: `ReadOnlyMany` when `readOnly`,
|
||
`ReadWriteMany` otherwise. `ReadWriteOnce` cannot express either.
|
||
- `mountOptions` are a **platform** decision held in the renderer, not a per-service field:
|
||
`nfsvers=4.1,hard,noatime,timeo=600,retrans=2`. NFSv4 carries mount and lock in-protocol, so
|
||
the only port that must be reachable is **tcp/2049** — no rpcbind/111, mountd or statd, which is
|
||
what keeps the firewall surface to a single rule. `hard` is chosen over `soft` because a soft
|
||
mount converts a server hiccup into a short read handed to the application.
|
||
- `size` on a bulk entry is the PV's **advertised capacity and nothing more** — no component
|
||
enforces it (the ZFS pool is the real limit), so it should state the dataset's actual size so
|
||
`kubectl get pv` reports something true.
|
||
- **fsGroup does not apply.** The in-tree NFS volume plugin performs no ownership management, so
|
||
`podSecurityContext.fsGroup` is a **no-op** on a bulk volume. A bulk mount that must be written
|
||
therefore requires the container's own uid/gid to be one the export's ownership permits (e.g.
|
||
Plex's `PLEX_UID`/`PLEX_GID`, LinuxServer images' `PUID`/`PGID`) — or a server-side squash. A
|
||
mount that succeeds but cannot write is the characteristic failure of this tier.
|
||
- `mountPath` (any class) overrides the `/data/<name>` default verbatim.
|
||
- `subPath` (any class) mounts a subtree instead of the volume root. **Two `subPath` mounts of one
|
||
export are two separate mounts**, so `link()` between them fails `EXDEV`; a workload that needs
|
||
hardlinks or atomic moves between two directories (Sonarr/Radarr importing from a download path
|
||
into a library) must mount their common ancestor **once** instead.
|
||
- `readOnly` (any class) marks the volumeMount read-only, and on a bulk entry also sets the PV's
|
||
`nfs.readOnly` so the kernel mount really is `ro`. Note this is a **client-side** guarantee: one
|
||
export serving both a reader and a writer that may land on any node cannot distinguish them
|
||
server-side.
|
||
- State separation rule: anything that must survive a rebuild lives on a named volume here, never on
|
||
ephemeral fs/hostPath — so the workload can be destroyed and re-applied from git with data intact.
|
||
- **Requirements:** **R-STATE-01** (rebuild survival), **R-STATE-02** (Longhorn replication),
|
||
**R-STATE-04** (bulk tier). (docs/04)
|
||
|
||
### `ingressRoutes[]` (`host` / `port`) — HTTPRoute + Certificate + DNS
|
||
- Each entry renders an **HTTPRoute** attaching `host` → the service `port` on the shared Gateway.
|
||
- TLS: the Gateway terminates with the cert-manager **wildcard `*.home.keithjackson.dev`** certificate
|
||
(Let's Encrypt via **DNS-01**, scoped Cloudflare token), so no per-service cert step — trusted,
|
||
non-self-signed, auto-renewed, including for `internal` services. Per-service `Certificate` only
|
||
where isolation warrants.
|
||
- DNS: **ExternalDNS** watches the HTTPRoute and manages the `<host>` record (public IP for `public`,
|
||
internal Gateway VIP via PiHole split-horizon for `internal`).
|
||
- The `host` pattern is enforced to `<label>.home.keithjackson.dev` by the schema.
|
||
- **Requirements:** **R-TLS-01/02/03** (docs/08), **R-DNS-01/02/03** (docs/09).
|
||
|
||
### `auth` (`mode` / `groups[]`) — identity integration (Keycloak / oauth2-proxy)
|
||
- `oidc` → register the app as a **Keycloak OIDC client** (declarative realm/client config, GitOps;
|
||
client secret SOPS-encrypted). App does its own login redirect.
|
||
- `forward-auth` → protect the service's `ingressRoutes` at the **Gateway** with **oauth2-proxy →
|
||
Keycloak** so auth happens before traffic reaches the app (bare UIs without usable OIDC).
|
||
- `native` → no platform SSO wiring; app owns its auth (documented exception, e.g. Plex/Seerr; and
|
||
Keycloak **itself** as the IdP, admin login local + MFA).
|
||
- `none` → no auth wiring (internal-only, discouraged).
|
||
- `groups[]` → the Keycloak groups permitted access, enforced by the OIDC client mappers /
|
||
oauth2-proxy `allowed_groups`. **Schema requires ≥1 group for `oidc`/`forward-auth`** — access is
|
||
"in an authorized group," not merely "logged in."
|
||
- Keycloak, oauth2-proxy, and app→Keycloak / proxy→Keycloak / Keycloak→DB flows are ordinary
|
||
workloads under default-deny with explicit allows (ties back to `dependencies`).
|
||
- **Requirements:** **R-IDP-02** (protect OIDC-capable and bare UIs), **R-IDP-06** (group-based
|
||
authorization). (docs/15, ADR-0014)
|
||
|
||
### `update` (`channel` / `range` / `cadence` / `autoRollbackOnFailedHealth` / `pin`)
|
||
- container `kind` → a Flux **ImagePolicy** (+ ImageRepository) encoding `channel`/`range`; Flux
|
||
image-automation advances the running version on its scan interval (`cadence`). `pin` freezes the
|
||
service (stops automation). No human/AI edits a tag.
|
||
- `vm` `kind` → in-guest unattended package updates on `cadence` (cloud-init), per docs/11 §3.
|
||
- `autoRollbackOnFailedHealth: true` → health-gated rollout that reverts on failed health and alerts.
|
||
- **Requirements:** **R-UPD-01** (stable only), **R-UPD-02** (no human/AI authoring), **R-UPD-03**
|
||
(health-gated auto-rollback). (docs/11, ADR-0009)
|
||
|
||
### `observability` (`metrics` / `slis[]`) — ServiceMonitor + alerts
|
||
- `metrics: true` → a **ServiceMonitor/PodMonitor** so Prometheus scrapes the service; logs are
|
||
shipped to Loki by the node/pod agent by default.
|
||
- each entry in `slis[]` → at least one **alert rule** (e.g. `up` → service-down alert) routed by
|
||
Alertmanager to email; a per-service Grafana dashboard row.
|
||
- **Requirements:** **R-OBS-01** (per-service SLIs/alerts). Platform-health alerts (R-OBS-04) are
|
||
rendered platform-wide, not per ServiceDefinition. (docs/10)
|
||
|
||
### `backup` (`policy`)
|
||
- a schedule ref (e.g. `daily-offcluster`) → **Velero + Longhorn** off-cluster backup schedule for
|
||
this service's volumes/namespace.
|
||
- literal `reacquirable` → **no** 1:1 backup (bulk media only). **Irreplaceable** state (configs, DBs,
|
||
Nextcloud, Gitea, PiHole config, Keycloak DB) MUST NOT be `reacquirable`.
|
||
- **Requirements:** **R-STATE-03** (off-cluster DR backup). (docs/04)
|
||
|
||
---
|
||
|
||
## Cross-field invariants the renderer / L0-L1 tests rely on
|
||
1. **kind ↔ source:** container⇒`image`, vm⇒`vmTemplate` (schema-enforced).
|
||
2. **auth ↔ groups:** `oidc`/`forward-auth` ⇒ non-empty `groups` (schema-enforced; R-IDP-06).
|
||
3. **default-deny is unconditional:** every namespace gets the deny baseline regardless of
|
||
`dependencies`; `dependencies` only *adds* allows (R-NET-01/02).
|
||
4. **egress is enforced outside the workload:** `vpn` renders gateway steering + kill-switch, never a
|
||
per-service sidecar the app could bypass (R-NET-05/06).
|
||
5. **TLS is never self-signed and needs no per-service step:** the wildcard Gateway cert covers every
|
||
`ingressRoutes.host`, `internal` included (R-TLS-01/03).
|
||
6. **irreplaceable ⇒ off-cluster backup:** DR-critical services must not carry `backup.policy:
|
||
reacquirable` (R-STATE-03). *(Advisory: not enforced by the schema — an L0 policy check should
|
||
assert it; see "Ambiguities / escalations" in schemas/README.md.)*
|
||
|
||
## Requirement traceability (contract-level)
|
||
Primary **R-GITOPS-02** (single reviewable object wires every requirement). Field-level: R-NET-01/02
|
||
(dependencies), R-NET-05/06 (egress), R-STATE-01/02/04 (storage), R-STATE-03 (backup),
|
||
R-TLS-01/02/03 + R-DNS-01/02/03 (ingressRoutes + exposure), R-OBS-01 (observability),
|
||
R-UPD-01/02/03 (update), R-IDP-02/06 (auth), R-PLAT-02/03 (kind/source).
|