jakman85andClaude Opus 5
cf7d97e546
Twelve `<app>-down` SLI rules were STRUCTURALLY INERT. Each read `up{job="<app>"} == 0`,
and the ServiceMonitor rendered beside it named `endpoints[].port: metrics` — a port NO
rendered app Service defined. Every monitor discovered ZERO targets, so no `up{job="<app>"}`
series existed and the rule could not fire. Verified on prod before this change:
`count by (job) (up)` listed 24 jobs and not one of the twelve appeared; `up{job="plex"}`
and `up{job="gitea"}` both returned nothing. `PrometheusTargetsMissing` could not see it
either — a target that was never discovered has no `up` series to be 0.
THE PORT NAME WAS NEVER THE MISSING PIECE. Ground truth per app: Plex, Pi-hole, Jackett,
qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export NO Prometheus metrics
without a separate exporter sidecar; Gitea can but only with app.ini metrics enabled;
Keycloak does, on the operator's management port, already scraped by
platform/keycloak/servicemonitor.yaml; vm-demo is a KubeVirt VM with no Service at all, so
its monitor selected nothing whatsoever. `metrics: true` was asserting a fact none of them
had. There was also a THIRD layer of inertness nobody had hit yet: every app namespace ships
default-deny with `ingress: []` and the renderer emitted no monitoring allow, so even a
correct port would have been dropped by policy — while renderer-contract.md claimed the
baseline carried such an allow.
FIXED IN THE GENERATOR, not in twelve rendered files:
* `slis: [up]` now renders a `<app>-down` alert against kube-state-metrics readiness of the
service's own workload object —
`kube_deployment_status_replicas_available{namespace,deployment} == 0 or absent(...)`,
`kube_statefulset_status_replicas_ready` for a StatefulSet, and
`absent(kubevirt_vmi_info{namespace,name,phase="running"})` for a VM. The same idiom
KeycloakDown has used since it was fixed. kube-state-metrics IS scraped, so this needs no
exporter and no new target. The `or absent(...)` half is load-bearing: the signal
DISAPPEARING now fires the alert instead of silencing it.
* schema 1.6.0 adds `observability.metricsEndpoint` — WHERE metrics are served. A
ServiceMonitor is rendered ONLY from it, and it renders all three halves of a working
scrape together: the named Service port, the monitor endpoint, and the
monitoring-namespace CiliumNetworkPolicy allow. No catalog entry declares one, so twelve
dead ServiceMonitors are gone. `metrics: true` keeps meaning logs-to-Loki plus
kube-state-metrics liveness.
* schema 1.6.0 also adds `observability.livenessWorkload`, and catalog/keycloak.yaml uses
it. Keycloak's server is a StatefulSet run by the operator; the default Deployment shape
would have made its `absent()` half fire FOREVER — the other failure mode this repo has
shipped.
* renderer selftest gains cases 9/9a/9b/9c: no metricsEndpoint => no monitor; the up SLI
must not read `up{job=...}`; endpoint port must exist on the Service; the metrics port
must NOT reach the Gateway allow; an unknown livenessWorkload.kind is REFUSED.
PROOF, both directions, for all twelve (apps/wave-{0,1,2}/tests/app-slis.test.yaml, run by
platform/monitoring/tests/run.sh over the RENDERED manifests):
* quiet on the healthy state; firing on no-ready-replica; firing when the series is ABSENT;
the VM asserted both gone and in a non-running phase; per-app isolation asserted, and
keycloak asserted un-maskable by its neighbours (keycloak-operator Deployment,
keycloak-db StatefulSet ready in the same namespace).
* mutation-checked: six mutations, each turning its suite red — reverting plex to
`up{job=...} == 0`; dropping seerr's `absent()` half; pointing keycloak at a Deployment;
capitalising the VMI phase to `Running`; widening keycloak's matcher to the namespace;
inverting plex's `== 0` to `!= 0`. Control run green.
* live on prod with `promtool query instant`: all ten deployed app Deployments export
`kube_deployment_status_replicas_available`,
`kube_statefulset_status_replicas_ready{namespace="keycloak",statefulset="keycloak"}` = 2,
`kubevirt_vmi_info` present with a lowercase `phase="running"`; and evaluating the twelve
SHIPPED expressions returns no series for eleven (quiet, not permanently firing). The
twelfth is vm-demo, which is not in apps/wave-0/kustomization.yaml and so is not deployed
— its rule is not applied either.
DELETION WAS NOT THE ANSWER, and the evidence is recorded rather than assumed:
kube-prometheus-stack's `kubernetes-apps` group IS live on prod (17 rules, KubePodNotReady /
KubeDeploymentReplicasMismatch among them), so app pods are not unmonitored — but that is a
generic warning keyed to a namespace, not the per-service CRITICAL SLI R-OBS-01 declares.
platform/monitoring/alerts.yaml carries no generic app-readiness rule at all.
GUARD SO IT CANNOT RECUR SILENTLY: new whole-repo L0 invariant
conformance/tests/l0-static/servicemonitor-ports-resolve. It walks the MONITORS, not the
alerts, so it needs no `up{job=...}` rule to reach one, and resolves each endpoint by SELECTOR
across the whole tree rather than by same-file adjacency. Fixing the alerts DELETED the only
thing that made the broken monitors visible to the sibling check, which is exactly why this
axis needed its own home. It declares `conformance.home.k3s/scope: whole-repo` and adds no
ci/l0.sh stage. Proven to FAIL on the pre-fix tree: 15 failures — eleven port disagreements,
vm-demo's monitor matching no Service at all, and the three generator/contract assertions —
and to pass after. Its resolver is additionally proven on in-check fixtures (agreeing pair
accepted, disagreeing pair rejected, target-less endpoint rejected), because after the fix
there are zero in-repo monitors left for it to judge and a guard whose only evidence is
"nothing failed" cannot be told apart from one that stopped working. It also asserts
r_servicemonitor does not HARDCODE a port name — the mechanism by which one edit gave twelve
services the same wrong one.
Also fixed, found on the way:
* platform/monitoring/tests/run.sh discovered the SUITE convention only inside
platform/monitoring/tests/ while documenting it as general — a suite added anywhere else
was never run and still looked like coverage. Now discovered tree-wide; a misplaced
*.test.yaml fails loudly instead of vanishing.
* KNOWN_UNWIRED_JOBS in alert-metrics-have-scrape-targets is now EMPTY, and the twelve
apps/* entries in alert-rules-unit-tested's UNTESTED_FILES are gone — both because the
gaps were CLOSED, not waived. Both registers are shrink-only and a stale entry fails.
* renderer-contract.md claimed the default-deny baseline carried a standing monitoring
scrape allow. It never did; the claim is corrected and the allow is now real, per declared
metrics endpoint and scoped to that one port.
Co-Authored-By: Claude Opus 5 <[email protected]>
251 lines
19 KiB
Markdown
251 lines
19 KiB
Markdown
# Renderer Contract — ServiceDefinition → rendered resources
|
||
|
||
**Status:** authoritative Phase-0 interface contract. **Consumed by** the manifest renderer
|
||
(WS-GITOPS), the network/egress renderer (WS-NET), storage (WS-STATE), TLS/DNS (WS-EDGE), identity
|
||
(WS-SEC), observability (WS-OBS), updates (WS-UPD), and the conformance/policy/CI agents (WS-TEST).
|
||
|
||
**Source of truth:** spec [`docs/13-service-catalog.md`](../../home-network-k3s/docs/13-service-catalog.md)
|
||
(the ServiceDefinition schema and default posture). This document maps each field of
|
||
[`servicedefinition.schema.json`](./servicedefinition.schema.json) to the concrete resources the
|
||
platform must render, and cites the requirement each mapping satisfies. Changing this contract is an
|
||
**interface change** — coordinate per the spec's `agents/coordination-protocol.md` (§2).
|
||
|
||
> The renderer MUST first materialize the schema **defaults** (below), then apply this mapping. A
|
||
> field absent from a ServiceDefinition renders from its default posture, not "not at all."
|
||
|
||
## Default posture (applied before mapping) — docs/13 §"Default posture"
|
||
| Field | Default | Requirement |
|
||
|---|---|---|
|
||
| `kind` | `container` | R-PLAT-02/03 |
|
||
| `exposure` | `internal` | R-DNS/R-NET |
|
||
| `egress` | `internet` | R-NET-05 |
|
||
| `storage` | `[]` (config PVC still expected for stateful svc) | R-STATE-02 |
|
||
| `dependencies` | `[]` → reaches **only DNS** (default-deny) | R-NET-01 |
|
||
| `inClusterClients` | `[]` → no direct east-west client; only the Gateway may reach an exposed port | R-NET-01/02 |
|
||
| `auth.mode` | `forward-auth` for any exposed UI | R-IDP-02 |
|
||
| `update.channel` | `stable` | R-UPD-01 |
|
||
| `update.autoRollbackOnFailedHealth` | `true` | R-UPD-03 |
|
||
| `observability.metrics` | `true` | R-OBS-01 |
|
||
| `observability.metricsEndpoint` | **absent** → the workload serves no Prometheus endpoint; no ServiceMonitor | R-OBS-01 |
|
||
| `observability.livenessWorkload` | the object this SD renders (`Deployment` named after the service; `VirtualMachineInstance` for `kind: vm`) | R-OBS-01 |
|
||
| backup of irreplaceable state | off-cluster (never `reacquirable`) | R-STATE-03 |
|
||
|
||
---
|
||
|
||
## Field → resource mapping
|
||
|
||
### `name`
|
||
- **Renders:** the service Namespace (trust zone), Deployment/VM name, Service name, and the DNS label
|
||
`<name>.home.keithjackson.dev`.
|
||
- **Requirements:** R-DNS-01 (naming model, docs/09), R-NET (namespace = isolation boundary, docs/03).
|
||
|
||
### `kind` + `source`
|
||
- `container` + `source.image` → **Deployment** (+ Service) using the image ref.
|
||
- `vm` + `source.vmTemplate` → **KubeVirt VirtualMachine** with a **CDI**-imported root disk.
|
||
- The schema enforces exactly-one-of (`image` for container, `vmTemplate` for vm).
|
||
- **Requirements:** R-PLAT-02/03 (PaaS/IaaS, docs/13); VM disks inherit Longhorn durability (docs/04).
|
||
|
||
### `exposure` (`internal` | `public`)
|
||
- Both classes render an **HTTPRoute** on the shared Gateway and are reachable on the LAN via internal
|
||
split-horizon DNS.
|
||
- `public` **additionally** signals the documented **manual edge port-forward** of 80/443 (and any
|
||
service-specific port, e.g. Plex 32400, Minecraft 25565) to the ingress VIP, and a **public**
|
||
ExternalDNS record for `<name>.home.keithjackson.dev`.
|
||
- `internal` renders **no public DNS record** and is never port-forwarded (PiHole resolves it to the
|
||
internal Gateway VIP).
|
||
- Exposure does **not** relax east-west default-deny (docs/09 §"Interaction with default-deny").
|
||
- **Requirements:** R-DNS-02/03 (docs/09), R-NET-04 (docs/03).
|
||
|
||
### `egress` (`internet` | `vpn`) — egress steering
|
||
- `internet` → pod egress allow to the default WAN route (node → SOHO edge → internet).
|
||
- `vpn` → the pod is **steered through the shared in-cluster Gluetun/PIA (WireGuard) egress gateway**
|
||
(Cilium Egress Gateway / routing). The pod gets **no independent egress path**; the
|
||
**kill-switch (fail-closed)** and allow rules are enforced at the gateway + namespace/policy level,
|
||
**outside** the workload.
|
||
- **Requirements:** **R-NET-05** (egress class), **R-NET-06** (VPN kill-switch, mandatory & central).
|
||
VPN-class set is normative: sonarr, radarr, jackett, qbittorrent (docs/03/13).
|
||
|
||
### `dependencies[]` (`to` / `port` / `protocol`) — east-west policy
|
||
- Each namespace first gets the **default-deny baseline** (deny all ingress + egress) applied by
|
||
GitOps at namespace creation, with a standing allow only for **DNS to the in-cluster resolver**.
|
||
**Observability scraping is NOT in the baseline** — it is an additive allow rendered per declared
|
||
`observability.metricsEndpoint` (see below), scoped to that one port. This paragraph used to claim
|
||
the baseline carried a standing monitoring allow; it never did, and an app metrics port would have
|
||
been dropped by policy even with the Service port and monitor endpoint in agreement.
|
||
- Each `dependencies[]` entry renders one **CiliumNetworkPolicy** egress allow: this service → `to`
|
||
service on `port`/`protocol`. Everything undeclared stays denied.
|
||
- Empty `dependencies` ⇒ the service can reach **only DNS**.
|
||
- **Requirements:** **R-NET-01** (default-deny baseline), **R-NET-02** (explicit allow / deny
|
||
undeclared). Auditable via Hubble (R-SEC-03, R-OBS-04).
|
||
|
||
### `storage[]` (`name` / `size` / `class` / `mountPath` / `export` / `subPath` / `readOnly`) — PVs + PVCs
|
||
- `class: longhorn` → a **PVC** on the 3-replica Longhorn StorageClass sized by `size` (replicated
|
||
config/DB/app state), `ReadWriteOnce`, dynamically provisioned. For a `vm`, the disk is a Longhorn
|
||
volume via CDI.
|
||
- `class: bulk` → a **static, pre-bound external NFS PersistentVolume _and_ its PVC**. The entry's
|
||
`export` is the absolute path on the bulk server; the SERVER is never named in a ServiceDefinition
|
||
— it is the `${BULK_NFS_SERVER}` cluster-vars knob, so one authored definition promotes across
|
||
environments that reach the same storage by different addresses (**R-ENV-02**). Not 3× replicated:
|
||
durability is the storage host's ZFS pool, not Longhorn.
|
||
- `storageClassName: ""` on **both** PV and PVC (explicitly empty ⇒ statically provisioned; no
|
||
provisioner is ever asked to create anything, and the default-StorageClass annotation is
|
||
suppressed). Deliberately NOT the `bulk` StorageClass: a StorageClass `provisioner` is
|
||
**immutable**, so re-pointing that stub would make Flux fail permanently on it.
|
||
- Binding is pinned from both ends — `claimRef` on the PV, `volumeName` on the PVC — so no other
|
||
claim of matching size/mode can win a race for the volume.
|
||
- `persistentVolumeReclaimPolicy: Retain`, always. Bulk data pre-exists the cluster, is far larger
|
||
than the cluster, and may have consumers outside it; nothing here may reclaim it.
|
||
- Access mode follows the read/write posture and is always `…Many`, because one export is
|
||
consumed by several namespaces and several nodes at once: `ReadOnlyMany` when `readOnly`,
|
||
`ReadWriteMany` otherwise. `ReadWriteOnce` cannot express either.
|
||
- `mountOptions` are a **platform** decision held in the renderer, not a per-service field:
|
||
`nfsvers=4.1,hard,noatime,timeo=600,retrans=2`. NFSv4 carries mount and lock in-protocol, so
|
||
the only port that must be reachable is **tcp/2049** — no rpcbind/111, mountd or statd, which is
|
||
what keeps the firewall surface to a single rule. `hard` is chosen over `soft` because a soft
|
||
mount converts a server hiccup into a short read handed to the application.
|
||
- `size` on a bulk entry is the PV's **advertised capacity and nothing more** — no component
|
||
enforces it (the ZFS pool is the real limit), so it should state the dataset's actual size so
|
||
`kubectl get pv` reports something true.
|
||
- **fsGroup does not apply.** The in-tree NFS volume plugin performs no ownership management, so
|
||
`podSecurityContext.fsGroup` is a **no-op** on a bulk volume. A bulk mount that must be written
|
||
therefore requires the container's own uid/gid to be one the export's ownership permits (e.g.
|
||
Plex's `PLEX_UID`/`PLEX_GID`, LinuxServer images' `PUID`/`PGID`) — or a server-side squash. A
|
||
mount that succeeds but cannot write is the characteristic failure of this tier.
|
||
- `mountPath` (any class) overrides the `/data/<name>` default verbatim.
|
||
- `subPath` (any class) mounts a subtree instead of the volume root. **Two `subPath` mounts of one
|
||
export are two separate mounts**, so `link()` between them fails `EXDEV`; a workload that needs
|
||
hardlinks or atomic moves between two directories (Sonarr/Radarr importing from a download path
|
||
into a library) must mount their common ancestor **once** instead.
|
||
- `readOnly` (any class) marks the volumeMount read-only, and on a bulk entry also sets the PV's
|
||
`nfs.readOnly` so the kernel mount really is `ro`. Note this is a **client-side** guarantee: one
|
||
export serving both a reader and a writer that may land on any node cannot distinguish them
|
||
server-side.
|
||
- State separation rule: anything that must survive a rebuild lives on a named volume here, never on
|
||
ephemeral fs/hostPath — so the workload can be destroyed and re-applied from git with data intact.
|
||
- **Requirements:** **R-STATE-01** (rebuild survival), **R-STATE-02** (Longhorn replication),
|
||
**R-STATE-04** (bulk tier). (docs/04)
|
||
|
||
### `ingressRoutes[]` (`host` / `port`) — HTTPRoute + Certificate + DNS
|
||
- Each entry renders an **HTTPRoute** attaching `host` → the service `port` on the shared Gateway.
|
||
- TLS: the Gateway terminates with the cert-manager **wildcard `*.home.keithjackson.dev`** certificate
|
||
(Let's Encrypt via **DNS-01**, scoped Cloudflare token), so no per-service cert step — trusted,
|
||
non-self-signed, auto-renewed, including for `internal` services. Per-service `Certificate` only
|
||
where isolation warrants.
|
||
- DNS: **ExternalDNS** watches the HTTPRoute and manages the `<host>` record (public IP for `public`,
|
||
internal Gateway VIP via PiHole split-horizon for `internal`).
|
||
- The `host` pattern is enforced to `<label>.home.keithjackson.dev` by the schema.
|
||
- **Requirements:** **R-TLS-01/02/03** (docs/08), **R-DNS-01/02/03** (docs/09).
|
||
|
||
### `inClusterClients[]` (`namespace` / `port` / `protocol` / `name`) — direct east-west clients
|
||
- Each entry renders ONE **CiliumNetworkPolicy** ingress allow on this service:
|
||
`endpointSelector` = this service, `fromEndpoints` matching **only** the reserved
|
||
`k8s:io.kubernetes.pod.namespace: <namespace>` label, `toPorts` carrying **only** the declared
|
||
`port`/`protocol`. Name: `<service>-allow-<name>-ingress`, `name` defaulting to `namespace`.
|
||
- **Why the field exists.** `ingressRoutes` opens the **north-south** path only (the gateway allow
|
||
admits `fromEntities: [ingress]`, the Gateway proxy identity, and nothing else), and
|
||
`dependencies` compiles edges **only between catalog services** — both halves come from a catalog
|
||
entry's own declaration. A **platform controller** that dials this service's ClusterIP directly
|
||
was therefore unauthorable, and under default-deny it is dropped. Prod hit exactly that: its
|
||
`GitRepository` points at the in-cluster Gitea Service
|
||
(`http://gitea.gitea.svc.cluster.local:3000/...`, clusters/prod/kustomization.yaml) so GitOps
|
||
depends on no public DNS/Cloudflare/Gateway/`${BASE_DOMAIN}`, but `flux-system`'s
|
||
source-controller was denied — and the HTTPRoute that would have served a public name lives in a
|
||
**downstream** tier that cannot apply while the source is down (source down → routes never rename
|
||
→ source stays down).
|
||
- **Least privilege is part of the contract:** a namespace and a port, never `fromEntities: all`,
|
||
never a port-less allow. The client is selected by **namespace** rather than pod labels because an
|
||
upstream controller's labels are its chart's to change, not this platform's contract — so name a
|
||
platform-controller namespace whose entire membership may hold that reach, never a shared one.
|
||
- **Additive, and not a dependency:** the namespace default-deny baseline is unchanged
|
||
(`ingress: []`), every undeclared source/port stays denied, and the entry creates **no wave
|
||
ordering edge** — the client is a platform component, not a catalog service.
|
||
- **Requirements:** **R-NET-01** (default-deny baseline stands), **R-NET-02** (explicit allow / deny
|
||
undeclared), **R-GITOPS-02** (the allowance is authored in the one reviewable object rather than
|
||
surviving as a live-applied cluster object). Auditable via Hubble (R-SEC-03, R-OBS-04).
|
||
|
||
### `auth` (`mode` / `groups[]`) — identity integration (Keycloak / oauth2-proxy)
|
||
- `oidc` → register the app as a **Keycloak OIDC client** (declarative realm/client config, GitOps;
|
||
client secret SOPS-encrypted). App does its own login redirect.
|
||
- `forward-auth` → protect the service's `ingressRoutes` at the **Gateway** with **oauth2-proxy →
|
||
Keycloak** so auth happens before traffic reaches the app (bare UIs without usable OIDC).
|
||
- `native` → no platform SSO wiring; app owns its auth (documented exception, e.g. Plex/Seerr; and
|
||
Keycloak **itself** as the IdP, admin login local + MFA).
|
||
- `none` → no auth wiring (internal-only, discouraged).
|
||
- `groups[]` → the Keycloak groups permitted access, enforced by the OIDC client mappers /
|
||
oauth2-proxy `allowed_groups`. **Schema requires ≥1 group for `oidc`/`forward-auth`** — access is
|
||
"in an authorized group," not merely "logged in."
|
||
- Keycloak, oauth2-proxy, and app→Keycloak / proxy→Keycloak / Keycloak→DB flows are ordinary
|
||
workloads under default-deny with explicit allows (ties back to `dependencies`).
|
||
- **Requirements:** **R-IDP-02** (protect OIDC-capable and bare UIs), **R-IDP-06** (group-based
|
||
authorization). (docs/15, ADR-0014)
|
||
|
||
### `update` (`channel` / `range` / `cadence` / `autoRollbackOnFailedHealth` / `pin`)
|
||
- container `kind` → a Flux **ImagePolicy** (+ ImageRepository) encoding `channel`/`range`; Flux
|
||
image-automation advances the running version on its scan interval (`cadence`). `pin` freezes the
|
||
service (stops automation). No human/AI edits a tag.
|
||
- `vm` `kind` → in-guest unattended package updates on `cadence` (cloud-init), per docs/11 §3.
|
||
- `autoRollbackOnFailedHealth: true` → health-gated rollout that reverts on failed health and alerts.
|
||
- **Requirements:** **R-UPD-01** (stable only), **R-UPD-02** (no human/AI authoring), **R-UPD-03**
|
||
(health-gated auto-rollback). (docs/11, ADR-0009)
|
||
|
||
### `observability` (`metrics` / `slis[]` / `metricsEndpoint` / `livenessWorkload`) — ServiceMonitor + alerts
|
||
- `metrics: true` → the platform **collects signals** for this service: logs to Loki via the node
|
||
agent, and kube-state-metrics readiness for the `up` SLI below. Both are platform-wide scrapes;
|
||
neither needs a per-service CR. **It does not assert that the workload serves a Prometheus
|
||
endpoint.** *(Corrected 2026-09-13. It used to render a ServiceMonitor unconditionally, hardcoded
|
||
to a Service port named `metrics`. No rendered Service ever defined such a port, so all twelve app
|
||
monitors discovered ZERO targets — verified on prod: 24 jobs in `up`, not one of the twelve — and
|
||
every `up{job=...}` SLI built on them was structurally inert. `metrics: true` could not have been
|
||
right: Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud serve no
|
||
Prometheus endpoint at all without a separate exporter.)*
|
||
- `metricsEndpoint: {port, name?, path?, interval?}` → **the workload does serve metrics, here.**
|
||
Renders all three halves of a working scrape from the one declaration, because any one of them
|
||
missing makes the scrape silently yield nothing:
|
||
1. a **named Service port** (`name`, default `metrics`) on that `port`;
|
||
2. a **ServiceMonitor** endpoint selecting that port name, on `path` at `interval`;
|
||
3. a **CiliumNetworkPolicy** admitting the `monitoring` namespace to that port — the namespace
|
||
default-deny baseline is `ingress: []`, so without it the scrape is dropped by policy.
|
||
The metrics port is deliberately **not** added to the Gateway ingress allow: it is east-west for
|
||
Prometheus, never north-south.
|
||
- each entry in `slis[]` → at least one **alert rule** routed by Alertmanager to email; a per-service
|
||
Grafana dashboard row. `up` → a `<name>-down` **critical** alert reading the readiness of
|
||
`livenessWorkload` from **kube-state-metrics** (`kube_deployment_status_replicas_available`,
|
||
`kube_statefulset_status_replicas_ready`, `kube_daemonset_status_number_available`) or, for a
|
||
`VirtualMachineInstance`, the existence of `kubevirt_vmi_info{phase="running"}`. Each rule carries
|
||
an `or absent(...)` half, so the signal DISAPPEARING fires the alert rather than silencing it.
|
||
- `livenessWorkload: {kind, name}` → which object's readiness **is** this service. Defaults to the
|
||
object this SD renders; override it when the running workload is a different one (keycloak's server
|
||
is a StatefulSet owned by the Keycloak operator — a Deployment-shaped rule would `absent()` and
|
||
fire forever).
|
||
- **Requirements:** **R-OBS-01** (per-service SLIs/alerts). Platform-health alerts (R-OBS-04) are
|
||
rendered platform-wide, not per ServiceDefinition. (docs/10)
|
||
|
||
### `backup` (`policy`)
|
||
- a schedule ref (e.g. `daily-offcluster`) → **Velero + Longhorn** off-cluster backup schedule for
|
||
this service's volumes/namespace.
|
||
- literal `reacquirable` → **no** 1:1 backup (bulk media only). **Irreplaceable** state (configs, DBs,
|
||
Nextcloud, Gitea, PiHole config, Keycloak DB) MUST NOT be `reacquirable`.
|
||
- **Requirements:** **R-STATE-03** (off-cluster DR backup). (docs/04)
|
||
|
||
---
|
||
|
||
## Cross-field invariants the renderer / L0-L1 tests rely on
|
||
1. **kind ↔ source:** container⇒`image`, vm⇒`vmTemplate` (schema-enforced).
|
||
2. **auth ↔ groups:** `oidc`/`forward-auth` ⇒ non-empty `groups` (schema-enforced; R-IDP-06).
|
||
3. **default-deny is unconditional:** every namespace gets the deny baseline regardless of
|
||
`dependencies`; `dependencies` only *adds* allows (R-NET-01/02). The same holds for
|
||
`inClusterClients`: it adds one namespace+port ingress allow and never relaxes the baseline.
|
||
4. **egress is enforced outside the workload:** `vpn` renders gateway steering + kill-switch, never a
|
||
per-service sidecar the app could bypass (R-NET-05/06).
|
||
5. **TLS is never self-signed and needs no per-service step:** the wildcard Gateway cert covers every
|
||
`ingressRoutes.host`, `internal` included (R-TLS-01/03).
|
||
6. **irreplaceable ⇒ off-cluster backup:** DR-critical services must not carry `backup.policy:
|
||
reacquirable` (R-STATE-03). *(Advisory: not enforced by the schema — an L0 policy check should
|
||
assert it; see "Ambiguities / escalations" in schemas/README.md.)*
|
||
|
||
## Requirement traceability (contract-level)
|
||
Primary **R-GITOPS-02** (single reviewable object wires every requirement). Field-level: R-NET-01/02
|
||
(dependencies, inClusterClients), R-NET-05/06 (egress), R-STATE-01/02/04 (storage), R-STATE-03 (backup),
|
||
R-TLS-01/02/03 + R-DNS-01/02/03 (ingressRoutes + exposure), R-OBS-01 (observability),
|
||
R-UPD-01/02/03 (update), R-IDP-02/06 (auth), R-PLAT-02/03 (kind/source).
|