feat(pihole): two replicas on different nodes, and fix a PVC mounted where pihole never writes

Phase 2 of making LAN DNS survive a node failure. Pihole was one pod on one
node; measured 2026-10-04, losing that node meant roughly 6-10 minutes with no
LAN resolution (tolerationSeconds 300 + NotReady detection + a Longhorn RWO
force-detach), while losing the node holding the pihole-dns L2 lease -- a
DIFFERENT node -- meant tens of seconds. Two of three nodes were single points
of failure with wildly different blast radii.

Adds `availability.replicas` to the ServiceDefinition schema (1.10.0). Intent
only; the mechanism is render.py's, documented in the renderer contract. 1 or
absent renders a Deployment exactly as before -- verified byte-identical across
the whole apps/ tree. >1 renders a StatefulSet (a longhorn PVC is RWO, so a
multi-replica Deployment never schedules the second pod), per-replica
volumeClaimTemplates, REQUIRED hostname anti-affinity, podManagementPolicy
Parallel, a minAvailable:1 PodDisruptionBudget, and an SLI that follows the
rendered kind (a Deployment-shaped rule against a StatefulSet absent()s forever).

Pinned by ~20 new --selftest assertions rather than a conformance change.

ALSO FIXES A LIVE BUG. Pihole's config PVC was mounted at the renderer default
/data/config, but pihole writes to /etc/pihole. Measured: /etc/pihole (7.1M --
gravity.db, pihole.toml, query DB) was on the OVERLAY, while the 2Gi Longhorn
volume held lost+found and 24K of nothing. So every restart discarded all state
and re-downloaded 72k gravity domains, and backup.policy has been backing up an
empty volume while reporting success. Fixed here because otherwise the new
per-replica volumes would hold nothing while state stayed ephemeral -- and
because persisting gravity.db is what makes a restart cheap.

The /etc/pihole mount is safe on the image's own evidence: start.sh is PID 1 as
uid 0 and chowns the directory recursively (no fsGroup needed), and it calls
migrate_gravity which creates the database when absent.

The gate caught a real regression on the first run: the wave-0 alert unit test
fed pihole a Deployment-shaped series. Updated; promtool SUCCESS.

Touches no sensitive paths, so this one needs no override.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-04 01:31:52 -05:00
1 parent 17f0c11403
commit 8e05635c33
7 files changed
+890 -114

No files matched your search

+309 -70
View File
@@ -2,10 +2,13 @@
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://home.keithjackson.dev/schemas/servicedefinition.schema.json",
"title": "ServiceDefinition",
"description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.9.0 (adds optional `auth.apiRoutes` -- a per-service list of anchored path regexes the forward-auth oauth2-proxy sidecar answers with 401 instead of a 302 to Keycloak, rendered as one repeatable `--api-route` per entry. The `auth` block stays `additionalProperties: false`; this is one named, documented key, not an opened door. Without it there was NO per-app override hook at all, so the only way to answer an app's XHRs with 401 was a renderer-wide constant -- and the path shape is a property of the app: jackett's is `/api/` (33 x 302 vs 2 x 401 measured 2026-09-30, 4 of 6 POSTs answered by the proxy and never reaching the app), radarr's is `/signalr/` (232 x 302 measured 2026-10-01, every one a POST from a left-open browser tab that is never re-prompted), and pihole serves no `/api/` at all. Each entry must be anchored at `^/` and continue with literal path characters, because a too-broad regex turns a plain page load into a 401 and the service looks permanently broken to a logged-out user. Enforced on every gate run by ci/l0.sh's schema stage over catalog/*.yaml, by catalog/render.py refusing to render an unanchored entry, and by conformance/tests/l0-static/oauth2-proxy-api-routes-per-app)). 1.8.0 (adds optional `hardware` - a host HARDWARE CAPABILITY declaration for the container kind, currently one member: `hardware.gpu: intel-quicksync`, meaning the workload needs the host iGPU render device for hardware video transcode. It is an ENUM OF INTENT, not a device path: the renderer satisfies it today with just two things -- a /dev/dri hostPath volume (`DirectoryOrCreate`) and a read-write /dev/dri volumeMount on the app container -- and a later move to the Intel GPU device plugin is then a render.py change with no catalog edit. It sets NO podSecurityContext.supplementalGroups: measured in-container on prod 2026-09-30, the pms-docker cont-init hook adds the app user to the device's group inside the container and `s6-setuidgid` then REPLACES the process group list from /etc/group, discarding anything Kubernetes supplied, so such an entry would be inert and misleading. Renders NOTHING when absent, so every existing app stays byte-identical. Emits no privileged securityContext (Kyverno restrict-host-namespaces would reject it and it is not needed), no nodeSelector (node placement is a separate concern with its own justification) and no resource requests). 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.md already said `internal` renders no public DNS record; nothing enforced it, and six internal services -- jackett, pihole, qbittorrent, radarr, semaphore, sonarr -- were publishing `<name>.home.keithjackson.dev` into the PUBLIC Cloudflare zone answering with the RFC1918 Gateway VIP, which published the internal subnet and a full service inventory and made each name a DNS-rebind primitive. Enforced on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns). 1.6.0 (splits `observability.metrics` from WHERE metrics live: adds optional `metricsEndpoint` (port/name/path/interval) and optional `livenessWorkload` (kind/name). `metrics: true` previously rendered a ServiceMonitor hardcoded to a port named `metrics` that no rendered Service ever defined, so all twelve app monitors discovered ZERO targets and every `up{job=...}` SLI was structurally inert on prod; a monitor is now rendered only from a declared `metricsEndpoint`, which also renders the Service port and the monitoring-namespace ingress allow, and the `up` SLI is expressed against kube-state-metrics -- a signal that exists for every workload without a per-app exporter. `livenessWorkload` names the object whose readiness IS the service, for the case where the running workload is not the one this SD renders). 1.5.0 (adds optional `inClusterClients` — additional in-cluster clients, named by NAMESPACE, admitted to a declared port of this service DIRECTLY, bypassing the shared Gateway. `ingressRoutes` admits only the Gateway proxy identity and `dependencies` only links catalog services, so a PLATFORM controller dialling a Service ClusterIP had no authorable form at all; that gap is what deadlocked proLine truncated
"description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.10.0 (adds optional `availability` for the container kind, one member: `availability.replicas`. It is AVAILABILITY INTENT, not a scheduling primitive -- it names no Kubernetes object, no topology key and no budget. 1 (the default) renders a Deployment exactly as before, so every pre-existing app stays byte-identical; >1 means the service must survive the loss of a single node, and the renderer satisfies that with a StatefulSet whose longhorn volumes become per-replica volumeClaimTemplates (the config PVC is ReadWriteOnce, so a multi-replica Deployment is UNSCHEDULABLE, not merely suboptimal), hard podAntiAffinity on kubernetes.io/hostname, podManagementPolicy Parallel, and a PodDisruptionBudget of minAvailable 1. Added for pihole, where LAN DNS -- a critical network service -- was one pod on one node: measured 2026-10-04, losing the node running it meant roughly 6-10 minutes with no LAN resolution (tolerationSeconds 300 plus node-NotReady detection plus a Longhorn RWO force-detach). It is deliberately NOT usable until a service's state is reproducible from git, because a replica with a fresh volume otherwise comes up serving but unconfigured -- for pihole that prerequisite was done first (decisions/2026-10-04-pihole-declare-state.md). Capped at 5: a count above the node count can never schedule under the anti-affinity this implies. Mechanism documented in schemas/renderer-contract.md `### availability`). 1.9.0 (adds optional `auth.apiRoutes` -- a per-service list of anchored path regexes the forward-auth oauth2-proxy sidecar answers with 401 instead of a 302 to Keycloak, rendered as one repeatable `--api-route` per entry. The `auth` block stays `additionalProperties: false`; this is one named, documented key, not an opened door. Without it there was NO per-app override hook at all, so the only way to answer an app's XHRs with 401 was a renderer-wide constant -- and the path shape is a property of the app: jackett's is `/api/` (33 x 302 vs 2 x 401 measured 2026-09-30, 4 of 6 POSTs answered by the proxy and never reaching the app), radarr's is `/signalr/` (232 x 302 measured 2026-10-01, every one a POST from a left-open browser tab that is never re-prompted), and pihole serves no `/api/` at all. Each entry must be anchored at `^/` and continue with literal path characters, because a too-broad regex turns a plain page load into a 401 and the service looks permanently broken to a logged-out user. Enforced on every gate run by ci/l0.sh's schema stage over catalog/*.yaml, by catalog/render.py refusing to render an unanchored entry, and by conformance/tests/l0-static/oauth2-proxy-api-routes-per-app)). 1.8.0 (adds optional `hardware` - a host HARDWARE CAPABILITY declaration for the container kind, currently one member: `hardware.gpu: intel-quicksync`, meaning the workload needs the host iGPU render device for hardware video transcode. It is an ENUM OF INTENT, not a device path: the renderer satisfies it today with just two things -- a /dev/dri hostPath volume (`DirectoryOrCreate`) and a read-write /dev/dri volumeMount on the app container -- and a later move to the Intel GPU device plugin is then a render.py change with no catalog edit. It sets NO podSecurityContext.supplementalGroups: measured in-container on prod 2026-09-30, the pms-docker cont-init hook adds the app user to the device's group inside the container and `s6-setuidgid` then REPLACES the process group list from /etc/group, discarding anything Kubernetes supplied, so such an entry would be inert and misleading. Renders NOTHING when absent, so every existing app stays byte-identical. Emits no privileged securityContext (Kyverno restrict-host-namespaces would reject it and it is not needed), no nodeSelector (node placement is a separate concern with its own justification) and no resource requests). 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.md already said `internal` renders no public DNS record; nothing enforced it, and six internal services -- jackett, pihole, qbittorrent, radarr, semaphore, sonarr -- were publishing `<name>.home.keithjackson.dev` into the PUBLIC Cloudflare zone answering with the RFC1918 Gateway VLine truncated
"type": "object",
"additionalProperties": false,
"required": ["name", "source"],
"required": [
"name",
"source"
],
"properties": {
"name": {
"type": "string",
@@ -17,7 +20,10 @@
"kind": {
"type": "string",
"description": "PaaS container (default) or IaaS VM via KubeVirt (docs/13; R-PLAT-02/03).",
"enum": ["container", "vm"],
"enum": [
"container",
"vm"
],
"default": "container"
},
"source": {
@@ -45,7 +51,9 @@
"items": {
"type": "object",
"additionalProperties": false,
"required": ["name"],
"required": [
"name"
],
"properties": {
"name": {
"type": "string",
@@ -66,27 +74,63 @@
"secretKeyRef": {
"type": "object",
"additionalProperties": false,
"required": ["name", "key"],
"required": [
"name",
"key"
],
"properties": {
"name": { "type": "string", "minLength": 1 },
"key": { "type": "string", "minLength": 1 }
"name": {
"type": "string",
"minLength": 1
},
"key": {
"type": "string",
"minLength": 1
}
}
},
"configMapKeyRef": {
"type": "object",
"additionalProperties": false,
"required": ["name", "key"],
"required": [
"name",
"key"
],
"properties": {
"name": { "type": "string", "minLength": 1 },
"key": { "type": "string", "minLength": 1 }
"name": {
"type": "string",
"minLength": 1
},
"key": {
"type": "string",
"minLength": 1
}
}
}
}
}
},
"oneOf": [
{ "required": ["value"], "not": { "required": ["valueFrom"] } },
{ "required": ["valueFrom"], "not": { "required": ["value"] } }
{
"required": [
"value"
],
"not": {
"required": [
"valueFrom"
]
}
},
{
"required": [
"valueFrom"
],
"not": {
"required": [
"value"
]
}
}
]
}
},
@@ -103,14 +147,28 @@
"secretRef": {
"type": "object",
"additionalProperties": false,
"required": ["name"],
"properties": { "name": { "type": "string", "minLength": 1 } }
"required": [
"name"
],
"properties": {
"name": {
"type": "string",
"minLength": 1
}
}
},
"configMapRef": {
"type": "object",
"additionalProperties": false,
"required": ["name"],
"properties": { "name": { "type": "string", "minLength": 1 } }
"required": [
"name"
],
"properties": {
"name": {
"type": "string",
"minLength": 1
}
}
}
}
}
@@ -121,8 +179,12 @@
"additionalProperties": false,
"minProperties": 1,
"properties": {
"postStart": { "$ref": "#/$defs/execHandler" },
"preStop": { "$ref": "#/$defs/execHandler" }
"postStart": {
"$ref": "#/$defs/execHandler"
},
"preStop": {
"$ref": "#/$defs/execHandler"
}
}
},
"podSecurityContext": {
@@ -131,12 +193,37 @@
"additionalProperties": false,
"minProperties": 1,
"properties": {
"runAsUser": { "type": "integer", "description": "UID the container processes run as." },
"runAsGroup": { "type": "integer", "description": "Primary GID the container processes run as." },
"runAsNonRoot": { "type": "boolean", "description": "Require the container to run as a non-root UID (validated at admission)." },
"fsGroup": { "type": "integer", "description": "Supplemental group applied to mounted volumes so they are group-owned + group-writable (chowned by the kubelet), letting a non-root image write a PVC." },
"fsGroupChangePolicy": { "type": "string", "enum": ["Always", "OnRootMismatch"], "description": "When the kubelet recursively applies fsGroup ownership to a volume." },
"supplementalGroups": { "type": "array", "items": { "type": "integer" }, "description": "Additional GIDs added to the first process in each container." }
"runAsUser": {
"type": "integer",
"description": "UID the container processes run as."
},
"runAsGroup": {
"type": "integer",
"description": "Primary GID the container processes run as."
},
"runAsNonRoot": {
"type": "boolean",
"description": "Require the container to run as a non-root UID (validated at admission)."
},
"fsGroup": {
"type": "integer",
"description": "Supplemental group applied to mounted volumes so they are group-owned + group-writable (chowned by the kubelet), letting a non-root image write a PVC."
},
"fsGroupChangePolicy": {
"type": "string",
"enum": [
"Always",
"OnRootMismatch"
],
"description": "When the kubelet recursively applies fsGroup ownership to a volume."
},
"supplementalGroups": {
"type": "array",
"items": {
"type": "integer"
},
"description": "Additional GIDs added to the first process in each container."
}
}
},
"hardware": {
@@ -148,20 +235,28 @@
"gpu": {
"type": "string",
"description": "GPU capability class this service needs. `intel-quicksync` = the host's Intel Quick Sync Video render device (the iGPU's /dev/dri/renderD* node) for hardware video transcode/decode. Enum, not a free-form device path, so the satisfying mechanism stays swappable (see the parent description). Extend the enum when a genuinely different capability class appears (e.g. nvidia-nvenc), never to name a different path to the same one.",
"enum": ["intel-quicksync"]
"enum": [
"intel-quicksync"
]
}
}
},
"exposure": {
"type": "string",
"description": "internal = LAN-only (internal DNS + Gateway on LAN); public = additionally edge port-forwarded 80/443 (docs/09; R-DNS/R-NET).",
"enum": ["internal", "public"],
"enum": [
"internal",
"public"
],
"default": "internal"
},
"egress": {
"type": "string",
"description": "Egress class (docs/03; R-NET-05). internet = default WAN route; vpn = steered by Cilium Egress Gateway out the egress-gateway node's host PIA VPN tunnel, with a mandatory host kill-switch (ADR-0015; R-NET-06).",
"enum": ["internet", "vpn"],
"enum": [
"internet",
"vpn"
],
"default": "internet"
},
"storage": {
@@ -171,7 +266,10 @@
"items": {
"type": "object",
"additionalProperties": false,
"required": ["name", "class"],
"required": [
"name",
"class"
],
"properties": {
"name": {
"type": "string",
@@ -191,7 +289,10 @@
"class": {
"type": "string",
"description": "longhorn = 3-replica block storage for app/config/DB state (R-STATE-02); bulk = large external media tier, NOT 3x replicated, backed by a STATIC NFS PersistentVolume the renderer emits (requires `export`).",
"enum": ["longhorn", "bulk"],
"enum": [
"longhorn",
"bulk"
],
"default": "longhorn"
},
"export": {
@@ -212,9 +313,28 @@
},
"allOf": [
{
"if": { "properties": { "class": { "const": "bulk" } }, "required": ["class"] },
"then": { "required": ["export"] },
"else": { "not": { "required": ["export"] } }
"if": {
"properties": {
"class": {
"const": "bulk"
}
},
"required": [
"class"
]
},
"then": {
"required": [
"export"
]
},
"else": {
"not": {
"required": [
"export"
]
}
}
}
]
}
@@ -226,7 +346,10 @@
"items": {
"type": "object",
"additionalProperties": false,
"required": ["to", "port"],
"required": [
"to",
"port"
],
"properties": {
"to": {
"type": "string",
@@ -241,7 +364,10 @@
},
"protocol": {
"type": "string",
"enum": ["tcp", "udp"],
"enum": [
"tcp",
"udp"
],
"default": "tcp"
}
}
@@ -254,11 +380,14 @@
"items": {
"type": "object",
"additionalProperties": false,
"required": ["host", "port"],
"required": [
"host",
"port"
],
"properties": {
"host": {
"type": "string",
"description": "Hostname under the platform zone, as a single DNS label plus the ${BASE_DOMAIN} token — e.g. 'gitea.${BASE_DOMAIN}' (docs/09). ${BASE_DOMAIN} is the per-environment hostname suffix from clusters/base/cluster-vars.yaml, resolved by Flux postBuild substitution at reconcile time (dev: home.keithjackson.dev, prod: prod.home.keithjackson.dev). AUTHOR THE TOKEN, NOT A LITERAL: a literal suffix is what let prod be authored to serve dev's public hostnames out of the shared Cloudflare zone, and prod's external-dns then deleted dev's records (infra/PROD-PROVISIONING.md §10). The literal 'home.keithjackson.dev' form stays accepted so the L1 render fixtures — which are renderer INPUTS, not reconciled manifests — keep validating unchanged.",
"description": "Hostname under the platform zone, as a single DNS label plus the ${BASE_DOMAIN} token \u2014 e.g. 'gitea.${BASE_DOMAIN}' (docs/09). ${BASE_DOMAIN} is the per-environment hostname suffix from clusters/base/cluster-vars.yaml, resolved by Flux postBuild substitution at reconcile time (dev: home.keithjackson.dev, prod: prod.home.keithjackson.dev). AUTHOR THE TOKEN, NOT A LITERAL: a literal suffix is what let prod be authored to serve dev's public hostnames out of the shared Cloudflare zone, and prod's external-dns then deleted dev's records (infra/PROD-PROVISIONING.md \u00a710). The literal 'home.keithjackson.dev' form stays accepted so the L1 render fixtures \u2014 which are renderer INPUTS, not reconciled manifests \u2014 keep validating unchanged.",
"pattern": "^[a-z0-9]([-a-z0-9]*[a-z0-9])?\\.(\\$\\{BASE_DOMAIN\\}|home\\.keithjackson\\.dev)$"
},
"port": {
@@ -272,13 +401,16 @@
},
"inClusterClients": {
"type": "array",
"description": "Additional IN-CLUSTER clients, identified by their NAMESPACE, permitted to reach this service's port DIRECTLY — east-west, bypassing the shared Gateway (docs/03; R-NET-01/02). Neither existing field can express this: `ingressRoutes` opens only the north-south path from the Gateway proxy identity (`fromEntities: [ingress]`), and `dependencies` compiles edges only between CATALOG services. A PLATFORM controller that dials this service's ClusterIP is therefore unauthorable — which is exactly how prod's GitOps deadlocked once its GitRepository moved to the in-cluster Gitea Service (source-controller DENIED => the source cannot reconcile => the downstream tier that would fix it never applies). Each entry renders ONE least-privilege ingress CiliumNetworkPolicy: this service's endpointSelector, `fromEndpoints` matching ONLY the named namespace, on ONLY the declared port — never `fromEntities: all`. Additive: the default-deny baseline still denies everything undeclared (R-NET-01). It is NOT a dependency edge and creates no wave ordering — the client is a platform component, not a catalog service.",
"description": "Additional IN-CLUSTER clients, identified by their NAMESPACE, permitted to reach this service's port DIRECTLY \u2014 east-west, bypassing the shared Gateway (docs/03; R-NET-01/02). Neither existing field can express this: `ingressRoutes` opens only the north-south path from the Gateway proxy identity (`fromEntities: [ingress]`), and `dependencies` compiles edges only between CATALOG services. A PLATFORM controller that dials this service's ClusterIP is therefore unauthorable \u2014 which is exactly how prod's GitOps deadlocked once its GitRepository moved to the in-cluster Gitea Service (source-controller DENIED => the source cannot reconcile => the downstream tier that would fix it never applies). Each entry renders ONE least-privilege ingress CiliumNetworkPolicy: this service's endpointSelector, `fromEndpoints` matching ONLY the named namespace, on ONLY the declared port \u2014 never `fromEntities: all`. Additive: the default-deny baseline still denies everything undeclared (R-NET-01). It is NOT a dependency edge and creates no wave ordering \u2014 the client is a platform component, not a catalog service.",
"default": [],
"uniqueItems": true,
"items": {
"type": "object",
"additionalProperties": false,
"required": ["namespace", "port"],
"required": [
"namespace",
"port"
],
"properties": {
"namespace": {
"type": "string",
@@ -294,12 +426,15 @@
},
"protocol": {
"type": "string",
"enum": ["tcp", "udp"],
"enum": [
"tcp",
"udp"
],
"default": "tcp"
},
"name": {
"type": "string",
"description": "Short client name used in the rendered policy name `<service>-allow-<name>-ingress`. Defaults to `namespace`. It exists because that name is operator-facing — it is what `kubectl get cnp` and a Hubble drop show — and because an allowance already applied live can then be authored under its EXISTING name, so Flux adopts the live object in place instead of creating a near-duplicate and leaving an unmanaged leftover (gitea/flux-system: `gitea-allow-flux-ingress`).",
"description": "Short client name used in the rendered policy name `<service>-allow-<name>-ingress`. Defaults to `namespace`. It exists because that name is operator-facing \u2014 it is what `kubectl get cnp` and a Hubble drop show \u2014 and because an allowance already applied live can then be authored under its EXISTING name, so Flux adopts the live object in place instead of creating a near-duplicate and leaving an unmanaged leftover (gitea/flux-system: `gitea-allow-flux-ingress`).",
"pattern": "^[a-z0-9]([-a-z0-9]*[a-z0-9])?$",
"maxLength": 40
}
@@ -310,18 +445,28 @@
"type": "object",
"description": "SSO integration (docs/15, ADR-0014; R-IDP-02/06). Default posture for any exposed UI is forward-auth unless it declares native OIDC (oidc) or a documented exception (native).",
"additionalProperties": false,
"required": ["mode"],
"required": [
"mode"
],
"properties": {
"mode": {
"type": "string",
"description": "oidc = app is a Keycloak OIDC client (own login); forward-auth = oauth2-proxy -> Keycloak at the Gateway; native = app manages its own auth (documented exception, e.g. Plex/Seerr, or Keycloak itself as the IdP); none = no auth (internal-only, discouraged).",
"enum": ["oidc", "forward-auth", "native", "none"],
"enum": [
"oidc",
"forward-auth",
"native",
"none"
],
"default": "forward-auth"
},
"groups": {
"type": "array",
"description": "Keycloak groups permitted access (R-IDP-06). Required and non-empty for oidc/forward-auth (least privilege: authenticated AND in an authorized group). N/A for native/none.",
"items": { "type": "string", "minLength": 1 },
"items": {
"type": "string",
"minLength": 1
},
"uniqueItems": true,
"default": []
},
@@ -373,20 +518,25 @@
"properties": {
"metrics": {
"type": "boolean",
"description": "Whether the platform collects signals for this service (logs to Loki + kube-state-metrics-derived liveness). It does NOT assert that the workload serves a Prometheus endpoint — say that with metricsEndpoint.",
"description": "Whether the platform collects signals for this service (logs to Loki + kube-state-metrics-derived liveness). It does NOT assert that the workload serves a Prometheus endpoint \u2014 say that with metricsEndpoint.",
"default": true
},
"slis": {
"type": "array",
"description": "Per-service alert intents (e.g. up). Each becomes at least one alert rule (R-OBS-01).",
"items": { "type": "string", "minLength": 1 },
"items": {
"type": "string",
"minLength": 1
},
"default": []
},
"metricsEndpoint": {
"type": "object",
"description": "WHERE this workload serves Prometheus metrics. Absent (the default for every catalog entry today) means it serves none, and NO ServiceMonitor is rendered — the renderer used to emit one naming port `metrics` regardless, which no rendered Service ever defined, so all twelve monitors discovered zero targets and every `up`-based SLI was structurally inert. When present it renders all three halves of a working scrape together: the named Service port, the ServiceMonitor endpoint, and the monitoring-namespace ingress allow that the default-deny baseline would otherwise drop.",
"description": "WHERE this workload serves Prometheus metrics. Absent (the default for every catalog entry today) means it serves none, and NO ServiceMonitor is rendered \u2014 the renderer used to emit one naming port `metrics` regardless, which no rendered Service ever defined, so all twelve monitors discovered zero targets and every `up`-based SLI was structurally inert. When present it renders all three halves of a working scrape together: the named Service port, the ServiceMonitor endpoint, and the monitoring-namespace ingress allow that the default-deny baseline would otherwise drop.",
"additionalProperties": false,
"required": ["port"],
"required": [
"port"
],
"properties": {
"port": {
"type": "integer",
@@ -416,13 +566,18 @@
},
"livenessWorkload": {
"type": "object",
"description": "WHICH Kubernetes object's readiness constitutes this service being up, for the `up` SLI. Defaults to the object this ServiceDefinition renders (Deployment named after the service for kind: container, VirtualMachineInstance for kind: vm). Override only when the object that actually runs on the cluster is a different one — keycloak's server is a StatefulSet owned by the Keycloak operator, and a Deployment-shaped rule would read an absent series and fire forever.",
"description": "WHICH Kubernetes object's readiness constitutes this service being up, for the `up` SLI. Defaults to the object this ServiceDefinition renders (Deployment named after the service for kind: container, VirtualMachineInstance for kind: vm). Override only when the object that actually runs on the cluster is a different one \u2014 keycloak's server is a StatefulSet owned by the Keycloak operator, and a Deployment-shaped rule would read an absent series and fire forever.",
"additionalProperties": false,
"properties": {
"kind": {
"type": "string",
"description": "Workload kind whose kube-state-metrics readiness series the SLI reads.",
"enum": ["Deployment", "StatefulSet", "DaemonSet", "VirtualMachineInstance"]
"enum": [
"Deployment",
"StatefulSet",
"DaemonSet",
"VirtualMachineInstance"
]
},
"name": {
"type": "string",
@@ -437,7 +592,9 @@
"type": "object",
"description": "Backup posture (docs/04; R-STATE-03). Irreplaceable state MUST be backed up off-cluster; bulk media may be 'reacquirable'.",
"additionalProperties": false,
"required": ["policy"],
"required": [
"policy"
],
"properties": {
"policy": {
"type": "string",
@@ -445,38 +602,88 @@
"minLength": 1
}
}
},
"availability": {
"type": "object",
"description": "Optional availability intent for the container kind: how many replicas of this workload must be running. INTENT ONLY -- it names no Kubernetes object and no scheduling primitive. How the platform satisfies it (workload kind, per-replica volumes, node spreading, disruption budget) is catalog/render.py's business, documented in schemas/renderer-contract.md `### availability`. Absent => replicas 1 and a Deployment, so every pre-existing app renders byte-identical. Ignored for kind=vm (a VM's availability is a different problem with different primitives).",
"additionalProperties": false,
"minProperties": 1,
"properties": {
"replicas": {
"type": "integer",
"description": "How many replicas must be running. 1 (default) renders a Deployment exactly as before. >1 declares that the service must survive the loss of a single node; the renderer then emits a StatefulSet with per-replica volumeClaimTemplates, hard node anti-affinity, podManagementPolicy Parallel and a PodDisruptionBudget. Only declare >1 for a service whose on-disk state is reproducible from git -- a replica with a fresh volume comes up serving whatever it can bootstrap itself, and nothing else.",
"minimum": 1,
"maximum": 5,
"default": 1
}
}
}
},
"allOf": [
{
"$comment": "kind determines which source field is required (docs/13). Absent kind defaults to container -> image.",
"if": {
"required": ["kind"],
"properties": { "kind": { "const": "vm" } }
"required": [
"kind"
],
"properties": {
"kind": {
"const": "vm"
}
}
},
"then": {
"properties": { "source": { "required": ["vmTemplate"] } }
"properties": {
"source": {
"required": [
"vmTemplate"
]
}
}
},
"else": {
"properties": { "source": { "required": ["image"] } }
"properties": {
"source": {
"required": [
"image"
]
}
}
}
},
{
"$comment": "oidc/forward-auth require at least one authorized group (R-IDP-06; docs/15 acceptance gate: restricted to the allowed group).",
"if": {
"required": ["auth"],
"required": [
"auth"
],
"properties": {
"auth": {
"required": ["mode"],
"properties": { "mode": { "enum": ["oidc", "forward-auth"] } }
"required": [
"mode"
],
"properties": {
"mode": {
"enum": [
"oidc",
"forward-auth"
]
}
}
}
}
},
"then": {
"properties": {
"auth": {
"required": ["groups"],
"properties": { "groups": { "minItems": 1 } }
"required": [
"groups"
],
"properties": {
"groups": {
"minItems": 1
}
}
}
}
}
@@ -484,19 +691,33 @@
{
"$comment": "auth.apiRoutes only has a consumer under forward-auth: it renders --api-route onto the per-app oauth2-proxy sidecar, and no sidecar is rendered for oidc/native/none. Declaring it elsewhere would be a SILENT NO-OP -- the operator believes the app's XHRs now answer 401 and nothing changed -- so it is rejected at the schema instead (R-IDP-02).",
"if": {
"required": ["auth"],
"required": [
"auth"
],
"properties": {
"auth": {
"required": ["apiRoutes"],
"properties": { "apiRoutes": { "minItems": 1 } }
"required": [
"apiRoutes"
],
"properties": {
"apiRoutes": {
"minItems": 1
}
}
}
}
},
"then": {
"properties": {
"auth": {
"required": ["mode"],
"properties": { "mode": { "const": "forward-auth" } }
"required": [
"mode"
],
"properties": {
"mode": {
"const": "forward-auth"
}
}
}
}
}
@@ -504,13 +725,25 @@
{
"$comment": "public exposure must not be unauthenticated: an explicit auth.mode 'none' is disallowed on public services (docs/13 v1.1, docs/15, R-IDP-02). Absent auth defaults to forward-auth at render time, so it is permitted here.",
"if": {
"required": ["exposure"],
"properties": { "exposure": { "const": "public" } }
"required": [
"exposure"
],
"properties": {
"exposure": {
"const": "public"
}
}
},
"then": {
"properties": {
"auth": {
"properties": { "mode": { "not": { "const": "none" } } }
"properties": {
"mode": {
"not": {
"const": "none"
}
}
}
}
}
}
@@ -521,18 +754,24 @@
"type": "object",
"description": "A container lifecycle hook. Only the exec form is supported (a command run inside the container); mirrors k8s LifecycleHandler.exec.",
"additionalProperties": false,
"required": ["exec"],
"required": [
"exec"
],
"properties": {
"exec": {
"type": "object",
"additionalProperties": false,
"required": ["command"],
"required": [
"command"
],
"properties": {
"command": {
"type": "array",
"description": "Command line to execute inside the container. Not run in a shell unless the command itself invokes one.",
"minItems": 1,
"items": { "type": "string" }
"items": {
"type": "string"
}
}
}
}