Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity
Files
fbaeb3e0386e624c5da6e9c513176dd40501c72f
home-network-k3s-impl/catalog/render.py
T
jakman85andClaude Opus 5 fbaeb3e038
gate / gate (pull_request) Successful in 1m49s
Details
gate / conformance (pull_request) Has been skipped
Details
feat(plex): expose tcp/32400 for remote access via a pinned LoadBalancer VIP 
Remote Plex clients had no path in, for two measured reasons: Plex advertised
no remote connection at all (customConnections and ManualPortMappingMode both
ABSENT while PublishServerOnPlexOnlineKey=1), and its Service was ClusterIP, so
32400 had no off-cluster path. The L7 Gateway was NOT the problem -- it serves
Plex correctly (200, claimed=1, 401 on protected endpoints) -- but Plex clients
discover servers through plex.tv, not DNS, so an HTTPRoute cannot put this
server on a client's connection list.

Operator chose the native path over ADVERTISE_IP: Plex terminates its own TLS
with *.plex.direct and streams bypass the Envoy L7 hop.

Adds a hand-written non-catalog unit (the SD schema cannot express a
LoadBalancer Service or a pinned lbipam VIP), registered in render.py's
EXTRA_UNITS so --emit-waves places it in plex's own wave.

externalTrafficPolicy: Cluster, inheriting crafty's 8-day-outage lesson rather
than re-deriving it. Known consequence, documented: SNAT puts remote clients
inside ALLOWED_NETWORKS, so Plex meters them as LAN. Not an auth change.

Remote access does NOT work until the operator does two manual steps that
cannot live in this repo -- the WAN tcp/32400 forward and Plex's "manually
specify public port" toggle (no env var exists for it; verified NONE in the
image's init surface). runbooks/plex-remote-access.md is the procedure.

Touches clusters/** -- a sensitive path -- so the approval-policy stage will
block pending an operator /approve-override.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-03 23:30:46 -05:00

3509 lines
171 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
#!/usr/bin/env python3
"""ServiceDefinition renderer (task T-0.3) — the platform's core GitOps contract.
Consumes ONE ServiceDefinition (schemas/servicedefinition.schema.json, catalog/*.yaml)
and emits the concrete manifest set the renderer contract specifies
(schemas/renderer-contract.md): Namespace + default-deny baseline, Deployment/Service or
KubeVirt VirtualMachine + CDI DataVolume, PVCs, per-dependency and per-in-cluster-client
CiliumNetworkPolicy allows,
egress-class steering label, HTTPRoute(s), ServiceMonitor/alerts, Flux image-automation,
and the auth-mode / backup wiring hints.
Requirements: R-GITOPS-02 (single reviewable object wires every requirement), R-NET-01/02
(default-deny + explicit allows), R-NET-05 (egress class), R-STATE-01/02/04 (storage),
R-TLS/R-DNS (ingressRoutes/exposure), R-OBS-01 (observability), R-UPD-01/02/03 (update),
R-IDP-02/06 (auth), R-PLAT-02/03 (kind/source).
Design (documented per the task's "if the contract is ambiguous, choose the simplest
reproducible option"): a deterministic Python renderer. The contract does not mandate a
mechanism (script / kustomize generator / templates); a render-then-consume script is the
simplest reproducible option the repo can run in CI and that Flux consumes from a rendered
output path. CI invokes it via `render.py <sd>` (or `--all`); the offline L1 render
conformance verifier diffs its deterministic output against committed goldens.
Determinism: fixed document order + fixed key order + PyYAML with sort_keys=False and a
wide line width, so identical input always yields byte-identical output (golden-diffable).
Dependency: PyYAML (input parse + output emit). This is the ONE renderer dependency; the
conformance harness stays stdlib-only and only shells out to this script + diffs text.
"""
from __future__ import annotations
import argparse
import re
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover - explicit, actionable failure
sys.stderr.write(
"render.py requires PyYAML (the renderer's single dependency). "
"Install with: pip install pyyaml\n"
)
raise SystemExit(2)
# --- Label / naming conventions (match platform/kyverno/policies + platform/*/) --------
LBL = "homelab.keithjackson.dev"
# The platform hostname suffix, emitted as the Flux post-build TOKEN rather than a literal.
# Every rendered manifest lands under a Flux path whose Kustomization declares
# postBuild.substituteFrom cluster-vars, so `${BASE_DOMAIN}` is resolved per environment at
# reconcile time: dev renders home.keithjackson.dev, prod renders prod.home.keithjackson.dev.
# It is a DECLARED cluster-vars knob (clusters/base/cluster-vars.yaml), so it is written BARE —
# the escape contract below reserves `$${...}` for expansions meant for a container's shell.
# Hardcoding the literal here is what let prod be authored to serve dev's public hostnames,
# and prod's external-dns then deleted dev's records (infra/PROD-PROVISIONING.md §10).
ZONE = "${BASE_DOMAIN}"
# The bulk (external NFS) tier's SERVER, emitted as a Flux post-build TOKEN for exactly the same
# reason as ZONE: dev and prod reach the same Proxmox ZFS host by DIFFERENT addresses (dev's nodes
# are on the host's own LAN and dial it directly; prod's nodes are on the far side of pfSense and
# must be given whichever address the still-pending routing decision produces). Hardcoding either
# one authors a manifest that is silently wrong in the other environment. Declared in
# clusters/base/cluster-vars.yaml and patched by BOTH overlays; written BARE (it is a real knob,
# not a shell expansion), so StrictPostBuildSubstitutions fails the tier if an overlay forgets it.
BULK_NFS_SERVER = "${BULK_NFS_SERVER}"
# Mount options for every bulk NFS volume. These are a PLATFORM decision, not a per-service one,
# so they live here rather than on the ServiceDefinition surface:
# nfsvers=4.1 NFSv4 carries mount+lock IN the protocol, so it needs ONE port (tcp/2049) and no
# rpcbind/111, mountd or statd. That is what makes the prod path a single pfSense
# rule instead of four, and it is why v4 is chosen over v3. The server advertises
# +3 +4 +4.1 +4.2; 4.1 is the conservative floor every node kernel already speaks.
# hard A `soft` mount turns a server hiccup into a short read / EIO handed to the
# application, which for a media library and an archive is silent corruption.
# `hard` blocks instead. The cost is explicit: an NFS outage HANGS I/O rather than
# failing it, so an outage shows up as stuck pods, not as damaged files. Chosen
# deliberately -- hung is recoverable, truncated is not.
# noatime The datasets are relatime, so every read would otherwise generate a metadata
# WRITE back to the pool. Plex walking an 8 T library must not do that.
# timeo=600/retrans=2 the TCP defaults, stated explicitly so a future reader knows the retry
# budget was considered rather than inherited.
BULK_NFS_MOUNT_OPTIONS = ["nfsvers=4.1", "hard", "noatime", "timeo=600", "retrans=2"]
GATEWAY_NAME = "shared"
GATEWAY_NS = "gateway"
# ExternalDNS per-resource scoping (R-DNS-03; see _externaldns_route_annotations for the full WHY
# and for the v0.21.0 source citation). The KEY is external-dns's own `ControllerKey`
# (annotationPrefix + "controller"); external-dns skips any source object whose value is PRESENT
# and != `ControllerValue` ("dns-controller"), so the VALUE only has to be some other string.
# `none` is chosen because it reads as intent in `kubectl get httproute -o yaml` rather than as a
# typo — "no DNS controller owns this route" — and because the one value that must never be
# emitted here is external-dns's own, which a literal `none` cannot drift into.
EXTERNALDNS_CONTROLLER_KEY = "external-dns.alpha.kubernetes.io/controller"
EXTERNALDNS_CONTROLLER_DISOWN = "none"
# The value external-dns claims for itself. Held here ONLY so the renderer and the L0 guard can
# assert the disown value is not it; never emitted.
EXTERNALDNS_CONTROLLER_CLAIMED = "dns-controller"
KP_RELEASE = "kube-prometheus-stack" # ServiceMonitor/PrometheusRule discovery label
MONITORING_NS = "monitoring" # where Prometheus runs; the only admitted scrape source
# The kube-state-metrics series a workload's readiness IS (r_prometheusrule). kube-state-metrics
# is scraped by the bundled kube-prometheus-stack ServiceMonitor, i.e. these exist for EVERY
# workload in the cluster with no per-service exporter — which is the whole point of using them
# for `slis: [up]`. Each entry is (metric, label naming the object). VERIFIED on prod 2026-09-13:
# kube_deployment_status_replicas_available returned a series for all ten deployed app
# Deployments, and kube_statefulset_status_replicas_ready{namespace="keycloak",
# statefulset="keycloak"} returned 2.
LIVENESS_METRICS = {
"Deployment": ("kube_deployment_status_replicas_available", "deployment"),
"StatefulSet": ("kube_statefulset_status_replicas_ready", "statefulset"),
"DaemonSet": ("kube_daemonset_status_number_available", "daemonset"),
}
# A KubeVirt VMI is not a replica count: kubevirt_vmi_info carries the phase as a LABEL, so
# liveness is the EXISTENCE of a running-phase series. The value is lowercase `running` — a
# capital-R matcher selects nothing and is just as inert (the same trap that made
# kubevirt_vmi_phase_count{phase="Failed"} a no-op; verified on prod, where the one live VMI
# reports phase="running").
VMI_METRIC = "kubevirt_vmi_info"
VMI_RUNNING_PHASE = "running"
# egress classes and auth modes fixed by docs/03 / docs/13 + the Kyverno enums.
EGRESS_CLASSES = ("internet", "vpn")
AUTH_MODES = ("oidc", "forward-auth", "native", "none")
# --- forward-auth (Option B) per-app oauth2-proxy sidecar ------------------------------
# A `forward-auth` app has a bare web UI with no usable native OIDC (PiHole, *arr, Jackett,
# qBittorrent). Cilium 1.16.x cannot inject ext_authz into the shared Gateway's Envoy config
# (docs/plans/cilium-native-externalauth-evaluation.md; platform/oauth2-proxy/README.md), so
# the platform's forward-auth CEC is an UNWIRED template. Instead each forward-auth app is
# gated by a PER-POD oauth2-proxy REVERSE-PROXY sidecar (Option B) — EXACTLY the pattern
# platform/crafty proves working: the shared Gateway routes <app>.home -> the sidecar (:4180),
# the sidecar authenticates the browser against Keycloak (homelab realm, the app's auth.groups)
# and only THEN proxies to the co-located app UI at http://127.0.0.1:<uiport>. The gate is IN
# the data path (fail-closed: no valid session in an authorized group => no upstream; R-IDP-02).
OAUTH2_PROXY_IMAGE = "quay.io/oauth2-proxy/oauth2-proxy:v7.8.1"
OAUTH2_PROXY_PORT = 4180
KEYCLOAK_ISSUER = f"https://keycloak.{ZONE}/realms/homelab"
# --- session lifetime -----------------------------------------------------------------
# Keycloak's `homelab` realm issues access tokens with a 300-SECOND lifetime. MEASURED on a
# live session, not read off a config page: `expires - created = 300.000s` exactly, with
# `refresh_token:true` on the same line — so the proxy HOLDS a refresh token it was never
# told to spend. `--cookie-refresh` defaults to 0, which the v7.8.1 binary's own help spells
# "0 to disable", so every one of these sidecars dropped its session five minutes after
# sign-in and sent the user back to Keycloak. Refresh at HALF the lifetime: two chances to
# renew before expiry, so one failed refresh is not an immediate logout.
#
# NOTE for anyone tempted to add `--session-cookie-minimal` to shrink the cookie: it is
# MUTUALLY EXCLUSIVE with this (provoked: "cookie_refresh > 0 requires oauth tokens in
# sessions"). Dropping --pass-access-token is a least-privilege argument, not a size fix.
OAUTH2_ACCESS_TOKEN_LIFETIME_S = 300
OAUTH2_COOKIE_REFRESH = "2m30s"
# --- metrics --------------------------------------------------------------------------
# oauth2-proxy serves /metrics only when --metrics-address is given; with it absent the
# listener never starts, so NO per-app sidecar was scrapable at all (measured: `ports` was
# `['http']` on all six). 44180 is the platform's existing convention, not a guess — the
# SHARED proxy already runs `--metrics-address=0.0.0.0:44180` with a `metrics` port and 2
# ready endpoints. The flag is confirmed present in the PINNED v7.8.1 binary (`--help`),
# which matters because the shared proxy is on v7.15.3 and could not have vouched for it.
#
# THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED. A container port is not a
# Service port: these sidecars' Services expose the app port and :4180, and each namespace's
# default-deny baseline has `ingress: []`, so Prometheus can neither resolve nor reach this
# port until a Service port (or PodMonitor), a monitoring-ingress allow and a ServiceMonitor
# are added. Do not read this flag as "metrics are working".
OAUTH2_PROXY_METRICS_PORT = 44180
# --- Hardware capability: `hardware.gpu: intel-quicksync` ------------------------------
# The ServiceDefinition declares the CAPABILITY (schemas/servicedefinition.schema.json
# `hardware.gpu`); everything below is the MECHANISM that currently satisfies it and is
# deliberately confined to this file. This file is the ONLY place the mechanism appears,
# which is what let it be swapped on 2026-10-01 -- from a `/dev/dri` hostPath to the Intel
# GPU device plugin's extended resource -- with ZERO ServiceDefinition edits.
#
# The extended resource is advertised by the `intel-gpu-plugin` DaemonSet
# (platform/intel-gpu-plugin/, pinned intel/intel-gpu-plugin:0.37.0). MEASURED on all three
# prod nodes 2026-10-01: `.status.allocatable["gpu.intel.com/i915"] == 1`.
QUICKSYNC_RESOURCE = "gpu.intel.com/i915"
# --- API routes: 401 instead of 302, PER APP ------------------------------------------
# oauth2-proxy answers an unauthenticated request with a 302 to Keycloak. Its built-in AJAX
# guard would send a 401 instead, but it fires only on an EXACT `Accept: application/json`,
# which these apps' JavaScript does not consistently send. So an unauthenticated XHR gets a
# 302, the browser follows it INVISIBLY, the live Keycloak SSO session satisfies it silently,
# and the call resolves 200 carrying the app's HTML where the JavaScript expected JSON — no
# prompt, no console error, nothing logged as a failure. On a WRITE it is worse: a browser
# following a 302 re-issues a POST as a GET and DROPS THE BODY, so the operator's change is
# discarded and the UI reports success. MEASURED, twice:
# jackett 2026-09-30: 33 x 302 against 2 x 401 on /api/ paths, and 4 of 6 POSTs answered by
# the proxy and never reaching the app (access-log upstream field `-`, not `/`).
# radarr 2026-10-01, 23h after the cookie fix rolled: 232 x 302, EVERY ONE a
# `POST /signalr/messages/negotiate` from the operator's own browser, upstream `-`.
# A left-open tab whose cookie the rename orphaned, polling forever, silently
# redirected every time, NEVER re-prompting. That is the reported symptom
# ("reloading never fixes it and I am never re-prompted") still happening in
# production, on a service nobody was looking at.
# `--api-route=<regex>` makes matching paths answer 401.
#
# WHY THIS IS NOT A RENDERER CONSTANT, AND MUST NEVER BECOME ONE. The path shape is a property
# of the APP, not of the forward-auth pattern: radarr's broken path is `/signalr/`, jackett's is
# `/api/`, and PIHOLE SERVES NO `/api/` AT ALL. One constant would apply one app's path shape to
# five, and the failure mode of a wrong one is not "no improvement" but "this service is
# permanently broken": a path a BROWSER NAVIGATES to, answered 401, shows a logged-out operator
# a bare error page and never offers a sign-in. So the routes are declared per service in
# `auth.apiRoutes` (schema 1.9.0) and rendered one `--api-route` per entry — the flag is
# repeatable, and it is SINGULAR: `--api-routes` does not exist in v7.8.1 and the proxy refuses
# to start with it (verified against the pinned binary's own --help, 2026-10-01:
# `--api-route strings return HTTP 401 instead of redirecting to authentication server...`).
#
# BREADTH RATCHET. Every entry must be anchored at `^/` and continue with literal path
# characters. That rejects `^/`, `^/.*`, `.*`, an unanchored `/api/`, and any alternation — the
# shapes that turn a page load into a 401. Alternation is refused rather than supported because
# the flag is repeatable, so one regex per route costs nothing and stays readable. The same
# pattern is in the schema (so ci/l0.sh's schema stage rejects it at the authoring surface) and
# enforced HERE (so ci/render-golden.sh fails even if the schema stage is skipped) — a breadth
# mistake must not need two independent gates to both be running.
OAUTH2_API_ROUTE_PATTERN = re.compile(r"^\^/[A-Za-z0-9._~/-]+(\.\*|\$)?$")
# =============================================================================
# Defaults (renderer-contract.md "Default posture") — materialize BEFORE mapping.
# =============================================================================
def apply_defaults(sd: dict) -> dict:
"""Return a copy of the ServiceDefinition with the contract's default posture filled.
A field absent from a ServiceDefinition renders from its default posture, not "not at
all" (renderer-contract.md).
"""
s = dict(sd)
s.setdefault("kind", "container")
s.setdefault("exposure", "internal")
s.setdefault("egress", "internet")
s.setdefault("storage", [])
s.setdefault("dependencies", [])
s.setdefault("ingressRoutes", [])
s.setdefault("inClusterClients", [])
# auth.mode default posture: forward-auth for any exposed UI, else none (internal, no
# UI). An explicit auth block always wins (R-IDP-02).
auth = dict(s.get("auth") or {})
if "mode" not in auth:
auth["mode"] = "forward-auth" if s["ingressRoutes"] else "none"
auth.setdefault("groups", [])
# apiRoutes default posture is EMPTY, deliberately, and that is the whole point of the field
# (schema 1.9.0): an app gets --api-route only where someone MEASURED that it serves a machine
# API, so a newly-onboarded app renders no --api-route at all rather than inheriting another
# app's path shape. See OAUTH2_API_ROUTE_PATTERN above for why a default of `^/api/` would be
# wrong rather than merely generous.
auth.setdefault("apiRoutes", [])
s["auth"] = auth
update = dict(s.get("update") or {})
update.setdefault("channel", "stable")
update.setdefault("autoRollbackOnFailedHealth", True)
s["update"] = update
obs = dict(s.get("observability") or {})
obs.setdefault("metrics", True)
obs.setdefault("slis", [])
# metricsEndpoint is NEVER defaulted (schema 1.6.0). `metrics: true` says the platform
# collects signals FOR this service; it cannot say the workload SERVES a Prometheus
# endpoint, because for most of this catalog it does not (see r_servicemonitor). When the
# endpoint IS declared, its name/path/interval carry defaults so the common case is
# `metricsEndpoint: {port: 9100}`.
ep = obs.get("metricsEndpoint")
if ep:
obs["metricsEndpoint"] = {
**{"name": "metrics", "path": "/metrics", "interval": "30s"},
**ep,
}
# livenessWorkload: WHICH Kubernetes object's readiness IS this service being up. Defaulted
# from `kind` — a container renders a Deployment named after the service, a vm renders a
# VirtualMachine whose VMI carries the same name — and overridable for the case where the
# object that actually runs on the cluster is not the one this SD renders (keycloak: the
# Keycloak operator runs a StatefulSet, so a Deployment-shaped rule would `absent()` forever).
lw = dict(obs.get("livenessWorkload") or {})
lw.setdefault("kind", "VirtualMachineInstance" if s["kind"] == "vm" else "Deployment")
lw.setdefault("name", s["name"])
obs["livenessWorkload"] = lw
s["observability"] = obs
# storage entry class default is longhorn (replicated tier); readOnly default is False
# (grant-nothing-extra is expressed by SETTING it, so the default stays back-compatible and
# every pre-existing golden renders byte-identical).
s["storage"] = [{**{"class": "longhorn", "readOnly": False}, **v} for v in s["storage"]]
# dependency protocol default is tcp.
s["dependencies"] = [{**{"protocol": "tcp"}, **d} for d in s["dependencies"]]
# in-cluster client protocol default is tcp (as for a dependency); its `name` — which only
# ever appears in the rendered POLICY name — defaults to the namespace it admits, so the
# common case declares just `namespace` + `port` and reads like a dependency edge.
s["inClusterClients"] = [
{**{"protocol": "tcp", "name": c.get("namespace", "")}, **c}
for c in s["inClusterClients"]
]
return s
# =============================================================================
# Field -> resource mapping (renderer-contract.md "Field -> resource mapping")
# =============================================================================
def r_namespace(s: dict) -> dict:
"""Namespace = the service trust zone / isolation boundary (R-NET, docs/03).
Carries app-namespace=true so Kyverno generates the default-deny baseline and the
egress-class / auth-mode admission policies apply. gateway-access=shared lets its
HTTPRoutes attach to the shared Gateway (only when it exposes routes).
"""
labels = {f"{LBL}/app-namespace": "true"}
if s["ingressRoutes"]:
labels["gateway-access"] = "shared"
ann = {}
backup = (s.get("backup") or {}).get("policy")
if backup:
ann[f"{LBL}/backup-policy"] = backup # R-STATE-03 off-cluster schedule ref (hint)
meta = {"name": s["name"], "labels": labels}
if ann:
meta["annotations"] = ann
return {"apiVersion": "v1", "kind": "Namespace", "metadata": meta}
def r_default_deny(s: dict) -> dict:
"""The default-deny baseline: deny all ingress+egress, allow ONLY DNS egress (R-NET-01).
Rendered explicitly (GitOps at namespace creation, renderer-contract §dependencies) —
it matches the Kyverno-generated baseline shape (platform/kyverno/.../generated-default
-deny.yaml) so the render carries the guarantee even before admission generates it.
"""
return {
"apiVersion": "networking.k8s.io/v1",
"kind": "NetworkPolicy",
"metadata": {
"name": "default-deny",
"namespace": s["name"],
"labels": {f"{LBL}/baseline": "default-deny"},
},
"spec": {
"podSelector": {},
"policyTypes": ["Ingress", "Egress"],
"ingress": [],
"egress": [
{
"to": [
{
"namespaceSelector": {
"matchLabels": {
"kubernetes.io/metadata.name": "kube-system"
}
}
}
],
"ports": [
{"protocol": "UDP", "port": 53},
{"protocol": "TCP", "port": 53},
],
}
],
},
}
def _pod_labels(s: dict) -> dict:
"""Workload pod labels: identity + the egress-class steering hook + auth-mode.
egress-class is what the shared Cilium Egress Gateway selects on to steer vpn-class
pods (enforcement is OUTSIDE the workload — renderer-contract §egress; ADR-0015);
auth-mode satisfies the Kyverno require-auth-mode-label policy (R-IDP-02).
"""
return {
"app.kubernetes.io/name": s["name"],
f"{LBL}/egress-class": s["egress"],
f"{LBL}/auth-mode": s["auth"]["mode"],
}
def _service_ports(s: dict) -> list:
ports = sorted({r["port"] for r in s["ingressRoutes"]})
return [{"name": f"port-{p}", "port": p, "targetPort": p} for p in ports]
def _bulk_access_mode(vol: dict) -> str:
"""A bulk volume's access mode follows its read/write posture.
Every bulk volume is `...Many`: the SAME export is consumed by several namespaces at once
(plex, sonarr and radarr all sit on /mnt/Storage/plex) and by several nodes at once, which is
precisely what ReadWriteOnce cannot express. WHICH `Many` it is says whether this consumer
writes -- so `kubectl get pv` reports the intent, not merely the topology.
"""
return "ReadOnlyMany" if vol.get("readOnly") else "ReadWriteMany"
def r_bulk_pvs(s: dict) -> list:
"""One STATIC PersistentVolume per `class: bulk` storage entry (docs/04 bulk tier).
WHY A PV AT ALL, RATHER THAN AN INLINE `nfs:` POD VOLUME
-------------------------------------------------------
An inline volume is fewer objects and it would work. It is rejected for four reasons, in
descending order of what they would cost to discover later:
1. RECLAIM POLICY. `persistentVolumeReclaimPolicy: Retain` is a declared, reviewable
property of a PV. There is no equivalent for an inline volume -- and the data behind
these mounts is 8.33 TiB of media and a 267 GiB archive that ALSO has live non-k3s
consumers. A statement in git that nothing in this cluster may reclaim it is worth an
object.
2. READ-ONLY LIVES IN ONE PLACE. `spec.nfs.readOnly: true` becomes a real `ro` mount option
for every pod that ever binds this volume. The same intent expressed on an inline volume
is one Deployment edit away from being dropped silently, with no diff anywhere near the
storage. (It is still a CLIENT-side guarantee -- the export cannot distinguish plex from
sonarr, since both may run on any node -- but it is one that lives in one reviewable
place.)
3. VISIBILITY. `kubectl get pv,pvc` is where an operator looks for "what external storage is
attached, and who holds it". An inline volume is invisible there, and invisible to
anything that reasons over PVCs.
4. IT IS THE CONTRACT THE REPO ALREADY HAS. Every other storage entry is a PVC; the
renderer, the Kyverno ServiceDefinition policy and the backup wiring all key on that. An
inline volume would be a second, parallel storage model inside the same renderer.
CROSS-NAMESPACE. A PV binds 1:1 to a PVC, so "plex, sonarr and radarr share one export" is NOT
one shared PV -- it is three PVs, one per namespace, each naming the same server+path. That is
the correct model rather than a workaround: the sharing happens on the NFS server, which is
built for concurrent clients, and each namespace keeps its own independently reviewable access
mode (plex ReadOnlyMany, the arrs ReadWriteMany). One shared PV would have forced a single
posture on all three.
BINDING IS PINNED FROM BOTH ENDS -- `claimRef` here and `volumeName` on the PVC. Either alone
leaves a race in which some other PVC of matching size and mode grabs the volume first;
together the binding is deterministic. `storageClassName: ""` (explicitly empty, NOT the `bulk`
StorageClass) is what tells the binder this volume is statically provisioned: it suppresses the
default-StorageClass annotation and guarantees no provisioner is ever asked to create anything.
It also sidesteps a trap -- a StorageClass's `provisioner` is IMMUTABLE, so re-pointing the
existing `bulk` class (whose stub names an NFS CSI driver that is not installed) at this data
would make Flux fail forever on that one object.
"""
out = []
for vol in s["storage"]:
if vol.get("class") != "bulk":
continue
name = f"{s['name']}-{vol['name']}"
# `export` and `size` are both structurally required of a bulk entry, and the JSON
# Schema only enforces the first. A PersistentVolume with no `capacity` is rejected by
# the API server, so a missing size must fail HERE with a sentence a human can act on
# rather than as a bare KeyError three frames down.
for field, why in (
("export", "the absolute NFS export path this volume attaches"),
("size", "the PV capacity (advisory, but the API requires the field)"),
):
if not vol.get(field):
raise ValueError(
f"{s['name']}: storage entry {vol['name']!r} has class 'bulk' but no "
f"'{field}' — {why}. A bulk volume attaches storage that already exists "
f"outside the cluster, so it cannot be rendered without it."
)
nfs = {"server": BULK_NFS_SERVER, "path": vol["export"]}
if vol.get("readOnly"):
nfs["readOnly"] = True
spec = {
"capacity": {"storage": vol["size"]},
"accessModes": [_bulk_access_mode(vol)],
# Retain: deleting the PVC must NEVER be able to reach the data. There is nothing to
# "reclaim" here in any case -- the volume was not provisioned by the cluster.
"persistentVolumeReclaimPolicy": "Retain",
"storageClassName": "",
"mountOptions": list(BULK_NFS_MOUNT_OPTIONS),
"nfs": nfs,
"claimRef": {"namespace": s["name"], "name": name},
}
out.append(
{
"apiVersion": "v1",
"kind": "PersistentVolume",
"metadata": {
"name": name,
"labels": {"app.kubernetes.io/name": s["name"]},
},
"spec": spec,
}
)
return out
def r_pvcs(s: dict) -> list:
"""One PVC per storage entry (R-STATE-01/02/04).
longhorn -> 3-replica replicated tier, dynamically provisioned, ReadWriteOnce.
bulk -> the STATIC external NFS PV rendered by r_bulk_pvs, pre-bound by volumeName.
"""
out = []
for vol in s["storage"]:
cls = vol.get("class", "longhorn")
if cls == "bulk":
# Mirror image of the PV: the same explicitly-empty class (so nothing dynamically
# provisions), the same access mode, and volumeName completing the two-sided binding.
out.append(
{
"apiVersion": "v1",
"kind": "PersistentVolumeClaim",
"metadata": {
"name": f"{s['name']}-{vol['name']}",
"namespace": s["name"],
"labels": {"app.kubernetes.io/name": s["name"]},
},
"spec": {
"accessModes": [_bulk_access_mode(vol)],
"storageClassName": "",
"volumeName": f"{s['name']}-{vol['name']}",
"resources": {"requests": {"storage": vol["size"]}},
},
}
)
continue
spec = {
"accessModes": ["ReadWriteOnce"],
"storageClassName": "longhorn",
}
if vol.get("size"):
spec["resources"] = {"requests": {"storage": vol["size"]}}
out.append(
{
"apiVersion": "v1",
"kind": "PersistentVolumeClaim",
"metadata": {
"name": f"{s['name']}-{vol['name']}",
"namespace": s["name"],
"labels": {"app.kubernetes.io/name": s["name"]},
},
"spec": spec,
}
)
return out
def r_deployment(s: dict) -> dict:
"""Deployment for a container workload (R-PLAT-02)."""
pod_labels = _pod_labels(s)
container = {
"name": s["name"],
"image": s["source"]["image"],
}
ports = _service_ports(s)
if ports:
container["ports"] = [{"containerPort": p["port"]} for p in ports]
# env / envFrom (schema-optional): the k8s EnvVar / EnvFromSource lists are emitted
# VERBATIM onto the app container. This is what lets a headless-install/DB-backed app
# (gitea, nextcloud, semaphore, unifi, minecraft) configure itself and pull a fixed
# secret (e.g. gitea's SECRET_KEY via secretKeyRef) without a supplementary patch.
# Absent => the key is never set, so apps that declare no env render byte-identical.
if s.get("env"):
container["env"] = s["env"]
if s.get("envFrom"):
container["envFrom"] = s["envFrom"]
volume_mounts, volumes = [], []
for vol in s["storage"]:
# Default mount is /data/<name>; an explicit `mountPath` (schema-optional) overrides
# it verbatim for apps that need a fixed on-disk layout (e.g. Gitea keeps all state
# directly under /data). Omitting it preserves the /data/<name> default (back-compat).
mount_path = vol.get("mountPath", f"/data/{vol['name']}")
mount = {"name": vol["name"], "mountPath": mount_path}
# subPath mounts a subtree of the volume instead of its root. Emitted only when declared,
# so every pre-existing golden stays byte-identical.
if vol.get("subPath"):
mount["subPath"] = vol["subPath"]
# readOnly on the mount as well as on the PV's nfs source. Belt and braces on purpose: the
# PV field is what makes the kernel mount `ro`; this one is what a reviewer reading the
# Deployment sees. An edit that drops one still leaves the other.
if vol.get("readOnly"):
mount["readOnly"] = True
volume_mounts.append(mount)
volumes.append(
{
"name": vol["name"],
"persistentVolumeClaim": {"claimName": f"{s['name']}-{vol['name']}"},
}
)
# hardware capability (schema-optional): `hardware.gpu: intel-quicksync` means this
# workload needs the host iGPU's render node for hardware video transcode. The SD says
# only that; the wiring below is the platform's current mechanism for satisfying it.
#
# THE MECHANISM IS AN EXTENDED RESOURCE, AND THE hostPath IT REPLACED WAS MEASURED NOT
# TO WORK. Do NOT "simplify" this back into a `/dev/dri` hostPath volume. That was the
# original implementation (merged 2026-09-30) and it was proven broken in the live Plex
# pod on k3s-prod-2 on 2026-10-01: the device node was VISIBLE in the container
# (`crw-rw---- root video2 226,128 renderD128`) and could not be opened --
# `uid 0 O_RDWR -> EPERM` and `uid 1000 O_RDWR -> EPERM`. EPERM from the DEVICE CGROUP,
# not EACCES from file permissions, proven three ways: root on the HOST opens the same
# device rc=0; the node is unified `cgroup2fs`; and the container's runc `config.json`
# read `linux.resources.devices: [{allow: False, access: 'rwm'}]` with
# `linux.devices: None` -- one deny-all rule, no allow entry for major 226. A hostPath is
# a filesystem operation: it binds the device node in but adds NO cgroup allow rule,
# because Kubernetes cannot infer one from a volume.
#
# The device plugin is what produces the allow rule. Confirmed in the v0.37.0 source:
# `createDeviceSpecsFromDrmFiles` emits `DeviceSpec{..., Permissions: "rw"}`, which the
# kubelet turns into both the device injection and the matching cgroup allow.
#
# `limits`, NOT `requests`, AND THAT IS NOT A STYLE CHOICE: an extended resource must be
# expressed in `limits`. Kubernetes mirrors `requests` from `limits` automatically, and a
# requests-only extended resource is invalid. App container only, never a sidecar -- the
# oauth2-proxy sidecar does no transcoding.
#
# THE FAILURE MODE THIS TRADED INTO, STATED PLAINLY -- it is worse in blast radius, and
# unavoidable. BEFORE: a missing device meant `DirectoryOrCreate` mounted an empty dir,
# Plex ran and SILENTLY software-transcoded (invisible, but harmless to everything else).
# NOW: if the resource is unavailable (plugin broken, removed, or not yet reconciled) the
# pod is UNSCHEDULABLE, and because `apps-wave-0` is `wait: true` with `apps-wave-1`
# dependsOn it, that STALLS THE APP TIER. There is no graceful-degradation variant: an
# extended resource is a hard scheduling constraint, and it is also the only thing that
# produces the cgroup allow rule. Plex's schedulability is now coupled to the
# intel-gpu-plugin DaemonSet; that is the price of hardware transcoding working at all.
# ESCAPE HATCH (one field, no code change): delete the `hardware:` block from
# `catalog/plex.yaml` and re-render -- Plex returns to software transcoding and the tier
# unblocks. runbooks/plex-quicksync.md carries it.
#
# `podSecurityContext.supplementalGroups` is DELIBERATELY NOT SET here, and adding it
# would be a regression, not a hardening. MEASURED in the running container on prod-2
# (2026-09-30, re-measured 2026-10-01): the image's own cont-init hook
# `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` `find`s every char device
# under /dev/dri, `stat -c '%g'` each one's GID, `groupadd`s a matching group INSIDE the
# container when none exists, and `usermod -a -G` the app user into it -- every start,
# nothing hardcoded. It does exactly the same thing to the PLUGIN-INJECTED device, which
# is why this swap needs no group work either. cont-init.d completes before services.d,
# so the account database s6 reads at exec time already carries the group: measured live,
# `Plex Media Server` ran with `Groups: 44 100 992 1000`, where 992 is the host `render`
# GID. The service is started with `exec s6-setuidgid plex ...`, and `s6-setuidgid`
# REPLACES the process group list from /etc/group rather than appending to it, so
# anything Kubernetes puts in supplementalGroups reaches s6-svscan and is discarded
# before the app ever starts. A supplementalGroups entry here would be inert AND would
# mislead whoever debugs this next.
hw_gpu = (s.get("hardware") or {}).get("gpu")
if hw_gpu == "intel-quicksync":
container["resources"] = {"limits": {QUICKSYNC_RESOURCE: 1}}
if volume_mounts:
container["volumeMounts"] = volume_mounts
# lifecycle hooks (schema-optional): the k8s Lifecycle (postStart/preStop exec) is
# emitted VERBATIM. Used for idempotent break-glass/bootstrap steps (e.g. gitea's
# postStart exec that seeds a local admin). Absent => byte-identical for other apps.
if s.get("lifecycle"):
container["lifecycle"] = s["lifecycle"]
pod_spec = {"containers": [container]}
# forward-auth (Option B): append the per-pod oauth2-proxy reverse-proxy sidecar AFTER the app
# container (app stays containers[0]). The shared Gateway routes to the sidecar (:4180); it gates
# the co-located UI against Keycloak and only then proxies to http://127.0.0.1:<uiport>. `oidc`
# apps (native client) and `native`/`none` render byte-identical (no sidecar).
if _needs_oauth2_sidecar(s):
pod_spec["containers"].append(_oauth2_sidecar_container(s))
# podSecurityContext (schema-optional): the k8s PodSecurityContext is emitted VERBATIM onto the
# pod spec. Its primary use is `fsGroup`, which makes a mounted Longhorn PVC group-writable by a
# non-root image (e.g. semaphore runs as UID 1001 and cannot otherwise write its config PVC).
# Absent => the key is never set, so apps that declare none render byte-identical.
if s.get("podSecurityContext"):
pod_spec["securityContext"] = s["podSecurityContext"]
if volumes:
pod_spec["volumes"] = volumes
ann = _auth_annotations(s)
tmpl_meta = {"labels": pod_labels}
if ann:
tmpl_meta["annotations"] = ann
spec = {"replicas": 1}
# An app with persistent storage mounts a ReadWriteOnce Longhorn PVC. The default
# RollingUpdate strategy tries to start the NEW pod before terminating the old one, but
# the old pod still holds the RWO volume -> the new pod is stuck ContainerCreating and
# the rollout deadlocks (single-replica app can never hand the volume over). Recreate
# terminates the old pod first, freeing the PVC for the new one (R-STATE-01/02).
if s["storage"]:
spec["strategy"] = {"type": "Recreate"}
spec["selector"] = {"matchLabels": {"app.kubernetes.io/name": s["name"]}}
spec["template"] = {"metadata": tmpl_meta, "spec": pod_spec}
return {
"apiVersion": "apps/v1",
"kind": "Deployment",
"metadata": {
"name": s["name"],
"namespace": s["name"],
"labels": {"app.kubernetes.io/name": s["name"]},
},
"spec": spec,
}
def r_service(s: dict):
ports = _service_ports(s)
# A declared metrics endpoint becomes a NAMED Service port, because that name is the only
# thing a ServiceMonitor endpoint can select on. Appended BELOW rather than inside
# _service_ports so it never leaks into r_gateway_ingress_policy's north-south allow list —
# the metrics port is for the monitoring namespace, not for the Gateway.
ep = s["observability"].get("metricsEndpoint")
if not ports and not ep:
return None
# forward-auth (Option B): the shared Gateway must reach the oauth2-proxy sidecar, so front it
# with a `authproxy` :4180 Service port (the HTTPRoute backendRef targets this). The app UI/API
# port(s) stay in the Service for EAST-WEST callers (e.g. arr -> qbittorrent:8080 / jackett:9117);
# north-south to those app ports is denied at the Gateway (see r_gateway_ingress_policy).
if _needs_oauth2_sidecar(s):
ports = ports + [
{"name": "authproxy", "port": OAUTH2_PROXY_PORT, "targetPort": OAUTH2_PROXY_PORT}
]
if ep:
ports = ports + [
{"name": ep["name"], "port": ep["port"], "targetPort": ep["port"]}
]
return {
"apiVersion": "v1",
"kind": "Service",
"metadata": {
"name": s["name"],
"namespace": s["name"],
"labels": {"app.kubernetes.io/name": s["name"]},
},
"spec": {
"selector": {"app.kubernetes.io/name": s["name"]},
"ports": ports,
},
}
def r_dependency_policies(s: dict) -> list:
"""One CiliumNetworkPolicy egress allow per dependency, on top of default-deny (R-NET-02).
Everything undeclared stays denied; empty dependencies => reaches only DNS.
Cross-namespace note: a CiliumNetworkPolicy's `toEndpoints` selector is scoped to the
POLICY's own namespace unless it carries the reserved `k8s:io.kubernetes.pod.namespace`
label. Every service lives in its own namespace named after it (r_namespace), and a
dependency always targets a DIFFERENT service/namespace, so the destination's namespace
(== dep["to"]) MUST be pinned here — otherwise Cilium looks for the destination's app
label inside the SOURCE namespace, finds nothing, and silently denies the egress (proven
in-cluster via Hubble: `EGRESS DENIED`). This pins it so the declared dependency's egress
half actually reaches the destination (verified: EGRESS ALLOWED / FORWARDED).
"""
out = []
for dep in s["dependencies"]:
proto = dep.get("protocol", "tcp").upper()
out.append(
{
"apiVersion": "cilium.io/v2",
"kind": "CiliumNetworkPolicy",
"metadata": {
"name": f"{s['name']}-allow-{dep['to']}-{dep['port']}",
"namespace": s["name"],
"labels": {"app.kubernetes.io/name": s["name"]},
},
"spec": {
"endpointSelector": {
"matchLabels": {"app.kubernetes.io/name": s["name"]}
},
"egress": [
{
"toEndpoints": [
{
"matchLabels": {
"app.kubernetes.io/name": dep["to"],
"k8s:io.kubernetes.pod.namespace": dep["to"],
}
}
],
"toPorts": [
{
"ports": [
{"port": str(dep["port"]), "protocol": proto}
]
}
],
}
],
},
}
)
return out
def build_reverse_map(sds: list) -> dict:
"""Build the cross-service reverse-dependency map (T-2.3 reciprocal ingress).
A `dependencies` edge is declared ONCE, in the SOURCE's catalog entry, as
`{to: X, port: P}`. With the default-deny-BOTH-directions baseline (R-NET-01), that one
declaration must compile into TWO matching halves (docs/03 "Explicit allow rules"; the
conformance fixtures spell it out: "egress permit on the source, ingress permit on the
destination"): (a) source egress-allow to X:P — rendered by r_dependency_policies on the
source — AND (b) X ingress-allow from source:P — the reciprocal half rendered here.
render.py otherwise processes each ServiceDefinition independently, so the destination X
never sees the SOURCE's declaration. This pass scans ALL ServiceDefinitions once and
returns `{destination_name: [{source, port, protocol}, ...]}` so that, when rendering X,
r_reverse_dependency_policies can emit the ingress half for every source that declares a
dependency ON X. Deterministic: each destination's inbound list is sorted by
(source, port, protocol) so the render stays byte-reproducible (golden-diffable).
"""
rev: dict = {}
for sd in sds:
if not isinstance(sd, dict) or "name" not in sd:
continue # tolerate non-ServiceDefinition siblings (e.g. a stray yaml)
s = apply_defaults(sd)
src = s["name"]
for dep in s["dependencies"]:
rev.setdefault(dep["to"], []).append(
{
"source": src,
"port": dep["port"],
"protocol": dep.get("protocol", "tcp"),
}
)
for dest in rev:
rev[dest].sort(key=lambda e: (e["source"], int(e["port"]), e["protocol"]))
return rev
def r_reverse_dependency_policies(s: dict, inbound: list) -> list:
"""Reciprocal ingress-allow CiliumNetworkPolicies on THIS service (R-NET-02).
`inbound` is this service's slice of the cross-service reverse map (build_reverse_map):
the list of sources that declared a dependency ON this service. Each becomes an ADDITIVE
ingress allow on top of default-deny, mirroring the source-side egress allow
(r_dependency_policies) exactly but inverted: `fromEndpoints` matching the SOURCE's
identity on the declared port/protocol. As with the egress half, the source lives in a
DIFFERENT namespace, and a CiliumNetworkPolicy's `fromEndpoints` selector is scoped to the
POLICY's own namespace unless it carries the reserved `k8s:io.kubernetes.pod.namespace`
label — so the source's namespace (== edge["source"]) is pinned alongside its app label.
Without both halves so pinned the destination's default-deny-ingress drops the source's
traffic and the declared dependency does not actually work (docs/03 R-NET-01/02; proven
in-cluster via Hubble: pinned => FORWARDED, unpinned => Policy denied DROPPED).
"""
out = []
for edge in inbound or []:
proto = edge.get("protocol", "tcp").upper()
out.append(
{
"apiVersion": "cilium.io/v2",
"kind": "CiliumNetworkPolicy",
"metadata": {
"name": f"{s['name']}-allow-{edge['source']}-{edge['port']}",
"namespace": s["name"],
"labels": {"app.kubernetes.io/name": s["name"]},
},
"spec": {
"endpointSelector": {
"matchLabels": {"app.kubernetes.io/name": s["name"]}
},
"ingress": [
{
"fromEndpoints": [
{
"matchLabels": {
"app.kubernetes.io/name": edge["source"],
"k8s:io.kubernetes.pod.namespace": edge["source"],
}
}
],
"toPorts": [
{
"ports": [
{"port": str(edge["port"]), "protocol": proto}
]
}
],
}
],
},
}
)
return out
def r_gateway_ingress_policy(s: dict):
"""North-south allow: let the shared Cilium Gateway reach an exposed service's port(s).
Any service that renders an HTTPRoute (non-empty ingressRoutes) is reachable through the
shared Gateway, but the namespace default-deny has `ingress: []` (deny all east-west and
north-south). This CiliumNetworkPolicy is the ADDITIVE, least-privilege allow that opens
ONLY the north-south path (docs/09 "Ingress opens north-south only — it does not relax
east-west default-deny"): ingress from Cilium's reserved `ingress` identity (the Gateway/
Ingress Envoy proxy, `reserved:ingress`, numeric identity 8 in Cilium 1.20.1) to ONLY the
exposed service port(s). Everything else stays denied; undeclared ports stay denied.
Verified empirically against the cluster: with default-deny in place, Hubble shows the
Gateway proxy reaching the backend as source identity `(ingress)` and the flow
`Policy denied DROPPED` on the service port — so `fromEntities: [ingress]` is exactly the
source the Cilium Gateway uses, and the port is the service targetPort (R-NET-01/02).
"""
if not s["ingressRoutes"]:
return None
# forward-auth (Option B): the ONLY north-south path is the oauth2-proxy sidecar. Open the
# Gateway to :4180 ONLY — never the app UI port — so the un-gated UI is unreachable from the
# Gateway (fail-closed; R-IDP-02). Every other app opens its exposed service port(s) as before.
if _needs_oauth2_sidecar(s):
allow_ports = [{"port": str(OAUTH2_PROXY_PORT), "protocol": "TCP"}]
else:
allow_ports = [
{"port": str(p["port"]), "protocol": "TCP"} for p in _service_ports(s)
]
return {
"apiVersion": "cilium.io/v2",
"kind": "CiliumNetworkPolicy",
"metadata": {
"name": f"{s['name']}-allow-gateway-ingress",
"namespace": s["name"],
"labels": {"app.kubernetes.io/name": s["name"]},
},
"spec": {
"endpointSelector": {
"matchLabels": {"app.kubernetes.io/name": s["name"]}
},
"ingress": [
{
"fromEntities": ["ingress"],
"toPorts": [{"ports": allow_ports}],
}
],
},
}
def r_in_cluster_client_policies(s: dict) -> list:
"""East-west allow: let a declared IN-CLUSTER client reach this service DIRECTLY (R-NET-02).
`ingressRoutes` opens the NORTH-SOUTH path only — r_gateway_ingress_policy admits the
reserved `ingress` identity (the Gateway proxy) and nothing else — and `dependencies`
compiles edges only BETWEEN CATALOG SERVICES (both halves are rendered from a catalog
entry's own declaration). A PLATFORM CONTROLLER that dials this service's ClusterIP
directly therefore had no authorable form at all, and under default-deny (R-NET-01) it is
DROPPED.
That gap is not theoretical: prod's `flux-system` source-controller clones the platform's
own repo from the in-cluster Gitea Service (clusters/prod/kustomization.yaml pins
`url: http://gitea.gitea.svc.cluster.local:3000/...` so prod's GitOps depends on neither
public DNS, Cloudflare, its own Gateway nor ${BASE_DOMAIN}). gitea-allow-gateway-ingress
admits `fromEntities: [ingress]` only, so the clone was denied, the source could not
reconcile — and the HTTPRoute that would have served a public name instead lives in a
DOWNSTREAM tier that cannot apply while the source is down. Source down -> routes never
rename -> source stays down. It had to be broken with a live `kubectl apply`, i.e. with a
cluster object outside git, which one `kubectl delete` or one rebuild would take away.
Each entry renders ONE least-privilege ingress CiliumNetworkPolicy, shaped exactly like the
reciprocal-dependency half (r_reverse_dependency_policies) but selecting the client by
NAMESPACE alone, because a platform controller's pod labels are the upstream chart's to
change and are not part of this platform's contract: `fromEndpoints` carrying ONLY the
reserved `k8s:io.kubernetes.pod.namespace` label, `toPorts` carrying ONLY the declared
port/protocol. Never `fromEntities: all`, and never the whole service. Additive: the
namespace default-deny baseline keeps denying every undeclared source and port.
The policy NAME is `<service>-allow-<name>-ingress`, `name` defaulting to the namespace —
it is operator-facing (`kubectl get cnp`, a Hubble drop), and letting an author pin it is
what allows an allowance that already had to be applied live to be authored under its
EXISTING name, so Flux adopts that object in place rather than creating a near-duplicate
beside it and leaving the live one unmanaged.
"""
out = []
for client in s["inClusterClients"]:
proto = client.get("protocol", "tcp").upper()
out.append(
{
"apiVersion": "cilium.io/v2",
"kind": "CiliumNetworkPolicy",
"metadata": {
"name": f"{s['name']}-allow-{client['name']}-ingress",
"namespace": s["name"],
"labels": {"app.kubernetes.io/name": s["name"]},
},
"spec": {
"endpointSelector": {
"matchLabels": {"app.kubernetes.io/name": s["name"]}
},
"ingress": [
{
"fromEndpoints": [
{
"matchLabels": {
"k8s:io.kubernetes.pod.namespace": client[
"namespace"
]
}
}
],
"toPorts": [
{
"ports": [
{"port": str(client["port"]), "protocol": proto}
]
}
],
}
],
},
}
)
return out
def _route_meta(s: dict, route: dict) -> dict:
"""HTTPRoute metadata: auth wiring hints PLUS the ExternalDNS scoping derived from
`exposure` (_externaldns_route_annotations — R-DNS-03: an `internal` service renders no
public DNS record, and now cannot)."""
meta = {
"name": f"{s['name']}-{route['port']}",
"namespace": s["name"],
"labels": {"app.kubernetes.io/name": s["name"]},
}
ann = {**_auth_annotations(s), **_externaldns_route_annotations(s)}
if ann:
meta["annotations"] = ann
return meta
def r_internet_egress(s: dict) -> dict:
"""North-south egress allow: let the app reach the INTERNET (Cilium reserved `world`
entity = everything outside the cluster), which is the whole point of the `egress` class
(docs/03: internet-class → WAN; vpn-class → steered through the node tun0 PIA tunnel with
the nftables kill-switch, ADR-0015). The default-deny baseline (R-NET-01) allows only DNS
so EAST-WEST stays denied; this ADDITIVE policy opens ONLY `world` (non-cluster) egress —
cluster/east-west traffic stays denied except the declared `dependencies`. Every app
declares an egress class (EGRESS_CLASSES), so every app gets this. The steering + kill-
switch (a cluster-wide singleton keyed on the egress-class pod label) decide the PATH; this
policy decides that north-south egress is ALLOWED at all.
"""
egress = [{"toEntities": ["world"]}]
# SSO apps (oidc/forward-auth) reach Keycloak at keycloak.home, which coredns split-horizon
# resolves to the shared Gateway VIP (an L2-announced LoadBalancer IP Cilium classifies as
# `world`), so the pod hairpins THROUGH the Gateway. Under Cilium >=1.16.8/1.17.2 (the
# CVE-2025-30162 fix, present in 1.20), the Gateway's L7 proxy re-evaluates THIS pod's egress
# policy against Keycloak's REAL backend cluster identity (enforce_policy_on_l7lb) — `world`
# matches the VIP at L3/L4 but NOT the resolved keycloak identity at L7, so OIDC discovery/
# token 403s ("Access denied") without an explicit allow to the keycloak backend. Scope it to
# the apps that actually talk to Keycloak (auth mode oidc or forward-auth).
if s.get("auth", {}).get("mode") in ("oidc", "forward-auth"):
egress.append(
{
"toEndpoints": [
{
"matchLabels": {
"io.kubernetes.pod.namespace": "keycloak",
"app": "keycloak",
}
}
]
}
)
return {
"apiVersion": "cilium.io/v2",
"kind": "CiliumNetworkPolicy",
"metadata": {
"name": f"{s['name']}-allow-internet-egress",
"namespace": s["name"],
"labels": {"app.kubernetes.io/name": s["name"]},
},
"spec": {
"endpointSelector": {"matchLabels": {"app.kubernetes.io/name": s["name"]}},
"egress": egress,
},
}
def r_httproutes(s: dict) -> list:
"""One HTTPRoute per ingressRoute, attaching host -> service port on the shared Gateway.
TLS is the Gateway's wildcard *.home.keithjackson.dev cert (no per-service cert step);
ExternalDNS manages the record from the HTTPRoute (R-TLS/R-DNS, renderer-contract).
"""
out = []
for route in s["ingressRoutes"]:
# forward-auth (Option B): the Gateway backendRef targets the oauth2-proxy sidecar (:4180),
# NOT the app UI port — so the gate is in the data path and the bare UI is never served
# un-authenticated. Other apps route straight to their exposed app port.
backend_port = OAUTH2_PROXY_PORT if _needs_oauth2_sidecar(s) else route["port"]
out.append(
{
"apiVersion": "gateway.networking.k8s.io/v1",
"kind": "HTTPRoute",
"metadata": _route_meta(s, route),
"spec": {
"parentRefs": [
{
"name": GATEWAY_NAME,
"namespace": GATEWAY_NS,
"sectionName": "https",
}
],
"hostnames": [route["host"]],
"rules": [
{
"matches": [
{"path": {"type": "PathPrefix", "value": "/"}}
],
"backendRefs": [
{"name": s["name"], "port": backend_port}
],
}
],
},
}
)
return out
def _needs_oauth2_sidecar(s: dict) -> bool:
"""True iff this service gets a per-pod oauth2-proxy reverse-proxy sidecar (Option B).
A `forward-auth` app with an exposed UI has no usable native OIDC, so its Gateway route is
gated IN the data path by a co-located oauth2-proxy sidecar (crafty's proven pattern). `oidc`
apps authenticate themselves (native Keycloak client) and `native`/`none` are unchanged."""
return s["auth"]["mode"] == "forward-auth" and bool(s["ingressRoutes"])
def _oauth2_sidecar_container(s: dict) -> dict:
"""The oauth2-proxy REVERSE-PROXY sidecar (Option B) that gates a forward-auth app's UI.
Mirrors platform/crafty/deployment.yaml's `oauth2-proxy` container EXACTLY, adapted per app:
the upstream is the co-located app UI over plain HTTP (http://127.0.0.1:<uiport>) — the *arr /
PiHole / Jackett / qBittorrent UIs are HTTP, so (unlike crafty's self-signed HTTPS :8443) there
is NO --ssl-upstream-insecure-skip-verify. --allowed-group is emitted once per declared
auth.group (R-IDP-06: a valid session is not enough — the user must be in an authorized group,
else /oauth2 returns 403). client-id/secret + cookie-secret come by name from the SOPS-encrypted
`<app>-oauth2-proxy` Secret (R-IDP-04). Single provider => --skip-provider-button 302s straight
to Keycloak. The sidecar->app hop is pod-local (127.0.0.1) and not subject to network policy.
--cookie-name and --cookie-domain are DERIVED per app (see the block comment on those two
args): every instance holds a different cookie-secret, so sharing one cookie name across the
zone makes the instances evict each other's sessions.
--api-route is DECLARED per app, from auth.apiRoutes, and is the one flag here that is
neither a constant nor derivable: it names the paths THIS app serves as a machine API, and
those differ per app (radarr `/signalr/`, jackett `/api/`, pihole none). An app that declares
none renders no --api-route at all."""
route = s["ingressRoutes"][0]
host = route["host"]
upstream_port = route["port"]
args = [
f"--http-address=0.0.0.0:{OAUTH2_PROXY_PORT}",
"--reverse-proxy=true",
"--provider=oidc",
f"--oidc-issuer-url={KEYCLOAK_ISSUER}",
f"--upstream=http://127.0.0.1:{upstream_port}",
"--email-domain=*",
"--scope=openid email profile",
"--oidc-groups-claim=groups",
]
# R-IDP-06: authorize by group. One flag per declared group (fail-closed to 403 otherwise).
for group in s["auth"].get("groups", []):
args.append(f"--allowed-group={group}")
args += [
"--skip-provider-button=true",
f"--redirect-url=https://{host}/oauth2/callback",
]
# ── API ROUTES: 401 instead of 302, ON THIS APP'S OWN PATHS ONLY ───────────────────────
# One flag per declared route, in declared order. An app that declares NONE renders NO
# --api-route, which is the correct and common case — see OAUTH2_API_ROUTE_PATTERN above for
# the measurements, for why this cannot be a renderer constant, and for the breadth ratchet.
# Validated here as well as in the schema so a breadth mistake is caught by render-golden
# even on a run where the schema stage did not execute.
for route in s["auth"].get("apiRoutes", []):
if not OAUTH2_API_ROUTE_PATTERN.match(str(route)):
raise SystemExit(
f"render.py: {s['name']}: auth.apiRoutes entry {route!r} is not an anchored path "
f"regex. Each entry must start with '^/' and continue with literal path "
f"characters (e.g. '^/api/', '^/signalr/', '^/api/v3/.*'). Rejected shapes: '^/' "
f"and '^/.*' (match every page load), an unanchored '/api/' (matches mid-path), "
f"and alternation (the flag is repeatable — declare one entry per route). A "
f"too-broad --api-route answers a PAGE LOAD with 401, so a logged-out operator "
f"sees a bare error page and is never offered a sign-in: the service looks "
f"permanently broken rather than merely unauthenticated."
)
args.append(f"--api-route={route}")
args += [
# ── COOKIE ISOLATION (fleet fix 2026-09-30; crafty's 2026-09-29 fix, generalized) ──────
# EVERY oauth2-proxy on this platform — this sidecar, its four siblings, and the SHARED
# forward-auth proxy (platform/oauth2-proxy) — is an INDEPENDENT instance holding its OWN
# `cookie-secret`. Measured on prod 2026-09-30: seven `*-oauth2-proxy` Secrets, seven
# DISTINCT cookie-secret values, so no instance can decrypt any other's cookie.
#
# Left at oauth2-proxy's DEFAULTS (`_oauth2_proxy` / `_oauth2_proxy_csrf`) and scoped to
# `.${BASE_DOMAIN}`, the browser therefore keeps ONE cookie pair for ALL of them:
# whichever proxy authenticates last OVERWRITES every other's session. The loser logs
# "No valid authentication in request" and bounces to Keycloak, which — the SSO session
# still being live — redirects straight back and overwrites the cookie again. The
# user-visible symptom is "I signed into Keycloak but I'm still not signed in", and it
# sends you hunting in Keycloak, where nothing is wrong. Silent, intermittent, and
# misattributed: the exact shape this repo keeps getting burned by.
#
# WHY IT WAS *INTERMITTENT*, AND WHY RELOADING NEVER HELPED. The loser does not merely
# fail to read the cookie — it calls ClearSession, which EXPIRES the cookie on its own
# --cookie-domain, i.e. the whole zone. And the *arr UIs poll: measured on prod, radarr's
# rejections are `/api/v3/command`, `/api/v3/queue/details` and `/signalr/negotiate`,
# 155 of them in 12h against 1 successful auth. Every poll carrying a cookie radarr
# cannot decrypt calls ClearSession again — so A LEFT-OPEN RADARR TAB DELETES JACKETT'S
# COOKIE EVERY FEW SECONDS, from another namespace, with no user action at all. That is
# the intermittency, and it is why signing in again fixed nothing for longer than the
# gap between two polls. Host-scoping the cookie (below) is what contains the blast
# radius of a ClearSession to the app that issued it.
#
# DERIVED, NOT LISTED. Both flags come from the ServiceDefinition (`s['name']` and the
# route host), so a NEW forward-auth app is isolated the moment it is onboarded. A
# per-service table would leave the next app defaulting into the collision, which is how
# twelve ServiceMonitors acquired one identical wrong port name in a single edit.
# conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct asserts both properties
# over the WHOLE tree and asserts THIS function derives rather than lists.
#
# --cookie-name a unique name per instance, so two sessions cannot alias at all.
# This is the fix; the one below is defence in depth.
# --cookie-domain HOST-scoped. A per-app reverse proxy has no reason to broadcast its
# session cookie to every other host in the zone — a zone-wide cookie
# is transmitted to every sibling app on `.${BASE_DOMAIN}`, so any
# FUTURE name collision would again be platform-wide instead of
# contained. The SHARED forward-auth proxy keeps the wide scope,
# because cross-host forward-auth is precisely its job.
#
# --whitelist-domain deliberately stays ZONE-WIDE: it governs permitted post-login
# REDIRECT targets, not cookie scope. Narrowing it here would break the sign-in bounce.
f"--cookie-name=_oauth2_proxy_{s['name']}",
f"--cookie-domain={host}",
f"--whitelist-domain=.{ZONE}",
"--cookie-secure=true",
# Spend the refresh token the proxy already holds, at half the 300s access-token
# lifetime. Without this the session dies 5 minutes after sign-in. See
# OAUTH2_ACCESS_TOKEN_LIFETIME_S above for the measurement and the
# --session-cookie-minimal incompatibility.
f"--cookie-refresh={OAUTH2_COOKIE_REFRESH}",
"--pass-access-token=true",
"--set-xauthrequest=true",
# Start the /metrics listener. Prerequisite only — not yet reachable by Prometheus;
# see OAUTH2_PROXY_METRICS_PORT above for exactly what is still missing.
f"--metrics-address=0.0.0.0:{OAUTH2_PROXY_METRICS_PORT}",
]
secret = f"{s['name']}-oauth2-proxy"
return {
"name": "oauth2-proxy",
"image": OAUTH2_PROXY_IMAGE,
"imagePullPolicy": "IfNotPresent",
"args": args,
"env": [
{
"name": "OAUTH2_PROXY_CLIENT_ID",
"valueFrom": {"secretKeyRef": {"name": secret, "key": "client-id"}},
},
{
"name": "OAUTH2_PROXY_CLIENT_SECRET",
"valueFrom": {"secretKeyRef": {"name": secret, "key": "client-secret"}},
},
{
"name": "OAUTH2_PROXY_COOKIE_SECRET",
"valueFrom": {"secretKeyRef": {"name": secret, "key": "cookie-secret"}},
},
],
"ports": [
{"name": "http", "containerPort": OAUTH2_PROXY_PORT, "protocol": "TCP"},
{"name": "metrics", "containerPort": OAUTH2_PROXY_METRICS_PORT, "protocol": "TCP"},
],
"resources": {
"requests": {"cpu": "25m", "memory": "64Mi"},
"limits": {"memory": "128Mi"},
},
"securityContext": {
"allowPrivilegeEscalation": False,
"readOnlyRootFilesystem": True,
"capabilities": {"drop": ["ALL"]},
},
"livenessProbe": {
"httpGet": {"path": "/ping", "port": "http"},
"initialDelaySeconds": 5,
"periodSeconds": 15,
},
"readinessProbe": {
"httpGet": {"path": "/ping", "port": "http"},
"initialDelaySeconds": 5,
"periodSeconds": 15,
},
}
def _auth_annotations(s: dict) -> dict:
"""Auth wiring hints (renderer-contract §auth): the platform (oauth2-proxy/Keycloak)
consumes these to protect ingressRoutes. oidc/forward-auth carry the authorized groups
(R-IDP-06: access is 'in an authorized group', not merely 'logged in')."""
mode = s["auth"]["mode"]
if mode in ("oidc", "forward-auth"):
groups = ",".join(s["auth"].get("groups", []))
return {
f"{LBL}/auth-mode": mode,
f"{LBL}/auth-groups": groups,
}
return {}
def _externaldns_route_annotations(s: dict) -> dict:
"""ExternalDNS scoping, rendered FROM `exposure` (renderer-contract §exposure; R-DNS-03).
`internal` renders NO public DNS record. That sentence was in the contract from the start and
nothing enforced it: ExternalDNS watched EVERY HTTPRoute the shared Gateway carried, so an
`internal` service's route published `<name>.${BASE_DOMAIN}` into the PUBLIC authoritative
Cloudflare zone, answering with the Gateway VIP — an RFC1918 address. Six of them did
(jackett, pihole, qbittorrent, radarr, semaphore, sonarr), which published the internal
subnet, the Gateway VIP and a full service inventory to anyone who asked, and made every one
of those names a DNS-rebind primitive that the SOHO router strips on the way back in.
THE MECHANISM, verified against the DEPLOYED version (chart >=1.21.0 <1.22.0, app v0.21.0 —
`registry.k8s.io/external-dns/external-dns:v0.21.0`, read off the live prod Deployment), not
from memory. external-dns v0.21.0 `source/annotations/processors.go`:
func IsControllerMismatch(entity metav1.ObjectMetaAccessor, rType string) bool {
value, ok := entity.GetObjectMeta().GetAnnotations()[ControllerKey]
if ok && value != ControllerValue { ...; return true }
return false
}
with `ControllerKey = "external-dns.alpha.kubernetes.io/controller"` and
`ControllerValue = "dns-controller"`. `source/gateway.go` calls it per route inside
`Endpoints()`, BEFORE any hostname is resolved:
if annotations.IsControllerMismatch(meta, src.rtKind) { continue }
so an HTTPRoute carrying any other value is not merely unpublished — it never becomes an
endpoint at all, for any hostname it names. `source/service.go` applies the identical check,
so the same annotation scopes the `service` source too if a rendered Service ever grows an
external-dns hostname.
WHY THIS IS STRUCTURAL AND NOT INCIDENTAL. The annotation is derived HERE, from `exposure`,
with no per-service list anywhere: every HTTPRoute this renderer emits for an `internal`
service carries it, and ci/render-golden.sh proves `apps/` reproduces byte-for-byte from
`catalog/`, so an `internal` service cannot EXIST in the applied tree without it. Flipping
`exposure: internal` to `public` is the only edit that can publish a name, which is exactly
the operator-facing declaration R-DNS-03 is written in terms of. The L0 check
conformance/tests/l0-static/internal-services-publish-no-public-dns re-proves all of that on
every gate run, including that this function reads `exposure` rather than naming services.
WHY NOT A CLUSTER-WIDE OPT-IN FILTER (`--label-filter`, chart `labelFilter`). It exists, and
v0.21.0's own `--help` names both sources in use as supporting it ("currently supported by
source types crd, gateway-httproute, ... service and ambassador-host"), so a "publish only
what is labelled" posture is available and is STRONGER by default. It is not taken here
because its blast radius is not this defect's: seven published names come from hand-authored
platform/apps manifests that carry no `exposure` field and that this renderer never sees
(auth/oauth2-proxy, crafty + crafty-mc, grafana, headlamp, the homepage apex, unifi). An
opt-in filter DELETES all seven the moment it is set, and whether each should stay published
is an operator decision, not a renderer one. Per-resource exclusion has exactly the blast
radius of the reported defect. The filter remains the available hardening once those seven
are adjudicated — see platform/externaldns/helmrelease.yaml.
"""
if s["exposure"] == "internal":
return {EXTERNALDNS_CONTROLLER_KEY: EXTERNALDNS_CONTROLLER_DISOWN}
return {}
def r_servicemonitor(s: dict):
"""ServiceMonitor — rendered ONLY when the SD says WHERE metrics are served.
THIS FUNCTION USED TO RENDER A MONITOR UNCONDITIONALLY FROM `metrics: true`, hardcoding
`endpoints: [{port: metrics}]`. No Service this renderer emits had ever defined a port of
that name, so all twelve monitors discovered ZERO targets: no `up{job="<app>"}` series
existed anywhere, `up{job="<app>"} == 0` could not fire, and `PrometheusTargetsMissing`
could not see the hole either, because a target that was never discovered has no `up` series
to be 0. Verified on prod 2026-09-13: `count by (job) (up)` listed 24 jobs and not one of
the twelve appeared; `up{job="plex"}` and `up{job="gitea"}` both returned nothing.
THE ROOT CAUSE WAS NOT A TYPO — it was that `metrics: true` cannot state a fact the workload
does not have. Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr, Semaphore and
Nextcloud serve no Prometheus endpoint at all without a separate exporter; Gitea can serve
one but only once its app.ini enables it. So the port name was never the missing piece. A
monitor is now rendered only from `observability.metricsEndpoint`, which names the port the
workload actually serves — and that same declaration renders the Service port (r_service) and
the monitoring-namespace ingress allow (r_monitoring_ingress_policy), so the three halves of
a working scrape cannot be authored apart.
`metrics: true` still means something without it: logs ship to Loki via the node agent, and
liveness comes from kube-state-metrics (r_prometheusrule) — neither needs a per-service CR.
"""
obs = s["observability"]
if not obs["metrics"]:
return None
ep = obs.get("metricsEndpoint")
if not ep:
return None
return {
"apiVersion": "monitoring.coreos.com/v1",
"kind": "ServiceMonitor",
"metadata": {
"name": s["name"],
"namespace": s["name"],
"labels": {"release": KP_RELEASE, "app.kubernetes.io/name": s["name"]},
},
"spec": {
"selector": {"matchLabels": {"app.kubernetes.io/name": s["name"]}},
"namespaceSelector": {"matchNames": [s["name"]]},
"endpoints": [
{"port": ep["name"], "path": ep["path"], "interval": ep["interval"]}
],
},
}
def r_monitoring_ingress_policy(s: dict):
"""Let the monitoring namespace reach a declared metrics port (R-NET-02, R-OBS-01).
renderer-contract.md has always said the default-deny baseline carries "a standing allow ...
for observability scraping from the monitoring namespace" — and no renderer output ever did.
Every app namespace ships `ingress: []`, so a scrape of an app metrics port would be DROPPED
by policy even with the Service port and the monitor endpoint in agreement. That is the same
trap Longhorn's own NetworkPolicies set (platform/longhorn/networkpolicy-metrics.yaml exists
for exactly this reason), and it is a THIRD independent way for a metrics SLI to be inert.
Rendered only alongside a declared metricsEndpoint, and it opens ONLY that port to ONLY the
monitoring namespace: an allow for a port nothing serves would be a standing hole for no
benefit.
"""
ep = s["observability"].get("metricsEndpoint")
if not ep or not s["observability"]["metrics"]:
return None
return {
"apiVersion": "cilium.io/v2",
"kind": "CiliumNetworkPolicy",
"metadata": {
"name": f"{s['name']}-allow-monitoring-scrape",
"namespace": s["name"],
"labels": {"app.kubernetes.io/name": s["name"]},
},
"spec": {
"endpointSelector": {"matchLabels": {"app.kubernetes.io/name": s["name"]}},
"ingress": [
{
"fromEndpoints": [
{"matchLabels": {"k8s:io.kubernetes.pod.namespace": MONITORING_NS}}
],
"toPorts": [
{"ports": [{"port": str(ep["port"]), "protocol": "TCP"}]}
],
}
],
},
}
def _up_sli_rule(s: dict) -> dict:
"""`slis: [up]` -> a `<name>-down` alert against a signal THAT EXISTS on this cluster.
IT USED TO READ `up{job="<name>"} == 0`. That is the textbook expression and it was inert
for all twelve services in the catalog, because `up` only exists for a target Prometheus
actually DISCOVERED and the rendered ServiceMonitor discovered none (see r_servicemonitor).
Worse than a wrong threshold: an inert rule and a correct rule are indistinguishable on a
healthy cluster, and the generic safety net cannot see the difference either — a target that
was never discovered has no `up` series for PrometheusTargetsMissing to find at 0.
Rewriting the SLI against the app's own /metrics was NOT an option for most of this catalog:
Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export no
Prometheus metrics without a separate exporter sidecar per app. So the `up` SLI is expressed
against kube-state-metrics, which IS scraped and DOES carry a series for every workload — the
same idiom KeycloakDown in platform/monitoring/alerts.yaml has used since it was fixed.
The `or absent(...)` half is deliberate and load-bearing: it makes the DISAPPEARANCE of the
signal fire the alert instead of silencing it. A rule that can only fire on `== 0` is a rule
that goes quiet when the workload object (or kube-state-metrics) vanishes, which is precisely
how the old expression failed. It is also why this rule cannot be permanently firing where the
workload exists: the series is present and non-zero on a healthy cluster (verified on prod).
"""
lw = s["observability"]["livenessWorkload"]
kind, obj = lw["kind"], lw["name"]
ns = s["name"]
if kind == "VirtualMachineInstance":
sel = f'{VMI_METRIC}{{namespace="{ns}", name="{obj}", phase="{VMI_RUNNING_PHASE}"}}'
expr = f"absent({sel})"
desc = (
f"{s['name']}: no KubeVirt VMI in phase {VMI_RUNNING_PHASE} for 5m "
f"(SLI: up, via {VMI_METRIC})."
)
else:
try:
metric, label = LIVENESS_METRICS[kind]
except KeyError:
raise ValueError(
f"{s['name']}: observability.livenessWorkload.kind {kind!r} has no liveness "
f"metric; known kinds: {sorted(LIVENESS_METRICS)} + VirtualMachineInstance"
) from None
sel = f'{metric}{{namespace="{ns}", {label}="{obj}"}}'
expr = f"{sel} == 0 or absent({sel})"
desc = (
f"{s['name']}: its {kind} has no available/ready replica for 5m, or the signal is "
f"gone entirely (SLI: up, via {metric})."
)
return {
"alert": f"{s['name']}-down".replace("_", "-"),
"expr": expr,
"for": "5m",
"labels": {"severity": "critical", "service": s["name"]},
"annotations": {
"summary": f"{s['name']} is down",
"description": desc,
},
}
def r_prometheusrule(s: dict):
"""Each SLI becomes at least one alert rule (R-OBS-01)."""
slis = s["observability"]["slis"]
if not slis:
return None
rules = []
for sli in slis:
if sli == "up":
rules.append(_up_sli_rule(s))
else:
rules.append(
{
"alert": f"{s['name']}-{sli}".replace("_", "-"),
"expr": f'absent({sli}{{service="{s["name"]}"}})',
"for": "10m",
"labels": {"severity": "warning", "service": s["name"]},
"annotations": {"summary": f"{s['name']} SLI {sli} unhealthy"},
}
)
return {
"apiVersion": "monitoring.coreos.com/v1",
"kind": "PrometheusRule",
"metadata": {
"name": f"{s['name']}-slis",
"namespace": s["name"],
"labels": {"release": KP_RELEASE, "app.kubernetes.io/name": s["name"]},
},
"spec": {"groups": [{"name": f"{s['name']}.slis", "rules": rules}]},
}
def r_image_automation(s: dict) -> list:
"""container kind -> Flux ImageRepository + ImagePolicy encoding channel/range (R-UPD).
Flux image-automation advances the running version on its scan interval; no human/AI
edits a tag (R-UPD-02). `pin` freezes automation. vm kind uses in-guest updates instead.
"""
if s["kind"] != "container":
return []
image = s["source"]["image"].split(":", 1)[0]
upd = s["update"]
interval = "1h0m0s"
imagerepo = {
"apiVersion": "image.toolkit.fluxcd.io/v1",
"kind": "ImageRepository",
"metadata": {"name": s["name"], "namespace": s["name"]},
"spec": {"image": image, "interval": interval},
}
policy_spec: dict = {"imageRepositoryRef": {"name": s["name"]}}
if upd.get("pin"):
# Operator freeze escape hatch: pin to a fixed tag, stopping automation.
policy_spec["filterTags"] = {"pattern": f"^{upd['pin']}$"}
policy_spec["policy"] = {"alphabetical": {"order": "asc"}}
elif upd.get("range"):
policy_spec["policy"] = {"semver": {"range": upd["range"]}}
else:
policy_spec["policy"] = {"semver": {"range": ">=0.0.0"}}
imagepolicy = {
"apiVersion": "image.toolkit.fluxcd.io/v1",
"kind": "ImagePolicy",
"metadata": {
"name": s["name"],
"namespace": s["name"],
"annotations": {
f"{LBL}/update-channel": upd.get("channel", "stable"),
f"{LBL}/auto-rollback": str(
upd.get("autoRollbackOnFailedHealth", True)
).lower(),
},
},
"spec": policy_spec,
}
return [imagerepo, imagepolicy]
# --- VM (kind: vm) path: KubeVirt VirtualMachine + CDI DataVolume ----------------------
def r_datavolume(s: dict) -> dict:
"""Standalone CDI DataVolume (NOT a dataVolumeTemplate) so the disk OUTLIVES the VM
and survives destroy/rebuild (R-STATE-01) — matches the vm-lifecycle fixture shape."""
root = s["storage"][0] if s["storage"] else {"name": "rootdisk", "size": "10Gi"}
return {
"apiVersion": "cdi.kubevirt.io/v1beta1",
"kind": "DataVolume",
"metadata": {
"name": f"{s['name']}-{root['name']}",
"namespace": s["name"],
},
"spec": {
"source": {"registry": {"url": f"docker://{s['source']['vmTemplate']}"}},
"storage": {
"accessModes": ["ReadWriteOnce"],
"storageClassName": "longhorn",
"resources": {"requests": {"storage": root.get("size", "10Gi")}},
},
},
}
VM_CLOUD_INIT = (
"#cloud-config\n"
"package_update: true\n"
"packages:\n"
" - dnf-automatic\n"
"write_files:\n"
" - path: /etc/dnf/automatic.conf\n"
" content: |\n"
" [commands]\n"
" upgrade_type = default\n"
" random_sleep = 0\n"
" download_updates = yes\n"
" apply_updates = yes\n"
"runcmd:\n"
" - [ systemctl, enable, --now, dnf-automatic.timer ]\n"
)
def r_virtualmachine(s: dict) -> dict:
"""KubeVirt VirtualMachine; egress-class propagates to the virt-launcher pod so the
shared Cilium egress steering + Kyverno policy see the class (R-PLAT-03; ADR-0015).
In-guest dnf-automatic delivers auto-updates with no human step (R-UPD-01, docs/11)."""
root = s["storage"][0] if s["storage"] else {"name": "rootdisk"}
pod_labels = {
"kubevirt.io/vm": s["name"],
f"{LBL}/egress-class": s["egress"],
f"{LBL}/auth-mode": s["auth"]["mode"],
}
return {
"apiVersion": "kubevirt.io/v1",
"kind": "VirtualMachine",
"metadata": {
"name": s["name"],
"namespace": s["name"],
"labels": {"app.kubernetes.io/name": s["name"]},
},
"spec": {
"running": False,
"template": {
"metadata": {"labels": pod_labels},
"spec": {
"domain": {
"cpu": {"cores": 1},
"memory": {"guest": "1536Mi"},
"devices": {
"disks": [
{"name": "rootdisk", "disk": {"bus": "virtio"}},
{"name": "cloudinit", "disk": {"bus": "virtio"}},
],
"interfaces": [{"name": "default", "masquerade": {}}],
},
"resources": {"requests": {"memory": "1536Mi"}},
},
"networks": [{"name": "default", "pod": {}}],
"volumes": [
{
"name": "rootdisk",
"dataVolume": {"name": f"{s['name']}-{root['name']}"},
},
{
"name": "cloudinit",
"cloudInitNoCloud": {"userData": VM_CLOUD_INIT},
},
],
},
},
},
}
# =============================================================================
# Substitution safety (R-ENV-02 / R-GITOPS-02) — refuse to emit a manifest that
# Flux's envsubst would silently eat before the workload ever runs.
# =============================================================================
# Every manifest this renderer writes lands under apps/wave-<N>/, and EVERY apps-wave-<N>
# Flux Kustomization declares `postBuild.substituteFrom: cluster-vars` (see
# flux_wave_kustomization_text below). Flux therefore runs envsubst over the WHOLE rendered
# document — including any shell script carried verbatim in a `lifecycle` hook or an env
# value — at reconcile time, long before a container sees it. The escape contract is stated
# in clusters/base/apps-vms.yaml and proven with `flux envsubst`:
#
# `${VAR}` -> SUBSTITUTED (empty when unset) => a shell brace must be written `$${VAR}`,
# which envsubst emits back as `${VAR}`.
# `$VAR` -> left alone => bare shell vars need NO escaping.
# `$(cmd)` -> left alone => command substitution needs NO escaping.
# `${A[0]}` -> HARD ERROR ("missing closing brace"), failing the whole Kustomization.
#
# This bit us for real (fixed 2026-09-01): gitea/nextcloud guarded their whole Keycloak OIDC
# registration with `if [ -z "${..._OIDC_CLIENT_SECRET:-}" ]`. Neither name is a cluster-var,
# so envsubst rendered the guard as `if [ -z "" ]` — ALWAYS TRUE — and every boot skipped the
# registration entirely. Flux's STRICT substitution mode cannot catch this class: it
# deliberately EXEMPTS the default-operator forms (`${V:-}`, `${V:=x}`, `${V:?}`), which is
# exactly the form a defensive shell script uses. So the renderer checks it itself.
CLUSTER_VARS_PATH = (
Path(__file__).resolve().parent.parent / "clusters" / "base" / "cluster-vars.yaml"
)
# A run of `$` followed by `{...}`. envsubst consumes `$$` as a literal `$`, so a run of N
# dollars leaves the brace SUBSTITUTED when N is odd (`${X}`, `$$${X}`) and LITERAL when N is
# even (`$${X}` — the escape). Deliberately greedy about the body so malformed/indexed forms
# (`${A[0]}`) are caught too rather than skipped.
_BRACED_EXPANSION_RE = re.compile(r"(\$+)\{([^{}]*)\}")
_PLAIN_IDENT_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_]*\Z")
_CLUSTER_VARS_CACHE: dict = {}
def cluster_vars_keys(path: Path | None = None) -> set:
"""The declared cluster-vars keys — the ONLY names envsubst may legitimately expand.
Read from clusters/base/cluster-vars.yaml (the single knob surface, R-ENV-02) so the
renderer never carries a stale hardcoded copy. Unreadable/absent => empty set, i.e.
FAIL-CLOSED: with no known knobs, every braced expansion must be escaped for the shell.
"""
key = str(path or CLUSTER_VARS_PATH)
if key not in _CLUSTER_VARS_CACHE:
try:
doc = yaml.safe_load(Path(key).read_text())
data = doc.get("data") if isinstance(doc, dict) else None
_CLUSTER_VARS_CACHE[key] = set(data) if isinstance(data, dict) else set()
except Exception: # pragma: no cover - absent/malformed => fail closed
_CLUSTER_VARS_CACHE[key] = set()
return _CLUSTER_VARS_CACHE[key]
def substitution_violations(text: str, known: set | None = None) -> list:
"""Every unescaped `${...}` in `text` that is NOT a bare, declared cluster-vars key.
Returns `(line_number, token, reason)` triples in line order. An expansion is ALLOWED only
when it is exactly `${KNOWN_KEY}` — a deliberate R-ENV-02 knob. Anything else is a bug:
a shell expansion that envsubst will eat (`${V:-}`), a typo'd/undeclared knob that
envsubst will blank, or an indexed form that hard-fails the whole Kustomization.
"""
keys = cluster_vars_keys() if known is None else known
out = []
for lineno, line in enumerate(text.splitlines(), start=1):
for m in _BRACED_EXPANSION_RE.finditer(line):
if len(m.group(1)) % 2 == 0:
continue # even run of `$` => escaped; envsubst emits it back literally
expr = m.group(2)
if _PLAIN_IDENT_RE.match(expr):
if expr in keys:
continue # a declared cluster-vars knob, intentionally substituted
reason = (
f"'{expr}' is not a declared cluster-vars key — envsubst will replace "
"it with the EMPTY STRING"
)
elif "[" in expr:
reason = (
"indexed expansion — envsubst hard-fails with 'missing closing brace' "
"and the whole Kustomization stops reconciling"
)
else:
reason = (
"shell expansion (default-operator/parameter form) — envsubst eats it "
"at reconcile time and STRICT mode exempts this form, so nothing else "
"will catch it"
)
out.append((lineno, m.group(0), reason))
return out
def assert_substitution_safe(text: str, origin: str = "<render>") -> None:
"""Refuse to emit a manifest carrying an unescaped braced expansion (see above).
Raises ValueError listing every offending token. The fix is almost always to escape the
shell brace as `$${...}` (envsubst emits a literal `${...}` for the shell), or — if the
value really is an environment knob — to declare it in clusters/base/cluster-vars.yaml
and write it as a bare `${KNOB}`.
"""
bad = substitution_violations(text)
if not bad:
return
lines = [
f"{origin}: refusing to emit — unescaped `${{...}}` in a manifest that Flux's "
"postBuild.substituteFrom (cluster-vars) will substitute at reconcile time:"
]
for lineno, token, reason in bad:
lines.append(f" line {lineno}: {token} -> {reason}")
lines.append(
" FIX: a SHELL expansion must be escaped `$${...}` (envsubst emits a literal "
"`${...}`); bare `$VAR` / `$(cmd)` need no escaping. A genuine environment knob "
"must be declared in clusters/base/cluster-vars.yaml and written bare `${KNOB}`. "
"Contract: clusters/base/apps-vms.yaml."
)
raise ValueError("\n".join(lines))
# =============================================================================
# Top-level render + emit
# =============================================================================
def render(sd: dict, image_automation: bool = True, inbound: list | None = None) -> list:
"""Return an ordered list of (header_comment, resource_dict) for a ServiceDefinition.
image_automation=False omits the Flux ImageRepository/ImagePolicy CRs — used when
onboarding apps before the Flux image-reflector/-automation controllers are installed
(their CRDs would otherwise be missing and the apps Kustomization would fail). The L1
render goldens use the default (True), so their expected output is unchanged.
inbound is this service's slice of the cross-service reverse-dependency map
(build_reverse_map): the sources that declared a dependency ON this service. It renders
the reciprocal ingress-allow half of each declared dependency (R-NET-02). None/[] (the
default, and every single-service render with no cross-catalog context) renders no
reciprocal ingress — so a lone render is unchanged and stays golden-diffable.
"""
s = apply_defaults(sd)
if s["kind"] not in ("container", "vm"):
raise ValueError(f"unknown kind: {s['kind']!r}")
if s["egress"] not in EGRESS_CLASSES:
raise ValueError(f"invalid egress class: {s['egress']!r}")
if s["auth"]["mode"] not in AUTH_MODES:
raise ValueError(f"invalid auth.mode: {s['auth']['mode']!r}")
docs: list = []
docs.append(("Namespace (trust zone; default-deny + policies apply)", r_namespace(s)))
docs.append(("Default-deny baseline (R-NET-01): deny all, allow only DNS", r_default_deny(s)))
if s["kind"] == "container":
# Container PVCs. (For a vm, the CDI DataVolume owns/creates the disk PVC, so we
# never render a plain PVC for the VM root disk — it would collide with CDI.)
# The static bulk PV is rendered BEFORE the PVC that names it. Document order inside one
# kustomize build is not an apply-ordering guarantee, but a PVC whose PV does not exist
# yet merely stays Pending and binds on the next reconcile, so this is a readability
# convention rather than a correctness requirement.
for pv in r_bulk_pvs(s):
docs.append(
(f"PersistentVolume {pv['metadata']['name']} - static external NFS (bulk tier)", pv)
)
for pvc in r_pvcs(s):
docs.append((f"PVC {pvc['metadata']['name']} (R-STATE)", pvc))
docs.append(("Deployment (R-PLAT-02)", r_deployment(s)))
svc = r_service(s)
if svc:
docs.append(("Service", svc))
else:
docs.append(("CDI DataVolume (standalone; survives rebuild, R-STATE-01)", r_datavolume(s)))
docs.append(("KubeVirt VirtualMachine (R-PLAT-03)", r_virtualmachine(s)))
for cnp in r_dependency_policies(s):
docs.append((f"CiliumNetworkPolicy egress allow (R-NET-02): {cnp['metadata']['name']}", cnp))
for cnp in r_reverse_dependency_policies(s, inbound or []):
docs.append(
(
"CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): "
f"{cnp['metadata']['name']}",
cnp,
)
)
gw_allow = r_gateway_ingress_policy(s)
if gw_allow:
docs.append(
(
"CiliumNetworkPolicy gateway-ingress allow (R-NET-01/02): "
f"{gw_allow['metadata']['name']}",
gw_allow,
)
)
for cnp in r_in_cluster_client_policies(s):
docs.append(
(
"CiliumNetworkPolicy in-cluster client ingress allow (R-NET-02): "
f"{cnp['metadata']['name']}",
cnp,
)
)
mon_allow = r_monitoring_ingress_policy(s)
if mon_allow:
docs.append(
(
"CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): "
f"{mon_allow['metadata']['name']}",
mon_allow,
)
)
ie = r_internet_egress(s)
docs.append(
(
"CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): "
f"{ie['metadata']['name']}",
ie,
)
)
for hr in r_httproutes(s):
docs.append((f"HTTPRoute {hr['metadata']['name']} (R-TLS/R-DNS)", hr))
sm = r_servicemonitor(s)
if sm:
docs.append(("ServiceMonitor (R-OBS-01)", sm))
pr = r_prometheusrule(s)
if pr:
docs.append(("PrometheusRule SLI alerts (R-OBS-01)", pr))
if image_automation:
for res in r_image_automation(s):
docs.append((f"Flux {res['kind']} (R-UPD)", res))
return docs
def emit(docs: list, origin: str = "<render>") -> str:
"""Deterministic multi-document YAML text (golden-diffable).
Every emitted document is checked against the substitution contract before it is handed
back (assert_substitution_safe): the rendered manifest lands on a Flux path that runs
envsubst over it, so an unescaped `${...}` meant for a SHELL is silently blanked. Raises
ValueError rather than emitting such a manifest.
"""
parts = []
for comment, resource in docs:
body = yaml.dump(
resource,
sort_keys=False,
default_flow_style=False,
width=4096,
)
parts.append(f"---\n# {comment}\n{body}".rstrip() + "\n")
text = "".join(parts)
assert_substitution_safe(text, origin)
return text
def load_sd(path: Path) -> dict:
with path.open() as fh:
return yaml.safe_load(fh)
def _load_catalog(dir_: Path) -> list:
"""Load every *.yaml ServiceDefinition in dir_ (tolerating unparseable siblings).
Used to build the cross-service reverse-dependency map for a single-file render, so
`render.py catalog/X.yaml` still emits X's reciprocal ingress for sources elsewhere in
the same catalog. Deterministic (sorted); render.py itself is not a *.yaml so it is skipped.
"""
sds = []
for p in sorted(dir_.glob("*.yaml")):
try:
doc = load_sd(p)
except Exception: # pragma: no cover - a malformed sibling must not break the render
continue
if isinstance(doc, dict):
sds.append(doc)
return sds
def render_file(path: Path, image_automation: bool = True, catalog_dir: Path | None = None) -> str:
"""Render one ServiceDefinition file, wiring in its reciprocal ingress allows.
The reverse-dependency map is built across the whole catalog (catalog_dir, defaulting to
the file's own directory) so this service receives the ingress half of every dependency
that OTHER services declare on it (R-NET-02). A directory with only this file (or with no
inbound edges — e.g. the L1 render fixtures) yields an empty inbound list, leaving the
render byte-identical to the pre-change output.
"""
sd = load_sd(path)
src_dir = catalog_dir if catalog_dir is not None else path.parent
rev = build_reverse_map(_load_catalog(src_dir))
inbound = rev.get(sd["name"], [])
return emit(
render(sd, image_automation=image_automation, inbound=inbound), origin=str(path)
)
# =============================================================================
# Wave partitioning — dependency-ordered app-tier delivery (WS-APPS wave delivery)
# =============================================================================
# A single monolithic `apps` Flux Kustomization health-gates (wait:true) the ENTIRE app
# set, so one crashlooping workload blocks ALL apps and every new revision. Instead we split
# the tier into dependency-ordered WAVES: each wave is its own Flux Kustomization
# `apps-wave-<N>` that dependsOn wave N-1 (wave 0 dependsOn platform-config), so a failure in
# a later wave can no longer stall an earlier one, while everything WITHIN a wave still
# reconciles in parallel. This module computes the wave assignment from the declared catalog
# dependency graph and emits the per-wave kustomize + Flux Kustomization CRs. It changes only
# WHERE a rendered manifest is placed — never its bytes (the L1 goldens stay green).
CO_DB_SUFFIX = "-db" # a co-located CNPG database for `<svc>` is the node `<svc>-db`.
# Non-catalog app-tier units (they carry NO ServiceDefinition, so compute_waves can't see
# them). Each is anchored to a catalog service; `above=True` forces it into the wave STRICTLY
# ABOVE its anchor because it is a hard runtime prerequisite it cannot start without (e.g. the
# Gitea Actions act_runner registers against + clones from Gitea, so Gitea must be healthy
# first); `above=False` would co-locate it in the anchor's own wave.
EXTRA_UNITS: dict = {
"gitea-actions": {
"anchor": "gitea",
"above": True,
"files": [
"gitea-actions.yaml",
"gitea-actions-runner-secret.sops.yaml",
],
},
# PiHole's DNS exposure: the udp/53 + tcp/53 LoadBalancer Services on a pinned VIP, plus the
# narrow LAN ingress CiliumNetworkPolicy that admits them (apps/wave-0/pihole-dns.yaml).
# `above: False` co-locates it in pihole's OWN wave: these are the pihole pod's own Services and
# its own policy, applied in the same wave as the Deployment they select, exactly as the
# rendered Service and CNPs in pihole.yaml are. It is NOT a runtime prerequisite of anything, so
# there is nothing for a higher wave to gate on.
#
# It is a non-catalog unit because the ServiceDefinition schema cannot express a LoadBalancer
# Service, a pinned lbipam VIP, a sharing key or UDP exposure at all (additionalProperties is
# false, and `ingressRoutes` only renders HTTPRoutes on the L7 Gateway) — the same reason
# apps/vms/unifi/50-service-l4.yaml and platform/crafty/service-l4.yaml are hand-written. The
# split-horizon ANSWERS are in the catalog, as FTLCONF env on catalog/pihole.yaml.
"pihole-dns": {
"anchor": "pihole",
"above": False,
"files": [
"pihole-dns.yaml",
],
},
# Plex's remote-access exposure: the raw tcp/32400 LoadBalancer Service on a pinned VIP, plus the
# CiliumNetworkPolicy that admits off-cluster clients to it (apps/wave-0/plex-remote.yaml).
# `above: False` co-locates it in plex's OWN wave — these are the plex pod's own Service and its
# own policy, applied in the same wave as the Deployment they select, exactly as the rendered
# Service and CNPs in plex.yaml are. Nothing in a higher wave gates on it.
#
# It is a non-catalog unit for the same reason pihole-dns is: the ServiceDefinition schema cannot
# express a LoadBalancer Service or a pinned lbipam VIP (additionalProperties is false, and
# `ingressRoutes` only renders HTTPRoutes on the L7 Gateway, which cannot carry Plex's raw port).
# Plex's own settings stay in the catalog as env on catalog/plex.yaml; the only thing living
# outside it is the exposure the schema has no word for.
"plex-remote": {
"anchor": "plex",
"above": False,
"files": [
"plex-remote.yaml",
],
},
}
def compute_waves(sds: list) -> dict:
"""Longest-dependency-path wave per node over the catalog dependency DAG.
Nodes = every catalog service ∪ every dependency target. A dependency target that has NO
ServiceDefinition (a co-located `<svc>-db` CNPG database) is a leaf. wave(n) = 0 if n
depends on nothing, else 1 + max(wave(dep)) over its declared dependencies. Because a
co-located DB is a leaf and its consumer declares a dependency ON it, the consumer always
lands in a strictly higher wave — the "DB strictly below its consumer" guarantee falls
straight out of the longest-path rule (no special case). Raises ValueError on a cycle.
"""
edges: dict = {} # node -> set(targets it depends on)
nodes: set = set()
for sd in sds:
if not isinstance(sd, dict) or "name" not in sd:
continue # tolerate non-ServiceDefinition siblings (e.g. render.py itself)
s = apply_defaults(sd)
name = s["name"]
nodes.add(name)
edges.setdefault(name, set())
for dep in s["dependencies"]:
edges[name].add(dep["to"])
nodes.add(dep["to"])
edges.setdefault(dep["to"], set())
wave: dict = {}
visiting: set = set()
def depth(n: str) -> int:
if n in wave:
return wave[n]
if n in visiting:
raise ValueError(f"dependency cycle through {n!r}")
visiting.add(n)
d = 0
for t in sorted(edges.get(n, ())):
d = max(d, depth(t) + 1)
visiting.discard(n)
wave[n] = d
return d
for n in sorted(nodes):
depth(n)
return wave
def _catalog_by_name(sds: list) -> dict:
out = {}
for sd in sds:
if isinstance(sd, dict) and "name" in sd:
s = apply_defaults(sd)
out[s["name"]] = s
return out
def service_wave(sds: list, name: str) -> int:
"""The wave a single service's rendered manifest belongs in (for wave-aware `-o`)."""
return compute_waves(sds).get(name, 0)
def plan_apps_waves(sds: list, present_files: set) -> dict:
"""Map the ONBOARDED app-tier file set into waves -> sorted resource basenames.
`present_files` is the set of manifest basenames currently committed under apps/ (flat or
already in wave-<N>/ subdirs), e.g. {"pihole.yaml", "nextcloud-db.yaml",
"nextcloud-db-app.sops.yaml", ...}. A catalog service is ONBOARDED iff `<svc>.yaml` is
present (the catalog also holds not-yet-onboarded SDs like keycloak/vm-demo, which must
NOT be placed). Each onboarded unit's sidecar files inherit the correct wave by naming
convention:
- `<svc>.yaml` -> wave(svc)
- `<svc>-bootstrap-secret.sops.yaml` -> wave(svc) (consumed by svc's env, same apply)
- `<svc>-claim-secret.sops.yaml` -> wave(svc) (a one-shot upstream-service claim token
consumed by svc's env — plex's plex.tv claim. Same wave as the app for the same reason the
bootstrap secret is: an env `secretKeyRef` has no `optional`, so the Secret must be applied
in the SAME reconcile as the workload or the pod is CreateContainerConfigError.)
- `<svc>-oauth2-proxy-credentials.sops.yaml` -> wave(svc) (forward-auth sidecar OIDC creds)
- `<svc>-db.yaml` -> wave(svc-db) (a strictly-lower wave; CNPG DB)
- `<svc>-db-app.sops.yaml` -> wave(svc-db) (CNPG initdb owner secret)
- `<svc>-db-backup-credentials.sops.yaml` -> wave(svc-db) (CNPG Barman S3 backup secret)
- `<svc>-db-backup-credentials-prod.sops.yaml` -> wave(svc-db) (the PROD half of that
secret: the two environments carry DIFFERENT object-store credentials, selected in
the Cluster CR by `${BACKUP_CREDENTIALS_SUFFIX}`. Both objects ship to both clusters,
so both must be placed in the DB wave — the Secret has to exist before the Cluster
that names it.)
Non-catalog units (EXTRA_UNITS, e.g. gitea-actions) place by their declared anchor+offset.
Only files that actually exist in `present_files` are placed, so the plan is robust to a
service having no bootstrap secret or no DB. Deterministic (sorted).
"""
waves = compute_waves(sds)
by_name = _catalog_by_name(sds)
plan: dict = {}
def add(w: int, f: str) -> None:
if f in present_files:
plan.setdefault(w, set()).add(f)
onboarded = {n for n in by_name if f"{n}.yaml" in present_files}
for name in sorted(onboarded):
w = waves[name]
add(w, f"{name}.yaml")
add(w, f"{name}-bootstrap-secret.sops.yaml")
# A one-shot claim token for an UPSTREAM service (plex's plex.tv claim). Same wave as the
# app: it is read by the app's env at pod start, and an env secretKeyRef has no `optional`,
# so a Secret applied in a later wave would leave the pod in CreateContainerConfigError.
# Placed only when the file exists, so services without one are unaffected.
add(w, f"{name}-claim-secret.sops.yaml")
# forward-auth (Option B): the per-app oauth2-proxy sidecar's OIDC client + cookie secret.
# Same wave as the app (the sidecar reads it at pod start); placed only when the file exists,
# so non-forward-auth apps are unaffected. Mirrors the bootstrap-secret naming convention.
add(w, f"{name}-oauth2-proxy-credentials.sops.yaml")
db = f"{name}{CO_DB_SUFFIX}"
if any(dep["to"] == db for dep in by_name[name]["dependencies"]):
wdb = waves[db]
add(wdb, f"{db}.yaml")
add(wdb, f"{db}-app.sops.yaml")
add(wdb, f"{db}-backup-credentials.sops.yaml") # CNPG Barman S3 creds (R-STATE-03)
# PROD half of the same credential (2026-09-07 per-env split). Placed only when
# the file exists, so a DB that has not been split is unaffected.
add(wdb, f"{db}-backup-credentials-prod.sops.yaml")
for unit, cfg in EXTRA_UNITS.items():
anchor = cfg["anchor"]
if anchor not in onboarded:
continue
w = waves[anchor] + (1 if cfg.get("above") else 0)
for f in cfg["files"]:
add(w, f)
return {w: sorted(fs) for w, fs in plan.items()}
def wave_kustomization_text(wave: int, files: list) -> str:
"""The kustomize Kustomization for one wave directory (apps/wave-<N>/kustomization.yaml)."""
dep = "platform-config" if wave == 0 else f"apps-wave-{wave - 1}"
header = (
f"# Wave {wave} of the app tier — GENERATED by `catalog/render.py --emit-waves`.\n"
f"# Reconciled by the Flux Kustomization `apps-wave-{wave}` "
f"(clusters/base/apps-wave-{wave}.yaml), which dependsOn `{dep}` and health-gates\n"
f"# (wait:true) ONLY this wave — a crashloop here cannot stall an earlier wave.\n"
f"# DO NOT hand-edit: re-run `render.py --emit-waves` after onboarding a service."
)
body = {
"apiVersion": "kustomize.config.k8s.io/v1beta1",
"kind": "Kustomization",
"resources": list(files),
}
dumped = yaml.dump(body, sort_keys=False, default_flow_style=False, width=4096)
return f"---\n{header}\n{dumped}"
def flux_wave_kustomization_text(wave: int) -> str:
"""The Flux Kustomization CR for one wave (clusters/base/apps-wave-<N>.yaml).
dependsOn wave N-1 (wave 0 dependsOn platform-config); prune+wait match the retired
monolith. SOPS decryption is added by the environment overlay patch, IDENTICALLY to the
former single `apps` Kustomization (clusters/dev patches each apps-wave-<N> by name) — the
base stays env-agnostic and references the out-of-band `sops-age` Secret by name only.
"""
dep = "platform-config" if wave == 0 else f"apps-wave-{wave - 1}"
header = (
f"# Flux Kustomization: app-tier WAVE {wave}. Reconciles ./apps/wave-{wave} AFTER "
f"`{dep}` is healthy.\n"
f"# GENERATED by `catalog/render.py --emit-waves`. Waves isolate a crashloop to its "
f"own wave + downstream:\n"
f"# wait:true health-gates ONLY this wave's resources, so a later wave can no longer "
f"stall an earlier one.\n"
f"# SOPS decryption is layered on per-environment (clusters/dev patches this "
f"Kustomization by name), identical to the retired `apps` Kustomization."
)
cr = {
"apiVersion": "kustomize.toolkit.fluxcd.io/v1",
"kind": "Kustomization",
"metadata": {"name": f"apps-wave-{wave}", "namespace": "flux-system"},
"spec": {
"interval": "10m",
"retryInterval": "2m",
"timeout": "5m",
"prune": True,
"wait": True,
"dependsOn": [{"name": dep}],
"sourceRef": {"kind": "GitRepository", "name": "flux-system"},
"path": f"./apps/wave-{wave}",
"postBuild": {
"substituteFrom": [{"kind": "ConfigMap", "name": "cluster-vars"}]
},
},
}
dumped = yaml.dump(cr, sort_keys=False, default_flow_style=False, width=4096)
return f"---\n{header}\n{dumped}"
def _discover_app_files(apps_dir: Path) -> set:
"""Basenames of every committed app-tier manifest under apps/ (flat OR in wave-<N>/).
Excludes the generated per-wave kustomization.yaml and any README. Deterministic.
"""
present = set()
for p in apps_dir.rglob("*.yaml"):
if p.name == "kustomization.yaml":
continue
present.add(p.name)
return present
def emit_waves(catalog_dir: Path, apps_dir: Path, clusters_dir: Path) -> dict:
"""Write apps/wave-<N>/kustomization.yaml + clusters/base/apps-wave-<N>.yaml.
Computes the wave plan from the catalog DAG and the app files present under apps/, then
(re)generates the per-wave kustomize files and the Flux Kustomization CRs. Idempotent:
it never moves or rewrites a service manifest (only its placement changes, done in git),
so the rendered bytes — and the L1 goldens — are untouched. Returns the wave plan.
"""
sds = _load_catalog(catalog_dir)
present = _discover_app_files(apps_dir)
plan = plan_apps_waves(sds, present)
for wave in sorted(plan):
wdir = apps_dir / f"wave-{wave}"
wdir.mkdir(parents=True, exist_ok=True)
(wdir / "kustomization.yaml").write_text(
wave_kustomization_text(wave, plan[wave])
)
(clusters_dir / f"apps-wave-{wave}.yaml").write_text(
flux_wave_kustomization_text(wave)
)
return plan
# =============================================================================
# Self-test (deterministic, no cluster) — `render.py --selftest`
# =============================================================================
def _selftest() -> int:
failures = []
def check(cond, msg):
if not cond:
failures.append(msg)
# 1) A minimal container SD renders the mandatory contract invariants.
sd = {"name": "demo", "source": {"image": "nginx:1"}}
docs = render(sd)
kinds = [d[1]["kind"] for d in docs]
check("Namespace" in kinds, "container: missing Namespace")
check("NetworkPolicy" in kinds, "container: missing default-deny NetworkPolicy")
check("Deployment" in kinds, "container: missing Deployment")
ns = next(d[1] for d in docs if d[1]["kind"] == "Namespace")
check(
ns["metadata"]["labels"].get(f"{LBL}/app-namespace") == "true",
"namespace missing app-namespace=true (Kyverno hook)",
)
dep = next(d[1] for d in docs if d[1]["kind"] == "Deployment")
plabels = dep["spec"]["template"]["metadata"]["labels"]
check(
plabels.get(f"{LBL}/egress-class") == "internet",
"default egress-class must be internet (R-NET-05)",
)
check(
plabels.get(f"{LBL}/auth-mode") in AUTH_MODES,
"pod must carry a valid auth-mode label (R-IDP-02)",
)
ndp = next(d[1] for d in docs if d[1]["kind"] == "NetworkPolicy")
check(
ndp["spec"]["policyTypes"] == ["Ingress", "Egress"],
"default-deny must cover both directions (R-NET-01)",
)
check(ndp["spec"]["ingress"] == [], "default-deny must deny all ingress (R-NET-01)")
# 2) egress: vpn flips ONLY the egress-class label (R-NET-07).
internet_docs = render({"name": "x", "source": {"image": "a:1"}, "egress": "internet"})
vpn_docs = render({"name": "x", "source": {"image": "a:1"}, "egress": "vpn"})
i_pod = next(d[1] for d in internet_docs if d[1]["kind"] == "Deployment")
v_pod = next(d[1] for d in vpn_docs if d[1]["kind"] == "Deployment")
check(
i_pod["spec"]["template"]["metadata"]["labels"][f"{LBL}/egress-class"] == "internet"
and v_pod["spec"]["template"]["metadata"]["labels"][f"{LBL}/egress-class"] == "vpn",
"egress class must render onto the pod label",
)
# Everything except the egress-class label must be identical (R-NET-07).
i_txt = emit(internet_docs).replace("egress-class: internet", "EGRESS")
v_txt = emit(vpn_docs).replace("egress-class: vpn", "EGRESS")
check(i_txt == v_txt, "flipping egress class changed more than the egress path (R-NET-07)")
# 3) Determinism: same input twice -> byte-identical output.
check(emit(render(sd)) == emit(render(sd)), "render is not deterministic")
# 4) dependencies -> one CiliumNetworkPolicy allow each (R-NET-02).
dep_docs = render(
{
"name": "y",
"source": {"image": "a:1"},
"dependencies": [{"to": "db", "port": 5432}],
}
)
cnps = [d[1] for d in dep_docs if d[1]["kind"] == "CiliumNetworkPolicy"]
dep_cnps = [c for c in cnps if c["metadata"]["name"] == "y-allow-db-5432"]
check(len(dep_cnps) == 1, "one dependency must render one CiliumNetworkPolicy allow")
if dep_cnps:
to_ep = dep_cnps[0]["spec"]["egress"][0]["toEndpoints"][0]["matchLabels"]
check(
to_ep.get("k8s:io.kubernetes.pod.namespace") == "db"
and to_ep.get("app.kubernetes.io/name") == "db",
"egress allow must pin the destination namespace (cross-ns Cilium selector)",
)
check(
any(c["metadata"]["name"] == "y-allow-internet-egress" for c in cnps),
"every app (egress class) must render a world/internet egress allow",
)
# 4a) reciprocal ingress: a dependency declared ONCE on the SOURCE must render the
# matching ingress-allow on the DESTINATION (R-NET-02). Without it, default-deny-
# ingress drops the source and the declared dependency does not actually work.
src_sd = {
"name": "src",
"source": {"image": "a:1"},
"dependencies": [{"to": "dst", "port": 8080, "protocol": "tcp"}],
}
dst_sd = {"name": "dst", "source": {"image": "b:1"}}
rev = build_reverse_map([src_sd, dst_sd])
check(
rev.get("dst") == [{"source": "src", "port": 8080, "protocol": "tcp"}],
"reverse map must record src->dst:8080 under the destination",
)
check("src" not in rev, "a service nothing depends on must have no inbound edges")
dst_docs = render(dst_sd, inbound=rev.get("dst", []))
rev_cnps = [
d[1]
for d in dst_docs
if d[1]["kind"] == "CiliumNetworkPolicy"
and d[1]["metadata"]["name"] == "dst-allow-src-8080"
]
check(len(rev_cnps) == 1, "destination must render one reciprocal ingress allow")
if rev_cnps:
ing = rev_cnps[0]["spec"]["ingress"]
from_ep = ing[0]["fromEndpoints"][0]["matchLabels"]
check(
from_ep.get("app.kubernetes.io/name") == "src"
and from_ep.get("k8s:io.kubernetes.pod.namespace") == "src",
"reciprocal ingress must be from the SOURCE's identity, namespace-pinned (cross-ns)",
)
allowed = {p["port"] for r in ing for tp in r["toPorts"] for p in tp["ports"]}
check(allowed == {"8080"}, "reciprocal ingress must open only the declared port")
# additive: default-deny ingress stays [] even with the reciprocal allow present.
dst_dd = next(d[1] for d in dst_docs if d[1]["kind"] == "NetworkPolicy")
check(
dst_dd["spec"]["ingress"] == [],
"reciprocal ingress must be ADDITIVE — default-deny ingress stays [] (R-NET-01)",
)
# a lone render (no inbound) must NOT gain any reciprocal ingress (golden-stable).
lone_docs = render(dst_sd)
check(
not any(
d[1]["kind"] == "CiliumNetworkPolicy"
and d[1]["metadata"]["name"].startswith("dst-allow-src")
for d in lone_docs
),
"a render with no inbound context must be unchanged (no reciprocal ingress)",
)
# 4b) exposure (ingressRoutes) -> an ADDITIVE gateway-ingress CiliumNetworkPolicy that
# allows ONLY the reserved `ingress` (Gateway proxy) identity to the exposed port,
# while default-deny still denies all other ingress (R-NET-01/02). Uses an explicit
# non-forward-auth mode so this exercises the direct app-port gateway path (a
# forward-auth app instead routes the Gateway to the :4180 sidecar — see 6c).
exp_docs = render(
{
"name": "z",
"source": {"image": "a:1"},
"ingressRoutes": [{"host": "z.home.keithjackson.dev", "port": 80}],
"auth": {"mode": "none"},
}
)
gw = [
d[1]
for d in exp_docs
if d[1]["kind"] == "CiliumNetworkPolicy"
and d[1]["metadata"]["name"].endswith("-allow-gateway-ingress")
]
check(len(gw) == 1, "an exposed service must render one gateway-ingress allow (R-NET-01)")
if gw:
ing = gw[0]["spec"]["ingress"]
check(
ing[0]["fromEntities"] == ["ingress"],
"gateway-ingress allow must be from the reserved `ingress` (Gateway) identity",
)
allowed_ports = {p["port"] for r in ing for tp in r["toPorts"] for p in tp["ports"]}
check(
allowed_ports == {"80"},
"gateway-ingress allow must be least-privilege (only the exposed port)",
)
# default-deny stays deny-all-ingress even when an exposed service adds the allow.
exp_dd = next(d[1] for d in exp_docs if d[1]["kind"] == "NetworkPolicy")
check(
exp_dd["spec"]["ingress"] == [],
"gateway-ingress allow must be ADDITIVE — default-deny ingress stays [] (R-NET-01)",
)
# an internal (no ingressRoutes) service must NOT get a gateway-ingress allow.
internal_docs = render({"name": "int", "source": {"image": "a:1"}})
check(
not any(
d[1]["kind"] == "CiliumNetworkPolicy"
and d[1]["metadata"]["name"].endswith("-allow-gateway-ingress")
for d in internal_docs
),
"a non-exposed service must not render a gateway-ingress allow (least privilege)",
)
# 4b-dns) exposure -> ExternalDNS SCOPING on the HTTPRoute (R-DNS-03). `internal` renders no
# public DNS record and must be INCAPABLE of rendering one: its route carries
# external-dns's own controller annotation set to something other than `dns-controller`,
# which v0.21.0's gateway-httproute source (source/gateway.go ->
# annotations.IsControllerMismatch) uses to skip the route before any hostname is
# resolved. `public` must NOT carry it, or the five public names would vanish instead.
# Both directions are asserted here because only asserting the internal half would pass
# just as happily if the renderer stamped EVERY route and silently unpublished the zone.
dns_int_docs = render(
{
"name": "di",
"source": {"image": "a:1"},
"exposure": "internal",
"ingressRoutes": [{"host": "di.home.keithjackson.dev", "port": 80}],
"auth": {"mode": "none"},
}
)
dns_int_routes = [d[1] for d in dns_int_docs if d[1]["kind"] == "HTTPRoute"]
check(
len(dns_int_routes) == 1,
"an internal exposed service must still render its HTTPRoute (LAN reachability is "
"unchanged — only the PUBLIC record is withdrawn)",
)
if dns_int_routes:
iann = dns_int_routes[0]["metadata"].get("annotations", {})
check(
iann.get(EXTERNALDNS_CONTROLLER_KEY) == EXTERNALDNS_CONTROLLER_DISOWN,
"an internal service's HTTPRoute must carry the ExternalDNS disown annotation",
)
check(
iann.get(EXTERNALDNS_CONTROLLER_KEY) != EXTERNALDNS_CONTROLLER_CLAIMED,
"the disown value must never be external-dns's own `dns-controller` (that CLAIMS "
"the route rather than releasing it — the one value that inverts the guarantee)",
)
dns_pub_docs = render(
{
"name": "dp",
"source": {"image": "a:1"},
"exposure": "public",
"ingressRoutes": [{"host": "dp.home.keithjackson.dev", "port": 80}],
"auth": {"mode": "native"},
}
)
dns_pub_routes = [d[1] for d in dns_pub_docs if d[1]["kind"] == "HTTPRoute"]
check(len(dns_pub_routes) == 1, "a public service must render its HTTPRoute")
if dns_pub_routes:
check(
EXTERNALDNS_CONTROLLER_KEY
not in dns_pub_routes[0]["metadata"].get("annotations", {}),
"a public service's HTTPRoute must NOT carry the ExternalDNS disown annotation",
)
# The DEFAULT posture is `internal` (renderer-contract "Default posture"), so a
# ServiceDefinition that simply omits `exposure` must also be unpublishable. Omitting the
# field is the likeliest way a new service arrives, and it is the case a per-service
# allow-list would miss.
dns_dflt_docs = render(
{
"name": "dd",
"source": {"image": "a:1"},
"ingressRoutes": [{"host": "dd.home.keithjackson.dev", "port": 80}],
"auth": {"mode": "none"},
}
)
dns_dflt_route = next(d[1] for d in dns_dflt_docs if d[1]["kind"] == "HTTPRoute")
check(
dns_dflt_route["metadata"]["annotations"].get(EXTERNALDNS_CONTROLLER_KEY)
== EXTERNALDNS_CONTROLLER_DISOWN,
"a ServiceDefinition that OMITS `exposure` defaults to internal and must also be "
"unpublishable (default posture, renderer-contract)",
)
# The scoping annotation must COEXIST with the auth hints rather than replace them: both are
# merged into one `annotations` map, and a forward-auth internal app is the overlap case.
dns_fa_docs = render(
{
"name": "df",
"source": {"image": "a:1"},
"exposure": "internal",
"ingressRoutes": [{"host": "df.home.keithjackson.dev", "port": 80}],
"auth": {"mode": "forward-auth", "groups": ["platform-admins"]},
}
)
dns_fa_ann = next(
d[1] for d in dns_fa_docs if d[1]["kind"] == "HTTPRoute"
)["metadata"]["annotations"]
check(
dns_fa_ann.get(EXTERNALDNS_CONTROLLER_KEY) == EXTERNALDNS_CONTROLLER_DISOWN
and dns_fa_ann.get(f"{LBL}/auth-mode") == "forward-auth",
"DNS scoping and auth-mode hints must COEXIST on one internal forward-auth route",
)
# 4c) inClusterClients -> one ADDITIVE, least-privilege ingress allow per entry that admits a
# PLATFORM controller's NAMESPACE (not the reserved `ingress` identity, not `all`) to ONE
# declared port. This is what keeps a cluster whose GitOps source is an in-cluster Service
# reconcilable: fromEntities[ingress] does not cover a controller dialling the ClusterIP.
icc_docs = render(
{
"name": "g",
"source": {"image": "a:1"},
"ingressRoutes": [{"host": "g.home.keithjackson.dev", "port": 3000}],
"auth": {"mode": "none"},
"inClusterClients": [
{"name": "flux", "namespace": "flux-system", "port": 3000}
],
}
)
icc = [
d[1]
for d in icc_docs
if d[1]["kind"] == "CiliumNetworkPolicy"
and d[1]["metadata"]["name"] == "g-allow-flux-ingress"
]
check(
len(icc) == 1,
"an inClusterClients entry must render one ingress CNP named "
"<service>-allow-<name>-ingress",
)
if icc:
p = icc[0]
check(
p["metadata"]["namespace"] == "g"
and p["spec"]["endpointSelector"]["matchLabels"]
== {"app.kubernetes.io/name": "g"},
"in-cluster client allow must live in the service's namespace and select the service",
)
ing = p["spec"]["ingress"]
check(
ing[0]["fromEndpoints"]
== [{"matchLabels": {"k8s:io.kubernetes.pod.namespace": "flux-system"}}],
"in-cluster client allow must match the client NAMESPACE only (reserved k8s: label)",
)
check(
"fromEntities" not in ing[0],
"in-cluster client allow must not widen to an entity (never fromEntities: all)",
)
icc_ports = {
(pt["port"], pt["protocol"])
for r in ing
for tp in r["toPorts"]
for pt in tp["ports"]
}
check(
icc_ports == {("3000", "TCP")},
"in-cluster client allow must be least-privilege (only the declared port/protocol)",
)
# ADDITIVE: default-deny ingress stays [] and the gateway allow is untouched.
icc_dd = next(d[1] for d in icc_docs if d[1]["kind"] == "NetworkPolicy")
check(
icc_dd["spec"]["ingress"] == [],
"in-cluster client allow must be ADDITIVE — default-deny ingress stays [] (R-NET-01)",
)
check(
any(
d[1]["metadata"].get("name") == "g-allow-gateway-ingress"
for d in icc_docs
if d[1]["kind"] == "CiliumNetworkPolicy"
),
"in-cluster client allow must not displace the north-south gateway allow",
)
# `name` is OPTIONAL and defaults to the namespace, so the common case is namespace + port.
icc_default = render(
{
"name": "g",
"source": {"image": "a:1"},
"inClusterClients": [{"namespace": "monitoring", "port": 9090}],
}
)
check(
any(
d[1]["metadata"].get("name") == "g-allow-monitoring-ingress"
for d in icc_default
if d[1]["kind"] == "CiliumNetworkPolicy"
),
"inClusterClients.name must default to the admitted namespace",
)
# A service that declares none must render NO such policy (grant-nothing-extra).
check(
not any(
d[1]["kind"] == "CiliumNetworkPolicy"
and d[1]["metadata"]["name"].endswith("-ingress")
and "fromEndpoints" in d[1]["spec"].get("ingress", [{}])[0]
for d in render({"name": "q", "source": {"image": "a:1"}})
),
"a service declaring no inClusterClients must render no in-cluster client allow",
)
# 5) vm kind -> VirtualMachine + standalone DataVolume.
vm_docs = render(
{
"name": "vmx",
"kind": "vm",
"source": {"vmTemplate": "quay.io/containerdisks/fedora:41"},
"storage": [{"name": "rootdisk", "size": "6Gi", "class": "longhorn"}],
}
)
vm_kinds = [d[1]["kind"] for d in vm_docs]
check("VirtualMachine" in vm_kinds, "vm: missing VirtualMachine")
check("DataVolume" in vm_kinds, "vm: missing CDI DataVolume")
# 6) env / envFrom / lifecycle -> emitted VERBATIM onto the app container (T-APPS).
envd_docs = render(
{
"name": "envd",
"source": {"image": "a:1"},
"env": [
{"name": "PLAIN", "value": "literal"},
{
"name": "SECRET",
"valueFrom": {"secretKeyRef": {"name": "sec", "key": "k"}},
},
],
"envFrom": [{"secretRef": {"name": "bulk"}}],
"lifecycle": {
"postStart": {"exec": {"command": ["/bin/sh", "-c", "echo hi"]}}
},
"podSecurityContext": {"fsGroup": 1001, "runAsUser": 1001},
}
)
envd_pod = next(
d[1] for d in envd_docs if d[1]["kind"] == "Deployment"
)["spec"]["template"]["spec"]
check(
envd_pod.get("securityContext") == {"fsGroup": 1001, "runAsUser": 1001},
"podSecurityContext must render verbatim onto the pod spec",
)
ec = envd_pod["containers"][0]
check(
ec.get("env") == [
{"name": "PLAIN", "value": "literal"},
{"name": "SECRET", "valueFrom": {"secretKeyRef": {"name": "sec", "key": "k"}}},
],
"env must render verbatim onto the container (literal + secretKeyRef)",
)
check(
ec.get("envFrom") == [{"secretRef": {"name": "bulk"}}],
"envFrom must render verbatim onto the container",
)
check(
ec.get("lifecycle")
== {"postStart": {"exec": {"command": ["/bin/sh", "-c", "echo hi"]}}},
"lifecycle must render verbatim onto the container",
)
# Absence => the keys are never set (existing apps stay byte-identical).
plain_c = next(
d[1] for d in render({"name": "plain", "source": {"image": "a:1"}})
if d[1]["kind"] == "Deployment"
)["spec"]["template"]["spec"]["containers"][0]
check(
"env" not in plain_c
and "envFrom" not in plain_c
and "lifecycle" not in plain_c,
"an SD without env/envFrom/lifecycle must not gain those container keys",
)
plain_pod = next(
d[1] for d in render({"name": "plain2", "source": {"image": "a:1"}})
if d[1]["kind"] == "Deployment"
)["spec"]["template"]["spec"]
check(
"securityContext" not in plain_pod,
"an SD without podSecurityContext must not gain a pod securityContext",
)
# 6b) hardware.gpu: intel-quicksync -> a `gpu.intel.com/i915` LIMIT on the app container
# and NOTHING ELSE: no volume, no volumeMount, no securityContext, no nodeSelector.
# The SD declares the CAPABILITY; these assertions pin the MECHANISM that currently
# satisfies it -- including what it deliberately does NOT emit -- so a change to it
# is a deliberate, visible edit. The previous mechanism (a /dev/dri hostPath) is
# asserted ABSENT: it was measured not to work (EPERM from the device cgroup; see
# the emission site), and the absence assertion is what stops it coming back.
hw_docs = render(
{
"name": "hw",
"source": {"image": "a:1"},
"hardware": {"gpu": "intel-quicksync"},
}
)
hw_pod = next(
d[1] for d in hw_docs if d[1]["kind"] == "Deployment"
)["spec"]["template"]["spec"]
hw_app = hw_pod["containers"][0]
# The whole mechanism, exactly: an extended resource in `limits`. `requests` is NOT set
# by the renderer -- Kubernetes mirrors it from `limits`, and a requests-only extended
# resource is invalid. Asserting the exact dict is deliberate: it fails if anyone adds a
# CPU/memory resources feature through this field (that would be scope creep, and this
# field is about the DEVICE only).
check(
hw_app.get("resources") == {"limits": {"gpu.intel.com/i915": 1}},
"hardware.gpu=intel-quicksync must render resources.limits "
"{gpu.intel.com/i915: 1} on the app container, and nothing else in resources",
)
# The retired mechanism must stay retired. A hostPath binds the device node in but adds
# no device-cgroup allow rule, so the open() was EPERM with the node plainly visible.
check(
not any(
"hostPath" in v or v.get("name") == "dri" for v in hw_pod.get("volumes", [])
)
and not any(
m.get("mountPath") == "/dev/dri" or m.get("name") == "dri"
for m in hw_app.get("volumeMounts", [])
),
"hardware.gpu=intel-quicksync must render NO /dev/dri hostPath volume and no "
"/dev/dri mount (the hostPath mechanism was measured not to work: device-cgroup "
"EPERM; the device plugin's DeviceSpec Permissions: rw is what grants access)",
)
# The resource is the WHOLE feature: no pod securityContext is invented. The image's own
# cont-init hook adds the app user to the (plugin-injected) device's group inside the
# container, and s6-setuidgid discards any Kubernetes-supplied supplementalGroups anyway
# (see the emission site). Asserting the absence is what stops someone adding an inert
# one back.
check(
"securityContext" not in hw_pod,
"hardware.gpu must NOT invent a pod securityContext (supplementalGroups is inert "
"under s6-setuidgid; the image's cont-init hook handles the device group)",
)
# Nothing privileged, and no node pinning: Kyverno restrict-host-namespaces (Enforce)
# rejects `privileged: true` in an app namespace, and placement is a separate concern
# with its own justification -- this field is about the DEVICE only. (The extended
# resource now constrains placement to nodes the plugin advertises, which is the
# scheduler's own doing, not a nodeSelector this renderer writes.)
check(
"privileged" not in hw_pod.get("securityContext", {})
and "securityContext" not in hw_app
and "nodeSelector" not in hw_pod,
"hardware.gpu must emit no privileged securityContext and no nodeSelector",
)
# The regression that matters: absence renders NOTHING. Mirrors the podSecurityContext
# negative above -- an SD that does not opt in must be byte-identical to before.
nohw_pod = next(
d[1] for d in render({"name": "nohw", "source": {"image": "a:1"}})
if d[1]["kind"] == "Deployment"
)["spec"]["template"]["spec"]
check(
"volumes" not in nohw_pod
and "volumeMounts" not in nohw_pod["containers"][0]
and "resources" not in nohw_pod["containers"][0]
and "securityContext" not in nohw_pod,
"an SD without `hardware` must gain no resources block, no volume, no mount and no "
"securityContext",
)
# An SD that declares BOTH keeps its own podSecurityContext exactly as written -- the
# hardware field neither adds to it nor rewrites it.
hwm_pod = next(
d[1]
for d in render(
{
"name": "hwm",
"source": {"image": "a:1"},
"hardware": {"gpu": "intel-quicksync"},
"podSecurityContext": {"fsGroup": 1001, "supplementalGroups": [44]},
}
)
if d[1]["kind"] == "Deployment"
)["spec"]["template"]["spec"]
check(
hwm_pod["securityContext"] == {"fsGroup": 1001, "supplementalGroups": [44]}
and hwm_pod["containers"][0].get("resources")
== {"limits": {"gpu.intel.com/i915": 1}},
"hardware.gpu must leave the SD's own podSecurityContext untouched",
)
# The resource goes on the APP container only. A forward-auth app renders an oauth2-proxy
# sidecar, which does no transcoding and must not consume one of the node's i915 slots.
hws_containers = next(
d[1]
for d in render(
{
"name": "hws",
"source": {"image": "a:1"},
"hardware": {"gpu": "intel-quicksync"},
"ingressRoutes": [{"host": "hws.home.keithjackson.dev", "port": 8080}],
"auth": {"mode": "forward-auth", "groups": ["media-users"]},
}
)
if d[1]["kind"] == "Deployment"
)["spec"]["template"]["spec"]["containers"]
check(
len(hws_containers) == 2
and hws_containers[0].get("resources")
== {"limits": {"gpu.intel.com/i915": 1}}
and "gpu.intel.com/i915"
not in hws_containers[1].get("resources", {}).get("limits", {}),
"hardware.gpu must put the i915 limit on the APP container only, never on the "
"oauth2-proxy sidecar",
)
# 6c) forward-auth (Option B): an exposed forward-auth app gets an oauth2-proxy sidecar that
# fronts the Gateway path (fail-closed), while oidc/native/none apps are unchanged.
fa_docs = render(
{
"name": "fa",
"source": {"image": "a:1"},
"ingressRoutes": [{"host": "fa.home.keithjackson.dev", "port": 9117}],
"auth": {"mode": "forward-auth", "groups": ["media-users"]},
}
)
fa_dep = next((d[1] for d in fa_docs if d[1]["kind"] == "Deployment"), None)
check(fa_dep is not None,
"a forward-auth app rendered NO Deployment, so there is no pod to gate at all")
fa_containers = (((fa_dep or {}).get("spec") or {}).get("template") or {}) \
.get("spec", {}).get("containers") or []
check(
len(fa_containers) == 2
and fa_containers[0]["name"] == "fa"
and fa_containers[1]["name"] == "oauth2-proxy",
f"forward-auth app must render the app container FIRST + an oauth2-proxy sidecar SECOND; "
f"rendered {[c.get('name') for c in fa_containers]!r}",
)
# EVERY assertion below reads `side`, so an absent sidecar must NOT be allowed to raise here.
# The whole point of the None-tolerant lookups in this block is that a missing sidecar (or a
# missing flag on it) reports as NAMED ASSERTIONS rather than as a traceback: an exception
# escaping _selftest() exits non-zero with the right verdict and ZERO diagnostics, and it
# hides every other assertion the same mutation broke. Measured by Desmond Oyelaran while
# wiring this selftest into the render-golden stage: deleting the --cookie-name emission gave
# rc 1 via an uncaught StopIteration and printed not one `SELFTEST FAIL:` line. Provoked
# again here with the sidecar removed entirely, which raised IndexError at this line.
side = fa_containers[1] if len(fa_containers) > 1 else {"args": [], "env": [], "ports": []}
check(
"--upstream=http://127.0.0.1:9117" in side["args"],
"sidecar upstream must be the co-located app UI over http on 127.0.0.1:<uiport>",
)
check(
"--allowed-group=media-users" in side["args"],
"sidecar must enforce the declared auth.group (R-IDP-06)",
)
check(
"--redirect-url=https://fa.home.keithjackson.dev/oauth2/callback" in side["args"],
"sidecar redirect-url must be the app host's /oauth2/callback",
)
check(
not any(a.startswith("--ssl-upstream-insecure-skip-verify") for a in side["args"]),
"http upstream must NOT skip TLS verify (that flag is only for crafty's self-signed https)",
)
# COOKIE ISOLATION. Every oauth2-proxy instance holds its own cookie-secret, so two instances
# sharing a cookie NAME on a shared DOMAIN evict each other's sessions silently. Both flags
# must be DERIVED from this service, not defaulted and not listed.
check(
"--cookie-name=_oauth2_proxy_fa" in side["args"],
"sidecar must declare a cookie name DERIVED from the service name — the oauth2-proxy "
"default (_oauth2_proxy) is shared with every other instance on the zone",
)
check(
"--cookie-domain=fa.home.keithjackson.dev" in side["args"],
"sidecar cookie-domain must be HOST-scoped (the app's own host), not zone-wide",
)
check(
f"--whitelist-domain=.{ZONE}" in side["args"],
"whitelist-domain must stay ZONE-wide — it governs post-login redirect targets, not "
"cookie scope; narrowing it breaks the sign-in bounce",
)
# SESSION LIFETIME. `--cookie-refresh` must be present AND strictly shorter than the
# measured 300s access-token lifetime, or the session dies mid-use. Asserted as a parsed
# duration rather than a string match, so "2m30s" -> "150s" -> "2.5m" stays green and a
# silent widening past the token lifetime does not.
refresh = next((a.split("=", 1)[1] for a in side["args"]
if a.startswith("--cookie-refresh=")), None)
check(refresh is not None,
"sidecar must set --cookie-refresh; the v7.8.1 default is 0 = DISABLED, so the "
"session dies when the access token expires and the user is bounced to Keycloak")
if refresh is not None:
_m = re.fullmatch(r"(?:(\d+)m)?(?:(\d+(?:\.\d+)?)s)?", refresh)
_secs = (int(_m.group(1) or 0) * 60 + float(_m.group(2) or 0)) if _m else -1
check(0 < _secs < OAUTH2_ACCESS_TOKEN_LIFETIME_S,
f"--cookie-refresh={refresh} parses to {_secs}s, which is not inside "
f"(0, {OAUTH2_ACCESS_TOKEN_LIFETIME_S}s). A refresh at or past the access-token "
f"lifetime never renews in time, which is the same logout it was added to stop")
check(
not any(a.startswith("--session-cookie-minimal") for a in side["args"]),
"--session-cookie-minimal is MUTUALLY EXCLUSIVE with --cookie-refresh "
"('cookie_refresh > 0 requires oauth tokens in sessions') — the proxy refuses to start",
)
# METRICS. The listener does not exist unless the flag is given, and the port must be
# DECLARED or nothing can ever select it (neither a Service port nor a PodMonitor).
check(
f"--metrics-address=0.0.0.0:{OAUTH2_PROXY_METRICS_PORT}" in side["args"],
"sidecar must start the /metrics listener; without --metrics-address it is not merely "
"unscraped, it is not serving at all",
)
check(
[p["name"] for p in side["ports"]] == ["http", "metrics"]
and {p["containerPort"] for p in side["ports"]}
== {OAUTH2_PROXY_PORT, OAUTH2_PROXY_METRICS_PORT},
"sidecar must declare BOTH the http and metrics container ports — an undeclared port "
"cannot be named by a Service port or a PodMonitor, so the flag alone is inert",
)
# API ROUTES (schema 1.9.0). Three properties, and the FIRST is the one that keeps the field
# honest: an app that declares nothing must render NO --api-route. `fa` above declares none.
check(
not any(a.startswith("--api-route") for a in side["args"]),
"a forward-auth app that declares no auth.apiRoutes must render NO --api-route. A "
"default or a renderer constant would answer PiHole's page loads with 401 — PiHole "
"serves no /api/ at all — so the service would look permanently broken to a "
"logged-out operator instead of signing them in",
)
# Same `next(..., None)` discipline as _cookie_name() below, for the same reason: if the
# sidecar stops being rendered at all, this must report as a named assertion rather than as a
# StopIteration that swallows every other SELFTEST FAIL line in the run.
_faapi_dep = next(
(
d[1]
for d in render(
{
"name": "faapi",
"source": {"image": "a:1"},
"ingressRoutes": [{"host": "faapi.home.keithjackson.dev", "port": 9119}],
"auth": {
"mode": "forward-auth",
"groups": ["media-users"],
"apiRoutes": ["^/api/", "^/signalr/"],
},
}
)
if d[1]["kind"] == "Deployment"
),
None,
)
fa_api_side = next(
(
c
for c in ((_faapi_dep or {}).get("spec", {}).get("template", {})
.get("spec", {}).get("containers") or [])
if c.get("name") == "oauth2-proxy"
),
None,
)
check(
fa_api_side is not None,
"a forward-auth app declaring auth.apiRoutes rendered NO oauth2-proxy sidecar, so the "
"routes have nowhere to land and the app is not gated at all (R-IDP-02)",
)
check(
fa_api_side is not None
and [a for a in fa_api_side["args"] if a.startswith("--api-route")]
== ["--api-route=^/api/", "--api-route=^/signalr/"],
"a declared auth.apiRoutes must render as one repeatable --api-route per entry, in "
"declared order, and NOTHING else. Note the flag is SINGULAR: --api-routes does not "
"exist in v7.8.1 and the proxy refuses to start with it",
)
# BREADTH. The renderer must REFUSE a regex that would match a page load, not merely
# discourage one, because the schema stage can be skipped and render-golden cannot.
for _bad in ("^/", "^/.*", ".*", "/api/", "^/api/|^/"):
try:
render(
{
"name": "fabad",
"source": {"image": "a:1"},
"ingressRoutes": [{"host": "fabad.home.keithjackson.dev", "port": 9120}],
"auth": {
"mode": "forward-auth",
"groups": ["media-users"],
"apiRoutes": [_bad],
},
}
)
_refused = False
except SystemExit:
_refused = True
check(
_refused,
f"the renderer ACCEPTED auth.apiRoutes entry {_bad!r}. A too-broad --api-route "
f"answers a plain page load with 401, so a logged-out operator sees a bare error "
f"page and is never offered a sign-in — the service looks permanently broken "
f"rather than merely unauthenticated, which is a worse failure than the 302 bug "
f"this flag exists to fix",
)
for _good in ("^/api/", "^/signalr/", "^/api/v3/.*", "^/api/v2.0/server/update$"):
check(
bool(OAUTH2_API_ROUTE_PATTERN.match(_good)),
f"the breadth ratchet rejected the legitimate route {_good!r}. A ratchet that "
f"blocks real API prefixes is a ratchet someone eventually deletes",
)
# Two forward-auth apps must not be able to collide. Renders a SECOND one and compares, so a
# future refactor to a constant (or to a per-service table with a copy-paste slip) fails here
# rather than silently in a browser six weeks later.
# Same None-tolerant shape as the `faapi` fixture above and for the same reason: a missing
# sidecar must surface as a named assertion, never as a traceback that suppresses the
# collected SELFTEST FAIL lines.
_fa2_dep = next(
(
d[1]
for d in render(
{
"name": "fa2",
"source": {"image": "a:1"},
"ingressRoutes": [{"host": "fa2.home.keithjackson.dev", "port": 9118}],
"auth": {"mode": "forward-auth", "groups": ["media-users"]},
}
)
if d[1]["kind"] == "Deployment"
),
None,
)
fa2_side = next(
(
c
for c in ((_fa2_dep or {}).get("spec", {}).get("template", {})
.get("spec", {}).get("containers") or [])
if c.get("name") == "oauth2-proxy"
),
{"args": []},
)
# `next(..., None)` — the DEFAULT IS LOAD-BEARING, not defensive style. With a bare
# `next(genexpr)` this raises StopIteration the moment a sidecar renders no --cookie-name,
# which is precisely the mutation this assertion exists to catch. The traceback then escapes
# _selftest() before its collected `SELFTEST FAIL:` lines are printed, so the run exits
# non-zero with the RIGHT VERDICT AND NO DIAGNOSTICS — and it masks every other assertion the
# same mutation broke. Measured by Desmond Oyelaran while wiring the selftest into the
# render-golden stage: deleting the --cookie-name emission gave rc 1 via an uncaught
# StopIteration and suppressed all of the messages. A missing flag must report AS AN
# ASSERTION, with its name in the output, which is what the None + explicit check below do.
def _cookie_name(container):
return next((a for a in container["args"] if a.startswith("--cookie-name=")), None)
_fa_cn, _fa2_cn = _cookie_name(side), _cookie_name(fa2_side)
check(
_fa_cn is not None and _fa2_cn is not None,
f"a forward-auth sidecar rendered NO --cookie-name at all (fa={_fa_cn!r}, "
f"fa2={_fa2_cn!r}). Omitting the flag is not neutral: oauth2-proxy then claims its "
f"built-in default `_oauth2_proxy`, which every other instance on the platform also "
f"claims, so they overwrite and ClearSession-delete each other's sessions",
)
check(
_fa_cn != _fa2_cn,
"two forward-auth apps rendered IDENTICAL cookie names — they would overwrite each "
"other's session on every sign-in (the 2026-09-29 crafty defect, fleet-wide)",
)
# Index-safe for the same reason as `side` above: an absent sidecar has no env, and an
# IndexError here would exit non-zero with no diagnostics at all.
_fa_env = (side.get("env") or [{}])[0]
check(
(((_fa_env.get("valueFrom") or {}).get("secretKeyRef") or {}).get("name")
== "fa-oauth2-proxy"),
"sidecar creds must come by name from the per-app <app>-oauth2-proxy Secret (R-IDP-04)",
)
fa_svc = next((d[1] for d in fa_docs if d[1]["kind"] == "Service"), None)
check(fa_svc is not None, "a forward-auth app must render a Service for the Gateway to reach")
fa_svc_ports = {p["port"] for p in (((fa_svc or {}).get("spec") or {}).get("ports") or [])}
check(
fa_svc_ports == {9117, 4180},
"forward-auth Service must expose the app port (east-west) AND :4180 (gateway->sidecar)",
)
fa_route = next(d[1] for d in fa_docs if d[1]["kind"] == "HTTPRoute")
check(
fa_route["spec"]["rules"][0]["backendRefs"][0]["port"] == 4180,
"forward-auth HTTPRoute backendRef must target the sidecar :4180, not the app UI port",
)
fa_gw = next(
d[1] for d in fa_docs
if d[1]["kind"] == "CiliumNetworkPolicy"
and d[1]["metadata"]["name"] == "fa-allow-gateway-ingress"
)
fa_gw_ports = {
p["port"] for r in fa_gw["spec"]["ingress"] for tp in r["toPorts"] for p in tp["ports"]
}
check(
fa_gw_ports == {"4180"},
"fail-closed: the Gateway may reach ONLY the sidecar :4180, never the un-gated app port",
)
# an oidc app (native client) must NOT gain a sidecar or a :4180 path.
oidc_docs = render(
{
"name": "oi",
"source": {"image": "a:1"},
"ingressRoutes": [{"host": "oi.home.keithjackson.dev", "port": 3000}],
"auth": {"mode": "oidc", "groups": ["platform-admins"]},
}
)
oi_containers = next(
d[1] for d in oidc_docs if d[1]["kind"] == "Deployment"
)["spec"]["template"]["spec"]["containers"]
check(
len(oi_containers) == 1
and not any("oauth2-proxy" in c["name"] for c in oi_containers),
"an oidc app must NOT gain an oauth2-proxy sidecar (native client authenticates itself)",
)
oi_route = next(d[1] for d in oidc_docs if d[1]["kind"] == "HTTPRoute")
check(
oi_route["spec"]["rules"][0]["backendRefs"][0]["port"] == 3000,
"an oidc app's HTTPRoute must still target its own app port (unchanged)",
)
# 7) wave partitioning: longest-path over the dependency DAG, CNPG-DB strictly below.
wsds = [
{"name": "a", "source": {"image": "x:1"}}, # leaf
{"name": "b", "source": {"image": "x:1"}}, # leaf
{"name": "c", "source": {"image": "x:1"},
"dependencies": [{"to": "a", "port": 1}, {"to": "b", "port": 1}]}, # wave 1
{"name": "d", "source": {"image": "x:1"},
"dependencies": [{"to": "c", "port": 1}]}, # wave 2
{"name": "svc", "source": {"image": "x:1"},
"dependencies": [{"to": "svc-db", "port": 5432}]}, # DB consumer
]
w = compute_waves(wsds)
check(w["a"] == 0 and w["b"] == 0, "leaf services must be wave 0")
check(w["c"] == 1, "a service depending on wave-0 nodes must be wave 1")
check(w["d"] == 2, "longest-path depth must be transitive (d -> c -> {a,b})")
check(
w["svc-db"] == 0 and w["svc"] == 1 and w["svc-db"] < w["svc"],
"a co-located <svc>-db must be a leaf STRICTLY BELOW its consumer (CNPG-DB rule)",
)
# cycle detection.
try:
compute_waves([
{"name": "p", "source": {"image": "x:1"}, "dependencies": [{"to": "q", "port": 1}]},
{"name": "q", "source": {"image": "x:1"}, "dependencies": [{"to": "p", "port": 1}]},
])
check(False, "compute_waves must raise on a dependency cycle")
except ValueError:
pass
# 7a) plan_apps_waves: sidecars inherit their unit's wave; not-onboarded SDs are skipped;
# EXTRA_UNITS anchor strictly above; a file absent from the tree is never placed.
present = {
"a.yaml", "a-claim-secret.sops.yaml",
"svc.yaml", "svc-bootstrap-secret.sops.yaml",
"svc-db.yaml", "svc-db-app.sops.yaml", "svc-db-backup-credentials.sops.yaml",
"svc-db-backup-credentials-prod.sops.yaml",
"gitea.yaml",
"gitea-actions.yaml", "gitea-actions-runner-secret.sops.yaml",
}
plan_sds = wsds + [{"name": "gitea", "source": {"image": "x:1"}},
{"name": "notonboarded", "source": {"image": "x:1"}}]
plan = plan_apps_waves(plan_sds, present)
flat = {f: wv for wv, fs in plan.items() for f in fs}
check(flat.get("a.yaml") == 0, "leaf app must land in wave 0")
check(flat.get("a-claim-secret.sops.yaml") == 0,
"a `<svc>-claim-secret.sops.yaml` (plex's plex.tv claim) must be placed in the SAME wave "
"as its app — an env secretKeyRef has no `optional`, so a later wave means "
"CreateContainerConfigError")
check(flat.get("svc-db.yaml") == 0 and flat.get("svc-db-app.sops.yaml") == 0
and flat.get("svc-db-backup-credentials.sops.yaml") == 0
and flat.get("svc-db-backup-credentials-prod.sops.yaml") == 0,
"CNPG DB manifest + its owner secret + BOTH halves of its per-env Barman backup secret "
"must be in the DB's (lower) wave")
check(flat.get("svc.yaml") == 1 and flat.get("svc-bootstrap-secret.sops.yaml") == 1,
"a DB-backed app + its bootstrap secret must be in the consumer wave, above the DB")
check(flat.get("gitea.yaml") == 0, "gitea (leaf) must be wave 0")
check(flat.get("gitea-actions.yaml") == 1
and flat.get("gitea-actions-runner-secret.sops.yaml") == 1,
"gitea-actions (EXTRA_UNITS, above=True) must be one wave above its anchor gitea")
check("notonboarded.yaml" not in flat,
"a catalog SD with no committed manifest must NOT be placed (not onboarded)")
check("b.yaml" not in flat, "a file absent from the tree must never be placed")
# 7b) generated Flux Kustomization CR wiring (dependsOn chain + prune/wait/path).
import yaml as _yaml # already imported at module top; local alias for clarity
fw0 = _yaml.safe_load(flux_wave_kustomization_text(0))
fw2 = _yaml.safe_load(flux_wave_kustomization_text(2))
check(fw0["metadata"]["name"] == "apps-wave-0"
and fw0["spec"]["dependsOn"] == [{"name": "platform-config"}]
and fw0["spec"]["path"] == "./apps/wave-0",
"wave 0 Flux Kustomization must dependsOn platform-config and point at ./apps/wave-0")
check(fw2["spec"]["dependsOn"] == [{"name": "apps-wave-1"}]
and fw2["spec"]["prune"] is True and fw2["spec"]["wait"] is True,
"wave N Flux Kustomization must dependsOn wave N-1 with prune+wait (health-gated)")
kt = _yaml.safe_load(wave_kustomization_text(1, ["svc.yaml", "sonarr.yaml"]))
check(kt["kind"] == "Kustomization" and kt["resources"] == ["svc.yaml", "sonarr.yaml"],
"wave kustomization must list exactly the wave's resources")
# 8) substitution safety (R-ENV-02): the renderer must REFUSE to emit a manifest whose
# shell script carries an unescaped `${...}`. Every rendered manifest lands under a
# Flux path with postBuild.substituteFrom cluster-vars, so envsubst rewrites it before
# the container runs — this is the gitea/nextcloud OIDC-guard bug (`[ -z "${X:-}" ]`
# -> `[ -z "" ]`, always true, whole registration block skipped on every boot).
known = {"BASE_DOMAIN", "GATEWAY_VIP"}
# allowed: a bare, DECLARED cluster-vars knob is a deliberate R-ENV-02 substitution.
check(
substitution_violations("host: app.${BASE_DOMAIN}", known) == [],
"a bare declared cluster-vars key must be allowed (that is what substitution is FOR)",
)
# allowed: escaped shell braces, and the forms envsubst leaves alone.
check(
substitution_violations('if [ -z "$${X:-}" ]; then', known) == [],
"an ESCAPED `$${...}` must be allowed (envsubst emits it back as a literal `${...}`)",
)
check(
substitution_violations('t="$(mktemp)"; echo "$HOME $1"', known) == [],
"bare `$VAR` / `$(cmd)` are left alone by envsubst and must NOT be flagged",
)
# `$$` is a literal `$` to envsubst, so an ODD run of dollars still leaves the brace live.
check(
len(substitution_violations("$$${X:-}", known)) == 1,
"an ODD run of `$` still substitutes ($$ -> literal $, then `${X:-}` is eaten)",
)
# flagged: the exact regression — a default-operator form STRICT mode deliberately exempts.
guard_bug = substitution_violations('if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then', known)
check(
len(guard_bug) == 1 and guard_bug[0][1] == "${GITEA_OIDC_CLIENT_SECRET:-}",
"an unescaped `${VAR:-}` shell guard must be flagged (strict mode exempts this form)",
)
# flagged: an undeclared bare knob (envsubst blanks it), and an indexed form (hard error).
check(
len(substitution_violations("value: ${NOT_A_KNOB}", known)) == 1,
"an undeclared bare `${VAR}` must be flagged (envsubst blanks it silently)",
)
idx = substitution_violations("echo ${PIPESTATUS[0]}", known)
check(
len(idx) == 1 and "missing closing brace" in idx[0][2],
"an indexed `${A[0]}` must be flagged (envsubst hard-fails the whole Kustomization)",
)
# the checker must be wired into emit(): rendering an SD whose lifecycle hook carries the
# bug RAISES rather than writing a manifest whose guard is dead on arrival.
bad_sd = {
"name": "subbad",
"source": {"image": "a:1"},
"lifecycle": {
"postStart": {
"exec": {
"command": [
"/bin/sh",
"-c",
'if [ -z "${SUBBAD_SECRET:-}" ]; then exit 0; fi',
]
}
}
},
}
try:
emit(render(bad_sd), origin="<selftest>")
check(False, "emit must REFUSE a manifest with an unescaped `${...}` shell expansion")
except ValueError as exc:
check(
"SUBBAD_SECRET" in str(exc) and "$${" in str(exc),
"the refusal must name the offending token AND state the `$${...}` fix",
)
# the ESCAPED form of the same hook renders normally (the fix must be emittable).
good_sd = dict(bad_sd, name="subgood")
good_sd["lifecycle"] = {
"postStart": {
"exec": {
"command": [
"/bin/sh",
"-c",
'if [ -z "$${SUBBAD_SECRET:-}" ]; then exit 0; fi',
]
}
}
}
check(
"$${SUBBAD_SECRET:-}" in emit(render(good_sd), origin="<selftest>"),
"the escaped `$${...}` form must render (and reach the shell as `${...}`)",
)
# 9) observability (R-OBS-01): `metrics: true` alone renders NO ServiceMonitor, and the `up`
# SLI reads a signal that exists. This is the regression test for the defect that made all
# twelve `<app>-down` rules structurally inert on prod (see r_servicemonitor).
obs_sd = {
"name": "obsdemo",
"source": {"image": "a:1"},
"ingressRoutes": [{"host": "obsdemo.${BASE_DOMAIN}", "port": 8080}],
"auth": {"mode": "none"},
"observability": {"metrics": True, "slis": ["up"]},
}
obs_docs = render(obs_sd)
check(
not [d for d in obs_docs if d[1]["kind"] == "ServiceMonitor"],
"metrics: true with NO metricsEndpoint must render no ServiceMonitor — a monitor naming a "
"port the Service does not define discovers zero targets and makes the SLI inert",
)
pr = next(d[1] for d in obs_docs if d[1]["kind"] == "PrometheusRule")
up_rule = pr["spec"]["groups"][0]["rules"][0]
check(up_rule["alert"] == "obsdemo-down", "slis: [up] must render <name>-down")
check(
"up{job=" not in up_rule["expr"],
"the up SLI must NOT read up{job=...}: no repo-rendered scrape target produces that job, "
"so the rule can never fire (verified inert on prod for all twelve apps)",
)
check(
up_rule["expr"]
== 'kube_deployment_status_replicas_available{namespace="obsdemo", deployment="obsdemo"} '
'== 0 or absent(kube_deployment_status_replicas_available{namespace="obsdemo", '
'deployment="obsdemo"})',
"container up SLI must read kube_deployment_status_replicas_available with an absent() "
f"half; got: {up_rule['expr']}",
)
# 9a) livenessWorkload override: the object that RUNS is not always the one this SD renders.
sts_docs = render(
dict(
obs_sd,
name="obssts",
observability={
"metrics": True,
"slis": ["up"],
"livenessWorkload": {"kind": "StatefulSet", "name": "obssts"},
},
)
)
sts_expr = next(
d[1] for d in sts_docs if d[1]["kind"] == "PrometheusRule"
)["spec"]["groups"][0]["rules"][0]["expr"]
check(
"kube_statefulset_status_replicas_ready" in sts_expr,
f"livenessWorkload kind StatefulSet must read the statefulset series; got: {sts_expr}",
)
# 9b) vm kind: a VMI has no replica count, so liveness is a running-phase series EXISTING.
vm_pr = next(
d[1]
for d in render(
{
"name": "obsvm",
"kind": "vm",
"source": {"vmTemplate": "quay.io/x/y:1"},
"storage": [{"name": "root", "size": "6Gi"}],
"observability": {"metrics": True, "slis": ["up"]},
}
)
if d[1]["kind"] == "PrometheusRule"
)
check(
vm_pr["spec"]["groups"][0]["rules"][0]["expr"]
== 'absent(kubevirt_vmi_info{namespace="obsvm", name="obsvm", phase="running"})',
"vm up SLI must read kubevirt_vmi_info with the LOWERCASE running phase: "
f"{vm_pr['spec']['groups'][0]['rules'][0]['expr']}",
)
# 9c) metricsEndpoint renders ALL THREE halves of a working scrape, and they AGREE. Any one of
# them missing is an independent way for the scrape to silently discover/deliver nothing.
ep_docs = render(
dict(
obs_sd,
name="obsep",
observability={
"metrics": True,
"slis": ["up"],
"metricsEndpoint": {"port": 9100},
},
)
)
ep_svc = next(d[1] for d in ep_docs if d[1]["kind"] == "Service")
ep_sm = next(d[1] for d in ep_docs if d[1]["kind"] == "ServiceMonitor")
svc_port_names = {p["name"] for p in ep_svc["spec"]["ports"]}
ep_ports = {e["port"] for e in ep_sm["spec"]["endpoints"]}
check(
ep_ports <= svc_port_names,
f"ServiceMonitor endpoint port(s) {sorted(ep_ports)} must all exist on the Service "
f"{sorted(svc_port_names)} — this exact disagreement is what made twelve monitors inert",
)
check(
{"name": "metrics", "port": 9100, "targetPort": 9100} in ep_svc["spec"]["ports"],
"metricsEndpoint must render the named Service port it declares",
)
mon_cnp = [
d[1]
for d in ep_docs
if d[1]["kind"] == "CiliumNetworkPolicy"
and d[1]["metadata"]["name"] == "obsep-allow-monitoring-scrape"
]
check(
len(mon_cnp) == 1,
"a declared metricsEndpoint must render the monitoring-namespace ingress allow — the "
"namespace default-deny has `ingress: []` and would DROP the scrape without it",
)
if mon_cnp:
ing = mon_cnp[0]["spec"]["ingress"][0]
check(
ing["fromEndpoints"][0]["matchLabels"]["k8s:io.kubernetes.pod.namespace"]
== MONITORING_NS
and ing["toPorts"][0]["ports"] == [{"port": "9100", "protocol": "TCP"}],
"the scrape allow must admit ONLY the monitoring namespace to ONLY the metrics port",
)
gw_cnp = next(
d[1]
for d in ep_docs
if d[1]["kind"] == "CiliumNetworkPolicy"
and d[1]["metadata"]["name"] == "obsep-allow-gateway-ingress"
)
gw_ports = {p["port"] for p in gw_cnp["spec"]["ingress"][0]["toPorts"][0]["ports"]}
check(
"9100" not in gw_ports,
"the metrics port must NOT be opened to the Gateway (north-south); it is for the "
f"monitoring namespace only. Gateway allows: {sorted(gw_ports)}",
)
# A metricsEndpoint the renderer cannot express must be REFUSED, never rendered inert.
try:
render(
dict(
obs_sd,
name="obsbad",
observability={
"metrics": True,
"slis": ["up"],
"livenessWorkload": {"kind": "CronJob", "name": "obsbad"},
},
)
)
check(False, "an unknown livenessWorkload.kind must raise, not render a bogus expression")
except ValueError as exc:
check("CronJob" in str(exc), "the refusal must name the unsupported kind")
if failures:
for f in failures:
sys.stderr.write(f"SELFTEST FAIL: {f}\n")
return 1
sys.stderr.write("render.py selftest: PASS\n")
return 0
def main(argv=None) -> int:
ap = argparse.ArgumentParser(description="Render a ServiceDefinition to manifests.")
ap.add_argument("input", nargs="?", help="ServiceDefinition YAML file")
ap.add_argument("--all", metavar="DIR", help="Render every *.yaml ServiceDefinition in DIR")
ap.add_argument("-o", "--out-dir", help="With --all: write <name>.yaml files into this dir")
ap.add_argument("--selftest", action="store_true", help="Run the built-in self-test")
ap.add_argument("--no-image-automation", action="store_true",
help="Omit Flux ImageRepository/ImagePolicy CRs (for onboarding before the "
"image-reflector/-automation controllers are installed)")
ap.add_argument("--wave-layout", action="store_true",
help="With -o: place each rendered file under <out>/wave-<N>/<name>.yaml, "
"N = its dependency-ordered wave (compute_waves over the catalog)")
ap.add_argument("--emit-waves", action="store_true",
help="Generate the per-wave kustomize + Flux Kustomization CRs from the "
"catalog DAG and the committed apps tree (no manifest is rewritten)")
ap.add_argument("--print-waves", action="store_true",
help="Print the computed wave table (node -> wave) for the catalog and exit")
ap.add_argument("--catalog", default="catalog",
help="Catalog dir for --emit-waves/--print-waves/--wave-layout (default: catalog)")
ap.add_argument("--apps-dir", default="apps",
help="App-tier dir for --emit-waves (default: apps)")
ap.add_argument("--clusters-dir", default="clusters/base",
help="Dir to write clusters/base/apps-wave-<N>.yaml (default: clusters/base)")
args = ap.parse_args(argv)
if args.selftest:
return _selftest()
if args.print_waves:
waves = compute_waves(_load_catalog(Path(args.catalog)))
for node in sorted(waves, key=lambda n: (waves[n], n)):
sys.stdout.write(f"wave {waves[node]} {node}\n")
return 0
if args.emit_waves:
plan = emit_waves(Path(args.catalog), Path(args.apps_dir), Path(args.clusters_dir))
for wave in sorted(plan):
sys.stderr.write(f"[emit-waves] wave {wave}: {', '.join(plan[wave])}\n")
sys.stderr.write(
f"[emit-waves] wrote {len(plan)} wave kustomization(s) under {args.apps_dir}/ "
f"and clusters CRs under {args.clusters_dir}/\n"
)
return 0
image_automation = not args.no_image_automation
def _out_path(outdir: Path, name: str, sds: list) -> Path:
if args.wave_layout:
wdir = outdir / f"wave-{service_wave(sds, name)}"
wdir.mkdir(parents=True, exist_ok=True)
return wdir / f"{name}.yaml"
outdir.mkdir(parents=True, exist_ok=True)
return outdir / f"{name}.yaml"
if args.all:
src = Path(args.all)
files = sorted(p for p in src.glob("*.yaml") if p.name != "render.py")
sds = _load_catalog(src) if args.wave_layout else []
for f in files:
text = render_file(f, image_automation=image_automation)
if args.out_dir:
_out_path(Path(args.out_dir), load_sd(f)["name"], sds).write_text(text)
else:
sys.stdout.write(text)
return 0
if not args.input:
ap.error("provide a ServiceDefinition file, --all DIR, --emit-waves, or --selftest")
inp = Path(args.input)
text = render_file(inp, image_automation=image_automation)
if args.out_dir:
sds = _load_catalog(Path(args.catalog)) if args.wave_layout else []
_out_path(Path(args.out_dir), load_sd(inp)["name"], sds).write_text(text)
else:
sys.stdout.write(text)
return 0
if __name__ == "__main__":
raise SystemExit(main())
Reference in new issue View Git Blame Copy Permalink