fix(cilium): point cgroup.hostRoot at the host's real cgroup2 root

hostRoot was /run/cilium/cgroupv2 with autoMount disabled, but nothing mounts cgroup2 there. The agent mounted one from inside its own private cgroup namespace, so socket-LB attached to the agent's own cgroup and host processes got no Service/LB translation. k3s-prod-1 (Gateway L2 lease holder + kube-vip VIP) therefore could not reach the Gateway VIP, and its apiserver's OIDC authenticator has failed since 2026-09-12. Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 18:07:18 -05:00
19 changed files with 155 additions and 1399 deletions
@@ -14,12 +14,10 @@ resources:
 - nextcloud-db-backup-credentials-prod.sops.yaml
 - nextcloud-db-backup-credentials.sops.yaml
 - nextcloud-db.yaml
 - pihole-bootstrap-secret.sops.yaml
 - pihole-dns.yaml
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
 - plex-claim-secret.sops.yaml
 - plex-remote.yaml
 - plex.yaml
 - qbittorrent-bootstrap-secret.sops.yaml
 - qbittorrent-oauth2-proxy-credentials.sops.yaml
@@ -1,72 +0,0 @@
 # =================================================================================================
 # PiHole WEB/API ADMIN PASSWORD — the one piece of PiHole state that was NOT declared anywhere.
 # =================================================================================================
 # WHY THIS FILE EXISTS. Measured on live prod 2026-10-04, `pihole.toml` marks every non-default
 # setting with `### CHANGED`, and names the ones that came from the environment `### CHANGED (env)`.
 # The complete list was:
 #
 #     dns.upstreams          ### CHANGED (env)   <- declared, catalog/pihole.yaml
 #     dns.listeningMode      ### CHANGED (env)   <- declared, catalog/pihole.yaml
 #     dns.rateLimit.count    ### CHANGED (env)   <- declared, catalog/pihole.yaml
 #     misc.dnsmasq_lines     ### CHANGED (env)   <- declared, catalog/pihole.yaml
 #     files.macvendor        ### CHANGED         <- set by the image itself, container path
 #     webserver.api.pwhash   ### CHANGED         <- SET OUT OF BAND. Declared NOWHERE. This file.
 #
 # So exactly ONE operator-set value lived only on the Longhorn PVC: the admin password, as a
 # one-way BALLOON-SHA256 hash. Nobody could recover it (a hash does not reverse), nothing in git
 # produced it, and a rebuild from git would have come up with a DIFFERENT password — or none.
 #
 # THE REASON THIS BLOCKS HA, which is why it is fixed first. The next step is running more than one
 # PiHole replica, and a replica with a fresh PVC would come up with its own unrelated password. The
 # admin UI would then accept or reject the same credential depending on which pod answered. Declaring
 # it here makes every replica, and every rebuild, identical by construction.
 #
 # MECHANISM: `webserver.api.password` is a real, settable FTL config key — verified in the running
 # image, where `pihole setpassword` is itself just `setFTLConfigValue "webserver.api.password"` with
 # the comment "pihole-FTL will automatically hash the password". So FTL takes CLEARTEXT on this key
 # and writes the hash into `webserver.api.pwhash` itself. catalog/pihole.yaml therefore passes this
 # value as `FTLCONF_webserver_api_password`, and the hash on the PVC becomes a derived artefact
 # rather than the source of truth.
 #
 # DEFENCE IN DEPTH, NOT THE ONLY GATE. The admin UI is already behind the oauth2-proxy sidecar
 # (`--allowed-group=platform-admins`), so this password is a SECOND gate. It is kept rather than set
 # empty because the `pihole` Service still publishes port 80 (PiHole direct, bypassing the proxy on
 # :4180) and only the CiliumNetworkPolicy closes it — and the default-deny policy in this repo has a
 # documented drift history. An unauthenticated admin UI that can rewrite LAN DNS is not a good thing
 # to leave resting on one policy staying correct. Narrowing that Service port is tracked follow-up.
 #
 # ROTATION: `sops` this file in your own terminal, change the value, commit. Flux re-applies the
 # Secret, the pod restarts, FTL re-hashes. Do NOT `sops -d` it into a shared terminal or a log
 # (CLAUDE.md section 6); read it in an editor. The value below was generated randomly at authoring
 # time and has never been printed to a transcript.
 apiVersion: v1
 kind: Secret
 metadata:
    name: pihole-bootstrap
    namespace: pihole
    labels:
        app.kubernetes.io/name: pihole
 type: Opaque
 stringData:
    #ENC[AES256_GCM,data:TMOuRyJeb5IVn3lZPXY7lgh1KGxu7GY9+WIOLHpMB50JqzmM2tHIaWUWKTt1JtZoX/E5/Y7WCUTnfkJLhRBN2zV2MwWRBSLYVy5CMg7xV8PTLRZJYAVUYS5Trg==,iv:q+DpfT6PQfY5ZuNsmP7Ddl+PDLjSm89fJICAhYoW64k=,tag:axijaO3DaxIZgZ3LeK2hXA==,type:comment]
    webui-password: ENC[AES256_GCM,data:7FcK/bUW79M46f/MlouAmATKbMnoUTNuMUjtqJ5vpU4=,iv:I8DBFB39uXZV42Ie2IPwSefmMgdKEtiDknfc9Tu7h5E=,tag:L423784UrDU3TvylpHc6KQ==,type:str]
 sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age:
        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0MkRSNjZLZlpXYkcvaVJL
            cjlqb2wrcWREZnZLdGpsTHNPWEpseHlRZnpjCnJVM1FDQ2M2dXlaS1JBZ2k2V1BP
            Yi9QUDZSMzZsL09pTi93RTdjdGxtY2MKLS0tIDIrQTZIVWdhMGIxSCtlUFN2Mlho
            SWVybld4VjJyaHRSKzZZU0xsUUV2SEEKbDYHV/qBZVspK7DpBW3vYjRMJ/HmAN+9
            G9bOlNEp0jAtcNZEn6lpQ9ORgvmIrhJHoBQRLabdMWzNWSZsNn56tA==
            -----END AGE ENCRYPTED FILE-----
    lastmodified: "2026-10-04T06:01:27Z"
    mac: ENC[AES256_GCM,data:rhTMTOwLOtHXHgqI5M2ZW/6w+UpLpxBlTk7NoN0wmGSqOcE+OGrXA1DhkZrJweyQT0RSNSSBm/6QMztLP81eepOzdp2rEkwvbQPK/l6K9g16Ww9c1IatS3NQsKEB8AoMDx7jBQ6exXYgggPx3MJsNYbE+WyloQ96U9KPdrecAM8=,iv:n34v2zX9D5xsaR1PFCH94mNMiuRfhUyI2lseUCoqiTA=,tag:9YmN9nIVLXfsGYIb0bUl2w==,type:str]
    pgp: []
    encrypted_regex: ^(data|stringData)$
    version: 3.9.4
@@ -35,18 +35,34 @@ spec:
    - protocol: TCP
      port: 53
 ---
-# StatefulSet (R-PLAT-02)
+# PVC pihole-config (R-STATE)
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
  name: pihole-config
  namespace: pihole
  labels:
    app.kubernetes.io/name: pihole
 spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 2Gi
 ---
 # Deployment (R-PLAT-02)
 apiVersion: apps/v1
-kind: StatefulSet
+kind: Deployment
 metadata:
  name: pihole
  namespace: pihole
  labels:
    app.kubernetes.io/name: pihole
 spec:
-  replicas: 2
+  replicas: 1
-  serviceName: pihole
+  strategy:
-  podManagementPolicy: Parallel
+    type: Recreate
  selector:
    matchLabels:
      app.kubernetes.io/name: pihole
@@ -74,21 +90,9 @@ spec:
          value: '0'
        - name: FTLCONF_misc_dnsmasq_lines
          value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}
        - name: FTLCONF_webserver_api_password
          valueFrom:
            secretKeyRef:
              name: pihole-bootstrap
              key: webui-password
        volumeMounts:
        - name: config
-          mountPath: /etc/pihole
+          mountPath: /data/config
        lifecycle:
          postStart:
            exec:
              command:
              - /bin/bash
              - -c
              - "# Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and\n# does NOT run a gravity rebuild.\nset -u\nGRAVITY=/etc/pihole/gravity.db\n\n# THE DECLARED SET \u2014 one URL per line, the source of truth for what PiHole blocks.\n# To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.\nDECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'\n\n# On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,\n# and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.\ni=0\nwhile [ ! -s \"$GRAVITY\" ] && [ \"$i\" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done\nif [ ! -s \"$GRAVITY\" ]; then\n  echo \"pihole-adlists: gravity.db absent after 180s - leaving adlists untouched\" >&2\n  exit 0\nfi\n\nsql() { pihole-FTL sqlite3 \"$GRAVITY\" \"$1\"; }\nCHANGED=0\n\n# 1) Every declared URL must be present AND enabled.\necho \"$DECLARED\" | while IFS= read -r url; do\n  [ -n \"$url\" ] || continue\n  printf '%s\\n' \"$url\"\ndone > /tmp/declared.txt\nwhile IFS= read -r url; do\n  [ -n \"$url\" ] || continue\n  # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.\n  have=$(sql \"select count(*) from adlist where address='$url';\")\n  if [ \"$have\" = \"0\" ]; then\n    echo \"pihole-adlists: adding $url\"\n    sql \"insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');\"\n    CHANGED=1\n  else\n    en=$(sql \"select enabled from adlist where address='$url';\")\n    if [ \"$en\" != \"1\" ]; then\n      echo \"pihole-adlists: re-enabling $url\"\n      sql \"update adlist set enabled=1 where address='$url';\"\n      CHANGED=1\n    fi\n  fi\ndone < /tmp/declared.txt\n\n# 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).\nsql \"select address from adlist where enabled=1;\" | while IFS= read -r addr; do\n  [ -n \"$addr\" ] || continue\n  if ! grep -qxF \"$addr\" /tmp/declared.txt; then\n    echo \"pihole-adlists: disabling undeclared $addr\"\n    sql \"update adlist set enabled=0 where address='$addr';\"\n    echo changed >> /tmp/adlist-changed\n  fi\ndone\n[ -s /tmp/adlist-changed ] && CHANGED=1\nrm -f /tmp/adlist-changed /tmp/declared.txt\n\n# 3) Rebuild gravity ONLY if the declared set actually moved.\nif [ \"$CHANGED\" = \"1\" ]; then\n  echo \"pihole-adlists: declared set changed - running gravity update\"\n  pihole -g || echo \"pihole-adlists: gravity update failed; existing gravity.db retained\" >&2\nelse\n  echo \"pihole-adlists: already matches the declared set - nothing to do\"\nfi\nexit 0\n"
      - name: oauth2-proxy
        image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
        imagePullPolicy: IfNotPresent
@@ -159,39 +163,10 @@ spec:
            port: http
          initialDelaySeconds: 5
          periodSeconds: 15
-      affinity:
+      volumes:
-        podAntiAffinity:
+      - name: config
-          requiredDuringSchedulingIgnoredDuringExecution:
+        persistentVolumeClaim:
-          - labelSelector:
+          claimName: pihole-config
              matchLabels:
                app.kubernetes.io/name: pihole
            topologyKey: kubernetes.io/hostname
  volumeClaimTemplates:
  - metadata:
      name: config
      labels:
        app.kubernetes.io/name: pihole
    spec:
      accessModes:
      - ReadWriteOnce
      storageClassName: longhorn
      resources:
        requests:
          storage: 2Gi
 ---
 # PodDisruptionBudget — keep one replica through a drain (availability.replicas)
 apiVersion: policy/v1
 kind: PodDisruptionBudget
 metadata:
  name: pihole
  namespace: pihole
  labels:
    app.kubernetes.io/name: pihole
 spec:
  minAvailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: pihole
 ---
 # Service
 apiVersion: v1
@@ -294,11 +269,11 @@ spec:
  - name: pihole.slis
    rules:
    - alert: pihole-down
-      expr: kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"} == 0 or absent(kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"})
+      expr: kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"} == 0 or absent(kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"})
      for: 5m
      labels:
        severity: critical
        service: pihole
      annotations:
        summary: pihole is down
-        description: 'pihole: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready).'
+        description: 'pihole: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
@@ -1,126 +0,0 @@
 ---
 # =================================================================================================
 # PLEX REMOTE ACCESS — raw tcp/32400 north-south exposure, and the policy that admits it.
 # =================================================================================================
 # HAND-WRITTEN, NOT RENDERED. A non-catalog app-tier unit anchored to the `plex` ServiceDefinition
 # and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS table — the same
 # mechanism gitea-actions and pihole-dns use. It is hand-written for the SAME reason those are:
 # schemas/servicedefinition.schema.json is `additionalProperties: false` and has NO field for a
 # type=LoadBalancer Service, for a pinned lbipam VIP, or for raw-TCP north-south exposure at all.
 # `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator.
 #
 # WHY THE L7 GATEWAY IS NOT ENOUGH HERE — and it is NOT because the Gateway fails to serve Plex.
 # Measured 2026-10-03 through the Gateway VIP: `GET https://plex.home.keithjackson.dev/identity`
 # returned HTTP 200 with `claimed="1"` and the real machineIdentifier, and the protected endpoints
 # (/library/sections, /status/sessions, /:/prefs) all returned 401. The L7 path WORKS.
 #
 # The problem is DISCOVERY, not proxying. Plex apps (phone, TV, Roku) do not resolve
 # `plex.${BASE_DOMAIN}`; they ask plex.tv for the server's connection list. A server gets onto that
 # list one of two ways: it advertises a custom URL (`customConnections`, the ADVERTISE_IP env), or it
 # publishes a reachable public IP:port for its own tcp/32400. Measured in the live Preferences.xml on
 # 2026-10-03: `customConnections` ABSENT, `ManualPortMappingMode` ABSENT, while
 # `PublishServerOnPlexOnlineKey="1"` — the server was trying to publish and had nothing reachable to
 # publish, so remote clients saw NO connection at all. The operator chose the NATIVE path (this
 # file) over advertising the Gateway URL: Plex then terminates its own TLS with its
 # `*.plex.direct` certificate and video streams bypass the Envoy L7 hop entirely.
 #
 # VIP: ${PLEX_L4_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.57, prod 192.168.2.24), PINNED via
 # lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
 # (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the existing
 # CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config is needed;
 # this Service just consumes the pool. PINNING IS NOT OPTIONAL: the address is typed into a WAN
 # port-forward on the SOHO router, which is hand-maintained and off-repo, so it can never move.
 # 192.168.2.24 verified FREE live 2026-10-03 — `default-pool` reported IPsTotal 11 / IPsUsed 4 /
 # IPsAvailable 7, and the only four allocations were .20 (cilium-gateway-shared), .21 (unifi-l4-*),
 # .22 (crafty-l4-*) and .23 (pihole-dns-*).
 #
 # ONE SERVICE, TCP ONLY. Unlike unifi/crafty/pihole-dns this needs no second Service and no
 # lbipam.cilium.io/sharing-key: Plex's remote access is tcp/32400 and nothing else. (Plex's
 # DLNA/GDM discovery uses UDP, but that is LAN-only multicast and is not part of remote access.)
 #
 # NO `loadBalancerSourceRanges`, DELIBERATELY — and this is the one place this file diverges from
 # its closest sibling. apps/wave-0/pihole-dns.yaml pins its source range to ${LAN_CIDR} because a
 # resolver must never answer the internet. This Service is the OPPOSITE case: admitting remote
 # clients from arbitrary internet addresses is its entire purpose, so there is no range to narrow
 # to. Plex authenticates every caller itself (measured above: 401 on every protected endpoint
 # without a token), which is what makes an unfiltered port defensible here.
 #
 # NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
 # `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their VIP as a public
 # Cloudflare record. This Service must not, for pihole-dns's reason: ${PLEX_L4_VIP} is an RFC1918
 # address that is useless in public DNS. Remote clients get this server's address from plex.tv, not
 # from the zone. The existing public `plex.${BASE_DOMAIN}` record is published from the rendered
 # HTTPRoute via the gateway source and is NOT affected by this file — the web UI keeps working
 # through the Gateway exactly as before.
 #
 # -------------------------------------------------------------------------------------------------
 # externalTrafficPolicy: Cluster — inherited from crafty's 8-DAY LAN OUTAGE, not re-derived.
 # -------------------------------------------------------------------------------------------------
 # `Local` is the tempting choice because it preserves the client source IP, which this service
 # actually has a use for (see the allowedNetworks note below). It is still WRONG here, for the
 # reason platform/crafty/service-l4.yaml documents at length: Cilium's L2 announcer elects ONE lease
 # holder per Service and does NOT follow the backend pod, so with `Local` the VIP can be announced
 # by a node that has no Plex pod, which black-holes the port while every object still reports
 # healthy. Plex is a single-replica Deployment with `strategy: Recreate` pinned to wherever the
 # scheduler put it, so the mismatch is the normal case, not a rare one. Take the SNAT.
 #
 # KNOWN CONSEQUENCE OF THAT CHOICE, stated rather than discovered later: `Cluster` SNATs the client
 # to the ingress NODE's address (192.168.2.1x), which falls inside the `192.168.0.0/16` entry of
 # catalog/plex.yaml's ALLOWED_NETWORKS. Plex will therefore classify every REMOTE client as LAN.
 # Measured 2026-10-03 on the Gateway path, which SNATs the same way: Plex logged the request as
 # `[10.0.2.214:36040 (Allowed Network (WAN))]`. This is a METERING effect, not a security one —
 # authorization is unchanged (401s above), but Plex's remote-quality/bandwidth limits will not bind
 # on these streams because it believes they are local. Narrowing ALLOWED_NETWORKS cannot fix it
 # without also de-classifying genuine LAN clients, so it is accepted.
 apiVersion: v1
 kind: Service
 metadata:
  name: plex-l4-tcp
  namespace: plex
  labels:
    app.kubernetes.io/name: plex
  annotations:
    lbipam.cilium.io/ips: "${PLEX_L4_VIP}"
 spec:
  type: LoadBalancer
  externalTrafficPolicy: Cluster
  selector:
    app.kubernetes.io/name: plex
  ports:
    - name: plex-remote
      port: 32400
      targetPort: 32400
      protocol: TCP
 ---
 # INGRESS allow for the raw port (R-NET-01/02).
 #
 # The rendered plex.yaml already carries `plex-allow-gateway-ingress` (fromEntities: ingress ->
 # 32400) and three reciprocal allows for sonarr/radarr/seerr. CiliumNetworkPolicies are ADDITIVE, so
 # this file adds the off-cluster case without touching those.
 #
 # `world` is Cilium's entity for anything outside the cluster, which is what a remote Plex client is.
 # `host` and `cluster` mirror platform/crafty/networkpolicy.yaml's game-port rule and are what make
 # this work under `externalTrafficPolicy: Cluster`: after SNAT the packet's source is the ingress
 # NODE, which Cilium classifies as `host` (same node) or `remote-node` (a different one) — NOT
 # `world`. Listing `world` alone would compile cleanly and then drop every remote stream, so all
 # four entities are named.
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
 metadata:
  name: plex-allow-remote-32400
  namespace: plex
  labels:
    app.kubernetes.io/name: plex
 spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: plex
  ingress:
    - fromEntities:
        - world
        - cluster
        - host
        - remote-node
      toPorts:
        - ports:
            - { port: "32400", protocol: TCP }
@@ -1,8 +1,6 @@
 # promtool rule-unit tests for the wave-0 per-app `<app>-down` SLI rules.
 # Manifests under test: every PrometheusRule in apps/wave-0/ — gitea, jackett, pihole, plex,
-# qbittorrent (container/Deployment), pihole (container/StatefulSet — `availability.replicas: 2`, so
+# qbittorrent (container/Deployment) and vm-demo (KubeVirt VMI). The harness discovers this file by
 # its SLI reads kube_statefulset_status_replicas_ready; see schemas/renderer-contract.md
 # `### availability`) and vm-demo (KubeVirt VMI). The harness discovers this file by
 # the SUITE convention (`<dir>/tests/*.test.yaml`) and flattens those RENDERED manifests, so these
 # tests run against the exact expressions Flux applies, never a copy.
 #
@@ -43,7 +41,7 @@ tests:
        values: '1+0x20'
      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
        values: '1+0x20'
-      - series: 'kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}'
+      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
        values: '1+0x20'
      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
        values: '1+0x20'
@@ -83,7 +81,7 @@ tests:
        values: '1+0x20'
      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
        values: '1+0x20'
-      - series: 'kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}'
+      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
        values: '1+0x20'
      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
        values: '0+0x20'
@@ -134,7 +132,7 @@ tests:
        values: '0+0x20'
      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
        values: '0+0x20'
-      - series: 'kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}'
+      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
        values: '0+0x20'
      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
        values: '1+0x20'
@@ -172,10 +170,10 @@ tests:
              severity: critical
              service: pihole
              namespace: pihole
-              statefulset: pihole
+              deployment: pihole
            exp_annotations:
              summary: "pihole is down"
-              description: "pihole: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
+              description: "pihole: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
      - eval_time: 10m
        alertname: qbittorrent-down
        exp_alerts:
@@ -12,42 +12,6 @@ storage:
  - name: config
    size: 2Gi
    class: longhorn           # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
    # THE PVC WAS MOUNTED SOMEWHERE PIHOLE NEVER WRITES, AND SO HELD NOTHING AT ALL. Without an
    # explicit mountPath the renderer defaults to /data/<name>, i.e. /data/config — but PiHole keeps
    # every piece of its state in /etc/pihole. MEASURED on live prod 2026-10-04: `/etc/pihole` (7.1 M
    # — gravity.db, pihole.toml, the query DB) was on the **overlay** filesystem, i.e. the
    # container's EPHEMERAL layer, while `/dev/longhorn/pvc-849edb86...` sat mounted at
    # /data/config holding `lost+found` and 24 K of nothing. Three consequences, all live until now:
    #   * every restart silently discarded PiHole's whole state, re-downloading 72 k gravity domains
    #     and regenerating the admin password hash;
    #   * `backup.policy: daily-offcluster` below has been faithfully backing up an EMPTY volume,
    #     which is worse than no backup because it reports success;
    #   * it is also why phase 1's password/adlist declaration mattered more than it first appeared —
    #     that state was not merely undeclared, it was ephemeral.
    # This is the same class of bug catalog/plex.yaml's `config` entry documents ("Without this the
    # PVC sat unused and Plex wrote to the container's EPHEMERAL layer").
    #
    # /etc/pihole IS THE IMAGE'S INTENDED MOUNT POINT, and an EMPTY volume there self-initialises.
    # Verified in the running image, not assumed:
    #   * `/usr/bin/start.sh` (PID 1) calls `migrate_gravity`, whose `migrate_to_database()` creates
    #     the gravity database when `[ ! -e "$gravityDBfile" ]` — the same branch the postStart hook
    #     below relies on, and the branch that produced the current ephemeral copy.
    #   * FTL writes `pihole.toml` itself from the `FTLCONF_*` env above (that is what the
    #     `### CHANGED (env)` markers in it are).
    #   * NO `podSecurityContext.fsGroup` is needed and adding one would be inert: `start.sh` line 5
    #     is `chown -R pihole:pihole /etc/pihole/ /var/log/pihole/` and PID 1 runs as **uid 0**, so
    #     the image fixes ownership of a fresh root-owned volume before anything else starts. Same
    #     arrangement as plex's image managing its own /config ownership.
    #   * the image's own default for `files.macvendor` was moved OUT of this directory (to
    #     `/macvendor.db`), which is what a maintainer does to a path they expect to be a volume.
    #
    # WHY IT IS FIXED IN THIS BRANCH rather than left for its own: `availability.replicas: 2` renders
    # per-replica volumeClaimTemplates, and with the wrong mountPath those would be two 2 Gi Longhorn
    # volumes (times three Longhorn replicas each) holding nothing but `lost+found` while all state
    # stayed ephemeral — paying the cost of persistence and getting none of it. It also REMOVES this
    # change's main operational risk: with gravity.db persisted, a pod restart is cheap, instead of
    # every restart re-downloading 72 k domains inside a wave that has a 5 m timeout.
    mountPath: /etc/pihole
 dependencies: []              # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
 # ------------------------------------------------------------------------------------------------
 # LAN SPLIT-HORIZON DNS — the half of R-DNS-03 that platform/coredns/coredns-custom.yaml defers to.
@@ -136,53 +100,6 @@ env:
  # while resolution for the zone does not work. Confirmed both ways against the live image.
  - name: FTLCONF_misc_dnsmasq_lines
    value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}"
  # THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
  # apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
  # carries the full rationale and the measurement that found it.
  #
  # `webserver.api.password` takes CLEARTEXT and FTL hashes it itself into `webserver.api.pwhash`.
  # Verified in the running image, not assumed: `pihole setpassword` is itself
  # `setFTLConfigValue "webserver.api.password" "$PASSWORD"`, commented upstream as "pihole-FTL will
  # automatically hash the password". So the BALLOON-SHA256 hash on the PVC is now a DERIVED artefact
  # and this is its source of truth. Rotating the Secret rotates the password.
  - name: FTLCONF_webserver_api_password
    valueFrom:
      secretKeyRef:
        name: pihole-bootstrap
        key: webui-password
 availability:
  # TWO REPLICAS, ON DIFFERENT NODES. LAN DNS is a critical network service and it was ONE pod on
  # ONE node. Measured 2026-10-04, that cost more than "until it reschedules":
  #   * the node running the pod dies -> ~40s to mark it NotReady, then `tolerationSeconds: 300`
  #     before the pod is even evicted, then reschedule, then Longhorn must force-detach the RWO
  #     volume from a dead node and reattach it. Roughly 6-10 MINUTES with no LAN resolution.
  #   * the node holding the `pihole-dns-*` L2 lease dies -> Cilium re-elects an announcer, tens of
  #     seconds. (These were DIFFERENT nodes: lease on k3s-prod-2, pod on k3s-prod-1. Two of three
  #     nodes were single points of failure for LAN DNS, with wildly different blast radii.)
  # The Services already tolerate this: `externalTrafficPolicy: Cluster` on the DNS LoadBalancers
  # (apps/wave-0/pihole-dns.yaml) means the L2 lease holder forwards to ANY ready backend, so a
  # second replica is picked up with no change there.
  #
  # WHAT THIS DOES NOT FIX, and it is the bigger risk: LAN DNS still depends on the CLUSTER. etcd
  # quorum loss, a Cilium regression, a Longhorn problem or a bad reconcile takes DNS down
  # everywhere regardless of replica count. The only thing that removes that dependency is a
  # resolver outside the cluster carrying the same `address=` lines. Considered and not chosen
  # (operator decision, 2026-10-04); see decisions/2026-10-04-pihole-multi-replica.md.
  #
  # THE PREREQUISITE WAS DONE FIRST, and this field would have been actively harmful without it. A
  # replica gets a FRESH volume, and until 2026-10-04 PiHole's admin password and adlists lived only
  # on the PVC — so replica 1 would have come up SERVING DNS AND BLOCKING NOTHING, with a different
  # password, and answers would have varied by which pod replied. Both are now declared (the
  # password as a SOPS Secret -> FTLCONF_webserver_api_password; the adlists reconciled into
  # gravity.db by the postStart hook below), which is what makes a fresh replica identical to its
  # sibling. See decisions/2026-10-04-pihole-declare-state.md.
  #
  # WHY 2 AND NOT 3. Two tolerates the loss of any single node, which is the stated goal, and leaves
  # a spare node for the evicted replica to land on under the HARD anti-affinity the renderer emits.
  # Three would pin one replica per node with no spare, so a single node failure would leave a
  # replica permanently Pending and — `apps-wave-0` being `wait: true` — stall the app tier for as
  # long as the node was down. Two is the count that degrades gracefully here.
  replicas: 2
 ingressRoutes:
  - host: pihole.${BASE_DOMAIN}
    port: 80                  # admin web UI, protected at the Gateway by forward-auth (docs/09)
@@ -197,110 +114,6 @@ auth:
  # rather than a renderer constant: the measurement that justifies the flag on two apps is
  # evidence about those two apps. Its oauth2-proxy log holds 0 unauthenticated requests across
  # the current pod's whole 23h lifetime (measured 2026-10-01), so there is nothing to act on.
 lifecycle:
  # LEVEL-TRIGGERED ADLIST ASSERTION, run on EVERY container start. Same pattern, and the same
  # reasoning, as catalog/qbittorrent.yaml's postStart: desired state lives in git and is
  # RE-ASSERTED unconditionally rather than set once by hand.
  #
  # WHY THIS IS NEEDED AT ALL — the blocklists were NOT reproducible from git. Measured on live prod
  # 2026-10-04: `gravity.db` held exactly ONE adlist (StevenBlack's hosts, enabled) and 72,234
  # gravity domains, and NOTHING in this repo named it. Worse, a FRESH database does not seed it:
  # the image's own schema template `/etc/.pihole/advanced/Templates/gravity.db.sql` creates the
  # `adlist` table and inserts NO rows into it (grepped in the running image — there is no
  # `INSERT INTO adlist` at all). So a rebuild from git, or a replica with a new PVC, would come up
  # SERVING DNS AND BLOCKING NOTHING, silently and with every object reporting healthy.
  #
  # WHY A postStart HOOK AND NOT A FILE. PiHole v6 keeps adlists as DB rows, not config. The
  # `/etc/pihole/adlists.list` file still exists in `gravity.sh` but is MIGRATION-ONLY: its import
  # sits inside `migrate_to_database()`, guarded by `if [ ! -e "$gravityDBfile" ]`, so it fires only
  # while CREATING a database and never again on an existing one. A ConfigMap mounted there would
  # therefore be silently ignored on every pod this cluster currently runs. There is also no
  # `pihole adlist` CLI subcommand (only allow/deny for domains), so SQL against the DB is the
  # actual interface. FTL ships an embedded shell (`pihole-FTL sqlite3`, SQLite 3.53.1) so no extra
  # tooling is needed in the image.
  #
  # IT RECONCILES, IT DOES NOT JUST ADD: adlists present in the DB but absent from DECLARED below are
  # DISABLED (not deleted — deleting cascades to `gravity` rows and would force a full re-download of
  # everything on a typo). So this list is the source of truth in both directions, and `pihole -g`
  # runs ONLY when something actually changed, which keeps the steady-state start free of a 70k-domain
  # rebuild.
  #
  # NO BRACED DOLLAR EXPANSION ANYWHERE IN THIS SCRIPT, DELIBERATELY — the same contract
  # catalog/qbittorrent.yaml documents at length. apps-wave-0 declares postBuild.substituteFrom
  # cluster-vars, so Flux runs envsubst over this whole manifest at reconcile time; a `${VAR}` form
  # would behave differently on the cluster than under a plain `kubectl apply` of the golden. Bare
  # `$VAR` and `$(cmd)` are untouched by envsubst and so behave identically however this is run.
  # `render.py`'s assert_substitution_safe refuses to emit a manifest that breaks this.
  postStart:
    exec:
      command:
        - /bin/bash
        - -c
        - |
          # Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and
          # does NOT run a gravity rebuild.
          set -u
          GRAVITY=/etc/pihole/gravity.db
          # THE DECLARED SET — one URL per line, the source of truth for what PiHole blocks.
          # To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.
          DECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'
          # On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,
          # and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.
          i=0
          while [ ! -s "$GRAVITY" ] && [ "$i" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done
          if [ ! -s "$GRAVITY" ]; then
            echo "pihole-adlists: gravity.db absent after 180s - leaving adlists untouched" >&2
            exit 0
          fi
          sql() { pihole-FTL sqlite3 "$GRAVITY" "$1"; }
          CHANGED=0
          # 1) Every declared URL must be present AND enabled.
          echo "$DECLARED" | while IFS= read -r url; do
            [ -n "$url" ] || continue
            printf '%s\n' "$url"
          done > /tmp/declared.txt
          while IFS= read -r url; do
            [ -n "$url" ] || continue
            # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.
            have=$(sql "select count(*) from adlist where address='$url';")
            if [ "$have" = "0" ]; then
              echo "pihole-adlists: adding $url"
              sql "insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');"
              CHANGED=1
            else
              en=$(sql "select enabled from adlist where address='$url';")
              if [ "$en" != "1" ]; then
                echo "pihole-adlists: re-enabling $url"
                sql "update adlist set enabled=1 where address='$url';"
                CHANGED=1
              fi
            fi
          done < /tmp/declared.txt
          # 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).
          sql "select address from adlist where enabled=1;" | while IFS= read -r addr; do
            [ -n "$addr" ] || continue
            if ! grep -qxF "$addr" /tmp/declared.txt; then
              echo "pihole-adlists: disabling undeclared $addr"
              sql "update adlist set enabled=0 where address='$addr';"
              echo changed >> /tmp/adlist-changed
            fi
          done
          [ -s /tmp/adlist-changed ] && CHANGED=1
          rm -f /tmp/adlist-changed /tmp/declared.txt
          # 3) Rebuild gravity ONLY if the declared set actually moved.
          if [ "$CHANGED" = "1" ]; then
            echo "pihole-adlists: declared set changed - running gravity update"
            pihole -g || echo "pihole-adlists: gravity update failed; existing gravity.db retained" >&2
          else
            echo "pihole-adlists: already matches the declared set - nothing to do"
          fi
          exit 0
 update:
  channel: stable             # R-UPD-01
  range: ">=2024.0.0"
@@ -272,15 +272,7 @@ def apply_defaults(sd: dict) -> dict:
    # object that actually runs on the cluster is not the one this SD renders (keycloak: the
    # Keycloak operator runs a StatefulSet, so a Deployment-shaped rule would `absent()` forever).
    lw = dict(obs.get("livenessWorkload") or {})
-    # The default must track what is ACTUALLY rendered, or the `up` SLI silently dies: a
+    lw.setdefault("kind", "VirtualMachineInstance" if s["kind"] == "vm" else "Deployment")
    # Deployment-shaped rule against a StatefulSet `absent()`s forever, which is an alert that can
    # never fire — the exact failure mode the keycloak override was added for. Deriving it from
    # `availability.replicas` means bumping the replica count cannot leave a stale rule behind.
    lw.setdefault(
        "kind",
        "VirtualMachineInstance" if s["kind"] == "vm"
        else ("StatefulSet" if _is_statefulset(s) else "Deployment"),
    )
    lw.setdefault("name", s["name"])
    obs["livenessWorkload"] = lw
    s["observability"] = obs
@@ -483,37 +475,6 @@ def r_bulk_pvs(s: dict) -> list:
    return out
 def _replicas(s: dict) -> int:
    """Declared replica count (schema 1.10.0 `availability.replicas`). Default 1."""
    return int((s.get("availability") or {}).get("replicas", 1) or 1)
 def _is_statefulset(s: dict) -> bool:
    """True when this container workload must render a StatefulSet rather than a Deployment.
    THE DECIDING CONSTRAINT IS THE VOLUME, not a preference. A longhorn storage entry renders
    ReadWriteOnce (r_pvcs), and one RWO PVC cannot be attached by pods on two different nodes --
    which is exactly what `availability.replicas > 1` plus the node anti-affinity below asks for.
    A 2-replica Deployment over a shared RWO claim does not degrade, it simply never schedules the
    second pod. A StatefulSet's volumeClaimTemplates give each replica its OWN claim, which is the
    only shape that satisfies both at once.
    A replicas-1 service keeps rendering a Deployment so every pre-existing golden is byte-identical.
    """
    return s["kind"] == "container" and _replicas(s) > 1
 def _sts_claim_volumes(s: dict) -> list:
    """Storage entries that become per-replica volumeClaimTemplates on a StatefulSet.
    Only the dynamically-provisioned (longhorn) entries. A `bulk` entry is a STATIC, pre-bound NFS
    PV over data that already exists outside the cluster and is shared read-only by every consumer
    (r_bulk_pvs) -- per-replica copies of it are neither possible nor wanted, so it stays a
    standalone PVC and an ordinary pod volume.
    """
    return [v for v in s["storage"] if v.get("class", "longhorn") != "bulk"]
 def r_pvcs(s: dict) -> list:
    """One PVC per storage entry (R-STATE-01/02/04).
@@ -521,15 +482,8 @@ def r_pvcs(s: dict) -> list:
    bulk     -> the STATIC external NFS PV rendered by r_bulk_pvs, pre-bound by volumeName.
    """
    out = []
    # On a StatefulSet the longhorn claims are NOT standalone objects — they are
    # volumeClaimTemplates, so each replica gets its own (see _is_statefulset). Emitting both would
    # leave an orphan PVC that nothing mounts and that `prune` would later delete out from under
    # nobody. `bulk` is unaffected and still rendered here.
    sts = _is_statefulset(s)
    for vol in s["storage"]:
        cls = vol.get("class", "longhorn")
        if sts and cls != "bulk":
            continue
        if cls == "bulk":
            # Mirror image of the PV: the same explicitly-empty class (so nothing dynamically
            # provisions), the same access mode, and volumeName completing the two-sided binding.
@@ -610,12 +564,6 @@ def r_deployment(s: dict) -> dict:
        if vol.get("readOnly"):
            mount["readOnly"] = True
        volume_mounts.append(mount)
        # On a StatefulSet a longhorn volume is supplied by its volumeClaimTemplate, whose
        # metadata.name IS the pod volume name. Adding it to pod.volumes as well is a conflict
        # (duplicate volume name), so it is skipped here and emitted below. The MOUNT is unchanged —
        # the container cannot tell the difference, which is what keeps this transparent to the app.
        if _is_statefulset(s) and vol.get("class", "longhorn") != "bulk":
            continue
        volumes.append(
            {
                "name": vol["name"],
@@ -713,80 +661,20 @@ def r_deployment(s: dict) -> dict:
    if ann:
        tmpl_meta["annotations"] = ann
-    sts = _is_statefulset(s)
+    spec = {"replicas": 1}
-
+    # An app with persistent storage mounts a ReadWriteOnce Longhorn PVC. The default
-    # NODE ANTI-AFFINITY, and it is REQUIRED rather than preferred. `availability.replicas > 1`
+    # RollingUpdate strategy tries to start the NEW pod before terminating the old one, but
-    # exists to survive the loss of one node; `preferred` would let both replicas land on the same
+    # the old pod still holds the RWO volume -> the new pod is stuck ContainerCreating and
-    # node and quietly deliver none of that, with every object reporting healthy. The cost is real
+    # the rollout deadlocks (single-replica app can never hand the volume over). Recreate
-    # and is accepted: with N replicas the workload needs N schedulable nodes, so on a 3-node
+    # terminates the old pod first, freeing the PVC for the new one (R-STATE-01/02).
-    # cluster losing TWO nodes leaves a replica Pending and — because `apps-wave-0` is `wait: true`
+    if s["storage"]:
    # with `apps-wave-1` dependsOn it — STALLS THE APP TIER. That coincides with having already lost
    # etcd quorum on a 3-node control plane, i.e. the cluster is not reconciling anything at that
    # point anyway, so the stall is a symptom rather than a new failure.
    if sts:
        pod_spec["affinity"] = {
            "podAntiAffinity": {
                "requiredDuringSchedulingIgnoredDuringExecution": [
                    {
                        "labelSelector": {
                            "matchLabels": {"app.kubernetes.io/name": s["name"]}
                        },
                        "topologyKey": "kubernetes.io/hostname",
                    }
                ]
            }
        }
    spec = {"replicas": _replicas(s)}
    if sts:
        # serviceName is a required StatefulSet field. It points at the service rendered by
        # r_service; that Service is a normal ClusterIP rather than headless, so per-pod DNS
        # (<pod>.<svc>) does NOT resolve — and nothing here needs it. These replicas are
        # interchangeable resolvers behind one VIP, not peers that address each other.
        spec["serviceName"] = s["name"]
        # Parallel, not the OrderedReady default. OrderedReady starts replica N+1 only after N is
        # Ready, so a cold start costs the sum of every replica's first-boot work instead of the
        # max — and `apps-wave-0` has `timeout: 5m`. There is no ordering requirement between
        # replicas to protect (no leader, no migration, no shared volume), so serialising buys
        # nothing and risks the wave timing out on a rebuild.
        spec["podManagementPolicy"] = "Parallel"
    elif s["storage"]:
        # An app with persistent storage mounts a ReadWriteOnce Longhorn PVC. The default
        # RollingUpdate strategy tries to start the NEW pod before terminating the old one, but
        # the old pod still holds the RWO volume -> the new pod is stuck ContainerCreating and
        # the rollout deadlocks (single-replica app can never hand the volume over). Recreate
        # terminates the old pod first, freeing the PVC for the new one (R-STATE-01/02).
        #
        # A StatefulSet needs none of this and cannot express it: each replica owns its own claim,
        # so a rolling update hands nothing over, and `strategy` is not a StatefulSet field at all.
        spec["strategy"] = {"type": "Recreate"}
    spec["selector"] = {"matchLabels": {"app.kubernetes.io/name": s["name"]}}
    spec["template"] = {"metadata": tmpl_meta, "spec": pod_spec}
    if sts:
        # Per-replica claims. metadata.name MUST equal the pod volume name the container mounts
        # (see the volumes loop above); the resulting PVCs are <name>-<sts>-<ordinal>, e.g.
        # config-pihole-0. Same tier and access mode a standalone longhorn PVC gets from r_pvcs —
        # this changes WHO owns the claim, never what it is.
        tmpls = []
        for vol in _sts_claim_volumes(s):
            vspec = {"accessModes": ["ReadWriteOnce"], "storageClassName": "longhorn"}
            if vol.get("size"):
                vspec["resources"] = {"requests": {"storage": vol["size"]}}
            tmpls.append(
                {
                    "metadata": {
                        "name": vol["name"],
                        "labels": {"app.kubernetes.io/name": s["name"]},
                    },
                    "spec": vspec,
                }
            )
        if tmpls:
            spec["volumeClaimTemplates"] = tmpls
    return {
        "apiVersion": "apps/v1",
-        "kind": "StatefulSet" if sts else "Deployment",
+        "kind": "Deployment",
        "metadata": {
            "name": s["name"],
            "namespace": s["name"],
@@ -796,35 +684,6 @@ def r_deployment(s: dict) -> dict:
    }
 def r_pdb(s: dict):
    """PodDisruptionBudget for a multi-replica workload (availability.replicas > 1).
    minAvailable 1, not `maxUnavailable: 1`. The budget's job here is to stop a VOLUNTARY
    disruption — a node drain for maintenance, a cluster upgrade — from taking the last replica
    with it; `minAvailable: 1` says that whatever else happens, one must keep serving. It does
    NOT protect against involuntary disruption (a node dying), which is what the anti-affinity
    and the replica count are for.
    Rendered only when replicas > 1: a PDB over a single-replica workload is actively harmful, as
    `minAvailable: 1` would then block every drain of its node forever.
    """
    if not _is_statefulset(s):
        return None
    return {
        "apiVersion": "policy/v1",
        "kind": "PodDisruptionBudget",
        "metadata": {
            "name": s["name"],
            "namespace": s["name"],
            "labels": {"app.kubernetes.io/name": s["name"]},
        },
        "spec": {
            "minAvailable": 1,
            "selector": {"matchLabels": {"app.kubernetes.io/name": s["name"]}},
        },
    }
 def r_service(s: dict):
    ports = _service_ports(s)
    # A declared metrics endpoint becomes a NAMED Service port, because that name is the only
@@ -1958,13 +1817,7 @@ def render(sd: dict, image_automation: bool = True, inbound: list | None = None)
            )
        for pvc in r_pvcs(s):
            docs.append((f"PVC {pvc['metadata']['name']} (R-STATE)", pvc))
-        wl = r_deployment(s)
+        docs.append(("Deployment (R-PLAT-02)", r_deployment(s)))
        docs.append((f"{wl['kind']} (R-PLAT-02)", wl))
        pdb = r_pdb(s)
        if pdb:
            docs.append(
                ("PodDisruptionBudget — keep one replica through a drain (availability.replicas)", pdb)
            )
        svc = r_service(s)
        if svc:
            docs.append(("Service", svc))
@@ -2149,24 +2002,6 @@ EXTRA_UNITS: dict = {
            "pihole-dns.yaml",
        ],
    },
    # Plex's remote-access exposure: the raw tcp/32400 LoadBalancer Service on a pinned VIP, plus the
    # CiliumNetworkPolicy that admits off-cluster clients to it (apps/wave-0/plex-remote.yaml).
    # `above: False` co-locates it in plex's OWN wave — these are the plex pod's own Service and its
    # own policy, applied in the same wave as the Deployment they select, exactly as the rendered
    # Service and CNPs in plex.yaml are. Nothing in a higher wave gates on it.
    #
    # It is a non-catalog unit for the same reason pihole-dns is: the ServiceDefinition schema cannot
    # express a LoadBalancer Service or a pinned lbipam VIP (additionalProperties is false, and
    # `ingressRoutes` only renders HTTPRoutes on the L7 Gateway, which cannot carry Plex's raw port).
    # Plex's own settings stay in the catalog as env on catalog/plex.yaml; the only thing living
    # outside it is the exposure the schema has no word for.
    "plex-remote": {
        "anchor": "plex",
        "above": False,
        "files": [
            "plex-remote.yaml",
        ],
    },
 }
@@ -2413,132 +2248,6 @@ def _selftest() -> int:
    check("Namespace" in kinds, "container: missing Namespace")
    check("NetworkPolicy" in kinds, "container: missing default-deny NetworkPolicy")
    check("Deployment" in kinds, "container: missing Deployment")
    check(
        "PodDisruptionBudget" not in kinds,
        "replicas-1 container must render NO PodDisruptionBudget (minAvailable 1 over a single "
        "replica blocks every drain of its node forever)",
    )
    # 1b) availability.replicas (schema 1.10.0). The DEFAULT path must be untouched, and the >1
    # path must change workload kind, claim ownership, spreading, budget AND the SLI metric
    # together — each of these has a silent-failure mode if it is the one that gets forgotten.
    base_av = {
        "name": "av",
        "source": {"image": "nginx:1"},
        "storage": [{"name": "config", "size": "1Gi"}],
        "observability": {"metrics": True, "slis": ["up"]},
    }
    one = render({**base_av, "availability": {"replicas": 1}})
    one_kinds = [d[1]["kind"] for d in one]
    check(
        "Deployment" in one_kinds and "StatefulSet" not in one_kinds,
        "availability.replicas=1 must still render a Deployment (back-compat)",
    )
    check(
        "PersistentVolumeClaim" in one_kinds,
        "availability.replicas=1 must still render a standalone PVC",
    )
    check(
        "PodDisruptionBudget" not in one_kinds,
        "availability.replicas=1 must render no PodDisruptionBudget",
    )
    many = render({**base_av, "availability": {"replicas": 2}})
    many_kinds = [d[1]["kind"] for d in many]
    check(
        "StatefulSet" in many_kinds and "Deployment" not in many_kinds,
        "availability.replicas>1 must render a StatefulSet, not a Deployment (a shared RWO claim "
        "cannot be attached on two nodes, so a multi-replica Deployment never schedules)",
    )
    check(
        "PersistentVolumeClaim" not in many_kinds,
        "availability.replicas>1 must NOT also render a standalone longhorn PVC — it would be an "
        "orphan that nothing mounts",
    )
    sts = next(d[1] for d in many if d[1]["kind"] == "StatefulSet")
    check(sts["spec"].get("replicas") == 2, "StatefulSet must carry the declared replica count")
    check(
        "strategy" not in sts["spec"],
        "StatefulSet must not carry `strategy` (not a StatefulSet field; Recreate is a "
        "Deployment-only workaround for handing over one RWO volume)",
    )
    check(
        sts["spec"].get("podManagementPolicy") == "Parallel",
        "StatefulSet must set podManagementPolicy=Parallel — OrderedReady serialises first-boot "
        "work across replicas and apps-wave-0 has a 5m timeout",
    )
    check(sts["spec"].get("serviceName") == "av", "StatefulSet must set serviceName")
    tmpls = sts["spec"].get("volumeClaimTemplates") or []
    check(
        [t["metadata"]["name"] for t in tmpls] == ["config"],
        "StatefulSet volumeClaimTemplate name must equal the pod volume name the container mounts, "
        "or the pod gets a duplicate/unresolvable volume",
    )
    check(
        tmpls and tmpls[0]["spec"]["accessModes"] == ["ReadWriteOnce"]
        and tmpls[0]["spec"]["storageClassName"] == "longhorn",
        "StatefulSet claim template must keep the same tier/access mode as the standalone PVC",
    )
    sts_vols = {v["name"] for v in sts["spec"]["template"]["spec"].get("volumes", [])}
    check(
        "config" not in sts_vols,
        "a claim-template volume must NOT also appear in pod.volumes (duplicate volume name)",
    )
    mounts = sts["spec"]["template"]["spec"]["containers"][0].get("volumeMounts") or []
    check(
        any(m["name"] == "config" for m in mounts),
        "the container must still mount the volume by name on a StatefulSet (transparent to the app)",
    )
    aff = (sts["spec"]["template"]["spec"].get("affinity") or {}).get("podAntiAffinity") or {}
    req = aff.get("requiredDuringSchedulingIgnoredDuringExecution") or []
    check(
        req and req[0].get("topologyKey") == "kubernetes.io/hostname",
        "availability.replicas>1 must render REQUIRED per-node anti-affinity — `preferred` can put "
        "every replica on one node and deliver none of the intended node-failure tolerance",
    )
    pdbs = [d[1] for d in many if d[1]["kind"] == "PodDisruptionBudget"]
    check(len(pdbs) == 1, "availability.replicas>1 must render exactly one PodDisruptionBudget")
    check(
        pdbs and pdbs[0]["spec"].get("minAvailable") == 1,
        "PodDisruptionBudget must be minAvailable=1 (keep one replica serving through a drain)",
    )
    pr_many = next((d[1] for d in many if d[1]["kind"] == "PrometheusRule"), None)
    check(
        pr_many is not None
        and "kube_statefulset_status_replicas_ready"
        in pr_many["spec"]["groups"][0]["rules"][0]["expr"],
        "the up SLI must follow the rendered workload kind — a Deployment-shaped rule against a "
        "StatefulSet absent()s forever and can never fire",
    )
    # A bulk volume is shared external NFS: it must stay a standalone PVC even on a StatefulSet.
    mixed = render(
        {
            **base_av,
            "availability": {"replicas": 2},
            "storage": [
                {"name": "config", "size": "1Gi"},
                {"name": "media", "class": "bulk", "export": "/mnt/x", "size": "1Ti"},
            ],
        }
    )
    m_sts = next(d[1] for d in mixed if d[1]["kind"] == "StatefulSet")
    check(
        [t["metadata"]["name"] for t in m_sts["spec"]["volumeClaimTemplates"]] == ["config"],
        "only longhorn volumes become claim templates; a bulk share must not be per-replica",
    )
    check(
        any(
            d[1]["kind"] == "PersistentVolumeClaim"
            and d[1]["metadata"]["name"] == "av-media"
            for d in mixed
        ),
        "a bulk PVC must still be rendered standalone on a StatefulSet",
    )
    check(
        "media" in {v["name"] for v in m_sts["spec"]["template"]["spec"].get("volumes", [])},
        "a bulk volume must still appear in pod.volumes on a StatefulSet",
    )
    ns = next(d[1] for d in docs if d[1]["kind"] == "Namespace")
    check(
@@ -100,8 +100,7 @@ data:
  # platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
  # tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
  # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
-  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers;
+  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers),
  # plex-remote: PLEX_L4_VIP),
  # velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
  # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
  # UNSUBSTITUTED path would ship the literal string into the cluster.
@@ -176,19 +175,6 @@ data:
  # dev  -> 192.168.1.56  (base default; next free after crafty .55 in dev's .53-.60 pool)
  # prod -> 192.168.2.23  (next free after crafty .22 in prod's .20-.30 pool; see clusters/prod)
  PIHOLE_DNS_VIP: "192.168.1.56"             # PiHole: authoritative LAN DNS for ${BASE_DOMAIN}
  # Plex's remote-access listener: raw tcp/32400 for REMOTE Plex clients (apps/wave-0/plex-remote.yaml).
  # A FOURTH pinned L4 VIP, for the usual reason — the `shared` Gateway is an L7 HTTPS terminator and
  # cannot carry Plex's own TLS. Plex clients do not resolve `plex.${BASE_DOMAIN}`; they fetch this
  # server's address from plex.tv, so the Gateway's L7 path (which still serves the web UI, and was
  # measured working) cannot put the server on a client's connection list at all.
  #
  # PINNED, AND THE PIN IS LOAD-BEARING OFF-REPO: the SOHO router forwards WAN tcp/32400 to this
  # address by hand. A VIP taken from the pool head would silently move on a reschedule and break
  # remote access with nothing in-cluster looking wrong. See runbooks/plex-remote-access.md.
  #
  # dev  -> 192.168.1.57  (base default; next free after pihole-dns .56 in dev's .53-.60 pool)
  # prod -> 192.168.2.24  (next free after pihole-dns .23 in prod's .20-.30 pool; see clusters/prod)
  PLEX_L4_VIP: "192.168.1.57"                # Plex: tcp/32400 remote access (NOT the web UI, which is L7)
  # The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
  # peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable
@@ -476,26 +476,6 @@ patches:
      - op: replace
        path: /data/PIHOLE_DNS_VIP
        value: "192.168.2.23"
      # Plex's remote-access listener VIP — raw tcp/32400 for REMOTE Plex clients
      # (apps/wave-0/plex-remote.yaml). 192.168.2.24 is the NEXT FREE address in the prod pool
      # 192.168.2.20-.30, verified live 2026-10-03: the CiliumLoadBalancerIPPool `default-pool`
      # reported IPsTotal 11 / IPsUsed 4 / IPsAvailable 7, and the only four allocations are .20
      # (cilium-gateway-shared), .21 (unifi-l4-tcp/-udp), .22 (crafty-l4-tcp/-udp) and .23
      # (pihole-dns-tcp/-udp). It is inside the pool block the operator already asserted sits outside
      # the SOHO router's DHCP scope, and it is not a node address (.10/.11/.12) or the
      # control-plane VIP (.9).
      #
      # UNLIKE .23, THIS ONE IS ALSO A WAN PORT-FORWARD TARGET. The router forwards WAN tcp/32400
      # here, by hand, off-repo — so changing this value is a two-place edit and the router is the
      # half that CI cannot check. runbooks/plex-remote-access.md is the procedure.
      #
      # NOT ARP-PROBED. .23 was verified free with `arping` from a LAN host before it was claimed;
      # for .24 only the in-cluster pool accounting above was checked, so an off-cluster squatter on
      # this address would not have been caught. Probe before trusting it if remote access comes up
      # with a conflict.
      - op: replace
        path: /data/PLEX_L4_VIP
        value: "192.168.2.24"
      # The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns —
      # NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path.
      # It scopes the ingress allow for PiHole's DNS listener to LAN clients only.
@@ -0,0 +1,76 @@
 ---
 date: 2026-10-03
 branch: fix/cilium-cgroup-hostroot
 title: Cilium socket-LB never covered host processes — cgroup.hostRoot pointed at an unmounted path
 ---
 ## What changed
 `cgroup.hostRoot` changes from `/run/cilium/cgroupv2` to `/sys/fs/cgroup` in all three parity
 copies: `platform/cilium/helmrelease.yaml` (authoritative), `platform/cilium/values.yaml`
 (mirror) and `infra/ansible/roles/cilium/templates/cilium-values.yaml.j2` (bootstrap).
 `autoMount` stays `false`.
 **The defect.** With `autoMount: false`, the Cilium chart only bind-mounts whatever exists at
 `hostRoot`, and expects the operator to have mounted cgroup2 there (cilium 1.20.1 values.yaml).
 The repo's rationale was "k3s provides the cgroupv2 mount". That is true only for
 `/sys/fs/cgroup`. Nothing mounts cgroup2 at `/run/cilium/cgroupv2` on these nodes; it is an
 empty `DirectoryOrCreate` hostPath. So the agent mounted cgroup2 there itself, from inside its
 own private cgroup namespace, and that mount's root is the agent container's own leaf cgroup.
 The socket-LB programs (`cil_sock4_connect` etc.) were attached to it and covered only the
 agent.
 **The consequence.** Every other host-netns process got no Service/LB translation:
 - Service ClusterIPs were unreachable from every node's host.
 - The node holding the Gateway's L2 lease could not reach the Gateway VIP (`no route to host`).
  Other nodes reached it only because their packets crossed the wire to the lease holder, where
  the tc datapath handles them.
 - k3s-prod-1 holds both the Gateway lease and the kube-vip API VIP. Its apiserver could not reach
  the OIDC issuer, logging `oidc: authenticator not initialized` since at least 2026-09-12
  (899 occurrences by 2026-10-03). kubectl SSO through the VIP therefore failed, and in-cluster
  OIDC clients failed on whichever calls landed on that apiserver.
 - Webhooks and aggregated APIs were unaffected only because k3s runs
  `--enable-aggregator-routing=true` (endpoint IPs, not ClusterIPs).
 Cilium's own source (v1.20.1 `bpf/bpf_sock.c`, `__sock4_xlate_fwd`) handles exactly this case
 when the hook does run: a host-netns connect to an L7-LB Service (the Gateway) is rewritten to
 `127.0.0.1:<l7_lb_proxy_port>`, the local Envoy listener. The fix is to make the hook cover the
 host.
 ## What I verified
 - `cilium-dbg status --verbose` on k3s-prod-1: `Socket LB: Enabled`, `Coverage: Hostns-only`.
  The live cilium-config shows `bpf-lb-sock=false`, but socket LB is implied by kube-proxy
  replacement. The Gateway Service frontends `192.168.2.20:{80,443}` and
  `10.43.225.79:{80,443}` map to `127.0.0.1:14337`.
 - Host: `findmnt /run/cilium/cgroupv2` is empty on all three nodes, and `/sys/fs/cgroup` is
  `cgroup2`.
 - In the agent: `/run/cilium/cgroupv2` is a `cgroup2 none` mount, the agent's own cgroup reads
  `0::/`, the mount has 0 child cgroups, and `bpftool cgroup show /run/cilium/cgroupv2` lists all
  `cil_sock*` programs. k3s runs in `0::/system.slice/k3s.service`, and the agent container in
  `/kubepods.slice/…/cri-containerd-3ad04cde….scope`.
 - **The decisive test**, run on k3s-prod-1 in the same netns: raw TCP connect to
  `192.168.2.20:443`, `10.43.225.79:443` and `10.43.59.141:8080` CONNECTED from inside the agent
  container (covered cgroup) and FAILED from the host (`No route to host` / timeout).
 - Blast radius: host-network pods are cilium, cilium-envoy, cilium-operator (API via
  `192.168.2.9:6443`, not a Service), kube-vip (`127.0.0.1:6443`, `svc_enable=false`) and
  node-exporter (no outbound). Nothing depends on host Service translation being absent; the
  change only makes Service/LB addresses reachable from the host where they currently fail.
 - `helm template cilium/cilium --version 1.20.1` with the new values, in the toolbox: the
  DaemonSet bind-mounts hostPath `/sys/fs/cgroup` into `cilium-agent` and `clean-cilium-state`,
  the cilium-config shows `cgroup-root: "/sys/fs/cgroup"`, and the init containers are unchanged
  (no `mount-cgroup`).
 - Toolbox gate: see the PR description.
 ## What I did NOT verify
 - **The rollout itself.** After merge, the agent DaemonSet rolls one node at a time
  (`maxUnavailable: 1`). On each node, afterwards: host TCP connects to the three targets above
  must succeed. On k3s-prod-1, the apiserver's background OIDC init retry must stop logging
  `authenticator not initialized` with no k3s restart.
 - That the old programs, attached to the previous agent container's cgroup, disappear with it.
  Expected, because cgroup BPF attachments die with the cgroup, but not observed.
 - An end-to-end kubectl OIDC login (needs a browser).
 - No regression test was added. A natural one is an L2 conformance check that a host-netns
  connect to the Gateway VIP succeeds on the lease holder, but `conformance/**` is a sensitive
  path. Left as a follow-up.
@@ -1,129 +0,0 @@
 ---
 date: 2026-10-03
 branch: feat/plex-remote-access-l4
 title: Plex remote access — raw tcp/32400 via a pinned LoadBalancer VIP, not the L7 Gateway
 ---
 ## What changed
 Reported symptom: users cannot reach Plex from outside the LAN. **Two independent causes**, measured
 rather than inferred:
 1. **Plex advertised no remote connection at all.** Live `Preferences.xml` on prod:
   `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while
   `PublishServerOnPlexOnlineKey="1"` — publishing was on and there was nothing reachable to publish.
   Plex's automatic UPnP/NAT-PMP path cannot work from a pod on the Cilium overlay.
 2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` refused;
   `136.51.115.252:443` answered with a **self-signed router cert** (`C=ZZ, CN=%D`), not the
   Gateway's LE wildcard. The Plex Service was `ClusterIP` only — no off-cluster path to 32400 by
   any route.
 This branch fixes the in-cluster half. **Remote access does not work until the operator also does
 two manual steps** (`runbooks/plex-remote-access.md`): the WAN tcp/32400 port-forward, and ticking
 "manually specify public port" in the Plex UI.
 - **`apps/wave-0/plex-remote.yaml` (new, hand-written).** A `type=LoadBalancer` Service on tcp/32400
  pinned to `${PLEX_L4_VIP}` via `lbipam.cilium.io/ips`, plus a `CiliumNetworkPolicy`
  (`plex-allow-remote-32400`) admitting `world`/`cluster`/`host`/`remote-node` to 32400. Non-catalog
  because `schemas/servicedefinition.schema.json` is `additionalProperties: false` and has no field
  for a LoadBalancer Service or a pinned VIP — the same reason `apps/wave-0/pihole-dns.yaml`,
  `platform/crafty/service-l4.yaml` and `apps/vms/unifi/50-service-l4.yaml` are hand-written.
 - **`catalog/render.py`:** an `EXTRA_UNITS` entry (`plex-remote`, anchored to `plex`, `above: False`)
  so `--emit-waves` places the unit in plex's own wave.
 - **`clusters/base/cluster-vars.yaml`:** new `PLEX_L4_VIP` (dev default `192.168.1.57`), plus the
  substitution inventory comment updated to name `plex-remote: PLEX_L4_VIP`.
 - **`clusters/prod/kustomization.yaml`:** prod override `PLEX_L4_VIP: 192.168.2.24`.
 - **`apps/wave-0/kustomization.yaml`:** `plex-remote.yaml` added to `resources`.
 - **`runbooks/plex-remote-access.md` (new):** the two manual steps, the four-step verification, why
  no alert is possible, and the two documented warts below.
 **`ADVERTISE_IP` was deliberately NOT set.** Setting it to `https://plex.home.keithjackson.dev:443`
 was the alternative — it needs no new WAN hole and the mechanism was verified working on an
 already-claimed server. **The operator chose the native path (2026-10-03)** so Plex terminates its own
 TLS (`*.plex.direct`) and streams bypass the Envoy L7 hop. `catalog/plex.yaml`'s deferral note for
 `ADVERTISE_IP` remains accurate and untouched.
 **`externalTrafficPolicy: Cluster`, with a known cost.** `Local` would preserve the client IP, but
 Cilium's L2 announcer does not follow the backend pod — the wart that cost this LAN an 8-day
 Minecraft outage (see `platform/crafty/service-l4.yaml`). Consequence accepted and documented: SNAT
 puts the client in `192.168.0.0/16`, inside `ALLOWED_NETWORKS`, so **Plex classifies remote streams
 as LAN**. That is a metering effect (remote bandwidth limits will not bind), not an authorization one.
 **SENSITIVE PATHS — this PR touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK
 it.** It needs the operator's `/approve-override <reason>` posted as a PR **review** of type
 "Comment". I have deliberately not posted it: CLAUDE.md §1 forbids an agent clearing this on its own
 PR, and the agent account holds `owner`, so the prohibition is a rule, not a limit.
 ## What I verified
 All measurements against live prod, 2026-10-03, read-only.
 - **Plex's advertised state**, the finding this branch rests on: `customConnections=<ABSENT>`,
  `ManualPortMappingMode=<ABSENT>`, `ManualPortMappingPort=<ABSENT>`,
  `PublishServerOnPlexOnlineKey="1"`, `allowedNetworks="10.0.0.0/8,192.168.0.0/16"`. Read by
  attribute grep, never by dumping `Preferences.xml` (it holds `PlexOnlineToken`).
 - **The L7 Gateway path works and is NOT the problem.** Via the Gateway VIP,
  `GET https://plex.home.keithjackson.dev/identity` → `HTTP 200`, `claimed="1"`, real
  `machineIdentifier`. HTTPRoute `plex-32400`: `Accepted=True`, `ResolvedRefs=True`.
 - **Plex enforces its own auth**, which is what makes an unfiltered WAN port defensible:
  `/library/sections`, `/status/sessions`, `/:/prefs` all → **401** unauthenticated through the
  Gateway.
 - **The SNAT/allowedNetworks wart is real, not theoretical.** A tagged probe through the Gateway was
  logged by Plex as `[10.0.2.214:36040 (Allowed Network (WAN))]` — the Envoy pod IP, classified as an
  allowed network.
 - **WAN state:** `136.51.115.252:32400` → connection refused. `136.51.115.252:443` → self-signed
  cert `C=ZZ, ST=Somewhere, O=%D342e7e6b, CN=%D`. Public DNS resolves correctly:
  `plex.home.keithjackson.dev` → CNAME `wan.home.keithjackson.dev` → `136.51.115.252`.
 - **`192.168.2.24` is free in-cluster.** `CiliumLoadBalancerIPPool default-pool`: IPsTotal 11 /
  IPsUsed 4 / IPsAvailable 7; the only allocations are `.20` (cilium-gateway-shared), `.21`
  (unifi-l4-*), `.22` (crafty-l4-*), `.23` (pihole-dns-*). Not a node address (`.10/.11/.12`) and not
  the kube-vip control-plane VIP (`.9`).
 - **The manual-port-mapping step genuinely cannot be automated.** Grep of the running image's whole
  init surface (`/etc/cont-init.d`, `/plex-common.sh`, `/usr/local/bin`) for `ManualPortMapping` and
  `PLEX_PREFERENCE` → **NONE**.
 - **`ADVERTISE_IP` would have worked on a claimed server** (recorded because the runbook's
  "why not" rests on it): `setPref "customConnections"` at line 126 of `40-plex-first-run` sits
  **outside** the `PLEX_CLAIM`/`PlexOnlineToken` guard, and the `/.firstRunComplete` early-exit
  marker is on the **overlay** FS, not the `/config` PVC, so the script re-runs every pod start.
 - **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
  `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
  **pass**. The rc 3 is two environmental skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`)
  and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3, not as pass.**
 - **`render-golden` passing is the specific proof** that the `render.py` `EXTRA_UNITS` change, the new
  hand-written unit and the edited wave kustomization are mutually consistent.
 ## What I did NOT verify
 - **THE ACTUAL FIX. Nothing in this branch has been proven to make remote access work** — it cannot
  be, because both remaining steps are off-repo and manual (router forward, Plex UI toggle). This
  branch removes two blockers and adds none; it does not demonstrate an outcome.
 - **Outside-in reachability, at all.** Every probe above was taken from inside the LAN, so the
  `:443` self-signed cert strictly proves only that there is **no NAT hairpin** — it does **not**
  prove outside-in failure. A router cert on WAN:443 is suggestive (its management UI would shadow a
  forward), but unproven. **This repo has no off-LAN vantage point.** Test 3 in the runbook
  (app.plex.tv from a cellular connection) is the only honest check.
 - **`192.168.2.24` was never ARP-probed off-cluster.** Only in-cluster pool accounting was checked.
  `.23` was `arping`-probed before it was claimed; `.24` was not, so an off-cluster squatter would
  not have been caught.
 - **The LoadBalancer has never been reconciled.** `plex-l4-tcp` does not exist in prod yet, so
  LB-IPAM has not actually assigned the pinned VIP, L2 announcement has not been observed for it, and
  the new CiliumNetworkPolicy has never admitted a real packet. Whether `world`/`host`/`cluster`/
  `remote-node` is the correct entity set **under this SNAT path** is reasoned from
  `platform/crafty/networkpolicy.yaml`'s working game-port rule, **not measured**.
 - **Whether Plex's remote path needs anything beyond the manual port tick.** If plex.tv still lists
  no remote connection after both manual steps, this branch is necessary-but-insufficient.
 - **The CI approval-policy outcome.** The local stage reported green, but that is an offline N/A, not
  a verdict: this PR touches `clusters/**`, so CI will block it pending an override.
 - **PRE-EXISTING DRIFT I FOUND AND DELIBERATELY LEFT ALONE — worth its own follow-up.** Running the
  documented `python3 catalog/render.py --emit-waves` regenerates the wave kustomizations and wants to
  add **`vm-demo.yaml`** to wave 0 and **`keycloak.yaml`** to wave 1. Both are listed as
  intentionally not onboarded in `ci/render-golden.sh`
  (`RENDER_GOLDEN_NOT_ONBOARDED="keycloak vm-demo"`); their manifests are tracked in `apps/` but
  deliberately absent from the `resources` lists so Flux does not apply them. `--emit-waves` places
  any file present on disk and has **no awareness of that exclusion**, while `render-golden.sh` uses
  `--wave-layout` and compares rendered manifests only — it never checks kustomization `resources`,
  which is why this has survived unnoticed. **So the header "DO NOT hand-edit: re-run
  `render.py --emit-waves`" is currently unsafe advice**: following it onboards an IdP and a KubeVirt
  VM. I therefore restored both kustomizations and hand-added only my one line. I did not fix
  `--emit-waves`, and I have not determined how long the drift has existed or whether either service
  is *meant* to be onboarded.
@@ -1,133 +0,0 @@
 ---
 date: 2026-10-04
 branch: feat/pihole-declare-state
 title: PiHole's operator-set state is now declared in git — the prerequisite for running more than one replica
 ---
 ## What changed
 Phase 1 of making LAN DNS survive a node failure. The operator asked for multi-replica PiHole;
 measurement showed that cannot be done safely first, because PiHole's important state was not
 reproducible from git. A second replica with a fresh PVC would have come up **serving DNS and
 blocking nothing, with a different admin password** — answers varying by which pod replied. This
 branch closes that; the replica work is phase 2 and is deliberately NOT here.
 **The undeclared-state inventory, measured not guessed.** `pihole.toml` marks every non-default
 setting `### CHANGED`, and those that came from the environment `### CHANGED (env)`. The complete
 live list on 2026-10-04:
 | Setting | Status |
 |---|---|
 | `dns.upstreams`, `dns.listeningMode`, `dns.rateLimit.count`, `misc.dnsmasq_lines` | `### CHANGED (env)` — already declared in `catalog/pihole.yaml` |
 | `files.macvendor` | `### CHANGED` — set by the image itself (container path), not operator state |
 | `webserver.api.pwhash` | **`### CHANGED` — set out of band. Declared NOWHERE.** |
 So exactly **one** operator-set value lived only on the PVC. Separately, the **adlists** were also
 undeclared: `gravity.db` held one list (StevenBlack's hosts) and 72,234 gravity domains, with nothing
 in this repo naming it.
 - **`apps/wave-0/pihole-bootstrap-secret.sops.yaml` (new, SOPS/age).** Secret `pihole-bootstrap`,
  key `webui-password`. Named to the `<svc>-bootstrap-secret.sops.yaml` convention so
  `render.py --emit-waves` places it with no renderer change. The value is a 32-char random password
  generated at authoring time and **never printed to a transcript**.
 - **`catalog/pihole.yaml`:** `FTLCONF_webserver_api_password` sourced from that Secret.
  `webserver.api.password` takes CLEARTEXT and FTL hashes it itself — verified in the running image,
  where `pihole setpassword` *is* `setFTLConfigValue "webserver.api.password"`, commented upstream
  "pihole-FTL will automatically hash the password". The PVC's `pwhash` becomes a derived artefact.
 - **`catalog/pihole.yaml`:** a `lifecycle.postStart` hook that **reconciles** the declared adlist set
  into `gravity.db` on every container start — adds missing, re-enables disabled, and **disables**
  (never deletes) anything enabled but undeclared, running `pihole -g` only when something moved.
  Same level-triggered idiom, and the same reasoning, as `catalog/qbittorrent.yaml`'s postStart.
 - **`apps/wave-0/pihole.yaml`, `apps/wave-0/kustomization.yaml`:** re-rendered / secret registered.
 **Why a postStart hook and not a file.** PiHole v6 keeps adlists as DB rows. `/etc/pihole/adlists.list`
 still exists in `gravity.sh` but is **migration-only**: its import sits inside `migrate_to_database()`
 behind `if [ ! -e "$gravityDBfile" ]`, so it fires only while *creating* a database. A ConfigMap
 mounted there would be silently ignored on every pod this cluster runs today. There is also no
 `pihole adlist` CLI subcommand, so SQL is the actual interface; FTL ships an embedded shell
 (`pihole-FTL sqlite3`) so nothing had to be added to the image.
 **The password is kept rather than set empty, deliberately.** The admin UI is already behind the
 oauth2-proxy sidecar (`--allowed-group=platform-admins`), so an empty password would have been
 simpler. But the `pihole` Service still publishes **port 80** (PiHole direct, bypassing the proxy on
 `:4180`) and only the CiliumNetworkPolicy closes it — and this repo has a documented default-deny
 drift history. An unauthenticated admin UI that can rewrite LAN DNS should not rest on one policy
 staying correct. Narrowing that Service port is follow-up, not done here.
 **SENSITIVE PATHS — this PR adds a `**/*.sops.yaml`, so `ci/approval-policy.sh` will BLOCK it.** It
 needs the operator's `/approve-override <reason>` as a PR **review** of type "Comment". Not posted by
 me: CLAUDE.md section 1 forbids an agent clearing this on its own PR.
 ## What I verified
 Live prod, 2026-10-04, read-only except where noted.
 - **The undeclared-state inventory above** — read from `pihole.toml`'s own `### CHANGED` markers, with
  the password value masked. This is the measurement the whole branch rests on.
 - **A fresh `gravity.db` seeds NO adlists.** The image's own schema template
  `/etc/.pihole/advanced/Templates/gravity.db.sql` creates the `adlist` table and contains no
  `INSERT INTO adlist` at all — grepped in the running image. Confirmed empirically below (0 rows).
  This is why the adlist declaration is load-bearing and not cosmetic.
 - **The adlists.list path really is migration-only** — read the guard in `/opt/pihole/gravity.sh`
  (`migrate_to_database()`, `if [ ! -e "$gravityDBfile" ]`).
 - **`webserver.api.password` is a real settable key** — `pihole-FTL --config webserver.api.password`
  returns a masked value, and `/usr/local/bin/pihole`'s `setpassword` sets exactly that key.
 - **THE RENDERED HOOK ACTUALLY WORKS — all four paths, extracted from `apps/wave-0/pihole.yaml` and
  run in the live pod against THROWAWAY COPIES of `gravity.db` in `/tmp`** (production `gravity.db`
  was never written, and `pihole -g` was stubbed so no 70k-domain download could start):
  1. *Steady state* → `already matches the declared set - nothing to do`. No writes, no rebuild.
  2. *True fresh DB built from the image's schema template* → 0 adlists before; hook added the
     declared list and triggered the rebuild. **This is the fresh-PVC/new-replica case.**
  3. *An undeclared list enabled* → disabled it (`enabled=0`), left the declared one alone, and did
     not delete the row.
  4. *Declared list present but disabled* → re-enabled it.
 - **Required tools exist in the image:** `bash`, `expr`, `grep`, `printf`, `sleep`, `pihole-FTL`.
 - **The Secret is genuinely encrypted** to the repo's age recipient; `stringData.webui-password`
  starts `ENC[`. The gate's `no-plaintext-secrets (R-SEC-01)` L0 check passes.
 - **Schema:** `check-jsonschema` against `schemas/servicedefinition.schema.json` passes.
 - **Render determinism:** the canonical
  `render.py --all catalog -o apps --wave-layout --no-image-automation` changed **only**
  `apps/wave-0/pihole.yaml`. `render-golden` passes, so the hook's text also satisfies
  `assert_substitution_safe` (no braced dollar expansion — the Flux envsubst contract).
 - **Toolbox gate:** rc **3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
  `render-golden`, `l1`, `approval-policy` all **pass**. The rc 3 is the two usual environmental
  skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`) and `conformance` L2/L3
  (`CLUSTER_AVAILABLE` unset). **Not green; reported as rc 3.**
 ## What I did NOT verify
 - **Nothing here has been reconciled.** Flux has not applied any of it, so the password has never
  been set by this mechanism and the hook has never run during a real container start. Everything
  above is the rendered artifact tested out-of-band, plus the image's own source.
 - **THE CURRENT ADMIN PASSWORD WILL STOP WORKING, and that is the intent.** FTL re-hashes
  `webserver.api.password` on start, overwriting the existing `pwhash`. Whoever knows the present
  password (nobody, which is why this started) loses it; the declared one replaces it. I have not
  observed that replacement happen.
 - **Cold-start cost is unmeasured and could matter.** `postStart` must complete before the container
  is reported Running. On a fresh PVC the hook waits for `gravity.db` (bounded, up to 180s) and then
  runs a real `pihole -g` (72k domains, network downloads). `apps-wave-0` is `wait: true` and
  `apps-wave-1` `dependsOn` it, so a slow cold start **delays the app tier**. The steady-state path
  exits in about a second (scenario 1), so this is a rebuild/first-boot cost only — but it is not
  measured, and the stub in testing means `pihole -g`'s real duration here is unknown.
 - **Whether `pihole -g` succeeds from inside the hook's environment.** Its failure is caught and the
  existing `gravity.db` retained, so a failure degrades rather than breaks — untested live.
 - **Concurrency.** Nothing serialises the hook against FTL's own scheduled gravity update. SQLite
  handles the locking, and the hook's writes are three small statements, but no interleaving was
  exercised.
 - **MULTI-REPLICA — PHASE 2, NOT STARTED.** This branch only removes the blocker. The remaining work,
  and it is substantial: a `replicas`/placement field on the ServiceDefinition schema (currently
  `additionalProperties: false` with no such field), `render.py` emitting a StatefulSet with
  `volumeClaimTemplates` (the config PVC is **ReadWriteOnce**, so a 2-replica Deployment cannot
  work), pod anti-affinity, a PodDisruptionBudget (there is none today), re-rendered goldens, and
  `conformance/**` updates — the last making phase 2 sensitive-path as well. The PVC rename a
  StatefulSet forces will discard the existing `gravity.db`, which is only safe **because** of this
  branch, and should be done only after this one is confirmed working in-cluster.
 - **The real risk the operator named is NOT fixed by phase 2 either.** Replicas address *node*
  failure. LAN DNS still depends on the cluster as a whole — etcd quorum, Cilium, Longhorn, a bad
  reconcile. Quantified on 2026-10-04: with `tolerationSeconds=300` plus node-NotReady detection
  plus a Longhorn RWO force-detach, a failure of the node running the pod is roughly **6-10 minutes
  of no LAN DNS**; a failure of the node holding the `pihole-dns-*` L2 lease (currently a
  *different* node — lease on k3s-prod-2, pod on k3s-prod-1) is tens of seconds. An out-of-cluster
  secondary resolver carrying the same `address=` lines is the only option that removes the
  dependency, and it was considered and not chosen.
 - **Port 80 on the `pihole` Service is still published** and still closed only by policy. Not narrowed.
@@ -1,145 +0,0 @@
 ---
 date: 2026-10-04
 branch: feat/pihole-multi-replica
 title: PiHole runs two replicas on different nodes — and its config PVC was mounted where PiHole never writes
 ---
 ## What changed
 Phase 2 of making LAN DNS survive a node failure (phase 1:
 `decisions/2026-10-04-pihole-declare-state.md`). PiHole was one pod on one node, which for a
 whole-house resolver cost more than "until it reschedules" — measured 2026-10-04:
 | Failure | Recovery |
 |---|---|
 | the node running the pod dies | ~40s to mark NotReady + `tolerationSeconds: 300` before eviction + reschedule + a Longhorn RWO force-detach → **roughly 6-10 minutes with no LAN DNS** |
 | the node holding the `pihole-dns-*` L2 lease dies | Cilium re-elects an announcer, tens of seconds |
 Those were **different** nodes (lease on k3s-prod-2, pod on k3s-prod-1), so two of three nodes were
 single points of failure for LAN DNS with wildly different blast radii.
 **A new schema field, because this is intent and not a primitive.** `availability.replicas`
 (schema **1.10.0**, mechanism in `schemas/renderer-contract.md` `### availability`). `1` or absent
 renders a Deployment exactly as before — every pre-existing golden is byte-identical, verified. `>1`
 renders, together:
 - a **StatefulSet** instead of a Deployment. This is forced, not preferred: a longhorn PVC is
  **ReadWriteOnce**, and one RWO claim cannot attach on two nodes, so a 2-replica Deployment does not
  degrade — the second pod never schedules.
 - longhorn storage as **per-replica `volumeClaimTemplates`** (`config-pihole-0`, `config-pihole-1`);
  the standalone PVC is no longer emitted, since it would be an orphan nothing mounts. A `bulk`
  entry stays a standalone PVC — it is a shared static NFS export, not per-replica state.
 - **required** `podAntiAffinity` on `kubernetes.io/hostname`. `preferred` would let both replicas
  land on one node and deliver none of the intended tolerance while reporting healthy.
 - `podManagementPolicy: Parallel`, so a cold start costs the max of the replicas' first-boot work
  rather than the sum — `apps-wave-0` has `timeout: 5m`.
 - a **PodDisruptionBudget**, `minAvailable: 1`, so a node drain cannot take the last replica.
 - `observability.livenessWorkload.kind` defaulting to `StatefulSet`, so the `up` SLI reads
  `kube_statefulset_status_replicas_ready`. A Deployment-shaped rule against a StatefulSet
  `absent()`s forever — an alert that can never fire.
 `catalog/pihole.yaml` sets `replicas: 2`, not 3: two tolerates any single node loss and leaves a
 spare node for the evicted replica under the hard anti-affinity. Three would pin one per node with no
 spare, so a single node failure would leave a replica permanently `Pending` and — `apps-wave-0` being
 `wait: true` — stall the app tier for as long as the node was down.
 ### The bug found on the way, and fixed here
 **PiHole's config PVC was mounted at a path PiHole never writes, so it held nothing and all of its
 state was ephemeral.** With no explicit `mountPath` the renderer defaults to `/data/<name>`; PiHole
 keeps everything in `/etc/pihole`. Measured on live prod 2026-10-04: `/etc/pihole` (7.1 M —
 `gravity.db`, `pihole.toml`, the query DB) was on the **overlay** filesystem, while the 2 Gi Longhorn
 volume sat at `/data/config` holding `lost+found` and 24 K of nothing. Consequences, all live until
 now:
 - every restart silently discarded PiHole's entire state and re-downloaded 72 k gravity domains;
 - `backup.policy: daily-offcluster` has been faithfully backing up an **empty** volume, which is
  worse than no backup because it reports success;
 - it also means phase 1 was understated: that state was not merely undeclared, it was **ephemeral**.
 Same class of bug `catalog/plex.yaml` documents for its own `config` entry. Fixed in this branch
 rather than its own because with the wrong path the new per-replica volumes would be two 2 Gi
 Longhorn volumes (×3 Longhorn replicas each) holding only `lost+found` while all state stayed
 ephemeral — paying for persistence and getting none. It also removes this change's main operational
 risk: with `gravity.db` persisted, a restart is cheap instead of re-downloading 72 k domains inside a
 wave with a 5 m timeout.
 ## What I verified
 - **Back-compatibility, the thing most likely to break quietly:** after the renderer change and
  before `catalog/pihole.yaml` declared anything, the canonical
  `render.py --all catalog -o apps --wave-layout --no-image-automation` produced **zero diff** across
  the whole `apps/` tree. `render-golden` passes.
 - **The new behaviour is pinned by ~20 new `render.py --selftest` assertions** (so `ci/render-golden.sh`
  hard-fails on a regression, and no `conformance/**` change was needed): `replicas: 1` still renders a
  Deployment + standalone PVC + no PDB; `replicas: 2` renders a StatefulSet and **no** standalone
  longhorn PVC, carries the replica count, omits `strategy`, sets `Parallel` and `serviceName`, names
  the claim template exactly as the mounted volume, keeps RWO/longhorn on it, does **not** duplicate
  that volume in `pod.volumes`, still mounts it on the container, renders required hostname
  anti-affinity, renders exactly one `minAvailable: 1` PDB, and switches the SLI to
  `kube_statefulset_status_replicas_ready`. A mixed longhorn+bulk case asserts the bulk share stays a
  standalone PVC and an ordinary pod volume.
 - **Rendered pihole output, read back:** `replicas: 2`, `serviceName: pihole`,
  `podManagementPolicy: Parallel`, no `strategy`, claim template `config` (RWO/longhorn/2Gi),
  anti-affinity `kubernetes.io/hostname`, `pod.volumes: []`, mount `config -> /etc/pihole`,
  PDB `minAvailable: 1`, SLI `kube_statefulset_status_replicas_ready`.
 - **The `/etc/pihole` mount is safe, evidenced from the image's own init rather than assumed:**
  `/usr/bin/start.sh` is PID 1 and runs as **uid 0**; its line 5 is
  `chown -R pihole:pihole /etc/pihole/ /var/log/pihole/`, so a fresh root-owned volume has its
  ownership fixed before anything else starts (**no `fsGroup` needed, and one would be inert**); it
  calls `migrate_gravity`, whose `migrate_to_database()` creates the gravity database when
  `[ ! -e "$gravityDBfile" ]`; FTL writes `pihole.toml` itself from the `FTLCONF_*` env; and the
  image's own default for `files.macvendor` was moved **out** of that directory, which is what a
  maintainer does to a path they expect to be a volume.
 - **The PVC really is unused:** `df` showed `/etc/pihole` on `overlay` and
  `/dev/longhorn/pvc-849edb86-...` on `/data/config`; `ls -A /data/config` → `lost+found` only.
 - **The gate caught a real regression and it is fixed.** The first run came back **rc 1**:
  `alert-rules-unit-tested` failed because `apps/wave-0/tests/app-slis.test.yaml` fed pihole a
  `kube_deployment_status_replicas_available` series and expected a `deployment: pihole` label. Three
  input series, the firing expectation and the file header were updated;
  `platform/monitoring/tests/run.sh apps/wave-0/tests/app-slis.test.yaml` → **SUCCESS** (6 rules).
 - **Schema:** `check-jsonschema` passes over all of `catalog/*.yaml`.
 - **Toolbox gate:** rc **3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
  `render-golden`, `l1`, `approval-policy` all **pass**. The rc 3 is the two environmental skips —
  `gitea-settings` (no local `SETTINGS_API_TOKEN`) and `conformance` L2/L3 (`CLUSTER_AVAILABLE`
  unset). **Not green; reported as rc 3.**
 - **No sensitive paths touched**, so unlike the previous two branches this one needs no
  `/approve-override` — confirmed against `infra/gitea/approval-policy.json`.
 ## What I did NOT verify
 - **Nothing has been reconciled.** No StatefulSet exists, no second replica has ever run, the
  anti-affinity has never been evaluated by the scheduler, the PDB has never been consulted, and the
  `/etc/pihole` mount has never been exercised on a real pod. Everything above is the rendered
  artifact plus the image's own source.
 - **THE ROLLOUT WILL BRIEFLY DROP LAN DNS, and I have not measured for how long.** `postStart` must
  complete before a container is reported Running, so a pod is not Ready — and the Service has no
  endpoints — until phase 1's adlist hook finishes. On this transition both replicas start with an
  empty `/etc/pihole`, so the hook finds a fresh `gravity.db` with zero adlists and runs a real
  `pihole -g` (72 k domains, network download) before either pod becomes Ready. **Do this at a quiet
  time.** Two mitigating facts: every pihole restart has *already* paid this cost (state was
  ephemeral), and after this change `gravity.db` persists so later restarts should not. If it turns
  out slow, backgrounding the gravity rebuild inside the hook would decouple readiness from it, at
  the cost of a window where DNS resolves but does not block — not done here.
 - **Whether `pihole -g` finishes inside `apps-wave-0`'s 5 m `timeout`.** `Parallel` was chosen to
  make this the max rather than the sum of two first boots, but the real duration is unmeasured (it
  was stubbed during phase 1's testing). If the wave times out, the Kustomization reports failed even
  though the pods may come up seconds later.
 - **The old PVC's deletion.** `apps-wave-0` has `prune: true`, so removing `pihole-config` from the
  rendered output should delete it. That destroys nothing (it holds `lost+found`), but I have not
  observed the prune, and I did not check whether Longhorn retains the backing volume afterwards.
 - **The transition overlap.** For a moment the pruned Deployment's pod and the new StatefulSet pods
  may both match the Service selector. Both are configured identically from git so this should be
  benign — unobserved.
 - **Longhorn capacity.** Two 2 Gi claims at 3 Longhorn replicas each is ~12 Gi of real storage where
  there was ~6 Gi. Not checked against free space.
 - **Whether two independent query databases are acceptable to the operator.** Stats and query history
  now split per replica, and the admin UI shows only the pod that answered. Adlists and password are
  identical (phase 1), so blocking and auth are consistent; the *reporting* is not.
 - **The residual risk is unchanged and is the larger one.** Replicas address node failure. LAN DNS
  still depends on the cluster — etcd quorum, Cilium, Longhorn, a bad reconcile — and only a resolver
  outside the cluster carrying the same `address=` lines removes that. Considered and not chosen
  (operator decision, 2026-10-04).
 - **`tolerationSeconds: 300` is untouched.** With two replicas the surviving one serves immediately, so
  the 5-minute eviction delay no longer gates recovery — but it still delays the *replacement* replica,
  and during that window there is no redundancy left.
@@ -51,11 +51,14 @@ hubble:
 l2announcements:
  enabled: true
-# k3s ships its own cgroup mount; point Cilium at it instead of auto-mounting.
+# Point Cilium at the host's real cgroup2 root (/sys/fs/cgroup, mounted by systemd) instead of
 # auto-mounting. NOT /run/cilium/cgroupv2: nothing mounts cgroup2 there, so the agent mounted its
 # OWN namespaced cgroup and socket-LB covered only itself — host processes (the apiserver) lost
 # Service/LB translation. [parity] platform/cilium/helmrelease.yaml explains (2026-10-03).
 cgroup:
  autoMount:
    enabled: false
-  hostRoot: /run/cilium/cgroupv2
+  hostRoot: /sys/fs/cgroup
 # Roll the cilium agent DaemonSet ONE node at a time. [parity] must match the
 # platform/cilium HelmRelease. The cilium 1.20.1 chart defaults this to 2, which on a
@@ -106,14 +106,33 @@ spec:
    # CiliumL2AnnouncementPolicy (reserved 10.73.0.240-.250) authored by WS-NET in Phase 2.
    l2announcements:
      enabled: true
-    # --- cgroup: k3s ships its own cgroup mount; point Cilium at it, don't auto-mount ---
+    # --- cgroup: point Cilium at the HOST's cgroup2 root, which systemd mounts at /sys/fs/cgroup ---
-    # [parity] MUST equal Ansible cgroup block. Not in the deliverable's headline list, but it IS
+    # [parity] MUST equal Ansible cgroup block (roles/cilium/templates/cilium-values.yaml.j2).
-    # part of the working CNI config on k3s: reconciling without it lets Cilium try to auto-mount
+    #
-    # cgroupv2 and break on adopt. Kept in parity for a safe Phase-2 adoption.
+    # FIXED 2026-10-03 — hostRoot WAS /run/cilium/cgroupv2, AND THAT SILENTLY BROKE SOCKET-LB FOR
    # EVERY HOST PROCESS. With autoMount disabled, the chart only bind-mounts whatever already exists
    # at hostRoot ("it's expected that users have mounted cgroup2 filesystem at the specified
    # cgroup.hostRoot" — cilium 1.20.1 values.yaml). Nothing ever mounted cgroup2 at
    # /run/cilium/cgroupv2 on these nodes (it is an empty DirectoryOrCreate hostPath), so the agent
    # mounted one ITSELF from inside its own private cgroup namespace. That mount's root is the
    # agent container's OWN leaf cgroup, so cil_sock4_connect & co. were attached there and covered
    # only the agent. Every other host-netns process (the k3s apiserver in
    # /system.slice/k3s.service, curl over SSH) was untranslated: Service ClusterIPs timed out from
    # every node, and the node holding the Gateway's L2 lease could not reach the Gateway VIP at all
    # ("no route to host"). That is why k3s-prod-1's apiserver logged `oidc: authenticator not
    # initialized` from 2026-09-12 on — kubectl SSO through the kube-vip VIP failed outright.
    # Measured on k3s-prod-1 before this change: TCP connect to 192.168.2.20:443, 10.43.225.79:443
    # and 10.43.59.141:8080 SUCCEEDED from inside the agent container (the covered cgroup) and
    # FAILED from the host, same node, same netns. Full write-up:
    # decisions/2026-10-03-cilium-cgroup-hostroot.md.
    #
    # /sys/fs/cgroup IS a cgroup2 mount in the host's root cgroup namespace on every node (verified
    # with findmnt on all three), so bind-mounting it gives the agent the real host root. autoMount
    # stays off because the mount it would create already exists.
    cgroup:
      autoMount:
        enabled: false
-      hostRoot: /run/cilium/cgroupv2
+      hostRoot: /sys/fs/cgroup
    # --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident ---
    # [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1.
    # The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate.
@@ -41,11 +41,13 @@ hubble:
 # [parity] ingress VIP announcement on the flat LAN (docs/09, OQ-1)
 l2announcements:
  enabled: true
-# [parity] k3s provides the cgroupv2 mount; do not auto-mount
+# [parity] the host's cgroup2 root is /sys/fs/cgroup (systemd mounts it); do not auto-mount.
 # NOT /run/cilium/cgroupv2: nothing mounts cgroup2 there, so socket-LB attached to the agent's own
 # cgroup and host processes lost Service/LB translation (2026-10-03 — see helmrelease.yaml).
 cgroup:
  autoMount:
    enabled: false
-  hostRoot: /run/cilium/cgroupv2
+  hostRoot: /sys/fs/cgroup
 # [parity] roll the agent DaemonSet ONE node at a time. Overrides the cilium 1.20.1 chart
 # default of 2, which on 3 nodes lets a single bad render break 2/3 agents at once — as the
 # 2026-09-01 incident did (k8sServiceHost: "" → the `config` init container fell back to the
@@ -1,148 +0,0 @@
 # plex-remote-access — getting Plex reachable from outside the LAN, and proving it
 **Status: the in-cluster half is shipped; the last mile is TWO MANUAL STEPS and remote access does
 NOT work until both are done.** Neither can live in this repo.
 **Alert:** none, and none is possible from in-cluster signals — see "Why no alert" below. This is an
 **operator procedure**, not an incident page.
 **Files:** `apps/wave-0/plex-remote.yaml` (the tcp/32400 LoadBalancer + its CiliumNetworkPolicy),
 `clusters/base/cluster-vars.yaml` + `clusters/prod/kustomization.yaml` (`PLEX_L4_VIP`),
 `catalog/render.py` (the `EXTRA_UNITS` entry that places the unit), `catalog/plex.yaml` (Plex's own
 env — note `ADVERTISE_IP` is deliberately NOT set; see "Why not the Gateway URL").
 **Related:** `runbooks/plex-quicksync.md` and `runbooks/plex-claim.md` — the other two Plex
 procedures with a manual last mile.
 ## What was actually broken (measured 2026-10-03)
 Plex was serving correctly and was reachable on the LAN. Two independent things stopped remote
 clients:
 1. **Plex advertised no remote connection at all.** In the live `Preferences.xml`:
   `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while
   `PublishServerOnPlexOnlineKey="1"`. The server was trying to publish itself to plex.tv and had
   nothing reachable to publish. Plex's automatic path (UPnP/NAT-PMP) cannot work here: the process
   lives in a pod on the Cilium overlay (`10.0.2.136`), it cannot speak UPnP to the SOHO router, and
   a mapping to the pod IP would be meaningless if it could.
 2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` → connection refused.
   `136.51.115.252:443` → a **self-signed router certificate** (`C=ZZ, CN=%D`), not the Gateway's
   Let's Encrypt wildcard.
 The Service was `ClusterIP` only, so port 32400 had no path off-cluster by any route.
 **Caveat on that second measurement, because it is easy to over-read:** it was taken from *inside*
 the LAN, so strictly it proves there is no NAT hairpin (reflection). It does not prove outside-in
 failure. But a router certificate answering 443 means the router's own management UI is bound to the
 WAN port, and that *would* shadow a port-forward for genuine outside traffic too. **Verify from a
 real outside vantage point** (phone on cellular, or an external port checker) before concluding
 either way — this repo has no such vantage point and must not pretend otherwise.
 ## Step 1 — forward WAN tcp/32400 on the router (operator, off-repo)
    WAN tcp/32400  ->  192.168.2.24  (${PLEX_L4_VIP})  port 32400
 `192.168.2.24` is a pinned Cilium LB VIP, ARP-announced on the flat LAN by the existing
 `CiliumL2AnnouncementPolicy default-l2`. It is **pinned on purpose**: this address is typed into the
 router by hand, so it must never move on a reschedule.
 While in the router, **check what else is bound to the WAN interface.** If remote management is
 listening on 443 (see the certificate measurement above), move it or disable it — otherwise the
 Gateway's 80/443 forward is shadowed and the Plex *web UI* over the internet stays broken even
 though remote *streaming* works. These are two separate paths and they fail independently.
 Confirm the VIP is actually answering on the LAN first, so a failure later is unambiguous:
    ./infra/devcontainer/dev.sh run bash -c \
      'kubectl --context k3s-prod -n plex get svc plex-l4-tcp -o wide'
 `EXTERNAL-IP` must read `192.168.2.24`. If it is `<pending>`, LB-IPAM could not assign — the address
 is outside the pool or already taken; see `clusters/prod/kustomization.yaml` for the pool accounting.
 **`192.168.2.24` was never ARP-probed off-cluster.** Only the in-cluster pool accounting was checked
 (IPsTotal 11 / IPsUsed 4 / IPsAvailable 7 on 2026-10-03). If the VIP behaves oddly, probe it from a
 LAN host — `arping 192.168.2.24` — the way `.23` was probed before it was claimed.
 ## Step 2 — tick "manually specify public port" in Plex (operator, UI only)
 **There is no GitOps path to this and no environment variable for it.** Measured in the running
 image on 2026-10-03: a grep of the entire init surface (`/etc/cont-init.d`, `/plex-common.sh`,
 `/usr/local/bin`) for `ManualPortMapping` and `PLEX_PREFERENCE` returned **NONE**. The image writes
 only six preferences, and this is not one of them. It is the same class of manual last mile as the
 `HardwareAcceleratedCodecs` toggle in `runbooks/plex-quicksync.md`.
 In the Plex web UI (`https://plex.home.keithjackson.dev`, on the LAN):
 **Settings → Remote Access**
 - Ensure **Enable Remote Access** is on. (`PublishServerOnPlexOnlineKey` was already `1`.)
 - Tick **Manually specify public port** and set it to **32400**.
 - **Apply**, then let the panel settle. It should report *"Fully accessible outside your network"*.
 Without this tick Plex waits for a NAT mapping it can never obtain and publishes no remote
 connection, so the port-forward in step 1 sits there unused.
 ## How to verify
 Do these in order; each one isolates a different half.
 **1. The port is open from outside.** From a phone on cellular, or any off-LAN host:
    curl -sk -o /dev/null -w '%{http_code}\n' https://136.51.115.252:32400/identity
 `200` means the forward and the policy are both good. A hang or refusal means step 1.
 **2. Plex itself believes it is reachable.** Settings → Remote Access must say *"Fully accessible
 outside your network"*. If it says *"not available outside your network"* while test 1 passes, step 2
 was not applied.
 **3. plex.tv is handing out a remote connection.** The decisive test, because it is what clients
 actually consume: sign in to [app.plex.tv](https://app.plex.tv) from an off-LAN network and confirm
 the server appears and plays. A client finds this server through plex.tv's connection list, never
 through DNS.
 **4. The policy is admitting the traffic.** If test 1 fails *after* the forward is confirmed, the
 CiliumNetworkPolicy is the next suspect:
    ./infra/devcontainer/dev.sh run bash -c \
      'kubectl --context k3s-prod -n plex get cnp plex-allow-remote-32400'
 See `runbooks/denied-flows.md` for reading actual drops.
 ## Why no alert
 Every in-cluster signal is green whether or not remote access works. The Service has endpoints, the
 pod is `Running`, the policy is applied, and `plex-down` watches
 `kube_deployment_status_replicas_available` — none of which can see a router with a missing
 port-forward or an unticked checkbox. An alert would need an off-LAN prober, which this cluster does
 not have. **Test 3 above is the only honest check, and it is manual.**
 ## Why not the Gateway URL (`ADVERTISE_IP`)
 The alternative was to leave Plex on `ClusterIP` and set
 `ADVERTISE_IP: https://plex.home.keithjackson.dev:443`, which writes Plex's `customConnections`
 preference and would have put the server on plex.tv's list via the existing L7 Gateway — no new WAN
 hole. That mechanism was verified to work on an already-claimed server (the `ADVERTISE_IP` block at
 line 126 of the image's `40-plex-first-run` sits **outside** the claim guard, and `/.firstRunComplete`
 lives on the overlay FS, not the `/config` PVC, so it re-runs every pod start), and the Gateway path
 itself was measured serving Plex correctly: `HTTP 200`, `claimed="1"`, with `401` on every protected
 endpoint.
 **The operator chose the native path anyway (2026-10-03), and the reasons are real:** Plex terminates
 its own TLS with its `*.plex.direct` certificate, and video streams bypass the Envoy L7 hop instead
 of being proxied through it. The cost is a new unfiltered WAN port — acceptable because Plex
 authenticates every caller itself. `ADVERTISE_IP` therefore stays unset in `catalog/plex.yaml`, and
 the deferral note there is still accurate.
 ## Known wart: remote clients are classified as LAN
 `apps/wave-0/plex-remote.yaml` uses `externalTrafficPolicy: Cluster`, which SNATs the client to the
 ingress node's address (`192.168.2.1x`). That falls inside the `192.168.0.0/16` entry of
 `ALLOWED_NETWORKS`, so **Plex treats every remote stream as local.** Measured on the Gateway path,
 which SNATs the same way: Plex logged `[10.0.2.214:36040 (Allowed Network (WAN))]`.
 This is a **metering** effect, not a security one — authorization is unchanged, as the 401s above
 show — but Plex's remote-quality and bandwidth limits will not bind on these streams.
 `Local` would preserve the client IP and fix it. **Do not switch to it.** Cilium's L2 announcer
 elects one lease holder per Service and does *not* follow the backend pod, so the VIP can be
 announced by a node with no Plex pod, black-holing the port while everything reports healthy. That
 exact change cost this LAN an 8-day Minecraft outage in September 2026 — the full write-up is at the
 top of `platform/crafty/service-l4.yaml`. Plex is a single-replica `Recreate` Deployment, so the
 node mismatch is the normal case here, not a rare one.
@@ -161,41 +161,6 @@ platform must render, and cites the requirement each mapping satisfies. Changing
 - **Requirements:** **R-STATE-01** (rebuild survival), **R-STATE-02** (Longhorn replication),
  **R-STATE-04** (bulk tier). (docs/04)
 ### `availability` (`replicas`) — workload kind, per-replica volumes, spreading, budget
 **Schema 1.10.0.** Availability INTENT. It names no Kubernetes object, no topology key and no
 budget — exactly like `hardware.gpu`, and for the same reason: the mechanism below should be
 replaceable without a catalog edit.
 - **Absent, or `replicas: 1`** → a **Deployment**, standalone PVC(s), `strategy: Recreate` when there
  is storage, no PodDisruptionBudget. Byte-identical to pre-1.10.0 output, so every existing app is
  untouched.
 - **`replicas: N > 1`** → all of the following, together:
  | Rendered | Why |
  |---|---|
  | **StatefulSet** instead of Deployment | A longhorn PVC is **ReadWriteOnce**. One RWO claim cannot attach on two nodes, so a multi-replica Deployment does not degrade — the second pod never schedules. |
  | longhorn storage becomes **`volumeClaimTemplates`** | Gives each replica its own claim (`<vol>-<svc>-<ordinal>`, e.g. `config-pihole-0`). The template's `metadata.name` **must** equal the pod volume name the container mounts. The standalone PVC is then **not** emitted — it would be an orphan nothing mounts. |
  | `bulk` storage stays a **standalone PVC** | A bulk entry is a static, pre-bound NFS PV over data that already exists outside the cluster and is shared read-only. Per-replica copies are neither possible nor wanted. |
  | **required** `podAntiAffinity` on `kubernetes.io/hostname` | `preferred` would let every replica land on one node and deliver none of the intended node-failure tolerance, while every object still reported healthy. |
  | `podManagementPolicy: Parallel` | `OrderedReady` serialises first-boot work across replicas; `apps-wave-0` has `timeout: 5m`. There is no ordering requirement between replicas to protect. |
  | **PodDisruptionBudget**, `minAvailable: 1` | Stops a *voluntary* disruption (node drain, upgrade) taking the last replica. It does nothing about a node dying — that is what the replica count and anti-affinity are for. Never rendered at `replicas: 1`, where `minAvailable: 1` would block every drain of that node forever. |
  | `observability.livenessWorkload.kind` defaults to **`StatefulSet`** | The `up` SLI reads kube-state-metrics by workload kind. A Deployment-shaped rule against a StatefulSet `absent()`s forever — an alert that can never fire. Deriving the default means bumping the replica count cannot leave a stale rule behind. |
 **Two costs, both deliberate, neither hidden:**
 1. **N replicas need N schedulable nodes.** On a 3-node cluster, losing two nodes leaves a replica
   `Pending`; since `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, that **stalls the
   app tier**. That coincides with having lost etcd quorum on a 3-node control plane, so nothing is
   reconciling at that point anyway — the stall is a symptom, not a new failure mode.
 2. **Only declare `> 1` for a service whose on-disk state is reproducible from git.** A replica with
   a fresh volume comes up serving whatever it can bootstrap itself and nothing else. For pihole that
   prerequisite was done first — the admin password and the adlists were declared in
   `decisions/2026-10-04-pihole-declare-state.md` — because otherwise a second replica would have
   served DNS while blocking nothing, with a different password.
 **Requirements:** R-PLAT-02 (the workload), R-STATE-01/02 (per-replica replicated volumes),
 R-OBS-01 (the SLI must follow the rendered kind).
 ### `hardware` (`gpu`) — host device capability
 - **The field is INTENT, never MECHANISM.** `hardware.gpu` is an **enum of capability classes**
  (today: `intel-quicksync`), not a device path, not a hostPath, not a resource request. The
@@ -2,7 +2,7 @@
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://home.keithjackson.dev/schemas/servicedefinition.schema.json",
  "title": "ServiceDefinition",
-  "description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.10.0 (adds optional `availability` for the container kind, one member: `availability.replicas`. It is AVAILABILITY INTENT, not a scheduling primitive -- it names no Kubernetes object, no topology key and no budget. 1 (the default) renders a Deployment exactly as before, so every pre-existing app stays byte-identical; >1 means the service must survive the loss of a single node, and the renderer satisfies that with a StatefulSet whose longhorn volumes become per-replica volumeClaimTemplates (a longhorn PVC is ReadWriteOnce and cannot attach on two nodes, so a multi-replica Deployment is UNSCHEDULABLE, not merely suboptimal), REQUIRED podAntiAffinity on kubernetes.io/hostname (preferred would let every replica land on one node and deliver none of the intended tolerance while reporting healthy), podManagementPolicy Parallel, a PodDisruptionBudget of minAvailable 1, and an `up` SLI that follows the rendered kind (a Deployment-shaped rule against a StatefulSet absent()s forever and can never fire). Added for pihole, where LAN DNS -- a critical network service -- was one pod on one node: measured 2026-10-04, losing the node running it meant roughly 6-10 minutes with no LAN resolution (tolerationSeconds 300 plus node-NotReady detection plus a Longhorn RWO force-detach). Deliberately NOT usable until a service's state is reproducible from git, because a replica with a fresh volume otherwise comes up serving but unconfigured -- for pihole that prerequisite was done first (decisions/2026-10-04-pihole-declare-state.md). Capped at 5: a count above the node count can never schedule under the anti-affinity this implies. Mechanism documented in schemas/renderer-contract.md `### availability`). 1.9.0 (adds optional `auth.apiRoutes` -- a per-service list of anchored path regexes the forward-auth oauth2-proxy sidecar answers with 401 instead of a 302 to Keycloak, rendered as one repeatable `--api-route` per entry. The `auth` block stays `additionalProperties: false`; this is one named, documented key, not an opened door. Without it there was NO per-app override hook at all, so the only way to answer an app's XHRs with 401 was a renderer-wide constant -- and the path shape is a property of the app: jackett's is `/api/` (33 x 302 vs 2 x 401 measured 2026-09-30, 4 of 6 POSTs answered by the proxy and never reaching the app), radarr's is `/signalr/` (232 x 302 measured 2026-10-01, every one a POST from a left-open browser tab that is never re-prompted), and pihole serves no `/api/` at all. Each entry must be anchored at `^/` and continue with literal path characters, because a too-broad regex turns a plain page load into a 401 and the service looks permanently broken to a logged-out user. Enforced on every gate run by ci/l0.sh's schema stage over catalog/*.yaml, by catalog/render.py refusing to render an unanchored entry, and by conformance/tests/l0-static/oauth2-proxy-api-routes-per-app)). 1.8.0 (adds optional `hardware` - a host HARDWARE CAPABILITY declaration for the container kind, currently one member: `hardware.gpu: intel-quicksync`, meaning the workload needs the host iGPU render device for hardware video transcode. It is an ENUM OF INTENT, not a device path: the renderer satisfies it today with just two things -- a /dev/dri hostPath volume (`DirectoryOrCreate`) and a read-write /dev/dri volumeMount on the app container -- and a later move to the Intel GPU device plugin is then a render.py change with no catalog edit. It sets NO podSecurityContext.supplementalGroups: measured in-container on prod 2026-09-30, the pms-docker cont-init hook adds the app user to the device's group inside the container and `s6-setuidgid` then REPLACES the process group list from /etc/group, discarding anything Kubernetes supplied, so such an entry would be inert and misleading. Renders NOTHING when absent, so every existing app stays byte-identical. Emits no privileged securityContext (Kyverno restrict-host-namespaces would reject it and it is not needed), no nodeSelector (node placement is a separate concern with its own justification) and no resource requests). 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.mdLine truncated
+  "description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.9.0 (adds optional `auth.apiRoutes` -- a per-service list of anchored path regexes the forward-auth oauth2-proxy sidecar answers with 401 instead of a 302 to Keycloak, rendered as one repeatable `--api-route` per entry. The `auth` block stays `additionalProperties: false`; this is one named, documented key, not an opened door. Without it there was NO per-app override hook at all, so the only way to answer an app's XHRs with 401 was a renderer-wide constant -- and the path shape is a property of the app: jackett's is `/api/` (33 x 302 vs 2 x 401 measured 2026-09-30, 4 of 6 POSTs answered by the proxy and never reaching the app), radarr's is `/signalr/` (232 x 302 measured 2026-10-01, every one a POST from a left-open browser tab that is never re-prompted), and pihole serves no `/api/` at all. Each entry must be anchored at `^/` and continue with literal path characters, because a too-broad regex turns a plain page load into a 401 and the service looks permanently broken to a logged-out user. Enforced on every gate run by ci/l0.sh's schema stage over catalog/*.yaml, by catalog/render.py refusing to render an unanchored entry, and by conformance/tests/l0-static/oauth2-proxy-api-routes-per-app)). 1.8.0 (adds optional `hardware` - a host HARDWARE CAPABILITY declaration for the container kind, currently one member: `hardware.gpu: intel-quicksync`, meaning the workload needs the host iGPU render device for hardware video transcode. It is an ENUM OF INTENT, not a device path: the renderer satisfies it today with just two things -- a /dev/dri hostPath volume (`DirectoryOrCreate`) and a read-write /dev/dri volumeMount on the app container -- and a later move to the Intel GPU device plugin is then a render.py change with no catalog edit. It sets NO podSecurityContext.supplementalGroups: measured in-container on prod 2026-09-30, the pms-docker cont-init hook adds the app user to the device's group inside the container and `s6-setuidgid` then REPLACES the process group list from /etc/group, discarding anything Kubernetes supplied, so such an entry would be inert and misleading. Renders NOTHING when absent, so every existing app stays byte-identical. Emits no privileged securityContext (Kyverno restrict-host-namespaces would reject it and it is not needed), no nodeSelector (node placement is a separate concern with its own justification) and no resource requests). 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.md already said `internal` renders no public DNS record; nothing enforced it, and six internal services -- jackett, pihole, qbittorrent, radarr, semaphore, sonarr -- were publishing `<name>.home.keithjackson.dev` into the PUBLIC Cloudflare zone answering with the RFC1918 Gateway VIP, which published the internal subnet and a full service inventory and made each name a DNS-rebind primitive. Enforced on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns). 1.6.0 (splits `observability.metrics` from WHERE metrics live: adds optional `metricsEndpoint` (port/name/path/interval) and optional `livenessWorkload` (kind/name). `metrics: true` previously rendered a ServiceMonitor hardcoded to a port named `metrics` that no rendered Service ever defined, so all twelve app monitors discovered ZERO targets and every `up{job=...}` SLI was structurally inert on prod; a monitor is now rendered only from a declared `metricsEndpoint`, which also renders the Service port and the monitoring-namespace ingress allow, and the `up` SLI is expressed against kube-state-metrics -- a signal that exists for every workload without a per-app exporter. `livenessWorkload` names the object whose readiness IS the service, for the case where the running workload is not the one this SD renders). 1.5.0 (adds optional `inClusterClients` — additional in-cluster clients, named by NAMESPACE, admitted to a declared port of this service DIRECTLY, bypassing the shared Gateway. `ingressRoutes` admits only the Gateway proxy identity and `dependencies` only links catalog services, so a PLATFORM controller dialling a Service ClusterIP had no authorable form at all; that gap is what deadlocked proLine truncated
  "type": "object",
  "additionalProperties": false,
  "required": ["name", "source"],
@@ -139,21 +139,6 @@
        "supplementalGroups": { "type": "array", "items": { "type": "integer" }, "description": "Additional GIDs added to the first process in each container." }
      }
    },
    "availability": {
      "type": "object",
      "description": "Optional availability intent for the container kind: how many replicas of this workload must be running. INTENT ONLY -- it names no Kubernetes object and no scheduling primitive. How the platform satisfies it (workload kind, per-replica volumes, node spreading, disruption budget) is catalog/render.py's business, documented in schemas/renderer-contract.md `### availability`. Absent => replicas 1 and a Deployment, so every pre-existing app renders byte-identical. Ignored for kind=vm (a VM's availability is a different problem with different primitives).",
      "additionalProperties": false,
      "minProperties": 1,
      "properties": {
        "replicas": {
          "type": "integer",
          "description": "How many replicas must be running. 1 (default) renders a Deployment exactly as before. >1 declares that the service must survive the loss of a single node; the renderer then emits a StatefulSet with per-replica volumeClaimTemplates, required per-node anti-affinity, podManagementPolicy Parallel and a PodDisruptionBudget. Only declare >1 for a service whose on-disk state is reproducible from git -- a replica with a fresh volume comes up serving whatever it can bootstrap itself, and nothing else.",
          "minimum": 1,
          "maximum": 5,
          "default": 1
        }
      }
    },
    "hardware": {
      "type": "object",
      "description": "Optional host HARDWARE CAPABILITY the workload needs, for the container kind (docs/13). This field declares INTENT (`what device class this service needs`), never MECHANISM (`which host path to bind-mount`, `which extended resource to request`). That separation is the whole point of the enum: the platform currently satisfies `intel-quicksync` with a /dev/dri hostPath volume and a matching mount on the app container, but a later migration to the Intel GPU device plugin (`gpu.intel.com/i915` resource requests, no hostPath) must be a catalog/render.py change and NOTHING else - no ServiceDefinition may need editing, and no reviewer may have to re-derive per-service device wiring. Do NOT 'simplify' this into a raw device path, a hostPath, or a resource request; that would push a platform implementation detail into the operator-facing contract and make the migration a catalog-wide edit. Absent => no device wiring at all (existing apps render byte-identical). Ignored for kind=vm (a VM gets devices through KubeVirt's own domain.devices).",