docs(decisions): record that PiHole HA is verified live, failover drill included

No manifests change. The three entries merged earlier today each say, correctly at the time, that nothing had been reconciled -- which is now stale in exactly the way CLAUDE.md section 2b warns about: a reader would conclude the work is unproven. This entry is the current state and does not edit the earlier files. Measured on live prod after dce20b7: apps-wave-0 reconciled in ~90s (inside the 5m timeout the multi-replica entry flagged as a risk); pihole-0 on k3s-prod-3 and pihole-1 on k3s-prod-2, so the required anti-affinity was satisfied; both per-replica claims Bound; the old Deployment and the old unused PVC both pruned; PDB healthy with one disruption allowed. The mountPath fix is confirmed: /etc/pihole is now on /dev/longhorn with 6.9M used on both pods, where it was previously on the overlay. Phase 1's declarations reproduce onto a FRESH volume, which was the whole reason for sequencing it first: 72,234 gravity domains rebuilt on each replica from the declared adlist, and the declared SOPS secret authenticates on BOTH pods ("valid":true via each container's own env, value never printed). FAILOVER DRILL, observed by the operator: a replica was deleted, recreated in ~30 seconds, and DNS never dropped a request. First direct evidence the replica count delivers what it was added for. Re-deriving the SLI from the rendered workload kind mattered concretely: the StatefulSet series returns 2 while the old Deployment series now returns ZERO series, so a stale Deployment-shaped rule would have matched its own absent() branch and fired permanently -- a standing false page. Still unverified and stated as such: node failure (the drill deleted a pod, so the 300s toleration path is untouched), DNS during the cutover itself, two-node loss, and the cluster-wide dependency that replicas cannot address. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Merge pull request 'feat(pihole): two replicas on different nodes, and fix a PVC mounted where pihole never writes' (#45 ) from feat/pihole-multi-replica into main
2026-10-04 02:03:42 -05:00 · 2026-10-04 06:50:56 +00:00 · 2026-10-04 01:34:52 -05:00 · 2026-10-04 06:14:47 +00:00 · 2026-10-04 01:07:30 -05:00 · 2026-10-04 04:46:34 +00:00
49 changed files with 4163 additions and 351 deletions
@@ -36,6 +36,36 @@ sensitive path, so the policy cannot be widened or narrowed without a human.

 That stage is **fail-closed** — if it cannot evaluate, it blocks. Unlike `gitea-settings`, where a
 missing credential is a lock-out, here a skip would be a bypass.
+
+**The administrator override (2026-10-01).** The rule above was unsatisfiable for the one human
+who runs this repo: Gitea refuses to let an author approve their own PR, and `jakman85` is the
+only collaborator — so a sensitive-path PR the operator opened could never be merged, *including
+the PR that would fix that.* An administrator may therefore waive the review by posting this on
+the PR:
+
+    /approve-override <reason>
+
+**Post it as a PR *review* of type "Comment", not in the plain comment box.** CI's
+`SETTINGS_API_TOKEN` is minted `[read:repository, read:admin]`, and `/issues/{n}/comments` sits
+behind Gitea's *issue* scope — so an override typed into the comment box is **invisible to CI**,
+and the stage reports it as a missing approval rather than as a permission problem (measured on
+PR #38, 2026-10-02). `/pulls/{n}/reviews` is readable, so a review body works today. Both sources
+are accepted and the gate names which it used; the comment box starts working once
+`infra/gitea/set-ci-token.sh` is re-run to pick up `read:issue`.
+
+The directive must **start a line**, the **reason is mandatory**, the poster's `admin`/`owner`
+permission is re-checked **live** when the gate runs, and it must be **newer than the
+head commit** — push again and you must override again. The overrider, the timestamp and the
+reason are printed in the gate log and the job summary. For an administrator, "a change to the
+gate cannot be gated by the gate" is therefore **advisory, not structural**; what is kept is an
+attributed record of each waiver. Note that the agent account `gitea-admin` also holds `owner`
+here, so an agent *can* clear this on its own PR — **do not**: posting that directive to unblock
+your own agent work is exactly the self-approval the stage exists to prevent. Get the operator
+to post it, or move the change out of the sensitive paths.
+
+**`required_approvals` on `main` is `0`**, not 1 — the gate is the approver. That drops the
+approval *count* only: `block_admin_merge_override` and `enable_status_check` both remain `true`,
+so **a red gate is unmergeable by everyone, administrators included.**
 - **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
  anything applied by hand is drift that the next reconcile reverts or, worse, adopts.

@@ -53,12 +83,19 @@ Verified against the live Gitea on 2026-09-29:
 | `main` protected, direct push rejected | ✅ `protected: true`, `user_can_push: false` — **verified by a real rejected push**, not by the API's word |
 | PR blocked until the gate passes | ✅ `enable_status_check: true`, context `gate / gate (pull_request)` |
 | Gate runs on every PR | ✅ runner `incluster-dind` online; ~1m35s |
-| Approval required before merge | ✅ `required_approvals: 1` |
+| Gate cannot be overridden, by anyone | ✅ `block_admin_merge_override: true` |
+| Human approval required before merge | ⬜ **No — `required_approvals: 0` since 2026-10-01.** The gate is the approver; sensitive paths are gated by `ci/approval-policy.sh` instead. See §1 |

-All four were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
+The first three were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
 true. They are true now because each was measured, and the push rejection was provoked rather
 than assumed. **Re-measure before relying on any of them** — see §2b.

+The fourth row deliberately changed on 2026-10-01. `required_approvals: 1` was a **lock-out**, not
+a protection: with a single collaborator and Gitea refusing author self-approval, nothing could
+satisfy it (measured on PR #35 — green gate, zero reviews, unmergeable). It also contradicted the
+model already written into `ci/approval-policy.sh` and `ci/gate.sh`. Rationale and the live-change
+script: `infra/gitea/allow-admin-self-merge.sh`.
+
 Still not covered: `vars.CLUSTER_AVAILABLE` is unset, so **L2/L3 conformance does not run in
 CI**. A green PR therefore proves the L0/L1 offline subset, not runtime behaviour.

@@ -14,10 +14,12 @@ resources:
 - nextcloud-db-backup-credentials-prod.sops.yaml
 - nextcloud-db-backup-credentials.sops.yaml
 - nextcloud-db.yaml
+- pihole-bootstrap-secret.sops.yaml
 - pihole-dns.yaml
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
 - plex-claim-secret.sops.yaml
+- plex-remote.yaml
 - plex.yaml
 - qbittorrent-bootstrap-secret.sops.yaml
 - qbittorrent-oauth2-proxy-credentials.sops.yaml
@@ -0,0 +1,72 @@
+# =================================================================================================
+# PiHole WEB/API ADMIN PASSWORD — the one piece of PiHole state that was NOT declared anywhere.
+# =================================================================================================
+# WHY THIS FILE EXISTS. Measured on live prod 2026-10-04, `pihole.toml` marks every non-default
+# setting with `### CHANGED`, and names the ones that came from the environment `### CHANGED (env)`.
+# The complete list was:
+#
+#     dns.upstreams          ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     dns.listeningMode      ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     dns.rateLimit.count    ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     misc.dnsmasq_lines     ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     files.macvendor        ### CHANGED         <- set by the image itself, container path
+#     webserver.api.pwhash   ### CHANGED         <- SET OUT OF BAND. Declared NOWHERE. This file.
+#
+# So exactly ONE operator-set value lived only on the Longhorn PVC: the admin password, as a
+# one-way BALLOON-SHA256 hash. Nobody could recover it (a hash does not reverse), nothing in git
+# produced it, and a rebuild from git would have come up with a DIFFERENT password — or none.
+#
+# THE REASON THIS BLOCKS HA, which is why it is fixed first. The next step is running more than one
+# PiHole replica, and a replica with a fresh PVC would come up with its own unrelated password. The
+# admin UI would then accept or reject the same credential depending on which pod answered. Declaring
+# it here makes every replica, and every rebuild, identical by construction.
+#
+# MECHANISM: `webserver.api.password` is a real, settable FTL config key — verified in the running
+# image, where `pihole setpassword` is itself just `setFTLConfigValue "webserver.api.password"` with
+# the comment "pihole-FTL will automatically hash the password". So FTL takes CLEARTEXT on this key
+# and writes the hash into `webserver.api.pwhash` itself. catalog/pihole.yaml therefore passes this
+# value as `FTLCONF_webserver_api_password`, and the hash on the PVC becomes a derived artefact
+# rather than the source of truth.
+#
+# DEFENCE IN DEPTH, NOT THE ONLY GATE. The admin UI is already behind the oauth2-proxy sidecar
+# (`--allowed-group=platform-admins`), so this password is a SECOND gate. It is kept rather than set
+# empty because the `pihole` Service still publishes port 80 (PiHole direct, bypassing the proxy on
+# :4180) and only the CiliumNetworkPolicy closes it — and the default-deny policy in this repo has a
+# documented drift history. An unauthenticated admin UI that can rewrite LAN DNS is not a good thing
+# to leave resting on one policy staying correct. Narrowing that Service port is tracked follow-up.
+#
+# ROTATION: `sops` this file in your own terminal, change the value, commit. Flux re-applies the
+# Secret, the pod restarts, FTL re-hashes. Do NOT `sops -d` it into a shared terminal or a log
+# (CLAUDE.md section 6); read it in an editor. The value below was generated randomly at authoring
+# time and has never been printed to a transcript.
+apiVersion: v1
+kind: Secret
+metadata:
+    name: pihole-bootstrap
+    namespace: pihole
+    labels:
+        app.kubernetes.io/name: pihole
+type: Opaque
+stringData:
+    #ENC[AES256_GCM,data:TMOuRyJeb5IVn3lZPXY7lgh1KGxu7GY9+WIOLHpMB50JqzmM2tHIaWUWKTt1JtZoX/E5/Y7WCUTnfkJLhRBN2zV2MwWRBSLYVy5CMg7xV8PTLRZJYAVUYS5Trg==,iv:q+DpfT6PQfY5ZuNsmP7Ddl+PDLjSm89fJICAhYoW64k=,tag:axijaO3DaxIZgZ3LeK2hXA==,type:comment]
+    webui-password: ENC[AES256_GCM,data:7FcK/bUW79M46f/MlouAmATKbMnoUTNuMUjtqJ5vpU4=,iv:I8DBFB39uXZV42Ie2IPwSefmMgdKEtiDknfc9Tu7h5E=,tag:L423784UrDU3TvylpHc6KQ==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0MkRSNjZLZlpXYkcvaVJL
+            cjlqb2wrcWREZnZLdGpsTHNPWEpseHlRZnpjCnJVM1FDQ2M2dXlaS1JBZ2k2V1BP
+            Yi9QUDZSMzZsL09pTi93RTdjdGxtY2MKLS0tIDIrQTZIVWdhMGIxSCtlUFN2Mlho
+            SWVybld4VjJyaHRSKzZZU0xsUUV2SEEKbDYHV/qBZVspK7DpBW3vYjRMJ/HmAN+9
+            G9bOlNEp0jAtcNZEn6lpQ9ORgvmIrhJHoBQRLabdMWzNWSZsNn56tA==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-10-04T06:01:27Z"
+    mac: ENC[AES256_GCM,data:rhTMTOwLOtHXHgqI5M2ZW/6w+UpLpxBlTk7NoN0wmGSqOcE+OGrXA1DhkZrJweyQT0RSNSSBm/6QMztLP81eepOzdp2rEkwvbQPK/l6K9g16Ww9c1IatS3NQsKEB8AoMDx7jBQ6exXYgggPx3MJsNYbE+WyloQ96U9KPdrecAM8=,iv:n34v2zX9D5xsaR1PFCH94mNMiuRfhUyI2lseUCoqiTA=,tag:9YmN9nIVLXfsGYIb0bUl2w==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4
@@ -35,34 +35,18 @@ spec:
    - protocol: TCP
      port: 53
 ---
-# PVC pihole-config (R-STATE)
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: pihole-config
-  namespace: pihole
-  labels:
-    app.kubernetes.io/name: pihole
-spec:
-  accessModes:
-  - ReadWriteOnce
-  storageClassName: longhorn
-  resources:
-    requests:
-      storage: 2Gi
---
-# Deployment (R-PLAT-02)
+# StatefulSet (R-PLAT-02)
 apiVersion: apps/v1
-kind: Deployment
+kind: StatefulSet
 metadata:
  name: pihole
  namespace: pihole
  labels:
    app.kubernetes.io/name: pihole
 spec:
-  replicas: 1
-  strategy:
-    type: Recreate
+  replicas: 2
+  serviceName: pihole
+  podManagementPolicy: Parallel
  selector:
    matchLabels:
      app.kubernetes.io/name: pihole
@@ -90,9 +74,21 @@ spec:
          value: '0'
        - name: FTLCONF_misc_dnsmasq_lines
          value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}
+        - name: FTLCONF_webserver_api_password
+          valueFrom:
+            secretKeyRef:
+              name: pihole-bootstrap
+              key: webui-password
        volumeMounts:
        - name: config
-          mountPath: /data/config
+          mountPath: /etc/pihole
+        lifecycle:
+          postStart:
+            exec:
+              command:
+              - /bin/bash
+              - -c
+              - "# Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and\n# does NOT run a gravity rebuild.\nset -u\nGRAVITY=/etc/pihole/gravity.db\n\n# THE DECLARED SET \u2014 one URL per line, the source of truth for what PiHole blocks.\n# To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.\nDECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'\n\n# On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,\n# and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.\ni=0\nwhile [ ! -s \"$GRAVITY\" ] && [ \"$i\" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done\nif [ ! -s \"$GRAVITY\" ]; then\n  echo \"pihole-adlists: gravity.db absent after 180s - leaving adlists untouched\" >&2\n  exit 0\nfi\n\nsql() { pihole-FTL sqlite3 \"$GRAVITY\" \"$1\"; }\nCHANGED=0\n\n# 1) Every declared URL must be present AND enabled.\necho \"$DECLARED\" | while IFS= read -r url; do\n  [ -n \"$url\" ] || continue\n  printf '%s\\n' \"$url\"\ndone > /tmp/declared.txt\nwhile IFS= read -r url; do\n  [ -n \"$url\" ] || continue\n  # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.\n  have=$(sql \"select count(*) from adlist where address='$url';\")\n  if [ \"$have\" = \"0\" ]; then\n    echo \"pihole-adlists: adding $url\"\n    sql \"insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');\"\n    CHANGED=1\n  else\n    en=$(sql \"select enabled from adlist where address='$url';\")\n    if [ \"$en\" != \"1\" ]; then\n      echo \"pihole-adlists: re-enabling $url\"\n      sql \"update adlist set enabled=1 where address='$url';\"\n      CHANGED=1\n    fi\n  fi\ndone < /tmp/declared.txt\n\n# 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).\nsql \"select address from adlist where enabled=1;\" | while IFS= read -r addr; do\n  [ -n \"$addr\" ] || continue\n  if ! grep -qxF \"$addr\" /tmp/declared.txt; then\n    echo \"pihole-adlists: disabling undeclared $addr\"\n    sql \"update adlist set enabled=0 where address='$addr';\"\n    echo changed >> /tmp/adlist-changed\n  fi\ndone\n[ -s /tmp/adlist-changed ] && CHANGED=1\nrm -f /tmp/adlist-changed /tmp/declared.txt\n\n# 3) Rebuild gravity ONLY if the declared set actually moved.\nif [ \"$CHANGED\" = \"1\" ]; then\n  echo \"pihole-adlists: declared set changed - running gravity update\"\n  pihole -g || echo \"pihole-adlists: gravity update failed; existing gravity.db retained\" >&2\nelse\n  echo \"pihole-adlists: already matches the declared set - nothing to do\"\nfi\nexit 0\n"
      - name: oauth2-proxy
        image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
        imagePullPolicy: IfNotPresent
@@ -163,10 +159,39 @@ spec:
            port: http
          initialDelaySeconds: 5
          periodSeconds: 15
-      volumes:
-      - name: config
-        persistentVolumeClaim:
-          claimName: pihole-config
+      affinity:
+        podAntiAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+          - labelSelector:
+              matchLabels:
+                app.kubernetes.io/name: pihole
+            topologyKey: kubernetes.io/hostname
+  volumeClaimTemplates:
+  - metadata:
+      name: config
+      labels:
+        app.kubernetes.io/name: pihole
+    spec:
+      accessModes:
+      - ReadWriteOnce
+      storageClassName: longhorn
+      resources:
+        requests:
+          storage: 2Gi
+---
+# PodDisruptionBudget — keep one replica through a drain (availability.replicas)
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+  name: pihole
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+spec:
+  minAvailable: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: pihole
 ---
 # Service
 apiVersion: v1
@@ -269,11 +294,11 @@ spec:
  - name: pihole.slis
    rules:
    - alert: pihole-down
-      expr: kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"} == 0 or absent(kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"})
+      expr: kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"} == 0 or absent(kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"})
      for: 5m
      labels:
        severity: critical
        service: pihole
      annotations:
        summary: pihole is down
-        description: 'pihole: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
+        description: 'pihole: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready).'
@@ -0,0 +1,126 @@
+---
+# =================================================================================================
+# PLEX REMOTE ACCESS — raw tcp/32400 north-south exposure, and the policy that admits it.
+# =================================================================================================
+# HAND-WRITTEN, NOT RENDERED. A non-catalog app-tier unit anchored to the `plex` ServiceDefinition
+# and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS table — the same
+# mechanism gitea-actions and pihole-dns use. It is hand-written for the SAME reason those are:
+# schemas/servicedefinition.schema.json is `additionalProperties: false` and has NO field for a
+# type=LoadBalancer Service, for a pinned lbipam VIP, or for raw-TCP north-south exposure at all.
+# `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator.
+#
+# WHY THE L7 GATEWAY IS NOT ENOUGH HERE — and it is NOT because the Gateway fails to serve Plex.
+# Measured 2026-10-03 through the Gateway VIP: `GET https://plex.home.keithjackson.dev/identity`
+# returned HTTP 200 with `claimed="1"` and the real machineIdentifier, and the protected endpoints
+# (/library/sections, /status/sessions, /:/prefs) all returned 401. The L7 path WORKS.
+#
+# The problem is DISCOVERY, not proxying. Plex apps (phone, TV, Roku) do not resolve
+# `plex.${BASE_DOMAIN}`; they ask plex.tv for the server's connection list. A server gets onto that
+# list one of two ways: it advertises a custom URL (`customConnections`, the ADVERTISE_IP env), or it
+# publishes a reachable public IP:port for its own tcp/32400. Measured in the live Preferences.xml on
+# 2026-10-03: `customConnections` ABSENT, `ManualPortMappingMode` ABSENT, while
+# `PublishServerOnPlexOnlineKey="1"` — the server was trying to publish and had nothing reachable to
+# publish, so remote clients saw NO connection at all. The operator chose the NATIVE path (this
+# file) over advertising the Gateway URL: Plex then terminates its own TLS with its
+# `*.plex.direct` certificate and video streams bypass the Envoy L7 hop entirely.
+#
+# VIP: ${PLEX_L4_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.57, prod 192.168.2.24), PINNED via
+# lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
+# (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the existing
+# CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config is needed;
+# this Service just consumes the pool. PINNING IS NOT OPTIONAL: the address is typed into a WAN
+# port-forward on the SOHO router, which is hand-maintained and off-repo, so it can never move.
+# 192.168.2.24 verified FREE live 2026-10-03 — `default-pool` reported IPsTotal 11 / IPsUsed 4 /
+# IPsAvailable 7, and the only four allocations were .20 (cilium-gateway-shared), .21 (unifi-l4-*),
+# .22 (crafty-l4-*) and .23 (pihole-dns-*).
+#
+# ONE SERVICE, TCP ONLY. Unlike unifi/crafty/pihole-dns this needs no second Service and no
+# lbipam.cilium.io/sharing-key: Plex's remote access is tcp/32400 and nothing else. (Plex's
+# DLNA/GDM discovery uses UDP, but that is LAN-only multicast and is not part of remote access.)
+#
+# NO `loadBalancerSourceRanges`, DELIBERATELY — and this is the one place this file diverges from
+# its closest sibling. apps/wave-0/pihole-dns.yaml pins its source range to ${LAN_CIDR} because a
+# resolver must never answer the internet. This Service is the OPPOSITE case: admitting remote
+# clients from arbitrary internet addresses is its entire purpose, so there is no range to narrow
+# to. Plex authenticates every caller itself (measured above: 401 on every protected endpoint
+# without a token), which is what makes an unfiltered port defensible here.
+#
+# NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
+# `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their VIP as a public
+# Cloudflare record. This Service must not, for pihole-dns's reason: ${PLEX_L4_VIP} is an RFC1918
+# address that is useless in public DNS. Remote clients get this server's address from plex.tv, not
+# from the zone. The existing public `plex.${BASE_DOMAIN}` record is published from the rendered
+# HTTPRoute via the gateway source and is NOT affected by this file — the web UI keeps working
+# through the Gateway exactly as before.
+#
+# -------------------------------------------------------------------------------------------------
+# externalTrafficPolicy: Cluster — inherited from crafty's 8-DAY LAN OUTAGE, not re-derived.
+# -------------------------------------------------------------------------------------------------
+# `Local` is the tempting choice because it preserves the client source IP, which this service
+# actually has a use for (see the allowedNetworks note below). It is still WRONG here, for the
+# reason platform/crafty/service-l4.yaml documents at length: Cilium's L2 announcer elects ONE lease
+# holder per Service and does NOT follow the backend pod, so with `Local` the VIP can be announced
+# by a node that has no Plex pod, which black-holes the port while every object still reports
+# healthy. Plex is a single-replica Deployment with `strategy: Recreate` pinned to wherever the
+# scheduler put it, so the mismatch is the normal case, not a rare one. Take the SNAT.
+#
+# KNOWN CONSEQUENCE OF THAT CHOICE, stated rather than discovered later: `Cluster` SNATs the client
+# to the ingress NODE's address (192.168.2.1x), which falls inside the `192.168.0.0/16` entry of
+# catalog/plex.yaml's ALLOWED_NETWORKS. Plex will therefore classify every REMOTE client as LAN.
+# Measured 2026-10-03 on the Gateway path, which SNATs the same way: Plex logged the request as
+# `[10.0.2.214:36040 (Allowed Network (WAN))]`. This is a METERING effect, not a security one —
+# authorization is unchanged (401s above), but Plex's remote-quality/bandwidth limits will not bind
+# on these streams because it believes they are local. Narrowing ALLOWED_NETWORKS cannot fix it
+# without also de-classifying genuine LAN clients, so it is accepted.
+apiVersion: v1
+kind: Service
+metadata:
+  name: plex-l4-tcp
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+  annotations:
+    lbipam.cilium.io/ips: "${PLEX_L4_VIP}"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Cluster
+  selector:
+    app.kubernetes.io/name: plex
+  ports:
+    - name: plex-remote
+      port: 32400
+      targetPort: 32400
+      protocol: TCP
+---
+# INGRESS allow for the raw port (R-NET-01/02).
+#
+# The rendered plex.yaml already carries `plex-allow-gateway-ingress` (fromEntities: ingress ->
+# 32400) and three reciprocal allows for sonarr/radarr/seerr. CiliumNetworkPolicies are ADDITIVE, so
+# this file adds the off-cluster case without touching those.
+#
+# `world` is Cilium's entity for anything outside the cluster, which is what a remote Plex client is.
+# `host` and `cluster` mirror platform/crafty/networkpolicy.yaml's game-port rule and are what make
+# this work under `externalTrafficPolicy: Cluster`: after SNAT the packet's source is the ingress
+# NODE, which Cilium classifies as `host` (same node) or `remote-node` (a different one) — NOT
+# `world`. Listing `world` alone would compile cleanly and then drop every remote stream, so all
+# four entities are named.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: plex-allow-remote-32400
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: plex
+  ingress:
+    - fromEntities:
+        - world
+        - cluster
+        - host
+        - remote-node
+      toPorts:
+        - ports:
+            - { port: "32400", protocol: TCP }
@@ -137,14 +137,15 @@ spec:
            secretKeyRef:
              name: plex-claim
              key: PLEX_CLAIM
+        resources:
+          limits:
+            gpu.intel.com/i915: 1
        volumeMounts:
        - name: config
          mountPath: /config
        - name: media
          mountPath: /media
          readOnly: true
-        - name: dri
-          mountPath: /dev/dri
      volumes:
      - name: config
        persistentVolumeClaim:
@@ -152,10 +153,6 @@ spec:
      - name: media
        persistentVolumeClaim:
          claimName: plex-media
-      - name: dri
-        hostPath:
-          path: /dev/dri
-          type: DirectoryOrCreate
 ---
 # Service
 apiVersion: v1
@@ -173,6 +170,28 @@ spec:
    port: 32400
    targetPort: 32400
 ---
+# CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): plex-allow-radarr-32400
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: plex-allow-radarr-32400
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: plex
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        app.kubernetes.io/name: radarr
+        k8s:io.kubernetes.pod.namespace: radarr
+    toPorts:
+    - ports:
+      - port: '32400'
+        protocol: TCP
+---
 # CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): plex-allow-seerr-32400
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -195,6 +214,28 @@ spec:
      - port: '32400'
        protocol: TCP
 ---
+# CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): plex-allow-sonarr-32400
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: plex-allow-sonarr-32400
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: plex
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        app.kubernetes.io/name: sonarr
+        k8s:io.kubernetes.pod.namespace: sonarr
+    toPorts:
+    - ports:
+      - port: '32400'
+        protocol: TCP
+---
 # CiliumNetworkPolicy gateway-ingress allow (R-NET-01/02): plex-allow-gateway-ingress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -1,6 +1,8 @@
 # promtool rule-unit tests for the wave-0 per-app `<app>-down` SLI rules.
 # Manifests under test: every PrometheusRule in apps/wave-0/ — gitea, jackett, pihole, plex,
-# qbittorrent (container/Deployment) and vm-demo (KubeVirt VMI). The harness discovers this file by
+# qbittorrent (container/Deployment), pihole (container/StatefulSet — `availability.replicas: 2`, so
+# its SLI reads kube_statefulset_status_replicas_ready; see schemas/renderer-contract.md
+# `### availability`) and vm-demo (KubeVirt VMI). The harness discovers this file by
 # the SUITE convention (`<dir>/tests/*.test.yaml`) and flattens those RENDERED manifests, so these
 # tests run against the exact expressions Flux applies, never a copy.
 #
@@ -41,7 +43,7 @@ tests:
        values: '1+0x20'
      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
        values: '1+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}'
        values: '1+0x20'
      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
        values: '1+0x20'
@@ -81,7 +83,7 @@ tests:
        values: '1+0x20'
      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
        values: '1+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}'
        values: '1+0x20'
      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
        values: '0+0x20'
@@ -132,7 +134,7 @@ tests:
        values: '0+0x20'
      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
        values: '0+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}'
        values: '0+0x20'
      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
        values: '1+0x20'
@@ -170,10 +172,10 @@ tests:
              severity: critical
              service: pihole
              namespace: pihole
-              deployment: pihole
+              statefulset: pihole
            exp_annotations:
              summary: "pihole is down"
-              description: "pihole: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+              description: "pihole: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
      - eval_time: 10m
        alertname: qbittorrent-down
        exp_alerts:
@@ -283,6 +283,28 @@ spec:
      - port: '9117'
        protocol: TCP
 ---
+# CiliumNetworkPolicy egress allow (R-NET-02): radarr-allow-plex-32400
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: radarr-allow-plex-32400
+  namespace: radarr
+  labels:
+    app.kubernetes.io/name: radarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: radarr
+  egress:
+  - toEndpoints:
+    - matchLabels:
+        app.kubernetes.io/name: plex
+        k8s:io.kubernetes.pod.namespace: plex
+    toPorts:
+    - ports:
+      - port: '32400'
+        protocol: TCP
+---
 # CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): radarr-allow-seerr-7878
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -281,6 +281,28 @@ spec:
      - port: '9117'
        protocol: TCP
 ---
+# CiliumNetworkPolicy egress allow (R-NET-02): sonarr-allow-plex-32400
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: sonarr-allow-plex-32400
+  namespace: sonarr
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: sonarr
+  egress:
+  - toEndpoints:
+    - matchLabels:
+        app.kubernetes.io/name: plex
+        k8s:io.kubernetes.pod.namespace: plex
+    toPorts:
+    - ports:
+      - port: '32400'
+        protocol: TCP
+---
 # CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): sonarr-allow-seerr-8989
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -12,6 +12,42 @@ storage:
  - name: config
    size: 2Gi
    class: longhorn           # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
+    # THE PVC WAS MOUNTED SOMEWHERE PIHOLE NEVER WRITES, AND SO HELD NOTHING AT ALL. Without an
+    # explicit mountPath the renderer defaults to /data/<name>, i.e. /data/config — but PiHole keeps
+    # every piece of its state in /etc/pihole. MEASURED on live prod 2026-10-04: `/etc/pihole` (7.1 M
+    # — gravity.db, pihole.toml, the query DB) was on the **overlay** filesystem, i.e. the
+    # container's EPHEMERAL layer, while `/dev/longhorn/pvc-849edb86...` sat mounted at
+    # /data/config holding `lost+found` and 24 K of nothing. Three consequences, all live until now:
+    #   * every restart silently discarded PiHole's whole state, re-downloading 72 k gravity domains
+    #     and regenerating the admin password hash;
+    #   * `backup.policy: daily-offcluster` below has been faithfully backing up an EMPTY volume,
+    #     which is worse than no backup because it reports success;
+    #   * it is also why phase 1's password/adlist declaration mattered more than it first appeared —
+    #     that state was not merely undeclared, it was ephemeral.
+    # This is the same class of bug catalog/plex.yaml's `config` entry documents ("Without this the
+    # PVC sat unused and Plex wrote to the container's EPHEMERAL layer").
+    #
+    # /etc/pihole IS THE IMAGE'S INTENDED MOUNT POINT, and an EMPTY volume there self-initialises.
+    # Verified in the running image, not assumed:
+    #   * `/usr/bin/start.sh` (PID 1) calls `migrate_gravity`, whose `migrate_to_database()` creates
+    #     the gravity database when `[ ! -e "$gravityDBfile" ]` — the same branch the postStart hook
+    #     below relies on, and the branch that produced the current ephemeral copy.
+    #   * FTL writes `pihole.toml` itself from the `FTLCONF_*` env above (that is what the
+    #     `### CHANGED (env)` markers in it are).
+    #   * NO `podSecurityContext.fsGroup` is needed and adding one would be inert: `start.sh` line 5
+    #     is `chown -R pihole:pihole /etc/pihole/ /var/log/pihole/` and PID 1 runs as **uid 0**, so
+    #     the image fixes ownership of a fresh root-owned volume before anything else starts. Same
+    #     arrangement as plex's image managing its own /config ownership.
+    #   * the image's own default for `files.macvendor` was moved OUT of this directory (to
+    #     `/macvendor.db`), which is what a maintainer does to a path they expect to be a volume.
+    #
+    # WHY IT IS FIXED IN THIS BRANCH rather than left for its own: `availability.replicas: 2` renders
+    # per-replica volumeClaimTemplates, and with the wrong mountPath those would be two 2 Gi Longhorn
+    # volumes (times three Longhorn replicas each) holding nothing but `lost+found` while all state
+    # stayed ephemeral — paying the cost of persistence and getting none of it. It also REMOVES this
+    # change's main operational risk: with gravity.db persisted, a pod restart is cheap, instead of
+    # every restart re-downloading 72 k domains inside a wave that has a 5 m timeout.
+    mountPath: /etc/pihole
 dependencies: []              # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
 # ------------------------------------------------------------------------------------------------
 # LAN SPLIT-HORIZON DNS — the half of R-DNS-03 that platform/coredns/coredns-custom.yaml defers to.
@@ -100,6 +136,53 @@ env:
  # while resolution for the zone does not work. Confirmed both ways against the live image.
  - name: FTLCONF_misc_dnsmasq_lines
    value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}"
+  # THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
+  # apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
+  # carries the full rationale and the measurement that found it.
+  #
+  # `webserver.api.password` takes CLEARTEXT and FTL hashes it itself into `webserver.api.pwhash`.
+  # Verified in the running image, not assumed: `pihole setpassword` is itself
+  # `setFTLConfigValue "webserver.api.password" "$PASSWORD"`, commented upstream as "pihole-FTL will
+  # automatically hash the password". So the BALLOON-SHA256 hash on the PVC is now a DERIVED artefact
+  # and this is its source of truth. Rotating the Secret rotates the password.
+  - name: FTLCONF_webserver_api_password
+    valueFrom:
+      secretKeyRef:
+        name: pihole-bootstrap
+        key: webui-password
+availability:
+  # TWO REPLICAS, ON DIFFERENT NODES. LAN DNS is a critical network service and it was ONE pod on
+  # ONE node. Measured 2026-10-04, that cost more than "until it reschedules":
+  #   * the node running the pod dies -> ~40s to mark it NotReady, then `tolerationSeconds: 300`
+  #     before the pod is even evicted, then reschedule, then Longhorn must force-detach the RWO
+  #     volume from a dead node and reattach it. Roughly 6-10 MINUTES with no LAN resolution.
+  #   * the node holding the `pihole-dns-*` L2 lease dies -> Cilium re-elects an announcer, tens of
+  #     seconds. (These were DIFFERENT nodes: lease on k3s-prod-2, pod on k3s-prod-1. Two of three
+  #     nodes were single points of failure for LAN DNS, with wildly different blast radii.)
+  # The Services already tolerate this: `externalTrafficPolicy: Cluster` on the DNS LoadBalancers
+  # (apps/wave-0/pihole-dns.yaml) means the L2 lease holder forwards to ANY ready backend, so a
+  # second replica is picked up with no change there.
+  #
+  # WHAT THIS DOES NOT FIX, and it is the bigger risk: LAN DNS still depends on the CLUSTER. etcd
+  # quorum loss, a Cilium regression, a Longhorn problem or a bad reconcile takes DNS down
+  # everywhere regardless of replica count. The only thing that removes that dependency is a
+  # resolver outside the cluster carrying the same `address=` lines. Considered and not chosen
+  # (operator decision, 2026-10-04); see decisions/2026-10-04-pihole-multi-replica.md.
+  #
+  # THE PREREQUISITE WAS DONE FIRST, and this field would have been actively harmful without it. A
+  # replica gets a FRESH volume, and until 2026-10-04 PiHole's admin password and adlists lived only
+  # on the PVC — so replica 1 would have come up SERVING DNS AND BLOCKING NOTHING, with a different
+  # password, and answers would have varied by which pod replied. Both are now declared (the
+  # password as a SOPS Secret -> FTLCONF_webserver_api_password; the adlists reconciled into
+  # gravity.db by the postStart hook below), which is what makes a fresh replica identical to its
+  # sibling. See decisions/2026-10-04-pihole-declare-state.md.
+  #
+  # WHY 2 AND NOT 3. Two tolerates the loss of any single node, which is the stated goal, and leaves
+  # a spare node for the evicted replica to land on under the HARD anti-affinity the renderer emits.
+  # Three would pin one replica per node with no spare, so a single node failure would leave a
+  # replica permanently Pending and — `apps-wave-0` being `wait: true` — stall the app tier for as
+  # long as the node was down. Two is the count that degrades gracefully here.
+  replicas: 2
 ingressRoutes:
  - host: pihole.${BASE_DOMAIN}
    port: 80                  # admin web UI, protected at the Gateway by forward-auth (docs/09)
@@ -114,6 +197,110 @@ auth:
  # rather than a renderer constant: the measurement that justifies the flag on two apps is
  # evidence about those two apps. Its oauth2-proxy log holds 0 unauthenticated requests across
  # the current pod's whole 23h lifetime (measured 2026-10-01), so there is nothing to act on.
+lifecycle:
+  # LEVEL-TRIGGERED ADLIST ASSERTION, run on EVERY container start. Same pattern, and the same
+  # reasoning, as catalog/qbittorrent.yaml's postStart: desired state lives in git and is
+  # RE-ASSERTED unconditionally rather than set once by hand.
+  #
+  # WHY THIS IS NEEDED AT ALL — the blocklists were NOT reproducible from git. Measured on live prod
+  # 2026-10-04: `gravity.db` held exactly ONE adlist (StevenBlack's hosts, enabled) and 72,234
+  # gravity domains, and NOTHING in this repo named it. Worse, a FRESH database does not seed it:
+  # the image's own schema template `/etc/.pihole/advanced/Templates/gravity.db.sql` creates the
+  # `adlist` table and inserts NO rows into it (grepped in the running image — there is no
+  # `INSERT INTO adlist` at all). So a rebuild from git, or a replica with a new PVC, would come up
+  # SERVING DNS AND BLOCKING NOTHING, silently and with every object reporting healthy.
+  #
+  # WHY A postStart HOOK AND NOT A FILE. PiHole v6 keeps adlists as DB rows, not config. The
+  # `/etc/pihole/adlists.list` file still exists in `gravity.sh` but is MIGRATION-ONLY: its import
+  # sits inside `migrate_to_database()`, guarded by `if [ ! -e "$gravityDBfile" ]`, so it fires only
+  # while CREATING a database and never again on an existing one. A ConfigMap mounted there would
+  # therefore be silently ignored on every pod this cluster currently runs. There is also no
+  # `pihole adlist` CLI subcommand (only allow/deny for domains), so SQL against the DB is the
+  # actual interface. FTL ships an embedded shell (`pihole-FTL sqlite3`, SQLite 3.53.1) so no extra
+  # tooling is needed in the image.
+  #
+  # IT RECONCILES, IT DOES NOT JUST ADD: adlists present in the DB but absent from DECLARED below are
+  # DISABLED (not deleted — deleting cascades to `gravity` rows and would force a full re-download of
+  # everything on a typo). So this list is the source of truth in both directions, and `pihole -g`
+  # runs ONLY when something actually changed, which keeps the steady-state start free of a 70k-domain
+  # rebuild.
+  #
+  # NO BRACED DOLLAR EXPANSION ANYWHERE IN THIS SCRIPT, DELIBERATELY — the same contract
+  # catalog/qbittorrent.yaml documents at length. apps-wave-0 declares postBuild.substituteFrom
+  # cluster-vars, so Flux runs envsubst over this whole manifest at reconcile time; a `${VAR}` form
+  # would behave differently on the cluster than under a plain `kubectl apply` of the golden. Bare
+  # `$VAR` and `$(cmd)` are untouched by envsubst and so behave identically however this is run.
+  # `render.py`'s assert_substitution_safe refuses to emit a manifest that breaks this.
+  postStart:
+    exec:
+      command:
+        - /bin/bash
+        - -c
+        - |
+          # Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and
+          # does NOT run a gravity rebuild.
+          set -u
+          GRAVITY=/etc/pihole/gravity.db
+
+          # THE DECLARED SET — one URL per line, the source of truth for what PiHole blocks.
+          # To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.
+          DECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'
+
+          # On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,
+          # and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.
+          i=0
+          while [ ! -s "$GRAVITY" ] && [ "$i" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done
+          if [ ! -s "$GRAVITY" ]; then
+            echo "pihole-adlists: gravity.db absent after 180s - leaving adlists untouched" >&2
+            exit 0
+          fi
+
+          sql() { pihole-FTL sqlite3 "$GRAVITY" "$1"; }
+          CHANGED=0
+
+          # 1) Every declared URL must be present AND enabled.
+          echo "$DECLARED" | while IFS= read -r url; do
+            [ -n "$url" ] || continue
+            printf '%s\n' "$url"
+          done > /tmp/declared.txt
+          while IFS= read -r url; do
+            [ -n "$url" ] || continue
+            # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.
+            have=$(sql "select count(*) from adlist where address='$url';")
+            if [ "$have" = "0" ]; then
+              echo "pihole-adlists: adding $url"
+              sql "insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');"
+              CHANGED=1
+            else
+              en=$(sql "select enabled from adlist where address='$url';")
+              if [ "$en" != "1" ]; then
+                echo "pihole-adlists: re-enabling $url"
+                sql "update adlist set enabled=1 where address='$url';"
+                CHANGED=1
+              fi
+            fi
+          done < /tmp/declared.txt
+
+          # 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).
+          sql "select address from adlist where enabled=1;" | while IFS= read -r addr; do
+            [ -n "$addr" ] || continue
+            if ! grep -qxF "$addr" /tmp/declared.txt; then
+              echo "pihole-adlists: disabling undeclared $addr"
+              sql "update adlist set enabled=0 where address='$addr';"
+              echo changed >> /tmp/adlist-changed
+            fi
+          done
+          [ -s /tmp/adlist-changed ] && CHANGED=1
+          rm -f /tmp/adlist-changed /tmp/declared.txt
+
+          # 3) Rebuild gravity ONLY if the declared set actually moved.
+          if [ "$CHANGED" = "1" ]; then
+            echo "pihole-adlists: declared set changed - running gravity update"
+            pihole -g || echo "pihole-adlists: gravity update failed; existing gravity.db retained" >&2
+          else
+            echo "pihole-adlists: already matches the declared set - nothing to do"
+          fi
+          exit 0
 update:
  channel: stable             # R-UPD-01
  range: ">=2024.0.0"
@@ -81,46 +81,49 @@ env:
  # clients, so per "prefer safety over completeness" it is left unset. FOLLOW-UP: once remote
  # reachability through the Gateway is confirmed, set ADVERTISE_IP: "https://plex.home.keithjackson.dev:443".
 # HARDWARE TRANSCODING — the host iGPU's render device (schema 1.8.0 `hardware`; R-PLAT-02).
-# INTENT ONLY. This says "this workload needs an Intel Quick Sync render device"; it does NOT name a
-# device path. The renderer currently satisfies it with exactly two things — a `/dev/dri` hostPath
-# volume (`type: DirectoryOrCreate`) and a read-write `/dev/dri` volumeMount on the app container —
-# and nothing else: no nodeSelector, no privileged, no resources (schemas/renderer-contract.md
-# `### hardware`). If that mechanism is later replaced by the Intel GPU device plugin
-# (`gpu.intel.com/i915`), it is a `catalog/render.py` change and THIS BLOCK DOES NOT MOVE. Do not
-# "simplify" it into a raw `/dev/...` path; that is the whole point of the enum.
+# INTENT ONLY, AND THE MECHANISM DELIBERATELY DOES NOT LIVE HERE. This says "this workload needs an
+# Intel Quick Sync render device"; it does NOT name a device path and it does NOT name a resource.
+# How the platform hands the device over is `catalog/render.py`'s business and is documented in
+# `schemas/renderer-contract.md` `### hardware` — read that for the mechanism, never this comment.
+# The payoff is proven, not theoretical: on 2026-10-01 the mechanism was swapped outright (hostPath
+# -> Intel GPU device plugin extended resource) and THIS BLOCK DID NOT MOVE — a `render.py` change
+# plus a re-render, zero catalog edits. Do not "simplify" it into a raw `/dev/...` path; that is the
+# whole point of the enum.
 #
 # THE DEVICE IS PRESENT ON EVERY PROD NODE, so nothing had to be installed and Plex can land
 # anywhere. Measured on all three nodes 2026-09-30: `/dev/dri/renderD128` present, `root:render`,
 # GID 992, mode 0660, driver `i915` bound (NOT `xe`), UHD Graphics 630 on an i7-9700T, media engines
-# `vcs0`/`vecs0` live, iGPU enabled in BIOS on each. Plex happens to run on k3s-prod-2 today, but
-# that is scheduler history rather than a requirement — this field deliberately renders no
-# nodeSelector, because placement is a separate concern with its own justification.
+# `vcs0`/`vecs0` live, iGPU enabled in BIOS on each. Measured again 2026-10-01, one layer up: all
+# three nodes advertise `gpu.intel.com/i915: 1` in `.status.allocatable`. This field renders no
+# nodeSelector; placement is a separate concern with its own justification.
+#
+# PLEX'S SCHEDULABILITY IS NOW COUPLED TO THE `intel-gpu-plugin` DaemonSet, and that is a real cost,
+# not a footnote. The current mechanism is a hard scheduling constraint: if the resource is
+# unavailable (plugin broken, removed, or not yet reconciled) this pod is UNSCHEDULABLE, and because
+# `apps-wave-0` is `wait: true` with `apps-wave-1` `dependsOn` it, that STALLS THE WHOLE APP TIER.
+# The previous hostPath mechanism degraded silently to software transcoding instead — but it never
+# actually granted access (device-cgroup EPERM, measured 2026-10-01), so there was nothing to keep.
+# THE ESCAPE HATCH IS THIS BLOCK: delete these two lines and re-render, and Plex goes back to
+# software transcoding and the tier unblocks. One field, no code change. `runbooks/plex-quicksync.md`
+# carries that procedure for whoever needs it at 2am.
 #
 # NO `podSecurityContext.supplementalGroups`, AND ADDING ONE WOULD BE INERT — do not "fix" this.
 # The device is `root:render` 0660 and this process is uid/gid 1000, so the obvious move is to push
-# the host `render` GID in as a supplemental group. It cannot work here. MEASURED IN THE RUNNING
-# CONTAINER, 2026-09-30: the image's own
-# `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` stats every char device under
-# `/dev/dri`, `groupadd`s a matching group INSIDE the container when none exists, and
+# the host `render` GID in as a supplemental group. It cannot work here, and this survived the
+# mechanism swap unchanged. MEASURED IN THE RUNNING CONTAINER, 2026-09-30 and again 2026-10-01: the
+# image's own `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` stats every char device
+# under `/dev/dri`, `groupadd`s a matching group INSIDE the container when none exists, and
 # `usermod -a -G`s the `plex` user into it — every start, nothing hardcoded, which also makes a host
-# GID renumbering self-healing (no GID is pinned anywhere in this repo). The service then execs via
-# `s6-setuidgid plex`, and `s6-setuidgid` REPLACES the process group list from `/etc/group` rather
-# than appending to it: `Plex Media Server` runs with `Groups: 100 1000`, and root's gid `0` — which
-# PID 1 (`s6-svscan`) does hold — is GONE. Anything Kubernetes supplies therefore reaches s6-svscan
-# and is discarded before Plex ever starts. Mounting the device is sufficient; the group is the
-# image's job and it does it correctly. `render.py --selftest` asserts the ABSENCE of a pod
-# securityContext for this field precisely so nobody adds an inert one back later.
+# GID renumbering self-healing (no GID is pinned anywhere in this repo) and which works the same way
+# on a PLUGIN-INJECTED device as on a bind-mounted one. Measured live, `Plex Media Server` ran with
+# `Groups: 44 100 992 1000` — 992 IS the host `render` GID, i.e. the hook had already done the job.
+# The service execs via `s6-setuidgid plex`, and `s6-setuidgid` REPLACES the process group list from
+# `/etc/group` rather than appending to it, so root's gid `0` — which PID 1 (`s6-svscan`) does hold —
+# is GONE, and anything Kubernetes supplies reaches s6-svscan and is discarded before Plex ever
+# starts. The group is the image's job and it does it correctly. `render.py --selftest` asserts the
+# ABSENCE of a pod securityContext for this field precisely so nobody adds an inert one back later.
 #
-# `DirectoryOrCreate` IS THE SAFE FAILURE MODE, not laziness. A node that comes back without the
-# device (rebuild, BIOS reset, driver regression, hardware swap) gets an empty directory and Plex
-# degrades to SOFTWARE transcoding. `Directory` would fail the mount instead — and because
-# `apps-wave-0` is `wait: true` with `apps-wave-1` `dependsOn` it, that would stall the WHOLE app
-# tier for a feature that is not load-bearing. The price, stated plainly: a missing device is
-# SILENT. This SD renders no `metricsEndpoint`, so there is no ServiceMonitor, no transcode-mode
-# metric, and no alert that could ever fire. The only proof is an operator reading the Plex
-# dashboard.
-#
-# THE LAST MILE IS A MANUAL OPERATOR STEP AND CANNOT LIVE IN THIS REPO. Mounting the device does
+# THE LAST MILE IS A MANUAL OPERATOR STEP AND CANNOT LIVE IN THIS REPO. Handing over the device does
 # not enable hardware transcoding — Plex's own `HardwareAcceleratedCodecs` preference does, and it
 # is ABSENT from `Preferences.xml` today (the toggle has never been on). NO environment variable can
 # set it: a grep of the running image's entire init surface (`/etc/cont-init.d`, `/etc/services.d`,
@@ -128,8 +131,7 @@ env:
 # and the six prefs the image does write (`MachineIdentifier`, `ProcessedMachineIdentifier`,
 # `PlexOnlineToken`, `customConnections`, `allowedNetworks`, `TranscoderTempDirectory`) are all
 # unrelated. So there is no GitOps path to the toggle; an operator clicks it once in the web UI.
-# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took and the
-# device-cgroup failure mode to expect first.
+# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took.
 hardware:
  gpu: intel-quicksync
 exposure: public              # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET)
@@ -100,6 +100,9 @@ dependencies:
  - to: jackett                 # arr → indexer proxy (docs/13 dependency graph; R-NET-02 explicit allow)
    port: 9117
    protocol: tcp
+  - to: plex
+    port: 32400                 # arr -> plex: required since library is on NFS which does not provide file CRUD notifications
+    protocol: tcp
 ingressRoutes:
  - host: radarr.${BASE_DOMAIN}
    port: 7878                  # Radarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
@@ -166,11 +166,14 @@ OAUTH2_PROXY_METRICS_PORT = 44180
 # --- Hardware capability: `hardware.gpu: intel-quicksync` ------------------------------
 # The ServiceDefinition declares the CAPABILITY (schemas/servicedefinition.schema.json
 # `hardware.gpu`); everything below is the MECHANISM that currently satisfies it and is
-# deliberately confined to this file. Swapping to the Intel GPU device plugin
-# (`gpu.intel.com/i915` resource requests instead of a hostPath) is a change here and
-# nowhere else -- no catalog entry moves.
-DRI_VOLUME_NAME = "dri"
-DRI_DEVICE_DIR = "/dev/dri"
+# deliberately confined to this file. This file is the ONLY place the mechanism appears,
+# which is what let it be swapped on 2026-10-01 -- from a `/dev/dri` hostPath to the Intel
+# GPU device plugin's extended resource -- with ZERO ServiceDefinition edits.
+#
+# The extended resource is advertised by the `intel-gpu-plugin` DaemonSet
+# (platform/intel-gpu-plugin/, pinned intel/intel-gpu-plugin:0.37.0). MEASURED on all three
+# prod nodes 2026-10-01: `.status.allocatable["gpu.intel.com/i915"] == 1`.
+QUICKSYNC_RESOURCE = "gpu.intel.com/i915"

 # --- API routes: 401 instead of 302, PER APP ------------------------------------------
 # oauth2-proxy answers an unauthenticated request with a 302 to Keycloak. Its built-in AJAX
@@ -269,7 +272,15 @@ def apply_defaults(sd: dict) -> dict:
    # object that actually runs on the cluster is not the one this SD renders (keycloak: the
    # Keycloak operator runs a StatefulSet, so a Deployment-shaped rule would `absent()` forever).
    lw = dict(obs.get("livenessWorkload") or {})
-    lw.setdefault("kind", "VirtualMachineInstance" if s["kind"] == "vm" else "Deployment")
+    # The default must track what is ACTUALLY rendered, or the `up` SLI silently dies: a
+    # Deployment-shaped rule against a StatefulSet `absent()`s forever, which is an alert that can
+    # never fire — the exact failure mode the keycloak override was added for. Deriving it from
+    # `availability.replicas` means bumping the replica count cannot leave a stale rule behind.
+    lw.setdefault(
+        "kind",
+        "VirtualMachineInstance" if s["kind"] == "vm"
+        else ("StatefulSet" if _is_statefulset(s) else "Deployment"),
+    )
    lw.setdefault("name", s["name"])
    obs["livenessWorkload"] = lw
    s["observability"] = obs
@@ -472,6 +483,37 @@ def r_bulk_pvs(s: dict) -> list:
    return out


+def _replicas(s: dict) -> int:
+    """Declared replica count (schema 1.10.0 `availability.replicas`). Default 1."""
+    return int((s.get("availability") or {}).get("replicas", 1) or 1)
+
+
+def _is_statefulset(s: dict) -> bool:
+    """True when this container workload must render a StatefulSet rather than a Deployment.
+
+    THE DECIDING CONSTRAINT IS THE VOLUME, not a preference. A longhorn storage entry renders
+    ReadWriteOnce (r_pvcs), and one RWO PVC cannot be attached by pods on two different nodes --
+    which is exactly what `availability.replicas > 1` plus the node anti-affinity below asks for.
+    A 2-replica Deployment over a shared RWO claim does not degrade, it simply never schedules the
+    second pod. A StatefulSet's volumeClaimTemplates give each replica its OWN claim, which is the
+    only shape that satisfies both at once.
+
+    A replicas-1 service keeps rendering a Deployment so every pre-existing golden is byte-identical.
+    """
+    return s["kind"] == "container" and _replicas(s) > 1
+
+
+def _sts_claim_volumes(s: dict) -> list:
+    """Storage entries that become per-replica volumeClaimTemplates on a StatefulSet.
+
+    Only the dynamically-provisioned (longhorn) entries. A `bulk` entry is a STATIC, pre-bound NFS
+    PV over data that already exists outside the cluster and is shared read-only by every consumer
+    (r_bulk_pvs) -- per-replica copies of it are neither possible nor wanted, so it stays a
+    standalone PVC and an ordinary pod volume.
+    """
+    return [v for v in s["storage"] if v.get("class", "longhorn") != "bulk"]
+
+
 def r_pvcs(s: dict) -> list:
    """One PVC per storage entry (R-STATE-01/02/04).

@@ -479,8 +521,15 @@ def r_pvcs(s: dict) -> list:
    bulk     -> the STATIC external NFS PV rendered by r_bulk_pvs, pre-bound by volumeName.
    """
    out = []
+    # On a StatefulSet the longhorn claims are NOT standalone objects — they are
+    # volumeClaimTemplates, so each replica gets its own (see _is_statefulset). Emitting both would
+    # leave an orphan PVC that nothing mounts and that `prune` would later delete out from under
+    # nobody. `bulk` is unaffected and still rendered here.
+    sts = _is_statefulset(s)
    for vol in s["storage"]:
        cls = vol.get("class", "longhorn")
+        if sts and cls != "bulk":
+            continue
        if cls == "bulk":
            # Mirror image of the PV: the same explicitly-empty class (so nothing dynamically
            # provisions), the same access mode, and volumeName completing the two-sided binding.
@@ -561,6 +610,12 @@ def r_deployment(s: dict) -> dict:
        if vol.get("readOnly"):
            mount["readOnly"] = True
        volume_mounts.append(mount)
+        # On a StatefulSet a longhorn volume is supplied by its volumeClaimTemplate, whose
+        # metadata.name IS the pod volume name. Adding it to pod.volumes as well is a conflict
+        # (duplicate volume name), so it is skipped here and emitted below. The MOUNT is unchanged —
+        # the container cannot tell the difference, which is what keeps this transparent to the app.
+        if _is_statefulset(s) and vol.get("class", "longhorn") != "bulk":
+            continue
        volumes.append(
            {
                "name": vol["name"],
@@ -571,41 +626,59 @@ def r_deployment(s: dict) -> dict:
    # workload needs the host iGPU's render node for hardware video transcode. The SD says
    # only that; the wiring below is the platform's current mechanism for satisfying it.
    #
-    # `type: DirectoryOrCreate` is DELIBERATE and load-bearing -- do NOT "tighten" it to
-    # `Directory`. A node that comes back without the render device (rebuild, BIOS change,
-    # driver regression, hardware swap) must degrade the workload to SOFTWARE transcoding,
-    # not fail the mount: `apps-wave-0` is `wait: true` and `apps-wave-1` dependsOn it, so a
-    # failed hostPath mount stalls the entire app tier for the sake of a feature that is not
-    # load-bearing. `DirectoryOrCreate` turns that failure mode into "transcodes in
-    # software" instead of "the app tier does not reconcile". /dev is tmpfs, so the empty
-    # directory the kubelet creates on such a node leaves nothing durable behind.
+    # THE MECHANISM IS AN EXTENDED RESOURCE, AND THE hostPath IT REPLACED WAS MEASURED NOT
+    # TO WORK. Do NOT "simplify" this back into a `/dev/dri` hostPath volume. That was the
+    # original implementation (merged 2026-09-30) and it was proven broken in the live Plex
+    # pod on k3s-prod-2 on 2026-10-01: the device node was VISIBLE in the container
+    # (`crw-rw---- root video2 226,128 renderD128`) and could not be opened --
+    # `uid 0 O_RDWR -> EPERM` and `uid 1000 O_RDWR -> EPERM`. EPERM from the DEVICE CGROUP,
+    # not EACCES from file permissions, proven three ways: root on the HOST opens the same
+    # device rc=0; the node is unified `cgroup2fs`; and the container's runc `config.json`
+    # read `linux.resources.devices: [{allow: False, access: 'rwm'}]` with
+    # `linux.devices: None` -- one deny-all rule, no allow entry for major 226. A hostPath is
+    # a filesystem operation: it binds the device node in but adds NO cgroup allow rule,
+    # because Kubernetes cannot infer one from a volume.
    #
-    # The mount is deliberately NOT readOnly: VA-API opens the render node read-write.
+    # The device plugin is what produces the allow rule. Confirmed in the v0.37.0 source:
+    # `createDeviceSpecsFromDrmFiles` emits `DeviceSpec{..., Permissions: "rw"}`, which the
+    # kubelet turns into both the device injection and the matching cgroup allow.
+    #
+    # `limits`, NOT `requests`, AND THAT IS NOT A STYLE CHOICE: an extended resource must be
+    # expressed in `limits`. Kubernetes mirrors `requests` from `limits` automatically, and a
+    # requests-only extended resource is invalid. App container only, never a sidecar -- the
+    # oauth2-proxy sidecar does no transcoding.
+    #
+    # THE FAILURE MODE THIS TRADED INTO, STATED PLAINLY -- it is worse in blast radius, and
+    # unavoidable. BEFORE: a missing device meant `DirectoryOrCreate` mounted an empty dir,
+    # Plex ran and SILENTLY software-transcoded (invisible, but harmless to everything else).
+    # NOW: if the resource is unavailable (plugin broken, removed, or not yet reconciled) the
+    # pod is UNSCHEDULABLE, and because `apps-wave-0` is `wait: true` with `apps-wave-1`
+    # dependsOn it, that STALLS THE APP TIER. There is no graceful-degradation variant: an
+    # extended resource is a hard scheduling constraint, and it is also the only thing that
+    # produces the cgroup allow rule. Plex's schedulability is now coupled to the
+    # intel-gpu-plugin DaemonSet; that is the price of hardware transcoding working at all.
+    # ESCAPE HATCH (one field, no code change): delete the `hardware:` block from
+    # `catalog/plex.yaml` and re-render -- Plex returns to software transcoding and the tier
+    # unblocks. runbooks/plex-quicksync.md carries it.
    #
    # `podSecurityContext.supplementalGroups` is DELIBERATELY NOT SET here, and adding it
-    # would be a regression, not a hardening. MEASURED in the running container on prod-2,
-    # 2026-09-30: the image's own cont-init hook
+    # would be a regression, not a hardening. MEASURED in the running container on prod-2
+    # (2026-09-30, re-measured 2026-10-01): the image's own cont-init hook
    # `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` `find`s every char device
    # under /dev/dri, `stat -c '%g'` each one's GID, `groupadd`s a matching group INSIDE the
    # container when none exists, and `usermod -a -G` the app user into it -- every start,
-    # nothing hardcoded. cont-init.d completes before services.d, so the account database
-    # s6 reads at exec time already carries the group. The service is then started with
-    # `exec s6-setuidgid plex ...`, and `s6-setuidgid` REPLACES the process group list from
-    # /etc/group rather than appending to it: `Plex Media Server` runs with
-    # `Uid: 1000  Gid: 1000  Groups: 100 1000` -- root's gid 0, which PID 1 (s6-svscan) does
-    # have, is GONE. Anything Kubernetes puts in supplementalGroups reaches s6-svscan and is
-    # then discarded before the app ever starts. MOUNTING THE DEVICE IS SUFFICIENT; the
-    # group handling is the image's job and it does it correctly. A supplementalGroups entry
-    # here would be inert AND would mislead whoever debugs this next.
+    # nothing hardcoded. It does exactly the same thing to the PLUGIN-INJECTED device, which
+    # is why this swap needs no group work either. cont-init.d completes before services.d,
+    # so the account database s6 reads at exec time already carries the group: measured live,
+    # `Plex Media Server` ran with `Groups: 44 100 992 1000`, where 992 is the host `render`
+    # GID. The service is started with `exec s6-setuidgid plex ...`, and `s6-setuidgid`
+    # REPLACES the process group list from /etc/group rather than appending to it, so
+    # anything Kubernetes puts in supplementalGroups reaches s6-svscan and is discarded
+    # before the app ever starts. A supplementalGroups entry here would be inert AND would
+    # mislead whoever debugs this next.
    hw_gpu = (s.get("hardware") or {}).get("gpu")
    if hw_gpu == "intel-quicksync":
-        volume_mounts.append({"name": DRI_VOLUME_NAME, "mountPath": DRI_DEVICE_DIR})
-        volumes.append(
-            {
-                "name": DRI_VOLUME_NAME,
-                "hostPath": {"path": DRI_DEVICE_DIR, "type": "DirectoryOrCreate"},
-            }
-        )
+        container["resources"] = {"limits": {QUICKSYNC_RESOURCE: 1}}

    if volume_mounts:
        container["volumeMounts"] = volume_mounts
@@ -640,20 +713,80 @@ def r_deployment(s: dict) -> dict:
    if ann:
        tmpl_meta["annotations"] = ann

-    spec = {"replicas": 1}
-    # An app with persistent storage mounts a ReadWriteOnce Longhorn PVC. The default
-    # RollingUpdate strategy tries to start the NEW pod before terminating the old one, but
-    # the old pod still holds the RWO volume -> the new pod is stuck ContainerCreating and
-    # the rollout deadlocks (single-replica app can never hand the volume over). Recreate
-    # terminates the old pod first, freeing the PVC for the new one (R-STATE-01/02).
-    if s["storage"]:
+    sts = _is_statefulset(s)
+
+    # NODE ANTI-AFFINITY, and it is REQUIRED rather than preferred. `availability.replicas > 1`
+    # exists to survive the loss of one node; `preferred` would let both replicas land on the same
+    # node and quietly deliver none of that, with every object reporting healthy. The cost is real
+    # and is accepted: with N replicas the workload needs N schedulable nodes, so on a 3-node
+    # cluster losing TWO nodes leaves a replica Pending and — because `apps-wave-0` is `wait: true`
+    # with `apps-wave-1` dependsOn it — STALLS THE APP TIER. That coincides with having already lost
+    # etcd quorum on a 3-node control plane, i.e. the cluster is not reconciling anything at that
+    # point anyway, so the stall is a symptom rather than a new failure.
+    if sts:
+        pod_spec["affinity"] = {
+            "podAntiAffinity": {
+                "requiredDuringSchedulingIgnoredDuringExecution": [
+                    {
+                        "labelSelector": {
+                            "matchLabels": {"app.kubernetes.io/name": s["name"]}
+                        },
+                        "topologyKey": "kubernetes.io/hostname",
+                    }
+                ]
+            }
+        }
+
+    spec = {"replicas": _replicas(s)}
+    if sts:
+        # serviceName is a required StatefulSet field. It points at the service rendered by
+        # r_service; that Service is a normal ClusterIP rather than headless, so per-pod DNS
+        # (<pod>.<svc>) does NOT resolve — and nothing here needs it. These replicas are
+        # interchangeable resolvers behind one VIP, not peers that address each other.
+        spec["serviceName"] = s["name"]
+        # Parallel, not the OrderedReady default. OrderedReady starts replica N+1 only after N is
+        # Ready, so a cold start costs the sum of every replica's first-boot work instead of the
+        # max — and `apps-wave-0` has `timeout: 5m`. There is no ordering requirement between
+        # replicas to protect (no leader, no migration, no shared volume), so serialising buys
+        # nothing and risks the wave timing out on a rebuild.
+        spec["podManagementPolicy"] = "Parallel"
+    elif s["storage"]:
+        # An app with persistent storage mounts a ReadWriteOnce Longhorn PVC. The default
+        # RollingUpdate strategy tries to start the NEW pod before terminating the old one, but
+        # the old pod still holds the RWO volume -> the new pod is stuck ContainerCreating and
+        # the rollout deadlocks (single-replica app can never hand the volume over). Recreate
+        # terminates the old pod first, freeing the PVC for the new one (R-STATE-01/02).
+        #
+        # A StatefulSet needs none of this and cannot express it: each replica owns its own claim,
+        # so a rolling update hands nothing over, and `strategy` is not a StatefulSet field at all.
        spec["strategy"] = {"type": "Recreate"}
    spec["selector"] = {"matchLabels": {"app.kubernetes.io/name": s["name"]}}
    spec["template"] = {"metadata": tmpl_meta, "spec": pod_spec}
+    if sts:
+        # Per-replica claims. metadata.name MUST equal the pod volume name the container mounts
+        # (see the volumes loop above); the resulting PVCs are <name>-<sts>-<ordinal>, e.g.
+        # config-pihole-0. Same tier and access mode a standalone longhorn PVC gets from r_pvcs —
+        # this changes WHO owns the claim, never what it is.
+        tmpls = []
+        for vol in _sts_claim_volumes(s):
+            vspec = {"accessModes": ["ReadWriteOnce"], "storageClassName": "longhorn"}
+            if vol.get("size"):
+                vspec["resources"] = {"requests": {"storage": vol["size"]}}
+            tmpls.append(
+                {
+                    "metadata": {
+                        "name": vol["name"],
+                        "labels": {"app.kubernetes.io/name": s["name"]},
+                    },
+                    "spec": vspec,
+                }
+            )
+        if tmpls:
+            spec["volumeClaimTemplates"] = tmpls

    return {
        "apiVersion": "apps/v1",
-        "kind": "Deployment",
+        "kind": "StatefulSet" if sts else "Deployment",
        "metadata": {
            "name": s["name"],
            "namespace": s["name"],
@@ -663,6 +796,35 @@ def r_deployment(s: dict) -> dict:
    }


+def r_pdb(s: dict):
+    """PodDisruptionBudget for a multi-replica workload (availability.replicas > 1).
+
+    minAvailable 1, not `maxUnavailable: 1`. The budget's job here is to stop a VOLUNTARY
+    disruption — a node drain for maintenance, a cluster upgrade — from taking the last replica
+    with it; `minAvailable: 1` says that whatever else happens, one must keep serving. It does
+    NOT protect against involuntary disruption (a node dying), which is what the anti-affinity
+    and the replica count are for.
+
+    Rendered only when replicas > 1: a PDB over a single-replica workload is actively harmful, as
+    `minAvailable: 1` would then block every drain of its node forever.
+    """
+    if not _is_statefulset(s):
+        return None
+    return {
+        "apiVersion": "policy/v1",
+        "kind": "PodDisruptionBudget",
+        "metadata": {
+            "name": s["name"],
+            "namespace": s["name"],
+            "labels": {"app.kubernetes.io/name": s["name"]},
+        },
+        "spec": {
+            "minAvailable": 1,
+            "selector": {"matchLabels": {"app.kubernetes.io/name": s["name"]}},
+        },
+    }
+
+
 def r_service(s: dict):
    ports = _service_ports(s)
    # A declared metrics endpoint becomes a NAMED Service port, because that name is the only
@@ -1796,7 +1958,13 @@ def render(sd: dict, image_automation: bool = True, inbound: list | None = None)
            )
        for pvc in r_pvcs(s):
            docs.append((f"PVC {pvc['metadata']['name']} (R-STATE)", pvc))
-        docs.append(("Deployment (R-PLAT-02)", r_deployment(s)))
+        wl = r_deployment(s)
+        docs.append((f"{wl['kind']} (R-PLAT-02)", wl))
+        pdb = r_pdb(s)
+        if pdb:
+            docs.append(
+                ("PodDisruptionBudget — keep one replica through a drain (availability.replicas)", pdb)
+            )
        svc = r_service(s)
        if svc:
            docs.append(("Service", svc))
@@ -1981,6 +2149,24 @@ EXTRA_UNITS: dict = {
            "pihole-dns.yaml",
        ],
    },
+    # Plex's remote-access exposure: the raw tcp/32400 LoadBalancer Service on a pinned VIP, plus the
+    # CiliumNetworkPolicy that admits off-cluster clients to it (apps/wave-0/plex-remote.yaml).
+    # `above: False` co-locates it in plex's OWN wave — these are the plex pod's own Service and its
+    # own policy, applied in the same wave as the Deployment they select, exactly as the rendered
+    # Service and CNPs in plex.yaml are. Nothing in a higher wave gates on it.
+    #
+    # It is a non-catalog unit for the same reason pihole-dns is: the ServiceDefinition schema cannot
+    # express a LoadBalancer Service or a pinned lbipam VIP (additionalProperties is false, and
+    # `ingressRoutes` only renders HTTPRoutes on the L7 Gateway, which cannot carry Plex's raw port).
+    # Plex's own settings stay in the catalog as env on catalog/plex.yaml; the only thing living
+    # outside it is the exposure the schema has no word for.
+    "plex-remote": {
+        "anchor": "plex",
+        "above": False,
+        "files": [
+            "plex-remote.yaml",
+        ],
+    },
 }


@@ -2227,6 +2413,132 @@ def _selftest() -> int:
    check("Namespace" in kinds, "container: missing Namespace")
    check("NetworkPolicy" in kinds, "container: missing default-deny NetworkPolicy")
    check("Deployment" in kinds, "container: missing Deployment")
+    check(
+        "PodDisruptionBudget" not in kinds,
+        "replicas-1 container must render NO PodDisruptionBudget (minAvailable 1 over a single "
+        "replica blocks every drain of its node forever)",
+    )
+
+    # 1b) availability.replicas (schema 1.10.0). The DEFAULT path must be untouched, and the >1
+    # path must change workload kind, claim ownership, spreading, budget AND the SLI metric
+    # together — each of these has a silent-failure mode if it is the one that gets forgotten.
+    base_av = {
+        "name": "av",
+        "source": {"image": "nginx:1"},
+        "storage": [{"name": "config", "size": "1Gi"}],
+        "observability": {"metrics": True, "slis": ["up"]},
+    }
+    one = render({**base_av, "availability": {"replicas": 1}})
+    one_kinds = [d[1]["kind"] for d in one]
+    check(
+        "Deployment" in one_kinds and "StatefulSet" not in one_kinds,
+        "availability.replicas=1 must still render a Deployment (back-compat)",
+    )
+    check(
+        "PersistentVolumeClaim" in one_kinds,
+        "availability.replicas=1 must still render a standalone PVC",
+    )
+    check(
+        "PodDisruptionBudget" not in one_kinds,
+        "availability.replicas=1 must render no PodDisruptionBudget",
+    )
+
+    many = render({**base_av, "availability": {"replicas": 2}})
+    many_kinds = [d[1]["kind"] for d in many]
+    check(
+        "StatefulSet" in many_kinds and "Deployment" not in many_kinds,
+        "availability.replicas>1 must render a StatefulSet, not a Deployment (a shared RWO claim "
+        "cannot be attached on two nodes, so a multi-replica Deployment never schedules)",
+    )
+    check(
+        "PersistentVolumeClaim" not in many_kinds,
+        "availability.replicas>1 must NOT also render a standalone longhorn PVC — it would be an "
+        "orphan that nothing mounts",
+    )
+    sts = next(d[1] for d in many if d[1]["kind"] == "StatefulSet")
+    check(sts["spec"].get("replicas") == 2, "StatefulSet must carry the declared replica count")
+    check(
+        "strategy" not in sts["spec"],
+        "StatefulSet must not carry `strategy` (not a StatefulSet field; Recreate is a "
+        "Deployment-only workaround for handing over one RWO volume)",
+    )
+    check(
+        sts["spec"].get("podManagementPolicy") == "Parallel",
+        "StatefulSet must set podManagementPolicy=Parallel — OrderedReady serialises first-boot "
+        "work across replicas and apps-wave-0 has a 5m timeout",
+    )
+    check(sts["spec"].get("serviceName") == "av", "StatefulSet must set serviceName")
+    tmpls = sts["spec"].get("volumeClaimTemplates") or []
+    check(
+        [t["metadata"]["name"] for t in tmpls] == ["config"],
+        "StatefulSet volumeClaimTemplate name must equal the pod volume name the container mounts, "
+        "or the pod gets a duplicate/unresolvable volume",
+    )
+    check(
+        tmpls and tmpls[0]["spec"]["accessModes"] == ["ReadWriteOnce"]
+        and tmpls[0]["spec"]["storageClassName"] == "longhorn",
+        "StatefulSet claim template must keep the same tier/access mode as the standalone PVC",
+    )
+    sts_vols = {v["name"] for v in sts["spec"]["template"]["spec"].get("volumes", [])}
+    check(
+        "config" not in sts_vols,
+        "a claim-template volume must NOT also appear in pod.volumes (duplicate volume name)",
+    )
+    mounts = sts["spec"]["template"]["spec"]["containers"][0].get("volumeMounts") or []
+    check(
+        any(m["name"] == "config" for m in mounts),
+        "the container must still mount the volume by name on a StatefulSet (transparent to the app)",
+    )
+    aff = (sts["spec"]["template"]["spec"].get("affinity") or {}).get("podAntiAffinity") or {}
+    req = aff.get("requiredDuringSchedulingIgnoredDuringExecution") or []
+    check(
+        req and req[0].get("topologyKey") == "kubernetes.io/hostname",
+        "availability.replicas>1 must render REQUIRED per-node anti-affinity — `preferred` can put "
+        "every replica on one node and deliver none of the intended node-failure tolerance",
+    )
+    pdbs = [d[1] for d in many if d[1]["kind"] == "PodDisruptionBudget"]
+    check(len(pdbs) == 1, "availability.replicas>1 must render exactly one PodDisruptionBudget")
+    check(
+        pdbs and pdbs[0]["spec"].get("minAvailable") == 1,
+        "PodDisruptionBudget must be minAvailable=1 (keep one replica serving through a drain)",
+    )
+    pr_many = next((d[1] for d in many if d[1]["kind"] == "PrometheusRule"), None)
+    check(
+        pr_many is not None
+        and "kube_statefulset_status_replicas_ready"
+        in pr_many["spec"]["groups"][0]["rules"][0]["expr"],
+        "the up SLI must follow the rendered workload kind — a Deployment-shaped rule against a "
+        "StatefulSet absent()s forever and can never fire",
+    )
+
+    # A bulk volume is shared external NFS: it must stay a standalone PVC even on a StatefulSet.
+    mixed = render(
+        {
+            **base_av,
+            "availability": {"replicas": 2},
+            "storage": [
+                {"name": "config", "size": "1Gi"},
+                {"name": "media", "class": "bulk", "export": "/mnt/x", "size": "1Ti"},
+            ],
+        }
+    )
+    m_sts = next(d[1] for d in mixed if d[1]["kind"] == "StatefulSet")
+    check(
+        [t["metadata"]["name"] for t in m_sts["spec"]["volumeClaimTemplates"]] == ["config"],
+        "only longhorn volumes become claim templates; a bulk share must not be per-replica",
+    )
+    check(
+        any(
+            d[1]["kind"] == "PersistentVolumeClaim"
+            and d[1]["metadata"]["name"] == "av-media"
+            for d in mixed
+        ),
+        "a bulk PVC must still be rendered standalone on a StatefulSet",
+    )
+    check(
+        "media" in {v["name"] for v in m_sts["spec"]["template"]["spec"].get("volumes", [])},
+        "a bulk volume must still appear in pod.volumes on a StatefulSet",
+    )

    ns = next(d[1] for d in docs if d[1]["kind"] == "Namespace")
    check(
@@ -2650,10 +2962,13 @@ def _selftest() -> int:
        "an SD without podSecurityContext must not gain a pod securityContext",
    )

-    # 6b) hardware.gpu: intel-quicksync -> a /dev/dri hostPath volume and a read-write mount
-    #     on the app container, and NOTHING ELSE. The SD declares the CAPABILITY; these
-    #     assertions pin the MECHANISM that currently satisfies it -- including what it
-    #     deliberately does NOT emit -- so a change to it is a deliberate, visible edit.
+    # 6b) hardware.gpu: intel-quicksync -> a `gpu.intel.com/i915` LIMIT on the app container
+    #     and NOTHING ELSE: no volume, no volumeMount, no securityContext, no nodeSelector.
+    #     The SD declares the CAPABILITY; these assertions pin the MECHANISM that currently
+    #     satisfies it -- including what it deliberately does NOT emit -- so a change to it
+    #     is a deliberate, visible edit. The previous mechanism (a /dev/dri hostPath) is
+    #     asserted ABSENT: it was measured not to work (EPERM from the device cgroup; see
+    #     the emission site), and the absence assertion is what stops it coming back.
    hw_docs = render(
        {
            "name": "hw",
@@ -2664,24 +2979,36 @@ def _selftest() -> int:
    hw_pod = next(
        d[1] for d in hw_docs if d[1]["kind"] == "Deployment"
    )["spec"]["template"]["spec"]
+    hw_app = hw_pod["containers"][0]
+    # The whole mechanism, exactly: an extended resource in `limits`. `requests` is NOT set
+    # by the renderer -- Kubernetes mirrors it from `limits`, and a requests-only extended
+    # resource is invalid. Asserting the exact dict is deliberate: it fails if anyone adds a
+    # CPU/memory resources feature through this field (that would be scope creep, and this
+    # field is about the DEVICE only).
    check(
-        {
-            "name": "dri",
-            "hostPath": {"path": "/dev/dri", "type": "DirectoryOrCreate"},
-        }
-        in hw_pod.get("volumes", []),
-        "hardware.gpu=intel-quicksync must render a /dev/dri hostPath volume "
-        "with type DirectoryOrCreate (a GPU-less node must degrade, not fail the mount)",
+        hw_app.get("resources") == {"limits": {"gpu.intel.com/i915": 1}},
+        "hardware.gpu=intel-quicksync must render resources.limits "
+        "{gpu.intel.com/i915: 1} on the app container, and nothing else in resources",
    )
-    hw_mounts = hw_pod["containers"][0].get("volumeMounts", [])
+    # The retired mechanism must stay retired. A hostPath binds the device node in but adds
+    # no device-cgroup allow rule, so the open() was EPERM with the node plainly visible.
    check(
-        {"name": "dri", "mountPath": "/dev/dri"} in hw_mounts,
-        "hardware.gpu=intel-quicksync must mount /dev/dri READ-WRITE on the app container",
+        not any(
+            "hostPath" in v or v.get("name") == "dri" for v in hw_pod.get("volumes", [])
+        )
+        and not any(
+            m.get("mountPath") == "/dev/dri" or m.get("name") == "dri"
+            for m in hw_app.get("volumeMounts", [])
+        ),
+        "hardware.gpu=intel-quicksync must render NO /dev/dri hostPath volume and no "
+        "/dev/dri mount (the hostPath mechanism was measured not to work: device-cgroup "
+        "EPERM; the device plugin's DeviceSpec Permissions: rw is what grants access)",
    )
-    # The device mount is the WHOLE feature: no pod securityContext is invented. The image's
-    # own cont-init hook adds the app user to the device's group inside the container, and
-    # s6-setuidgid discards any Kubernetes-supplied supplementalGroups anyway (see the
-    # emission site). Asserting the absence is what stops someone adding an inert one back.
+    # The resource is the WHOLE feature: no pod securityContext is invented. The image's own
+    # cont-init hook adds the app user to the (plugin-injected) device's group inside the
+    # container, and s6-setuidgid discards any Kubernetes-supplied supplementalGroups anyway
+    # (see the emission site). Asserting the absence is what stops someone adding an inert
+    # one back.
    check(
        "securityContext" not in hw_pod,
        "hardware.gpu must NOT invent a pod securityContext (supplementalGroups is inert "
@@ -2689,10 +3016,12 @@ def _selftest() -> int:
    )
    # Nothing privileged, and no node pinning: Kyverno restrict-host-namespaces (Enforce)
    # rejects `privileged: true` in an app namespace, and placement is a separate concern
-    # with its own justification -- this field is about the DEVICE only.
+    # with its own justification -- this field is about the DEVICE only. (The extended
+    # resource now constrains placement to nodes the plugin advertises, which is the
+    # scheduler's own doing, not a nodeSelector this renderer writes.)
    check(
        "privileged" not in hw_pod.get("securityContext", {})
-        and "securityContext" not in hw_pod["containers"][0]
+        and "securityContext" not in hw_app
        and "nodeSelector" not in hw_pod,
        "hardware.gpu must emit no privileged securityContext and no nodeSelector",
    )
@@ -2705,8 +3034,10 @@ def _selftest() -> int:
    check(
        "volumes" not in nohw_pod
        and "volumeMounts" not in nohw_pod["containers"][0]
+        and "resources" not in nohw_pod["containers"][0]
        and "securityContext" not in nohw_pod,
-        "an SD without `hardware` must gain no dri volume, no mount and no securityContext",
+        "an SD without `hardware` must gain no resources block, no volume, no mount and no "
+        "securityContext",
    )
    # An SD that declares BOTH keeps its own podSecurityContext exactly as written -- the
    # hardware field neither adds to it nor rewrites it.
@@ -2723,9 +3054,35 @@ def _selftest() -> int:
        if d[1]["kind"] == "Deployment"
    )["spec"]["template"]["spec"]
    check(
-        hwm_pod["securityContext"] == {"fsGroup": 1001, "supplementalGroups": [44]},
+        hwm_pod["securityContext"] == {"fsGroup": 1001, "supplementalGroups": [44]}
+        and hwm_pod["containers"][0].get("resources")
+        == {"limits": {"gpu.intel.com/i915": 1}},
        "hardware.gpu must leave the SD's own podSecurityContext untouched",
    )
+    # The resource goes on the APP container only. A forward-auth app renders an oauth2-proxy
+    # sidecar, which does no transcoding and must not consume one of the node's i915 slots.
+    hws_containers = next(
+        d[1]
+        for d in render(
+            {
+                "name": "hws",
+                "source": {"image": "a:1"},
+                "hardware": {"gpu": "intel-quicksync"},
+                "ingressRoutes": [{"host": "hws.home.keithjackson.dev", "port": 8080}],
+                "auth": {"mode": "forward-auth", "groups": ["media-users"]},
+            }
+        )
+        if d[1]["kind"] == "Deployment"
+    )["spec"]["template"]["spec"]["containers"]
+    check(
+        len(hws_containers) == 2
+        and hws_containers[0].get("resources")
+        == {"limits": {"gpu.intel.com/i915": 1}}
+        and "gpu.intel.com/i915"
+        not in hws_containers[1].get("resources", {}).get("limits", {}),
+        "hardware.gpu must put the i915 limit on the APP container only, never on the "
+        "oauth2-proxy sidecar",
+    )

    # 6c) forward-auth (Option B): an exposed forward-auth app gets an oauth2-proxy sidecar that
    #     fronts the Gateway path (fail-closed), while oidc/native/none apps are unchanged.
@@ -100,6 +100,9 @@ dependencies:
  - to: jackett                 # arr → indexer proxy (docs/13 dependency graph; R-NET-02 explicit allow)
    port: 9117
    protocol: tcp
+  - to: plex                    # arr -> plex: required since library is on NFS which does not provide file CRUD notifications
+    port: 32400
+    protocol: tcp
 ingressRoutes:
  - host: sonarr.${BASE_DOMAIN}
    port: 8989                  # Sonarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
@@ -0,0 +1,194 @@
+#!/usr/bin/env python3
+"""ci/approval-override.py — select the admin-override on a PR, if there is a valid one.
+
+This script decides NOTHING about authority. It parses a PR's comments and reviews and returns
+the single override *candidate* that ci/approval-policy.sh should then try to authorise against
+the live Gitea permission API. Keeping the two apart is the point:
+
+  * parsing is pure, offline, and therefore TESTABLE (ci/tests/approval-override.test.sh);
+  * authorisation needs the API and must stay in the stage that fails closed.
+
+A candidate is a comment OR a review whose body contains, AT THE START OF A LINE, the directive
+followed by a non-empty reason:
+
+    /approve-override the gate cannot gate itself and I am the only human reviewer
+
+TWO SOURCES, AND WHY THAT IS NOT BELT-AND-BRACES. Measured 2026-10-02 on PR #38: the override was
+posted as an ordinary PR comment, was well-formed, was newer than the head commit — and the stage
+still blocked, because CI could not READ it. `SETTINGS_API_TOKEN` is minted with scopes
+`[read:repository, read:admin]`, and Gitea puts /issues/{n}/comments behind the *issue* scope, so
+the request came back as an error OBJECT rather than an array. `/pulls/{n}/reviews` is under
+read:repository and was readable in the same run. So:
+
+  * a REVIEW body works with the token CI has today (leave a review of type "Comment");
+  * a COMMENT body works once the token carries read:issue (see infra/gitea/set-ci-token.sh).
+
+Accepting both means the hatch is not hostage to one API scope. A source that cannot be read
+contributes no candidates and is reported on stderr; it can only cause a false BLOCK, never a
+false pass, which is why degrading it is safe and blocking on it was not necessary.
+
+WHY EACH RULE IS HERE
+
+  * START OF A LINE, not "anywhere in the body". Otherwise quoting someone else's override
+    ("you could just /approve-override this") re-arms it. Lines beginning with '>' are DROPPED
+    rather than unquoted, for exactly that reason.
+  * A REASON IS MANDATORY. A bare directive records that a review was waived but not why, which
+    is the difference between an audit trail and a rubber stamp.
+  * STALE OVERRIDES ARE DISCARDED. An override earned at commit A must not silently cover commit
+    B. Any candidate created at or before the head commit's timestamp is dropped — the same
+    semantics Gitea's own `dismiss_stale_approvals` applies to approvals. Push new commits and
+    you must re-override.
+  * A PENDING REVIEW IS NOT SUBMITTED. It is a private draft visible only to its author, so it is
+    not an act of anything and is dropped.
+  * THE LATEST CANDIDATE WINS, across both sources, so a later override supersedes an earlier one.
+
+Output: one TSV line (login, timestamp, source, reason) on stdout, or nothing at all when there is
+no valid candidate. Never exits non-zero for "no override" — absence is a normal answer and the
+caller must not be able to confuse it with a parse crash.
+
+Input, all via env so nothing lands in `ps`:
+  COMMENTS_JSON  the Gitea /issues/{n}/comments array (or an error object; or absent)
+  REVIEWS_JSON   the Gitea /pulls/{n}/reviews array   (or an error object; or absent)
+  HEAD_TIME      ISO-8601 timestamp of the PR head commit
+  DIRECTIVE      the override directive, e.g. /approve-override
+"""
+import json
+import os
+import re
+import sys
+from datetime import datetime, timezone
+
+
+def parse_ts(value):
+    """Gitea emits RFC3339. Return an aware datetime, or None if it is unparseable.
+
+    Unparseable is NOT 'treat as fresh': the caller drops such a candidate, because a
+    timestamp we cannot read is a staleness check we cannot perform.
+    """
+    if not value:
+        return None
+    text = str(value).strip()
+    # Python < 3.11 cannot read a trailing 'Z' in fromisoformat.
+    if text.endswith(("Z", "z")):
+        text = text[:-1] + "+00:00"
+    try:
+        parsed = datetime.fromisoformat(text)
+    except ValueError:
+        return None
+    if parsed.tzinfo is None:
+        parsed = parsed.replace(tzinfo=timezone.utc)
+    return parsed
+
+
+def extract_reason(body, directive):
+    """Return the reason from the first line that STARTS with the directive, else None."""
+    pattern = re.compile(
+        r"^[ \t]*" + re.escape(directive) + r"(?=[ \t:]|$)[ \t:]*(.*)$"
+    )
+    for raw in (body or "").splitlines():
+        # A quoted line is somebody repeating an override, not casting one.
+        if raw.lstrip().startswith(">"):
+            continue
+        match = pattern.match(raw)
+        if match:
+            reason = match.group(1).strip()
+            return reason or None
+    return None
+
+
+def load_source(env_name, label):
+    """Return a list of items for one source, or [] with a stderr note if it is unusable."""
+    raw = os.environ.get(env_name)
+    if not raw or not raw.strip():
+        return []
+    try:
+        parsed = json.loads(raw)
+    except (ValueError, TypeError):
+        print(
+            "approval-override: %s is not valid JSON — ignoring that source" % label,
+            file=sys.stderr,
+        )
+        return []
+    if isinstance(parsed, list):
+        return parsed
+    # An error object, typically {"message": "token does not have ... scope"}. Say which,
+    # because "no override found" and "could not read the overrides" look identical from
+    # the outside and the first wasted a real debugging session.
+    detail = ""
+    if isinstance(parsed, dict):
+        detail = str(parsed.get("message") or parsed.get("errors") or "")[:200]
+    print(
+        "approval-override: %s is not a list (Gitea returned %s%s) — ignoring that source"
+        % (label, type(parsed).__name__, ": " + detail if detail else ""),
+        file=sys.stderr,
+    )
+    return []
+
+
+def candidates_from(items, source, directive, head_time):
+    """Yield (when, login, reason, raw_when, source) for every valid override in one source."""
+    found = []
+    for item in items:
+        if not isinstance(item, dict):
+            continue
+        login = ((item.get("user") or {}).get("login") or "").strip()
+        reason = extract_reason(item.get("body"), directive)
+        if not login or not reason:
+            continue
+        # A PENDING review is an unsubmitted private draft — not an act of anything.
+        state = str(item.get("state") or "").upper()
+        if source == "review" and state == "PENDING":
+            print(
+                "approval-override: dropping PENDING (unsubmitted) review by '%s'" % login,
+                file=sys.stderr,
+            )
+            continue
+        raw_when = (
+            item.get("submitted_at") or item.get("created_at") or item.get("updated_at") or ""
+        )
+        when = parse_ts(raw_when)
+        if when is None:
+            print(
+                "approval-override: dropping %s override by '%s' — unreadable timestamp %r"
+                % (source, login, raw_when),
+                file=sys.stderr,
+            )
+            continue
+        # An override must post-date the commit it covers.
+        if head_time is not None and when <= head_time:
+            print(
+                "approval-override: dropping STALE %s override by '%s' (%s) — not newer than "
+                "head commit (%s). Re-post it to override the current commit."
+                % (source, login, when.isoformat(), head_time.isoformat()),
+                file=sys.stderr,
+            )
+            continue
+        found.append((when, login, reason, raw_when, source))
+    return found
+
+
+def main():
+    directive = os.environ.get("DIRECTIVE", "/approve-override").strip()
+    head_time = parse_ts(os.environ.get("HEAD_TIME", ""))
+
+    found = []
+    found += candidates_from(
+        load_source("COMMENTS_JSON", "COMMENTS_JSON"), "comment", directive, head_time
+    )
+    found += candidates_from(
+        load_source("REVIEWS_JSON", "REVIEWS_JSON"), "review", directive, head_time
+    )
+
+    if not found:
+        return 0
+
+    # Latest wins, across both sources: a later override supersedes an earlier one.
+    when, login, reason, raw_when, source = max(found, key=lambda c: c[0])
+    # Keep the reason on one line — this becomes a TSV field.
+    reason = " ".join(reason.split())
+    print("%s\t%s\t%s\t%s" % (login, raw_when, source, reason))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())
@@ -25,10 +25,45 @@
 # true: if this cannot evaluate, an unreviewed change to the gate itself could merge. A skip is a
 # BYPASS, not a gap. So: cannot determine -> BLOCK.
 #
+# THE ADMIN OVERRIDE (added 2026-10-01), and the lock-out it clears.
+#
+# The rule above was UNSATISFIABLE for the one human who runs this repo. Gitea refuses to let a
+# pull request's author approve their own PR, and `jakman85` is the only collaborator — so "an
+# APPROVED review from someone who is not its author" is a condition the sole administrator can
+# never meet. Every sensitive-path PR they opened was permanently unmergeable, INCLUDING the PR
+# that would fix it. Measured 2026-10-01: PR #35, gate green in 1m41s, zero reviews, blocked.
+#
+# So an administrator may waive the review by COMMENTING on the PR:
+#
+#     /approve-override <reason>
+#
+# Authority comes from the LIVE Gitea permission API, never from the comment — the commenter must
+# hold a permission listed in `.override.authorized_permissions` (admin/owner) at the moment the
+# gate runs, so an old comment from a since-demoted account is worth nothing. The directive must
+# start a line, a reason is mandatory, and the comment must POST-DATE the head commit: an override
+# earned at commit A does not cover commit B. Choosing the candidate comment is the job of the
+# pure, offline, unit-tested ci/approval-override.py; AUTHORISING it is the job of this stage,
+# which fails closed.
+#
+# WHAT THIS COSTS, STATED PLAINLY. For an administrator, "a change to the gate cannot be gated by
+# the gate" stops being structural and becomes advisory. That is the unavoidable consequence of
+# there being one human; the alternative is a repository its owner cannot change. What is bought
+# back is a RECORD — the waiver is an explicit act, attributed to an account, carrying a stated
+# reason, in the PR history and in this stage's output. A silent bypass keyed on "is the author an
+# admin?" would have been less code and is exactly the fail-open shape this repo keeps finding.
+#
+# AND THE PART THAT IS NOT COMFORTABLE. Authorisation is by permission ALONE, with no allowlist of
+# human accounts (operator decision, 2026-10-01, taken with this consequence on the table). The
+# AGENT account `gitea-admin` also reports permission `owner` here, so an agent can clear the
+# sensitive-path review on its own PR by posting the directive — and agents author most PRs in
+# this repo, so that is the common case, not an edge case. To narrow it without touching this
+# logic, add an `override_admins` list to infra/gitea/approval-policy.json: the check below
+# already requires membership whenever that key is present.
+#
 # EXIT CODES
-#   0  no sensitive paths touched (gate alone is sufficient), OR sensitive paths touched and a
-#      non-author APPROVED review is present
-#   1  sensitive paths touched without such a review, OR the policy could not be evaluated
+#   0  no sensitive paths touched (gate alone is sufficient), OR sensitive paths touched and
+#      either a non-author APPROVED review or a valid admin override is present
+#   1  sensitive paths touched with neither, OR the policy could not be evaluated
 set -euo pipefail
 CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
@@ -65,6 +100,17 @@ api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
 FILES_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/files?limit=1000" || true)"
 PR_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}" || true)"
 REVIEWS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/reviews" || true)"
+# PR conversation comments live under the ISSUES endpoint. `/pulls/{n}/comments` returns review
+# (diff) comments ONLY, so an override posted in the normal comment box would never appear in it.
+# Verified against this Gitea (1.24.3) rather than assumed.
+#
+# This one is NOT in the must-parse loop below, and that is deliberate. /issues/{n}/comments sits
+# behind Gitea's *issue* token scope, which `SETTINGS_API_TOKEN` does not carry (it is minted
+# `[read:repository, read:admin]`) — so it returns an error OBJECT, not an array. Blocking on that
+# turned "CI cannot read your override" into a message that said "you did not post one", and cost a
+# real debugging session on PR #38. It is handled explicitly, and loudly, further down: an
+# unreadable source can only cause a false BLOCK, never a false pass.
+COMMENTS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/issues/${PR}/comments?limit=1000" || true)"
 for v in FILES_JSON PR_JSON REVIEWS_JSON; do
  jq -e . >/dev/null 2>&1 <<<"${!v}" || {
    err "could not read ${v} from Gitea for PR #${PR} — BLOCKING rather than assuming."
@@ -107,11 +153,122 @@ if [ -n "$APPROVER" ]; then
  exit 0
 fi

+# ─────────────────────────────────────────────────────────────────────────────────────────
+# No non-author approval exists. Before refusing, look for an administrator's explicit
+# override. See the header for why this hatch exists and what it costs.
+# ─────────────────────────────────────────────────────────────────────────────────────────
+OVERRIDE_ENABLED="$(jq -r '.override.enabled // false' "$DECL")"
+DIRECTIVE="$(jq -r '.override.comment_directive // "/approve-override"' "$DECL")"
+HEAD_SHA="$(jq -r '.head.sha // empty' <<<"$PR_JSON")"
+
+if [ "$OVERRIDE_ENABLED" = "true" ]; then
+  # The head commit's timestamp anchors staleness. Read it from GITEA, not from the local
+  # tree: CI checks out a merge commit, whose date is not the head commit's date.
+  HEAD_TIME=""
+  if [ -n "$HEAD_SHA" ]; then
+    HEAD_COMMIT_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/git/commits/${HEAD_SHA}" || true)"
+    HEAD_TIME="$(jq -r '.commit.committer.date // .created // empty' <<<"${HEAD_COMMIT_JSON:-null}" 2>/dev/null || true)"
+  fi
+  if [ -z "$HEAD_TIME" ]; then
+    err "could not read the head commit's timestamp for PR #${PR} — BLOCKING."
+    err "  Without it an override cannot be aged, and an unaged override is one that silently"
+    err "  covers commits nobody waived review on."
+    summary "- approval-policy: **BLOCKED — head commit timestamp unreadable** (cannot age an override)"
+    exit 1
+  fi
+
+  # Is the comments source actually readable? Say so plainly if not — a scope problem and "you
+  # never posted an override" are indistinguishable from the outside, and conflating them is what
+  # made PR #38 look like a code bug.
+  if ! jq -e 'type == "array"' >/dev/null 2>&1 <<<"${COMMENTS_JSON:-}"; then
+    SCOPE_MSG="$(jq -r '.message // empty' <<<"${COMMENTS_JSON:-null}" 2>/dev/null || true)"
+    warn "cannot read PR COMMENTS from Gitea${SCOPE_MSG:+ — ${SCOPE_MSG}}."
+    warn "  /issues/${PR}/comments needs the *issue* token scope; SETTINGS_API_TOKEN is minted"
+    warn "  [read:repository, read:admin] (infra/gitea/set-ci-token.sh). An override posted as an"
+    warn "  ordinary PR COMMENT is therefore invisible to CI — post it as a REVIEW of type"
+    warn "  'Comment' instead, which /pulls/${PR}/reviews exposes under read:repository."
+    summary "- approval-policy: note — PR comments unreadable (token lacks the issue scope); use a **review** to override"
+    COMMENTS_JSON='[]'
+  fi
+
+  # Pure, offline, unit-tested selection: which comment or review, if any, is a live override?
+  OVERRIDE_TSV="$(COMMENTS_JSON="$COMMENTS_JSON" REVIEWS_JSON="$REVIEWS_JSON" \
+    HEAD_TIME="$HEAD_TIME" DIRECTIVE="$DIRECTIVE" \
+    python3 "${CI_DIR}/approval-override.py")" || {
+      err "ci/approval-override.py failed — BLOCKING rather than assuming there is no override."
+      summary "- approval-policy: **BLOCKED — override parser failed**"
+      exit 1; }
+
+  if [ -n "$OVERRIDE_TSV" ]; then
+    OV_USER="$(cut -f1 <<<"$OVERRIDE_TSV")"
+    OV_WHEN="$(cut -f2 <<<"$OVERRIDE_TSV")"
+    OV_SRC="$(cut -f3 <<<"$OVERRIDE_TSV")"
+    OV_WHY="$(cut -f4- <<<"$OVERRIDE_TSV")"
+
+    # AUTHORITY IS MEASURED LIVE, never inferred from the comment or from a name in git.
+    PERM_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/collaborators/${OV_USER}/permission" || true)"
+    OV_PERM="$(jq -r '.permission // empty' <<<"${PERM_JSON:-null}" 2>/dev/null || true)"
+    if [ -z "$OV_PERM" ]; then
+      err "'${OV_USER}' posted ${DIRECTIVE}, but their permission on ${GITEA_REPO} could not be"
+      err "  read — BLOCKING. An override whose authority is unverified is not an override."
+      summary "- approval-policy: **BLOCKED — cannot verify \`${OV_USER}\`'s permission**"
+      exit 1
+    fi
+
+    # Authorised when the LIVE permission is one of the declared ones and — only if an
+    # `override_admins` allowlist exists at all — the account is on it. No allowlist means
+    # permission alone, which is the operator's standing decision; adding the key narrows
+    # this with no change to any code.
+    AUTHORISED="$(DECL="$DECL" OV_PERM="$OV_PERM" OV_USER="$OV_USER" python3 -c '
+import json, os
+d = json.load(open(os.environ["DECL"])).get("override", {})
+perms = d.get("authorized_permissions") or ["admin", "owner"]
+allow = d.get("override_admins")
+ok_perm = os.environ["OV_PERM"] in perms
+ok_list = True if allow is None else os.environ["OV_USER"] in allow
+print("yes" if (ok_perm and ok_list) else "no")
+')"
+
+    if [ "$AUTHORISED" = "yes" ]; then
+      if [ "$OV_USER" = "$AUTHOR" ]; then
+        warn "ADMIN OVERRIDE — SELF-WAIVED. '${OV_USER}' authored this PR and waived the"
+        warn "  non-author review requirement on their own change. Permitted (permission"
+        warn "  '${OV_PERM}'), deliberate, and shouted about here precisely because it is not free."
+      else
+        warn "ADMIN OVERRIDE by '${OV_USER}' (permission '${OV_PERM}'); author is '${AUTHOR}'."
+      fi
+      warn "  via:    PR ${OV_SRC}"
+      warn "  when:   ${OV_WHEN}  (head commit ${HEAD_SHA:0:8}, ${HEAD_TIME})"
+      warn "  reason: ${OV_WHY}"
+      warn "  review waived on: $(printf '%s ' $SENSITIVE)"
+      ok "sensitive paths cleared by administrator override — this stage does not block."
+      SELFNOTE=""; [ "$OV_USER" = "$AUTHOR" ] && SELFNOTE=", **self-waived as author**"
+      summary "- approval-policy: **OVERRIDDEN by \`${OV_USER}\`** (permission \`${OV_PERM}\`, via PR ${OV_SRC}${SELFNOTE})"
+      summary "  - reason: ${OV_WHY}"
+      summary "  - review waived on: $(printf '\`%s\` ' $SENSITIVE)"
+      exit 0
+    fi
+
+    err "'${OV_USER}' posted ${DIRECTIVE} but is NOT authorised to override: live permission is"
+    err "  '${OV_PERM}'. The authorised set is declared in infra/gitea/approval-policy.json."
+    summary "- approval-policy: **BLOCKED — \`${OV_USER}\` may not override** (permission \`${OV_PERM}\`)"
+    exit 1
+  fi
+fi
+
 err "This PR changes the pipeline or the security posture it rests on, and has NO approving"
 err "  review from anyone other than its author ('${AUTHOR}')."
 err "  A change to the gate cannot be gated by the gate — that is the one thing CI cannot"
 err "  honestly approve for itself. Get a human review, or move the change out of:"
 printf '    %s\n' $SENSITIVE >&2
 err "  The list lives in infra/gitea/approval-policy.json, with a reason per entry."
-summary "- approval-policy: **BLOCKED — sensitive paths need a non-author approval**"
+if [ "$OVERRIDE_ENABLED" = "true" ]; then
+  err "  Or, if you hold admin/owner on this repository, post this on the PR:"
+  err "      ${DIRECTIVE} <reason>"
+  err "  as a REVIEW of type 'Comment' (not the plain comment box — CI's token cannot read"
+  err "  /issues/${PR}/comments without the issue scope; see infra/gitea/set-ci-token.sh)."
+  err "  The reason is mandatory, the directive must START a line, and it must be NEWER than"
+  err "  head commit ${HEAD_SHA:0:8} — push again and you must override again."
+fi
+summary "- approval-policy: **BLOCKED — needs a non-author approval, or an admin \`${DIRECTIVE}\` comment**"
 exit 1
@@ -0,0 +1,232 @@
+#!/usr/bin/env bash
+# ci/tests/approval-override.test.sh — proves ci/approval-override.py selects the override it
+# claims to select, and REJECTS the ones it claims to reject.
+#
+# This parser decides whether a human review requirement gets waived. Every rule in it exists
+# because the obvious implementation ("does the body contain the directive?") is wrong in a way
+# that is invisible until abused: a quoted directive would re-arm an override, a bare directive
+# would waive review with no stated reason, and an override posted against commit A would go on
+# covering commit B forever. A guard is worth nothing until somebody has watched it refuse, which
+# is the same reasoning as ci/tests/decisions-log.test.sh one layer up.
+#
+# The parser is pure and offline BY DESIGN — authorisation (the live permission lookup) lives in
+# ci/approval-policy.sh, which cannot be tested without Gitea. That split is what makes this file
+# possible, and it is also why a green run here does NOT prove the stage authorises correctly.
+set -uo pipefail
+CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+SUT="${CI_DIR}/approval-override.py"
+
+PASS=0 FAIL=0
+ok()  { printf '  ok   %s\n' "$*"; PASS=$((PASS+1)); }
+bad() { printf '  FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
+
+HEAD_TIME="2026-10-01T12:00:00Z"
+AFTER="2026-10-01T12:30:00Z"
+BEFORE="2026-10-01T11:30:00Z"
+
+# run <comments-json> -> stdout of the parser (stderr is diagnostic only)
+run() {
+  COMMENTS_JSON="$1" REVIEWS_JSON="" HEAD_TIME="${2:-$HEAD_TIME}" DIRECTIVE="${3:-/approve-override}" \
+    python3 "$SUT" 2>/dev/null
+}
+
+# run_rev <reviews-json> — the same, but through the REVIEWS source.
+run_rev() {
+  COMMENTS_JSON="" REVIEWS_JSON="$1" HEAD_TIME="${2:-$HEAD_TIME}" DIRECTIVE="${3:-/approve-override}" \
+    python3 "$SUT" 2>/dev/null
+}
+
+# comment <login> <created_at> <body>
+comment() {
+  python3 -c '
+import json, sys
+print(json.dumps([{"user": {"login": sys.argv[1]}, "created_at": sys.argv[2], "body": sys.argv[3]}]))
+' "$1" "$2" "$3"
+}
+
+expect_none() {
+  local label="$1" out
+  out="$(run "$2")"
+  if [ -z "$out" ]; then ok "$label"; else bad "$label (got: ${out})"; fi
+}
+
+expect_user() {
+  local label="$1" json="$2" want="$3" out got
+  out="$(run "$json")"
+  got="$(cut -f1 <<<"$out")"
+  if [ "$got" = "$want" ]; then ok "$label"; else bad "$label (wanted ${want}, got '${got}')"; fi
+}
+
+expect_reason() {
+  local label="$1" json="$2" want="$3" out got
+  out="$(run "$json")"
+  got="$(cut -f4- <<<"$out")"
+  if [ "$got" = "$want" ]; then ok "$label"; else bad "$label (wanted '${want}', got '${got}')"; fi
+}
+
+# check <label> <actual> <wanted>
+check() {
+  if [ "$2" = "$3" ]; then ok "$1"; else bad "$1 (wanted '${3}', got '${2}')"; fi
+}
+
+# review <login> <submitted_at> <state> <body>
+review() {
+  python3 -c '
+import json, sys
+print(json.dumps([{"user": {"login": sys.argv[1]}, "submitted_at": sys.argv[2],
+                   "state": sys.argv[3], "body": sys.argv[4]}]))
+' "$1" "$2" "$3" "$4"
+}
+
+echo "== ci/approval-override.py"
+
+# --- the happy path -------------------------------------------------------------------
+expect_user "a directive with a reason, posted after the head commit, is selected" \
+  "$(comment jakman85 "$AFTER" '/approve-override sole admin, gate is green')" jakman85
+
+expect_reason "the reason is returned verbatim" \
+  "$(comment jakman85 "$AFTER" '/approve-override sole admin, gate is green')" \
+  'sole admin, gate is green'
+
+# --- absence is a normal answer, not an error -----------------------------------------
+expect_none "no comments at all" '[]'
+expect_none "an ordinary comment" "$(comment jakman85 "$AFTER" 'looks good to me')"
+
+# --- a reason is MANDATORY ------------------------------------------------------------
+expect_none "a bare directive (no reason) is refused" \
+  "$(comment jakman85 "$AFTER" '/approve-override')"
+expect_none "a directive followed by only whitespace is refused" \
+  "$(comment jakman85 "$AFTER" '/approve-override    ')"
+
+# --- the directive must START a line --------------------------------------------------
+expect_none "the directive mid-sentence does not count" \
+  "$(comment jakman85 "$AFTER" 'you could just /approve-override this one')"
+expect_none "a QUOTED directive does not re-arm an override" \
+  "$(comment jakman85 "$AFTER" '> /approve-override quoting someone else')"
+expect_user "leading whitespace before the directive is fine" \
+  "$(comment jakman85 "$AFTER" '   /approve-override indented but mine')" jakman85
+expect_user "the directive on a later line of a longer comment counts" \
+  "$(comment jakman85 "$AFTER" 'some preamble
+/approve-override the real reason here')" jakman85
+
+# --- staleness: an override must post-date the commit it waives review on -------------
+expect_none "an override OLDER than the head commit is dropped as stale" \
+  "$(comment jakman85 "$BEFORE" '/approve-override stale, predates the push')"
+expect_none "an override exactly AT the head commit time is dropped (strictly newer)" \
+  "$(comment jakman85 "$HEAD_TIME" '/approve-override same instant')"
+
+# --- malformed input is never a free pass --------------------------------------------
+expect_none "an unreadable timestamp drops the candidate" \
+  "$(comment jakman85 'not-a-date' '/approve-override bad clock')"
+expect_none "a comment with no author is dropped" \
+  '[{"user": {}, "created_at": "2026-10-01T12:30:00Z", "body": "/approve-override no login"}]'
+expect_none "invalid JSON yields no override (and never a traceback into stdout)" \
+  'this is not json'
+expect_none "a JSON object instead of a list yields no override" \
+  '{"user": "nope"}'
+
+# --- latest wins ----------------------------------------------------------------------
+TWO='[
+  {"user": {"login": "early"},  "created_at": "2026-10-01T12:10:00Z", "body": "/approve-override first"},
+  {"user": {"login": "latest"}, "created_at": "2026-10-01T12:40:00Z", "body": "/approve-override second"}
+]'
+expect_user "the LATEST override supersedes an earlier one" "$TWO" latest
+expect_reason "and brings its own reason" "$TWO" second
+
+# --- offsets, not just Z --------------------------------------------------------------
+# Gitea emits RFC3339 WITH AN OFFSET (measured: 2026-10-01T23:09:08-05:00), not a trailing Z.
+# A parser that only handles 'Z' silently drops every real override, so this case is load-bearing.
+expect_user "an RFC3339 offset timestamp (-05:00), as Gitea actually emits, parses" \
+  '[{"user": {"login": "jakman85"}, "created_at": "2026-10-01T08:30:00-05:00", "body": "/approve-override offset form"}]' \
+  jakman85
+expect_none "and an offset timestamp that is genuinely earlier is still stale" \
+  '[{"user": {"login": "jakman85"}, "created_at": "2026-10-01T06:00:00-05:00", "body": "/approve-override offset, stale"}]'
+
+# --- the directive is configurable ----------------------------------------------------
+out="$(run "$(comment jakman85 "$AFTER" '/admin-waive because policy says so')" "$HEAD_TIME" '/admin-waive')"
+if [ "$(cut -f1 <<<"$out")" = "jakman85" ]; then
+  ok "a custom directive from the declaration is honoured"
+else
+  bad "a custom directive from the declaration is honoured (got '${out}')"
+fi
+expect_none "the default directive does not match a custom-directive comment" \
+  "$(comment jakman85 "$AFTER" '/admin-waive because policy says so')"
+
+# --- no staleness anchor: do not invent one -------------------------------------------
+out="$(COMMENTS_JSON="$(comment jakman85 "$BEFORE" '/approve-override no anchor')" \
+       HEAD_TIME="" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+if [ "$(cut -f1 <<<"$out")" = "jakman85" ]; then
+  ok "with no HEAD_TIME the staleness test is skipped, not faked (caller blocks instead)"
+else
+  bad "with no HEAD_TIME the candidate should still be returned (got '${out}')"
+fi
+
+# --- the REVIEWS source ---------------------------------------------------------------
+# Load-bearing, not redundant. Measured on PR #38: SETTINGS_API_TOKEN is minted
+# [read:repository, read:admin], /issues/{n}/comments needs the ISSUE scope, so a well-formed
+# override in the comment box came back as an error object and was invisible. /pulls/{n}/reviews
+# IS readable with that token, so a review body is the route that works today.
+check "a review body carries an override" \
+  "$(cut -f1 <<<"$(run_rev "$(review jakman85 "$AFTER" COMMENT '/approve-override via a review')")")" \
+  jakman85
+check "the source is reported as 'review'" \
+  "$(cut -f3 <<<"$(run_rev "$(review jakman85 "$AFTER" COMMENT '/approve-override via a review')")")" \
+  review
+check "the source is reported as 'comment' for a comment" \
+  "$(cut -f3 <<<"$(run "$(comment jakman85 "$AFTER" '/approve-override via a comment')")")" \
+  comment
+
+out="$(run_rev "$(review jakman85 "$BEFORE" COMMENT '/approve-override stale review')")"
+check "a stale review override is dropped too" "${out:-<none>}" "<none>"
+
+out="$(run_rev "$(review jakman85 "$AFTER" PENDING '/approve-override unsubmitted draft')")"
+check "a PENDING (unsubmitted) review is NOT an act of anything" "${out:-<none>}" "<none>"
+
+check "an APPROVED review carrying the directive still counts as an override" \
+  "$(cut -f1 <<<"$(run_rev "$(review jakman85 "$AFTER" APPROVED '/approve-override belt and braces')")")" \
+  jakman85
+
+out="$(run_rev "$(review jakman85 "$AFTER" COMMENT 'ordinary review prose, no directive')")"
+check "an ordinary review body is not an override" "${out:-<none>}" "<none>"
+
+# --- an unreadable source degrades to 'no candidates', never to a pass ----------------
+# The real-world shape: Gitea answers with {"message": "token does not have ... scope"}.
+ERR_OBJ='{"message":"token does not have at least one of required scope(s): [read:issue]"}'
+out="$(COMMENTS_JSON="$ERR_OBJ" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
+       DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "an error OBJECT from the comments API yields no override" "${out:-<none>}" "<none>"
+
+out="$(COMMENTS_JSON="$ERR_OBJ" \
+       REVIEWS_JSON="$(review jakman85 "$AFTER" COMMENT '/approve-override reviews still work')" \
+       HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "an unreadable COMMENTS source does not suppress a valid REVIEW override" \
+  "$(cut -f1 <<<"$out")" jakman85
+
+# and it must SAY so, or a scope problem reads as 'you never posted one' (it did, on PR #38)
+errout="$(COMMENTS_JSON="$ERR_OBJ" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
+          DIRECTIVE=/approve-override python3 "$SUT" 2>&1 >/dev/null)"
+case "$errout" in
+  *"not a list"*read:issue*) ok "the diagnostic names the scope Gitea complained about" ;;
+  *) bad "the diagnostic should quote Gitea's message (got: ${errout})" ;;
+esac
+
+# --- latest wins ACROSS the two sources ----------------------------------------------
+out="$(COMMENTS_JSON="$(comment early "2026-10-01T12:10:00Z" '/approve-override from a comment')" \
+       REVIEWS_JSON="$(review later "2026-10-01T12:50:00Z" COMMENT '/approve-override from a review')" \
+       HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "the latest override wins across both sources (review newer)" "$(cut -f1 <<<"$out")" later
+check "and its source is labelled" "$(cut -f3 <<<"$out")" review
+
+out="$(COMMENTS_JSON="$(comment later "2026-10-01T12:50:00Z" '/approve-override from a comment')" \
+       REVIEWS_JSON="$(review early "2026-10-01T12:10:00Z" COMMENT '/approve-override from a review')" \
+       HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "the latest override wins across both sources (comment newer)" "$(cut -f1 <<<"$out")" later
+check "and that source is labelled too" "$(cut -f3 <<<"$out")" comment
+
+# --- both sources absent entirely -----------------------------------------------------
+out="$(COMMENTS_JSON="" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
+       DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "no sources at all yields no override, and no crash" "${out:-<none>}" "<none>"
+
+printf '\n  %d passed, %d failed\n' "$PASS" "$FAIL"
+[ "$FAIL" -eq 0 ] || exit 1
@@ -42,7 +42,16 @@ cp "${REAL_CI_DIR}/gate.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env
 # 15 of 47 assertions failed on `main` for that reason alone, including every one that expects a
 # PASS/UNDETERMINED/NOT-APPLICABLE headline. The suite was measuring the missing stub, not the
 # decision logic. `decisions-log` is added here in the same change that adds it to gate.sh.
-for stage in spec-immutability decisions-log lint l0 render-golden l1 gitea-settings conformance; do
+#
+# AND IT HAPPENED AGAIN, with `approval-policy` (fixed 2026-10-01). Same shape, same cost: the
+# stage was added to gate.sh by PR #17, never added here, so 15 of 50 assertions failed on `main`
+# — measured on 0c7e8dd before touching anything, to be sure it was not the change under test.
+# The note above was written about the previous instance and did not prevent the next one, which
+# says the comment is not the fix. THE FIX IS TO DERIVE THE LIST: it now comes from gate.sh's own
+# `run_stage` calls, so a stage added there can no longer be silently missing here.
+mapfile -t STAGES < <(sed -n 's/^run_stage[[:space:]]\+\([a-z0-9-]\+\).*/\1/p' "${REAL_CI_DIR}/gate.sh")
+[ "${#STAGES[@]}" -ge 8 ] || { echo "could not parse stage names out of gate.sh (got ${#STAGES[@]})" >&2; exit 1; }
+for stage in "${STAGES[@]}"; do
  cat > "${CI}/${stage}.sh" <<'STUB'
 #!/usr/bin/env bash
 n="$(basename "$0" .sh)"; v="STAGE_RC_${n//-/_}"
@@ -100,7 +100,8 @@ data:
  # platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
  # tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
  # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
-  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers),
+  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers;
+  # plex-remote: PLEX_L4_VIP),
  # velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
  # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
  # UNSUBSTITUTED path would ship the literal string into the cluster.
@@ -175,6 +176,19 @@ data:
  # dev  -> 192.168.1.56  (base default; next free after crafty .55 in dev's .53-.60 pool)
  # prod -> 192.168.2.23  (next free after crafty .22 in prod's .20-.30 pool; see clusters/prod)
  PIHOLE_DNS_VIP: "192.168.1.56"             # PiHole: authoritative LAN DNS for ${BASE_DOMAIN}
+  # Plex's remote-access listener: raw tcp/32400 for REMOTE Plex clients (apps/wave-0/plex-remote.yaml).
+  # A FOURTH pinned L4 VIP, for the usual reason — the `shared` Gateway is an L7 HTTPS terminator and
+  # cannot carry Plex's own TLS. Plex clients do not resolve `plex.${BASE_DOMAIN}`; they fetch this
+  # server's address from plex.tv, so the Gateway's L7 path (which still serves the web UI, and was
+  # measured working) cannot put the server on a client's connection list at all.
+  #
+  # PINNED, AND THE PIN IS LOAD-BEARING OFF-REPO: the SOHO router forwards WAN tcp/32400 to this
+  # address by hand. A VIP taken from the pool head would silently move on a reschedule and break
+  # remote access with nothing in-cluster looking wrong. See runbooks/plex-remote-access.md.
+  #
+  # dev  -> 192.168.1.57  (base default; next free after pihole-dns .56 in dev's .53-.60 pool)
+  # prod -> 192.168.2.24  (next free after pihole-dns .23 in prod's .20-.30 pool; see clusters/prod)
+  PLEX_L4_VIP: "192.168.1.57"                # Plex: tcp/32400 remote access (NOT the web UI, which is L7)

  # The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
  # peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable
@@ -476,6 +476,26 @@ patches:
      - op: replace
        path: /data/PIHOLE_DNS_VIP
        value: "192.168.2.23"
+      # Plex's remote-access listener VIP — raw tcp/32400 for REMOTE Plex clients
+      # (apps/wave-0/plex-remote.yaml). 192.168.2.24 is the NEXT FREE address in the prod pool
+      # 192.168.2.20-.30, verified live 2026-10-03: the CiliumLoadBalancerIPPool `default-pool`
+      # reported IPsTotal 11 / IPsUsed 4 / IPsAvailable 7, and the only four allocations are .20
+      # (cilium-gateway-shared), .21 (unifi-l4-tcp/-udp), .22 (crafty-l4-tcp/-udp) and .23
+      # (pihole-dns-tcp/-udp). It is inside the pool block the operator already asserted sits outside
+      # the SOHO router's DHCP scope, and it is not a node address (.10/.11/.12) or the
+      # control-plane VIP (.9).
+      #
+      # UNLIKE .23, THIS ONE IS ALSO A WAN PORT-FORWARD TARGET. The router forwards WAN tcp/32400
+      # here, by hand, off-repo — so changing this value is a two-place edit and the router is the
+      # half that CI cannot check. runbooks/plex-remote-access.md is the procedure.
+      #
+      # NOT ARP-PROBED. .23 was verified free with `arping` from a LAN host before it was claimed;
+      # for .24 only the in-cluster pool accounting above was checked, so an off-cluster squatter on
+      # this address would not have been caught. Probe before trusting it if remote access comes up
+      # with a conflict.
+      - op: replace
+        path: /data/PLEX_L4_VIP
+        value: "192.168.2.24"
      # The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns —
      # NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path.
      # It scopes the ingress allow for PiHole's DNS listener to LAN clients only.
@@ -0,0 +1,256 @@
+---
+date: 2026-10-01
+branch: fix/admin-approval-override
+title: Administrators can merge their own green PR — required_approvals 1 → 0, plus an attributed /approve-override for sensitive paths
+---
+
+## What changed
+
+**The problem, measured first.** `required_approvals: 1` on `main` was a lock-out, not a
+protection. Gitea refuses to let a pull request's author approve their own PR, and `jakman85` is
+the **only** collaborator (permission `owner`). So no account in existence could approve a PR the
+operator opened. Measured on the live API, 2026-10-01:
+
+| | |
+|---|---|
+| PR #35 (`bugfix/arr-to-plex`, author `jakman85`) | `gate / gate (pull_request)` = **success, 1m41s** |
+| its reviews | `[]` |
+| live `required_approvals` | `1` |
+| live `block_admin_merge_override` | `true` |
+
+Green gate, zero reviews, no path to a merge — and the same was true of any PR that would have
+fixed it.
+
+It was also never the design this repo wrote down. `ci/approval-policy.sh`'s own header and
+`ci/gate.sh:194` both already described the model as *"`required_approvals` drops to 0 and the
+GATE decides"*, and `infra/gitea/require-gate.sh`'s closing note named this exact failure and
+prescribed this exact remedy. `decisions/2026-09-30-flux-deploy-notification-inhibited.md` had
+flagged the 1-vs-0 disagreement as unresolved and not that session's to settle. This settles it
+at 0.
+
+**Two layers were blocking, and both are addressed.**
+
+*Layer 1 — Gitea branch protection.* `required_approvals: 1 → 0` in
+`infra/gitea/expected-branch-protection.json`, with the rationale in the file, plus
+`infra/gitea/allow-admin-self-merge.sh` to apply it to the live rule (it reads the target **from
+the declaration**, so the script cannot drift from what the gate checks). `require-gate.sh` was
+changed to 0 as well — left at 1 it would silently re-impose the lock-out on its next run.
+
+The narrow choice matters: the *other* way to unblock an owner is
+`block_admin_merge_override: false`, and that would have re-opened precisely what
+`infra/gitea/block-admin-override.sh` was written to close (PRs #5–#9 merged with zero approvals
+while the settings read as enforced). This change drops the approval **count** only.
+`block_admin_merge_override` and `enable_status_check` both stay `true`, and the script asserts
+both afterwards and fails loudly if the PATCH disturbed either. **A red gate remains unmergeable
+by everyone, administrators included.**
+
+*Layer 2 — `ci/approval-policy.sh`.* Sensitive paths (`ci/**`, `infra/gitea/**`, `CLAUDE.md`,
+kyverno, rbac, sops, Flux wiring) still require a non-author `APPROVED` review, which the sole
+administrator still cannot obtain. An administrator may now waive it by commenting on the PR:
+
+    /approve-override <reason>
+
+- the directive must **start a line** (a quoted or mid-sentence mention does not count, or
+  repeating someone else's override would re-arm it);
+- a **reason is mandatory** — a bare directive records that review was waived but not why;
+- the commenter's permission is **re-checked live** against
+  `/collaborators/{login}/permission` when the gate runs, so an old comment from a demoted
+  account carries nothing;
+- it is **invalidated by a new commit** (the comment must post-date the head commit), mirroring
+  Gitea's own `dismiss_stale_approvals`;
+- the overrider, timestamp and reason are printed in the gate log and the job summary, and a
+  self-waiver by the PR's own author is called out as such.
+
+Selection of the candidate comment is `ci/approval-override.py` — pure, offline, no authority of
+its own — so it could be unit-tested; authorisation stays in the fail-closed stage.
+`ci/tests/approval-override.test.sh` adds **23 assertions**, all passing, each constructing a
+case the parser must refuse. Every new failure mode blocks: unreadable head-commit timestamp,
+unreadable permission, parser error.
+
+**The cost, stated rather than buried.** For an administrator, *"a change to the gate cannot be
+gated by the gate"* is now **advisory, not structural**. That is unavoidable with one human — the
+alternative is a repository its owner cannot change — and what is bought back is an attributed,
+reasoned, commit-scoped record of each waiver instead of a silent bypass.
+
+**The uncomfortable part, accepted with eyes open (operator decision).** Authorisation is by live
+permission **alone**, with no allowlist of human accounts. The **agent account `gitea-admin` also
+reports permission `owner`** on this repo (measured), so an agent *can* clear the sensitive-path
+review on its own PR — and agents author most PRs here, so that is the common case, not an edge
+case. I raised this before implementing; the operator chose permission-only. It is documented in
+three places rather than left to be discovered: `approval-policy.json`
+(`_known_consequence_agent_accounts`), the stage's header, and `CLAUDE.md`, which now tells
+agents plainly **not** to use it on their own work. The narrowing path needs no code change —
+add an `override_admins` list to `approval-policy.json` and the stage already requires membership
+in addition to the permission check.
+
+Also updated: `CLAUDE.md` §1 and the §2 enforcement table (the "approval required" row is now
+explicitly ⬜ with the reason, rather than a stale ✅), and `infra/gitea/README.md` gains §6 and
+marks its historical step 4 as reversed — do not re-apply.
+
+**Second pass, 2026-10-02 — the override did not work the first time, and why.** The operator ran
+the live PATCH (`required_approvals: 0` confirmed live, with `block_admin_merge_override` and
+`enable_status_check` still `true`), merged PR #35, and posted `/approve-override Need to ensure
+approval override work as expected` on PR #38. The gate re-ran (run 203) and **still blocked.**
+
+The comment was not at fault: posted by `jakman85`, directive at the start of the line, reason
+present, and 29 minutes **newer** than head commit `616a497`, so it cleared the staleness rule.
+What failed was that **CI could not read it.** Run 203's log:
+
+    approval-override: COMMENTS_JSON is not a list
+
+`SETTINGS_API_TOKEN` is minted `["read:repository", "read:admin"]` by
+`infra/gitea/set-ci-token.sh`, and Gitea puts `/issues/{n}/comments` behind the **issue** scope. So
+Gitea returned an error *object*, `jq -e .` accepted it as valid JSON, and the stage fell through
+to "no approving review" — **a permission failure wearing the costume of a missing override.**
+`/pulls/{n}/files` and `/pulls/{n}/reviews` are under `read:repository`, which is exactly why every
+other part of the same stage worked and this one did not.
+
+Three fixes, and the first is the one that mattered most:
+
+1. **The diagnosis is no longer misleading.** `COMMENTS_JSON` is out of the must-parse-or-block
+   loop and is explicitly checked for being an *array*; when it is not, the stage prints Gitea's
+   own `.message`, names the missing scope, and says where to post the override instead. An
+   unreadable source can only cause a false BLOCK, never a false pass, so degrading it to a loud
+   warning is safe — and blocking on it was never the honest behaviour.
+2. **The override is accepted from a PR REVIEW body as well as a comment**, so the hatch is not
+   hostage to one API scope. A review of type "Comment" works on the token CI has *today*. The
+   gate names which source it used. `PENDING` (unsubmitted, author-only draft) reviews are dropped.
+3. **`set-ci-token.sh` now requests `read:issue`**, which makes the ordinary comment box work —
+   the thing anybody reaches for first. It takes effect only on a **re-run**: an existing token's
+   scopes cannot be widened in place.
+
+`ci/tests/approval-override.test.sh` grows from 23 to **38 assertions**, covering the reviews
+source, `PENDING` reviews, an error *object* from the comments API, "an unreadable comments source
+must not suppress a valid review override", latest-wins across both sources, and that the
+diagnostic actually quotes Gitea's message.
+
+**One adjacent fix, deliberately in scope: `ci/tests/gate-decision.test.sh` was already broken.**
+It hand-maintains a list of stub stage scripts that "must name EVERY stage `gate.sh` runs", and it
+did not name `approval-policy` — added to `gate.sh` by PR #17 and never added here. `run_stage`
+therefore got exit 127 from a missing file and recorded a FAILED stage in every case, so **15 of
+50 assertions failed**, including every one expecting a PASS/UNDETERMINED headline. Measured on
+pristine `origin/main` (`0c7e8dd`) before touching it, so the attribution is not a guess.
+
+I fixed it because `make -C ci test` is the *only* runner for the suite this change adds — leaving
+it red would have made the new tests unverifiable in practice. The file's own comment already
+narrated this exact failure for `gitea-settings` and did not prevent the repeat, which says the
+comment was not the fix. So the stub list is now **derived from `gate.sh`'s own `run_stage` lines**
+and asserts it parsed at least 8 of them: a stage added to the gate can no longer be silently
+missing from its test. `make -C ci test` goes from **FAILED to all suites passed**.
+
+## What I verified
+
+- **The lock-out is real, and PR #35 is the proof.** Queried live: head sha
+  `8a094c46`, statuses `gate / gate (pull_request) = success :: Successful in 1m41s`,
+  `/pulls/35/reviews` → `[]`, `/branch_protections/main` → `required_approvals: 1`. I read the
+  **rule object**, not `/branches/main`, per §2b.
+- **`jakman85` is the only collaborator**: `/collaborators` → one entry. Permission `owner`.
+- **`gitea-admin` also reports `owner`** — this is why the agent-account consequence above is a
+  statement of fact and not a hypothetical.
+- **PR #35 touches no sensitive paths** (`apps/wave-0/plex.yaml`, `apps/wave-1/{radarr,sonarr}.yaml`,
+  `catalog/{radarr,sonarr}.yaml`, its decisions entry) — so `ci/approval-policy.sh` was never its
+  blocker. Layer 1 alone unblocks it.
+- **API shapes verified before depending on them**, rather than assumed: `/git/commits/{sha}`
+  returns `.commit.committer.date` as RFC3339 **with an offset** (`2026-10-01T23:09:08-05:00`),
+  not a trailing `Z` — a parser handling only `Z` would have silently dropped every real
+  override, so there is a test for the offset form. PR conversation comments are under
+  `/issues/{n}/comments`; `/pulls/{n}/comments` returns diff comments only and would never have
+  seen an override.
+- **`ci/tests/approval-override.test.sh`: 23 passed, 0 failed.** Covers the happy path, bare
+  directive, mid-sentence and `>`-quoted directives, stale and equal-timestamp overrides,
+  unparseable timestamps, missing login, invalid JSON, non-list JSON, latest-wins, offset
+  timestamps, and a configurable directive.
+- `bash -n` clean on `ci/approval-policy.sh`, `infra/gitea/allow-admin-self-merge.sh`,
+  `infra/gitea/require-gate.sh`; `jq -e .` clean on both JSON declarations.
+- **`gate-decision.test.sh`'s 15 failures are pre-existing, measured not assumed.** Extracted
+  `origin/main`'s `ci/` at `0c7e8dd` into a throwaway directory and ran the suite from there:
+  **FAIL, 15 failed / 35 passed**. After deriving the stub list from `gate.sh`: **PASS, 50
+  assertions**.
+- **`make -C ci test` in the pinned toolbox: `ci/tests: all suites passed`, rc 0** — approval-override
+  **38**, gate-decision 50, touched-requirements-base 31, test-attribution 15, touched-requirements 26,
+  decisions-log pass. It was rc 2 (`ci/tests: FAILED`) before the stub-list fix.
+- **The live flip worked, and the drift detector proved it.** Run 202 (before): `gitea-settings:
+  FAIL`, `required_approvals: DECLARED 0 but LIVE is 1`. Run 203 (after the operator ran the
+  PATCH): `gitea-settings: pass`, with `block_admin_merge_override = true` among the keys it
+  checked. So the narrow change held — the approval count moved and the gate's teeth did not.
+- **The first override attempt failed for a measured reason, not a guessed one.** Run 203 log:
+  `approval-override: COMMENTS_JSON is not a list`. Token scopes read from `set-ci-token.sh`:
+  `[read:repository, read:admin]`. The same run successfully read `/pulls/38` and
+  `/pulls/38/files` (author and sensitive paths both resolved) and got past the reviews
+  validation, which is how I know `/pulls/{n}/reviews` IS readable with that token — from
+  behaviour, not from documentation.
+- **The comment itself was valid.** `created_at` 2026-10-02T05:31:24Z vs head commit `616a497`
+  committer date 2026-10-02T00:02:20-05:00 → the comment is 29m04s newer, so it passed the
+  staleness rule. Ruling the comment out is what left the token scope as the only candidate.
+- **`?limit=1000` is not the culprit.** Queried `/issues/38/comments` with `limit=1000`, `limit=50`
+  and no limit using my own credentials: `type=array` in all three. So the difference is the
+  credential, not the query.
+- **The root cause is confirmed by Gitea, in its own words.** Run 206 (the fix in place):
+  `[warn] cannot read PR COMMENTS from Gitea — token does not have at least one of required
+  scope(s), required=[read:issue], token scope=read:admin,read:repository.` followed by the
+  instruction to post the override as a review. That is the diagnosis the first attempt should
+  have produced, and `gitea-settings: pass` in the same run. The stage still ends `FAIL`, correctly
+  — there is no valid override on the PR yet.
+- **Toolbox merge gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`), run on the
+  rebased branch: **rc 3, `MERGE GATE: UNDETERMINED`**. Green stages: spec-immutability,
+  decisions-log, lint, l0 (23 offline cases, 0 failures), render-golden including the new
+  `catalog/render.py --selftest`, l1 (3 cases), approval-policy (N/A offline). Skipped:
+  `gitea-settings` (no `SETTINGS_API_TOKEN` on this host) and `conformance` (no cluster).
+  **This is NOT a green run and I am not reporting it as one** — rc 3 is UNDETERMINED per the
+  CLAUDE.md §3 contract.
+
+## What I did NOT verify
+
+- **The override has still never been ACCEPTED by a real run.** This is the second attempt at
+  this bullet, which is itself the point: the first pass called the hatch unverified, and the
+  first real test found a defect that no amount of local testing would have (an API scope CI has
+  and I did not). What is now measured is that the parse → *reject* path works end to end and
+  diagnoses itself. The **accepting** branch — directive found → live permission lookup → exit 0 —
+  has never executed. Posting the override as a **review** on PR #38 is its first real test, and
+  I would not be surprised by a third round.
+- **`read:issue` is the right scope name — Gitea said so itself, so this is no longer a guess.**
+  Run 206 printed Gitea's own message: `token does not have at least one of required scope(s),
+  required=[read:issue], token scope=read:admin,read:repository`. That both confirms the root
+  cause and names the string, which matters because this repo has been burned by a token *name*
+  that could not exist (`GITEA_SETTINGS_TOKEN`). What is **still** unverified is that the token
+  *mint* accepts it — `POST /users/{u}/tokens` with that scope has not been run. The review route
+  does not depend on it.
+- **I did not verify that Gitea lets a PR's own author submit a `COMMENT`-type review.** Gitea
+  blocks author self-*approval*; "Comment" is expected to be allowed, and that expectation is now
+  load-bearing for the recommended route. It is one click to find out, and I deliberately did not
+  test it by posting an override myself: doing so as `gitea-admin` is precisely the agent
+  self-override that `CLAUDE.md` now forbids.
+- **The whole re-run mechanism is manual.** `gate.yml` triggers on `pull_request` and `push: main`
+  only — there is no `issue_comment` trigger, so posting an override does **not** re-run the gate.
+  Re-run the job from the Actions UI, or close and reopen the PR (which fires `pull_request` and
+  leaves the head SHA alone). **Do not push an empty commit to re-trigger it:** that moves the head
+  commit and makes the override stale by the rule above, so it would have to be posted again. A
+  comment trigger was considered and not built — it would need to check out PR head code while the
+  job holds `SETTINGS_API_TOKEN`, which is the classic pwn-request shape and not worth it here.
+- **I have not seen the gate still REFUSE a merge** now that approvals are 0. The claim that
+  `block_admin_merge_override: true` keeps a red gate unmergeable rests on Gitea's documented
+  behaviour and on the 2026-09-29 session, **not** on a provoked refusal in this one. That probe
+  is written out at the foot of `allow-admin-self-merge.sh` and should be done, exactly as the
+  push rejection was provoked rather than assumed.
+- **Clock skew in the staleness check is unhandled.** Staleness compares the comment's
+  server-side `created_at` against the head commit's *committer* date, which comes from the
+  committer's machine. A commit dated in the future would reject legitimate overrides until real
+  time catches up (fail-closed, so loud, but annoying). Gitea's own `dismiss_stale_approvals`
+  compares SHAs instead; a comment cannot. Not hit here — PR #35's commit date is ~21 min behind
+  server time — and not fixed.
+- **`ci/tests/approval-override.test.sh` does not run in the merge gate.** `ci/gate.sh` runs no
+  `ci/tests/*`; they run only via `make -C ci test`. So a regression in the parser would not be
+  caught by CI. PR #33 (merged while this branch was being written, and rebased onto here) fixed
+  the *same class* of gap for `catalog/render.py --selftest` by invoking it from
+  `ci/render-golden.sh` — it did **not** wire `ci/tests/*` in, which I confirmed after rebasing
+  rather than assuming. Wiring them is a one-line stage and I did not widen scope to it.
+- **Authorisation itself is untested code.** `ci/approval-override.py` is unit-tested; the live
+  permission lookup, the `override_admins` allowlist branch, and every new BLOCK path in
+  `ci/approval-policy.sh` need Gitea and have never been exercised. The allowlist branch in
+  particular is implemented but **unused** — `override_admins` is absent by design.
+- **The other open PRs will go red** once the live value changes, until they are rebased onto the
+  merged declaration: as of writing those are **#29** and **#37** (both `gitea-admin`), plus
+  **#35**. I did not rebase them. Merging them first avoids it entirely; the script's header gives
+  the order. PRs #33, #34 and #36 merged while this branch was being written — this branch is
+  rebased onto `0c7e8dd`, so the list was re-measured rather than carried forward.
+- No cluster state was read or changed; nothing here touches Flux, Kyverno or any workload.
@@ -0,0 +1,120 @@
+---
+date: 2026-10-01
+branch: bugfix/arr-to-plex
+title: Radarr and Sonarr may now reach Plex on 32400 — the network path only, not the Plex connection itself
+---
+
+## What changed
+
+Radarr and Sonarr had no route to Plex. The media library lives on NFS
+(`/mnt/Storage/plex`, the ZFS dataset `Storage/plex` on `${BULK_NFS_SERVER}`), and NFS
+does not deliver filesystem CRUD notifications, so Plex never learns that an arr has
+written a new file. Without a push from the arrs, the library only reflects new media
+when somebody scans it by hand. That is the bug this branch addresses.
+
+The fix is one `dependencies:` entry per app, added to the catalog — the source of truth:
+
+    - to: plex
+      port: 32400
+      protocol: tcp
+
+in both `catalog/radarr.yaml` and `catalog/sonarr.yaml` (these two *are* the arr suite
+here; `jackett`, `qbittorrent` and `seerr` are different roles and are untouched).
+
+The renderer expands each entry into a **pair** of Cilium policies, which is why a
+three-line catalog change produces 88 lines under `apps/`:
+
+| Side | Policy | File |
+|---|---|---|
+| egress | `radarr-allow-plex-32400` | `apps/wave-1/radarr.yaml` |
+| egress | `sonarr-allow-plex-32400` | `apps/wave-1/sonarr.yaml` |
+| reciprocal ingress | `plex-allow-radarr-32400` | `apps/wave-0/plex.yaml` |
+| reciprocal ingress | `plex-allow-sonarr-32400` | `apps/wave-0/plex.yaml` |
+
+Three commits, and the middle two are worth recording because both were self-inflicted
+gate failures rather than design changes:
+
+- `753c48b` — the catalog edit alone, with `apps/` left unrendered.
+- `9048c2f` — re-rendered `apps/` from the catalog, which `render==golden` requires.
+- `58f02fe` — removed 20 trailing spaces on `catalog/radarr.yaml:103`, left behind as
+  comment-alignment padding when the comment moved down to the `port:` line. Sonarr's
+  equivalent line kept its comment in place, which is why only radarr tripped.
+
+## What I verified
+
+**The lint failure, root-caused.** Gate run 193 on `9048c2f` failed with
+`[fail] lint: FAILED -> blocks merge`, and lint was the only failing stage. yamllint
+emitted 419 findings, but **418 were `line-length`**, which `ci/yamllint.yaml` pins at
+`level: warning` and which therefore cannot change the exit code. The single error-level
+finding in the entire repository was:
+
+    catalog/radarr.yaml
+      103:13    error    trailing spaces  (trailing-spaces)
+
+`trailing-spaces` inherits the default severity (error), so that one line returned rc 1.
+After the fix, `git grep -nIE ' +$' -- '*.yaml' '*.yml'` matches nothing repo-wide.
+
+**Gate run 195 on `58f02fe`** (read from the job log, not from the status tick):
+
+    [ ok ] approval-policy: green     [ ok ] l0: green
+    [ ok ] gitea-settings: green      [ ok ] render-golden: green
+    [ ok ] decisions-log: green       [ ok ] l1: green
+    [ ok ] lint: green
+    [fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
+    [fail]   Stages that did not run: spec-immutability
+
+The Gitea check reports `success` / "Successful in 1m41s", but the gate's own verdict is
+**UNDETERMINED (rc 3)**, not PASS. That tick is `gate.yml` tolerating exit 3 for the one
+stage it documents as unverifiable on this runner (the spec repo is not checked out
+there). Nothing else produced a 3.
+
+**`render==golden` green** on `58f02fe`, so `apps/` is a faithful canonical render of
+`catalog/` — the four generated policies are not hand-written drift.
+
+**DNS egress was already open** and is not a new gap: `apps/wave-1/radarr.yaml` carries
+an existing allow for port 53 UDP+TCP, so resolving the Plex service name does not need
+anything added here.
+
+**Live branch protection on `main`**, measured against
+`/api/v1/repos/Home-Network/home-network-k3s-impl/branches/main` rather than assumed:
+`protected: true`, `user_can_push: false`, `enable_status_check: true`,
+`status_check_contexts: ['gate / gate (pull_request)']`, `required_approvals: 1`.
+
+## What I did NOT verify
+
+**This change does not accomplish what the PR set out to do.** It opens the network path
+and nothing else. Triggering a library update also requires each arr's
+*Settings → Connect → Plex Media Server* entry plus a Plex token, and **none of that
+exists anywhere in `catalog/` or `apps/`** — I grepped both for `plex`, `token` and
+`connect` and found only NFS paths, a uid and a timezone. Radarr's and Sonarr's config
+lives on a 5Gi Longhorn PVC mounted at `/config`, so that connection is UI-only state
+that exists solely on the PVC and never in git, in the same way Crafty's local login
+does. **Until somebody adds it in each app's UI, Plex still will not see library
+updates, and this branch will look like it did nothing.** That follow-up is not tracked
+by any file in this PR.
+
+**L2/L3 conformance never ran.** `CLUSTER_AVAILABLE` is unset on the runner, so the
+eight requirements this diff touches — `R-NET-01`, `R-NET-05`, `R-NET-06`, `R-NET-07`,
+`R-IDP-02`, `R-IDP-06`, `R-GITOPS-02`, `R-PLAT-02` — are **UNVERIFIED**. That gap is
+unusually load-bearing here: the entire substance of this change is four
+CiliumNetworkPolicies, and no test has confirmed that traffic from the `radarr` or
+`sonarr` namespaces actually reaches `plex:32400`. The green gate proves the offline
+render subset only.
+
+**`spec-immutability` did not run** (`R-TEST-06` not verified), for the documented
+reason — the sibling spec repo is absent from the runner.
+
+**Nothing was checked on the cluster.** I ran no `kubectl`, so I did not confirm that
+the `plex` Service exposes 32400 as the arrs would address it, that the pod labels the
+policies select (`app.kubernetes.io/name`, `k8s:io.kubernetes.pod.namespace`) match the
+running pods, or that Cilium accepted the four new objects. Flux has not applied any of
+this: the branch is unmerged, so `.status.lastAppliedRevision` does not contain it.
+
+**I did not verify Plex's own side** — whether its library-update endpoint on 32400 is
+the right target for the arrs' notification, or whether Plex requires
+"Update my library automatically" enabled to act on one.
+
+**I am not the approver and did not approve this PR.** `required_approvals: 1` is live,
+and the only credential on the authoring workstation belongs to the PR's author
+(`jakman85`), whose own review Gitea does not count. Per CLAUDE.md §1 the merge is a
+human action regardless.
@@ -0,0 +1,211 @@
+---
+date: 2026-10-01
+branch: feat/quicksync-via-device-plugin
+title: "`hardware.gpu: intel-quicksync` is now satisfied by `gpu.intel.com/i915`, not a `/dev/dri` hostPath — the hostPath was measured not to work (device-cgroup EPERM), and the swap took ZERO ServiceDefinition edits"
+---
+
+## What changed
+
+The mechanism behind `hardware.gpu: intel-quicksync` was replaced. **No ServiceDefinition field
+moved** — that was the whole design bet when this field was made an enum of capability classes
+instead of a device path, and this is the change that collects on it.
+
+**Why the old mechanism had to go.** The `/dev/dri` hostPath shipped on 2026-09-30 was measured
+**non-functional** in the live Plex pod on k3s-prod-2 on 2026-10-01. The device node was *visible*
+in the container — `crw-rw---- root video2 226,128 renderD128` — and could not be opened:
+
+| attempt | result |
+|---|---|
+| `uid 0`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+| `uid 1000`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+
+`EPERM` is the **device cgroup**, not `EACCES` from file permissions, proven three ways: root on the
+**host** opens the same device `rc=0`; the node is unified `cgroup2fs`; and the container's runc
+`config.json` read `linux.resources.devices: [{allow: False, access: 'rwm'}]` with
+`linux.devices: None` — one deny-all rule, no allow entry for major 226. A hostPath is a *filesystem*
+operation: it binds the device node in and adds **no cgroup allow rule**, because Kubernetes cannot
+infer one from a volume. `uid 0` failing matters: root does not bypass the cgroup device gate, so
+"run it as root" was never going to work either, and `privileged: true` is unavailable (Kyverno
+`restrict-host-namespaces` is Enforce on app namespaces).
+
+**`catalog/render.py`** — retired `DRI_VOLUME_NAME` / `DRI_DEVICE_DIR` for `QUICKSYNC_RESOURCE`
+(`gpu.intel.com/i915`). The `hardware.gpu == intel-quicksync` branch now emits exactly one thing on
+the **app container only** (never a sidecar):
+
+```yaml
+resources:
+  limits:
+    gpu.intel.com/i915: 1
+```
+
+and no volume, no volumeMount, no `securityContext`, no `nodeSelector`. `limits` rather than
+`requests` is mandatory, not stylistic: Kubernetes mirrors `requests` from `limits` automatically and
+a requests-only extended resource is invalid. This is the **only** app-container `resources` render
+path in the file and it is deliberately narrow — no general CPU/memory resources feature was invented
+along the way.
+
+**Self-tests** — the three existing ones updated plus one new:
+- the emitted `resources` dict is asserted **exactly** (`{"limits": {"gpu.intel.com/i915": 1}}`), so
+  a stray CPU/memory addition through this field fails loudly;
+- the retired hostPath volume and `/dev/dri` mount are asserted **absent**;
+- the no-`hardware` negative extended to cover `resources` (absence still renders nothing at all);
+- the `podSecurityContext` pass-through still holds, and the "no synthesised `securityContext`"
+  assertion is kept verbatim;
+- **new:** the i915 limit must never reach the oauth2-proxy sidecar.
+
+**`schemas/renderer-contract.md`** — the mechanism half of `### hardware` and invariant 8 rewritten.
+The intent half is unchanged and still correct. The `EPERM` measurement is recorded as the *reason*
+the design is what it is, so a future reader cannot "simplify" it back without knowing it was tried
+and measured. The `DirectoryOrCreate` rationale is kept, explicitly labelled historical.
+
+**`catalog/plex.yaml`** — comment correction only; the `hardware:` field is byte-identical.
+`git diff` on that file contains **no non-comment lines**. Its old comment restated the mechanism
+("exactly two things — a `/dev/dri` hostPath volume…"), which is now false; it states intent and
+points at the contract, and keeps the measured facts that remain true (device on all three nodes,
+`i915` bound, the cont-init hook / `supplementalGroups` explanation).
+
+**`runbooks/plex-quicksync.md`** — the diagnostic sequence now checks
+`.status.allocatable["gpu.intel.com/i915"]` and the plugin DaemonSet instead of a hostPath; the
+decisive open test is specified **as uid 1000 via `s6-setuidgid plex`**; the cgroup `EPERM` is
+recast from a *risk* to the *reason for the design*; a new top-of-page "STOP HERE IF PLEX IS
+`Pending`" section and a "Plex will not schedule" procedure carry the new failure mode and the escape
+hatch. The manual-toggle section (`HardwareAcceleratedCodecs`, Settings → Transcoder, still not
+settable by any env var) is unchanged.
+
+**`apps/wave-0/plex.yaml`** — re-rendered: six hostPath lines removed, a three-line `resources` block
+added. Nothing else in `apps/` changed.
+
+### The one real regression, stated rather than sold
+
+This trades failure modes, and the new one is **worse in blast radius**:
+
+| | device/resource unavailable ⇒ |
+|---|---|
+| **Before** | `DirectoryOrCreate` mounted an empty dir; Plex ran and **silently software-transcoded**. Invisible, harmless to everything else. |
+| **After** | Plex is **Unschedulable**. `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so this **stalls the app tier**. |
+
+It is **unavoidable**: an extended resource is a hard scheduling constraint, and it is also the only
+thing that produces the cgroup allow rule. There is no graceful-degradation variant. Plex's
+schedulability is now coupled to the `intel-gpu-plugin` DaemonSet — broken, removed, or merely not
+yet reconciled, it takes the app tier with it. That is the price of hardware transcoding working at
+all.
+
+**Escape hatch — one field, no code change:** delete the `hardware:` block from `catalog/plex.yaml`
+and re-render. Plex returns to software transcoding and the tier unblocks. It is in the contract, in
+the runbook's "Plex will not schedule → remedy B" with exact commands, and here, because it is what
+someone will need at 2am.
+
+## What I verified
+
+**The precondition, re-read from the branch rather than assumed:** the plugin is a standalone
+DaemonSet at `platform/intel-gpu-plugin/` with its own Flux Kustomization
+(`clusters/base/intel-gpu-plugin.yaml`, named `intel-gpu-plugin` in `flux-system`), namespace
+`intel-gpu-plugin`, DaemonSet `intel-gpu-plugin`, pinned `intel/intel-gpu-plugin:0.37.0`. Every
+`kubectl` command I wrote into the runbook names those exact objects. The node/DaemonSet readings
+themselves (`i915=1` on all three nodes; 3 desired / 3 ready / 3 available) were **measured by the
+PM, not by me** — see "What I did NOT verify".
+
+**`render.py --selftest`: PASS**, and every new assertion is **mutation-tested — 8/8 mutants caught**
+(a test that cannot fail is not a test):
+
+| mutant | caught by |
+|---|---|
+| `requests` instead of `limits` | exact-resources assertion |
+| emission dropped entirely | exact-resources assertion |
+| `/dev/dri` hostPath + mount re-added | hostPath-absent assertion |
+| emitted unconditionally (no `hardware` key) | no-`hardware` negative |
+| `supplementalGroups` synthesised | no-pod-`securityContext` assertion |
+| resource renamed `gpu.intel.com/xe` | exact-resources assertion |
+| quantity as the string `"1"` | exact-resources assertion |
+| limit leaked onto the oauth2-proxy sidecar | new app-container-only assertion |
+
+**Re-render is clean and scoped.** `python3 catalog/render.py --all catalog -o apps --wave-layout
+--no-image-automation` then `git status --porcelain apps/` ⇒ `M apps/wave-0/plex.yaml` and nothing
+else. `git diff --stat apps/` ⇒ `1 file changed, 3 insertions(+), 6 deletions(-)` — exactly the six
+hostPath lines out, the three-line `resources` block in. **`apps/wave-1/keycloak.yaml` and
+`apps/wave-0/vm-demo.yaml` are untouched** (checked explicitly: they are git-tracked but excluded
+from the golden comparison by `RENDER_GOLDEN_NOT_ONBOARDED` in `ci/render-golden.sh`, so a spurious
+diff there would land ungated). `clusters/` is untouched; `--emit-waves` was not run.
+
+**Invariant 8 holds, mechanically checked rather than eyeballed.** `git diff -U0 catalog/plex.yaml`
+filtered to non-comment changed lines returns **empty**, and `yaml.safe_load` gives
+`hardware == {'gpu': 'intel-quicksync'}`. Zero ServiceDefinition field edits.
+
+**The gate: `rc 3` UNDETERMINED, which is this environment's calibrated best — NOT green, and I am
+not calling it green.** Run from this worktree in the pinned toolbox
+(`DEV_KUBECONFIG=/dev/null`, plus an `ENGINE` shim bind-mounting the main checkout's `.git`
+read-only, per CLAUDE.md §3; `infra/ansible/.cache/prod.kubeconfig` was **not** copied):
+
+| stage | result |
+|---|---|
+| `spec-immutability` | **SKIPPED** — spec repo not present at `/work/../home-network-k3s` (R-TEST-06 unverified) |
+| `decisions-log` | green |
+| `lint` | green |
+| `l0` | green |
+| `render-golden` | green |
+| `l1` | green |
+| `approval-policy` | green (`event='none'`, N/A off a PR event) |
+| `gitea-settings` | **SKIPPED** — `SETTINGS_API_TOKEN` unset |
+| `conformance` | **SKIPPED** — `CLUSTER_AVAILABLE != true`, L2/L3 not run |
+
+Three skips, the three expected ones; **nothing failed**. `diff base: origin/main @ 8214642d87de …
+HEAD @ 78bf68493d7d`, touched requirements resolved (`R-PLAT-02` among them), so the R-TEST-03
+coverage gate actually applied — an earlier run on the uncommitted tree reported the touched set
+"NOT DETERMINABLE (on-base)" and that is *not* the run I am quoting.
+
+## What I did NOT verify
+
+**HARDWARE TRANSCODING IS STILL NOT OBSERVED WORKING, and nothing in this branch could establish
+it.** A green gate proves the **manifest renders**, nothing more.
+
+- **The `open()` test has not been re-run under the new mechanism, and cannot be until this merges
+  and Flux reconciles.** The whole case for this change is that the device plugin's
+  `DeviceSpec{… Permissions: "rw"}` produces the cgroup allow rule the hostPath could not. That is
+  read from the v0.37.0 **source**, not observed on this cluster. If the plugin does not in fact
+  deliver it here, this change buys nothing and costs schedulability.
+- **The Plex-side toggle is still unset.** `HardwareAcceleratedCodecs` was **absent** from
+  `Preferences.xml` as of 2026-09-30 and nothing in this branch touches it; no env var, ConfigMap or
+  `postBuild` substitution can. It is a manual UI click, and until it happens Plex software-transcodes
+  no matter how the device arrives.
+- **I did not measure the cluster at all.** `i915=1` on all three nodes and the DaemonSet's
+  3/3/3 come from the PM's brief, dated 2026-10-01. Per CLAUDE.md §2b this repo is not evidence about
+  the live platform, and neither is a handoff entry: **re-measure before relying on them** (the
+  commands are first-command #1 in the runbook). The DaemonSet was 2m15s old when read — young enough
+  that it had not been observed stable for any length of time.
+- **The new failure mode has not been exercised.** "Plugin unavailable ⇒ Plex Unschedulable ⇒
+  `apps-wave-0` stalls ⇒ `apps-wave-1` blocked" is derived from `wait: true` and `dependsOn` in the
+  Flux wiring, not from a drill. The escape hatch (delete `hardware:`, re-render) is likewise
+  reasoned, not rehearsed — though it is a strict subset of the renderer path the selftest covers.
+- **No L2/L3 conformance.** `CLUSTER_AVAILABLE` is unset, so the gate's runtime layer did not run
+  here and does not run in CI either. `spec-immutability` and `gitea-settings` also skipped.
+- **CI's own tick, when it appears, will not distinguish rc 0 from rc 3**:
+  `.gitea/workflows/gate.yml:138` is `./ci/gate.sh || [ "$?" -eq 3 ]`. Read the stage output, not the
+  colour — and take the newest row per context from `GET /statuses/{sha}`, which appends.
+- **`imagePullPolicy: Always` on `plexinc/pms-docker:latest` is still a live hazard** to the
+  cont-init-hook reasoning this feature leans on. Unchanged by this branch, recorded in the runbook's
+  floating-image caveat.
+
+### Post-merge sequence for the operator (all mutating steps are yours, not an agent's)
+
+1. Merge the PR once the gate reports. Confirm Flux applied it — compare
+   `kubectl --context k3s-prod -n flux-system get kustomization apps-wave-0
+   -o jsonpath='{.status.lastAppliedRevision}'` against the merge commit.
+2. Confirm Plex **scheduled** (this is the new risk):
+   `kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide`.
+   `Pending` ⇒ runbook "Plex will not schedule".
+3. Confirm it consumed the resource:
+   `kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex
+   -o jsonpath='{.items[0].spec.containers[0].resources.limits}'`.
+4. **The decisive test — as uid 1000, via `s6-setuidgid plex`. `rc=0` is the pass condition:**
+   `kubectl --context k3s-prod -n plex exec deploy/plex -- s6-setuidgid plex sh -c
+   'exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED'`.
+   **Passing as `uid 0` is weaker evidence and must not be accepted as proof** — Plex runs as 1000,
+   and uid 0 was one of the two `EPERM`s that killed the hostPath.
+5. **Only then** the Plex UI toggle: Settings → Transcoder → "Use hardware acceleration when
+   available" (under *Show Advanced*) → Save. Then force a transcode and read Dashboard → Activity:
+   `Transcode (hw)` is the proof; plain `Transcode` is software.
+
+Step 4 failing means the plugin is not delivering the cgroup allow rule on this cluster. Do **not**
+reach for `privileged: true` (Kyverno Enforce rejects it, and uid 0 failed anyway) and do **not**
+re-add a `/dev/dri` hostPath (that is the mechanism this replaced, measured non-functional). The
+escape hatch is to delete the `hardware:` block from `catalog/plex.yaml` and re-render.
@@ -0,0 +1,149 @@
+---
+date: 2026-10-02
+branch: fix/plex-quicksync-verification
+title: Hardware transcoding works — and `runbooks/plex-quicksync.md`'s verification section was wrong
+---
+
+## What changed
+
+**Hardware transcoding is confirmed working on prod.** Measured 2026-10-02 from
+`Plex Media Server.log` in the running pod: `vaapi` on `/dev/dri/renderD128`
+(`Intel CoffeeLake-S GT2 [UHD Graphics 630]`), `using hardware decode accelerator vaapi`,
+`zero-copy support present`, `using zero-copy transcoding`, and
+`final decoder: vaapi, final encoder: vaapi`. VAAPI for **both** directions, zero-copy, so frames
+never round-trip through the CPU. The device-plugin mechanism from PR #37 works end to end.
+
+**And `runbooks/plex-quicksync.md`'s verification section was wrong, which is the correction this
+branch exists to publish.** The page said the Plex dashboard showing `Transcode (hw)` was "the *only*
+proof available, and nothing else will ever tell you". The operator had hardware transcoding fully
+working while **Dashboard → Activity showed nothing of the sort** — so the page's authoritative check
+was silent on a working system, and sent someone looking for a fault that did not exist. Three
+inference errors are now corrected on the page, two of them recorded as named false leads so the next
+reader does not re-walk them:
+
+1. **The dashboard is not proof.** Replaced as the primary check by Plex's own log — the grep is in
+   the page, in its **redacting** form, because `Plex Media Server.log` carries API URLs containing
+   `X-Plex-Token` (CLAUDE.md §6; a credential has been leaked into a transcript here before). The
+   page also now says explicitly that **`Plex Transcoder Statistics.log` is the wrong file** — it
+   holds XML `<Variant .../>` records, not ffmpeg output, and the name invites the mistake — and why
+   `grep -a` is required. The dashboard is **kept but demoted** to a secondary, best-effort signal,
+   with its absence stated to prove nothing.
+2. **An fd snapshot is not evidence.** A `/proc/*/fd` sweep inside the pod found no holder of
+   `renderD128` while hardware transcoding was active; transcode sessions are short and the scan
+   landed in a gap. Recorded as a documented false lead, with the reliable process-level check in its
+   place: the transcoder's **argv** (`-hwaccel`, `-hwaccel_device`, `vaapi`, `renderD128`), found via
+   `ps -eo pid,user,comm | grep -i transcoder`.
+3. **A missing preference key does not mean the setting is off.** The page told readers to treat an
+   absent `HardwareAcceleratedCodecs` in `Preferences.xml` as "never enabled". With hardware
+   acceleration demonstrably on and working, that key was **absent from all 27 attributes** while
+   `HardwareDevicePath` **was** written and the log said `hardware transcoding: enabled`. First-command
+   #4 now reads both keys and states plainly that `HardwareDevicePath` present is a positive signal,
+   absence of `HardwareAcceleratedCodecs` means nothing either way, only `="0"` is conclusive, and the
+   log settles it.
+
+Also on the page:
+
+- **GID `992` is no longer presented as a number to expect.** Under the plugin the injected device
+  arrives `root:root`, so the image's `cont-init.d` hook resolves GID **`0`** and adds `plex` to the
+  **root group** — measured `Groups: 0 100 1000`, `id plex` → `groups=1000(plex),0(root),100(users)`.
+  992 was the hostPath-era host `render` GID. The page now says so, marks the root-group membership as
+  an inherent property of the plugin's `DeviceSpec` rather than a defect, and repeats that it is
+  **not** a reason to reintroduce `supplementalGroups` (still a no-op: `s6-setuidgid` rebuilds the
+  group list from `/etc/group`). The stale `Groups: 100 1000` claim in the escalation section is gone.
+- **A new troubleshooting entry for legitimate per-title fallbacks**, marked as a **hypothesis**.
+  Between two fully hardware sessions, one logged `enabled, but no hardware decode accelerator found`
+  and empty final decoder/encoder. UHD 630 decodes H.264, HEVC 8/10-bit and VP9 but **not AV1** and
+  not 10-bit 4:2:2 HEVC, so the likely cause is the source file, not the platform — explicitly
+  written as a lead to check first, **not** a confirmed diagnosis of that item.
+- The status is stated at the top of the page and in a rewritten provenance footer that separates the
+  2026-10-02 confirmations from the superseded 2026-09-30/2026-10-01 hostPath-era readings.
+- `runbooks/README.md`'s index entry for the page was stale in the same two ways (it still described a
+  `/dev/dri` hostPath and the dashboard-only proof) and is corrected to match.
+
+Unchanged on purpose: the "STOP HERE IF PLEX IS `Pending`" section and the escape hatch (delete the
+`hardware:` block from `catalog/plex.yaml`, re-render) keep exactly their existing prominence — they
+are correct and they are the 2am material. The "no alert is possible" reasoning is also kept verbatim
+in substance: Plex declares no `metricsEndpoint`, renders no ServiceMonitor, and a fallback alert
+would be a rule evaluating against nothing.
+
+**Documentation only.** No change to `catalog/`, `schemas/`, `apps/` or `platform/`.
+
+## What I verified
+
+- **Gate, in the pinned toolbox from this worktree** (`DEV_KUBECONFIG=/dev/null`, `ENGINE` shim
+  bind-mounting the main checkout's `.git`): **rc 3 UNDETERMINED**, diff base
+  `origin/main @ 948875aad37d`, with exactly the three expected skips and **no failures**. Green:
+  `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, `approval-policy`. Skipped:
+  `spec-immutability` (spec repo unavailable → R-TEST-06 not verified), `gitea-settings`
+  (`SETTINGS_API_TOKEN` unset), `conformance` (`CLUSTER_AVAILABLE!=true` → L2/L3 not run).
+  Touched requirements: **R-OBS-04**, reported `UNVERIFIED` for lack of a cluster. `approval-policy`
+  passed as `N/A` locally (`event='none'`, not a `pull_request`) — it evaluates for real on the PR.
+  **rc 3 is not green**, it is the calibrated worktree result, and it is reported here as rc 3.
+- **The gate was run three times, with the same rc 3 / three-skip result on each committed tip** —
+  including the final one. Worth recording from the first attempt: run *before* committing, with
+  `HEAD == origin/main`, the touched-requirement set is `NOT DETERMINABLE (on-base)` and the
+  R-TEST-03 coverage gate cannot be applied at all. Commit before gating from a fresh worktree branch.
+- **`runbooks/kustomization.yaml` still carries `- plex-quicksync=plex-quicksync.md`** in the
+  `configMapGenerator` `files:` list (line 110) — checked explicitly, because without it the page
+  merges and still 404s. `README=README.md` is likewise present, which is what serves the corrected
+  index entry.
+- **This change touches no `sensitive_paths`.** `infra/gitea/approval-policy.json` lists `ci/**`,
+  `.gitea/**`, `conformance/**`, `clusters/**`, `bootstrap/**`, `infra/gitea/**`,
+  `platform/kyverno/**`, `platform-config/kustomization.yaml`, `.sops.yaml`, `**/*.sops.yaml`,
+  `**/rbac.yaml`, `CLAUDE.md`. `runbooks/**` and `decisions/**` are not among them, and the
+  `approval-policy` stage passed.
+- **No YAML was added or edited**, so `indent-sequences: consistent` (which blocked a sibling branch
+  today) is not in play here; `lint` passed over the full tracked YAML set regardless.
+- **`DECISIONS-LOG.md` is untouched** — this handoff is its own file, per CLAUDE.md §7.
+- **The PR's own CI run was read, not assumed** (PR #39, run 208, task 261, head `1677dff5688b`).
+  **It is RED, and not because of this branch.** Every stage this change could affect is green —
+  `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, `approval-policy` (`approval-policy` evaluated
+  for real on the `pull_request` event and passed: no sensitive path touched). The single failure is
+  **`gitea-settings`**, quoting the measured field:
+
+      [fail]   required_approvals: DECLARED 1 but LIVE is 0
+      [fail] Gitea has DRIFTED from infra/gitea/expected-branch-protection.json.
+      [fail] MERGE GATE: BLOCKED
+
+  Every other declared setting matched (`block_admin_merge_override`, `block_on_rejected_reviews`,
+  `dismiss_stale_approvals`, `enable_status_check`, `enable_push=false`,
+  `status_check_contexts = ["gate / gate (pull_request)"]`, `runners online: 1`). So **live `main`
+  currently requires ZERO approvals**, the declaration still says 1, and with
+  `REQUIRE_GITEA_SETTINGS: "true"` in `.gitea/workflows/gate.yml` that drift **blocks every PR in
+  the repo**, not just this one. Note the shape of it: the check that would notice approvals are
+  unenforced is the check that is failing. Local branch `fix/admin-approval-override` (`616a497`,
+  "fix(ci): let an administrator merge their own green PR") looks like the in-flight work that means
+  to change the declaration. **I did not touch it** — `infra/gitea/**` is a sensitive path, needs a
+  non-author approval, and is outside this branch's documentation-only scope. It is an operator
+  decision: re-assert the protection, or change the declaration deliberately in its own PR.
+- The repository-wide grep for `992` to find what else still asserts it: remaining occurrences are in
+  `catalog/plex.yaml`, `catalog/render.py`, `schemas/renderer-contract.md`,
+  `platform/intel-gpu-plugin/README.md`, `platform/intel-gpu-plugin/kustomization.yaml`, and three
+  older `decisions/` entries.
+
+## What I did NOT verify
+
+- **I did not re-measure anything on the live cluster in this session.** Every platform number on the
+  page — the VAAPI/zero-copy log lines, `RW-OPEN-OK` as uid 1000, the `gpu.intel.com/i915` limits and
+  requests on the pod spec, `Groups: 0 100 1000`, `id plex`, the 27 `Preferences.xml` attributes, the
+  empty `/proc/*/fd` sweep — came to me as measurements taken earlier on 2026-10-02 and was
+  transcribed, not re-run. Per CLAUDE.md §2b, treat the page as reporting those readings rather than
+  as proof of current state; re-run the primary check before relying on it.
+- **The per-title fallback cause is unverified and labelled as such.** The source codec of the title
+  that logged `no hardware decode accelerator found` was never read. The AV1 / 10-bit-4:2:2-HEVC
+  explanation is inferred from UHD 630's capability set alone.
+- **Whether Plex's dashboard *ever* shows `Transcode (hw)` on this deployment is unknown.** It was
+  observed silent during working hardware transcoding; whether that is a client/version quirk, a
+  reporting bug, or something about this setup was not investigated. The page therefore demotes it
+  rather than explaining it.
+- **The commands as written in the page were not executed from the page.** They are the forms that
+  were run, re-wrapped for `dev.sh`; in particular the `exec -it deploy/plex -- sh` route for the argv
+  check is noted as possibly lacking a TTY under `dev.sh run` and was not exercised in that shape.
+- **Stale `992` copies outside `runbooks/` were left alone**, deliberately — scope was documentation
+  only, and `catalog/plex.yaml`, `catalog/render.py`, `schemas/renderer-contract.md` and
+  `platform/intel-gpu-plugin/` all still carry the hostPath-era `Groups: 44 100 992 1000` reading as
+  if current. `render.py --selftest` asserts around that comment, so correcting it is a real change
+  with gate consequences, not a doc edit. **That is open work.** Older `decisions/` entries also cite
+  992; those are historical records and should stay as written.
+- **CI on the PR.** The branch is pushed and the PR opened; I did not watch the Gitea run report, so
+  nothing here should be read as "CI green". The only gate result I can quote is the local rc 3 above.
@@ -0,0 +1,45 @@
+---
+date: 2026-10-03
+branch: fix/node-keycloak-hosts-pin
+title: Pin the OIDC issuer to the Gateway VIP on prod nodes — the WAN answer does not hairpin
+---
+
+## What changed
+
+Follow-up to PR #40 (`decisions/2026-10-03-public-dns-wan-target.md`). Public DNS for
+`*.home.keithjackson.dev` now CNAMEs to the WAN IP. The prod nodes resolve through public DNS,
+and the router does not hairpin, so the kube-apiserver could no longer reach its OIDC issuer
+`keycloak.home.keithjackson.dev`.
+
+- `roles/node_baseline`: a new `node_baseline_hosts_pins` list (default `[]`) and one
+  `blockinfile` task (tags `dns`, `hosts`). It writes the pins into `/etc/hosts` as a single
+  marked block. An empty list removes the block.
+- `group_vars/k3s_prod/vars.yml`: pins `keycloak.{{ k3s_base_domain }}` → `192.168.2.20`, which
+  must stay equal to `GATEWAY_VIP` (parity comment added). It also replaces the stale "no
+  /etc/hosts pin is needed" paragraph with the measured reason the pin is now required.
+
+Dev is unaffected (empty default).
+
+## What I verified
+
+- After PR #40 reconciled (`platform` lastAppliedRevision `576e1ab`, Healthy=True), on
+  k3s-prod-2 and k3s-prod-3: `getent hosts keycloak.home.keithjackson.dev` →
+  `136.51.115.252 wan.home.keithjackson.dev`. `curl https://keycloak…/realms/homelab/.well-known/openid-configuration`
+  → exit 60, `ssl_verify_result=18` (the router's self-signed cert, not the Gateway).
+- `journalctl -u k3s` (last 20 min) showed no OIDC/JWKS errors yet. The apiserver still holds
+  the keys it cached before the flip, so the failure is latent until a k3s restart or key rotation.
+- Neither node has cloud-init (`/etc/cloud` absent), so nothing rewrites `/etc/hosts` at boot.
+- `--check --diff --tags hosts --limit k3s-prod-2,k3s-prod-3`: each node gets exactly the
+  three-line marked block `192.168.2.20	keycloak.home.keithjackson.dev`.
+- Outside-in, PR #40 works: check-host.net probes from 3 external nodes got `200 OK` from
+  `https://plex.home.keithjackson.dev/identity` at `136.51.115.252`.
+
+## What I did NOT verify
+
+- **k3s-prod-1**: Ansible cannot reach it. SSH host key verification fails (`known_hosts:94`,
+  ED25519 `SHA256:zgUS8OeWznzCX9hlN7XdZ8iLNkU1G2loXjtJFfEUYpE`), which is consistent with
+  the 2026-09-11 SSD replacement and reinstall. The operator must confirm the fingerprint
+  before the pin can be applied there.
+- The playbook has not been applied for real. It is an operator run after merge:
+  `ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`.
+- An end-to-end kubectl OIDC login after the pin (needs a browser).
@@ -0,0 +1,129 @@
+---
+date: 2026-10-03
+branch: feat/plex-remote-access-l4
+title: Plex remote access — raw tcp/32400 via a pinned LoadBalancer VIP, not the L7 Gateway
+---
+
+## What changed
+
+Reported symptom: users cannot reach Plex from outside the LAN. **Two independent causes**, measured
+rather than inferred:
+
+1. **Plex advertised no remote connection at all.** Live `Preferences.xml` on prod:
+   `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while
+   `PublishServerOnPlexOnlineKey="1"` — publishing was on and there was nothing reachable to publish.
+   Plex's automatic UPnP/NAT-PMP path cannot work from a pod on the Cilium overlay.
+2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` refused;
+   `136.51.115.252:443` answered with a **self-signed router cert** (`C=ZZ, CN=%D`), not the
+   Gateway's LE wildcard. The Plex Service was `ClusterIP` only — no off-cluster path to 32400 by
+   any route.
+
+This branch fixes the in-cluster half. **Remote access does not work until the operator also does
+two manual steps** (`runbooks/plex-remote-access.md`): the WAN tcp/32400 port-forward, and ticking
+"manually specify public port" in the Plex UI.
+
+- **`apps/wave-0/plex-remote.yaml` (new, hand-written).** A `type=LoadBalancer` Service on tcp/32400
+  pinned to `${PLEX_L4_VIP}` via `lbipam.cilium.io/ips`, plus a `CiliumNetworkPolicy`
+  (`plex-allow-remote-32400`) admitting `world`/`cluster`/`host`/`remote-node` to 32400. Non-catalog
+  because `schemas/servicedefinition.schema.json` is `additionalProperties: false` and has no field
+  for a LoadBalancer Service or a pinned VIP — the same reason `apps/wave-0/pihole-dns.yaml`,
+  `platform/crafty/service-l4.yaml` and `apps/vms/unifi/50-service-l4.yaml` are hand-written.
+- **`catalog/render.py`:** an `EXTRA_UNITS` entry (`plex-remote`, anchored to `plex`, `above: False`)
+  so `--emit-waves` places the unit in plex's own wave.
+- **`clusters/base/cluster-vars.yaml`:** new `PLEX_L4_VIP` (dev default `192.168.1.57`), plus the
+  substitution inventory comment updated to name `plex-remote: PLEX_L4_VIP`.
+- **`clusters/prod/kustomization.yaml`:** prod override `PLEX_L4_VIP: 192.168.2.24`.
+- **`apps/wave-0/kustomization.yaml`:** `plex-remote.yaml` added to `resources`.
+- **`runbooks/plex-remote-access.md` (new):** the two manual steps, the four-step verification, why
+  no alert is possible, and the two documented warts below.
+
+**`ADVERTISE_IP` was deliberately NOT set.** Setting it to `https://plex.home.keithjackson.dev:443`
+was the alternative — it needs no new WAN hole and the mechanism was verified working on an
+already-claimed server. **The operator chose the native path (2026-10-03)** so Plex terminates its own
+TLS (`*.plex.direct`) and streams bypass the Envoy L7 hop. `catalog/plex.yaml`'s deferral note for
+`ADVERTISE_IP` remains accurate and untouched.
+
+**`externalTrafficPolicy: Cluster`, with a known cost.** `Local` would preserve the client IP, but
+Cilium's L2 announcer does not follow the backend pod — the wart that cost this LAN an 8-day
+Minecraft outage (see `platform/crafty/service-l4.yaml`). Consequence accepted and documented: SNAT
+puts the client in `192.168.0.0/16`, inside `ALLOWED_NETWORKS`, so **Plex classifies remote streams
+as LAN**. That is a metering effect (remote bandwidth limits will not bind), not an authorization one.
+
+**SENSITIVE PATHS — this PR touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK
+it.** It needs the operator's `/approve-override <reason>` posted as a PR **review** of type
+"Comment". I have deliberately not posted it: CLAUDE.md §1 forbids an agent clearing this on its own
+PR, and the agent account holds `owner`, so the prohibition is a rule, not a limit.
+
+## What I verified
+
+All measurements against live prod, 2026-10-03, read-only.
+
+- **Plex's advertised state**, the finding this branch rests on: `customConnections=<ABSENT>`,
+  `ManualPortMappingMode=<ABSENT>`, `ManualPortMappingPort=<ABSENT>`,
+  `PublishServerOnPlexOnlineKey="1"`, `allowedNetworks="10.0.0.0/8,192.168.0.0/16"`. Read by
+  attribute grep, never by dumping `Preferences.xml` (it holds `PlexOnlineToken`).
+- **The L7 Gateway path works and is NOT the problem.** Via the Gateway VIP,
+  `GET https://plex.home.keithjackson.dev/identity` → `HTTP 200`, `claimed="1"`, real
+  `machineIdentifier`. HTTPRoute `plex-32400`: `Accepted=True`, `ResolvedRefs=True`.
+- **Plex enforces its own auth**, which is what makes an unfiltered WAN port defensible:
+  `/library/sections`, `/status/sessions`, `/:/prefs` all → **401** unauthenticated through the
+  Gateway.
+- **The SNAT/allowedNetworks wart is real, not theoretical.** A tagged probe through the Gateway was
+  logged by Plex as `[10.0.2.214:36040 (Allowed Network (WAN))]` — the Envoy pod IP, classified as an
+  allowed network.
+- **WAN state:** `136.51.115.252:32400` → connection refused. `136.51.115.252:443` → self-signed
+  cert `C=ZZ, ST=Somewhere, O=%D342e7e6b, CN=%D`. Public DNS resolves correctly:
+  `plex.home.keithjackson.dev` → CNAME `wan.home.keithjackson.dev` → `136.51.115.252`.
+- **`192.168.2.24` is free in-cluster.** `CiliumLoadBalancerIPPool default-pool`: IPsTotal 11 /
+  IPsUsed 4 / IPsAvailable 7; the only allocations are `.20` (cilium-gateway-shared), `.21`
+  (unifi-l4-*), `.22` (crafty-l4-*), `.23` (pihole-dns-*). Not a node address (`.10/.11/.12`) and not
+  the kube-vip control-plane VIP (`.9`).
+- **The manual-port-mapping step genuinely cannot be automated.** Grep of the running image's whole
+  init surface (`/etc/cont-init.d`, `/plex-common.sh`, `/usr/local/bin`) for `ManualPortMapping` and
+  `PLEX_PREFERENCE` → **NONE**.
+- **`ADVERTISE_IP` would have worked on a claimed server** (recorded because the runbook's
+  "why not" rests on it): `setPref "customConnections"` at line 126 of `40-plex-first-run` sits
+  **outside** the `PLEX_CLAIM`/`PlexOnlineToken` guard, and the `/.firstRunComplete` early-exit
+  marker is on the **overlay** FS, not the `/config` PVC, so the script re-runs every pod start.
+- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
+  `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
+  **pass**. The rc 3 is two environmental skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`)
+  and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3, not as pass.**
+- **`render-golden` passing is the specific proof** that the `render.py` `EXTRA_UNITS` change, the new
+  hand-written unit and the edited wave kustomization are mutually consistent.
+
+## What I did NOT verify
+
+- **THE ACTUAL FIX. Nothing in this branch has been proven to make remote access work** — it cannot
+  be, because both remaining steps are off-repo and manual (router forward, Plex UI toggle). This
+  branch removes two blockers and adds none; it does not demonstrate an outcome.
+- **Outside-in reachability, at all.** Every probe above was taken from inside the LAN, so the
+  `:443` self-signed cert strictly proves only that there is **no NAT hairpin** — it does **not**
+  prove outside-in failure. A router cert on WAN:443 is suggestive (its management UI would shadow a
+  forward), but unproven. **This repo has no off-LAN vantage point.** Test 3 in the runbook
+  (app.plex.tv from a cellular connection) is the only honest check.
+- **`192.168.2.24` was never ARP-probed off-cluster.** Only in-cluster pool accounting was checked.
+  `.23` was `arping`-probed before it was claimed; `.24` was not, so an off-cluster squatter would
+  not have been caught.
+- **The LoadBalancer has never been reconciled.** `plex-l4-tcp` does not exist in prod yet, so
+  LB-IPAM has not actually assigned the pinned VIP, L2 announcement has not been observed for it, and
+  the new CiliumNetworkPolicy has never admitted a real packet. Whether `world`/`host`/`cluster`/
+  `remote-node` is the correct entity set **under this SNAT path** is reasoned from
+  `platform/crafty/networkpolicy.yaml`'s working game-port rule, **not measured**.
+- **Whether Plex's remote path needs anything beyond the manual port tick.** If plex.tv still lists
+  no remote connection after both manual steps, this branch is necessary-but-insufficient.
+- **The CI approval-policy outcome.** The local stage reported green, but that is an offline N/A, not
+  a verdict: this PR touches `clusters/**`, so CI will block it pending an override.
+- **PRE-EXISTING DRIFT I FOUND AND DELIBERATELY LEFT ALONE — worth its own follow-up.** Running the
+  documented `python3 catalog/render.py --emit-waves` regenerates the wave kustomizations and wants to
+  add **`vm-demo.yaml`** to wave 0 and **`keycloak.yaml`** to wave 1. Both are listed as
+  intentionally not onboarded in `ci/render-golden.sh`
+  (`RENDER_GOLDEN_NOT_ONBOARDED="keycloak vm-demo"`); their manifests are tracked in `apps/` but
+  deliberately absent from the `resources` lists so Flux does not apply them. `--emit-waves` places
+  any file present on disk and has **no awareness of that exclusion**, while `render-golden.sh` uses
+  `--wave-layout` and compares rendered manifests only — it never checks kustomization `resources`,
+  which is why this has survived unnoticed. **So the header "DO NOT hand-edit: re-run
+  `render.py --emit-waves`" is currently unsafe advice**: following it onboards an IdP and a KubeVirt
+  VM. I therefore restored both kustomizations and hand-added only my one line. I did not fix
+  `--emit-waves`, and I have not determined how long the drift has existed or whether either service
+  is *meant* to be onboarded.
@@ -0,0 +1,73 @@
+---
+date: 2026-10-03
+branch: feat/public-dns-wan-target
+title: Public DNS now points at the WAN — Gateway-published names CNAME to an in-cluster DDNS record
+---
+
+## What changed
+
+The operator closed the open "public-target gap" recorded in
+`platform/externaldns/helmrelease.yaml`. The WAN NAT / port-forwarding (80/443 → Gateway VIP
+192.168.2.20) was done by the operator outside this repo.
+
+Operator decisions (2026-10-03):
+
+1. **Scope: every Gateway-published name goes public.** That covers the five `exposure: public`
+   apps (plex, gitea, keycloak, nextcloud, seerr) and also the hand-authored admin UIs (auth,
+   crafty, grafana, headlamp, the `home.keithjackson.dev` apex). `internal` services still publish
+   nothing: their `controller: none` route annotation is checked before the target is read.
+2. **The WAN IP is kept current in-cluster**, not hardcoded.
+
+Implementation:
+
+- `platform/gateway/gateway.yaml`: `external-dns.alpha.kubernetes.io/target: wan.${BASE_DOMAIN}` on
+  the shared Gateway. Every gateway-sourced record changes from `A 192.168.2.20` to
+  `CNAME wan.home.keithjackson.dev`.
+- `platform/externaldns/ddns.yaml` (new): `favonia/cloudflare-ddns:1.17.1` Deployment in the
+  `external-dns` namespace. It reuses `cloudflare-api-token` and keeps `wan.${BASE_DOMAIN}` an
+  unproxied IPv4 A record at the WAN IP (detected via Cloudflare trace, checked every 5 min,
+  `DELETE_ON_STOP=false`). external-dns has no TXT registry entry for it, so it never owns or
+  deletes it.
+- The helmrelease comment block and the README now describe the decision instead of the gap.
+
+Known consequence: **k3s nodes resolve through public DNS** (`node_baseline_dns_servers` = 8.8.8.8 /
+1.1.1.1). The kube-apiserver's OIDC issuer `keycloak.home.keithjackson.dev` therefore resolves to the
+WAN IP from the nodes, and kubectl SSO works only if the router hairpins WAN:443 back to the Gateway.
+A LAN-side probe of `136.51.115.252:443` on 2026-10-03 returned a **self-signed placeholder cert**,
+not the Gateway's Let's Encrypt wildcard. That suggests no hairpin. The cert-based admin kubeconfig
+is unaffected.
+
+## What I verified
+
+- Live prod, 2026-10-03, before the change: Gateway `gateway/shared` ADDRESS `192.168.2.20`.
+  External-dns args include `--source=gateway-httproute --source=service --policy=sync
+  --txt-owner-id=prod --domain-filter=keithjackson.dev`, and it logs "All records are already up to
+  date".
+- Public answers at 1.1.1.1: plex/gitea/keycloak/grafana/auth/apex → `192.168.2.20`;
+  `crafty-mc` → `.22` and `unifi` → `.21` (service source, not affected by this change); jackett →
+  none (internal, correctly unpublished).
+- The authority (betty.ns.cloudflare.com) has no unprefixed TXT, AAAA or MX at `plex.` or the apex,
+  so the A→CNAME switch has no co-existing record to collide with.
+- WAN IP from the workstation: `136.51.115.252`.
+- No CiliumNetworkPolicy or NetworkPolicy in `external-dns`. The Kyverno label and default-deny
+  policies match only `app-namespace` namespaces. The vpn egress policy selects only
+  `egress-class: vpn` pods.
+- Toolbox gate result: see the PR description (run after this commit).
+
+## What I did NOT verify
+
+- **The outside-in path.** I have no vantage point outside the LAN. Whether
+  `https://<svc>.home.keithjackson.dev` from the internet reaches the Gateway with the LE cert is
+  UNVERIFIED.
+- **Hairpin / kubectl SSO after the flip.** Likely broken per the probe above. Fix candidates: a
+  node `/etc/hosts` pin for `keycloak.${BASE_DOMAIN}` → Gateway VIP via `node_baseline`, or router
+  NAT reflection.
+- **The record-type flip itself**, until Flux reconciles: whether Cloudflare accepts external-dns's
+  create-CNAME in the same batch as the delete-A, or converges one interval later.
+- `favonia/cloudflare-ddns` under this pod securityContext. The README warns that
+  `no-new-privileges` misbehaves on some Proxmox-style hosts. These are bare-metal nodes, but it is
+  untested until the pod runs.
+- Whether the L4 service-source names (crafty-mc, unifi) should also go public. Not decided, not
+  changed.
+- `conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh` header lines 39–40
+  still describe the target as open. That is a sensitive path, so it is left for a follow-up.
@@ -0,0 +1,133 @@
+---
+date: 2026-10-04
+branch: feat/pihole-declare-state
+title: PiHole's operator-set state is now declared in git — the prerequisite for running more than one replica
+---
+
+## What changed
+
+Phase 1 of making LAN DNS survive a node failure. The operator asked for multi-replica PiHole;
+measurement showed that cannot be done safely first, because PiHole's important state was not
+reproducible from git. A second replica with a fresh PVC would have come up **serving DNS and
+blocking nothing, with a different admin password** — answers varying by which pod replied. This
+branch closes that; the replica work is phase 2 and is deliberately NOT here.
+
+**The undeclared-state inventory, measured not guessed.** `pihole.toml` marks every non-default
+setting `### CHANGED`, and those that came from the environment `### CHANGED (env)`. The complete
+live list on 2026-10-04:
+
+| Setting | Status |
+|---|---|
+| `dns.upstreams`, `dns.listeningMode`, `dns.rateLimit.count`, `misc.dnsmasq_lines` | `### CHANGED (env)` — already declared in `catalog/pihole.yaml` |
+| `files.macvendor` | `### CHANGED` — set by the image itself (container path), not operator state |
+| `webserver.api.pwhash` | **`### CHANGED` — set out of band. Declared NOWHERE.** |
+
+So exactly **one** operator-set value lived only on the PVC. Separately, the **adlists** were also
+undeclared: `gravity.db` held one list (StevenBlack's hosts) and 72,234 gravity domains, with nothing
+in this repo naming it.
+
+- **`apps/wave-0/pihole-bootstrap-secret.sops.yaml` (new, SOPS/age).** Secret `pihole-bootstrap`,
+  key `webui-password`. Named to the `<svc>-bootstrap-secret.sops.yaml` convention so
+  `render.py --emit-waves` places it with no renderer change. The value is a 32-char random password
+  generated at authoring time and **never printed to a transcript**.
+- **`catalog/pihole.yaml`:** `FTLCONF_webserver_api_password` sourced from that Secret.
+  `webserver.api.password` takes CLEARTEXT and FTL hashes it itself — verified in the running image,
+  where `pihole setpassword` *is* `setFTLConfigValue "webserver.api.password"`, commented upstream
+  "pihole-FTL will automatically hash the password". The PVC's `pwhash` becomes a derived artefact.
+- **`catalog/pihole.yaml`:** a `lifecycle.postStart` hook that **reconciles** the declared adlist set
+  into `gravity.db` on every container start — adds missing, re-enables disabled, and **disables**
+  (never deletes) anything enabled but undeclared, running `pihole -g` only when something moved.
+  Same level-triggered idiom, and the same reasoning, as `catalog/qbittorrent.yaml`'s postStart.
+- **`apps/wave-0/pihole.yaml`, `apps/wave-0/kustomization.yaml`:** re-rendered / secret registered.
+
+**Why a postStart hook and not a file.** PiHole v6 keeps adlists as DB rows. `/etc/pihole/adlists.list`
+still exists in `gravity.sh` but is **migration-only**: its import sits inside `migrate_to_database()`
+behind `if [ ! -e "$gravityDBfile" ]`, so it fires only while *creating* a database. A ConfigMap
+mounted there would be silently ignored on every pod this cluster runs today. There is also no
+`pihole adlist` CLI subcommand, so SQL is the actual interface; FTL ships an embedded shell
+(`pihole-FTL sqlite3`) so nothing had to be added to the image.
+
+**The password is kept rather than set empty, deliberately.** The admin UI is already behind the
+oauth2-proxy sidecar (`--allowed-group=platform-admins`), so an empty password would have been
+simpler. But the `pihole` Service still publishes **port 80** (PiHole direct, bypassing the proxy on
+`:4180`) and only the CiliumNetworkPolicy closes it — and this repo has a documented default-deny
+drift history. An unauthenticated admin UI that can rewrite LAN DNS should not rest on one policy
+staying correct. Narrowing that Service port is follow-up, not done here.
+
+**SENSITIVE PATHS — this PR adds a `**/*.sops.yaml`, so `ci/approval-policy.sh` will BLOCK it.** It
+needs the operator's `/approve-override <reason>` as a PR **review** of type "Comment". Not posted by
+me: CLAUDE.md section 1 forbids an agent clearing this on its own PR.
+
+## What I verified
+
+Live prod, 2026-10-04, read-only except where noted.
+
+- **The undeclared-state inventory above** — read from `pihole.toml`'s own `### CHANGED` markers, with
+  the password value masked. This is the measurement the whole branch rests on.
+- **A fresh `gravity.db` seeds NO adlists.** The image's own schema template
+  `/etc/.pihole/advanced/Templates/gravity.db.sql` creates the `adlist` table and contains no
+  `INSERT INTO adlist` at all — grepped in the running image. Confirmed empirically below (0 rows).
+  This is why the adlist declaration is load-bearing and not cosmetic.
+- **The adlists.list path really is migration-only** — read the guard in `/opt/pihole/gravity.sh`
+  (`migrate_to_database()`, `if [ ! -e "$gravityDBfile" ]`).
+- **`webserver.api.password` is a real settable key** — `pihole-FTL --config webserver.api.password`
+  returns a masked value, and `/usr/local/bin/pihole`'s `setpassword` sets exactly that key.
+- **THE RENDERED HOOK ACTUALLY WORKS — all four paths, extracted from `apps/wave-0/pihole.yaml` and
+  run in the live pod against THROWAWAY COPIES of `gravity.db` in `/tmp`** (production `gravity.db`
+  was never written, and `pihole -g` was stubbed so no 70k-domain download could start):
+  1. *Steady state* → `already matches the declared set - nothing to do`. No writes, no rebuild.
+  2. *True fresh DB built from the image's schema template* → 0 adlists before; hook added the
+     declared list and triggered the rebuild. **This is the fresh-PVC/new-replica case.**
+  3. *An undeclared list enabled* → disabled it (`enabled=0`), left the declared one alone, and did
+     not delete the row.
+  4. *Declared list present but disabled* → re-enabled it.
+- **Required tools exist in the image:** `bash`, `expr`, `grep`, `printf`, `sleep`, `pihole-FTL`.
+- **The Secret is genuinely encrypted** to the repo's age recipient; `stringData.webui-password`
+  starts `ENC[`. The gate's `no-plaintext-secrets (R-SEC-01)` L0 check passes.
+- **Schema:** `check-jsonschema` against `schemas/servicedefinition.schema.json` passes.
+- **Render determinism:** the canonical
+  `render.py --all catalog -o apps --wave-layout --no-image-automation` changed **only**
+  `apps/wave-0/pihole.yaml`. `render-golden` passes, so the hook's text also satisfies
+  `assert_substitution_safe` (no braced dollar expansion — the Flux envsubst contract).
+- **Toolbox gate:** rc **3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
+  `render-golden`, `l1`, `approval-policy` all **pass**. The rc 3 is the two usual environmental
+  skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`) and `conformance` L2/L3
+  (`CLUSTER_AVAILABLE` unset). **Not green; reported as rc 3.**
+
+## What I did NOT verify
+
+- **Nothing here has been reconciled.** Flux has not applied any of it, so the password has never
+  been set by this mechanism and the hook has never run during a real container start. Everything
+  above is the rendered artifact tested out-of-band, plus the image's own source.
+- **THE CURRENT ADMIN PASSWORD WILL STOP WORKING, and that is the intent.** FTL re-hashes
+  `webserver.api.password` on start, overwriting the existing `pwhash`. Whoever knows the present
+  password (nobody, which is why this started) loses it; the declared one replaces it. I have not
+  observed that replacement happen.
+- **Cold-start cost is unmeasured and could matter.** `postStart` must complete before the container
+  is reported Running. On a fresh PVC the hook waits for `gravity.db` (bounded, up to 180s) and then
+  runs a real `pihole -g` (72k domains, network downloads). `apps-wave-0` is `wait: true` and
+  `apps-wave-1` `dependsOn` it, so a slow cold start **delays the app tier**. The steady-state path
+  exits in about a second (scenario 1), so this is a rebuild/first-boot cost only — but it is not
+  measured, and the stub in testing means `pihole -g`'s real duration here is unknown.
+- **Whether `pihole -g` succeeds from inside the hook's environment.** Its failure is caught and the
+  existing `gravity.db` retained, so a failure degrades rather than breaks — untested live.
+- **Concurrency.** Nothing serialises the hook against FTL's own scheduled gravity update. SQLite
+  handles the locking, and the hook's writes are three small statements, but no interleaving was
+  exercised.
+- **MULTI-REPLICA — PHASE 2, NOT STARTED.** This branch only removes the blocker. The remaining work,
+  and it is substantial: a `replicas`/placement field on the ServiceDefinition schema (currently
+  `additionalProperties: false` with no such field), `render.py` emitting a StatefulSet with
+  `volumeClaimTemplates` (the config PVC is **ReadWriteOnce**, so a 2-replica Deployment cannot
+  work), pod anti-affinity, a PodDisruptionBudget (there is none today), re-rendered goldens, and
+  `conformance/**` updates — the last making phase 2 sensitive-path as well. The PVC rename a
+  StatefulSet forces will discard the existing `gravity.db`, which is only safe **because** of this
+  branch, and should be done only after this one is confirmed working in-cluster.
+- **The real risk the operator named is NOT fixed by phase 2 either.** Replicas address *node*
+  failure. LAN DNS still depends on the cluster as a whole — etcd quorum, Cilium, Longhorn, a bad
+  reconcile. Quantified on 2026-10-04: with `tolerationSeconds=300` plus node-NotReady detection
+  plus a Longhorn RWO force-detach, a failure of the node running the pod is roughly **6-10 minutes
+  of no LAN DNS**; a failure of the node holding the `pihole-dns-*` L2 lease (currently a
+  *different* node — lease on k3s-prod-2, pod on k3s-prod-1) is tens of seconds. An out-of-cluster
+  secondary resolver carrying the same `address=` lines is the only option that removes the
+  dependency, and it was considered and not chosen.
+- **Port 80 on the `pihole` Service is still published** and still closed only by policy. Not narrowed.
@@ -0,0 +1,117 @@
+---
+date: 2026-10-04
+branch: docs/pihole-ha-verified
+title: PiHole HA is VERIFIED on live prod — including a failover drill with zero dropped DNS queries
+---
+
+## What changed
+
+**No manifests changed. This entry exists to correct the log.** The three PiHole/Plex entries merged
+earlier today each say, correctly at the time, that nothing had been reconciled and that the outcome
+was unverified:
+
+- `decisions/2026-10-04-pihole-declare-state.md` — "Nothing here has been reconciled."
+- `decisions/2026-10-04-pihole-multi-replica.md` — "Nothing has been reconciled. No StatefulSet
+  exists, no second replica has ever run, the anti-affinity has never been evaluated..."
+
+All three are now **merged, reconciled and measured**, so those sentences are stale in exactly the way
+§2b of CLAUDE.md warns about — a reader who found them would conclude the work is unproven. This entry
+is the current state; it does not edit the earlier files (separate files, nothing to merge).
+
+Delivered and live as of 2026-10-04, from PRs #43 (plex tcp/32400), #44 (declare PiHole state) and
+#45 (PiHole 2 replicas + the mountPath bug):
+
+- PiHole is a **2-replica StatefulSet**, one pod per node, with per-replica Longhorn volumes, a
+  PodDisruptionBudget, and an `up` SLI that follows the rendered workload kind.
+- PiHole's admin password and adlists are **declared in git** and reproduce onto a fresh volume.
+- PiHole's config PVC is mounted where PiHole actually writes (`/etc/pihole`), which it was not before.
+
+## What I verified
+
+All measured against live prod on 2026-10-04, after the merge of #45 (`dce20b7`).
+
+**Reconcile.** `GitRepository` artifact `main@sha1:dce20b7`; `apps-wave-0`
+`lastAppliedRevision=main@sha1:dce20b7`, `Ready=True/ReconciliationSucceeded`. It completed in roughly
+**90 seconds**, well inside the Kustomization's `timeout: 5m` — the risk flagged in the multi-replica
+entry ("whether `pihole -g` finishes inside the 5 m timeout") did **not** materialise. A transient
+`Ready=False/DependencyNotReady: platform-config` seen mid-flight was the dependency cascade, not a
+failure; `platform-config` was already `Ready=True` at the new revision moments later.
+
+**Topology and objects.**
+
+| Measured | Value |
+|---|---|
+| pods / nodes | `pihole-0` → **k3s-prod-3**, `pihole-1` → **k3s-prod-2** (2/2 Running each) — the required anti-affinity was satisfied |
+| per-replica claims | `config-pihole-0`, `config-pihole-1` — both `Bound`, 2Gi |
+| old `pihole-config` PVC | **pruned** (it held only `lost+found`, so nothing was lost) |
+| old `pihole` Deployment | **pruned** — `No resources found in pihole namespace` |
+| PodDisruptionBudget | `minAvailable 1`, `disruptionsAllowed 1`, `currentHealthy 2` |
+
+**The mountPath bug is fixed.** `df` inside both pods reports `/etc/pihole` on
+`/dev/longhorn/pvc-...` with **6.9 M used** — previously it was on `overlay` while the Longhorn volume
+sat unused at `/data/config`. `backup.policy: daily-offcluster` now has real content to back up
+instead of reporting success over an empty volume.
+
+**Phase 1's declarations reproduce onto a FRESH volume — the point of doing it first.** Both replicas
+booted with an empty `/etc/pihole`:
+
+- **72,234 gravity domains** on each, rebuilt from the adlist declared in `catalog/pihole.yaml`.
+- **The declared SOPS secret authenticates on both pods.** `POST /api/auth` using the container's own
+  `FTLCONF_webserver_api_password` returned `"valid":true` on `pihole-0` **and** `pihole-1` — so
+  Secret → env → FTL's own hashing → working login, identically on both. (The secret's value was never
+  printed; an earlier reading of `pwhash` as "unset" was a quoting bug in the check, disproven by an
+  empty password being rejected with `password incorrect` and then by this positive test.)
+
+**User-facing behaviour through the VIP (192.168.2.23), from a LAN host.**
+
+    plex.home.keithjackson.dev   -> 192.168.2.20      split-horizon intact
+    home.keithjackson.dev        -> 192.168.2.20
+    unifi.home.keithjackson.dev  -> 192.168.2.21
+    google.com                   -> 142.250.115.102   recursion to 8.8.8.8 works
+    doubleclick.net              -> 0.0.0.0           blocking LIVE in FTL, not just in the DB
+    googleadservices.com         -> 0.0.0.0
+    12 consecutive queries       -> 12/12 answered
+
+**THE FAILOVER DRILL — observed by the operator, 2026-10-04.** A single StatefulSet replica was
+deleted. It was recreated in **~30 seconds**, and **DNS never dropped a request** throughout. This is
+the first direct evidence that the replica count delivers what it was added for, rather than merely
+rendering correctly.
+
+**The SLI is not inert, and re-deriving it mattered.** Measured in Prometheus:
+`kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}` returns **1 series,
+value 2**; `kube_deployment_status_replicas_available{namespace="pihole"}` now returns **0 series**;
+and Prometheus has `pihole-down` loaded as `state=inactive health=ok` with the StatefulSet expression.
+Had the rule stayed Deployment-shaped, its `or absent(...)` branch would have matched the now-absent
+series and **fired permanently** — a standing false page rather than silence. That is the concrete
+payoff of defaulting `livenessWorkload.kind` from the rendered kind.
+
+## What I did NOT verify
+
+- **NODE failure, as opposed to pod failure.** The drill deleted a pod, which does not exercise the
+  `tolerationSeconds: 300` path at all — that timer only applies when a node goes NotReady. The
+  mechanism protecting DNS is the same one just proven (the surviving replica plus
+  `externalTrafficPolicy: Cluster` on the DNS LoadBalancers), so user-visible impact should likewise be
+  nil; what the 300 s still delays is the *replacement* replica, and during that window there is no
+  redundancy left. Untested. Pulling power from a node is the only real test.
+- **DNS during the cutover itself.** I was not sampling resolution while #45 reconciled, so whether
+  there was a brief gap between the Deployment being pruned and the StatefulSet becoming Ready is
+  **unmeasured**. The drill above covers the steady-state failover case, not that one-off transition.
+- **Two-node loss.** The hard anti-affinity means `replicas: 2` needs two schedulable nodes; losing
+  two would leave a replica `Pending` and, with `wait: true`, stall the app tier. Reasoned, not
+  exercised — and it coincides with etcd quorum loss on a 3-node control plane.
+- **Longhorn capacity headroom.** Two 2Gi claims at 3 replicas each is ~12Gi where there was ~6Gi.
+  Both claims are `Bound` and healthy; free space was not checked.
+- **Query history is now split per replica** and the admin UI shows only whichever pod answered.
+  Adlists and password are identical, so blocking and auth are consistent; the *reporting* is not, and
+  whether that is acceptable long-term is the operator's call.
+- **The residual risk is unchanged, and it is the larger one.** Replicas address node failure. LAN DNS
+  still depends on the cluster — etcd quorum, Cilium, Longhorn, a bad reconcile — and only a resolver
+  outside the cluster carrying the same `address=` lines removes that dependency. Considered and not
+  chosen (operator decision, 2026-10-04).
+- **Plex remote access (#43) is NOT covered by this entry.** Its two manual steps (the WAN tcp/32400
+  forward and Plex's "manually specify public port") and the outside-in path remain as
+  `decisions/2026-10-03-plex-remote-access-l4.md` describes them. The operator reported it working;
+  I have not measured it from off-LAN.
+- **`--emit-waves` drift is still open.** Re-running it still wants to onboard `vm-demo.yaml` and
+  `keycloak.yaml`, both listed as intentionally not onboarded in `ci/render-golden.sh`. Unfixed; see
+  `decisions/2026-10-03-plex-remote-access-l4.md`.
@@ -0,0 +1,145 @@
+---
+date: 2026-10-04
+branch: feat/pihole-multi-replica
+title: PiHole runs two replicas on different nodes — and its config PVC was mounted where PiHole never writes
+---
+
+## What changed
+
+Phase 2 of making LAN DNS survive a node failure (phase 1:
+`decisions/2026-10-04-pihole-declare-state.md`). PiHole was one pod on one node, which for a
+whole-house resolver cost more than "until it reschedules" — measured 2026-10-04:
+
+| Failure | Recovery |
+|---|---|
+| the node running the pod dies | ~40s to mark NotReady + `tolerationSeconds: 300` before eviction + reschedule + a Longhorn RWO force-detach → **roughly 6-10 minutes with no LAN DNS** |
+| the node holding the `pihole-dns-*` L2 lease dies | Cilium re-elects an announcer, tens of seconds |
+
+Those were **different** nodes (lease on k3s-prod-2, pod on k3s-prod-1), so two of three nodes were
+single points of failure for LAN DNS with wildly different blast radii.
+
+**A new schema field, because this is intent and not a primitive.** `availability.replicas`
+(schema **1.10.0**, mechanism in `schemas/renderer-contract.md` `### availability`). `1` or absent
+renders a Deployment exactly as before — every pre-existing golden is byte-identical, verified. `>1`
+renders, together:
+
+- a **StatefulSet** instead of a Deployment. This is forced, not preferred: a longhorn PVC is
+  **ReadWriteOnce**, and one RWO claim cannot attach on two nodes, so a 2-replica Deployment does not
+  degrade — the second pod never schedules.
+- longhorn storage as **per-replica `volumeClaimTemplates`** (`config-pihole-0`, `config-pihole-1`);
+  the standalone PVC is no longer emitted, since it would be an orphan nothing mounts. A `bulk`
+  entry stays a standalone PVC — it is a shared static NFS export, not per-replica state.
+- **required** `podAntiAffinity` on `kubernetes.io/hostname`. `preferred` would let both replicas
+  land on one node and deliver none of the intended tolerance while reporting healthy.
+- `podManagementPolicy: Parallel`, so a cold start costs the max of the replicas' first-boot work
+  rather than the sum — `apps-wave-0` has `timeout: 5m`.
+- a **PodDisruptionBudget**, `minAvailable: 1`, so a node drain cannot take the last replica.
+- `observability.livenessWorkload.kind` defaulting to `StatefulSet`, so the `up` SLI reads
+  `kube_statefulset_status_replicas_ready`. A Deployment-shaped rule against a StatefulSet
+  `absent()`s forever — an alert that can never fire.
+
+`catalog/pihole.yaml` sets `replicas: 2`, not 3: two tolerates any single node loss and leaves a
+spare node for the evicted replica under the hard anti-affinity. Three would pin one per node with no
+spare, so a single node failure would leave a replica permanently `Pending` and — `apps-wave-0` being
+`wait: true` — stall the app tier for as long as the node was down.
+
+### The bug found on the way, and fixed here
+
+**PiHole's config PVC was mounted at a path PiHole never writes, so it held nothing and all of its
+state was ephemeral.** With no explicit `mountPath` the renderer defaults to `/data/<name>`; PiHole
+keeps everything in `/etc/pihole`. Measured on live prod 2026-10-04: `/etc/pihole` (7.1 M —
+`gravity.db`, `pihole.toml`, the query DB) was on the **overlay** filesystem, while the 2 Gi Longhorn
+volume sat at `/data/config` holding `lost+found` and 24 K of nothing. Consequences, all live until
+now:
+
+- every restart silently discarded PiHole's entire state and re-downloaded 72 k gravity domains;
+- `backup.policy: daily-offcluster` has been faithfully backing up an **empty** volume, which is
+  worse than no backup because it reports success;
+- it also means phase 1 was understated: that state was not merely undeclared, it was **ephemeral**.
+
+Same class of bug `catalog/plex.yaml` documents for its own `config` entry. Fixed in this branch
+rather than its own because with the wrong path the new per-replica volumes would be two 2 Gi
+Longhorn volumes (×3 Longhorn replicas each) holding only `lost+found` while all state stayed
+ephemeral — paying for persistence and getting none. It also removes this change's main operational
+risk: with `gravity.db` persisted, a restart is cheap instead of re-downloading 72 k domains inside a
+wave with a 5 m timeout.
+
+## What I verified
+
+- **Back-compatibility, the thing most likely to break quietly:** after the renderer change and
+  before `catalog/pihole.yaml` declared anything, the canonical
+  `render.py --all catalog -o apps --wave-layout --no-image-automation` produced **zero diff** across
+  the whole `apps/` tree. `render-golden` passes.
+- **The new behaviour is pinned by ~20 new `render.py --selftest` assertions** (so `ci/render-golden.sh`
+  hard-fails on a regression, and no `conformance/**` change was needed): `replicas: 1` still renders a
+  Deployment + standalone PVC + no PDB; `replicas: 2` renders a StatefulSet and **no** standalone
+  longhorn PVC, carries the replica count, omits `strategy`, sets `Parallel` and `serviceName`, names
+  the claim template exactly as the mounted volume, keeps RWO/longhorn on it, does **not** duplicate
+  that volume in `pod.volumes`, still mounts it on the container, renders required hostname
+  anti-affinity, renders exactly one `minAvailable: 1` PDB, and switches the SLI to
+  `kube_statefulset_status_replicas_ready`. A mixed longhorn+bulk case asserts the bulk share stays a
+  standalone PVC and an ordinary pod volume.
+- **Rendered pihole output, read back:** `replicas: 2`, `serviceName: pihole`,
+  `podManagementPolicy: Parallel`, no `strategy`, claim template `config` (RWO/longhorn/2Gi),
+  anti-affinity `kubernetes.io/hostname`, `pod.volumes: []`, mount `config -> /etc/pihole`,
+  PDB `minAvailable: 1`, SLI `kube_statefulset_status_replicas_ready`.
+- **The `/etc/pihole` mount is safe, evidenced from the image's own init rather than assumed:**
+  `/usr/bin/start.sh` is PID 1 and runs as **uid 0**; its line 5 is
+  `chown -R pihole:pihole /etc/pihole/ /var/log/pihole/`, so a fresh root-owned volume has its
+  ownership fixed before anything else starts (**no `fsGroup` needed, and one would be inert**); it
+  calls `migrate_gravity`, whose `migrate_to_database()` creates the gravity database when
+  `[ ! -e "$gravityDBfile" ]`; FTL writes `pihole.toml` itself from the `FTLCONF_*` env; and the
+  image's own default for `files.macvendor` was moved **out** of that directory, which is what a
+  maintainer does to a path they expect to be a volume.
+- **The PVC really is unused:** `df` showed `/etc/pihole` on `overlay` and
+  `/dev/longhorn/pvc-849edb86-...` on `/data/config`; `ls -A /data/config` → `lost+found` only.
+- **The gate caught a real regression and it is fixed.** The first run came back **rc 1**:
+  `alert-rules-unit-tested` failed because `apps/wave-0/tests/app-slis.test.yaml` fed pihole a
+  `kube_deployment_status_replicas_available` series and expected a `deployment: pihole` label. Three
+  input series, the firing expectation and the file header were updated;
+  `platform/monitoring/tests/run.sh apps/wave-0/tests/app-slis.test.yaml` → **SUCCESS** (6 rules).
+- **Schema:** `check-jsonschema` passes over all of `catalog/*.yaml`.
+- **Toolbox gate:** rc **3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
+  `render-golden`, `l1`, `approval-policy` all **pass**. The rc 3 is the two environmental skips —
+  `gitea-settings` (no local `SETTINGS_API_TOKEN`) and `conformance` L2/L3 (`CLUSTER_AVAILABLE`
+  unset). **Not green; reported as rc 3.**
+- **No sensitive paths touched**, so unlike the previous two branches this one needs no
+  `/approve-override` — confirmed against `infra/gitea/approval-policy.json`.
+
+## What I did NOT verify
+
+- **Nothing has been reconciled.** No StatefulSet exists, no second replica has ever run, the
+  anti-affinity has never been evaluated by the scheduler, the PDB has never been consulted, and the
+  `/etc/pihole` mount has never been exercised on a real pod. Everything above is the rendered
+  artifact plus the image's own source.
+- **THE ROLLOUT WILL BRIEFLY DROP LAN DNS, and I have not measured for how long.** `postStart` must
+  complete before a container is reported Running, so a pod is not Ready — and the Service has no
+  endpoints — until phase 1's adlist hook finishes. On this transition both replicas start with an
+  empty `/etc/pihole`, so the hook finds a fresh `gravity.db` with zero adlists and runs a real
+  `pihole -g` (72 k domains, network download) before either pod becomes Ready. **Do this at a quiet
+  time.** Two mitigating facts: every pihole restart has *already* paid this cost (state was
+  ephemeral), and after this change `gravity.db` persists so later restarts should not. If it turns
+  out slow, backgrounding the gravity rebuild inside the hook would decouple readiness from it, at
+  the cost of a window where DNS resolves but does not block — not done here.
+- **Whether `pihole -g` finishes inside `apps-wave-0`'s 5 m `timeout`.** `Parallel` was chosen to
+  make this the max rather than the sum of two first boots, but the real duration is unmeasured (it
+  was stubbed during phase 1's testing). If the wave times out, the Kustomization reports failed even
+  though the pods may come up seconds later.
+- **The old PVC's deletion.** `apps-wave-0` has `prune: true`, so removing `pihole-config` from the
+  rendered output should delete it. That destroys nothing (it holds `lost+found`), but I have not
+  observed the prune, and I did not check whether Longhorn retains the backing volume afterwards.
+- **The transition overlap.** For a moment the pruned Deployment's pod and the new StatefulSet pods
+  may both match the Service selector. Both are configured identically from git so this should be
+  benign — unobserved.
+- **Longhorn capacity.** Two 2 Gi claims at 3 Longhorn replicas each is ~12 Gi of real storage where
+  there was ~6 Gi. Not checked against free space.
+- **Whether two independent query databases are acceptable to the operator.** Stats and query history
+  now split per replica, and the admin UI shows only the pod that answered. Adlists and password are
+  identical (phase 1), so blocking and auth are consistent; the *reporting* is not.
+- **The residual risk is unchanged and is the larger one.** Replicas address node failure. LAN DNS
+  still depends on the cluster — etcd quorum, Cilium, Longhorn, a bad reconcile — and only a resolver
+  outside the cluster carrying the same `address=` lines removes that. Considered and not chosen
+  (operator decision, 2026-10-04).
+- **`tolerationSeconds: 300` is untouched.** With two replicas the surviving one serves immediately, so
+  the 5-minute eviction delay no longer gates recovery — but it still delays the *replacement* replica,
+  and during that window there is no redundancy left.
@@ -306,19 +306,27 @@ node_baseline_dns_servers:
  # k3s_base_domain, below). It reaches the issuer through the NODE resolver, i.e. through
  # these PUBLIC upstreams — not through CoreDNS's split-horizon answer.
  #
-  # THIS USED TO BE BROKEN, AND THE 2026-09-04 SUBDOMAIN SPLIT FIXED IT. While prod was
-  # authored to serve dev's hostnames, a prod node resolving keycloak.home.keithjackson.dev
-  # publicly got DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN, so
-  # kubectl OIDC SSO on prod could not work at all. Now prod's issuer is
-  # keycloak.prod.home.keithjackson.dev, and the public record for that name is published by
-  # PROD's own external-dns pointing at PROD's Gateway VIP (192.168.2.20). The public answer
-  # and the split-horizon answer therefore AGREE, and no /etc/hosts pin is needed.
+  # THE PUBLIC ANSWER IS NOW THE WAN, SO THE ISSUER IS PINNED (2026-10-03). Since
+  # platform/gateway/gateway.yaml set an external-dns `target`, every public name is a CNAME
+  # to wan.home.keithjackson.dev = the home WAN IP. The SOHO router does NOT hairpin: from the
+  # LAN, WAN:443 answers with the router's own self-signed placeholder cert (measured from
+  # k3s-prod-2/-3 2026-10-03: `curl` exit 60, ssl_verify_result=18). The apiserver would then
+  # be unable to fetch its issuer's discovery/JWKS, so kubectl OIDC SSO would fail on the
+  # next k3s restart or Keycloak key rotation. node_baseline_hosts_pins (below) pins the
+  # issuer host to the Gateway VIP in /etc/hosts, which glibc consults before these
+  # resolvers. (History: before the 2026-09-04 subdomain split the public answer was DEV's
+  # VIP; after it, the public answer was prod's own VIP and no pin was needed.)
  #
-  # STILL TO VERIFY on prod bring-up (PROD-PROVISIONING.md §10): this holds only once prod's
-  # external-dns has actually published the record. Until then the issuer does not resolve
-  # from the nodes and kubectl SSO fails — the cert-based admin kubeconfig is the break-glass
-  # path, and it always works. Ordering, not a design gap.
+  # The cert-based admin kubeconfig never depends on any of this. It is the break-glass path.
 node_baseline_dns_search: []
+# /etc/hosts pins: node-process names that must reach the in-cluster Gateway directly rather
+# than via the public (WAN, non-hairpinned) answer. See "THE PUBLIC ANSWER IS NOW THE WAN" above.
+# [parity] the IP MUST equal GATEWAY_VIP in clusters/prod/kustomization.yaml (192.168.2.20),
+# and the name MUST follow k3s_oidc_issuer_url's host (roles/k3s_server/defaults/main.yml).
+node_baseline_hosts_pins:
+  - ip: "192.168.2.20"
+    names:
+      - "keycloak.{{ k3s_base_domain }}"

 # --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
 # PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/
@@ -242,3 +242,9 @@ node_baseline_dns_search:
 node_baseline_dns_options:
  - timeout:2
  - attempts:2
+# Static /etc/hosts entries (tags: dns, hosts), as a list of {ip, names}. EMPTY by default:
+# dev needs none. Prod pins its OIDC issuer host to the Gateway VIP because the public
+# answer for it is the non-hairpinned WAN IP (inventory/group_vars/k3s_prod/vars.yml).
+# Written as ONE marked block, so removing an entry here removes it from the node, and
+# an empty list removes the block entirely.
+node_baseline_hosts_pins: []
@@ -1090,3 +1090,22 @@
      {% endif %}
  when: node_baseline_dns_mechanism == 'resolv-conf-file'
  tags: [dns]
+
+# /etc/hosts pins — tag `dns` (and `hosts` on its own). Some node processes must reach a
+# public hostname at the in-cluster Gateway, not at its public answer. On prod, the
+# kube-apiserver's OIDC issuer resolves publicly to the WAN IP, which the router does not
+# hairpin. glibc reads /etc/hosts before any nameserver (nsswitch `files dns`), so a pin wins
+# regardless of which resolver mechanism above is in effect. One marked block: entries
+# dropped from node_baseline_hosts_pins disappear from the node, and an empty list
+# removes the block.
+- name: Pin node-process hostnames in /etc/hosts
+  ansible.builtin.blockinfile:
+    path: /etc/hosts
+    marker: "# {mark} node_baseline hosts pins (Ansible, tags: hosts)"
+    block: |
+      {% for pin in node_baseline_hosts_pins %}
+      {{ pin.ip }}	{{ pin.names | join(' ') }}
+      {% endfor %}
+    state: "{{ 'present' if node_baseline_hosts_pins | length > 0 else 'absent' }}"
+    backup: true
+  tags: [dns, hosts]
@@ -52,7 +52,7 @@ All four steps are complete and each was verified, not assumed:
 | 1. Runner registered | ✅ `incluster-dind` online. Root cause of its absence was a stale `/data/.runner` identity; clearing the PVC forced re-registration |
 | 2. Gate observed green on a real PR | ✅ `gate / gate (pull_request)` ~1m35s |
 | 3. `enable_status_check: true` | ✅ context `gate / gate (pull_request)` — that one ONLY |
-| 4. Collaborator + approvals | ✅ `jakman85` is owner; `required_approvals: 1` |
+| 4. Collaborator + approvals | ✅ `jakman85` is owner. `required_approvals` was raised to 1 here and **lowered back to 0 on 2026-10-01** — see §6, it was a lock-out |

 Direct push to `main` was **provoked and rejected** (`pre-receive hook declined`) as
 `gitea-admin`, a site admin — so push protection is not merely declared, it holds for the
@@ -124,6 +124,66 @@ the stage goes **green on a real PR**, and only then set `REQUIRE_GITEA_SETTINGS
 **Changing the declaration is how you change policy.** If the live state is what you now want,
 edit this JSON in the same commit — deliberately, in review — rather than letting the two drift.

+### 6. `allow-admin-self-merge.sh` — undoing the lock-out step 4 created (2026-10-01)
+
+Step 4 above raised `required_approvals` to 1. That was a **lock-out, not a protection**, and the
+note at the bottom of `require-gate.sh` had already said why: *Gitea will not let an author approve
+their own PR.* With `jakman85` the only collaborator, there was no account in existence that could
+approve a PR `jakman85` opened.
+
+Measured 2026-10-01 — **PR #35**: `gate / gate (pull_request)` **success in 1m41s**, reviews `[]`,
+`required_approvals: 1`, `block_admin_merge_override: true`. Unmergeable, permanently, with no
+action available to its author. The same was true of any PR fixing it.
+
+It was also never the documented design. `ci/approval-policy.sh`'s header and `ci/gate.sh:194`
+both already describe the model as *"`required_approvals` drops to 0 and the GATE decides"*, and
+`decisions/2026-09-30-flux-deploy-notification-inhibited.md` flagged the 1-vs-0 disagreement as
+unresolved and not that session's to settle. So this is a correction toward the stated intent, not
+a weakening away from it.
+
+    ./infra/gitea/allow-admin-self-merge.sh      # required_approvals -> 0 (read from the declaration)
+
+**What stays switched on, and why that is the point.** The other way to unblock an owner is
+`block_admin_merge_override: false`, and it would have been the wrong one — that re-opens exactly
+what §5 above closed, letting the one account that merges bypass the gate itself. This script
+touches the approval **count** only and *verifies* afterwards that `block_admin_merge_override` and
+`enable_status_check` are both still `true`, failing loudly if the PATCH disturbed either:
+
+| | before | after |
+|---|---|---|
+| `required_approvals` | 1 | **0** |
+| `block_admin_merge_override` | true | true (asserted) |
+| `enable_status_check` | true | true (asserted) |
+
+So a **red gate is still unmergeable by everyone, administrators included**. What changed is that
+a *green* one no longer needs a signature that cannot exist.
+
+**Security-relevant changes are still reviewed** — by `ci/approval-policy.sh` against
+`approval-policy.json` `sensitive_paths`, not by this field. An administrator waives that review
+only by posting `/approve-override <reason>` on the PR: reason mandatory, permission re-checked
+live, invalidated by a new commit, and recorded by account in the gate summary.
+
+> **Post the override as a PR *review* (type "Comment"), not in the plain comment box** — until
+> the CI token is re-minted. `SETTINGS_API_TOKEN` is `[read:repository, read:admin]`, and
+> `/issues/{n}/comments` needs Gitea's *issue* scope, so Gitea answers with an error **object**
+> instead of an array and the override is invisible. Measured on PR #38 (2026-10-02): a correct,
+> fresh, well-formed override in the comment box produced `approval-policy: FAIL` reading as "no
+> approving review" — a permission failure disguised as a missing one. `/pulls/{n}/reviews` is
+> under `read:repository` and worked in the same run. The stage now reads **both** sources, names
+> which one it used, and says plainly when the comments API is unreadable. `set-ci-token.sh`
+> requests `read:issue` as of 2026-10-02, so **re-run it** and the comment box works too — token
+> scopes cannot be widened in place. Note that the
+**agent account `gitea-admin` also reports permission `owner`**, so it can clear its own PRs that
+way; that is an accepted consequence of authorising by permission alone, and the route to
+narrowing it (an `override_admins` allowlist, already honoured by the code) is written down in
+`approval-policy.json`.
+
+**Sequencing.** `ci/gitea-settings.sh` compares live against the declaration *from the branch under
+test*, and `REQUIRE_GITEA_SETTINGS=true`. So merge any other open PRs **before** running this, then
+run it, then re-run the gate on the branch carrying the new declaration. Any PR left open across
+the change declares 1 against a live 0 and goes **red until rebased** — the detector working, not
+a fault. The script's header spells the order out.
+
 ## `conformance-trigger-kubeconfig.sh` — the narrow L2/L3 trigger credential

 Implements `docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 6 (= §5 step 7): mint a
@@ -204,6 +264,8 @@ it. The name matters — see the note under the behaviour table above.)
 2. **Watch the gate actually report on a real PR**, and note the context name it produced.
 3. **Only then set `enable_status_check: true`** with that exact context.
 4. Add the human reviewer as a repo collaborator, then raise `required_approvals` to 1.
+   — **Step 4 was REVERSED on 2026-10-01 (§6).** It assumed a reviewer distinct from the author
+   would exist. One never did, so this step produced a lock-out. Do not re-apply it.

 > **Do not do step 3 before step 1.** A required status check with no runner leaves every
 > PR permanently pending — including the PR that would fix it. That half is measured: PR #1
@@ -0,0 +1,105 @@
+#!/usr/bin/env bash
+# Let the sole administrator merge their own green PR: required_approvals 1 -> 0 on `main`.
+#
+# WHY. `required_approvals: 1` plus Gitea's refusal to let an author approve their own pull
+# request plus exactly ONE collaborator (`jakman85`, permission `owner`) is a lock-out, not a
+# policy. Every PR the operator opened was permanently unmergeable, including the PR that would
+# fix it. Measured 2026-10-01: PR #35 — `gate / gate (pull_request)` success in 1m41s, zero
+# reviews, `required_approvals: 1`, no account in existence able to satisfy it.
+#
+# The approvals field was also never what the design said it was. `ci/approval-policy.sh`'s
+# header and `ci/gate.sh:194` both already described the model as "required_approvals: 0 and the
+# GATE decides", and `require-gate.sh`'s own closing note named this exact failure and this exact
+# remedy. decisions/2026-09-30-flux-deploy-notification-inhibited.md flagged the 1-vs-0
+# disagreement as unresolved. This resolves it at 0.
+#
+# WHAT THIS DOES *NOT* DO, and the distinction is the whole point.
+#
+#   * It does NOT touch `block_admin_merge_override`, which stays TRUE. The required status check
+#     still binds ABSOLUTELY — a red gate is unmergeable by everyone, administrators included.
+#     Setting `block_admin_merge_override: false` would have been the other way to unblock an
+#     owner, and it would have re-opened precisely what block-admin-override.sh was written to
+#     close: PRs #5-#9 merged with zero approvals while the settings read as enforced.
+#   * It does NOT stop reviewing security-relevant changes. Those are gated by
+#     ci/approval-policy.sh against infra/gitea/approval-policy.json `sensitive_paths`, which an
+#     administrator can waive only with an explicit, attributed `/approve-override <reason>`
+#     comment on the PR.
+#
+# ORDER MATTERS — READ THIS BEFORE RUNNING IT.
+#
+# ci/gitea-settings.sh fails the merge gate on ANY divergence between the live rule and
+# infra/gitea/expected-branch-protection.json, and `gate.yml` sets REQUIRE_GITEA_SETTINGS=true.
+# The declaration in git now says 0. So between running this and merging that declaration, the
+# live value and the declared value must be compared from a branch that CARRIES the new
+# declaration — otherwise the stage goes red, correctly, for drift.
+#
+#   1. MERGE any other open PRs FIRST, while approvals still work for them (their branches
+#      declare 1, and a PR you did not author can still be approved normally). Skipping this
+#      only costs a rebase later; it does not break anything.
+#   2. Run this script.
+#   3. Re-run the gate on the branch that carries the new declaration. `gitea-settings` goes
+#      green because live 0 == declared 0.
+#   4. Merge that branch.
+#   5. Any PR still open from before step 2 now declares 1 against a live 0 and will be RED
+#      until it is rebased onto the new main. That is the detector doing its job, not a fault.
+#
+# Recoverable: `required_approvals` can be PATCHed back to 1 by an owner at any time. If you do,
+# change expected-branch-protection.json in the same breath or every PR goes red on drift.
+set -euo pipefail
+HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
+DECL="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/expected-branch-protection.json"
+
+command -v jq >/dev/null || { echo "jq is required" >&2; exit 1; }
+[ -f "$DECL" ] || { echo "declaration missing: ${DECL}" >&2; exit 1; }
+
+# Take the target from the DECLARATION, not from a literal here. If the two could disagree,
+# this script would be one more thing re-asserting a value nothing checks — which is the
+# failure mode the whole of infra/gitea/ exists to stop.
+WANT="$(jq -r '.protection.required_approvals' "$DECL")"
+[[ "$WANT" =~ ^[0-9]+$ ]] || { echo "declaration has no numeric required_approvals" >&2; exit 1; }
+
+creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
+U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
+[ -n "$U" ] && [ -n "$P" ] || { echo "no stored credential for $HOST" >&2; exit 1; }
+echo "authenticating as '${U}' (secret not shown)"
+api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
+B="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main"
+
+# Read the RULE OBJECT, never /branches/main: the branch view does not expose
+# block_admin_merge_override and showed all-green for hours while the rule was overridable.
+echo "==> before"
+api "$B" | jq -r '{required_approvals, block_admin_merge_override, enable_status_check, status_check_contexts}'
+
+echo "==> setting required_approvals = ${WANT} (declared in expected-branch-protection.json)"
+api -X PATCH -H 'Content-Type: application/json' \
+  -d "{\"required_approvals\": ${WANT}}" "$B" >/dev/null
+
+echo "==> after"
+AFTER_JSON="$(api "$B")"
+jq -r '{required_approvals, block_admin_merge_override, enable_status_check, status_check_contexts}' <<<"$AFTER_JSON"
+
+GOT="$(jq -r '.required_approvals' <<<"$AFTER_JSON")"
+ADMIN_BLOCK="$(jq -r '.block_admin_merge_override' <<<"$AFTER_JSON")"
+STATUS="$(jq -r '.enable_status_check' <<<"$AFTER_JSON")"
+
+fail=0
+[ "$GOT" = "$WANT" ] || { echo "PATCH did not take: required_approvals is ${GOT}, wanted ${WANT}" >&2; fail=1; }
+# These two are what keeps this change narrow. If the PATCH disturbed either, the gate has
+# stopped binding and that must be loud, not discovered later.
+[ "$ADMIN_BLOCK" = true ] || { echo "block_admin_merge_override is ${ADMIN_BLOCK}, MUST be true — the gate is now bypassable" >&2; fail=1; }
+[ "$STATUS" = true ]      || { echo "enable_status_check is ${STATUS}, MUST be true — the gate is no longer required" >&2; fail=1; }
+[ "$fail" -eq 0 ] || { echo "do NOT record this as applied." >&2; exit 1; }
+
+cat <<'NOTE'
+
+NOW PROVE IT -- the setting is not the evidence. Two things to see, not assume:
+
+  1. An APPROVAL IS NO LONGER NEEDED. On your own PR with a green gate, the merge button is
+     live with zero reviews on it.
+  2. THE GATE STILL BINDS. On a PR whose gate is RED or still pending, Gitea must REFUSE the
+     merge -- for you, as owner. If it lets you through, block_admin_merge_override is not
+     doing its job and this change has traded a lock-out for a bypass.
+
+Then confirm the detector agrees: run ci/gate.sh from a branch carrying the new
+expected-branch-protection.json and check that `gitea-settings` is green rather than DRIFT.
+NOTE
@@ -1,6 +1,6 @@
 {
  "_intent": "The CI/CD pipeline is the approver. A human approval is required ONLY for changes that alter what the pipeline itself verifies, or the security posture it rests on -- because those are the changes CI cannot honestly gate: a PR that weakens the gate would be approved by the weakened gate.",
-  "_rule": "If a PR touches any sensitive_paths glob, it needs an APPROVED review from someone who is not its author. Everything else merges on a green gate alone.",
+  "_rule": "If a PR touches any sensitive_paths glob, it needs an APPROVED review from someone who is not its author, OR an admin override (see .override). Everything else merges on a green gate alone.",
  "_on_failure_to_evaluate": "BLOCK. Unlike gitea-settings, a skip here is a bypass, not a gap.",
  "sensitive_paths": [
    "ci/**",
@@ -24,5 +24,18 @@
    "platform/kyverno/**": "the admission policy engine",
    ".sops.yaml, **/*.sops.yaml, **/rbac.yaml": "encryption recipients, secrets, and cluster authority",
    "CLAUDE.md": "the delivery contract agents follow"
+  },
+  "override": {
+    "_why": "This repository has ONE human. Gitea refuses to let an author approve their own pull request, so 'a non-author APPROVED review' is a requirement the sole administrator can never satisfy -- every sensitive-path PR they open was permanently unmergeable, including the PR that would fix it. Measured 2026-10-01: PR #35 green gate, zero reviews, unmergeable. An escape hatch that the administrator can actually reach is therefore REQUIRED for the repository to function; the design question is only whether it leaves a record. This one does.",
+    "enabled": true,
+    "comment_directive": "/approve-override",
+    "requires_reason": true,
+    "authorized_permissions": ["admin", "owner"],
+    "invalidated_by_new_commits": true,
+    "_how": "Post the directive at the START of a line, followed by a reason: '/approve-override <why>'. ci/approval-policy.sh resolves the poster's LIVE repo permission via /collaborators/{login}/permission and accepts the override only if it is in authorized_permissions. The overrider, the source, the timestamp and the reason are printed in the gate log and the job summary.",
+    "_where_to_post_it": "EITHER a PR REVIEW body (type 'Comment') OR the ordinary PR comment box. Prefer the REVIEW: measured 2026-10-02 on PR #38, a well-formed override in the comment box was INVISIBLE to CI, because /issues/{n}/comments sits behind Gitea's *issue* token scope and SETTINGS_API_TOKEN is minted [read:repository, read:admin]. Gitea returned an error object rather than an array, and the stage reported 'no approving review' -- a permission problem wearing the costume of a missing override. /pulls/{n}/reviews is under read:repository and was readable in the same run, so a review body is the route that works on today's token. infra/gitea/set-ci-token.sh now also requests read:issue, which makes the comment box work -- but only after that script is RE-RUN, since an existing token's scopes cannot be widened in place. The stage accepts both sources and names which one it used, so neither route is load-bearing alone.",
+    "_self_override_is_allowed": "The author may override their own PR if they hold an authorized permission. That is the entire point -- with one human there is no one else -- but it means the 'a change to the gate cannot be gated by the gate' protection is, for an administrator, advisory rather than structural. It is traded for an auditable record: the waiver is an explicit act, attributed, with a stated reason, in the PR's comment history and the gate output.",
+    "_known_consequence_agent_accounts": "ACCEPTED, EYES OPEN (operator decision, 2026-10-01). Authorisation is by live permission ALONE, with no allowlist of human accounts. The agent account `gitea-admin` reports permission `owner` on this repo (measured 2026-10-01), so an AGENT can clear the sensitive-path review on its own PR by posting the directive. Agents author most PRs here, so this is the broad case, not the edge case. To narrow it later without changing any logic, add an `override_admins` allowlist here and require membership in addition to the permission check -- ci/approval-policy.sh reads both from this file.",
+    "_staleness": "An override is dropped unless its comment is NEWER than the PR head commit, mirroring Gitea's own dismiss_stale_approvals. An override earned at commit A therefore does not silently cover commit B; push again and you must override again."
  }
 }
@@ -1,6 +1,9 @@
 {
  "_comment": "DECLARED desired state of Gitea branch protection on main. ci/gitea-settings.sh fails the merge gate when the LIVE Gitea diverges from this. Gitea settings live in Gitea database - invisible to Flux, unreadable by every other check - so this file plus that check are the only drift detection that exists for them.",
  "_generated": "2026-09-29 from the live rule, after the session that found protection OFF and zero runners for a month",
+  "_amended": "2026-10-01: required_approvals 1 -> 0. See _why_required_approvals_is_zero.",
+  "_why_required_approvals_is_zero": "The GATE is the approver (operator intent, 2026-09-29). required_approvals: 1 was never consistent with that intent: ci/approval-policy.sh's header and ci/gate.sh:194 both already described the model as 'required_approvals: 0 and the GATE decides', and infra/gitea/require-gate.sh's own closing note named the consequence of 1 -- 'Gitea will not let an author approve their own PR'. With exactly ONE collaborator (jakman85, owner), that made every PR the operator opened permanently unmergeable. Measured 2026-10-01: PR #35, gate green in 1m41s, zero reviews, no path to a merge. decisions/2026-09-30-flux-deploy-notification-inhibited.md had already flagged the 1-vs-0 disagreement as unresolved; this settles it at 0.",
+  "_what_still_holds_at_zero": "This drops the APPROVAL COUNT only. block_admin_merge_override stays TRUE and enable_status_check stays TRUE, so the required gate context still binds absolutely -- nobody, administrator included, can merge a red gate. That distinction is the whole reason this is the chosen fix rather than block_admin_merge_override: false, which would have handed admins a bypass over the gate itself and re-opened what infra/gitea/block-admin-override.sh was written to close (PRs #5-#9, merged with zero approvals). Architecturally and security-relevant paths are still reviewed, by ci/approval-policy.sh rather than by this field.",
  "branch": "main",
  "runners_online_min": 1,
  "protection": {
@@ -11,7 +14,7 @@
    "status_check_contexts": [
      "gate / gate (pull_request)"
    ],
-    "required_approvals": 1,
+    "required_approvals": 0,
    "enable_approvals_whitelist": false,
    "block_admin_merge_override": true,
    "block_on_rejected_reviews": true,
@@ -34,7 +34,7 @@ jq -n '{
  enable_push: false,
  enable_status_check: true,
  status_check_contexts: ["gate / gate (pull_request)"],
-  required_approvals: 1,
+  required_approvals: 0,
  block_on_rejected_reviews: true,
  dismiss_stale_approvals: true,
  block_on_outdated_branch: false,
@@ -48,11 +48,21 @@ api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \

 cat <<'NOTE'

-NOTE ON required_approvals: 1 -- Gitea will not let an author approve their own PR.
-  * Agent-opened PRs (author gitea-admin) are approved by jakman85. Fine.
-  * PRs YOU open as jakman85 need gitea-admin to approve. If that is awkward, either open
-    agent work as gitea-admin and review as jakman85, or set required_approvals back to 0
-    and rely on the status check alone.
+NOTE ON required_approvals: 0 -- CHANGED FROM 1 ON 2026-10-01, and this script is the reason
+the change had to land here too: re-running it with 1 would silently re-impose the lock-out.
+
+The note that used to sit here called the problem and its own fix: "Gitea will not let an
+author approve their own PR ... if that is awkward ... set required_approvals back to 0 and
+rely on the status check alone." It was awkward. With jakman85 the ONLY collaborator, every PR
+the operator opened was unmergeable -- measured on PR #35, gate green in 1m41s, zero reviews,
+nothing that could ever satisfy the rule. So: 0, and the status check alone, as written.
+
+What did NOT change, deliberately:
+  * block_admin_merge_override stays TRUE. This drops the approval COUNT; it does not hand
+    anyone a bypass over the gate. A red gate is still unmergeable by everybody.
+  * Architecturally/security-relevant paths are still human-reviewed -- by ci/approval-policy.sh
+    (sensitive_paths in infra/gitea/approval-policy.json), which an administrator can waive
+    only by posting an explicit, attributed `/approve-override <reason>` comment.

 Then prove it, rather than trusting the API:
  git commit --allow-empty -m probe && git push origin main   # EXPECT: rejected
@@ -5,9 +5,23 @@
 # The token value is NEVER printed. It goes from the token API into the secrets API and
 # nowhere else -- not to stdout, not to a file, not into an agent transcript.
 #
-# Scope: `read:repository` + `read:admin` are what the check needs (the rule object, and the
-# runner list). Gitea token scopes are coarse; grant the least that works on your version and
-# confirm the check goes green rather than assuming the scope name.
+# Scope: `read:repository` + `read:admin` are what ci/gitea-settings.sh needs (the rule object,
+# and the runner list). `read:issue` is for ci/approval-policy.sh, and it was MISSING — see below.
+# Gitea token scopes are coarse; grant the least that works on your version and confirm the check
+# goes green rather than assuming the scope name.
+#
+# WHY read:issue IS HERE (added 2026-10-02, measured on PR #38). ci/approval-policy.sh reads the
+# PR's comments to find an admin `/approve-override`. /issues/{n}/comments sits behind the *issue*
+# scope, which this script did not request — so Gitea answered with an error OBJECT instead of an
+# array, and a perfectly well-formed override posted in the PR comment box was invisible to CI.
+# The failure did not look like a permission problem from the outside; it looked like the override
+# had not been posted. `/pulls/{n}/files` and `/pulls/{n}/reviews` are under read:repository, which
+# is why every other part of the same stage worked and this one did not.
+#
+# The stage now accepts an override from a REVIEW body as well, so it still functions on a token
+# WITHOUT read:issue. This scope is what makes the ordinary comment box work too — the thing
+# anybody would reach for first. **Re-run this script to pick it up; an existing token's scopes
+# cannot be widened in place.**
 set -euo pipefail
 HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
 # NOT GITEA_* — Gitea reserves that prefix for secret names and returns HTTP 400
@@ -21,7 +35,7 @@ basic() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }

 echo "==> minting token '${NAME}' as ${U}"
 TOKEN="$(basic -X POST -H 'Content-Type: application/json' \
-  -d "{\"name\":\"${NAME}\",\"scopes\":[\"read:repository\",\"read:admin\"]}" \
+  -d "{\"name\":\"${NAME}\",\"scopes\":[\"read:repository\",\"read:admin\",\"read:issue\"]}" \
  "https://${HOST}/api/v1/users/${U}/tokens" | jq -r '.sha1 // empty')"
 [ -n "$TOKEN" ] || { echo "token creation failed (check scopes for your Gitea version)" >&2; exit 1; }
 echo "==> got a ${#TOKEN}-character token (value not shown)"
@@ -20,6 +20,30 @@ A wildcard `*.keithjackson.dev` matches only one label deep, so `<svc>.home.keit
 its own record. **`internal` services need NO public record** — PiHole resolves them to the internal
 Gateway VIP (split-horizon). ExternalDNS manages **`public`** services' records only.

+## Public target + in-cluster DDNS (2026-10-03)
+
+Gateway-published records are **CNAMEs to `wan.home.keithjackson.dev`**, not the Gateway VIP:
+
+```
+plex.home.keithjackson.dev  CNAME  wan.home.keithjackson.dev   <- external-dns (Gateway `target` annotation)
+wan.home.keithjackson.dev   A      <current WAN IP>            <- cloudflare-ddns (ddns.yaml), every 5m
+```
+
+- The `target` annotation is on the **shared Gateway** (`platform/gateway/gateway.yaml`), so it
+  applies to every route that publishes at all. `internal` routes (`controller: none`) still publish
+  nothing.
+- `ddns.yaml` runs `favonia/cloudflare-ddns` in this namespace with the same `cloudflare-api-token`.
+  It detects the WAN IP through Cloudflare's trace endpoint, so the pod **must not** carry
+  `egress-class: vpn`.
+- LAN and pods resolve `*.home.keithjackson.dev` to the Gateway VIP via PiHole and CoreDNS
+  (split-horizon) and never use the public path.
+- **Nodes resolve through public DNS** (`node_baseline_dns_servers`), so from a node, the apiserver's
+  OIDC issuer `keycloak.home.keithjackson.dev` resolves to the WAN IP. kubectl SSO therefore relies on
+  the router hairpinning (NAT reflection) WAN:443 back to the Gateway.
+
+Check: `dig +short @1.1.1.1 plex.home.keithjackson.dev` → `wan.home.keithjackson.dev.` then the WAN IP;
+`kubectl -n external-dns logs deploy/cloudflare-ddns`.
+
 ## Secrets

 Cloudflare API token referenced **by name** (`cloudflare-api-token`, shared with cert-manager),
@@ -0,0 +1,87 @@
+---
+# In-cluster dynamic DNS for the home WAN (2026-10-03).
+#
+# Keeps ONE record, `wan.${BASE_DOMAIN}` (A, IPv4 only, not proxied), pointed at the home WAN's
+# current public IP. Every public name external-dns publishes from the shared Gateway is a CNAME to
+# it (platform/gateway/gateway.yaml `external-dns.alpha.kubernetes.io/target`), so a WAN IP change
+# moves all of them through this one record, with no Git change and no external-dns involvement.
+#
+# OWNERSHIP SPLIT WITH EXTERNAL-DNS. This record carries no external-dns TXT registry entry, so
+# external-dns (policy: sync, owner `prod`) never treats it as its own and never deletes it.
+# Conversely this updater manages only the names in DOMAINS and never touches external-dns's records.
+#
+# IP DETECTION. Default `cloudflare.trace`: an HTTPS request to Cloudflare's trace endpoint, which
+# reports the address the request arrived from. Correct here only because this pod is NOT in the
+# `vpn` egress class — the CiliumEgressGatewayPolicy (platform/vpn-egress) selects pods by
+# `homelab.keithjackson.dev/egress-class: vpn` alone, cluster-wide. Labelling this pod `vpn` would
+# publish the VPN exit IP instead of the WAN IP.
+#
+# TOKEN. Reuses this namespace's `cloudflare-api-token` (same Zone:DNS:Edit token as external-dns
+# and cert-manager) — no new secret.
+#
+# TIER NOTE. This lives in the gating `platform` Flux Kustomization (wait: true) next to
+# external-dns. If it ever crashloops, `platform` goes not-Ready and apps stop reconciling (no
+# outage). Kept here rather than in its own non-gating Kustomization to avoid a clusters/** change.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: cloudflare-ddns
+  namespace: external-dns
+  labels:
+    app.kubernetes.io/name: cloudflare-ddns
+spec:
+  replicas: 1                 # exactly one writer for the record
+  revisionHistoryLimit: 3
+  strategy:
+    type: Recreate            # never two updaters at once
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: cloudflare-ddns
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: cloudflare-ddns
+    spec:
+      automountServiceAccountToken: false
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: cloudflare-ddns
+          image: favonia/cloudflare-ddns:1.17.1
+          imagePullPolicy: IfNotPresent
+          env:
+            - name: CLOUDFLARE_API_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: cloudflare-api-token
+                  key: api-token
+            - name: DOMAINS
+              value: "wan.${BASE_DOMAIN}"
+            - name: IP6_PROVIDER
+              value: "none"         # IPv4 WAN only; leaves any AAAA alone
+            - name: PROXIED
+              value: "false"        # DNS-only: Plex/Gitea/etc. terminate TLS on the Gateway
+            - name: UPDATE_CRON
+              value: "@every 5m"
+            - name: DELETE_ON_STOP
+              value: "false"        # a pod restart must not blank the record every CNAME depends on
+            - name: RECORD_COMMENT
+              value: "managed by k3s cloudflare-ddns (platform/externaldns/ddns.yaml)"
+            - name: EMOJI
+              value: "false"
+          resources:
+            requests:
+              cpu: 5m
+              memory: 16Mi
+            limits:
+              memory: 64Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop:
+                - ALL
@@ -81,38 +81,29 @@ spec:
    # what this instance can see or own; per-resource exclusion only ever REMOVES an object from
    # this instance's view, so it cannot make prod capable of deleting a record it does not own.
    # ---------------------------------------------------------------------------------------------
-    # THE PUBLIC-TARGET GAP — STILL OPEN, AND NOT FIXABLE HERE ON v0.21.0 (operator decision)
+    # THE PUBLIC TARGET — DECIDED 2026-10-03 (operator): every Gateway-published name goes public
    #
-    # The five `exposure: public` services (plex, gitea, seerr, keycloak, nextcloud) publish records
-    # that point at 192.168.2.20 — the Gateway VIP, an unroutable RFC1918 address — so they are not
-    # actually reachable from the internet either. Their records are LEFT EXACTLY AS THEY ARE by
-    # this change, for two independent reasons:
+    # Until 2026-10-03 every gateway-sourced record answered with the Gateway VIP (192.168.2.20,
+    # RFC1918), so nothing published here was reachable from the internet. Two facts constrained the
+    # fix, and both still hold:
+    #   * `external-dns.alpha.kubernetes.io/target` is read in v0.21.0's gateway-httproute source ONLY
+    #     from the GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
+    #     gw.Annotations)`) and from a ListenerSet listener — NEVER from an HTTPRoute. A target on the
+    #     shared Gateway therefore moves EVERY gateway-sourced record at once.
+    #   * The home WAN IP is residential and can change, so a literal IP in Git would rot.
    #
-    #   1. THE VALUE IS NOT ESTABLISHED. The one public IP this repo knows is 136.51.115.252
-    #      (clusters/prod/kustomization.yaml, where the zone's `*.keithjackson.dev` wildcard also
-    #      points). It belongs to the LEGACY non-k3s stack and currently serves a placeholder
-    #      certificate. Whether prod's public services belong behind it, behind a new forward, or
-    #      are not meant to be public at all is an operator decision. Breaking five services' DNS
-    #      on a guess is worse than leaving a known-wrong-but-stable state.
-    #   2. THE MECHANISM CANNOT BE SCOPED TO THEM. `external-dns.alpha.kubernetes.io/target` is a
-    #      real override, but in v0.21.0's gateway-httproute source it is read ONLY from the
-    #      GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
-    #      gw.Annotations)`) and from a ListenerSet listener — NEVER from the HTTPRoute. So there is
-    #      no per-service target override while every public service shares the `shared` Gateway.
-    #      Annotating the Gateway would move EVERY gateway-sourced record, which after this change
-    #      still includes auth, crafty, grafana, headlamp and the apex — i.e. it would publish a
-    #      routable address for admin UIs. That is strictly worse than the status quo.
-    #
-    # SO THE DECISION THE OPERATOR OWNS IS TWO-PART, and no knob is added here until the first half
-    # is answered, because a knob that cannot take effect invites someone to set it and expect one:
-    #   (a) WHAT address should the five public names answer with — 136.51.115.252 as-is, a new
-    #       port-forward/WAN address, or nothing (drop `exposure: public` to `internal` instead)?
-    #   (b) HOW should they be separated from the non-public names that share the Gateway? Options,
-    #       roughly in ascending cost: a SECOND Gateway (or a ListenerSet) carrying the target
-    #       annotation, with only the public services' routes attached; or publishing the five names
-    #       from the `crd` source (DNSEndpoint objects the renderer emits for `public` services),
-    #       which restores per-service targeting without a second Gateway.
-    # Only (b) is a manifest change; (a) is a fact about the home WAN that has to be checked.
+    # The operator's answers: (a) yes, ALL of them — the five `exposure: public` apps AND the admin
+    # UIs (auth, crafty, grafana, headlamp, the apex), which sit behind Keycloak / oauth2-proxy;
+    # (b) the address is kept current IN-CLUSTER. Implementation:
+    #   * platform/gateway/gateway.yaml sets `target: wan.${BASE_DOMAIN}` on the shared Gateway, so
+    #     each published name becomes a CNAME to that one hostname;
+    #   * platform/externaldns/ddns.yaml (favonia/cloudflare-ddns) keeps `wan.${BASE_DOMAIN}` an A
+    #     record at the current WAN IP. That record has no external-dns TXT registry entry, so this
+    #     instance never owns or garbage-collects it.
+    # Internal services are unchanged: their routes carry `controller: none` (above), which is
+    # checked BEFORE the target is read, so they still publish nothing. The `service` source (the L4
+    # LoadBalancers crafty-mc, unifi) does not read the Gateway's annotation and still publishes its
+    # LB VIP. Whether those should go public is a separate, still-open question.
    # ---------------------------------------------------------------------------------------------
    # CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below.
    #
@@ -8,3 +8,5 @@ resources:
  # SOPS-encrypted Cloudflare token (env CF_API_TOKEN) — SAME token as cert-manager, second copy in
  # the external-dns namespace. Flux decrypts at reconcile.
  - cloudflare-api-token.sops.yaml
+  # DDNS for `wan.${BASE_DOMAIN}` — the CNAME target every Gateway-published record points at.
+  - ddns.yaml
@@ -7,6 +7,23 @@ kind: Gateway
 metadata:
  name: shared
  namespace: gateway
+  annotations:
+    # PUBLIC DNS TARGET (2026-10-03, operator decision). Without this, external-dns publishes every
+    # route on this Gateway with the Gateway's OWN address (${GATEWAY_VIP}, RFC1918), which is
+    # unreachable from the internet. With it, every published `<svc>.${BASE_DOMAIN}` (and the apex)
+    # becomes a CNAME to `wan.${BASE_DOMAIN}`, an A record that platform/externaldns/ddns.yaml keeps
+    # pointed at the home WAN's current public IP. The WAN forwards 80/443 to ${GATEWAY_VIP}.
+    #
+    # Scope is EVERY route on this Gateway that publishes at all — the operator chose to make the
+    # admin UIs (auth, grafana, headlamp, crafty, apex) publicly resolvable alongside the five
+    # `exposure: public` apps. `internal` services stay unpublished via their
+    # `external-dns.alpha.kubernetes.io/controller: none` route annotation, which external-dns checks
+    # BEFORE it reads this target. external-dns v0.21.0 reads `target` from the Gateway only, never
+    # from an HTTPRoute — see platform/externaldns/helmrelease.yaml.
+    #
+    # LAN and in-cluster clients are unaffected: PiHole and CoreDNS answer ${BASE_DOMAIN} with
+    # ${GATEWAY_VIP} directly (split-horizon), so they never take this public path.
+    external-dns.alpha.kubernetes.io/target: "wan.${BASE_DOMAIN}"
 spec:
  gatewayClassName: cilium
  listeners:
@@ -153,14 +153,18 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
  plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
  token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
  Includes why a changed Secret does not restart anything, and why an expired token is harmless.
- **`plex-quicksync`** — hardware transcoding on the host iGPU. GitOps puts `/dev/dri` in the pod
-  (`hardware.gpu: intel-quicksync`); **a human still has to tick Settings → Transcoder → "Use hardware
-  acceleration when available"**, and no environment variable can do it (the image supports no
-  `PLEX_PREFERENCE*`, verified). The page names the real preference key (`HardwareAcceleratedCodecs`,
-  read out of the binary), says which widely-cited key **does not exist**, and is explicit that the
-  **only** proof is `Transcode (hw)` on the Plex dashboard — Plex renders no ServiceMonitor, so no
-  alert could ever fire. Also: the device-cgroup failure to expect first and its device-plugin
-  fallback, and why a `:latest` pull can silently break it.
+- **`plex-quicksync`** — hardware transcoding on the host iGPU. **Confirmed working 2026-10-02**
+  (VAAPI decode *and* encode, zero-copy). GitOps requests `gpu.intel.com/i915` on the pod
+  (`hardware.gpu: intel-quicksync`, satisfied by the `intel-gpu-plugin` DaemonSet — a `/dev/dri`
+  hostPath was measured not to work: device-cgroup `EPERM`); **a human still has to tick Settings →
+  Transcoder → "Use hardware acceleration when available"**, and no environment variable can do it
+  (the image supports no `PLEX_PREFERENCE*`, verified). **The proof is `Plex Media Server.log`'s
+  `final decoder: / final encoder:` line, not the Plex dashboard** — the dashboard showed nothing
+  while hardware transcoding was working, so its silence proves nothing; the page carries the
+  redacting grep (`X-Plex-Token` lives in that log), the WRONG log file to avoid, and two more
+  documented false leads (an empty `/proc/*/fd` scan, and a missing `HardwareAcceleratedCodecs` key).
+  Plex renders no ServiceMonitor, so no alert could ever fire. Also: **`Pending` stalls the whole app
+  tier** and its one-field escape hatch, and why a `:latest` pull can silently break it.
 - **`oauth2-proxy-session`** — a gated UI loads but its saves quietly do nothing, and you are never
  re-prompted to log in. Tells the three stacked auth layers apart from the sidecar's own access log
  (the four lines that matter, including the upstream `-` that means the proxy answered a request the
@@ -1,14 +1,33 @@
 # plex-quicksync — making Plex actually use Intel Quick Sync, and proving it did

+**Status: hardware transcoding is CONFIRMED WORKING on prod as of 2026-10-02** — VAAPI for both
+decode *and* encode, zero-copy, measured in Plex's own server log (see "How to verify"). The manual
+toggle has been ticked. This page is therefore now mostly a **verification and regression**
+procedure, and its old verification section was **wrong** — see the correction at the top of
+"How to verify".
 **Alert:** none, and none is possible — see "How to verify" below. This is an **operator procedure**,
 not an incident page. It is indexed here because the feature it completes is **inert until a human
 clicks one button**, and a switched-off feature that looks shipped is exactly what this page exists
 to prevent.
 **Files:** `catalog/plex.yaml` (`hardware.gpu: intel-quicksync`), `apps/wave-0/plex.yaml` (the
-rendered golden — the `dri` hostPath volume and the `/dev/dri` mount), `catalog/render.py` (the
-mechanism), `schemas/renderer-contract.md` `### hardware` (the contract).
+rendered golden — the `resources.limits: gpu.intel.com/i915: 1` block), `catalog/render.py` (the
+mechanism), `schemas/renderer-contract.md` `### hardware` (the contract),
+`platform/intel-gpu-plugin/` (the DaemonSet that advertises the resource).
 **Related:** `runbooks/plex-claim.md` — the other Plex procedure with a manual last mile.

+## STOP HERE IF PLEX IS `Pending` / `Unschedulable`
+
+That is this feature's **one serious failure mode** and it **stalls the whole app tier**
+(`apps-wave-0` is `wait: true`; `apps-wave-1` `dependsOn` it). Since 2026-10-01 Plex requests the
+extended resource `gpu.intel.com/i915: 1`, which is a **hard scheduling constraint** — if the
+`intel-gpu-plugin` DaemonSet is not advertising it on at least one schedulable node, Plex does not
+start at all. Go to **"Plex will not schedule"** below; it is the first troubleshooting section on
+purpose.
+
+**The escape hatch, in one sentence:** delete the two `hardware:` lines from `catalog/plex.yaml`,
+re-render, and merge — Plex returns to software transcoding and the tier unblocks. Exact commands
+are in **"Plex will not schedule" → remedy B**.
+
 ## What this means here

 Plex transcodes in **software** by default. On the UHD Graphics 630 iGPU of an i7-9700T, Quick Sync
@@ -18,8 +37,8 @@ is in git**:

 | Half | Who does it | State |
 |---|---|---|
-| The render device is inside the container at `/dev/dri` | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders a `dri` hostPath volume (`/dev/dri`, `type: DirectoryOrCreate`) and a read-write `/dev/dri` volumeMount on the app container. Flux applies it. | Done by merging that change. |
-| Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **NOT done.** `HardwareAcceleratedCodecs` was **absent** from `Preferences.xml` when this page was written — the toggle has never been on. |
+| The render device is inside the container **and openable** | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders `resources.limits: {gpu.intel.com/i915: 1}` on the app container. The `intel-gpu-plugin` DaemonSet (`platform/intel-gpu-plugin/`) injects the device **and the matching device-cgroup allow rule**. Flux applies it. | Done by merging that change. |
+| Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **Done, 2026-10-02**, and confirmed from the **log** rather than from a preference key. `HardwareAcceleratedCodecs` is **still absent** from `Preferences.xml` with hardware transcoding working, so its absence proves nothing — see first-command #4. |

 **There is no GitOps path to the second half, and that is verified rather than assumed.** A bounded
 grep across the running image's entire init surface — `/etc/cont-init.d`, `/etc/services.d`,
@@ -30,10 +49,44 @@ Every `setPref` the image performs is one of six hardcoded keys (`MachineIdentif
 and no `postBuild` substitution can set it. So do not go looking for the clever way to automate this;
 there isn't one, and the search has already been done.

-What the device mount does **not** do, so nobody reads more into a green gate than is there: it does
-not enable hardware transcoding, it does not prove the device can be *opened* from inside a
-non-privileged pod (see "Likely causes" #2), and it renders no `nodeSelector` — Plex can land on any
-of the three nodes and all three carry the device.
+### Why a resource request and not a `/dev/dri` hostPath
+
+Because the hostPath was **tried and measured not to work**, and knowing that is the only thing that
+will stop someone "simplifying" it back. Merged 2026-09-30, measured in the live pod on k3s-prod-2 on
+**2026-10-01**: the device node was plainly **visible** in the container
+(`crw-rw---- root video2 226,128 renderD128`) and **could not be opened** —
+
+| attempt | result |
+|---|---|
+| `uid 0`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+| `uid 1000`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+
+`EPERM` is the **device cgroup** refusing, not `EACCES` from file permissions. The two look identical
+from the outside and are not the same problem. Proven three ways: root on the **host** opened the
+same device `rc=0`; the node is unified `cgroup2fs`; and the container's runc `config.json` read
+`linux.resources.devices: [{allow: False, access: 'rwm'}]` with `linux.devices: None` — **one
+deny-all rule, no allow entry for major 226**. A hostPath is a *filesystem* operation: it binds the
+device node in and adds **no cgroup allow rule**, because Kubernetes cannot infer one from a volume.
+
+Two non-workarounds, so nobody spends an evening on them: **`uid 0` failed too**, so running as root
+does not bypass the cgroup gate; and **`privileged: true` is not available** — Kyverno
+`restrict-host-namespaces` is **Enforce** on app namespaces and rejects it.
+
+The device plugin is what produces the allow rule — in the v0.37.0 source,
+`createDeviceSpecsFromDrmFiles` emits `DeviceSpec{…, Permissions: "rw"}`, which the kubelet turns
+into both the device injection and the matching cgroup allow.
+
+**The failure mode that trade bought, stated plainly.** Before, a missing device meant the pod ran
+and **silently software-transcoded** — invisible, but harmless to everything else. Now, a missing
+*resource* means Plex is **Unschedulable** and the app tier stalls. That is strictly worse in blast
+radius and it is **unavoidable**: an extended resource is a hard scheduling constraint, and it is
+also the only thing that produces the cgroup allow rule. There is no graceful-degradation variant.
+The one improvement is that the loss is no longer silent.
+
+What this does **not** do, so nobody reads more into a green gate than is there: it does not enable
+hardware transcoding (that is the manual toggle below — done 2026-10-02), a green gate only proves
+the **manifest renders**, and it writes no `nodeSelector` — the resource itself constrains Plex to nodes the plugin
+advertises, which is the scheduler's doing and not a selector in this repo.

 ## Running the commands in this runbook

@@ -51,15 +104,46 @@ read of that file below greps a single named key — do not widen one.

 ## First commands

-**1. Is the device mounted into the pod at all?** (Proves Flux applied the change, not that it works.)
+**0. Is Plex running at all?** Ask this first now — the current mechanism can stop it starting.
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide'
+```
+
+`Pending` ⇒ go straight to **"Plex will not schedule"**. Running ⇒ carry on.
+
+**1. Do the nodes advertise the resource, and is the plugin Ready?** This replaces the old "is the
+hostPath mounted" check, and it is now the first thing that can be wrong.
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod get nodes -o json \
+   | jq -r ".items[] | \"\(.metadata.name) i915=\(.status.allocatable[\"gpu.intel.com/i915\"] // \"ABSENT\")\""'
+```
+
+Expect `i915=1` on each of the three prod nodes (measured 2026-10-01: all three). Any `ABSENT` ⇒ that
+node cannot run Plex. Then the DaemonSet behind it:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n intel-gpu-plugin get daemonset intel-gpu-plugin'
+```
+
+Expect **3 desired / 3 ready / 3 available** (measured 2026-10-01). Anything less is the cause, not a
+symptom — fix it there, in `platform/intel-gpu-plugin/`, not here.
+
+**2. Did Flux apply the current manifest?** `gpu.intel.com/i915` on the app container means the
+GitOps half landed:

 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
-     -o jsonpath="{.items[0].spec.containers[0].volumeMounts[*].mountPath}"; echo'
+     -o jsonpath="{.items[0].spec.containers[0].resources.limits}"; echo'
 ```

-`/dev/dri` in that list ⇒ the GitOps half landed. Absent ⇒ the running pod predates the merge; compare
+A `/dev/dri` **volumeMount** instead ⇒ the running pod predates the 2026-10-01 merge and is on the
+mechanism that does not work. Compare

 ```
 infra/devcontainer/dev.sh run bash -c \
@@ -69,46 +153,52 @@ infra/devcontainer/dev.sh run bash -c \

 against the merge commit before blaming anything else.

-**2. Is the device actually visible inside the container?**
+**3. Is the device visible inside the container, and can Plex *open* it?** This is the decisive test
+of the whole mechanism, and **it must be run as uid 1000**, because that is what Plex runs as.

 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- ls -l /dev/dri'
 ```

-Expect `renderD128` (usually `card0` too), mode `crw-rw----`, group `render`. An **empty directory**
-here is the `DirectoryOrCreate` degradation: the kubelet created an empty `/dev/dri` because the node
-had none. That is the designed-safe outcome — the app tier still reconciles — but it means software
-transcoding, silently. Find which node the pod is on and ask why that node lost its iGPU:
-
-```
-infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex -o wide'
-```
-
-**3. Can Plex *open* it?** This is the question the manifest cannot answer — see "Likely causes" #2.
+Expect `renderD128`, mode `crw-rw----`. The plugin injects it; there is no longer any
+`DirectoryOrCreate` empty-directory case — if the resource were missing the pod would not be running
+to exec into. Then the open, **as the app user**:

 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
-     bash -c "exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED"'
+     s6-setuidgid plex sh -c "exec 3<>/dev/dri/renderD128 && echo RW-OPEN-OK || echo RW-OPEN-DENIED"'
 ```

-`RW-OPEN-DENIED` (or `Operation not permitted` / `Permission denied`) with the device plainly visible
-in step 2 is the **device-cgroup** failure, not a file-permission failure. The two look identical from
-the outside and are not the same problem.
+`RW-OPEN-OK` **as uid 1000** is the pass condition. **Passing as `uid 0` is weaker evidence and must
+not be accepted as proof** — Plex runs as 1000, and the whole point of the 2026-10-01 change was a
+cgroup rule that uid 0 could not buy its way past either. `RW-OPEN-DENIED` with the device plainly
+visible is the **device-cgroup** failure — i.e. the plugin's allow rule did not arrive; see "Likely
+causes" #2.

-**4. Is the toggle on?** Named key only — never the whole file:
+**4. Is the toggle on?** **You cannot answer this from `Preferences.xml` alone.** That mistake is on
+this page's record: it told readers to treat a missing `HardwareAcceleratedCodecs` as "never
+enabled", and on 2026-10-02 that key was **absent from all 27 attributes** while hardware
+transcoding was demonstrably working. Named keys only — never the whole file, it holds
+`PlexOnlineToken`:

 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
-     grep -o "HardwareAcceleratedCodecs=\"[01]\"" \
-     "/config/Library/Application Support/Plex Media Server/Preferences.xml" || echo ABSENT'
+     grep -o -e "HardwareAcceleratedCodecs=\"[01]\"" -e "HardwareDevicePath=\"[^\"]*\"" \
+     "/config/Library/Application Support/Plex Media Server/Preferences.xml" || echo NEITHER-KEY'
 ```

-`ABSENT` ⇒ never enabled (the state this page was written in). `="0"` ⇒ explicitly off. `="1"` ⇒ on,
-and the remaining question is whether it *works* — go to "How to verify".
+| Reading | What it actually means |
+|---|---|
+| `HardwareDevicePath="…"` **present** | **Positive signal.** Plex writes this when hardware acceleration is configured; it was present on 2026-10-02 alongside a working VAAPI pipeline. |
+| `HardwareAcceleratedCodecs` **absent** | **Nothing. Neither on nor off.** Measured absent with the feature working (2026-10-02). Do not diagnose from it, and do not write it into the file to "fix" it. |
+| `HardwareAcceleratedCodecs="0"` | explicitly **off** — the only conclusive reading, and only in the negative. |
+| `HardwareAcceleratedCodecs="1"` | on. |
+
+**The log settles it, this file does not.** Go to "How to verify"; the `hardware transcoding:
+enabled` and `final decoder/final encoder` lines are the verdict.

 ## The one manual step

@@ -117,8 +207,11 @@ Plex web UI → **Settings → Transcoder → "Use hardware acceleration when av

 The underlying preference key is **`HardwareAcceleratedCodecs`**. That name was confirmed by
 extracting strings from the 23 MB `Plex Media Server` binary in the running container — **not** from
-recall and not from a forum post. Real neighbouring keys from the same binary, so you can recognise
-them and not invent them:
+recall and not from a forum post. **But do not expect to see it in `Preferences.xml` after ticking
+the box:** on 2026-10-02, with the toggle on and hardware transcoding working, the key was absent and
+`HardwareDevicePath` was what had been written. The key exists in the binary; Plex does not
+necessarily persist it. Real neighbouring keys from the same binary, so you can recognise them and
+not invent them:

 | Key | What it is |
 |---|---|
@@ -135,73 +228,223 @@ binary. Recorded explicitly so nobody spends an evening hunting for it or, worse
 Editing `Preferences.xml` by hand is **not** the procedure. It is app state on the replicated Longhorn
 tier, it is in no git commit, and the UI writes the key correctly. Use the UI.

-## How to verify — and why there is no alert
+## How to verify — the log, not the dashboard

-Start a playback that **forces a transcode** (a client that cannot direct-play the source, or pick a
-lower quality explicitly), then read Plex's own **Dashboard → Activity**:
+**Correction, 2026-10-02.** Until today this page said the Plex dashboard showing `Transcode (hw)`
+was "the *only* proof available, and nothing else will ever tell you". **That was false and it cost
+the operator real time:** hardware transcoding was fully working — VAAPI decode *and* encode, with
+zero-copy — while **Dashboard → Activity showed nothing of the sort**. A check that is silent while
+the feature works does not just fail to prove anything; it sends someone hunting a fault that does
+not exist. The authoritative check is **Plex's own server log**.

- **`Transcode (hw)`** on the session ⇒ hardware transcoding. **This is the proof.**
- plain **`Transcode`** ⇒ software. The feature is not working, whatever the manifest says.
+### The primary check — run this one

-**That dashboard is the *only* proof available, and nothing else will ever tell you.**
-`catalog/plex.yaml` declares no `metricsEndpoint`, so Plex renders **no ServiceMonitor at all** —
-there is no scrape, no transcode-mode metric, and therefore no alert that could fire. Do not open work
-to "alert on software-transcode fallback": with no metric behind it such a rule would be permanently
-inert, and this repo has already been burned at scale by rules that evaluated against nothing while
-looking like coverage (`runbooks/README.md`, "Alerts you should not trust the silence of"). If that
-gap is ever worth closing, the honest order is a real exporter first and a rule second.
+```
+./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n plex exec deploy/plex -- \
+  grep -aiE "hardware transcoding: (final|using)" \
+  "/config/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log" \
+  | sed -E "s/(X-Plex-Token|token)=[^&[:space:]]*/\1=REDACTED/g" | tail -5'
+```

-Corroborating, not a substitute — the server log on the config PVC:
+| Line | Verdict |
+|---|---|
+| `hardware transcoding: final decoder: vaapi, final encoder: vaapi` | **PASS.** Hardware in **both** directions. This is the pass condition. |
+| `hardware transcoding: using zero-copy transcoding` | Best case — frames never round-trip through the CPU. Observed on prod 2026-10-02. |
+| `hardware transcoding: final decoder: , final encoder: ` (both empty) | It **fell back to software** for that session. |
+| no matching line at all | No transcode has been attempted since the log rolled — force one (below), then look again. Absence is not a failure. |
+
+The 2026-10-02 reading on prod, in full, for the shape to recognise:
+
+```
+Codecs: hardware transcoding: testing API vaapi for device '/dev/dri/renderD128'
+                              (Intel CoffeeLake-S GT2 [UHD Graphics 630])
+hardware transcoding: using hardware decode accelerator vaapi
+hardware transcoding: zero-copy support present
+hardware transcoding: using zero-copy transcoding
+hardware transcoding: final decoder: vaapi, final encoder: vaapi
+```
+
+To make a line appear, start a playback that **forces a transcode**: a client that cannot
+direct-play the source, or an explicitly lower quality. Direct Play / Direct Stream never reaches
+the transcoder and logs nothing.
+
+### Three things to get right about that command
+
+1. **Always keep the `sed`.** `Plex Media Server.log` carries API URLs containing
+   **`X-Plex-Token=…`**, a live plex.tv credential. Any output that might land in a ticket, a
+   transcript or a chat window must be redacted — CLAUDE.md §6, and a credential has already been
+   leaked into a transcript in this repo exactly this way. The redacting form above is the form to
+   copy; do not "simplify" the pipe out of it.
+2. **`Plex Transcoder Statistics.log` is the WRONG FILE.** The name is irresistible and someone will
+   reach for it. It holds XML `<Variant .../>` session records — **not** ffmpeg/VAAPI output. There
+   is no decoder/encoder verdict anywhere in it. The file you want is `Plex Media Server.log`.
+3. `grep -a` is deliberate: the log carries occasional non-UTF-8 bytes, and without `-a` GNU grep
+   reports "binary file matches" and prints nothing — which reads exactly like "no hardware
+   transcoding".
+
+### Secondary and unreliable: the Plex dashboard
+
+**Dashboard → Activity** *may* label a transcoding session `Transcode (hw)`. If it does, it agrees
+with the log and you are done. **If it does not, that proves nothing at all** — measured 2026-10-02:
+hardware transcoding active and zero-copy, dashboard silent. Never conclude "software transcoding"
+from the dashboard and never open an investigation on its say-so. Kept here only because it is the
+fastest glance available; it is a convenience, **not evidence**.
+
+### Process-level, while a transcode is in flight: the transcoder's argv
+
+This shows Plex *requesting* hardware, as distinct from reporting it afterwards. Get a shell in the
+pod (if `-it` has no TTY under `dev.sh run`, wrap each line in `sh -c "…"` instead):

 ```
 infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
-     sh -c "grep -i -m20 -e hardware -e vaapi -e hwdevice \
-     \"/config/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log\""'
+  'kubectl --context k3s-prod -n plex exec -it deploy/plex -- sh'
 ```

-Treat those strings as advisory: they are Plex's, not this repo's, and they move between versions.
-Absence of a match is **not** evidence of software transcoding; a VA-API device open is good evidence
-of hardware. The dashboard remains the verdict.
+then, inside the pod:
+
+```
+ps -eo pid,user,comm | grep -i transcoder
+tr '\0' ' ' < /proc/<transcoder-pid>/cmdline | grep -oE '(-hwaccel[a-z_]*|vaapi|renderD[0-9]+)'
+```
+
+`-hwaccel`, `-hwaccel_device`, `vaapi`, `renderD128` present ⇒ Plex is asking for hardware. Keep the
+`grep -oE`: the raw argv contains media paths and session identifiers you do not want in a
+transcript. This only works while a session is actually transcoding.
+
+### Documented false lead: scanning `/proc/*/fd` for a holder of `renderD128`
+
+**It returns nothing even when hardware transcoding is working.** Measured 2026-10-02: a
+`/proc/*/fd` sweep inside the pod found **no** process holding `/dev/dri/renderD128`, while the log
+showed `final decoder: vaapi, final encoder: vaapi` for sessions either side of the scan. Transcode
+sessions are **short** and the scan landed in a gap between two of them. An empty fd scan is a
+**timing artefact, not a finding** — do not report it as evidence of software transcoding, and do
+not escalate on it. Use the argv check above (which has the same timing constraint but tells you
+something when it does catch a session), or just read the log, which is retrospective and therefore
+does not care about timing.
+
+### Why there is no alert — this half was always right
+
+`catalog/plex.yaml` declares no `metricsEndpoint`, so Plex renders **no ServiceMonitor at all** —
+there is no scrape, no transcode-mode metric, and therefore no alert that could fire. Do not open
+work to "alert on software-transcode fallback": with no metric behind it such a rule would be
+permanently inert, and this repo has already been burned at scale by rules that evaluated against
+nothing while looking like coverage (`runbooks/README.md`, "Alerts you should not trust the silence
+of"). If that gap is ever worth closing, the honest order is a real exporter first and a rule
+second.

 ## Likely causes, most likely first

-When the dashboard still says plain `Transcode`:
+When the **log** says it fell back to software — `final decoder: , final encoder: ` with both
+fields empty. (Not when the dashboard is quiet: the dashboard's silence means nothing, see "How to
+verify".)

-1. **The toggle is off.** Overwhelmingly the most likely answer, and the reason this page exists.
-   First-command #4 returns `ABSENT` or `="0"`. Remedy: the manual step above. Check this **before**
-   investigating anything technical.
+1. **The toggle is off.** Historically the most likely answer and the reason this page exists. It was
+   ticked on 2026-10-02 and should stay ticked, but a `/config` restore or a stray UI save can unset
+   it. Remedy: the manual step above. **Do not diagnose this from a missing
+   `HardwareAcceleratedCodecs`** — see first-command #4; only `="0"` is conclusive, and the log line
+   `hardware transcoding: enabled` is the real signal that the toggle took.

-2. **containerd's device cgroup is refusing the `open()` — UNVERIFIED, and the largest remaining
-   technical risk in this feature.** Whether a non-privileged pod may `open()` a bind-mounted
-   character device on this k3s/containerd build **has not been established**; it can only be settled
-   by deploying and trying. The container inits as uid 0, which settles *file* permissions, but
-   **uid 0 does not bypass the cgroup device gate.** Symptom: step 2 shows `renderD128` present and
-   correctly owned, step 3 returns `RW-OPEN-DENIED`.
-   **The documented fallback is the Intel Device Plugin for Kubernetes** (`gpu.intel.com/i915` as a
-   resource request, no hostPath). Because the ServiceDefinition field expresses *intent* rather than
-   a device path, that migration is a **`catalog/render.py` change plus a re-render — with no
-   ServiceDefinition edit anywhere**; `catalog/plex.yaml` keeps saying exactly what it says today.
-   **`privileged: true` is not an available escape hatch:** Kyverno `restrict-host-namespaces` is in
-   **Enforce** mode and rejects it in app namespaces. Do not reach for it.
+2. **The device cannot be `open()`ed — the plugin's cgroup allow rule did not arrive.** Symptom:
+   first-command #3 shows `renderD128` present and correctly owned, and the open returns
+   `RW-OPEN-DENIED` / `Operation not permitted`. This is `EPERM` from the **device cgroup**, not
+   `EACCES` from file permissions. **This is no longer a design risk — it is the measured reason the
+   design is what it is** (2026-10-01; see "Why a resource request and not a `/dev/dri` hostPath").
+   If it recurs *under the current mechanism*, the plugin is the suspect, not the manifest: confirm
+   the pod really did consume the resource

-3. **The pod landed on a node without the device.** Step 2 shows an empty `/dev/dri`. All three prod
-   nodes carried `renderD128` with `i915` bound when this was measured (2026-09-30), so this means a
-   node changed — a rebuild, a BIOS reset, or a driver regression. `DirectoryOrCreate` is why the app
-   tier still came up; the cost is that the loss is silent. **Fix the node, do not tighten the mount:**
-   `Directory` would stall `apps-wave-0` (`wait: true`, with `apps-wave-1` depending on it) for a
-   feature that is not load-bearing.
+   ```
+   infra/devcontainer/dev.sh run bash -c \
+     'kubectl --context k3s-prod -n plex get pod -l app.kubernetes.io/name=plex \
+        -o jsonpath="{.items[0].spec.containers[0].resources}"; echo'
+   ```
+
+   and then check the plugin pod on Plex's node (first-command #1). **Do not reach for
+   `privileged: true`** — Kyverno `restrict-host-namespaces` is Enforce and rejects it, and uid 0
+   failed the same open in 2026-09-30's measurement anyway. **Do not re-add a `/dev/dri` hostPath**;
+   that is the mechanism this replaced, and it was measured not to work.
+
+3. **The pod landed on a node whose GPU went away.** Under the current mechanism this does **not**
+   present as an empty `/dev/dri` — the node simply stops advertising `gpu.intel.com/i915` and Plex
+   either never schedules there or gets evicted off. So this cause now shows up as **`Pending`**, not
+   as silent software transcoding: see "Plex will not schedule". All three prod nodes carried
+   `renderD128` with `i915` bound when measured (2026-09-30) and all three advertised the resource on
+   2026-10-01, so this means a node changed — a rebuild, a BIOS reset, or a driver regression.
+   **Fix the node.**

 4. **The image changed under you.** See "The floating-image caveat". A `:latest` pull can remove the
   mechanism this feature leans on with no commit in this repo.

-5. **The session was never transcoding.** Direct Play / Direct Stream never touches the transcoder, so
-   `Transcode (hw)` cannot appear. Confirm the session actually says `Transcode` before treating it as
-   a hardware failure.
+5. **The session was never transcoding.** Direct Play / Direct Stream never reaches the transcoder,
+   so no `hardware transcoding:` line is written for it at all. An *absent* line is this, not a
+   failure — confirm a transcode actually happened before treating anything as a hardware fault.

-6. **The source needs a codec Quick Sync on this generation cannot handle** — e.g. AV1 decode, which
-   Coffee Lake's UHD 630 has no engine for. Plex falls back to software *per stream*. Symptom: some
-   sessions show `(hw)` and others do not. Not a fault.
+6. **The *source* codec may be one this iGPU cannot decode — HYPOTHESIS, not a confirmed
+   diagnosis.** The symptom that points here is fallback on **particular titles** while other
+   sessions are hardware. Measured on prod 2026-10-02, *between* two fully hardware sessions, one
+   session logged:
+
+   ```
+   hardware transcoding: enabled, but no hardware decode accelerator found
+   hardware transcoding: final decoder: , final encoder:
+   ```
+
+   The platform was demonstrably working either side of it, so the likely explanation is the **file**,
+   not the cluster: UHD 630's media engine decodes H.264, HEVC 8-bit and 10-bit, and VP9 — but **not
+   AV1**, and not 10-bit 4:2:2 HEVC. **This was NOT verified against that particular title.** It is
+   inferred from the hardware's capability set and is written here as the lead to check first, not as
+   a finding. So: fallback on *all* files is a platform problem (causes 1-4); fallback on *some*
+   files, check the source codec before suspecting the device, and expect occasional legitimate
+   per-title fallbacks to be normal rather than a fault.
+
+## Plex will not schedule
+
+**Symptom:** the Plex pod sits `Pending`, and `apps-wave-0` does not go Ready, which holds
+`apps-wave-1` behind it. This is the cost of the 2026-10-01 mechanism and the section you want at
+2am.
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n plex describe pod -l app.kubernetes.io/name=plex | tail -20'
+```
+
+`0/3 nodes are available: 3 Insufficient gpu.intel.com/i915` (or similar) confirms it: **nothing is
+advertising the resource Plex requires**. Plex's schedulability is coupled to the `intel-gpu-plugin`
+DaemonSet, deliberately and unavoidably — the plugin is also the only thing that produces the
+device-cgroup allow rule, so there is no variant of this feature that degrades gracefully.
+
+**Remedy A — fix the plugin (preferred).** Run first-command #1. If the DaemonSet is not 3/3, or the
+nodes show `i915=ABSENT`, the fault is in `platform/intel-gpu-plugin/` or on the node, and that is
+where to fix it. Check its Flux Kustomization reconciled:
+
+```
+infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n flux-system get kustomization intel-gpu-plugin'
+```
+
+A plugin that is merely **not yet reconciled** resolves itself; one that is failing does not.
+
+**Remedy B — the escape hatch: give up hardware transcoding and unblock the tier.** One field, no
+code change. In a branch (never on `main`, never `kubectl edit` — Flux reconciles `main` and would
+revert you):
+
+1. delete these two lines from `catalog/plex.yaml`:
+
+   ```yaml
+   hardware:
+     gpu: intel-quicksync
+   ```
+
+2. re-render the golden:
+
+   ```
+   python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation
+   ```
+
+   The only change must be `apps/wave-0/plex.yaml`, losing its `resources` block.
+3. gate, PR, merge. Plex schedules anywhere and **software-transcodes**; the app tier unblocks.
+
+Re-adding the field later is the same edit in reverse. Nothing about the Plex-side toggle or the
+`/config` PVC is affected either way — do **not** delete the PVC as part of this.

 ## The floating-image caveat

@@ -229,21 +472,37 @@ infra/devcontainer/dev.sh run bash -c \
 ```

 `45-plex-hw-transcode-and-connected-tuner` still present ⇒ the group mechanism is still the one
-described here. Gone or renamed ⇒ stop and re-measure. In particular the "no `supplementalGroups`
-needed" conclusion in `catalog/plex.yaml` and `schemas/renderer-contract.md` is only true while that
-hook exists.
+described here: the hook reads the injected device's **own** ownership, which under the device plugin
+is `root:root`, so it resolves GID `0` (see "When to escalate or stop"). Gone or renamed ⇒ stop and
+re-measure. In particular the "no `supplementalGroups` needed" conclusion in `catalog/plex.yaml` and
+`schemas/renderer-contract.md` is only true while that hook exists.

 ## When to escalate or stop

 - **Do not add `supplementalGroups` or a `podSecurityContext` to `catalog/plex.yaml`.** It would be
-  **inert**, and it would mislead the next person. Measured in-container 2026-09-30: the image's
-  `cont-init.d` hook stats each `/dev/dri` char device and `usermod -a -G`s the `plex` user into its
-  group *inside* the container, and the service then execs via `s6-setuidgid plex`, which **replaces**
-  the process group list from `/etc/group` rather than appending — `Plex Media Server` runs with
-  `Groups: 100 1000`, root's gid `0` (which PID 1 does hold) gone. Anything Kubernetes supplies is
+  **inert**, and it would mislead the next person. The image's `cont-init.d` hook stats each
+  `/dev/dri` char device and `usermod -a -G`s the `plex` user into **that device's** group, inside
+  the container; the service then execs via `s6-setuidgid plex`, which **rebuilds** the process group
+  list from `/etc/group` rather than appending to it. Anything Kubernetes supplies is therefore
  discarded before Plex starts. `render.py --selftest` asserts that absence deliberately.
+- **GID `992` is a hostPath-era number — do not expect it, and do not read the root group as a
+  defect.** Under the device plugin the injected `/dev/dri/renderD128` arrives **`root:root`**, so
+  the hook resolves GID **`0`** and adds `plex` to the **root group**. Measured 2026-10-02:
+  `Plex Media Server` runs with `Groups: 0 100 1000`, and `id plex` reports
+  `groups=1000(plex),0(root),100(users)`. Under the old hostPath the device was `root:render`
+  (GID 992 on these nodes) and the reading was `Groups: 44 100 992 1000`. **992 is now irrelevant**:
+  if any page, comment or older handoff tells you to expect it, that text is stale and the system
+  you are looking at is working. The root-group membership is an inherent property of the plugin's
+  `DeviceSpec` — it injects the host device with its host ownership — **not** a defect, and **not** a
+  reason to reintroduce `supplementalGroups`, which would still be a no-op for the reason in the
+  bullet above.
 - **Do not reach for `privileged: true`.** Kyverno `restrict-host-namespaces` is Enforce and will
-  reject it. If the cgroup gate is the blocker, the answer is the Intel device plugin (cause #2).
+  reject it — and it would not help: `uid 0` failed the same `open()` in the 2026-09-30 measurement,
+  because root does not bypass the cgroup device gate.
+- **Do not re-add a `/dev/dri` hostPath, and do not "simplify" the resource request into one.** That
+  was the original mechanism and it was measured non-functional on 2026-10-01 (device-cgroup `EPERM`
+  with the device plainly visible). The device plugin's `DeviceSpec{… Permissions: "rw"}` is what
+  grants access; a volume cannot.
 - **Do not `kubectl edit`/`patch` the Deployment to add or change the device mount.** Flux reconciles
  `main`; a hand-applied change is drift that the next reconcile reverts or, worse, adopts. Mutating
  `kubectl` and `flux reconcile` are **operator** actions, and the durable change is a git change
@@ -253,9 +512,10 @@ hook exists.
 - **Do not delete the `plex-config` PVC to "start clean".** It is the library database, metadata and
  watch state (`policy: daily-offcluster`). The transcode toggle is one preference, not a reason to
  lose the DB.
- **Stop and get the operator** if the device cannot be opened from the pod (cause #2 confirmed). The
-  next step is a platform change — the Intel device plugin — with its own PR, not more poking at this
-  workload.
+- **Stop and get the operator** if the device cannot be opened from the pod as uid 1000 (cause #2
+  confirmed) *under the current mechanism* — note that this open **passed** on 2026-10-02
+  (`RW-OPEN-OK` as uid 1000 via `s6-setuidgid plex`), so a failure now is a regression. The next step is a platform change in
+  `platform/intel-gpu-plugin/`, with its own PR, not more poking at this workload.

 ## Known gaps, deliberately not fixed here

@@ -277,7 +537,25 @@ Recorded so they are findable, and so nobody assumes they were overlooked:

 ---
 Source of truth: `runbooks/plex-quicksync.md`. Index: `/runbooks/`
-The node measurements (all three prod nodes) and the in-container image measurements cited here were
-taken **2026-09-30** against `plexinc/pms-docker` digest `sha256:e0ab27395614…`
-(`plexmediaserver 1.43.4.10903`). Hardware transcoding had **not** been observed working at the time
-of writing, and the Plex-side toggle was still unset.
+
+**Hardware transcoding is CONFIRMED WORKING on prod, measured 2026-10-02.** From
+`Plex Media Server.log` in the running pod: API `vaapi` on `/dev/dri/renderD128`
+(`Intel CoffeeLake-S GT2 [UHD Graphics 630]`), `using hardware decode accelerator vaapi`,
+`zero-copy support present`, `using zero-copy transcoding`, and
+`final decoder: vaapi, final encoder: vaapi`. Also measured in the same pod: `RW-OPEN-OK` opening
+`/dev/dri/renderD128` as uid 1000 via `s6-setuidgid plex`; the pod spec carrying
+`{"limits":{"gpu.intel.com/i915":"1"},"requests":{"gpu.intel.com/i915":"1"}}`; `Groups: 0 100 1000`
+and `id plex` → `groups=1000(plex),0(root),100(users)`; and `HardwareDevicePath` present in
+`Preferences.xml` while `HardwareAcceleratedCodecs` was absent from all 27 attributes.
+
+Earlier readings, kept for provenance and **superseded** where they conflict: the node measurements
+(all three prod nodes) and the in-container image measurements taken **2026-09-30** against
+`plexinc/pms-docker` digest `sha256:e0ab27395614…` (`plexmediaserver 1.43.4.10903`); the hostPath
+`EPERM` open results and the `Groups: 44 100 992 1000` / GID-992 group reading, **2026-10-01, under
+the hostPath mechanism that no longer exists**; and the `gpu.intel.com/i915: 1` allocatable and
+`intel-gpu-plugin` DaemonSet readings, **2026-10-01**.
+
+What this page still does **not** establish: that any particular title's software fallback was caused
+by its source codec (cause 6 is an unverified hypothesis), and that Plex's dashboard *ever* labels a
+session `Transcode (hw)` on this deployment — it has not been observed doing so, which is exactly why
+the log is now the primary check.
@@ -0,0 +1,148 @@
+# plex-remote-access — getting Plex reachable from outside the LAN, and proving it
+
+**Status: the in-cluster half is shipped; the last mile is TWO MANUAL STEPS and remote access does
+NOT work until both are done.** Neither can live in this repo.
+**Alert:** none, and none is possible from in-cluster signals — see "Why no alert" below. This is an
+**operator procedure**, not an incident page.
+**Files:** `apps/wave-0/plex-remote.yaml` (the tcp/32400 LoadBalancer + its CiliumNetworkPolicy),
+`clusters/base/cluster-vars.yaml` + `clusters/prod/kustomization.yaml` (`PLEX_L4_VIP`),
+`catalog/render.py` (the `EXTRA_UNITS` entry that places the unit), `catalog/plex.yaml` (Plex's own
+env — note `ADVERTISE_IP` is deliberately NOT set; see "Why not the Gateway URL").
+**Related:** `runbooks/plex-quicksync.md` and `runbooks/plex-claim.md` — the other two Plex
+procedures with a manual last mile.
+
+## What was actually broken (measured 2026-10-03)
+
+Plex was serving correctly and was reachable on the LAN. Two independent things stopped remote
+clients:
+
+1. **Plex advertised no remote connection at all.** In the live `Preferences.xml`:
+   `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while
+   `PublishServerOnPlexOnlineKey="1"`. The server was trying to publish itself to plex.tv and had
+   nothing reachable to publish. Plex's automatic path (UPnP/NAT-PMP) cannot work here: the process
+   lives in a pod on the Cilium overlay (`10.0.2.136`), it cannot speak UPnP to the SOHO router, and
+   a mapping to the pod IP would be meaningless if it could.
+2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` → connection refused.
+   `136.51.115.252:443` → a **self-signed router certificate** (`C=ZZ, CN=%D`), not the Gateway's
+   Let's Encrypt wildcard.
+
+The Service was `ClusterIP` only, so port 32400 had no path off-cluster by any route.
+
+**Caveat on that second measurement, because it is easy to over-read:** it was taken from *inside*
+the LAN, so strictly it proves there is no NAT hairpin (reflection). It does not prove outside-in
+failure. But a router certificate answering 443 means the router's own management UI is bound to the
+WAN port, and that *would* shadow a port-forward for genuine outside traffic too. **Verify from a
+real outside vantage point** (phone on cellular, or an external port checker) before concluding
+either way — this repo has no such vantage point and must not pretend otherwise.
+
+## Step 1 — forward WAN tcp/32400 on the router (operator, off-repo)
+
+    WAN tcp/32400  ->  192.168.2.24  (${PLEX_L4_VIP})  port 32400
+
+`192.168.2.24` is a pinned Cilium LB VIP, ARP-announced on the flat LAN by the existing
+`CiliumL2AnnouncementPolicy default-l2`. It is **pinned on purpose**: this address is typed into the
+router by hand, so it must never move on a reschedule.
+
+While in the router, **check what else is bound to the WAN interface.** If remote management is
+listening on 443 (see the certificate measurement above), move it or disable it — otherwise the
+Gateway's 80/443 forward is shadowed and the Plex *web UI* over the internet stays broken even
+though remote *streaming* works. These are two separate paths and they fail independently.
+
+Confirm the VIP is actually answering on the LAN first, so a failure later is unambiguous:
+
+    ./infra/devcontainer/dev.sh run bash -c \
+      'kubectl --context k3s-prod -n plex get svc plex-l4-tcp -o wide'
+
+`EXTERNAL-IP` must read `192.168.2.24`. If it is `<pending>`, LB-IPAM could not assign — the address
+is outside the pool or already taken; see `clusters/prod/kustomization.yaml` for the pool accounting.
+
+**`192.168.2.24` was never ARP-probed off-cluster.** Only the in-cluster pool accounting was checked
+(IPsTotal 11 / IPsUsed 4 / IPsAvailable 7 on 2026-10-03). If the VIP behaves oddly, probe it from a
+LAN host — `arping 192.168.2.24` — the way `.23` was probed before it was claimed.
+
+## Step 2 — tick "manually specify public port" in Plex (operator, UI only)
+
+**There is no GitOps path to this and no environment variable for it.** Measured in the running
+image on 2026-10-03: a grep of the entire init surface (`/etc/cont-init.d`, `/plex-common.sh`,
+`/usr/local/bin`) for `ManualPortMapping` and `PLEX_PREFERENCE` returned **NONE**. The image writes
+only six preferences, and this is not one of them. It is the same class of manual last mile as the
+`HardwareAcceleratedCodecs` toggle in `runbooks/plex-quicksync.md`.
+
+In the Plex web UI (`https://plex.home.keithjackson.dev`, on the LAN):
+
+**Settings → Remote Access**
+- Ensure **Enable Remote Access** is on. (`PublishServerOnPlexOnlineKey` was already `1`.)
+- Tick **Manually specify public port** and set it to **32400**.
+- **Apply**, then let the panel settle. It should report *"Fully accessible outside your network"*.
+
+Without this tick Plex waits for a NAT mapping it can never obtain and publishes no remote
+connection, so the port-forward in step 1 sits there unused.
+
+## How to verify
+
+Do these in order; each one isolates a different half.
+
+**1. The port is open from outside.** From a phone on cellular, or any off-LAN host:
+
+    curl -sk -o /dev/null -w '%{http_code}\n' https://136.51.115.252:32400/identity
+
+`200` means the forward and the policy are both good. A hang or refusal means step 1.
+
+**2. Plex itself believes it is reachable.** Settings → Remote Access must say *"Fully accessible
+outside your network"*. If it says *"not available outside your network"* while test 1 passes, step 2
+was not applied.
+
+**3. plex.tv is handing out a remote connection.** The decisive test, because it is what clients
+actually consume: sign in to [app.plex.tv](https://app.plex.tv) from an off-LAN network and confirm
+the server appears and plays. A client finds this server through plex.tv's connection list, never
+through DNS.
+
+**4. The policy is admitting the traffic.** If test 1 fails *after* the forward is confirmed, the
+CiliumNetworkPolicy is the next suspect:
+
+    ./infra/devcontainer/dev.sh run bash -c \
+      'kubectl --context k3s-prod -n plex get cnp plex-allow-remote-32400'
+
+See `runbooks/denied-flows.md` for reading actual drops.
+
+## Why no alert
+
+Every in-cluster signal is green whether or not remote access works. The Service has endpoints, the
+pod is `Running`, the policy is applied, and `plex-down` watches
+`kube_deployment_status_replicas_available` — none of which can see a router with a missing
+port-forward or an unticked checkbox. An alert would need an off-LAN prober, which this cluster does
+not have. **Test 3 above is the only honest check, and it is manual.**
+
+## Why not the Gateway URL (`ADVERTISE_IP`)
+
+The alternative was to leave Plex on `ClusterIP` and set
+`ADVERTISE_IP: https://plex.home.keithjackson.dev:443`, which writes Plex's `customConnections`
+preference and would have put the server on plex.tv's list via the existing L7 Gateway — no new WAN
+hole. That mechanism was verified to work on an already-claimed server (the `ADVERTISE_IP` block at
+line 126 of the image's `40-plex-first-run` sits **outside** the claim guard, and `/.firstRunComplete`
+lives on the overlay FS, not the `/config` PVC, so it re-runs every pod start), and the Gateway path
+itself was measured serving Plex correctly: `HTTP 200`, `claimed="1"`, with `401` on every protected
+endpoint.
+
+**The operator chose the native path anyway (2026-10-03), and the reasons are real:** Plex terminates
+its own TLS with its `*.plex.direct` certificate, and video streams bypass the Envoy L7 hop instead
+of being proxied through it. The cost is a new unfiltered WAN port — acceptable because Plex
+authenticates every caller itself. `ADVERTISE_IP` therefore stays unset in `catalog/plex.yaml`, and
+the deferral note there is still accurate.
+
+## Known wart: remote clients are classified as LAN
+
+`apps/wave-0/plex-remote.yaml` uses `externalTrafficPolicy: Cluster`, which SNATs the client to the
+ingress node's address (`192.168.2.1x`). That falls inside the `192.168.0.0/16` entry of
+`ALLOWED_NETWORKS`, so **Plex treats every remote stream as local.** Measured on the Gateway path,
+which SNATs the same way: Plex logged `[10.0.2.214:36040 (Allowed Network (WAN))]`.
+
+This is a **metering** effect, not a security one — authorization is unchanged, as the 401s above
+show — but Plex's remote-quality and bandwidth limits will not bind on these streams.
+
+`Local` would preserve the client IP and fix it. **Do not switch to it.** Cilium's L2 announcer
+elects one lease holder per Service and does *not* follow the backend pod, so the VIP can be
+announced by a node with no Plex pod, black-holing the port while everything reports healthy. That
+exact change cost this LAN an 8-day Minecraft outage in September 2026 — the full write-up is at the
+top of `platform/crafty/service-l4.yaml`. Plex is a single-replica `Recreate` Deployment, so the
+node mismatch is the normal case here, not a rare one.
@@ -161,61 +161,155 @@ platform must render, and cites the requirement each mapping satisfies. Changing
 - **Requirements:** **R-STATE-01** (rebuild survival), **R-STATE-02** (Longhorn replication),
  **R-STATE-04** (bulk tier). (docs/04)

+### `availability` (`replicas`) — workload kind, per-replica volumes, spreading, budget
+
+**Schema 1.10.0.** Availability INTENT. It names no Kubernetes object, no topology key and no
+budget — exactly like `hardware.gpu`, and for the same reason: the mechanism below should be
+replaceable without a catalog edit.
+
+- **Absent, or `replicas: 1`** → a **Deployment**, standalone PVC(s), `strategy: Recreate` when there
+  is storage, no PodDisruptionBudget. Byte-identical to pre-1.10.0 output, so every existing app is
+  untouched.
+- **`replicas: N > 1`** → all of the following, together:
+  | Rendered | Why |
+  |---|---|
+  | **StatefulSet** instead of Deployment | A longhorn PVC is **ReadWriteOnce**. One RWO claim cannot attach on two nodes, so a multi-replica Deployment does not degrade — the second pod never schedules. |
+  | longhorn storage becomes **`volumeClaimTemplates`** | Gives each replica its own claim (`<vol>-<svc>-<ordinal>`, e.g. `config-pihole-0`). The template's `metadata.name` **must** equal the pod volume name the container mounts. The standalone PVC is then **not** emitted — it would be an orphan nothing mounts. |
+  | `bulk` storage stays a **standalone PVC** | A bulk entry is a static, pre-bound NFS PV over data that already exists outside the cluster and is shared read-only. Per-replica copies are neither possible nor wanted. |
+  | **required** `podAntiAffinity` on `kubernetes.io/hostname` | `preferred` would let every replica land on one node and deliver none of the intended node-failure tolerance, while every object still reported healthy. |
+  | `podManagementPolicy: Parallel` | `OrderedReady` serialises first-boot work across replicas; `apps-wave-0` has `timeout: 5m`. There is no ordering requirement between replicas to protect. |
+  | **PodDisruptionBudget**, `minAvailable: 1` | Stops a *voluntary* disruption (node drain, upgrade) taking the last replica. It does nothing about a node dying — that is what the replica count and anti-affinity are for. Never rendered at `replicas: 1`, where `minAvailable: 1` would block every drain of that node forever. |
+  | `observability.livenessWorkload.kind` defaults to **`StatefulSet`** | The `up` SLI reads kube-state-metrics by workload kind. A Deployment-shaped rule against a StatefulSet `absent()`s forever — an alert that can never fire. Deriving the default means bumping the replica count cannot leave a stale rule behind. |
+
+**Two costs, both deliberate, neither hidden:**
+
+1. **N replicas need N schedulable nodes.** On a 3-node cluster, losing two nodes leaves a replica
+   `Pending`; since `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, that **stalls the
+   app tier**. That coincides with having lost etcd quorum on a 3-node control plane, so nothing is
+   reconciling at that point anyway — the stall is a symptom, not a new failure mode.
+2. **Only declare `> 1` for a service whose on-disk state is reproducible from git.** A replica with
+   a fresh volume comes up serving whatever it can bootstrap itself and nothing else. For pihole that
+   prerequisite was done first — the admin password and the adlists were declared in
+   `decisions/2026-10-04-pihole-declare-state.md` — because otherwise a second replica would have
+   served DNS while blocking nothing, with a different password.
+
+**Requirements:** R-PLAT-02 (the workload), R-STATE-01/02 (per-replica replicated volumes),
+R-OBS-01 (the SLI must follow the rendered kind).
+
 ### `hardware` (`gpu`) — host device capability
 - **The field is INTENT, never MECHANISM.** `hardware.gpu` is an **enum of capability classes**
  (today: `intel-quicksync`), not a device path, not a hostPath, not a resource request. The
  ServiceDefinition says *what class of host device this workload needs*; how the platform hands it
  over is `catalog/render.py`'s business and lives nowhere else. **Do not "simplify" this into a raw
-  `/dev/...` path.** The entire payoff is that migrating from the hostPath mechanism below to the
-  Intel GPU device plugin (`resources.limits: gpu.intel.com/i915`) is a renderer change with **zero**
-  catalog edits and zero re-review of per-service device wiring. Extend the enum for a genuinely
-  different capability class (e.g. `nvidia-nvenc`); never to name a second route to the same one.
- `gpu: intel-quicksync` → on the **app container only** (never a sidecar), exactly two things:
-  1. a **hostPath volume** `dri` → `/dev/dri`, **`type: DirectoryOrCreate`**;
-  2. a **read-write volumeMount** of `dri` at `/dev/dri` on the app container (VA-API opens the
-     render node read-write, so it is deliberately not `readOnly`).
+  `/dev/...` path.** The payoff is no longer hypothetical: on **2026-10-01** the mechanism was
+  swapped outright — from a `/dev/dri` hostPath to the Intel GPU device plugin's extended resource —
+  as a **`catalog/render.py` change plus a re-render, with ZERO catalog edits** and no re-review of
+  per-service device wiring. `catalog/plex.yaml`'s `hardware:` block came through byte-identical.
+  Extend the enum for a genuinely different capability class (e.g. `nvidia-nvenc`); never to name a
+  second route to the same one.
+- `gpu: intel-quicksync` → on the **app container only** (never a sidecar), **exactly one thing**:

-  **Mounting the device is sufficient. That is the whole mapping.**
+      resources:
+        limits:
+          gpu.intel.com/i915: 1
+
+  **That is the whole mapping.** No volume, no volumeMount, no `securityContext`, no `nodeSelector`,
+  no privileged anything.
+  - **`limits`, not `requests`, and that is not a style choice.** An extended resource must be
+    expressed in `limits`; Kubernetes mirrors `requests` from `limits` automatically, and a
+    requests-only extended resource is rejected as invalid.
+  - The resource is advertised by the **`intel-gpu-plugin` DaemonSet** (`platform/intel-gpu-plugin/`,
+    its own Flux Kustomization at `clusters/base/intel-gpu-plugin.yaml`, pinned
+    `intel/intel-gpu-plugin:0.37.0`). **Measured 2026-10-01:** all three prod nodes report
+    `.status.allocatable["gpu.intel.com/i915"] == 1`; DaemonSet 3 desired / 3 ready / 3 available.
+  - This is the **only** app-container `resources` render path in the file, and it is deliberately
+    narrow. **Do not generalise it into a CPU/memory resources feature** while passing through — that
+    is a separate field with a separate justification, and `render.py --selftest` asserts the emitted
+    dict *exactly* so the attempt fails loudly.
+- **WHY THE hostPath MECHANISM WAS ABANDONED — read this before "simplifying" it back.** The first
+  implementation (merged 2026-09-30) rendered a `dri` hostPath volume at `/dev/dri` plus a read-write
+  mount. It was **measured not to work** in the live Plex pod on k3s-prod-2 on **2026-10-01**. The
+  device node was **visible** in the container — `crw-rw---- root video2 226,128 renderD128` — and
+  **could not be opened**:
+
+  | attempt | result |
+  |---|---|
+  | `uid 0`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+  | `uid 1000`, `O_RDWR` | **`EPERM`** "Operation not permitted" |
+
+  `EPERM` is the **device cgroup** refusing, not `EACCES` from file permissions — the two look
+  identical from the outside and are not the same problem. Proven three ways: root on the **host**
+  opens the same device `rc=0`; the node is unified `cgroup2fs`; and the container's runc
+  `config.json` reads `linux.resources.devices: [{allow: False, access: 'rwm'}]` with
+  `linux.devices: None` — **one deny-all rule, no allow entry for major 226**. A hostPath is a
+  *filesystem* operation: it binds the device node in but adds **no cgroup allow rule**, because
+  Kubernetes cannot infer one from a volume. **Note that `uid 0` failed too** — root does not bypass
+  the cgroup device gate, so "run it as root" is not a workaround. Neither is `privileged: true`:
+  Kyverno `restrict-host-namespaces` is **Enforce** on app namespaces and rejects it.
+  The device plugin is what produces the allow rule — confirmed in the v0.37.0 source,
+  `createDeviceSpecsFromDrmFiles` emits `DeviceSpec{…, Permissions: "rw"}`, which the kubelet turns
+  into both the device injection and the matching cgroup allow.
+- **THE FAILURE MODE THIS TRADED INTO IS WORSE IN BLAST RADIUS, AND IT IS UNAVOIDABLE.** Stated
+  plainly rather than sold:
+
+  | | device/resource unavailable ⇒ |
+  |---|---|
+  | **Before** (hostPath, `DirectoryOrCreate`) | kubelet mounts an empty dir; Plex runs and **silently software-transcodes**. Invisible, but harmless to everything else. |
+  | **Now** (extended resource) | the pod is **Unschedulable**. `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, so this **stalls the app tier**. |
+
+  There is **no graceful-degradation variant**: an extended resource is a hard scheduling
+  constraint, and it is also the only thing that produces the cgroup allow rule. The workload's
+  schedulability is now **coupled to the `intel-gpu-plugin` DaemonSet** — a plugin that is broken,
+  removed, or merely not yet reconciled takes the app tier with it. That is the price of hardware
+  transcoding working at all.
+  - **Escape hatch, one field and no code change:** delete the `hardware:` block from the
+    ServiceDefinition (`catalog/plex.yaml`) and re-render. The workload returns to software
+    transcoding and the tier unblocks. `runbooks/plex-quicksync.md` carries it as a first-class
+    procedure, because it is what someone will need at 2am.
+  - The old mechanism's cost is **gone**, and that is the one improvement: a missing device is no
+    longer silent. There is still no metric and no alert (media apps here render no
+    `metricsEndpoint`, so no ServiceMonitor exists), but "Plex is Unschedulable" is loud, whereas
+    "Plex is quietly software-transcoding" never was.
 - **`podSecurityContext.supplementalGroups` is deliberately NOT rendered, and adding it would be a
-  regression rather than a hardening.** `/dev/dri/renderD128` is `root:render` `0660`, so the obvious
+  regression rather than a hardening.** This survives the mechanism swap unchanged, and it was
+  re-measured against the plugin's behaviour. `renderD128` is `root:render` `0660`, so the obvious
  move is to push the host `render` GID in — and it does not work. **Measured in the running
-  container on prod-2, 2026-09-30:** the image's own
+  container on prod-2 (2026-09-30, re-measured 2026-10-01):** the image's own
  `/etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner` hook `find`s every char device under
  `/dev/dri`, `stat -c '%g'` each one's GID, `groupadd`s a matching group *inside the container* when
-  none exists, and `usermod -a -G` the app user into it — every start, nothing hardcoded.
-  `cont-init.d` completes before `services.d`, so the account database is already correct when the
-  service execs. The service then starts via `exec s6-setuidgid plex …`, and `s6-setuidgid`
-  **replaces** the process group list from `/etc/group` rather than appending to it:
-  `Plex Media Server` runs as `Uid: 1000  Gid: 1000  Groups: 100 1000`, with root's gid `0` — which
-  PID 1 (`s6-svscan`) does hold — **gone**. A Kubernetes `supplementalGroups` therefore reaches
-  `s6-svscan` and is discarded before the app ever starts. It would be inert *and* it would mislead
-  whoever debugs this next. The renderer asserts its own absence in `render.py --selftest`.
+  none exists, and `usermod -a -G` the app user into it — every start, nothing hardcoded. It does
+  exactly the same thing to a **plugin-injected** device as to a bind-mounted one, which is why this
+  swap needed no group work. `cont-init.d` completes before `services.d`, so the account database is
+  already correct when the service execs. The service then starts via `exec s6-setuidgid plex …`, and
+  `s6-setuidgid` **replaces** the process group list from `/etc/group` rather than appending to it —
+  measured live, `Plex Media Server` ran with `Groups: 44 100 992 1000`, where **992 is the host
+  `render` GID**, while root's gid `0` (which PID 1, `s6-svscan`, does hold) was **gone**. A
+  Kubernetes `supplementalGroups` therefore reaches `s6-svscan` and is discarded before the app ever
+  starts. It would be inert *and* it would mislead whoever debugs this next. The renderer asserts its
+  own absence in `render.py --selftest`.
  - A corollary worth having: because the hook re-`stat`s the device on every start, the host GID
    **drifting** (a node rebuild renumbering the *systemd dynamic* `render` group) is self-healing.
    No GID is pinned anywhere in this repo.
-  - **Scope of that measurement, stated so nobody over-reads it.** "Mounting the device is
-    sufficient" was established for an image that resolves the device group itself. A FUTURE
-    consumer that runs non-root and has no equivalent init hook would still need the group solved
-    somehow — and the answer for that case is a **renderer** change (per-image handling, or the
-    device plugin), not a new ServiceDefinition field. The enum is what keeps that option open. The
-    first such consumer must re-measure rather than assume this bullet covers it.
- **`DirectoryOrCreate` is load-bearing — do not tighten it to `Directory`.** A node that comes back
-  without the render device (rebuild, BIOS change, driver regression, hardware swap) must degrade the
-  workload to **software transcoding**, not fail the mount. `apps-wave-0` is `wait: true` and
-  `apps-wave-1` `dependsOn` it, so a failed hostPath mount stalls the **whole app tier** for the sake
-  of a feature that is not load-bearing. `/dev` is tmpfs, so the empty directory the kubelet creates
-  on such a node leaves nothing durable behind.
-  - **The cost of that choice, stated plainly:** a missing device is **silent**. The workload
-    software-transcodes and nothing reports it. There is no metric to alert on — media apps here
-    render no `metricsEndpoint`, so no ServiceMonitor exists — and an alert that cannot fire is worse
-    than none (see §`observability`'s 2026-09-13 correction). The only proof of hardware transcode is
-    an operator observing it in the application's own dashboard.
+  - **Scope of that measurement, stated so nobody over-reads it.** "The group is the image's job" was
+    established for an image that resolves the device group itself. A FUTURE consumer that runs
+    non-root and has no equivalent init hook would still need the group solved somehow — and the
+    answer for that case is a **renderer** change (per-image handling), not a new ServiceDefinition
+    field. The enum is what keeps that option open, and the 2026-10-01 swap is the proof it works.
+    The first such consumer must re-measure rather than assume this bullet covers it.
 - **Deliberately NOT rendered** from this field: no `privileged: true` (Kyverno
-  `restrict-host-namespaces` is Enforce on app namespaces and would reject it, and the render node
-  does not need it); no `nodeSelector` (placement is a separate concern with its own justification —
-  this field is about the device, not about where the pod lands); no resource requests/limits (there
-  is no app-container resource render path, and inventing one here would be scope creep).
- Absent ⇒ **nothing at all** is rendered; every service that does not opt in stays byte-identical.
+  `restrict-host-namespaces` is Enforce on app namespaces and would reject it, and it would not have
+  helped anyway — see the `EPERM` table); no `nodeSelector` (placement is a separate concern with its
+  own justification — this field is about the device, not about where the pod lands; the extended
+  resource does constrain placement to nodes the plugin advertises, but that is the **scheduler's**
+  doing, not a selector this renderer writes); no CPU/memory requests or limits (see above — the
+  `resources` block carries the extended resource and nothing else).
+- **Historical, kept so a reader recognises it rather than reintroduces it:**
+  `type: DirectoryOrCreate` on the retired hostPath was load-bearing *for that mechanism* — it is
+  what made a GPU-less node degrade to software transcoding instead of failing the mount and
+  stalling the app tier. That rationale died with the hostPath; it is **not** an argument for bringing it back, since
+  the mechanism it protected never granted access in the first place.
+- Absent ⇒ **nothing at all** is rendered; every service that does not opt in stays byte-identical
+  (no `resources` key, no volume, no mount, no `securityContext`).
  Ignored for `kind: vm` — a VM receives devices through KubeVirt's own `domain.devices`.
 - **Requirements:** **R-PLAT-02** (the PaaS provides the host capability; the app declares need).

@@ -365,11 +459,18 @@ platform must render, and cites the requirement each mapping satisfies. Changing
   `external-dns.alpha.kubernetes.io/controller: none`, and no `public` service's route carries it.
   Derived from `exposure` alone; enforced on every gate run by
   `conformance/tests/l0-static/internal-services-publish-no-public-dns` (R-DNS-03).
-8. **`hardware` is inert until opted into, and its mechanism is renderer-local.** An SD with no
-   `hardware` key renders no volume and no volumeMount; an SD *with* it gains no `securityContext`
-   either. Both directions are proven by `render.py --selftest`. The `/dev/dri` hostPath and the
-   `DirectoryOrCreate` type appear in `catalog/render.py` and nowhere else, so the capability's
-   implementation can be replaced without touching a single ServiceDefinition.
+8. **`hardware` is inert until opted into, and its mechanism is renderer-local — now DEMONSTRATED,
+   not merely claimed.** An SD with no `hardware` key renders no `resources` block, no volume and no
+   volumeMount; an SD *with* it gains a `gpu.intel.com/i915: 1` **limit on the app container only**
+   (never a sidecar) and no `securityContext` either. All directions are proven by
+   `render.py --selftest`, and each assertion is mutation-tested. The resource name
+   `gpu.intel.com/i915` appears in `catalog/render.py` and nowhere else in the catalog path, so the
+   capability's implementation can be replaced without touching a single ServiceDefinition — which
+   is exactly what happened on **2026-10-01**, when the mechanism moved from a `/dev/dri` hostPath
+   (measured non-functional: device-cgroup `EPERM`) to the device plugin's extended resource with
+   **zero** catalog edits. The cost of that mechanism is recorded in §`hardware`: the workload is now
+   **Unschedulable** when the resource is absent, which stalls the app tier, and the escape hatch is
+   to delete the `hardware:` block and re-render.

 ## Requirement traceability (contract-level)
 Primary **R-GITOPS-02** (single reviewable object wires every requirement). Field-level: R-NET-01/02
@@ -2,7 +2,7 @@
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://home.keithjackson.dev/schemas/servicedefinition.schema.json",
  "title": "ServiceDefinition",
-  "description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.9.0 (adds optional `auth.apiRoutes` -- a per-service list of anchored path regexes the forward-auth oauth2-proxy sidecar answers with 401 instead of a 302 to Keycloak, rendered as one repeatable `--api-route` per entry. The `auth` block stays `additionalProperties: false`; this is one named, documented key, not an opened door. Without it there was NO per-app override hook at all, so the only way to answer an app's XHRs with 401 was a renderer-wide constant -- and the path shape is a property of the app: jackett's is `/api/` (33 x 302 vs 2 x 401 measured 2026-09-30, 4 of 6 POSTs answered by the proxy and never reaching the app), radarr's is `/signalr/` (232 x 302 measured 2026-10-01, every one a POST from a left-open browser tab that is never re-prompted), and pihole serves no `/api/` at all. Each entry must be anchored at `^/` and continue with literal path characters, because a too-broad regex turns a plain page load into a 401 and the service looks permanently broken to a logged-out user. Enforced on every gate run by ci/l0.sh's schema stage over catalog/*.yaml, by catalog/render.py refusing to render an unanchored entry, and by conformance/tests/l0-static/oauth2-proxy-api-routes-per-app)). 1.8.0 (adds optional `hardware` - a host HARDWARE CAPABILITY declaration for the container kind, currently one member: `hardware.gpu: intel-quicksync`, meaning the workload needs the host iGPU render device for hardware video transcode. It is an ENUM OF INTENT, not a device path: the renderer satisfies it today with just two things -- a /dev/dri hostPath volume (`DirectoryOrCreate`) and a read-write /dev/dri volumeMount on the app container -- and a later move to the Intel GPU device plugin is then a render.py change with no catalog edit. It sets NO podSecurityContext.supplementalGroups: measured in-container on prod 2026-09-30, the pms-docker cont-init hook adds the app user to the device's group inside the container and `s6-setuidgid` then REPLACES the process group list from /etc/group, discarding anything Kubernetes supplied, so such an entry would be inert and misleading. Renders NOTHING when absent, so every existing app stays byte-identical. Emits no privileged securityContext (Kyverno restrict-host-namespaces would reject it and it is not needed), no nodeSelector (node placement is a separate concern with its own justification) and no resource requests). 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.md already said `internal` renders no public DNS record; nothing enforced it, and six internal services -- jackett, pihole, qbittorrent, radarr, semaphore, sonarr -- were publishing `<name>.home.keithjackson.dev` into the PUBLIC Cloudflare zone answering with the RFC1918 Gateway VIP, which published the internal subnet and a full service inventory and made each name a DNS-rebind primitive. Enforced on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns). 1.6.0 (splits `observability.metrics` from WHERE metrics live: adds optional `metricsEndpoint` (port/name/path/interval) and optional `livenessWorkload` (kind/name). `metrics: true` previously rendered a ServiceMonitor hardcoded to a port named `metrics` that no rendered Service ever defined, so all twelve app monitors discovered ZERO targets and every `up{job=...}` SLI was structurally inert on prod; a monitor is now rendered only from a declared `metricsEndpoint`, which also renders the Service port and the monitoring-namespace ingress allow, and the `up` SLI is expressed against kube-state-metrics -- a signal that exists for every workload without a per-app exporter. `livenessWorkload` names the object whose readiness IS the service, for the case where the running workload is not the one this SD renders). 1.5.0 (adds optional `inClusterClients` — additional in-cluster clients, named by NAMESPACE, admitted to a declared port of this service DIRECTLY, bypassing the shared Gateway. `ingressRoutes` admits only the Gateway proxy identity and `dependencies` only links catalog services, so a PLATFORM controller dialling a Service ClusterIP had no authorable form at all; that gap is what deadlocked proLine truncated
+  "description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.10.0 (adds optional `availability` for the container kind, one member: `availability.replicas`. It is AVAILABILITY INTENT, not a scheduling primitive -- it names no Kubernetes object, no topology key and no budget. 1 (the default) renders a Deployment exactly as before, so every pre-existing app stays byte-identical; >1 means the service must survive the loss of a single node, and the renderer satisfies that with a StatefulSet whose longhorn volumes become per-replica volumeClaimTemplates (a longhorn PVC is ReadWriteOnce and cannot attach on two nodes, so a multi-replica Deployment is UNSCHEDULABLE, not merely suboptimal), REQUIRED podAntiAffinity on kubernetes.io/hostname (preferred would let every replica land on one node and deliver none of the intended tolerance while reporting healthy), podManagementPolicy Parallel, a PodDisruptionBudget of minAvailable 1, and an `up` SLI that follows the rendered kind (a Deployment-shaped rule against a StatefulSet absent()s forever and can never fire). Added for pihole, where LAN DNS -- a critical network service -- was one pod on one node: measured 2026-10-04, losing the node running it meant roughly 6-10 minutes with no LAN resolution (tolerationSeconds 300 plus node-NotReady detection plus a Longhorn RWO force-detach). Deliberately NOT usable until a service's state is reproducible from git, because a replica with a fresh volume otherwise comes up serving but unconfigured -- for pihole that prerequisite was done first (decisions/2026-10-04-pihole-declare-state.md). Capped at 5: a count above the node count can never schedule under the anti-affinity this implies. Mechanism documented in schemas/renderer-contract.md `### availability`). 1.9.0 (adds optional `auth.apiRoutes` -- a per-service list of anchored path regexes the forward-auth oauth2-proxy sidecar answers with 401 instead of a 302 to Keycloak, rendered as one repeatable `--api-route` per entry. The `auth` block stays `additionalProperties: false`; this is one named, documented key, not an opened door. Without it there was NO per-app override hook at all, so the only way to answer an app's XHRs with 401 was a renderer-wide constant -- and the path shape is a property of the app: jackett's is `/api/` (33 x 302 vs 2 x 401 measured 2026-09-30, 4 of 6 POSTs answered by the proxy and never reaching the app), radarr's is `/signalr/` (232 x 302 measured 2026-10-01, every one a POST from a left-open browser tab that is never re-prompted), and pihole serves no `/api/` at all. Each entry must be anchored at `^/` and continue with literal path characters, because a too-broad regex turns a plain page load into a 401 and the service looks permanently broken to a logged-out user. Enforced on every gate run by ci/l0.sh's schema stage over catalog/*.yaml, by catalog/render.py refusing to render an unanchored entry, and by conformance/tests/l0-static/oauth2-proxy-api-routes-per-app)). 1.8.0 (adds optional `hardware` - a host HARDWARE CAPABILITY declaration for the container kind, currently one member: `hardware.gpu: intel-quicksync`, meaning the workload needs the host iGPU render device for hardware video transcode. It is an ENUM OF INTENT, not a device path: the renderer satisfies it today with just two things -- a /dev/dri hostPath volume (`DirectoryOrCreate`) and a read-write /dev/dri volumeMount on the app container -- and a later move to the Intel GPU device plugin is then a render.py change with no catalog edit. It sets NO podSecurityContext.supplementalGroups: measured in-container on prod 2026-09-30, the pms-docker cont-init hook adds the app user to the device's group inside the container and `s6-setuidgid` then REPLACES the process group list from /etc/group, discarding anything Kubernetes supplied, so such an entry would be inert and misleading. Renders NOTHING when absent, so every existing app stays byte-identical. Emits no privileged securityContext (Kyverno restrict-host-namespaces would reject it and it is not needed), no nodeSelector (node placement is a separate concern with its own justification) and no resource requests). 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.mdLine truncated
  "type": "object",
  "additionalProperties": false,
  "required": ["name", "source"],
@@ -139,6 +139,21 @@
        "supplementalGroups": { "type": "array", "items": { "type": "integer" }, "description": "Additional GIDs added to the first process in each container." }
      }
    },
+    "availability": {
+      "type": "object",
+      "description": "Optional availability intent for the container kind: how many replicas of this workload must be running. INTENT ONLY -- it names no Kubernetes object and no scheduling primitive. How the platform satisfies it (workload kind, per-replica volumes, node spreading, disruption budget) is catalog/render.py's business, documented in schemas/renderer-contract.md `### availability`. Absent => replicas 1 and a Deployment, so every pre-existing app renders byte-identical. Ignored for kind=vm (a VM's availability is a different problem with different primitives).",
+      "additionalProperties": false,
+      "minProperties": 1,
+      "properties": {
+        "replicas": {
+          "type": "integer",
+          "description": "How many replicas must be running. 1 (default) renders a Deployment exactly as before. >1 declares that the service must survive the loss of a single node; the renderer then emits a StatefulSet with per-replica volumeClaimTemplates, required per-node anti-affinity, podManagementPolicy Parallel and a PodDisruptionBudget. Only declare >1 for a service whose on-disk state is reproducible from git -- a replica with a fresh volume comes up serving whatever it can bootstrap itself, and nothing else.",
+          "minimum": 1,
+          "maximum": 5,
+          "default": 1
+        }
+      }
+    },
    "hardware": {
      "type": "object",
      "description": "Optional host HARDWARE CAPABILITY the workload needs, for the container kind (docs/13). This field declares INTENT (`what device class this service needs`), never MECHANISM (`which host path to bind-mount`, `which extended resource to request`). That separation is the whole point of the enum: the platform currently satisfies `intel-quicksync` with a /dev/dri hostPath volume and a matching mount on the app container, but a later migration to the Intel GPU device plugin (`gpu.intel.com/i915` resource requests, no hostPath) must be a catalog/render.py change and NOTHING else - no ServiceDefinition may need editing, and no reviewer may have to re-derive per-service device wiring. Do NOT 'simplify' this into a raw device path, a hostPath, or a resource request; that would push a platform implementation detail into the operator-facing contract and make the migration a catalog-wide edit. Absent => no device wiring at all (existing apps render byte-identical). Ignored for kind=vm (a VM gets devices through KubeVirt's own domain.devices).",