feat(pihole): declare the admin password and adlists in git

Prerequisite for running more than one PiHole replica. A second replica with a fresh PVC would have come up serving DNS and blocking nothing, with a different admin password -- answers varying by which pod replied. pihole.toml's own ### CHANGED markers gave the exact inventory: four settings already came from env (declared), one came from the image, and exactly ONE operator-set value lived only on the PVC -- webserver.api.pwhash, a one-way hash nobody could recover. Separately the adlists were undeclared: one list and 72,234 gravity domains that nothing in this repo named. Declares both: - A SOPS secret feeding FTLCONF_webserver_api_password. webserver.api.password takes cleartext and FTL hashes it itself (verified: pihole setpassword IS setFTLConfigValue on that key), so the PVC's pwhash becomes derived state. - A level-triggered postStart hook that reconciles the declared adlist set into gravity.db -- adds, re-enables, and disables (never deletes) undeclared lists, running pihole -g only when something moved. A file would not work: v6's adlists.list import is migration-only, guarded on the DB not existing. The rendered hook was run in the live pod against throwaway copies of gravity.db, exercising all four paths. A fresh DB built from the image's own schema template has ZERO adlists, confirming this is load-bearing rather than cosmetic. Password kept rather than set empty: the pihole Service still publishes port 80 past the oauth2-proxy, closed only by policy, and default-deny has drifted here before. Multi-replica is phase 2 and deliberately not in this branch -- it needs a schema field, a StatefulSet (the config PVC is RWO), anti-affinity and a PDB. Adds a *.sops.yaml -- a sensitive path -- so approval-policy will block pending an operator /approve-override. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Merge pull request 'feat(plex): expose tcp/32400 for remote access via a pinned LoadBalancer VIP' (#43 ) from feat/plex-remote-access-l4 into main
2026-10-04 01:07:30 -05:00 · 2026-10-04 04:46:34 +00:00 · 2026-10-03 23:30:46 -05:00 · 2026-10-03 22:46:38 +00:00 · 2026-10-03 17:39:37 -05:00 · 2026-10-03 22:25:13 +00:00
32 changed files with 1818 additions and 189 deletions
@@ -40,13 +40,21 @@ missing credential is a lock-out, here a skip would be a bypass.
 **The administrator override (2026-10-01).** The rule above was unsatisfiable for the one human
 who runs this repo: Gitea refuses to let an author approve their own PR, and `jakman85` is the
 only collaborator — so a sensitive-path PR the operator opened could never be merged, *including
-the PR that would fix that.* An administrator may therefore waive the review by commenting on
+the PR that would fix that.* An administrator may therefore waive the review by posting this on
 the PR:

    /approve-override <reason>

-The directive must **start a line**, the **reason is mandatory**, the commenter's `admin`/`owner`
-permission is re-checked **live** when the gate runs, and the comment must be **newer than the
+**Post it as a PR *review* of type "Comment", not in the plain comment box.** CI's
+`SETTINGS_API_TOKEN` is minted `[read:repository, read:admin]`, and `/issues/{n}/comments` sits
+behind Gitea's *issue* scope — so an override typed into the comment box is **invisible to CI**,
+and the stage reports it as a missing approval rather than as a permission problem (measured on
+PR #38, 2026-10-02). `/pulls/{n}/reviews` is readable, so a review body works today. Both sources
+are accepted and the gate names which it used; the comment box starts working once
+`infra/gitea/set-ci-token.sh` is re-run to pick up `read:issue`.
+
+The directive must **start a line**, the **reason is mandatory**, the poster's `admin`/`owner`
+permission is re-checked **live** when the gate runs, and it must be **newer than the
 head commit** — push again and you must override again. The overrider, the timestamp and the
 reason are printed in the gate log and the job summary. For an administrator, "a change to the
 gate cannot be gated by the gate" is therefore **advisory, not structural**; what is kept is an
@@ -14,10 +14,12 @@ resources:
 - nextcloud-db-backup-credentials-prod.sops.yaml
 - nextcloud-db-backup-credentials.sops.yaml
 - nextcloud-db.yaml
+- pihole-bootstrap-secret.sops.yaml
 - pihole-dns.yaml
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
 - plex-claim-secret.sops.yaml
+- plex-remote.yaml
 - plex.yaml
 - qbittorrent-bootstrap-secret.sops.yaml
 - qbittorrent-oauth2-proxy-credentials.sops.yaml
@@ -0,0 +1,72 @@
+# =================================================================================================
+# PiHole WEB/API ADMIN PASSWORD — the one piece of PiHole state that was NOT declared anywhere.
+# =================================================================================================
+# WHY THIS FILE EXISTS. Measured on live prod 2026-10-04, `pihole.toml` marks every non-default
+# setting with `### CHANGED`, and names the ones that came from the environment `### CHANGED (env)`.
+# The complete list was:
+#
+#     dns.upstreams          ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     dns.listeningMode      ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     dns.rateLimit.count    ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     misc.dnsmasq_lines     ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     files.macvendor        ### CHANGED         <- set by the image itself, container path
+#     webserver.api.pwhash   ### CHANGED         <- SET OUT OF BAND. Declared NOWHERE. This file.
+#
+# So exactly ONE operator-set value lived only on the Longhorn PVC: the admin password, as a
+# one-way BALLOON-SHA256 hash. Nobody could recover it (a hash does not reverse), nothing in git
+# produced it, and a rebuild from git would have come up with a DIFFERENT password — or none.
+#
+# THE REASON THIS BLOCKS HA, which is why it is fixed first. The next step is running more than one
+# PiHole replica, and a replica with a fresh PVC would come up with its own unrelated password. The
+# admin UI would then accept or reject the same credential depending on which pod answered. Declaring
+# it here makes every replica, and every rebuild, identical by construction.
+#
+# MECHANISM: `webserver.api.password` is a real, settable FTL config key — verified in the running
+# image, where `pihole setpassword` is itself just `setFTLConfigValue "webserver.api.password"` with
+# the comment "pihole-FTL will automatically hash the password". So FTL takes CLEARTEXT on this key
+# and writes the hash into `webserver.api.pwhash` itself. catalog/pihole.yaml therefore passes this
+# value as `FTLCONF_webserver_api_password`, and the hash on the PVC becomes a derived artefact
+# rather than the source of truth.
+#
+# DEFENCE IN DEPTH, NOT THE ONLY GATE. The admin UI is already behind the oauth2-proxy sidecar
+# (`--allowed-group=platform-admins`), so this password is a SECOND gate. It is kept rather than set
+# empty because the `pihole` Service still publishes port 80 (PiHole direct, bypassing the proxy on
+# :4180) and only the CiliumNetworkPolicy closes it — and the default-deny policy in this repo has a
+# documented drift history. An unauthenticated admin UI that can rewrite LAN DNS is not a good thing
+# to leave resting on one policy staying correct. Narrowing that Service port is tracked follow-up.
+#
+# ROTATION: `sops` this file in your own terminal, change the value, commit. Flux re-applies the
+# Secret, the pod restarts, FTL re-hashes. Do NOT `sops -d` it into a shared terminal or a log
+# (CLAUDE.md section 6); read it in an editor. The value below was generated randomly at authoring
+# time and has never been printed to a transcript.
+apiVersion: v1
+kind: Secret
+metadata:
+    name: pihole-bootstrap
+    namespace: pihole
+    labels:
+        app.kubernetes.io/name: pihole
+type: Opaque
+stringData:
+    #ENC[AES256_GCM,data:TMOuRyJeb5IVn3lZPXY7lgh1KGxu7GY9+WIOLHpMB50JqzmM2tHIaWUWKTt1JtZoX/E5/Y7WCUTnfkJLhRBN2zV2MwWRBSLYVy5CMg7xV8PTLRZJYAVUYS5Trg==,iv:q+DpfT6PQfY5ZuNsmP7Ddl+PDLjSm89fJICAhYoW64k=,tag:axijaO3DaxIZgZ3LeK2hXA==,type:comment]
+    webui-password: ENC[AES256_GCM,data:7FcK/bUW79M46f/MlouAmATKbMnoUTNuMUjtqJ5vpU4=,iv:I8DBFB39uXZV42Ie2IPwSefmMgdKEtiDknfc9Tu7h5E=,tag:L423784UrDU3TvylpHc6KQ==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0MkRSNjZLZlpXYkcvaVJL
+            cjlqb2wrcWREZnZLdGpsTHNPWEpseHlRZnpjCnJVM1FDQ2M2dXlaS1JBZ2k2V1BP
+            Yi9QUDZSMzZsL09pTi93RTdjdGxtY2MKLS0tIDIrQTZIVWdhMGIxSCtlUFN2Mlho
+            SWVybld4VjJyaHRSKzZZU0xsUUV2SEEKbDYHV/qBZVspK7DpBW3vYjRMJ/HmAN+9
+            G9bOlNEp0jAtcNZEn6lpQ9ORgvmIrhJHoBQRLabdMWzNWSZsNn56tA==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-10-04T06:01:27Z"
+    mac: ENC[AES256_GCM,data:rhTMTOwLOtHXHgqI5M2ZW/6w+UpLpxBlTk7NoN0wmGSqOcE+OGrXA1DhkZrJweyQT0RSNSSBm/6QMztLP81eepOzdp2rEkwvbQPK/l6K9g16Ww9c1IatS3NQsKEB8AoMDx7jBQ6exXYgggPx3MJsNYbE+WyloQ96U9KPdrecAM8=,iv:n34v2zX9D5xsaR1PFCH94mNMiuRfhUyI2lseUCoqiTA=,tag:9YmN9nIVLXfsGYIb0bUl2w==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4
@@ -90,9 +90,21 @@ spec:
          value: '0'
        - name: FTLCONF_misc_dnsmasq_lines
          value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}
+        - name: FTLCONF_webserver_api_password
+          valueFrom:
+            secretKeyRef:
+              name: pihole-bootstrap
+              key: webui-password
        volumeMounts:
        - name: config
          mountPath: /data/config
+        lifecycle:
+          postStart:
+            exec:
+              command:
+              - /bin/bash
+              - -c
+              - "# Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and\n# does NOT run a gravity rebuild.\nset -u\nGRAVITY=/etc/pihole/gravity.db\n\n# THE DECLARED SET \u2014 one URL per line, the source of truth for what PiHole blocks.\n# To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.\nDECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'\n\n# On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,\n# and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.\ni=0\nwhile [ ! -s \"$GRAVITY\" ] && [ \"$i\" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done\nif [ ! -s \"$GRAVITY\" ]; then\n  echo \"pihole-adlists: gravity.db absent after 180s - leaving adlists untouched\" >&2\n  exit 0\nfi\n\nsql() { pihole-FTL sqlite3 \"$GRAVITY\" \"$1\"; }\nCHANGED=0\n\n# 1) Every declared URL must be present AND enabled.\necho \"$DECLARED\" | while IFS= read -r url; do\n  [ -n \"$url\" ] || continue\n  printf '%s\\n' \"$url\"\ndone > /tmp/declared.txt\nwhile IFS= read -r url; do\n  [ -n \"$url\" ] || continue\n  # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.\n  have=$(sql \"select count(*) from adlist where address='$url';\")\n  if [ \"$have\" = \"0\" ]; then\n    echo \"pihole-adlists: adding $url\"\n    sql \"insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');\"\n    CHANGED=1\n  else\n    en=$(sql \"select enabled from adlist where address='$url';\")\n    if [ \"$en\" != \"1\" ]; then\n      echo \"pihole-adlists: re-enabling $url\"\n      sql \"update adlist set enabled=1 where address='$url';\"\n      CHANGED=1\n    fi\n  fi\ndone < /tmp/declared.txt\n\n# 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).\nsql \"select address from adlist where enabled=1;\" | while IFS= read -r addr; do\n  [ -n \"$addr\" ] || continue\n  if ! grep -qxF \"$addr\" /tmp/declared.txt; then\n    echo \"pihole-adlists: disabling undeclared $addr\"\n    sql \"update adlist set enabled=0 where address='$addr';\"\n    echo changed >> /tmp/adlist-changed\n  fi\ndone\n[ -s /tmp/adlist-changed ] && CHANGED=1\nrm -f /tmp/adlist-changed /tmp/declared.txt\n\n# 3) Rebuild gravity ONLY if the declared set actually moved.\nif [ \"$CHANGED\" = \"1\" ]; then\n  echo \"pihole-adlists: declared set changed - running gravity update\"\n  pihole -g || echo \"pihole-adlists: gravity update failed; existing gravity.db retained\" >&2\nelse\n  echo \"pihole-adlists: already matches the declared set - nothing to do\"\nfi\nexit 0\n"
      - name: oauth2-proxy
        image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
        imagePullPolicy: IfNotPresent
@@ -0,0 +1,126 @@
+---
+# =================================================================================================
+# PLEX REMOTE ACCESS — raw tcp/32400 north-south exposure, and the policy that admits it.
+# =================================================================================================
+# HAND-WRITTEN, NOT RENDERED. A non-catalog app-tier unit anchored to the `plex` ServiceDefinition
+# and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS table — the same
+# mechanism gitea-actions and pihole-dns use. It is hand-written for the SAME reason those are:
+# schemas/servicedefinition.schema.json is `additionalProperties: false` and has NO field for a
+# type=LoadBalancer Service, for a pinned lbipam VIP, or for raw-TCP north-south exposure at all.
+# `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator.
+#
+# WHY THE L7 GATEWAY IS NOT ENOUGH HERE — and it is NOT because the Gateway fails to serve Plex.
+# Measured 2026-10-03 through the Gateway VIP: `GET https://plex.home.keithjackson.dev/identity`
+# returned HTTP 200 with `claimed="1"` and the real machineIdentifier, and the protected endpoints
+# (/library/sections, /status/sessions, /:/prefs) all returned 401. The L7 path WORKS.
+#
+# The problem is DISCOVERY, not proxying. Plex apps (phone, TV, Roku) do not resolve
+# `plex.${BASE_DOMAIN}`; they ask plex.tv for the server's connection list. A server gets onto that
+# list one of two ways: it advertises a custom URL (`customConnections`, the ADVERTISE_IP env), or it
+# publishes a reachable public IP:port for its own tcp/32400. Measured in the live Preferences.xml on
+# 2026-10-03: `customConnections` ABSENT, `ManualPortMappingMode` ABSENT, while
+# `PublishServerOnPlexOnlineKey="1"` — the server was trying to publish and had nothing reachable to
+# publish, so remote clients saw NO connection at all. The operator chose the NATIVE path (this
+# file) over advertising the Gateway URL: Plex then terminates its own TLS with its
+# `*.plex.direct` certificate and video streams bypass the Envoy L7 hop entirely.
+#
+# VIP: ${PLEX_L4_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.57, prod 192.168.2.24), PINNED via
+# lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
+# (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the existing
+# CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config is needed;
+# this Service just consumes the pool. PINNING IS NOT OPTIONAL: the address is typed into a WAN
+# port-forward on the SOHO router, which is hand-maintained and off-repo, so it can never move.
+# 192.168.2.24 verified FREE live 2026-10-03 — `default-pool` reported IPsTotal 11 / IPsUsed 4 /
+# IPsAvailable 7, and the only four allocations were .20 (cilium-gateway-shared), .21 (unifi-l4-*),
+# .22 (crafty-l4-*) and .23 (pihole-dns-*).
+#
+# ONE SERVICE, TCP ONLY. Unlike unifi/crafty/pihole-dns this needs no second Service and no
+# lbipam.cilium.io/sharing-key: Plex's remote access is tcp/32400 and nothing else. (Plex's
+# DLNA/GDM discovery uses UDP, but that is LAN-only multicast and is not part of remote access.)
+#
+# NO `loadBalancerSourceRanges`, DELIBERATELY — and this is the one place this file diverges from
+# its closest sibling. apps/wave-0/pihole-dns.yaml pins its source range to ${LAN_CIDR} because a
+# resolver must never answer the internet. This Service is the OPPOSITE case: admitting remote
+# clients from arbitrary internet addresses is its entire purpose, so there is no range to narrow
+# to. Plex authenticates every caller itself (measured above: 401 on every protected endpoint
+# without a token), which is what makes an unfiltered port defensible here.
+#
+# NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
+# `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their VIP as a public
+# Cloudflare record. This Service must not, for pihole-dns's reason: ${PLEX_L4_VIP} is an RFC1918
+# address that is useless in public DNS. Remote clients get this server's address from plex.tv, not
+# from the zone. The existing public `plex.${BASE_DOMAIN}` record is published from the rendered
+# HTTPRoute via the gateway source and is NOT affected by this file — the web UI keeps working
+# through the Gateway exactly as before.
+#
+# -------------------------------------------------------------------------------------------------
+# externalTrafficPolicy: Cluster — inherited from crafty's 8-DAY LAN OUTAGE, not re-derived.
+# -------------------------------------------------------------------------------------------------
+# `Local` is the tempting choice because it preserves the client source IP, which this service
+# actually has a use for (see the allowedNetworks note below). It is still WRONG here, for the
+# reason platform/crafty/service-l4.yaml documents at length: Cilium's L2 announcer elects ONE lease
+# holder per Service and does NOT follow the backend pod, so with `Local` the VIP can be announced
+# by a node that has no Plex pod, which black-holes the port while every object still reports
+# healthy. Plex is a single-replica Deployment with `strategy: Recreate` pinned to wherever the
+# scheduler put it, so the mismatch is the normal case, not a rare one. Take the SNAT.
+#
+# KNOWN CONSEQUENCE OF THAT CHOICE, stated rather than discovered later: `Cluster` SNATs the client
+# to the ingress NODE's address (192.168.2.1x), which falls inside the `192.168.0.0/16` entry of
+# catalog/plex.yaml's ALLOWED_NETWORKS. Plex will therefore classify every REMOTE client as LAN.
+# Measured 2026-10-03 on the Gateway path, which SNATs the same way: Plex logged the request as
+# `[10.0.2.214:36040 (Allowed Network (WAN))]`. This is a METERING effect, not a security one —
+# authorization is unchanged (401s above), but Plex's remote-quality/bandwidth limits will not bind
+# on these streams because it believes they are local. Narrowing ALLOWED_NETWORKS cannot fix it
+# without also de-classifying genuine LAN clients, so it is accepted.
+apiVersion: v1
+kind: Service
+metadata:
+  name: plex-l4-tcp
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+  annotations:
+    lbipam.cilium.io/ips: "${PLEX_L4_VIP}"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Cluster
+  selector:
+    app.kubernetes.io/name: plex
+  ports:
+    - name: plex-remote
+      port: 32400
+      targetPort: 32400
+      protocol: TCP
+---
+# INGRESS allow for the raw port (R-NET-01/02).
+#
+# The rendered plex.yaml already carries `plex-allow-gateway-ingress` (fromEntities: ingress ->
+# 32400) and three reciprocal allows for sonarr/radarr/seerr. CiliumNetworkPolicies are ADDITIVE, so
+# this file adds the off-cluster case without touching those.
+#
+# `world` is Cilium's entity for anything outside the cluster, which is what a remote Plex client is.
+# `host` and `cluster` mirror platform/crafty/networkpolicy.yaml's game-port rule and are what make
+# this work under `externalTrafficPolicy: Cluster`: after SNAT the packet's source is the ingress
+# NODE, which Cilium classifies as `host` (same node) or `remote-node` (a different one) — NOT
+# `world`. Listing `world` alone would compile cleanly and then drop every remote stream, so all
+# four entities are named.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: plex-allow-remote-32400
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: plex
+  ingress:
+    - fromEntities:
+        - world
+        - cluster
+        - host
+        - remote-node
+      toPorts:
+        - ports:
+            - { port: "32400", protocol: TCP }
@@ -100,6 +100,20 @@ env:
  # while resolution for the zone does not work. Confirmed both ways against the live image.
  - name: FTLCONF_misc_dnsmasq_lines
    value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}"
+  # THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
+  # apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
+  # carries the full rationale and the measurement that found it.
+  #
+  # `webserver.api.password` takes CLEARTEXT and FTL hashes it itself into `webserver.api.pwhash`.
+  # Verified in the running image, not assumed: `pihole setpassword` is itself
+  # `setFTLConfigValue "webserver.api.password" "$PASSWORD"`, commented upstream as "pihole-FTL will
+  # automatically hash the password". So the BALLOON-SHA256 hash on the PVC is now a DERIVED artefact
+  # and this is its source of truth. Rotating the Secret rotates the password.
+  - name: FTLCONF_webserver_api_password
+    valueFrom:
+      secretKeyRef:
+        name: pihole-bootstrap
+        key: webui-password
 ingressRoutes:
  - host: pihole.${BASE_DOMAIN}
    port: 80                  # admin web UI, protected at the Gateway by forward-auth (docs/09)
@@ -114,6 +128,110 @@ auth:
  # rather than a renderer constant: the measurement that justifies the flag on two apps is
  # evidence about those two apps. Its oauth2-proxy log holds 0 unauthenticated requests across
  # the current pod's whole 23h lifetime (measured 2026-10-01), so there is nothing to act on.
+lifecycle:
+  # LEVEL-TRIGGERED ADLIST ASSERTION, run on EVERY container start. Same pattern, and the same
+  # reasoning, as catalog/qbittorrent.yaml's postStart: desired state lives in git and is
+  # RE-ASSERTED unconditionally rather than set once by hand.
+  #
+  # WHY THIS IS NEEDED AT ALL — the blocklists were NOT reproducible from git. Measured on live prod
+  # 2026-10-04: `gravity.db` held exactly ONE adlist (StevenBlack's hosts, enabled) and 72,234
+  # gravity domains, and NOTHING in this repo named it. Worse, a FRESH database does not seed it:
+  # the image's own schema template `/etc/.pihole/advanced/Templates/gravity.db.sql` creates the
+  # `adlist` table and inserts NO rows into it (grepped in the running image — there is no
+  # `INSERT INTO adlist` at all). So a rebuild from git, or a replica with a new PVC, would come up
+  # SERVING DNS AND BLOCKING NOTHING, silently and with every object reporting healthy.
+  #
+  # WHY A postStart HOOK AND NOT A FILE. PiHole v6 keeps adlists as DB rows, not config. The
+  # `/etc/pihole/adlists.list` file still exists in `gravity.sh` but is MIGRATION-ONLY: its import
+  # sits inside `migrate_to_database()`, guarded by `if [ ! -e "$gravityDBfile" ]`, so it fires only
+  # while CREATING a database and never again on an existing one. A ConfigMap mounted there would
+  # therefore be silently ignored on every pod this cluster currently runs. There is also no
+  # `pihole adlist` CLI subcommand (only allow/deny for domains), so SQL against the DB is the
+  # actual interface. FTL ships an embedded shell (`pihole-FTL sqlite3`, SQLite 3.53.1) so no extra
+  # tooling is needed in the image.
+  #
+  # IT RECONCILES, IT DOES NOT JUST ADD: adlists present in the DB but absent from DECLARED below are
+  # DISABLED (not deleted — deleting cascades to `gravity` rows and would force a full re-download of
+  # everything on a typo). So this list is the source of truth in both directions, and `pihole -g`
+  # runs ONLY when something actually changed, which keeps the steady-state start free of a 70k-domain
+  # rebuild.
+  #
+  # NO BRACED DOLLAR EXPANSION ANYWHERE IN THIS SCRIPT, DELIBERATELY — the same contract
+  # catalog/qbittorrent.yaml documents at length. apps-wave-0 declares postBuild.substituteFrom
+  # cluster-vars, so Flux runs envsubst over this whole manifest at reconcile time; a `${VAR}` form
+  # would behave differently on the cluster than under a plain `kubectl apply` of the golden. Bare
+  # `$VAR` and `$(cmd)` are untouched by envsubst and so behave identically however this is run.
+  # `render.py`'s assert_substitution_safe refuses to emit a manifest that breaks this.
+  postStart:
+    exec:
+      command:
+        - /bin/bash
+        - -c
+        - |
+          # Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and
+          # does NOT run a gravity rebuild.
+          set -u
+          GRAVITY=/etc/pihole/gravity.db
+
+          # THE DECLARED SET — one URL per line, the source of truth for what PiHole blocks.
+          # To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.
+          DECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'
+
+          # On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,
+          # and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.
+          i=0
+          while [ ! -s "$GRAVITY" ] && [ "$i" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done
+          if [ ! -s "$GRAVITY" ]; then
+            echo "pihole-adlists: gravity.db absent after 180s - leaving adlists untouched" >&2
+            exit 0
+          fi
+
+          sql() { pihole-FTL sqlite3 "$GRAVITY" "$1"; }
+          CHANGED=0
+
+          # 1) Every declared URL must be present AND enabled.
+          echo "$DECLARED" | while IFS= read -r url; do
+            [ -n "$url" ] || continue
+            printf '%s\n' "$url"
+          done > /tmp/declared.txt
+          while IFS= read -r url; do
+            [ -n "$url" ] || continue
+            # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.
+            have=$(sql "select count(*) from adlist where address='$url';")
+            if [ "$have" = "0" ]; then
+              echo "pihole-adlists: adding $url"
+              sql "insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');"
+              CHANGED=1
+            else
+              en=$(sql "select enabled from adlist where address='$url';")
+              if [ "$en" != "1" ]; then
+                echo "pihole-adlists: re-enabling $url"
+                sql "update adlist set enabled=1 where address='$url';"
+                CHANGED=1
+              fi
+            fi
+          done < /tmp/declared.txt
+
+          # 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).
+          sql "select address from adlist where enabled=1;" | while IFS= read -r addr; do
+            [ -n "$addr" ] || continue
+            if ! grep -qxF "$addr" /tmp/declared.txt; then
+              echo "pihole-adlists: disabling undeclared $addr"
+              sql "update adlist set enabled=0 where address='$addr';"
+              echo changed >> /tmp/adlist-changed
+            fi
+          done
+          [ -s /tmp/adlist-changed ] && CHANGED=1
+          rm -f /tmp/adlist-changed /tmp/declared.txt
+
+          # 3) Rebuild gravity ONLY if the declared set actually moved.
+          if [ "$CHANGED" = "1" ]; then
+            echo "pihole-adlists: declared set changed - running gravity update"
+            pihole -g || echo "pihole-adlists: gravity update failed; existing gravity.db retained" >&2
+          else
+            echo "pihole-adlists: already matches the declared set - nothing to do"
+          fi
+          exit 0
 update:
  channel: stable             # R-UPD-01
  range: ">=2024.0.0"
@@ -2002,6 +2002,24 @@ EXTRA_UNITS: dict = {
            "pihole-dns.yaml",
        ],
    },
+    # Plex's remote-access exposure: the raw tcp/32400 LoadBalancer Service on a pinned VIP, plus the
+    # CiliumNetworkPolicy that admits off-cluster clients to it (apps/wave-0/plex-remote.yaml).
+    # `above: False` co-locates it in plex's OWN wave — these are the plex pod's own Service and its
+    # own policy, applied in the same wave as the Deployment they select, exactly as the rendered
+    # Service and CNPs in plex.yaml are. Nothing in a higher wave gates on it.
+    #
+    # It is a non-catalog unit for the same reason pihole-dns is: the ServiceDefinition schema cannot
+    # express a LoadBalancer Service or a pinned lbipam VIP (additionalProperties is false, and
+    # `ingressRoutes` only renders HTTPRoutes on the L7 Gateway, which cannot carry Plex's raw port).
+    # Plex's own settings stay in the catalog as env on catalog/plex.yaml; the only thing living
+    # outside it is the exposure the schema has no word for.
+    "plex-remote": {
+        "anchor": "plex",
+        "above": False,
+        "files": [
+            "plex-remote.yaml",
+        ],
+    },
 }


@@ -1,39 +1,54 @@
 #!/usr/bin/env python3
-"""ci/approval-override.py — select the admin-override comment on a PR, if there is a valid one.
+"""ci/approval-override.py — select the admin-override on a PR, if there is a valid one.

-This script decides NOTHING about authority. It parses a PR's comment list and returns the
-single override *candidate* that ci/approval-policy.sh should then try to authorise against
+This script decides NOTHING about authority. It parses a PR's comments and reviews and returns
+the single override *candidate* that ci/approval-policy.sh should then try to authorise against
 the live Gitea permission API. Keeping the two apart is the point:

  * parsing is pure, offline, and therefore TESTABLE (ci/tests/approval-override.test.sh);
  * authorisation needs the API and must stay in the stage that fails closed.

-A candidate is a comment whose body contains, AT THE START OF A LINE, the directive followed
-by a non-empty reason:
+A candidate is a comment OR a review whose body contains, AT THE START OF A LINE, the directive
+followed by a non-empty reason:

    /approve-override the gate cannot gate itself and I am the only human reviewer

+TWO SOURCES, AND WHY THAT IS NOT BELT-AND-BRACES. Measured 2026-10-02 on PR #38: the override was
+posted as an ordinary PR comment, was well-formed, was newer than the head commit — and the stage
+still blocked, because CI could not READ it. `SETTINGS_API_TOKEN` is minted with scopes
+`[read:repository, read:admin]`, and Gitea puts /issues/{n}/comments behind the *issue* scope, so
+the request came back as an error OBJECT rather than an array. `/pulls/{n}/reviews` is under
+read:repository and was readable in the same run. So:
+
+  * a REVIEW body works with the token CI has today (leave a review of type "Comment");
+  * a COMMENT body works once the token carries read:issue (see infra/gitea/set-ci-token.sh).
+
+Accepting both means the hatch is not hostage to one API scope. A source that cannot be read
+contributes no candidates and is reported on stderr; it can only cause a false BLOCK, never a
+false pass, which is why degrading it is safe and blocking on it was not necessary.
+
 WHY EACH RULE IS HERE

  * START OF A LINE, not "anywhere in the body". Otherwise quoting someone else's override
-    ("you could just /approve-override this") re-arms it. Markdown quotes are stripped of a
-    leading '>' before matching for the same reason -- a quoted directive must NOT count, so
-    lines beginning with '>' are DROPPED, not unquoted.
-  * A REASON IS MANDATORY. A bare directive records that a review was waived but not why,
-    which is the difference between an audit trail and a rubber stamp.
-  * STALE OVERRIDES ARE DISCARDED. An override earned at commit A must not silently cover
-    commit B. Any candidate created at or before the head commit's timestamp is dropped --
-    the same semantics Gitea's own `dismiss_stale_approvals` applies to reviews. Push new
-    commits and you must re-override.
-  * THE LATEST CANDIDATE WINS, so a second override comment supersedes the first rather than
-    both being live.
+    ("you could just /approve-override this") re-arms it. Lines beginning with '>' are DROPPED
+    rather than unquoted, for exactly that reason.
+  * A REASON IS MANDATORY. A bare directive records that a review was waived but not why, which
+    is the difference between an audit trail and a rubber stamp.
+  * STALE OVERRIDES ARE DISCARDED. An override earned at commit A must not silently cover commit
+    B. Any candidate created at or before the head commit's timestamp is dropped — the same
+    semantics Gitea's own `dismiss_stale_approvals` applies to approvals. Push new commits and
+    you must re-override.
+  * A PENDING REVIEW IS NOT SUBMITTED. It is a private draft visible only to its author, so it is
+    not an act of anything and is dropped.
+  * THE LATEST CANDIDATE WINS, across both sources, so a later override supersedes an earlier one.

-Output: one TSV line (login, created_at, reason) on stdout, or nothing at all when there is no
-valid candidate. Never exits non-zero for "no override" -- absence is a normal answer, and the
+Output: one TSV line (login, timestamp, source, reason) on stdout, or nothing at all when there is
+no valid candidate. Never exits non-zero for "no override" — absence is a normal answer and the
 caller must not be able to confuse it with a parse crash.

 Input, all via env so nothing lands in `ps`:
-  COMMENTS_JSON  the Gitea /issues/{n}/comments array
+  COMMENTS_JSON  the Gitea /issues/{n}/comments array (or an error object; or absent)
+  REVIEWS_JSON   the Gitea /pulls/{n}/reviews array   (or an error object; or absent)
  HEAD_TIME      ISO-8601 timestamp of the PR head commit
  DIRECTIVE      the override directive, e.g. /approve-override
 """
@@ -81,57 +96,97 @@ def extract_reason(body, directive):
    return None


-def main():
-    directive = os.environ.get("DIRECTIVE", "/approve-override").strip()
-    head_time = parse_ts(os.environ.get("HEAD_TIME", ""))
-
+def load_source(env_name, label):
+    """Return a list of items for one source, or [] with a stderr note if it is unusable."""
+    raw = os.environ.get(env_name)
+    if not raw or not raw.strip():
+        return []
    try:
-        comments = json.loads(os.environ.get("COMMENTS_JSON") or "[]")
+        parsed = json.loads(raw)
    except (ValueError, TypeError):
-        # The caller validated this JSON already; if it is broken here, say so loudly on
-        # stderr and emit no candidate. Silence on stdout means "no override".
-        print("approval-override: COMMENTS_JSON is not valid JSON", file=sys.stderr)
-        return 0
-    if not isinstance(comments, list):
-        print("approval-override: COMMENTS_JSON is not a list", file=sys.stderr)
-        return 0
+        print(
+            "approval-override: %s is not valid JSON — ignoring that source" % label,
+            file=sys.stderr,
+        )
+        return []
+    if isinstance(parsed, list):
+        return parsed
+    # An error object, typically {"message": "token does not have ... scope"}. Say which,
+    # because "no override found" and "could not read the overrides" look identical from
+    # the outside and the first wasted a real debugging session.
+    detail = ""
+    if isinstance(parsed, dict):
+        detail = str(parsed.get("message") or parsed.get("errors") or "")[:200]
+    print(
+        "approval-override: %s is not a list (Gitea returned %s%s) — ignoring that source"
+        % (label, type(parsed).__name__, ": " + detail if detail else ""),
+        file=sys.stderr,
+    )
+    return []

-    candidates = []
-    for comment in comments:
-        if not isinstance(comment, dict):
+
+def candidates_from(items, source, directive, head_time):
+    """Yield (when, login, reason, raw_when, source) for every valid override in one source."""
+    found = []
+    for item in items:
+        if not isinstance(item, dict):
            continue
-        login = ((comment.get("user") or {}).get("login") or "").strip()
-        reason = extract_reason(comment.get("body"), directive)
+        login = ((item.get("user") or {}).get("login") or "").strip()
+        reason = extract_reason(item.get("body"), directive)
        if not login or not reason:
            continue
-        created_raw = comment.get("created_at") or comment.get("updated_at") or ""
-        created = parse_ts(created_raw)
-        if created is None:
+        # A PENDING review is an unsubmitted private draft — not an act of anything.
+        state = str(item.get("state") or "").upper()
+        if source == "review" and state == "PENDING":
            print(
-                "approval-override: dropping override by '%s' — unreadable timestamp %r"
-                % (login, created_raw),
+                "approval-override: dropping PENDING (unsubmitted) review by '%s'" % login,
+                file=sys.stderr,
+            )
+            continue
+        raw_when = (
+            item.get("submitted_at") or item.get("created_at") or item.get("updated_at") or ""
+        )
+        when = parse_ts(raw_when)
+        if when is None:
+            print(
+                "approval-override: dropping %s override by '%s' — unreadable timestamp %r"
+                % (source, login, raw_when),
                file=sys.stderr,
            )
            continue
        # An override must post-date the commit it covers.
-        if head_time is not None and created <= head_time:
+        if head_time is not None and when <= head_time:
            print(
-                "approval-override: dropping STALE override by '%s' (%s) — not newer than "
-                "head commit (%s). Re-comment to override the current commit."
-                % (login, created.isoformat(), head_time.isoformat()),
+                "approval-override: dropping STALE %s override by '%s' (%s) — not newer than "
+                "head commit (%s). Re-post it to override the current commit."
+                % (source, login, when.isoformat(), head_time.isoformat()),
                file=sys.stderr,
            )
            continue
-        candidates.append((created, login, reason, created_raw))
+        found.append((when, login, reason, raw_when, source))
+    return found

-    if not candidates:
+
+def main():
+    directive = os.environ.get("DIRECTIVE", "/approve-override").strip()
+    head_time = parse_ts(os.environ.get("HEAD_TIME", ""))
+
+    found = []
+    found += candidates_from(
+        load_source("COMMENTS_JSON", "COMMENTS_JSON"), "comment", directive, head_time
+    )
+    found += candidates_from(
+        load_source("REVIEWS_JSON", "REVIEWS_JSON"), "review", directive, head_time
+    )
+
+    if not found:
        return 0

-    # Latest wins: a later override supersedes an earlier one.
-    created, login, reason, created_raw = max(candidates, key=lambda c: c[0])
+    # Latest wins, across both sources: a later override supersedes an earlier one.
+    when, login, reason, raw_when, source = max(found, key=lambda c: c[0])
    # Keep the reason on one line — this becomes a TSV field.
    reason = " ".join(reason.split())
-    print("%s\t%s\t%s" % (login, created_raw, reason))
+    print("%s\t%s\t%s\t%s" % (login, raw_when, source, reason))
    return 0


@@ -103,8 +103,15 @@ REVIEWS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/reviews
 # PR conversation comments live under the ISSUES endpoint. `/pulls/{n}/comments` returns review
 # (diff) comments ONLY, so an override posted in the normal comment box would never appear in it.
 # Verified against this Gitea (1.24.3) rather than assumed.
+#
+# This one is NOT in the must-parse loop below, and that is deliberate. /issues/{n}/comments sits
+# behind Gitea's *issue* token scope, which `SETTINGS_API_TOKEN` does not carry (it is minted
+# `[read:repository, read:admin]`) — so it returns an error OBJECT, not an array. Blocking on that
+# turned "CI cannot read your override" into a message that said "you did not post one", and cost a
+# real debugging session on PR #38. It is handled explicitly, and loudly, further down: an
+# unreadable source can only cause a false BLOCK, never a false pass.
 COMMENTS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/issues/${PR}/comments?limit=1000" || true)"
-for v in FILES_JSON PR_JSON REVIEWS_JSON COMMENTS_JSON; do
+for v in FILES_JSON PR_JSON REVIEWS_JSON; do
  jq -e . >/dev/null 2>&1 <<<"${!v}" || {
    err "could not read ${v} from Gitea for PR #${PR} — BLOCKING rather than assuming."
    summary "- approval-policy: **BLOCKED — Gitea unreadable**"
@@ -170,8 +177,23 @@ if [ "$OVERRIDE_ENABLED" = "true" ]; then
    exit 1
  fi

-  # Pure, offline, unit-tested selection: which comment, if any, is a live override?
-  OVERRIDE_TSV="$(COMMENTS_JSON="$COMMENTS_JSON" HEAD_TIME="$HEAD_TIME" DIRECTIVE="$DIRECTIVE" \
+  # Is the comments source actually readable? Say so plainly if not — a scope problem and "you
+  # never posted an override" are indistinguishable from the outside, and conflating them is what
+  # made PR #38 look like a code bug.
+  if ! jq -e 'type == "array"' >/dev/null 2>&1 <<<"${COMMENTS_JSON:-}"; then
+    SCOPE_MSG="$(jq -r '.message // empty' <<<"${COMMENTS_JSON:-null}" 2>/dev/null || true)"
+    warn "cannot read PR COMMENTS from Gitea${SCOPE_MSG:+ — ${SCOPE_MSG}}."
+    warn "  /issues/${PR}/comments needs the *issue* token scope; SETTINGS_API_TOKEN is minted"
+    warn "  [read:repository, read:admin] (infra/gitea/set-ci-token.sh). An override posted as an"
+    warn "  ordinary PR COMMENT is therefore invisible to CI — post it as a REVIEW of type"
+    warn "  'Comment' instead, which /pulls/${PR}/reviews exposes under read:repository."
+    summary "- approval-policy: note — PR comments unreadable (token lacks the issue scope); use a **review** to override"
+    COMMENTS_JSON='[]'
+  fi
+
+  # Pure, offline, unit-tested selection: which comment or review, if any, is a live override?
+  OVERRIDE_TSV="$(COMMENTS_JSON="$COMMENTS_JSON" REVIEWS_JSON="$REVIEWS_JSON" \
+    HEAD_TIME="$HEAD_TIME" DIRECTIVE="$DIRECTIVE" \
    python3 "${CI_DIR}/approval-override.py")" || {
      err "ci/approval-override.py failed — BLOCKING rather than assuming there is no override."
      summary "- approval-policy: **BLOCKED — override parser failed**"
@@ -180,7 +202,8 @@ if [ "$OVERRIDE_ENABLED" = "true" ]; then
  if [ -n "$OVERRIDE_TSV" ]; then
    OV_USER="$(cut -f1 <<<"$OVERRIDE_TSV")"
    OV_WHEN="$(cut -f2 <<<"$OVERRIDE_TSV")"
-    OV_WHY="$(cut -f3- <<<"$OVERRIDE_TSV")"
+    OV_SRC="$(cut -f3 <<<"$OVERRIDE_TSV")"
+    OV_WHY="$(cut -f4- <<<"$OVERRIDE_TSV")"

    # AUTHORITY IS MEASURED LIVE, never inferred from the comment or from a name in git.
    PERM_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/collaborators/${OV_USER}/permission" || true)"
@@ -214,12 +237,13 @@ print("yes" if (ok_perm and ok_list) else "no")
      else
        warn "ADMIN OVERRIDE by '${OV_USER}' (permission '${OV_PERM}'); author is '${AUTHOR}'."
      fi
+      warn "  via:    PR ${OV_SRC}"
      warn "  when:   ${OV_WHEN}  (head commit ${HEAD_SHA:0:8}, ${HEAD_TIME})"
      warn "  reason: ${OV_WHY}"
      warn "  review waived on: $(printf '%s ' $SENSITIVE)"
      ok "sensitive paths cleared by administrator override — this stage does not block."
      SELFNOTE=""; [ "$OV_USER" = "$AUTHOR" ] && SELFNOTE=", **self-waived as author**"
-      summary "- approval-policy: **OVERRIDDEN by \`${OV_USER}\`** (permission \`${OV_PERM}\`${SELFNOTE})"
+      summary "- approval-policy: **OVERRIDDEN by \`${OV_USER}\`** (permission \`${OV_PERM}\`, via PR ${OV_SRC}${SELFNOTE})"
      summary "  - reason: ${OV_WHY}"
      summary "  - review waived on: $(printf '\`%s\` ' $SENSITIVE)"
      exit 0
@@ -239,10 +263,12 @@ err "  honestly approve for itself. Get a human review, or move the change out o
 printf '    %s\n' $SENSITIVE >&2
 err "  The list lives in infra/gitea/approval-policy.json, with a reason per entry."
 if [ "$OVERRIDE_ENABLED" = "true" ]; then
-  err "  Or, if you hold admin/owner on this repository, comment on the PR:"
+  err "  Or, if you hold admin/owner on this repository, post this on the PR:"
  err "      ${DIRECTIVE} <reason>"
-  err "  The reason is mandatory, the directive must START a line, and the comment must be"
-  err "  NEWER than head commit ${HEAD_SHA:0:8} — push again and you must override again."
+  err "  as a REVIEW of type 'Comment' (not the plain comment box — CI's token cannot read"
+  err "  /issues/${PR}/comments without the issue scope; see infra/gitea/set-ci-token.sh)."
+  err "  The reason is mandatory, the directive must START a line, and it must be NEWER than"
+  err "  head commit ${HEAD_SHA:0:8} — push again and you must override again."
 fi
 summary "- approval-policy: **BLOCKED — needs a non-author approval, or an admin \`${DIRECTIVE}\` comment**"
 exit 1
@@ -26,7 +26,13 @@ BEFORE="2026-10-01T11:30:00Z"

 # run <comments-json> -> stdout of the parser (stderr is diagnostic only)
 run() {
-  COMMENTS_JSON="$1" HEAD_TIME="${2:-$HEAD_TIME}" DIRECTIVE="${3:-/approve-override}" \
+  COMMENTS_JSON="$1" REVIEWS_JSON="" HEAD_TIME="${2:-$HEAD_TIME}" DIRECTIVE="${3:-/approve-override}" \
+    python3 "$SUT" 2>/dev/null
+}
+
+# run_rev <reviews-json> — the same, but through the REVIEWS source.
+run_rev() {
+  COMMENTS_JSON="" REVIEWS_JSON="$1" HEAD_TIME="${2:-$HEAD_TIME}" DIRECTIVE="${3:-/approve-override}" \
    python3 "$SUT" 2>/dev/null
 }

@@ -54,10 +60,24 @@ expect_user() {
 expect_reason() {
  local label="$1" json="$2" want="$3" out got
  out="$(run "$json")"
-  got="$(cut -f3- <<<"$out")"
+  got="$(cut -f4- <<<"$out")"
  if [ "$got" = "$want" ]; then ok "$label"; else bad "$label (wanted '${want}', got '${got}')"; fi
 }

+# check <label> <actual> <wanted>
+check() {
+  if [ "$2" = "$3" ]; then ok "$1"; else bad "$1 (wanted '${3}', got '${2}')"; fi
+}
+
+# review <login> <submitted_at> <state> <body>
+review() {
+  python3 -c '
+import json, sys
+print(json.dumps([{"user": {"login": sys.argv[1]}, "submitted_at": sys.argv[2],
+                   "state": sys.argv[3], "body": sys.argv[4]}]))
+' "$1" "$2" "$3" "$4"
+}
+
 echo "== ci/approval-override.py"

 # --- the happy path -------------------------------------------------------------------
@@ -141,5 +161,72 @@ else
  bad "with no HEAD_TIME the candidate should still be returned (got '${out}')"
 fi

+# --- the REVIEWS source ---------------------------------------------------------------
+# Load-bearing, not redundant. Measured on PR #38: SETTINGS_API_TOKEN is minted
+# [read:repository, read:admin], /issues/{n}/comments needs the ISSUE scope, so a well-formed
+# override in the comment box came back as an error object and was invisible. /pulls/{n}/reviews
+# IS readable with that token, so a review body is the route that works today.
+check "a review body carries an override" \
+  "$(cut -f1 <<<"$(run_rev "$(review jakman85 "$AFTER" COMMENT '/approve-override via a review')")")" \
+  jakman85
+check "the source is reported as 'review'" \
+  "$(cut -f3 <<<"$(run_rev "$(review jakman85 "$AFTER" COMMENT '/approve-override via a review')")")" \
+  review
+check "the source is reported as 'comment' for a comment" \
+  "$(cut -f3 <<<"$(run "$(comment jakman85 "$AFTER" '/approve-override via a comment')")")" \
+  comment
+
+out="$(run_rev "$(review jakman85 "$BEFORE" COMMENT '/approve-override stale review')")"
+check "a stale review override is dropped too" "${out:-<none>}" "<none>"
+
+out="$(run_rev "$(review jakman85 "$AFTER" PENDING '/approve-override unsubmitted draft')")"
+check "a PENDING (unsubmitted) review is NOT an act of anything" "${out:-<none>}" "<none>"
+
+check "an APPROVED review carrying the directive still counts as an override" \
+  "$(cut -f1 <<<"$(run_rev "$(review jakman85 "$AFTER" APPROVED '/approve-override belt and braces')")")" \
+  jakman85
+
+out="$(run_rev "$(review jakman85 "$AFTER" COMMENT 'ordinary review prose, no directive')")"
+check "an ordinary review body is not an override" "${out:-<none>}" "<none>"
+
+# --- an unreadable source degrades to 'no candidates', never to a pass ----------------
+# The real-world shape: Gitea answers with {"message": "token does not have ... scope"}.
+ERR_OBJ='{"message":"token does not have at least one of required scope(s): [read:issue]"}'
+out="$(COMMENTS_JSON="$ERR_OBJ" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
+       DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "an error OBJECT from the comments API yields no override" "${out:-<none>}" "<none>"
+
+out="$(COMMENTS_JSON="$ERR_OBJ" \
+       REVIEWS_JSON="$(review jakman85 "$AFTER" COMMENT '/approve-override reviews still work')" \
+       HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "an unreadable COMMENTS source does not suppress a valid REVIEW override" \
+  "$(cut -f1 <<<"$out")" jakman85
+
+# and it must SAY so, or a scope problem reads as 'you never posted one' (it did, on PR #38)
+errout="$(COMMENTS_JSON="$ERR_OBJ" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
+          DIRECTIVE=/approve-override python3 "$SUT" 2>&1 >/dev/null)"
+case "$errout" in
+  *"not a list"*read:issue*) ok "the diagnostic names the scope Gitea complained about" ;;
+  *) bad "the diagnostic should quote Gitea's message (got: ${errout})" ;;
+esac
+
+# --- latest wins ACROSS the two sources ----------------------------------------------
+out="$(COMMENTS_JSON="$(comment early "2026-10-01T12:10:00Z" '/approve-override from a comment')" \
+       REVIEWS_JSON="$(review later "2026-10-01T12:50:00Z" COMMENT '/approve-override from a review')" \
+       HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "the latest override wins across both sources (review newer)" "$(cut -f1 <<<"$out")" later
+check "and its source is labelled" "$(cut -f3 <<<"$out")" review
+
+out="$(COMMENTS_JSON="$(comment later "2026-10-01T12:50:00Z" '/approve-override from a comment')" \
+       REVIEWS_JSON="$(review early "2026-10-01T12:10:00Z" COMMENT '/approve-override from a review')" \
+       HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "the latest override wins across both sources (comment newer)" "$(cut -f1 <<<"$out")" later
+check "and that source is labelled too" "$(cut -f3 <<<"$out")" comment
+
+# --- both sources absent entirely -----------------------------------------------------
+out="$(COMMENTS_JSON="" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
+       DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
+check "no sources at all yields no override, and no crash" "${out:-<none>}" "<none>"
+
 printf '\n  %d passed, %d failed\n' "$PASS" "$FAIL"
 [ "$FAIL" -eq 0 ] || exit 1
@@ -100,7 +100,8 @@ data:
  # platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
  # tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
  # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
-  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers),
+  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers;
+  # plex-remote: PLEX_L4_VIP),
  # velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
  # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
  # UNSUBSTITUTED path would ship the literal string into the cluster.
@@ -175,6 +176,19 @@ data:
  # dev  -> 192.168.1.56  (base default; next free after crafty .55 in dev's .53-.60 pool)
  # prod -> 192.168.2.23  (next free after crafty .22 in prod's .20-.30 pool; see clusters/prod)
  PIHOLE_DNS_VIP: "192.168.1.56"             # PiHole: authoritative LAN DNS for ${BASE_DOMAIN}
+  # Plex's remote-access listener: raw tcp/32400 for REMOTE Plex clients (apps/wave-0/plex-remote.yaml).
+  # A FOURTH pinned L4 VIP, for the usual reason — the `shared` Gateway is an L7 HTTPS terminator and
+  # cannot carry Plex's own TLS. Plex clients do not resolve `plex.${BASE_DOMAIN}`; they fetch this
+  # server's address from plex.tv, so the Gateway's L7 path (which still serves the web UI, and was
+  # measured working) cannot put the server on a client's connection list at all.
+  #
+  # PINNED, AND THE PIN IS LOAD-BEARING OFF-REPO: the SOHO router forwards WAN tcp/32400 to this
+  # address by hand. A VIP taken from the pool head would silently move on a reschedule and break
+  # remote access with nothing in-cluster looking wrong. See runbooks/plex-remote-access.md.
+  #
+  # dev  -> 192.168.1.57  (base default; next free after pihole-dns .56 in dev's .53-.60 pool)
+  # prod -> 192.168.2.24  (next free after pihole-dns .23 in prod's .20-.30 pool; see clusters/prod)
+  PLEX_L4_VIP: "192.168.1.57"                # Plex: tcp/32400 remote access (NOT the web UI, which is L7)

  # The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
  # peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable
@@ -476,6 +476,26 @@ patches:
      - op: replace
        path: /data/PIHOLE_DNS_VIP
        value: "192.168.2.23"
+      # Plex's remote-access listener VIP — raw tcp/32400 for REMOTE Plex clients
+      # (apps/wave-0/plex-remote.yaml). 192.168.2.24 is the NEXT FREE address in the prod pool
+      # 192.168.2.20-.30, verified live 2026-10-03: the CiliumLoadBalancerIPPool `default-pool`
+      # reported IPsTotal 11 / IPsUsed 4 / IPsAvailable 7, and the only four allocations are .20
+      # (cilium-gateway-shared), .21 (unifi-l4-tcp/-udp), .22 (crafty-l4-tcp/-udp) and .23
+      # (pihole-dns-tcp/-udp). It is inside the pool block the operator already asserted sits outside
+      # the SOHO router's DHCP scope, and it is not a node address (.10/.11/.12) or the
+      # control-plane VIP (.9).
+      #
+      # UNLIKE .23, THIS ONE IS ALSO A WAN PORT-FORWARD TARGET. The router forwards WAN tcp/32400
+      # here, by hand, off-repo — so changing this value is a two-place edit and the router is the
+      # half that CI cannot check. runbooks/plex-remote-access.md is the procedure.
+      #
+      # NOT ARP-PROBED. .23 was verified free with `arping` from a LAN host before it was claimed;
+      # for .24 only the in-cluster pool accounting above was checked, so an off-cluster squatter on
+      # this address would not have been caught. Probe before trusting it if remote access comes up
+      # with a conflict.
+      - op: replace
+        path: /data/PLEX_L4_VIP
+        value: "192.168.2.24"
      # The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns —
      # NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path.
      # It scopes the ingress allow for PiHole's DNS listener to LAN clients only.
@@ -87,6 +87,43 @@ Also updated: `CLAUDE.md` §1 and the §2 enforcement table (the "approval requi
 explicitly ⬜ with the reason, rather than a stale ✅), and `infra/gitea/README.md` gains §6 and
 marks its historical step 4 as reversed — do not re-apply.

+**Second pass, 2026-10-02 — the override did not work the first time, and why.** The operator ran
+the live PATCH (`required_approvals: 0` confirmed live, with `block_admin_merge_override` and
+`enable_status_check` still `true`), merged PR #35, and posted `/approve-override Need to ensure
+approval override work as expected` on PR #38. The gate re-ran (run 203) and **still blocked.**
+
+The comment was not at fault: posted by `jakman85`, directive at the start of the line, reason
+present, and 29 minutes **newer** than head commit `616a497`, so it cleared the staleness rule.
+What failed was that **CI could not read it.** Run 203's log:
+
+    approval-override: COMMENTS_JSON is not a list
+
+`SETTINGS_API_TOKEN` is minted `["read:repository", "read:admin"]` by
+`infra/gitea/set-ci-token.sh`, and Gitea puts `/issues/{n}/comments` behind the **issue** scope. So
+Gitea returned an error *object*, `jq -e .` accepted it as valid JSON, and the stage fell through
+to "no approving review" — **a permission failure wearing the costume of a missing override.**
+`/pulls/{n}/files` and `/pulls/{n}/reviews` are under `read:repository`, which is exactly why every
+other part of the same stage worked and this one did not.
+
+Three fixes, and the first is the one that mattered most:
+
+1. **The diagnosis is no longer misleading.** `COMMENTS_JSON` is out of the must-parse-or-block
+   loop and is explicitly checked for being an *array*; when it is not, the stage prints Gitea's
+   own `.message`, names the missing scope, and says where to post the override instead. An
+   unreadable source can only cause a false BLOCK, never a false pass, so degrading it to a loud
+   warning is safe — and blocking on it was never the honest behaviour.
+2. **The override is accepted from a PR REVIEW body as well as a comment**, so the hatch is not
+   hostage to one API scope. A review of type "Comment" works on the token CI has *today*. The
+   gate names which source it used. `PENDING` (unsubmitted, author-only draft) reviews are dropped.
+3. **`set-ci-token.sh` now requests `read:issue`**, which makes the ordinary comment box work —
+   the thing anybody reaches for first. It takes effect only on a **re-run**: an existing token's
+   scopes cannot be widened in place.
+
+`ci/tests/approval-override.test.sh` grows from 23 to **38 assertions**, covering the reviews
+source, `PENDING` reviews, an error *object* from the comments API, "an unreadable comments source
+must not suppress a valid review override", latest-wins across both sources, and that the
+diagnostic actually quotes Gitea's message.
+
 **One adjacent fix, deliberately in scope: `ci/tests/gate-decision.test.sh` was already broken.**
 It hand-maintains a list of stub stage scripts that "must name EVERY stage `gate.sh` runs", and it
 did not name `approval-policy` — added to `gate.sh` by PR #17 and never added here. `run_stage`
@@ -130,8 +167,30 @@ missing from its test. `make -C ci test` goes from **FAILED to all suites passed
  **FAIL, 15 failed / 35 passed**. After deriving the stub list from `gate.sh`: **PASS, 50
  assertions**.
 - **`make -C ci test` in the pinned toolbox: `ci/tests: all suites passed`, rc 0** — approval-override
-  23, gate-decision 50, touched-requirements-base 31, test-attribution 15, touched-requirements 26,
+  **38**, gate-decision 50, touched-requirements-base 31, test-attribution 15, touched-requirements 26,
  decisions-log pass. It was rc 2 (`ci/tests: FAILED`) before the stub-list fix.
+- **The live flip worked, and the drift detector proved it.** Run 202 (before): `gitea-settings:
+  FAIL`, `required_approvals: DECLARED 0 but LIVE is 1`. Run 203 (after the operator ran the
+  PATCH): `gitea-settings: pass`, with `block_admin_merge_override = true` among the keys it
+  checked. So the narrow change held — the approval count moved and the gate's teeth did not.
+- **The first override attempt failed for a measured reason, not a guessed one.** Run 203 log:
+  `approval-override: COMMENTS_JSON is not a list`. Token scopes read from `set-ci-token.sh`:
+  `[read:repository, read:admin]`. The same run successfully read `/pulls/38` and
+  `/pulls/38/files` (author and sensitive paths both resolved) and got past the reviews
+  validation, which is how I know `/pulls/{n}/reviews` IS readable with that token — from
+  behaviour, not from documentation.
+- **The comment itself was valid.** `created_at` 2026-10-02T05:31:24Z vs head commit `616a497`
+  committer date 2026-10-02T00:02:20-05:00 → the comment is 29m04s newer, so it passed the
+  staleness rule. Ruling the comment out is what left the token scope as the only candidate.
+- **`?limit=1000` is not the culprit.** Queried `/issues/38/comments` with `limit=1000`, `limit=50`
+  and no limit using my own credentials: `type=array` in all three. So the difference is the
+  credential, not the query.
+- **The root cause is confirmed by Gitea, in its own words.** Run 206 (the fix in place):
+  `[warn] cannot read PR COMMENTS from Gitea — token does not have at least one of required
+  scope(s), required=[read:issue], token scope=read:admin,read:repository.` followed by the
+  instruction to post the override as a review. That is the diagnosis the first attempt should
+  have produced, and `gitea-settings: pass` in the same run. The stage still ends `FAIL`, correctly
+  — there is no valid override on the PR yet.
 - **Toolbox merge gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`), run on the
  rebased branch: **rc 3, `MERGE GATE: UNDETERMINED`**. Green stages: spec-immutability,
  decisions-log, lint, l0 (23 offline cases, 0 failures), render-golden including the new
@@ -142,20 +201,32 @@ missing from its test. `make -C ci test` goes from **FAILED to all suites passed

 ## What I did NOT verify

- **The live Gitea rule still says `required_approvals: 1`. I did not change it.** Mutating live
-  platform settings is an operator action; `infra/gitea/allow-admin-self-merge.sh` is written,
-  syntax-checked, and **unrun**. The declaration and the live rule must move together.
-  *Now measured, not predicted:* PR #38's CI run (run 202, job 0) fails at **1m44s** with
-  `required_approvals: DECLARED 0 but LIVE is 1` → `gitea-settings: FAIL`. Correct behaviour, and
-  the PR stays red until the script is run.
- **The override has never been ACCEPTED by a real run** — only offered. Measured in the same CI
-  log: the stage reached `approval-policy: FAIL` on the sensitive paths and printed the *new*
-  code's guidance, `Or, if you hold admin/owner on this repository, comment on the PR:
-  /approve-override <reason> ... NEWER than head commit 7b9f733c`. So the new stage **is** what
-  runs on this PR (self-bootstrapping confirmed by measurement, not by reasoning), and the
-  authorising branch — comment parse → live permission lookup → exit 0 — is still **unexercised**.
-  Posting the comment on PR #38 is its first real test. If it does not work, the recovery is to
-  PATCH `required_approvals` and read run 202's successor log.
+- **The override has still never been ACCEPTED by a real run.** This is the second attempt at
+  this bullet, which is itself the point: the first pass called the hatch unverified, and the
+  first real test found a defect that no amount of local testing would have (an API scope CI has
+  and I did not). What is now measured is that the parse → *reject* path works end to end and
+  diagnoses itself. The **accepting** branch — directive found → live permission lookup → exit 0 —
+  has never executed. Posting the override as a **review** on PR #38 is its first real test, and
+  I would not be surprised by a third round.
+- **`read:issue` is the right scope name — Gitea said so itself, so this is no longer a guess.**
+  Run 206 printed Gitea's own message: `token does not have at least one of required scope(s),
+  required=[read:issue], token scope=read:admin,read:repository`. That both confirms the root
+  cause and names the string, which matters because this repo has been burned by a token *name*
+  that could not exist (`GITEA_SETTINGS_TOKEN`). What is **still** unverified is that the token
+  *mint* accepts it — `POST /users/{u}/tokens` with that scope has not been run. The review route
+  does not depend on it.
+- **I did not verify that Gitea lets a PR's own author submit a `COMMENT`-type review.** Gitea
+  blocks author self-*approval*; "Comment" is expected to be allowed, and that expectation is now
+  load-bearing for the recommended route. It is one click to find out, and I deliberately did not
+  test it by posting an override myself: doing so as `gitea-admin` is precisely the agent
+  self-override that `CLAUDE.md` now forbids.
+- **The whole re-run mechanism is manual.** `gate.yml` triggers on `pull_request` and `push: main`
+  only — there is no `issue_comment` trigger, so posting an override does **not** re-run the gate.
+  Re-run the job from the Actions UI, or close and reopen the PR (which fires `pull_request` and
+  leaves the head SHA alone). **Do not push an empty commit to re-trigger it:** that moves the head
+  commit and makes the override stale by the rule above, so it would have to be posted again. A
+  comment trigger was considered and not built — it would need to check out PR head code while the
+  job holds `SETTINGS_API_TOKEN`, which is the classic pwn-request shape and not worth it here.
 - **I have not seen the gate still REFUSE a merge** now that approvals are 0. The claim that
  `block_admin_merge_override: true` keeps a red gate unmergeable rests on Gitea's documented
  behaviour and on the 2026-09-29 session, **not** on a provoked refusal in this one. That probe
@@ -0,0 +1,149 @@
+---
+date: 2026-10-02
+branch: fix/plex-quicksync-verification
+title: Hardware transcoding works — and `runbooks/plex-quicksync.md`'s verification section was wrong
+---
+
+## What changed
+
+**Hardware transcoding is confirmed working on prod.** Measured 2026-10-02 from
+`Plex Media Server.log` in the running pod: `vaapi` on `/dev/dri/renderD128`
+(`Intel CoffeeLake-S GT2 [UHD Graphics 630]`), `using hardware decode accelerator vaapi`,
+`zero-copy support present`, `using zero-copy transcoding`, and
+`final decoder: vaapi, final encoder: vaapi`. VAAPI for **both** directions, zero-copy, so frames
+never round-trip through the CPU. The device-plugin mechanism from PR #37 works end to end.
+
+**And `runbooks/plex-quicksync.md`'s verification section was wrong, which is the correction this
+branch exists to publish.** The page said the Plex dashboard showing `Transcode (hw)` was "the *only*
+proof available, and nothing else will ever tell you". The operator had hardware transcoding fully
+working while **Dashboard → Activity showed nothing of the sort** — so the page's authoritative check
+was silent on a working system, and sent someone looking for a fault that did not exist. Three
+inference errors are now corrected on the page, two of them recorded as named false leads so the next
+reader does not re-walk them:
+
+1. **The dashboard is not proof.** Replaced as the primary check by Plex's own log — the grep is in
+   the page, in its **redacting** form, because `Plex Media Server.log` carries API URLs containing
+   `X-Plex-Token` (CLAUDE.md §6; a credential has been leaked into a transcript here before). The
+   page also now says explicitly that **`Plex Transcoder Statistics.log` is the wrong file** — it
+   holds XML `<Variant .../>` records, not ffmpeg output, and the name invites the mistake — and why
+   `grep -a` is required. The dashboard is **kept but demoted** to a secondary, best-effort signal,
+   with its absence stated to prove nothing.
+2. **An fd snapshot is not evidence.** A `/proc/*/fd` sweep inside the pod found no holder of
+   `renderD128` while hardware transcoding was active; transcode sessions are short and the scan
+   landed in a gap. Recorded as a documented false lead, with the reliable process-level check in its
+   place: the transcoder's **argv** (`-hwaccel`, `-hwaccel_device`, `vaapi`, `renderD128`), found via
+   `ps -eo pid,user,comm | grep -i transcoder`.
+3. **A missing preference key does not mean the setting is off.** The page told readers to treat an
+   absent `HardwareAcceleratedCodecs` in `Preferences.xml` as "never enabled". With hardware
+   acceleration demonstrably on and working, that key was **absent from all 27 attributes** while
+   `HardwareDevicePath` **was** written and the log said `hardware transcoding: enabled`. First-command
+   #4 now reads both keys and states plainly that `HardwareDevicePath` present is a positive signal,
+   absence of `HardwareAcceleratedCodecs` means nothing either way, only `="0"` is conclusive, and the
+   log settles it.
+
+Also on the page:
+
+- **GID `992` is no longer presented as a number to expect.** Under the plugin the injected device
+  arrives `root:root`, so the image's `cont-init.d` hook resolves GID **`0`** and adds `plex` to the
+  **root group** — measured `Groups: 0 100 1000`, `id plex` → `groups=1000(plex),0(root),100(users)`.
+  992 was the hostPath-era host `render` GID. The page now says so, marks the root-group membership as
+  an inherent property of the plugin's `DeviceSpec` rather than a defect, and repeats that it is
+  **not** a reason to reintroduce `supplementalGroups` (still a no-op: `s6-setuidgid` rebuilds the
+  group list from `/etc/group`). The stale `Groups: 100 1000` claim in the escalation section is gone.
+- **A new troubleshooting entry for legitimate per-title fallbacks**, marked as a **hypothesis**.
+  Between two fully hardware sessions, one logged `enabled, but no hardware decode accelerator found`
+  and empty final decoder/encoder. UHD 630 decodes H.264, HEVC 8/10-bit and VP9 but **not AV1** and
+  not 10-bit 4:2:2 HEVC, so the likely cause is the source file, not the platform — explicitly
+  written as a lead to check first, **not** a confirmed diagnosis of that item.
+- The status is stated at the top of the page and in a rewritten provenance footer that separates the
+  2026-10-02 confirmations from the superseded 2026-09-30/2026-10-01 hostPath-era readings.
+- `runbooks/README.md`'s index entry for the page was stale in the same two ways (it still described a
+  `/dev/dri` hostPath and the dashboard-only proof) and is corrected to match.
+
+Unchanged on purpose: the "STOP HERE IF PLEX IS `Pending`" section and the escape hatch (delete the
+`hardware:` block from `catalog/plex.yaml`, re-render) keep exactly their existing prominence — they
+are correct and they are the 2am material. The "no alert is possible" reasoning is also kept verbatim
+in substance: Plex declares no `metricsEndpoint`, renders no ServiceMonitor, and a fallback alert
+would be a rule evaluating against nothing.
+
+**Documentation only.** No change to `catalog/`, `schemas/`, `apps/` or `platform/`.
+
+## What I verified
+
+- **Gate, in the pinned toolbox from this worktree** (`DEV_KUBECONFIG=/dev/null`, `ENGINE` shim
+  bind-mounting the main checkout's `.git`): **rc 3 UNDETERMINED**, diff base
+  `origin/main @ 948875aad37d`, with exactly the three expected skips and **no failures**. Green:
+  `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, `approval-policy`. Skipped:
+  `spec-immutability` (spec repo unavailable → R-TEST-06 not verified), `gitea-settings`
+  (`SETTINGS_API_TOKEN` unset), `conformance` (`CLUSTER_AVAILABLE!=true` → L2/L3 not run).
+  Touched requirements: **R-OBS-04**, reported `UNVERIFIED` for lack of a cluster. `approval-policy`
+  passed as `N/A` locally (`event='none'`, not a `pull_request`) — it evaluates for real on the PR.
+  **rc 3 is not green**, it is the calibrated worktree result, and it is reported here as rc 3.
+- **The gate was run three times, with the same rc 3 / three-skip result on each committed tip** —
+  including the final one. Worth recording from the first attempt: run *before* committing, with
+  `HEAD == origin/main`, the touched-requirement set is `NOT DETERMINABLE (on-base)` and the
+  R-TEST-03 coverage gate cannot be applied at all. Commit before gating from a fresh worktree branch.
+- **`runbooks/kustomization.yaml` still carries `- plex-quicksync=plex-quicksync.md`** in the
+  `configMapGenerator` `files:` list (line 110) — checked explicitly, because without it the page
+  merges and still 404s. `README=README.md` is likewise present, which is what serves the corrected
+  index entry.
+- **This change touches no `sensitive_paths`.** `infra/gitea/approval-policy.json` lists `ci/**`,
+  `.gitea/**`, `conformance/**`, `clusters/**`, `bootstrap/**`, `infra/gitea/**`,
+  `platform/kyverno/**`, `platform-config/kustomization.yaml`, `.sops.yaml`, `**/*.sops.yaml`,
+  `**/rbac.yaml`, `CLAUDE.md`. `runbooks/**` and `decisions/**` are not among them, and the
+  `approval-policy` stage passed.
+- **No YAML was added or edited**, so `indent-sequences: consistent` (which blocked a sibling branch
+  today) is not in play here; `lint` passed over the full tracked YAML set regardless.
+- **`DECISIONS-LOG.md` is untouched** — this handoff is its own file, per CLAUDE.md §7.
+- **The PR's own CI run was read, not assumed** (PR #39, run 208, task 261, head `1677dff5688b`).
+  **It is RED, and not because of this branch.** Every stage this change could affect is green —
+  `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, `approval-policy` (`approval-policy` evaluated
+  for real on the `pull_request` event and passed: no sensitive path touched). The single failure is
+  **`gitea-settings`**, quoting the measured field:
+
+      [fail]   required_approvals: DECLARED 1 but LIVE is 0
+      [fail] Gitea has DRIFTED from infra/gitea/expected-branch-protection.json.
+      [fail] MERGE GATE: BLOCKED
+
+  Every other declared setting matched (`block_admin_merge_override`, `block_on_rejected_reviews`,
+  `dismiss_stale_approvals`, `enable_status_check`, `enable_push=false`,
+  `status_check_contexts = ["gate / gate (pull_request)"]`, `runners online: 1`). So **live `main`
+  currently requires ZERO approvals**, the declaration still says 1, and with
+  `REQUIRE_GITEA_SETTINGS: "true"` in `.gitea/workflows/gate.yml` that drift **blocks every PR in
+  the repo**, not just this one. Note the shape of it: the check that would notice approvals are
+  unenforced is the check that is failing. Local branch `fix/admin-approval-override` (`616a497`,
+  "fix(ci): let an administrator merge their own green PR") looks like the in-flight work that means
+  to change the declaration. **I did not touch it** — `infra/gitea/**` is a sensitive path, needs a
+  non-author approval, and is outside this branch's documentation-only scope. It is an operator
+  decision: re-assert the protection, or change the declaration deliberately in its own PR.
+- The repository-wide grep for `992` to find what else still asserts it: remaining occurrences are in
+  `catalog/plex.yaml`, `catalog/render.py`, `schemas/renderer-contract.md`,
+  `platform/intel-gpu-plugin/README.md`, `platform/intel-gpu-plugin/kustomization.yaml`, and three
+  older `decisions/` entries.
+
+## What I did NOT verify
+
+- **I did not re-measure anything on the live cluster in this session.** Every platform number on the
+  page — the VAAPI/zero-copy log lines, `RW-OPEN-OK` as uid 1000, the `gpu.intel.com/i915` limits and
+  requests on the pod spec, `Groups: 0 100 1000`, `id plex`, the 27 `Preferences.xml` attributes, the
+  empty `/proc/*/fd` sweep — came to me as measurements taken earlier on 2026-10-02 and was
+  transcribed, not re-run. Per CLAUDE.md §2b, treat the page as reporting those readings rather than
+  as proof of current state; re-run the primary check before relying on it.
+- **The per-title fallback cause is unverified and labelled as such.** The source codec of the title
+  that logged `no hardware decode accelerator found` was never read. The AV1 / 10-bit-4:2:2-HEVC
+  explanation is inferred from UHD 630's capability set alone.
+- **Whether Plex's dashboard *ever* shows `Transcode (hw)` on this deployment is unknown.** It was
+  observed silent during working hardware transcoding; whether that is a client/version quirk, a
+  reporting bug, or something about this setup was not investigated. The page therefore demotes it
+  rather than explaining it.
+- **The commands as written in the page were not executed from the page.** They are the forms that
+  were run, re-wrapped for `dev.sh`; in particular the `exec -it deploy/plex -- sh` route for the argv
+  check is noted as possibly lacking a TTY under `dev.sh run` and was not exercised in that shape.
+- **Stale `992` copies outside `runbooks/` were left alone**, deliberately — scope was documentation
+  only, and `catalog/plex.yaml`, `catalog/render.py`, `schemas/renderer-contract.md` and
+  `platform/intel-gpu-plugin/` all still carry the hostPath-era `Groups: 44 100 992 1000` reading as
+  if current. `render.py --selftest` asserts around that comment, so correcting it is a real change
+  with gate consequences, not a doc edit. **That is open work.** Older `decisions/` entries also cite
+  992; those are historical records and should stay as written.
+- **CI on the PR.** The branch is pushed and the PR opened; I did not watch the Gitea run report, so
+  nothing here should be read as "CI green". The only gate result I can quote is the local rc 3 above.
@@ -0,0 +1,45 @@
+---
+date: 2026-10-03
+branch: fix/node-keycloak-hosts-pin
+title: Pin the OIDC issuer to the Gateway VIP on prod nodes — the WAN answer does not hairpin
+---
+
+## What changed
+
+Follow-up to PR #40 (`decisions/2026-10-03-public-dns-wan-target.md`). Public DNS for
+`*.home.keithjackson.dev` now CNAMEs to the WAN IP. The prod nodes resolve through public DNS,
+and the router does not hairpin, so the kube-apiserver could no longer reach its OIDC issuer
+`keycloak.home.keithjackson.dev`.
+
+- `roles/node_baseline`: a new `node_baseline_hosts_pins` list (default `[]`) and one
+  `blockinfile` task (tags `dns`, `hosts`). It writes the pins into `/etc/hosts` as a single
+  marked block. An empty list removes the block.
+- `group_vars/k3s_prod/vars.yml`: pins `keycloak.{{ k3s_base_domain }}` → `192.168.2.20`, which
+  must stay equal to `GATEWAY_VIP` (parity comment added). It also replaces the stale "no
+  /etc/hosts pin is needed" paragraph with the measured reason the pin is now required.
+
+Dev is unaffected (empty default).
+
+## What I verified
+
+- After PR #40 reconciled (`platform` lastAppliedRevision `576e1ab`, Healthy=True), on
+  k3s-prod-2 and k3s-prod-3: `getent hosts keycloak.home.keithjackson.dev` →
+  `136.51.115.252 wan.home.keithjackson.dev`. `curl https://keycloak…/realms/homelab/.well-known/openid-configuration`
+  → exit 60, `ssl_verify_result=18` (the router's self-signed cert, not the Gateway).
+- `journalctl -u k3s` (last 20 min) showed no OIDC/JWKS errors yet. The apiserver still holds
+  the keys it cached before the flip, so the failure is latent until a k3s restart or key rotation.
+- Neither node has cloud-init (`/etc/cloud` absent), so nothing rewrites `/etc/hosts` at boot.
+- `--check --diff --tags hosts --limit k3s-prod-2,k3s-prod-3`: each node gets exactly the
+  three-line marked block `192.168.2.20	keycloak.home.keithjackson.dev`.
+- Outside-in, PR #40 works: check-host.net probes from 3 external nodes got `200 OK` from
+  `https://plex.home.keithjackson.dev/identity` at `136.51.115.252`.
+
+## What I did NOT verify
+
+- **k3s-prod-1**: Ansible cannot reach it. SSH host key verification fails (`known_hosts:94`,
+  ED25519 `SHA256:zgUS8OeWznzCX9hlN7XdZ8iLNkU1G2loXjtJFfEUYpE`), which is consistent with
+  the 2026-09-11 SSD replacement and reinstall. The operator must confirm the fingerprint
+  before the pin can be applied there.
+- The playbook has not been applied for real. It is an operator run after merge:
+  `ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`.
+- An end-to-end kubectl OIDC login after the pin (needs a browser).
@@ -0,0 +1,129 @@
+---
+date: 2026-10-03
+branch: feat/plex-remote-access-l4
+title: Plex remote access — raw tcp/32400 via a pinned LoadBalancer VIP, not the L7 Gateway
+---
+
+## What changed
+
+Reported symptom: users cannot reach Plex from outside the LAN. **Two independent causes**, measured
+rather than inferred:
+
+1. **Plex advertised no remote connection at all.** Live `Preferences.xml` on prod:
+   `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while
+   `PublishServerOnPlexOnlineKey="1"` — publishing was on and there was nothing reachable to publish.
+   Plex's automatic UPnP/NAT-PMP path cannot work from a pod on the Cilium overlay.
+2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` refused;
+   `136.51.115.252:443` answered with a **self-signed router cert** (`C=ZZ, CN=%D`), not the
+   Gateway's LE wildcard. The Plex Service was `ClusterIP` only — no off-cluster path to 32400 by
+   any route.
+
+This branch fixes the in-cluster half. **Remote access does not work until the operator also does
+two manual steps** (`runbooks/plex-remote-access.md`): the WAN tcp/32400 port-forward, and ticking
+"manually specify public port" in the Plex UI.
+
+- **`apps/wave-0/plex-remote.yaml` (new, hand-written).** A `type=LoadBalancer` Service on tcp/32400
+  pinned to `${PLEX_L4_VIP}` via `lbipam.cilium.io/ips`, plus a `CiliumNetworkPolicy`
+  (`plex-allow-remote-32400`) admitting `world`/`cluster`/`host`/`remote-node` to 32400. Non-catalog
+  because `schemas/servicedefinition.schema.json` is `additionalProperties: false` and has no field
+  for a LoadBalancer Service or a pinned VIP — the same reason `apps/wave-0/pihole-dns.yaml`,
+  `platform/crafty/service-l4.yaml` and `apps/vms/unifi/50-service-l4.yaml` are hand-written.
+- **`catalog/render.py`:** an `EXTRA_UNITS` entry (`plex-remote`, anchored to `plex`, `above: False`)
+  so `--emit-waves` places the unit in plex's own wave.
+- **`clusters/base/cluster-vars.yaml`:** new `PLEX_L4_VIP` (dev default `192.168.1.57`), plus the
+  substitution inventory comment updated to name `plex-remote: PLEX_L4_VIP`.
+- **`clusters/prod/kustomization.yaml`:** prod override `PLEX_L4_VIP: 192.168.2.24`.
+- **`apps/wave-0/kustomization.yaml`:** `plex-remote.yaml` added to `resources`.
+- **`runbooks/plex-remote-access.md` (new):** the two manual steps, the four-step verification, why
+  no alert is possible, and the two documented warts below.
+
+**`ADVERTISE_IP` was deliberately NOT set.** Setting it to `https://plex.home.keithjackson.dev:443`
+was the alternative — it needs no new WAN hole and the mechanism was verified working on an
+already-claimed server. **The operator chose the native path (2026-10-03)** so Plex terminates its own
+TLS (`*.plex.direct`) and streams bypass the Envoy L7 hop. `catalog/plex.yaml`'s deferral note for
+`ADVERTISE_IP` remains accurate and untouched.
+
+**`externalTrafficPolicy: Cluster`, with a known cost.** `Local` would preserve the client IP, but
+Cilium's L2 announcer does not follow the backend pod — the wart that cost this LAN an 8-day
+Minecraft outage (see `platform/crafty/service-l4.yaml`). Consequence accepted and documented: SNAT
+puts the client in `192.168.0.0/16`, inside `ALLOWED_NETWORKS`, so **Plex classifies remote streams
+as LAN**. That is a metering effect (remote bandwidth limits will not bind), not an authorization one.
+
+**SENSITIVE PATHS — this PR touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK
+it.** It needs the operator's `/approve-override <reason>` posted as a PR **review** of type
+"Comment". I have deliberately not posted it: CLAUDE.md §1 forbids an agent clearing this on its own
+PR, and the agent account holds `owner`, so the prohibition is a rule, not a limit.
+
+## What I verified
+
+All measurements against live prod, 2026-10-03, read-only.
+
+- **Plex's advertised state**, the finding this branch rests on: `customConnections=<ABSENT>`,
+  `ManualPortMappingMode=<ABSENT>`, `ManualPortMappingPort=<ABSENT>`,
+  `PublishServerOnPlexOnlineKey="1"`, `allowedNetworks="10.0.0.0/8,192.168.0.0/16"`. Read by
+  attribute grep, never by dumping `Preferences.xml` (it holds `PlexOnlineToken`).
+- **The L7 Gateway path works and is NOT the problem.** Via the Gateway VIP,
+  `GET https://plex.home.keithjackson.dev/identity` → `HTTP 200`, `claimed="1"`, real
+  `machineIdentifier`. HTTPRoute `plex-32400`: `Accepted=True`, `ResolvedRefs=True`.
+- **Plex enforces its own auth**, which is what makes an unfiltered WAN port defensible:
+  `/library/sections`, `/status/sessions`, `/:/prefs` all → **401** unauthenticated through the
+  Gateway.
+- **The SNAT/allowedNetworks wart is real, not theoretical.** A tagged probe through the Gateway was
+  logged by Plex as `[10.0.2.214:36040 (Allowed Network (WAN))]` — the Envoy pod IP, classified as an
+  allowed network.
+- **WAN state:** `136.51.115.252:32400` → connection refused. `136.51.115.252:443` → self-signed
+  cert `C=ZZ, ST=Somewhere, O=%D342e7e6b, CN=%D`. Public DNS resolves correctly:
+  `plex.home.keithjackson.dev` → CNAME `wan.home.keithjackson.dev` → `136.51.115.252`.
+- **`192.168.2.24` is free in-cluster.** `CiliumLoadBalancerIPPool default-pool`: IPsTotal 11 /
+  IPsUsed 4 / IPsAvailable 7; the only allocations are `.20` (cilium-gateway-shared), `.21`
+  (unifi-l4-*), `.22` (crafty-l4-*), `.23` (pihole-dns-*). Not a node address (`.10/.11/.12`) and not
+  the kube-vip control-plane VIP (`.9`).
+- **The manual-port-mapping step genuinely cannot be automated.** Grep of the running image's whole
+  init surface (`/etc/cont-init.d`, `/plex-common.sh`, `/usr/local/bin`) for `ManualPortMapping` and
+  `PLEX_PREFERENCE` → **NONE**.
+- **`ADVERTISE_IP` would have worked on a claimed server** (recorded because the runbook's
+  "why not" rests on it): `setPref "customConnections"` at line 126 of `40-plex-first-run` sits
+  **outside** the `PLEX_CLAIM`/`PlexOnlineToken` guard, and the `/.firstRunComplete` early-exit
+  marker is on the **overlay** FS, not the `/config` PVC, so the script re-runs every pod start.
+- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
+  `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
+  **pass**. The rc 3 is two environmental skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`)
+  and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3, not as pass.**
+- **`render-golden` passing is the specific proof** that the `render.py` `EXTRA_UNITS` change, the new
+  hand-written unit and the edited wave kustomization are mutually consistent.
+
+## What I did NOT verify
+
+- **THE ACTUAL FIX. Nothing in this branch has been proven to make remote access work** — it cannot
+  be, because both remaining steps are off-repo and manual (router forward, Plex UI toggle). This
+  branch removes two blockers and adds none; it does not demonstrate an outcome.
+- **Outside-in reachability, at all.** Every probe above was taken from inside the LAN, so the
+  `:443` self-signed cert strictly proves only that there is **no NAT hairpin** — it does **not**
+  prove outside-in failure. A router cert on WAN:443 is suggestive (its management UI would shadow a
+  forward), but unproven. **This repo has no off-LAN vantage point.** Test 3 in the runbook
+  (app.plex.tv from a cellular connection) is the only honest check.
+- **`192.168.2.24` was never ARP-probed off-cluster.** Only in-cluster pool accounting was checked.
+  `.23` was `arping`-probed before it was claimed; `.24` was not, so an off-cluster squatter would
+  not have been caught.
+- **The LoadBalancer has never been reconciled.** `plex-l4-tcp` does not exist in prod yet, so
+  LB-IPAM has not actually assigned the pinned VIP, L2 announcement has not been observed for it, and
+  the new CiliumNetworkPolicy has never admitted a real packet. Whether `world`/`host`/`cluster`/
+  `remote-node` is the correct entity set **under this SNAT path** is reasoned from
+  `platform/crafty/networkpolicy.yaml`'s working game-port rule, **not measured**.
+- **Whether Plex's remote path needs anything beyond the manual port tick.** If plex.tv still lists
+  no remote connection after both manual steps, this branch is necessary-but-insufficient.
+- **The CI approval-policy outcome.** The local stage reported green, but that is an offline N/A, not
+  a verdict: this PR touches `clusters/**`, so CI will block it pending an override.
+- **PRE-EXISTING DRIFT I FOUND AND DELIBERATELY LEFT ALONE — worth its own follow-up.** Running the
+  documented `python3 catalog/render.py --emit-waves` regenerates the wave kustomizations and wants to
+  add **`vm-demo.yaml`** to wave 0 and **`keycloak.yaml`** to wave 1. Both are listed as
+  intentionally not onboarded in `ci/render-golden.sh`
+  (`RENDER_GOLDEN_NOT_ONBOARDED="keycloak vm-demo"`); their manifests are tracked in `apps/` but
+  deliberately absent from the `resources` lists so Flux does not apply them. `--emit-waves` places
+  any file present on disk and has **no awareness of that exclusion**, while `render-golden.sh` uses
+  `--wave-layout` and compares rendered manifests only — it never checks kustomization `resources`,
+  which is why this has survived unnoticed. **So the header "DO NOT hand-edit: re-run
+  `render.py --emit-waves`" is currently unsafe advice**: following it onboards an IdP and a KubeVirt
+  VM. I therefore restored both kustomizations and hand-added only my one line. I did not fix
+  `--emit-waves`, and I have not determined how long the drift has existed or whether either service
+  is *meant* to be onboarded.
@@ -0,0 +1,73 @@
+---
+date: 2026-10-03
+branch: feat/public-dns-wan-target
+title: Public DNS now points at the WAN — Gateway-published names CNAME to an in-cluster DDNS record
+---
+
+## What changed
+
+The operator closed the open "public-target gap" recorded in
+`platform/externaldns/helmrelease.yaml`. The WAN NAT / port-forwarding (80/443 → Gateway VIP
+192.168.2.20) was done by the operator outside this repo.
+
+Operator decisions (2026-10-03):
+
+1. **Scope: every Gateway-published name goes public.** That covers the five `exposure: public`
+   apps (plex, gitea, keycloak, nextcloud, seerr) and also the hand-authored admin UIs (auth,
+   crafty, grafana, headlamp, the `home.keithjackson.dev` apex). `internal` services still publish
+   nothing: their `controller: none` route annotation is checked before the target is read.
+2. **The WAN IP is kept current in-cluster**, not hardcoded.
+
+Implementation:
+
+- `platform/gateway/gateway.yaml`: `external-dns.alpha.kubernetes.io/target: wan.${BASE_DOMAIN}` on
+  the shared Gateway. Every gateway-sourced record changes from `A 192.168.2.20` to
+  `CNAME wan.home.keithjackson.dev`.
+- `platform/externaldns/ddns.yaml` (new): `favonia/cloudflare-ddns:1.17.1` Deployment in the
+  `external-dns` namespace. It reuses `cloudflare-api-token` and keeps `wan.${BASE_DOMAIN}` an
+  unproxied IPv4 A record at the WAN IP (detected via Cloudflare trace, checked every 5 min,
+  `DELETE_ON_STOP=false`). external-dns has no TXT registry entry for it, so it never owns or
+  deletes it.
+- The helmrelease comment block and the README now describe the decision instead of the gap.
+
+Known consequence: **k3s nodes resolve through public DNS** (`node_baseline_dns_servers` = 8.8.8.8 /
+1.1.1.1). The kube-apiserver's OIDC issuer `keycloak.home.keithjackson.dev` therefore resolves to the
+WAN IP from the nodes, and kubectl SSO works only if the router hairpins WAN:443 back to the Gateway.
+A LAN-side probe of `136.51.115.252:443` on 2026-10-03 returned a **self-signed placeholder cert**,
+not the Gateway's Let's Encrypt wildcard. That suggests no hairpin. The cert-based admin kubeconfig
+is unaffected.
+
+## What I verified
+
+- Live prod, 2026-10-03, before the change: Gateway `gateway/shared` ADDRESS `192.168.2.20`.
+  External-dns args include `--source=gateway-httproute --source=service --policy=sync
+  --txt-owner-id=prod --domain-filter=keithjackson.dev`, and it logs "All records are already up to
+  date".
+- Public answers at 1.1.1.1: plex/gitea/keycloak/grafana/auth/apex → `192.168.2.20`;
+  `crafty-mc` → `.22` and `unifi` → `.21` (service source, not affected by this change); jackett →
+  none (internal, correctly unpublished).
+- The authority (betty.ns.cloudflare.com) has no unprefixed TXT, AAAA or MX at `plex.` or the apex,
+  so the A→CNAME switch has no co-existing record to collide with.
+- WAN IP from the workstation: `136.51.115.252`.
+- No CiliumNetworkPolicy or NetworkPolicy in `external-dns`. The Kyverno label and default-deny
+  policies match only `app-namespace` namespaces. The vpn egress policy selects only
+  `egress-class: vpn` pods.
+- Toolbox gate result: see the PR description (run after this commit).
+
+## What I did NOT verify
+
+- **The outside-in path.** I have no vantage point outside the LAN. Whether
+  `https://<svc>.home.keithjackson.dev` from the internet reaches the Gateway with the LE cert is
+  UNVERIFIED.
+- **Hairpin / kubectl SSO after the flip.** Likely broken per the probe above. Fix candidates: a
+  node `/etc/hosts` pin for `keycloak.${BASE_DOMAIN}` → Gateway VIP via `node_baseline`, or router
+  NAT reflection.
+- **The record-type flip itself**, until Flux reconciles: whether Cloudflare accepts external-dns's
+  create-CNAME in the same batch as the delete-A, or converges one interval later.
+- `favonia/cloudflare-ddns` under this pod securityContext. The README warns that
+  `no-new-privileges` misbehaves on some Proxmox-style hosts. These are bare-metal nodes, but it is
+  untested until the pod runs.
+- Whether the L4 service-source names (crafty-mc, unifi) should also go public. Not decided, not
+  changed.
+- `conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh` header lines 39–40
+  still describe the target as open. That is a sensitive path, so it is left for a follow-up.
@@ -0,0 +1,133 @@
+---
+date: 2026-10-04
+branch: feat/pihole-declare-state
+title: PiHole's operator-set state is now declared in git — the prerequisite for running more than one replica
+---
+
+## What changed
+
+Phase 1 of making LAN DNS survive a node failure. The operator asked for multi-replica PiHole;
+measurement showed that cannot be done safely first, because PiHole's important state was not
+reproducible from git. A second replica with a fresh PVC would have come up **serving DNS and
+blocking nothing, with a different admin password** — answers varying by which pod replied. This
+branch closes that; the replica work is phase 2 and is deliberately NOT here.
+
+**The undeclared-state inventory, measured not guessed.** `pihole.toml` marks every non-default
+setting `### CHANGED`, and those that came from the environment `### CHANGED (env)`. The complete
+live list on 2026-10-04:
+
+| Setting | Status |
+|---|---|
+| `dns.upstreams`, `dns.listeningMode`, `dns.rateLimit.count`, `misc.dnsmasq_lines` | `### CHANGED (env)` — already declared in `catalog/pihole.yaml` |
+| `files.macvendor` | `### CHANGED` — set by the image itself (container path), not operator state |
+| `webserver.api.pwhash` | **`### CHANGED` — set out of band. Declared NOWHERE.** |
+
+So exactly **one** operator-set value lived only on the PVC. Separately, the **adlists** were also
+undeclared: `gravity.db` held one list (StevenBlack's hosts) and 72,234 gravity domains, with nothing
+in this repo naming it.
+
+- **`apps/wave-0/pihole-bootstrap-secret.sops.yaml` (new, SOPS/age).** Secret `pihole-bootstrap`,
+  key `webui-password`. Named to the `<svc>-bootstrap-secret.sops.yaml` convention so
+  `render.py --emit-waves` places it with no renderer change. The value is a 32-char random password
+  generated at authoring time and **never printed to a transcript**.
+- **`catalog/pihole.yaml`:** `FTLCONF_webserver_api_password` sourced from that Secret.
+  `webserver.api.password` takes CLEARTEXT and FTL hashes it itself — verified in the running image,
+  where `pihole setpassword` *is* `setFTLConfigValue "webserver.api.password"`, commented upstream
+  "pihole-FTL will automatically hash the password". The PVC's `pwhash` becomes a derived artefact.
+- **`catalog/pihole.yaml`:** a `lifecycle.postStart` hook that **reconciles** the declared adlist set
+  into `gravity.db` on every container start — adds missing, re-enables disabled, and **disables**
+  (never deletes) anything enabled but undeclared, running `pihole -g` only when something moved.
+  Same level-triggered idiom, and the same reasoning, as `catalog/qbittorrent.yaml`'s postStart.
+- **`apps/wave-0/pihole.yaml`, `apps/wave-0/kustomization.yaml`:** re-rendered / secret registered.
+
+**Why a postStart hook and not a file.** PiHole v6 keeps adlists as DB rows. `/etc/pihole/adlists.list`
+still exists in `gravity.sh` but is **migration-only**: its import sits inside `migrate_to_database()`
+behind `if [ ! -e "$gravityDBfile" ]`, so it fires only while *creating* a database. A ConfigMap
+mounted there would be silently ignored on every pod this cluster runs today. There is also no
+`pihole adlist` CLI subcommand, so SQL is the actual interface; FTL ships an embedded shell
+(`pihole-FTL sqlite3`) so nothing had to be added to the image.
+
+**The password is kept rather than set empty, deliberately.** The admin UI is already behind the
+oauth2-proxy sidecar (`--allowed-group=platform-admins`), so an empty password would have been
+simpler. But the `pihole` Service still publishes **port 80** (PiHole direct, bypassing the proxy on
+`:4180`) and only the CiliumNetworkPolicy closes it — and this repo has a documented default-deny
+drift history. An unauthenticated admin UI that can rewrite LAN DNS should not rest on one policy
+staying correct. Narrowing that Service port is follow-up, not done here.
+
+**SENSITIVE PATHS — this PR adds a `**/*.sops.yaml`, so `ci/approval-policy.sh` will BLOCK it.** It
+needs the operator's `/approve-override <reason>` as a PR **review** of type "Comment". Not posted by
+me: CLAUDE.md section 1 forbids an agent clearing this on its own PR.
+
+## What I verified
+
+Live prod, 2026-10-04, read-only except where noted.
+
+- **The undeclared-state inventory above** — read from `pihole.toml`'s own `### CHANGED` markers, with
+  the password value masked. This is the measurement the whole branch rests on.
+- **A fresh `gravity.db` seeds NO adlists.** The image's own schema template
+  `/etc/.pihole/advanced/Templates/gravity.db.sql` creates the `adlist` table and contains no
+  `INSERT INTO adlist` at all — grepped in the running image. Confirmed empirically below (0 rows).
+  This is why the adlist declaration is load-bearing and not cosmetic.
+- **The adlists.list path really is migration-only** — read the guard in `/opt/pihole/gravity.sh`
+  (`migrate_to_database()`, `if [ ! -e "$gravityDBfile" ]`).
+- **`webserver.api.password` is a real settable key** — `pihole-FTL --config webserver.api.password`
+  returns a masked value, and `/usr/local/bin/pihole`'s `setpassword` sets exactly that key.
+- **THE RENDERED HOOK ACTUALLY WORKS — all four paths, extracted from `apps/wave-0/pihole.yaml` and
+  run in the live pod against THROWAWAY COPIES of `gravity.db` in `/tmp`** (production `gravity.db`
+  was never written, and `pihole -g` was stubbed so no 70k-domain download could start):
+  1. *Steady state* → `already matches the declared set - nothing to do`. No writes, no rebuild.
+  2. *True fresh DB built from the image's schema template* → 0 adlists before; hook added the
+     declared list and triggered the rebuild. **This is the fresh-PVC/new-replica case.**
+  3. *An undeclared list enabled* → disabled it (`enabled=0`), left the declared one alone, and did
+     not delete the row.
+  4. *Declared list present but disabled* → re-enabled it.
+- **Required tools exist in the image:** `bash`, `expr`, `grep`, `printf`, `sleep`, `pihole-FTL`.
+- **The Secret is genuinely encrypted** to the repo's age recipient; `stringData.webui-password`
+  starts `ENC[`. The gate's `no-plaintext-secrets (R-SEC-01)` L0 check passes.
+- **Schema:** `check-jsonschema` against `schemas/servicedefinition.schema.json` passes.
+- **Render determinism:** the canonical
+  `render.py --all catalog -o apps --wave-layout --no-image-automation` changed **only**
+  `apps/wave-0/pihole.yaml`. `render-golden` passes, so the hook's text also satisfies
+  `assert_substitution_safe` (no braced dollar expansion — the Flux envsubst contract).
+- **Toolbox gate:** rc **3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
+  `render-golden`, `l1`, `approval-policy` all **pass**. The rc 3 is the two usual environmental
+  skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`) and `conformance` L2/L3
+  (`CLUSTER_AVAILABLE` unset). **Not green; reported as rc 3.**
+
+## What I did NOT verify
+
+- **Nothing here has been reconciled.** Flux has not applied any of it, so the password has never
+  been set by this mechanism and the hook has never run during a real container start. Everything
+  above is the rendered artifact tested out-of-band, plus the image's own source.
+- **THE CURRENT ADMIN PASSWORD WILL STOP WORKING, and that is the intent.** FTL re-hashes
+  `webserver.api.password` on start, overwriting the existing `pwhash`. Whoever knows the present
+  password (nobody, which is why this started) loses it; the declared one replaces it. I have not
+  observed that replacement happen.
+- **Cold-start cost is unmeasured and could matter.** `postStart` must complete before the container
+  is reported Running. On a fresh PVC the hook waits for `gravity.db` (bounded, up to 180s) and then
+  runs a real `pihole -g` (72k domains, network downloads). `apps-wave-0` is `wait: true` and
+  `apps-wave-1` `dependsOn` it, so a slow cold start **delays the app tier**. The steady-state path
+  exits in about a second (scenario 1), so this is a rebuild/first-boot cost only — but it is not
+  measured, and the stub in testing means `pihole -g`'s real duration here is unknown.
+- **Whether `pihole -g` succeeds from inside the hook's environment.** Its failure is caught and the
+  existing `gravity.db` retained, so a failure degrades rather than breaks — untested live.
+- **Concurrency.** Nothing serialises the hook against FTL's own scheduled gravity update. SQLite
+  handles the locking, and the hook's writes are three small statements, but no interleaving was
+  exercised.
+- **MULTI-REPLICA — PHASE 2, NOT STARTED.** This branch only removes the blocker. The remaining work,
+  and it is substantial: a `replicas`/placement field on the ServiceDefinition schema (currently
+  `additionalProperties: false` with no such field), `render.py` emitting a StatefulSet with
+  `volumeClaimTemplates` (the config PVC is **ReadWriteOnce**, so a 2-replica Deployment cannot
+  work), pod anti-affinity, a PodDisruptionBudget (there is none today), re-rendered goldens, and
+  `conformance/**` updates — the last making phase 2 sensitive-path as well. The PVC rename a
+  StatefulSet forces will discard the existing `gravity.db`, which is only safe **because** of this
+  branch, and should be done only after this one is confirmed working in-cluster.
+- **The real risk the operator named is NOT fixed by phase 2 either.** Replicas address *node*
+  failure. LAN DNS still depends on the cluster as a whole — etcd quorum, Cilium, Longhorn, a bad
+  reconcile. Quantified on 2026-10-04: with `tolerationSeconds=300` plus node-NotReady detection
+  plus a Longhorn RWO force-detach, a failure of the node running the pod is roughly **6-10 minutes
+  of no LAN DNS**; a failure of the node holding the `pihole-dns-*` L2 lease (currently a
+  *different* node — lease on k3s-prod-2, pod on k3s-prod-1) is tens of seconds. An out-of-cluster
+  secondary resolver carrying the same `address=` lines is the only option that removes the
+  dependency, and it was considered and not chosen.
+- **Port 80 on the `pihole` Service is still published** and still closed only by policy. Not narrowed.
@@ -306,19 +306,27 @@ node_baseline_dns_servers:
  # k3s_base_domain, below). It reaches the issuer through the NODE resolver, i.e. through
  # these PUBLIC upstreams — not through CoreDNS's split-horizon answer.
  #
-  # THIS USED TO BE BROKEN, AND THE 2026-09-04 SUBDOMAIN SPLIT FIXED IT. While prod was
-  # authored to serve dev's hostnames, a prod node resolving keycloak.home.keithjackson.dev
-  # publicly got DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN, so
-  # kubectl OIDC SSO on prod could not work at all. Now prod's issuer is
-  # keycloak.prod.home.keithjackson.dev, and the public record for that name is published by
-  # PROD's own external-dns pointing at PROD's Gateway VIP (192.168.2.20). The public answer
-  # and the split-horizon answer therefore AGREE, and no /etc/hosts pin is needed.
+  # THE PUBLIC ANSWER IS NOW THE WAN, SO THE ISSUER IS PINNED (2026-10-03). Since
+  # platform/gateway/gateway.yaml set an external-dns `target`, every public name is a CNAME
+  # to wan.home.keithjackson.dev = the home WAN IP. The SOHO router does NOT hairpin: from the
+  # LAN, WAN:443 answers with the router's own self-signed placeholder cert (measured from
+  # k3s-prod-2/-3 2026-10-03: `curl` exit 60, ssl_verify_result=18). The apiserver would then
+  # be unable to fetch its issuer's discovery/JWKS, so kubectl OIDC SSO would fail on the
+  # next k3s restart or Keycloak key rotation. node_baseline_hosts_pins (below) pins the
+  # issuer host to the Gateway VIP in /etc/hosts, which glibc consults before these
+  # resolvers. (History: before the 2026-09-04 subdomain split the public answer was DEV's
+  # VIP; after it, the public answer was prod's own VIP and no pin was needed.)
  #
-  # STILL TO VERIFY on prod bring-up (PROD-PROVISIONING.md §10): this holds only once prod's
-  # external-dns has actually published the record. Until then the issuer does not resolve
-  # from the nodes and kubectl SSO fails — the cert-based admin kubeconfig is the break-glass
-  # path, and it always works. Ordering, not a design gap.
+  # The cert-based admin kubeconfig never depends on any of this. It is the break-glass path.
 node_baseline_dns_search: []
+# /etc/hosts pins: node-process names that must reach the in-cluster Gateway directly rather
+# than via the public (WAN, non-hairpinned) answer. See "THE PUBLIC ANSWER IS NOW THE WAN" above.
+# [parity] the IP MUST equal GATEWAY_VIP in clusters/prod/kustomization.yaml (192.168.2.20),
+# and the name MUST follow k3s_oidc_issuer_url's host (roles/k3s_server/defaults/main.yml).
+node_baseline_hosts_pins:
+  - ip: "192.168.2.20"
+    names:
+      - "keycloak.{{ k3s_base_domain }}"

 # --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
 # PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/
@@ -242,3 +242,9 @@ node_baseline_dns_search:
 node_baseline_dns_options:
  - timeout:2
  - attempts:2
+# Static /etc/hosts entries (tags: dns, hosts), as a list of {ip, names}. EMPTY by default:
+# dev needs none. Prod pins its OIDC issuer host to the Gateway VIP because the public
+# answer for it is the non-hairpinned WAN IP (inventory/group_vars/k3s_prod/vars.yml).
+# Written as ONE marked block, so removing an entry here removes it from the node, and
+# an empty list removes the block entirely.
+node_baseline_hosts_pins: []
@@ -1090,3 +1090,22 @@
      {% endif %}
  when: node_baseline_dns_mechanism == 'resolv-conf-file'
  tags: [dns]
+
+# /etc/hosts pins — tag `dns` (and `hosts` on its own). Some node processes must reach a
+# public hostname at the in-cluster Gateway, not at its public answer. On prod, the
+# kube-apiserver's OIDC issuer resolves publicly to the WAN IP, which the router does not
+# hairpin. glibc reads /etc/hosts before any nameserver (nsswitch `files dns`), so a pin wins
+# regardless of which resolver mechanism above is in effect. One marked block: entries
+# dropped from node_baseline_hosts_pins disappear from the node, and an empty list
+# removes the block.
+- name: Pin node-process hostnames in /etc/hosts
+  ansible.builtin.blockinfile:
+    path: /etc/hosts
+    marker: "# {mark} node_baseline hosts pins (Ansible, tags: hosts)"
+    block: |
+      {% for pin in node_baseline_hosts_pins %}
+      {{ pin.ip }}	{{ pin.names | join(' ') }}
+      {% endfor %}
+    state: "{{ 'present' if node_baseline_hosts_pins | length > 0 else 'absent' }}"
+    backup: true
+  tags: [dns, hosts]
@@ -161,7 +161,18 @@ a *green* one no longer needs a signature that cannot exist.
 **Security-relevant changes are still reviewed** — by `ci/approval-policy.sh` against
 `approval-policy.json` `sensitive_paths`, not by this field. An administrator waives that review
 only by posting `/approve-override <reason>` on the PR: reason mandatory, permission re-checked
-live, invalidated by a new commit, and recorded by account in the gate summary. Note that the
+live, invalidated by a new commit, and recorded by account in the gate summary.
+
+> **Post the override as a PR *review* (type "Comment"), not in the plain comment box** — until
+> the CI token is re-minted. `SETTINGS_API_TOKEN` is `[read:repository, read:admin]`, and
+> `/issues/{n}/comments` needs Gitea's *issue* scope, so Gitea answers with an error **object**
+> instead of an array and the override is invisible. Measured on PR #38 (2026-10-02): a correct,
+> fresh, well-formed override in the comment box produced `approval-policy: FAIL` reading as "no
+> approving review" — a permission failure disguised as a missing one. `/pulls/{n}/reviews` is
+> under `read:repository` and worked in the same run. The stage now reads **both** sources, names
+> which one it used, and says plainly when the comments API is unreadable. `set-ci-token.sh`
+> requests `read:issue` as of 2026-10-02, so **re-run it** and the comment box works too — token
+> scopes cannot be widened in place. Note that the
 **agent account `gitea-admin` also reports permission `owner`**, so it can clear its own PRs that
 way; that is an accepted consequence of authorising by permission alone, and the route to
 narrowing it (an `override_admins` allowlist, already honoured by the code) is written down in
@@ -32,7 +32,8 @@
    "requires_reason": true,
    "authorized_permissions": ["admin", "owner"],
    "invalidated_by_new_commits": true,
-    "_how": "Comment on the PR with the directive at the START of a line, followed by a reason: '/approve-override <why>'. ci/approval-policy.sh resolves the commenter's LIVE repo permission via /collaborators/{login}/permission and accepts the override only if it is in authorized_permissions. The overrider, the timestamp and the reason are printed in the gate log and the job summary.",
+    "_how": "Post the directive at the START of a line, followed by a reason: '/approve-override <why>'. ci/approval-policy.sh resolves the poster's LIVE repo permission via /collaborators/{login}/permission and accepts the override only if it is in authorized_permissions. The overrider, the source, the timestamp and the reason are printed in the gate log and the job summary.",
+    "_where_to_post_it": "EITHER a PR REVIEW body (type 'Comment') OR the ordinary PR comment box. Prefer the REVIEW: measured 2026-10-02 on PR #38, a well-formed override in the comment box was INVISIBLE to CI, because /issues/{n}/comments sits behind Gitea's *issue* token scope and SETTINGS_API_TOKEN is minted [read:repository, read:admin]. Gitea returned an error object rather than an array, and the stage reported 'no approving review' -- a permission problem wearing the costume of a missing override. /pulls/{n}/reviews is under read:repository and was readable in the same run, so a review body is the route that works on today's token. infra/gitea/set-ci-token.sh now also requests read:issue, which makes the comment box work -- but only after that script is RE-RUN, since an existing token's scopes cannot be widened in place. The stage accepts both sources and names which one it used, so neither route is load-bearing alone.",
    "_self_override_is_allowed": "The author may override their own PR if they hold an authorized permission. That is the entire point -- with one human there is no one else -- but it means the 'a change to the gate cannot be gated by the gate' protection is, for an administrator, advisory rather than structural. It is traded for an auditable record: the waiver is an explicit act, attributed, with a stated reason, in the PR's comment history and the gate output.",
    "_known_consequence_agent_accounts": "ACCEPTED, EYES OPEN (operator decision, 2026-10-01). Authorisation is by live permission ALONE, with no allowlist of human accounts. The agent account `gitea-admin` reports permission `owner` on this repo (measured 2026-10-01), so an AGENT can clear the sensitive-path review on its own PR by posting the directive. Agents author most PRs here, so this is the broad case, not the edge case. To narrow it later without changing any logic, add an `override_admins` allowlist here and require membership in addition to the permission check -- ci/approval-policy.sh reads both from this file.",
    "_staleness": "An override is dropped unless its comment is NEWER than the PR head commit, mirroring Gitea's own dismiss_stale_approvals. An override earned at commit A therefore does not silently cover commit B; push again and you must override again."
@@ -5,9 +5,23 @@
 # The token value is NEVER printed. It goes from the token API into the secrets API and
 # nowhere else -- not to stdout, not to a file, not into an agent transcript.
 #
-# Scope: `read:repository` + `read:admin` are what the check needs (the rule object, and the
-# runner list). Gitea token scopes are coarse; grant the least that works on your version and
-# confirm the check goes green rather than assuming the scope name.
+# Scope: `read:repository` + `read:admin` are what ci/gitea-settings.sh needs (the rule object,
+# and the runner list). `read:issue` is for ci/approval-policy.sh, and it was MISSING — see below.
+# Gitea token scopes are coarse; grant the least that works on your version and confirm the check
+# goes green rather than assuming the scope name.
+#
+# WHY read:issue IS HERE (added 2026-10-02, measured on PR #38). ci/approval-policy.sh reads the
+# PR's comments to find an admin `/approve-override`. /issues/{n}/comments sits behind the *issue*
+# scope, which this script did not request — so Gitea answered with an error OBJECT instead of an
+# array, and a perfectly well-formed override posted in the PR comment box was invisible to CI.
+# The failure did not look like a permission problem from the outside; it looked like the override
+# had not been posted. `/pulls/{n}/files` and `/pulls/{n}/reviews` are under read:repository, which
+# is why every other part of the same stage worked and this one did not.
+#
+# The stage now accepts an override from a REVIEW body as well, so it still functions on a token
+# WITHOUT read:issue. This scope is what makes the ordinary comment box work too — the thing
+# anybody would reach for first. **Re-run this script to pick it up; an existing token's scopes
+# cannot be widened in place.**
 set -euo pipefail
 HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
 # NOT GITEA_* — Gitea reserves that prefix for secret names and returns HTTP 400
@@ -21,7 +35,7 @@ basic() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }

 echo "==> minting token '${NAME}' as ${U}"
 TOKEN="$(basic -X POST -H 'Content-Type: application/json' \
-  -d "{\"name\":\"${NAME}\",\"scopes\":[\"read:repository\",\"read:admin\"]}" \
+  -d "{\"name\":\"${NAME}\",\"scopes\":[\"read:repository\",\"read:admin\",\"read:issue\"]}" \
  "https://${HOST}/api/v1/users/${U}/tokens" | jq -r '.sha1 // empty')"
 [ -n "$TOKEN" ] || { echo "token creation failed (check scopes for your Gitea version)" >&2; exit 1; }
 echo "==> got a ${#TOKEN}-character token (value not shown)"
@@ -20,6 +20,30 @@ A wildcard `*.keithjackson.dev` matches only one label deep, so `<svc>.home.keit
 its own record. **`internal` services need NO public record** — PiHole resolves them to the internal
 Gateway VIP (split-horizon). ExternalDNS manages **`public`** services' records only.

+## Public target + in-cluster DDNS (2026-10-03)
+
+Gateway-published records are **CNAMEs to `wan.home.keithjackson.dev`**, not the Gateway VIP:
+
+```
+plex.home.keithjackson.dev  CNAME  wan.home.keithjackson.dev   <- external-dns (Gateway `target` annotation)
+wan.home.keithjackson.dev   A      <current WAN IP>            <- cloudflare-ddns (ddns.yaml), every 5m
+```
+
+- The `target` annotation is on the **shared Gateway** (`platform/gateway/gateway.yaml`), so it
+  applies to every route that publishes at all. `internal` routes (`controller: none`) still publish
+  nothing.
+- `ddns.yaml` runs `favonia/cloudflare-ddns` in this namespace with the same `cloudflare-api-token`.
+  It detects the WAN IP through Cloudflare's trace endpoint, so the pod **must not** carry
+  `egress-class: vpn`.
+- LAN and pods resolve `*.home.keithjackson.dev` to the Gateway VIP via PiHole and CoreDNS
+  (split-horizon) and never use the public path.
+- **Nodes resolve through public DNS** (`node_baseline_dns_servers`), so from a node, the apiserver's
+  OIDC issuer `keycloak.home.keithjackson.dev` resolves to the WAN IP. kubectl SSO therefore relies on
+  the router hairpinning (NAT reflection) WAN:443 back to the Gateway.
+
+Check: `dig +short @1.1.1.1 plex.home.keithjackson.dev` → `wan.home.keithjackson.dev.` then the WAN IP;
+`kubectl -n external-dns logs deploy/cloudflare-ddns`.
+
 ## Secrets

 Cloudflare API token referenced **by name** (`cloudflare-api-token`, shared with cert-manager),
@@ -0,0 +1,87 @@
+---
+# In-cluster dynamic DNS for the home WAN (2026-10-03).
+#
+# Keeps ONE record, `wan.${BASE_DOMAIN}` (A, IPv4 only, not proxied), pointed at the home WAN's
+# current public IP. Every public name external-dns publishes from the shared Gateway is a CNAME to
+# it (platform/gateway/gateway.yaml `external-dns.alpha.kubernetes.io/target`), so a WAN IP change
+# moves all of them through this one record, with no Git change and no external-dns involvement.
+#
+# OWNERSHIP SPLIT WITH EXTERNAL-DNS. This record carries no external-dns TXT registry entry, so
+# external-dns (policy: sync, owner `prod`) never treats it as its own and never deletes it.
+# Conversely this updater manages only the names in DOMAINS and never touches external-dns's records.
+#
+# IP DETECTION. Default `cloudflare.trace`: an HTTPS request to Cloudflare's trace endpoint, which
+# reports the address the request arrived from. Correct here only because this pod is NOT in the
+# `vpn` egress class — the CiliumEgressGatewayPolicy (platform/vpn-egress) selects pods by
+# `homelab.keithjackson.dev/egress-class: vpn` alone, cluster-wide. Labelling this pod `vpn` would
+# publish the VPN exit IP instead of the WAN IP.
+#
+# TOKEN. Reuses this namespace's `cloudflare-api-token` (same Zone:DNS:Edit token as external-dns
+# and cert-manager) — no new secret.
+#
+# TIER NOTE. This lives in the gating `platform` Flux Kustomization (wait: true) next to
+# external-dns. If it ever crashloops, `platform` goes not-Ready and apps stop reconciling (no
+# outage). Kept here rather than in its own non-gating Kustomization to avoid a clusters/** change.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: cloudflare-ddns
+  namespace: external-dns
+  labels:
+    app.kubernetes.io/name: cloudflare-ddns
+spec:
+  replicas: 1                 # exactly one writer for the record
+  revisionHistoryLimit: 3
+  strategy:
+    type: Recreate            # never two updaters at once
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: cloudflare-ddns
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: cloudflare-ddns
+    spec:
+      automountServiceAccountToken: false
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: cloudflare-ddns
+          image: favonia/cloudflare-ddns:1.17.1
+          imagePullPolicy: IfNotPresent
+          env:
+            - name: CLOUDFLARE_API_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: cloudflare-api-token
+                  key: api-token
+            - name: DOMAINS
+              value: "wan.${BASE_DOMAIN}"
+            - name: IP6_PROVIDER
+              value: "none"         # IPv4 WAN only; leaves any AAAA alone
+            - name: PROXIED
+              value: "false"        # DNS-only: Plex/Gitea/etc. terminate TLS on the Gateway
+            - name: UPDATE_CRON
+              value: "@every 5m"
+            - name: DELETE_ON_STOP
+              value: "false"        # a pod restart must not blank the record every CNAME depends on
+            - name: RECORD_COMMENT
+              value: "managed by k3s cloudflare-ddns (platform/externaldns/ddns.yaml)"
+            - name: EMOJI
+              value: "false"
+          resources:
+            requests:
+              cpu: 5m
+              memory: 16Mi
+            limits:
+              memory: 64Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop:
+                - ALL
@@ -81,38 +81,29 @@ spec:
    # what this instance can see or own; per-resource exclusion only ever REMOVES an object from
    # this instance's view, so it cannot make prod capable of deleting a record it does not own.
    # ---------------------------------------------------------------------------------------------
-    # THE PUBLIC-TARGET GAP — STILL OPEN, AND NOT FIXABLE HERE ON v0.21.0 (operator decision)
+    # THE PUBLIC TARGET — DECIDED 2026-10-03 (operator): every Gateway-published name goes public
    #
-    # The five `exposure: public` services (plex, gitea, seerr, keycloak, nextcloud) publish records
-    # that point at 192.168.2.20 — the Gateway VIP, an unroutable RFC1918 address — so they are not
-    # actually reachable from the internet either. Their records are LEFT EXACTLY AS THEY ARE by
-    # this change, for two independent reasons:
+    # Until 2026-10-03 every gateway-sourced record answered with the Gateway VIP (192.168.2.20,
+    # RFC1918), so nothing published here was reachable from the internet. Two facts constrained the
+    # fix, and both still hold:
+    #   * `external-dns.alpha.kubernetes.io/target` is read in v0.21.0's gateway-httproute source ONLY
+    #     from the GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
+    #     gw.Annotations)`) and from a ListenerSet listener — NEVER from an HTTPRoute. A target on the
+    #     shared Gateway therefore moves EVERY gateway-sourced record at once.
+    #   * The home WAN IP is residential and can change, so a literal IP in Git would rot.
    #
-    #   1. THE VALUE IS NOT ESTABLISHED. The one public IP this repo knows is 136.51.115.252
-    #      (clusters/prod/kustomization.yaml, where the zone's `*.keithjackson.dev` wildcard also
-    #      points). It belongs to the LEGACY non-k3s stack and currently serves a placeholder
-    #      certificate. Whether prod's public services belong behind it, behind a new forward, or
-    #      are not meant to be public at all is an operator decision. Breaking five services' DNS
-    #      on a guess is worse than leaving a known-wrong-but-stable state.
-    #   2. THE MECHANISM CANNOT BE SCOPED TO THEM. `external-dns.alpha.kubernetes.io/target` is a
-    #      real override, but in v0.21.0's gateway-httproute source it is read ONLY from the
-    #      GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
-    #      gw.Annotations)`) and from a ListenerSet listener — NEVER from the HTTPRoute. So there is
-    #      no per-service target override while every public service shares the `shared` Gateway.
-    #      Annotating the Gateway would move EVERY gateway-sourced record, which after this change
-    #      still includes auth, crafty, grafana, headlamp and the apex — i.e. it would publish a
-    #      routable address for admin UIs. That is strictly worse than the status quo.
-    #
-    # SO THE DECISION THE OPERATOR OWNS IS TWO-PART, and no knob is added here until the first half
-    # is answered, because a knob that cannot take effect invites someone to set it and expect one:
-    #   (a) WHAT address should the five public names answer with — 136.51.115.252 as-is, a new
-    #       port-forward/WAN address, or nothing (drop `exposure: public` to `internal` instead)?
-    #   (b) HOW should they be separated from the non-public names that share the Gateway? Options,
-    #       roughly in ascending cost: a SECOND Gateway (or a ListenerSet) carrying the target
-    #       annotation, with only the public services' routes attached; or publishing the five names
-    #       from the `crd` source (DNSEndpoint objects the renderer emits for `public` services),
-    #       which restores per-service targeting without a second Gateway.
-    # Only (b) is a manifest change; (a) is a fact about the home WAN that has to be checked.
+    # The operator's answers: (a) yes, ALL of them — the five `exposure: public` apps AND the admin
+    # UIs (auth, crafty, grafana, headlamp, the apex), which sit behind Keycloak / oauth2-proxy;
+    # (b) the address is kept current IN-CLUSTER. Implementation:
+    #   * platform/gateway/gateway.yaml sets `target: wan.${BASE_DOMAIN}` on the shared Gateway, so
+    #     each published name becomes a CNAME to that one hostname;
+    #   * platform/externaldns/ddns.yaml (favonia/cloudflare-ddns) keeps `wan.${BASE_DOMAIN}` an A
+    #     record at the current WAN IP. That record has no external-dns TXT registry entry, so this
+    #     instance never owns or garbage-collects it.
+    # Internal services are unchanged: their routes carry `controller: none` (above), which is
+    # checked BEFORE the target is read, so they still publish nothing. The `service` source (the L4
+    # LoadBalancers crafty-mc, unifi) does not read the Gateway's annotation and still publishes its
+    # LB VIP. Whether those should go public is a separate, still-open question.
    # ---------------------------------------------------------------------------------------------
    # CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below.
    #
@@ -8,3 +8,5 @@ resources:
  # SOPS-encrypted Cloudflare token (env CF_API_TOKEN) — SAME token as cert-manager, second copy in
  # the external-dns namespace. Flux decrypts at reconcile.
  - cloudflare-api-token.sops.yaml
+  # DDNS for `wan.${BASE_DOMAIN}` — the CNAME target every Gateway-published record points at.
+  - ddns.yaml
@@ -7,6 +7,23 @@ kind: Gateway
 metadata:
  name: shared
  namespace: gateway
+  annotations:
+    # PUBLIC DNS TARGET (2026-10-03, operator decision). Without this, external-dns publishes every
+    # route on this Gateway with the Gateway's OWN address (${GATEWAY_VIP}, RFC1918), which is
+    # unreachable from the internet. With it, every published `<svc>.${BASE_DOMAIN}` (and the apex)
+    # becomes a CNAME to `wan.${BASE_DOMAIN}`, an A record that platform/externaldns/ddns.yaml keeps
+    # pointed at the home WAN's current public IP. The WAN forwards 80/443 to ${GATEWAY_VIP}.
+    #
+    # Scope is EVERY route on this Gateway that publishes at all — the operator chose to make the
+    # admin UIs (auth, grafana, headlamp, crafty, apex) publicly resolvable alongside the five
+    # `exposure: public` apps. `internal` services stay unpublished via their
+    # `external-dns.alpha.kubernetes.io/controller: none` route annotation, which external-dns checks
+    # BEFORE it reads this target. external-dns v0.21.0 reads `target` from the Gateway only, never
+    # from an HTTPRoute — see platform/externaldns/helmrelease.yaml.
+    #
+    # LAN and in-cluster clients are unaffected: PiHole and CoreDNS answer ${BASE_DOMAIN} with
+    # ${GATEWAY_VIP} directly (split-horizon), so they never take this public path.
+    external-dns.alpha.kubernetes.io/target: "wan.${BASE_DOMAIN}"
 spec:
  gatewayClassName: cilium
  listeners:
@@ -153,14 +153,18 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
  plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
  token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
  Includes why a changed Secret does not restart anything, and why an expired token is harmless.
- **`plex-quicksync`** — hardware transcoding on the host iGPU. GitOps puts `/dev/dri` in the pod
-  (`hardware.gpu: intel-quicksync`); **a human still has to tick Settings → Transcoder → "Use hardware
-  acceleration when available"**, and no environment variable can do it (the image supports no
-  `PLEX_PREFERENCE*`, verified). The page names the real preference key (`HardwareAcceleratedCodecs`,
-  read out of the binary), says which widely-cited key **does not exist**, and is explicit that the
-  **only** proof is `Transcode (hw)` on the Plex dashboard — Plex renders no ServiceMonitor, so no
-  alert could ever fire. Also: the device-cgroup failure to expect first and its device-plugin
-  fallback, and why a `:latest` pull can silently break it.
+- **`plex-quicksync`** — hardware transcoding on the host iGPU. **Confirmed working 2026-10-02**
+  (VAAPI decode *and* encode, zero-copy). GitOps requests `gpu.intel.com/i915` on the pod
+  (`hardware.gpu: intel-quicksync`, satisfied by the `intel-gpu-plugin` DaemonSet — a `/dev/dri`
+  hostPath was measured not to work: device-cgroup `EPERM`); **a human still has to tick Settings →
+  Transcoder → "Use hardware acceleration when available"**, and no environment variable can do it
+  (the image supports no `PLEX_PREFERENCE*`, verified). **The proof is `Plex Media Server.log`'s
+  `final decoder: / final encoder:` line, not the Plex dashboard** — the dashboard showed nothing
+  while hardware transcoding was working, so its silence proves nothing; the page carries the
+  redacting grep (`X-Plex-Token` lives in that log), the WRONG log file to avoid, and two more
+  documented false leads (an empty `/proc/*/fd` scan, and a missing `HardwareAcceleratedCodecs` key).
+  Plex renders no ServiceMonitor, so no alert could ever fire. Also: **`Pending` stalls the whole app
+  tier** and its one-field escape hatch, and why a `:latest` pull can silently break it.
 - **`oauth2-proxy-session`** — a gated UI loads but its saves quietly do nothing, and you are never
  re-prompted to log in. Tells the three stacked auth layers apart from the sidecar's own access log
  (the four lines that matter, including the upstream `-` that means the proxy answered a request the
@@ -1,5 +1,10 @@
 # plex-quicksync — making Plex actually use Intel Quick Sync, and proving it did

+**Status: hardware transcoding is CONFIRMED WORKING on prod as of 2026-10-02** — VAAPI for both
+decode *and* encode, zero-copy, measured in Plex's own server log (see "How to verify"). The manual
+toggle has been ticked. This page is therefore now mostly a **verification and regression**
+procedure, and its old verification section was **wrong** — see the correction at the top of
+"How to verify".
 **Alert:** none, and none is possible — see "How to verify" below. This is an **operator procedure**,
 not an incident page. It is indexed here because the feature it completes is **inert until a human
 clicks one button**, and a switched-off feature that looks shipped is exactly what this page exists
@@ -33,7 +38,7 @@ is in git**:
 | Half | Who does it | State |
 |---|---|---|
 | The render device is inside the container **and openable** | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders `resources.limits: {gpu.intel.com/i915: 1}` on the app container. The `intel-gpu-plugin` DaemonSet (`platform/intel-gpu-plugin/`) injects the device **and the matching device-cgroup allow rule**. Flux applies it. | Done by merging that change. |
-| Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **NOT done.** `HardwareAcceleratedCodecs` was **absent** from `Preferences.xml` when this page was written — the toggle has never been on. |
+| Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **Done, 2026-10-02**, and confirmed from the **log** rather than from a preference key. `HardwareAcceleratedCodecs` is **still absent** from `Preferences.xml` with hardware transcoding working, so its absence proves nothing — see first-command #4. |

 **There is no GitOps path to the second half, and that is verified rather than assumed.** A bounded
 grep across the running image's entire init surface — `/etc/cont-init.d`, `/etc/services.d`,
@@ -79,8 +84,8 @@ also the only thing that produces the cgroup allow rule. There is no graceful-de
 The one improvement is that the loss is no longer silent.

 What this does **not** do, so nobody reads more into a green gate than is there: it does not enable
-hardware transcoding (that is the manual toggle below), a green gate only proves the **manifest
-renders**, and it writes no `nodeSelector` — the resource itself constrains Plex to nodes the plugin
+hardware transcoding (that is the manual toggle below — done 2026-10-02), a green gate only proves
+the **manifest renders**, and it writes no `nodeSelector` — the resource itself constrains Plex to nodes the plugin
 advertises, which is the scheduler's doing and not a selector in this repo.

 ## Running the commands in this runbook
@@ -172,17 +177,28 @@ cgroup rule that uid 0 could not buy its way past either. `RW-OPEN-DENIED` with
 visible is the **device-cgroup** failure — i.e. the plugin's allow rule did not arrive; see "Likely
 causes" #2.

-**4. Is the toggle on?** Named key only — never the whole file:
+**4. Is the toggle on?** **You cannot answer this from `Preferences.xml` alone.** That mistake is on
+this page's record: it told readers to treat a missing `HardwareAcceleratedCodecs` as "never
+enabled", and on 2026-10-02 that key was **absent from all 27 attributes** while hardware
+transcoding was demonstrably working. Named keys only — never the whole file, it holds
+`PlexOnlineToken`:

 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
-     grep -o "HardwareAcceleratedCodecs=\"[01]\"" \
-     "/config/Library/Application Support/Plex Media Server/Preferences.xml" || echo ABSENT'
+     grep -o -e "HardwareAcceleratedCodecs=\"[01]\"" -e "HardwareDevicePath=\"[^\"]*\"" \
+     "/config/Library/Application Support/Plex Media Server/Preferences.xml" || echo NEITHER-KEY'
 ```

-`ABSENT` ⇒ never enabled (the state this page was written in). `="0"` ⇒ explicitly off. `="1"` ⇒ on,
-and the remaining question is whether it *works* — go to "How to verify".
+| Reading | What it actually means |
+|---|---|
+| `HardwareDevicePath="…"` **present** | **Positive signal.** Plex writes this when hardware acceleration is configured; it was present on 2026-10-02 alongside a working VAAPI pipeline. |
+| `HardwareAcceleratedCodecs` **absent** | **Nothing. Neither on nor off.** Measured absent with the feature working (2026-10-02). Do not diagnose from it, and do not write it into the file to "fix" it. |
+| `HardwareAcceleratedCodecs="0"` | explicitly **off** — the only conclusive reading, and only in the negative. |
+| `HardwareAcceleratedCodecs="1"` | on. |
+
+**The log settles it, this file does not.** Go to "How to verify"; the `hardware transcoding:
+enabled` and `final decoder/final encoder` lines are the verdict.

 ## The one manual step

@@ -191,8 +207,11 @@ Plex web UI → **Settings → Transcoder → "Use hardware acceleration when av

 The underlying preference key is **`HardwareAcceleratedCodecs`**. That name was confirmed by
 extracting strings from the 23 MB `Plex Media Server` binary in the running container — **not** from
-recall and not from a forum post. Real neighbouring keys from the same binary, so you can recognise
-them and not invent them:
+recall and not from a forum post. **But do not expect to see it in `Preferences.xml` after ticking
+the box:** on 2026-10-02, with the toggle on and hardware transcoding working, the key was absent and
+`HardwareDevicePath` was what had been written. The key exists in the binary; Plex does not
+necessarily persist it. Real neighbouring keys from the same binary, so you can recognise them and
+not invent them:

 | Key | What it is |
 |---|---|
@@ -209,42 +228,121 @@ binary. Recorded explicitly so nobody spends an evening hunting for it or, worse
 Editing `Preferences.xml` by hand is **not** the procedure. It is app state on the replicated Longhorn
 tier, it is in no git commit, and the UI writes the key correctly. Use the UI.

-## How to verify — and why there is no alert
+## How to verify — the log, not the dashboard

-Start a playback that **forces a transcode** (a client that cannot direct-play the source, or pick a
-lower quality explicitly), then read Plex's own **Dashboard → Activity**:
+**Correction, 2026-10-02.** Until today this page said the Plex dashboard showing `Transcode (hw)`
+was "the *only* proof available, and nothing else will ever tell you". **That was false and it cost
+the operator real time:** hardware transcoding was fully working — VAAPI decode *and* encode, with
+zero-copy — while **Dashboard → Activity showed nothing of the sort**. A check that is silent while
+the feature works does not just fail to prove anything; it sends someone hunting a fault that does
+not exist. The authoritative check is **Plex's own server log**.

- **`Transcode (hw)`** on the session ⇒ hardware transcoding. **This is the proof.**
- plain **`Transcode`** ⇒ software. The feature is not working, whatever the manifest says.
+### The primary check — run this one

-**That dashboard is the *only* proof available, and nothing else will ever tell you.**
-`catalog/plex.yaml` declares no `metricsEndpoint`, so Plex renders **no ServiceMonitor at all** —
-there is no scrape, no transcode-mode metric, and therefore no alert that could fire. Do not open work
-to "alert on software-transcode fallback": with no metric behind it such a rule would be permanently
-inert, and this repo has already been burned at scale by rules that evaluated against nothing while
-looking like coverage (`runbooks/README.md`, "Alerts you should not trust the silence of"). If that
-gap is ever worth closing, the honest order is a real exporter first and a rule second.
+```
+./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n plex exec deploy/plex -- \
+  grep -aiE "hardware transcoding: (final|using)" \
+  "/config/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log" \
+  | sed -E "s/(X-Plex-Token|token)=[^&[:space:]]*/\1=REDACTED/g" | tail -5'
+```

-Corroborating, not a substitute — the server log on the config PVC:
+| Line | Verdict |
+|---|---|
+| `hardware transcoding: final decoder: vaapi, final encoder: vaapi` | **PASS.** Hardware in **both** directions. This is the pass condition. |
+| `hardware transcoding: using zero-copy transcoding` | Best case — frames never round-trip through the CPU. Observed on prod 2026-10-02. |
+| `hardware transcoding: final decoder: , final encoder: ` (both empty) | It **fell back to software** for that session. |
+| no matching line at all | No transcode has been attempted since the log rolled — force one (below), then look again. Absence is not a failure. |
+
+The 2026-10-02 reading on prod, in full, for the shape to recognise:
+
+```
+Codecs: hardware transcoding: testing API vaapi for device '/dev/dri/renderD128'
+                              (Intel CoffeeLake-S GT2 [UHD Graphics 630])
+hardware transcoding: using hardware decode accelerator vaapi
+hardware transcoding: zero-copy support present
+hardware transcoding: using zero-copy transcoding
+hardware transcoding: final decoder: vaapi, final encoder: vaapi
+```
+
+To make a line appear, start a playback that **forces a transcode**: a client that cannot
+direct-play the source, or an explicitly lower quality. Direct Play / Direct Stream never reaches
+the transcoder and logs nothing.
+
+### Three things to get right about that command
+
+1. **Always keep the `sed`.** `Plex Media Server.log` carries API URLs containing
+   **`X-Plex-Token=…`**, a live plex.tv credential. Any output that might land in a ticket, a
+   transcript or a chat window must be redacted — CLAUDE.md §6, and a credential has already been
+   leaked into a transcript in this repo exactly this way. The redacting form above is the form to
+   copy; do not "simplify" the pipe out of it.
+2. **`Plex Transcoder Statistics.log` is the WRONG FILE.** The name is irresistible and someone will
+   reach for it. It holds XML `<Variant .../>` session records — **not** ffmpeg/VAAPI output. There
+   is no decoder/encoder verdict anywhere in it. The file you want is `Plex Media Server.log`.
+3. `grep -a` is deliberate: the log carries occasional non-UTF-8 bytes, and without `-a` GNU grep
+   reports "binary file matches" and prints nothing — which reads exactly like "no hardware
+   transcoding".
+
+### Secondary and unreliable: the Plex dashboard
+
+**Dashboard → Activity** *may* label a transcoding session `Transcode (hw)`. If it does, it agrees
+with the log and you are done. **If it does not, that proves nothing at all** — measured 2026-10-02:
+hardware transcoding active and zero-copy, dashboard silent. Never conclude "software transcoding"
+from the dashboard and never open an investigation on its say-so. Kept here only because it is the
+fastest glance available; it is a convenience, **not evidence**.
+
+### Process-level, while a transcode is in flight: the transcoder's argv
+
+This shows Plex *requesting* hardware, as distinct from reporting it afterwards. Get a shell in the
+pod (if `-it` has no TTY under `dev.sh run`, wrap each line in `sh -c "…"` instead):

 ```
 infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
-     sh -c "grep -i -m20 -e hardware -e vaapi -e hwdevice \
-     \"/config/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log\""'
+  'kubectl --context k3s-prod -n plex exec -it deploy/plex -- sh'
 ```

-Treat those strings as advisory: they are Plex's, not this repo's, and they move between versions.
-Absence of a match is **not** evidence of software transcoding; a VA-API device open is good evidence
-of hardware. The dashboard remains the verdict.
+then, inside the pod:
+
+```
+ps -eo pid,user,comm | grep -i transcoder
+tr '\0' ' ' < /proc/<transcoder-pid>/cmdline | grep -oE '(-hwaccel[a-z_]*|vaapi|renderD[0-9]+)'
+```
+
+`-hwaccel`, `-hwaccel_device`, `vaapi`, `renderD128` present ⇒ Plex is asking for hardware. Keep the
+`grep -oE`: the raw argv contains media paths and session identifiers you do not want in a
+transcript. This only works while a session is actually transcoding.
+
+### Documented false lead: scanning `/proc/*/fd` for a holder of `renderD128`
+
+**It returns nothing even when hardware transcoding is working.** Measured 2026-10-02: a
+`/proc/*/fd` sweep inside the pod found **no** process holding `/dev/dri/renderD128`, while the log
+showed `final decoder: vaapi, final encoder: vaapi` for sessions either side of the scan. Transcode
+sessions are **short** and the scan landed in a gap between two of them. An empty fd scan is a
+**timing artefact, not a finding** — do not report it as evidence of software transcoding, and do
+not escalate on it. Use the argv check above (which has the same timing constraint but tells you
+something when it does catch a session), or just read the log, which is retrospective and therefore
+does not care about timing.
+
+### Why there is no alert — this half was always right
+
+`catalog/plex.yaml` declares no `metricsEndpoint`, so Plex renders **no ServiceMonitor at all** —
+there is no scrape, no transcode-mode metric, and therefore no alert that could fire. Do not open
+work to "alert on software-transcode fallback": with no metric behind it such a rule would be
+permanently inert, and this repo has already been burned at scale by rules that evaluated against
+nothing while looking like coverage (`runbooks/README.md`, "Alerts you should not trust the silence
+of"). If that gap is ever worth closing, the honest order is a real exporter first and a rule
+second.

 ## Likely causes, most likely first

-When the dashboard still says plain `Transcode`:
+When the **log** says it fell back to software — `final decoder: , final encoder: ` with both
+fields empty. (Not when the dashboard is quiet: the dashboard's silence means nothing, see "How to
+verify".)

-1. **The toggle is off.** Overwhelmingly the most likely answer, and the reason this page exists.
-   First-command #4 returns `ABSENT` or `="0"`. Remedy: the manual step above. Check this **before**
-   investigating anything technical.
+1. **The toggle is off.** Historically the most likely answer and the reason this page exists. It was
+   ticked on 2026-10-02 and should stay ticked, but a `/config` restore or a stray UI save can unset
+   it. Remedy: the manual step above. **Do not diagnose this from a missing
+   `HardwareAcceleratedCodecs`** — see first-command #4; only `="0"` is conclusive, and the log line
+   `hardware transcoding: enabled` is the real signal that the toggle took.

 2. **The device cannot be `open()`ed — the plugin's cgroup allow rule did not arrive.** Symptom:
   first-command #3 shows `renderD128` present and correctly owned, and the open returns
@@ -276,13 +374,27 @@ When the dashboard still says plain `Transcode`:
 4. **The image changed under you.** See "The floating-image caveat". A `:latest` pull can remove the
   mechanism this feature leans on with no commit in this repo.

-5. **The session was never transcoding.** Direct Play / Direct Stream never touches the transcoder, so
-   `Transcode (hw)` cannot appear. Confirm the session actually says `Transcode` before treating it as
-   a hardware failure.
+5. **The session was never transcoding.** Direct Play / Direct Stream never reaches the transcoder,
+   so no `hardware transcoding:` line is written for it at all. An *absent* line is this, not a
+   failure — confirm a transcode actually happened before treating anything as a hardware fault.

-6. **The source needs a codec Quick Sync on this generation cannot handle** — e.g. AV1 decode, which
-   Coffee Lake's UHD 630 has no engine for. Plex falls back to software *per stream*. Symptom: some
-   sessions show `(hw)` and others do not. Not a fault.
+6. **The *source* codec may be one this iGPU cannot decode — HYPOTHESIS, not a confirmed
+   diagnosis.** The symptom that points here is fallback on **particular titles** while other
+   sessions are hardware. Measured on prod 2026-10-02, *between* two fully hardware sessions, one
+   session logged:
+
+   ```
+   hardware transcoding: enabled, but no hardware decode accelerator found
+   hardware transcoding: final decoder: , final encoder:
+   ```
+
+   The platform was demonstrably working either side of it, so the likely explanation is the **file**,
+   not the cluster: UHD 630's media engine decodes H.264, HEVC 8-bit and 10-bit, and VP9 — but **not
+   AV1**, and not 10-bit 4:2:2 HEVC. **This was NOT verified against that particular title.** It is
+   inferred from the hardware's capability set and is written here as the lead to check first, not as
+   a finding. So: fallback on *all* files is a platform problem (causes 1-4); fallback on *some*
+   files, check the source codec before suspecting the device, and expect occasional legitimate
+   per-title fallbacks to be normal rather than a fault.

 ## Plex will not schedule

@@ -360,19 +472,30 @@ infra/devcontainer/dev.sh run bash -c \
 ```

 `45-plex-hw-transcode-and-connected-tuner` still present ⇒ the group mechanism is still the one
-described here. Gone or renamed ⇒ stop and re-measure. In particular the "no `supplementalGroups`
-needed" conclusion in `catalog/plex.yaml` and `schemas/renderer-contract.md` is only true while that
-hook exists.
+described here: the hook reads the injected device's **own** ownership, which under the device plugin
+is `root:root`, so it resolves GID `0` (see "When to escalate or stop"). Gone or renamed ⇒ stop and
+re-measure. In particular the "no `supplementalGroups` needed" conclusion in `catalog/plex.yaml` and
+`schemas/renderer-contract.md` is only true while that hook exists.

 ## When to escalate or stop

 - **Do not add `supplementalGroups` or a `podSecurityContext` to `catalog/plex.yaml`.** It would be
-  **inert**, and it would mislead the next person. Measured in-container 2026-09-30: the image's
-  `cont-init.d` hook stats each `/dev/dri` char device and `usermod -a -G`s the `plex` user into its
-  group *inside* the container, and the service then execs via `s6-setuidgid plex`, which **replaces**
-  the process group list from `/etc/group` rather than appending — `Plex Media Server` runs with
-  `Groups: 100 1000`, root's gid `0` (which PID 1 does hold) gone. Anything Kubernetes supplies is
+  **inert**, and it would mislead the next person. The image's `cont-init.d` hook stats each
+  `/dev/dri` char device and `usermod -a -G`s the `plex` user into **that device's** group, inside
+  the container; the service then execs via `s6-setuidgid plex`, which **rebuilds** the process group
+  list from `/etc/group` rather than appending to it. Anything Kubernetes supplies is therefore
  discarded before Plex starts. `render.py --selftest` asserts that absence deliberately.
+- **GID `992` is a hostPath-era number — do not expect it, and do not read the root group as a
+  defect.** Under the device plugin the injected `/dev/dri/renderD128` arrives **`root:root`**, so
+  the hook resolves GID **`0`** and adds `plex` to the **root group**. Measured 2026-10-02:
+  `Plex Media Server` runs with `Groups: 0 100 1000`, and `id plex` reports
+  `groups=1000(plex),0(root),100(users)`. Under the old hostPath the device was `root:render`
+  (GID 992 on these nodes) and the reading was `Groups: 44 100 992 1000`. **992 is now irrelevant**:
+  if any page, comment or older handoff tells you to expect it, that text is stale and the system
+  you are looking at is working. The root-group membership is an inherent property of the plugin's
+  `DeviceSpec` — it injects the host device with its host ownership — **not** a defect, and **not** a
+  reason to reintroduce `supplementalGroups`, which would still be a no-op for the reason in the
+  bullet above.
 - **Do not reach for `privileged: true`.** Kyverno `restrict-host-namespaces` is Enforce and will
  reject it — and it would not help: `uid 0` failed the same `open()` in the 2026-09-30 measurement,
  because root does not bypass the cgroup device gate.
@@ -390,7 +513,8 @@ hook exists.
  watch state (`policy: daily-offcluster`). The transcode toggle is one preference, not a reason to
  lose the DB.
 - **Stop and get the operator** if the device cannot be opened from the pod as uid 1000 (cause #2
-  confirmed) *under the current mechanism*. The next step is a platform change in
+  confirmed) *under the current mechanism* — note that this open **passed** on 2026-10-02
+  (`RW-OPEN-OK` as uid 1000 via `s6-setuidgid plex`), so a failure now is a regression. The next step is a platform change in
  `platform/intel-gpu-plugin/`, with its own PR, not more poking at this workload.

 ## Known gaps, deliberately not fixed here
@@ -413,10 +537,25 @@ Recorded so they are findable, and so nobody assumes they were overlooked:

 ---
 Source of truth: `runbooks/plex-quicksync.md`. Index: `/runbooks/`
-The node measurements (all three prod nodes) and the in-container image measurements cited here were
-taken **2026-09-30** against `plexinc/pms-docker` digest `sha256:e0ab27395614…`
-(`plexmediaserver 1.43.4.10903`). The `EPERM` open measurements, the `Groups: 44 100 992 1000`
-reading and the `gpu.intel.com/i915: 1` allocatable/DaemonSet readings were taken **2026-10-01**.
-Hardware transcoding has **still not been observed working**: the open test has not been re-run under
-the device-plugin mechanism (it cannot be, until that merges and Flux reconciles), and the Plex-side
-toggle is still unset.
+
+**Hardware transcoding is CONFIRMED WORKING on prod, measured 2026-10-02.** From
+`Plex Media Server.log` in the running pod: API `vaapi` on `/dev/dri/renderD128`
+(`Intel CoffeeLake-S GT2 [UHD Graphics 630]`), `using hardware decode accelerator vaapi`,
+`zero-copy support present`, `using zero-copy transcoding`, and
+`final decoder: vaapi, final encoder: vaapi`. Also measured in the same pod: `RW-OPEN-OK` opening
+`/dev/dri/renderD128` as uid 1000 via `s6-setuidgid plex`; the pod spec carrying
+`{"limits":{"gpu.intel.com/i915":"1"},"requests":{"gpu.intel.com/i915":"1"}}`; `Groups: 0 100 1000`
+and `id plex` → `groups=1000(plex),0(root),100(users)`; and `HardwareDevicePath` present in
+`Preferences.xml` while `HardwareAcceleratedCodecs` was absent from all 27 attributes.
+
+Earlier readings, kept for provenance and **superseded** where they conflict: the node measurements
+(all three prod nodes) and the in-container image measurements taken **2026-09-30** against
+`plexinc/pms-docker` digest `sha256:e0ab27395614…` (`plexmediaserver 1.43.4.10903`); the hostPath
+`EPERM` open results and the `Groups: 44 100 992 1000` / GID-992 group reading, **2026-10-01, under
+the hostPath mechanism that no longer exists**; and the `gpu.intel.com/i915: 1` allocatable and
+`intel-gpu-plugin` DaemonSet readings, **2026-10-01**.
+
+What this page still does **not** establish: that any particular title's software fallback was caused
+by its source codec (cause 6 is an unverified hypothesis), and that Plex's dashboard *ever* labels a
+session `Transcode (hw)` on this deployment — it has not been observed doing so, which is exactly why
+the log is now the primary check.
@@ -0,0 +1,148 @@
+# plex-remote-access — getting Plex reachable from outside the LAN, and proving it
+
+**Status: the in-cluster half is shipped; the last mile is TWO MANUAL STEPS and remote access does
+NOT work until both are done.** Neither can live in this repo.
+**Alert:** none, and none is possible from in-cluster signals — see "Why no alert" below. This is an
+**operator procedure**, not an incident page.
+**Files:** `apps/wave-0/plex-remote.yaml` (the tcp/32400 LoadBalancer + its CiliumNetworkPolicy),
+`clusters/base/cluster-vars.yaml` + `clusters/prod/kustomization.yaml` (`PLEX_L4_VIP`),
+`catalog/render.py` (the `EXTRA_UNITS` entry that places the unit), `catalog/plex.yaml` (Plex's own
+env — note `ADVERTISE_IP` is deliberately NOT set; see "Why not the Gateway URL").
+**Related:** `runbooks/plex-quicksync.md` and `runbooks/plex-claim.md` — the other two Plex
+procedures with a manual last mile.
+
+## What was actually broken (measured 2026-10-03)
+
+Plex was serving correctly and was reachable on the LAN. Two independent things stopped remote
+clients:
+
+1. **Plex advertised no remote connection at all.** In the live `Preferences.xml`:
+   `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while
+   `PublishServerOnPlexOnlineKey="1"`. The server was trying to publish itself to plex.tv and had
+   nothing reachable to publish. Plex's automatic path (UPnP/NAT-PMP) cannot work here: the process
+   lives in a pod on the Cilium overlay (`10.0.2.136`), it cannot speak UPnP to the SOHO router, and
+   a mapping to the pod IP would be meaningless if it could.
+2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` → connection refused.
+   `136.51.115.252:443` → a **self-signed router certificate** (`C=ZZ, CN=%D`), not the Gateway's
+   Let's Encrypt wildcard.
+
+The Service was `ClusterIP` only, so port 32400 had no path off-cluster by any route.
+
+**Caveat on that second measurement, because it is easy to over-read:** it was taken from *inside*
+the LAN, so strictly it proves there is no NAT hairpin (reflection). It does not prove outside-in
+failure. But a router certificate answering 443 means the router's own management UI is bound to the
+WAN port, and that *would* shadow a port-forward for genuine outside traffic too. **Verify from a
+real outside vantage point** (phone on cellular, or an external port checker) before concluding
+either way — this repo has no such vantage point and must not pretend otherwise.
+
+## Step 1 — forward WAN tcp/32400 on the router (operator, off-repo)
+
+    WAN tcp/32400  ->  192.168.2.24  (${PLEX_L4_VIP})  port 32400
+
+`192.168.2.24` is a pinned Cilium LB VIP, ARP-announced on the flat LAN by the existing
+`CiliumL2AnnouncementPolicy default-l2`. It is **pinned on purpose**: this address is typed into the
+router by hand, so it must never move on a reschedule.
+
+While in the router, **check what else is bound to the WAN interface.** If remote management is
+listening on 443 (see the certificate measurement above), move it or disable it — otherwise the
+Gateway's 80/443 forward is shadowed and the Plex *web UI* over the internet stays broken even
+though remote *streaming* works. These are two separate paths and they fail independently.
+
+Confirm the VIP is actually answering on the LAN first, so a failure later is unambiguous:
+
+    ./infra/devcontainer/dev.sh run bash -c \
+      'kubectl --context k3s-prod -n plex get svc plex-l4-tcp -o wide'
+
+`EXTERNAL-IP` must read `192.168.2.24`. If it is `<pending>`, LB-IPAM could not assign — the address
+is outside the pool or already taken; see `clusters/prod/kustomization.yaml` for the pool accounting.
+
+**`192.168.2.24` was never ARP-probed off-cluster.** Only the in-cluster pool accounting was checked
+(IPsTotal 11 / IPsUsed 4 / IPsAvailable 7 on 2026-10-03). If the VIP behaves oddly, probe it from a
+LAN host — `arping 192.168.2.24` — the way `.23` was probed before it was claimed.
+
+## Step 2 — tick "manually specify public port" in Plex (operator, UI only)
+
+**There is no GitOps path to this and no environment variable for it.** Measured in the running
+image on 2026-10-03: a grep of the entire init surface (`/etc/cont-init.d`, `/plex-common.sh`,
+`/usr/local/bin`) for `ManualPortMapping` and `PLEX_PREFERENCE` returned **NONE**. The image writes
+only six preferences, and this is not one of them. It is the same class of manual last mile as the
+`HardwareAcceleratedCodecs` toggle in `runbooks/plex-quicksync.md`.
+
+In the Plex web UI (`https://plex.home.keithjackson.dev`, on the LAN):
+
+**Settings → Remote Access**
+- Ensure **Enable Remote Access** is on. (`PublishServerOnPlexOnlineKey` was already `1`.)
+- Tick **Manually specify public port** and set it to **32400**.
+- **Apply**, then let the panel settle. It should report *"Fully accessible outside your network"*.
+
+Without this tick Plex waits for a NAT mapping it can never obtain and publishes no remote
+connection, so the port-forward in step 1 sits there unused.
+
+## How to verify
+
+Do these in order; each one isolates a different half.
+
+**1. The port is open from outside.** From a phone on cellular, or any off-LAN host:
+
+    curl -sk -o /dev/null -w '%{http_code}\n' https://136.51.115.252:32400/identity
+
+`200` means the forward and the policy are both good. A hang or refusal means step 1.
+
+**2. Plex itself believes it is reachable.** Settings → Remote Access must say *"Fully accessible
+outside your network"*. If it says *"not available outside your network"* while test 1 passes, step 2
+was not applied.
+
+**3. plex.tv is handing out a remote connection.** The decisive test, because it is what clients
+actually consume: sign in to [app.plex.tv](https://app.plex.tv) from an off-LAN network and confirm
+the server appears and plays. A client finds this server through plex.tv's connection list, never
+through DNS.
+
+**4. The policy is admitting the traffic.** If test 1 fails *after* the forward is confirmed, the
+CiliumNetworkPolicy is the next suspect:
+
+    ./infra/devcontainer/dev.sh run bash -c \
+      'kubectl --context k3s-prod -n plex get cnp plex-allow-remote-32400'
+
+See `runbooks/denied-flows.md` for reading actual drops.
+
+## Why no alert
+
+Every in-cluster signal is green whether or not remote access works. The Service has endpoints, the
+pod is `Running`, the policy is applied, and `plex-down` watches
+`kube_deployment_status_replicas_available` — none of which can see a router with a missing
+port-forward or an unticked checkbox. An alert would need an off-LAN prober, which this cluster does
+not have. **Test 3 above is the only honest check, and it is manual.**
+
+## Why not the Gateway URL (`ADVERTISE_IP`)
+
+The alternative was to leave Plex on `ClusterIP` and set
+`ADVERTISE_IP: https://plex.home.keithjackson.dev:443`, which writes Plex's `customConnections`
+preference and would have put the server on plex.tv's list via the existing L7 Gateway — no new WAN
+hole. That mechanism was verified to work on an already-claimed server (the `ADVERTISE_IP` block at
+line 126 of the image's `40-plex-first-run` sits **outside** the claim guard, and `/.firstRunComplete`
+lives on the overlay FS, not the `/config` PVC, so it re-runs every pod start), and the Gateway path
+itself was measured serving Plex correctly: `HTTP 200`, `claimed="1"`, with `401` on every protected
+endpoint.
+
+**The operator chose the native path anyway (2026-10-03), and the reasons are real:** Plex terminates
+its own TLS with its `*.plex.direct` certificate, and video streams bypass the Envoy L7 hop instead
+of being proxied through it. The cost is a new unfiltered WAN port — acceptable because Plex
+authenticates every caller itself. `ADVERTISE_IP` therefore stays unset in `catalog/plex.yaml`, and
+the deferral note there is still accurate.
+
+## Known wart: remote clients are classified as LAN
+
+`apps/wave-0/plex-remote.yaml` uses `externalTrafficPolicy: Cluster`, which SNATs the client to the
+ingress node's address (`192.168.2.1x`). That falls inside the `192.168.0.0/16` entry of
+`ALLOWED_NETWORKS`, so **Plex treats every remote stream as local.** Measured on the Gateway path,
+which SNATs the same way: Plex logged `[10.0.2.214:36040 (Allowed Network (WAN))]`.
+
+This is a **metering** effect, not a security one — authorization is unchanged, as the 401s above
+show — but Plex's remote-quality and bandwidth limits will not bind on these streams.
+
+`Local` would preserve the client IP and fix it. **Do not switch to it.** Cilium's L2 announcer
+elects one lease holder per Service and does *not* follow the backend pod, so the VIP can be
+announced by a node with no Plex pod, black-holing the port while everything reports healthy. That
+exact change cost this LAN an 8-day Minecraft outage in September 2026 — the full write-up is at the
+top of `platform/crafty/service-l4.yaml`. Plex is a single-replica `Recreate` Deployment, so the
+node mismatch is the normal case here, not a rare one.
Author	SHA1	Message	Date
jakman85andClaude Opus 5	3b1d6ae115	feat(pihole): declare the admin password and adlists in git gate / gate (pull_request) Successful in 1m47s Details gate / conformance (pull_request) Has been skipped Details Prerequisite for running more than one PiHole replica. A second replica with a fresh PVC would have come up serving DNS and blocking nothing, with a different admin password -- answers varying by which pod replied. pihole.toml's own ### CHANGED markers gave the exact inventory: four settings already came from env (declared), one came from the image, and exactly ONE operator-set value lived only on the PVC -- webserver.api.pwhash, a one-way hash nobody could recover. Separately the adlists were undeclared: one list and 72,234 gravity domains that nothing in this repo named. Declares both: - A SOPS secret feeding FTLCONF_webserver_api_password. webserver.api.password takes cleartext and FTL hashes it itself (verified: pihole setpassword IS setFTLConfigValue on that key), so the PVC's pwhash becomes derived state. - A level-triggered postStart hook that reconciles the declared adlist set into gravity.db -- adds, re-enables, and disables (never deletes) undeclared lists, running pihole -g only when something moved. A file would not work: v6's adlists.list import is migration-only, guarded on the DB not existing. The rendered hook was run in the live pod against throwaway copies of gravity.db, exercising all four paths. A fresh DB built from the image's own schema template has ZERO adlists, confirming this is load-bearing rather than cosmetic. Password kept rather than set empty: the pihole Service still publishes port 80 past the oauth2-proxy, closed only by policy, and default-deny has drifted here before. Multi-replica is phase 2 and deliberately not in this branch -- it needs a schema field, a StatefulSet (the config PVC is RWO), anti-affinity and a PDB. Adds a *.sops.yaml -- a sensitive path -- so approval-policy will block pending an operator /approve-override. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-04 01:07:30 -05:00
jakman85	939461120a	Merge pull request 'feat(plex): expose tcp/32400 for remote access via a pinned LoadBalancer VIP' (#43 ) from feat/plex-remote-access-l4 into main gate / gate (push) Failing after 1m56s Details gate / conformance (push) Has been skipped Details Reviewed-on: #43 Reviewed-by: Keith Jackson <[email protected]>	2026-10-04 04:46:34 +00:00
jakman85andClaude Opus 5	fbaeb3e038	feat(plex): expose tcp/32400 for remote access via a pinned LoadBalancer VIP gate / gate (pull_request) Successful in 1m49s Details gate / conformance (pull_request) Has been skipped Details Remote Plex clients had no path in, for two measured reasons: Plex advertised no remote connection at all (customConnections and ManualPortMappingMode both ABSENT while PublishServerOnPlexOnlineKey=1), and its Service was ClusterIP, so 32400 had no off-cluster path. The L7 Gateway was NOT the problem -- it serves Plex correctly (200, claimed=1, 401 on protected endpoints) -- but Plex clients discover servers through plex.tv, not DNS, so an HTTPRoute cannot put this server on a client's connection list. Operator chose the native path over ADVERTISE_IP: Plex terminates its own TLS with .plex.direct and streams bypass the Envoy L7 hop. Adds a hand-written non-catalog unit (the SD schema cannot express a LoadBalancer Service or a pinned lbipam VIP), registered in render.py's EXTRA_UNITS so --emit-waves places it in plex's own wave. externalTrafficPolicy: Cluster, inheriting crafty's 8-day-outage lesson rather than re-deriving it. Known consequence, documented: SNAT puts remote clients inside ALLOWED_NETWORKS, so Plex meters them as LAN. Not an auth change. Remote access does NOT work until the operator does two manual steps that cannot live in this repo -- the WAN tcp/32400 forward and Plex's "manually specify public port" toggle (no env var exists for it; verified NONE in the image's init surface). runbooks/plex-remote-access.md is the procedure. Touches clusters/* -- a sensitive path -- so the approval-policy stage will block pending an operator /approve-override. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-03 23:30:46 -05:00
jakman85	7ef2af205d	Merge pull request 'fix(node-baseline): pin the OIDC issuer to the Gateway VIP on prod nodes' (#41 ) from fix/node-keycloak-hosts-pin into main gate / gate (push) Successful in 1m45s Details gate / conformance (push) Has been skipped Details Reviewed-on: #41	2026-10-03 22:46:38 +00:00
jakman85andClaude Opus 5.5	788eaeae62	fix(node-baseline): pin the OIDC issuer to the Gateway VIP on prod nodes gate / gate (pull_request) Successful in 1m46s Details gate / conformance (pull_request) Has been skipped Details After public DNS moved to the WAN IP (#40), prod nodes resolve keycloak.home.keithjackson.dev to a WAN address the router does not hairpin, so the apiserver could not reach its OIDC issuer. Adds an opt-in node_baseline_hosts_pins /etc/hosts block; prod pins keycloak to GATEWAY_VIP. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-10-03 17:39:37 -05:00
jakman85	576e1abb19	Merge pull request 'feat(dns): publish Gateway names to the WAN via in-cluster DDNS' (#40 ) from feat/public-dns-wan-target into main gate / gate (push) Successful in 1m45s Details gate / conformance (push) Has been skipped Details Reviewed-on: #40	2026-10-03 22:25:13 +00:00
jakman85andClaude Opus 5.5	a1e808e697	feat(dns): publish Gateway names to the WAN via in-cluster DDNS gate / gate (pull_request) Successful in 1m46s Details gate / conformance (pull_request) Has been skipped Details Gateway-published records now CNAME to wan.${BASE_DOMAIN}, which a favonia/cloudflare-ddns Deployment keeps at the home WAN's current public IP. Operator decision: every Gateway-published name goes public; internal services (controller: none) remain unpublished. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-10-03 16:59:53 -05:00
jakman85	bcb8931b4f	Merge pull request 'fix(runbooks): Plex hw transcoding WORKS — the dashboard was never the proof' (#39 ) from fix/plex-quicksync-verification into main gate / gate (push) Successful in 1m44s Details gate / conformance (push) Has been skipped Details Reviewed-on: #39 Reviewed-by: Keith Jackson <[email protected]>	2026-10-02 06:43:04 +00:00
jakman85	0d7f94098b	Merge branch 'main' into fix/plex-quicksync-verification gate / gate (pull_request) Successful in 1m46s Details gate / conformance (pull_request) Has been skipped Details	2026-10-02 06:37:04 +00:00
jakman85	5ffd0c98bc	Merge pull request 'fix(ci): let an administrator merge their own green PR — required_approvals 1 → 0, plus an attributed /approve-override' (#38 ) from fix/admin-approval-override into main gate / gate (push) Successful in 1m44s Details gate / conformance (push) Has been skipped Details Reviewed-on: #38 Reviewed-by: Keith Jackson <[email protected]>	2026-10-02 06:17:17 +00:00
jakman85andClaude Opus 5	f0994c11a0	docs(decisions): record the PR's measured CI result — gitea-settings has drifted gate / gate (pull_request) Failing after 1m44s Details gate / conformance (pull_request) Has been skipped Details Read PR #39's own gate log rather than the tick (CLAUDE.md §2b). Every stage this branch could affect is green: decisions-log, lint, l0, render-golden, l1, and approval-policy (which evaluated for real on the pull_request event and passed — no sensitive path touched). The one failure is `gitea-settings`: required_approvals: DECLARED 1 but LIVE is 0 Gitea has DRIFTED from infra/gitea/expected-branch-protection.json Every other declared setting matched, and one runner is online. So live `main` currently requires ZERO approvals while the declaration says 1, and because .gitea/workflows/gate.yml sets REQUIRE_GITEA_SETTINGS=true, that drift blocks EVERY PR in the repo, not only this one. The check that would notice approvals are unenforced is the check that is failing. Not touched here: infra/gitea/** is a sensitive path (non-author approval required) and outside this branch's documentation-only scope. Re-asserting the protection or changing the declaration is an operator decision in its own PR; local branch fix/admin-approval-override (`616a497`) looks like that work in flight. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-02 01:01:13 -05:00
jakman85andClaude Opus 5	b52c8d1af6	fix(ci): accept the override from a PR review, and stop misdiagnosing an unreadable comments API gate / gate (pull_request) Successful in 1m46s Details gate / conformance (pull_request) Has been skipped Details The override did not work on its first real test, and the reason was invisible from the outside. Run 203 on PR #38: a well-formed `/approve-override` posted by jakman85 in the PR comment box, 29 minutes newer than the head commit, and the stage still blocked with: approval-override: COMMENTS_JSON is not a list SETTINGS_API_TOKEN is minted [read:repository, read:admin], and Gitea puts /issues/{n}/comments behind the issue scope — so Gitea answered with an error OBJECT, `jq -e .` accepted it as valid JSON, and the stage fell through to 'no approving review'. A permission failure wearing the costume of a missing override. /pulls/{n}/files and /pulls/{n}/reviews are under read:repository, which is why every other part of the same stage worked. 1. The diagnosis no longer lies. COMMENTS_JSON leaves the must-parse-or-block loop and is checked for being an ARRAY; when it is not, the stage prints Gitea's own .message, names the missing scope and says where to post instead. An unreadable source can only cause a false BLOCK, never a false pass, so a loud warning is the honest behaviour and blocking was not. 2. An override is accepted from a PR REVIEW body as well as a comment, so the hatch is not hostage to one API scope — a review of type 'Comment' works on the token CI has today. The gate names which source it used; PENDING (unsubmitted, author-only) reviews are dropped. 3. set-ci-token.sh requests read:issue, making the ordinary comment box work — on a RE-RUN only, since an existing token's scopes cannot be widened. ci/tests/approval-override.test.sh: 23 -> 38 assertions, covering the reviews source, PENDING reviews, an error object from the comments API, 'an unreadable comments source must not suppress a valid review override', latest-wins across both sources, and that the diagnostic quotes Gitea's message. Also measured and recorded: run 203 shows gitea-settings PASS after the operator ran the live PATCH, with block_admin_merge_override still true — the narrow change held, the approval count moved and the gate's teeth did not. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-02 00:50:20 -05:00
jakman85andClaude Opus 5	1677dff568	fix(runbooks): Plex hw transcoding WORKS — the dashboard was never the proof gate / gate (pull_request) Failing after 1m46s Details gate / conformance (pull_request) Has been skipped Details Hardware transcoding is confirmed working on prod (2026-10-02): VAAPI for both decode and encode on /dev/dri/renderD128, with zero-copy, measured in `Plex Media Server.log` in the running pod. The device-plugin mechanism from #37 works end to end. `runbooks/plex-quicksync.md` said the Plex dashboard showing `Transcode (hw)` was "the only proof available, and nothing else will ever tell you". That is false: Dashboard -> Activity showed nothing while the feature was fully working, so the page's authoritative check was silent on a healthy system and sent the operator hunting a fault that did not exist. Three inference errors corrected, two kept as named false leads: 1. The LOG is the primary check, not the dashboard. The grep ships in its REDACTING form -- that log carries API URLs with `X-Plex-Token` (CLAUDE.md §6). Also recorded: `Plex Transcoder Statistics.log` is the WRONG file (XML <Variant/> records, no ffmpeg output), and why `grep -a` is required. The dashboard is kept but demoted to best-effort, with its silence stated to prove nothing. 2. An fd snapshot is not evidence. A /proc/*/fd sweep found no holder of renderD128 while hardware transcoding was active -- sessions are short and the scan landed in a gap. The reliable process-level check is the transcoder's argv (-hwaccel, -hwaccel_device, vaapi, renderD128). 3. A missing preference key means nothing. `HardwareAcceleratedCodecs` was absent from all 27 attributes with the toggle on and working, while `HardwareDevicePath` was written and the log said "hardware transcoding: enabled". Only `="0"` is conclusive; the log settles it. Also stale and now fixed: GID 992 is a hostPath-era number. Under the plugin the injected device is root:root, so the image's hook resolves GID 0 and adds `plex` to the root group (`Groups: 0 100 1000`, `groups=1000(plex),0(root),100(users)`). Marked as an inherent property of the plugin's DeviceSpec, not a defect, and explicitly not a reason to reintroduce supplementalGroups (still a no-op -- s6-setuidgid rebuilds the group list from /etc/group). Added a per-title fallback entry (UHD 630 has no AV1 decode), labelled a HYPOTHESIS: the title that fell back was never checked. `runbooks/README.md`'s index entry was stale the same two ways and is corrected. Unchanged on purpose: the "STOP HERE IF PLEX IS Pending" section, the escape hatch, and the no-alert-is-possible reasoning (no metricsEndpoint, no ServiceMonitor, nothing a rule could evaluate). Documentation only -- no catalog/, schemas/, apps/ or platform/ change. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-02 00:49:53 -05:00