feat(plex): expose tcp/32400 for remote access via a pinned LoadBalancer VIP

Remote Plex clients had no path in, for two measured reasons: Plex advertised no remote connection at all (customConnections and ManualPortMappingMode both ABSENT while PublishServerOnPlexOnlineKey=1), and its Service was ClusterIP, so 32400 had no off-cluster path. The L7 Gateway was NOT the problem -- it serves Plex correctly (200, claimed=1, 401 on protected endpoints) -- but Plex clients discover servers through plex.tv, not DNS, so an HTTPRoute cannot put this server on a client's connection list. Operator chose the native path over ADVERTISE_IP: Plex terminates its own TLS with *.plex.direct and streams bypass the Envoy L7 hop. Adds a hand-written non-catalog unit (the SD schema cannot express a LoadBalancer Service or a pinned lbipam VIP), registered in render.py's EXTRA_UNITS so --emit-waves places it in plex's own wave. externalTrafficPolicy: Cluster, inheriting crafty's 8-day-outage lesson rather than re-deriving it. Known consequence, documented: SNAT puts remote clients inside ALLOWED_NETWORKS, so Plex meters them as LAN. Not an auth change. Remote access does NOT work until the operator does two manual steps that cannot live in this repo -- the WAN tcp/32400 forward and Plex's "manually specify public port" toggle (no env var exists for it; verified NONE in the image's init surface). runbooks/plex-remote-access.md is the procedure. Touches clusters/** -- a sensitive path -- so the approval-policy stage will block pending an operator /approve-override. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Merge pull request 'fix(node-baseline): pin the OIDC issuer to the Gateway VIP on prod nodes' (#41 ) from fix/node-keycloak-hosts-pin into main
2026-10-03 23:30:46 -05:00 · 2026-10-03 22:46:38 +00:00 · 2026-10-03 17:39:37 -05:00 · 2026-10-03 22:25:13 +00:00 · 2026-10-03 16:59:53 -05:00 · 2026-10-02 06:43:04 +00:00
20 changed files with 1125 additions and 105 deletions
@@ -18,6 +18,7 @@ resources:
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
 - plex-claim-secret.sops.yaml
 - plex-remote.yaml
 - plex.yaml
 - qbittorrent-bootstrap-secret.sops.yaml
 - qbittorrent-oauth2-proxy-credentials.sops.yaml
@@ -0,0 +1,126 @@
 ---
 # =================================================================================================
 # PLEX REMOTE ACCESS — raw tcp/32400 north-south exposure, and the policy that admits it.
 # =================================================================================================
 # HAND-WRITTEN, NOT RENDERED. A non-catalog app-tier unit anchored to the `plex` ServiceDefinition
 # and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS table — the same
 # mechanism gitea-actions and pihole-dns use. It is hand-written for the SAME reason those are:
 # schemas/servicedefinition.schema.json is `additionalProperties: false` and has NO field for a
 # type=LoadBalancer Service, for a pinned lbipam VIP, or for raw-TCP north-south exposure at all.
 # `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator.
 #
 # WHY THE L7 GATEWAY IS NOT ENOUGH HERE — and it is NOT because the Gateway fails to serve Plex.
 # Measured 2026-10-03 through the Gateway VIP: `GET https://plex.home.keithjackson.dev/identity`
 # returned HTTP 200 with `claimed="1"` and the real machineIdentifier, and the protected endpoints
 # (/library/sections, /status/sessions, /:/prefs) all returned 401. The L7 path WORKS.
 #
 # The problem is DISCOVERY, not proxying. Plex apps (phone, TV, Roku) do not resolve
 # `plex.${BASE_DOMAIN}`; they ask plex.tv for the server's connection list. A server gets onto that
 # list one of two ways: it advertises a custom URL (`customConnections`, the ADVERTISE_IP env), or it
 # publishes a reachable public IP:port for its own tcp/32400. Measured in the live Preferences.xml on
 # 2026-10-03: `customConnections` ABSENT, `ManualPortMappingMode` ABSENT, while
 # `PublishServerOnPlexOnlineKey="1"` — the server was trying to publish and had nothing reachable to
 # publish, so remote clients saw NO connection at all. The operator chose the NATIVE path (this
 # file) over advertising the Gateway URL: Plex then terminates its own TLS with its
 # `*.plex.direct` certificate and video streams bypass the Envoy L7 hop entirely.
 #
 # VIP: ${PLEX_L4_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.57, prod 192.168.2.24), PINNED via
 # lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
 # (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the existing
 # CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config is needed;
 # this Service just consumes the pool. PINNING IS NOT OPTIONAL: the address is typed into a WAN
 # port-forward on the SOHO router, which is hand-maintained and off-repo, so it can never move.
 # 192.168.2.24 verified FREE live 2026-10-03 — `default-pool` reported IPsTotal 11 / IPsUsed 4 /
 # IPsAvailable 7, and the only four allocations were .20 (cilium-gateway-shared), .21 (unifi-l4-*),
 # .22 (crafty-l4-*) and .23 (pihole-dns-*).
 #
 # ONE SERVICE, TCP ONLY. Unlike unifi/crafty/pihole-dns this needs no second Service and no
 # lbipam.cilium.io/sharing-key: Plex's remote access is tcp/32400 and nothing else. (Plex's
 # DLNA/GDM discovery uses UDP, but that is LAN-only multicast and is not part of remote access.)
 #
 # NO `loadBalancerSourceRanges`, DELIBERATELY — and this is the one place this file diverges from
 # its closest sibling. apps/wave-0/pihole-dns.yaml pins its source range to ${LAN_CIDR} because a
 # resolver must never answer the internet. This Service is the OPPOSITE case: admitting remote
 # clients from arbitrary internet addresses is its entire purpose, so there is no range to narrow
 # to. Plex authenticates every caller itself (measured above: 401 on every protected endpoint
 # without a token), which is what makes an unfiltered port defensible here.
 #
 # NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
 # `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their VIP as a public
 # Cloudflare record. This Service must not, for pihole-dns's reason: ${PLEX_L4_VIP} is an RFC1918
 # address that is useless in public DNS. Remote clients get this server's address from plex.tv, not
 # from the zone. The existing public `plex.${BASE_DOMAIN}` record is published from the rendered
 # HTTPRoute via the gateway source and is NOT affected by this file — the web UI keeps working
 # through the Gateway exactly as before.
 #
 # -------------------------------------------------------------------------------------------------
 # externalTrafficPolicy: Cluster — inherited from crafty's 8-DAY LAN OUTAGE, not re-derived.
 # -------------------------------------------------------------------------------------------------
 # `Local` is the tempting choice because it preserves the client source IP, which this service
 # actually has a use for (see the allowedNetworks note below). It is still WRONG here, for the
 # reason platform/crafty/service-l4.yaml documents at length: Cilium's L2 announcer elects ONE lease
 # holder per Service and does NOT follow the backend pod, so with `Local` the VIP can be announced
 # by a node that has no Plex pod, which black-holes the port while every object still reports
 # healthy. Plex is a single-replica Deployment with `strategy: Recreate` pinned to wherever the
 # scheduler put it, so the mismatch is the normal case, not a rare one. Take the SNAT.
 #
 # KNOWN CONSEQUENCE OF THAT CHOICE, stated rather than discovered later: `Cluster` SNATs the client
 # to the ingress NODE's address (192.168.2.1x), which falls inside the `192.168.0.0/16` entry of
 # catalog/plex.yaml's ALLOWED_NETWORKS. Plex will therefore classify every REMOTE client as LAN.
 # Measured 2026-10-03 on the Gateway path, which SNATs the same way: Plex logged the request as
 # `[10.0.2.214:36040 (Allowed Network (WAN))]`. This is a METERING effect, not a security one —
 # authorization is unchanged (401s above), but Plex's remote-quality/bandwidth limits will not bind
 # on these streams because it believes they are local. Narrowing ALLOWED_NETWORKS cannot fix it
 # without also de-classifying genuine LAN clients, so it is accepted.
 apiVersion: v1
 kind: Service
 metadata:
  name: plex-l4-tcp
  namespace: plex
  labels:
    app.kubernetes.io/name: plex
  annotations:
    lbipam.cilium.io/ips: "${PLEX_L4_VIP}"
 spec:
  type: LoadBalancer
  externalTrafficPolicy: Cluster
  selector:
    app.kubernetes.io/name: plex
  ports:
    - name: plex-remote
      port: 32400
      targetPort: 32400
      protocol: TCP
 ---
 # INGRESS allow for the raw port (R-NET-01/02).
 #
 # The rendered plex.yaml already carries `plex-allow-gateway-ingress` (fromEntities: ingress ->
 # 32400) and three reciprocal allows for sonarr/radarr/seerr. CiliumNetworkPolicies are ADDITIVE, so
 # this file adds the off-cluster case without touching those.
 #
 # `world` is Cilium's entity for anything outside the cluster, which is what a remote Plex client is.
 # `host` and `cluster` mirror platform/crafty/networkpolicy.yaml's game-port rule and are what make
 # this work under `externalTrafficPolicy: Cluster`: after SNAT the packet's source is the ingress
 # NODE, which Cilium classifies as `host` (same node) or `remote-node` (a different one) — NOT
 # `world`. Listing `world` alone would compile cleanly and then drop every remote stream, so all
 # four entities are named.
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
 metadata:
  name: plex-allow-remote-32400
  namespace: plex
  labels:
    app.kubernetes.io/name: plex
 spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: plex
  ingress:
    - fromEntities:
        - world
        - cluster
        - host
        - remote-node
      toPorts:
        - ports:
            - { port: "32400", protocol: TCP }
@@ -2002,6 +2002,24 @@ EXTRA_UNITS: dict = {
            "pihole-dns.yaml",
        ],
    },
    # Plex's remote-access exposure: the raw tcp/32400 LoadBalancer Service on a pinned VIP, plus the
    # CiliumNetworkPolicy that admits off-cluster clients to it (apps/wave-0/plex-remote.yaml).
    # `above: False` co-locates it in plex's OWN wave — these are the plex pod's own Service and its
    # own policy, applied in the same wave as the Deployment they select, exactly as the rendered
    # Service and CNPs in plex.yaml are. Nothing in a higher wave gates on it.
    #
    # It is a non-catalog unit for the same reason pihole-dns is: the ServiceDefinition schema cannot
    # express a LoadBalancer Service or a pinned lbipam VIP (additionalProperties is false, and
    # `ingressRoutes` only renders HTTPRoutes on the L7 Gateway, which cannot carry Plex's raw port).
    # Plex's own settings stay in the catalog as env on catalog/plex.yaml; the only thing living
    # outside it is the exposure the schema has no word for.
    "plex-remote": {
        "anchor": "plex",
        "above": False,
        "files": [
            "plex-remote.yaml",
        ],
    },
 }
@@ -100,7 +100,8 @@ data:
  # platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
  # tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
  # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
-  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers),
+  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers;
  # plex-remote: PLEX_L4_VIP),
  # velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
  # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
  # UNSUBSTITUTED path would ship the literal string into the cluster.
@@ -175,6 +176,19 @@ data:
  # dev  -> 192.168.1.56  (base default; next free after crafty .55 in dev's .53-.60 pool)
  # prod -> 192.168.2.23  (next free after crafty .22 in prod's .20-.30 pool; see clusters/prod)
  PIHOLE_DNS_VIP: "192.168.1.56"             # PiHole: authoritative LAN DNS for ${BASE_DOMAIN}
  # Plex's remote-access listener: raw tcp/32400 for REMOTE Plex clients (apps/wave-0/plex-remote.yaml).
  # A FOURTH pinned L4 VIP, for the usual reason — the `shared` Gateway is an L7 HTTPS terminator and
  # cannot carry Plex's own TLS. Plex clients do not resolve `plex.${BASE_DOMAIN}`; they fetch this
  # server's address from plex.tv, so the Gateway's L7 path (which still serves the web UI, and was
  # measured working) cannot put the server on a client's connection list at all.
  #
  # PINNED, AND THE PIN IS LOAD-BEARING OFF-REPO: the SOHO router forwards WAN tcp/32400 to this
  # address by hand. A VIP taken from the pool head would silently move on a reschedule and break
  # remote access with nothing in-cluster looking wrong. See runbooks/plex-remote-access.md.
  #
  # dev  -> 192.168.1.57  (base default; next free after pihole-dns .56 in dev's .53-.60 pool)
  # prod -> 192.168.2.24  (next free after pihole-dns .23 in prod's .20-.30 pool; see clusters/prod)
  PLEX_L4_VIP: "192.168.1.57"                # Plex: tcp/32400 remote access (NOT the web UI, which is L7)
  # The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
  # peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable
@@ -476,6 +476,26 @@ patches:
      - op: replace
        path: /data/PIHOLE_DNS_VIP
        value: "192.168.2.23"
      # Plex's remote-access listener VIP — raw tcp/32400 for REMOTE Plex clients
      # (apps/wave-0/plex-remote.yaml). 192.168.2.24 is the NEXT FREE address in the prod pool
      # 192.168.2.20-.30, verified live 2026-10-03: the CiliumLoadBalancerIPPool `default-pool`
      # reported IPsTotal 11 / IPsUsed 4 / IPsAvailable 7, and the only four allocations are .20
      # (cilium-gateway-shared), .21 (unifi-l4-tcp/-udp), .22 (crafty-l4-tcp/-udp) and .23
      # (pihole-dns-tcp/-udp). It is inside the pool block the operator already asserted sits outside
      # the SOHO router's DHCP scope, and it is not a node address (.10/.11/.12) or the
      # control-plane VIP (.9).
      #
      # UNLIKE .23, THIS ONE IS ALSO A WAN PORT-FORWARD TARGET. The router forwards WAN tcp/32400
      # here, by hand, off-repo — so changing this value is a two-place edit and the router is the
      # half that CI cannot check. runbooks/plex-remote-access.md is the procedure.
      #
      # NOT ARP-PROBED. .23 was verified free with `arping` from a LAN host before it was claimed;
      # for .24 only the in-cluster pool accounting above was checked, so an off-cluster squatter on
      # this address would not have been caught. Probe before trusting it if remote access comes up
      # with a conflict.
      - op: replace
        path: /data/PLEX_L4_VIP
        value: "192.168.2.24"
      # The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns —
      # NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path.
      # It scopes the ingress allow for PiHole's DNS listener to LAN clients only.
@@ -0,0 +1,149 @@
 ---
 date: 2026-10-02
 branch: fix/plex-quicksync-verification
 title: Hardware transcoding works — and `runbooks/plex-quicksync.md`'s verification section was wrong
 ---
 ## What changed
 **Hardware transcoding is confirmed working on prod.** Measured 2026-10-02 from
 `Plex Media Server.log` in the running pod: `vaapi` on `/dev/dri/renderD128`
 (`Intel CoffeeLake-S GT2 [UHD Graphics 630]`), `using hardware decode accelerator vaapi`,
 `zero-copy support present`, `using zero-copy transcoding`, and
 `final decoder: vaapi, final encoder: vaapi`. VAAPI for **both** directions, zero-copy, so frames
 never round-trip through the CPU. The device-plugin mechanism from PR #37 works end to end.
 **And `runbooks/plex-quicksync.md`'s verification section was wrong, which is the correction this
 branch exists to publish.** The page said the Plex dashboard showing `Transcode (hw)` was "the *only*
 proof available, and nothing else will ever tell you". The operator had hardware transcoding fully
 working while **Dashboard → Activity showed nothing of the sort** — so the page's authoritative check
 was silent on a working system, and sent someone looking for a fault that did not exist. Three
 inference errors are now corrected on the page, two of them recorded as named false leads so the next
 reader does not re-walk them:
 1. **The dashboard is not proof.** Replaced as the primary check by Plex's own log — the grep is in
   the page, in its **redacting** form, because `Plex Media Server.log` carries API URLs containing
   `X-Plex-Token` (CLAUDE.md §6; a credential has been leaked into a transcript here before). The
   page also now says explicitly that **`Plex Transcoder Statistics.log` is the wrong file** — it
   holds XML `<Variant .../>` records, not ffmpeg output, and the name invites the mistake — and why
   `grep -a` is required. The dashboard is **kept but demoted** to a secondary, best-effort signal,
   with its absence stated to prove nothing.
 2. **An fd snapshot is not evidence.** A `/proc/*/fd` sweep inside the pod found no holder of
   `renderD128` while hardware transcoding was active; transcode sessions are short and the scan
   landed in a gap. Recorded as a documented false lead, with the reliable process-level check in its
   place: the transcoder's **argv** (`-hwaccel`, `-hwaccel_device`, `vaapi`, `renderD128`), found via
   `ps -eo pid,user,comm | grep -i transcoder`.
 3. **A missing preference key does not mean the setting is off.** The page told readers to treat an
   absent `HardwareAcceleratedCodecs` in `Preferences.xml` as "never enabled". With hardware
   acceleration demonstrably on and working, that key was **absent from all 27 attributes** while
   `HardwareDevicePath` **was** written and the log said `hardware transcoding: enabled`. First-command
   #4 now reads both keys and states plainly that `HardwareDevicePath` present is a positive signal,
   absence of `HardwareAcceleratedCodecs` means nothing either way, only `="0"` is conclusive, and the
   log settles it.
 Also on the page:
 - **GID `992` is no longer presented as a number to expect.** Under the plugin the injected device
  arrives `root:root`, so the image's `cont-init.d` hook resolves GID **`0`** and adds `plex` to the
  **root group** — measured `Groups: 0 100 1000`, `id plex` → `groups=1000(plex),0(root),100(users)`.
  992 was the hostPath-era host `render` GID. The page now says so, marks the root-group membership as
  an inherent property of the plugin's `DeviceSpec` rather than a defect, and repeats that it is
  **not** a reason to reintroduce `supplementalGroups` (still a no-op: `s6-setuidgid` rebuilds the
  group list from `/etc/group`). The stale `Groups: 100 1000` claim in the escalation section is gone.
 - **A new troubleshooting entry for legitimate per-title fallbacks**, marked as a **hypothesis**.
  Between two fully hardware sessions, one logged `enabled, but no hardware decode accelerator found`
  and empty final decoder/encoder. UHD 630 decodes H.264, HEVC 8/10-bit and VP9 but **not AV1** and
  not 10-bit 4:2:2 HEVC, so the likely cause is the source file, not the platform — explicitly
  written as a lead to check first, **not** a confirmed diagnosis of that item.
 - The status is stated at the top of the page and in a rewritten provenance footer that separates the
  2026-10-02 confirmations from the superseded 2026-09-30/2026-10-01 hostPath-era readings.
 - `runbooks/README.md`'s index entry for the page was stale in the same two ways (it still described a
  `/dev/dri` hostPath and the dashboard-only proof) and is corrected to match.
 Unchanged on purpose: the "STOP HERE IF PLEX IS `Pending`" section and the escape hatch (delete the
 `hardware:` block from `catalog/plex.yaml`, re-render) keep exactly their existing prominence — they
 are correct and they are the 2am material. The "no alert is possible" reasoning is also kept verbatim
 in substance: Plex declares no `metricsEndpoint`, renders no ServiceMonitor, and a fallback alert
 would be a rule evaluating against nothing.
 **Documentation only.** No change to `catalog/`, `schemas/`, `apps/` or `platform/`.
 ## What I verified
 - **Gate, in the pinned toolbox from this worktree** (`DEV_KUBECONFIG=/dev/null`, `ENGINE` shim
  bind-mounting the main checkout's `.git`): **rc 3 UNDETERMINED**, diff base
  `origin/main @ 948875aad37d`, with exactly the three expected skips and **no failures**. Green:
  `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, `approval-policy`. Skipped:
  `spec-immutability` (spec repo unavailable → R-TEST-06 not verified), `gitea-settings`
  (`SETTINGS_API_TOKEN` unset), `conformance` (`CLUSTER_AVAILABLE!=true` → L2/L3 not run).
  Touched requirements: **R-OBS-04**, reported `UNVERIFIED` for lack of a cluster. `approval-policy`
  passed as `N/A` locally (`event='none'`, not a `pull_request`) — it evaluates for real on the PR.
  **rc 3 is not green**, it is the calibrated worktree result, and it is reported here as rc 3.
 - **The gate was run three times, with the same rc 3 / three-skip result on each committed tip** —
  including the final one. Worth recording from the first attempt: run *before* committing, with
  `HEAD == origin/main`, the touched-requirement set is `NOT DETERMINABLE (on-base)` and the
  R-TEST-03 coverage gate cannot be applied at all. Commit before gating from a fresh worktree branch.
 - **`runbooks/kustomization.yaml` still carries `- plex-quicksync=plex-quicksync.md`** in the
  `configMapGenerator` `files:` list (line 110) — checked explicitly, because without it the page
  merges and still 404s. `README=README.md` is likewise present, which is what serves the corrected
  index entry.
 - **This change touches no `sensitive_paths`.** `infra/gitea/approval-policy.json` lists `ci/**`,
  `.gitea/**`, `conformance/**`, `clusters/**`, `bootstrap/**`, `infra/gitea/**`,
  `platform/kyverno/**`, `platform-config/kustomization.yaml`, `.sops.yaml`, `**/*.sops.yaml`,
  `**/rbac.yaml`, `CLAUDE.md`. `runbooks/**` and `decisions/**` are not among them, and the
  `approval-policy` stage passed.
 - **No YAML was added or edited**, so `indent-sequences: consistent` (which blocked a sibling branch
  today) is not in play here; `lint` passed over the full tracked YAML set regardless.
 - **`DECISIONS-LOG.md` is untouched** — this handoff is its own file, per CLAUDE.md §7.
 - **The PR's own CI run was read, not assumed** (PR #39, run 208, task 261, head `1677dff5688b`).
  **It is RED, and not because of this branch.** Every stage this change could affect is green —
  `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, `approval-policy` (`approval-policy` evaluated
  for real on the `pull_request` event and passed: no sensitive path touched). The single failure is
  **`gitea-settings`**, quoting the measured field:
      [fail]   required_approvals: DECLARED 1 but LIVE is 0
      [fail] Gitea has DRIFTED from infra/gitea/expected-branch-protection.json.
      [fail] MERGE GATE: BLOCKED
  Every other declared setting matched (`block_admin_merge_override`, `block_on_rejected_reviews`,
  `dismiss_stale_approvals`, `enable_status_check`, `enable_push=false`,
  `status_check_contexts = ["gate / gate (pull_request)"]`, `runners online: 1`). So **live `main`
  currently requires ZERO approvals**, the declaration still says 1, and with
  `REQUIRE_GITEA_SETTINGS: "true"` in `.gitea/workflows/gate.yml` that drift **blocks every PR in
  the repo**, not just this one. Note the shape of it: the check that would notice approvals are
  unenforced is the check that is failing. Local branch `fix/admin-approval-override` (`616a497`,
  "fix(ci): let an administrator merge their own green PR") looks like the in-flight work that means
  to change the declaration. **I did not touch it** — `infra/gitea/**` is a sensitive path, needs a
  non-author approval, and is outside this branch's documentation-only scope. It is an operator
  decision: re-assert the protection, or change the declaration deliberately in its own PR.
 - The repository-wide grep for `992` to find what else still asserts it: remaining occurrences are in
  `catalog/plex.yaml`, `catalog/render.py`, `schemas/renderer-contract.md`,
  `platform/intel-gpu-plugin/README.md`, `platform/intel-gpu-plugin/kustomization.yaml`, and three
  older `decisions/` entries.
 ## What I did NOT verify
 - **I did not re-measure anything on the live cluster in this session.** Every platform number on the
  page — the VAAPI/zero-copy log lines, `RW-OPEN-OK` as uid 1000, the `gpu.intel.com/i915` limits and
  requests on the pod spec, `Groups: 0 100 1000`, `id plex`, the 27 `Preferences.xml` attributes, the
  empty `/proc/*/fd` sweep — came to me as measurements taken earlier on 2026-10-02 and was
  transcribed, not re-run. Per CLAUDE.md §2b, treat the page as reporting those readings rather than
  as proof of current state; re-run the primary check before relying on it.
 - **The per-title fallback cause is unverified and labelled as such.** The source codec of the title
  that logged `no hardware decode accelerator found` was never read. The AV1 / 10-bit-4:2:2-HEVC
  explanation is inferred from UHD 630's capability set alone.
 - **Whether Plex's dashboard *ever* shows `Transcode (hw)` on this deployment is unknown.** It was
  observed silent during working hardware transcoding; whether that is a client/version quirk, a
  reporting bug, or something about this setup was not investigated. The page therefore demotes it
  rather than explaining it.
 - **The commands as written in the page were not executed from the page.** They are the forms that
  were run, re-wrapped for `dev.sh`; in particular the `exec -it deploy/plex -- sh` route for the argv
  check is noted as possibly lacking a TTY under `dev.sh run` and was not exercised in that shape.
 - **Stale `992` copies outside `runbooks/` were left alone**, deliberately — scope was documentation
  only, and `catalog/plex.yaml`, `catalog/render.py`, `schemas/renderer-contract.md` and
  `platform/intel-gpu-plugin/` all still carry the hostPath-era `Groups: 44 100 992 1000` reading as
  if current. `render.py --selftest` asserts around that comment, so correcting it is a real change
  with gate consequences, not a doc edit. **That is open work.** Older `decisions/` entries also cite
  992; those are historical records and should stay as written.
 - **CI on the PR.** The branch is pushed and the PR opened; I did not watch the Gitea run report, so
  nothing here should be read as "CI green". The only gate result I can quote is the local rc 3 above.
@@ -0,0 +1,45 @@
 ---
 date: 2026-10-03
 branch: fix/node-keycloak-hosts-pin
 title: Pin the OIDC issuer to the Gateway VIP on prod nodes — the WAN answer does not hairpin
 ---
 ## What changed
 Follow-up to PR #40 (`decisions/2026-10-03-public-dns-wan-target.md`). Public DNS for
 `*.home.keithjackson.dev` now CNAMEs to the WAN IP. The prod nodes resolve through public DNS,
 and the router does not hairpin, so the kube-apiserver could no longer reach its OIDC issuer
 `keycloak.home.keithjackson.dev`.
 - `roles/node_baseline`: a new `node_baseline_hosts_pins` list (default `[]`) and one
  `blockinfile` task (tags `dns`, `hosts`). It writes the pins into `/etc/hosts` as a single
  marked block. An empty list removes the block.
 - `group_vars/k3s_prod/vars.yml`: pins `keycloak.{{ k3s_base_domain }}` → `192.168.2.20`, which
  must stay equal to `GATEWAY_VIP` (parity comment added). It also replaces the stale "no
  /etc/hosts pin is needed" paragraph with the measured reason the pin is now required.
 Dev is unaffected (empty default).
 ## What I verified
 - After PR #40 reconciled (`platform` lastAppliedRevision `576e1ab`, Healthy=True), on
  k3s-prod-2 and k3s-prod-3: `getent hosts keycloak.home.keithjackson.dev` →
  `136.51.115.252 wan.home.keithjackson.dev`. `curl https://keycloak…/realms/homelab/.well-known/openid-configuration`
  → exit 60, `ssl_verify_result=18` (the router's self-signed cert, not the Gateway).
 - `journalctl -u k3s` (last 20 min) showed no OIDC/JWKS errors yet. The apiserver still holds
  the keys it cached before the flip, so the failure is latent until a k3s restart or key rotation.
 - Neither node has cloud-init (`/etc/cloud` absent), so nothing rewrites `/etc/hosts` at boot.
 - `--check --diff --tags hosts --limit k3s-prod-2,k3s-prod-3`: each node gets exactly the
  three-line marked block `192.168.2.20	keycloak.home.keithjackson.dev`.
 - Outside-in, PR #40 works: check-host.net probes from 3 external nodes got `200 OK` from
  `https://plex.home.keithjackson.dev/identity` at `136.51.115.252`.
 ## What I did NOT verify
 - **k3s-prod-1**: Ansible cannot reach it. SSH host key verification fails (`known_hosts:94`,
  ED25519 `SHA256:zgUS8OeWznzCX9hlN7XdZ8iLNkU1G2loXjtJFfEUYpE`), which is consistent with
  the 2026-09-11 SSD replacement and reinstall. The operator must confirm the fingerprint
  before the pin can be applied there.
 - The playbook has not been applied for real. It is an operator run after merge:
  `ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`.
 - An end-to-end kubectl OIDC login after the pin (needs a browser).
@@ -0,0 +1,129 @@
 ---
 date: 2026-10-03
 branch: feat/plex-remote-access-l4
 title: Plex remote access — raw tcp/32400 via a pinned LoadBalancer VIP, not the L7 Gateway
 ---
 ## What changed
 Reported symptom: users cannot reach Plex from outside the LAN. **Two independent causes**, measured
 rather than inferred:
 1. **Plex advertised no remote connection at all.** Live `Preferences.xml` on prod:
   `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while
   `PublishServerOnPlexOnlineKey="1"` — publishing was on and there was nothing reachable to publish.
   Plex's automatic UPnP/NAT-PMP path cannot work from a pod on the Cilium overlay.
 2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` refused;
   `136.51.115.252:443` answered with a **self-signed router cert** (`C=ZZ, CN=%D`), not the
   Gateway's LE wildcard. The Plex Service was `ClusterIP` only — no off-cluster path to 32400 by
   any route.
 This branch fixes the in-cluster half. **Remote access does not work until the operator also does
 two manual steps** (`runbooks/plex-remote-access.md`): the WAN tcp/32400 port-forward, and ticking
 "manually specify public port" in the Plex UI.
 - **`apps/wave-0/plex-remote.yaml` (new, hand-written).** A `type=LoadBalancer` Service on tcp/32400
  pinned to `${PLEX_L4_VIP}` via `lbipam.cilium.io/ips`, plus a `CiliumNetworkPolicy`
  (`plex-allow-remote-32400`) admitting `world`/`cluster`/`host`/`remote-node` to 32400. Non-catalog
  because `schemas/servicedefinition.schema.json` is `additionalProperties: false` and has no field
  for a LoadBalancer Service or a pinned VIP — the same reason `apps/wave-0/pihole-dns.yaml`,
  `platform/crafty/service-l4.yaml` and `apps/vms/unifi/50-service-l4.yaml` are hand-written.
 - **`catalog/render.py`:** an `EXTRA_UNITS` entry (`plex-remote`, anchored to `plex`, `above: False`)
  so `--emit-waves` places the unit in plex's own wave.
 - **`clusters/base/cluster-vars.yaml`:** new `PLEX_L4_VIP` (dev default `192.168.1.57`), plus the
  substitution inventory comment updated to name `plex-remote: PLEX_L4_VIP`.
 - **`clusters/prod/kustomization.yaml`:** prod override `PLEX_L4_VIP: 192.168.2.24`.
 - **`apps/wave-0/kustomization.yaml`:** `plex-remote.yaml` added to `resources`.
 - **`runbooks/plex-remote-access.md` (new):** the two manual steps, the four-step verification, why
  no alert is possible, and the two documented warts below.
 **`ADVERTISE_IP` was deliberately NOT set.** Setting it to `https://plex.home.keithjackson.dev:443`
 was the alternative — it needs no new WAN hole and the mechanism was verified working on an
 already-claimed server. **The operator chose the native path (2026-10-03)** so Plex terminates its own
 TLS (`*.plex.direct`) and streams bypass the Envoy L7 hop. `catalog/plex.yaml`'s deferral note for
 `ADVERTISE_IP` remains accurate and untouched.
 **`externalTrafficPolicy: Cluster`, with a known cost.** `Local` would preserve the client IP, but
 Cilium's L2 announcer does not follow the backend pod — the wart that cost this LAN an 8-day
 Minecraft outage (see `platform/crafty/service-l4.yaml`). Consequence accepted and documented: SNAT
 puts the client in `192.168.0.0/16`, inside `ALLOWED_NETWORKS`, so **Plex classifies remote streams
 as LAN**. That is a metering effect (remote bandwidth limits will not bind), not an authorization one.
 **SENSITIVE PATHS — this PR touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK
 it.** It needs the operator's `/approve-override <reason>` posted as a PR **review** of type
 "Comment". I have deliberately not posted it: CLAUDE.md §1 forbids an agent clearing this on its own
 PR, and the agent account holds `owner`, so the prohibition is a rule, not a limit.
 ## What I verified
 All measurements against live prod, 2026-10-03, read-only.
 - **Plex's advertised state**, the finding this branch rests on: `customConnections=<ABSENT>`,
  `ManualPortMappingMode=<ABSENT>`, `ManualPortMappingPort=<ABSENT>`,
  `PublishServerOnPlexOnlineKey="1"`, `allowedNetworks="10.0.0.0/8,192.168.0.0/16"`. Read by
  attribute grep, never by dumping `Preferences.xml` (it holds `PlexOnlineToken`).
 - **The L7 Gateway path works and is NOT the problem.** Via the Gateway VIP,
  `GET https://plex.home.keithjackson.dev/identity` → `HTTP 200`, `claimed="1"`, real
  `machineIdentifier`. HTTPRoute `plex-32400`: `Accepted=True`, `ResolvedRefs=True`.
 - **Plex enforces its own auth**, which is what makes an unfiltered WAN port defensible:
  `/library/sections`, `/status/sessions`, `/:/prefs` all → **401** unauthenticated through the
  Gateway.
 - **The SNAT/allowedNetworks wart is real, not theoretical.** A tagged probe through the Gateway was
  logged by Plex as `[10.0.2.214:36040 (Allowed Network (WAN))]` — the Envoy pod IP, classified as an
  allowed network.
 - **WAN state:** `136.51.115.252:32400` → connection refused. `136.51.115.252:443` → self-signed
  cert `C=ZZ, ST=Somewhere, O=%D342e7e6b, CN=%D`. Public DNS resolves correctly:
  `plex.home.keithjackson.dev` → CNAME `wan.home.keithjackson.dev` → `136.51.115.252`.
 - **`192.168.2.24` is free in-cluster.** `CiliumLoadBalancerIPPool default-pool`: IPsTotal 11 /
  IPsUsed 4 / IPsAvailable 7; the only allocations are `.20` (cilium-gateway-shared), `.21`
  (unifi-l4-*), `.22` (crafty-l4-*), `.23` (pihole-dns-*). Not a node address (`.10/.11/.12`) and not
  the kube-vip control-plane VIP (`.9`).
 - **The manual-port-mapping step genuinely cannot be automated.** Grep of the running image's whole
  init surface (`/etc/cont-init.d`, `/plex-common.sh`, `/usr/local/bin`) for `ManualPortMapping` and
  `PLEX_PREFERENCE` → **NONE**.
 - **`ADVERTISE_IP` would have worked on a claimed server** (recorded because the runbook's
  "why not" rests on it): `setPref "customConnections"` at line 126 of `40-plex-first-run` sits
  **outside** the `PLEX_CLAIM`/`PlexOnlineToken` guard, and the `/.firstRunComplete` early-exit
  marker is on the **overlay** FS, not the `/config` PVC, so the script re-runs every pod start.
 - **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
  `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
  **pass**. The rc 3 is two environmental skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`)
  and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3, not as pass.**
 - **`render-golden` passing is the specific proof** that the `render.py` `EXTRA_UNITS` change, the new
  hand-written unit and the edited wave kustomization are mutually consistent.
 ## What I did NOT verify
 - **THE ACTUAL FIX. Nothing in this branch has been proven to make remote access work** — it cannot
  be, because both remaining steps are off-repo and manual (router forward, Plex UI toggle). This
  branch removes two blockers and adds none; it does not demonstrate an outcome.
 - **Outside-in reachability, at all.** Every probe above was taken from inside the LAN, so the
  `:443` self-signed cert strictly proves only that there is **no NAT hairpin** — it does **not**
  prove outside-in failure. A router cert on WAN:443 is suggestive (its management UI would shadow a
  forward), but unproven. **This repo has no off-LAN vantage point.** Test 3 in the runbook
  (app.plex.tv from a cellular connection) is the only honest check.
 - **`192.168.2.24` was never ARP-probed off-cluster.** Only in-cluster pool accounting was checked.
  `.23` was `arping`-probed before it was claimed; `.24` was not, so an off-cluster squatter would
  not have been caught.
 - **The LoadBalancer has never been reconciled.** `plex-l4-tcp` does not exist in prod yet, so
  LB-IPAM has not actually assigned the pinned VIP, L2 announcement has not been observed for it, and
  the new CiliumNetworkPolicy has never admitted a real packet. Whether `world`/`host`/`cluster`/
  `remote-node` is the correct entity set **under this SNAT path** is reasoned from
  `platform/crafty/networkpolicy.yaml`'s working game-port rule, **not measured**.
 - **Whether Plex's remote path needs anything beyond the manual port tick.** If plex.tv still lists
  no remote connection after both manual steps, this branch is necessary-but-insufficient.
 - **The CI approval-policy outcome.** The local stage reported green, but that is an offline N/A, not
  a verdict: this PR touches `clusters/**`, so CI will block it pending an override.
 - **PRE-EXISTING DRIFT I FOUND AND DELIBERATELY LEFT ALONE — worth its own follow-up.** Running the
  documented `python3 catalog/render.py --emit-waves` regenerates the wave kustomizations and wants to
  add **`vm-demo.yaml`** to wave 0 and **`keycloak.yaml`** to wave 1. Both are listed as
  intentionally not onboarded in `ci/render-golden.sh`
  (`RENDER_GOLDEN_NOT_ONBOARDED="keycloak vm-demo"`); their manifests are tracked in `apps/` but
  deliberately absent from the `resources` lists so Flux does not apply them. `--emit-waves` places
  any file present on disk and has **no awareness of that exclusion**, while `render-golden.sh` uses
  `--wave-layout` and compares rendered manifests only — it never checks kustomization `resources`,
  which is why this has survived unnoticed. **So the header "DO NOT hand-edit: re-run
  `render.py --emit-waves`" is currently unsafe advice**: following it onboards an IdP and a KubeVirt
  VM. I therefore restored both kustomizations and hand-added only my one line. I did not fix
  `--emit-waves`, and I have not determined how long the drift has existed or whether either service
  is *meant* to be onboarded.
@@ -0,0 +1,73 @@
 ---
 date: 2026-10-03
 branch: feat/public-dns-wan-target
 title: Public DNS now points at the WAN — Gateway-published names CNAME to an in-cluster DDNS record
 ---
 ## What changed
 The operator closed the open "public-target gap" recorded in
 `platform/externaldns/helmrelease.yaml`. The WAN NAT / port-forwarding (80/443 → Gateway VIP
 192.168.2.20) was done by the operator outside this repo.
 Operator decisions (2026-10-03):
 1. **Scope: every Gateway-published name goes public.** That covers the five `exposure: public`
   apps (plex, gitea, keycloak, nextcloud, seerr) and also the hand-authored admin UIs (auth,
   crafty, grafana, headlamp, the `home.keithjackson.dev` apex). `internal` services still publish
   nothing: their `controller: none` route annotation is checked before the target is read.
 2. **The WAN IP is kept current in-cluster**, not hardcoded.
 Implementation:
 - `platform/gateway/gateway.yaml`: `external-dns.alpha.kubernetes.io/target: wan.${BASE_DOMAIN}` on
  the shared Gateway. Every gateway-sourced record changes from `A 192.168.2.20` to
  `CNAME wan.home.keithjackson.dev`.
 - `platform/externaldns/ddns.yaml` (new): `favonia/cloudflare-ddns:1.17.1` Deployment in the
  `external-dns` namespace. It reuses `cloudflare-api-token` and keeps `wan.${BASE_DOMAIN}` an
  unproxied IPv4 A record at the WAN IP (detected via Cloudflare trace, checked every 5 min,
  `DELETE_ON_STOP=false`). external-dns has no TXT registry entry for it, so it never owns or
  deletes it.
 - The helmrelease comment block and the README now describe the decision instead of the gap.
 Known consequence: **k3s nodes resolve through public DNS** (`node_baseline_dns_servers` = 8.8.8.8 /
 1.1.1.1). The kube-apiserver's OIDC issuer `keycloak.home.keithjackson.dev` therefore resolves to the
 WAN IP from the nodes, and kubectl SSO works only if the router hairpins WAN:443 back to the Gateway.
 A LAN-side probe of `136.51.115.252:443` on 2026-10-03 returned a **self-signed placeholder cert**,
 not the Gateway's Let's Encrypt wildcard. That suggests no hairpin. The cert-based admin kubeconfig
 is unaffected.
 ## What I verified
 - Live prod, 2026-10-03, before the change: Gateway `gateway/shared` ADDRESS `192.168.2.20`.
  External-dns args include `--source=gateway-httproute --source=service --policy=sync
  --txt-owner-id=prod --domain-filter=keithjackson.dev`, and it logs "All records are already up to
  date".
 - Public answers at 1.1.1.1: plex/gitea/keycloak/grafana/auth/apex → `192.168.2.20`;
  `crafty-mc` → `.22` and `unifi` → `.21` (service source, not affected by this change); jackett →
  none (internal, correctly unpublished).
 - The authority (betty.ns.cloudflare.com) has no unprefixed TXT, AAAA or MX at `plex.` or the apex,
  so the A→CNAME switch has no co-existing record to collide with.
 - WAN IP from the workstation: `136.51.115.252`.
 - No CiliumNetworkPolicy or NetworkPolicy in `external-dns`. The Kyverno label and default-deny
  policies match only `app-namespace` namespaces. The vpn egress policy selects only
  `egress-class: vpn` pods.
 - Toolbox gate result: see the PR description (run after this commit).
 ## What I did NOT verify
 - **The outside-in path.** I have no vantage point outside the LAN. Whether
  `https://<svc>.home.keithjackson.dev` from the internet reaches the Gateway with the LE cert is
  UNVERIFIED.
 - **Hairpin / kubectl SSO after the flip.** Likely broken per the probe above. Fix candidates: a
  node `/etc/hosts` pin for `keycloak.${BASE_DOMAIN}` → Gateway VIP via `node_baseline`, or router
  NAT reflection.
 - **The record-type flip itself**, until Flux reconciles: whether Cloudflare accepts external-dns's
  create-CNAME in the same batch as the delete-A, or converges one interval later.
 - `favonia/cloudflare-ddns` under this pod securityContext. The README warns that
  `no-new-privileges` misbehaves on some Proxmox-style hosts. These are bare-metal nodes, but it is
  untested until the pod runs.
 - Whether the L4 service-source names (crafty-mc, unifi) should also go public. Not decided, not
  changed.
 - `conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh` header lines 39–40
  still describe the target as open. That is a sensitive path, so it is left for a follow-up.
@@ -306,19 +306,27 @@ node_baseline_dns_servers:
  # k3s_base_domain, below). It reaches the issuer through the NODE resolver, i.e. through
  # these PUBLIC upstreams — not through CoreDNS's split-horizon answer.
  #
-  # THIS USED TO BE BROKEN, AND THE 2026-09-04 SUBDOMAIN SPLIT FIXED IT. While prod was
+  # THE PUBLIC ANSWER IS NOW THE WAN, SO THE ISSUER IS PINNED (2026-10-03). Since
-  # authored to serve dev's hostnames, a prod node resolving keycloak.home.keithjackson.dev
+  # platform/gateway/gateway.yaml set an external-dns `target`, every public name is a CNAME
-  # publicly got DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN, so
+  # to wan.home.keithjackson.dev = the home WAN IP. The SOHO router does NOT hairpin: from the
-  # kubectl OIDC SSO on prod could not work at all. Now prod's issuer is
+  # LAN, WAN:443 answers with the router's own self-signed placeholder cert (measured from
-  # keycloak.prod.home.keithjackson.dev, and the public record for that name is published by
+  # k3s-prod-2/-3 2026-10-03: `curl` exit 60, ssl_verify_result=18). The apiserver would then
-  # PROD's own external-dns pointing at PROD's Gateway VIP (192.168.2.20). The public answer
+  # be unable to fetch its issuer's discovery/JWKS, so kubectl OIDC SSO would fail on the
-  # and the split-horizon answer therefore AGREE, and no /etc/hosts pin is needed.
+  # next k3s restart or Keycloak key rotation. node_baseline_hosts_pins (below) pins the
  # issuer host to the Gateway VIP in /etc/hosts, which glibc consults before these
  # resolvers. (History: before the 2026-09-04 subdomain split the public answer was DEV's
  # VIP; after it, the public answer was prod's own VIP and no pin was needed.)
  #
-  # STILL TO VERIFY on prod bring-up (PROD-PROVISIONING.md §10): this holds only once prod's
+  # The cert-based admin kubeconfig never depends on any of this. It is the break-glass path.
  # external-dns has actually published the record. Until then the issuer does not resolve
  # from the nodes and kubectl SSO fails — the cert-based admin kubeconfig is the break-glass
  # path, and it always works. Ordering, not a design gap.
 node_baseline_dns_search: []
 # /etc/hosts pins: node-process names that must reach the in-cluster Gateway directly rather
 # than via the public (WAN, non-hairpinned) answer. See "THE PUBLIC ANSWER IS NOW THE WAN" above.
 # [parity] the IP MUST equal GATEWAY_VIP in clusters/prod/kustomization.yaml (192.168.2.20),
 # and the name MUST follow k3s_oidc_issuer_url's host (roles/k3s_server/defaults/main.yml).
 node_baseline_hosts_pins:
  - ip: "192.168.2.20"
    names:
      - "keycloak.{{ k3s_base_domain }}"
 # --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
 # PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/
@@ -242,3 +242,9 @@ node_baseline_dns_search:
 node_baseline_dns_options:
  - timeout:2
  - attempts:2
 # Static /etc/hosts entries (tags: dns, hosts), as a list of {ip, names}. EMPTY by default:
 # dev needs none. Prod pins its OIDC issuer host to the Gateway VIP because the public
 # answer for it is the non-hairpinned WAN IP (inventory/group_vars/k3s_prod/vars.yml).
 # Written as ONE marked block, so removing an entry here removes it from the node, and
 # an empty list removes the block entirely.
 node_baseline_hosts_pins: []
@@ -1090,3 +1090,22 @@
      {% endif %}
  when: node_baseline_dns_mechanism == 'resolv-conf-file'
  tags: [dns]
 # /etc/hosts pins — tag `dns` (and `hosts` on its own). Some node processes must reach a
 # public hostname at the in-cluster Gateway, not at its public answer. On prod, the
 # kube-apiserver's OIDC issuer resolves publicly to the WAN IP, which the router does not
 # hairpin. glibc reads /etc/hosts before any nameserver (nsswitch `files dns`), so a pin wins
 # regardless of which resolver mechanism above is in effect. One marked block: entries
 # dropped from node_baseline_hosts_pins disappear from the node, and an empty list
 # removes the block.
 - name: Pin node-process hostnames in /etc/hosts
  ansible.builtin.blockinfile:
    path: /etc/hosts
    marker: "# {mark} node_baseline hosts pins (Ansible, tags: hosts)"
    block: |
      {% for pin in node_baseline_hosts_pins %}
      {{ pin.ip }}	{{ pin.names | join(' ') }}
      {% endfor %}
    state: "{{ 'present' if node_baseline_hosts_pins | length > 0 else 'absent' }}"
    backup: true
  tags: [dns, hosts]
@@ -20,6 +20,30 @@ A wildcard `*.keithjackson.dev` matches only one label deep, so `<svc>.home.keit
 its own record. **`internal` services need NO public record** — PiHole resolves them to the internal
 Gateway VIP (split-horizon). ExternalDNS manages **`public`** services' records only.
 ## Public target + in-cluster DDNS (2026-10-03)
 Gateway-published records are **CNAMEs to `wan.home.keithjackson.dev`**, not the Gateway VIP:
 ```
 plex.home.keithjackson.dev  CNAME  wan.home.keithjackson.dev   <- external-dns (Gateway `target` annotation)
 wan.home.keithjackson.dev   A      <current WAN IP>            <- cloudflare-ddns (ddns.yaml), every 5m
 ```
 - The `target` annotation is on the **shared Gateway** (`platform/gateway/gateway.yaml`), so it
  applies to every route that publishes at all. `internal` routes (`controller: none`) still publish
  nothing.
 - `ddns.yaml` runs `favonia/cloudflare-ddns` in this namespace with the same `cloudflare-api-token`.
  It detects the WAN IP through Cloudflare's trace endpoint, so the pod **must not** carry
  `egress-class: vpn`.
 - LAN and pods resolve `*.home.keithjackson.dev` to the Gateway VIP via PiHole and CoreDNS
  (split-horizon) and never use the public path.
 - **Nodes resolve through public DNS** (`node_baseline_dns_servers`), so from a node, the apiserver's
  OIDC issuer `keycloak.home.keithjackson.dev` resolves to the WAN IP. kubectl SSO therefore relies on
  the router hairpinning (NAT reflection) WAN:443 back to the Gateway.
 Check: `dig +short @1.1.1.1 plex.home.keithjackson.dev` → `wan.home.keithjackson.dev.` then the WAN IP;
 `kubectl -n external-dns logs deploy/cloudflare-ddns`.
 ## Secrets
 Cloudflare API token referenced **by name** (`cloudflare-api-token`, shared with cert-manager),
@@ -0,0 +1,87 @@
 ---
 # In-cluster dynamic DNS for the home WAN (2026-10-03).
 #
 # Keeps ONE record, `wan.${BASE_DOMAIN}` (A, IPv4 only, not proxied), pointed at the home WAN's
 # current public IP. Every public name external-dns publishes from the shared Gateway is a CNAME to
 # it (platform/gateway/gateway.yaml `external-dns.alpha.kubernetes.io/target`), so a WAN IP change
 # moves all of them through this one record, with no Git change and no external-dns involvement.
 #
 # OWNERSHIP SPLIT WITH EXTERNAL-DNS. This record carries no external-dns TXT registry entry, so
 # external-dns (policy: sync, owner `prod`) never treats it as its own and never deletes it.
 # Conversely this updater manages only the names in DOMAINS and never touches external-dns's records.
 #
 # IP DETECTION. Default `cloudflare.trace`: an HTTPS request to Cloudflare's trace endpoint, which
 # reports the address the request arrived from. Correct here only because this pod is NOT in the
 # `vpn` egress class — the CiliumEgressGatewayPolicy (platform/vpn-egress) selects pods by
 # `homelab.keithjackson.dev/egress-class: vpn` alone, cluster-wide. Labelling this pod `vpn` would
 # publish the VPN exit IP instead of the WAN IP.
 #
 # TOKEN. Reuses this namespace's `cloudflare-api-token` (same Zone:DNS:Edit token as external-dns
 # and cert-manager) — no new secret.
 #
 # TIER NOTE. This lives in the gating `platform` Flux Kustomization (wait: true) next to
 # external-dns. If it ever crashloops, `platform` goes not-Ready and apps stop reconciling (no
 # outage). Kept here rather than in its own non-gating Kustomization to avoid a clusters/** change.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: cloudflare-ddns
  namespace: external-dns
  labels:
    app.kubernetes.io/name: cloudflare-ddns
 spec:
  replicas: 1                 # exactly one writer for the record
  revisionHistoryLimit: 3
  strategy:
    type: Recreate            # never two updaters at once
  selector:
    matchLabels:
      app.kubernetes.io/name: cloudflare-ddns
  template:
    metadata:
      labels:
        app.kubernetes.io/name: cloudflare-ddns
    spec:
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: cloudflare-ddns
          image: favonia/cloudflare-ddns:1.17.1
          imagePullPolicy: IfNotPresent
          env:
            - name: CLOUDFLARE_API_TOKEN
              valueFrom:
                secretKeyRef:
                  name: cloudflare-api-token
                  key: api-token
            - name: DOMAINS
              value: "wan.${BASE_DOMAIN}"
            - name: IP6_PROVIDER
              value: "none"         # IPv4 WAN only; leaves any AAAA alone
            - name: PROXIED
              value: "false"        # DNS-only: Plex/Gitea/etc. terminate TLS on the Gateway
            - name: UPDATE_CRON
              value: "@every 5m"
            - name: DELETE_ON_STOP
              value: "false"        # a pod restart must not blank the record every CNAME depends on
            - name: RECORD_COMMENT
              value: "managed by k3s cloudflare-ddns (platform/externaldns/ddns.yaml)"
            - name: EMOJI
              value: "false"
          resources:
            requests:
              cpu: 5m
              memory: 16Mi
            limits:
              memory: 64Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL
@@ -81,38 +81,29 @@ spec:
    # what this instance can see or own; per-resource exclusion only ever REMOVES an object from
    # this instance's view, so it cannot make prod capable of deleting a record it does not own.
    # ---------------------------------------------------------------------------------------------
-    # THE PUBLIC-TARGET GAP — STILL OPEN, AND NOT FIXABLE HERE ON v0.21.0 (operator decision)
+    # THE PUBLIC TARGET — DECIDED 2026-10-03 (operator): every Gateway-published name goes public
    #
-    # The five `exposure: public` services (plex, gitea, seerr, keycloak, nextcloud) publish records
+    # Until 2026-10-03 every gateway-sourced record answered with the Gateway VIP (192.168.2.20,
-    # that point at 192.168.2.20 — the Gateway VIP, an unroutable RFC1918 address — so they are not
+    # RFC1918), so nothing published here was reachable from the internet. Two facts constrained the
-    # actually reachable from the internet either. Their records are LEFT EXACTLY AS THEY ARE by
+    # fix, and both still hold:
-    # this change, for two independent reasons:
+    #   * `external-dns.alpha.kubernetes.io/target` is read in v0.21.0's gateway-httproute source ONLY
    #     from the GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
    #     gw.Annotations)`) and from a ListenerSet listener — NEVER from an HTTPRoute. A target on the
    #     shared Gateway therefore moves EVERY gateway-sourced record at once.
    #   * The home WAN IP is residential and can change, so a literal IP in Git would rot.
    #
-    #   1. THE VALUE IS NOT ESTABLISHED. The one public IP this repo knows is 136.51.115.252
+    # The operator's answers: (a) yes, ALL of them — the five `exposure: public` apps AND the admin
-    #      (clusters/prod/kustomization.yaml, where the zone's `*.keithjackson.dev` wildcard also
+    # UIs (auth, crafty, grafana, headlamp, the apex), which sit behind Keycloak / oauth2-proxy;
-    #      points). It belongs to the LEGACY non-k3s stack and currently serves a placeholder
+    # (b) the address is kept current IN-CLUSTER. Implementation:
-    #      certificate. Whether prod's public services belong behind it, behind a new forward, or
+    #   * platform/gateway/gateway.yaml sets `target: wan.${BASE_DOMAIN}` on the shared Gateway, so
-    #      are not meant to be public at all is an operator decision. Breaking five services' DNS
+    #     each published name becomes a CNAME to that one hostname;
-    #      on a guess is worse than leaving a known-wrong-but-stable state.
+    #   * platform/externaldns/ddns.yaml (favonia/cloudflare-ddns) keeps `wan.${BASE_DOMAIN}` an A
-    #   2. THE MECHANISM CANNOT BE SCOPED TO THEM. `external-dns.alpha.kubernetes.io/target` is a
+    #     record at the current WAN IP. That record has no external-dns TXT registry entry, so this
-    #      real override, but in v0.21.0's gateway-httproute source it is read ONLY from the
+    #     instance never owns or garbage-collects it.
-    #      GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
+    # Internal services are unchanged: their routes carry `controller: none` (above), which is
-    #      gw.Annotations)`) and from a ListenerSet listener — NEVER from the HTTPRoute. So there is
+    # checked BEFORE the target is read, so they still publish nothing. The `service` source (the L4
-    #      no per-service target override while every public service shares the `shared` Gateway.
+    # LoadBalancers crafty-mc, unifi) does not read the Gateway's annotation and still publishes its
-    #      Annotating the Gateway would move EVERY gateway-sourced record, which after this change
+    # LB VIP. Whether those should go public is a separate, still-open question.
    #      still includes auth, crafty, grafana, headlamp and the apex — i.e. it would publish a
    #      routable address for admin UIs. That is strictly worse than the status quo.
    #
    # SO THE DECISION THE OPERATOR OWNS IS TWO-PART, and no knob is added here until the first half
    # is answered, because a knob that cannot take effect invites someone to set it and expect one:
    #   (a) WHAT address should the five public names answer with — 136.51.115.252 as-is, a new
    #       port-forward/WAN address, or nothing (drop `exposure: public` to `internal` instead)?
    #   (b) HOW should they be separated from the non-public names that share the Gateway? Options,
    #       roughly in ascending cost: a SECOND Gateway (or a ListenerSet) carrying the target
    #       annotation, with only the public services' routes attached; or publishing the five names
    #       from the `crd` source (DNSEndpoint objects the renderer emits for `public` services),
    #       which restores per-service targeting without a second Gateway.
    # Only (b) is a manifest change; (a) is a fact about the home WAN that has to be checked.
    # ---------------------------------------------------------------------------------------------
    # CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below.
    #
@@ -8,3 +8,5 @@ resources:
  # SOPS-encrypted Cloudflare token (env CF_API_TOKEN) — SAME token as cert-manager, second copy in
  # the external-dns namespace. Flux decrypts at reconcile.
  - cloudflare-api-token.sops.yaml
  # DDNS for `wan.${BASE_DOMAIN}` — the CNAME target every Gateway-published record points at.
  - ddns.yaml
@@ -7,6 +7,23 @@ kind: Gateway
 metadata:
  name: shared
  namespace: gateway
  annotations:
    # PUBLIC DNS TARGET (2026-10-03, operator decision). Without this, external-dns publishes every
    # route on this Gateway with the Gateway's OWN address (${GATEWAY_VIP}, RFC1918), which is
    # unreachable from the internet. With it, every published `<svc>.${BASE_DOMAIN}` (and the apex)
    # becomes a CNAME to `wan.${BASE_DOMAIN}`, an A record that platform/externaldns/ddns.yaml keeps
    # pointed at the home WAN's current public IP. The WAN forwards 80/443 to ${GATEWAY_VIP}.
    #
    # Scope is EVERY route on this Gateway that publishes at all — the operator chose to make the
    # admin UIs (auth, grafana, headlamp, crafty, apex) publicly resolvable alongside the five
    # `exposure: public` apps. `internal` services stay unpublished via their
    # `external-dns.alpha.kubernetes.io/controller: none` route annotation, which external-dns checks
    # BEFORE it reads this target. external-dns v0.21.0 reads `target` from the Gateway only, never
    # from an HTTPRoute — see platform/externaldns/helmrelease.yaml.
    #
    # LAN and in-cluster clients are unaffected: PiHole and CoreDNS answer ${BASE_DOMAIN} with
    # ${GATEWAY_VIP} directly (split-horizon), so they never take this public path.
    external-dns.alpha.kubernetes.io/target: "wan.${BASE_DOMAIN}"
 spec:
  gatewayClassName: cilium
  listeners:
@@ -153,14 +153,18 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
  plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
  token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
  Includes why a changed Secret does not restart anything, and why an expired token is harmless.
- **`plex-quicksync`** — hardware transcoding on the host iGPU. GitOps puts `/dev/dri` in the pod
+- **`plex-quicksync`** — hardware transcoding on the host iGPU. **Confirmed working 2026-10-02**
-  (`hardware.gpu: intel-quicksync`); **a human still has to tick Settings → Transcoder → "Use hardware
+  (VAAPI decode *and* encode, zero-copy). GitOps requests `gpu.intel.com/i915` on the pod
-  acceleration when available"**, and no environment variable can do it (the image supports no
+  (`hardware.gpu: intel-quicksync`, satisfied by the `intel-gpu-plugin` DaemonSet — a `/dev/dri`
-  `PLEX_PREFERENCE*`, verified). The page names the real preference key (`HardwareAcceleratedCodecs`,
+  hostPath was measured not to work: device-cgroup `EPERM`); **a human still has to tick Settings →
-  read out of the binary), says which widely-cited key **does not exist**, and is explicit that the
+  Transcoder → "Use hardware acceleration when available"**, and no environment variable can do it
-  **only** proof is `Transcode (hw)` on the Plex dashboard — Plex renders no ServiceMonitor, so no
+  (the image supports no `PLEX_PREFERENCE*`, verified). **The proof is `Plex Media Server.log`'s
-  alert could ever fire. Also: the device-cgroup failure to expect first and its device-plugin
+  `final decoder: / final encoder:` line, not the Plex dashboard** — the dashboard showed nothing
-  fallback, and why a `:latest` pull can silently break it.
+  while hardware transcoding was working, so its silence proves nothing; the page carries the
  redacting grep (`X-Plex-Token` lives in that log), the WRONG log file to avoid, and two more
  documented false leads (an empty `/proc/*/fd` scan, and a missing `HardwareAcceleratedCodecs` key).
  Plex renders no ServiceMonitor, so no alert could ever fire. Also: **`Pending` stalls the whole app
  tier** and its one-field escape hatch, and why a `:latest` pull can silently break it.
 - **`oauth2-proxy-session`** — a gated UI loads but its saves quietly do nothing, and you are never
  re-prompted to log in. Tells the three stacked auth layers apart from the sidecar's own access log
  (the four lines that matter, including the upstream `-` that means the proxy answered a request the
@@ -1,5 +1,10 @@
 # plex-quicksync — making Plex actually use Intel Quick Sync, and proving it did
 **Status: hardware transcoding is CONFIRMED WORKING on prod as of 2026-10-02** — VAAPI for both
 decode *and* encode, zero-copy, measured in Plex's own server log (see "How to verify"). The manual
 toggle has been ticked. This page is therefore now mostly a **verification and regression**
 procedure, and its old verification section was **wrong** — see the correction at the top of
 "How to verify".
 **Alert:** none, and none is possible — see "How to verify" below. This is an **operator procedure**,
 not an incident page. It is indexed here because the feature it completes is **inert until a human
 clicks one button**, and a switched-off feature that looks shipped is exactly what this page exists
@@ -33,7 +38,7 @@ is in git**:
 | Half | Who does it | State |
 |---|---|---|
 | The render device is inside the container **and openable** | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders `resources.limits: {gpu.intel.com/i915: 1}` on the app container. The `intel-gpu-plugin` DaemonSet (`platform/intel-gpu-plugin/`) injects the device **and the matching device-cgroup allow rule**. Flux applies it. | Done by merging that change. |
-| Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **NOT done.** `HardwareAcceleratedCodecs` was **absent** from `Preferences.xml` when this page was written — the toggle has never been on. |
+| Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **Done, 2026-10-02**, and confirmed from the **log** rather than from a preference key. `HardwareAcceleratedCodecs` is **still absent** from `Preferences.xml` with hardware transcoding working, so its absence proves nothing — see first-command #4. |
 **There is no GitOps path to the second half, and that is verified rather than assumed.** A bounded
 grep across the running image's entire init surface — `/etc/cont-init.d`, `/etc/services.d`,
@@ -79,8 +84,8 @@ also the only thing that produces the cgroup allow rule. There is no graceful-de
 The one improvement is that the loss is no longer silent.
 What this does **not** do, so nobody reads more into a green gate than is there: it does not enable
-hardware transcoding (that is the manual toggle below), a green gate only proves the **manifest
+hardware transcoding (that is the manual toggle below — done 2026-10-02), a green gate only proves
-renders**, and it writes no `nodeSelector` — the resource itself constrains Plex to nodes the plugin
+the **manifest renders**, and it writes no `nodeSelector` — the resource itself constrains Plex to nodes the plugin
 advertises, which is the scheduler's doing and not a selector in this repo.
 ## Running the commands in this runbook
@@ -172,17 +177,28 @@ cgroup rule that uid 0 could not buy its way past either. `RW-OPEN-DENIED` with
 visible is the **device-cgroup** failure — i.e. the plugin's allow rule did not arrive; see "Likely
 causes" #2.
-**4. Is the toggle on?** Named key only — never the whole file:
+**4. Is the toggle on?** **You cannot answer this from `Preferences.xml` alone.** That mistake is on
 this page's record: it told readers to treat a missing `HardwareAcceleratedCodecs` as "never
 enabled", and on 2026-10-02 that key was **absent from all 27 attributes** while hardware
 transcoding was demonstrably working. Named keys only — never the whole file, it holds
 `PlexOnlineToken`:
 ```
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
-     grep -o "HardwareAcceleratedCodecs=\"[01]\"" \
+     grep -o -e "HardwareAcceleratedCodecs=\"[01]\"" -e "HardwareDevicePath=\"[^\"]*\"" \
-     "/config/Library/Application Support/Plex Media Server/Preferences.xml" || echo ABSENT'
+     "/config/Library/Application Support/Plex Media Server/Preferences.xml" || echo NEITHER-KEY'
 ```
-`ABSENT` ⇒ never enabled (the state this page was written in). `="0"` ⇒ explicitly off. `="1"` ⇒ on,
+| Reading | What it actually means |
-and the remaining question is whether it *works* — go to "How to verify".
+|---|---|
 | `HardwareDevicePath="…"` **present** | **Positive signal.** Plex writes this when hardware acceleration is configured; it was present on 2026-10-02 alongside a working VAAPI pipeline. |
 | `HardwareAcceleratedCodecs` **absent** | **Nothing. Neither on nor off.** Measured absent with the feature working (2026-10-02). Do not diagnose from it, and do not write it into the file to "fix" it. |
 | `HardwareAcceleratedCodecs="0"` | explicitly **off** — the only conclusive reading, and only in the negative. |
 | `HardwareAcceleratedCodecs="1"` | on. |
 **The log settles it, this file does not.** Go to "How to verify"; the `hardware transcoding:
 enabled` and `final decoder/final encoder` lines are the verdict.
 ## The one manual step
@@ -191,8 +207,11 @@ Plex web UI → **Settings → Transcoder → "Use hardware acceleration when av
 The underlying preference key is **`HardwareAcceleratedCodecs`**. That name was confirmed by
 extracting strings from the 23 MB `Plex Media Server` binary in the running container — **not** from
-recall and not from a forum post. Real neighbouring keys from the same binary, so you can recognise
+recall and not from a forum post. **But do not expect to see it in `Preferences.xml` after ticking
-them and not invent them:
+the box:** on 2026-10-02, with the toggle on and hardware transcoding working, the key was absent and
 `HardwareDevicePath` was what had been written. The key exists in the binary; Plex does not
 necessarily persist it. Real neighbouring keys from the same binary, so you can recognise them and
 not invent them:
 | Key | What it is |
 |---|---|
@@ -209,42 +228,121 @@ binary. Recorded explicitly so nobody spends an evening hunting for it or, worse
 Editing `Preferences.xml` by hand is **not** the procedure. It is app state on the replicated Longhorn
 tier, it is in no git commit, and the UI writes the key correctly. Use the UI.
-## How to verify — and why there is no alert
+## How to verify — the log, not the dashboard
-Start a playback that **forces a transcode** (a client that cannot direct-play the source, or pick a
+**Correction, 2026-10-02.** Until today this page said the Plex dashboard showing `Transcode (hw)`
-lower quality explicitly), then read Plex's own **Dashboard → Activity**:
+was "the *only* proof available, and nothing else will ever tell you". **That was false and it cost
 the operator real time:** hardware transcoding was fully working — VAAPI decode *and* encode, with
 zero-copy — while **Dashboard → Activity showed nothing of the sort**. A check that is silent while
 the feature works does not just fail to prove anything; it sends someone hunting a fault that does
 not exist. The authoritative check is **Plex's own server log**.
- **`Transcode (hw)`** on the session ⇒ hardware transcoding. **This is the proof.**
+### The primary check — run this one
 - plain **`Transcode`** ⇒ software. The feature is not working, whatever the manifest says.
-**That dashboard is the *only* proof available, and nothing else will ever tell you.**
+```
-`catalog/plex.yaml` declares no `metricsEndpoint`, so Plex renders **no ServiceMonitor at all** —
+./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n plex exec deploy/plex -- \
-there is no scrape, no transcode-mode metric, and therefore no alert that could fire. Do not open work
+  grep -aiE "hardware transcoding: (final|using)" \
-to "alert on software-transcode fallback": with no metric behind it such a rule would be permanently
+  "/config/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log" \
-inert, and this repo has already been burned at scale by rules that evaluated against nothing while
+  | sed -E "s/(X-Plex-Token|token)=[^&[:space:]]*/\1=REDACTED/g" | tail -5'
-looking like coverage (`runbooks/README.md`, "Alerts you should not trust the silence of"). If that
+```
 gap is ever worth closing, the honest order is a real exporter first and a rule second.
-Corroborating, not a substitute — the server log on the config PVC:
+| Line | Verdict |
 |---|---|
 | `hardware transcoding: final decoder: vaapi, final encoder: vaapi` | **PASS.** Hardware in **both** directions. This is the pass condition. |
 | `hardware transcoding: using zero-copy transcoding` | Best case — frames never round-trip through the CPU. Observed on prod 2026-10-02. |
 | `hardware transcoding: final decoder: , final encoder: ` (both empty) | It **fell back to software** for that session. |
 | no matching line at all | No transcode has been attempted since the log rolled — force one (below), then look again. Absence is not a failure. |
 The 2026-10-02 reading on prod, in full, for the shape to recognise:
 ```
 Codecs: hardware transcoding: testing API vaapi for device '/dev/dri/renderD128'
                              (Intel CoffeeLake-S GT2 [UHD Graphics 630])
 hardware transcoding: using hardware decode accelerator vaapi
 hardware transcoding: zero-copy support present
 hardware transcoding: using zero-copy transcoding
 hardware transcoding: final decoder: vaapi, final encoder: vaapi
 ```
 To make a line appear, start a playback that **forces a transcode**: a client that cannot
 direct-play the source, or an explicitly lower quality. Direct Play / Direct Stream never reaches
 the transcoder and logs nothing.
 ### Three things to get right about that command
 1. **Always keep the `sed`.** `Plex Media Server.log` carries API URLs containing
   **`X-Plex-Token=…`**, a live plex.tv credential. Any output that might land in a ticket, a
   transcript or a chat window must be redacted — CLAUDE.md §6, and a credential has already been
   leaked into a transcript in this repo exactly this way. The redacting form above is the form to
   copy; do not "simplify" the pipe out of it.
 2. **`Plex Transcoder Statistics.log` is the WRONG FILE.** The name is irresistible and someone will
   reach for it. It holds XML `<Variant .../>` session records — **not** ffmpeg/VAAPI output. There
   is no decoder/encoder verdict anywhere in it. The file you want is `Plex Media Server.log`.
 3. `grep -a` is deliberate: the log carries occasional non-UTF-8 bytes, and without `-a` GNU grep
   reports "binary file matches" and prints nothing — which reads exactly like "no hardware
   transcoding".
 ### Secondary and unreliable: the Plex dashboard
 **Dashboard → Activity** *may* label a transcoding session `Transcode (hw)`. If it does, it agrees
 with the log and you are done. **If it does not, that proves nothing at all** — measured 2026-10-02:
 hardware transcoding active and zero-copy, dashboard silent. Never conclude "software transcoding"
 from the dashboard and never open an investigation on its say-so. Kept here only because it is the
 fastest glance available; it is a convenience, **not evidence**.
 ### Process-level, while a transcode is in flight: the transcoder's argv
 This shows Plex *requesting* hardware, as distinct from reporting it afterwards. Get a shell in the
 pod (if `-it` has no TTY under `dev.sh run`, wrap each line in `sh -c "…"` instead):
 ```
 infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n plex exec deploy/plex -- \
+  'kubectl --context k3s-prod -n plex exec -it deploy/plex -- sh'
     sh -c "grep -i -m20 -e hardware -e vaapi -e hwdevice \
     \"/config/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log\""'
 ```
-Treat those strings as advisory: they are Plex's, not this repo's, and they move between versions.
+then, inside the pod:
-Absence of a match is **not** evidence of software transcoding; a VA-API device open is good evidence
+
-of hardware. The dashboard remains the verdict.
+```
 ps -eo pid,user,comm | grep -i transcoder
 tr '\0' ' ' < /proc/<transcoder-pid>/cmdline | grep -oE '(-hwaccel[a-z_]*|vaapi|renderD[0-9]+)'
 ```
 `-hwaccel`, `-hwaccel_device`, `vaapi`, `renderD128` present ⇒ Plex is asking for hardware. Keep the
 `grep -oE`: the raw argv contains media paths and session identifiers you do not want in a
 transcript. This only works while a session is actually transcoding.
 ### Documented false lead: scanning `/proc/*/fd` for a holder of `renderD128`
 **It returns nothing even when hardware transcoding is working.** Measured 2026-10-02: a
 `/proc/*/fd` sweep inside the pod found **no** process holding `/dev/dri/renderD128`, while the log
 showed `final decoder: vaapi, final encoder: vaapi` for sessions either side of the scan. Transcode
 sessions are **short** and the scan landed in a gap between two of them. An empty fd scan is a
 **timing artefact, not a finding** — do not report it as evidence of software transcoding, and do
 not escalate on it. Use the argv check above (which has the same timing constraint but tells you
 something when it does catch a session), or just read the log, which is retrospective and therefore
 does not care about timing.
 ### Why there is no alert — this half was always right
 `catalog/plex.yaml` declares no `metricsEndpoint`, so Plex renders **no ServiceMonitor at all** —
 there is no scrape, no transcode-mode metric, and therefore no alert that could fire. Do not open
 work to "alert on software-transcode fallback": with no metric behind it such a rule would be
 permanently inert, and this repo has already been burned at scale by rules that evaluated against
 nothing while looking like coverage (`runbooks/README.md`, "Alerts you should not trust the silence
 of"). If that gap is ever worth closing, the honest order is a real exporter first and a rule
 second.
 ## Likely causes, most likely first
-When the dashboard still says plain `Transcode`:
+When the **log** says it fell back to software — `final decoder: , final encoder: ` with both
 fields empty. (Not when the dashboard is quiet: the dashboard's silence means nothing, see "How to
 verify".)
-1. **The toggle is off.** Overwhelmingly the most likely answer, and the reason this page exists.
+1. **The toggle is off.** Historically the most likely answer and the reason this page exists. It was
-   First-command #4 returns `ABSENT` or `="0"`. Remedy: the manual step above. Check this **before**
+   ticked on 2026-10-02 and should stay ticked, but a `/config` restore or a stray UI save can unset
-   investigating anything technical.
+   it. Remedy: the manual step above. **Do not diagnose this from a missing
   `HardwareAcceleratedCodecs`** — see first-command #4; only `="0"` is conclusive, and the log line
   `hardware transcoding: enabled` is the real signal that the toggle took.
 2. **The device cannot be `open()`ed — the plugin's cgroup allow rule did not arrive.** Symptom:
   first-command #3 shows `renderD128` present and correctly owned, and the open returns
@@ -276,13 +374,27 @@ When the dashboard still says plain `Transcode`:
 4. **The image changed under you.** See "The floating-image caveat". A `:latest` pull can remove the
   mechanism this feature leans on with no commit in this repo.
-5. **The session was never transcoding.** Direct Play / Direct Stream never touches the transcoder, so
+5. **The session was never transcoding.** Direct Play / Direct Stream never reaches the transcoder,
-   `Transcode (hw)` cannot appear. Confirm the session actually says `Transcode` before treating it as
+   so no `hardware transcoding:` line is written for it at all. An *absent* line is this, not a
-   a hardware failure.
+   failure — confirm a transcode actually happened before treating anything as a hardware fault.
-6. **The source needs a codec Quick Sync on this generation cannot handle** — e.g. AV1 decode, which
+6. **The *source* codec may be one this iGPU cannot decode — HYPOTHESIS, not a confirmed
-   Coffee Lake's UHD 630 has no engine for. Plex falls back to software *per stream*. Symptom: some
+   diagnosis.** The symptom that points here is fallback on **particular titles** while other
-   sessions show `(hw)` and others do not. Not a fault.
+   sessions are hardware. Measured on prod 2026-10-02, *between* two fully hardware sessions, one
   session logged:
   ```
   hardware transcoding: enabled, but no hardware decode accelerator found
   hardware transcoding: final decoder: , final encoder:
   ```
   The platform was demonstrably working either side of it, so the likely explanation is the **file**,
   not the cluster: UHD 630's media engine decodes H.264, HEVC 8-bit and 10-bit, and VP9 — but **not
   AV1**, and not 10-bit 4:2:2 HEVC. **This was NOT verified against that particular title.** It is
   inferred from the hardware's capability set and is written here as the lead to check first, not as
   a finding. So: fallback on *all* files is a platform problem (causes 1-4); fallback on *some*
   files, check the source codec before suspecting the device, and expect occasional legitimate
   per-title fallbacks to be normal rather than a fault.
 ## Plex will not schedule
@@ -360,19 +472,30 @@ infra/devcontainer/dev.sh run bash -c \
 ```
 `45-plex-hw-transcode-and-connected-tuner` still present ⇒ the group mechanism is still the one
-described here. Gone or renamed ⇒ stop and re-measure. In particular the "no `supplementalGroups`
+described here: the hook reads the injected device's **own** ownership, which under the device plugin
-needed" conclusion in `catalog/plex.yaml` and `schemas/renderer-contract.md` is only true while that
+is `root:root`, so it resolves GID `0` (see "When to escalate or stop"). Gone or renamed ⇒ stop and
-hook exists.
+re-measure. In particular the "no `supplementalGroups` needed" conclusion in `catalog/plex.yaml` and
 `schemas/renderer-contract.md` is only true while that hook exists.
 ## When to escalate or stop
 - **Do not add `supplementalGroups` or a `podSecurityContext` to `catalog/plex.yaml`.** It would be
-  **inert**, and it would mislead the next person. Measured in-container 2026-09-30: the image's
+  **inert**, and it would mislead the next person. The image's `cont-init.d` hook stats each
-  `cont-init.d` hook stats each `/dev/dri` char device and `usermod -a -G`s the `plex` user into its
+  `/dev/dri` char device and `usermod -a -G`s the `plex` user into **that device's** group, inside
-  group *inside* the container, and the service then execs via `s6-setuidgid plex`, which **replaces**
+  the container; the service then execs via `s6-setuidgid plex`, which **rebuilds** the process group
-  the process group list from `/etc/group` rather than appending — `Plex Media Server` runs with
+  list from `/etc/group` rather than appending to it. Anything Kubernetes supplies is therefore
  `Groups: 100 1000`, root's gid `0` (which PID 1 does hold) gone. Anything Kubernetes supplies is
  discarded before Plex starts. `render.py --selftest` asserts that absence deliberately.
 - **GID `992` is a hostPath-era number — do not expect it, and do not read the root group as a
  defect.** Under the device plugin the injected `/dev/dri/renderD128` arrives **`root:root`**, so
  the hook resolves GID **`0`** and adds `plex` to the **root group**. Measured 2026-10-02:
  `Plex Media Server` runs with `Groups: 0 100 1000`, and `id plex` reports
  `groups=1000(plex),0(root),100(users)`. Under the old hostPath the device was `root:render`
  (GID 992 on these nodes) and the reading was `Groups: 44 100 992 1000`. **992 is now irrelevant**:
  if any page, comment or older handoff tells you to expect it, that text is stale and the system
  you are looking at is working. The root-group membership is an inherent property of the plugin's
  `DeviceSpec` — it injects the host device with its host ownership — **not** a defect, and **not** a
  reason to reintroduce `supplementalGroups`, which would still be a no-op for the reason in the
  bullet above.
 - **Do not reach for `privileged: true`.** Kyverno `restrict-host-namespaces` is Enforce and will
  reject it — and it would not help: `uid 0` failed the same `open()` in the 2026-09-30 measurement,
  because root does not bypass the cgroup device gate.
@@ -390,7 +513,8 @@ hook exists.
  watch state (`policy: daily-offcluster`). The transcode toggle is one preference, not a reason to
  lose the DB.
 - **Stop and get the operator** if the device cannot be opened from the pod as uid 1000 (cause #2
-  confirmed) *under the current mechanism*. The next step is a platform change in
+  confirmed) *under the current mechanism* — note that this open **passed** on 2026-10-02
  (`RW-OPEN-OK` as uid 1000 via `s6-setuidgid plex`), so a failure now is a regression. The next step is a platform change in
  `platform/intel-gpu-plugin/`, with its own PR, not more poking at this workload.
 ## Known gaps, deliberately not fixed here
@@ -413,10 +537,25 @@ Recorded so they are findable, and so nobody assumes they were overlooked:
 ---
 Source of truth: `runbooks/plex-quicksync.md`. Index: `/runbooks/`
-The node measurements (all three prod nodes) and the in-container image measurements cited here were
+
-taken **2026-09-30** against `plexinc/pms-docker` digest `sha256:e0ab27395614…`
+**Hardware transcoding is CONFIRMED WORKING on prod, measured 2026-10-02.** From
-(`plexmediaserver 1.43.4.10903`). The `EPERM` open measurements, the `Groups: 44 100 992 1000`
+`Plex Media Server.log` in the running pod: API `vaapi` on `/dev/dri/renderD128`
-reading and the `gpu.intel.com/i915: 1` allocatable/DaemonSet readings were taken **2026-10-01**.
+(`Intel CoffeeLake-S GT2 [UHD Graphics 630]`), `using hardware decode accelerator vaapi`,
-Hardware transcoding has **still not been observed working**: the open test has not been re-run under
+`zero-copy support present`, `using zero-copy transcoding`, and
-the device-plugin mechanism (it cannot be, until that merges and Flux reconciles), and the Plex-side
+`final decoder: vaapi, final encoder: vaapi`. Also measured in the same pod: `RW-OPEN-OK` opening
-toggle is still unset.
+`/dev/dri/renderD128` as uid 1000 via `s6-setuidgid plex`; the pod spec carrying
 `{"limits":{"gpu.intel.com/i915":"1"},"requests":{"gpu.intel.com/i915":"1"}}`; `Groups: 0 100 1000`
 and `id plex` → `groups=1000(plex),0(root),100(users)`; and `HardwareDevicePath` present in
 `Preferences.xml` while `HardwareAcceleratedCodecs` was absent from all 27 attributes.
 Earlier readings, kept for provenance and **superseded** where they conflict: the node measurements
 (all three prod nodes) and the in-container image measurements taken **2026-09-30** against
 `plexinc/pms-docker` digest `sha256:e0ab27395614…` (`plexmediaserver 1.43.4.10903`); the hostPath
 `EPERM` open results and the `Groups: 44 100 992 1000` / GID-992 group reading, **2026-10-01, under
 the hostPath mechanism that no longer exists**; and the `gpu.intel.com/i915: 1` allocatable and
 `intel-gpu-plugin` DaemonSet readings, **2026-10-01**.
 What this page still does **not** establish: that any particular title's software fallback was caused
 by its source codec (cause 6 is an unverified hypothesis), and that Plex's dashboard *ever* labels a
 session `Transcode (hw)` on this deployment — it has not been observed doing so, which is exactly why
 the log is now the primary check.
@@ -0,0 +1,148 @@
 # plex-remote-access — getting Plex reachable from outside the LAN, and proving it
 **Status: the in-cluster half is shipped; the last mile is TWO MANUAL STEPS and remote access does
 NOT work until both are done.** Neither can live in this repo.
 **Alert:** none, and none is possible from in-cluster signals — see "Why no alert" below. This is an
 **operator procedure**, not an incident page.
 **Files:** `apps/wave-0/plex-remote.yaml` (the tcp/32400 LoadBalancer + its CiliumNetworkPolicy),
 `clusters/base/cluster-vars.yaml` + `clusters/prod/kustomization.yaml` (`PLEX_L4_VIP`),
 `catalog/render.py` (the `EXTRA_UNITS` entry that places the unit), `catalog/plex.yaml` (Plex's own
 env — note `ADVERTISE_IP` is deliberately NOT set; see "Why not the Gateway URL").
 **Related:** `runbooks/plex-quicksync.md` and `runbooks/plex-claim.md` — the other two Plex
 procedures with a manual last mile.
 ## What was actually broken (measured 2026-10-03)
 Plex was serving correctly and was reachable on the LAN. Two independent things stopped remote
 clients:
 1. **Plex advertised no remote connection at all.** In the live `Preferences.xml`:
   `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while
   `PublishServerOnPlexOnlineKey="1"`. The server was trying to publish itself to plex.tv and had
   nothing reachable to publish. Plex's automatic path (UPnP/NAT-PMP) cannot work here: the process
   lives in a pod on the Cilium overlay (`10.0.2.136`), it cannot speak UPnP to the SOHO router, and
   a mapping to the pod IP would be meaningless if it could.
 2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` → connection refused.
   `136.51.115.252:443` → a **self-signed router certificate** (`C=ZZ, CN=%D`), not the Gateway's
   Let's Encrypt wildcard.
 The Service was `ClusterIP` only, so port 32400 had no path off-cluster by any route.
 **Caveat on that second measurement, because it is easy to over-read:** it was taken from *inside*
 the LAN, so strictly it proves there is no NAT hairpin (reflection). It does not prove outside-in
 failure. But a router certificate answering 443 means the router's own management UI is bound to the
 WAN port, and that *would* shadow a port-forward for genuine outside traffic too. **Verify from a
 real outside vantage point** (phone on cellular, or an external port checker) before concluding
 either way — this repo has no such vantage point and must not pretend otherwise.
 ## Step 1 — forward WAN tcp/32400 on the router (operator, off-repo)
    WAN tcp/32400  ->  192.168.2.24  (${PLEX_L4_VIP})  port 32400
 `192.168.2.24` is a pinned Cilium LB VIP, ARP-announced on the flat LAN by the existing
 `CiliumL2AnnouncementPolicy default-l2`. It is **pinned on purpose**: this address is typed into the
 router by hand, so it must never move on a reschedule.
 While in the router, **check what else is bound to the WAN interface.** If remote management is
 listening on 443 (see the certificate measurement above), move it or disable it — otherwise the
 Gateway's 80/443 forward is shadowed and the Plex *web UI* over the internet stays broken even
 though remote *streaming* works. These are two separate paths and they fail independently.
 Confirm the VIP is actually answering on the LAN first, so a failure later is unambiguous:
    ./infra/devcontainer/dev.sh run bash -c \
      'kubectl --context k3s-prod -n plex get svc plex-l4-tcp -o wide'
 `EXTERNAL-IP` must read `192.168.2.24`. If it is `<pending>`, LB-IPAM could not assign — the address
 is outside the pool or already taken; see `clusters/prod/kustomization.yaml` for the pool accounting.
 **`192.168.2.24` was never ARP-probed off-cluster.** Only the in-cluster pool accounting was checked
 (IPsTotal 11 / IPsUsed 4 / IPsAvailable 7 on 2026-10-03). If the VIP behaves oddly, probe it from a
 LAN host — `arping 192.168.2.24` — the way `.23` was probed before it was claimed.
 ## Step 2 — tick "manually specify public port" in Plex (operator, UI only)
 **There is no GitOps path to this and no environment variable for it.** Measured in the running
 image on 2026-10-03: a grep of the entire init surface (`/etc/cont-init.d`, `/plex-common.sh`,
 `/usr/local/bin`) for `ManualPortMapping` and `PLEX_PREFERENCE` returned **NONE**. The image writes
 only six preferences, and this is not one of them. It is the same class of manual last mile as the
 `HardwareAcceleratedCodecs` toggle in `runbooks/plex-quicksync.md`.
 In the Plex web UI (`https://plex.home.keithjackson.dev`, on the LAN):
 **Settings → Remote Access**
 - Ensure **Enable Remote Access** is on. (`PublishServerOnPlexOnlineKey` was already `1`.)
 - Tick **Manually specify public port** and set it to **32400**.
 - **Apply**, then let the panel settle. It should report *"Fully accessible outside your network"*.
 Without this tick Plex waits for a NAT mapping it can never obtain and publishes no remote
 connection, so the port-forward in step 1 sits there unused.
 ## How to verify
 Do these in order; each one isolates a different half.
 **1. The port is open from outside.** From a phone on cellular, or any off-LAN host:
    curl -sk -o /dev/null -w '%{http_code}\n' https://136.51.115.252:32400/identity
 `200` means the forward and the policy are both good. A hang or refusal means step 1.
 **2. Plex itself believes it is reachable.** Settings → Remote Access must say *"Fully accessible
 outside your network"*. If it says *"not available outside your network"* while test 1 passes, step 2
 was not applied.
 **3. plex.tv is handing out a remote connection.** The decisive test, because it is what clients
 actually consume: sign in to [app.plex.tv](https://app.plex.tv) from an off-LAN network and confirm
 the server appears and plays. A client finds this server through plex.tv's connection list, never
 through DNS.
 **4. The policy is admitting the traffic.** If test 1 fails *after* the forward is confirmed, the
 CiliumNetworkPolicy is the next suspect:
    ./infra/devcontainer/dev.sh run bash -c \
      'kubectl --context k3s-prod -n plex get cnp plex-allow-remote-32400'
 See `runbooks/denied-flows.md` for reading actual drops.
 ## Why no alert
 Every in-cluster signal is green whether or not remote access works. The Service has endpoints, the
 pod is `Running`, the policy is applied, and `plex-down` watches
 `kube_deployment_status_replicas_available` — none of which can see a router with a missing
 port-forward or an unticked checkbox. An alert would need an off-LAN prober, which this cluster does
 not have. **Test 3 above is the only honest check, and it is manual.**
 ## Why not the Gateway URL (`ADVERTISE_IP`)
 The alternative was to leave Plex on `ClusterIP` and set
 `ADVERTISE_IP: https://plex.home.keithjackson.dev:443`, which writes Plex's `customConnections`
 preference and would have put the server on plex.tv's list via the existing L7 Gateway — no new WAN
 hole. That mechanism was verified to work on an already-claimed server (the `ADVERTISE_IP` block at
 line 126 of the image's `40-plex-first-run` sits **outside** the claim guard, and `/.firstRunComplete`
 lives on the overlay FS, not the `/config` PVC, so it re-runs every pod start), and the Gateway path
 itself was measured serving Plex correctly: `HTTP 200`, `claimed="1"`, with `401` on every protected
 endpoint.
 **The operator chose the native path anyway (2026-10-03), and the reasons are real:** Plex terminates
 its own TLS with its `*.plex.direct` certificate, and video streams bypass the Envoy L7 hop instead
 of being proxied through it. The cost is a new unfiltered WAN port — acceptable because Plex
 authenticates every caller itself. `ADVERTISE_IP` therefore stays unset in `catalog/plex.yaml`, and
 the deferral note there is still accurate.
 ## Known wart: remote clients are classified as LAN
 `apps/wave-0/plex-remote.yaml` uses `externalTrafficPolicy: Cluster`, which SNATs the client to the
 ingress node's address (`192.168.2.1x`). That falls inside the `192.168.0.0/16` entry of
 `ALLOWED_NETWORKS`, so **Plex treats every remote stream as local.** Measured on the Gateway path,
 which SNATs the same way: Plex logged `[10.0.2.214:36040 (Allowed Network (WAN))]`.
 This is a **metering** effect, not a security one — authorization is unchanged, as the 401s above
 show — but Plex's remote-quality and bandwidth limits will not bind on these streams.
 `Local` would preserve the client IP and fix it. **Do not switch to it.** Cilium's L2 announcer
 elects one lease holder per Service and does *not* follow the backend pod, so the VIP can be
 announced by a node with no Plex pod, black-holing the port while everything reports healthy. That
 exact change cost this LAN an 8-day Minecraft outage in September 2026 — the full write-up is at the
 top of `platform/crafty/service-l4.yaml`. Plex is a single-replica `Recreate` Deployment, so the
 node mismatch is the normal case here, not a rare one.
Author	SHA1	Message	Date
jakman85andClaude Opus 5	fbaeb3e038	feat(plex): expose tcp/32400 for remote access via a pinned LoadBalancer VIP gate / gate (pull_request) Successful in 1m49s Details gate / conformance (pull_request) Has been skipped Details Remote Plex clients had no path in, for two measured reasons: Plex advertised no remote connection at all (customConnections and ManualPortMappingMode both ABSENT while PublishServerOnPlexOnlineKey=1), and its Service was ClusterIP, so 32400 had no off-cluster path. The L7 Gateway was NOT the problem -- it serves Plex correctly (200, claimed=1, 401 on protected endpoints) -- but Plex clients discover servers through plex.tv, not DNS, so an HTTPRoute cannot put this server on a client's connection list. Operator chose the native path over ADVERTISE_IP: Plex terminates its own TLS with .plex.direct and streams bypass the Envoy L7 hop. Adds a hand-written non-catalog unit (the SD schema cannot express a LoadBalancer Service or a pinned lbipam VIP), registered in render.py's EXTRA_UNITS so --emit-waves places it in plex's own wave. externalTrafficPolicy: Cluster, inheriting crafty's 8-day-outage lesson rather than re-deriving it. Known consequence, documented: SNAT puts remote clients inside ALLOWED_NETWORKS, so Plex meters them as LAN. Not an auth change. Remote access does NOT work until the operator does two manual steps that cannot live in this repo -- the WAN tcp/32400 forward and Plex's "manually specify public port" toggle (no env var exists for it; verified NONE in the image's init surface). runbooks/plex-remote-access.md is the procedure. Touches clusters/* -- a sensitive path -- so the approval-policy stage will block pending an operator /approve-override. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-03 23:30:46 -05:00
jakman85	7ef2af205d	Merge pull request 'fix(node-baseline): pin the OIDC issuer to the Gateway VIP on prod nodes' (#41 ) from fix/node-keycloak-hosts-pin into main gate / gate (push) Successful in 1m45s Details gate / conformance (push) Has been skipped Details Reviewed-on: #41	2026-10-03 22:46:38 +00:00
jakman85andClaude Opus 5.5	788eaeae62	fix(node-baseline): pin the OIDC issuer to the Gateway VIP on prod nodes gate / gate (pull_request) Successful in 1m46s Details gate / conformance (pull_request) Has been skipped Details After public DNS moved to the WAN IP (#40), prod nodes resolve keycloak.home.keithjackson.dev to a WAN address the router does not hairpin, so the apiserver could not reach its OIDC issuer. Adds an opt-in node_baseline_hosts_pins /etc/hosts block; prod pins keycloak to GATEWAY_VIP. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-10-03 17:39:37 -05:00
jakman85	576e1abb19	Merge pull request 'feat(dns): publish Gateway names to the WAN via in-cluster DDNS' (#40 ) from feat/public-dns-wan-target into main gate / gate (push) Successful in 1m45s Details gate / conformance (push) Has been skipped Details Reviewed-on: #40	2026-10-03 22:25:13 +00:00
jakman85andClaude Opus 5.5	a1e808e697	feat(dns): publish Gateway names to the WAN via in-cluster DDNS gate / gate (pull_request) Successful in 1m46s Details gate / conformance (pull_request) Has been skipped Details Gateway-published records now CNAME to wan.${BASE_DOMAIN}, which a favonia/cloudflare-ddns Deployment keeps at the home WAN's current public IP. Operator decision: every Gateway-published name goes public; internal services (controller: none) remain unpublished. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-10-03 16:59:53 -05:00
jakman85	bcb8931b4f	Merge pull request 'fix(runbooks): Plex hw transcoding WORKS — the dashboard was never the proof' (#39 ) from fix/plex-quicksync-verification into main gate / gate (push) Successful in 1m44s Details gate / conformance (push) Has been skipped Details Reviewed-on: #39 Reviewed-by: Keith Jackson <[email protected]>	2026-10-02 06:43:04 +00:00
jakman85	0d7f94098b	Merge branch 'main' into fix/plex-quicksync-verification gate / gate (pull_request) Successful in 1m46s Details gate / conformance (pull_request) Has been skipped Details	2026-10-02 06:37:04 +00:00
jakman85andClaude Opus 5	f0994c11a0	docs(decisions): record the PR's measured CI result — gitea-settings has drifted gate / gate (pull_request) Failing after 1m44s Details gate / conformance (pull_request) Has been skipped Details Read PR #39's own gate log rather than the tick (CLAUDE.md §2b). Every stage this branch could affect is green: decisions-log, lint, l0, render-golden, l1, and approval-policy (which evaluated for real on the pull_request event and passed — no sensitive path touched). The one failure is `gitea-settings`: required_approvals: DECLARED 1 but LIVE is 0 Gitea has DRIFTED from infra/gitea/expected-branch-protection.json Every other declared setting matched, and one runner is online. So live `main` currently requires ZERO approvals while the declaration says 1, and because .gitea/workflows/gate.yml sets REQUIRE_GITEA_SETTINGS=true, that drift blocks EVERY PR in the repo, not only this one. The check that would notice approvals are unenforced is the check that is failing. Not touched here: infra/gitea/** is a sensitive path (non-author approval required) and outside this branch's documentation-only scope. Re-asserting the protection or changing the declaration is an operator decision in its own PR; local branch fix/admin-approval-override (`616a497`) looks like that work in flight. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-02 01:01:13 -05:00
jakman85andClaude Opus 5	1677dff568	fix(runbooks): Plex hw transcoding WORKS — the dashboard was never the proof gate / gate (pull_request) Failing after 1m46s Details gate / conformance (pull_request) Has been skipped Details Hardware transcoding is confirmed working on prod (2026-10-02): VAAPI for both decode and encode on /dev/dri/renderD128, with zero-copy, measured in `Plex Media Server.log` in the running pod. The device-plugin mechanism from #37 works end to end. `runbooks/plex-quicksync.md` said the Plex dashboard showing `Transcode (hw)` was "the only proof available, and nothing else will ever tell you". That is false: Dashboard -> Activity showed nothing while the feature was fully working, so the page's authoritative check was silent on a healthy system and sent the operator hunting a fault that did not exist. Three inference errors corrected, two kept as named false leads: 1. The LOG is the primary check, not the dashboard. The grep ships in its REDACTING form -- that log carries API URLs with `X-Plex-Token` (CLAUDE.md §6). Also recorded: `Plex Transcoder Statistics.log` is the WRONG file (XML <Variant/> records, no ffmpeg output), and why `grep -a` is required. The dashboard is kept but demoted to best-effort, with its silence stated to prove nothing. 2. An fd snapshot is not evidence. A /proc/*/fd sweep found no holder of renderD128 while hardware transcoding was active -- sessions are short and the scan landed in a gap. The reliable process-level check is the transcoder's argv (-hwaccel, -hwaccel_device, vaapi, renderD128). 3. A missing preference key means nothing. `HardwareAcceleratedCodecs` was absent from all 27 attributes with the toggle on and working, while `HardwareDevicePath` was written and the log said "hardware transcoding: enabled". Only `="0"` is conclusive; the log settles it. Also stale and now fixed: GID 992 is a hostPath-era number. Under the plugin the injected device is root:root, so the image's hook resolves GID 0 and adds `plex` to the root group (`Groups: 0 100 1000`, `groups=1000(plex),0(root),100(users)`). Marked as an inherent property of the plugin's DeviceSpec, not a defect, and explicitly not a reason to reintroduce supplementalGroups (still a no-op -- s6-setuidgid rebuilds the group list from /etc/group). Added a per-title fallback entry (UHD 630 has no AV1 decode), labelled a HYPOTHESIS: the title that fell back was never checked. `runbooks/README.md`'s index entry was stale the same two ways and is corrected. Unchanged on purpose: the "STOP HERE IF PLEX IS Pending" section, the escape hatch, and the no-alert-is-possible reasoning (no metricsEndpoint, no ServiceMonitor, nothing a rule could evaluate). Documentation only -- no catalog/, schemas/, apps/ or platform/ change. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-02 00:49:53 -05:00