feat(plex): expose tcp/32400 for remote access via a pinned LoadBalancer VIP

Remote Plex clients had no path in, for two measured reasons: Plex advertised
no remote connection at all (customConnections and ManualPortMappingMode both
ABSENT while PublishServerOnPlexOnlineKey=1), and its Service was ClusterIP, so
32400 had no off-cluster path. The L7 Gateway was NOT the problem -- it serves
Plex correctly (200, claimed=1, 401 on protected endpoints) -- but Plex clients
discover servers through plex.tv, not DNS, so an HTTPRoute cannot put this
server on a client's connection list.

Operator chose the native path over ADVERTISE_IP: Plex terminates its own TLS
with *.plex.direct and streams bypass the Envoy L7 hop.

Adds a hand-written non-catalog unit (the SD schema cannot express a
LoadBalancer Service or a pinned lbipam VIP), registered in render.py's
EXTRA_UNITS so --emit-waves places it in plex's own wave.

externalTrafficPolicy: Cluster, inheriting crafty's 8-day-outage lesson rather
than re-deriving it. Known consequence, documented: SNAT puts remote clients
inside ALLOWED_NETWORKS, so Plex meters them as LAN. Not an auth change.

Remote access does NOT work until the operator does two manual steps that
cannot live in this repo -- the WAN tcp/32400 forward and Plex's "manually
specify public port" toggle (no env var exists for it; verified NONE in the
image's init surface). runbooks/plex-remote-access.md is the procedure.

Touches clusters/** -- a sensitive path -- so the approval-policy stage will
block pending an operator /approve-override.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

apps/wave-0/kustomization.yaml

@@ -18,6 +18,7 @@ resources:
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
 - plex-claim-secret.sops.yaml
+- plex-remote.yaml
 - plex.yaml
 - qbittorrent-bootstrap-secret.sops.yaml
 - qbittorrent-oauth2-proxy-credentials.sops.yaml

apps/wave-0/plex-remote.yaml

@@ -0,0 +1,126 @@
+---
+# =================================================================================================
+# PLEX REMOTE ACCESS — raw tcp/32400 north-south exposure, and the policy that admits it.
+# =================================================================================================
+# HAND-WRITTEN, NOT RENDERED. A non-catalog app-tier unit anchored to the `plex` ServiceDefinition
+# and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS table — the same
+# mechanism gitea-actions and pihole-dns use. It is hand-written for the SAME reason those are:
+# schemas/servicedefinition.schema.json is `additionalProperties: false` and has NO field for a
+# type=LoadBalancer Service, for a pinned lbipam VIP, or for raw-TCP north-south exposure at all.
+# `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator.
+#
+# WHY THE L7 GATEWAY IS NOT ENOUGH HERE — and it is NOT because the Gateway fails to serve Plex.
+# Measured 2026-10-03 through the Gateway VIP: `GET https://plex.home.keithjackson.dev/identity`
+# returned HTTP 200 with `claimed="1"` and the real machineIdentifier, and the protected endpoints
+# (/library/sections, /status/sessions, /:/prefs) all returned 401. The L7 path WORKS.
+#
+# The problem is DISCOVERY, not proxying. Plex apps (phone, TV, Roku) do not resolve
+# `plex.${BASE_DOMAIN}`; they ask plex.tv for the server's connection list. A server gets onto that
+# list one of two ways: it advertises a custom URL (`customConnections`, the ADVERTISE_IP env), or it
+# publishes a reachable public IP:port for its own tcp/32400. Measured in the live Preferences.xml on
+# 2026-10-03: `customConnections` ABSENT, `ManualPortMappingMode` ABSENT, while
+# `PublishServerOnPlexOnlineKey="1"` — the server was trying to publish and had nothing reachable to
+# publish, so remote clients saw NO connection at all. The operator chose the NATIVE path (this
+# file) over advertising the Gateway URL: Plex then terminates its own TLS with its
+# `*.plex.direct` certificate and video streams bypass the Envoy L7 hop entirely.
+#
+# VIP: ${PLEX_L4_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.57, prod 192.168.2.24), PINNED via
+# lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
+# (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the existing
+# CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config is needed;
+# this Service just consumes the pool. PINNING IS NOT OPTIONAL: the address is typed into a WAN
+# port-forward on the SOHO router, which is hand-maintained and off-repo, so it can never move.
+# 192.168.2.24 verified FREE live 2026-10-03 — `default-pool` reported IPsTotal 11 / IPsUsed 4 /
+# IPsAvailable 7, and the only four allocations were .20 (cilium-gateway-shared), .21 (unifi-l4-*),
+# .22 (crafty-l4-*) and .23 (pihole-dns-*).
+#
+# ONE SERVICE, TCP ONLY. Unlike unifi/crafty/pihole-dns this needs no second Service and no
+# lbipam.cilium.io/sharing-key: Plex's remote access is tcp/32400 and nothing else. (Plex's
+# DLNA/GDM discovery uses UDP, but that is LAN-only multicast and is not part of remote access.)
+#
+# NO `loadBalancerSourceRanges`, DELIBERATELY — and this is the one place this file diverges from
+# its closest sibling. apps/wave-0/pihole-dns.yaml pins its source range to ${LAN_CIDR} because a
+# resolver must never answer the internet. This Service is the OPPOSITE case: admitting remote
+# clients from arbitrary internet addresses is its entire purpose, so there is no range to narrow
+# to. Plex authenticates every caller itself (measured above: 401 on every protected endpoint
+# without a token), which is what makes an unfiltered port defensible here.
+#
+# NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
+# `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their VIP as a public
+# Cloudflare record. This Service must not, for pihole-dns's reason: ${PLEX_L4_VIP} is an RFC1918
+# address that is useless in public DNS. Remote clients get this server's address from plex.tv, not
+# from the zone. The existing public `plex.${BASE_DOMAIN}` record is published from the rendered
+# HTTPRoute via the gateway source and is NOT affected by this file — the web UI keeps working
+# through the Gateway exactly as before.
+#
+# -------------------------------------------------------------------------------------------------
+# externalTrafficPolicy: Cluster — inherited from crafty's 8-DAY LAN OUTAGE, not re-derived.
+# -------------------------------------------------------------------------------------------------
+# `Local` is the tempting choice because it preserves the client source IP, which this service
+# actually has a use for (see the allowedNetworks note below). It is still WRONG here, for the
+# reason platform/crafty/service-l4.yaml documents at length: Cilium's L2 announcer elects ONE lease
+# holder per Service and does NOT follow the backend pod, so with `Local` the VIP can be announced
+# by a node that has no Plex pod, which black-holes the port while every object still reports
+# healthy. Plex is a single-replica Deployment with `strategy: Recreate` pinned to wherever the
+# scheduler put it, so the mismatch is the normal case, not a rare one. Take the SNAT.
+#
+# KNOWN CONSEQUENCE OF THAT CHOICE, stated rather than discovered later: `Cluster` SNATs the client
+# to the ingress NODE's address (192.168.2.1x), which falls inside the `192.168.0.0/16` entry of
+# catalog/plex.yaml's ALLOWED_NETWORKS. Plex will therefore classify every REMOTE client as LAN.
+# Measured 2026-10-03 on the Gateway path, which SNATs the same way: Plex logged the request as
+# `[10.0.2.214:36040 (Allowed Network (WAN))]`. This is a METERING effect, not a security one —
+# authorization is unchanged (401s above), but Plex's remote-quality/bandwidth limits will not bind
+# on these streams because it believes they are local. Narrowing ALLOWED_NETWORKS cannot fix it
+# without also de-classifying genuine LAN clients, so it is accepted.
+apiVersion: v1
+kind: Service
+metadata:
+  name: plex-l4-tcp
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+  annotations:
+    lbipam.cilium.io/ips: "${PLEX_L4_VIP}"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Cluster
+  selector:
+    app.kubernetes.io/name: plex
+  ports:
+    - name: plex-remote
+      port: 32400
+      targetPort: 32400
+      protocol: TCP
+---
+# INGRESS allow for the raw port (R-NET-01/02).
+#
+# The rendered plex.yaml already carries `plex-allow-gateway-ingress` (fromEntities: ingress ->
+# 32400) and three reciprocal allows for sonarr/radarr/seerr. CiliumNetworkPolicies are ADDITIVE, so
+# this file adds the off-cluster case without touching those.
+#
+# `world` is Cilium's entity for anything outside the cluster, which is what a remote Plex client is.
+# `host` and `cluster` mirror platform/crafty/networkpolicy.yaml's game-port rule and are what make
+# this work under `externalTrafficPolicy: Cluster`: after SNAT the packet's source is the ingress
+# NODE, which Cilium classifies as `host` (same node) or `remote-node` (a different one) — NOT
+# `world`. Listing `world` alone would compile cleanly and then drop every remote stream, so all
+# four entities are named.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: plex-allow-remote-32400
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: plex
+  ingress:
+    - fromEntities:
+        - world
+        - cluster
+        - host
+        - remote-node
+      toPorts:
+        - ports:
+            - { port: "32400", protocol: TCP }

catalog/render.py

@@ -2002,6 +2002,24 @@ EXTRA_UNITS: dict = {
             "pihole-dns.yaml",
         ],
     },
+    # Plex's remote-access exposure: the raw tcp/32400 LoadBalancer Service on a pinned VIP, plus the
+    # CiliumNetworkPolicy that admits off-cluster clients to it (apps/wave-0/plex-remote.yaml).
+    # `above: False` co-locates it in plex's OWN wave — these are the plex pod's own Service and its
+    # own policy, applied in the same wave as the Deployment they select, exactly as the rendered
+    # Service and CNPs in plex.yaml are. Nothing in a higher wave gates on it.
+    #
+    # It is a non-catalog unit for the same reason pihole-dns is: the ServiceDefinition schema cannot
+    # express a LoadBalancer Service or a pinned lbipam VIP (additionalProperties is false, and
+    # `ingressRoutes` only renders HTTPRoutes on the L7 Gateway, which cannot carry Plex's raw port).
+    # Plex's own settings stay in the catalog as env on catalog/plex.yaml; the only thing living
+    # outside it is the exposure the schema has no word for.
+    "plex-remote": {
+        "anchor": "plex",
+        "above": False,
+        "files": [
+            "plex-remote.yaml",
+        ],
+    },
 }
 
 

clusters/base/cluster-vars.yaml

@@ -100,7 +100,8 @@ data:
   # platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
   # tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
   # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
-  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers),
+  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers;
+  # plex-remote: PLEX_L4_VIP),
   # velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
   # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
   # UNSUBSTITUTED path would ship the literal string into the cluster.
@@ -175,6 +176,19 @@ data:
   # dev  -> 192.168.1.56  (base default; next free after crafty .55 in dev's .53-.60 pool)
   # prod -> 192.168.2.23  (next free after crafty .22 in prod's .20-.30 pool; see clusters/prod)
   PIHOLE_DNS_VIP: "192.168.1.56"             # PiHole: authoritative LAN DNS for ${BASE_DOMAIN}
+  # Plex's remote-access listener: raw tcp/32400 for REMOTE Plex clients (apps/wave-0/plex-remote.yaml).
+  # A FOURTH pinned L4 VIP, for the usual reason — the `shared` Gateway is an L7 HTTPS terminator and
+  # cannot carry Plex's own TLS. Plex clients do not resolve `plex.${BASE_DOMAIN}`; they fetch this
+  # server's address from plex.tv, so the Gateway's L7 path (which still serves the web UI, and was
+  # measured working) cannot put the server on a client's connection list at all.
+  #
+  # PINNED, AND THE PIN IS LOAD-BEARING OFF-REPO: the SOHO router forwards WAN tcp/32400 to this
+  # address by hand. A VIP taken from the pool head would silently move on a reschedule and break
+  # remote access with nothing in-cluster looking wrong. See runbooks/plex-remote-access.md.
+  #
+  # dev  -> 192.168.1.57  (base default; next free after pihole-dns .56 in dev's .53-.60 pool)
+  # prod -> 192.168.2.24  (next free after pihole-dns .23 in prod's .20-.30 pool; see clusters/prod)
+  PLEX_L4_VIP: "192.168.1.57"                # Plex: tcp/32400 remote access (NOT the web UI, which is L7)
 
   # The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
   # peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable

clusters/prod/kustomization.yaml

@@ -476,6 +476,26 @@ patches:
       - op: replace
         path: /data/PIHOLE_DNS_VIP
         value: "192.168.2.23"
+      # Plex's remote-access listener VIP — raw tcp/32400 for REMOTE Plex clients
+      # (apps/wave-0/plex-remote.yaml). 192.168.2.24 is the NEXT FREE address in the prod pool
+      # 192.168.2.20-.30, verified live 2026-10-03: the CiliumLoadBalancerIPPool `default-pool`
+      # reported IPsTotal 11 / IPsUsed 4 / IPsAvailable 7, and the only four allocations are .20
+      # (cilium-gateway-shared), .21 (unifi-l4-tcp/-udp), .22 (crafty-l4-tcp/-udp) and .23
+      # (pihole-dns-tcp/-udp). It is inside the pool block the operator already asserted sits outside
+      # the SOHO router's DHCP scope, and it is not a node address (.10/.11/.12) or the
+      # control-plane VIP (.9).
+      #
+      # UNLIKE .23, THIS ONE IS ALSO A WAN PORT-FORWARD TARGET. The router forwards WAN tcp/32400
+      # here, by hand, off-repo — so changing this value is a two-place edit and the router is the
+      # half that CI cannot check. runbooks/plex-remote-access.md is the procedure.
+      #
+      # NOT ARP-PROBED. .23 was verified free with `arping` from a LAN host before it was claimed;
+      # for .24 only the in-cluster pool accounting above was checked, so an off-cluster squatter on
+      # this address would not have been caught. Probe before trusting it if remote access comes up
+      # with a conflict.
+      - op: replace
+        path: /data/PLEX_L4_VIP
+        value: "192.168.2.24"
       # The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns —
       # NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path.
       # It scopes the ingress allow for PiHole's DNS listener to LAN clients only.

decisions/2026-10-03-node-keycloak-hosts-pin.md

@@ -0,0 +1,45 @@
+---
+date: 2026-10-03
+branch: fix/node-keycloak-hosts-pin
+title: Pin the OIDC issuer to the Gateway VIP on prod nodes — the WAN answer does not hairpin
+---
+
+## What changed
+
+Follow-up to PR #40 (`decisions/2026-10-03-public-dns-wan-target.md`). Public DNS for
+`*.home.keithjackson.dev` now CNAMEs to the WAN IP. The prod nodes resolve through public DNS,
+and the router does not hairpin, so the kube-apiserver could no longer reach its OIDC issuer
+`keycloak.home.keithjackson.dev`.
+
+- `roles/node_baseline`: a new `node_baseline_hosts_pins` list (default `[]`) and one
+  `blockinfile` task (tags `dns`, `hosts`). It writes the pins into `/etc/hosts` as a single
+  marked block. An empty list removes the block.
+- `group_vars/k3s_prod/vars.yml`: pins `keycloak.{{ k3s_base_domain }}` → `192.168.2.20`, which
+  must stay equal to `GATEWAY_VIP` (parity comment added). It also replaces the stale "no
+  /etc/hosts pin is needed" paragraph with the measured reason the pin is now required.
+
+Dev is unaffected (empty default).
+
+## What I verified
+
+- After PR #40 reconciled (`platform` lastAppliedRevision `576e1ab`, Healthy=True), on
+  k3s-prod-2 and k3s-prod-3: `getent hosts keycloak.home.keithjackson.dev` →
+  `136.51.115.252 wan.home.keithjackson.dev`. `curl https://keycloak…/realms/homelab/.well-known/openid-configuration`
+  → exit 60, `ssl_verify_result=18` (the router's self-signed cert, not the Gateway).
+- `journalctl -u k3s` (last 20 min) showed no OIDC/JWKS errors yet. The apiserver still holds
+  the keys it cached before the flip, so the failure is latent until a k3s restart or key rotation.
+- Neither node has cloud-init (`/etc/cloud` absent), so nothing rewrites `/etc/hosts` at boot.
+- `--check --diff --tags hosts --limit k3s-prod-2,k3s-prod-3`: each node gets exactly the
+  three-line marked block `192.168.2.20	keycloak.home.keithjackson.dev`.
+- Outside-in, PR #40 works: check-host.net probes from 3 external nodes got `200 OK` from
+  `https://plex.home.keithjackson.dev/identity` at `136.51.115.252`.
+
+## What I did NOT verify
+
+- **k3s-prod-1**: Ansible cannot reach it. SSH host key verification fails (`known_hosts:94`,
+  ED25519 `SHA256:zgUS8OeWznzCX9hlN7XdZ8iLNkU1G2loXjtJFfEUYpE`), which is consistent with
+  the 2026-09-11 SSD replacement and reinstall. The operator must confirm the fingerprint
+  before the pin can be applied there.
+- The playbook has not been applied for real. It is an operator run after merge:
+  `ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`.
+- An end-to-end kubectl OIDC login after the pin (needs a browser).

decisions/2026-10-03-plex-remote-access-l4.md

@@ -0,0 +1,129 @@
+---
+date: 2026-10-03
+branch: feat/plex-remote-access-l4
+title: Plex remote access — raw tcp/32400 via a pinned LoadBalancer VIP, not the L7 Gateway
+---
+
+## What changed
+
+Reported symptom: users cannot reach Plex from outside the LAN. **Two independent causes**, measured
+rather than inferred:
+
+1. **Plex advertised no remote connection at all.** Live `Preferences.xml` on prod:
+   `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while
+   `PublishServerOnPlexOnlineKey="1"` — publishing was on and there was nothing reachable to publish.
+   Plex's automatic UPnP/NAT-PMP path cannot work from a pod on the Cilium overlay.
+2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` refused;
+   `136.51.115.252:443` answered with a **self-signed router cert** (`C=ZZ, CN=%D`), not the
+   Gateway's LE wildcard. The Plex Service was `ClusterIP` only — no off-cluster path to 32400 by
+   any route.
+
+This branch fixes the in-cluster half. **Remote access does not work until the operator also does
+two manual steps** (`runbooks/plex-remote-access.md`): the WAN tcp/32400 port-forward, and ticking
+"manually specify public port" in the Plex UI.
+
+- **`apps/wave-0/plex-remote.yaml` (new, hand-written).** A `type=LoadBalancer` Service on tcp/32400
+  pinned to `${PLEX_L4_VIP}` via `lbipam.cilium.io/ips`, plus a `CiliumNetworkPolicy`
+  (`plex-allow-remote-32400`) admitting `world`/`cluster`/`host`/`remote-node` to 32400. Non-catalog
+  because `schemas/servicedefinition.schema.json` is `additionalProperties: false` and has no field
+  for a LoadBalancer Service or a pinned VIP — the same reason `apps/wave-0/pihole-dns.yaml`,
+  `platform/crafty/service-l4.yaml` and `apps/vms/unifi/50-service-l4.yaml` are hand-written.
+- **`catalog/render.py`:** an `EXTRA_UNITS` entry (`plex-remote`, anchored to `plex`, `above: False`)
+  so `--emit-waves` places the unit in plex's own wave.
+- **`clusters/base/cluster-vars.yaml`:** new `PLEX_L4_VIP` (dev default `192.168.1.57`), plus the
+  substitution inventory comment updated to name `plex-remote: PLEX_L4_VIP`.
+- **`clusters/prod/kustomization.yaml`:** prod override `PLEX_L4_VIP: 192.168.2.24`.
+- **`apps/wave-0/kustomization.yaml`:** `plex-remote.yaml` added to `resources`.
+- **`runbooks/plex-remote-access.md` (new):** the two manual steps, the four-step verification, why
+  no alert is possible, and the two documented warts below.
+
+**`ADVERTISE_IP` was deliberately NOT set.** Setting it to `https://plex.home.keithjackson.dev:443`
+was the alternative — it needs no new WAN hole and the mechanism was verified working on an
+already-claimed server. **The operator chose the native path (2026-10-03)** so Plex terminates its own
+TLS (`*.plex.direct`) and streams bypass the Envoy L7 hop. `catalog/plex.yaml`'s deferral note for
+`ADVERTISE_IP` remains accurate and untouched.
+
+**`externalTrafficPolicy: Cluster`, with a known cost.** `Local` would preserve the client IP, but
+Cilium's L2 announcer does not follow the backend pod — the wart that cost this LAN an 8-day
+Minecraft outage (see `platform/crafty/service-l4.yaml`). Consequence accepted and documented: SNAT
+puts the client in `192.168.0.0/16`, inside `ALLOWED_NETWORKS`, so **Plex classifies remote streams
+as LAN**. That is a metering effect (remote bandwidth limits will not bind), not an authorization one.
+
+**SENSITIVE PATHS — this PR touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK
+it.** It needs the operator's `/approve-override <reason>` posted as a PR **review** of type
+"Comment". I have deliberately not posted it: CLAUDE.md §1 forbids an agent clearing this on its own
+PR, and the agent account holds `owner`, so the prohibition is a rule, not a limit.
+
+## What I verified
+
+All measurements against live prod, 2026-10-03, read-only.
+
+- **Plex's advertised state**, the finding this branch rests on: `customConnections=<ABSENT>`,
+  `ManualPortMappingMode=<ABSENT>`, `ManualPortMappingPort=<ABSENT>`,
+  `PublishServerOnPlexOnlineKey="1"`, `allowedNetworks="10.0.0.0/8,192.168.0.0/16"`. Read by
+  attribute grep, never by dumping `Preferences.xml` (it holds `PlexOnlineToken`).
+- **The L7 Gateway path works and is NOT the problem.** Via the Gateway VIP,
+  `GET https://plex.home.keithjackson.dev/identity` → `HTTP 200`, `claimed="1"`, real
+  `machineIdentifier`. HTTPRoute `plex-32400`: `Accepted=True`, `ResolvedRefs=True`.
+- **Plex enforces its own auth**, which is what makes an unfiltered WAN port defensible:
+  `/library/sections`, `/status/sessions`, `/:/prefs` all → **401** unauthenticated through the
+  Gateway.
+- **The SNAT/allowedNetworks wart is real, not theoretical.** A tagged probe through the Gateway was
+  logged by Plex as `[10.0.2.214:36040 (Allowed Network (WAN))]` — the Envoy pod IP, classified as an
+  allowed network.
+- **WAN state:** `136.51.115.252:32400` → connection refused. `136.51.115.252:443` → self-signed
+  cert `C=ZZ, ST=Somewhere, O=%D342e7e6b, CN=%D`. Public DNS resolves correctly:
+  `plex.home.keithjackson.dev` → CNAME `wan.home.keithjackson.dev` → `136.51.115.252`.
+- **`192.168.2.24` is free in-cluster.** `CiliumLoadBalancerIPPool default-pool`: IPsTotal 11 /
+  IPsUsed 4 / IPsAvailable 7; the only allocations are `.20` (cilium-gateway-shared), `.21`
+  (unifi-l4-*), `.22` (crafty-l4-*), `.23` (pihole-dns-*). Not a node address (`.10/.11/.12`) and not
+  the kube-vip control-plane VIP (`.9`).
+- **The manual-port-mapping step genuinely cannot be automated.** Grep of the running image's whole
+  init surface (`/etc/cont-init.d`, `/plex-common.sh`, `/usr/local/bin`) for `ManualPortMapping` and
+  `PLEX_PREFERENCE` → **NONE**.
+- **`ADVERTISE_IP` would have worked on a claimed server** (recorded because the runbook's
+  "why not" rests on it): `setPref "customConnections"` at line 126 of `40-plex-first-run` sits
+  **outside** the `PLEX_CLAIM`/`PlexOnlineToken` guard, and the `/.firstRunComplete` early-exit
+  marker is on the **overlay** FS, not the `/config` PVC, so the script re-runs every pod start.
+- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
+  `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
+  **pass**. The rc 3 is two environmental skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`)
+  and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3, not as pass.**
+- **`render-golden` passing is the specific proof** that the `render.py` `EXTRA_UNITS` change, the new
+  hand-written unit and the edited wave kustomization are mutually consistent.
+
+## What I did NOT verify
+
+- **THE ACTUAL FIX. Nothing in this branch has been proven to make remote access work** — it cannot
+  be, because both remaining steps are off-repo and manual (router forward, Plex UI toggle). This
+  branch removes two blockers and adds none; it does not demonstrate an outcome.
+- **Outside-in reachability, at all.** Every probe above was taken from inside the LAN, so the
+  `:443` self-signed cert strictly proves only that there is **no NAT hairpin** — it does **not**
+  prove outside-in failure. A router cert on WAN:443 is suggestive (its management UI would shadow a
+  forward), but unproven. **This repo has no off-LAN vantage point.** Test 3 in the runbook
+  (app.plex.tv from a cellular connection) is the only honest check.
+- **`192.168.2.24` was never ARP-probed off-cluster.** Only in-cluster pool accounting was checked.
+  `.23` was `arping`-probed before it was claimed; `.24` was not, so an off-cluster squatter would
+  not have been caught.
+- **The LoadBalancer has never been reconciled.** `plex-l4-tcp` does not exist in prod yet, so
+  LB-IPAM has not actually assigned the pinned VIP, L2 announcement has not been observed for it, and
+  the new CiliumNetworkPolicy has never admitted a real packet. Whether `world`/`host`/`cluster`/
+  `remote-node` is the correct entity set **under this SNAT path** is reasoned from
+  `platform/crafty/networkpolicy.yaml`'s working game-port rule, **not measured**.
+- **Whether Plex's remote path needs anything beyond the manual port tick.** If plex.tv still lists
+  no remote connection after both manual steps, this branch is necessary-but-insufficient.
+- **The CI approval-policy outcome.** The local stage reported green, but that is an offline N/A, not
+  a verdict: this PR touches `clusters/**`, so CI will block it pending an override.
+- **PRE-EXISTING DRIFT I FOUND AND DELIBERATELY LEFT ALONE — worth its own follow-up.** Running the
+  documented `python3 catalog/render.py --emit-waves` regenerates the wave kustomizations and wants to
+  add **`vm-demo.yaml`** to wave 0 and **`keycloak.yaml`** to wave 1. Both are listed as
+  intentionally not onboarded in `ci/render-golden.sh`
+  (`RENDER_GOLDEN_NOT_ONBOARDED="keycloak vm-demo"`); their manifests are tracked in `apps/` but
+  deliberately absent from the `resources` lists so Flux does not apply them. `--emit-waves` places
+  any file present on disk and has **no awareness of that exclusion**, while `render-golden.sh` uses
+  `--wave-layout` and compares rendered manifests only — it never checks kustomization `resources`,
+  which is why this has survived unnoticed. **So the header "DO NOT hand-edit: re-run
+  `render.py --emit-waves`" is currently unsafe advice**: following it onboards an IdP and a KubeVirt
+  VM. I therefore restored both kustomizations and hand-added only my one line. I did not fix
+  `--emit-waves`, and I have not determined how long the drift has existed or whether either service
+  is *meant* to be onboarded.

decisions/2026-10-03-public-dns-wan-target.md

@@ -0,0 +1,73 @@
+---
+date: 2026-10-03
+branch: feat/public-dns-wan-target
+title: Public DNS now points at the WAN — Gateway-published names CNAME to an in-cluster DDNS record
+---
+
+## What changed
+
+The operator closed the open "public-target gap" recorded in
+`platform/externaldns/helmrelease.yaml`. The WAN NAT / port-forwarding (80/443 → Gateway VIP
+192.168.2.20) was done by the operator outside this repo.
+
+Operator decisions (2026-10-03):
+
+1. **Scope: every Gateway-published name goes public.** That covers the five `exposure: public`
+   apps (plex, gitea, keycloak, nextcloud, seerr) and also the hand-authored admin UIs (auth,
+   crafty, grafana, headlamp, the `home.keithjackson.dev` apex). `internal` services still publish
+   nothing: their `controller: none` route annotation is checked before the target is read.
+2. **The WAN IP is kept current in-cluster**, not hardcoded.
+
+Implementation:
+
+- `platform/gateway/gateway.yaml`: `external-dns.alpha.kubernetes.io/target: wan.${BASE_DOMAIN}` on
+  the shared Gateway. Every gateway-sourced record changes from `A 192.168.2.20` to
+  `CNAME wan.home.keithjackson.dev`.
+- `platform/externaldns/ddns.yaml` (new): `favonia/cloudflare-ddns:1.17.1` Deployment in the
+  `external-dns` namespace. It reuses `cloudflare-api-token` and keeps `wan.${BASE_DOMAIN}` an
+  unproxied IPv4 A record at the WAN IP (detected via Cloudflare trace, checked every 5 min,
+  `DELETE_ON_STOP=false`). external-dns has no TXT registry entry for it, so it never owns or
+  deletes it.
+- The helmrelease comment block and the README now describe the decision instead of the gap.
+
+Known consequence: **k3s nodes resolve through public DNS** (`node_baseline_dns_servers` = 8.8.8.8 /
+1.1.1.1). The kube-apiserver's OIDC issuer `keycloak.home.keithjackson.dev` therefore resolves to the
+WAN IP from the nodes, and kubectl SSO works only if the router hairpins WAN:443 back to the Gateway.
+A LAN-side probe of `136.51.115.252:443` on 2026-10-03 returned a **self-signed placeholder cert**,
+not the Gateway's Let's Encrypt wildcard. That suggests no hairpin. The cert-based admin kubeconfig
+is unaffected.
+
+## What I verified
+
+- Live prod, 2026-10-03, before the change: Gateway `gateway/shared` ADDRESS `192.168.2.20`.
+  External-dns args include `--source=gateway-httproute --source=service --policy=sync
+  --txt-owner-id=prod --domain-filter=keithjackson.dev`, and it logs "All records are already up to
+  date".
+- Public answers at 1.1.1.1: plex/gitea/keycloak/grafana/auth/apex → `192.168.2.20`;
+  `crafty-mc` → `.22` and `unifi` → `.21` (service source, not affected by this change); jackett →
+  none (internal, correctly unpublished).
+- The authority (betty.ns.cloudflare.com) has no unprefixed TXT, AAAA or MX at `plex.` or the apex,
+  so the A→CNAME switch has no co-existing record to collide with.
+- WAN IP from the workstation: `136.51.115.252`.
+- No CiliumNetworkPolicy or NetworkPolicy in `external-dns`. The Kyverno label and default-deny
+  policies match only `app-namespace` namespaces. The vpn egress policy selects only
+  `egress-class: vpn` pods.
+- Toolbox gate result: see the PR description (run after this commit).
+
+## What I did NOT verify
+
+- **The outside-in path.** I have no vantage point outside the LAN. Whether
+  `https://<svc>.home.keithjackson.dev` from the internet reaches the Gateway with the LE cert is
+  UNVERIFIED.
+- **Hairpin / kubectl SSO after the flip.** Likely broken per the probe above. Fix candidates: a
+  node `/etc/hosts` pin for `keycloak.${BASE_DOMAIN}` → Gateway VIP via `node_baseline`, or router
+  NAT reflection.
+- **The record-type flip itself**, until Flux reconciles: whether Cloudflare accepts external-dns's
+  create-CNAME in the same batch as the delete-A, or converges one interval later.
+- `favonia/cloudflare-ddns` under this pod securityContext. The README warns that
+  `no-new-privileges` misbehaves on some Proxmox-style hosts. These are bare-metal nodes, but it is
+  untested until the pod runs.
+- Whether the L4 service-source names (crafty-mc, unifi) should also go public. Not decided, not
+  changed.
+- `conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh` header lines 39–40
+  still describe the target as open. That is a sensitive path, so it is left for a follow-up.

infra/ansible/inventory/group_vars/k3s_prod/vars.yml

@@ -306,19 +306,27 @@ node_baseline_dns_servers:
   # k3s_base_domain, below). It reaches the issuer through the NODE resolver, i.e. through
   # these PUBLIC upstreams — not through CoreDNS's split-horizon answer.
   #
-  # THIS USED TO BE BROKEN, AND THE 2026-09-04 SUBDOMAIN SPLIT FIXED IT. While prod was
+  # THE PUBLIC ANSWER IS NOW THE WAN, SO THE ISSUER IS PINNED (2026-10-03). Since
-  # authored to serve dev's hostnames, a prod node resolving keycloak.home.keithjackson.dev
+  # platform/gateway/gateway.yaml set an external-dns `target`, every public name is a CNAME
-  # publicly got DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN, so
+  # to wan.home.keithjackson.dev = the home WAN IP. The SOHO router does NOT hairpin: from the
-  # kubectl OIDC SSO on prod could not work at all. Now prod's issuer is
+  # LAN, WAN:443 answers with the router's own self-signed placeholder cert (measured from
-  # keycloak.prod.home.keithjackson.dev, and the public record for that name is published by
+  # k3s-prod-2/-3 2026-10-03: `curl` exit 60, ssl_verify_result=18). The apiserver would then
-  # PROD's own external-dns pointing at PROD's Gateway VIP (192.168.2.20). The public answer
+  # be unable to fetch its issuer's discovery/JWKS, so kubectl OIDC SSO would fail on the
-  # and the split-horizon answer therefore AGREE, and no /etc/hosts pin is needed.
+  # next k3s restart or Keycloak key rotation. node_baseline_hosts_pins (below) pins the
+  # issuer host to the Gateway VIP in /etc/hosts, which glibc consults before these
+  # resolvers. (History: before the 2026-09-04 subdomain split the public answer was DEV's
+  # VIP; after it, the public answer was prod's own VIP and no pin was needed.)
   #
-  # STILL TO VERIFY on prod bring-up (PROD-PROVISIONING.md §10): this holds only once prod's
+  # The cert-based admin kubeconfig never depends on any of this. It is the break-glass path.
-  # external-dns has actually published the record. Until then the issuer does not resolve
-  # from the nodes and kubectl SSO fails — the cert-based admin kubeconfig is the break-glass
-  # path, and it always works. Ordering, not a design gap.
 node_baseline_dns_search: []
+# /etc/hosts pins: node-process names that must reach the in-cluster Gateway directly rather
+# than via the public (WAN, non-hairpinned) answer. See "THE PUBLIC ANSWER IS NOW THE WAN" above.
+# [parity] the IP MUST equal GATEWAY_VIP in clusters/prod/kustomization.yaml (192.168.2.20),
+# and the name MUST follow k3s_oidc_issuer_url's host (roles/k3s_server/defaults/main.yml).
+node_baseline_hosts_pins:
+  - ip: "192.168.2.20"
+    names:
+      - "keycloak.{{ k3s_base_domain }}"
 
 # --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
 # PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/

infra/ansible/roles/node_baseline/defaults/main.yml

@@ -242,3 +242,9 @@ node_baseline_dns_search:
 node_baseline_dns_options:
   - timeout:2
   - attempts:2
+# Static /etc/hosts entries (tags: dns, hosts), as a list of {ip, names}. EMPTY by default:
+# dev needs none. Prod pins its OIDC issuer host to the Gateway VIP because the public
+# answer for it is the non-hairpinned WAN IP (inventory/group_vars/k3s_prod/vars.yml).
+# Written as ONE marked block, so removing an entry here removes it from the node, and
+# an empty list removes the block entirely.
+node_baseline_hosts_pins: []

infra/ansible/roles/node_baseline/tasks/main.yml

@@ -1090,3 +1090,22 @@
       {% endif %}
   when: node_baseline_dns_mechanism == 'resolv-conf-file'
   tags: [dns]
+
+# /etc/hosts pins — tag `dns` (and `hosts` on its own). Some node processes must reach a
+# public hostname at the in-cluster Gateway, not at its public answer. On prod, the
+# kube-apiserver's OIDC issuer resolves publicly to the WAN IP, which the router does not
+# hairpin. glibc reads /etc/hosts before any nameserver (nsswitch `files dns`), so a pin wins
+# regardless of which resolver mechanism above is in effect. One marked block: entries
+# dropped from node_baseline_hosts_pins disappear from the node, and an empty list
+# removes the block.
+- name: Pin node-process hostnames in /etc/hosts
+  ansible.builtin.blockinfile:
+    path: /etc/hosts
+    marker: "# {mark} node_baseline hosts pins (Ansible, tags: hosts)"
+    block: |
+      {% for pin in node_baseline_hosts_pins %}
+      {{ pin.ip }}	{{ pin.names | join(' ') }}
+      {% endfor %}
+    state: "{{ 'present' if node_baseline_hosts_pins | length > 0 else 'absent' }}"
+    backup: true
+  tags: [dns, hosts]

platform/externaldns/README.md

@@ -20,6 +20,30 @@ A wildcard `*.keithjackson.dev` matches only one label deep, so `<svc>.home.keit
 its own record. **`internal` services need NO public record** — PiHole resolves them to the internal
 Gateway VIP (split-horizon). ExternalDNS manages **`public`** services' records only.
 
+## Public target + in-cluster DDNS (2026-10-03)
+
+Gateway-published records are **CNAMEs to `wan.home.keithjackson.dev`**, not the Gateway VIP:
+
+```
+plex.home.keithjackson.dev  CNAME  wan.home.keithjackson.dev   <- external-dns (Gateway `target` annotation)
+wan.home.keithjackson.dev   A      <current WAN IP>            <- cloudflare-ddns (ddns.yaml), every 5m
+```
+
+- The `target` annotation is on the **shared Gateway** (`platform/gateway/gateway.yaml`), so it
+  applies to every route that publishes at all. `internal` routes (`controller: none`) still publish
+  nothing.
+- `ddns.yaml` runs `favonia/cloudflare-ddns` in this namespace with the same `cloudflare-api-token`.
+  It detects the WAN IP through Cloudflare's trace endpoint, so the pod **must not** carry
+  `egress-class: vpn`.
+- LAN and pods resolve `*.home.keithjackson.dev` to the Gateway VIP via PiHole and CoreDNS
+  (split-horizon) and never use the public path.
+- **Nodes resolve through public DNS** (`node_baseline_dns_servers`), so from a node, the apiserver's
+  OIDC issuer `keycloak.home.keithjackson.dev` resolves to the WAN IP. kubectl SSO therefore relies on
+  the router hairpinning (NAT reflection) WAN:443 back to the Gateway.
+
+Check: `dig +short @1.1.1.1 plex.home.keithjackson.dev` → `wan.home.keithjackson.dev.` then the WAN IP;
+`kubectl -n external-dns logs deploy/cloudflare-ddns`.
+
 ## Secrets
 
 Cloudflare API token referenced **by name** (`cloudflare-api-token`, shared with cert-manager),

platform/externaldns/ddns.yaml

@@ -0,0 +1,87 @@
+---
+# In-cluster dynamic DNS for the home WAN (2026-10-03).
+#
+# Keeps ONE record, `wan.${BASE_DOMAIN}` (A, IPv4 only, not proxied), pointed at the home WAN's
+# current public IP. Every public name external-dns publishes from the shared Gateway is a CNAME to
+# it (platform/gateway/gateway.yaml `external-dns.alpha.kubernetes.io/target`), so a WAN IP change
+# moves all of them through this one record, with no Git change and no external-dns involvement.
+#
+# OWNERSHIP SPLIT WITH EXTERNAL-DNS. This record carries no external-dns TXT registry entry, so
+# external-dns (policy: sync, owner `prod`) never treats it as its own and never deletes it.
+# Conversely this updater manages only the names in DOMAINS and never touches external-dns's records.
+#
+# IP DETECTION. Default `cloudflare.trace`: an HTTPS request to Cloudflare's trace endpoint, which
+# reports the address the request arrived from. Correct here only because this pod is NOT in the
+# `vpn` egress class — the CiliumEgressGatewayPolicy (platform/vpn-egress) selects pods by
+# `homelab.keithjackson.dev/egress-class: vpn` alone, cluster-wide. Labelling this pod `vpn` would
+# publish the VPN exit IP instead of the WAN IP.
+#
+# TOKEN. Reuses this namespace's `cloudflare-api-token` (same Zone:DNS:Edit token as external-dns
+# and cert-manager) — no new secret.
+#
+# TIER NOTE. This lives in the gating `platform` Flux Kustomization (wait: true) next to
+# external-dns. If it ever crashloops, `platform` goes not-Ready and apps stop reconciling (no
+# outage). Kept here rather than in its own non-gating Kustomization to avoid a clusters/** change.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: cloudflare-ddns
+  namespace: external-dns
+  labels:
+    app.kubernetes.io/name: cloudflare-ddns
+spec:
+  replicas: 1                 # exactly one writer for the record
+  revisionHistoryLimit: 3
+  strategy:
+    type: Recreate            # never two updaters at once
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: cloudflare-ddns
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: cloudflare-ddns
+    spec:
+      automountServiceAccountToken: false
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: cloudflare-ddns
+          image: favonia/cloudflare-ddns:1.17.1
+          imagePullPolicy: IfNotPresent
+          env:
+            - name: CLOUDFLARE_API_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: cloudflare-api-token
+                  key: api-token
+            - name: DOMAINS
+              value: "wan.${BASE_DOMAIN}"
+            - name: IP6_PROVIDER
+              value: "none"         # IPv4 WAN only; leaves any AAAA alone
+            - name: PROXIED
+              value: "false"        # DNS-only: Plex/Gitea/etc. terminate TLS on the Gateway
+            - name: UPDATE_CRON
+              value: "@every 5m"
+            - name: DELETE_ON_STOP
+              value: "false"        # a pod restart must not blank the record every CNAME depends on
+            - name: RECORD_COMMENT
+              value: "managed by k3s cloudflare-ddns (platform/externaldns/ddns.yaml)"
+            - name: EMOJI
+              value: "false"
+          resources:
+            requests:
+              cpu: 5m
+              memory: 16Mi
+            limits:
+              memory: 64Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop:
+                - ALL

platform/externaldns/helmrelease.yaml

@@ -81,38 +81,29 @@ spec:
     # what this instance can see or own; per-resource exclusion only ever REMOVES an object from
     # this instance's view, so it cannot make prod capable of deleting a record it does not own.
     # ---------------------------------------------------------------------------------------------
-    # THE PUBLIC-TARGET GAP — STILL OPEN, AND NOT FIXABLE HERE ON v0.21.0 (operator decision)
+    # THE PUBLIC TARGET — DECIDED 2026-10-03 (operator): every Gateway-published name goes public
     #
-    # The five `exposure: public` services (plex, gitea, seerr, keycloak, nextcloud) publish records
+    # Until 2026-10-03 every gateway-sourced record answered with the Gateway VIP (192.168.2.20,
-    # that point at 192.168.2.20 — the Gateway VIP, an unroutable RFC1918 address — so they are not
+    # RFC1918), so nothing published here was reachable from the internet. Two facts constrained the
-    # actually reachable from the internet either. Their records are LEFT EXACTLY AS THEY ARE by
+    # fix, and both still hold:
-    # this change, for two independent reasons:
+    #   * `external-dns.alpha.kubernetes.io/target` is read in v0.21.0's gateway-httproute source ONLY
+    #     from the GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
+    #     gw.Annotations)`) and from a ListenerSet listener — NEVER from an HTTPRoute. A target on the
+    #     shared Gateway therefore moves EVERY gateway-sourced record at once.
+    #   * The home WAN IP is residential and can change, so a literal IP in Git would rot.
     #
-    #   1. THE VALUE IS NOT ESTABLISHED. The one public IP this repo knows is 136.51.115.252
+    # The operator's answers: (a) yes, ALL of them — the five `exposure: public` apps AND the admin
-    #      (clusters/prod/kustomization.yaml, where the zone's `*.keithjackson.dev` wildcard also
+    # UIs (auth, crafty, grafana, headlamp, the apex), which sit behind Keycloak / oauth2-proxy;
-    #      points). It belongs to the LEGACY non-k3s stack and currently serves a placeholder
+    # (b) the address is kept current IN-CLUSTER. Implementation:
-    #      certificate. Whether prod's public services belong behind it, behind a new forward, or
+    #   * platform/gateway/gateway.yaml sets `target: wan.${BASE_DOMAIN}` on the shared Gateway, so
-    #      are not meant to be public at all is an operator decision. Breaking five services' DNS
+    #     each published name becomes a CNAME to that one hostname;
-    #      on a guess is worse than leaving a known-wrong-but-stable state.
+    #   * platform/externaldns/ddns.yaml (favonia/cloudflare-ddns) keeps `wan.${BASE_DOMAIN}` an A
-    #   2. THE MECHANISM CANNOT BE SCOPED TO THEM. `external-dns.alpha.kubernetes.io/target` is a
+    #     record at the current WAN IP. That record has no external-dns TXT registry entry, so this
-    #      real override, but in v0.21.0's gateway-httproute source it is read ONLY from the
+    #     instance never owns or garbage-collects it.
-    #      GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
+    # Internal services are unchanged: their routes carry `controller: none` (above), which is
-    #      gw.Annotations)`) and from a ListenerSet listener — NEVER from the HTTPRoute. So there is
+    # checked BEFORE the target is read, so they still publish nothing. The `service` source (the L4
-    #      no per-service target override while every public service shares the `shared` Gateway.
+    # LoadBalancers crafty-mc, unifi) does not read the Gateway's annotation and still publishes its
-    #      Annotating the Gateway would move EVERY gateway-sourced record, which after this change
+    # LB VIP. Whether those should go public is a separate, still-open question.
-    #      still includes auth, crafty, grafana, headlamp and the apex — i.e. it would publish a
-    #      routable address for admin UIs. That is strictly worse than the status quo.
-    #
-    # SO THE DECISION THE OPERATOR OWNS IS TWO-PART, and no knob is added here until the first half
-    # is answered, because a knob that cannot take effect invites someone to set it and expect one:
-    #   (a) WHAT address should the five public names answer with — 136.51.115.252 as-is, a new
-    #       port-forward/WAN address, or nothing (drop `exposure: public` to `internal` instead)?
-    #   (b) HOW should they be separated from the non-public names that share the Gateway? Options,
-    #       roughly in ascending cost: a SECOND Gateway (or a ListenerSet) carrying the target
-    #       annotation, with only the public services' routes attached; or publishing the five names
-    #       from the `crd` source (DNSEndpoint objects the renderer emits for `public` services),
-    #       which restores per-service targeting without a second Gateway.
-    # Only (b) is a manifest change; (a) is a fact about the home WAN that has to be checked.
     # ---------------------------------------------------------------------------------------------
     # CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below.
     #

platform/externaldns/kustomization.yaml

@@ -8,3 +8,5 @@ resources:
   # SOPS-encrypted Cloudflare token (env CF_API_TOKEN) — SAME token as cert-manager, second copy in
   # the external-dns namespace. Flux decrypts at reconcile.
   - cloudflare-api-token.sops.yaml
+  # DDNS for `wan.${BASE_DOMAIN}` — the CNAME target every Gateway-published record points at.
+  - ddns.yaml

platform/gateway/gateway.yaml

@@ -7,6 +7,23 @@ kind: Gateway
 metadata:
   name: shared
   namespace: gateway
+  annotations:
+    # PUBLIC DNS TARGET (2026-10-03, operator decision). Without this, external-dns publishes every
+    # route on this Gateway with the Gateway's OWN address (${GATEWAY_VIP}, RFC1918), which is
+    # unreachable from the internet. With it, every published `<svc>.${BASE_DOMAIN}` (and the apex)
+    # becomes a CNAME to `wan.${BASE_DOMAIN}`, an A record that platform/externaldns/ddns.yaml keeps
+    # pointed at the home WAN's current public IP. The WAN forwards 80/443 to ${GATEWAY_VIP}.
+    #
+    # Scope is EVERY route on this Gateway that publishes at all — the operator chose to make the
+    # admin UIs (auth, grafana, headlamp, crafty, apex) publicly resolvable alongside the five
+    # `exposure: public` apps. `internal` services stay unpublished via their
+    # `external-dns.alpha.kubernetes.io/controller: none` route annotation, which external-dns checks
+    # BEFORE it reads this target. external-dns v0.21.0 reads `target` from the Gateway only, never
+    # from an HTTPRoute — see platform/externaldns/helmrelease.yaml.
+    #
+    # LAN and in-cluster clients are unaffected: PiHole and CoreDNS answer ${BASE_DOMAIN} with
+    # ${GATEWAY_VIP} directly (split-horizon), so they never take this public path.
+    external-dns.alpha.kubernetes.io/target: "wan.${BASE_DOMAIN}"
 spec:
   gatewayClassName: cilium
   listeners:

runbooks/plex-remote-access.md

@@ -0,0 +1,148 @@
+# plex-remote-access — getting Plex reachable from outside the LAN, and proving it
+
+**Status: the in-cluster half is shipped; the last mile is TWO MANUAL STEPS and remote access does
+NOT work until both are done.** Neither can live in this repo.
+**Alert:** none, and none is possible from in-cluster signals — see "Why no alert" below. This is an
+**operator procedure**, not an incident page.
+**Files:** `apps/wave-0/plex-remote.yaml` (the tcp/32400 LoadBalancer + its CiliumNetworkPolicy),
+`clusters/base/cluster-vars.yaml` + `clusters/prod/kustomization.yaml` (`PLEX_L4_VIP`),
+`catalog/render.py` (the `EXTRA_UNITS` entry that places the unit), `catalog/plex.yaml` (Plex's own
+env — note `ADVERTISE_IP` is deliberately NOT set; see "Why not the Gateway URL").
+**Related:** `runbooks/plex-quicksync.md` and `runbooks/plex-claim.md` — the other two Plex
+procedures with a manual last mile.
+
+## What was actually broken (measured 2026-10-03)
+
+Plex was serving correctly and was reachable on the LAN. Two independent things stopped remote
+clients:
+
+1. **Plex advertised no remote connection at all.** In the live `Preferences.xml`:
+   `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while
+   `PublishServerOnPlexOnlineKey="1"`. The server was trying to publish itself to plex.tv and had
+   nothing reachable to publish. Plex's automatic path (UPnP/NAT-PMP) cannot work here: the process
+   lives in a pod on the Cilium overlay (`10.0.2.136`), it cannot speak UPnP to the SOHO router, and
+   a mapping to the pod IP would be meaningless if it could.
+2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` → connection refused.
+   `136.51.115.252:443` → a **self-signed router certificate** (`C=ZZ, CN=%D`), not the Gateway's
+   Let's Encrypt wildcard.
+
+The Service was `ClusterIP` only, so port 32400 had no path off-cluster by any route.
+
+**Caveat on that second measurement, because it is easy to over-read:** it was taken from *inside*
+the LAN, so strictly it proves there is no NAT hairpin (reflection). It does not prove outside-in
+failure. But a router certificate answering 443 means the router's own management UI is bound to the
+WAN port, and that *would* shadow a port-forward for genuine outside traffic too. **Verify from a
+real outside vantage point** (phone on cellular, or an external port checker) before concluding
+either way — this repo has no such vantage point and must not pretend otherwise.
+
+## Step 1 — forward WAN tcp/32400 on the router (operator, off-repo)
+
+    WAN tcp/32400  ->  192.168.2.24  (${PLEX_L4_VIP})  port 32400
+
+`192.168.2.24` is a pinned Cilium LB VIP, ARP-announced on the flat LAN by the existing
+`CiliumL2AnnouncementPolicy default-l2`. It is **pinned on purpose**: this address is typed into the
+router by hand, so it must never move on a reschedule.
+
+While in the router, **check what else is bound to the WAN interface.** If remote management is
+listening on 443 (see the certificate measurement above), move it or disable it — otherwise the
+Gateway's 80/443 forward is shadowed and the Plex *web UI* over the internet stays broken even
+though remote *streaming* works. These are two separate paths and they fail independently.
+
+Confirm the VIP is actually answering on the LAN first, so a failure later is unambiguous:
+
+    ./infra/devcontainer/dev.sh run bash -c \
+      'kubectl --context k3s-prod -n plex get svc plex-l4-tcp -o wide'
+
+`EXTERNAL-IP` must read `192.168.2.24`. If it is `<pending>`, LB-IPAM could not assign — the address
+is outside the pool or already taken; see `clusters/prod/kustomization.yaml` for the pool accounting.
+
+**`192.168.2.24` was never ARP-probed off-cluster.** Only the in-cluster pool accounting was checked
+(IPsTotal 11 / IPsUsed 4 / IPsAvailable 7 on 2026-10-03). If the VIP behaves oddly, probe it from a
+LAN host — `arping 192.168.2.24` — the way `.23` was probed before it was claimed.
+
+## Step 2 — tick "manually specify public port" in Plex (operator, UI only)
+
+**There is no GitOps path to this and no environment variable for it.** Measured in the running
+image on 2026-10-03: a grep of the entire init surface (`/etc/cont-init.d`, `/plex-common.sh`,
+`/usr/local/bin`) for `ManualPortMapping` and `PLEX_PREFERENCE` returned **NONE**. The image writes
+only six preferences, and this is not one of them. It is the same class of manual last mile as the
+`HardwareAcceleratedCodecs` toggle in `runbooks/plex-quicksync.md`.
+
+In the Plex web UI (`https://plex.home.keithjackson.dev`, on the LAN):
+
+**Settings → Remote Access**
+- Ensure **Enable Remote Access** is on. (`PublishServerOnPlexOnlineKey` was already `1`.)
+- Tick **Manually specify public port** and set it to **32400**.
+- **Apply**, then let the panel settle. It should report *"Fully accessible outside your network"*.
+
+Without this tick Plex waits for a NAT mapping it can never obtain and publishes no remote
+connection, so the port-forward in step 1 sits there unused.
+
+## How to verify
+
+Do these in order; each one isolates a different half.
+
+**1. The port is open from outside.** From a phone on cellular, or any off-LAN host:
+
+    curl -sk -o /dev/null -w '%{http_code}\n' https://136.51.115.252:32400/identity
+
+`200` means the forward and the policy are both good. A hang or refusal means step 1.
+
+**2. Plex itself believes it is reachable.** Settings → Remote Access must say *"Fully accessible
+outside your network"*. If it says *"not available outside your network"* while test 1 passes, step 2
+was not applied.
+
+**3. plex.tv is handing out a remote connection.** The decisive test, because it is what clients
+actually consume: sign in to [app.plex.tv](https://app.plex.tv) from an off-LAN network and confirm
+the server appears and plays. A client finds this server through plex.tv's connection list, never
+through DNS.
+
+**4. The policy is admitting the traffic.** If test 1 fails *after* the forward is confirmed, the
+CiliumNetworkPolicy is the next suspect:
+
+    ./infra/devcontainer/dev.sh run bash -c \
+      'kubectl --context k3s-prod -n plex get cnp plex-allow-remote-32400'
+
+See `runbooks/denied-flows.md` for reading actual drops.
+
+## Why no alert
+
+Every in-cluster signal is green whether or not remote access works. The Service has endpoints, the
+pod is `Running`, the policy is applied, and `plex-down` watches
+`kube_deployment_status_replicas_available` — none of which can see a router with a missing
+port-forward or an unticked checkbox. An alert would need an off-LAN prober, which this cluster does
+not have. **Test 3 above is the only honest check, and it is manual.**
+
+## Why not the Gateway URL (`ADVERTISE_IP`)
+
+The alternative was to leave Plex on `ClusterIP` and set
+`ADVERTISE_IP: https://plex.home.keithjackson.dev:443`, which writes Plex's `customConnections`
+preference and would have put the server on plex.tv's list via the existing L7 Gateway — no new WAN
+hole. That mechanism was verified to work on an already-claimed server (the `ADVERTISE_IP` block at
+line 126 of the image's `40-plex-first-run` sits **outside** the claim guard, and `/.firstRunComplete`
+lives on the overlay FS, not the `/config` PVC, so it re-runs every pod start), and the Gateway path
+itself was measured serving Plex correctly: `HTTP 200`, `claimed="1"`, with `401` on every protected
+endpoint.
+
+**The operator chose the native path anyway (2026-10-03), and the reasons are real:** Plex terminates
+its own TLS with its `*.plex.direct` certificate, and video streams bypass the Envoy L7 hop instead
+of being proxied through it. The cost is a new unfiltered WAN port — acceptable because Plex
+authenticates every caller itself. `ADVERTISE_IP` therefore stays unset in `catalog/plex.yaml`, and
+the deferral note there is still accurate.
+
+## Known wart: remote clients are classified as LAN
+
+`apps/wave-0/plex-remote.yaml` uses `externalTrafficPolicy: Cluster`, which SNATs the client to the
+ingress node's address (`192.168.2.1x`). That falls inside the `192.168.0.0/16` entry of
+`ALLOWED_NETWORKS`, so **Plex treats every remote stream as local.** Measured on the Gateway path,
+which SNATs the same way: Plex logged `[10.0.2.214:36040 (Allowed Network (WAN))]`.
+
+This is a **metering** effect, not a security one — authorization is unchanged, as the 401s above
+show — but Plex's remote-quality and bandwidth limits will not bind on these streams.
+
+`Local` would preserve the client IP and fix it. **Do not switch to it.** Cilium's L2 announcer
+elects one lease holder per Service and does *not* follow the backend pod, so the VIP can be
+announced by a node with no Plex pod, black-holing the port while everything reports healthy. That
+exact change cost this LAN an 8-day Minecraft outage in September 2026 — the full write-up is at the
+top of `platform/crafty/service-l4.yaml`. Plex is a single-replica `Recreate` Deployment, so the
+node mismatch is the normal case here, not a rare one.