fix(cilium): point cgroup.hostRoot at the host's real cgroup2 root

hostRoot was /run/cilium/cgroupv2 with autoMount disabled, but nothing mounts cgroup2 there. The agent mounted one from inside its own private cgroup namespace, so socket-LB attached to the agent's own cgroup and host processes got no Service/LB translation. k3s-prod-1 (Gateway L2 lease holder + kube-vip VIP) therefore could not reach the Gateway VIP, and its apiserver's OIDC authenticator has failed since 2026-09-12. Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 18:07:18 -05:00
11 changed files with 110 additions and 466 deletions
@@ -18,7 +18,6 @@ resources:
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
 - plex-claim-secret.sops.yaml
 - plex-remote.yaml
 - plex.yaml
 - qbittorrent-bootstrap-secret.sops.yaml
 - qbittorrent-oauth2-proxy-credentials.sops.yaml
@@ -1,126 +0,0 @@
 ---
 # =================================================================================================
 # PLEX REMOTE ACCESS — raw tcp/32400 north-south exposure, and the policy that admits it.
 # =================================================================================================
 # HAND-WRITTEN, NOT RENDERED. A non-catalog app-tier unit anchored to the `plex` ServiceDefinition
 # and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS table — the same
 # mechanism gitea-actions and pihole-dns use. It is hand-written for the SAME reason those are:
 # schemas/servicedefinition.schema.json is `additionalProperties: false` and has NO field for a
 # type=LoadBalancer Service, for a pinned lbipam VIP, or for raw-TCP north-south exposure at all.
 # `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator.
 #
 # WHY THE L7 GATEWAY IS NOT ENOUGH HERE — and it is NOT because the Gateway fails to serve Plex.
 # Measured 2026-10-03 through the Gateway VIP: `GET https://plex.home.keithjackson.dev/identity`
 # returned HTTP 200 with `claimed="1"` and the real machineIdentifier, and the protected endpoints
 # (/library/sections, /status/sessions, /:/prefs) all returned 401. The L7 path WORKS.
 #
 # The problem is DISCOVERY, not proxying. Plex apps (phone, TV, Roku) do not resolve
 # `plex.${BASE_DOMAIN}`; they ask plex.tv for the server's connection list. A server gets onto that
 # list one of two ways: it advertises a custom URL (`customConnections`, the ADVERTISE_IP env), or it
 # publishes a reachable public IP:port for its own tcp/32400. Measured in the live Preferences.xml on
 # 2026-10-03: `customConnections` ABSENT, `ManualPortMappingMode` ABSENT, while
 # `PublishServerOnPlexOnlineKey="1"` — the server was trying to publish and had nothing reachable to
 # publish, so remote clients saw NO connection at all. The operator chose the NATIVE path (this
 # file) over advertising the Gateway URL: Plex then terminates its own TLS with its
 # `*.plex.direct` certificate and video streams bypass the Envoy L7 hop entirely.
 #
 # VIP: ${PLEX_L4_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.57, prod 192.168.2.24), PINNED via
 # lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
 # (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the existing
 # CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config is needed;
 # this Service just consumes the pool. PINNING IS NOT OPTIONAL: the address is typed into a WAN
 # port-forward on the SOHO router, which is hand-maintained and off-repo, so it can never move.
 # 192.168.2.24 verified FREE live 2026-10-03 — `default-pool` reported IPsTotal 11 / IPsUsed 4 /
 # IPsAvailable 7, and the only four allocations were .20 (cilium-gateway-shared), .21 (unifi-l4-*),
 # .22 (crafty-l4-*) and .23 (pihole-dns-*).
 #
 # ONE SERVICE, TCP ONLY. Unlike unifi/crafty/pihole-dns this needs no second Service and no
 # lbipam.cilium.io/sharing-key: Plex's remote access is tcp/32400 and nothing else. (Plex's
 # DLNA/GDM discovery uses UDP, but that is LAN-only multicast and is not part of remote access.)
 #
 # NO `loadBalancerSourceRanges`, DELIBERATELY — and this is the one place this file diverges from
 # its closest sibling. apps/wave-0/pihole-dns.yaml pins its source range to ${LAN_CIDR} because a
 # resolver must never answer the internet. This Service is the OPPOSITE case: admitting remote
 # clients from arbitrary internet addresses is its entire purpose, so there is no range to narrow
 # to. Plex authenticates every caller itself (measured above: 401 on every protected endpoint
 # without a token), which is what makes an unfiltered port defensible here.
 #
 # NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
 # `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their VIP as a public
 # Cloudflare record. This Service must not, for pihole-dns's reason: ${PLEX_L4_VIP} is an RFC1918
 # address that is useless in public DNS. Remote clients get this server's address from plex.tv, not
 # from the zone. The existing public `plex.${BASE_DOMAIN}` record is published from the rendered
 # HTTPRoute via the gateway source and is NOT affected by this file — the web UI keeps working
 # through the Gateway exactly as before.
 #
 # -------------------------------------------------------------------------------------------------
 # externalTrafficPolicy: Cluster — inherited from crafty's 8-DAY LAN OUTAGE, not re-derived.
 # -------------------------------------------------------------------------------------------------
 # `Local` is the tempting choice because it preserves the client source IP, which this service
 # actually has a use for (see the allowedNetworks note below). It is still WRONG here, for the
 # reason platform/crafty/service-l4.yaml documents at length: Cilium's L2 announcer elects ONE lease
 # holder per Service and does NOT follow the backend pod, so with `Local` the VIP can be announced
 # by a node that has no Plex pod, which black-holes the port while every object still reports
 # healthy. Plex is a single-replica Deployment with `strategy: Recreate` pinned to wherever the
 # scheduler put it, so the mismatch is the normal case, not a rare one. Take the SNAT.
 #
 # KNOWN CONSEQUENCE OF THAT CHOICE, stated rather than discovered later: `Cluster` SNATs the client
 # to the ingress NODE's address (192.168.2.1x), which falls inside the `192.168.0.0/16` entry of
 # catalog/plex.yaml's ALLOWED_NETWORKS. Plex will therefore classify every REMOTE client as LAN.
 # Measured 2026-10-03 on the Gateway path, which SNATs the same way: Plex logged the request as
 # `[10.0.2.214:36040 (Allowed Network (WAN))]`. This is a METERING effect, not a security one —
 # authorization is unchanged (401s above), but Plex's remote-quality/bandwidth limits will not bind
 # on these streams because it believes they are local. Narrowing ALLOWED_NETWORKS cannot fix it
 # without also de-classifying genuine LAN clients, so it is accepted.
 apiVersion: v1
 kind: Service
 metadata:
  name: plex-l4-tcp
  namespace: plex
  labels:
    app.kubernetes.io/name: plex
  annotations:
    lbipam.cilium.io/ips: "${PLEX_L4_VIP}"
 spec:
  type: LoadBalancer
  externalTrafficPolicy: Cluster
  selector:
    app.kubernetes.io/name: plex
  ports:
    - name: plex-remote
      port: 32400
      targetPort: 32400
      protocol: TCP
 ---
 # INGRESS allow for the raw port (R-NET-01/02).
 #
 # The rendered plex.yaml already carries `plex-allow-gateway-ingress` (fromEntities: ingress ->
 # 32400) and three reciprocal allows for sonarr/radarr/seerr. CiliumNetworkPolicies are ADDITIVE, so
 # this file adds the off-cluster case without touching those.
 #
 # `world` is Cilium's entity for anything outside the cluster, which is what a remote Plex client is.
 # `host` and `cluster` mirror platform/crafty/networkpolicy.yaml's game-port rule and are what make
 # this work under `externalTrafficPolicy: Cluster`: after SNAT the packet's source is the ingress
 # NODE, which Cilium classifies as `host` (same node) or `remote-node` (a different one) — NOT
 # `world`. Listing `world` alone would compile cleanly and then drop every remote stream, so all
 # four entities are named.
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
 metadata:
  name: plex-allow-remote-32400
  namespace: plex
  labels:
    app.kubernetes.io/name: plex
 spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: plex
  ingress:
    - fromEntities:
        - world
        - cluster
        - host
        - remote-node
      toPorts:
        - ports:
            - { port: "32400", protocol: TCP }
@@ -2002,24 +2002,6 @@ EXTRA_UNITS: dict = {
            "pihole-dns.yaml",
        ],
    },
    # Plex's remote-access exposure: the raw tcp/32400 LoadBalancer Service on a pinned VIP, plus the
    # CiliumNetworkPolicy that admits off-cluster clients to it (apps/wave-0/plex-remote.yaml).
    # `above: False` co-locates it in plex's OWN wave — these are the plex pod's own Service and its
    # own policy, applied in the same wave as the Deployment they select, exactly as the rendered
    # Service and CNPs in plex.yaml are. Nothing in a higher wave gates on it.
    #
    # It is a non-catalog unit for the same reason pihole-dns is: the ServiceDefinition schema cannot
    # express a LoadBalancer Service or a pinned lbipam VIP (additionalProperties is false, and
    # `ingressRoutes` only renders HTTPRoutes on the L7 Gateway, which cannot carry Plex's raw port).
    # Plex's own settings stay in the catalog as env on catalog/plex.yaml; the only thing living
    # outside it is the exposure the schema has no word for.
    "plex-remote": {
        "anchor": "plex",
        "above": False,
        "files": [
            "plex-remote.yaml",
        ],
    },
 }
@@ -100,8 +100,7 @@ data:
  # platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
  # tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
  # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
-  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers;
+  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers),
  # plex-remote: PLEX_L4_VIP),
  # velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
  # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
  # UNSUBSTITUTED path would ship the literal string into the cluster.
@@ -176,19 +175,6 @@ data:
  # dev  -> 192.168.1.56  (base default; next free after crafty .55 in dev's .53-.60 pool)
  # prod -> 192.168.2.23  (next free after crafty .22 in prod's .20-.30 pool; see clusters/prod)
  PIHOLE_DNS_VIP: "192.168.1.56"             # PiHole: authoritative LAN DNS for ${BASE_DOMAIN}
  # Plex's remote-access listener: raw tcp/32400 for REMOTE Plex clients (apps/wave-0/plex-remote.yaml).
  # A FOURTH pinned L4 VIP, for the usual reason — the `shared` Gateway is an L7 HTTPS terminator and
  # cannot carry Plex's own TLS. Plex clients do not resolve `plex.${BASE_DOMAIN}`; they fetch this
  # server's address from plex.tv, so the Gateway's L7 path (which still serves the web UI, and was
  # measured working) cannot put the server on a client's connection list at all.
  #
  # PINNED, AND THE PIN IS LOAD-BEARING OFF-REPO: the SOHO router forwards WAN tcp/32400 to this
  # address by hand. A VIP taken from the pool head would silently move on a reschedule and break
  # remote access with nothing in-cluster looking wrong. See runbooks/plex-remote-access.md.
  #
  # dev  -> 192.168.1.57  (base default; next free after pihole-dns .56 in dev's .53-.60 pool)
  # prod -> 192.168.2.24  (next free after pihole-dns .23 in prod's .20-.30 pool; see clusters/prod)
  PLEX_L4_VIP: "192.168.1.57"                # Plex: tcp/32400 remote access (NOT the web UI, which is L7)
  # The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
  # peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable
@@ -476,26 +476,6 @@ patches:
      - op: replace
        path: /data/PIHOLE_DNS_VIP
        value: "192.168.2.23"
      # Plex's remote-access listener VIP — raw tcp/32400 for REMOTE Plex clients
      # (apps/wave-0/plex-remote.yaml). 192.168.2.24 is the NEXT FREE address in the prod pool
      # 192.168.2.20-.30, verified live 2026-10-03: the CiliumLoadBalancerIPPool `default-pool`
      # reported IPsTotal 11 / IPsUsed 4 / IPsAvailable 7, and the only four allocations are .20
      # (cilium-gateway-shared), .21 (unifi-l4-tcp/-udp), .22 (crafty-l4-tcp/-udp) and .23
      # (pihole-dns-tcp/-udp). It is inside the pool block the operator already asserted sits outside
      # the SOHO router's DHCP scope, and it is not a node address (.10/.11/.12) or the
      # control-plane VIP (.9).
      #
      # UNLIKE .23, THIS ONE IS ALSO A WAN PORT-FORWARD TARGET. The router forwards WAN tcp/32400
      # here, by hand, off-repo — so changing this value is a two-place edit and the router is the
      # half that CI cannot check. runbooks/plex-remote-access.md is the procedure.
      #
      # NOT ARP-PROBED. .23 was verified free with `arping` from a LAN host before it was claimed;
      # for .24 only the in-cluster pool accounting above was checked, so an off-cluster squatter on
      # this address would not have been caught. Probe before trusting it if remote access comes up
      # with a conflict.
      - op: replace
        path: /data/PLEX_L4_VIP
        value: "192.168.2.24"
      # The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns —
      # NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path.
      # It scopes the ingress allow for PiHole's DNS listener to LAN clients only.
@@ -0,0 +1,76 @@
 ---
 date: 2026-10-03
 branch: fix/cilium-cgroup-hostroot
 title: Cilium socket-LB never covered host processes — cgroup.hostRoot pointed at an unmounted path
 ---
 ## What changed
 `cgroup.hostRoot` changes from `/run/cilium/cgroupv2` to `/sys/fs/cgroup` in all three parity
 copies: `platform/cilium/helmrelease.yaml` (authoritative), `platform/cilium/values.yaml`
 (mirror) and `infra/ansible/roles/cilium/templates/cilium-values.yaml.j2` (bootstrap).
 `autoMount` stays `false`.
 **The defect.** With `autoMount: false`, the Cilium chart only bind-mounts whatever exists at
 `hostRoot`, and expects the operator to have mounted cgroup2 there (cilium 1.20.1 values.yaml).
 The repo's rationale was "k3s provides the cgroupv2 mount". That is true only for
 `/sys/fs/cgroup`. Nothing mounts cgroup2 at `/run/cilium/cgroupv2` on these nodes; it is an
 empty `DirectoryOrCreate` hostPath. So the agent mounted cgroup2 there itself, from inside its
 own private cgroup namespace, and that mount's root is the agent container's own leaf cgroup.
 The socket-LB programs (`cil_sock4_connect` etc.) were attached to it and covered only the
 agent.
 **The consequence.** Every other host-netns process got no Service/LB translation:
 - Service ClusterIPs were unreachable from every node's host.
 - The node holding the Gateway's L2 lease could not reach the Gateway VIP (`no route to host`).
  Other nodes reached it only because their packets crossed the wire to the lease holder, where
  the tc datapath handles them.
 - k3s-prod-1 holds both the Gateway lease and the kube-vip API VIP. Its apiserver could not reach
  the OIDC issuer, logging `oidc: authenticator not initialized` since at least 2026-09-12
  (899 occurrences by 2026-10-03). kubectl SSO through the VIP therefore failed, and in-cluster
  OIDC clients failed on whichever calls landed on that apiserver.
 - Webhooks and aggregated APIs were unaffected only because k3s runs
  `--enable-aggregator-routing=true` (endpoint IPs, not ClusterIPs).
 Cilium's own source (v1.20.1 `bpf/bpf_sock.c`, `__sock4_xlate_fwd`) handles exactly this case
 when the hook does run: a host-netns connect to an L7-LB Service (the Gateway) is rewritten to
 `127.0.0.1:<l7_lb_proxy_port>`, the local Envoy listener. The fix is to make the hook cover the
 host.
 ## What I verified
 - `cilium-dbg status --verbose` on k3s-prod-1: `Socket LB: Enabled`, `Coverage: Hostns-only`.
  The live cilium-config shows `bpf-lb-sock=false`, but socket LB is implied by kube-proxy
  replacement. The Gateway Service frontends `192.168.2.20:{80,443}` and
  `10.43.225.79:{80,443}` map to `127.0.0.1:14337`.
 - Host: `findmnt /run/cilium/cgroupv2` is empty on all three nodes, and `/sys/fs/cgroup` is
  `cgroup2`.
 - In the agent: `/run/cilium/cgroupv2` is a `cgroup2 none` mount, the agent's own cgroup reads
  `0::/`, the mount has 0 child cgroups, and `bpftool cgroup show /run/cilium/cgroupv2` lists all
  `cil_sock*` programs. k3s runs in `0::/system.slice/k3s.service`, and the agent container in
  `/kubepods.slice/…/cri-containerd-3ad04cde….scope`.
 - **The decisive test**, run on k3s-prod-1 in the same netns: raw TCP connect to
  `192.168.2.20:443`, `10.43.225.79:443` and `10.43.59.141:8080` CONNECTED from inside the agent
  container (covered cgroup) and FAILED from the host (`No route to host` / timeout).
 - Blast radius: host-network pods are cilium, cilium-envoy, cilium-operator (API via
  `192.168.2.9:6443`, not a Service), kube-vip (`127.0.0.1:6443`, `svc_enable=false`) and
  node-exporter (no outbound). Nothing depends on host Service translation being absent; the
  change only makes Service/LB addresses reachable from the host where they currently fail.
 - `helm template cilium/cilium --version 1.20.1` with the new values, in the toolbox: the
  DaemonSet bind-mounts hostPath `/sys/fs/cgroup` into `cilium-agent` and `clean-cilium-state`,
  the cilium-config shows `cgroup-root: "/sys/fs/cgroup"`, and the init containers are unchanged
  (no `mount-cgroup`).
 - Toolbox gate: see the PR description.
 ## What I did NOT verify
 - **The rollout itself.** After merge, the agent DaemonSet rolls one node at a time
  (`maxUnavailable: 1`). On each node, afterwards: host TCP connects to the three targets above
  must succeed. On k3s-prod-1, the apiserver's background OIDC init retry must stop logging
  `authenticator not initialized` with no k3s restart.
 - That the old programs, attached to the previous agent container's cgroup, disappear with it.
  Expected, because cgroup BPF attachments die with the cgroup, but not observed.
 - An end-to-end kubectl OIDC login (needs a browser).
 - No regression test was added. A natural one is an L2 conformance check that a host-netns
  connect to the Gateway VIP succeeds on the lease holder, but `conformance/**` is a sensitive
  path. Left as a follow-up.
@@ -1,129 +0,0 @@
 ---
 date: 2026-10-03
 branch: feat/plex-remote-access-l4
 title: Plex remote access — raw tcp/32400 via a pinned LoadBalancer VIP, not the L7 Gateway
 ---
 ## What changed
 Reported symptom: users cannot reach Plex from outside the LAN. **Two independent causes**, measured
 rather than inferred:
 1. **Plex advertised no remote connection at all.** Live `Preferences.xml` on prod:
   `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while
   `PublishServerOnPlexOnlineKey="1"` — publishing was on and there was nothing reachable to publish.
   Plex's automatic UPnP/NAT-PMP path cannot work from a pod on the Cilium overlay.
 2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` refused;
   `136.51.115.252:443` answered with a **self-signed router cert** (`C=ZZ, CN=%D`), not the
   Gateway's LE wildcard. The Plex Service was `ClusterIP` only — no off-cluster path to 32400 by
   any route.
 This branch fixes the in-cluster half. **Remote access does not work until the operator also does
 two manual steps** (`runbooks/plex-remote-access.md`): the WAN tcp/32400 port-forward, and ticking
 "manually specify public port" in the Plex UI.
 - **`apps/wave-0/plex-remote.yaml` (new, hand-written).** A `type=LoadBalancer` Service on tcp/32400
  pinned to `${PLEX_L4_VIP}` via `lbipam.cilium.io/ips`, plus a `CiliumNetworkPolicy`
  (`plex-allow-remote-32400`) admitting `world`/`cluster`/`host`/`remote-node` to 32400. Non-catalog
  because `schemas/servicedefinition.schema.json` is `additionalProperties: false` and has no field
  for a LoadBalancer Service or a pinned VIP — the same reason `apps/wave-0/pihole-dns.yaml`,
  `platform/crafty/service-l4.yaml` and `apps/vms/unifi/50-service-l4.yaml` are hand-written.
 - **`catalog/render.py`:** an `EXTRA_UNITS` entry (`plex-remote`, anchored to `plex`, `above: False`)
  so `--emit-waves` places the unit in plex's own wave.
 - **`clusters/base/cluster-vars.yaml`:** new `PLEX_L4_VIP` (dev default `192.168.1.57`), plus the
  substitution inventory comment updated to name `plex-remote: PLEX_L4_VIP`.
 - **`clusters/prod/kustomization.yaml`:** prod override `PLEX_L4_VIP: 192.168.2.24`.
 - **`apps/wave-0/kustomization.yaml`:** `plex-remote.yaml` added to `resources`.
 - **`runbooks/plex-remote-access.md` (new):** the two manual steps, the four-step verification, why
  no alert is possible, and the two documented warts below.
 **`ADVERTISE_IP` was deliberately NOT set.** Setting it to `https://plex.home.keithjackson.dev:443`
 was the alternative — it needs no new WAN hole and the mechanism was verified working on an
 already-claimed server. **The operator chose the native path (2026-10-03)** so Plex terminates its own
 TLS (`*.plex.direct`) and streams bypass the Envoy L7 hop. `catalog/plex.yaml`'s deferral note for
 `ADVERTISE_IP` remains accurate and untouched.
 **`externalTrafficPolicy: Cluster`, with a known cost.** `Local` would preserve the client IP, but
 Cilium's L2 announcer does not follow the backend pod — the wart that cost this LAN an 8-day
 Minecraft outage (see `platform/crafty/service-l4.yaml`). Consequence accepted and documented: SNAT
 puts the client in `192.168.0.0/16`, inside `ALLOWED_NETWORKS`, so **Plex classifies remote streams
 as LAN**. That is a metering effect (remote bandwidth limits will not bind), not an authorization one.
 **SENSITIVE PATHS — this PR touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK
 it.** It needs the operator's `/approve-override <reason>` posted as a PR **review** of type
 "Comment". I have deliberately not posted it: CLAUDE.md §1 forbids an agent clearing this on its own
 PR, and the agent account holds `owner`, so the prohibition is a rule, not a limit.
 ## What I verified
 All measurements against live prod, 2026-10-03, read-only.
 - **Plex's advertised state**, the finding this branch rests on: `customConnections=<ABSENT>`,
  `ManualPortMappingMode=<ABSENT>`, `ManualPortMappingPort=<ABSENT>`,
  `PublishServerOnPlexOnlineKey="1"`, `allowedNetworks="10.0.0.0/8,192.168.0.0/16"`. Read by
  attribute grep, never by dumping `Preferences.xml` (it holds `PlexOnlineToken`).
 - **The L7 Gateway path works and is NOT the problem.** Via the Gateway VIP,
  `GET https://plex.home.keithjackson.dev/identity` → `HTTP 200`, `claimed="1"`, real
  `machineIdentifier`. HTTPRoute `plex-32400`: `Accepted=True`, `ResolvedRefs=True`.
 - **Plex enforces its own auth**, which is what makes an unfiltered WAN port defensible:
  `/library/sections`, `/status/sessions`, `/:/prefs` all → **401** unauthenticated through the
  Gateway.
 - **The SNAT/allowedNetworks wart is real, not theoretical.** A tagged probe through the Gateway was
  logged by Plex as `[10.0.2.214:36040 (Allowed Network (WAN))]` — the Envoy pod IP, classified as an
  allowed network.
 - **WAN state:** `136.51.115.252:32400` → connection refused. `136.51.115.252:443` → self-signed
  cert `C=ZZ, ST=Somewhere, O=%D342e7e6b, CN=%D`. Public DNS resolves correctly:
  `plex.home.keithjackson.dev` → CNAME `wan.home.keithjackson.dev` → `136.51.115.252`.
 - **`192.168.2.24` is free in-cluster.** `CiliumLoadBalancerIPPool default-pool`: IPsTotal 11 /
  IPsUsed 4 / IPsAvailable 7; the only allocations are `.20` (cilium-gateway-shared), `.21`
  (unifi-l4-*), `.22` (crafty-l4-*), `.23` (pihole-dns-*). Not a node address (`.10/.11/.12`) and not
  the kube-vip control-plane VIP (`.9`).
 - **The manual-port-mapping step genuinely cannot be automated.** Grep of the running image's whole
  init surface (`/etc/cont-init.d`, `/plex-common.sh`, `/usr/local/bin`) for `ManualPortMapping` and
  `PLEX_PREFERENCE` → **NONE**.
 - **`ADVERTISE_IP` would have worked on a claimed server** (recorded because the runbook's
  "why not" rests on it): `setPref "customConnections"` at line 126 of `40-plex-first-run` sits
  **outside** the `PLEX_CLAIM`/`PlexOnlineToken` guard, and the `/.firstRunComplete` early-exit
  marker is on the **overlay** FS, not the `/config` PVC, so the script re-runs every pod start.
 - **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
  `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
  **pass**. The rc 3 is two environmental skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`)
  and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3, not as pass.**
 - **`render-golden` passing is the specific proof** that the `render.py` `EXTRA_UNITS` change, the new
  hand-written unit and the edited wave kustomization are mutually consistent.
 ## What I did NOT verify
 - **THE ACTUAL FIX. Nothing in this branch has been proven to make remote access work** — it cannot
  be, because both remaining steps are off-repo and manual (router forward, Plex UI toggle). This
  branch removes two blockers and adds none; it does not demonstrate an outcome.
 - **Outside-in reachability, at all.** Every probe above was taken from inside the LAN, so the
  `:443` self-signed cert strictly proves only that there is **no NAT hairpin** — it does **not**
  prove outside-in failure. A router cert on WAN:443 is suggestive (its management UI would shadow a
  forward), but unproven. **This repo has no off-LAN vantage point.** Test 3 in the runbook
  (app.plex.tv from a cellular connection) is the only honest check.
 - **`192.168.2.24` was never ARP-probed off-cluster.** Only in-cluster pool accounting was checked.
  `.23` was `arping`-probed before it was claimed; `.24` was not, so an off-cluster squatter would
  not have been caught.
 - **The LoadBalancer has never been reconciled.** `plex-l4-tcp` does not exist in prod yet, so
  LB-IPAM has not actually assigned the pinned VIP, L2 announcement has not been observed for it, and
  the new CiliumNetworkPolicy has never admitted a real packet. Whether `world`/`host`/`cluster`/
  `remote-node` is the correct entity set **under this SNAT path** is reasoned from
  `platform/crafty/networkpolicy.yaml`'s working game-port rule, **not measured**.
 - **Whether Plex's remote path needs anything beyond the manual port tick.** If plex.tv still lists
  no remote connection after both manual steps, this branch is necessary-but-insufficient.
 - **The CI approval-policy outcome.** The local stage reported green, but that is an offline N/A, not
  a verdict: this PR touches `clusters/**`, so CI will block it pending an override.
 - **PRE-EXISTING DRIFT I FOUND AND DELIBERATELY LEFT ALONE — worth its own follow-up.** Running the
  documented `python3 catalog/render.py --emit-waves` regenerates the wave kustomizations and wants to
  add **`vm-demo.yaml`** to wave 0 and **`keycloak.yaml`** to wave 1. Both are listed as
  intentionally not onboarded in `ci/render-golden.sh`
  (`RENDER_GOLDEN_NOT_ONBOARDED="keycloak vm-demo"`); their manifests are tracked in `apps/` but
  deliberately absent from the `resources` lists so Flux does not apply them. `--emit-waves` places
  any file present on disk and has **no awareness of that exclusion**, while `render-golden.sh` uses
  `--wave-layout` and compares rendered manifests only — it never checks kustomization `resources`,
  which is why this has survived unnoticed. **So the header "DO NOT hand-edit: re-run
  `render.py --emit-waves`" is currently unsafe advice**: following it onboards an IdP and a KubeVirt
  VM. I therefore restored both kustomizations and hand-added only my one line. I did not fix
  `--emit-waves`, and I have not determined how long the drift has existed or whether either service
  is *meant* to be onboarded.
@@ -51,11 +51,14 @@ hubble:
 l2announcements:
  enabled: true
-# k3s ships its own cgroup mount; point Cilium at it instead of auto-mounting.
+# Point Cilium at the host's real cgroup2 root (/sys/fs/cgroup, mounted by systemd) instead of
 # auto-mounting. NOT /run/cilium/cgroupv2: nothing mounts cgroup2 there, so the agent mounted its
 # OWN namespaced cgroup and socket-LB covered only itself — host processes (the apiserver) lost
 # Service/LB translation. [parity] platform/cilium/helmrelease.yaml explains (2026-10-03).
 cgroup:
  autoMount:
    enabled: false
-  hostRoot: /run/cilium/cgroupv2
+  hostRoot: /sys/fs/cgroup
 # Roll the cilium agent DaemonSet ONE node at a time. [parity] must match the
 # platform/cilium HelmRelease. The cilium 1.20.1 chart defaults this to 2, which on a
@@ -106,14 +106,33 @@ spec:
    # CiliumL2AnnouncementPolicy (reserved 10.73.0.240-.250) authored by WS-NET in Phase 2.
    l2announcements:
      enabled: true
-    # --- cgroup: k3s ships its own cgroup mount; point Cilium at it, don't auto-mount ---
+    # --- cgroup: point Cilium at the HOST's cgroup2 root, which systemd mounts at /sys/fs/cgroup ---
-    # [parity] MUST equal Ansible cgroup block. Not in the deliverable's headline list, but it IS
+    # [parity] MUST equal Ansible cgroup block (roles/cilium/templates/cilium-values.yaml.j2).
-    # part of the working CNI config on k3s: reconciling without it lets Cilium try to auto-mount
+    #
-    # cgroupv2 and break on adopt. Kept in parity for a safe Phase-2 adoption.
+    # FIXED 2026-10-03 — hostRoot WAS /run/cilium/cgroupv2, AND THAT SILENTLY BROKE SOCKET-LB FOR
    # EVERY HOST PROCESS. With autoMount disabled, the chart only bind-mounts whatever already exists
    # at hostRoot ("it's expected that users have mounted cgroup2 filesystem at the specified
    # cgroup.hostRoot" — cilium 1.20.1 values.yaml). Nothing ever mounted cgroup2 at
    # /run/cilium/cgroupv2 on these nodes (it is an empty DirectoryOrCreate hostPath), so the agent
    # mounted one ITSELF from inside its own private cgroup namespace. That mount's root is the
    # agent container's OWN leaf cgroup, so cil_sock4_connect & co. were attached there and covered
    # only the agent. Every other host-netns process (the k3s apiserver in
    # /system.slice/k3s.service, curl over SSH) was untranslated: Service ClusterIPs timed out from
    # every node, and the node holding the Gateway's L2 lease could not reach the Gateway VIP at all
    # ("no route to host"). That is why k3s-prod-1's apiserver logged `oidc: authenticator not
    # initialized` from 2026-09-12 on — kubectl SSO through the kube-vip VIP failed outright.
    # Measured on k3s-prod-1 before this change: TCP connect to 192.168.2.20:443, 10.43.225.79:443
    # and 10.43.59.141:8080 SUCCEEDED from inside the agent container (the covered cgroup) and
    # FAILED from the host, same node, same netns. Full write-up:
    # decisions/2026-10-03-cilium-cgroup-hostroot.md.
    #
    # /sys/fs/cgroup IS a cgroup2 mount in the host's root cgroup namespace on every node (verified
    # with findmnt on all three), so bind-mounting it gives the agent the real host root. autoMount
    # stays off because the mount it would create already exists.
    cgroup:
      autoMount:
        enabled: false
-      hostRoot: /run/cilium/cgroupv2
+      hostRoot: /sys/fs/cgroup
    # --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident ---
    # [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1.
    # The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate.
@@ -41,11 +41,13 @@ hubble:
 # [parity] ingress VIP announcement on the flat LAN (docs/09, OQ-1)
 l2announcements:
  enabled: true
-# [parity] k3s provides the cgroupv2 mount; do not auto-mount
+# [parity] the host's cgroup2 root is /sys/fs/cgroup (systemd mounts it); do not auto-mount.
 # NOT /run/cilium/cgroupv2: nothing mounts cgroup2 there, so socket-LB attached to the agent's own
 # cgroup and host processes lost Service/LB translation (2026-10-03 — see helmrelease.yaml).
 cgroup:
  autoMount:
    enabled: false
-  hostRoot: /run/cilium/cgroupv2
+  hostRoot: /sys/fs/cgroup
 # [parity] roll the agent DaemonSet ONE node at a time. Overrides the cilium 1.20.1 chart
 # default of 2, which on 3 nodes lets a single bad render break 2/3 agents at once — as the
 # 2026-09-01 incident did (k8sServiceHost: "" → the `config` init container fell back to the
@@ -1,148 +0,0 @@
 # plex-remote-access — getting Plex reachable from outside the LAN, and proving it
 **Status: the in-cluster half is shipped; the last mile is TWO MANUAL STEPS and remote access does
 NOT work until both are done.** Neither can live in this repo.
 **Alert:** none, and none is possible from in-cluster signals — see "Why no alert" below. This is an
 **operator procedure**, not an incident page.
 **Files:** `apps/wave-0/plex-remote.yaml` (the tcp/32400 LoadBalancer + its CiliumNetworkPolicy),
 `clusters/base/cluster-vars.yaml` + `clusters/prod/kustomization.yaml` (`PLEX_L4_VIP`),
 `catalog/render.py` (the `EXTRA_UNITS` entry that places the unit), `catalog/plex.yaml` (Plex's own
 env — note `ADVERTISE_IP` is deliberately NOT set; see "Why not the Gateway URL").
 **Related:** `runbooks/plex-quicksync.md` and `runbooks/plex-claim.md` — the other two Plex
 procedures with a manual last mile.
 ## What was actually broken (measured 2026-10-03)
 Plex was serving correctly and was reachable on the LAN. Two independent things stopped remote
 clients:
 1. **Plex advertised no remote connection at all.** In the live `Preferences.xml`:
   `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while
   `PublishServerOnPlexOnlineKey="1"`. The server was trying to publish itself to plex.tv and had
   nothing reachable to publish. Plex's automatic path (UPnP/NAT-PMP) cannot work here: the process
   lives in a pod on the Cilium overlay (`10.0.2.136`), it cannot speak UPnP to the SOHO router, and
   a mapping to the pod IP would be meaningless if it could.
 2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` → connection refused.
   `136.51.115.252:443` → a **self-signed router certificate** (`C=ZZ, CN=%D`), not the Gateway's
   Let's Encrypt wildcard.
 The Service was `ClusterIP` only, so port 32400 had no path off-cluster by any route.
 **Caveat on that second measurement, because it is easy to over-read:** it was taken from *inside*
 the LAN, so strictly it proves there is no NAT hairpin (reflection). It does not prove outside-in
 failure. But a router certificate answering 443 means the router's own management UI is bound to the
 WAN port, and that *would* shadow a port-forward for genuine outside traffic too. **Verify from a
 real outside vantage point** (phone on cellular, or an external port checker) before concluding
 either way — this repo has no such vantage point and must not pretend otherwise.
 ## Step 1 — forward WAN tcp/32400 on the router (operator, off-repo)
    WAN tcp/32400  ->  192.168.2.24  (${PLEX_L4_VIP})  port 32400
 `192.168.2.24` is a pinned Cilium LB VIP, ARP-announced on the flat LAN by the existing
 `CiliumL2AnnouncementPolicy default-l2`. It is **pinned on purpose**: this address is typed into the
 router by hand, so it must never move on a reschedule.
 While in the router, **check what else is bound to the WAN interface.** If remote management is
 listening on 443 (see the certificate measurement above), move it or disable it — otherwise the
 Gateway's 80/443 forward is shadowed and the Plex *web UI* over the internet stays broken even
 though remote *streaming* works. These are two separate paths and they fail independently.
 Confirm the VIP is actually answering on the LAN first, so a failure later is unambiguous:
    ./infra/devcontainer/dev.sh run bash -c \
      'kubectl --context k3s-prod -n plex get svc plex-l4-tcp -o wide'
 `EXTERNAL-IP` must read `192.168.2.24`. If it is `<pending>`, LB-IPAM could not assign — the address
 is outside the pool or already taken; see `clusters/prod/kustomization.yaml` for the pool accounting.
 **`192.168.2.24` was never ARP-probed off-cluster.** Only the in-cluster pool accounting was checked
 (IPsTotal 11 / IPsUsed 4 / IPsAvailable 7 on 2026-10-03). If the VIP behaves oddly, probe it from a
 LAN host — `arping 192.168.2.24` — the way `.23` was probed before it was claimed.
 ## Step 2 — tick "manually specify public port" in Plex (operator, UI only)
 **There is no GitOps path to this and no environment variable for it.** Measured in the running
 image on 2026-10-03: a grep of the entire init surface (`/etc/cont-init.d`, `/plex-common.sh`,
 `/usr/local/bin`) for `ManualPortMapping` and `PLEX_PREFERENCE` returned **NONE**. The image writes
 only six preferences, and this is not one of them. It is the same class of manual last mile as the
 `HardwareAcceleratedCodecs` toggle in `runbooks/plex-quicksync.md`.
 In the Plex web UI (`https://plex.home.keithjackson.dev`, on the LAN):
 **Settings → Remote Access**
 - Ensure **Enable Remote Access** is on. (`PublishServerOnPlexOnlineKey` was already `1`.)
 - Tick **Manually specify public port** and set it to **32400**.
 - **Apply**, then let the panel settle. It should report *"Fully accessible outside your network"*.
 Without this tick Plex waits for a NAT mapping it can never obtain and publishes no remote
 connection, so the port-forward in step 1 sits there unused.
 ## How to verify
 Do these in order; each one isolates a different half.
 **1. The port is open from outside.** From a phone on cellular, or any off-LAN host:
    curl -sk -o /dev/null -w '%{http_code}\n' https://136.51.115.252:32400/identity
 `200` means the forward and the policy are both good. A hang or refusal means step 1.
 **2. Plex itself believes it is reachable.** Settings → Remote Access must say *"Fully accessible
 outside your network"*. If it says *"not available outside your network"* while test 1 passes, step 2
 was not applied.
 **3. plex.tv is handing out a remote connection.** The decisive test, because it is what clients
 actually consume: sign in to [app.plex.tv](https://app.plex.tv) from an off-LAN network and confirm
 the server appears and plays. A client finds this server through plex.tv's connection list, never
 through DNS.
 **4. The policy is admitting the traffic.** If test 1 fails *after* the forward is confirmed, the
 CiliumNetworkPolicy is the next suspect:
    ./infra/devcontainer/dev.sh run bash -c \
      'kubectl --context k3s-prod -n plex get cnp plex-allow-remote-32400'
 See `runbooks/denied-flows.md` for reading actual drops.
 ## Why no alert
 Every in-cluster signal is green whether or not remote access works. The Service has endpoints, the
 pod is `Running`, the policy is applied, and `plex-down` watches
 `kube_deployment_status_replicas_available` — none of which can see a router with a missing
 port-forward or an unticked checkbox. An alert would need an off-LAN prober, which this cluster does
 not have. **Test 3 above is the only honest check, and it is manual.**
 ## Why not the Gateway URL (`ADVERTISE_IP`)
 The alternative was to leave Plex on `ClusterIP` and set
 `ADVERTISE_IP: https://plex.home.keithjackson.dev:443`, which writes Plex's `customConnections`
 preference and would have put the server on plex.tv's list via the existing L7 Gateway — no new WAN
 hole. That mechanism was verified to work on an already-claimed server (the `ADVERTISE_IP` block at
 line 126 of the image's `40-plex-first-run` sits **outside** the claim guard, and `/.firstRunComplete`
 lives on the overlay FS, not the `/config` PVC, so it re-runs every pod start), and the Gateway path
 itself was measured serving Plex correctly: `HTTP 200`, `claimed="1"`, with `401` on every protected
 endpoint.
 **The operator chose the native path anyway (2026-10-03), and the reasons are real:** Plex terminates
 its own TLS with its `*.plex.direct` certificate, and video streams bypass the Envoy L7 hop instead
 of being proxied through it. The cost is a new unfiltered WAN port — acceptable because Plex
 authenticates every caller itself. `ADVERTISE_IP` therefore stays unset in `catalog/plex.yaml`, and
 the deferral note there is still accurate.
 ## Known wart: remote clients are classified as LAN
 `apps/wave-0/plex-remote.yaml` uses `externalTrafficPolicy: Cluster`, which SNATs the client to the
 ingress node's address (`192.168.2.1x`). That falls inside the `192.168.0.0/16` entry of
 `ALLOWED_NETWORKS`, so **Plex treats every remote stream as local.** Measured on the Gateway path,
 which SNATs the same way: Plex logged `[10.0.2.214:36040 (Allowed Network (WAN))]`.
 This is a **metering** effect, not a security one — authorization is unchanged, as the 401s above
 show — but Plex's remote-quality and bandwidth limits will not bind on these streams.
 `Local` would preserve the client IP and fix it. **Do not switch to it.** Cilium's L2 announcer
 elects one lease holder per Service and does *not* follow the backend pod, so the VIP can be
 announced by a node with no Plex pod, black-holing the port while everything reports healthy. That
 exact change cost this LAN an 8-day Minecraft outage in September 2026 — the full write-up is at the
 top of `platform/crafty/service-l4.yaml`. Plex is a single-replica `Recreate` Deployment, so the
 node mismatch is the normal case here, not a rare one.