fix(cilium): point cgroup.hostRoot at the host's real cgroup2 root

hostRoot was /run/cilium/cgroupv2 with autoMount disabled, but nothing
mounts cgroup2 there. The agent mounted one from inside its own private
cgroup namespace, so socket-LB attached to the agent's own cgroup and
host processes got no Service/LB translation. k3s-prod-1 (Gateway L2
lease holder + kube-vip VIP) therefore could not reach the Gateway VIP,
and its apiserver's OIDC authenticator has failed since 2026-09-12.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

apps/wave-0/kustomization.yaml

@@ -18,7 +18,6 @@ resources:
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
 - plex-claim-secret.sops.yaml
-- plex-remote.yaml
 - plex.yaml
 - qbittorrent-bootstrap-secret.sops.yaml
 - qbittorrent-oauth2-proxy-credentials.sops.yaml

apps/wave-0/plex-remote.yaml

@@ -1,126 +0,0 @@
----
-# =================================================================================================
-# PLEX REMOTE ACCESS — raw tcp/32400 north-south exposure, and the policy that admits it.
-# =================================================================================================
-# HAND-WRITTEN, NOT RENDERED. A non-catalog app-tier unit anchored to the `plex` ServiceDefinition
-# and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS table — the same
-# mechanism gitea-actions and pihole-dns use. It is hand-written for the SAME reason those are:
-# schemas/servicedefinition.schema.json is `additionalProperties: false` and has NO field for a
-# type=LoadBalancer Service, for a pinned lbipam VIP, or for raw-TCP north-south exposure at all.
-# `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator.
-#
-# WHY THE L7 GATEWAY IS NOT ENOUGH HERE — and it is NOT because the Gateway fails to serve Plex.
-# Measured 2026-10-03 through the Gateway VIP: `GET https://plex.home.keithjackson.dev/identity`
-# returned HTTP 200 with `claimed="1"` and the real machineIdentifier, and the protected endpoints
-# (/library/sections, /status/sessions, /:/prefs) all returned 401. The L7 path WORKS.
-#
-# The problem is DISCOVERY, not proxying. Plex apps (phone, TV, Roku) do not resolve
-# `plex.${BASE_DOMAIN}`; they ask plex.tv for the server's connection list. A server gets onto that
-# list one of two ways: it advertises a custom URL (`customConnections`, the ADVERTISE_IP env), or it
-# publishes a reachable public IP:port for its own tcp/32400. Measured in the live Preferences.xml on
-# 2026-10-03: `customConnections` ABSENT, `ManualPortMappingMode` ABSENT, while
-# `PublishServerOnPlexOnlineKey="1"` — the server was trying to publish and had nothing reachable to
-# publish, so remote clients saw NO connection at all. The operator chose the NATIVE path (this
-# file) over advertising the Gateway URL: Plex then terminates its own TLS with its
-# `*.plex.direct` certificate and video streams bypass the Envoy L7 hop entirely.
-#
-# VIP: ${PLEX_L4_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.57, prod 192.168.2.24), PINNED via
-# lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
-# (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the existing
-# CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config is needed;
-# this Service just consumes the pool. PINNING IS NOT OPTIONAL: the address is typed into a WAN
-# port-forward on the SOHO router, which is hand-maintained and off-repo, so it can never move.
-# 192.168.2.24 verified FREE live 2026-10-03 — `default-pool` reported IPsTotal 11 / IPsUsed 4 /
-# IPsAvailable 7, and the only four allocations were .20 (cilium-gateway-shared), .21 (unifi-l4-*),
-# .22 (crafty-l4-*) and .23 (pihole-dns-*).
-#
-# ONE SERVICE, TCP ONLY. Unlike unifi/crafty/pihole-dns this needs no second Service and no
-# lbipam.cilium.io/sharing-key: Plex's remote access is tcp/32400 and nothing else. (Plex's
-# DLNA/GDM discovery uses UDP, but that is LAN-only multicast and is not part of remote access.)
-#
-# NO `loadBalancerSourceRanges`, DELIBERATELY — and this is the one place this file diverges from
-# its closest sibling. apps/wave-0/pihole-dns.yaml pins its source range to ${LAN_CIDR} because a
-# resolver must never answer the internet. This Service is the OPPOSITE case: admitting remote
-# clients from arbitrary internet addresses is its entire purpose, so there is no range to narrow
-# to. Plex authenticates every caller itself (measured above: 401 on every protected endpoint
-# without a token), which is what makes an unfiltered port defensible here.
-#
-# NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
-# `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their VIP as a public
-# Cloudflare record. This Service must not, for pihole-dns's reason: ${PLEX_L4_VIP} is an RFC1918
-# address that is useless in public DNS. Remote clients get this server's address from plex.tv, not
-# from the zone. The existing public `plex.${BASE_DOMAIN}` record is published from the rendered
-# HTTPRoute via the gateway source and is NOT affected by this file — the web UI keeps working
-# through the Gateway exactly as before.
-#
-# -------------------------------------------------------------------------------------------------
-# externalTrafficPolicy: Cluster — inherited from crafty's 8-DAY LAN OUTAGE, not re-derived.
-# -------------------------------------------------------------------------------------------------
-# `Local` is the tempting choice because it preserves the client source IP, which this service
-# actually has a use for (see the allowedNetworks note below). It is still WRONG here, for the
-# reason platform/crafty/service-l4.yaml documents at length: Cilium's L2 announcer elects ONE lease
-# holder per Service and does NOT follow the backend pod, so with `Local` the VIP can be announced
-# by a node that has no Plex pod, which black-holes the port while every object still reports
-# healthy. Plex is a single-replica Deployment with `strategy: Recreate` pinned to wherever the
-# scheduler put it, so the mismatch is the normal case, not a rare one. Take the SNAT.
-#
-# KNOWN CONSEQUENCE OF THAT CHOICE, stated rather than discovered later: `Cluster` SNATs the client
-# to the ingress NODE's address (192.168.2.1x), which falls inside the `192.168.0.0/16` entry of
-# catalog/plex.yaml's ALLOWED_NETWORKS. Plex will therefore classify every REMOTE client as LAN.
-# Measured 2026-10-03 on the Gateway path, which SNATs the same way: Plex logged the request as
-# `[10.0.2.214:36040 (Allowed Network (WAN))]`. This is a METERING effect, not a security one —
-# authorization is unchanged (401s above), but Plex's remote-quality/bandwidth limits will not bind
-# on these streams because it believes they are local. Narrowing ALLOWED_NETWORKS cannot fix it
-# without also de-classifying genuine LAN clients, so it is accepted.
-apiVersion: v1
-kind: Service
-metadata:
-  name: plex-l4-tcp
-  namespace: plex
-  labels:
-    app.kubernetes.io/name: plex
-  annotations:
-    lbipam.cilium.io/ips: "${PLEX_L4_VIP}"
-spec:
-  type: LoadBalancer
-  externalTrafficPolicy: Cluster
-  selector:
-    app.kubernetes.io/name: plex
-  ports:
-    - name: plex-remote
-      port: 32400
-      targetPort: 32400
-      protocol: TCP
----
-# INGRESS allow for the raw port (R-NET-01/02).
-#
-# The rendered plex.yaml already carries `plex-allow-gateway-ingress` (fromEntities: ingress ->
-# 32400) and three reciprocal allows for sonarr/radarr/seerr. CiliumNetworkPolicies are ADDITIVE, so
-# this file adds the off-cluster case without touching those.
-#
-# `world` is Cilium's entity for anything outside the cluster, which is what a remote Plex client is.
-# `host` and `cluster` mirror platform/crafty/networkpolicy.yaml's game-port rule and are what make
-# this work under `externalTrafficPolicy: Cluster`: after SNAT the packet's source is the ingress
-# NODE, which Cilium classifies as `host` (same node) or `remote-node` (a different one) — NOT
-# `world`. Listing `world` alone would compile cleanly and then drop every remote stream, so all
-# four entities are named.
-apiVersion: cilium.io/v2
-kind: CiliumNetworkPolicy
-metadata:
-  name: plex-allow-remote-32400
-  namespace: plex
-  labels:
-    app.kubernetes.io/name: plex
-spec:
-  endpointSelector:
-    matchLabels:
-      app.kubernetes.io/name: plex
-  ingress:
-    - fromEntities:
-        - world
-        - cluster
-        - host
-        - remote-node
-      toPorts:
-        - ports:
-            - { port: "32400", protocol: TCP }

catalog/render.py

@@ -2002,24 +2002,6 @@ EXTRA_UNITS: dict = {
             "pihole-dns.yaml",
         ],
     },
-    # Plex's remote-access exposure: the raw tcp/32400 LoadBalancer Service on a pinned VIP, plus the
-    # CiliumNetworkPolicy that admits off-cluster clients to it (apps/wave-0/plex-remote.yaml).
-    # `above: False` co-locates it in plex's OWN wave — these are the plex pod's own Service and its
-    # own policy, applied in the same wave as the Deployment they select, exactly as the rendered
-    # Service and CNPs in plex.yaml are. Nothing in a higher wave gates on it.
-    #
-    # It is a non-catalog unit for the same reason pihole-dns is: the ServiceDefinition schema cannot
-    # express a LoadBalancer Service or a pinned lbipam VIP (additionalProperties is false, and
-    # `ingressRoutes` only renders HTTPRoutes on the L7 Gateway, which cannot carry Plex's raw port).
-    # Plex's own settings stay in the catalog as env on catalog/plex.yaml; the only thing living
-    # outside it is the exposure the schema has no word for.
-    "plex-remote": {
-        "anchor": "plex",
-        "above": False,
-        "files": [
-            "plex-remote.yaml",
-        ],
-    },
 }
 
 

clusters/base/cluster-vars.yaml

@@ -100,8 +100,7 @@ data:
   # platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
   # tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
   # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
-  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers;
-  # plex-remote: PLEX_L4_VIP),
+  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers),
   # velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
   # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
   # UNSUBSTITUTED path would ship the literal string into the cluster.
@@ -176,19 +175,6 @@ data:
   # dev  -> 192.168.1.56  (base default; next free after crafty .55 in dev's .53-.60 pool)
   # prod -> 192.168.2.23  (next free after crafty .22 in prod's .20-.30 pool; see clusters/prod)
   PIHOLE_DNS_VIP: "192.168.1.56"             # PiHole: authoritative LAN DNS for ${BASE_DOMAIN}
-  # Plex's remote-access listener: raw tcp/32400 for REMOTE Plex clients (apps/wave-0/plex-remote.yaml).
-  # A FOURTH pinned L4 VIP, for the usual reason — the `shared` Gateway is an L7 HTTPS terminator and
-  # cannot carry Plex's own TLS. Plex clients do not resolve `plex.${BASE_DOMAIN}`; they fetch this
-  # server's address from plex.tv, so the Gateway's L7 path (which still serves the web UI, and was
-  # measured working) cannot put the server on a client's connection list at all.
-  #
-  # PINNED, AND THE PIN IS LOAD-BEARING OFF-REPO: the SOHO router forwards WAN tcp/32400 to this
-  # address by hand. A VIP taken from the pool head would silently move on a reschedule and break
-  # remote access with nothing in-cluster looking wrong. See runbooks/plex-remote-access.md.
-  #
-  # dev  -> 192.168.1.57  (base default; next free after pihole-dns .56 in dev's .53-.60 pool)
-  # prod -> 192.168.2.24  (next free after pihole-dns .23 in prod's .20-.30 pool; see clusters/prod)
-  PLEX_L4_VIP: "192.168.1.57"                # Plex: tcp/32400 remote access (NOT the web UI, which is L7)
 
   # The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
   # peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable

clusters/prod/kustomization.yaml

@@ -476,26 +476,6 @@ patches:
       - op: replace
         path: /data/PIHOLE_DNS_VIP
         value: "192.168.2.23"
-      # Plex's remote-access listener VIP — raw tcp/32400 for REMOTE Plex clients
-      # (apps/wave-0/plex-remote.yaml). 192.168.2.24 is the NEXT FREE address in the prod pool
-      # 192.168.2.20-.30, verified live 2026-10-03: the CiliumLoadBalancerIPPool `default-pool`
-      # reported IPsTotal 11 / IPsUsed 4 / IPsAvailable 7, and the only four allocations are .20
-      # (cilium-gateway-shared), .21 (unifi-l4-tcp/-udp), .22 (crafty-l4-tcp/-udp) and .23
-      # (pihole-dns-tcp/-udp). It is inside the pool block the operator already asserted sits outside
-      # the SOHO router's DHCP scope, and it is not a node address (.10/.11/.12) or the
-      # control-plane VIP (.9).
-      #
-      # UNLIKE .23, THIS ONE IS ALSO A WAN PORT-FORWARD TARGET. The router forwards WAN tcp/32400
-      # here, by hand, off-repo — so changing this value is a two-place edit and the router is the
-      # half that CI cannot check. runbooks/plex-remote-access.md is the procedure.
-      #
-      # NOT ARP-PROBED. .23 was verified free with `arping` from a LAN host before it was claimed;
-      # for .24 only the in-cluster pool accounting above was checked, so an off-cluster squatter on
-      # this address would not have been caught. Probe before trusting it if remote access comes up
-      # with a conflict.
-      - op: replace
-        path: /data/PLEX_L4_VIP
-        value: "192.168.2.24"
       # The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns —
       # NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path.
       # It scopes the ingress allow for PiHole's DNS listener to LAN clients only.

decisions/2026-10-03-cilium-cgroup-hostroot.md

@@ -0,0 +1,76 @@
+---
+date: 2026-10-03
+branch: fix/cilium-cgroup-hostroot
+title: Cilium socket-LB never covered host processes — cgroup.hostRoot pointed at an unmounted path
+---
+
+## What changed
+
+`cgroup.hostRoot` changes from `/run/cilium/cgroupv2` to `/sys/fs/cgroup` in all three parity
+copies: `platform/cilium/helmrelease.yaml` (authoritative), `platform/cilium/values.yaml`
+(mirror) and `infra/ansible/roles/cilium/templates/cilium-values.yaml.j2` (bootstrap).
+`autoMount` stays `false`.
+
+**The defect.** With `autoMount: false`, the Cilium chart only bind-mounts whatever exists at
+`hostRoot`, and expects the operator to have mounted cgroup2 there (cilium 1.20.1 values.yaml).
+The repo's rationale was "k3s provides the cgroupv2 mount". That is true only for
+`/sys/fs/cgroup`. Nothing mounts cgroup2 at `/run/cilium/cgroupv2` on these nodes; it is an
+empty `DirectoryOrCreate` hostPath. So the agent mounted cgroup2 there itself, from inside its
+own private cgroup namespace, and that mount's root is the agent container's own leaf cgroup.
+The socket-LB programs (`cil_sock4_connect` etc.) were attached to it and covered only the
+agent.
+
+**The consequence.** Every other host-netns process got no Service/LB translation:
+- Service ClusterIPs were unreachable from every node's host.
+- The node holding the Gateway's L2 lease could not reach the Gateway VIP (`no route to host`).
+  Other nodes reached it only because their packets crossed the wire to the lease holder, where
+  the tc datapath handles them.
+- k3s-prod-1 holds both the Gateway lease and the kube-vip API VIP. Its apiserver could not reach
+  the OIDC issuer, logging `oidc: authenticator not initialized` since at least 2026-09-12
+  (899 occurrences by 2026-10-03). kubectl SSO through the VIP therefore failed, and in-cluster
+  OIDC clients failed on whichever calls landed on that apiserver.
+- Webhooks and aggregated APIs were unaffected only because k3s runs
+  `--enable-aggregator-routing=true` (endpoint IPs, not ClusterIPs).
+
+Cilium's own source (v1.20.1 `bpf/bpf_sock.c`, `__sock4_xlate_fwd`) handles exactly this case
+when the hook does run: a host-netns connect to an L7-LB Service (the Gateway) is rewritten to
+`127.0.0.1:<l7_lb_proxy_port>`, the local Envoy listener. The fix is to make the hook cover the
+host.
+
+## What I verified
+
+- `cilium-dbg status --verbose` on k3s-prod-1: `Socket LB: Enabled`, `Coverage: Hostns-only`.
+  The live cilium-config shows `bpf-lb-sock=false`, but socket LB is implied by kube-proxy
+  replacement. The Gateway Service frontends `192.168.2.20:{80,443}` and
+  `10.43.225.79:{80,443}` map to `127.0.0.1:14337`.
+- Host: `findmnt /run/cilium/cgroupv2` is empty on all three nodes, and `/sys/fs/cgroup` is
+  `cgroup2`.
+- In the agent: `/run/cilium/cgroupv2` is a `cgroup2 none` mount, the agent's own cgroup reads
+  `0::/`, the mount has 0 child cgroups, and `bpftool cgroup show /run/cilium/cgroupv2` lists all
+  `cil_sock*` programs. k3s runs in `0::/system.slice/k3s.service`, and the agent container in
+  `/kubepods.slice/…/cri-containerd-3ad04cde….scope`.
+- **The decisive test**, run on k3s-prod-1 in the same netns: raw TCP connect to
+  `192.168.2.20:443`, `10.43.225.79:443` and `10.43.59.141:8080` CONNECTED from inside the agent
+  container (covered cgroup) and FAILED from the host (`No route to host` / timeout).
+- Blast radius: host-network pods are cilium, cilium-envoy, cilium-operator (API via
+  `192.168.2.9:6443`, not a Service), kube-vip (`127.0.0.1:6443`, `svc_enable=false`) and
+  node-exporter (no outbound). Nothing depends on host Service translation being absent; the
+  change only makes Service/LB addresses reachable from the host where they currently fail.
+- `helm template cilium/cilium --version 1.20.1` with the new values, in the toolbox: the
+  DaemonSet bind-mounts hostPath `/sys/fs/cgroup` into `cilium-agent` and `clean-cilium-state`,
+  the cilium-config shows `cgroup-root: "/sys/fs/cgroup"`, and the init containers are unchanged
+  (no `mount-cgroup`).
+- Toolbox gate: see the PR description.
+
+## What I did NOT verify
+
+- **The rollout itself.** After merge, the agent DaemonSet rolls one node at a time
+  (`maxUnavailable: 1`). On each node, afterwards: host TCP connects to the three targets above
+  must succeed. On k3s-prod-1, the apiserver's background OIDC init retry must stop logging
+  `authenticator not initialized` with no k3s restart.
+- That the old programs, attached to the previous agent container's cgroup, disappear with it.
+  Expected, because cgroup BPF attachments die with the cgroup, but not observed.
+- An end-to-end kubectl OIDC login (needs a browser).
+- No regression test was added. A natural one is an L2 conformance check that a host-netns
+  connect to the Gateway VIP succeeds on the lease holder, but `conformance/**` is a sensitive
+  path. Left as a follow-up.

decisions/2026-10-03-plex-remote-access-l4.md

@@ -1,129 +0,0 @@
----
-date: 2026-10-03
-branch: feat/plex-remote-access-l4
-title: Plex remote access — raw tcp/32400 via a pinned LoadBalancer VIP, not the L7 Gateway
----
-
-## What changed
-
-Reported symptom: users cannot reach Plex from outside the LAN. **Two independent causes**, measured
-rather than inferred:
-
-1. **Plex advertised no remote connection at all.** Live `Preferences.xml` on prod:
-   `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while
-   `PublishServerOnPlexOnlineKey="1"` — publishing was on and there was nothing reachable to publish.
-   Plex's automatic UPnP/NAT-PMP path cannot work from a pod on the Cilium overlay.
-2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` refused;
-   `136.51.115.252:443` answered with a **self-signed router cert** (`C=ZZ, CN=%D`), not the
-   Gateway's LE wildcard. The Plex Service was `ClusterIP` only — no off-cluster path to 32400 by
-   any route.
-
-This branch fixes the in-cluster half. **Remote access does not work until the operator also does
-two manual steps** (`runbooks/plex-remote-access.md`): the WAN tcp/32400 port-forward, and ticking
-"manually specify public port" in the Plex UI.
-
-- **`apps/wave-0/plex-remote.yaml` (new, hand-written).** A `type=LoadBalancer` Service on tcp/32400
-  pinned to `${PLEX_L4_VIP}` via `lbipam.cilium.io/ips`, plus a `CiliumNetworkPolicy`
-  (`plex-allow-remote-32400`) admitting `world`/`cluster`/`host`/`remote-node` to 32400. Non-catalog
-  because `schemas/servicedefinition.schema.json` is `additionalProperties: false` and has no field
-  for a LoadBalancer Service or a pinned VIP — the same reason `apps/wave-0/pihole-dns.yaml`,
-  `platform/crafty/service-l4.yaml` and `apps/vms/unifi/50-service-l4.yaml` are hand-written.
-- **`catalog/render.py`:** an `EXTRA_UNITS` entry (`plex-remote`, anchored to `plex`, `above: False`)
-  so `--emit-waves` places the unit in plex's own wave.
-- **`clusters/base/cluster-vars.yaml`:** new `PLEX_L4_VIP` (dev default `192.168.1.57`), plus the
-  substitution inventory comment updated to name `plex-remote: PLEX_L4_VIP`.
-- **`clusters/prod/kustomization.yaml`:** prod override `PLEX_L4_VIP: 192.168.2.24`.
-- **`apps/wave-0/kustomization.yaml`:** `plex-remote.yaml` added to `resources`.
-- **`runbooks/plex-remote-access.md` (new):** the two manual steps, the four-step verification, why
-  no alert is possible, and the two documented warts below.
-
-**`ADVERTISE_IP` was deliberately NOT set.** Setting it to `https://plex.home.keithjackson.dev:443`
-was the alternative — it needs no new WAN hole and the mechanism was verified working on an
-already-claimed server. **The operator chose the native path (2026-10-03)** so Plex terminates its own
-TLS (`*.plex.direct`) and streams bypass the Envoy L7 hop. `catalog/plex.yaml`'s deferral note for
-`ADVERTISE_IP` remains accurate and untouched.
-
-**`externalTrafficPolicy: Cluster`, with a known cost.** `Local` would preserve the client IP, but
-Cilium's L2 announcer does not follow the backend pod — the wart that cost this LAN an 8-day
-Minecraft outage (see `platform/crafty/service-l4.yaml`). Consequence accepted and documented: SNAT
-puts the client in `192.168.0.0/16`, inside `ALLOWED_NETWORKS`, so **Plex classifies remote streams
-as LAN**. That is a metering effect (remote bandwidth limits will not bind), not an authorization one.
-
-**SENSITIVE PATHS — this PR touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK
-it.** It needs the operator's `/approve-override <reason>` posted as a PR **review** of type
-"Comment". I have deliberately not posted it: CLAUDE.md §1 forbids an agent clearing this on its own
-PR, and the agent account holds `owner`, so the prohibition is a rule, not a limit.
-
-## What I verified
-
-All measurements against live prod, 2026-10-03, read-only.
-
-- **Plex's advertised state**, the finding this branch rests on: `customConnections=<ABSENT>`,
-  `ManualPortMappingMode=<ABSENT>`, `ManualPortMappingPort=<ABSENT>`,
-  `PublishServerOnPlexOnlineKey="1"`, `allowedNetworks="10.0.0.0/8,192.168.0.0/16"`. Read by
-  attribute grep, never by dumping `Preferences.xml` (it holds `PlexOnlineToken`).
-- **The L7 Gateway path works and is NOT the problem.** Via the Gateway VIP,
-  `GET https://plex.home.keithjackson.dev/identity` → `HTTP 200`, `claimed="1"`, real
-  `machineIdentifier`. HTTPRoute `plex-32400`: `Accepted=True`, `ResolvedRefs=True`.
-- **Plex enforces its own auth**, which is what makes an unfiltered WAN port defensible:
-  `/library/sections`, `/status/sessions`, `/:/prefs` all → **401** unauthenticated through the
-  Gateway.
-- **The SNAT/allowedNetworks wart is real, not theoretical.** A tagged probe through the Gateway was
-  logged by Plex as `[10.0.2.214:36040 (Allowed Network (WAN))]` — the Envoy pod IP, classified as an
-  allowed network.
-- **WAN state:** `136.51.115.252:32400` → connection refused. `136.51.115.252:443` → self-signed
-  cert `C=ZZ, ST=Somewhere, O=%D342e7e6b, CN=%D`. Public DNS resolves correctly:
-  `plex.home.keithjackson.dev` → CNAME `wan.home.keithjackson.dev` → `136.51.115.252`.
-- **`192.168.2.24` is free in-cluster.** `CiliumLoadBalancerIPPool default-pool`: IPsTotal 11 /
-  IPsUsed 4 / IPsAvailable 7; the only allocations are `.20` (cilium-gateway-shared), `.21`
-  (unifi-l4-*), `.22` (crafty-l4-*), `.23` (pihole-dns-*). Not a node address (`.10/.11/.12`) and not
-  the kube-vip control-plane VIP (`.9`).
-- **The manual-port-mapping step genuinely cannot be automated.** Grep of the running image's whole
-  init surface (`/etc/cont-init.d`, `/plex-common.sh`, `/usr/local/bin`) for `ManualPortMapping` and
-  `PLEX_PREFERENCE` → **NONE**.
-- **`ADVERTISE_IP` would have worked on a claimed server** (recorded because the runbook's
-  "why not" rests on it): `setPref "customConnections"` at line 126 of `40-plex-first-run` sits
-  **outside** the `PLEX_CLAIM`/`PlexOnlineToken` guard, and the `/.firstRunComplete` early-exit
-  marker is on the **overlay** FS, not the `/config` PVC, so the script re-runs every pod start.
-- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
-  `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
-  **pass**. The rc 3 is two environmental skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`)
-  and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3, not as pass.**
-- **`render-golden` passing is the specific proof** that the `render.py` `EXTRA_UNITS` change, the new
-  hand-written unit and the edited wave kustomization are mutually consistent.
-
-## What I did NOT verify
-
-- **THE ACTUAL FIX. Nothing in this branch has been proven to make remote access work** — it cannot
-  be, because both remaining steps are off-repo and manual (router forward, Plex UI toggle). This
-  branch removes two blockers and adds none; it does not demonstrate an outcome.
-- **Outside-in reachability, at all.** Every probe above was taken from inside the LAN, so the
-  `:443` self-signed cert strictly proves only that there is **no NAT hairpin** — it does **not**
-  prove outside-in failure. A router cert on WAN:443 is suggestive (its management UI would shadow a
-  forward), but unproven. **This repo has no off-LAN vantage point.** Test 3 in the runbook
-  (app.plex.tv from a cellular connection) is the only honest check.
-- **`192.168.2.24` was never ARP-probed off-cluster.** Only in-cluster pool accounting was checked.
-  `.23` was `arping`-probed before it was claimed; `.24` was not, so an off-cluster squatter would
-  not have been caught.
-- **The LoadBalancer has never been reconciled.** `plex-l4-tcp` does not exist in prod yet, so
-  LB-IPAM has not actually assigned the pinned VIP, L2 announcement has not been observed for it, and
-  the new CiliumNetworkPolicy has never admitted a real packet. Whether `world`/`host`/`cluster`/
-  `remote-node` is the correct entity set **under this SNAT path** is reasoned from
-  `platform/crafty/networkpolicy.yaml`'s working game-port rule, **not measured**.
-- **Whether Plex's remote path needs anything beyond the manual port tick.** If plex.tv still lists
-  no remote connection after both manual steps, this branch is necessary-but-insufficient.
-- **The CI approval-policy outcome.** The local stage reported green, but that is an offline N/A, not
-  a verdict: this PR touches `clusters/**`, so CI will block it pending an override.
-- **PRE-EXISTING DRIFT I FOUND AND DELIBERATELY LEFT ALONE — worth its own follow-up.** Running the
-  documented `python3 catalog/render.py --emit-waves` regenerates the wave kustomizations and wants to
-  add **`vm-demo.yaml`** to wave 0 and **`keycloak.yaml`** to wave 1. Both are listed as
-  intentionally not onboarded in `ci/render-golden.sh`
-  (`RENDER_GOLDEN_NOT_ONBOARDED="keycloak vm-demo"`); their manifests are tracked in `apps/` but
-  deliberately absent from the `resources` lists so Flux does not apply them. `--emit-waves` places
-  any file present on disk and has **no awareness of that exclusion**, while `render-golden.sh` uses
-  `--wave-layout` and compares rendered manifests only — it never checks kustomization `resources`,
-  which is why this has survived unnoticed. **So the header "DO NOT hand-edit: re-run
-  `render.py --emit-waves`" is currently unsafe advice**: following it onboards an IdP and a KubeVirt
-  VM. I therefore restored both kustomizations and hand-added only my one line. I did not fix
-  `--emit-waves`, and I have not determined how long the drift has existed or whether either service
-  is *meant* to be onboarded.

infra/ansible/roles/cilium/templates/cilium-values.yaml.j2

@@ -51,11 +51,14 @@ hubble:
 l2announcements:
   enabled: true
 
-# k3s ships its own cgroup mount; point Cilium at it instead of auto-mounting.
+# Point Cilium at the host's real cgroup2 root (/sys/fs/cgroup, mounted by systemd) instead of
+# auto-mounting. NOT /run/cilium/cgroupv2: nothing mounts cgroup2 there, so the agent mounted its
+# OWN namespaced cgroup and socket-LB covered only itself — host processes (the apiserver) lost
+# Service/LB translation. [parity] platform/cilium/helmrelease.yaml explains (2026-10-03).
 cgroup:
   autoMount:
     enabled: false
-  hostRoot: /run/cilium/cgroupv2
+  hostRoot: /sys/fs/cgroup
 
 # Roll the cilium agent DaemonSet ONE node at a time. [parity] must match the
 # platform/cilium HelmRelease. The cilium 1.20.1 chart defaults this to 2, which on a

platform/cilium/helmrelease.yaml

@@ -106,14 +106,33 @@ spec:
     # CiliumL2AnnouncementPolicy (reserved 10.73.0.240-.250) authored by WS-NET in Phase 2.
     l2announcements:
       enabled: true
-    # --- cgroup: k3s ships its own cgroup mount; point Cilium at it, don't auto-mount ---
-    # [parity] MUST equal Ansible cgroup block. Not in the deliverable's headline list, but it IS
-    # part of the working CNI config on k3s: reconciling without it lets Cilium try to auto-mount
-    # cgroupv2 and break on adopt. Kept in parity for a safe Phase-2 adoption.
+    # --- cgroup: point Cilium at the HOST's cgroup2 root, which systemd mounts at /sys/fs/cgroup ---
+    # [parity] MUST equal Ansible cgroup block (roles/cilium/templates/cilium-values.yaml.j2).
+    #
+    # FIXED 2026-10-03 — hostRoot WAS /run/cilium/cgroupv2, AND THAT SILENTLY BROKE SOCKET-LB FOR
+    # EVERY HOST PROCESS. With autoMount disabled, the chart only bind-mounts whatever already exists
+    # at hostRoot ("it's expected that users have mounted cgroup2 filesystem at the specified
+    # cgroup.hostRoot" — cilium 1.20.1 values.yaml). Nothing ever mounted cgroup2 at
+    # /run/cilium/cgroupv2 on these nodes (it is an empty DirectoryOrCreate hostPath), so the agent
+    # mounted one ITSELF from inside its own private cgroup namespace. That mount's root is the
+    # agent container's OWN leaf cgroup, so cil_sock4_connect & co. were attached there and covered
+    # only the agent. Every other host-netns process (the k3s apiserver in
+    # /system.slice/k3s.service, curl over SSH) was untranslated: Service ClusterIPs timed out from
+    # every node, and the node holding the Gateway's L2 lease could not reach the Gateway VIP at all
+    # ("no route to host"). That is why k3s-prod-1's apiserver logged `oidc: authenticator not
+    # initialized` from 2026-09-12 on — kubectl SSO through the kube-vip VIP failed outright.
+    # Measured on k3s-prod-1 before this change: TCP connect to 192.168.2.20:443, 10.43.225.79:443
+    # and 10.43.59.141:8080 SUCCEEDED from inside the agent container (the covered cgroup) and
+    # FAILED from the host, same node, same netns. Full write-up:
+    # decisions/2026-10-03-cilium-cgroup-hostroot.md.
+    #
+    # /sys/fs/cgroup IS a cgroup2 mount in the host's root cgroup namespace on every node (verified
+    # with findmnt on all three), so bind-mounting it gives the agent the real host root. autoMount
+    # stays off because the mount it would create already exists.
     cgroup:
       autoMount:
         enabled: false
-      hostRoot: /run/cilium/cgroupv2
+      hostRoot: /sys/fs/cgroup
     # --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident ---
     # [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1.
     # The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate.

platform/cilium/values.yaml

@@ -41,11 +41,13 @@ hubble:
 # [parity] ingress VIP announcement on the flat LAN (docs/09, OQ-1)
 l2announcements:
   enabled: true
-# [parity] k3s provides the cgroupv2 mount; do not auto-mount
+# [parity] the host's cgroup2 root is /sys/fs/cgroup (systemd mounts it); do not auto-mount.
+# NOT /run/cilium/cgroupv2: nothing mounts cgroup2 there, so socket-LB attached to the agent's own
+# cgroup and host processes lost Service/LB translation (2026-10-03 — see helmrelease.yaml).
 cgroup:
   autoMount:
     enabled: false
-  hostRoot: /run/cilium/cgroupv2
+  hostRoot: /sys/fs/cgroup
 # [parity] roll the agent DaemonSet ONE node at a time. Overrides the cilium 1.20.1 chart
 # default of 2, which on 3 nodes lets a single bad render break 2/3 agents at once — as the
 # 2026-09-01 incident did (k8sServiceHost: "" → the `config` init container fell back to the

runbooks/plex-remote-access.md

@@ -1,148 +0,0 @@
-# plex-remote-access — getting Plex reachable from outside the LAN, and proving it
-
-**Status: the in-cluster half is shipped; the last mile is TWO MANUAL STEPS and remote access does
-NOT work until both are done.** Neither can live in this repo.
-**Alert:** none, and none is possible from in-cluster signals — see "Why no alert" below. This is an
-**operator procedure**, not an incident page.
-**Files:** `apps/wave-0/plex-remote.yaml` (the tcp/32400 LoadBalancer + its CiliumNetworkPolicy),
-`clusters/base/cluster-vars.yaml` + `clusters/prod/kustomization.yaml` (`PLEX_L4_VIP`),
-`catalog/render.py` (the `EXTRA_UNITS` entry that places the unit), `catalog/plex.yaml` (Plex's own
-env — note `ADVERTISE_IP` is deliberately NOT set; see "Why not the Gateway URL").
-**Related:** `runbooks/plex-quicksync.md` and `runbooks/plex-claim.md` — the other two Plex
-procedures with a manual last mile.
-
-## What was actually broken (measured 2026-10-03)
-
-Plex was serving correctly and was reachable on the LAN. Two independent things stopped remote
-clients:
-
-1. **Plex advertised no remote connection at all.** In the live `Preferences.xml`:
-   `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while
-   `PublishServerOnPlexOnlineKey="1"`. The server was trying to publish itself to plex.tv and had
-   nothing reachable to publish. Plex's automatic path (UPnP/NAT-PMP) cannot work here: the process
-   lives in a pod on the Cilium overlay (`10.0.2.136`), it cannot speak UPnP to the SOHO router, and
-   a mapping to the pod IP would be meaningless if it could.
-2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` → connection refused.
-   `136.51.115.252:443` → a **self-signed router certificate** (`C=ZZ, CN=%D`), not the Gateway's
-   Let's Encrypt wildcard.
-
-The Service was `ClusterIP` only, so port 32400 had no path off-cluster by any route.
-
-**Caveat on that second measurement, because it is easy to over-read:** it was taken from *inside*
-the LAN, so strictly it proves there is no NAT hairpin (reflection). It does not prove outside-in
-failure. But a router certificate answering 443 means the router's own management UI is bound to the
-WAN port, and that *would* shadow a port-forward for genuine outside traffic too. **Verify from a
-real outside vantage point** (phone on cellular, or an external port checker) before concluding
-either way — this repo has no such vantage point and must not pretend otherwise.
-
-## Step 1 — forward WAN tcp/32400 on the router (operator, off-repo)
-
-    WAN tcp/32400  ->  192.168.2.24  (${PLEX_L4_VIP})  port 32400
-
-`192.168.2.24` is a pinned Cilium LB VIP, ARP-announced on the flat LAN by the existing
-`CiliumL2AnnouncementPolicy default-l2`. It is **pinned on purpose**: this address is typed into the
-router by hand, so it must never move on a reschedule.
-
-While in the router, **check what else is bound to the WAN interface.** If remote management is
-listening on 443 (see the certificate measurement above), move it or disable it — otherwise the
-Gateway's 80/443 forward is shadowed and the Plex *web UI* over the internet stays broken even
-though remote *streaming* works. These are two separate paths and they fail independently.
-
-Confirm the VIP is actually answering on the LAN first, so a failure later is unambiguous:
-
-    ./infra/devcontainer/dev.sh run bash -c \
-      'kubectl --context k3s-prod -n plex get svc plex-l4-tcp -o wide'
-
-`EXTERNAL-IP` must read `192.168.2.24`. If it is `<pending>`, LB-IPAM could not assign — the address
-is outside the pool or already taken; see `clusters/prod/kustomization.yaml` for the pool accounting.
-
-**`192.168.2.24` was never ARP-probed off-cluster.** Only the in-cluster pool accounting was checked
-(IPsTotal 11 / IPsUsed 4 / IPsAvailable 7 on 2026-10-03). If the VIP behaves oddly, probe it from a
-LAN host — `arping 192.168.2.24` — the way `.23` was probed before it was claimed.
-
-## Step 2 — tick "manually specify public port" in Plex (operator, UI only)
-
-**There is no GitOps path to this and no environment variable for it.** Measured in the running
-image on 2026-10-03: a grep of the entire init surface (`/etc/cont-init.d`, `/plex-common.sh`,
-`/usr/local/bin`) for `ManualPortMapping` and `PLEX_PREFERENCE` returned **NONE**. The image writes
-only six preferences, and this is not one of them. It is the same class of manual last mile as the
-`HardwareAcceleratedCodecs` toggle in `runbooks/plex-quicksync.md`.
-
-In the Plex web UI (`https://plex.home.keithjackson.dev`, on the LAN):
-
-**Settings → Remote Access**
-- Ensure **Enable Remote Access** is on. (`PublishServerOnPlexOnlineKey` was already `1`.)
-- Tick **Manually specify public port** and set it to **32400**.
-- **Apply**, then let the panel settle. It should report *"Fully accessible outside your network"*.
-
-Without this tick Plex waits for a NAT mapping it can never obtain and publishes no remote
-connection, so the port-forward in step 1 sits there unused.
-
-## How to verify
-
-Do these in order; each one isolates a different half.
-
-**1. The port is open from outside.** From a phone on cellular, or any off-LAN host:
-
-    curl -sk -o /dev/null -w '%{http_code}\n' https://136.51.115.252:32400/identity
-
-`200` means the forward and the policy are both good. A hang or refusal means step 1.
-
-**2. Plex itself believes it is reachable.** Settings → Remote Access must say *"Fully accessible
-outside your network"*. If it says *"not available outside your network"* while test 1 passes, step 2
-was not applied.
-
-**3. plex.tv is handing out a remote connection.** The decisive test, because it is what clients
-actually consume: sign in to [app.plex.tv](https://app.plex.tv) from an off-LAN network and confirm
-the server appears and plays. A client finds this server through plex.tv's connection list, never
-through DNS.
-
-**4. The policy is admitting the traffic.** If test 1 fails *after* the forward is confirmed, the
-CiliumNetworkPolicy is the next suspect:
-
-    ./infra/devcontainer/dev.sh run bash -c \
-      'kubectl --context k3s-prod -n plex get cnp plex-allow-remote-32400'
-
-See `runbooks/denied-flows.md` for reading actual drops.
-
-## Why no alert
-
-Every in-cluster signal is green whether or not remote access works. The Service has endpoints, the
-pod is `Running`, the policy is applied, and `plex-down` watches
-`kube_deployment_status_replicas_available` — none of which can see a router with a missing
-port-forward or an unticked checkbox. An alert would need an off-LAN prober, which this cluster does
-not have. **Test 3 above is the only honest check, and it is manual.**
-
-## Why not the Gateway URL (`ADVERTISE_IP`)
-
-The alternative was to leave Plex on `ClusterIP` and set
-`ADVERTISE_IP: https://plex.home.keithjackson.dev:443`, which writes Plex's `customConnections`
-preference and would have put the server on plex.tv's list via the existing L7 Gateway — no new WAN
-hole. That mechanism was verified to work on an already-claimed server (the `ADVERTISE_IP` block at
-line 126 of the image's `40-plex-first-run` sits **outside** the claim guard, and `/.firstRunComplete`
-lives on the overlay FS, not the `/config` PVC, so it re-runs every pod start), and the Gateway path
-itself was measured serving Plex correctly: `HTTP 200`, `claimed="1"`, with `401` on every protected
-endpoint.
-
-**The operator chose the native path anyway (2026-10-03), and the reasons are real:** Plex terminates
-its own TLS with its `*.plex.direct` certificate, and video streams bypass the Envoy L7 hop instead
-of being proxied through it. The cost is a new unfiltered WAN port — acceptable because Plex
-authenticates every caller itself. `ADVERTISE_IP` therefore stays unset in `catalog/plex.yaml`, and
-the deferral note there is still accurate.
-
-## Known wart: remote clients are classified as LAN
-
-`apps/wave-0/plex-remote.yaml` uses `externalTrafficPolicy: Cluster`, which SNATs the client to the
-ingress node's address (`192.168.2.1x`). That falls inside the `192.168.0.0/16` entry of
-`ALLOWED_NETWORKS`, so **Plex treats every remote stream as local.** Measured on the Gateway path,
-which SNATs the same way: Plex logged `[10.0.2.214:36040 (Allowed Network (WAN))]`.
-
-This is a **metering** effect, not a security one — authorization is unchanged, as the 401s above
-show — but Plex's remote-quality and bandwidth limits will not bind on these streams.
-
-`Local` would preserve the client IP and fix it. **Do not switch to it.** Cilium's L2 announcer
-elects one lease holder per Service and does *not* follow the backend pod, so the VIP can be
-announced by a node with no Plex pod, black-holing the port while everything reports healthy. That
-exact change cost this LAN an 8-day Minecraft outage in September 2026 — the full write-up is at the
-top of `platform/crafty/service-l4.yaml`. Plex is a single-replica `Recreate` Deployment, so the
-node mismatch is the normal case here, not a rare one.