fix(plex): advertise the plex.direct name, not the bare VIP (server vanished from the plex.tv account)

customConnections held two hand-typed URLs and neither could ever complete, so Plex had nothing publishable and the server disappeared from the account: - https://192.168.2.24:32400 -- the server's cert is *.130072f20c334d6c8f63d0dfef87309b.plex.direct with NO IP SAN, so bare-IP https fails TLS for every client. Measured: a continuous storm of "CERT: incomplete TLS handshake ... certificate unknown" from real LAN clients arriving through the plex-l4-tcp VIP. - https://plex.home.keithjackson.dev:32400 -- nothing listens; the Gateway serves that host on 443 (measured http=000 against 192.168.2.20:32400). With no completable connection plex.tv published nothing ("Mapped - Not Published"), Plex fell back to Relay, and clients reported "not authorized". Fix: advertise <dashed-ip>.<serverhash>.plex.direct -- public DNS that resolves to the embedded address, so it points at the LAN VIP and matches the wildcard cert. Verified with full TLS validation (no curl -k): 200 + verify OK, where the bare-IP form fails to connect. Declared as ADVERTISE_IP so git owns the value instead of a web form: the image's 40-plex-first-run writes it into customConnections on every pod start, outside the claim guard. Does NOT fix off-LAN access -- the advertised address is RFC1918 and the WAN tcp/32400 forward remains unverified. ManualPortMappingMode=1 with an absent port is left as-is (no GitOps path exists for it). Touches clusters/** -> sensitive paths -> needs the operator's /approve-override, deliberately not self-posted (CLAUDE.md 1). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Merge pull request 'fix(cilium): point cgroup.hostRoot at the host cgroup2 root (socket-LB never covered host processes)' (#42 ) from fix/cilium-cgroup-hostroot into main
2026-10-04 02:53:01 -05:00 · 2026-10-04 07:10:31 +00:00 · 2026-10-04 07:08:19 +00:00 · 2026-10-04 02:03:42 -05:00 · 2026-10-04 06:50:56 +00:00 · 2026-10-04 01:34:52 -05:00
27 changed files with 1930 additions and 74 deletions
@@ -14,10 +14,12 @@ resources:
 - nextcloud-db-backup-credentials-prod.sops.yaml
 - nextcloud-db-backup-credentials.sops.yaml
 - nextcloud-db.yaml
+- pihole-bootstrap-secret.sops.yaml
 - pihole-dns.yaml
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
 - plex-claim-secret.sops.yaml
+- plex-remote.yaml
 - plex.yaml
 - qbittorrent-bootstrap-secret.sops.yaml
 - qbittorrent-oauth2-proxy-credentials.sops.yaml
@@ -0,0 +1,72 @@
+# =================================================================================================
+# PiHole WEB/API ADMIN PASSWORD — the one piece of PiHole state that was NOT declared anywhere.
+# =================================================================================================
+# WHY THIS FILE EXISTS. Measured on live prod 2026-10-04, `pihole.toml` marks every non-default
+# setting with `### CHANGED`, and names the ones that came from the environment `### CHANGED (env)`.
+# The complete list was:
+#
+#     dns.upstreams          ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     dns.listeningMode      ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     dns.rateLimit.count    ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     misc.dnsmasq_lines     ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     files.macvendor        ### CHANGED         <- set by the image itself, container path
+#     webserver.api.pwhash   ### CHANGED         <- SET OUT OF BAND. Declared NOWHERE. This file.
+#
+# So exactly ONE operator-set value lived only on the Longhorn PVC: the admin password, as a
+# one-way BALLOON-SHA256 hash. Nobody could recover it (a hash does not reverse), nothing in git
+# produced it, and a rebuild from git would have come up with a DIFFERENT password — or none.
+#
+# THE REASON THIS BLOCKS HA, which is why it is fixed first. The next step is running more than one
+# PiHole replica, and a replica with a fresh PVC would come up with its own unrelated password. The
+# admin UI would then accept or reject the same credential depending on which pod answered. Declaring
+# it here makes every replica, and every rebuild, identical by construction.
+#
+# MECHANISM: `webserver.api.password` is a real, settable FTL config key — verified in the running
+# image, where `pihole setpassword` is itself just `setFTLConfigValue "webserver.api.password"` with
+# the comment "pihole-FTL will automatically hash the password". So FTL takes CLEARTEXT on this key
+# and writes the hash into `webserver.api.pwhash` itself. catalog/pihole.yaml therefore passes this
+# value as `FTLCONF_webserver_api_password`, and the hash on the PVC becomes a derived artefact
+# rather than the source of truth.
+#
+# DEFENCE IN DEPTH, NOT THE ONLY GATE. The admin UI is already behind the oauth2-proxy sidecar
+# (`--allowed-group=platform-admins`), so this password is a SECOND gate. It is kept rather than set
+# empty because the `pihole` Service still publishes port 80 (PiHole direct, bypassing the proxy on
+# :4180) and only the CiliumNetworkPolicy closes it — and the default-deny policy in this repo has a
+# documented drift history. An unauthenticated admin UI that can rewrite LAN DNS is not a good thing
+# to leave resting on one policy staying correct. Narrowing that Service port is tracked follow-up.
+#
+# ROTATION: `sops` this file in your own terminal, change the value, commit. Flux re-applies the
+# Secret, the pod restarts, FTL re-hashes. Do NOT `sops -d` it into a shared terminal or a log
+# (CLAUDE.md section 6); read it in an editor. The value below was generated randomly at authoring
+# time and has never been printed to a transcript.
+apiVersion: v1
+kind: Secret
+metadata:
+    name: pihole-bootstrap
+    namespace: pihole
+    labels:
+        app.kubernetes.io/name: pihole
+type: Opaque
+stringData:
+    #ENC[AES256_GCM,data:TMOuRyJeb5IVn3lZPXY7lgh1KGxu7GY9+WIOLHpMB50JqzmM2tHIaWUWKTt1JtZoX/E5/Y7WCUTnfkJLhRBN2zV2MwWRBSLYVy5CMg7xV8PTLRZJYAVUYS5Trg==,iv:q+DpfT6PQfY5ZuNsmP7Ddl+PDLjSm89fJICAhYoW64k=,tag:axijaO3DaxIZgZ3LeK2hXA==,type:comment]
+    webui-password: ENC[AES256_GCM,data:7FcK/bUW79M46f/MlouAmATKbMnoUTNuMUjtqJ5vpU4=,iv:I8DBFB39uXZV42Ie2IPwSefmMgdKEtiDknfc9Tu7h5E=,tag:L423784UrDU3TvylpHc6KQ==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0MkRSNjZLZlpXYkcvaVJL
+            cjlqb2wrcWREZnZLdGpsTHNPWEpseHlRZnpjCnJVM1FDQ2M2dXlaS1JBZ2k2V1BP
+            Yi9QUDZSMzZsL09pTi93RTdjdGxtY2MKLS0tIDIrQTZIVWdhMGIxSCtlUFN2Mlho
+            SWVybld4VjJyaHRSKzZZU0xsUUV2SEEKbDYHV/qBZVspK7DpBW3vYjRMJ/HmAN+9
+            G9bOlNEp0jAtcNZEn6lpQ9ORgvmIrhJHoBQRLabdMWzNWSZsNn56tA==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-10-04T06:01:27Z"
+    mac: ENC[AES256_GCM,data:rhTMTOwLOtHXHgqI5M2ZW/6w+UpLpxBlTk7NoN0wmGSqOcE+OGrXA1DhkZrJweyQT0RSNSSBm/6QMztLP81eepOzdp2rEkwvbQPK/l6K9g16Ww9c1IatS3NQsKEB8AoMDx7jBQ6exXYgggPx3MJsNYbE+WyloQ96U9KPdrecAM8=,iv:n34v2zX9D5xsaR1PFCH94mNMiuRfhUyI2lseUCoqiTA=,tag:9YmN9nIVLXfsGYIb0bUl2w==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4
@@ -35,34 +35,18 @@ spec:
    - protocol: TCP
      port: 53
 ---
-# PVC pihole-config (R-STATE)
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: pihole-config
-  namespace: pihole
-  labels:
-    app.kubernetes.io/name: pihole
-spec:
-  accessModes:
-  - ReadWriteOnce
-  storageClassName: longhorn
-  resources:
-    requests:
-      storage: 2Gi
---
-# Deployment (R-PLAT-02)
+# StatefulSet (R-PLAT-02)
 apiVersion: apps/v1
-kind: Deployment
+kind: StatefulSet
 metadata:
  name: pihole
  namespace: pihole
  labels:
    app.kubernetes.io/name: pihole
 spec:
-  replicas: 1
-  strategy:
-    type: Recreate
+  replicas: 2
+  serviceName: pihole
+  podManagementPolicy: Parallel
  selector:
    matchLabels:
      app.kubernetes.io/name: pihole
@@ -90,9 +74,21 @@ spec:
          value: '0'
        - name: FTLCONF_misc_dnsmasq_lines
          value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}
+        - name: FTLCONF_webserver_api_password
+          valueFrom:
+            secretKeyRef:
+              name: pihole-bootstrap
+              key: webui-password
        volumeMounts:
        - name: config
-          mountPath: /data/config
+          mountPath: /etc/pihole
+        lifecycle:
+          postStart:
+            exec:
+              command:
+              - /bin/bash
+              - -c
+              - "# Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and\n# does NOT run a gravity rebuild.\nset -u\nGRAVITY=/etc/pihole/gravity.db\n\n# THE DECLARED SET \u2014 one URL per line, the source of truth for what PiHole blocks.\n# To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.\nDECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'\n\n# On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,\n# and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.\ni=0\nwhile [ ! -s \"$GRAVITY\" ] && [ \"$i\" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done\nif [ ! -s \"$GRAVITY\" ]; then\n  echo \"pihole-adlists: gravity.db absent after 180s - leaving adlists untouched\" >&2\n  exit 0\nfi\n\nsql() { pihole-FTL sqlite3 \"$GRAVITY\" \"$1\"; }\nCHANGED=0\n\n# 1) Every declared URL must be present AND enabled.\necho \"$DECLARED\" | while IFS= read -r url; do\n  [ -n \"$url\" ] || continue\n  printf '%s\\n' \"$url\"\ndone > /tmp/declared.txt\nwhile IFS= read -r url; do\n  [ -n \"$url\" ] || continue\n  # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.\n  have=$(sql \"select count(*) from adlist where address='$url';\")\n  if [ \"$have\" = \"0\" ]; then\n    echo \"pihole-adlists: adding $url\"\n    sql \"insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');\"\n    CHANGED=1\n  else\n    en=$(sql \"select enabled from adlist where address='$url';\")\n    if [ \"$en\" != \"1\" ]; then\n      echo \"pihole-adlists: re-enabling $url\"\n      sql \"update adlist set enabled=1 where address='$url';\"\n      CHANGED=1\n    fi\n  fi\ndone < /tmp/declared.txt\n\n# 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).\nsql \"select address from adlist where enabled=1;\" | while IFS= read -r addr; do\n  [ -n \"$addr\" ] || continue\n  if ! grep -qxF \"$addr\" /tmp/declared.txt; then\n    echo \"pihole-adlists: disabling undeclared $addr\"\n    sql \"update adlist set enabled=0 where address='$addr';\"\n    echo changed >> /tmp/adlist-changed\n  fi\ndone\n[ -s /tmp/adlist-changed ] && CHANGED=1\nrm -f /tmp/adlist-changed /tmp/declared.txt\n\n# 3) Rebuild gravity ONLY if the declared set actually moved.\nif [ \"$CHANGED\" = \"1\" ]; then\n  echo \"pihole-adlists: declared set changed - running gravity update\"\n  pihole -g || echo \"pihole-adlists: gravity update failed; existing gravity.db retained\" >&2\nelse\n  echo \"pihole-adlists: already matches the declared set - nothing to do\"\nfi\nexit 0\n"
      - name: oauth2-proxy
        image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
        imagePullPolicy: IfNotPresent
@@ -163,10 +159,39 @@ spec:
            port: http
          initialDelaySeconds: 5
          periodSeconds: 15
-      volumes:
-      - name: config
-        persistentVolumeClaim:
-          claimName: pihole-config
+      affinity:
+        podAntiAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+          - labelSelector:
+              matchLabels:
+                app.kubernetes.io/name: pihole
+            topologyKey: kubernetes.io/hostname
+  volumeClaimTemplates:
+  - metadata:
+      name: config
+      labels:
+        app.kubernetes.io/name: pihole
+    spec:
+      accessModes:
+      - ReadWriteOnce
+      storageClassName: longhorn
+      resources:
+        requests:
+          storage: 2Gi
+---
+# PodDisruptionBudget — keep one replica through a drain (availability.replicas)
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+  name: pihole
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+spec:
+  minAvailable: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: pihole
 ---
 # Service
 apiVersion: v1
@@ -269,11 +294,11 @@ spec:
  - name: pihole.slis
    rules:
    - alert: pihole-down
-      expr: kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"} == 0 or absent(kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"})
+      expr: kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"} == 0 or absent(kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"})
      for: 5m
      labels:
        severity: critical
        service: pihole
      annotations:
        summary: pihole is down
-        description: 'pihole: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
+        description: 'pihole: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready).'
@@ -0,0 +1,126 @@
+---
+# =================================================================================================
+# PLEX REMOTE ACCESS — raw tcp/32400 north-south exposure, and the policy that admits it.
+# =================================================================================================
+# HAND-WRITTEN, NOT RENDERED. A non-catalog app-tier unit anchored to the `plex` ServiceDefinition
+# and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS table — the same
+# mechanism gitea-actions and pihole-dns use. It is hand-written for the SAME reason those are:
+# schemas/servicedefinition.schema.json is `additionalProperties: false` and has NO field for a
+# type=LoadBalancer Service, for a pinned lbipam VIP, or for raw-TCP north-south exposure at all.
+# `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator.
+#
+# WHY THE L7 GATEWAY IS NOT ENOUGH HERE — and it is NOT because the Gateway fails to serve Plex.
+# Measured 2026-10-03 through the Gateway VIP: `GET https://plex.home.keithjackson.dev/identity`
+# returned HTTP 200 with `claimed="1"` and the real machineIdentifier, and the protected endpoints
+# (/library/sections, /status/sessions, /:/prefs) all returned 401. The L7 path WORKS.
+#
+# The problem is DISCOVERY, not proxying. Plex apps (phone, TV, Roku) do not resolve
+# `plex.${BASE_DOMAIN}`; they ask plex.tv for the server's connection list. A server gets onto that
+# list one of two ways: it advertises a custom URL (`customConnections`, the ADVERTISE_IP env), or it
+# publishes a reachable public IP:port for its own tcp/32400. Measured in the live Preferences.xml on
+# 2026-10-03: `customConnections` ABSENT, `ManualPortMappingMode` ABSENT, while
+# `PublishServerOnPlexOnlineKey="1"` — the server was trying to publish and had nothing reachable to
+# publish, so remote clients saw NO connection at all. The operator chose the NATIVE path (this
+# file) over advertising the Gateway URL: Plex then terminates its own TLS with its
+# `*.plex.direct` certificate and video streams bypass the Envoy L7 hop entirely.
+#
+# VIP: ${PLEX_L4_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.57, prod 192.168.2.24), PINNED via
+# lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
+# (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the existing
+# CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config is needed;
+# this Service just consumes the pool. PINNING IS NOT OPTIONAL: the address is typed into a WAN
+# port-forward on the SOHO router, which is hand-maintained and off-repo, so it can never move.
+# 192.168.2.24 verified FREE live 2026-10-03 — `default-pool` reported IPsTotal 11 / IPsUsed 4 /
+# IPsAvailable 7, and the only four allocations were .20 (cilium-gateway-shared), .21 (unifi-l4-*),
+# .22 (crafty-l4-*) and .23 (pihole-dns-*).
+#
+# ONE SERVICE, TCP ONLY. Unlike unifi/crafty/pihole-dns this needs no second Service and no
+# lbipam.cilium.io/sharing-key: Plex's remote access is tcp/32400 and nothing else. (Plex's
+# DLNA/GDM discovery uses UDP, but that is LAN-only multicast and is not part of remote access.)
+#
+# NO `loadBalancerSourceRanges`, DELIBERATELY — and this is the one place this file diverges from
+# its closest sibling. apps/wave-0/pihole-dns.yaml pins its source range to ${LAN_CIDR} because a
+# resolver must never answer the internet. This Service is the OPPOSITE case: admitting remote
+# clients from arbitrary internet addresses is its entire purpose, so there is no range to narrow
+# to. Plex authenticates every caller itself (measured above: 401 on every protected endpoint
+# without a token), which is what makes an unfiltered port defensible here.
+#
+# NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
+# `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their VIP as a public
+# Cloudflare record. This Service must not, for pihole-dns's reason: ${PLEX_L4_VIP} is an RFC1918
+# address that is useless in public DNS. Remote clients get this server's address from plex.tv, not
+# from the zone. The existing public `plex.${BASE_DOMAIN}` record is published from the rendered
+# HTTPRoute via the gateway source and is NOT affected by this file — the web UI keeps working
+# through the Gateway exactly as before.
+#
+# -------------------------------------------------------------------------------------------------
+# externalTrafficPolicy: Cluster — inherited from crafty's 8-DAY LAN OUTAGE, not re-derived.
+# -------------------------------------------------------------------------------------------------
+# `Local` is the tempting choice because it preserves the client source IP, which this service
+# actually has a use for (see the allowedNetworks note below). It is still WRONG here, for the
+# reason platform/crafty/service-l4.yaml documents at length: Cilium's L2 announcer elects ONE lease
+# holder per Service and does NOT follow the backend pod, so with `Local` the VIP can be announced
+# by a node that has no Plex pod, which black-holes the port while every object still reports
+# healthy. Plex is a single-replica Deployment with `strategy: Recreate` pinned to wherever the
+# scheduler put it, so the mismatch is the normal case, not a rare one. Take the SNAT.
+#
+# KNOWN CONSEQUENCE OF THAT CHOICE, stated rather than discovered later: `Cluster` SNATs the client
+# to the ingress NODE's address (192.168.2.1x), which falls inside the `192.168.0.0/16` entry of
+# catalog/plex.yaml's ALLOWED_NETWORKS. Plex will therefore classify every REMOTE client as LAN.
+# Measured 2026-10-03 on the Gateway path, which SNATs the same way: Plex logged the request as
+# `[10.0.2.214:36040 (Allowed Network (WAN))]`. This is a METERING effect, not a security one —
+# authorization is unchanged (401s above), but Plex's remote-quality/bandwidth limits will not bind
+# on these streams because it believes they are local. Narrowing ALLOWED_NETWORKS cannot fix it
+# without also de-classifying genuine LAN clients, so it is accepted.
+apiVersion: v1
+kind: Service
+metadata:
+  name: plex-l4-tcp
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+  annotations:
+    lbipam.cilium.io/ips: "${PLEX_L4_VIP}"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Cluster
+  selector:
+    app.kubernetes.io/name: plex
+  ports:
+    - name: plex-remote
+      port: 32400
+      targetPort: 32400
+      protocol: TCP
+---
+# INGRESS allow for the raw port (R-NET-01/02).
+#
+# The rendered plex.yaml already carries `plex-allow-gateway-ingress` (fromEntities: ingress ->
+# 32400) and three reciprocal allows for sonarr/radarr/seerr. CiliumNetworkPolicies are ADDITIVE, so
+# this file adds the off-cluster case without touching those.
+#
+# `world` is Cilium's entity for anything outside the cluster, which is what a remote Plex client is.
+# `host` and `cluster` mirror platform/crafty/networkpolicy.yaml's game-port rule and are what make
+# this work under `externalTrafficPolicy: Cluster`: after SNAT the packet's source is the ingress
+# NODE, which Cilium classifies as `host` (same node) or `remote-node` (a different one) — NOT
+# `world`. Listing `world` alone would compile cleanly and then drop every remote stream, so all
+# four entities are named.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: plex-allow-remote-32400
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: plex
+  ingress:
+    - fromEntities:
+        - world
+        - cluster
+        - host
+        - remote-node
+      toPorts:
+        - ports:
+            - { port: "32400", protocol: TCP }
@@ -137,6 +137,8 @@ spec:
            secretKeyRef:
              name: plex-claim
              key: PLEX_CLAIM
+        - name: ADVERTISE_IP
+          value: ${PLEX_ADVERTISE_URL}
        resources:
          limits:
            gpu.intel.com/i915: 1
@@ -1,6 +1,8 @@
 # promtool rule-unit tests for the wave-0 per-app `<app>-down` SLI rules.
 # Manifests under test: every PrometheusRule in apps/wave-0/ — gitea, jackett, pihole, plex,
-# qbittorrent (container/Deployment) and vm-demo (KubeVirt VMI). The harness discovers this file by
+# qbittorrent (container/Deployment), pihole (container/StatefulSet — `availability.replicas: 2`, so
+# its SLI reads kube_statefulset_status_replicas_ready; see schemas/renderer-contract.md
+# `### availability`) and vm-demo (KubeVirt VMI). The harness discovers this file by
 # the SUITE convention (`<dir>/tests/*.test.yaml`) and flattens those RENDERED manifests, so these
 # tests run against the exact expressions Flux applies, never a copy.
 #
@@ -41,7 +43,7 @@ tests:
        values: '1+0x20'
      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
        values: '1+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}'
        values: '1+0x20'
      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
        values: '1+0x20'
@@ -81,7 +83,7 @@ tests:
        values: '1+0x20'
      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
        values: '1+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}'
        values: '1+0x20'
      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
        values: '0+0x20'
@@ -132,7 +134,7 @@ tests:
        values: '0+0x20'
      - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
        values: '0+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}'
        values: '0+0x20'
      - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
        values: '1+0x20'
@@ -170,10 +172,10 @@ tests:
              severity: critical
              service: pihole
              namespace: pihole
-              deployment: pihole
+              statefulset: pihole
            exp_annotations:
              summary: "pihole is down"
-              description: "pihole: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+              description: "pihole: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
      - eval_time: 10m
        alertname: qbittorrent-down
        exp_alerts:
@@ -12,6 +12,42 @@ storage:
  - name: config
    size: 2Gi
    class: longhorn           # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
+    # THE PVC WAS MOUNTED SOMEWHERE PIHOLE NEVER WRITES, AND SO HELD NOTHING AT ALL. Without an
+    # explicit mountPath the renderer defaults to /data/<name>, i.e. /data/config — but PiHole keeps
+    # every piece of its state in /etc/pihole. MEASURED on live prod 2026-10-04: `/etc/pihole` (7.1 M
+    # — gravity.db, pihole.toml, the query DB) was on the **overlay** filesystem, i.e. the
+    # container's EPHEMERAL layer, while `/dev/longhorn/pvc-849edb86...` sat mounted at
+    # /data/config holding `lost+found` and 24 K of nothing. Three consequences, all live until now:
+    #   * every restart silently discarded PiHole's whole state, re-downloading 72 k gravity domains
+    #     and regenerating the admin password hash;
+    #   * `backup.policy: daily-offcluster` below has been faithfully backing up an EMPTY volume,
+    #     which is worse than no backup because it reports success;
+    #   * it is also why phase 1's password/adlist declaration mattered more than it first appeared —
+    #     that state was not merely undeclared, it was ephemeral.
+    # This is the same class of bug catalog/plex.yaml's `config` entry documents ("Without this the
+    # PVC sat unused and Plex wrote to the container's EPHEMERAL layer").
+    #
+    # /etc/pihole IS THE IMAGE'S INTENDED MOUNT POINT, and an EMPTY volume there self-initialises.
+    # Verified in the running image, not assumed:
+    #   * `/usr/bin/start.sh` (PID 1) calls `migrate_gravity`, whose `migrate_to_database()` creates
+    #     the gravity database when `[ ! -e "$gravityDBfile" ]` — the same branch the postStart hook
+    #     below relies on, and the branch that produced the current ephemeral copy.
+    #   * FTL writes `pihole.toml` itself from the `FTLCONF_*` env above (that is what the
+    #     `### CHANGED (env)` markers in it are).
+    #   * NO `podSecurityContext.fsGroup` is needed and adding one would be inert: `start.sh` line 5
+    #     is `chown -R pihole:pihole /etc/pihole/ /var/log/pihole/` and PID 1 runs as **uid 0**, so
+    #     the image fixes ownership of a fresh root-owned volume before anything else starts. Same
+    #     arrangement as plex's image managing its own /config ownership.
+    #   * the image's own default for `files.macvendor` was moved OUT of this directory (to
+    #     `/macvendor.db`), which is what a maintainer does to a path they expect to be a volume.
+    #
+    # WHY IT IS FIXED IN THIS BRANCH rather than left for its own: `availability.replicas: 2` renders
+    # per-replica volumeClaimTemplates, and with the wrong mountPath those would be two 2 Gi Longhorn
+    # volumes (times three Longhorn replicas each) holding nothing but `lost+found` while all state
+    # stayed ephemeral — paying the cost of persistence and getting none of it. It also REMOVES this
+    # change's main operational risk: with gravity.db persisted, a pod restart is cheap, instead of
+    # every restart re-downloading 72 k domains inside a wave that has a 5 m timeout.
+    mountPath: /etc/pihole
 dependencies: []              # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
 # ------------------------------------------------------------------------------------------------
 # LAN SPLIT-HORIZON DNS — the half of R-DNS-03 that platform/coredns/coredns-custom.yaml defers to.
@@ -100,6 +136,53 @@ env:
  # while resolution for the zone does not work. Confirmed both ways against the live image.
  - name: FTLCONF_misc_dnsmasq_lines
    value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}"
+  # THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
+  # apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
+  # carries the full rationale and the measurement that found it.
+  #
+  # `webserver.api.password` takes CLEARTEXT and FTL hashes it itself into `webserver.api.pwhash`.
+  # Verified in the running image, not assumed: `pihole setpassword` is itself
+  # `setFTLConfigValue "webserver.api.password" "$PASSWORD"`, commented upstream as "pihole-FTL will
+  # automatically hash the password". So the BALLOON-SHA256 hash on the PVC is now a DERIVED artefact
+  # and this is its source of truth. Rotating the Secret rotates the password.
+  - name: FTLCONF_webserver_api_password
+    valueFrom:
+      secretKeyRef:
+        name: pihole-bootstrap
+        key: webui-password
+availability:
+  # TWO REPLICAS, ON DIFFERENT NODES. LAN DNS is a critical network service and it was ONE pod on
+  # ONE node. Measured 2026-10-04, that cost more than "until it reschedules":
+  #   * the node running the pod dies -> ~40s to mark it NotReady, then `tolerationSeconds: 300`
+  #     before the pod is even evicted, then reschedule, then Longhorn must force-detach the RWO
+  #     volume from a dead node and reattach it. Roughly 6-10 MINUTES with no LAN resolution.
+  #   * the node holding the `pihole-dns-*` L2 lease dies -> Cilium re-elects an announcer, tens of
+  #     seconds. (These were DIFFERENT nodes: lease on k3s-prod-2, pod on k3s-prod-1. Two of three
+  #     nodes were single points of failure for LAN DNS, with wildly different blast radii.)
+  # The Services already tolerate this: `externalTrafficPolicy: Cluster` on the DNS LoadBalancers
+  # (apps/wave-0/pihole-dns.yaml) means the L2 lease holder forwards to ANY ready backend, so a
+  # second replica is picked up with no change there.
+  #
+  # WHAT THIS DOES NOT FIX, and it is the bigger risk: LAN DNS still depends on the CLUSTER. etcd
+  # quorum loss, a Cilium regression, a Longhorn problem or a bad reconcile takes DNS down
+  # everywhere regardless of replica count. The only thing that removes that dependency is a
+  # resolver outside the cluster carrying the same `address=` lines. Considered and not chosen
+  # (operator decision, 2026-10-04); see decisions/2026-10-04-pihole-multi-replica.md.
+  #
+  # THE PREREQUISITE WAS DONE FIRST, and this field would have been actively harmful without it. A
+  # replica gets a FRESH volume, and until 2026-10-04 PiHole's admin password and adlists lived only
+  # on the PVC — so replica 1 would have come up SERVING DNS AND BLOCKING NOTHING, with a different
+  # password, and answers would have varied by which pod replied. Both are now declared (the
+  # password as a SOPS Secret -> FTLCONF_webserver_api_password; the adlists reconciled into
+  # gravity.db by the postStart hook below), which is what makes a fresh replica identical to its
+  # sibling. See decisions/2026-10-04-pihole-declare-state.md.
+  #
+  # WHY 2 AND NOT 3. Two tolerates the loss of any single node, which is the stated goal, and leaves
+  # a spare node for the evicted replica to land on under the HARD anti-affinity the renderer emits.
+  # Three would pin one replica per node with no spare, so a single node failure would leave a
+  # replica permanently Pending and — `apps-wave-0` being `wait: true` — stall the app tier for as
+  # long as the node was down. Two is the count that degrades gracefully here.
+  replicas: 2
 ingressRoutes:
  - host: pihole.${BASE_DOMAIN}
    port: 80                  # admin web UI, protected at the Gateway by forward-auth (docs/09)
@@ -114,6 +197,110 @@ auth:
  # rather than a renderer constant: the measurement that justifies the flag on two apps is
  # evidence about those two apps. Its oauth2-proxy log holds 0 unauthenticated requests across
  # the current pod's whole 23h lifetime (measured 2026-10-01), so there is nothing to act on.
+lifecycle:
+  # LEVEL-TRIGGERED ADLIST ASSERTION, run on EVERY container start. Same pattern, and the same
+  # reasoning, as catalog/qbittorrent.yaml's postStart: desired state lives in git and is
+  # RE-ASSERTED unconditionally rather than set once by hand.
+  #
+  # WHY THIS IS NEEDED AT ALL — the blocklists were NOT reproducible from git. Measured on live prod
+  # 2026-10-04: `gravity.db` held exactly ONE adlist (StevenBlack's hosts, enabled) and 72,234
+  # gravity domains, and NOTHING in this repo named it. Worse, a FRESH database does not seed it:
+  # the image's own schema template `/etc/.pihole/advanced/Templates/gravity.db.sql` creates the
+  # `adlist` table and inserts NO rows into it (grepped in the running image — there is no
+  # `INSERT INTO adlist` at all). So a rebuild from git, or a replica with a new PVC, would come up
+  # SERVING DNS AND BLOCKING NOTHING, silently and with every object reporting healthy.
+  #
+  # WHY A postStart HOOK AND NOT A FILE. PiHole v6 keeps adlists as DB rows, not config. The
+  # `/etc/pihole/adlists.list` file still exists in `gravity.sh` but is MIGRATION-ONLY: its import
+  # sits inside `migrate_to_database()`, guarded by `if [ ! -e "$gravityDBfile" ]`, so it fires only
+  # while CREATING a database and never again on an existing one. A ConfigMap mounted there would
+  # therefore be silently ignored on every pod this cluster currently runs. There is also no
+  # `pihole adlist` CLI subcommand (only allow/deny for domains), so SQL against the DB is the
+  # actual interface. FTL ships an embedded shell (`pihole-FTL sqlite3`, SQLite 3.53.1) so no extra
+  # tooling is needed in the image.
+  #
+  # IT RECONCILES, IT DOES NOT JUST ADD: adlists present in the DB but absent from DECLARED below are
+  # DISABLED (not deleted — deleting cascades to `gravity` rows and would force a full re-download of
+  # everything on a typo). So this list is the source of truth in both directions, and `pihole -g`
+  # runs ONLY when something actually changed, which keeps the steady-state start free of a 70k-domain
+  # rebuild.
+  #
+  # NO BRACED DOLLAR EXPANSION ANYWHERE IN THIS SCRIPT, DELIBERATELY — the same contract
+  # catalog/qbittorrent.yaml documents at length. apps-wave-0 declares postBuild.substituteFrom
+  # cluster-vars, so Flux runs envsubst over this whole manifest at reconcile time; a `${VAR}` form
+  # would behave differently on the cluster than under a plain `kubectl apply` of the golden. Bare
+  # `$VAR` and `$(cmd)` are untouched by envsubst and so behave identically however this is run.
+  # `render.py`'s assert_substitution_safe refuses to emit a manifest that breaks this.
+  postStart:
+    exec:
+      command:
+        - /bin/bash
+        - -c
+        - |
+          # Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and
+          # does NOT run a gravity rebuild.
+          set -u
+          GRAVITY=/etc/pihole/gravity.db
+
+          # THE DECLARED SET — one URL per line, the source of truth for what PiHole blocks.
+          # To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.
+          DECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'
+
+          # On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,
+          # and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.
+          i=0
+          while [ ! -s "$GRAVITY" ] && [ "$i" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done
+          if [ ! -s "$GRAVITY" ]; then
+            echo "pihole-adlists: gravity.db absent after 180s - leaving adlists untouched" >&2
+            exit 0
+          fi
+
+          sql() { pihole-FTL sqlite3 "$GRAVITY" "$1"; }
+          CHANGED=0
+
+          # 1) Every declared URL must be present AND enabled.
+          echo "$DECLARED" | while IFS= read -r url; do
+            [ -n "$url" ] || continue
+            printf '%s\n' "$url"
+          done > /tmp/declared.txt
+          while IFS= read -r url; do
+            [ -n "$url" ] || continue
+            # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.
+            have=$(sql "select count(*) from adlist where address='$url';")
+            if [ "$have" = "0" ]; then
+              echo "pihole-adlists: adding $url"
+              sql "insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');"
+              CHANGED=1
+            else
+              en=$(sql "select enabled from adlist where address='$url';")
+              if [ "$en" != "1" ]; then
+                echo "pihole-adlists: re-enabling $url"
+                sql "update adlist set enabled=1 where address='$url';"
+                CHANGED=1
+              fi
+            fi
+          done < /tmp/declared.txt
+
+          # 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).
+          sql "select address from adlist where enabled=1;" | while IFS= read -r addr; do
+            [ -n "$addr" ] || continue
+            if ! grep -qxF "$addr" /tmp/declared.txt; then
+              echo "pihole-adlists: disabling undeclared $addr"
+              sql "update adlist set enabled=0 where address='$addr';"
+              echo changed >> /tmp/adlist-changed
+            fi
+          done
+          [ -s /tmp/adlist-changed ] && CHANGED=1
+          rm -f /tmp/adlist-changed /tmp/declared.txt
+
+          # 3) Rebuild gravity ONLY if the declared set actually moved.
+          if [ "$CHANGED" = "1" ]; then
+            echo "pihole-adlists: declared set changed - running gravity update"
+            pihole -g || echo "pihole-adlists: gravity update failed; existing gravity.db retained" >&2
+          else
+            echo "pihole-adlists: already matches the declared set - nothing to do"
+          fi
+          exit 0
 update:
  channel: stable             # R-UPD-01
  range: ">=2024.0.0"
@@ -72,14 +72,54 @@ env:
      secretKeyRef:
        name: plex-claim
        key: PLEX_CLAIM
-  # ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
-  # which URL to advertise to plex.tv for REMOTE clients. The only candidate is
-  # https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
-  # Gateway VIP (conformance/tests/l2-behavioral/dns-internal-resolution), and both its public
-  # reachability and whether the TLS-terminating Cilium Gateway proxies Plex's 32400 API correctly for
-  # remote access are UNVERIFIED. Advertising a wrong/internal-only URL breaks remote access for ALL
-  # clients, so per "prefer safety over completeness" it is left unset. FOLLOW-UP: once remote
-  # reachability through the Gateway is confirmed, set ADVERTISE_IP: "https://plex.home.keithjackson.dev:443".
+  # ADVERTISE_IP ("Custom server access URLs") — NOW SET. This replaces a DEFERRAL note that stood
+  # here until 2026-10-04, and the deferral's premise has changed: it weighed only ONE candidate,
+  # `https://plex.home.keithjackson.dev` through the L7 Gateway. The operator chose the NATIVE L4
+  # path instead on 2026-10-03 (apps/wave-0/plex-remote.yaml) so streams go straight to
+  # ${PLEX_L4_VIP}:32400 and bypass the Envoy hop. What was still missing was a URL whose TLS
+  # CERTIFICATE validates — and getting that wrong is what took the service down.
+  #
+  # THE VALUE IS A `plex.direct` NAME, NOT THE RAW VIP, AND THAT IS THE WHOLE POINT. Plex serves
+  # 32400 with its own certificate; measured live 2026-10-04 it carries exactly one name:
+  #     CN = *.<serverhash>.plex.direct      (one SAN, no IP SANs at all)
+  # so `https://<ip>:32400` can NEVER validate. `<dashed-ip>.<serverhash>.plex.direct` is Plex's own
+  # mechanism for this: it is PUBLIC DNS that resolves to the embedded address, so the name points at
+  # the LAN VIP while the wildcard cert above matches it. Measured 2026-10-04 with FULL verification
+  # (no `curl -k`):
+  #     https://192-168-2-24.<hash>.plex.direct:32400/identity -> 200, TLS verify OK
+  #     https://192.168.2.24:32400/identity                    -> 000, cert validation FAILED
+  #
+  # THIS FIELD EXISTS BECAUSE AUTO-DISCOVERY CANNOT WORK HERE. Left empty, Plex publishes the
+  # addresses it can see itself — which from a pod is the overlay IP (`10.0.2.136`), useless to a LAN
+  # client. The VIP has to be advertised explicitly; there is no "just clear it" fix.
+  #
+  # WHAT WENT WRONG, recorded so it is not re-introduced by hand. customConnections OVERRIDES Plex's
+  # own publishing, and on 2026-10-04 it held two hand-typed URLs, both uncompletable:
+  #   - `https://192.168.2.24:32400`              -> cert cannot cover an IP. Measured: a storm of
+  #     `CERT: incomplete TLS handshake ... sslv3 alert certificate unknown` in Plex's log every few
+  #     seconds, from REAL LAN clients arriving through the VIP.
+  #   - `https://plex.home.keithjackson.dev:32400` -> nothing listens. The Gateway serves this host on
+  #     443; measured `http=000` against 192.168.2.20:32400.
+  # With every advertised connection dead, plex.tv had nothing publishable, sat at
+  # `mapping state 'Mapped - Not Published'`, fell back to Plex Relay, and THE SERVER DISAPPEARED
+  # FROM THE ACCOUNT — clients reported "not authorized".
+  #
+  # DECLARING IT HERE rather than in the web UI is the durable half of the fix. The image's
+  # /etc/cont-init.d/40-plex-first-run runs `setPref "customConnections" "${ADVERTISE_IP}"` (line 127)
+  # OUTSIDE the PLEX_CLAIM/PlexOnlineToken guard, and its `/.firstRunComplete` marker sits on the
+  # container's OVERLAY fs, not the /config PVC — so this is re-applied on EVERY pod start and git,
+  # not a form field, owns the value. Editing this env rolls the Deployment, which is what applies it.
+  #
+  # TWO LIMITS, stated rather than discovered later:
+  #   - THE HASH IS SERVER IDENTITY, which is why the whole URL is a cluster-var and not built from
+  #     ${PLEX_L4_VIP}: envsubst cannot dash an IP, and the hash is not derivable from anything in
+  #     this tree. It is not a secret (it ships in the cert and in the published connection list), but
+  #     if Plex reissues under a new hash this URL goes stale and must be re-read from the live cert.
+  #   - THIS RESTORES LAN CLIENTS AND THE ACCOUNT LISTING, NOT WAN. Remote clients still need the
+  #     off-repo tcp/32400 port-forward (runbooks/plex-remote-access.md). Once that exists, the WAN
+  #     twin of this URL can be appended comma-separated — customConnections takes a list.
+  - name: ADVERTISE_IP
+    value: ${PLEX_ADVERTISE_URL}
 # HARDWARE TRANSCODING — the host iGPU's render device (schema 1.8.0 `hardware`; R-PLAT-02).
 # INTENT ONLY, AND THE MECHANISM DELIBERATELY DOES NOT LIVE HERE. This says "this workload needs an
 # Intel Quick Sync render device"; it does NOT name a device path and it does NOT name a resource.
@@ -272,7 +272,15 @@ def apply_defaults(sd: dict) -> dict:
    # object that actually runs on the cluster is not the one this SD renders (keycloak: the
    # Keycloak operator runs a StatefulSet, so a Deployment-shaped rule would `absent()` forever).
    lw = dict(obs.get("livenessWorkload") or {})
-    lw.setdefault("kind", "VirtualMachineInstance" if s["kind"] == "vm" else "Deployment")
+    # The default must track what is ACTUALLY rendered, or the `up` SLI silently dies: a
+    # Deployment-shaped rule against a StatefulSet `absent()`s forever, which is an alert that can
+    # never fire — the exact failure mode the keycloak override was added for. Deriving it from
+    # `availability.replicas` means bumping the replica count cannot leave a stale rule behind.
+    lw.setdefault(
+        "kind",
+        "VirtualMachineInstance" if s["kind"] == "vm"
+        else ("StatefulSet" if _is_statefulset(s) else "Deployment"),
+    )
    lw.setdefault("name", s["name"])
    obs["livenessWorkload"] = lw
    s["observability"] = obs
@@ -475,6 +483,37 @@ def r_bulk_pvs(s: dict) -> list:
    return out


+def _replicas(s: dict) -> int:
+    """Declared replica count (schema 1.10.0 `availability.replicas`). Default 1."""
+    return int((s.get("availability") or {}).get("replicas", 1) or 1)
+
+
+def _is_statefulset(s: dict) -> bool:
+    """True when this container workload must render a StatefulSet rather than a Deployment.
+
+    THE DECIDING CONSTRAINT IS THE VOLUME, not a preference. A longhorn storage entry renders
+    ReadWriteOnce (r_pvcs), and one RWO PVC cannot be attached by pods on two different nodes --
+    which is exactly what `availability.replicas > 1` plus the node anti-affinity below asks for.
+    A 2-replica Deployment over a shared RWO claim does not degrade, it simply never schedules the
+    second pod. A StatefulSet's volumeClaimTemplates give each replica its OWN claim, which is the
+    only shape that satisfies both at once.
+
+    A replicas-1 service keeps rendering a Deployment so every pre-existing golden is byte-identical.
+    """
+    return s["kind"] == "container" and _replicas(s) > 1
+
+
+def _sts_claim_volumes(s: dict) -> list:
+    """Storage entries that become per-replica volumeClaimTemplates on a StatefulSet.
+
+    Only the dynamically-provisioned (longhorn) entries. A `bulk` entry is a STATIC, pre-bound NFS
+    PV over data that already exists outside the cluster and is shared read-only by every consumer
+    (r_bulk_pvs) -- per-replica copies of it are neither possible nor wanted, so it stays a
+    standalone PVC and an ordinary pod volume.
+    """
+    return [v for v in s["storage"] if v.get("class", "longhorn") != "bulk"]
+
+
 def r_pvcs(s: dict) -> list:
    """One PVC per storage entry (R-STATE-01/02/04).

@@ -482,8 +521,15 @@ def r_pvcs(s: dict) -> list:
    bulk     -> the STATIC external NFS PV rendered by r_bulk_pvs, pre-bound by volumeName.
    """
    out = []
+    # On a StatefulSet the longhorn claims are NOT standalone objects — they are
+    # volumeClaimTemplates, so each replica gets its own (see _is_statefulset). Emitting both would
+    # leave an orphan PVC that nothing mounts and that `prune` would later delete out from under
+    # nobody. `bulk` is unaffected and still rendered here.
+    sts = _is_statefulset(s)
    for vol in s["storage"]:
        cls = vol.get("class", "longhorn")
+        if sts and cls != "bulk":
+            continue
        if cls == "bulk":
            # Mirror image of the PV: the same explicitly-empty class (so nothing dynamically
            # provisions), the same access mode, and volumeName completing the two-sided binding.
@@ -564,6 +610,12 @@ def r_deployment(s: dict) -> dict:
        if vol.get("readOnly"):
            mount["readOnly"] = True
        volume_mounts.append(mount)
+        # On a StatefulSet a longhorn volume is supplied by its volumeClaimTemplate, whose
+        # metadata.name IS the pod volume name. Adding it to pod.volumes as well is a conflict
+        # (duplicate volume name), so it is skipped here and emitted below. The MOUNT is unchanged —
+        # the container cannot tell the difference, which is what keeps this transparent to the app.
+        if _is_statefulset(s) and vol.get("class", "longhorn") != "bulk":
+            continue
        volumes.append(
            {
                "name": vol["name"],
@@ -661,20 +713,80 @@ def r_deployment(s: dict) -> dict:
    if ann:
        tmpl_meta["annotations"] = ann

-    spec = {"replicas": 1}
-    # An app with persistent storage mounts a ReadWriteOnce Longhorn PVC. The default
-    # RollingUpdate strategy tries to start the NEW pod before terminating the old one, but
-    # the old pod still holds the RWO volume -> the new pod is stuck ContainerCreating and
-    # the rollout deadlocks (single-replica app can never hand the volume over). Recreate
-    # terminates the old pod first, freeing the PVC for the new one (R-STATE-01/02).
-    if s["storage"]:
+    sts = _is_statefulset(s)
+
+    # NODE ANTI-AFFINITY, and it is REQUIRED rather than preferred. `availability.replicas > 1`
+    # exists to survive the loss of one node; `preferred` would let both replicas land on the same
+    # node and quietly deliver none of that, with every object reporting healthy. The cost is real
+    # and is accepted: with N replicas the workload needs N schedulable nodes, so on a 3-node
+    # cluster losing TWO nodes leaves a replica Pending and — because `apps-wave-0` is `wait: true`
+    # with `apps-wave-1` dependsOn it — STALLS THE APP TIER. That coincides with having already lost
+    # etcd quorum on a 3-node control plane, i.e. the cluster is not reconciling anything at that
+    # point anyway, so the stall is a symptom rather than a new failure.
+    if sts:
+        pod_spec["affinity"] = {
+            "podAntiAffinity": {
+                "requiredDuringSchedulingIgnoredDuringExecution": [
+                    {
+                        "labelSelector": {
+                            "matchLabels": {"app.kubernetes.io/name": s["name"]}
+                        },
+                        "topologyKey": "kubernetes.io/hostname",
+                    }
+                ]
+            }
+        }
+
+    spec = {"replicas": _replicas(s)}
+    if sts:
+        # serviceName is a required StatefulSet field. It points at the service rendered by
+        # r_service; that Service is a normal ClusterIP rather than headless, so per-pod DNS
+        # (<pod>.<svc>) does NOT resolve — and nothing here needs it. These replicas are
+        # interchangeable resolvers behind one VIP, not peers that address each other.
+        spec["serviceName"] = s["name"]
+        # Parallel, not the OrderedReady default. OrderedReady starts replica N+1 only after N is
+        # Ready, so a cold start costs the sum of every replica's first-boot work instead of the
+        # max — and `apps-wave-0` has `timeout: 5m`. There is no ordering requirement between
+        # replicas to protect (no leader, no migration, no shared volume), so serialising buys
+        # nothing and risks the wave timing out on a rebuild.
+        spec["podManagementPolicy"] = "Parallel"
+    elif s["storage"]:
+        # An app with persistent storage mounts a ReadWriteOnce Longhorn PVC. The default
+        # RollingUpdate strategy tries to start the NEW pod before terminating the old one, but
+        # the old pod still holds the RWO volume -> the new pod is stuck ContainerCreating and
+        # the rollout deadlocks (single-replica app can never hand the volume over). Recreate
+        # terminates the old pod first, freeing the PVC for the new one (R-STATE-01/02).
+        #
+        # A StatefulSet needs none of this and cannot express it: each replica owns its own claim,
+        # so a rolling update hands nothing over, and `strategy` is not a StatefulSet field at all.
        spec["strategy"] = {"type": "Recreate"}
    spec["selector"] = {"matchLabels": {"app.kubernetes.io/name": s["name"]}}
    spec["template"] = {"metadata": tmpl_meta, "spec": pod_spec}
+    if sts:
+        # Per-replica claims. metadata.name MUST equal the pod volume name the container mounts
+        # (see the volumes loop above); the resulting PVCs are <name>-<sts>-<ordinal>, e.g.
+        # config-pihole-0. Same tier and access mode a standalone longhorn PVC gets from r_pvcs —
+        # this changes WHO owns the claim, never what it is.
+        tmpls = []
+        for vol in _sts_claim_volumes(s):
+            vspec = {"accessModes": ["ReadWriteOnce"], "storageClassName": "longhorn"}
+            if vol.get("size"):
+                vspec["resources"] = {"requests": {"storage": vol["size"]}}
+            tmpls.append(
+                {
+                    "metadata": {
+                        "name": vol["name"],
+                        "labels": {"app.kubernetes.io/name": s["name"]},
+                    },
+                    "spec": vspec,
+                }
+            )
+        if tmpls:
+            spec["volumeClaimTemplates"] = tmpls

    return {
        "apiVersion": "apps/v1",
-        "kind": "Deployment",
+        "kind": "StatefulSet" if sts else "Deployment",
        "metadata": {
            "name": s["name"],
            "namespace": s["name"],
@@ -684,6 +796,35 @@ def r_deployment(s: dict) -> dict:
    }


+def r_pdb(s: dict):
+    """PodDisruptionBudget for a multi-replica workload (availability.replicas > 1).
+
+    minAvailable 1, not `maxUnavailable: 1`. The budget's job here is to stop a VOLUNTARY
+    disruption — a node drain for maintenance, a cluster upgrade — from taking the last replica
+    with it; `minAvailable: 1` says that whatever else happens, one must keep serving. It does
+    NOT protect against involuntary disruption (a node dying), which is what the anti-affinity
+    and the replica count are for.
+
+    Rendered only when replicas > 1: a PDB over a single-replica workload is actively harmful, as
+    `minAvailable: 1` would then block every drain of its node forever.
+    """
+    if not _is_statefulset(s):
+        return None
+    return {
+        "apiVersion": "policy/v1",
+        "kind": "PodDisruptionBudget",
+        "metadata": {
+            "name": s["name"],
+            "namespace": s["name"],
+            "labels": {"app.kubernetes.io/name": s["name"]},
+        },
+        "spec": {
+            "minAvailable": 1,
+            "selector": {"matchLabels": {"app.kubernetes.io/name": s["name"]}},
+        },
+    }
+
+
 def r_service(s: dict):
    ports = _service_ports(s)
    # A declared metrics endpoint becomes a NAMED Service port, because that name is the only
@@ -1817,7 +1958,13 @@ def render(sd: dict, image_automation: bool = True, inbound: list | None = None)
            )
        for pvc in r_pvcs(s):
            docs.append((f"PVC {pvc['metadata']['name']} (R-STATE)", pvc))
-        docs.append(("Deployment (R-PLAT-02)", r_deployment(s)))
+        wl = r_deployment(s)
+        docs.append((f"{wl['kind']} (R-PLAT-02)", wl))
+        pdb = r_pdb(s)
+        if pdb:
+            docs.append(
+                ("PodDisruptionBudget — keep one replica through a drain (availability.replicas)", pdb)
+            )
        svc = r_service(s)
        if svc:
            docs.append(("Service", svc))
@@ -2002,6 +2149,24 @@ EXTRA_UNITS: dict = {
            "pihole-dns.yaml",
        ],
    },
+    # Plex's remote-access exposure: the raw tcp/32400 LoadBalancer Service on a pinned VIP, plus the
+    # CiliumNetworkPolicy that admits off-cluster clients to it (apps/wave-0/plex-remote.yaml).
+    # `above: False` co-locates it in plex's OWN wave — these are the plex pod's own Service and its
+    # own policy, applied in the same wave as the Deployment they select, exactly as the rendered
+    # Service and CNPs in plex.yaml are. Nothing in a higher wave gates on it.
+    #
+    # It is a non-catalog unit for the same reason pihole-dns is: the ServiceDefinition schema cannot
+    # express a LoadBalancer Service or a pinned lbipam VIP (additionalProperties is false, and
+    # `ingressRoutes` only renders HTTPRoutes on the L7 Gateway, which cannot carry Plex's raw port).
+    # Plex's own settings stay in the catalog as env on catalog/plex.yaml; the only thing living
+    # outside it is the exposure the schema has no word for.
+    "plex-remote": {
+        "anchor": "plex",
+        "above": False,
+        "files": [
+            "plex-remote.yaml",
+        ],
+    },
 }


@@ -2248,6 +2413,132 @@ def _selftest() -> int:
    check("Namespace" in kinds, "container: missing Namespace")
    check("NetworkPolicy" in kinds, "container: missing default-deny NetworkPolicy")
    check("Deployment" in kinds, "container: missing Deployment")
+    check(
+        "PodDisruptionBudget" not in kinds,
+        "replicas-1 container must render NO PodDisruptionBudget (minAvailable 1 over a single "
+        "replica blocks every drain of its node forever)",
+    )
+
+    # 1b) availability.replicas (schema 1.10.0). The DEFAULT path must be untouched, and the >1
+    # path must change workload kind, claim ownership, spreading, budget AND the SLI metric
+    # together — each of these has a silent-failure mode if it is the one that gets forgotten.
+    base_av = {
+        "name": "av",
+        "source": {"image": "nginx:1"},
+        "storage": [{"name": "config", "size": "1Gi"}],
+        "observability": {"metrics": True, "slis": ["up"]},
+    }
+    one = render({**base_av, "availability": {"replicas": 1}})
+    one_kinds = [d[1]["kind"] for d in one]
+    check(
+        "Deployment" in one_kinds and "StatefulSet" not in one_kinds,
+        "availability.replicas=1 must still render a Deployment (back-compat)",
+    )
+    check(
+        "PersistentVolumeClaim" in one_kinds,
+        "availability.replicas=1 must still render a standalone PVC",
+    )
+    check(
+        "PodDisruptionBudget" not in one_kinds,
+        "availability.replicas=1 must render no PodDisruptionBudget",
+    )
+
+    many = render({**base_av, "availability": {"replicas": 2}})
+    many_kinds = [d[1]["kind"] for d in many]
+    check(
+        "StatefulSet" in many_kinds and "Deployment" not in many_kinds,
+        "availability.replicas>1 must render a StatefulSet, not a Deployment (a shared RWO claim "
+        "cannot be attached on two nodes, so a multi-replica Deployment never schedules)",
+    )
+    check(
+        "PersistentVolumeClaim" not in many_kinds,
+        "availability.replicas>1 must NOT also render a standalone longhorn PVC — it would be an "
+        "orphan that nothing mounts",
+    )
+    sts = next(d[1] for d in many if d[1]["kind"] == "StatefulSet")
+    check(sts["spec"].get("replicas") == 2, "StatefulSet must carry the declared replica count")
+    check(
+        "strategy" not in sts["spec"],
+        "StatefulSet must not carry `strategy` (not a StatefulSet field; Recreate is a "
+        "Deployment-only workaround for handing over one RWO volume)",
+    )
+    check(
+        sts["spec"].get("podManagementPolicy") == "Parallel",
+        "StatefulSet must set podManagementPolicy=Parallel — OrderedReady serialises first-boot "
+        "work across replicas and apps-wave-0 has a 5m timeout",
+    )
+    check(sts["spec"].get("serviceName") == "av", "StatefulSet must set serviceName")
+    tmpls = sts["spec"].get("volumeClaimTemplates") or []
+    check(
+        [t["metadata"]["name"] for t in tmpls] == ["config"],
+        "StatefulSet volumeClaimTemplate name must equal the pod volume name the container mounts, "
+        "or the pod gets a duplicate/unresolvable volume",
+    )
+    check(
+        tmpls and tmpls[0]["spec"]["accessModes"] == ["ReadWriteOnce"]
+        and tmpls[0]["spec"]["storageClassName"] == "longhorn",
+        "StatefulSet claim template must keep the same tier/access mode as the standalone PVC",
+    )
+    sts_vols = {v["name"] for v in sts["spec"]["template"]["spec"].get("volumes", [])}
+    check(
+        "config" not in sts_vols,
+        "a claim-template volume must NOT also appear in pod.volumes (duplicate volume name)",
+    )
+    mounts = sts["spec"]["template"]["spec"]["containers"][0].get("volumeMounts") or []
+    check(
+        any(m["name"] == "config" for m in mounts),
+        "the container must still mount the volume by name on a StatefulSet (transparent to the app)",
+    )
+    aff = (sts["spec"]["template"]["spec"].get("affinity") or {}).get("podAntiAffinity") or {}
+    req = aff.get("requiredDuringSchedulingIgnoredDuringExecution") or []
+    check(
+        req and req[0].get("topologyKey") == "kubernetes.io/hostname",
+        "availability.replicas>1 must render REQUIRED per-node anti-affinity — `preferred` can put "
+        "every replica on one node and deliver none of the intended node-failure tolerance",
+    )
+    pdbs = [d[1] for d in many if d[1]["kind"] == "PodDisruptionBudget"]
+    check(len(pdbs) == 1, "availability.replicas>1 must render exactly one PodDisruptionBudget")
+    check(
+        pdbs and pdbs[0]["spec"].get("minAvailable") == 1,
+        "PodDisruptionBudget must be minAvailable=1 (keep one replica serving through a drain)",
+    )
+    pr_many = next((d[1] for d in many if d[1]["kind"] == "PrometheusRule"), None)
+    check(
+        pr_many is not None
+        and "kube_statefulset_status_replicas_ready"
+        in pr_many["spec"]["groups"][0]["rules"][0]["expr"],
+        "the up SLI must follow the rendered workload kind — a Deployment-shaped rule against a "
+        "StatefulSet absent()s forever and can never fire",
+    )
+
+    # A bulk volume is shared external NFS: it must stay a standalone PVC even on a StatefulSet.
+    mixed = render(
+        {
+            **base_av,
+            "availability": {"replicas": 2},
+            "storage": [
+                {"name": "config", "size": "1Gi"},
+                {"name": "media", "class": "bulk", "export": "/mnt/x", "size": "1Ti"},
+            ],
+        }
+    )
+    m_sts = next(d[1] for d in mixed if d[1]["kind"] == "StatefulSet")
+    check(
+        [t["metadata"]["name"] for t in m_sts["spec"]["volumeClaimTemplates"]] == ["config"],
+        "only longhorn volumes become claim templates; a bulk share must not be per-replica",
+    )
+    check(
+        any(
+            d[1]["kind"] == "PersistentVolumeClaim"
+            and d[1]["metadata"]["name"] == "av-media"
+            for d in mixed
+        ),
+        "a bulk PVC must still be rendered standalone on a StatefulSet",
+    )
+    check(
+        "media" in {v["name"] for v in m_sts["spec"]["template"]["spec"].get("volumes", [])},
+        "a bulk volume must still appear in pod.volumes on a StatefulSet",
+    )

    ns = next(d[1] for d in docs if d[1]["kind"] == "Namespace")
    check(
@@ -100,7 +100,8 @@ data:
  # platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
  # tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
  # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
-  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers),
+  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers;
+  # plex-remote: PLEX_L4_VIP; plex: PLEX_ADVERTISE_URL),
  # velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
  # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
  # UNSUBSTITUTED path would ship the literal string into the cluster.
@@ -175,6 +176,45 @@ data:
  # dev  -> 192.168.1.56  (base default; next free after crafty .55 in dev's .53-.60 pool)
  # prod -> 192.168.2.23  (next free after crafty .22 in prod's .20-.30 pool; see clusters/prod)
  PIHOLE_DNS_VIP: "192.168.1.56"             # PiHole: authoritative LAN DNS for ${BASE_DOMAIN}
+  # Plex's remote-access listener: raw tcp/32400 for REMOTE Plex clients (apps/wave-0/plex-remote.yaml).
+  # A FOURTH pinned L4 VIP, for the usual reason — the `shared` Gateway is an L7 HTTPS terminator and
+  # cannot carry Plex's own TLS. Plex clients do not resolve `plex.${BASE_DOMAIN}`; they fetch this
+  # server's address from plex.tv, so the Gateway's L7 path (which still serves the web UI, and was
+  # measured working) cannot put the server on a client's connection list at all.
+  #
+  # PINNED, AND THE PIN IS LOAD-BEARING OFF-REPO: the SOHO router forwards WAN tcp/32400 to this
+  # address by hand. A VIP taken from the pool head would silently move on a reschedule and break
+  # remote access with nothing in-cluster looking wrong. See runbooks/plex-remote-access.md.
+  #
+  # dev  -> 192.168.1.57  (base default; next free after pihole-dns .56 in dev's .53-.60 pool)
+  # prod -> 192.168.2.24  (next free after pihole-dns .23 in prod's .20-.30 pool; see clusters/prod)
+  PLEX_L4_VIP: "192.168.1.57"                # Plex: tcp/32400 remote access (NOT the web UI, which is L7)
+
+  # PLEX_ADVERTISE_URL — the connection URL Plex publishes to plex.tv ("Custom server access URLs",
+  # the ADVERTISE_IP env in catalog/plex.yaml, which the image writes into the `customConnections`
+  # preference on every pod start). Read catalog/plex.yaml's ADVERTISE_IP block for the full why;
+  # the short version is the only thing that matters when editing this line:
+  #
+  #   IT MUST BE A `<dashed-ip>.<serverhash>.plex.direct` NAME, NEVER A BARE IP. Plex serves 32400
+  #   with a `*.<serverhash>.plex.direct` wildcard certificate and NO IP SAN, so `https://<ip>:32400`
+  #   fails TLS validation for every client. That exact mistake is what broke this service on
+  #   2026-10-04: with no completable connection, plex.tv published nothing, the server vanished from
+  #   the account and clients reported "not authorized". The dashed name is public DNS that resolves
+  #   to the embedded address, so it points at the LAN VIP AND matches the cert.
+  #
+  # WHY IT IS A WHOLE URL AND NOT BUILT FROM ${PLEX_L4_VIP}: postBuild substitution is plain
+  # envsubst — it cannot turn 192.168.2.24 into 192-168-2-24 — and `<serverhash>` is this server's
+  # certificate identity, derivable from nothing in this tree. Re-read it from the live cert
+  # (`openssl s_client -connect <vip>:32400 | openssl x509 -noout -subject`) if Plex ever reissues.
+  #
+  # The dashed IP here MUST track ${PLEX_L4_VIP} above. They are two halves of one address and
+  # nothing checks that they agree.
+  #
+  # dev  -> PLACEHOLDER, non-functional. dev is decommissioned and has no claimed Plex server, so
+  #         there is no real hash to put here; the base default only has to be a defined, correctly
+  #         shaped string (StrictPostBuildSubstitutions makes an UNDEFINED var a hard failure).
+  # prod -> the real 192.168.2.24 + live server hash; see clusters/prod/kustomization.yaml.
+  PLEX_ADVERTISE_URL: "https://192-168-1-57.0000000000000000000000000000000f.plex.direct:32400"

  # The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
  # peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable
@@ -476,6 +476,36 @@ patches:
      - op: replace
        path: /data/PIHOLE_DNS_VIP
        value: "192.168.2.23"
+      # Plex's remote-access listener VIP — raw tcp/32400 for REMOTE Plex clients
+      # (apps/wave-0/plex-remote.yaml). 192.168.2.24 is the NEXT FREE address in the prod pool
+      # 192.168.2.20-.30, verified live 2026-10-03: the CiliumLoadBalancerIPPool `default-pool`
+      # reported IPsTotal 11 / IPsUsed 4 / IPsAvailable 7, and the only four allocations are .20
+      # (cilium-gateway-shared), .21 (unifi-l4-tcp/-udp), .22 (crafty-l4-tcp/-udp) and .23
+      # (pihole-dns-tcp/-udp). It is inside the pool block the operator already asserted sits outside
+      # the SOHO router's DHCP scope, and it is not a node address (.10/.11/.12) or the
+      # control-plane VIP (.9).
+      #
+      # UNLIKE .23, THIS ONE IS ALSO A WAN PORT-FORWARD TARGET. The router forwards WAN tcp/32400
+      # here, by hand, off-repo — so changing this value is a two-place edit and the router is the
+      # half that CI cannot check. runbooks/plex-remote-access.md is the procedure.
+      #
+      # NOT ARP-PROBED. .23 was verified free with `arping` from a LAN host before it was claimed;
+      # for .24 only the in-cluster pool accounting above was checked, so an off-cluster squatter on
+      # this address would not have been caught. Probe before trusting it if remote access comes up
+      # with a conflict.
+      - op: replace
+        path: /data/PLEX_L4_VIP
+        value: "192.168.2.24"
+      # PLEX_ADVERTISE_URL — the prod VIP above in Plex's `plex.direct` form, which is the ONLY form
+      # that validates: the server's cert is `*.130072f20c334d6c8f63d0dfef87309b.plex.direct` with no
+      # IP SAN (read off the live listener 2026-10-04), so a bare `https://192.168.2.24:32400` fails
+      # TLS for every client — the 2026-10-04 outage. Measured with full verification, no `curl -k`:
+      # this URL answers /identity with 200 and TLS verify OK; the bare-IP form fails to connect.
+      # THE DASHED IP MUST MATCH /data/PLEX_L4_VIP ABOVE (192.168.2.24 -> 192-168-2-24); nothing
+      # enforces that. The hash is server identity, not a secret, and changes only if Plex reissues.
+      - op: replace
+        path: /data/PLEX_ADVERTISE_URL
+        value: "https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400"
      # The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns —
      # NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path.
      # It scopes the ingress allow for PiHole's DNS listener to LAN clients only.
@@ -0,0 +1,76 @@
+---
+date: 2026-10-03
+branch: fix/cilium-cgroup-hostroot
+title: Cilium socket-LB never covered host processes — cgroup.hostRoot pointed at an unmounted path
+---
+
+## What changed
+
+`cgroup.hostRoot` changes from `/run/cilium/cgroupv2` to `/sys/fs/cgroup` in all three parity
+copies: `platform/cilium/helmrelease.yaml` (authoritative), `platform/cilium/values.yaml`
+(mirror) and `infra/ansible/roles/cilium/templates/cilium-values.yaml.j2` (bootstrap).
+`autoMount` stays `false`.
+
+**The defect.** With `autoMount: false`, the Cilium chart only bind-mounts whatever exists at
+`hostRoot`, and expects the operator to have mounted cgroup2 there (cilium 1.20.1 values.yaml).
+The repo's rationale was "k3s provides the cgroupv2 mount". That is true only for
+`/sys/fs/cgroup`. Nothing mounts cgroup2 at `/run/cilium/cgroupv2` on these nodes; it is an
+empty `DirectoryOrCreate` hostPath. So the agent mounted cgroup2 there itself, from inside its
+own private cgroup namespace, and that mount's root is the agent container's own leaf cgroup.
+The socket-LB programs (`cil_sock4_connect` etc.) were attached to it and covered only the
+agent.
+
+**The consequence.** Every other host-netns process got no Service/LB translation:
+- Service ClusterIPs were unreachable from every node's host.
+- The node holding the Gateway's L2 lease could not reach the Gateway VIP (`no route to host`).
+  Other nodes reached it only because their packets crossed the wire to the lease holder, where
+  the tc datapath handles them.
+- k3s-prod-1 holds both the Gateway lease and the kube-vip API VIP. Its apiserver could not reach
+  the OIDC issuer, logging `oidc: authenticator not initialized` since at least 2026-09-12
+  (899 occurrences by 2026-10-03). kubectl SSO through the VIP therefore failed, and in-cluster
+  OIDC clients failed on whichever calls landed on that apiserver.
+- Webhooks and aggregated APIs were unaffected only because k3s runs
+  `--enable-aggregator-routing=true` (endpoint IPs, not ClusterIPs).
+
+Cilium's own source (v1.20.1 `bpf/bpf_sock.c`, `__sock4_xlate_fwd`) handles exactly this case
+when the hook does run: a host-netns connect to an L7-LB Service (the Gateway) is rewritten to
+`127.0.0.1:<l7_lb_proxy_port>`, the local Envoy listener. The fix is to make the hook cover the
+host.
+
+## What I verified
+
+- `cilium-dbg status --verbose` on k3s-prod-1: `Socket LB: Enabled`, `Coverage: Hostns-only`.
+  The live cilium-config shows `bpf-lb-sock=false`, but socket LB is implied by kube-proxy
+  replacement. The Gateway Service frontends `192.168.2.20:{80,443}` and
+  `10.43.225.79:{80,443}` map to `127.0.0.1:14337`.
+- Host: `findmnt /run/cilium/cgroupv2` is empty on all three nodes, and `/sys/fs/cgroup` is
+  `cgroup2`.
+- In the agent: `/run/cilium/cgroupv2` is a `cgroup2 none` mount, the agent's own cgroup reads
+  `0::/`, the mount has 0 child cgroups, and `bpftool cgroup show /run/cilium/cgroupv2` lists all
+  `cil_sock*` programs. k3s runs in `0::/system.slice/k3s.service`, and the agent container in
+  `/kubepods.slice/…/cri-containerd-3ad04cde….scope`.
+- **The decisive test**, run on k3s-prod-1 in the same netns: raw TCP connect to
+  `192.168.2.20:443`, `10.43.225.79:443` and `10.43.59.141:8080` CONNECTED from inside the agent
+  container (covered cgroup) and FAILED from the host (`No route to host` / timeout).
+- Blast radius: host-network pods are cilium, cilium-envoy, cilium-operator (API via
+  `192.168.2.9:6443`, not a Service), kube-vip (`127.0.0.1:6443`, `svc_enable=false`) and
+  node-exporter (no outbound). Nothing depends on host Service translation being absent; the
+  change only makes Service/LB addresses reachable from the host where they currently fail.
+- `helm template cilium/cilium --version 1.20.1` with the new values, in the toolbox: the
+  DaemonSet bind-mounts hostPath `/sys/fs/cgroup` into `cilium-agent` and `clean-cilium-state`,
+  the cilium-config shows `cgroup-root: "/sys/fs/cgroup"`, and the init containers are unchanged
+  (no `mount-cgroup`).
+- Toolbox gate: see the PR description.
+
+## What I did NOT verify
+
+- **The rollout itself.** After merge, the agent DaemonSet rolls one node at a time
+  (`maxUnavailable: 1`). On each node, afterwards: host TCP connects to the three targets above
+  must succeed. On k3s-prod-1, the apiserver's background OIDC init retry must stop logging
+  `authenticator not initialized` with no k3s restart.
+- That the old programs, attached to the previous agent container's cgroup, disappear with it.
+  Expected, because cgroup BPF attachments die with the cgroup, but not observed.
+- An end-to-end kubectl OIDC login (needs a browser).
+- No regression test was added. A natural one is an L2 conformance check that a host-netns
+  connect to the Gateway VIP succeeds on the lease holder, but `conformance/**` is a sensitive
+  path. Left as a follow-up.
@@ -0,0 +1,45 @@
+---
+date: 2026-10-03
+branch: fix/node-keycloak-hosts-pin
+title: Pin the OIDC issuer to the Gateway VIP on prod nodes — the WAN answer does not hairpin
+---
+
+## What changed
+
+Follow-up to PR #40 (`decisions/2026-10-03-public-dns-wan-target.md`). Public DNS for
+`*.home.keithjackson.dev` now CNAMEs to the WAN IP. The prod nodes resolve through public DNS,
+and the router does not hairpin, so the kube-apiserver could no longer reach its OIDC issuer
+`keycloak.home.keithjackson.dev`.
+
+- `roles/node_baseline`: a new `node_baseline_hosts_pins` list (default `[]`) and one
+  `blockinfile` task (tags `dns`, `hosts`). It writes the pins into `/etc/hosts` as a single
+  marked block. An empty list removes the block.
+- `group_vars/k3s_prod/vars.yml`: pins `keycloak.{{ k3s_base_domain }}` → `192.168.2.20`, which
+  must stay equal to `GATEWAY_VIP` (parity comment added). It also replaces the stale "no
+  /etc/hosts pin is needed" paragraph with the measured reason the pin is now required.
+
+Dev is unaffected (empty default).
+
+## What I verified
+
+- After PR #40 reconciled (`platform` lastAppliedRevision `576e1ab`, Healthy=True), on
+  k3s-prod-2 and k3s-prod-3: `getent hosts keycloak.home.keithjackson.dev` →
+  `136.51.115.252 wan.home.keithjackson.dev`. `curl https://keycloak…/realms/homelab/.well-known/openid-configuration`
+  → exit 60, `ssl_verify_result=18` (the router's self-signed cert, not the Gateway).
+- `journalctl -u k3s` (last 20 min) showed no OIDC/JWKS errors yet. The apiserver still holds
+  the keys it cached before the flip, so the failure is latent until a k3s restart or key rotation.
+- Neither node has cloud-init (`/etc/cloud` absent), so nothing rewrites `/etc/hosts` at boot.
+- `--check --diff --tags hosts --limit k3s-prod-2,k3s-prod-3`: each node gets exactly the
+  three-line marked block `192.168.2.20	keycloak.home.keithjackson.dev`.
+- Outside-in, PR #40 works: check-host.net probes from 3 external nodes got `200 OK` from
+  `https://plex.home.keithjackson.dev/identity` at `136.51.115.252`.
+
+## What I did NOT verify
+
+- **k3s-prod-1**: Ansible cannot reach it. SSH host key verification fails (`known_hosts:94`,
+  ED25519 `SHA256:zgUS8OeWznzCX9hlN7XdZ8iLNkU1G2loXjtJFfEUYpE`), which is consistent with
+  the 2026-09-11 SSD replacement and reinstall. The operator must confirm the fingerprint
+  before the pin can be applied there.
+- The playbook has not been applied for real. It is an operator run after merge:
+  `ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`.
+- An end-to-end kubectl OIDC login after the pin (needs a browser).
@@ -0,0 +1,129 @@
+---
+date: 2026-10-03
+branch: feat/plex-remote-access-l4
+title: Plex remote access — raw tcp/32400 via a pinned LoadBalancer VIP, not the L7 Gateway
+---
+
+## What changed
+
+Reported symptom: users cannot reach Plex from outside the LAN. **Two independent causes**, measured
+rather than inferred:
+
+1. **Plex advertised no remote connection at all.** Live `Preferences.xml` on prod:
+   `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while
+   `PublishServerOnPlexOnlineKey="1"` — publishing was on and there was nothing reachable to publish.
+   Plex's automatic UPnP/NAT-PMP path cannot work from a pod on the Cilium overlay.
+2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` refused;
+   `136.51.115.252:443` answered with a **self-signed router cert** (`C=ZZ, CN=%D`), not the
+   Gateway's LE wildcard. The Plex Service was `ClusterIP` only — no off-cluster path to 32400 by
+   any route.
+
+This branch fixes the in-cluster half. **Remote access does not work until the operator also does
+two manual steps** (`runbooks/plex-remote-access.md`): the WAN tcp/32400 port-forward, and ticking
+"manually specify public port" in the Plex UI.
+
+- **`apps/wave-0/plex-remote.yaml` (new, hand-written).** A `type=LoadBalancer` Service on tcp/32400
+  pinned to `${PLEX_L4_VIP}` via `lbipam.cilium.io/ips`, plus a `CiliumNetworkPolicy`
+  (`plex-allow-remote-32400`) admitting `world`/`cluster`/`host`/`remote-node` to 32400. Non-catalog
+  because `schemas/servicedefinition.schema.json` is `additionalProperties: false` and has no field
+  for a LoadBalancer Service or a pinned VIP — the same reason `apps/wave-0/pihole-dns.yaml`,
+  `platform/crafty/service-l4.yaml` and `apps/vms/unifi/50-service-l4.yaml` are hand-written.
+- **`catalog/render.py`:** an `EXTRA_UNITS` entry (`plex-remote`, anchored to `plex`, `above: False`)
+  so `--emit-waves` places the unit in plex's own wave.
+- **`clusters/base/cluster-vars.yaml`:** new `PLEX_L4_VIP` (dev default `192.168.1.57`), plus the
+  substitution inventory comment updated to name `plex-remote: PLEX_L4_VIP`.
+- **`clusters/prod/kustomization.yaml`:** prod override `PLEX_L4_VIP: 192.168.2.24`.
+- **`apps/wave-0/kustomization.yaml`:** `plex-remote.yaml` added to `resources`.
+- **`runbooks/plex-remote-access.md` (new):** the two manual steps, the four-step verification, why
+  no alert is possible, and the two documented warts below.
+
+**`ADVERTISE_IP` was deliberately NOT set.** Setting it to `https://plex.home.keithjackson.dev:443`
+was the alternative — it needs no new WAN hole and the mechanism was verified working on an
+already-claimed server. **The operator chose the native path (2026-10-03)** so Plex terminates its own
+TLS (`*.plex.direct`) and streams bypass the Envoy L7 hop. `catalog/plex.yaml`'s deferral note for
+`ADVERTISE_IP` remains accurate and untouched.
+
+**`externalTrafficPolicy: Cluster`, with a known cost.** `Local` would preserve the client IP, but
+Cilium's L2 announcer does not follow the backend pod — the wart that cost this LAN an 8-day
+Minecraft outage (see `platform/crafty/service-l4.yaml`). Consequence accepted and documented: SNAT
+puts the client in `192.168.0.0/16`, inside `ALLOWED_NETWORKS`, so **Plex classifies remote streams
+as LAN**. That is a metering effect (remote bandwidth limits will not bind), not an authorization one.
+
+**SENSITIVE PATHS — this PR touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK
+it.** It needs the operator's `/approve-override <reason>` posted as a PR **review** of type
+"Comment". I have deliberately not posted it: CLAUDE.md §1 forbids an agent clearing this on its own
+PR, and the agent account holds `owner`, so the prohibition is a rule, not a limit.
+
+## What I verified
+
+All measurements against live prod, 2026-10-03, read-only.
+
+- **Plex's advertised state**, the finding this branch rests on: `customConnections=<ABSENT>`,
+  `ManualPortMappingMode=<ABSENT>`, `ManualPortMappingPort=<ABSENT>`,
+  `PublishServerOnPlexOnlineKey="1"`, `allowedNetworks="10.0.0.0/8,192.168.0.0/16"`. Read by
+  attribute grep, never by dumping `Preferences.xml` (it holds `PlexOnlineToken`).
+- **The L7 Gateway path works and is NOT the problem.** Via the Gateway VIP,
+  `GET https://plex.home.keithjackson.dev/identity` → `HTTP 200`, `claimed="1"`, real
+  `machineIdentifier`. HTTPRoute `plex-32400`: `Accepted=True`, `ResolvedRefs=True`.
+- **Plex enforces its own auth**, which is what makes an unfiltered WAN port defensible:
+  `/library/sections`, `/status/sessions`, `/:/prefs` all → **401** unauthenticated through the
+  Gateway.
+- **The SNAT/allowedNetworks wart is real, not theoretical.** A tagged probe through the Gateway was
+  logged by Plex as `[10.0.2.214:36040 (Allowed Network (WAN))]` — the Envoy pod IP, classified as an
+  allowed network.
+- **WAN state:** `136.51.115.252:32400` → connection refused. `136.51.115.252:443` → self-signed
+  cert `C=ZZ, ST=Somewhere, O=%D342e7e6b, CN=%D`. Public DNS resolves correctly:
+  `plex.home.keithjackson.dev` → CNAME `wan.home.keithjackson.dev` → `136.51.115.252`.
+- **`192.168.2.24` is free in-cluster.** `CiliumLoadBalancerIPPool default-pool`: IPsTotal 11 /
+  IPsUsed 4 / IPsAvailable 7; the only allocations are `.20` (cilium-gateway-shared), `.21`
+  (unifi-l4-*), `.22` (crafty-l4-*), `.23` (pihole-dns-*). Not a node address (`.10/.11/.12`) and not
+  the kube-vip control-plane VIP (`.9`).
+- **The manual-port-mapping step genuinely cannot be automated.** Grep of the running image's whole
+  init surface (`/etc/cont-init.d`, `/plex-common.sh`, `/usr/local/bin`) for `ManualPortMapping` and
+  `PLEX_PREFERENCE` → **NONE**.
+- **`ADVERTISE_IP` would have worked on a claimed server** (recorded because the runbook's
+  "why not" rests on it): `setPref "customConnections"` at line 126 of `40-plex-first-run` sits
+  **outside** the `PLEX_CLAIM`/`PlexOnlineToken` guard, and the `/.firstRunComplete` early-exit
+  marker is on the **overlay** FS, not the `/config` PVC, so the script re-runs every pod start.
+- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
+  `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
+  **pass**. The rc 3 is two environmental skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`)
+  and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3, not as pass.**
+- **`render-golden` passing is the specific proof** that the `render.py` `EXTRA_UNITS` change, the new
+  hand-written unit and the edited wave kustomization are mutually consistent.
+
+## What I did NOT verify
+
+- **THE ACTUAL FIX. Nothing in this branch has been proven to make remote access work** — it cannot
+  be, because both remaining steps are off-repo and manual (router forward, Plex UI toggle). This
+  branch removes two blockers and adds none; it does not demonstrate an outcome.
+- **Outside-in reachability, at all.** Every probe above was taken from inside the LAN, so the
+  `:443` self-signed cert strictly proves only that there is **no NAT hairpin** — it does **not**
+  prove outside-in failure. A router cert on WAN:443 is suggestive (its management UI would shadow a
+  forward), but unproven. **This repo has no off-LAN vantage point.** Test 3 in the runbook
+  (app.plex.tv from a cellular connection) is the only honest check.
+- **`192.168.2.24` was never ARP-probed off-cluster.** Only in-cluster pool accounting was checked.
+  `.23` was `arping`-probed before it was claimed; `.24` was not, so an off-cluster squatter would
+  not have been caught.
+- **The LoadBalancer has never been reconciled.** `plex-l4-tcp` does not exist in prod yet, so
+  LB-IPAM has not actually assigned the pinned VIP, L2 announcement has not been observed for it, and
+  the new CiliumNetworkPolicy has never admitted a real packet. Whether `world`/`host`/`cluster`/
+  `remote-node` is the correct entity set **under this SNAT path** is reasoned from
+  `platform/crafty/networkpolicy.yaml`'s working game-port rule, **not measured**.
+- **Whether Plex's remote path needs anything beyond the manual port tick.** If plex.tv still lists
+  no remote connection after both manual steps, this branch is necessary-but-insufficient.
+- **The CI approval-policy outcome.** The local stage reported green, but that is an offline N/A, not
+  a verdict: this PR touches `clusters/**`, so CI will block it pending an override.
+- **PRE-EXISTING DRIFT I FOUND AND DELIBERATELY LEFT ALONE — worth its own follow-up.** Running the
+  documented `python3 catalog/render.py --emit-waves` regenerates the wave kustomizations and wants to
+  add **`vm-demo.yaml`** to wave 0 and **`keycloak.yaml`** to wave 1. Both are listed as
+  intentionally not onboarded in `ci/render-golden.sh`
+  (`RENDER_GOLDEN_NOT_ONBOARDED="keycloak vm-demo"`); their manifests are tracked in `apps/` but
+  deliberately absent from the `resources` lists so Flux does not apply them. `--emit-waves` places
+  any file present on disk and has **no awareness of that exclusion**, while `render-golden.sh` uses
+  `--wave-layout` and compares rendered manifests only — it never checks kustomization `resources`,
+  which is why this has survived unnoticed. **So the header "DO NOT hand-edit: re-run
+  `render.py --emit-waves`" is currently unsafe advice**: following it onboards an IdP and a KubeVirt
+  VM. I therefore restored both kustomizations and hand-added only my one line. I did not fix
+  `--emit-waves`, and I have not determined how long the drift has existed or whether either service
+  is *meant* to be onboarded.
@@ -0,0 +1,133 @@
+---
+date: 2026-10-04
+branch: feat/pihole-declare-state
+title: PiHole's operator-set state is now declared in git — the prerequisite for running more than one replica
+---
+
+## What changed
+
+Phase 1 of making LAN DNS survive a node failure. The operator asked for multi-replica PiHole;
+measurement showed that cannot be done safely first, because PiHole's important state was not
+reproducible from git. A second replica with a fresh PVC would have come up **serving DNS and
+blocking nothing, with a different admin password** — answers varying by which pod replied. This
+branch closes that; the replica work is phase 2 and is deliberately NOT here.
+
+**The undeclared-state inventory, measured not guessed.** `pihole.toml` marks every non-default
+setting `### CHANGED`, and those that came from the environment `### CHANGED (env)`. The complete
+live list on 2026-10-04:
+
+| Setting | Status |
+|---|---|
+| `dns.upstreams`, `dns.listeningMode`, `dns.rateLimit.count`, `misc.dnsmasq_lines` | `### CHANGED (env)` — already declared in `catalog/pihole.yaml` |
+| `files.macvendor` | `### CHANGED` — set by the image itself (container path), not operator state |
+| `webserver.api.pwhash` | **`### CHANGED` — set out of band. Declared NOWHERE.** |
+
+So exactly **one** operator-set value lived only on the PVC. Separately, the **adlists** were also
+undeclared: `gravity.db` held one list (StevenBlack's hosts) and 72,234 gravity domains, with nothing
+in this repo naming it.
+
+- **`apps/wave-0/pihole-bootstrap-secret.sops.yaml` (new, SOPS/age).** Secret `pihole-bootstrap`,
+  key `webui-password`. Named to the `<svc>-bootstrap-secret.sops.yaml` convention so
+  `render.py --emit-waves` places it with no renderer change. The value is a 32-char random password
+  generated at authoring time and **never printed to a transcript**.
+- **`catalog/pihole.yaml`:** `FTLCONF_webserver_api_password` sourced from that Secret.
+  `webserver.api.password` takes CLEARTEXT and FTL hashes it itself — verified in the running image,
+  where `pihole setpassword` *is* `setFTLConfigValue "webserver.api.password"`, commented upstream
+  "pihole-FTL will automatically hash the password". The PVC's `pwhash` becomes a derived artefact.
+- **`catalog/pihole.yaml`:** a `lifecycle.postStart` hook that **reconciles** the declared adlist set
+  into `gravity.db` on every container start — adds missing, re-enables disabled, and **disables**
+  (never deletes) anything enabled but undeclared, running `pihole -g` only when something moved.
+  Same level-triggered idiom, and the same reasoning, as `catalog/qbittorrent.yaml`'s postStart.
+- **`apps/wave-0/pihole.yaml`, `apps/wave-0/kustomization.yaml`:** re-rendered / secret registered.
+
+**Why a postStart hook and not a file.** PiHole v6 keeps adlists as DB rows. `/etc/pihole/adlists.list`
+still exists in `gravity.sh` but is **migration-only**: its import sits inside `migrate_to_database()`
+behind `if [ ! -e "$gravityDBfile" ]`, so it fires only while *creating* a database. A ConfigMap
+mounted there would be silently ignored on every pod this cluster runs today. There is also no
+`pihole adlist` CLI subcommand, so SQL is the actual interface; FTL ships an embedded shell
+(`pihole-FTL sqlite3`) so nothing had to be added to the image.
+
+**The password is kept rather than set empty, deliberately.** The admin UI is already behind the
+oauth2-proxy sidecar (`--allowed-group=platform-admins`), so an empty password would have been
+simpler. But the `pihole` Service still publishes **port 80** (PiHole direct, bypassing the proxy on
+`:4180`) and only the CiliumNetworkPolicy closes it — and this repo has a documented default-deny
+drift history. An unauthenticated admin UI that can rewrite LAN DNS should not rest on one policy
+staying correct. Narrowing that Service port is follow-up, not done here.
+
+**SENSITIVE PATHS — this PR adds a `**/*.sops.yaml`, so `ci/approval-policy.sh` will BLOCK it.** It
+needs the operator's `/approve-override <reason>` as a PR **review** of type "Comment". Not posted by
+me: CLAUDE.md section 1 forbids an agent clearing this on its own PR.
+
+## What I verified
+
+Live prod, 2026-10-04, read-only except where noted.
+
+- **The undeclared-state inventory above** — read from `pihole.toml`'s own `### CHANGED` markers, with
+  the password value masked. This is the measurement the whole branch rests on.
+- **A fresh `gravity.db` seeds NO adlists.** The image's own schema template
+  `/etc/.pihole/advanced/Templates/gravity.db.sql` creates the `adlist` table and contains no
+  `INSERT INTO adlist` at all — grepped in the running image. Confirmed empirically below (0 rows).
+  This is why the adlist declaration is load-bearing and not cosmetic.
+- **The adlists.list path really is migration-only** — read the guard in `/opt/pihole/gravity.sh`
+  (`migrate_to_database()`, `if [ ! -e "$gravityDBfile" ]`).
+- **`webserver.api.password` is a real settable key** — `pihole-FTL --config webserver.api.password`
+  returns a masked value, and `/usr/local/bin/pihole`'s `setpassword` sets exactly that key.
+- **THE RENDERED HOOK ACTUALLY WORKS — all four paths, extracted from `apps/wave-0/pihole.yaml` and
+  run in the live pod against THROWAWAY COPIES of `gravity.db` in `/tmp`** (production `gravity.db`
+  was never written, and `pihole -g` was stubbed so no 70k-domain download could start):
+  1. *Steady state* → `already matches the declared set - nothing to do`. No writes, no rebuild.
+  2. *True fresh DB built from the image's schema template* → 0 adlists before; hook added the
+     declared list and triggered the rebuild. **This is the fresh-PVC/new-replica case.**
+  3. *An undeclared list enabled* → disabled it (`enabled=0`), left the declared one alone, and did
+     not delete the row.
+  4. *Declared list present but disabled* → re-enabled it.
+- **Required tools exist in the image:** `bash`, `expr`, `grep`, `printf`, `sleep`, `pihole-FTL`.
+- **The Secret is genuinely encrypted** to the repo's age recipient; `stringData.webui-password`
+  starts `ENC[`. The gate's `no-plaintext-secrets (R-SEC-01)` L0 check passes.
+- **Schema:** `check-jsonschema` against `schemas/servicedefinition.schema.json` passes.
+- **Render determinism:** the canonical
+  `render.py --all catalog -o apps --wave-layout --no-image-automation` changed **only**
+  `apps/wave-0/pihole.yaml`. `render-golden` passes, so the hook's text also satisfies
+  `assert_substitution_safe` (no braced dollar expansion — the Flux envsubst contract).
+- **Toolbox gate:** rc **3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
+  `render-golden`, `l1`, `approval-policy` all **pass**. The rc 3 is the two usual environmental
+  skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`) and `conformance` L2/L3
+  (`CLUSTER_AVAILABLE` unset). **Not green; reported as rc 3.**
+
+## What I did NOT verify
+
+- **Nothing here has been reconciled.** Flux has not applied any of it, so the password has never
+  been set by this mechanism and the hook has never run during a real container start. Everything
+  above is the rendered artifact tested out-of-band, plus the image's own source.
+- **THE CURRENT ADMIN PASSWORD WILL STOP WORKING, and that is the intent.** FTL re-hashes
+  `webserver.api.password` on start, overwriting the existing `pwhash`. Whoever knows the present
+  password (nobody, which is why this started) loses it; the declared one replaces it. I have not
+  observed that replacement happen.
+- **Cold-start cost is unmeasured and could matter.** `postStart` must complete before the container
+  is reported Running. On a fresh PVC the hook waits for `gravity.db` (bounded, up to 180s) and then
+  runs a real `pihole -g` (72k domains, network downloads). `apps-wave-0` is `wait: true` and
+  `apps-wave-1` `dependsOn` it, so a slow cold start **delays the app tier**. The steady-state path
+  exits in about a second (scenario 1), so this is a rebuild/first-boot cost only — but it is not
+  measured, and the stub in testing means `pihole -g`'s real duration here is unknown.
+- **Whether `pihole -g` succeeds from inside the hook's environment.** Its failure is caught and the
+  existing `gravity.db` retained, so a failure degrades rather than breaks — untested live.
+- **Concurrency.** Nothing serialises the hook against FTL's own scheduled gravity update. SQLite
+  handles the locking, and the hook's writes are three small statements, but no interleaving was
+  exercised.
+- **MULTI-REPLICA — PHASE 2, NOT STARTED.** This branch only removes the blocker. The remaining work,
+  and it is substantial: a `replicas`/placement field on the ServiceDefinition schema (currently
+  `additionalProperties: false` with no such field), `render.py` emitting a StatefulSet with
+  `volumeClaimTemplates` (the config PVC is **ReadWriteOnce**, so a 2-replica Deployment cannot
+  work), pod anti-affinity, a PodDisruptionBudget (there is none today), re-rendered goldens, and
+  `conformance/**` updates — the last making phase 2 sensitive-path as well. The PVC rename a
+  StatefulSet forces will discard the existing `gravity.db`, which is only safe **because** of this
+  branch, and should be done only after this one is confirmed working in-cluster.
+- **The real risk the operator named is NOT fixed by phase 2 either.** Replicas address *node*
+  failure. LAN DNS still depends on the cluster as a whole — etcd quorum, Cilium, Longhorn, a bad
+  reconcile. Quantified on 2026-10-04: with `tolerationSeconds=300` plus node-NotReady detection
+  plus a Longhorn RWO force-detach, a failure of the node running the pod is roughly **6-10 minutes
+  of no LAN DNS**; a failure of the node holding the `pihole-dns-*` L2 lease (currently a
+  *different* node — lease on k3s-prod-2, pod on k3s-prod-1) is tens of seconds. An out-of-cluster
+  secondary resolver carrying the same `address=` lines is the only option that removes the
+  dependency, and it was considered and not chosen.
+- **Port 80 on the `pihole` Service is still published** and still closed only by policy. Not narrowed.
@@ -0,0 +1,117 @@
+---
+date: 2026-10-04
+branch: docs/pihole-ha-verified
+title: PiHole HA is VERIFIED on live prod — including a failover drill with zero dropped DNS queries
+---
+
+## What changed
+
+**No manifests changed. This entry exists to correct the log.** The three PiHole/Plex entries merged
+earlier today each say, correctly at the time, that nothing had been reconciled and that the outcome
+was unverified:
+
+- `decisions/2026-10-04-pihole-declare-state.md` — "Nothing here has been reconciled."
+- `decisions/2026-10-04-pihole-multi-replica.md` — "Nothing has been reconciled. No StatefulSet
+  exists, no second replica has ever run, the anti-affinity has never been evaluated..."
+
+All three are now **merged, reconciled and measured**, so those sentences are stale in exactly the way
+§2b of CLAUDE.md warns about — a reader who found them would conclude the work is unproven. This entry
+is the current state; it does not edit the earlier files (separate files, nothing to merge).
+
+Delivered and live as of 2026-10-04, from PRs #43 (plex tcp/32400), #44 (declare PiHole state) and
+#45 (PiHole 2 replicas + the mountPath bug):
+
+- PiHole is a **2-replica StatefulSet**, one pod per node, with per-replica Longhorn volumes, a
+  PodDisruptionBudget, and an `up` SLI that follows the rendered workload kind.
+- PiHole's admin password and adlists are **declared in git** and reproduce onto a fresh volume.
+- PiHole's config PVC is mounted where PiHole actually writes (`/etc/pihole`), which it was not before.
+
+## What I verified
+
+All measured against live prod on 2026-10-04, after the merge of #45 (`dce20b7`).
+
+**Reconcile.** `GitRepository` artifact `main@sha1:dce20b7`; `apps-wave-0`
+`lastAppliedRevision=main@sha1:dce20b7`, `Ready=True/ReconciliationSucceeded`. It completed in roughly
+**90 seconds**, well inside the Kustomization's `timeout: 5m` — the risk flagged in the multi-replica
+entry ("whether `pihole -g` finishes inside the 5 m timeout") did **not** materialise. A transient
+`Ready=False/DependencyNotReady: platform-config` seen mid-flight was the dependency cascade, not a
+failure; `platform-config` was already `Ready=True` at the new revision moments later.
+
+**Topology and objects.**
+
+| Measured | Value |
+|---|---|
+| pods / nodes | `pihole-0` → **k3s-prod-3**, `pihole-1` → **k3s-prod-2** (2/2 Running each) — the required anti-affinity was satisfied |
+| per-replica claims | `config-pihole-0`, `config-pihole-1` — both `Bound`, 2Gi |
+| old `pihole-config` PVC | **pruned** (it held only `lost+found`, so nothing was lost) |
+| old `pihole` Deployment | **pruned** — `No resources found in pihole namespace` |
+| PodDisruptionBudget | `minAvailable 1`, `disruptionsAllowed 1`, `currentHealthy 2` |
+
+**The mountPath bug is fixed.** `df` inside both pods reports `/etc/pihole` on
+`/dev/longhorn/pvc-...` with **6.9 M used** — previously it was on `overlay` while the Longhorn volume
+sat unused at `/data/config`. `backup.policy: daily-offcluster` now has real content to back up
+instead of reporting success over an empty volume.
+
+**Phase 1's declarations reproduce onto a FRESH volume — the point of doing it first.** Both replicas
+booted with an empty `/etc/pihole`:
+
+- **72,234 gravity domains** on each, rebuilt from the adlist declared in `catalog/pihole.yaml`.
+- **The declared SOPS secret authenticates on both pods.** `POST /api/auth` using the container's own
+  `FTLCONF_webserver_api_password` returned `"valid":true` on `pihole-0` **and** `pihole-1` — so
+  Secret → env → FTL's own hashing → working login, identically on both. (The secret's value was never
+  printed; an earlier reading of `pwhash` as "unset" was a quoting bug in the check, disproven by an
+  empty password being rejected with `password incorrect` and then by this positive test.)
+
+**User-facing behaviour through the VIP (192.168.2.23), from a LAN host.**
+
+    plex.home.keithjackson.dev   -> 192.168.2.20      split-horizon intact
+    home.keithjackson.dev        -> 192.168.2.20
+    unifi.home.keithjackson.dev  -> 192.168.2.21
+    google.com                   -> 142.250.115.102   recursion to 8.8.8.8 works
+    doubleclick.net              -> 0.0.0.0           blocking LIVE in FTL, not just in the DB
+    googleadservices.com         -> 0.0.0.0
+    12 consecutive queries       -> 12/12 answered
+
+**THE FAILOVER DRILL — observed by the operator, 2026-10-04.** A single StatefulSet replica was
+deleted. It was recreated in **~30 seconds**, and **DNS never dropped a request** throughout. This is
+the first direct evidence that the replica count delivers what it was added for, rather than merely
+rendering correctly.
+
+**The SLI is not inert, and re-deriving it mattered.** Measured in Prometheus:
+`kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}` returns **1 series,
+value 2**; `kube_deployment_status_replicas_available{namespace="pihole"}` now returns **0 series**;
+and Prometheus has `pihole-down` loaded as `state=inactive health=ok` with the StatefulSet expression.
+Had the rule stayed Deployment-shaped, its `or absent(...)` branch would have matched the now-absent
+series and **fired permanently** — a standing false page rather than silence. That is the concrete
+payoff of defaulting `livenessWorkload.kind` from the rendered kind.
+
+## What I did NOT verify
+
+- **NODE failure, as opposed to pod failure.** The drill deleted a pod, which does not exercise the
+  `tolerationSeconds: 300` path at all — that timer only applies when a node goes NotReady. The
+  mechanism protecting DNS is the same one just proven (the surviving replica plus
+  `externalTrafficPolicy: Cluster` on the DNS LoadBalancers), so user-visible impact should likewise be
+  nil; what the 300 s still delays is the *replacement* replica, and during that window there is no
+  redundancy left. Untested. Pulling power from a node is the only real test.
+- **DNS during the cutover itself.** I was not sampling resolution while #45 reconciled, so whether
+  there was a brief gap between the Deployment being pruned and the StatefulSet becoming Ready is
+  **unmeasured**. The drill above covers the steady-state failover case, not that one-off transition.
+- **Two-node loss.** The hard anti-affinity means `replicas: 2` needs two schedulable nodes; losing
+  two would leave a replica `Pending` and, with `wait: true`, stall the app tier. Reasoned, not
+  exercised — and it coincides with etcd quorum loss on a 3-node control plane.
+- **Longhorn capacity headroom.** Two 2Gi claims at 3 replicas each is ~12Gi where there was ~6Gi.
+  Both claims are `Bound` and healthy; free space was not checked.
+- **Query history is now split per replica** and the admin UI shows only whichever pod answered.
+  Adlists and password are identical, so blocking and auth are consistent; the *reporting* is not, and
+  whether that is acceptable long-term is the operator's call.
+- **The residual risk is unchanged, and it is the larger one.** Replicas address node failure. LAN DNS
+  still depends on the cluster — etcd quorum, Cilium, Longhorn, a bad reconcile — and only a resolver
+  outside the cluster carrying the same `address=` lines removes that dependency. Considered and not
+  chosen (operator decision, 2026-10-04).
+- **Plex remote access (#43) is NOT covered by this entry.** Its two manual steps (the WAN tcp/32400
+  forward and Plex's "manually specify public port") and the outside-in path remain as
+  `decisions/2026-10-03-plex-remote-access-l4.md` describes them. The operator reported it working;
+  I have not measured it from off-LAN.
+- **`--emit-waves` drift is still open.** Re-running it still wants to onboard `vm-demo.yaml` and
+  `keycloak.yaml`, both listed as intentionally not onboarded in `ci/render-golden.sh`. Unfixed; see
+  `decisions/2026-10-03-plex-remote-access-l4.md`.
@@ -0,0 +1,145 @@
+---
+date: 2026-10-04
+branch: feat/pihole-multi-replica
+title: PiHole runs two replicas on different nodes — and its config PVC was mounted where PiHole never writes
+---
+
+## What changed
+
+Phase 2 of making LAN DNS survive a node failure (phase 1:
+`decisions/2026-10-04-pihole-declare-state.md`). PiHole was one pod on one node, which for a
+whole-house resolver cost more than "until it reschedules" — measured 2026-10-04:
+
+| Failure | Recovery |
+|---|---|
+| the node running the pod dies | ~40s to mark NotReady + `tolerationSeconds: 300` before eviction + reschedule + a Longhorn RWO force-detach → **roughly 6-10 minutes with no LAN DNS** |
+| the node holding the `pihole-dns-*` L2 lease dies | Cilium re-elects an announcer, tens of seconds |
+
+Those were **different** nodes (lease on k3s-prod-2, pod on k3s-prod-1), so two of three nodes were
+single points of failure for LAN DNS with wildly different blast radii.
+
+**A new schema field, because this is intent and not a primitive.** `availability.replicas`
+(schema **1.10.0**, mechanism in `schemas/renderer-contract.md` `### availability`). `1` or absent
+renders a Deployment exactly as before — every pre-existing golden is byte-identical, verified. `>1`
+renders, together:
+
+- a **StatefulSet** instead of a Deployment. This is forced, not preferred: a longhorn PVC is
+  **ReadWriteOnce**, and one RWO claim cannot attach on two nodes, so a 2-replica Deployment does not
+  degrade — the second pod never schedules.
+- longhorn storage as **per-replica `volumeClaimTemplates`** (`config-pihole-0`, `config-pihole-1`);
+  the standalone PVC is no longer emitted, since it would be an orphan nothing mounts. A `bulk`
+  entry stays a standalone PVC — it is a shared static NFS export, not per-replica state.
+- **required** `podAntiAffinity` on `kubernetes.io/hostname`. `preferred` would let both replicas
+  land on one node and deliver none of the intended tolerance while reporting healthy.
+- `podManagementPolicy: Parallel`, so a cold start costs the max of the replicas' first-boot work
+  rather than the sum — `apps-wave-0` has `timeout: 5m`.
+- a **PodDisruptionBudget**, `minAvailable: 1`, so a node drain cannot take the last replica.
+- `observability.livenessWorkload.kind` defaulting to `StatefulSet`, so the `up` SLI reads
+  `kube_statefulset_status_replicas_ready`. A Deployment-shaped rule against a StatefulSet
+  `absent()`s forever — an alert that can never fire.
+
+`catalog/pihole.yaml` sets `replicas: 2`, not 3: two tolerates any single node loss and leaves a
+spare node for the evicted replica under the hard anti-affinity. Three would pin one per node with no
+spare, so a single node failure would leave a replica permanently `Pending` and — `apps-wave-0` being
+`wait: true` — stall the app tier for as long as the node was down.
+
+### The bug found on the way, and fixed here
+
+**PiHole's config PVC was mounted at a path PiHole never writes, so it held nothing and all of its
+state was ephemeral.** With no explicit `mountPath` the renderer defaults to `/data/<name>`; PiHole
+keeps everything in `/etc/pihole`. Measured on live prod 2026-10-04: `/etc/pihole` (7.1 M —
+`gravity.db`, `pihole.toml`, the query DB) was on the **overlay** filesystem, while the 2 Gi Longhorn
+volume sat at `/data/config` holding `lost+found` and 24 K of nothing. Consequences, all live until
+now:
+
+- every restart silently discarded PiHole's entire state and re-downloaded 72 k gravity domains;
+- `backup.policy: daily-offcluster` has been faithfully backing up an **empty** volume, which is
+  worse than no backup because it reports success;
+- it also means phase 1 was understated: that state was not merely undeclared, it was **ephemeral**.
+
+Same class of bug `catalog/plex.yaml` documents for its own `config` entry. Fixed in this branch
+rather than its own because with the wrong path the new per-replica volumes would be two 2 Gi
+Longhorn volumes (×3 Longhorn replicas each) holding only `lost+found` while all state stayed
+ephemeral — paying for persistence and getting none. It also removes this change's main operational
+risk: with `gravity.db` persisted, a restart is cheap instead of re-downloading 72 k domains inside a
+wave with a 5 m timeout.
+
+## What I verified
+
+- **Back-compatibility, the thing most likely to break quietly:** after the renderer change and
+  before `catalog/pihole.yaml` declared anything, the canonical
+  `render.py --all catalog -o apps --wave-layout --no-image-automation` produced **zero diff** across
+  the whole `apps/` tree. `render-golden` passes.
+- **The new behaviour is pinned by ~20 new `render.py --selftest` assertions** (so `ci/render-golden.sh`
+  hard-fails on a regression, and no `conformance/**` change was needed): `replicas: 1` still renders a
+  Deployment + standalone PVC + no PDB; `replicas: 2` renders a StatefulSet and **no** standalone
+  longhorn PVC, carries the replica count, omits `strategy`, sets `Parallel` and `serviceName`, names
+  the claim template exactly as the mounted volume, keeps RWO/longhorn on it, does **not** duplicate
+  that volume in `pod.volumes`, still mounts it on the container, renders required hostname
+  anti-affinity, renders exactly one `minAvailable: 1` PDB, and switches the SLI to
+  `kube_statefulset_status_replicas_ready`. A mixed longhorn+bulk case asserts the bulk share stays a
+  standalone PVC and an ordinary pod volume.
+- **Rendered pihole output, read back:** `replicas: 2`, `serviceName: pihole`,
+  `podManagementPolicy: Parallel`, no `strategy`, claim template `config` (RWO/longhorn/2Gi),
+  anti-affinity `kubernetes.io/hostname`, `pod.volumes: []`, mount `config -> /etc/pihole`,
+  PDB `minAvailable: 1`, SLI `kube_statefulset_status_replicas_ready`.
+- **The `/etc/pihole` mount is safe, evidenced from the image's own init rather than assumed:**
+  `/usr/bin/start.sh` is PID 1 and runs as **uid 0**; its line 5 is
+  `chown -R pihole:pihole /etc/pihole/ /var/log/pihole/`, so a fresh root-owned volume has its
+  ownership fixed before anything else starts (**no `fsGroup` needed, and one would be inert**); it
+  calls `migrate_gravity`, whose `migrate_to_database()` creates the gravity database when
+  `[ ! -e "$gravityDBfile" ]`; FTL writes `pihole.toml` itself from the `FTLCONF_*` env; and the
+  image's own default for `files.macvendor` was moved **out** of that directory, which is what a
+  maintainer does to a path they expect to be a volume.
+- **The PVC really is unused:** `df` showed `/etc/pihole` on `overlay` and
+  `/dev/longhorn/pvc-849edb86-...` on `/data/config`; `ls -A /data/config` → `lost+found` only.
+- **The gate caught a real regression and it is fixed.** The first run came back **rc 1**:
+  `alert-rules-unit-tested` failed because `apps/wave-0/tests/app-slis.test.yaml` fed pihole a
+  `kube_deployment_status_replicas_available` series and expected a `deployment: pihole` label. Three
+  input series, the firing expectation and the file header were updated;
+  `platform/monitoring/tests/run.sh apps/wave-0/tests/app-slis.test.yaml` → **SUCCESS** (6 rules).
+- **Schema:** `check-jsonschema` passes over all of `catalog/*.yaml`.
+- **Toolbox gate:** rc **3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
+  `render-golden`, `l1`, `approval-policy` all **pass**. The rc 3 is the two environmental skips —
+  `gitea-settings` (no local `SETTINGS_API_TOKEN`) and `conformance` L2/L3 (`CLUSTER_AVAILABLE`
+  unset). **Not green; reported as rc 3.**
+- **No sensitive paths touched**, so unlike the previous two branches this one needs no
+  `/approve-override` — confirmed against `infra/gitea/approval-policy.json`.
+
+## What I did NOT verify
+
+- **Nothing has been reconciled.** No StatefulSet exists, no second replica has ever run, the
+  anti-affinity has never been evaluated by the scheduler, the PDB has never been consulted, and the
+  `/etc/pihole` mount has never been exercised on a real pod. Everything above is the rendered
+  artifact plus the image's own source.
+- **THE ROLLOUT WILL BRIEFLY DROP LAN DNS, and I have not measured for how long.** `postStart` must
+  complete before a container is reported Running, so a pod is not Ready — and the Service has no
+  endpoints — until phase 1's adlist hook finishes. On this transition both replicas start with an
+  empty `/etc/pihole`, so the hook finds a fresh `gravity.db` with zero adlists and runs a real
+  `pihole -g` (72 k domains, network download) before either pod becomes Ready. **Do this at a quiet
+  time.** Two mitigating facts: every pihole restart has *already* paid this cost (state was
+  ephemeral), and after this change `gravity.db` persists so later restarts should not. If it turns
+  out slow, backgrounding the gravity rebuild inside the hook would decouple readiness from it, at
+  the cost of a window where DNS resolves but does not block — not done here.
+- **Whether `pihole -g` finishes inside `apps-wave-0`'s 5 m `timeout`.** `Parallel` was chosen to
+  make this the max rather than the sum of two first boots, but the real duration is unmeasured (it
+  was stubbed during phase 1's testing). If the wave times out, the Kustomization reports failed even
+  though the pods may come up seconds later.
+- **The old PVC's deletion.** `apps-wave-0` has `prune: true`, so removing `pihole-config` from the
+  rendered output should delete it. That destroys nothing (it holds `lost+found`), but I have not
+  observed the prune, and I did not check whether Longhorn retains the backing volume afterwards.
+- **The transition overlap.** For a moment the pruned Deployment's pod and the new StatefulSet pods
+  may both match the Service selector. Both are configured identically from git so this should be
+  benign — unobserved.
+- **Longhorn capacity.** Two 2 Gi claims at 3 Longhorn replicas each is ~12 Gi of real storage where
+  there was ~6 Gi. Not checked against free space.
+- **Whether two independent query databases are acceptable to the operator.** Stats and query history
+  now split per replica, and the admin UI shows only the pod that answered. Adlists and password are
+  identical (phase 1), so blocking and auth are consistent; the *reporting* is not.
+- **The residual risk is unchanged and is the larger one.** Replicas address node failure. LAN DNS
+  still depends on the cluster — etcd quorum, Cilium, Longhorn, a bad reconcile — and only a resolver
+  outside the cluster carrying the same `address=` lines removes that. Considered and not chosen
+  (operator decision, 2026-10-04).
+- **`tolerationSeconds: 300` is untouched.** With two replicas the surviving one serves immediately, so
+  the 5-minute eviction delay no longer gates recovery — but it still delays the *replacement* replica,
+  and during that window there is no redundancy left.
@@ -0,0 +1,139 @@
+---
+date: 2026-10-04
+branch: fix/plex-advertise-plexdirect
+title: Plex vanished from the plex.tv account — every advertised connection was uncompletable; advertise the plex.direct name instead of the bare VIP
+---
+
+## What changed
+
+Reported symptom: the server no longer appears on the operator's plex.tv account; `plex.home.keithjackson.dev`
+shows "not authorized"; connecting straight to the VIP + port works.
+
+**Root cause, measured: `customConnections` held two hand-typed URLs and NEITHER could ever complete,
+so Plex had nothing publishable.** Because `customConnections` OVERRIDES Plex's own connection
+publishing, the server ended up advertising only dead endpoints. Live `Preferences.xml`, 2026-10-04:
+
+    customConnections="https://plex.home.keithjackson.dev:32400/,https://192.168.2.24:32400/"
+    ManualPortMappingMode="1"   ManualPortMappingPort=<ABSENT>
+
+- `https://192.168.2.24:32400` — **TLS can never validate.** Plex serves 32400 with
+  `CN=*.130072f20c334d6c8f63d0dfef87309b.plex.direct` and **no IP SAN**, so a bare-IP https URL fails
+  for every client. Not theoretical: Plex's log carries a continuous storm of
+  `CERT: incomplete TLS handshake from <node> ... sslv3 alert certificate unknown`, every few seconds,
+  from **real LAN clients** arriving through the new `plex-l4-tcp` VIP.
+- `https://plex.home.keithjackson.dev:32400` — **nothing listens.** The shared Gateway serves that host
+  on 443; measured `http=000` (connect failed) against `192.168.2.20:32400`.
+
+Consequence chain, each link measured: no completable connection -> plex.tv could publish nothing
+(`MyPlex: mapping state set to 'Mapped - Not Published'`, repeatedly) -> Plex fell back to **Plex Relay**
+(`startRelay`, remote forward to 127.0.0.1:32401) -> **the server disappeared from the account**, and
+clients reported "not authorized".
+
+The fix advertises Plex's own `<dashed-ip>.<serverhash>.plex.direct` form, which is public DNS resolving
+to the embedded address — so it points at the LAN VIP *and* matches the wildcard cert.
+
+- **`catalog/plex.yaml`:** `ADVERTISE_IP: ${PLEX_ADVERTISE_URL}` added, replacing the long-standing
+  DEFERRAL note. The deferral had weighed only the L7 Gateway URL; the operator chose the native L4 path
+  on 2026-10-03, so the open question was never "Gateway or not" but "which URL has a valid cert".
+- **`clusters/base/cluster-vars.yaml`:** new `PLEX_ADVERTISE_URL` knob (+ substitution-inventory comment
+  updated to name `plex: PLEX_ADVERTISE_URL`). The base default is an explicit **non-functional
+  placeholder** — dev is decommissioned and has no claimed server, and
+  `StrictPostBuildSubstitutions` makes an undefined var a hard failure, so the default only has to exist
+  and be correctly shaped.
+- **`clusters/prod/kustomization.yaml`:** prod value
+  `https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400`.
+- **`apps/wave-0/plex.yaml`:** re-rendered with the canonical command
+  (`python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation`). Two added
+  lines, nothing else.
+
+**Why a whole-URL cluster-var and not something built from `${PLEX_L4_VIP}`:** postBuild substitution is
+plain envsubst and cannot turn `192.168.2.24` into `192-168-2-24`, and `<serverhash>` is the server's
+certificate identity, derivable from nothing in this tree. The dashed IP and `${PLEX_L4_VIP}` are two
+halves of one address and **nothing checks that they agree** — noted in both files.
+
+**SENSITIVE PATHS — this PR touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK it.**
+It needs the operator's `/approve-override <reason>` posted as a PR **review** of type "Comment".
+Deliberately not posted here: CLAUDE.md §1 forbids an agent clearing this on its own PR, and the agent
+account holds `owner`, so the prohibition is a rule rather than a limit.
+
+## What I verified
+
+All measurements against live prod, 2026-10-04, read-only. Secrets handled per CLAUDE.md §6: preference
+attributes were read by name-scoped grep, never by dumping `Preferences.xml` (it holds `PlexOnlineToken`),
+and the token was reported only as `<PRESENT,len=22>`.
+
+- **THE FIX ITSELF, with full TLS verification (no `curl -k`)** — the one measurement the whole change
+  rests on:
+  - `https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400/identity` -> **`http=200`, TLS verify OK**
+  - `https://192.168.2.24:32400/identity` -> **`http=000`, cert validation FAILED**
+  - that name resolves to `192.168.2.24` (`getent hosts`), confirming the dashed-IP mechanism works from
+    inside this network and is not blocked by DNS-rebinding protection on the resolver path tested.
+- **The server is still claimed — this was never a lost claim.** `/identity` -> `claimed="1"` with the real
+  `machineIdentifier`; `PlexOnlineToken` present, `PlexOnlineUsername="jakman85"`.
+- **The certificate, read off the live listener:** exactly one SAN,
+  `DNS:*.130072f20c334d6c8f63d0dfef87309b.plex.direct`. No IP SAN. This is why the bare-IP URL fails.
+- **The publish payload Plex actually sent to plex.tv**, which is what proves the connection list was the
+  problem rather than a guess: `PUT https://servers.plex.tv/devices/<id>?Connection[][uri]=https://plex.home.keithjackson.dev:32400&Connection[][uri]=https://192.168.2.24:32400&Connection[][uri]=http://136.51.115.252:32400&Connection[][uri]=http://10.0.2.136:32400&httpsEnabled=1...`
+  — all four unreachable from a LAN client or from plex.tv.
+- **`Mapped - Not Published`** at every reachability check across the whole log, and an active **Plex Relay**
+  session — the documented fallback when nothing is directly publishable.
+- **The `certificate unknown` TLS failures come from real client traffic, not a probe.** The source
+  `10.0.1.78` resolves via `cilium-dbg bpf ipcache get` to **identity 1 = `reserved:host`**, i.e. LAN
+  clients SNAT'd through `plex-l4-tcp` (`externalTrafficPolicy: Cluster`) on prod-3.
+- **The Gateway is NOT broken, and my first reading of it was wrong.** A 403 `Access denied` from
+  `server: envoy` on the Gateway path turned out to be an artifact of probing from inside the plex pod:
+  **all 16** Gateway hostnames return 403 from there, because Cilium's Gateway preserves the original
+  source identity and enforces the destination's ingress policy. Plex's own log shows a concurrent,
+  **working, signed-in** browser session through the Gateway returning 200s. The HTTPRoute is
+  `Accepted=True`/`ResolvedRefs=True` with no filters, and it is the only route claiming that hostname.
+- **Cilium is not denying the Gateway->Plex hop:** `cilium-dbg bpf policy get 3796` shows
+  `Allow Ingress reserved:ingress 32400/TCP`, PROXY PORT `NONE`, with 141k packets counted. No L7/HTTP
+  rules and no CCNPs exist anywhere in the cluster.
+- **In-cluster consumers were never affected:** Seerr's recently-added scans and watchlist syncs complete
+  cleanly with no auth errors; there are **zero** HTTP 401s in Plex's log (the five ` 401 ` hits are
+  transcoder segment numbers).
+- **`ADVERTISE_IP`'s mechanism, in the running image rather than from docs:**
+  `/etc/cont-init.d/40-plex-first-run` line 127 does `setPref "customConnections" "${ADVERTISE_IP}"`,
+  outside the claim guard, and `/.firstRunComplete` (lines 9/139) is on the container overlay, not the
+  `/config` PVC — so it re-applies on every pod start. Editing the env rolls the Deployment, which applies it.
+- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
+  `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
+  **pass**. The rc 3 is the two known environmental skips — `gitea-settings` (no local
+  `SETTINGS_API_TOKEN`) and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3.**
+- **`render-golden` passing is the specific proof** that the catalog edit and the committed golden agree;
+  the rendered diff is exactly two lines and no other service's manifest moved.
+
+## What I did NOT verify
+
+- **THE OUTCOME. Nothing here has been observed to put the server back on the plex.tv account.** The
+  change is unreconciled: `ADVERTISE_IP` is absent from the live Deployment, the live
+  `customConnections` still holds the two broken URLs, and Plex has not been restarted. What is proven is
+  that the *new URL answers 200 with a valid certificate* and that *every currently advertised URL cannot
+  complete* — a necessary-and-plausibly-sufficient argument, not a demonstrated fix.
+- **Whether plex.tv will publish and list the server once this lands.** plex.tv's decision is made
+  off-network and cannot be measured from here. A LAN-only connection may still leave the account showing
+  the server as remotely unreachable even when local clients work.
+- **REMOTE (off-LAN) ACCESS IS NOT FIXED BY THIS, and may still be fully broken.** The advertised URL
+  resolves to an RFC1918 address, so it serves LAN clients only. The off-repo WAN tcp/32400 port-forward
+  (`runbooks/plex-remote-access.md`) is still **unverified** — `http://136.51.115.252:32400` was
+  unreachable from inside the LAN, but 2026-10-03 established there is no NAT hairpin here, so that is
+  **inconclusive**, not proof of absence. This repo has no off-LAN vantage point.
+- **`ManualPortMappingMode="1"` with `ManualPortMappingPort` ABSENT is left as-is.** Manual public-port
+  mapping is enabled with no port set — a half-configured state I did not change and whose exact effect on
+  publishing I did not isolate. There is **no GitOps path** to it (2026-10-03: the image supports no
+  `PLEX_PREFERENCE*` of any kind), so it is an operator UI step.
+- **Whether the operator intended the hand-typed URLs** to override the 2026-10-03 native-L4 design, or
+  typed them while debugging. I did not alter live preferences; the env var will overwrite
+  `customConnections` on the next pod start, which is a deliberate transfer of ownership from the web form
+  to git and will **silently discard** anything typed there since.
+- **The `plex.direct` hash's stability.** Treated as server identity and read once from the live cert. I
+  did not establish what reissues it, so this URL could go stale with no alert; nothing in CI or the
+  conformance suite checks that the dashed IP still matches `${PLEX_L4_VIP}` or that the hash is current.
+- **Resolver behaviour for the whole LAN.** `plex.direct` resolved correctly from a *pod* (cluster DNS
+  path). LAN clients resolve via PiHole, and dnsmasq's `stop-dns-rebind` would reject a public answer
+  carrying a private address. I did **not** test resolution from a LAN client, and this is the most likely
+  way the fix still fails.
+- **The CI approval-policy outcome.** The local stage reported green, but that is an offline N/A, not a
+  verdict: this PR touches `clusters/**`, so CI will block pending the operator's override.
+- **Why PMS restarted.** The process came up at 05:26 UTC while the pod is 2d2h old with 0 restarts, so
+  something restarted Plex inside the container. Unexplained and not investigated.
@@ -306,19 +306,27 @@ node_baseline_dns_servers:
  # k3s_base_domain, below). It reaches the issuer through the NODE resolver, i.e. through
  # these PUBLIC upstreams — not through CoreDNS's split-horizon answer.
  #
-  # THIS USED TO BE BROKEN, AND THE 2026-09-04 SUBDOMAIN SPLIT FIXED IT. While prod was
-  # authored to serve dev's hostnames, a prod node resolving keycloak.home.keithjackson.dev
-  # publicly got DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN, so
-  # kubectl OIDC SSO on prod could not work at all. Now prod's issuer is
-  # keycloak.prod.home.keithjackson.dev, and the public record for that name is published by
-  # PROD's own external-dns pointing at PROD's Gateway VIP (192.168.2.20). The public answer
-  # and the split-horizon answer therefore AGREE, and no /etc/hosts pin is needed.
+  # THE PUBLIC ANSWER IS NOW THE WAN, SO THE ISSUER IS PINNED (2026-10-03). Since
+  # platform/gateway/gateway.yaml set an external-dns `target`, every public name is a CNAME
+  # to wan.home.keithjackson.dev = the home WAN IP. The SOHO router does NOT hairpin: from the
+  # LAN, WAN:443 answers with the router's own self-signed placeholder cert (measured from
+  # k3s-prod-2/-3 2026-10-03: `curl` exit 60, ssl_verify_result=18). The apiserver would then
+  # be unable to fetch its issuer's discovery/JWKS, so kubectl OIDC SSO would fail on the
+  # next k3s restart or Keycloak key rotation. node_baseline_hosts_pins (below) pins the
+  # issuer host to the Gateway VIP in /etc/hosts, which glibc consults before these
+  # resolvers. (History: before the 2026-09-04 subdomain split the public answer was DEV's
+  # VIP; after it, the public answer was prod's own VIP and no pin was needed.)
  #
-  # STILL TO VERIFY on prod bring-up (PROD-PROVISIONING.md §10): this holds only once prod's
-  # external-dns has actually published the record. Until then the issuer does not resolve
-  # from the nodes and kubectl SSO fails — the cert-based admin kubeconfig is the break-glass
-  # path, and it always works. Ordering, not a design gap.
+  # The cert-based admin kubeconfig never depends on any of this. It is the break-glass path.
 node_baseline_dns_search: []
+# /etc/hosts pins: node-process names that must reach the in-cluster Gateway directly rather
+# than via the public (WAN, non-hairpinned) answer. See "THE PUBLIC ANSWER IS NOW THE WAN" above.
+# [parity] the IP MUST equal GATEWAY_VIP in clusters/prod/kustomization.yaml (192.168.2.20),
+# and the name MUST follow k3s_oidc_issuer_url's host (roles/k3s_server/defaults/main.yml).
+node_baseline_hosts_pins:
+  - ip: "192.168.2.20"
+    names:
+      - "keycloak.{{ k3s_base_domain }}"

 # --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
 # PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/
@@ -51,11 +51,14 @@ hubble:
 l2announcements:
  enabled: true

-# k3s ships its own cgroup mount; point Cilium at it instead of auto-mounting.
+# Point Cilium at the host's real cgroup2 root (/sys/fs/cgroup, mounted by systemd) instead of
+# auto-mounting. NOT /run/cilium/cgroupv2: nothing mounts cgroup2 there, so the agent mounted its
+# OWN namespaced cgroup and socket-LB covered only itself — host processes (the apiserver) lost
+# Service/LB translation. [parity] platform/cilium/helmrelease.yaml explains (2026-10-03).
 cgroup:
  autoMount:
    enabled: false
-  hostRoot: /run/cilium/cgroupv2
+  hostRoot: /sys/fs/cgroup

 # Roll the cilium agent DaemonSet ONE node at a time. [parity] must match the
 # platform/cilium HelmRelease. The cilium 1.20.1 chart defaults this to 2, which on a
@@ -242,3 +242,9 @@ node_baseline_dns_search:
 node_baseline_dns_options:
  - timeout:2
  - attempts:2
+# Static /etc/hosts entries (tags: dns, hosts), as a list of {ip, names}. EMPTY by default:
+# dev needs none. Prod pins its OIDC issuer host to the Gateway VIP because the public
+# answer for it is the non-hairpinned WAN IP (inventory/group_vars/k3s_prod/vars.yml).
+# Written as ONE marked block, so removing an entry here removes it from the node, and
+# an empty list removes the block entirely.
+node_baseline_hosts_pins: []
@@ -1090,3 +1090,22 @@
      {% endif %}
  when: node_baseline_dns_mechanism == 'resolv-conf-file'
  tags: [dns]
+
+# /etc/hosts pins — tag `dns` (and `hosts` on its own). Some node processes must reach a
+# public hostname at the in-cluster Gateway, not at its public answer. On prod, the
+# kube-apiserver's OIDC issuer resolves publicly to the WAN IP, which the router does not
+# hairpin. glibc reads /etc/hosts before any nameserver (nsswitch `files dns`), so a pin wins
+# regardless of which resolver mechanism above is in effect. One marked block: entries
+# dropped from node_baseline_hosts_pins disappear from the node, and an empty list
+# removes the block.
+- name: Pin node-process hostnames in /etc/hosts
+  ansible.builtin.blockinfile:
+    path: /etc/hosts
+    marker: "# {mark} node_baseline hosts pins (Ansible, tags: hosts)"
+    block: |
+      {% for pin in node_baseline_hosts_pins %}
+      {{ pin.ip }}	{{ pin.names | join(' ') }}
+      {% endfor %}
+    state: "{{ 'present' if node_baseline_hosts_pins | length > 0 else 'absent' }}"
+    backup: true
+  tags: [dns, hosts]
@@ -106,14 +106,33 @@ spec:
    # CiliumL2AnnouncementPolicy (reserved 10.73.0.240-.250) authored by WS-NET in Phase 2.
    l2announcements:
      enabled: true
-    # --- cgroup: k3s ships its own cgroup mount; point Cilium at it, don't auto-mount ---
-    # [parity] MUST equal Ansible cgroup block. Not in the deliverable's headline list, but it IS
-    # part of the working CNI config on k3s: reconciling without it lets Cilium try to auto-mount
-    # cgroupv2 and break on adopt. Kept in parity for a safe Phase-2 adoption.
+    # --- cgroup: point Cilium at the HOST's cgroup2 root, which systemd mounts at /sys/fs/cgroup ---
+    # [parity] MUST equal Ansible cgroup block (roles/cilium/templates/cilium-values.yaml.j2).
+    #
+    # FIXED 2026-10-03 — hostRoot WAS /run/cilium/cgroupv2, AND THAT SILENTLY BROKE SOCKET-LB FOR
+    # EVERY HOST PROCESS. With autoMount disabled, the chart only bind-mounts whatever already exists
+    # at hostRoot ("it's expected that users have mounted cgroup2 filesystem at the specified
+    # cgroup.hostRoot" — cilium 1.20.1 values.yaml). Nothing ever mounted cgroup2 at
+    # /run/cilium/cgroupv2 on these nodes (it is an empty DirectoryOrCreate hostPath), so the agent
+    # mounted one ITSELF from inside its own private cgroup namespace. That mount's root is the
+    # agent container's OWN leaf cgroup, so cil_sock4_connect & co. were attached there and covered
+    # only the agent. Every other host-netns process (the k3s apiserver in
+    # /system.slice/k3s.service, curl over SSH) was untranslated: Service ClusterIPs timed out from
+    # every node, and the node holding the Gateway's L2 lease could not reach the Gateway VIP at all
+    # ("no route to host"). That is why k3s-prod-1's apiserver logged `oidc: authenticator not
+    # initialized` from 2026-09-12 on — kubectl SSO through the kube-vip VIP failed outright.
+    # Measured on k3s-prod-1 before this change: TCP connect to 192.168.2.20:443, 10.43.225.79:443
+    # and 10.43.59.141:8080 SUCCEEDED from inside the agent container (the covered cgroup) and
+    # FAILED from the host, same node, same netns. Full write-up:
+    # decisions/2026-10-03-cilium-cgroup-hostroot.md.
+    #
+    # /sys/fs/cgroup IS a cgroup2 mount in the host's root cgroup namespace on every node (verified
+    # with findmnt on all three), so bind-mounting it gives the agent the real host root. autoMount
+    # stays off because the mount it would create already exists.
    cgroup:
      autoMount:
        enabled: false
-      hostRoot: /run/cilium/cgroupv2
+      hostRoot: /sys/fs/cgroup
    # --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident ---
    # [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1.
    # The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate.
@@ -41,11 +41,13 @@ hubble:
 # [parity] ingress VIP announcement on the flat LAN (docs/09, OQ-1)
 l2announcements:
  enabled: true
-# [parity] k3s provides the cgroupv2 mount; do not auto-mount
+# [parity] the host's cgroup2 root is /sys/fs/cgroup (systemd mounts it); do not auto-mount.
+# NOT /run/cilium/cgroupv2: nothing mounts cgroup2 there, so socket-LB attached to the agent's own
+# cgroup and host processes lost Service/LB translation (2026-10-03 — see helmrelease.yaml).
 cgroup:
  autoMount:
    enabled: false
-  hostRoot: /run/cilium/cgroupv2
+  hostRoot: /sys/fs/cgroup
 # [parity] roll the agent DaemonSet ONE node at a time. Overrides the cilium 1.20.1 chart
 # default of 2, which on 3 nodes lets a single bad render break 2/3 agents at once — as the
 # 2026-09-01 incident did (k8sServiceHost: "" → the `config` init container fell back to the
@@ -0,0 +1,148 @@
+# plex-remote-access — getting Plex reachable from outside the LAN, and proving it
+
+**Status: the in-cluster half is shipped; the last mile is TWO MANUAL STEPS and remote access does
+NOT work until both are done.** Neither can live in this repo.
+**Alert:** none, and none is possible from in-cluster signals — see "Why no alert" below. This is an
+**operator procedure**, not an incident page.
+**Files:** `apps/wave-0/plex-remote.yaml` (the tcp/32400 LoadBalancer + its CiliumNetworkPolicy),
+`clusters/base/cluster-vars.yaml` + `clusters/prod/kustomization.yaml` (`PLEX_L4_VIP`),
+`catalog/render.py` (the `EXTRA_UNITS` entry that places the unit), `catalog/plex.yaml` (Plex's own
+env — note `ADVERTISE_IP` is deliberately NOT set; see "Why not the Gateway URL").
+**Related:** `runbooks/plex-quicksync.md` and `runbooks/plex-claim.md` — the other two Plex
+procedures with a manual last mile.
+
+## What was actually broken (measured 2026-10-03)
+
+Plex was serving correctly and was reachable on the LAN. Two independent things stopped remote
+clients:
+
+1. **Plex advertised no remote connection at all.** In the live `Preferences.xml`:
+   `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while
+   `PublishServerOnPlexOnlineKey="1"`. The server was trying to publish itself to plex.tv and had
+   nothing reachable to publish. Plex's automatic path (UPnP/NAT-PMP) cannot work here: the process
+   lives in a pod on the Cilium overlay (`10.0.2.136`), it cannot speak UPnP to the SOHO router, and
+   a mapping to the pod IP would be meaningless if it could.
+2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` → connection refused.
+   `136.51.115.252:443` → a **self-signed router certificate** (`C=ZZ, CN=%D`), not the Gateway's
+   Let's Encrypt wildcard.
+
+The Service was `ClusterIP` only, so port 32400 had no path off-cluster by any route.
+
+**Caveat on that second measurement, because it is easy to over-read:** it was taken from *inside*
+the LAN, so strictly it proves there is no NAT hairpin (reflection). It does not prove outside-in
+failure. But a router certificate answering 443 means the router's own management UI is bound to the
+WAN port, and that *would* shadow a port-forward for genuine outside traffic too. **Verify from a
+real outside vantage point** (phone on cellular, or an external port checker) before concluding
+either way — this repo has no such vantage point and must not pretend otherwise.
+
+## Step 1 — forward WAN tcp/32400 on the router (operator, off-repo)
+
+    WAN tcp/32400  ->  192.168.2.24  (${PLEX_L4_VIP})  port 32400
+
+`192.168.2.24` is a pinned Cilium LB VIP, ARP-announced on the flat LAN by the existing
+`CiliumL2AnnouncementPolicy default-l2`. It is **pinned on purpose**: this address is typed into the
+router by hand, so it must never move on a reschedule.
+
+While in the router, **check what else is bound to the WAN interface.** If remote management is
+listening on 443 (see the certificate measurement above), move it or disable it — otherwise the
+Gateway's 80/443 forward is shadowed and the Plex *web UI* over the internet stays broken even
+though remote *streaming* works. These are two separate paths and they fail independently.
+
+Confirm the VIP is actually answering on the LAN first, so a failure later is unambiguous:
+
+    ./infra/devcontainer/dev.sh run bash -c \
+      'kubectl --context k3s-prod -n plex get svc plex-l4-tcp -o wide'
+
+`EXTERNAL-IP` must read `192.168.2.24`. If it is `<pending>`, LB-IPAM could not assign — the address
+is outside the pool or already taken; see `clusters/prod/kustomization.yaml` for the pool accounting.
+
+**`192.168.2.24` was never ARP-probed off-cluster.** Only the in-cluster pool accounting was checked
+(IPsTotal 11 / IPsUsed 4 / IPsAvailable 7 on 2026-10-03). If the VIP behaves oddly, probe it from a
+LAN host — `arping 192.168.2.24` — the way `.23` was probed before it was claimed.
+
+## Step 2 — tick "manually specify public port" in Plex (operator, UI only)
+
+**There is no GitOps path to this and no environment variable for it.** Measured in the running
+image on 2026-10-03: a grep of the entire init surface (`/etc/cont-init.d`, `/plex-common.sh`,
+`/usr/local/bin`) for `ManualPortMapping` and `PLEX_PREFERENCE` returned **NONE**. The image writes
+only six preferences, and this is not one of them. It is the same class of manual last mile as the
+`HardwareAcceleratedCodecs` toggle in `runbooks/plex-quicksync.md`.
+
+In the Plex web UI (`https://plex.home.keithjackson.dev`, on the LAN):
+
+**Settings → Remote Access**
+- Ensure **Enable Remote Access** is on. (`PublishServerOnPlexOnlineKey` was already `1`.)
+- Tick **Manually specify public port** and set it to **32400**.
+- **Apply**, then let the panel settle. It should report *"Fully accessible outside your network"*.
+
+Without this tick Plex waits for a NAT mapping it can never obtain and publishes no remote
+connection, so the port-forward in step 1 sits there unused.
+
+## How to verify
+
+Do these in order; each one isolates a different half.
+
+**1. The port is open from outside.** From a phone on cellular, or any off-LAN host:
+
+    curl -sk -o /dev/null -w '%{http_code}\n' https://136.51.115.252:32400/identity
+
+`200` means the forward and the policy are both good. A hang or refusal means step 1.
+
+**2. Plex itself believes it is reachable.** Settings → Remote Access must say *"Fully accessible
+outside your network"*. If it says *"not available outside your network"* while test 1 passes, step 2
+was not applied.
+
+**3. plex.tv is handing out a remote connection.** The decisive test, because it is what clients
+actually consume: sign in to [app.plex.tv](https://app.plex.tv) from an off-LAN network and confirm
+the server appears and plays. A client finds this server through plex.tv's connection list, never
+through DNS.
+
+**4. The policy is admitting the traffic.** If test 1 fails *after* the forward is confirmed, the
+CiliumNetworkPolicy is the next suspect:
+
+    ./infra/devcontainer/dev.sh run bash -c \
+      'kubectl --context k3s-prod -n plex get cnp plex-allow-remote-32400'
+
+See `runbooks/denied-flows.md` for reading actual drops.
+
+## Why no alert
+
+Every in-cluster signal is green whether or not remote access works. The Service has endpoints, the
+pod is `Running`, the policy is applied, and `plex-down` watches
+`kube_deployment_status_replicas_available` — none of which can see a router with a missing
+port-forward or an unticked checkbox. An alert would need an off-LAN prober, which this cluster does
+not have. **Test 3 above is the only honest check, and it is manual.**
+
+## Why not the Gateway URL (`ADVERTISE_IP`)
+
+The alternative was to leave Plex on `ClusterIP` and set
+`ADVERTISE_IP: https://plex.home.keithjackson.dev:443`, which writes Plex's `customConnections`
+preference and would have put the server on plex.tv's list via the existing L7 Gateway — no new WAN
+hole. That mechanism was verified to work on an already-claimed server (the `ADVERTISE_IP` block at
+line 126 of the image's `40-plex-first-run` sits **outside** the claim guard, and `/.firstRunComplete`
+lives on the overlay FS, not the `/config` PVC, so it re-runs every pod start), and the Gateway path
+itself was measured serving Plex correctly: `HTTP 200`, `claimed="1"`, with `401` on every protected
+endpoint.
+
+**The operator chose the native path anyway (2026-10-03), and the reasons are real:** Plex terminates
+its own TLS with its `*.plex.direct` certificate, and video streams bypass the Envoy L7 hop instead
+of being proxied through it. The cost is a new unfiltered WAN port — acceptable because Plex
+authenticates every caller itself. `ADVERTISE_IP` therefore stays unset in `catalog/plex.yaml`, and
+the deferral note there is still accurate.
+
+## Known wart: remote clients are classified as LAN
+
+`apps/wave-0/plex-remote.yaml` uses `externalTrafficPolicy: Cluster`, which SNATs the client to the
+ingress node's address (`192.168.2.1x`). That falls inside the `192.168.0.0/16` entry of
+`ALLOWED_NETWORKS`, so **Plex treats every remote stream as local.** Measured on the Gateway path,
+which SNATs the same way: Plex logged `[10.0.2.214:36040 (Allowed Network (WAN))]`.
+
+This is a **metering** effect, not a security one — authorization is unchanged, as the 401s above
+show — but Plex's remote-quality and bandwidth limits will not bind on these streams.
+
+`Local` would preserve the client IP and fix it. **Do not switch to it.** Cilium's L2 announcer
+elects one lease holder per Service and does *not* follow the backend pod, so the VIP can be
+announced by a node with no Plex pod, black-holing the port while everything reports healthy. That
+exact change cost this LAN an 8-day Minecraft outage in September 2026 — the full write-up is at the
+top of `platform/crafty/service-l4.yaml`. Plex is a single-replica `Recreate` Deployment, so the
+node mismatch is the normal case here, not a rare one.
@@ -161,6 +161,41 @@ platform must render, and cites the requirement each mapping satisfies. Changing
 - **Requirements:** **R-STATE-01** (rebuild survival), **R-STATE-02** (Longhorn replication),
  **R-STATE-04** (bulk tier). (docs/04)

+### `availability` (`replicas`) — workload kind, per-replica volumes, spreading, budget
+
+**Schema 1.10.0.** Availability INTENT. It names no Kubernetes object, no topology key and no
+budget — exactly like `hardware.gpu`, and for the same reason: the mechanism below should be
+replaceable without a catalog edit.
+
+- **Absent, or `replicas: 1`** → a **Deployment**, standalone PVC(s), `strategy: Recreate` when there
+  is storage, no PodDisruptionBudget. Byte-identical to pre-1.10.0 output, so every existing app is
+  untouched.
+- **`replicas: N > 1`** → all of the following, together:
+  | Rendered | Why |
+  |---|---|
+  | **StatefulSet** instead of Deployment | A longhorn PVC is **ReadWriteOnce**. One RWO claim cannot attach on two nodes, so a multi-replica Deployment does not degrade — the second pod never schedules. |
+  | longhorn storage becomes **`volumeClaimTemplates`** | Gives each replica its own claim (`<vol>-<svc>-<ordinal>`, e.g. `config-pihole-0`). The template's `metadata.name` **must** equal the pod volume name the container mounts. The standalone PVC is then **not** emitted — it would be an orphan nothing mounts. |
+  | `bulk` storage stays a **standalone PVC** | A bulk entry is a static, pre-bound NFS PV over data that already exists outside the cluster and is shared read-only. Per-replica copies are neither possible nor wanted. |
+  | **required** `podAntiAffinity` on `kubernetes.io/hostname` | `preferred` would let every replica land on one node and deliver none of the intended node-failure tolerance, while every object still reported healthy. |
+  | `podManagementPolicy: Parallel` | `OrderedReady` serialises first-boot work across replicas; `apps-wave-0` has `timeout: 5m`. There is no ordering requirement between replicas to protect. |
+  | **PodDisruptionBudget**, `minAvailable: 1` | Stops a *voluntary* disruption (node drain, upgrade) taking the last replica. It does nothing about a node dying — that is what the replica count and anti-affinity are for. Never rendered at `replicas: 1`, where `minAvailable: 1` would block every drain of that node forever. |
+  | `observability.livenessWorkload.kind` defaults to **`StatefulSet`** | The `up` SLI reads kube-state-metrics by workload kind. A Deployment-shaped rule against a StatefulSet `absent()`s forever — an alert that can never fire. Deriving the default means bumping the replica count cannot leave a stale rule behind. |
+
+**Two costs, both deliberate, neither hidden:**
+
+1. **N replicas need N schedulable nodes.** On a 3-node cluster, losing two nodes leaves a replica
+   `Pending`; since `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, that **stalls the
+   app tier**. That coincides with having lost etcd quorum on a 3-node control plane, so nothing is
+   reconciling at that point anyway — the stall is a symptom, not a new failure mode.
+2. **Only declare `> 1` for a service whose on-disk state is reproducible from git.** A replica with
+   a fresh volume comes up serving whatever it can bootstrap itself and nothing else. For pihole that
+   prerequisite was done first — the admin password and the adlists were declared in
+   `decisions/2026-10-04-pihole-declare-state.md` — because otherwise a second replica would have
+   served DNS while blocking nothing, with a different password.
+
+**Requirements:** R-PLAT-02 (the workload), R-STATE-01/02 (per-replica replicated volumes),
+R-OBS-01 (the SLI must follow the rendered kind).
+
 ### `hardware` (`gpu`) — host device capability
 - **The field is INTENT, never MECHANISM.** `hardware.gpu` is an **enum of capability classes**
  (today: `intel-quicksync`), not a device path, not a hostPath, not a resource request. The
@@ -2,7 +2,7 @@
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://home.keithjackson.dev/schemas/servicedefinition.schema.json",
  "title": "ServiceDefinition",
-  "description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.9.0 (adds optional `auth.apiRoutes` -- a per-service list of anchored path regexes the forward-auth oauth2-proxy sidecar answers with 401 instead of a 302 to Keycloak, rendered as one repeatable `--api-route` per entry. The `auth` block stays `additionalProperties: false`; this is one named, documented key, not an opened door. Without it there was NO per-app override hook at all, so the only way to answer an app's XHRs with 401 was a renderer-wide constant -- and the path shape is a property of the app: jackett's is `/api/` (33 x 302 vs 2 x 401 measured 2026-09-30, 4 of 6 POSTs answered by the proxy and never reaching the app), radarr's is `/signalr/` (232 x 302 measured 2026-10-01, every one a POST from a left-open browser tab that is never re-prompted), and pihole serves no `/api/` at all. Each entry must be anchored at `^/` and continue with literal path characters, because a too-broad regex turns a plain page load into a 401 and the service looks permanently broken to a logged-out user. Enforced on every gate run by ci/l0.sh's schema stage over catalog/*.yaml, by catalog/render.py refusing to render an unanchored entry, and by conformance/tests/l0-static/oauth2-proxy-api-routes-per-app)). 1.8.0 (adds optional `hardware` - a host HARDWARE CAPABILITY declaration for the container kind, currently one member: `hardware.gpu: intel-quicksync`, meaning the workload needs the host iGPU render device for hardware video transcode. It is an ENUM OF INTENT, not a device path: the renderer satisfies it today with just two things -- a /dev/dri hostPath volume (`DirectoryOrCreate`) and a read-write /dev/dri volumeMount on the app container -- and a later move to the Intel GPU device plugin is then a render.py change with no catalog edit. It sets NO podSecurityContext.supplementalGroups: measured in-container on prod 2026-09-30, the pms-docker cont-init hook adds the app user to the device's group inside the container and `s6-setuidgid` then REPLACES the process group list from /etc/group, discarding anything Kubernetes supplied, so such an entry would be inert and misleading. Renders NOTHING when absent, so every existing app stays byte-identical. Emits no privileged securityContext (Kyverno restrict-host-namespaces would reject it and it is not needed), no nodeSelector (node placement is a separate concern with its own justification) and no resource requests). 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.md already said `internal` renders no public DNS record; nothing enforced it, and six internal services -- jackett, pihole, qbittorrent, radarr, semaphore, sonarr -- were publishing `<name>.home.keithjackson.dev` into the PUBLIC Cloudflare zone answering with the RFC1918 Gateway VIP, which published the internal subnet and a full service inventory and made each name a DNS-rebind primitive. Enforced on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns). 1.6.0 (splits `observability.metrics` from WHERE metrics live: adds optional `metricsEndpoint` (port/name/path/interval) and optional `livenessWorkload` (kind/name). `metrics: true` previously rendered a ServiceMonitor hardcoded to a port named `metrics` that no rendered Service ever defined, so all twelve app monitors discovered ZERO targets and every `up{job=...}` SLI was structurally inert on prod; a monitor is now rendered only from a declared `metricsEndpoint`, which also renders the Service port and the monitoring-namespace ingress allow, and the `up` SLI is expressed against kube-state-metrics -- a signal that exists for every workload without a per-app exporter. `livenessWorkload` names the object whose readiness IS the service, for the case where the running workload is not the one this SD renders). 1.5.0 (adds optional `inClusterClients` — additional in-cluster clients, named by NAMESPACE, admitted to a declared port of this service DIRECTLY, bypassing the shared Gateway. `ingressRoutes` admits only the Gateway proxy identity and `dependencies` only links catalog services, so a PLATFORM controller dialling a Service ClusterIP had no authorable form at all; that gap is what deadlocked proLine truncated
+  "description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.10.0 (adds optional `availability` for the container kind, one member: `availability.replicas`. It is AVAILABILITY INTENT, not a scheduling primitive -- it names no Kubernetes object, no topology key and no budget. 1 (the default) renders a Deployment exactly as before, so every pre-existing app stays byte-identical; >1 means the service must survive the loss of a single node, and the renderer satisfies that with a StatefulSet whose longhorn volumes become per-replica volumeClaimTemplates (a longhorn PVC is ReadWriteOnce and cannot attach on two nodes, so a multi-replica Deployment is UNSCHEDULABLE, not merely suboptimal), REQUIRED podAntiAffinity on kubernetes.io/hostname (preferred would let every replica land on one node and deliver none of the intended tolerance while reporting healthy), podManagementPolicy Parallel, a PodDisruptionBudget of minAvailable 1, and an `up` SLI that follows the rendered kind (a Deployment-shaped rule against a StatefulSet absent()s forever and can never fire). Added for pihole, where LAN DNS -- a critical network service -- was one pod on one node: measured 2026-10-04, losing the node running it meant roughly 6-10 minutes with no LAN resolution (tolerationSeconds 300 plus node-NotReady detection plus a Longhorn RWO force-detach). Deliberately NOT usable until a service's state is reproducible from git, because a replica with a fresh volume otherwise comes up serving but unconfigured -- for pihole that prerequisite was done first (decisions/2026-10-04-pihole-declare-state.md). Capped at 5: a count above the node count can never schedule under the anti-affinity this implies. Mechanism documented in schemas/renderer-contract.md `### availability`). 1.9.0 (adds optional `auth.apiRoutes` -- a per-service list of anchored path regexes the forward-auth oauth2-proxy sidecar answers with 401 instead of a 302 to Keycloak, rendered as one repeatable `--api-route` per entry. The `auth` block stays `additionalProperties: false`; this is one named, documented key, not an opened door. Without it there was NO per-app override hook at all, so the only way to answer an app's XHRs with 401 was a renderer-wide constant -- and the path shape is a property of the app: jackett's is `/api/` (33 x 302 vs 2 x 401 measured 2026-09-30, 4 of 6 POSTs answered by the proxy and never reaching the app), radarr's is `/signalr/` (232 x 302 measured 2026-10-01, every one a POST from a left-open browser tab that is never re-prompted), and pihole serves no `/api/` at all. Each entry must be anchored at `^/` and continue with literal path characters, because a too-broad regex turns a plain page load into a 401 and the service looks permanently broken to a logged-out user. Enforced on every gate run by ci/l0.sh's schema stage over catalog/*.yaml, by catalog/render.py refusing to render an unanchored entry, and by conformance/tests/l0-static/oauth2-proxy-api-routes-per-app)). 1.8.0 (adds optional `hardware` - a host HARDWARE CAPABILITY declaration for the container kind, currently one member: `hardware.gpu: intel-quicksync`, meaning the workload needs the host iGPU render device for hardware video transcode. It is an ENUM OF INTENT, not a device path: the renderer satisfies it today with just two things -- a /dev/dri hostPath volume (`DirectoryOrCreate`) and a read-write /dev/dri volumeMount on the app container -- and a later move to the Intel GPU device plugin is then a render.py change with no catalog edit. It sets NO podSecurityContext.supplementalGroups: measured in-container on prod 2026-09-30, the pms-docker cont-init hook adds the app user to the device's group inside the container and `s6-setuidgid` then REPLACES the process group list from /etc/group, discarding anything Kubernetes supplied, so such an entry would be inert and misleading. Renders NOTHING when absent, so every existing app stays byte-identical. Emits no privileged securityContext (Kyverno restrict-host-namespaces would reject it and it is not needed), no nodeSelector (node placement is a separate concern with its own justification) and no resource requests). 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.mdLine truncated
  "type": "object",
  "additionalProperties": false,
  "required": ["name", "source"],
@@ -139,6 +139,21 @@
        "supplementalGroups": { "type": "array", "items": { "type": "integer" }, "description": "Additional GIDs added to the first process in each container." }
      }
    },
+    "availability": {
+      "type": "object",
+      "description": "Optional availability intent for the container kind: how many replicas of this workload must be running. INTENT ONLY -- it names no Kubernetes object and no scheduling primitive. How the platform satisfies it (workload kind, per-replica volumes, node spreading, disruption budget) is catalog/render.py's business, documented in schemas/renderer-contract.md `### availability`. Absent => replicas 1 and a Deployment, so every pre-existing app renders byte-identical. Ignored for kind=vm (a VM's availability is a different problem with different primitives).",
+      "additionalProperties": false,
+      "minProperties": 1,
+      "properties": {
+        "replicas": {
+          "type": "integer",
+          "description": "How many replicas must be running. 1 (default) renders a Deployment exactly as before. >1 declares that the service must survive the loss of a single node; the renderer then emits a StatefulSet with per-replica volumeClaimTemplates, required per-node anti-affinity, podManagementPolicy Parallel and a PodDisruptionBudget. Only declare >1 for a service whose on-disk state is reproducible from git -- a replica with a fresh volume comes up serving whatever it can bootstrap itself, and nothing else.",
+          "minimum": 1,
+          "maximum": 5,
+          "default": 1
+        }
+      }
+    },
    "hardware": {
      "type": "object",
      "description": "Optional host HARDWARE CAPABILITY the workload needs, for the container kind (docs/13). This field declares INTENT (`what device class this service needs`), never MECHANISM (`which host path to bind-mount`, `which extended resource to request`). That separation is the whole point of the enum: the platform currently satisfies `intel-quicksync` with a /dev/dri hostPath volume and a matching mount on the app container, but a later migration to the Intel GPU device plugin (`gpu.intel.com/i915` resource requests, no hostPath) must be a catalog/render.py change and NOTHING else - no ServiceDefinition may need editing, and no reviewer may have to re-derive per-service device wiring. Do NOT 'simplify' this into a raw device path, a hostPath, or a resource request; that would push a platform implementation detail into the operator-facing contract and make the migration a catalog-wide edit. Absent => no device wiring at all (existing apps render byte-identical). Ignored for kind=vm (a VM gets devices through KubeVirt's own domain.devices).",
Author	SHA1	Message	Date
jakman85andClaude Opus 5	1f4552d409	fix(plex): advertise the plex.direct name, not the bare VIP (server vanished from the plex.tv account) gate / gate (pull_request) Successful in 1m48s Details gate / conformance (pull_request) Has been skipped Details customConnections held two hand-typed URLs and neither could ever complete, so Plex had nothing publishable and the server disappeared from the account: - https://192.168.2.24:32400 -- the server's cert is .130072f20c334d6c8f63d0dfef87309b.plex.direct with NO IP SAN, so bare-IP https fails TLS for every client. Measured: a continuous storm of "CERT: incomplete TLS handshake ... certificate unknown" from real LAN clients arriving through the plex-l4-tcp VIP. - https://plex.home.keithjackson.dev:32400 -- nothing listens; the Gateway serves that host on 443 (measured http=000 against 192.168.2.20:32400). With no completable connection plex.tv published nothing ("Mapped - Not Published"), Plex fell back to Relay, and clients reported "not authorized". Fix: advertise <dashed-ip>.<serverhash>.plex.direct -- public DNS that resolves to the embedded address, so it points at the LAN VIP and matches the wildcard cert. Verified with full TLS validation (no curl -k): 200 + verify OK, where the bare-IP form fails to connect. Declared as ADVERTISE_IP so git owns the value instead of a web form: the image's 40-plex-first-run writes it into customConnections on every pod start, outside the claim guard. Does NOT fix off-LAN access -- the advertised address is RFC1918 and the WAN tcp/32400 forward remains unverified. ManualPortMappingMode=1 with an absent port is left as-is (no GitOps path exists for it). Touches clusters/* -> sensitive paths -> needs the operator's /approve-override, deliberately not self-posted (CLAUDE.md 1). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-04 02:53:01 -05:00
jakman85	c95b1c7121	Merge pull request 'fix(cilium): point cgroup.hostRoot at the host cgroup2 root (socket-LB never covered host processes)' (#42 ) from fix/cilium-cgroup-hostroot into main gate / gate (push) Failing after 1m45s Details gate / conformance (push) Has been skipped Details Reviewed-on: #42	2026-10-04 07:10:31 +00:00
jakman85	ed6afc3d3f	Merge pull request 'docs(decisions): record that PiHole HA is verified live, failover drill included' (#46 ) from docs/pihole-ha-verified into main gate / gate (push) Successful in 1m46s Details gate / conformance (push) Has been skipped Details Reviewed-on: #46	2026-10-04 07:08:19 +00:00
jakman85andClaude Opus 5	254985baa6	docs(decisions): record that PiHole HA is verified live, failover drill included gate / gate (pull_request) Successful in 1m46s Details gate / conformance (pull_request) Has been skipped Details No manifests change. The three entries merged earlier today each say, correctly at the time, that nothing had been reconciled -- which is now stale in exactly the way CLAUDE.md section 2b warns about: a reader would conclude the work is unproven. This entry is the current state and does not edit the earlier files. Measured on live prod after `dce20b7`: apps-wave-0 reconciled in ~90s (inside the 5m timeout the multi-replica entry flagged as a risk); pihole-0 on k3s-prod-3 and pihole-1 on k3s-prod-2, so the required anti-affinity was satisfied; both per-replica claims Bound; the old Deployment and the old unused PVC both pruned; PDB healthy with one disruption allowed. The mountPath fix is confirmed: /etc/pihole is now on /dev/longhorn with 6.9M used on both pods, where it was previously on the overlay. Phase 1's declarations reproduce onto a FRESH volume, which was the whole reason for sequencing it first: 72,234 gravity domains rebuilt on each replica from the declared adlist, and the declared SOPS secret authenticates on BOTH pods ("valid":true via each container's own env, value never printed). FAILOVER DRILL, observed by the operator: a replica was deleted, recreated in ~30 seconds, and DNS never dropped a request. First direct evidence the replica count delivers what it was added for. Re-deriving the SLI from the rendered workload kind mattered concretely: the StatefulSet series returns 2 while the old Deployment series now returns ZERO series, so a stale Deployment-shaped rule would have matched its own absent() branch and fired permanently -- a standing false page. Still unverified and stated as such: node failure (the drill deleted a pod, so the 300s toleration path is untouched), DNS during the cutover itself, two-node loss, and the cluster-wide dependency that replicas cannot address. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-04 02:03:42 -05:00
jakman85	dce20b7be6	Merge pull request 'feat(pihole): two replicas on different nodes, and fix a PVC mounted where pihole never writes' (#45 ) from feat/pihole-multi-replica into main gate / gate (push) Successful in 1m47s Details gate / conformance (push) Has been skipped Details Reviewed-on: #45	2026-10-04 06:50:56 +00:00
jakman85andClaude Opus 5	028853c87c	feat(pihole): two replicas on different nodes, and fix a PVC mounted where pihole never writes gate / gate (pull_request) Successful in 1m47s Details gate / conformance (pull_request) Has been skipped Details Phase 2 of making LAN DNS survive a node failure. Pihole was one pod on one node; measured 2026-10-04, losing that node meant roughly 6-10 minutes with no LAN resolution (tolerationSeconds 300 + NotReady detection + a Longhorn RWO force-detach), while losing the node holding the pihole-dns L2 lease -- a DIFFERENT node -- meant tens of seconds. Two of three nodes were single points of failure with wildly different blast radii. Adds `availability.replicas` to the ServiceDefinition schema (1.10.0). Intent only; the mechanism is render.py's, documented in the renderer contract. 1 or absent renders a Deployment exactly as before -- verified byte-identical across the whole apps/ tree. >1 renders a StatefulSet (a longhorn PVC is RWO, so a multi-replica Deployment never schedules the second pod), per-replica volumeClaimTemplates, REQUIRED hostname anti-affinity, podManagementPolicy Parallel, a minAvailable:1 PodDisruptionBudget, and an SLI that follows the rendered kind (a Deployment-shaped rule against a StatefulSet absent()s forever). Pinned by ~20 new --selftest assertions rather than a conformance change. ALSO FIXES A LIVE BUG. Pihole's config PVC was mounted at the renderer default /data/config, but pihole writes to /etc/pihole. Measured: /etc/pihole (7.1M -- gravity.db, pihole.toml, query DB) was on the OVERLAY, while the 2Gi Longhorn volume held lost+found and 24K of nothing. So every restart discarded all state and re-downloaded 72k gravity domains, and backup.policy has been backing up an empty volume while reporting success. Fixed here because otherwise the new per-replica volumes would hold nothing while state stayed ephemeral -- and because persisting gravity.db is what makes a restart cheap. The /etc/pihole mount is safe on the image's own evidence: start.sh is PID 1 as uid 0 and chowns the directory recursively (no fsGroup needed), and it calls migrate_gravity which creates the database when absent. The gate caught a real regression on the first run: the wave-0 alert unit test fed pihole a Deployment-shaped series. Updated; promtool SUCCESS. Touches no sensitive paths, so this one needs no override. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-04 01:34:52 -05:00
jakman85	17f0c11403	Merge pull request 'feat(pihole): declare the admin password and adlists in git' (#44 ) from feat/pihole-declare-state into main gate / gate (push) Successful in 1m45s Details gate / conformance (push) Has been skipped Details Reviewed-on: #44 Reviewed-by: Keith Jackson <[email protected]>	2026-10-04 06:14:47 +00:00
jakman85andClaude Opus 5	3b1d6ae115	feat(pihole): declare the admin password and adlists in git gate / gate (pull_request) Successful in 1m47s Details gate / conformance (pull_request) Has been skipped Details Prerequisite for running more than one PiHole replica. A second replica with a fresh PVC would have come up serving DNS and blocking nothing, with a different admin password -- answers varying by which pod replied. pihole.toml's own ### CHANGED markers gave the exact inventory: four settings already came from env (declared), one came from the image, and exactly ONE operator-set value lived only on the PVC -- webserver.api.pwhash, a one-way hash nobody could recover. Separately the adlists were undeclared: one list and 72,234 gravity domains that nothing in this repo named. Declares both: - A SOPS secret feeding FTLCONF_webserver_api_password. webserver.api.password takes cleartext and FTL hashes it itself (verified: pihole setpassword IS setFTLConfigValue on that key), so the PVC's pwhash becomes derived state. - A level-triggered postStart hook that reconciles the declared adlist set into gravity.db -- adds, re-enables, and disables (never deletes) undeclared lists, running pihole -g only when something moved. A file would not work: v6's adlists.list import is migration-only, guarded on the DB not existing. The rendered hook was run in the live pod against throwaway copies of gravity.db, exercising all four paths. A fresh DB built from the image's own schema template has ZERO adlists, confirming this is load-bearing rather than cosmetic. Password kept rather than set empty: the pihole Service still publishes port 80 past the oauth2-proxy, closed only by policy, and default-deny has drifted here before. Multi-replica is phase 2 and deliberately not in this branch -- it needs a schema field, a StatefulSet (the config PVC is RWO), anti-affinity and a PDB. Adds a *.sops.yaml -- a sensitive path -- so approval-policy will block pending an operator /approve-override. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-04 01:07:30 -05:00
jakman85	939461120a	Merge pull request 'feat(plex): expose tcp/32400 for remote access via a pinned LoadBalancer VIP' (#43 ) from feat/plex-remote-access-l4 into main gate / gate (push) Failing after 1m56s Details gate / conformance (push) Has been skipped Details Reviewed-on: #43 Reviewed-by: Keith Jackson <[email protected]>	2026-10-04 04:46:34 +00:00
jakman85andClaude Opus 5	fbaeb3e038	feat(plex): expose tcp/32400 for remote access via a pinned LoadBalancer VIP gate / gate (pull_request) Successful in 1m49s Details gate / conformance (pull_request) Has been skipped Details Remote Plex clients had no path in, for two measured reasons: Plex advertised no remote connection at all (customConnections and ManualPortMappingMode both ABSENT while PublishServerOnPlexOnlineKey=1), and its Service was ClusterIP, so 32400 had no off-cluster path. The L7 Gateway was NOT the problem -- it serves Plex correctly (200, claimed=1, 401 on protected endpoints) -- but Plex clients discover servers through plex.tv, not DNS, so an HTTPRoute cannot put this server on a client's connection list. Operator chose the native path over ADVERTISE_IP: Plex terminates its own TLS with .plex.direct and streams bypass the Envoy L7 hop. Adds a hand-written non-catalog unit (the SD schema cannot express a LoadBalancer Service or a pinned lbipam VIP), registered in render.py's EXTRA_UNITS so --emit-waves places it in plex's own wave. externalTrafficPolicy: Cluster, inheriting crafty's 8-day-outage lesson rather than re-deriving it. Known consequence, documented: SNAT puts remote clients inside ALLOWED_NETWORKS, so Plex meters them as LAN. Not an auth change. Remote access does NOT work until the operator does two manual steps that cannot live in this repo -- the WAN tcp/32400 forward and Plex's "manually specify public port" toggle (no env var exists for it; verified NONE in the image's init surface). runbooks/plex-remote-access.md is the procedure. Touches clusters/* -- a sensitive path -- so the approval-policy stage will block pending an operator /approve-override. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-03 23:30:46 -05:00
jakman85andClaude Opus 5.5	9a69f6906a	fix(cilium): point cgroup.hostRoot at the host's real cgroup2 root gate / gate (pull_request) Successful in 1m47s Details gate / conformance (pull_request) Has been skipped Details hostRoot was /run/cilium/cgroupv2 with autoMount disabled, but nothing mounts cgroup2 there. The agent mounted one from inside its own private cgroup namespace, so socket-LB attached to the agent's own cgroup and host processes got no Service/LB translation. k3s-prod-1 (Gateway L2 lease holder + kube-vip VIP) therefore could not reach the Gateway VIP, and its apiserver's OIDC authenticator has failed since 2026-09-12. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-10-03 18:07:18 -05:00
jakman85	7ef2af205d	Merge pull request 'fix(node-baseline): pin the OIDC issuer to the Gateway VIP on prod nodes' (#41 ) from fix/node-keycloak-hosts-pin into main gate / gate (push) Successful in 1m45s Details gate / conformance (push) Has been skipped Details Reviewed-on: #41	2026-10-03 22:46:38 +00:00
jakman85andClaude Opus 5.5	788eaeae62	fix(node-baseline): pin the OIDC issuer to the Gateway VIP on prod nodes gate / gate (pull_request) Successful in 1m46s Details gate / conformance (pull_request) Has been skipped Details After public DNS moved to the WAN IP (#40), prod nodes resolve keycloak.home.keithjackson.dev to a WAN address the router does not hairpin, so the apiserver could not reach its OIDC issuer. Adds an opt-in node_baseline_hosts_pins /etc/hosts block; prod pins keycloak to GATEWAY_VIP. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-10-03 17:39:37 -05:00
jakman85	576e1abb19	Merge pull request 'feat(dns): publish Gateway names to the WAN via in-cluster DDNS' (#40 ) from feat/public-dns-wan-target into main gate / gate (push) Successful in 1m45s Details gate / conformance (push) Has been skipped Details Reviewed-on: #40	2026-10-03 22:25:13 +00:00