docs(obs): record that the gate-sidecar scrape works — four places still said it did not

PR #49 (merged 21c88b0) fixed the blocked scrape and it is verified working on prod
2026-10-05: all 6 targets up == 1, health=up, lastError='', lastScrapeDuration 0.001-0.006s
against 10.0s and "context deadline exceeded" before. oauth2_proxy_* is in the TSDB for
these namespaces for the first time (5 metric names, 148 series).

Four places in the tree still asserted the pre-fix state as current, in the present tense,
in the files a reader consults first:

  platform/oauth2-proxy/podmonitor.yaml  "READ THIS BEFORE BELIEVING THE TARGETS ARE UP:
    THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY" and "SO THE TARGETS BELOW WILL READ
    up == 0 UNTIL AN INGRESS ALLOWANCE LANDS". The first actively instructs the reader to
    disbelieve a correct up == 1. Now a STATUS block with the post-fix numbers, with the
    diagnosis kept below it under an explicit HISTORY heading because it is why the fix is
    shaped the way it is.
  catalog/render.py:159  "THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED".
  catalog/render.py:1365  the same claim on the sidecar arg.
  platform/oauth2-proxy/README.md  its subsections are dated and so are honest records, but
    a reader hits the stale body long before any correction. A SUPERSEDED pointer now sits
    under the heading and an Update 2026-10-05 section carries the before/after.

The "not scraped" claims are corrected; the "a flag alone is not enough" warning is KEPT
deliberately, because that part is still true and "flag without ingress allowance" is
exactly what caused the 63.7h outage.

Two substantive errors fixed while there, not just tense. The README's "three things are
needed" listed a Service port and a ServiceMonitor — wrong, a PodMonitor targets pod ports
directly, which is why no Service exposes 44180 and the renderer needed no Service change;
following it would have sent someone to edit Service rendering for nothing. And
podmonitor.yaml deferred the fix to "the catalog/renderer, in flight on another branch" —
no such branch existed; it now names r_monitoring_ingress_policy, with a new CRAFTY section
explaining the one namespace this file cannot help.

Comments only in render.py, so the rendered output is byte-identical: no apps/ file and no
golden changes, and therefore no sensitive path and no human approval needed — the opposite
of the PR this documents.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

apps/wave-0/jackett.yaml

@@ -244,6 +244,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): jackett-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: jackett-allow-monitoring-scrape
+  namespace: jackett
+  labels:
+    app.kubernetes.io/name: jackett
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: jackett
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): jackett-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

apps/wave-0/kustomization.yaml

@@ -14,10 +14,12 @@ resources:
 - nextcloud-db-backup-credentials-prod.sops.yaml
 - nextcloud-db-backup-credentials.sops.yaml
 - nextcloud-db.yaml
+- pihole-bootstrap-secret.sops.yaml
 - pihole-dns.yaml
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
 - plex-claim-secret.sops.yaml
+- plex-remote.yaml
 - plex.yaml
 - qbittorrent-bootstrap-secret.sops.yaml
 - qbittorrent-oauth2-proxy-credentials.sops.yaml

apps/wave-0/pihole-bootstrap-secret.sops.yaml

@@ -0,0 +1,72 @@
+# =================================================================================================
+# PiHole WEB/API ADMIN PASSWORD — the one piece of PiHole state that was NOT declared anywhere.
+# =================================================================================================
+# WHY THIS FILE EXISTS. Measured on live prod 2026-10-04, `pihole.toml` marks every non-default
+# setting with `### CHANGED`, and names the ones that came from the environment `### CHANGED (env)`.
+# The complete list was:
+#
+#     dns.upstreams          ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     dns.listeningMode      ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     dns.rateLimit.count    ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     misc.dnsmasq_lines     ### CHANGED (env)   <- declared, catalog/pihole.yaml
+#     files.macvendor        ### CHANGED         <- set by the image itself, container path
+#     webserver.api.pwhash   ### CHANGED         <- SET OUT OF BAND. Declared NOWHERE. This file.
+#
+# So exactly ONE operator-set value lived only on the Longhorn PVC: the admin password, as a
+# one-way BALLOON-SHA256 hash. Nobody could recover it (a hash does not reverse), nothing in git
+# produced it, and a rebuild from git would have come up with a DIFFERENT password — or none.
+#
+# THE REASON THIS BLOCKS HA, which is why it is fixed first. The next step is running more than one
+# PiHole replica, and a replica with a fresh PVC would come up with its own unrelated password. The
+# admin UI would then accept or reject the same credential depending on which pod answered. Declaring
+# it here makes every replica, and every rebuild, identical by construction.
+#
+# MECHANISM: `webserver.api.password` is a real, settable FTL config key — verified in the running
+# image, where `pihole setpassword` is itself just `setFTLConfigValue "webserver.api.password"` with
+# the comment "pihole-FTL will automatically hash the password". So FTL takes CLEARTEXT on this key
+# and writes the hash into `webserver.api.pwhash` itself. catalog/pihole.yaml therefore passes this
+# value as `FTLCONF_webserver_api_password`, and the hash on the PVC becomes a derived artefact
+# rather than the source of truth.
+#
+# DEFENCE IN DEPTH, NOT THE ONLY GATE. The admin UI is already behind the oauth2-proxy sidecar
+# (`--allowed-group=platform-admins`), so this password is a SECOND gate. It is kept rather than set
+# empty because the `pihole` Service still publishes port 80 (PiHole direct, bypassing the proxy on
+# :4180) and only the CiliumNetworkPolicy closes it — and the default-deny policy in this repo has a
+# documented drift history. An unauthenticated admin UI that can rewrite LAN DNS is not a good thing
+# to leave resting on one policy staying correct. Narrowing that Service port is tracked follow-up.
+#
+# ROTATION: `sops` this file in your own terminal, change the value, commit. Flux re-applies the
+# Secret, the pod restarts, FTL re-hashes. Do NOT `sops -d` it into a shared terminal or a log
+# (CLAUDE.md section 6); read it in an editor. The value below was generated randomly at authoring
+# time and has never been printed to a transcript.
+apiVersion: v1
+kind: Secret
+metadata:
+    name: pihole-bootstrap
+    namespace: pihole
+    labels:
+        app.kubernetes.io/name: pihole
+type: Opaque
+stringData:
+    #ENC[AES256_GCM,data:TMOuRyJeb5IVn3lZPXY7lgh1KGxu7GY9+WIOLHpMB50JqzmM2tHIaWUWKTt1JtZoX/E5/Y7WCUTnfkJLhRBN2zV2MwWRBSLYVy5CMg7xV8PTLRZJYAVUYS5Trg==,iv:q+DpfT6PQfY5ZuNsmP7Ddl+PDLjSm89fJICAhYoW64k=,tag:axijaO3DaxIZgZ3LeK2hXA==,type:comment]
+    webui-password: ENC[AES256_GCM,data:7FcK/bUW79M46f/MlouAmATKbMnoUTNuMUjtqJ5vpU4=,iv:I8DBFB39uXZV42Ie2IPwSefmMgdKEtiDknfc9Tu7h5E=,tag:L423784UrDU3TvylpHc6KQ==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0MkRSNjZLZlpXYkcvaVJL
+            cjlqb2wrcWREZnZLdGpsTHNPWEpseHlRZnpjCnJVM1FDQ2M2dXlaS1JBZ2k2V1BP
+            Yi9QUDZSMzZsL09pTi93RTdjdGxtY2MKLS0tIDIrQTZIVWdhMGIxSCtlUFN2Mlho
+            SWVybld4VjJyaHRSKzZZU0xsUUV2SEEKbDYHV/qBZVspK7DpBW3vYjRMJ/HmAN+9
+            G9bOlNEp0jAtcNZEn6lpQ9ORgvmIrhJHoBQRLabdMWzNWSZsNn56tA==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-10-04T06:01:27Z"
+    mac: ENC[AES256_GCM,data:rhTMTOwLOtHXHgqI5M2ZW/6w+UpLpxBlTk7NoN0wmGSqOcE+OGrXA1DhkZrJweyQT0RSNSSBm/6QMztLP81eepOzdp2rEkwvbQPK/l6K9g16Ww9c1IatS3NQsKEB8AoMDx7jBQ6exXYgggPx3MJsNYbE+WyloQ96U9KPdrecAM8=,iv:n34v2zX9D5xsaR1PFCH94mNMiuRfhUyI2lseUCoqiTA=,tag:9YmN9nIVLXfsGYIb0bUl2w==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

apps/wave-0/pihole.yaml

@@ -35,34 +35,18 @@ spec:
     - protocol: TCP
       port: 53
 ---
-# PVC pihole-config (R-STATE)
+# StatefulSet (R-PLAT-02)
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: pihole-config
-  namespace: pihole
-  labels:
-    app.kubernetes.io/name: pihole
-spec:
-  accessModes:
-  - ReadWriteOnce
-  storageClassName: longhorn
-  resources:
-    requests:
-      storage: 2Gi
----
-# Deployment (R-PLAT-02)
 apiVersion: apps/v1
-kind: Deployment
+kind: StatefulSet
 metadata:
   name: pihole
   namespace: pihole
   labels:
     app.kubernetes.io/name: pihole
 spec:
-  replicas: 1
+  replicas: 2
-  strategy:
+  serviceName: pihole
-    type: Recreate
+  podManagementPolicy: Parallel
   selector:
     matchLabels:
       app.kubernetes.io/name: pihole
@@ -89,10 +73,22 @@ spec:
         - name: FTLCONF_dns_rateLimit_count
           value: '0'
         - name: FTLCONF_misc_dnsmasq_lines
-          value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}
+          value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
+        - name: FTLCONF_webserver_api_password
+          valueFrom:
+            secretKeyRef:
+              name: pihole-bootstrap
+              key: webui-password
         volumeMounts:
         - name: config
-          mountPath: /data/config
+          mountPath: /etc/pihole
+        lifecycle:
+          postStart:
+            exec:
+              command:
+              - /bin/bash
+              - -c
+              - "# Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and\n# does NOT run a gravity rebuild.\nset -u\nGRAVITY=/etc/pihole/gravity.db\n\n# THE DECLARED SET \u2014 one URL per line, the source of truth for what PiHole blocks.\n# To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.\nDECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'\n\n# On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,\n# and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.\ni=0\nwhile [ ! -s \"$GRAVITY\" ] && [ \"$i\" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done\nif [ ! -s \"$GRAVITY\" ]; then\n  echo \"pihole-adlists: gravity.db absent after 180s - leaving adlists untouched\" >&2\n  exit 0\nfi\n\nsql() { pihole-FTL sqlite3 \"$GRAVITY\" \"$1\"; }\nCHANGED=0\n\n# 1) Every declared URL must be present AND enabled.\necho \"$DECLARED\" | while IFS= read -r url; do\n  [ -n \"$url\" ] || continue\n  printf '%s\\n' \"$url\"\ndone > /tmp/declared.txt\nwhile IFS= read -r url; do\n  [ -n \"$url\" ] || continue\n  # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.\n  have=$(sql \"select count(*) from adlist where address='$url';\")\n  if [ \"$have\" = \"0\" ]; then\n    echo \"pihole-adlists: adding $url\"\n    sql \"insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');\"\n    CHANGED=1\n  else\n    en=$(sql \"select enabled from adlist where address='$url';\")\n    if [ \"$en\" != \"1\" ]; then\n      echo \"pihole-adlists: re-enabling $url\"\n      sql \"update adlist set enabled=1 where address='$url';\"\n      CHANGED=1\n    fi\n  fi\ndone < /tmp/declared.txt\n\n# 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).\nsql \"select address from adlist where enabled=1;\" | while IFS= read -r addr; do\n  [ -n \"$addr\" ] || continue\n  if ! grep -qxF \"$addr\" /tmp/declared.txt; then\n    echo \"pihole-adlists: disabling undeclared $addr\"\n    sql \"update adlist set enabled=0 where address='$addr';\"\n    echo changed >> /tmp/adlist-changed\n  fi\ndone\n[ -s /tmp/adlist-changed ] && CHANGED=1\nrm -f /tmp/adlist-changed /tmp/declared.txt\n\n# 3) Rebuild gravity ONLY if the declared set actually moved.\nif [ \"$CHANGED\" = \"1\" ]; then\n  echo \"pihole-adlists: declared set changed - running gravity update\"\n  pihole -g || echo \"pihole-adlists: gravity update failed; existing gravity.db retained\" >&2\nelse\n  echo \"pihole-adlists: already matches the declared set - nothing to do\"\nfi\nexit 0\n"
       - name: oauth2-proxy
         image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
         imagePullPolicy: IfNotPresent
@@ -163,10 +159,39 @@ spec:
             port: http
           initialDelaySeconds: 5
           periodSeconds: 15
-      volumes:
+      affinity:
-      - name: config
+        podAntiAffinity:
-        persistentVolumeClaim:
+          requiredDuringSchedulingIgnoredDuringExecution:
-          claimName: pihole-config
+          - labelSelector:
+              matchLabels:
+                app.kubernetes.io/name: pihole
+            topologyKey: kubernetes.io/hostname
+  volumeClaimTemplates:
+  - metadata:
+      name: config
+      labels:
+        app.kubernetes.io/name: pihole
+    spec:
+      accessModes:
+      - ReadWriteOnce
+      storageClassName: longhorn
+      resources:
+        requests:
+          storage: 2Gi
+---
+# PodDisruptionBudget — keep one replica through a drain (availability.replicas)
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+  name: pihole
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+spec:
+  minAvailable: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: pihole
 ---
 # Service
 apiVersion: v1
@@ -207,6 +232,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): pihole-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: pihole-allow-monitoring-scrape
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: pihole
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): pihole-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -269,11 +315,11 @@ spec:
   - name: pihole.slis
     rules:
     - alert: pihole-down
-      expr: kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"} == 0 or absent(kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"})
+      expr: kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"} == 0 or absent(kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"})
       for: 5m
       labels:
         severity: critical
         service: pihole
       annotations:
         summary: pihole is down
-        description: 'pihole: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
+        description: 'pihole: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready).'

apps/wave-0/plex-remote.yaml

@@ -0,0 +1,126 @@
+---
+# =================================================================================================
+# PLEX REMOTE ACCESS — raw tcp/32400 north-south exposure, and the policy that admits it.
+# =================================================================================================
+# HAND-WRITTEN, NOT RENDERED. A non-catalog app-tier unit anchored to the `plex` ServiceDefinition
+# and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS table — the same
+# mechanism gitea-actions and pihole-dns use. It is hand-written for the SAME reason those are:
+# schemas/servicedefinition.schema.json is `additionalProperties: false` and has NO field for a
+# type=LoadBalancer Service, for a pinned lbipam VIP, or for raw-TCP north-south exposure at all.
+# `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator.
+#
+# WHY THE L7 GATEWAY IS NOT ENOUGH HERE — and it is NOT because the Gateway fails to serve Plex.
+# Measured 2026-10-03 through the Gateway VIP: `GET https://plex.home.keithjackson.dev/identity`
+# returned HTTP 200 with `claimed="1"` and the real machineIdentifier, and the protected endpoints
+# (/library/sections, /status/sessions, /:/prefs) all returned 401. The L7 path WORKS.
+#
+# The problem is DISCOVERY, not proxying. Plex apps (phone, TV, Roku) do not resolve
+# `plex.${BASE_DOMAIN}`; they ask plex.tv for the server's connection list. A server gets onto that
+# list one of two ways: it advertises a custom URL (`customConnections`, the ADVERTISE_IP env), or it
+# publishes a reachable public IP:port for its own tcp/32400. Measured in the live Preferences.xml on
+# 2026-10-03: `customConnections` ABSENT, `ManualPortMappingMode` ABSENT, while
+# `PublishServerOnPlexOnlineKey="1"` — the server was trying to publish and had nothing reachable to
+# publish, so remote clients saw NO connection at all. The operator chose the NATIVE path (this
+# file) over advertising the Gateway URL: Plex then terminates its own TLS with its
+# `*.plex.direct` certificate and video streams bypass the Envoy L7 hop entirely.
+#
+# VIP: ${PLEX_L4_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.57, prod 192.168.2.24), PINNED via
+# lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
+# (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the existing
+# CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config is needed;
+# this Service just consumes the pool. PINNING IS NOT OPTIONAL: the address is typed into a WAN
+# port-forward on the SOHO router, which is hand-maintained and off-repo, so it can never move.
+# 192.168.2.24 verified FREE live 2026-10-03 — `default-pool` reported IPsTotal 11 / IPsUsed 4 /
+# IPsAvailable 7, and the only four allocations were .20 (cilium-gateway-shared), .21 (unifi-l4-*),
+# .22 (crafty-l4-*) and .23 (pihole-dns-*).
+#
+# ONE SERVICE, TCP ONLY. Unlike unifi/crafty/pihole-dns this needs no second Service and no
+# lbipam.cilium.io/sharing-key: Plex's remote access is tcp/32400 and nothing else. (Plex's
+# DLNA/GDM discovery uses UDP, but that is LAN-only multicast and is not part of remote access.)
+#
+# NO `loadBalancerSourceRanges`, DELIBERATELY — and this is the one place this file diverges from
+# its closest sibling. apps/wave-0/pihole-dns.yaml pins its source range to ${LAN_CIDR} because a
+# resolver must never answer the internet. This Service is the OPPOSITE case: admitting remote
+# clients from arbitrary internet addresses is its entire purpose, so there is no range to narrow
+# to. Plex authenticates every caller itself (measured above: 401 on every protected endpoint
+# without a token), which is what makes an unfiltered port defensible here.
+#
+# NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
+# `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their VIP as a public
+# Cloudflare record. This Service must not, for pihole-dns's reason: ${PLEX_L4_VIP} is an RFC1918
+# address that is useless in public DNS. Remote clients get this server's address from plex.tv, not
+# from the zone. The existing public `plex.${BASE_DOMAIN}` record is published from the rendered
+# HTTPRoute via the gateway source and is NOT affected by this file — the web UI keeps working
+# through the Gateway exactly as before.
+#
+# -------------------------------------------------------------------------------------------------
+# externalTrafficPolicy: Cluster — inherited from crafty's 8-DAY LAN OUTAGE, not re-derived.
+# -------------------------------------------------------------------------------------------------
+# `Local` is the tempting choice because it preserves the client source IP, which this service
+# actually has a use for (see the allowedNetworks note below). It is still WRONG here, for the
+# reason platform/crafty/service-l4.yaml documents at length: Cilium's L2 announcer elects ONE lease
+# holder per Service and does NOT follow the backend pod, so with `Local` the VIP can be announced
+# by a node that has no Plex pod, which black-holes the port while every object still reports
+# healthy. Plex is a single-replica Deployment with `strategy: Recreate` pinned to wherever the
+# scheduler put it, so the mismatch is the normal case, not a rare one. Take the SNAT.
+#
+# KNOWN CONSEQUENCE OF THAT CHOICE, stated rather than discovered later: `Cluster` SNATs the client
+# to the ingress NODE's address (192.168.2.1x), which falls inside the `192.168.0.0/16` entry of
+# catalog/plex.yaml's ALLOWED_NETWORKS. Plex will therefore classify every REMOTE client as LAN.
+# Measured 2026-10-03 on the Gateway path, which SNATs the same way: Plex logged the request as
+# `[10.0.2.214:36040 (Allowed Network (WAN))]`. This is a METERING effect, not a security one —
+# authorization is unchanged (401s above), but Plex's remote-quality/bandwidth limits will not bind
+# on these streams because it believes they are local. Narrowing ALLOWED_NETWORKS cannot fix it
+# without also de-classifying genuine LAN clients, so it is accepted.
+apiVersion: v1
+kind: Service
+metadata:
+  name: plex-l4-tcp
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+  annotations:
+    lbipam.cilium.io/ips: "${PLEX_L4_VIP}"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Cluster
+  selector:
+    app.kubernetes.io/name: plex
+  ports:
+    - name: plex-remote
+      port: 32400
+      targetPort: 32400
+      protocol: TCP
+---
+# INGRESS allow for the raw port (R-NET-01/02).
+#
+# The rendered plex.yaml already carries `plex-allow-gateway-ingress` (fromEntities: ingress ->
+# 32400) and three reciprocal allows for sonarr/radarr/seerr. CiliumNetworkPolicies are ADDITIVE, so
+# this file adds the off-cluster case without touching those.
+#
+# `world` is Cilium's entity for anything outside the cluster, which is what a remote Plex client is.
+# `host` and `cluster` mirror platform/crafty/networkpolicy.yaml's game-port rule and are what make
+# this work under `externalTrafficPolicy: Cluster`: after SNAT the packet's source is the ingress
+# NODE, which Cilium classifies as `host` (same node) or `remote-node` (a different one) — NOT
+# `world`. Listing `world` alone would compile cleanly and then drop every remote stream, so all
+# four entities are named.
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: plex-allow-remote-32400
+  namespace: plex
+  labels:
+    app.kubernetes.io/name: plex
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: plex
+  ingress:
+    - fromEntities:
+        - world
+        - cluster
+        - host
+        - remote-node
+      toPorts:
+        - ports:
+            - { port: "32400", protocol: TCP }

apps/wave-0/plex.yaml

@@ -137,6 +137,20 @@ spec:
             secretKeyRef:
               name: plex-claim
               key: PLEX_CLAIM
+        - name: ADVERTISE_IP
+          value: https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400
+        - name: PLEX_TLS_CRT
+          valueFrom:
+            secretKeyRef:
+              name: plex-vip-tls
+              key: tls.crt
+              optional: true
+        - name: PLEX_TLS_KEY
+          valueFrom:
+            secretKeyRef:
+              name: plex-vip-tls
+              key: tls.key
+              optional: true
         resources:
           limits:
             gpu.intel.com/i915: 1
@@ -146,6 +160,14 @@ spec:
         - name: media
           mountPath: /media
           readOnly: true
+        lifecycle:
+          postStart:
+            exec:
+              command:
+              - /bin/bash
+              - -c
+              - "# NOTE \u2014 LOAD-BEARING: SHELL VARIABLES HERE ARE WRITTEN BARE, NEVER BRACE-WRAPPED.\n# This whole script is rewritten by Flux postBuild envsubst BEFORE it ever runs, so a\n# brace-wrapped name is read as a cluster-vars knob: an undeclared one is a HARD\n# Kustomization failure under StrictPostBuildSubstitutions, and a declared one would be\n# replaced with its value. Shell vars are therefore `$PX`, `$TOKEN`, `$P12` \u2014 bare. Only\n# BASE_DOMAIN below is brace-wrapped, because it IS a knob and IS meant to be\n# substituted. Command substitution `$(cmd)` is left alone and needs no escaping. A shell\n# variable written with braces would be blanked or would fail the whole Kustomization \u2014\n# this repo has already lost a gitea OIDC bootstrap to exactly that mistake, and\n# catalog/render.py now refuses to emit it (which is how this very comment got caught).\n# DELIBERATELY NO `set -u`. The TLS env vars are declared `optional: true`, so when the\n# certificate has not issued yet they are genuinely UNSET, not empty \u2014 and under `set -u`\n# the first `[ -n \"$PLEX_TLS_CRT\" ]` would abort the script, which for a postStart hook\n# means kubelet KILLS THE CONTAINER. That is the precise failure the `optional` flag exists\n# to prevent, so turning it back on here would defeat the whole design. (The alternative,\n# a brace-wrapped `:-` default, would need envsubst escaping \u2014 the trap the note above\n# describes.) Every value below is explicitly guarded instead.\nPX=\"/config/Library/Application Support/Plex Media Server/Preferences.xml\"\nCERTDIR=/config/certs\nP12=$CERTDIR/plex-vip.p12\n# The PKCS#12 passphrase protects nothing and is deliberately not a secret: Plex needs it\n# in cleartext in customCertificateKey, and it and the bundle sit on the same PVC. It is a\n# container-format requirement, not an access control.\nP12PASS=plexvip\nLANNETS='10.0.0.0/8,192.168.0.0/16'\nCERTDOMAIN=plex-vip.${BASE_DOMAIN}\nAPI=http://127.0.0.1:32400\n\n# (1) Build the PKCS#12 bundle from the PEM the Secret supplies. Rebuilt on every start so\n# a cert-manager RENEWAL is picked up with no extra moving part.\nif [ -n \"$PLEX_TLS_CRT\" ] && [ -n \"$PLEX_TLS_KEY\" ]; then\n  mkdir -p $CERTDIR && chmod 700 $CERTDIR\n  printf '%s' \"$PLEX_TLS_CRT\" > $CERTDIR/tls.crt\n  printf '%s' \"$PLEX_TLS_KEY\" > $CERTDIR/tls.key\n  chmod 600 $CERTDIR/tls.key\n  if openssl pkcs12 -export -out $P12.tmp -inkey $CERTDIR/tls.key \\\n       -in $CERTDIR/tls.crt -passout pass:$P12PASS 2>/dev/null; then\n    mv -f $P12.tmp $P12\n    chown 1000:1000 $P12 2>/dev/null || true\n    chmod 600 $P12\n    echo \"[plex-settings] PKCS#12 bundle written\"\n  else\n    rm -f $P12.tmp\n    echo \"[plex-settings] WARN: pkcs12 export failed; keeping any previous bundle\" >&2\n  fi\n  rm -f $CERTDIR/tls.key\nelse\n  echo \"[plex-settings] WARN: TLS env empty; certificate not installed this start\" >&2\nfi\n\n# (2) Wait, bounded (~90s), for the local API. /identity needs no auth.\ni=0\nwhile [ \"$i\" -lt 45 ]; do\n  if [ \"$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 $API/identity 2>/dev/null)\" = \"200\" ]; then\n    break\n  fi\n  i=$(expr $i + 1); sleep 2\ndone\n\n# (3) The prefs API needs the server token. It already lives in Preferences.xml on this\n# PVC; it is never echoed here, and Plex redacts tokens in its own request log. An\n# UNCLAIMED server has none \u2014 a normal state on a fresh /config \u2014 so skip, do not fail.\nTOKEN=$(grep -o 'PlexOnlineToken=\"[^\"]*\"' \"$PX\" 2>/dev/null | head -1 | cut -d'\"' -f2)\nif [ -z \"$TOKEN\" ]; then\n  echo \"[plex-settings] not claimed yet \u2014 skipping prefs, will retry next start\" >&2\n  exit 0\nfi\n\n# Percent-encode what our values actually contain. Plex takes prefs as QUERY parameters on\n# a PUT, so a raw '/' or ',' would be read as structure rather than data.\nenc() {\n  printf '%s' \"$1\" | sed -e 's|%|%25|g' -e 's|/|%2F|g' -e 's|,|%2C|g' \\\n                         -e 's|:|%3A|g' -e 's| |%20|g'\n}\nsetpref() {\n  code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 -X PUT\
+                \ \\\n    -H \"X-Plex-Token: $TOKEN\" \"$API/:/prefs?$1=$(enc \"$2\")\" 2>/dev/null)\n  if [ \"$code\" = \"200\" ]; then\n    echo \"[plex-settings] set $1\"\n  else\n    echo \"[plex-settings] WARN: $1 -> HTTP $code\" >&2\n  fi\n}\n\n# (4) THE LAN DEFINITION \u2014 the reason this hook was written. Without it Plex has NO notion\n# of a local network and tags EVERY client remote. Measured 2026-10-04: 468 requests\n# logged \"Allowed Network (WAN)\" and ZERO \"(LAN)\", including Seerr from inside the\n# cluster, which is why the server showed as Remote to LAN clients. Note allowedNetworks\n# (the ALLOWED_NETWORKS env) only waives AUTH \u2014 it does NOT make a client local. Two\n# different settings; only one of them has an env hook.\nsetpref LanNetworksBandwidth \"$LANNETS\"\n\n# (5) THE CERTIFICATE, which is what makes the `https` in ADVERTISE_IP servable. Set only\n# if the bundle exists \u2014 pointing Plex at a missing file would break its TLS listener.\nif [ -s \"$P12\" ]; then\n  setpref customCertificatePath \"$P12\"\n  setpref customCertificateKey \"$P12PASS\"\n  setpref customCertificateDomain \"$CERTDOMAIN\"\nelse\n  echo \"[plex-settings] WARN: no bundle; leaving certificate prefs untouched\" >&2\nfi\nexit 0\n"
       volumes:
       - name: config
         persistentVolumeClaim:

apps/wave-0/qbittorrent.yaml

@@ -319,6 +319,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): qbittorrent-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: qbittorrent-allow-monitoring-scrape
+  namespace: qbittorrent
+  labels:
+    app.kubernetes.io/name: qbittorrent
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: qbittorrent
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): qbittorrent-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

apps/wave-0/tests/app-slis.test.yaml

@@ -1,6 +1,8 @@
 # promtool rule-unit tests for the wave-0 per-app `<app>-down` SLI rules.
 # Manifests under test: every PrometheusRule in apps/wave-0/ — gitea, jackett, pihole, plex,
-# qbittorrent (container/Deployment) and vm-demo (KubeVirt VMI). The harness discovers this file by
+# qbittorrent (container/Deployment), pihole (container/StatefulSet — `availability.replicas: 2`, so
+# its SLI reads kube_statefulset_status_replicas_ready; see schemas/renderer-contract.md
+# `### availability`) and vm-demo (KubeVirt VMI). The harness discovers this file by
 # the SUITE convention (`<dir>/tests/*.test.yaml`) and flattens those RENDERED manifests, so these
 # tests run against the exact expressions Flux applies, never a copy.
 #
@@ -41,7 +43,7 @@ tests:
         values: '1+0x20'
       - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
         values: '1+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}'
         values: '1+0x20'
       - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
         values: '1+0x20'
@@ -81,7 +83,7 @@ tests:
         values: '1+0x20'
       - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
         values: '1+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}'
         values: '1+0x20'
       - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
         values: '0+0x20'
@@ -132,7 +134,7 @@ tests:
         values: '0+0x20'
       - series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
         values: '0+0x20'
-      - series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
+      - series: 'kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}'
         values: '0+0x20'
       - series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
         values: '1+0x20'
@@ -170,10 +172,10 @@ tests:
               severity: critical
               service: pihole
               namespace: pihole
-              deployment: pihole
+              statefulset: pihole
             exp_annotations:
               summary: "pihole is down"
-              description: "pihole: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
+              description: "pihole: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
       - eval_time: 10m
         alertname: qbittorrent-down
         exp_alerts:

apps/wave-1/radarr.yaml

@@ -347,6 +347,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): radarr-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: radarr-allow-monitoring-scrape
+  namespace: radarr
+  labels:
+    app.kubernetes.io/name: radarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: radarr
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): radarr-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

apps/wave-1/sonarr.yaml

@@ -345,6 +345,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: sonarr-allow-monitoring-scrape
+  namespace: sonarr
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: sonarr
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

catalog/pihole.yaml

@@ -12,6 +12,42 @@ storage:
   - name: config
     size: 2Gi
     class: longhorn           # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
+    # THE PVC WAS MOUNTED SOMEWHERE PIHOLE NEVER WRITES, AND SO HELD NOTHING AT ALL. Without an
+    # explicit mountPath the renderer defaults to /data/<name>, i.e. /data/config — but PiHole keeps
+    # every piece of its state in /etc/pihole. MEASURED on live prod 2026-10-04: `/etc/pihole` (7.1 M
+    # — gravity.db, pihole.toml, the query DB) was on the **overlay** filesystem, i.e. the
+    # container's EPHEMERAL layer, while `/dev/longhorn/pvc-849edb86...` sat mounted at
+    # /data/config holding `lost+found` and 24 K of nothing. Three consequences, all live until now:
+    #   * every restart silently discarded PiHole's whole state, re-downloading 72 k gravity domains
+    #     and regenerating the admin password hash;
+    #   * `backup.policy: daily-offcluster` below has been faithfully backing up an EMPTY volume,
+    #     which is worse than no backup because it reports success;
+    #   * it is also why phase 1's password/adlist declaration mattered more than it first appeared —
+    #     that state was not merely undeclared, it was ephemeral.
+    # This is the same class of bug catalog/plex.yaml's `config` entry documents ("Without this the
+    # PVC sat unused and Plex wrote to the container's EPHEMERAL layer").
+    #
+    # /etc/pihole IS THE IMAGE'S INTENDED MOUNT POINT, and an EMPTY volume there self-initialises.
+    # Verified in the running image, not assumed:
+    #   * `/usr/bin/start.sh` (PID 1) calls `migrate_gravity`, whose `migrate_to_database()` creates
+    #     the gravity database when `[ ! -e "$gravityDBfile" ]` — the same branch the postStart hook
+    #     below relies on, and the branch that produced the current ephemeral copy.
+    #   * FTL writes `pihole.toml` itself from the `FTLCONF_*` env above (that is what the
+    #     `### CHANGED (env)` markers in it are).
+    #   * NO `podSecurityContext.fsGroup` is needed and adding one would be inert: `start.sh` line 5
+    #     is `chown -R pihole:pihole /etc/pihole/ /var/log/pihole/` and PID 1 runs as **uid 0**, so
+    #     the image fixes ownership of a fresh root-owned volume before anything else starts. Same
+    #     arrangement as plex's image managing its own /config ownership.
+    #   * the image's own default for `files.macvendor` was moved OUT of this directory (to
+    #     `/macvendor.db`), which is what a maintainer does to a path they expect to be a volume.
+    #
+    # WHY IT IS FIXED IN THIS BRANCH rather than left for its own: `availability.replicas: 2` renders
+    # per-replica volumeClaimTemplates, and with the wrong mountPath those would be two 2 Gi Longhorn
+    # volumes (times three Longhorn replicas each) holding nothing but `lost+found` while all state
+    # stayed ephemeral — paying the cost of persistence and getting none of it. It also REMOVES this
+    # change's main operational risk: with gravity.db persisted, a pod restart is cheap, instead of
+    # every restart re-downloading 72 k domains inside a wave that has a 5 m timeout.
+    mountPath: /etc/pihole
 dependencies: []              # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
 # ------------------------------------------------------------------------------------------------
 # LAN SPLIT-HORIZON DNS — the half of R-DNS-03 that platform/coredns/coredns-custom.yaml defers to.
@@ -98,8 +134,64 @@ env:
   # then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal
   # characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid"
   # while resolution for the zone does not work. Confirmed both ways against the live image.
+  # plex-vip -> ${PLEX_L4_VIP} (added 2026-10-04) is the LAN HALF of Plex's single advertised
+  # hostname. Plex publishes ONE URL to plex.tv for both local and remote clients
+  # (catalog/plex.yaml ADVERTISE_IP): on the LAN this answer points it at the tcp/32400
+  # LoadBalancer, while Cloudflare answers the same name publicly with the WAN address, NATed to
+  # the same VIP. WITHOUT THIS LINE the entry above answers it with ${GATEWAY_VIP} via the zone
+  # wildcard, where NOTHING listens on 32400 — measured 2026-10-04, plex-vip.${BASE_DOMAIN}
+  # resolved to the Gateway VIP and a connect to :32400 there returned http=000. That is a silent
+  # failure: the Plex web UI on plex.${BASE_DOMAIN} keeps working through the Gateway on 443 while
+  # every client that takes the ADVERTISED connection cannot reach the server at all.
   - name: FTLCONF_misc_dnsmasq_lines
-    value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}"
+    value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
+  # THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
+  # apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
+  # carries the full rationale and the measurement that found it.
+  #
+  # `webserver.api.password` takes CLEARTEXT and FTL hashes it itself into `webserver.api.pwhash`.
+  # Verified in the running image, not assumed: `pihole setpassword` is itself
+  # `setFTLConfigValue "webserver.api.password" "$PASSWORD"`, commented upstream as "pihole-FTL will
+  # automatically hash the password". So the BALLOON-SHA256 hash on the PVC is now a DERIVED artefact
+  # and this is its source of truth. Rotating the Secret rotates the password.
+  - name: FTLCONF_webserver_api_password
+    valueFrom:
+      secretKeyRef:
+        name: pihole-bootstrap
+        key: webui-password
+availability:
+  # TWO REPLICAS, ON DIFFERENT NODES. LAN DNS is a critical network service and it was ONE pod on
+  # ONE node. Measured 2026-10-04, that cost more than "until it reschedules":
+  #   * the node running the pod dies -> ~40s to mark it NotReady, then `tolerationSeconds: 300`
+  #     before the pod is even evicted, then reschedule, then Longhorn must force-detach the RWO
+  #     volume from a dead node and reattach it. Roughly 6-10 MINUTES with no LAN resolution.
+  #   * the node holding the `pihole-dns-*` L2 lease dies -> Cilium re-elects an announcer, tens of
+  #     seconds. (These were DIFFERENT nodes: lease on k3s-prod-2, pod on k3s-prod-1. Two of three
+  #     nodes were single points of failure for LAN DNS, with wildly different blast radii.)
+  # The Services already tolerate this: `externalTrafficPolicy: Cluster` on the DNS LoadBalancers
+  # (apps/wave-0/pihole-dns.yaml) means the L2 lease holder forwards to ANY ready backend, so a
+  # second replica is picked up with no change there.
+  #
+  # WHAT THIS DOES NOT FIX, and it is the bigger risk: LAN DNS still depends on the CLUSTER. etcd
+  # quorum loss, a Cilium regression, a Longhorn problem or a bad reconcile takes DNS down
+  # everywhere regardless of replica count. The only thing that removes that dependency is a
+  # resolver outside the cluster carrying the same `address=` lines. Considered and not chosen
+  # (operator decision, 2026-10-04); see decisions/2026-10-04-pihole-multi-replica.md.
+  #
+  # THE PREREQUISITE WAS DONE FIRST, and this field would have been actively harmful without it. A
+  # replica gets a FRESH volume, and until 2026-10-04 PiHole's admin password and adlists lived only
+  # on the PVC — so replica 1 would have come up SERVING DNS AND BLOCKING NOTHING, with a different
+  # password, and answers would have varied by which pod replied. Both are now declared (the
+  # password as a SOPS Secret -> FTLCONF_webserver_api_password; the adlists reconciled into
+  # gravity.db by the postStart hook below), which is what makes a fresh replica identical to its
+  # sibling. See decisions/2026-10-04-pihole-declare-state.md.
+  #
+  # WHY 2 AND NOT 3. Two tolerates the loss of any single node, which is the stated goal, and leaves
+  # a spare node for the evicted replica to land on under the HARD anti-affinity the renderer emits.
+  # Three would pin one replica per node with no spare, so a single node failure would leave a
+  # replica permanently Pending and — `apps-wave-0` being `wait: true` — stall the app tier for as
+  # long as the node was down. Two is the count that degrades gracefully here.
+  replicas: 2
 ingressRoutes:
   - host: pihole.${BASE_DOMAIN}
     port: 80                  # admin web UI, protected at the Gateway by forward-auth (docs/09)
@@ -114,6 +206,110 @@ auth:
   # rather than a renderer constant: the measurement that justifies the flag on two apps is
   # evidence about those two apps. Its oauth2-proxy log holds 0 unauthenticated requests across
   # the current pod's whole 23h lifetime (measured 2026-10-01), so there is nothing to act on.
+lifecycle:
+  # LEVEL-TRIGGERED ADLIST ASSERTION, run on EVERY container start. Same pattern, and the same
+  # reasoning, as catalog/qbittorrent.yaml's postStart: desired state lives in git and is
+  # RE-ASSERTED unconditionally rather than set once by hand.
+  #
+  # WHY THIS IS NEEDED AT ALL — the blocklists were NOT reproducible from git. Measured on live prod
+  # 2026-10-04: `gravity.db` held exactly ONE adlist (StevenBlack's hosts, enabled) and 72,234
+  # gravity domains, and NOTHING in this repo named it. Worse, a FRESH database does not seed it:
+  # the image's own schema template `/etc/.pihole/advanced/Templates/gravity.db.sql` creates the
+  # `adlist` table and inserts NO rows into it (grepped in the running image — there is no
+  # `INSERT INTO adlist` at all). So a rebuild from git, or a replica with a new PVC, would come up
+  # SERVING DNS AND BLOCKING NOTHING, silently and with every object reporting healthy.
+  #
+  # WHY A postStart HOOK AND NOT A FILE. PiHole v6 keeps adlists as DB rows, not config. The
+  # `/etc/pihole/adlists.list` file still exists in `gravity.sh` but is MIGRATION-ONLY: its import
+  # sits inside `migrate_to_database()`, guarded by `if [ ! -e "$gravityDBfile" ]`, so it fires only
+  # while CREATING a database and never again on an existing one. A ConfigMap mounted there would
+  # therefore be silently ignored on every pod this cluster currently runs. There is also no
+  # `pihole adlist` CLI subcommand (only allow/deny for domains), so SQL against the DB is the
+  # actual interface. FTL ships an embedded shell (`pihole-FTL sqlite3`, SQLite 3.53.1) so no extra
+  # tooling is needed in the image.
+  #
+  # IT RECONCILES, IT DOES NOT JUST ADD: adlists present in the DB but absent from DECLARED below are
+  # DISABLED (not deleted — deleting cascades to `gravity` rows and would force a full re-download of
+  # everything on a typo). So this list is the source of truth in both directions, and `pihole -g`
+  # runs ONLY when something actually changed, which keeps the steady-state start free of a 70k-domain
+  # rebuild.
+  #
+  # NO BRACED DOLLAR EXPANSION ANYWHERE IN THIS SCRIPT, DELIBERATELY — the same contract
+  # catalog/qbittorrent.yaml documents at length. apps-wave-0 declares postBuild.substituteFrom
+  # cluster-vars, so Flux runs envsubst over this whole manifest at reconcile time; a `${VAR}` form
+  # would behave differently on the cluster than under a plain `kubectl apply` of the golden. Bare
+  # `$VAR` and `$(cmd)` are untouched by envsubst and so behave identically however this is run.
+  # `render.py`'s assert_substitution_safe refuses to emit a manifest that breaks this.
+  postStart:
+    exec:
+      command:
+        - /bin/bash
+        - -c
+        - |
+          # Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and
+          # does NOT run a gravity rebuild.
+          set -u
+          GRAVITY=/etc/pihole/gravity.db
+
+          # THE DECLARED SET — one URL per line, the source of truth for what PiHole blocks.
+          # To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.
+          DECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'
+
+          # On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,
+          # and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.
+          i=0
+          while [ ! -s "$GRAVITY" ] && [ "$i" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done
+          if [ ! -s "$GRAVITY" ]; then
+            echo "pihole-adlists: gravity.db absent after 180s - leaving adlists untouched" >&2
+            exit 0
+          fi
+
+          sql() { pihole-FTL sqlite3 "$GRAVITY" "$1"; }
+          CHANGED=0
+
+          # 1) Every declared URL must be present AND enabled.
+          echo "$DECLARED" | while IFS= read -r url; do
+            [ -n "$url" ] || continue
+            printf '%s\n' "$url"
+          done > /tmp/declared.txt
+          while IFS= read -r url; do
+            [ -n "$url" ] || continue
+            # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.
+            have=$(sql "select count(*) from adlist where address='$url';")
+            if [ "$have" = "0" ]; then
+              echo "pihole-adlists: adding $url"
+              sql "insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');"
+              CHANGED=1
+            else
+              en=$(sql "select enabled from adlist where address='$url';")
+              if [ "$en" != "1" ]; then
+                echo "pihole-adlists: re-enabling $url"
+                sql "update adlist set enabled=1 where address='$url';"
+                CHANGED=1
+              fi
+            fi
+          done < /tmp/declared.txt
+
+          # 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).
+          sql "select address from adlist where enabled=1;" | while IFS= read -r addr; do
+            [ -n "$addr" ] || continue
+            if ! grep -qxF "$addr" /tmp/declared.txt; then
+              echo "pihole-adlists: disabling undeclared $addr"
+              sql "update adlist set enabled=0 where address='$addr';"
+              echo changed >> /tmp/adlist-changed
+            fi
+          done
+          [ -s /tmp/adlist-changed ] && CHANGED=1
+          rm -f /tmp/adlist-changed /tmp/declared.txt
+
+          # 3) Rebuild gravity ONLY if the declared set actually moved.
+          if [ "$CHANGED" = "1" ]; then
+            echo "pihole-adlists: declared set changed - running gravity update"
+            pihole -g || echo "pihole-adlists: gravity update failed; existing gravity.db retained" >&2
+          else
+            echo "pihole-adlists: already matches the declared set - nothing to do"
+          fi
+          exit 0
 update:
   channel: stable             # R-UPD-01
   range: ">=2024.0.0"

catalog/plex.yaml

@@ -72,14 +72,227 @@ env:
       secretKeyRef:
         name: plex-claim
         key: PLEX_CLAIM
-  # ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
+  # ADVERTISE_IP ("Custom server access URLs") — the connection Plex publishes to plex.tv.
-  # which URL to advertise to plex.tv for REMOTE clients. The only candidate is
+  #
-  # https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
+  # THE VALUE MUST BE RESOLVABLE BY EVERY CLIENT, NOT JUST BY CLIENTS THAT USE OUR DNS. That single
-  # Gateway VIP (conformance/tests/l2-behavioral/dns-internal-resolution), and both its public
+  # property is what this field keeps getting wrong, so it is the first thing to check before
-  # reachability and whether the TLS-terminating Cilium Gateway proxies Plex's 32400 API correctly for
+  # changing it. A Plex client resolves the advertised host ITSELF, with whatever resolver it
-  # remote access are UNVERIFIED. Advertising a wrong/internal-only URL breaks remote access for ALL
+  # happens to use — and on a real LAN that is frequently NOT PiHole: browsers ship DoH on by
-  # clients, so per "prefer safety over completeness" it is left unset. FOLLOW-UP: once remote
+  # default, phones have "private DNS", guest and IoT devices often take the router's DNS, and some
-  # reachability through the Gateway is confirmed, set ADVERTISE_IP: "https://plex.home.keithjackson.dev:443".
+  # clients hard-code a public resolver.
+  #
+  # THAT IS NOT THEORETICAL — IT TOOK PLEX DOWN FOR THE WHOLE LAN ON 2026-10-05. This field briefly
+  # advertised `plex-vip.${BASE_DOMAIN}`, a name that exists ONLY in PiHole's split-horizon answers
+  # (catalog/pihole.yaml) and in no public zone. Clients that did not use PiHole could not resolve
+  # it; the SOHO router answered the failed lookup with ITSELF, and every one of them landed on the
+  # router's own web UI and reported a certificate error for `CN = %D`. Measured 2026-10-05: the
+  # router serves that self-signed cert (`C = ZZ, O = %D342e7e6b, CN = %D`) on BOTH `192.168.2.1:443`
+  # and the WAN address, while PiHole itself was answering perfectly — both replicas returned
+  # `plex-vip -> ${PLEX_L4_VIP}` and `plex -> ${GATEWAY_VIP}`. The DNS was right; the DEPENDENCY on
+  # it was the bug. Nothing Plex advertises listens on 443, which is how we know the clients had left
+  # the Plex path entirely and were talking to the router.
+  #
+  # SO: `<dashed-ip>.<serverhash>.plex.direct`, WHICH IS RESOLVER-INDEPENDENT. This is Plex's own
+  # mechanism and the reason it exists: `plex.direct` is a PUBLIC zone that returns the private
+  # address embedded in the label, so EVERY resolver on earth — PiHole, the router, 1.1.1.1, a DoH
+  # browser — answers `192-168-2-24.<hash>.plex.direct` with `192.168.2.24`. It needs no
+  # split-horizon entry, no Cloudflare record, and no certificate work, because the wildcard cert
+  # Plex already serves on 32400 covers exactly this name. Re-verified live 2026-10-05 with FULL
+  # validation (no `curl -k`): resolves to 192.168.2.24 and `/identity` returns **200, TLS verified**.
+  #
+  # WHY NOT THE NICER HOSTNAME, since a readable name in our own zone is obviously more pleasant:
+  # it cannot be made to work for clients that bypass PiHole. A public record for it would resolve
+  # to the WAN address, and with no NAT hairpin on this router a LAN client sent there fails too —
+  # so the hostname is correct ONLY for clients using PiHole, and wrong for the rest. There is no
+  # configuration of that name that is right for both. `plex.direct` is right for both.
+  # (The PiHole `plex-vip` answer is KEPT — it is harmless, costs nothing, and is genuinely useful
+  # for reaching the VIP by name from a LAN shell. It is simply not what Plex advertises.)
+  #
+  # THE TWO EMBEDDED VALUES, and the honest cost of this choice. The dashed IP must track
+  # ${PLEX_L4_VIP} and `<serverhash>` is this server's certificate identity — read it from the live
+  # listener with
+  #     openssl s_client -connect <vip>:32400 </dev/null | openssl x509 -noout -subject
+  # Nothing in this tree can derive either, and nothing checks that they still agree: envsubst
+  # cannot dash an IP, so this is a literal. That fragility is real and was the reason the hostname
+  # was tried in the first place — it is accepted here because an unresolvable name is a WORSE
+  # failure (silent, LAN-wide, and it points users at the router) than a stale literal, which at
+  # least fails loudly and only after someone changes the VIP.
+  #
+  # The image's /etc/cont-init.d/40-plex-first-run runs `setPref "customConnections"
+  # "${ADVERTISE_IP}"` (line 127, verified in the running image), OUTSIDE the
+  # PLEX_CLAIM/PlexOnlineToken guard, with its `/.firstRunComplete` marker on the container OVERLAY
+  # fs rather than the /config PVC — so this is re-applied on EVERY pod start and git, not a web
+  # form, owns the value.
+  - name: ADVERTISE_IP
+    value: https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400
+  # THE TLS MATERIAL FOR THE NAME ABOVE, as PEM text in env rather than a mounted volume.
+  # `plex-vip-tls` is issued by cert-manager from ClusterIssuer `letsencrypt-prod` for exactly
+  # `plex-vip.${BASE_DOMAIN}` (apps/wave-0/plex-cert.yaml, same wave => same reconcile as this
+  # workload). The postStart hook converts it to the PKCS#12 bundle Plex requires.
+  #
+  # WHY ENV AND NOT A VOLUME, which would be the conventional choice. MEASURED 2026-10-04:
+  # `schemas/servicedefinition.schema.json` `storage` accepts only
+  # [class, export, mountPath, name, readOnly, size, subPath] with `additionalProperties: false`,
+  # `catalog/render.py` emits NO secret volumes, and no rendered app mounts a Secret as a volume.
+  # Mounting one therefore needs a schema + renderer change; `env.valueFrom.secretKeyRef` is
+  # already in the schema and already used by PLEX_CLAIM above. PEM is text, so it survives an env
+  # var intact (a raw PKCS#12 would not — it is binary and carries NUL bytes).
+  #
+  # COST, STATED PLAINLY: the private key is readable from this container's environment
+  # (/proc/1/environ, `kubectl exec env`) by anything that can already exec into the pod. That is
+  # the same audience that can read the converted bundle under /config, so the marginal exposure is
+  # small — but a secret VOLUME would still be the better shape, and adding secret-volume support
+  # to the schema is the clean follow-up that removes this note.
+  #
+  # `optional: true` IS LOAD-BEARING, NOT DEFENSIVE CLUTTER (schema 1.11.0, added for this). The
+  # Certificate is issued INTO THIS NAMESPACE, and the namespace is created by the rendered
+  # apps/wave-0/plex.yaml itself — so the cert CANNOT be issued in an earlier tier, and the
+  # Certificate and this Deployment land in the SAME reconcile. On a cold bootstrap the Secret is
+  # therefore briefly absent while cert-manager completes a DNS01 challenge. With a HARD ref, Plex
+  # would sit in CreateContainerConfigError until a certificate issued — and because `apps-wave-0`
+  # is `wait: true` with `apps-wave-1` dependsOn it, a failed or slow issuance would STALL THE WHOLE
+  # APP TIER (the same hazard the `hardware` block below documents for the GPU resource). Optional
+  # instead means: no cert -> Plex still starts and serves, the postStart hook logs that the
+  # certificate was not installed, and the next start after issuance converges. Plex falls back to
+  # its own `*.plex.direct` certificate in the meantime, so the degraded state is "advertised URL
+  # not yet servable over TLS", not "media server down".
+  - name: PLEX_TLS_CRT
+    valueFrom:
+      secretKeyRef:
+        name: plex-vip-tls
+        key: tls.crt
+        optional: true
+  - name: PLEX_TLS_KEY
+    valueFrom:
+      secretKeyRef:
+        name: plex-vip-tls
+        key: tls.key
+        optional: true
+lifecycle:
+  # LEVEL-TRIGGERED PLEX SETTINGS, re-asserted on EVERY container start — the same pattern and the
+  # same reasoning as catalog/pihole.yaml's and catalog/qbittorrent.yaml's postStart: desired state
+  # lives in git and is re-asserted unconditionally rather than typed into a web form once. Typing
+  # settings into the web form is precisely what took this service down on 2026-10-04.
+  #
+  # IT EXISTS BECAUSE THE IMAGE HAS NO ENV FOR THESE SETTINGS. Measured 2026-10-04 across the whole
+  # init surface (/etc/cont-init.d, /etc/services.d, /plex-common.sh, /usr/local/bin): the image
+  # calls setPref for exactly SIX prefs — MachineIdentifier, ProcessedMachineIdentifier,
+  # PlexOnlineToken, customConnections, allowedNetworks, TranscoderTempDirectory — and supports NO
+  # generic PLEX_PREFERENCE* hook of any kind. The two things we need (a LAN definition and a custom
+  # certificate) are reachable ONLY through Plex's own API, so this hook is the only GitOps path.
+  #
+  # IT MUST NEVER FAIL. A postStart hook that exits non-zero makes kubelet KILL the container, so
+  # every step is bounded and the script ends with an unconditional `exit 0`. A missing token, an
+  # unissued certificate or an unreachable API degrades to "not applied this start" and is retried
+  # on the next one; it never crashloops Plex.
+  postStart:
+    exec:
+      command:
+        - /bin/bash
+        - -c
+        - |
+          # NOTE — LOAD-BEARING: SHELL VARIABLES HERE ARE WRITTEN BARE, NEVER BRACE-WRAPPED.
+          # This whole script is rewritten by Flux postBuild envsubst BEFORE it ever runs, so a
+          # brace-wrapped name is read as a cluster-vars knob: an undeclared one is a HARD
+          # Kustomization failure under StrictPostBuildSubstitutions, and a declared one would be
+          # replaced with its value. Shell vars are therefore `$PX`, `$TOKEN`, `$P12` — bare. Only
+          # BASE_DOMAIN below is brace-wrapped, because it IS a knob and IS meant to be
+          # substituted. Command substitution `$(cmd)` is left alone and needs no escaping. A shell
+          # variable written with braces would be blanked or would fail the whole Kustomization —
+          # this repo has already lost a gitea OIDC bootstrap to exactly that mistake, and
+          # catalog/render.py now refuses to emit it (which is how this very comment got caught).
+          # DELIBERATELY NO `set -u`. The TLS env vars are declared `optional: true`, so when the
+          # certificate has not issued yet they are genuinely UNSET, not empty — and under `set -u`
+          # the first `[ -n "$PLEX_TLS_CRT" ]` would abort the script, which for a postStart hook
+          # means kubelet KILLS THE CONTAINER. That is the precise failure the `optional` flag exists
+          # to prevent, so turning it back on here would defeat the whole design. (The alternative,
+          # a brace-wrapped `:-` default, would need envsubst escaping — the trap the note above
+          # describes.) Every value below is explicitly guarded instead.
+          PX="/config/Library/Application Support/Plex Media Server/Preferences.xml"
+          CERTDIR=/config/certs
+          P12=$CERTDIR/plex-vip.p12
+          # The PKCS#12 passphrase protects nothing and is deliberately not a secret: Plex needs it
+          # in cleartext in customCertificateKey, and it and the bundle sit on the same PVC. It is a
+          # container-format requirement, not an access control.
+          P12PASS=plexvip
+          LANNETS='10.0.0.0/8,192.168.0.0/16'
+          CERTDOMAIN=plex-vip.${BASE_DOMAIN}
+          API=http://127.0.0.1:32400
+
+          # (1) Build the PKCS#12 bundle from the PEM the Secret supplies. Rebuilt on every start so
+          # a cert-manager RENEWAL is picked up with no extra moving part.
+          if [ -n "$PLEX_TLS_CRT" ] && [ -n "$PLEX_TLS_KEY" ]; then
+            mkdir -p $CERTDIR && chmod 700 $CERTDIR
+            printf '%s' "$PLEX_TLS_CRT" > $CERTDIR/tls.crt
+            printf '%s' "$PLEX_TLS_KEY" > $CERTDIR/tls.key
+            chmod 600 $CERTDIR/tls.key
+            if openssl pkcs12 -export -out $P12.tmp -inkey $CERTDIR/tls.key \
+                 -in $CERTDIR/tls.crt -passout pass:$P12PASS 2>/dev/null; then
+              mv -f $P12.tmp $P12
+              chown 1000:1000 $P12 2>/dev/null || true
+              chmod 600 $P12
+              echo "[plex-settings] PKCS#12 bundle written"
+            else
+              rm -f $P12.tmp
+              echo "[plex-settings] WARN: pkcs12 export failed; keeping any previous bundle" >&2
+            fi
+            rm -f $CERTDIR/tls.key
+          else
+            echo "[plex-settings] WARN: TLS env empty; certificate not installed this start" >&2
+          fi
+
+          # (2) Wait, bounded (~90s), for the local API. /identity needs no auth.
+          i=0
+          while [ "$i" -lt 45 ]; do
+            if [ "$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 $API/identity 2>/dev/null)" = "200" ]; then
+              break
+            fi
+            i=$(expr $i + 1); sleep 2
+          done
+
+          # (3) The prefs API needs the server token. It already lives in Preferences.xml on this
+          # PVC; it is never echoed here, and Plex redacts tokens in its own request log. An
+          # UNCLAIMED server has none — a normal state on a fresh /config — so skip, do not fail.
+          TOKEN=$(grep -o 'PlexOnlineToken="[^"]*"' "$PX" 2>/dev/null | head -1 | cut -d'"' -f2)
+          if [ -z "$TOKEN" ]; then
+            echo "[plex-settings] not claimed yet — skipping prefs, will retry next start" >&2
+            exit 0
+          fi
+
+          # Percent-encode what our values actually contain. Plex takes prefs as QUERY parameters on
+          # a PUT, so a raw '/' or ',' would be read as structure rather than data.
+          enc() {
+            printf '%s' "$1" | sed -e 's|%|%25|g' -e 's|/|%2F|g' -e 's|,|%2C|g' \
+                                   -e 's|:|%3A|g' -e 's| |%20|g'
+          }
+          setpref() {
+            code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 -X PUT \
+              -H "X-Plex-Token: $TOKEN" "$API/:/prefs?$1=$(enc "$2")" 2>/dev/null)
+            if [ "$code" = "200" ]; then
+              echo "[plex-settings] set $1"
+            else
+              echo "[plex-settings] WARN: $1 -> HTTP $code" >&2
+            fi
+          }
+
+          # (4) THE LAN DEFINITION — the reason this hook was written. Without it Plex has NO notion
+          # of a local network and tags EVERY client remote. Measured 2026-10-04: 468 requests
+          # logged "Allowed Network (WAN)" and ZERO "(LAN)", including Seerr from inside the
+          # cluster, which is why the server showed as Remote to LAN clients. Note allowedNetworks
+          # (the ALLOWED_NETWORKS env) only waives AUTH — it does NOT make a client local. Two
+          # different settings; only one of them has an env hook.
+          setpref LanNetworksBandwidth "$LANNETS"
+
+          # (5) THE CERTIFICATE, which is what makes the `https` in ADVERTISE_IP servable. Set only
+          # if the bundle exists — pointing Plex at a missing file would break its TLS listener.
+          if [ -s "$P12" ]; then
+            setpref customCertificatePath "$P12"
+            setpref customCertificateKey "$P12PASS"
+            setpref customCertificateDomain "$CERTDOMAIN"
+          else
+            echo "[plex-settings] WARN: no bundle; leaving certificate prefs untouched" >&2
+          fi
+          exit 0
 # HARDWARE TRANSCODING — the host iGPU's render device (schema 1.8.0 `hardware`; R-PLAT-02).
 # INTENT ONLY, AND THE MECHANISM DELIBERATELY DOES NOT LIVE HERE. This says "this workload needs an
 # Intel Quick Sync render device"; it does NOT name a device path and it does NOT name a resource.

catalog/render.py

@@ -156,11 +156,17 @@ OAUTH2_COOKIE_REFRESH = "2m30s"
 # ready endpoints. The flag is confirmed present in the PINNED v7.8.1 binary (`--help`),
 # which matters because the shared proxy is on v7.15.3 and could not have vouched for it.
 #
-# THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED. A container port is not a
+# THE SCRAPE WORKS AS OF 2026-10-05 — but it took THREE independent pieces, and this flag is
-# Service port: these sidecars' Services expose the app port and :4180, and each namespace's
+# only the first. A container port is not a Service port, and each namespace's default-deny
-# default-deny baseline has `ingress: []`, so Prometheus can neither resolve nor reach this
+# baseline denies all ingress, so the flag alone bought nothing for 63.7h:
-# port until a Service port (or PodMonitor), a monitoring-ingress allow and a ServiceMonitor
+#   1. this flag + a named `metrics` container port   (PR #28)
-# are added. Do not read this flag as "metrics are working".
+#   2. a monitor that targets POD ports, not Service ports — platform/oauth2-proxy/podmonitor.yaml
+#      is a PodMonitor for exactly that reason (no Service exposes 44180)
+#   3. an ingress allowance from ns monitoring        (PR #49, r_monitoring_ingress_policy below)
+# Verified on prod 2026-10-05: all 6 targets up == 1, scraping in 1-6ms.
+# STILL do not read this flag alone as "metrics are working" — 1 without 3 is precisely the state
+# that produced 5 x TargetDown for 63.7h. If you add a sidecar metrics port anywhere else, check
+# that r_monitoring_ingress_policy covers it.
 OAUTH2_PROXY_METRICS_PORT = 44180
 
 # --- Hardware capability: `hardware.gpu: intel-quicksync` ------------------------------
@@ -272,7 +278,15 @@ def apply_defaults(sd: dict) -> dict:
     # object that actually runs on the cluster is not the one this SD renders (keycloak: the
     # Keycloak operator runs a StatefulSet, so a Deployment-shaped rule would `absent()` forever).
     lw = dict(obs.get("livenessWorkload") or {})
-    lw.setdefault("kind", "VirtualMachineInstance" if s["kind"] == "vm" else "Deployment")
+    # The default must track what is ACTUALLY rendered, or the `up` SLI silently dies: a
+    # Deployment-shaped rule against a StatefulSet `absent()`s forever, which is an alert that can
+    # never fire — the exact failure mode the keycloak override was added for. Deriving it from
+    # `availability.replicas` means bumping the replica count cannot leave a stale rule behind.
+    lw.setdefault(
+        "kind",
+        "VirtualMachineInstance" if s["kind"] == "vm"
+        else ("StatefulSet" if _is_statefulset(s) else "Deployment"),
+    )
     lw.setdefault("name", s["name"])
     obs["livenessWorkload"] = lw
     s["observability"] = obs
@@ -475,6 +489,37 @@ def r_bulk_pvs(s: dict) -> list:
     return out
 
 
+def _replicas(s: dict) -> int:
+    """Declared replica count (schema 1.10.0 `availability.replicas`). Default 1."""
+    return int((s.get("availability") or {}).get("replicas", 1) or 1)
+
+
+def _is_statefulset(s: dict) -> bool:
+    """True when this container workload must render a StatefulSet rather than a Deployment.
+
+    THE DECIDING CONSTRAINT IS THE VOLUME, not a preference. A longhorn storage entry renders
+    ReadWriteOnce (r_pvcs), and one RWO PVC cannot be attached by pods on two different nodes --
+    which is exactly what `availability.replicas > 1` plus the node anti-affinity below asks for.
+    A 2-replica Deployment over a shared RWO claim does not degrade, it simply never schedules the
+    second pod. A StatefulSet's volumeClaimTemplates give each replica its OWN claim, which is the
+    only shape that satisfies both at once.
+
+    A replicas-1 service keeps rendering a Deployment so every pre-existing golden is byte-identical.
+    """
+    return s["kind"] == "container" and _replicas(s) > 1
+
+
+def _sts_claim_volumes(s: dict) -> list:
+    """Storage entries that become per-replica volumeClaimTemplates on a StatefulSet.
+
+    Only the dynamically-provisioned (longhorn) entries. A `bulk` entry is a STATIC, pre-bound NFS
+    PV over data that already exists outside the cluster and is shared read-only by every consumer
+    (r_bulk_pvs) -- per-replica copies of it are neither possible nor wanted, so it stays a
+    standalone PVC and an ordinary pod volume.
+    """
+    return [v for v in s["storage"] if v.get("class", "longhorn") != "bulk"]
+
+
 def r_pvcs(s: dict) -> list:
     """One PVC per storage entry (R-STATE-01/02/04).
 
@@ -482,8 +527,15 @@ def r_pvcs(s: dict) -> list:
     bulk     -> the STATIC external NFS PV rendered by r_bulk_pvs, pre-bound by volumeName.
     """
     out = []
+    # On a StatefulSet the longhorn claims are NOT standalone objects — they are
+    # volumeClaimTemplates, so each replica gets its own (see _is_statefulset). Emitting both would
+    # leave an orphan PVC that nothing mounts and that `prune` would later delete out from under
+    # nobody. `bulk` is unaffected and still rendered here.
+    sts = _is_statefulset(s)
     for vol in s["storage"]:
         cls = vol.get("class", "longhorn")
+        if sts and cls != "bulk":
+            continue
         if cls == "bulk":
             # Mirror image of the PV: the same explicitly-empty class (so nothing dynamically
             # provisions), the same access mode, and volumeName completing the two-sided binding.
@@ -564,6 +616,12 @@ def r_deployment(s: dict) -> dict:
         if vol.get("readOnly"):
             mount["readOnly"] = True
         volume_mounts.append(mount)
+        # On a StatefulSet a longhorn volume is supplied by its volumeClaimTemplate, whose
+        # metadata.name IS the pod volume name. Adding it to pod.volumes as well is a conflict
+        # (duplicate volume name), so it is skipped here and emitted below. The MOUNT is unchanged —
+        # the container cannot tell the difference, which is what keeps this transparent to the app.
+        if _is_statefulset(s) and vol.get("class", "longhorn") != "bulk":
+            continue
         volumes.append(
             {
                 "name": vol["name"],
@@ -661,20 +719,80 @@ def r_deployment(s: dict) -> dict:
     if ann:
         tmpl_meta["annotations"] = ann
 
-    spec = {"replicas": 1}
+    sts = _is_statefulset(s)
-    # An app with persistent storage mounts a ReadWriteOnce Longhorn PVC. The default
+
-    # RollingUpdate strategy tries to start the NEW pod before terminating the old one, but
+    # NODE ANTI-AFFINITY, and it is REQUIRED rather than preferred. `availability.replicas > 1`
-    # the old pod still holds the RWO volume -> the new pod is stuck ContainerCreating and
+    # exists to survive the loss of one node; `preferred` would let both replicas land on the same
-    # the rollout deadlocks (single-replica app can never hand the volume over). Recreate
+    # node and quietly deliver none of that, with every object reporting healthy. The cost is real
-    # terminates the old pod first, freeing the PVC for the new one (R-STATE-01/02).
+    # and is accepted: with N replicas the workload needs N schedulable nodes, so on a 3-node
-    if s["storage"]:
+    # cluster losing TWO nodes leaves a replica Pending and — because `apps-wave-0` is `wait: true`
+    # with `apps-wave-1` dependsOn it — STALLS THE APP TIER. That coincides with having already lost
+    # etcd quorum on a 3-node control plane, i.e. the cluster is not reconciling anything at that
+    # point anyway, so the stall is a symptom rather than a new failure.
+    if sts:
+        pod_spec["affinity"] = {
+            "podAntiAffinity": {
+                "requiredDuringSchedulingIgnoredDuringExecution": [
+                    {
+                        "labelSelector": {
+                            "matchLabels": {"app.kubernetes.io/name": s["name"]}
+                        },
+                        "topologyKey": "kubernetes.io/hostname",
+                    }
+                ]
+            }
+        }
+
+    spec = {"replicas": _replicas(s)}
+    if sts:
+        # serviceName is a required StatefulSet field. It points at the service rendered by
+        # r_service; that Service is a normal ClusterIP rather than headless, so per-pod DNS
+        # (<pod>.<svc>) does NOT resolve — and nothing here needs it. These replicas are
+        # interchangeable resolvers behind one VIP, not peers that address each other.
+        spec["serviceName"] = s["name"]
+        # Parallel, not the OrderedReady default. OrderedReady starts replica N+1 only after N is
+        # Ready, so a cold start costs the sum of every replica's first-boot work instead of the
+        # max — and `apps-wave-0` has `timeout: 5m`. There is no ordering requirement between
+        # replicas to protect (no leader, no migration, no shared volume), so serialising buys
+        # nothing and risks the wave timing out on a rebuild.
+        spec["podManagementPolicy"] = "Parallel"
+    elif s["storage"]:
+        # An app with persistent storage mounts a ReadWriteOnce Longhorn PVC. The default
+        # RollingUpdate strategy tries to start the NEW pod before terminating the old one, but
+        # the old pod still holds the RWO volume -> the new pod is stuck ContainerCreating and
+        # the rollout deadlocks (single-replica app can never hand the volume over). Recreate
+        # terminates the old pod first, freeing the PVC for the new one (R-STATE-01/02).
+        #
+        # A StatefulSet needs none of this and cannot express it: each replica owns its own claim,
+        # so a rolling update hands nothing over, and `strategy` is not a StatefulSet field at all.
         spec["strategy"] = {"type": "Recreate"}
     spec["selector"] = {"matchLabels": {"app.kubernetes.io/name": s["name"]}}
     spec["template"] = {"metadata": tmpl_meta, "spec": pod_spec}
+    if sts:
+        # Per-replica claims. metadata.name MUST equal the pod volume name the container mounts
+        # (see the volumes loop above); the resulting PVCs are <name>-<sts>-<ordinal>, e.g.
+        # config-pihole-0. Same tier and access mode a standalone longhorn PVC gets from r_pvcs —
+        # this changes WHO owns the claim, never what it is.
+        tmpls = []
+        for vol in _sts_claim_volumes(s):
+            vspec = {"accessModes": ["ReadWriteOnce"], "storageClassName": "longhorn"}
+            if vol.get("size"):
+                vspec["resources"] = {"requests": {"storage": vol["size"]}}
+            tmpls.append(
+                {
+                    "metadata": {
+                        "name": vol["name"],
+                        "labels": {"app.kubernetes.io/name": s["name"]},
+                    },
+                    "spec": vspec,
+                }
+            )
+        if tmpls:
+            spec["volumeClaimTemplates"] = tmpls
 
     return {
         "apiVersion": "apps/v1",
-        "kind": "Deployment",
+        "kind": "StatefulSet" if sts else "Deployment",
         "metadata": {
             "name": s["name"],
             "namespace": s["name"],
@@ -684,6 +802,35 @@ def r_deployment(s: dict) -> dict:
     }
 
 
+def r_pdb(s: dict):
+    """PodDisruptionBudget for a multi-replica workload (availability.replicas > 1).
+
+    minAvailable 1, not `maxUnavailable: 1`. The budget's job here is to stop a VOLUNTARY
+    disruption — a node drain for maintenance, a cluster upgrade — from taking the last replica
+    with it; `minAvailable: 1` says that whatever else happens, one must keep serving. It does
+    NOT protect against involuntary disruption (a node dying), which is what the anti-affinity
+    and the replica count are for.
+
+    Rendered only when replicas > 1: a PDB over a single-replica workload is actively harmful, as
+    `minAvailable: 1` would then block every drain of its node forever.
+    """
+    if not _is_statefulset(s):
+        return None
+    return {
+        "apiVersion": "policy/v1",
+        "kind": "PodDisruptionBudget",
+        "metadata": {
+            "name": s["name"],
+            "namespace": s["name"],
+            "labels": {"app.kubernetes.io/name": s["name"]},
+        },
+        "spec": {
+            "minAvailable": 1,
+            "selector": {"matchLabels": {"app.kubernetes.io/name": s["name"]}},
+        },
+    }
+
+
 def r_service(s: dict):
     ports = _service_ports(s)
     # A declared metrics endpoint becomes a NAMED Service port, because that name is the only
@@ -1221,8 +1368,9 @@ def _oauth2_sidecar_container(s: dict) -> dict:
         f"--cookie-refresh={OAUTH2_COOKIE_REFRESH}",
         "--pass-access-token=true",
         "--set-xauthrequest=true",
-        # Start the /metrics listener. Prerequisite only — not yet reachable by Prometheus;
+        # Start the /metrics listener. Scraped as of 2026-10-05, but only because
-        # see OAUTH2_PROXY_METRICS_PORT above for exactly what is still missing.
+        # r_monitoring_ingress_policy also opens this port to ns monitoring — the flag alone was
+        # inert for 63.7h. See OAUTH2_PROXY_METRICS_PORT above for the three pieces required.
         f"--metrics-address=0.0.0.0:{OAUTH2_PROXY_METRICS_PORT}",
     ]
     secret = f"{s['name']}-oauth2-proxy"
@@ -1400,12 +1548,54 @@ def r_monitoring_ingress_policy(s: dict):
     trap Longhorn's own NetworkPolicies set (platform/longhorn/networkpolicy-metrics.yaml exists
     for exactly this reason), and it is a THIRD independent way for a metrics SLI to be inert.
 
-    Rendered only alongside a declared metricsEndpoint, and it opens ONLY that port to ONLY the
+    Rendered only alongside a port something actually serves, and it opens ONLY that port to ONLY
-    monitoring namespace: an allow for a port nothing serves would be a standing hole for no
+    the monitoring namespace: an allow for a port nothing serves would be a standing hole for no
     benefit.
+
+    TWO INDEPENDENT SOURCES OF A SERVED METRICS PORT, and this used to cover only the first:
+
+      1. `observability.metricsEndpoint` — the APP's own exporter. NOTE, measured 2026-10-04: no
+         ServiceDefinition in this catalog declares one, so before the sidecar clause below this
+         function had NEVER emitted a single object on any branch. It is correct but, so far,
+         entirely latent — do not read its presence as evidence that scrape ingress is solved.
+
+      2. THE forward-auth GATE SIDECAR's own `/metrics` on 44180 (added 2026-10-04). Every
+         `_needs_oauth2_sidecar` app gets `--metrics-address=0.0.0.0:{OAUTH2_PROXY_METRICS_PORT}`
+         and a named `metrics` container port from `_oauth2_sidecar_container`, so the port is
+         served whenever the sidecar exists — unconditionally, which is what makes it safe to
+         open without a per-app declaration.
+
+    WHY CLAUSE 2 KEYS ON `_needs_oauth2_sidecar` AND NOT ON `observability.metrics`: it must match
+    the condition the SCRAPE keys on, exactly. platform/oauth2-proxy/podmonitor.yaml selects pods
+    by the `homelab.keithjackson.dev/auth-mode: forward-auth` POD LABEL and nothing else, so a
+    target exists iff the sidecar exists. Keying the allowance on `observability.metrics` instead
+    would let `metrics: false` produce a discovered-but-blocked target — the same "two halves
+    authored apart" failure this function exists to prevent, just moved one field over.
+
+    THIS CLAUSE IS A BUG FIX, and the bug was live for 63.7h. Measured on prod 2026-10-04: all 6
+    sidecar targets across jackett/radarr/sonarr/qbittorrent/pihole read `health=down`,
+    `lastError: context deadline exceeded`, `lastScrapeDuration` exactly 10.0s — the scrapeTimeout,
+    i.e. packets DROPPED, not refused. Each app namespace ships a `default-deny` NetworkPolicy
+    with `policyTypes: [Ingress, Egress]` and NO `ingress` key at all (deny-all-ingress), and the
+    only ingress allowance is `<app>-allow-gateway-ingress`: `fromEntities: [ingress]` -> 4180.
+    Prometheus matched no clause. 5 x `TargetDown` + 6 x `OAuth2ProxyGateUnobserved` had been
+    re-notifying at ~22 emails/day since the PodMonitor landed on 2026-10-02.
+
+    WHY THE POLICY BELONGS HERE AND NOT IN platform/oauth2-proxy: podmonitor.yaml's header records
+    the objection — a policy in that tier would name app namespaces created by the SIBLING
+    apps-wave-0/1 tiers (measured Flux graph: `oauth2-proxy` and `apps-wave-0` both
+    `dependsOn=[platform-config]`, no ordering between them), so it would apply on a running
+    cluster and fail on a cold bootstrap. Rendered here it ships in the same tier that creates the
+    namespace, which dissolves that ordering problem entirely.
     """
-    ep = s["observability"].get("metricsEndpoint")
+    obs = s["observability"]
-    if not ep or not s["observability"]["metrics"]:
+    ep = obs.get("metricsEndpoint")
+    ports: list[str] = []
+    if ep and obs["metrics"]:
+        ports.append(str(ep["port"]))
+    if _needs_oauth2_sidecar(s):
+        ports.append(str(OAUTH2_PROXY_METRICS_PORT))
+    if not ports:
         return None
     return {
         "apiVersion": "cilium.io/v2",
@@ -1423,7 +1613,7 @@ def r_monitoring_ingress_policy(s: dict):
                         {"matchLabels": {"k8s:io.kubernetes.pod.namespace": MONITORING_NS}}
                     ],
                     "toPorts": [
-                        {"ports": [{"port": str(ep["port"]), "protocol": "TCP"}]}
+                        {"ports": [{"port": p, "protocol": "TCP"} for p in ports]}
                     ],
                 }
             ],
@@ -1817,7 +2007,13 @@ def render(sd: dict, image_automation: bool = True, inbound: list | None = None)
             )
         for pvc in r_pvcs(s):
             docs.append((f"PVC {pvc['metadata']['name']} (R-STATE)", pvc))
-        docs.append(("Deployment (R-PLAT-02)", r_deployment(s)))
+        wl = r_deployment(s)
+        docs.append((f"{wl['kind']} (R-PLAT-02)", wl))
+        pdb = r_pdb(s)
+        if pdb:
+            docs.append(
+                ("PodDisruptionBudget — keep one replica through a drain (availability.replicas)", pdb)
+            )
         svc = r_service(s)
         if svc:
             docs.append(("Service", svc))
@@ -2002,6 +2198,24 @@ EXTRA_UNITS: dict = {
             "pihole-dns.yaml",
         ],
     },
+    # Plex's remote-access exposure: the raw tcp/32400 LoadBalancer Service on a pinned VIP, plus the
+    # CiliumNetworkPolicy that admits off-cluster clients to it (apps/wave-0/plex-remote.yaml).
+    # `above: False` co-locates it in plex's OWN wave — these are the plex pod's own Service and its
+    # own policy, applied in the same wave as the Deployment they select, exactly as the rendered
+    # Service and CNPs in plex.yaml are. Nothing in a higher wave gates on it.
+    #
+    # It is a non-catalog unit for the same reason pihole-dns is: the ServiceDefinition schema cannot
+    # express a LoadBalancer Service or a pinned lbipam VIP (additionalProperties is false, and
+    # `ingressRoutes` only renders HTTPRoutes on the L7 Gateway, which cannot carry Plex's raw port).
+    # Plex's own settings stay in the catalog as env on catalog/plex.yaml; the only thing living
+    # outside it is the exposure the schema has no word for.
+    "plex-remote": {
+        "anchor": "plex",
+        "above": False,
+        "files": [
+            "plex-remote.yaml",
+        ],
+    },
 }
 
 
@@ -2248,6 +2462,132 @@ def _selftest() -> int:
     check("Namespace" in kinds, "container: missing Namespace")
     check("NetworkPolicy" in kinds, "container: missing default-deny NetworkPolicy")
     check("Deployment" in kinds, "container: missing Deployment")
+    check(
+        "PodDisruptionBudget" not in kinds,
+        "replicas-1 container must render NO PodDisruptionBudget (minAvailable 1 over a single "
+        "replica blocks every drain of its node forever)",
+    )
+
+    # 1b) availability.replicas (schema 1.10.0). The DEFAULT path must be untouched, and the >1
+    # path must change workload kind, claim ownership, spreading, budget AND the SLI metric
+    # together — each of these has a silent-failure mode if it is the one that gets forgotten.
+    base_av = {
+        "name": "av",
+        "source": {"image": "nginx:1"},
+        "storage": [{"name": "config", "size": "1Gi"}],
+        "observability": {"metrics": True, "slis": ["up"]},
+    }
+    one = render({**base_av, "availability": {"replicas": 1}})
+    one_kinds = [d[1]["kind"] for d in one]
+    check(
+        "Deployment" in one_kinds and "StatefulSet" not in one_kinds,
+        "availability.replicas=1 must still render a Deployment (back-compat)",
+    )
+    check(
+        "PersistentVolumeClaim" in one_kinds,
+        "availability.replicas=1 must still render a standalone PVC",
+    )
+    check(
+        "PodDisruptionBudget" not in one_kinds,
+        "availability.replicas=1 must render no PodDisruptionBudget",
+    )
+
+    many = render({**base_av, "availability": {"replicas": 2}})
+    many_kinds = [d[1]["kind"] for d in many]
+    check(
+        "StatefulSet" in many_kinds and "Deployment" not in many_kinds,
+        "availability.replicas>1 must render a StatefulSet, not a Deployment (a shared RWO claim "
+        "cannot be attached on two nodes, so a multi-replica Deployment never schedules)",
+    )
+    check(
+        "PersistentVolumeClaim" not in many_kinds,
+        "availability.replicas>1 must NOT also render a standalone longhorn PVC — it would be an "
+        "orphan that nothing mounts",
+    )
+    sts = next(d[1] for d in many if d[1]["kind"] == "StatefulSet")
+    check(sts["spec"].get("replicas") == 2, "StatefulSet must carry the declared replica count")
+    check(
+        "strategy" not in sts["spec"],
+        "StatefulSet must not carry `strategy` (not a StatefulSet field; Recreate is a "
+        "Deployment-only workaround for handing over one RWO volume)",
+    )
+    check(
+        sts["spec"].get("podManagementPolicy") == "Parallel",
+        "StatefulSet must set podManagementPolicy=Parallel — OrderedReady serialises first-boot "
+        "work across replicas and apps-wave-0 has a 5m timeout",
+    )
+    check(sts["spec"].get("serviceName") == "av", "StatefulSet must set serviceName")
+    tmpls = sts["spec"].get("volumeClaimTemplates") or []
+    check(
+        [t["metadata"]["name"] for t in tmpls] == ["config"],
+        "StatefulSet volumeClaimTemplate name must equal the pod volume name the container mounts, "
+        "or the pod gets a duplicate/unresolvable volume",
+    )
+    check(
+        tmpls and tmpls[0]["spec"]["accessModes"] == ["ReadWriteOnce"]
+        and tmpls[0]["spec"]["storageClassName"] == "longhorn",
+        "StatefulSet claim template must keep the same tier/access mode as the standalone PVC",
+    )
+    sts_vols = {v["name"] for v in sts["spec"]["template"]["spec"].get("volumes", [])}
+    check(
+        "config" not in sts_vols,
+        "a claim-template volume must NOT also appear in pod.volumes (duplicate volume name)",
+    )
+    mounts = sts["spec"]["template"]["spec"]["containers"][0].get("volumeMounts") or []
+    check(
+        any(m["name"] == "config" for m in mounts),
+        "the container must still mount the volume by name on a StatefulSet (transparent to the app)",
+    )
+    aff = (sts["spec"]["template"]["spec"].get("affinity") or {}).get("podAntiAffinity") or {}
+    req = aff.get("requiredDuringSchedulingIgnoredDuringExecution") or []
+    check(
+        req and req[0].get("topologyKey") == "kubernetes.io/hostname",
+        "availability.replicas>1 must render REQUIRED per-node anti-affinity — `preferred` can put "
+        "every replica on one node and deliver none of the intended node-failure tolerance",
+    )
+    pdbs = [d[1] for d in many if d[1]["kind"] == "PodDisruptionBudget"]
+    check(len(pdbs) == 1, "availability.replicas>1 must render exactly one PodDisruptionBudget")
+    check(
+        pdbs and pdbs[0]["spec"].get("minAvailable") == 1,
+        "PodDisruptionBudget must be minAvailable=1 (keep one replica serving through a drain)",
+    )
+    pr_many = next((d[1] for d in many if d[1]["kind"] == "PrometheusRule"), None)
+    check(
+        pr_many is not None
+        and "kube_statefulset_status_replicas_ready"
+        in pr_many["spec"]["groups"][0]["rules"][0]["expr"],
+        "the up SLI must follow the rendered workload kind — a Deployment-shaped rule against a "
+        "StatefulSet absent()s forever and can never fire",
+    )
+
+    # A bulk volume is shared external NFS: it must stay a standalone PVC even on a StatefulSet.
+    mixed = render(
+        {
+            **base_av,
+            "availability": {"replicas": 2},
+            "storage": [
+                {"name": "config", "size": "1Gi"},
+                {"name": "media", "class": "bulk", "export": "/mnt/x", "size": "1Ti"},
+            ],
+        }
+    )
+    m_sts = next(d[1] for d in mixed if d[1]["kind"] == "StatefulSet")
+    check(
+        [t["metadata"]["name"] for t in m_sts["spec"]["volumeClaimTemplates"]] == ["config"],
+        "only longhorn volumes become claim templates; a bulk share must not be per-replica",
+    )
+    check(
+        any(
+            d[1]["kind"] == "PersistentVolumeClaim"
+            and d[1]["metadata"]["name"] == "av-media"
+            for d in mixed
+        ),
+        "a bulk PVC must still be rendered standalone on a StatefulSet",
+    )
+    check(
+        "media" in {v["name"] for v in m_sts["spec"]["template"]["spec"].get("volumes", [])},
+        "a bulk volume must still appear in pod.volumes on a StatefulSet",
+    )
 
     ns = next(d[1] for d in docs if d[1]["kind"] == "Namespace")
     check(

clusters/base/cluster-vars.yaml

@@ -100,7 +100,8 @@ data:
   # platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
   # tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
   # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
-  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers),
+  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers;
+  # plex-remote + pihole: PLEX_L4_VIP),
   # velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
   # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
   # UNSUBSTITUTED path would ship the literal string into the cluster.
@@ -175,6 +176,26 @@ data:
   # dev  -> 192.168.1.56  (base default; next free after crafty .55 in dev's .53-.60 pool)
   # prod -> 192.168.2.23  (next free after crafty .22 in prod's .20-.30 pool; see clusters/prod)
   PIHOLE_DNS_VIP: "192.168.1.56"             # PiHole: authoritative LAN DNS for ${BASE_DOMAIN}
+  # Plex's remote-access listener: raw tcp/32400 for REMOTE Plex clients (apps/wave-0/plex-remote.yaml).
+  # A FOURTH pinned L4 VIP, for the usual reason — the `shared` Gateway is an L7 HTTPS terminator and
+  # cannot carry Plex's own TLS. Plex clients do not resolve `plex.${BASE_DOMAIN}`; they fetch this
+  # server's address from plex.tv, so the Gateway's L7 path (which still serves the web UI, and was
+  # measured working) cannot put the server on a client's connection list at all.
+  #
+  # PINNED, AND THE PIN IS LOAD-BEARING OFF-REPO: the SOHO router forwards WAN tcp/32400 to this
+  # address by hand. A VIP taken from the pool head would silently move on a reschedule and break
+  # remote access with nothing in-cluster looking wrong. See runbooks/plex-remote-access.md.
+  #
+  # dev  -> 192.168.1.57  (base default; next free after pihole-dns .56 in dev's .53-.60 pool)
+  # prod -> 192.168.2.24  (next free after pihole-dns .23 in prod's .20-.30 pool; see clusters/prod)
+  PLEX_L4_VIP: "192.168.1.57"                # Plex: tcp/32400 remote access (NOT the web UI, which is L7)
+  # SECOND CONSUMER since 2026-10-04: catalog/pihole.yaml reads this knob to answer
+  # plex-vip.${BASE_DOMAIN} on the LAN, which is the local half of the single hostname Plex
+  # advertises to plex.tv. Without that per-host answer the zone wildcard sends the name to
+  # ${GATEWAY_VIP}, where nothing listens on 32400. So this value is now typed into THREE places
+  # that must agree: this knob, the WAN port-forward on the SOHO router (off-repo), and PiHole's
+  # answer -- and only the first two were true before.
+
 
   # The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
   # peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable

clusters/prod/kustomization.yaml

@@ -476,6 +476,26 @@ patches:
       - op: replace
         path: /data/PIHOLE_DNS_VIP
         value: "192.168.2.23"
+      # Plex's remote-access listener VIP — raw tcp/32400 for REMOTE Plex clients
+      # (apps/wave-0/plex-remote.yaml). 192.168.2.24 is the NEXT FREE address in the prod pool
+      # 192.168.2.20-.30, verified live 2026-10-03: the CiliumLoadBalancerIPPool `default-pool`
+      # reported IPsTotal 11 / IPsUsed 4 / IPsAvailable 7, and the only four allocations are .20
+      # (cilium-gateway-shared), .21 (unifi-l4-tcp/-udp), .22 (crafty-l4-tcp/-udp) and .23
+      # (pihole-dns-tcp/-udp). It is inside the pool block the operator already asserted sits outside
+      # the SOHO router's DHCP scope, and it is not a node address (.10/.11/.12) or the
+      # control-plane VIP (.9).
+      #
+      # UNLIKE .23, THIS ONE IS ALSO A WAN PORT-FORWARD TARGET. The router forwards WAN tcp/32400
+      # here, by hand, off-repo — so changing this value is a two-place edit and the router is the
+      # half that CI cannot check. runbooks/plex-remote-access.md is the procedure.
+      #
+      # NOT ARP-PROBED. .23 was verified free with `arping` from a LAN host before it was claimed;
+      # for .24 only the in-cluster pool accounting above was checked, so an off-cluster squatter on
+      # this address would not have been caught. Probe before trusting it if remote access comes up
+      # with a conflict.
+      - op: replace
+        path: /data/PLEX_L4_VIP
+        value: "192.168.2.24"
       # The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns —
       # NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path.
       # It scopes the ingress allow for PiHole's DNS listener to LAN clients only.

conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml

@@ -171,6 +171,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): paas-demo-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: paas-demo-allow-monitoring-scrape
+  namespace: paas-demo
+  labels:
+    app.kubernetes.io/name: paas-demo
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: paas-demo
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): paas-demo-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml

@@ -198,6 +198,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: sonarr-allow-monitoring-scrape
+  namespace: sonarr
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: sonarr
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml

@@ -198,6 +198,27 @@ spec:
       - port: '4180'
         protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: sonarr-allow-monitoring-scrape
+  namespace: sonarr
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: sonarr
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy

decisions/2026-10-03-cilium-cgroup-hostroot.md

@@ -0,0 +1,76 @@
+---
+date: 2026-10-03
+branch: fix/cilium-cgroup-hostroot
+title: Cilium socket-LB never covered host processes — cgroup.hostRoot pointed at an unmounted path
+---
+
+## What changed
+
+`cgroup.hostRoot` changes from `/run/cilium/cgroupv2` to `/sys/fs/cgroup` in all three parity
+copies: `platform/cilium/helmrelease.yaml` (authoritative), `platform/cilium/values.yaml`
+(mirror) and `infra/ansible/roles/cilium/templates/cilium-values.yaml.j2` (bootstrap).
+`autoMount` stays `false`.
+
+**The defect.** With `autoMount: false`, the Cilium chart only bind-mounts whatever exists at
+`hostRoot`, and expects the operator to have mounted cgroup2 there (cilium 1.20.1 values.yaml).
+The repo's rationale was "k3s provides the cgroupv2 mount". That is true only for
+`/sys/fs/cgroup`. Nothing mounts cgroup2 at `/run/cilium/cgroupv2` on these nodes; it is an
+empty `DirectoryOrCreate` hostPath. So the agent mounted cgroup2 there itself, from inside its
+own private cgroup namespace, and that mount's root is the agent container's own leaf cgroup.
+The socket-LB programs (`cil_sock4_connect` etc.) were attached to it and covered only the
+agent.
+
+**The consequence.** Every other host-netns process got no Service/LB translation:
+- Service ClusterIPs were unreachable from every node's host.
+- The node holding the Gateway's L2 lease could not reach the Gateway VIP (`no route to host`).
+  Other nodes reached it only because their packets crossed the wire to the lease holder, where
+  the tc datapath handles them.
+- k3s-prod-1 holds both the Gateway lease and the kube-vip API VIP. Its apiserver could not reach
+  the OIDC issuer, logging `oidc: authenticator not initialized` since at least 2026-09-12
+  (899 occurrences by 2026-10-03). kubectl SSO through the VIP therefore failed, and in-cluster
+  OIDC clients failed on whichever calls landed on that apiserver.
+- Webhooks and aggregated APIs were unaffected only because k3s runs
+  `--enable-aggregator-routing=true` (endpoint IPs, not ClusterIPs).
+
+Cilium's own source (v1.20.1 `bpf/bpf_sock.c`, `__sock4_xlate_fwd`) handles exactly this case
+when the hook does run: a host-netns connect to an L7-LB Service (the Gateway) is rewritten to
+`127.0.0.1:<l7_lb_proxy_port>`, the local Envoy listener. The fix is to make the hook cover the
+host.
+
+## What I verified
+
+- `cilium-dbg status --verbose` on k3s-prod-1: `Socket LB: Enabled`, `Coverage: Hostns-only`.
+  The live cilium-config shows `bpf-lb-sock=false`, but socket LB is implied by kube-proxy
+  replacement. The Gateway Service frontends `192.168.2.20:{80,443}` and
+  `10.43.225.79:{80,443}` map to `127.0.0.1:14337`.
+- Host: `findmnt /run/cilium/cgroupv2` is empty on all three nodes, and `/sys/fs/cgroup` is
+  `cgroup2`.
+- In the agent: `/run/cilium/cgroupv2` is a `cgroup2 none` mount, the agent's own cgroup reads
+  `0::/`, the mount has 0 child cgroups, and `bpftool cgroup show /run/cilium/cgroupv2` lists all
+  `cil_sock*` programs. k3s runs in `0::/system.slice/k3s.service`, and the agent container in
+  `/kubepods.slice/…/cri-containerd-3ad04cde….scope`.
+- **The decisive test**, run on k3s-prod-1 in the same netns: raw TCP connect to
+  `192.168.2.20:443`, `10.43.225.79:443` and `10.43.59.141:8080` CONNECTED from inside the agent
+  container (covered cgroup) and FAILED from the host (`No route to host` / timeout).
+- Blast radius: host-network pods are cilium, cilium-envoy, cilium-operator (API via
+  `192.168.2.9:6443`, not a Service), kube-vip (`127.0.0.1:6443`, `svc_enable=false`) and
+  node-exporter (no outbound). Nothing depends on host Service translation being absent; the
+  change only makes Service/LB addresses reachable from the host where they currently fail.
+- `helm template cilium/cilium --version 1.20.1` with the new values, in the toolbox: the
+  DaemonSet bind-mounts hostPath `/sys/fs/cgroup` into `cilium-agent` and `clean-cilium-state`,
+  the cilium-config shows `cgroup-root: "/sys/fs/cgroup"`, and the init containers are unchanged
+  (no `mount-cgroup`).
+- Toolbox gate: see the PR description.
+
+## What I did NOT verify
+
+- **The rollout itself.** After merge, the agent DaemonSet rolls one node at a time
+  (`maxUnavailable: 1`). On each node, afterwards: host TCP connects to the three targets above
+  must succeed. On k3s-prod-1, the apiserver's background OIDC init retry must stop logging
+  `authenticator not initialized` with no k3s restart.
+- That the old programs, attached to the previous agent container's cgroup, disappear with it.
+  Expected, because cgroup BPF attachments die with the cgroup, but not observed.
+- An end-to-end kubectl OIDC login (needs a browser).
+- No regression test was added. A natural one is an L2 conformance check that a host-netns
+  connect to the Gateway VIP succeeds on the lease holder, but `conformance/**` is a sensitive
+  path. Left as a follow-up.

decisions/2026-10-03-node-keycloak-hosts-pin.md

@@ -0,0 +1,45 @@
+---
+date: 2026-10-03
+branch: fix/node-keycloak-hosts-pin
+title: Pin the OIDC issuer to the Gateway VIP on prod nodes — the WAN answer does not hairpin
+---
+
+## What changed
+
+Follow-up to PR #40 (`decisions/2026-10-03-public-dns-wan-target.md`). Public DNS for
+`*.home.keithjackson.dev` now CNAMEs to the WAN IP. The prod nodes resolve through public DNS,
+and the router does not hairpin, so the kube-apiserver could no longer reach its OIDC issuer
+`keycloak.home.keithjackson.dev`.
+
+- `roles/node_baseline`: a new `node_baseline_hosts_pins` list (default `[]`) and one
+  `blockinfile` task (tags `dns`, `hosts`). It writes the pins into `/etc/hosts` as a single
+  marked block. An empty list removes the block.
+- `group_vars/k3s_prod/vars.yml`: pins `keycloak.{{ k3s_base_domain }}` → `192.168.2.20`, which
+  must stay equal to `GATEWAY_VIP` (parity comment added). It also replaces the stale "no
+  /etc/hosts pin is needed" paragraph with the measured reason the pin is now required.
+
+Dev is unaffected (empty default).
+
+## What I verified
+
+- After PR #40 reconciled (`platform` lastAppliedRevision `576e1ab`, Healthy=True), on
+  k3s-prod-2 and k3s-prod-3: `getent hosts keycloak.home.keithjackson.dev` →
+  `136.51.115.252 wan.home.keithjackson.dev`. `curl https://keycloak…/realms/homelab/.well-known/openid-configuration`
+  → exit 60, `ssl_verify_result=18` (the router's self-signed cert, not the Gateway).
+- `journalctl -u k3s` (last 20 min) showed no OIDC/JWKS errors yet. The apiserver still holds
+  the keys it cached before the flip, so the failure is latent until a k3s restart or key rotation.
+- Neither node has cloud-init (`/etc/cloud` absent), so nothing rewrites `/etc/hosts` at boot.
+- `--check --diff --tags hosts --limit k3s-prod-2,k3s-prod-3`: each node gets exactly the
+  three-line marked block `192.168.2.20	keycloak.home.keithjackson.dev`.
+- Outside-in, PR #40 works: check-host.net probes from 3 external nodes got `200 OK` from
+  `https://plex.home.keithjackson.dev/identity` at `136.51.115.252`.
+
+## What I did NOT verify
+
+- **k3s-prod-1**: Ansible cannot reach it. SSH host key verification fails (`known_hosts:94`,
+  ED25519 `SHA256:zgUS8OeWznzCX9hlN7XdZ8iLNkU1G2loXjtJFfEUYpE`), which is consistent with
+  the 2026-09-11 SSD replacement and reinstall. The operator must confirm the fingerprint
+  before the pin can be applied there.
+- The playbook has not been applied for real. It is an operator run after merge:
+  `ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`.
+- An end-to-end kubectl OIDC login after the pin (needs a browser).

decisions/2026-10-03-plex-remote-access-l4.md

@@ -0,0 +1,129 @@
+---
+date: 2026-10-03
+branch: feat/plex-remote-access-l4
+title: Plex remote access — raw tcp/32400 via a pinned LoadBalancer VIP, not the L7 Gateway
+---
+
+## What changed
+
+Reported symptom: users cannot reach Plex from outside the LAN. **Two independent causes**, measured
+rather than inferred:
+
+1. **Plex advertised no remote connection at all.** Live `Preferences.xml` on prod:
+   `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while
+   `PublishServerOnPlexOnlineKey="1"` — publishing was on and there was nothing reachable to publish.
+   Plex's automatic UPnP/NAT-PMP path cannot work from a pod on the Cilium overlay.
+2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` refused;
+   `136.51.115.252:443` answered with a **self-signed router cert** (`C=ZZ, CN=%D`), not the
+   Gateway's LE wildcard. The Plex Service was `ClusterIP` only — no off-cluster path to 32400 by
+   any route.
+
+This branch fixes the in-cluster half. **Remote access does not work until the operator also does
+two manual steps** (`runbooks/plex-remote-access.md`): the WAN tcp/32400 port-forward, and ticking
+"manually specify public port" in the Plex UI.
+
+- **`apps/wave-0/plex-remote.yaml` (new, hand-written).** A `type=LoadBalancer` Service on tcp/32400
+  pinned to `${PLEX_L4_VIP}` via `lbipam.cilium.io/ips`, plus a `CiliumNetworkPolicy`
+  (`plex-allow-remote-32400`) admitting `world`/`cluster`/`host`/`remote-node` to 32400. Non-catalog
+  because `schemas/servicedefinition.schema.json` is `additionalProperties: false` and has no field
+  for a LoadBalancer Service or a pinned VIP — the same reason `apps/wave-0/pihole-dns.yaml`,
+  `platform/crafty/service-l4.yaml` and `apps/vms/unifi/50-service-l4.yaml` are hand-written.
+- **`catalog/render.py`:** an `EXTRA_UNITS` entry (`plex-remote`, anchored to `plex`, `above: False`)
+  so `--emit-waves` places the unit in plex's own wave.
+- **`clusters/base/cluster-vars.yaml`:** new `PLEX_L4_VIP` (dev default `192.168.1.57`), plus the
+  substitution inventory comment updated to name `plex-remote: PLEX_L4_VIP`.
+- **`clusters/prod/kustomization.yaml`:** prod override `PLEX_L4_VIP: 192.168.2.24`.
+- **`apps/wave-0/kustomization.yaml`:** `plex-remote.yaml` added to `resources`.
+- **`runbooks/plex-remote-access.md` (new):** the two manual steps, the four-step verification, why
+  no alert is possible, and the two documented warts below.
+
+**`ADVERTISE_IP` was deliberately NOT set.** Setting it to `https://plex.home.keithjackson.dev:443`
+was the alternative — it needs no new WAN hole and the mechanism was verified working on an
+already-claimed server. **The operator chose the native path (2026-10-03)** so Plex terminates its own
+TLS (`*.plex.direct`) and streams bypass the Envoy L7 hop. `catalog/plex.yaml`'s deferral note for
+`ADVERTISE_IP` remains accurate and untouched.
+
+**`externalTrafficPolicy: Cluster`, with a known cost.** `Local` would preserve the client IP, but
+Cilium's L2 announcer does not follow the backend pod — the wart that cost this LAN an 8-day
+Minecraft outage (see `platform/crafty/service-l4.yaml`). Consequence accepted and documented: SNAT
+puts the client in `192.168.0.0/16`, inside `ALLOWED_NETWORKS`, so **Plex classifies remote streams
+as LAN**. That is a metering effect (remote bandwidth limits will not bind), not an authorization one.
+
+**SENSITIVE PATHS — this PR touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK
+it.** It needs the operator's `/approve-override <reason>` posted as a PR **review** of type
+"Comment". I have deliberately not posted it: CLAUDE.md §1 forbids an agent clearing this on its own
+PR, and the agent account holds `owner`, so the prohibition is a rule, not a limit.
+
+## What I verified
+
+All measurements against live prod, 2026-10-03, read-only.
+
+- **Plex's advertised state**, the finding this branch rests on: `customConnections=<ABSENT>`,
+  `ManualPortMappingMode=<ABSENT>`, `ManualPortMappingPort=<ABSENT>`,
+  `PublishServerOnPlexOnlineKey="1"`, `allowedNetworks="10.0.0.0/8,192.168.0.0/16"`. Read by
+  attribute grep, never by dumping `Preferences.xml` (it holds `PlexOnlineToken`).
+- **The L7 Gateway path works and is NOT the problem.** Via the Gateway VIP,
+  `GET https://plex.home.keithjackson.dev/identity` → `HTTP 200`, `claimed="1"`, real
+  `machineIdentifier`. HTTPRoute `plex-32400`: `Accepted=True`, `ResolvedRefs=True`.
+- **Plex enforces its own auth**, which is what makes an unfiltered WAN port defensible:
+  `/library/sections`, `/status/sessions`, `/:/prefs` all → **401** unauthenticated through the
+  Gateway.
+- **The SNAT/allowedNetworks wart is real, not theoretical.** A tagged probe through the Gateway was
+  logged by Plex as `[10.0.2.214:36040 (Allowed Network (WAN))]` — the Envoy pod IP, classified as an
+  allowed network.
+- **WAN state:** `136.51.115.252:32400` → connection refused. `136.51.115.252:443` → self-signed
+  cert `C=ZZ, ST=Somewhere, O=%D342e7e6b, CN=%D`. Public DNS resolves correctly:
+  `plex.home.keithjackson.dev` → CNAME `wan.home.keithjackson.dev` → `136.51.115.252`.
+- **`192.168.2.24` is free in-cluster.** `CiliumLoadBalancerIPPool default-pool`: IPsTotal 11 /
+  IPsUsed 4 / IPsAvailable 7; the only allocations are `.20` (cilium-gateway-shared), `.21`
+  (unifi-l4-*), `.22` (crafty-l4-*), `.23` (pihole-dns-*). Not a node address (`.10/.11/.12`) and not
+  the kube-vip control-plane VIP (`.9`).
+- **The manual-port-mapping step genuinely cannot be automated.** Grep of the running image's whole
+  init surface (`/etc/cont-init.d`, `/plex-common.sh`, `/usr/local/bin`) for `ManualPortMapping` and
+  `PLEX_PREFERENCE` → **NONE**.
+- **`ADVERTISE_IP` would have worked on a claimed server** (recorded because the runbook's
+  "why not" rests on it): `setPref "customConnections"` at line 126 of `40-plex-first-run` sits
+  **outside** the `PLEX_CLAIM`/`PlexOnlineToken` guard, and the `/.firstRunComplete` early-exit
+  marker is on the **overlay** FS, not the `/config` PVC, so the script re-runs every pod start.
+- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
+  `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
+  **pass**. The rc 3 is two environmental skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`)
+  and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3, not as pass.**
+- **`render-golden` passing is the specific proof** that the `render.py` `EXTRA_UNITS` change, the new
+  hand-written unit and the edited wave kustomization are mutually consistent.
+
+## What I did NOT verify
+
+- **THE ACTUAL FIX. Nothing in this branch has been proven to make remote access work** — it cannot
+  be, because both remaining steps are off-repo and manual (router forward, Plex UI toggle). This
+  branch removes two blockers and adds none; it does not demonstrate an outcome.
+- **Outside-in reachability, at all.** Every probe above was taken from inside the LAN, so the
+  `:443` self-signed cert strictly proves only that there is **no NAT hairpin** — it does **not**
+  prove outside-in failure. A router cert on WAN:443 is suggestive (its management UI would shadow a
+  forward), but unproven. **This repo has no off-LAN vantage point.** Test 3 in the runbook
+  (app.plex.tv from a cellular connection) is the only honest check.
+- **`192.168.2.24` was never ARP-probed off-cluster.** Only in-cluster pool accounting was checked.
+  `.23` was `arping`-probed before it was claimed; `.24` was not, so an off-cluster squatter would
+  not have been caught.
+- **The LoadBalancer has never been reconciled.** `plex-l4-tcp` does not exist in prod yet, so
+  LB-IPAM has not actually assigned the pinned VIP, L2 announcement has not been observed for it, and
+  the new CiliumNetworkPolicy has never admitted a real packet. Whether `world`/`host`/`cluster`/
+  `remote-node` is the correct entity set **under this SNAT path** is reasoned from
+  `platform/crafty/networkpolicy.yaml`'s working game-port rule, **not measured**.
+- **Whether Plex's remote path needs anything beyond the manual port tick.** If plex.tv still lists
+  no remote connection after both manual steps, this branch is necessary-but-insufficient.
+- **The CI approval-policy outcome.** The local stage reported green, but that is an offline N/A, not
+  a verdict: this PR touches `clusters/**`, so CI will block it pending an override.
+- **PRE-EXISTING DRIFT I FOUND AND DELIBERATELY LEFT ALONE — worth its own follow-up.** Running the
+  documented `python3 catalog/render.py --emit-waves` regenerates the wave kustomizations and wants to
+  add **`vm-demo.yaml`** to wave 0 and **`keycloak.yaml`** to wave 1. Both are listed as
+  intentionally not onboarded in `ci/render-golden.sh`
+  (`RENDER_GOLDEN_NOT_ONBOARDED="keycloak vm-demo"`); their manifests are tracked in `apps/` but
+  deliberately absent from the `resources` lists so Flux does not apply them. `--emit-waves` places
+  any file present on disk and has **no awareness of that exclusion**, while `render-golden.sh` uses
+  `--wave-layout` and compares rendered manifests only — it never checks kustomization `resources`,
+  which is why this has survived unnoticed. **So the header "DO NOT hand-edit: re-run
+  `render.py --emit-waves`" is currently unsafe advice**: following it onboards an IdP and a KubeVirt
+  VM. I therefore restored both kustomizations and hand-added only my one line. I did not fix
+  `--emit-waves`, and I have not determined how long the drift has existed or whether either service
+  is *meant* to be onboarded.

decisions/2026-10-03-public-dns-wan-target.md

@@ -0,0 +1,73 @@
+---
+date: 2026-10-03
+branch: feat/public-dns-wan-target
+title: Public DNS now points at the WAN — Gateway-published names CNAME to an in-cluster DDNS record
+---
+
+## What changed
+
+The operator closed the open "public-target gap" recorded in
+`platform/externaldns/helmrelease.yaml`. The WAN NAT / port-forwarding (80/443 → Gateway VIP
+192.168.2.20) was done by the operator outside this repo.
+
+Operator decisions (2026-10-03):
+
+1. **Scope: every Gateway-published name goes public.** That covers the five `exposure: public`
+   apps (plex, gitea, keycloak, nextcloud, seerr) and also the hand-authored admin UIs (auth,
+   crafty, grafana, headlamp, the `home.keithjackson.dev` apex). `internal` services still publish
+   nothing: their `controller: none` route annotation is checked before the target is read.
+2. **The WAN IP is kept current in-cluster**, not hardcoded.
+
+Implementation:
+
+- `platform/gateway/gateway.yaml`: `external-dns.alpha.kubernetes.io/target: wan.${BASE_DOMAIN}` on
+  the shared Gateway. Every gateway-sourced record changes from `A 192.168.2.20` to
+  `CNAME wan.home.keithjackson.dev`.
+- `platform/externaldns/ddns.yaml` (new): `favonia/cloudflare-ddns:1.17.1` Deployment in the
+  `external-dns` namespace. It reuses `cloudflare-api-token` and keeps `wan.${BASE_DOMAIN}` an
+  unproxied IPv4 A record at the WAN IP (detected via Cloudflare trace, checked every 5 min,
+  `DELETE_ON_STOP=false`). external-dns has no TXT registry entry for it, so it never owns or
+  deletes it.
+- The helmrelease comment block and the README now describe the decision instead of the gap.
+
+Known consequence: **k3s nodes resolve through public DNS** (`node_baseline_dns_servers` = 8.8.8.8 /
+1.1.1.1). The kube-apiserver's OIDC issuer `keycloak.home.keithjackson.dev` therefore resolves to the
+WAN IP from the nodes, and kubectl SSO works only if the router hairpins WAN:443 back to the Gateway.
+A LAN-side probe of `136.51.115.252:443` on 2026-10-03 returned a **self-signed placeholder cert**,
+not the Gateway's Let's Encrypt wildcard. That suggests no hairpin. The cert-based admin kubeconfig
+is unaffected.
+
+## What I verified
+
+- Live prod, 2026-10-03, before the change: Gateway `gateway/shared` ADDRESS `192.168.2.20`.
+  External-dns args include `--source=gateway-httproute --source=service --policy=sync
+  --txt-owner-id=prod --domain-filter=keithjackson.dev`, and it logs "All records are already up to
+  date".
+- Public answers at 1.1.1.1: plex/gitea/keycloak/grafana/auth/apex → `192.168.2.20`;
+  `crafty-mc` → `.22` and `unifi` → `.21` (service source, not affected by this change); jackett →
+  none (internal, correctly unpublished).
+- The authority (betty.ns.cloudflare.com) has no unprefixed TXT, AAAA or MX at `plex.` or the apex,
+  so the A→CNAME switch has no co-existing record to collide with.
+- WAN IP from the workstation: `136.51.115.252`.
+- No CiliumNetworkPolicy or NetworkPolicy in `external-dns`. The Kyverno label and default-deny
+  policies match only `app-namespace` namespaces. The vpn egress policy selects only
+  `egress-class: vpn` pods.
+- Toolbox gate result: see the PR description (run after this commit).
+
+## What I did NOT verify
+
+- **The outside-in path.** I have no vantage point outside the LAN. Whether
+  `https://<svc>.home.keithjackson.dev` from the internet reaches the Gateway with the LE cert is
+  UNVERIFIED.
+- **Hairpin / kubectl SSO after the flip.** Likely broken per the probe above. Fix candidates: a
+  node `/etc/hosts` pin for `keycloak.${BASE_DOMAIN}` → Gateway VIP via `node_baseline`, or router
+  NAT reflection.
+- **The record-type flip itself**, until Flux reconciles: whether Cloudflare accepts external-dns's
+  create-CNAME in the same batch as the delete-A, or converges one interval later.
+- `favonia/cloudflare-ddns` under this pod securityContext. The README warns that
+  `no-new-privileges` misbehaves on some Proxmox-style hosts. These are bare-metal nodes, but it is
+  untested until the pod runs.
+- Whether the L4 service-source names (crafty-mc, unifi) should also go public. Not decided, not
+  changed.
+- `conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh` header lines 39–40
+  still describe the target as open. That is a sensitive path, so it is left for a follow-up.

decisions/2026-10-04-hubble-flow-export-plex.md

@@ -0,0 +1,157 @@
+---
+date: 2026-10-04
+branch: feat/hubble-flow-export-plex
+title: Record the pre-SNAT client address for Plex via Hubble static flow export, instead of breaking externalTrafficPolicy to get it
+---
+
+## What changed
+
+Operator requirement, narrowed explicitly: **this is 100% about log visibility** — knowing which
+address connected to Plex. Not bandwidth metering, which does not matter here.
+
+**The problem, measured.** Plex's own log cannot show it. Every LoadBalancer Service in this cluster
+is `externalTrafficPolicy: Cluster`, so the node holding the L2 lease for the VIP SNATs the client
+before forwarding to the backend pod:
+
+| Measured 2026-10-04 | Value |
+|---|---|
+| Node announcing `192.168.2.24` | `k3s-prod-3` (also holds `.20` Gateway, `.23` pihole-dns) |
+| Node running the Plex pod | `k3s-prod-1` (`10.0.0.135`) |
+| Sources in Plex's log | **3,214 × `10.0.1.x`**, 105 × `10.0.2.x`, 120 × loopback |
+| `192.168.x` sources in Plex's log, ever | **0** |
+| `X-Forwarded-For` in Plex's log | **0** (it serves raw tcp/32400, not HTTP via the Gateway) |
+| `10.0.1.78` identity (`cilium-dbg bpf ipcache`) | `identity=1` → `reserved:host` = prod-3 itself |
+
+**Nothing on the router is involved** — the operator correctly pointed out there is no hairpin NAT
+here. The rewrite is entirely in-cluster, done by Cilium's BPF load-balancer on the lease holder,
+because the backend pod is on a different node.
+
+**The address is not lost, only unrecorded.** Cilium observes the flow at ingress, *before* its own
+translation, so Hubble already holds the true source:
+
+    192.168.2.133 -> 10.0.0.135    (real LAN client -> the Plex pod)
+    10.0.1.78     -> 10.0.0.135    (the same traffic post-SNAT — what Plex sees)
+
+What was missing is **retention**: the ring buffer was **4095/4095 flows at ~303 flows/s — about
+thirteen seconds of history** — with hubble metrics disabled and no exporter configured.
+
+So: enable the static flow exporter, scoped to Plex's port, writing to the agent's stdout.
+
+- **`platform/cilium/helmrelease.yaml`** (authoritative): `hubble.export.static.enabled: true`,
+  `filePath: stdout`, `allowList` of `{"destination_port":["32400"]}` and
+  `{"source_port":["32400"]}`.
+- **`platform/cilium/values.yaml`** (reference mirror) and
+  **`infra/ansible/roles/cilium/templates/cilium-values.yaml.j2`** (the `[parity]` source of truth):
+  the same block, so a bootstrap render does not differ from what Flux reconciles — a mismatch would
+  be a permanent Helm diff and an avoidable agent rollout.
+
+**`filePath: stdout`, not a file,** because Grafana Alloy already runs as a DaemonSet tailing every
+pod's logs into Loki (`platform/monitoring/alloy.yaml`). stdout therefore reaches Loki with **no new
+hostPath mount, no file-rotation handling, and no Alloy change at all**. Upstream documents the
+value: *"To get Hubble flows directly exported to the logs instead of written to a rotated file,
+`stdout` can be defined as `hubble-export-file-path`."* Query:
+
+    {namespace="kube-system", app="cilium"} |= "32400"
+
+**SCOPED BY PORT, NOT BY NAMESPACE — a measured decision, not a preference.** A
+`--to-namespace plex` query returned **only the post-SNAT flows**; the flow carrying the real client
+address did not match it, because at the point of observation the destination identity is not yet
+resolved to the plex pod. A namespace filter would have dropped exactly the record this change
+exists to capture.
+
+**WHY NOT FIX THE SNAT INSTEAD — both alternatives rejected on evidence.**
+- `externalTrafficPolicy: Local` would put the real address in Plex's own log, but cilium 1.20's L2
+  announcer is **explicitly incompatible** with it. Upstream docs for this exact version:
+  *"incompatible with the `externalTrafficPolicy: Local` on services as it may cause service IPs to
+  be announced on nodes without pods causing traffic drops"* and *"L2 announcements isn't currently
+  aware of this setting"*. Live proof it is not theoretical: the VIP is announced by prod-3 while the
+  pod runs on prod-1, so the flip would black-hole tcp/32400 **immediately**. Making it safe means
+  pinning both the pod and the announcement to one node and surrendering Plex's ability to
+  reschedule — a real availability loss to buy a log field. Same trap
+  `platform/crafty/service-l4.yaml` documents from an 8-day Minecraft outage.
+- **DSR** also preserves the client address, but `routing-mode` here is **vxlan**, not
+  native/geneve, and DSR returns bypass the ingress node, which breaks the SOHO router's stateful
+  NAT on the remote path. Cluster-wide blast radius across all eight LB Services for one service's
+  logging.
+
+**A correction to the repo's own record.** `decisions/2026-10-03-plex-remote-access-l4.md` states the
+SNAT puts the client inside `192.168.0.0/16` (the ingress node's LAN address), and I repeated that in
+the previous entry. **It is wrong**: measured, the SNAT source is `10.0.1.x` — the node's
+`cilium_host` inside the pod CIDR. This matters, because it means `LanNetworksBandwidth` can never
+discriminate local from remote while eTP is `Cluster`: `10.0.0.0/8` must be in the LAN set for the
+in-cluster consumers (Seerr) to count as local, and every external client arrives in that same range.
+PR #48's LAN definition therefore fixes the Remote badge by making *everything* local — acceptable
+only because metering is explicitly not a goal here.
+
+## What I verified
+
+Live prod, 2026-10-04, read-only.
+
+- **Every measurement in the table above**, including the VIP-lease-to-node map
+  (`cilium-dbg shell -- db/show l2-announce` on all three agents) and the pod's node.
+- **Hubble already carries the pre-SNAT source** — the single finding this whole approach rests on:
+  `hubble observe --port 32400` on the lease holder returned `192.168.2.133 -> 10.0.0.135` **and**
+  its return flow, alongside the post-SNAT `10.0.1.78 -> 10.0.0.135`.
+- **The retention gap is real:** `cilium-dbg status` reported `Hubble: Ok  Current/Max Flows:
+  4095/4095 (100.00%), Flows/s: 303.62  Metrics: Disabled`, and the `cilium-config` ConfigMap
+  contains no `hubble-export-*` key at all.
+- **The namespace-filter trap**, directly: `--to-namespace plex` returned only `10.0.1.78` and
+  `10.0.1.71`; the `192.168.2.133` flow was absent. This is why the allowList is port-scoped.
+- **All eight LoadBalancer Services are `eTP=Cluster`** (crafty ×2, gateway, pihole-dns ×2, plex,
+  unifi ×2) — so this is a cluster-wide pattern, not a Plex-specific choice.
+- **The eTP:Local incompatibility is upstream-documented for v1.20 specifically**, quoted above from
+  the docs for the running version — not inferred from the crafty incident.
+- **`routing-mode: tunnel`** and **`bpf-lb-mode-annotation: false`** in the live `cilium-config`,
+  which is what rules DSR out.
+- **The chart's value paths and defaults**, read from cilium v1.20.1's own `values.yaml`:
+  `hubble.export.static.{enabled,filePath,fieldMask,fieldAggregate,aggregationInterval,allowList,denyList,fileMaxSizeMb,fileMaxBackups,fileCompress}`,
+  with `filePath` defaulting to `/var/run/cilium/hubble/events.log`. The keys used here exist.
+- **Alloy's pipeline shape**, read from `platform/monitoring/alloy.yaml`: a DaemonSet with
+  `loki.source.kubernetes "pods"` → `loki.process` → `loki.write` at
+  `http://loki.monitoring.svc.cluster.local:3100`. It already tails kube-system pods, which is why
+  stdout needs no Alloy change.
+- **All three value files agree** (one `destination_port` occurrence each) and both YAML files parse,
+  with `export.static` resolving to the intended object.
+- **Toolbox gate**: **rc 3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
+  `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two known environmental skips
+  (`gitea-settings` — no local `SETTINGS_API_TOKEN`; `conformance` L2/L3 — `CLUSTER_AVAILABLE`
+  unset). **Not green. Reported as rc 3.**
+- **No sensitive paths touched** — `platform/cilium/**` and `infra/ansible/**` are not in
+  `infra/gitea/approval-policy.json`, so this needs no approval override, unlike PR #48.
+
+## What I did NOT verify
+
+- **THE OUTCOME. Nothing has been reconciled and no flow has ever been exported.** `cilium-config`
+  still has no `hubble-export-*` key. Every claim is about inputs.
+- **Whether `filePath: stdout` is accepted as written.** This is the single largest risk. The value
+  comes from a verbatim upstream doc sentence (*"`stdout` can be defined as
+  `hubble-export-file-path`"*), but I did not confirm the exact notation in cilium 1.20.1's source,
+  and the chart default is a real filesystem path. If the literal string is not special-cased, the
+  agent may create a FILE named `stdout` in its working directory and nothing reaches Loki — a
+  silent no-op, not a crash. **Fallback if so:** leave `filePath` at the chart default and add a
+  `local.file_match` + `loki.source.file` target to Alloy with a hostPath mount for
+  `/var/run/cilium/hubble`.
+- **Whether the FlowFilter JSON is accepted.** `{"destination_port":["32400"]}` follows the proto
+  (`destination_port` is a repeated string) and the documented shape (`'{"verdict":["DROPPED"]}'`),
+  but upstream gives **no port example** and I did not validate it against the running agent. A
+  malformed filter could be rejected — or, worse, be silently treated as "no match" and export
+  nothing.
+- **THIS CHANGE ROLLS CILIUM.** Editing the HelmRelease restarts the agent DaemonSet across all
+  three nodes. BPF maps are preserved so established traffic should survive, but prod-3 currently
+  announces the Gateway VIP, **PiHole's DNS VIP** and Plex's VIP, so a lease change during the
+  rollout could briefly interrupt LAN DNS for the whole house. Not measured, and worth doing at a
+  quiet moment.
+- **Log volume in practice.** Estimated "a few hundred to a few thousand lines a day" from the
+  observed request counts; never measured as flows. Both directions are exported and no `fieldMask`
+  is set, so each record is a full flow object. If it is noisy, narrow with `fieldMask` rather than
+  by dropping a filter.
+- **Loki's retention for this data**, and whether the `{namespace="kube-system", app="cilium"}` label
+  set actually matches (it depends on the cilium pods carrying
+  `app.kubernetes.io/name`, which Alloy maps to `app` — I did not confirm that label exists on the
+  cilium DaemonSet pods). The query in the comment may need adjusting to `pod=~"cilium-.*"`.
+- **That the exported flow actually contains the pre-SNAT address.** Hubble's *live* stream does
+  (measured). I am assuming the exporter sees the same flow objects; I did not confirm the exporter
+  is not fed a post-translation view.
+- **The parity convention has no automated check.** I kept all three files in sync by hand because
+  the comments demand it; nothing in `ci/` or `conformance/` verifies cilium Helm↔Ansible parity, so
+  a future edit can still drift them silently.

decisions/2026-10-04-oauth2-sidecar-metrics-scrape-netpol.md

@@ -0,0 +1,155 @@
+---
+date: 2026-10-04
+branch: fix/oauth2-sidecar-metrics-scrape-netpol
+title: The forward-auth gate sidecars' /metrics was NetworkPolicy-blocked — the renderer now opens 44180 to ns monitoring
+---
+
+> **Added 2026-10-05, after merge:** the top item under *What I did NOT verify* below — "that the
+> scrape now succeeds" — **has since been verified on prod**. All 6 targets read `up == 1`. The
+> original text is left exactly as written; the measurements are in
+> [[2026-10-05-oauth2-sidecar-scrape-verified]].
+
+## What changed
+
+`catalog/render.py`'s `r_monitoring_ingress_policy` now emits its CiliumNetworkPolicy for a
+**second** reason: the forward-auth gate sidecar's own `/metrics` port (44180), keyed on
+`_needs_oauth2_sidecar(s)`. Previously it fired only from `observability.metricsEndpoint`.
+
+That matters more than it sounds, because **no ServiceDefinition in this catalog declares a
+`metricsEndpoint`** (measured: zero grep hits across `catalog/*.yaml`). So the function was wired
+in at the call site and had **never emitted a single object on any branch** — correct, and
+entirely latent. Its presence in the tree read as "scrape ingress is handled". It was not.
+
+Re-rendered `apps/` from that change: five new CNPs, one per forward-auth app
+(`jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`), each opening **only** 44180 to **only**
+`fromEndpoints: [k8s:io.kubernetes.pod.namespace: monitoring]`.
+
+The bug being fixed: `platform/oauth2-proxy/podmonitor.yaml` landed 2026-10-02T04:58:44Z and
+discovered 6 targets that have **never once scraped successfully**. Each app namespace ships a
+`default-deny` NetworkPolicy with `policyTypes: [Ingress, Egress]` and **no `ingress` key at all**
+— deny-all-ingress in NetworkPolicy semantics — and the only ingress allowance is
+`<app>-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180. Prometheus matched no
+clause, so its packets were dropped.
+
+Two deliberate choices, both recorded in the function's docstring:
+
+- **Clause 2 keys on `_needs_oauth2_sidecar`, not on `observability.metrics`.** It has to match
+  the condition the *scrape* keys on: `podmonitor.yaml` selects pods by the
+  `homelab.keithjackson.dev/auth-mode: forward-auth` pod label and nothing else, so a target
+  exists iff the sidecar exists. Keying on `observability.metrics` would let `metrics: false`
+  produce a discovered-but-blocked target — the same "two halves authored apart" failure this
+  function exists to prevent, moved one field over.
+- **The policy is rendered here, not added to `platform/oauth2-proxy/`.** podmonitor.yaml's own
+  header records why that tier cannot host it: it would name app namespaces created by the
+  *sibling* `apps-wave-0/1` tiers, so it would apply on a running cluster and fail on a cold
+  bootstrap. Rendered per-app it ships in the same tier that creates the namespace.
+
+## What I verified
+
+Measured on prod 2026-10-04, read-only, via the pinned toolbox:
+
+- **The cause, not a guess.** All 6 sidecar targets in pool
+  `podMonitor/oauth2-proxy/oauth2-proxy-sidecars/0`: `health=down`,
+  `lastError: context deadline exceeded`, `lastScrapeDuration` exactly `10.0s` — the configured
+  `scrapeTimeout`. Hitting the timeout precisely means packets **dropped, not refused**.
+- **The listener is fine.** jackett's `oauth2-proxy` container carries
+  `--metrics-address=0.0.0.0:44180` and a named port `metrics: 44180`.
+- **The deny is universal.** All six namespaces (`jackett radarr sonarr qbittorrent pihole
+  crafty`): `default-deny` with `ingress` key absent, `nIngress=0`; the only ingress CNP allowance
+  is 4180 `fromEntities: [ingress]`. **Zero** namespaces had any `-allow-monitoring-scrape` object.
+- **Frequency.** Single continuous episode, no flapping: `ALERTS_FOR_STATE` = 229343s (63.7h),
+  firing since 2026-10-02 05:09 UTC — 11 minutes after the PodMonitor was created, i.e. TargetDown's
+  `for: 10m`. A 15-day `query_range` shows **1 episode** per namespace, not many. Alertmanager:
+  `group_by: [alertname, namespace]`, `repeat_interval: 12h` on `email-warning` → 5 TargetDown
+  groups = 10 emails/day, plus 6 `OAuth2ProxyGateUnobserved` groups = 12/day, ~22/day total.
+  0 silences.
+- **The re-render is deterministic and the diff is exactly the intended one.** `apps/` is
+  byte-identical to `render.py --all catalog -o apps --wave-layout --no-image-automation`
+  (12/12 SAME before my edit; the 11-file "drift" against a default render is solely the
+  ImageRepository/ImagePolicy pair, which `apps/` is generated without). After the edit:
+  5 files changed, +21 lines each, nothing else touched.
+- `render.py --selftest`: **PASS**.
+- **Three L1 goldens needed regenerating, and the diff is +63 lines / -0.** The first gate run
+  caught `container-workload-render` (R-PLAT-02) and `egress-class-render` (R-NET-05/06/07) as
+  FAIL with exactly the new CNP as the drift. Regenerated with the same invocations the checks
+  use (`render.py <fixture>`, no flags). Evidence the regen is faithful rather than a rubber
+  stamp: `default-deny-baseline-render`'s golden came back **byte-identical** (its fixture is not
+  forward-auth), and the other three gained only the one CNP, port 44180, ns monitoring.
+- Toolbox gate `./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`: **rc 3 — UNDETERMINED,
+  NOT rc 0, and I am not reporting it as green.** Every stage that ran is green
+  (spec-immutability, approval-policy, decisions-log, lint, l0, render-golden, l1); `l1` went
+  FAIL -> pass once the goldens were regenerated. The gate's own words: "Stages that did not run:
+  gitea-settings" — skipped for want of `GITEA_SETTINGS_TOKEN`, which is an environment gap on
+  every local run and unrelated to this change. L2/L3 conformance is a `[warn]`, not a not-run
+  stage: UNVERIFIED for all 8 touched requirements (R-GITOPS-02 R-IDP-02 R-IDP-06 R-NET-01
+  R-NET-05 R-NET-06 R-NET-07 R-PLAT-02), no cluster.
+- `--emit-waves` was **not** run (it re-onboards `vm-demo` + `keycloak`); only the per-service
+  render path was used, so no wave kustomization changed.
+
+## This PR needs a human approval (sensitive path) — MEASURED, not predicted
+
+CI run #230 on PR #49, head `f9446be3`: **`MERGE GATE: BLOCKED`, and `approval-policy` is the ONLY
+failing stage.** Read from the job log, not from the status tick:
+
+    [skip] stage spec-immutability: SKIPPED (not run)   <- sibling spec repo absent in CI; expected
+    [ ok ] stage decisions-log: pass
+    [ ok ] stage lint: pass
+    [ ok ] stage l0: pass
+    [ ok ] stage render-golden: pass
+    [ ok ] stage l1: pass
+    [fail] stage approval-policy: FAIL
+    [ ok ] stage gitea-settings: pass                   <- could NOT run locally; green in CI
+    [skip] stage conformance: SKIPPED (not run)
+
+    sensitive paths touched:
+      conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml
+      conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml
+      conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml
+
+Exactly the three regenerated goldens, nothing else. `l1` and `render-golden` both pass in CI,
+which is the independent confirmation that the regenerated goldens are right rather than merely
+self-consistent with my local run.
+
+One local-reproduction note worth recording, because it wasted a step: running
+`PR_NUMBER=49 SETTINGS_API_TOKEN=... ci/approval-policy.sh` with the credential from
+`git credential fill` reports `could not determine PR author — BLOCKING`. That is an **artifact**,
+not a finding — `~/.git-credentials` holds a *password*, and `Authorization: token <password>`
+returns 401 `user does not exist [uid: 0]`. The stage needs a real API token. I nearly reported
+that 401 as the CI failure; the job log is what settled it.
+
+Regenerating the L1 goldens means this branch touches `conformance/**`, which is in
+`sensitive_paths` in `infra/gitea/approval-policy.json` ("the tests and the harness; changing them
+changes what green MEANS"). So `ci/approval-policy.sh` will require an approving review from a
+non-author, and the local gate **cannot** see that: it reports `approval-policy: green` because
+offline it evaluates N/A.
+
+The golden update is not optional — a renderer change that adds an object necessarily drifts the
+goldens that pin the renderer's output — but it does mean the gate correctly refuses to
+self-approve this one. **An agent must not clear this with `/approve-override`**, even though the
+`gitea-admin` account holds `owner`: that is precisely the self-approval the stage exists to
+prevent. The operator posts the override as a PR *review* of type "Comment", or approves normally.
+
+## What I did NOT verify
+
+- **That the scrape now succeeds.** This is a git-only change and nothing is applied until Flux
+  reconciles `main`. The 5 CNPs have never existed on the cluster, so `up == 1` for those targets
+  is **UNVERIFIED** — it is a prediction from the measured drop signature, not an observation.
+  After merge, confirm `up{container="oauth2-proxy"}` goes to 1 for all 6 targets and that the 11
+  alerts resolve. `send_resolved: true` is set, so expect ~11 resolved emails.
+- **That the CNP `endpointSelector` matches pihole's pods.** It selects
+  `app.kubernetes.io/name: pihole`, identical to the shape of `pihole-allow-gateway-ingress`
+  which demonstrably works for 4180 — so this is inference from a working sibling, not a
+  measurement of the new object.
+- **`crafty` is still unobserved and will keep firing `OAuth2ProxyGateUnobserved`.** It is
+  hand-managed under `platform/crafty`, outside the renderer, and has **no `--metrics-address`
+  at all** — so there is no port to open and this change cannot help it. Separate work.
+- **Whether 44180 is reachable in the Cilium data path once allowed.** Only the policy was
+  reasoned about; no L2 probe was run (CI has `vars.CLUSTER_AVAILABLE` unset, so L2/L3 did not
+  run here either — the gate proves the L0/L1 offline subset only).
+- **Why `OAuth2ProxyGateUnobserved` started at 2026-10-04 06:44** rather than 10-02 with
+  TargetDown. Alertmanager did not restart (TargetDown's `startsAt` is still 10-02) and the rule's
+  `for` is 1h, so neither explains it. Something changed on the monitoring side that morning which
+  I did not identify. It does not affect this fix.
+- I did not re-run a port-forward to 44180 to prove the endpoint serves *today*; I confirmed the
+  flag and named port on the live pod spec. The 200/10208-byte measurement is podmonitor.yaml's,
+  from 2026-10-01.

decisions/2026-10-04-pihole-declare-state.md

@@ -0,0 +1,133 @@
+---
+date: 2026-10-04
+branch: feat/pihole-declare-state
+title: PiHole's operator-set state is now declared in git — the prerequisite for running more than one replica
+---
+
+## What changed
+
+Phase 1 of making LAN DNS survive a node failure. The operator asked for multi-replica PiHole;
+measurement showed that cannot be done safely first, because PiHole's important state was not
+reproducible from git. A second replica with a fresh PVC would have come up **serving DNS and
+blocking nothing, with a different admin password** — answers varying by which pod replied. This
+branch closes that; the replica work is phase 2 and is deliberately NOT here.
+
+**The undeclared-state inventory, measured not guessed.** `pihole.toml` marks every non-default
+setting `### CHANGED`, and those that came from the environment `### CHANGED (env)`. The complete
+live list on 2026-10-04:
+
+| Setting | Status |
+|---|---|
+| `dns.upstreams`, `dns.listeningMode`, `dns.rateLimit.count`, `misc.dnsmasq_lines` | `### CHANGED (env)` — already declared in `catalog/pihole.yaml` |
+| `files.macvendor` | `### CHANGED` — set by the image itself (container path), not operator state |
+| `webserver.api.pwhash` | **`### CHANGED` — set out of band. Declared NOWHERE.** |
+
+So exactly **one** operator-set value lived only on the PVC. Separately, the **adlists** were also
+undeclared: `gravity.db` held one list (StevenBlack's hosts) and 72,234 gravity domains, with nothing
+in this repo naming it.
+
+- **`apps/wave-0/pihole-bootstrap-secret.sops.yaml` (new, SOPS/age).** Secret `pihole-bootstrap`,
+  key `webui-password`. Named to the `<svc>-bootstrap-secret.sops.yaml` convention so
+  `render.py --emit-waves` places it with no renderer change. The value is a 32-char random password
+  generated at authoring time and **never printed to a transcript**.
+- **`catalog/pihole.yaml`:** `FTLCONF_webserver_api_password` sourced from that Secret.
+  `webserver.api.password` takes CLEARTEXT and FTL hashes it itself — verified in the running image,
+  where `pihole setpassword` *is* `setFTLConfigValue "webserver.api.password"`, commented upstream
+  "pihole-FTL will automatically hash the password". The PVC's `pwhash` becomes a derived artefact.
+- **`catalog/pihole.yaml`:** a `lifecycle.postStart` hook that **reconciles** the declared adlist set
+  into `gravity.db` on every container start — adds missing, re-enables disabled, and **disables**
+  (never deletes) anything enabled but undeclared, running `pihole -g` only when something moved.
+  Same level-triggered idiom, and the same reasoning, as `catalog/qbittorrent.yaml`'s postStart.
+- **`apps/wave-0/pihole.yaml`, `apps/wave-0/kustomization.yaml`:** re-rendered / secret registered.
+
+**Why a postStart hook and not a file.** PiHole v6 keeps adlists as DB rows. `/etc/pihole/adlists.list`
+still exists in `gravity.sh` but is **migration-only**: its import sits inside `migrate_to_database()`
+behind `if [ ! -e "$gravityDBfile" ]`, so it fires only while *creating* a database. A ConfigMap
+mounted there would be silently ignored on every pod this cluster runs today. There is also no
+`pihole adlist` CLI subcommand, so SQL is the actual interface; FTL ships an embedded shell
+(`pihole-FTL sqlite3`) so nothing had to be added to the image.
+
+**The password is kept rather than set empty, deliberately.** The admin UI is already behind the
+oauth2-proxy sidecar (`--allowed-group=platform-admins`), so an empty password would have been
+simpler. But the `pihole` Service still publishes **port 80** (PiHole direct, bypassing the proxy on
+`:4180`) and only the CiliumNetworkPolicy closes it — and this repo has a documented default-deny
+drift history. An unauthenticated admin UI that can rewrite LAN DNS should not rest on one policy
+staying correct. Narrowing that Service port is follow-up, not done here.
+
+**SENSITIVE PATHS — this PR adds a `**/*.sops.yaml`, so `ci/approval-policy.sh` will BLOCK it.** It
+needs the operator's `/approve-override <reason>` as a PR **review** of type "Comment". Not posted by
+me: CLAUDE.md section 1 forbids an agent clearing this on its own PR.
+
+## What I verified
+
+Live prod, 2026-10-04, read-only except where noted.
+
+- **The undeclared-state inventory above** — read from `pihole.toml`'s own `### CHANGED` markers, with
+  the password value masked. This is the measurement the whole branch rests on.
+- **A fresh `gravity.db` seeds NO adlists.** The image's own schema template
+  `/etc/.pihole/advanced/Templates/gravity.db.sql` creates the `adlist` table and contains no
+  `INSERT INTO adlist` at all — grepped in the running image. Confirmed empirically below (0 rows).
+  This is why the adlist declaration is load-bearing and not cosmetic.
+- **The adlists.list path really is migration-only** — read the guard in `/opt/pihole/gravity.sh`
+  (`migrate_to_database()`, `if [ ! -e "$gravityDBfile" ]`).
+- **`webserver.api.password` is a real settable key** — `pihole-FTL --config webserver.api.password`
+  returns a masked value, and `/usr/local/bin/pihole`'s `setpassword` sets exactly that key.
+- **THE RENDERED HOOK ACTUALLY WORKS — all four paths, extracted from `apps/wave-0/pihole.yaml` and
+  run in the live pod against THROWAWAY COPIES of `gravity.db` in `/tmp`** (production `gravity.db`
+  was never written, and `pihole -g` was stubbed so no 70k-domain download could start):
+  1. *Steady state* → `already matches the declared set - nothing to do`. No writes, no rebuild.
+  2. *True fresh DB built from the image's schema template* → 0 adlists before; hook added the
+     declared list and triggered the rebuild. **This is the fresh-PVC/new-replica case.**
+  3. *An undeclared list enabled* → disabled it (`enabled=0`), left the declared one alone, and did
+     not delete the row.
+  4. *Declared list present but disabled* → re-enabled it.
+- **Required tools exist in the image:** `bash`, `expr`, `grep`, `printf`, `sleep`, `pihole-FTL`.
+- **The Secret is genuinely encrypted** to the repo's age recipient; `stringData.webui-password`
+  starts `ENC[`. The gate's `no-plaintext-secrets (R-SEC-01)` L0 check passes.
+- **Schema:** `check-jsonschema` against `schemas/servicedefinition.schema.json` passes.
+- **Render determinism:** the canonical
+  `render.py --all catalog -o apps --wave-layout --no-image-automation` changed **only**
+  `apps/wave-0/pihole.yaml`. `render-golden` passes, so the hook's text also satisfies
+  `assert_substitution_safe` (no braced dollar expansion — the Flux envsubst contract).
+- **Toolbox gate:** rc **3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
+  `render-golden`, `l1`, `approval-policy` all **pass**. The rc 3 is the two usual environmental
+  skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`) and `conformance` L2/L3
+  (`CLUSTER_AVAILABLE` unset). **Not green; reported as rc 3.**
+
+## What I did NOT verify
+
+- **Nothing here has been reconciled.** Flux has not applied any of it, so the password has never
+  been set by this mechanism and the hook has never run during a real container start. Everything
+  above is the rendered artifact tested out-of-band, plus the image's own source.
+- **THE CURRENT ADMIN PASSWORD WILL STOP WORKING, and that is the intent.** FTL re-hashes
+  `webserver.api.password` on start, overwriting the existing `pwhash`. Whoever knows the present
+  password (nobody, which is why this started) loses it; the declared one replaces it. I have not
+  observed that replacement happen.
+- **Cold-start cost is unmeasured and could matter.** `postStart` must complete before the container
+  is reported Running. On a fresh PVC the hook waits for `gravity.db` (bounded, up to 180s) and then
+  runs a real `pihole -g` (72k domains, network downloads). `apps-wave-0` is `wait: true` and
+  `apps-wave-1` `dependsOn` it, so a slow cold start **delays the app tier**. The steady-state path
+  exits in about a second (scenario 1), so this is a rebuild/first-boot cost only — but it is not
+  measured, and the stub in testing means `pihole -g`'s real duration here is unknown.
+- **Whether `pihole -g` succeeds from inside the hook's environment.** Its failure is caught and the
+  existing `gravity.db` retained, so a failure degrades rather than breaks — untested live.
+- **Concurrency.** Nothing serialises the hook against FTL's own scheduled gravity update. SQLite
+  handles the locking, and the hook's writes are three small statements, but no interleaving was
+  exercised.
+- **MULTI-REPLICA — PHASE 2, NOT STARTED.** This branch only removes the blocker. The remaining work,
+  and it is substantial: a `replicas`/placement field on the ServiceDefinition schema (currently
+  `additionalProperties: false` with no such field), `render.py` emitting a StatefulSet with
+  `volumeClaimTemplates` (the config PVC is **ReadWriteOnce**, so a 2-replica Deployment cannot
+  work), pod anti-affinity, a PodDisruptionBudget (there is none today), re-rendered goldens, and
+  `conformance/**` updates — the last making phase 2 sensitive-path as well. The PVC rename a
+  StatefulSet forces will discard the existing `gravity.db`, which is only safe **because** of this
+  branch, and should be done only after this one is confirmed working in-cluster.
+- **The real risk the operator named is NOT fixed by phase 2 either.** Replicas address *node*
+  failure. LAN DNS still depends on the cluster as a whole — etcd quorum, Cilium, Longhorn, a bad
+  reconcile. Quantified on 2026-10-04: with `tolerationSeconds=300` plus node-NotReady detection
+  plus a Longhorn RWO force-detach, a failure of the node running the pod is roughly **6-10 minutes
+  of no LAN DNS**; a failure of the node holding the `pihole-dns-*` L2 lease (currently a
+  *different* node — lease on k3s-prod-2, pod on k3s-prod-1) is tens of seconds. An out-of-cluster
+  secondary resolver carrying the same `address=` lines is the only option that removes the
+  dependency, and it was considered and not chosen.
+- **Port 80 on the `pihole` Service is still published** and still closed only by policy. Not narrowed.

decisions/2026-10-04-pihole-ha-verified.md

@@ -0,0 +1,117 @@
+---
+date: 2026-10-04
+branch: docs/pihole-ha-verified
+title: PiHole HA is VERIFIED on live prod — including a failover drill with zero dropped DNS queries
+---
+
+## What changed
+
+**No manifests changed. This entry exists to correct the log.** The three PiHole/Plex entries merged
+earlier today each say, correctly at the time, that nothing had been reconciled and that the outcome
+was unverified:
+
+- `decisions/2026-10-04-pihole-declare-state.md` — "Nothing here has been reconciled."
+- `decisions/2026-10-04-pihole-multi-replica.md` — "Nothing has been reconciled. No StatefulSet
+  exists, no second replica has ever run, the anti-affinity has never been evaluated..."
+
+All three are now **merged, reconciled and measured**, so those sentences are stale in exactly the way
+§2b of CLAUDE.md warns about — a reader who found them would conclude the work is unproven. This entry
+is the current state; it does not edit the earlier files (separate files, nothing to merge).
+
+Delivered and live as of 2026-10-04, from PRs #43 (plex tcp/32400), #44 (declare PiHole state) and
+#45 (PiHole 2 replicas + the mountPath bug):
+
+- PiHole is a **2-replica StatefulSet**, one pod per node, with per-replica Longhorn volumes, a
+  PodDisruptionBudget, and an `up` SLI that follows the rendered workload kind.
+- PiHole's admin password and adlists are **declared in git** and reproduce onto a fresh volume.
+- PiHole's config PVC is mounted where PiHole actually writes (`/etc/pihole`), which it was not before.
+
+## What I verified
+
+All measured against live prod on 2026-10-04, after the merge of #45 (`dce20b7`).
+
+**Reconcile.** `GitRepository` artifact `main@sha1:dce20b7`; `apps-wave-0`
+`lastAppliedRevision=main@sha1:dce20b7`, `Ready=True/ReconciliationSucceeded`. It completed in roughly
+**90 seconds**, well inside the Kustomization's `timeout: 5m` — the risk flagged in the multi-replica
+entry ("whether `pihole -g` finishes inside the 5 m timeout") did **not** materialise. A transient
+`Ready=False/DependencyNotReady: platform-config` seen mid-flight was the dependency cascade, not a
+failure; `platform-config` was already `Ready=True` at the new revision moments later.
+
+**Topology and objects.**
+
+| Measured | Value |
+|---|---|
+| pods / nodes | `pihole-0` → **k3s-prod-3**, `pihole-1` → **k3s-prod-2** (2/2 Running each) — the required anti-affinity was satisfied |
+| per-replica claims | `config-pihole-0`, `config-pihole-1` — both `Bound`, 2Gi |
+| old `pihole-config` PVC | **pruned** (it held only `lost+found`, so nothing was lost) |
+| old `pihole` Deployment | **pruned** — `No resources found in pihole namespace` |
+| PodDisruptionBudget | `minAvailable 1`, `disruptionsAllowed 1`, `currentHealthy 2` |
+
+**The mountPath bug is fixed.** `df` inside both pods reports `/etc/pihole` on
+`/dev/longhorn/pvc-...` with **6.9 M used** — previously it was on `overlay` while the Longhorn volume
+sat unused at `/data/config`. `backup.policy: daily-offcluster` now has real content to back up
+instead of reporting success over an empty volume.
+
+**Phase 1's declarations reproduce onto a FRESH volume — the point of doing it first.** Both replicas
+booted with an empty `/etc/pihole`:
+
+- **72,234 gravity domains** on each, rebuilt from the adlist declared in `catalog/pihole.yaml`.
+- **The declared SOPS secret authenticates on both pods.** `POST /api/auth` using the container's own
+  `FTLCONF_webserver_api_password` returned `"valid":true` on `pihole-0` **and** `pihole-1` — so
+  Secret → env → FTL's own hashing → working login, identically on both. (The secret's value was never
+  printed; an earlier reading of `pwhash` as "unset" was a quoting bug in the check, disproven by an
+  empty password being rejected with `password incorrect` and then by this positive test.)
+
+**User-facing behaviour through the VIP (192.168.2.23), from a LAN host.**
+
+    plex.home.keithjackson.dev   -> 192.168.2.20      split-horizon intact
+    home.keithjackson.dev        -> 192.168.2.20
+    unifi.home.keithjackson.dev  -> 192.168.2.21
+    google.com                   -> 142.250.115.102   recursion to 8.8.8.8 works
+    doubleclick.net              -> 0.0.0.0           blocking LIVE in FTL, not just in the DB
+    googleadservices.com         -> 0.0.0.0
+    12 consecutive queries       -> 12/12 answered
+
+**THE FAILOVER DRILL — observed by the operator, 2026-10-04.** A single StatefulSet replica was
+deleted. It was recreated in **~30 seconds**, and **DNS never dropped a request** throughout. This is
+the first direct evidence that the replica count delivers what it was added for, rather than merely
+rendering correctly.
+
+**The SLI is not inert, and re-deriving it mattered.** Measured in Prometheus:
+`kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}` returns **1 series,
+value 2**; `kube_deployment_status_replicas_available{namespace="pihole"}` now returns **0 series**;
+and Prometheus has `pihole-down` loaded as `state=inactive health=ok` with the StatefulSet expression.
+Had the rule stayed Deployment-shaped, its `or absent(...)` branch would have matched the now-absent
+series and **fired permanently** — a standing false page rather than silence. That is the concrete
+payoff of defaulting `livenessWorkload.kind` from the rendered kind.
+
+## What I did NOT verify
+
+- **NODE failure, as opposed to pod failure.** The drill deleted a pod, which does not exercise the
+  `tolerationSeconds: 300` path at all — that timer only applies when a node goes NotReady. The
+  mechanism protecting DNS is the same one just proven (the surviving replica plus
+  `externalTrafficPolicy: Cluster` on the DNS LoadBalancers), so user-visible impact should likewise be
+  nil; what the 300 s still delays is the *replacement* replica, and during that window there is no
+  redundancy left. Untested. Pulling power from a node is the only real test.
+- **DNS during the cutover itself.** I was not sampling resolution while #45 reconciled, so whether
+  there was a brief gap between the Deployment being pruned and the StatefulSet becoming Ready is
+  **unmeasured**. The drill above covers the steady-state failover case, not that one-off transition.
+- **Two-node loss.** The hard anti-affinity means `replicas: 2` needs two schedulable nodes; losing
+  two would leave a replica `Pending` and, with `wait: true`, stall the app tier. Reasoned, not
+  exercised — and it coincides with etcd quorum loss on a 3-node control plane.
+- **Longhorn capacity headroom.** Two 2Gi claims at 3 replicas each is ~12Gi where there was ~6Gi.
+  Both claims are `Bound` and healthy; free space was not checked.
+- **Query history is now split per replica** and the admin UI shows only whichever pod answered.
+  Adlists and password are identical, so blocking and auth are consistent; the *reporting* is not, and
+  whether that is acceptable long-term is the operator's call.
+- **The residual risk is unchanged, and it is the larger one.** Replicas address node failure. LAN DNS
+  still depends on the cluster — etcd quorum, Cilium, Longhorn, a bad reconcile — and only a resolver
+  outside the cluster carrying the same `address=` lines removes that dependency. Considered and not
+  chosen (operator decision, 2026-10-04).
+- **Plex remote access (#43) is NOT covered by this entry.** Its two manual steps (the WAN tcp/32400
+  forward and Plex's "manually specify public port") and the outside-in path remain as
+  `decisions/2026-10-03-plex-remote-access-l4.md` describes them. The operator reported it working;
+  I have not measured it from off-LAN.
+- **`--emit-waves` drift is still open.** Re-running it still wants to onboard `vm-demo.yaml` and
+  `keycloak.yaml`, both listed as intentionally not onboarded in `ci/render-golden.sh`. Unfixed; see
+  `decisions/2026-10-03-plex-remote-access-l4.md`.

decisions/2026-10-04-pihole-multi-replica.md

@@ -0,0 +1,145 @@
+---
+date: 2026-10-04
+branch: feat/pihole-multi-replica
+title: PiHole runs two replicas on different nodes — and its config PVC was mounted where PiHole never writes
+---
+
+## What changed
+
+Phase 2 of making LAN DNS survive a node failure (phase 1:
+`decisions/2026-10-04-pihole-declare-state.md`). PiHole was one pod on one node, which for a
+whole-house resolver cost more than "until it reschedules" — measured 2026-10-04:
+
+| Failure | Recovery |
+|---|---|
+| the node running the pod dies | ~40s to mark NotReady + `tolerationSeconds: 300` before eviction + reschedule + a Longhorn RWO force-detach → **roughly 6-10 minutes with no LAN DNS** |
+| the node holding the `pihole-dns-*` L2 lease dies | Cilium re-elects an announcer, tens of seconds |
+
+Those were **different** nodes (lease on k3s-prod-2, pod on k3s-prod-1), so two of three nodes were
+single points of failure for LAN DNS with wildly different blast radii.
+
+**A new schema field, because this is intent and not a primitive.** `availability.replicas`
+(schema **1.10.0**, mechanism in `schemas/renderer-contract.md` `### availability`). `1` or absent
+renders a Deployment exactly as before — every pre-existing golden is byte-identical, verified. `>1`
+renders, together:
+
+- a **StatefulSet** instead of a Deployment. This is forced, not preferred: a longhorn PVC is
+  **ReadWriteOnce**, and one RWO claim cannot attach on two nodes, so a 2-replica Deployment does not
+  degrade — the second pod never schedules.
+- longhorn storage as **per-replica `volumeClaimTemplates`** (`config-pihole-0`, `config-pihole-1`);
+  the standalone PVC is no longer emitted, since it would be an orphan nothing mounts. A `bulk`
+  entry stays a standalone PVC — it is a shared static NFS export, not per-replica state.
+- **required** `podAntiAffinity` on `kubernetes.io/hostname`. `preferred` would let both replicas
+  land on one node and deliver none of the intended tolerance while reporting healthy.
+- `podManagementPolicy: Parallel`, so a cold start costs the max of the replicas' first-boot work
+  rather than the sum — `apps-wave-0` has `timeout: 5m`.
+- a **PodDisruptionBudget**, `minAvailable: 1`, so a node drain cannot take the last replica.
+- `observability.livenessWorkload.kind` defaulting to `StatefulSet`, so the `up` SLI reads
+  `kube_statefulset_status_replicas_ready`. A Deployment-shaped rule against a StatefulSet
+  `absent()`s forever — an alert that can never fire.
+
+`catalog/pihole.yaml` sets `replicas: 2`, not 3: two tolerates any single node loss and leaves a
+spare node for the evicted replica under the hard anti-affinity. Three would pin one per node with no
+spare, so a single node failure would leave a replica permanently `Pending` and — `apps-wave-0` being
+`wait: true` — stall the app tier for as long as the node was down.
+
+### The bug found on the way, and fixed here
+
+**PiHole's config PVC was mounted at a path PiHole never writes, so it held nothing and all of its
+state was ephemeral.** With no explicit `mountPath` the renderer defaults to `/data/<name>`; PiHole
+keeps everything in `/etc/pihole`. Measured on live prod 2026-10-04: `/etc/pihole` (7.1 M —
+`gravity.db`, `pihole.toml`, the query DB) was on the **overlay** filesystem, while the 2 Gi Longhorn
+volume sat at `/data/config` holding `lost+found` and 24 K of nothing. Consequences, all live until
+now:
+
+- every restart silently discarded PiHole's entire state and re-downloaded 72 k gravity domains;
+- `backup.policy: daily-offcluster` has been faithfully backing up an **empty** volume, which is
+  worse than no backup because it reports success;
+- it also means phase 1 was understated: that state was not merely undeclared, it was **ephemeral**.
+
+Same class of bug `catalog/plex.yaml` documents for its own `config` entry. Fixed in this branch
+rather than its own because with the wrong path the new per-replica volumes would be two 2 Gi
+Longhorn volumes (×3 Longhorn replicas each) holding only `lost+found` while all state stayed
+ephemeral — paying for persistence and getting none. It also removes this change's main operational
+risk: with `gravity.db` persisted, a restart is cheap instead of re-downloading 72 k domains inside a
+wave with a 5 m timeout.
+
+## What I verified
+
+- **Back-compatibility, the thing most likely to break quietly:** after the renderer change and
+  before `catalog/pihole.yaml` declared anything, the canonical
+  `render.py --all catalog -o apps --wave-layout --no-image-automation` produced **zero diff** across
+  the whole `apps/` tree. `render-golden` passes.
+- **The new behaviour is pinned by ~20 new `render.py --selftest` assertions** (so `ci/render-golden.sh`
+  hard-fails on a regression, and no `conformance/**` change was needed): `replicas: 1` still renders a
+  Deployment + standalone PVC + no PDB; `replicas: 2` renders a StatefulSet and **no** standalone
+  longhorn PVC, carries the replica count, omits `strategy`, sets `Parallel` and `serviceName`, names
+  the claim template exactly as the mounted volume, keeps RWO/longhorn on it, does **not** duplicate
+  that volume in `pod.volumes`, still mounts it on the container, renders required hostname
+  anti-affinity, renders exactly one `minAvailable: 1` PDB, and switches the SLI to
+  `kube_statefulset_status_replicas_ready`. A mixed longhorn+bulk case asserts the bulk share stays a
+  standalone PVC and an ordinary pod volume.
+- **Rendered pihole output, read back:** `replicas: 2`, `serviceName: pihole`,
+  `podManagementPolicy: Parallel`, no `strategy`, claim template `config` (RWO/longhorn/2Gi),
+  anti-affinity `kubernetes.io/hostname`, `pod.volumes: []`, mount `config -> /etc/pihole`,
+  PDB `minAvailable: 1`, SLI `kube_statefulset_status_replicas_ready`.
+- **The `/etc/pihole` mount is safe, evidenced from the image's own init rather than assumed:**
+  `/usr/bin/start.sh` is PID 1 and runs as **uid 0**; its line 5 is
+  `chown -R pihole:pihole /etc/pihole/ /var/log/pihole/`, so a fresh root-owned volume has its
+  ownership fixed before anything else starts (**no `fsGroup` needed, and one would be inert**); it
+  calls `migrate_gravity`, whose `migrate_to_database()` creates the gravity database when
+  `[ ! -e "$gravityDBfile" ]`; FTL writes `pihole.toml` itself from the `FTLCONF_*` env; and the
+  image's own default for `files.macvendor` was moved **out** of that directory, which is what a
+  maintainer does to a path they expect to be a volume.
+- **The PVC really is unused:** `df` showed `/etc/pihole` on `overlay` and
+  `/dev/longhorn/pvc-849edb86-...` on `/data/config`; `ls -A /data/config` → `lost+found` only.
+- **The gate caught a real regression and it is fixed.** The first run came back **rc 1**:
+  `alert-rules-unit-tested` failed because `apps/wave-0/tests/app-slis.test.yaml` fed pihole a
+  `kube_deployment_status_replicas_available` series and expected a `deployment: pihole` label. Three
+  input series, the firing expectation and the file header were updated;
+  `platform/monitoring/tests/run.sh apps/wave-0/tests/app-slis.test.yaml` → **SUCCESS** (6 rules).
+- **Schema:** `check-jsonschema` passes over all of `catalog/*.yaml`.
+- **Toolbox gate:** rc **3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
+  `render-golden`, `l1`, `approval-policy` all **pass**. The rc 3 is the two environmental skips —
+  `gitea-settings` (no local `SETTINGS_API_TOKEN`) and `conformance` L2/L3 (`CLUSTER_AVAILABLE`
+  unset). **Not green; reported as rc 3.**
+- **No sensitive paths touched**, so unlike the previous two branches this one needs no
+  `/approve-override` — confirmed against `infra/gitea/approval-policy.json`.
+
+## What I did NOT verify
+
+- **Nothing has been reconciled.** No StatefulSet exists, no second replica has ever run, the
+  anti-affinity has never been evaluated by the scheduler, the PDB has never been consulted, and the
+  `/etc/pihole` mount has never been exercised on a real pod. Everything above is the rendered
+  artifact plus the image's own source.
+- **THE ROLLOUT WILL BRIEFLY DROP LAN DNS, and I have not measured for how long.** `postStart` must
+  complete before a container is reported Running, so a pod is not Ready — and the Service has no
+  endpoints — until phase 1's adlist hook finishes. On this transition both replicas start with an
+  empty `/etc/pihole`, so the hook finds a fresh `gravity.db` with zero adlists and runs a real
+  `pihole -g` (72 k domains, network download) before either pod becomes Ready. **Do this at a quiet
+  time.** Two mitigating facts: every pihole restart has *already* paid this cost (state was
+  ephemeral), and after this change `gravity.db` persists so later restarts should not. If it turns
+  out slow, backgrounding the gravity rebuild inside the hook would decouple readiness from it, at
+  the cost of a window where DNS resolves but does not block — not done here.
+- **Whether `pihole -g` finishes inside `apps-wave-0`'s 5 m `timeout`.** `Parallel` was chosen to
+  make this the max rather than the sum of two first boots, but the real duration is unmeasured (it
+  was stubbed during phase 1's testing). If the wave times out, the Kustomization reports failed even
+  though the pods may come up seconds later.
+- **The old PVC's deletion.** `apps-wave-0` has `prune: true`, so removing `pihole-config` from the
+  rendered output should delete it. That destroys nothing (it holds `lost+found`), but I have not
+  observed the prune, and I did not check whether Longhorn retains the backing volume afterwards.
+- **The transition overlap.** For a moment the pruned Deployment's pod and the new StatefulSet pods
+  may both match the Service selector. Both are configured identically from git so this should be
+  benign — unobserved.
+- **Longhorn capacity.** Two 2 Gi claims at 3 Longhorn replicas each is ~12 Gi of real storage where
+  there was ~6 Gi. Not checked against free space.
+- **Whether two independent query databases are acceptable to the operator.** Stats and query history
+  now split per replica, and the admin UI shows only the pod that answered. Adlists and password are
+  identical (phase 1), so blocking and auth are consistent; the *reporting* is not.
+- **The residual risk is unchanged and is the larger one.** Replicas address node failure. LAN DNS
+  still depends on the cluster — etcd quorum, Cilium, Longhorn, a bad reconcile — and only a resolver
+  outside the cluster carrying the same `address=` lines removes that. Considered and not chosen
+  (operator decision, 2026-10-04).
+- **`tolerationSeconds: 300` is untouched.** With two replicas the surviving one serves immediately, so
+  the 5-minute eviction delay no longer gates recovery — but it still delays the *replacement* replica,
+  and during that window there is no redundancy left.

decisions/2026-10-04-plex-advertise-plexdirect.md

@@ -0,0 +1,139 @@
+---
+date: 2026-10-04
+branch: fix/plex-advertise-plexdirect
+title: Plex vanished from the plex.tv account — every advertised connection was uncompletable; advertise the plex.direct name instead of the bare VIP
+---
+
+## What changed
+
+Reported symptom: the server no longer appears on the operator's plex.tv account; `plex.home.keithjackson.dev`
+shows "not authorized"; connecting straight to the VIP + port works.
+
+**Root cause, measured: `customConnections` held two hand-typed URLs and NEITHER could ever complete,
+so Plex had nothing publishable.** Because `customConnections` OVERRIDES Plex's own connection
+publishing, the server ended up advertising only dead endpoints. Live `Preferences.xml`, 2026-10-04:
+
+    customConnections="https://plex.home.keithjackson.dev:32400/,https://192.168.2.24:32400/"
+    ManualPortMappingMode="1"   ManualPortMappingPort=<ABSENT>
+
+- `https://192.168.2.24:32400` — **TLS can never validate.** Plex serves 32400 with
+  `CN=*.130072f20c334d6c8f63d0dfef87309b.plex.direct` and **no IP SAN**, so a bare-IP https URL fails
+  for every client. Not theoretical: Plex's log carries a continuous storm of
+  `CERT: incomplete TLS handshake from <node> ... sslv3 alert certificate unknown`, every few seconds,
+  from **real LAN clients** arriving through the new `plex-l4-tcp` VIP.
+- `https://plex.home.keithjackson.dev:32400` — **nothing listens.** The shared Gateway serves that host
+  on 443; measured `http=000` (connect failed) against `192.168.2.20:32400`.
+
+Consequence chain, each link measured: no completable connection -> plex.tv could publish nothing
+(`MyPlex: mapping state set to 'Mapped - Not Published'`, repeatedly) -> Plex fell back to **Plex Relay**
+(`startRelay`, remote forward to 127.0.0.1:32401) -> **the server disappeared from the account**, and
+clients reported "not authorized".
+
+The fix advertises Plex's own `<dashed-ip>.<serverhash>.plex.direct` form, which is public DNS resolving
+to the embedded address — so it points at the LAN VIP *and* matches the wildcard cert.
+
+- **`catalog/plex.yaml`:** `ADVERTISE_IP: ${PLEX_ADVERTISE_URL}` added, replacing the long-standing
+  DEFERRAL note. The deferral had weighed only the L7 Gateway URL; the operator chose the native L4 path
+  on 2026-10-03, so the open question was never "Gateway or not" but "which URL has a valid cert".
+- **`clusters/base/cluster-vars.yaml`:** new `PLEX_ADVERTISE_URL` knob (+ substitution-inventory comment
+  updated to name `plex: PLEX_ADVERTISE_URL`). The base default is an explicit **non-functional
+  placeholder** — dev is decommissioned and has no claimed server, and
+  `StrictPostBuildSubstitutions` makes an undefined var a hard failure, so the default only has to exist
+  and be correctly shaped.
+- **`clusters/prod/kustomization.yaml`:** prod value
+  `https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400`.
+- **`apps/wave-0/plex.yaml`:** re-rendered with the canonical command
+  (`python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation`). Two added
+  lines, nothing else.
+
+**Why a whole-URL cluster-var and not something built from `${PLEX_L4_VIP}`:** postBuild substitution is
+plain envsubst and cannot turn `192.168.2.24` into `192-168-2-24`, and `<serverhash>` is the server's
+certificate identity, derivable from nothing in this tree. The dashed IP and `${PLEX_L4_VIP}` are two
+halves of one address and **nothing checks that they agree** — noted in both files.
+
+**SENSITIVE PATHS — this PR touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK it.**
+It needs the operator's `/approve-override <reason>` posted as a PR **review** of type "Comment".
+Deliberately not posted here: CLAUDE.md §1 forbids an agent clearing this on its own PR, and the agent
+account holds `owner`, so the prohibition is a rule rather than a limit.
+
+## What I verified
+
+All measurements against live prod, 2026-10-04, read-only. Secrets handled per CLAUDE.md §6: preference
+attributes were read by name-scoped grep, never by dumping `Preferences.xml` (it holds `PlexOnlineToken`),
+and the token was reported only as `<PRESENT,len=22>`.
+
+- **THE FIX ITSELF, with full TLS verification (no `curl -k`)** — the one measurement the whole change
+  rests on:
+  - `https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400/identity` -> **`http=200`, TLS verify OK**
+  - `https://192.168.2.24:32400/identity` -> **`http=000`, cert validation FAILED**
+  - that name resolves to `192.168.2.24` (`getent hosts`), confirming the dashed-IP mechanism works from
+    inside this network and is not blocked by DNS-rebinding protection on the resolver path tested.
+- **The server is still claimed — this was never a lost claim.** `/identity` -> `claimed="1"` with the real
+  `machineIdentifier`; `PlexOnlineToken` present, `PlexOnlineUsername="jakman85"`.
+- **The certificate, read off the live listener:** exactly one SAN,
+  `DNS:*.130072f20c334d6c8f63d0dfef87309b.plex.direct`. No IP SAN. This is why the bare-IP URL fails.
+- **The publish payload Plex actually sent to plex.tv**, which is what proves the connection list was the
+  problem rather than a guess: `PUT https://servers.plex.tv/devices/<id>?Connection[][uri]=https://plex.home.keithjackson.dev:32400&Connection[][uri]=https://192.168.2.24:32400&Connection[][uri]=http://136.51.115.252:32400&Connection[][uri]=http://10.0.2.136:32400&httpsEnabled=1...`
+  — all four unreachable from a LAN client or from plex.tv.
+- **`Mapped - Not Published`** at every reachability check across the whole log, and an active **Plex Relay**
+  session — the documented fallback when nothing is directly publishable.
+- **The `certificate unknown` TLS failures come from real client traffic, not a probe.** The source
+  `10.0.1.78` resolves via `cilium-dbg bpf ipcache get` to **identity 1 = `reserved:host`**, i.e. LAN
+  clients SNAT'd through `plex-l4-tcp` (`externalTrafficPolicy: Cluster`) on prod-3.
+- **The Gateway is NOT broken, and my first reading of it was wrong.** A 403 `Access denied` from
+  `server: envoy` on the Gateway path turned out to be an artifact of probing from inside the plex pod:
+  **all 16** Gateway hostnames return 403 from there, because Cilium's Gateway preserves the original
+  source identity and enforces the destination's ingress policy. Plex's own log shows a concurrent,
+  **working, signed-in** browser session through the Gateway returning 200s. The HTTPRoute is
+  `Accepted=True`/`ResolvedRefs=True` with no filters, and it is the only route claiming that hostname.
+- **Cilium is not denying the Gateway->Plex hop:** `cilium-dbg bpf policy get 3796` shows
+  `Allow Ingress reserved:ingress 32400/TCP`, PROXY PORT `NONE`, with 141k packets counted. No L7/HTTP
+  rules and no CCNPs exist anywhere in the cluster.
+- **In-cluster consumers were never affected:** Seerr's recently-added scans and watchlist syncs complete
+  cleanly with no auth errors; there are **zero** HTTP 401s in Plex's log (the five ` 401 ` hits are
+  transcoder segment numbers).
+- **`ADVERTISE_IP`'s mechanism, in the running image rather than from docs:**
+  `/etc/cont-init.d/40-plex-first-run` line 127 does `setPref "customConnections" "${ADVERTISE_IP}"`,
+  outside the claim guard, and `/.firstRunComplete` (lines 9/139) is on the container overlay, not the
+  `/config` PVC — so it re-applies on every pod start. Editing the env rolls the Deployment, which applies it.
+- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
+  `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
+  **pass**. The rc 3 is the two known environmental skips — `gitea-settings` (no local
+  `SETTINGS_API_TOKEN`) and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3.**
+- **`render-golden` passing is the specific proof** that the catalog edit and the committed golden agree;
+  the rendered diff is exactly two lines and no other service's manifest moved.
+
+## What I did NOT verify
+
+- **THE OUTCOME. Nothing here has been observed to put the server back on the plex.tv account.** The
+  change is unreconciled: `ADVERTISE_IP` is absent from the live Deployment, the live
+  `customConnections` still holds the two broken URLs, and Plex has not been restarted. What is proven is
+  that the *new URL answers 200 with a valid certificate* and that *every currently advertised URL cannot
+  complete* — a necessary-and-plausibly-sufficient argument, not a demonstrated fix.
+- **Whether plex.tv will publish and list the server once this lands.** plex.tv's decision is made
+  off-network and cannot be measured from here. A LAN-only connection may still leave the account showing
+  the server as remotely unreachable even when local clients work.
+- **REMOTE (off-LAN) ACCESS IS NOT FIXED BY THIS, and may still be fully broken.** The advertised URL
+  resolves to an RFC1918 address, so it serves LAN clients only. The off-repo WAN tcp/32400 port-forward
+  (`runbooks/plex-remote-access.md`) is still **unverified** — `http://136.51.115.252:32400` was
+  unreachable from inside the LAN, but 2026-10-03 established there is no NAT hairpin here, so that is
+  **inconclusive**, not proof of absence. This repo has no off-LAN vantage point.
+- **`ManualPortMappingMode="1"` with `ManualPortMappingPort` ABSENT is left as-is.** Manual public-port
+  mapping is enabled with no port set — a half-configured state I did not change and whose exact effect on
+  publishing I did not isolate. There is **no GitOps path** to it (2026-10-03: the image supports no
+  `PLEX_PREFERENCE*` of any kind), so it is an operator UI step.
+- **Whether the operator intended the hand-typed URLs** to override the 2026-10-03 native-L4 design, or
+  typed them while debugging. I did not alter live preferences; the env var will overwrite
+  `customConnections` on the next pod start, which is a deliberate transfer of ownership from the web form
+  to git and will **silently discard** anything typed there since.
+- **The `plex.direct` hash's stability.** Treated as server identity and read once from the live cert. I
+  did not establish what reissues it, so this URL could go stale with no alert; nothing in CI or the
+  conformance suite checks that the dashed IP still matches `${PLEX_L4_VIP}` or that the hash is current.
+- **Resolver behaviour for the whole LAN.** `plex.direct` resolved correctly from a *pod* (cluster DNS
+  path). LAN clients resolve via PiHole, and dnsmasq's `stop-dns-rebind` would reject a public answer
+  carrying a private address. I did **not** test resolution from a LAN client, and this is the most likely
+  way the fix still fails.
+- **The CI approval-policy outcome.** The local stage reported green, but that is an offline N/A, not a
+  verdict: this PR touches `clusters/**`, so CI will block pending the operator's override.
+- **Why PMS restarted.** The process came up at 05:26 UTC while the pod is 2d2h old with 0 restarts, so
+  something restarted Plex inside the container. Unexplained and not investigated.

decisions/2026-10-04-plex-unblock-app-tier.md

@@ -0,0 +1,126 @@
+---
+date: 2026-10-04
+branch: fix/plex-unblock-app-tier
+title: Unblock the app tier — my Certificate stalled apps-wave-0, and the advertised https URL had no servable certificate
+---
+
+## What changed
+
+**I broke the app tier with PR #48 and this reverts the part that did it.** Two faults, both mine,
+both measured after the merge:
+
+1. **Kyverno blocks cert-manager in app namespaces.** The temporary private-key Secrets
+   (`plex-vip-<random>`) were denied by `secret-provenance/secrets-must-be-flux-managed`:
+
+       resource Secret/plex/plex-vip-rp4qj was blocked due to the following policies
+       secret-provenance:
+         secrets-must-be-flux-managed: Secrets in application namespaces must be delivered by
+         Flux from SOPS-encrypted git ... or be exempt controller-generated Secrets.
+
+   The policy matches every Secret in a namespace labelled
+   `homelab.keithjackson.dev/app-namespace: true` and excludes only Secrets labelled
+   `homelab.keithjackson.dev/secret-managed-by: controller` or carrying a Flux label. **The `plex`
+   namespace has that label; the `gateway` namespace — where the working wildcard cert lives — does
+   NOT.** That asymmetry is why the existing Certificate was never affected and mine was, and I did
+   not check it. cert-manager's temp key Secrets carry neither exempting label, and `secretTemplate`
+   cannot label them (the keymanager controller creates them, not the Certificate template). Result:
+   **no CertificateRequest, Order or Challenge was ever created.**
+
+2. **Flux health-checks the Certificate — guarding the pod was not enough.** `optional: true` did
+   exactly what it was designed to do: Plex rolled and ran fine without the Secret (verified). But
+   `apps-wave-0` is `wait: true`, so Flux waits on every applied object with a known health type:
+
+       health check failed after 5m0.045564329s: timeout waiting for:
+       [Certificate/plex/plex-vip status: 'InProgress']
+
+   apps-wave-0 never went Ready, and apps-wave-1/-2 plus everything downstream of platform-config
+   sat in `DependencyNotReady` behind it. I reasoned about the pod and missed the object.
+
+A third, independent problem the same PR introduced: **the advertised URL was `https://` with no
+servable certificate.** Plex answers 32400 with its `*.<serverhash>.plex.direct` cert, which cannot
+cover `plex-vip.${BASE_DOMAIN}` — the precise failure mode of the outage earlier that day.
+
+- **`apps/wave-0/plex-cert.yaml` DELETED**, and `plex-cert.yaml` removed from
+  `apps/wave-0/kustomization.yaml` `resources`, so Flux prunes the stuck Certificate and the health
+  gate clears. **Deleted rather than merely unregistered** — my first attempt kept the file as
+  "tracked but not onboarded" (the shape `keycloak.yaml` and `vm-demo.yaml` use) and **the repo's own
+  gate rejected it**: `tls-lifecycle-automated` (R-TLS-02) fails any Certificate in the tree that no
+  Flux path reconciles — *"nothing reconciles it, so a deleted Certificate/issuer/controller would
+  stay deleted until a human re-applied it."* That pattern is legal for a Deployment and not for a
+  TLS object, which is a distinction I had not known.
+- **`catalog/render.py`:** the now-dangling `plex-cert` EXTRA_UNITS entry removed.
+- **The re-enable recipe is recorded at the call site** — in `catalog/plex.yaml` beside the scheme
+  it is coupled to — rather than in the deleted file: the Kyverno exemption, the non-gating
+  Kustomization, and the `https` flip, as one change.
+- **`catalog/plex.yaml`:** `ADVERTISE_IP` → **`http://`**`plex-vip.${BASE_DOMAIN}:32400`, explicitly
+  temporary, with the restore condition written at the call site.
+
+**Everything else from PR #48 is kept and working** — the hostname design, the PiHole split-horizon
+answer, the `LanNetworksBandwidth` hook, the `optional` schema field, and the TLS env (inert without
+the Secret, so re-enabling is additive).
+
+## What I verified
+
+Live prod, 2026-10-04, read-only, after the #48/#50 merges.
+
+- **The stall and its exact cause**: the health-check timeout message above, and the full dependency
+  cascade (`cilium` → "dependency 'flux-system/flux-system' revision is not up to date", everything
+  else `DependencyNotReady`). The root `cilium` wait was a normal revision-gated cascade; the
+  *Certificate* was the real blocker.
+- **The Kyverno denial text, verbatim**, from the cert-manager controller log, repeating every ~30s
+  across at least 15 distinct Secret names.
+- **The policy's own structure**, read from `platform/kyverno/admission/secret-provenance.yaml`:
+  match = Secrets in `app-namespace: true` namespaces; exclude = `secret-managed-by: controller` OR
+  a `kustomize.toolkit.fluxcd.io/name` label; validate = unconditional `deny`.
+- **The namespace-label asymmetry**: `plex` carries
+  `homelab.keithjackson.dev/app-namespace: "true"`; `gateway` does not.
+- **cert-manager's controller ServiceAccount is `cert-manager`** in namespace `cert-manager` — the
+  identity a future exemption should name.
+- **`optional: true` worked**: the new pod (`plex-c4f87c84d-zgjq5`) is `1/1 Running` with no Secret
+  present, and `/config/certs` was never created.
+- **The postStart hook RAN and took effect** — the thing the whole hook was written for:
+  `LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16"` is now set in the live `Preferences.xml`, and
+  `customConnections="https://plex-vip.home.keithjackson.dev:32400"` was applied from the env. The
+  certificate prefs are correctly **ABSENT**, i.e. the hook's "no bundle → leave them alone" guard
+  behaved as designed.
+- **PiHole's split-horizon answer is correct**, asked of `pihole-0` directly:
+  `plex-vip.home.keithjackson.dev -> 192.168.2.24`, alongside `plex -> .20` and `crafty-mc -> .22`,
+  and the live `/etc/pihole/dnsmasq.conf` carries
+  `address=/plex-vip.home.keithjackson.dev/192.168.2.24`. **This corrects my own earlier check**,
+  which read `.20` and which I nearly reported as a failure — that test resolved via CLUSTER DNS
+  (CoreDNS) from inside a pod and never consulted PiHole at all.
+- **The gate caught my first attempt and I fixed it rather than working around it.** Keeping the
+  file unregistered produced `MERGE GATE: BLOCKED` (rc 1) with
+  `CHECK FAIL (tls-lifecycle-automated): 1 of 612 assertions failed`. After deleting the file and
+  the dangling EXTRA_UNITS entry: **rc 3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
+  `render-golden`, `l1` and `approval-policy` all **pass** — the rc 3 being only the two known
+  environmental skips (`gitea-settings`, `conformance` L2/L3). **Not green. Reported as rc 3.**
+
+## What I did NOT verify
+
+- **Whether this actually unblocks the tier.** Unreconciled at time of writing. The reasoning is
+  that pruning the Certificate removes the only failing health check, but I have not seen
+  apps-wave-0 go Ready.
+- **Whether `http://` restores client connectivity.** PiHole resolves the name and Plex serves 32400
+  (both measured), but I have not watched a Plex client complete a connection over the advertised
+  URL, and I did not re-check whether plex.tv publishes it.
+- **THE REMOTE BADGE — the symptom that started this.** `LanNetworksBandwidth` is set, but the log
+  still tags traffic `Allowed Network (WAN)`: **82 WAN, 0 LAN**, with entries continuing after the
+  pref was written. So either Plex needs a restart for it to take effect, or that log tag is not
+  driven by this pref at all. **I do not know which, and the user-visible badge is the only real
+  test** — it is in the Plex client, which I cannot see.
+- **The Hubble flow export is NOT active.** The cilium agents are still the pre-merge pods, so the
+  `filePath: stdout` question is still open and no flow has been exported. The Cilium rollout — and
+  its LAN-DNS risk, since prod-3 holds the Gateway, PiHole-DNS and Plex VIPs — is still AHEAD.
+- **Whether a cert-manager exemption is the right shape**, or whether it widens
+  `secret-provenance` more than the operator wants. It is `platform/kyverno/**`, a sensitive path,
+  and deliberately not attempted here.
+- **Whether the pruned Certificate leaves residue** — I did not check for orphaned
+  `plex-vip-<random>` Secrets or CertificateRequests after the prune. Note Kyverno DENIED every one
+  of those Secrets, so there may be nothing to clean, but that is an inference, not a check.
+- **Whether `tls-lifecycle-automated` would also have objected to the Certificate living in a
+  `wait: false` Kustomization** (the recommended re-enable shape). It demands a reconciling Flux
+  path, which that provides, but I did not test it.
+- **The unencrypted advertised URL.** `http://` is tolerable only while remote access does not work
+  (no WAN forward). Nothing enforces that assumption, and nothing will complain if `http` is still
+  there the day the port-forward appears.

decisions/2026-10-04-plex-vip-hostname-and-lan-networks.md

@@ -0,0 +1,151 @@
+---
+date: 2026-10-04
+branch: feat/plex-vip-hostname-and-lan-networks
+title: Plex — one advertised hostname for local and remote, its own LE certificate, and a LAN definition so LAN clients stop being reported as remote
+---
+
+## What changed
+
+Two operator-reported symptoms, after the 2026-10-04 "server missing from the account" fix
+(decisions/2026-10-04-plex-advertise-plexdirect.md) had restored publishing:
+
+1. **LAN clients were shown as Remote.** Root cause, measured: `LanNetworksBandwidth` was **ABSENT**,
+   so Plex had no definition of a local network and classified *everything* as WAN — **468 requests
+   logged `Allowed Network (WAN)` and ZERO `(LAN)`**, including Seerr dialling from inside the
+   cluster. `allowedNetworks` (the `ALLOWED_NETWORKS` env) only waives **auth**; it does not make a
+   client local. Two different settings, and only one of them has an env hook.
+2. **The advertised URL was a single-purpose, fragile value.** The previous fix advertised
+   `https://<dashed-ip>.<serverhash>.plex.direct:32400`, which works, but hard-codes the VIP *and*
+   the server's certificate hash into a cluster-var — two values nothing could keep in sync.
+
+**Operator design adopted (2026-10-04): ONE hostname, `plex-vip.${BASE_DOMAIN}`, serving both
+audiences by split-horizon DNS** — PiHole answers it with `${PLEX_L4_VIP}` on the LAN, Cloudflare
+answers it publicly with the WAN address, NATed to the same VIP.
+
+- **`catalog/plex.yaml`:** `ADVERTISE_IP` → `https://plex-vip.${BASE_DOMAIN}:32400`. New
+  `lifecycle.postStart` hook (level-triggered, same idiom as pihole's and qbittorrent's) that
+  re-asserts on **every** start: `LanNetworksBandwidth`, and the three `customCertificate*` prefs
+  after converting the PEM to the PKCS#12 bundle Plex requires. New `PLEX_TLS_CRT`/`PLEX_TLS_KEY`
+  env from the `plex-vip-tls` Secret, both `optional: true`.
+- **`apps/wave-0/plex-cert.yaml` (new, hand-written):** a cert-manager `Certificate` for exactly
+  `plex-vip.${BASE_DOMAIN}` from ClusterIssuer `letsencrypt-prod`. Non-catalog because the schema has
+  no field for a Certificate. A single name, not the wildcard: the wildcard's Secret lives in
+  `gateway` and Secrets are not readable across namespaces, and re-requesting `*.${BASE_DOMAIN}`
+  would burn a duplicate-certificate slot for no benefit.
+- **`schemas/servicedefinition.schema.json` → 1.11.0:** adds `optional` to
+  `env.valueFrom.secretKeyRef` / `configMapKeyRef`. **This is load-bearing, not tidiness.** The
+  Certificate must be issued into the `plex` namespace, and that namespace is created by the
+  rendered `apps/wave-0/plex.yaml` itself — so it cannot be issued in an earlier tier and lands in
+  the *same* reconcile as the workload. With hard refs, Plex would sit in
+  `CreateContainerConfigError` until a certificate issued, and `apps-wave-0` is `wait: true` with
+  `apps-wave-1` dependsOn it, so a slow or failed DNS01 challenge would have **stalled the whole app
+  tier** — the hazard `catalog/plex.yaml` already documents for its GPU resource.
+- **`catalog/pihole.yaml`:** `address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}` added to
+  `FTLCONF_misc_dnsmasq_lines`. Required, not cosmetic: measured 2026-10-04,
+  `plex-vip.${BASE_DOMAIN}` resolved to **`192.168.2.20`** (the Gateway VIP) via the zone wildcard,
+  and a connect to `:32400` there returned **`http=000`**.
+- **`clusters/base/cluster-vars.yaml` + `clusters/prod/kustomization.yaml`:** `PLEX_ADVERTISE_URL`
+  **deleted** — the hostname makes it unnecessary, which removes the cert-hash fragility outright.
+  `PLEX_L4_VIP`'s note now records its second consumer (pihole).
+- **`catalog/render.py`:** `plex-cert` EXTRA_UNITS entry; **`apps/wave-0/kustomization.yaml`:**
+  `plex-cert.yaml` hand-added (NOT via `--emit-waves`, which would also onboard `keycloak` and
+  `vm-demo` — the pre-existing drift recorded on 2026-10-03).
+
+**A correction to my own earlier reasoning, recorded because it was wrong in the handoff log.** The
+previous entry asserted a bare-IP URL "can NEVER validate" and treated a SAN match as a
+*requirement*. The operator pointed out it is not: `httpsRequired=0` and `secureConnections` default
+to Preferred, so clients can fall back to cleartext. The measurements still stand (a validating
+client did fail, and real clients did send `certificate unknown`) — but the correct framing is that a
+matching certificate keeps the remote leg **encrypted**, not that it is required for Plex to
+function. That is why this branch obtains a real certificate for the advertised name rather than
+choosing the name to suit Plex's built-in one.
+
+**SENSITIVE PATHS — touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK until an
+approving review (or the operator's `/approve-override <reason>` posted as a PR *review* of type
+"Comment"). Deliberately not self-posted: CLAUDE.md §1.**
+
+## What I verified
+
+Live prod, 2026-10-04, read-only. Secrets per CLAUDE.md §6: prefs were read by name-scoped grep,
+the server token was used inside the pod and never printed, and no `Preferences.xml` was dumped.
+
+- **The LAN-classification root cause, directly:** `LanNetworksBandwidth=<ABSENT>`, and request
+  tagging of **468 × `Allowed Network (WAN)` vs 0 × `(LAN)`** — including `10.0.2.33` (Seerr), an
+  in-cluster pod. This is the measurement the whole hook rests on.
+- **All four prefs the hook sets EXIST on this build**, read from the server's own `/:/prefs`
+  (token-authenticated, output filtered to ids): `LanNetworksBandwidth`, `customCertificatePath`,
+  `customCertificateKey`, `customCertificateDomain`, all `type="text"`. I had earlier been unable to
+  confirm this (an unauthenticated `GET /:/prefs` returns **401**) and said so rather than assuming.
+- **There is NO env path for these settings** — the reason the hook exists. Grep of the image's
+  entire init surface: `setPref` is called for exactly six prefs (`MachineIdentifier`,
+  `ProcessedMachineIdentifier`, `PlexOnlineToken`, `customConnections`, `allowedNetworks`,
+  `TranscoderTempDirectory`) and the only `PLEX_*` env it reads are CLAIM/CLAIM_FILE/UID/GID/
+  UPDATE_CHANNEL and five INFO/HOME vars. No `PLEX_PREFERENCE*` of any kind; zero files mention
+  `LanNetworks`.
+- **The DNS gap was real:** `plex-vip.${BASE_DOMAIN}` resolved to `192.168.2.20` (Gateway VIP), and
+  `https://192.168.2.20:32400` returned `http=000` (nothing listening). PiHole's wildcard is why.
+- **`openssl` is present in the image** (`/usr/bin/openssl`, 3.0.13) — the hook's PEM→PKCS#12
+  conversion has a tool to run.
+- **The issuer exists and works:** ClusterIssuer `letsencrypt-prod` `READY=True` (28d), and the
+  existing `gateway/wildcard-home` Certificate is `Ready`/not-expired, which is what makes a DNS01
+  issuance for a new name in this zone a reasonable expectation.
+- **The renderer's substitution guard caught a real defect in my own work**, which is recorded
+  because it is the second time this class of bug has appeared here: my hook's explanatory comment
+  contained brace-wrapped names, and `assert_substitution_safe` refused to emit
+  (`'PX' is not a declared cluster-vars key — envsubst will replace it with the EMPTY STRING`).
+  Reworded; the rendered hook now contains exactly one brace-wrapped name, `BASE_DOMAIN`, verified
+  by parsing the emitted manifest.
+- **A `set -u` bug I introduced and then removed.** With `optional: true` the TLS env vars are
+  genuinely UNSET when no certificate has issued; under `set -u` the first `[ -n ... ]` would abort
+  the hook, and a postStart hook exiting non-zero makes kubelet KILL the container — the exact
+  failure `optional` exists to prevent. Verified by parsing the rendered manifest that no `set -u`
+  **command** remains (only two mentions in comments).
+- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
+  `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
+  **pass**; the rc 3 is the two known environmental skips (`gitea-settings` — no local
+  `SETTINGS_API_TOKEN`; `conformance` L2/L3 — `CLUSTER_AVAILABLE` unset). **Not green. Reported as
+  rc 3.** `l0` passing is the specific proof the schema edit admits `optional` and still validates
+  every catalog file; `render-golden` passing is the proof catalog and committed goldens agree.
+- **Only plex and pihole goldens moved.** No other service's manifest changed, and no service was
+  silently onboarded.
+
+## What I did NOT verify
+
+- **THE OUTCOME, ENTIRELY. Nothing in this branch has been observed working.** It is unreconciled:
+  no Certificate exists in prod, `plex-vip` still resolves to the Gateway VIP, `LanNetworksBandwidth`
+  is still absent, and the hook has NEVER RUN. Every claim above is about the inputs, not the result.
+- **Whether the certificate will actually ISSUE.** DNS01 for a new name in this zone is inferred from
+  the wildcard's success, not demonstrated. If the Cloudflare credential or solver is scoped in a way
+  that fails for this name, Plex will start (by design) with no TLS and the advertised `https://` URL
+  will be unservable — i.e. **this branch could leave the advertised URL broken in a new way**. First
+  reconcile must be watched.
+- **Whether `openssl pkcs12 -export` produces a bundle PLEX ACCEPTS.** OpenSSL 3 defaults to AES-256
+  + SHA-256, which some consumers reject. Untested against Plex. If it is rejected, Plex may fail to
+  load its TLS listener — the hook guards against a *missing* file, NOT against a present-but-
+  unreadable one. **This is the single largest unverified risk in the change.**
+- **Whether the certificate prefs take effect without a restart.** Plex may need to re-initialise its
+  listener after `customCertificatePath` changes. If so, the first reconcile applies the prefs and
+  the SECOND start serves the cert. Converges either way, but the first start may look wrong.
+- **Whether `LanNetworksBandwidth` actually flips the Remote badge.** It is the correct setting by
+  Plex's own documentation and the measured absence explains the symptom exactly, but I did not set it
+  live and observe a client reclassify.
+- **SNAT makes the LAN definition coarser than it looks.** With `externalTrafficPolicy: Cluster`, Plex
+  sees the ingress NODE's address for everything arriving via the VIP, so once `10.0.0.0/8` and
+  `192.168.0.0/16` are LAN, GENUINE REMOTE clients arriving that way will ALSO be classified local
+  and remote bandwidth limits will not bind on them. This is the metering consequence already
+  accepted on 2026-10-03, now made stronger rather than weaker. Not measured.
+- **The public Cloudflare record for `plex-vip` is off-repo and absent from this branch**, as is the
+  WAN tcp/32400 port-forward. Remote access cannot work until the operator creates both; nothing here
+  can verify either, and this repo has no off-LAN vantage point.
+- **The private key is exposed in the container's environment** (`/proc/1/environ`) because the schema
+  and renderer support no secret VOLUME — measured: `storage` accepts only
+  [class, export, mountPath, name, readOnly, size, subPath] with `additionalProperties: false`, and
+  no rendered app mounts a Secret. A volume would be the better shape; adding secret-volume support
+  is the clean follow-up and I did not do it.
+- **The PKCS#12 passphrase is a hardcoded literal** (`plexvip`) in the rendered manifest. It protects
+  nothing — Plex needs it in cleartext in `customCertificateKey` and it sits beside the bundle on the
+  same PVC — but it is worth knowing it is not a secret and was never treated as one.
+- **Four places must agree and nothing enforces it:** `ADVERTISE_IP`, the hook's
+  `customCertificateDomain`, `plex-cert.yaml`'s `dnsNames`, and PiHole's answer. No gate stage or
+  conformance test checks that they match, or that PiHole's `plex-vip` answer still equals
+  `${PLEX_L4_VIP}`.

decisions/2026-10-05-oauth2-sidecar-scrape-verified.md

@@ -0,0 +1,90 @@
+---
+date: 2026-10-05
+branch: docs/oauth2-sidecar-scrape-verified
+title: The gate-sidecar scrape is verified working on prod — and four places in the tree still said it wasn't
+---
+
+## What changed
+
+Documentation only. No manifest, no rendered output, no behaviour.
+
+PR #49 merged as `21c88b0` and the fix works, but four places in the tree still asserted the
+*pre-fix* state as current — in the present tense, in capitals, in the files a reader would consult
+first. This is the stale-documentation failure mode this repo has been burned by twice already
+(`git push origin main` in the runbooks, "append to DECISIONS-LOG.md" in the old handoffs), so:
+
+- **`platform/oauth2-proxy/podmonitor.yaml`** — the banner read *"READ THIS BEFORE BELIEVING THE
+  TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY"* and *"SO THE TARGETS BELOW WILL
+  READ `up == 0` UNTIL AN INGRESS ALLOWANCE LANDS"*. Both now false, and the first one actively
+  instructs the reader to disbelieve a correct `up == 1`. Replaced with a `STATUS: THE SCRAPE WORKS`
+  block carrying the post-fix numbers; the diagnosis is kept below it under an explicit `HISTORY`
+  heading, because it is why the fix is shaped the way it is. Added the drop fingerprint to
+  recognise if it regresses — `lastScrapeDuration` pinned at *exactly* the `scrapeTimeout` with
+  `context deadline exceeded`, versus 1–6 ms when healthy.
+- **`catalog/render.py:159`** — *"THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED … Do not
+  read this flag as 'metrics are working'."* Rewritten to enumerate the three pieces a working
+  scrape needs and which PR landed each. The warning is **kept, deliberately**: the flag alone
+  really is insufficient, and "1 without 3" is exactly what caused the 63.7h outage. Only the
+  "not yet scraped" claim was false.
+- **`catalog/render.py:1365`** — the sidecar arg comment, same correction, one line.
+- **`platform/oauth2-proxy/README.md`** — its two observability subsections are *dated*
+  ("measured 2026-09-30", "Update 2026-10-01"), so they are honest records rather than lies. But a
+  reader hits the stale body long before any correction, so a `> SUPERSEDED` pointer now sits
+  directly under the heading, and an `### Update 2026-10-05` section at the end carries the
+  before/after table, the TSDB series counts and the alert outcome.
+
+Two substantive errors fixed while there, not just tense:
+
+1. The README's *"To close the gap for an app, three things are needed"* listed **a Service port
+   and a ServiceMonitor**. Wrong — a PodMonitor targets pod ports directly, which is the whole
+   reason no Service exposes 44180 and the renderer needed no Service change. Following that
+   paragraph would have sent someone to edit `catalog/render.py`'s Service rendering for no reason.
+2. `podmonitor.yaml` ended by deferring the fix to *"the catalog/renderer, which is in flight on
+   another branch"* — that branch does not exist and never did; the work happened in PR #49. It now
+   names `r_monitoring_ingress_policy` directly, and a new `CRAFTY` section explains why crafty is
+   the one namespace this file cannot help.
+
+## What I verified
+
+- **The fix itself, on prod 2026-10-05, read-only** (this is the evidence the docs now quote):
+  Flux applied `21c88b0` through the real path — `apps-wave-0` `lastAppliedRevision` at 21:37:01Z,
+  `apps-wave-1` at 21:37:46Z, in dependency order — and all five `<app>-allow-monitoring-scrape`
+  CNPs exist, each `ports=['44180']`, `from=[['monitoring']]`.
+- **All 6 targets `up == 1`**, `health=up`, `lastError=''`, `lastScrapeDuration` 0.001–0.006s —
+  against 10.0s and `context deadline exceeded` before.
+- **`oauth2_proxy_*` is in the TSDB for these namespaces for the first time:** 5 metric names,
+  148 series; `sum by (namespace) (oauth2_proxy_requests_total)` = qbittorrent 35072, radarr 23560,
+  sonarr 1284. (Queried with `count by (__name__)` plus a matcher, not label-values.)
+- **Alerts:** all 5 `TargetDown` and 5 of 6 `OAuth2ProxyGateUnobserved` resolved; one series
+  remains, `OAuth2ProxyGateUnobserved{namespace="crafty"}`, firing.
+- **The rendered output is byte-identical.** `catalog/render.py` changed only in comments, so this
+  branch touches no `apps/` file and no golden. Confirmed by `render-golden` and `l1` both passing
+  with no regeneration, and by `git status` showing nothing under `apps/`.
+- **This branch touches no sensitive path**, unlike PR #49: `platform/oauth2-proxy/**`,
+  `catalog/**` and `decisions/**` are none of them in `infra/gitea/approval-policy.json`. So it
+  should merge on a green gate with no human approval — the opposite of the PR it documents, and
+  the reason the two were kept apart.
+- `render.py --selftest`: PASS.
+- Toolbox gate: see the handoff's own rule — reported below rather than claimed here.
+
+## What I did NOT verify
+
+- **I did not re-measure the 63.7h figure or the ~22 emails/day rate**; both are carried forward
+  from the 2026-10-04 measurements, now quoted inside the docs as historical. The 15-day
+  `query_range` that established "one episode, not flapping" was run once, on 2026-10-04.
+- **`oauth2_proxy_requests_total` is still absent for jackett and pihole.** I assert in the README
+  that this is because the counter is not created until the gate serves its first request. That is
+  the documented behaviour of a Prometheus counter with no initialisation, and it is consistent
+  with the other three namespaces having traffic — but I did not force a request through jackett's
+  gate and watch the series appear. If it is still absent after real use, that is a finding.
+- **The auth-churn rule is still unwritten.** The README now says it is "writable rather than
+  decorative"; that is a claim about the metrics being present, not a rule anyone has authored or
+  unit-tested. The thing the 2026-09-30 incident actually needed does not exist yet.
+- **crafty remains unobserved.** I read its container spec on 2026-10-05 and confirmed no
+  `--metrics-address`; I did not attempt the fix, and nothing here changes its alert.
+- **I did not audit the whole tree for other stale "metrics are not scraped" claims** beyond the
+  four sites a targeted grep found (`not yet reachable`, `UNTIL AN INGRESS`, `up == 0`,
+  `not reachable by Prometheus` and neighbours). `runbooks/oauth2-proxy-auth-failures.md` is
+  described in the README as "the by-hand substitute meanwhile" and I left it alone — the by-hand
+  log procedure still works and is not wrong, but whether it should now be rewritten against the
+  new metrics is unexamined.

decisions/2026-10-05-plex-resolver-independent-advertise.md

@@ -0,0 +1,105 @@
+---
+date: 2026-10-05
+branch: fix/plex-resolver-independent-advertise
+title: Plex down LAN-wide — the advertised name existed only in PiHole, so clients that bypass it landed on the router's %D cert
+---
+
+## What changed
+
+**I took Plex down for the whole LAN.** Reported symptom: every internal client showed a certificate
+error with `%D` as the subject name and could not connect.
+
+**Root cause: the advertised connection depended on our own resolver.** `ADVERTISE_IP` was
+`http://plex-vip.${BASE_DOMAIN}:32400` — a name that exists ONLY in PiHole's split-horizon answers
+and in no public zone. A Plex client resolves the advertised host *itself*, with whatever resolver it
+happens to use, and on a real LAN that is frequently not PiHole: browsers ship DoH on by default,
+phones have "private DNS", guest/IoT devices take the router's DNS. Those clients could not resolve
+the name, the SOHO router answered the failed lookup with **itself**, and they all landed on the
+router's web UI.
+
+| Measured 2026-10-05 | Result |
+|---|---|
+| `192.168.2.1:443` (router, LAN side) | `C = ZZ, ST = Somewhere, O = %D342e7e6b, CN = %D` |
+| `136.51.115.252:443` (router, WAN side) | the same `%D` cert |
+| PiHole **pihole-0** | `plex -> 192.168.2.20`, `plex-vip -> 192.168.2.24` ✅ |
+| PiHole **pihole-1** | identical ✅ (both behind the `.23` VIP) |
+| Gateway `.20:443` cert | `CN = *.home.keithjackson.dev`, valid to 2026-12-12 ✅ |
+| `http://192.168.2.24:32400/identity` | `200` ✅ |
+| Plex's published connections | all port **32400** — nothing on 443 |
+
+**The DNS was right; the dependency on it was the bug.** PiHole answered correctly from both
+replicas the entire time. And because nothing Plex advertises listens on 443, the clients had plainly
+left the Plex path altogether and were talking to the router.
+
+**The fix: go back to `<dashed-ip>.<serverhash>.plex.direct`, which is RESOLVER-INDEPENDENT.** This
+is Plex's own mechanism and the reason it exists — `plex.direct` is a public zone that returns the
+private address embedded in the label, so *every* resolver (PiHole, the router, 1.1.1.1, a DoH
+browser) answers `192-168-2-24.<hash>.plex.direct` with `192.168.2.24`. It needs no split-horizon
+entry, no Cloudflare record and no certificate work, because the wildcard cert Plex already serves on
+32400 covers exactly that name.
+
+- **`catalog/plex.yaml`:** `ADVERTISE_IP` →
+  `https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400`, with the
+  resolver-independence requirement written at the top of the block as the first thing to check
+  before anyone changes this field again.
+- **PiHole's `plex-vip` answer is KEPT.** It is harmless, costs nothing, and is useful for reaching
+  the VIP by name from a LAN shell. It is simply not what Plex advertises.
+
+**Why not keep the nicer hostname.** It cannot be made correct for clients that bypass PiHole: a
+public record for it resolves to the WAN address, and with no NAT hairpin on this router a LAN client
+sent there fails too. The name is right only for PiHole users and wrong for everyone else, and there
+is no configuration of it that is right for both. `plex.direct` is right for both.
+
+**The honest cost, restated rather than buried:** the value embeds both the VIP and the server's
+certificate hash as a literal — envsubst cannot dash an IP and nothing in the tree can derive the
+hash, so nothing checks that they still agree. That fragility is exactly why the hostname was tried.
+It is accepted because an unresolvable name is a *worse* failure: silent, LAN-wide, and it points
+users at their router.
+
+## What I verified
+
+Live prod, 2026-10-05, read-only.
+
+- **Every row of the table above**, including the router's `%D` cert on both its LAN and WAN sides —
+  which is what identifies the router as what the clients were reaching.
+- **THE FIX ITSELF, with full TLS validation (no `curl -k`)**: the plex.direct name resolves to
+  `192.168.2.24` and `/identity` returns **200, TLS verified**. Re-checked today, not carried over
+  from yesterday's measurement.
+- **Both PiHole replicas are identical and correct**, with the same five `address=` lines live in
+  `/etc/pihole/dnsmasq.conf`, and both IPs present behind the `pihole-dns-tcp`/`-udp` endpoints. I
+  checked only pihole-0 at first and deliberately went back for pihole-1, because a split-brain pair
+  would have produced exactly this intermittent-looking symptom.
+- **The cluster was never the problem:** all eight LoadBalancer VIPs still assigned, L2
+  announcements intact (prod-3 holds `.20`/`.23`/`.24`), `wildcard-home` Certificate `Ready=True`,
+  and PR #51 had reconciled (`customConnections="http://plex-vip.home.keithjackson.dev:32400"` was
+  live, confirming the bad value was in effect).
+- **`LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16"` is still set**, so the postStart hook survived
+  the roll and keeps re-asserting.
+- **The Cilium rollout from #50 STILL has not happened** — agents are unchanged since
+  2026-10-04T07:11, so the Hubble export is inactive and `filePath: stdout` remains unproven.
+
+## What I did NOT verify
+
+- **THE OUTCOME. Unreconciled.** No client has been observed connecting after this change. The
+  argument is that the advertised name now resolves for every resolver and serves a matching
+  certificate — both measured — but I have not seen a Plex client complete a connection, and I
+  cannot see the user's clients at all.
+- **THE CLIENT-SIDE RESOLVER PATH IS INFERRED, NOT OBSERVED.** I could not query a public resolver
+  from here — my attempt returned NXDOMAIN even for a `plex.direct` name that certainly resolves, so
+  the probe was broken (egress policy or a missing tool) and I discarded it rather than report it.
+  The chain "client bypasses PiHole → NXDOMAIN → router answers with itself → `%D`" is the only
+  explanation consistent with all the measurements, but the NXDOMAIN-hijack step in particular is
+  deduced from the router serving `%D` on its LAN address, not from watching a lookup.
+- **Which clients bypass PiHole, or how many.** If in fact every client uses PiHole, then my
+  explanation is wrong and something else sent them to the router — in which case the next thing to
+  check is whether the clients were using `plex.home.keithjackson.dev` (port 443, the Gateway) rather
+  than an advertised connection at all.
+- **Whether plex.tv will re-publish promptly** after the value changes, and whether stale connection
+  lists are cached on clients. A client holding the old list may keep failing until it refreshes.
+- **Mixed content.** The previous `http://` value would also have been blocked by the browser for
+  anyone using `https://app.plex.tv`; that is fixed incidentally by returning to `https`, but I did
+  not test it.
+- **The Remote badge — the symptom that began this whole sequence — remains unconfirmed.**
+  `LanNetworksBandwidth` is set, but Plex still tags traffic `Allowed Network (WAN)`. Unchanged and
+  still unexplained.
+- **`ManualPortMappingMode="1"` with an absent port** is still there, untouched.

infra/ansible/inventory/group_vars/k3s_prod/vars.yml

@@ -306,19 +306,27 @@ node_baseline_dns_servers:
   # k3s_base_domain, below). It reaches the issuer through the NODE resolver, i.e. through
   # these PUBLIC upstreams — not through CoreDNS's split-horizon answer.
   #
-  # THIS USED TO BE BROKEN, AND THE 2026-09-04 SUBDOMAIN SPLIT FIXED IT. While prod was
+  # THE PUBLIC ANSWER IS NOW THE WAN, SO THE ISSUER IS PINNED (2026-10-03). Since
-  # authored to serve dev's hostnames, a prod node resolving keycloak.home.keithjackson.dev
+  # platform/gateway/gateway.yaml set an external-dns `target`, every public name is a CNAME
-  # publicly got DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN, so
+  # to wan.home.keithjackson.dev = the home WAN IP. The SOHO router does NOT hairpin: from the
-  # kubectl OIDC SSO on prod could not work at all. Now prod's issuer is
+  # LAN, WAN:443 answers with the router's own self-signed placeholder cert (measured from
-  # keycloak.prod.home.keithjackson.dev, and the public record for that name is published by
+  # k3s-prod-2/-3 2026-10-03: `curl` exit 60, ssl_verify_result=18). The apiserver would then
-  # PROD's own external-dns pointing at PROD's Gateway VIP (192.168.2.20). The public answer
+  # be unable to fetch its issuer's discovery/JWKS, so kubectl OIDC SSO would fail on the
-  # and the split-horizon answer therefore AGREE, and no /etc/hosts pin is needed.
+  # next k3s restart or Keycloak key rotation. node_baseline_hosts_pins (below) pins the
+  # issuer host to the Gateway VIP in /etc/hosts, which glibc consults before these
+  # resolvers. (History: before the 2026-09-04 subdomain split the public answer was DEV's
+  # VIP; after it, the public answer was prod's own VIP and no pin was needed.)
   #
-  # STILL TO VERIFY on prod bring-up (PROD-PROVISIONING.md §10): this holds only once prod's
+  # The cert-based admin kubeconfig never depends on any of this. It is the break-glass path.
-  # external-dns has actually published the record. Until then the issuer does not resolve
-  # from the nodes and kubectl SSO fails — the cert-based admin kubeconfig is the break-glass
-  # path, and it always works. Ordering, not a design gap.
 node_baseline_dns_search: []
+# /etc/hosts pins: node-process names that must reach the in-cluster Gateway directly rather
+# than via the public (WAN, non-hairpinned) answer. See "THE PUBLIC ANSWER IS NOW THE WAN" above.
+# [parity] the IP MUST equal GATEWAY_VIP in clusters/prod/kustomization.yaml (192.168.2.20),
+# and the name MUST follow k3s_oidc_issuer_url's host (roles/k3s_server/defaults/main.yml).
+node_baseline_hosts_pins:
+  - ip: "192.168.2.20"
+    names:
+      - "keycloak.{{ k3s_base_domain }}"
 
 # --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
 # PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/

infra/ansible/roles/cilium/templates/cilium-values.yaml.j2

@@ -45,17 +45,34 @@ hubble:
     enabled: true
   ui:
     enabled: true
+  # Static flow export -> cilium-agent stdout. Kept EQUAL to the Flux values
+  # (platform/cilium/helmrelease.yaml), which carry the full rationale: it records the PRE-SNAT
+  # client address for Plex's raw tcp/32400, which Plex itself cannot log because every
+  # LoadBalancer Service here is externalTrafficPolicy: Cluster and the VIP lease holder SNATs the
+  # client. Present here so a BOOTSTRAP render does not differ from what Flux reconciles -- a
+  # mismatch would show up as a permanent Helm diff and an avoidable agent rollout. Scoped by PORT,
+  # not namespace: the flow carrying the real client address does not match a namespace filter.
+  export:
+    static:
+      enabled: true
+      filePath: stdout
+      allowList:
+        - '{"destination_port":["32400"]}'
+        - '{"source_port":["32400"]}'
 
 # Ingress VIP announcement on the NAT LAN. IP pool / CiliumL2AnnouncementPolicy
 # (from the reserved 10.73.0.240-.250 range) is authored by WS-NET in Phase 2.
 l2announcements:
   enabled: true
 
-# k3s ships its own cgroup mount; point Cilium at it instead of auto-mounting.
+# Point Cilium at the host's real cgroup2 root (/sys/fs/cgroup, mounted by systemd) instead of
+# auto-mounting. NOT /run/cilium/cgroupv2: nothing mounts cgroup2 there, so the agent mounted its
+# OWN namespaced cgroup and socket-LB covered only itself — host processes (the apiserver) lost
+# Service/LB translation. [parity] platform/cilium/helmrelease.yaml explains (2026-10-03).
 cgroup:
   autoMount:
     enabled: false
-  hostRoot: /run/cilium/cgroupv2
+  hostRoot: /sys/fs/cgroup
 
 # Roll the cilium agent DaemonSet ONE node at a time. [parity] must match the
 # platform/cilium HelmRelease. The cilium 1.20.1 chart defaults this to 2, which on a

infra/ansible/roles/node_baseline/defaults/main.yml

@@ -242,3 +242,9 @@ node_baseline_dns_search:
 node_baseline_dns_options:
   - timeout:2
   - attempts:2
+# Static /etc/hosts entries (tags: dns, hosts), as a list of {ip, names}. EMPTY by default:
+# dev needs none. Prod pins its OIDC issuer host to the Gateway VIP because the public
+# answer for it is the non-hairpinned WAN IP (inventory/group_vars/k3s_prod/vars.yml).
+# Written as ONE marked block, so removing an entry here removes it from the node, and
+# an empty list removes the block entirely.
+node_baseline_hosts_pins: []

infra/ansible/roles/node_baseline/tasks/main.yml

@@ -1090,3 +1090,22 @@
       {% endif %}
   when: node_baseline_dns_mechanism == 'resolv-conf-file'
   tags: [dns]
+
+# /etc/hosts pins — tag `dns` (and `hosts` on its own). Some node processes must reach a
+# public hostname at the in-cluster Gateway, not at its public answer. On prod, the
+# kube-apiserver's OIDC issuer resolves publicly to the WAN IP, which the router does not
+# hairpin. glibc reads /etc/hosts before any nameserver (nsswitch `files dns`), so a pin wins
+# regardless of which resolver mechanism above is in effect. One marked block: entries
+# dropped from node_baseline_hosts_pins disappear from the node, and an empty list
+# removes the block.
+- name: Pin node-process hostnames in /etc/hosts
+  ansible.builtin.blockinfile:
+    path: /etc/hosts
+    marker: "# {mark} node_baseline hosts pins (Ansible, tags: hosts)"
+    block: |
+      {% for pin in node_baseline_hosts_pins %}
+      {{ pin.ip }}	{{ pin.names | join(' ') }}
+      {% endfor %}
+    state: "{{ 'present' if node_baseline_hosts_pins | length > 0 else 'absent' }}"
+    backup: true
+  tags: [dns, hosts]

platform/cilium/helmrelease.yaml

@@ -101,19 +101,92 @@ spec:
         enabled: true
       ui:
         enabled: true   # exposure gated behind oauth2-proxy forward-auth later (ADR-0014)
+      # --- STATIC FLOW EXPORT: persist the PRE-SNAT client address for Plex (2026-10-04) ---
+      # [parity] MUST equal Ansible: hubble.export.static.{enabled,filePath,allowList}
+      #
+      # WHY THIS EXISTS. Plex's own log cannot show who connected. Every LoadBalancer Service in
+      # this cluster is `externalTrafficPolicy: Cluster`, so the node holding the L2 lease for the
+      # VIP SNATs the client before forwarding to the backend pod. Measured 2026-10-04: Plex had
+      # logged 3,214 requests from `10.0.1.x` (the lease holder k3s-prod-3's own address, Cilium
+      # identity 1 = reserved:host) and ZERO from any 192.168.x address, ever. No X-Forwarded-For
+      # either -- it serves raw tcp/32400, not HTTP through the Gateway.
+      #
+      # THE ADDRESS IS NOT LOST, ONLY UNRECORDED. Cilium observes the flow at ingress, BEFORE its
+      # own translation, so Hubble already has the true source. Measured on the lease holder:
+      #     192.168.2.133 -> 10.0.0.135   (real LAN client -> the Plex pod)
+      #     10.0.1.78     -> 10.0.0.135   (the same traffic, post-SNAT -- what Plex sees)
+      # What was missing is RETENTION: the ring buffer was 4095/4095 flows at ~303 flows/s, i.e.
+      # about THIRTEEN SECONDS of history, with hubble metrics disabled and no exporter configured.
+      #
+      # WHY NOT FIX THE SNAT INSTEAD. `externalTrafficPolicy: Local` would preserve the address in
+      # Plex's own log, but cilium 1.20's L2 announcer is explicitly incompatible with it -- upstream
+      # docs for this exact version: "incompatible with the externalTrafficPolicy: Local on services
+      # as it may cause service IPs to be announced on nodes without pods causing traffic drops",
+      # and "L2 announcements isn't currently aware of this setting". Live proof that this is not
+      # theoretical: the VIP is announced by k3s-prod-3 while the pod runs on k3s-prod-1, so the
+      # flip would black-hole tcp/32400 immediately. Making it safe would mean pinning BOTH the pod
+      # and the announcement to one node and giving up Plex's ability to reschedule -- a real
+      # availability loss to buy a log field. (It is the same trap platform/crafty/service-l4.yaml
+      # documents from an 8-day Minecraft outage.) DSR was also rejected: routing-mode here is
+      # vxlan, not native/geneve, and DSR returns bypass the ingress node, which breaks the SOHO
+      # router's stateful NAT on the remote path.
+      #
+      # `filePath: stdout` RATHER THAN A FILE, deliberately. Grafana Alloy already runs as a
+      # DaemonSet tailing every pod's logs into Loki (platform/monitoring/alloy.yaml), so stdout
+      # reaches Loki with NO new hostPath mount, no file-rotation handling and no Alloy change.
+      # Upstream documents it: "To get Hubble flows directly exported to the logs instead of written
+      # to a rotated file, `stdout` can be defined as `hubble-export-file-path`." fileMaxSizeMb /
+      # fileMaxBackups are therefore left at chart defaults and do not apply.
+      #   Query:  {namespace="kube-system", app="cilium"} |= "32400"
+      #
+      # SCOPED BY PORT, NOT BY NAMESPACE -- and that is a measured decision, not a preference. A
+      # `--to-namespace plex` query returned ONLY the post-SNAT flows; the flow carrying the real
+      # client address did NOT match it, because at the point of observation the destination
+      # identity is not yet resolved to the plex pod. Filtering by namespace would therefore drop
+      # exactly the record this change exists to capture. Port it is.
+      #
+      # Both directions are listed so the return path is captured too. Volume is small (a few
+      # hundred to a few thousand lines a day); if it ever becomes noisy, narrow it with
+      # `fieldMask` rather than by removing a filter.
+      export:
+        static:
+          enabled: true
+          filePath: stdout
+          allowList:
+            - '{"destination_port":["32400"]}'
+            - '{"source_port":["32400"]}'
     # --- L2 announcement for the ingress VIP on the flat LAN (docs/09, OQ-1 resolved) ---
     # [parity] MUST equal Ansible: l2announcements.enabled: true. IP pool +
     # CiliumL2AnnouncementPolicy (reserved 10.73.0.240-.250) authored by WS-NET in Phase 2.
     l2announcements:
       enabled: true
-    # --- cgroup: k3s ships its own cgroup mount; point Cilium at it, don't auto-mount ---
+    # --- cgroup: point Cilium at the HOST's cgroup2 root, which systemd mounts at /sys/fs/cgroup ---
-    # [parity] MUST equal Ansible cgroup block. Not in the deliverable's headline list, but it IS
+    # [parity] MUST equal Ansible cgroup block (roles/cilium/templates/cilium-values.yaml.j2).
-    # part of the working CNI config on k3s: reconciling without it lets Cilium try to auto-mount
+    #
-    # cgroupv2 and break on adopt. Kept in parity for a safe Phase-2 adoption.
+    # FIXED 2026-10-03 — hostRoot WAS /run/cilium/cgroupv2, AND THAT SILENTLY BROKE SOCKET-LB FOR
+    # EVERY HOST PROCESS. With autoMount disabled, the chart only bind-mounts whatever already exists
+    # at hostRoot ("it's expected that users have mounted cgroup2 filesystem at the specified
+    # cgroup.hostRoot" — cilium 1.20.1 values.yaml). Nothing ever mounted cgroup2 at
+    # /run/cilium/cgroupv2 on these nodes (it is an empty DirectoryOrCreate hostPath), so the agent
+    # mounted one ITSELF from inside its own private cgroup namespace. That mount's root is the
+    # agent container's OWN leaf cgroup, so cil_sock4_connect & co. were attached there and covered
+    # only the agent. Every other host-netns process (the k3s apiserver in
+    # /system.slice/k3s.service, curl over SSH) was untranslated: Service ClusterIPs timed out from
+    # every node, and the node holding the Gateway's L2 lease could not reach the Gateway VIP at all
+    # ("no route to host"). That is why k3s-prod-1's apiserver logged `oidc: authenticator not
+    # initialized` from 2026-09-12 on — kubectl SSO through the kube-vip VIP failed outright.
+    # Measured on k3s-prod-1 before this change: TCP connect to 192.168.2.20:443, 10.43.225.79:443
+    # and 10.43.59.141:8080 SUCCEEDED from inside the agent container (the covered cgroup) and
+    # FAILED from the host, same node, same netns. Full write-up:
+    # decisions/2026-10-03-cilium-cgroup-hostroot.md.
+    #
+    # /sys/fs/cgroup IS a cgroup2 mount in the host's root cgroup namespace on every node (verified
+    # with findmnt on all three), so bind-mounting it gives the agent the real host root. autoMount
+    # stays off because the mount it would create already exists.
     cgroup:
       autoMount:
         enabled: false
-      hostRoot: /run/cilium/cgroupv2
+      hostRoot: /sys/fs/cgroup
     # --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident ---
     # [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1.
     # The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate.

platform/cilium/values.yaml

@@ -38,14 +38,28 @@ hubble:
     enabled: true
   ui:
     enabled: true
+  # [parity] static flow export -> cilium-agent stdout -> Alloy -> Loki. Records the PRE-SNAT
+  # client address for Plex's raw tcp/32400, which Plex's own log cannot show (every LB Service here
+  # is externalTrafficPolicy: Cluster, so the VIP lease holder SNATs the client). Scoped BY PORT,
+  # not namespace: the flow carrying the real client IP does not match a namespace filter. Full
+  # rationale, measurements and the rejected alternatives are in helmrelease.yaml.
+  export:
+    static:
+      enabled: true
+      filePath: stdout
+      allowList:
+        - '{"destination_port":["32400"]}'
+        - '{"source_port":["32400"]}'
 # [parity] ingress VIP announcement on the flat LAN (docs/09, OQ-1)
 l2announcements:
   enabled: true
-# [parity] k3s provides the cgroupv2 mount; do not auto-mount
+# [parity] the host's cgroup2 root is /sys/fs/cgroup (systemd mounts it); do not auto-mount.
+# NOT /run/cilium/cgroupv2: nothing mounts cgroup2 there, so socket-LB attached to the agent's own
+# cgroup and host processes lost Service/LB translation (2026-10-03 — see helmrelease.yaml).
 cgroup:
   autoMount:
     enabled: false
-  hostRoot: /run/cilium/cgroupv2
+  hostRoot: /sys/fs/cgroup
 # [parity] roll the agent DaemonSet ONE node at a time. Overrides the cilium 1.20.1 chart
 # default of 2, which on 3 nodes lets a single bad render break 2/3 agents at once — as the
 # 2026-09-01 incident did (k8sServiceHost: "" → the `config` init container fell back to the

platform/externaldns/README.md

@@ -20,6 +20,30 @@ A wildcard `*.keithjackson.dev` matches only one label deep, so `<svc>.home.keit
 its own record. **`internal` services need NO public record** — PiHole resolves them to the internal
 Gateway VIP (split-horizon). ExternalDNS manages **`public`** services' records only.
 
+## Public target + in-cluster DDNS (2026-10-03)
+
+Gateway-published records are **CNAMEs to `wan.home.keithjackson.dev`**, not the Gateway VIP:
+
+```
+plex.home.keithjackson.dev  CNAME  wan.home.keithjackson.dev   <- external-dns (Gateway `target` annotation)
+wan.home.keithjackson.dev   A      <current WAN IP>            <- cloudflare-ddns (ddns.yaml), every 5m
+```
+
+- The `target` annotation is on the **shared Gateway** (`platform/gateway/gateway.yaml`), so it
+  applies to every route that publishes at all. `internal` routes (`controller: none`) still publish
+  nothing.
+- `ddns.yaml` runs `favonia/cloudflare-ddns` in this namespace with the same `cloudflare-api-token`.
+  It detects the WAN IP through Cloudflare's trace endpoint, so the pod **must not** carry
+  `egress-class: vpn`.
+- LAN and pods resolve `*.home.keithjackson.dev` to the Gateway VIP via PiHole and CoreDNS
+  (split-horizon) and never use the public path.
+- **Nodes resolve through public DNS** (`node_baseline_dns_servers`), so from a node, the apiserver's
+  OIDC issuer `keycloak.home.keithjackson.dev` resolves to the WAN IP. kubectl SSO therefore relies on
+  the router hairpinning (NAT reflection) WAN:443 back to the Gateway.
+
+Check: `dig +short @1.1.1.1 plex.home.keithjackson.dev` → `wan.home.keithjackson.dev.` then the WAN IP;
+`kubectl -n external-dns logs deploy/cloudflare-ddns`.
+
 ## Secrets
 
 Cloudflare API token referenced **by name** (`cloudflare-api-token`, shared with cert-manager),

platform/externaldns/ddns.yaml

@@ -0,0 +1,87 @@
+---
+# In-cluster dynamic DNS for the home WAN (2026-10-03).
+#
+# Keeps ONE record, `wan.${BASE_DOMAIN}` (A, IPv4 only, not proxied), pointed at the home WAN's
+# current public IP. Every public name external-dns publishes from the shared Gateway is a CNAME to
+# it (platform/gateway/gateway.yaml `external-dns.alpha.kubernetes.io/target`), so a WAN IP change
+# moves all of them through this one record, with no Git change and no external-dns involvement.
+#
+# OWNERSHIP SPLIT WITH EXTERNAL-DNS. This record carries no external-dns TXT registry entry, so
+# external-dns (policy: sync, owner `prod`) never treats it as its own and never deletes it.
+# Conversely this updater manages only the names in DOMAINS and never touches external-dns's records.
+#
+# IP DETECTION. Default `cloudflare.trace`: an HTTPS request to Cloudflare's trace endpoint, which
+# reports the address the request arrived from. Correct here only because this pod is NOT in the
+# `vpn` egress class — the CiliumEgressGatewayPolicy (platform/vpn-egress) selects pods by
+# `homelab.keithjackson.dev/egress-class: vpn` alone, cluster-wide. Labelling this pod `vpn` would
+# publish the VPN exit IP instead of the WAN IP.
+#
+# TOKEN. Reuses this namespace's `cloudflare-api-token` (same Zone:DNS:Edit token as external-dns
+# and cert-manager) — no new secret.
+#
+# TIER NOTE. This lives in the gating `platform` Flux Kustomization (wait: true) next to
+# external-dns. If it ever crashloops, `platform` goes not-Ready and apps stop reconciling (no
+# outage). Kept here rather than in its own non-gating Kustomization to avoid a clusters/** change.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: cloudflare-ddns
+  namespace: external-dns
+  labels:
+    app.kubernetes.io/name: cloudflare-ddns
+spec:
+  replicas: 1                 # exactly one writer for the record
+  revisionHistoryLimit: 3
+  strategy:
+    type: Recreate            # never two updaters at once
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: cloudflare-ddns
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: cloudflare-ddns
+    spec:
+      automountServiceAccountToken: false
+      securityContext:
+        runAsNonRoot: true
+        runAsUser: 1000
+        runAsGroup: 1000
+        seccompProfile:
+          type: RuntimeDefault
+      containers:
+        - name: cloudflare-ddns
+          image: favonia/cloudflare-ddns:1.17.1
+          imagePullPolicy: IfNotPresent
+          env:
+            - name: CLOUDFLARE_API_TOKEN
+              valueFrom:
+                secretKeyRef:
+                  name: cloudflare-api-token
+                  key: api-token
+            - name: DOMAINS
+              value: "wan.${BASE_DOMAIN}"
+            - name: IP6_PROVIDER
+              value: "none"         # IPv4 WAN only; leaves any AAAA alone
+            - name: PROXIED
+              value: "false"        # DNS-only: Plex/Gitea/etc. terminate TLS on the Gateway
+            - name: UPDATE_CRON
+              value: "@every 5m"
+            - name: DELETE_ON_STOP
+              value: "false"        # a pod restart must not blank the record every CNAME depends on
+            - name: RECORD_COMMENT
+              value: "managed by k3s cloudflare-ddns (platform/externaldns/ddns.yaml)"
+            - name: EMOJI
+              value: "false"
+          resources:
+            requests:
+              cpu: 5m
+              memory: 16Mi
+            limits:
+              memory: 64Mi
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop:
+                - ALL

platform/externaldns/helmrelease.yaml

@@ -81,38 +81,29 @@ spec:
     # what this instance can see or own; per-resource exclusion only ever REMOVES an object from
     # this instance's view, so it cannot make prod capable of deleting a record it does not own.
     # ---------------------------------------------------------------------------------------------
-    # THE PUBLIC-TARGET GAP — STILL OPEN, AND NOT FIXABLE HERE ON v0.21.0 (operator decision)
+    # THE PUBLIC TARGET — DECIDED 2026-10-03 (operator): every Gateway-published name goes public
     #
-    # The five `exposure: public` services (plex, gitea, seerr, keycloak, nextcloud) publish records
+    # Until 2026-10-03 every gateway-sourced record answered with the Gateway VIP (192.168.2.20,
-    # that point at 192.168.2.20 — the Gateway VIP, an unroutable RFC1918 address — so they are not
+    # RFC1918), so nothing published here was reachable from the internet. Two facts constrained the
-    # actually reachable from the internet either. Their records are LEFT EXACTLY AS THEY ARE by
+    # fix, and both still hold:
-    # this change, for two independent reasons:
+    #   * `external-dns.alpha.kubernetes.io/target` is read in v0.21.0's gateway-httproute source ONLY
+    #     from the GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
+    #     gw.Annotations)`) and from a ListenerSet listener — NEVER from an HTTPRoute. A target on the
+    #     shared Gateway therefore moves EVERY gateway-sourced record at once.
+    #   * The home WAN IP is residential and can change, so a literal IP in Git would rot.
     #
-    #   1. THE VALUE IS NOT ESTABLISHED. The one public IP this repo knows is 136.51.115.252
+    # The operator's answers: (a) yes, ALL of them — the five `exposure: public` apps AND the admin
-    #      (clusters/prod/kustomization.yaml, where the zone's `*.keithjackson.dev` wildcard also
+    # UIs (auth, crafty, grafana, headlamp, the apex), which sit behind Keycloak / oauth2-proxy;
-    #      points). It belongs to the LEGACY non-k3s stack and currently serves a placeholder
+    # (b) the address is kept current IN-CLUSTER. Implementation:
-    #      certificate. Whether prod's public services belong behind it, behind a new forward, or
+    #   * platform/gateway/gateway.yaml sets `target: wan.${BASE_DOMAIN}` on the shared Gateway, so
-    #      are not meant to be public at all is an operator decision. Breaking five services' DNS
+    #     each published name becomes a CNAME to that one hostname;
-    #      on a guess is worse than leaving a known-wrong-but-stable state.
+    #   * platform/externaldns/ddns.yaml (favonia/cloudflare-ddns) keeps `wan.${BASE_DOMAIN}` an A
-    #   2. THE MECHANISM CANNOT BE SCOPED TO THEM. `external-dns.alpha.kubernetes.io/target` is a
+    #     record at the current WAN IP. That record has no external-dns TXT registry entry, so this
-    #      real override, but in v0.21.0's gateway-httproute source it is read ONLY from the
+    #     instance never owns or garbage-collects it.
-    #      GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
+    # Internal services are unchanged: their routes carry `controller: none` (above), which is
-    #      gw.Annotations)`) and from a ListenerSet listener — NEVER from the HTTPRoute. So there is
+    # checked BEFORE the target is read, so they still publish nothing. The `service` source (the L4
-    #      no per-service target override while every public service shares the `shared` Gateway.
+    # LoadBalancers crafty-mc, unifi) does not read the Gateway's annotation and still publishes its
-    #      Annotating the Gateway would move EVERY gateway-sourced record, which after this change
+    # LB VIP. Whether those should go public is a separate, still-open question.
-    #      still includes auth, crafty, grafana, headlamp and the apex — i.e. it would publish a
-    #      routable address for admin UIs. That is strictly worse than the status quo.
-    #
-    # SO THE DECISION THE OPERATOR OWNS IS TWO-PART, and no knob is added here until the first half
-    # is answered, because a knob that cannot take effect invites someone to set it and expect one:
-    #   (a) WHAT address should the five public names answer with — 136.51.115.252 as-is, a new
-    #       port-forward/WAN address, or nothing (drop `exposure: public` to `internal` instead)?
-    #   (b) HOW should they be separated from the non-public names that share the Gateway? Options,
-    #       roughly in ascending cost: a SECOND Gateway (or a ListenerSet) carrying the target
-    #       annotation, with only the public services' routes attached; or publishing the five names
-    #       from the `crd` source (DNSEndpoint objects the renderer emits for `public` services),
-    #       which restores per-service targeting without a second Gateway.
-    # Only (b) is a manifest change; (a) is a fact about the home WAN that has to be checked.
     # ---------------------------------------------------------------------------------------------
     # CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below.
     #

platform/externaldns/kustomization.yaml

@@ -8,3 +8,5 @@ resources:
   # SOPS-encrypted Cloudflare token (env CF_API_TOKEN) — SAME token as cert-manager, second copy in
   # the external-dns namespace. Flux decrypts at reconcile.
   - cloudflare-api-token.sops.yaml
+  # DDNS for `wan.${BASE_DOMAIN}` — the CNAME target every Gateway-published record points at.
+  - ddns.yaml

platform/gateway/gateway.yaml

@@ -7,6 +7,23 @@ kind: Gateway
 metadata:
   name: shared
   namespace: gateway
+  annotations:
+    # PUBLIC DNS TARGET (2026-10-03, operator decision). Without this, external-dns publishes every
+    # route on this Gateway with the Gateway's OWN address (${GATEWAY_VIP}, RFC1918), which is
+    # unreachable from the internet. With it, every published `<svc>.${BASE_DOMAIN}` (and the apex)
+    # becomes a CNAME to `wan.${BASE_DOMAIN}`, an A record that platform/externaldns/ddns.yaml keeps
+    # pointed at the home WAN's current public IP. The WAN forwards 80/443 to ${GATEWAY_VIP}.
+    #
+    # Scope is EVERY route on this Gateway that publishes at all — the operator chose to make the
+    # admin UIs (auth, grafana, headlamp, crafty, apex) publicly resolvable alongside the five
+    # `exposure: public` apps. `internal` services stay unpublished via their
+    # `external-dns.alpha.kubernetes.io/controller: none` route annotation, which external-dns checks
+    # BEFORE it reads this target. external-dns v0.21.0 reads `target` from the Gateway only, never
+    # from an HTTPRoute — see platform/externaldns/helmrelease.yaml.
+    #
+    # LAN and in-cluster clients are unaffected: PiHole and CoreDNS answer ${BASE_DOMAIN} with
+    # ${GATEWAY_VIP} directly (split-horizon), so they never take this public path.
+    external-dns.alpha.kubernetes.io/target: "wan.${BASE_DOMAIN}"
 spec:
   gatewayClassName: cilium
   listeners:

platform/oauth2-proxy/README.md

@@ -91,6 +91,10 @@ This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on
 
 ## Observability: what this component can and cannot see (measured 2026-09-30)
 
+> **SUPERSEDED — read "Update 2026-10-05" at the end of this section first.** The gate sidecars
+> **are** scraped now and the TSDB **does** hold `oauth2_proxy_*` for them. The two subsections
+> below are kept as the dated record of how this was diagnosed, not as current state.
+
 `servicemonitor.yaml` scrapes **this namespace's** deployment and nothing else. On prod that is
 two targets, `job="oauth2-proxy"`, both `up=1`. The complete set of oauth2-proxy-specific series
 in the TSDB is:
@@ -157,3 +161,55 @@ present `up` series. A discovered-but-failing target publishes `up` with value 0
 would have cleared `OAuth2ProxyGateUnobserved` the moment the PodMonitor landed — reporting the gates
 as observed while nothing was being scraped. The unit test pins that scenario; deleting the `== 1`
 fails it and nothing else.
+
+### Update 2026-10-05: the scrape works
+
+PR #49 (merged `21c88b0`) added the missing third piece. `catalog/render.py`'s
+`r_monitoring_ingress_policy` now also fires for the gate sidecar's metrics port, keyed on
+`_needs_oauth2_sidecar`, rendering `<app>-allow-monitoring-scrape` per app: ns `monitoring` → that
+app's `endpointSelector` → TCP 44180 only. Five objects, one each for jackett, pihole, qbittorrent,
+radarr, sonarr.
+
+Measured on prod 2026-10-05 after Flux applied it (`apps-wave-0` 21:37:01Z, `apps-wave-1`
+21:37:46Z):
+
+| | before | after |
+|---|---|---|
+| target health (6) | `down` | `up` |
+| `lastError` | `context deadline exceeded` | `''` |
+| `lastScrapeDuration` | **10.0s** — exactly the `scrapeTimeout` | **0.001–0.006s** |
+| `up` | `0` × 6 | `1` × 6 |
+
+    count by (__name__) ({__name__=~"oauth2_proxy_.*",
+                          namespace=~"jackett|radarr|sonarr|qbittorrent|pihole"})
+      oauth2_proxy_requests_in_flight              => 6
+      oauth2_proxy_requests_total                  => 16
+      oauth2_proxy_response_duration_seconds_bucket => 108
+      oauth2_proxy_response_duration_seconds_sum   => 9
+      oauth2_proxy_response_duration_seconds_count => 9
+
+    sum by (namespace) (oauth2_proxy_requests_total)
+      qbittorrent => 35072    radarr => 23560    sonarr => 1284
+
+`oauth2_proxy_requests_total` is absent for jackett and pihole simply because that counter is not
+created until the gate has served its first request.
+
+**So the auth-churn rule is now writable rather than decorative** — sustained `code="302"` as a
+share of `oauth2_proxy_requests_total`, held across a window longer than the session lifetime. That
+is the rule the 2026-09-30 incident above could not be caught by. It is still unwritten.
+
+**Alert outcome:** all 5 `TargetDown` resolved, and 5 of 6 `OAuth2ProxyGateUnobserved` with them.
+At `repeat_interval: 12h` with `group_by: [alertname, namespace]`, those 11 groups had been
+re-notifying at roughly **22 emails/day** since 2026-10-02; that drops to 2/day.
+
+**`crafty` still fires, and that is correct.** It is a hand-maintained Deployment under
+`platform/crafty`, outside the renderer, whose `oauth2-proxy` container has **no
+`--metrics-address` at all** — no listener, so no port to open and nothing for the PodMonitor's
+pod-label selector to match. Closing it needs the flag and a `metrics` containerPort on crafty's
+own manifest first, then a hand-written allowance beside it. Open follow-up work.
+
+**The paragraph above headed "To close the gap for an app" is superseded**: a *Service* port and a
+ServiceMonitor are **not** required. A PodMonitor targets pod ports directly, which is why no
+Service exposes 44180 and why the renderer needed no Service change. The three things actually
+needed are `--metrics-address` + named container port, a PodMonitor, and the monitoring-ingress
+allowance.

platform/oauth2-proxy/podmonitor.yaml

@@ -27,10 +27,28 @@
 # created by the apps-wave-0/1 tiers, which this tier does not and must not depend on.
 #
 # ============================================================================================
-# READ THIS BEFORE BELIEVING THE TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY
+# STATUS: THE SCRAPE WORKS. Verified on prod 2026-10-05 (PR #49 merged as 21c88b0).
 # ============================================================================================
-# This PodMonitor is NECESSARY AND NOT SUFFICIENT. The endpoint serves, and the path from ns
+# All 6 targets in pool podMonitor/oauth2-proxy/oauth2-proxy-sidecars/0 read `up == 1`,
-# monitoring is blocked. Both halves measured on prod 2026-10-01, read-only:
+# health=up, lastError='', lastScrapeDuration 0.001-0.006s. The TSDB holds oauth2_proxy_* for
+# these namespaces for the first time: 5 metric names / 148 series, with live counters
+# (oauth2_proxy_requests_total 35072 qbittorrent, 23560 radarr, 1284 sonarr). All 5 TargetDown
+# and 5 of 6 OAuth2ProxyGateUnobserved resolved; only namespace="crafty" still fires, and that
+# is correct — see CRAFTY below.
+#
+# The missing piece was an ingress allowance, and it now ships WITH EACH APP, rendered by
+# catalog/render.py's r_monitoring_ingress_policy as `<app>-allow-monitoring-scrape`:
+# ns monitoring -> the app's endpointSelector -> TCP 44180 only. See WHY NOT IN THIS DIRECTORY
+# below for why it is rendered per-app instead of shipped from this tier.
+#
+# Keep the history below. It is why the fix is shaped the way it is, and `up == 1` in
+# auth-visibility-alerts.yaml is load-bearing precisely because of it.
+#
+# --------------------------------------------------------------------------------------------
+# HISTORY: 2026-10-02 -> 2026-10-05, THE SCRAPE WAS DENIED BY NETWORKPOLICY FOR 63.7h
+# --------------------------------------------------------------------------------------------
+# This PodMonitor was NECESSARY AND NOT SUFFICIENT: the endpoint served, and the path from ns
+# monitoring was blocked. Both halves measured on prod 2026-10-01, read-only:
 #
 #   THE ENDPOINT IS HEALTHY — port-forward straight to the jackett pod's 44180:
 #     GET /metrics -> http=200 bytes=10208
@@ -54,19 +72,24 @@
 #   `jackett-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180 — the Gateway's Envoy
 #   and nothing else. Prometheus matches no clause and its packets are dropped, not refused.
 #
-# SO THE TARGETS BELOW WILL READ `up == 0` UNTIL AN INGRESS ALLOWANCE LANDS. That is a known,
+# THE TARGETS READ `up == 0` FROM 2026-10-02 05:09 UTC UNTIL THE ALLOWANCE LANDED — one
-# measured, documented state — NOT a regression introduced here, and NOT something to silence.
+# continuous 63.7h episode (a 15d query_range shows 1 episode per namespace, not flapping),
-# It is deliberately left visible: `OAuth2ProxyGateUnobserved` in auth-visibility-alerts.yaml
+# starting 11 minutes after this file was created, i.e. TargetDown's `for: 10m`. It was never
+# silenced, and that was the right call: `OAuth2ProxyGateUnobserved` in auth-visibility-alerts.yaml
 # requires `up == 1`, precisely so a discovered-but-failing target does not read as "observed"
 # (see that file's header — this exact trap is why the `== 1` is there).
 #
+# The signature to recognise if it ever returns: lastScrapeDuration pinned at EXACTLY the
+# scrapeTimeout (10.0s) with `context deadline exceeded`. That is a DROP, not a refusal — a
+# refused connection fails in milliseconds. Post-fix the same targets scrape in 1-6ms.
+#
 # THIS IS THE SAME BUG LONGHORN HAD, and the platform has a documented fix idiom for it:
 # platform/longhorn/networkpolicy-metrics.yaml, whose header records the identical fingerprint
 # measured on 2026-09-13 — ServiceMonitor in place, 3 targets DISCOVERED, `up` 0/0/0, TSDB empty,
 # `lastError: context deadline exceeded` (a drop, not a refusal). The fix there is a narrow
 # additive `networking.k8s.io/v1` NetworkPolicy: ns monitoring -> the pod label, that one port.
 #
-# WHY THAT POLICY IS NOT IN THIS DIRECTORY. It would have to be five objects, one per app
+# WHY NOT IN THIS DIRECTORY (the reasoning the fix followed). It would have to be five objects, one per app
 # namespace, and those namespaces are created by apps-wave-0/1. Measured Flux graph, 2026-10-01:
 #   oauth2-proxy  dependsOn=['platform-config']        <- this tier
 #   apps-wave-0   dependsOn=['platform-config']        <- SIBLING, no ordering between them
@@ -76,8 +99,19 @@
 # running cluster and break on a rebuild — the "works now, broken on restore" shape this repo
 # keeps paying for. Longhorn's precedent is safe only because longhorn-system is created by the
 # same tier that ships its policy. The allowance therefore belongs WITH THE APPS, rendered next to
-# the rest of each app's network policy — i.e. in the catalog/renderer, which is in flight on
+# the rest of each app's network policy — i.e. in the catalog/renderer. THAT IS WHERE IT LANDED:
-# another branch. Tracked in the handoff; it is one narrow rule per app, not a redesign.
+# `r_monitoring_ingress_policy`, keyed on `_needs_oauth2_sidecar`, one narrow rule per app.
+#
+# --------------------------------------------------------------------------------------------
+# CRAFTY: STILL UNOBSERVED, AND THIS FILE CANNOT HELP IT
+# --------------------------------------------------------------------------------------------
+# `OAuth2ProxyGateUnobserved{namespace="crafty"}` still fires after the fix, correctly. crafty is
+# a hand-maintained Deployment under platform/crafty, OUTSIDE the renderer, and its oauth2-proxy
+# container has NO `--metrics-address` at all (measured 2026-10-05) — so its metrics listener
+# never starts and there is no port to open. The selector below therefore does not match it, and
+# the per-app allowance is never rendered for it. Closing crafty's gap needs the flag plus a
+# `metrics` containerPort added to its own manifest FIRST; the allowance is the second half, and
+# it would have to be written by hand there rather than rendered. Open follow-up work.
 apiVersion: monitoring.coreos.com/v1
 kind: PodMonitor
 metadata:

runbooks/plex-remote-access.md

@@ -0,0 +1,148 @@
+# plex-remote-access — getting Plex reachable from outside the LAN, and proving it
+
+**Status: the in-cluster half is shipped; the last mile is TWO MANUAL STEPS and remote access does
+NOT work until both are done.** Neither can live in this repo.
+**Alert:** none, and none is possible from in-cluster signals — see "Why no alert" below. This is an
+**operator procedure**, not an incident page.
+**Files:** `apps/wave-0/plex-remote.yaml` (the tcp/32400 LoadBalancer + its CiliumNetworkPolicy),
+`clusters/base/cluster-vars.yaml` + `clusters/prod/kustomization.yaml` (`PLEX_L4_VIP`),
+`catalog/render.py` (the `EXTRA_UNITS` entry that places the unit), `catalog/plex.yaml` (Plex's own
+env — note `ADVERTISE_IP` is deliberately NOT set; see "Why not the Gateway URL").
+**Related:** `runbooks/plex-quicksync.md` and `runbooks/plex-claim.md` — the other two Plex
+procedures with a manual last mile.
+
+## What was actually broken (measured 2026-10-03)
+
+Plex was serving correctly and was reachable on the LAN. Two independent things stopped remote
+clients:
+
+1. **Plex advertised no remote connection at all.** In the live `Preferences.xml`:
+   `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while
+   `PublishServerOnPlexOnlineKey="1"`. The server was trying to publish itself to plex.tv and had
+   nothing reachable to publish. Plex's automatic path (UPnP/NAT-PMP) cannot work here: the process
+   lives in a pod on the Cilium overlay (`10.0.2.136`), it cannot speak UPnP to the SOHO router, and
+   a mapping to the pod IP would be meaningless if it could.
+2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` → connection refused.
+   `136.51.115.252:443` → a **self-signed router certificate** (`C=ZZ, CN=%D`), not the Gateway's
+   Let's Encrypt wildcard.
+
+The Service was `ClusterIP` only, so port 32400 had no path off-cluster by any route.
+
+**Caveat on that second measurement, because it is easy to over-read:** it was taken from *inside*
+the LAN, so strictly it proves there is no NAT hairpin (reflection). It does not prove outside-in
+failure. But a router certificate answering 443 means the router's own management UI is bound to the
+WAN port, and that *would* shadow a port-forward for genuine outside traffic too. **Verify from a
+real outside vantage point** (phone on cellular, or an external port checker) before concluding
+either way — this repo has no such vantage point and must not pretend otherwise.
+
+## Step 1 — forward WAN tcp/32400 on the router (operator, off-repo)
+
+    WAN tcp/32400  ->  192.168.2.24  (${PLEX_L4_VIP})  port 32400
+
+`192.168.2.24` is a pinned Cilium LB VIP, ARP-announced on the flat LAN by the existing
+`CiliumL2AnnouncementPolicy default-l2`. It is **pinned on purpose**: this address is typed into the
+router by hand, so it must never move on a reschedule.
+
+While in the router, **check what else is bound to the WAN interface.** If remote management is
+listening on 443 (see the certificate measurement above), move it or disable it — otherwise the
+Gateway's 80/443 forward is shadowed and the Plex *web UI* over the internet stays broken even
+though remote *streaming* works. These are two separate paths and they fail independently.
+
+Confirm the VIP is actually answering on the LAN first, so a failure later is unambiguous:
+
+    ./infra/devcontainer/dev.sh run bash -c \
+      'kubectl --context k3s-prod -n plex get svc plex-l4-tcp -o wide'
+
+`EXTERNAL-IP` must read `192.168.2.24`. If it is `<pending>`, LB-IPAM could not assign — the address
+is outside the pool or already taken; see `clusters/prod/kustomization.yaml` for the pool accounting.
+
+**`192.168.2.24` was never ARP-probed off-cluster.** Only the in-cluster pool accounting was checked
+(IPsTotal 11 / IPsUsed 4 / IPsAvailable 7 on 2026-10-03). If the VIP behaves oddly, probe it from a
+LAN host — `arping 192.168.2.24` — the way `.23` was probed before it was claimed.
+
+## Step 2 — tick "manually specify public port" in Plex (operator, UI only)
+
+**There is no GitOps path to this and no environment variable for it.** Measured in the running
+image on 2026-10-03: a grep of the entire init surface (`/etc/cont-init.d`, `/plex-common.sh`,
+`/usr/local/bin`) for `ManualPortMapping` and `PLEX_PREFERENCE` returned **NONE**. The image writes
+only six preferences, and this is not one of them. It is the same class of manual last mile as the
+`HardwareAcceleratedCodecs` toggle in `runbooks/plex-quicksync.md`.
+
+In the Plex web UI (`https://plex.home.keithjackson.dev`, on the LAN):
+
+**Settings → Remote Access**
+- Ensure **Enable Remote Access** is on. (`PublishServerOnPlexOnlineKey` was already `1`.)
+- Tick **Manually specify public port** and set it to **32400**.
+- **Apply**, then let the panel settle. It should report *"Fully accessible outside your network"*.
+
+Without this tick Plex waits for a NAT mapping it can never obtain and publishes no remote
+connection, so the port-forward in step 1 sits there unused.
+
+## How to verify
+
+Do these in order; each one isolates a different half.
+
+**1. The port is open from outside.** From a phone on cellular, or any off-LAN host:
+
+    curl -sk -o /dev/null -w '%{http_code}\n' https://136.51.115.252:32400/identity
+
+`200` means the forward and the policy are both good. A hang or refusal means step 1.
+
+**2. Plex itself believes it is reachable.** Settings → Remote Access must say *"Fully accessible
+outside your network"*. If it says *"not available outside your network"* while test 1 passes, step 2
+was not applied.
+
+**3. plex.tv is handing out a remote connection.** The decisive test, because it is what clients
+actually consume: sign in to [app.plex.tv](https://app.plex.tv) from an off-LAN network and confirm
+the server appears and plays. A client finds this server through plex.tv's connection list, never
+through DNS.
+
+**4. The policy is admitting the traffic.** If test 1 fails *after* the forward is confirmed, the
+CiliumNetworkPolicy is the next suspect:
+
+    ./infra/devcontainer/dev.sh run bash -c \
+      'kubectl --context k3s-prod -n plex get cnp plex-allow-remote-32400'
+
+See `runbooks/denied-flows.md` for reading actual drops.
+
+## Why no alert
+
+Every in-cluster signal is green whether or not remote access works. The Service has endpoints, the
+pod is `Running`, the policy is applied, and `plex-down` watches
+`kube_deployment_status_replicas_available` — none of which can see a router with a missing
+port-forward or an unticked checkbox. An alert would need an off-LAN prober, which this cluster does
+not have. **Test 3 above is the only honest check, and it is manual.**
+
+## Why not the Gateway URL (`ADVERTISE_IP`)
+
+The alternative was to leave Plex on `ClusterIP` and set
+`ADVERTISE_IP: https://plex.home.keithjackson.dev:443`, which writes Plex's `customConnections`
+preference and would have put the server on plex.tv's list via the existing L7 Gateway — no new WAN
+hole. That mechanism was verified to work on an already-claimed server (the `ADVERTISE_IP` block at
+line 126 of the image's `40-plex-first-run` sits **outside** the claim guard, and `/.firstRunComplete`
+lives on the overlay FS, not the `/config` PVC, so it re-runs every pod start), and the Gateway path
+itself was measured serving Plex correctly: `HTTP 200`, `claimed="1"`, with `401` on every protected
+endpoint.
+
+**The operator chose the native path anyway (2026-10-03), and the reasons are real:** Plex terminates
+its own TLS with its `*.plex.direct` certificate, and video streams bypass the Envoy L7 hop instead
+of being proxied through it. The cost is a new unfiltered WAN port — acceptable because Plex
+authenticates every caller itself. `ADVERTISE_IP` therefore stays unset in `catalog/plex.yaml`, and
+the deferral note there is still accurate.
+
+## Known wart: remote clients are classified as LAN
+
+`apps/wave-0/plex-remote.yaml` uses `externalTrafficPolicy: Cluster`, which SNATs the client to the
+ingress node's address (`192.168.2.1x`). That falls inside the `192.168.0.0/16` entry of
+`ALLOWED_NETWORKS`, so **Plex treats every remote stream as local.** Measured on the Gateway path,
+which SNATs the same way: Plex logged `[10.0.2.214:36040 (Allowed Network (WAN))]`.
+
+This is a **metering** effect, not a security one — authorization is unchanged, as the 401s above
+show — but Plex's remote-quality and bandwidth limits will not bind on these streams.
+
+`Local` would preserve the client IP and fix it. **Do not switch to it.** Cilium's L2 announcer
+elects one lease holder per Service and does *not* follow the backend pod, so the VIP can be
+announced by a node with no Plex pod, black-holing the port while everything reports healthy. That
+exact change cost this LAN an 8-day Minecraft outage in September 2026 — the full write-up is at the
+top of `platform/crafty/service-l4.yaml`. Plex is a single-replica `Recreate` Deployment, so the
+node mismatch is the normal case here, not a rare one.

schemas/renderer-contract.md

@@ -161,6 +161,41 @@ platform must render, and cites the requirement each mapping satisfies. Changing
 - **Requirements:** **R-STATE-01** (rebuild survival), **R-STATE-02** (Longhorn replication),
   **R-STATE-04** (bulk tier). (docs/04)
 
+### `availability` (`replicas`) — workload kind, per-replica volumes, spreading, budget
+
+**Schema 1.10.0.** Availability INTENT. It names no Kubernetes object, no topology key and no
+budget — exactly like `hardware.gpu`, and for the same reason: the mechanism below should be
+replaceable without a catalog edit.
+
+- **Absent, or `replicas: 1`** → a **Deployment**, standalone PVC(s), `strategy: Recreate` when there
+  is storage, no PodDisruptionBudget. Byte-identical to pre-1.10.0 output, so every existing app is
+  untouched.
+- **`replicas: N > 1`** → all of the following, together:
+  | Rendered | Why |
+  |---|---|
+  | **StatefulSet** instead of Deployment | A longhorn PVC is **ReadWriteOnce**. One RWO claim cannot attach on two nodes, so a multi-replica Deployment does not degrade — the second pod never schedules. |
+  | longhorn storage becomes **`volumeClaimTemplates`** | Gives each replica its own claim (`<vol>-<svc>-<ordinal>`, e.g. `config-pihole-0`). The template's `metadata.name` **must** equal the pod volume name the container mounts. The standalone PVC is then **not** emitted — it would be an orphan nothing mounts. |
+  | `bulk` storage stays a **standalone PVC** | A bulk entry is a static, pre-bound NFS PV over data that already exists outside the cluster and is shared read-only. Per-replica copies are neither possible nor wanted. |
+  | **required** `podAntiAffinity` on `kubernetes.io/hostname` | `preferred` would let every replica land on one node and deliver none of the intended node-failure tolerance, while every object still reported healthy. |
+  | `podManagementPolicy: Parallel` | `OrderedReady` serialises first-boot work across replicas; `apps-wave-0` has `timeout: 5m`. There is no ordering requirement between replicas to protect. |
+  | **PodDisruptionBudget**, `minAvailable: 1` | Stops a *voluntary* disruption (node drain, upgrade) taking the last replica. It does nothing about a node dying — that is what the replica count and anti-affinity are for. Never rendered at `replicas: 1`, where `minAvailable: 1` would block every drain of that node forever. |
+  | `observability.livenessWorkload.kind` defaults to **`StatefulSet`** | The `up` SLI reads kube-state-metrics by workload kind. A Deployment-shaped rule against a StatefulSet `absent()`s forever — an alert that can never fire. Deriving the default means bumping the replica count cannot leave a stale rule behind. |
+
+**Two costs, both deliberate, neither hidden:**
+
+1. **N replicas need N schedulable nodes.** On a 3-node cluster, losing two nodes leaves a replica
+   `Pending`; since `apps-wave-0` is `wait: true` and `apps-wave-1` `dependsOn` it, that **stalls the
+   app tier**. That coincides with having lost etcd quorum on a 3-node control plane, so nothing is
+   reconciling at that point anyway — the stall is a symptom, not a new failure mode.
+2. **Only declare `> 1` for a service whose on-disk state is reproducible from git.** A replica with
+   a fresh volume comes up serving whatever it can bootstrap itself and nothing else. For pihole that
+   prerequisite was done first — the admin password and the adlists were declared in
+   `decisions/2026-10-04-pihole-declare-state.md` — because otherwise a second replica would have
+   served DNS while blocking nothing, with a different password.
+
+**Requirements:** R-PLAT-02 (the workload), R-STATE-01/02 (per-replica replicated volumes),
+R-OBS-01 (the SLI must follow the rendered kind).
+
 ### `hardware` (`gpu`) — host device capability
 - **The field is INTENT, never MECHANISM.** `hardware.gpu` is an **enum of capability classes**
   (today: `intel-quicksync`), not a device path, not a hostPath, not a resource request. The

schemas/servicedefinition.schema.json

@@ -2,7 +2,7 @@
   "$schema": "https://json-schema.org/draft/2020-12/schema",
   "$id": "https://home.keithjackson.dev/schemas/servicedefinition.schema.json",
   "title": "ServiceDefinition",
-  "description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.9.0 (adds optional `auth.apiRoutes` -- a per-service list of anchored path regexes the forward-auth oauth2-proxy sidecar answers with 401 instead of a 302 to Keycloak, rendered as one repeatable `--api-route` per entry. The `auth` block stays `additionalProperties: false`; this is one named, documented key, not an opened door. Without it there was NO per-app override hook at all, so the only way to answer an app's XHRs with 401 was a renderer-wide constant -- and the path shape is a property of the app: jackett's is `/api/` (33 x 302 vs 2 x 401 measured 2026-09-30, 4 of 6 POSTs answered by the proxy and never reaching the app), radarr's is `/signalr/` (232 x 302 measured 2026-10-01, every one a POST from a left-open browser tab that is never re-prompted), and pihole serves no `/api/` at all. Each entry must be anchored at `^/` and continue with literal path characters, because a too-broad regex turns a plain page load into a 401 and the service looks permanently broken to a logged-out user. Enforced on every gate run by ci/l0.sh's schema stage over catalog/*.yaml, by catalog/render.py refusing to render an unanchored entry, and by conformance/tests/l0-static/oauth2-proxy-api-routes-per-app)). 1.8.0 (adds optional `hardware` - a host HARDWARE CAPABILITY declaration for the container kind, currently one member: `hardware.gpu: intel-quicksync`, meaning the workload needs the host iGPU render device for hardware video transcode. It is an ENUM OF INTENT, not a device path: the renderer satisfies it today with just two things -- a /dev/dri hostPath volume (`DirectoryOrCreate`) and a read-write /dev/dri volumeMount on the app container -- and a later move to the Intel GPU device plugin is then a render.py change with no catalog edit. It sets NO podSecurityContext.supplementalGroups: measured in-container on prod 2026-09-30, the pms-docker cont-init hook adds the app user to the device's group inside the container and `s6-setuidgid` then REPLACES the process group list from /etc/group, discarding anything Kubernetes supplied, so such an entry would be inert and misleading. Renders NOTHING when absent, so every existing app stays byte-identical. Emits no privileged securityContext (Kyverno restrict-host-namespaces would reject it and it is not needed), no nodeSelector (node placement is a separate concern with its own justification) and no resource requests). 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.md already said `internal` renders no public DNS record; nothing enforced it, and six internal services -- jackett, pihole, qbittorrent, radarr, semaphore, sonarr -- were publishing `<name>.home.keithjackson.dev` into the PUBLIC Cloudflare zone answering with the RFC1918 Gateway VIP, which published the internal subnet and a full service inventory and made each name a DNS-rebind primitive. Enforced on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns). 1.6.0 (splits `observability.metrics` from WHERE metrics live: adds optional `metricsEndpoint` (port/name/path/interval) and optional `livenessWorkload` (kind/name). `metrics: true` previously rendered a ServiceMonitor hardcoded to a port named `metrics` that no rendered Service ever defined, so all twelve app monitors discovered ZERO targets and every `up{job=...}` SLI was structurally inert on prod; a monitor is now rendered only from a declared `metricsEndpoint`, which also renders the Service port and the monitoring-namespace ingress allow, and the `up` SLI is expressed against kube-state-metrics -- a signal that exists for every workload without a per-app exporter. `livenessWorkload` names the object whose readiness IS the service, for the case where the running workload is not the one this SD renders). 1.5.0 (adds optional `inClusterClients` — additional in-cluster clients, named by NAMESPACE, admitted to a declared port of this service DIRECTLY, bypassing the shared Gateway. `ingressRoutes` admits only the Gateway proxy identity and `dependencies` only links catalog services, so a PLATFORM controller dialling a Service ClusterIP had no authorable form at all; that gap is what deadlocked proLine truncated
+  "description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.11.0 (adds optional `optional` to env.valueFrom.secretKeyRef and env.valueFrom.configMapKeyRef -- the k8s EnvVarSource field, emitted verbatim. Without it a secretKeyRef was ALWAYS hard: a missing Secret meant the pod would not start at all. That is the right default and stays the default, but it cannot express a value the workload is designed to run without. Added for plex, whose TLS material is issued by cert-manager INTO ITS OWN NAMESPACE and therefore cannot exist before the app wave that creates that namespace -- the Certificate and the Deployment land in the same reconcile, so on a cold bootstrap the Secret is briefly absent while a DNS01 challenge completes. Hard refs there would have made the media server unstartable until a certificate issued, and `apps-wave-0` is `wait: true`, so a failed or slow issuance would have STALLED THE WHOLE APP TIER -- the same hazard catalog/plex.yaml already documents for its GPU resource. With `optional: true` Plex starts regardless, its postStart hook logs that the certificate is not installed, and the next start after issuance converges. Renders nothing when absent, so every existing app stays byte-identical). 1.10.0 (adds optional `availability` for the container kind, one member: `availability.replicas`. It is AVAILABILITY INTENT, not a scheduling primitive -- it names no Kubernetes object, no topology key and no budget. 1 (the default) renders a Deployment exactly as before, so every pre-existing app stays byte-identical; >1 means the service must survive the loss of a single node, and the renderer satisfies that with a StatefulSet whose longhorn volumes become per-replica volumeClaimTemplates (a longhorn PVC is ReadWriteOnce and cannot attach on two nodes, so a multi-replica Deployment is UNSCHEDULABLE, not merely suboptimal), REQUIRED podAntiAffinity on kubernetes.io/hostname (preferred would let every replica land on one node and deliver none of the intended tolerance while reporting healthy), podManagementPolicy Parallel, a PodDisruptionBudget of minAvailable 1, and an `up` SLI that follows the rendered kind (a Deployment-shaped rule against a StatefulSet absent()s forever and can never fire). Added for pihole, where LAN DNS -- a critical network service -- was one pod on one node: measured 2026-10-04, losing the node running it meant roughly 6-10 minutes with no LAN resolution (tolerationSeconds 300 plus node-NotReady detection plus a Longhorn RWO force-detach). Deliberately NOT usable until a service's state is reproducible from git, because a replica with a fresh volume otherwise comes up serving but unconfigured -- for pihole that prerequisite was done first (decisions/2026-10-04-pihole-declare-state.md). Capped at 5: a count above the node count can never schedule under the anti-affinity this implies. Mechanism documented in schemas/renderer-contract.md `### availability`). 1.9.0 (adds optional `auth.apiRoutes` -- a per-service list of anchored path regexes the forward-auth oauth2-proxy sidecar answers with 401 instead of a 302 to Keycloak, rendered as one repeatable `--api-route` per entry. The `auth` block stays `additionalProperties: false`; this is one named, documented key, not an opened door. Without it there was NO per-app override hook at all, so the only way to answer an app's XHRs with 401 was a renderer-wide constant -- and the path shape is a property of the app: jackett's is `/api/` (33 x 302 vs 2 x 401 measured 2026-09-30, 4 of 6 POSTs answered by the proxy and never reaching the app), radarr's is `/signalr/` (232 x 302 measured 2026-10-01, every one a POST from a left-open browser tab that is never re-prompted), and pihole serves no `/api/` at all. Each entry must be anchored at `^/` and continue with literal path characters, because a too-broad regex turns a plain page load into a 401 and the service looks permanently broken to a logged-out user. Enforced on every gate run by ci/l0.sh's schema stage over catalog/*.yaml, by catalog/render.py refusing to render an unanchored entry, and by conformance/tests/l0-static/oauth2-proxy-api-routes-per-app)). 1.8.0 (adds optional `hardware` - a host HARDWARE CAPABILITY declaration for the container kind, currently one member: `hardware.gpu: intel-quicksync`, meaning the workload needs the host iGPU render device for hardware video transcode. It is an ENUM OF INTENT, not a device path: the renderer satisfies it today with just two things -- a /dev/dri hostPath volume (`DirectoryOrCreate`) and a read-write /dev/dri volumeMount on the app container -- and a later move to the Intel GPU device pluLine truncated
   "type": "object",
   "additionalProperties": false,
   "required": ["name", "source"],
@@ -69,7 +69,8 @@
                 "required": ["name", "key"],
                 "properties": {
                   "name": { "type": "string", "minLength": 1 },
-                  "key": { "type": "string", "minLength": 1 }
+                  "key": { "type": "string", "minLength": 1 },
+                  "optional": { "type": "boolean", "description": "Tolerate the key being ABSENT: the env var is simply unset instead of the pod refusing to start (kubelet CreateContainerConfigError). Default false, which is the pre-1.11.0 behaviour. Use ONLY where the workload degrades gracefully without the value, and say so at the call site -- a silently-empty credential is usually worse than a pod that will not start." }
                 }
               },
               "configMapKeyRef": {
@@ -78,7 +79,8 @@
                 "required": ["name", "key"],
                 "properties": {
                   "name": { "type": "string", "minLength": 1 },
-                  "key": { "type": "string", "minLength": 1 }
+                  "key": { "type": "string", "minLength": 1 },
+                  "optional": { "type": "boolean", "description": "Tolerate the key being ABSENT: the env var is simply unset instead of the pod refusing to start (kubelet CreateContainerConfigError). Default false, which is the pre-1.11.0 behaviour. Use ONLY where the workload degrades gracefully without the value, and say so at the call site -- a silently-empty credential is usually worse than a pod that will not start." }
                 }
               }
             }
@@ -139,6 +141,21 @@
         "supplementalGroups": { "type": "array", "items": { "type": "integer" }, "description": "Additional GIDs added to the first process in each container." }
       }
     },
+    "availability": {
+      "type": "object",
+      "description": "Optional availability intent for the container kind: how many replicas of this workload must be running. INTENT ONLY -- it names no Kubernetes object and no scheduling primitive. How the platform satisfies it (workload kind, per-replica volumes, node spreading, disruption budget) is catalog/render.py's business, documented in schemas/renderer-contract.md `### availability`. Absent => replicas 1 and a Deployment, so every pre-existing app renders byte-identical. Ignored for kind=vm (a VM's availability is a different problem with different primitives).",
+      "additionalProperties": false,
+      "minProperties": 1,
+      "properties": {
+        "replicas": {
+          "type": "integer",
+          "description": "How many replicas must be running. 1 (default) renders a Deployment exactly as before. >1 declares that the service must survive the loss of a single node; the renderer then emits a StatefulSet with per-replica volumeClaimTemplates, required per-node anti-affinity, podManagementPolicy Parallel and a PodDisruptionBudget. Only declare >1 for a service whose on-disk state is reproducible from git -- a replica with a fresh volume comes up serving whatever it can bootstrap itself, and nothing else.",
+          "minimum": 1,
+          "maximum": 5,
+          "default": 1
+        }
+      }
+    },
     "hardware": {
       "type": "object",
       "description": "Optional host HARDWARE CAPABILITY the workload needs, for the container kind (docs/13). This field declares INTENT (`what device class this service needs`), never MECHANISM (`which host path to bind-mount`, `which extended resource to request`). That separation is the whole point of the enum: the platform currently satisfies `intel-quicksync` with a /dev/dri hostPath volume and a matching mount on the app container, but a later migration to the Intel GPU device plugin (`gpu.intel.com/i915` resource requests, no hostPath) must be a catalog/render.py change and NOTHING else - no ServiceDefinition may need editing, and no reviewer may have to re-derive per-service device wiring. Do NOT 'simplify' this into a raw device path, a hostPath, or a resource request; that would push a platform implementation detail into the operator-facing contract and make the migration a catalog-wide edit. Absent => no device wiring at all (existing apps render byte-identical). Ignored for kind=vm (a VM gets devices through KubeVirt's own domain.devices).",