docs(agents): one workspace per agent run — the shared checkout is off-limits

Agents have been working in the operator's checkout and clobbering each other. Nothing in the tree told them not to: worktrees appeared in CLAUDE.md only as §3's gate workarounds, and decisions/README.md described worktree isolation as something agents already do rather than as a rule. New CLAUDE.md §0 states the rule, with the commands and an explicit do-not list for the shared checkout. Placed before §1 so every existing §-reference in the repo keeps its meaning. Measured in the shared checkout on 2026-10-05, each row a distinct failure mode: 107 `checkout: moving` reflog entries; sitting on feat/pm-agent-issue-to-pr, not main; two other runs' untracked trees left in it; local main two commits behind origin/main; 42 registered worktrees of which 31 are on already-merged branches; 9 gate scripts carrying a `.claude/` exclusion because an in-tree worktree made find-based discovery count the same manifest up to 18 times. Corrects a claim carried in earlier briefs: `git worktree remove` is NOT blocked by the agent sandbox. Provoked on a throwaway worktree — rc 0. §3 reframed as the normal path and grown from two breakages to three: spec-immutability's skip is avoidable, not merely tolerated, with the spec repo mounted read-only in the same ENGINE shim plus -e SPEC_REPO=/home-network-k3s. CLAUDE.md is a sensitive path, so this PR needs a non-author review or an administrator /approve-override posted as a PR review. Not posted by me: §1 forbids an agent clearing that on its own PR. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Merge pull request 'feat(cilium): roll the agents when cilium-config changes (merged config was silently inert)' (#53 ) from feat/cilium-roll-out-on-config-change into main
2026-10-05 22:23:50 -05:00 · 2026-10-06 02:30:31 +00:00 · 2026-10-05 21:19:13 -05:00 · 2026-10-05 21:18:33 -05:00 · 2026-10-06 02:06:00 +00:00 · 2026-10-05 20:58:55 -05:00
33 changed files with 2241 additions and 50 deletions
@@ -4,6 +4,117 @@ Read this before doing anything durable. It is the delivery contract for AI codi
 sessions and agents. The newest entry in `decisions/` is the current PM handoff and
 takes precedence on *what* to work on; this file governs *how* it lands.

+**Start with §0.** It is first because it is the rule you can break before you have read
+anything else: the very first `git checkout -b` in the wrong directory has already done the
+damage.
+
+## 0. Work in your OWN workspace. Never in the shared checkout.
+
+Before you create a branch, edit a file, or run `git` anything: **make yourself a git
+worktree outside the repository root, and do all of your work there.** One workspace per
+agent run, no exceptions.
+
+    # from the shared checkout, ONCE, at the start of your run
+    git fetch origin main
+    git worktree add -b <type>/<slug> /path/outside/the/repo/wt-<slug> origin/main
+    cd /path/outside/the/repo/wt-<slug>        # everything after this happens HERE
+
+`/path/outside/the/repo` means outside this repository's tree — your session scratchpad,
+or a sibling like `/var/home/keith/source/wt-<slug>`. Branch from **`origin/main`**, never
+from `main` and never from `HEAD`; see "stale base" below.
+
+**The shared checkout is `/var/home/keith/source/home-network-k3s-impl`** — the operator's
+working copy, the one you are probably started in. In it, do **not**:
+
+- **`git checkout` / `git switch` / `git checkout -b`.** Its HEAD is shared state; moving
+  it moves the tree out from under whoever else is using it.
+- **edit, create or delete a file.** Including "just one doc", including a file you intend
+  to move later.
+- **`git add` / `git commit` / `git stash` / `git restore`.** `git add -A` there stages
+  whatever another agent left lying around; `git stash` *removes* their uncommitted work
+  into a place they will not look.
+- **`git worktree remove` anything you did not create.**
+
+Read-only commands there are fine, and the one write you do need is creating your own
+worktree.
+
+### Why — measured in the shared checkout, 2026-10-05
+
+| What was measured | Value |
+|---|---|
+| `checkout: moving` entries in its reflog | **107** — the tree has been yanked between branches 107 times |
+| the branch it was sitting on | `feat/pm-agent-issue-to-pr` — **not `main`** |
+| uncommitted trees left in it by other runs | **2** (`infra/agent/`, `platform/pm-agent/`), mtimes five minutes apart the same evening |
+| its local `main` vs `origin/main` | **two commits behind** (`4386d14` vs `22c94ef`) |
+| registered worktrees | **42**, of which **31** are on branches already merged to `main` |
+| gate scripts carrying an explicit `.claude/` exclusion | **9** |
+
+Each row is a distinct way the shared checkout damages work:
+
+- **Dirty tree, foreign files.** Two other runs' uncommitted directories were sitting in
+  it. An agent that edits there and runs `git add -A` ships someone else's half-finished
+  work inside its own PR, and the PR's author has no idea what those files are.
+- **Stale base.** Its `main` was two commits behind `origin/main`, so
+  `git checkout -b x main` there branches off a tree that is already not what CI will merge
+  into — the gate you run locally proves a base nobody else has. Worse, it was not even *on*
+  `main`: a bare `git checkout -b x` would have branched off `feat/pm-agent-issue-to-pr` and
+  silently inherited that work into your PR. `git fetch` first, and name `origin/main`
+  explicitly as the start point.
+- **Shared HEAD.** 107 branch switches in one tree is 107 chances to switch it while
+  another agent is mid-edit. The loser's changes are either carried onto the wrong branch
+  or lost to a checkout conflict — and the symptom surfaces in *their* session, not yours.
+
+### Put the workspace OUTSIDE the repo root
+
+A worktree *inside* the tree (`.claude/worktrees/agent-*/` — which is where Claude Code's
+own default puts it) is a full second copy of the repo that every tree-walking check then
+finds. Measured 2026-09-12: `find`-based discovery reported **the same manifest under test
+once per worktree, up to 18 times**, and the L0 check comparing that listing against the
+tree failed on paths that are not authored surface at all
+(`platform/monitoring/tests/run.sh:115-123`). Nine gate scripts now carry an explicit
+`.claude/` prune for this reason — which means the hazard is *worked around*, not fixed,
+and the next check that walks the tree will hit it again. Keep the copy out of the tree and
+none of that applies.
+
+### Clean up — and expect not to be able to
+
+31 of the 42 registered worktrees are on branches already merged to `main`. A registered
+worktree **pins its branch**: while it exists, that branch cannot be deleted or checked out
+anywhere else. So a left-behind workspace is a lock on a branch name, and the cost lands on
+whoever next needs it.
+
+Remove yours when you are done:
+
+    git worktree remove /path/outside/the/repo/wt-<slug>
+
+Measured 2026-10-05: that command **succeeds** under the agent sandbox (rc 0, on a throwaway
+worktree in the session scratchpad) — earlier sessions recorded it as blocked, and that
+record was wrong, or at least no longer true. Try it; it probably works. If it is refused,
+say so in your handoff and leave the worktree registered — do **not** `rm -rf` the directory,
+which leaves a registered worktree pointing at nothing and still holding the branch.
+
+If someone *else's* stale worktree is holding the branch you need, do not try to reclaim the
+branch name. Push a `rework/<pr-number>` branch to the PR's refspec instead.
+
+### If you are already standing in the shared checkout with edits
+
+Stop. Do not `git stash` (that moves other agents' work too). Create your worktree as
+above, redo the edits there, and leave the shared checkout's HEAD and working tree exactly
+as you found them.
+
+### Gating from a worktree
+
+Three known breakages, all with workarounds, all required every run: see §3, "Running the
+gate from a worktree".
+
+### If you are writing a brief for another agent
+
+The brief must name **its own explicit branch** and tell it to create **its own worktree**
+from `origin/main`. "Work on the current branch" and "continue in the checkout" are how
+branch sprawl and cross-contamination got here: an agent given no workspace uses the one it
+is started in, which is the shared one. Also carry §3's three gate workarounds into the
+brief — an agent that cannot gate will report rc 3 as green.
+
 ## 1. Never push to `main`. Ever.

 Every durable change reaches `main` exactly one way:
@@ -147,15 +258,21 @@ Exit codes are a contract — do not flatten them:
 | 3 | UNDETERMINED — a stage did not run. **Never report this as green.** |

 `spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out
-beside this one; that single skip is expected and tolerated. Any *other* skip is a
-real gap in your run.
+beside this one; that single skip is expected and tolerated — and from a worktree it is
+avoidable outright (§3's worktree list, item 3). `gitea-settings` is the other expected
+skip: it needs `SETTINGS_API_TOKEN`, which exists only in CI, so **a local run cannot
+return 0** — the honest best local result is rc 3 with `gitea-settings` as the only stage
+that did not run. Any *further* skip is a real gap in your run. Report the rc and name the
+skipped stages; never round rc 3 up.

 **Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`,
 so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts.

-### Running the gate from a git worktree
+### Running the gate from a worktree

-Two unrelated failures, both with known workarounds — put both in any agent brief:
+This is the **normal** path, not an edge case — §0 requires you to be in a worktree, so
+every gate run you do hits all of these. Three known breakages, all with
+workarounds, all of them yours to apply every run — put them in any agent brief:

 1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked
   and exists only in the main checkout. Do **not** copy it (it is a credential).
@@ -166,10 +283,17 @@ Two unrelated failures, both with known workarounds — put both in any agent br
   the `ENGINE` hook — a shim that intercepts `run` and re-execs
   `podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in
   `tracked_files()` is this bug, not a broken tree.
+3. **`spec-immutability` skips** — the skip called tolerated above is also *avoidable*. `dev.sh`
+   derives the sibling spec path from its own `REPO_ROOT`, which in a worktree is your
+   workspace, so `../home-network-k3s` does not resolve and it adds no mount. Mount the spec
+   repo read-only in the same `ENGINE` shim and pass `-e SPEC_REPO=/home-network-k3s`: with
+   both mounts the stage reports `pass`, not SKIPPED, and R-TEST-06 is actually covered
+   (measured — `decisions/2026-10-01-render-selftest-runs-in-gate.md`).

 ## 4. Branch naming

-Use `feat/`, `fix/`, `docs/`, `chore/`. Note that `.gitea/workflows/gate.yml`'s **push**
+Use `feat/`, `fix/`, `docs/`, `chore/`. **One branch, one workspace, one PR** — if you
+need a second branch, make a second worktree (§0); do not re-point an existing one. Note that `.gitea/workflows/gate.yml`'s **push**
 trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run.
 The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected.

@@ -11,6 +11,9 @@ specification** at [`../home-network-k3s`](../home-network-k3s).
 - **Done = a green, reproducible, requirement-tagged conformance test** — never "merged," never a
  manual demo (spec `docs/14`, `agents/definition-of-done.md`).
 - **No work on `main`.** Every change is a feature branch merged into `main` (spec `CONTRIBUTING.md`).
+- **No work in a shared checkout.** Every agent run gets its own git worktree, outside the repo
+  root, branched from `origin/main`. This checkout is shared mutable state — see
+  [`CLAUDE.md` §0](CLAUDE.md).
 - **No plaintext secrets** — SOPS-encrypted only (spec `docs/07`).
 - **Phase discipline** — only build what the active roadmap phase authorizes (spec `roadmap/phases.md`).

@@ -94,6 +97,20 @@ The running narrative — decisions, root causes, incidents, follow-ups — is

 ## Working on this repo

+**First, get out of this checkout.** It is the operator's working copy and several agents
+share it; concurrent runs here have overwritten each other's branches and left each other's
+uncommitted files behind. One workspace per change:
+
+```bash
+git fetch origin main
+git worktree add -b <type>/<slug> /path/outside/the/repo/wt-<slug> origin/main
+cd /path/outside/the/repo/wt-<slug>
+```
+
+Outside the repo root specifically — a worktree under `.claude/worktrees/` is a second full
+copy that tree-walking gate checks then count twice. The rule, the measurements behind it, and
+the three gate workarounds a worktree needs are in [`CLAUDE.md` §0](CLAUDE.md) and §3.
+
 Everything the gate needs is baked into the pinned dev container — **do not install the
 toolchain on the host**:

@@ -244,6 +244,27 @@ spec:
      - port: '4180'
        protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): jackett-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: jackett-allow-monitoring-scrape
+  namespace: jackett
+  labels:
+    app.kubernetes.io/name: jackett
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: jackett
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): jackett-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -73,7 +73,7 @@ spec:
        - name: FTLCONF_dns_rateLimit_count
          value: '0'
        - name: FTLCONF_misc_dnsmasq_lines
-          value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}
+          value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
        - name: FTLCONF_webserver_api_password
          valueFrom:
            secretKeyRef:
@@ -232,6 +232,27 @@ spec:
      - port: '4180'
        protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): pihole-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: pihole-allow-monitoring-scrape
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: pihole
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): pihole-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -137,6 +137,20 @@ spec:
            secretKeyRef:
              name: plex-claim
              key: PLEX_CLAIM
+        - name: ADVERTISE_IP
+          value: https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400
+        - name: PLEX_TLS_CRT
+          valueFrom:
+            secretKeyRef:
+              name: plex-vip-tls
+              key: tls.crt
+              optional: true
+        - name: PLEX_TLS_KEY
+          valueFrom:
+            secretKeyRef:
+              name: plex-vip-tls
+              key: tls.key
+              optional: true
        resources:
          limits:
            gpu.intel.com/i915: 1
@@ -146,6 +160,14 @@ spec:
        - name: media
          mountPath: /media
          readOnly: true
+        lifecycle:
+          postStart:
+            exec:
+              command:
+              - /bin/bash
+              - -c
+              - "# NOTE \u2014 LOAD-BEARING: SHELL VARIABLES HERE ARE WRITTEN BARE, NEVER BRACE-WRAPPED.\n# This whole script is rewritten by Flux postBuild envsubst BEFORE it ever runs, so a\n# brace-wrapped name is read as a cluster-vars knob: an undeclared one is a HARD\n# Kustomization failure under StrictPostBuildSubstitutions, and a declared one would be\n# replaced with its value. Shell vars are therefore `$PX`, `$TOKEN`, `$P12` \u2014 bare. Only\n# BASE_DOMAIN below is brace-wrapped, because it IS a knob and IS meant to be\n# substituted. Command substitution `$(cmd)` is left alone and needs no escaping. A shell\n# variable written with braces would be blanked or would fail the whole Kustomization \u2014\n# this repo has already lost a gitea OIDC bootstrap to exactly that mistake, and\n# catalog/render.py now refuses to emit it (which is how this very comment got caught).\n# DELIBERATELY NO `set -u`. The TLS env vars are declared `optional: true`, so when the\n# certificate has not issued yet they are genuinely UNSET, not empty \u2014 and under `set -u`\n# the first `[ -n \"$PLEX_TLS_CRT\" ]` would abort the script, which for a postStart hook\n# means kubelet KILLS THE CONTAINER. That is the precise failure the `optional` flag exists\n# to prevent, so turning it back on here would defeat the whole design. (The alternative,\n# a brace-wrapped `:-` default, would need envsubst escaping \u2014 the trap the note above\n# describes.) Every value below is explicitly guarded instead.\nPX=\"/config/Library/Application Support/Plex Media Server/Preferences.xml\"\nCERTDIR=/config/certs\nP12=$CERTDIR/plex-vip.p12\n# The PKCS#12 passphrase protects nothing and is deliberately not a secret: Plex needs it\n# in cleartext in customCertificateKey, and it and the bundle sit on the same PVC. It is a\n# container-format requirement, not an access control.\nP12PASS=plexvip\nLANNETS='10.0.0.0/8,192.168.0.0/16'\nCERTDOMAIN=plex-vip.${BASE_DOMAIN}\nAPI=http://127.0.0.1:32400\n\n# (1) Build the PKCS#12 bundle from the PEM the Secret supplies. Rebuilt on every start so\n# a cert-manager RENEWAL is picked up with no extra moving part.\nif [ -n \"$PLEX_TLS_CRT\" ] && [ -n \"$PLEX_TLS_KEY\" ]; then\n  mkdir -p $CERTDIR && chmod 700 $CERTDIR\n  printf '%s' \"$PLEX_TLS_CRT\" > $CERTDIR/tls.crt\n  printf '%s' \"$PLEX_TLS_KEY\" > $CERTDIR/tls.key\n  chmod 600 $CERTDIR/tls.key\n  if openssl pkcs12 -export -out $P12.tmp -inkey $CERTDIR/tls.key \\\n       -in $CERTDIR/tls.crt -passout pass:$P12PASS 2>/dev/null; then\n    mv -f $P12.tmp $P12\n    chown 1000:1000 $P12 2>/dev/null || true\n    chmod 600 $P12\n    echo \"[plex-settings] PKCS#12 bundle written\"\n  else\n    rm -f $P12.tmp\n    echo \"[plex-settings] WARN: pkcs12 export failed; keeping any previous bundle\" >&2\n  fi\n  rm -f $CERTDIR/tls.key\nelse\n  echo \"[plex-settings] WARN: TLS env empty; certificate not installed this start\" >&2\nfi\n\n# (2) Wait, bounded (~90s), for the local API. /identity needs no auth.\ni=0\nwhile [ \"$i\" -lt 45 ]; do\n  if [ \"$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 $API/identity 2>/dev/null)\" = \"200\" ]; then\n    break\n  fi\n  i=$(expr $i + 1); sleep 2\ndone\n\n# (3) The prefs API needs the server token. It already lives in Preferences.xml on this\n# PVC; it is never echoed here, and Plex redacts tokens in its own request log. An\n# UNCLAIMED server has none \u2014 a normal state on a fresh /config \u2014 so skip, do not fail.\nTOKEN=$(grep -o 'PlexOnlineToken=\"[^\"]*\"' \"$PX\" 2>/dev/null | head -1 | cut -d'\"' -f2)\nif [ -z \"$TOKEN\" ]; then\n  echo \"[plex-settings] not claimed yet \u2014 skipping prefs, will retry next start\" >&2\n  exit 0\nfi\n\n# Percent-encode what our values actually contain. Plex takes prefs as QUERY parameters on\n# a PUT, so a raw '/' or ',' would be read as structure rather than data.\nenc() {\n  printf '%s' \"$1\" | sed -e 's|%|%25|g' -e 's|/|%2F|g' -e 's|,|%2C|g' \\\n                         -e 's|:|%3A|g' -e 's| |%20|g'\n}\nsetpref() {\n  code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 -X PUT\
+                \ \\\n    -H \"X-Plex-Token: $TOKEN\" \"$API/:/prefs?$1=$(enc \"$2\")\" 2>/dev/null)\n  if [ \"$code\" = \"200\" ]; then\n    echo \"[plex-settings] set $1\"\n  else\n    echo \"[plex-settings] WARN: $1 -> HTTP $code\" >&2\n  fi\n}\n\n# (4) THE LAN DEFINITION \u2014 the reason this hook was written. Without it Plex has NO notion\n# of a local network and tags EVERY client remote. Measured 2026-10-04: 468 requests\n# logged \"Allowed Network (WAN)\" and ZERO \"(LAN)\", including Seerr from inside the\n# cluster, which is why the server showed as Remote to LAN clients. Note allowedNetworks\n# (the ALLOWED_NETWORKS env) only waives AUTH \u2014 it does NOT make a client local. Two\n# different settings; only one of them has an env hook.\nsetpref LanNetworksBandwidth \"$LANNETS\"\n\n# (5) THE CERTIFICATE, which is what makes the `https` in ADVERTISE_IP servable. Set only\n# if the bundle exists \u2014 pointing Plex at a missing file would break its TLS listener.\nif [ -s \"$P12\" ]; then\n  setpref customCertificatePath \"$P12\"\n  setpref customCertificateKey \"$P12PASS\"\n  setpref customCertificateDomain \"$CERTDOMAIN\"\nelse\n  echo \"[plex-settings] WARN: no bundle; leaving certificate prefs untouched\" >&2\nfi\nexit 0\n"
      volumes:
      - name: config
        persistentVolumeClaim:
@@ -319,6 +319,27 @@ spec:
      - port: '4180'
        protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): qbittorrent-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: qbittorrent-allow-monitoring-scrape
+  namespace: qbittorrent
+  labels:
+    app.kubernetes.io/name: qbittorrent
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: qbittorrent
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): qbittorrent-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -347,6 +347,27 @@ spec:
      - port: '4180'
        protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): radarr-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: radarr-allow-monitoring-scrape
+  namespace: radarr
+  labels:
+    app.kubernetes.io/name: radarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: radarr
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): radarr-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -345,6 +345,27 @@ spec:
      - port: '4180'
        protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: sonarr-allow-monitoring-scrape
+  namespace: sonarr
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: sonarr
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -134,8 +134,17 @@ env:
  # then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal
  # characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid"
  # while resolution for the zone does not work. Confirmed both ways against the live image.
+  # plex-vip -> ${PLEX_L4_VIP} (added 2026-10-04) is the LAN HALF of Plex's single advertised
+  # hostname. Plex publishes ONE URL to plex.tv for both local and remote clients
+  # (catalog/plex.yaml ADVERTISE_IP): on the LAN this answer points it at the tcp/32400
+  # LoadBalancer, while Cloudflare answers the same name publicly with the WAN address, NATed to
+  # the same VIP. WITHOUT THIS LINE the entry above answers it with ${GATEWAY_VIP} via the zone
+  # wildcard, where NOTHING listens on 32400 — measured 2026-10-04, plex-vip.${BASE_DOMAIN}
+  # resolved to the Gateway VIP and a connect to :32400 there returned http=000. That is a silent
+  # failure: the Plex web UI on plex.${BASE_DOMAIN} keeps working through the Gateway on 443 while
+  # every client that takes the ADVERTISED connection cannot reach the server at all.
  - name: FTLCONF_misc_dnsmasq_lines
-    value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}"
+    value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
  # THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
  # apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
  # carries the full rationale and the measurement that found it.
@@ -72,14 +72,227 @@ env:
      secretKeyRef:
        name: plex-claim
        key: PLEX_CLAIM
-  # ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
-  # which URL to advertise to plex.tv for REMOTE clients. The only candidate is
-  # https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
-  # Gateway VIP (conformance/tests/l2-behavioral/dns-internal-resolution), and both its public
-  # reachability and whether the TLS-terminating Cilium Gateway proxies Plex's 32400 API correctly for
-  # remote access are UNVERIFIED. Advertising a wrong/internal-only URL breaks remote access for ALL
-  # clients, so per "prefer safety over completeness" it is left unset. FOLLOW-UP: once remote
-  # reachability through the Gateway is confirmed, set ADVERTISE_IP: "https://plex.home.keithjackson.dev:443".
+  # ADVERTISE_IP ("Custom server access URLs") — the connection Plex publishes to plex.tv.
+  #
+  # THE VALUE MUST BE RESOLVABLE BY EVERY CLIENT, NOT JUST BY CLIENTS THAT USE OUR DNS. That single
+  # property is what this field keeps getting wrong, so it is the first thing to check before
+  # changing it. A Plex client resolves the advertised host ITSELF, with whatever resolver it
+  # happens to use — and on a real LAN that is frequently NOT PiHole: browsers ship DoH on by
+  # default, phones have "private DNS", guest and IoT devices often take the router's DNS, and some
+  # clients hard-code a public resolver.
+  #
+  # THAT IS NOT THEORETICAL — IT TOOK PLEX DOWN FOR THE WHOLE LAN ON 2026-10-05. This field briefly
+  # advertised `plex-vip.${BASE_DOMAIN}`, a name that exists ONLY in PiHole's split-horizon answers
+  # (catalog/pihole.yaml) and in no public zone. Clients that did not use PiHole could not resolve
+  # it; the SOHO router answered the failed lookup with ITSELF, and every one of them landed on the
+  # router's own web UI and reported a certificate error for `CN = %D`. Measured 2026-10-05: the
+  # router serves that self-signed cert (`C = ZZ, O = %D342e7e6b, CN = %D`) on BOTH `192.168.2.1:443`
+  # and the WAN address, while PiHole itself was answering perfectly — both replicas returned
+  # `plex-vip -> ${PLEX_L4_VIP}` and `plex -> ${GATEWAY_VIP}`. The DNS was right; the DEPENDENCY on
+  # it was the bug. Nothing Plex advertises listens on 443, which is how we know the clients had left
+  # the Plex path entirely and were talking to the router.
+  #
+  # SO: `<dashed-ip>.<serverhash>.plex.direct`, WHICH IS RESOLVER-INDEPENDENT. This is Plex's own
+  # mechanism and the reason it exists: `plex.direct` is a PUBLIC zone that returns the private
+  # address embedded in the label, so EVERY resolver on earth — PiHole, the router, 1.1.1.1, a DoH
+  # browser — answers `192-168-2-24.<hash>.plex.direct` with `192.168.2.24`. It needs no
+  # split-horizon entry, no Cloudflare record, and no certificate work, because the wildcard cert
+  # Plex already serves on 32400 covers exactly this name. Re-verified live 2026-10-05 with FULL
+  # validation (no `curl -k`): resolves to 192.168.2.24 and `/identity` returns **200, TLS verified**.
+  #
+  # WHY NOT THE NICER HOSTNAME, since a readable name in our own zone is obviously more pleasant:
+  # it cannot be made to work for clients that bypass PiHole. A public record for it would resolve
+  # to the WAN address, and with no NAT hairpin on this router a LAN client sent there fails too —
+  # so the hostname is correct ONLY for clients using PiHole, and wrong for the rest. There is no
+  # configuration of that name that is right for both. `plex.direct` is right for both.
+  # (The PiHole `plex-vip` answer is KEPT — it is harmless, costs nothing, and is genuinely useful
+  # for reaching the VIP by name from a LAN shell. It is simply not what Plex advertises.)
+  #
+  # THE TWO EMBEDDED VALUES, and the honest cost of this choice. The dashed IP must track
+  # ${PLEX_L4_VIP} and `<serverhash>` is this server's certificate identity — read it from the live
+  # listener with
+  #     openssl s_client -connect <vip>:32400 </dev/null | openssl x509 -noout -subject
+  # Nothing in this tree can derive either, and nothing checks that they still agree: envsubst
+  # cannot dash an IP, so this is a literal. That fragility is real and was the reason the hostname
+  # was tried in the first place — it is accepted here because an unresolvable name is a WORSE
+  # failure (silent, LAN-wide, and it points users at the router) than a stale literal, which at
+  # least fails loudly and only after someone changes the VIP.
+  #
+  # The image's /etc/cont-init.d/40-plex-first-run runs `setPref "customConnections"
+  # "${ADVERTISE_IP}"` (line 127, verified in the running image), OUTSIDE the
+  # PLEX_CLAIM/PlexOnlineToken guard, with its `/.firstRunComplete` marker on the container OVERLAY
+  # fs rather than the /config PVC — so this is re-applied on EVERY pod start and git, not a web
+  # form, owns the value.
+  - name: ADVERTISE_IP
+    value: https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400
+  # THE TLS MATERIAL FOR THE NAME ABOVE, as PEM text in env rather than a mounted volume.
+  # `plex-vip-tls` is issued by cert-manager from ClusterIssuer `letsencrypt-prod` for exactly
+  # `plex-vip.${BASE_DOMAIN}` (apps/wave-0/plex-cert.yaml, same wave => same reconcile as this
+  # workload). The postStart hook converts it to the PKCS#12 bundle Plex requires.
+  #
+  # WHY ENV AND NOT A VOLUME, which would be the conventional choice. MEASURED 2026-10-04:
+  # `schemas/servicedefinition.schema.json` `storage` accepts only
+  # [class, export, mountPath, name, readOnly, size, subPath] with `additionalProperties: false`,
+  # `catalog/render.py` emits NO secret volumes, and no rendered app mounts a Secret as a volume.
+  # Mounting one therefore needs a schema + renderer change; `env.valueFrom.secretKeyRef` is
+  # already in the schema and already used by PLEX_CLAIM above. PEM is text, so it survives an env
+  # var intact (a raw PKCS#12 would not — it is binary and carries NUL bytes).
+  #
+  # COST, STATED PLAINLY: the private key is readable from this container's environment
+  # (/proc/1/environ, `kubectl exec env`) by anything that can already exec into the pod. That is
+  # the same audience that can read the converted bundle under /config, so the marginal exposure is
+  # small — but a secret VOLUME would still be the better shape, and adding secret-volume support
+  # to the schema is the clean follow-up that removes this note.
+  #
+  # `optional: true` IS LOAD-BEARING, NOT DEFENSIVE CLUTTER (schema 1.11.0, added for this). The
+  # Certificate is issued INTO THIS NAMESPACE, and the namespace is created by the rendered
+  # apps/wave-0/plex.yaml itself — so the cert CANNOT be issued in an earlier tier, and the
+  # Certificate and this Deployment land in the SAME reconcile. On a cold bootstrap the Secret is
+  # therefore briefly absent while cert-manager completes a DNS01 challenge. With a HARD ref, Plex
+  # would sit in CreateContainerConfigError until a certificate issued — and because `apps-wave-0`
+  # is `wait: true` with `apps-wave-1` dependsOn it, a failed or slow issuance would STALL THE WHOLE
+  # APP TIER (the same hazard the `hardware` block below documents for the GPU resource). Optional
+  # instead means: no cert -> Plex still starts and serves, the postStart hook logs that the
+  # certificate was not installed, and the next start after issuance converges. Plex falls back to
+  # its own `*.plex.direct` certificate in the meantime, so the degraded state is "advertised URL
+  # not yet servable over TLS", not "media server down".
+  - name: PLEX_TLS_CRT
+    valueFrom:
+      secretKeyRef:
+        name: plex-vip-tls
+        key: tls.crt
+        optional: true
+  - name: PLEX_TLS_KEY
+    valueFrom:
+      secretKeyRef:
+        name: plex-vip-tls
+        key: tls.key
+        optional: true
+lifecycle:
+  # LEVEL-TRIGGERED PLEX SETTINGS, re-asserted on EVERY container start — the same pattern and the
+  # same reasoning as catalog/pihole.yaml's and catalog/qbittorrent.yaml's postStart: desired state
+  # lives in git and is re-asserted unconditionally rather than typed into a web form once. Typing
+  # settings into the web form is precisely what took this service down on 2026-10-04.
+  #
+  # IT EXISTS BECAUSE THE IMAGE HAS NO ENV FOR THESE SETTINGS. Measured 2026-10-04 across the whole
+  # init surface (/etc/cont-init.d, /etc/services.d, /plex-common.sh, /usr/local/bin): the image
+  # calls setPref for exactly SIX prefs — MachineIdentifier, ProcessedMachineIdentifier,
+  # PlexOnlineToken, customConnections, allowedNetworks, TranscoderTempDirectory — and supports NO
+  # generic PLEX_PREFERENCE* hook of any kind. The two things we need (a LAN definition and a custom
+  # certificate) are reachable ONLY through Plex's own API, so this hook is the only GitOps path.
+  #
+  # IT MUST NEVER FAIL. A postStart hook that exits non-zero makes kubelet KILL the container, so
+  # every step is bounded and the script ends with an unconditional `exit 0`. A missing token, an
+  # unissued certificate or an unreachable API degrades to "not applied this start" and is retried
+  # on the next one; it never crashloops Plex.
+  postStart:
+    exec:
+      command:
+        - /bin/bash
+        - -c
+        - |
+          # NOTE — LOAD-BEARING: SHELL VARIABLES HERE ARE WRITTEN BARE, NEVER BRACE-WRAPPED.
+          # This whole script is rewritten by Flux postBuild envsubst BEFORE it ever runs, so a
+          # brace-wrapped name is read as a cluster-vars knob: an undeclared one is a HARD
+          # Kustomization failure under StrictPostBuildSubstitutions, and a declared one would be
+          # replaced with its value. Shell vars are therefore `$PX`, `$TOKEN`, `$P12` — bare. Only
+          # BASE_DOMAIN below is brace-wrapped, because it IS a knob and IS meant to be
+          # substituted. Command substitution `$(cmd)` is left alone and needs no escaping. A shell
+          # variable written with braces would be blanked or would fail the whole Kustomization —
+          # this repo has already lost a gitea OIDC bootstrap to exactly that mistake, and
+          # catalog/render.py now refuses to emit it (which is how this very comment got caught).
+          # DELIBERATELY NO `set -u`. The TLS env vars are declared `optional: true`, so when the
+          # certificate has not issued yet they are genuinely UNSET, not empty — and under `set -u`
+          # the first `[ -n "$PLEX_TLS_CRT" ]` would abort the script, which for a postStart hook
+          # means kubelet KILLS THE CONTAINER. That is the precise failure the `optional` flag exists
+          # to prevent, so turning it back on here would defeat the whole design. (The alternative,
+          # a brace-wrapped `:-` default, would need envsubst escaping — the trap the note above
+          # describes.) Every value below is explicitly guarded instead.
+          PX="/config/Library/Application Support/Plex Media Server/Preferences.xml"
+          CERTDIR=/config/certs
+          P12=$CERTDIR/plex-vip.p12
+          # The PKCS#12 passphrase protects nothing and is deliberately not a secret: Plex needs it
+          # in cleartext in customCertificateKey, and it and the bundle sit on the same PVC. It is a
+          # container-format requirement, not an access control.
+          P12PASS=plexvip
+          LANNETS='10.0.0.0/8,192.168.0.0/16'
+          CERTDOMAIN=plex-vip.${BASE_DOMAIN}
+          API=http://127.0.0.1:32400
+
+          # (1) Build the PKCS#12 bundle from the PEM the Secret supplies. Rebuilt on every start so
+          # a cert-manager RENEWAL is picked up with no extra moving part.
+          if [ -n "$PLEX_TLS_CRT" ] && [ -n "$PLEX_TLS_KEY" ]; then
+            mkdir -p $CERTDIR && chmod 700 $CERTDIR
+            printf '%s' "$PLEX_TLS_CRT" > $CERTDIR/tls.crt
+            printf '%s' "$PLEX_TLS_KEY" > $CERTDIR/tls.key
+            chmod 600 $CERTDIR/tls.key
+            if openssl pkcs12 -export -out $P12.tmp -inkey $CERTDIR/tls.key \
+                 -in $CERTDIR/tls.crt -passout pass:$P12PASS 2>/dev/null; then
+              mv -f $P12.tmp $P12
+              chown 1000:1000 $P12 2>/dev/null || true
+              chmod 600 $P12
+              echo "[plex-settings] PKCS#12 bundle written"
+            else
+              rm -f $P12.tmp
+              echo "[plex-settings] WARN: pkcs12 export failed; keeping any previous bundle" >&2
+            fi
+            rm -f $CERTDIR/tls.key
+          else
+            echo "[plex-settings] WARN: TLS env empty; certificate not installed this start" >&2
+          fi
+
+          # (2) Wait, bounded (~90s), for the local API. /identity needs no auth.
+          i=0
+          while [ "$i" -lt 45 ]; do
+            if [ "$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 $API/identity 2>/dev/null)" = "200" ]; then
+              break
+            fi
+            i=$(expr $i + 1); sleep 2
+          done
+
+          # (3) The prefs API needs the server token. It already lives in Preferences.xml on this
+          # PVC; it is never echoed here, and Plex redacts tokens in its own request log. An
+          # UNCLAIMED server has none — a normal state on a fresh /config — so skip, do not fail.
+          TOKEN=$(grep -o 'PlexOnlineToken="[^"]*"' "$PX" 2>/dev/null | head -1 | cut -d'"' -f2)
+          if [ -z "$TOKEN" ]; then
+            echo "[plex-settings] not claimed yet — skipping prefs, will retry next start" >&2
+            exit 0
+          fi
+
+          # Percent-encode what our values actually contain. Plex takes prefs as QUERY parameters on
+          # a PUT, so a raw '/' or ',' would be read as structure rather than data.
+          enc() {
+            printf '%s' "$1" | sed -e 's|%|%25|g' -e 's|/|%2F|g' -e 's|,|%2C|g' \
+                                   -e 's|:|%3A|g' -e 's| |%20|g'
+          }
+          setpref() {
+            code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 -X PUT \
+              -H "X-Plex-Token: $TOKEN" "$API/:/prefs?$1=$(enc "$2")" 2>/dev/null)
+            if [ "$code" = "200" ]; then
+              echo "[plex-settings] set $1"
+            else
+              echo "[plex-settings] WARN: $1 -> HTTP $code" >&2
+            fi
+          }
+
+          # (4) THE LAN DEFINITION — the reason this hook was written. Without it Plex has NO notion
+          # of a local network and tags EVERY client remote. Measured 2026-10-04: 468 requests
+          # logged "Allowed Network (WAN)" and ZERO "(LAN)", including Seerr from inside the
+          # cluster, which is why the server showed as Remote to LAN clients. Note allowedNetworks
+          # (the ALLOWED_NETWORKS env) only waives AUTH — it does NOT make a client local. Two
+          # different settings; only one of them has an env hook.
+          setpref LanNetworksBandwidth "$LANNETS"
+
+          # (5) THE CERTIFICATE, which is what makes the `https` in ADVERTISE_IP servable. Set only
+          # if the bundle exists — pointing Plex at a missing file would break its TLS listener.
+          if [ -s "$P12" ]; then
+            setpref customCertificatePath "$P12"
+            setpref customCertificateKey "$P12PASS"
+            setpref customCertificateDomain "$CERTDOMAIN"
+          else
+            echo "[plex-settings] WARN: no bundle; leaving certificate prefs untouched" >&2
+          fi
+          exit 0
 # HARDWARE TRANSCODING — the host iGPU's render device (schema 1.8.0 `hardware`; R-PLAT-02).
 # INTENT ONLY, AND THE MECHANISM DELIBERATELY DOES NOT LIVE HERE. This says "this workload needs an
 # Intel Quick Sync render device"; it does NOT name a device path and it does NOT name a resource.
@@ -156,11 +156,17 @@ OAUTH2_COOKIE_REFRESH = "2m30s"
 # ready endpoints. The flag is confirmed present in the PINNED v7.8.1 binary (`--help`),
 # which matters because the shared proxy is on v7.15.3 and could not have vouched for it.
 #
-# THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED. A container port is not a
-# Service port: these sidecars' Services expose the app port and :4180, and each namespace's
-# default-deny baseline has `ingress: []`, so Prometheus can neither resolve nor reach this
-# port until a Service port (or PodMonitor), a monitoring-ingress allow and a ServiceMonitor
-# are added. Do not read this flag as "metrics are working".
+# THE SCRAPE WORKS AS OF 2026-10-05 — but it took THREE independent pieces, and this flag is
+# only the first. A container port is not a Service port, and each namespace's default-deny
+# baseline denies all ingress, so the flag alone bought nothing for 63.7h:
+#   1. this flag + a named `metrics` container port   (PR #28)
+#   2. a monitor that targets POD ports, not Service ports — platform/oauth2-proxy/podmonitor.yaml
+#      is a PodMonitor for exactly that reason (no Service exposes 44180)
+#   3. an ingress allowance from ns monitoring        (PR #49, r_monitoring_ingress_policy below)
+# Verified on prod 2026-10-05: all 6 targets up == 1, scraping in 1-6ms.
+# STILL do not read this flag alone as "metrics are working" — 1 without 3 is precisely the state
+# that produced 5 x TargetDown for 63.7h. If you add a sidecar metrics port anywhere else, check
+# that r_monitoring_ingress_policy covers it.
 OAUTH2_PROXY_METRICS_PORT = 44180

 # --- Hardware capability: `hardware.gpu: intel-quicksync` ------------------------------
@@ -1362,8 +1368,9 @@ def _oauth2_sidecar_container(s: dict) -> dict:
        f"--cookie-refresh={OAUTH2_COOKIE_REFRESH}",
        "--pass-access-token=true",
        "--set-xauthrequest=true",
-        # Start the /metrics listener. Prerequisite only — not yet reachable by Prometheus;
-        # see OAUTH2_PROXY_METRICS_PORT above for exactly what is still missing.
+        # Start the /metrics listener. Scraped as of 2026-10-05, but only because
+        # r_monitoring_ingress_policy also opens this port to ns monitoring — the flag alone was
+        # inert for 63.7h. See OAUTH2_PROXY_METRICS_PORT above for the three pieces required.
        f"--metrics-address=0.0.0.0:{OAUTH2_PROXY_METRICS_PORT}",
    ]
    secret = f"{s['name']}-oauth2-proxy"
@@ -1541,12 +1548,54 @@ def r_monitoring_ingress_policy(s: dict):
    trap Longhorn's own NetworkPolicies set (platform/longhorn/networkpolicy-metrics.yaml exists
    for exactly this reason), and it is a THIRD independent way for a metrics SLI to be inert.

-    Rendered only alongside a declared metricsEndpoint, and it opens ONLY that port to ONLY the
-    monitoring namespace: an allow for a port nothing serves would be a standing hole for no
+    Rendered only alongside a port something actually serves, and it opens ONLY that port to ONLY
+    the monitoring namespace: an allow for a port nothing serves would be a standing hole for no
    benefit.
+
+    TWO INDEPENDENT SOURCES OF A SERVED METRICS PORT, and this used to cover only the first:
+
+      1. `observability.metricsEndpoint` — the APP's own exporter. NOTE, measured 2026-10-04: no
+         ServiceDefinition in this catalog declares one, so before the sidecar clause below this
+         function had NEVER emitted a single object on any branch. It is correct but, so far,
+         entirely latent — do not read its presence as evidence that scrape ingress is solved.
+
+      2. THE forward-auth GATE SIDECAR's own `/metrics` on 44180 (added 2026-10-04). Every
+         `_needs_oauth2_sidecar` app gets `--metrics-address=0.0.0.0:{OAUTH2_PROXY_METRICS_PORT}`
+         and a named `metrics` container port from `_oauth2_sidecar_container`, so the port is
+         served whenever the sidecar exists — unconditionally, which is what makes it safe to
+         open without a per-app declaration.
+
+    WHY CLAUSE 2 KEYS ON `_needs_oauth2_sidecar` AND NOT ON `observability.metrics`: it must match
+    the condition the SCRAPE keys on, exactly. platform/oauth2-proxy/podmonitor.yaml selects pods
+    by the `homelab.keithjackson.dev/auth-mode: forward-auth` POD LABEL and nothing else, so a
+    target exists iff the sidecar exists. Keying the allowance on `observability.metrics` instead
+    would let `metrics: false` produce a discovered-but-blocked target — the same "two halves
+    authored apart" failure this function exists to prevent, just moved one field over.
+
+    THIS CLAUSE IS A BUG FIX, and the bug was live for 63.7h. Measured on prod 2026-10-04: all 6
+    sidecar targets across jackett/radarr/sonarr/qbittorrent/pihole read `health=down`,
+    `lastError: context deadline exceeded`, `lastScrapeDuration` exactly 10.0s — the scrapeTimeout,
+    i.e. packets DROPPED, not refused. Each app namespace ships a `default-deny` NetworkPolicy
+    with `policyTypes: [Ingress, Egress]` and NO `ingress` key at all (deny-all-ingress), and the
+    only ingress allowance is `<app>-allow-gateway-ingress`: `fromEntities: [ingress]` -> 4180.
+    Prometheus matched no clause. 5 x `TargetDown` + 6 x `OAuth2ProxyGateUnobserved` had been
+    re-notifying at ~22 emails/day since the PodMonitor landed on 2026-10-02.
+
+    WHY THE POLICY BELONGS HERE AND NOT IN platform/oauth2-proxy: podmonitor.yaml's header records
+    the objection — a policy in that tier would name app namespaces created by the SIBLING
+    apps-wave-0/1 tiers (measured Flux graph: `oauth2-proxy` and `apps-wave-0` both
+    `dependsOn=[platform-config]`, no ordering between them), so it would apply on a running
+    cluster and fail on a cold bootstrap. Rendered here it ships in the same tier that creates the
+    namespace, which dissolves that ordering problem entirely.
    """
-    ep = s["observability"].get("metricsEndpoint")
-    if not ep or not s["observability"]["metrics"]:
+    obs = s["observability"]
+    ep = obs.get("metricsEndpoint")
+    ports: list[str] = []
+    if ep and obs["metrics"]:
+        ports.append(str(ep["port"]))
+    if _needs_oauth2_sidecar(s):
+        ports.append(str(OAUTH2_PROXY_METRICS_PORT))
+    if not ports:
        return None
    return {
        "apiVersion": "cilium.io/v2",
@@ -1564,7 +1613,7 @@ def r_monitoring_ingress_policy(s: dict):
                        {"matchLabels": {"k8s:io.kubernetes.pod.namespace": MONITORING_NS}}
                    ],
                    "toPorts": [
-                        {"ports": [{"port": str(ep["port"]), "protocol": "TCP"}]}
+                        {"ports": [{"port": p, "protocol": "TCP"} for p in ports]}
                    ],
                }
            ],
@@ -101,7 +101,7 @@ data:
  # tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
  # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers;
-  # plex-remote: PLEX_L4_VIP),
+  # plex-remote + pihole: PLEX_L4_VIP),
  # velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
  # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
  # UNSUBSTITUTED path would ship the literal string into the cluster.
@@ -189,6 +189,13 @@ data:
  # dev  -> 192.168.1.57  (base default; next free after pihole-dns .56 in dev's .53-.60 pool)
  # prod -> 192.168.2.24  (next free after pihole-dns .23 in prod's .20-.30 pool; see clusters/prod)
  PLEX_L4_VIP: "192.168.1.57"                # Plex: tcp/32400 remote access (NOT the web UI, which is L7)
+  # SECOND CONSUMER since 2026-10-04: catalog/pihole.yaml reads this knob to answer
+  # plex-vip.${BASE_DOMAIN} on the LAN, which is the local half of the single hostname Plex
+  # advertises to plex.tv. Without that per-host answer the zone wildcard sends the name to
+  # ${GATEWAY_VIP}, where nothing listens on 32400. So this value is now typed into THREE places
+  # that must agree: this knob, the WAN port-forward on the SOHO router (off-repo), and PiHole's
+  # answer -- and only the first two were true before.
+

  # The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
  # peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable
@@ -171,6 +171,27 @@ spec:
      - port: '4180'
        protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): paas-demo-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: paas-demo-allow-monitoring-scrape
+  namespace: paas-demo
+  labels:
+    app.kubernetes.io/name: paas-demo
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: paas-demo
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): paas-demo-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -198,6 +198,27 @@ spec:
      - port: '4180'
        protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: sonarr-allow-monitoring-scrape
+  namespace: sonarr
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: sonarr
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -198,6 +198,27 @@ spec:
      - port: '4180'
        protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: sonarr-allow-monitoring-scrape
+  namespace: sonarr
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: sonarr
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -0,0 +1,76 @@
+---
+date: 2026-10-03
+branch: fix/cilium-cgroup-hostroot
+title: Cilium socket-LB never covered host processes — cgroup.hostRoot pointed at an unmounted path
+---
+
+## What changed
+
+`cgroup.hostRoot` changes from `/run/cilium/cgroupv2` to `/sys/fs/cgroup` in all three parity
+copies: `platform/cilium/helmrelease.yaml` (authoritative), `platform/cilium/values.yaml`
+(mirror) and `infra/ansible/roles/cilium/templates/cilium-values.yaml.j2` (bootstrap).
+`autoMount` stays `false`.
+
+**The defect.** With `autoMount: false`, the Cilium chart only bind-mounts whatever exists at
+`hostRoot`, and expects the operator to have mounted cgroup2 there (cilium 1.20.1 values.yaml).
+The repo's rationale was "k3s provides the cgroupv2 mount". That is true only for
+`/sys/fs/cgroup`. Nothing mounts cgroup2 at `/run/cilium/cgroupv2` on these nodes; it is an
+empty `DirectoryOrCreate` hostPath. So the agent mounted cgroup2 there itself, from inside its
+own private cgroup namespace, and that mount's root is the agent container's own leaf cgroup.
+The socket-LB programs (`cil_sock4_connect` etc.) were attached to it and covered only the
+agent.
+
+**The consequence.** Every other host-netns process got no Service/LB translation:
+- Service ClusterIPs were unreachable from every node's host.
+- The node holding the Gateway's L2 lease could not reach the Gateway VIP (`no route to host`).
+  Other nodes reached it only because their packets crossed the wire to the lease holder, where
+  the tc datapath handles them.
+- k3s-prod-1 holds both the Gateway lease and the kube-vip API VIP. Its apiserver could not reach
+  the OIDC issuer, logging `oidc: authenticator not initialized` since at least 2026-09-12
+  (899 occurrences by 2026-10-03). kubectl SSO through the VIP therefore failed, and in-cluster
+  OIDC clients failed on whichever calls landed on that apiserver.
+- Webhooks and aggregated APIs were unaffected only because k3s runs
+  `--enable-aggregator-routing=true` (endpoint IPs, not ClusterIPs).
+
+Cilium's own source (v1.20.1 `bpf/bpf_sock.c`, `__sock4_xlate_fwd`) handles exactly this case
+when the hook does run: a host-netns connect to an L7-LB Service (the Gateway) is rewritten to
+`127.0.0.1:<l7_lb_proxy_port>`, the local Envoy listener. The fix is to make the hook cover the
+host.
+
+## What I verified
+
+- `cilium-dbg status --verbose` on k3s-prod-1: `Socket LB: Enabled`, `Coverage: Hostns-only`.
+  The live cilium-config shows `bpf-lb-sock=false`, but socket LB is implied by kube-proxy
+  replacement. The Gateway Service frontends `192.168.2.20:{80,443}` and
+  `10.43.225.79:{80,443}` map to `127.0.0.1:14337`.
+- Host: `findmnt /run/cilium/cgroupv2` is empty on all three nodes, and `/sys/fs/cgroup` is
+  `cgroup2`.
+- In the agent: `/run/cilium/cgroupv2` is a `cgroup2 none` mount, the agent's own cgroup reads
+  `0::/`, the mount has 0 child cgroups, and `bpftool cgroup show /run/cilium/cgroupv2` lists all
+  `cil_sock*` programs. k3s runs in `0::/system.slice/k3s.service`, and the agent container in
+  `/kubepods.slice/…/cri-containerd-3ad04cde….scope`.
+- **The decisive test**, run on k3s-prod-1 in the same netns: raw TCP connect to
+  `192.168.2.20:443`, `10.43.225.79:443` and `10.43.59.141:8080` CONNECTED from inside the agent
+  container (covered cgroup) and FAILED from the host (`No route to host` / timeout).
+- Blast radius: host-network pods are cilium, cilium-envoy, cilium-operator (API via
+  `192.168.2.9:6443`, not a Service), kube-vip (`127.0.0.1:6443`, `svc_enable=false`) and
+  node-exporter (no outbound). Nothing depends on host Service translation being absent; the
+  change only makes Service/LB addresses reachable from the host where they currently fail.
+- `helm template cilium/cilium --version 1.20.1` with the new values, in the toolbox: the
+  DaemonSet bind-mounts hostPath `/sys/fs/cgroup` into `cilium-agent` and `clean-cilium-state`,
+  the cilium-config shows `cgroup-root: "/sys/fs/cgroup"`, and the init containers are unchanged
+  (no `mount-cgroup`).
+- Toolbox gate: see the PR description.
+
+## What I did NOT verify
+
+- **The rollout itself.** After merge, the agent DaemonSet rolls one node at a time
+  (`maxUnavailable: 1`). On each node, afterwards: host TCP connects to the three targets above
+  must succeed. On k3s-prod-1, the apiserver's background OIDC init retry must stop logging
+  `authenticator not initialized` with no k3s restart.
+- That the old programs, attached to the previous agent container's cgroup, disappear with it.
+  Expected, because cgroup BPF attachments die with the cgroup, but not observed.
+- An end-to-end kubectl OIDC login (needs a browser).
+- No regression test was added. A natural one is an L2 conformance check that a host-netns
+  connect to the Gateway VIP succeeds on the lease holder, but `conformance/**` is a sensitive
+  path. Left as a follow-up.
@@ -0,0 +1,157 @@
+---
+date: 2026-10-04
+branch: feat/hubble-flow-export-plex
+title: Record the pre-SNAT client address for Plex via Hubble static flow export, instead of breaking externalTrafficPolicy to get it
+---
+
+## What changed
+
+Operator requirement, narrowed explicitly: **this is 100% about log visibility** — knowing which
+address connected to Plex. Not bandwidth metering, which does not matter here.
+
+**The problem, measured.** Plex's own log cannot show it. Every LoadBalancer Service in this cluster
+is `externalTrafficPolicy: Cluster`, so the node holding the L2 lease for the VIP SNATs the client
+before forwarding to the backend pod:
+
+| Measured 2026-10-04 | Value |
+|---|---|
+| Node announcing `192.168.2.24` | `k3s-prod-3` (also holds `.20` Gateway, `.23` pihole-dns) |
+| Node running the Plex pod | `k3s-prod-1` (`10.0.0.135`) |
+| Sources in Plex's log | **3,214 × `10.0.1.x`**, 105 × `10.0.2.x`, 120 × loopback |
+| `192.168.x` sources in Plex's log, ever | **0** |
+| `X-Forwarded-For` in Plex's log | **0** (it serves raw tcp/32400, not HTTP via the Gateway) |
+| `10.0.1.78` identity (`cilium-dbg bpf ipcache`) | `identity=1` → `reserved:host` = prod-3 itself |
+
+**Nothing on the router is involved** — the operator correctly pointed out there is no hairpin NAT
+here. The rewrite is entirely in-cluster, done by Cilium's BPF load-balancer on the lease holder,
+because the backend pod is on a different node.
+
+**The address is not lost, only unrecorded.** Cilium observes the flow at ingress, *before* its own
+translation, so Hubble already holds the true source:
+
+    192.168.2.133 -> 10.0.0.135    (real LAN client -> the Plex pod)
+    10.0.1.78     -> 10.0.0.135    (the same traffic post-SNAT — what Plex sees)
+
+What was missing is **retention**: the ring buffer was **4095/4095 flows at ~303 flows/s — about
+thirteen seconds of history** — with hubble metrics disabled and no exporter configured.
+
+So: enable the static flow exporter, scoped to Plex's port, writing to the agent's stdout.
+
+- **`platform/cilium/helmrelease.yaml`** (authoritative): `hubble.export.static.enabled: true`,
+  `filePath: stdout`, `allowList` of `{"destination_port":["32400"]}` and
+  `{"source_port":["32400"]}`.
+- **`platform/cilium/values.yaml`** (reference mirror) and
+  **`infra/ansible/roles/cilium/templates/cilium-values.yaml.j2`** (the `[parity]` source of truth):
+  the same block, so a bootstrap render does not differ from what Flux reconciles — a mismatch would
+  be a permanent Helm diff and an avoidable agent rollout.
+
+**`filePath: stdout`, not a file,** because Grafana Alloy already runs as a DaemonSet tailing every
+pod's logs into Loki (`platform/monitoring/alloy.yaml`). stdout therefore reaches Loki with **no new
+hostPath mount, no file-rotation handling, and no Alloy change at all**. Upstream documents the
+value: *"To get Hubble flows directly exported to the logs instead of written to a rotated file,
+`stdout` can be defined as `hubble-export-file-path`."* Query:
+
+    {namespace="kube-system", app="cilium"} |= "32400"
+
+**SCOPED BY PORT, NOT BY NAMESPACE — a measured decision, not a preference.** A
+`--to-namespace plex` query returned **only the post-SNAT flows**; the flow carrying the real client
+address did not match it, because at the point of observation the destination identity is not yet
+resolved to the plex pod. A namespace filter would have dropped exactly the record this change
+exists to capture.
+
+**WHY NOT FIX THE SNAT INSTEAD — both alternatives rejected on evidence.**
+- `externalTrafficPolicy: Local` would put the real address in Plex's own log, but cilium 1.20's L2
+  announcer is **explicitly incompatible** with it. Upstream docs for this exact version:
+  *"incompatible with the `externalTrafficPolicy: Local` on services as it may cause service IPs to
+  be announced on nodes without pods causing traffic drops"* and *"L2 announcements isn't currently
+  aware of this setting"*. Live proof it is not theoretical: the VIP is announced by prod-3 while the
+  pod runs on prod-1, so the flip would black-hole tcp/32400 **immediately**. Making it safe means
+  pinning both the pod and the announcement to one node and surrendering Plex's ability to
+  reschedule — a real availability loss to buy a log field. Same trap
+  `platform/crafty/service-l4.yaml` documents from an 8-day Minecraft outage.
+- **DSR** also preserves the client address, but `routing-mode` here is **vxlan**, not
+  native/geneve, and DSR returns bypass the ingress node, which breaks the SOHO router's stateful
+  NAT on the remote path. Cluster-wide blast radius across all eight LB Services for one service's
+  logging.
+
+**A correction to the repo's own record.** `decisions/2026-10-03-plex-remote-access-l4.md` states the
+SNAT puts the client inside `192.168.0.0/16` (the ingress node's LAN address), and I repeated that in
+the previous entry. **It is wrong**: measured, the SNAT source is `10.0.1.x` — the node's
+`cilium_host` inside the pod CIDR. This matters, because it means `LanNetworksBandwidth` can never
+discriminate local from remote while eTP is `Cluster`: `10.0.0.0/8` must be in the LAN set for the
+in-cluster consumers (Seerr) to count as local, and every external client arrives in that same range.
+PR #48's LAN definition therefore fixes the Remote badge by making *everything* local — acceptable
+only because metering is explicitly not a goal here.
+
+## What I verified
+
+Live prod, 2026-10-04, read-only.
+
+- **Every measurement in the table above**, including the VIP-lease-to-node map
+  (`cilium-dbg shell -- db/show l2-announce` on all three agents) and the pod's node.
+- **Hubble already carries the pre-SNAT source** — the single finding this whole approach rests on:
+  `hubble observe --port 32400` on the lease holder returned `192.168.2.133 -> 10.0.0.135` **and**
+  its return flow, alongside the post-SNAT `10.0.1.78 -> 10.0.0.135`.
+- **The retention gap is real:** `cilium-dbg status` reported `Hubble: Ok  Current/Max Flows:
+  4095/4095 (100.00%), Flows/s: 303.62  Metrics: Disabled`, and the `cilium-config` ConfigMap
+  contains no `hubble-export-*` key at all.
+- **The namespace-filter trap**, directly: `--to-namespace plex` returned only `10.0.1.78` and
+  `10.0.1.71`; the `192.168.2.133` flow was absent. This is why the allowList is port-scoped.
+- **All eight LoadBalancer Services are `eTP=Cluster`** (crafty ×2, gateway, pihole-dns ×2, plex,
+  unifi ×2) — so this is a cluster-wide pattern, not a Plex-specific choice.
+- **The eTP:Local incompatibility is upstream-documented for v1.20 specifically**, quoted above from
+  the docs for the running version — not inferred from the crafty incident.
+- **`routing-mode: tunnel`** and **`bpf-lb-mode-annotation: false`** in the live `cilium-config`,
+  which is what rules DSR out.
+- **The chart's value paths and defaults**, read from cilium v1.20.1's own `values.yaml`:
+  `hubble.export.static.{enabled,filePath,fieldMask,fieldAggregate,aggregationInterval,allowList,denyList,fileMaxSizeMb,fileMaxBackups,fileCompress}`,
+  with `filePath` defaulting to `/var/run/cilium/hubble/events.log`. The keys used here exist.
+- **Alloy's pipeline shape**, read from `platform/monitoring/alloy.yaml`: a DaemonSet with
+  `loki.source.kubernetes "pods"` → `loki.process` → `loki.write` at
+  `http://loki.monitoring.svc.cluster.local:3100`. It already tails kube-system pods, which is why
+  stdout needs no Alloy change.
+- **All three value files agree** (one `destination_port` occurrence each) and both YAML files parse,
+  with `export.static` resolving to the intended object.
+- **Toolbox gate**: **rc 3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
+  `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two known environmental skips
+  (`gitea-settings` — no local `SETTINGS_API_TOKEN`; `conformance` L2/L3 — `CLUSTER_AVAILABLE`
+  unset). **Not green. Reported as rc 3.**
+- **No sensitive paths touched** — `platform/cilium/**` and `infra/ansible/**` are not in
+  `infra/gitea/approval-policy.json`, so this needs no approval override, unlike PR #48.
+
+## What I did NOT verify
+
+- **THE OUTCOME. Nothing has been reconciled and no flow has ever been exported.** `cilium-config`
+  still has no `hubble-export-*` key. Every claim is about inputs.
+- **Whether `filePath: stdout` is accepted as written.** This is the single largest risk. The value
+  comes from a verbatim upstream doc sentence (*"`stdout` can be defined as
+  `hubble-export-file-path`"*), but I did not confirm the exact notation in cilium 1.20.1's source,
+  and the chart default is a real filesystem path. If the literal string is not special-cased, the
+  agent may create a FILE named `stdout` in its working directory and nothing reaches Loki — a
+  silent no-op, not a crash. **Fallback if so:** leave `filePath` at the chart default and add a
+  `local.file_match` + `loki.source.file` target to Alloy with a hostPath mount for
+  `/var/run/cilium/hubble`.
+- **Whether the FlowFilter JSON is accepted.** `{"destination_port":["32400"]}` follows the proto
+  (`destination_port` is a repeated string) and the documented shape (`'{"verdict":["DROPPED"]}'`),
+  but upstream gives **no port example** and I did not validate it against the running agent. A
+  malformed filter could be rejected — or, worse, be silently treated as "no match" and export
+  nothing.
+- **THIS CHANGE ROLLS CILIUM.** Editing the HelmRelease restarts the agent DaemonSet across all
+  three nodes. BPF maps are preserved so established traffic should survive, but prod-3 currently
+  announces the Gateway VIP, **PiHole's DNS VIP** and Plex's VIP, so a lease change during the
+  rollout could briefly interrupt LAN DNS for the whole house. Not measured, and worth doing at a
+  quiet moment.
+- **Log volume in practice.** Estimated "a few hundred to a few thousand lines a day" from the
+  observed request counts; never measured as flows. Both directions are exported and no `fieldMask`
+  is set, so each record is a full flow object. If it is noisy, narrow with `fieldMask` rather than
+  by dropping a filter.
+- **Loki's retention for this data**, and whether the `{namespace="kube-system", app="cilium"}` label
+  set actually matches (it depends on the cilium pods carrying
+  `app.kubernetes.io/name`, which Alloy maps to `app` — I did not confirm that label exists on the
+  cilium DaemonSet pods). The query in the comment may need adjusting to `pod=~"cilium-.*"`.
+- **That the exported flow actually contains the pre-SNAT address.** Hubble's *live* stream does
+  (measured). I am assuming the exporter sees the same flow objects; I did not confirm the exporter
+  is not fed a post-translation view.
+- **The parity convention has no automated check.** I kept all three files in sync by hand because
+  the comments demand it; nothing in `ci/` or `conformance/` verifies cilium Helm↔Ansible parity, so
+  a future edit can still drift them silently.
@@ -0,0 +1,155 @@
+---
+date: 2026-10-04
+branch: fix/oauth2-sidecar-metrics-scrape-netpol
+title: The forward-auth gate sidecars' /metrics was NetworkPolicy-blocked — the renderer now opens 44180 to ns monitoring
+---
+
+> **Added 2026-10-05, after merge:** the top item under *What I did NOT verify* below — "that the
+> scrape now succeeds" — **has since been verified on prod**. All 6 targets read `up == 1`. The
+> original text is left exactly as written; the measurements are in
+> [[2026-10-05-oauth2-sidecar-scrape-verified]].
+
+## What changed
+
+`catalog/render.py`'s `r_monitoring_ingress_policy` now emits its CiliumNetworkPolicy for a
+**second** reason: the forward-auth gate sidecar's own `/metrics` port (44180), keyed on
+`_needs_oauth2_sidecar(s)`. Previously it fired only from `observability.metricsEndpoint`.
+
+That matters more than it sounds, because **no ServiceDefinition in this catalog declares a
+`metricsEndpoint`** (measured: zero grep hits across `catalog/*.yaml`). So the function was wired
+in at the call site and had **never emitted a single object on any branch** — correct, and
+entirely latent. Its presence in the tree read as "scrape ingress is handled". It was not.
+
+Re-rendered `apps/` from that change: five new CNPs, one per forward-auth app
+(`jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`), each opening **only** 44180 to **only**
+`fromEndpoints: [k8s:io.kubernetes.pod.namespace: monitoring]`.
+
+The bug being fixed: `platform/oauth2-proxy/podmonitor.yaml` landed 2026-10-02T04:58:44Z and
+discovered 6 targets that have **never once scraped successfully**. Each app namespace ships a
+`default-deny` NetworkPolicy with `policyTypes: [Ingress, Egress]` and **no `ingress` key at all**
+— deny-all-ingress in NetworkPolicy semantics — and the only ingress allowance is
+`<app>-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180. Prometheus matched no
+clause, so its packets were dropped.
+
+Two deliberate choices, both recorded in the function's docstring:
+
+- **Clause 2 keys on `_needs_oauth2_sidecar`, not on `observability.metrics`.** It has to match
+  the condition the *scrape* keys on: `podmonitor.yaml` selects pods by the
+  `homelab.keithjackson.dev/auth-mode: forward-auth` pod label and nothing else, so a target
+  exists iff the sidecar exists. Keying on `observability.metrics` would let `metrics: false`
+  produce a discovered-but-blocked target — the same "two halves authored apart" failure this
+  function exists to prevent, moved one field over.
+- **The policy is rendered here, not added to `platform/oauth2-proxy/`.** podmonitor.yaml's own
+  header records why that tier cannot host it: it would name app namespaces created by the
+  *sibling* `apps-wave-0/1` tiers, so it would apply on a running cluster and fail on a cold
+  bootstrap. Rendered per-app it ships in the same tier that creates the namespace.
+
+## What I verified
+
+Measured on prod 2026-10-04, read-only, via the pinned toolbox:
+
+- **The cause, not a guess.** All 6 sidecar targets in pool
+  `podMonitor/oauth2-proxy/oauth2-proxy-sidecars/0`: `health=down`,
+  `lastError: context deadline exceeded`, `lastScrapeDuration` exactly `10.0s` — the configured
+  `scrapeTimeout`. Hitting the timeout precisely means packets **dropped, not refused**.
+- **The listener is fine.** jackett's `oauth2-proxy` container carries
+  `--metrics-address=0.0.0.0:44180` and a named port `metrics: 44180`.
+- **The deny is universal.** All six namespaces (`jackett radarr sonarr qbittorrent pihole
+  crafty`): `default-deny` with `ingress` key absent, `nIngress=0`; the only ingress CNP allowance
+  is 4180 `fromEntities: [ingress]`. **Zero** namespaces had any `-allow-monitoring-scrape` object.
+- **Frequency.** Single continuous episode, no flapping: `ALERTS_FOR_STATE` = 229343s (63.7h),
+  firing since 2026-10-02 05:09 UTC — 11 minutes after the PodMonitor was created, i.e. TargetDown's
+  `for: 10m`. A 15-day `query_range` shows **1 episode** per namespace, not many. Alertmanager:
+  `group_by: [alertname, namespace]`, `repeat_interval: 12h` on `email-warning` → 5 TargetDown
+  groups = 10 emails/day, plus 6 `OAuth2ProxyGateUnobserved` groups = 12/day, ~22/day total.
+  0 silences.
+- **The re-render is deterministic and the diff is exactly the intended one.** `apps/` is
+  byte-identical to `render.py --all catalog -o apps --wave-layout --no-image-automation`
+  (12/12 SAME before my edit; the 11-file "drift" against a default render is solely the
+  ImageRepository/ImagePolicy pair, which `apps/` is generated without). After the edit:
+  5 files changed, +21 lines each, nothing else touched.
+- `render.py --selftest`: **PASS**.
+- **Three L1 goldens needed regenerating, and the diff is +63 lines / -0.** The first gate run
+  caught `container-workload-render` (R-PLAT-02) and `egress-class-render` (R-NET-05/06/07) as
+  FAIL with exactly the new CNP as the drift. Regenerated with the same invocations the checks
+  use (`render.py <fixture>`, no flags). Evidence the regen is faithful rather than a rubber
+  stamp: `default-deny-baseline-render`'s golden came back **byte-identical** (its fixture is not
+  forward-auth), and the other three gained only the one CNP, port 44180, ns monitoring.
+- Toolbox gate `./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`: **rc 3 — UNDETERMINED,
+  NOT rc 0, and I am not reporting it as green.** Every stage that ran is green
+  (spec-immutability, approval-policy, decisions-log, lint, l0, render-golden, l1); `l1` went
+  FAIL -> pass once the goldens were regenerated. The gate's own words: "Stages that did not run:
+  gitea-settings" — skipped for want of `GITEA_SETTINGS_TOKEN`, which is an environment gap on
+  every local run and unrelated to this change. L2/L3 conformance is a `[warn]`, not a not-run
+  stage: UNVERIFIED for all 8 touched requirements (R-GITOPS-02 R-IDP-02 R-IDP-06 R-NET-01
+  R-NET-05 R-NET-06 R-NET-07 R-PLAT-02), no cluster.
+- `--emit-waves` was **not** run (it re-onboards `vm-demo` + `keycloak`); only the per-service
+  render path was used, so no wave kustomization changed.
+
+## This PR needs a human approval (sensitive path) — MEASURED, not predicted
+
+CI run #230 on PR #49, head `f9446be3`: **`MERGE GATE: BLOCKED`, and `approval-policy` is the ONLY
+failing stage.** Read from the job log, not from the status tick:
+
+    [skip] stage spec-immutability: SKIPPED (not run)   <- sibling spec repo absent in CI; expected
+    [ ok ] stage decisions-log: pass
+    [ ok ] stage lint: pass
+    [ ok ] stage l0: pass
+    [ ok ] stage render-golden: pass
+    [ ok ] stage l1: pass
+    [fail] stage approval-policy: FAIL
+    [ ok ] stage gitea-settings: pass                   <- could NOT run locally; green in CI
+    [skip] stage conformance: SKIPPED (not run)
+
+    sensitive paths touched:
+      conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml
+      conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml
+      conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml
+
+Exactly the three regenerated goldens, nothing else. `l1` and `render-golden` both pass in CI,
+which is the independent confirmation that the regenerated goldens are right rather than merely
+self-consistent with my local run.
+
+One local-reproduction note worth recording, because it wasted a step: running
+`PR_NUMBER=49 SETTINGS_API_TOKEN=... ci/approval-policy.sh` with the credential from
+`git credential fill` reports `could not determine PR author — BLOCKING`. That is an **artifact**,
+not a finding — `~/.git-credentials` holds a *password*, and `Authorization: token <password>`
+returns 401 `user does not exist [uid: 0]`. The stage needs a real API token. I nearly reported
+that 401 as the CI failure; the job log is what settled it.
+
+Regenerating the L1 goldens means this branch touches `conformance/**`, which is in
+`sensitive_paths` in `infra/gitea/approval-policy.json` ("the tests and the harness; changing them
+changes what green MEANS"). So `ci/approval-policy.sh` will require an approving review from a
+non-author, and the local gate **cannot** see that: it reports `approval-policy: green` because
+offline it evaluates N/A.
+
+The golden update is not optional — a renderer change that adds an object necessarily drifts the
+goldens that pin the renderer's output — but it does mean the gate correctly refuses to
+self-approve this one. **An agent must not clear this with `/approve-override`**, even though the
+`gitea-admin` account holds `owner`: that is precisely the self-approval the stage exists to
+prevent. The operator posts the override as a PR *review* of type "Comment", or approves normally.
+
+## What I did NOT verify
+
+- **That the scrape now succeeds.** This is a git-only change and nothing is applied until Flux
+  reconciles `main`. The 5 CNPs have never existed on the cluster, so `up == 1` for those targets
+  is **UNVERIFIED** — it is a prediction from the measured drop signature, not an observation.
+  After merge, confirm `up{container="oauth2-proxy"}` goes to 1 for all 6 targets and that the 11
+  alerts resolve. `send_resolved: true` is set, so expect ~11 resolved emails.
+- **That the CNP `endpointSelector` matches pihole's pods.** It selects
+  `app.kubernetes.io/name: pihole`, identical to the shape of `pihole-allow-gateway-ingress`
+  which demonstrably works for 4180 — so this is inference from a working sibling, not a
+  measurement of the new object.
+- **`crafty` is still unobserved and will keep firing `OAuth2ProxyGateUnobserved`.** It is
+  hand-managed under `platform/crafty`, outside the renderer, and has **no `--metrics-address`
+  at all** — so there is no port to open and this change cannot help it. Separate work.
+- **Whether 44180 is reachable in the Cilium data path once allowed.** Only the policy was
+  reasoned about; no L2 probe was run (CI has `vars.CLUSTER_AVAILABLE` unset, so L2/L3 did not
+  run here either — the gate proves the L0/L1 offline subset only).
+- **Why `OAuth2ProxyGateUnobserved` started at 2026-10-04 06:44** rather than 10-02 with
+  TargetDown. Alertmanager did not restart (TargetDown's `startsAt` is still 10-02) and the rule's
+  `for` is 1h, so neither explains it. Something changed on the monitoring side that morning which
+  I did not identify. It does not affect this fix.
+- I did not re-run a port-forward to 44180 to prove the endpoint serves *today*; I confirmed the
+  flag and named port on the live pod spec. The 200/10208-byte measurement is podmonitor.yaml's,
+  from 2026-10-01.
@@ -0,0 +1,117 @@
+---
+date: 2026-10-04
+branch: docs/pihole-ha-verified
+title: PiHole HA is VERIFIED on live prod — including a failover drill with zero dropped DNS queries
+---
+
+## What changed
+
+**No manifests changed. This entry exists to correct the log.** The three PiHole/Plex entries merged
+earlier today each say, correctly at the time, that nothing had been reconciled and that the outcome
+was unverified:
+
+- `decisions/2026-10-04-pihole-declare-state.md` — "Nothing here has been reconciled."
+- `decisions/2026-10-04-pihole-multi-replica.md` — "Nothing has been reconciled. No StatefulSet
+  exists, no second replica has ever run, the anti-affinity has never been evaluated..."
+
+All three are now **merged, reconciled and measured**, so those sentences are stale in exactly the way
+§2b of CLAUDE.md warns about — a reader who found them would conclude the work is unproven. This entry
+is the current state; it does not edit the earlier files (separate files, nothing to merge).
+
+Delivered and live as of 2026-10-04, from PRs #43 (plex tcp/32400), #44 (declare PiHole state) and
+#45 (PiHole 2 replicas + the mountPath bug):
+
+- PiHole is a **2-replica StatefulSet**, one pod per node, with per-replica Longhorn volumes, a
+  PodDisruptionBudget, and an `up` SLI that follows the rendered workload kind.
+- PiHole's admin password and adlists are **declared in git** and reproduce onto a fresh volume.
+- PiHole's config PVC is mounted where PiHole actually writes (`/etc/pihole`), which it was not before.
+
+## What I verified
+
+All measured against live prod on 2026-10-04, after the merge of #45 (`dce20b7`).
+
+**Reconcile.** `GitRepository` artifact `main@sha1:dce20b7`; `apps-wave-0`
+`lastAppliedRevision=main@sha1:dce20b7`, `Ready=True/ReconciliationSucceeded`. It completed in roughly
+**90 seconds**, well inside the Kustomization's `timeout: 5m` — the risk flagged in the multi-replica
+entry ("whether `pihole -g` finishes inside the 5 m timeout") did **not** materialise. A transient
+`Ready=False/DependencyNotReady: platform-config` seen mid-flight was the dependency cascade, not a
+failure; `platform-config` was already `Ready=True` at the new revision moments later.
+
+**Topology and objects.**
+
+| Measured | Value |
+|---|---|
+| pods / nodes | `pihole-0` → **k3s-prod-3**, `pihole-1` → **k3s-prod-2** (2/2 Running each) — the required anti-affinity was satisfied |
+| per-replica claims | `config-pihole-0`, `config-pihole-1` — both `Bound`, 2Gi |
+| old `pihole-config` PVC | **pruned** (it held only `lost+found`, so nothing was lost) |
+| old `pihole` Deployment | **pruned** — `No resources found in pihole namespace` |
+| PodDisruptionBudget | `minAvailable 1`, `disruptionsAllowed 1`, `currentHealthy 2` |
+
+**The mountPath bug is fixed.** `df` inside both pods reports `/etc/pihole` on
+`/dev/longhorn/pvc-...` with **6.9 M used** — previously it was on `overlay` while the Longhorn volume
+sat unused at `/data/config`. `backup.policy: daily-offcluster` now has real content to back up
+instead of reporting success over an empty volume.
+
+**Phase 1's declarations reproduce onto a FRESH volume — the point of doing it first.** Both replicas
+booted with an empty `/etc/pihole`:
+
+- **72,234 gravity domains** on each, rebuilt from the adlist declared in `catalog/pihole.yaml`.
+- **The declared SOPS secret authenticates on both pods.** `POST /api/auth` using the container's own
+  `FTLCONF_webserver_api_password` returned `"valid":true` on `pihole-0` **and** `pihole-1` — so
+  Secret → env → FTL's own hashing → working login, identically on both. (The secret's value was never
+  printed; an earlier reading of `pwhash` as "unset" was a quoting bug in the check, disproven by an
+  empty password being rejected with `password incorrect` and then by this positive test.)
+
+**User-facing behaviour through the VIP (192.168.2.23), from a LAN host.**
+
+    plex.home.keithjackson.dev   -> 192.168.2.20      split-horizon intact
+    home.keithjackson.dev        -> 192.168.2.20
+    unifi.home.keithjackson.dev  -> 192.168.2.21
+    google.com                   -> 142.250.115.102   recursion to 8.8.8.8 works
+    doubleclick.net              -> 0.0.0.0           blocking LIVE in FTL, not just in the DB
+    googleadservices.com         -> 0.0.0.0
+    12 consecutive queries       -> 12/12 answered
+
+**THE FAILOVER DRILL — observed by the operator, 2026-10-04.** A single StatefulSet replica was
+deleted. It was recreated in **~30 seconds**, and **DNS never dropped a request** throughout. This is
+the first direct evidence that the replica count delivers what it was added for, rather than merely
+rendering correctly.
+
+**The SLI is not inert, and re-deriving it mattered.** Measured in Prometheus:
+`kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"}` returns **1 series,
+value 2**; `kube_deployment_status_replicas_available{namespace="pihole"}` now returns **0 series**;
+and Prometheus has `pihole-down` loaded as `state=inactive health=ok` with the StatefulSet expression.
+Had the rule stayed Deployment-shaped, its `or absent(...)` branch would have matched the now-absent
+series and **fired permanently** — a standing false page rather than silence. That is the concrete
+payoff of defaulting `livenessWorkload.kind` from the rendered kind.
+
+## What I did NOT verify
+
+- **NODE failure, as opposed to pod failure.** The drill deleted a pod, which does not exercise the
+  `tolerationSeconds: 300` path at all — that timer only applies when a node goes NotReady. The
+  mechanism protecting DNS is the same one just proven (the surviving replica plus
+  `externalTrafficPolicy: Cluster` on the DNS LoadBalancers), so user-visible impact should likewise be
+  nil; what the 300 s still delays is the *replacement* replica, and during that window there is no
+  redundancy left. Untested. Pulling power from a node is the only real test.
+- **DNS during the cutover itself.** I was not sampling resolution while #45 reconciled, so whether
+  there was a brief gap between the Deployment being pruned and the StatefulSet becoming Ready is
+  **unmeasured**. The drill above covers the steady-state failover case, not that one-off transition.
+- **Two-node loss.** The hard anti-affinity means `replicas: 2` needs two schedulable nodes; losing
+  two would leave a replica `Pending` and, with `wait: true`, stall the app tier. Reasoned, not
+  exercised — and it coincides with etcd quorum loss on a 3-node control plane.
+- **Longhorn capacity headroom.** Two 2Gi claims at 3 replicas each is ~12Gi where there was ~6Gi.
+  Both claims are `Bound` and healthy; free space was not checked.
+- **Query history is now split per replica** and the admin UI shows only whichever pod answered.
+  Adlists and password are identical, so blocking and auth are consistent; the *reporting* is not, and
+  whether that is acceptable long-term is the operator's call.
+- **The residual risk is unchanged, and it is the larger one.** Replicas address node failure. LAN DNS
+  still depends on the cluster — etcd quorum, Cilium, Longhorn, a bad reconcile — and only a resolver
+  outside the cluster carrying the same `address=` lines removes that dependency. Considered and not
+  chosen (operator decision, 2026-10-04).
+- **Plex remote access (#43) is NOT covered by this entry.** Its two manual steps (the WAN tcp/32400
+  forward and Plex's "manually specify public port") and the outside-in path remain as
+  `decisions/2026-10-03-plex-remote-access-l4.md` describes them. The operator reported it working;
+  I have not measured it from off-LAN.
+- **`--emit-waves` drift is still open.** Re-running it still wants to onboard `vm-demo.yaml` and
+  `keycloak.yaml`, both listed as intentionally not onboarded in `ci/render-golden.sh`. Unfixed; see
+  `decisions/2026-10-03-plex-remote-access-l4.md`.
@@ -0,0 +1,139 @@
+---
+date: 2026-10-04
+branch: fix/plex-advertise-plexdirect
+title: Plex vanished from the plex.tv account — every advertised connection was uncompletable; advertise the plex.direct name instead of the bare VIP
+---
+
+## What changed
+
+Reported symptom: the server no longer appears on the operator's plex.tv account; `plex.home.keithjackson.dev`
+shows "not authorized"; connecting straight to the VIP + port works.
+
+**Root cause, measured: `customConnections` held two hand-typed URLs and NEITHER could ever complete,
+so Plex had nothing publishable.** Because `customConnections` OVERRIDES Plex's own connection
+publishing, the server ended up advertising only dead endpoints. Live `Preferences.xml`, 2026-10-04:
+
+    customConnections="https://plex.home.keithjackson.dev:32400/,https://192.168.2.24:32400/"
+    ManualPortMappingMode="1"   ManualPortMappingPort=<ABSENT>
+
+- `https://192.168.2.24:32400` — **TLS can never validate.** Plex serves 32400 with
+  `CN=*.130072f20c334d6c8f63d0dfef87309b.plex.direct` and **no IP SAN**, so a bare-IP https URL fails
+  for every client. Not theoretical: Plex's log carries a continuous storm of
+  `CERT: incomplete TLS handshake from <node> ... sslv3 alert certificate unknown`, every few seconds,
+  from **real LAN clients** arriving through the new `plex-l4-tcp` VIP.
+- `https://plex.home.keithjackson.dev:32400` — **nothing listens.** The shared Gateway serves that host
+  on 443; measured `http=000` (connect failed) against `192.168.2.20:32400`.
+
+Consequence chain, each link measured: no completable connection -> plex.tv could publish nothing
+(`MyPlex: mapping state set to 'Mapped - Not Published'`, repeatedly) -> Plex fell back to **Plex Relay**
+(`startRelay`, remote forward to 127.0.0.1:32401) -> **the server disappeared from the account**, and
+clients reported "not authorized".
+
+The fix advertises Plex's own `<dashed-ip>.<serverhash>.plex.direct` form, which is public DNS resolving
+to the embedded address — so it points at the LAN VIP *and* matches the wildcard cert.
+
+- **`catalog/plex.yaml`:** `ADVERTISE_IP: ${PLEX_ADVERTISE_URL}` added, replacing the long-standing
+  DEFERRAL note. The deferral had weighed only the L7 Gateway URL; the operator chose the native L4 path
+  on 2026-10-03, so the open question was never "Gateway or not" but "which URL has a valid cert".
+- **`clusters/base/cluster-vars.yaml`:** new `PLEX_ADVERTISE_URL` knob (+ substitution-inventory comment
+  updated to name `plex: PLEX_ADVERTISE_URL`). The base default is an explicit **non-functional
+  placeholder** — dev is decommissioned and has no claimed server, and
+  `StrictPostBuildSubstitutions` makes an undefined var a hard failure, so the default only has to exist
+  and be correctly shaped.
+- **`clusters/prod/kustomization.yaml`:** prod value
+  `https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400`.
+- **`apps/wave-0/plex.yaml`:** re-rendered with the canonical command
+  (`python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation`). Two added
+  lines, nothing else.
+
+**Why a whole-URL cluster-var and not something built from `${PLEX_L4_VIP}`:** postBuild substitution is
+plain envsubst and cannot turn `192.168.2.24` into `192-168-2-24`, and `<serverhash>` is the server's
+certificate identity, derivable from nothing in this tree. The dashed IP and `${PLEX_L4_VIP}` are two
+halves of one address and **nothing checks that they agree** — noted in both files.
+
+**SENSITIVE PATHS — this PR touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK it.**
+It needs the operator's `/approve-override <reason>` posted as a PR **review** of type "Comment".
+Deliberately not posted here: CLAUDE.md §1 forbids an agent clearing this on its own PR, and the agent
+account holds `owner`, so the prohibition is a rule rather than a limit.
+
+## What I verified
+
+All measurements against live prod, 2026-10-04, read-only. Secrets handled per CLAUDE.md §6: preference
+attributes were read by name-scoped grep, never by dumping `Preferences.xml` (it holds `PlexOnlineToken`),
+and the token was reported only as `<PRESENT,len=22>`.
+
+- **THE FIX ITSELF, with full TLS verification (no `curl -k`)** — the one measurement the whole change
+  rests on:
+  - `https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400/identity` -> **`http=200`, TLS verify OK**
+  - `https://192.168.2.24:32400/identity` -> **`http=000`, cert validation FAILED**
+  - that name resolves to `192.168.2.24` (`getent hosts`), confirming the dashed-IP mechanism works from
+    inside this network and is not blocked by DNS-rebinding protection on the resolver path tested.
+- **The server is still claimed — this was never a lost claim.** `/identity` -> `claimed="1"` with the real
+  `machineIdentifier`; `PlexOnlineToken` present, `PlexOnlineUsername="jakman85"`.
+- **The certificate, read off the live listener:** exactly one SAN,
+  `DNS:*.130072f20c334d6c8f63d0dfef87309b.plex.direct`. No IP SAN. This is why the bare-IP URL fails.
+- **The publish payload Plex actually sent to plex.tv**, which is what proves the connection list was the
+  problem rather than a guess: `PUT https://servers.plex.tv/devices/<id>?Connection[][uri]=https://plex.home.keithjackson.dev:32400&Connection[][uri]=https://192.168.2.24:32400&Connection[][uri]=http://136.51.115.252:32400&Connection[][uri]=http://10.0.2.136:32400&httpsEnabled=1...`
+  — all four unreachable from a LAN client or from plex.tv.
+- **`Mapped - Not Published`** at every reachability check across the whole log, and an active **Plex Relay**
+  session — the documented fallback when nothing is directly publishable.
+- **The `certificate unknown` TLS failures come from real client traffic, not a probe.** The source
+  `10.0.1.78` resolves via `cilium-dbg bpf ipcache get` to **identity 1 = `reserved:host`**, i.e. LAN
+  clients SNAT'd through `plex-l4-tcp` (`externalTrafficPolicy: Cluster`) on prod-3.
+- **The Gateway is NOT broken, and my first reading of it was wrong.** A 403 `Access denied` from
+  `server: envoy` on the Gateway path turned out to be an artifact of probing from inside the plex pod:
+  **all 16** Gateway hostnames return 403 from there, because Cilium's Gateway preserves the original
+  source identity and enforces the destination's ingress policy. Plex's own log shows a concurrent,
+  **working, signed-in** browser session through the Gateway returning 200s. The HTTPRoute is
+  `Accepted=True`/`ResolvedRefs=True` with no filters, and it is the only route claiming that hostname.
+- **Cilium is not denying the Gateway->Plex hop:** `cilium-dbg bpf policy get 3796` shows
+  `Allow Ingress reserved:ingress 32400/TCP`, PROXY PORT `NONE`, with 141k packets counted. No L7/HTTP
+  rules and no CCNPs exist anywhere in the cluster.
+- **In-cluster consumers were never affected:** Seerr's recently-added scans and watchlist syncs complete
+  cleanly with no auth errors; there are **zero** HTTP 401s in Plex's log (the five ` 401 ` hits are
+  transcoder segment numbers).
+- **`ADVERTISE_IP`'s mechanism, in the running image rather than from docs:**
+  `/etc/cont-init.d/40-plex-first-run` line 127 does `setPref "customConnections" "${ADVERTISE_IP}"`,
+  outside the claim guard, and `/.firstRunComplete` (lines 9/139) is on the container overlay, not the
+  `/config` PVC — so it re-applies on every pod start. Editing the env rolls the Deployment, which applies it.
+- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
+  `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
+  **pass**. The rc 3 is the two known environmental skips — `gitea-settings` (no local
+  `SETTINGS_API_TOKEN`) and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3.**
+- **`render-golden` passing is the specific proof** that the catalog edit and the committed golden agree;
+  the rendered diff is exactly two lines and no other service's manifest moved.
+
+## What I did NOT verify
+
+- **THE OUTCOME. Nothing here has been observed to put the server back on the plex.tv account.** The
+  change is unreconciled: `ADVERTISE_IP` is absent from the live Deployment, the live
+  `customConnections` still holds the two broken URLs, and Plex has not been restarted. What is proven is
+  that the *new URL answers 200 with a valid certificate* and that *every currently advertised URL cannot
+  complete* — a necessary-and-plausibly-sufficient argument, not a demonstrated fix.
+- **Whether plex.tv will publish and list the server once this lands.** plex.tv's decision is made
+  off-network and cannot be measured from here. A LAN-only connection may still leave the account showing
+  the server as remotely unreachable even when local clients work.
+- **REMOTE (off-LAN) ACCESS IS NOT FIXED BY THIS, and may still be fully broken.** The advertised URL
+  resolves to an RFC1918 address, so it serves LAN clients only. The off-repo WAN tcp/32400 port-forward
+  (`runbooks/plex-remote-access.md`) is still **unverified** — `http://136.51.115.252:32400` was
+  unreachable from inside the LAN, but 2026-10-03 established there is no NAT hairpin here, so that is
+  **inconclusive**, not proof of absence. This repo has no off-LAN vantage point.
+- **`ManualPortMappingMode="1"` with `ManualPortMappingPort` ABSENT is left as-is.** Manual public-port
+  mapping is enabled with no port set — a half-configured state I did not change and whose exact effect on
+  publishing I did not isolate. There is **no GitOps path** to it (2026-10-03: the image supports no
+  `PLEX_PREFERENCE*` of any kind), so it is an operator UI step.
+- **Whether the operator intended the hand-typed URLs** to override the 2026-10-03 native-L4 design, or
+  typed them while debugging. I did not alter live preferences; the env var will overwrite
+  `customConnections` on the next pod start, which is a deliberate transfer of ownership from the web form
+  to git and will **silently discard** anything typed there since.
+- **The `plex.direct` hash's stability.** Treated as server identity and read once from the live cert. I
+  did not establish what reissues it, so this URL could go stale with no alert; nothing in CI or the
+  conformance suite checks that the dashed IP still matches `${PLEX_L4_VIP}` or that the hash is current.
+- **Resolver behaviour for the whole LAN.** `plex.direct` resolved correctly from a *pod* (cluster DNS
+  path). LAN clients resolve via PiHole, and dnsmasq's `stop-dns-rebind` would reject a public answer
+  carrying a private address. I did **not** test resolution from a LAN client, and this is the most likely
+  way the fix still fails.
+- **The CI approval-policy outcome.** The local stage reported green, but that is an offline N/A, not a
+  verdict: this PR touches `clusters/**`, so CI will block pending the operator's override.
+- **Why PMS restarted.** The process came up at 05:26 UTC while the pod is 2d2h old with 0 restarts, so
+  something restarted Plex inside the container. Unexplained and not investigated.
@@ -0,0 +1,126 @@
+---
+date: 2026-10-04
+branch: fix/plex-unblock-app-tier
+title: Unblock the app tier — my Certificate stalled apps-wave-0, and the advertised https URL had no servable certificate
+---
+
+## What changed
+
+**I broke the app tier with PR #48 and this reverts the part that did it.** Two faults, both mine,
+both measured after the merge:
+
+1. **Kyverno blocks cert-manager in app namespaces.** The temporary private-key Secrets
+   (`plex-vip-<random>`) were denied by `secret-provenance/secrets-must-be-flux-managed`:
+
+       resource Secret/plex/plex-vip-rp4qj was blocked due to the following policies
+       secret-provenance:
+         secrets-must-be-flux-managed: Secrets in application namespaces must be delivered by
+         Flux from SOPS-encrypted git ... or be exempt controller-generated Secrets.
+
+   The policy matches every Secret in a namespace labelled
+   `homelab.keithjackson.dev/app-namespace: true` and excludes only Secrets labelled
+   `homelab.keithjackson.dev/secret-managed-by: controller` or carrying a Flux label. **The `plex`
+   namespace has that label; the `gateway` namespace — where the working wildcard cert lives — does
+   NOT.** That asymmetry is why the existing Certificate was never affected and mine was, and I did
+   not check it. cert-manager's temp key Secrets carry neither exempting label, and `secretTemplate`
+   cannot label them (the keymanager controller creates them, not the Certificate template). Result:
+   **no CertificateRequest, Order or Challenge was ever created.**
+
+2. **Flux health-checks the Certificate — guarding the pod was not enough.** `optional: true` did
+   exactly what it was designed to do: Plex rolled and ran fine without the Secret (verified). But
+   `apps-wave-0` is `wait: true`, so Flux waits on every applied object with a known health type:
+
+       health check failed after 5m0.045564329s: timeout waiting for:
+       [Certificate/plex/plex-vip status: 'InProgress']
+
+   apps-wave-0 never went Ready, and apps-wave-1/-2 plus everything downstream of platform-config
+   sat in `DependencyNotReady` behind it. I reasoned about the pod and missed the object.
+
+A third, independent problem the same PR introduced: **the advertised URL was `https://` with no
+servable certificate.** Plex answers 32400 with its `*.<serverhash>.plex.direct` cert, which cannot
+cover `plex-vip.${BASE_DOMAIN}` — the precise failure mode of the outage earlier that day.
+
+- **`apps/wave-0/plex-cert.yaml` DELETED**, and `plex-cert.yaml` removed from
+  `apps/wave-0/kustomization.yaml` `resources`, so Flux prunes the stuck Certificate and the health
+  gate clears. **Deleted rather than merely unregistered** — my first attempt kept the file as
+  "tracked but not onboarded" (the shape `keycloak.yaml` and `vm-demo.yaml` use) and **the repo's own
+  gate rejected it**: `tls-lifecycle-automated` (R-TLS-02) fails any Certificate in the tree that no
+  Flux path reconciles — *"nothing reconciles it, so a deleted Certificate/issuer/controller would
+  stay deleted until a human re-applied it."* That pattern is legal for a Deployment and not for a
+  TLS object, which is a distinction I had not known.
+- **`catalog/render.py`:** the now-dangling `plex-cert` EXTRA_UNITS entry removed.
+- **The re-enable recipe is recorded at the call site** — in `catalog/plex.yaml` beside the scheme
+  it is coupled to — rather than in the deleted file: the Kyverno exemption, the non-gating
+  Kustomization, and the `https` flip, as one change.
+- **`catalog/plex.yaml`:** `ADVERTISE_IP` → **`http://`**`plex-vip.${BASE_DOMAIN}:32400`, explicitly
+  temporary, with the restore condition written at the call site.
+
+**Everything else from PR #48 is kept and working** — the hostname design, the PiHole split-horizon
+answer, the `LanNetworksBandwidth` hook, the `optional` schema field, and the TLS env (inert without
+the Secret, so re-enabling is additive).
+
+## What I verified
+
+Live prod, 2026-10-04, read-only, after the #48/#50 merges.
+
+- **The stall and its exact cause**: the health-check timeout message above, and the full dependency
+  cascade (`cilium` → "dependency 'flux-system/flux-system' revision is not up to date", everything
+  else `DependencyNotReady`). The root `cilium` wait was a normal revision-gated cascade; the
+  *Certificate* was the real blocker.
+- **The Kyverno denial text, verbatim**, from the cert-manager controller log, repeating every ~30s
+  across at least 15 distinct Secret names.
+- **The policy's own structure**, read from `platform/kyverno/admission/secret-provenance.yaml`:
+  match = Secrets in `app-namespace: true` namespaces; exclude = `secret-managed-by: controller` OR
+  a `kustomize.toolkit.fluxcd.io/name` label; validate = unconditional `deny`.
+- **The namespace-label asymmetry**: `plex` carries
+  `homelab.keithjackson.dev/app-namespace: "true"`; `gateway` does not.
+- **cert-manager's controller ServiceAccount is `cert-manager`** in namespace `cert-manager` — the
+  identity a future exemption should name.
+- **`optional: true` worked**: the new pod (`plex-c4f87c84d-zgjq5`) is `1/1 Running` with no Secret
+  present, and `/config/certs` was never created.
+- **The postStart hook RAN and took effect** — the thing the whole hook was written for:
+  `LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16"` is now set in the live `Preferences.xml`, and
+  `customConnections="https://plex-vip.home.keithjackson.dev:32400"` was applied from the env. The
+  certificate prefs are correctly **ABSENT**, i.e. the hook's "no bundle → leave them alone" guard
+  behaved as designed.
+- **PiHole's split-horizon answer is correct**, asked of `pihole-0` directly:
+  `plex-vip.home.keithjackson.dev -> 192.168.2.24`, alongside `plex -> .20` and `crafty-mc -> .22`,
+  and the live `/etc/pihole/dnsmasq.conf` carries
+  `address=/plex-vip.home.keithjackson.dev/192.168.2.24`. **This corrects my own earlier check**,
+  which read `.20` and which I nearly reported as a failure — that test resolved via CLUSTER DNS
+  (CoreDNS) from inside a pod and never consulted PiHole at all.
+- **The gate caught my first attempt and I fixed it rather than working around it.** Keeping the
+  file unregistered produced `MERGE GATE: BLOCKED` (rc 1) with
+  `CHECK FAIL (tls-lifecycle-automated): 1 of 612 assertions failed`. After deleting the file and
+  the dangling EXTRA_UNITS entry: **rc 3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
+  `render-golden`, `l1` and `approval-policy` all **pass** — the rc 3 being only the two known
+  environmental skips (`gitea-settings`, `conformance` L2/L3). **Not green. Reported as rc 3.**
+
+## What I did NOT verify
+
+- **Whether this actually unblocks the tier.** Unreconciled at time of writing. The reasoning is
+  that pruning the Certificate removes the only failing health check, but I have not seen
+  apps-wave-0 go Ready.
+- **Whether `http://` restores client connectivity.** PiHole resolves the name and Plex serves 32400
+  (both measured), but I have not watched a Plex client complete a connection over the advertised
+  URL, and I did not re-check whether plex.tv publishes it.
+- **THE REMOTE BADGE — the symptom that started this.** `LanNetworksBandwidth` is set, but the log
+  still tags traffic `Allowed Network (WAN)`: **82 WAN, 0 LAN**, with entries continuing after the
+  pref was written. So either Plex needs a restart for it to take effect, or that log tag is not
+  driven by this pref at all. **I do not know which, and the user-visible badge is the only real
+  test** — it is in the Plex client, which I cannot see.
+- **The Hubble flow export is NOT active.** The cilium agents are still the pre-merge pods, so the
+  `filePath: stdout` question is still open and no flow has been exported. The Cilium rollout — and
+  its LAN-DNS risk, since prod-3 holds the Gateway, PiHole-DNS and Plex VIPs — is still AHEAD.
+- **Whether a cert-manager exemption is the right shape**, or whether it widens
+  `secret-provenance` more than the operator wants. It is `platform/kyverno/**`, a sensitive path,
+  and deliberately not attempted here.
+- **Whether the pruned Certificate leaves residue** — I did not check for orphaned
+  `plex-vip-<random>` Secrets or CertificateRequests after the prune. Note Kyverno DENIED every one
+  of those Secrets, so there may be nothing to clean, but that is an inference, not a check.
+- **Whether `tls-lifecycle-automated` would also have objected to the Certificate living in a
+  `wait: false` Kustomization** (the recommended re-enable shape). It demands a reconciling Flux
+  path, which that provides, but I did not test it.
+- **The unencrypted advertised URL.** `http://` is tolerable only while remote access does not work
+  (no WAN forward). Nothing enforces that assumption, and nothing will complain if `http` is still
+  there the day the port-forward appears.
@@ -0,0 +1,151 @@
+---
+date: 2026-10-04
+branch: feat/plex-vip-hostname-and-lan-networks
+title: Plex — one advertised hostname for local and remote, its own LE certificate, and a LAN definition so LAN clients stop being reported as remote
+---
+
+## What changed
+
+Two operator-reported symptoms, after the 2026-10-04 "server missing from the account" fix
+(decisions/2026-10-04-plex-advertise-plexdirect.md) had restored publishing:
+
+1. **LAN clients were shown as Remote.** Root cause, measured: `LanNetworksBandwidth` was **ABSENT**,
+   so Plex had no definition of a local network and classified *everything* as WAN — **468 requests
+   logged `Allowed Network (WAN)` and ZERO `(LAN)`**, including Seerr dialling from inside the
+   cluster. `allowedNetworks` (the `ALLOWED_NETWORKS` env) only waives **auth**; it does not make a
+   client local. Two different settings, and only one of them has an env hook.
+2. **The advertised URL was a single-purpose, fragile value.** The previous fix advertised
+   `https://<dashed-ip>.<serverhash>.plex.direct:32400`, which works, but hard-codes the VIP *and*
+   the server's certificate hash into a cluster-var — two values nothing could keep in sync.
+
+**Operator design adopted (2026-10-04): ONE hostname, `plex-vip.${BASE_DOMAIN}`, serving both
+audiences by split-horizon DNS** — PiHole answers it with `${PLEX_L4_VIP}` on the LAN, Cloudflare
+answers it publicly with the WAN address, NATed to the same VIP.
+
+- **`catalog/plex.yaml`:** `ADVERTISE_IP` → `https://plex-vip.${BASE_DOMAIN}:32400`. New
+  `lifecycle.postStart` hook (level-triggered, same idiom as pihole's and qbittorrent's) that
+  re-asserts on **every** start: `LanNetworksBandwidth`, and the three `customCertificate*` prefs
+  after converting the PEM to the PKCS#12 bundle Plex requires. New `PLEX_TLS_CRT`/`PLEX_TLS_KEY`
+  env from the `plex-vip-tls` Secret, both `optional: true`.
+- **`apps/wave-0/plex-cert.yaml` (new, hand-written):** a cert-manager `Certificate` for exactly
+  `plex-vip.${BASE_DOMAIN}` from ClusterIssuer `letsencrypt-prod`. Non-catalog because the schema has
+  no field for a Certificate. A single name, not the wildcard: the wildcard's Secret lives in
+  `gateway` and Secrets are not readable across namespaces, and re-requesting `*.${BASE_DOMAIN}`
+  would burn a duplicate-certificate slot for no benefit.
+- **`schemas/servicedefinition.schema.json` → 1.11.0:** adds `optional` to
+  `env.valueFrom.secretKeyRef` / `configMapKeyRef`. **This is load-bearing, not tidiness.** The
+  Certificate must be issued into the `plex` namespace, and that namespace is created by the
+  rendered `apps/wave-0/plex.yaml` itself — so it cannot be issued in an earlier tier and lands in
+  the *same* reconcile as the workload. With hard refs, Plex would sit in
+  `CreateContainerConfigError` until a certificate issued, and `apps-wave-0` is `wait: true` with
+  `apps-wave-1` dependsOn it, so a slow or failed DNS01 challenge would have **stalled the whole app
+  tier** — the hazard `catalog/plex.yaml` already documents for its GPU resource.
+- **`catalog/pihole.yaml`:** `address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}` added to
+  `FTLCONF_misc_dnsmasq_lines`. Required, not cosmetic: measured 2026-10-04,
+  `plex-vip.${BASE_DOMAIN}` resolved to **`192.168.2.20`** (the Gateway VIP) via the zone wildcard,
+  and a connect to `:32400` there returned **`http=000`**.
+- **`clusters/base/cluster-vars.yaml` + `clusters/prod/kustomization.yaml`:** `PLEX_ADVERTISE_URL`
+  **deleted** — the hostname makes it unnecessary, which removes the cert-hash fragility outright.
+  `PLEX_L4_VIP`'s note now records its second consumer (pihole).
+- **`catalog/render.py`:** `plex-cert` EXTRA_UNITS entry; **`apps/wave-0/kustomization.yaml`:**
+  `plex-cert.yaml` hand-added (NOT via `--emit-waves`, which would also onboard `keycloak` and
+  `vm-demo` — the pre-existing drift recorded on 2026-10-03).
+
+**A correction to my own earlier reasoning, recorded because it was wrong in the handoff log.** The
+previous entry asserted a bare-IP URL "can NEVER validate" and treated a SAN match as a
+*requirement*. The operator pointed out it is not: `httpsRequired=0` and `secureConnections` default
+to Preferred, so clients can fall back to cleartext. The measurements still stand (a validating
+client did fail, and real clients did send `certificate unknown`) — but the correct framing is that a
+matching certificate keeps the remote leg **encrypted**, not that it is required for Plex to
+function. That is why this branch obtains a real certificate for the advertised name rather than
+choosing the name to suit Plex's built-in one.
+
+**SENSITIVE PATHS — touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK until an
+approving review (or the operator's `/approve-override <reason>` posted as a PR *review* of type
+"Comment"). Deliberately not self-posted: CLAUDE.md §1.**
+
+## What I verified
+
+Live prod, 2026-10-04, read-only. Secrets per CLAUDE.md §6: prefs were read by name-scoped grep,
+the server token was used inside the pod and never printed, and no `Preferences.xml` was dumped.
+
+- **The LAN-classification root cause, directly:** `LanNetworksBandwidth=<ABSENT>`, and request
+  tagging of **468 × `Allowed Network (WAN)` vs 0 × `(LAN)`** — including `10.0.2.33` (Seerr), an
+  in-cluster pod. This is the measurement the whole hook rests on.
+- **All four prefs the hook sets EXIST on this build**, read from the server's own `/:/prefs`
+  (token-authenticated, output filtered to ids): `LanNetworksBandwidth`, `customCertificatePath`,
+  `customCertificateKey`, `customCertificateDomain`, all `type="text"`. I had earlier been unable to
+  confirm this (an unauthenticated `GET /:/prefs` returns **401**) and said so rather than assuming.
+- **There is NO env path for these settings** — the reason the hook exists. Grep of the image's
+  entire init surface: `setPref` is called for exactly six prefs (`MachineIdentifier`,
+  `ProcessedMachineIdentifier`, `PlexOnlineToken`, `customConnections`, `allowedNetworks`,
+  `TranscoderTempDirectory`) and the only `PLEX_*` env it reads are CLAIM/CLAIM_FILE/UID/GID/
+  UPDATE_CHANNEL and five INFO/HOME vars. No `PLEX_PREFERENCE*` of any kind; zero files mention
+  `LanNetworks`.
+- **The DNS gap was real:** `plex-vip.${BASE_DOMAIN}` resolved to `192.168.2.20` (Gateway VIP), and
+  `https://192.168.2.20:32400` returned `http=000` (nothing listening). PiHole's wildcard is why.
+- **`openssl` is present in the image** (`/usr/bin/openssl`, 3.0.13) — the hook's PEM→PKCS#12
+  conversion has a tool to run.
+- **The issuer exists and works:** ClusterIssuer `letsencrypt-prod` `READY=True` (28d), and the
+  existing `gateway/wildcard-home` Certificate is `Ready`/not-expired, which is what makes a DNS01
+  issuance for a new name in this zone a reasonable expectation.
+- **The renderer's substitution guard caught a real defect in my own work**, which is recorded
+  because it is the second time this class of bug has appeared here: my hook's explanatory comment
+  contained brace-wrapped names, and `assert_substitution_safe` refused to emit
+  (`'PX' is not a declared cluster-vars key — envsubst will replace it with the EMPTY STRING`).
+  Reworded; the rendered hook now contains exactly one brace-wrapped name, `BASE_DOMAIN`, verified
+  by parsing the emitted manifest.
+- **A `set -u` bug I introduced and then removed.** With `optional: true` the TLS env vars are
+  genuinely UNSET when no certificate has issued; under `set -u` the first `[ -n ... ]` would abort
+  the hook, and a postStart hook exiting non-zero makes kubelet KILL the container — the exact
+  failure `optional` exists to prevent. Verified by parsing the rendered manifest that no `set -u`
+  **command** remains (only two mentions in comments).
+- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
+  `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
+  **pass**; the rc 3 is the two known environmental skips (`gitea-settings` — no local
+  `SETTINGS_API_TOKEN`; `conformance` L2/L3 — `CLUSTER_AVAILABLE` unset). **Not green. Reported as
+  rc 3.** `l0` passing is the specific proof the schema edit admits `optional` and still validates
+  every catalog file; `render-golden` passing is the proof catalog and committed goldens agree.
+- **Only plex and pihole goldens moved.** No other service's manifest changed, and no service was
+  silently onboarded.
+
+## What I did NOT verify
+
+- **THE OUTCOME, ENTIRELY. Nothing in this branch has been observed working.** It is unreconciled:
+  no Certificate exists in prod, `plex-vip` still resolves to the Gateway VIP, `LanNetworksBandwidth`
+  is still absent, and the hook has NEVER RUN. Every claim above is about the inputs, not the result.
+- **Whether the certificate will actually ISSUE.** DNS01 for a new name in this zone is inferred from
+  the wildcard's success, not demonstrated. If the Cloudflare credential or solver is scoped in a way
+  that fails for this name, Plex will start (by design) with no TLS and the advertised `https://` URL
+  will be unservable — i.e. **this branch could leave the advertised URL broken in a new way**. First
+  reconcile must be watched.
+- **Whether `openssl pkcs12 -export` produces a bundle PLEX ACCEPTS.** OpenSSL 3 defaults to AES-256
+  + SHA-256, which some consumers reject. Untested against Plex. If it is rejected, Plex may fail to
+  load its TLS listener — the hook guards against a *missing* file, NOT against a present-but-
+  unreadable one. **This is the single largest unverified risk in the change.**
+- **Whether the certificate prefs take effect without a restart.** Plex may need to re-initialise its
+  listener after `customCertificatePath` changes. If so, the first reconcile applies the prefs and
+  the SECOND start serves the cert. Converges either way, but the first start may look wrong.
+- **Whether `LanNetworksBandwidth` actually flips the Remote badge.** It is the correct setting by
+  Plex's own documentation and the measured absence explains the symptom exactly, but I did not set it
+  live and observe a client reclassify.
+- **SNAT makes the LAN definition coarser than it looks.** With `externalTrafficPolicy: Cluster`, Plex
+  sees the ingress NODE's address for everything arriving via the VIP, so once `10.0.0.0/8` and
+  `192.168.0.0/16` are LAN, GENUINE REMOTE clients arriving that way will ALSO be classified local
+  and remote bandwidth limits will not bind on them. This is the metering consequence already
+  accepted on 2026-10-03, now made stronger rather than weaker. Not measured.
+- **The public Cloudflare record for `plex-vip` is off-repo and absent from this branch**, as is the
+  WAN tcp/32400 port-forward. Remote access cannot work until the operator creates both; nothing here
+  can verify either, and this repo has no off-LAN vantage point.
+- **The private key is exposed in the container's environment** (`/proc/1/environ`) because the schema
+  and renderer support no secret VOLUME — measured: `storage` accepts only
+  [class, export, mountPath, name, readOnly, size, subPath] with `additionalProperties: false`, and
+  no rendered app mounts a Secret. A volume would be the better shape; adding secret-volume support
+  is the clean follow-up and I did not do it.
+- **The PKCS#12 passphrase is a hardcoded literal** (`plexvip`) in the rendered manifest. It protects
+  nothing — Plex needs it in cleartext in `customCertificateKey` and it sits beside the bundle on the
+  same PVC — but it is worth knowing it is not a secret and was never treated as one.
+- **Four places must agree and nothing enforces it:** `ADVERTISE_IP`, the hook's
+  `customCertificateDomain`, `plex-cert.yaml`'s `dnsNames`, and PiHole's answer. No gate stage or
+  conformance test checks that they match, or that PiHole's `plex-vip` answer still equals
+  `${PLEX_L4_VIP}`.
@@ -0,0 +1,135 @@
+---
+date: 2026-10-05
+branch: docs/agents-work-in-their-own-workspace
+title: One workspace per agent run — the shared checkout is now documented as off-limits
+---
+
+## What changed
+
+Agents have been doing their work in the operator's checkout at
+`/var/home/keith/source/home-network-k3s-impl` and clobbering each other. Nothing in the tree
+told them not to: worktrees appeared in CLAUDE.md only as §3's *gate workarounds*, i.e. as an
+awkward edge case, and `decisions/README.md` asserted worktree isolation as a fact about how
+agents already behave rather than as a rule they must follow. Nowhere was the rule stated.
+
+Documentation only — no code, no manifests, no pipeline logic.
+
+- **`CLAUDE.md`: new `## 0. Work in your OWN workspace. Never in the shared checkout.`**
+  Placed before §1 so the section numbers every other file and handoff entry cites (§1, §2b,
+  §3, §7) all keep their meaning. It gives the three commands (`git fetch origin main`;
+  `git worktree add -b <type>/<slug> <path outside the repo> origin/main`; `cd`), an explicit
+  do-not list for the shared checkout (no `checkout`/`switch`, no edits, no
+  `add`/`commit`/`stash`/`restore`, no removing others' worktrees), a measurement table, the
+  "put it outside the repo root" rule, cleanup, what to do if you are already standing in the
+  shared checkout with edits, and what a PM must put in a brief it writes for another agent.
+- **`CLAUDE.md` intro** now says "Start with §0", with the reason: it is the rule you can
+  break before you have read anything else.
+- **`CLAUDE.md` §3** retitled "Running the gate from a worktree" and reframed as the normal
+  path rather than an edge case, since §0 now makes every gate run a worktree run. Its list
+  grew from two breakages to three: `spec-immutability`'s skip is *avoidable*, not merely
+  tolerated, if the `ENGINE` shim also mounts the sibling spec repo read-only and passes
+  `-e SPEC_REPO=/home-network-k3s`.
+- **`CLAUDE.md` §4** gained "One branch, one workspace, one PR".
+- **`README.md`**: a fourth non-negotiable rule ("No work in a shared checkout"), and
+  "Working on this repo" now opens with how to get out of the checkout before describing the
+  dev container that runs in it.
+- **`decisions/README.md`**: the worktree-isolation sentence now cites §0 as the requirement
+  instead of describing it as something agents happen to do.
+
+I did this work the way the new §0 requires — a worktree at
+`<session scratchpad>/wt-workspace-rule`, outside the repo root, branched from `origin/main`,
+with the shared checkout left on the branch and with the untracked files I found in it.
+
+## What I verified
+
+Every number in the new §0 table was measured in the shared checkout on 2026-10-05, not
+recalled:
+
+| Claim | Command | Value |
+|---|---|---|
+| shared HEAD is yanked between branches | `git reflog \| grep -c 'checkout: moving'` | **107** |
+| it is not sitting on `main` | `git worktree list` | `feat/pm-agent-issue-to-pr` |
+| other runs' uncommitted work is in it | `git status --porcelain` | **2** trees: `infra/agent/` (22:04), `platform/pm-agent/` (22:09) — `stat -c %y` |
+| its `main` is stale | `git rev-parse main` vs `origin/main` | `4386d14` vs `22c94ef`, **2 behind** |
+| worktrees registered / already merged | `git worktree list --porcelain` + `git merge-base --is-ancestor <b> main` | **42** branches, **31** merged |
+| nested worktrees are a gate hazard | `grep -rl '\.claude' conformance/tests platform/monitoring/tests` | **9** scripts carry an explicit exclusion |
+| the 18× double-count | read `platform/monitoring/tests/run.sh:115-123` | in-tree comment, observed 2026-09-12 |
+
+**A claim of mine was wrong and is corrected in the doc.** I had it recorded that
+`git worktree remove` is blocked by the agent sandbox. Provoked rather than assumed: created
+a throwaway detached worktree in the scratchpad and removed it — **rc 0**, and
+`git worktree list` no longer shows it. §0 therefore tells agents to try it, and the
+`rework/<pr-number>` escape hatch is now scoped to *someone else's* stale worktree holding a
+branch, which is a real and separate problem (31 of them).
+
+`spec-immutability` is avoidable-not-tolerated: taken from
+`decisions/2026-10-01-render-selftest-runs-in-gate.md`, which records
+`stage spec-immutability: pass` under the two-mount shim plus `-e SPEC_REPO=/home-network-k3s`.
+
+**Gate — `rc 3`, UNDETERMINED, and I am not calling it green.** Pinned toolbox, from this
+worktree, `DEV_KUBECONFIG=/dev/null` and an `ENGINE` shim bind-mounting both the main
+checkout's `.git` (at its own host path) and the sibling spec repo read-only with
+`-e SPEC_REPO=/home-network-k3s`:
+
+    ENGINE=<two-mount shim> DEV_KUBECONFIG=/dev/null       ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'   # rc 3
+
+| stage | result |
+|---|---|
+| spec-immutability | **pass** — it genuinely RAN; the second mount is what makes it run from a worktree |
+| approval-policy | pass, but **N/A**: `event='none'`. It cannot see a PR locally, so this green says nothing. See below |
+| decisions-log | pass — the three required headings and the frozen-archive pin |
+| lint · l0 · render-golden · l1 | pass (L1: 3 offline checks, 0 failures, 0 skips, 0 unwired) |
+| gitea-settings | **SKIPPED** — `SETTINGS_API_TOKEN` unset; it exists only in CI |
+| conformance (L2/L3) | SKIPPED — `CLUSTER_AVAILABLE!=true`; the decision notes no cluster-needing requirement was touched |
+
+`gitea-settings` is the *only* stage that did not run, and it cannot be made to run from a
+workstation, so rc 3 is the ceiling for any local run — not a defect in this branch. The
+first gate attempt returned rc 3 for a different and real reason (`HEAD is not ahead of
+origin/main`: I had not committed yet, so the touched-requirement set was empty by default
+rather than by verification); it was re-run after committing, and the table above is the
+post-commit run. §3 now records both expected skips, because it previously named only
+`spec-immutability`.
+
+## What I did NOT verify
+
+- **GATE: UNVERIFIED at the time of writing.** The pinned-toolbox result for this branch and
+  its exit code are reported in the PR, not predicted here. **A local rc 3 is not green** and
+  will not be described as one.
+- **CI gate: RED, measured — `MERGE GATE: BLOCKED`, one failing stage.** Run 245 on PR #56,
+  head `b988da8b`, 1m47s. The only failure is `approval-policy`: *"a sensitive change lacks a
+  non-author approval"*, `sensitive paths touched: CLAUDE.md`, author `gitea-admin`. Every other
+  stage passed — `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, and `gitea-settings`
+  (which can only run in CI, and reported `required_approvals = 0`,
+  `dismiss_stale_approvals = true`). `spec-immutability` SKIPPED in CI (no sibling spec repo in
+  the CI workspace — the inverse of the local run, where the two-mount shim made it pass) and
+  `conformance` SKIPPED. **Note `gate / conformance (pull_request)` reports status `success`
+  with description "Has been skipped"** — CLAUDE.md §2b's trap, live on this very PR. Nothing
+  in this branch is broken; the red is the structural human gate, below. *Measured at
+  `b988da8b`; this entry was then amended to record it, so the head SHA in the PR is newer than
+  the run quoted here and CI re-ran on the amended commit.*
+- **This PR is a sensitive-path PR and is CI-RED on `approval-policy` until a human acts.**
+  `CLAUDE.md` is in `infra/gitea/approval-policy.json`'s `sensitive_paths` ("the delivery
+  contract agents follow"), so the stage requires an approving review from someone who is not
+  the author, or an `/approve-override <reason>` posted by an administrator **as a PR review
+  of type "Comment"**, not in the plain comment box. `ci/approval-policy.sh` run locally
+  reports N/A (it cannot see a PR), so the local gate cannot catch this and a green local run
+  says nothing about it. **I have not posted an override and must not** — CLAUDE.md §1 forbids
+  an agent clearing this on its own PR even though `gitea-admin` holds `owner` and could.
+- **Nothing here is enforced.** §0 is prose. There is no gate stage that detects work done in
+  the shared checkout, no check that a worktree is outside the repo root, and no check that a
+  branch was cut from `origin/main`. Every other rule in CLAUDE.md that agents kept breaking
+  (`DECISIONS-LOG.md` appends, the three handoff headings) ended up as a gate stage, and that
+  is the honest follow-up here. I did not write one: the ask was documentation, and an
+  enforcement stage lands in `ci/**`, which is itself a sensitive path and deserves its own PR.
+- **I did not clean up the 31 stale merged worktrees or the 35 nested under
+  `.claude/worktrees/`.** They are other sessions' workspaces; removing them is an operator
+  action, and §0 explicitly tells agents not to remove worktrees they did not create.
+- **I did not touch the untracked `infra/agent/` and `platform/pm-agent/` trees** sitting in the
+  shared checkout. They are another run's in-flight work. Whose, and whether they are wanted,
+  is unknown to me.
+- **`docs/plans/ssd-etcd-execution-runbook.md:18` still points at a `.claude/worktrees/` path**
+  inside the repo. Left as-is deliberately: that runbook is marked COMPLETED/historical and
+  describes the now-decommissioned dev cluster, so the path is a record of what was done, not
+  an instruction. It does mean a grep for in-repo worktree paths is not yet clean.
+- **No runtime, cluster or Flux behaviour was touched or checked.** This branch changes three
+  Markdown files; `kubectl` was not run.
@@ -0,0 +1,88 @@
+---
+date: 2026-10-05
+branch: feat/cilium-roll-out-on-config-change
+title: Cilium config changes were silently inert — roll the agents when cilium-config changes, which also finally activates the Hubble exporter
+---
+
+## What changed
+
+**A merged Cilium config change did nothing, and everything reported success.** The Hubble flow
+exporter (PR #50, merged 2026-10-04) wrote its settings into `cilium-config` and never took effect.
+
+| Measured 2026-10-05, over a day after the merge | |
+|---|---|
+| HelmRelease `cilium` | `Ready=True`, `UpgradeSucceeded`, *"Helm upgrade succeeded for release kube-system/cilium.v6"* |
+| `cilium-config` keys | `hubble-export-file-path: stdout`, `hubble-export-allowlist: {"destination_port":["32400"]} {"source_port":["32400"]}` — **all present** |
+| cilium agent pods | **all three still the PRE-MERGE pods**, `startTime 2026-10-04T07:11Z` |
+| exported flow lines in agent stdout | **0, 0, 0** |
+| Flux Kustomizations | all `Ready=True` |
+
+Cause: the chart's `rollOutCiliumPods` defaults to **`false`** (cilium 1.20.1 `values.yaml`: *"Roll
+out cilium agent pods automatically when configmap is updated"*), so Helm rewrites the ConfigMap and
+leaves the DaemonSet's pod template untouched — and **cilium-agent reads that ConfigMap only at
+startup**. The feature was merged, believed delivered, green everywhere, and doing nothing at all.
+
+That is the same class of failure this repo already guards against elsewhere: a green signal sitting
+over something that never ran.
+
+- **`platform/cilium/helmrelease.yaml`** (authoritative): `rollOutCiliumPods: true`.
+- **`platform/cilium/values.yaml`** (reference mirror) and
+  **`infra/ansible/roles/cilium/templates/cilium-values.yaml.j2`** (the `[parity]` source of truth):
+  the same, so a bootstrap render does not differ from what Flux reconciles.
+
+**The cost is real and is the point.** Every future Cilium config change now triggers a cluster-wide
+agent rollout instead of waiting for someone to remember a manual restart. That is strictly more
+honest — the change applies when merged, or it fails visibly — but it makes config edits disruptive
+by default, so they want a quiet moment. Mitigation already in place:
+`updateStrategy.rollingUpdate.maxUnavailable: 1` means the DaemonSet will not proceed past an unready
+agent, so a bad render costs one node rather than the cluster.
+
+**Merging this is itself one rollout**, because adding the ConfigMap checksum annotation changes the
+pod template — and that rollout is what finally activates the Hubble exporter.
+
+## What I verified
+
+- **Every row of the table above**, read from the live HelmRelease status, the live `cilium-config`
+  ConfigMap, the agent pods' `startTime`, and a grep for `"flow"` in each agent's stdout.
+- **The chart value name, default and semantics**, quoted from cilium **v1.20.1**'s own
+  `values.yaml` — the version actually running here (`quay.io/cilium/cilium:v1.20.1`), not a guess
+  from another release: `rollOutCiliumPods: false` with the comment *"Roll out cilium agent pods
+  automatically when configmap is updated."*
+- **All three value files carry the key** and both YAML files still parse, with
+  `spec.values.rollOutCiliumPods` resolving to `true`.
+- **The blast-radius mitigation is genuinely present**, not assumed:
+  `updateStrategy.rollingUpdate.maxUnavailable: 1` is already set in the HelmRelease, with a long
+  existing comment explaining that a CNI rollout has no healthy-peer fallback.
+- **THE MECHANISM IS CONFIRMED BY CONTRAST, not just reasoned.** The cgroup fix (PR #42,
+  2026-10-03) DID take effect: both the DaemonSet and the running pod mount
+  `cilium-cgroup hostPath=/sys/fs/cgroup`, matching what the HelmRelease declares. The difference is
+  exactly the one claimed above — #42 changed the POD TEMPLATE (a volume hostPath), which rolls the
+  DaemonSet on its own, while #50 changed ONLY the ConfigMap, which does not. It also explains the
+  agents' `startTime 2026-10-04T07:11Z`: that was #42's rollout, and nothing has rolled them since.
+  So the inertness is specific to ConfigMap-only changes, which is precisely what this flag fixes.
+- **Flux is healthy before this change** — every Kustomization `Ready=True`, so this lands on a clean
+  graph rather than into an existing cascade.
+- **Toolbox gate**: recorded in the PR.
+
+## What I did NOT verify
+
+- **THE OUTCOME, ENTIRELY. Unreconciled.** No agent has rolled, no flow has been exported. This
+  change is an argument that the rollout will happen, not evidence that it did.
+- **Whether `filePath: stdout` actually works** — still the open question from #50, and it cannot be
+  answered until the agents restart. If the literal is not special-cased, the agent may create a
+  FILE named `stdout` and nothing reaches Loki: a silent no-op, not a crash. Fallback documented in
+  #50: keep the chart's default path and add a `loki.source.file` target to Alloy with a hostPath
+  mount. **Check for flow lines in the agent logs immediately after the rollout.**
+- **THE ROLLOUT'S IMPACT ON LAN DNS IS UNMEASURED AND IS THE MAIN RISK.** The node holding the L2
+  lease for the Gateway VIP, **PiHole's DNS VIP** and Plex's VIP loses them briefly as it cycles.
+  `maxUnavailable: 1` bounds it to one node at a time, and BPF maps persist across an agent restart,
+  but I have not measured how long a VIP is unreachable during a lease move, nor whether the two
+  PiHole replicas cover the gap. Merge it at a quiet moment.
+- **Whether enabling this is the operator's preferred trade.** The alternative — leaving it `false`
+  and restarting by hand — keeps config edits non-disruptive at the cost of them being silently
+  inert. I judged silent inertness the worse failure, but it is a judgement, not a measurement.
+- **Whether anything ELSE in `cilium-config` is currently inert.** I checked the one other recent
+  Cilium change and it is fine (see below), but I did not audit every key against the running
+  agents' actual state.
+- **`hubble.relay.rollOutPods` / `hubble.ui.rollOutPods`** are left at their `false` defaults. Both
+  have the same silently-inert property for their own ConfigMaps; not addressed here.
@@ -0,0 +1,90 @@
+---
+date: 2026-10-05
+branch: docs/oauth2-sidecar-scrape-verified
+title: The gate-sidecar scrape is verified working on prod — and four places in the tree still said it wasn't
+---
+
+## What changed
+
+Documentation only. No manifest, no rendered output, no behaviour.
+
+PR #49 merged as `21c88b0` and the fix works, but four places in the tree still asserted the
+*pre-fix* state as current — in the present tense, in capitals, in the files a reader would consult
+first. This is the stale-documentation failure mode this repo has been burned by twice already
+(`git push origin main` in the runbooks, "append to DECISIONS-LOG.md" in the old handoffs), so:
+
+- **`platform/oauth2-proxy/podmonitor.yaml`** — the banner read *"READ THIS BEFORE BELIEVING THE
+  TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY"* and *"SO THE TARGETS BELOW WILL
+  READ `up == 0` UNTIL AN INGRESS ALLOWANCE LANDS"*. Both now false, and the first one actively
+  instructs the reader to disbelieve a correct `up == 1`. Replaced with a `STATUS: THE SCRAPE WORKS`
+  block carrying the post-fix numbers; the diagnosis is kept below it under an explicit `HISTORY`
+  heading, because it is why the fix is shaped the way it is. Added the drop fingerprint to
+  recognise if it regresses — `lastScrapeDuration` pinned at *exactly* the `scrapeTimeout` with
+  `context deadline exceeded`, versus 1–6 ms when healthy.
+- **`catalog/render.py:159`** — *"THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED … Do not
+  read this flag as 'metrics are working'."* Rewritten to enumerate the three pieces a working
+  scrape needs and which PR landed each. The warning is **kept, deliberately**: the flag alone
+  really is insufficient, and "1 without 3" is exactly what caused the 63.7h outage. Only the
+  "not yet scraped" claim was false.
+- **`catalog/render.py:1365`** — the sidecar arg comment, same correction, one line.
+- **`platform/oauth2-proxy/README.md`** — its two observability subsections are *dated*
+  ("measured 2026-09-30", "Update 2026-10-01"), so they are honest records rather than lies. But a
+  reader hits the stale body long before any correction, so a `> SUPERSEDED` pointer now sits
+  directly under the heading, and an `### Update 2026-10-05` section at the end carries the
+  before/after table, the TSDB series counts and the alert outcome.
+
+Two substantive errors fixed while there, not just tense:
+
+1. The README's *"To close the gap for an app, three things are needed"* listed **a Service port
+   and a ServiceMonitor**. Wrong — a PodMonitor targets pod ports directly, which is the whole
+   reason no Service exposes 44180 and the renderer needed no Service change. Following that
+   paragraph would have sent someone to edit `catalog/render.py`'s Service rendering for no reason.
+2. `podmonitor.yaml` ended by deferring the fix to *"the catalog/renderer, which is in flight on
+   another branch"* — that branch does not exist and never did; the work happened in PR #49. It now
+   names `r_monitoring_ingress_policy` directly, and a new `CRAFTY` section explains why crafty is
+   the one namespace this file cannot help.
+
+## What I verified
+
+- **The fix itself, on prod 2026-10-05, read-only** (this is the evidence the docs now quote):
+  Flux applied `21c88b0` through the real path — `apps-wave-0` `lastAppliedRevision` at 21:37:01Z,
+  `apps-wave-1` at 21:37:46Z, in dependency order — and all five `<app>-allow-monitoring-scrape`
+  CNPs exist, each `ports=['44180']`, `from=[['monitoring']]`.
+- **All 6 targets `up == 1`**, `health=up`, `lastError=''`, `lastScrapeDuration` 0.001–0.006s —
+  against 10.0s and `context deadline exceeded` before.
+- **`oauth2_proxy_*` is in the TSDB for these namespaces for the first time:** 5 metric names,
+  148 series; `sum by (namespace) (oauth2_proxy_requests_total)` = qbittorrent 35072, radarr 23560,
+  sonarr 1284. (Queried with `count by (__name__)` plus a matcher, not label-values.)
+- **Alerts:** all 5 `TargetDown` and 5 of 6 `OAuth2ProxyGateUnobserved` resolved; one series
+  remains, `OAuth2ProxyGateUnobserved{namespace="crafty"}`, firing.
+- **The rendered output is byte-identical.** `catalog/render.py` changed only in comments, so this
+  branch touches no `apps/` file and no golden. Confirmed by `render-golden` and `l1` both passing
+  with no regeneration, and by `git status` showing nothing under `apps/`.
+- **This branch touches no sensitive path**, unlike PR #49: `platform/oauth2-proxy/**`,
+  `catalog/**` and `decisions/**` are none of them in `infra/gitea/approval-policy.json`. So it
+  should merge on a green gate with no human approval — the opposite of the PR it documents, and
+  the reason the two were kept apart.
+- `render.py --selftest`: PASS.
+- Toolbox gate: see the handoff's own rule — reported below rather than claimed here.
+
+## What I did NOT verify
+
+- **I did not re-measure the 63.7h figure or the ~22 emails/day rate**; both are carried forward
+  from the 2026-10-04 measurements, now quoted inside the docs as historical. The 15-day
+  `query_range` that established "one episode, not flapping" was run once, on 2026-10-04.
+- **`oauth2_proxy_requests_total` is still absent for jackett and pihole.** I assert in the README
+  that this is because the counter is not created until the gate serves its first request. That is
+  the documented behaviour of a Prometheus counter with no initialisation, and it is consistent
+  with the other three namespaces having traffic — but I did not force a request through jackett's
+  gate and watch the series appear. If it is still absent after real use, that is a finding.
+- **The auth-churn rule is still unwritten.** The README now says it is "writable rather than
+  decorative"; that is a claim about the metrics being present, not a rule anyone has authored or
+  unit-tested. The thing the 2026-09-30 incident actually needed does not exist yet.
+- **crafty remains unobserved.** I read its container spec on 2026-10-05 and confirmed no
+  `--metrics-address`; I did not attempt the fix, and nothing here changes its alert.
+- **I did not audit the whole tree for other stale "metrics are not scraped" claims** beyond the
+  four sites a targeted grep found (`not yet reachable`, `UNTIL AN INGRESS`, `up == 0`,
+  `not reachable by Prometheus` and neighbours). `runbooks/oauth2-proxy-auth-failures.md` is
+  described in the README as "the by-hand substitute meanwhile" and I left it alone — the by-hand
+  log procedure still works and is not wrong, but whether it should now be rewritten against the
+  new metrics is unexamined.
@@ -0,0 +1,105 @@
+---
+date: 2026-10-05
+branch: fix/plex-resolver-independent-advertise
+title: Plex down LAN-wide — the advertised name existed only in PiHole, so clients that bypass it landed on the router's %D cert
+---
+
+## What changed
+
+**I took Plex down for the whole LAN.** Reported symptom: every internal client showed a certificate
+error with `%D` as the subject name and could not connect.
+
+**Root cause: the advertised connection depended on our own resolver.** `ADVERTISE_IP` was
+`http://plex-vip.${BASE_DOMAIN}:32400` — a name that exists ONLY in PiHole's split-horizon answers
+and in no public zone. A Plex client resolves the advertised host *itself*, with whatever resolver it
+happens to use, and on a real LAN that is frequently not PiHole: browsers ship DoH on by default,
+phones have "private DNS", guest/IoT devices take the router's DNS. Those clients could not resolve
+the name, the SOHO router answered the failed lookup with **itself**, and they all landed on the
+router's web UI.
+
+| Measured 2026-10-05 | Result |
+|---|---|
+| `192.168.2.1:443` (router, LAN side) | `C = ZZ, ST = Somewhere, O = %D342e7e6b, CN = %D` |
+| `136.51.115.252:443` (router, WAN side) | the same `%D` cert |
+| PiHole **pihole-0** | `plex -> 192.168.2.20`, `plex-vip -> 192.168.2.24` ✅ |
+| PiHole **pihole-1** | identical ✅ (both behind the `.23` VIP) |
+| Gateway `.20:443` cert | `CN = *.home.keithjackson.dev`, valid to 2026-12-12 ✅ |
+| `http://192.168.2.24:32400/identity` | `200` ✅ |
+| Plex's published connections | all port **32400** — nothing on 443 |
+
+**The DNS was right; the dependency on it was the bug.** PiHole answered correctly from both
+replicas the entire time. And because nothing Plex advertises listens on 443, the clients had plainly
+left the Plex path altogether and were talking to the router.
+
+**The fix: go back to `<dashed-ip>.<serverhash>.plex.direct`, which is RESOLVER-INDEPENDENT.** This
+is Plex's own mechanism and the reason it exists — `plex.direct` is a public zone that returns the
+private address embedded in the label, so *every* resolver (PiHole, the router, 1.1.1.1, a DoH
+browser) answers `192-168-2-24.<hash>.plex.direct` with `192.168.2.24`. It needs no split-horizon
+entry, no Cloudflare record and no certificate work, because the wildcard cert Plex already serves on
+32400 covers exactly that name.
+
+- **`catalog/plex.yaml`:** `ADVERTISE_IP` →
+  `https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400`, with the
+  resolver-independence requirement written at the top of the block as the first thing to check
+  before anyone changes this field again.
+- **PiHole's `plex-vip` answer is KEPT.** It is harmless, costs nothing, and is useful for reaching
+  the VIP by name from a LAN shell. It is simply not what Plex advertises.
+
+**Why not keep the nicer hostname.** It cannot be made correct for clients that bypass PiHole: a
+public record for it resolves to the WAN address, and with no NAT hairpin on this router a LAN client
+sent there fails too. The name is right only for PiHole users and wrong for everyone else, and there
+is no configuration of it that is right for both. `plex.direct` is right for both.
+
+**The honest cost, restated rather than buried:** the value embeds both the VIP and the server's
+certificate hash as a literal — envsubst cannot dash an IP and nothing in the tree can derive the
+hash, so nothing checks that they still agree. That fragility is exactly why the hostname was tried.
+It is accepted because an unresolvable name is a *worse* failure: silent, LAN-wide, and it points
+users at their router.
+
+## What I verified
+
+Live prod, 2026-10-05, read-only.
+
+- **Every row of the table above**, including the router's `%D` cert on both its LAN and WAN sides —
+  which is what identifies the router as what the clients were reaching.
+- **THE FIX ITSELF, with full TLS validation (no `curl -k`)**: the plex.direct name resolves to
+  `192.168.2.24` and `/identity` returns **200, TLS verified**. Re-checked today, not carried over
+  from yesterday's measurement.
+- **Both PiHole replicas are identical and correct**, with the same five `address=` lines live in
+  `/etc/pihole/dnsmasq.conf`, and both IPs present behind the `pihole-dns-tcp`/`-udp` endpoints. I
+  checked only pihole-0 at first and deliberately went back for pihole-1, because a split-brain pair
+  would have produced exactly this intermittent-looking symptom.
+- **The cluster was never the problem:** all eight LoadBalancer VIPs still assigned, L2
+  announcements intact (prod-3 holds `.20`/`.23`/`.24`), `wildcard-home` Certificate `Ready=True`,
+  and PR #51 had reconciled (`customConnections="http://plex-vip.home.keithjackson.dev:32400"` was
+  live, confirming the bad value was in effect).
+- **`LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16"` is still set**, so the postStart hook survived
+  the roll and keeps re-asserting.
+- **The Cilium rollout from #50 STILL has not happened** — agents are unchanged since
+  2026-10-04T07:11, so the Hubble export is inactive and `filePath: stdout` remains unproven.
+
+## What I did NOT verify
+
+- **THE OUTCOME. Unreconciled.** No client has been observed connecting after this change. The
+  argument is that the advertised name now resolves for every resolver and serves a matching
+  certificate — both measured — but I have not seen a Plex client complete a connection, and I
+  cannot see the user's clients at all.
+- **THE CLIENT-SIDE RESOLVER PATH IS INFERRED, NOT OBSERVED.** I could not query a public resolver
+  from here — my attempt returned NXDOMAIN even for a `plex.direct` name that certainly resolves, so
+  the probe was broken (egress policy or a missing tool) and I discarded it rather than report it.
+  The chain "client bypasses PiHole → NXDOMAIN → router answers with itself → `%D`" is the only
+  explanation consistent with all the measurements, but the NXDOMAIN-hijack step in particular is
+  deduced from the router serving `%D` on its LAN address, not from watching a lookup.
+- **Which clients bypass PiHole, or how many.** If in fact every client uses PiHole, then my
+  explanation is wrong and something else sent them to the router — in which case the next thing to
+  check is whether the clients were using `plex.home.keithjackson.dev` (port 443, the Gateway) rather
+  than an advertised connection at all.
+- **Whether plex.tv will re-publish promptly** after the value changes, and whether stale connection
+  lists are cached on clients. A client holding the old list may keep failing until it refreshes.
+- **Mixed content.** The previous `http://` value would also have been blocked by the browser for
+  anyone using `https://app.plex.tv`; that is fixed incidentally by returning to `https`, but I did
+  not test it.
+- **The Remote badge — the symptom that began this whole sequence — remains unconfirmed.**
+  `LanNetworksBandwidth` is set, but Plex still tags traffic `Allowed Network (WAN)`. Unchanged and
+  still unexplained.
+- **`ManualPortMappingMode="1"` with an absent port** is still there, untouched.
@@ -26,7 +26,8 @@ The fix is the obvious one for concurrent appenders: **stop sharing a file**. Ea
 handoff is its own file. Two agents writing handoffs touch disjoint paths, so there is
 nothing for git to merge and nothing to conflict. This scales to any number of parallel
 branches with zero coordination — which is the point, since the agents are deliberately
-isolated from each other in separate worktrees and cannot coordinate.
+isolated from each other in separate worktrees (required by `CLAUDE.md` §0) and cannot
+coordinate.

 There is deliberately **no index file** and no counter. An index would be a shared
 mutable file — exactly the thing that just failed — and a sequence number would have
@@ -39,23 +39,46 @@ bpf:
  masquerade: true

 # Flow observability / isolation audit (ADR-0002).
+# Roll the agents when cilium-config changes. Kept EQUAL to the Flux values
+# (platform/cilium/helmrelease.yaml), which carry the full rationale: the chart default of false
+# leaves the pod template untouched on a config change, and cilium-agent reads that ConfigMap only
+# at startup, so the change is silently inert until someone restarts the DaemonSet by hand.
+rollOutCiliumPods: true
+
 hubble:
  enabled: true
  relay:
    enabled: true
  ui:
    enabled: true
+  # Static flow export -> cilium-agent stdout. Kept EQUAL to the Flux values
+  # (platform/cilium/helmrelease.yaml), which carry the full rationale: it records the PRE-SNAT
+  # client address for Plex's raw tcp/32400, which Plex itself cannot log because every
+  # LoadBalancer Service here is externalTrafficPolicy: Cluster and the VIP lease holder SNATs the
+  # client. Present here so a BOOTSTRAP render does not differ from what Flux reconciles -- a
+  # mismatch would show up as a permanent Helm diff and an avoidable agent rollout. Scoped by PORT,
+  # not namespace: the flow carrying the real client address does not match a namespace filter.
+  export:
+    static:
+      enabled: true
+      filePath: stdout
+      allowList:
+        - '{"destination_port":["32400"]}'
+        - '{"source_port":["32400"]}'

 # Ingress VIP announcement on the NAT LAN. IP pool / CiliumL2AnnouncementPolicy
 # (from the reserved 10.73.0.240-.250 range) is authored by WS-NET in Phase 2.
 l2announcements:
  enabled: true

-# k3s ships its own cgroup mount; point Cilium at it instead of auto-mounting.
+# Point Cilium at the host's real cgroup2 root (/sys/fs/cgroup, mounted by systemd) instead of
+# auto-mounting. NOT /run/cilium/cgroupv2: nothing mounts cgroup2 there, so the agent mounted its
+# OWN namespaced cgroup and socket-LB covered only itself — host processes (the apiserver) lost
+# Service/LB translation. [parity] platform/cilium/helmrelease.yaml explains (2026-10-03).
 cgroup:
  autoMount:
    enabled: false
-  hostRoot: /run/cilium/cgroupv2
+  hostRoot: /sys/fs/cgroup

 # Roll the cilium agent DaemonSet ONE node at a time. [parity] must match the
 # platform/cilium HelmRelease. The cilium 1.20.1 chart defaults this to 2, which on a
@@ -101,20 +101,124 @@ spec:
        enabled: true
      ui:
        enabled: true   # exposure gated behind oauth2-proxy forward-auth later (ADR-0014)
+      # --- STATIC FLOW EXPORT: persist the PRE-SNAT client address for Plex (2026-10-04) ---
+      # [parity] MUST equal Ansible: hubble.export.static.{enabled,filePath,allowList}
+      #
+      # WHY THIS EXISTS. Plex's own log cannot show who connected. Every LoadBalancer Service in
+      # this cluster is `externalTrafficPolicy: Cluster`, so the node holding the L2 lease for the
+      # VIP SNATs the client before forwarding to the backend pod. Measured 2026-10-04: Plex had
+      # logged 3,214 requests from `10.0.1.x` (the lease holder k3s-prod-3's own address, Cilium
+      # identity 1 = reserved:host) and ZERO from any 192.168.x address, ever. No X-Forwarded-For
+      # either -- it serves raw tcp/32400, not HTTP through the Gateway.
+      #
+      # THE ADDRESS IS NOT LOST, ONLY UNRECORDED. Cilium observes the flow at ingress, BEFORE its
+      # own translation, so Hubble already has the true source. Measured on the lease holder:
+      #     192.168.2.133 -> 10.0.0.135   (real LAN client -> the Plex pod)
+      #     10.0.1.78     -> 10.0.0.135   (the same traffic, post-SNAT -- what Plex sees)
+      # What was missing is RETENTION: the ring buffer was 4095/4095 flows at ~303 flows/s, i.e.
+      # about THIRTEEN SECONDS of history, with hubble metrics disabled and no exporter configured.
+      #
+      # WHY NOT FIX THE SNAT INSTEAD. `externalTrafficPolicy: Local` would preserve the address in
+      # Plex's own log, but cilium 1.20's L2 announcer is explicitly incompatible with it -- upstream
+      # docs for this exact version: "incompatible with the externalTrafficPolicy: Local on services
+      # as it may cause service IPs to be announced on nodes without pods causing traffic drops",
+      # and "L2 announcements isn't currently aware of this setting". Live proof that this is not
+      # theoretical: the VIP is announced by k3s-prod-3 while the pod runs on k3s-prod-1, so the
+      # flip would black-hole tcp/32400 immediately. Making it safe would mean pinning BOTH the pod
+      # and the announcement to one node and giving up Plex's ability to reschedule -- a real
+      # availability loss to buy a log field. (It is the same trap platform/crafty/service-l4.yaml
+      # documents from an 8-day Minecraft outage.) DSR was also rejected: routing-mode here is
+      # vxlan, not native/geneve, and DSR returns bypass the ingress node, which breaks the SOHO
+      # router's stateful NAT on the remote path.
+      #
+      # `filePath: stdout` RATHER THAN A FILE, deliberately. Grafana Alloy already runs as a
+      # DaemonSet tailing every pod's logs into Loki (platform/monitoring/alloy.yaml), so stdout
+      # reaches Loki with NO new hostPath mount, no file-rotation handling and no Alloy change.
+      # Upstream documents it: "To get Hubble flows directly exported to the logs instead of written
+      # to a rotated file, `stdout` can be defined as `hubble-export-file-path`." fileMaxSizeMb /
+      # fileMaxBackups are therefore left at chart defaults and do not apply.
+      #   Query:  {namespace="kube-system", app="cilium"} |= "32400"
+      #
+      # SCOPED BY PORT, NOT BY NAMESPACE -- and that is a measured decision, not a preference. A
+      # `--to-namespace plex` query returned ONLY the post-SNAT flows; the flow carrying the real
+      # client address did NOT match it, because at the point of observation the destination
+      # identity is not yet resolved to the plex pod. Filtering by namespace would therefore drop
+      # exactly the record this change exists to capture. Port it is.
+      #
+      # Both directions are listed so the return path is captured too. Volume is small (a few
+      # hundred to a few thousand lines a day); if it ever becomes noisy, narrow it with
+      # `fieldMask` rather than by removing a filter.
+      export:
+        static:
+          enabled: true
+          filePath: stdout
+          allowList:
+            - '{"destination_port":["32400"]}'
+            - '{"source_port":["32400"]}'
    # --- L2 announcement for the ingress VIP on the flat LAN (docs/09, OQ-1 resolved) ---
    # [parity] MUST equal Ansible: l2announcements.enabled: true. IP pool +
    # CiliumL2AnnouncementPolicy (reserved 10.73.0.240-.250) authored by WS-NET in Phase 2.
    l2announcements:
      enabled: true
-    # --- cgroup: k3s ships its own cgroup mount; point Cilium at it, don't auto-mount ---
-    # [parity] MUST equal Ansible cgroup block. Not in the deliverable's headline list, but it IS
-    # part of the working CNI config on k3s: reconciling without it lets Cilium try to auto-mount
-    # cgroupv2 and break on adopt. Kept in parity for a safe Phase-2 adoption.
+    # --- cgroup: point Cilium at the HOST's cgroup2 root, which systemd mounts at /sys/fs/cgroup ---
+    # [parity] MUST equal Ansible cgroup block (roles/cilium/templates/cilium-values.yaml.j2).
+    #
+    # FIXED 2026-10-03 — hostRoot WAS /run/cilium/cgroupv2, AND THAT SILENTLY BROKE SOCKET-LB FOR
+    # EVERY HOST PROCESS. With autoMount disabled, the chart only bind-mounts whatever already exists
+    # at hostRoot ("it's expected that users have mounted cgroup2 filesystem at the specified
+    # cgroup.hostRoot" — cilium 1.20.1 values.yaml). Nothing ever mounted cgroup2 at
+    # /run/cilium/cgroupv2 on these nodes (it is an empty DirectoryOrCreate hostPath), so the agent
+    # mounted one ITSELF from inside its own private cgroup namespace. That mount's root is the
+    # agent container's OWN leaf cgroup, so cil_sock4_connect & co. were attached there and covered
+    # only the agent. Every other host-netns process (the k3s apiserver in
+    # /system.slice/k3s.service, curl over SSH) was untranslated: Service ClusterIPs timed out from
+    # every node, and the node holding the Gateway's L2 lease could not reach the Gateway VIP at all
+    # ("no route to host"). That is why k3s-prod-1's apiserver logged `oidc: authenticator not
+    # initialized` from 2026-09-12 on — kubectl SSO through the kube-vip VIP failed outright.
+    # Measured on k3s-prod-1 before this change: TCP connect to 192.168.2.20:443, 10.43.225.79:443
+    # and 10.43.59.141:8080 SUCCEEDED from inside the agent container (the covered cgroup) and
+    # FAILED from the host, same node, same netns. Full write-up:
+    # decisions/2026-10-03-cilium-cgroup-hostroot.md.
+    #
+    # /sys/fs/cgroup IS a cgroup2 mount in the host's root cgroup namespace on every node (verified
+    # with findmnt on all three), so bind-mounting it gives the agent the real host root. autoMount
+    # stays off because the mount it would create already exists.
    cgroup:
      autoMount:
        enabled: false
-      hostRoot: /run/cilium/cgroupv2
+      hostRoot: /sys/fs/cgroup
    # --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident ---
+    # --- Roll the agents when their ConfigMap changes (2026-10-05) ---
+    # [parity] MUST equal Ansible: rollOutCiliumPods: true
+    #
+    # WHY: a merged Cilium config change DID NOTHING. The chart default is `false`
+    # (cilium 1.20.1 values.yaml: "Roll out cilium agent pods automatically when configmap is
+    # updated. rollOutCiliumPods: false"), so Helm rewrites `cilium-config` and leaves the
+    # DaemonSet's pod template untouched — and cilium-agent reads that ConfigMap only at STARTUP.
+    # The result is a silently inert change: Flux green, HelmRelease `UpgradeSucceeded`, the new
+    # keys visibly present in the ConfigMap, and the running agents still on the old settings
+    # until a human remembers to restart them.
+    #
+    # MEASURED, and it is why this exists: the Hubble flow exporter merged 2026-10-04 wrote
+    # `hubble-export-file-path: stdout` and its allowlist into `cilium-config`, the HelmRelease
+    # reported `Helm upgrade succeeded ... cilium.v6`, and over a DAY later all three agents were
+    # still the pre-merge pods (startTime 2026-10-04T07:11Z) with ZERO exported flow lines. The
+    # feature was merged, believed delivered, and doing nothing at all. That is the same class of
+    # failure this repo already documents elsewhere — a green signal over something that never ran.
+    #
+    # THE COST IS REAL AND IS THE POINT: every future Cilium config change now triggers a
+    # cluster-wide agent rollout instead of waiting for a manual restart. That is strictly more
+    # honest — the change applies when it is merged, or it fails visibly — but it means config
+    # edits are disruptive by default, so they want a quiet moment. One node at a time:
+    # `updateStrategy.rollingUpdate.maxUnavailable: 1` below means the DaemonSet will not proceed
+    # past an unready agent, so a bad render costs one node rather than the cluster. Note that the
+    # node holding the L2 lease for the Gateway, PiHole-DNS and Plex VIPs loses them briefly as it
+    # cycles, so a rollout can blip LAN DNS.
+    #
+    # ENABLING THIS IS ITSELF ONE ROLLOUT: adding the ConfigMap checksum annotation changes the pod
+    # template, so merging this rolls the agents once — which is what finally activates the Hubble
+    # exporter that has been inert since 2026-10-04.
+    rollOutCiliumPods: true
    # [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1.
    # The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate.
    # maxUnavailable: 2) — NOT a local choice, and on a 3-node cluster that lets a single bad
@@ -32,20 +32,43 @@ enableIPv4Masquerade: true
 bpf:
  masquerade: true
 # [parity] flow observability / isolation audit
+# [parity] roll the agents when cilium-config changes. The chart default (false) rewrites the
+# ConfigMap but leaves the pod template alone, and cilium-agent only reads it at STARTUP — so a
+# merged config change is SILENTLY INERT. Measured 2026-10-04..05: the Hubble exporter keys were
+# live in cilium-config and the HelmRelease said UpgradeSucceeded, while all three agents were
+# still pre-merge pods exporting nothing for over a day. Cost: config edits now roll the agents
+# (one node at a time via maxUnavailable: 1), which can briefly blip the VIPs on the lease-holding
+# node. Full rationale in helmrelease.yaml.
+rollOutCiliumPods: true
+
 hubble:
  enabled: true
  relay:
    enabled: true
  ui:
    enabled: true
+  # [parity] static flow export -> cilium-agent stdout -> Alloy -> Loki. Records the PRE-SNAT
+  # client address for Plex's raw tcp/32400, which Plex's own log cannot show (every LB Service here
+  # is externalTrafficPolicy: Cluster, so the VIP lease holder SNATs the client). Scoped BY PORT,
+  # not namespace: the flow carrying the real client IP does not match a namespace filter. Full
+  # rationale, measurements and the rejected alternatives are in helmrelease.yaml.
+  export:
+    static:
+      enabled: true
+      filePath: stdout
+      allowList:
+        - '{"destination_port":["32400"]}'
+        - '{"source_port":["32400"]}'
 # [parity] ingress VIP announcement on the flat LAN (docs/09, OQ-1)
 l2announcements:
  enabled: true
-# [parity] k3s provides the cgroupv2 mount; do not auto-mount
+# [parity] the host's cgroup2 root is /sys/fs/cgroup (systemd mounts it); do not auto-mount.
+# NOT /run/cilium/cgroupv2: nothing mounts cgroup2 there, so socket-LB attached to the agent's own
+# cgroup and host processes lost Service/LB translation (2026-10-03 — see helmrelease.yaml).
 cgroup:
  autoMount:
    enabled: false
-  hostRoot: /run/cilium/cgroupv2
+  hostRoot: /sys/fs/cgroup
 # [parity] roll the agent DaemonSet ONE node at a time. Overrides the cilium 1.20.1 chart
 # default of 2, which on 3 nodes lets a single bad render break 2/3 agents at once — as the
 # 2026-09-01 incident did (k8sServiceHost: "" → the `config` init container fell back to the
@@ -91,6 +91,10 @@ This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on

 ## Observability: what this component can and cannot see (measured 2026-09-30)

+> **SUPERSEDED — read "Update 2026-10-05" at the end of this section first.** The gate sidecars
+> **are** scraped now and the TSDB **does** hold `oauth2_proxy_*` for them. The two subsections
+> below are kept as the dated record of how this was diagnosed, not as current state.
+
 `servicemonitor.yaml` scrapes **this namespace's** deployment and nothing else. On prod that is
 two targets, `job="oauth2-proxy"`, both `up=1`. The complete set of oauth2-proxy-specific series
 in the TSDB is:
@@ -157,3 +161,55 @@ present `up` series. A discovered-but-failing target publishes `up` with value 0
 would have cleared `OAuth2ProxyGateUnobserved` the moment the PodMonitor landed — reporting the gates
 as observed while nothing was being scraped. The unit test pins that scenario; deleting the `== 1`
 fails it and nothing else.
+
+### Update 2026-10-05: the scrape works
+
+PR #49 (merged `21c88b0`) added the missing third piece. `catalog/render.py`'s
+`r_monitoring_ingress_policy` now also fires for the gate sidecar's metrics port, keyed on
+`_needs_oauth2_sidecar`, rendering `<app>-allow-monitoring-scrape` per app: ns `monitoring` → that
+app's `endpointSelector` → TCP 44180 only. Five objects, one each for jackett, pihole, qbittorrent,
+radarr, sonarr.
+
+Measured on prod 2026-10-05 after Flux applied it (`apps-wave-0` 21:37:01Z, `apps-wave-1`
+21:37:46Z):
+
+| | before | after |
+|---|---|---|
+| target health (6) | `down` | `up` |
+| `lastError` | `context deadline exceeded` | `''` |
+| `lastScrapeDuration` | **10.0s** — exactly the `scrapeTimeout` | **0.001–0.006s** |
+| `up` | `0` × 6 | `1` × 6 |
+
+    count by (__name__) ({__name__=~"oauth2_proxy_.*",
+                          namespace=~"jackett|radarr|sonarr|qbittorrent|pihole"})
+      oauth2_proxy_requests_in_flight              => 6
+      oauth2_proxy_requests_total                  => 16
+      oauth2_proxy_response_duration_seconds_bucket => 108
+      oauth2_proxy_response_duration_seconds_sum   => 9
+      oauth2_proxy_response_duration_seconds_count => 9
+
+    sum by (namespace) (oauth2_proxy_requests_total)
+      qbittorrent => 35072    radarr => 23560    sonarr => 1284
+
+`oauth2_proxy_requests_total` is absent for jackett and pihole simply because that counter is not
+created until the gate has served its first request.
+
+**So the auth-churn rule is now writable rather than decorative** — sustained `code="302"` as a
+share of `oauth2_proxy_requests_total`, held across a window longer than the session lifetime. That
+is the rule the 2026-09-30 incident above could not be caught by. It is still unwritten.
+
+**Alert outcome:** all 5 `TargetDown` resolved, and 5 of 6 `OAuth2ProxyGateUnobserved` with them.
+At `repeat_interval: 12h` with `group_by: [alertname, namespace]`, those 11 groups had been
+re-notifying at roughly **22 emails/day** since 2026-10-02; that drops to 2/day.
+
+**`crafty` still fires, and that is correct.** It is a hand-maintained Deployment under
+`platform/crafty`, outside the renderer, whose `oauth2-proxy` container has **no
+`--metrics-address` at all** — no listener, so no port to open and nothing for the PodMonitor's
+pod-label selector to match. Closing it needs the flag and a `metrics` containerPort on crafty's
+own manifest first, then a hand-written allowance beside it. Open follow-up work.
+
+**The paragraph above headed "To close the gap for an app" is superseded**: a *Service* port and a
+ServiceMonitor are **not** required. A PodMonitor targets pod ports directly, which is why no
+Service exposes 44180 and why the renderer needed no Service change. The three things actually
+needed are `--metrics-address` + named container port, a PodMonitor, and the monitoring-ingress
+allowance.
@@ -27,10 +27,28 @@
 # created by the apps-wave-0/1 tiers, which this tier does not and must not depend on.
 #
 # ============================================================================================
-# READ THIS BEFORE BELIEVING THE TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY
+# STATUS: THE SCRAPE WORKS. Verified on prod 2026-10-05 (PR #49 merged as 21c88b0).
 # ============================================================================================
-# This PodMonitor is NECESSARY AND NOT SUFFICIENT. The endpoint serves, and the path from ns
-# monitoring is blocked. Both halves measured on prod 2026-10-01, read-only:
+# All 6 targets in pool podMonitor/oauth2-proxy/oauth2-proxy-sidecars/0 read `up == 1`,
+# health=up, lastError='', lastScrapeDuration 0.001-0.006s. The TSDB holds oauth2_proxy_* for
+# these namespaces for the first time: 5 metric names / 148 series, with live counters
+# (oauth2_proxy_requests_total 35072 qbittorrent, 23560 radarr, 1284 sonarr). All 5 TargetDown
+# and 5 of 6 OAuth2ProxyGateUnobserved resolved; only namespace="crafty" still fires, and that
+# is correct — see CRAFTY below.
+#
+# The missing piece was an ingress allowance, and it now ships WITH EACH APP, rendered by
+# catalog/render.py's r_monitoring_ingress_policy as `<app>-allow-monitoring-scrape`:
+# ns monitoring -> the app's endpointSelector -> TCP 44180 only. See WHY NOT IN THIS DIRECTORY
+# below for why it is rendered per-app instead of shipped from this tier.
+#
+# Keep the history below. It is why the fix is shaped the way it is, and `up == 1` in
+# auth-visibility-alerts.yaml is load-bearing precisely because of it.
+#
+# --------------------------------------------------------------------------------------------
+# HISTORY: 2026-10-02 -> 2026-10-05, THE SCRAPE WAS DENIED BY NETWORKPOLICY FOR 63.7h
+# --------------------------------------------------------------------------------------------
+# This PodMonitor was NECESSARY AND NOT SUFFICIENT: the endpoint served, and the path from ns
+# monitoring was blocked. Both halves measured on prod 2026-10-01, read-only:
 #
 #   THE ENDPOINT IS HEALTHY — port-forward straight to the jackett pod's 44180:
 #     GET /metrics -> http=200 bytes=10208
@@ -54,19 +72,24 @@
 #   `jackett-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180 — the Gateway's Envoy
 #   and nothing else. Prometheus matches no clause and its packets are dropped, not refused.
 #
-# SO THE TARGETS BELOW WILL READ `up == 0` UNTIL AN INGRESS ALLOWANCE LANDS. That is a known,
-# measured, documented state — NOT a regression introduced here, and NOT something to silence.
-# It is deliberately left visible: `OAuth2ProxyGateUnobserved` in auth-visibility-alerts.yaml
+# THE TARGETS READ `up == 0` FROM 2026-10-02 05:09 UTC UNTIL THE ALLOWANCE LANDED — one
+# continuous 63.7h episode (a 15d query_range shows 1 episode per namespace, not flapping),
+# starting 11 minutes after this file was created, i.e. TargetDown's `for: 10m`. It was never
+# silenced, and that was the right call: `OAuth2ProxyGateUnobserved` in auth-visibility-alerts.yaml
 # requires `up == 1`, precisely so a discovered-but-failing target does not read as "observed"
 # (see that file's header — this exact trap is why the `== 1` is there).
 #
+# The signature to recognise if it ever returns: lastScrapeDuration pinned at EXACTLY the
+# scrapeTimeout (10.0s) with `context deadline exceeded`. That is a DROP, not a refusal — a
+# refused connection fails in milliseconds. Post-fix the same targets scrape in 1-6ms.
+#
 # THIS IS THE SAME BUG LONGHORN HAD, and the platform has a documented fix idiom for it:
 # platform/longhorn/networkpolicy-metrics.yaml, whose header records the identical fingerprint
 # measured on 2026-09-13 — ServiceMonitor in place, 3 targets DISCOVERED, `up` 0/0/0, TSDB empty,
 # `lastError: context deadline exceeded` (a drop, not a refusal). The fix there is a narrow
 # additive `networking.k8s.io/v1` NetworkPolicy: ns monitoring -> the pod label, that one port.
 #
-# WHY THAT POLICY IS NOT IN THIS DIRECTORY. It would have to be five objects, one per app
+# WHY NOT IN THIS DIRECTORY (the reasoning the fix followed). It would have to be five objects, one per app
 # namespace, and those namespaces are created by apps-wave-0/1. Measured Flux graph, 2026-10-01:
 #   oauth2-proxy  dependsOn=['platform-config']        <- this tier
 #   apps-wave-0   dependsOn=['platform-config']        <- SIBLING, no ordering between them
@@ -76,8 +99,19 @@
 # running cluster and break on a rebuild — the "works now, broken on restore" shape this repo
 # keeps paying for. Longhorn's precedent is safe only because longhorn-system is created by the
 # same tier that ships its policy. The allowance therefore belongs WITH THE APPS, rendered next to
-# the rest of each app's network policy — i.e. in the catalog/renderer, which is in flight on
-# another branch. Tracked in the handoff; it is one narrow rule per app, not a redesign.
+# the rest of each app's network policy — i.e. in the catalog/renderer. THAT IS WHERE IT LANDED:
+# `r_monitoring_ingress_policy`, keyed on `_needs_oauth2_sidecar`, one narrow rule per app.
+#
+# --------------------------------------------------------------------------------------------
+# CRAFTY: STILL UNOBSERVED, AND THIS FILE CANNOT HELP IT
+# --------------------------------------------------------------------------------------------
+# `OAuth2ProxyGateUnobserved{namespace="crafty"}` still fires after the fix, correctly. crafty is
+# a hand-maintained Deployment under platform/crafty, OUTSIDE the renderer, and its oauth2-proxy
+# container has NO `--metrics-address` at all (measured 2026-10-05) — so its metrics listener
+# never starts and there is no port to open. The selector below therefore does not match it, and
+# the per-app allowance is never rendered for it. Closing crafty's gap needs the flag plus a
+# `metrics` containerPort added to its own manifest FIRST; the allowance is the second half, and
+# it would have to be written by hand there rather than rendered. Open follow-up work.
 apiVersion: monitoring.coreos.com/v1
 kind: PodMonitor
 metadata:
@@ -2,7 +2,7 @@
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://home.keithjackson.dev/schemas/servicedefinition.schema.json",
  "title": "ServiceDefinition",
-  "description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.10.0 (adds optional `availability` for the container kind, one member: `availability.replicas`. It is AVAILABILITY INTENT, not a scheduling primitive -- it names no Kubernetes object, no topology key and no budget. 1 (the default) renders a Deployment exactly as before, so every pre-existing app stays byte-identical; >1 means the service must survive the loss of a single node, and the renderer satisfies that with a StatefulSet whose longhorn volumes become per-replica volumeClaimTemplates (a longhorn PVC is ReadWriteOnce and cannot attach on two nodes, so a multi-replica Deployment is UNSCHEDULABLE, not merely suboptimal), REQUIRED podAntiAffinity on kubernetes.io/hostname (preferred would let every replica land on one node and deliver none of the intended tolerance while reporting healthy), podManagementPolicy Parallel, a PodDisruptionBudget of minAvailable 1, and an `up` SLI that follows the rendered kind (a Deployment-shaped rule against a StatefulSet absent()s forever and can never fire). Added for pihole, where LAN DNS -- a critical network service -- was one pod on one node: measured 2026-10-04, losing the node running it meant roughly 6-10 minutes with no LAN resolution (tolerationSeconds 300 plus node-NotReady detection plus a Longhorn RWO force-detach). Deliberately NOT usable until a service's state is reproducible from git, because a replica with a fresh volume otherwise comes up serving but unconfigured -- for pihole that prerequisite was done first (decisions/2026-10-04-pihole-declare-state.md). Capped at 5: a count above the node count can never schedule under the anti-affinity this implies. Mechanism documented in schemas/renderer-contract.md `### availability`). 1.9.0 (adds optional `auth.apiRoutes` -- a per-service list of anchored path regexes the forward-auth oauth2-proxy sidecar answers with 401 instead of a 302 to Keycloak, rendered as one repeatable `--api-route` per entry. The `auth` block stays `additionalProperties: false`; this is one named, documented key, not an opened door. Without it there was NO per-app override hook at all, so the only way to answer an app's XHRs with 401 was a renderer-wide constant -- and the path shape is a property of the app: jackett's is `/api/` (33 x 302 vs 2 x 401 measured 2026-09-30, 4 of 6 POSTs answered by the proxy and never reaching the app), radarr's is `/signalr/` (232 x 302 measured 2026-10-01, every one a POST from a left-open browser tab that is never re-prompted), and pihole serves no `/api/` at all. Each entry must be anchored at `^/` and continue with literal path characters, because a too-broad regex turns a plain page load into a 401 and the service looks permanently broken to a logged-out user. Enforced on every gate run by ci/l0.sh's schema stage over catalog/*.yaml, by catalog/render.py refusing to render an unanchored entry, and by conformance/tests/l0-static/oauth2-proxy-api-routes-per-app)). 1.8.0 (adds optional `hardware` - a host HARDWARE CAPABILITY declaration for the container kind, currently one member: `hardware.gpu: intel-quicksync`, meaning the workload needs the host iGPU render device for hardware video transcode. It is an ENUM OF INTENT, not a device path: the renderer satisfies it today with just two things -- a /dev/dri hostPath volume (`DirectoryOrCreate`) and a read-write /dev/dri volumeMount on the app container -- and a later move to the Intel GPU device plugin is then a render.py change with no catalog edit. It sets NO podSecurityContext.supplementalGroups: measured in-container on prod 2026-09-30, the pms-docker cont-init hook adds the app user to the device's group inside the container and `s6-setuidgid` then REPLACES the process group list from /etc/group, discarding anything Kubernetes supplied, so such an entry would be inert and misleading. Renders NOTHING when absent, so every existing app stays byte-identical. Emits no privileged securityContext (Kyverno restrict-host-namespaces would reject it and it is not needed), no nodeSelector (node placement is a separate concern with its own justification) and no resource requests). 1.7.0 (no field changes; a MAPPING change to `exposure`, which had never had an effect on DNS. `exposure: internal` now renders ExternalDNS's own controller annotation onto every HTTPRoute it emits, set to a value other than `dns-controller`, so external-dns (v0.21.0, source/gateway.go -> annotations.IsControllerMismatch) skips the route before resolving any hostname and the service is INCAPABLE of producing a public record. Both docs/09 R-DNS-03 and schemas/renderer-contract.mdLine truncated
+  "description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.11.0 (adds optional `optional` to env.valueFrom.secretKeyRef and env.valueFrom.configMapKeyRef -- the k8s EnvVarSource field, emitted verbatim. Without it a secretKeyRef was ALWAYS hard: a missing Secret meant the pod would not start at all. That is the right default and stays the default, but it cannot express a value the workload is designed to run without. Added for plex, whose TLS material is issued by cert-manager INTO ITS OWN NAMESPACE and therefore cannot exist before the app wave that creates that namespace -- the Certificate and the Deployment land in the same reconcile, so on a cold bootstrap the Secret is briefly absent while a DNS01 challenge completes. Hard refs there would have made the media server unstartable until a certificate issued, and `apps-wave-0` is `wait: true`, so a failed or slow issuance would have STALLED THE WHOLE APP TIER -- the same hazard catalog/plex.yaml already documents for its GPU resource. With `optional: true` Plex starts regardless, its postStart hook logs that the certificate is not installed, and the next start after issuance converges. Renders nothing when absent, so every existing app stays byte-identical). 1.10.0 (adds optional `availability` for the container kind, one member: `availability.replicas`. It is AVAILABILITY INTENT, not a scheduling primitive -- it names no Kubernetes object, no topology key and no budget. 1 (the default) renders a Deployment exactly as before, so every pre-existing app stays byte-identical; >1 means the service must survive the loss of a single node, and the renderer satisfies that with a StatefulSet whose longhorn volumes become per-replica volumeClaimTemplates (a longhorn PVC is ReadWriteOnce and cannot attach on two nodes, so a multi-replica Deployment is UNSCHEDULABLE, not merely suboptimal), REQUIRED podAntiAffinity on kubernetes.io/hostname (preferred would let every replica land on one node and deliver none of the intended tolerance while reporting healthy), podManagementPolicy Parallel, a PodDisruptionBudget of minAvailable 1, and an `up` SLI that follows the rendered kind (a Deployment-shaped rule against a StatefulSet absent()s forever and can never fire). Added for pihole, where LAN DNS -- a critical network service -- was one pod on one node: measured 2026-10-04, losing the node running it meant roughly 6-10 minutes with no LAN resolution (tolerationSeconds 300 plus node-NotReady detection plus a Longhorn RWO force-detach). Deliberately NOT usable until a service's state is reproducible from git, because a replica with a fresh volume otherwise comes up serving but unconfigured -- for pihole that prerequisite was done first (decisions/2026-10-04-pihole-declare-state.md). Capped at 5: a count above the node count can never schedule under the anti-affinity this implies. Mechanism documented in schemas/renderer-contract.md `### availability`). 1.9.0 (adds optional `auth.apiRoutes` -- a per-service list of anchored path regexes the forward-auth oauth2-proxy sidecar answers with 401 instead of a 302 to Keycloak, rendered as one repeatable `--api-route` per entry. The `auth` block stays `additionalProperties: false`; this is one named, documented key, not an opened door. Without it there was NO per-app override hook at all, so the only way to answer an app's XHRs with 401 was a renderer-wide constant -- and the path shape is a property of the app: jackett's is `/api/` (33 x 302 vs 2 x 401 measured 2026-09-30, 4 of 6 POSTs answered by the proxy and never reaching the app), radarr's is `/signalr/` (232 x 302 measured 2026-10-01, every one a POST from a left-open browser tab that is never re-prompted), and pihole serves no `/api/` at all. Each entry must be anchored at `^/` and continue with literal path characters, because a too-broad regex turns a plain page load into a 401 and the service looks permanently broken to a logged-out user. Enforced on every gate run by ci/l0.sh's schema stage over catalog/*.yaml, by catalog/render.py refusing to render an unanchored entry, and by conformance/tests/l0-static/oauth2-proxy-api-routes-per-app)). 1.8.0 (adds optional `hardware` - a host HARDWARE CAPABILITY declaration for the container kind, currently one member: `hardware.gpu: intel-quicksync`, meaning the workload needs the host iGPU render device for hardware video transcode. It is an ENUM OF INTENT, not a device path: the renderer satisfies it today with just two things -- a /dev/dri hostPath volume (`DirectoryOrCreate`) and a read-write /dev/dri volumeMount on the app container -- and a later move to the Intel GPU device pluLine truncated
  "type": "object",
  "additionalProperties": false,
  "required": ["name", "source"],
@@ -69,7 +69,8 @@
                "required": ["name", "key"],
                "properties": {
                  "name": { "type": "string", "minLength": 1 },
-                  "key": { "type": "string", "minLength": 1 }
+                  "key": { "type": "string", "minLength": 1 },
+                  "optional": { "type": "boolean", "description": "Tolerate the key being ABSENT: the env var is simply unset instead of the pod refusing to start (kubelet CreateContainerConfigError). Default false, which is the pre-1.11.0 behaviour. Use ONLY where the workload degrades gracefully without the value, and say so at the call site -- a silently-empty credential is usually worse than a pod that will not start." }
                }
              },
              "configMapKeyRef": {
@@ -78,7 +79,8 @@
                "required": ["name", "key"],
                "properties": {
                  "name": { "type": "string", "minLength": 1 },
-                  "key": { "type": "string", "minLength": 1 }
+                  "key": { "type": "string", "minLength": 1 },
+                  "optional": { "type": "boolean", "description": "Tolerate the key being ABSENT: the env var is simply unset instead of the pod refusing to start (kubelet CreateContainerConfigError). Default false, which is the pre-1.11.0 behaviour. Use ONLY where the workload degrades gracefully without the value, and say so at the call site -- a silently-empty credential is usually worse than a pod that will not start." }
                }
              }
            }