docs(agents): one workspace per agent run — the shared checkout is off-limits

Agents have been working in the operator's checkout and clobbering each other.
Nothing in the tree told them not to: worktrees appeared in CLAUDE.md only as
§3's gate workarounds, and decisions/README.md described worktree isolation as
something agents already do rather than as a rule.

New CLAUDE.md §0 states the rule, with the commands and an explicit do-not list
for the shared checkout. Placed before §1 so every existing §-reference in the
repo keeps its meaning.

Measured in the shared checkout on 2026-10-05, each row a distinct failure mode:
107 `checkout: moving` reflog entries; sitting on feat/pm-agent-issue-to-pr, not
main; two other runs' untracked trees left in it; local main two commits behind
origin/main; 42 registered worktrees of which 31 are on already-merged branches;
9 gate scripts carrying a `.claude/` exclusion because an in-tree worktree made
find-based discovery count the same manifest up to 18 times.

Corrects a claim carried in earlier briefs: `git worktree remove` is NOT blocked
by the agent sandbox. Provoked on a throwaway worktree — rc 0.

§3 reframed as the normal path and grown from two breakages to three:
spec-immutability's skip is avoidable, not merely tolerated, with the spec repo
mounted read-only in the same ENGINE shim plus -e SPEC_REPO=/home-network-k3s.

CLAUDE.md is a sensitive path, so this PR needs a non-author review or an
administrator /approve-override posted as a PR review. Not posted by me: §1
forbids an agent clearing that on its own PR.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

CLAUDE.md

@@ -4,6 +4,117 @@ Read this before doing anything durable. It is the delivery contract for AI codi
 sessions and agents. The newest entry in `decisions/` is the current PM handoff and
 takes precedence on *what* to work on; this file governs *how* it lands.
 
+**Start with §0.** It is first because it is the rule you can break before you have read
+anything else: the very first `git checkout -b` in the wrong directory has already done the
+damage.
+
+## 0. Work in your OWN workspace. Never in the shared checkout.
+
+Before you create a branch, edit a file, or run `git` anything: **make yourself a git
+worktree outside the repository root, and do all of your work there.** One workspace per
+agent run, no exceptions.
+
+    # from the shared checkout, ONCE, at the start of your run
+    git fetch origin main
+    git worktree add -b <type>/<slug> /path/outside/the/repo/wt-<slug> origin/main
+    cd /path/outside/the/repo/wt-<slug>        # everything after this happens HERE
+
+`/path/outside/the/repo` means outside this repository's tree — your session scratchpad,
+or a sibling like `/var/home/keith/source/wt-<slug>`. Branch from **`origin/main`**, never
+from `main` and never from `HEAD`; see "stale base" below.
+
+**The shared checkout is `/var/home/keith/source/home-network-k3s-impl`** — the operator's
+working copy, the one you are probably started in. In it, do **not**:
+
+- **`git checkout` / `git switch` / `git checkout -b`.** Its HEAD is shared state; moving
+  it moves the tree out from under whoever else is using it.
+- **edit, create or delete a file.** Including "just one doc", including a file you intend
+  to move later.
+- **`git add` / `git commit` / `git stash` / `git restore`.** `git add -A` there stages
+  whatever another agent left lying around; `git stash` *removes* their uncommitted work
+  into a place they will not look.
+- **`git worktree remove` anything you did not create.**
+
+Read-only commands there are fine, and the one write you do need is creating your own
+worktree.
+
+### Why — measured in the shared checkout, 2026-10-05
+
+| What was measured | Value |
+|---|---|
+| `checkout: moving` entries in its reflog | **107** — the tree has been yanked between branches 107 times |
+| the branch it was sitting on | `feat/pm-agent-issue-to-pr` — **not `main`** |
+| uncommitted trees left in it by other runs | **2** (`infra/agent/`, `platform/pm-agent/`), mtimes five minutes apart the same evening |
+| its local `main` vs `origin/main` | **two commits behind** (`4386d14` vs `22c94ef`) |
+| registered worktrees | **42**, of which **31** are on branches already merged to `main` |
+| gate scripts carrying an explicit `.claude/` exclusion | **9** |
+
+Each row is a distinct way the shared checkout damages work:
+
+- **Dirty tree, foreign files.** Two other runs' uncommitted directories were sitting in
+  it. An agent that edits there and runs `git add -A` ships someone else's half-finished
+  work inside its own PR, and the PR's author has no idea what those files are.
+- **Stale base.** Its `main` was two commits behind `origin/main`, so
+  `git checkout -b x main` there branches off a tree that is already not what CI will merge
+  into — the gate you run locally proves a base nobody else has. Worse, it was not even *on*
+  `main`: a bare `git checkout -b x` would have branched off `feat/pm-agent-issue-to-pr` and
+  silently inherited that work into your PR. `git fetch` first, and name `origin/main`
+  explicitly as the start point.
+- **Shared HEAD.** 107 branch switches in one tree is 107 chances to switch it while
+  another agent is mid-edit. The loser's changes are either carried onto the wrong branch
+  or lost to a checkout conflict — and the symptom surfaces in *their* session, not yours.
+
+### Put the workspace OUTSIDE the repo root
+
+A worktree *inside* the tree (`.claude/worktrees/agent-*/` — which is where Claude Code's
+own default puts it) is a full second copy of the repo that every tree-walking check then
+finds. Measured 2026-09-12: `find`-based discovery reported **the same manifest under test
+once per worktree, up to 18 times**, and the L0 check comparing that listing against the
+tree failed on paths that are not authored surface at all
+(`platform/monitoring/tests/run.sh:115-123`). Nine gate scripts now carry an explicit
+`.claude/` prune for this reason — which means the hazard is *worked around*, not fixed,
+and the next check that walks the tree will hit it again. Keep the copy out of the tree and
+none of that applies.
+
+### Clean up — and expect not to be able to
+
+31 of the 42 registered worktrees are on branches already merged to `main`. A registered
+worktree **pins its branch**: while it exists, that branch cannot be deleted or checked out
+anywhere else. So a left-behind workspace is a lock on a branch name, and the cost lands on
+whoever next needs it.
+
+Remove yours when you are done:
+
+    git worktree remove /path/outside/the/repo/wt-<slug>
+
+Measured 2026-10-05: that command **succeeds** under the agent sandbox (rc 0, on a throwaway
+worktree in the session scratchpad) — earlier sessions recorded it as blocked, and that
+record was wrong, or at least no longer true. Try it; it probably works. If it is refused,
+say so in your handoff and leave the worktree registered — do **not** `rm -rf` the directory,
+which leaves a registered worktree pointing at nothing and still holding the branch.
+
+If someone *else's* stale worktree is holding the branch you need, do not try to reclaim the
+branch name. Push a `rework/<pr-number>` branch to the PR's refspec instead.
+
+### If you are already standing in the shared checkout with edits
+
+Stop. Do not `git stash` (that moves other agents' work too). Create your worktree as
+above, redo the edits there, and leave the shared checkout's HEAD and working tree exactly
+as you found them.
+
+### Gating from a worktree
+
+Three known breakages, all with workarounds, all required every run: see §3, "Running the
+gate from a worktree".
+
+### If you are writing a brief for another agent
+
+The brief must name **its own explicit branch** and tell it to create **its own worktree**
+from `origin/main`. "Work on the current branch" and "continue in the checkout" are how
+branch sprawl and cross-contamination got here: an agent given no workspace uses the one it
+is started in, which is the shared one. Also carry §3's three gate workarounds into the
+brief — an agent that cannot gate will report rc 3 as green.
+
 ## 1. Never push to `main`. Ever.
 
 Every durable change reaches `main` exactly one way:
@@ -147,15 +258,21 @@ Exit codes are a contract — do not flatten them:
 | 3 | UNDETERMINED — a stage did not run. **Never report this as green.** |
 
 `spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out
-beside this one; that single skip is expected and tolerated. Any *other* skip is a
-real gap in your run.
+beside this one; that single skip is expected and tolerated — and from a worktree it is
+avoidable outright (§3's worktree list, item 3). `gitea-settings` is the other expected
+skip: it needs `SETTINGS_API_TOKEN`, which exists only in CI, so **a local run cannot
+return 0** — the honest best local result is rc 3 with `gitea-settings` as the only stage
+that did not run. Any *further* skip is a real gap in your run. Report the rc and name the
+skipped stages; never round rc 3 up.
 
 **Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`,
 so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts.
 
-### Running the gate from a git worktree
+### Running the gate from a worktree
 
-Two unrelated failures, both with known workarounds — put both in any agent brief:
+This is the **normal** path, not an edge case — §0 requires you to be in a worktree, so
+every gate run you do hits all of these. Three known breakages, all with
+workarounds, all of them yours to apply every run — put them in any agent brief:
 
 1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked
    and exists only in the main checkout. Do **not** copy it (it is a credential).
@@ -166,10 +283,17 @@ Two unrelated failures, both with known workarounds — put both in any agent br
    the `ENGINE` hook — a shim that intercepts `run` and re-execs
    `podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in
    `tracked_files()` is this bug, not a broken tree.
+3. **`spec-immutability` skips** — the skip called tolerated above is also *avoidable*. `dev.sh`
+   derives the sibling spec path from its own `REPO_ROOT`, which in a worktree is your
+   workspace, so `../home-network-k3s` does not resolve and it adds no mount. Mount the spec
+   repo read-only in the same `ENGINE` shim and pass `-e SPEC_REPO=/home-network-k3s`: with
+   both mounts the stage reports `pass`, not SKIPPED, and R-TEST-06 is actually covered
+   (measured — `decisions/2026-10-01-render-selftest-runs-in-gate.md`).
 
 ## 4. Branch naming
 
-Use `feat/`, `fix/`, `docs/`, `chore/`. Note that `.gitea/workflows/gate.yml`'s **push**
+Use `feat/`, `fix/`, `docs/`, `chore/`. **One branch, one workspace, one PR** — if you
+need a second branch, make a second worktree (§0); do not re-point an existing one. Note that `.gitea/workflows/gate.yml`'s **push**
 trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run.
 The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected.
 

README.md

@@ -11,6 +11,9 @@ specification** at [`../home-network-k3s`](../home-network-k3s).
 - **Done = a green, reproducible, requirement-tagged conformance test** — never "merged," never a
   manual demo (spec `docs/14`, `agents/definition-of-done.md`).
 - **No work on `main`.** Every change is a feature branch merged into `main` (spec `CONTRIBUTING.md`).
+- **No work in a shared checkout.** Every agent run gets its own git worktree, outside the repo
+  root, branched from `origin/main`. This checkout is shared mutable state — see
+  [`CLAUDE.md` §0](CLAUDE.md).
 - **No plaintext secrets** — SOPS-encrypted only (spec `docs/07`).
 - **Phase discipline** — only build what the active roadmap phase authorizes (spec `roadmap/phases.md`).
 
@@ -94,6 +97,20 @@ The running narrative — decisions, root causes, incidents, follow-ups — is
 
 ## Working on this repo
 
+**First, get out of this checkout.** It is the operator's working copy and several agents
+share it; concurrent runs here have overwritten each other's branches and left each other's
+uncommitted files behind. One workspace per change:
+
+```bash
+git fetch origin main
+git worktree add -b <type>/<slug> /path/outside/the/repo/wt-<slug> origin/main
+cd /path/outside/the/repo/wt-<slug>
+```
+
+Outside the repo root specifically — a worktree under `.claude/worktrees/` is a second full
+copy that tree-walking gate checks then count twice. The rule, the measurements behind it, and
+the three gate workarounds a worktree needs are in [`CLAUDE.md` §0](CLAUDE.md) and §3.
+
 Everything the gate needs is baked into the pinned dev container — **do not install the
 toolchain on the host**:
 

apps/wave-0/plex.yaml

@@ -138,7 +138,7 @@ spec:
               name: plex-claim
               key: PLEX_CLAIM
         - name: ADVERTISE_IP
-          value: http://plex-vip.${BASE_DOMAIN}:32400
+          value: https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400
         - name: PLEX_TLS_CRT
           valueFrom:
             secretKeyRef:

catalog/plex.yaml

@@ -72,77 +72,59 @@ env:
       secretKeyRef:
         name: plex-claim
         key: PLEX_CLAIM
-  # ADVERTISE_IP ("Custom server access URLs") — the ONE hostname this server publishes to plex.tv,
-  # serving BOTH local and remote clients by split-horizon DNS (operator design, 2026-10-04):
-  #   LAN    -> PiHole answers plex-vip.${BASE_DOMAIN} with ${PLEX_L4_VIP} (the tcp/32400 LoadBalancer
-  #             in apps/wave-0/plex-remote.yaml). See catalog/pihole.yaml's split-horizon line, which
-  #             needs a per-host override because the zone wildcard otherwise points at the GATEWAY
-  #             VIP — measured 2026-10-04: plex-vip.${BASE_DOMAIN} resolved to the Gateway, where
-  #             nothing listens on 32400.
-  #   PUBLIC -> Cloudflare answers with the WAN address, NATed to the same VIP (off-repo).
+  # ADVERTISE_IP ("Custom server access URLs") — the connection Plex publishes to plex.tv.
   #
-  # WHY A HOSTNAME AND NOT THE `plex.direct` FORM THIS BRIEFLY USED. The first fix for the
-  # 2026-10-04 outage advertised `https://<dashed-ip>.<serverhash>.plex.direct:32400`, which works
-  # (and is Plex's own mechanism), but it hard-codes BOTH the VIP and the server's certificate hash
-  # into a cluster-var — two values nothing could keep in sync, with no alert if either drifted. A
-  # hostname in our own zone removes that whole class of problem: ${BASE_DOMAIN} is already a knob,
-  # the VIP lives only in PiHole's answer, and `PLEX_ADVERTISE_URL` is deleted outright.
+  # THE VALUE MUST BE RESOLVABLE BY EVERY CLIENT, NOT JUST BY CLIENTS THAT USE OUR DNS. That single
+  # property is what this field keeps getting wrong, so it is the first thing to check before
+  # changing it. A Plex client resolves the advertised host ITSELF, with whatever resolver it
+  # happens to use — and on a real LAN that is frequently NOT PiHole: browsers ship DoH on by
+  # default, phones have "private DNS", guest and IoT devices often take the router's DNS, and some
+  # clients hard-code a public resolver.
   #
-  # THIS IS `https`, WHICH ONLY WORKS BECAUSE WE HAND PLEX OUR OWN CERTIFICATE. Plex serves 32400
-  # with its `*.<serverhash>.plex.direct` cert, which does NOT cover this name — advertising
-  # `https://plex-vip...` against the stock cert reproduces the exact failure that caused the
-  # outage (measured: a storm of `CERT: incomplete TLS handshake ... certificate unknown` from real
-  # clients, and `curl` without `-k` failing to connect). The `lifecycle.postStart` hook below
-  # installs the Let's Encrypt cert for this name and points Plex at it, so the advertised URL and
-  # the served certificate agree. Do not change this to a bare IP, and do not drop the hook and
-  # leave the `https` here.
+  # THAT IS NOT THEORETICAL — IT TOOK PLEX DOWN FOR THE WHOLE LAN ON 2026-10-05. This field briefly
+  # advertised `plex-vip.${BASE_DOMAIN}`, a name that exists ONLY in PiHole's split-horizon answers
+  # (catalog/pihole.yaml) and in no public zone. Clients that did not use PiHole could not resolve
+  # it; the SOHO router answered the failed lookup with ITSELF, and every one of them landed on the
+  # router's own web UI and reported a certificate error for `CN = %D`. Measured 2026-10-05: the
+  # router serves that self-signed cert (`C = ZZ, O = %D342e7e6b, CN = %D`) on BOTH `192.168.2.1:443`
+  # and the WAN address, while PiHole itself was answering perfectly — both replicas returned
+  # `plex-vip -> ${PLEX_L4_VIP}` and `plex -> ${GATEWAY_VIP}`. The DNS was right; the DEPENDENCY on
+  # it was the bug. Nothing Plex advertises listens on 443, which is how we know the clients had left
+  # the Plex path entirely and were talking to the router.
   #
-  # (A SAN match is not what makes Plex *work* — `httpsRequired=0` and `secureConnections` default
-  # to Preferred, so clients can fall back to cleartext. It is what keeps the remote leg ENCRYPTED
-  # instead of silently downgrading traffic that crosses the internet.)
+  # SO: `<dashed-ip>.<serverhash>.plex.direct`, WHICH IS RESOLVER-INDEPENDENT. This is Plex's own
+  # mechanism and the reason it exists: `plex.direct` is a PUBLIC zone that returns the private
+  # address embedded in the label, so EVERY resolver on earth — PiHole, the router, 1.1.1.1, a DoH
+  # browser — answers `192-168-2-24.<hash>.plex.direct` with `192.168.2.24`. It needs no
+  # split-horizon entry, no Cloudflare record, and no certificate work, because the wildcard cert
+  # Plex already serves on 32400 covers exactly this name. Re-verified live 2026-10-05 with FULL
+  # validation (no `curl -k`): resolves to 192.168.2.24 and `/identity` returns **200, TLS verified**.
+  #
+  # WHY NOT THE NICER HOSTNAME, since a readable name in our own zone is obviously more pleasant:
+  # it cannot be made to work for clients that bypass PiHole. A public record for it would resolve
+  # to the WAN address, and with no NAT hairpin on this router a LAN client sent there fails too —
+  # so the hostname is correct ONLY for clients using PiHole, and wrong for the rest. There is no
+  # configuration of that name that is right for both. `plex.direct` is right for both.
+  # (The PiHole `plex-vip` answer is KEPT — it is harmless, costs nothing, and is genuinely useful
+  # for reaching the VIP by name from a LAN shell. It is simply not what Plex advertises.)
+  #
+  # THE TWO EMBEDDED VALUES, and the honest cost of this choice. The dashed IP must track
+  # ${PLEX_L4_VIP} and `<serverhash>` is this server's certificate identity — read it from the live
+  # listener with
+  #     openssl s_client -connect <vip>:32400 </dev/null | openssl x509 -noout -subject
+  # Nothing in this tree can derive either, and nothing checks that they still agree: envsubst
+  # cannot dash an IP, so this is a literal. That fragility is real and was the reason the hostname
+  # was tried in the first place — it is accepted here because an unresolvable name is a WORSE
+  # failure (silent, LAN-wide, and it points users at the router) than a stale literal, which at
+  # least fails loudly and only after someone changes the VIP.
   #
   # The image's /etc/cont-init.d/40-plex-first-run runs `setPref "customConnections"
   # "${ADVERTISE_IP}"` (line 127, verified in the running image), OUTSIDE the
-  # PLEX_CLAIM/PlexOnlineToken guard, and its `/.firstRunComplete` marker is on the container
-  # OVERLAY fs rather than the /config PVC — so this is re-applied on EVERY pod start and git, not
-  # a web form, owns the value. Hand-typed custom URLs are what broke this service; they are now
-  # overwritten on every start.
-  #
-  # SCHEME IS `http` AND THAT IS TEMPORARY, 2026-10-04. It was `https`, which requires Plex to serve
-  # a certificate for this name — and the Certificate that would have supplied one is NOT APPLIED
-  # (see apps/wave-0/plex-cert.yaml's header). Advertising `https` without it reproduces the
-  # 2026-10-04 outage exactly: Plex answers 32400 with its `*.<serverhash>.plex.direct` cert, every
-  # TLS-validating client rejects the hostname, and plex.tv can publish nothing. `http` works TODAY
-  # and is MEASURED to: PiHole answers plex-vip.${BASE_DOMAIN} with ${PLEX_L4_VIP} (verified live
-  # against pihole-0, 2026-10-04) and Plex serves 32400 there.
-  #
-  # THE COST, STATED: the remote leg is unencrypted. That is tolerable only because remote access
-  # does not work yet at all (the WAN tcp/32400 forward is still absent), so today this carries LAN
-  # traffic only. RESTORE `https` IN THE SAME COMMIT that re-applies the Certificate — the two are
-  # one change, and splitting them is what broke this.
-  #
-  # HOW TO GET `https` BACK. The Certificate that would serve this name was applied on 2026-10-04
-  # and STALLED THE ENTIRE APP TIER; the file was DELETED rather than merely unregistered, because
-  # `tls-lifecycle-automated` (R-TLS-02) fails any Certificate left in the tree that no Flux path
-  # reconciles. Full post-mortem: decisions/2026-10-04-plex-unblock-app-tier.md. Three things must
-  # land TOGETHER:
-  #   1. EXEMPT cert-manager FROM `secret-provenance`. Its temporary private-key Secrets
-  #      (`plex-vip-<random>`) are denied by `secrets-must-be-flux-managed`, which matches every
-  #      Secret in a namespace labelled `homelab.keithjackson.dev/app-namespace: true` — `plex` HAS
-  #      that label, `gateway` (where the working wildcard cert lives) does NOT, which is why this
-  #      never bit the existing Certificate. `secretTemplate` cannot fix it: those Secrets come from
-  #      the keymanager controller, not the Certificate's template. Narrow fix: an
-  #      `exclude.any[].subjects` entry for ServiceAccount `cert-manager`/`cert-manager` (confirmed
-  #      live as the controller's SA). `platform/kyverno/**` is a SENSITIVE path — non-author approval.
-  #   2. GIVE THE CERTIFICATE A NON-GATING HOME. `apps-wave-0` is `wait: true`, so Flux health-checks
-  #      the Certificate OBJECT and a pending issuance stalls the whole tier
-  #      (`timeout waiting for: [Certificate/plex/plex-vip status: 'InProgress']`). `optional: true`
-  #      on the env refs below protects the POD, not the Kustomization — that distinction is the
-  #      whole bug. Use a separate Kustomization with `wait: false`, the shape
-  #      clusters/base/backup.yaml already uses for exactly this reason.
-  #   3. Flip this value back to `https`.
+  # PLEX_CLAIM/PlexOnlineToken guard, with its `/.firstRunComplete` marker on the container OVERLAY
+  # fs rather than the /config PVC — so this is re-applied on EVERY pod start and git, not a web
+  # form, owns the value.
   - name: ADVERTISE_IP
-    value: http://plex-vip.${BASE_DOMAIN}:32400
+    value: https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400
   # THE TLS MATERIAL FOR THE NAME ABOVE, as PEM text in env rather than a mounted volume.
   # `plex-vip-tls` is issued by cert-manager from ClusterIssuer `letsencrypt-prod` for exactly
   # `plex-vip.${BASE_DOMAIN}` (apps/wave-0/plex-cert.yaml, same wave => same reconcile as this

catalog/render.py

@@ -156,11 +156,17 @@ OAUTH2_COOKIE_REFRESH = "2m30s"
 # ready endpoints. The flag is confirmed present in the PINNED v7.8.1 binary (`--help`),
 # which matters because the shared proxy is on v7.15.3 and could not have vouched for it.
 #
-# THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED. A container port is not a
-# Service port: these sidecars' Services expose the app port and :4180, and each namespace's
-# default-deny baseline has `ingress: []`, so Prometheus can neither resolve nor reach this
-# port until a Service port (or PodMonitor), a monitoring-ingress allow and a ServiceMonitor
-# are added. Do not read this flag as "metrics are working".
+# THE SCRAPE WORKS AS OF 2026-10-05 — but it took THREE independent pieces, and this flag is
+# only the first. A container port is not a Service port, and each namespace's default-deny
+# baseline denies all ingress, so the flag alone bought nothing for 63.7h:
+#   1. this flag + a named `metrics` container port   (PR #28)
+#   2. a monitor that targets POD ports, not Service ports — platform/oauth2-proxy/podmonitor.yaml
+#      is a PodMonitor for exactly that reason (no Service exposes 44180)
+#   3. an ingress allowance from ns monitoring        (PR #49, r_monitoring_ingress_policy below)
+# Verified on prod 2026-10-05: all 6 targets up == 1, scraping in 1-6ms.
+# STILL do not read this flag alone as "metrics are working" — 1 without 3 is precisely the state
+# that produced 5 x TargetDown for 63.7h. If you add a sidecar metrics port anywhere else, check
+# that r_monitoring_ingress_policy covers it.
 OAUTH2_PROXY_METRICS_PORT = 44180
 
 # --- Hardware capability: `hardware.gpu: intel-quicksync` ------------------------------
@@ -1362,8 +1368,9 @@ def _oauth2_sidecar_container(s: dict) -> dict:
         f"--cookie-refresh={OAUTH2_COOKIE_REFRESH}",
         "--pass-access-token=true",
         "--set-xauthrequest=true",
-        # Start the /metrics listener. Prerequisite only — not yet reachable by Prometheus;
-        # see OAUTH2_PROXY_METRICS_PORT above for exactly what is still missing.
+        # Start the /metrics listener. Scraped as of 2026-10-05, but only because
+        # r_monitoring_ingress_policy also opens this port to ns monitoring — the flag alone was
+        # inert for 63.7h. See OAUTH2_PROXY_METRICS_PORT above for the three pieces required.
         f"--metrics-address=0.0.0.0:{OAUTH2_PROXY_METRICS_PORT}",
     ]
     secret = f"{s['name']}-oauth2-proxy"

decisions/2026-10-04-oauth2-sidecar-metrics-scrape-netpol.md

@@ -4,6 +4,11 @@ branch: fix/oauth2-sidecar-metrics-scrape-netpol
 title: The forward-auth gate sidecars' /metrics was NetworkPolicy-blocked — the renderer now opens 44180 to ns monitoring
 ---
 
+> **Added 2026-10-05, after merge:** the top item under *What I did NOT verify* below — "that the
+> scrape now succeeds" — **has since been verified on prod**. All 6 targets read `up == 1`. The
+> original text is left exactly as written; the measurements are in
+> [[2026-10-05-oauth2-sidecar-scrape-verified]].
+
 ## What changed
 
 `catalog/render.py`'s `r_monitoring_ingress_policy` now emits its CiliumNetworkPolicy for a

decisions/2026-10-05-agents-work-in-their-own-workspace.md

@@ -0,0 +1,135 @@
+---
+date: 2026-10-05
+branch: docs/agents-work-in-their-own-workspace
+title: One workspace per agent run — the shared checkout is now documented as off-limits
+---
+
+## What changed
+
+Agents have been doing their work in the operator's checkout at
+`/var/home/keith/source/home-network-k3s-impl` and clobbering each other. Nothing in the tree
+told them not to: worktrees appeared in CLAUDE.md only as §3's *gate workarounds*, i.e. as an
+awkward edge case, and `decisions/README.md` asserted worktree isolation as a fact about how
+agents already behave rather than as a rule they must follow. Nowhere was the rule stated.
+
+Documentation only — no code, no manifests, no pipeline logic.
+
+- **`CLAUDE.md`: new `## 0. Work in your OWN workspace. Never in the shared checkout.`**
+  Placed before §1 so the section numbers every other file and handoff entry cites (§1, §2b,
+  §3, §7) all keep their meaning. It gives the three commands (`git fetch origin main`;
+  `git worktree add -b <type>/<slug> <path outside the repo> origin/main`; `cd`), an explicit
+  do-not list for the shared checkout (no `checkout`/`switch`, no edits, no
+  `add`/`commit`/`stash`/`restore`, no removing others' worktrees), a measurement table, the
+  "put it outside the repo root" rule, cleanup, what to do if you are already standing in the
+  shared checkout with edits, and what a PM must put in a brief it writes for another agent.
+- **`CLAUDE.md` intro** now says "Start with §0", with the reason: it is the rule you can
+  break before you have read anything else.
+- **`CLAUDE.md` §3** retitled "Running the gate from a worktree" and reframed as the normal
+  path rather than an edge case, since §0 now makes every gate run a worktree run. Its list
+  grew from two breakages to three: `spec-immutability`'s skip is *avoidable*, not merely
+  tolerated, if the `ENGINE` shim also mounts the sibling spec repo read-only and passes
+  `-e SPEC_REPO=/home-network-k3s`.
+- **`CLAUDE.md` §4** gained "One branch, one workspace, one PR".
+- **`README.md`**: a fourth non-negotiable rule ("No work in a shared checkout"), and
+  "Working on this repo" now opens with how to get out of the checkout before describing the
+  dev container that runs in it.
+- **`decisions/README.md`**: the worktree-isolation sentence now cites §0 as the requirement
+  instead of describing it as something agents happen to do.
+
+I did this work the way the new §0 requires — a worktree at
+`<session scratchpad>/wt-workspace-rule`, outside the repo root, branched from `origin/main`,
+with the shared checkout left on the branch and with the untracked files I found in it.
+
+## What I verified
+
+Every number in the new §0 table was measured in the shared checkout on 2026-10-05, not
+recalled:
+
+| Claim | Command | Value |
+|---|---|---|
+| shared HEAD is yanked between branches | `git reflog \| grep -c 'checkout: moving'` | **107** |
+| it is not sitting on `main` | `git worktree list` | `feat/pm-agent-issue-to-pr` |
+| other runs' uncommitted work is in it | `git status --porcelain` | **2** trees: `infra/agent/` (22:04), `platform/pm-agent/` (22:09) — `stat -c %y` |
+| its `main` is stale | `git rev-parse main` vs `origin/main` | `4386d14` vs `22c94ef`, **2 behind** |
+| worktrees registered / already merged | `git worktree list --porcelain` + `git merge-base --is-ancestor <b> main` | **42** branches, **31** merged |
+| nested worktrees are a gate hazard | `grep -rl '\.claude' conformance/tests platform/monitoring/tests` | **9** scripts carry an explicit exclusion |
+| the 18× double-count | read `platform/monitoring/tests/run.sh:115-123` | in-tree comment, observed 2026-09-12 |
+
+**A claim of mine was wrong and is corrected in the doc.** I had it recorded that
+`git worktree remove` is blocked by the agent sandbox. Provoked rather than assumed: created
+a throwaway detached worktree in the scratchpad and removed it — **rc 0**, and
+`git worktree list` no longer shows it. §0 therefore tells agents to try it, and the
+`rework/<pr-number>` escape hatch is now scoped to *someone else's* stale worktree holding a
+branch, which is a real and separate problem (31 of them).
+
+`spec-immutability` is avoidable-not-tolerated: taken from
+`decisions/2026-10-01-render-selftest-runs-in-gate.md`, which records
+`stage spec-immutability: pass` under the two-mount shim plus `-e SPEC_REPO=/home-network-k3s`.
+
+**Gate — `rc 3`, UNDETERMINED, and I am not calling it green.** Pinned toolbox, from this
+worktree, `DEV_KUBECONFIG=/dev/null` and an `ENGINE` shim bind-mounting both the main
+checkout's `.git` (at its own host path) and the sibling spec repo read-only with
+`-e SPEC_REPO=/home-network-k3s`:
+
+    ENGINE=<two-mount shim> DEV_KUBECONFIG=/dev/null       ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'   # rc 3
+
+| stage | result |
+|---|---|
+| spec-immutability | **pass** — it genuinely RAN; the second mount is what makes it run from a worktree |
+| approval-policy | pass, but **N/A**: `event='none'`. It cannot see a PR locally, so this green says nothing. See below |
+| decisions-log | pass — the three required headings and the frozen-archive pin |
+| lint · l0 · render-golden · l1 | pass (L1: 3 offline checks, 0 failures, 0 skips, 0 unwired) |
+| gitea-settings | **SKIPPED** — `SETTINGS_API_TOKEN` unset; it exists only in CI |
+| conformance (L2/L3) | SKIPPED — `CLUSTER_AVAILABLE!=true`; the decision notes no cluster-needing requirement was touched |
+
+`gitea-settings` is the *only* stage that did not run, and it cannot be made to run from a
+workstation, so rc 3 is the ceiling for any local run — not a defect in this branch. The
+first gate attempt returned rc 3 for a different and real reason (`HEAD is not ahead of
+origin/main`: I had not committed yet, so the touched-requirement set was empty by default
+rather than by verification); it was re-run after committing, and the table above is the
+post-commit run. §3 now records both expected skips, because it previously named only
+`spec-immutability`.
+
+## What I did NOT verify
+
+- **GATE: UNVERIFIED at the time of writing.** The pinned-toolbox result for this branch and
+  its exit code are reported in the PR, not predicted here. **A local rc 3 is not green** and
+  will not be described as one.
+- **CI gate: RED, measured — `MERGE GATE: BLOCKED`, one failing stage.** Run 245 on PR #56,
+  head `b988da8b`, 1m47s. The only failure is `approval-policy`: *"a sensitive change lacks a
+  non-author approval"*, `sensitive paths touched: CLAUDE.md`, author `gitea-admin`. Every other
+  stage passed — `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, and `gitea-settings`
+  (which can only run in CI, and reported `required_approvals = 0`,
+  `dismiss_stale_approvals = true`). `spec-immutability` SKIPPED in CI (no sibling spec repo in
+  the CI workspace — the inverse of the local run, where the two-mount shim made it pass) and
+  `conformance` SKIPPED. **Note `gate / conformance (pull_request)` reports status `success`
+  with description "Has been skipped"** — CLAUDE.md §2b's trap, live on this very PR. Nothing
+  in this branch is broken; the red is the structural human gate, below. *Measured at
+  `b988da8b`; this entry was then amended to record it, so the head SHA in the PR is newer than
+  the run quoted here and CI re-ran on the amended commit.*
+- **This PR is a sensitive-path PR and is CI-RED on `approval-policy` until a human acts.**
+  `CLAUDE.md` is in `infra/gitea/approval-policy.json`'s `sensitive_paths` ("the delivery
+  contract agents follow"), so the stage requires an approving review from someone who is not
+  the author, or an `/approve-override <reason>` posted by an administrator **as a PR review
+  of type "Comment"**, not in the plain comment box. `ci/approval-policy.sh` run locally
+  reports N/A (it cannot see a PR), so the local gate cannot catch this and a green local run
+  says nothing about it. **I have not posted an override and must not** — CLAUDE.md §1 forbids
+  an agent clearing this on its own PR even though `gitea-admin` holds `owner` and could.
+- **Nothing here is enforced.** §0 is prose. There is no gate stage that detects work done in
+  the shared checkout, no check that a worktree is outside the repo root, and no check that a
+  branch was cut from `origin/main`. Every other rule in CLAUDE.md that agents kept breaking
+  (`DECISIONS-LOG.md` appends, the three handoff headings) ended up as a gate stage, and that
+  is the honest follow-up here. I did not write one: the ask was documentation, and an
+  enforcement stage lands in `ci/**`, which is itself a sensitive path and deserves its own PR.
+- **I did not clean up the 31 stale merged worktrees or the 35 nested under
+  `.claude/worktrees/`.** They are other sessions' workspaces; removing them is an operator
+  action, and §0 explicitly tells agents not to remove worktrees they did not create.
+- **I did not touch the untracked `infra/agent/` and `platform/pm-agent/` trees** sitting in the
+  shared checkout. They are another run's in-flight work. Whose, and whether they are wanted,
+  is unknown to me.
+- **`docs/plans/ssd-etcd-execution-runbook.md:18` still points at a `.claude/worktrees/` path**
+  inside the repo. Left as-is deliberately: that runbook is marked COMPLETED/historical and
+  describes the now-decommissioned dev cluster, so the path is a record of what was done, not
+  an instruction. It does mean a grep for in-repo worktree paths is not yet clean.
+- **No runtime, cluster or Flux behaviour was touched or checked.** This branch changes three
+  Markdown files; `kubectl` was not run.

decisions/2026-10-05-cilium-roll-out-on-config-change.md

@@ -0,0 +1,88 @@
+---
+date: 2026-10-05
+branch: feat/cilium-roll-out-on-config-change
+title: Cilium config changes were silently inert — roll the agents when cilium-config changes, which also finally activates the Hubble exporter
+---
+
+## What changed
+
+**A merged Cilium config change did nothing, and everything reported success.** The Hubble flow
+exporter (PR #50, merged 2026-10-04) wrote its settings into `cilium-config` and never took effect.
+
+| Measured 2026-10-05, over a day after the merge | |
+|---|---|
+| HelmRelease `cilium` | `Ready=True`, `UpgradeSucceeded`, *"Helm upgrade succeeded for release kube-system/cilium.v6"* |
+| `cilium-config` keys | `hubble-export-file-path: stdout`, `hubble-export-allowlist: {"destination_port":["32400"]} {"source_port":["32400"]}` — **all present** |
+| cilium agent pods | **all three still the PRE-MERGE pods**, `startTime 2026-10-04T07:11Z` |
+| exported flow lines in agent stdout | **0, 0, 0** |
+| Flux Kustomizations | all `Ready=True` |
+
+Cause: the chart's `rollOutCiliumPods` defaults to **`false`** (cilium 1.20.1 `values.yaml`: *"Roll
+out cilium agent pods automatically when configmap is updated"*), so Helm rewrites the ConfigMap and
+leaves the DaemonSet's pod template untouched — and **cilium-agent reads that ConfigMap only at
+startup**. The feature was merged, believed delivered, green everywhere, and doing nothing at all.
+
+That is the same class of failure this repo already guards against elsewhere: a green signal sitting
+over something that never ran.
+
+- **`platform/cilium/helmrelease.yaml`** (authoritative): `rollOutCiliumPods: true`.
+- **`platform/cilium/values.yaml`** (reference mirror) and
+  **`infra/ansible/roles/cilium/templates/cilium-values.yaml.j2`** (the `[parity]` source of truth):
+  the same, so a bootstrap render does not differ from what Flux reconciles.
+
+**The cost is real and is the point.** Every future Cilium config change now triggers a cluster-wide
+agent rollout instead of waiting for someone to remember a manual restart. That is strictly more
+honest — the change applies when merged, or it fails visibly — but it makes config edits disruptive
+by default, so they want a quiet moment. Mitigation already in place:
+`updateStrategy.rollingUpdate.maxUnavailable: 1` means the DaemonSet will not proceed past an unready
+agent, so a bad render costs one node rather than the cluster.
+
+**Merging this is itself one rollout**, because adding the ConfigMap checksum annotation changes the
+pod template — and that rollout is what finally activates the Hubble exporter.
+
+## What I verified
+
+- **Every row of the table above**, read from the live HelmRelease status, the live `cilium-config`
+  ConfigMap, the agent pods' `startTime`, and a grep for `"flow"` in each agent's stdout.
+- **The chart value name, default and semantics**, quoted from cilium **v1.20.1**'s own
+  `values.yaml` — the version actually running here (`quay.io/cilium/cilium:v1.20.1`), not a guess
+  from another release: `rollOutCiliumPods: false` with the comment *"Roll out cilium agent pods
+  automatically when configmap is updated."*
+- **All three value files carry the key** and both YAML files still parse, with
+  `spec.values.rollOutCiliumPods` resolving to `true`.
+- **The blast-radius mitigation is genuinely present**, not assumed:
+  `updateStrategy.rollingUpdate.maxUnavailable: 1` is already set in the HelmRelease, with a long
+  existing comment explaining that a CNI rollout has no healthy-peer fallback.
+- **THE MECHANISM IS CONFIRMED BY CONTRAST, not just reasoned.** The cgroup fix (PR #42,
+  2026-10-03) DID take effect: both the DaemonSet and the running pod mount
+  `cilium-cgroup hostPath=/sys/fs/cgroup`, matching what the HelmRelease declares. The difference is
+  exactly the one claimed above — #42 changed the POD TEMPLATE (a volume hostPath), which rolls the
+  DaemonSet on its own, while #50 changed ONLY the ConfigMap, which does not. It also explains the
+  agents' `startTime 2026-10-04T07:11Z`: that was #42's rollout, and nothing has rolled them since.
+  So the inertness is specific to ConfigMap-only changes, which is precisely what this flag fixes.
+- **Flux is healthy before this change** — every Kustomization `Ready=True`, so this lands on a clean
+  graph rather than into an existing cascade.
+- **Toolbox gate**: recorded in the PR.
+
+## What I did NOT verify
+
+- **THE OUTCOME, ENTIRELY. Unreconciled.** No agent has rolled, no flow has been exported. This
+  change is an argument that the rollout will happen, not evidence that it did.
+- **Whether `filePath: stdout` actually works** — still the open question from #50, and it cannot be
+  answered until the agents restart. If the literal is not special-cased, the agent may create a
+  FILE named `stdout` and nothing reaches Loki: a silent no-op, not a crash. Fallback documented in
+  #50: keep the chart's default path and add a `loki.source.file` target to Alloy with a hostPath
+  mount. **Check for flow lines in the agent logs immediately after the rollout.**
+- **THE ROLLOUT'S IMPACT ON LAN DNS IS UNMEASURED AND IS THE MAIN RISK.** The node holding the L2
+  lease for the Gateway VIP, **PiHole's DNS VIP** and Plex's VIP loses them briefly as it cycles.
+  `maxUnavailable: 1` bounds it to one node at a time, and BPF maps persist across an agent restart,
+  but I have not measured how long a VIP is unreachable during a lease move, nor whether the two
+  PiHole replicas cover the gap. Merge it at a quiet moment.
+- **Whether enabling this is the operator's preferred trade.** The alternative — leaving it `false`
+  and restarting by hand — keeps config edits non-disruptive at the cost of them being silently
+  inert. I judged silent inertness the worse failure, but it is a judgement, not a measurement.
+- **Whether anything ELSE in `cilium-config` is currently inert.** I checked the one other recent
+  Cilium change and it is fine (see below), but I did not audit every key against the running
+  agents' actual state.
+- **`hubble.relay.rollOutPods` / `hubble.ui.rollOutPods`** are left at their `false` defaults. Both
+  have the same silently-inert property for their own ConfigMaps; not addressed here.

decisions/2026-10-05-oauth2-sidecar-scrape-verified.md

@@ -0,0 +1,90 @@
+---
+date: 2026-10-05
+branch: docs/oauth2-sidecar-scrape-verified
+title: The gate-sidecar scrape is verified working on prod — and four places in the tree still said it wasn't
+---
+
+## What changed
+
+Documentation only. No manifest, no rendered output, no behaviour.
+
+PR #49 merged as `21c88b0` and the fix works, but four places in the tree still asserted the
+*pre-fix* state as current — in the present tense, in capitals, in the files a reader would consult
+first. This is the stale-documentation failure mode this repo has been burned by twice already
+(`git push origin main` in the runbooks, "append to DECISIONS-LOG.md" in the old handoffs), so:
+
+- **`platform/oauth2-proxy/podmonitor.yaml`** — the banner read *"READ THIS BEFORE BELIEVING THE
+  TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY"* and *"SO THE TARGETS BELOW WILL
+  READ `up == 0` UNTIL AN INGRESS ALLOWANCE LANDS"*. Both now false, and the first one actively
+  instructs the reader to disbelieve a correct `up == 1`. Replaced with a `STATUS: THE SCRAPE WORKS`
+  block carrying the post-fix numbers; the diagnosis is kept below it under an explicit `HISTORY`
+  heading, because it is why the fix is shaped the way it is. Added the drop fingerprint to
+  recognise if it regresses — `lastScrapeDuration` pinned at *exactly* the `scrapeTimeout` with
+  `context deadline exceeded`, versus 1–6 ms when healthy.
+- **`catalog/render.py:159`** — *"THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED … Do not
+  read this flag as 'metrics are working'."* Rewritten to enumerate the three pieces a working
+  scrape needs and which PR landed each. The warning is **kept, deliberately**: the flag alone
+  really is insufficient, and "1 without 3" is exactly what caused the 63.7h outage. Only the
+  "not yet scraped" claim was false.
+- **`catalog/render.py:1365`** — the sidecar arg comment, same correction, one line.
+- **`platform/oauth2-proxy/README.md`** — its two observability subsections are *dated*
+  ("measured 2026-09-30", "Update 2026-10-01"), so they are honest records rather than lies. But a
+  reader hits the stale body long before any correction, so a `> SUPERSEDED` pointer now sits
+  directly under the heading, and an `### Update 2026-10-05` section at the end carries the
+  before/after table, the TSDB series counts and the alert outcome.
+
+Two substantive errors fixed while there, not just tense:
+
+1. The README's *"To close the gap for an app, three things are needed"* listed **a Service port
+   and a ServiceMonitor**. Wrong — a PodMonitor targets pod ports directly, which is the whole
+   reason no Service exposes 44180 and the renderer needed no Service change. Following that
+   paragraph would have sent someone to edit `catalog/render.py`'s Service rendering for no reason.
+2. `podmonitor.yaml` ended by deferring the fix to *"the catalog/renderer, which is in flight on
+   another branch"* — that branch does not exist and never did; the work happened in PR #49. It now
+   names `r_monitoring_ingress_policy` directly, and a new `CRAFTY` section explains why crafty is
+   the one namespace this file cannot help.
+
+## What I verified
+
+- **The fix itself, on prod 2026-10-05, read-only** (this is the evidence the docs now quote):
+  Flux applied `21c88b0` through the real path — `apps-wave-0` `lastAppliedRevision` at 21:37:01Z,
+  `apps-wave-1` at 21:37:46Z, in dependency order — and all five `<app>-allow-monitoring-scrape`
+  CNPs exist, each `ports=['44180']`, `from=[['monitoring']]`.
+- **All 6 targets `up == 1`**, `health=up`, `lastError=''`, `lastScrapeDuration` 0.001–0.006s —
+  against 10.0s and `context deadline exceeded` before.
+- **`oauth2_proxy_*` is in the TSDB for these namespaces for the first time:** 5 metric names,
+  148 series; `sum by (namespace) (oauth2_proxy_requests_total)` = qbittorrent 35072, radarr 23560,
+  sonarr 1284. (Queried with `count by (__name__)` plus a matcher, not label-values.)
+- **Alerts:** all 5 `TargetDown` and 5 of 6 `OAuth2ProxyGateUnobserved` resolved; one series
+  remains, `OAuth2ProxyGateUnobserved{namespace="crafty"}`, firing.
+- **The rendered output is byte-identical.** `catalog/render.py` changed only in comments, so this
+  branch touches no `apps/` file and no golden. Confirmed by `render-golden` and `l1` both passing
+  with no regeneration, and by `git status` showing nothing under `apps/`.
+- **This branch touches no sensitive path**, unlike PR #49: `platform/oauth2-proxy/**`,
+  `catalog/**` and `decisions/**` are none of them in `infra/gitea/approval-policy.json`. So it
+  should merge on a green gate with no human approval — the opposite of the PR it documents, and
+  the reason the two were kept apart.
+- `render.py --selftest`: PASS.
+- Toolbox gate: see the handoff's own rule — reported below rather than claimed here.
+
+## What I did NOT verify
+
+- **I did not re-measure the 63.7h figure or the ~22 emails/day rate**; both are carried forward
+  from the 2026-10-04 measurements, now quoted inside the docs as historical. The 15-day
+  `query_range` that established "one episode, not flapping" was run once, on 2026-10-04.
+- **`oauth2_proxy_requests_total` is still absent for jackett and pihole.** I assert in the README
+  that this is because the counter is not created until the gate serves its first request. That is
+  the documented behaviour of a Prometheus counter with no initialisation, and it is consistent
+  with the other three namespaces having traffic — but I did not force a request through jackett's
+  gate and watch the series appear. If it is still absent after real use, that is a finding.
+- **The auth-churn rule is still unwritten.** The README now says it is "writable rather than
+  decorative"; that is a claim about the metrics being present, not a rule anyone has authored or
+  unit-tested. The thing the 2026-09-30 incident actually needed does not exist yet.
+- **crafty remains unobserved.** I read its container spec on 2026-10-05 and confirmed no
+  `--metrics-address`; I did not attempt the fix, and nothing here changes its alert.
+- **I did not audit the whole tree for other stale "metrics are not scraped" claims** beyond the
+  four sites a targeted grep found (`not yet reachable`, `UNTIL AN INGRESS`, `up == 0`,
+  `not reachable by Prometheus` and neighbours). `runbooks/oauth2-proxy-auth-failures.md` is
+  described in the README as "the by-hand substitute meanwhile" and I left it alone — the by-hand
+  log procedure still works and is not wrong, but whether it should now be rewritten against the
+  new metrics is unexamined.

decisions/2026-10-05-plex-resolver-independent-advertise.md

@@ -0,0 +1,105 @@
+---
+date: 2026-10-05
+branch: fix/plex-resolver-independent-advertise
+title: Plex down LAN-wide — the advertised name existed only in PiHole, so clients that bypass it landed on the router's %D cert
+---
+
+## What changed
+
+**I took Plex down for the whole LAN.** Reported symptom: every internal client showed a certificate
+error with `%D` as the subject name and could not connect.
+
+**Root cause: the advertised connection depended on our own resolver.** `ADVERTISE_IP` was
+`http://plex-vip.${BASE_DOMAIN}:32400` — a name that exists ONLY in PiHole's split-horizon answers
+and in no public zone. A Plex client resolves the advertised host *itself*, with whatever resolver it
+happens to use, and on a real LAN that is frequently not PiHole: browsers ship DoH on by default,
+phones have "private DNS", guest/IoT devices take the router's DNS. Those clients could not resolve
+the name, the SOHO router answered the failed lookup with **itself**, and they all landed on the
+router's web UI.
+
+| Measured 2026-10-05 | Result |
+|---|---|
+| `192.168.2.1:443` (router, LAN side) | `C = ZZ, ST = Somewhere, O = %D342e7e6b, CN = %D` |
+| `136.51.115.252:443` (router, WAN side) | the same `%D` cert |
+| PiHole **pihole-0** | `plex -> 192.168.2.20`, `plex-vip -> 192.168.2.24` ✅ |
+| PiHole **pihole-1** | identical ✅ (both behind the `.23` VIP) |
+| Gateway `.20:443` cert | `CN = *.home.keithjackson.dev`, valid to 2026-12-12 ✅ |
+| `http://192.168.2.24:32400/identity` | `200` ✅ |
+| Plex's published connections | all port **32400** — nothing on 443 |
+
+**The DNS was right; the dependency on it was the bug.** PiHole answered correctly from both
+replicas the entire time. And because nothing Plex advertises listens on 443, the clients had plainly
+left the Plex path altogether and were talking to the router.
+
+**The fix: go back to `<dashed-ip>.<serverhash>.plex.direct`, which is RESOLVER-INDEPENDENT.** This
+is Plex's own mechanism and the reason it exists — `plex.direct` is a public zone that returns the
+private address embedded in the label, so *every* resolver (PiHole, the router, 1.1.1.1, a DoH
+browser) answers `192-168-2-24.<hash>.plex.direct` with `192.168.2.24`. It needs no split-horizon
+entry, no Cloudflare record and no certificate work, because the wildcard cert Plex already serves on
+32400 covers exactly that name.
+
+- **`catalog/plex.yaml`:** `ADVERTISE_IP` →
+  `https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400`, with the
+  resolver-independence requirement written at the top of the block as the first thing to check
+  before anyone changes this field again.
+- **PiHole's `plex-vip` answer is KEPT.** It is harmless, costs nothing, and is useful for reaching
+  the VIP by name from a LAN shell. It is simply not what Plex advertises.
+
+**Why not keep the nicer hostname.** It cannot be made correct for clients that bypass PiHole: a
+public record for it resolves to the WAN address, and with no NAT hairpin on this router a LAN client
+sent there fails too. The name is right only for PiHole users and wrong for everyone else, and there
+is no configuration of it that is right for both. `plex.direct` is right for both.
+
+**The honest cost, restated rather than buried:** the value embeds both the VIP and the server's
+certificate hash as a literal — envsubst cannot dash an IP and nothing in the tree can derive the
+hash, so nothing checks that they still agree. That fragility is exactly why the hostname was tried.
+It is accepted because an unresolvable name is a *worse* failure: silent, LAN-wide, and it points
+users at their router.
+
+## What I verified
+
+Live prod, 2026-10-05, read-only.
+
+- **Every row of the table above**, including the router's `%D` cert on both its LAN and WAN sides —
+  which is what identifies the router as what the clients were reaching.
+- **THE FIX ITSELF, with full TLS validation (no `curl -k`)**: the plex.direct name resolves to
+  `192.168.2.24` and `/identity` returns **200, TLS verified**. Re-checked today, not carried over
+  from yesterday's measurement.
+- **Both PiHole replicas are identical and correct**, with the same five `address=` lines live in
+  `/etc/pihole/dnsmasq.conf`, and both IPs present behind the `pihole-dns-tcp`/`-udp` endpoints. I
+  checked only pihole-0 at first and deliberately went back for pihole-1, because a split-brain pair
+  would have produced exactly this intermittent-looking symptom.
+- **The cluster was never the problem:** all eight LoadBalancer VIPs still assigned, L2
+  announcements intact (prod-3 holds `.20`/`.23`/`.24`), `wildcard-home` Certificate `Ready=True`,
+  and PR #51 had reconciled (`customConnections="http://plex-vip.home.keithjackson.dev:32400"` was
+  live, confirming the bad value was in effect).
+- **`LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16"` is still set**, so the postStart hook survived
+  the roll and keeps re-asserting.
+- **The Cilium rollout from #50 STILL has not happened** — agents are unchanged since
+  2026-10-04T07:11, so the Hubble export is inactive and `filePath: stdout` remains unproven.
+
+## What I did NOT verify
+
+- **THE OUTCOME. Unreconciled.** No client has been observed connecting after this change. The
+  argument is that the advertised name now resolves for every resolver and serves a matching
+  certificate — both measured — but I have not seen a Plex client complete a connection, and I
+  cannot see the user's clients at all.
+- **THE CLIENT-SIDE RESOLVER PATH IS INFERRED, NOT OBSERVED.** I could not query a public resolver
+  from here — my attempt returned NXDOMAIN even for a `plex.direct` name that certainly resolves, so
+  the probe was broken (egress policy or a missing tool) and I discarded it rather than report it.
+  The chain "client bypasses PiHole → NXDOMAIN → router answers with itself → `%D`" is the only
+  explanation consistent with all the measurements, but the NXDOMAIN-hijack step in particular is
+  deduced from the router serving `%D` on its LAN address, not from watching a lookup.
+- **Which clients bypass PiHole, or how many.** If in fact every client uses PiHole, then my
+  explanation is wrong and something else sent them to the router — in which case the next thing to
+  check is whether the clients were using `plex.home.keithjackson.dev` (port 443, the Gateway) rather
+  than an advertised connection at all.
+- **Whether plex.tv will re-publish promptly** after the value changes, and whether stale connection
+  lists are cached on clients. A client holding the old list may keep failing until it refreshes.
+- **Mixed content.** The previous `http://` value would also have been blocked by the browser for
+  anyone using `https://app.plex.tv`; that is fixed incidentally by returning to `https`, but I did
+  not test it.
+- **The Remote badge — the symptom that began this whole sequence — remains unconfirmed.**
+  `LanNetworksBandwidth` is set, but Plex still tags traffic `Allowed Network (WAN)`. Unchanged and
+  still unexplained.
+- **`ManualPortMappingMode="1"` with an absent port** is still there, untouched.

decisions/README.md

@@ -26,7 +26,8 @@ The fix is the obvious one for concurrent appenders: **stop sharing a file**. Ea
 handoff is its own file. Two agents writing handoffs touch disjoint paths, so there is
 nothing for git to merge and nothing to conflict. This scales to any number of parallel
 branches with zero coordination — which is the point, since the agents are deliberately
-isolated from each other in separate worktrees and cannot coordinate.
+isolated from each other in separate worktrees (required by `CLAUDE.md` §0) and cannot
+coordinate.
 
 There is deliberately **no index file** and no counter. An index would be a shared
 mutable file — exactly the thing that just failed — and a sequence number would have

infra/ansible/roles/cilium/templates/cilium-values.yaml.j2

@@ -39,6 +39,12 @@ bpf:
   masquerade: true
 
 # Flow observability / isolation audit (ADR-0002).
+# Roll the agents when cilium-config changes. Kept EQUAL to the Flux values
+# (platform/cilium/helmrelease.yaml), which carry the full rationale: the chart default of false
+# leaves the pod template untouched on a config change, and cilium-agent reads that ConfigMap only
+# at startup, so the change is silently inert until someone restarts the DaemonSet by hand.
+rollOutCiliumPods: true
+
 hubble:
   enabled: true
   relay:

platform/cilium/helmrelease.yaml

@@ -188,6 +188,37 @@ spec:
         enabled: false
       hostRoot: /sys/fs/cgroup
     # --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident ---
+    # --- Roll the agents when their ConfigMap changes (2026-10-05) ---
+    # [parity] MUST equal Ansible: rollOutCiliumPods: true
+    #
+    # WHY: a merged Cilium config change DID NOTHING. The chart default is `false`
+    # (cilium 1.20.1 values.yaml: "Roll out cilium agent pods automatically when configmap is
+    # updated. rollOutCiliumPods: false"), so Helm rewrites `cilium-config` and leaves the
+    # DaemonSet's pod template untouched — and cilium-agent reads that ConfigMap only at STARTUP.
+    # The result is a silently inert change: Flux green, HelmRelease `UpgradeSucceeded`, the new
+    # keys visibly present in the ConfigMap, and the running agents still on the old settings
+    # until a human remembers to restart them.
+    #
+    # MEASURED, and it is why this exists: the Hubble flow exporter merged 2026-10-04 wrote
+    # `hubble-export-file-path: stdout` and its allowlist into `cilium-config`, the HelmRelease
+    # reported `Helm upgrade succeeded ... cilium.v6`, and over a DAY later all three agents were
+    # still the pre-merge pods (startTime 2026-10-04T07:11Z) with ZERO exported flow lines. The
+    # feature was merged, believed delivered, and doing nothing at all. That is the same class of
+    # failure this repo already documents elsewhere — a green signal over something that never ran.
+    #
+    # THE COST IS REAL AND IS THE POINT: every future Cilium config change now triggers a
+    # cluster-wide agent rollout instead of waiting for a manual restart. That is strictly more
+    # honest — the change applies when it is merged, or it fails visibly — but it means config
+    # edits are disruptive by default, so they want a quiet moment. One node at a time:
+    # `updateStrategy.rollingUpdate.maxUnavailable: 1` below means the DaemonSet will not proceed
+    # past an unready agent, so a bad render costs one node rather than the cluster. Note that the
+    # node holding the L2 lease for the Gateway, PiHole-DNS and Plex VIPs loses them briefly as it
+    # cycles, so a rollout can blip LAN DNS.
+    #
+    # ENABLING THIS IS ITSELF ONE ROLLOUT: adding the ConfigMap checksum annotation changes the pod
+    # template, so merging this rolls the agents once — which is what finally activates the Hubble
+    # exporter that has been inert since 2026-10-04.
+    rollOutCiliumPods: true
     # [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1.
     # The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate.
     # maxUnavailable: 2) — NOT a local choice, and on a 3-node cluster that lets a single bad

platform/cilium/values.yaml

@@ -32,6 +32,15 @@ enableIPv4Masquerade: true
 bpf:
   masquerade: true
 # [parity] flow observability / isolation audit
+# [parity] roll the agents when cilium-config changes. The chart default (false) rewrites the
+# ConfigMap but leaves the pod template alone, and cilium-agent only reads it at STARTUP — so a
+# merged config change is SILENTLY INERT. Measured 2026-10-04..05: the Hubble exporter keys were
+# live in cilium-config and the HelmRelease said UpgradeSucceeded, while all three agents were
+# still pre-merge pods exporting nothing for over a day. Cost: config edits now roll the agents
+# (one node at a time via maxUnavailable: 1), which can briefly blip the VIPs on the lease-holding
+# node. Full rationale in helmrelease.yaml.
+rollOutCiliumPods: true
+
 hubble:
   enabled: true
   relay:

platform/oauth2-proxy/README.md

@@ -91,6 +91,10 @@ This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on
 
 ## Observability: what this component can and cannot see (measured 2026-09-30)
 
+> **SUPERSEDED — read "Update 2026-10-05" at the end of this section first.** The gate sidecars
+> **are** scraped now and the TSDB **does** hold `oauth2_proxy_*` for them. The two subsections
+> below are kept as the dated record of how this was diagnosed, not as current state.
+
 `servicemonitor.yaml` scrapes **this namespace's** deployment and nothing else. On prod that is
 two targets, `job="oauth2-proxy"`, both `up=1`. The complete set of oauth2-proxy-specific series
 in the TSDB is:
@@ -157,3 +161,55 @@ present `up` series. A discovered-but-failing target publishes `up` with value 0
 would have cleared `OAuth2ProxyGateUnobserved` the moment the PodMonitor landed — reporting the gates
 as observed while nothing was being scraped. The unit test pins that scenario; deleting the `== 1`
 fails it and nothing else.
+
+### Update 2026-10-05: the scrape works
+
+PR #49 (merged `21c88b0`) added the missing third piece. `catalog/render.py`'s
+`r_monitoring_ingress_policy` now also fires for the gate sidecar's metrics port, keyed on
+`_needs_oauth2_sidecar`, rendering `<app>-allow-monitoring-scrape` per app: ns `monitoring` → that
+app's `endpointSelector` → TCP 44180 only. Five objects, one each for jackett, pihole, qbittorrent,
+radarr, sonarr.
+
+Measured on prod 2026-10-05 after Flux applied it (`apps-wave-0` 21:37:01Z, `apps-wave-1`
+21:37:46Z):
+
+| | before | after |
+|---|---|---|
+| target health (6) | `down` | `up` |
+| `lastError` | `context deadline exceeded` | `''` |
+| `lastScrapeDuration` | **10.0s** — exactly the `scrapeTimeout` | **0.001–0.006s** |
+| `up` | `0` × 6 | `1` × 6 |
+
+    count by (__name__) ({__name__=~"oauth2_proxy_.*",
+                          namespace=~"jackett|radarr|sonarr|qbittorrent|pihole"})
+      oauth2_proxy_requests_in_flight              => 6
+      oauth2_proxy_requests_total                  => 16
+      oauth2_proxy_response_duration_seconds_bucket => 108
+      oauth2_proxy_response_duration_seconds_sum   => 9
+      oauth2_proxy_response_duration_seconds_count => 9
+
+    sum by (namespace) (oauth2_proxy_requests_total)
+      qbittorrent => 35072    radarr => 23560    sonarr => 1284
+
+`oauth2_proxy_requests_total` is absent for jackett and pihole simply because that counter is not
+created until the gate has served its first request.
+
+**So the auth-churn rule is now writable rather than decorative** — sustained `code="302"` as a
+share of `oauth2_proxy_requests_total`, held across a window longer than the session lifetime. That
+is the rule the 2026-09-30 incident above could not be caught by. It is still unwritten.
+
+**Alert outcome:** all 5 `TargetDown` resolved, and 5 of 6 `OAuth2ProxyGateUnobserved` with them.
+At `repeat_interval: 12h` with `group_by: [alertname, namespace]`, those 11 groups had been
+re-notifying at roughly **22 emails/day** since 2026-10-02; that drops to 2/day.
+
+**`crafty` still fires, and that is correct.** It is a hand-maintained Deployment under
+`platform/crafty`, outside the renderer, whose `oauth2-proxy` container has **no
+`--metrics-address` at all** — no listener, so no port to open and nothing for the PodMonitor's
+pod-label selector to match. Closing it needs the flag and a `metrics` containerPort on crafty's
+own manifest first, then a hand-written allowance beside it. Open follow-up work.
+
+**The paragraph above headed "To close the gap for an app" is superseded**: a *Service* port and a
+ServiceMonitor are **not** required. A PodMonitor targets pod ports directly, which is why no
+Service exposes 44180 and why the renderer needed no Service change. The three things actually
+needed are `--metrics-address` + named container port, a PodMonitor, and the monitoring-ingress
+allowance.

platform/oauth2-proxy/podmonitor.yaml

@@ -27,10 +27,28 @@
 # created by the apps-wave-0/1 tiers, which this tier does not and must not depend on.
 #
 # ============================================================================================
-# READ THIS BEFORE BELIEVING THE TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY
+# STATUS: THE SCRAPE WORKS. Verified on prod 2026-10-05 (PR #49 merged as 21c88b0).
 # ============================================================================================
-# This PodMonitor is NECESSARY AND NOT SUFFICIENT. The endpoint serves, and the path from ns
-# monitoring is blocked. Both halves measured on prod 2026-10-01, read-only:
+# All 6 targets in pool podMonitor/oauth2-proxy/oauth2-proxy-sidecars/0 read `up == 1`,
+# health=up, lastError='', lastScrapeDuration 0.001-0.006s. The TSDB holds oauth2_proxy_* for
+# these namespaces for the first time: 5 metric names / 148 series, with live counters
+# (oauth2_proxy_requests_total 35072 qbittorrent, 23560 radarr, 1284 sonarr). All 5 TargetDown
+# and 5 of 6 OAuth2ProxyGateUnobserved resolved; only namespace="crafty" still fires, and that
+# is correct — see CRAFTY below.
+#
+# The missing piece was an ingress allowance, and it now ships WITH EACH APP, rendered by
+# catalog/render.py's r_monitoring_ingress_policy as `<app>-allow-monitoring-scrape`:
+# ns monitoring -> the app's endpointSelector -> TCP 44180 only. See WHY NOT IN THIS DIRECTORY
+# below for why it is rendered per-app instead of shipped from this tier.
+#
+# Keep the history below. It is why the fix is shaped the way it is, and `up == 1` in
+# auth-visibility-alerts.yaml is load-bearing precisely because of it.
+#
+# --------------------------------------------------------------------------------------------
+# HISTORY: 2026-10-02 -> 2026-10-05, THE SCRAPE WAS DENIED BY NETWORKPOLICY FOR 63.7h
+# --------------------------------------------------------------------------------------------
+# This PodMonitor was NECESSARY AND NOT SUFFICIENT: the endpoint served, and the path from ns
+# monitoring was blocked. Both halves measured on prod 2026-10-01, read-only:
 #
 #   THE ENDPOINT IS HEALTHY — port-forward straight to the jackett pod's 44180:
 #     GET /metrics -> http=200 bytes=10208
@@ -54,19 +72,24 @@
 #   `jackett-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180 — the Gateway's Envoy
 #   and nothing else. Prometheus matches no clause and its packets are dropped, not refused.
 #
-# SO THE TARGETS BELOW WILL READ `up == 0` UNTIL AN INGRESS ALLOWANCE LANDS. That is a known,
-# measured, documented state — NOT a regression introduced here, and NOT something to silence.
-# It is deliberately left visible: `OAuth2ProxyGateUnobserved` in auth-visibility-alerts.yaml
+# THE TARGETS READ `up == 0` FROM 2026-10-02 05:09 UTC UNTIL THE ALLOWANCE LANDED — one
+# continuous 63.7h episode (a 15d query_range shows 1 episode per namespace, not flapping),
+# starting 11 minutes after this file was created, i.e. TargetDown's `for: 10m`. It was never
+# silenced, and that was the right call: `OAuth2ProxyGateUnobserved` in auth-visibility-alerts.yaml
 # requires `up == 1`, precisely so a discovered-but-failing target does not read as "observed"
 # (see that file's header — this exact trap is why the `== 1` is there).
 #
+# The signature to recognise if it ever returns: lastScrapeDuration pinned at EXACTLY the
+# scrapeTimeout (10.0s) with `context deadline exceeded`. That is a DROP, not a refusal — a
+# refused connection fails in milliseconds. Post-fix the same targets scrape in 1-6ms.
+#
 # THIS IS THE SAME BUG LONGHORN HAD, and the platform has a documented fix idiom for it:
 # platform/longhorn/networkpolicy-metrics.yaml, whose header records the identical fingerprint
 # measured on 2026-09-13 — ServiceMonitor in place, 3 targets DISCOVERED, `up` 0/0/0, TSDB empty,
 # `lastError: context deadline exceeded` (a drop, not a refusal). The fix there is a narrow
 # additive `networking.k8s.io/v1` NetworkPolicy: ns monitoring -> the pod label, that one port.
 #
-# WHY THAT POLICY IS NOT IN THIS DIRECTORY. It would have to be five objects, one per app
+# WHY NOT IN THIS DIRECTORY (the reasoning the fix followed). It would have to be five objects, one per app
 # namespace, and those namespaces are created by apps-wave-0/1. Measured Flux graph, 2026-10-01:
 #   oauth2-proxy  dependsOn=['platform-config']        <- this tier
 #   apps-wave-0   dependsOn=['platform-config']        <- SIBLING, no ordering between them
@@ -76,8 +99,19 @@
 # running cluster and break on a rebuild — the "works now, broken on restore" shape this repo
 # keeps paying for. Longhorn's precedent is safe only because longhorn-system is created by the
 # same tier that ships its policy. The allowance therefore belongs WITH THE APPS, rendered next to
-# the rest of each app's network policy — i.e. in the catalog/renderer, which is in flight on
-# another branch. Tracked in the handoff; it is one narrow rule per app, not a redesign.
+# the rest of each app's network policy — i.e. in the catalog/renderer. THAT IS WHERE IT LANDED:
+# `r_monitoring_ingress_policy`, keyed on `_needs_oauth2_sidecar`, one narrow rule per app.
+#
+# --------------------------------------------------------------------------------------------
+# CRAFTY: STILL UNOBSERVED, AND THIS FILE CANNOT HELP IT
+# --------------------------------------------------------------------------------------------
+# `OAuth2ProxyGateUnobserved{namespace="crafty"}` still fires after the fix, correctly. crafty is
+# a hand-maintained Deployment under platform/crafty, OUTSIDE the renderer, and its oauth2-proxy
+# container has NO `--metrics-address` at all (measured 2026-10-05) — so its metrics listener
+# never starts and there is no port to open. The selector below therefore does not match it, and
+# the per-app allowance is never rendered for it. Closing crafty's gap needs the flag plus a
+# `metrics` containerPort added to its own manifest FIRST; the allowance is the second half, and
+# it would have to be written by hand there rather than rendered. Open follow-up work.
 apiVersion: monitoring.coreos.com/v1
 kind: PodMonitor
 metadata: