fix(runbooks): vpn-remote-access step 1 named a nonexistent script; step 2 cited the wrong gitignore rule

Both reported by the operator on first use of the runbook merged in #55. Step 1 said `cd infra/ansible && ./site.sh --tags system --limit k3s_prod`. There is no site.sh anywhere in this repo. It was wrong four ways at once: this repo drives Ansible with `ansible-playbook <playbook>` and never a wrapper (playbooks/site.yml is the full bring-up chain, not this); the module work is in playbooks/node-baseline.yml; targeting here is `-e k3s_hosts=...`, not `--limit`; and Ansible exists only in the pinned toolbox, with ansible.cfg read only from infra/ansible. Replaced with the real command plus why `-e k3s_hosts=k3s_prod` matters (the default group is decommissioned dev) and why `--tags system` must never be dropped (an untagged run also executes the `disks` tasks, which assert on prod's deliberate placeholder devices). Step 2 told the operator to stage the server key at /work/.cache/server.key "inside the repo's gitignored cache". .cache/ is NOT gitignored -- the file was ignored only incidentally by `*.key` (.gitignore:11). So the path was safe for a reason I had not identified, and renaming it (server.pem, wg-server) would have left a committable WireGuard private key in the checkout, which no-plaintext-secrets does not recognise. The step now names the real rule, marks the suffix as the load-bearing part, and gives a check-ignore verification to run before trusting it. Docs only -- no manifest and no behaviour change. Touches no sensitive path. Also records the post-merge measurements for #55, whose own entry predates the merge: Flux applied ad3f56e by the real path, all 18 Kustomizations Ready, hostNetwork confirmed (pod IPs are the node IPs), and the designed fail-closed state (Init:0/1 on the missing wg-server-key Secret). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Merge pull request 'docs(agents): one workspace per agent run — the shared checkout is off-limits' (#56 ) from docs/agents-work-in-their-own-workspace into main
2026-10-05 23:08:19 -05:00 · 2026-10-06 03:49:28 +00:00 · 2026-10-06 03:41:48 +00:00 · 2026-10-05 22:31:40 -05:00 · 2026-10-05 22:29:46 -05:00
20 changed files with 1850 additions and 7 deletions
@@ -4,6 +4,125 @@ Read this before doing anything durable. It is the delivery contract for AI codi
 sessions and agents. The newest entry in `decisions/` is the current PM handoff and
 takes precedence on *what* to work on; this file governs *how* it lands.

+**Start with §0.** It is first because it is the rule you can break before you have read
+anything else: the very first `git checkout -b` in the wrong directory has already done the
+damage.
+
+## 0. Work in your OWN workspace. Never in the shared checkout.
+
+Before you create a branch, edit a file, or run `git` anything: **make yourself a git
+worktree outside the repository root, and do all of your work there.** One workspace per
+agent run, no exceptions.
+
+    # from the shared checkout, ONCE, at the start of your run
+    git fetch origin main
+    git worktree add -b <type>/<slug> /path/outside/the/repo/wt-<slug> origin/main
+    cd /path/outside/the/repo/wt-<slug>        # everything after this happens HERE
+
+`/path/outside/the/repo` means outside this repository's tree — your session scratchpad,
+or a sibling like `/var/home/keith/source/wt-<slug>`. Branch from **`origin/main`**, never
+from `main` and never from `HEAD`; see "stale base" below.
+
+**The shared checkout is `/var/home/keith/source/home-network-k3s-impl`** — the operator's
+working copy, the one you are probably started in. In it, do **not**:
+
+- **`git checkout` / `git switch` / `git checkout -b`.** Its HEAD is shared state; moving
+  it moves the tree out from under whoever else is using it.
+- **edit, create or delete a file.** Including "just one doc", including a file you intend
+  to move later.
+- **`git add` / `git commit` / `git stash` / `git restore`.** `git add -A` there stages
+  whatever another agent left lying around; `git stash` *removes* their uncommitted work
+  into a place they will not look.
+- **`git worktree remove` anything you did not create.**
+
+Read-only commands there are fine, and the one write you do need is creating your own
+worktree.
+
+### Why — measured in the shared checkout, 2026-10-05
+
+| What was measured | Value |
+|---|---|
+| `checkout: moving` entries in its reflog | **107** — the tree has been yanked between branches 107 times |
+| the branch it was sitting on | `feat/pm-agent-issue-to-pr` — **not `main`** |
+| uncommitted trees left in it by other runs | **2** (`infra/agent/`, `platform/pm-agent/`), mtimes five minutes apart the same evening |
+| its local `main` vs `origin/main` | **two commits behind** (`4386d14` vs `22c94ef`) |
+| registered worktrees | **42**, of which **31** are on branches already merged to `main` |
+| gate scripts carrying an explicit `.claude/` exclusion | **9** |
+
+Each row is a distinct way the shared checkout damages work:
+
+- **Dirty tree, foreign files.** Two other runs' uncommitted directories were sitting in
+  it. An agent that edits there and runs `git add -A` ships someone else's half-finished
+  work inside its own PR, and the PR's author has no idea what those files are. This is not
+  hypothetical and it is not historical: **eleven minutes after those two directories were
+  measured, while this very section was being written, another run committed them** —
+  `5334abe feat(pm-agent): ...`, 1,000+ lines across `infra/agent/`, `clusters/`, `catalog/`
+  and `apps/` — onto the shared checkout's checked-out branch, moving its HEAD under anyone
+  reading it. Had this rule's author been working in that tree rather than in a worktree,
+  that commit and this one would have collided.
+- **Stale base.** Its `main` was two commits behind `origin/main`, so
+  `git checkout -b x main` there branches off a tree that is already not what CI will merge
+  into — the gate you run locally proves a base nobody else has. Worse, it was not even *on*
+  `main`: a bare `git checkout -b x` would have branched off `feat/pm-agent-issue-to-pr` and
+  silently inherited that work into your PR. `git fetch` first, and name `origin/main`
+  explicitly as the start point.
+- **Shared HEAD.** 107 branch switches in one tree is 107 chances to switch it while
+  another agent is mid-edit. The loser's changes are either carried onto the wrong branch
+  or lost to a checkout conflict — and the symptom surfaces in *their* session, not yours.
+
+### Put the workspace OUTSIDE the repo root
+
+A worktree *inside* the tree (`.claude/worktrees/agent-*/` — which is where Claude Code's
+own default puts it) is a full second copy of the repo that every tree-walking check then
+finds. Measured 2026-09-12: `find`-based discovery reported **the same manifest under test
+once per worktree, up to 18 times**, and the L0 check comparing that listing against the
+tree failed on paths that are not authored surface at all
+(`platform/monitoring/tests/run.sh:115-123`). Nine gate scripts now carry an explicit
+`.claude/` prune for this reason — which means the hazard is *worked around*, not fixed,
+and the next check that walks the tree will hit it again. Keep the copy out of the tree and
+none of that applies.
+
+### Clean up — and expect not to be able to
+
+31 of the 42 registered worktrees are on branches already merged to `main`. A registered
+worktree **pins its branch**: while it exists, that branch cannot be deleted or checked out
+anywhere else. So a left-behind workspace is a lock on a branch name, and the cost lands on
+whoever next needs it.
+
+Remove yours when you are done:
+
+    git worktree remove /path/outside/the/repo/wt-<slug>
+
+Measured 2026-10-05: on a worktree **you created yourself, in the session scratchpad**, that
+command **succeeds** — rc 0, no prompt, with or without a branch checked out. The "worktree
+removal is sandbox-blocked" note in circulation is about the `.claude/worktrees/agent-*` ones
+belonging to *other* sessions, some of which are additionally `locked` on git's own terms. So:
+remove yours, and only yours. If removal is refused, say so in your handoff and leave the
+worktree registered — do **not** `rm -rf` the directory, which leaves a registered worktree
+pointing at nothing and still holding the branch.
+
+If someone *else's* stale worktree is holding the branch you need, do not try to reclaim the
+branch name. Push a `rework/<pr-number>` branch to the PR's refspec instead.
+
+### If you are already standing in the shared checkout with edits
+
+Stop. Do not `git stash` (that moves other agents' work too). Create your worktree as
+above, redo the edits there, and leave the shared checkout's HEAD and working tree exactly
+as you found them.
+
+### Gating from a worktree
+
+Three known breakages, all with workarounds, all required every run: see §3, "Running the
+gate from a worktree".
+
+### If you are writing a brief for another agent
+
+The brief must name **its own explicit branch** and tell it to create **its own worktree**
+from `origin/main`. "Work on the current branch" and "continue in the checkout" are how
+branch sprawl and cross-contamination got here: an agent given no workspace uses the one it
+is started in, which is the shared one. Also carry §3's three gate workarounds into the
+brief — an agent that cannot gate will report rc 3 as green.
+
 ## 1. Never push to `main`. Ever.

 Every durable change reaches `main` exactly one way:
@@ -147,15 +266,21 @@ Exit codes are a contract — do not flatten them:
 | 3 | UNDETERMINED — a stage did not run. **Never report this as green.** |

 `spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out
-beside this one; that single skip is expected and tolerated. Any *other* skip is a
-real gap in your run.
+beside this one; that single skip is expected and tolerated — and from a worktree it is
+avoidable outright (§3's worktree list, item 3). `gitea-settings` is the other expected
+skip: it needs `SETTINGS_API_TOKEN`, which exists only in CI, so **a local run cannot
+return 0** — the honest best local result is rc 3 with `gitea-settings` as the only stage
+that did not run. Any *further* skip is a real gap in your run. Report the rc and name the
+skipped stages; never round rc 3 up.

 **Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`,
 so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts.

-### Running the gate from a git worktree
+### Running the gate from a worktree

-Two unrelated failures, both with known workarounds — put both in any agent brief:
+This is the **normal** path, not an edge case — §0 requires you to be in a worktree, so
+every gate run you do hits all of these. Three known breakages, all with
+workarounds, all of them yours to apply every run — put them in any agent brief:

 1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked
   and exists only in the main checkout. Do **not** copy it (it is a credential).
@@ -166,10 +291,17 @@ Two unrelated failures, both with known workarounds — put both in any agent br
   the `ENGINE` hook — a shim that intercepts `run` and re-execs
   `podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in
   `tracked_files()` is this bug, not a broken tree.
+3. **`spec-immutability` skips** — the skip called tolerated above is also *avoidable*. `dev.sh`
+   derives the sibling spec path from its own `REPO_ROOT`, which in a worktree is your
+   workspace, so `../home-network-k3s` does not resolve and it adds no mount. Mount the spec
+   repo read-only in the same `ENGINE` shim and pass `-e SPEC_REPO=/home-network-k3s`: with
+   both mounts the stage reports `pass`, not SKIPPED, and R-TEST-06 is actually covered
+   (measured — `decisions/2026-10-01-render-selftest-runs-in-gate.md`).

 ## 4. Branch naming

-Use `feat/`, `fix/`, `docs/`, `chore/`. Note that `.gitea/workflows/gate.yml`'s **push**
+Use `feat/`, `fix/`, `docs/`, `chore/`. **One branch, one workspace, one PR** — if you
+need a second branch, make a second worktree (§0); do not re-point an existing one. Note that `.gitea/workflows/gate.yml`'s **push**
 trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run.
 The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected.

@@ -11,6 +11,9 @@ specification** at [`../home-network-k3s`](../home-network-k3s).
 - **Done = a green, reproducible, requirement-tagged conformance test** — never "merged," never a
  manual demo (spec `docs/14`, `agents/definition-of-done.md`).
 - **No work on `main`.** Every change is a feature branch merged into `main` (spec `CONTRIBUTING.md`).
+- **No work in a shared checkout.** Every agent run gets its own git worktree, outside the repo
+  root, branched from `origin/main`. This checkout is shared mutable state — see
+  [`CLAUDE.md` §0](CLAUDE.md).
 - **No plaintext secrets** — SOPS-encrypted only (spec `docs/07`).
 - **Phase discipline** — only build what the active roadmap phase authorizes (spec `roadmap/phases.md`).

@@ -21,7 +24,8 @@ infra/          node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible,
 bootstrap/      one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
 platform/       platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
                longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
-                vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
+                vpn-egress, vpn-ingress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
+                (vpn-egress = pods OUT via PIA; vpn-ingress = remote clients IN via WireGuard)
 platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
                (ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
                Split out because CRs and the CRDs they use cannot apply in one pass.
@@ -94,6 +98,20 @@ The running narrative — decisions, root causes, incidents, follow-ups — is

 ## Working on this repo

+**First, get out of this checkout.** It is the operator's working copy and several agents
+share it; concurrent runs here have overwritten each other's branches and left each other's
+uncommitted files behind. One workspace per change:
+
+```bash
+git fetch origin main
+git worktree add -b <type>/<slug> /path/outside/the/repo/wt-<slug> origin/main
+cd /path/outside/the/repo/wt-<slug>
+```
+
+Outside the repo root specifically — a worktree under `.claude/worktrees/` is a second full
+copy that tree-walking gate checks then count twice. The rule, the measurements behind it, and
+the three gate workarounds a worktree needs are in [`CLAUDE.md` §0](CLAUDE.md) and §3.
+
 Everything the gate needs is baked into the pinned dev container — **do not install the
 toolchain on the host**:

@@ -102,6 +102,7 @@ data:
  # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers;
  # plex-remote + pihole: PLEX_L4_VIP),
+  # vpn-ingress (WG_CLIENT_CIDR),
  # velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
  # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
  # UNSUBSTITUTED path would ship the literal string into the cluster.
@@ -207,6 +208,31 @@ data:
  # prod -> 192.168.2.0/24  (prod LAN; nodes .10/.11/.12)
  LAN_CIDR: "192.168.1.0/24"                 # ingress scope for the PiHole DNS listener

+  # --- Remote-access VPN client subnet (platform/vpn-ingress, WireGuard) --------------------------
+  # The tunnel subnet handed to REMOTE VPN clients. `.1` is the server (the hostNetwork DaemonSet,
+  # on every node); `.2`-`.254` are devices, allocated one /32 each in
+  # platform/vpn-ingress/peers.yaml. Consumed twice in the pod's config-assembly script: to derive
+  # the server's own address, and as the `-s` match of the MASQUERADE rule that makes LAN replies
+  # routable.
+  #
+  # IT IS A KNOB RATHER THAN A LITERAL FOR ONE REASON: it is the only address range in this cluster
+  # that must be checked against EVERY OTHER range at once, so it belongs where those are declared.
+  # 10.90.0.0/24 was chosen to collide with none of them:
+  #     10.0.0.0/16     Cilium cluster-pool pod CIDR     (${CLUSTER_POD_CIDR})
+  #     10.43.0.0/16    k3s service CIDR                 (measured live 2026-10-05: kubernetes svc
+  #                                                       clusterIP 10.43.0.1)
+  #     10.72.192.0/18  the PIA tunnel's pushed addresses (platform/vpn-egress)
+  #     192.168.1.0/24  dev LAN / pfSense LAN            (${LAN_CIDR} base default)
+  #     192.168.2.0/24  prod LAN                         (${LAN_CIDR} on prod)
+  #
+  # NO PER-ENVIRONMENT OVERRIDE, and that is deliberate rather than an omission: unlike every VIP
+  # above, this range is NOT drawn from the LAN. It is private to the tunnel, so the same value is
+  # correct in both environments and a prod patch would be a no-op that implies otherwise.
+  #
+  # IF YOU CHANGE IT, every already-issued client profile breaks (its `Address` is from this range)
+  # and must be re-rendered — so treat it as fixed once the first device is enrolled.
+  WG_CLIENT_CIDR: "10.90.0.0/24"             # remote VPN clients; .1 = server. NOT a LAN range
+
  # --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
  # THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media
  # library (Storage/plex, 8.33 T — plex, sonarr, radarr) and the legacy Nextcloud archive
@@ -71,4 +71,11 @@ resources:
  # controller must NEVER gate the platform chain; same non-gating rationale as headlamp/oauth2-proxy.
  # Carries velero-cloud.sops.yaml, so clusters/dev adds a SOPS decryption patch for it (by name).
  - backup.yaml          # Flux Kustomization -> ./platform/velero (dependsOn platform-config)
+  # WireGuard remote-access VPN (north-south INGRESS, udp/51820) — POST-platform tier
+  # (dependsOn platform-config, wait:false). NOT the same thing as platform/vpn-egress, which sends
+  # pods OUT through PIA; this lets remote CLIENTS IN. Non-gating is load-bearing rather than
+  # conventional here: its DaemonSet cannot start until the operator creates the out-of-band
+  # `wg-server-key` Secret, so in the gating `platform` tier an unfinished manual step would have
+  # stalled every apps wave. Carries no SOPS secret, so neither overlay adds a decryption patch.
+  - vpn-ingress.yaml     # Flux Kustomization -> ./platform/vpn-ingress (dependsOn platform-config)
  - cluster-vars.yaml    # overlay knob surface (base defaults)
@@ -0,0 +1,56 @@
+---
+# Flux Kustomization: WireGuard remote-access VPN — reconciles ./platform/vpn-ingress.
+#
+# POST-PLATFORM TIER (dependsOn platform-config), NON-GATING (wait:false). Not a line in
+# platform/kustomization.yaml, for the standard reason plus one specific to this component:
+#
+#   * STANDARD: a leaf must never sit in the gating `platform` tier. While `platform` is not-Ready
+#     NOTHING downstream reconciles — the incident behind the headlamp/oauth2-proxy/homepage moves
+#     (DECISIONS-LOG: "Headlamp gating stall").
+#   * SPECIFIC, AND THE REASON THIS IS NOT A JUDGEMENT CALL: this DaemonSet cannot start until the
+#     operator creates the out-of-band `wg-server-key` Secret (runbooks/vpn-remote-access.md
+#     step 1). That is the correct fail-closed state — an unenrolled VPN must not come up — but in
+#     the gating tier it would have been a self-inflicted platform-wide outage ON MERGE, with the
+#     cause (a missing manual step) invisible from any Flux condition.
+#
+# WHY dependsOn platform-config AND NOT cilium. This component needs no Gateway, no TLS cert, no
+# Keycloak and no LB-IPAM pool — it has no HTTPRoute and no LoadBalancer Service, and hostNetwork
+# means it does not even need a Cilium endpoint. It could legally reconcile much earlier. It is
+# pinned behind platform-config anyway so that the ONE thing remote clients actually consume
+# through it — split-horizon DNS and the shared Gateway — exists by the time the tunnel does.
+# A tunnel that comes up before the things on the far side of it is a support call, not a feature.
+#
+# wait:false: nothing in the graph dependsOn this, and its pods are legitimately not-Ready whenever
+# the server key is absent or the `wireguard` kernel module has not been loaded on a node. Neither
+# condition should be able to gate anything else, even if a future resource is mistakenly pointed
+# at it.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: vpn-ingress
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: false
+  dependsOn:
+    - name: platform-config
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/vpn-ingress
+  # REQUIRED, not boilerplate: platform/vpn-ingress/daemonset.yaml renders ${WG_CLIENT_CIDR} into
+  # the config-assembly script. Without this block that token becomes the EMPTY STRING and the
+  # Kustomization still reports Ready (Flux's substitution fails OPEN — ci:
+  # flux-postbuild-substitution), leaving an initContainer that derives a server address of ".1/"
+  # and a MASQUERADE rule with no source. Both fail loudly at pod start, but the Kustomization
+  # would look healthy.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # NO `decryption:` stanza is needed in the overlays: this path carries no *.sops.yaml. The server
+  # key is an out-of-band Secret (platform/vpn-ingress/kustomization.yaml says what to do if that
+  # ever changes).
@@ -0,0 +1,147 @@
+---
+date: 2026-10-05
+branch: docs/agents-work-in-their-own-workspace
+title: One workspace per agent run — the shared checkout is now documented as off-limits
+---
+
+## What changed
+
+Agents have been doing their work in the operator's checkout at
+`/var/home/keith/source/home-network-k3s-impl` and clobbering each other. Nothing in the tree
+told them not to: worktrees appeared in CLAUDE.md only as §3's *gate workarounds*, i.e. as an
+awkward edge case, and `decisions/README.md` asserted worktree isolation as a fact about how
+agents already behave rather than as a rule they must follow. Nowhere was the rule stated.
+
+Documentation only — no code, no manifests, no pipeline logic.
+
+- **`CLAUDE.md`: new `## 0. Work in your OWN workspace. Never in the shared checkout.`**
+  Placed before §1 so the section numbers every other file and handoff entry cites (§1, §2b,
+  §3, §7) all keep their meaning. It gives the three commands (`git fetch origin main`;
+  `git worktree add -b <type>/<slug> <path outside the repo> origin/main`; `cd`), an explicit
+  do-not list for the shared checkout (no `checkout`/`switch`, no edits, no
+  `add`/`commit`/`stash`/`restore`, no removing others' worktrees), a measurement table, the
+  "put it outside the repo root" rule, cleanup, what to do if you are already standing in the
+  shared checkout with edits, and what a PM must put in a brief it writes for another agent.
+- **`CLAUDE.md` intro** now says "Start with §0", with the reason: it is the rule you can
+  break before you have read anything else.
+- **`CLAUDE.md` §3** retitled "Running the gate from a worktree" and reframed as the normal
+  path rather than an edge case, since §0 now makes every gate run a worktree run. Its list
+  grew from two breakages to three: `spec-immutability`'s skip is *avoidable*, not merely
+  tolerated, if the `ENGINE` shim also mounts the sibling spec repo read-only and passes
+  `-e SPEC_REPO=/home-network-k3s`.
+- **`CLAUDE.md` §4** gained "One branch, one workspace, one PR".
+- **`README.md`**: a fourth non-negotiable rule ("No work in a shared checkout"), and
+  "Working on this repo" now opens with how to get out of the checkout before describing the
+  dev container that runs in it.
+- **`decisions/README.md`**: the worktree-isolation sentence now cites §0 as the requirement
+  instead of describing it as something agents happen to do.
+
+I did this work the way the new §0 requires — a worktree at
+`<session scratchpad>/wt-workspace-rule`, outside the repo root, branched from `origin/main`,
+with the shared checkout left on the branch and with the untracked files I found in it.
+
+## What I verified
+
+Every number in the new §0 table was measured in the shared checkout on 2026-10-05, not
+recalled:
+
+| Claim | Command | Value |
+|---|---|---|
+| shared HEAD is yanked between branches | `git reflog \| grep -c 'checkout: moving'` | **107** |
+| it is not sitting on `main` | `git worktree list` | `feat/pm-agent-issue-to-pr` |
+| other runs' uncommitted work is in it | `git status --porcelain` | **2** trees: `infra/agent/` (22:04), `platform/pm-agent/` (22:09) — `stat -c %y` |
+| its `main` is stale | `git rev-parse main` vs `origin/main` | `4386d14` vs `22c94ef`, **2 behind** |
+| worktrees registered / already merged | `git worktree list --porcelain` + `git merge-base --is-ancestor <b> main` | **42** branches, **31** merged |
+| nested worktrees are a gate hazard | `grep -rl '\.claude' conformance/tests platform/monitoring/tests` | **9** scripts carry an explicit exclusion |
+| the 18× double-count | read `platform/monitoring/tests/run.sh:115-123` | in-tree comment, observed 2026-09-12 |
+
+**The failure mode was observed live, mid-session, not just reconstructed.** The two untracked
+trees I measured at 22:04 and 22:09 were committed at 22:20 by another run — `5334abe`
+`feat(pm-agent): an in-cluster AI Program Manager...`, 1,000+ lines across `infra/agent/`,
+`clusters/`, `catalog/`, `apps/`, `docs/plans/` and its own `decisions/` entry — onto the shared
+checkout's checked-out branch. Measured after the fact: the shared checkout's HEAD moved
+`22c94ef` → `5334abe` and its working tree went clean between my first `git status` and my last,
+with `git log -1` naming a commit that is not mine. That is §0's "shared HEAD" row happening
+during the session that wrote §0, and it is now cited in the doc.
+
+**A claim of mine was wrong and is corrected in the doc.** I had it recorded that
+`git worktree remove` is blocked by the agent sandbox. Provoked rather than assumed: created
+a throwaway detached worktree in the scratchpad and removed it — **rc 0**, and
+`git worktree list` no longer shows it. §0 therefore tells agents to remove their own, and
+scopes the "blocked" note to what it actually covers: *other* sessions'
+`.claude/worktrees/agent-*`, some of which are additionally `locked` on git's own terms. The
+`rework/<pr-number>` escape hatch is now scoped to someone else's stale worktree holding a
+branch, which is a real and separate problem (31 of them). Both of my own worktrees in this
+session were removed cleanly at rc 0 after pushing.
+
+`spec-immutability` is avoidable-not-tolerated: taken from
+`decisions/2026-10-01-render-selftest-runs-in-gate.md`, which records
+`stage spec-immutability: pass` under the two-mount shim plus `-e SPEC_REPO=/home-network-k3s`.
+
+**Gate — `rc 3`, UNDETERMINED, and I am not calling it green.** Pinned toolbox, from this
+worktree, `DEV_KUBECONFIG=/dev/null` and an `ENGINE` shim bind-mounting both the main
+checkout's `.git` (at its own host path) and the sibling spec repo read-only with
+`-e SPEC_REPO=/home-network-k3s`:
+
+    ENGINE=<two-mount shim> DEV_KUBECONFIG=/dev/null       ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'   # rc 3
+
+| stage | result |
+|---|---|
+| spec-immutability | **pass** — it genuinely RAN; the second mount is what makes it run from a worktree |
+| approval-policy | pass, but **N/A**: `event='none'`. It cannot see a PR locally, so this green says nothing. See below |
+| decisions-log | pass — the three required headings and the frozen-archive pin |
+| lint · l0 · render-golden · l1 | pass (L1: 3 offline checks, 0 failures, 0 skips, 0 unwired) |
+| gitea-settings | **SKIPPED** — `SETTINGS_API_TOKEN` unset; it exists only in CI |
+| conformance (L2/L3) | SKIPPED — `CLUSTER_AVAILABLE!=true`; the decision notes no cluster-needing requirement was touched |
+
+`gitea-settings` is the *only* stage that did not run, and it cannot be made to run from a
+workstation, so rc 3 is the ceiling for any local run — not a defect in this branch. The
+first gate attempt returned rc 3 for a different and real reason (`HEAD is not ahead of
+origin/main`: I had not committed yet, so the touched-requirement set was empty by default
+rather than by verification); it was re-run after committing, and the table above is the
+post-commit run. §3 now records both expected skips, because it previously named only
+`spec-immutability`.
+
+## What I did NOT verify
+
+- **GATE: UNVERIFIED at the time of writing.** The pinned-toolbox result for this branch and
+  its exit code are reported in the PR, not predicted here. **A local rc 3 is not green** and
+  will not be described as one.
+- **CI gate: RED, measured — `MERGE GATE: BLOCKED`, one failing stage.** Run 245 on PR #56,
+  head `b988da8b`, 1m47s. The only failure is `approval-policy`: *"a sensitive change lacks a
+  non-author approval"*, `sensitive paths touched: CLAUDE.md`, author `gitea-admin`. Every other
+  stage passed — `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, and `gitea-settings`
+  (which can only run in CI, and reported `required_approvals = 0`,
+  `dismiss_stale_approvals = true`). `spec-immutability` SKIPPED in CI (no sibling spec repo in
+  the CI workspace — the inverse of the local run, where the two-mount shim made it pass) and
+  `conformance` SKIPPED. **Note `gate / conformance (pull_request)` reports status `success`
+  with description "Has been skipped"** — CLAUDE.md §2b's trap, live on this very PR. Nothing
+  in this branch is broken; the red is the structural human gate, below. *Measured at
+  `b988da8b`; this entry was then amended to record it, so the head SHA in the PR is newer than
+  the run quoted here and CI re-ran on the amended commit.*
+- **This PR is a sensitive-path PR and is CI-RED on `approval-policy` until a human acts.**
+  `CLAUDE.md` is in `infra/gitea/approval-policy.json`'s `sensitive_paths` ("the delivery
+  contract agents follow"), so the stage requires an approving review from someone who is not
+  the author, or an `/approve-override <reason>` posted by an administrator **as a PR review
+  of type "Comment"**, not in the plain comment box. `ci/approval-policy.sh` run locally
+  reports N/A (it cannot see a PR), so the local gate cannot catch this and a green local run
+  says nothing about it. **I have not posted an override and must not** — CLAUDE.md §1 forbids
+  an agent clearing this on its own PR even though `gitea-admin` holds `owner` and could.
+- **Nothing here is enforced.** §0 is prose. There is no gate stage that detects work done in
+  the shared checkout, no check that a worktree is outside the repo root, and no check that a
+  branch was cut from `origin/main`. Every other rule in CLAUDE.md that agents kept breaking
+  (`DECISIONS-LOG.md` appends, the three handoff headings) ended up as a gate stage, and that
+  is the honest follow-up here. I did not write one: the ask was documentation, and an
+  enforcement stage lands in `ci/**`, which is itself a sensitive path and deserves its own PR.
+- **I did not clean up the 31 stale merged worktrees or the 35 nested under
+  `.claude/worktrees/`.** They are other sessions' workspaces; removing them is an operator
+  action, and §0 explicitly tells agents not to remove worktrees they did not create.
+- **I did not touch the untracked `infra/agent/` and `platform/pm-agent/` trees** sitting in the
+  shared checkout. They are another run's in-flight work. Whose, and whether they are wanted,
+  is unknown to me.
+- **`docs/plans/ssd-etcd-execution-runbook.md:18` still points at a `.claude/worktrees/` path**
+  inside the repo. Left as-is deliberately: that runbook is marked COMPLETED/historical and
+  describes the now-decommissioned dev cluster, so the path is a record of what was done, not
+  an instruction. It does mean a grep for in-repo worktree paths is not yet clean.
+- **No runtime, cluster or Flux behaviour was touched or checked.** This branch changes three
+  Markdown files; `kubectl` was not run.
@@ -0,0 +1,225 @@
+---
+date: 2026-10-05
+branch: feat/vpn-remote-access-wireguard
+title: Remote-access VPN — WireGuard on udp/51820, hostNetwork because a pod identity would 403 every Gateway hostname
+---
+
+## What changed
+
+New platform component `platform/vpn-ingress/`: a **WireGuard** remote-access VPN so the operator's
+Android / Linux / Windows / macOS devices can dial in from anywhere on the internet and behave like
+machines on the LAN — full reach to `192.168.2.0/24` plus internet egress out the house WAN.
+
+This is the **opposite direction** to the existing `platform/vpn-egress` (pods *out* through PIA,
+ADR-0015). They share nothing but the word "VPN"; both READMEs now say so, and `README.md`'s layout
+block labels the two.
+
+**Transport: WireGuard, and the security argument is the handshake's failure mode, not "it is
+encrypted".** WireGuard's first handshake message carries a MAC keyed with the responder's own
+public key, so a sender that does not already know it gets **no reply at all** — no error, no reset,
+no banner. From the internet, udp/51820 is indistinguishable from a closed port: nothing to
+brute-force, no version string, no amplification. That is what makes an unfiltered WAN port
+defensible here, and it is why there is deliberately no `loadBalancerSourceRanges`-style allowlist
+(which would also break roaming clients). It also has first-party clients on all four named
+platforms, which OpenVPN (needs a CA + per-device certs + a CRL) and IKEv2 do not.
+
+**`hostNetwork: true`, and this is THE decision — forced by a measured property of this cluster.**
+A pod-networked WireGuard server must SNAT client traffic to its own pod IP (Cilium drops a packet
+whose source is not the endpoint's address), so every remote client would have arrived wearing *that
+pod's* security identity. `decisions/2026-10-04-plex-advertise-plexdirect.md` measured what that
+costs: a request to the shared Gateway VIP **from inside a pod returns `403 Access denied` from
+Envoy for all 16 Gateway hostnames**, because Cilium's Gateway preserves the original source
+identity and enforces the *destination's* ingress policy, and every app admits `reserved:ingress`
+plus named peers — never an arbitrary pod. So the obvious build would have produced a tunnel that
+came up healthy, showed connected peers, routed packets, and 403'd every cluster web UI — and
+`*.home.keithjackson.dev` would not even have resolved, because `apps/wave-0/pihole-dns.yaml`'s
+ingress policy deliberately omits the `cluster` entity. In the host netns there is no pod identity
+to carry: forwarded traffic is SNATed to the node address and arrives as `host`/`remote-node`,
+byte-for-byte what a LAN client looks like. Three further problems vanish with it: `ip_forward` is
+already 1 on these nodes (so no `--allowed-unsafe-sysctls` kubelet change — it is *not* a safe
+sysctl); no LoadBalancer VIP, so no L2-announcement lease lottery and no `externalTrafficPolicy`
+trade-off; and no double encapsulation, which is what makes `MTU = 1420` simply correct.
+
+`hostNetwork` is permitted by `restrict-host-namespaces`' own stated carve-out (it matches
+`app-namespace=true` only, and names Cilium/node-exporter as the precedent); `namespace.yaml` is
+deliberately unlabelled. **The cost is stated, not buried:** a hostNetwork pod has no Cilium
+endpoint, so **no NetworkPolicy constrains this workload**. `networkpolicy.yaml` says that at
+length rather than shipping an `allow world → 51820` rule that would read as the control and
+enforce nothing; the real boundary is WireGuard's peer authentication.
+
+**A DaemonSet, not a Deployment.** The WAN forward names one node address by hand, off-repo (same
+shape as the plex tcp/32400 forward). A floating single replica would silently invalidate it on
+every reschedule; an identical listener on all three nodes makes **failover a one-line router
+change** with no cluster work and no re-issued client profiles. A pinned LoadBalancer VIP was the
+considered alternative and was rejected: it reintroduces the L2 lease, SNAT and an extra node hop
+into the one path whose failure mode is "I am away from home and cannot get in".
+
+**Authorisation is split so that it stays reviewable.** Peer **public** keys live in
+`platform/vpn-ingress/peers.yaml` in **plaintext** — a public key grants nothing by itself, and this
+makes "who can get into my house" a PR diff with `git log` as its audit trail. The only real secret,
+the server's private key, is the `wg-server-key` Secret, **created out of band by the operator**: an
+agent must not mint a long-lived private key (CLAUDE.md §6), and a key that passed through a session
+transcript would have to be treated as compromised. The two halves are joined only at pod start, by
+an init container, into a `medium: Memory` emptyDir — so the assembled config holding the key never
+touches a node's disk and dies with the pod.
+
+**There are no peers yet, and that is the correct initial state**, for the same reason. The VPN ships
+INERT-BUT-HEALTHY: a zero-peer interface listens and refuses every handshake.
+
+Files:
+
+- `platform/vpn-ingress/` (new): `namespace.yaml`, `peers.yaml` (the plaintext peer register),
+  `daemonset.yaml` (hostNetwork listener + config assembly), `networkpolicy.yaml` (namespace
+  baseline, with the honest caveat), `kustomization.yaml`, `README.md` (the design record).
+- `clusters/base/vpn-ingress.yaml` (new): its own **non-gating post-platform** Flux Kustomization
+  (`dependsOn platform-config`, `wait:false`). Non-gating is load-bearing rather than conventional
+  here: the DaemonSet *cannot* start until the operator creates the server key, so in the gating
+  `platform` tier an unfinished manual step would have stalled every apps wave on merge.
+- `clusters/base/kustomization.yaml`: the new tier added.
+- `clusters/base/cluster-vars.yaml`: new `WG_CLIENT_CIDR` (`10.90.0.0/24`), declared next to
+  `LAN_CIDR` because it is the one range that must be collision-checked against every other; the
+  comment enumerates the five it avoids. **No prod override, deliberately** — unlike every VIP, this
+  range is private to the tunnel, so the same value is correct in both environments and a prod patch
+  would imply otherwise. Substitution inventory updated.
+- `infra/ansible/roles/node_baseline/defaults/main.yml`: `wireguard` added to
+  `node_baseline_kernel_modules`. The pod has `NET_ADMIN` but deliberately **not** `SYS_MODULE`, so
+  it cannot autoload the module — it CrashLoops with a clear error rather than quietly degrading to
+  userspace wireguard-go (which would need `/dev/net/tun`, a device a hostPath mount cannot actually
+  grant, as the `/dev/dri` attempt proved).
+- `infra/vpn-ingress/new-peer.sh` (new): enrols a device. Generates the keypair on the operator's
+  machine, writes a 0600 profile **outside** the checkout (it refuses to write into the repo),
+  prints a QR code for phones, and prints only the public `[Peer]` stanza. It also refuses a
+  duplicate address, because two devices on one address does not error anywhere — the server routes
+  to whichever handshook last, so both appear to work intermittently.
+- `runbooks/vpn-remote-access.md` (new) + its `kustomization.yaml` generator entry + `README.md`
+  index line: the three manual steps, the four off-LAN verification checks, revocation, and why no
+  connection alert exists.
+- `README.md`: `vpn-ingress` added to the layout, with the egress/ingress distinction spelled out.
+
+**THREE MANUAL STEPS REMAIN AND THE VPN DOES NOT WORK UNTIL ALL THREE ARE DONE** (all off-repo by
+necessity; `runbooks/vpn-remote-access.md`): run `node_baseline` to load the kernel module, create
+the `wg-server-key` Secret, and forward WAN udp/51820 to one node address on the SOHO router.
+
+**SENSITIVE PATHS — this PR touches `clusters/**` (three files), so `ci/approval-policy.sh` will
+BLOCK it.** It needs the operator's `/approve-override <reason>` posted as a PR **review** of type
+"Comment". I have deliberately not posted it: CLAUDE.md §1 forbids an agent clearing this on its own
+PR, and the agent account holds `owner`, so the prohibition is a rule, not a limit.
+
+## What I verified
+
+Live prod, read-only, 2026-10-05, plus the offline gate.
+
+- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
+  `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
+  **pass**. The rc 3 is two environmental skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`)
+  and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3, not as pass.**
+  Run from an isolated git worktree with both CLAUDE.md §3 workarounds applied
+  (`DEV_KUBECONFIG=/dev/null` plus an `ENGINE` shim bind-mounting the main checkout's `.git` and
+  the sibling spec repo) — which is why `spec-immutability` is **green here and skipped in CI**,
+  where the runner has no spec checkout. `approval-policy` passing locally is an offline N/A, not a
+  verdict; the real verdict is the CI run below.
+- **The CI gate ran on PR #55 (run 244, head `1a67fc1b`) and `approval-policy` is the ONLY failing
+  stage** — read from the job log, not from the red tick. Verbatim: "This PR changes the pipeline or
+  the security posture it rests on, and has NO approving review from anyone other than its author
+  ('gitea-admin')", naming exactly the three files I predicted:
+  `clusters/base/cluster-vars.yaml`, `clusters/base/kustomization.yaml`,
+  `clusters/base/vpn-ingress.yaml`. Every other stage that ran reported green —
+  `gitea-settings`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1`. So the branch is blocked
+  on the operator's `/approve-override <reason>` (posted as a PR **review** of type "Comment") and
+  on nothing else. **I have not posted it** — CLAUDE.md §1.
+- **Branch protection is live, measured by CI's own `gitea-settings` stage** (it has the token this
+  workstation lacks): `required_approvals = 0`, `enable_status_check = true`,
+  `status_check_contexts = ["gate / gate (pull_request)"]`, `block_admin_merge_override = true`,
+  `enable_push = false`, `dismiss_stale_approvals = true`, and **1 runner online**. That is the §2b
+  re-measurement of CLAUDE.md §2's table for this branch, not a restatement of it.
+- **`flux-postbuild-substitution` (L0) actually exercised the new wiring, and caught a real defect
+  in it.** On the first run it FAILED, naming `platform/vpn-ingress/daemonset.yaml:118` and
+  attributing it to "Flux Kustomization 'vpn-ingress' (clusters/base/vpn-ingress.yaml, path
+  ./platform/vpn-ingress)". Two things are proven by that message: the new tier's
+  `postBuild.substituteFrom` declaration is correctly detected (so `${WG_CLIENT_CIDR}` will be
+  substituted, not shipped literally), and the check reads comments as well as values — the hazard
+  was a comment that spelled out the forbidden braced form literally while claiming not to use it.
+  Fixed; the check now passes 444/444 with 0 hazards.
+- **The `wireguard` kernel module is available on these nodes by construction**: all three run
+  Debian 13 (trixie) on kernel `7.1.8+deb13-amd64` (measured), and WireGuard has been in-tree since
+  5.6, so it is present in the distro kernel and only needs loading.
+- **`net.ipv4.ip_forward = 1` is already written by `node_baseline`**
+  (`roles/node_baseline/tasks/main.yml`, `/etc/sysctl.d/99-k3s-cilium.conf`), and **no kubelet
+  carries `--allowed-unsafe-sysctls`** (grepped across `roles/k3s_server`, `roles/node_baseline` and
+  `group_vars`: zero hits). That pair is what makes the host netns the only option that does not
+  require a node-restart change.
+- **Cilium's datapath, measured from `cilium-config` rather than from the values file**:
+  `enable-ipv4-masquerade=true`, `enable-bpf-masquerade=true`, `routing-mode=tunnel`,
+  `tunnel-protocol=vxlan`, `ipv4-native-routing-cidr` EMPTY, Cilium `v1.20.1`. The vxlan underlay is
+  where `MTU = 1420` comes from, and the empty native-routing CIDR is why pod→LAN is masqueraded
+  today.
+- **`WG_CLIENT_CIDR` collides with nothing.** Pod CIDR `10.0.0.0/16` (`CLUSTER_POD_CIDR`), service
+  CIDR `10.43.0.0/16` (measured: `kubernetes` Service clusterIP is `10.43.0.1`), the PIA tunnel's
+  `10.72.192.0/18` (`platform/vpn-egress/README.md`), and both LANs `192.168.1.0/24` /
+  `192.168.2.0/24`. `10.90.0.0/24` is outside all five.
+- **The egress-class label matters and is set to `internet`.** Read live:
+  `CiliumEgressGatewayPolicy vpn-egress-steering` selects `podSelector:
+  homelab.keithjackson.dev/egress-class: vpn` with **no namespace scope**, so `vpn` here would have
+  steered every remote client's internet traffic out PIA.
+- **All three nodes are untainted** (`spec.taints: None` on each), so the DaemonSet covers the
+  cluster with no tolerations.
+- **`new-peer.sh` was run end to end** against a throwaway server key in the scratchpad (deleted
+  immediately after; it never existed in the repo). It rendered a correct profile — `Address
+  10.90.0.7/32`, `DNS 192.168.2.23`, `Endpoint wan.home.keithjackson.dev:51820`, `MTU 1420`,
+  `PersistentKeepalive 25`, `AllowedIPs 0.0.0.0/0` — at mode 0600, printed the matching public-key
+  stanza, and both argument-validation paths (missing name, out-of-range octet) refuse correctly.
+  The private key was redacted from every line I displayed.
+- **The endpoint name is resolver-independent, which is the specific trap this avoids.**
+  `wan.${BASE_DOMAIN}` is an unproxied public Cloudflare A record kept current by
+  `platform/externaldns/ddns.yaml` (checked every 5 min), so it resolves from a cellular connection
+  — unlike any name PiHole answers, which a remote client cannot resolve until it is already
+  connected. (This is the `plex.direct` lesson applied.)
+- **Velero does cover this namespace.** `platform/velero/helmrelease.yaml`'s `daily` schedule
+  excludes only `monitoring` and `unifi`, so `vpn-ingress` and the `wg-server-key` Secret are
+  captured off-cluster to R2. Read from the manifest, not from a backup listing.
+
+## What I did NOT verify
+
+- **THE ACTUAL FIX. Nothing in this branch has been proven to carry a single packet.** The
+  DaemonSet has never been reconciled, no `wg0` has ever existed, and no handshake has ever been
+  attempted. Three blockers are off-repo and manual (kernel module, server key, WAN forward), and
+  this branch removes none of them — it only makes them the *only* remaining work.
+- **Outside-in reachability, at all.** This repo has **no off-LAN vantage point**, which is exactly
+  the gap the plex remote-access work hit. Step 6 of the runbook (a phone on cellular) is the only
+  honest test and it has not been run.
+- **The in-pod `iptables` → host-netfilter interaction, which is my biggest single doubt.** Debian 13
+  uses nf_tables; the container writes MASQUERADE/FORWARD rules into the **host's** tables via the
+  image's own `iptables`. If that binary is the legacy backend while the host is nft, the rules can
+  apply to a table the kernel does not consult — and the symptom would be the nastiest available:
+  handshakes succeed, `wg show` looks perfect, and every forwarded connection hangs. **Check this
+  first if the tunnel comes up but nothing is reachable:** `iptables-save -t nat | grep MASQUERADE`
+  and `nft list table ip nat` on the node, from the host, not from the pod.
+- **Whether `allowPrivilegeEscalation: false` is compatible with the image's s6 init.** It is the
+  correct posture and I set it, but this repo has already been bitten by `no-new-privileges`
+  misbehaving (`platform/externaldns/ddns.yaml`'s README note). If the pod fails before
+  `wg-quick` runs, that flag is the first thing to try flipping.
+- **Whether `NET_ADMIN` alone is sufficient.** Reasoned from what `wg-quick` does (create the link,
+  set a sysctl, write two netfilter rules), not measured. If `iptables` complains, `NET_RAW` is the
+  likely addition — deliberately not granted pre-emptively.
+- **The image's custom-config path.** `lscr.io/linuxserver/wireguard:1.0.20260223-r0-ls124` (tag
+  confirmed to exist and to be the current one, published 2026-10-01) is expected to bring up
+  whatever is in `/config/wg_confs/` when `PEERS` is unset. That is its documented behaviour; it has
+  not been run here, and the init container's assembly of that file has never executed.
+- **Whether the probe is right.** `wg show wg0` is asserted to exit non-zero when the interface is
+  absent. Not executed.
+- **Cilium's reaction to a new `wg0` in the host netns.** `platform/cilium/values.yaml` does not pin
+  `devices`, so Cilium auto-detects; whether it attaches to `wg0` — and whether that matters — is
+  unknown. `wg` interfaces are already filtered out by `node_health_collect.py`, so something in
+  the estate anticipates them, but that is not evidence about Cilium.
+- **Zero peers means the end-to-end authorisation path is untested**, including revocation. The
+  first enrolment is also the first test of `peers.yaml`'s format being what `wg-quick` accepts.
+- **No peer-level observability exists.** An exporter publishing
+  `wireguard_latest_handshake_seconds` is the top follow-up; until then "who is connected" is an
+  on-demand `wg show` and nothing alerts on an unexpected session. Pod-down is covered only by
+  kube-prometheus-stack's generic `KubePodNotReady` / `KubeDeploymentReplicasMismatch`, which I did
+  not confirm fire for a DaemonSet in this namespace.
+- **The security consequence I am most keen for a human to review rather than take from me:** an
+  enrolled device reaches the three k3s nodes themselves — apiserver 6443, kubelet 10250, SSH 22 —
+  because "reach the entire local network" includes them. That is stated in the README and the
+  runbook with the narrowing recipe, but it is a posture decision, not a verified fact, and it is
+  the right thing to argue about on this PR.
@@ -0,0 +1,112 @@
+---
+date: 2026-10-05
+branch: fix/vpn-runbook-ansible-command
+title: vpn-remote-access runbook — step 1 named a script that does not exist, and step 2 cited the wrong reason the key is gitignored
+---
+
+## What changed
+
+Two defects in `runbooks/vpn-remote-access.md` (merged in #55), both authored by me, both reported
+by the operator the first time they tried to follow it.
+
+**1. Step 1 named a command that does not exist.** It said:
+
+    cd infra/ansible && ./site.sh --tags system --limit k3s_prod
+
+There is **no `site.sh` anywhere in this repo** (`find . -name site.sh` → nothing). The invocation
+was wrong in four separate ways at once, which is what makes it worth recording rather than just
+patching: wrong kind of thing (this repo drives Ansible with `ansible-playbook <playbook>`, never a
+wrapper script — `playbooks/site.yml` exists and is the full bring-up chain, not this); wrong
+playbook (the module work lives in `playbooks/node-baseline.yml`); wrong targeting flag (`--limit`
+instead of this repo's `-e k3s_hosts=...` convention, which `infra/ansible/README.md` "Targeting
+prod" documents for every playbook); and no toolbox wrapper, though Ansible exists only inside the
+pinned container and `ansible.cfg` is only read from `infra/ansible`.
+
+Replaced with the real command, plus the three things about it that are load-bearing:
+
+    ./infra/devcontainer/dev.sh run bash -c \
+      'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system'
+
+- `-e k3s_hosts=k3s_prod` — the playbook defaults to `k3s_servers`, which is the **dev** group;
+  dev is decommissioned, so an unqualified run does nothing useful.
+- `--tags system`, never untagged — the two kernel-module tasks carry that tag, and an untagged run
+  would also hit the `disks` tasks, which assert on block-device variables prod deliberately leaves
+  as placeholders (the playbook's own header warns about this). `--tags system` is broader than the
+  modules alone (sysctls, iscsid, qemu-guest-agent, swap masking) but each is idempotent and
+  already in its desired state on a live node.
+
+**2. Step 2 gave the wrong reason the server key is safe from git — and the right reason is
+fragile.** It told the operator to stage the key at `/work/.cache/server.key` "inside the repo's
+gitignored cache". `.cache/` is **not** gitignored: `git check-ignore -v .cache/serverkey.txt`
+matches nothing. The file was ignored only incidentally, by `.gitignore` line 11 `*.key` — so the
+path happened to be safe for a reason I had not identified, and **any operator who renamed it
+(`server.pem`, `wg-server`, `privatekey`) would have had a committable WireGuard private key sitting
+in the checkout.** The gate would not have caught it either; `no-plaintext-secrets` does not
+recognise a bare WireGuard key.
+
+The step now states the real rule, marks the `.key` suffix as the thing doing the work, and gives
+the operator a verification they can run before trusting it (`git check-ignore -v` must print a
+rule, and `git status --short` must not list the file).
+
+Nothing outside `runbooks/vpn-remote-access.md` changed; no manifest, no behaviour.
+
+## What I verified
+
+- **`site.sh` does not exist**: `find . -name 'site.sh' -not -path './.git/*'` → no output.
+- **The real surface**: `infra/ansible/playbooks/` contains `node-baseline.yml` and `site.yml` (and
+  11 others); `node-baseline.yml`'s own header documents both
+  `-e k3s_hosts=k3s_prod` and the tag-scoping hazard. The `-e <playbook>_hosts=` convention is
+  confirmed by `infra/ansible/README.md` §"Targeting prod" and by the existing
+  `runbooks/node-memory-swap.md`, which invokes `ansible-playbook playbooks/node-health.yml -e
+  node_health_hosts=...`.
+- **`k3s_prod` is a real inventory group**: `[k3s_prod:children]` in `inventory/hosts.ini`, with
+  `[k3s_prod_primary]`/`[k3s_prod_secondary]`.
+- **`--tags system` covers the kernel-module tasks**: both "Persist kernel modules…" and "Load
+  kernel modules now" carry `tags: [system]` in `roles/node_baseline/tasks/main.yml`.
+- **The gitignore claim, measured both directions**:
+  `git check-ignore -v .cache/server.key` → `.gitignore:11:*.key`;
+  `git check-ignore -v .cache/serverkey.txt` → no match. So the suffix is the only thing ignoring it.
+- **Every path the component's docs reference resolves**: all six relative markdown links in
+  `platform/vpn-ingress/README.md` (checked from that file's own directory), and both Ansible
+  playbook paths.
+- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
+  `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; the rc 3 is the two
+  environmental skips (`gitea-settings` — no local token; conformance L2/L3 — no `CLUSTER_AVAILABLE`).
+  **Not green. Reported as rc 3, not as pass.**
+- **This branch touches NO sensitive path** (`runbooks/**` and `decisions/**` are not in
+  `infra/gitea/approval-policy.json`), so unlike #55 it needs no `/approve-override`.
+
+## What I did NOT verify
+
+- **The corrected command has not been RUN.** Ansible is an operator action, and this is the
+  decisive gap: I have verified the playbook, the group, the tag and the flag all exist and are
+  the documented convention, but not that the run succeeds end to end or that
+  `lsmod | grep '^wireguard'` then reports the module. It is a better-evidenced command, not a
+  proven one.
+- **Whether `--tags system` is entirely side-effect-free on a live prod node.** Each task it runs is
+  idempotent by construction, but I did not dry-run it (`--check`) to confirm it reports zero
+  changes beyond the module load.
+- **The rest of the runbook's commands past step 2** are unchanged and still unexercised — in
+  particular `kubectl exec ds/wireguard -- wg show`, which cannot be reached while the pods sit in
+  `Init` waiting on the server key.
+- **Whether any OTHER document I wrote in #55 contains an invented command.** I re-checked every
+  file path and markdown link in the component (all resolve) and every script the docs tell an
+  operator to run, but "the referenced file exists" is weaker than "the command is correct" — which
+  is exactly the gap that produced `site.sh`.
+
+## Post-merge state of #55, recorded here since its own entry predates the merge
+
+Measured on live prod after #55 merged as `ad3f56e`:
+
+- Flux applied it by the real path: Kustomization `vpn-ingress` `lastAppliedRevision:
+  main@sha1:ad3f56e`, equal to the GitRepository's artifact revision, `Ready=True`.
+- All **18** Flux Kustomizations `Ready=True` — the new tier disturbed nothing, and `wait: false`
+  behaved as intended (`vpn-ingress` is Ready while its own pods are not, so nothing downstream
+  stalled on it).
+- **`hostNetwork` is confirmed in effect, not merely configured**: the three pods report pod IPs
+  `192.168.2.10/.11/.12` — the node addresses, one per node.
+- **It fails closed exactly as designed**, with a legible cause: all three pods `Init:0/1`,
+  `MountVolume.SetUp failed for volume "wg-server-key" : secret "wg-server-key" not found`, and no
+  Secret in the namespace. The #55 entry's "the DaemonSet has never been reconciled" is superseded
+  by this paragraph; its remaining unverified claims (the tunnel carrying a packet, the
+  iptables/nftables backend question) still stand.
@@ -26,7 +26,8 @@ The fix is the obvious one for concurrent appenders: **stop sharing a file**. Ea
 handoff is its own file. Two agents writing handoffs touch disjoint paths, so there is
 nothing for git to merge and nothing to conflict. This scales to any number of parallel
 branches with zero coordination — which is the point, since the agents are deliberately
-isolated from each other in separate worktrees and cannot coordinate.
+isolated from each other in separate worktrees (required by `CLAUDE.md` §0) and cannot
+coordinate.

 There is deliberately **no index file** and no counter. An index would be a shared
 mutable file — exactly the thing that just failed — and a sequence number would have
@@ -74,12 +74,23 @@ node_baseline_backports_kernel_packages:
 # kernel, and how do I undo it?" from the machine itself.
 node_baseline_backports_kernel_stamp: /etc/node-baseline-backports-kernel

+# Modules written to /etc/modules-load.d/k3s-cilium-longhorn.conf (persistent) AND modprobe'd now.
 node_baseline_kernel_modules:
  - overlay
  - br_netfilter
  - iscsi_tcp
  - dm_crypt
  - nfs
+  # Remote-access VPN (platform/vpn-ingress). The WireGuard DaemonSet creates its wg0 link in the
+  # HOST netns with CAP_NET_ADMIN only -- it is deliberately NOT granted CAP_SYS_MODULE, so it
+  # cannot autoload this itself: `ip link add wg0 type wireguard` fails outright when the module is
+  # absent, and the pod CrashLoops. Loading it here is therefore a PREREQUISITE of that component,
+  # not a nicety. In-tree since kernel 5.6, so Debian 13's 7.1.8 ships it and there is nothing to
+  # install. Naming it here also means the kernel does the crypto: without the module a WireGuard
+  # container's only fallback is slow userspace wireguard-go, which we do not use (and which would
+  # need /dev/net/tun -- a device a hostPath mount cannot actually grant, as the /dev/dri attempt
+  # proved: visible but EPERM from the device cgroup).
+  - wireguard

 # --- DATA DISKS (tasks/disks.yml, tag `disks`) ----------------------------------------
 # The SHAPE of a node's storage is DECLARED per environment, never inferred from whether
@@ -0,0 +1,195 @@
+#!/usr/bin/env bash
+# =================================================================================================
+# new-peer.sh — enrol one device on the remote-access VPN (platform/vpn-ingress).
+#
+# Generates a WireGuard keypair ON THIS MACHINE, renders the device's client profile, optionally
+# prints a QR code for a phone, and prints the one `[Peer]` stanza to paste into
+# platform/vpn-ingress/peers.yaml. The private half NEVER leaves this machine: it is not sent to the
+# cluster, not committed, and not written to any log.
+#
+# WHY A SCRIPT AND NOT A RUNBOOK STEP. Three of these fields are easy to get wrong in ways that are
+# either silently insecure or silently broken, and all three are mechanical:
+#   * AllowedIPs on the SERVER side must be a /32. On the server, AllowedIPs is simultaneously the
+#     cryptokey routing table AND the inbound source filter, so a /24 here lets one device spoof
+#     every other device's tunnel address.
+#   * PersistentKeepalive must be set on the CLIENT. Without it the NAT mapping at the client's end
+#     expires after ~30s of idle and nothing can reach the device until it speaks first — which
+#     presents as "the VPN works for a minute and then half-works".
+#   * The endpoint must be the DDNS name, never the current WAN IP (see --endpoint below).
+#
+# USAGE
+#   ./infra/vpn-ingress/new-peer.sh <device-name> <last-octet> [--split] [--out DIR]
+#
+#   <device-name>   short, lowercase, e.g. pixel-8, thinkpad, work-laptop. Used for the filename
+#                   and the comment that ties the key to a human-recognisable device.
+#   <last-octet>    2-254, the device's address inside WG_CLIENT_CIDR. Must be unused.
+#   --split         split tunnel: route ONLY the LAN and the tunnel subnet. Default is a FULL
+#                   tunnel (0.0.0.0/0) so the device also egresses the house WAN.
+#   --out DIR       where to write the profile (default: a 0700 dir under $TMPDIR).
+#
+# ENVIRONMENT
+#   WG_SERVER_KEY_FILE  server PRIVATE key (default ~/.config/wireguard-home/server.key); its public
+#                       half is derived from it. Alternatively set WG_SERVER_PUBKEY directly.
+#   WG_ENDPOINT_HOST    default wan.home.keithjackson.dev
+#   WG_CLIENT_CIDR      default 10.90.0.0/24   (must match clusters/base/cluster-vars.yaml)
+#   WG_LAN_CIDR         default 192.168.2.0/24 (prod LAN_CIDR)
+#   WG_DNS              default 192.168.2.23   (prod PIHOLE_DNS_VIP)
+#
+# REQUIRES: wireguard-tools (`wg`). `qrencode` is optional and only used for the QR code.
+# =================================================================================================
+set -euo pipefail
+
+die() { printf 'new-peer.sh: %s\n' "$*" >&2; exit 1; }
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.yaml"
+
+# --- defaults, each mirroring a value that is DECLARED elsewhere in this repo -------------------
+# These are duplicated here rather than parsed out of the YAML on purpose: this script is meant to
+# run on an operator's laptop with no checkout-shaped assumptions beyond its own location, and a
+# silent mis-parse would be worse than a visible default. If you change the knob, change it here.
+: "${WG_CLIENT_CIDR:=10.90.0.0/24}"                 # clusters/base/cluster-vars.yaml
+: "${WG_LAN_CIDR:=192.168.2.0/24}"                  # clusters/prod LAN_CIDR
+: "${WG_DNS:=192.168.2.23}"                         # clusters/prod PIHOLE_DNS_VIP
+: "${WG_SERVER_KEY_FILE:=${HOME}/.config/wireguard-home/server.key}"
+# wan.<base-domain> is an UNPROXIED A record that platform/externaldns/ddns.yaml keeps pointed at
+# the current WAN IP (checked every 5 min). Use it, never a literal address: this house has a
+# DYNAMIC WAN IP, and a profile pinned to today's address stops working without warning and with no
+# symptom on the server side. It is a PUBLIC Cloudflare record, so it also resolves from a cellular
+# connection — unlike any `*.home.keithjackson.dev` name PiHole answers, which a remote client
+# cannot resolve until it is already connected.
+: "${WG_ENDPOINT_HOST:=wan.home.keithjackson.dev}"
+: "${WG_LISTEN_PORT:=51820}"
+: "${WG_MTU:=1420}"
+
+SPLIT=0
+OUT_DIR=""
+NAME=""
+OCTET=""
+while [ $# -gt 0 ]; do
+  case "$1" in
+    --split) SPLIT=1; shift ;;
+    --out)   OUT_DIR="${2:-}"; [ -n "$OUT_DIR" ] || die "--out needs a directory"; shift 2 ;;
+    -h|--help) sed -n '2,40p' "${BASH_SOURCE[0]}"; exit 0 ;;
+    -*) die "unknown option: $1" ;;
+    *) if [ -z "$NAME" ]; then NAME="$1"; elif [ -z "$OCTET" ]; then OCTET="$1"; else die "unexpected argument: $1"; fi; shift ;;
+  esac
+done
+
+[ -n "$NAME" ]  || die "need a device name. See --help"
+[ -n "$OCTET" ] || die "need a last octet (2-254). See --help"
+printf '%s' "$NAME" | grep -Eq '^[a-z0-9][a-z0-9-]{0,30}$' \
+  || die "device name must be lowercase letters/digits/hyphens: got '$NAME'"
+printf '%s' "$OCTET" | grep -Eq '^[0-9]+$' || die "octet must be a number: got '$OCTET'"
+[ "$OCTET" -ge 2 ] && [ "$OCTET" -le 254 ] || die "octet must be 2-254 (.1 is the server): got '$OCTET'"
+
+command -v wg >/dev/null || die "wireguard-tools not installed ('wg' not on PATH). Debian/Ubuntu: apt install wireguard-tools"
+
+# --- the server's public key --------------------------------------------------------------------
+if [ -z "${WG_SERVER_PUBKEY:-}" ]; then
+  [ -r "$WG_SERVER_KEY_FILE" ] || die "cannot read server key '$WG_SERVER_KEY_FILE'.
+  Either point WG_SERVER_KEY_FILE at it, or set WG_SERVER_PUBKEY to the public half.
+  If the server key does not exist yet, see runbooks/vpn-remote-access.md step 1."
+  WG_SERVER_PUBKEY="$(tr -d ' \r\n' < "$WG_SERVER_KEY_FILE" | wg pubkey)"
+fi
+
+CLIENT_BASE="${WG_CLIENT_CIDR%/*}"; CLIENT_BASE="${CLIENT_BASE%.*}"
+CLIENT_ADDR="${CLIENT_BASE}.${OCTET}"
+
+# --- refuse a collision, because the failure is otherwise invisible -----------------------------
+# Two devices sharing an address does not error anywhere: the server routes that address to
+# whichever peer handshook most recently, so BOTH devices appear to work intermittently.
+if [ -r "$PEERS_FILE" ] && grep -q "AllowedIPs *= *${CLIENT_ADDR}/32" "$PEERS_FILE"; then
+  die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet."
+fi
+if [ -r "$PEERS_FILE" ] && grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
+  printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.yaml.\n' "$NAME" >&2
+fi
+
+# --- output location: 0700, and never inside the repo -------------------------------------------
+if [ -z "$OUT_DIR" ]; then
+  OUT_DIR="$(mktemp -d "${TMPDIR:-/tmp}/wg-peer-XXXXXX")"
+else
+  mkdir -p "$OUT_DIR"
+fi
+chmod 700 "$OUT_DIR"
+case "$(cd "$OUT_DIR" && pwd)" in
+  "$REPO_ROOT"|"$REPO_ROOT"/*) die "refusing to write a private key inside the git checkout ($OUT_DIR)" ;;
+esac
+PROFILE="${OUT_DIR}/${NAME}.conf"
+
+umask 077
+PRIVKEY="$(wg genkey)"
+PUBKEY="$(printf '%s' "$PRIVKEY" | wg pubkey)"
+
+if [ "$SPLIT" -eq 1 ]; then
+  ALLOWED="${WG_LAN_CIDR}, ${WG_CLIENT_CIDR}"
+  TUNNEL_KIND="split tunnel (LAN only; the device keeps using its own internet)"
+else
+  ALLOWED="0.0.0.0/0"
+  TUNNEL_KIND="FULL tunnel (all IPv4 traffic egresses the house WAN)"
+fi
+
+cat > "$PROFILE" <<EOF
+# ${NAME} — remote-access VPN profile for home.keithjackson.dev
+# Generated $(date -u +%Y-%m-%dT%H:%M:%SZ) by infra/vpn-ingress/new-peer.sh
+# ${TUNNEL_KIND}
+#
+# THIS FILE CONTAINS A PRIVATE KEY. Move it to the device, then delete it from here.
+# It is the ONLY copy: the server never sees it and it cannot be recovered or re-derived.
+[Interface]
+PrivateKey = ${PRIVKEY}
+Address = ${CLIENT_ADDR}/32
+DNS = ${WG_DNS}
+MTU = ${WG_MTU}
+
+[Peer]
+PublicKey = ${WG_SERVER_PUBKEY}
+Endpoint = ${WG_ENDPOINT_HOST}:${WG_LISTEN_PORT}
+AllowedIPs = ${ALLOWED}
+# Hold the NAT mapping open from the client side. Required for the server to be able to reach the
+# device at all once it goes idle; harmless otherwise (one small packet every 25s).
+PersistentKeepalive = 25
+EOF
+chmod 600 "$PROFILE"
+
+cat <<EOF
+
+  Device      : ${NAME}
+  Address     : ${CLIENT_ADDR}/32
+  Routing     : ${TUNNEL_KIND}
+  DNS         : ${WG_DNS}  (PiHole — gives the device LAN answers for *.home.keithjackson.dev)
+  Profile     : ${PROFILE}   (0600; contains the private key)
+
+  ------------------------------------------------------------------------------------------
+  STEP 1 — add this to platform/vpn-ingress/peers.yaml, under data.peers.conf, then PR it:
+
+    # ${NAME} — <owner>, enrolled $(date -u +%Y-%m-%d)
+    [Peer]
+    PublicKey = ${PUBKEY}
+    AllowedIPs = ${CLIENT_ADDR}/32
+
+  Keep the comment line: \`wg show\` can only print the key, so it is the only record of which
+  stanza belongs to which device when one has to be revoked.
+
+  STEP 2 — get the profile onto the device:
+    Android / iOS  : scan the QR code below in the WireGuard app ("+" -> Scan from QR code)
+    Windows / macOS: WireGuard app -> "Import tunnel(s) from file" -> ${NAME}.conf
+    Linux          : install as /etc/wireguard/${NAME}.conf, then
+                     sudo systemctl enable --now wg-quick@${NAME}
+
+  STEP 3 — delete the profile from this machine once the device has it:
+    shred -u ${PROFILE} 2>/dev/null || rm -f ${PROFILE}
+
+  The tunnel will not work until the peers.yaml PR is MERGED and Flux has rolled the DaemonSet
+  (~1 min). Until then the device handshakes and gets silence, exactly like an unknown key.
+  ------------------------------------------------------------------------------------------
+EOF
+
+if command -v qrencode >/dev/null; then
+  printf '  QR code (this IS the private key — your terminal scrollback now holds a credential):\n\n'
+  qrencode -t ansiutf8 < "$PROFILE"
+else
+  printf '  qrencode not installed, so no QR code was printed (Debian/Ubuntu: apt install qrencode).\n'
+  printf '  Transfer %s to the device by another route instead.\n' "$PROFILE"
+fi
@@ -0,0 +1,199 @@
+# platform/vpn-ingress — WireGuard remote access (north-south INGRESS)
+
+Lets the operator's own devices — Android, Linux, Windows, macOS, iOS — dial in from anywhere on
+the internet and behave as though plugged into the home LAN: full reach to `192.168.2.0/24` and
+egress out the house WAN.
+
+> **This is not `platform/vpn-egress`.** They are opposite directions and share nothing but the
+> word "VPN". `vpn-egress` pushes *pods out* to the internet through a PIA tunnel (ADR-0015,
+> R-NET-05/06/07). This component lets *remote clients in*. Changing one never affects the other.
+
+| | |
+|---|---|
+| Transport | WireGuard, **udp/51820**, kernel implementation |
+| Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node |
+| Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices |
+| Authorisation | a device's **public key** is in [`peers.yaml`](./peers.yaml), or it cannot connect |
+| The one secret | the server's private key, Secret `wg-server-key`, **created out of band** |
+| Exposure | one hand-made WAN port-forward on the SOHO router, off-repo |
+| Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) |
+| Procedure / bring-up | [`runbooks/vpn-remote-access.md`](../../runbooks/vpn-remote-access.md) |
+
+## Why WireGuard
+
+The requirement names four client platforms. WireGuard is the only option that has a *first-party*
+client on all of them (official apps for Android, iOS, Windows and macOS; in-kernel on Linux since
+5.6, so a Linux client needs no userspace daemon at all). It is also the smallest thing to expose:
+one UDP port, no cipher negotiation, no PKI, no user database, no TLS stack.
+
+OpenVPN would have needed a CA, per-device certificates and a revocation list — three pieces of
+long-lived state this repo would then have to protect and rotate. IKEv2/IPsec has native clients but
+is painful to run in a container and brittle behind double NAT. Neither buys anything here.
+
+### Why this is safe to expose to the whole internet
+
+The honest answer is not "because it is encrypted" — it is the handshake's failure mode:
+
+**An unauthenticated packet gets no reply at all.** WireGuard's first handshake message carries a
+MAC keyed with the *responder's own public key*. If the sender does not already know it, the packet
+is dropped silently — no error, no reset, no banner, nothing a scanner can fingerprint. From the
+outside, udp/51820 is indistinguishable from a closed port. There is no login prompt to brute-force,
+no version string to match against a CVE, and no amplification to abuse.
+
+That is the whole security argument, and it is why the exposed port carries no IP allowlist (which
+would also have made roaming clients unusable — a phone's address changes with every cell tower).
+Compare `apps/wave-0/pihole-dns.yaml`, which *does* pin `loadBalancerSourceRanges` to the LAN: a
+resolver answers anybody, so it must be scoped by address. WireGuard answers nobody.
+
+What it does **not** protect against is a stolen device — anyone holding a device's private key
+*is* that device. That is what revocation is for: delete the stanza, merge, done (~1 min).
+
+## Architecture, and the one decision that drives everything
+
+```
+  phone on 5G                    SOHO router                 k3s-prod-N (host netns)
+ ┌──────────────┐  udp/51820   ┌───────────────┐  forward   ┌────────────────────────────┐
+ │ WireGuard app│─────────────▶│ WAN 51820 ──▶ │───────────▶│ wg0 10.90.0.1/24           │
+ │ 10.90.0.2/32 │   over the   │ node IP:51820 │            │   decrypt, then FORWARD    │
+ └──────────────┘   internet   └───────────────┘            │   SNAT to the node's addr  │
+                                                            └─────────────┬──────────────┘
+                                                                          │ as `host`/`remote-node`
+                                        ┌─────────────────────────────────┼──────────────────────┐
+                                        ▼                                 ▼                      ▼
+                               LAN 192.168.2.0/24            Gateway VIP .20 (16 web UIs)   WAN / internet
+                               (NAS, router, printers)        PiHole DNS VIP .23
+```
+
+### `hostNetwork: true` is required, not convenient
+
+This is the only genuinely interesting decision in the component, and it was forced by a *measured*
+property of this cluster rather than chosen for convenience.
+
+In Cilium, what a packet may reach is decided by the **security identity of its source**. A
+WireGuard server running as an ordinary pod must SNAT client traffic to its own pod IP (Cilium drops
+a packet whose source is not the endpoint's own address), so every remote client would have arrived
+everywhere wearing *that pod's* identity. On 2026-10-04 this cluster measured what that costs
+(`decisions/2026-10-04-plex-advertise-plexdirect.md`): a request to the shared Gateway VIP **from
+inside a pod returns `403 Access denied` from Envoy for all 16 Gateway hostnames**, because Cilium's
+Gateway preserves the original source identity and enforces the *destination's* ingress policy —
+and every app admits `reserved:ingress` plus named peers, never an arbitrary pod. LAN browsers work
+only because they arrive as `reserved:world` / `reserved:host`.
+
+So the pod-networked version of this component would have produced a tunnel that came up healthy,
+reported connected peers, routed packets — and 403'd **every single cluster web UI**: plex, gitea,
+keycloak, nextcloud, grafana, headlamp, the apex. `*.home.keithjackson.dev` would not even have
+resolved, because `apps/wave-0/pihole-dns.yaml`'s ingress policy deliberately omits the `cluster`
+entity. A VPN that looks up and is useless is the worst available outcome, and nothing in the
+manifests would have hinted at it.
+
+In the **host** netns there is no pod identity to carry. Forwarded client traffic is SNATed to the
+node's own address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` —
+byte-for-byte what a LAN client looks like. Three further problems vanish with it:
+
+* `net.ipv4.ip_forward` is **already 1** on these nodes (`node_baseline` writes
+  `/etc/sysctl.d/99-k3s-cilium.conf`). It is *not* in the kubelet's safe-sysctl allowlist, so a
+  pod-netns build would have needed `--allowed-unsafe-sysctls` on all three kubelets — a
+  node-restart change to the k3s service, to run a VPN.
+* **No LoadBalancer VIP**, so no `CiliumL2AnnouncementPolicy` lease lottery (the backend-blind
+  announcer that cost this LAN an 8-day Minecraft outage — see
+  `conformance/tests/l0-static/l2-announced-vips-serve-from-any-node`) and no
+  `externalTrafficPolicy` trade-off to make at all.
+* **No double encapsulation.** Client UDP lands on the node's own NIC, so the underlay is a flat
+  1500-byte path, not Cilium's 1450-byte VXLAN. That is what makes `MTU = 1420` simply correct.
+
+`hostNetwork` is *permitted* here, and not by oversight:
+`platform/kyverno/policies/restrict-host-namespaces.yaml` matches `namespaceSelector:
+app-namespace=true` only, and names this exact carve-out in its own comment — "platform components
+that do … e.g. Cilium, node-exporter … live in platform namespaces that do NOT carry the
+app-namespace label". [`namespace.yaml`](./namespace.yaml) is deliberately unlabelled.
+
+**The cost, stated rather than buried:** a `hostNetwork` pod has no Cilium endpoint, so **no
+NetworkPolicy constrains this workload** — not [`networkpolicy.yaml`](./networkpolicy.yaml), not
+anything. That file says so at length instead of implying a protection it cannot provide. The
+boundary around this port is WireGuard's peer authentication, full stop.
+
+### Why a DaemonSet
+
+The WAN forward names **one node address**, by hand, off-repo (same shape as the plex tcp/32400
+forward). A single-replica Deployment would float on every reschedule and silently invalidate it.
+A DaemonSet instead puts an identical listener on all three nodes — every node answers the same
+keys — so **losing the forwarded node is a one-line router change**, with no cluster work and no
+re-issued client profiles. The two idle listeners cost ~15 MiB each.
+
+A pinned `LoadBalancer` VIP was the considered alternative (it would make the forward immune to node
+replacement). Rejected: it reintroduces the L2-announcement lease, SNAT, and an extra node-to-node
+hop into the one path whose failure mode is *"I am away from home and cannot get in"*. Fewer
+translations win here.
+
+### The split that keeps authorisation reviewable
+
+| half | where | why |
+|---|---|---|
+| peer **public** keys + addresses | [`peers.yaml`](./peers.yaml), **plaintext in git** | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
+| the server's **private** key | Secret `wg-server-key`, **out of band** | the only real secret; an agent must not mint a long-lived private key (CLAUDE.md §6) |
+
+The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** —
+so the assembled config holding the private key never touches any node's disk and dies with the pod.
+
+Encrypting the peer register would have hidden exactly the change a reviewer most needs to see: a
+new line granting a new device access to the whole LAN.
+
+## What a client can reach — and how to narrow it
+
+By design, **everything a LAN machine can reach**: all of `192.168.2.0/24`, the shared Gateway's 16
+hostnames, PiHole, the NAS, the router's admin UI, and the internet via the house WAN.
+
+**That includes the three k3s nodes themselves** — `192.168.2.10/.11/.12`, and therefore the
+kube-apiserver on 6443, kubelet on 10250 and SSH on 22. An enrolled device is as privileged as a
+machine sitting in the house, which is what "reach the entire local network" asks for; it is
+recorded here because it is the largest consequence of saying yes to that, and because nobody should
+discover it later. **It is also why a lost device must be revoked promptly** (delete its stanza,
+merge; ~1 min to take effect).
+
+To grant less, narrow it at the **client** profile first — `new-peer.sh --split` emits a
+split-tunnel profile routing only the LAN and the tunnel subnet, so the device keeps its own
+internet path. To enforce less *server-side*, add filtering to the `PostUp` chain in
+[`daemonset.yaml`](./daemonset.yaml) (e.g. `iptables -A FORWARD -i wg0 -d 192.168.2.10 -j DROP`
+before the accept rules) — the FORWARD chain there is the only enforcement point, since no
+NetworkPolicy applies.
+
+## Operating it
+
+* **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
+  stanza into `peers.yaml` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
+* **Revoke a device:** delete its stanza → PR → merge. No other step; the key stops working.
+* **Is it up?** `kubectl --context k3s-prod -n vpn-ingress get ds wireguard` — three Ready.
+* **Who is connected?** `kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show`
+  (`latest handshake` under ~3 min means live). Never dump `wg showconf` — it prints the private key.
+* **Full bring-up and verification:** `runbooks/vpn-remote-access.md`.
+
+## Prerequisites outside this directory
+
+1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
+   (`infra/ansible/roles/node_baseline/defaults/main.yml`); the operator runs the role. The pod has
+   `NET_ADMIN` but deliberately **not** `SYS_MODULE`, so it cannot load the module itself — it
+   CrashLoops with a clear `wg-quick` error instead of quietly degrading to userspace WireGuard.
+2. **The `wg-server-key` Secret.** Runbook step 1. Until it exists the DaemonSet cannot start —
+   the correct fail-closed state for an unenrolled VPN.
+3. **The WAN port-forward**, udp/51820 → one node address. Hand-made on the SOHO router, off-repo,
+   and the half CI can never check.
+
+## Not built (deliberate, with reasons)
+
+* **No metrics / no peer-level alerting.** A WireGuard exporter would expose
+  `wireguard_latest_handshake_seconds` per peer — genuinely useful for spotting unexpected
+  activity — but it needs a second `NET_ADMIN` container in the pod that terminates the
+  internet-facing tunnel, and that trade was not worth making unverified. Pod-down is already
+  covered generically by `KubeDeploymentReplicasMismatch`/`KubePodNotReady` from
+  kube-prometheus-stack. **This is the top follow-up.**
+* **No preshared keys.** They would add a symmetric post-quantum hedge on top of the Noise
+  handshake — and would be a real secret, forcing `peers.yaml` into SOPS and costing the
+  reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
+  `wg-server-key` Secret.
+* **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6
+  still reaches IPv6 sites *outside* the tunnel. Functionally harmless, a privacy leak if the point
+  was to hide the client's network. Fix is an `fd..` ULA on `wg0` plus `::/0` in the client's
+  `AllowedIPs`; not done because nothing here needs it.
+* **No in-cluster-service reach beyond what the LAN has.** Clients reach cluster apps the way a LAN
+  browser does — through the Gateway VIP. Pod IPs and `10.43.0.0/16` service IPs are not routed
+  into the tunnel.
@@ -0,0 +1,295 @@
+---
+# =================================================================================================
+# WireGuard remote-access listener — udp/51820 in the HOST network namespace, on every node.
+# =================================================================================================
+# WHY WireGuard AND NOT OpenVPN/IPsec. The requirement is "Android, Linux, Windows and other
+# computing clients", from anywhere on the internet, securely. WireGuard is the only option here
+# with a first-party client on all four platforms (official apps for Android/iOS/Windows/macOS, and
+# in-kernel on Linux since 5.6 — so Linux clients need no userspace daemon at all). It is also the
+# smallest attack surface of the three: one UDP port, ~4k lines of kernel code, no cipher
+# negotiation, no PKI, no user database, and — the property that actually matters for a port
+# exposed to the whole internet — it NEVER REPLIES to an unauthenticated packet. See README.
+#
+# -------------------------------------------------------------------------------------------------
+# hostNetwork: true — REQUIRED, not convenient. This is THE decision in this file.
+# -------------------------------------------------------------------------------------------------
+# A VPN client must behave like a machine plugged into the LAN. In Cilium, what a packet is ALLOWED
+# to reach is decided by the SECURITY IDENTITY of its source, and after any in-pod SNAT that
+# identity is the VPN pod's — not `world`. That distinction is not academic; it was MEASURED on
+# this cluster on 2026-10-04 (decisions/2026-10-04-plex-advertise-plexdirect.md): a curl of the
+# shared Gateway VIP from inside a pod returns `403 Access denied` from Envoy for **all 16**
+# Gateway hostnames, because Cilium's Gateway preserves the original source identity and enforces
+# the DESTINATION's ingress CiliumNetworkPolicy, and every app admits `reserved:ingress` plus named
+# peers — never an arbitrary pod. LAN browsers are allowed only because they arrive as
+# `reserved:world` / `reserved:host`.
+#
+# So a pod-networked WireGuard server would have handed remote clients a tunnel on which every
+# single cluster web UI returned 403 — plex, gitea, keycloak, nextcloud, grafana, headlamp, the
+# apex, all of them — while `wg show` reported a perfectly healthy tunnel. PiHole would have failed
+# the same way and worse: apps/wave-0/pihole-dns.yaml's ingress policy deliberately OMITS the
+# `cluster` entity, so VPN clients would not have been able to resolve `*.${BASE_DOMAIN}` either.
+# The VPN would have looked up and been useless, which is the worst failure shape available.
+#
+# In the host netns there is no pod identity to carry: forwarded client traffic is SNATed to the
+# NODE's address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` —
+# exactly what a LAN client looks like. Three further problems disappear with it:
+#   * net.ipv4.ip_forward is ALREADY 1 on these nodes (node_baseline writes it to
+#     /etc/sysctl.d/99-k3s-cilium.conf), so no pod sysctl is needed — and `net.ipv4.ip_forward`
+#     is NOT in the kubelet's safe-sysctl allowlist, so a pod-netns version of this would have
+#     required `--allowed-unsafe-sysctls` on all three kubelets (a node-restart change).
+#   * no LoadBalancer VIP, so no CiliumL2AnnouncementPolicy lease lottery (the defect that cost
+#     this LAN an 8-day Minecraft outage) and no `externalTrafficPolicy` trade-off at all.
+#   * no double encapsulation. WireGuard's UDP lands on the node's own NIC, so the underlay is a
+#     flat 1500-byte LAN path, not Cilium's 1450-byte VXLAN.
+#
+# hostNetwork is PERMITTED here, and not by accident: platform/kyverno/policies/
+# restrict-host-namespaces.yaml matches `namespaceSelector: app-namespace=true` only, and names
+# this exact carve-out in its own comment ("platform components that do -- e.g. Cilium,
+# node-exporter -- live in platform namespaces that do NOT carry the app-namespace label").
+# namespace.yaml is deliberately unlabelled. THE COST IS REAL AND IS STATED: a hostNetwork pod
+# bypasses the Cilium datapath, so NO NetworkPolicy constrains this workload (see
+# networkpolicy.yaml, which says so rather than pretending otherwise). The boundary protecting
+# this port is WireGuard's own cryptographic peer authentication, not the CNI.
+#
+# -------------------------------------------------------------------------------------------------
+# A DaemonSet, not a single-replica Deployment.
+# -------------------------------------------------------------------------------------------------
+# The WAN port-forward on the SOHO router points at ONE node address, by hand, off-repo (same shape
+# as the plex tcp/32400 forward — runbooks/plex-remote-access.md). A Deployment would float to
+# whichever node the scheduler picked and silently invalidate that forward on every reschedule. A
+# DaemonSet instead puts an identical warm listener on ALL THREE nodes: the forward names one of
+# them, and if that node is lost, FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work and no
+# new client profiles — every node answers the same keys. Each node binds udp/51820 in its own
+# netns, so there is no conflict, and the two idle listeners cost ~15 MiB each.
+#
+# All three nodes carry no taints (measured 2026-10-05), so no tolerations are needed for the
+# DaemonSet to cover the cluster; add them if a node is ever tainted.
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: wireguard
+  namespace: vpn-ingress
+  labels:
+    app.kubernetes.io/name: vpn-ingress
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: vpn-ingress
+  # One node at a time, so a rollout never takes every listener down at once. The node named by the
+  # router's forward does drop for a few seconds during its own replacement; clients re-handshake
+  # automatically (WireGuard retries every 5s, and rekeys every 2 minutes regardless).
+  updateStrategy:
+    type: RollingUpdate
+    rollingUpdate:
+      maxUnavailable: 1
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: vpn-ingress
+        # Declared for documentation and for symmetry with the app tier. It MUST be `internet`, and
+        # the value is load-bearing rather than cosmetic: platform/vpn-egress/egress-steering.yaml's
+        # CiliumEgressGatewayPolicy selects `egress-class: vpn` pods CLUSTER-WIDE, with no namespace
+        # scope. Setting `vpn` here would steer every remote client's internet traffic out the PIA
+        # tunnel instead of the house WAN — the opposite of the requirement, and it would look like
+        # a working VPN with a mysterious foreign exit IP. (The selector matches on the pod label,
+        # so hostNetwork does not exempt this pod from being selected.)
+        homelab.keithjackson.dev/egress-class: internet
+        # WireGuard authenticates its own peers by public key; there is no UI and no IdP hop.
+        homelab.keithjackson.dev/auth-mode: native
+    spec:
+      # See the long rationale at the head of this file. ClusterFirstWithHostNet keeps in-cluster
+      # DNS working for the pod itself despite hostNetwork (without it the pod inherits the NODE's
+      # resolv.conf, which points at public resolvers).
+      hostNetwork: true
+      dnsPolicy: ClusterFirstWithHostNet
+      automountServiceAccountToken: false
+      # ── Assemble the running config: PUBLIC peer register (git) + PRIVATE server key (Secret) ──
+      # The two halves are deliberately stored apart (peers.yaml explains why) and are only ever
+      # joined here, at pod start, into a tmpfs that is destroyed with the pod. The server's private
+      # key is therefore never written to any node's disk, never committed, and never logged.
+      initContainers:
+        - name: assemble-config
+          image: lscr.io/linuxserver/wireguard:1.0.20260223-r0-ls124
+          imagePullPolicy: IfNotPresent
+          # POSIX sh. Note on $ handling: this script is rendered through Flux's envsubst (the
+          # `vpn-ingress` Kustomization declares postBuild.substituteFrom), so `${WG_CLIENT_CIDR}`
+          # below IS substituted on purpose — it is a declared cluster-vars knob. Every SHELL
+          # variable is written bare (`$net`, `$(cmd)`), which envsubst leaves untouched. There is
+          # deliberately NO braced shell-variable form anywhere in here: braces are what envsubst
+          # consumes, so a shell variable written that way silently renders to the empty string.
+          # (ci: flux-postbuild-substitution enforces this, and caught this very comment when it
+          # first spelled the hazard out literally — the check reads comments too, correctly.)
+          command:
+            - /bin/sh
+            - -c
+            - |
+              set -eu
+              umask 077
+              mkdir -p /config/wg_confs
+
+              # Derive the server's own tunnel address from the single CIDR knob, so there is no
+              # second knob to keep in agreement: 10.90.0.0/24 -> 10.90.0.1/24.
+              net=${WG_CLIENT_CIDR}
+              mask=$(echo "$net" | cut -d/ -f2)
+              base=$(echo "$net" | cut -d/ -f1 | cut -d. -f1-3)
+
+              # The key arrives as a Secret file; strip any trailing newline the operator's shell
+              # added, or wg-quick rejects the config with "Key is not the correct length or format".
+              key=$(tr -d ' \r\n' < /run/wg-server-key/privatekey)
+              if [ -z "$key" ]; then
+                echo "FATAL: /run/wg-server-key/privatekey is empty." >&2
+                echo "       See runbooks/vpn-remote-access.md step 1 (create the server key)." >&2
+                exit 1
+              fi
+
+              {
+                echo '[Interface]'
+                echo "Address = $base.1/$mask"
+                echo 'ListenPort = 51820'
+                # 1420 = 1500 (flat LAN/WAN path to the node's own NIC) - 60 bytes of WireGuard
+                # encapsulation (20 IPv4 + 8 UDP + 32 WireGuard). This is also what wg-quick would
+                # auto-derive, but it is pinned because the symptom of getting it wrong is the
+                # nastiest kind: handshakes and SSH work while large packets (HTTPS, file copies)
+                # hang. Inner packets stay under Cilium's 1450-byte VXLAN MTU too, so reaching a
+                # cluster pod does not fragment either.
+                echo 'MTU = 1420'
+                printf 'PrivateKey = %s\n' "$key"
+                # ── Forwarding + SNAT, applied in the HOST netns when wg0 comes up ──
+                # ip_forward is already 1 from node_baseline; re-asserted so this component does not
+                # silently depend on another role's sysctl file surviving.
+                echo 'PostUp = sysctl -q -w net.ipv4.ip_forward=1'
+                # SNAT client traffic to the node's address. REQUIRED: without it a LAN host would
+                # reply to 10.90.0.x, find no route, hand the packet to the router, and the router
+                # would drop it — so every connection would hang rather than fail. Matching on
+                # `! -o wg0` rather than a named `-o eth0` keeps this independent of the node's NIC
+                # name. Each rule is add-if-absent so a pod killed without PostDown does not
+                # accumulate duplicates in the host's tables across restarts.
+                echo "PostUp = iptables -t nat -C POSTROUTING -s $net ! -o wg0 -j MASQUERADE 2>/dev/null || iptables -t nat -A POSTROUTING -s $net ! -o wg0 -j MASQUERADE"
+                echo 'PostUp = iptables -C FORWARD -i wg0 -j ACCEPT 2>/dev/null || iptables -A FORWARD -i wg0 -j ACCEPT'
+                echo 'PostUp = iptables -C FORWARD -o wg0 -j ACCEPT 2>/dev/null || iptables -A FORWARD -o wg0 -j ACCEPT'
+                echo "PostDown = iptables -t nat -D POSTROUTING -s $net ! -o wg0 -j MASQUERADE 2>/dev/null || true"
+                echo 'PostDown = iptables -D FORWARD -i wg0 -j ACCEPT 2>/dev/null || true'
+                echo 'PostDown = iptables -D FORWARD -o wg0 -j ACCEPT 2>/dev/null || true'
+                echo ''
+              } > /config/wg_confs/wg0.conf
+
+              cat /run/wg-peers/peers.conf >> /config/wg_confs/wg0.conf
+              chmod 600 /config/wg_confs/wg0.conf
+
+              # Report the SHAPE of the result, never its content — the file now holds the server's
+              # private key, and a log line is a transcript (CLAUDE.md §6).
+              echo "wg0.conf assembled: $(grep -c '^\[Peer\]' /config/wg_confs/wg0.conf || true) peer(s), server $base.1/$mask, udp/51820."
+          securityContext:
+            runAsUser: 0
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop:
+                - ALL
+          resources:
+            requests:
+              cpu: 10m
+              memory: 16Mi
+            limits:
+              memory: 64Mi
+          volumeMounts:
+            - { name: wg-config, mountPath: /config }
+            - { name: wg-server-key, mountPath: /run/wg-server-key, readOnly: true }
+            - { name: wg-peers, mountPath: /run/wg-peers, readOnly: true }
+      containers:
+        - name: wireguard
+          # linuxserver/wireguard — provides wg, wg-quick and iptables on a maintained, multi-arch
+          # base. Pinned to the full upstream+build tag (1.0.20260223, published 2026-10-01) rather
+          # than `latest`, per the crafty/oauth2-proxy convention; Renovate maintains it.
+          #
+          # NO `PEERS` ENV IS SET, DELIBERATELY. With PEERS the image runs in "server mode" and
+          # GENERATES keypairs at runtime — which would put the authoritative peer list on a
+          # container filesystem instead of in git, and (worse) print client private keys and QR
+          # codes into the POD LOG, where they would be readable by anyone with log access and
+          # shipped to Loki. Supplying our own /config/wg_confs/wg0.conf selects the image's
+          # bring-up-what-you-are-given path instead.
+          image: lscr.io/linuxserver/wireguard:1.0.20260223-r0-ls124
+          imagePullPolicy: IfNotPresent
+          env:
+            - name: TZ
+              value: "America/Chicago"
+          ports:
+            # hostNetwork, so containerPort IS the node port. Declared for documentation and so
+            # `kubectl get pod -o wide` shows what this listener owns.
+            - { name: wireguard, containerPort: 51820, hostPort: 51820, protocol: UDP }
+          securityContext:
+            # NET_ADMIN is the whole privilege budget: create the wg0 link, set the host sysctl, and
+            # write the two netfilter rules above. Everything else is dropped, and the container is
+            # NOT privileged — which also keeps it clear of restrict-host-namespaces' second rule.
+            # SYS_MODULE is deliberately NOT granted: the `wireguard` module is loaded by the node
+            # (infra/ansible/roles/node_baseline, node_baseline_kernel_modules), so this container
+            # never needs to load a kernel module. If the module is missing the pod CrashLoops with
+            # a clear wg-quick error rather than quietly falling back to slow userspace WireGuard.
+            allowPrivilegeEscalation: false
+            capabilities:
+              drop:
+                - ALL
+              add:
+                - NET_ADMIN
+            # readOnlyRootFilesystem intentionally UNSET: the image's s6 init writes under /run and
+            # /var during startup. Same deferral as crafty/headlamp.
+          resources:
+            requests:
+              cpu: 25m
+              memory: 32Mi
+            limits:
+              # Kernel WireGuard does its crypto in the KERNEL, not in this container, so throughput
+              # does not consume the container's budget. The limit covers s6 plus the tmpfs holding
+              # wg0.conf (a memory-medium emptyDir counts against this).
+              memory: 128Mi
+          # `wg show wg0` exits non-zero when the interface is absent, which is the one failure that
+          # matters and the one that a port probe cannot see: a crashed or never-created wg0 leaves
+          # udp/51820 silent, and WireGuard is silent to unauthenticated traffic ANYWAY, so "nothing
+          # answers" is indistinguishable from "working" from the outside. This probe is the only
+          # thing that tells the difference.
+          startupProbe:
+            exec:
+              command: ["sh", "-c", "wg show wg0 >/dev/null"]
+            periodSeconds: 5
+            failureThreshold: 24
+          livenessProbe:
+            exec:
+              command: ["sh", "-c", "wg show wg0 >/dev/null"]
+            periodSeconds: 30
+            failureThreshold: 3
+          volumeMounts:
+            - { name: wg-config, mountPath: /config }
+            # Read-only so the image can SEE whether the kernel module is present and fail with a
+            # real message. It cannot load one (no SYS_MODULE) — that is the node's job.
+            - { name: lib-modules, mountPath: /lib/modules, readOnly: true }
+      volumes:
+        # tmpfs. The assembled wg0.conf contains the server's private key, so it must never land on
+        # a node's disk — `medium: Memory` guarantees that and guarantees it is gone when the pod is.
+        - name: wg-config
+          emptyDir:
+            medium: Memory
+            # 8Mi, not 1Mi: /config is this image's whole config ROOT, not just our one file — its
+            # s6 init also writes templates and state there, and a tmpfs that small would ENOSPC
+            # during startup rather than at a point anyone would connect to the volume size. Still
+            # counted against the container's 128Mi limit, and still far more than wg0.conf needs.
+            sizeLimit: 8Mi
+        # The ONLY secret in this component. Created OUT OF BAND by the operator and referenced by
+        # name (R-IDP-04) — runbooks/vpn-remote-access.md step 1. It is NOT a *.sops.yaml in this
+        # tree on purpose: an agent must not mint a long-lived private key (CLAUDE.md §6), and a key
+        # that passed through a session transcript would have to be treated as compromised.
+        # DR: Velero's `daily` schedule excludes only `monitoring` and `unifi`, so this namespace
+        # and this Secret ARE captured off-cluster (platform/velero/helmrelease.yaml). Keep a copy
+        # in the operator's password manager as well — restoring it is what makes every already
+        # enrolled client work again without re-enrolling.
+        - name: wg-server-key
+          secret:
+            secretName: wg-server-key
+            defaultMode: 0400
+            items:
+              - { key: privatekey, path: privatekey }
+        - name: wg-peers
+          configMap:
+            name: wg-peers
+        - name: lib-modules
+          hostPath:
+            path: /lib/modules
+            type: Directory
@@ -0,0 +1,30 @@
+---
+# WireGuard remote-access VPN (north-south INGRESS). Reconciled by its OWN non-gating post-platform
+# Flux Kustomization (clusters/base/vpn-ingress.yaml, dependsOn platform-config, wait:false) — the
+# same placement as crafty/headlamp/homepage/runbooks, for the same reason: a leaf whose pod can be
+# legitimately unhealthy must never sit in the gating `platform` tier, where one component's
+# unhealth stalls platform-config and every apps wave behind it (DECISIONS-LOG: "Headlamp gating
+# stall"). That rationale is unusually concrete here: until the operator creates the `wg-server-key`
+# Secret (runbooks/vpn-remote-access.md step 1) this DaemonSet CANNOT start, so putting it in the
+# gating tier would have taken the whole platform down on merge.
+#
+# NOT in platform/kustomization.yaml — see clusters/base/vpn-ingress.yaml.
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - namespace.yaml
+  # The peer register: which devices may dial in. PLAINTEXT and reviewable on purpose (see the file).
+  - peers.yaml
+  # The listener itself: hostNetwork DaemonSet, udp/51820 on every node.
+  - daemonset.yaml
+  # Namespace default-deny baseline. NOTE: does NOT constrain the hostNetwork DaemonSet — the file
+  # says so at length rather than implying a protection it cannot provide.
+  - networkpolicy.yaml
+#
+# DELIBERATELY ABSENT: a Secret. The server's private key (`wg-server-key`) is created out of band
+# by the operator and referenced BY NAME from daemonset.yaml. There is no `*.sops.yaml` here, so
+# this component needs NO SOPS decryption stanza in clusters/dev or clusters/prod — unlike
+# crafty/headlamp/velero. If that key is ever migrated into SOPS, BOTH halves must land together:
+# the file here, and a `decryption:` patch naming the `vpn-ingress` Kustomization in each overlay.
+# Without the patch, kustomize hands Flux the raw ciphertext and the pod mounts a key full of
+# ENC[...] — which applies cleanly and fails only at handshake time.
@@ -0,0 +1,26 @@
+---
+# WireGuard REMOTE-ACCESS VPN (north-south INGRESS) — the counterpart to platform/vpn-egress.
+#
+# Do not confuse the two. They are opposite directions and share nothing but the word "VPN":
+#   * platform/vpn-egress  — pods OUT to the internet through a PIA tunnel (ADR-0015, R-NET-05/06/07).
+#   * platform/vpn-ingress — remote CLIENTS IN from the internet to this LAN. THIS directory.
+#
+# NOTE: intentionally NOT homelab.keithjackson.dev/app-namespace=true. This is PLATFORM network
+# infrastructure (like cilium / vpn-egress), not an application, so Kyverno does NOT generate the app
+# default-deny baseline here and the app-scoped auth-mode / egress-class / host-namespace policies do
+# not apply — every Kyverno policy in platform/kyverno/policies matches namespaceSelector
+# app-namespace=true. THAT EXEMPTION IS LOAD-BEARING HERE, not incidental: this workload runs with
+# `hostNetwork: true`, which `restrict-host-namespaces` forbids in an app namespace. It is permitted
+# here by that policy's OWN stated carve-out — "platform components that do [need host namespaces] --
+# e.g. Cilium, node-exporter -- live in platform namespaces that do NOT carry the app-namespace
+# label, so they are out of scope". daemonset.yaml documents why hostNetwork is REQUIRED rather than
+# convenient; it is the single most consequential decision in this component.
+#
+# NO `gateway-access` label. This component serves no HTTP and attaches no HTTPRoute to the shared
+# Gateway — it is a raw UDP listener. There is deliberately nothing here for the L7 Gateway to route.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: vpn-ingress
+  labels:
+    app.kubernetes.io/name: vpn-ingress
@@ -0,0 +1,56 @@
+---
+# =================================================================================================
+# READ THIS BEFORE TRUSTING THIS FILE: it does NOT constrain the WireGuard listener.
+# =================================================================================================
+# Every other platform namespace in this repo (crafty, headlamp, homepage, runbooks) ships an
+# explicit default-deny baseline as defense in depth, because Kyverno's generator only covers
+# `app-namespace=true` namespaces. This file keeps that convention — but it would be dishonest to
+# leave it at that, because the convention does not actually bite here:
+#
+#   THE `wireguard` DaemonSet RUNS WITH hostNetwork: true, SO IT HAS NO CILIUM ENDPOINT, AND NO
+#   NetworkPolicy OR CiliumNetworkPolicy APPLIES TO IT — NEITHER THIS ONE NOR ANY OTHER.
+#
+# That is the documented reason restrict-host-namespaces forbids hostNetwork in app namespaces at
+# all ("a pod on hostNetwork bypasses the CNI (Cilium) datapath and therefore ESCAPES the
+# default-deny NetworkPolicy fabric -- it is the most direct way to defeat R-NET-01"). Here that
+# escape is the deliberate, required design (daemonset.yaml explains why: without it, remote
+# clients carry a pod identity and are 403'd by every Gateway hostname). A CiliumNetworkPolicy
+# admitting `world` to udp/51820 would therefore be theatre: it would read as the control that
+# protects this port, and it would be enforcing nothing. So there isn't one.
+#
+# WHAT ACTUALLY PROTECTS THE EXPOSED PORT, then, since the CNI does not:
+#   1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.yaml gets
+#      NO REPLY — see README "Why this is safe to expose". This is the real control.
+#   2. The SOHO router's forward: only udp/51820 is forwarded inbound, to one node.
+#   3. The blast radius once authenticated is deliberately the whole LAN — that IS the requirement
+#      ("clients should reach the entire local network"). README "What a client can reach" states
+#      the consequence in full, including reach to the nodes' own control plane, and gives the
+#      narrowing recipe for an operator who wants less than that.
+#
+# WHAT THIS FILE IS STILL FOR: it is the namespace's standing posture for any FUTURE pod-networked
+# workload added here (a metrics exporter is the likely one — README "Not built"). Without it, such
+# a pod would be born unrestricted, because Kyverno will not generate a baseline in a namespace
+# that is not labelled `app-namespace=true`, and this one deliberately is not.
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: default-deny
+  namespace: vpn-ingress
+  labels:
+    homelab.keithjackson.dev/baseline: default-deny
+spec:
+  podSelector: {}
+  policyTypes:
+    - Ingress
+    - Egress
+  ingress: []
+  egress:
+    - to:
+        - namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: kube-system
+      ports:
+        - protocol: UDP
+          port: 53
+        - protocol: TCP
+          port: 53
@@ -0,0 +1,73 @@
+---
+# ============================================================================================
+# THE PEER REGISTER — every device allowed to dial in, in PLAINTEXT, in git, on purpose.
+# ============================================================================================
+# WireGuard has no usernames, no passwords and no PKI. Authorisation is exactly one thing: a
+# device's PUBLIC key appears in this file, or it cannot complete a handshake. An unlisted key
+# gets no reply at all — not a rejection, silence (see README "Why this is safe to expose").
+# Revoking a device is therefore deleting its stanza, and nothing else.
+#
+# WHY PLAINTEXT, AND WHY THIS IS NOT A SECRET LEAK. A WireGuard peer's PUBLIC key is public by
+# construction — it is the half you are supposed to hand to the server, and holding it grants
+# nothing. The only secret in the whole scheme is the SERVER's private key, which is NOT here:
+# it lives in the `wg-server-key` Secret and is assembled into the running config at pod start
+# (daemonset.yaml). Keeping the register in cleartext buys the property that matters most for an
+# access-control list: **"who can get into my house" is reviewable in a pull-request diff**, and
+# its history is `git log` on this file. A SOPS-encrypted register would hide exactly the change
+# a reviewer most needs to see — a new line granting a new device entry to the whole LAN.
+#
+# This is the same division of labour the repo already uses for the opposite case: a Secret's KEY
+# NAMES are cleartext while values are encrypted (.sops.yaml `encrypted_regex`), so diffs stay
+# readable. Here the whole file is the key-name half.
+#
+# ADDING A DEVICE — never hand-roll this, and never generate a key in a chat transcript.
+#   ./infra/vpn-ingress/new-peer.sh <device-name> <last-octet>
+# It generates the keypair ON THE OPERATOR'S MACHINE, writes/renders the client profile (and a
+# QR code for phones), prints ONLY the public half for pasting below, and never transmits or
+# logs the private half. Then: add the stanza, PR, merge; Flux rolls the DaemonSet.
+#
+# ADDRESS ALLOCATION. ${WG_CLIENT_CIDR} is the tunnel subnet; .1 is the server (every node).
+# Give each device ONE address as a /32 in AllowedIPs. A /32 is not a formality — on the SERVER
+# side AllowedIPs is the cryptokey ROUTING table AND the inbound source filter: a peer may only
+# send from, and receive traffic for, the addresses listed in its own stanza. A sloppy /24 here
+# would let any one device impersonate every other device's tunnel address.
+#
+#   .1          the server (this DaemonSet, on every node)
+#   .2 - .254   devices, one each, allocated below. Keep this list dense and in order so the
+#               next free octet is obvious; never re-use an octet after revoking a device
+#               (an old client profile would silently start working again as someone else).
+#
+# PRESHARED KEYS are deliberately NOT used. They add a symmetric post-quantum hedge on top of
+# the Noise handshake and would be a genuine secret — which would force this register into SOPS
+# and cost the reviewability above. Revisit only if the threat model changes; if it does, the PSK
+# belongs in the `wg-server-key` Secret, not here. See README "Hardening not taken".
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: wg-peers
+  namespace: vpn-ingress
+  labels:
+    app.kubernetes.io/name: vpn-ingress
+data:
+  # Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
+  #
+  # *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
+  # An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
+  # authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
+  # DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
+  # The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
+  # (`wg show` reports an interface with no peers).
+  #
+  # COPY THIS TEMPLATE, uncomment, one block per device:
+  #
+  #   # pixel-8 — Keith's phone, enrolled 2026-10-05
+  #   [Peer]
+  #   PublicKey = <the public key new-peer.sh printed>
+  #   AllowedIPs = 10.90.0.2/32
+  #
+  # Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
+  # tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
+  # it an incident responder cannot tell which stanza to delete.
+  peers.conf: |
+    # No peers enrolled yet. See the template in this file's comments and
+    # ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
@@ -120,6 +120,9 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
 - **`vpn-egress-stale`** — the tunnel is UP but `egress-class=vpn` pods have no egress: Cilium's cached
  ifindex outlived a reconnect. Also covers the watchdog for that rule going inert.
 - **`denied-flows`** — policy-denied flow spike. The Cilium hairpin/identity regression class.
+- **`vpn-remote-access`** — the INBOUND remote-access VPN (WireGuard, udp/51820): standing it up,
+  enrolling and revoking devices, and the off-LAN checks that are the only honest proof it works.
+  The opposite direction to the two pages above — do not confuse them.

 ### Backups and DR
 - **`backup-failure`** — a Velero backup failed. The R2 object-tagging incompatibility is documented here.
@@ -113,3 +113,8 @@ configMapGenerator:
      - update-rollback=update-rollback.md
      - vpn-down=vpn-down.md
      - vpn-egress-stale=vpn-egress-stale.md
+      # Operator procedure, not an alert page. Served because the component it completes is INERT
+      # until three off-repo manual steps are done (kernel module, server key, WAN port-forward) —
+      # and because the one check that actually proves it cannot be run from inside the house.
+      # Peer-level alerting does not exist yet (it needs an exporter); this page is the substitute.
+      - vpn-remote-access=vpn-remote-access.md
@@ -0,0 +1,226 @@
+# vpn-remote-access — standing up the WireGuard remote-access VPN, and proving it
+
+**Status: the in-cluster half is shipped; the VPN is INERT until three manual steps are done.**
+All three are off-repo by necessity — a kernel module, a private key, and a router port-forward.
+**Alert:** pod-down is covered generically (`KubePodNotReady` / `KubeDeploymentReplicasMismatch`).
+There is **no peer-level alerting** — see "Why there is no connection alert". This is an **operator
+procedure**, not an incident page.
+**Files:** `platform/vpn-ingress/` (namespace, peer register, DaemonSet, baseline policy),
+`clusters/base/vpn-ingress.yaml` (its Flux tier), `clusters/base/cluster-vars.yaml`
+(`WG_CLIENT_CIDR`), `infra/ansible/roles/node_baseline/defaults/main.yml` (the `wireguard` module),
+`infra/vpn-ingress/new-peer.sh` (device enrolment).
+**Design rationale:** `platform/vpn-ingress/README.md` — read "Why `hostNetwork` is required"
+before changing anything about how this is wired.
+**Not to be confused with** `vpn-down` / `vpn-egress-stale`, which are about `platform/vpn-egress`
+(pods *out* through PIA). This page is remote clients *in*.
+
+## What this gives you
+
+A device — Android, Linux, Windows, macOS, iOS — dials in over **udp/51820** from anywhere and
+behaves like a machine on the LAN: full reach to `192.168.2.0/24`, the Gateway's web UIs, PiHole
+DNS, and internet egress out the house WAN.
+
+Authorisation is one thing only: the device's **public key is in
+`platform/vpn-ingress/peers.yaml`**. An unlisted key gets no reply at all.
+
+> **An enrolled device is as privileged as a machine sitting in the house** — including the three
+> k3s nodes (`192.168.2.10/.11/.12`: apiserver 6443, kubelet 10250, SSH 22). That is what "reach
+> the entire local network" means. Revoke a lost device promptly: delete its stanza, PR, merge.
+
+## Step 1 — load the `wireguard` module on every node (once)
+
+The DaemonSet creates its `wg0` link in the host netns with `CAP_NET_ADMIN` only; it is
+deliberately **not** granted `CAP_SYS_MODULE`, so it cannot autoload the module. Without it
+`wg-quick` fails outright and the pod CrashLoops.
+
+```bash
+# operator action (Ansible is not an agent action). Ansible lives ONLY in the pinned toolbox,
+# and ansible.cfg is only picked up from infra/ansible, so cd there inside the container.
+./infra/devcontainer/dev.sh run bash -c \
+  'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system'
+```
+
+Three details in that command are load-bearing, not style:
+
+- **`-e k3s_hosts=k3s_prod`.** `node-baseline.yml` targets `k3s_servers` by default, which is the
+  **dev** group — and dev was decommissioned, so an unqualified run does nothing useful. This is
+  the targeting pattern `infra/ansible/README.md` "Targeting prod" documents for every playbook.
+- **`--tags system`, and never an untagged run.** The two kernel-module tasks carry the `system`
+  tag. An untagged run would also execute the `disks` tasks, which assert on block-device variables
+  prod has deliberately left as placeholders (`infra/PROD-PROVISIONING.md` §2) — the playbook's own
+  header warns about this. `--tags system` is broader than the modules alone (it also re-asserts
+  sysctls, iscsid, qemu-guest-agent and the swap masking) but every one of those is idempotent and
+  already in its desired state on a live node.
+- **It is a playbook, not a wrapper script.** There is no `site.sh` in this repo; `playbooks/site.yml`
+  is the full bring-up chain and is the wrong tool here.
+
+Verify on each node — the module is in-tree since kernel 5.6, so nothing is installed, only loaded:
+
+```bash
+# expected: a `wireguard` line, and the modules-load.d file naming it for the next boot
+lsmod | grep '^wireguard'
+grep -w wireguard /etc/modules-load.d/k3s-cilium-longhorn.conf
+```
+
+## Step 2 — create the server key (once)
+
+This is the **only** secret in the component, and the one thing that must never be generated in an
+agent session or pasted into a transcript. Generate it on your own machine:
+
+```bash
+mkdir -p ~/.config/wireguard-home && chmod 700 ~/.config/wireguard-home
+umask 077
+wg genkey > ~/.config/wireguard-home/server.key          # keep this; see "Back it up" below
+```
+
+Load it into the cluster. **Do not echo it, do not pass it on an interactive command line that
+lands in shell history, and never print the resulting Secret's `.data`** (CLAUDE.md §6):
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n vpn-ingress create secret generic wg-server-key \
+     --from-file=privatekey=/work/.cache/server.key'
+```
+
+…where `/work/.cache/server.key` is a path you copied the key to (`.cache/` is not tracked; create
+it if absent). Delete that copy as soon as the Secret exists: `shred -u .cache/server.key`.
+
+> **KEEP THE `.key` SUFFIX.** That filename is what keeps the key out of git: `.gitignore` line 11
+> is `*.key`, and it is the **only** rule covering this path — `.cache/` itself is NOT ignored
+> (`git check-ignore -v .cache/serverkey.txt` matches nothing). So a copy named `server.pem`,
+> `wg-server` or `privatekey` sitting in that directory is **committable**, and the gate's
+> `no-plaintext-secrets` stage does not know a WireGuard private key when it sees one. Verify before
+> you trust it: `git check-ignore -v .cache/<your-filename>` must print a rule, and
+> `git status --short` must not list the file.
+
+Confirm it exists **by key name only**:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n vpn-ingress get secret wg-server-key -o jsonpath="{.data}" | tr "," "\n" | cut -d: -f1'
+# expected: {"privatekey   -- a NAME. If you see base64 payload, you printed the key. Rotate it.
+```
+
+**Why out of band and not SOPS.** An agent must not mint a long-lived private key, and a key that
+passed through a session transcript would have to be treated as compromised. The trade-off is that
+this Secret is **not reconstructible from git** — so:
+
+**Back it up.** Keep `server.key` in your password manager. Velero's `daily` schedule excludes only
+`monitoring` and `unifi` (`platform/velero/helmrelease.yaml`), so this namespace *is* captured
+off-cluster to R2 as well. Restoring this one key is what makes every already-enrolled device work
+again after a cluster rebuild — **losing it means re-enrolling every device.**
+
+## Step 3 — forward udp/51820 on the router (once)
+
+On the SOHO router at `192.168.2.1`, forward **WAN udp/51820** to **one node address**, e.g.
+`192.168.2.10:51820`. This is hand-made and off-repo; it is the half CI can never check.
+
+Pick any node — all three run an identical listener. Note which one you chose here:
+
+```
+# WAN udp/51820 -> 192.168.2.__  (k3s-prod-_)   set <date>
+```
+
+**Node loss = failover is this one line.** Re-point the forward at another node address; every
+device keeps working with the same profile, because all three nodes answer the same keys.
+
+Do **not** also expose this on tcp, and do not change the port hoping for obscurity — WireGuard is
+already silent to unauthenticated probes (README, "Why this is safe to expose").
+
+## Step 4 — confirm the listener is healthy
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n vpn-ingress get ds wireguard -o wide'
+# expected: DESIRED 3  CURRENT 3  READY 3
+
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show'
+# expected: `interface: wg0`, `listening port: 51820`, and (before step 5) NO peers.
+```
+
+A zero-peer interface that is Ready is the **correct** state before any device is enrolled: it
+listens, and refuses every handshake because no key is trusted.
+
+The readiness/liveness probe is `wg show wg0`, and that is load-bearing rather than cosmetic: a
+port probe cannot distinguish "wg0 never came up" from "working", because WireGuard is silent to
+unauthenticated traffic either way. If the pod is Ready, the interface exists.
+
+## Step 5 — enrol a device
+
+```bash
+./infra/vpn-ingress/new-peer.sh pixel-8 2              # full tunnel  (LAN + WAN egress)
+./infra/vpn-ingress/new-peer.sh thinkpad 3 --split     # split tunnel (LAN only)
+```
+
+It generates the keypair **on your machine**, writes a `0600` client profile outside the checkout,
+prints a QR code if `qrencode` is installed, and prints the `[Peer]` stanza to add. Then:
+
+1. Paste the stanza into `platform/vpn-ingress/peers.yaml` under `data.peers.conf`, keeping the
+   `# <name> — <owner>, enrolled <date>` comment. `wg show` can only ever print the key, so that
+   comment is the only record of which stanza belongs to which device.
+2. PR → merge. Flux rolls the DaemonSet (~1 min).
+3. Get the profile onto the device (QR scan for phones; "Import tunnel from file" on
+   Windows/macOS; `/etc/wireguard/<name>.conf` + `wg-quick@<name>` on Linux).
+4. **Delete the profile from your machine**: `shred -u <path>`. It is the only copy of that
+   device's private key — the server never sees it and it cannot be re-derived.
+
+Until the PR is merged the device handshakes and gets silence, exactly like an unknown key.
+
+## Step 6 — verify it end to end (from OFF the LAN)
+
+**This is the only honest test, and it cannot be run from inside the house.** Put the phone on
+cellular, with Wi-Fi off, and check all four — each proves a different hop:
+
+| # | Check | Proves |
+|---|---|---|
+| 1 | tunnel shows a recent handshake in the WireGuard app | the WAN forward + the server key + the peer stanza |
+| 2 | `ping 192.168.2.1` (the router) | decrypt → FORWARD → LAN, and the SNAT return path |
+| 3 | open `https://plex.home.keithjackson.dev` | PiHole DNS over the tunnel **and** the Gateway accepting a `host`-identity source — the specific thing a pod-networked VPN would have 403'd |
+| 4 | visit an IP-echo site; it should show `136.51.115.252` | full-tunnel WAN egress out the house, not the carrier |
+
+Then confirm server-side:
+
+```bash
+./infra/devcontainer/dev.sh run bash -c \
+  'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show'
+# a `latest handshake` under ~3 min, and non-zero transfer, on the expected peer.
+```
+
+**If check 3 fails while 2 passes**, the tunnel is fine and the problem is DNS or the Gateway, not
+WireGuard. Check the client's `DNS = 192.168.2.23` and that PiHole is up.
+
+**If 1 and 2 pass but large transfers hang**, it is MTU. The profile ships `MTU = 1420` (1500 − 60
+bytes of WireGuard encapsulation); drop the client to `1280` and re-test.
+
+**If nothing handshakes at all**, you cannot tell a wrong forward from a wrong key from outside —
+by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes
+if the router is not delivering the packets, and a `handshake` with no completed session if it is.
+
+## Revoking a device
+
+Delete its `[Peer]` stanza from `peers.yaml`, PR, merge. Nothing else — the key stops working on the
+next reconcile. **Never re-use that address octet**: an old profile would silently start working
+again as a different device.
+
+## Why there is no connection alert
+
+A WireGuard exporter would publish `wireguard_latest_handshake_seconds` per peer and make
+"something connected at 03:00" alertable. It needs a second `NET_ADMIN` container inside the pod
+that terminates the internet-facing tunnel, and that was not worth shipping unverified. **It is the
+top follow-up for this component.** Until then:
+
+* pod-down is covered generically by kube-prometheus-stack's `KubePodNotReady` /
+  `KubeDeploymentReplicasMismatch`;
+* "who is connected" is an on-demand question — `wg show`, above;
+* **never** `wg showconf` — it prints the server's private key.
+
+## Known limitations
+
+* **No IPv6 in the tunnel.** A full-tunnel client with native IPv6 still reaches IPv6 sites outside
+  the tunnel. Functionally harmless; a privacy leak if hiding the client's network was the point.
+* **Client IPs are not preserved** on the LAN side: everything arrives SNATed as the node's
+  address. A static route for `10.90.0.0/24` → the forwarded node, added on the router, would
+  preserve them — at the cost of another hand-maintained off-repo route.
+* **Rollouts briefly drop the forwarded node.** `maxUnavailable: 1`, so one node at a time; clients
+  re-handshake automatically (retry every 5s, rekey every 2 min).
Author	SHA1	Message	Date
jakman85andClaude Opus 5	18f741d744	fix(runbooks): vpn-remote-access step 1 named a nonexistent script; step 2 cited the wrong gitignore rule gate / gate (pull_request) Successful in 1m47s Details gate / conformance (pull_request) Has been skipped Details Both reported by the operator on first use of the runbook merged in #55. Step 1 said `cd infra/ansible && ./site.sh --tags system --limit k3s_prod`. There is no site.sh anywhere in this repo. It was wrong four ways at once: this repo drives Ansible with `ansible-playbook <playbook>` and never a wrapper (playbooks/site.yml is the full bring-up chain, not this); the module work is in playbooks/node-baseline.yml; targeting here is `-e k3s_hosts=...`, not `--limit`; and Ansible exists only in the pinned toolbox, with ansible.cfg read only from infra/ansible. Replaced with the real command plus why `-e k3s_hosts=k3s_prod` matters (the default group is decommissioned dev) and why `--tags system` must never be dropped (an untagged run also executes the `disks` tasks, which assert on prod's deliberate placeholder devices). Step 2 told the operator to stage the server key at /work/.cache/server.key "inside the repo's gitignored cache". .cache/ is NOT gitignored -- the file was ignored only incidentally by `*.key` (.gitignore:11). So the path was safe for a reason I had not identified, and renaming it (server.pem, wg-server) would have left a committable WireGuard private key in the checkout, which no-plaintext-secrets does not recognise. The step now names the real rule, marks the suffix as the load-bearing part, and gives a check-ignore verification to run before trusting it. Docs only -- no manifest and no behaviour change. Touches no sensitive path. Also records the post-merge measurements for #55, whose own entry predates the merge: Flux applied `ad3f56e` by the real path, all 18 Kustomizations Ready, hostNetwork confirmed (pod IPs are the node IPs), and the designed fail-closed state (Init:0/1 on the missing wg-server-key Secret). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-05 23:08:19 -05:00
jakman85	3ee663dbcc	Merge pull request 'docs(agents): one workspace per agent run — the shared checkout is off-limits' (#56 ) from docs/agents-work-in-their-own-workspace into main gate / gate (push) Successful in 1m46s Details gate / conformance (push) Has been skipped Details Reviewed-on: #56 Reviewed-by: Keith Jackson <[email protected]>	2026-10-06 03:49:28 +00:00
jakman85	ad3f56eee8	Merge pull request 'feat(vpn-ingress): WireGuard remote access on udp/51820 — hostNetwork, because a pod identity would 403 every Gateway hostname' (#55 ) from feat/vpn-remote-access-wireguard into main gate / gate (push) Successful in 1m47s Details gate / conformance (push) Has been skipped Details Reviewed-on: #55 Reviewed-by: Keith Jackson <[email protected]>	2026-10-06 03:41:48 +00:00
jakman85andClaude Opus 5	2cb891c05d	docs(agents): one workspace per agent run — the shared checkout is off-limits gate / gate (pull_request) Successful in 1m48s Details gate / conformance (pull_request) Has been skipped Details Agents have been working in the operator's checkout and clobbering each other. Nothing in the tree told them not to: worktrees appeared in CLAUDE.md only as §3's gate workarounds, and decisions/README.md described worktree isolation as something agents already do rather than as a rule. New CLAUDE.md §0 states the rule, with the commands and an explicit do-not list for the shared checkout. Placed before §1 so every existing §-reference in the repo keeps its meaning. Measured in the shared checkout on 2026-10-05, each row a distinct failure mode: 107 `checkout: moving` reflog entries; sitting on feat/pm-agent-issue-to-pr, not main; two other runs' untracked trees left in it; local main two commits behind origin/main; 42 registered worktrees of which 31 are on already-merged branches; 9 gate scripts carrying a `.claude/` exclusion because an in-tree worktree made find-based discovery count the same manifest up to 18 times. Corrects a claim carried in earlier briefs: `git worktree remove` is NOT blocked by the agent sandbox. Provoked on a throwaway worktree — rc 0. §3 reframed as the normal path and grown from two breakages to three: spec-immutability's skip is avoidable, not merely tolerated, with the spec repo mounted read-only in the same ENGINE shim plus -e SPEC_REPO=/home-network-k3s. CLAUDE.md is a sensitive path, so this PR needs a non-author review or an administrator /approve-override posted as a PR review. Not posted by me: §1 forbids an agent clearing that on its own PR. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-05 22:31:40 -05:00
jakman85andClaude Opus 5	f92707f38c	feat(vpn-ingress): WireGuard remote access on udp/51820 (hostNetwork, declarative peers) gate / gate (pull_request) Successful in 1m47s Details gate / conformance (pull_request) Has been skipped Details Lets the operator's Android / Linux / Windows / macOS devices dial in from anywhere and behave like LAN machines: full reach to 192.168.2.0/24 plus internet egress out the house WAN. Opposite direction to platform/vpn-egress, which sends pods OUT through PIA; the two share nothing but the word "VPN". WireGuard because it is the only option with a first-party client on all four named platforms, and because its handshake MAC is keyed with the responder's own public key: an unauthenticated packet gets NO reply, so udp/51820 is indistinguishable from a closed port. That, not "it is encrypted", is what makes an unfiltered WAN port defensible, and why there is no source allowlist. hostNetwork: true is REQUIRED, not convenient. A pod-networked server must SNAT clients to its own pod IP, so every remote client would carry that pod's Cilium identity -- and a pod-sourced request to the shared Gateway VIP returns 403 from Envoy for ALL 16 hostnames (measured 2026-10-04), because Cilium's Gateway preserves the original source identity and enforces the destination's ingress policy. The obvious build would have shipped a healthy-looking tunnel on which every cluster web UI was 403 and .home.keithjackson.dev did not even resolve. In the host netns traffic is SNATed to the node and arrives as host/remote-node, exactly like a LAN client. It also sidesteps the unsafe ip_forward sysctl, the L2-announcement lease lottery, and double encapsulation (hence MTU 1420). Permitted by restrict-host-namespaces' own carve-out for platform namespaces. The cost is stated in networkpolicy.yaml rather than papered over: no NetworkPolicy constrains a hostNetwork pod. A DaemonSet so the hand-made WAN forward cannot be invalidated by a reschedule; all three nodes answer the same keys, so failover is a one-line router change. Peer PUBLIC keys are plaintext in peers.yaml, which makes "who can get into my house" a reviewable PR diff. The only secret -- the server's private key -- is an out-of-band Secret: an agent must not mint a long-lived private key, and the two halves are joined only at pod start into a tmpfs that dies with the pod. Ships INERT-BUT-HEALTHY with zero peers. THREE off-repo manual steps remain (kernel module, server key, WAN forward) -- runbooks/vpn-remote-access.md. Touches clusters/* , so ci/approval-policy.sh will block pending the operator's /approve-override. Deliberately not posted by me (CLAUDE.md §1). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-05 22:29:46 -05:00