Added my phone to peers

Reviewed-on: #64

.gitignore

@@ -37,6 +37,14 @@ coverage/
 infra/tofu/backend.hcl
 infra/tofu/.r2.env
 
+# Operator scratch, writable from inside the dev container (the repo is its only persistent
+# read-write mount). Holds things that must never be committed: a staged copy of the SOPS age or
+# WireGuard key, rendered VPN client profiles (infra/vpn-ingress/new-peer.sh writes here by
+# default). `infra/ansible/.gitignore` already does this for its own .cache/; this is the same
+# convention at the root. Do NOT rely on `*.key` alone to protect a staged key — that only covers
+# one filename shape, which is why this rule exists.
+.cache/
+
 # Claude Code local scratch: agent worktrees + session state. Nothing here is tracked;
 # ignoring it keeps `git add -A` from staging embedded worktree repos and keeps the lint
 # stage (which scans untracked files) from tripping on .claude/worktrees/.

clusters/base/vpn-ingress.yaml

@@ -8,8 +8,8 @@
 #     NOTHING downstream reconciles — the incident behind the headlamp/oauth2-proxy/homepage moves
 #     (DECISIONS-LOG: "Headlamp gating stall").
 #   * SPECIFIC, AND THE REASON THIS IS NOT A JUDGEMENT CALL: this DaemonSet cannot start until the
-#     operator creates the out-of-band `wg-server-key` Secret (runbooks/vpn-remote-access.md
-#     step 1). That is the correct fail-closed state — an unenrolled VPN must not come up — but in
+#     operator replaces the `REPLACE-ME` placeholder in platform/vpn-ingress/wg-server-key.sops.yaml
+#     with a real key (runbooks/vpn-remote-access.md step 2). That is the correct fail-closed state — an unenrolled VPN must not come up — but in
 #     the gating tier it would have been a self-inflicted platform-wide outage ON MERGE, with the
 #     cause (a missing manual step) invisible from any Flux condition.
 #
@@ -51,6 +51,9 @@ spec:
     substituteFrom:
       - kind: ConfigMap
         name: cluster-vars
-  # NO `decryption:` stanza is needed in the overlays: this path carries no *.sops.yaml. The server
-  # key is an out-of-band Secret (platform/vpn-ingress/kustomization.yaml says what to do if that
-  # ever changes).
+  # SOPS decryption is layered per-environment by the overlay (clusters/dev and clusters/prod each
+  # patch THIS Kustomization by name), identical to headlamp/crafty/velero — this path carries
+  # platform/vpn-ingress/wg-server-key.sops.yaml, the WireGuard server private key. The `sops-age`
+  # Secret is created out of band and referenced by NAME only. Without the overlay patch the Secret
+  # applies cleanly full of ENC[...] and only the tunnel is broken, so the patch is as load-bearing
+  # as the encrypted file itself.

clusters/dev/kustomization.yaml

@@ -122,6 +122,24 @@ patches:
           provider: sops
           secretRef:
             name: sops-age
+  # vpn-ingress (the WireGuard remote-access VPN, clusters/base/vpn-ingress.yaml) carries
+  # platform/vpn-ingress/wg-server-key.sops.yaml — the WireGuard SERVER private key, the one
+  # secret in that component. Without this stanza kustomize hands Flux the raw ciphertext and the
+  # Secret applies full of ENC[...]: the APPLY SUCCEEDS and every Flux condition stays green, and
+  # the only symptom is a VPN that no client can handshake with. The DaemonSet's init container
+  # validates the value with `wg pubkey` specifically so that state is a named CrashLoop instead.
+  # The `sops-age` Secret is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: vpn-ingress
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
   # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
   # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
   # account and the Anthropic subscription token from `claude setup-token`. Without this

clusters/prod/kustomization.yaml

@@ -113,6 +113,24 @@ patches:
           provider: sops
           secretRef:
             name: sops-age
+  # vpn-ingress (the WireGuard remote-access VPN, clusters/base/vpn-ingress.yaml) carries
+  # platform/vpn-ingress/wg-server-key.sops.yaml — the WireGuard SERVER private key, the one
+  # secret in that component. Without this stanza kustomize hands Flux the raw ciphertext and the
+  # Secret applies full of ENC[...]: the APPLY SUCCEEDS and every Flux condition stays green, and
+  # the only symptom is a VPN that no client can handshake with. The DaemonSet's init container
+  # validates the value with `wg pubkey` specifically so that state is a named CrashLoop instead.
+  # The `sops-age` Secret is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: vpn-ingress
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
   # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
   # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
   # account and the Anthropic subscription token from `claude setup-token`. Without this

decisions/2026-10-05-pm-agent-registry-pull-prereq.md

@@ -0,0 +1,127 @@
+---
+date: 2026-10-05
+branch: docs/pm-agent-registry-pull-prereq
+title: pm-agent reconciled exactly as designed — and is the first workload ever to pull from the in-cluster registry, which does not work yet
+---
+
+## What changed
+
+Documentation only. No manifest, no policy, no code.
+
+PR #57 merged and `pm-agent` reconciled. I measured what landed, and the measurement turned
+up **two prerequisites for the next operator step that nothing in this repository
+declares** — and that the handoff entry for #57 could not have known, because it listed the
+image pull as unverified and that is exactly where it bit.
+
+So this entry records the post-merge measurement, and
+`platform/pm-agent/README.md`, `docs/plans/pm-agent-issue-to-pr.md` and
+`runbooks/pm-agent.md` now name both prerequisites where they say "build, push, pin the
+digest". Before this, that step read as one action; it is three, and the two new ones are
+not an agent's to perform.
+
+**The finding is not specific to pm-agent.** `conformance-runner` has the identical
+registry path (`gitea.home.keithjackson.dev/home-network/...`), the identical "operator
+pushes, a digest goes in a PR" model, and has never been pulled either — so the same two
+blockers sit in front of the L2/L3 conformance wiring. Measured: **no pod anywhere in the
+cluster references that registry** except the pm-agent dispatcher, which is therefore the
+first workload ever to try it.
+
+## What I verified
+
+Read-only, through the pinned toolbox, against the live prod cluster.
+
+**The merge reconciled, completely and exactly as designed.**
+
+- `GitRepository/flux-system` artifact revision: `main@sha1:95597bd6883637…` — the merge
+  commit. `platform-config`, `pm-agent` and `apps-wave-0` all report that same
+  `lastAppliedRevision` with `Ready=True`. Compared field to field, per CLAUDE.md §2b,
+  rather than assumed from the merge.
+- **The security boundary is live.** `ClusterPolicy/pm-agent-job-shape`:
+  `validationFailureAction=Enforce`, `failurePolicy=Fail`, `Ready=True`, and all six rules
+  present (`worker-must-be-the-committed-shape`,
+  `worker-must-be-one-container-with-no-args`,
+  `worker-volumes-must-be-workspace-or-tmp-emptydirs`,
+  `worker-may-not-declare-init-or-ephemeral-containers`,
+  `worker-env-names-must-be-the-committed-set`, `worker-may-not-use-envfrom`). This is the
+  check that mattered most: a ClusterPolicy added to the wrong kustomization list ships
+  green and never reaches the cluster, which is what happened to the conformance boundary
+  on 2026-09-29.
+- Namespace contents: both ServiceAccounts, `pm-agent-config` (17 keys), the
+  `pm-agent-credentials` Secret, the `default-deny` NetworkPolicy, and both
+  CiliumNetworkPolicies with `VALID=True`.
+- **SOPS decryption works through the new overlay patch.** Checked by KEY NAMES and VALUE
+  LENGTHS only, never values (CLAUDE.md §6): keys `gitea-token`, `claude-code-oauth-token`,
+  lengths 76 and 52 bytes — which are the base64 lengths of the committed placeholder
+  strings, so the Secret holds the decrypted placeholders and not ciphertext. A
+  ciphertext apply succeeds silently and would have left `ENC[AES256_GCM,…]` values an
+  order of magnitude longer.
+- **`pm-agent-worker` really is bound to nothing.** The namespace has exactly one
+  RoleBinding (`pm-agent` → Role `pm-agent`, subject `ServiceAccount/pm-agent`), and no
+  ClusterRoleBinding anywhere names either pm-agent ServiceAccount.
+- `gitea-allow-pm-agent-ingress` is applied in the `gitea` namespace, matching
+  `io.kubernetes.pod.namespace: pm-agent`.
+- The dispatcher is `ImagePullBackOff` on the all-zero placeholder digest — the designed,
+  visible failure.
+
+**The two prerequisites, measured.**
+
+1. **The nodes cannot verify the registry's TLS certificate.** containerd's error on the
+   pull is not about the digest:
+
+       failed to resolve reference "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:000…":
+       tls: failed to verify certificate: x509: certificate is not valid for any names,
+       but wanted to match gitea.home.keithjackson.dev
+
+   "not valid for **any** names" means the certificate presented to the node carries no
+   matching SAN at all. From the operator's workstation the same name is healthy: it
+   resolves to `192.168.2.20` and serves a Let's Encrypt certificate
+   `CN=*.home.keithjackson.dev` with SANs `*.home.keithjackson.dev, home.keithjackson.dev`
+   — and `Certificate gateway/wildcard-home` is `Ready=True` for exactly those names. So
+   the certificate that *should* be served is fine, and what the NODE reaches is not what
+   the workstation reaches. **Correcting the digest will not clear this.**
+2. **Anonymous pulls from this registry are refused.** Not inferred from a bare 401 — that
+   would have been wrong, because this Gitea answers 401 for *everything* anonymously:
+   `/v2/home-network/agent-runner/manifests/latest`, `/v2/home-network/conformance-runner/…`,
+   a randomly-named nonexistent repository, and `nosuchowner/nosuchimage` all return 401,
+   and `/api/v1/packages/Home-Network` returns `{"message":"token is required"}`. The
+   registry *does* issue a token to an anonymous client, but its `access` claim is `null` —
+   no pull scope. So the conclusion that holds is the simple one: **an unauthenticated
+   client gets nothing from this registry**, so a pull with no credential cannot succeed.
+   And `grep` over the tree finds **no `imagePullSecrets` anywhere** — not in
+   `platform/pm-agent/deployment.yaml`, not in `infra/agent/worker-job-template.yaml`, not
+   in `conformance/incluster/`.
+
+## What I did NOT verify
+
+- **Why the node sees a nameless certificate.** I measured that it does, and that the LAN
+  does not. I did not diagnose it: I did not read the nodes' resolver or
+  `/etc/rancher/k3s/registries.yaml` (node files, not reachable read-only through the API),
+  and I did not check whether the node resolves the name to the same `192.168.2.20` the
+  workstation does or to something else.
+
+  **A correction to a remark I first wrote here:** I cited
+  `cluster-vars`'s `GATEWAY_VIP: 192.168.1.53` as suspicious next to the nodes'
+  `192.168.2.x`. That was noise — the operator confirms those base `cluster-vars` values are
+  **dev-era leftovers for a cluster that is decommissioned**, so the 192.168.1.x addresses in
+  that file say nothing about prod and should not be read as evidence about anything. The
+  measured facts stand on their own and none of them depended on that knob.
+- **Whether the package would be pullable once authenticated.** Nothing has ever been
+  pushed to that path, so "a credential is sufficient" is untested. The only thing
+  established is that no credential is insufficient.
+- **Which fix the operator wants.** Two routes, and they are not equivalent:
+  (a) an `imagePullSecret` — a `dockerconfigjson` holding a Gitea token, SOPS-encrypted
+  like `pm-agent-credentials`, referenced from the Deployment *and* from the committed
+  worker Job template; note the Job template's shape is pinned by admission, and
+  `imagePullSecrets` is a field that policy does not currently constrain, so adding one is
+  a template change that needs a deliberate look at whether the policy should pin it too.
+  (b) allowing anonymous reads of that package in Gitea — no manifest change at all, but it
+  makes the image world-readable to anything that can reach the host.
+  I have implemented neither: both need an operator action I must not take, and (b) is a
+  posture decision. The docs now present both.
+- **Everything the #57 entry listed as unverified about RUNTIME remains unverified.** The
+  image has still never been built or pushed. No dispatcher has authenticated to Gitea, no
+  issue has been triaged, no worker Job has been created — so the admission policy, though
+  now provably applied and Enforce, has still never refused or admitted a real Job. The
+  network policies have not carried traffic.
+- **I did not re-run the merge gate against `main` after the merge.** This branch's own gate
+  run is the evidence for this change; the merged state's is CI run on PR #57.

decisions/2026-10-05-vpn-runbook-ansible-command.md

@@ -0,0 +1,112 @@
+---
+date: 2026-10-05
+branch: fix/vpn-runbook-ansible-command
+title: vpn-remote-access runbook — step 1 named a script that does not exist, and step 2 cited the wrong reason the key is gitignored
+---
+
+## What changed
+
+Two defects in `runbooks/vpn-remote-access.md` (merged in #55), both authored by me, both reported
+by the operator the first time they tried to follow it.
+
+**1. Step 1 named a command that does not exist.** It said:
+
+    cd infra/ansible && ./site.sh --tags system --limit k3s_prod
+
+There is **no `site.sh` anywhere in this repo** (`find . -name site.sh` → nothing). The invocation
+was wrong in four separate ways at once, which is what makes it worth recording rather than just
+patching: wrong kind of thing (this repo drives Ansible with `ansible-playbook <playbook>`, never a
+wrapper script — `playbooks/site.yml` exists and is the full bring-up chain, not this); wrong
+playbook (the module work lives in `playbooks/node-baseline.yml`); wrong targeting flag (`--limit`
+instead of this repo's `-e k3s_hosts=...` convention, which `infra/ansible/README.md` "Targeting
+prod" documents for every playbook); and no toolbox wrapper, though Ansible exists only inside the
+pinned container and `ansible.cfg` is only read from `infra/ansible`.
+
+Replaced with the real command, plus the three things about it that are load-bearing:
+
+    ./infra/devcontainer/dev.sh run bash -c \
+      'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system'
+
+- `-e k3s_hosts=k3s_prod` — the playbook defaults to `k3s_servers`, which is the **dev** group;
+  dev is decommissioned, so an unqualified run does nothing useful.
+- `--tags system`, never untagged — the two kernel-module tasks carry that tag, and an untagged run
+  would also hit the `disks` tasks, which assert on block-device variables prod deliberately leaves
+  as placeholders (the playbook's own header warns about this). `--tags system` is broader than the
+  modules alone (sysctls, iscsid, qemu-guest-agent, swap masking) but each is idempotent and
+  already in its desired state on a live node.
+
+**2. Step 2 gave the wrong reason the server key is safe from git — and the right reason is
+fragile.** It told the operator to stage the key at `/work/.cache/server.key` "inside the repo's
+gitignored cache". `.cache/` is **not** gitignored: `git check-ignore -v .cache/serverkey.txt`
+matches nothing. The file was ignored only incidentally, by `.gitignore` line 11 `*.key` — so the
+path happened to be safe for a reason I had not identified, and **any operator who renamed it
+(`server.pem`, `wg-server`, `privatekey`) would have had a committable WireGuard private key sitting
+in the checkout.** The gate would not have caught it either; `no-plaintext-secrets` does not
+recognise a bare WireGuard key.
+
+The step now states the real rule, marks the `.key` suffix as the thing doing the work, and gives
+the operator a verification they can run before trusting it (`git check-ignore -v` must print a
+rule, and `git status --short` must not list the file).
+
+Nothing outside `runbooks/vpn-remote-access.md` changed; no manifest, no behaviour.
+
+## What I verified
+
+- **`site.sh` does not exist**: `find . -name 'site.sh' -not -path './.git/*'` → no output.
+- **The real surface**: `infra/ansible/playbooks/` contains `node-baseline.yml` and `site.yml` (and
+  11 others); `node-baseline.yml`'s own header documents both
+  `-e k3s_hosts=k3s_prod` and the tag-scoping hazard. The `-e <playbook>_hosts=` convention is
+  confirmed by `infra/ansible/README.md` §"Targeting prod" and by the existing
+  `runbooks/node-memory-swap.md`, which invokes `ansible-playbook playbooks/node-health.yml -e
+  node_health_hosts=...`.
+- **`k3s_prod` is a real inventory group**: `[k3s_prod:children]` in `inventory/hosts.ini`, with
+  `[k3s_prod_primary]`/`[k3s_prod_secondary]`.
+- **`--tags system` covers the kernel-module tasks**: both "Persist kernel modules…" and "Load
+  kernel modules now" carry `tags: [system]` in `roles/node_baseline/tasks/main.yml`.
+- **The gitignore claim, measured both directions**:
+  `git check-ignore -v .cache/server.key` → `.gitignore:11:*.key`;
+  `git check-ignore -v .cache/serverkey.txt` → no match. So the suffix is the only thing ignoring it.
+- **Every path the component's docs reference resolves**: all six relative markdown links in
+  `platform/vpn-ingress/README.md` (checked from that file's own directory), and both Ansible
+  playbook paths.
+- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
+  `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; the rc 3 is the two
+  environmental skips (`gitea-settings` — no local token; conformance L2/L3 — no `CLUSTER_AVAILABLE`).
+  **Not green. Reported as rc 3, not as pass.**
+- **This branch touches NO sensitive path** (`runbooks/**` and `decisions/**` are not in
+  `infra/gitea/approval-policy.json`), so unlike #55 it needs no `/approve-override`.
+
+## What I did NOT verify
+
+- **The corrected command has not been RUN.** Ansible is an operator action, and this is the
+  decisive gap: I have verified the playbook, the group, the tag and the flag all exist and are
+  the documented convention, but not that the run succeeds end to end or that
+  `lsmod | grep '^wireguard'` then reports the module. It is a better-evidenced command, not a
+  proven one.
+- **Whether `--tags system` is entirely side-effect-free on a live prod node.** Each task it runs is
+  idempotent by construction, but I did not dry-run it (`--check`) to confirm it reports zero
+  changes beyond the module load.
+- **The rest of the runbook's commands past step 2** are unchanged and still unexercised — in
+  particular `kubectl exec ds/wireguard -- wg show`, which cannot be reached while the pods sit in
+  `Init` waiting on the server key.
+- **Whether any OTHER document I wrote in #55 contains an invented command.** I re-checked every
+  file path and markdown link in the component (all resolve) and every script the docs tell an
+  operator to run, but "the referenced file exists" is weaker than "the command is correct" — which
+  is exactly the gap that produced `site.sh`.
+
+## Post-merge state of #55, recorded here since its own entry predates the merge
+
+Measured on live prod after #55 merged as `ad3f56e`:
+
+- Flux applied it by the real path: Kustomization `vpn-ingress` `lastAppliedRevision:
+  main@sha1:ad3f56e`, equal to the GitRepository's artifact revision, `Ready=True`.
+- All **18** Flux Kustomizations `Ready=True` — the new tier disturbed nothing, and `wait: false`
+  behaved as intended (`vpn-ingress` is Ready while its own pods are not, so nothing downstream
+  stalled on it).
+- **`hostNetwork` is confirmed in effect, not merely configured**: the three pods report pod IPs
+  `192.168.2.10/.11/.12` — the node addresses, one per node.
+- **It fails closed exactly as designed**, with a legible cause: all three pods `Init:0/1`,
+  `MountVolume.SetUp failed for volume "wg-server-key" : secret "wg-server-key" not found`, and no
+  Secret in the namespace. The #55 entry's "the DaemonSet has never been reconciled" is superseded
+  by this paragraph; its remaining unverified claims (the tunnel carrying a packet, the
+  iptables/nftables backend question) still stand.

decisions/2026-10-05-vpn-server-key-in-sops.md

@@ -0,0 +1,187 @@
+---
+date: 2026-10-05
+branch: fix/vpn-server-key-in-sops
+title: VPN server key moves into SOPS-encrypted git — the out-of-band Secret broke R-SEC-01 and the DR story
+---
+
+## What changed
+
+**Operator correction (2026-10-05): "secrets are kept in source through sops".** They are right, and
+#55 shipped the WireGuard server private key the wrong way: as a Secret the operator was told to
+`kubectl create` out of band, referenced by name only. This branch moves it into the tree as
+`platform/vpn-ingress/wg-server-key.sops.yaml`, where every other secret in this repo lives.
+
+**Why the original choice was wrong, not merely unconventional.** Two concrete costs, both of which
+#55's own README half-admitted and then shipped anyway:
+
+1. **R-SEC-01 / spec docs/07 says secrets live in source, SOPS-encrypted, reaching the cluster only
+   through Flux.** An out-of-band Secret is outside the GitOps tree by construction.
+2. **It broke DR for the one credential that cannot be re-derived.** A cluster rebuild restores
+   everything in git; it does not restore a hand-made Secret. Losing this key means **re-enrolling
+   every device**, because each client profile pins its public half. #55 papered over this with
+   "keep a copy in your password manager", which is a workaround for a self-inflicted gap.
+
+Note `platform/kyverno/admission/secret-provenance.yaml` did **not** forbid the original shape —
+it scopes to `app-namespace=true` namespaces and `vpn-ingress` is deliberately a platform namespace,
+so admission would have accepted the hand-applied Secret. The rule it violated was the written
+requirement, not an enforced control. Recording that because "the gate let it through" was never
+evidence it was correct.
+
+**How the unfillable-key problem is solved.** An agent still must not mint a long-lived private key
+(CLAUDE.md §6), so the file is committed **already encrypted, with `privatekey: REPLACE-ME`**. That
+makes the wiring complete and the gate green from the first commit — the resource line and both
+overlay decryption patches are present — and leaves the operator exactly one act: replace the value.
+The alternative, a commented-out resource line to uncomment later, is precisely the "file in git with
+no generator entry: reviewed, merged, apparently done, still broken" shape this repo keeps getting
+bitten by (`alert-runbooks-resolvable` exists because of it).
+
+**The init-container guard is now a validation, not an emptiness check.** It was `[ -z "$key" ]`.
+Because the committed placeholder is non-empty, the likely failure is now a value that is present
+and *wrong*, so the guard pipes the value through `wg pubkey` — which accepts only a real 32-byte
+curve25519 private key — and names which of four cases it hit:
+
+| case | why it matters |
+|---|---|
+| `REPLACE-ME` placeholder | the expected fresh-install state |
+| empty | blanked value |
+| **starts with `ENC[`** | **the Secret reached the cluster UNDECRYPTED.** A missing overlay decryption patch applies cleanly and leaves every Flux condition green — this is the silent failure the guard exists for |
+| rejected, none of the above | mangled value, or a *public* key pasted by mistake |
+
+Files:
+
+- `platform/vpn-ingress/wg-server-key.sops.yaml` (new) — SOPS-encrypted to the repo's age recipient,
+  `stringData.privatekey` only, placeholder value.
+- `platform/vpn-ingress/wg-server-key.sops.yaml.example` (new) — plaintext template + operator
+  hand-off, following the `crafty-oauth2-proxy-credentials.sops.yaml.example` convention.
+- `platform/vpn-ingress/kustomization.yaml` — the resource added; the "DELIBERATELY ABSENT: a
+  Secret" block replaced with the overlay-patch warning.
+- `platform/vpn-ingress/daemonset.yaml` — the `wg pubkey` guard; volume comment rewritten.
+- `clusters/dev/kustomization.yaml`, `clusters/prod/kustomization.yaml` — **the `vpn-ingress`
+  `decryption:` patch, in both.** This is the load-bearing half: #55's own kustomization comment
+  predicted that forgetting it applies ciphertext cleanly.
+- `clusters/base/vpn-ingress.yaml` — its "NO decryption stanza is needed" comment was now false.
+- `platform/vpn-ingress/README.md`, `runbooks/vpn-remote-access.md` — step 2 rewritten.
+- `infra/vpn-ingress/new-peer.sh` — stale "step 1" pointer.
+
+**A second finding: `wg` and `sops` were not in the same place — and the first fix for that was
+also wrong.** Measured: `wg` existed only on the host (`/usr/bin/wg`), `sops` only in the dev
+container, `qrencode` on neither. So `sops -d … | wg pubkey` could not run in one shell, and #55's
+step 2 — which assumed it could — would have failed.
+
+My first fix was to **split every step by where it runs**, and to tell the operator to
+`dnf install qrencode` on their workstation. **Operator correction: "sop, wg, and any other tool
+requirement is supposed to be baked into the devcontainer."** That is CLAUDE.md §5 and the
+devcontainer README's "Don't install the toolchain on the host", and the split fix broke the
+invariant rather than honouring it — it also produced a procedure that could never be reproduced in
+CI, and which no one should have to follow.
+
+**The real fix:** `wireguard-tools` and `qrencode` are now distro packages in
+`infra/devcontainer/Containerfile`, alongside ansible/openssh/diffutils. Both are in Fedora 44's
+repos (`wireguard-tools` 1.0.20260223 — the same upstream version as the container image this
+component runs — and `qrencode` 4.1.1), unpinned like every other dnf package there, because the
+pinned thing is the Fedora base and neither is a gate tool whose version changes what green means.
+
+Three things got *simpler* rather than merely relocated, which is the tell that the operator's rule
+was the right one:
+
+1. **Step 2 is now one pipeline in one shell:** `sops set … "\"$(wg genkey)\""` generates the key
+   straight into the already-encrypted document. The plaintext never lands on disk, never reaches a
+   terminal, and there is no staged copy to shred. The previous version needed a copy in
+   `.cache/server.key`, a `git check-ignore` warning about the `*.key` suffix being the only thing
+   protecting it, and a sha256 comparison to prove the right bytes arrived. All of that is gone.
+2. **`new-peer.sh` now derives the server's public key from `wg-server-key.sops.yaml` itself**
+   (`sops -d --extract … | wg pubkey`), so there is no local key copy to keep in step with the
+   encrypted one. It previously required the operator to keep the private key at a magic host path.
+3. **`.cache/` is now gitignored at the repo root** (mirroring `infra/ansible/.gitignore`), so the
+   container has a persistent, non-committable scratch directory. `new-peer.sh` writes client
+   profiles to `.cache/wg-peers/` by default — necessary because the repo bind-mount is the
+   container's ONLY persistent read-write mount, so a profile written to `/tmp` in there is gone
+   before it can be moved to a desktop. Its old blanket "refuse to write inside the checkout" guard
+   is replaced by `git check-ignore`, which tests the actual property instead of a proxy.
+
+**SENSITIVE PATHS — this PR touches `clusters/**` (two overlays) AND adds a `**/*.sops.yaml`, so
+`ci/approval-policy.sh` will BLOCK it.** It needs the operator's `/approve-override <reason>` as a
+PR **review** of type "Comment". I have deliberately not posted it (CLAUDE.md §1).
+
+## What I verified
+
+- **The encrypted file is real and correctly targeted.** `sops -e -i` in the pinned toolbox
+  produced `stringData.privatekey: ENC[AES256_GCM,...]` with cleartext keys, `encrypted_regex
+  ^(data|stringData)$`, and exactly one age recipient:
+  `age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg` — the recipient in the root
+  `.sops.yaml`, i.e. the key Flux holds as `sops-age`.
+- **Round-trip, without exposing anything**: `sops -d` then a Python read of key names only →
+  `stringData keys: ['privatekey']`, and `value is still the placeholder: True`.
+- **Both overlay patches are present and name the right Kustomization** (`name: vpn-ingress`,
+  `namespace: flux-system`, `decryption.secretRef.name: sops-age`), inserted after each overlay's
+  `velero` stanza, mirroring headlamp/crafty/velero.
+- **`kubectl kustomize platform/vpn-ingress` builds** and now emits five objects — Namespace,
+  ConfigMap, **Secret**, DaemonSet, NetworkPolicy — with `${WG_CLIENT_CIDR}` still an unexpanded
+  token for Flux.
+- **The tool gap, measured on both host and container before the fix**: `wg` host-only, `sops`
+  container-only, `qrencode` on neither.
+- **Both packages exist in the image's repos**, queried from inside the running container:
+  `wireguard-tools` 1.0.20260223 (repo `updates`) and `qrencode` 4.1.1 (repo `fedora`).
+- **The image REBUILDS with them and both tools resolve inside it** —
+  `./infra/devcontainer/dev.sh build` succeeded, and `wg --version` / `qrencode --version` /
+  `sops --version` all answer in the rebuilt container. This is the claim the change rests on, so
+  it was executed rather than reasoned.
+- **`new-peer.sh`'s new SOPS path fails correctly against the placeholder.** Run in the rebuilt
+  container it refused with "could not derive the server public key … it still holds the REPLACE-ME
+  placeholder", which is both the expected pre-key behaviour and a live test of the
+  `sops -d | wg pubkey` pipeline the whole design now depends on.
+- **`.cache/` is ignored regardless of filename** now: `git check-ignore -v .cache/anything.conf`
+  and `.cache/serverkey.txt` both match `.gitignore:46:.cache/`. Before this change only `*.key`
+  matched, which is the fragility the previous runbook had to warn about.
+- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
+  `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two
+  environmental skips (`gitea-settings` — no local token; conformance L2/L3 — no cluster).
+  **Not green. Reported as rc 3, not as pass.**
+- **The L0 gate caught THREE real defects in this branch before it could merge — including one
+  in the fix for another — and I had asserted one of them was fine without running it.** Both are recorded because the gate's value here is the
+  whole point of the exercise:
+  1. **`no-plaintext-secrets` FAILED**: "`wg-server-key.sops.yaml.example`: an unencrypted .example
+     template is referenced by a kustomization — it would be applied as a plaintext Secret". My
+     kustomization named the template in a *comment*, and the check matches comments too (correctly:
+     naming one is how a plaintext Secret gets applied by accident). `platform/crafty/kustomization.yaml`
+     never names its own example either — the convention was there to copy and I missed it. Fixed by
+     not naming the file; the hand-off lives inside the example, where it belongs.
+     **An earlier draft of this entry claimed this check passed. It did not. The claim was written
+     from expectation rather than from a run.**
+  2. **`flux-postbuild-substitution` FAILED** on `${key#ENC[}` in my new guard
+     (`daemonset.yaml:155`), flagged `[SILENT-EMPTY]`: envsubst consumes every `${...}`, so my
+     shell prefix-strip would have rendered to the empty string — the ciphertext branch of the guard
+     would never have fired, which is precisely the silent failure the guard was added to catch.
+     Rewritten with `case`, which uses no braces and so has nothing to escape.
+  3. **And then it failed a THIRD time, on the fix's own comment** — `[FATAL-PARSE]
+     daemonset.yaml:156 ${...}`, the prose explaining the rule, written with the literal form. That
+     class is worse than it looks: a `FATAL-PARSE` hazard fails the **entire** Kustomization, every
+     resource in the path, not just the line. Rewritten as "dollar-brace expansion". This is the
+     second time in this component that writing the hazard literally inside a comment about the
+     hazard has been the defect (the first was in #55), so the habit to build is: **in any
+     substituted path, describe the construct in words, never type it.**
+- **After both fixes, `no-plaintext-secrets` and `flux-postbuild-substitution` pass** (446 and 198
+  assertions respectively), and `l0` is green.
+
+## What I did NOT verify
+
+- **That Flux decrypts this file on prod.** The overlay patches have never reconciled. This is the
+  whole risk of the change and it has exactly one observable outcome: after merge the pods should
+  CrashLoop reporting the **`REPLACE-ME` placeholder** (proving decryption worked), *not* reporting
+  **`ENC[`** (which would prove a patch is wrong). I built the guard to tell those apart because I
+  cannot test them.
+- **That `wg pubkey` accepts the operator's real key through this path.** The guard is reasoned from
+  `wg pubkey`'s behaviour and has never run — `wg` is absent from both this workstation's container
+  and the gate, so the first execution will be in the pod.
+- **The `sops set` command in the hand-off.** I verified `sops -e -i` and `sops -d` in the toolbox;
+  I did **not** run `sops set` with the quoting shown, and nested quoting inside
+  `dev.sh run bash -c` is exactly where a documented command tends to break. If it misbehaves, the
+  editor form (`sops <file>`) is the fallback and does not depend on quoting.
+- **The sha256 comparison in step 2 step 4** is untested end to end (it needs the real key).
+- **Whether anything else in #55 assumed a tool is available where it is not.** I checked `wg`,
+  `sops` and `qrencode` because step 2 depended on them. I did not audit every command in the
+  runbook the same way — and that class of defect has now produced two corrections in a row
+  (`site.sh`, then `sops | wg`), so assume more remain until each step has actually been run.
+- **The live VPN still carries no traffic**, and the earlier gaps stand unchanged: the
+  iptables-vs-nftables question, `allowPrivilegeEscalation: false` versus the image's s6 init, and
+  any off-LAN reachability.

decisions/2026-10-06-new-peer-ignores-template-comments.md

@@ -0,0 +1,73 @@
+---
+date: 2026-10-06
+branch: fix/new-peer-ignores-template-comments
+title: new-peer.sh refused the first VPN address because its duplicate check matched its own commented-out example
+---
+
+## What changed
+
+The operator's first real enrolment failed:
+
+```
+$ ./infra/vpn-ingress/new-peer.sh keith-phone 2
+new-peer.sh: 10.90.0.2/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet.
+```
+
+**Nothing was allocated.** `peers.yaml` had zero peers — the pod's own init container reported
+`wg0.conf assembled: 0 peer(s)`. The duplicate-address check grepped the WHOLE file, and
+`peers.yaml` line 66 is a **commented-out example stanza** I wrote, containing literally
+`#   AllowedIPs = 10.90.0.2/32`. So `.2` — the first address anybody would ask for, and the one my
+own runbook uses in its example — was permanently un-allocatable, with a message asserting the
+opposite of the truth.
+
+Both halves fixed, because either alone leaves the trap half-set:
+
+- **`infra/vpn-ingress/new-peer.sh`**: the check strips comment lines before matching
+  (`grep -vE '^[[:space:]]*#' | grep -q …`). That is the correct predicate rather than a patch: a
+  real stanza's `AllowedIPs` line is never commented, while both the template and every peer's
+  `# <name> — <owner>` label always are.
+- **`platform/vpn-ingress/peers.yaml`**: the example now uses placeholders that cannot match
+  anything real — `# <device-name> — <owner>, enrolled <YYYY-MM-DD>` and
+  `AllowedIPs = 10.90.0.<octet>/32`. A future grep cannot be fooled by it either.
+
+The bug class is worth naming: **a documentation example that is byte-identical to real data will
+eventually be parsed as real data.** The same shape caused two earlier defects in this component —
+`flux-postbuild-substitution` flagged a `${...}` written inside a comment *explaining* that hazard,
+and `no-plaintext-secrets` flagged the `.sops.yaml.example` because the kustomization named it in a
+comment. Three instances now; the rule is to make examples structurally unmatchable, not merely
+commented out.
+
+No sensitive paths (`platform/vpn-ingress/**`, `infra/vpn-ingress/**`, `decisions/**`), so this
+merges on a green gate with no override.
+
+## What I verified
+
+Run in the pinned toolbox against the live tree, 2026-10-06.
+
+- **`.2` now works**, and it went through the REAL key path: `new-peer.sh keith-phone 2` rendered a
+  profile with `Address 10.90.0.2/32`, `AllowedIPs = 10.90.0.2/32` in the printed stanza, full
+  tunnel, at `.cache/wg-peers/keith-phone.conf` (0600). The throwaway keypair was deleted after.
+- **This incidentally closes a gap left open in #60**: the profile could only have been rendered by
+  deriving the server's public key via `sops -d --extract … | wg pubkey` from
+  `wg-server-key.sops.yaml` — so that pipeline is now proven **against the operator's real key**,
+  not just against a placeholder it was expected to reject.
+- **A real duplicate is still refused.** Appending an uncommented
+  `AllowedIPs = 10.90.0.5/32` stanza and asking for octet 5 produced the correct refusal; the tree
+  was restored afterwards and `git diff --stat` showed only the intended template edit.
+- **Zero active allocations remain**: `grep -vE '^[[:space:]]*#' peers.yaml | grep -c AllowedIPs`
+  → `0`.
+- **Toolbox gate**: rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`,
+  `l1` and `approval-policy` all pass; rc 3 is the two environmental skips (`gitea-settings`,
+  conformance L2/L3). **Not green. Reported as rc 3.**
+
+## What I did NOT verify
+
+- **No peer is enrolled by this branch.** The keypair I generated was a throwaway and was deleted;
+  the operator must run the script themselves, since the device's private key must exist only on
+  the device. So the end of the enrolment path — a stanza merged into `peers.yaml`, Flux rolling
+  the DaemonSet, and `wg show` listing a peer — is still unexercised.
+- **The duplicate-NAME warning** (`grep -qi "^ *# *${NAME} "`) deliberately still inspects
+  comments, because per-peer labels are comments. It is no longer tripped by the template, but I
+  did not test it against a real enrolled peer.
+- **Nothing about client traffic.** The WAN port-forward is still not configured and there is still
+  no off-LAN vantage point, so the tunnel has carried no packet.

decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md

@@ -0,0 +1,126 @@
+---
+date: 2026-10-06
+branch: docs/pm-agent-registry-pull-prereq
+title: The nodes resolve gitea to the WAN, so the failing certificate is the router's — the second node process to need the issuer pin
+---
+
+## What changed
+
+The image pull for `pm-agent` failed TLS verification, and the previous entry on this branch
+honestly recorded it as **not diagnosed**. The operator asked for the diagnosis before any
+registry design. This is it, and the fix turns out to be a pattern this repository already
+has.
+
+`infra/ansible/inventory/group_vars/k3s_prod/vars.yml` now pins
+`gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `node_baseline_hosts_pins`, beside the
+Keycloak issuer pin that is there for the identical reason.
+
+**It reaches the nodes the way every other node setting in this project does — through
+Ansible, from the inventory.** Node configuration is Ansible's (and, for the dev VMs,
+OpenTofu's); nothing here touches a node by hand. For prod specifically, Ansible is the
+*only* surface: `infra/PROD-PROVISIONING.md` records that prod is 3× bare metal and that the
+dev path (OpenTofu → Proxmox VM → cloud-init) does not apply, so there is no
+cloud-init/tofu half to keep in parity for this. The one scoped, idempotent command:
+
+    cd infra/ansible && ansible-playbook playbooks/node-baseline.yml \
+      -e k3s_hosts=k3s_prod --tags hosts
+
+`--tags hosts` runs ONLY the `/etc/hosts` task, so applying this pin cannot drift any other
+baseline setting. Running the play is an operator action (it has become/root on prod), but
+the configuration itself is declared, reviewed and version-controlled — not typed on a node.
+
+The docs that described this as an undiagnosed blocker now describe the cause and the fix:
+`platform/pm-agent/README.md`, `runbooks/pm-agent.md` (its symptom row now names the pin and
+the play), and `docs/plans/pm-agent-issue-to-pr.md`.
+
+Also recorded, in the two places an operator reads when creating the credential: **the agent
+authenticates to Gitea with a local account and a scoped PAT, by decision** (operator,
+2026-10-06), not with a federated identity. Written down because the alternative looks
+obviously better until you check whether it is possible — see below.
+
+## What I verified
+
+Every step measured, read-only, in this order.
+
+**The failing certificate is the router's, not the Gateway's.**
+
+- containerd's error names TLS, not the digest: `tls: failed to verify certificate: x509:
+  certificate is not valid for any names, but wanted to match gitea.home.keithjackson.dev`.
+  "not valid for **any** names" means the presented certificate carries no SANs at all.
+- `192.168.2.20` is the shared Gateway's own LoadBalancer VIP
+  (`gateway/cilium-gateway-shared`, from the Cilium pool `192.168.2.20-30`), and its `https`
+  listener serves `*.home.keithjackson.dev` with `wildcard-home-tls`. From the operator's
+  workstation the name resolves there and serves the real Let's Encrypt
+  `CN=*.home.keithjackson.dev` with correct SANs. **So the certificate and the Gateway are
+  healthy.**
+- **The node asks a different resolver and gets a different answer.** Prod's
+  `node_baseline_dns_servers` is the LAN gateway, not Pi-hole — deliberately, as that file
+  explains. Public DNS for this zone is a CNAME to `wan.home.keithjackson.dev`: measured via
+  both `1.1.1.1` and `8.8.8.8`, `gitea.home.keithjackson.dev` →
+  `wan.home.keithjackson.dev.` → `136.51.115.252`, and the apex answers the same.
+- **That address serves a certificate with no names.** Measured from the workstation against
+  the WAN address with the right SNI:
+
+      subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
+      issuer=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
+      No extensions in certificate
+
+  No extensions means no `subjectAltName`, which is exactly "not valid for any names". The
+  `CN=%D` is the router's placeholder.
+
+**This repository had already measured the same thing, for Keycloak, and I did not need to
+re-derive it.** `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` says, above the pin:
+"Since platform/gateway/gateway.yaml set an external-dns `target`, every public name is a
+CNAME to wan.home.keithjackson.dev = the home WAN IP. The SOHO router does NOT hairpin: from
+the LAN, WAN:443 answers with the router's own self-signed placeholder cert (measured from
+k3s-prod-2/-3 2026-10-03: `curl` exit 60, ssl_verify_result=18)." Same cause, same symptom,
+one name later. The pin exists precisely for node processes that must reach the Gateway
+directly; containerd is one, and nobody had added it.
+
+**The parity the pin has to satisfy:** `192.168.2.20` equals the live
+`GATEWAY_VIP`, and `k3s_base_domain` equals the live `BASE_DOMAIN` — both read from the
+cluster, not from the comment that claims prod serves `prod.home.keithjackson.dev` (it does
+not; `flux-system/cluster-vars` reports `BASE_DOMAIN=home.keithjackson.dev` and
+`GATEWAY_VIP=192.168.2.20`, and Ansible's `k3s_base_domain` is `home.keithjackson.dev`).
+That comment is stale history.
+
+**The credential decision, and why the obvious alternative is not available.** Gitea's API
+accepts basic auth, PATs, SSH signatures and its own OAuth2 bearer tokens; as an OAuth2
+provider Gitea implements **only the Authorization Code Grant** (plus PKCE/OIDC), so there
+is no client-credentials grant, and nothing in Gitea validates a token issued by a
+third-party OIDC provider. A Keycloak service-account token therefore cannot authenticate to
+Gitea's API or to git-over-HTTP. Read from Gitea's own documentation
+(`docs.gitea.com/development/api-usage`, `/development/oauth2-provider`), not assumed.
+
+**Gate:** `./ci/gate.sh` in the pinned toolbox from a §0 worktree — see the branch's run; rc 3
+with `gitea-settings` the only stage that did not run.
+
+## What I did NOT verify
+
+- **That the pin fixes the pull.** It is IaC, and nothing has applied it: `/etc/hosts` on the
+  nodes is written by the `node_baseline` play's `hosts`-tagged task, which is an operator
+  action. I did confirm the mechanism is not inert — `roles/node_baseline/tasks/main.yml`
+  renders `node_baseline_hosts_pins` into a marked `blockinfile` with `backup: true`, and
+  `state: absent` when the list empties, so a removed entry really leaves the node. Until that runs
+  and an image exists, "the node now reaches the Gateway" is a design claim. The evidence
+  that it *will* is the Keycloak pin working for the same cause — which I also did not
+  re-measure.
+- **Why public DNS answers at all for an `internal` service.** `catalog/gitea.yaml` sets no
+  `exposure`, so it inherits `internal`, and the renderer contract says an internal service's
+  HTTPRoute carries `external-dns.alpha.kubernetes.io/controller: none` and publishes no
+  public record. Yet `gitea.home.keithjackson.dev` resolves publicly to the WAN CNAME. I did
+  not chase that: it may be a zone-level wildcard rather than a published record, and it did
+  not change the diagnosis. It is worth someone's attention, because "internal services
+  publish no public DNS" is an L0-checked claim and this looks like a case where the *name*
+  resolves regardless of what the renderer withheld.
+- **Registry authentication.** Unchanged and still open: an unauthenticated client gets
+  nothing from this registry, and no `imagePullSecret` exists anywhere in the tree. With the
+  credential model now decided (local Gitea accounts with PATs) the shape is a
+  `dockerconfigjson` for an account whose token carries `read:package`, referenced from the
+  Deployment and the committed worker Job template — and the worker Job template's shape is
+  pinned by admission, so whether the policy should also pin `imagePullSecrets` deserves a
+  deliberate look rather than a quiet addition. Not built: minting that token is an operator
+  action, and I did not want to guess at a second credential's scope.
+- **Anything about runtime.** The image still does not exist. No dispatcher has
+  authenticated, no issue has been triaged, no worker Job has been created, and the admission
+  policy — live, `Enforce`, six rules — has still never admitted or refused a real Job.

decisions/2026-10-06-vpn-ingress-ip-forward.md

@@ -0,0 +1,113 @@
+---
+date: 2026-10-06
+branch: fix/vpn-ingress-ip-forward
+title: The VPN tunnel never came up — a hostNetwork pod cannot write net.ipv4.ip_forward, and wg-quick treats a failed PostUp as fatal
+---
+
+## What changed
+
+The remote-access VPN reconciled for the first time with a real server key (#60 then #62) and
+**the tunnel did not come up**. One line in my own manifest was the whole cause.
+
+Measured in the `wireguard` container on prod, 2026-10-06:
+
+```
+[#] ip link add dev wg0 type wireguard
+[#] wg addconf wg0 /dev/fd/63
+[#] ip -4 address add 10.90.0.1/24 dev wg0
+[#] ip link set mtu 1420 up dev wg0
+[#] sysctl -q -w net.ipv4.ip_forward=1
+sysctl: permission denied on key "net.ipv4.ip_forward"
+[#] ip link delete dev wg0
+**** Tunnel /config/wg_confs/wg0.conf failed, will stop all others! ****
+```
+
+Everything up to that point worked. **The cause is not a missing capability.** With
+`hostNetwork: true` the runtime mounts `/proc/sys` **read-only** in the container — confirmed in the
+pod's own `/proc/mounts`: `proc /proc/sys proc ro,nosuid,nodev,noexec,relatime` — precisely because
+those sysctls are the *host's* globals and a container must not rewrite them. `CAP_NET_ADMIN` cannot
+lift a read-only mount, so this can never succeed from here. And `wg-quick` treats any failed
+`PostUp` as fatal, so it deleted the interface it had just finished building.
+
+**The line was never needed.** `node_baseline` writes `net.ipv4.ip_forward=1` to
+`/etc/sysctl.d/99-k3s-cilium.conf`, and reading the value from inside the pod returns `1`. I had
+written the header comment *correctly* — "ip_forward is ALREADY 1 on these nodes … so no pod sysctl
+is needed" — and then emitted the sysctl anyway, justified in its own comment as "re-asserted so
+this component does not silently depend on another role's sysctl file surviving". That defensive
+re-assertion is the entire outage. **The lesson is narrow and reusable: in a hostNetwork pod,
+verify host state, never write it.**
+
+Fix:
+
+- **`platform/vpn-ingress/daemonset.yaml`**: the `PostUp = sysctl …` line is gone, replaced by a
+  **read** in the init container — `grep -qx 1 /proc/sys/net/ipv4/ip_forward` — which fails closed
+  with a message naming the `node_baseline` role. Reading `/proc/sys` is permitted; only writing is
+  not. Fail-closed still matters: a tunnel that comes up but cannot forward would satisfy every
+  probe in this component and silently drop every client's traffic.
+- The header bullet that said "no pod sysctl is needed" now also says **do not attempt one**, with
+  the mechanism, so the next author does not re-add it for the same reason I did.
+- The `securityContext` comment claimed NET_ADMIN was for "create the wg0 link, set the host sysctl,
+  and write the netfilter rules". The middle clause was false; corrected.
+- The image's `chown: … Operation not permitted` startup run — plus its own "Permissions could not
+  be set … we will not provide support for it" — is now documented as **expected and harmless**, in
+  both the manifest and the runbook, so nobody chases it. Capabilities are dropped to `NET_ADMIN`
+  only, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs as root,
+  and `wg-quick` proceeds past it (as it did here — the failure was 30 lines later).
+- **`runbooks/vpn-remote-access.md`**: a "what a healthy start looks like" block with the real log
+  lines; `ip_forward is 0` added to the init-failure table; and the note that `0/1 Running` with
+  `Startup probe failed: Unable to access interface: No such device` is the signature of `wg-quick`
+  tearing the tunnel down — read the `[#]` line it died on.
+- **`platform/vpn-ingress/README.md`**: a table of what the first reconcile settled (below).
+
+No sensitive paths: `platform/vpn-ingress/**`, `runbooks/**` and `decisions/**` only. This merges on
+a green gate with no override.
+
+## What I verified
+
+Live prod, read-only, 2026-10-06. **Four questions this component's design rested on are now
+measured rather than assumed — three went the right way, one was the defect above.**
+
+- **The GitOps path is proven end to end.** `Kustomization vpn-ingress` `lastAppliedRevision:
+  main@sha1:cf7d5e4` equals the GitRepository artifact revision. The Secret carries
+  `kustomize.toolkit.fluxcd.io/name: vpn-ingress`, so Flux delivered it.
+- **SOPS decryption works, proven by digest rather than by eye.** The value in
+  `platform/vpn-ingress/wg-server-key.sops.yaml` on `main`, decrypted, is 44 bytes, is **not** the
+  placeholder, and **`wg pubkey` accepts it**. Its SHA-256 equals the SHA-256 of the live Secret's
+  decoded `data.privatekey` (`d7c8f8de6f252649…` both sides). No key material was printed at any
+  point — lengths, booleans and digests only.
+- **The `wg pubkey` guard fired correctly, and its `ENC[` branch was never needed.** During the
+  kubelet window before the updated Secret propagated, all three pods reported *"It is still the
+  committed REPLACE-ME PLACEHOLDER"* — the guard distinguishing a stale/unfilled value from
+  ciphertext, exactly as designed. It cleared on its own once the mount refreshed; **a transient
+  `REPLACE-ME` CrashLoop for a minute or two after the key PR merges is expected, not a fault.**
+- **The kernel module reaches the pod**: `ip link add dev wg0 type wireguard` succeeded and
+  `/proc/modules` lists `wireguard`. So the `node_baseline` addition works and `CAP_SYS_MODULE` is
+  correctly not required.
+- **`allowPrivilegeEscalation: false` does NOT break the image's s6 init** — the container reaches
+  `wg-quick` normally. The flag stays; the fear recorded in #55 is closed.
+- **The iptables backends MATCH, which was my single biggest documented doubt.** The image ships
+  `iptables v1.8.13 (nf_tables)` via `/usr/sbin/xtables-nft-multi`, and Debian 13 is nf_tables. The
+  nightmare case — rules accepted into a table the kernel never consults, so handshakes succeed and
+  every forwarded connection hangs — cannot occur on this pairing.
+- **`net.ipv4.ip_forward` is `1`** in the host netns the pod shares, and `/proc/sys` is `ro` there.
+  Both read directly from the pod.
+- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
+  `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two
+  environmental skips (`gitea-settings`, conformance L2/L3). **Not green. Reported as rc 3.**
+
+## What I did NOT verify
+
+- **That the tunnel now comes up.** This fix has not reconciled. The next reconcile is the test, and
+  its success criterion is specific: pods `1/1 Running`, and `wg show` reporting `interface: wg0`
+  with `listening port: 51820`.
+- **That the MASQUERADE and FORWARD rules actually apply.** They are the next statements after the
+  one that failed, so **they have never executed**. The backends matching (above) removes the worst
+  failure mode but proves nothing about the rules themselves. First check after the pods go Ready:
+  `iptables -t nat -S POSTROUTING | grep 10.90.0.0/24` on a node.
+- **Anything about client traffic.** Still zero peers enrolled, still no WAN port-forward, still no
+  off-LAN vantage point. The VPN has carried no packet.
+- **Whether the transient `REPLACE-ME` window has an upper bound.** It cleared within ~2 minutes
+  here; I did not determine the kubelet refresh behaviour, so a longer stall after a key change is
+  uncharacterised.
+- **The `sops set` hand-off command**, still. The operator filled the key in by their own route
+  (#62), so the exact quoting I documented inside `dev.sh run bash -c` remains unexercised.

docs/plans/pm-agent-issue-to-pr.md

@@ -233,9 +233,21 @@ about runtime behaviour in this document is a design claim, not a measurement.
 
 ## Follow-ups, in the order they are worth doing
 
-1. **Build, push, pin.** Operator runs `infra/agent/build-image.sh`, pushes, and one PR
-   replaces both placeholder digests. Expect the build to need iteration: the nodejs module
-   stream and the npm install have never run against this base.
+1. **Make the registry pullable, THEN build, push, pin.** Measured after the merge, and
+   neither half is declared anywhere in this repository
+   (`decisions/2026-10-05-pm-agent-registry-pull-prereq.md`):
+   (i) the nodes resolved the registry hostname to the WAN IP and failed TLS against the
+   router's `CN=%D` placeholder certificate — **diagnosed and fixed** by pinning
+   `gitea.{{ k3s_base_domain }}` to the Gateway VIP in `node_baseline_hosts_pins`, the same
+   pin the Keycloak issuer already carries, applied by
+   `ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`;
+   (ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with
+   the credential decision now being local Gitea accounts with PATs, that means a
+   `dockerconfigjson` for an account whose token carries `read:package`. Only then does the
+   digest matter. Operator runs `infra/agent/build-image.sh`, pushes, and one PR replaces both
+   placeholder digests. Expect the build itself to need iteration too: the nodejs module
+   stream and the npm install have never run against this base. The same two blockers sit in
+   front of `conformance-runner`, which shares the registry path and has never been pulled.
 2. **Create the credentials** per `pm-agent-credentials.sops.yaml.example`, and **verify the
    bot's permission is `write`, not `admin`** — that measurement is the control that keeps
    the agent out of its own approval gate.

infra/agent/pm-worker

@@ -90,9 +90,19 @@ esac
 # pointing at an external host would be REACHABLE: it could clone a repository that is not
 # this one and push this repository's work somewhere else. The allowlist is baked in here,
 # where changing it requires rebuilding the image whose digest admission pins.
+#
+# ONE ENTRY, AND IT IS THE CLUSTER-LOCAL SERVICE NAME. This agent runs INSIDE the cluster, so
+# it reaches Gitea the way every other in-cluster consumer does — `*.svc.cluster.local` —
+# which is by design a different path from how a client outside the cluster reaches the same
+# service. The external hostname `gitea.home.keithjackson.dev` used to be a second accepted
+# value here; it is removed deliberately. It was never needed (nothing sets it), it routes
+# in-cluster traffic out through the Gateway and back for no reason, and as an ACCEPTED value
+# it widened this allowlist to a host reachable over the `world:443` egress rule — which is
+# the exact shape of injection the allowlist exists to refuse. The cluster-local name also
+# depends on none of public DNS, Cloudflare, the Gateway or the certificate.
 case "${PM_GITEA_URL}" in
-  http://gitea.gitea.svc.cluster.local:3000|https://gitea.home.keithjackson.dev) ;;
-  *) die "PM_GITEA_URL '${PM_GITEA_URL}' is not an allowed git host (see this script's header)" ;;
+  http://gitea.gitea.svc.cluster.local:3000) ;;
+  *) die "PM_GITEA_URL '${PM_GITEA_URL}' is not the allowed in-cluster git host (see this script's header)" ;;
 esac
 [[ "${PM_GITEA_OWNER}" =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,38}$ ]] || die "PM_GITEA_OWNER is malformed"
 [[ "${PM_GITEA_REPO}"  =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,99}$ ]] || die "PM_GITEA_REPO is malformed"

infra/agent/probe-validation.sh

@@ -86,12 +86,22 @@ refuses "branch equal to the base branch"       "must not be the base branch" \
 # --- where code comes FROM and goes TO -------------------------------------
 # The namespace's egress policy permits tcp/443 to the world (the model needs it), so an
 # injected host would be REACHABLE. These are the cases that matter most.
-refuses "a redirected git host"                 "is not an allowed git host" \
+refuses "a redirected git host"                 "is not the allowed in-cluster git host" \
         PM_GITEA_URL=https://evil.invalid
-refuses "a real, reachable, WRONG git host"     "is not an allowed git host" \
+refuses "a real, reachable, WRONG git host"     "is not the allowed in-cluster git host" \
         PM_GITEA_URL=https://github.com
-refuses "the right host on a smuggled path"     "is not an allowed git host" \
+refuses "the right host on a smuggled path"     "is not the allowed in-cluster git host" \
         PM_GITEA_URL=http://gitea.gitea.svc.cluster.local:3000/../..
+# This agent is IN the cluster, so it reaches Gitea by its cluster-local service name. The
+# EXTERNAL hostname is refused too — it used to be a second accepted value, and as an
+# accepted value it widened this allowlist to a host reachable over the `world:443` egress
+# rule, which is the injection shape the allowlist exists to refuse.
+refuses "this Gitea's EXTERNAL hostname"        "is not the allowed in-cluster git host" \
+        PM_GITEA_URL=https://gitea.home.keithjackson.dev
+refuses "the cluster-local name over https"     "is not the allowed in-cluster git host" \
+        PM_GITEA_URL=https://gitea.gitea.svc.cluster.local:3000
+refuses "the cluster-local name, wrong port"    "is not the allowed in-cluster git host" \
+        PM_GITEA_URL=http://gitea.gitea.svc.cluster.local:8080
 refuses "a malformed owner"                     "PM_GITEA_OWNER is malformed" \
         'PM_GITEA_OWNER=../other-org'
 refuses "a malformed repository"                "PM_GITEA_REPO is malformed" \

infra/ansible/inventory/group_vars/k3s_prod/vars.yml

@@ -327,6 +327,29 @@ node_baseline_hosts_pins:
   - ip: "192.168.2.20"
     names:
       - "keycloak.{{ k3s_base_domain }}"
+      # gitea: the SECOND node process to need this pin, for the identical reason, found
+      # 2026-10-06. containerd pulls images as a NODE process, so it resolves through the
+      # public upstreams above and gets the CNAME to wan.home.keithjackson.dev — the WAN IP,
+      # which this router does not hairpin. The pull then fails on the router's own
+      # placeholder certificate rather than on anything to do with the image:
+      #
+      #   failed to resolve reference "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:…":
+      #   tls: failed to verify certificate: x509: certificate is not valid for any names,
+      #   but wanted to match gitea.home.keithjackson.dev
+      #
+      # Measured from the workstation the same day, against the WAN address with that SNI:
+      #   subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
+      #   No extensions in certificate          <- no SANs at all, hence "not valid for ANY names"
+      # while the same name via PiHole answers 192.168.2.20 and serves the real
+      # Let's Encrypt *.home.keithjackson.dev. So the certificate and the Gateway are fine;
+      # only the node's ANSWER is wrong. Same root cause, same fix, as the issuer pin above.
+      #
+      # WHO NEEDS IT: any workload whose image lives in the in-cluster Gitea registry —
+      # today platform/pm-agent (agent-runner) and conformance/incluster
+      # (conformance-runner), neither of which has ever been pulled. A pull is still
+      # UNAUTHENTICATED after this pin, which is a separate thing to settle; this only makes
+      # the node reach the right endpoint.
+      - "gitea.{{ k3s_base_domain }}"
 
 # --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
 # PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/

infra/devcontainer/Containerfile

@@ -32,9 +32,23 @@ ARG TARGETARCH=amd64
 # Distro packages: libvirt client libs (harmless disk-image utilities), ansible, ssh,
 # disk-image + cloud-init seed tooling, python3-pip (for the pip-installed test tools),
 # diffutils (provides `diff`, used by L1 golden render checks), and common utilities.
+#
+# wireguard-tools + qrencode serve the REMOTE-ACCESS VPN (platform/vpn-ingress):
+#   * wireguard-tools gives `wg genkey` / `wg pubkey`, needed to mint the server key and to derive
+#     every client profile's `[Peer] PublicKey` (infra/vpn-ingress/new-peer.sh).
+#   * qrencode renders a client profile as a QR code, which is how a PHONE gets enrolled.
+# They are here rather than on the operator's host because this image is THE toolchain
+# (CLAUDE.md §5; README "Don't install the toolchain on the host"). That rule was briefly broken
+# by a runbook that told the operator to run `wg` on the host and `sops` in here, and then split
+# every step across the boundary to work around it — a procedure nobody should have to follow, and
+# one that cannot be reproduced in CI. Both tools in one image means `sops -d … | wg pubkey` is
+# just a pipeline again.
+# Unpinned on purpose, like every other dnf package here: the Fedora base is what is pinned
+# (FEDORA_VERSION), and these are not gate tools whose version changes what green means.
 RUN dnf -y install --setopt=install_weak_deps=False \
       libvirt-client qemu-img cloud-utils genisoimage openssh-clients \
       ansible-core python3-libvirt python3-pip diffutils \
+      wireguard-tools qrencode \
       git make jq unzip tar gzip xz which ca-certificates curl \
  && dnf clean all
 

infra/devcontainer/README.md

@@ -14,6 +14,7 @@ baseline, and the *same* binaries CI uses.
 | Provisioning | OpenTofu (`bpg/proxmox` provider), `ansible-core`, `openssh-clients` |
 | GitOps / cluster | `flux`, `kubectl`, `helm` |
 | Secrets | `sops`, `age` + `age-keygen` |
+| VPN (`platform/vpn-ingress`) | `wg` (wireguard-tools) — mint the server key, derive client public keys; `qrencode` — render a client profile as a phone-scannable QR |
 | **CI gate** | `yamllint`, `check-jsonschema`, `kyverno` CLI, `kyverno-json`, `chainsaw`, `yq`, `diffutils`, `make` |
 | Misc | `qemu-img`, `cloud-utils`/`genisoimage`, `libvirt-client` (disk-image utilities only — leftovers from the libvirt era, harmless) |
 

infra/vpn-ingress/new-peer.sh

@@ -2,10 +2,24 @@
 # =================================================================================================
 # new-peer.sh — enrol one device on the remote-access VPN (platform/vpn-ingress).
 #
-# Generates a WireGuard keypair ON THIS MACHINE, renders the device's client profile, optionally
-# prints a QR code for a phone, and prints the one `[Peer]` stanza to paste into
-# platform/vpn-ingress/peers.yaml. The private half NEVER leaves this machine: it is not sent to the
-# cluster, not committed, and not written to any log.
+# Generates a WireGuard keypair, renders the device's client profile, prints a QR code for a phone,
+# and prints the one `[Peer]` stanza to paste into platform/vpn-ingress/peers.yaml. The device's
+# private half is written to ONE 0600 file and nowhere else: not sent to the cluster, not committed,
+# never logged.
+#
+# RUN IT IN THE DEV CONTAINER — that is where the toolchain lives (CLAUDE.md §5):
+#
+#     ./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh pixel-8 2'
+#
+# It needs `wg` (wireguard-tools), `qrencode` and `sops`, all three baked into the image. It will
+# run on a host that happens to have them too, but the container is the supported way and the only
+# one that is reproducible.
+#
+# THE SERVER'S PUBLIC KEY IS DERIVED FROM THE ENCRYPTED FILE, not from a copy you keep lying around:
+# `sops -d` the private half out of platform/vpn-ingress/wg-server-key.sops.yaml and pipe it
+# straight into `wg pubkey`. The plaintext never touches disk and never reaches a terminal. (This
+# is why both tools being in ONE image matters — see the Containerfile comment.) Override with
+# WG_SERVER_PUBKEY or WG_SERVER_KEY_FILE if you are working off-tree.
 #
 # WHY A SCRIPT AND NOT A RUNBOOK STEP. Three of these fields are easy to get wrong in ways that are
 # either silently insecure or silently broken, and all three are mechanical:
@@ -25,17 +39,23 @@
 #   <last-octet>    2-254, the device's address inside WG_CLIENT_CIDR. Must be unused.
 #   --split         split tunnel: route ONLY the LAN and the tunnel subnet. Default is a FULL
 #                   tunnel (0.0.0.0/0) so the device also egresses the house WAN.
-#   --out DIR       where to write the profile (default: a 0700 dir under $TMPDIR).
+#   --out DIR       where to write the profile. Default: `.cache/wg-peers/` in the repo, which is
+#                   GITIGNORED (`.gitignore` `.cache/`) and, unlike /tmp inside the container,
+#                   survives the container exiting — the repo is its only persistent read-write
+#                   mount, so a profile written anywhere else is gone before you can move it to a
+#                   desktop. The script refuses any destination git does not ignore.
 #
 # ENVIRONMENT
-#   WG_SERVER_KEY_FILE  server PRIVATE key (default ~/.config/wireguard-home/server.key); its public
-#                       half is derived from it. Alternatively set WG_SERVER_PUBKEY directly.
+#   WG_SERVER_PUBKEY    the server's public key, if you want to skip the SOPS decrypt entirely.
+#   WG_SERVER_KEY_FILE  a local copy of the server PRIVATE key to derive the public half from,
+#                       instead of reading the SOPS file.
+#   WG_SERVER_SOPS_FILE default platform/vpn-ingress/wg-server-key.sops.yaml
 #   WG_ENDPOINT_HOST    default wan.home.keithjackson.dev
 #   WG_CLIENT_CIDR      default 10.90.0.0/24   (must match clusters/base/cluster-vars.yaml)
 #   WG_LAN_CIDR         default 192.168.2.0/24 (prod LAN_CIDR)
 #   WG_DNS              default 192.168.2.23   (prod PIHOLE_DNS_VIP)
 #
-# REQUIRES: wireguard-tools (`wg`). `qrencode` is optional and only used for the QR code.
+# REQUIRES: `wg`, `qrencode`, `sops` — all three in the dev container.
 # =================================================================================================
 set -euo pipefail
 
@@ -51,7 +71,7 @@ PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.yaml"
 : "${WG_CLIENT_CIDR:=10.90.0.0/24}"                 # clusters/base/cluster-vars.yaml
 : "${WG_LAN_CIDR:=192.168.2.0/24}"                  # clusters/prod LAN_CIDR
 : "${WG_DNS:=192.168.2.23}"                         # clusters/prod PIHOLE_DNS_VIP
-: "${WG_SERVER_KEY_FILE:=${HOME}/.config/wireguard-home/server.key}"
+: "${WG_SERVER_SOPS_FILE:=${REPO_ROOT}/platform/vpn-ingress/wg-server-key.sops.yaml}"
 # wan.<base-domain> is an UNPROXIED A record that platform/externaldns/ddns.yaml keeps pointed at
 # the current WAN IP (checked every 5 min). Use it, never a literal address: this house has a
 # DYNAMIC WAN IP, and a profile pinned to today's address stops working without warning and with no
@@ -83,14 +103,28 @@ printf '%s' "$NAME" | grep -Eq '^[a-z0-9][a-z0-9-]{0,30}$' \
 printf '%s' "$OCTET" | grep -Eq '^[0-9]+$' || die "octet must be a number: got '$OCTET'"
 [ "$OCTET" -ge 2 ] && [ "$OCTET" -le 254 ] || die "octet must be 2-254 (.1 is the server): got '$OCTET'"
 
-command -v wg >/dev/null || die "wireguard-tools not installed ('wg' not on PATH). Debian/Ubuntu: apt install wireguard-tools"
+command -v wg >/dev/null || die "'wg' not on PATH. Run this inside the dev container:
+  ./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh $NAME $OCTET'"
 
 # --- the server's public key --------------------------------------------------------------------
+# Order: an explicit pubkey, then a local private-key file, then the SOPS file in the tree. The
+# last is the normal path and needs no local copy of anything.
 if [ -z "${WG_SERVER_PUBKEY:-}" ]; then
-  [ -r "$WG_SERVER_KEY_FILE" ] || die "cannot read server key '$WG_SERVER_KEY_FILE'.
-  Either point WG_SERVER_KEY_FILE at it, or set WG_SERVER_PUBKEY to the public half.
-  If the server key does not exist yet, see runbooks/vpn-remote-access.md step 1."
-  WG_SERVER_PUBKEY="$(tr -d ' \r\n' < "$WG_SERVER_KEY_FILE" | wg pubkey)"
+  if [ -n "${WG_SERVER_KEY_FILE:-}" ]; then
+    [ -r "$WG_SERVER_KEY_FILE" ] || die "cannot read WG_SERVER_KEY_FILE '$WG_SERVER_KEY_FILE'"
+    WG_SERVER_PUBKEY="$(tr -d ' \r\n' < "$WG_SERVER_KEY_FILE" | wg pubkey)"
+  else
+    command -v sops >/dev/null || die "'sops' not on PATH. Run this inside the dev container (see --help)."
+    [ -r "$WG_SERVER_SOPS_FILE" ] || die "cannot read '$WG_SERVER_SOPS_FILE'"
+    # The decrypted private key goes straight into `wg pubkey` — never to a file, never to a tty.
+    # `wg pubkey` fails on the committed REPLACE-ME placeholder, which is the useful error here.
+    WG_SERVER_PUBKEY="$(sops -d --extract '["stringData"]["privatekey"]' "$WG_SERVER_SOPS_FILE" \
+                          | tr -d ' \r\n' | wg pubkey 2>/dev/null)" || WG_SERVER_PUBKEY=""
+    [ -n "$WG_SERVER_PUBKEY" ] || die "could not derive the server public key from
+  $WG_SERVER_SOPS_FILE
+  Either it still holds the REPLACE-ME placeholder (see runbooks/vpn-remote-access.md step 2),
+  or this shell cannot decrypt it (the age keyring is mounted at ~/.config/sops in the container)."
+  fi
 fi
 
 CLIENT_BASE="${WG_CLIENT_CIDR%/*}"; CLIENT_BASE="${CLIENT_BASE%.*}"
@@ -99,7 +133,13 @@ CLIENT_ADDR="${CLIENT_BASE}.${OCTET}"
 # --- refuse a collision, because the failure is otherwise invisible -----------------------------
 # Two devices sharing an address does not error anywhere: the server routes that address to
 # whichever peer handshook most recently, so BOTH devices appear to work intermittently.
-if [ -r "$PEERS_FILE" ] && grep -q "AllowedIPs *= *${CLIENT_ADDR}/32" "$PEERS_FILE"; then
+# COMMENT LINES ARE NOT ALLOCATIONS. This used to grep the whole file and so matched the
+# COMMENTED-OUT EXAMPLE STANZA in peers.yaml, which made `.2` — the first address anyone would
+# ask for — permanently un-allocatable with a message claiming it was already taken. Strip
+# comments first: a real stanza's `AllowedIPs` line is never commented, while both the template
+# and every peer's `# <name> — <owner>` label are.
+if [ -r "$PEERS_FILE" ] \
+   && grep -vE '^[[:space:]]*#' "$PEERS_FILE" | grep -q "AllowedIPs *= *${CLIENT_ADDR}/32"; then
   die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet."
 fi
 if [ -r "$PEERS_FILE" ] && grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
@@ -107,14 +147,22 @@ if [ -r "$PEERS_FILE" ] && grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
 fi
 
 # --- output location: 0700, and never inside the repo -------------------------------------------
-if [ -z "$OUT_DIR" ]; then
-  OUT_DIR="$(mktemp -d "${TMPDIR:-/tmp}/wg-peer-XXXXXX")"
-else
-  mkdir -p "$OUT_DIR"
-fi
+[ -n "$OUT_DIR" ] || OUT_DIR="${REPO_ROOT}/.cache/wg-peers"
+mkdir -p "$OUT_DIR"
 chmod 700 "$OUT_DIR"
-case "$(cd "$OUT_DIR" && pwd)" in
-  "$REPO_ROOT"|"$REPO_ROOT"/*) die "refusing to write a private key inside the git checkout ($OUT_DIR)" ;;
+OUT_ABS="$(cd "$OUT_DIR" && pwd)"
+# A profile holds a private key, so the destination must not be committable. Inside the repo that
+# means git must IGNORE it — test the real property rather than a proxy, because the repo is the
+# container's only persistent read-write mount and so the only sane place to write. Outside the
+# repo, anywhere goes.
+case "$OUT_ABS" in
+  "$REPO_ROOT"|"$REPO_ROOT"/*)
+    if ! git -C "$REPO_ROOT" check-ignore -q "$OUT_ABS/probe.conf" 2>/dev/null; then
+      die "refusing to write a private key to '$OUT_ABS': it is inside the git checkout and NOT
+  gitignored, so the profile would be committable. Use the default (.cache/wg-peers) or add an
+  ignore rule first. Verify with: git check-ignore -v <path>"
+    fi
+    ;;
 esac
 PROFILE="${OUT_DIR}/${NAME}.conf"
 
@@ -190,6 +238,8 @@ if command -v qrencode >/dev/null; then
   printf '  QR code (this IS the private key — your terminal scrollback now holds a credential):\n\n'
   qrencode -t ansiutf8 < "$PROFILE"
 else
-  printf '  qrencode not installed, so no QR code was printed (Debian/Ubuntu: apt install qrencode).\n'
-  printf '  Transfer %s to the device by another route instead.\n' "$PROFILE"
+  # Should not happen in the dev container: qrencode is baked into the image. Kept as a real
+  # fallback for a host run, rather than failing after the profile has already been written.
+  printf '  qrencode not on PATH, so no QR code was printed. It is in the dev container —\n'
+  printf '  rerun there, or transfer %s to the device by another route.\n' "$PROFILE"
 fi

platform/pm-agent/README.md

@@ -22,6 +22,25 @@ requires.
 
 ---
 
+## Why the agent authenticates to Gitea with a local account and a PAT
+
+Not Keycloak, and that is a decision rather than an omission (operator, 2026-10-06): the
+agents only ever talk to Gitea, so a local Gitea account with a scoped personal access token
+is the whole requirement, and it is drastically simpler than federating a machine identity.
+
+It is also what Gitea can actually do. Keycloak service accounts were considered and are not
+available for this: Gitea's API accepts basic auth, PATs, SSH signatures and **its own**
+OAuth2 bearer tokens, and as an OAuth2 provider Gitea implements **only the Authorization
+Code Grant** — there is no client-credentials grant, and nothing in Gitea validates a token
+issued by a third-party OIDC provider. So a Keycloak service-account token cannot
+authenticate to Gitea's API or to git-over-HTTP at all. Keycloak remains what it is for human
+access: the browser login source registered on Gitea (`catalog/gitea.yaml`).
+
+What still matters, and is enforced by the hand-off rather than by Keycloak: **the account
+must not be an administrator** — `infra/gitea/approval-policy.json` authorises
+`/approve-override` on live repository permission alone, so an admin-permission bot could
+waive the sensitive-path human review on its own pull request.
+
 ## It does nothing at all until an operator does two things
 
 Both are deliberate, and both fail **visibly** rather than silently:
@@ -31,6 +50,36 @@ Both are deliberate, and both fail **visibly** rather than silently:
    `infra/agent/build-image.sh` (which verifies it and refuses to push), push it as the
    operator, then update **both** places with the digest `podman push --digestfile` returned,
    in one pull request. Until then: `ImagePullBackOff`.
+
+   **Two things have to be true before any digest will pull, and neither is declared in this
+   repository.** Both were measured after PR #57 merged — this service is the first workload
+   in the cluster ever to reference the in-cluster registry, so nothing had exercised the
+   path. Details and the exact commands:
+   [`decisions/2026-10-05-pm-agent-registry-pull-prereq.md`](../../decisions/2026-10-05-pm-agent-registry-pull-prereq.md).
+
+   - **The nodes resolve the registry hostname to the WAN, so the TLS they fail to verify is
+     the router's — DIAGNOSED 2026-10-06, and fixed in this change.** containerd pulls as a
+     *node* process, so it resolves through the node's public upstreams
+     (`node_baseline_dns_servers` = the LAN gateway) rather than through Pi-hole. The public
+     answer for every name in this zone is a CNAME to `wan.home.keithjackson.dev` — the WAN
+     IP — which this router does not hairpin; it answers :443 with its own placeholder
+     certificate `CN=%D` carrying **no extensions, therefore no SANs**, which is precisely
+     `x509: certificate is not valid for any names`. The Gateway, the wildcard certificate
+     and the digest were never implicated. The fix is the pin this repository already uses
+     for the Keycloak issuer, for the identical reason: `gitea.{{ k3s_base_domain }}` is now
+     in `node_baseline_hosts_pins` — node configuration is Ansible's, declared in the
+     inventory, never typed on a node. Apply it scoped to that one task:
+     `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod
+     --tags hosts`.
+   - **The pull is still unauthenticated** once the node reaches the right endpoint. An
+     unauthenticated client gets nothing from this Gitea's registry, and there is no
+     `imagePullSecret` anywhere in the tree. Consistent with the agent credential decision
+     (local Gitea accounts with PATs), the fix is a `dockerconfigjson` for a local Gitea
+     account whose token carries `read:package`, referenced from the Deployment and the
+     committed worker Job template — not built yet, because minting that token is yours.
+
+   The same two apply to `conformance-runner`: same registry path, same push-then-pin model,
+   never pulled either.
 2. **The credentials are placeholders.** `pm-agent-credentials.sops.yaml` is encrypted to the
    repository's age recipient but holds `REPLACE-ME-…` strings. The hand-off —
    which Gitea account to create, which scopes, and why it must *not* be an administrator —
@@ -96,7 +145,7 @@ Job from a committed template. **The model's output is data, never a command.**
 | **cannot** | merge anything — the gate is the approver, and the service makes no merge call |
 | **cannot** | reach the Kubernetes API: no ServiceAccount token is mounted, the `pm-agent-worker` identity is bound to nothing, and the egress policy does not include the API server. `kubectl` is on its `PATH` (the gate's toolchain needs it) and is inert |
 | **cannot** | reach anything except this Gitea and tcp/443 — no package registry, no GitHub |
-| **cannot** | clone from or push to any host but this Gitea: `pm-worker` validates `PM_GITEA_URL` against an allowlist baked into the image |
+| **cannot** | clone from or push to anywhere but `gitea.gitea.svc.cluster.local:3000`: `pm-worker` validates `PM_GITEA_URL` against a one-entry allowlist baked into the image. In-cluster consumers reach a service by its cluster-local name — a different path by design from how a client outside the cluster reaches it — so the external hostname is not an accepted value |
 | **cannot** | authenticate a `git push` **during the model session**: `GIT_ASKPASS` is unset for its duration, so the restriction is structural rather than a request in a prompt |
 | **cannot** | be created in any shape but the committed one — see the admission policy |
 

platform/pm-agent/pm-agent-credentials.sops.yaml.example

@@ -51,6 +51,15 @@
 # (2) MINT THE GITEA TOKEN as that user (log in as `pm-agent`, or use the admin API to
 #     create a token for it): Settings -> Applications -> Generate New Token.
 #
+#     A LOCAL GITEA ACCOUNT WITH A PAT IS THE DECIDED MODEL (operator, 2026-10-06), not
+#     Keycloak. The agents only ever talk to Gitea, so this is the whole requirement and it
+#     is far simpler than federating a machine identity. It is also the only thing that
+#     works: Gitea's API accepts basic auth, PATs, SSH signatures and its OWN OAuth2 bearer
+#     tokens, and as a provider Gitea implements only the Authorization Code Grant — no
+#     client-credentials grant, and nothing that validates a third-party OIDC token. A
+#     Keycloak service-account token therefore cannot authenticate to Gitea's API or to
+#     git-over-HTTP. Do not "upgrade" this to Keycloak without re-reading that.
+#
 #       Scopes: `write:repository` and `write:issue`. Nothing else — in particular NOT
 #       `write:admin`, NOT `write:organization`, NOT `write:user`.
 #

platform/vpn-ingress/README.md

@@ -14,7 +14,7 @@ egress out the house WAN.
 | Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node |
 | Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices |
 | Authorisation | a device's **public key** is in [`peers.yaml`](./peers.yaml), or it cannot connect |
-| The one secret | the server's private key, Secret `wg-server-key`, **created out of band** |
+| The one secret | the server's private key — **SOPS-encrypted in this tree**, `wg-server-key.sops.yaml` |
 | Exposure | one hand-made WAN port-forward on the SOHO router, off-repo |
 | Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) |
 | Procedure / bring-up | [`runbooks/vpn-remote-access.md`](../../runbooks/vpn-remote-access.md) |
@@ -130,11 +130,17 @@ translations win here.
 | half | where | why |
 |---|---|---|
 | peer **public** keys + addresses | [`peers.yaml`](./peers.yaml), **plaintext in git** | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
-| the server's **private** key | Secret `wg-server-key`, **out of band** | the only real secret; an agent must not mint a long-lived private key (CLAUDE.md §6) |
+| the server's **private** key | `wg-server-key.sops.yaml`, **SOPS-encrypted in git** | the only real secret (R-SEC-01, spec docs/07). Ships with a `REPLACE-ME` placeholder: an agent must not mint a long-lived private key (CLAUDE.md §6), so the operator `sops set`s it in |
 
 The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** —
 so the assembled config holding the private key never touches any node's disk and dies with the pod.
 
+Both halves are therefore in git, which is the point: a cluster rebuild restores the server key with
+everything else. An out-of-band `kubectl create secret` — the first shape this component shipped in,
+corrected after review — would have put the one credential every enrolled device depends on *outside*
+the GitOps tree, and losing it means re-enrolling every device, because each client profile pins this
+key's public half.
+
 Encrypting the peer register would have hidden exactly the change a reviewer most needs to see: a
 new line granting a new device access to the whole LAN.
 
@@ -167,14 +173,32 @@ NetworkPolicy applies.
   (`latest handshake` under ~3 min means live). Never dump `wg showconf` — it prints the private key.
 * **Full bring-up and verification:** `runbooks/vpn-remote-access.md`.
 
+## Measured on prod, 2026-10-06 — what is now known rather than assumed
+
+The first real reconcile settled four things this component's design rested on. Three went the
+right way; one was a defect in my own manifest.
+
+| question | answer |
+|---|---|
+| Does the **kernel module** reach the pod? | **Yes.** `ip link add dev wg0 type wireguard` succeeds, and `/proc/modules` lists `wireguard`, after the `node_baseline` run. `CAP_SYS_MODULE` is correctly not needed. |
+| Does `allowPrivilegeEscalation: false` break the image's **s6 init**? | **No.** The container reaches `wg-quick` normally. The flag stays. |
+| Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
+| Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
+
 ## Prerequisites outside this directory
 
 1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
    (`infra/ansible/roles/node_baseline/defaults/main.yml`); the operator runs the role. The pod has
    `NET_ADMIN` but deliberately **not** `SYS_MODULE`, so it cannot load the module itself — it
    CrashLoops with a clear `wg-quick` error instead of quietly degrading to userspace WireGuard.
-2. **The `wg-server-key` Secret.** Runbook step 1. Until it exists the DaemonSet cannot start —
-   the correct fail-closed state for an unenrolled VPN.
+2. **The server private key.** `wg-server-key.sops.yaml` is committed, encrypted, with a
+   `REPLACE-ME` placeholder, and both overlays already carry its decryption patch — so the wiring
+   is complete and the only outstanding act is replacing the value (runbook step 2). Until then the
+   DaemonSet CrashLoops with an explicit FATAL naming the file: the init container validates the
+   value with `wg pubkey`, which rejects the placeholder, an empty value, a public key pasted by
+   mistake, **and** a Secret that arrived as undecrypted `ENC[...]` ciphertext. That last case is
+   the one worth having a guard for — a missing overlay decryption patch applies cleanly and leaves
+   every Flux condition green.
 3. **The WAN port-forward**, udp/51820 → one node address. Hand-made on the SOHO router, off-repo,
    and the half CI can never check.
 

platform/vpn-ingress/daemonset.yaml

@@ -37,6 +37,10 @@
 #     /etc/sysctl.d/99-k3s-cilium.conf), so no pod sysctl is needed — and `net.ipv4.ip_forward`
 #     is NOT in the kubelet's safe-sysctl allowlist, so a pod-netns version of this would have
 #     required `--allowed-unsafe-sysctls` on all three kubelets (a node-restart change).
+#     COROLLARY LEARNED THE HARD WAY (prod, 2026-10-06): this pod must not merely avoid NEEDING
+#     the sysctl, it must not ATTEMPT it. hostNetwork implies /proc/sys is mounted read-only, so
+#     `sysctl -w` fails with EPERM regardless of capabilities, and wg-quick treats a failed PostUp
+#     as fatal and deletes the interface. Verify the value; never write it.
 #   * no LoadBalancer VIP, so no CiliumL2AnnouncementPolicy lease lottery (the defect that cost
 #     this LAN an 8-day Minecraft outage) and no `externalTrafficPolicy` trade-off at all.
 #   * no double encapsulation. WireGuard's UDP lands on the node's own NIC, so the underlay is a
@@ -136,9 +140,51 @@ spec:
               # The key arrives as a Secret file; strip any trailing newline the operator's shell
               # added, or wg-quick rejects the config with "Key is not the correct length or format".
               key=$(tr -d ' \r\n' < /run/wg-server-key/privatekey)
-              if [ -z "$key" ]; then
-                echo "FATAL: /run/wg-server-key/privatekey is empty." >&2
-                echo "       See runbooks/vpn-remote-access.md step 1 (create the server key)." >&2
+              # VALIDATE, don't just check for empty. The Secret ships committed with a REPLACE-ME
+              # placeholder so the GitOps wiring is complete before the key exists, so the common
+              # failure is a value that is present and WRONG — a placeholder, a public key pasted by
+              # mistake, or a SOPS document that reached the cluster undecrypted (ENC[...], which
+              # happens if an overlay is missing its `vpn-ingress` decryption patch). `wg pubkey`
+              # accepts a value only if it is a real 32-byte curve25519 private key, so it rejects
+              # all four. Without this the bad value would be written into wg0.conf and the pod would
+              # fail later and less legibly, or come up with a server nobody can handshake with.
+              if ! printf '%s' "$key" | wg pubkey >/dev/null 2>&1; then
+                echo "FATAL: the server private key in Secret 'wg-server-key' is not a valid" >&2
+                echo "       WireGuard private key (wg pubkey rejected it)." >&2
+                if [ -z "$key" ]; then
+                  echo "       It is EMPTY." >&2
+                elif [ "$key" = "REPLACE-ME" ]; then
+                  echo "       It is still the committed REPLACE-ME PLACEHOLDER — the key has not" >&2
+                  echo "       been generated yet. This is the expected state on a fresh install." >&2
+                else
+                  # `case`, NOT a dollar-brace prefix-strip. This manifest is rendered through
+                  # envsubst, which consumes every dollar-brace expansion it sees, so a shell
+                  # parameter expansion here renders to the empty string and silently disables this
+                  # very branch. `case` uses no braces, so there is nothing to escape.
+                  # (ci: flux-postbuild-substitution caught both the original expansion AND the
+                  # first version of this comment, which spelled the hazard out literally.)
+                  case "$key" in
+                    ENC\[*)
+                      echo "       It is still SOPS CIPHERTEXT (starts with ENC[). The Secret" >&2
+                      echo "       reached the cluster undecrypted — check that this environment's" >&2
+                      echo "       overlay carries the 'vpn-ingress' decryption patch." >&2
+                      ;;
+                  esac
+                fi
+                echo "       Fix: runbooks/vpn-remote-access.md step 2." >&2
+                exit 1
+              fi
+
+              # The pod shares the HOST netns, so this reads the node's real setting. Reading
+              # /proc/sys is permitted even though writing is not (see the PostUp note below).
+              # Fail closed: a tunnel that comes up but cannot forward would look healthy to every
+              # probe here and simply drop every client's traffic.
+              if ! grep -qx 1 /proc/sys/net/ipv4/ip_forward; then
+                echo "FATAL: net.ipv4.ip_forward is 0 on this node, so forwarded VPN traffic would" >&2
+                echo "       be dropped. This pod CANNOT set it: with hostNetwork the runtime mounts" >&2
+                echo "       /proc/sys read-only, because it is the host's global setting." >&2
+                echo "       Fix on the NODE: run the node_baseline role (it writes" >&2
+                echo "       /etc/sysctl.d/99-k3s-cilium.conf) — runbooks/vpn-remote-access.md step 1." >&2
                 exit 1
               fi
 
@@ -155,9 +201,20 @@ spec:
                 echo 'MTU = 1420'
                 printf 'PrivateKey = %s\n' "$key"
                 # ── Forwarding + SNAT, applied in the HOST netns when wg0 comes up ──
-                # ip_forward is already 1 from node_baseline; re-asserted so this component does not
-                # silently depend on another role's sysctl file surviving.
-                echo 'PostUp = sysctl -q -w net.ipv4.ip_forward=1'
+                # NO `sysctl -w net.ipv4.ip_forward=1` HERE. It is IMPOSSIBLE from this pod and it was
+                # the only thing stopping the tunnel coming up — MEASURED on prod 2026-10-06:
+                #     [#] sysctl -q -w net.ipv4.ip_forward=1
+                #     sysctl: permission denied on key "net.ipv4.ip_forward"
+                #     [#] ip link delete dev wg0
+                # and wg-quick then tore the whole tunnel down. The cause is not a missing
+                # capability: with `hostNetwork: true` the runtime mounts /proc/sys READ-ONLY in the
+                # container (confirmed in /proc/mounts: `proc /proc/sys proc ro,...`), precisely
+                # because those sysctls are the HOST's globals and a container must not rewrite
+                # them. CAP_NET_ADMIN does not and cannot lift a read-only mount.
+                # It was never needed anyway: node_baseline writes net.ipv4.ip_forward=1 to
+                # /etc/sysctl.d/99-k3s-cilium.conf, and reading it from this pod returns 1. The
+                # init container now VERIFIES that (a read, which /proc/sys does allow) and fails
+                # closed with a pointed message, which is what the write was pretending to achieve.
                 # SNAT client traffic to the node's address. REQUIRED: without it a LAN host would
                 # reply to 10.90.0.x, find no route, hand the packet to the router, and the router
                 # would drop it — so every connection would hang rather than fail. Matching on
@@ -217,8 +274,10 @@ spec:
             # `kubectl get pod -o wide` shows what this listener owns.
             - { name: wireguard, containerPort: 51820, hostPort: 51820, protocol: UDP }
           securityContext:
-            # NET_ADMIN is the whole privilege budget: create the wg0 link, set the host sysctl, and
-            # write the two netfilter rules above. Everything else is dropped, and the container is
+            # NET_ADMIN is the whole privilege budget: create the wg0 link and write the netfilter
+            # rules above. (It does NOT let the pod set a host sysctl — /proc/sys is read-only under
+            # hostNetwork — which is why nothing here tries.) Everything else is dropped, and the
+            # container is
             # NOT privileged — which also keeps it clear of restrict-host-namespaces' second rule.
             # SYS_MODULE is deliberately NOT granted: the `wireguard` module is loaded by the node
             # (infra/ansible/roles/node_baseline, node_baseline_kernel_modules), so this container
@@ -232,6 +291,16 @@ spec:
                 - NET_ADMIN
             # readOnlyRootFilesystem intentionally UNSET: the image's s6 init writes under /run and
             # /var during startup. Same deferral as crafty/headlamp.
+            #
+            # EXPECTED, HARMLESS STARTUP NOISE — do not "fix" it. Because capabilities are dropped
+            # to NET_ADMIN only, the image's s6 init cannot chown /config (no CAP_CHOWN) and logs a
+            # run of `chown: ... Operation not permitted` plus "Permissions could not be set. This
+            # is probably because your volume mounts are remote or read-only" and a scary "we will
+            # not provide support for it". It is wrong about the cause and it does not matter:
+            # /config is a writable tmpfs emptyDir, everything here runs as root (uid 0) anyway, and
+            # wg-quick proceeds normally — verified on prod 2026-10-06, where the tunnel's only
+            # failure was the sysctl above, well past these warnings. Granting CAP_CHOWN to silence
+            # log noise would widen the privilege set of an internet-facing pod for nothing.
           resources:
             requests:
               cpu: 25m
@@ -272,14 +341,22 @@ spec:
             # during startup rather than at a point anyone would connect to the volume size. Still
             # counted against the container's 128Mi limit, and still far more than wg0.conf needs.
             sizeLimit: 8Mi
-        # The ONLY secret in this component. Created OUT OF BAND by the operator and referenced by
-        # name (R-IDP-04) — runbooks/vpn-remote-access.md step 1. It is NOT a *.sops.yaml in this
-        # tree on purpose: an agent must not mint a long-lived private key (CLAUDE.md §6), and a key
-        # that passed through a session transcript would have to be treated as compromised.
-        # DR: Velero's `daily` schedule excludes only `monitoring` and `unifi`, so this namespace
-        # and this Secret ARE captured off-cluster (platform/velero/helmrelease.yaml). Keep a copy
-        # in the operator's password manager as well — restoring it is what makes every already
-        # enrolled client work again without re-enrolling.
+        # The ONLY secret in this component. SOPS-encrypted IN THIS TREE (R-SEC-01, spec docs/07) as
+        # wg-server-key.sops.yaml, delivered by Flux, referenced here by name only (R-IDP-04).
+        #
+        # It ships committed with a REPLACE-ME placeholder: an agent must not mint a long-lived
+        # private key (CLAUDE.md §6), so the operator generates it and `sops set`s it in
+        # (runbooks/vpn-remote-access.md step 2). Shipping the encrypted placeholder rather than no
+        # file at all is what lets the resource line and both overlay decryption patches be present
+        # and GREEN from the first commit — the alternative, a commented-out resource to uncomment
+        # later, is exactly the "file in git with no generator entry: reviewed, merged, apparently
+        # done, still broken" shape this repo keeps getting bitten by.
+        #
+        # DR: in git means a cluster rebuild restores it with everything else — the property an
+        # out-of-band `kubectl create secret` would NOT have had, and losing this key means
+        # re-enrolling every device because each client profile pins its public half. Velero also
+        # captures this namespace (its `daily` schedule excludes only `monitoring` and `unifi`).
+        # Keep a password-manager copy too: that is the one that survives losing cluster AND backups.
         - name: wg-server-key
           secret:
             secretName: wg-server-key

platform/vpn-ingress/kustomization.yaml

@@ -4,9 +4,9 @@
 # same placement as crafty/headlamp/homepage/runbooks, for the same reason: a leaf whose pod can be
 # legitimately unhealthy must never sit in the gating `platform` tier, where one component's
 # unhealth stalls platform-config and every apps wave behind it (DECISIONS-LOG: "Headlamp gating
-# stall"). That rationale is unusually concrete here: until the operator creates the `wg-server-key`
-# Secret (runbooks/vpn-remote-access.md step 1) this DaemonSet CANNOT start, so putting it in the
-# gating tier would have taken the whole platform down on merge.
+# stall"). That rationale is unusually concrete here: until the operator replaces the `REPLACE-ME`
+# placeholder in wg-server-key.sops.yaml (runbooks/vpn-remote-access.md step 2) this DaemonSet
+# CANNOT start, so putting it in the gating tier would have taken the whole platform down on merge.
 #
 # NOT in platform/kustomization.yaml — see clusters/base/vpn-ingress.yaml.
 apiVersion: kustomize.config.k8s.io/v1beta1
@@ -19,12 +19,22 @@ resources:
   - daemonset.yaml
   # Namespace default-deny baseline. NOTE: does NOT constrain the hostNetwork DaemonSet — the file
   # says so at length rather than implying a protection it cannot provide.
-  - networkpolicy.yaml
+  - networkpolicy.yaml  # The server's WireGuard private key — the ONE secret here. SOPS-encrypted (R-SEC-01, spec
+  # docs/07), decrypted per-env by the `vpn-ingress` Kustomization, referenced by name only from
+  # daemonset.yaml (R-IDP-04). Its plaintext `.example` sibling carries the operator hand-off --
+  # deliberately NOT named here: conformance/tests/l0-static/no-plaintext-secrets fails any
+  # kustomization that mentions an unencrypted template, because naming one is how a plaintext
+  # Secret gets applied by accident. (It matches comments too. That is correct, not overzealous.)
+  #
+  # SHIPS WITH A `REPLACE-ME` PLACEHOLDER, already encrypted. An agent must not
+  # mint a long-lived private key (CLAUDE.md §6), and the init container validates the value with
+  # `wg pubkey`, so the unfilled state is a loud CrashLoop naming this file, never a server that
+  # quietly cannot handshake.
+  - wg-server-key.sops.yaml
 #
-# DELIBERATELY ABSENT: a Secret. The server's private key (`wg-server-key`) is created out of band
-# by the operator and referenced BY NAME from daemonset.yaml. There is no `*.sops.yaml` here, so
-# this component needs NO SOPS decryption stanza in clusters/dev or clusters/prod — unlike
-# crafty/headlamp/velero. If that key is ever migrated into SOPS, BOTH halves must land together:
-# the file here, and a `decryption:` patch naming the `vpn-ingress` Kustomization in each overlay.
-# Without the patch, kustomize hands Flux the raw ciphertext and the pod mounts a key full of
-# ENC[...] — which applies cleanly and fails only at handshake time.
+# THIS COMPONENT CARRIES AN ENCRYPTED SECRET, so BOTH overlays must patch the `vpn-ingress`
+# Kustomization with a `decryption:` stanza — clusters/dev and clusters/prod both do, added by the
+# same change as this file. That patch is as load-bearing as the file itself: without it kustomize
+# hands Flux the raw ciphertext, the Secret APPLIES CLEANLY full of ENC[...], and nothing in any
+# Flux condition looks wrong. The init container's `wg pubkey` guard exists partly to turn exactly
+# that into a named error rather than a silently non-working tunnel.

platform/vpn-ingress/peers.yaml

@@ -60,10 +60,16 @@ data:
   #
   # COPY THIS TEMPLATE, uncomment, one block per device:
   #
-  #   # pixel-8 — Keith's phone, enrolled 2026-10-05
+  #   # <device-name> — <owner>, enrolled <YYYY-MM-DD>
   #   [Peer]
   #   PublicKey = <the public key new-peer.sh printed>
-  #   AllowedIPs = 10.90.0.2/32
+  #   AllowedIPs = 10.90.0.<octet>/32
+  #
+  # The placeholders above are deliberately NOT valid values. An earlier version of this template
+  # used a real-looking `10.90.0.2/32`, and new-peer.sh's duplicate-address check — which grepped
+  # the whole file — matched it, so `.2` could never be allocated and the script insisted it was
+  # already taken. The check now ignores comments; keeping the example unmatchable too means a
+  # future grep cannot be fooled either.
   #
   # Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
   # tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
@@ -71,3 +77,7 @@ data:
   peers.conf: |
     # No peers enrolled yet. See the template in this file's comments and
     # ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
+    # keith-phone — <owner>, enrolled 2026-10-06
+    [Peer]
+    PublicKey = pBRSHrRyG2hnsuZ6FvHfEzF8hOVTF5stHUb/7fW2Km8=
+    AllowedIPs = 10.90.0.2/32

platform/vpn-ingress/wg-server-key.sops.yaml

@@ -0,0 +1,34 @@
+# WireGuard SERVER private key for platform/vpn-ingress — SOPS-ENCRYPTED (R-SEC-01, spec docs/07).
+# Keys stay cleartext for reviewable diffs; only `stringData` values are encrypted (.sops.yaml
+# `encrypted_regex`). Plaintext template + operator hand-off: wg-server-key.sops.yaml.example.
+apiVersion: v1
+kind: Secret
+metadata:
+    name: wg-server-key
+    namespace: vpn-ingress
+type: Opaque
+stringData:
+    #ENC[AES256_GCM,data:fUoPkfWvasisdTtwdhjeOVOlmdQkTl6Bp9YxYyji8g6k+ClATds3BnHJWOj2iplaGpT2XaNakq1+D4YyOi7QbFvNJui+e4qRpqcY8hWnazvZIHJYxjxfAfzW,iv:xtKd6oMZQumAQJAsNLHqJB+o0wdQJ+vL1kI1mc9jL7w=,tag:aImqLgbSmhsRI36/t5kpyg==,type:comment]
+    #ENC[AES256_GCM,data:npDsCW6rQleolIpur5gFkX2jSRrIQzBlGaIPs2Schv0YU3q8AanG39z4QqT3ASDmskV5YuH29Bo0ujEAgBf52YIjQCp58PDDETofrVKOJzhHMa7bV+HCxUFE,iv:7E7thyFBleyqZpqJRGApKvIq3ubCFOkshF/mxim3YPc=,tag:joIrhMe2I6kUnx1ouOPdZw==,type:comment]
+    #ENC[AES256_GCM,data:wHUFx09FgP/2+LTyw61EFbKA5UaqfjU7CeGk6Xc5av2y4r4AvQJRefDM5EGlahKecDFEFsf7fSQRFRHtUJqrQTtw+6tcj3ubSCPW4iJvhyJJv0xdb/0puRmcgj+2NfgF,iv:xgDOSDLqMQ1wcI9vrkKqfNMgIDFEusFnnO7Rzq/aOa4=,tag:m1m/fgjkBsG91c+3daqOpA==,type:comment]
+    privatekey: ENC[AES256_GCM,data:ML8CPNm4JwhQHUTUUxZO/I+9X15BCIGqOb1Uw9EmX5RTAsxQ9N97YvkYlbY=,iv:5ZUsFdMMmeGsy4fNbk6qbJkzweMVWXZ5g6E7NciAGUg=,tag:pAHv3ieLjoEyFbqbWXsVgg==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBMays2OXY0VkpRc2FQQmhU
+            Q0owM3N0VkhYR0lYTUdNZWRqeTFIMjNHOVJ3CnJSMnc5NFEyU0IzWTFtUkw2YVFS
+            MU02VFo1TE14YVpZRzNhVjZwRTZwd0UKLS0tIDFaMWF4NUZjZmhRdXRuZFJYNkg3
+            SUUzN1FnMlI4bkkzbVNlMWhUVnJZODAKQ+Iy86CIv/Oj9E9j4E/0y5hDbcctRuqB
+            OR825KRPRaTdF/G2hudE1R1JbM4wGbPLMzv078R4ZXHDPGvDttSEkA==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-10-06T06:10:31Z"
+    mac: ENC[AES256_GCM,data:6wAGAwWL7jI/NEeswBnUiWoRty+bb5DEsGEAPXajKNNYCwtm5yuXBrcOI/q4w6H8FG+nqVAECkdpAOI3S9mjDfc2zlWAzfCJJdDic92RKPXOC0tOH/M1BBSGJ1fVdJGAZZT1yAVxmtYy9WohJpawv3kmsVv+JnD+TIqmZ1CJHXo=,iv:srs4o0olSwmraQQgmdPxb+4h/t6fHHeFKJ2z914kGgY=,tag:jjoIKhU8Tu9i5SJkvkfcCg==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

platform/vpn-ingress/wg-server-key.sops.yaml.example

@@ -0,0 +1,76 @@
+# =============================================================================
+# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
+# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
+#
+# WHAT THIS IS: the WireGuard SERVER's long-term private key for the remote-access VPN
+# (platform/vpn-ingress). It is the ONE secret in the whole component. The init container in
+# daemonset.yaml reads it at `/run/wg-server-key/privatekey` and joins it to the PLAINTEXT peer
+# register (peers.yaml) in a tmpfs, so the assembled wg0.conf never touches a node's disk.
+#
+# WHY IT IS IN GIT, SOPS-ENCRYPTED, AND NOT HAND-CREATED. R-SEC-01 / spec docs/07: secrets live in
+# source, encrypted, and reach the cluster only through Flux. An out-of-band `kubectl create secret`
+# would put the single credential every enrolled device depends on OUTSIDE the GitOps tree, where a
+# cluster rebuild silently loses it — and losing it means RE-ENROLLING EVERY DEVICE, because each
+# client profile pins this key's public half.
+#
+# *** THE REAL FILE IS COMMITTED WITH `privatekey: REPLACE-ME`, ALREADY ENCRYPTED. ***
+# That is deliberate, so the wiring is complete and green before the key exists: the resource is
+# already listed in kustomization.yaml and both overlays already carry the `vpn-ingress` decryption
+# patch. Your job is ONLY to replace the value (step 3 below). Until you do, the DaemonSet
+# CrashLoops with an explicit FATAL naming this file — the init container validates the value with
+# `wg pubkey` before writing wg0.conf, so a placeholder fails CLOSED and LOUDLY rather than
+# producing a server that silently cannot complete a handshake.
+#
+# ---------------------------------------------------------------------------------------------
+# EVERYTHING RUNS IN THE DEV CONTAINER. `wg`, `sops` and `qrencode` are all baked into the image
+# (infra/devcontainer/Containerfile) because that image IS the toolchain — CLAUDE.md §5, and the
+# devcontainer README's "Don't install the toolchain on the host". One shell, one toolchain, so
+# `wg genkey | sops set` is just a pipeline.
+# ---------------------------------------------------------------------------------------------
+#
+# HAND-OFF STEPS (operator — an agent must not mint a long-lived private key, CLAUDE.md §6).
+# Full procedure with verification: runbooks/vpn-remote-access.md step 2.
+#
+#  1. Generate the key STRAIGHT INTO this file. The plaintext never lands on disk and never
+#     reaches a terminal; `sops set` edits the already-encrypted document in place:
+#         ./infra/devcontainer/dev.sh run bash -c '
+#           cd /work
+#           umask 077
+#           sops set platform/vpn-ingress/wg-server-key.sops.yaml \
+#             "[\"stringData\"][\"privatekey\"]" "\"$(wg genkey)\""
+#         '
+#     (Or edit it: `./infra/devcontainer/dev.sh run sops platform/vpn-ingress/wg-server-key.sops.yaml`.)
+#
+#  2. Record BOTH halves in your PASSWORD MANAGER. The encrypted file is in git and Velero captures
+#     this namespace to R2, so this is belt-and-braces — but it is the only copy that survives
+#     losing the cluster AND the backups, and the key cannot be regenerated without invalidating
+#     every client profile.
+#         # public half (safe to display; it is what every client profile carries)
+#         ./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d \
+#           --extract "[\"stringData\"][\"privatekey\"]" \
+#           platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey'
+#
+#  3. Confirm the value is a real key without printing it — `wg pubkey` succeeds only on a valid
+#     32-byte curve25519 private key, which is exactly the check the pod's init container runs:
+#         ./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d \
+#           --extract "[\"stringData\"][\"privatekey\"]" \
+#           platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey >/dev/null \
+#           && echo "valid WireGuard key"'
+#
+#  4. PR it. Nothing else to wire: the resource line and both overlay decryption patches already
+#     exist. Flux rolls the DaemonSet and the pods leave CrashLoop.
+#
+# NOTE infra/vpn-ingress/new-peer.sh reads THIS FILE to derive each client profile's
+# `[Peer] PublicKey`, so there is no separate copy of the key to keep in step with anything.
+# =============================================================================
+apiVersion: v1
+kind: Secret
+metadata:
+  name: wg-server-key
+  namespace: vpn-ingress
+type: Opaque
+stringData:
+  # The server's WireGuard private key: `wg genkey` output — 44 chars of base64 ending in '='.
+  # NOT a passphrase, and NOT the public key. The init container validates it with `wg pubkey`
+  # before writing wg0.conf, so a placeholder or a mangled value fails closed with a clear message.
+  privatekey: REPLACE-ME

runbooks/pm-agent.md

@@ -67,7 +67,9 @@ Work through these in order; the first three are by far the most likely.
 | Symptom in the log | Cause | Fix |
 |---|---|---|
 | `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
-| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) |
+| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first |
+| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
+| `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours |
 | `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` |
 | `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |
 | `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) |

runbooks/vpn-remote-access.md

@@ -34,10 +34,26 @@ deliberately **not** granted `CAP_SYS_MODULE`, so it cannot autoload the module.
 `wg-quick` fails outright and the pod CrashLoops.
 
 ```bash
-# operator action (Ansible is not an agent action)
-cd infra/ansible && ./site.sh --tags system --limit k3s_prod
+# operator action (Ansible is not an agent action). Ansible lives ONLY in the pinned toolbox,
+# and ansible.cfg is only picked up from infra/ansible, so cd there inside the container.
+./infra/devcontainer/dev.sh run bash -c \
+  'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system'
 ```
 
+Three details in that command are load-bearing, not style:
+
+- **`-e k3s_hosts=k3s_prod`.** `node-baseline.yml` targets `k3s_servers` by default, which is the
+  **dev** group — and dev was decommissioned, so an unqualified run does nothing useful. This is
+  the targeting pattern `infra/ansible/README.md` "Targeting prod" documents for every playbook.
+- **`--tags system`, and never an untagged run.** The two kernel-module tasks carry the `system`
+  tag. An untagged run would also execute the `disks` tasks, which assert on block-device variables
+  prod has deliberately left as placeholders (`infra/PROD-PROVISIONING.md` §2) — the playbook's own
+  header warns about this. `--tags system` is broader than the modules alone (it also re-asserts
+  sysctls, iscsid, qemu-guest-agent and the swap masking) but every one of those is idempotent and
+  already in its desired state on a live node.
+- **It is a playbook, not a wrapper script.** There is no `site.sh` in this repo; `playbooks/site.yml`
+  is the full bring-up chain and is the wrong tool here.
+
 Verify on each node — the module is in-tree since kernel 5.6, so nothing is installed, only loaded:
 
 ```bash
@@ -46,46 +62,108 @@ lsmod | grep '^wireguard'
 grep -w wireguard /etc/modules-load.d/k3s-cilium-longhorn.conf
 ```
 
-## Step 2 — create the server key (once)
+## Step 2 — fill in the server key (once)
 
-This is the **only** secret in the component, and the one thing that must never be generated in an
-agent session or pasted into a transcript. Generate it on your own machine:
+The key is **SOPS-encrypted in this tree** (R-SEC-01, spec docs/07), not created by hand:
+`platform/vpn-ingress/wg-server-key.sops.yaml`. It is already committed — encrypted, with a
+`REPLACE-ME` placeholder — and it is already listed in the component's `resources:` with a
+`decryption:` patch in both overlays. **Nothing needs wiring. You are only replacing one value.**
+
+Why in git rather than `kubectl create secret`: an out-of-band Secret puts the one credential every
+enrolled device depends on outside the GitOps tree, where a cluster rebuild silently loses it — and
+losing it means re-enrolling every device, because each client profile pins this key's public half.
+
+Everything below runs **inside the dev container** — `wg`, `sops` and `qrencode` are all baked into
+the image (CLAUDE.md §5: the toolchain lives only in the pinned container; don't install it on the
+host). One shell, one toolchain.
+
+**1. Generate the key straight into the encrypted file.** One pipeline: `wg genkey` makes it,
+`sops set` writes it into the already-encrypted document in place. The plaintext never lands on
+disk and never reaches a terminal.
 
 ```bash
-mkdir -p ~/.config/wireguard-home && chmod 700 ~/.config/wireguard-home
-umask 077
-wg genkey > ~/.config/wireguard-home/server.key          # keep this; see "Back it up" below
+./infra/devcontainer/dev.sh run bash -c '
+  cd /work
+  umask 077
+  sops set platform/vpn-ingress/wg-server-key.sops.yaml \
+    "[\"stringData\"][\"privatekey\"]" "\"$(wg genkey)\""
+'
 ```
 
-Load it into the cluster. **Do not echo it, do not pass it on an interactive command line that
-lands in shell history, and never print the resulting Secret's `.data`** (CLAUDE.md §6):
+**2. Record it in your PASSWORD MANAGER.** Derive both halves from the file you just wrote:
 
 ```bash
-./infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n vpn-ingress create secret generic wg-server-key \
-     --from-file=privatekey=/work/.cache/server.key'
+# the PUBLIC half — safe to display, and what every client profile carries
+./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
+  platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey'
+
+# the PRIVATE half — paste into the password manager, then clear your scrollback
+./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
+  platform/vpn-ingress/wg-server-key.sops.yaml'
 ```
 
-…where `/work/.cache/server.key` is a path you copied the key to **inside the repo's gitignored
-cache** (or mount it however you prefer — the requirement is only that the bytes never pass through
-a log). Delete that copy afterwards: `shred -u .cache/server.key`.
+The second command is the one place this key is deliberately shown. The encrypted file is in git and
+Velero captures this namespace to R2, so the password-manager copy is belt-and-braces — but it is
+the only copy that survives losing the cluster *and* the backups, and the key cannot be regenerated
+without invalidating every client profile.
 
-Confirm it exists **by key name only**:
+**3. Confirm the value is a real key**, without printing it — if `wg pubkey` prints a key, the
+value is valid; if it errors, it is still wrong:
 
 ```bash
-./infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n vpn-ingress get secret wg-server-key -o jsonpath="{.data}" | tr "," "\n" | cut -d: -f1'
-# expected: {"privatekey   -- a NAME. If you see base64 payload, you printed the key. Rotate it.
+./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
+  platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey >/dev/null && echo "valid WireGuard key"'
 ```
 
-**Why out of band and not SOPS.** An agent must not mint a long-lived private key, and a key that
-passed through a session transcript would have to be treated as compromised. The trade-off is that
-this Secret is **not reconstructible from git** — so:
+That is the same check the pod's init container runs, so a pass here predicts a pass there.
 
-**Back it up.** Keep `server.key` in your password manager. Velero's `daily` schedule excludes only
-`monitoring` and `unifi` (`platform/velero/helmrelease.yaml`), so this namespace *is* captured
-off-cluster to R2 as well. Restoring this one key is what makes every already-enrolled device work
-again after a cluster rebuild — **losing it means re-enrolling every device.**
+**4. PR it, and merge.** Flux rolls the DaemonSet and the pods leave CrashLoop.
+
+If you would rather use an editor than `sops set`:
+`./infra/devcontainer/dev.sh run sops platform/vpn-ingress/wg-server-key.sops.yaml` (replace
+`REPLACE-ME`, save, exit — it re-encrypts on write).
+
+### What the pod tells you if this went wrong
+
+The init container validates the value with `wg pubkey` before writing `wg0.conf`, and names the
+specific failure in its log — so you never get a server that silently cannot complete a handshake:
+
+| pod log says | meaning |
+|---|---|
+| `still the committed REPLACE-ME PLACEHOLDER` | step 3 not done (the expected fresh-install state) |
+| `It is EMPTY` | the value was blanked |
+| `still SOPS CIPHERTEXT (starts with ENC[)` | the Secret reached the cluster undecrypted — this environment's overlay is missing its `vpn-ingress` decryption patch. **This is the dangerous one:** the Secret applies cleanly and every Flux condition stays green |
+| `wg pubkey rejected it` with none of the above | a mangled value, or a *public* key pasted by mistake |
+| `net.ipv4.ip_forward is 0 on this node` | the node_baseline role has not run here (step 1). The pod **cannot** set this itself — under `hostNetwork` the runtime mounts `/proc/sys` read-only, because it is the host's global setting |
+
+### What a healthy start looks like
+
+Measured on prod 2026-10-06. The init container logs one line, naming the shape and never the key:
+
+```
+wg0.conf assembled: 0 peer(s), server 10.90.0.1/24, udp/51820.
+```
+
+then the `wireguard` container brings the tunnel up:
+
+```
+**** Found WG conf /config/wg_confs/wg0.conf, adding to list ****
+**** Activating tunnel /config/wg_confs/wg0.conf ****
+[#] ip link add dev wg0 type wireguard
+[#] wg addconf wg0 /dev/fd/63
+[#] ip -4 address add 10.90.0.1/24 dev wg0
+[#] ip link set mtu 1420 up dev wg0
+```
+
+**Ignore the `chown: ... Operation not permitted` run before that**, and the image's own
+"Permissions could not be set ... we will not provide support for it". Capabilities are dropped to
+`NET_ADMIN` only, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs
+as root, and `wg-quick` proceeds. It is wrong about the cause and it does not matter.
+
+**A pod stuck `0/1 Running` with `Startup probe failed: Unable to access interface: No such device`
+means `wg-quick` tore the tunnel down** — read the `wireguard` container log for the `[#]` line it
+died on. That is the shape of a real failure; the probe is doing its job of refusing to call a
+server without a `wg0` "ready".
 
 ## Step 3 — forward udp/51820 on the router (once)
 
@@ -126,12 +204,19 @@ unauthenticated traffic either way. If the pod is Ready, the interface exists.
 ## Step 5 — enrol a device
 
 ```bash
-./infra/vpn-ingress/new-peer.sh pixel-8 2              # full tunnel  (LAN + WAN egress)
-./infra/vpn-ingress/new-peer.sh thinkpad 3 --split     # split tunnel (LAN only)
+# in the dev container, like everything else
+./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh pixel-8 2'            # full tunnel
+./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh thinkpad 3 --split'   # LAN only
 ```
 
-It generates the keypair **on your machine**, writes a `0600` client profile outside the checkout,
-prints a QR code if `qrencode` is installed, and prints the `[Peer]` stanza to add. Then:
+It derives the server's public key from `wg-server-key.sops.yaml` itself, so there is no local key
+copy to keep in step with anything. Profiles land in `.cache/wg-peers/` — gitignored, and persistent
+because the repo is the container's only read-write mount.
+
+It generates the device's keypair, writes a `0600` client profile, prints a QR code, and prints the
+`[Peer]` stanza to add. It refuses to write a profile anywhere git does not ignore.
+
+Then:
 
 1. Paste the stanza into `platform/vpn-ingress/peers.yaml` under `data.peers.conf`, keeping the
    `# <name> — <owner>, enrolled <date>` comment. `wg show` can only ever print the key, so that