fix(vpn-ingress): keep the server key in source through SOPS, not as an out-of-band Secret

Operator correction: secrets are kept in source through sops. #55 shipped the
WireGuard server private key as a Secret to `kubectl create` by hand. Two real
costs, not just a convention break:

  * R-SEC-01 / spec docs/07 -- secrets live in source, SOPS-encrypted, and
    reach the cluster only through Flux.
  * It broke DR for the one credential that cannot be re-derived. A rebuild
    restores everything in git; it does not restore a hand-made Secret, and
    losing this key means re-enrolling EVERY device, because each client
    profile pins its public half. #55 papered over that with "keep a copy in
    your password manager".

(Note secret-provenance did NOT forbid the original shape -- it scopes to
app-namespace=true and vpn-ingress is deliberately a platform namespace. The
rule broken was the written requirement, not an enforced control.)

An agent still must not mint a long-lived private key (CLAUDE.md §6), so the
file is committed ALREADY ENCRYPTED with `privatekey: REPLACE-ME`. That makes
the wiring complete and green from the first commit -- resource line plus both
overlay decryption patches -- leaving the operator exactly one act: replace the
value. A commented-out resource to uncomment later is the "in git, no generator
entry, reviewed, merged, still broken" shape this repo keeps hitting.

The init guard is now a validation rather than an emptiness check: the value is
piped through `wg pubkey`, which accepts only a real curve25519 private key, and
the pod names which case it hit -- placeholder, empty, mangled, or (the silent
one) still `ENC[` ciphertext because an overlay is missing its decryption patch,
which otherwise applies cleanly with every Flux condition green.

Second, independent finding that also fixes #55's step 2: `wg` is on the HOST
only and `sops` is in the CONTAINER only, so `sops -d | wg pubkey` cannot run in
one shell -- the command #55 documented could not have worked. Every step is now
split by where it runs, and the written key is verified by comparing sha256
digests rather than printing anything. `qrencode` is on neither, so the QR step
says so and gives the install command.

Touches clusters/** and adds a **/*.sops.yaml, so approval-policy will block
pending the operator's /approve-override. Deliberately not posted by me.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

runbooks/vpn-remote-access.md

@@ -62,53 +62,90 @@ lsmod | grep '^wireguard'
 grep -w wireguard /etc/modules-load.d/k3s-cilium-longhorn.conf
 ```
 
-## Step 2 — create the server key (once)
+## Step 2 — fill in the server key (once)
 
-This is the **only** secret in the component, and the one thing that must never be generated in an
-agent session or pasted into a transcript. Generate it on your own machine:
+The key is **SOPS-encrypted in this tree** (R-SEC-01, spec docs/07), not created by hand:
+`platform/vpn-ingress/wg-server-key.sops.yaml`. It is already committed — encrypted, with a
+`REPLACE-ME` placeholder — and it is already listed in the component's `resources:` with a
+`decryption:` patch in both overlays. **Nothing needs wiring. You are only replacing one value.**
+
+Why in git rather than `kubectl create secret`: an out-of-band Secret puts the one credential every
+enrolled device depends on outside the GitOps tree, where a cluster rebuild silently loses it — and
+losing it means re-enrolling every device, because each client profile pins this key's public half.
+
+> **MIND WHERE EACH COMMAND RUNS.** The two tools are not in the same place (measured 2026-10-05):
+> `wg` is on the **HOST** only (`/usr/bin/wg`, absent from the dev container); `sops` is in the
+> **CONTAINER** only (`/usr/local/bin/sops`, absent from the host). You cannot pipe
+> `sops -d … | wg pubkey` in one shell. The steps below are split accordingly and never put the key
+> on a terminal.
+
+**1. HOST — generate it.** Keep it at this path: it is where `new-peer.sh` looks by default to
+derive the public half for each client profile.
 
 ```bash
-mkdir -p ~/.config/wireguard-home && chmod 700 ~/.config/wireguard-home
 umask 077
-wg genkey > ~/.config/wireguard-home/server.key          # keep this; see "Back it up" below
+mkdir -p ~/.config/wireguard-home && chmod 700 ~/.config/wireguard-home
+wg genkey > ~/.config/wireguard-home/server.key
 ```
 
-Load it into the cluster. **Do not echo it, do not pass it on an interactive command line that
-lands in shell history, and never print the resulting Secret's `.data`** (CLAUDE.md §6):
+**2. Store it in your PASSWORD MANAGER**, together with its public half (safe to display):
+
+```bash
+wg pubkey < ~/.config/wireguard-home/server.key
+```
+
+This copy is the one that survives losing the cluster *and* the backups. Velero does capture this
+namespace to R2 (its `daily` schedule excludes only `monitoring` and `unifi`), and the encrypted
+file is in git — but the key itself cannot be regenerated without invalidating every client profile.
+
+**3. Write it into the encrypted file in place.**
+
+```bash
+mkdir -p .cache && cp ~/.config/wireguard-home/server.key .cache/server.key
+git check-ignore -v .cache/server.key    # MUST print `.gitignore:11:*.key` — see the warning below
+
+./infra/devcontainer/dev.sh run bash -c \
+  'cd /work && sops set platform/vpn-ingress/wg-server-key.sops.yaml \
+     "[\"stringData\"][\"privatekey\"]" "\"$(tr -d "\r\n" < /work/.cache/server.key)\""'
+
+shred -u .cache/server.key
+```
+
+`sops set` edits the **encrypted** document directly, so no plaintext copy of it is ever written.
+(`sops platform/vpn-ingress/wg-server-key.sops.yaml` opens an editor instead, if you prefer.)
+
+> **KEEP THE `.key` SUFFIX** on the staged copy. `.gitignore` line 11 is `*.key`, and it is the
+> **only** rule covering that path — `.cache/` itself is NOT ignored
+> (`git check-ignore -v .cache/serverkey.txt` matches nothing). A copy named `server.pem` or
+> `privatekey` in that directory is **committable**, and the gate's `no-plaintext-secrets` stage
+> does not recognise a bare WireGuard key.
+
+**4. Verify you wrote the right bytes, without printing them** — compare digests, not values:
 
 ```bash
 ./infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n vpn-ingress create secret generic wg-server-key \
-     --from-file=privatekey=/work/.cache/server.key'
+  'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
+     platform/vpn-ingress/wg-server-key.sops.yaml | tr -d "\r\n" | sha256sum'
+
+tr -d '\r\n' < ~/.config/wireguard-home/server.key | sha256sum
 ```
 
-…where `/work/.cache/server.key` is a path you copied the key to (`.cache/` is not tracked; create
-it if absent). Delete that copy as soon as the Secret exists: `shred -u .cache/server.key`.
+The two digests must match. This is the only sanctioned decrypt of this file, and its output goes
+straight into `sha256sum` — never onto a terminal (CLAUDE.md §6).
 
-> **KEEP THE `.key` SUFFIX.** That filename is what keeps the key out of git: `.gitignore` line 11
-> is `*.key`, and it is the **only** rule covering this path — `.cache/` itself is NOT ignored
-> (`git check-ignore -v .cache/serverkey.txt` matches nothing). So a copy named `server.pem`,
-> `wg-server` or `privatekey` sitting in that directory is **committable**, and the gate's
-> `no-plaintext-secrets` stage does not know a WireGuard private key when it sees one. Verify before
-> you trust it: `git check-ignore -v .cache/<your-filename>` must print a rule, and
-> `git status --short` must not list the file.
+**5. PR it, and merge.** Flux rolls the DaemonSet and the pods leave CrashLoop.
 
-Confirm it exists **by key name only**:
+### What the pod tells you if this went wrong
 
-```bash
-./infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n vpn-ingress get secret wg-server-key -o jsonpath="{.data}" | tr "," "\n" | cut -d: -f1'
-# expected: {"privatekey   -- a NAME. If you see base64 payload, you printed the key. Rotate it.
-```
+The init container validates the value with `wg pubkey` before writing `wg0.conf`, and names the
+specific failure in its log — so you never get a server that silently cannot complete a handshake:
 
-**Why out of band and not SOPS.** An agent must not mint a long-lived private key, and a key that
-passed through a session transcript would have to be treated as compromised. The trade-off is that
-this Secret is **not reconstructible from git** — so:
-
-**Back it up.** Keep `server.key` in your password manager. Velero's `daily` schedule excludes only
-`monitoring` and `unifi` (`platform/velero/helmrelease.yaml`), so this namespace *is* captured
-off-cluster to R2 as well. Restoring this one key is what makes every already-enrolled device work
-again after a cluster rebuild — **losing it means re-enrolling every device.**
+| pod log says | meaning |
+|---|---|
+| `still the committed REPLACE-ME PLACEHOLDER` | step 3 not done (the expected fresh-install state) |
+| `It is EMPTY` | the value was blanked |
+| `still SOPS CIPHERTEXT (starts with ENC[)` | the Secret reached the cluster undecrypted — this environment's overlay is missing its `vpn-ingress` decryption patch. **This is the dangerous one:** the Secret applies cleanly and every Flux condition stays green |
+| `wg pubkey rejected it` with none of the above | a mangled value, or a *public* key pasted by mistake |
 
 ## Step 3 — forward udp/51820 on the router (once)
 
@@ -153,8 +190,16 @@ unauthenticated traffic either way. If the pod is Ready, the interface exists.
 ./infra/vpn-ingress/new-peer.sh thinkpad 3 --split     # split tunnel (LAN only)
 ```
 
-It generates the keypair **on your machine**, writes a `0600` client profile outside the checkout,
-prints a QR code if `qrencode` is installed, and prints the `[Peer]` stanza to add. Then:
+It generates the keypair **on your machine**, writes a `0600` client profile outside the checkout
+(it refuses to write inside the git checkout), prints a QR code, and prints the `[Peer]` stanza to
+add.
+
+> **`qrencode` is installed neither on this workstation nor in the dev container** (measured
+> 2026-10-05), so the QR path needs it first: `sudo dnf install qrencode` (or `apt install
+> qrencode`). Without it the script says so plainly and leaves you to move the `.conf` to the
+> device by another route — fine for a desktop, awkward for a phone.
+
+Then:
 
 1. Paste the stanza into `platform/vpn-ingress/peers.yaml` under `data.peers.conf`, keeping the
    `# <name> — <owner>, enrolled <date>` comment. `wg show` can only ever print the key, so that