fix(vpn-ingress): keep the server key in source through SOPS, not as an out-of-band Secret

Operator correction: secrets are kept in source through sops. #55 shipped the
WireGuard server private key as a Secret to `kubectl create` by hand. Two real
costs, not just a convention break:

  * R-SEC-01 / spec docs/07 -- secrets live in source, SOPS-encrypted, and
    reach the cluster only through Flux.
  * It broke DR for the one credential that cannot be re-derived. A rebuild
    restores everything in git; it does not restore a hand-made Secret, and
    losing this key means re-enrolling EVERY device, because each client
    profile pins its public half. #55 papered over that with "keep a copy in
    your password manager".

(Note secret-provenance did NOT forbid the original shape -- it scopes to
app-namespace=true and vpn-ingress is deliberately a platform namespace. The
rule broken was the written requirement, not an enforced control.)

An agent still must not mint a long-lived private key (CLAUDE.md §6), so the
file is committed ALREADY ENCRYPTED with `privatekey: REPLACE-ME`. That makes
the wiring complete and green from the first commit -- resource line plus both
overlay decryption patches -- leaving the operator exactly one act: replace the
value. A commented-out resource to uncomment later is the "in git, no generator
entry, reviewed, merged, still broken" shape this repo keeps hitting.

The init guard is now a validation rather than an emptiness check: the value is
piped through `wg pubkey`, which accepts only a real curve25519 private key, and
the pod names which case it hit -- placeholder, empty, mangled, or (the silent
one) still `ENC[` ciphertext because an overlay is missing its decryption patch,
which otherwise applies cleanly with every Flux condition green.

Second, independent finding that also fixes #55's step 2: `wg` is on the HOST
only and `sops` is in the CONTAINER only, so `sops -d | wg pubkey` cannot run in
one shell -- the command #55 documented could not have worked. Every step is now
split by where it runs, and the written key is verified by comparing sha256
digests rather than printing anything. `qrencode` is on neither, so the QR step
says so and gives the install command.

Touches clusters/** and adds a **/*.sops.yaml, so approval-policy will block
pending the operator's /approve-override. Deliberately not posted by me.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

clusters/base/vpn-ingress.yaml

@@ -8,8 +8,8 @@
 #     NOTHING downstream reconciles — the incident behind the headlamp/oauth2-proxy/homepage moves
 #     (DECISIONS-LOG: "Headlamp gating stall").
 #   * SPECIFIC, AND THE REASON THIS IS NOT A JUDGEMENT CALL: this DaemonSet cannot start until the
-#     operator creates the out-of-band `wg-server-key` Secret (runbooks/vpn-remote-access.md
+#     operator replaces the `REPLACE-ME` placeholder in platform/vpn-ingress/wg-server-key.sops.yaml
-#     step 1). That is the correct fail-closed state — an unenrolled VPN must not come up — but in
+#     with a real key (runbooks/vpn-remote-access.md step 2). That is the correct fail-closed state — an unenrolled VPN must not come up — but in
 #     the gating tier it would have been a self-inflicted platform-wide outage ON MERGE, with the
 #     cause (a missing manual step) invisible from any Flux condition.
 #
@@ -51,6 +51,9 @@ spec:
     substituteFrom:
       - kind: ConfigMap
         name: cluster-vars
-  # NO `decryption:` stanza is needed in the overlays: this path carries no *.sops.yaml. The server
+  # SOPS decryption is layered per-environment by the overlay (clusters/dev and clusters/prod each
-  # key is an out-of-band Secret (platform/vpn-ingress/kustomization.yaml says what to do if that
+  # patch THIS Kustomization by name), identical to headlamp/crafty/velero — this path carries
-  # ever changes).
+  # platform/vpn-ingress/wg-server-key.sops.yaml, the WireGuard server private key. The `sops-age`
+  # Secret is created out of band and referenced by NAME only. Without the overlay patch the Secret
+  # applies cleanly full of ENC[...] and only the tunnel is broken, so the patch is as load-bearing
+  # as the encrypted file itself.

clusters/dev/kustomization.yaml

@@ -122,6 +122,24 @@ patches:
           provider: sops
           secretRef:
             name: sops-age
+  # vpn-ingress (the WireGuard remote-access VPN, clusters/base/vpn-ingress.yaml) carries
+  # platform/vpn-ingress/wg-server-key.sops.yaml — the WireGuard SERVER private key, the one
+  # secret in that component. Without this stanza kustomize hands Flux the raw ciphertext and the
+  # Secret applies full of ENC[...]: the APPLY SUCCEEDS and every Flux condition stays green, and
+  # the only symptom is a VPN that no client can handshake with. The DaemonSet's init container
+  # validates the value with `wg pubkey` specifically so that state is a named CrashLoop instead.
+  # The `sops-age` Secret is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: vpn-ingress
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
   # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
   # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
   # account and the Anthropic subscription token from `claude setup-token`. Without this

clusters/prod/kustomization.yaml

@@ -113,6 +113,24 @@ patches:
           provider: sops
           secretRef:
             name: sops-age
+  # vpn-ingress (the WireGuard remote-access VPN, clusters/base/vpn-ingress.yaml) carries
+  # platform/vpn-ingress/wg-server-key.sops.yaml — the WireGuard SERVER private key, the one
+  # secret in that component. Without this stanza kustomize hands Flux the raw ciphertext and the
+  # Secret applies full of ENC[...]: the APPLY SUCCEEDS and every Flux condition stays green, and
+  # the only symptom is a VPN that no client can handshake with. The DaemonSet's init container
+  # validates the value with `wg pubkey` specifically so that state is a named CrashLoop instead.
+  # The `sops-age` Secret is out-of-band (name only).
+  - patch: |-
+      apiVersion: kustomize.toolkit.fluxcd.io/v1
+      kind: Kustomization
+      metadata:
+        name: vpn-ingress
+        namespace: flux-system
+      spec:
+        decryption:
+          provider: sops
+          secretRef:
+            name: sops-age
   # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
   # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
   # account and the Anthropic subscription token from `claude setup-token`. Without this

decisions/2026-10-05-vpn-server-key-in-sops.md

@@ -0,0 +1,146 @@
+---
+date: 2026-10-05
+branch: fix/vpn-server-key-in-sops
+title: VPN server key moves into SOPS-encrypted git — the out-of-band Secret broke R-SEC-01 and the DR story
+---
+
+## What changed
+
+**Operator correction (2026-10-05): "secrets are kept in source through sops".** They are right, and
+#55 shipped the WireGuard server private key the wrong way: as a Secret the operator was told to
+`kubectl create` out of band, referenced by name only. This branch moves it into the tree as
+`platform/vpn-ingress/wg-server-key.sops.yaml`, where every other secret in this repo lives.
+
+**Why the original choice was wrong, not merely unconventional.** Two concrete costs, both of which
+#55's own README half-admitted and then shipped anyway:
+
+1. **R-SEC-01 / spec docs/07 says secrets live in source, SOPS-encrypted, reaching the cluster only
+   through Flux.** An out-of-band Secret is outside the GitOps tree by construction.
+2. **It broke DR for the one credential that cannot be re-derived.** A cluster rebuild restores
+   everything in git; it does not restore a hand-made Secret. Losing this key means **re-enrolling
+   every device**, because each client profile pins its public half. #55 papered over this with
+   "keep a copy in your password manager", which is a workaround for a self-inflicted gap.
+
+Note `platform/kyverno/admission/secret-provenance.yaml` did **not** forbid the original shape —
+it scopes to `app-namespace=true` namespaces and `vpn-ingress` is deliberately a platform namespace,
+so admission would have accepted the hand-applied Secret. The rule it violated was the written
+requirement, not an enforced control. Recording that because "the gate let it through" was never
+evidence it was correct.
+
+**How the unfillable-key problem is solved.** An agent still must not mint a long-lived private key
+(CLAUDE.md §6), so the file is committed **already encrypted, with `privatekey: REPLACE-ME`**. That
+makes the wiring complete and the gate green from the first commit — the resource line and both
+overlay decryption patches are present — and leaves the operator exactly one act: replace the value.
+The alternative, a commented-out resource line to uncomment later, is precisely the "file in git with
+no generator entry: reviewed, merged, apparently done, still broken" shape this repo keeps getting
+bitten by (`alert-runbooks-resolvable` exists because of it).
+
+**The init-container guard is now a validation, not an emptiness check.** It was `[ -z "$key" ]`.
+Because the committed placeholder is non-empty, the likely failure is now a value that is present
+and *wrong*, so the guard pipes the value through `wg pubkey` — which accepts only a real 32-byte
+curve25519 private key — and names which of four cases it hit:
+
+| case | why it matters |
+|---|---|
+| `REPLACE-ME` placeholder | the expected fresh-install state |
+| empty | blanked value |
+| **starts with `ENC[`** | **the Secret reached the cluster UNDECRYPTED.** A missing overlay decryption patch applies cleanly and leaves every Flux condition green — this is the silent failure the guard exists for |
+| rejected, none of the above | mangled value, or a *public* key pasted by mistake |
+
+Files:
+
+- `platform/vpn-ingress/wg-server-key.sops.yaml` (new) — SOPS-encrypted to the repo's age recipient,
+  `stringData.privatekey` only, placeholder value.
+- `platform/vpn-ingress/wg-server-key.sops.yaml.example` (new) — plaintext template + operator
+  hand-off, following the `crafty-oauth2-proxy-credentials.sops.yaml.example` convention.
+- `platform/vpn-ingress/kustomization.yaml` — the resource added; the "DELIBERATELY ABSENT: a
+  Secret" block replaced with the overlay-patch warning.
+- `platform/vpn-ingress/daemonset.yaml` — the `wg pubkey` guard; volume comment rewritten.
+- `clusters/dev/kustomization.yaml`, `clusters/prod/kustomization.yaml` — **the `vpn-ingress`
+  `decryption:` patch, in both.** This is the load-bearing half: #55's own kustomization comment
+  predicted that forgetting it applies ciphertext cleanly.
+- `clusters/base/vpn-ingress.yaml` — its "NO decryption stanza is needed" comment was now false.
+- `platform/vpn-ingress/README.md`, `runbooks/vpn-remote-access.md` — step 2 rewritten.
+- `infra/vpn-ingress/new-peer.sh` — stale "step 1" pointer.
+
+**A second finding, which changes the procedure independently of SOPS: `wg` and `sops` are not in
+the same place.** Measured: `wg` is on the **host** only (`/usr/bin/wg`, absent from the dev
+container); `sops` is in the **container** only (`/usr/local/bin/sops`, absent from the host). So
+`sops -d … | wg pubkey` cannot be run in one shell, and #55's step 2 — which assumed it could —
+would have failed. The runbook and the `.example` now split every command by where it runs, and
+verify the written key by comparing **sha256 digests** of the decrypted value and the operator's
+local copy, which never puts key material on a terminal. `qrencode` is on neither, so the QR
+instruction now says so and gives the install command.
+
+**SENSITIVE PATHS — this PR touches `clusters/**` (two overlays) AND adds a `**/*.sops.yaml`, so
+`ci/approval-policy.sh` will BLOCK it.** It needs the operator's `/approve-override <reason>` as a
+PR **review** of type "Comment". I have deliberately not posted it (CLAUDE.md §1).
+
+## What I verified
+
+- **The encrypted file is real and correctly targeted.** `sops -e -i` in the pinned toolbox
+  produced `stringData.privatekey: ENC[AES256_GCM,...]` with cleartext keys, `encrypted_regex
+  ^(data|stringData)$`, and exactly one age recipient:
+  `age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg` — the recipient in the root
+  `.sops.yaml`, i.e. the key Flux holds as `sops-age`.
+- **Round-trip, without exposing anything**: `sops -d` then a Python read of key names only →
+  `stringData keys: ['privatekey']`, and `value is still the placeholder: True`.
+- **Both overlay patches are present and name the right Kustomization** (`name: vpn-ingress`,
+  `namespace: flux-system`, `decryption.secretRef.name: sops-age`), inserted after each overlay's
+  `velero` stanza, mirroring headlamp/crafty/velero.
+- **`kubectl kustomize platform/vpn-ingress` builds** and now emits five objects — Namespace,
+  ConfigMap, **Secret**, DaemonSet, NetworkPolicy — with `${WG_CLIENT_CIDR}` still an unexpanded
+  token for Flux.
+- **`wg` / `sops` / `qrencode` locations**, measured on both host and container, as described above.
+- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
+  `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two
+  environmental skips (`gitea-settings` — no local token; conformance L2/L3 — no cluster).
+  **Not green. Reported as rc 3, not as pass.**
+- **The L0 gate caught THREE real defects in this branch before it could merge — including one
+  in the fix for another — and I had asserted one of them was fine without running it.** Both are recorded because the gate's value here is the
+  whole point of the exercise:
+  1. **`no-plaintext-secrets` FAILED**: "`wg-server-key.sops.yaml.example`: an unencrypted .example
+     template is referenced by a kustomization — it would be applied as a plaintext Secret". My
+     kustomization named the template in a *comment*, and the check matches comments too (correctly:
+     naming one is how a plaintext Secret gets applied by accident). `platform/crafty/kustomization.yaml`
+     never names its own example either — the convention was there to copy and I missed it. Fixed by
+     not naming the file; the hand-off lives inside the example, where it belongs.
+     **An earlier draft of this entry claimed this check passed. It did not. The claim was written
+     from expectation rather than from a run.**
+  2. **`flux-postbuild-substitution` FAILED** on `${key#ENC[}` in my new guard
+     (`daemonset.yaml:155`), flagged `[SILENT-EMPTY]`: envsubst consumes every `${...}`, so my
+     shell prefix-strip would have rendered to the empty string — the ciphertext branch of the guard
+     would never have fired, which is precisely the silent failure the guard was added to catch.
+     Rewritten with `case`, which uses no braces and so has nothing to escape.
+  3. **And then it failed a THIRD time, on the fix's own comment** — `[FATAL-PARSE]
+     daemonset.yaml:156 ${...}`, the prose explaining the rule, written with the literal form. That
+     class is worse than it looks: a `FATAL-PARSE` hazard fails the **entire** Kustomization, every
+     resource in the path, not just the line. Rewritten as "dollar-brace expansion". This is the
+     second time in this component that writing the hazard literally inside a comment about the
+     hazard has been the defect (the first was in #55), so the habit to build is: **in any
+     substituted path, describe the construct in words, never type it.**
+- **After both fixes, `no-plaintext-secrets` and `flux-postbuild-substitution` pass** (446 and 198
+  assertions respectively), and `l0` is green.
+
+## What I did NOT verify
+
+- **That Flux decrypts this file on prod.** The overlay patches have never reconciled. This is the
+  whole risk of the change and it has exactly one observable outcome: after merge the pods should
+  CrashLoop reporting the **`REPLACE-ME` placeholder** (proving decryption worked), *not* reporting
+  **`ENC[`** (which would prove a patch is wrong). I built the guard to tell those apart because I
+  cannot test them.
+- **That `wg pubkey` accepts the operator's real key through this path.** The guard is reasoned from
+  `wg pubkey`'s behaviour and has never run — `wg` is absent from both this workstation's container
+  and the gate, so the first execution will be in the pod.
+- **The `sops set` command in the hand-off.** I verified `sops -e -i` and `sops -d` in the toolbox;
+  I did **not** run `sops set` with the quoting shown, and nested quoting inside
+  `dev.sh run bash -c` is exactly where a documented command tends to break. If it misbehaves, the
+  editor form (`sops <file>`) is the fallback and does not depend on quoting.
+- **The sha256 comparison in step 2 step 4** is untested end to end (it needs the real key).
+- **Whether anything else in #55 assumed a tool is available where it is not.** I checked `wg`,
+  `sops` and `qrencode` because step 2 depended on them. I did not audit every command in the
+  runbook the same way — and that class of defect has now produced two corrections in a row
+  (`site.sh`, then `sops | wg`), so assume more remain until each step has actually been run.
+- **The live VPN still carries no traffic**, and the earlier gaps stand unchanged: the
+  iptables-vs-nftables question, `allowPrivilegeEscalation: false` versus the image's s6 init, and
+  any off-LAN reachability.

infra/vpn-ingress/new-peer.sh

@@ -89,7 +89,7 @@ command -v wg >/dev/null || die "wireguard-tools not installed ('wg' not on PATH
 if [ -z "${WG_SERVER_PUBKEY:-}" ]; then
   [ -r "$WG_SERVER_KEY_FILE" ] || die "cannot read server key '$WG_SERVER_KEY_FILE'.
   Either point WG_SERVER_KEY_FILE at it, or set WG_SERVER_PUBKEY to the public half.
-  If the server key does not exist yet, see runbooks/vpn-remote-access.md step 1."
+  If the server key does not exist yet, see runbooks/vpn-remote-access.md step 2."
   WG_SERVER_PUBKEY="$(tr -d ' \r\n' < "$WG_SERVER_KEY_FILE" | wg pubkey)"
 fi
 

platform/vpn-ingress/README.md

@@ -14,7 +14,7 @@ egress out the house WAN.
 | Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node |
 | Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices |
 | Authorisation | a device's **public key** is in [`peers.yaml`](./peers.yaml), or it cannot connect |
-| The one secret | the server's private key, Secret `wg-server-key`, **created out of band** |
+| The one secret | the server's private key — **SOPS-encrypted in this tree**, `wg-server-key.sops.yaml` |
 | Exposure | one hand-made WAN port-forward on the SOHO router, off-repo |
 | Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) |
 | Procedure / bring-up | [`runbooks/vpn-remote-access.md`](../../runbooks/vpn-remote-access.md) |
@@ -130,11 +130,17 @@ translations win here.
 | half | where | why |
 |---|---|---|
 | peer **public** keys + addresses | [`peers.yaml`](./peers.yaml), **plaintext in git** | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
-| the server's **private** key | Secret `wg-server-key`, **out of band** | the only real secret; an agent must not mint a long-lived private key (CLAUDE.md §6) |
+| the server's **private** key | `wg-server-key.sops.yaml`, **SOPS-encrypted in git** | the only real secret (R-SEC-01, spec docs/07). Ships with a `REPLACE-ME` placeholder: an agent must not mint a long-lived private key (CLAUDE.md §6), so the operator `sops set`s it in |
 
 The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** —
 so the assembled config holding the private key never touches any node's disk and dies with the pod.
 
+Both halves are therefore in git, which is the point: a cluster rebuild restores the server key with
+everything else. An out-of-band `kubectl create secret` — the first shape this component shipped in,
+corrected after review — would have put the one credential every enrolled device depends on *outside*
+the GitOps tree, and losing it means re-enrolling every device, because each client profile pins this
+key's public half.
+
 Encrypting the peer register would have hidden exactly the change a reviewer most needs to see: a
 new line granting a new device access to the whole LAN.
 
@@ -173,8 +179,14 @@ NetworkPolicy applies.
    (`infra/ansible/roles/node_baseline/defaults/main.yml`); the operator runs the role. The pod has
    `NET_ADMIN` but deliberately **not** `SYS_MODULE`, so it cannot load the module itself — it
    CrashLoops with a clear `wg-quick` error instead of quietly degrading to userspace WireGuard.
-2. **The `wg-server-key` Secret.** Runbook step 1. Until it exists the DaemonSet cannot start —
+2. **The server private key.** `wg-server-key.sops.yaml` is committed, encrypted, with a
-   the correct fail-closed state for an unenrolled VPN.
+   `REPLACE-ME` placeholder, and both overlays already carry its decryption patch — so the wiring
+   is complete and the only outstanding act is replacing the value (runbook step 2). Until then the
+   DaemonSet CrashLoops with an explicit FATAL naming the file: the init container validates the
+   value with `wg pubkey`, which rejects the placeholder, an empty value, a public key pasted by
+   mistake, **and** a Secret that arrived as undecrypted `ENC[...]` ciphertext. That last case is
+   the one worth having a guard for — a missing overlay decryption patch applies cleanly and leaves
+   every Flux condition green.
 3. **The WAN port-forward**, udp/51820 → one node address. Hand-made on the SOHO router, off-repo,
    and the half CI can never check.
 

platform/vpn-ingress/daemonset.yaml

@@ -136,9 +136,38 @@ spec:
               # The key arrives as a Secret file; strip any trailing newline the operator's shell
               # added, or wg-quick rejects the config with "Key is not the correct length or format".
               key=$(tr -d ' \r\n' < /run/wg-server-key/privatekey)
-              if [ -z "$key" ]; then
+              # VALIDATE, don't just check for empty. The Secret ships committed with a REPLACE-ME
-                echo "FATAL: /run/wg-server-key/privatekey is empty." >&2
+              # placeholder so the GitOps wiring is complete before the key exists, so the common
-                echo "       See runbooks/vpn-remote-access.md step 1 (create the server key)." >&2
+              # failure is a value that is present and WRONG — a placeholder, a public key pasted by
+              # mistake, or a SOPS document that reached the cluster undecrypted (ENC[...], which
+              # happens if an overlay is missing its `vpn-ingress` decryption patch). `wg pubkey`
+              # accepts a value only if it is a real 32-byte curve25519 private key, so it rejects
+              # all four. Without this the bad value would be written into wg0.conf and the pod would
+              # fail later and less legibly, or come up with a server nobody can handshake with.
+              if ! printf '%s' "$key" | wg pubkey >/dev/null 2>&1; then
+                echo "FATAL: the server private key in Secret 'wg-server-key' is not a valid" >&2
+                echo "       WireGuard private key (wg pubkey rejected it)." >&2
+                if [ -z "$key" ]; then
+                  echo "       It is EMPTY." >&2
+                elif [ "$key" = "REPLACE-ME" ]; then
+                  echo "       It is still the committed REPLACE-ME PLACEHOLDER — the key has not" >&2
+                  echo "       been generated yet. This is the expected state on a fresh install." >&2
+                else
+                  # `case`, NOT a dollar-brace prefix-strip. This manifest is rendered through
+                  # envsubst, which consumes every dollar-brace expansion it sees, so a shell
+                  # parameter expansion here renders to the empty string and silently disables this
+                  # very branch. `case` uses no braces, so there is nothing to escape.
+                  # (ci: flux-postbuild-substitution caught both the original expansion AND the
+                  # first version of this comment, which spelled the hazard out literally.)
+                  case "$key" in
+                    ENC\[*)
+                      echo "       It is still SOPS CIPHERTEXT (starts with ENC[). The Secret" >&2
+                      echo "       reached the cluster undecrypted — check that this environment's" >&2
+                      echo "       overlay carries the 'vpn-ingress' decryption patch." >&2
+                      ;;
+                  esac
+                fi
+                echo "       Fix: runbooks/vpn-remote-access.md step 2." >&2
                 exit 1
               fi
 
@@ -272,14 +301,22 @@ spec:
             # during startup rather than at a point anyone would connect to the volume size. Still
             # counted against the container's 128Mi limit, and still far more than wg0.conf needs.
             sizeLimit: 8Mi
-        # The ONLY secret in this component. Created OUT OF BAND by the operator and referenced by
+        # The ONLY secret in this component. SOPS-encrypted IN THIS TREE (R-SEC-01, spec docs/07) as
-        # name (R-IDP-04) — runbooks/vpn-remote-access.md step 1. It is NOT a *.sops.yaml in this
+        # wg-server-key.sops.yaml, delivered by Flux, referenced here by name only (R-IDP-04).
-        # tree on purpose: an agent must not mint a long-lived private key (CLAUDE.md §6), and a key
+        #
-        # that passed through a session transcript would have to be treated as compromised.
+        # It ships committed with a REPLACE-ME placeholder: an agent must not mint a long-lived
-        # DR: Velero's `daily` schedule excludes only `monitoring` and `unifi`, so this namespace
+        # private key (CLAUDE.md §6), so the operator generates it and `sops set`s it in
-        # and this Secret ARE captured off-cluster (platform/velero/helmrelease.yaml). Keep a copy
+        # (runbooks/vpn-remote-access.md step 2). Shipping the encrypted placeholder rather than no
-        # in the operator's password manager as well — restoring it is what makes every already
+        # file at all is what lets the resource line and both overlay decryption patches be present
-        # enrolled client work again without re-enrolling.
+        # and GREEN from the first commit — the alternative, a commented-out resource to uncomment
+        # later, is exactly the "file in git with no generator entry: reviewed, merged, apparently
+        # done, still broken" shape this repo keeps getting bitten by.
+        #
+        # DR: in git means a cluster rebuild restores it with everything else — the property an
+        # out-of-band `kubectl create secret` would NOT have had, and losing this key means
+        # re-enrolling every device because each client profile pins its public half. Velero also
+        # captures this namespace (its `daily` schedule excludes only `monitoring` and `unifi`).
+        # Keep a password-manager copy too: that is the one that survives losing cluster AND backups.
         - name: wg-server-key
           secret:
             secretName: wg-server-key

platform/vpn-ingress/kustomization.yaml

@@ -4,9 +4,9 @@
 # same placement as crafty/headlamp/homepage/runbooks, for the same reason: a leaf whose pod can be
 # legitimately unhealthy must never sit in the gating `platform` tier, where one component's
 # unhealth stalls platform-config and every apps wave behind it (DECISIONS-LOG: "Headlamp gating
-# stall"). That rationale is unusually concrete here: until the operator creates the `wg-server-key`
+# stall"). That rationale is unusually concrete here: until the operator replaces the `REPLACE-ME`
-# Secret (runbooks/vpn-remote-access.md step 1) this DaemonSet CANNOT start, so putting it in the
+# placeholder in wg-server-key.sops.yaml (runbooks/vpn-remote-access.md step 2) this DaemonSet
-# gating tier would have taken the whole platform down on merge.
+# CANNOT start, so putting it in the gating tier would have taken the whole platform down on merge.
 #
 # NOT in platform/kustomization.yaml — see clusters/base/vpn-ingress.yaml.
 apiVersion: kustomize.config.k8s.io/v1beta1
@@ -19,12 +19,22 @@ resources:
   - daemonset.yaml
   # Namespace default-deny baseline. NOTE: does NOT constrain the hostNetwork DaemonSet — the file
   # says so at length rather than implying a protection it cannot provide.
-  - networkpolicy.yaml
+  - networkpolicy.yaml  # The server's WireGuard private key — the ONE secret here. SOPS-encrypted (R-SEC-01, spec
+  # docs/07), decrypted per-env by the `vpn-ingress` Kustomization, referenced by name only from
+  # daemonset.yaml (R-IDP-04). Its plaintext `.example` sibling carries the operator hand-off --
+  # deliberately NOT named here: conformance/tests/l0-static/no-plaintext-secrets fails any
+  # kustomization that mentions an unencrypted template, because naming one is how a plaintext
+  # Secret gets applied by accident. (It matches comments too. That is correct, not overzealous.)
+  #
+  # SHIPS WITH A `REPLACE-ME` PLACEHOLDER, already encrypted. An agent must not
+  # mint a long-lived private key (CLAUDE.md §6), and the init container validates the value with
+  # `wg pubkey`, so the unfilled state is a loud CrashLoop naming this file, never a server that
+  # quietly cannot handshake.
+  - wg-server-key.sops.yaml
 #
-# DELIBERATELY ABSENT: a Secret. The server's private key (`wg-server-key`) is created out of band
+# THIS COMPONENT CARRIES AN ENCRYPTED SECRET, so BOTH overlays must patch the `vpn-ingress`
-# by the operator and referenced BY NAME from daemonset.yaml. There is no `*.sops.yaml` here, so
+# Kustomization with a `decryption:` stanza — clusters/dev and clusters/prod both do, added by the
-# this component needs NO SOPS decryption stanza in clusters/dev or clusters/prod — unlike
+# same change as this file. That patch is as load-bearing as the file itself: without it kustomize
-# crafty/headlamp/velero. If that key is ever migrated into SOPS, BOTH halves must land together:
+# hands Flux the raw ciphertext, the Secret APPLIES CLEANLY full of ENC[...], and nothing in any
-# the file here, and a `decryption:` patch naming the `vpn-ingress` Kustomization in each overlay.
+# Flux condition looks wrong. The init container's `wg pubkey` guard exists partly to turn exactly
-# Without the patch, kustomize hands Flux the raw ciphertext and the pod mounts a key full of
+# that into a named error rather than a silently non-working tunnel.
-# ENC[...] — which applies cleanly and fails only at handshake time.

platform/vpn-ingress/wg-server-key.sops.yaml

@@ -0,0 +1,34 @@
+# WireGuard SERVER private key for platform/vpn-ingress — SOPS-ENCRYPTED (R-SEC-01, spec docs/07).
+# Keys stay cleartext for reviewable diffs; only `stringData` values are encrypted (.sops.yaml
+# `encrypted_regex`). Plaintext template + operator hand-off: wg-server-key.sops.yaml.example.
+apiVersion: v1
+kind: Secret
+metadata:
+    name: wg-server-key
+    namespace: vpn-ingress
+type: Opaque
+stringData:
+    #ENC[AES256_GCM,data:fUoPkfWvasisdTtwdhjeOVOlmdQkTl6Bp9YxYyji8g6k+ClATds3BnHJWOj2iplaGpT2XaNakq1+D4YyOi7QbFvNJui+e4qRpqcY8hWnazvZIHJYxjxfAfzW,iv:xtKd6oMZQumAQJAsNLHqJB+o0wdQJ+vL1kI1mc9jL7w=,tag:aImqLgbSmhsRI36/t5kpyg==,type:comment]
+    #ENC[AES256_GCM,data:npDsCW6rQleolIpur5gFkX2jSRrIQzBlGaIPs2Schv0YU3q8AanG39z4QqT3ASDmskV5YuH29Bo0ujEAgBf52YIjQCp58PDDETofrVKOJzhHMa7bV+HCxUFE,iv:7E7thyFBleyqZpqJRGApKvIq3ubCFOkshF/mxim3YPc=,tag:joIrhMe2I6kUnx1ouOPdZw==,type:comment]
+    #ENC[AES256_GCM,data:wHUFx09FgP/2+LTyw61EFbKA5UaqfjU7CeGk6Xc5av2y4r4AvQJRefDM5EGlahKecDFEFsf7fSQRFRHtUJqrQTtw+6tcj3ubSCPW4iJvhyJJv0xdb/0puRmcgj+2NfgF,iv:xgDOSDLqMQ1wcI9vrkKqfNMgIDFEusFnnO7Rzq/aOa4=,tag:m1m/fgjkBsG91c+3daqOpA==,type:comment]
+    privatekey: ENC[AES256_GCM,data:irLENWa1Z3IJSQ==,iv:Qu2Yr6F5+M6P8cQRuhB5J7joxCcudUhwoEWDiL941Lo=,tag:ZNwcBZEkswtnkX0vGKfMAQ==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBMays2OXY0VkpRc2FQQmhU
+            Q0owM3N0VkhYR0lYTUdNZWRqeTFIMjNHOVJ3CnJSMnc5NFEyU0IzWTFtUkw2YVFS
+            MU02VFo1TE14YVpZRzNhVjZwRTZwd0UKLS0tIDFaMWF4NUZjZmhRdXRuZFJYNkg3
+            SUUzN1FnMlI4bkkzbVNlMWhUVnJZODAKQ+Iy86CIv/Oj9E9j4E/0y5hDbcctRuqB
+            OR825KRPRaTdF/G2hudE1R1JbM4wGbPLMzv078R4ZXHDPGvDttSEkA==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-10-06T04:20:58Z"
+    mac: ENC[AES256_GCM,data:GpP/zTX4FtJyrlaCS/R0/sAtFbkM0W8A5sqNxb5ZziSgLUXI+oUKYB09igN6vx5cjRNH7rSmXduS/actaT5jhP8kuF1n/iQfG56PsCJHL2vneefzcEOKXeGquAwDOY+iGmNsZ5R8ySrM/Fq7HYLMJasWosyTh3YlBs8GRYiHmU4=,iv:438+pDzHLGyh+OMQCMzGBEBrZUPNp5BbjrrMdiEiqJ8=,tag:JqLu6zZWKtcoB8mp4HSZig==,type:str]
+    pgp: []
+    encrypted_regex: ^(data|stringData)$
+    version: 3.9.4

platform/vpn-ingress/wg-server-key.sops.yaml.example

@@ -0,0 +1,82 @@
+# =============================================================================
+# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
+# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
+#
+# WHAT THIS IS: the WireGuard SERVER's long-term private key for the remote-access VPN
+# (platform/vpn-ingress). It is the ONE secret in the whole component. The init container in
+# daemonset.yaml reads it at `/run/wg-server-key/privatekey` and joins it to the PLAINTEXT peer
+# register (peers.yaml) in a tmpfs, so the assembled wg0.conf never touches a node's disk.
+#
+# WHY IT IS IN GIT, SOPS-ENCRYPTED, AND NOT HAND-CREATED. R-SEC-01 / spec docs/07: secrets live in
+# source, encrypted, and reach the cluster only through Flux. An out-of-band `kubectl create secret`
+# would put the single credential every enrolled device depends on OUTSIDE the GitOps tree, where a
+# cluster rebuild silently loses it — and losing it means RE-ENROLLING EVERY DEVICE, because each
+# client profile pins this key's public half.
+#
+# *** THE REAL FILE IS COMMITTED WITH `privatekey: REPLACE-ME`, ALREADY ENCRYPTED. ***
+# That is deliberate, so the wiring is complete and green before the key exists: the resource is
+# already listed in kustomization.yaml and both overlays already carry the `vpn-ingress` decryption
+# patch. Your job is ONLY to replace the value (step 3 below). Until you do, the DaemonSet
+# CrashLoops with an explicit FATAL naming this file — the init container validates the value with
+# `wg pubkey` before writing wg0.conf, so a placeholder fails CLOSED and LOUDLY rather than
+# producing a server that silently cannot complete a handshake.
+#
+# ---------------------------------------------------------------------------------------------
+# MIND WHERE EACH COMMAND RUNS. The two tools do not live in the same place (measured 2026-10-05):
+#     wg     -> the HOST only          (/usr/bin/wg; NOT in the dev container)
+#     sops   -> the dev CONTAINER only (/usr/local/bin/sops; NOT on the host)
+# So you cannot pipe `sops -d ... | wg pubkey` in one shell. The steps below are split accordingly,
+# and they never put the key on a terminal or in a pipeline that crosses the boundary.
+# ---------------------------------------------------------------------------------------------
+#
+# HAND-OFF STEPS (operator — an agent must not mint a long-lived private key, CLAUDE.md §6).
+# Full procedure with verification: runbooks/vpn-remote-access.md step 2.
+#
+#  1. HOST — generate it. This file is also what infra/vpn-ingress/new-peer.sh reads to derive the
+#     public half for each client profile, so keep it where the script's default expects it:
+#         umask 077
+#         mkdir -p ~/.config/wireguard-home && chmod 700 ~/.config/wireguard-home
+#         wg genkey > ~/.config/wireguard-home/server.key
+#
+#  2. Put that value in your PASSWORD MANAGER. It cannot be regenerated without invalidating every
+#     client profile. (Velero backs this namespace up to R2 too, but the password-manager copy is
+#     the one that survives losing the cluster AND the backups.)
+#     Record the public half alongside it — safe to display, and useful when debugging a client:
+#         wg pubkey < ~/.config/wireguard-home/server.key
+#
+#  3. Stage it where the CONTAINER can read it, then write it into the encrypted file in place:
+#         mkdir -p .cache && cp ~/.config/wireguard-home/server.key .cache/server.key
+#         git check-ignore -v .cache/server.key    # MUST print `.gitignore:11:*.key` — see below
+#         ./infra/devcontainer/dev.sh run bash -c \
+#           'cd /work && sops set platform/vpn-ingress/wg-server-key.sops.yaml \
+#              "[\"stringData\"][\"privatekey\"]" "\"$(tr -d "\r\n" < /work/.cache/server.key)\""'
+#         shred -u .cache/server.key
+#     `sops set` edits the ENCRYPTED document directly — no plaintext copy of it is ever written.
+#
+#     KEEP THE `.key` SUFFIX on that staged copy. `.gitignore` line 11 is `*.key`, and it is the
+#     ONLY rule covering that path: `.cache/` itself is NOT ignored. A copy named `server.pem` or
+#     `privatekey` in the same directory is COMMITTABLE, and the gate's `no-plaintext-secrets` stage
+#     does not recognise a bare WireGuard key.
+#
+#  4. Verify you wrote the right bytes WITHOUT printing them — compare digests, not values:
+#         ./infra/devcontainer/dev.sh run bash -c \
+#           'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
+#              platform/vpn-ingress/wg-server-key.sops.yaml | tr -d "\r\n" | sha256sum'
+#         tr -d '\r\n' < ~/.config/wireguard-home/server.key | sha256sum
+#     The two digests must match. (This is the only sanctioned decrypt of this file, and its output
+#     goes straight into sha256sum — never onto a terminal. CLAUDE.md §6.)
+#
+#  5. PR it. Nothing else to wire: the resource line and both overlay decryption patches already
+#     exist. Flux rolls the DaemonSet and the pods leave CrashLoop.
+# =============================================================================
+apiVersion: v1
+kind: Secret
+metadata:
+  name: wg-server-key
+  namespace: vpn-ingress
+type: Opaque
+stringData:
+  # The server's WireGuard private key: `wg genkey` output — 44 chars of base64 ending in '='.
+  # NOT a passphrase, and NOT the public key. The init container validates it with `wg pubkey`
+  # before writing wg0.conf, so a placeholder or a mangled value fails closed with a clear message.
+  privatekey: REPLACE-ME

runbooks/vpn-remote-access.md

@@ -62,53 +62,90 @@ lsmod | grep '^wireguard'
 grep -w wireguard /etc/modules-load.d/k3s-cilium-longhorn.conf
 ```
 
-## Step 2 — create the server key (once)
+## Step 2 — fill in the server key (once)
 
-This is the **only** secret in the component, and the one thing that must never be generated in an
+The key is **SOPS-encrypted in this tree** (R-SEC-01, spec docs/07), not created by hand:
-agent session or pasted into a transcript. Generate it on your own machine:
+`platform/vpn-ingress/wg-server-key.sops.yaml`. It is already committed — encrypted, with a
+`REPLACE-ME` placeholder — and it is already listed in the component's `resources:` with a
+`decryption:` patch in both overlays. **Nothing needs wiring. You are only replacing one value.**
+
+Why in git rather than `kubectl create secret`: an out-of-band Secret puts the one credential every
+enrolled device depends on outside the GitOps tree, where a cluster rebuild silently loses it — and
+losing it means re-enrolling every device, because each client profile pins this key's public half.
+
+> **MIND WHERE EACH COMMAND RUNS.** The two tools are not in the same place (measured 2026-10-05):
+> `wg` is on the **HOST** only (`/usr/bin/wg`, absent from the dev container); `sops` is in the
+> **CONTAINER** only (`/usr/local/bin/sops`, absent from the host). You cannot pipe
+> `sops -d … | wg pubkey` in one shell. The steps below are split accordingly and never put the key
+> on a terminal.
+
+**1. HOST — generate it.** Keep it at this path: it is where `new-peer.sh` looks by default to
+derive the public half for each client profile.
 
 ```bash
-mkdir -p ~/.config/wireguard-home && chmod 700 ~/.config/wireguard-home
 umask 077
-wg genkey > ~/.config/wireguard-home/server.key          # keep this; see "Back it up" below
+mkdir -p ~/.config/wireguard-home && chmod 700 ~/.config/wireguard-home
+wg genkey > ~/.config/wireguard-home/server.key
 ```
 
-Load it into the cluster. **Do not echo it, do not pass it on an interactive command line that
+**2. Store it in your PASSWORD MANAGER**, together with its public half (safe to display):
-lands in shell history, and never print the resulting Secret's `.data`** (CLAUDE.md §6):
+
+```bash
+wg pubkey < ~/.config/wireguard-home/server.key
+```
+
+This copy is the one that survives losing the cluster *and* the backups. Velero does capture this
+namespace to R2 (its `daily` schedule excludes only `monitoring` and `unifi`), and the encrypted
+file is in git — but the key itself cannot be regenerated without invalidating every client profile.
+
+**3. Write it into the encrypted file in place.**
+
+```bash
+mkdir -p .cache && cp ~/.config/wireguard-home/server.key .cache/server.key
+git check-ignore -v .cache/server.key    # MUST print `.gitignore:11:*.key` — see the warning below
+
+./infra/devcontainer/dev.sh run bash -c \
+  'cd /work && sops set platform/vpn-ingress/wg-server-key.sops.yaml \
+     "[\"stringData\"][\"privatekey\"]" "\"$(tr -d "\r\n" < /work/.cache/server.key)\""'
+
+shred -u .cache/server.key
+```
+
+`sops set` edits the **encrypted** document directly, so no plaintext copy of it is ever written.
+(`sops platform/vpn-ingress/wg-server-key.sops.yaml` opens an editor instead, if you prefer.)
+
+> **KEEP THE `.key` SUFFIX** on the staged copy. `.gitignore` line 11 is `*.key`, and it is the
+> **only** rule covering that path — `.cache/` itself is NOT ignored
+> (`git check-ignore -v .cache/serverkey.txt` matches nothing). A copy named `server.pem` or
+> `privatekey` in that directory is **committable**, and the gate's `no-plaintext-secrets` stage
+> does not recognise a bare WireGuard key.
+
+**4. Verify you wrote the right bytes, without printing them** — compare digests, not values:
 
 ```bash
 ./infra/devcontainer/dev.sh run bash -c \
-  'kubectl --context k3s-prod -n vpn-ingress create secret generic wg-server-key \
+  'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
-     --from-file=privatekey=/work/.cache/server.key'
+     platform/vpn-ingress/wg-server-key.sops.yaml | tr -d "\r\n" | sha256sum'
+
+tr -d '\r\n' < ~/.config/wireguard-home/server.key | sha256sum
 ```
 
-…where `/work/.cache/server.key` is a path you copied the key to (`.cache/` is not tracked; create
+The two digests must match. This is the only sanctioned decrypt of this file, and its output goes
-it if absent). Delete that copy as soon as the Secret exists: `shred -u .cache/server.key`.
+straight into `sha256sum` — never onto a terminal (CLAUDE.md §6).
 
-> **KEEP THE `.key` SUFFIX.** That filename is what keeps the key out of git: `.gitignore` line 11
+**5. PR it, and merge.** Flux rolls the DaemonSet and the pods leave CrashLoop.
-> is `*.key`, and it is the **only** rule covering this path — `.cache/` itself is NOT ignored
-> (`git check-ignore -v .cache/serverkey.txt` matches nothing). So a copy named `server.pem`,
-> `wg-server` or `privatekey` sitting in that directory is **committable**, and the gate's
-> `no-plaintext-secrets` stage does not know a WireGuard private key when it sees one. Verify before
-> you trust it: `git check-ignore -v .cache/<your-filename>` must print a rule, and
-> `git status --short` must not list the file.
 
-Confirm it exists **by key name only**:
+### What the pod tells you if this went wrong
 
-```bash
+The init container validates the value with `wg pubkey` before writing `wg0.conf`, and names the
-./infra/devcontainer/dev.sh run bash -c \
+specific failure in its log — so you never get a server that silently cannot complete a handshake:
-  'kubectl --context k3s-prod -n vpn-ingress get secret wg-server-key -o jsonpath="{.data}" | tr "," "\n" | cut -d: -f1'
-# expected: {"privatekey   -- a NAME. If you see base64 payload, you printed the key. Rotate it.
-```
 
-**Why out of band and not SOPS.** An agent must not mint a long-lived private key, and a key that
+| pod log says | meaning |
-passed through a session transcript would have to be treated as compromised. The trade-off is that
+|---|---|
-this Secret is **not reconstructible from git** — so:
+| `still the committed REPLACE-ME PLACEHOLDER` | step 3 not done (the expected fresh-install state) |
-
+| `It is EMPTY` | the value was blanked |
-**Back it up.** Keep `server.key` in your password manager. Velero's `daily` schedule excludes only
+| `still SOPS CIPHERTEXT (starts with ENC[)` | the Secret reached the cluster undecrypted — this environment's overlay is missing its `vpn-ingress` decryption patch. **This is the dangerous one:** the Secret applies cleanly and every Flux condition stays green |
-`monitoring` and `unifi` (`platform/velero/helmrelease.yaml`), so this namespace *is* captured
+| `wg pubkey rejected it` with none of the above | a mangled value, or a *public* key pasted by mistake |
-off-cluster to R2 as well. Restoring this one key is what makes every already-enrolled device work
-again after a cluster rebuild — **losing it means re-enrolling every device.**
 
 ## Step 3 — forward udp/51820 on the router (once)
 
@@ -153,8 +190,16 @@ unauthenticated traffic either way. If the pod is Ready, the interface exists.
 ./infra/vpn-ingress/new-peer.sh thinkpad 3 --split     # split tunnel (LAN only)
 ```
 
-It generates the keypair **on your machine**, writes a `0600` client profile outside the checkout,
+It generates the keypair **on your machine**, writes a `0600` client profile outside the checkout
-prints a QR code if `qrencode` is installed, and prints the `[Peer]` stanza to add. Then:
+(it refuses to write inside the git checkout), prints a QR code, and prints the `[Peer]` stanza to
+add.
+
+> **`qrencode` is installed neither on this workstation nor in the dev container** (measured
+> 2026-10-05), so the QR path needs it first: `sudo dnf install qrencode` (or `apt install
+> qrencode`). Without it the script says so plainly and leaves you to move the `.conf` to the
+> device by another route — fine for a desktop, awkward for a phone.
+
+Then:
 
 1. Paste the stanza into `platform/vpn-ingress/peers.yaml` under `data.peers.conf`, keeping the
    `# <name> — <owner>, enrolled <date>` comment. `wg show` can only ever print the key, so that