home-network-k3s-impl/decisions/2026-10-05-vpn-server-key-in-sops.md

---
date: 2026-10-05
branch: fix/vpn-server-key-in-sops
title: VPN server key moves into SOPS-encrypted git — the out-of-band Secret broke R-SEC-01 and the DR story
---

## What changed

**Operator correction (2026-10-05): "secrets are kept in source through sops".** They are right, and
#55 shipped the WireGuard server private key the wrong way: as a Secret the operator was told to
`kubectl create` out of band, referenced by name only. This branch moves it into the tree as
`platform/vpn-ingress/wg-server-key.sops.yaml`, where every other secret in this repo lives.

**Why the original choice was wrong, not merely unconventional.** Two concrete costs, both of which
#55's own README half-admitted and then shipped anyway:

1. **R-SEC-01 / spec docs/07 says secrets live in source, SOPS-encrypted, reaching the cluster only
   through Flux.** An out-of-band Secret is outside the GitOps tree by construction.
2. **It broke DR for the one credential that cannot be re-derived.** A cluster rebuild restores
   everything in git; it does not restore a hand-made Secret. Losing this key means **re-enrolling
   every device**, because each client profile pins its public half. #55 papered over this with
   "keep a copy in your password manager", which is a workaround for a self-inflicted gap.

Note `platform/kyverno/admission/secret-provenance.yaml` did **not** forbid the original shape —
it scopes to `app-namespace=true` namespaces and `vpn-ingress` is deliberately a platform namespace,
so admission would have accepted the hand-applied Secret. The rule it violated was the written
requirement, not an enforced control. Recording that because "the gate let it through" was never
evidence it was correct.

**How the unfillable-key problem is solved.** An agent still must not mint a long-lived private key
(CLAUDE.md §6), so the file is committed **already encrypted, with `privatekey: REPLACE-ME`**. That
makes the wiring complete and the gate green from the first commit — the resource line and both
overlay decryption patches are present — and leaves the operator exactly one act: replace the value.
The alternative, a commented-out resource line to uncomment later, is precisely the "file in git with
no generator entry: reviewed, merged, apparently done, still broken" shape this repo keeps getting
bitten by (`alert-runbooks-resolvable` exists because of it).

**The init-container guard is now a validation, not an emptiness check.** It was `[ -z "$key" ]`.
Because the committed placeholder is non-empty, the likely failure is now a value that is present
and *wrong*, so the guard pipes the value through `wg pubkey` — which accepts only a real 32-byte
curve25519 private key — and names which of four cases it hit:

| case | why it matters |
|---|---|
| `REPLACE-ME` placeholder | the expected fresh-install state |
| empty | blanked value |
| **starts with `ENC[`** | **the Secret reached the cluster UNDECRYPTED.** A missing overlay decryption patch applies cleanly and leaves every Flux condition green — this is the silent failure the guard exists for |
| rejected, none of the above | mangled value, or a *public* key pasted by mistake |

Files:

- `platform/vpn-ingress/wg-server-key.sops.yaml` (new) — SOPS-encrypted to the repo's age recipient,
  `stringData.privatekey` only, placeholder value.
- `platform/vpn-ingress/wg-server-key.sops.yaml.example` (new) — plaintext template + operator
  hand-off, following the `crafty-oauth2-proxy-credentials.sops.yaml.example` convention.
- `platform/vpn-ingress/kustomization.yaml` — the resource added; the "DELIBERATELY ABSENT: a
  Secret" block replaced with the overlay-patch warning.
- `platform/vpn-ingress/daemonset.yaml` — the `wg pubkey` guard; volume comment rewritten.
- `clusters/dev/kustomization.yaml`, `clusters/prod/kustomization.yaml` — **the `vpn-ingress`
  `decryption:` patch, in both.** This is the load-bearing half: #55's own kustomization comment
  predicted that forgetting it applies ciphertext cleanly.
- `clusters/base/vpn-ingress.yaml` — its "NO decryption stanza is needed" comment was now false.
- `platform/vpn-ingress/README.md`, `runbooks/vpn-remote-access.md` — step 2 rewritten.
- `infra/vpn-ingress/new-peer.sh` — stale "step 1" pointer.

**A second finding: `wg` and `sops` were not in the same place — and the first fix for that was
also wrong.** Measured: `wg` existed only on the host (`/usr/bin/wg`), `sops` only in the dev
container, `qrencode` on neither. So `sops -d … | wg pubkey` could not run in one shell, and #55's
step 2 — which assumed it could — would have failed.

My first fix was to **split every step by where it runs**, and to tell the operator to
`dnf install qrencode` on their workstation. **Operator correction: "sop, wg, and any other tool
requirement is supposed to be baked into the devcontainer."** That is CLAUDE.md §5 and the
devcontainer README's "Don't install the toolchain on the host", and the split fix broke the
invariant rather than honouring it — it also produced a procedure that could never be reproduced in
CI, and which no one should have to follow.

**The real fix:** `wireguard-tools` and `qrencode` are now distro packages in
`infra/devcontainer/Containerfile`, alongside ansible/openssh/diffutils. Both are in Fedora 44's
repos (`wireguard-tools` 1.0.20260223 — the same upstream version as the container image this
component runs — and `qrencode` 4.1.1), unpinned like every other dnf package there, because the
pinned thing is the Fedora base and neither is a gate tool whose version changes what green means.

Three things got *simpler* rather than merely relocated, which is the tell that the operator's rule
was the right one:

1. **Step 2 is now one pipeline in one shell:** `sops set … "\"$(wg genkey)\""` generates the key
   straight into the already-encrypted document. The plaintext never lands on disk, never reaches a
   terminal, and there is no staged copy to shred. The previous version needed a copy in
   `.cache/server.key`, a `git check-ignore` warning about the `*.key` suffix being the only thing
   protecting it, and a sha256 comparison to prove the right bytes arrived. All of that is gone.
2. **`new-peer.sh` now derives the server's public key from `wg-server-key.sops.yaml` itself**
   (`sops -d --extract … | wg pubkey`), so there is no local key copy to keep in step with the
   encrypted one. It previously required the operator to keep the private key at a magic host path.
3. **`.cache/` is now gitignored at the repo root** (mirroring `infra/ansible/.gitignore`), so the
   container has a persistent, non-committable scratch directory. `new-peer.sh` writes client
   profiles to `.cache/wg-peers/` by default — necessary because the repo bind-mount is the
   container's ONLY persistent read-write mount, so a profile written to `/tmp` in there is gone
   before it can be moved to a desktop. Its old blanket "refuse to write inside the checkout" guard
   is replaced by `git check-ignore`, which tests the actual property instead of a proxy.

**SENSITIVE PATHS — this PR touches `clusters/**` (two overlays) AND adds a `**/*.sops.yaml`, so
`ci/approval-policy.sh` will BLOCK it.** It needs the operator's `/approve-override <reason>` as a
PR **review** of type "Comment". I have deliberately not posted it (CLAUDE.md §1).

## What I verified

- **The encrypted file is real and correctly targeted.** `sops -e -i` in the pinned toolbox
  produced `stringData.privatekey: ENC[AES256_GCM,...]` with cleartext keys, `encrypted_regex
  ^(data|stringData)$`, and exactly one age recipient:
  `age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg` — the recipient in the root
  `.sops.yaml`, i.e. the key Flux holds as `sops-age`.
- **Round-trip, without exposing anything**: `sops -d` then a Python read of key names only →
  `stringData keys: ['privatekey']`, and `value is still the placeholder: True`.
- **Both overlay patches are present and name the right Kustomization** (`name: vpn-ingress`,
  `namespace: flux-system`, `decryption.secretRef.name: sops-age`), inserted after each overlay's
  `velero` stanza, mirroring headlamp/crafty/velero.
- **`kubectl kustomize platform/vpn-ingress` builds** and now emits five objects — Namespace,
  ConfigMap, **Secret**, DaemonSet, NetworkPolicy — with `${WG_CLIENT_CIDR}` still an unexpanded
  token for Flux.
- **The tool gap, measured on both host and container before the fix**: `wg` host-only, `sops`
  container-only, `qrencode` on neither.
- **Both packages exist in the image's repos**, queried from inside the running container:
  `wireguard-tools` 1.0.20260223 (repo `updates`) and `qrencode` 4.1.1 (repo `fedora`).
- **The image REBUILDS with them and both tools resolve inside it** —
  `./infra/devcontainer/dev.sh build` succeeded, and `wg --version` / `qrencode --version` /
  `sops --version` all answer in the rebuilt container. This is the claim the change rests on, so
  it was executed rather than reasoned.
- **`new-peer.sh`'s new SOPS path fails correctly against the placeholder.** Run in the rebuilt
  container it refused with "could not derive the server public key … it still holds the REPLACE-ME
  placeholder", which is both the expected pre-key behaviour and a live test of the
  `sops -d | wg pubkey` pipeline the whole design now depends on.
- **`.cache/` is ignored regardless of filename** now: `git check-ignore -v .cache/anything.conf`
  and `.cache/serverkey.txt` both match `.gitignore:46:.cache/`. Before this change only `*.key`
  matched, which is the fragility the previous runbook had to warn about.
- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
  `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two
  environmental skips (`gitea-settings` — no local token; conformance L2/L3 — no cluster).
  **Not green. Reported as rc 3, not as pass.**
- **The L0 gate caught THREE real defects in this branch before it could merge — including one
  in the fix for another — and I had asserted one of them was fine without running it.** Both are recorded because the gate's value here is the
  whole point of the exercise:
  1. **`no-plaintext-secrets` FAILED**: "`wg-server-key.sops.yaml.example`: an unencrypted .example
     template is referenced by a kustomization — it would be applied as a plaintext Secret". My
     kustomization named the template in a *comment*, and the check matches comments too (correctly:
     naming one is how a plaintext Secret gets applied by accident). `platform/crafty/kustomization.yaml`
     never names its own example either — the convention was there to copy and I missed it. Fixed by
     not naming the file; the hand-off lives inside the example, where it belongs.
     **An earlier draft of this entry claimed this check passed. It did not. The claim was written
     from expectation rather than from a run.**
  2. **`flux-postbuild-substitution` FAILED** on `${key#ENC[}` in my new guard
     (`daemonset.yaml:155`), flagged `[SILENT-EMPTY]`: envsubst consumes every `${...}`, so my
     shell prefix-strip would have rendered to the empty string — the ciphertext branch of the guard
     would never have fired, which is precisely the silent failure the guard was added to catch.
     Rewritten with `case`, which uses no braces and so has nothing to escape.
  3. **And then it failed a THIRD time, on the fix's own comment** — `[FATAL-PARSE]
     daemonset.yaml:156 ${...}`, the prose explaining the rule, written with the literal form. That
     class is worse than it looks: a `FATAL-PARSE` hazard fails the **entire** Kustomization, every
     resource in the path, not just the line. Rewritten as "dollar-brace expansion". This is the
     second time in this component that writing the hazard literally inside a comment about the
     hazard has been the defect (the first was in #55), so the habit to build is: **in any
     substituted path, describe the construct in words, never type it.**
- **After both fixes, `no-plaintext-secrets` and `flux-postbuild-substitution` pass** (446 and 198
  assertions respectively), and `l0` is green.

## What I did NOT verify

- **That Flux decrypts this file on prod.** The overlay patches have never reconciled. This is the
  whole risk of the change and it has exactly one observable outcome: after merge the pods should
  CrashLoop reporting the **`REPLACE-ME` placeholder** (proving decryption worked), *not* reporting
  **`ENC[`** (which would prove a patch is wrong). I built the guard to tell those apart because I
  cannot test them.
- **That `wg pubkey` accepts the operator's real key through this path.** The guard is reasoned from
  `wg pubkey`'s behaviour and has never run — `wg` is absent from both this workstation's container
  and the gate, so the first execution will be in the pod.
- **The `sops set` command in the hand-off.** I verified `sops -e -i` and `sops -d` in the toolbox;
  I did **not** run `sops set` with the quoting shown, and nested quoting inside
  `dev.sh run bash -c` is exactly where a documented command tends to break. If it misbehaves, the
  editor form (`sops <file>`) is the fallback and does not depend on quoting.
- **The sha256 comparison in step 2 step 4** is untested end to end (it needs the real key).
- **Whether anything else in #55 assumed a tool is available where it is not.** I checked `wg`,
  `sops` and `qrencode` because step 2 depended on them. I did not audit every command in the
  runbook the same way — and that class of defect has now produced two corrections in a row
  (`site.sh`, then `sops | wg`), so assume more remain until each step has actually been run.
- **The live VPN still carries no traffic**, and the earlier gaps stand unchanged: the
  iptables-vs-nftables question, `allowPrivilegeEscalation: false` versus the image's s6 init, and
  any off-LAN reachability.