home-network-k3s-impl/decisions/2026-10-05-vpn-server-key-in-sops.md

date, branch, title

date	branch	title
2026-10-05	fix/vpn-server-key-in-sops	VPN server key moves into SOPS-encrypted git — the out-of-band Secret broke R-SEC-01 and the DR story

What changed

Operator correction (2026-10-05): "secrets are kept in source through sops". They are right, and #55 shipped the WireGuard server private key the wrong way: as a Secret the operator was told to kubectl create out of band, referenced by name only. This branch moves it into the tree as platform/vpn-ingress/wg-server-key.sops.yaml, where every other secret in this repo lives.

Why the original choice was wrong, not merely unconventional. Two concrete costs, both of which #55's own README half-admitted and then shipped anyway:

1. R-SEC-01 / spec docs/07 says secrets live in source, SOPS-encrypted, reaching the cluster only through Flux. An out-of-band Secret is outside the GitOps tree by construction.
2. It broke DR for the one credential that cannot be re-derived. A cluster rebuild restores everything in git; it does not restore a hand-made Secret. Losing this key means re-enrolling every device, because each client profile pins its public half. #55 papered over this with "keep a copy in your password manager", which is a workaround for a self-inflicted gap.

Note platform/kyverno/admission/secret-provenance.yaml did not forbid the original shape — it scopes to app-namespace=true namespaces and vpn-ingress is deliberately a platform namespace, so admission would have accepted the hand-applied Secret. The rule it violated was the written requirement, not an enforced control. Recording that because "the gate let it through" was never evidence it was correct.

How the unfillable-key problem is solved. An agent still must not mint a long-lived private key (CLAUDE.md §6), so the file is committed already encrypted, with privatekey: REPLACE-ME. That makes the wiring complete and the gate green from the first commit — the resource line and both overlay decryption patches are present — and leaves the operator exactly one act: replace the value. The alternative, a commented-out resource line to uncomment later, is precisely the "file in git with no generator entry: reviewed, merged, apparently done, still broken" shape this repo keeps getting bitten by (alert-runbooks-resolvable exists because of it).

The init-container guard is now a validation, not an emptiness check. It was [ -z "$key" ]. Because the committed placeholder is non-empty, the likely failure is now a value that is present and wrong, so the guard pipes the value through wg pubkey — which accepts only a real 32-byte curve25519 private key — and names which of four cases it hit:

case	why it matters
REPLACE-ME placeholder	the expected fresh-install state
empty	blanked value
starts with ENC[	the Secret reached the cluster UNDECRYPTED. A missing overlay decryption patch applies cleanly and leaves every Flux condition green — this is the silent failure the guard exists for
rejected, none of the above	mangled value, or a public key pasted by mistake

Files:

• platform/vpn-ingress/wg-server-key.sops.yaml (new) — SOPS-encrypted to the repo's age recipient, stringData.privatekey only, placeholder value.
• platform/vpn-ingress/wg-server-key.sops.yaml.example (new) — plaintext template + operator hand-off, following the crafty-oauth2-proxy-credentials.sops.yaml.example convention.
• platform/vpn-ingress/kustomization.yaml — the resource added; the "DELIBERATELY ABSENT: a Secret" block replaced with the overlay-patch warning.
• platform/vpn-ingress/daemonset.yaml — the wg pubkey guard; volume comment rewritten.
• clusters/dev/kustomization.yaml, clusters/prod/kustomization.yaml — the vpn-ingress decryption: patch, in both. This is the load-bearing half: #55's own kustomization comment predicted that forgetting it applies ciphertext cleanly.
• clusters/base/vpn-ingress.yaml — its "NO decryption stanza is needed" comment was now false.
• platform/vpn-ingress/README.md, runbooks/vpn-remote-access.md — step 2 rewritten.
• infra/vpn-ingress/new-peer.sh — stale "step 1" pointer.

A second finding: wg and sops were not in the same place — and the first fix for that was also wrong. Measured: wg existed only on the host (/usr/bin/wg), sops only in the dev container, qrencode on neither. So sops -d … | wg pubkey could not run in one shell, and #55's step 2 — which assumed it could — would have failed.

My first fix was to split every step by where it runs, and to tell the operator to dnf install qrencode on their workstation. Operator correction: "sop, wg, and any other tool requirement is supposed to be baked into the devcontainer." That is CLAUDE.md §5 and the devcontainer README's "Don't install the toolchain on the host", and the split fix broke the invariant rather than honouring it — it also produced a procedure that could never be reproduced in CI, and which no one should have to follow.

The real fix: wireguard-tools and qrencode are now distro packages in infra/devcontainer/Containerfile, alongside ansible/openssh/diffutils. Both are in Fedora 44's repos (wireguard-tools 1.0.20260223 — the same upstream version as the container image this component runs — and qrencode 4.1.1), unpinned like every other dnf package there, because the pinned thing is the Fedora base and neither is a gate tool whose version changes what green means.

Three things got simpler rather than merely relocated, which is the tell that the operator's rule was the right one:

1. Step 2 is now one pipeline in one shell: sops set … "\"$(wg genkey)\"" generates the key straight into the already-encrypted document. The plaintext never lands on disk, never reaches a terminal, and there is no staged copy to shred. The previous version needed a copy in .cache/server.key, a git check-ignore warning about the *.key suffix being the only thing protecting it, and a sha256 comparison to prove the right bytes arrived. All of that is gone.
2. new-peer.sh now derives the server's public key from wg-server-key.sops.yaml itself (sops -d --extract … | wg pubkey), so there is no local key copy to keep in step with the encrypted one. It previously required the operator to keep the private key at a magic host path.
3. .cache/ is now gitignored at the repo root (mirroring infra/ansible/.gitignore), so the container has a persistent, non-committable scratch directory. new-peer.sh writes client profiles to .cache/wg-peers/ by default — necessary because the repo bind-mount is the container's ONLY persistent read-write mount, so a profile written to /tmp in there is gone before it can be moved to a desktop. Its old blanket "refuse to write inside the checkout" guard is replaced by git check-ignore, which tests the actual property instead of a proxy.

SENSITIVE PATHS — this PR touches clusters/** (two overlays) AND adds a **/*.sops.yaml, so ci/approval-policy.sh will BLOCK it. It needs the operator's /approve-override <reason> as a PR review of type "Comment". I have deliberately not posted it (CLAUDE.md §1).

What I verified

• The encrypted file is real and correctly targeted. sops -e -i in the pinned toolbox produced stringData.privatekey: ENC[AES256_GCM,...] with cleartext keys, encrypted_regex ^(data|stringData)$, and exactly one age recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg — the recipient in the root .sops.yaml, i.e. the key Flux holds as sops-age.
• Round-trip, without exposing anything: sops -d then a Python read of key names only → stringData keys: ['privatekey'], and value is still the placeholder: True.
• Both overlay patches are present and name the right Kustomization (name: vpn-ingress, namespace: flux-system, decryption.secretRef.name: sops-age), inserted after each overlay's velero stanza, mirroring headlamp/crafty/velero.
• kubectl kustomize platform/vpn-ingress builds and now emits five objects — Namespace, ConfigMap, Secret, DaemonSet, NetworkPolicy — with ${WG_CLIENT_CIDR} still an unexpanded token for Flux.
• The tool gap, measured on both host and container before the fix: wg host-only, sops container-only, qrencode on neither.
• Both packages exist in the image's repos, queried from inside the running container: wireguard-tools 1.0.20260223 (repo updates) and qrencode 4.1.1 (repo fedora).
• The image REBUILDS with them and both tools resolve inside it — ./infra/devcontainer/dev.sh build succeeded, and wg --version / qrencode --version / sops --version all answer in the rebuilt container. This is the claim the change rests on, so it was executed rather than reasoned.
• new-peer.sh's new SOPS path fails correctly against the placeholder. Run in the rebuilt container it refused with "could not derive the server public key … it still holds the REPLACE-ME placeholder", which is both the expected pre-key behaviour and a live test of the sops -d | wg pubkey pipeline the whole design now depends on.
• .cache/ is ignored regardless of filename now: git check-ignore -v .cache/anything.conf and .cache/serverkey.txt both match .gitignore:46:.cache/. Before this change only *.key matched, which is the fragility the previous runbook had to warn about.
• Toolbox gate from an isolated worktree: rc 3, with spec-immutability, decisions-log, lint, l0, render-golden, l1 and approval-policy all pass; rc 3 is the two environmental skips (gitea-settings — no local token; conformance L2/L3 — no cluster). Not green. Reported as rc 3, not as pass.
• The L0 gate caught THREE real defects in this branch before it could merge — including one in the fix for another — and I had asserted one of them was fine without running it. Both are recorded because the gate's value here is the whole point of the exercise:
  1. no-plaintext-secrets FAILED: "wg-server-key.sops.yaml.example: an unencrypted .example template is referenced by a kustomization — it would be applied as a plaintext Secret". My kustomization named the template in a comment, and the check matches comments too (correctly: naming one is how a plaintext Secret gets applied by accident). platform/crafty/kustomization.yaml never names its own example either — the convention was there to copy and I missed it. Fixed by not naming the file; the hand-off lives inside the example, where it belongs. An earlier draft of this entry claimed this check passed. It did not. The claim was written from expectation rather than from a run.
  2. flux-postbuild-substitution FAILED on ${key#ENC[} in my new guard (daemonset.yaml:155), flagged [SILENT-EMPTY]: envsubst consumes every ${...}, so my shell prefix-strip would have rendered to the empty string — the ciphertext branch of the guard would never have fired, which is precisely the silent failure the guard was added to catch. Rewritten with case, which uses no braces and so has nothing to escape.
  3. And then it failed a THIRD time, on the fix's own comment — [FATAL-PARSE] daemonset.yaml:156 ${...}, the prose explaining the rule, written with the literal form. That class is worse than it looks: a FATAL-PARSE hazard fails the entire Kustomization, every resource in the path, not just the line. Rewritten as "dollar-brace expansion". This is the second time in this component that writing the hazard literally inside a comment about the hazard has been the defect (the first was in #55), so the habit to build is: in any substituted path, describe the construct in words, never type it.
• After both fixes, no-plaintext-secrets and flux-postbuild-substitution pass (446 and 198 assertions respectively), and l0 is green.

What I did NOT verify

• That Flux decrypts this file on prod. The overlay patches have never reconciled. This is the whole risk of the change and it has exactly one observable outcome: after merge the pods should CrashLoop reporting the REPLACE-ME placeholder (proving decryption worked), not reporting ENC[ (which would prove a patch is wrong). I built the guard to tell those apart because I cannot test them.
• That wg pubkey accepts the operator's real key through this path. The guard is reasoned from wg pubkey's behaviour and has never run — wg is absent from both this workstation's container and the gate, so the first execution will be in the pod.
• The sops set command in the hand-off. I verified sops -e -i and sops -d in the toolbox; I did not run sops set with the quoting shown, and nested quoting inside dev.sh run bash -c is exactly where a documented command tends to break. If it misbehaves, the editor form (sops <file>) is the fallback and does not depend on quoting.
• The sha256 comparison in step 2 step 4 is untested end to end (it needs the real key).
• Whether anything else in #55 assumed a tool is available where it is not. I checked wg, sops and qrencode because step 2 depended on them. I did not audit every command in the runbook the same way — and that class of defect has now produced two corrections in a row (site.sh, then sops | wg), so assume more remain until each step has actually been run.
• The live VPN still carries no traffic, and the earlier gaps stand unchanged: the iptables-vs-nftables question, allowPrivilegeEscalation: false versus the image's s6 init, and any off-LAN reachability.