Files
home-network-k3s-impl/decisions/2026-10-05-vpn-server-key-in-sops.md
T
jakman85andClaude Opus 5 e35612dc7e
gate / gate (pull_request) Failing after 1m51s
gate / conformance (pull_request) Has been skipped
fix(vpn-ingress): keep the server key in source through SOPS, not as an out-of-band Secret
Operator correction: secrets are kept in source through sops. #55 shipped the
WireGuard server private key as a Secret to `kubectl create` by hand. Two real
costs, not just a convention break:

  * R-SEC-01 / spec docs/07 -- secrets live in source, SOPS-encrypted, and
    reach the cluster only through Flux.
  * It broke DR for the one credential that cannot be re-derived. A rebuild
    restores everything in git; it does not restore a hand-made Secret, and
    losing this key means re-enrolling EVERY device, because each client
    profile pins its public half. #55 papered over that with "keep a copy in
    your password manager".

(Note secret-provenance did NOT forbid the original shape -- it scopes to
app-namespace=true and vpn-ingress is deliberately a platform namespace. The
rule broken was the written requirement, not an enforced control.)

An agent still must not mint a long-lived private key (CLAUDE.md §6), so the
file is committed ALREADY ENCRYPTED with `privatekey: REPLACE-ME`. That makes
the wiring complete and green from the first commit -- resource line plus both
overlay decryption patches -- leaving the operator exactly one act: replace the
value. A commented-out resource to uncomment later is the "in git, no generator
entry, reviewed, merged, still broken" shape this repo keeps hitting.

The init guard is now a validation rather than an emptiness check: the value is
piped through `wg pubkey`, which accepts only a real curve25519 private key, and
the pod names which case it hit -- placeholder, empty, mangled, or (the silent
one) still `ENC[` ciphertext because an overlay is missing its decryption patch,
which otherwise applies cleanly with every Flux condition green.

Second, independent finding that also fixes #55's step 2: `wg` is on the HOST
only and `sops` is in the CONTAINER only, so `sops -d | wg pubkey` cannot run in
one shell -- the command #55 documented could not have worked. Every step is now
split by where it runs, and the written key is verified by comparing sha256
digests rather than printing anything. `qrencode` is on neither, so the QR step
says so and gives the install command.

Touches clusters/** and adds a **/*.sops.yaml, so approval-policy will block
pending the operator's /approve-override. Deliberately not posted by me.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-05 23:55:29 -05:00

11 KiB

date, branch, title
date branch title
2026-10-05 fix/vpn-server-key-in-sops VPN server key moves into SOPS-encrypted git — the out-of-band Secret broke R-SEC-01 and the DR story

What changed

Operator correction (2026-10-05): "secrets are kept in source through sops". They are right, and #55 shipped the WireGuard server private key the wrong way: as a Secret the operator was told to kubectl create out of band, referenced by name only. This branch moves it into the tree as platform/vpn-ingress/wg-server-key.sops.yaml, where every other secret in this repo lives.

Why the original choice was wrong, not merely unconventional. Two concrete costs, both of which #55's own README half-admitted and then shipped anyway:

  1. R-SEC-01 / spec docs/07 says secrets live in source, SOPS-encrypted, reaching the cluster only through Flux. An out-of-band Secret is outside the GitOps tree by construction.
  2. It broke DR for the one credential that cannot be re-derived. A cluster rebuild restores everything in git; it does not restore a hand-made Secret. Losing this key means re-enrolling every device, because each client profile pins its public half. #55 papered over this with "keep a copy in your password manager", which is a workaround for a self-inflicted gap.

Note platform/kyverno/admission/secret-provenance.yaml did not forbid the original shape — it scopes to app-namespace=true namespaces and vpn-ingress is deliberately a platform namespace, so admission would have accepted the hand-applied Secret. The rule it violated was the written requirement, not an enforced control. Recording that because "the gate let it through" was never evidence it was correct.

How the unfillable-key problem is solved. An agent still must not mint a long-lived private key (CLAUDE.md §6), so the file is committed already encrypted, with privatekey: REPLACE-ME. That makes the wiring complete and the gate green from the first commit — the resource line and both overlay decryption patches are present — and leaves the operator exactly one act: replace the value. The alternative, a commented-out resource line to uncomment later, is precisely the "file in git with no generator entry: reviewed, merged, apparently done, still broken" shape this repo keeps getting bitten by (alert-runbooks-resolvable exists because of it).

The init-container guard is now a validation, not an emptiness check. It was [ -z "$key" ]. Because the committed placeholder is non-empty, the likely failure is now a value that is present and wrong, so the guard pipes the value through wg pubkey — which accepts only a real 32-byte curve25519 private key — and names which of four cases it hit:

case why it matters
REPLACE-ME placeholder the expected fresh-install state
empty blanked value
starts with ENC[ the Secret reached the cluster UNDECRYPTED. A missing overlay decryption patch applies cleanly and leaves every Flux condition green — this is the silent failure the guard exists for
rejected, none of the above mangled value, or a public key pasted by mistake

Files:

  • platform/vpn-ingress/wg-server-key.sops.yaml (new) — SOPS-encrypted to the repo's age recipient, stringData.privatekey only, placeholder value.
  • platform/vpn-ingress/wg-server-key.sops.yaml.example (new) — plaintext template + operator hand-off, following the crafty-oauth2-proxy-credentials.sops.yaml.example convention.
  • platform/vpn-ingress/kustomization.yaml — the resource added; the "DELIBERATELY ABSENT: a Secret" block replaced with the overlay-patch warning.
  • platform/vpn-ingress/daemonset.yaml — the wg pubkey guard; volume comment rewritten.
  • clusters/dev/kustomization.yaml, clusters/prod/kustomization.yaml — the vpn-ingress decryption: patch, in both. This is the load-bearing half: #55's own kustomization comment predicted that forgetting it applies ciphertext cleanly.
  • clusters/base/vpn-ingress.yaml — its "NO decryption stanza is needed" comment was now false.
  • platform/vpn-ingress/README.md, runbooks/vpn-remote-access.md — step 2 rewritten.
  • infra/vpn-ingress/new-peer.sh — stale "step 1" pointer.

A second finding, which changes the procedure independently of SOPS: wg and sops are not in the same place. Measured: wg is on the host only (/usr/bin/wg, absent from the dev container); sops is in the container only (/usr/local/bin/sops, absent from the host). So sops -d … | wg pubkey cannot be run in one shell, and #55's step 2 — which assumed it could — would have failed. The runbook and the .example now split every command by where it runs, and verify the written key by comparing sha256 digests of the decrypted value and the operator's local copy, which never puts key material on a terminal. qrencode is on neither, so the QR instruction now says so and gives the install command.

SENSITIVE PATHS — this PR touches clusters/** (two overlays) AND adds a **/*.sops.yaml, so ci/approval-policy.sh will BLOCK it. It needs the operator's /approve-override <reason> as a PR review of type "Comment". I have deliberately not posted it (CLAUDE.md §1).

What I verified

  • The encrypted file is real and correctly targeted. sops -e -i in the pinned toolbox produced stringData.privatekey: ENC[AES256_GCM,...] with cleartext keys, encrypted_regex ^(data|stringData)$, and exactly one age recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg — the recipient in the root .sops.yaml, i.e. the key Flux holds as sops-age.
  • Round-trip, without exposing anything: sops -d then a Python read of key names only → stringData keys: ['privatekey'], and value is still the placeholder: True.
  • Both overlay patches are present and name the right Kustomization (name: vpn-ingress, namespace: flux-system, decryption.secretRef.name: sops-age), inserted after each overlay's velero stanza, mirroring headlamp/crafty/velero.
  • kubectl kustomize platform/vpn-ingress builds and now emits five objects — Namespace, ConfigMap, Secret, DaemonSet, NetworkPolicy — with ${WG_CLIENT_CIDR} still an unexpanded token for Flux.
  • wg / sops / qrencode locations, measured on both host and container, as described above.
  • Toolbox gate from an isolated worktree: rc 3, with spec-immutability, decisions-log, lint, l0, render-golden, l1 and approval-policy all pass; rc 3 is the two environmental skips (gitea-settings — no local token; conformance L2/L3 — no cluster). Not green. Reported as rc 3, not as pass.
  • The L0 gate caught THREE real defects in this branch before it could merge — including one in the fix for another — and I had asserted one of them was fine without running it. Both are recorded because the gate's value here is the whole point of the exercise:
    1. no-plaintext-secrets FAILED: "wg-server-key.sops.yaml.example: an unencrypted .example template is referenced by a kustomization — it would be applied as a plaintext Secret". My kustomization named the template in a comment, and the check matches comments too (correctly: naming one is how a plaintext Secret gets applied by accident). platform/crafty/kustomization.yaml never names its own example either — the convention was there to copy and I missed it. Fixed by not naming the file; the hand-off lives inside the example, where it belongs. An earlier draft of this entry claimed this check passed. It did not. The claim was written from expectation rather than from a run.
    2. flux-postbuild-substitution FAILED on ${key#ENC[} in my new guard (daemonset.yaml:155), flagged [SILENT-EMPTY]: envsubst consumes every ${...}, so my shell prefix-strip would have rendered to the empty string — the ciphertext branch of the guard would never have fired, which is precisely the silent failure the guard was added to catch. Rewritten with case, which uses no braces and so has nothing to escape.
    3. And then it failed a THIRD time, on the fix's own comment — [FATAL-PARSE] daemonset.yaml:156 ${...}, the prose explaining the rule, written with the literal form. That class is worse than it looks: a FATAL-PARSE hazard fails the entire Kustomization, every resource in the path, not just the line. Rewritten as "dollar-brace expansion". This is the second time in this component that writing the hazard literally inside a comment about the hazard has been the defect (the first was in #55), so the habit to build is: in any substituted path, describe the construct in words, never type it.
  • After both fixes, no-plaintext-secrets and flux-postbuild-substitution pass (446 and 198 assertions respectively), and l0 is green.

What I did NOT verify

  • That Flux decrypts this file on prod. The overlay patches have never reconciled. This is the whole risk of the change and it has exactly one observable outcome: after merge the pods should CrashLoop reporting the REPLACE-ME placeholder (proving decryption worked), not reporting ENC[ (which would prove a patch is wrong). I built the guard to tell those apart because I cannot test them.
  • That wg pubkey accepts the operator's real key through this path. The guard is reasoned from wg pubkey's behaviour and has never run — wg is absent from both this workstation's container and the gate, so the first execution will be in the pod.
  • The sops set command in the hand-off. I verified sops -e -i and sops -d in the toolbox; I did not run sops set with the quoting shown, and nested quoting inside dev.sh run bash -c is exactly where a documented command tends to break. If it misbehaves, the editor form (sops <file>) is the fallback and does not depend on quoting.
  • The sha256 comparison in step 2 step 4 is untested end to end (it needs the real key).
  • Whether anything else in #55 assumed a tool is available where it is not. I checked wg, sops and qrencode because step 2 depended on them. I did not audit every command in the runbook the same way — and that class of defect has now produced two corrections in a row (site.sh, then sops | wg), so assume more remain until each step has actually been run.
  • The live VPN still carries no traffic, and the earlier gaps stand unchanged: the iptables-vs-nftables question, allowPrivilegeEscalation: false versus the image's s6 init, and any off-LAN reachability.