Operator correction: secrets are kept in source through sops. #55 shipped the WireGuard server private key as a Secret to `kubectl create` by hand. Two real costs, not just a convention break: * R-SEC-01 / spec docs/07 -- secrets live in source, SOPS-encrypted, and reach the cluster only through Flux. * It broke DR for the one credential that cannot be re-derived. A rebuild restores everything in git; it does not restore a hand-made Secret, and losing this key means re-enrolling EVERY device, because each client profile pins its public half. #55 papered over that with "keep a copy in your password manager". (Note secret-provenance did NOT forbid the original shape -- it scopes to app-namespace=true and vpn-ingress is deliberately a platform namespace. The rule broken was the written requirement, not an enforced control.) An agent still must not mint a long-lived private key (CLAUDE.md §6), so the file is committed ALREADY ENCRYPTED with `privatekey: REPLACE-ME`. That makes the wiring complete and green from the first commit -- resource line plus both overlay decryption patches -- leaving the operator exactly one act: replace the value. A commented-out resource to uncomment later is the "in git, no generator entry, reviewed, merged, still broken" shape this repo keeps hitting. The init guard is now a validation rather than an emptiness check: the value is piped through `wg pubkey`, which accepts only a real curve25519 private key, and the pod names which case it hit -- placeholder, empty, mangled, or (the silent one) still `ENC[` ciphertext because an overlay is missing its decryption patch, which otherwise applies cleanly with every Flux condition green. Second, independent finding that also fixes #55's step 2: `wg` is on the HOST only and `sops` is in the CONTAINER only, so `sops -d | wg pubkey` cannot run in one shell -- the command #55 documented could not have worked. Every step is now split by where it runs, and the written key is verified by comparing sha256 digests rather than printing anything. `qrencode` is on neither, so the QR step says so and gives the install command. Touches clusters/** and adds a **/*.sops.yaml, so approval-policy will block pending the operator's /approve-override. Deliberately not posted by me. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
11 KiB
date, branch, title
| date | branch | title |
|---|---|---|
| 2026-10-05 | fix/vpn-server-key-in-sops | VPN server key moves into SOPS-encrypted git — the out-of-band Secret broke R-SEC-01 and the DR story |
What changed
Operator correction (2026-10-05): "secrets are kept in source through sops". They are right, and
#55 shipped the WireGuard server private key the wrong way: as a Secret the operator was told to
kubectl create out of band, referenced by name only. This branch moves it into the tree as
platform/vpn-ingress/wg-server-key.sops.yaml, where every other secret in this repo lives.
Why the original choice was wrong, not merely unconventional. Two concrete costs, both of which #55's own README half-admitted and then shipped anyway:
- R-SEC-01 / spec docs/07 says secrets live in source, SOPS-encrypted, reaching the cluster only through Flux. An out-of-band Secret is outside the GitOps tree by construction.
- It broke DR for the one credential that cannot be re-derived. A cluster rebuild restores everything in git; it does not restore a hand-made Secret. Losing this key means re-enrolling every device, because each client profile pins its public half. #55 papered over this with "keep a copy in your password manager", which is a workaround for a self-inflicted gap.
Note platform/kyverno/admission/secret-provenance.yaml did not forbid the original shape —
it scopes to app-namespace=true namespaces and vpn-ingress is deliberately a platform namespace,
so admission would have accepted the hand-applied Secret. The rule it violated was the written
requirement, not an enforced control. Recording that because "the gate let it through" was never
evidence it was correct.
How the unfillable-key problem is solved. An agent still must not mint a long-lived private key
(CLAUDE.md §6), so the file is committed already encrypted, with privatekey: REPLACE-ME. That
makes the wiring complete and the gate green from the first commit — the resource line and both
overlay decryption patches are present — and leaves the operator exactly one act: replace the value.
The alternative, a commented-out resource line to uncomment later, is precisely the "file in git with
no generator entry: reviewed, merged, apparently done, still broken" shape this repo keeps getting
bitten by (alert-runbooks-resolvable exists because of it).
The init-container guard is now a validation, not an emptiness check. It was [ -z "$key" ].
Because the committed placeholder is non-empty, the likely failure is now a value that is present
and wrong, so the guard pipes the value through wg pubkey — which accepts only a real 32-byte
curve25519 private key — and names which of four cases it hit:
| case | why it matters |
|---|---|
REPLACE-ME placeholder |
the expected fresh-install state |
| empty | blanked value |
starts with ENC[ |
the Secret reached the cluster UNDECRYPTED. A missing overlay decryption patch applies cleanly and leaves every Flux condition green — this is the silent failure the guard exists for |
| rejected, none of the above | mangled value, or a public key pasted by mistake |
Files:
platform/vpn-ingress/wg-server-key.sops.yaml(new) — SOPS-encrypted to the repo's age recipient,stringData.privatekeyonly, placeholder value.platform/vpn-ingress/wg-server-key.sops.yaml.example(new) — plaintext template + operator hand-off, following thecrafty-oauth2-proxy-credentials.sops.yaml.exampleconvention.platform/vpn-ingress/kustomization.yaml— the resource added; the "DELIBERATELY ABSENT: a Secret" block replaced with the overlay-patch warning.platform/vpn-ingress/daemonset.yaml— thewg pubkeyguard; volume comment rewritten.clusters/dev/kustomization.yaml,clusters/prod/kustomization.yaml— thevpn-ingressdecryption:patch, in both. This is the load-bearing half: #55's own kustomization comment predicted that forgetting it applies ciphertext cleanly.clusters/base/vpn-ingress.yaml— its "NO decryption stanza is needed" comment was now false.platform/vpn-ingress/README.md,runbooks/vpn-remote-access.md— step 2 rewritten.infra/vpn-ingress/new-peer.sh— stale "step 1" pointer.
A second finding, which changes the procedure independently of SOPS: wg and sops are not in
the same place. Measured: wg is on the host only (/usr/bin/wg, absent from the dev
container); sops is in the container only (/usr/local/bin/sops, absent from the host). So
sops -d … | wg pubkey cannot be run in one shell, and #55's step 2 — which assumed it could —
would have failed. The runbook and the .example now split every command by where it runs, and
verify the written key by comparing sha256 digests of the decrypted value and the operator's
local copy, which never puts key material on a terminal. qrencode is on neither, so the QR
instruction now says so and gives the install command.
SENSITIVE PATHS — this PR touches clusters/** (two overlays) AND adds a **/*.sops.yaml, so
ci/approval-policy.sh will BLOCK it. It needs the operator's /approve-override <reason> as a
PR review of type "Comment". I have deliberately not posted it (CLAUDE.md §1).
What I verified
- The encrypted file is real and correctly targeted.
sops -e -iin the pinned toolbox producedstringData.privatekey: ENC[AES256_GCM,...]with cleartext keys,encrypted_regex ^(data|stringData)$, and exactly one age recipient:age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg— the recipient in the root.sops.yaml, i.e. the key Flux holds assops-age. - Round-trip, without exposing anything:
sops -dthen a Python read of key names only →stringData keys: ['privatekey'], andvalue is still the placeholder: True. - Both overlay patches are present and name the right Kustomization (
name: vpn-ingress,namespace: flux-system,decryption.secretRef.name: sops-age), inserted after each overlay'svelerostanza, mirroring headlamp/crafty/velero. kubectl kustomize platform/vpn-ingressbuilds and now emits five objects — Namespace, ConfigMap, Secret, DaemonSet, NetworkPolicy — with${WG_CLIENT_CIDR}still an unexpanded token for Flux.wg/sops/qrencodelocations, measured on both host and container, as described above.- Toolbox gate from an isolated worktree: rc 3, with
spec-immutability,decisions-log,lint,l0,render-golden,l1andapproval-policyall pass; rc 3 is the two environmental skips (gitea-settings— no local token; conformance L2/L3 — no cluster). Not green. Reported as rc 3, not as pass. - The L0 gate caught THREE real defects in this branch before it could merge — including one
in the fix for another — and I had asserted one of them was fine without running it. Both are recorded because the gate's value here is the
whole point of the exercise:
no-plaintext-secretsFAILED: "wg-server-key.sops.yaml.example: an unencrypted .example template is referenced by a kustomization — it would be applied as a plaintext Secret". My kustomization named the template in a comment, and the check matches comments too (correctly: naming one is how a plaintext Secret gets applied by accident).platform/crafty/kustomization.yamlnever names its own example either — the convention was there to copy and I missed it. Fixed by not naming the file; the hand-off lives inside the example, where it belongs. An earlier draft of this entry claimed this check passed. It did not. The claim was written from expectation rather than from a run.flux-postbuild-substitutionFAILED on${key#ENC[}in my new guard (daemonset.yaml:155), flagged[SILENT-EMPTY]: envsubst consumes every${...}, so my shell prefix-strip would have rendered to the empty string — the ciphertext branch of the guard would never have fired, which is precisely the silent failure the guard was added to catch. Rewritten withcase, which uses no braces and so has nothing to escape.- And then it failed a THIRD time, on the fix's own comment —
[FATAL-PARSE] daemonset.yaml:156 ${...}, the prose explaining the rule, written with the literal form. That class is worse than it looks: aFATAL-PARSEhazard fails the entire Kustomization, every resource in the path, not just the line. Rewritten as "dollar-brace expansion". This is the second time in this component that writing the hazard literally inside a comment about the hazard has been the defect (the first was in #55), so the habit to build is: in any substituted path, describe the construct in words, never type it.
- After both fixes,
no-plaintext-secretsandflux-postbuild-substitutionpass (446 and 198 assertions respectively), andl0is green.
What I did NOT verify
- That Flux decrypts this file on prod. The overlay patches have never reconciled. This is the
whole risk of the change and it has exactly one observable outcome: after merge the pods should
CrashLoop reporting the
REPLACE-MEplaceholder (proving decryption worked), not reportingENC[(which would prove a patch is wrong). I built the guard to tell those apart because I cannot test them. - That
wg pubkeyaccepts the operator's real key through this path. The guard is reasoned fromwg pubkey's behaviour and has never run —wgis absent from both this workstation's container and the gate, so the first execution will be in the pod. - The
sops setcommand in the hand-off. I verifiedsops -e -iandsops -din the toolbox; I did not runsops setwith the quoting shown, and nested quoting insidedev.sh run bash -cis exactly where a documented command tends to break. If it misbehaves, the editor form (sops <file>) is the fallback and does not depend on quoting. - The sha256 comparison in step 2 step 4 is untested end to end (it needs the real key).
- Whether anything else in #55 assumed a tool is available where it is not. I checked
wg,sopsandqrencodebecause step 2 depended on them. I did not audit every command in the runbook the same way — and that class of defect has now produced two corrections in a row (site.sh, thensops | wg), so assume more remain until each step has actually been run. - The live VPN still carries no traffic, and the earlier gaps stand unchanged: the
iptables-vs-nftables question,
allowPrivilegeEscalation: falseversus the image's s6 init, and any off-LAN reachability.