Merge pull request 'fix(vpn-ingress): keep the server key in source through SOPS, not as an out-of-band Secret' (#60) from fix/vpn-server-key-in-sops into main
Reviewed-on: #60 Reviewed-by: Keith Jackson <[email protected]>
This commit was merged in pull request #60.
This commit is contained in:
jakman85 committed
commit
4791653f2b
14 files changed
+584
-90
No files matched your search
@@ -14,7 +14,7 @@ egress out the house WAN.
|
||||
| Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node |
|
||||
| Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices |
|
||||
| Authorisation | a device's **public key** is in [`peers.yaml`](./peers.yaml), or it cannot connect |
|
||||
| The one secret | the server's private key, Secret `wg-server-key`, **created out of band** |
|
||||
| The one secret | the server's private key — **SOPS-encrypted in this tree**, `wg-server-key.sops.yaml` |
|
||||
| Exposure | one hand-made WAN port-forward on the SOHO router, off-repo |
|
||||
| Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) |
|
||||
| Procedure / bring-up | [`runbooks/vpn-remote-access.md`](../../runbooks/vpn-remote-access.md) |
|
||||
@@ -130,11 +130,17 @@ translations win here.
|
||||
| half | where | why |
|
||||
|---|---|---|
|
||||
| peer **public** keys + addresses | [`peers.yaml`](./peers.yaml), **plaintext in git** | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
|
||||
| the server's **private** key | Secret `wg-server-key`, **out of band** | the only real secret; an agent must not mint a long-lived private key (CLAUDE.md §6) |
|
||||
| the server's **private** key | `wg-server-key.sops.yaml`, **SOPS-encrypted in git** | the only real secret (R-SEC-01, spec docs/07). Ships with a `REPLACE-ME` placeholder: an agent must not mint a long-lived private key (CLAUDE.md §6), so the operator `sops set`s it in |
|
||||
|
||||
The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** —
|
||||
so the assembled config holding the private key never touches any node's disk and dies with the pod.
|
||||
|
||||
Both halves are therefore in git, which is the point: a cluster rebuild restores the server key with
|
||||
everything else. An out-of-band `kubectl create secret` — the first shape this component shipped in,
|
||||
corrected after review — would have put the one credential every enrolled device depends on *outside*
|
||||
the GitOps tree, and losing it means re-enrolling every device, because each client profile pins this
|
||||
key's public half.
|
||||
|
||||
Encrypting the peer register would have hidden exactly the change a reviewer most needs to see: a
|
||||
new line granting a new device access to the whole LAN.
|
||||
|
||||
@@ -173,8 +179,14 @@ NetworkPolicy applies.
|
||||
(`infra/ansible/roles/node_baseline/defaults/main.yml`); the operator runs the role. The pod has
|
||||
`NET_ADMIN` but deliberately **not** `SYS_MODULE`, so it cannot load the module itself — it
|
||||
CrashLoops with a clear `wg-quick` error instead of quietly degrading to userspace WireGuard.
|
||||
2. **The `wg-server-key` Secret.** Runbook step 1. Until it exists the DaemonSet cannot start —
|
||||
the correct fail-closed state for an unenrolled VPN.
|
||||
2. **The server private key.** `wg-server-key.sops.yaml` is committed, encrypted, with a
|
||||
`REPLACE-ME` placeholder, and both overlays already carry its decryption patch — so the wiring
|
||||
is complete and the only outstanding act is replacing the value (runbook step 2). Until then the
|
||||
DaemonSet CrashLoops with an explicit FATAL naming the file: the init container validates the
|
||||
value with `wg pubkey`, which rejects the placeholder, an empty value, a public key pasted by
|
||||
mistake, **and** a Secret that arrived as undecrypted `ENC[...]` ciphertext. That last case is
|
||||
the one worth having a guard for — a missing overlay decryption patch applies cleanly and leaves
|
||||
every Flux condition green.
|
||||
3. **The WAN port-forward**, udp/51820 → one node address. Hand-made on the SOHO router, off-repo,
|
||||
and the half CI can never check.
|
||||
|
||||
|
||||
@@ -136,9 +136,38 @@ spec:
|
||||
# The key arrives as a Secret file; strip any trailing newline the operator's shell
|
||||
# added, or wg-quick rejects the config with "Key is not the correct length or format".
|
||||
key=$(tr -d ' \r\n' < /run/wg-server-key/privatekey)
|
||||
if [ -z "$key" ]; then
|
||||
echo "FATAL: /run/wg-server-key/privatekey is empty." >&2
|
||||
echo " See runbooks/vpn-remote-access.md step 1 (create the server key)." >&2
|
||||
# VALIDATE, don't just check for empty. The Secret ships committed with a REPLACE-ME
|
||||
# placeholder so the GitOps wiring is complete before the key exists, so the common
|
||||
# failure is a value that is present and WRONG — a placeholder, a public key pasted by
|
||||
# mistake, or a SOPS document that reached the cluster undecrypted (ENC[...], which
|
||||
# happens if an overlay is missing its `vpn-ingress` decryption patch). `wg pubkey`
|
||||
# accepts a value only if it is a real 32-byte curve25519 private key, so it rejects
|
||||
# all four. Without this the bad value would be written into wg0.conf and the pod would
|
||||
# fail later and less legibly, or come up with a server nobody can handshake with.
|
||||
if ! printf '%s' "$key" | wg pubkey >/dev/null 2>&1; then
|
||||
echo "FATAL: the server private key in Secret 'wg-server-key' is not a valid" >&2
|
||||
echo " WireGuard private key (wg pubkey rejected it)." >&2
|
||||
if [ -z "$key" ]; then
|
||||
echo " It is EMPTY." >&2
|
||||
elif [ "$key" = "REPLACE-ME" ]; then
|
||||
echo " It is still the committed REPLACE-ME PLACEHOLDER — the key has not" >&2
|
||||
echo " been generated yet. This is the expected state on a fresh install." >&2
|
||||
else
|
||||
# `case`, NOT a dollar-brace prefix-strip. This manifest is rendered through
|
||||
# envsubst, which consumes every dollar-brace expansion it sees, so a shell
|
||||
# parameter expansion here renders to the empty string and silently disables this
|
||||
# very branch. `case` uses no braces, so there is nothing to escape.
|
||||
# (ci: flux-postbuild-substitution caught both the original expansion AND the
|
||||
# first version of this comment, which spelled the hazard out literally.)
|
||||
case "$key" in
|
||||
ENC\[*)
|
||||
echo " It is still SOPS CIPHERTEXT (starts with ENC[). The Secret" >&2
|
||||
echo " reached the cluster undecrypted — check that this environment's" >&2
|
||||
echo " overlay carries the 'vpn-ingress' decryption patch." >&2
|
||||
;;
|
||||
esac
|
||||
fi
|
||||
echo " Fix: runbooks/vpn-remote-access.md step 2." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
@@ -272,14 +301,22 @@ spec:
|
||||
# during startup rather than at a point anyone would connect to the volume size. Still
|
||||
# counted against the container's 128Mi limit, and still far more than wg0.conf needs.
|
||||
sizeLimit: 8Mi
|
||||
# The ONLY secret in this component. Created OUT OF BAND by the operator and referenced by
|
||||
# name (R-IDP-04) — runbooks/vpn-remote-access.md step 1. It is NOT a *.sops.yaml in this
|
||||
# tree on purpose: an agent must not mint a long-lived private key (CLAUDE.md §6), and a key
|
||||
# that passed through a session transcript would have to be treated as compromised.
|
||||
# DR: Velero's `daily` schedule excludes only `monitoring` and `unifi`, so this namespace
|
||||
# and this Secret ARE captured off-cluster (platform/velero/helmrelease.yaml). Keep a copy
|
||||
# in the operator's password manager as well — restoring it is what makes every already
|
||||
# enrolled client work again without re-enrolling.
|
||||
# The ONLY secret in this component. SOPS-encrypted IN THIS TREE (R-SEC-01, spec docs/07) as
|
||||
# wg-server-key.sops.yaml, delivered by Flux, referenced here by name only (R-IDP-04).
|
||||
#
|
||||
# It ships committed with a REPLACE-ME placeholder: an agent must not mint a long-lived
|
||||
# private key (CLAUDE.md §6), so the operator generates it and `sops set`s it in
|
||||
# (runbooks/vpn-remote-access.md step 2). Shipping the encrypted placeholder rather than no
|
||||
# file at all is what lets the resource line and both overlay decryption patches be present
|
||||
# and GREEN from the first commit — the alternative, a commented-out resource to uncomment
|
||||
# later, is exactly the "file in git with no generator entry: reviewed, merged, apparently
|
||||
# done, still broken" shape this repo keeps getting bitten by.
|
||||
#
|
||||
# DR: in git means a cluster rebuild restores it with everything else — the property an
|
||||
# out-of-band `kubectl create secret` would NOT have had, and losing this key means
|
||||
# re-enrolling every device because each client profile pins its public half. Velero also
|
||||
# captures this namespace (its `daily` schedule excludes only `monitoring` and `unifi`).
|
||||
# Keep a password-manager copy too: that is the one that survives losing cluster AND backups.
|
||||
- name: wg-server-key
|
||||
secret:
|
||||
secretName: wg-server-key
|
||||
|
||||
@@ -4,9 +4,9 @@
|
||||
# same placement as crafty/headlamp/homepage/runbooks, for the same reason: a leaf whose pod can be
|
||||
# legitimately unhealthy must never sit in the gating `platform` tier, where one component's
|
||||
# unhealth stalls platform-config and every apps wave behind it (DECISIONS-LOG: "Headlamp gating
|
||||
# stall"). That rationale is unusually concrete here: until the operator creates the `wg-server-key`
|
||||
# Secret (runbooks/vpn-remote-access.md step 1) this DaemonSet CANNOT start, so putting it in the
|
||||
# gating tier would have taken the whole platform down on merge.
|
||||
# stall"). That rationale is unusually concrete here: until the operator replaces the `REPLACE-ME`
|
||||
# placeholder in wg-server-key.sops.yaml (runbooks/vpn-remote-access.md step 2) this DaemonSet
|
||||
# CANNOT start, so putting it in the gating tier would have taken the whole platform down on merge.
|
||||
#
|
||||
# NOT in platform/kustomization.yaml — see clusters/base/vpn-ingress.yaml.
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
@@ -19,12 +19,22 @@ resources:
|
||||
- daemonset.yaml
|
||||
# Namespace default-deny baseline. NOTE: does NOT constrain the hostNetwork DaemonSet — the file
|
||||
# says so at length rather than implying a protection it cannot provide.
|
||||
- networkpolicy.yaml
|
||||
- networkpolicy.yaml # The server's WireGuard private key — the ONE secret here. SOPS-encrypted (R-SEC-01, spec
|
||||
# docs/07), decrypted per-env by the `vpn-ingress` Kustomization, referenced by name only from
|
||||
# daemonset.yaml (R-IDP-04). Its plaintext `.example` sibling carries the operator hand-off --
|
||||
# deliberately NOT named here: conformance/tests/l0-static/no-plaintext-secrets fails any
|
||||
# kustomization that mentions an unencrypted template, because naming one is how a plaintext
|
||||
# Secret gets applied by accident. (It matches comments too. That is correct, not overzealous.)
|
||||
#
|
||||
# SHIPS WITH A `REPLACE-ME` PLACEHOLDER, already encrypted. An agent must not
|
||||
# mint a long-lived private key (CLAUDE.md §6), and the init container validates the value with
|
||||
# `wg pubkey`, so the unfilled state is a loud CrashLoop naming this file, never a server that
|
||||
# quietly cannot handshake.
|
||||
- wg-server-key.sops.yaml
|
||||
#
|
||||
# DELIBERATELY ABSENT: a Secret. The server's private key (`wg-server-key`) is created out of band
|
||||
# by the operator and referenced BY NAME from daemonset.yaml. There is no `*.sops.yaml` here, so
|
||||
# this component needs NO SOPS decryption stanza in clusters/dev or clusters/prod — unlike
|
||||
# crafty/headlamp/velero. If that key is ever migrated into SOPS, BOTH halves must land together:
|
||||
# the file here, and a `decryption:` patch naming the `vpn-ingress` Kustomization in each overlay.
|
||||
# Without the patch, kustomize hands Flux the raw ciphertext and the pod mounts a key full of
|
||||
# ENC[...] — which applies cleanly and fails only at handshake time.
|
||||
# THIS COMPONENT CARRIES AN ENCRYPTED SECRET, so BOTH overlays must patch the `vpn-ingress`
|
||||
# Kustomization with a `decryption:` stanza — clusters/dev and clusters/prod both do, added by the
|
||||
# same change as this file. That patch is as load-bearing as the file itself: without it kustomize
|
||||
# hands Flux the raw ciphertext, the Secret APPLIES CLEANLY full of ENC[...], and nothing in any
|
||||
# Flux condition looks wrong. The init container's `wg pubkey` guard exists partly to turn exactly
|
||||
# that into a named error rather than a silently non-working tunnel.
|
||||
@@ -0,0 +1,34 @@
|
||||
# WireGuard SERVER private key for platform/vpn-ingress — SOPS-ENCRYPTED (R-SEC-01, spec docs/07).
|
||||
# Keys stay cleartext for reviewable diffs; only `stringData` values are encrypted (.sops.yaml
|
||||
# `encrypted_regex`). Plaintext template + operator hand-off: wg-server-key.sops.yaml.example.
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: wg-server-key
|
||||
namespace: vpn-ingress
|
||||
type: Opaque
|
||||
stringData:
|
||||
#ENC[AES256_GCM,data:fUoPkfWvasisdTtwdhjeOVOlmdQkTl6Bp9YxYyji8g6k+ClATds3BnHJWOj2iplaGpT2XaNakq1+D4YyOi7QbFvNJui+e4qRpqcY8hWnazvZIHJYxjxfAfzW,iv:xtKd6oMZQumAQJAsNLHqJB+o0wdQJ+vL1kI1mc9jL7w=,tag:aImqLgbSmhsRI36/t5kpyg==,type:comment]
|
||||
#ENC[AES256_GCM,data:npDsCW6rQleolIpur5gFkX2jSRrIQzBlGaIPs2Schv0YU3q8AanG39z4QqT3ASDmskV5YuH29Bo0ujEAgBf52YIjQCp58PDDETofrVKOJzhHMa7bV+HCxUFE,iv:7E7thyFBleyqZpqJRGApKvIq3ubCFOkshF/mxim3YPc=,tag:joIrhMe2I6kUnx1ouOPdZw==,type:comment]
|
||||
#ENC[AES256_GCM,data:wHUFx09FgP/2+LTyw61EFbKA5UaqfjU7CeGk6Xc5av2y4r4AvQJRefDM5EGlahKecDFEFsf7fSQRFRHtUJqrQTtw+6tcj3ubSCPW4iJvhyJJv0xdb/0puRmcgj+2NfgF,iv:xgDOSDLqMQ1wcI9vrkKqfNMgIDFEusFnnO7Rzq/aOa4=,tag:m1m/fgjkBsG91c+3daqOpA==,type:comment]
|
||||
privatekey: ENC[AES256_GCM,data:irLENWa1Z3IJSQ==,iv:Qu2Yr6F5+M6P8cQRuhB5J7joxCcudUhwoEWDiL941Lo=,tag:ZNwcBZEkswtnkX0vGKfMAQ==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBMays2OXY0VkpRc2FQQmhU
|
||||
Q0owM3N0VkhYR0lYTUdNZWRqeTFIMjNHOVJ3CnJSMnc5NFEyU0IzWTFtUkw2YVFS
|
||||
MU02VFo1TE14YVpZRzNhVjZwRTZwd0UKLS0tIDFaMWF4NUZjZmhRdXRuZFJYNkg3
|
||||
SUUzN1FnMlI4bkkzbVNlMWhUVnJZODAKQ+Iy86CIv/Oj9E9j4E/0y5hDbcctRuqB
|
||||
OR825KRPRaTdF/G2hudE1R1JbM4wGbPLMzv078R4ZXHDPGvDttSEkA==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-10-06T04:20:58Z"
|
||||
mac: ENC[AES256_GCM,data:GpP/zTX4FtJyrlaCS/R0/sAtFbkM0W8A5sqNxb5ZziSgLUXI+oUKYB09igN6vx5cjRNH7rSmXduS/actaT5jhP8kuF1n/iQfG56PsCJHL2vneefzcEOKXeGquAwDOY+iGmNsZ5R8ySrM/Fq7HYLMJasWosyTh3YlBs8GRYiHmU4=,iv:438+pDzHLGyh+OMQCMzGBEBrZUPNp5BbjrrMdiEiqJ8=,tag:JqLu6zZWKtcoB8mp4HSZig==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
@@ -0,0 +1,76 @@
|
||||
# =============================================================================
|
||||
# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
|
||||
# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
|
||||
#
|
||||
# WHAT THIS IS: the WireGuard SERVER's long-term private key for the remote-access VPN
|
||||
# (platform/vpn-ingress). It is the ONE secret in the whole component. The init container in
|
||||
# daemonset.yaml reads it at `/run/wg-server-key/privatekey` and joins it to the PLAINTEXT peer
|
||||
# register (peers.yaml) in a tmpfs, so the assembled wg0.conf never touches a node's disk.
|
||||
#
|
||||
# WHY IT IS IN GIT, SOPS-ENCRYPTED, AND NOT HAND-CREATED. R-SEC-01 / spec docs/07: secrets live in
|
||||
# source, encrypted, and reach the cluster only through Flux. An out-of-band `kubectl create secret`
|
||||
# would put the single credential every enrolled device depends on OUTSIDE the GitOps tree, where a
|
||||
# cluster rebuild silently loses it — and losing it means RE-ENROLLING EVERY DEVICE, because each
|
||||
# client profile pins this key's public half.
|
||||
#
|
||||
# *** THE REAL FILE IS COMMITTED WITH `privatekey: REPLACE-ME`, ALREADY ENCRYPTED. ***
|
||||
# That is deliberate, so the wiring is complete and green before the key exists: the resource is
|
||||
# already listed in kustomization.yaml and both overlays already carry the `vpn-ingress` decryption
|
||||
# patch. Your job is ONLY to replace the value (step 3 below). Until you do, the DaemonSet
|
||||
# CrashLoops with an explicit FATAL naming this file — the init container validates the value with
|
||||
# `wg pubkey` before writing wg0.conf, so a placeholder fails CLOSED and LOUDLY rather than
|
||||
# producing a server that silently cannot complete a handshake.
|
||||
#
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# EVERYTHING RUNS IN THE DEV CONTAINER. `wg`, `sops` and `qrencode` are all baked into the image
|
||||
# (infra/devcontainer/Containerfile) because that image IS the toolchain — CLAUDE.md §5, and the
|
||||
# devcontainer README's "Don't install the toolchain on the host". One shell, one toolchain, so
|
||||
# `wg genkey | sops set` is just a pipeline.
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
#
|
||||
# HAND-OFF STEPS (operator — an agent must not mint a long-lived private key, CLAUDE.md §6).
|
||||
# Full procedure with verification: runbooks/vpn-remote-access.md step 2.
|
||||
#
|
||||
# 1. Generate the key STRAIGHT INTO this file. The plaintext never lands on disk and never
|
||||
# reaches a terminal; `sops set` edits the already-encrypted document in place:
|
||||
# ./infra/devcontainer/dev.sh run bash -c '
|
||||
# cd /work
|
||||
# umask 077
|
||||
# sops set platform/vpn-ingress/wg-server-key.sops.yaml \
|
||||
# "[\"stringData\"][\"privatekey\"]" "\"$(wg genkey)\""
|
||||
# '
|
||||
# (Or edit it: `./infra/devcontainer/dev.sh run sops platform/vpn-ingress/wg-server-key.sops.yaml`.)
|
||||
#
|
||||
# 2. Record BOTH halves in your PASSWORD MANAGER. The encrypted file is in git and Velero captures
|
||||
# this namespace to R2, so this is belt-and-braces — but it is the only copy that survives
|
||||
# losing the cluster AND the backups, and the key cannot be regenerated without invalidating
|
||||
# every client profile.
|
||||
# # public half (safe to display; it is what every client profile carries)
|
||||
# ./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d \
|
||||
# --extract "[\"stringData\"][\"privatekey\"]" \
|
||||
# platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey'
|
||||
#
|
||||
# 3. Confirm the value is a real key without printing it — `wg pubkey` succeeds only on a valid
|
||||
# 32-byte curve25519 private key, which is exactly the check the pod's init container runs:
|
||||
# ./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d \
|
||||
# --extract "[\"stringData\"][\"privatekey\"]" \
|
||||
# platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey >/dev/null \
|
||||
# && echo "valid WireGuard key"'
|
||||
#
|
||||
# 4. PR it. Nothing else to wire: the resource line and both overlay decryption patches already
|
||||
# exist. Flux rolls the DaemonSet and the pods leave CrashLoop.
|
||||
#
|
||||
# NOTE infra/vpn-ingress/new-peer.sh reads THIS FILE to derive each client profile's
|
||||
# `[Peer] PublicKey`, so there is no separate copy of the key to keep in step with anything.
|
||||
# =============================================================================
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: wg-server-key
|
||||
namespace: vpn-ingress
|
||||
type: Opaque
|
||||
stringData:
|
||||
# The server's WireGuard private key: `wg genkey` output — 44 chars of base64 ending in '='.
|
||||
# NOT a passphrase, and NOT the public key. The init container validates it with `wg pubkey`
|
||||
# before writing wg0.conf, so a placeholder or a mangled value fails closed with a clear message.
|
||||
privatekey: REPLACE-ME
|
||||
Reference in new issue
Block a user