diff --git a/clusters/base/vpn-ingress.yaml b/clusters/base/vpn-ingress.yaml index c34816a..e9dd108 100644 --- a/clusters/base/vpn-ingress.yaml +++ b/clusters/base/vpn-ingress.yaml @@ -8,8 +8,8 @@ # NOTHING downstream reconciles — the incident behind the headlamp/oauth2-proxy/homepage moves # (DECISIONS-LOG: "Headlamp gating stall"). # * SPECIFIC, AND THE REASON THIS IS NOT A JUDGEMENT CALL: this DaemonSet cannot start until the -# operator creates the out-of-band `wg-server-key` Secret (runbooks/vpn-remote-access.md -# step 1). That is the correct fail-closed state — an unenrolled VPN must not come up — but in +# operator replaces the `REPLACE-ME` placeholder in platform/vpn-ingress/wg-server-key.sops.yaml +# with a real key (runbooks/vpn-remote-access.md step 2). That is the correct fail-closed state — an unenrolled VPN must not come up — but in # the gating tier it would have been a self-inflicted platform-wide outage ON MERGE, with the # cause (a missing manual step) invisible from any Flux condition. # @@ -51,6 +51,9 @@ spec: substituteFrom: - kind: ConfigMap name: cluster-vars - # NO `decryption:` stanza is needed in the overlays: this path carries no *.sops.yaml. The server - # key is an out-of-band Secret (platform/vpn-ingress/kustomization.yaml says what to do if that - # ever changes). + # SOPS decryption is layered per-environment by the overlay (clusters/dev and clusters/prod each + # patch THIS Kustomization by name), identical to headlamp/crafty/velero — this path carries + # platform/vpn-ingress/wg-server-key.sops.yaml, the WireGuard server private key. The `sops-age` + # Secret is created out of band and referenced by NAME only. Without the overlay patch the Secret + # applies cleanly full of ENC[...] and only the tunnel is broken, so the patch is as load-bearing + # as the encrypted file itself. diff --git a/clusters/dev/kustomization.yaml b/clusters/dev/kustomization.yaml index db65056..2ed0b54 100644 --- a/clusters/dev/kustomization.yaml +++ b/clusters/dev/kustomization.yaml @@ -122,6 +122,24 @@ patches: provider: sops secretRef: name: sops-age + # vpn-ingress (the WireGuard remote-access VPN, clusters/base/vpn-ingress.yaml) carries + # platform/vpn-ingress/wg-server-key.sops.yaml — the WireGuard SERVER private key, the one + # secret in that component. Without this stanza kustomize hands Flux the raw ciphertext and the + # Secret applies full of ENC[...]: the APPLY SUCCEEDS and every Flux condition stays green, and + # the only symptom is a VPN that no client can handshake with. The DaemonSet's init container + # validates the value with `wg pubkey` specifically so that state is a named CrashLoop instead. + # The `sops-age` Secret is out-of-band (name only). + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: vpn-ingress + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot # account and the Anthropic subscription token from `claude setup-token`. Without this diff --git a/clusters/prod/kustomization.yaml b/clusters/prod/kustomization.yaml index 0153f67..dab3d63 100644 --- a/clusters/prod/kustomization.yaml +++ b/clusters/prod/kustomization.yaml @@ -113,6 +113,24 @@ patches: provider: sops secretRef: name: sops-age + # vpn-ingress (the WireGuard remote-access VPN, clusters/base/vpn-ingress.yaml) carries + # platform/vpn-ingress/wg-server-key.sops.yaml — the WireGuard SERVER private key, the one + # secret in that component. Without this stanza kustomize hands Flux the raw ciphertext and the + # Secret applies full of ENC[...]: the APPLY SUCCEEDS and every Flux condition stays green, and + # the only symptom is a VPN that no client can handshake with. The DaemonSet's init container + # validates the value with `wg pubkey` specifically so that state is a named CrashLoop instead. + # The `sops-age` Secret is out-of-band (name only). + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: vpn-ingress + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot # account and the Anthropic subscription token from `claude setup-token`. Without this diff --git a/decisions/2026-10-05-vpn-server-key-in-sops.md b/decisions/2026-10-05-vpn-server-key-in-sops.md new file mode 100644 index 0000000..4519128 --- /dev/null +++ b/decisions/2026-10-05-vpn-server-key-in-sops.md @@ -0,0 +1,146 @@ +--- +date: 2026-10-05 +branch: fix/vpn-server-key-in-sops +title: VPN server key moves into SOPS-encrypted git — the out-of-band Secret broke R-SEC-01 and the DR story +--- + +## What changed + +**Operator correction (2026-10-05): "secrets are kept in source through sops".** They are right, and +#55 shipped the WireGuard server private key the wrong way: as a Secret the operator was told to +`kubectl create` out of band, referenced by name only. This branch moves it into the tree as +`platform/vpn-ingress/wg-server-key.sops.yaml`, where every other secret in this repo lives. + +**Why the original choice was wrong, not merely unconventional.** Two concrete costs, both of which +#55's own README half-admitted and then shipped anyway: + +1. **R-SEC-01 / spec docs/07 says secrets live in source, SOPS-encrypted, reaching the cluster only + through Flux.** An out-of-band Secret is outside the GitOps tree by construction. +2. **It broke DR for the one credential that cannot be re-derived.** A cluster rebuild restores + everything in git; it does not restore a hand-made Secret. Losing this key means **re-enrolling + every device**, because each client profile pins its public half. #55 papered over this with + "keep a copy in your password manager", which is a workaround for a self-inflicted gap. + +Note `platform/kyverno/admission/secret-provenance.yaml` did **not** forbid the original shape — +it scopes to `app-namespace=true` namespaces and `vpn-ingress` is deliberately a platform namespace, +so admission would have accepted the hand-applied Secret. The rule it violated was the written +requirement, not an enforced control. Recording that because "the gate let it through" was never +evidence it was correct. + +**How the unfillable-key problem is solved.** An agent still must not mint a long-lived private key +(CLAUDE.md §6), so the file is committed **already encrypted, with `privatekey: REPLACE-ME`**. That +makes the wiring complete and the gate green from the first commit — the resource line and both +overlay decryption patches are present — and leaves the operator exactly one act: replace the value. +The alternative, a commented-out resource line to uncomment later, is precisely the "file in git with +no generator entry: reviewed, merged, apparently done, still broken" shape this repo keeps getting +bitten by (`alert-runbooks-resolvable` exists because of it). + +**The init-container guard is now a validation, not an emptiness check.** It was `[ -z "$key" ]`. +Because the committed placeholder is non-empty, the likely failure is now a value that is present +and *wrong*, so the guard pipes the value through `wg pubkey` — which accepts only a real 32-byte +curve25519 private key — and names which of four cases it hit: + +| case | why it matters | +|---|---| +| `REPLACE-ME` placeholder | the expected fresh-install state | +| empty | blanked value | +| **starts with `ENC[`** | **the Secret reached the cluster UNDECRYPTED.** A missing overlay decryption patch applies cleanly and leaves every Flux condition green — this is the silent failure the guard exists for | +| rejected, none of the above | mangled value, or a *public* key pasted by mistake | + +Files: + +- `platform/vpn-ingress/wg-server-key.sops.yaml` (new) — SOPS-encrypted to the repo's age recipient, + `stringData.privatekey` only, placeholder value. +- `platform/vpn-ingress/wg-server-key.sops.yaml.example` (new) — plaintext template + operator + hand-off, following the `crafty-oauth2-proxy-credentials.sops.yaml.example` convention. +- `platform/vpn-ingress/kustomization.yaml` — the resource added; the "DELIBERATELY ABSENT: a + Secret" block replaced with the overlay-patch warning. +- `platform/vpn-ingress/daemonset.yaml` — the `wg pubkey` guard; volume comment rewritten. +- `clusters/dev/kustomization.yaml`, `clusters/prod/kustomization.yaml` — **the `vpn-ingress` + `decryption:` patch, in both.** This is the load-bearing half: #55's own kustomization comment + predicted that forgetting it applies ciphertext cleanly. +- `clusters/base/vpn-ingress.yaml` — its "NO decryption stanza is needed" comment was now false. +- `platform/vpn-ingress/README.md`, `runbooks/vpn-remote-access.md` — step 2 rewritten. +- `infra/vpn-ingress/new-peer.sh` — stale "step 1" pointer. + +**A second finding, which changes the procedure independently of SOPS: `wg` and `sops` are not in +the same place.** Measured: `wg` is on the **host** only (`/usr/bin/wg`, absent from the dev +container); `sops` is in the **container** only (`/usr/local/bin/sops`, absent from the host). So +`sops -d … | wg pubkey` cannot be run in one shell, and #55's step 2 — which assumed it could — +would have failed. The runbook and the `.example` now split every command by where it runs, and +verify the written key by comparing **sha256 digests** of the decrypted value and the operator's +local copy, which never puts key material on a terminal. `qrencode` is on neither, so the QR +instruction now says so and gives the install command. + +**SENSITIVE PATHS — this PR touches `clusters/**` (two overlays) AND adds a `**/*.sops.yaml`, so +`ci/approval-policy.sh` will BLOCK it.** It needs the operator's `/approve-override ` as a +PR **review** of type "Comment". I have deliberately not posted it (CLAUDE.md §1). + +## What I verified + +- **The encrypted file is real and correctly targeted.** `sops -e -i` in the pinned toolbox + produced `stringData.privatekey: ENC[AES256_GCM,...]` with cleartext keys, `encrypted_regex + ^(data|stringData)$`, and exactly one age recipient: + `age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg` — the recipient in the root + `.sops.yaml`, i.e. the key Flux holds as `sops-age`. +- **Round-trip, without exposing anything**: `sops -d` then a Python read of key names only → + `stringData keys: ['privatekey']`, and `value is still the placeholder: True`. +- **Both overlay patches are present and name the right Kustomization** (`name: vpn-ingress`, + `namespace: flux-system`, `decryption.secretRef.name: sops-age`), inserted after each overlay's + `velero` stanza, mirroring headlamp/crafty/velero. +- **`kubectl kustomize platform/vpn-ingress` builds** and now emits five objects — Namespace, + ConfigMap, **Secret**, DaemonSet, NetworkPolicy — with `${WG_CLIENT_CIDR}` still an unexpanded + token for Flux. +- **`wg` / `sops` / `qrencode` locations**, measured on both host and container, as described above. +- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`, + `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two + environmental skips (`gitea-settings` — no local token; conformance L2/L3 — no cluster). + **Not green. Reported as rc 3, not as pass.** +- **The L0 gate caught THREE real defects in this branch before it could merge — including one + in the fix for another — and I had asserted one of them was fine without running it.** Both are recorded because the gate's value here is the + whole point of the exercise: + 1. **`no-plaintext-secrets` FAILED**: "`wg-server-key.sops.yaml.example`: an unencrypted .example + template is referenced by a kustomization — it would be applied as a plaintext Secret". My + kustomization named the template in a *comment*, and the check matches comments too (correctly: + naming one is how a plaintext Secret gets applied by accident). `platform/crafty/kustomization.yaml` + never names its own example either — the convention was there to copy and I missed it. Fixed by + not naming the file; the hand-off lives inside the example, where it belongs. + **An earlier draft of this entry claimed this check passed. It did not. The claim was written + from expectation rather than from a run.** + 2. **`flux-postbuild-substitution` FAILED** on `${key#ENC[}` in my new guard + (`daemonset.yaml:155`), flagged `[SILENT-EMPTY]`: envsubst consumes every `${...}`, so my + shell prefix-strip would have rendered to the empty string — the ciphertext branch of the guard + would never have fired, which is precisely the silent failure the guard was added to catch. + Rewritten with `case`, which uses no braces and so has nothing to escape. + 3. **And then it failed a THIRD time, on the fix's own comment** — `[FATAL-PARSE] + daemonset.yaml:156 ${...}`, the prose explaining the rule, written with the literal form. That + class is worse than it looks: a `FATAL-PARSE` hazard fails the **entire** Kustomization, every + resource in the path, not just the line. Rewritten as "dollar-brace expansion". This is the + second time in this component that writing the hazard literally inside a comment about the + hazard has been the defect (the first was in #55), so the habit to build is: **in any + substituted path, describe the construct in words, never type it.** +- **After both fixes, `no-plaintext-secrets` and `flux-postbuild-substitution` pass** (446 and 198 + assertions respectively), and `l0` is green. + +## What I did NOT verify + +- **That Flux decrypts this file on prod.** The overlay patches have never reconciled. This is the + whole risk of the change and it has exactly one observable outcome: after merge the pods should + CrashLoop reporting the **`REPLACE-ME` placeholder** (proving decryption worked), *not* reporting + **`ENC[`** (which would prove a patch is wrong). I built the guard to tell those apart because I + cannot test them. +- **That `wg pubkey` accepts the operator's real key through this path.** The guard is reasoned from + `wg pubkey`'s behaviour and has never run — `wg` is absent from both this workstation's container + and the gate, so the first execution will be in the pod. +- **The `sops set` command in the hand-off.** I verified `sops -e -i` and `sops -d` in the toolbox; + I did **not** run `sops set` with the quoting shown, and nested quoting inside + `dev.sh run bash -c` is exactly where a documented command tends to break. If it misbehaves, the + editor form (`sops `) is the fallback and does not depend on quoting. +- **The sha256 comparison in step 2 step 4** is untested end to end (it needs the real key). +- **Whether anything else in #55 assumed a tool is available where it is not.** I checked `wg`, + `sops` and `qrencode` because step 2 depended on them. I did not audit every command in the + runbook the same way — and that class of defect has now produced two corrections in a row + (`site.sh`, then `sops | wg`), so assume more remain until each step has actually been run. +- **The live VPN still carries no traffic**, and the earlier gaps stand unchanged: the + iptables-vs-nftables question, `allowPrivilegeEscalation: false` versus the image's s6 init, and + any off-LAN reachability. diff --git a/infra/vpn-ingress/new-peer.sh b/infra/vpn-ingress/new-peer.sh index 31d370b..54f296b 100755 --- a/infra/vpn-ingress/new-peer.sh +++ b/infra/vpn-ingress/new-peer.sh @@ -89,7 +89,7 @@ command -v wg >/dev/null || die "wireguard-tools not installed ('wg' not on PATH if [ -z "${WG_SERVER_PUBKEY:-}" ]; then [ -r "$WG_SERVER_KEY_FILE" ] || die "cannot read server key '$WG_SERVER_KEY_FILE'. Either point WG_SERVER_KEY_FILE at it, or set WG_SERVER_PUBKEY to the public half. - If the server key does not exist yet, see runbooks/vpn-remote-access.md step 1." + If the server key does not exist yet, see runbooks/vpn-remote-access.md step 2." WG_SERVER_PUBKEY="$(tr -d ' \r\n' < "$WG_SERVER_KEY_FILE" | wg pubkey)" fi diff --git a/platform/vpn-ingress/README.md b/platform/vpn-ingress/README.md index 4ed1016..36a664d 100644 --- a/platform/vpn-ingress/README.md +++ b/platform/vpn-ingress/README.md @@ -14,7 +14,7 @@ egress out the house WAN. | Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node | | Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices | | Authorisation | a device's **public key** is in [`peers.yaml`](./peers.yaml), or it cannot connect | -| The one secret | the server's private key, Secret `wg-server-key`, **created out of band** | +| The one secret | the server's private key — **SOPS-encrypted in this tree**, `wg-server-key.sops.yaml` | | Exposure | one hand-made WAN port-forward on the SOHO router, off-repo | | Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) | | Procedure / bring-up | [`runbooks/vpn-remote-access.md`](../../runbooks/vpn-remote-access.md) | @@ -130,11 +130,17 @@ translations win here. | half | where | why | |---|---|---| | peer **public** keys + addresses | [`peers.yaml`](./peers.yaml), **plaintext in git** | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail | -| the server's **private** key | Secret `wg-server-key`, **out of band** | the only real secret; an agent must not mint a long-lived private key (CLAUDE.md §6) | +| the server's **private** key | `wg-server-key.sops.yaml`, **SOPS-encrypted in git** | the only real secret (R-SEC-01, spec docs/07). Ships with a `REPLACE-ME` placeholder: an agent must not mint a long-lived private key (CLAUDE.md §6), so the operator `sops set`s it in | The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** — so the assembled config holding the private key never touches any node's disk and dies with the pod. +Both halves are therefore in git, which is the point: a cluster rebuild restores the server key with +everything else. An out-of-band `kubectl create secret` — the first shape this component shipped in, +corrected after review — would have put the one credential every enrolled device depends on *outside* +the GitOps tree, and losing it means re-enrolling every device, because each client profile pins this +key's public half. + Encrypting the peer register would have hidden exactly the change a reviewer most needs to see: a new line granting a new device access to the whole LAN. @@ -173,8 +179,14 @@ NetworkPolicy applies. (`infra/ansible/roles/node_baseline/defaults/main.yml`); the operator runs the role. The pod has `NET_ADMIN` but deliberately **not** `SYS_MODULE`, so it cannot load the module itself — it CrashLoops with a clear `wg-quick` error instead of quietly degrading to userspace WireGuard. -2. **The `wg-server-key` Secret.** Runbook step 1. Until it exists the DaemonSet cannot start — - the correct fail-closed state for an unenrolled VPN. +2. **The server private key.** `wg-server-key.sops.yaml` is committed, encrypted, with a + `REPLACE-ME` placeholder, and both overlays already carry its decryption patch — so the wiring + is complete and the only outstanding act is replacing the value (runbook step 2). Until then the + DaemonSet CrashLoops with an explicit FATAL naming the file: the init container validates the + value with `wg pubkey`, which rejects the placeholder, an empty value, a public key pasted by + mistake, **and** a Secret that arrived as undecrypted `ENC[...]` ciphertext. That last case is + the one worth having a guard for — a missing overlay decryption patch applies cleanly and leaves + every Flux condition green. 3. **The WAN port-forward**, udp/51820 → one node address. Hand-made on the SOHO router, off-repo, and the half CI can never check. diff --git a/platform/vpn-ingress/daemonset.yaml b/platform/vpn-ingress/daemonset.yaml index 4a72602..801b4ef 100644 --- a/platform/vpn-ingress/daemonset.yaml +++ b/platform/vpn-ingress/daemonset.yaml @@ -136,9 +136,38 @@ spec: # The key arrives as a Secret file; strip any trailing newline the operator's shell # added, or wg-quick rejects the config with "Key is not the correct length or format". key=$(tr -d ' \r\n' < /run/wg-server-key/privatekey) - if [ -z "$key" ]; then - echo "FATAL: /run/wg-server-key/privatekey is empty." >&2 - echo " See runbooks/vpn-remote-access.md step 1 (create the server key)." >&2 + # VALIDATE, don't just check for empty. The Secret ships committed with a REPLACE-ME + # placeholder so the GitOps wiring is complete before the key exists, so the common + # failure is a value that is present and WRONG — a placeholder, a public key pasted by + # mistake, or a SOPS document that reached the cluster undecrypted (ENC[...], which + # happens if an overlay is missing its `vpn-ingress` decryption patch). `wg pubkey` + # accepts a value only if it is a real 32-byte curve25519 private key, so it rejects + # all four. Without this the bad value would be written into wg0.conf and the pod would + # fail later and less legibly, or come up with a server nobody can handshake with. + if ! printf '%s' "$key" | wg pubkey >/dev/null 2>&1; then + echo "FATAL: the server private key in Secret 'wg-server-key' is not a valid" >&2 + echo " WireGuard private key (wg pubkey rejected it)." >&2 + if [ -z "$key" ]; then + echo " It is EMPTY." >&2 + elif [ "$key" = "REPLACE-ME" ]; then + echo " It is still the committed REPLACE-ME PLACEHOLDER — the key has not" >&2 + echo " been generated yet. This is the expected state on a fresh install." >&2 + else + # `case`, NOT a dollar-brace prefix-strip. This manifest is rendered through + # envsubst, which consumes every dollar-brace expansion it sees, so a shell + # parameter expansion here renders to the empty string and silently disables this + # very branch. `case` uses no braces, so there is nothing to escape. + # (ci: flux-postbuild-substitution caught both the original expansion AND the + # first version of this comment, which spelled the hazard out literally.) + case "$key" in + ENC\[*) + echo " It is still SOPS CIPHERTEXT (starts with ENC[). The Secret" >&2 + echo " reached the cluster undecrypted — check that this environment's" >&2 + echo " overlay carries the 'vpn-ingress' decryption patch." >&2 + ;; + esac + fi + echo " Fix: runbooks/vpn-remote-access.md step 2." >&2 exit 1 fi @@ -272,14 +301,22 @@ spec: # during startup rather than at a point anyone would connect to the volume size. Still # counted against the container's 128Mi limit, and still far more than wg0.conf needs. sizeLimit: 8Mi - # The ONLY secret in this component. Created OUT OF BAND by the operator and referenced by - # name (R-IDP-04) — runbooks/vpn-remote-access.md step 1. It is NOT a *.sops.yaml in this - # tree on purpose: an agent must not mint a long-lived private key (CLAUDE.md §6), and a key - # that passed through a session transcript would have to be treated as compromised. - # DR: Velero's `daily` schedule excludes only `monitoring` and `unifi`, so this namespace - # and this Secret ARE captured off-cluster (platform/velero/helmrelease.yaml). Keep a copy - # in the operator's password manager as well — restoring it is what makes every already - # enrolled client work again without re-enrolling. + # The ONLY secret in this component. SOPS-encrypted IN THIS TREE (R-SEC-01, spec docs/07) as + # wg-server-key.sops.yaml, delivered by Flux, referenced here by name only (R-IDP-04). + # + # It ships committed with a REPLACE-ME placeholder: an agent must not mint a long-lived + # private key (CLAUDE.md §6), so the operator generates it and `sops set`s it in + # (runbooks/vpn-remote-access.md step 2). Shipping the encrypted placeholder rather than no + # file at all is what lets the resource line and both overlay decryption patches be present + # and GREEN from the first commit — the alternative, a commented-out resource to uncomment + # later, is exactly the "file in git with no generator entry: reviewed, merged, apparently + # done, still broken" shape this repo keeps getting bitten by. + # + # DR: in git means a cluster rebuild restores it with everything else — the property an + # out-of-band `kubectl create secret` would NOT have had, and losing this key means + # re-enrolling every device because each client profile pins its public half. Velero also + # captures this namespace (its `daily` schedule excludes only `monitoring` and `unifi`). + # Keep a password-manager copy too: that is the one that survives losing cluster AND backups. - name: wg-server-key secret: secretName: wg-server-key diff --git a/platform/vpn-ingress/kustomization.yaml b/platform/vpn-ingress/kustomization.yaml index 96a158c..924f9b5 100644 --- a/platform/vpn-ingress/kustomization.yaml +++ b/platform/vpn-ingress/kustomization.yaml @@ -4,9 +4,9 @@ # same placement as crafty/headlamp/homepage/runbooks, for the same reason: a leaf whose pod can be # legitimately unhealthy must never sit in the gating `platform` tier, where one component's # unhealth stalls platform-config and every apps wave behind it (DECISIONS-LOG: "Headlamp gating -# stall"). That rationale is unusually concrete here: until the operator creates the `wg-server-key` -# Secret (runbooks/vpn-remote-access.md step 1) this DaemonSet CANNOT start, so putting it in the -# gating tier would have taken the whole platform down on merge. +# stall"). That rationale is unusually concrete here: until the operator replaces the `REPLACE-ME` +# placeholder in wg-server-key.sops.yaml (runbooks/vpn-remote-access.md step 2) this DaemonSet +# CANNOT start, so putting it in the gating tier would have taken the whole platform down on merge. # # NOT in platform/kustomization.yaml — see clusters/base/vpn-ingress.yaml. apiVersion: kustomize.config.k8s.io/v1beta1 @@ -19,12 +19,22 @@ resources: - daemonset.yaml # Namespace default-deny baseline. NOTE: does NOT constrain the hostNetwork DaemonSet — the file # says so at length rather than implying a protection it cannot provide. - - networkpolicy.yaml + - networkpolicy.yaml # The server's WireGuard private key — the ONE secret here. SOPS-encrypted (R-SEC-01, spec + # docs/07), decrypted per-env by the `vpn-ingress` Kustomization, referenced by name only from + # daemonset.yaml (R-IDP-04). Its plaintext `.example` sibling carries the operator hand-off -- + # deliberately NOT named here: conformance/tests/l0-static/no-plaintext-secrets fails any + # kustomization that mentions an unencrypted template, because naming one is how a plaintext + # Secret gets applied by accident. (It matches comments too. That is correct, not overzealous.) + # + # SHIPS WITH A `REPLACE-ME` PLACEHOLDER, already encrypted. An agent must not + # mint a long-lived private key (CLAUDE.md §6), and the init container validates the value with + # `wg pubkey`, so the unfilled state is a loud CrashLoop naming this file, never a server that + # quietly cannot handshake. + - wg-server-key.sops.yaml # -# DELIBERATELY ABSENT: a Secret. The server's private key (`wg-server-key`) is created out of band -# by the operator and referenced BY NAME from daemonset.yaml. There is no `*.sops.yaml` here, so -# this component needs NO SOPS decryption stanza in clusters/dev or clusters/prod — unlike -# crafty/headlamp/velero. If that key is ever migrated into SOPS, BOTH halves must land together: -# the file here, and a `decryption:` patch naming the `vpn-ingress` Kustomization in each overlay. -# Without the patch, kustomize hands Flux the raw ciphertext and the pod mounts a key full of -# ENC[...] — which applies cleanly and fails only at handshake time. +# THIS COMPONENT CARRIES AN ENCRYPTED SECRET, so BOTH overlays must patch the `vpn-ingress` +# Kustomization with a `decryption:` stanza — clusters/dev and clusters/prod both do, added by the +# same change as this file. That patch is as load-bearing as the file itself: without it kustomize +# hands Flux the raw ciphertext, the Secret APPLIES CLEANLY full of ENC[...], and nothing in any +# Flux condition looks wrong. The init container's `wg pubkey` guard exists partly to turn exactly +# that into a named error rather than a silently non-working tunnel. diff --git a/platform/vpn-ingress/wg-server-key.sops.yaml b/platform/vpn-ingress/wg-server-key.sops.yaml new file mode 100644 index 0000000..7347899 --- /dev/null +++ b/platform/vpn-ingress/wg-server-key.sops.yaml @@ -0,0 +1,34 @@ +# WireGuard SERVER private key for platform/vpn-ingress — SOPS-ENCRYPTED (R-SEC-01, spec docs/07). +# Keys stay cleartext for reviewable diffs; only `stringData` values are encrypted (.sops.yaml +# `encrypted_regex`). Plaintext template + operator hand-off: wg-server-key.sops.yaml.example. +apiVersion: v1 +kind: Secret +metadata: + name: wg-server-key + namespace: vpn-ingress +type: Opaque +stringData: + #ENC[AES256_GCM,data:fUoPkfWvasisdTtwdhjeOVOlmdQkTl6Bp9YxYyji8g6k+ClATds3BnHJWOj2iplaGpT2XaNakq1+D4YyOi7QbFvNJui+e4qRpqcY8hWnazvZIHJYxjxfAfzW,iv:xtKd6oMZQumAQJAsNLHqJB+o0wdQJ+vL1kI1mc9jL7w=,tag:aImqLgbSmhsRI36/t5kpyg==,type:comment] + #ENC[AES256_GCM,data:npDsCW6rQleolIpur5gFkX2jSRrIQzBlGaIPs2Schv0YU3q8AanG39z4QqT3ASDmskV5YuH29Bo0ujEAgBf52YIjQCp58PDDETofrVKOJzhHMa7bV+HCxUFE,iv:7E7thyFBleyqZpqJRGApKvIq3ubCFOkshF/mxim3YPc=,tag:joIrhMe2I6kUnx1ouOPdZw==,type:comment] + #ENC[AES256_GCM,data:wHUFx09FgP/2+LTyw61EFbKA5UaqfjU7CeGk6Xc5av2y4r4AvQJRefDM5EGlahKecDFEFsf7fSQRFRHtUJqrQTtw+6tcj3ubSCPW4iJvhyJJv0xdb/0puRmcgj+2NfgF,iv:xgDOSDLqMQ1wcI9vrkKqfNMgIDFEusFnnO7Rzq/aOa4=,tag:m1m/fgjkBsG91c+3daqOpA==,type:comment] + privatekey: ENC[AES256_GCM,data:irLENWa1Z3IJSQ==,iv:Qu2Yr6F5+M6P8cQRuhB5J7joxCcudUhwoEWDiL941Lo=,tag:ZNwcBZEkswtnkX0vGKfMAQ==,type:str] +sops: + kms: [] + gcp_kms: [] + azure_kv: [] + hc_vault: [] + age: + - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg + enc: | + -----BEGIN AGE ENCRYPTED FILE----- + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBMays2OXY0VkpRc2FQQmhU + Q0owM3N0VkhYR0lYTUdNZWRqeTFIMjNHOVJ3CnJSMnc5NFEyU0IzWTFtUkw2YVFS + MU02VFo1TE14YVpZRzNhVjZwRTZwd0UKLS0tIDFaMWF4NUZjZmhRdXRuZFJYNkg3 + SUUzN1FnMlI4bkkzbVNlMWhUVnJZODAKQ+Iy86CIv/Oj9E9j4E/0y5hDbcctRuqB + OR825KRPRaTdF/G2hudE1R1JbM4wGbPLMzv078R4ZXHDPGvDttSEkA== + -----END AGE ENCRYPTED FILE----- + lastmodified: "2026-10-06T04:20:58Z" + mac: ENC[AES256_GCM,data:GpP/zTX4FtJyrlaCS/R0/sAtFbkM0W8A5sqNxb5ZziSgLUXI+oUKYB09igN6vx5cjRNH7rSmXduS/actaT5jhP8kuF1n/iQfG56PsCJHL2vneefzcEOKXeGquAwDOY+iGmNsZ5R8ySrM/Fq7HYLMJasWosyTh3YlBs8GRYiHmU4=,iv:438+pDzHLGyh+OMQCMzGBEBrZUPNp5BbjrrMdiEiqJ8=,tag:JqLu6zZWKtcoB8mp4HSZig==,type:str] + pgp: [] + encrypted_regex: ^(data|stringData)$ + version: 3.9.4 diff --git a/platform/vpn-ingress/wg-server-key.sops.yaml.example b/platform/vpn-ingress/wg-server-key.sops.yaml.example new file mode 100644 index 0000000..57f7662 --- /dev/null +++ b/platform/vpn-ingress/wg-server-key.sops.yaml.example @@ -0,0 +1,82 @@ +# ============================================================================= +# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never +# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF. +# +# WHAT THIS IS: the WireGuard SERVER's long-term private key for the remote-access VPN +# (platform/vpn-ingress). It is the ONE secret in the whole component. The init container in +# daemonset.yaml reads it at `/run/wg-server-key/privatekey` and joins it to the PLAINTEXT peer +# register (peers.yaml) in a tmpfs, so the assembled wg0.conf never touches a node's disk. +# +# WHY IT IS IN GIT, SOPS-ENCRYPTED, AND NOT HAND-CREATED. R-SEC-01 / spec docs/07: secrets live in +# source, encrypted, and reach the cluster only through Flux. An out-of-band `kubectl create secret` +# would put the single credential every enrolled device depends on OUTSIDE the GitOps tree, where a +# cluster rebuild silently loses it — and losing it means RE-ENROLLING EVERY DEVICE, because each +# client profile pins this key's public half. +# +# *** THE REAL FILE IS COMMITTED WITH `privatekey: REPLACE-ME`, ALREADY ENCRYPTED. *** +# That is deliberate, so the wiring is complete and green before the key exists: the resource is +# already listed in kustomization.yaml and both overlays already carry the `vpn-ingress` decryption +# patch. Your job is ONLY to replace the value (step 3 below). Until you do, the DaemonSet +# CrashLoops with an explicit FATAL naming this file — the init container validates the value with +# `wg pubkey` before writing wg0.conf, so a placeholder fails CLOSED and LOUDLY rather than +# producing a server that silently cannot complete a handshake. +# +# --------------------------------------------------------------------------------------------- +# MIND WHERE EACH COMMAND RUNS. The two tools do not live in the same place (measured 2026-10-05): +# wg -> the HOST only (/usr/bin/wg; NOT in the dev container) +# sops -> the dev CONTAINER only (/usr/local/bin/sops; NOT on the host) +# So you cannot pipe `sops -d ... | wg pubkey` in one shell. The steps below are split accordingly, +# and they never put the key on a terminal or in a pipeline that crosses the boundary. +# --------------------------------------------------------------------------------------------- +# +# HAND-OFF STEPS (operator — an agent must not mint a long-lived private key, CLAUDE.md §6). +# Full procedure with verification: runbooks/vpn-remote-access.md step 2. +# +# 1. HOST — generate it. This file is also what infra/vpn-ingress/new-peer.sh reads to derive the +# public half for each client profile, so keep it where the script's default expects it: +# umask 077 +# mkdir -p ~/.config/wireguard-home && chmod 700 ~/.config/wireguard-home +# wg genkey > ~/.config/wireguard-home/server.key +# +# 2. Put that value in your PASSWORD MANAGER. It cannot be regenerated without invalidating every +# client profile. (Velero backs this namespace up to R2 too, but the password-manager copy is +# the one that survives losing the cluster AND the backups.) +# Record the public half alongside it — safe to display, and useful when debugging a client: +# wg pubkey < ~/.config/wireguard-home/server.key +# +# 3. Stage it where the CONTAINER can read it, then write it into the encrypted file in place: +# mkdir -p .cache && cp ~/.config/wireguard-home/server.key .cache/server.key +# git check-ignore -v .cache/server.key # MUST print `.gitignore:11:*.key` — see below +# ./infra/devcontainer/dev.sh run bash -c \ +# 'cd /work && sops set platform/vpn-ingress/wg-server-key.sops.yaml \ +# "[\"stringData\"][\"privatekey\"]" "\"$(tr -d "\r\n" < /work/.cache/server.key)\""' +# shred -u .cache/server.key +# `sops set` edits the ENCRYPTED document directly — no plaintext copy of it is ever written. +# +# KEEP THE `.key` SUFFIX on that staged copy. `.gitignore` line 11 is `*.key`, and it is the +# ONLY rule covering that path: `.cache/` itself is NOT ignored. A copy named `server.pem` or +# `privatekey` in the same directory is COMMITTABLE, and the gate's `no-plaintext-secrets` stage +# does not recognise a bare WireGuard key. +# +# 4. Verify you wrote the right bytes WITHOUT printing them — compare digests, not values: +# ./infra/devcontainer/dev.sh run bash -c \ +# 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \ +# platform/vpn-ingress/wg-server-key.sops.yaml | tr -d "\r\n" | sha256sum' +# tr -d '\r\n' < ~/.config/wireguard-home/server.key | sha256sum +# The two digests must match. (This is the only sanctioned decrypt of this file, and its output +# goes straight into sha256sum — never onto a terminal. CLAUDE.md §6.) +# +# 5. PR it. Nothing else to wire: the resource line and both overlay decryption patches already +# exist. Flux rolls the DaemonSet and the pods leave CrashLoop. +# ============================================================================= +apiVersion: v1 +kind: Secret +metadata: + name: wg-server-key + namespace: vpn-ingress +type: Opaque +stringData: + # The server's WireGuard private key: `wg genkey` output — 44 chars of base64 ending in '='. + # NOT a passphrase, and NOT the public key. The init container validates it with `wg pubkey` + # before writing wg0.conf, so a placeholder or a mangled value fails closed with a clear message. + privatekey: REPLACE-ME diff --git a/runbooks/vpn-remote-access.md b/runbooks/vpn-remote-access.md index cde254c..21a0eac 100644 --- a/runbooks/vpn-remote-access.md +++ b/runbooks/vpn-remote-access.md @@ -62,53 +62,90 @@ lsmod | grep '^wireguard' grep -w wireguard /etc/modules-load.d/k3s-cilium-longhorn.conf ``` -## Step 2 — create the server key (once) +## Step 2 — fill in the server key (once) -This is the **only** secret in the component, and the one thing that must never be generated in an -agent session or pasted into a transcript. Generate it on your own machine: +The key is **SOPS-encrypted in this tree** (R-SEC-01, spec docs/07), not created by hand: +`platform/vpn-ingress/wg-server-key.sops.yaml`. It is already committed — encrypted, with a +`REPLACE-ME` placeholder — and it is already listed in the component's `resources:` with a +`decryption:` patch in both overlays. **Nothing needs wiring. You are only replacing one value.** + +Why in git rather than `kubectl create secret`: an out-of-band Secret puts the one credential every +enrolled device depends on outside the GitOps tree, where a cluster rebuild silently loses it — and +losing it means re-enrolling every device, because each client profile pins this key's public half. + +> **MIND WHERE EACH COMMAND RUNS.** The two tools are not in the same place (measured 2026-10-05): +> `wg` is on the **HOST** only (`/usr/bin/wg`, absent from the dev container); `sops` is in the +> **CONTAINER** only (`/usr/local/bin/sops`, absent from the host). You cannot pipe +> `sops -d … | wg pubkey` in one shell. The steps below are split accordingly and never put the key +> on a terminal. + +**1. HOST — generate it.** Keep it at this path: it is where `new-peer.sh` looks by default to +derive the public half for each client profile. ```bash -mkdir -p ~/.config/wireguard-home && chmod 700 ~/.config/wireguard-home umask 077 -wg genkey > ~/.config/wireguard-home/server.key # keep this; see "Back it up" below +mkdir -p ~/.config/wireguard-home && chmod 700 ~/.config/wireguard-home +wg genkey > ~/.config/wireguard-home/server.key ``` -Load it into the cluster. **Do not echo it, do not pass it on an interactive command line that -lands in shell history, and never print the resulting Secret's `.data`** (CLAUDE.md §6): +**2. Store it in your PASSWORD MANAGER**, together with its public half (safe to display): + +```bash +wg pubkey < ~/.config/wireguard-home/server.key +``` + +This copy is the one that survives losing the cluster *and* the backups. Velero does capture this +namespace to R2 (its `daily` schedule excludes only `monitoring` and `unifi`), and the encrypted +file is in git — but the key itself cannot be regenerated without invalidating every client profile. + +**3. Write it into the encrypted file in place.** + +```bash +mkdir -p .cache && cp ~/.config/wireguard-home/server.key .cache/server.key +git check-ignore -v .cache/server.key # MUST print `.gitignore:11:*.key` — see the warning below + +./infra/devcontainer/dev.sh run bash -c \ + 'cd /work && sops set platform/vpn-ingress/wg-server-key.sops.yaml \ + "[\"stringData\"][\"privatekey\"]" "\"$(tr -d "\r\n" < /work/.cache/server.key)\""' + +shred -u .cache/server.key +``` + +`sops set` edits the **encrypted** document directly, so no plaintext copy of it is ever written. +(`sops platform/vpn-ingress/wg-server-key.sops.yaml` opens an editor instead, if you prefer.) + +> **KEEP THE `.key` SUFFIX** on the staged copy. `.gitignore` line 11 is `*.key`, and it is the +> **only** rule covering that path — `.cache/` itself is NOT ignored +> (`git check-ignore -v .cache/serverkey.txt` matches nothing). A copy named `server.pem` or +> `privatekey` in that directory is **committable**, and the gate's `no-plaintext-secrets` stage +> does not recognise a bare WireGuard key. + +**4. Verify you wrote the right bytes, without printing them** — compare digests, not values: ```bash ./infra/devcontainer/dev.sh run bash -c \ - 'kubectl --context k3s-prod -n vpn-ingress create secret generic wg-server-key \ - --from-file=privatekey=/work/.cache/server.key' + 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \ + platform/vpn-ingress/wg-server-key.sops.yaml | tr -d "\r\n" | sha256sum' + +tr -d '\r\n' < ~/.config/wireguard-home/server.key | sha256sum ``` -…where `/work/.cache/server.key` is a path you copied the key to (`.cache/` is not tracked; create -it if absent). Delete that copy as soon as the Secret exists: `shred -u .cache/server.key`. +The two digests must match. This is the only sanctioned decrypt of this file, and its output goes +straight into `sha256sum` — never onto a terminal (CLAUDE.md §6). -> **KEEP THE `.key` SUFFIX.** That filename is what keeps the key out of git: `.gitignore` line 11 -> is `*.key`, and it is the **only** rule covering this path — `.cache/` itself is NOT ignored -> (`git check-ignore -v .cache/serverkey.txt` matches nothing). So a copy named `server.pem`, -> `wg-server` or `privatekey` sitting in that directory is **committable**, and the gate's -> `no-plaintext-secrets` stage does not know a WireGuard private key when it sees one. Verify before -> you trust it: `git check-ignore -v .cache/` must print a rule, and -> `git status --short` must not list the file. +**5. PR it, and merge.** Flux rolls the DaemonSet and the pods leave CrashLoop. -Confirm it exists **by key name only**: +### What the pod tells you if this went wrong -```bash -./infra/devcontainer/dev.sh run bash -c \ - 'kubectl --context k3s-prod -n vpn-ingress get secret wg-server-key -o jsonpath="{.data}" | tr "," "\n" | cut -d: -f1' -# expected: {"privatekey -- a NAME. If you see base64 payload, you printed the key. Rotate it. -``` +The init container validates the value with `wg pubkey` before writing `wg0.conf`, and names the +specific failure in its log — so you never get a server that silently cannot complete a handshake: -**Why out of band and not SOPS.** An agent must not mint a long-lived private key, and a key that -passed through a session transcript would have to be treated as compromised. The trade-off is that -this Secret is **not reconstructible from git** — so: - -**Back it up.** Keep `server.key` in your password manager. Velero's `daily` schedule excludes only -`monitoring` and `unifi` (`platform/velero/helmrelease.yaml`), so this namespace *is* captured -off-cluster to R2 as well. Restoring this one key is what makes every already-enrolled device work -again after a cluster rebuild — **losing it means re-enrolling every device.** +| pod log says | meaning | +|---|---| +| `still the committed REPLACE-ME PLACEHOLDER` | step 3 not done (the expected fresh-install state) | +| `It is EMPTY` | the value was blanked | +| `still SOPS CIPHERTEXT (starts with ENC[)` | the Secret reached the cluster undecrypted — this environment's overlay is missing its `vpn-ingress` decryption patch. **This is the dangerous one:** the Secret applies cleanly and every Flux condition stays green | +| `wg pubkey rejected it` with none of the above | a mangled value, or a *public* key pasted by mistake | ## Step 3 — forward udp/51820 on the router (once) @@ -153,8 +190,16 @@ unauthenticated traffic either way. If the pod is Ready, the interface exists. ./infra/vpn-ingress/new-peer.sh thinkpad 3 --split # split tunnel (LAN only) ``` -It generates the keypair **on your machine**, writes a `0600` client profile outside the checkout, -prints a QR code if `qrencode` is installed, and prints the `[Peer]` stanza to add. Then: +It generates the keypair **on your machine**, writes a `0600` client profile outside the checkout +(it refuses to write inside the git checkout), prints a QR code, and prints the `[Peer]` stanza to +add. + +> **`qrencode` is installed neither on this workstation nor in the dev container** (measured +> 2026-10-05), so the QR path needs it first: `sudo dnf install qrencode` (or `apt install +> qrencode`). Without it the script says so plainly and leaves you to move the `.conf` to the +> device by another route — fine for a desktop, awkward for a phone. + +Then: 1. Paste the stanza into `platform/vpn-ingress/peers.yaml` under `data.peers.conf`, keeping the `# — , enrolled ` comment. `wg show` can only ever print the key, so that