From 2acfed460ab7a57e28bd66a6c36c7a809e994a0d Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Tue, 6 Oct 2026 02:55:59 -0500 Subject: [PATCH] fix(vpn-ingress): content-hash the peer register, or a merged peer never reaches the tunnel The operator enrolled a phone, merged the peer, and `wg show` listed none. Measured on prod 2026-10-06 with the peer already on main: live wg-peers ConfigMap, active AllowedIPs .. 0 DaemonSet volume reference ................. configMap.name: wg-peers (FIXED) DaemonSet generation/observedGeneration .... 1 / 1 (never rolled) init container, all three pods ............. "0 peer(s)" Part of it was just the 10m reconcile interval. The defect is the fixed name: Flux updates a fixed-name ConfigMap IN PLACE, so the pod template is byte-identical, so no rollout is triggered -- and the init container assembles wg0.conf ONCE at pod start into a tmpfs, so the running tunnel keeps the old peer list indefinitely. Adding or revoking a peer had no effect until something unrelated restarted the pods. For a REVOCATION that is a security defect, not an inconvenience: delete a stanza, see a green PR, device keeps working. The runbook and README both claimed "PR -> merge -> Flux rolls the DaemonSet". Never true. And this repo already documents the fix, in a file I had read -- runbooks/kustomization.yaml: "The generated name carries kustomize's content hash, so editing any runbook rolls the server automatically and a stale projection is impossible." So: peers.yaml (inline ConfigMap) -> peers.conf (plain file) + a configMapGenerator. The operator's enrolled peer is carried across verbatim. Every line is WireGuard config and `#` is a legal comment there, so the documentation now lives inside the file that is actually mounted. `namespace: vpn-ingress` on the kustomization is REQUIRED, not tidiness: a generated ConfigMap inherits no namespace, everything else here declares vpn-ingress, and kustomize's nameReference will not rewrite across a namespace boundary -- without it the build mounts the unhashed name and the pods wedge on a missing volume. Caught only by building the output and reading the volumes. new-peer.sh: PEERS_FILE repointed, and a missing register is now FATAL rather than skipped -- the old `[ -r "$PEERS_FILE" ] && grep ...` guard meant that after the rename the only symptom would have been duplicate detection quietly never firing again. Stale peers.yaml references swept from six files including the script's own user-facing output. Verified: the DaemonSet now mounts wg-peers-m9kcckcb5t; the hash tracks content (edit -> dgg7ht9c74 -> restore -> m9kcckcb5t); the enrolled peer is byte-identical to main's; and `new-peer.sh dupe 2` is now correctly refused by the operator's real allocation. No sensitive paths; merges on a green gate. Co-Authored-By: Claude Opus 5 (1M context) --- ...2026-10-06-peers-configmap-content-hash.md | 103 ++++++++++++++++++ infra/vpn-ingress/new-peer.sh | 25 +++-- platform/vpn-ingress/README.md | 11 +- platform/vpn-ingress/daemonset.yaml | 2 +- platform/vpn-ingress/kustomization.yaml | 33 +++++- platform/vpn-ingress/networkpolicy.yaml | 2 +- .../vpn-ingress/{peers.yaml => peers.conf} | 91 +++++++++------- .../wg-server-key.sops.yaml.example | 2 +- runbooks/vpn-remote-access.md | 22 +++- 9 files changed, 226 insertions(+), 65 deletions(-) create mode 100644 decisions/2026-10-06-peers-configmap-content-hash.md rename platform/vpn-ingress/{peers.yaml => peers.conf} (53%) diff --git a/decisions/2026-10-06-peers-configmap-content-hash.md b/decisions/2026-10-06-peers-configmap-content-hash.md new file mode 100644 index 0000000..0a4dc40 --- /dev/null +++ b/decisions/2026-10-06-peers-configmap-content-hash.md @@ -0,0 +1,103 @@ +--- +date: 2026-10-06 +branch: fix/peers-configmap-content-hash +title: A merged VPN peer never reached the tunnel — the peer register was a fixed-name ConfigMap, so nothing rolled the pods +--- + +## What changed + +The operator enrolled a phone, merged the peer, and **`wg show` listed no peers.** Measured on prod +2026-10-06, with the peer already on `main`: + +| observation | value | +|---|---| +| peer stanza on `main` | present (`AllowedIPs = 10.90.0.2/32`) | +| GitRepository artifact | `main@sha1:e1bebb8` — Flux had fetched it | +| `vpn-ingress` Kustomization `lastAppliedRevision` | `main@sha1:5928d81` — **behind**, 10m interval | +| live `wg-peers` ConfigMap, active `AllowedIPs` lines | **0** | +| DaemonSet volume reference | `configMap.name: wg-peers` — a **fixed** name | +| DaemonSet `generation` / `observedGeneration` | **1 / 1** — it had **never rolled** | +| init container, all three pods | `wg0.conf assembled: 0 peer(s)` | + +The lag was only the reconcile interval. **The defect is the fixed name.** When Flux does update a +fixed-name ConfigMap it updates it *in place*: the DaemonSet's pod template is byte-identical, so +no rollout is triggered — and because the init container assembles `wg0.conf` **once at pod start** +into a tmpfs, the running tunnel keeps the old peer list indefinitely. Adding or revoking a peer +would have had **no effect at all** until something unrelated happened to restart the pods. For a +*revocation* that is a security defect, not an inconvenience: the operator would delete a stanza, +see a green PR, and the device would keep working. + +My runbook and README both asserted "PR → merge → Flux rolls the DaemonSet (~1 min)". That was +never true. + +**This repo already documents the fix, and I had read the file.** `runbooks/kustomization.yaml`: +*"The generated name carries kustomize's content hash, so editing any runbook rolls the server +automatically and a stale projection is impossible."* + +So the register becomes a **content-hashed generator**: + +- `platform/vpn-ingress/peers.yaml` (inline ConfigMap) → **`platform/vpn-ingress/peers.conf`**, a + plain file, plus a `configMapGenerator` in `kustomization.yaml`. The operator's enrolled peer was + carried across verbatim. Every line is WireGuard config and `#` is a legal comment there, so the + documentation now travels *inside the mounted file* instead of in a YAML wrapper that gets + stripped. +- **`namespace: vpn-ingress` added to the kustomization — REQUIRED, not tidiness.** A generated + ConfigMap inherits no namespace; every other object here declares `vpn-ingress`; and kustomize's + `nameReference` transformer will not rewrite a reference across a namespace boundary. Without it + the build emitted `ConfigMap wg-peers-m9kcckcb5t` while the DaemonSet still mounted the unhashed + `wg-peers`, which would not exist — the pods would wedge on a missing volume. I caught this only + by building the output and reading the volumes block. +- **`infra/vpn-ingress/new-peer.sh`**: `PEERS_FILE` repointed to `peers.conf`, and a missing + register is now **fatal** instead of skipped. The old guard was + `[ -r "$PEERS_FILE" ] && grep …`, so after the rename the only symptom would have been + duplicate-address detection quietly never firing again — the silent-skip shape this repo treats + as a defect in its own right. +- Stale `peers.yaml` references swept from the README, runbook, `daemonset.yaml`, + `networkpolicy.yaml`, the SOPS example, and the script's own **user-facing output** (it told the + operator to paste into "peers.yaml, under data.peers.conf", which would no longer exist). +- The runbook gained the verification that actually distinguishes this failure — the DaemonSet's + `generation` must *increase* after the merge — and a troubleshooting row for `0 peer(s)` after a + merge. + +Flux's `prune: true` removes the orphaned fixed-name `wg-peers` ConfigMap on the next reconcile. + +No sensitive paths (`platform/vpn-ingress/**`, `infra/vpn-ingress/**`, `runbooks/**`, +`decisions/**`), so this merges on a green gate with no override. + +## What I verified + +Built and run in the pinned toolbox, 2026-10-06 (kustomize v5.8.1 via `kubectl kustomize`). + +- **The reference is now rewritten end to end**: generated `ConfigMap wg-peers-m9kcckcb5t` in + namespace `vpn-ingress`, and the DaemonSet mounts **`wg-peers-m9kcckcb5t`**. Before adding + `namespace:` the same build mounted the unhashed `wg-peers` — so the namespace line is load + bearing, measured both ways. +- **The hash tracks content, which is the whole mechanism.** `wg-peers-m9kcckcb5t` → + append a second peer → `wg-peers-dgg7ht9c74` → restore → `wg-peers-m9kcckcb5t`. Deterministic + and reversible, so an edit necessarily changes the pod template. +- **The operator's enrolled peer survived the file conversion byte-for-byte**: the active stanza in + `peers.conf` (`PublicKey = pBRSHrRy…`, `AllowedIPs = 10.90.0.2/32`) is identical to the one on + `main`, compared directly against `git show origin/main:platform/vpn-ingress/peers.yaml`. +- **The build still emits all five objects**: Namespace, ConfigMap, Secret, DaemonSet, + NetworkPolicy — and the `Namespace` object's own name is still `vpn-ingress` (the namespace + transformer did not rewrite it). +- **`new-peer.sh` works against the new register**: `spare-laptop 7` renders a profile and prints + "append this to platform/vpn-ingress/peers.conf"; `dupe 2` is **refused** because the operator's + real `.2` now occupies it — the first time the duplicate check has been exercised against real + data rather than against its own example. +- **Toolbox gate**: rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, + `l1` and `approval-policy` all pass; rc 3 is the two environmental skips. **Not green. Reported + as rc 3.** + +## What I did NOT verify + +- **That the roll actually happens on the cluster.** The whole point of the change is unproven + until Flux applies it: the success criterion is the DaemonSet's `generation` increasing and + `wg0.conf assembled: 1 peer(s)` in the init log, followed by `wg show` listing the peer. +- **That the tunnel then carries traffic.** No handshake has ever been observed. The WAN + port-forward is reportedly configured but has never been exercised by a packet I can see, and I + have no off-LAN vantage point — runbook step 6 from cellular remains the only honest test. +- **Revocation**, which is the case where this defect was a security problem. Deleting a stanza + should now roll the pods and drop the peer; untested. +- **Whether the orphaned fixed-name `wg-peers` ConfigMap is actually pruned.** Reasoned from + `prune: true` on the Kustomization, not observed. diff --git a/infra/vpn-ingress/new-peer.sh b/infra/vpn-ingress/new-peer.sh index 00e3cdb..494404d 100755 --- a/infra/vpn-ingress/new-peer.sh +++ b/infra/vpn-ingress/new-peer.sh @@ -3,7 +3,7 @@ # new-peer.sh — enrol one device on the remote-access VPN (platform/vpn-ingress). # # Generates a WireGuard keypair, renders the device's client profile, prints a QR code for a phone, -# and prints the one `[Peer]` stanza to paste into platform/vpn-ingress/peers.yaml. The device's +# and prints the one `[Peer]` stanza to paste into platform/vpn-ingress/peers.conf. The device's # private half is written to ONE 0600 file and nowhere else: not sent to the cluster, not committed, # never logged. # @@ -62,7 +62,7 @@ set -euo pipefail die() { printf 'new-peer.sh: %s\n' "$*" >&2; exit 1; } REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" -PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.yaml" +PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.conf" # --- defaults, each mirroring a value that is DECLARED elsewhere in this repo ------------------- # These are duplicated here rather than parsed out of the YAML on purpose: this script is meant to @@ -134,16 +134,21 @@ CLIENT_ADDR="${CLIENT_BASE}.${OCTET}" # Two devices sharing an address does not error anywhere: the server routes that address to # whichever peer handshook most recently, so BOTH devices appear to work intermittently. # COMMENT LINES ARE NOT ALLOCATIONS. This used to grep the whole file and so matched the -# COMMENTED-OUT EXAMPLE STANZA in peers.yaml, which made `.2` — the first address anyone would +# COMMENTED-OUT EXAMPLE STANZA in the register, which made `.2` — the first address anyone would # ask for — permanently un-allocatable with a message claiming it was already taken. Strip # comments first: a real stanza's `AllowedIPs` line is never commented, while both the template # and every peer's `# — ` label are. -if [ -r "$PEERS_FILE" ] \ - && grep -vE '^[[:space:]]*#' "$PEERS_FILE" | grep -q "AllowedIPs *= *${CLIENT_ADDR}/32"; then - die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet." +# FAIL if the register is missing rather than skipping the check. The path was silently wrong once +# (it still said peers.yaml after the file became peers.conf), and because the old guard was +# `[ -r "$PEERS_FILE" ] && ...` the only symptom would have been duplicate detection quietly +# never firing again — the exact silent-skip shape this repo treats as a defect. +[ -r "$PEERS_FILE" ] || die "cannot read the peer register '$PEERS_FILE'. + Duplicate-address detection cannot run, so this script refuses to issue a profile." +if grep -vE '^[[:space:]]*#' "$PEERS_FILE" | grep -q "AllowedIPs *= *${CLIENT_ADDR}/32"; then + die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.conf. Pick another octet." fi -if [ -r "$PEERS_FILE" ] && grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then - printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.yaml.\n' "$NAME" >&2 +if grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then + printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.conf.\n' "$NAME" >&2 fi # --- output location: 0700, and never inside the repo ------------------------------------------- @@ -210,7 +215,7 @@ cat <, enrolled $(date -u +%Y-%m-%d) [Peer] @@ -229,7 +234,7 @@ cat </dev/null || rm -f ${PROFILE} - The tunnel will not work until the peers.yaml PR is MERGED and Flux has rolled the DaemonSet + The tunnel will not work until the peers.conf PR is MERGED and Flux has rolled the DaemonSet (~1 min). Until then the device handshakes and gets silence, exactly like an unknown key. ------------------------------------------------------------------------------------------ EOF diff --git a/platform/vpn-ingress/README.md b/platform/vpn-ingress/README.md index 6ed96e0..3cfc916 100644 --- a/platform/vpn-ingress/README.md +++ b/platform/vpn-ingress/README.md @@ -13,7 +13,7 @@ egress out the house WAN. | Transport | WireGuard, **udp/51820**, kernel implementation | | Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node | | Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices | -| Authorisation | a device's **public key** is in [`peers.yaml`](./peers.yaml), or it cannot connect | +| Authorisation | a device's **public key** is in [`peers.conf`](./peers.conf), or it cannot connect | | The one secret | the server's private key — **SOPS-encrypted in this tree**, `wg-server-key.sops.yaml` | | Exposure | one hand-made WAN port-forward on the SOHO router, off-repo | | Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) | @@ -129,7 +129,7 @@ translations win here. | half | where | why | |---|---|---| -| peer **public** keys + addresses | [`peers.yaml`](./peers.yaml), **plaintext in git** | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail | +| peer **public** keys + addresses | [`peers.conf`](./peers.conf), **plaintext in git**, as a content-hashed `configMapGenerator` so an edit rolls the pods | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail | | the server's **private** key | `wg-server-key.sops.yaml`, **SOPS-encrypted in git** | the only real secret (R-SEC-01, spec docs/07). Ships with a `REPLACE-ME` placeholder: an agent must not mint a long-lived private key (CLAUDE.md §6), so the operator `sops set`s it in | The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** — @@ -166,8 +166,9 @@ NetworkPolicy applies. ## Operating it * **Enrol a device:** `./infra/vpn-ingress/new-peer.sh ` → paste the printed `[Peer]` - stanza into `peers.yaml` → PR → merge. Phones scan a QR code; desktops import a `.conf`. -* **Revoke a device:** delete its stanza → PR → merge. No other step; the key stops working. + stanza into `peers.conf` → PR → merge. Phones scan a QR code; desktops import a `.conf`. +* **Revoke a device:** delete its stanza from `peers.conf` → PR → merge. No other step; the key + stops working as soon as the pods roll (the generator's content hash guarantees they do). * **Is it up?** `kubectl --context k3s-prod -n vpn-ingress get ds wireguard` — three Ready. * **Who is connected?** `kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show` (`latest handshake` under ~3 min means live). Never dump `wg showconf` — it prints the private key. @@ -211,7 +212,7 @@ right way; one was a defect in my own manifest. covered generically by `KubeDeploymentReplicasMismatch`/`KubePodNotReady` from kube-prometheus-stack. **This is the top follow-up.** * **No preshared keys.** They would add a symmetric post-quantum hedge on top of the Noise - handshake — and would be a real secret, forcing `peers.yaml` into SOPS and costing the + handshake — and would be a real secret, forcing `peers.conf` into SOPS and costing the reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the `wg-server-key` Secret. * **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6 diff --git a/platform/vpn-ingress/daemonset.yaml b/platform/vpn-ingress/daemonset.yaml index fb6aeb6..c1a203d 100644 --- a/platform/vpn-ingress/daemonset.yaml +++ b/platform/vpn-ingress/daemonset.yaml @@ -108,7 +108,7 @@ spec: dnsPolicy: ClusterFirstWithHostNet automountServiceAccountToken: false # ── Assemble the running config: PUBLIC peer register (git) + PRIVATE server key (Secret) ── - # The two halves are deliberately stored apart (peers.yaml explains why) and are only ever + # The two halves are deliberately stored apart (peers.conf explains why) and are only ever # joined here, at pod start, into a tmpfs that is destroyed with the pod. The server's private # key is therefore never written to any node's disk, never committed, and never logged. initContainers: diff --git a/platform/vpn-ingress/kustomization.yaml b/platform/vpn-ingress/kustomization.yaml index 924f9b5..69f4600 100644 --- a/platform/vpn-ingress/kustomization.yaml +++ b/platform/vpn-ingress/kustomization.yaml @@ -11,10 +11,16 @@ # NOT in platform/kustomization.yaml — see clusters/base/vpn-ingress.yaml. apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization +# REQUIRED for the configMapGenerator below to be wired up at all, not merely tidy. A generated +# ConfigMap inherits NO namespace, every other object here declares `vpn-ingress`, and kustomize's +# nameReference transformer will not rewrite a reference across a namespace boundary — so without +# this the DaemonSet keeps mounting the unhashed name `wg-peers`, which does not exist, and the +# pods wedge on a missing volume. Verified by building it both ways (kustomize v5.8.1). +# Same placement as runbooks/kustomization.yaml, which uses `namespace: runbooks` for its own +# content generator. +namespace: vpn-ingress resources: - namespace.yaml - # The peer register: which devices may dial in. PLAINTEXT and reviewable on purpose (see the file). - - peers.yaml # The listener itself: hostNetwork DaemonSet, udp/51820 on every node. - daemonset.yaml # Namespace default-deny baseline. NOTE: does NOT constrain the hostNetwork DaemonSet — the file @@ -38,3 +44,26 @@ resources: # hands Flux the raw ciphertext, the Secret APPLIES CLEANLY full of ENC[...], and nothing in any # Flux condition looks wrong. The init container's `wg pubkey` guard exists partly to turn exactly # that into a named error rather than a silently non-working tunnel. + +# ── THE PEER REGISTER, as a CONTENT-HASHED ConfigMap ───────────────────────────────────────── +# peers.conf (plaintext, reviewable — see the file) becomes the `wg-peers` ConfigMap. It is a +# GENERATOR, not a plain resource, and that distinction is load-bearing rather than stylistic: +# kustomize appends a hash of the content to the generated name and rewrites the DaemonSet's +# volume reference to match, so adding or revoking a peer changes the pod template and ROLLS THE +# PODS automatically. +# +# A fixed-name ConfigMap (which is what this component shipped first) does not do that. Flux +# updates it in place, the pod template is byte-identical, no rollout is triggered — and because +# the init container assembles wg0.conf ONCE at pod start into a tmpfs, the running tunnel keeps +# the old peer list indefinitely. Measured on prod 2026-10-06: a merged peer reached the +# ConfigMap while `wg show` listed no peers and the DaemonSet still read generation=1, having +# never rolled. The runbook's "PR -> merge -> Flux rolls the DaemonSet" was false until this. +# +# DO NOT set `disableNameSuffixHash` here. The hash IS the mechanism. +configMapGenerator: + - name: wg-peers + files: + - peers.conf + options: + labels: + app.kubernetes.io/name: vpn-ingress diff --git a/platform/vpn-ingress/networkpolicy.yaml b/platform/vpn-ingress/networkpolicy.yaml index 7bf8ed8..b7a6991 100644 --- a/platform/vpn-ingress/networkpolicy.yaml +++ b/platform/vpn-ingress/networkpolicy.yaml @@ -19,7 +19,7 @@ # protects this port, and it would be enforcing nothing. So there isn't one. # # WHAT ACTUALLY PROTECTS THE EXPOSED PORT, then, since the CNI does not: -# 1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.yaml gets +# 1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.conf gets # NO REPLY — see README "Why this is safe to expose". This is the real control. # 2. The SOHO router's forward: only udp/51820 is forwarded inbound, to one node. # 3. The blast radius once authenticated is deliberately the whole LAN — that IS the requirement diff --git a/platform/vpn-ingress/peers.yaml b/platform/vpn-ingress/peers.conf similarity index 53% rename from platform/vpn-ingress/peers.yaml rename to platform/vpn-ingress/peers.conf index b110520..f4f048d 100644 --- a/platform/vpn-ingress/peers.yaml +++ b/platform/vpn-ingress/peers.conf @@ -1,4 +1,16 @@ ---- +# THIS FILE IS THE MOUNTED CONTENT. It is turned into the `wg-peers` ConfigMap by +# `configMapGenerator` in kustomization.yaml, whose generated name carries a CONTENT HASH — so +# editing this file changes the ConfigMap name, which changes the DaemonSet's pod template, which +# ROLLS THE PODS. That is the whole reason it is a generator and not an inline ConfigMap manifest: +# a fixed-name ConfigMap updates in place, the pod template never changes, no rollout happens, and +# the init container's wg0.conf (built once at pod start, in tmpfs) keeps the OLD peer list +# forever. Measured 2026-10-06: a merged peer sat in the ConfigMap while `wg show` listed none and +# the DaemonSet reported generation=1 — it had never rolled. Same mechanism and same fix as +# runbooks/kustomization.yaml. +# +# Every line here is WireGuard config. `#` is a legal comment in it, so the documentation below +# travels inside the file that is actually mounted, rather than in a wrapper that gets stripped. +# # ============================================================================================ # THE PEER REGISTER — every device allowed to dial in, in PLAINTEXT, in git, on purpose. # ============================================================================================ @@ -41,43 +53,40 @@ # the Noise handshake and would be a genuine secret — which would force this register into SOPS # and cost the reviewability above. Revisit only if the threat model changes; if it does, the PSK # belongs in the `wg-server-key` Secret, not here. See README "Hardening not taken". -apiVersion: v1 -kind: ConfigMap -metadata: - name: wg-peers - namespace: vpn-ingress - labels: - app.kubernetes.io/name: vpn-ingress -data: - # Appended VERBATIM to the generated [Interface] block, in this order, at pod start. - # - # *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. *** - # An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of - # authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the - # DaemonSet will run and listen, and every handshake will be refused because no key is trusted. - # The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible - # (`wg show` reports an interface with no peers). - # - # COPY THIS TEMPLATE, uncomment, one block per device: - # - # # — , enrolled - # [Peer] - # PublicKey = - # AllowedIPs = 10.90.0./32 - # - # The placeholders above are deliberately NOT valid values. An earlier version of this template - # used a real-looking `10.90.0.2/32`, and new-peer.sh's duplicate-address check — which grepped - # the whole file — matched it, so `.2` could never be allocated and the script insisted it was - # already taken. The check now ignores comments; keeping the example unmatchable too means a - # future grep cannot be fooled either. - # - # Keep the `# — , enrolled ` comment line. It is the only place a key is - # tied to a human-recognisable device; `wg show` can only ever print the key itself, so without - # it an incident responder cannot tell which stanza to delete. - peers.conf: | - # No peers enrolled yet. See the template in this file's comments and - # ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device. - # keith-phone — , enrolled 2026-10-06 - [Peer] - PublicKey = pBRSHrRyG2hnsuZ6FvHfEzF8hOVTF5stHUb/7fW2Km8= - AllowedIPs = 10.90.0.2/32 +# +# name: wg-peers +# namespace: vpn-ingress +# labels: +# app.kubernetes.io/name: vpn-ingress +# Appended VERBATIM to the generated [Interface] block, in this order, at pod start. +# +# *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. *** +# An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of +# authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the +# DaemonSet will run and listen, and every handshake will be refused because no key is trusted. +# The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible +# (`wg show` reports an interface with no peers). +# +# COPY THIS TEMPLATE, uncomment, one block per device: +# +# # — , enrolled +# [Peer] +# PublicKey = +# AllowedIPs = 10.90.0./32 +# +# The placeholders above are deliberately NOT valid values. An earlier version of this template +# used a real-looking `10.90.0.2/32`, and new-peer.sh's duplicate-address check — which grepped +# the whole file — matched it, so `.2` could never be allocated and the script insisted it was +# already taken. The check now ignores comments; keeping the example unmatchable too means a +# future grep cannot be fooled either. +# +# Keep the `# — , enrolled ` comment line. It is the only place a key is +# tied to a human-recognisable device; `wg show` can only ever print the key itself, so without +# it an incident responder cannot tell which stanza to delete. +# +# No peers enrolled yet. See the template in this file's comments and +# ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device. +# keith-phone — , enrolled 2026-10-06 +[Peer] +PublicKey = pBRSHrRyG2hnsuZ6FvHfEzF8hOVTF5stHUb/7fW2Km8= +AllowedIPs = 10.90.0.2/32 diff --git a/platform/vpn-ingress/wg-server-key.sops.yaml.example b/platform/vpn-ingress/wg-server-key.sops.yaml.example index a0c9950..5304849 100644 --- a/platform/vpn-ingress/wg-server-key.sops.yaml.example +++ b/platform/vpn-ingress/wg-server-key.sops.yaml.example @@ -5,7 +5,7 @@ # WHAT THIS IS: the WireGuard SERVER's long-term private key for the remote-access VPN # (platform/vpn-ingress). It is the ONE secret in the whole component. The init container in # daemonset.yaml reads it at `/run/wg-server-key/privatekey` and joins it to the PLAINTEXT peer -# register (peers.yaml) in a tmpfs, so the assembled wg0.conf never touches a node's disk. +# register (peers.conf) in a tmpfs, so the assembled wg0.conf never touches a node's disk. # # WHY IT IS IN GIT, SOPS-ENCRYPTED, AND NOT HAND-CREATED. R-SEC-01 / spec docs/07: secrets live in # source, encrypted, and reach the cluster only through Flux. An out-of-band `kubectl create secret` diff --git a/runbooks/vpn-remote-access.md b/runbooks/vpn-remote-access.md index e280402..9f123ab 100644 --- a/runbooks/vpn-remote-access.md +++ b/runbooks/vpn-remote-access.md @@ -21,7 +21,7 @@ behaves like a machine on the LAN: full reach to `192.168.2.0/24`, the Gateway's DNS, and internet egress out the house WAN. Authorisation is one thing only: the device's **public key is in -`platform/vpn-ingress/peers.yaml`**. An unlisted key gets no reply at all. +`platform/vpn-ingress/peers.conf`**. An unlisted key gets no reply at all. > **An enrolled device is as privileged as a machine sitting in the house** — including the three > k3s nodes (`192.168.2.10/.11/.12`: apiserver 6443, kubelet 10250, SSH 22). That is what "reach @@ -135,6 +135,7 @@ specific failure in its log — so you never get a server that silently cannot c | `still SOPS CIPHERTEXT (starts with ENC[)` | the Secret reached the cluster undecrypted — this environment's overlay is missing its `vpn-ingress` decryption patch. **This is the dangerous one:** the Secret applies cleanly and every Flux condition stays green | | `wg pubkey rejected it` with none of the above | a mangled value, or a *public* key pasted by mistake | | `net.ipv4.ip_forward is 0 on this node` | the node_baseline role has not run here (step 1). The pod **cannot** set this itself — under `hostNetwork` the runtime mounts `/proc/sys` read-only, because it is the host's global setting | +| `0 peer(s)` after you merged a peer | the DaemonSet did not roll. Check its `generation` (above). Pre-2026-10-06 this was guaranteed, because the register was a fixed-name ConfigMap; if it recurs, something has reverted `peers.conf` to a plain resource instead of a `configMapGenerator` | ### What a healthy start looks like @@ -218,10 +219,23 @@ It generates the device's keypair, writes a `0600` client profile, prints a QR c Then: -1. Paste the stanza into `platform/vpn-ingress/peers.yaml` under `data.peers.conf`, keeping the +1. Append the stanza to `platform/vpn-ingress/peers.conf`, keeping the `# — , enrolled ` comment. `wg show` can only ever print the key, so that comment is the only record of which stanza belongs to which device. -2. PR → merge. Flux rolls the DaemonSet (~1 min). +2. PR → merge. Flux rolls the DaemonSet (~1 min), because `peers.conf` is a + **content-hashed** `configMapGenerator`: editing it changes the ConfigMap's generated name, + which changes the pod template, which forces the roll. (It did **not** work this way at first — + a fixed-name ConfigMap updates in place, the pod template never changes, and the init container + keeps the old peer list forever. Measured on prod 2026-10-06.) Confirm it actually rolled: + + ```bash + ./infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n vpn-ingress get ds wireguard \ + -o jsonpath="{.metadata.generation}{\"/\"}{.status.observedGeneration}{\"\n\"}"' + ``` + + The generation must have **increased** since before the merge. If it is still `1`, the roll did + not happen and `wg show` will list no peer no matter how long you wait. 3. Get the profile onto the device (QR scan for phones; "Import tunnel from file" on Windows/macOS; `/etc/wireguard/.conf` + `wg-quick@` on Linux). 4. **Delete the profile from your machine**: `shred -u `. It is the only copy of that @@ -261,7 +275,7 @@ if the router is not delivering the packets, and a `handshake` with no completed ## Revoking a device -Delete its `[Peer]` stanza from `peers.yaml`, PR, merge. Nothing else — the key stops working on the +Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the next reconcile. **Never re-use that address octet**: an old profile would silently start working again as a different device. -- 2.54.0