fix(vpn-ingress): content-hash the peer register, or a merged peer never reaches the tunnel
gate / gate (pull_request) Successful in 1m52s
gate / conformance (pull_request) Has been skipped

The operator enrolled a phone, merged the peer, and `wg show` listed none.
Measured on prod 2026-10-06 with the peer already on main:

  live wg-peers ConfigMap, active AllowedIPs .. 0
  DaemonSet volume reference ................. configMap.name: wg-peers (FIXED)
  DaemonSet generation/observedGeneration .... 1 / 1   (never rolled)
  init container, all three pods ............. "0 peer(s)"

Part of it was just the 10m reconcile interval. The defect is the fixed name:
Flux updates a fixed-name ConfigMap IN PLACE, so the pod template is
byte-identical, so no rollout is triggered -- and the init container assembles
wg0.conf ONCE at pod start into a tmpfs, so the running tunnel keeps the old
peer list indefinitely. Adding or revoking a peer had no effect until something
unrelated restarted the pods. For a REVOCATION that is a security defect, not
an inconvenience: delete a stanza, see a green PR, device keeps working.

The runbook and README both claimed "PR -> merge -> Flux rolls the DaemonSet".
Never true. And this repo already documents the fix, in a file I had read --
runbooks/kustomization.yaml: "The generated name carries kustomize's content
hash, so editing any runbook rolls the server automatically and a stale
projection is impossible."

So: peers.yaml (inline ConfigMap) -> peers.conf (plain file) + a
configMapGenerator. The operator's enrolled peer is carried across verbatim.
Every line is WireGuard config and `#` is a legal comment there, so the
documentation now lives inside the file that is actually mounted.

`namespace: vpn-ingress` on the kustomization is REQUIRED, not tidiness: a
generated ConfigMap inherits no namespace, everything else here declares
vpn-ingress, and kustomize's nameReference will not rewrite across a namespace
boundary -- without it the build mounts the unhashed name and the pods wedge on
a missing volume. Caught only by building the output and reading the volumes.

new-peer.sh: PEERS_FILE repointed, and a missing register is now FATAL rather
than skipped -- the old `[ -r "$PEERS_FILE" ] && grep ...` guard meant that
after the rename the only symptom would have been duplicate detection quietly
never firing again. Stale peers.yaml references swept from six files including
the script's own user-facing output.

Verified: the DaemonSet now mounts wg-peers-m9kcckcb5t; the hash tracks content
(edit -> dgg7ht9c74 -> restore -> m9kcckcb5t); the enrolled peer is byte-identical
to main's; and `new-peer.sh dupe 2` is now correctly refused by the operator's
real allocation.

No sensitive paths; merges on a green gate.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-06 02:55:59 -05:00
1 parent e1bebb857e
commit 2acfed460a
9 files changed
+226 -65

No files matched your search

+15 -10
View File
@@ -3,7 +3,7 @@
# new-peer.sh — enrol one device on the remote-access VPN (platform/vpn-ingress).
#
# Generates a WireGuard keypair, renders the device's client profile, prints a QR code for a phone,
# and prints the one `[Peer]` stanza to paste into platform/vpn-ingress/peers.yaml. The device's
# and prints the one `[Peer]` stanza to paste into platform/vpn-ingress/peers.conf. The device's
# private half is written to ONE 0600 file and nowhere else: not sent to the cluster, not committed,
# never logged.
#
@@ -62,7 +62,7 @@ set -euo pipefail
die() { printf 'new-peer.sh: %s\n' "$*" >&2; exit 1; }
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.yaml"
PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.conf"
# --- defaults, each mirroring a value that is DECLARED elsewhere in this repo -------------------
# These are duplicated here rather than parsed out of the YAML on purpose: this script is meant to
@@ -134,16 +134,21 @@ CLIENT_ADDR="${CLIENT_BASE}.${OCTET}"
# Two devices sharing an address does not error anywhere: the server routes that address to
# whichever peer handshook most recently, so BOTH devices appear to work intermittently.
# COMMENT LINES ARE NOT ALLOCATIONS. This used to grep the whole file and so matched the
# COMMENTED-OUT EXAMPLE STANZA in peers.yaml, which made `.2` — the first address anyone would
# COMMENTED-OUT EXAMPLE STANZA in the register, which made `.2` — the first address anyone would
# ask for — permanently un-allocatable with a message claiming it was already taken. Strip
# comments first: a real stanza's `AllowedIPs` line is never commented, while both the template
# and every peer's `# <name> — <owner>` label are.
if [ -r "$PEERS_FILE" ] \
&& grep -vE '^[[:space:]]*#' "$PEERS_FILE" | grep -q "AllowedIPs *= *${CLIENT_ADDR}/32"; then
die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet."
# FAIL if the register is missing rather than skipping the check. The path was silently wrong once
# (it still said peers.yaml after the file became peers.conf), and because the old guard was
# `[ -r "$PEERS_FILE" ] && ...` the only symptom would have been duplicate detection quietly
# never firing again — the exact silent-skip shape this repo treats as a defect.
[ -r "$PEERS_FILE" ] || die "cannot read the peer register '$PEERS_FILE'.
Duplicate-address detection cannot run, so this script refuses to issue a profile."
if grep -vE '^[[:space:]]*#' "$PEERS_FILE" | grep -q "AllowedIPs *= *${CLIENT_ADDR}/32"; then
die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.conf. Pick another octet."
fi
if [ -r "$PEERS_FILE" ] && grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.yaml.\n' "$NAME" >&2
if grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.conf.\n' "$NAME" >&2
fi
# --- output location: 0700, and never inside the repo -------------------------------------------
@@ -210,7 +215,7 @@ cat <<EOF
Profile : ${PROFILE} (0600; contains the private key)
------------------------------------------------------------------------------------------
STEP 1 — add this to platform/vpn-ingress/peers.yaml, under data.peers.conf, then PR it:
STEP 1 — append this to platform/vpn-ingress/peers.conf, then PR it:
# ${NAME} — <owner>, enrolled $(date -u +%Y-%m-%d)
[Peer]
@@ -229,7 +234,7 @@ cat <<EOF
STEP 3 — delete the profile from this machine once the device has it:
shred -u ${PROFILE} 2>/dev/null || rm -f ${PROFILE}
The tunnel will not work until the peers.yaml PR is MERGED and Flux has rolled the DaemonSet
The tunnel will not work until the peers.conf PR is MERGED and Flux has rolled the DaemonSet
(~1 min). Until then the device handshakes and gets silence, exactly like an unknown key.
------------------------------------------------------------------------------------------
EOF