fix(vpn-ingress): content-hash the peer register, or a merged peer never reaches the tunnel
gate / gate (pull_request) Successful in 1m52s
gate / conformance (pull_request) Has been skipped

The operator enrolled a phone, merged the peer, and `wg show` listed none.
Measured on prod 2026-10-06 with the peer already on main:

  live wg-peers ConfigMap, active AllowedIPs .. 0
  DaemonSet volume reference ................. configMap.name: wg-peers (FIXED)
  DaemonSet generation/observedGeneration .... 1 / 1   (never rolled)
  init container, all three pods ............. "0 peer(s)"

Part of it was just the 10m reconcile interval. The defect is the fixed name:
Flux updates a fixed-name ConfigMap IN PLACE, so the pod template is
byte-identical, so no rollout is triggered -- and the init container assembles
wg0.conf ONCE at pod start into a tmpfs, so the running tunnel keeps the old
peer list indefinitely. Adding or revoking a peer had no effect until something
unrelated restarted the pods. For a REVOCATION that is a security defect, not
an inconvenience: delete a stanza, see a green PR, device keeps working.

The runbook and README both claimed "PR -> merge -> Flux rolls the DaemonSet".
Never true. And this repo already documents the fix, in a file I had read --
runbooks/kustomization.yaml: "The generated name carries kustomize's content
hash, so editing any runbook rolls the server automatically and a stale
projection is impossible."

So: peers.yaml (inline ConfigMap) -> peers.conf (plain file) + a
configMapGenerator. The operator's enrolled peer is carried across verbatim.
Every line is WireGuard config and `#` is a legal comment there, so the
documentation now lives inside the file that is actually mounted.

`namespace: vpn-ingress` on the kustomization is REQUIRED, not tidiness: a
generated ConfigMap inherits no namespace, everything else here declares
vpn-ingress, and kustomize's nameReference will not rewrite across a namespace
boundary -- without it the build mounts the unhashed name and the pods wedge on
a missing volume. Caught only by building the output and reading the volumes.

new-peer.sh: PEERS_FILE repointed, and a missing register is now FATAL rather
than skipped -- the old `[ -r "$PEERS_FILE" ] && grep ...` guard meant that
after the rename the only symptom would have been duplicate detection quietly
never firing again. Stale peers.yaml references swept from six files including
the script's own user-facing output.

Verified: the DaemonSet now mounts wg-peers-m9kcckcb5t; the hash tracks content
(edit -> dgg7ht9c74 -> restore -> m9kcckcb5t); the enrolled peer is byte-identical
to main's; and `new-peer.sh dupe 2` is now correctly refused by the operator's
real allocation.

No sensitive paths; merges on a green gate.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-06 02:55:59 -05:00
1 parent e1bebb857e
commit 2acfed460a
9 files changed
+226 -65

No files matched your search

+18 -4
View File
@@ -21,7 +21,7 @@ behaves like a machine on the LAN: full reach to `192.168.2.0/24`, the Gateway's
DNS, and internet egress out the house WAN.
Authorisation is one thing only: the device's **public key is in
`platform/vpn-ingress/peers.yaml`**. An unlisted key gets no reply at all.
`platform/vpn-ingress/peers.conf`**. An unlisted key gets no reply at all.
> **An enrolled device is as privileged as a machine sitting in the house** — including the three
> k3s nodes (`192.168.2.10/.11/.12`: apiserver 6443, kubelet 10250, SSH 22). That is what "reach
@@ -135,6 +135,7 @@ specific failure in its log — so you never get a server that silently cannot c
| `still SOPS CIPHERTEXT (starts with ENC[)` | the Secret reached the cluster undecrypted — this environment's overlay is missing its `vpn-ingress` decryption patch. **This is the dangerous one:** the Secret applies cleanly and every Flux condition stays green |
| `wg pubkey rejected it` with none of the above | a mangled value, or a *public* key pasted by mistake |
| `net.ipv4.ip_forward is 0 on this node` | the node_baseline role has not run here (step 1). The pod **cannot** set this itself — under `hostNetwork` the runtime mounts `/proc/sys` read-only, because it is the host's global setting |
| `0 peer(s)` after you merged a peer | the DaemonSet did not roll. Check its `generation` (above). Pre-2026-10-06 this was guaranteed, because the register was a fixed-name ConfigMap; if it recurs, something has reverted `peers.conf` to a plain resource instead of a `configMapGenerator` |
### What a healthy start looks like
@@ -218,10 +219,23 @@ It generates the device's keypair, writes a `0600` client profile, prints a QR c
Then:
1. Paste the stanza into `platform/vpn-ingress/peers.yaml` under `data.peers.conf`, keeping the
1. Append the stanza to `platform/vpn-ingress/peers.conf`, keeping the
`# <name> — <owner>, enrolled <date>` comment. `wg show` can only ever print the key, so that
comment is the only record of which stanza belongs to which device.
2. PR → merge. Flux rolls the DaemonSet (~1 min).
2. PR → merge. Flux rolls the DaemonSet (~1 min), because `peers.conf` is a
**content-hashed** `configMapGenerator`: editing it changes the ConfigMap's generated name,
which changes the pod template, which forces the roll. (It did **not** work this way at first —
a fixed-name ConfigMap updates in place, the pod template never changes, and the init container
keeps the old peer list forever. Measured on prod 2026-10-06.) Confirm it actually rolled:
```bash
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n vpn-ingress get ds wireguard \
-o jsonpath="{.metadata.generation}{\"/\"}{.status.observedGeneration}{\"\n\"}"'
```
The generation must have **increased** since before the merge. If it is still `1`, the roll did
not happen and `wg show` will list no peer no matter how long you wait.
3. Get the profile onto the device (QR scan for phones; "Import tunnel from file" on
Windows/macOS; `/etc/wireguard/<name>.conf` + `wg-quick@<name>` on Linux).
4. **Delete the profile from your machine**: `shred -u <path>`. It is the only copy of that
@@ -261,7 +275,7 @@ if the router is not delivering the packets, and a `handshake` with no completed
## Revoking a device
Delete its `[Peer]` stanza from `peers.yaml`, PR, merge. Nothing else — the key stops working on the
Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the
next reconcile. **Never re-use that address octet**: an old profile would silently start working
again as a different device.