fix(vpn-ingress): content-hash the peer register, or a merged peer never reaches the tunnel
The operator enrolled a phone, merged the peer, and `wg show` listed none. Measured on prod 2026-10-06 with the peer already on main: live wg-peers ConfigMap, active AllowedIPs .. 0 DaemonSet volume reference ................. configMap.name: wg-peers (FIXED) DaemonSet generation/observedGeneration .... 1 / 1 (never rolled) init container, all three pods ............. "0 peer(s)" Part of it was just the 10m reconcile interval. The defect is the fixed name: Flux updates a fixed-name ConfigMap IN PLACE, so the pod template is byte-identical, so no rollout is triggered -- and the init container assembles wg0.conf ONCE at pod start into a tmpfs, so the running tunnel keeps the old peer list indefinitely. Adding or revoking a peer had no effect until something unrelated restarted the pods. For a REVOCATION that is a security defect, not an inconvenience: delete a stanza, see a green PR, device keeps working. The runbook and README both claimed "PR -> merge -> Flux rolls the DaemonSet". Never true. And this repo already documents the fix, in a file I had read -- runbooks/kustomization.yaml: "The generated name carries kustomize's content hash, so editing any runbook rolls the server automatically and a stale projection is impossible." So: peers.yaml (inline ConfigMap) -> peers.conf (plain file) + a configMapGenerator. The operator's enrolled peer is carried across verbatim. Every line is WireGuard config and `#` is a legal comment there, so the documentation now lives inside the file that is actually mounted. `namespace: vpn-ingress` on the kustomization is REQUIRED, not tidiness: a generated ConfigMap inherits no namespace, everything else here declares vpn-ingress, and kustomize's nameReference will not rewrite across a namespace boundary -- without it the build mounts the unhashed name and the pods wedge on a missing volume. Caught only by building the output and reading the volumes. new-peer.sh: PEERS_FILE repointed, and a missing register is now FATAL rather than skipped -- the old `[ -r "$PEERS_FILE" ] && grep ...` guard meant that after the rename the only symptom would have been duplicate detection quietly never firing again. Stale peers.yaml references swept from six files including the script's own user-facing output. Verified: the DaemonSet now mounts wg-peers-m9kcckcb5t; the hash tracks content (edit -> dgg7ht9c74 -> restore -> m9kcckcb5t); the enrolled peer is byte-identical to main's; and `new-peer.sh dupe 2` is now correctly refused by the operator's real allocation. No sensitive paths; merges on a green gate. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
e1bebb857e
commit
2acfed460a
9 files changed
+226
-65
No files matched your search
@@ -21,7 +21,7 @@ behaves like a machine on the LAN: full reach to `192.168.2.0/24`, the Gateway's
|
||||
DNS, and internet egress out the house WAN.
|
||||
|
||||
Authorisation is one thing only: the device's **public key is in
|
||||
`platform/vpn-ingress/peers.yaml`**. An unlisted key gets no reply at all.
|
||||
`platform/vpn-ingress/peers.conf`**. An unlisted key gets no reply at all.
|
||||
|
||||
> **An enrolled device is as privileged as a machine sitting in the house** — including the three
|
||||
> k3s nodes (`192.168.2.10/.11/.12`: apiserver 6443, kubelet 10250, SSH 22). That is what "reach
|
||||
@@ -135,6 +135,7 @@ specific failure in its log — so you never get a server that silently cannot c
|
||||
| `still SOPS CIPHERTEXT (starts with ENC[)` | the Secret reached the cluster undecrypted — this environment's overlay is missing its `vpn-ingress` decryption patch. **This is the dangerous one:** the Secret applies cleanly and every Flux condition stays green |
|
||||
| `wg pubkey rejected it` with none of the above | a mangled value, or a *public* key pasted by mistake |
|
||||
| `net.ipv4.ip_forward is 0 on this node` | the node_baseline role has not run here (step 1). The pod **cannot** set this itself — under `hostNetwork` the runtime mounts `/proc/sys` read-only, because it is the host's global setting |
|
||||
| `0 peer(s)` after you merged a peer | the DaemonSet did not roll. Check its `generation` (above). Pre-2026-10-06 this was guaranteed, because the register was a fixed-name ConfigMap; if it recurs, something has reverted `peers.conf` to a plain resource instead of a `configMapGenerator` |
|
||||
|
||||
### What a healthy start looks like
|
||||
|
||||
@@ -218,10 +219,23 @@ It generates the device's keypair, writes a `0600` client profile, prints a QR c
|
||||
|
||||
Then:
|
||||
|
||||
1. Paste the stanza into `platform/vpn-ingress/peers.yaml` under `data.peers.conf`, keeping the
|
||||
1. Append the stanza to `platform/vpn-ingress/peers.conf`, keeping the
|
||||
`# <name> — <owner>, enrolled <date>` comment. `wg show` can only ever print the key, so that
|
||||
comment is the only record of which stanza belongs to which device.
|
||||
2. PR → merge. Flux rolls the DaemonSet (~1 min).
|
||||
2. PR → merge. Flux rolls the DaemonSet (~1 min), because `peers.conf` is a
|
||||
**content-hashed** `configMapGenerator`: editing it changes the ConfigMap's generated name,
|
||||
which changes the pod template, which forces the roll. (It did **not** work this way at first —
|
||||
a fixed-name ConfigMap updates in place, the pod template never changes, and the init container
|
||||
keeps the old peer list forever. Measured on prod 2026-10-06.) Confirm it actually rolled:
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress get ds wireguard \
|
||||
-o jsonpath="{.metadata.generation}{\"/\"}{.status.observedGeneration}{\"\n\"}"'
|
||||
```
|
||||
|
||||
The generation must have **increased** since before the merge. If it is still `1`, the roll did
|
||||
not happen and `wg show` will list no peer no matter how long you wait.
|
||||
3. Get the profile onto the device (QR scan for phones; "Import tunnel from file" on
|
||||
Windows/macOS; `/etc/wireguard/<name>.conf` + `wg-quick@<name>` on Linux).
|
||||
4. **Delete the profile from your machine**: `shred -u <path>`. It is the only copy of that
|
||||
@@ -261,7 +275,7 @@ if the router is not delivering the packets, and a `handshake` with no completed
|
||||
|
||||
## Revoking a device
|
||||
|
||||
Delete its `[Peer]` stanza from `peers.yaml`, PR, merge. Nothing else — the key stops working on the
|
||||
Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the
|
||||
next reconcile. **Never re-use that address octet**: an old profile would silently start working
|
||||
again as a different device.
|
||||
|
||||
|
||||
Reference in new issue
Block a user