fix(vpn-ingress): content-hash the peer register, or a merged peer never reaches the tunnel
The operator enrolled a phone, merged the peer, and `wg show` listed none. Measured on prod 2026-10-06 with the peer already on main: live wg-peers ConfigMap, active AllowedIPs .. 0 DaemonSet volume reference ................. configMap.name: wg-peers (FIXED) DaemonSet generation/observedGeneration .... 1 / 1 (never rolled) init container, all three pods ............. "0 peer(s)" Part of it was just the 10m reconcile interval. The defect is the fixed name: Flux updates a fixed-name ConfigMap IN PLACE, so the pod template is byte-identical, so no rollout is triggered -- and the init container assembles wg0.conf ONCE at pod start into a tmpfs, so the running tunnel keeps the old peer list indefinitely. Adding or revoking a peer had no effect until something unrelated restarted the pods. For a REVOCATION that is a security defect, not an inconvenience: delete a stanza, see a green PR, device keeps working. The runbook and README both claimed "PR -> merge -> Flux rolls the DaemonSet". Never true. And this repo already documents the fix, in a file I had read -- runbooks/kustomization.yaml: "The generated name carries kustomize's content hash, so editing any runbook rolls the server automatically and a stale projection is impossible." So: peers.yaml (inline ConfigMap) -> peers.conf (plain file) + a configMapGenerator. The operator's enrolled peer is carried across verbatim. Every line is WireGuard config and `#` is a legal comment there, so the documentation now lives inside the file that is actually mounted. `namespace: vpn-ingress` on the kustomization is REQUIRED, not tidiness: a generated ConfigMap inherits no namespace, everything else here declares vpn-ingress, and kustomize's nameReference will not rewrite across a namespace boundary -- without it the build mounts the unhashed name and the pods wedge on a missing volume. Caught only by building the output and reading the volumes. new-peer.sh: PEERS_FILE repointed, and a missing register is now FATAL rather than skipped -- the old `[ -r "$PEERS_FILE" ] && grep ...` guard meant that after the rename the only symptom would have been duplicate detection quietly never firing again. Stale peers.yaml references swept from six files including the script's own user-facing output. Verified: the DaemonSet now mounts wg-peers-m9kcckcb5t; the hash tracks content (edit -> dgg7ht9c74 -> restore -> m9kcckcb5t); the enrolled peer is byte-identical to main's; and `new-peer.sh dupe 2` is now correctly refused by the operator's real allocation. No sensitive paths; merges on a green gate. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
e1bebb857e
commit
2acfed460a
9 files changed
+226
-65
No files matched your search
@@ -13,7 +13,7 @@ egress out the house WAN.
|
||||
| Transport | WireGuard, **udp/51820**, kernel implementation |
|
||||
| Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node |
|
||||
| Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices |
|
||||
| Authorisation | a device's **public key** is in [`peers.yaml`](./peers.yaml), or it cannot connect |
|
||||
| Authorisation | a device's **public key** is in [`peers.conf`](./peers.conf), or it cannot connect |
|
||||
| The one secret | the server's private key — **SOPS-encrypted in this tree**, `wg-server-key.sops.yaml` |
|
||||
| Exposure | one hand-made WAN port-forward on the SOHO router, off-repo |
|
||||
| Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) |
|
||||
@@ -129,7 +129,7 @@ translations win here.
|
||||
|
||||
| half | where | why |
|
||||
|---|---|---|
|
||||
| peer **public** keys + addresses | [`peers.yaml`](./peers.yaml), **plaintext in git** | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
|
||||
| peer **public** keys + addresses | [`peers.conf`](./peers.conf), **plaintext in git**, as a content-hashed `configMapGenerator` so an edit rolls the pods | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
|
||||
| the server's **private** key | `wg-server-key.sops.yaml`, **SOPS-encrypted in git** | the only real secret (R-SEC-01, spec docs/07). Ships with a `REPLACE-ME` placeholder: an agent must not mint a long-lived private key (CLAUDE.md §6), so the operator `sops set`s it in |
|
||||
|
||||
The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** —
|
||||
@@ -166,8 +166,9 @@ NetworkPolicy applies.
|
||||
## Operating it
|
||||
|
||||
* **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
|
||||
stanza into `peers.yaml` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
|
||||
* **Revoke a device:** delete its stanza → PR → merge. No other step; the key stops working.
|
||||
stanza into `peers.conf` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
|
||||
* **Revoke a device:** delete its stanza from `peers.conf` → PR → merge. No other step; the key
|
||||
stops working as soon as the pods roll (the generator's content hash guarantees they do).
|
||||
* **Is it up?** `kubectl --context k3s-prod -n vpn-ingress get ds wireguard` — three Ready.
|
||||
* **Who is connected?** `kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show`
|
||||
(`latest handshake` under ~3 min means live). Never dump `wg showconf` — it prints the private key.
|
||||
@@ -211,7 +212,7 @@ right way; one was a defect in my own manifest.
|
||||
covered generically by `KubeDeploymentReplicasMismatch`/`KubePodNotReady` from
|
||||
kube-prometheus-stack. **This is the top follow-up.**
|
||||
* **No preshared keys.** They would add a symmetric post-quantum hedge on top of the Noise
|
||||
handshake — and would be a real secret, forcing `peers.yaml` into SOPS and costing the
|
||||
handshake — and would be a real secret, forcing `peers.conf` into SOPS and costing the
|
||||
reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
|
||||
`wg-server-key` Secret.
|
||||
* **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6
|
||||
|
||||
Reference in new issue
Block a user