fix(vpn-ingress): content-hash the peer register, or a merged peer never reaches the tunnel
The operator enrolled a phone, merged the peer, and `wg show` listed none. Measured on prod 2026-10-06 with the peer already on main: live wg-peers ConfigMap, active AllowedIPs .. 0 DaemonSet volume reference ................. configMap.name: wg-peers (FIXED) DaemonSet generation/observedGeneration .... 1 / 1 (never rolled) init container, all three pods ............. "0 peer(s)" Part of it was just the 10m reconcile interval. The defect is the fixed name: Flux updates a fixed-name ConfigMap IN PLACE, so the pod template is byte-identical, so no rollout is triggered -- and the init container assembles wg0.conf ONCE at pod start into a tmpfs, so the running tunnel keeps the old peer list indefinitely. Adding or revoking a peer had no effect until something unrelated restarted the pods. For a REVOCATION that is a security defect, not an inconvenience: delete a stanza, see a green PR, device keeps working. The runbook and README both claimed "PR -> merge -> Flux rolls the DaemonSet". Never true. And this repo already documents the fix, in a file I had read -- runbooks/kustomization.yaml: "The generated name carries kustomize's content hash, so editing any runbook rolls the server automatically and a stale projection is impossible." So: peers.yaml (inline ConfigMap) -> peers.conf (plain file) + a configMapGenerator. The operator's enrolled peer is carried across verbatim. Every line is WireGuard config and `#` is a legal comment there, so the documentation now lives inside the file that is actually mounted. `namespace: vpn-ingress` on the kustomization is REQUIRED, not tidiness: a generated ConfigMap inherits no namespace, everything else here declares vpn-ingress, and kustomize's nameReference will not rewrite across a namespace boundary -- without it the build mounts the unhashed name and the pods wedge on a missing volume. Caught only by building the output and reading the volumes. new-peer.sh: PEERS_FILE repointed, and a missing register is now FATAL rather than skipped -- the old `[ -r "$PEERS_FILE" ] && grep ...` guard meant that after the rename the only symptom would have been duplicate detection quietly never firing again. Stale peers.yaml references swept from six files including the script's own user-facing output. Verified: the DaemonSet now mounts wg-peers-m9kcckcb5t; the hash tracks content (edit -> dgg7ht9c74 -> restore -> m9kcckcb5t); the enrolled peer is byte-identical to main's; and `new-peer.sh dupe 2` is now correctly refused by the operator's real allocation. No sensitive paths; merges on a green gate. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
e1bebb857e
commit
2acfed460a
9 files changed
+226
-65
No files matched your search
@@ -11,10 +11,16 @@
|
||||
# NOT in platform/kustomization.yaml — see clusters/base/vpn-ingress.yaml.
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
# REQUIRED for the configMapGenerator below to be wired up at all, not merely tidy. A generated
|
||||
# ConfigMap inherits NO namespace, every other object here declares `vpn-ingress`, and kustomize's
|
||||
# nameReference transformer will not rewrite a reference across a namespace boundary — so without
|
||||
# this the DaemonSet keeps mounting the unhashed name `wg-peers`, which does not exist, and the
|
||||
# pods wedge on a missing volume. Verified by building it both ways (kustomize v5.8.1).
|
||||
# Same placement as runbooks/kustomization.yaml, which uses `namespace: runbooks` for its own
|
||||
# content generator.
|
||||
namespace: vpn-ingress
|
||||
resources:
|
||||
- namespace.yaml
|
||||
# The peer register: which devices may dial in. PLAINTEXT and reviewable on purpose (see the file).
|
||||
- peers.yaml
|
||||
# The listener itself: hostNetwork DaemonSet, udp/51820 on every node.
|
||||
- daemonset.yaml
|
||||
# Namespace default-deny baseline. NOTE: does NOT constrain the hostNetwork DaemonSet — the file
|
||||
@@ -38,3 +44,26 @@ resources:
|
||||
# hands Flux the raw ciphertext, the Secret APPLIES CLEANLY full of ENC[...], and nothing in any
|
||||
# Flux condition looks wrong. The init container's `wg pubkey` guard exists partly to turn exactly
|
||||
# that into a named error rather than a silently non-working tunnel.
|
||||
|
||||
# ── THE PEER REGISTER, as a CONTENT-HASHED ConfigMap ─────────────────────────────────────────
|
||||
# peers.conf (plaintext, reviewable — see the file) becomes the `wg-peers` ConfigMap. It is a
|
||||
# GENERATOR, not a plain resource, and that distinction is load-bearing rather than stylistic:
|
||||
# kustomize appends a hash of the content to the generated name and rewrites the DaemonSet's
|
||||
# volume reference to match, so adding or revoking a peer changes the pod template and ROLLS THE
|
||||
# PODS automatically.
|
||||
#
|
||||
# A fixed-name ConfigMap (which is what this component shipped first) does not do that. Flux
|
||||
# updates it in place, the pod template is byte-identical, no rollout is triggered — and because
|
||||
# the init container assembles wg0.conf ONCE at pod start into a tmpfs, the running tunnel keeps
|
||||
# the old peer list indefinitely. Measured on prod 2026-10-06: a merged peer reached the
|
||||
# ConfigMap while `wg show` listed no peers and the DaemonSet still read generation=1, having
|
||||
# never rolled. The runbook's "PR -> merge -> Flux rolls the DaemonSet" was false until this.
|
||||
#
|
||||
# DO NOT set `disableNameSuffixHash` here. The hash IS the mechanism.
|
||||
configMapGenerator:
|
||||
- name: wg-peers
|
||||
files:
|
||||
- peers.conf
|
||||
options:
|
||||
labels:
|
||||
app.kubernetes.io/name: vpn-ingress
|
||||
Reference in new issue
Block a user