fix(vpn-ingress): content-hash the peer register, or a merged peer never reaches the tunnel
gate / gate (pull_request) Successful in 1m52s
gate / conformance (pull_request) Has been skipped

The operator enrolled a phone, merged the peer, and `wg show` listed none.
Measured on prod 2026-10-06 with the peer already on main:

  live wg-peers ConfigMap, active AllowedIPs .. 0
  DaemonSet volume reference ................. configMap.name: wg-peers (FIXED)
  DaemonSet generation/observedGeneration .... 1 / 1   (never rolled)
  init container, all three pods ............. "0 peer(s)"

Part of it was just the 10m reconcile interval. The defect is the fixed name:
Flux updates a fixed-name ConfigMap IN PLACE, so the pod template is
byte-identical, so no rollout is triggered -- and the init container assembles
wg0.conf ONCE at pod start into a tmpfs, so the running tunnel keeps the old
peer list indefinitely. Adding or revoking a peer had no effect until something
unrelated restarted the pods. For a REVOCATION that is a security defect, not
an inconvenience: delete a stanza, see a green PR, device keeps working.

The runbook and README both claimed "PR -> merge -> Flux rolls the DaemonSet".
Never true. And this repo already documents the fix, in a file I had read --
runbooks/kustomization.yaml: "The generated name carries kustomize's content
hash, so editing any runbook rolls the server automatically and a stale
projection is impossible."

So: peers.yaml (inline ConfigMap) -> peers.conf (plain file) + a
configMapGenerator. The operator's enrolled peer is carried across verbatim.
Every line is WireGuard config and `#` is a legal comment there, so the
documentation now lives inside the file that is actually mounted.

`namespace: vpn-ingress` on the kustomization is REQUIRED, not tidiness: a
generated ConfigMap inherits no namespace, everything else here declares
vpn-ingress, and kustomize's nameReference will not rewrite across a namespace
boundary -- without it the build mounts the unhashed name and the pods wedge on
a missing volume. Caught only by building the output and reading the volumes.

new-peer.sh: PEERS_FILE repointed, and a missing register is now FATAL rather
than skipped -- the old `[ -r "$PEERS_FILE" ] && grep ...` guard meant that
after the rename the only symptom would have been duplicate detection quietly
never firing again. Stale peers.yaml references swept from six files including
the script's own user-facing output.

Verified: the DaemonSet now mounts wg-peers-m9kcckcb5t; the hash tracks content
(edit -> dgg7ht9c74 -> restore -> m9kcckcb5t); the enrolled peer is byte-identical
to main's; and `new-peer.sh dupe 2` is now correctly refused by the operator's
real allocation.

No sensitive paths; merges on a green gate.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-06 02:55:59 -05:00
1 parent e1bebb857e
commit 2acfed460a
9 files changed
+226 -65

No files matched your search

@@ -0,0 +1,103 @@
---
date: 2026-10-06
branch: fix/peers-configmap-content-hash
title: A merged VPN peer never reached the tunnel — the peer register was a fixed-name ConfigMap, so nothing rolled the pods
---
## What changed
The operator enrolled a phone, merged the peer, and **`wg show` listed no peers.** Measured on prod
2026-10-06, with the peer already on `main`:
| observation | value |
|---|---|
| peer stanza on `main` | present (`AllowedIPs = 10.90.0.2/32`) |
| GitRepository artifact | `main@sha1:e1bebb8` — Flux had fetched it |
| `vpn-ingress` Kustomization `lastAppliedRevision` | `main@sha1:5928d81` — **behind**, 10m interval |
| live `wg-peers` ConfigMap, active `AllowedIPs` lines | **0** |
| DaemonSet volume reference | `configMap.name: wg-peers` — a **fixed** name |
| DaemonSet `generation` / `observedGeneration` | **1 / 1** — it had **never rolled** |
| init container, all three pods | `wg0.conf assembled: 0 peer(s)` |
The lag was only the reconcile interval. **The defect is the fixed name.** When Flux does update a
fixed-name ConfigMap it updates it *in place*: the DaemonSet's pod template is byte-identical, so
no rollout is triggered — and because the init container assembles `wg0.conf` **once at pod start**
into a tmpfs, the running tunnel keeps the old peer list indefinitely. Adding or revoking a peer
would have had **no effect at all** until something unrelated happened to restart the pods. For a
*revocation* that is a security defect, not an inconvenience: the operator would delete a stanza,
see a green PR, and the device would keep working.
My runbook and README both asserted "PR → merge → Flux rolls the DaemonSet (~1 min)". That was
never true.
**This repo already documents the fix, and I had read the file.** `runbooks/kustomization.yaml`:
*"The generated name carries kustomize's content hash, so editing any runbook rolls the server
automatically and a stale projection is impossible."*
So the register becomes a **content-hashed generator**:
- `platform/vpn-ingress/peers.yaml` (inline ConfigMap) → **`platform/vpn-ingress/peers.conf`**, a
plain file, plus a `configMapGenerator` in `kustomization.yaml`. The operator's enrolled peer was
carried across verbatim. Every line is WireGuard config and `#` is a legal comment there, so the
documentation now travels *inside the mounted file* instead of in a YAML wrapper that gets
stripped.
- **`namespace: vpn-ingress` added to the kustomization — REQUIRED, not tidiness.** A generated
ConfigMap inherits no namespace; every other object here declares `vpn-ingress`; and kustomize's
`nameReference` transformer will not rewrite a reference across a namespace boundary. Without it
the build emitted `ConfigMap wg-peers-m9kcckcb5t` while the DaemonSet still mounted the unhashed
`wg-peers`, which would not exist — the pods would wedge on a missing volume. I caught this only
by building the output and reading the volumes block.
- **`infra/vpn-ingress/new-peer.sh`**: `PEERS_FILE` repointed to `peers.conf`, and a missing
register is now **fatal** instead of skipped. The old guard was
`[ -r "$PEERS_FILE" ] && grep …`, so after the rename the only symptom would have been
duplicate-address detection quietly never firing again — the silent-skip shape this repo treats
as a defect in its own right.
- Stale `peers.yaml` references swept from the README, runbook, `daemonset.yaml`,
`networkpolicy.yaml`, the SOPS example, and the script's own **user-facing output** (it told the
operator to paste into "peers.yaml, under data.peers.conf", which would no longer exist).
- The runbook gained the verification that actually distinguishes this failure — the DaemonSet's
`generation` must *increase* after the merge — and a troubleshooting row for `0 peer(s)` after a
merge.
Flux's `prune: true` removes the orphaned fixed-name `wg-peers` ConfigMap on the next reconcile.
No sensitive paths (`platform/vpn-ingress/**`, `infra/vpn-ingress/**`, `runbooks/**`,
`decisions/**`), so this merges on a green gate with no override.
## What I verified
Built and run in the pinned toolbox, 2026-10-06 (kustomize v5.8.1 via `kubectl kustomize`).
- **The reference is now rewritten end to end**: generated `ConfigMap wg-peers-m9kcckcb5t` in
namespace `vpn-ingress`, and the DaemonSet mounts **`wg-peers-m9kcckcb5t`**. Before adding
`namespace:` the same build mounted the unhashed `wg-peers` — so the namespace line is load
bearing, measured both ways.
- **The hash tracks content, which is the whole mechanism.** `wg-peers-m9kcckcb5t` →
append a second peer → `wg-peers-dgg7ht9c74` → restore → `wg-peers-m9kcckcb5t`. Deterministic
and reversible, so an edit necessarily changes the pod template.
- **The operator's enrolled peer survived the file conversion byte-for-byte**: the active stanza in
`peers.conf` (`PublicKey = pBRSHrRy…`, `AllowedIPs = 10.90.0.2/32`) is identical to the one on
`main`, compared directly against `git show origin/main:platform/vpn-ingress/peers.yaml`.
- **The build still emits all five objects**: Namespace, ConfigMap, Secret, DaemonSet,
NetworkPolicy — and the `Namespace` object's own name is still `vpn-ingress` (the namespace
transformer did not rewrite it).
- **`new-peer.sh` works against the new register**: `spare-laptop 7` renders a profile and prints
"append this to platform/vpn-ingress/peers.conf"; `dupe 2` is **refused** because the operator's
real `.2` now occupies it — the first time the duplicate check has been exercised against real
data rather than against its own example.
- **Toolbox gate**: rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`,
`l1` and `approval-policy` all pass; rc 3 is the two environmental skips. **Not green. Reported
as rc 3.**
## What I did NOT verify
- **That the roll actually happens on the cluster.** The whole point of the change is unproven
until Flux applies it: the success criterion is the DaemonSet's `generation` increasing and
`wg0.conf assembled: 1 peer(s)` in the init log, followed by `wg show` listing the peer.
- **That the tunnel then carries traffic.** No handshake has ever been observed. The WAN
port-forward is reportedly configured but has never been exercised by a packet I can see, and I
have no off-LAN vantage point — runbook step 6 from cellular remains the only honest test.
- **Revocation**, which is the case where this defect was a security problem. Deleting a stanza
should now roll the pods and drop the peer; untested.
- **Whether the orphaned fixed-name `wg-peers` ConfigMap is actually pruned.** Reasoned from
`prune: true` on the Kustomization, not observed.