fix(vpn-ingress): content-hash the peer register, or a merged peer never reaches the tunnel
The operator enrolled a phone, merged the peer, and `wg show` listed none. Measured on prod 2026-10-06 with the peer already on main: live wg-peers ConfigMap, active AllowedIPs .. 0 DaemonSet volume reference ................. configMap.name: wg-peers (FIXED) DaemonSet generation/observedGeneration .... 1 / 1 (never rolled) init container, all three pods ............. "0 peer(s)" Part of it was just the 10m reconcile interval. The defect is the fixed name: Flux updates a fixed-name ConfigMap IN PLACE, so the pod template is byte-identical, so no rollout is triggered -- and the init container assembles wg0.conf ONCE at pod start into a tmpfs, so the running tunnel keeps the old peer list indefinitely. Adding or revoking a peer had no effect until something unrelated restarted the pods. For a REVOCATION that is a security defect, not an inconvenience: delete a stanza, see a green PR, device keeps working. The runbook and README both claimed "PR -> merge -> Flux rolls the DaemonSet". Never true. And this repo already documents the fix, in a file I had read -- runbooks/kustomization.yaml: "The generated name carries kustomize's content hash, so editing any runbook rolls the server automatically and a stale projection is impossible." So: peers.yaml (inline ConfigMap) -> peers.conf (plain file) + a configMapGenerator. The operator's enrolled peer is carried across verbatim. Every line is WireGuard config and `#` is a legal comment there, so the documentation now lives inside the file that is actually mounted. `namespace: vpn-ingress` on the kustomization is REQUIRED, not tidiness: a generated ConfigMap inherits no namespace, everything else here declares vpn-ingress, and kustomize's nameReference will not rewrite across a namespace boundary -- without it the build mounts the unhashed name and the pods wedge on a missing volume. Caught only by building the output and reading the volumes. new-peer.sh: PEERS_FILE repointed, and a missing register is now FATAL rather than skipped -- the old `[ -r "$PEERS_FILE" ] && grep ...` guard meant that after the rename the only symptom would have been duplicate detection quietly never firing again. Stale peers.yaml references swept from six files including the script's own user-facing output. Verified: the DaemonSet now mounts wg-peers-m9kcckcb5t; the hash tracks content (edit -> dgg7ht9c74 -> restore -> m9kcckcb5t); the enrolled peer is byte-identical to main's; and `new-peer.sh dupe 2` is now correctly refused by the operator's real allocation. No sensitive paths; merges on a green gate. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
e1bebb857e
commit
2acfed460a
9 files changed
+226
-65
No files matched your search
@@ -0,0 +1,103 @@
|
||||
---
|
||||
date: 2026-10-06
|
||||
branch: fix/peers-configmap-content-hash
|
||||
title: A merged VPN peer never reached the tunnel — the peer register was a fixed-name ConfigMap, so nothing rolled the pods
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
The operator enrolled a phone, merged the peer, and **`wg show` listed no peers.** Measured on prod
|
||||
2026-10-06, with the peer already on `main`:
|
||||
|
||||
| observation | value |
|
||||
|---|---|
|
||||
| peer stanza on `main` | present (`AllowedIPs = 10.90.0.2/32`) |
|
||||
| GitRepository artifact | `main@sha1:e1bebb8` — Flux had fetched it |
|
||||
| `vpn-ingress` Kustomization `lastAppliedRevision` | `main@sha1:5928d81` — **behind**, 10m interval |
|
||||
| live `wg-peers` ConfigMap, active `AllowedIPs` lines | **0** |
|
||||
| DaemonSet volume reference | `configMap.name: wg-peers` — a **fixed** name |
|
||||
| DaemonSet `generation` / `observedGeneration` | **1 / 1** — it had **never rolled** |
|
||||
| init container, all three pods | `wg0.conf assembled: 0 peer(s)` |
|
||||
|
||||
The lag was only the reconcile interval. **The defect is the fixed name.** When Flux does update a
|
||||
fixed-name ConfigMap it updates it *in place*: the DaemonSet's pod template is byte-identical, so
|
||||
no rollout is triggered — and because the init container assembles `wg0.conf` **once at pod start**
|
||||
into a tmpfs, the running tunnel keeps the old peer list indefinitely. Adding or revoking a peer
|
||||
would have had **no effect at all** until something unrelated happened to restart the pods. For a
|
||||
*revocation* that is a security defect, not an inconvenience: the operator would delete a stanza,
|
||||
see a green PR, and the device would keep working.
|
||||
|
||||
My runbook and README both asserted "PR → merge → Flux rolls the DaemonSet (~1 min)". That was
|
||||
never true.
|
||||
|
||||
**This repo already documents the fix, and I had read the file.** `runbooks/kustomization.yaml`:
|
||||
*"The generated name carries kustomize's content hash, so editing any runbook rolls the server
|
||||
automatically and a stale projection is impossible."*
|
||||
|
||||
So the register becomes a **content-hashed generator**:
|
||||
|
||||
- `platform/vpn-ingress/peers.yaml` (inline ConfigMap) → **`platform/vpn-ingress/peers.conf`**, a
|
||||
plain file, plus a `configMapGenerator` in `kustomization.yaml`. The operator's enrolled peer was
|
||||
carried across verbatim. Every line is WireGuard config and `#` is a legal comment there, so the
|
||||
documentation now travels *inside the mounted file* instead of in a YAML wrapper that gets
|
||||
stripped.
|
||||
- **`namespace: vpn-ingress` added to the kustomization — REQUIRED, not tidiness.** A generated
|
||||
ConfigMap inherits no namespace; every other object here declares `vpn-ingress`; and kustomize's
|
||||
`nameReference` transformer will not rewrite a reference across a namespace boundary. Without it
|
||||
the build emitted `ConfigMap wg-peers-m9kcckcb5t` while the DaemonSet still mounted the unhashed
|
||||
`wg-peers`, which would not exist — the pods would wedge on a missing volume. I caught this only
|
||||
by building the output and reading the volumes block.
|
||||
- **`infra/vpn-ingress/new-peer.sh`**: `PEERS_FILE` repointed to `peers.conf`, and a missing
|
||||
register is now **fatal** instead of skipped. The old guard was
|
||||
`[ -r "$PEERS_FILE" ] && grep …`, so after the rename the only symptom would have been
|
||||
duplicate-address detection quietly never firing again — the silent-skip shape this repo treats
|
||||
as a defect in its own right.
|
||||
- Stale `peers.yaml` references swept from the README, runbook, `daemonset.yaml`,
|
||||
`networkpolicy.yaml`, the SOPS example, and the script's own **user-facing output** (it told the
|
||||
operator to paste into "peers.yaml, under data.peers.conf", which would no longer exist).
|
||||
- The runbook gained the verification that actually distinguishes this failure — the DaemonSet's
|
||||
`generation` must *increase* after the merge — and a troubleshooting row for `0 peer(s)` after a
|
||||
merge.
|
||||
|
||||
Flux's `prune: true` removes the orphaned fixed-name `wg-peers` ConfigMap on the next reconcile.
|
||||
|
||||
No sensitive paths (`platform/vpn-ingress/**`, `infra/vpn-ingress/**`, `runbooks/**`,
|
||||
`decisions/**`), so this merges on a green gate with no override.
|
||||
|
||||
## What I verified
|
||||
|
||||
Built and run in the pinned toolbox, 2026-10-06 (kustomize v5.8.1 via `kubectl kustomize`).
|
||||
|
||||
- **The reference is now rewritten end to end**: generated `ConfigMap wg-peers-m9kcckcb5t` in
|
||||
namespace `vpn-ingress`, and the DaemonSet mounts **`wg-peers-m9kcckcb5t`**. Before adding
|
||||
`namespace:` the same build mounted the unhashed `wg-peers` — so the namespace line is load
|
||||
bearing, measured both ways.
|
||||
- **The hash tracks content, which is the whole mechanism.** `wg-peers-m9kcckcb5t` →
|
||||
append a second peer → `wg-peers-dgg7ht9c74` → restore → `wg-peers-m9kcckcb5t`. Deterministic
|
||||
and reversible, so an edit necessarily changes the pod template.
|
||||
- **The operator's enrolled peer survived the file conversion byte-for-byte**: the active stanza in
|
||||
`peers.conf` (`PublicKey = pBRSHrRy…`, `AllowedIPs = 10.90.0.2/32`) is identical to the one on
|
||||
`main`, compared directly against `git show origin/main:platform/vpn-ingress/peers.yaml`.
|
||||
- **The build still emits all five objects**: Namespace, ConfigMap, Secret, DaemonSet,
|
||||
NetworkPolicy — and the `Namespace` object's own name is still `vpn-ingress` (the namespace
|
||||
transformer did not rewrite it).
|
||||
- **`new-peer.sh` works against the new register**: `spare-laptop 7` renders a profile and prints
|
||||
"append this to platform/vpn-ingress/peers.conf"; `dupe 2` is **refused** because the operator's
|
||||
real `.2` now occupies it — the first time the duplicate check has been exercised against real
|
||||
data rather than against its own example.
|
||||
- **Toolbox gate**: rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`,
|
||||
`l1` and `approval-policy` all pass; rc 3 is the two environmental skips. **Not green. Reported
|
||||
as rc 3.**
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **That the roll actually happens on the cluster.** The whole point of the change is unproven
|
||||
until Flux applies it: the success criterion is the DaemonSet's `generation` increasing and
|
||||
`wg0.conf assembled: 1 peer(s)` in the init log, followed by `wg show` listing the peer.
|
||||
- **That the tunnel then carries traffic.** No handshake has ever been observed. The WAN
|
||||
port-forward is reportedly configured but has never been exercised by a packet I can see, and I
|
||||
have no off-LAN vantage point — runbook step 6 from cellular remains the only honest test.
|
||||
- **Revocation**, which is the case where this defect was a security problem. Deleting a stanza
|
||||
should now roll the pods and drop the peer; untested.
|
||||
- **Whether the orphaned fixed-name `wg-peers` ConfigMap is actually pruned.** Reasoned from
|
||||
`prune: true` on the Kustomization, not observed.
|
||||
Reference in new issue
Block a user