jakman85andClaude Opus 5
2acfed460a
The operator enrolled a phone, merged the peer, and `wg show` listed none. Measured on prod 2026-10-06 with the peer already on main: live wg-peers ConfigMap, active AllowedIPs .. 0 DaemonSet volume reference ................. configMap.name: wg-peers (FIXED) DaemonSet generation/observedGeneration .... 1 / 1 (never rolled) init container, all three pods ............. "0 peer(s)" Part of it was just the 10m reconcile interval. The defect is the fixed name: Flux updates a fixed-name ConfigMap IN PLACE, so the pod template is byte-identical, so no rollout is triggered -- and the init container assembles wg0.conf ONCE at pod start into a tmpfs, so the running tunnel keeps the old peer list indefinitely. Adding or revoking a peer had no effect until something unrelated restarted the pods. For a REVOCATION that is a security defect, not an inconvenience: delete a stanza, see a green PR, device keeps working. The runbook and README both claimed "PR -> merge -> Flux rolls the DaemonSet". Never true. And this repo already documents the fix, in a file I had read -- runbooks/kustomization.yaml: "The generated name carries kustomize's content hash, so editing any runbook rolls the server automatically and a stale projection is impossible." So: peers.yaml (inline ConfigMap) -> peers.conf (plain file) + a configMapGenerator. The operator's enrolled peer is carried across verbatim. Every line is WireGuard config and `#` is a legal comment there, so the documentation now lives inside the file that is actually mounted. `namespace: vpn-ingress` on the kustomization is REQUIRED, not tidiness: a generated ConfigMap inherits no namespace, everything else here declares vpn-ingress, and kustomize's nameReference will not rewrite across a namespace boundary -- without it the build mounts the unhashed name and the pods wedge on a missing volume. Caught only by building the output and reading the volumes. new-peer.sh: PEERS_FILE repointed, and a missing register is now FATAL rather than skipped -- the old `[ -r "$PEERS_FILE" ] && grep ...` guard meant that after the rename the only symptom would have been duplicate detection quietly never firing again. Stale peers.yaml references swept from six files including the script's own user-facing output. Verified: the DaemonSet now mounts wg-peers-m9kcckcb5t; the hash tracks content (edit -> dgg7ht9c74 -> restore -> m9kcckcb5t); the enrolled peer is byte-identical to main's; and `new-peer.sh dupe 2` is now correctly refused by the operator's real allocation. No sensitive paths; merges on a green gate. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
104 lines
6.5 KiB
Markdown
104 lines
6.5 KiB
Markdown
---
|
|
date: 2026-10-06
|
|
branch: fix/peers-configmap-content-hash
|
|
title: A merged VPN peer never reached the tunnel — the peer register was a fixed-name ConfigMap, so nothing rolled the pods
|
|
---
|
|
|
|
## What changed
|
|
|
|
The operator enrolled a phone, merged the peer, and **`wg show` listed no peers.** Measured on prod
|
|
2026-10-06, with the peer already on `main`:
|
|
|
|
| observation | value |
|
|
|---|---|
|
|
| peer stanza on `main` | present (`AllowedIPs = 10.90.0.2/32`) |
|
|
| GitRepository artifact | `main@sha1:e1bebb8` — Flux had fetched it |
|
|
| `vpn-ingress` Kustomization `lastAppliedRevision` | `main@sha1:5928d81` — **behind**, 10m interval |
|
|
| live `wg-peers` ConfigMap, active `AllowedIPs` lines | **0** |
|
|
| DaemonSet volume reference | `configMap.name: wg-peers` — a **fixed** name |
|
|
| DaemonSet `generation` / `observedGeneration` | **1 / 1** — it had **never rolled** |
|
|
| init container, all three pods | `wg0.conf assembled: 0 peer(s)` |
|
|
|
|
The lag was only the reconcile interval. **The defect is the fixed name.** When Flux does update a
|
|
fixed-name ConfigMap it updates it *in place*: the DaemonSet's pod template is byte-identical, so
|
|
no rollout is triggered — and because the init container assembles `wg0.conf` **once at pod start**
|
|
into a tmpfs, the running tunnel keeps the old peer list indefinitely. Adding or revoking a peer
|
|
would have had **no effect at all** until something unrelated happened to restart the pods. For a
|
|
*revocation* that is a security defect, not an inconvenience: the operator would delete a stanza,
|
|
see a green PR, and the device would keep working.
|
|
|
|
My runbook and README both asserted "PR → merge → Flux rolls the DaemonSet (~1 min)". That was
|
|
never true.
|
|
|
|
**This repo already documents the fix, and I had read the file.** `runbooks/kustomization.yaml`:
|
|
*"The generated name carries kustomize's content hash, so editing any runbook rolls the server
|
|
automatically and a stale projection is impossible."*
|
|
|
|
So the register becomes a **content-hashed generator**:
|
|
|
|
- `platform/vpn-ingress/peers.yaml` (inline ConfigMap) → **`platform/vpn-ingress/peers.conf`**, a
|
|
plain file, plus a `configMapGenerator` in `kustomization.yaml`. The operator's enrolled peer was
|
|
carried across verbatim. Every line is WireGuard config and `#` is a legal comment there, so the
|
|
documentation now travels *inside the mounted file* instead of in a YAML wrapper that gets
|
|
stripped.
|
|
- **`namespace: vpn-ingress` added to the kustomization — REQUIRED, not tidiness.** A generated
|
|
ConfigMap inherits no namespace; every other object here declares `vpn-ingress`; and kustomize's
|
|
`nameReference` transformer will not rewrite a reference across a namespace boundary. Without it
|
|
the build emitted `ConfigMap wg-peers-m9kcckcb5t` while the DaemonSet still mounted the unhashed
|
|
`wg-peers`, which would not exist — the pods would wedge on a missing volume. I caught this only
|
|
by building the output and reading the volumes block.
|
|
- **`infra/vpn-ingress/new-peer.sh`**: `PEERS_FILE` repointed to `peers.conf`, and a missing
|
|
register is now **fatal** instead of skipped. The old guard was
|
|
`[ -r "$PEERS_FILE" ] && grep …`, so after the rename the only symptom would have been
|
|
duplicate-address detection quietly never firing again — the silent-skip shape this repo treats
|
|
as a defect in its own right.
|
|
- Stale `peers.yaml` references swept from the README, runbook, `daemonset.yaml`,
|
|
`networkpolicy.yaml`, the SOPS example, and the script's own **user-facing output** (it told the
|
|
operator to paste into "peers.yaml, under data.peers.conf", which would no longer exist).
|
|
- The runbook gained the verification that actually distinguishes this failure — the DaemonSet's
|
|
`generation` must *increase* after the merge — and a troubleshooting row for `0 peer(s)` after a
|
|
merge.
|
|
|
|
Flux's `prune: true` removes the orphaned fixed-name `wg-peers` ConfigMap on the next reconcile.
|
|
|
|
No sensitive paths (`platform/vpn-ingress/**`, `infra/vpn-ingress/**`, `runbooks/**`,
|
|
`decisions/**`), so this merges on a green gate with no override.
|
|
|
|
## What I verified
|
|
|
|
Built and run in the pinned toolbox, 2026-10-06 (kustomize v5.8.1 via `kubectl kustomize`).
|
|
|
|
- **The reference is now rewritten end to end**: generated `ConfigMap wg-peers-m9kcckcb5t` in
|
|
namespace `vpn-ingress`, and the DaemonSet mounts **`wg-peers-m9kcckcb5t`**. Before adding
|
|
`namespace:` the same build mounted the unhashed `wg-peers` — so the namespace line is load
|
|
bearing, measured both ways.
|
|
- **The hash tracks content, which is the whole mechanism.** `wg-peers-m9kcckcb5t` →
|
|
append a second peer → `wg-peers-dgg7ht9c74` → restore → `wg-peers-m9kcckcb5t`. Deterministic
|
|
and reversible, so an edit necessarily changes the pod template.
|
|
- **The operator's enrolled peer survived the file conversion byte-for-byte**: the active stanza in
|
|
`peers.conf` (`PublicKey = pBRSHrRy…`, `AllowedIPs = 10.90.0.2/32`) is identical to the one on
|
|
`main`, compared directly against `git show origin/main:platform/vpn-ingress/peers.yaml`.
|
|
- **The build still emits all five objects**: Namespace, ConfigMap, Secret, DaemonSet,
|
|
NetworkPolicy — and the `Namespace` object's own name is still `vpn-ingress` (the namespace
|
|
transformer did not rewrite it).
|
|
- **`new-peer.sh` works against the new register**: `spare-laptop 7` renders a profile and prints
|
|
"append this to platform/vpn-ingress/peers.conf"; `dupe 2` is **refused** because the operator's
|
|
real `.2` now occupies it — the first time the duplicate check has been exercised against real
|
|
data rather than against its own example.
|
|
- **Toolbox gate**: rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`,
|
|
`l1` and `approval-policy` all pass; rc 3 is the two environmental skips. **Not green. Reported
|
|
as rc 3.**
|
|
|
|
## What I did NOT verify
|
|
|
|
- **That the roll actually happens on the cluster.** The whole point of the change is unproven
|
|
until Flux applies it: the success criterion is the DaemonSet's `generation` increasing and
|
|
`wg0.conf assembled: 1 peer(s)` in the init log, followed by `wg show` listing the peer.
|
|
- **That the tunnel then carries traffic.** No handshake has ever been observed. The WAN
|
|
port-forward is reportedly configured but has never been exercised by a packet I can see, and I
|
|
have no off-LAN vantage point — runbook step 6 from cellular remains the only honest test.
|
|
- **Revocation**, which is the case where this defect was a security problem. Deleting a stanza
|
|
should now roll the pods and drop the peer; untested.
|
|
- **Whether the orphaned fixed-name `wg-peers` ConfigMap is actually pruned.** Reasoned from
|
|
`prune: true` on the Kustomization, not observed.
|