home-network-k3s-impl/decisions/2026-10-06-peers-configmap-content-hash.md

date, branch, title

date	branch	title
2026-10-06	fix/peers-configmap-content-hash	A merged VPN peer never reached the tunnel — the peer register was a fixed-name ConfigMap, so nothing rolled the pods

What changed

The operator enrolled a phone, merged the peer, and wg show listed no peers. Measured on prod 2026-10-06, with the peer already on main:

observation	value
peer stanza on main	present (AllowedIPs = 10.90.0.2/32)
GitRepository artifact	main@sha1:e1bebb8 — Flux had fetched it
vpn-ingress Kustomization lastAppliedRevision	main@sha1:5928d81 — behind, 10m interval
live wg-peers ConfigMap, active AllowedIPs lines	0
DaemonSet volume reference	configMap.name: wg-peers — a fixed name
DaemonSet generation / observedGeneration	1 / 1 — it had never rolled
init container, all three pods	wg0.conf assembled: 0 peer(s)

The lag was only the reconcile interval. The defect is the fixed name. When Flux does update a fixed-name ConfigMap it updates it in place: the DaemonSet's pod template is byte-identical, so no rollout is triggered — and because the init container assembles wg0.conf once at pod start into a tmpfs, the running tunnel keeps the old peer list indefinitely. Adding or revoking a peer would have had no effect at all until something unrelated happened to restart the pods. For a revocation that is a security defect, not an inconvenience: the operator would delete a stanza, see a green PR, and the device would keep working.

My runbook and README both asserted "PR → merge → Flux rolls the DaemonSet (~1 min)". That was never true.

This repo already documents the fix, and I had read the file. runbooks/kustomization.yaml: "The generated name carries kustomize's content hash, so editing any runbook rolls the server automatically and a stale projection is impossible."

So the register becomes a content-hashed generator:

• platform/vpn-ingress/peers.yaml (inline ConfigMap) → platform/vpn-ingress/peers.conf, a plain file, plus a configMapGenerator in kustomization.yaml. The operator's enrolled peer was carried across verbatim. Every line is WireGuard config and # is a legal comment there, so the documentation now travels inside the mounted file instead of in a YAML wrapper that gets stripped.
• namespace: vpn-ingress added to the kustomization — REQUIRED, not tidiness. A generated ConfigMap inherits no namespace; every other object here declares vpn-ingress; and kustomize's nameReference transformer will not rewrite a reference across a namespace boundary. Without it the build emitted ConfigMap wg-peers-m9kcckcb5t while the DaemonSet still mounted the unhashed wg-peers, which would not exist — the pods would wedge on a missing volume. I caught this only by building the output and reading the volumes block.
• infra/vpn-ingress/new-peer.sh: PEERS_FILE repointed to peers.conf, and a missing register is now fatal instead of skipped. The old guard was [ -r "$PEERS_FILE" ] && grep …, so after the rename the only symptom would have been duplicate-address detection quietly never firing again — the silent-skip shape this repo treats as a defect in its own right.
• Stale peers.yaml references swept from the README, runbook, daemonset.yaml, networkpolicy.yaml, the SOPS example, and the script's own user-facing output (it told the operator to paste into "peers.yaml, under data.peers.conf", which would no longer exist).
• The runbook gained the verification that actually distinguishes this failure — the DaemonSet's generation must increase after the merge — and a troubleshooting row for 0 peer(s) after a merge.

Flux's prune: true removes the orphaned fixed-name wg-peers ConfigMap on the next reconcile.

No sensitive paths (platform/vpn-ingress/**, infra/vpn-ingress/**, runbooks/**, decisions/**), so this merges on a green gate with no override.

What I verified

Built and run in the pinned toolbox, 2026-10-06 (kustomize v5.8.1 via kubectl kustomize).

• The reference is now rewritten end to end: generated ConfigMap wg-peers-m9kcckcb5t in namespace vpn-ingress, and the DaemonSet mounts wg-peers-m9kcckcb5t. Before adding namespace: the same build mounted the unhashed wg-peers — so the namespace line is load bearing, measured both ways.
• The hash tracks content, which is the whole mechanism. wg-peers-m9kcckcb5t → append a second peer → wg-peers-dgg7ht9c74 → restore → wg-peers-m9kcckcb5t. Deterministic and reversible, so an edit necessarily changes the pod template.
• The operator's enrolled peer survived the file conversion byte-for-byte: the active stanza in peers.conf (PublicKey = pBRSHrRy…, AllowedIPs = 10.90.0.2/32) is identical to the one on main, compared directly against git show origin/main:platform/vpn-ingress/peers.yaml.
• The build still emits all five objects: Namespace, ConfigMap, Secret, DaemonSet, NetworkPolicy — and the Namespace object's own name is still vpn-ingress (the namespace transformer did not rewrite it).
• new-peer.sh works against the new register: spare-laptop 7 renders a profile and prints "append this to platform/vpn-ingress/peers.conf"; dupe 2 is refused because the operator's real .2 now occupies it — the first time the duplicate check has been exercised against real data rather than against its own example.
• Toolbox gate: rc 3, with spec-immutability, decisions-log, lint, l0, render-golden, l1 and approval-policy all pass; rc 3 is the two environmental skips. Not green. Reported as rc 3.

What I did NOT verify

• That the roll actually happens on the cluster. The whole point of the change is unproven until Flux applies it: the success criterion is the DaemonSet's generation increasing and wg0.conf assembled: 1 peer(s) in the init log, followed by wg show listing the peer.
• That the tunnel then carries traffic. No handshake has ever been observed. The WAN port-forward is reportedly configured but has never been exercised by a packet I can see, and I have no off-LAN vantage point — runbook step 6 from cellular remains the only honest test.
• Revocation, which is the case where this defect was a security problem. Deleting a stanza should now roll the pods and drop the peer; untested.
• Whether the orphaned fixed-name wg-peers ConfigMap is actually pruned. Reasoned from prune: true on the Kustomization, not observed.