jakman85andClaude Opus 5
2acfed460a
The operator enrolled a phone, merged the peer, and `wg show` listed none. Measured on prod 2026-10-06 with the peer already on main: live wg-peers ConfigMap, active AllowedIPs .. 0 DaemonSet volume reference ................. configMap.name: wg-peers (FIXED) DaemonSet generation/observedGeneration .... 1 / 1 (never rolled) init container, all three pods ............. "0 peer(s)" Part of it was just the 10m reconcile interval. The defect is the fixed name: Flux updates a fixed-name ConfigMap IN PLACE, so the pod template is byte-identical, so no rollout is triggered -- and the init container assembles wg0.conf ONCE at pod start into a tmpfs, so the running tunnel keeps the old peer list indefinitely. Adding or revoking a peer had no effect until something unrelated restarted the pods. For a REVOCATION that is a security defect, not an inconvenience: delete a stanza, see a green PR, device keeps working. The runbook and README both claimed "PR -> merge -> Flux rolls the DaemonSet". Never true. And this repo already documents the fix, in a file I had read -- runbooks/kustomization.yaml: "The generated name carries kustomize's content hash, so editing any runbook rolls the server automatically and a stale projection is impossible." So: peers.yaml (inline ConfigMap) -> peers.conf (plain file) + a configMapGenerator. The operator's enrolled peer is carried across verbatim. Every line is WireGuard config and `#` is a legal comment there, so the documentation now lives inside the file that is actually mounted. `namespace: vpn-ingress` on the kustomization is REQUIRED, not tidiness: a generated ConfigMap inherits no namespace, everything else here declares vpn-ingress, and kustomize's nameReference will not rewrite across a namespace boundary -- without it the build mounts the unhashed name and the pods wedge on a missing volume. Caught only by building the output and reading the volumes. new-peer.sh: PEERS_FILE repointed, and a missing register is now FATAL rather than skipped -- the old `[ -r "$PEERS_FILE" ] && grep ...` guard meant that after the rename the only symptom would have been duplicate detection quietly never firing again. Stale peers.yaml references swept from six files including the script's own user-facing output. Verified: the DaemonSet now mounts wg-peers-m9kcckcb5t; the hash tracks content (edit -> dgg7ht9c74 -> restore -> m9kcckcb5t); the enrolled peer is byte-identical to main's; and `new-peer.sh dupe 2` is now correctly refused by the operator's real allocation. No sensitive paths; merges on a green gate. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
70 lines
4.6 KiB
YAML
70 lines
4.6 KiB
YAML
---
|
|
# WireGuard remote-access VPN (north-south INGRESS). Reconciled by its OWN non-gating post-platform
|
|
# Flux Kustomization (clusters/base/vpn-ingress.yaml, dependsOn platform-config, wait:false) — the
|
|
# same placement as crafty/headlamp/homepage/runbooks, for the same reason: a leaf whose pod can be
|
|
# legitimately unhealthy must never sit in the gating `platform` tier, where one component's
|
|
# unhealth stalls platform-config and every apps wave behind it (DECISIONS-LOG: "Headlamp gating
|
|
# stall"). That rationale is unusually concrete here: until the operator replaces the `REPLACE-ME`
|
|
# placeholder in wg-server-key.sops.yaml (runbooks/vpn-remote-access.md step 2) this DaemonSet
|
|
# CANNOT start, so putting it in the gating tier would have taken the whole platform down on merge.
|
|
#
|
|
# NOT in platform/kustomization.yaml — see clusters/base/vpn-ingress.yaml.
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
# REQUIRED for the configMapGenerator below to be wired up at all, not merely tidy. A generated
|
|
# ConfigMap inherits NO namespace, every other object here declares `vpn-ingress`, and kustomize's
|
|
# nameReference transformer will not rewrite a reference across a namespace boundary — so without
|
|
# this the DaemonSet keeps mounting the unhashed name `wg-peers`, which does not exist, and the
|
|
# pods wedge on a missing volume. Verified by building it both ways (kustomize v5.8.1).
|
|
# Same placement as runbooks/kustomization.yaml, which uses `namespace: runbooks` for its own
|
|
# content generator.
|
|
namespace: vpn-ingress
|
|
resources:
|
|
- namespace.yaml
|
|
# The listener itself: hostNetwork DaemonSet, udp/51820 on every node.
|
|
- daemonset.yaml
|
|
# Namespace default-deny baseline. NOTE: does NOT constrain the hostNetwork DaemonSet — the file
|
|
# says so at length rather than implying a protection it cannot provide.
|
|
- networkpolicy.yaml # The server's WireGuard private key — the ONE secret here. SOPS-encrypted (R-SEC-01, spec
|
|
# docs/07), decrypted per-env by the `vpn-ingress` Kustomization, referenced by name only from
|
|
# daemonset.yaml (R-IDP-04). Its plaintext `.example` sibling carries the operator hand-off --
|
|
# deliberately NOT named here: conformance/tests/l0-static/no-plaintext-secrets fails any
|
|
# kustomization that mentions an unencrypted template, because naming one is how a plaintext
|
|
# Secret gets applied by accident. (It matches comments too. That is correct, not overzealous.)
|
|
#
|
|
# SHIPS WITH A `REPLACE-ME` PLACEHOLDER, already encrypted. An agent must not
|
|
# mint a long-lived private key (CLAUDE.md §6), and the init container validates the value with
|
|
# `wg pubkey`, so the unfilled state is a loud CrashLoop naming this file, never a server that
|
|
# quietly cannot handshake.
|
|
- wg-server-key.sops.yaml
|
|
#
|
|
# THIS COMPONENT CARRIES AN ENCRYPTED SECRET, so BOTH overlays must patch the `vpn-ingress`
|
|
# Kustomization with a `decryption:` stanza — clusters/dev and clusters/prod both do, added by the
|
|
# same change as this file. That patch is as load-bearing as the file itself: without it kustomize
|
|
# hands Flux the raw ciphertext, the Secret APPLIES CLEANLY full of ENC[...], and nothing in any
|
|
# Flux condition looks wrong. The init container's `wg pubkey` guard exists partly to turn exactly
|
|
# that into a named error rather than a silently non-working tunnel.
|
|
|
|
# ── THE PEER REGISTER, as a CONTENT-HASHED ConfigMap ─────────────────────────────────────────
|
|
# peers.conf (plaintext, reviewable — see the file) becomes the `wg-peers` ConfigMap. It is a
|
|
# GENERATOR, not a plain resource, and that distinction is load-bearing rather than stylistic:
|
|
# kustomize appends a hash of the content to the generated name and rewrites the DaemonSet's
|
|
# volume reference to match, so adding or revoking a peer changes the pod template and ROLLS THE
|
|
# PODS automatically.
|
|
#
|
|
# A fixed-name ConfigMap (which is what this component shipped first) does not do that. Flux
|
|
# updates it in place, the pod template is byte-identical, no rollout is triggered — and because
|
|
# the init container assembles wg0.conf ONCE at pod start into a tmpfs, the running tunnel keeps
|
|
# the old peer list indefinitely. Measured on prod 2026-10-06: a merged peer reached the
|
|
# ConfigMap while `wg show` listed no peers and the DaemonSet still read generation=1, having
|
|
# never rolled. The runbook's "PR -> merge -> Flux rolls the DaemonSet" was false until this.
|
|
#
|
|
# DO NOT set `disableNameSuffixHash` here. The hash IS the mechanism.
|
|
configMapGenerator:
|
|
- name: wg-peers
|
|
files:
|
|
- peers.conf
|
|
options:
|
|
labels:
|
|
app.kubernetes.io/name: vpn-ingress
|