Compare commits

..
Author SHA1 Message Date
jakman85andClaude Opus 5 2acfed460a fix(vpn-ingress): content-hash the peer register, or a merged peer never reaches the tunnel
gate / gate (pull_request) Successful in 1m52s
gate / conformance (pull_request) Has been skipped
The operator enrolled a phone, merged the peer, and `wg show` listed none.
Measured on prod 2026-10-06 with the peer already on main:

  live wg-peers ConfigMap, active AllowedIPs .. 0
  DaemonSet volume reference ................. configMap.name: wg-peers (FIXED)
  DaemonSet generation/observedGeneration .... 1 / 1   (never rolled)
  init container, all three pods ............. "0 peer(s)"

Part of it was just the 10m reconcile interval. The defect is the fixed name:
Flux updates a fixed-name ConfigMap IN PLACE, so the pod template is
byte-identical, so no rollout is triggered -- and the init container assembles
wg0.conf ONCE at pod start into a tmpfs, so the running tunnel keeps the old
peer list indefinitely. Adding or revoking a peer had no effect until something
unrelated restarted the pods. For a REVOCATION that is a security defect, not
an inconvenience: delete a stanza, see a green PR, device keeps working.

The runbook and README both claimed "PR -> merge -> Flux rolls the DaemonSet".
Never true. And this repo already documents the fix, in a file I had read --
runbooks/kustomization.yaml: "The generated name carries kustomize's content
hash, so editing any runbook rolls the server automatically and a stale
projection is impossible."

So: peers.yaml (inline ConfigMap) -> peers.conf (plain file) + a
configMapGenerator. The operator's enrolled peer is carried across verbatim.
Every line is WireGuard config and `#` is a legal comment there, so the
documentation now lives inside the file that is actually mounted.

`namespace: vpn-ingress` on the kustomization is REQUIRED, not tidiness: a
generated ConfigMap inherits no namespace, everything else here declares
vpn-ingress, and kustomize's nameReference will not rewrite across a namespace
boundary -- without it the build mounts the unhashed name and the pods wedge on
a missing volume. Caught only by building the output and reading the volumes.

new-peer.sh: PEERS_FILE repointed, and a missing register is now FATAL rather
than skipped -- the old `[ -r "$PEERS_FILE" ] && grep ...` guard meant that
after the rename the only symptom would have been duplicate detection quietly
never firing again. Stale peers.yaml references swept from six files including
the script's own user-facing output.

Verified: the DaemonSet now mounts wg-peers-m9kcckcb5t; the hash tracks content
(edit -> dgg7ht9c74 -> restore -> m9kcckcb5t); the enrolled peer is byte-identical
to main's; and `new-peer.sh dupe 2` is now correctly refused by the operator's
real allocation.

No sensitive paths; merges on a green gate.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-06 02:55:59 -05:00
jakman85 e1bebb857e Merge pull request 'Added my phone to peers' (#65) from vpn/add-keith-phone into main
gate / gate (push) Successful in 1m53s
gate / conformance (push) Has been skipped
Reviewed-on: #65
2026-10-06 07:42:04 +00:00
jakman85 23db97b958 Added my phone to peers
gate / gate (pull_request) Successful in 1m53s
gate / conformance (pull_request) Has been skipped
2026-10-06 02:39:23 -05:00
jakman85 5928d8153c Merge pull request 'fix(vpn-ingress): new-peer.sh counted its own commented-out example as an allocated address' (#64) from fix/new-peer-ignores-template-comments into main
gate / gate (push) Successful in 1m50s
gate / conformance (push) Has been skipped
Reviewed-on: #64
2026-10-06 07:31:03 +00:00
9 changed files with 226 additions and 61 deletions

No files matched your search

@@ -0,0 +1,103 @@
---
date: 2026-10-06
branch: fix/peers-configmap-content-hash
title: A merged VPN peer never reached the tunnel — the peer register was a fixed-name ConfigMap, so nothing rolled the pods
---
## What changed
The operator enrolled a phone, merged the peer, and **`wg show` listed no peers.** Measured on prod
2026-10-06, with the peer already on `main`:
| observation | value |
|---|---|
| peer stanza on `main` | present (`AllowedIPs = 10.90.0.2/32`) |
| GitRepository artifact | `main@sha1:e1bebb8` — Flux had fetched it |
| `vpn-ingress` Kustomization `lastAppliedRevision` | `main@sha1:5928d81` — **behind**, 10m interval |
| live `wg-peers` ConfigMap, active `AllowedIPs` lines | **0** |
| DaemonSet volume reference | `configMap.name: wg-peers` — a **fixed** name |
| DaemonSet `generation` / `observedGeneration` | **1 / 1** — it had **never rolled** |
| init container, all three pods | `wg0.conf assembled: 0 peer(s)` |
The lag was only the reconcile interval. **The defect is the fixed name.** When Flux does update a
fixed-name ConfigMap it updates it *in place*: the DaemonSet's pod template is byte-identical, so
no rollout is triggered — and because the init container assembles `wg0.conf` **once at pod start**
into a tmpfs, the running tunnel keeps the old peer list indefinitely. Adding or revoking a peer
would have had **no effect at all** until something unrelated happened to restart the pods. For a
*revocation* that is a security defect, not an inconvenience: the operator would delete a stanza,
see a green PR, and the device would keep working.
My runbook and README both asserted "PR → merge → Flux rolls the DaemonSet (~1 min)". That was
never true.
**This repo already documents the fix, and I had read the file.** `runbooks/kustomization.yaml`:
*"The generated name carries kustomize's content hash, so editing any runbook rolls the server
automatically and a stale projection is impossible."*
So the register becomes a **content-hashed generator**:
- `platform/vpn-ingress/peers.yaml` (inline ConfigMap) → **`platform/vpn-ingress/peers.conf`**, a
plain file, plus a `configMapGenerator` in `kustomization.yaml`. The operator's enrolled peer was
carried across verbatim. Every line is WireGuard config and `#` is a legal comment there, so the
documentation now travels *inside the mounted file* instead of in a YAML wrapper that gets
stripped.
- **`namespace: vpn-ingress` added to the kustomization — REQUIRED, not tidiness.** A generated
ConfigMap inherits no namespace; every other object here declares `vpn-ingress`; and kustomize's
`nameReference` transformer will not rewrite a reference across a namespace boundary. Without it
the build emitted `ConfigMap wg-peers-m9kcckcb5t` while the DaemonSet still mounted the unhashed
`wg-peers`, which would not exist — the pods would wedge on a missing volume. I caught this only
by building the output and reading the volumes block.
- **`infra/vpn-ingress/new-peer.sh`**: `PEERS_FILE` repointed to `peers.conf`, and a missing
register is now **fatal** instead of skipped. The old guard was
`[ -r "$PEERS_FILE" ] && grep …`, so after the rename the only symptom would have been
duplicate-address detection quietly never firing again — the silent-skip shape this repo treats
as a defect in its own right.
- Stale `peers.yaml` references swept from the README, runbook, `daemonset.yaml`,
`networkpolicy.yaml`, the SOPS example, and the script's own **user-facing output** (it told the
operator to paste into "peers.yaml, under data.peers.conf", which would no longer exist).
- The runbook gained the verification that actually distinguishes this failure — the DaemonSet's
`generation` must *increase* after the merge — and a troubleshooting row for `0 peer(s)` after a
merge.
Flux's `prune: true` removes the orphaned fixed-name `wg-peers` ConfigMap on the next reconcile.
No sensitive paths (`platform/vpn-ingress/**`, `infra/vpn-ingress/**`, `runbooks/**`,
`decisions/**`), so this merges on a green gate with no override.
## What I verified
Built and run in the pinned toolbox, 2026-10-06 (kustomize v5.8.1 via `kubectl kustomize`).
- **The reference is now rewritten end to end**: generated `ConfigMap wg-peers-m9kcckcb5t` in
namespace `vpn-ingress`, and the DaemonSet mounts **`wg-peers-m9kcckcb5t`**. Before adding
`namespace:` the same build mounted the unhashed `wg-peers` — so the namespace line is load
bearing, measured both ways.
- **The hash tracks content, which is the whole mechanism.** `wg-peers-m9kcckcb5t` →
append a second peer → `wg-peers-dgg7ht9c74` → restore → `wg-peers-m9kcckcb5t`. Deterministic
and reversible, so an edit necessarily changes the pod template.
- **The operator's enrolled peer survived the file conversion byte-for-byte**: the active stanza in
`peers.conf` (`PublicKey = pBRSHrRy…`, `AllowedIPs = 10.90.0.2/32`) is identical to the one on
`main`, compared directly against `git show origin/main:platform/vpn-ingress/peers.yaml`.
- **The build still emits all five objects**: Namespace, ConfigMap, Secret, DaemonSet,
NetworkPolicy — and the `Namespace` object's own name is still `vpn-ingress` (the namespace
transformer did not rewrite it).
- **`new-peer.sh` works against the new register**: `spare-laptop 7` renders a profile and prints
"append this to platform/vpn-ingress/peers.conf"; `dupe 2` is **refused** because the operator's
real `.2` now occupies it — the first time the duplicate check has been exercised against real
data rather than against its own example.
- **Toolbox gate**: rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`,
`l1` and `approval-policy` all pass; rc 3 is the two environmental skips. **Not green. Reported
as rc 3.**
## What I did NOT verify
- **That the roll actually happens on the cluster.** The whole point of the change is unproven
until Flux applies it: the success criterion is the DaemonSet's `generation` increasing and
`wg0.conf assembled: 1 peer(s)` in the init log, followed by `wg show` listing the peer.
- **That the tunnel then carries traffic.** No handshake has ever been observed. The WAN
port-forward is reportedly configured but has never been exercised by a packet I can see, and I
have no off-LAN vantage point — runbook step 6 from cellular remains the only honest test.
- **Revocation**, which is the case where this defect was a security problem. Deleting a stanza
should now roll the pods and drop the peer; untested.
- **Whether the orphaned fixed-name `wg-peers` ConfigMap is actually pruned.** Reasoned from
`prune: true` on the Kustomization, not observed.
+15 -10
View File
@@ -3,7 +3,7 @@
# new-peer.sh — enrol one device on the remote-access VPN (platform/vpn-ingress).
#
# Generates a WireGuard keypair, renders the device's client profile, prints a QR code for a phone,
# and prints the one `[Peer]` stanza to paste into platform/vpn-ingress/peers.yaml. The device's
# and prints the one `[Peer]` stanza to paste into platform/vpn-ingress/peers.conf. The device's
# private half is written to ONE 0600 file and nowhere else: not sent to the cluster, not committed,
# never logged.
#
@@ -62,7 +62,7 @@ set -euo pipefail
die() { printf 'new-peer.sh: %s\n' "$*" >&2; exit 1; }
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.yaml"
PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.conf"
# --- defaults, each mirroring a value that is DECLARED elsewhere in this repo -------------------
# These are duplicated here rather than parsed out of the YAML on purpose: this script is meant to
@@ -134,16 +134,21 @@ CLIENT_ADDR="${CLIENT_BASE}.${OCTET}"
# Two devices sharing an address does not error anywhere: the server routes that address to
# whichever peer handshook most recently, so BOTH devices appear to work intermittently.
# COMMENT LINES ARE NOT ALLOCATIONS. This used to grep the whole file and so matched the
# COMMENTED-OUT EXAMPLE STANZA in peers.yaml, which made `.2` — the first address anyone would
# COMMENTED-OUT EXAMPLE STANZA in the register, which made `.2` — the first address anyone would
# ask for — permanently un-allocatable with a message claiming it was already taken. Strip
# comments first: a real stanza's `AllowedIPs` line is never commented, while both the template
# and every peer's `# <name> — <owner>` label are.
if [ -r "$PEERS_FILE" ] \
&& grep -vE '^[[:space:]]*#' "$PEERS_FILE" | grep -q "AllowedIPs *= *${CLIENT_ADDR}/32"; then
die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet."
# FAIL if the register is missing rather than skipping the check. The path was silently wrong once
# (it still said peers.yaml after the file became peers.conf), and because the old guard was
# `[ -r "$PEERS_FILE" ] && ...` the only symptom would have been duplicate detection quietly
# never firing again — the exact silent-skip shape this repo treats as a defect.
[ -r "$PEERS_FILE" ] || die "cannot read the peer register '$PEERS_FILE'.
Duplicate-address detection cannot run, so this script refuses to issue a profile."
if grep -vE '^[[:space:]]*#' "$PEERS_FILE" | grep -q "AllowedIPs *= *${CLIENT_ADDR}/32"; then
die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.conf. Pick another octet."
fi
if [ -r "$PEERS_FILE" ] && grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.yaml.\n' "$NAME" >&2
if grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.conf.\n' "$NAME" >&2
fi
# --- output location: 0700, and never inside the repo -------------------------------------------
@@ -210,7 +215,7 @@ cat <<EOF
Profile : ${PROFILE} (0600; contains the private key)
------------------------------------------------------------------------------------------
STEP 1 — add this to platform/vpn-ingress/peers.yaml, under data.peers.conf, then PR it:
STEP 1 — append this to platform/vpn-ingress/peers.conf, then PR it:
# ${NAME} — <owner>, enrolled $(date -u +%Y-%m-%d)
[Peer]
@@ -229,7 +234,7 @@ cat <<EOF
STEP 3 — delete the profile from this machine once the device has it:
shred -u ${PROFILE} 2>/dev/null || rm -f ${PROFILE}
The tunnel will not work until the peers.yaml PR is MERGED and Flux has rolled the DaemonSet
The tunnel will not work until the peers.conf PR is MERGED and Flux has rolled the DaemonSet
(~1 min). Until then the device handshakes and gets silence, exactly like an unknown key.
------------------------------------------------------------------------------------------
EOF
+6 -5
View File
@@ -13,7 +13,7 @@ egress out the house WAN.
| Transport | WireGuard, **udp/51820**, kernel implementation |
| Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node |
| Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices |
| Authorisation | a device's **public key** is in [`peers.yaml`](./peers.yaml), or it cannot connect |
| Authorisation | a device's **public key** is in [`peers.conf`](./peers.conf), or it cannot connect |
| The one secret | the server's private key — **SOPS-encrypted in this tree**, `wg-server-key.sops.yaml` |
| Exposure | one hand-made WAN port-forward on the SOHO router, off-repo |
| Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) |
@@ -129,7 +129,7 @@ translations win here.
| half | where | why |
|---|---|---|
| peer **public** keys + addresses | [`peers.yaml`](./peers.yaml), **plaintext in git** | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
| peer **public** keys + addresses | [`peers.conf`](./peers.conf), **plaintext in git**, as a content-hashed `configMapGenerator` so an edit rolls the pods | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
| the server's **private** key | `wg-server-key.sops.yaml`, **SOPS-encrypted in git** | the only real secret (R-SEC-01, spec docs/07). Ships with a `REPLACE-ME` placeholder: an agent must not mint a long-lived private key (CLAUDE.md §6), so the operator `sops set`s it in |
The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** —
@@ -166,8 +166,9 @@ NetworkPolicy applies.
## Operating it
* **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
stanza into `peers.yaml` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
* **Revoke a device:** delete its stanza → PR → merge. No other step; the key stops working.
stanza into `peers.conf` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
* **Revoke a device:** delete its stanza from `peers.conf` → PR → merge. No other step; the key
stops working as soon as the pods roll (the generator's content hash guarantees they do).
* **Is it up?** `kubectl --context k3s-prod -n vpn-ingress get ds wireguard` — three Ready.
* **Who is connected?** `kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show`
(`latest handshake` under ~3 min means live). Never dump `wg showconf` — it prints the private key.
@@ -211,7 +212,7 @@ right way; one was a defect in my own manifest.
covered generically by `KubeDeploymentReplicasMismatch`/`KubePodNotReady` from
kube-prometheus-stack. **This is the top follow-up.**
* **No preshared keys.** They would add a symmetric post-quantum hedge on top of the Noise
handshake — and would be a real secret, forcing `peers.yaml` into SOPS and costing the
handshake — and would be a real secret, forcing `peers.conf` into SOPS and costing the
reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
`wg-server-key` Secret.
* **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6
+1 -1
View File
@@ -108,7 +108,7 @@ spec:
dnsPolicy: ClusterFirstWithHostNet
automountServiceAccountToken: false
# ── Assemble the running config: PUBLIC peer register (git) + PRIVATE server key (Secret) ──
# The two halves are deliberately stored apart (peers.yaml explains why) and are only ever
# The two halves are deliberately stored apart (peers.conf explains why) and are only ever
# joined here, at pod start, into a tmpfs that is destroyed with the pod. The server's private
# key is therefore never written to any node's disk, never committed, and never logged.
initContainers:
+31 -2
View File
@@ -11,10 +11,16 @@
# NOT in platform/kustomization.yaml — see clusters/base/vpn-ingress.yaml.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# REQUIRED for the configMapGenerator below to be wired up at all, not merely tidy. A generated
# ConfigMap inherits NO namespace, every other object here declares `vpn-ingress`, and kustomize's
# nameReference transformer will not rewrite a reference across a namespace boundary — so without
# this the DaemonSet keeps mounting the unhashed name `wg-peers`, which does not exist, and the
# pods wedge on a missing volume. Verified by building it both ways (kustomize v5.8.1).
# Same placement as runbooks/kustomization.yaml, which uses `namespace: runbooks` for its own
# content generator.
namespace: vpn-ingress
resources:
- namespace.yaml
# The peer register: which devices may dial in. PLAINTEXT and reviewable on purpose (see the file).
- peers.yaml
# The listener itself: hostNetwork DaemonSet, udp/51820 on every node.
- daemonset.yaml
# Namespace default-deny baseline. NOTE: does NOT constrain the hostNetwork DaemonSet — the file
@@ -38,3 +44,26 @@ resources:
# hands Flux the raw ciphertext, the Secret APPLIES CLEANLY full of ENC[...], and nothing in any
# Flux condition looks wrong. The init container's `wg pubkey` guard exists partly to turn exactly
# that into a named error rather than a silently non-working tunnel.
# ── THE PEER REGISTER, as a CONTENT-HASHED ConfigMap ─────────────────────────────────────────
# peers.conf (plaintext, reviewable — see the file) becomes the `wg-peers` ConfigMap. It is a
# GENERATOR, not a plain resource, and that distinction is load-bearing rather than stylistic:
# kustomize appends a hash of the content to the generated name and rewrites the DaemonSet's
# volume reference to match, so adding or revoking a peer changes the pod template and ROLLS THE
# PODS automatically.
#
# A fixed-name ConfigMap (which is what this component shipped first) does not do that. Flux
# updates it in place, the pod template is byte-identical, no rollout is triggered — and because
# the init container assembles wg0.conf ONCE at pod start into a tmpfs, the running tunnel keeps
# the old peer list indefinitely. Measured on prod 2026-10-06: a merged peer reached the
# ConfigMap while `wg show` listed no peers and the DaemonSet still read generation=1, having
# never rolled. The runbook's "PR -> merge -> Flux rolls the DaemonSet" was false until this.
#
# DO NOT set `disableNameSuffixHash` here. The hash IS the mechanism.
configMapGenerator:
- name: wg-peers
files:
- peers.conf
options:
labels:
app.kubernetes.io/name: vpn-ingress
+1 -1
View File
@@ -19,7 +19,7 @@
# protects this port, and it would be enforcing nothing. So there isn't one.
#
# WHAT ACTUALLY PROTECTS THE EXPOSED PORT, then, since the CNI does not:
# 1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.yaml gets
# 1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.conf gets
# NO REPLY — see README "Why this is safe to expose". This is the real control.
# 2. The SOHO router's forward: only udp/51820 is forwarded inbound, to one node.
# 3. The blast radius once authenticated is deliberately the whole LAN — that IS the requirement
@@ -1,4 +1,16 @@
---
# THIS FILE IS THE MOUNTED CONTENT. It is turned into the `wg-peers` ConfigMap by
# `configMapGenerator` in kustomization.yaml, whose generated name carries a CONTENT HASH — so
# editing this file changes the ConfigMap name, which changes the DaemonSet's pod template, which
# ROLLS THE PODS. That is the whole reason it is a generator and not an inline ConfigMap manifest:
# a fixed-name ConfigMap updates in place, the pod template never changes, no rollout happens, and
# the init container's wg0.conf (built once at pod start, in tmpfs) keeps the OLD peer list
# forever. Measured 2026-10-06: a merged peer sat in the ConfigMap while `wg show` listed none and
# the DaemonSet reported generation=1 — it had never rolled. Same mechanism and same fix as
# runbooks/kustomization.yaml.
#
# Every line here is WireGuard config. `#` is a legal comment in it, so the documentation below
# travels inside the file that is actually mounted, rather than in a wrapper that gets stripped.
#
# ============================================================================================
# THE PEER REGISTER — every device allowed to dial in, in PLAINTEXT, in git, on purpose.
# ============================================================================================
@@ -41,39 +53,40 @@
# the Noise handshake and would be a genuine secret — which would force this register into SOPS
# and cost the reviewability above. Revisit only if the threat model changes; if it does, the PSK
# belongs in the `wg-server-key` Secret, not here. See README "Hardening not taken".
apiVersion: v1
kind: ConfigMap
metadata:
name: wg-peers
namespace: vpn-ingress
labels:
app.kubernetes.io/name: vpn-ingress
data:
# Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
#
# *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
# An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
# authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
# DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
# The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
# (`wg show` reports an interface with no peers).
#
# COPY THIS TEMPLATE, uncomment, one block per device:
#
# # <device-name> — <owner>, enrolled <YYYY-MM-DD>
# [Peer]
# PublicKey = <the public key new-peer.sh printed>
# AllowedIPs = 10.90.0.<octet>/32
#
# The placeholders above are deliberately NOT valid values. An earlier version of this template
# used a real-looking `10.90.0.2/32`, and new-peer.sh's duplicate-address check — which grepped
# the whole file — matched it, so `.2` could never be allocated and the script insisted it was
# already taken. The check now ignores comments; keeping the example unmatchable too means a
# future grep cannot be fooled either.
#
# Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
# tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
# it an incident responder cannot tell which stanza to delete.
peers.conf: |
# No peers enrolled yet. See the template in this file's comments and
# ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
#
# name: wg-peers
# namespace: vpn-ingress
# labels:
# app.kubernetes.io/name: vpn-ingress
# Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
#
# *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
# An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
# authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
# DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
# The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
# (`wg show` reports an interface with no peers).
#
# COPY THIS TEMPLATE, uncomment, one block per device:
#
# # <device-name> — <owner>, enrolled <YYYY-MM-DD>
# [Peer]
# PublicKey = <the public key new-peer.sh printed>
# AllowedIPs = 10.90.0.<octet>/32
#
# The placeholders above are deliberately NOT valid values. An earlier version of this template
# used a real-looking `10.90.0.2/32`, and new-peer.sh's duplicate-address check — which grepped
# the whole file — matched it, so `.2` could never be allocated and the script insisted it was
# already taken. The check now ignores comments; keeping the example unmatchable too means a
# future grep cannot be fooled either.
#
# Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
# tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
# it an incident responder cannot tell which stanza to delete.
#
# No peers enrolled yet. See the template in this file's comments and
# ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
# keith-phone — <owner>, enrolled 2026-10-06
[Peer]
PublicKey = pBRSHrRyG2hnsuZ6FvHfEzF8hOVTF5stHUb/7fW2Km8=
AllowedIPs = 10.90.0.2/32
@@ -5,7 +5,7 @@
# WHAT THIS IS: the WireGuard SERVER's long-term private key for the remote-access VPN
# (platform/vpn-ingress). It is the ONE secret in the whole component. The init container in
# daemonset.yaml reads it at `/run/wg-server-key/privatekey` and joins it to the PLAINTEXT peer
# register (peers.yaml) in a tmpfs, so the assembled wg0.conf never touches a node's disk.
# register (peers.conf) in a tmpfs, so the assembled wg0.conf never touches a node's disk.
#
# WHY IT IS IN GIT, SOPS-ENCRYPTED, AND NOT HAND-CREATED. R-SEC-01 / spec docs/07: secrets live in
# source, encrypted, and reach the cluster only through Flux. An out-of-band `kubectl create secret`
+18 -4
View File
@@ -21,7 +21,7 @@ behaves like a machine on the LAN: full reach to `192.168.2.0/24`, the Gateway's
DNS, and internet egress out the house WAN.
Authorisation is one thing only: the device's **public key is in
`platform/vpn-ingress/peers.yaml`**. An unlisted key gets no reply at all.
`platform/vpn-ingress/peers.conf`**. An unlisted key gets no reply at all.
> **An enrolled device is as privileged as a machine sitting in the house** — including the three
> k3s nodes (`192.168.2.10/.11/.12`: apiserver 6443, kubelet 10250, SSH 22). That is what "reach
@@ -135,6 +135,7 @@ specific failure in its log — so you never get a server that silently cannot c
| `still SOPS CIPHERTEXT (starts with ENC[)` | the Secret reached the cluster undecrypted — this environment's overlay is missing its `vpn-ingress` decryption patch. **This is the dangerous one:** the Secret applies cleanly and every Flux condition stays green |
| `wg pubkey rejected it` with none of the above | a mangled value, or a *public* key pasted by mistake |
| `net.ipv4.ip_forward is 0 on this node` | the node_baseline role has not run here (step 1). The pod **cannot** set this itself — under `hostNetwork` the runtime mounts `/proc/sys` read-only, because it is the host's global setting |
| `0 peer(s)` after you merged a peer | the DaemonSet did not roll. Check its `generation` (above). Pre-2026-10-06 this was guaranteed, because the register was a fixed-name ConfigMap; if it recurs, something has reverted `peers.conf` to a plain resource instead of a `configMapGenerator` |
### What a healthy start looks like
@@ -218,10 +219,23 @@ It generates the device's keypair, writes a `0600` client profile, prints a QR c
Then:
1. Paste the stanza into `platform/vpn-ingress/peers.yaml` under `data.peers.conf`, keeping the
1. Append the stanza to `platform/vpn-ingress/peers.conf`, keeping the
`# <name> — <owner>, enrolled <date>` comment. `wg show` can only ever print the key, so that
comment is the only record of which stanza belongs to which device.
2. PR → merge. Flux rolls the DaemonSet (~1 min).
2. PR → merge. Flux rolls the DaemonSet (~1 min), because `peers.conf` is a
**content-hashed** `configMapGenerator`: editing it changes the ConfigMap's generated name,
which changes the pod template, which forces the roll. (It did **not** work this way at first —
a fixed-name ConfigMap updates in place, the pod template never changes, and the init container
keeps the old peer list forever. Measured on prod 2026-10-06.) Confirm it actually rolled:
```bash
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n vpn-ingress get ds wireguard \
-o jsonpath="{.metadata.generation}{\"/\"}{.status.observedGeneration}{\"\n\"}"'
```
The generation must have **increased** since before the merge. If it is still `1`, the roll did
not happen and `wg show` will list no peer no matter how long you wait.
3. Get the profile onto the device (QR scan for phones; "Import tunnel from file" on
Windows/macOS; `/etc/wireguard/<name>.conf` + `wg-quick@<name>` on Linux).
4. **Delete the profile from your machine**: `shred -u <path>`. It is the only copy of that
@@ -261,7 +275,7 @@ if the router is not delivering the packets, and a `handshake` with no completed
## Revoking a device
Delete its `[Peer]` stanza from `peers.yaml`, PR, merge. Nothing else — the key stops working on the
Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the
next reconcile. **Never re-use that address octet**: an old profile would silently start working
again as a different device.