Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity

fix(vpn-ingress): content-hash the peer register, or a merged peer never reaches the tunnel
gate / gate (pull_request) Successful in 1m52s
Details
gate / conformance (pull_request) Has been skipped
Details

Browse Source 
The operator enrolled a phone, merged the peer, and `wg show` listed none.
Measured on prod 2026-10-06 with the peer already on main:

  live wg-peers ConfigMap, active AllowedIPs .. 0
  DaemonSet volume reference ................. configMap.name: wg-peers (FIXED)
  DaemonSet generation/observedGeneration .... 1 / 1   (never rolled)
  init container, all three pods ............. "0 peer(s)"

Part of it was just the 10m reconcile interval. The defect is the fixed name:
Flux updates a fixed-name ConfigMap IN PLACE, so the pod template is
byte-identical, so no rollout is triggered -- and the init container assembles
wg0.conf ONCE at pod start into a tmpfs, so the running tunnel keeps the old
peer list indefinitely. Adding or revoking a peer had no effect until something
unrelated restarted the pods. For a REVOCATION that is a security defect, not
an inconvenience: delete a stanza, see a green PR, device keeps working.

The runbook and README both claimed "PR -> merge -> Flux rolls the DaemonSet".
Never true. And this repo already documents the fix, in a file I had read --
runbooks/kustomization.yaml: "The generated name carries kustomize's content
hash, so editing any runbook rolls the server automatically and a stale
projection is impossible."

So: peers.yaml (inline ConfigMap) -> peers.conf (plain file) + a
configMapGenerator. The operator's enrolled peer is carried across verbatim.
Every line is WireGuard config and `#` is a legal comment there, so the
documentation now lives inside the file that is actually mounted.

`namespace: vpn-ingress` on the kustomization is REQUIRED, not tidiness: a
generated ConfigMap inherits no namespace, everything else here declares
vpn-ingress, and kustomize's nameReference will not rewrite across a namespace
boundary -- without it the build mounts the unhashed name and the pods wedge on
a missing volume. Caught only by building the output and reading the volumes.

new-peer.sh: PEERS_FILE repointed, and a missing register is now FATAL rather
than skipped -- the old `[ -r "$PEERS_FILE" ] && grep ...` guard meant that
after the rename the only symptom would have been duplicate detection quietly
never firing again. Stale peers.yaml references swept from six files including
the script's own user-facing output.

Verified: the DaemonSet now mounts wg-peers-m9kcckcb5t; the hash tracks content
(edit -> dgg7ht9c74 -> restore -> m9kcckcb5t); the enrolled peer is byte-identical
to main's; and `new-peer.sh dupe 2` is now correctly refused by the operator's
real allocation.

No sensitive paths; merges on a green gate.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-10-06 02:55:59 -05:00
1 parent e1bebb857e
commit 2acfed460a
9 files changed
+226 -65
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

platform/vpn-ingress/README.md
+6 -5
View File
@@ -13,7 +13,7 @@ egress out the house WAN.
| Transport | WireGuard, **udp/51820**, kernel implementation |
| Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node |
| Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices |
| Authorisation | a device's **public key** is in [`peers.yaml`](./peers.yaml), or it cannot connect |
| Authorisation | a device's **public key** is in [`peers.conf`](./peers.conf), or it cannot connect |
| The one secret | the server's private key — **SOPS-encrypted in this tree**, `wg-server-key.sops.yaml` |
| Exposure | one hand-made WAN port-forward on the SOHO router, off-repo |
| Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) |
@@ -129,7 +129,7 @@ translations win here.
| half | where | why |
|---|---|---|
| peer **public** keys + addresses | [`peers.yaml`](./peers.yaml), **plaintext in git** | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
| peer **public** keys + addresses | [`peers.conf`](./peers.conf), **plaintext in git**, as a content-hashed `configMapGenerator` so an edit rolls the pods | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
| the server's **private** key | `wg-server-key.sops.yaml`, **SOPS-encrypted in git** | the only real secret (R-SEC-01, spec docs/07). Ships with a `REPLACE-ME` placeholder: an agent must not mint a long-lived private key (CLAUDE.md §6), so the operator `sops set`s it in |
The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** —
@@ -166,8 +166,9 @@ NetworkPolicy applies.
## Operating it
* **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
stanza into `peers.yaml` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
* **Revoke a device:** delete its stanza → PR → merge. No other step; the key stops working.
stanza into `peers.conf` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
* **Revoke a device:** delete its stanza from `peers.conf` → PR → merge. No other step; the key
stops working as soon as the pods roll (the generator's content hash guarantees they do).
* **Is it up?** `kubectl --context k3s-prod -n vpn-ingress get ds wireguard` — three Ready.
* **Who is connected?** `kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show`
(`latest handshake` under ~3 min means live). Never dump `wg showconf` — it prints the private key.
@@ -211,7 +212,7 @@ right way; one was a defect in my own manifest.
covered generically by `KubeDeploymentReplicasMismatch`/`KubePodNotReady` from
kube-prometheus-stack. **This is the top follow-up.**
* **No preshared keys.** They would add a symmetric post-quantum hedge on top of the Noise
handshake — and would be a real secret, forcing `peers.yaml` into SOPS and costing the
handshake — and would be a real secret, forcing `peers.conf` into SOPS and costing the
reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
`wg-server-key` Secret.
* **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6
platform/vpn-ingress/daemonset.yaml
+1 -1
View File
@@ -108,7 +108,7 @@ spec:
dnsPolicy: ClusterFirstWithHostNet
automountServiceAccountToken: false
# ── Assemble the running config: PUBLIC peer register (git) + PRIVATE server key (Secret) ──
# The two halves are deliberately stored apart (peers.yaml explains why) and are only ever
# The two halves are deliberately stored apart (peers.conf explains why) and are only ever
# joined here, at pod start, into a tmpfs that is destroyed with the pod. The server's private
# key is therefore never written to any node's disk, never committed, and never logged.
initContainers:
platform/vpn-ingress/kustomization.yaml
+31 -2
View File
@@ -11,10 +11,16 @@
# NOT in platform/kustomization.yaml — see clusters/base/vpn-ingress.yaml.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# REQUIRED for the configMapGenerator below to be wired up at all, not merely tidy. A generated
# ConfigMap inherits NO namespace, every other object here declares `vpn-ingress`, and kustomize's
# nameReference transformer will not rewrite a reference across a namespace boundary — so without
# this the DaemonSet keeps mounting the unhashed name `wg-peers`, which does not exist, and the
# pods wedge on a missing volume. Verified by building it both ways (kustomize v5.8.1).
# Same placement as runbooks/kustomization.yaml, which uses `namespace: runbooks` for its own
# content generator.
namespace: vpn-ingress
resources:
- namespace.yaml
# The peer register: which devices may dial in. PLAINTEXT and reviewable on purpose (see the file).
- peers.yaml
# The listener itself: hostNetwork DaemonSet, udp/51820 on every node.
- daemonset.yaml
# Namespace default-deny baseline. NOTE: does NOT constrain the hostNetwork DaemonSet — the file
@@ -38,3 +44,26 @@ resources:
# hands Flux the raw ciphertext, the Secret APPLIES CLEANLY full of ENC[...], and nothing in any
# Flux condition looks wrong. The init container's `wg pubkey` guard exists partly to turn exactly
# that into a named error rather than a silently non-working tunnel.
# ── THE PEER REGISTER, as a CONTENT-HASHED ConfigMap ─────────────────────────────────────────
# peers.conf (plaintext, reviewable — see the file) becomes the `wg-peers` ConfigMap. It is a
# GENERATOR, not a plain resource, and that distinction is load-bearing rather than stylistic:
# kustomize appends a hash of the content to the generated name and rewrites the DaemonSet's
# volume reference to match, so adding or revoking a peer changes the pod template and ROLLS THE
# PODS automatically.
#
# A fixed-name ConfigMap (which is what this component shipped first) does not do that. Flux
# updates it in place, the pod template is byte-identical, no rollout is triggered — and because
# the init container assembles wg0.conf ONCE at pod start into a tmpfs, the running tunnel keeps
# the old peer list indefinitely. Measured on prod 2026-10-06: a merged peer reached the
# ConfigMap while `wg show` listed no peers and the DaemonSet still read generation=1, having
# never rolled. The runbook's "PR -> merge -> Flux rolls the DaemonSet" was false until this.
#
# DO NOT set `disableNameSuffixHash` here. The hash IS the mechanism.
configMapGenerator:
- name: wg-peers
files:
- peers.conf
options:
labels:
app.kubernetes.io/name: vpn-ingress
platform/vpn-ingress/networkpolicy.yaml
+1 -1
View File
@@ -19,7 +19,7 @@
# protects this port, and it would be enforcing nothing. So there isn't one.
#
# WHAT ACTUALLY PROTECTS THE EXPOSED PORT, then, since the CNI does not:
# 1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.yaml gets
# 1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.conf gets
# NO REPLY — see README "Why this is safe to expose". This is the real control.
# 2. The SOHO router's forward: only udp/51820 is forwarded inbound, to one node.
# 3. The blast radius once authenticated is deliberately the whole LAN — that IS the requirement
platform/vpn-ingress/peers.yaml → platform/vpn-ingress/peers.conf
+50 -41
View File
@@ -1,4 +1,16 @@
---
# THIS FILE IS THE MOUNTED CONTENT. It is turned into the `wg-peers` ConfigMap by
# `configMapGenerator` in kustomization.yaml, whose generated name carries a CONTENT HASH — so
# editing this file changes the ConfigMap name, which changes the DaemonSet's pod template, which
# ROLLS THE PODS. That is the whole reason it is a generator and not an inline ConfigMap manifest:
# a fixed-name ConfigMap updates in place, the pod template never changes, no rollout happens, and
# the init container's wg0.conf (built once at pod start, in tmpfs) keeps the OLD peer list
# forever. Measured 2026-10-06: a merged peer sat in the ConfigMap while `wg show` listed none and
# the DaemonSet reported generation=1 — it had never rolled. Same mechanism and same fix as
# runbooks/kustomization.yaml.
#
# Every line here is WireGuard config. `#` is a legal comment in it, so the documentation below
# travels inside the file that is actually mounted, rather than in a wrapper that gets stripped.
#
# ============================================================================================
# THE PEER REGISTER — every device allowed to dial in, in PLAINTEXT, in git, on purpose.
# ============================================================================================
@@ -41,43 +53,40 @@
# the Noise handshake and would be a genuine secret — which would force this register into SOPS
# and cost the reviewability above. Revisit only if the threat model changes; if it does, the PSK
# belongs in the `wg-server-key` Secret, not here. See README "Hardening not taken".
apiVersion: v1
kind: ConfigMap
metadata:
name: wg-peers
namespace: vpn-ingress
labels:
app.kubernetes.io/name: vpn-ingress
data:
# Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
#
# *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
# An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
# authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
# DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
# The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
# (`wg show` reports an interface with no peers).
#
# COPY THIS TEMPLATE, uncomment, one block per device:
#
# # <device-name> — <owner>, enrolled <YYYY-MM-DD>
# [Peer]
# PublicKey = <the public key new-peer.sh printed>
# AllowedIPs = 10.90.0.<octet>/32
#
# The placeholders above are deliberately NOT valid values. An earlier version of this template
# used a real-looking `10.90.0.2/32`, and new-peer.sh's duplicate-address check — which grepped
# the whole file — matched it, so `.2` could never be allocated and the script insisted it was
# already taken. The check now ignores comments; keeping the example unmatchable too means a
# future grep cannot be fooled either.
#
# Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
# tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
# it an incident responder cannot tell which stanza to delete.
peers.conf: |
# No peers enrolled yet. See the template in this file's comments and
# ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
# keith-phone — <owner>, enrolled 2026-10-06
[Peer]
PublicKey = pBRSHrRyG2hnsuZ6FvHfEzF8hOVTF5stHUb/7fW2Km8=
AllowedIPs = 10.90.0.2/32
#
# name: wg-peers
# namespace: vpn-ingress
# labels:
# app.kubernetes.io/name: vpn-ingress
# Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
#
# *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
# An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
# authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
# DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
# The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
# (`wg show` reports an interface with no peers).
#
# COPY THIS TEMPLATE, uncomment, one block per device:
#
# # <device-name> — <owner>, enrolled <YYYY-MM-DD>
# [Peer]
# PublicKey = <the public key new-peer.sh printed>
# AllowedIPs = 10.90.0.<octet>/32
#
# The placeholders above are deliberately NOT valid values. An earlier version of this template
# used a real-looking `10.90.0.2/32`, and new-peer.sh's duplicate-address check — which grepped
# the whole file — matched it, so `.2` could never be allocated and the script insisted it was
# already taken. The check now ignores comments; keeping the example unmatchable too means a
# future grep cannot be fooled either.
#
# Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
# tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
# it an incident responder cannot tell which stanza to delete.
#
# No peers enrolled yet. See the template in this file's comments and
# ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
# keith-phone — <owner>, enrolled 2026-10-06
[Peer]
PublicKey = pBRSHrRyG2hnsuZ6FvHfEzF8hOVTF5stHUb/7fW2Km8=
AllowedIPs = 10.90.0.2/32
platform/vpn-ingress/wg-server-key.sops.yaml.example
+1 -1
View File
@@ -5,7 +5,7 @@
# WHAT THIS IS: the WireGuard SERVER's long-term private key for the remote-access VPN
# (platform/vpn-ingress). It is the ONE secret in the whole component. The init container in
# daemonset.yaml reads it at `/run/wg-server-key/privatekey` and joins it to the PLAINTEXT peer
# register (peers.yaml) in a tmpfs, so the assembled wg0.conf never touches a node's disk.
# register (peers.conf) in a tmpfs, so the assembled wg0.conf never touches a node's disk.
#
# WHY IT IS IN GIT, SOPS-ENCRYPTED, AND NOT HAND-CREATED. R-SEC-01 / spec docs/07: secrets live in
# source, encrypted, and reach the cluster only through Flux. An out-of-band `kubectl create secret`