Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2acfed460a | ||
|
|
e1bebb857e | ||
|
|
23db97b958 | ||
|
|
5928d8153c | ||
|
|
146dd9a906 | ||
|
|
d1c07fe021 | ||
|
|
5c43fbac65 | ||
|
|
cf7d5e4316 |
No files matched your search
@@ -0,0 +1,73 @@
|
||||
---
|
||||
date: 2026-10-06
|
||||
branch: fix/new-peer-ignores-template-comments
|
||||
title: new-peer.sh refused the first VPN address because its duplicate check matched its own commented-out example
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
The operator's first real enrolment failed:
|
||||
|
||||
```
|
||||
$ ./infra/vpn-ingress/new-peer.sh keith-phone 2
|
||||
new-peer.sh: 10.90.0.2/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet.
|
||||
```
|
||||
|
||||
**Nothing was allocated.** `peers.yaml` had zero peers — the pod's own init container reported
|
||||
`wg0.conf assembled: 0 peer(s)`. The duplicate-address check grepped the WHOLE file, and
|
||||
`peers.yaml` line 66 is a **commented-out example stanza** I wrote, containing literally
|
||||
`# AllowedIPs = 10.90.0.2/32`. So `.2` — the first address anybody would ask for, and the one my
|
||||
own runbook uses in its example — was permanently un-allocatable, with a message asserting the
|
||||
opposite of the truth.
|
||||
|
||||
Both halves fixed, because either alone leaves the trap half-set:
|
||||
|
||||
- **`infra/vpn-ingress/new-peer.sh`**: the check strips comment lines before matching
|
||||
(`grep -vE '^[[:space:]]*#' | grep -q …`). That is the correct predicate rather than a patch: a
|
||||
real stanza's `AllowedIPs` line is never commented, while both the template and every peer's
|
||||
`# <name> — <owner>` label always are.
|
||||
- **`platform/vpn-ingress/peers.yaml`**: the example now uses placeholders that cannot match
|
||||
anything real — `# <device-name> — <owner>, enrolled <YYYY-MM-DD>` and
|
||||
`AllowedIPs = 10.90.0.<octet>/32`. A future grep cannot be fooled by it either.
|
||||
|
||||
The bug class is worth naming: **a documentation example that is byte-identical to real data will
|
||||
eventually be parsed as real data.** The same shape caused two earlier defects in this component —
|
||||
`flux-postbuild-substitution` flagged a `${...}` written inside a comment *explaining* that hazard,
|
||||
and `no-plaintext-secrets` flagged the `.sops.yaml.example` because the kustomization named it in a
|
||||
comment. Three instances now; the rule is to make examples structurally unmatchable, not merely
|
||||
commented out.
|
||||
|
||||
No sensitive paths (`platform/vpn-ingress/**`, `infra/vpn-ingress/**`, `decisions/**`), so this
|
||||
merges on a green gate with no override.
|
||||
|
||||
## What I verified
|
||||
|
||||
Run in the pinned toolbox against the live tree, 2026-10-06.
|
||||
|
||||
- **`.2` now works**, and it went through the REAL key path: `new-peer.sh keith-phone 2` rendered a
|
||||
profile with `Address 10.90.0.2/32`, `AllowedIPs = 10.90.0.2/32` in the printed stanza, full
|
||||
tunnel, at `.cache/wg-peers/keith-phone.conf` (0600). The throwaway keypair was deleted after.
|
||||
- **This incidentally closes a gap left open in #60**: the profile could only have been rendered by
|
||||
deriving the server's public key via `sops -d --extract … | wg pubkey` from
|
||||
`wg-server-key.sops.yaml` — so that pipeline is now proven **against the operator's real key**,
|
||||
not just against a placeholder it was expected to reject.
|
||||
- **A real duplicate is still refused.** Appending an uncommented
|
||||
`AllowedIPs = 10.90.0.5/32` stanza and asking for octet 5 produced the correct refusal; the tree
|
||||
was restored afterwards and `git diff --stat` showed only the intended template edit.
|
||||
- **Zero active allocations remain**: `grep -vE '^[[:space:]]*#' peers.yaml | grep -c AllowedIPs`
|
||||
→ `0`.
|
||||
- **Toolbox gate**: rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`,
|
||||
`l1` and `approval-policy` all pass; rc 3 is the two environmental skips (`gitea-settings`,
|
||||
conformance L2/L3). **Not green. Reported as rc 3.**
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **No peer is enrolled by this branch.** The keypair I generated was a throwaway and was deleted;
|
||||
the operator must run the script themselves, since the device's private key must exist only on
|
||||
the device. So the end of the enrolment path — a stanza merged into `peers.yaml`, Flux rolling
|
||||
the DaemonSet, and `wg show` listing a peer — is still unexercised.
|
||||
- **The duplicate-NAME warning** (`grep -qi "^ *# *${NAME} "`) deliberately still inspects
|
||||
comments, because per-peer labels are comments. It is no longer tripped by the template, but I
|
||||
did not test it against a real enrolled peer.
|
||||
- **Nothing about client traffic.** The WAN port-forward is still not configured and there is still
|
||||
no off-LAN vantage point, so the tunnel has carried no packet.
|
||||
@@ -0,0 +1,103 @@
|
||||
---
|
||||
date: 2026-10-06
|
||||
branch: fix/peers-configmap-content-hash
|
||||
title: A merged VPN peer never reached the tunnel — the peer register was a fixed-name ConfigMap, so nothing rolled the pods
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
The operator enrolled a phone, merged the peer, and **`wg show` listed no peers.** Measured on prod
|
||||
2026-10-06, with the peer already on `main`:
|
||||
|
||||
| observation | value |
|
||||
|---|---|
|
||||
| peer stanza on `main` | present (`AllowedIPs = 10.90.0.2/32`) |
|
||||
| GitRepository artifact | `main@sha1:e1bebb8` — Flux had fetched it |
|
||||
| `vpn-ingress` Kustomization `lastAppliedRevision` | `main@sha1:5928d81` — **behind**, 10m interval |
|
||||
| live `wg-peers` ConfigMap, active `AllowedIPs` lines | **0** |
|
||||
| DaemonSet volume reference | `configMap.name: wg-peers` — a **fixed** name |
|
||||
| DaemonSet `generation` / `observedGeneration` | **1 / 1** — it had **never rolled** |
|
||||
| init container, all three pods | `wg0.conf assembled: 0 peer(s)` |
|
||||
|
||||
The lag was only the reconcile interval. **The defect is the fixed name.** When Flux does update a
|
||||
fixed-name ConfigMap it updates it *in place*: the DaemonSet's pod template is byte-identical, so
|
||||
no rollout is triggered — and because the init container assembles `wg0.conf` **once at pod start**
|
||||
into a tmpfs, the running tunnel keeps the old peer list indefinitely. Adding or revoking a peer
|
||||
would have had **no effect at all** until something unrelated happened to restart the pods. For a
|
||||
*revocation* that is a security defect, not an inconvenience: the operator would delete a stanza,
|
||||
see a green PR, and the device would keep working.
|
||||
|
||||
My runbook and README both asserted "PR → merge → Flux rolls the DaemonSet (~1 min)". That was
|
||||
never true.
|
||||
|
||||
**This repo already documents the fix, and I had read the file.** `runbooks/kustomization.yaml`:
|
||||
*"The generated name carries kustomize's content hash, so editing any runbook rolls the server
|
||||
automatically and a stale projection is impossible."*
|
||||
|
||||
So the register becomes a **content-hashed generator**:
|
||||
|
||||
- `platform/vpn-ingress/peers.yaml` (inline ConfigMap) → **`platform/vpn-ingress/peers.conf`**, a
|
||||
plain file, plus a `configMapGenerator` in `kustomization.yaml`. The operator's enrolled peer was
|
||||
carried across verbatim. Every line is WireGuard config and `#` is a legal comment there, so the
|
||||
documentation now travels *inside the mounted file* instead of in a YAML wrapper that gets
|
||||
stripped.
|
||||
- **`namespace: vpn-ingress` added to the kustomization — REQUIRED, not tidiness.** A generated
|
||||
ConfigMap inherits no namespace; every other object here declares `vpn-ingress`; and kustomize's
|
||||
`nameReference` transformer will not rewrite a reference across a namespace boundary. Without it
|
||||
the build emitted `ConfigMap wg-peers-m9kcckcb5t` while the DaemonSet still mounted the unhashed
|
||||
`wg-peers`, which would not exist — the pods would wedge on a missing volume. I caught this only
|
||||
by building the output and reading the volumes block.
|
||||
- **`infra/vpn-ingress/new-peer.sh`**: `PEERS_FILE` repointed to `peers.conf`, and a missing
|
||||
register is now **fatal** instead of skipped. The old guard was
|
||||
`[ -r "$PEERS_FILE" ] && grep …`, so after the rename the only symptom would have been
|
||||
duplicate-address detection quietly never firing again — the silent-skip shape this repo treats
|
||||
as a defect in its own right.
|
||||
- Stale `peers.yaml` references swept from the README, runbook, `daemonset.yaml`,
|
||||
`networkpolicy.yaml`, the SOPS example, and the script's own **user-facing output** (it told the
|
||||
operator to paste into "peers.yaml, under data.peers.conf", which would no longer exist).
|
||||
- The runbook gained the verification that actually distinguishes this failure — the DaemonSet's
|
||||
`generation` must *increase* after the merge — and a troubleshooting row for `0 peer(s)` after a
|
||||
merge.
|
||||
|
||||
Flux's `prune: true` removes the orphaned fixed-name `wg-peers` ConfigMap on the next reconcile.
|
||||
|
||||
No sensitive paths (`platform/vpn-ingress/**`, `infra/vpn-ingress/**`, `runbooks/**`,
|
||||
`decisions/**`), so this merges on a green gate with no override.
|
||||
|
||||
## What I verified
|
||||
|
||||
Built and run in the pinned toolbox, 2026-10-06 (kustomize v5.8.1 via `kubectl kustomize`).
|
||||
|
||||
- **The reference is now rewritten end to end**: generated `ConfigMap wg-peers-m9kcckcb5t` in
|
||||
namespace `vpn-ingress`, and the DaemonSet mounts **`wg-peers-m9kcckcb5t`**. Before adding
|
||||
`namespace:` the same build mounted the unhashed `wg-peers` — so the namespace line is load
|
||||
bearing, measured both ways.
|
||||
- **The hash tracks content, which is the whole mechanism.** `wg-peers-m9kcckcb5t` →
|
||||
append a second peer → `wg-peers-dgg7ht9c74` → restore → `wg-peers-m9kcckcb5t`. Deterministic
|
||||
and reversible, so an edit necessarily changes the pod template.
|
||||
- **The operator's enrolled peer survived the file conversion byte-for-byte**: the active stanza in
|
||||
`peers.conf` (`PublicKey = pBRSHrRy…`, `AllowedIPs = 10.90.0.2/32`) is identical to the one on
|
||||
`main`, compared directly against `git show origin/main:platform/vpn-ingress/peers.yaml`.
|
||||
- **The build still emits all five objects**: Namespace, ConfigMap, Secret, DaemonSet,
|
||||
NetworkPolicy — and the `Namespace` object's own name is still `vpn-ingress` (the namespace
|
||||
transformer did not rewrite it).
|
||||
- **`new-peer.sh` works against the new register**: `spare-laptop 7` renders a profile and prints
|
||||
"append this to platform/vpn-ingress/peers.conf"; `dupe 2` is **refused** because the operator's
|
||||
real `.2` now occupies it — the first time the duplicate check has been exercised against real
|
||||
data rather than against its own example.
|
||||
- **Toolbox gate**: rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`,
|
||||
`l1` and `approval-policy` all pass; rc 3 is the two environmental skips. **Not green. Reported
|
||||
as rc 3.**
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **That the roll actually happens on the cluster.** The whole point of the change is unproven
|
||||
until Flux applies it: the success criterion is the DaemonSet's `generation` increasing and
|
||||
`wg0.conf assembled: 1 peer(s)` in the init log, followed by `wg show` listing the peer.
|
||||
- **That the tunnel then carries traffic.** No handshake has ever been observed. The WAN
|
||||
port-forward is reportedly configured but has never been exercised by a packet I can see, and I
|
||||
have no off-LAN vantage point — runbook step 6 from cellular remains the only honest test.
|
||||
- **Revocation**, which is the case where this defect was a security problem. Deleting a stanza
|
||||
should now roll the pods and drop the peer; untested.
|
||||
- **Whether the orphaned fixed-name `wg-peers` ConfigMap is actually pruned.** Reasoned from
|
||||
`prune: true` on the Kustomization, not observed.
|
||||
@@ -0,0 +1,113 @@
|
||||
---
|
||||
date: 2026-10-06
|
||||
branch: fix/vpn-ingress-ip-forward
|
||||
title: The VPN tunnel never came up — a hostNetwork pod cannot write net.ipv4.ip_forward, and wg-quick treats a failed PostUp as fatal
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
The remote-access VPN reconciled for the first time with a real server key (#60 then #62) and
|
||||
**the tunnel did not come up**. One line in my own manifest was the whole cause.
|
||||
|
||||
Measured in the `wireguard` container on prod, 2026-10-06:
|
||||
|
||||
```
|
||||
[#] ip link add dev wg0 type wireguard
|
||||
[#] wg addconf wg0 /dev/fd/63
|
||||
[#] ip -4 address add 10.90.0.1/24 dev wg0
|
||||
[#] ip link set mtu 1420 up dev wg0
|
||||
[#] sysctl -q -w net.ipv4.ip_forward=1
|
||||
sysctl: permission denied on key "net.ipv4.ip_forward"
|
||||
[#] ip link delete dev wg0
|
||||
**** Tunnel /config/wg_confs/wg0.conf failed, will stop all others! ****
|
||||
```
|
||||
|
||||
Everything up to that point worked. **The cause is not a missing capability.** With
|
||||
`hostNetwork: true` the runtime mounts `/proc/sys` **read-only** in the container — confirmed in the
|
||||
pod's own `/proc/mounts`: `proc /proc/sys proc ro,nosuid,nodev,noexec,relatime` — precisely because
|
||||
those sysctls are the *host's* globals and a container must not rewrite them. `CAP_NET_ADMIN` cannot
|
||||
lift a read-only mount, so this can never succeed from here. And `wg-quick` treats any failed
|
||||
`PostUp` as fatal, so it deleted the interface it had just finished building.
|
||||
|
||||
**The line was never needed.** `node_baseline` writes `net.ipv4.ip_forward=1` to
|
||||
`/etc/sysctl.d/99-k3s-cilium.conf`, and reading the value from inside the pod returns `1`. I had
|
||||
written the header comment *correctly* — "ip_forward is ALREADY 1 on these nodes … so no pod sysctl
|
||||
is needed" — and then emitted the sysctl anyway, justified in its own comment as "re-asserted so
|
||||
this component does not silently depend on another role's sysctl file surviving". That defensive
|
||||
re-assertion is the entire outage. **The lesson is narrow and reusable: in a hostNetwork pod,
|
||||
verify host state, never write it.**
|
||||
|
||||
Fix:
|
||||
|
||||
- **`platform/vpn-ingress/daemonset.yaml`**: the `PostUp = sysctl …` line is gone, replaced by a
|
||||
**read** in the init container — `grep -qx 1 /proc/sys/net/ipv4/ip_forward` — which fails closed
|
||||
with a message naming the `node_baseline` role. Reading `/proc/sys` is permitted; only writing is
|
||||
not. Fail-closed still matters: a tunnel that comes up but cannot forward would satisfy every
|
||||
probe in this component and silently drop every client's traffic.
|
||||
- The header bullet that said "no pod sysctl is needed" now also says **do not attempt one**, with
|
||||
the mechanism, so the next author does not re-add it for the same reason I did.
|
||||
- The `securityContext` comment claimed NET_ADMIN was for "create the wg0 link, set the host sysctl,
|
||||
and write the netfilter rules". The middle clause was false; corrected.
|
||||
- The image's `chown: … Operation not permitted` startup run — plus its own "Permissions could not
|
||||
be set … we will not provide support for it" — is now documented as **expected and harmless**, in
|
||||
both the manifest and the runbook, so nobody chases it. Capabilities are dropped to `NET_ADMIN`
|
||||
only, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs as root,
|
||||
and `wg-quick` proceeds past it (as it did here — the failure was 30 lines later).
|
||||
- **`runbooks/vpn-remote-access.md`**: a "what a healthy start looks like" block with the real log
|
||||
lines; `ip_forward is 0` added to the init-failure table; and the note that `0/1 Running` with
|
||||
`Startup probe failed: Unable to access interface: No such device` is the signature of `wg-quick`
|
||||
tearing the tunnel down — read the `[#]` line it died on.
|
||||
- **`platform/vpn-ingress/README.md`**: a table of what the first reconcile settled (below).
|
||||
|
||||
No sensitive paths: `platform/vpn-ingress/**`, `runbooks/**` and `decisions/**` only. This merges on
|
||||
a green gate with no override.
|
||||
|
||||
## What I verified
|
||||
|
||||
Live prod, read-only, 2026-10-06. **Four questions this component's design rested on are now
|
||||
measured rather than assumed — three went the right way, one was the defect above.**
|
||||
|
||||
- **The GitOps path is proven end to end.** `Kustomization vpn-ingress` `lastAppliedRevision:
|
||||
main@sha1:cf7d5e4` equals the GitRepository artifact revision. The Secret carries
|
||||
`kustomize.toolkit.fluxcd.io/name: vpn-ingress`, so Flux delivered it.
|
||||
- **SOPS decryption works, proven by digest rather than by eye.** The value in
|
||||
`platform/vpn-ingress/wg-server-key.sops.yaml` on `main`, decrypted, is 44 bytes, is **not** the
|
||||
placeholder, and **`wg pubkey` accepts it**. Its SHA-256 equals the SHA-256 of the live Secret's
|
||||
decoded `data.privatekey` (`d7c8f8de6f252649…` both sides). No key material was printed at any
|
||||
point — lengths, booleans and digests only.
|
||||
- **The `wg pubkey` guard fired correctly, and its `ENC[` branch was never needed.** During the
|
||||
kubelet window before the updated Secret propagated, all three pods reported *"It is still the
|
||||
committed REPLACE-ME PLACEHOLDER"* — the guard distinguishing a stale/unfilled value from
|
||||
ciphertext, exactly as designed. It cleared on its own once the mount refreshed; **a transient
|
||||
`REPLACE-ME` CrashLoop for a minute or two after the key PR merges is expected, not a fault.**
|
||||
- **The kernel module reaches the pod**: `ip link add dev wg0 type wireguard` succeeded and
|
||||
`/proc/modules` lists `wireguard`. So the `node_baseline` addition works and `CAP_SYS_MODULE` is
|
||||
correctly not required.
|
||||
- **`allowPrivilegeEscalation: false` does NOT break the image's s6 init** — the container reaches
|
||||
`wg-quick` normally. The flag stays; the fear recorded in #55 is closed.
|
||||
- **The iptables backends MATCH, which was my single biggest documented doubt.** The image ships
|
||||
`iptables v1.8.13 (nf_tables)` via `/usr/sbin/xtables-nft-multi`, and Debian 13 is nf_tables. The
|
||||
nightmare case — rules accepted into a table the kernel never consults, so handshakes succeed and
|
||||
every forwarded connection hangs — cannot occur on this pairing.
|
||||
- **`net.ipv4.ip_forward` is `1`** in the host netns the pod shares, and `/proc/sys` is `ro` there.
|
||||
Both read directly from the pod.
|
||||
- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
|
||||
`lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two
|
||||
environmental skips (`gitea-settings`, conformance L2/L3). **Not green. Reported as rc 3.**
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **That the tunnel now comes up.** This fix has not reconciled. The next reconcile is the test, and
|
||||
its success criterion is specific: pods `1/1 Running`, and `wg show` reporting `interface: wg0`
|
||||
with `listening port: 51820`.
|
||||
- **That the MASQUERADE and FORWARD rules actually apply.** They are the next statements after the
|
||||
one that failed, so **they have never executed**. The backends matching (above) removes the worst
|
||||
failure mode but proves nothing about the rules themselves. First check after the pods go Ready:
|
||||
`iptables -t nat -S POSTROUTING | grep 10.90.0.0/24` on a node.
|
||||
- **Anything about client traffic.** Still zero peers enrolled, still no WAN port-forward, still no
|
||||
off-LAN vantage point. The VPN has carried no packet.
|
||||
- **Whether the transient `REPLACE-ME` window has an upper bound.** It cleared within ~2 minutes
|
||||
here; I did not determine the kubelet refresh behaviour, so a longer stall after a key change is
|
||||
uncharacterised.
|
||||
- **The `sops set` hand-off command**, still. The operator filled the key in by their own route
|
||||
(#62), so the exact quoting I documented inside `dev.sh run bash -c` remains unexercised.
|
||||
@@ -3,7 +3,7 @@
|
||||
# new-peer.sh — enrol one device on the remote-access VPN (platform/vpn-ingress).
|
||||
#
|
||||
# Generates a WireGuard keypair, renders the device's client profile, prints a QR code for a phone,
|
||||
# and prints the one `[Peer]` stanza to paste into platform/vpn-ingress/peers.yaml. The device's
|
||||
# and prints the one `[Peer]` stanza to paste into platform/vpn-ingress/peers.conf. The device's
|
||||
# private half is written to ONE 0600 file and nowhere else: not sent to the cluster, not committed,
|
||||
# never logged.
|
||||
#
|
||||
@@ -62,7 +62,7 @@ set -euo pipefail
|
||||
die() { printf 'new-peer.sh: %s\n' "$*" >&2; exit 1; }
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.yaml"
|
||||
PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.conf"
|
||||
|
||||
# --- defaults, each mirroring a value that is DECLARED elsewhere in this repo -------------------
|
||||
# These are duplicated here rather than parsed out of the YAML on purpose: this script is meant to
|
||||
@@ -133,11 +133,22 @@ CLIENT_ADDR="${CLIENT_BASE}.${OCTET}"
|
||||
# --- refuse a collision, because the failure is otherwise invisible -----------------------------
|
||||
# Two devices sharing an address does not error anywhere: the server routes that address to
|
||||
# whichever peer handshook most recently, so BOTH devices appear to work intermittently.
|
||||
if [ -r "$PEERS_FILE" ] && grep -q "AllowedIPs *= *${CLIENT_ADDR}/32" "$PEERS_FILE"; then
|
||||
die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet."
|
||||
# COMMENT LINES ARE NOT ALLOCATIONS. This used to grep the whole file and so matched the
|
||||
# COMMENTED-OUT EXAMPLE STANZA in the register, which made `.2` — the first address anyone would
|
||||
# ask for — permanently un-allocatable with a message claiming it was already taken. Strip
|
||||
# comments first: a real stanza's `AllowedIPs` line is never commented, while both the template
|
||||
# and every peer's `# <name> — <owner>` label are.
|
||||
# FAIL if the register is missing rather than skipping the check. The path was silently wrong once
|
||||
# (it still said peers.yaml after the file became peers.conf), and because the old guard was
|
||||
# `[ -r "$PEERS_FILE" ] && ...` the only symptom would have been duplicate detection quietly
|
||||
# never firing again — the exact silent-skip shape this repo treats as a defect.
|
||||
[ -r "$PEERS_FILE" ] || die "cannot read the peer register '$PEERS_FILE'.
|
||||
Duplicate-address detection cannot run, so this script refuses to issue a profile."
|
||||
if grep -vE '^[[:space:]]*#' "$PEERS_FILE" | grep -q "AllowedIPs *= *${CLIENT_ADDR}/32"; then
|
||||
die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.conf. Pick another octet."
|
||||
fi
|
||||
if [ -r "$PEERS_FILE" ] && grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
|
||||
printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.yaml.\n' "$NAME" >&2
|
||||
if grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
|
||||
printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.conf.\n' "$NAME" >&2
|
||||
fi
|
||||
|
||||
# --- output location: 0700, and never inside the repo -------------------------------------------
|
||||
@@ -204,7 +215,7 @@ cat <<EOF
|
||||
Profile : ${PROFILE} (0600; contains the private key)
|
||||
|
||||
------------------------------------------------------------------------------------------
|
||||
STEP 1 — add this to platform/vpn-ingress/peers.yaml, under data.peers.conf, then PR it:
|
||||
STEP 1 — append this to platform/vpn-ingress/peers.conf, then PR it:
|
||||
|
||||
# ${NAME} — <owner>, enrolled $(date -u +%Y-%m-%d)
|
||||
[Peer]
|
||||
@@ -223,7 +234,7 @@ cat <<EOF
|
||||
STEP 3 — delete the profile from this machine once the device has it:
|
||||
shred -u ${PROFILE} 2>/dev/null || rm -f ${PROFILE}
|
||||
|
||||
The tunnel will not work until the peers.yaml PR is MERGED and Flux has rolled the DaemonSet
|
||||
The tunnel will not work until the peers.conf PR is MERGED and Flux has rolled the DaemonSet
|
||||
(~1 min). Until then the device handshakes and gets silence, exactly like an unknown key.
|
||||
------------------------------------------------------------------------------------------
|
||||
EOF
|
||||
|
||||
@@ -13,7 +13,7 @@ egress out the house WAN.
|
||||
| Transport | WireGuard, **udp/51820**, kernel implementation |
|
||||
| Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node |
|
||||
| Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices |
|
||||
| Authorisation | a device's **public key** is in [`peers.yaml`](./peers.yaml), or it cannot connect |
|
||||
| Authorisation | a device's **public key** is in [`peers.conf`](./peers.conf), or it cannot connect |
|
||||
| The one secret | the server's private key — **SOPS-encrypted in this tree**, `wg-server-key.sops.yaml` |
|
||||
| Exposure | one hand-made WAN port-forward on the SOHO router, off-repo |
|
||||
| Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) |
|
||||
@@ -129,7 +129,7 @@ translations win here.
|
||||
|
||||
| half | where | why |
|
||||
|---|---|---|
|
||||
| peer **public** keys + addresses | [`peers.yaml`](./peers.yaml), **plaintext in git** | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
|
||||
| peer **public** keys + addresses | [`peers.conf`](./peers.conf), **plaintext in git**, as a content-hashed `configMapGenerator` so an edit rolls the pods | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
|
||||
| the server's **private** key | `wg-server-key.sops.yaml`, **SOPS-encrypted in git** | the only real secret (R-SEC-01, spec docs/07). Ships with a `REPLACE-ME` placeholder: an agent must not mint a long-lived private key (CLAUDE.md §6), so the operator `sops set`s it in |
|
||||
|
||||
The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** —
|
||||
@@ -166,13 +166,26 @@ NetworkPolicy applies.
|
||||
## Operating it
|
||||
|
||||
* **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
|
||||
stanza into `peers.yaml` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
|
||||
* **Revoke a device:** delete its stanza → PR → merge. No other step; the key stops working.
|
||||
stanza into `peers.conf` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
|
||||
* **Revoke a device:** delete its stanza from `peers.conf` → PR → merge. No other step; the key
|
||||
stops working as soon as the pods roll (the generator's content hash guarantees they do).
|
||||
* **Is it up?** `kubectl --context k3s-prod -n vpn-ingress get ds wireguard` — three Ready.
|
||||
* **Who is connected?** `kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show`
|
||||
(`latest handshake` under ~3 min means live). Never dump `wg showconf` — it prints the private key.
|
||||
* **Full bring-up and verification:** `runbooks/vpn-remote-access.md`.
|
||||
|
||||
## Measured on prod, 2026-10-06 — what is now known rather than assumed
|
||||
|
||||
The first real reconcile settled four things this component's design rested on. Three went the
|
||||
right way; one was a defect in my own manifest.
|
||||
|
||||
| question | answer |
|
||||
|---|---|
|
||||
| Does the **kernel module** reach the pod? | **Yes.** `ip link add dev wg0 type wireguard` succeeds, and `/proc/modules` lists `wireguard`, after the `node_baseline` run. `CAP_SYS_MODULE` is correctly not needed. |
|
||||
| Does `allowPrivilegeEscalation: false` break the image's **s6 init**? | **No.** The container reaches `wg-quick` normally. The flag stays. |
|
||||
| Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
|
||||
| Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
|
||||
|
||||
## Prerequisites outside this directory
|
||||
|
||||
1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
|
||||
@@ -199,7 +212,7 @@ NetworkPolicy applies.
|
||||
covered generically by `KubeDeploymentReplicasMismatch`/`KubePodNotReady` from
|
||||
kube-prometheus-stack. **This is the top follow-up.**
|
||||
* **No preshared keys.** They would add a symmetric post-quantum hedge on top of the Noise
|
||||
handshake — and would be a real secret, forcing `peers.yaml` into SOPS and costing the
|
||||
handshake — and would be a real secret, forcing `peers.conf` into SOPS and costing the
|
||||
reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
|
||||
`wg-server-key` Secret.
|
||||
* **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6
|
||||
|
||||
@@ -37,6 +37,10 @@
|
||||
# /etc/sysctl.d/99-k3s-cilium.conf), so no pod sysctl is needed — and `net.ipv4.ip_forward`
|
||||
# is NOT in the kubelet's safe-sysctl allowlist, so a pod-netns version of this would have
|
||||
# required `--allowed-unsafe-sysctls` on all three kubelets (a node-restart change).
|
||||
# COROLLARY LEARNED THE HARD WAY (prod, 2026-10-06): this pod must not merely avoid NEEDING
|
||||
# the sysctl, it must not ATTEMPT it. hostNetwork implies /proc/sys is mounted read-only, so
|
||||
# `sysctl -w` fails with EPERM regardless of capabilities, and wg-quick treats a failed PostUp
|
||||
# as fatal and deletes the interface. Verify the value; never write it.
|
||||
# * no LoadBalancer VIP, so no CiliumL2AnnouncementPolicy lease lottery (the defect that cost
|
||||
# this LAN an 8-day Minecraft outage) and no `externalTrafficPolicy` trade-off at all.
|
||||
# * no double encapsulation. WireGuard's UDP lands on the node's own NIC, so the underlay is a
|
||||
@@ -104,7 +108,7 @@ spec:
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
automountServiceAccountToken: false
|
||||
# ── Assemble the running config: PUBLIC peer register (git) + PRIVATE server key (Secret) ──
|
||||
# The two halves are deliberately stored apart (peers.yaml explains why) and are only ever
|
||||
# The two halves are deliberately stored apart (peers.conf explains why) and are only ever
|
||||
# joined here, at pod start, into a tmpfs that is destroyed with the pod. The server's private
|
||||
# key is therefore never written to any node's disk, never committed, and never logged.
|
||||
initContainers:
|
||||
@@ -171,6 +175,19 @@ spec:
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# The pod shares the HOST netns, so this reads the node's real setting. Reading
|
||||
# /proc/sys is permitted even though writing is not (see the PostUp note below).
|
||||
# Fail closed: a tunnel that comes up but cannot forward would look healthy to every
|
||||
# probe here and simply drop every client's traffic.
|
||||
if ! grep -qx 1 /proc/sys/net/ipv4/ip_forward; then
|
||||
echo "FATAL: net.ipv4.ip_forward is 0 on this node, so forwarded VPN traffic would" >&2
|
||||
echo " be dropped. This pod CANNOT set it: with hostNetwork the runtime mounts" >&2
|
||||
echo " /proc/sys read-only, because it is the host's global setting." >&2
|
||||
echo " Fix on the NODE: run the node_baseline role (it writes" >&2
|
||||
echo " /etc/sysctl.d/99-k3s-cilium.conf) — runbooks/vpn-remote-access.md step 1." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
{
|
||||
echo '[Interface]'
|
||||
echo "Address = $base.1/$mask"
|
||||
@@ -184,9 +201,20 @@ spec:
|
||||
echo 'MTU = 1420'
|
||||
printf 'PrivateKey = %s\n' "$key"
|
||||
# ── Forwarding + SNAT, applied in the HOST netns when wg0 comes up ──
|
||||
# ip_forward is already 1 from node_baseline; re-asserted so this component does not
|
||||
# silently depend on another role's sysctl file surviving.
|
||||
echo 'PostUp = sysctl -q -w net.ipv4.ip_forward=1'
|
||||
# NO `sysctl -w net.ipv4.ip_forward=1` HERE. It is IMPOSSIBLE from this pod and it was
|
||||
# the only thing stopping the tunnel coming up — MEASURED on prod 2026-10-06:
|
||||
# [#] sysctl -q -w net.ipv4.ip_forward=1
|
||||
# sysctl: permission denied on key "net.ipv4.ip_forward"
|
||||
# [#] ip link delete dev wg0
|
||||
# and wg-quick then tore the whole tunnel down. The cause is not a missing
|
||||
# capability: with `hostNetwork: true` the runtime mounts /proc/sys READ-ONLY in the
|
||||
# container (confirmed in /proc/mounts: `proc /proc/sys proc ro,...`), precisely
|
||||
# because those sysctls are the HOST's globals and a container must not rewrite
|
||||
# them. CAP_NET_ADMIN does not and cannot lift a read-only mount.
|
||||
# It was never needed anyway: node_baseline writes net.ipv4.ip_forward=1 to
|
||||
# /etc/sysctl.d/99-k3s-cilium.conf, and reading it from this pod returns 1. The
|
||||
# init container now VERIFIES that (a read, which /proc/sys does allow) and fails
|
||||
# closed with a pointed message, which is what the write was pretending to achieve.
|
||||
# SNAT client traffic to the node's address. REQUIRED: without it a LAN host would
|
||||
# reply to 10.90.0.x, find no route, hand the packet to the router, and the router
|
||||
# would drop it — so every connection would hang rather than fail. Matching on
|
||||
@@ -246,8 +274,10 @@ spec:
|
||||
# `kubectl get pod -o wide` shows what this listener owns.
|
||||
- { name: wireguard, containerPort: 51820, hostPort: 51820, protocol: UDP }
|
||||
securityContext:
|
||||
# NET_ADMIN is the whole privilege budget: create the wg0 link, set the host sysctl, and
|
||||
# write the two netfilter rules above. Everything else is dropped, and the container is
|
||||
# NET_ADMIN is the whole privilege budget: create the wg0 link and write the netfilter
|
||||
# rules above. (It does NOT let the pod set a host sysctl — /proc/sys is read-only under
|
||||
# hostNetwork — which is why nothing here tries.) Everything else is dropped, and the
|
||||
# container is
|
||||
# NOT privileged — which also keeps it clear of restrict-host-namespaces' second rule.
|
||||
# SYS_MODULE is deliberately NOT granted: the `wireguard` module is loaded by the node
|
||||
# (infra/ansible/roles/node_baseline, node_baseline_kernel_modules), so this container
|
||||
@@ -261,6 +291,16 @@ spec:
|
||||
- NET_ADMIN
|
||||
# readOnlyRootFilesystem intentionally UNSET: the image's s6 init writes under /run and
|
||||
# /var during startup. Same deferral as crafty/headlamp.
|
||||
#
|
||||
# EXPECTED, HARMLESS STARTUP NOISE — do not "fix" it. Because capabilities are dropped
|
||||
# to NET_ADMIN only, the image's s6 init cannot chown /config (no CAP_CHOWN) and logs a
|
||||
# run of `chown: ... Operation not permitted` plus "Permissions could not be set. This
|
||||
# is probably because your volume mounts are remote or read-only" and a scary "we will
|
||||
# not provide support for it". It is wrong about the cause and it does not matter:
|
||||
# /config is a writable tmpfs emptyDir, everything here runs as root (uid 0) anyway, and
|
||||
# wg-quick proceeds normally — verified on prod 2026-10-06, where the tunnel's only
|
||||
# failure was the sysctl above, well past these warnings. Granting CAP_CHOWN to silence
|
||||
# log noise would widen the privilege set of an internet-facing pod for nothing.
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
|
||||
@@ -11,10 +11,16 @@
|
||||
# NOT in platform/kustomization.yaml — see clusters/base/vpn-ingress.yaml.
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
# REQUIRED for the configMapGenerator below to be wired up at all, not merely tidy. A generated
|
||||
# ConfigMap inherits NO namespace, every other object here declares `vpn-ingress`, and kustomize's
|
||||
# nameReference transformer will not rewrite a reference across a namespace boundary — so without
|
||||
# this the DaemonSet keeps mounting the unhashed name `wg-peers`, which does not exist, and the
|
||||
# pods wedge on a missing volume. Verified by building it both ways (kustomize v5.8.1).
|
||||
# Same placement as runbooks/kustomization.yaml, which uses `namespace: runbooks` for its own
|
||||
# content generator.
|
||||
namespace: vpn-ingress
|
||||
resources:
|
||||
- namespace.yaml
|
||||
# The peer register: which devices may dial in. PLAINTEXT and reviewable on purpose (see the file).
|
||||
- peers.yaml
|
||||
# The listener itself: hostNetwork DaemonSet, udp/51820 on every node.
|
||||
- daemonset.yaml
|
||||
# Namespace default-deny baseline. NOTE: does NOT constrain the hostNetwork DaemonSet — the file
|
||||
@@ -38,3 +44,26 @@ resources:
|
||||
# hands Flux the raw ciphertext, the Secret APPLIES CLEANLY full of ENC[...], and nothing in any
|
||||
# Flux condition looks wrong. The init container's `wg pubkey` guard exists partly to turn exactly
|
||||
# that into a named error rather than a silently non-working tunnel.
|
||||
|
||||
# ── THE PEER REGISTER, as a CONTENT-HASHED ConfigMap ─────────────────────────────────────────
|
||||
# peers.conf (plaintext, reviewable — see the file) becomes the `wg-peers` ConfigMap. It is a
|
||||
# GENERATOR, not a plain resource, and that distinction is load-bearing rather than stylistic:
|
||||
# kustomize appends a hash of the content to the generated name and rewrites the DaemonSet's
|
||||
# volume reference to match, so adding or revoking a peer changes the pod template and ROLLS THE
|
||||
# PODS automatically.
|
||||
#
|
||||
# A fixed-name ConfigMap (which is what this component shipped first) does not do that. Flux
|
||||
# updates it in place, the pod template is byte-identical, no rollout is triggered — and because
|
||||
# the init container assembles wg0.conf ONCE at pod start into a tmpfs, the running tunnel keeps
|
||||
# the old peer list indefinitely. Measured on prod 2026-10-06: a merged peer reached the
|
||||
# ConfigMap while `wg show` listed no peers and the DaemonSet still read generation=1, having
|
||||
# never rolled. The runbook's "PR -> merge -> Flux rolls the DaemonSet" was false until this.
|
||||
#
|
||||
# DO NOT set `disableNameSuffixHash` here. The hash IS the mechanism.
|
||||
configMapGenerator:
|
||||
- name: wg-peers
|
||||
files:
|
||||
- peers.conf
|
||||
options:
|
||||
labels:
|
||||
app.kubernetes.io/name: vpn-ingress
|
||||
@@ -19,7 +19,7 @@
|
||||
# protects this port, and it would be enforcing nothing. So there isn't one.
|
||||
#
|
||||
# WHAT ACTUALLY PROTECTS THE EXPOSED PORT, then, since the CNI does not:
|
||||
# 1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.yaml gets
|
||||
# 1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.conf gets
|
||||
# NO REPLY — see README "Why this is safe to expose". This is the real control.
|
||||
# 2. The SOHO router's forward: only udp/51820 is forwarded inbound, to one node.
|
||||
# 3. The blast radius once authenticated is deliberately the whole LAN — that IS the requirement
|
||||
|
||||
@@ -1,4 +1,16 @@
|
||||
---
|
||||
# THIS FILE IS THE MOUNTED CONTENT. It is turned into the `wg-peers` ConfigMap by
|
||||
# `configMapGenerator` in kustomization.yaml, whose generated name carries a CONTENT HASH — so
|
||||
# editing this file changes the ConfigMap name, which changes the DaemonSet's pod template, which
|
||||
# ROLLS THE PODS. That is the whole reason it is a generator and not an inline ConfigMap manifest:
|
||||
# a fixed-name ConfigMap updates in place, the pod template never changes, no rollout happens, and
|
||||
# the init container's wg0.conf (built once at pod start, in tmpfs) keeps the OLD peer list
|
||||
# forever. Measured 2026-10-06: a merged peer sat in the ConfigMap while `wg show` listed none and
|
||||
# the DaemonSet reported generation=1 — it had never rolled. Same mechanism and same fix as
|
||||
# runbooks/kustomization.yaml.
|
||||
#
|
||||
# Every line here is WireGuard config. `#` is a legal comment in it, so the documentation below
|
||||
# travels inside the file that is actually mounted, rather than in a wrapper that gets stripped.
|
||||
#
|
||||
# ============================================================================================
|
||||
# THE PEER REGISTER — every device allowed to dial in, in PLAINTEXT, in git, on purpose.
|
||||
# ============================================================================================
|
||||
@@ -41,33 +53,40 @@
|
||||
# the Noise handshake and would be a genuine secret — which would force this register into SOPS
|
||||
# and cost the reviewability above. Revisit only if the threat model changes; if it does, the PSK
|
||||
# belongs in the `wg-server-key` Secret, not here. See README "Hardening not taken".
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: wg-peers
|
||||
namespace: vpn-ingress
|
||||
labels:
|
||||
app.kubernetes.io/name: vpn-ingress
|
||||
data:
|
||||
# Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
|
||||
#
|
||||
# *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
|
||||
# An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
|
||||
# authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
|
||||
# DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
|
||||
# The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
|
||||
# (`wg show` reports an interface with no peers).
|
||||
#
|
||||
# COPY THIS TEMPLATE, uncomment, one block per device:
|
||||
#
|
||||
# # pixel-8 — Keith's phone, enrolled 2026-10-05
|
||||
# [Peer]
|
||||
# PublicKey = <the public key new-peer.sh printed>
|
||||
# AllowedIPs = 10.90.0.2/32
|
||||
#
|
||||
# Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
|
||||
# tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
|
||||
# it an incident responder cannot tell which stanza to delete.
|
||||
peers.conf: |
|
||||
# No peers enrolled yet. See the template in this file's comments and
|
||||
# ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
|
||||
#
|
||||
# name: wg-peers
|
||||
# namespace: vpn-ingress
|
||||
# labels:
|
||||
# app.kubernetes.io/name: vpn-ingress
|
||||
# Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
|
||||
#
|
||||
# *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
|
||||
# An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
|
||||
# authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
|
||||
# DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
|
||||
# The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
|
||||
# (`wg show` reports an interface with no peers).
|
||||
#
|
||||
# COPY THIS TEMPLATE, uncomment, one block per device:
|
||||
#
|
||||
# # <device-name> — <owner>, enrolled <YYYY-MM-DD>
|
||||
# [Peer]
|
||||
# PublicKey = <the public key new-peer.sh printed>
|
||||
# AllowedIPs = 10.90.0.<octet>/32
|
||||
#
|
||||
# The placeholders above are deliberately NOT valid values. An earlier version of this template
|
||||
# used a real-looking `10.90.0.2/32`, and new-peer.sh's duplicate-address check — which grepped
|
||||
# the whole file — matched it, so `.2` could never be allocated and the script insisted it was
|
||||
# already taken. The check now ignores comments; keeping the example unmatchable too means a
|
||||
# future grep cannot be fooled either.
|
||||
#
|
||||
# Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
|
||||
# tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
|
||||
# it an incident responder cannot tell which stanza to delete.
|
||||
#
|
||||
# No peers enrolled yet. See the template in this file's comments and
|
||||
# ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
|
||||
# keith-phone — <owner>, enrolled 2026-10-06
|
||||
[Peer]
|
||||
PublicKey = pBRSHrRyG2hnsuZ6FvHfEzF8hOVTF5stHUb/7fW2Km8=
|
||||
AllowedIPs = 10.90.0.2/32
|
||||
@@ -5,7 +5,7 @@
|
||||
# WHAT THIS IS: the WireGuard SERVER's long-term private key for the remote-access VPN
|
||||
# (platform/vpn-ingress). It is the ONE secret in the whole component. The init container in
|
||||
# daemonset.yaml reads it at `/run/wg-server-key/privatekey` and joins it to the PLAINTEXT peer
|
||||
# register (peers.yaml) in a tmpfs, so the assembled wg0.conf never touches a node's disk.
|
||||
# register (peers.conf) in a tmpfs, so the assembled wg0.conf never touches a node's disk.
|
||||
#
|
||||
# WHY IT IS IN GIT, SOPS-ENCRYPTED, AND NOT HAND-CREATED. R-SEC-01 / spec docs/07: secrets live in
|
||||
# source, encrypted, and reach the cluster only through Flux. An out-of-band `kubectl create secret`
|
||||
|
||||
@@ -21,7 +21,7 @@ behaves like a machine on the LAN: full reach to `192.168.2.0/24`, the Gateway's
|
||||
DNS, and internet egress out the house WAN.
|
||||
|
||||
Authorisation is one thing only: the device's **public key is in
|
||||
`platform/vpn-ingress/peers.yaml`**. An unlisted key gets no reply at all.
|
||||
`platform/vpn-ingress/peers.conf`**. An unlisted key gets no reply at all.
|
||||
|
||||
> **An enrolled device is as privileged as a machine sitting in the house** — including the three
|
||||
> k3s nodes (`192.168.2.10/.11/.12`: apiserver 6443, kubelet 10250, SSH 22). That is what "reach
|
||||
@@ -134,6 +134,37 @@ specific failure in its log — so you never get a server that silently cannot c
|
||||
| `It is EMPTY` | the value was blanked |
|
||||
| `still SOPS CIPHERTEXT (starts with ENC[)` | the Secret reached the cluster undecrypted — this environment's overlay is missing its `vpn-ingress` decryption patch. **This is the dangerous one:** the Secret applies cleanly and every Flux condition stays green |
|
||||
| `wg pubkey rejected it` with none of the above | a mangled value, or a *public* key pasted by mistake |
|
||||
| `net.ipv4.ip_forward is 0 on this node` | the node_baseline role has not run here (step 1). The pod **cannot** set this itself — under `hostNetwork` the runtime mounts `/proc/sys` read-only, because it is the host's global setting |
|
||||
| `0 peer(s)` after you merged a peer | the DaemonSet did not roll. Check its `generation` (above). Pre-2026-10-06 this was guaranteed, because the register was a fixed-name ConfigMap; if it recurs, something has reverted `peers.conf` to a plain resource instead of a `configMapGenerator` |
|
||||
|
||||
### What a healthy start looks like
|
||||
|
||||
Measured on prod 2026-10-06. The init container logs one line, naming the shape and never the key:
|
||||
|
||||
```
|
||||
wg0.conf assembled: 0 peer(s), server 10.90.0.1/24, udp/51820.
|
||||
```
|
||||
|
||||
then the `wireguard` container brings the tunnel up:
|
||||
|
||||
```
|
||||
**** Found WG conf /config/wg_confs/wg0.conf, adding to list ****
|
||||
**** Activating tunnel /config/wg_confs/wg0.conf ****
|
||||
[#] ip link add dev wg0 type wireguard
|
||||
[#] wg addconf wg0 /dev/fd/63
|
||||
[#] ip -4 address add 10.90.0.1/24 dev wg0
|
||||
[#] ip link set mtu 1420 up dev wg0
|
||||
```
|
||||
|
||||
**Ignore the `chown: ... Operation not permitted` run before that**, and the image's own
|
||||
"Permissions could not be set ... we will not provide support for it". Capabilities are dropped to
|
||||
`NET_ADMIN` only, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs
|
||||
as root, and `wg-quick` proceeds. It is wrong about the cause and it does not matter.
|
||||
|
||||
**A pod stuck `0/1 Running` with `Startup probe failed: Unable to access interface: No such device`
|
||||
means `wg-quick` tore the tunnel down** — read the `wireguard` container log for the `[#]` line it
|
||||
died on. That is the shape of a real failure; the probe is doing its job of refusing to call a
|
||||
server without a `wg0` "ready".
|
||||
|
||||
## Step 3 — forward udp/51820 on the router (once)
|
||||
|
||||
@@ -188,10 +219,23 @@ It generates the device's keypair, writes a `0600` client profile, prints a QR c
|
||||
|
||||
Then:
|
||||
|
||||
1. Paste the stanza into `platform/vpn-ingress/peers.yaml` under `data.peers.conf`, keeping the
|
||||
1. Append the stanza to `platform/vpn-ingress/peers.conf`, keeping the
|
||||
`# <name> — <owner>, enrolled <date>` comment. `wg show` can only ever print the key, so that
|
||||
comment is the only record of which stanza belongs to which device.
|
||||
2. PR → merge. Flux rolls the DaemonSet (~1 min).
|
||||
2. PR → merge. Flux rolls the DaemonSet (~1 min), because `peers.conf` is a
|
||||
**content-hashed** `configMapGenerator`: editing it changes the ConfigMap's generated name,
|
||||
which changes the pod template, which forces the roll. (It did **not** work this way at first —
|
||||
a fixed-name ConfigMap updates in place, the pod template never changes, and the init container
|
||||
keeps the old peer list forever. Measured on prod 2026-10-06.) Confirm it actually rolled:
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress get ds wireguard \
|
||||
-o jsonpath="{.metadata.generation}{\"/\"}{.status.observedGeneration}{\"\n\"}"'
|
||||
```
|
||||
|
||||
The generation must have **increased** since before the merge. If it is still `1`, the roll did
|
||||
not happen and `wg show` will list no peer no matter how long you wait.
|
||||
3. Get the profile onto the device (QR scan for phones; "Import tunnel from file" on
|
||||
Windows/macOS; `/etc/wireguard/<name>.conf` + `wg-quick@<name>` on Linux).
|
||||
4. **Delete the profile from your machine**: `shred -u <path>`. It is the only copy of that
|
||||
@@ -231,7 +275,7 @@ if the router is not delivering the packets, and a `handshake` with no completed
|
||||
|
||||
## Revoking a device
|
||||
|
||||
Delete its `[Peer]` stanza from `peers.yaml`, PR, merge. Nothing else — the key stops working on the
|
||||
Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the
|
||||
next reconcile. **Never re-use that address octet**: an old profile would silently start working
|
||||
again as a different device.
|
||||
|
||||
|
||||
Reference in new issue
Block a user