Compare commits

...
Author SHA1 Message Date
jakman85andClaude Opus 5 2acfed460a fix(vpn-ingress): content-hash the peer register, or a merged peer never reaches the tunnel
gate / gate (pull_request) Successful in 1m52s
gate / conformance (pull_request) Has been skipped
The operator enrolled a phone, merged the peer, and `wg show` listed none.
Measured on prod 2026-10-06 with the peer already on main:

  live wg-peers ConfigMap, active AllowedIPs .. 0
  DaemonSet volume reference ................. configMap.name: wg-peers (FIXED)
  DaemonSet generation/observedGeneration .... 1 / 1   (never rolled)
  init container, all three pods ............. "0 peer(s)"

Part of it was just the 10m reconcile interval. The defect is the fixed name:
Flux updates a fixed-name ConfigMap IN PLACE, so the pod template is
byte-identical, so no rollout is triggered -- and the init container assembles
wg0.conf ONCE at pod start into a tmpfs, so the running tunnel keeps the old
peer list indefinitely. Adding or revoking a peer had no effect until something
unrelated restarted the pods. For a REVOCATION that is a security defect, not
an inconvenience: delete a stanza, see a green PR, device keeps working.

The runbook and README both claimed "PR -> merge -> Flux rolls the DaemonSet".
Never true. And this repo already documents the fix, in a file I had read --
runbooks/kustomization.yaml: "The generated name carries kustomize's content
hash, so editing any runbook rolls the server automatically and a stale
projection is impossible."

So: peers.yaml (inline ConfigMap) -> peers.conf (plain file) + a
configMapGenerator. The operator's enrolled peer is carried across verbatim.
Every line is WireGuard config and `#` is a legal comment there, so the
documentation now lives inside the file that is actually mounted.

`namespace: vpn-ingress` on the kustomization is REQUIRED, not tidiness: a
generated ConfigMap inherits no namespace, everything else here declares
vpn-ingress, and kustomize's nameReference will not rewrite across a namespace
boundary -- without it the build mounts the unhashed name and the pods wedge on
a missing volume. Caught only by building the output and reading the volumes.

new-peer.sh: PEERS_FILE repointed, and a missing register is now FATAL rather
than skipped -- the old `[ -r "$PEERS_FILE" ] && grep ...` guard meant that
after the rename the only symptom would have been duplicate detection quietly
never firing again. Stale peers.yaml references swept from six files including
the script's own user-facing output.

Verified: the DaemonSet now mounts wg-peers-m9kcckcb5t; the hash tracks content
(edit -> dgg7ht9c74 -> restore -> m9kcckcb5t); the enrolled peer is byte-identical
to main's; and `new-peer.sh dupe 2` is now correctly refused by the operator's
real allocation.

No sensitive paths; merges on a green gate.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-06 02:55:59 -05:00
jakman85 e1bebb857e Merge pull request 'Added my phone to peers' (#65) from vpn/add-keith-phone into main
gate / gate (push) Successful in 1m53s
gate / conformance (push) Has been skipped
Reviewed-on: #65
2026-10-06 07:42:04 +00:00
jakman85 23db97b958 Added my phone to peers
gate / gate (pull_request) Successful in 1m53s
gate / conformance (pull_request) Has been skipped
2026-10-06 02:39:23 -05:00
jakman85 5928d8153c Merge pull request 'fix(vpn-ingress): new-peer.sh counted its own commented-out example as an allocated address' (#64) from fix/new-peer-ignores-template-comments into main
gate / gate (push) Successful in 1m50s
gate / conformance (push) Has been skipped
Reviewed-on: #64
2026-10-06 07:31:03 +00:00
jakman85andClaude Opus 5 146dd9a906 fix(vpn-ingress): new-peer.sh counted its own commented-out example as an allocated address
gate / gate (pull_request) Successful in 1m51s
gate / conformance (pull_request) Has been skipped
The first real enrolment failed:

    $ ./infra/vpn-ingress/new-peer.sh keith-phone 2
    new-peer.sh: 10.90.0.2/32 is already allocated ... Pick another octet.

Nothing was allocated -- peers.yaml had zero peers, as the pod's init
container reported (`0 peer(s)`). The duplicate-address check grepped the
whole file, and peers.yaml line 66 is a COMMENTED-OUT example stanza I wrote
containing literally `#   AllowedIPs = 10.90.0.2/32`. So .2 -- the first
address anyone would ask for, and the one my own runbook uses as its example
-- was permanently un-allocatable, with a message asserting the opposite of
the truth.

Fixed on both sides, because either alone leaves the trap half-set:

  * the check strips comment lines before matching, which is the correct
    predicate rather than a patch: a real stanza's AllowedIPs line is never
    commented, while the template and every peer's `# <name> — <owner>` label
    always are.
  * the example now uses placeholders that cannot match anything real
    (`10.90.0.<octet>/32`, `<device-name>`), so a future grep cannot be fooled
    by it either.

The bug class is worth naming: a documentation example that is byte-identical
to real data will eventually be parsed as real data. This is its third
instance in this component -- flux-postbuild-substitution flagged a `${...}`
written inside a comment explaining that hazard, and no-plaintext-secrets
flagged the .sops.yaml.example because a kustomization named it in a comment.
Make examples structurally unmatchable, not merely commented out.

Verified by running it: .2 now renders a profile (via the real
`sops -d | wg pubkey` path, which this also proves against the operator's
actual key for the first time), and an uncommented duplicate stanza is still
correctly refused.

No sensitive paths; merges on a green gate.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-06 02:28:37 -05:00
jakman85 d1c07fe021 Merge pull request 'fix(vpn-ingress): stop trying to set net.ipv4.ip_forward — a hostNetwork pod cannot, and wg-quick dies on a failed PostUp' (#63) from fix/vpn-ingress-ip-forward into main
gate / gate (push) Successful in 1m51s
gate / conformance (push) Has been skipped
Reviewed-on: #63
2026-10-06 07:09:34 +00:00
jakman85andClaude Opus 5 5c43fbac65 fix(vpn-ingress): stop trying to set net.ipv4.ip_forward — a hostNetwork pod cannot, and wg-quick dies on a failed PostUp
gate / gate (pull_request) Successful in 1m51s
gate / conformance (pull_request) Has been skipped
The tunnel never came up on its first reconcile with a real key. One line of my
own manifest was the whole cause. Measured on prod 2026-10-06:

    [#] ip link add dev wg0 type wireguard        <- fine
    [#] wg addconf wg0 /dev/fd/63                 <- fine
    [#] ip -4 address add 10.90.0.1/24 dev wg0    <- fine
    [#] ip link set mtu 1420 up dev wg0           <- fine
    [#] sysctl -q -w net.ipv4.ip_forward=1
    sysctl: permission denied on key "net.ipv4.ip_forward"
    [#] ip link delete dev wg0
    **** Tunnel failed, will stop all others! ****

Not a missing capability: with hostNetwork the runtime mounts /proc/sys
READ-ONLY (confirmed in the pod's /proc/mounts), because those sysctls are the
HOST's globals. CAP_NET_ADMIN cannot lift a read-only mount, so the write can
never succeed here -- and wg-quick treats a failed PostUp as fatal, so it
deleted the interface it had just built.

The line was never needed. node_baseline already sets ip_forward=1, and reading
it from the pod returns 1. My header comment said exactly that ("so no pod
sysctl is needed") and then emitted the sysctl anyway, justified as "re-asserted
so this component does not silently depend on another role's sysctl file". That
defensive re-assertion is the entire outage. In a hostNetwork pod: VERIFY host
state, never write it.

Replaced with a read in the init container -- `grep -qx 1
/proc/sys/net/ipv4/ip_forward` -- failing closed with a message naming the
node_baseline role. Still fail-closed on purpose: a tunnel that comes up unable
to forward satisfies every probe here and silently drops all client traffic.

Also documents, in the manifest and the runbook, that the image's `chown:
Operation not permitted` run and its "we will not provide support for it"
warning are expected and harmless -- capabilities are dropped to NET_ADMIN so
its s6 init has no CAP_CHOWN, /config is a writable tmpfs, everything runs as
root, and wg-quick proceeds (it did; the failure was 30 lines later).

Three other unknowns closed by the same reconcile, now recorded in the README:
the kernel module reaches the pod (ip link add type wireguard works, no
CAP_SYS_MODULE needed); allowPrivilegeEscalation:false does NOT break the s6
init; and the image's iptables is nf_tables (xtables-nft-multi) matching Debian
13, which closes the worst documented fear -- rules landing in a table the
kernel never consults.

No sensitive paths; merges on a green gate with no override.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-06 01:30:08 -05:00
jakman85 cf7d5e4316 Merge pull request 'Changed secret' (#62) from feature/add-private-key-for-wireguard into main
gate / gate (push) Successful in 1m50s
gate / conformance (push) Has been skipped
Reviewed-on: #62
2026-10-06 06:18:13 +00:00
11 changed files with 503 additions and 58 deletions

No files matched your search

@@ -0,0 +1,73 @@
---
date: 2026-10-06
branch: fix/new-peer-ignores-template-comments
title: new-peer.sh refused the first VPN address because its duplicate check matched its own commented-out example
---
## What changed
The operator's first real enrolment failed:
```
$ ./infra/vpn-ingress/new-peer.sh keith-phone 2
new-peer.sh: 10.90.0.2/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet.
```
**Nothing was allocated.** `peers.yaml` had zero peers — the pod's own init container reported
`wg0.conf assembled: 0 peer(s)`. The duplicate-address check grepped the WHOLE file, and
`peers.yaml` line 66 is a **commented-out example stanza** I wrote, containing literally
`# AllowedIPs = 10.90.0.2/32`. So `.2` — the first address anybody would ask for, and the one my
own runbook uses in its example — was permanently un-allocatable, with a message asserting the
opposite of the truth.
Both halves fixed, because either alone leaves the trap half-set:
- **`infra/vpn-ingress/new-peer.sh`**: the check strips comment lines before matching
(`grep -vE '^[[:space:]]*#' | grep -q …`). That is the correct predicate rather than a patch: a
real stanza's `AllowedIPs` line is never commented, while both the template and every peer's
`# <name> — <owner>` label always are.
- **`platform/vpn-ingress/peers.yaml`**: the example now uses placeholders that cannot match
anything real — `# <device-name> — <owner>, enrolled <YYYY-MM-DD>` and
`AllowedIPs = 10.90.0.<octet>/32`. A future grep cannot be fooled by it either.
The bug class is worth naming: **a documentation example that is byte-identical to real data will
eventually be parsed as real data.** The same shape caused two earlier defects in this component —
`flux-postbuild-substitution` flagged a `${...}` written inside a comment *explaining* that hazard,
and `no-plaintext-secrets` flagged the `.sops.yaml.example` because the kustomization named it in a
comment. Three instances now; the rule is to make examples structurally unmatchable, not merely
commented out.
No sensitive paths (`platform/vpn-ingress/**`, `infra/vpn-ingress/**`, `decisions/**`), so this
merges on a green gate with no override.
## What I verified
Run in the pinned toolbox against the live tree, 2026-10-06.
- **`.2` now works**, and it went through the REAL key path: `new-peer.sh keith-phone 2` rendered a
profile with `Address 10.90.0.2/32`, `AllowedIPs = 10.90.0.2/32` in the printed stanza, full
tunnel, at `.cache/wg-peers/keith-phone.conf` (0600). The throwaway keypair was deleted after.
- **This incidentally closes a gap left open in #60**: the profile could only have been rendered by
deriving the server's public key via `sops -d --extract … | wg pubkey` from
`wg-server-key.sops.yaml` — so that pipeline is now proven **against the operator's real key**,
not just against a placeholder it was expected to reject.
- **A real duplicate is still refused.** Appending an uncommented
`AllowedIPs = 10.90.0.5/32` stanza and asking for octet 5 produced the correct refusal; the tree
was restored afterwards and `git diff --stat` showed only the intended template edit.
- **Zero active allocations remain**: `grep -vE '^[[:space:]]*#' peers.yaml | grep -c AllowedIPs`
→ `0`.
- **Toolbox gate**: rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`,
`l1` and `approval-policy` all pass; rc 3 is the two environmental skips (`gitea-settings`,
conformance L2/L3). **Not green. Reported as rc 3.**
## What I did NOT verify
- **No peer is enrolled by this branch.** The keypair I generated was a throwaway and was deleted;
the operator must run the script themselves, since the device's private key must exist only on
the device. So the end of the enrolment path — a stanza merged into `peers.yaml`, Flux rolling
the DaemonSet, and `wg show` listing a peer — is still unexercised.
- **The duplicate-NAME warning** (`grep -qi "^ *# *${NAME} "`) deliberately still inspects
comments, because per-peer labels are comments. It is no longer tripped by the template, but I
did not test it against a real enrolled peer.
- **Nothing about client traffic.** The WAN port-forward is still not configured and there is still
no off-LAN vantage point, so the tunnel has carried no packet.
@@ -0,0 +1,103 @@
---
date: 2026-10-06
branch: fix/peers-configmap-content-hash
title: A merged VPN peer never reached the tunnel — the peer register was a fixed-name ConfigMap, so nothing rolled the pods
---
## What changed
The operator enrolled a phone, merged the peer, and **`wg show` listed no peers.** Measured on prod
2026-10-06, with the peer already on `main`:
| observation | value |
|---|---|
| peer stanza on `main` | present (`AllowedIPs = 10.90.0.2/32`) |
| GitRepository artifact | `main@sha1:e1bebb8` — Flux had fetched it |
| `vpn-ingress` Kustomization `lastAppliedRevision` | `main@sha1:5928d81` — **behind**, 10m interval |
| live `wg-peers` ConfigMap, active `AllowedIPs` lines | **0** |
| DaemonSet volume reference | `configMap.name: wg-peers` — a **fixed** name |
| DaemonSet `generation` / `observedGeneration` | **1 / 1** — it had **never rolled** |
| init container, all three pods | `wg0.conf assembled: 0 peer(s)` |
The lag was only the reconcile interval. **The defect is the fixed name.** When Flux does update a
fixed-name ConfigMap it updates it *in place*: the DaemonSet's pod template is byte-identical, so
no rollout is triggered — and because the init container assembles `wg0.conf` **once at pod start**
into a tmpfs, the running tunnel keeps the old peer list indefinitely. Adding or revoking a peer
would have had **no effect at all** until something unrelated happened to restart the pods. For a
*revocation* that is a security defect, not an inconvenience: the operator would delete a stanza,
see a green PR, and the device would keep working.
My runbook and README both asserted "PR → merge → Flux rolls the DaemonSet (~1 min)". That was
never true.
**This repo already documents the fix, and I had read the file.** `runbooks/kustomization.yaml`:
*"The generated name carries kustomize's content hash, so editing any runbook rolls the server
automatically and a stale projection is impossible."*
So the register becomes a **content-hashed generator**:
- `platform/vpn-ingress/peers.yaml` (inline ConfigMap) → **`platform/vpn-ingress/peers.conf`**, a
plain file, plus a `configMapGenerator` in `kustomization.yaml`. The operator's enrolled peer was
carried across verbatim. Every line is WireGuard config and `#` is a legal comment there, so the
documentation now travels *inside the mounted file* instead of in a YAML wrapper that gets
stripped.
- **`namespace: vpn-ingress` added to the kustomization — REQUIRED, not tidiness.** A generated
ConfigMap inherits no namespace; every other object here declares `vpn-ingress`; and kustomize's
`nameReference` transformer will not rewrite a reference across a namespace boundary. Without it
the build emitted `ConfigMap wg-peers-m9kcckcb5t` while the DaemonSet still mounted the unhashed
`wg-peers`, which would not exist — the pods would wedge on a missing volume. I caught this only
by building the output and reading the volumes block.
- **`infra/vpn-ingress/new-peer.sh`**: `PEERS_FILE` repointed to `peers.conf`, and a missing
register is now **fatal** instead of skipped. The old guard was
`[ -r "$PEERS_FILE" ] && grep …`, so after the rename the only symptom would have been
duplicate-address detection quietly never firing again — the silent-skip shape this repo treats
as a defect in its own right.
- Stale `peers.yaml` references swept from the README, runbook, `daemonset.yaml`,
`networkpolicy.yaml`, the SOPS example, and the script's own **user-facing output** (it told the
operator to paste into "peers.yaml, under data.peers.conf", which would no longer exist).
- The runbook gained the verification that actually distinguishes this failure — the DaemonSet's
`generation` must *increase* after the merge — and a troubleshooting row for `0 peer(s)` after a
merge.
Flux's `prune: true` removes the orphaned fixed-name `wg-peers` ConfigMap on the next reconcile.
No sensitive paths (`platform/vpn-ingress/**`, `infra/vpn-ingress/**`, `runbooks/**`,
`decisions/**`), so this merges on a green gate with no override.
## What I verified
Built and run in the pinned toolbox, 2026-10-06 (kustomize v5.8.1 via `kubectl kustomize`).
- **The reference is now rewritten end to end**: generated `ConfigMap wg-peers-m9kcckcb5t` in
namespace `vpn-ingress`, and the DaemonSet mounts **`wg-peers-m9kcckcb5t`**. Before adding
`namespace:` the same build mounted the unhashed `wg-peers` — so the namespace line is load
bearing, measured both ways.
- **The hash tracks content, which is the whole mechanism.** `wg-peers-m9kcckcb5t` →
append a second peer → `wg-peers-dgg7ht9c74` → restore → `wg-peers-m9kcckcb5t`. Deterministic
and reversible, so an edit necessarily changes the pod template.
- **The operator's enrolled peer survived the file conversion byte-for-byte**: the active stanza in
`peers.conf` (`PublicKey = pBRSHrRy…`, `AllowedIPs = 10.90.0.2/32`) is identical to the one on
`main`, compared directly against `git show origin/main:platform/vpn-ingress/peers.yaml`.
- **The build still emits all five objects**: Namespace, ConfigMap, Secret, DaemonSet,
NetworkPolicy — and the `Namespace` object's own name is still `vpn-ingress` (the namespace
transformer did not rewrite it).
- **`new-peer.sh` works against the new register**: `spare-laptop 7` renders a profile and prints
"append this to platform/vpn-ingress/peers.conf"; `dupe 2` is **refused** because the operator's
real `.2` now occupies it — the first time the duplicate check has been exercised against real
data rather than against its own example.
- **Toolbox gate**: rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`,
`l1` and `approval-policy` all pass; rc 3 is the two environmental skips. **Not green. Reported
as rc 3.**
## What I did NOT verify
- **That the roll actually happens on the cluster.** The whole point of the change is unproven
until Flux applies it: the success criterion is the DaemonSet's `generation` increasing and
`wg0.conf assembled: 1 peer(s)` in the init log, followed by `wg show` listing the peer.
- **That the tunnel then carries traffic.** No handshake has ever been observed. The WAN
port-forward is reportedly configured but has never been exercised by a packet I can see, and I
have no off-LAN vantage point — runbook step 6 from cellular remains the only honest test.
- **Revocation**, which is the case where this defect was a security problem. Deleting a stanza
should now roll the pods and drop the peer; untested.
- **Whether the orphaned fixed-name `wg-peers` ConfigMap is actually pruned.** Reasoned from
`prune: true` on the Kustomization, not observed.
@@ -0,0 +1,113 @@
---
date: 2026-10-06
branch: fix/vpn-ingress-ip-forward
title: The VPN tunnel never came up — a hostNetwork pod cannot write net.ipv4.ip_forward, and wg-quick treats a failed PostUp as fatal
---
## What changed
The remote-access VPN reconciled for the first time with a real server key (#60 then #62) and
**the tunnel did not come up**. One line in my own manifest was the whole cause.
Measured in the `wireguard` container on prod, 2026-10-06:
```
[#] ip link add dev wg0 type wireguard
[#] wg addconf wg0 /dev/fd/63
[#] ip -4 address add 10.90.0.1/24 dev wg0
[#] ip link set mtu 1420 up dev wg0
[#] sysctl -q -w net.ipv4.ip_forward=1
sysctl: permission denied on key "net.ipv4.ip_forward"
[#] ip link delete dev wg0
**** Tunnel /config/wg_confs/wg0.conf failed, will stop all others! ****
```
Everything up to that point worked. **The cause is not a missing capability.** With
`hostNetwork: true` the runtime mounts `/proc/sys` **read-only** in the container — confirmed in the
pod's own `/proc/mounts`: `proc /proc/sys proc ro,nosuid,nodev,noexec,relatime` — precisely because
those sysctls are the *host's* globals and a container must not rewrite them. `CAP_NET_ADMIN` cannot
lift a read-only mount, so this can never succeed from here. And `wg-quick` treats any failed
`PostUp` as fatal, so it deleted the interface it had just finished building.
**The line was never needed.** `node_baseline` writes `net.ipv4.ip_forward=1` to
`/etc/sysctl.d/99-k3s-cilium.conf`, and reading the value from inside the pod returns `1`. I had
written the header comment *correctly* — "ip_forward is ALREADY 1 on these nodes … so no pod sysctl
is needed" — and then emitted the sysctl anyway, justified in its own comment as "re-asserted so
this component does not silently depend on another role's sysctl file surviving". That defensive
re-assertion is the entire outage. **The lesson is narrow and reusable: in a hostNetwork pod,
verify host state, never write it.**
Fix:
- **`platform/vpn-ingress/daemonset.yaml`**: the `PostUp = sysctl …` line is gone, replaced by a
**read** in the init container — `grep -qx 1 /proc/sys/net/ipv4/ip_forward` — which fails closed
with a message naming the `node_baseline` role. Reading `/proc/sys` is permitted; only writing is
not. Fail-closed still matters: a tunnel that comes up but cannot forward would satisfy every
probe in this component and silently drop every client's traffic.
- The header bullet that said "no pod sysctl is needed" now also says **do not attempt one**, with
the mechanism, so the next author does not re-add it for the same reason I did.
- The `securityContext` comment claimed NET_ADMIN was for "create the wg0 link, set the host sysctl,
and write the netfilter rules". The middle clause was false; corrected.
- The image's `chown: … Operation not permitted` startup run — plus its own "Permissions could not
be set … we will not provide support for it" — is now documented as **expected and harmless**, in
both the manifest and the runbook, so nobody chases it. Capabilities are dropped to `NET_ADMIN`
only, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs as root,
and `wg-quick` proceeds past it (as it did here — the failure was 30 lines later).
- **`runbooks/vpn-remote-access.md`**: a "what a healthy start looks like" block with the real log
lines; `ip_forward is 0` added to the init-failure table; and the note that `0/1 Running` with
`Startup probe failed: Unable to access interface: No such device` is the signature of `wg-quick`
tearing the tunnel down — read the `[#]` line it died on.
- **`platform/vpn-ingress/README.md`**: a table of what the first reconcile settled (below).
No sensitive paths: `platform/vpn-ingress/**`, `runbooks/**` and `decisions/**` only. This merges on
a green gate with no override.
## What I verified
Live prod, read-only, 2026-10-06. **Four questions this component's design rested on are now
measured rather than assumed — three went the right way, one was the defect above.**
- **The GitOps path is proven end to end.** `Kustomization vpn-ingress` `lastAppliedRevision:
main@sha1:cf7d5e4` equals the GitRepository artifact revision. The Secret carries
`kustomize.toolkit.fluxcd.io/name: vpn-ingress`, so Flux delivered it.
- **SOPS decryption works, proven by digest rather than by eye.** The value in
`platform/vpn-ingress/wg-server-key.sops.yaml` on `main`, decrypted, is 44 bytes, is **not** the
placeholder, and **`wg pubkey` accepts it**. Its SHA-256 equals the SHA-256 of the live Secret's
decoded `data.privatekey` (`d7c8f8de6f252649…` both sides). No key material was printed at any
point — lengths, booleans and digests only.
- **The `wg pubkey` guard fired correctly, and its `ENC[` branch was never needed.** During the
kubelet window before the updated Secret propagated, all three pods reported *"It is still the
committed REPLACE-ME PLACEHOLDER"* — the guard distinguishing a stale/unfilled value from
ciphertext, exactly as designed. It cleared on its own once the mount refreshed; **a transient
`REPLACE-ME` CrashLoop for a minute or two after the key PR merges is expected, not a fault.**
- **The kernel module reaches the pod**: `ip link add dev wg0 type wireguard` succeeded and
`/proc/modules` lists `wireguard`. So the `node_baseline` addition works and `CAP_SYS_MODULE` is
correctly not required.
- **`allowPrivilegeEscalation: false` does NOT break the image's s6 init** — the container reaches
`wg-quick` normally. The flag stays; the fear recorded in #55 is closed.
- **The iptables backends MATCH, which was my single biggest documented doubt.** The image ships
`iptables v1.8.13 (nf_tables)` via `/usr/sbin/xtables-nft-multi`, and Debian 13 is nf_tables. The
nightmare case — rules accepted into a table the kernel never consults, so handshakes succeed and
every forwarded connection hangs — cannot occur on this pairing.
- **`net.ipv4.ip_forward` is `1`** in the host netns the pod shares, and `/proc/sys` is `ro` there.
Both read directly from the pod.
- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
`lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two
environmental skips (`gitea-settings`, conformance L2/L3). **Not green. Reported as rc 3.**
## What I did NOT verify
- **That the tunnel now comes up.** This fix has not reconciled. The next reconcile is the test, and
its success criterion is specific: pods `1/1 Running`, and `wg show` reporting `interface: wg0`
with `listening port: 51820`.
- **That the MASQUERADE and FORWARD rules actually apply.** They are the next statements after the
one that failed, so **they have never executed**. The backends matching (above) removes the worst
failure mode but proves nothing about the rules themselves. First check after the pods go Ready:
`iptables -t nat -S POSTROUTING | grep 10.90.0.0/24` on a node.
- **Anything about client traffic.** Still zero peers enrolled, still no WAN port-forward, still no
off-LAN vantage point. The VPN has carried no packet.
- **Whether the transient `REPLACE-ME` window has an upper bound.** It cleared within ~2 minutes
here; I did not determine the kubelet refresh behaviour, so a longer stall after a key change is
uncharacterised.
- **The `sops set` hand-off command**, still. The operator filled the key in by their own route
(#62), so the exact quoting I documented inside `dev.sh run bash -c` remains unexercised.
+19 -8
View File
@@ -3,7 +3,7 @@
# new-peer.sh — enrol one device on the remote-access VPN (platform/vpn-ingress).
#
# Generates a WireGuard keypair, renders the device's client profile, prints a QR code for a phone,
# and prints the one `[Peer]` stanza to paste into platform/vpn-ingress/peers.yaml. The device's
# and prints the one `[Peer]` stanza to paste into platform/vpn-ingress/peers.conf. The device's
# private half is written to ONE 0600 file and nowhere else: not sent to the cluster, not committed,
# never logged.
#
@@ -62,7 +62,7 @@ set -euo pipefail
die() { printf 'new-peer.sh: %s\n' "$*" >&2; exit 1; }
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.yaml"
PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.conf"
# --- defaults, each mirroring a value that is DECLARED elsewhere in this repo -------------------
# These are duplicated here rather than parsed out of the YAML on purpose: this script is meant to
@@ -133,11 +133,22 @@ CLIENT_ADDR="${CLIENT_BASE}.${OCTET}"
# --- refuse a collision, because the failure is otherwise invisible -----------------------------
# Two devices sharing an address does not error anywhere: the server routes that address to
# whichever peer handshook most recently, so BOTH devices appear to work intermittently.
if [ -r "$PEERS_FILE" ] && grep -q "AllowedIPs *= *${CLIENT_ADDR}/32" "$PEERS_FILE"; then
die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet."
# COMMENT LINES ARE NOT ALLOCATIONS. This used to grep the whole file and so matched the
# COMMENTED-OUT EXAMPLE STANZA in the register, which made `.2` — the first address anyone would
# ask for — permanently un-allocatable with a message claiming it was already taken. Strip
# comments first: a real stanza's `AllowedIPs` line is never commented, while both the template
# and every peer's `# <name> — <owner>` label are.
# FAIL if the register is missing rather than skipping the check. The path was silently wrong once
# (it still said peers.yaml after the file became peers.conf), and because the old guard was
# `[ -r "$PEERS_FILE" ] && ...` the only symptom would have been duplicate detection quietly
# never firing again — the exact silent-skip shape this repo treats as a defect.
[ -r "$PEERS_FILE" ] || die "cannot read the peer register '$PEERS_FILE'.
Duplicate-address detection cannot run, so this script refuses to issue a profile."
if grep -vE '^[[:space:]]*#' "$PEERS_FILE" | grep -q "AllowedIPs *= *${CLIENT_ADDR}/32"; then
die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.conf. Pick another octet."
fi
if [ -r "$PEERS_FILE" ] && grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.yaml.\n' "$NAME" >&2
if grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.conf.\n' "$NAME" >&2
fi
# --- output location: 0700, and never inside the repo -------------------------------------------
@@ -204,7 +215,7 @@ cat <<EOF
Profile : ${PROFILE} (0600; contains the private key)
------------------------------------------------------------------------------------------
STEP 1 — add this to platform/vpn-ingress/peers.yaml, under data.peers.conf, then PR it:
STEP 1 — append this to platform/vpn-ingress/peers.conf, then PR it:
# ${NAME} — <owner>, enrolled $(date -u +%Y-%m-%d)
[Peer]
@@ -223,7 +234,7 @@ cat <<EOF
STEP 3 — delete the profile from this machine once the device has it:
shred -u ${PROFILE} 2>/dev/null || rm -f ${PROFILE}
The tunnel will not work until the peers.yaml PR is MERGED and Flux has rolled the DaemonSet
The tunnel will not work until the peers.conf PR is MERGED and Flux has rolled the DaemonSet
(~1 min). Until then the device handshakes and gets silence, exactly like an unknown key.
------------------------------------------------------------------------------------------
EOF
+18 -5
View File
@@ -13,7 +13,7 @@ egress out the house WAN.
| Transport | WireGuard, **udp/51820**, kernel implementation |
| Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node |
| Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices |
| Authorisation | a device's **public key** is in [`peers.yaml`](./peers.yaml), or it cannot connect |
| Authorisation | a device's **public key** is in [`peers.conf`](./peers.conf), or it cannot connect |
| The one secret | the server's private key — **SOPS-encrypted in this tree**, `wg-server-key.sops.yaml` |
| Exposure | one hand-made WAN port-forward on the SOHO router, off-repo |
| Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) |
@@ -129,7 +129,7 @@ translations win here.
| half | where | why |
|---|---|---|
| peer **public** keys + addresses | [`peers.yaml`](./peers.yaml), **plaintext in git** | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
| peer **public** keys + addresses | [`peers.conf`](./peers.conf), **plaintext in git**, as a content-hashed `configMapGenerator` so an edit rolls the pods | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
| the server's **private** key | `wg-server-key.sops.yaml`, **SOPS-encrypted in git** | the only real secret (R-SEC-01, spec docs/07). Ships with a `REPLACE-ME` placeholder: an agent must not mint a long-lived private key (CLAUDE.md §6), so the operator `sops set`s it in |
The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** —
@@ -166,13 +166,26 @@ NetworkPolicy applies.
## Operating it
* **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
stanza into `peers.yaml` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
* **Revoke a device:** delete its stanza → PR → merge. No other step; the key stops working.
stanza into `peers.conf` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
* **Revoke a device:** delete its stanza from `peers.conf` → PR → merge. No other step; the key
stops working as soon as the pods roll (the generator's content hash guarantees they do).
* **Is it up?** `kubectl --context k3s-prod -n vpn-ingress get ds wireguard` — three Ready.
* **Who is connected?** `kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show`
(`latest handshake` under ~3 min means live). Never dump `wg showconf` — it prints the private key.
* **Full bring-up and verification:** `runbooks/vpn-remote-access.md`.
## Measured on prod, 2026-10-06 — what is now known rather than assumed
The first real reconcile settled four things this component's design rested on. Three went the
right way; one was a defect in my own manifest.
| question | answer |
|---|---|
| Does the **kernel module** reach the pod? | **Yes.** `ip link add dev wg0 type wireguard` succeeds, and `/proc/modules` lists `wireguard`, after the `node_baseline` run. `CAP_SYS_MODULE` is correctly not needed. |
| Does `allowPrivilegeEscalation: false` break the image's **s6 init**? | **No.** The container reaches `wg-quick` normally. The flag stays. |
| Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
| Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
## Prerequisites outside this directory
1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
@@ -199,7 +212,7 @@ NetworkPolicy applies.
covered generically by `KubeDeploymentReplicasMismatch`/`KubePodNotReady` from
kube-prometheus-stack. **This is the top follow-up.**
* **No preshared keys.** They would add a symmetric post-quantum hedge on top of the Noise
handshake — and would be a real secret, forcing `peers.yaml` into SOPS and costing the
handshake — and would be a real secret, forcing `peers.conf` into SOPS and costing the
reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
`wg-server-key` Secret.
* **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6
+46 -6
View File
@@ -37,6 +37,10 @@
# /etc/sysctl.d/99-k3s-cilium.conf), so no pod sysctl is needed — and `net.ipv4.ip_forward`
# is NOT in the kubelet's safe-sysctl allowlist, so a pod-netns version of this would have
# required `--allowed-unsafe-sysctls` on all three kubelets (a node-restart change).
# COROLLARY LEARNED THE HARD WAY (prod, 2026-10-06): this pod must not merely avoid NEEDING
# the sysctl, it must not ATTEMPT it. hostNetwork implies /proc/sys is mounted read-only, so
# `sysctl -w` fails with EPERM regardless of capabilities, and wg-quick treats a failed PostUp
# as fatal and deletes the interface. Verify the value; never write it.
# * no LoadBalancer VIP, so no CiliumL2AnnouncementPolicy lease lottery (the defect that cost
# this LAN an 8-day Minecraft outage) and no `externalTrafficPolicy` trade-off at all.
# * no double encapsulation. WireGuard's UDP lands on the node's own NIC, so the underlay is a
@@ -104,7 +108,7 @@ spec:
dnsPolicy: ClusterFirstWithHostNet
automountServiceAccountToken: false
# ── Assemble the running config: PUBLIC peer register (git) + PRIVATE server key (Secret) ──
# The two halves are deliberately stored apart (peers.yaml explains why) and are only ever
# The two halves are deliberately stored apart (peers.conf explains why) and are only ever
# joined here, at pod start, into a tmpfs that is destroyed with the pod. The server's private
# key is therefore never written to any node's disk, never committed, and never logged.
initContainers:
@@ -171,6 +175,19 @@ spec:
exit 1
fi
# The pod shares the HOST netns, so this reads the node's real setting. Reading
# /proc/sys is permitted even though writing is not (see the PostUp note below).
# Fail closed: a tunnel that comes up but cannot forward would look healthy to every
# probe here and simply drop every client's traffic.
if ! grep -qx 1 /proc/sys/net/ipv4/ip_forward; then
echo "FATAL: net.ipv4.ip_forward is 0 on this node, so forwarded VPN traffic would" >&2
echo " be dropped. This pod CANNOT set it: with hostNetwork the runtime mounts" >&2
echo " /proc/sys read-only, because it is the host's global setting." >&2
echo " Fix on the NODE: run the node_baseline role (it writes" >&2
echo " /etc/sysctl.d/99-k3s-cilium.conf) — runbooks/vpn-remote-access.md step 1." >&2
exit 1
fi
{
echo '[Interface]'
echo "Address = $base.1/$mask"
@@ -184,9 +201,20 @@ spec:
echo 'MTU = 1420'
printf 'PrivateKey = %s\n' "$key"
# ── Forwarding + SNAT, applied in the HOST netns when wg0 comes up ──
# ip_forward is already 1 from node_baseline; re-asserted so this component does not
# silently depend on another role's sysctl file surviving.
echo 'PostUp = sysctl -q -w net.ipv4.ip_forward=1'
# NO `sysctl -w net.ipv4.ip_forward=1` HERE. It is IMPOSSIBLE from this pod and it was
# the only thing stopping the tunnel coming up — MEASURED on prod 2026-10-06:
# [#] sysctl -q -w net.ipv4.ip_forward=1
# sysctl: permission denied on key "net.ipv4.ip_forward"
# [#] ip link delete dev wg0
# and wg-quick then tore the whole tunnel down. The cause is not a missing
# capability: with `hostNetwork: true` the runtime mounts /proc/sys READ-ONLY in the
# container (confirmed in /proc/mounts: `proc /proc/sys proc ro,...`), precisely
# because those sysctls are the HOST's globals and a container must not rewrite
# them. CAP_NET_ADMIN does not and cannot lift a read-only mount.
# It was never needed anyway: node_baseline writes net.ipv4.ip_forward=1 to
# /etc/sysctl.d/99-k3s-cilium.conf, and reading it from this pod returns 1. The
# init container now VERIFIES that (a read, which /proc/sys does allow) and fails
# closed with a pointed message, which is what the write was pretending to achieve.
# SNAT client traffic to the node's address. REQUIRED: without it a LAN host would
# reply to 10.90.0.x, find no route, hand the packet to the router, and the router
# would drop it — so every connection would hang rather than fail. Matching on
@@ -246,8 +274,10 @@ spec:
# `kubectl get pod -o wide` shows what this listener owns.
- { name: wireguard, containerPort: 51820, hostPort: 51820, protocol: UDP }
securityContext:
# NET_ADMIN is the whole privilege budget: create the wg0 link, set the host sysctl, and
# write the two netfilter rules above. Everything else is dropped, and the container is
# NET_ADMIN is the whole privilege budget: create the wg0 link and write the netfilter
# rules above. (It does NOT let the pod set a host sysctl — /proc/sys is read-only under
# hostNetwork — which is why nothing here tries.) Everything else is dropped, and the
# container is
# NOT privileged — which also keeps it clear of restrict-host-namespaces' second rule.
# SYS_MODULE is deliberately NOT granted: the `wireguard` module is loaded by the node
# (infra/ansible/roles/node_baseline, node_baseline_kernel_modules), so this container
@@ -261,6 +291,16 @@ spec:
- NET_ADMIN
# readOnlyRootFilesystem intentionally UNSET: the image's s6 init writes under /run and
# /var during startup. Same deferral as crafty/headlamp.
#
# EXPECTED, HARMLESS STARTUP NOISE — do not "fix" it. Because capabilities are dropped
# to NET_ADMIN only, the image's s6 init cannot chown /config (no CAP_CHOWN) and logs a
# run of `chown: ... Operation not permitted` plus "Permissions could not be set. This
# is probably because your volume mounts are remote or read-only" and a scary "we will
# not provide support for it". It is wrong about the cause and it does not matter:
# /config is a writable tmpfs emptyDir, everything here runs as root (uid 0) anyway, and
# wg-quick proceeds normally — verified on prod 2026-10-06, where the tunnel's only
# failure was the sysctl above, well past these warnings. Granting CAP_CHOWN to silence
# log noise would widen the privilege set of an internet-facing pod for nothing.
resources:
requests:
cpu: 25m
+31 -2
View File
@@ -11,10 +11,16 @@
# NOT in platform/kustomization.yaml — see clusters/base/vpn-ingress.yaml.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# REQUIRED for the configMapGenerator below to be wired up at all, not merely tidy. A generated
# ConfigMap inherits NO namespace, every other object here declares `vpn-ingress`, and kustomize's
# nameReference transformer will not rewrite a reference across a namespace boundary — so without
# this the DaemonSet keeps mounting the unhashed name `wg-peers`, which does not exist, and the
# pods wedge on a missing volume. Verified by building it both ways (kustomize v5.8.1).
# Same placement as runbooks/kustomization.yaml, which uses `namespace: runbooks` for its own
# content generator.
namespace: vpn-ingress
resources:
- namespace.yaml
# The peer register: which devices may dial in. PLAINTEXT and reviewable on purpose (see the file).
- peers.yaml
# The listener itself: hostNetwork DaemonSet, udp/51820 on every node.
- daemonset.yaml
# Namespace default-deny baseline. NOTE: does NOT constrain the hostNetwork DaemonSet — the file
@@ -38,3 +44,26 @@ resources:
# hands Flux the raw ciphertext, the Secret APPLIES CLEANLY full of ENC[...], and nothing in any
# Flux condition looks wrong. The init container's `wg pubkey` guard exists partly to turn exactly
# that into a named error rather than a silently non-working tunnel.
# ── THE PEER REGISTER, as a CONTENT-HASHED ConfigMap ─────────────────────────────────────────
# peers.conf (plaintext, reviewable — see the file) becomes the `wg-peers` ConfigMap. It is a
# GENERATOR, not a plain resource, and that distinction is load-bearing rather than stylistic:
# kustomize appends a hash of the content to the generated name and rewrites the DaemonSet's
# volume reference to match, so adding or revoking a peer changes the pod template and ROLLS THE
# PODS automatically.
#
# A fixed-name ConfigMap (which is what this component shipped first) does not do that. Flux
# updates it in place, the pod template is byte-identical, no rollout is triggered — and because
# the init container assembles wg0.conf ONCE at pod start into a tmpfs, the running tunnel keeps
# the old peer list indefinitely. Measured on prod 2026-10-06: a merged peer reached the
# ConfigMap while `wg show` listed no peers and the DaemonSet still read generation=1, having
# never rolled. The runbook's "PR -> merge -> Flux rolls the DaemonSet" was false until this.
#
# DO NOT set `disableNameSuffixHash` here. The hash IS the mechanism.
configMapGenerator:
- name: wg-peers
files:
- peers.conf
options:
labels:
app.kubernetes.io/name: vpn-ingress
+1 -1
View File
@@ -19,7 +19,7 @@
# protects this port, and it would be enforcing nothing. So there isn't one.
#
# WHAT ACTUALLY PROTECTS THE EXPOSED PORT, then, since the CNI does not:
# 1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.yaml gets
# 1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.conf gets
# NO REPLY — see README "Why this is safe to expose". This is the real control.
# 2. The SOHO router's forward: only udp/51820 is forwarded inbound, to one node.
# 3. The blast radius once authenticated is deliberately the whole LAN — that IS the requirement
@@ -1,4 +1,16 @@
---
# THIS FILE IS THE MOUNTED CONTENT. It is turned into the `wg-peers` ConfigMap by
# `configMapGenerator` in kustomization.yaml, whose generated name carries a CONTENT HASH — so
# editing this file changes the ConfigMap name, which changes the DaemonSet's pod template, which
# ROLLS THE PODS. That is the whole reason it is a generator and not an inline ConfigMap manifest:
# a fixed-name ConfigMap updates in place, the pod template never changes, no rollout happens, and
# the init container's wg0.conf (built once at pod start, in tmpfs) keeps the OLD peer list
# forever. Measured 2026-10-06: a merged peer sat in the ConfigMap while `wg show` listed none and
# the DaemonSet reported generation=1 — it had never rolled. Same mechanism and same fix as
# runbooks/kustomization.yaml.
#
# Every line here is WireGuard config. `#` is a legal comment in it, so the documentation below
# travels inside the file that is actually mounted, rather than in a wrapper that gets stripped.
#
# ============================================================================================
# THE PEER REGISTER — every device allowed to dial in, in PLAINTEXT, in git, on purpose.
# ============================================================================================
@@ -41,33 +53,40 @@
# the Noise handshake and would be a genuine secret — which would force this register into SOPS
# and cost the reviewability above. Revisit only if the threat model changes; if it does, the PSK
# belongs in the `wg-server-key` Secret, not here. See README "Hardening not taken".
apiVersion: v1
kind: ConfigMap
metadata:
name: wg-peers
namespace: vpn-ingress
labels:
app.kubernetes.io/name: vpn-ingress
data:
# Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
#
# *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
# An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
# authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
# DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
# The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
# (`wg show` reports an interface with no peers).
#
# COPY THIS TEMPLATE, uncomment, one block per device:
#
# # pixel-8 — Keith's phone, enrolled 2026-10-05
# [Peer]
# PublicKey = <the public key new-peer.sh printed>
# AllowedIPs = 10.90.0.2/32
#
# Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
# tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
# it an incident responder cannot tell which stanza to delete.
peers.conf: |
# No peers enrolled yet. See the template in this file's comments and
# ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
#
# name: wg-peers
# namespace: vpn-ingress
# labels:
# app.kubernetes.io/name: vpn-ingress
# Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
#
# *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
# An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
# authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
# DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
# The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
# (`wg show` reports an interface with no peers).
#
# COPY THIS TEMPLATE, uncomment, one block per device:
#
# # <device-name> — <owner>, enrolled <YYYY-MM-DD>
# [Peer]
# PublicKey = <the public key new-peer.sh printed>
# AllowedIPs = 10.90.0.<octet>/32
#
# The placeholders above are deliberately NOT valid values. An earlier version of this template
# used a real-looking `10.90.0.2/32`, and new-peer.sh's duplicate-address check — which grepped
# the whole file — matched it, so `.2` could never be allocated and the script insisted it was
# already taken. The check now ignores comments; keeping the example unmatchable too means a
# future grep cannot be fooled either.
#
# Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
# tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
# it an incident responder cannot tell which stanza to delete.
#
# No peers enrolled yet. See the template in this file's comments and
# ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
# keith-phone — <owner>, enrolled 2026-10-06
[Peer]
PublicKey = pBRSHrRyG2hnsuZ6FvHfEzF8hOVTF5stHUb/7fW2Km8=
AllowedIPs = 10.90.0.2/32
@@ -5,7 +5,7 @@
# WHAT THIS IS: the WireGuard SERVER's long-term private key for the remote-access VPN
# (platform/vpn-ingress). It is the ONE secret in the whole component. The init container in
# daemonset.yaml reads it at `/run/wg-server-key/privatekey` and joins it to the PLAINTEXT peer
# register (peers.yaml) in a tmpfs, so the assembled wg0.conf never touches a node's disk.
# register (peers.conf) in a tmpfs, so the assembled wg0.conf never touches a node's disk.
#
# WHY IT IS IN GIT, SOPS-ENCRYPTED, AND NOT HAND-CREATED. R-SEC-01 / spec docs/07: secrets live in
# source, encrypted, and reach the cluster only through Flux. An out-of-band `kubectl create secret`
+48 -4
View File
@@ -21,7 +21,7 @@ behaves like a machine on the LAN: full reach to `192.168.2.0/24`, the Gateway's
DNS, and internet egress out the house WAN.
Authorisation is one thing only: the device's **public key is in
`platform/vpn-ingress/peers.yaml`**. An unlisted key gets no reply at all.
`platform/vpn-ingress/peers.conf`**. An unlisted key gets no reply at all.
> **An enrolled device is as privileged as a machine sitting in the house** — including the three
> k3s nodes (`192.168.2.10/.11/.12`: apiserver 6443, kubelet 10250, SSH 22). That is what "reach
@@ -134,6 +134,37 @@ specific failure in its log — so you never get a server that silently cannot c
| `It is EMPTY` | the value was blanked |
| `still SOPS CIPHERTEXT (starts with ENC[)` | the Secret reached the cluster undecrypted — this environment's overlay is missing its `vpn-ingress` decryption patch. **This is the dangerous one:** the Secret applies cleanly and every Flux condition stays green |
| `wg pubkey rejected it` with none of the above | a mangled value, or a *public* key pasted by mistake |
| `net.ipv4.ip_forward is 0 on this node` | the node_baseline role has not run here (step 1). The pod **cannot** set this itself — under `hostNetwork` the runtime mounts `/proc/sys` read-only, because it is the host's global setting |
| `0 peer(s)` after you merged a peer | the DaemonSet did not roll. Check its `generation` (above). Pre-2026-10-06 this was guaranteed, because the register was a fixed-name ConfigMap; if it recurs, something has reverted `peers.conf` to a plain resource instead of a `configMapGenerator` |
### What a healthy start looks like
Measured on prod 2026-10-06. The init container logs one line, naming the shape and never the key:
```
wg0.conf assembled: 0 peer(s), server 10.90.0.1/24, udp/51820.
```
then the `wireguard` container brings the tunnel up:
```
**** Found WG conf /config/wg_confs/wg0.conf, adding to list ****
**** Activating tunnel /config/wg_confs/wg0.conf ****
[#] ip link add dev wg0 type wireguard
[#] wg addconf wg0 /dev/fd/63
[#] ip -4 address add 10.90.0.1/24 dev wg0
[#] ip link set mtu 1420 up dev wg0
```
**Ignore the `chown: ... Operation not permitted` run before that**, and the image's own
"Permissions could not be set ... we will not provide support for it". Capabilities are dropped to
`NET_ADMIN` only, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs
as root, and `wg-quick` proceeds. It is wrong about the cause and it does not matter.
**A pod stuck `0/1 Running` with `Startup probe failed: Unable to access interface: No such device`
means `wg-quick` tore the tunnel down** — read the `wireguard` container log for the `[#]` line it
died on. That is the shape of a real failure; the probe is doing its job of refusing to call a
server without a `wg0` "ready".
## Step 3 — forward udp/51820 on the router (once)
@@ -188,10 +219,23 @@ It generates the device's keypair, writes a `0600` client profile, prints a QR c
Then:
1. Paste the stanza into `platform/vpn-ingress/peers.yaml` under `data.peers.conf`, keeping the
1. Append the stanza to `platform/vpn-ingress/peers.conf`, keeping the
`# <name> — <owner>, enrolled <date>` comment. `wg show` can only ever print the key, so that
comment is the only record of which stanza belongs to which device.
2. PR → merge. Flux rolls the DaemonSet (~1 min).
2. PR → merge. Flux rolls the DaemonSet (~1 min), because `peers.conf` is a
**content-hashed** `configMapGenerator`: editing it changes the ConfigMap's generated name,
which changes the pod template, which forces the roll. (It did **not** work this way at first —
a fixed-name ConfigMap updates in place, the pod template never changes, and the init container
keeps the old peer list forever. Measured on prod 2026-10-06.) Confirm it actually rolled:
```bash
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n vpn-ingress get ds wireguard \
-o jsonpath="{.metadata.generation}{\"/\"}{.status.observedGeneration}{\"\n\"}"'
```
The generation must have **increased** since before the merge. If it is still `1`, the roll did
not happen and `wg show` will list no peer no matter how long you wait.
3. Get the profile onto the device (QR scan for phones; "Import tunnel from file" on
Windows/macOS; `/etc/wireguard/<name>.conf` + `wg-quick@<name>` on Linux).
4. **Delete the profile from your machine**: `shred -u <path>`. It is the only copy of that
@@ -231,7 +275,7 @@ if the router is not delivering the packets, and a `handshake` with no completed
## Revoking a device
Delete its `[Peer]` stanza from `peers.yaml`, PR, merge. Nothing else — the key stops working on the
Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the
next reconcile. **Never re-use that address octet**: an old profile would silently start working
again as a different device.