From f13d336630c715def60c199beaf65e119af1b67d Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 8 Nov 2026 20:44:50 -0600 Subject: [PATCH] fix(vpn): enrol keith-phone-home and admit the WireGuard subnet to PiHole DNS Add the at-home peer (10.90.0.3) and list ${WG_CLIENT_CIDR} in the PiHole DNS frontends' loadBalancerSourceRanges. wg0 is a Cilium device, so tunnel clients reach the source-range check untranslated; a LAN-only list is predicted to drop their DNS. Adds an L0 guard that both ranges are present and every range is RFC1918, and corrects the README's SNAT and IPv6 claims. Co-Authored-By: Claude Opus 5.5 --- apps/wave-0/pihole-dns.yaml | 24 ++- catalog/pihole.yaml | 2 +- .../chainsaw-test.yaml | 39 +++++ .../pihole-dns-source-ranges/check.sh | 146 ++++++++++++++++++ .../2026-11-08-wg-home-peer-pihole-sources.md | 103 ++++++++++++ platform/vpn-ingress/README.md | 62 ++++++-- platform/vpn-ingress/peers.conf | 4 + 7 files changed, 363 insertions(+), 17 deletions(-) create mode 100644 conformance/tests/l0-static/pihole-dns-source-ranges/chainsaw-test.yaml create mode 100755 conformance/tests/l0-static/pihole-dns-source-ranges/check.sh create mode 100644 decisions/2026-11-08-wg-home-peer-pihole-sources.md diff --git a/apps/wave-0/pihole-dns.yaml b/apps/wave-0/pihole-dns.yaml index 2bfe140..36333ff 100644 --- a/apps/wave-0/pihole-dns.yaml +++ b/apps/wave-0/pihole-dns.yaml @@ -83,15 +83,25 @@ # and would also fix the same latent defect in unifi and crafty; it belongs in its own change. # # ------------------------------------------------------------------------------------------------- -# loadBalancerSourceRanges — THE LAN RESTRICTION, ENFORCED IN THE DATAPATH. +# loadBalancerSourceRanges — THE LAN + VPN RESTRICTION, ENFORCED IN THE DATAPATH. # ------------------------------------------------------------------------------------------------- # Cilium enforces this list in eBPF on the ingress node, on the VIP, BEFORE load-balancing and -# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} is dropped by the -# datapath — it never reaches a backend and never reaches a network policy. This is what makes +# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} and ${WG_CLIENT_CIDR} is +# dropped by the datapath — it never reaches a backend and never reaches a network policy. This is what makes # "LAN clients only" a real, tested restriction rather than a comment: VERIFIED by narrowing the # range to a /24 that excludes the test client, after which the same query that had been answering # stopped being answered at all, and restoring it, after which it answered again. # +# ${WG_CLIENT_CIDR} IS LISTED BECAUSE VPN CLIENTS REACH THIS CHECK UNTRANSLATED. wg0 is one of +# Cilium's kube-proxy-replacement devices (measured 2026-11-08: `cilium-dbg status` lists eno1, wg0, +# tun0), so a tunnel client's query is load-balanced as it enters from wg0, ahead of the iptables +# MASQUERADE in platform/vpn-ingress/daemonset.yaml, and the source tested here is still 10.90.0.x +# rather than the node's address. Without this entry a full-tunnel client's DNS is predicted to die +# with DROP_NOT_IN_SRC_RANGE (derived from Cilium's source; NOT observed live, because no peer has +# ever completed a handshake). The range is private and reachable only by a key in peers.conf, so +# PiHole stays off the internet; conformance/tests/l0-static/pihole-dns-source-ranges fails the gate +# if either entry goes missing or any range here stops being RFC1918. +# # It is also the layer that survives the SNAT above. A CiliumNetworkPolicy cannot express this # restriction once the source has been rewritten: measured on this cluster, the forwarded query # arrives from the ingress node's internal router address and Cilium resolves that source to the @@ -113,6 +123,7 @@ spec: externalTrafficPolicy: Cluster loadBalancerSourceRanges: - ${LAN_CIDR} + - ${WG_CLIENT_CIDR} selector: app.kubernetes.io/name: pihole ports: @@ -133,6 +144,7 @@ spec: externalTrafficPolicy: Cluster loadBalancerSourceRanges: - ${LAN_CIDR} + - ${WG_CLIENT_CIDR} selector: app.kubernetes.io/name: pihole ports: @@ -158,7 +170,9 @@ spec: # "Policy denied ... identity world" drops in `cilium-dbg monitor`). So `world` is the only peer # selector that can describe LAN traffic that has been through the LoadBalancer, and the set of # addresses that can actually BE that traffic is fixed by loadBalancerSourceRanges before this -# policy is consulted. Together: ${LAN_CIDR} in the datapath, udp/53 + tcp/53 here. +# policy is consulted. Together: ${LAN_CIDR} + ${WG_CLIENT_CIDR} in the datapath, udp/53 + tcp/53 +# here. A VPN client delivered to a backend on its own ingress node keeps its 10.90.0.x source, +# which Cilium also classifies as `world`, so this rule needs no new peer for it. # # This is NOT wider than the precedent it follows. unifi-allow-l4-ingress and crafty-allow-ingress # admit `fromEntities: [world, cluster, host]` on their L4 ports with NO source-range filter at all; @@ -187,7 +201,7 @@ spec: app.kubernetes.io/name: pihole ingress: - fromEntities: - - world # LAN clients, post-LoadBalancer: see the block above for why not fromCIDR + - world # LAN + VPN clients, post-LoadBalancer: see the block above for why not fromCIDR - host # this node's kubelet probes - remote-node # the node-to-node hop that forwards a query to wherever the pod runs toPorts: diff --git a/catalog/pihole.yaml b/catalog/pihole.yaml index 76aa595..82697ef 100644 --- a/catalog/pihole.yaml +++ b/catalog/pihole.yaml @@ -85,7 +85,7 @@ env: # Default is LOCAL, which answers only clients on the same subnet as the listening interface. The # pod's interface is on the CLUSTER pod CIDR, so every LAN client would be refused. ALL is # required for a LoadBalancer-fronted resolver; what keeps PiHole from being an OPEN resolver is - # the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}]` on the DNS Services, + # the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}, ${WG_CLIENT_CIDR}]` on the DNS Services, # enforced by Cilium in the datapath before load-balancing, plus a CiliumNetworkPolicy that admits # nothing but udp/53 and tcp/53. See apps/wave-0/pihole-dns.yaml. - name: FTLCONF_dns_listeningMode diff --git a/conformance/tests/l0-static/pihole-dns-source-ranges/chainsaw-test.yaml b/conformance/tests/l0-static/pihole-dns-source-ranges/chainsaw-test.yaml new file mode 100644 index 0000000..1ca03bd --- /dev/null +++ b/conformance/tests/l0-static/pihole-dns-source-ranges/chainsaw-test.yaml @@ -0,0 +1,39 @@ +# ============================================================================= +# L0 STATIC — PiHole's DNS frontends admit the LAN and the VPN, and nothing public +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# +# loadBalancerSourceRanges is the only thing keeping PiHole from being an open resolver, and it +# must also carry the WireGuard client subnet, because Cilium load-balances tunnel traffic on wg0 +# before the iptables MASQUERADE (so the source it tests is still 10.90.0.x). check.sh's header +# states the assertions and what they do not prove. +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library + PyYAML); +# conformance/run.sh --layer l0 runs it directly and this Chainsaw Test wraps the SAME script. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: pihole-dns-source-ranges + labels: + conformance.home.k3s/R-DNS-03: "true" + annotations: + conformance.home.k3s/requirements: "R-DNS-03" + conformance.home.k3s/layer: "L0" + # WHOLE-REPO SCOPE: ci/touched-requirements.sh maps apps/* to no R-DNS requirement, so a change + # to these Services would otherwise select nothing. + conformance.home.k3s/scope: "whole-repo" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Every PiHole LoadBalancer Service lists both the LAN and the WireGuard client subnet in + loadBalancerSourceRanges, and every range resolves to RFC1918, so VPN clients can resolve + and PiHole stays unreachable from the internet. +spec: + steps: + - name: assert-pihole-source-ranges + description: >- + Collect the PiHole LoadBalancer Services, resolve each source range through cluster-vars, + and assert both required ranges are present and every range is private. + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/pihole-dns-source-ranges/check.sh b/conformance/tests/l0-static/pihole-dns-source-ranges/check.sh new file mode 100755 index 0000000..d35c5cd --- /dev/null +++ b/conformance/tests/l0-static/pihole-dns-source-ranges/check.sh @@ -0,0 +1,146 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — pihole-dns-source-ranges +# Serves: R-DNS-03 (LAN split-horizon DNS — spec docs/09). +# +# THE BUG CLASS THIS CLOSES. PiHole's LoadBalancer frontends are the only thing stopping it being +# an open resolver, and their `loadBalancerSourceRanges` is pulled in two opposite directions: +# * Too WIDE, and the resolver is open: a public range, or 0.0.0.0/0, in this list is the whole +# restriction gone, with nothing else in the path to catch it (the CiliumNetworkPolicy admits +# `world` on :53, because that is the only peer that can describe post-LoadBalancer traffic). +# * Too NARROW, and VPN DNS dies: wg0 is a Cilium kube-proxy-replacement device, so a tunnel +# client's query is load-balanced with its 10.90.0.x source intact, ahead of the iptables +# MASQUERADE. A LAN-only list is predicted to drop it as DROP_NOT_IN_SRC_RANGE, and the +# tunnel would still report a healthy handshake. +# +# GIVEN the repository as git tracks it, +# WHEN every type=LoadBalancer Service in namespace `pihole` is collected and each source range +# is resolved through cluster-vars (base value AND the prod overlay's replacement), +# THEN (1) the frontends exist and cover both udp/53 and tcp/53; +# (2) every frontend lists ${LAN_CIDR} and ${WG_CLIENT_CIDR}; +# (3) every range resolves to a valid IPv4 CIDR inside RFC1918 — so no 0.0.0.0/0, no public +# range, and no unresolvable token that could hide one; +# AND (4) the check is NON-VACUOUS: a real checkout and a floor on the assertion count. +# +# WHAT THIS DOES NOT PROVE: that a query over the tunnel is actually answered. That needs a +# handshake from a real client; see platform/vpn-ingress/README.md. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" + +cd "${ROOT}" +"${PYTHON}" - "${ROOT}" <<'PY' +import ipaddress, re, subprocess, sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + print("check FAIL: PyYAML is required (same dependency as catalog/render.py)", + file=sys.stderr) + raise SystemExit(1) + +ROOT = Path(sys.argv[1]) +FAILURES: list[str] = [] +CHECKS = 0 +REQUIRED_TOKENS = ("LAN_CIDR", "WG_CLIENT_CIDR") +RFC1918 = [ipaddress.ip_network(n) for n in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16")] +TOKEN = re.compile(r"^\$\{([A-Z0-9_]+)\}$") + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +def docs_of(rel: str): + try: + return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace")) + if isinstance(d, dict)] + except Exception: + return [] + + +FILES = [p for p in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"], + capture_output=True, text=True, check=True).stdout.split("\0") + if p] +YAML_FILES = [f for f in FILES if re.search(r"\.ya?ml$", f)] +check(len(YAML_FILES) > 50, f"only {len(YAML_FILES)} YAML files seen — not a real checkout") + +# ------------------------------------------------- every value a ${TOKEN} can take on this tree +values: dict[str, set[str]] = {} +for d in docs_of("clusters/base/cluster-vars.yaml"): + if d.get("kind") == "ConfigMap": + for k, v in (d.get("data") or {}).items(): + values.setdefault(k, set()).add(str(v)) +for d in docs_of("clusters/prod/kustomization.yaml"): + for p in d.get("patches") or []: + ops = p.get("patch") if isinstance(p, dict) else None + ops = yaml.safe_load(ops) if isinstance(ops, str) else ops + for op in ops if isinstance(ops, list) else []: + m = re.fullmatch(r"/data/([A-Z0-9_]+)", str(op.get("path", ""))) + if m and op.get("op") in ("add", "replace"): + values.setdefault(m.group(1), set()).add(str(op.get("value"))) +for tok in REQUIRED_TOKENS: + check(values.get(tok), f"clusters/base/cluster-vars.yaml defines no {tok} (R-DNS-03)") + + +def assert_private(where: str, cidr: str): + try: + net = ipaddress.ip_network(cidr, strict=False) + except ValueError: + check(False, f"{where}: {cidr!r} is not a valid CIDR (R-DNS-03)") + return + check(net.version == 4 and any(net.subnet_of(r) for r in RFC1918), + f"{where}: {cidr} is not inside RFC1918, so it opens PiHole beyond the LAN and the VPN " + "(R-DNS-03)") + + +# ---------------------------------------------------------------------------- the frontends +frontends = [] +for rel in YAML_FILES: + for d in docs_of(rel): + md, spec = d.get("metadata") or {}, d.get("spec") or {} + if d.get("kind") == "Service" and md.get("namespace") == "pihole" \ + and spec.get("type") == "LoadBalancer": + frontends.append((rel, md.get("name"), spec)) + +protocols = {(p.get("protocol", "TCP"), p.get("port")) + for _, _, spec in frontends for p in spec.get("ports") or []} +for want in (("UDP", 53), ("TCP", 53)): + check(want in protocols, f"no PiHole LoadBalancer Service serves {want[0]}/{want[1]} — this " + "check's corpus has moved and must be re-derived (R-DNS-03)") + +for rel, name, spec in frontends: + where = f"{rel}: Service/{name}" + ranges = spec.get("loadBalancerSourceRanges") or [] + check(ranges, f"{where} has no loadBalancerSourceRanges, so it answers anyone who can reach " + "the VIP (R-DNS-03)") + tokens = {m.group(1) for m in (TOKEN.match(str(r)) for r in ranges) if m} + for tok in REQUIRED_TOKENS: + check(tok in tokens, f"{where} does not list ${{{tok}}} in loadBalancerSourceRanges " + "(R-DNS-03)") + for r in ranges: + m = TOKEN.match(str(r)) + if not m: + assert_private(where, str(r)) + continue + resolved = values.get(m.group(1)) + check(resolved, f"{where}: {r} is not defined in cluster-vars, so its value cannot be " + "shown to be private (R-DNS-03)") + for v in sorted(resolved or []): + assert_private(f"{where} ({r})", v) + +check(CHECKS >= 15, f"only {CHECKS} assertions ran — this check has gone vacuous (R-DNS-03)") + +if FAILURES: + for f in FAILURES: + print(f"check FAIL: {f}", file=sys.stderr) + raise SystemExit(1) +print(f"check OK: pihole-dns-source-ranges — {CHECKS} assertions over " + f"{len(frontends)} PiHole LoadBalancer Service(s)") +PY diff --git a/decisions/2026-11-08-wg-home-peer-pihole-sources.md b/decisions/2026-11-08-wg-home-peer-pihole-sources.md new file mode 100644 index 0000000..3f4586d --- /dev/null +++ b/decisions/2026-11-08-wg-home-peer-pihole-sources.md @@ -0,0 +1,103 @@ +--- +date: 2026-11-08 +branch: fix/wg-home-peer-pihole-sources +title: Enrol keith-phone-home and admit the WireGuard subnet to PiHole's DNS frontends +--- + +## What changed + +- **`platform/vpn-ingress/peers.conf`**: new peer `keith-phone-home`, `10.90.0.3/32`, key generated + on the phone. It is the `home` half of a two-tunnel profile (Endpoint `192.168.2.10:51820`), + because the router has no hairpin NAT and `wan.` is unusable from inside the house. +- **`apps/wave-0/pihole-dns.yaml`**: `${WG_CLIENT_CIDR}` added after `${LAN_CIDR}` on both + `pihole-dns-udp` and `pihole-dns-tcp`. This file is hand-written, not rendered (`catalog/render.py` + only lists it in `EXTRA_UNITS`), so it is the source of truth and `--emit-waves` was not run. + `catalog/pihole.yaml` changes in one comment only, so nothing renders differently. +- **New L0 guard `conformance/tests/l0-static/pihole-dns-source-ranges`** (whole-repo scope, + R-DNS-03): every PiHole LoadBalancer Service lists both tokens, and every range resolves through + cluster-vars (base and prod overlay) to RFC1918. +- **`platform/vpn-ingress/README.md`**: corrected "clients arrive SNATed" (false for LoadBalancer + VIPs) and the IPv6 claim (Android's `GoBackend` blocks IPv6 for a single-peer default route); + documented the home/away pattern, why `wan.` fails from inside, and the prod-1 dependency. + +Rollout on merge: the `wg-peers` ConfigMap hash changes (`m9kcckcb5t` -> `29mf6k867t`), so the +DaemonSet rolls, `maxUnavailable: 1`, `maxSurge: 0`, one node at a time in an order the controller +chooses. Each roll deletes and recreates that node's `wg0`, so an established session on that node +drops and must re-handshake. `keith-phone` has no session today (see below), so nothing is +interrupted in practice. + +**The one risk worth watching:** `wg0` is in Cilium's auto-detected device list and `devices` is +still unpinned. On 2026-10-06 a crash-looping `wg0` churned that list and all three nodes lost their +main-table default route. A rollout is one delete/create per node, not a loop, so it should not +repeat that, but after merge check `ip route get 1.1.1.1` on each node. + +The Service change is an in-place update: same VIP, same Service objects, no pod restart. + +Checked on the rest of the path and left unchanged: + +- Gateway VIP `.20` (`gateway/cilium-gateway-shared`): no source ranges, live. Nothing needed. +- `pihole-allow-lan-dns-ingress` admits `world` on :53, which covers an untranslated 10.90.0.x + source. `platform/vpn-ingress/networkpolicy.yaml` does not apply to a hostNetwork pod. +- No L0 check pinned these ranges to the LAN; none needed updating. + +Flagged, not changed (out of scope): + +- `runbooks/vpn-remote-access.md` "Known limitations" repeats both corrected claims, and step 3 of + its verification table says the Gateway sees a `host`-identity source. +- `platform/vpn-ingress/daemonset.yaml`'s header repeats the SNAT claim. +- `clusters/base/cluster-vars.yaml`'s substitution-coverage comment does not list + `WG_CLIENT_CIDR` under apps-wave-0. The variable is defined and substituted there regardless. +- `peers.conf` still says "No peers enrolled yet" above the enrolled peers. +- PiHole's NodePorts are not covered by source ranges (already acknowledged in `pihole-dns.yaml`). + Tunnel clients can reach node addresses, so they can reach those too; they are enrolled devices. +- Pinning Cilium's `devices` remains the open hardening for the churn risk above. + +## What I verified + +Live, read-only, 2026-11-08: + +- `cilium-dbg status` on one agent: `KubeProxyReplacement: True [eno1 ..., wg0 10.90.0.1, tun0 ...]` + and `Masquerading: BPF [eno1, wg0, tun0]`. So `wg0` really is a load-balancing device. +- `pihole-dns-tcp` and `-udp` carry `ranges=[192.168.2.0/24]`; no other LoadBalancer Service has + source ranges. Live `cluster-vars`: `WG_CLIENT_CIDR=10.90.0.0/24`, `LAN_CIDR=192.168.2.0/24`. +- `wg show wg0 latest-handshakes` on all three pods: `keith-phone` = `0`. +- `wan.home.keithjackson.dev` resolves publicly to a single A record (via 1.1.1.1). + +Offline, in the pinned toolbox: + +- `flux envsubst --strict` renders both Services to `[192.168.2.0/24, 10.90.0.0/24]`. +- `kubectl kustomize platform/vpn-ingress`: `wg-peers-m9kcckcb5t` on the old `peers.conf`, + `wg-peers-29mf6k867t` on the new one. +- Guard mutation test, each restored afterwards. Baseline passes (22 assertions, 2 Services). All + seven fail with rc 1 and a named message: WG range dropped from one Service; `0.0.0.0/0` added; + `8.8.8.0/24` added; `WG_CLIENT_CIDR` made public in cluster-vars; prod overlay sets `LAN_CIDR` to + `0.0.0.0/0`; ranges block removed; undefined token added. +- The Android claim was read from upstream `GoBackend.java` (`wireguard-android`, master): the + `allowFamily` calls are skipped when `sawDefaultRoute && config.getPeers().size() == 1`. +- Gate (pinned toolbox, from this worktree, on the committed tree): **rc 3, UNDETERMINED, not + green.** `gitea-settings` is the only stage counted as not run (no token locally). `conformance` + (L2/L3) also skipped as the declared no-cluster gap. `spec-immutability`, `decisions-log`, `lint`, + `l0` (including the new guard), `render-golden` and `l1` passed. `approval-policy` reported N/A + offline; it will be red in CI until the operator overrides, because `conformance/**` is sensitive. +- `git merge-tree` against `feat/pihole-explicit-lan-names` (#157): no conflicts. + +## What I did NOT verify + +- **`DROP_NOT_IN_SRC_RANGE` was NOT observed live.** It is a prediction from Cilium's source. Only + the `wg0` device membership was measured; the drop itself has never been seen. +- **Full-tunnel behaviour was NOT observed live**, on either profile. **No handshake has ever been + recorded for any peer**: `keith-phone` reads `0` on all three pods (a counter that resets when a + pod restarts, last 22 days ago, so it covers only the time since; earlier entries here also + record none). +- That tunnel DNS works after this change, including the reply path from a backend on another node + and from one on the ingress node. That needs the phone: `dig` over the tunnel, while watching + `cilium-dbg monitor --type drop`. +- That the Gateway serves tunnel clients. Its source is predicted to be `world`, as for a LAN + browser, which is what the apps admit; not tested. +- That the new peer's key is the one on the phone. I copied the public key from the brief. +- That the rollout leaves node default routes intact. Reasoned from the 2026-10-06 incident being + a loop, not measured. +- That a LAN client is unaffected while the Service's source ranges are updated. Expected, not + measured. +- IPv6 behaviour on any non-Android client. +- That the WAN forward targets prod-1: it is off-repo. diff --git a/platform/vpn-ingress/README.md b/platform/vpn-ingress/README.md index 3cfc916..84b23c6 100644 --- a/platform/vpn-ingress/README.md +++ b/platform/vpn-ingress/README.md @@ -42,8 +42,9 @@ no version string to match against a CVE, and no amplification to abuse. That is the whole security argument, and it is why the exposed port carries no IP allowlist (which would also have made roaming clients unusable — a phone's address changes with every cell tower). -Compare `apps/wave-0/pihole-dns.yaml`, which *does* pin `loadBalancerSourceRanges` to the LAN: a -resolver answers anybody, so it must be scoped by address. WireGuard answers nobody. +Compare `apps/wave-0/pihole-dns.yaml`, which *does* pin `loadBalancerSourceRanges` to the LAN and +this tunnel's client subnet: a resolver answers anybody, so it must be scoped by address. WireGuard +answers nobody. What it does **not** protect against is a stolen device — anyone holding a device's private key *is* that device. That is what revocation is for: delete the stanza, merge, done (~1 min). @@ -55,9 +56,9 @@ What it does **not** protect against is a stolen device — anyone holding a dev ┌──────────────┐ udp/51820 ┌───────────────┐ forward ┌────────────────────────────┐ │ WireGuard app│─────────────▶│ WAN 51820 ──▶ │───────────▶│ wg0 10.90.0.1/24 │ │ 10.90.0.2/32 │ over the │ node IP:51820 │ │ decrypt, then FORWARD │ - └──────────────┘ internet └───────────────┘ │ SNAT to the node's addr │ + └──────────────┘ internet └───────────────┘ │ SNAT (LAN), eBPF LB (VIPs)│ └─────────────┬──────────────┘ - │ as `host`/`remote-node` + │ translated per destination (below) ┌─────────────────────────────────┼──────────────────────┐ ▼ ▼ ▼ LAN 192.168.2.0/24 Gateway VIP .20 (16 web UIs) WAN / internet @@ -86,9 +87,21 @@ resolved, because `apps/wave-0/pihole-dns.yaml`'s ingress policy deliberately om entity. A VPN that looks up and is useless is the worst available outcome, and nothing in the manifests would have hinted at it. -In the **host** netns there is no pod identity to carry. Forwarded client traffic is SNATed to the -node's own address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` — -byte-for-byte what a LAN client looks like. Three further problems vanish with it: +In the **host** netns there is no pod identity to carry, which is what avoids the 403s. Clients do +**not**, however, all arrive SNATed to the node. Measured 2026-11-08: Cilium counts `wg0` among its +kube-proxy-replacement devices (`cilium-dbg status`: `eno1`, `wg0`, `tun0`), so what a packet looks +like depends on where it is going: + +| destination | handled by | source the destination sees | +|---|---|---| +| LAN hosts, the internet | kernel FORWARD, then the iptables MASQUERADE in [`daemonset.yaml`](./daemonset.yaml) | the node's address | +| LoadBalancer VIPs (Gateway `.20`, PiHole `.23`, ...) | Cilium's eBPF load balancer, as the packet enters from `wg0`, **before** MASQUERADE | `10.90.0.x` at the VIP's source-range check, identity `world` | + +The second row is why PiHole's `loadBalancerSourceRanges` has to list `${WG_CLIENT_CIDR}` as well as +the LAN: left LAN-only, tunnel DNS is predicted to die with `DROP_NOT_IN_SRC_RANGE`. That prediction +and the row's source column come from reading Cilium's datapath source; **no peer has ever completed +a handshake, so neither has been observed live.** The Gateway VIP carries no source ranges and needs +no change. Three further problems vanish with the host netns: * `net.ipv4.ip_forward` is **already 1** on these nodes (`node_baseline` writes `/etc/sysctl.d/99-k3s-cilium.conf`). It is *not* in the kubelet's safe-sysctl allowlist, so a @@ -163,6 +176,30 @@ internet path. To enforce less *server-side*, add filtering to the `PostUp` chai before the accept rules) — the FORWARD chain there is the only enforcement point, since no NetworkPolicy applies. +## Phone profiles: one tunnel for home, one for away + +A phone that needs the LAN-only apps runs full tunnel (`AllowedIPs = 0.0.0.0/0`, `DNS = 192.168.2.23`) +everywhere, so it needs **two tunnels**, each its own peer with its own key and address: + +| tunnel | peer | address | Endpoint | use when | +|---|---|---|---|---| +| `away` | `keith-phone` | `10.90.0.2` | `wan.home.keithjackson.dev:51820` | off the home network | +| `home` | `keith-phone-home` | `10.90.0.3` | `192.168.2.10:51820` | on the home Wi-Fi | + +**`wan.` cannot work from inside the house.** It is an unproxied A record holding the house's WAN +address (`platform/externaldns/README.md`), and the router has no hairpin NAT, so a LAN client that +sends to it never comes back in through the port-forward. The Cloudflare Tunnel is no way round: it +carries HTTP(S) to the Gateway and WireGuard is UDP to a node, so `wan.` has to stay DNS-only and +`away` has to keep using it. + +**`home` depends on `k3s-prod-1`.** Its Endpoint is that node's address, with no failover: if +prod-1 is down, `home` cannot handshake even though the other two listeners are healthy. The fix +is client-side: edit the tunnel's Endpoint to `192.168.2.11` or `.12`, since every node answers the +same keys. No server change and no new key is needed. The same dependency applies to `away` if the +WAN forward points at prod-1. + +Android allows one active VPN at a time, so switch tunnels by hand when leaving or arriving. + ## Operating it * **Enrol a device:** `./infra/vpn-ingress/new-peer.sh ` → paste the printed `[Peer]` @@ -215,10 +252,13 @@ right way; one was a defect in my own manifest. handshake — and would be a real secret, forcing `peers.conf` into SOPS and costing the reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the `wg-server-key` Secret. -* **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6 - still reaches IPv6 sites *outside* the tunnel. Functionally harmless, a privacy leak if the point - was to hide the client's network. Fix is an `fd..` ULA on `wg0` plus `::/0` in the client's - `AllowedIPs`; not done because nothing here needs it. +* **No IPv6.** The server offers no IPv6 tunnel address and `new-peer.sh` emits only `0.0.0.0/0`. + What happens to a client's IPv6 then depends on the client. The **Android app does not leak it**: + its `GoBackend` skips `allowFamily()` when the config is one peer with a default route, so Android + blocks IPv6 for the VPN's duration and apps fall back to IPv4. Other clients are not covered by + that code; expect `wg-quick` on a dual-stack Linux host to leave IPv6 outside the tunnel. Fix is + an `fd..` ULA on `wg0` plus `::/0` in the client's `AllowedIPs`; not done because nothing here + needs it. * **No in-cluster-service reach beyond what the LAN has.** Clients reach cluster apps the way a LAN browser does — through the Gateway VIP. Pod IPs and `10.43.0.0/16` service IPs are not routed into the tunnel. diff --git a/platform/vpn-ingress/peers.conf b/platform/vpn-ingress/peers.conf index f4f048d..c4dea14 100644 --- a/platform/vpn-ingress/peers.conf +++ b/platform/vpn-ingress/peers.conf @@ -90,3 +90,7 @@ [Peer] PublicKey = pBRSHrRyG2hnsuZ6FvHfEzF8hOVTF5stHUb/7fW2Km8= AllowedIPs = 10.90.0.2/32 +# keith-phone-home — , enrolled 2026-11-08 (at-home profile, LAN Endpoint; no hairpin NAT) +[Peer] +PublicKey = AT8lBJHqrVkv8RL9n0qWDsh5nyl0e3qgrfTAaYKgBz0= +AllowedIPs = 10.90.0.3/32 -- 2.54.0