diff --git a/apps/wave-0/pihole-dns.yaml b/apps/wave-0/pihole-dns.yaml index 2bfe140..36333ff 100644 --- a/apps/wave-0/pihole-dns.yaml +++ b/apps/wave-0/pihole-dns.yaml @@ -83,15 +83,25 @@ # and would also fix the same latent defect in unifi and crafty; it belongs in its own change. # # ------------------------------------------------------------------------------------------------- -# loadBalancerSourceRanges — THE LAN RESTRICTION, ENFORCED IN THE DATAPATH. +# loadBalancerSourceRanges — THE LAN + VPN RESTRICTION, ENFORCED IN THE DATAPATH. # ------------------------------------------------------------------------------------------------- # Cilium enforces this list in eBPF on the ingress node, on the VIP, BEFORE load-balancing and -# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} is dropped by the -# datapath — it never reaches a backend and never reaches a network policy. This is what makes +# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} and ${WG_CLIENT_CIDR} is +# dropped by the datapath — it never reaches a backend and never reaches a network policy. This is what makes # "LAN clients only" a real, tested restriction rather than a comment: VERIFIED by narrowing the # range to a /24 that excludes the test client, after which the same query that had been answering # stopped being answered at all, and restoring it, after which it answered again. # +# ${WG_CLIENT_CIDR} IS LISTED BECAUSE VPN CLIENTS REACH THIS CHECK UNTRANSLATED. wg0 is one of +# Cilium's kube-proxy-replacement devices (measured 2026-11-08: `cilium-dbg status` lists eno1, wg0, +# tun0), so a tunnel client's query is load-balanced as it enters from wg0, ahead of the iptables +# MASQUERADE in platform/vpn-ingress/daemonset.yaml, and the source tested here is still 10.90.0.x +# rather than the node's address. Without this entry a full-tunnel client's DNS is predicted to die +# with DROP_NOT_IN_SRC_RANGE (derived from Cilium's source; NOT observed live, because no peer has +# ever completed a handshake). The range is private and reachable only by a key in peers.conf, so +# PiHole stays off the internet; conformance/tests/l0-static/pihole-dns-source-ranges fails the gate +# if either entry goes missing or any range here stops being RFC1918. +# # It is also the layer that survives the SNAT above. A CiliumNetworkPolicy cannot express this # restriction once the source has been rewritten: measured on this cluster, the forwarded query # arrives from the ingress node's internal router address and Cilium resolves that source to the @@ -113,6 +123,7 @@ spec: externalTrafficPolicy: Cluster loadBalancerSourceRanges: - ${LAN_CIDR} + - ${WG_CLIENT_CIDR} selector: app.kubernetes.io/name: pihole ports: @@ -133,6 +144,7 @@ spec: externalTrafficPolicy: Cluster loadBalancerSourceRanges: - ${LAN_CIDR} + - ${WG_CLIENT_CIDR} selector: app.kubernetes.io/name: pihole ports: @@ -158,7 +170,9 @@ spec: # "Policy denied ... identity world" drops in `cilium-dbg monitor`). So `world` is the only peer # selector that can describe LAN traffic that has been through the LoadBalancer, and the set of # addresses that can actually BE that traffic is fixed by loadBalancerSourceRanges before this -# policy is consulted. Together: ${LAN_CIDR} in the datapath, udp/53 + tcp/53 here. +# policy is consulted. Together: ${LAN_CIDR} + ${WG_CLIENT_CIDR} in the datapath, udp/53 + tcp/53 +# here. A VPN client delivered to a backend on its own ingress node keeps its 10.90.0.x source, +# which Cilium also classifies as `world`, so this rule needs no new peer for it. # # This is NOT wider than the precedent it follows. unifi-allow-l4-ingress and crafty-allow-ingress # admit `fromEntities: [world, cluster, host]` on their L4 ports with NO source-range filter at all; @@ -187,7 +201,7 @@ spec: app.kubernetes.io/name: pihole ingress: - fromEntities: - - world # LAN clients, post-LoadBalancer: see the block above for why not fromCIDR + - world # LAN + VPN clients, post-LoadBalancer: see the block above for why not fromCIDR - host # this node's kubelet probes - remote-node # the node-to-node hop that forwards a query to wherever the pod runs toPorts: diff --git a/catalog/pihole.yaml b/catalog/pihole.yaml index 76aa595..82697ef 100644 --- a/catalog/pihole.yaml +++ b/catalog/pihole.yaml @@ -85,7 +85,7 @@ env: # Default is LOCAL, which answers only clients on the same subnet as the listening interface. The # pod's interface is on the CLUSTER pod CIDR, so every LAN client would be refused. ALL is # required for a LoadBalancer-fronted resolver; what keeps PiHole from being an OPEN resolver is - # the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}]` on the DNS Services, + # the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}, ${WG_CLIENT_CIDR}]` on the DNS Services, # enforced by Cilium in the datapath before load-balancing, plus a CiliumNetworkPolicy that admits # nothing but udp/53 and tcp/53. See apps/wave-0/pihole-dns.yaml. - name: FTLCONF_dns_listeningMode diff --git a/conformance/tests/l0-static/pihole-dns-source-ranges/chainsaw-test.yaml b/conformance/tests/l0-static/pihole-dns-source-ranges/chainsaw-test.yaml new file mode 100644 index 0000000..1ca03bd --- /dev/null +++ b/conformance/tests/l0-static/pihole-dns-source-ranges/chainsaw-test.yaml @@ -0,0 +1,39 @@ +# ============================================================================= +# L0 STATIC — PiHole's DNS frontends admit the LAN and the VPN, and nothing public +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# +# loadBalancerSourceRanges is the only thing keeping PiHole from being an open resolver, and it +# must also carry the WireGuard client subnet, because Cilium load-balances tunnel traffic on wg0 +# before the iptables MASQUERADE (so the source it tests is still 10.90.0.x). check.sh's header +# states the assertions and what they do not prove. +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library + PyYAML); +# conformance/run.sh --layer l0 runs it directly and this Chainsaw Test wraps the SAME script. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: pihole-dns-source-ranges + labels: + conformance.home.k3s/R-DNS-03: "true" + annotations: + conformance.home.k3s/requirements: "R-DNS-03" + conformance.home.k3s/layer: "L0" + # WHOLE-REPO SCOPE: ci/touched-requirements.sh maps apps/* to no R-DNS requirement, so a change + # to these Services would otherwise select nothing. + conformance.home.k3s/scope: "whole-repo" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + Every PiHole LoadBalancer Service lists both the LAN and the WireGuard client subnet in + loadBalancerSourceRanges, and every range resolves to RFC1918, so VPN clients can resolve + and PiHole stays unreachable from the internet. +spec: + steps: + - name: assert-pihole-source-ranges + description: >- + Collect the PiHole LoadBalancer Services, resolve each source range through cluster-vars, + and assert both required ranges are present and every range is private. + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/pihole-dns-source-ranges/check.sh b/conformance/tests/l0-static/pihole-dns-source-ranges/check.sh new file mode 100755 index 0000000..d35c5cd --- /dev/null +++ b/conformance/tests/l0-static/pihole-dns-source-ranges/check.sh @@ -0,0 +1,146 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — pihole-dns-source-ranges +# Serves: R-DNS-03 (LAN split-horizon DNS — spec docs/09). +# +# THE BUG CLASS THIS CLOSES. PiHole's LoadBalancer frontends are the only thing stopping it being +# an open resolver, and their `loadBalancerSourceRanges` is pulled in two opposite directions: +# * Too WIDE, and the resolver is open: a public range, or 0.0.0.0/0, in this list is the whole +# restriction gone, with nothing else in the path to catch it (the CiliumNetworkPolicy admits +# `world` on :53, because that is the only peer that can describe post-LoadBalancer traffic). +# * Too NARROW, and VPN DNS dies: wg0 is a Cilium kube-proxy-replacement device, so a tunnel +# client's query is load-balanced with its 10.90.0.x source intact, ahead of the iptables +# MASQUERADE. A LAN-only list is predicted to drop it as DROP_NOT_IN_SRC_RANGE, and the +# tunnel would still report a healthy handshake. +# +# GIVEN the repository as git tracks it, +# WHEN every type=LoadBalancer Service in namespace `pihole` is collected and each source range +# is resolved through cluster-vars (base value AND the prod overlay's replacement), +# THEN (1) the frontends exist and cover both udp/53 and tcp/53; +# (2) every frontend lists ${LAN_CIDR} and ${WG_CLIENT_CIDR}; +# (3) every range resolves to a valid IPv4 CIDR inside RFC1918 — so no 0.0.0.0/0, no public +# range, and no unresolvable token that could hide one; +# AND (4) the check is NON-VACUOUS: a real checkout and a floor on the assertion count. +# +# WHAT THIS DOES NOT PROVE: that a query over the tunnel is actually answered. That needs a +# handshake from a real client; see platform/vpn-ingress/README.md. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" + +cd "${ROOT}" +"${PYTHON}" - "${ROOT}" <<'PY' +import ipaddress, re, subprocess, sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + print("check FAIL: PyYAML is required (same dependency as catalog/render.py)", + file=sys.stderr) + raise SystemExit(1) + +ROOT = Path(sys.argv[1]) +FAILURES: list[str] = [] +CHECKS = 0 +REQUIRED_TOKENS = ("LAN_CIDR", "WG_CLIENT_CIDR") +RFC1918 = [ipaddress.ip_network(n) for n in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16")] +TOKEN = re.compile(r"^\$\{([A-Z0-9_]+)\}$") + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +def docs_of(rel: str): + try: + return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace")) + if isinstance(d, dict)] + except Exception: + return [] + + +FILES = [p for p in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"], + capture_output=True, text=True, check=True).stdout.split("\0") + if p] +YAML_FILES = [f for f in FILES if re.search(r"\.ya?ml$", f)] +check(len(YAML_FILES) > 50, f"only {len(YAML_FILES)} YAML files seen — not a real checkout") + +# ------------------------------------------------- every value a ${TOKEN} can take on this tree +values: dict[str, set[str]] = {} +for d in docs_of("clusters/base/cluster-vars.yaml"): + if d.get("kind") == "ConfigMap": + for k, v in (d.get("data") or {}).items(): + values.setdefault(k, set()).add(str(v)) +for d in docs_of("clusters/prod/kustomization.yaml"): + for p in d.get("patches") or []: + ops = p.get("patch") if isinstance(p, dict) else None + ops = yaml.safe_load(ops) if isinstance(ops, str) else ops + for op in ops if isinstance(ops, list) else []: + m = re.fullmatch(r"/data/([A-Z0-9_]+)", str(op.get("path", ""))) + if m and op.get("op") in ("add", "replace"): + values.setdefault(m.group(1), set()).add(str(op.get("value"))) +for tok in REQUIRED_TOKENS: + check(values.get(tok), f"clusters/base/cluster-vars.yaml defines no {tok} (R-DNS-03)") + + +def assert_private(where: str, cidr: str): + try: + net = ipaddress.ip_network(cidr, strict=False) + except ValueError: + check(False, f"{where}: {cidr!r} is not a valid CIDR (R-DNS-03)") + return + check(net.version == 4 and any(net.subnet_of(r) for r in RFC1918), + f"{where}: {cidr} is not inside RFC1918, so it opens PiHole beyond the LAN and the VPN " + "(R-DNS-03)") + + +# ---------------------------------------------------------------------------- the frontends +frontends = [] +for rel in YAML_FILES: + for d in docs_of(rel): + md, spec = d.get("metadata") or {}, d.get("spec") or {} + if d.get("kind") == "Service" and md.get("namespace") == "pihole" \ + and spec.get("type") == "LoadBalancer": + frontends.append((rel, md.get("name"), spec)) + +protocols = {(p.get("protocol", "TCP"), p.get("port")) + for _, _, spec in frontends for p in spec.get("ports") or []} +for want in (("UDP", 53), ("TCP", 53)): + check(want in protocols, f"no PiHole LoadBalancer Service serves {want[0]}/{want[1]} — this " + "check's corpus has moved and must be re-derived (R-DNS-03)") + +for rel, name, spec in frontends: + where = f"{rel}: Service/{name}" + ranges = spec.get("loadBalancerSourceRanges") or [] + check(ranges, f"{where} has no loadBalancerSourceRanges, so it answers anyone who can reach " + "the VIP (R-DNS-03)") + tokens = {m.group(1) for m in (TOKEN.match(str(r)) for r in ranges) if m} + for tok in REQUIRED_TOKENS: + check(tok in tokens, f"{where} does not list ${{{tok}}} in loadBalancerSourceRanges " + "(R-DNS-03)") + for r in ranges: + m = TOKEN.match(str(r)) + if not m: + assert_private(where, str(r)) + continue + resolved = values.get(m.group(1)) + check(resolved, f"{where}: {r} is not defined in cluster-vars, so its value cannot be " + "shown to be private (R-DNS-03)") + for v in sorted(resolved or []): + assert_private(f"{where} ({r})", v) + +check(CHECKS >= 15, f"only {CHECKS} assertions ran — this check has gone vacuous (R-DNS-03)") + +if FAILURES: + for f in FAILURES: + print(f"check FAIL: {f}", file=sys.stderr) + raise SystemExit(1) +print(f"check OK: pihole-dns-source-ranges — {CHECKS} assertions over " + f"{len(frontends)} PiHole LoadBalancer Service(s)") +PY diff --git a/decisions/2026-11-08-wg-home-peer-pihole-sources.md b/decisions/2026-11-08-wg-home-peer-pihole-sources.md new file mode 100644 index 0000000..3f4586d --- /dev/null +++ b/decisions/2026-11-08-wg-home-peer-pihole-sources.md @@ -0,0 +1,103 @@ +--- +date: 2026-11-08 +branch: fix/wg-home-peer-pihole-sources +title: Enrol keith-phone-home and admit the WireGuard subnet to PiHole's DNS frontends +--- + +## What changed + +- **`platform/vpn-ingress/peers.conf`**: new peer `keith-phone-home`, `10.90.0.3/32`, key generated + on the phone. It is the `home` half of a two-tunnel profile (Endpoint `192.168.2.10:51820`), + because the router has no hairpin NAT and `wan.` is unusable from inside the house. +- **`apps/wave-0/pihole-dns.yaml`**: `${WG_CLIENT_CIDR}` added after `${LAN_CIDR}` on both + `pihole-dns-udp` and `pihole-dns-tcp`. This file is hand-written, not rendered (`catalog/render.py` + only lists it in `EXTRA_UNITS`), so it is the source of truth and `--emit-waves` was not run. + `catalog/pihole.yaml` changes in one comment only, so nothing renders differently. +- **New L0 guard `conformance/tests/l0-static/pihole-dns-source-ranges`** (whole-repo scope, + R-DNS-03): every PiHole LoadBalancer Service lists both tokens, and every range resolves through + cluster-vars (base and prod overlay) to RFC1918. +- **`platform/vpn-ingress/README.md`**: corrected "clients arrive SNATed" (false for LoadBalancer + VIPs) and the IPv6 claim (Android's `GoBackend` blocks IPv6 for a single-peer default route); + documented the home/away pattern, why `wan.` fails from inside, and the prod-1 dependency. + +Rollout on merge: the `wg-peers` ConfigMap hash changes (`m9kcckcb5t` -> `29mf6k867t`), so the +DaemonSet rolls, `maxUnavailable: 1`, `maxSurge: 0`, one node at a time in an order the controller +chooses. Each roll deletes and recreates that node's `wg0`, so an established session on that node +drops and must re-handshake. `keith-phone` has no session today (see below), so nothing is +interrupted in practice. + +**The one risk worth watching:** `wg0` is in Cilium's auto-detected device list and `devices` is +still unpinned. On 2026-10-06 a crash-looping `wg0` churned that list and all three nodes lost their +main-table default route. A rollout is one delete/create per node, not a loop, so it should not +repeat that, but after merge check `ip route get 1.1.1.1` on each node. + +The Service change is an in-place update: same VIP, same Service objects, no pod restart. + +Checked on the rest of the path and left unchanged: + +- Gateway VIP `.20` (`gateway/cilium-gateway-shared`): no source ranges, live. Nothing needed. +- `pihole-allow-lan-dns-ingress` admits `world` on :53, which covers an untranslated 10.90.0.x + source. `platform/vpn-ingress/networkpolicy.yaml` does not apply to a hostNetwork pod. +- No L0 check pinned these ranges to the LAN; none needed updating. + +Flagged, not changed (out of scope): + +- `runbooks/vpn-remote-access.md` "Known limitations" repeats both corrected claims, and step 3 of + its verification table says the Gateway sees a `host`-identity source. +- `platform/vpn-ingress/daemonset.yaml`'s header repeats the SNAT claim. +- `clusters/base/cluster-vars.yaml`'s substitution-coverage comment does not list + `WG_CLIENT_CIDR` under apps-wave-0. The variable is defined and substituted there regardless. +- `peers.conf` still says "No peers enrolled yet" above the enrolled peers. +- PiHole's NodePorts are not covered by source ranges (already acknowledged in `pihole-dns.yaml`). + Tunnel clients can reach node addresses, so they can reach those too; they are enrolled devices. +- Pinning Cilium's `devices` remains the open hardening for the churn risk above. + +## What I verified + +Live, read-only, 2026-11-08: + +- `cilium-dbg status` on one agent: `KubeProxyReplacement: True [eno1 ..., wg0 10.90.0.1, tun0 ...]` + and `Masquerading: BPF [eno1, wg0, tun0]`. So `wg0` really is a load-balancing device. +- `pihole-dns-tcp` and `-udp` carry `ranges=[192.168.2.0/24]`; no other LoadBalancer Service has + source ranges. Live `cluster-vars`: `WG_CLIENT_CIDR=10.90.0.0/24`, `LAN_CIDR=192.168.2.0/24`. +- `wg show wg0 latest-handshakes` on all three pods: `keith-phone` = `0`. +- `wan.home.keithjackson.dev` resolves publicly to a single A record (via 1.1.1.1). + +Offline, in the pinned toolbox: + +- `flux envsubst --strict` renders both Services to `[192.168.2.0/24, 10.90.0.0/24]`. +- `kubectl kustomize platform/vpn-ingress`: `wg-peers-m9kcckcb5t` on the old `peers.conf`, + `wg-peers-29mf6k867t` on the new one. +- Guard mutation test, each restored afterwards. Baseline passes (22 assertions, 2 Services). All + seven fail with rc 1 and a named message: WG range dropped from one Service; `0.0.0.0/0` added; + `8.8.8.0/24` added; `WG_CLIENT_CIDR` made public in cluster-vars; prod overlay sets `LAN_CIDR` to + `0.0.0.0/0`; ranges block removed; undefined token added. +- The Android claim was read from upstream `GoBackend.java` (`wireguard-android`, master): the + `allowFamily` calls are skipped when `sawDefaultRoute && config.getPeers().size() == 1`. +- Gate (pinned toolbox, from this worktree, on the committed tree): **rc 3, UNDETERMINED, not + green.** `gitea-settings` is the only stage counted as not run (no token locally). `conformance` + (L2/L3) also skipped as the declared no-cluster gap. `spec-immutability`, `decisions-log`, `lint`, + `l0` (including the new guard), `render-golden` and `l1` passed. `approval-policy` reported N/A + offline; it will be red in CI until the operator overrides, because `conformance/**` is sensitive. +- `git merge-tree` against `feat/pihole-explicit-lan-names` (#157): no conflicts. + +## What I did NOT verify + +- **`DROP_NOT_IN_SRC_RANGE` was NOT observed live.** It is a prediction from Cilium's source. Only + the `wg0` device membership was measured; the drop itself has never been seen. +- **Full-tunnel behaviour was NOT observed live**, on either profile. **No handshake has ever been + recorded for any peer**: `keith-phone` reads `0` on all three pods (a counter that resets when a + pod restarts, last 22 days ago, so it covers only the time since; earlier entries here also + record none). +- That tunnel DNS works after this change, including the reply path from a backend on another node + and from one on the ingress node. That needs the phone: `dig` over the tunnel, while watching + `cilium-dbg monitor --type drop`. +- That the Gateway serves tunnel clients. Its source is predicted to be `world`, as for a LAN + browser, which is what the apps admit; not tested. +- That the new peer's key is the one on the phone. I copied the public key from the brief. +- That the rollout leaves node default routes intact. Reasoned from the 2026-10-06 incident being + a loop, not measured. +- That a LAN client is unaffected while the Service's source ranges are updated. Expected, not + measured. +- IPv6 behaviour on any non-Android client. +- That the WAN forward targets prod-1: it is off-repo. diff --git a/platform/vpn-ingress/README.md b/platform/vpn-ingress/README.md index 3cfc916..84b23c6 100644 --- a/platform/vpn-ingress/README.md +++ b/platform/vpn-ingress/README.md @@ -42,8 +42,9 @@ no version string to match against a CVE, and no amplification to abuse. That is the whole security argument, and it is why the exposed port carries no IP allowlist (which would also have made roaming clients unusable — a phone's address changes with every cell tower). -Compare `apps/wave-0/pihole-dns.yaml`, which *does* pin `loadBalancerSourceRanges` to the LAN: a -resolver answers anybody, so it must be scoped by address. WireGuard answers nobody. +Compare `apps/wave-0/pihole-dns.yaml`, which *does* pin `loadBalancerSourceRanges` to the LAN and +this tunnel's client subnet: a resolver answers anybody, so it must be scoped by address. WireGuard +answers nobody. What it does **not** protect against is a stolen device — anyone holding a device's private key *is* that device. That is what revocation is for: delete the stanza, merge, done (~1 min). @@ -55,9 +56,9 @@ What it does **not** protect against is a stolen device — anyone holding a dev ┌──────────────┐ udp/51820 ┌───────────────┐ forward ┌────────────────────────────┐ │ WireGuard app│─────────────▶│ WAN 51820 ──▶ │───────────▶│ wg0 10.90.0.1/24 │ │ 10.90.0.2/32 │ over the │ node IP:51820 │ │ decrypt, then FORWARD │ - └──────────────┘ internet └───────────────┘ │ SNAT to the node's addr │ + └──────────────┘ internet └───────────────┘ │ SNAT (LAN), eBPF LB (VIPs)│ └─────────────┬──────────────┘ - │ as `host`/`remote-node` + │ translated per destination (below) ┌─────────────────────────────────┼──────────────────────┐ ▼ ▼ ▼ LAN 192.168.2.0/24 Gateway VIP .20 (16 web UIs) WAN / internet @@ -86,9 +87,21 @@ resolved, because `apps/wave-0/pihole-dns.yaml`'s ingress policy deliberately om entity. A VPN that looks up and is useless is the worst available outcome, and nothing in the manifests would have hinted at it. -In the **host** netns there is no pod identity to carry. Forwarded client traffic is SNATed to the -node's own address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` — -byte-for-byte what a LAN client looks like. Three further problems vanish with it: +In the **host** netns there is no pod identity to carry, which is what avoids the 403s. Clients do +**not**, however, all arrive SNATed to the node. Measured 2026-11-08: Cilium counts `wg0` among its +kube-proxy-replacement devices (`cilium-dbg status`: `eno1`, `wg0`, `tun0`), so what a packet looks +like depends on where it is going: + +| destination | handled by | source the destination sees | +|---|---|---| +| LAN hosts, the internet | kernel FORWARD, then the iptables MASQUERADE in [`daemonset.yaml`](./daemonset.yaml) | the node's address | +| LoadBalancer VIPs (Gateway `.20`, PiHole `.23`, ...) | Cilium's eBPF load balancer, as the packet enters from `wg0`, **before** MASQUERADE | `10.90.0.x` at the VIP's source-range check, identity `world` | + +The second row is why PiHole's `loadBalancerSourceRanges` has to list `${WG_CLIENT_CIDR}` as well as +the LAN: left LAN-only, tunnel DNS is predicted to die with `DROP_NOT_IN_SRC_RANGE`. That prediction +and the row's source column come from reading Cilium's datapath source; **no peer has ever completed +a handshake, so neither has been observed live.** The Gateway VIP carries no source ranges and needs +no change. Three further problems vanish with the host netns: * `net.ipv4.ip_forward` is **already 1** on these nodes (`node_baseline` writes `/etc/sysctl.d/99-k3s-cilium.conf`). It is *not* in the kubelet's safe-sysctl allowlist, so a @@ -163,6 +176,30 @@ internet path. To enforce less *server-side*, add filtering to the `PostUp` chai before the accept rules) — the FORWARD chain there is the only enforcement point, since no NetworkPolicy applies. +## Phone profiles: one tunnel for home, one for away + +A phone that needs the LAN-only apps runs full tunnel (`AllowedIPs = 0.0.0.0/0`, `DNS = 192.168.2.23`) +everywhere, so it needs **two tunnels**, each its own peer with its own key and address: + +| tunnel | peer | address | Endpoint | use when | +|---|---|---|---|---| +| `away` | `keith-phone` | `10.90.0.2` | `wan.home.keithjackson.dev:51820` | off the home network | +| `home` | `keith-phone-home` | `10.90.0.3` | `192.168.2.10:51820` | on the home Wi-Fi | + +**`wan.` cannot work from inside the house.** It is an unproxied A record holding the house's WAN +address (`platform/externaldns/README.md`), and the router has no hairpin NAT, so a LAN client that +sends to it never comes back in through the port-forward. The Cloudflare Tunnel is no way round: it +carries HTTP(S) to the Gateway and WireGuard is UDP to a node, so `wan.` has to stay DNS-only and +`away` has to keep using it. + +**`home` depends on `k3s-prod-1`.** Its Endpoint is that node's address, with no failover: if +prod-1 is down, `home` cannot handshake even though the other two listeners are healthy. The fix +is client-side: edit the tunnel's Endpoint to `192.168.2.11` or `.12`, since every node answers the +same keys. No server change and no new key is needed. The same dependency applies to `away` if the +WAN forward points at prod-1. + +Android allows one active VPN at a time, so switch tunnels by hand when leaving or arriving. + ## Operating it * **Enrol a device:** `./infra/vpn-ingress/new-peer.sh ` → paste the printed `[Peer]` @@ -215,10 +252,13 @@ right way; one was a defect in my own manifest. handshake — and would be a real secret, forcing `peers.conf` into SOPS and costing the reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the `wg-server-key` Secret. -* **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6 - still reaches IPv6 sites *outside* the tunnel. Functionally harmless, a privacy leak if the point - was to hide the client's network. Fix is an `fd..` ULA on `wg0` plus `::/0` in the client's - `AllowedIPs`; not done because nothing here needs it. +* **No IPv6.** The server offers no IPv6 tunnel address and `new-peer.sh` emits only `0.0.0.0/0`. + What happens to a client's IPv6 then depends on the client. The **Android app does not leak it**: + its `GoBackend` skips `allowFamily()` when the config is one peer with a default route, so Android + blocks IPv6 for the VPN's duration and apps fall back to IPv4. Other clients are not covered by + that code; expect `wg-quick` on a dual-stack Linux host to leave IPv6 outside the tunnel. Fix is + an `fd..` ULA on `wg0` plus `::/0` in the client's `AllowedIPs`; not done because nothing here + needs it. * **No in-cluster-service reach beyond what the LAN has.** Clients reach cluster apps the way a LAN browser does — through the Gateway VIP. Pod IPs and `10.43.0.0/16` service IPs are not routed into the tunnel. diff --git a/platform/vpn-ingress/peers.conf b/platform/vpn-ingress/peers.conf index f4f048d..c4dea14 100644 --- a/platform/vpn-ingress/peers.conf +++ b/platform/vpn-ingress/peers.conf @@ -90,3 +90,7 @@ [Peer] PublicKey = pBRSHrRyG2hnsuZ6FvHfEzF8hOVTF5stHUb/7fW2Km8= AllowedIPs = 10.90.0.2/32 +# keith-phone-home — , enrolled 2026-11-08 (at-home profile, LAN Endpoint; no hairpin NAT) +[Peer] +PublicKey = AT8lBJHqrVkv8RL9n0qWDsh5nyl0e3qgrfTAaYKgBz0= +AllowedIPs = 10.90.0.3/32