fix(vpn): enrol keith-phone-home and admit the WireGuard subnet to PiHole DNS #158
No files matched your search
@@ -83,15 +83,25 @@
|
||||
# and would also fix the same latent defect in unifi and crafty; it belongs in its own change.
|
||||
#
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# loadBalancerSourceRanges — THE LAN RESTRICTION, ENFORCED IN THE DATAPATH.
|
||||
# loadBalancerSourceRanges — THE LAN + VPN RESTRICTION, ENFORCED IN THE DATAPATH.
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# Cilium enforces this list in eBPF on the ingress node, on the VIP, BEFORE load-balancing and
|
||||
# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} is dropped by the
|
||||
# datapath — it never reaches a backend and never reaches a network policy. This is what makes
|
||||
# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} and ${WG_CLIENT_CIDR} is
|
||||
# dropped by the datapath — it never reaches a backend and never reaches a network policy. This is what makes
|
||||
# "LAN clients only" a real, tested restriction rather than a comment: VERIFIED by narrowing the
|
||||
# range to a /24 that excludes the test client, after which the same query that had been answering
|
||||
# stopped being answered at all, and restoring it, after which it answered again.
|
||||
#
|
||||
# ${WG_CLIENT_CIDR} IS LISTED BECAUSE VPN CLIENTS REACH THIS CHECK UNTRANSLATED. wg0 is one of
|
||||
# Cilium's kube-proxy-replacement devices (measured 2026-11-08: `cilium-dbg status` lists eno1, wg0,
|
||||
# tun0), so a tunnel client's query is load-balanced as it enters from wg0, ahead of the iptables
|
||||
# MASQUERADE in platform/vpn-ingress/daemonset.yaml, and the source tested here is still 10.90.0.x
|
||||
# rather than the node's address. Without this entry a full-tunnel client's DNS is predicted to die
|
||||
# with DROP_NOT_IN_SRC_RANGE (derived from Cilium's source; NOT observed live, because no peer has
|
||||
# ever completed a handshake). The range is private and reachable only by a key in peers.conf, so
|
||||
# PiHole stays off the internet; conformance/tests/l0-static/pihole-dns-source-ranges fails the gate
|
||||
# if either entry goes missing or any range here stops being RFC1918.
|
||||
#
|
||||
# It is also the layer that survives the SNAT above. A CiliumNetworkPolicy cannot express this
|
||||
# restriction once the source has been rewritten: measured on this cluster, the forwarded query
|
||||
# arrives from the ingress node's internal router address and Cilium resolves that source to the
|
||||
@@ -113,6 +123,7 @@ spec:
|
||||
externalTrafficPolicy: Cluster
|
||||
loadBalancerSourceRanges:
|
||||
- ${LAN_CIDR}
|
||||
- ${WG_CLIENT_CIDR}
|
||||
selector:
|
||||
app.kubernetes.io/name: pihole
|
||||
ports:
|
||||
@@ -133,6 +144,7 @@ spec:
|
||||
externalTrafficPolicy: Cluster
|
||||
loadBalancerSourceRanges:
|
||||
- ${LAN_CIDR}
|
||||
- ${WG_CLIENT_CIDR}
|
||||
selector:
|
||||
app.kubernetes.io/name: pihole
|
||||
ports:
|
||||
@@ -158,7 +170,9 @@ spec:
|
||||
# "Policy denied ... identity world" drops in `cilium-dbg monitor`). So `world` is the only peer
|
||||
# selector that can describe LAN traffic that has been through the LoadBalancer, and the set of
|
||||
# addresses that can actually BE that traffic is fixed by loadBalancerSourceRanges before this
|
||||
# policy is consulted. Together: ${LAN_CIDR} in the datapath, udp/53 + tcp/53 here.
|
||||
# policy is consulted. Together: ${LAN_CIDR} + ${WG_CLIENT_CIDR} in the datapath, udp/53 + tcp/53
|
||||
# here. A VPN client delivered to a backend on its own ingress node keeps its 10.90.0.x source,
|
||||
# which Cilium also classifies as `world`, so this rule needs no new peer for it.
|
||||
#
|
||||
# This is NOT wider than the precedent it follows. unifi-allow-l4-ingress and crafty-allow-ingress
|
||||
# admit `fromEntities: [world, cluster, host]` on their L4 ports with NO source-range filter at all;
|
||||
@@ -187,7 +201,7 @@ spec:
|
||||
app.kubernetes.io/name: pihole
|
||||
ingress:
|
||||
- fromEntities:
|
||||
- world # LAN clients, post-LoadBalancer: see the block above for why not fromCIDR
|
||||
- world # LAN + VPN clients, post-LoadBalancer: see the block above for why not fromCIDR
|
||||
- host # this node's kubelet probes
|
||||
- remote-node # the node-to-node hop that forwards a query to wherever the pod runs
|
||||
toPorts:
|
||||
|
||||
+1
-1
@@ -85,7 +85,7 @@ env:
|
||||
# Default is LOCAL, which answers only clients on the same subnet as the listening interface. The
|
||||
# pod's interface is on the CLUSTER pod CIDR, so every LAN client would be refused. ALL is
|
||||
# required for a LoadBalancer-fronted resolver; what keeps PiHole from being an OPEN resolver is
|
||||
# the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}]` on the DNS Services,
|
||||
# the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}, ${WG_CLIENT_CIDR}]` on the DNS Services,
|
||||
# enforced by Cilium in the datapath before load-balancing, plus a CiliumNetworkPolicy that admits
|
||||
# nothing but udp/53 and tcp/53. See apps/wave-0/pihole-dns.yaml.
|
||||
- name: FTLCONF_dns_listeningMode
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — PiHole's DNS frontends admit the LAN and the VPN, and nothing public
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# loadBalancerSourceRanges is the only thing keeping PiHole from being an open resolver, and it
|
||||
# must also carry the WireGuard client subnet, because Cilium load-balances tunnel traffic on wg0
|
||||
# before the iptables MASQUERADE (so the source it tests is still 10.90.0.x). check.sh's header
|
||||
# states the assertions and what they do not prove.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library + PyYAML);
|
||||
# conformance/run.sh --layer l0 runs it directly and this Chainsaw Test wraps the SAME script.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: pihole-dns-source-ranges
|
||||
labels:
|
||||
conformance.home.k3s/R-DNS-03: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-DNS-03"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE: ci/touched-requirements.sh maps apps/* to no R-DNS requirement, so a change
|
||||
# to these Services would otherwise select nothing.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Every PiHole LoadBalancer Service lists both the LAN and the WireGuard client subnet in
|
||||
loadBalancerSourceRanges, and every range resolves to RFC1918, so VPN clients can resolve
|
||||
and PiHole stays unreachable from the internet.
|
||||
spec:
|
||||
steps:
|
||||
- name: assert-pihole-source-ranges
|
||||
description: >-
|
||||
Collect the PiHole LoadBalancer Services, resolve each source range through cluster-vars,
|
||||
and assert both required ranges are present and every range is private.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,146 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — pihole-dns-source-ranges
|
||||
# Serves: R-DNS-03 (LAN split-horizon DNS — spec docs/09).
|
||||
#
|
||||
# THE BUG CLASS THIS CLOSES. PiHole's LoadBalancer frontends are the only thing stopping it being
|
||||
# an open resolver, and their `loadBalancerSourceRanges` is pulled in two opposite directions:
|
||||
# * Too WIDE, and the resolver is open: a public range, or 0.0.0.0/0, in this list is the whole
|
||||
# restriction gone, with nothing else in the path to catch it (the CiliumNetworkPolicy admits
|
||||
# `world` on :53, because that is the only peer that can describe post-LoadBalancer traffic).
|
||||
# * Too NARROW, and VPN DNS dies: wg0 is a Cilium kube-proxy-replacement device, so a tunnel
|
||||
# client's query is load-balanced with its 10.90.0.x source intact, ahead of the iptables
|
||||
# MASQUERADE. A LAN-only list is predicted to drop it as DROP_NOT_IN_SRC_RANGE, and the
|
||||
# tunnel would still report a healthy handshake.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every type=LoadBalancer Service in namespace `pihole` is collected and each source range
|
||||
# is resolved through cluster-vars (base value AND the prod overlay's replacement),
|
||||
# THEN (1) the frontends exist and cover both udp/53 and tcp/53;
|
||||
# (2) every frontend lists ${LAN_CIDR} and ${WG_CLIENT_CIDR};
|
||||
# (3) every range resolves to a valid IPv4 CIDR inside RFC1918 — so no 0.0.0.0/0, no public
|
||||
# range, and no unresolvable token that could hide one;
|
||||
# AND (4) the check is NON-VACUOUS: a real checkout and a floor on the assertion count.
|
||||
#
|
||||
# WHAT THIS DOES NOT PROVE: that a query over the tunnel is actually answered. That needs a
|
||||
# handshake from a real client; see platform/vpn-ingress/README.md.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
import ipaddress, re, subprocess, sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
REQUIRED_TOKENS = ("LAN_CIDR", "WG_CLIENT_CIDR")
|
||||
RFC1918 = [ipaddress.ip_network(n) for n in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16")]
|
||||
TOKEN = re.compile(r"^\$\{([A-Z0-9_]+)\}$")
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def docs_of(rel: str):
|
||||
try:
|
||||
return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace"))
|
||||
if isinstance(d, dict)]
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
|
||||
FILES = [p for p in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
|
||||
capture_output=True, text=True, check=True).stdout.split("\0")
|
||||
if p]
|
||||
YAML_FILES = [f for f in FILES if re.search(r"\.ya?ml$", f)]
|
||||
check(len(YAML_FILES) > 50, f"only {len(YAML_FILES)} YAML files seen — not a real checkout")
|
||||
|
||||
# ------------------------------------------------- every value a ${TOKEN} can take on this tree
|
||||
values: dict[str, set[str]] = {}
|
||||
for d in docs_of("clusters/base/cluster-vars.yaml"):
|
||||
if d.get("kind") == "ConfigMap":
|
||||
for k, v in (d.get("data") or {}).items():
|
||||
values.setdefault(k, set()).add(str(v))
|
||||
for d in docs_of("clusters/prod/kustomization.yaml"):
|
||||
for p in d.get("patches") or []:
|
||||
ops = p.get("patch") if isinstance(p, dict) else None
|
||||
ops = yaml.safe_load(ops) if isinstance(ops, str) else ops
|
||||
for op in ops if isinstance(ops, list) else []:
|
||||
m = re.fullmatch(r"/data/([A-Z0-9_]+)", str(op.get("path", "")))
|
||||
if m and op.get("op") in ("add", "replace"):
|
||||
values.setdefault(m.group(1), set()).add(str(op.get("value")))
|
||||
for tok in REQUIRED_TOKENS:
|
||||
check(values.get(tok), f"clusters/base/cluster-vars.yaml defines no {tok} (R-DNS-03)")
|
||||
|
||||
|
||||
def assert_private(where: str, cidr: str):
|
||||
try:
|
||||
net = ipaddress.ip_network(cidr, strict=False)
|
||||
except ValueError:
|
||||
check(False, f"{where}: {cidr!r} is not a valid CIDR (R-DNS-03)")
|
||||
return
|
||||
check(net.version == 4 and any(net.subnet_of(r) for r in RFC1918),
|
||||
f"{where}: {cidr} is not inside RFC1918, so it opens PiHole beyond the LAN and the VPN "
|
||||
"(R-DNS-03)")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------- the frontends
|
||||
frontends = []
|
||||
for rel in YAML_FILES:
|
||||
for d in docs_of(rel):
|
||||
md, spec = d.get("metadata") or {}, d.get("spec") or {}
|
||||
if d.get("kind") == "Service" and md.get("namespace") == "pihole" \
|
||||
and spec.get("type") == "LoadBalancer":
|
||||
frontends.append((rel, md.get("name"), spec))
|
||||
|
||||
protocols = {(p.get("protocol", "TCP"), p.get("port"))
|
||||
for _, _, spec in frontends for p in spec.get("ports") or []}
|
||||
for want in (("UDP", 53), ("TCP", 53)):
|
||||
check(want in protocols, f"no PiHole LoadBalancer Service serves {want[0]}/{want[1]} — this "
|
||||
"check's corpus has moved and must be re-derived (R-DNS-03)")
|
||||
|
||||
for rel, name, spec in frontends:
|
||||
where = f"{rel}: Service/{name}"
|
||||
ranges = spec.get("loadBalancerSourceRanges") or []
|
||||
check(ranges, f"{where} has no loadBalancerSourceRanges, so it answers anyone who can reach "
|
||||
"the VIP (R-DNS-03)")
|
||||
tokens = {m.group(1) for m in (TOKEN.match(str(r)) for r in ranges) if m}
|
||||
for tok in REQUIRED_TOKENS:
|
||||
check(tok in tokens, f"{where} does not list ${{{tok}}} in loadBalancerSourceRanges "
|
||||
"(R-DNS-03)")
|
||||
for r in ranges:
|
||||
m = TOKEN.match(str(r))
|
||||
if not m:
|
||||
assert_private(where, str(r))
|
||||
continue
|
||||
resolved = values.get(m.group(1))
|
||||
check(resolved, f"{where}: {r} is not defined in cluster-vars, so its value cannot be "
|
||||
"shown to be private (R-DNS-03)")
|
||||
for v in sorted(resolved or []):
|
||||
assert_private(f"{where} ({r})", v)
|
||||
|
||||
check(CHECKS >= 15, f"only {CHECKS} assertions ran — this check has gone vacuous (R-DNS-03)")
|
||||
|
||||
if FAILURES:
|
||||
for f in FAILURES:
|
||||
print(f"check FAIL: {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(f"check OK: pihole-dns-source-ranges — {CHECKS} assertions over "
|
||||
f"{len(frontends)} PiHole LoadBalancer Service(s)")
|
||||
PY
|
||||
@@ -0,0 +1,103 @@
|
||||
---
|
||||
date: 2026-11-08
|
||||
branch: fix/wg-home-peer-pihole-sources
|
||||
title: Enrol keith-phone-home and admit the WireGuard subnet to PiHole's DNS frontends
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
- **`platform/vpn-ingress/peers.conf`**: new peer `keith-phone-home`, `10.90.0.3/32`, key generated
|
||||
on the phone. It is the `home` half of a two-tunnel profile (Endpoint `192.168.2.10:51820`),
|
||||
because the router has no hairpin NAT and `wan.` is unusable from inside the house.
|
||||
- **`apps/wave-0/pihole-dns.yaml`**: `${WG_CLIENT_CIDR}` added after `${LAN_CIDR}` on both
|
||||
`pihole-dns-udp` and `pihole-dns-tcp`. This file is hand-written, not rendered (`catalog/render.py`
|
||||
only lists it in `EXTRA_UNITS`), so it is the source of truth and `--emit-waves` was not run.
|
||||
`catalog/pihole.yaml` changes in one comment only, so nothing renders differently.
|
||||
- **New L0 guard `conformance/tests/l0-static/pihole-dns-source-ranges`** (whole-repo scope,
|
||||
R-DNS-03): every PiHole LoadBalancer Service lists both tokens, and every range resolves through
|
||||
cluster-vars (base and prod overlay) to RFC1918.
|
||||
- **`platform/vpn-ingress/README.md`**: corrected "clients arrive SNATed" (false for LoadBalancer
|
||||
VIPs) and the IPv6 claim (Android's `GoBackend` blocks IPv6 for a single-peer default route);
|
||||
documented the home/away pattern, why `wan.` fails from inside, and the prod-1 dependency.
|
||||
|
||||
Rollout on merge: the `wg-peers` ConfigMap hash changes (`m9kcckcb5t` -> `29mf6k867t`), so the
|
||||
DaemonSet rolls, `maxUnavailable: 1`, `maxSurge: 0`, one node at a time in an order the controller
|
||||
chooses. Each roll deletes and recreates that node's `wg0`, so an established session on that node
|
||||
drops and must re-handshake. `keith-phone` has no session today (see below), so nothing is
|
||||
interrupted in practice.
|
||||
|
||||
**The one risk worth watching:** `wg0` is in Cilium's auto-detected device list and `devices` is
|
||||
still unpinned. On 2026-10-06 a crash-looping `wg0` churned that list and all three nodes lost their
|
||||
main-table default route. A rollout is one delete/create per node, not a loop, so it should not
|
||||
repeat that, but after merge check `ip route get 1.1.1.1` on each node.
|
||||
|
||||
The Service change is an in-place update: same VIP, same Service objects, no pod restart.
|
||||
|
||||
Checked on the rest of the path and left unchanged:
|
||||
|
||||
- Gateway VIP `.20` (`gateway/cilium-gateway-shared`): no source ranges, live. Nothing needed.
|
||||
- `pihole-allow-lan-dns-ingress` admits `world` on :53, which covers an untranslated 10.90.0.x
|
||||
source. `platform/vpn-ingress/networkpolicy.yaml` does not apply to a hostNetwork pod.
|
||||
- No L0 check pinned these ranges to the LAN; none needed updating.
|
||||
|
||||
Flagged, not changed (out of scope):
|
||||
|
||||
- `runbooks/vpn-remote-access.md` "Known limitations" repeats both corrected claims, and step 3 of
|
||||
its verification table says the Gateway sees a `host`-identity source.
|
||||
- `platform/vpn-ingress/daemonset.yaml`'s header repeats the SNAT claim.
|
||||
- `clusters/base/cluster-vars.yaml`'s substitution-coverage comment does not list
|
||||
`WG_CLIENT_CIDR` under apps-wave-0. The variable is defined and substituted there regardless.
|
||||
- `peers.conf` still says "No peers enrolled yet" above the enrolled peers.
|
||||
- PiHole's NodePorts are not covered by source ranges (already acknowledged in `pihole-dns.yaml`).
|
||||
Tunnel clients can reach node addresses, so they can reach those too; they are enrolled devices.
|
||||
- Pinning Cilium's `devices` remains the open hardening for the churn risk above.
|
||||
|
||||
## What I verified
|
||||
|
||||
Live, read-only, 2026-11-08:
|
||||
|
||||
- `cilium-dbg status` on one agent: `KubeProxyReplacement: True [eno1 ..., wg0 10.90.0.1, tun0 ...]`
|
||||
and `Masquerading: BPF [eno1, wg0, tun0]`. So `wg0` really is a load-balancing device.
|
||||
- `pihole-dns-tcp` and `-udp` carry `ranges=[192.168.2.0/24]`; no other LoadBalancer Service has
|
||||
source ranges. Live `cluster-vars`: `WG_CLIENT_CIDR=10.90.0.0/24`, `LAN_CIDR=192.168.2.0/24`.
|
||||
- `wg show wg0 latest-handshakes` on all three pods: `keith-phone` = `0`.
|
||||
- `wan.home.keithjackson.dev` resolves publicly to a single A record (via 1.1.1.1).
|
||||
|
||||
Offline, in the pinned toolbox:
|
||||
|
||||
- `flux envsubst --strict` renders both Services to `[192.168.2.0/24, 10.90.0.0/24]`.
|
||||
- `kubectl kustomize platform/vpn-ingress`: `wg-peers-m9kcckcb5t` on the old `peers.conf`,
|
||||
`wg-peers-29mf6k867t` on the new one.
|
||||
- Guard mutation test, each restored afterwards. Baseline passes (22 assertions, 2 Services). All
|
||||
seven fail with rc 1 and a named message: WG range dropped from one Service; `0.0.0.0/0` added;
|
||||
`8.8.8.0/24` added; `WG_CLIENT_CIDR` made public in cluster-vars; prod overlay sets `LAN_CIDR` to
|
||||
`0.0.0.0/0`; ranges block removed; undefined token added.
|
||||
- The Android claim was read from upstream `GoBackend.java` (`wireguard-android`, master): the
|
||||
`allowFamily` calls are skipped when `sawDefaultRoute && config.getPeers().size() == 1`.
|
||||
- Gate (pinned toolbox, from this worktree, on the committed tree): **rc 3, UNDETERMINED, not
|
||||
green.** `gitea-settings` is the only stage counted as not run (no token locally). `conformance`
|
||||
(L2/L3) also skipped as the declared no-cluster gap. `spec-immutability`, `decisions-log`, `lint`,
|
||||
`l0` (including the new guard), `render-golden` and `l1` passed. `approval-policy` reported N/A
|
||||
offline; it will be red in CI until the operator overrides, because `conformance/**` is sensitive.
|
||||
- `git merge-tree` against `feat/pihole-explicit-lan-names` (#157): no conflicts.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **`DROP_NOT_IN_SRC_RANGE` was NOT observed live.** It is a prediction from Cilium's source. Only
|
||||
the `wg0` device membership was measured; the drop itself has never been seen.
|
||||
- **Full-tunnel behaviour was NOT observed live**, on either profile. **No handshake has ever been
|
||||
recorded for any peer**: `keith-phone` reads `0` on all three pods (a counter that resets when a
|
||||
pod restarts, last 22 days ago, so it covers only the time since; earlier entries here also
|
||||
record none).
|
||||
- That tunnel DNS works after this change, including the reply path from a backend on another node
|
||||
and from one on the ingress node. That needs the phone: `dig` over the tunnel, while watching
|
||||
`cilium-dbg monitor --type drop`.
|
||||
- That the Gateway serves tunnel clients. Its source is predicted to be `world`, as for a LAN
|
||||
browser, which is what the apps admit; not tested.
|
||||
- That the new peer's key is the one on the phone. I copied the public key from the brief.
|
||||
- That the rollout leaves node default routes intact. Reasoned from the 2026-10-06 incident being
|
||||
a loop, not measured.
|
||||
- That a LAN client is unaffected while the Service's source ranges are updated. Expected, not
|
||||
measured.
|
||||
- IPv6 behaviour on any non-Android client.
|
||||
- That the WAN forward targets prod-1: it is off-repo.
|
||||
@@ -42,8 +42,9 @@ no version string to match against a CVE, and no amplification to abuse.
|
||||
|
||||
That is the whole security argument, and it is why the exposed port carries no IP allowlist (which
|
||||
would also have made roaming clients unusable — a phone's address changes with every cell tower).
|
||||
Compare `apps/wave-0/pihole-dns.yaml`, which *does* pin `loadBalancerSourceRanges` to the LAN: a
|
||||
resolver answers anybody, so it must be scoped by address. WireGuard answers nobody.
|
||||
Compare `apps/wave-0/pihole-dns.yaml`, which *does* pin `loadBalancerSourceRanges` to the LAN and
|
||||
this tunnel's client subnet: a resolver answers anybody, so it must be scoped by address. WireGuard
|
||||
answers nobody.
|
||||
|
||||
What it does **not** protect against is a stolen device — anyone holding a device's private key
|
||||
*is* that device. That is what revocation is for: delete the stanza, merge, done (~1 min).
|
||||
@@ -55,9 +56,9 @@ What it does **not** protect against is a stolen device — anyone holding a dev
|
||||
┌──────────────┐ udp/51820 ┌───────────────┐ forward ┌────────────────────────────┐
|
||||
│ WireGuard app│─────────────▶│ WAN 51820 ──▶ │───────────▶│ wg0 10.90.0.1/24 │
|
||||
│ 10.90.0.2/32 │ over the │ node IP:51820 │ │ decrypt, then FORWARD │
|
||||
└──────────────┘ internet └───────────────┘ │ SNAT to the node's addr │
|
||||
└──────────────┘ internet └───────────────┘ │ SNAT (LAN), eBPF LB (VIPs)│
|
||||
└─────────────┬──────────────┘
|
||||
│ as `host`/`remote-node`
|
||||
│ translated per destination (below)
|
||||
┌─────────────────────────────────┼──────────────────────┐
|
||||
▼ ▼ ▼
|
||||
LAN 192.168.2.0/24 Gateway VIP .20 (16 web UIs) WAN / internet
|
||||
@@ -86,9 +87,21 @@ resolved, because `apps/wave-0/pihole-dns.yaml`'s ingress policy deliberately om
|
||||
entity. A VPN that looks up and is useless is the worst available outcome, and nothing in the
|
||||
manifests would have hinted at it.
|
||||
|
||||
In the **host** netns there is no pod identity to carry. Forwarded client traffic is SNATed to the
|
||||
node's own address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` —
|
||||
byte-for-byte what a LAN client looks like. Three further problems vanish with it:
|
||||
In the **host** netns there is no pod identity to carry, which is what avoids the 403s. Clients do
|
||||
**not**, however, all arrive SNATed to the node. Measured 2026-11-08: Cilium counts `wg0` among its
|
||||
kube-proxy-replacement devices (`cilium-dbg status`: `eno1`, `wg0`, `tun0`), so what a packet looks
|
||||
like depends on where it is going:
|
||||
|
||||
| destination | handled by | source the destination sees |
|
||||
|---|---|---|
|
||||
| LAN hosts, the internet | kernel FORWARD, then the iptables MASQUERADE in [`daemonset.yaml`](./daemonset.yaml) | the node's address |
|
||||
| LoadBalancer VIPs (Gateway `.20`, PiHole `.23`, ...) | Cilium's eBPF load balancer, as the packet enters from `wg0`, **before** MASQUERADE | `10.90.0.x` at the VIP's source-range check, identity `world` |
|
||||
|
||||
The second row is why PiHole's `loadBalancerSourceRanges` has to list `${WG_CLIENT_CIDR}` as well as
|
||||
the LAN: left LAN-only, tunnel DNS is predicted to die with `DROP_NOT_IN_SRC_RANGE`. That prediction
|
||||
and the row's source column come from reading Cilium's datapath source; **no peer has ever completed
|
||||
a handshake, so neither has been observed live.** The Gateway VIP carries no source ranges and needs
|
||||
no change. Three further problems vanish with the host netns:
|
||||
|
||||
* `net.ipv4.ip_forward` is **already 1** on these nodes (`node_baseline` writes
|
||||
`/etc/sysctl.d/99-k3s-cilium.conf`). It is *not* in the kubelet's safe-sysctl allowlist, so a
|
||||
@@ -163,6 +176,30 @@ internet path. To enforce less *server-side*, add filtering to the `PostUp` chai
|
||||
before the accept rules) — the FORWARD chain there is the only enforcement point, since no
|
||||
NetworkPolicy applies.
|
||||
|
||||
## Phone profiles: one tunnel for home, one for away
|
||||
|
||||
A phone that needs the LAN-only apps runs full tunnel (`AllowedIPs = 0.0.0.0/0`, `DNS = 192.168.2.23`)
|
||||
everywhere, so it needs **two tunnels**, each its own peer with its own key and address:
|
||||
|
||||
| tunnel | peer | address | Endpoint | use when |
|
||||
|---|---|---|---|---|
|
||||
| `away` | `keith-phone` | `10.90.0.2` | `wan.home.keithjackson.dev:51820` | off the home network |
|
||||
| `home` | `keith-phone-home` | `10.90.0.3` | `192.168.2.10:51820` | on the home Wi-Fi |
|
||||
|
||||
**`wan.` cannot work from inside the house.** It is an unproxied A record holding the house's WAN
|
||||
address (`platform/externaldns/README.md`), and the router has no hairpin NAT, so a LAN client that
|
||||
sends to it never comes back in through the port-forward. The Cloudflare Tunnel is no way round: it
|
||||
carries HTTP(S) to the Gateway and WireGuard is UDP to a node, so `wan.` has to stay DNS-only and
|
||||
`away` has to keep using it.
|
||||
|
||||
**`home` depends on `k3s-prod-1`.** Its Endpoint is that node's address, with no failover: if
|
||||
prod-1 is down, `home` cannot handshake even though the other two listeners are healthy. The fix
|
||||
is client-side: edit the tunnel's Endpoint to `192.168.2.11` or `.12`, since every node answers the
|
||||
same keys. No server change and no new key is needed. The same dependency applies to `away` if the
|
||||
WAN forward points at prod-1.
|
||||
|
||||
Android allows one active VPN at a time, so switch tunnels by hand when leaving or arriving.
|
||||
|
||||
## Operating it
|
||||
|
||||
* **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
|
||||
@@ -215,10 +252,13 @@ right way; one was a defect in my own manifest.
|
||||
handshake — and would be a real secret, forcing `peers.conf` into SOPS and costing the
|
||||
reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
|
||||
`wg-server-key` Secret.
|
||||
* **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6
|
||||
still reaches IPv6 sites *outside* the tunnel. Functionally harmless, a privacy leak if the point
|
||||
was to hide the client's network. Fix is an `fd..` ULA on `wg0` plus `::/0` in the client's
|
||||
`AllowedIPs`; not done because nothing here needs it.
|
||||
* **No IPv6.** The server offers no IPv6 tunnel address and `new-peer.sh` emits only `0.0.0.0/0`.
|
||||
What happens to a client's IPv6 then depends on the client. The **Android app does not leak it**:
|
||||
its `GoBackend` skips `allowFamily()` when the config is one peer with a default route, so Android
|
||||
blocks IPv6 for the VPN's duration and apps fall back to IPv4. Other clients are not covered by
|
||||
that code; expect `wg-quick` on a dual-stack Linux host to leave IPv6 outside the tunnel. Fix is
|
||||
an `fd..` ULA on `wg0` plus `::/0` in the client's `AllowedIPs`; not done because nothing here
|
||||
needs it.
|
||||
* **No in-cluster-service reach beyond what the LAN has.** Clients reach cluster apps the way a LAN
|
||||
browser does — through the Gateway VIP. Pod IPs and `10.43.0.0/16` service IPs are not routed
|
||||
into the tunnel.
|
||||
@@ -90,3 +90,7 @@
|
||||
[Peer]
|
||||
PublicKey = pBRSHrRyG2hnsuZ6FvHfEzF8hOVTF5stHUb/7fW2Km8=
|
||||
AllowedIPs = 10.90.0.2/32
|
||||
# keith-phone-home — <owner>, enrolled 2026-11-08 (at-home profile, LAN Endpoint; no hairpin NAT)
|
||||
[Peer]
|
||||
PublicKey = AT8lBJHqrVkv8RL9n0qWDsh5nyl0e3qgrfTAaYKgBz0=
|
||||
AllowedIPs = 10.90.0.3/32
|
||||
Reference in new issue
Block a user