Compare commits

...
Author SHA1 Message Date
jakman85andClaude Opus 5.5 8cd5ab2e41 chore(spec): adopt the spec sync: registry parity and ADR cross-references
gate / gate (pull_request) Failing after 1m4s
gate / conformance (pull_request) Skipped
Add R-DNS-04, R-DNS-05 and R-SEC-04 to the requirement mirror and tag the
L0 tests that verify them. Align the R-ENV and R-UPD-02 summaries with the
amended requirements. Point the VPN egress docs at ADR-0020; the drafted
"ADR-0016" number went to the Cloudflare Tunnel.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 22:47:54 -06:00
jakman85 50c8719b12 Merge pull request 'fix(vpn): default client profiles to MTU 1280; correct the VPN runbook' (#159) from fix/wg-client-mtu-docs into main
gate / gate (push) Successful in 57s
gate / conformance (push) Skipped
Reviewed-on: #159
2026-11-09 03:26:02 +00:00
jakman85andClaude Opus 5.5 8f60e6103a fix(vpn): default client profiles to MTU 1280; correct the VPN runbook
gate / gate (pull_request) Successful in 57s
gate / conformance (pull_request) Skipped
new-peer.sh generated MTU 1420, which stalled traffic on mobile data
despite a healthy handshake (one phone, one carrier, 2026-11-09). Default
to 1280, overridable through WG_MTU.

Correct stale claims in the runbook and the DaemonSet header against
live BPF conntrack: LoadBalancer VIPs see 10.90.0.x as world, not a
SNATed host identity. Add a troubleshooting section.

The DaemonSet render is unchanged, so WireGuard pods do not roll.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 21:15:20 -06:00
jakman85 da0515c8c8 Merge pull request 'fix(vpn): enrol keith-phone-home and admit the WireGuard subnet to PiHole DNS' (#158) from fix/wg-home-peer-pihole-sources into main
gate / gate (push) Successful in 55s
gate / conformance (push) Skipped
Reviewed-on: #158
2026-11-09 02:50:04 +00:00
jakman85 3a2ec5a12c Merge pull request 'feat(dns): PiHole answers an explicit LAN name list, not a zone wildcard' (#157) from feat/pihole-explicit-lan-names into main
gate / gate (push) Canceled after 25s
gate / conformance (push) Canceled after 0s
Reviewed-on: #157
Reviewed-by: Keith Jackson <[email protected]>
2026-11-09 02:49:39 +00:00
jakman85andClaude Opus 5.5 f13d336630 fix(vpn): enrol keith-phone-home and admit the WireGuard subnet to PiHole DNS
gate / gate (pull_request) Successful in 1m12s
gate / conformance (pull_request) Skipped
Add the at-home peer (10.90.0.3) and list ${WG_CLIENT_CIDR} in the PiHole
DNS frontends' loadBalancerSourceRanges. wg0 is a Cilium device, so tunnel
clients reach the source-range check untranslated; a LAN-only list is
predicted to drop their DNS. Adds an L0 guard that both ranges are present
and every range is RFC1918, and corrects the README's SNAT and IPv6 claims.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 20:45:48 -06:00
18 changed files with 627 additions and 45 deletions

No files matched your search

+19 -5
View File
@@ -83,15 +83,25 @@
# and would also fix the same latent defect in unifi and crafty; it belongs in its own change.
#
# -------------------------------------------------------------------------------------------------
# loadBalancerSourceRanges — THE LAN RESTRICTION, ENFORCED IN THE DATAPATH.
# loadBalancerSourceRanges — THE LAN + VPN RESTRICTION, ENFORCED IN THE DATAPATH.
# -------------------------------------------------------------------------------------------------
# Cilium enforces this list in eBPF on the ingress node, on the VIP, BEFORE load-balancing and
# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} is dropped by the
# datapath — it never reaches a backend and never reaches a network policy. This is what makes
# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} and ${WG_CLIENT_CIDR} is
# dropped by the datapath — it never reaches a backend and never reaches a network policy. This is what makes
# "LAN clients only" a real, tested restriction rather than a comment: VERIFIED by narrowing the
# range to a /24 that excludes the test client, after which the same query that had been answering
# stopped being answered at all, and restoring it, after which it answered again.
#
# ${WG_CLIENT_CIDR} IS LISTED BECAUSE VPN CLIENTS REACH THIS CHECK UNTRANSLATED. wg0 is one of
# Cilium's kube-proxy-replacement devices (measured 2026-11-08: `cilium-dbg status` lists eno1, wg0,
# tun0), so a tunnel client's query is load-balanced as it enters from wg0, ahead of the iptables
# MASQUERADE in platform/vpn-ingress/daemonset.yaml, and the source tested here is still 10.90.0.x
# rather than the node's address. Without this entry a full-tunnel client's DNS is predicted to die
# with DROP_NOT_IN_SRC_RANGE (derived from Cilium's source; NOT observed live, because no peer has
# ever completed a handshake). The range is private and reachable only by a key in peers.conf, so
# PiHole stays off the internet; conformance/tests/l0-static/pihole-dns-source-ranges fails the gate
# if either entry goes missing or any range here stops being RFC1918.
#
# It is also the layer that survives the SNAT above. A CiliumNetworkPolicy cannot express this
# restriction once the source has been rewritten: measured on this cluster, the forwarded query
# arrives from the ingress node's internal router address and Cilium resolves that source to the
@@ -113,6 +123,7 @@ spec:
externalTrafficPolicy: Cluster
loadBalancerSourceRanges:
- ${LAN_CIDR}
- ${WG_CLIENT_CIDR}
selector:
app.kubernetes.io/name: pihole
ports:
@@ -133,6 +144,7 @@ spec:
externalTrafficPolicy: Cluster
loadBalancerSourceRanges:
- ${LAN_CIDR}
- ${WG_CLIENT_CIDR}
selector:
app.kubernetes.io/name: pihole
ports:
@@ -158,7 +170,9 @@ spec:
# "Policy denied ... identity world" drops in `cilium-dbg monitor`). So `world` is the only peer
# selector that can describe LAN traffic that has been through the LoadBalancer, and the set of
# addresses that can actually BE that traffic is fixed by loadBalancerSourceRanges before this
# policy is consulted. Together: ${LAN_CIDR} in the datapath, udp/53 + tcp/53 here.
# policy is consulted. Together: ${LAN_CIDR} + ${WG_CLIENT_CIDR} in the datapath, udp/53 + tcp/53
# here. A VPN client delivered to a backend on its own ingress node keeps its 10.90.0.x source,
# which Cilium also classifies as `world`, so this rule needs no new peer for it.
#
# This is NOT wider than the precedent it follows. unifi-allow-l4-ingress and crafty-allow-ingress
# admit `fromEntities: [world, cluster, host]` on their L4 ports with NO source-range filter at all;
@@ -187,7 +201,7 @@ spec:
app.kubernetes.io/name: pihole
ingress:
- fromEntities:
- world # LAN clients, post-LoadBalancer: see the block above for why not fromCIDR
- world # LAN + VPN clients, post-LoadBalancer: see the block above for why not fromCIDR
- host # this node's kubelet probes
- remote-node # the node-to-node hop that forwards a query to wherever the pod runs
toPorts:
+1 -1
View File
@@ -85,7 +85,7 @@ env:
# Default is LOCAL, which answers only clients on the same subnet as the listening interface. The
# pod's interface is on the CLUSTER pod CIDR, so every LAN client would be refused. ALL is
# required for a LoadBalancer-fronted resolver; what keeps PiHole from being an OPEN resolver is
# the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}]` on the DNS Services,
# the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}, ${WG_CLIENT_CIDR}]` on the DNS Services,
# enforced by Cilium in the datapath before load-balancing, plus a CiliumNetworkPolicy that admits
# nothing but udp/53 and tcp/53. See apps/wave-0/pihole-dns.yaml.
- name: FTLCONF_dns_listeningMode
+16 -4
View File
@@ -120,6 +120,14 @@ requirements:
doc: docs/09-dns-and-ingress.md
safety_critical: false
summary: "Internal clients resolve to the internal Gateway (split-horizon)."
- id: R-DNS-04
doc: docs/09-dns-and-ingress.md
safety_critical: false
summary: "Public HTTP(S) reachable with no inbound 80/443 forward; LAN works whichever resolver answers."
- id: R-DNS-05
doc: docs/09-dns-and-ingress.md
safety_critical: false
summary: "Only declared-public names reachable through public ingress; LAN/VPN-only names are refused."
# --- Automated updates ---
- id: R-UPD-01
@@ -129,7 +137,7 @@ requirements:
- id: R-UPD-02
doc: docs/11-automated-updates.md
safety_critical: true
summary: "Updates need no human/AI/hand-authored change; bad update auto-rolls-back."
summary: "Allowed-class updates need no human/AI/hand-authored change; bad update auto-rolls-back."
- id: R-UPD-03
doc: docs/11-automated-updates.md
safety_critical: false
@@ -175,15 +183,15 @@ requirements:
- id: R-ENV-01
doc: docs/02-environments-and-provisioning.md
safety_critical: false
summary: "dev/test provisioned as local KVMs via IaC (create/baseline/rebuild)."
summary: "Nodes baselined, cluster-joined and managed as code; bare-metal imaging is checklist-driven."
- id: R-ENV-02
doc: docs/02-environments-and-provisioning.md
safety_critical: false
summary: "Same GitOps config promotes dev/test -> prod (authored once)."
summary: "GitOps config authored once as base plus overlay; a future environment needs overrides only."
- id: R-ENV-03
doc: docs/02-environments-and-provisioning.md
safety_critical: false
summary: "prod is 3x mini-PC bare-metal k3s with local storage."
summary: "Production is 3x mini-PC bare metal; changes reach it only through the PR gate and Flux."
# --- GitOps & delivery ---
- id: R-GITOPS-01
@@ -212,6 +220,10 @@ requirements:
doc: docs/07-security-and-secrets.md
safety_critical: false
summary: "All security properties demonstrable/auditable."
- id: R-SEC-04
doc: docs/07-security-and-secrets.md
safety_critical: false
summary: "Kubernetes API not reachable from the public internet by any path; LAN/VPN-only."
# --- Identity & access ---
- id: R-IDP-01
@@ -22,8 +22,11 @@ metadata:
labels:
conformance.home.k3s/R-NET-01: "true"
conformance.home.k3s/R-NET-02: "true"
conformance.home.k3s/R-DNS-04: "true"
conformance.home.k3s/R-DNS-05: "true"
conformance.home.k3s/R-SEC-04: "true"
annotations:
conformance.home.k3s/requirements: "R-NET-01, R-NET-02"
conformance.home.k3s/requirements: "R-NET-01, R-NET-02, R-DNS-04, R-DNS-05, R-SEC-04"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "true"
@@ -105,8 +105,9 @@ metadata:
labels:
conformance.home.k3s/R-DNS-02: "true"
conformance.home.k3s/R-DNS-03: "true"
conformance.home.k3s/R-DNS-05: "true"
annotations:
conformance.home.k3s/requirements: "R-DNS-02, R-DNS-03"
conformance.home.k3s/requirements: "R-DNS-02, R-DNS-03, R-DNS-05"
conformance.home.k3s/layer: "L0"
# WHOLE-REPO SCOPE — this invariant quantifies over every ServiceDefinition in catalog/, every
# HTTPRoute in apps/wave-* and platform/* and runbooks/, and every external-dns-annotated
@@ -0,0 +1,39 @@
# =============================================================================
# L0 STATIC — PiHole's DNS frontends admit the LAN and the VPN, and nothing public
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# loadBalancerSourceRanges is the only thing keeping PiHole from being an open resolver, and it
# must also carry the WireGuard client subnet, because Cilium load-balances tunnel traffic on wg0
# before the iptables MASQUERADE (so the source it tests is still 10.90.0.x). check.sh's header
# states the assertions and what they do not prove.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library + PyYAML);
# conformance/run.sh --layer l0 runs it directly and this Chainsaw Test wraps the SAME script.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: pihole-dns-source-ranges
labels:
conformance.home.k3s/R-DNS-03: "true"
annotations:
conformance.home.k3s/requirements: "R-DNS-03"
conformance.home.k3s/layer: "L0"
# WHOLE-REPO SCOPE: ci/touched-requirements.sh maps apps/* to no R-DNS requirement, so a change
# to these Services would otherwise select nothing.
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Every PiHole LoadBalancer Service lists both the LAN and the WireGuard client subnet in
loadBalancerSourceRanges, and every range resolves to RFC1918, so VPN clients can resolve
and PiHole stays unreachable from the internet.
spec:
steps:
- name: assert-pihole-source-ranges
description: >-
Collect the PiHole LoadBalancer Services, resolve each source range through cluster-vars,
and assert both required ranges are present and every range is private.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,146 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — pihole-dns-source-ranges
# Serves: R-DNS-03 (LAN split-horizon DNS — spec docs/09).
#
# THE BUG CLASS THIS CLOSES. PiHole's LoadBalancer frontends are the only thing stopping it being
# an open resolver, and their `loadBalancerSourceRanges` is pulled in two opposite directions:
# * Too WIDE, and the resolver is open: a public range, or 0.0.0.0/0, in this list is the whole
# restriction gone, with nothing else in the path to catch it (the CiliumNetworkPolicy admits
# `world` on :53, because that is the only peer that can describe post-LoadBalancer traffic).
# * Too NARROW, and VPN DNS dies: wg0 is a Cilium kube-proxy-replacement device, so a tunnel
# client's query is load-balanced with its 10.90.0.x source intact, ahead of the iptables
# MASQUERADE. A LAN-only list is predicted to drop it as DROP_NOT_IN_SRC_RANGE, and the
# tunnel would still report a healthy handshake.
#
# GIVEN the repository as git tracks it,
# WHEN every type=LoadBalancer Service in namespace `pihole` is collected and each source range
# is resolved through cluster-vars (base value AND the prod overlay's replacement),
# THEN (1) the frontends exist and cover both udp/53 and tcp/53;
# (2) every frontend lists ${LAN_CIDR} and ${WG_CLIENT_CIDR};
# (3) every range resolves to a valid IPv4 CIDR inside RFC1918 — so no 0.0.0.0/0, no public
# range, and no unresolvable token that could hide one;
# AND (4) the check is NON-VACUOUS: a real checkout and a floor on the assertion count.
#
# WHAT THIS DOES NOT PROVE: that a query over the tunnel is actually answered. That needs a
# handshake from a real client; see platform/vpn-ingress/README.md.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
import ipaddress, re, subprocess, sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
FAILURES: list[str] = []
CHECKS = 0
REQUIRED_TOKENS = ("LAN_CIDR", "WG_CLIENT_CIDR")
RFC1918 = [ipaddress.ip_network(n) for n in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16")]
TOKEN = re.compile(r"^\$\{([A-Z0-9_]+)\}$")
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def docs_of(rel: str):
try:
return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace"))
if isinstance(d, dict)]
except Exception:
return []
FILES = [p for p in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
capture_output=True, text=True, check=True).stdout.split("\0")
if p]
YAML_FILES = [f for f in FILES if re.search(r"\.ya?ml$", f)]
check(len(YAML_FILES) > 50, f"only {len(YAML_FILES)} YAML files seen — not a real checkout")
# ------------------------------------------------- every value a ${TOKEN} can take on this tree
values: dict[str, set[str]] = {}
for d in docs_of("clusters/base/cluster-vars.yaml"):
if d.get("kind") == "ConfigMap":
for k, v in (d.get("data") or {}).items():
values.setdefault(k, set()).add(str(v))
for d in docs_of("clusters/prod/kustomization.yaml"):
for p in d.get("patches") or []:
ops = p.get("patch") if isinstance(p, dict) else None
ops = yaml.safe_load(ops) if isinstance(ops, str) else ops
for op in ops if isinstance(ops, list) else []:
m = re.fullmatch(r"/data/([A-Z0-9_]+)", str(op.get("path", "")))
if m and op.get("op") in ("add", "replace"):
values.setdefault(m.group(1), set()).add(str(op.get("value")))
for tok in REQUIRED_TOKENS:
check(values.get(tok), f"clusters/base/cluster-vars.yaml defines no {tok} (R-DNS-03)")
def assert_private(where: str, cidr: str):
try:
net = ipaddress.ip_network(cidr, strict=False)
except ValueError:
check(False, f"{where}: {cidr!r} is not a valid CIDR (R-DNS-03)")
return
check(net.version == 4 and any(net.subnet_of(r) for r in RFC1918),
f"{where}: {cidr} is not inside RFC1918, so it opens PiHole beyond the LAN and the VPN "
"(R-DNS-03)")
# ---------------------------------------------------------------------------- the frontends
frontends = []
for rel in YAML_FILES:
for d in docs_of(rel):
md, spec = d.get("metadata") or {}, d.get("spec") or {}
if d.get("kind") == "Service" and md.get("namespace") == "pihole" \
and spec.get("type") == "LoadBalancer":
frontends.append((rel, md.get("name"), spec))
protocols = {(p.get("protocol", "TCP"), p.get("port"))
for _, _, spec in frontends for p in spec.get("ports") or []}
for want in (("UDP", 53), ("TCP", 53)):
check(want in protocols, f"no PiHole LoadBalancer Service serves {want[0]}/{want[1]} — this "
"check's corpus has moved and must be re-derived (R-DNS-03)")
for rel, name, spec in frontends:
where = f"{rel}: Service/{name}"
ranges = spec.get("loadBalancerSourceRanges") or []
check(ranges, f"{where} has no loadBalancerSourceRanges, so it answers anyone who can reach "
"the VIP (R-DNS-03)")
tokens = {m.group(1) for m in (TOKEN.match(str(r)) for r in ranges) if m}
for tok in REQUIRED_TOKENS:
check(tok in tokens, f"{where} does not list ${{{tok}}} in loadBalancerSourceRanges "
"(R-DNS-03)")
for r in ranges:
m = TOKEN.match(str(r))
if not m:
assert_private(where, str(r))
continue
resolved = values.get(m.group(1))
check(resolved, f"{where}: {r} is not defined in cluster-vars, so its value cannot be "
"shown to be private (R-DNS-03)")
for v in sorted(resolved or []):
assert_private(f"{where} ({r})", v)
check(CHECKS >= 15, f"only {CHECKS} assertions ran — this check has gone vacuous (R-DNS-03)")
if FAILURES:
for f in FAILURES:
print(f"check FAIL: {f}", file=sys.stderr)
raise SystemExit(1)
print(f"check OK: pihole-dns-source-ranges — {CHECKS} assertions over "
f"{len(frontends)} PiHole LoadBalancer Service(s)")
PY
@@ -20,8 +20,9 @@ metadata:
name: pihole-lan-names
labels:
conformance.home.k3s/R-DNS-03: "true"
conformance.home.k3s/R-DNS-04: "true"
annotations:
conformance.home.k3s/requirements: "R-DNS-03"
conformance.home.k3s/requirements: "R-DNS-03, R-DNS-04"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "true"
+46
View File
@@ -0,0 +1,46 @@
---
date: 2026-11-08
branch: chore/adopt-spec-sync
title: Adopt the spec sync (registry parity, ADR cross-references, spec pin)
---
## What changed
The operator ruled this repo's `main` (`50c8719`) the accepted state and had the System Architect
bring the spec repo in line with it (spec PR `jakman85/home-network-k3s#2`). This is the
implementation half of that sync.
- `conformance/requirements/registry.yaml`: added R-DNS-04, R-DNS-05 and R-SEC-04, which the spec
defined and this mirror lacked. Reworded the R-ENV-01/02/03 and R-UPD-02 summaries to match the
amended requirements.
- Tagged existing L0 tests against those IDs, because a registry entry with no tagged test blocks
the merge: `cloudflared-ingress-rules` (R-DNS-04, R-DNS-05, R-SEC-04), `pihole-lan-names`
(R-DNS-04), and `internal-services-publish-no-public-dns` (R-DNS-05).
- `platform/vpn-egress/README.md` and a comment in `egress-steering.yaml` pointed at a drafted
"ADR-0016 supersede". That number went to the Cloudflare Tunnel; the accepted ADR is ADR-0020.
- `ci/SPEC_PIN` must move to the spec merge commit. That SHA does not exist until the spec PR
merges, so this PR carries a `WIP:` title until the bump is added.
## What I verified
- Read each tagged test's assertions before tagging it. `cloudflared-ingress-rules` asserts every
LAN-only hostname maps to a 404, the catch-all exists, no rule targets the API, and the egress
policy admits no API or host entity. `pihole-lan-names` asserts LAN names resolve independently
of the public path. `internal-services-publish-no-public-dns` asserts internal routes carry the
disown annotation and pins the declared L4 set.
- The toolbox gate result for this commit is quoted in the PR description, with the spec branch
mounted so that the checks parsing spec text ran against the new spec.
## What I did NOT verify
- **These tags are static coverage, not proof.** No L2 test touches the tunnel, and nothing here
shows router 80/443 closed or 6443 unforwarded (spec `docs/16-known-gaps.md`).
- R-SEC-04 is registered `safety_critical: false`. Making it `true` needs an L3 fault-injection
test that does not exist. UNVERIFIED whether the operator wants that.
- No live cluster, tunnel or DNS state was measured.
- `spec-immutability` did not run against a pinned checkout: the pin is not bumped yet.
- Stale implementation prose found by the audit was deliberately left out of this PR, because the
operator ruled this repo the accepted state and none of it changes behaviour. Examples:
`catalog/gitea.yaml` still says "+ mirrored"; `platform/longhorn` still promises a backup target;
`README.md` says `node-loss` is outstanding; `hands-off-update` exercises Flux image policy rather
than Renovate; `dev-nodes-provisioned-as-code` is named for an environment that no longer exists.
@@ -0,0 +1,74 @@
---
date: 2026-11-08
branch: fix/wg-client-mtu-docs
title: WireGuard client profiles default to MTU 1280; VPN runbook corrected and given a troubleshooting section
---
## What changed
- **`infra/vpn-ingress/new-peer.sh`**: `WG_MTU` default 1420 -> 1280, still overridable, reason in a
comment. Only new profiles change; the phone's existing profiles are not touched.
- **`runbooks/vpn-remote-access.md`**: the verification table no longer says the Gateway sees a
`host`-identity source; the limitations no longer say everything arrives SNATed or that IPv6 leaks
on Android; the "if nothing handshakes" advice no longer claims `wg show` separates a bad forward
from a bad key. New "Troubleshooting" section covering the four cases in the brief.
- **`platform/vpn-ingress/daemonset.yaml`**: header comment only, SNAT claim corrected.
- **`platform/vpn-ingress/README.md`**: one sentence, so `MTU = 1420` reads as the server's value.
Rollout on merge: the WireGuard DaemonSet does **not** roll. The `runbooks` ConfigMap is content
hashed and `vpn-remote-access.md` is in it, so the runbooks site rolls.
Left unchanged, for the operator:
- **`platform/vpn-ingress/peers.conf`** still says "No peers enrolled yet". A comment-only edit on a
throwaway copy moved the ConfigMap from `wg-peers-29mf6k867t` to `wg-peers-gggkk5d4tm` and changed
the DaemonSet, so correcting it restarts every `wg0`. Fix it alongside the next peer change.
- **`clusters/base/cluster-vars.yaml`** coverage comment still omits `WG_CLIENT_CIDR` (sensitive path).
- **Server `wg0` MTU stays 1420**, because changing it restarts all WireGuard pods. For TCP a 1280
client should be enough on its own: it advertises an MSS of 1240, which caps what the far end
sends. That protection does not reach UDP. A server-to-phone datagram with an inner size of
1281-1420 bytes is still sent as up to 1480 outer bytes and could be lost on the same path; QUIC
should recover by probing, large DNS answers are the likeliest casualty. If that shows up, the
fix is a lower server MTU, as a deliberate rollout.
- **Other stale text, outside this brief:** `platform/vpn-ingress/README.md` still says no peer has
ever completed a handshake, and the runbook's status line still calls the VPN inert.
## What I verified
- Upstream `wireguard-android` `GoBackend.java`, master (last commit to that file `ecc3c0a`,
2026-03-15): `builder.setMtu(config.getInterface().getMtu().orElse(1280))`, and `allowFamily()` is
skipped when `sawDefaultRoute && config.getPeers().size() == 1`.
- Live, read-only, 2026-11-09 UTC, on the `k3s-prod-1` pod and agent:
- `wg0` MTU is 1420 on all three pods. Both peers have handshakes and transfer on prod-1 only; the
other two pods read `0`.
- BPF conntrack holds `UDP SVC 10.90.0.x -> 192.168.2.23:53` and flows to `192.168.2.24:32400`.
Their backend entries carry `SourceSecurityID=2` (`world`). So the source stays `10.90.0.x`
through the load balancer on those two VIPs.
- `/proc/net/nf_conntrack` is readable from the pod, which has no `conntrack` binary.
- Offline, pinned toolbox: `kubectl kustomize platform/vpn-ingress` gives identical ConfigMap and
DaemonSet hashes on `origin/main` and on this branch.
- `new-peer.sh` writes `MTU = 1280` by default and `MTU = 1420` with `WG_MTU=1420`. Run with
throwaway keys and stdout discarded; profiles shredded; nothing enrolled.
- No test or conformance check pins the client MTU (`grep` for `WG_MTU` and `1420`).
- Gate (pinned toolbox, from this worktree, on the committed tree): **rc 3, UNDETERMINED, not green.**
`gitea-settings` is the only stage counted as not run (no token locally). `conformance` (L2/L3)
also skipped as the declared no-cluster gap, so R-ENV-01 and R-OBS-04 are not certified.
`spec-immutability`, `approval-policy`, `decisions-log`, `lint`, `l0`, `render-golden` and `l1`
passed. No sensitive path is touched.
## What I did NOT verify
- **That MTU was the cause.** It is inferred from one before-and-after, on one phone and one carrier,
reported by the operator. The path MTU was never measured.
- **That 1280 is the best value.** It is the value that worked, and the IPv6 minimum. A higher one
may work too.
- That the phone's app is on `GoBackend` rather than the kernel backend, so that a blank field meant
1280 on this device. Only the upstream default was read.
- That TCP MSS behaves as described, or that large UDP replies are actually lost. No packet capture
was taken.
- That the Gateway VIP `.20` sees `10.90.0.x` / `world`. Conntrack held no tunnel flows to it when
checked.
- Android's IPv6 blocking on a real device; read from source only.
- `wg-quick` IPv6 behaviour on Linux, and any non-Android client at 1280.
- `/proc/net/nf_conntrack` output during a key mismatch. The `[UNREPLIED]` reading comes from the
earlier measurement in the brief; I saw only healthy entries.
@@ -0,0 +1,103 @@
---
date: 2026-11-08
branch: fix/wg-home-peer-pihole-sources
title: Enrol keith-phone-home and admit the WireGuard subnet to PiHole's DNS frontends
---
## What changed
- **`platform/vpn-ingress/peers.conf`**: new peer `keith-phone-home`, `10.90.0.3/32`, key generated
on the phone. It is the `home` half of a two-tunnel profile (Endpoint `192.168.2.10:51820`),
because the router has no hairpin NAT and `wan.` is unusable from inside the house.
- **`apps/wave-0/pihole-dns.yaml`**: `${WG_CLIENT_CIDR}` added after `${LAN_CIDR}` on both
`pihole-dns-udp` and `pihole-dns-tcp`. This file is hand-written, not rendered (`catalog/render.py`
only lists it in `EXTRA_UNITS`), so it is the source of truth and `--emit-waves` was not run.
`catalog/pihole.yaml` changes in one comment only, so nothing renders differently.
- **New L0 guard `conformance/tests/l0-static/pihole-dns-source-ranges`** (whole-repo scope,
R-DNS-03): every PiHole LoadBalancer Service lists both tokens, and every range resolves through
cluster-vars (base and prod overlay) to RFC1918.
- **`platform/vpn-ingress/README.md`**: corrected "clients arrive SNATed" (false for LoadBalancer
VIPs) and the IPv6 claim (Android's `GoBackend` blocks IPv6 for a single-peer default route);
documented the home/away pattern, why `wan.` fails from inside, and the prod-1 dependency.
Rollout on merge: the `wg-peers` ConfigMap hash changes (`m9kcckcb5t` -> `29mf6k867t`), so the
DaemonSet rolls, `maxUnavailable: 1`, `maxSurge: 0`, one node at a time in an order the controller
chooses. Each roll deletes and recreates that node's `wg0`, so an established session on that node
drops and must re-handshake. `keith-phone` has no session today (see below), so nothing is
interrupted in practice.
**The one risk worth watching:** `wg0` is in Cilium's auto-detected device list and `devices` is
still unpinned. On 2026-10-06 a crash-looping `wg0` churned that list and all three nodes lost their
main-table default route. A rollout is one delete/create per node, not a loop, so it should not
repeat that, but after merge check `ip route get 1.1.1.1` on each node.
The Service change is an in-place update: same VIP, same Service objects, no pod restart.
Checked on the rest of the path and left unchanged:
- Gateway VIP `.20` (`gateway/cilium-gateway-shared`): no source ranges, live. Nothing needed.
- `pihole-allow-lan-dns-ingress` admits `world` on :53, which covers an untranslated 10.90.0.x
source. `platform/vpn-ingress/networkpolicy.yaml` does not apply to a hostNetwork pod.
- No L0 check pinned these ranges to the LAN; none needed updating.
Flagged, not changed (out of scope):
- `runbooks/vpn-remote-access.md` "Known limitations" repeats both corrected claims, and step 3 of
its verification table says the Gateway sees a `host`-identity source.
- `platform/vpn-ingress/daemonset.yaml`'s header repeats the SNAT claim.
- `clusters/base/cluster-vars.yaml`'s substitution-coverage comment does not list
`WG_CLIENT_CIDR` under apps-wave-0. The variable is defined and substituted there regardless.
- `peers.conf` still says "No peers enrolled yet" above the enrolled peers.
- PiHole's NodePorts are not covered by source ranges (already acknowledged in `pihole-dns.yaml`).
Tunnel clients can reach node addresses, so they can reach those too; they are enrolled devices.
- Pinning Cilium's `devices` remains the open hardening for the churn risk above.
## What I verified
Live, read-only, 2026-11-08:
- `cilium-dbg status` on one agent: `KubeProxyReplacement: True [eno1 ..., wg0 10.90.0.1, tun0 ...]`
and `Masquerading: BPF [eno1, wg0, tun0]`. So `wg0` really is a load-balancing device.
- `pihole-dns-tcp` and `-udp` carry `ranges=[192.168.2.0/24]`; no other LoadBalancer Service has
source ranges. Live `cluster-vars`: `WG_CLIENT_CIDR=10.90.0.0/24`, `LAN_CIDR=192.168.2.0/24`.
- `wg show wg0 latest-handshakes` on all three pods: `keith-phone` = `0`.
- `wan.home.keithjackson.dev` resolves publicly to a single A record (via 1.1.1.1).
Offline, in the pinned toolbox:
- `flux envsubst --strict` renders both Services to `[192.168.2.0/24, 10.90.0.0/24]`.
- `kubectl kustomize platform/vpn-ingress`: `wg-peers-m9kcckcb5t` on the old `peers.conf`,
`wg-peers-29mf6k867t` on the new one.
- Guard mutation test, each restored afterwards. Baseline passes (22 assertions, 2 Services). All
seven fail with rc 1 and a named message: WG range dropped from one Service; `0.0.0.0/0` added;
`8.8.8.0/24` added; `WG_CLIENT_CIDR` made public in cluster-vars; prod overlay sets `LAN_CIDR` to
`0.0.0.0/0`; ranges block removed; undefined token added.
- The Android claim was read from upstream `GoBackend.java` (`wireguard-android`, master): the
`allowFamily` calls are skipped when `sawDefaultRoute && config.getPeers().size() == 1`.
- Gate (pinned toolbox, from this worktree, on the committed tree): **rc 3, UNDETERMINED, not
green.** `gitea-settings` is the only stage counted as not run (no token locally). `conformance`
(L2/L3) also skipped as the declared no-cluster gap. `spec-immutability`, `decisions-log`, `lint`,
`l0` (including the new guard), `render-golden` and `l1` passed. `approval-policy` reported N/A
offline; it will be red in CI until the operator overrides, because `conformance/**` is sensitive.
- `git merge-tree` against `feat/pihole-explicit-lan-names` (#157): no conflicts.
## What I did NOT verify
- **`DROP_NOT_IN_SRC_RANGE` was NOT observed live.** It is a prediction from Cilium's source. Only
the `wg0` device membership was measured; the drop itself has never been seen.
- **Full-tunnel behaviour was NOT observed live**, on either profile. **No handshake has ever been
recorded for any peer**: `keith-phone` reads `0` on all three pods (a counter that resets when a
pod restarts, last 22 days ago, so it covers only the time since; earlier entries here also
record none).
- That tunnel DNS works after this change, including the reply path from a backend on another node
and from one on the ingress node. That needs the phone: `dig` over the tunnel, while watching
`cilium-dbg monitor --type drop`.
- That the Gateway serves tunnel clients. Its source is predicted to be `world`, as for a LAN
browser, which is what the apps admit; not tested.
- That the new peer's key is the one on the phone. I copied the public key from the brief.
- That the rollout leaves node default routes intact. Reasoned from the 2026-10-06 incident being
a loop, not measured.
- That a LAN client is unaffected while the Service's source ranges are updated. Expected, not
measured.
- IPv6 behaviour on any non-Android client.
- That the WAN forward targets prod-1: it is off-repo.
+9 -1
View File
@@ -53,6 +53,7 @@
# WG_ENDPOINT_HOST default wan.home.keithjackson.dev
# WG_CLIENT_CIDR default 10.90.0.0/24 (must match clusters/base/cluster-vars.yaml)
# WG_LAN_CIDR default 192.168.2.0/24 (prod LAN_CIDR)
# WG_MTU default 1280 (client tunnel MTU; see the comment on the default below)
# WG_DNS default 192.168.2.23 (prod PIHOLE_DNS_VIP)
#
# REQUIRES: `wg`, `qrencode`, `sops` — all three in the dev container.
@@ -80,7 +81,14 @@ PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.conf"
# cannot resolve until it is already connected.
: "${WG_ENDPOINT_HOST:=wan.home.keithjackson.dev}"
: "${WG_LISTEN_PORT:=51820}"
: "${WG_MTU:=1420}"
# The CLIENT's tunnel MTU. Deliberately lower than the server's 1420 (daemonset.yaml), which only
# describes the 1500-byte path at the node's end. A phone's path is the carrier's, often behind
# carrier NAT, and is not known to carry 1500. Measured 2026-11-09, one Android phone on one
# carrier: at 1420 the handshake was healthy but names and pages failed; clearing the app's MTU
# field (its userspace backend then uses 1280) fixed it. 1280 is also the IPv6 minimum link MTU, so
# it stays valid if the tunnel ever gains IPv6. For a device only ever on a wired or known-1500
# path, WG_MTU=1420 recovers about 10% of per-packet payload.
: "${WG_MTU:=1280}"
SPLIT=0
OUT_DIR=""
+3 -3
View File
@@ -217,9 +217,9 @@ Consequences, stated plainly:
**Governance:** ADR-0015 and `docs/03` are specifications and are immutable to agents
(spec `AGENTS.md` §11, R-TEST-06, enforced by `ci/spec-immutability.sh`) — they are **not** edited
here. The correction is raised as an **escalation with a drafted ADR-0016 supersede** in
`DECISIONS-LOG.md` (entry of 2026-09-06) for the operator/architect to accept. Until then this file
and `egress-steering.yaml` are the accurate statement of what the implementation can actually do.
here. The correction, first escalated in `DECISIONS-LOG.md` (entry of 2026-09-06), is recorded in
spec **ADR-0020**, which supersedes ADR-0015's Kill-switch and Resilience bullets. This file and
`egress-steering.yaml` state what the implementation does.
## Secrets — host-side now
PIA credentials + the `.ovpn` live **on the gateway node via Ansible vault** (see spec `docs/07`), **not**
+1 -1
View File
@@ -109,7 +109,7 @@ spec:
# that shard and re-shards the rest (breaking live connections), and each shard exits from a
# DIFFERENT public IP. Upstream: cilium/cilium#30157, cilium/cilium#18230.
# ADR-0015's "Resilience: two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)"
# is therefore NOT IMPLEMENTABLE on Cilium OSS — escalated for an ADR-0016 supersede; see
# is therefore NOT IMPLEMENTABLE on Cilium OSS — superseded by spec ADR-0020; see
# platform/vpn-egress/README.md and the DECISIONS-LOG entry of 2026-09-06.
#
# `interface: tun0` IS RESOLVED ONCE — the gateway node has to tell Cilium when tun0 changes.
+54 -12
View File
@@ -42,8 +42,9 @@ no version string to match against a CVE, and no amplification to abuse.
That is the whole security argument, and it is why the exposed port carries no IP allowlist (which
would also have made roaming clients unusable — a phone's address changes with every cell tower).
Compare `apps/wave-0/pihole-dns.yaml`, which *does* pin `loadBalancerSourceRanges` to the LAN: a
resolver answers anybody, so it must be scoped by address. WireGuard answers nobody.
Compare `apps/wave-0/pihole-dns.yaml`, which *does* pin `loadBalancerSourceRanges` to the LAN and
this tunnel's client subnet: a resolver answers anybody, so it must be scoped by address. WireGuard
answers nobody.
What it does **not** protect against is a stolen device — anyone holding a device's private key
*is* that device. That is what revocation is for: delete the stanza, merge, done (~1 min).
@@ -55,9 +56,9 @@ What it does **not** protect against is a stolen device — anyone holding a dev
┌──────────────┐ udp/51820 ┌───────────────┐ forward ┌────────────────────────────┐
│ WireGuard app│─────────────▶│ WAN 51820 ──▶ │───────────▶│ wg0 10.90.0.1/24 │
│ 10.90.0.2/32 │ over the │ node IP:51820 │ │ decrypt, then FORWARD │
└──────────────┘ internet └───────────────┘ │ SNAT to the node's addr │
└──────────────┘ internet └───────────────┘ │ SNAT (LAN), eBPF LB (VIPs)│
└─────────────┬──────────────┘
│ as `host`/`remote-node`
│ translated per destination (below)
┌─────────────────────────────────┼──────────────────────┐
▼ ▼ ▼
LAN 192.168.2.0/24 Gateway VIP .20 (16 web UIs) WAN / internet
@@ -86,9 +87,21 @@ resolved, because `apps/wave-0/pihole-dns.yaml`'s ingress policy deliberately om
entity. A VPN that looks up and is useless is the worst available outcome, and nothing in the
manifests would have hinted at it.
In the **host** netns there is no pod identity to carry. Forwarded client traffic is SNATed to the
node's own address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` —
byte-for-byte what a LAN client looks like. Three further problems vanish with it:
In the **host** netns there is no pod identity to carry, which is what avoids the 403s. Clients do
**not**, however, all arrive SNATed to the node. Measured 2026-11-08: Cilium counts `wg0` among its
kube-proxy-replacement devices (`cilium-dbg status`: `eno1`, `wg0`, `tun0`), so what a packet looks
like depends on where it is going:
| destination | handled by | source the destination sees |
|---|---|---|
| LAN hosts, the internet | kernel FORWARD, then the iptables MASQUERADE in [`daemonset.yaml`](./daemonset.yaml) | the node's address |
| LoadBalancer VIPs (Gateway `.20`, PiHole `.23`, ...) | Cilium's eBPF load balancer, as the packet enters from `wg0`, **before** MASQUERADE | `10.90.0.x` at the VIP's source-range check, identity `world` |
The second row is why PiHole's `loadBalancerSourceRanges` has to list `${WG_CLIENT_CIDR}` as well as
the LAN: left LAN-only, tunnel DNS is predicted to die with `DROP_NOT_IN_SRC_RANGE`. That prediction
and the row's source column come from reading Cilium's datapath source; **no peer has ever completed
a handshake, so neither has been observed live.** The Gateway VIP carries no source ranges and needs
no change. Three further problems vanish with the host netns:
* `net.ipv4.ip_forward` is **already 1** on these nodes (`node_baseline` writes
`/etc/sysctl.d/99-k3s-cilium.conf`). It is *not* in the kubelet's safe-sysctl allowlist, so a
@@ -99,7 +112,9 @@ byte-for-byte what a LAN client looks like. Three further problems vanish with i
`conformance/tests/l0-static/l2-announced-vips-serve-from-any-node`) and no
`externalTrafficPolicy` trade-off to make at all.
* **No double encapsulation.** Client UDP lands on the node's own NIC, so the underlay is a flat
1500-byte path, not Cilium's 1450-byte VXLAN. That is what makes `MTU = 1420` simply correct.
1500-byte path, not Cilium's 1450-byte VXLAN. That is what makes `MTU = 1420` correct on the
server's `wg0`. It says nothing about a client's own path: `new-peer.sh` defaults client profiles
to `1280` (see "Troubleshooting" in `runbooks/vpn-remote-access.md`).
`hostNetwork` is *permitted* here, and not by oversight:
`platform/kyverno/policies/restrict-host-namespaces.yaml` matches `namespaceSelector:
@@ -163,6 +178,30 @@ internet path. To enforce less *server-side*, add filtering to the `PostUp` chai
before the accept rules) — the FORWARD chain there is the only enforcement point, since no
NetworkPolicy applies.
## Phone profiles: one tunnel for home, one for away
A phone that needs the LAN-only apps runs full tunnel (`AllowedIPs = 0.0.0.0/0`, `DNS = 192.168.2.23`)
everywhere, so it needs **two tunnels**, each its own peer with its own key and address:
| tunnel | peer | address | Endpoint | use when |
|---|---|---|---|---|
| `away` | `keith-phone` | `10.90.0.2` | `wan.home.keithjackson.dev:51820` | off the home network |
| `home` | `keith-phone-home` | `10.90.0.3` | `192.168.2.10:51820` | on the home Wi-Fi |
**`wan.` cannot work from inside the house.** It is an unproxied A record holding the house's WAN
address (`platform/externaldns/README.md`), and the router has no hairpin NAT, so a LAN client that
sends to it never comes back in through the port-forward. The Cloudflare Tunnel is no way round: it
carries HTTP(S) to the Gateway and WireGuard is UDP to a node, so `wan.` has to stay DNS-only and
`away` has to keep using it.
**`home` depends on `k3s-prod-1`.** Its Endpoint is that node's address, with no failover: if
prod-1 is down, `home` cannot handshake even though the other two listeners are healthy. The fix
is client-side: edit the tunnel's Endpoint to `192.168.2.11` or `.12`, since every node answers the
same keys. No server change and no new key is needed. The same dependency applies to `away` if the
WAN forward points at prod-1.
Android allows one active VPN at a time, so switch tunnels by hand when leaving or arriving.
## Operating it
* **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
@@ -215,10 +254,13 @@ right way; one was a defect in my own manifest.
handshake — and would be a real secret, forcing `peers.conf` into SOPS and costing the
reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
`wg-server-key` Secret.
* **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6
still reaches IPv6 sites *outside* the tunnel. Functionally harmless, a privacy leak if the point
was to hide the client's network. Fix is an `fd..` ULA on `wg0` plus `::/0` in the client's
`AllowedIPs`; not done because nothing here needs it.
* **No IPv6.** The server offers no IPv6 tunnel address and `new-peer.sh` emits only `0.0.0.0/0`.
What happens to a client's IPv6 then depends on the client. The **Android app does not leak it**:
its `GoBackend` skips `allowFamily()` when the config is one peer with a default route, so Android
blocks IPv6 for the VPN's duration and apps fall back to IPv4. Other clients are not covered by
that code; expect `wg-quick` on a dual-stack Linux host to leave IPv6 outside the tunnel. Fix is
an `fd..` ULA on `wg0` plus `::/0` in the client's `AllowedIPs`; not done because nothing here
needs it.
* **No in-cluster-service reach beyond what the LAN has.** Clients reach cluster apps the way a LAN
browser does — through the Gateway VIP. Pod IPs and `10.43.0.0/16` service IPs are not routed
into the tunnel.
+6 -3
View File
@@ -30,9 +30,12 @@
# `cluster` entity, so VPN clients would not have been able to resolve `*.${BASE_DOMAIN}` either.
# The VPN would have looked up and been useless, which is the worst failure shape available.
#
# In the host netns there is no pod identity to carry: forwarded client traffic is SNATed to the
# NODE's address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` —
# exactly what a LAN client looks like. Three further problems disappear with it:
# In the host netns there is no pod identity to carry, so a client never looks like a pod. What a
# destination sees depends on how the packet is handled. LAN hosts and the internet: forwarded,
# then MASQUERADEd to the NODE's address by the PostUp rule below. LoadBalancer VIPs: Cilium's load
# balancer takes the packet as it enters from wg0, before that rule, so the source is still
# 10.90.0.x with identity `world`, like a LAN browser (measured 2026-11-09 for PiHole .23 and Plex
# .24; see README). Three further problems disappear with it:
# * net.ipv4.ip_forward is ALREADY 1 on these nodes (node_baseline writes it to
# /etc/sysctl.d/99-k3s-cilium.conf), so no pod sysctl is needed — and `net.ipv4.ip_forward`
# is NOT in the kubelet's safe-sysctl allowlist, so a pod-netns version of this would have
+4
View File
@@ -90,3 +90,7 @@
[Peer]
PublicKey = pBRSHrRyG2hnsuZ6FvHfEzF8hOVTF5stHUb/7fW2Km8=
AllowedIPs = 10.90.0.2/32
# keith-phone-home — <owner>, enrolled 2026-11-08 (at-home profile, LAN Endpoint; no hairpin NAT)
[Peer]
PublicKey = AT8lBJHqrVkv8RL9n0qWDsh5nyl0e3qgrfTAaYKgBz0=
AllowedIPs = 10.90.0.3/32
+98 -12
View File
@@ -252,7 +252,7 @@ cellular, with Wi-Fi off, and check all four — each proves a different hop:
|---|---|---|
| 1 | tunnel shows a recent handshake in the WireGuard app | the WAN forward + the server key + the peer stanza |
| 2 | `ping 192.168.2.1` (the router) | decrypt → FORWARD → LAN, and the SNAT return path |
| 3 | open `https://plex.home.keithjackson.dev` | PiHole DNS over the tunnel **and** the Gateway accepting a `host`-identity source — the specific thing a pod-networked VPN would have 403'd |
| 3 | open `https://plex.home.keithjackson.dev` | PiHole DNS over the tunnel **and** the Gateway admitting a tunnel client, which a pod-networked VPN would have 403'd. Expected source identity is `world`, like a LAN browser: measured 2026-11-09 on VIPs `.23` and `.24`, not yet observed on the Gateway's `.20` |
| 4 | visit an IP-echo site; it should show `136.51.115.252` | full-tunnel WAN egress out the house, not the carrier |
Then confirm server-side:
@@ -260,18 +260,97 @@ Then confirm server-side:
```bash
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show'
# a `latest handshake` under ~3 min, and non-zero transfer, on the expected peer.
# a `latest handshake` under ~3 min, and non-zero transfer, on the expected peer. `ds/wireguard`
# picks ONE pod; if it reads 0, ask the pod on the node holding the forward (see Troubleshooting).
```
**If check 3 fails while 2 passes**, the tunnel is fine and the problem is DNS or the Gateway, not
WireGuard. Check the client's `DNS = 192.168.2.23` and that PiHole is up.
**If 1 and 2 pass but large transfers hang**, it is MTU. The profile ships `MTU = 1420` (1500 − 60
bytes of WireGuard encapsulation); drop the client to `1280` and re-test.
**If 1 and 2 pass but pages or large transfers hang**, suspect the client MTU; see
Troubleshooting.
**If nothing handshakes at all**, you cannot tell a wrong forward from a wrong key from outside —
by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes
if the router is not delivering the packets, and a `handshake` with no completed session if it is.
**If nothing handshakes at all**, you cannot tell a wrong forward from a wrong key from the
device, by design. `wg show` cannot tell them apart either: it reads the same in both cases. The
node's conntrack table can; see Troubleshooting.
## Troubleshooting
Ask the pod on **the node carrying the traffic**: the node the WAN forward names for `away`,
`k3s-prod-1` for `home`. Sessions are per node, and the other two pods read `0`.
```bash
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n vpn-ingress get pods -o wide'
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n vpn-ingress exec <pod> -c wireguard -- wg show wg0 latest-handshakes'
# likewise: `wg show wg0 endpoints`, `wg show wg0 transfer`
```
**Use only those three subcommands to diagnose.** Never `wg showconf`, `wg show wg0 private-key` or
`wg show wg0 preshared-keys`: they print key material into your terminal and transcript.
| output | reading |
|---|---|
| `latest-handshakes`: `<public key> 0` | no handshake with this peer since the pod started |
| `endpoints`: `<public key> (none)` | the server has not accepted a packet from this peer since the pod started |
| `transfer`: `<public key> <rx> <tx>` | bytes, as the server counts them |
### The app shows TX, no RX and no "Latest handshake"
**Symptom.** The app's transmit counter climbs and receive stays at zero. On the server,
`endpoints` is `(none)` and `latest-handshakes` is `0` for that peer.
**Cause.** A key mismatch: the server's public key in the profile, or the device's public key in
`peers.conf`. WireGuard drops an initiation it cannot authenticate and sends nothing back, so
neither end logs an error. Measured 2026-11-09: a mistyped server public key in the phone's profile.
**Check.** First prove the packets arrive. The pod has no `conntrack` binary, but in `hostNetwork`
it can read the host's table:
```bash
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n vpn-ingress exec <pod> -c wireguard -- grep dport=51820 /proc/net/nf_conntrack'
```
- A line whose first tuple is `src=<the phone's public address> ... dport=51820`, marked
`[UNREPLIED]`: the datagrams reach the node and the server stays silent. It is the keys.
- No such line: the packets are not arriving. Look at the router forward, the DNS name, or the
device's network.
Then compare the keys **by copy and paste, never by eye and never retyped**: the server's public
half (the first command in Step 2's "Record it in your PASSWORD MANAGER") against the profile's
`PublicKey`, and the device's public key from the app against its stanza in `peers.conf`.
### Works on Wi-Fi, stalls on mobile data, handshake healthy
**Symptom.** `latest-handshakes` is recent and some traffic flows, but names or pages fail on
mobile data. The same device is fine on Wi-Fi.
**Cause.** Most likely a client MTU too large for the carrier's path: a tunnel MTU of `1420` means
outer packets of up to 1480 bytes. This is inferred from one before-and-after, on one phone and one
carrier (2026-11-09): `1420` failed and clearing the field fixed it. The path MTU itself was not
measured.
**Check.** In the Android WireGuard app, edit the tunnel, clear the `MTU` field or set it to `1280`,
save, then turn the tunnel off and on. A blank field means `1280` in the app's default userspace
backend (`GoBackend`: `getMtu().orElse(1280)`). On other clients set `MTU = 1280` in the profile.
`new-peer.sh` now generates `MTU = 1280` unless `WG_MTU` overrides it. A profile already on a device
does not change, so edit it by hand. If `1280` does not help, it is not the MTU: go back to
Step 6's check 3.
### The `away` profile cannot connect from home
**Symptom.** On the home network, `away` never handshakes.
**Cause.** Its Endpoint is `wan.home.keithjackson.dev`, the house's WAN address, and the router has
no hairpin NAT. WireGuard cannot use the Cloudflare Tunnel instead.
**Check.** Turn Wi-Fi off. If `away` handshakes on mobile data, the profile is fine: use the `home`
tunnel indoors, as described in
[Phone profiles: one tunnel for home, one for away](../platform/vpn-ingress/README.md#phone-profiles-one-tunnel-for-home-one-for-away).
## Revoking a device
@@ -293,10 +372,17 @@ top follow-up for this component.** Until then:
## Known limitations
* **No IPv6 in the tunnel.** A full-tunnel client with native IPv6 still reaches IPv6 sites outside
the tunnel. Functionally harmless; a privacy leak if hiding the client's network was the point.
* **Client IPs are not preserved** on the LAN side: everything arrives SNATed as the node's
address. A static route for `10.90.0.0/24` → the forwarded node, added on the router, would
preserve them — at the cost of another hand-maintained off-repo route.
* **No IPv6 in the tunnel.** What a client's IPv6 does then depends on the client. In the Android
app's default `GoBackend`, a config with one peer and a default route skips `allowFamily()`, so
Android should block IPv6 while the tunnel is up and apps fall back to IPv4. That is read from
upstream source, not observed on a device. A `--split` profile has no default route, so its IPv6
stays outside the tunnel. Expect `wg-quick` on a dual-stack Linux host to leave IPv6 outside too
(not tested).
* **Client IPs are only partly preserved.** LAN hosts and the internet see the node's address
(MASQUERADE). A static route for `10.90.0.0/24` → the forwarded node, added on the router, would
preserve them there, at the cost of another hand-maintained off-repo route. LoadBalancer VIPs are
different: Cilium handles the packet before MASQUERADE, so the source stays `10.90.0.x`, identity
`world` (BPF conntrack on prod-1, 2026-11-09, VIPs `.23` and `.24`). Any `loadBalancerSourceRanges`
must therefore list the tunnel subnet, as PiHole's do.
* **Rollouts briefly drop the forwarded node.** `maxUnavailable: 1`, so one node at a time; clients
re-handshake automatically (retry every 5s, rekey every 2 min).