Compare commits

..
Author SHA1 Message Date
jakman85andClaude Opus 5.5 8cd5ab2e41 chore(spec): adopt the spec sync: registry parity and ADR cross-references
gate / gate (pull_request) Failing after 1m4s
gate / conformance (pull_request) Skipped
Add R-DNS-04, R-DNS-05 and R-SEC-04 to the requirement mirror and tag the
L0 tests that verify them. Align the R-ENV and R-UPD-02 summaries with the
amended requirements. Point the VPN egress docs at ADR-0020; the drafted
"ADR-0016" number went to the Cloudflare Tunnel.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 22:47:54 -06:00
jakman85 50c8719b12 Merge pull request 'fix(vpn): default client profiles to MTU 1280; correct the VPN runbook' (#159) from fix/wg-client-mtu-docs into main
gate / gate (push) Successful in 57s
gate / conformance (push) Skipped
Reviewed-on: #159
2026-11-09 03:26:02 +00:00
jakman85andClaude Opus 5.5 8f60e6103a fix(vpn): default client profiles to MTU 1280; correct the VPN runbook
gate / gate (pull_request) Successful in 57s
gate / conformance (pull_request) Skipped
new-peer.sh generated MTU 1420, which stalled traffic on mobile data
despite a healthy handshake (one phone, one carrier, 2026-11-09). Default
to 1280, overridable through WG_MTU.

Correct stale claims in the runbook and the DaemonSet header against
live BPF conntrack: LoadBalancer VIPs see 10.90.0.x as world, not a
SNATed host identity. Add a troubleshooting section.

The DaemonSet render is unchanged, so WireGuard pods do not roll.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 21:15:20 -06:00
jakman85 da0515c8c8 Merge pull request 'fix(vpn): enrol keith-phone-home and admit the WireGuard subnet to PiHole DNS' (#158) from fix/wg-home-peer-pihole-sources into main
gate / gate (push) Successful in 55s
gate / conformance (push) Skipped
Reviewed-on: #158
2026-11-09 02:50:04 +00:00
jakman85 3a2ec5a12c Merge pull request 'feat(dns): PiHole answers an explicit LAN name list, not a zone wildcard' (#157) from feat/pihole-explicit-lan-names into main
gate / gate (push) Canceled after 25s
gate / conformance (push) Canceled after 0s
Reviewed-on: #157
Reviewed-by: Keith Jackson <[email protected]>
2026-11-09 02:49:39 +00:00
jakman85andClaude Opus 5.5 8fe8d5df8a feat(dns): PiHole answers an explicit LAN name list, not a zone wildcard
gate / gate (pull_request) Successful in 1m14s
gate / conformance (pull_request) Skipped
Public names are proxied CNAMEs to the Cloudflare Tunnel since PR #156 and
work from the LAN, so they should resolve publicly there too. Replace the
address=/${BASE_DOMAIN}/ wildcard with explicit entries: the LAN-only names
(derived from every controller:none HTTPRoute), gitea/nextcloud (off
Cloudflare's 100 MB / 100 s limits), keycloak/auth, and the L4 VIPs.

Add L0 check pihole-lan-names, mutation-tested. Correct docs that still
described public names as CNAME to wan. and the cloudflared entrypoint comment.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 19:51:49 -06:00
27 changed files with 776 additions and 87 deletions

No files matched your search

+1 -1
View File
@@ -76,7 +76,7 @@ bug but the honest answer, because proofs are claims about a specific tree on a
|---|---|
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
| 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; **now 3× mini-PC bare metal** (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13) |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the authoritative LAN resolver** for `${BASE_DOMAIN}` on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the LAN resolver**, answering an explicit list of `${BASE_DOMAIN}` names (LAN-only apps, gitea/nextcloud/keycloak/auth, the L4 VIPs — everything else resolves publicly to Cloudflare, since 2026-11-09) on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled |
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |
+2 -2
View File
@@ -176,8 +176,8 @@ or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:
**UI access:** `https://unifi.home.keithjackson.dev:11443`. That name resolves to the VIP and not to
the Gateway on purpose: both `platform/coredns/coredns-custom.yaml` (in-cluster) and
`apps/wave-0/pihole.yaml` (LAN split-horizon) carve `unifi.${BASE_DOMAIN}` out of the
zone-wide `${GATEWAY_VIP}` answer and point it at `${UNIFI_L4_VIP}`. `${BASE_DOMAIN}` on prod is the
`apps/wave-0/pihole.yaml` (LAN split-horizon, an explicit name list since 2026-11-09) point
`unifi.${BASE_DOMAIN}` at `${UNIFI_L4_VIP}` rather than `${GATEWAY_VIP}`. `${BASE_DOMAIN}` on prod is the
apex `home.keithjackson.dev` (confirmed live in the `cluster-vars` ConfigMap) after the 2026-09-13
flip, so this hostname is current, not a dev leftover.
+1 -1
View File
@@ -73,7 +73,7 @@ spec:
- name: FTLCONF_dns_rateLimit_count
value: '0'
- name: FTLCONF_misc_dnsmasq_lines
value: local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
value: local-ttl=60;address=/gitea.${BASE_DOMAIN}/${GATEWAY_VIP};address=/nextcloud.${BASE_DOMAIN}/${GATEWAY_VIP};address=/keycloak.${BASE_DOMAIN}/${GATEWAY_VIP};address=/auth.${BASE_DOMAIN}/${GATEWAY_VIP};address=/headlamp.${BASE_DOMAIN}/${GATEWAY_VIP};address=/jackett.${BASE_DOMAIN}/${GATEWAY_VIP};address=/pihole.${BASE_DOMAIN}/${GATEWAY_VIP};address=/plex.${BASE_DOMAIN}/${GATEWAY_VIP};address=/qbittorrent.${BASE_DOMAIN}/${GATEWAY_VIP};address=/radarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/semaphore.${BASE_DOMAIN}/${GATEWAY_VIP};address=/sonarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
- name: FTLCONF_webserver_api_password
valueFrom:
secretKeyRef:
+57 -30
View File
@@ -108,41 +108,61 @@ env:
# `loadBalancer.mode: dsr`, which preserves the client address.
- name: FTLCONF_dns_rateLimit_count
value: "0"
# THE SPLIT-HORIZON ANSWERS. A dnsmasq `address=/<domain>/<ip>` entry answers that domain AND
# every subdomain of it, and dnsmasq resolves overlapping entries by LONGEST MATCHING DOMAIN —
# so the first line is a wildcard for the whole zone and each later line is a specific override.
# PRECEDENCE IS THEREFORE STRUCTURAL, not order-dependent: `unifi.${BASE_DOMAIN}` matches both
# line 1 and line 2, line 2 is the longer domain, line 2 wins. Rewriting these in any order
# changes nothing.
# THE LAN ANSWERS — AN EXPLICIT LIST, NOT A ZONE WILDCARD (2026-11-09). A dnsmasq
# `address=/<domain>/<ip>` entry answers that domain AND every subdomain of it (on LABEL
# boundaries: `pihole.` does not match `pihole-dns.`), and dnsmasq resolves overlapping entries by
# LONGEST MATCHING DOMAIN, so the order below is irrelevant. Every name NOT listed here is
# FORWARDED to the upstreams above and gets the PUBLIC answer.
#
# A NEWLY ONBOARDED SERVICE NEEDS NO CHANGE HERE. Every catalog ingressRoute host is
# `<label>.${BASE_DOMAIN}` served by the shared Gateway, so the wildcard already answers it with
# ${GATEWAY_VIP} — the same guarantee, and the same reasoning, as the `template IN A` block in
# platform/coredns/coredns-custom.yaml. Add a line here ONLY for a service that does not live
# behind the Gateway: an L4 LoadBalancer on its own pinned VIP. Those are enumerated from the
# repo, not from memory — grep `external-dns.alpha.kubernetes.io/hostname` on a Service, which
# is the annotation that means "this Service publishes its own hostname at its own address":
# apps/vms/unifi/50-service-l4.yaml (unifi -> ${UNIFI_L4_VIP}) and
# platform/crafty/service-l4.yaml (crafty-mc -> ${CRAFTY_L4_VIP}). Those two, plus PiHole's own
# resolver address, are the complete set of exceptions today.
# WHY NO WILDCARD ANY MORE. Until 2026-11-09 the first entry was `address=/${BASE_DOMAIN}/
# ${GATEWAY_VIP}`, answering the whole zone. That made sense while every public record was a
# DNS-only CNAME to `wan.${BASE_DOMAIN}` (no NAT hairpin on this router, so the LAN NEEDED a
# private answer). Since PR #156 every public name is a PROXIED CNAME to the Cloudflare Tunnel
# (platform/externaldns/README.md), which works identically from the LAN — so a name that is
# public now SHOULD resolve publicly from the LAN too, and the LAN no longer depends on PiHole for
# it: a client that bypasses PiHole (DoH browser, phone private DNS, the router's fallback) gets
# the same working answer. The wildcard had also become actively misleading: it answered names
# that exist nowhere (`wan.` itself resolved to the Gateway VIP on the LAN).
#
# AAAA needs nothing: dnsmasq answers a AAAA query for an `address=`d name with NODATA
# (NOERROR, no answer), so IPv6-capable clients fall back to the A record instead of getting
# NXDOMAIN — the same behaviour coredns-custom hand-writes a `template IN AAAA` block for.
# WHAT IS LISTED, and why each — enforced by conformance/tests/l0-static/pihole-lan-names:
# -> ${GATEWAY_VIP}, LAN-ONLY: every HTTPRoute disowned from public DNS
# (`external-dns.alpha.kubernetes.io/controller: none`, i.e. catalog `exposure: internal`,
# plus plex and headlamp) — jackett, pihole, plex, qbittorrent, radarr, semaphore, sonarr,
# headlamp. These have NO public record and the tunnel 404s them, so THIS LINE IS THEIR ONLY
# PATH: drop one and the app is unreachable everywhere. A NEWLY ONBOARDED `internal` SERVICE
# MUST BE ADDED HERE (the L0 check fails until it is). A new `public` service needs nothing.
# -> ${GATEWAY_VIP}, PUBLIC BUT KEPT ON THE LAN PATH: gitea (git push, LFS, container registry)
# and nextcloud (uploads) — Cloudflare's proxy caps a request body at 100 MB and a response
# at 100 s, which a large push or upload exceeds; keycloak and auth (oauth2-proxy) — so the
# OIDC login of those two LAN-path apps stays on the same path as the apps themselves.
# -> their own L4 VIPs: unifi (${UNIFI_L4_VIP}) and crafty-mc (${CRAFTY_L4_VIP}) are not behind
# the Gateway (grep `external-dns.alpha.kubernetes.io/hostname` on a Service:
# apps/vms/unifi/50-service-l4.yaml, platform/crafty/service-l4.yaml); their public records
# are DNS-only A records to the same RFC1918 VIPs, which this router's rebind protection
# strips, so the LAN answer is still required. pihole-dns (${PIHOLE_DNS_VIP}) is PiHole's own
# resolver address; plex-vip (${PLEX_L4_VIP}) is below.
# NOT listed, deliberately, so they resolve via Cloudflare from the LAN as from anywhere: the apex,
# grafana, crafty, seerr. In-cluster (platform/coredns/coredns-custom.yaml) and node
# (/etc/hosts pins, node_baseline_hosts_pins) resolution are separate and unchanged.
#
# AAAA: dnsmasq answers a AAAA query for an `address=`d name with NODATA (NOERROR, no answer),
# so IPv6-capable clients fall back to the A record instead of getting NXDOMAIN — the same
# behaviour coredns-custom hand-writes a `template IN AAAA` block for. Unlisted (proxied) names
# now get Cloudflare's real A and AAAA answers.
#
# Array-valued FTLCONF variables are SEMICOLON-separated. A JSON-looking list is accepted and
# then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal
# characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid"
# while resolution for the zone does not work. Confirmed both ways against the live image.
# plex-vip -> ${PLEX_L4_VIP} (added 2026-10-04) is the LAN HALF of Plex's single advertised
# hostname. Plex publishes ONE URL to plex.tv for both local and remote clients
# (catalog/plex.yaml ADVERTISE_IP): on the LAN this answer points it at the tcp/32400
# LoadBalancer, while Cloudflare answers the same name publicly with the WAN address, NATed to
# the same VIP. WITHOUT THIS LINE the entry above answers it with ${GATEWAY_VIP} via the zone
# wildcard, where NOTHING listens on 32400 — measured 2026-10-04, plex-vip.${BASE_DOMAIN}
# resolved to the Gateway VIP and a connect to :32400 there returned http=000. That is a silent
# failure: the Plex web UI on plex.${BASE_DOMAIN} keeps working through the Gateway on 443 while
# every client that takes the ADVERTISED connection cannot reach the server at all.
#
# plex-vip -> ${PLEX_L4_VIP} (added 2026-10-04) is the name of the tcp/32400 LoadBalancer
# (apps/wave-0/plex-remote.yaml) and the domain of the certificate Plex serves there
# (catalog/plex.yaml CERTDOMAIN / customCertificateDomain, Secret `plex-vip-tls`). It is NOT what
# Plex advertises any more (ADVERTISE_IP is a resolver-independent plex.direct name since
# 2026-10-05) and it has NO public record, so this entry is its only resolution — kept for
# reaching the VIP by name with a matching certificate from a LAN shell. Measured 2026-10-04,
# before it existed the zone wildcard answered it with ${GATEWAY_VIP}, where nothing listens on
# 32400; without it now, the name is NXDOMAIN.
#
# ── local-ttl=60 — THE SPLIT-HORIZON ANSWER MUST BE CACHEABLE ────────────────────────────────
# MEASURED 2026-10-18, diagnosing "DNS flapping between the cluster ingress and the public IP":
@@ -172,10 +192,17 @@ env:
# cache expires, the router falls back, and the public answer latches again. That is a real
# outage, and the fix for it is PiHole availability, not DNS caching.
#
# SINCE 2026-11-09 THE LATCH IS MOSTLY HARMLESS. The measurement above predates the tunnel: the
# public answer was then `wan.`, unreachable from the LAN. Now the public answer for every
# proxied name is Cloudflare, which works from the LAN, and those names are not listed below at
# all. A latch still bites only the listed names: gitea/nextcloud/keycloak/auth fall back to
# Cloudflare (working, but under its 100 MB / 100 s limits); the LAN-only names and the L4 VIPs
# get NXDOMAIN or a stripped answer until the router re-queries PiHole.
#
# Applies to every `address=` reply below (dnsmasq: local-ttl governs replies made from
# configuration, /etc/hosts and DHCP leases), so one directive covers all five names.
# configuration, /etc/hosts and DHCP leases), so one directive covers every listed name.
- name: FTLCONF_misc_dnsmasq_lines
value: "local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
value: "local-ttl=60;address=/gitea.${BASE_DOMAIN}/${GATEWAY_VIP};address=/nextcloud.${BASE_DOMAIN}/${GATEWAY_VIP};address=/keycloak.${BASE_DOMAIN}/${GATEWAY_VIP};address=/auth.${BASE_DOMAIN}/${GATEWAY_VIP};address=/headlamp.${BASE_DOMAIN}/${GATEWAY_VIP};address=/jackett.${BASE_DOMAIN}/${GATEWAY_VIP};address=/pihole.${BASE_DOMAIN}/${GATEWAY_VIP};address=/plex.${BASE_DOMAIN}/${GATEWAY_VIP};address=/qbittorrent.${BASE_DOMAIN}/${GATEWAY_VIP};address=/radarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/semaphore.${BASE_DOMAIN}/${GATEWAY_VIP};address=/sonarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
# THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
# apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
# carries the full rationale and the measurement that found it.
+16 -4
View File
@@ -120,6 +120,14 @@ requirements:
doc: docs/09-dns-and-ingress.md
safety_critical: false
summary: "Internal clients resolve to the internal Gateway (split-horizon)."
- id: R-DNS-04
doc: docs/09-dns-and-ingress.md
safety_critical: false
summary: "Public HTTP(S) reachable with no inbound 80/443 forward; LAN works whichever resolver answers."
- id: R-DNS-05
doc: docs/09-dns-and-ingress.md
safety_critical: false
summary: "Only declared-public names reachable through public ingress; LAN/VPN-only names are refused."
# --- Automated updates ---
- id: R-UPD-01
@@ -129,7 +137,7 @@ requirements:
- id: R-UPD-02
doc: docs/11-automated-updates.md
safety_critical: true
summary: "Updates need no human/AI/hand-authored change; bad update auto-rolls-back."
summary: "Allowed-class updates need no human/AI/hand-authored change; bad update auto-rolls-back."
- id: R-UPD-03
doc: docs/11-automated-updates.md
safety_critical: false
@@ -175,15 +183,15 @@ requirements:
- id: R-ENV-01
doc: docs/02-environments-and-provisioning.md
safety_critical: false
summary: "dev/test provisioned as local KVMs via IaC (create/baseline/rebuild)."
summary: "Nodes baselined, cluster-joined and managed as code; bare-metal imaging is checklist-driven."
- id: R-ENV-02
doc: docs/02-environments-and-provisioning.md
safety_critical: false
summary: "Same GitOps config promotes dev/test -> prod (authored once)."
summary: "GitOps config authored once as base plus overlay; a future environment needs overrides only."
- id: R-ENV-03
doc: docs/02-environments-and-provisioning.md
safety_critical: false
summary: "prod is 3x mini-PC bare-metal k3s with local storage."
summary: "Production is 3x mini-PC bare metal; changes reach it only through the PR gate and Flux."
# --- GitOps & delivery ---
- id: R-GITOPS-01
@@ -212,6 +220,10 @@ requirements:
doc: docs/07-security-and-secrets.md
safety_critical: false
summary: "All security properties demonstrable/auditable."
- id: R-SEC-04
doc: docs/07-security-and-secrets.md
safety_critical: false
summary: "Kubernetes API not reachable from the public internet by any path; LAN/VPN-only."
# --- Identity & access ---
- id: R-IDP-01
@@ -22,8 +22,11 @@ metadata:
labels:
conformance.home.k3s/R-NET-01: "true"
conformance.home.k3s/R-NET-02: "true"
conformance.home.k3s/R-DNS-04: "true"
conformance.home.k3s/R-DNS-05: "true"
conformance.home.k3s/R-SEC-04: "true"
annotations:
conformance.home.k3s/requirements: "R-NET-01, R-NET-02"
conformance.home.k3s/requirements: "R-NET-01, R-NET-02, R-DNS-04, R-DNS-05, R-SEC-04"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "true"
@@ -105,8 +105,9 @@ metadata:
labels:
conformance.home.k3s/R-DNS-02: "true"
conformance.home.k3s/R-DNS-03: "true"
conformance.home.k3s/R-DNS-05: "true"
annotations:
conformance.home.k3s/requirements: "R-DNS-02, R-DNS-03"
conformance.home.k3s/requirements: "R-DNS-02, R-DNS-03, R-DNS-05"
conformance.home.k3s/layer: "L0"
# WHOLE-REPO SCOPE — this invariant quantifies over every ServiceDefinition in catalog/, every
# HTTPRoute in apps/wave-* and platform/* and runbooks/, and every external-dns-annotated
@@ -0,0 +1,46 @@
# =============================================================================
# L0 STATIC — PiHole answers an explicit LAN name list, and no LAN-only name is left out of it
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# Since PR #156 every public name is a proxied CNAME to the Cloudflare Tunnel and works from the
# LAN, so PiHole's zone wildcard was replaced by an explicit list (catalog/pihole.yaml). That makes
# the list load-bearing: a route disowned from public DNS and absent from it resolves nowhere. This
# test derives the required names from the tree (every `controller: none` HTTPRoute, every hostname
# the tunnel 404s, the L4 Services' own VIPs) and asserts the list covers them, carries no
# zone-wide entry, and pins gitea and nextcloud to the Gateway VIP. It proves itself non-vacuous by
# mutation. Rationale and the honest NOT-proven list: check.sh.
#
# WHY SCOPE, NOT REQUIREMENTS: the property spans catalog/, apps/, platform/ and runbooks/; a new
# internal service in any of them can break it, and ci/touched-requirements.sh maps apps/* to no
# requirement at all, so a requirement-scoped selection would skip it on precisely those changes.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: pihole-lan-names
labels:
conformance.home.k3s/R-DNS-03: "true"
conformance.home.k3s/R-DNS-04: "true"
annotations:
conformance.home.k3s/requirements: "R-DNS-03, R-DNS-04"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "true"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
PiHole answers an explicit list of LAN names rather than the whole zone; every HTTPRoute
disowned from public DNS has an entry to the Gateway VIP; every hostname the Cloudflare
Tunnel 404s has an entry, with the L4 names on their own VIPs; and gitea and nextcloud stay
on the LAN path, off Cloudflare's request-size and timeout limits.
spec:
steps:
- name: assert-pihole-explicit-lan-names
description: >-
Parse FTLCONF_misc_dnsmasq_lines from catalog/pihole.yaml and the rendered
apps/wave-0/pihole.yaml, derive the required names from every `controller: none` HTTPRoute,
the http_status:404 hostnames in platform/cloudflared/config.yaml and the L4 Services'
lbipam annotations, assert the list covers them and contains no zone-wide entry, and run
the evaluator on mutated lists to prove it is not vacuous.
try:
- script:
content: bash ./check.sh
+269
View File
@@ -0,0 +1,269 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — pihole-lan-names
# Serves: R-DNS-03 (LAN split-horizon answers), and decisions/2026-11-09-pihole-explicit-lan-names.md.
#
# WHY THIS EXISTS. Since PR #156 every PUBLIC name is a proxied CNAME to the Cloudflare Tunnel and
# works from the LAN, so PiHole no longer answers the whole zone with a wildcard: it answers an
# EXPLICIT list (catalog/pihole.yaml, FTLCONF_misc_dnsmasq_lines) and forwards everything else to
# the public resolvers. That inverts the old failure mode. With the wildcard, a forgotten name still
# resolved; now a forgotten LAN-ONLY name — a route disowned from public DNS, which the tunnel also
# 404s — resolves NOWHERE, and the app is silently unreachable from every client. Nothing in the
# cluster turns red when that happens. So the list is derived from the tree and asserted here.
#
# GIVEN the repository as authored,
# THEN (0) the dnsmasq value parses (only `local-ttl=` and `address=/<domain>/<${VAR}>` entries,
# every VAR is a cluster-var), the catalog value and the rendered apps/wave-0/pihole.yaml
# value are identical, and the inventories below are non-empty (anti-vacuous-pass);
# (a) NO entry covers the whole zone: no `address=/${BASE_DOMAIN}/`, no `address=/#/`, and no
# entry for any parent of the zone (dnsmasq matches every subdomain of an entry);
# (b) every HTTPRoute hostname disowned from public DNS
# (`external-dns.alpha.kubernetes.io/controller: none`, anywhere in apps/ platform/
# runbooks/) has an explicit entry to ${GATEWAY_VIP} — a disowned route with no
# `hostnames:` is itself a FAIL, because it cannot be checked;
# (c) every hostname platform/cloudflared/config.yaml maps to http_status:404 has an entry;
# the L4 DNS-only names (derived: Services carrying both
# `external-dns.alpha.kubernetes.io/hostname` and `lbipam.cilium.io/ips` — unifi,
# crafty-mc) must map to THAT Service's VIP, not the Gateway;
# (d) gitea and nextcloud are pinned to ${GATEWAY_VIP} (Cloudflare's 100 MB body / 100 s
# limits break git push/LFS and large uploads);
# AND (e) the evaluator is not vacuous: it is run against mutated copies of the real entry list
# (zone wildcard added; a disowned host dropped; a 404'd host dropped; gitea repointed;
# unifi repointed to the Gateway) and must flag each one.
#
# NOT proven (static): that the running PiHole pods carry this value (FTL applies FTLCONF_* at
# start; the env change rolls the StatefulSet), that public resolution of the unlisted names
# reaches Cloudflare, or the substituted IPs. Runtime check: `dig @${PIHOLE_DNS_VIP} <name>`.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
exec python3 - "$ROOT" <<'PY'
import copy
import re
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("CHECK FAIL (pihole-lan-names): PyYAML is required", file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
FAILURES, CHECKS = [], 0
NAME = "pihole-lan-names"
ZONE = "${BASE_DOMAIN}"
GW = "${GATEWAY_VIP}"
DISOWN = "external-dns.alpha.kubernetes.io/controller"
EDNS_HOST = "external-dns.alpha.kubernetes.io/hostname"
LB_IPS = "lbipam.cilium.io/ips"
# Public, but must bypass Cloudflare's 100 MB request body / 100 s response limits.
LAN_PINNED = ("gitea", "nextcloud")
# Trees that hold applied manifests. conformance/ fixtures and .claude/ worktrees are not authored
# surface (CLAUDE.md §0); decisions/ is prose.
SCAN = ("apps", "platform", "runbooks")
def check(cond, desc):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(desc)
def docs(path):
try:
return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)]
except yaml.YAMLError:
return []
def dnsmasq_value(objs):
"""The FTLCONF_misc_dnsmasq_lines value from a list of docs (a ServiceDefinition or manifests)."""
found = []
for d in objs:
envs = list(d.get("env") or [])
tmpl = (((d.get("spec") or {}).get("template") or {}).get("spec") or {})
for c in tmpl.get("containers") or []:
envs += c.get("env") or []
found += [e.get("value") for e in envs if e.get("name") == "FTLCONF_misc_dnsmasq_lines"]
return found
def die(msg):
print(f"CHECK FAIL ({NAME}): {msg}", file=sys.stderr)
raise SystemExit(1)
CATALOG = ROOT / "catalog" / "pihole.yaml"
RENDERED = ROOT / "apps" / "wave-0" / "pihole.yaml"
CF = ROOT / "platform" / "cloudflared" / "config.yaml"
for p in (CATALOG, RENDERED, CF):
if not p.is_file():
die(f"{p.relative_to(ROOT)} is absent — the surface this check guards moved; re-point or "
"delete this check in the SAME change")
cat_vals = dnsmasq_value(docs(CATALOG))
ren_vals = dnsmasq_value(docs(RENDERED))
check(len(cat_vals) == 1, f"(0) catalog/pihole.yaml declares FTLCONF_misc_dnsmasq_lines exactly "
f"once (found {len(cat_vals)})")
check(len(ren_vals) == 1, f"(0) apps/wave-0/pihole.yaml carries FTLCONF_misc_dnsmasq_lines exactly "
f"once (found {len(ren_vals)})")
if not cat_vals:
die("no dnsmasq value to check")
check(cat_vals[:1] == ren_vals[:1],
"(0) the rendered apps/wave-0/pihole.yaml dnsmasq value differs from catalog/pihole.yaml — "
"re-render pihole (catalog/render.py --no-image-automation catalog/pihole.yaml)")
cv = next(d for d in docs(ROOT / "clusters" / "base" / "cluster-vars.yaml")
if d.get("kind") == "ConfigMap")
CLUSTER_VARS = set(cv["data"])
BASE_DOMAIN_VALUE = cv["data"]["BASE_DOMAIN"]
ENTRY = re.compile(r"^address=/([^/]+)/(\$\{[A-Z0-9_]+\})$")
def parse(value):
"""Return (entries {domain: target}, problems)."""
entries, problems = {}, []
for raw in value.split(";"):
line = raw.strip()
if raw != line:
problems.append(f"entry {raw!r} has surrounding whitespace (FTL writes it verbatim)")
if not line:
problems.append("empty entry (doubled or trailing `;`)")
continue
if re.fullmatch(r"local-ttl=\d+", line):
continue
m = ENTRY.match(line)
if not m:
problems.append(f"entry {line!r} is not `local-ttl=N` or `address=/<domain>/${{VAR}}` — "
f"this check cannot reason about it")
continue
dom, tgt = m.groups()
if tgt[2:-1] not in CLUSTER_VARS:
problems.append(f"{dom} -> {tgt}: not a key of clusters/base/cluster-vars.yaml")
if dom in entries and entries[dom] != tgt:
problems.append(f"{dom} is listed twice with different targets")
entries[dom] = tgt
return entries, problems
entries, problems = parse(cat_vals[0])
for pr in problems:
check(False, f"(0) {pr}")
# ── inventories, derived from the tree ──────────────────────────────────────────────────────────
disowned, unnamed = set(), []
l4 = {}
for top in SCAN:
for f in sorted((ROOT / top).rglob("*.yaml")):
for d in docs(f):
meta = d.get("metadata") or {}
ann = meta.get("annotations") or {}
if d.get("kind") == "HTTPRoute" and str(ann.get(DISOWN, "")).strip() == "none":
hosts = (d.get("spec") or {}).get("hostnames") or []
if not hosts:
unnamed.append(f"{f.relative_to(ROOT)}:{meta.get('name')}")
disowned.update(hosts)
if d.get("kind") == "Service" and EDNS_HOST in ann and LB_IPS in ann:
for h in str(ann[EDNS_HOST]).split(","):
l4[h.strip()] = str(ann[LB_IPS]).strip()
for u in unnamed:
check(False, f"(b) disowned HTTPRoute {u} declares no `hostnames:` — it inherits a listener "
f"hostname this check cannot see; name it explicitly")
check(len(disowned) >= 5, f"(0) only {len(disowned)} disowned HTTPRoute hostnames found — the scan "
f"is vacuous (the disown annotation was renamed?)")
check(len(l4) >= 2, f"(0) only {len(l4)} L4 Services publishing their own hostname found (expected "
f"unifi and crafty-mc) — the scan is vacuous")
cf = yaml.safe_load(CF.read_text())
cf_404 = {r["hostname"] for r in cf.get("ingress") or []
if r.get("hostname") and r.get("service") == "http_status:404"}
check(len(cf_404) >= 5, f"(0) only {len(cf_404)} http_status:404 hostnames in the cloudflared "
f"config — the scan is vacuous")
def covers_zone(dom):
"""True if an entry for `dom` answers every name in the zone (dom is the zone or a parent)."""
if dom in ("#", "", "."):
return True
if dom == ZONE:
return True
# A literal parent (e.g. `keithjackson.dev`, `dev`) of the zone's substituted value.
real = dom.replace(ZONE, BASE_DOMAIN_VALUE)
return real == BASE_DOMAIN_VALUE or BASE_DOMAIN_VALUE.endswith("." + real)
def evaluate(entries):
bad = []
# (a)
for dom in entries:
if covers_zone(dom):
bad.append(f"(a) `address=/{dom}/` answers EVERY name in the zone — the wildcard is gone "
f"on purpose (public names must resolve to Cloudflare from the LAN)")
# (b)
for h in sorted(disowned):
if entries.get(h) != GW:
bad.append(f"(b) {h} is disowned from public DNS but PiHole maps it to "
f"{entries.get(h, 'NOTHING')} — expected {GW}; without it the app is "
f"unreachable everywhere")
# (c)
for h in sorted(cf_404):
if h in l4:
if entries.get(h) != l4[h]:
bad.append(f"(c) {h} is an L4 DNS-only name; PiHole maps it to "
f"{entries.get(h, 'NOTHING')}, expected its own VIP {l4[h]}")
elif h not in entries:
bad.append(f"(c) the tunnel 404s {h} but PiHole has no entry for it — it would resolve "
f"only publicly, and the tunnel refuses it")
# (d)
for n in LAN_PINNED:
h = f"{n}.{ZONE}"
if entries.get(h) != GW:
bad.append(f"(d) {h} maps to {entries.get(h, 'NOTHING (public/Cloudflare)')}; it must be "
f"pinned to {GW} — Cloudflare caps request bodies at 100 MB and responses at "
f"100 s")
return bad
for v in evaluate(entries):
check(False, v)
# (e) mutation self-test
mutants = {}
m = dict(entries); m[ZONE] = GW
mutants["zone wildcard added"] = m
m = dict(entries); m[BASE_DOMAIN_VALUE.split(".", 1)[1]] = GW
mutants["parent-of-zone wildcard added"] = m
m = dict(entries); m.pop(sorted(disowned)[0], None)
mutants[f"disowned host {sorted(disowned)[0]} dropped"] = m
pure_404 = sorted(h for h in cf_404 if h not in l4 and h not in disowned)
if pure_404:
m = dict(entries); m.pop(pure_404[0], None)
mutants[f"404'd host {pure_404[0]} dropped"] = m
m = dict(entries); m.pop(f"gitea.{ZONE}", None)
mutants["gitea unpinned"] = m
m = dict(entries); m[f"nextcloud.{ZONE}"] = "${UNIFI_L4_VIP}"
mutants["nextcloud repointed"] = m
if l4:
h = sorted(l4)[0]
m = dict(entries); m[h] = GW
mutants[f"{h} repointed to the Gateway"] = m
for name, mut in mutants.items():
check(evaluate(mut), f"(e) self-test: the evaluator did NOT flag the mutant '{name}' — vacuous")
check(len(mutants) >= 5, f"(e) only {len(mutants)} mutants built — the self-test degenerated")
_, pr = parse("local-ttl=60;address=/x.${BASE_DOMAIN}/${NOT_A_CLUSTER_VAR}; address=/y/${GATEWAY_VIP}")
check(len(pr) >= 2, "(e) self-test: the parser accepted an unknown VAR or a whitespace-padded entry")
if CHECKS < 15:
FAILURES.append(f"only {CHECKS} assertions ran — the check degenerated")
if FAILURES:
print(f"CHECK FAIL ({NAME}): {len(FAILURES)} of {CHECKS} assertions failed", file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
print(f"PASS ({NAME}): {CHECKS} assertions; {len(entries)} PiHole names, {len(disowned)} disowned "
f"routes, {len(cf_404)} tunnel-404 names, {len(l4)} L4 names")
PY
+46
View File
@@ -0,0 +1,46 @@
---
date: 2026-11-08
branch: chore/adopt-spec-sync
title: Adopt the spec sync (registry parity, ADR cross-references, spec pin)
---
## What changed
The operator ruled this repo's `main` (`50c8719`) the accepted state and had the System Architect
bring the spec repo in line with it (spec PR `jakman85/home-network-k3s#2`). This is the
implementation half of that sync.
- `conformance/requirements/registry.yaml`: added R-DNS-04, R-DNS-05 and R-SEC-04, which the spec
defined and this mirror lacked. Reworded the R-ENV-01/02/03 and R-UPD-02 summaries to match the
amended requirements.
- Tagged existing L0 tests against those IDs, because a registry entry with no tagged test blocks
the merge: `cloudflared-ingress-rules` (R-DNS-04, R-DNS-05, R-SEC-04), `pihole-lan-names`
(R-DNS-04), and `internal-services-publish-no-public-dns` (R-DNS-05).
- `platform/vpn-egress/README.md` and a comment in `egress-steering.yaml` pointed at a drafted
"ADR-0016 supersede". That number went to the Cloudflare Tunnel; the accepted ADR is ADR-0020.
- `ci/SPEC_PIN` must move to the spec merge commit. That SHA does not exist until the spec PR
merges, so this PR carries a `WIP:` title until the bump is added.
## What I verified
- Read each tagged test's assertions before tagging it. `cloudflared-ingress-rules` asserts every
LAN-only hostname maps to a 404, the catch-all exists, no rule targets the API, and the egress
policy admits no API or host entity. `pihole-lan-names` asserts LAN names resolve independently
of the public path. `internal-services-publish-no-public-dns` asserts internal routes carry the
disown annotation and pins the declared L4 set.
- The toolbox gate result for this commit is quoted in the PR description, with the spec branch
mounted so that the checks parsing spec text ran against the new spec.
## What I did NOT verify
- **These tags are static coverage, not proof.** No L2 test touches the tunnel, and nothing here
shows router 80/443 closed or 6443 unforwarded (spec `docs/16-known-gaps.md`).
- R-SEC-04 is registered `safety_critical: false`. Making it `true` needs an L3 fault-injection
test that does not exist. UNVERIFIED whether the operator wants that.
- No live cluster, tunnel or DNS state was measured.
- `spec-immutability` did not run against a pinned checkout: the pin is not bumped yet.
- Stale implementation prose found by the audit was deliberately left out of this PR, because the
operator ruled this repo the accepted state and none of it changes behaviour. Examples:
`catalog/gitea.yaml` still says "+ mirrored"; `platform/longhorn` still promises a backup target;
`README.md` says `node-loss` is outstanding; `hands-off-update` exercises Flux image policy rather
than Renovate; `dev-nodes-provisioned-as-code` is named for an environment that no longer exists.
@@ -0,0 +1,74 @@
---
date: 2026-11-08
branch: fix/wg-client-mtu-docs
title: WireGuard client profiles default to MTU 1280; VPN runbook corrected and given a troubleshooting section
---
## What changed
- **`infra/vpn-ingress/new-peer.sh`**: `WG_MTU` default 1420 -> 1280, still overridable, reason in a
comment. Only new profiles change; the phone's existing profiles are not touched.
- **`runbooks/vpn-remote-access.md`**: the verification table no longer says the Gateway sees a
`host`-identity source; the limitations no longer say everything arrives SNATed or that IPv6 leaks
on Android; the "if nothing handshakes" advice no longer claims `wg show` separates a bad forward
from a bad key. New "Troubleshooting" section covering the four cases in the brief.
- **`platform/vpn-ingress/daemonset.yaml`**: header comment only, SNAT claim corrected.
- **`platform/vpn-ingress/README.md`**: one sentence, so `MTU = 1420` reads as the server's value.
Rollout on merge: the WireGuard DaemonSet does **not** roll. The `runbooks` ConfigMap is content
hashed and `vpn-remote-access.md` is in it, so the runbooks site rolls.
Left unchanged, for the operator:
- **`platform/vpn-ingress/peers.conf`** still says "No peers enrolled yet". A comment-only edit on a
throwaway copy moved the ConfigMap from `wg-peers-29mf6k867t` to `wg-peers-gggkk5d4tm` and changed
the DaemonSet, so correcting it restarts every `wg0`. Fix it alongside the next peer change.
- **`clusters/base/cluster-vars.yaml`** coverage comment still omits `WG_CLIENT_CIDR` (sensitive path).
- **Server `wg0` MTU stays 1420**, because changing it restarts all WireGuard pods. For TCP a 1280
client should be enough on its own: it advertises an MSS of 1240, which caps what the far end
sends. That protection does not reach UDP. A server-to-phone datagram with an inner size of
1281-1420 bytes is still sent as up to 1480 outer bytes and could be lost on the same path; QUIC
should recover by probing, large DNS answers are the likeliest casualty. If that shows up, the
fix is a lower server MTU, as a deliberate rollout.
- **Other stale text, outside this brief:** `platform/vpn-ingress/README.md` still says no peer has
ever completed a handshake, and the runbook's status line still calls the VPN inert.
## What I verified
- Upstream `wireguard-android` `GoBackend.java`, master (last commit to that file `ecc3c0a`,
2026-03-15): `builder.setMtu(config.getInterface().getMtu().orElse(1280))`, and `allowFamily()` is
skipped when `sawDefaultRoute && config.getPeers().size() == 1`.
- Live, read-only, 2026-11-09 UTC, on the `k3s-prod-1` pod and agent:
- `wg0` MTU is 1420 on all three pods. Both peers have handshakes and transfer on prod-1 only; the
other two pods read `0`.
- BPF conntrack holds `UDP SVC 10.90.0.x -> 192.168.2.23:53` and flows to `192.168.2.24:32400`.
Their backend entries carry `SourceSecurityID=2` (`world`). So the source stays `10.90.0.x`
through the load balancer on those two VIPs.
- `/proc/net/nf_conntrack` is readable from the pod, which has no `conntrack` binary.
- Offline, pinned toolbox: `kubectl kustomize platform/vpn-ingress` gives identical ConfigMap and
DaemonSet hashes on `origin/main` and on this branch.
- `new-peer.sh` writes `MTU = 1280` by default and `MTU = 1420` with `WG_MTU=1420`. Run with
throwaway keys and stdout discarded; profiles shredded; nothing enrolled.
- No test or conformance check pins the client MTU (`grep` for `WG_MTU` and `1420`).
- Gate (pinned toolbox, from this worktree, on the committed tree): **rc 3, UNDETERMINED, not green.**
`gitea-settings` is the only stage counted as not run (no token locally). `conformance` (L2/L3)
also skipped as the declared no-cluster gap, so R-ENV-01 and R-OBS-04 are not certified.
`spec-immutability`, `approval-policy`, `decisions-log`, `lint`, `l0`, `render-golden` and `l1`
passed. No sensitive path is touched.
## What I did NOT verify
- **That MTU was the cause.** It is inferred from one before-and-after, on one phone and one carrier,
reported by the operator. The path MTU was never measured.
- **That 1280 is the best value.** It is the value that worked, and the IPv6 minimum. A higher one
may work too.
- That the phone's app is on `GoBackend` rather than the kernel backend, so that a blank field meant
1280 on this device. Only the upstream default was read.
- That TCP MSS behaves as described, or that large UDP replies are actually lost. No packet capture
was taken.
- That the Gateway VIP `.20` sees `10.90.0.x` / `world`. Conntrack held no tunnel flows to it when
checked.
- Android's IPv6 blocking on a real device; read from source only.
- `wg-quick` IPv6 behaviour on Linux, and any non-Android client at 1280.
- `/proc/net/nf_conntrack` output during a key mismatch. The `[UNREPLIED]` reading comes from the
earlier measurement in the brief; I saw only healthy entries.
@@ -0,0 +1,97 @@
---
date: 2026-11-09
branch: feat/pihole-explicit-lan-names
title: PiHole answers an explicit LAN name list instead of a zone wildcard (public names resolve to Cloudflare from the LAN)
---
## What changed
**`catalog/pihole.yaml` → `apps/wave-0/pihole.yaml`.** `FTLCONF_misc_dnsmasq_lines` no longer carries
`address=/${BASE_DOMAIN}/${GATEWAY_VIP}`. Every name not listed is now forwarded to 8.8.8.8/8.8.4.4 and
gets the public answer. The final list (`local-ttl=60` kept):
| Name (`.${BASE_DOMAIN}`) | Answer | Why |
|---|---|---|
| jackett, pihole, plex, qbittorrent, radarr, semaphore, sonarr, headlamp | `${GATEWAY_VIP}` | LAN-only: every HTTPRoute with `external-dns.alpha.kubernetes.io/controller: none`, derived from the tree. No public record, tunnel 404s them; this entry is their only path |
| gitea, nextcloud | `${GATEWAY_VIP}` | public, but Cloudflare's proxy caps bodies at 100 MB / responses at 100 s (git push, LFS, uploads) |
| keycloak, auth | `${GATEWAY_VIP}` | keep the OIDC flow of the LAN-path apps on one path |
| unifi | `${UNIFI_L4_VIP}` | L4 VIP, not behind the Gateway |
| crafty-mc | `${CRAFTY_L4_VIP}` | L4 VIP, not behind the Gateway |
| pihole-dns | `${PIHOLE_DNS_VIP}` | the resolver's own address |
| plex-vip | `${PLEX_L4_VIP}` | kept, see below |
Not listed, so they resolve to Cloudflare from the LAN: the apex, grafana, crafty, seerr.
**The derived LAN-only set matches the brief's list exactly** (8 names; the tree was walked, not the
list trusted). The `runbooks/serving` route has no `hostnames:` but is not disowned (it inherits the
apex, public).
**plex-vip is kept.** Nothing advertises it any more (`ADVERTISE_IP` is a `plex.direct` name since
2026-10-05), but it is not unused: it is Plex's `customCertificateDomain` (`CERTDOMAIN`, Secret
`plex-vip-tls`), the name of the tcp/32400 VIP, and the 404'd name in the tunnel config. It has no
public record, so this entry is its only resolution.
**New L0 check `conformance/tests/l0-static/pihole-lan-names`** (`scope: whole-repo`, so picked up by
the runner's scope selection with no other wiring). Asserts: (a) no zone-wide entry (zone, `#`, or any
parent of the zone); (b) every `controller: none` hostname → `${GATEWAY_VIP}`, and a disowned route
with no `hostnames:` fails; (c) every tunnel-404 hostname has an entry, with the L4 names (derived
from Services carrying both the external-dns hostname and an lbipam VIP) on their own VIP; (d) gitea
and nextcloud → `${GATEWAY_VIP}`; plus catalog == rendered value and a built-in mutation self-test.
**Stale docs corrected:** `platform/pm-agent/README.md`, `runbooks/pm-agent.md`,
`platform/externaldns/README.md`, `platform/cloudflared/README.md`, `platform/gateway/gateway.yaml`
(comment), `platform/headlamp/httproute.yaml` (comment), `apps/vms/unifi/README.md`, `README.md`,
`schemas/renderer-contract.md`; `runbooks/cloudflare-tunnel.md` gains the two consequences below.
`platform/cloudflared/deployment.yaml`: the image entrypoint comment was wrong (comment-only).
## What I verified
- **Render:** the baseline `render.py --no-image-automation catalog/pihole.yaml` was byte-identical to
the committed `apps/wave-0/pihole.yaml` before the change, and the file was replaced by that
command's output after it; `--emit-waves` was not run.
- **Mutation, against the real files (10 mutants, each rc 1 with the right assertion, each restored):**
(a) zone wildcard restored; (b) headlamp and sonarr entries removed; (b) a disowned route with no
`hostnames:`; (c) a new tunnel-404 host with no entry; (c) pihole-dns removed; (c) unifi pointed at
the Gateway; (d) gitea removed; (d) nextcloud repointed; (0) rendered value drifted from the catalog.
- **Rollout mechanics, measured live (read-only):** StatefulSet `updateStrategy` RollingUpdate
partition 0, `podManagementPolicy: Parallel`, no `minReadySeconds`, 2 replicas, PDB minAvailable 1.
The last rollout (2026-11-08 06:47–06:48Z) replaced pihole-1 then pihole-0: FTL logged "started" ~1 s
after the container, the pod went Ready ~16 s after the container, and pihole-0 started ~9 s later.
- **Expected LAN DNS blip:** no total outage; one replica answers throughout. Each of the two pod
replacements can lose a few UDP queries for ~1–2 s while the terminating pod leaves the Service
(clients retry). For roughly 30–60 s the two pods answer differently for the apex, grafana, crafty
and seerr; both answers work. Cached answers age out in 60 s (`local-ttl`).
- **Live DNS before the change:** @8.8.8.8 returns Cloudflare (104.21.51.91 / 172.67.178.27) for the
apex, grafana, seerr, crafty and gitea, and nothing for jackett; @192.168.2.23 returns 192.168.2.20
for all of them (the wildcard).
- **Cloudflared image:** `skopeo inspect --config` on the pinned digest (amd64): Entrypoint
`["cloudflared"]`, Cmd `["version"]`, User `65532:65532`.
## What I did NOT verify
- **The post-merge behaviour.** Nothing here ran against a live PiHole with the new list; the
expected answers are from dnsmasq semantics and the tree. After reconcile, check
`dig @192.168.2.23` for one name per row, and grafana returning Cloudflare.
- **Whether every LAN client path works via Cloudflare.** Measured 2xx/3xx through Cloudflare by the
brief, not re-measured from a LAN client here.
- **The FTL DNS-listening moment.** "FTL started" is a log line, not a successful query; the rollout
timing above assumes DNS is up shortly after it.
**Accepted consequences, stated plainly:**
- The apex, grafana, crafty and seerr now depend on the internet and the tunnel from the LAN too. An
ISP or Cloudflare outage, or stopping the connectors, makes them unreachable from the LAN where they
previously kept working. If grafana in particular should survive an internet outage, add it to the
list (one entry; the check permits it).
- The tunnel rollback (revert to `wan.`) now also requires restoring the wildcard; recorded in
`runbooks/cloudflare-tunnel.md`.
**Flagged, not changed:**
- The `pihole` container has no readiness probe; only the `oauth2-proxy` sidecar does, so "Ready" says
nothing about DNS. The measured margin (FTL ~1 s, Ready ~16 s) makes this benign in a normal rollout
but a slow FTL start would not hold the rollout back. A DNS readiness probe needs a schema/renderer
decision.
**Stale statements left alone:** `platform/coredns/coredns-custom.yaml` header ("both resolve the whole
zone ... by wildcard", brief said not to touch CoreDNS); `clusters/base/cluster-vars.yaml` comments
(sensitive path); `schemas/renderer-contract.md` §`exposure` first bullets (manual edge port-forward
wording); historical `infra/PROD-PROVISIONING.md`, `platform/externaldns/README.md` TODO list.
+9 -1
View File
@@ -53,6 +53,7 @@
# WG_ENDPOINT_HOST default wan.home.keithjackson.dev
# WG_CLIENT_CIDR default 10.90.0.0/24 (must match clusters/base/cluster-vars.yaml)
# WG_LAN_CIDR default 192.168.2.0/24 (prod LAN_CIDR)
# WG_MTU default 1280 (client tunnel MTU; see the comment on the default below)
# WG_DNS default 192.168.2.23 (prod PIHOLE_DNS_VIP)
#
# REQUIRES: `wg`, `qrencode`, `sops` — all three in the dev container.
@@ -80,7 +81,14 @@ PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.conf"
# cannot resolve until it is already connected.
: "${WG_ENDPOINT_HOST:=wan.home.keithjackson.dev}"
: "${WG_LISTEN_PORT:=51820}"
: "${WG_MTU:=1420}"
# The CLIENT's tunnel MTU. Deliberately lower than the server's 1420 (daemonset.yaml), which only
# describes the 1500-byte path at the node's end. A phone's path is the carrier's, often behind
# carrier NAT, and is not known to carry 1500. Measured 2026-11-09, one Android phone on one
# carrier: at 1420 the handshake was healthy but names and pages failed; clearing the app's MTU
# field (its userspace backend then uses 1280) fixed it. 1280 is also the IPv6 minimum link MTU, so
# it stays valid if the tunnel ever gains IPv6. For a device only ever on a wired or known-1500
# path, WG_MTU=1420 recovers about 10% of per-packet payload.
: "${WG_MTU:=1280}"
SPLIT=0
OUT_DIR=""
+5 -3
View File
@@ -17,6 +17,8 @@ outbound-only from the cluster (no router port forward, no hairpin, no split-DNS
| `networkpolicy.yaml` | default-deny (single owner — not an app namespace) + Prometheus scrape allow + egress CNP: world 7844 tcp/udp and 443, the Gateway (`ingress` entity), and Gateway backends by `gateway-access=shared` minus the LAN-only namespaces. |
| `servicemonitor.yaml`, `alerts.yaml` | Headless metrics Service + ServiceMonitor scraping :2000, and the two alerts, both written to fire (not go quiet) when their series vanish. |
**Phase 1 only.** This component changes no live traffic: Cloudflare sends a hostname through the
tunnel only when its DNS record is a CNAME to `<UUID>.cfargotunnel.com`, and nothing here creates
one. Moving DNS (external-dns, Gateway annotations, Pi-hole/CoreDNS split horizon) is a later change.
Cloudflare sends a hostname through the tunnel only when its DNS record is a proxied CNAME to
`<UUID>.cfargotunnel.com`; nothing in this directory creates one. Those records are published by
external-dns from the shared Gateway's target annotation (Phase 2, PR #156; see
`platform/externaldns/README.md`). LAN resolution: `catalog/pihole.yaml` (explicit names only since
2026-11-09), so public names reach Cloudflare from the LAN too.
+3 -2
View File
@@ -70,8 +70,9 @@ spec:
- name: cloudflared
# Tag + digest: Renovate's `kubernetes` manager (renovate.json5, platform/**) bumps both.
image: docker.io/cloudflare/cloudflared:2026.10.7@sha256:4955524aeaa379e154023d5a027c250f7b35041a2e7e950cf13f144bc9b9c5ba
# The image ENTRYPOINT is `cloudflared --no-autoupdate`; these replace its CMD. The tunnel
# UUID, credentials path, metrics address and ingress rules all come from the file.
# The image ENTRYPOINT is `cloudflared` and its CMD is `version` (read from the pinned
# digest's image config, 2026-11-09); these args replace that CMD. `no-autoupdate`, the
# tunnel UUID, credentials path, metrics address and ingress rules all come from the file.
args:
- tunnel
- --config
+7 -2
View File
@@ -44,8 +44,13 @@ wan.home.keithjackson.dev A (DNS-only) <current WAN IP> <- cloudflare-ddn
- `internal` routes (`controller: none`) still publish nothing. Plex's web UI and Headlamp joined that
set on 2026-11-08 (LAN/VPN-only); the tunnel also 404s every such name (9c/9d).
- Public answers are Cloudflare anycast **A and AAAA** — Cloudflare's edge, not an origin address.
- LAN and pods resolve `*.home.keithjackson.dev` to the Gateway VIP via PiHole and CoreDNS
(split-horizon) and never use the public path.
- **Pods** resolve the whole zone to the Gateway VIP via CoreDNS (split-horizon) and never use the
public path. **LAN clients** use PiHole, which since 2026-11-09 answers an EXPLICIT list
(`catalog/pihole.yaml`): the LAN-only names, plus gitea/nextcloud/keycloak/auth (kept off
Cloudflare's 100 MB / 100 s limits and on one OIDC path), go to the Gateway VIP and the L4 names to
their own VIPs; every other public name (apex, grafana, crafty, seerr) resolves to Cloudflare from
the LAN exactly as from anywhere else, so no split-DNS is needed for them. Guard:
`conformance/tests/l0-static/pihole-lan-names`.
- **Nodes resolve through public DNS** (`node_baseline_dns_servers`), so from a node the apiserver's
OIDC issuer `keycloak.home.keithjackson.dev` now resolves to Cloudflare and its discovery/JWKS
fetches go through the tunnel (publicly trusted edge cert; no `oidc-ca-file` is configured). The
+4 -2
View File
@@ -30,8 +30,10 @@ metadata:
# applies BEFORE it reads this target. Those same names map to `http_status:404` in the tunnel's
# ingress rules, so even a hand-made record cannot reach them through the tunnel.
#
# LAN and in-cluster clients are unaffected: PiHole and CoreDNS answer ${BASE_DOMAIN} with
# ${GATEWAY_VIP} directly (split-horizon), so they never take this public path.
# In-cluster clients are unaffected: CoreDNS answers ${BASE_DOMAIN} with ${GATEWAY_VIP} directly
# (split-horizon). LAN clients use PiHole's EXPLICIT list (catalog/pihole.yaml, since 2026-11-09):
# LAN-only names and gitea/nextcloud/keycloak/auth get ${GATEWAY_VIP}; every other published
# name takes this public path through Cloudflare, from the LAN as from anywhere.
#
# `wan.${BASE_DOMAIN}` (platform/externaldns/ddns.yaml) is NO LONGER a target of anything here;
# it stays only as the WireGuard udp/51820 endpoint.
+1 -1
View File
@@ -20,7 +20,7 @@ metadata:
homelab.keithjackson.dev/auth-groups: platform-admins
# LAN/VPN-ONLY (operator decision 2026-11-08). Disowned from external-dns, so the public
# headlamp record and its TXT are DELETED and nothing republishes them; LAN/VPN clients keep
# resolving the name via PiHole's split-horizon wildcard (-> the Gateway VIP). The Cloudflare
# resolving the name via PiHole's explicit entry for it (-> the Gateway VIP). The Cloudflare
# Tunnel also maps headlamp.${BASE_DOMAIN} to http_status:404 (platform/cloudflared/config.yaml).
# Same mechanism the renderer emits for `exposure: internal` services; asserted by
# conformance/tests/l0-static/internal-services-publish-no-public-dns (9c).
+7 -4
View File
@@ -72,11 +72,14 @@ Both are deliberate, and both fail **visibly** rather than silently:
- **The nodes resolve the registry hostname to the WAN, so the TLS they fail to verify is
the router's — DIAGNOSED 2026-10-06, and fixed in this change.** containerd pulls as a
*node* process, so it resolves through the node's public upstreams
(`node_baseline_dns_servers` = the LAN gateway) rather than through Pi-hole. The public
answer for every name in this zone is a CNAME to `wan.home.keithjackson.dev` — the WAN
IP — which this router does not hairpin; it answers :443 with its own placeholder
(`node_baseline_dns_servers` = the LAN gateway) rather than through Pi-hole. At the time,
the public answer for every name in this zone was a CNAME to `wan.home.keithjackson.dev`
— the WAN IP — which this router does not hairpin; it answered :443 with its own placeholder
certificate `CN=%D` carrying **no extensions, therefore no SANs**, which is precisely
`x509: certificate is not valid for any names`. The Gateway, the wildcard certificate
`x509: certificate is not valid for any names`. (Since 2026-11-08 the public answer is
Cloudflare's edge via the Tunnel, not the WAN IP — see `platform/externaldns/README.md`.
The pin is still the right path: it keeps registry pulls on the LAN and off Cloudflare's
proxy.) The Gateway, the wildcard certificate
and the digest were never implicated. The fix is the pin this repository already uses
for the Keycloak issuer, for the identical reason: `gitea.{{ k3s_base_domain }}` is now
in `node_baseline_hosts_pins` — node configuration is Ansible's, declared in the
+3 -3
View File
@@ -217,9 +217,9 @@ Consequences, stated plainly:
**Governance:** ADR-0015 and `docs/03` are specifications and are immutable to agents
(spec `AGENTS.md` §11, R-TEST-06, enforced by `ci/spec-immutability.sh`) — they are **not** edited
here. The correction is raised as an **escalation with a drafted ADR-0016 supersede** in
`DECISIONS-LOG.md` (entry of 2026-09-06) for the operator/architect to accept. Until then this file
and `egress-steering.yaml` are the accurate statement of what the implementation can actually do.
here. The correction, first escalated in `DECISIONS-LOG.md` (entry of 2026-09-06), is recorded in
spec **ADR-0020**, which supersedes ADR-0015's Kill-switch and Resilience bullets. This file and
`egress-steering.yaml` state what the implementation does.
## Secrets — host-side now
PIA credentials + the `.ovpn` live **on the gateway node via Ansible vault** (see spec `docs/07`), **not**
+1 -1
View File
@@ -109,7 +109,7 @@ spec:
# that shard and re-shards the rest (breaking live connections), and each shard exits from a
# DIFFERENT public IP. Upstream: cilium/cilium#30157, cilium/cilium#18230.
# ADR-0015's "Resilience: two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)"
# is therefore NOT IMPLEMENTABLE on Cilium OSS — escalated for an ADR-0016 supersede; see
# is therefore NOT IMPLEMENTABLE on Cilium OSS — superseded by spec ADR-0020; see
# platform/vpn-egress/README.md and the DECISIONS-LOG entry of 2026-09-06.
#
# `interface: tun0` IS RESOLVED ONCE — the gateway node has to tell Cilium when tun0 changes.
+3 -1
View File
@@ -112,7 +112,9 @@ no change. Three further problems vanish with the host netns:
`conformance/tests/l0-static/l2-announced-vips-serve-from-any-node`) and no
`externalTrafficPolicy` trade-off to make at all.
* **No double encapsulation.** Client UDP lands on the node's own NIC, so the underlay is a flat
1500-byte path, not Cilium's 1450-byte VXLAN. That is what makes `MTU = 1420` simply correct.
1500-byte path, not Cilium's 1450-byte VXLAN. That is what makes `MTU = 1420` correct on the
server's `wg0`. It says nothing about a client's own path: `new-peer.sh` defaults client profiles
to `1280` (see "Troubleshooting" in `runbooks/vpn-remote-access.md`).
`hostNetwork` is *permitted* here, and not by oversight:
`platform/kyverno/policies/restrict-host-namespaces.yaml` matches `namespaceSelector:
+6 -3
View File
@@ -30,9 +30,12 @@
# `cluster` entity, so VPN clients would not have been able to resolve `*.${BASE_DOMAIN}` either.
# The VPN would have looked up and been useless, which is the worst failure shape available.
#
# In the host netns there is no pod identity to carry: forwarded client traffic is SNATed to the
# NODE's address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` —
# exactly what a LAN client looks like. Three further problems disappear with it:
# In the host netns there is no pod identity to carry, so a client never looks like a pod. What a
# destination sees depends on how the packet is handled. LAN hosts and the internet: forwarded,
# then MASQUERADEd to the NODE's address by the PostUp rule below. LoadBalancer VIPs: Cilium's load
# balancer takes the packet as it enters from wg0, before that rule, so the source is still
# 10.90.0.x with identity `world`, like a LAN browser (measured 2026-11-09 for PiHole .23 and Plex
# .24; see README). Three further problems disappear with it:
# * net.ipv4.ip_forward is ALREADY 1 on these nodes (node_baseline writes it to
# /etc/sysctl.d/99-k3s-cilium.conf), so no pod sysctl is needed — and `net.ipv4.ip_forward`
# is NOT in the kubelet's safe-sysctl allowlist, so a pod-netns version of this would have
+6 -2
View File
@@ -353,11 +353,15 @@ is deleted, which is why the last step is not optional.
'flux --context k3s-prod suspend kustomization cloudflared &&
kubectl --context k3s-prod -n cloudflared scale deploy/cloudflared --replicas=0'
```
Cloudflare then answers tunnel hostnames with a 530/1033 error. Follow up in git, then
Cloudflare then answers tunnel hostnames with a 530/1033 error — **from the LAN too** for the
names PiHole does not list (apex, grafana, crafty, seerr; since 2026-11-09). Follow up in git, then
`flux resume kustomization cloudflared`.
- **After the DNS flip (Phase 2):** the path back is a revert PR of `feat/dns-via-cloudflare-tunnel`
(Gateway target back to `wan.${BASE_DOMAIN}`, drop `--cloudflare-proxied`) — external-dns then
rewrites every record DNS-only to `wan.`. That only works if the router's WAN tcp/80+443 forwards
are **re-opened** first (they are closed after Phase 2). Never edit the records by hand:
are **re-opened** first (they are closed after Phase 2). **Also restore PiHole's zone wildcard**
(`address=/${BASE_DOMAIN}/${GATEWAY_VIP}` in `catalog/pihole.yaml`, and drop the
`pihole-lan-names` L0 assertion (a)): since 2026-11-09 the apex, grafana, crafty and seerr are not
listed there and resolve publicly, and a public `wan.` answer does not hairpin from the LAN. Never edit the records by hand:
external-dns owns them and `policy: sync` reverts a hand edit. Revoking the tunnel credential (`cf tunnel delete`, or rotating it
in the Zero Trust dashboard) cuts every connector at once if the credential is believed leaked.
+1 -1
View File
@@ -74,7 +74,7 @@ Work through these in order; the first three are by far the most likely.
|---|---|---|
| `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
| `ImagePullBackOff`, digest `sha256:0000…` | (historical: before 2026-10-30 the committed digest was an all-zero placeholder) the image has never been published at the digest named | CI publishes it, not a person: check the latest `images` workflow run on `main` (it fails loudly if the `REGISTRY_PUSH_TOKEN` secret is missing; re-run it with `workflow_dispatch`), then merge Renovate's digest-bump PR, which updates **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) together — but read the row below first |
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and (before the 2026-11-08 Tunnel cutover; the public answer is now Cloudflare) got the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
| `ImagePullBackOff` with `401 Unauthorized` / `unauthorized` | the pull credential is wrong, absent or scoped without `read:package`. The `pm-agent-registry` Secret exists and is referenced; its *value* is what starts as a placeholder | check the key names only, never values: `kubectl -n pm-agent get secret pm-agent-registry -o go-template='{{range $k, $v := .data}}{{$k}}{{"\n"}}{{end}}'` should list `.dockerconfigjson`. Then confirm the token carries `read:package`, that its account can read the org's packages (a team with the Packages unit — see the `.example`), and that the registry key inside the JSON is the bare host `gitea.home.keithjackson.dev` — with a scheme or a port it matches nothing and the pull goes out anonymously, which looks identical to a missing Secret. Steps: `platform/pm-agent/pm-agent-registry.sops.yaml.example` |
| `GET /user` answers `403` / `token does not have at least one of required scope(s): [read:user]` at startup, then a CrashLoop | the token lacks `read:user` — Gitea 1.24.3 scopes the whole `/api/v1/user` group, and the dispatcher's first call is `GET /user` | mint a new token with all four scopes (`read:user`, `write:repository`, `write:issue`, `read:package`) and put it in **both** Secrets. Add `read:notification` if you want the cheap pull-request mention query (`platform/pm-agent/README.md`) |
| a worker Job is refused by admission (`pm-agent-job-shape`, `imagePullSecrets`) | `PM_WORKER_IMAGE` is a digest built before the template carried `imagePullSecrets` (PR #61), so its baked template lacks the entry admission requires | pin the agent-runner digest CI published after #61 merged (Renovate's bump PR); until then keep `PM_DRY_RUN=true` |
+98 -12
View File
@@ -252,7 +252,7 @@ cellular, with Wi-Fi off, and check all four — each proves a different hop:
|---|---|---|
| 1 | tunnel shows a recent handshake in the WireGuard app | the WAN forward + the server key + the peer stanza |
| 2 | `ping 192.168.2.1` (the router) | decrypt → FORWARD → LAN, and the SNAT return path |
| 3 | open `https://plex.home.keithjackson.dev` | PiHole DNS over the tunnel **and** the Gateway accepting a `host`-identity source — the specific thing a pod-networked VPN would have 403'd |
| 3 | open `https://plex.home.keithjackson.dev` | PiHole DNS over the tunnel **and** the Gateway admitting a tunnel client, which a pod-networked VPN would have 403'd. Expected source identity is `world`, like a LAN browser: measured 2026-11-09 on VIPs `.23` and `.24`, not yet observed on the Gateway's `.20` |
| 4 | visit an IP-echo site; it should show `136.51.115.252` | full-tunnel WAN egress out the house, not the carrier |
Then confirm server-side:
@@ -260,18 +260,97 @@ Then confirm server-side:
```bash
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show'
# a `latest handshake` under ~3 min, and non-zero transfer, on the expected peer.
# a `latest handshake` under ~3 min, and non-zero transfer, on the expected peer. `ds/wireguard`
# picks ONE pod; if it reads 0, ask the pod on the node holding the forward (see Troubleshooting).
```
**If check 3 fails while 2 passes**, the tunnel is fine and the problem is DNS or the Gateway, not
WireGuard. Check the client's `DNS = 192.168.2.23` and that PiHole is up.
**If 1 and 2 pass but large transfers hang**, it is MTU. The profile ships `MTU = 1420` (1500 − 60
bytes of WireGuard encapsulation); drop the client to `1280` and re-test.
**If 1 and 2 pass but pages or large transfers hang**, suspect the client MTU; see
Troubleshooting.
**If nothing handshakes at all**, you cannot tell a wrong forward from a wrong key from outside —
by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes
if the router is not delivering the packets, and a `handshake` with no completed session if it is.
**If nothing handshakes at all**, you cannot tell a wrong forward from a wrong key from the
device, by design. `wg show` cannot tell them apart either: it reads the same in both cases. The
node's conntrack table can; see Troubleshooting.
## Troubleshooting
Ask the pod on **the node carrying the traffic**: the node the WAN forward names for `away`,
`k3s-prod-1` for `home`. Sessions are per node, and the other two pods read `0`.
```bash
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n vpn-ingress get pods -o wide'
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n vpn-ingress exec <pod> -c wireguard -- wg show wg0 latest-handshakes'
# likewise: `wg show wg0 endpoints`, `wg show wg0 transfer`
```
**Use only those three subcommands to diagnose.** Never `wg showconf`, `wg show wg0 private-key` or
`wg show wg0 preshared-keys`: they print key material into your terminal and transcript.
| output | reading |
|---|---|
| `latest-handshakes`: `<public key> 0` | no handshake with this peer since the pod started |
| `endpoints`: `<public key> (none)` | the server has not accepted a packet from this peer since the pod started |
| `transfer`: `<public key> <rx> <tx>` | bytes, as the server counts them |
### The app shows TX, no RX and no "Latest handshake"
**Symptom.** The app's transmit counter climbs and receive stays at zero. On the server,
`endpoints` is `(none)` and `latest-handshakes` is `0` for that peer.
**Cause.** A key mismatch: the server's public key in the profile, or the device's public key in
`peers.conf`. WireGuard drops an initiation it cannot authenticate and sends nothing back, so
neither end logs an error. Measured 2026-11-09: a mistyped server public key in the phone's profile.
**Check.** First prove the packets arrive. The pod has no `conntrack` binary, but in `hostNetwork`
it can read the host's table:
```bash
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n vpn-ingress exec <pod> -c wireguard -- grep dport=51820 /proc/net/nf_conntrack'
```
- A line whose first tuple is `src=<the phone's public address> ... dport=51820`, marked
`[UNREPLIED]`: the datagrams reach the node and the server stays silent. It is the keys.
- No such line: the packets are not arriving. Look at the router forward, the DNS name, or the
device's network.
Then compare the keys **by copy and paste, never by eye and never retyped**: the server's public
half (the first command in Step 2's "Record it in your PASSWORD MANAGER") against the profile's
`PublicKey`, and the device's public key from the app against its stanza in `peers.conf`.
### Works on Wi-Fi, stalls on mobile data, handshake healthy
**Symptom.** `latest-handshakes` is recent and some traffic flows, but names or pages fail on
mobile data. The same device is fine on Wi-Fi.
**Cause.** Most likely a client MTU too large for the carrier's path: a tunnel MTU of `1420` means
outer packets of up to 1480 bytes. This is inferred from one before-and-after, on one phone and one
carrier (2026-11-09): `1420` failed and clearing the field fixed it. The path MTU itself was not
measured.
**Check.** In the Android WireGuard app, edit the tunnel, clear the `MTU` field or set it to `1280`,
save, then turn the tunnel off and on. A blank field means `1280` in the app's default userspace
backend (`GoBackend`: `getMtu().orElse(1280)`). On other clients set `MTU = 1280` in the profile.
`new-peer.sh` now generates `MTU = 1280` unless `WG_MTU` overrides it. A profile already on a device
does not change, so edit it by hand. If `1280` does not help, it is not the MTU: go back to
Step 6's check 3.
### The `away` profile cannot connect from home
**Symptom.** On the home network, `away` never handshakes.
**Cause.** Its Endpoint is `wan.home.keithjackson.dev`, the house's WAN address, and the router has
no hairpin NAT. WireGuard cannot use the Cloudflare Tunnel instead.
**Check.** Turn Wi-Fi off. If `away` handshakes on mobile data, the profile is fine: use the `home`
tunnel indoors, as described in
[Phone profiles: one tunnel for home, one for away](../platform/vpn-ingress/README.md#phone-profiles-one-tunnel-for-home-one-for-away).
## Revoking a device
@@ -293,10 +372,17 @@ top follow-up for this component.** Until then:
## Known limitations
* **No IPv6 in the tunnel.** A full-tunnel client with native IPv6 still reaches IPv6 sites outside
the tunnel. Functionally harmless; a privacy leak if hiding the client's network was the point.
* **Client IPs are not preserved** on the LAN side: everything arrives SNATed as the node's
address. A static route for `10.90.0.0/24` → the forwarded node, added on the router, would
preserve them — at the cost of another hand-maintained off-repo route.
* **No IPv6 in the tunnel.** What a client's IPv6 does then depends on the client. In the Android
app's default `GoBackend`, a config with one peer and a default route skips `allowFamily()`, so
Android should block IPv6 while the tunnel is up and apps fall back to IPv4. That is read from
upstream source, not observed on a device. A `--split` profile has no default route, so its IPv6
stays outside the tunnel. Expect `wg-quick` on a dual-stack Linux host to leave IPv6 outside too
(not tested).
* **Client IPs are only partly preserved.** LAN hosts and the internet see the node's address
(MASQUERADE). A static route for `10.90.0.0/24` → the forwarded node, added on the router, would
preserve them there, at the cost of another hand-maintained off-repo route. LoadBalancer VIPs are
different: Cilium handles the packet before MASQUERADE, so the source stays `10.90.0.x`, identity
`world` (BPF conntrack on prod-1, 2026-11-09, VIPs `.23` and `.24`). Any `loadBalancerSourceRanges`
must therefore list the tunnel subnet, as PiHole's do.
* **Rollouts briefly drop the forwarded node.** `maxUnavailable: 1`, so one node at a time; clients
re-handshake automatically (retry every 5s, rekey every 2 min).
+7 -9
View File
@@ -82,13 +82,10 @@ platform must render, and cites the requirement each mapping satisfies. Changing
all seven the moment it is set. Per-resource exclusion has exactly the blast radius of the
defect. The filter stays available as hardening once those seven are adjudicated — see
`platform/externaldns/helmrelease.yaml` and the register in the L0 check.
- `public` renders the record but the renderer does **not** yet set a target: the record still
resolves to the shared Gateway VIP, i.e. an RFC1918 address, so a `public` service is not in fact
publicly reachable. Fixing that is **not** a renderer change on v0.21.0 —
`source/gateway.go` reads `external-dns.alpha.kubernetes.io/target` only from the **Gateway** (and
a ListenerSet), never from an HTTPRoute, so a per-service target override is not expressible while
the public services share the `shared` Gateway. The open decision is recorded in
`platform/externaldns/helmrelease.yaml`.
- `public` renders the record but the renderer does **not** set a target: external-dns reads
`external-dns.alpha.kubernetes.io/target` only from the **Gateway** (and a ListenerSet), never from
an HTTPRoute, so the target is set once, on the `shared` Gateway — since 2026-11-08 the Cloudflare
Tunnel, proxied (`platform/gateway/gateway.yaml`, `platform/externaldns/README.md`).
- Exposure does **not** relax east-west default-deny (docs/09 §"Interaction with default-deny").
- **Requirements:** R-DNS-02/03 (docs/09), R-NET-04 (docs/03).
@@ -323,8 +320,9 @@ R-OBS-01 (the SLI must follow the rendered kind).
`public` service**. An `internal` service's route is rendered with ExternalDNS's controller
annotation set to `none`, which removes it from ExternalDNS's view entirely (see §`exposure`), so
its `<host>` exists **only** in the internal split-horizon answer PiHole serves, never in the
public zone. A `public` record's TARGET is still the Gateway VIP, not a routable address — open
operator decision, see §`exposure`.
public zone. A `public` record's TARGET is the Cloudflare Tunnel (a proxied CNAME, from the
shared Gateway's target annotation; `platform/externaldns/README.md`), and a `public` name is
not listed in PiHole unless it must stay on the LAN path.
- The `host` pattern is enforced to `<label>.home.keithjackson.dev` by the schema.
- **Requirements:** **R-TLS-01/02/03** (docs/08), **R-DNS-01/02/03** (docs/09).