Compare commits

..
Author SHA1 Message Date
renovate-bot 112ed3b6b4 chore(deps): Update Helm release kube-prometheus-stack to v93
gate / gate (pull_request) Successful in 55s
gate / conformance (pull_request) Skipped
2026-11-09 08:22:27 +00:00
jakman85 50c8719b12 Merge pull request 'fix(vpn): default client profiles to MTU 1280; correct the VPN runbook' (#159) from fix/wg-client-mtu-docs into main
gate / gate (push) Successful in 57s
gate / conformance (push) Skipped
Reviewed-on: #159
2026-11-09 03:26:02 +00:00
jakman85andClaude Opus 5.5 8f60e6103a fix(vpn): default client profiles to MTU 1280; correct the VPN runbook
gate / gate (pull_request) Successful in 57s
gate / conformance (pull_request) Skipped
new-peer.sh generated MTU 1420, which stalled traffic on mobile data
despite a healthy handshake (one phone, one carrier, 2026-11-09). Default
to 1280, overridable through WG_MTU.

Correct stale claims in the runbook and the DaemonSet header against
live BPF conntrack: LoadBalancer VIPs see 10.90.0.x as world, not a
SNATed host identity. Add a troubleshooting section.

The DaemonSet render is unchanged, so WireGuard pods do not roll.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 21:15:20 -06:00
jakman85 da0515c8c8 Merge pull request 'fix(vpn): enrol keith-phone-home and admit the WireGuard subnet to PiHole DNS' (#158) from fix/wg-home-peer-pihole-sources into main
gate / gate (push) Successful in 55s
gate / conformance (push) Skipped
Reviewed-on: #158
2026-11-09 02:50:04 +00:00
jakman85 3a2ec5a12c Merge pull request 'feat(dns): PiHole answers an explicit LAN name list, not a zone wildcard' (#157) from feat/pihole-explicit-lan-names into main
gate / gate (push) Canceled after 25s
gate / conformance (push) Canceled after 0s
Reviewed-on: #157
Reviewed-by: Keith Jackson <[email protected]>
2026-11-09 02:49:39 +00:00
jakman85andClaude Opus 5.5 f13d336630 fix(vpn): enrol keith-phone-home and admit the WireGuard subnet to PiHole DNS
gate / gate (pull_request) Successful in 1m12s
gate / conformance (pull_request) Skipped
Add the at-home peer (10.90.0.3) and list ${WG_CLIENT_CIDR} in the PiHole
DNS frontends' loadBalancerSourceRanges. wg0 is a Cilium device, so tunnel
clients reach the source-range check untranslated; a LAN-only list is
predicted to drop their DNS. Adds an L0 guard that both ranges are present
and every range is RFC1918, and corrects the README's SNAT and IPv6 claims.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 20:45:48 -06:00
jakman85andClaude Opus 5.5 8fe8d5df8a feat(dns): PiHole answers an explicit LAN name list, not a zone wildcard
gate / gate (pull_request) Successful in 1m14s
gate / conformance (pull_request) Skipped
Public names are proxied CNAMEs to the Cloudflare Tunnel since PR #156 and
work from the LAN, so they should resolve publicly there too. Replace the
address=/${BASE_DOMAIN}/ wildcard with explicit entries: the LAN-only names
(derived from every controller:none HTTPRoute), gitea/nextcloud (off
Cloudflare's 100 MB / 100 s limits), keycloak/auth, and the L4 VIPs.

Add L0 check pihole-lan-names, mutation-tested. Correct docs that still
described public names as CNAME to wan. and the cloudflared entrypoint comment.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 19:51:49 -06:00
jakman85 020e736541 Merge pull request 'feat(dns): public names via the Cloudflare Tunnel (Phase 2) — proxied CNAMEs, Plex/Headlamp LAN-only' (#156) from feat/dns-via-cloudflare-tunnel into main
gate / gate (push) Successful in 40s
gate / conformance (push) Skipped
Reviewed-on: #156
Reviewed-by: Keith Jackson <[email protected]>
2026-11-09 01:27:06 +00:00
jakman85andClaude Opus 5.5 92944486d1 feat(dns): publish public names as proxied CNAMEs to the Cloudflare Tunnel
gate / gate (pull_request) Successful in 45s
gate / conformance (pull_request) Skipped
Gateway target -> fc81938b-...cfargotunnel.com, --cloudflare-proxied globally,
L4 service publishers (unifi, crafty-mc) pinned DNS-only, Plex web UI and
Headlamp disowned (LAN/VPN-only). L0 internal-services-publish-no-public-dns
gains assertion (9) tying the Gateway target to the tunnel UUID, requiring
cloudflare-proxied=false on every service-source publisher, and keeping the
tunnel's 404 list and the unpublished set consistent. L2 dns-exposure fixture
opts out of proxying.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 17:55:41 -06:00
jakman85 2e777e13ca Merge pull request 'chore(cloudflared): rotate tunnel credential (old secret exposed in a transcript)' (#155) from feature/cloudflare-tunnel-config into main
gate / gate (push) Successful in 41s
gate / conformance (push) Skipped
Reviewed-on: #155
2026-11-08 22:11:24 +00:00
jakman85 063c8acf77 chore(cloudflared): rotate tunnel credential (old secret exposed in a transcript)
gate / gate (pull_request) Successful in 41s
gate / conformance (pull_request) Skipped
2026-11-08 16:05:11 -06:00
41 changed files with 1486 additions and 185 deletions

No files matched your search

+1 -1
View File
@@ -76,7 +76,7 @@ bug but the honest answer, because proofs are claims about a specific tree on a
|---|---|
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
| 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; **now 3× mini-PC bare metal** (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13) |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the authoritative LAN resolver** for `${BASE_DOMAIN}` on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the LAN resolver**, answering an explicit list of `${BASE_DOMAIN}` names (LAN-only apps, gitea/nextcloud/keycloak/auth, the L4 VIPs — everything else resolves publicly to Cloudflare, since 2026-11-09) on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled |
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |
+5
View File
@@ -100,6 +100,11 @@ metadata:
# external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP.
# This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443.
external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
# DNS-ONLY, NEVER PROXIED (2026-11-08). external-dns runs with --cloudflare-proxied globally (the
# gateway names point at the Cloudflare Tunnel); Cloudflare's proxy carries HTTP(S) only, so an
# orange-clouded A for these L4 ports would be broken. Required by the L0 check
# internal-services-publish-no-public-dns (9b) on every `service`-source publisher.
external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
spec:
type: LoadBalancer
externalTrafficPolicy: Cluster
+2 -2
View File
@@ -176,8 +176,8 @@ or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:
**UI access:** `https://unifi.home.keithjackson.dev:11443`. That name resolves to the VIP and not to
the Gateway on purpose: both `platform/coredns/coredns-custom.yaml` (in-cluster) and
`apps/wave-0/pihole.yaml` (LAN split-horizon) carve `unifi.${BASE_DOMAIN}` out of the
zone-wide `${GATEWAY_VIP}` answer and point it at `${UNIFI_L4_VIP}`. `${BASE_DOMAIN}` on prod is the
`apps/wave-0/pihole.yaml` (LAN split-horizon, an explicit name list since 2026-11-09) point
`unifi.${BASE_DOMAIN}` at `${UNIFI_L4_VIP}` rather than `${GATEWAY_VIP}`. `${BASE_DOMAIN}` on prod is the
apex `home.keithjackson.dev` (confirmed live in the `cluster-vars` ConfigMap) after the 2026-09-13
flip, so this hostname is current, not a dev leftover.
+19 -5
View File
@@ -83,15 +83,25 @@
# and would also fix the same latent defect in unifi and crafty; it belongs in its own change.
#
# -------------------------------------------------------------------------------------------------
# loadBalancerSourceRanges — THE LAN RESTRICTION, ENFORCED IN THE DATAPATH.
# loadBalancerSourceRanges — THE LAN + VPN RESTRICTION, ENFORCED IN THE DATAPATH.
# -------------------------------------------------------------------------------------------------
# Cilium enforces this list in eBPF on the ingress node, on the VIP, BEFORE load-balancing and
# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} is dropped by the
# datapath — it never reaches a backend and never reaches a network policy. This is what makes
# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} and ${WG_CLIENT_CIDR} is
# dropped by the datapath — it never reaches a backend and never reaches a network policy. This is what makes
# "LAN clients only" a real, tested restriction rather than a comment: VERIFIED by narrowing the
# range to a /24 that excludes the test client, after which the same query that had been answering
# stopped being answered at all, and restoring it, after which it answered again.
#
# ${WG_CLIENT_CIDR} IS LISTED BECAUSE VPN CLIENTS REACH THIS CHECK UNTRANSLATED. wg0 is one of
# Cilium's kube-proxy-replacement devices (measured 2026-11-08: `cilium-dbg status` lists eno1, wg0,
# tun0), so a tunnel client's query is load-balanced as it enters from wg0, ahead of the iptables
# MASQUERADE in platform/vpn-ingress/daemonset.yaml, and the source tested here is still 10.90.0.x
# rather than the node's address. Without this entry a full-tunnel client's DNS is predicted to die
# with DROP_NOT_IN_SRC_RANGE (derived from Cilium's source; NOT observed live, because no peer has
# ever completed a handshake). The range is private and reachable only by a key in peers.conf, so
# PiHole stays off the internet; conformance/tests/l0-static/pihole-dns-source-ranges fails the gate
# if either entry goes missing or any range here stops being RFC1918.
#
# It is also the layer that survives the SNAT above. A CiliumNetworkPolicy cannot express this
# restriction once the source has been rewritten: measured on this cluster, the forwarded query
# arrives from the ingress node's internal router address and Cilium resolves that source to the
@@ -113,6 +123,7 @@ spec:
externalTrafficPolicy: Cluster
loadBalancerSourceRanges:
- ${LAN_CIDR}
- ${WG_CLIENT_CIDR}
selector:
app.kubernetes.io/name: pihole
ports:
@@ -133,6 +144,7 @@ spec:
externalTrafficPolicy: Cluster
loadBalancerSourceRanges:
- ${LAN_CIDR}
- ${WG_CLIENT_CIDR}
selector:
app.kubernetes.io/name: pihole
ports:
@@ -158,7 +170,9 @@ spec:
# "Policy denied ... identity world" drops in `cilium-dbg monitor`). So `world` is the only peer
# selector that can describe LAN traffic that has been through the LoadBalancer, and the set of
# addresses that can actually BE that traffic is fixed by loadBalancerSourceRanges before this
# policy is consulted. Together: ${LAN_CIDR} in the datapath, udp/53 + tcp/53 here.
# policy is consulted. Together: ${LAN_CIDR} + ${WG_CLIENT_CIDR} in the datapath, udp/53 + tcp/53
# here. A VPN client delivered to a backend on its own ingress node keeps its 10.90.0.x source,
# which Cilium also classifies as `world`, so this rule needs no new peer for it.
#
# This is NOT wider than the precedent it follows. unifi-allow-l4-ingress and crafty-allow-ingress
# admit `fromEntities: [world, cluster, host]` on their L4 ports with NO source-range filter at all;
@@ -187,7 +201,7 @@ spec:
app.kubernetes.io/name: pihole
ingress:
- fromEntities:
- world # LAN clients, post-LoadBalancer: see the block above for why not fromCIDR
- world # LAN + VPN clients, post-LoadBalancer: see the block above for why not fromCIDR
- host # this node's kubelet probes
- remote-node # the node-to-node hop that forwards a query to wherever the pod runs
toPorts:
+1 -1
View File
@@ -73,7 +73,7 @@ spec:
- name: FTLCONF_dns_rateLimit_count
value: '0'
- name: FTLCONF_misc_dnsmasq_lines
value: local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
value: local-ttl=60;address=/gitea.${BASE_DOMAIN}/${GATEWAY_VIP};address=/nextcloud.${BASE_DOMAIN}/${GATEWAY_VIP};address=/keycloak.${BASE_DOMAIN}/${GATEWAY_VIP};address=/auth.${BASE_DOMAIN}/${GATEWAY_VIP};address=/headlamp.${BASE_DOMAIN}/${GATEWAY_VIP};address=/jackett.${BASE_DOMAIN}/${GATEWAY_VIP};address=/pihole.${BASE_DOMAIN}/${GATEWAY_VIP};address=/plex.${BASE_DOMAIN}/${GATEWAY_VIP};address=/qbittorrent.${BASE_DOMAIN}/${GATEWAY_VIP};address=/radarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/semaphore.${BASE_DOMAIN}/${GATEWAY_VIP};address=/sonarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
- name: FTLCONF_webserver_api_password
valueFrom:
secretKeyRef:
+2
View File
@@ -302,6 +302,8 @@ metadata:
namespace: plex
labels:
app.kubernetes.io/name: plex
annotations:
external-dns.alpha.kubernetes.io/controller: none
spec:
parentRefs:
- name: shared
+58 -31
View File
@@ -85,7 +85,7 @@ env:
# Default is LOCAL, which answers only clients on the same subnet as the listening interface. The
# pod's interface is on the CLUSTER pod CIDR, so every LAN client would be refused. ALL is
# required for a LoadBalancer-fronted resolver; what keeps PiHole from being an OPEN resolver is
# the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}]` on the DNS Services,
# the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}, ${WG_CLIENT_CIDR}]` on the DNS Services,
# enforced by Cilium in the datapath before load-balancing, plus a CiliumNetworkPolicy that admits
# nothing but udp/53 and tcp/53. See apps/wave-0/pihole-dns.yaml.
- name: FTLCONF_dns_listeningMode
@@ -108,41 +108,61 @@ env:
# `loadBalancer.mode: dsr`, which preserves the client address.
- name: FTLCONF_dns_rateLimit_count
value: "0"
# THE SPLIT-HORIZON ANSWERS. A dnsmasq `address=/<domain>/<ip>` entry answers that domain AND
# every subdomain of it, and dnsmasq resolves overlapping entries by LONGEST MATCHING DOMAIN —
# so the first line is a wildcard for the whole zone and each later line is a specific override.
# PRECEDENCE IS THEREFORE STRUCTURAL, not order-dependent: `unifi.${BASE_DOMAIN}` matches both
# line 1 and line 2, line 2 is the longer domain, line 2 wins. Rewriting these in any order
# changes nothing.
# THE LAN ANSWERS — AN EXPLICIT LIST, NOT A ZONE WILDCARD (2026-11-09). A dnsmasq
# `address=/<domain>/<ip>` entry answers that domain AND every subdomain of it (on LABEL
# boundaries: `pihole.` does not match `pihole-dns.`), and dnsmasq resolves overlapping entries by
# LONGEST MATCHING DOMAIN, so the order below is irrelevant. Every name NOT listed here is
# FORWARDED to the upstreams above and gets the PUBLIC answer.
#
# A NEWLY ONBOARDED SERVICE NEEDS NO CHANGE HERE. Every catalog ingressRoute host is
# `<label>.${BASE_DOMAIN}` served by the shared Gateway, so the wildcard already answers it with
# ${GATEWAY_VIP} — the same guarantee, and the same reasoning, as the `template IN A` block in
# platform/coredns/coredns-custom.yaml. Add a line here ONLY for a service that does not live
# behind the Gateway: an L4 LoadBalancer on its own pinned VIP. Those are enumerated from the
# repo, not from memory — grep `external-dns.alpha.kubernetes.io/hostname` on a Service, which
# is the annotation that means "this Service publishes its own hostname at its own address":
# apps/vms/unifi/50-service-l4.yaml (unifi -> ${UNIFI_L4_VIP}) and
# platform/crafty/service-l4.yaml (crafty-mc -> ${CRAFTY_L4_VIP}). Those two, plus PiHole's own
# resolver address, are the complete set of exceptions today.
# WHY NO WILDCARD ANY MORE. Until 2026-11-09 the first entry was `address=/${BASE_DOMAIN}/
# ${GATEWAY_VIP}`, answering the whole zone. That made sense while every public record was a
# DNS-only CNAME to `wan.${BASE_DOMAIN}` (no NAT hairpin on this router, so the LAN NEEDED a
# private answer). Since PR #156 every public name is a PROXIED CNAME to the Cloudflare Tunnel
# (platform/externaldns/README.md), which works identically from the LAN — so a name that is
# public now SHOULD resolve publicly from the LAN too, and the LAN no longer depends on PiHole for
# it: a client that bypasses PiHole (DoH browser, phone private DNS, the router's fallback) gets
# the same working answer. The wildcard had also become actively misleading: it answered names
# that exist nowhere (`wan.` itself resolved to the Gateway VIP on the LAN).
#
# AAAA needs nothing: dnsmasq answers a AAAA query for an `address=`d name with NODATA
# (NOERROR, no answer), so IPv6-capable clients fall back to the A record instead of getting
# NXDOMAIN — the same behaviour coredns-custom hand-writes a `template IN AAAA` block for.
# WHAT IS LISTED, and why each — enforced by conformance/tests/l0-static/pihole-lan-names:
# -> ${GATEWAY_VIP}, LAN-ONLY: every HTTPRoute disowned from public DNS
# (`external-dns.alpha.kubernetes.io/controller: none`, i.e. catalog `exposure: internal`,
# plus plex and headlamp) — jackett, pihole, plex, qbittorrent, radarr, semaphore, sonarr,
# headlamp. These have NO public record and the tunnel 404s them, so THIS LINE IS THEIR ONLY
# PATH: drop one and the app is unreachable everywhere. A NEWLY ONBOARDED `internal` SERVICE
# MUST BE ADDED HERE (the L0 check fails until it is). A new `public` service needs nothing.
# -> ${GATEWAY_VIP}, PUBLIC BUT KEPT ON THE LAN PATH: gitea (git push, LFS, container registry)
# and nextcloud (uploads) — Cloudflare's proxy caps a request body at 100 MB and a response
# at 100 s, which a large push or upload exceeds; keycloak and auth (oauth2-proxy) — so the
# OIDC login of those two LAN-path apps stays on the same path as the apps themselves.
# -> their own L4 VIPs: unifi (${UNIFI_L4_VIP}) and crafty-mc (${CRAFTY_L4_VIP}) are not behind
# the Gateway (grep `external-dns.alpha.kubernetes.io/hostname` on a Service:
# apps/vms/unifi/50-service-l4.yaml, platform/crafty/service-l4.yaml); their public records
# are DNS-only A records to the same RFC1918 VIPs, which this router's rebind protection
# strips, so the LAN answer is still required. pihole-dns (${PIHOLE_DNS_VIP}) is PiHole's own
# resolver address; plex-vip (${PLEX_L4_VIP}) is below.
# NOT listed, deliberately, so they resolve via Cloudflare from the LAN as from anywhere: the apex,
# grafana, crafty, seerr. In-cluster (platform/coredns/coredns-custom.yaml) and node
# (/etc/hosts pins, node_baseline_hosts_pins) resolution are separate and unchanged.
#
# AAAA: dnsmasq answers a AAAA query for an `address=`d name with NODATA (NOERROR, no answer),
# so IPv6-capable clients fall back to the A record instead of getting NXDOMAIN — the same
# behaviour coredns-custom hand-writes a `template IN AAAA` block for. Unlisted (proxied) names
# now get Cloudflare's real A and AAAA answers.
#
# Array-valued FTLCONF variables are SEMICOLON-separated. A JSON-looking list is accepted and
# then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal
# characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid"
# while resolution for the zone does not work. Confirmed both ways against the live image.
# plex-vip -> ${PLEX_L4_VIP} (added 2026-10-04) is the LAN HALF of Plex's single advertised
# hostname. Plex publishes ONE URL to plex.tv for both local and remote clients
# (catalog/plex.yaml ADVERTISE_IP): on the LAN this answer points it at the tcp/32400
# LoadBalancer, while Cloudflare answers the same name publicly with the WAN address, NATed to
# the same VIP. WITHOUT THIS LINE the entry above answers it with ${GATEWAY_VIP} via the zone
# wildcard, where NOTHING listens on 32400 — measured 2026-10-04, plex-vip.${BASE_DOMAIN}
# resolved to the Gateway VIP and a connect to :32400 there returned http=000. That is a silent
# failure: the Plex web UI on plex.${BASE_DOMAIN} keeps working through the Gateway on 443 while
# every client that takes the ADVERTISED connection cannot reach the server at all.
#
# plex-vip -> ${PLEX_L4_VIP} (added 2026-10-04) is the name of the tcp/32400 LoadBalancer
# (apps/wave-0/plex-remote.yaml) and the domain of the certificate Plex serves there
# (catalog/plex.yaml CERTDOMAIN / customCertificateDomain, Secret `plex-vip-tls`). It is NOT what
# Plex advertises any more (ADVERTISE_IP is a resolver-independent plex.direct name since
# 2026-10-05) and it has NO public record, so this entry is its only resolution — kept for
# reaching the VIP by name with a matching certificate from a LAN shell. Measured 2026-10-04,
# before it existed the zone wildcard answered it with ${GATEWAY_VIP}, where nothing listens on
# 32400; without it now, the name is NXDOMAIN.
#
# ── local-ttl=60 — THE SPLIT-HORIZON ANSWER MUST BE CACHEABLE ────────────────────────────────
# MEASURED 2026-10-18, diagnosing "DNS flapping between the cluster ingress and the public IP":
@@ -172,10 +192,17 @@ env:
# cache expires, the router falls back, and the public answer latches again. That is a real
# outage, and the fix for it is PiHole availability, not DNS caching.
#
# SINCE 2026-11-09 THE LATCH IS MOSTLY HARMLESS. The measurement above predates the tunnel: the
# public answer was then `wan.`, unreachable from the LAN. Now the public answer for every
# proxied name is Cloudflare, which works from the LAN, and those names are not listed below at
# all. A latch still bites only the listed names: gitea/nextcloud/keycloak/auth fall back to
# Cloudflare (working, but under its 100 MB / 100 s limits); the LAN-only names and the L4 VIPs
# get NXDOMAIN or a stripped answer until the router re-queries PiHole.
#
# Applies to every `address=` reply below (dnsmasq: local-ttl governs replies made from
# configuration, /etc/hosts and DHCP leases), so one directive covers all five names.
# configuration, /etc/hosts and DHCP leases), so one directive covers every listed name.
- name: FTLCONF_misc_dnsmasq_lines
value: "local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
value: "local-ttl=60;address=/gitea.${BASE_DOMAIN}/${GATEWAY_VIP};address=/nextcloud.${BASE_DOMAIN}/${GATEWAY_VIP};address=/keycloak.${BASE_DOMAIN}/${GATEWAY_VIP};address=/auth.${BASE_DOMAIN}/${GATEWAY_VIP};address=/headlamp.${BASE_DOMAIN}/${GATEWAY_VIP};address=/jackett.${BASE_DOMAIN}/${GATEWAY_VIP};address=/pihole.${BASE_DOMAIN}/${GATEWAY_VIP};address=/plex.${BASE_DOMAIN}/${GATEWAY_VIP};address=/qbittorrent.${BASE_DOMAIN}/${GATEWAY_VIP};address=/radarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/semaphore.${BASE_DOMAIN}/${GATEWAY_VIP};address=/sonarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
# THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
# apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
# carries the full rationale and the measurement that found it.
+8 -1
View File
@@ -347,7 +347,14 @@ lifecycle:
# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took.
hardware:
gpu: intel-quicksync
exposure: public # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET)
# EXPOSURE: internal (operator decision 2026-11-08, Cloudflare Tunnel cut-over). The plex.${BASE_DOMAIN}
# web UI is LAN/VPN-only: `internal` renders `external-dns ... controller: none` on the HTTPRoute, so
# external-dns DELETES the public plex record and its TXT, and platform/cloudflared/config.yaml maps the
# name to http_status:404 so it can never be served through the tunnel either. REMOTE STREAMING IS NOT
# AFFECTED: it never used this name. It rides plex.tv + the advertised plex.direct URL (ADVERTISE_IP
# above) to the raw tcp/32400 L4 VIP (${PLEX_L4_VIP}, apps/wave-0/plex-remote.yaml) via the WAN
# tcp/32400 forward — none of which this field touches. See runbooks/plex-remote-access.md.
exposure: internal # web UI LAN/VPN-only; remote streaming = plex.tv + WAN tcp/32400 (docs/09; R-DNS/R-NET)
egress: internet # normal WAN egress (docs/03; R-NET-05)
storage:
- name: config
@@ -92,9 +92,10 @@
# the NAME axis only. R-NET-04 / the edge port-forward posture owns the other.
# - Anything about the seven non-catalog publishers' own correctness. The register records WHAT
# each publishes so the claim is reviewable, which is all a static check can offer.
# - Anything about the five `public` services' TARGET. They still resolve to the Gateway VIP
# rather than a routable address; that is an open operator decision recorded in
# platform/externaldns/helmrelease.yaml, and asserting a target here would hard-code a guess.
# - That the public names RESOLVE to Cloudflare at runtime. check.sh (9) pins the TREE: the
# Gateway target is the tunnel UUID in platform/cloudflared/config.yaml, proxying is on, the L4
# `service` publishers opt out, and every name the tunnel 404s is unpublished. Whether Cloudflare
# then answers anycast for them is a live read (`dig @1.1.1.1`), not a static one.
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
@@ -29,7 +29,17 @@
# ci/l0.sh stage (whole-repo-invariants-wired fails the gate if one appears);
# (8) the external-dns HelmRelease declares `annotationPrefix` explicitly and it matches the
# prefix of EVERY external-dns annotation in the tree and in catalog/render.py — added after
# the 2026-11-07 v0.23.0 default-prefix change silently voided (1)-(3) at runtime.
# the 2026-11-07 v0.23.0 default-prefix change silently voided (1)-(3) at runtime;
# (9) THE CLOUDFLARE TUNNEL WIRING (2026-11-08) — every public name is a PROXIED CNAME to the tunnel:
# (a) the shared Gateway's `target` is EXACTLY `<tunnel:>.cfargotunnel.com`, the UUID read from
# platform/cloudflared/config.yaml, and the HelmRelease sets `--cloudflare-proxied`;
# (b) every `service`-source publisher carries `cloudflare-proxied: "false"` (L4 A records stay
# DNS-only), and no HTTPRoute opts out of proxying (a DNS-only tunnel CNAME is dead);
# (c) no hostname the tunnel maps to `http_status:404` has a public record — no undisowned route
# names it, and the only `service`-source publishers of one are the declared L4 DNS-only
# names (unifi, crafty-mc); conversely every disowned route's hostname IS 404'd by the
# tunnel, so the "unpublished" and "refused at the tunnel" lists cannot drift apart;
# (d) the operator's LAN/VPN-only names (plex, headlamp) are disowned everywhere they appear.
#
# WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC, offline, cluster-free):
# - That the record is gone from Cloudflare. Live read; belongs beside
@@ -39,8 +49,11 @@
# both the A and its `a-` TXT within ~60s) — but a manual proof is not a gate.
# - That an internal name is UNREACHABLE. A port-forward plus a literal IP needs no DNS record.
# This check owns the NAME axis only.
# - Anything about the five public services' TARGET. They still answer with the Gateway VIP; the
# routable-target decision is open and recorded in platform/externaldns/helmrelease.yaml.
# - That the tunnel actually SERVES the public names, or that Cloudflare honoured `proxied`. (9)
# proves the tree asks for a proxied CNAME to the right tunnel; the live answer (`dig @1.1.1.1`
# returning Cloudflare anycast) is a runtime read. Nor does (8)/(9) prove the RUNNING controller
# honours the prefix — a future chart bump that drops it would publish PROXIED A records at the
# Gateway VIP; (8) only keeps the tree and the HelmRelease value consistent.
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
#
# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns
@@ -141,9 +154,12 @@ def docs_of(p: Path):
# unadjudicated set cannot grow without a review event. Same ratchet reasoning as
# KNOWN_EXTERNAL_MONITORS in servicemonitor-ports-resolve.
#
# EVERY ENTRY HERE IS A NAME THAT STILL ANSWERS WITH AN RFC1918 ADDRESS IN A PUBLIC ZONE. That is
# the honest state after this change, and it is deliberate: converting them is an operator
# decision per name (is grafana meant to be publicly resolvable at all?), not a renderer change,
# WHAT THE ENTRIES PUBLISH NOW (2026-11-08). Every HTTPRoute entry that is not disowned publishes a
# PROXIED CNAME to the Cloudflare Tunnel (the shared Gateway's `target`, assertion 9a), so no origin
# address of ours is in the public zone for it. Headlamp is disowned (operator: LAN/VPN-only, 9d).
# The two `service`-source entries still publish their RFC1918 LB VIP as a DNS-only A record —
# deliberate (L4 ports cannot ride Cloudflare's HTTP proxy), and pinned DNS-only by 9b. Whether each
# remaining name should be public at all stays an operator decision per name, not a renderer change,
# and a cluster-wide ExternalDNS opt-in filter — the available hardening, see
# platform/externaldns/helmrelease.yaml — would DELETE all of them at once the moment it is set.
#
@@ -151,27 +167,28 @@ def docs_of(p: Path):
# =============================================================================================
KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = {
"platform/oauth2-proxy/httproute.yaml:oauth2-proxy/oauth2-proxy":
"publishes auth.${BASE_DOMAIN} -> the Gateway VIP. The oauth2-proxy login endpoint is a "
"publishes auth.${BASE_DOMAIN} -> the tunnel (proxied CNAME). The oauth2-proxy login endpoint is a "
"platform component, not a catalog service; it must be reachable by every browser that "
"any forward-auth app redirects, so whether it is PUBLICLY resolvable is coupled to "
"whether the five public services are — an operator decision, not a renderer one.",
"platform/keycloak/httproute.yaml:keycloak/keycloak":
"publishes keycloak.${BASE_DOMAIN} -> the Gateway VIP, and it is the route that actually "
"publishes keycloak.${BASE_DOMAIN} -> the tunnel (proxied CNAME), and it is the route that actually "
"OWNS that record (the TXT registry entry reads resource=httproute/keycloak/keycloak, not "
"the catalog-rendered keycloak-8080 route, because external-dns merges duplicate "
"hostnames and this one won). Keycloak is `exposure: public` in the catalog, so the name "
"is meant to be public either way; it is listed because the PUBLISHING object is the "
"hand-authored platform route, which carries no exposure field.",
"platform/monitoring/grafana-httproute.yaml:monitoring/grafana":
"publishes grafana.${BASE_DOMAIN} -> the Gateway VIP. An admin dashboard; almost "
"certainly should not be publicly resolvable, but it is not a catalog service and "
"withdrawing it is an operator call with the same LAN-resolution prerequisite as the six "
"internal services (Pi-hole must answer first).",
"publishes grafana.${BASE_DOMAIN} -> the tunnel (proxied CNAME). An admin dashboard "
"behind Keycloak; it is not a catalog service, so withdrawing it is an operator call per "
"name (it stayed public in the 2026-11-08 tunnel decision).",
"platform/headlamp/httproute.yaml:headlamp/headlamp":
"publishes headlamp.${BASE_DOMAIN} -> the Gateway VIP. A Kubernetes web UI with native "
"Keycloak OIDC; same adjudication as grafana.",
"publishes NOTHING since 2026-11-08: the route carries `controller: none` (operator "
"decision: Headlamp is LAN/VPN-only) and the tunnel 404s the name. A Kubernetes web UI "
"with native Keycloak OIDC; listed because it is a hand-authored route, and pinned "
"disowned by (9d).",
"platform/homepage/httproute.yaml:homepage/homepage":
"publishes the APEX ${BASE_DOMAIN} itself -> the Gateway VIP, via the https-apex "
"publishes the APEX ${BASE_DOMAIN} itself -> the tunnel (proxied CNAME), via the https-apex "
"listener. The apex is not a per-service name and cannot be withdrawn by an exposure "
"declaration.",
"runbooks/serving/httproute.yaml:runbooks/runbooks":
@@ -180,16 +197,16 @@ KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = {
"entry names the homepage route). Deliberately un-authenticated — see the manifest's own "
"header for why gating runbooks on Keycloak is a loop with no exit.",
"platform/crafty/httproute.yaml:crafty/crafty":
"publishes crafty.${BASE_DOMAIN} -> the Gateway VIP (the Minecraft admin UI, gated by an "
"publishes crafty.${BASE_DOMAIN} -> the tunnel (proxied CNAME) (the Minecraft admin UI, gated by an "
"oauth2-proxy sidecar). Platform component, no exposure field.",
"platform/crafty/service-l4.yaml:crafty/crafty-l4-tcp":
"the `service` SOURCE, not a route: publishes crafty-mc.${BASE_DOMAIN} -> "
"${CRAFTY_L4_VIP} (192.168.2.22) for the raw Minecraft TCP/UDP game ports the HTTPS "
"Gateway cannot express. A game server is the one case where a public name may be "
"genuinely wanted; the address it answers with is still RFC1918.",
"genuinely wanted; the address it answers with is still RFC1918, DNS-only (9b).",
"apps/vms/unifi/50-service-l4.yaml:unifi/unifi-l4-tcp":
"the `service` source: publishes unifi.${BASE_DOMAIN} -> ${UNIFI_L4_VIP} "
"(192.168.2.21) for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored "
"(192.168.2.21), DNS-only (9b), for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored "
"under apps/vms (a VM, not a rendered container service); the catalog's only VM entry is "
"vm-demo, which declares no ingressRoutes and publishes nothing.",
}
@@ -515,6 +532,113 @@ check(not bad_render,
f"(8d) catalog/render.py emits external-dns keys under {bad_render}, but the controller "
f"reads {ANNOTATION_PREFIX!r}")
# =============================================================================================
# (9) THE CLOUDFLARE TUNNEL WIRING (2026-11-08). The public names are PROXIED CNAMEs to the tunnel.
# =============================================================================================
CF_CONFIG = ROOT / "platform/cloudflared/config.yaml"
GATEWAY_FILE = ROOT / "platform/gateway/gateway.yaml"
TARGET_KEY = ANNOTATION_PREFIX + "target"
PROXIED_KEY = ANNOTATION_PREFIX + "cloudflare-proxied"
UUID_RE = re.compile(r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$")
# The L4 names the tunnel 404s that are NONETHELESS published — as DNS-only A records from the
# `service` source, at their own LB VIP, because they are not HTTP and never touch the tunnel
# (operator decision 2026-11-08). Shrink-only, and must agree with the tunnel's 404 list (9c).
L4_DNS_ONLY_NAMES = {"unifi.${BASE_DOMAIN}", "crafty-mc.${BASE_DOMAIN}"}
# The operator's LAN/VPN-only HTTP names (2026-11-08). Pinned here as well as in the tunnel config, so
# deleting one from config.yaml cannot quietly also release it from (9c).
OPERATOR_LAN_ONLY_NAMES = {"plex.${BASE_DOMAIN}", "headlamp.${BASE_DOMAIN}"}
cf = next(iter(docs_of(CF_CONFIG)), None)
check(cf is not None and isinstance(cf.get("ingress"), list),
f"(9) {CF_CONFIG.relative_to(ROOT)} unreadable or has no `ingress` list — cannot prove what "
f"the tunnel refuses or which tunnel the Gateway must target")
cf = cf or {}
tunnel = str(cf.get("tunnel") or "")
check(bool(UUID_RE.match(tunnel)),
f"(9a) platform/cloudflared/config.yaml `tunnel:` is {tunnel!r}, not a literal tunnel UUID — "
f"the Gateway target is derived from it")
gw = next((d for d in docs_of(GATEWAY_FILE) if d.get("kind") == "Gateway"
and ident(d) == ("gateway", "shared")), None)
check(gw is not None, f"(9a) {GATEWAY_FILE.relative_to(ROOT)} holds no Gateway gateway/shared")
want_target = f"{tunnel}.cfargotunnel.com"
got_target = annotations_of(gw or {}).get(TARGET_KEY)
check(got_target == want_target,
f"(9a) Gateway gateway/shared `{TARGET_KEY}` is {got_target!r}; it must be EXACTLY "
f"{want_target!r} (the tunnel in platform/cloudflared/config.yaml). Any other value publishes "
f"every public name at the wrong place — a dead cfargotunnel.com name (Cloudflare 1016), or "
f"an origin address of ours.")
check("--cloudflare-proxied" in [str(a) for a in (extra if isinstance(extra, list) else [])]
or (isinstance(extra, dict) and str(extra.get("cloudflare-proxied", "")).lower() == "true"),
"(9a) external-dns HelmRelease extraArgs must include `--cloudflare-proxied`: a DNS-only "
"CNAME to <uuid>.cfargotunnel.com resolves to nothing routable, so every public name is dead")
for rel, d in dns_services:
if is_disowned(d):
continue
if HOSTNAME_KEY not in annotations_of(d):
continue
v = annotations_of(d).get(PROXIED_KEY)
check(str(v).lower() == "false",
f"(9b) {rel} Service {'/'.join(ident(d))} publishes via the `service` source but carries "
f"`{PROXIED_KEY}: {v!r}`, not \"false\". With --cloudflare-proxied global it would become "
f"an orange-clouded A record, which cannot carry L4 traffic and hides nothing (the LB VIP "
f"is RFC1918). Every `service`-source publisher must opt out explicitly.")
for rel, d in routes:
v = annotations_of(d).get(PROXIED_KEY)
check(v is None or str(v).lower() == "true",
f"(9b) {rel} HTTPRoute {'/'.join(ident(d))} sets `{PROXIED_KEY}: {v!r}`. A gateway-sourced "
f"record is a CNAME to the tunnel; DNS-only, it is dead. Disown the route instead "
f"(`{CONTROLLER_KEY}: none`) if the name should not be public.")
refused = {str(r.get("hostname")) for r in cf.get("ingress") or []
if isinstance(r, dict) and r.get("hostname") and r.get("service") == "http_status:404"}
check(len(refused) > 0, "(9c) the tunnel config 404s no hostname — parse contract broken (the "
"operator's LAN/VPN-only list is known to be non-empty)")
published_l4: set[str] = set()
for rel, d in routes:
hosts = {str(h) for h in (d.get("spec") or {}).get("hostnames") or []}
leaked = sorted(hosts & refused)
if is_disowned(d):
unrefused = sorted(hosts - refused)
check(not unrefused,
f"(9c) {rel} HTTPRoute {'/'.join(ident(d))} is disowned (no public record) but the "
f"tunnel does not 404 {unrefused}: add them to platform/cloudflared/config.yaml above "
f"the wildcard, or the two halves of \"LAN/VPN-only\" have drifted apart.")
continue
check(not leaked,
f"(9c) {rel} HTTPRoute {'/'.join(ident(d))} PUBLISHES {leaked}, which the tunnel maps to "
f"http_status:404 (LAN/VPN-only). The name would be a public proxied record that serves "
f"only 404s — disown the route (`{CONTROLLER_KEY}: none`) or adjudicate the name.")
for rel, d in dns_services:
if is_disowned(d):
continue
hosts = {h.strip() for h in str(annotations_of(d).get(HOSTNAME_KEY) or "").split(",")
if h.strip()}
for h in sorted(hosts & refused):
published_l4.add(h)
check(h in L4_DNS_ONLY_NAMES,
f"(9c) {rel} Service {'/'.join(ident(d))} publishes {h}, which the tunnel 404s, and it "
f"is not a declared L4 DNS-only name {sorted(L4_DNS_ONLY_NAMES)}")
check(L4_DNS_ONLY_NAMES <= refused,
f"(9c) L4 DNS-only names {sorted(L4_DNS_ONLY_NAMES - refused)} are not 404'd by the tunnel; a "
f"stray CNAME to the tunnel for them would hit the Gateway's catch-all instead of a 404")
check(published_l4 == L4_DNS_ONLY_NAMES,
f"(9c) L4_DNS_ONLY_NAMES is shrink-only: {sorted(L4_DNS_ONLY_NAMES - published_l4)} no longer "
f"published by any `service`-source publisher — delete them from the list")
check(OPERATOR_LAN_ONLY_NAMES <= refused,
f"(9d) operator LAN/VPN-only names {sorted(OPERATOR_LAN_ONLY_NAMES - refused)} are missing "
f"from the tunnel's http_status:404 rules (platform/cloudflared/config.yaml)")
for name in sorted(OPERATOR_LAN_ONLY_NAMES):
naming = [(rel, d) for rel, d in routes
if name in {str(h) for h in (d.get("spec") or {}).get("hostnames") or []}]
check(len(naming) > 0, f"(9d) no HTTPRoute names {name} — stale OPERATOR_LAN_ONLY_NAMES entry")
for rel, d in naming:
check(is_disowned(d),
f"(9d) {rel} HTTPRoute {'/'.join(ident(d))} names LAN/VPN-only {name} but is not "
f"disowned, so external-dns publishes it publicly (operator decision 2026-11-08)")
# =============================================================================================
if FAILURES:
for f in FAILURES:
@@ -0,0 +1,39 @@
# =============================================================================
# L0 STATIC — PiHole's DNS frontends admit the LAN and the VPN, and nothing public
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# loadBalancerSourceRanges is the only thing keeping PiHole from being an open resolver, and it
# must also carry the WireGuard client subnet, because Cilium load-balances tunnel traffic on wg0
# before the iptables MASQUERADE (so the source it tests is still 10.90.0.x). check.sh's header
# states the assertions and what they do not prove.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library + PyYAML);
# conformance/run.sh --layer l0 runs it directly and this Chainsaw Test wraps the SAME script.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: pihole-dns-source-ranges
labels:
conformance.home.k3s/R-DNS-03: "true"
annotations:
conformance.home.k3s/requirements: "R-DNS-03"
conformance.home.k3s/layer: "L0"
# WHOLE-REPO SCOPE: ci/touched-requirements.sh maps apps/* to no R-DNS requirement, so a change
# to these Services would otherwise select nothing.
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Every PiHole LoadBalancer Service lists both the LAN and the WireGuard client subnet in
loadBalancerSourceRanges, and every range resolves to RFC1918, so VPN clients can resolve
and PiHole stays unreachable from the internet.
spec:
steps:
- name: assert-pihole-source-ranges
description: >-
Collect the PiHole LoadBalancer Services, resolve each source range through cluster-vars,
and assert both required ranges are present and every range is private.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,146 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — pihole-dns-source-ranges
# Serves: R-DNS-03 (LAN split-horizon DNS — spec docs/09).
#
# THE BUG CLASS THIS CLOSES. PiHole's LoadBalancer frontends are the only thing stopping it being
# an open resolver, and their `loadBalancerSourceRanges` is pulled in two opposite directions:
# * Too WIDE, and the resolver is open: a public range, or 0.0.0.0/0, in this list is the whole
# restriction gone, with nothing else in the path to catch it (the CiliumNetworkPolicy admits
# `world` on :53, because that is the only peer that can describe post-LoadBalancer traffic).
# * Too NARROW, and VPN DNS dies: wg0 is a Cilium kube-proxy-replacement device, so a tunnel
# client's query is load-balanced with its 10.90.0.x source intact, ahead of the iptables
# MASQUERADE. A LAN-only list is predicted to drop it as DROP_NOT_IN_SRC_RANGE, and the
# tunnel would still report a healthy handshake.
#
# GIVEN the repository as git tracks it,
# WHEN every type=LoadBalancer Service in namespace `pihole` is collected and each source range
# is resolved through cluster-vars (base value AND the prod overlay's replacement),
# THEN (1) the frontends exist and cover both udp/53 and tcp/53;
# (2) every frontend lists ${LAN_CIDR} and ${WG_CLIENT_CIDR};
# (3) every range resolves to a valid IPv4 CIDR inside RFC1918 — so no 0.0.0.0/0, no public
# range, and no unresolvable token that could hide one;
# AND (4) the check is NON-VACUOUS: a real checkout and a floor on the assertion count.
#
# WHAT THIS DOES NOT PROVE: that a query over the tunnel is actually answered. That needs a
# handshake from a real client; see platform/vpn-ingress/README.md.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
import ipaddress, re, subprocess, sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
FAILURES: list[str] = []
CHECKS = 0
REQUIRED_TOKENS = ("LAN_CIDR", "WG_CLIENT_CIDR")
RFC1918 = [ipaddress.ip_network(n) for n in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16")]
TOKEN = re.compile(r"^\$\{([A-Z0-9_]+)\}$")
def check(cond, msg):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(msg)
def docs_of(rel: str):
try:
return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace"))
if isinstance(d, dict)]
except Exception:
return []
FILES = [p for p in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
capture_output=True, text=True, check=True).stdout.split("\0")
if p]
YAML_FILES = [f for f in FILES if re.search(r"\.ya?ml$", f)]
check(len(YAML_FILES) > 50, f"only {len(YAML_FILES)} YAML files seen — not a real checkout")
# ------------------------------------------------- every value a ${TOKEN} can take on this tree
values: dict[str, set[str]] = {}
for d in docs_of("clusters/base/cluster-vars.yaml"):
if d.get("kind") == "ConfigMap":
for k, v in (d.get("data") or {}).items():
values.setdefault(k, set()).add(str(v))
for d in docs_of("clusters/prod/kustomization.yaml"):
for p in d.get("patches") or []:
ops = p.get("patch") if isinstance(p, dict) else None
ops = yaml.safe_load(ops) if isinstance(ops, str) else ops
for op in ops if isinstance(ops, list) else []:
m = re.fullmatch(r"/data/([A-Z0-9_]+)", str(op.get("path", "")))
if m and op.get("op") in ("add", "replace"):
values.setdefault(m.group(1), set()).add(str(op.get("value")))
for tok in REQUIRED_TOKENS:
check(values.get(tok), f"clusters/base/cluster-vars.yaml defines no {tok} (R-DNS-03)")
def assert_private(where: str, cidr: str):
try:
net = ipaddress.ip_network(cidr, strict=False)
except ValueError:
check(False, f"{where}: {cidr!r} is not a valid CIDR (R-DNS-03)")
return
check(net.version == 4 and any(net.subnet_of(r) for r in RFC1918),
f"{where}: {cidr} is not inside RFC1918, so it opens PiHole beyond the LAN and the VPN "
"(R-DNS-03)")
# ---------------------------------------------------------------------------- the frontends
frontends = []
for rel in YAML_FILES:
for d in docs_of(rel):
md, spec = d.get("metadata") or {}, d.get("spec") or {}
if d.get("kind") == "Service" and md.get("namespace") == "pihole" \
and spec.get("type") == "LoadBalancer":
frontends.append((rel, md.get("name"), spec))
protocols = {(p.get("protocol", "TCP"), p.get("port"))
for _, _, spec in frontends for p in spec.get("ports") or []}
for want in (("UDP", 53), ("TCP", 53)):
check(want in protocols, f"no PiHole LoadBalancer Service serves {want[0]}/{want[1]} — this "
"check's corpus has moved and must be re-derived (R-DNS-03)")
for rel, name, spec in frontends:
where = f"{rel}: Service/{name}"
ranges = spec.get("loadBalancerSourceRanges") or []
check(ranges, f"{where} has no loadBalancerSourceRanges, so it answers anyone who can reach "
"the VIP (R-DNS-03)")
tokens = {m.group(1) for m in (TOKEN.match(str(r)) for r in ranges) if m}
for tok in REQUIRED_TOKENS:
check(tok in tokens, f"{where} does not list ${{{tok}}} in loadBalancerSourceRanges "
"(R-DNS-03)")
for r in ranges:
m = TOKEN.match(str(r))
if not m:
assert_private(where, str(r))
continue
resolved = values.get(m.group(1))
check(resolved, f"{where}: {r} is not defined in cluster-vars, so its value cannot be "
"shown to be private (R-DNS-03)")
for v in sorted(resolved or []):
assert_private(f"{where} ({r})", v)
check(CHECKS >= 15, f"only {CHECKS} assertions ran — this check has gone vacuous (R-DNS-03)")
if FAILURES:
for f in FAILURES:
print(f"check FAIL: {f}", file=sys.stderr)
raise SystemExit(1)
print(f"check OK: pihole-dns-source-ranges — {CHECKS} assertions over "
f"{len(frontends)} PiHole LoadBalancer Service(s)")
PY
@@ -0,0 +1,45 @@
# =============================================================================
# L0 STATIC — PiHole answers an explicit LAN name list, and no LAN-only name is left out of it
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# Since PR #156 every public name is a proxied CNAME to the Cloudflare Tunnel and works from the
# LAN, so PiHole's zone wildcard was replaced by an explicit list (catalog/pihole.yaml). That makes
# the list load-bearing: a route disowned from public DNS and absent from it resolves nowhere. This
# test derives the required names from the tree (every `controller: none` HTTPRoute, every hostname
# the tunnel 404s, the L4 Services' own VIPs) and asserts the list covers them, carries no
# zone-wide entry, and pins gitea and nextcloud to the Gateway VIP. It proves itself non-vacuous by
# mutation. Rationale and the honest NOT-proven list: check.sh.
#
# WHY SCOPE, NOT REQUIREMENTS: the property spans catalog/, apps/, platform/ and runbooks/; a new
# internal service in any of them can break it, and ci/touched-requirements.sh maps apps/* to no
# requirement at all, so a requirement-scoped selection would skip it on precisely those changes.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: pihole-lan-names
labels:
conformance.home.k3s/R-DNS-03: "true"
annotations:
conformance.home.k3s/requirements: "R-DNS-03"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "true"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
PiHole answers an explicit list of LAN names rather than the whole zone; every HTTPRoute
disowned from public DNS has an entry to the Gateway VIP; every hostname the Cloudflare
Tunnel 404s has an entry, with the L4 names on their own VIPs; and gitea and nextcloud stay
on the LAN path, off Cloudflare's request-size and timeout limits.
spec:
steps:
- name: assert-pihole-explicit-lan-names
description: >-
Parse FTLCONF_misc_dnsmasq_lines from catalog/pihole.yaml and the rendered
apps/wave-0/pihole.yaml, derive the required names from every `controller: none` HTTPRoute,
the http_status:404 hostnames in platform/cloudflared/config.yaml and the L4 Services'
lbipam annotations, assert the list covers them and contains no zone-wide entry, and run
the evaluator on mutated lists to prove it is not vacuous.
try:
- script:
content: bash ./check.sh
+269
View File
@@ -0,0 +1,269 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — pihole-lan-names
# Serves: R-DNS-03 (LAN split-horizon answers), and decisions/2026-11-09-pihole-explicit-lan-names.md.
#
# WHY THIS EXISTS. Since PR #156 every PUBLIC name is a proxied CNAME to the Cloudflare Tunnel and
# works from the LAN, so PiHole no longer answers the whole zone with a wildcard: it answers an
# EXPLICIT list (catalog/pihole.yaml, FTLCONF_misc_dnsmasq_lines) and forwards everything else to
# the public resolvers. That inverts the old failure mode. With the wildcard, a forgotten name still
# resolved; now a forgotten LAN-ONLY name — a route disowned from public DNS, which the tunnel also
# 404s — resolves NOWHERE, and the app is silently unreachable from every client. Nothing in the
# cluster turns red when that happens. So the list is derived from the tree and asserted here.
#
# GIVEN the repository as authored,
# THEN (0) the dnsmasq value parses (only `local-ttl=` and `address=/<domain>/<${VAR}>` entries,
# every VAR is a cluster-var), the catalog value and the rendered apps/wave-0/pihole.yaml
# value are identical, and the inventories below are non-empty (anti-vacuous-pass);
# (a) NO entry covers the whole zone: no `address=/${BASE_DOMAIN}/`, no `address=/#/`, and no
# entry for any parent of the zone (dnsmasq matches every subdomain of an entry);
# (b) every HTTPRoute hostname disowned from public DNS
# (`external-dns.alpha.kubernetes.io/controller: none`, anywhere in apps/ platform/
# runbooks/) has an explicit entry to ${GATEWAY_VIP} — a disowned route with no
# `hostnames:` is itself a FAIL, because it cannot be checked;
# (c) every hostname platform/cloudflared/config.yaml maps to http_status:404 has an entry;
# the L4 DNS-only names (derived: Services carrying both
# `external-dns.alpha.kubernetes.io/hostname` and `lbipam.cilium.io/ips` — unifi,
# crafty-mc) must map to THAT Service's VIP, not the Gateway;
# (d) gitea and nextcloud are pinned to ${GATEWAY_VIP} (Cloudflare's 100 MB body / 100 s
# limits break git push/LFS and large uploads);
# AND (e) the evaluator is not vacuous: it is run against mutated copies of the real entry list
# (zone wildcard added; a disowned host dropped; a 404'd host dropped; gitea repointed;
# unifi repointed to the Gateway) and must flag each one.
#
# NOT proven (static): that the running PiHole pods carry this value (FTL applies FTLCONF_* at
# start; the env change rolls the StatefulSet), that public resolution of the unlisted names
# reaches Cloudflare, or the substituted IPs. Runtime check: `dig @${PIHOLE_DNS_VIP} <name>`.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
exec python3 - "$ROOT" <<'PY'
import copy
import re
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("CHECK FAIL (pihole-lan-names): PyYAML is required", file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
FAILURES, CHECKS = [], 0
NAME = "pihole-lan-names"
ZONE = "${BASE_DOMAIN}"
GW = "${GATEWAY_VIP}"
DISOWN = "external-dns.alpha.kubernetes.io/controller"
EDNS_HOST = "external-dns.alpha.kubernetes.io/hostname"
LB_IPS = "lbipam.cilium.io/ips"
# Public, but must bypass Cloudflare's 100 MB request body / 100 s response limits.
LAN_PINNED = ("gitea", "nextcloud")
# Trees that hold applied manifests. conformance/ fixtures and .claude/ worktrees are not authored
# surface (CLAUDE.md §0); decisions/ is prose.
SCAN = ("apps", "platform", "runbooks")
def check(cond, desc):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(desc)
def docs(path):
try:
return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)]
except yaml.YAMLError:
return []
def dnsmasq_value(objs):
"""The FTLCONF_misc_dnsmasq_lines value from a list of docs (a ServiceDefinition or manifests)."""
found = []
for d in objs:
envs = list(d.get("env") or [])
tmpl = (((d.get("spec") or {}).get("template") or {}).get("spec") or {})
for c in tmpl.get("containers") or []:
envs += c.get("env") or []
found += [e.get("value") for e in envs if e.get("name") == "FTLCONF_misc_dnsmasq_lines"]
return found
def die(msg):
print(f"CHECK FAIL ({NAME}): {msg}", file=sys.stderr)
raise SystemExit(1)
CATALOG = ROOT / "catalog" / "pihole.yaml"
RENDERED = ROOT / "apps" / "wave-0" / "pihole.yaml"
CF = ROOT / "platform" / "cloudflared" / "config.yaml"
for p in (CATALOG, RENDERED, CF):
if not p.is_file():
die(f"{p.relative_to(ROOT)} is absent — the surface this check guards moved; re-point or "
"delete this check in the SAME change")
cat_vals = dnsmasq_value(docs(CATALOG))
ren_vals = dnsmasq_value(docs(RENDERED))
check(len(cat_vals) == 1, f"(0) catalog/pihole.yaml declares FTLCONF_misc_dnsmasq_lines exactly "
f"once (found {len(cat_vals)})")
check(len(ren_vals) == 1, f"(0) apps/wave-0/pihole.yaml carries FTLCONF_misc_dnsmasq_lines exactly "
f"once (found {len(ren_vals)})")
if not cat_vals:
die("no dnsmasq value to check")
check(cat_vals[:1] == ren_vals[:1],
"(0) the rendered apps/wave-0/pihole.yaml dnsmasq value differs from catalog/pihole.yaml — "
"re-render pihole (catalog/render.py --no-image-automation catalog/pihole.yaml)")
cv = next(d for d in docs(ROOT / "clusters" / "base" / "cluster-vars.yaml")
if d.get("kind") == "ConfigMap")
CLUSTER_VARS = set(cv["data"])
BASE_DOMAIN_VALUE = cv["data"]["BASE_DOMAIN"]
ENTRY = re.compile(r"^address=/([^/]+)/(\$\{[A-Z0-9_]+\})$")
def parse(value):
"""Return (entries {domain: target}, problems)."""
entries, problems = {}, []
for raw in value.split(";"):
line = raw.strip()
if raw != line:
problems.append(f"entry {raw!r} has surrounding whitespace (FTL writes it verbatim)")
if not line:
problems.append("empty entry (doubled or trailing `;`)")
continue
if re.fullmatch(r"local-ttl=\d+", line):
continue
m = ENTRY.match(line)
if not m:
problems.append(f"entry {line!r} is not `local-ttl=N` or `address=/<domain>/${{VAR}}` — "
f"this check cannot reason about it")
continue
dom, tgt = m.groups()
if tgt[2:-1] not in CLUSTER_VARS:
problems.append(f"{dom} -> {tgt}: not a key of clusters/base/cluster-vars.yaml")
if dom in entries and entries[dom] != tgt:
problems.append(f"{dom} is listed twice with different targets")
entries[dom] = tgt
return entries, problems
entries, problems = parse(cat_vals[0])
for pr in problems:
check(False, f"(0) {pr}")
# ── inventories, derived from the tree ──────────────────────────────────────────────────────────
disowned, unnamed = set(), []
l4 = {}
for top in SCAN:
for f in sorted((ROOT / top).rglob("*.yaml")):
for d in docs(f):
meta = d.get("metadata") or {}
ann = meta.get("annotations") or {}
if d.get("kind") == "HTTPRoute" and str(ann.get(DISOWN, "")).strip() == "none":
hosts = (d.get("spec") or {}).get("hostnames") or []
if not hosts:
unnamed.append(f"{f.relative_to(ROOT)}:{meta.get('name')}")
disowned.update(hosts)
if d.get("kind") == "Service" and EDNS_HOST in ann and LB_IPS in ann:
for h in str(ann[EDNS_HOST]).split(","):
l4[h.strip()] = str(ann[LB_IPS]).strip()
for u in unnamed:
check(False, f"(b) disowned HTTPRoute {u} declares no `hostnames:` — it inherits a listener "
f"hostname this check cannot see; name it explicitly")
check(len(disowned) >= 5, f"(0) only {len(disowned)} disowned HTTPRoute hostnames found — the scan "
f"is vacuous (the disown annotation was renamed?)")
check(len(l4) >= 2, f"(0) only {len(l4)} L4 Services publishing their own hostname found (expected "
f"unifi and crafty-mc) — the scan is vacuous")
cf = yaml.safe_load(CF.read_text())
cf_404 = {r["hostname"] for r in cf.get("ingress") or []
if r.get("hostname") and r.get("service") == "http_status:404"}
check(len(cf_404) >= 5, f"(0) only {len(cf_404)} http_status:404 hostnames in the cloudflared "
f"config — the scan is vacuous")
def covers_zone(dom):
"""True if an entry for `dom` answers every name in the zone (dom is the zone or a parent)."""
if dom in ("#", "", "."):
return True
if dom == ZONE:
return True
# A literal parent (e.g. `keithjackson.dev`, `dev`) of the zone's substituted value.
real = dom.replace(ZONE, BASE_DOMAIN_VALUE)
return real == BASE_DOMAIN_VALUE or BASE_DOMAIN_VALUE.endswith("." + real)
def evaluate(entries):
bad = []
# (a)
for dom in entries:
if covers_zone(dom):
bad.append(f"(a) `address=/{dom}/` answers EVERY name in the zone — the wildcard is gone "
f"on purpose (public names must resolve to Cloudflare from the LAN)")
# (b)
for h in sorted(disowned):
if entries.get(h) != GW:
bad.append(f"(b) {h} is disowned from public DNS but PiHole maps it to "
f"{entries.get(h, 'NOTHING')} — expected {GW}; without it the app is "
f"unreachable everywhere")
# (c)
for h in sorted(cf_404):
if h in l4:
if entries.get(h) != l4[h]:
bad.append(f"(c) {h} is an L4 DNS-only name; PiHole maps it to "
f"{entries.get(h, 'NOTHING')}, expected its own VIP {l4[h]}")
elif h not in entries:
bad.append(f"(c) the tunnel 404s {h} but PiHole has no entry for it — it would resolve "
f"only publicly, and the tunnel refuses it")
# (d)
for n in LAN_PINNED:
h = f"{n}.{ZONE}"
if entries.get(h) != GW:
bad.append(f"(d) {h} maps to {entries.get(h, 'NOTHING (public/Cloudflare)')}; it must be "
f"pinned to {GW} — Cloudflare caps request bodies at 100 MB and responses at "
f"100 s")
return bad
for v in evaluate(entries):
check(False, v)
# (e) mutation self-test
mutants = {}
m = dict(entries); m[ZONE] = GW
mutants["zone wildcard added"] = m
m = dict(entries); m[BASE_DOMAIN_VALUE.split(".", 1)[1]] = GW
mutants["parent-of-zone wildcard added"] = m
m = dict(entries); m.pop(sorted(disowned)[0], None)
mutants[f"disowned host {sorted(disowned)[0]} dropped"] = m
pure_404 = sorted(h for h in cf_404 if h not in l4 and h not in disowned)
if pure_404:
m = dict(entries); m.pop(pure_404[0], None)
mutants[f"404'd host {pure_404[0]} dropped"] = m
m = dict(entries); m.pop(f"gitea.{ZONE}", None)
mutants["gitea unpinned"] = m
m = dict(entries); m[f"nextcloud.{ZONE}"] = "${UNIFI_L4_VIP}"
mutants["nextcloud repointed"] = m
if l4:
h = sorted(l4)[0]
m = dict(entries); m[h] = GW
mutants[f"{h} repointed to the Gateway"] = m
for name, mut in mutants.items():
check(evaluate(mut), f"(e) self-test: the evaluator did NOT flag the mutant '{name}' — vacuous")
check(len(mutants) >= 5, f"(e) only {len(mutants)} mutants built — the self-test degenerated")
_, pr = parse("local-ttl=60;address=/x.${BASE_DOMAIN}/${NOT_A_CLUSTER_VAR}; address=/y/${GATEWAY_VIP}")
check(len(pr) >= 2, "(e) self-test: the parser accepted an unknown VAR or a whitespace-padded entry")
if CHECKS < 15:
FAILURES.append(f"only {CHECKS} assertions ran — the check degenerated")
if FAILURES:
print(f"CHECK FAIL ({NAME}): {len(FAILURES)} of {CHECKS} assertions failed", file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
print(f"PASS ({NAME}): {CHECKS} assertions; {len(entries)} PiHole names, {len(disowned)} disowned "
f"routes, {len(cf_404)} tunnel-404 names, {len(l4)} L4 names")
PY
@@ -64,6 +64,12 @@ spec:
annotations:
# conformance-dns.<this cluster's BASE_DOMAIN> — never a literal apex.
external-dns.alpha.kubernetes.io/hostname: (join('.', ['conformance-dns', $baseDomain]))
# DNS-ONLY (2026-11-08). external-dns runs with --cloudflare-proxied globally (the
# gateway names point at the Cloudflare Tunnel). Without this opt-out the probe
# record is orange-clouded, 1.1.1.1 answers Cloudflare anycast instead of the LB IP,
# and the exact-IP poll below can never match. Same opt-out every real
# `service`-source publisher carries (unifi, crafty-mc).
external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
spec:
type: LoadBalancer
# No real backend needed — Cilium LB-IPAM assigns a VIP regardless, and
@@ -0,0 +1,77 @@
---
date: 2026-11-08
branch: feat/dns-via-cloudflare-tunnel
title: Public DNS via the Cloudflare Tunnel (Phase 2) — proxied CNAMEs, Plex/Headlamp LAN-only, L0 guard
---
## What changed
One atomic change, so the conformance guard lands with the DNS flip.
- `platform/gateway/gateway.yaml`: the shared Gateway's `external-dns.alpha.kubernetes.io/target`
is now the literal `fc81938b-5855-47f1-8a27-6286f56dda93.cfargotunnel.com` (was
`wan.${BASE_DOMAIN}`). Comment block rewritten.
- `platform/externaldns/helmrelease.yaml`: `extraArgs: [--cloudflare-proxied]` (global, because
v0.23.0 reads per-object `cloudflare-proxied` from the route/Service, never the Gateway). The
public-target comment block rewritten, including the prefix-regression risk.
- `apps/vms/unifi/50-service-l4.yaml`, `platform/crafty/service-l4.yaml`:
`external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"` (L4 A records stay DNS-only). They
are the only `service`-source publishers in the tree and live (measured).
- Plex web UI LAN/VPN-only: `catalog/plex.yaml` `exposure: public` → `internal`; `apps/wave-0/plex.yaml`
re-rendered with the canonical command into a scratch dir and ONLY plex.yaml copied back. The
render diff is exactly +2 lines (`annotations: external-dns.alpha.kubernetes.io/controller: none`
on HTTPRoute plex-32400). No other golden moved, and nothing was onboarded (vm-demo/keycloak files
were not touched). ADVERTISE_IP, the L4 VIP .24 and `plex-remote.yaml` are untouched.
- Headlamp LAN/VPN-only: `controller: none` on the hand-authored `platform/headlamp/httproute.yaml`.
- `conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh`: header and
register prose updated; new assertion (9):
(9a) Gateway target == `<tunnel: from platform/cloudflared/config.yaml>.cfargotunnel.com`, and
`--cloudflare-proxied` is set; (9b) every `service`-source publisher has `cloudflare-proxied:
"false"` and no HTTPRoute opts out of proxying; (9c) no undisowned route names a hostname the
tunnel 404s, the only `service` publishers of a 404'd name are the declared L4 set
{unifi, crafty-mc}, and every disowned route's hostname IS 404'd by the tunnel (lists kept
consistent both ways); (9d) plex and headlamp pinned disowned. Its `chainsaw-test.yaml` gap prose
updated.
- `conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml`: fixture Service gets
`cloudflare-proxied: "false"`, else the global flag proxies it and the exact-IP poll fails.
- Docs: `platform/externaldns/ddns.yaml` header (wan. is WireGuard-only), `platform/externaldns/README.md`,
`platform/gateway/README.md`, `runbooks/plex-remote-access.md`, `runbooks/cloudflare-tunnel.md`
(Phase 2 done; operator step to close WAN tcp/80+443 and keep udp/51820 + tcp/32400; rotation
now also swaps the Gateway UUID; rollback path).
## What I verified
- external-dns v0.23.0 source, read at the tag: `source/gateway.go` takes target overrides from
`gw.Annotations` and provider-specific (incl. `cloudflare-proxied`) from the ROUTE's annotations;
`annotations.SetAnnotationPrefix` re-derives `CloudflarePrefix`, so the alpha-prefixed key is the
one read here; `provider/cloudflare` `shouldBeProxied` starts from `--cloudflare-proxied` and a
parsed `"false"` annotation overrides it.
- (8b) only forbids `annotation-prefix` / `enable-legacy-annotation-prefix` in extraArgs; a mutation
confirmed `--cloudflare-proxied` passes and `--annotation-prefix=x/` fails.
- The L0 check passes (116 assertions) and each new assertion FAILS on its mutation: Gateway target
back to wan (9a), flag removed (9a), unifi proxied "true" (9b), headlamp undisowned (9c,9d), plex
golden undisowned (9c,9d,(1)), jackett removed from the tunnel's 404 list (9c).
`cloudflared-ingress-rules` still passes; `render.py --selftest` PASS.
- Live, read-only (kubectl k3s-prod, ~23:55Z): 17 HTTPRoutes; the six internal apps carry
`controller: none`; plex and headlamp do not (yet); the only Services with an external-dns
hostname are crafty/crafty-l4-tcp and unifi/unifi-l4-tcp; Gateway target is currently
`wan.home.keithjackson.dev`.
- Toolbox gate result: see the PR description (rc and skipped stages quoted there).
## What I did NOT verify
- Anything after merge: that external-dns actually UPDATEs the eight public names to proxied CNAMEs
to the tunnel, DELETEs plex + headlamp and their TXT, and leaves unifi/crafty-mc/wan untouched.
Expected effect is listed in the PR; the PM must read external-dns logs and `dig @1.1.1.1`.
- A rendered HelmRelease: I read the chart 1.23.0 `templates/deployment.yaml` (a list `extraArgs`
is emitted item-by-item via `tpl . $ | quote`), but did not `helm template` it.
- kubectl OIDC from nodes: nodes resolve publicly, so `keycloak.` discovery/JWKS now goes through
Cloudflare + the tunnel. No `oidc-ca-file` is set (system roots), so the edge cert should validate,
but Cloudflare bot/WAF settings on that path are untested.
- RISK: if the annotation prefix stops being honoured at runtime again (11-07 incident class),
`target` is ignored and every name becomes a PROXIED A record at 192.168.2.20, while internal
names publish. Assertion (8) only keeps the tree and `values.annotationPrefix` consistent; it
cannot catch a controller that ignores the flag. No runtime alert exists for it.
- The router WAN 80/443 close is an operator step, not done. The Pi-hole wildcard trim is a later PR.
- Follow-up: spec requirement IDs for this (proposed R-DNS-04/05, R-SEC-04) once the spec ADR merges;
`platform/pm-agent/README.md` and a comment in `catalog/pihole.yaml` still describe the wan. CNAME.
@@ -0,0 +1,74 @@
---
date: 2026-11-08
branch: fix/wg-client-mtu-docs
title: WireGuard client profiles default to MTU 1280; VPN runbook corrected and given a troubleshooting section
---
## What changed
- **`infra/vpn-ingress/new-peer.sh`**: `WG_MTU` default 1420 -> 1280, still overridable, reason in a
comment. Only new profiles change; the phone's existing profiles are not touched.
- **`runbooks/vpn-remote-access.md`**: the verification table no longer says the Gateway sees a
`host`-identity source; the limitations no longer say everything arrives SNATed or that IPv6 leaks
on Android; the "if nothing handshakes" advice no longer claims `wg show` separates a bad forward
from a bad key. New "Troubleshooting" section covering the four cases in the brief.
- **`platform/vpn-ingress/daemonset.yaml`**: header comment only, SNAT claim corrected.
- **`platform/vpn-ingress/README.md`**: one sentence, so `MTU = 1420` reads as the server's value.
Rollout on merge: the WireGuard DaemonSet does **not** roll. The `runbooks` ConfigMap is content
hashed and `vpn-remote-access.md` is in it, so the runbooks site rolls.
Left unchanged, for the operator:
- **`platform/vpn-ingress/peers.conf`** still says "No peers enrolled yet". A comment-only edit on a
throwaway copy moved the ConfigMap from `wg-peers-29mf6k867t` to `wg-peers-gggkk5d4tm` and changed
the DaemonSet, so correcting it restarts every `wg0`. Fix it alongside the next peer change.
- **`clusters/base/cluster-vars.yaml`** coverage comment still omits `WG_CLIENT_CIDR` (sensitive path).
- **Server `wg0` MTU stays 1420**, because changing it restarts all WireGuard pods. For TCP a 1280
client should be enough on its own: it advertises an MSS of 1240, which caps what the far end
sends. That protection does not reach UDP. A server-to-phone datagram with an inner size of
1281-1420 bytes is still sent as up to 1480 outer bytes and could be lost on the same path; QUIC
should recover by probing, large DNS answers are the likeliest casualty. If that shows up, the
fix is a lower server MTU, as a deliberate rollout.
- **Other stale text, outside this brief:** `platform/vpn-ingress/README.md` still says no peer has
ever completed a handshake, and the runbook's status line still calls the VPN inert.
## What I verified
- Upstream `wireguard-android` `GoBackend.java`, master (last commit to that file `ecc3c0a`,
2026-03-15): `builder.setMtu(config.getInterface().getMtu().orElse(1280))`, and `allowFamily()` is
skipped when `sawDefaultRoute && config.getPeers().size() == 1`.
- Live, read-only, 2026-11-09 UTC, on the `k3s-prod-1` pod and agent:
- `wg0` MTU is 1420 on all three pods. Both peers have handshakes and transfer on prod-1 only; the
other two pods read `0`.
- BPF conntrack holds `UDP SVC 10.90.0.x -> 192.168.2.23:53` and flows to `192.168.2.24:32400`.
Their backend entries carry `SourceSecurityID=2` (`world`). So the source stays `10.90.0.x`
through the load balancer on those two VIPs.
- `/proc/net/nf_conntrack` is readable from the pod, which has no `conntrack` binary.
- Offline, pinned toolbox: `kubectl kustomize platform/vpn-ingress` gives identical ConfigMap and
DaemonSet hashes on `origin/main` and on this branch.
- `new-peer.sh` writes `MTU = 1280` by default and `MTU = 1420` with `WG_MTU=1420`. Run with
throwaway keys and stdout discarded; profiles shredded; nothing enrolled.
- No test or conformance check pins the client MTU (`grep` for `WG_MTU` and `1420`).
- Gate (pinned toolbox, from this worktree, on the committed tree): **rc 3, UNDETERMINED, not green.**
`gitea-settings` is the only stage counted as not run (no token locally). `conformance` (L2/L3)
also skipped as the declared no-cluster gap, so R-ENV-01 and R-OBS-04 are not certified.
`spec-immutability`, `approval-policy`, `decisions-log`, `lint`, `l0`, `render-golden` and `l1`
passed. No sensitive path is touched.
## What I did NOT verify
- **That MTU was the cause.** It is inferred from one before-and-after, on one phone and one carrier,
reported by the operator. The path MTU was never measured.
- **That 1280 is the best value.** It is the value that worked, and the IPv6 minimum. A higher one
may work too.
- That the phone's app is on `GoBackend` rather than the kernel backend, so that a blank field meant
1280 on this device. Only the upstream default was read.
- That TCP MSS behaves as described, or that large UDP replies are actually lost. No packet capture
was taken.
- That the Gateway VIP `.20` sees `10.90.0.x` / `world`. Conntrack held no tunnel flows to it when
checked.
- Android's IPv6 blocking on a real device; read from source only.
- `wg-quick` IPv6 behaviour on Linux, and any non-Android client at 1280.
- `/proc/net/nf_conntrack` output during a key mismatch. The `[UNREPLIED]` reading comes from the
earlier measurement in the brief; I saw only healthy entries.
@@ -0,0 +1,103 @@
---
date: 2026-11-08
branch: fix/wg-home-peer-pihole-sources
title: Enrol keith-phone-home and admit the WireGuard subnet to PiHole's DNS frontends
---
## What changed
- **`platform/vpn-ingress/peers.conf`**: new peer `keith-phone-home`, `10.90.0.3/32`, key generated
on the phone. It is the `home` half of a two-tunnel profile (Endpoint `192.168.2.10:51820`),
because the router has no hairpin NAT and `wan.` is unusable from inside the house.
- **`apps/wave-0/pihole-dns.yaml`**: `${WG_CLIENT_CIDR}` added after `${LAN_CIDR}` on both
`pihole-dns-udp` and `pihole-dns-tcp`. This file is hand-written, not rendered (`catalog/render.py`
only lists it in `EXTRA_UNITS`), so it is the source of truth and `--emit-waves` was not run.
`catalog/pihole.yaml` changes in one comment only, so nothing renders differently.
- **New L0 guard `conformance/tests/l0-static/pihole-dns-source-ranges`** (whole-repo scope,
R-DNS-03): every PiHole LoadBalancer Service lists both tokens, and every range resolves through
cluster-vars (base and prod overlay) to RFC1918.
- **`platform/vpn-ingress/README.md`**: corrected "clients arrive SNATed" (false for LoadBalancer
VIPs) and the IPv6 claim (Android's `GoBackend` blocks IPv6 for a single-peer default route);
documented the home/away pattern, why `wan.` fails from inside, and the prod-1 dependency.
Rollout on merge: the `wg-peers` ConfigMap hash changes (`m9kcckcb5t` -> `29mf6k867t`), so the
DaemonSet rolls, `maxUnavailable: 1`, `maxSurge: 0`, one node at a time in an order the controller
chooses. Each roll deletes and recreates that node's `wg0`, so an established session on that node
drops and must re-handshake. `keith-phone` has no session today (see below), so nothing is
interrupted in practice.
**The one risk worth watching:** `wg0` is in Cilium's auto-detected device list and `devices` is
still unpinned. On 2026-10-06 a crash-looping `wg0` churned that list and all three nodes lost their
main-table default route. A rollout is one delete/create per node, not a loop, so it should not
repeat that, but after merge check `ip route get 1.1.1.1` on each node.
The Service change is an in-place update: same VIP, same Service objects, no pod restart.
Checked on the rest of the path and left unchanged:
- Gateway VIP `.20` (`gateway/cilium-gateway-shared`): no source ranges, live. Nothing needed.
- `pihole-allow-lan-dns-ingress` admits `world` on :53, which covers an untranslated 10.90.0.x
source. `platform/vpn-ingress/networkpolicy.yaml` does not apply to a hostNetwork pod.
- No L0 check pinned these ranges to the LAN; none needed updating.
Flagged, not changed (out of scope):
- `runbooks/vpn-remote-access.md` "Known limitations" repeats both corrected claims, and step 3 of
its verification table says the Gateway sees a `host`-identity source.
- `platform/vpn-ingress/daemonset.yaml`'s header repeats the SNAT claim.
- `clusters/base/cluster-vars.yaml`'s substitution-coverage comment does not list
`WG_CLIENT_CIDR` under apps-wave-0. The variable is defined and substituted there regardless.
- `peers.conf` still says "No peers enrolled yet" above the enrolled peers.
- PiHole's NodePorts are not covered by source ranges (already acknowledged in `pihole-dns.yaml`).
Tunnel clients can reach node addresses, so they can reach those too; they are enrolled devices.
- Pinning Cilium's `devices` remains the open hardening for the churn risk above.
## What I verified
Live, read-only, 2026-11-08:
- `cilium-dbg status` on one agent: `KubeProxyReplacement: True [eno1 ..., wg0 10.90.0.1, tun0 ...]`
and `Masquerading: BPF [eno1, wg0, tun0]`. So `wg0` really is a load-balancing device.
- `pihole-dns-tcp` and `-udp` carry `ranges=[192.168.2.0/24]`; no other LoadBalancer Service has
source ranges. Live `cluster-vars`: `WG_CLIENT_CIDR=10.90.0.0/24`, `LAN_CIDR=192.168.2.0/24`.
- `wg show wg0 latest-handshakes` on all three pods: `keith-phone` = `0`.
- `wan.home.keithjackson.dev` resolves publicly to a single A record (via 1.1.1.1).
Offline, in the pinned toolbox:
- `flux envsubst --strict` renders both Services to `[192.168.2.0/24, 10.90.0.0/24]`.
- `kubectl kustomize platform/vpn-ingress`: `wg-peers-m9kcckcb5t` on the old `peers.conf`,
`wg-peers-29mf6k867t` on the new one.
- Guard mutation test, each restored afterwards. Baseline passes (22 assertions, 2 Services). All
seven fail with rc 1 and a named message: WG range dropped from one Service; `0.0.0.0/0` added;
`8.8.8.0/24` added; `WG_CLIENT_CIDR` made public in cluster-vars; prod overlay sets `LAN_CIDR` to
`0.0.0.0/0`; ranges block removed; undefined token added.
- The Android claim was read from upstream `GoBackend.java` (`wireguard-android`, master): the
`allowFamily` calls are skipped when `sawDefaultRoute && config.getPeers().size() == 1`.
- Gate (pinned toolbox, from this worktree, on the committed tree): **rc 3, UNDETERMINED, not
green.** `gitea-settings` is the only stage counted as not run (no token locally). `conformance`
(L2/L3) also skipped as the declared no-cluster gap. `spec-immutability`, `decisions-log`, `lint`,
`l0` (including the new guard), `render-golden` and `l1` passed. `approval-policy` reported N/A
offline; it will be red in CI until the operator overrides, because `conformance/**` is sensitive.
- `git merge-tree` against `feat/pihole-explicit-lan-names` (#157): no conflicts.
## What I did NOT verify
- **`DROP_NOT_IN_SRC_RANGE` was NOT observed live.** It is a prediction from Cilium's source. Only
the `wg0` device membership was measured; the drop itself has never been seen.
- **Full-tunnel behaviour was NOT observed live**, on either profile. **No handshake has ever been
recorded for any peer**: `keith-phone` reads `0` on all three pods (a counter that resets when a
pod restarts, last 22 days ago, so it covers only the time since; earlier entries here also
record none).
- That tunnel DNS works after this change, including the reply path from a backend on another node
and from one on the ingress node. That needs the phone: `dig` over the tunnel, while watching
`cilium-dbg monitor --type drop`.
- That the Gateway serves tunnel clients. Its source is predicted to be `world`, as for a LAN
browser, which is what the apps admit; not tested.
- That the new peer's key is the one on the phone. I copied the public key from the brief.
- That the rollout leaves node default routes intact. Reasoned from the 2026-10-06 incident being
a loop, not measured.
- That a LAN client is unaffected while the Service's source ranges are updated. Expected, not
measured.
- IPv6 behaviour on any non-Android client.
- That the WAN forward targets prod-1: it is off-repo.
@@ -0,0 +1,97 @@
---
date: 2026-11-09
branch: feat/pihole-explicit-lan-names
title: PiHole answers an explicit LAN name list instead of a zone wildcard (public names resolve to Cloudflare from the LAN)
---
## What changed
**`catalog/pihole.yaml` → `apps/wave-0/pihole.yaml`.** `FTLCONF_misc_dnsmasq_lines` no longer carries
`address=/${BASE_DOMAIN}/${GATEWAY_VIP}`. Every name not listed is now forwarded to 8.8.8.8/8.8.4.4 and
gets the public answer. The final list (`local-ttl=60` kept):
| Name (`.${BASE_DOMAIN}`) | Answer | Why |
|---|---|---|
| jackett, pihole, plex, qbittorrent, radarr, semaphore, sonarr, headlamp | `${GATEWAY_VIP}` | LAN-only: every HTTPRoute with `external-dns.alpha.kubernetes.io/controller: none`, derived from the tree. No public record, tunnel 404s them; this entry is their only path |
| gitea, nextcloud | `${GATEWAY_VIP}` | public, but Cloudflare's proxy caps bodies at 100 MB / responses at 100 s (git push, LFS, uploads) |
| keycloak, auth | `${GATEWAY_VIP}` | keep the OIDC flow of the LAN-path apps on one path |
| unifi | `${UNIFI_L4_VIP}` | L4 VIP, not behind the Gateway |
| crafty-mc | `${CRAFTY_L4_VIP}` | L4 VIP, not behind the Gateway |
| pihole-dns | `${PIHOLE_DNS_VIP}` | the resolver's own address |
| plex-vip | `${PLEX_L4_VIP}` | kept, see below |
Not listed, so they resolve to Cloudflare from the LAN: the apex, grafana, crafty, seerr.
**The derived LAN-only set matches the brief's list exactly** (8 names; the tree was walked, not the
list trusted). The `runbooks/serving` route has no `hostnames:` but is not disowned (it inherits the
apex, public).
**plex-vip is kept.** Nothing advertises it any more (`ADVERTISE_IP` is a `plex.direct` name since
2026-10-05), but it is not unused: it is Plex's `customCertificateDomain` (`CERTDOMAIN`, Secret
`plex-vip-tls`), the name of the tcp/32400 VIP, and the 404'd name in the tunnel config. It has no
public record, so this entry is its only resolution.
**New L0 check `conformance/tests/l0-static/pihole-lan-names`** (`scope: whole-repo`, so picked up by
the runner's scope selection with no other wiring). Asserts: (a) no zone-wide entry (zone, `#`, or any
parent of the zone); (b) every `controller: none` hostname → `${GATEWAY_VIP}`, and a disowned route
with no `hostnames:` fails; (c) every tunnel-404 hostname has an entry, with the L4 names (derived
from Services carrying both the external-dns hostname and an lbipam VIP) on their own VIP; (d) gitea
and nextcloud → `${GATEWAY_VIP}`; plus catalog == rendered value and a built-in mutation self-test.
**Stale docs corrected:** `platform/pm-agent/README.md`, `runbooks/pm-agent.md`,
`platform/externaldns/README.md`, `platform/cloudflared/README.md`, `platform/gateway/gateway.yaml`
(comment), `platform/headlamp/httproute.yaml` (comment), `apps/vms/unifi/README.md`, `README.md`,
`schemas/renderer-contract.md`; `runbooks/cloudflare-tunnel.md` gains the two consequences below.
`platform/cloudflared/deployment.yaml`: the image entrypoint comment was wrong (comment-only).
## What I verified
- **Render:** the baseline `render.py --no-image-automation catalog/pihole.yaml` was byte-identical to
the committed `apps/wave-0/pihole.yaml` before the change, and the file was replaced by that
command's output after it; `--emit-waves` was not run.
- **Mutation, against the real files (10 mutants, each rc 1 with the right assertion, each restored):**
(a) zone wildcard restored; (b) headlamp and sonarr entries removed; (b) a disowned route with no
`hostnames:`; (c) a new tunnel-404 host with no entry; (c) pihole-dns removed; (c) unifi pointed at
the Gateway; (d) gitea removed; (d) nextcloud repointed; (0) rendered value drifted from the catalog.
- **Rollout mechanics, measured live (read-only):** StatefulSet `updateStrategy` RollingUpdate
partition 0, `podManagementPolicy: Parallel`, no `minReadySeconds`, 2 replicas, PDB minAvailable 1.
The last rollout (2026-11-08 06:47–06:48Z) replaced pihole-1 then pihole-0: FTL logged "started" ~1 s
after the container, the pod went Ready ~16 s after the container, and pihole-0 started ~9 s later.
- **Expected LAN DNS blip:** no total outage; one replica answers throughout. Each of the two pod
replacements can lose a few UDP queries for ~1–2 s while the terminating pod leaves the Service
(clients retry). For roughly 30–60 s the two pods answer differently for the apex, grafana, crafty
and seerr; both answers work. Cached answers age out in 60 s (`local-ttl`).
- **Live DNS before the change:** @8.8.8.8 returns Cloudflare (104.21.51.91 / 172.67.178.27) for the
apex, grafana, seerr, crafty and gitea, and nothing for jackett; @192.168.2.23 returns 192.168.2.20
for all of them (the wildcard).
- **Cloudflared image:** `skopeo inspect --config` on the pinned digest (amd64): Entrypoint
`["cloudflared"]`, Cmd `["version"]`, User `65532:65532`.
## What I did NOT verify
- **The post-merge behaviour.** Nothing here ran against a live PiHole with the new list; the
expected answers are from dnsmasq semantics and the tree. After reconcile, check
`dig @192.168.2.23` for one name per row, and grafana returning Cloudflare.
- **Whether every LAN client path works via Cloudflare.** Measured 2xx/3xx through Cloudflare by the
brief, not re-measured from a LAN client here.
- **The FTL DNS-listening moment.** "FTL started" is a log line, not a successful query; the rollout
timing above assumes DNS is up shortly after it.
**Accepted consequences, stated plainly:**
- The apex, grafana, crafty and seerr now depend on the internet and the tunnel from the LAN too. An
ISP or Cloudflare outage, or stopping the connectors, makes them unreachable from the LAN where they
previously kept working. If grafana in particular should survive an internet outage, add it to the
list (one entry; the check permits it).
- The tunnel rollback (revert to `wan.`) now also requires restoring the wildcard; recorded in
`runbooks/cloudflare-tunnel.md`.
**Flagged, not changed:**
- The `pihole` container has no readiness probe; only the `oauth2-proxy` sidecar does, so "Ready" says
nothing about DNS. The measured margin (FTL ~1 s, Ready ~16 s) makes this benign in a normal rollout
but a slow FTL start would not hold the rollout back. A DNS readiness probe needs a schema/renderer
decision.
**Stale statements left alone:** `platform/coredns/coredns-custom.yaml` header ("both resolve the whole
zone ... by wildcard", brief said not to touch CoreDNS); `clusters/base/cluster-vars.yaml` comments
(sensitive path); `schemas/renderer-contract.md` §`exposure` first bullets (manual edge port-forward
wording); historical `infra/PROD-PROVISIONING.md`, `platform/externaldns/README.md` TODO list.
+9 -1
View File
@@ -53,6 +53,7 @@
# WG_ENDPOINT_HOST default wan.home.keithjackson.dev
# WG_CLIENT_CIDR default 10.90.0.0/24 (must match clusters/base/cluster-vars.yaml)
# WG_LAN_CIDR default 192.168.2.0/24 (prod LAN_CIDR)
# WG_MTU default 1280 (client tunnel MTU; see the comment on the default below)
# WG_DNS default 192.168.2.23 (prod PIHOLE_DNS_VIP)
#
# REQUIRES: `wg`, `qrencode`, `sops` — all three in the dev container.
@@ -80,7 +81,14 @@ PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.conf"
# cannot resolve until it is already connected.
: "${WG_ENDPOINT_HOST:=wan.home.keithjackson.dev}"
: "${WG_LISTEN_PORT:=51820}"
: "${WG_MTU:=1420}"
# The CLIENT's tunnel MTU. Deliberately lower than the server's 1420 (daemonset.yaml), which only
# describes the 1500-byte path at the node's end. A phone's path is the carrier's, often behind
# carrier NAT, and is not known to carry 1500. Measured 2026-11-09, one Android phone on one
# carrier: at 1420 the handshake was healthy but names and pages failed; clearing the app's MTU
# field (its userspace backend then uses 1280) fixed it. 1280 is also the IPv6 minimum link MTU, so
# it stays valid if the tunnel ever gains IPv6. For a device only ever on a wired or known-1500
# path, WG_MTU=1420 recovers about 10% of per-packet payload.
: "${WG_MTU:=1280}"
SPLIT=0
OUT_DIR=""
+5 -3
View File
@@ -17,6 +17,8 @@ outbound-only from the cluster (no router port forward, no hairpin, no split-DNS
| `networkpolicy.yaml` | default-deny (single owner — not an app namespace) + Prometheus scrape allow + egress CNP: world 7844 tcp/udp and 443, the Gateway (`ingress` entity), and Gateway backends by `gateway-access=shared` minus the LAN-only namespaces. |
| `servicemonitor.yaml`, `alerts.yaml` | Headless metrics Service + ServiceMonitor scraping :2000, and the two alerts, both written to fire (not go quiet) when their series vanish. |
**Phase 1 only.** This component changes no live traffic: Cloudflare sends a hostname through the
tunnel only when its DNS record is a CNAME to `<UUID>.cfargotunnel.com`, and nothing here creates
one. Moving DNS (external-dns, Gateway annotations, Pi-hole/CoreDNS split horizon) is a later change.
Cloudflare sends a hostname through the tunnel only when its DNS record is a proxied CNAME to
`<UUID>.cfargotunnel.com`; nothing in this directory creates one. Those records are published by
external-dns from the shared Gateway's target annotation (Phase 2, PR #156; see
`platform/externaldns/README.md`). LAN resolution: `catalog/pihole.yaml` (explicit names only since
2026-11-09), so public names reach Cloudflare from the LAN too.
@@ -9,7 +9,7 @@ metadata:
namespace: cloudflared
type: Opaque
stringData:
credentials.json: ENC[AES256_GCM,data:iTCsCQ9g4zouELNa1KzFy8ZO5eLctBdKFaWDXlejpxjWs06sGaduYRPV9kmm1JT8A8s10OevqEEXM+9unOSrmOiUvsxMYAvtwvfyUs3dVbtOB/uCBzHsGK+4cJoC1iII6WzITVe5tlWMXYtyBnDo1Mm3BfO+H/6ytLf750BoABqohO2cfWRfN9MAXL6BFe7urZdKrMou23BooDUxG/pwTpnALbUCer/PhCmrB6cgOw==,iv:ltn/fN59x320pFgvb3/g0UgzBmQcdnnk8JvKpoFn0H0=,tag:fr8s1aDsZBAQYLGXgwG8WA==,type:str]
credentials.json: ENC[AES256_GCM,data:68G438/jsM4KqPt/3wjFp6jVmAVnsidx/FEdK6Lke2euQgkDqRcJjhoKZurapgP8zsP+FrQ98VgsQgUpERHXUCRuJp5wRIgTVmL23CweuZLacaLxBo6eoKmNC/H5DzGZsyhDxGjcXc+8BGpfDFzHR39Uo2h56XstiHtyZhnhlhia8nprLip9RinA9mbKRMJKTR/6IsZsdc91E22aEycDjZ3+nYmr4KP2+km3fd15Nw==,iv:CuAUscGmvMqT2ixFJZVzJWSvDYwbfQl79sQ+FHnXdVQ=,tag:H4fFjXPCFoxE1Mqwx0R5sQ==,type:str]
sops:
kms: []
gcp_kms: []
@@ -25,8 +25,8 @@ sops:
emhHaFpEZWpMN0JXL2NVaW5PU1ZwVW8KkCDX7ED039X12BCmiSqHr0fmZKSZmfjP
DIiR6DvWWw1TcB3CjD5q29nIthCDf9zlrpuMfQ+9ODAlRr5t6zP+cg==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-11-08T21:28:19Z"
mac: ENC[AES256_GCM,data:4wD/6YSBP/e6MTevGSOFTi/EamJRZoryf+aET/opTwjf7yMgi5iObXTnRapKwAFzeXkSkFVfnfRd8Ir714Uez3z2yLxQ6Ygte9VQxHUfkKfXfu+OmuCnjFGaTjsxm6551VPFGHZuto6wZjebwozWuxzHhZi/AjCo2QGyRjmymBM=,iv:hClSAe8r6fhnQzwQY3NZSYhqElzwSRPhI8Y2fH1+r64=,tag:LviWf5oMB6NUjTFNXy62zg==,type:str]
lastmodified: "2026-11-08T22:05:00Z"
mac: ENC[AES256_GCM,data:FaTWQhcdYD4sDu50K9NisBO81cOwHqWKt9b/0XBvZzV5FAMNHBIi4Md3by5XT16paMuS+kgz20/+CLX1PkJ6FNbCwnp7M0+a+xtT+UT67apgJ04B6GW7HZJ+iUS6sOIIFoefSDHfKpO4bYLDIoXWupLYFlbHHRJQqQXpU7yvVgI=,iv:YDSYmgc7YUsYVeh7tPm8Sztvr3+L6xb1lww4lZdzDaY=,tag:7IektWvjJpKG98edUqNadQ==,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.9.4
+1 -1
View File
@@ -20,7 +20,7 @@
# OPERATOR STEP: replace REPLACE-ME-TUNNEL-UUID with the UUID printed by `cloudflared tunnel create`
# (runbooks/cloudflare-tunnel.md step 3). Until then cloudflared treats the value as a tunnel NAME,
# finds no origin certificate to resolve it with, and exits — a CrashLoop in a non-gating tier.
tunnel: f26f1052-1bc0-4d9d-800b-4da0377472a3
tunnel: fc81938b-5855-47f1-8a27-6286f56dda93
credentials-file: /etc/cloudflared/creds/credentials.json
# Serves /metrics (Prometheus), /ready (200 only with >=1 live edge connection) and /healthcheck.
metrics: 0.0.0.0:2000
+3 -2
View File
@@ -70,8 +70,9 @@ spec:
- name: cloudflared
# Tag + digest: Renovate's `kubernetes` manager (renovate.json5, platform/**) bumps both.
image: docker.io/cloudflare/cloudflared:2026.10.7@sha256:4955524aeaa379e154023d5a027c250f7b35041a2e7e950cf13f144bc9b9c5ba
# The image ENTRYPOINT is `cloudflared --no-autoupdate`; these replace its CMD. The tunnel
# UUID, credentials path, metrics address and ingress rules all come from the file.
# The image ENTRYPOINT is `cloudflared` and its CMD is `version` (read from the pinned
# digest's image config, 2026-11-09); these args replace that CMD. `no-autoupdate`, the
# tunnel UUID, credentials path, metrics address and ingress rules all come from the file.
args:
- tunnel
- --config
+5
View File
@@ -101,6 +101,11 @@ metadata:
lbipam.cilium.io/ips: "${CRAFTY_L4_VIP}"
# external-dns publishes A crafty-mc.home.keithjackson.dev -> the pinned VIP (Java server target).
external-dns.alpha.kubernetes.io/hostname: "crafty-mc.${BASE_DOMAIN}"
# DNS-ONLY, NEVER PROXIED (2026-11-08). external-dns runs with --cloudflare-proxied globally (the
# gateway names point at the Cloudflare Tunnel); Cloudflare's proxy carries HTTP(S) only, so an
# orange-clouded A for these L4 ports would be broken. Required by the L0 check
# internal-services-publish-no-public-dns (9b) on every `service`-source publisher.
external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
spec:
type: LoadBalancer
externalTrafficPolicy: Cluster
+37 -17
View File
@@ -20,29 +20,49 @@ A wildcard `*.keithjackson.dev` matches only one label deep, so `<svc>.home.keit
its own record. **`internal` services need NO public record** — PiHole resolves them to the internal
Gateway VIP (split-horizon). ExternalDNS manages **`public`** services' records only.
## Public target + in-cluster DDNS (2026-10-03)
## Public target: the Cloudflare Tunnel, proxied (2026-11-08)
Gateway-published records are **CNAMEs to `wan.home.keithjackson.dev`**, not the Gateway VIP:
Gateway-published records are **proxied CNAMEs to the Cloudflare Tunnel**
(`platform/cloudflared`), not the WAN IP and not the Gateway VIP:
```
plex.home.keithjackson.dev CNAME wan.home.keithjackson.dev <- external-dns (Gateway `target` annotation)
wan.home.keithjackson.dev A <current WAN IP> <- cloudflare-ddns (ddns.yaml), every 5m
gitea.home.keithjackson.dev CNAME (proxied) fc81938b-5855-47f1-8a27-6286f56dda93.cfargotunnel.com <- external-dns
unifi.home.keithjackson.dev A (DNS-only) 192.168.2.21 <- external-dns, `service` source (L4)
crafty-mc.home... A (DNS-only) 192.168.2.22 <- external-dns, `service` source (L4)
wan.home.keithjackson.dev A (DNS-only) <current WAN IP> <- cloudflare-ddns (ddns.yaml): WireGuard udp/51820 ONLY
```
- The `target` annotation is on the **shared Gateway** (`platform/gateway/gateway.yaml`), so it
applies to every route that publishes at all. `internal` routes (`controller: none`) still publish
nothing.
- `ddns.yaml` runs `favonia/cloudflare-ddns` in this namespace with the same `cloudflare-api-token`.
It detects the WAN IP through Cloudflare's trace endpoint, so the pod **must not** carry
`egress-class: vpn`.
- LAN and pods resolve `*.home.keithjackson.dev` to the Gateway VIP via PiHole and CoreDNS
(split-horizon) and never use the public path.
- **Nodes resolve through public DNS** (`node_baseline_dns_servers`), so from a node, the apiserver's
OIDC issuer `keycloak.home.keithjackson.dev` resolves to the WAN IP. kubectl SSO therefore relies on
the router hairpinning (NAT reflection) WAN:443 back to the Gateway.
- The `target` annotation is on the **shared Gateway** (`platform/gateway/gateway.yaml`); external-dns
reads it from the Gateway only, so it applies to every route that publishes at all. Its UUID must
equal `tunnel:` in `platform/cloudflared/config.yaml` (L0 check
`internal-services-publish-no-public-dns` (9a)).
- `--cloudflare-proxied` (HelmRelease `extraArgs`) orange-clouds every record by default — the only
practical lever, because v0.23.0 reads the per-object `cloudflare-proxied` annotation from the
route/Service, never the Gateway. **Every `service`-source publisher must carry
`external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"`** (L4 ports cannot ride the HTTP
proxy); (9b) fails the gate otherwise.
- `internal` routes (`controller: none`) still publish nothing. Plex's web UI and Headlamp joined that
set on 2026-11-08 (LAN/VPN-only); the tunnel also 404s every such name (9c/9d).
- Public answers are Cloudflare anycast **A and AAAA** — Cloudflare's edge, not an origin address.
- **Pods** resolve the whole zone to the Gateway VIP via CoreDNS (split-horizon) and never use the
public path. **LAN clients** use PiHole, which since 2026-11-09 answers an EXPLICIT list
(`catalog/pihole.yaml`): the LAN-only names, plus gitea/nextcloud/keycloak/auth (kept off
Cloudflare's 100 MB / 100 s limits and on one OIDC path), go to the Gateway VIP and the L4 names to
their own VIPs; every other public name (apex, grafana, crafty, seerr) resolves to Cloudflare from
the LAN exactly as from anywhere else, so no split-DNS is needed for them. Guard:
`conformance/tests/l0-static/pihole-lan-names`.
- **Nodes resolve through public DNS** (`node_baseline_dns_servers`), so from a node the apiserver's
OIDC issuer `keycloak.home.keithjackson.dev` now resolves to Cloudflare and its discovery/JWKS
fetches go through the tunnel (publicly trusted edge cert; no `oidc-ca-file` is configured). The
router NAT hairpin is no longer on that path.
Check: `dig +short @1.1.1.1 plex.home.keithjackson.dev` → `wan.home.keithjackson.dev.` then the WAN IP;
`kubectl -n external-dns logs deploy/cloudflare-ddns`.
Check: `dig +short @1.1.1.1 gitea.home.keithjackson.dev` → Cloudflare anycast addresses (no CNAME is
visible for a proxied record); `kubectl -n external-dns logs deploy/external-dns`.
### History
2026-10-03 → 2026-11-08: records were DNS-only CNAMEs to `wan.home.keithjackson.dev`, with the router
forwarding WAN tcp/80+443 to the Gateway VIP. Before 2026-10-03 they answered the Gateway VIP itself.
## Secrets
+10 -4
View File
@@ -2,9 +2,15 @@
# In-cluster dynamic DNS for the home WAN (2026-10-03).
#
# Keeps ONE record, `wan.${BASE_DOMAIN}` (A, IPv4 only, not proxied), pointed at the home WAN's
# current public IP. Every public name external-dns publishes from the shared Gateway is a CNAME to
# it (platform/gateway/gateway.yaml `external-dns.alpha.kubernetes.io/target`), so a WAN IP change
# moves all of them through this one record, with no Git change and no external-dns involvement.
# current public IP.
#
# WIREGUARD-ONLY SINCE 2026-11-08. From 2026-10-03 every public name external-dns published from the
# shared Gateway was a CNAME to this record. They are now proxied CNAMEs to the Cloudflare Tunnel
# (platform/gateway/gateway.yaml `external-dns.alpha.kubernetes.io/target`), so NOTHING in the zone
# points here any more. The record stays for exactly one consumer: the WireGuard VPN endpoint
# (udp/51820, platform/vpn-ingress, infra/vpn-ingress/new-peer.sh), which cannot ride Cloudflare's
# HTTP proxy and needs the real WAN address. It must stay DNS-only (`PROXIED` unset/false below) —
# an orange-clouded A record would hand peers a Cloudflare anycast address that drops UDP.
#
# OWNERSHIP SPLIT WITH EXTERNAL-DNS. This record carries no external-dns TXT registry entry, so
# external-dns (policy: sync, owner `prod`) never treats it as its own and never deletes it.
@@ -64,7 +70,7 @@ spec:
- name: IP6_PROVIDER
value: "none" # IPv4 WAN only; leaves any AAAA alone
- name: PROXIED
value: "false" # DNS-only: Plex/Gitea/etc. terminate TLS on the Gateway
value: "false" # DNS-only: WireGuard udp/51820 needs the real WAN IP
- name: UPDATE_CRON
value: "@every 5m"
- name: DELETE_ON_STOP
+38 -20
View File
@@ -107,29 +107,47 @@ spec:
# what this instance can see or own; per-resource exclusion only ever REMOVES an object from
# this instance's view, so it cannot make prod capable of deleting a record it does not own.
# ---------------------------------------------------------------------------------------------
# THE PUBLIC TARGET — DECIDED 2026-10-03 (operator): every Gateway-published name goes public
# THE PUBLIC TARGET — THE CLOUDFLARE TUNNEL, PROXIED (2026-11-08; supersedes 2026-10-03)
#
# Until 2026-10-03 every gateway-sourced record answered with the Gateway VIP (192.168.2.20,
# RFC1918), so nothing published here was reachable from the internet. Two facts constrained the
# fix, and both still hold:
# * `external-dns.alpha.kubernetes.io/target` is read in v0.21.0's gateway-httproute source ONLY
# from the GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
# gw.Annotations)`) and from a ListenerSet listener — NEVER from an HTTPRoute. A target on the
# shared Gateway therefore moves EVERY gateway-sourced record at once.
# * The home WAN IP is residential and can change, so a literal IP in Git would rot.
# History: until 2026-10-03 every gateway-sourced record answered the Gateway VIP (RFC1918);
# from 2026-10-03 they were CNAMEs to `wan.${BASE_DOMAIN}` (DNS-only, the home WAN IP, with the
# router forwarding tcp/80+443 to the VIP). Both published an origin address of ours. Now:
# * platform/gateway/gateway.yaml sets `target: <tunnel-uuid>.cfargotunnel.com` on the shared
# Gateway. `target` is read in v0.23.0's gateway-httproute source ONLY from the GATEWAY
# (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(gw.Annotations)`),
# never from an HTTPRoute, so this one value moves every gateway-sourced record at once.
# * `--cloudflare-proxied` below orange-clouds every record by default. It is GLOBAL because
# that is the only practical lever: v0.23.0 reads the per-object `cloudflare-proxied`
# annotation from the ROUTE (or Service), never the Gateway (source/gateway.go:
# `annotations.ProviderSpecificAnnotations(annots)` with `annots := meta.Annotations` of the
# route), so a Gateway-level opt-in does not exist. A CNAME to cfargotunnel.com MUST be
# proxied — DNS-only it resolves to nothing routable — so "proxied by default" is the right
# default for this zone's gateway names.
# * The provider honours a per-object `"false"` over the global default (provider/cloudflare
# shouldBeProxied: ParseBool of the annotation overrides proxiedByDefault). The key it reads
# is `<annotationPrefix>cloudflare-proxied` (annotations.SetAnnotationPrefix re-derives
# CloudflarePrefix), i.e. `external-dns.alpha.kubernetes.io/cloudflare-proxied` here.
#
# THE `service` SOURCE MUST OPT OUT. The L4 LoadBalancers (crafty-mc, unifi) publish an A record
# at their own LB VIP; Cloudflare's proxy is HTTP(S)-only, so a proxied A for a game or UniFi
# inform port is broken by construction. Each carries `cloudflare-proxied: "false"` and stays
# DNS-only, exactly as before this change. conformance/tests/l0-static/
# internal-services-publish-no-public-dns (assertion 9b) FAILS on any `service`-source publisher
# without it, so a new L4 Service cannot silently become a proxied record.
#
# The operator's answers: (a) yes, ALL of them — the five `exposure: public` apps AND the admin
# UIs (auth, crafty, grafana, headlamp, the apex), which sit behind Keycloak / oauth2-proxy;
# (b) the address is kept current IN-CLUSTER. Implementation:
# * platform/gateway/gateway.yaml sets `target: wan.${BASE_DOMAIN}` on the shared Gateway, so
# each published name becomes a CNAME to that one hostname;
# * platform/externaldns/ddns.yaml (favonia/cloudflare-ddns) keeps `wan.${BASE_DOMAIN}` an A
# record at the current WAN IP. That record has no external-dns TXT registry entry, so this
# instance never owns or garbage-collects it.
# Internal services are unchanged: their routes carry `controller: none` (above), which is
# checked BEFORE the target is read, so they still publish nothing. The `service` source (the L4
# LoadBalancers crafty-mc, unifi) does not read the Gateway's annotation and still publishes its
# LB VIP. Whether those should go public is a separate, still-open question.
# checked BEFORE the target is read, so they still publish nothing. Plex's web UI and Headlamp
# joined that set on 2026-11-08 (operator: LAN/VPN-only). `wan.${BASE_DOMAIN}` (ddns.yaml) has no
# external-dns TXT registry entry and is untouched; it now serves only WireGuard udp/51820.
#
# THE RISK THIS ADDS, AND WHAT GUARDS IT. If the annotation prefix ever stops being honoured
# again (the 2026-11-07 v0.23.0 incident), `target` is ignored and every name falls back to the
# Gateway's own address — and with this flag it would be published as a PROXIED A record at
# ${GATEWAY_VIP} (RFC1918; Cloudflare 1003/522s for every public name) while `controller: none`
# is also ignored and internal names publish. Assertion (8) of the L0 check above ties the
# prefix this tree writes to `values.annotationPrefix`; (8b) forbids overriding it here.
extraArgs:
- --cloudflare-proxied
# ---------------------------------------------------------------------------------------------
# CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below.
#
+6 -2
View File
@@ -30,5 +30,9 @@ east-west default-deny (docs/09).
- **Phase 2 (WS-NET):** pin the L2-announced ingress VIP / IP pool (Cilium L2 announcement, docs/09);
add the HTTP→HTTPS redirect route; confirm the cross-namespace ReferenceGrant pattern with WS-APPS.
- **Public exposure** is a **manual edge port-forward** of 80/443 to the Gateway VIP (single SOHO
router, docs/09) — a documented manual step, the analogue of the old pfSense exception.
- **Public exposure** is via the **Cloudflare Tunnel** (2026-11-08, `platform/cloudflared`): the
Gateway's `external-dns.alpha.kubernetes.io/target` is `<tunnel-uuid>.cfargotunnel.com`, so every
published name is a proxied CNAME to the tunnel and cloudflared forwards to this Gateway's
ClusterIP. No inbound WAN 80/443 forward is needed (the old manual edge port-forward is retired —
runbooks/cloudflare-tunnel.md). LAN/VPN-only names (internal apps, plex, headlamp) carry
`controller: none` on their route and are 404'd by the tunnel.
+28 -14
View File
@@ -8,22 +8,36 @@ metadata:
name: shared
namespace: gateway
annotations:
# PUBLIC DNS TARGET (2026-10-03, operator decision). Without this, external-dns publishes every
# route on this Gateway with the Gateway's OWN address (${GATEWAY_VIP}, RFC1918), which is
# unreachable from the internet. With it, every published `<svc>.${BASE_DOMAIN}` (and the apex)
# becomes a CNAME to `wan.${BASE_DOMAIN}`, an A record that platform/externaldns/ddns.yaml keeps
# pointed at the home WAN's current public IP. The WAN forwards 80/443 to ${GATEWAY_VIP}.
# PUBLIC DNS TARGET = THE CLOUDFLARE TUNNEL (2026-11-08, operator decision; replaces the
# 2026-10-03 `wan.${BASE_DOMAIN}` target). external-dns v0.23.0's gateway-httproute source reads
# `target` from the GATEWAY only (source/gateway.go: `overrides: annotations.
# TargetsFromTargetAnnotation(gw.Annotations)`) — never from an HTTPRoute — so this one value moves
# EVERY published route on this Gateway at once. Each published `<svc>.${BASE_DOMAIN}` (and the
# apex) becomes a CNAME to `<tunnel-uuid>.cfargotunnel.com`, and platform/externaldns/helmrelease.yaml
# sets `--cloudflare-proxied` so the CNAME is orange-clouded: public clients resolve Cloudflare
# anycast, Cloudflare terminates edge TLS (Total TLS) and hands the request to the in-cluster
# cloudflared connectors (platform/cloudflared), which forward to this Gateway's ClusterIP. No
# origin address of ours appears in public DNS, and no inbound WAN port is needed for HTTP(S).
#
# Scope is EVERY route on this Gateway that publishes at all — the operator chose to make the
# admin UIs (auth, grafana, headlamp, crafty, apex) publicly resolvable alongside the five
# `exposure: public` apps. `internal` services stay unpublished via their
# `external-dns.alpha.kubernetes.io/controller: none` route annotation, which external-dns checks
# BEFORE it reads this target. external-dns v0.21.0 reads `target` from the Gateway only, never
# from an HTTPRoute — see platform/externaldns/helmrelease.yaml.
# THE UUID IS A LITERAL, AND IT MUST EQUAL `tunnel:` IN platform/cloudflared/config.yaml. A CNAME
# to any other <uuid>.cfargotunnel.com is a dead name (Cloudflare answers 1016). That equality is
# asserted on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns
# (assertion 9a). A literal rather than a cluster-var keeps this change off clusters/**.
#
# LAN and in-cluster clients are unaffected: PiHole and CoreDNS answer ${BASE_DOMAIN} with
# ${GATEWAY_VIP} directly (split-horizon), so they never take this public path.
external-dns.alpha.kubernetes.io/target: "wan.${BASE_DOMAIN}"
# Scope: every route on this Gateway that publishes at all. `internal` services (and the
# operator-designated LAN/VPN-only names: plex, headlamp) stay unpublished via their
# `external-dns.alpha.kubernetes.io/controller: none` route annotation, which external-dns
# applies BEFORE it reads this target. Those same names map to `http_status:404` in the tunnel's
# ingress rules, so even a hand-made record cannot reach them through the tunnel.
#
# In-cluster clients are unaffected: CoreDNS answers ${BASE_DOMAIN} with ${GATEWAY_VIP} directly
# (split-horizon). LAN clients use PiHole's EXPLICIT list (catalog/pihole.yaml, since 2026-11-09):
# LAN-only names and gitea/nextcloud/keycloak/auth get ${GATEWAY_VIP}; every other published
# name takes this public path through Cloudflare, from the LAN as from anywhere.
#
# `wan.${BASE_DOMAIN}` (platform/externaldns/ddns.yaml) is NO LONGER a target of anything here;
# it stays only as the WireGuard udp/51820 endpoint.
external-dns.alpha.kubernetes.io/target: "fc81938b-5855-47f1-8a27-6286f56dda93.cfargotunnel.com"
spec:
gatewayClassName: cilium
listeners:
+7
View File
@@ -18,6 +18,13 @@ metadata:
annotations:
homelab.keithjackson.dev/auth-mode: oidc
homelab.keithjackson.dev/auth-groups: platform-admins
# LAN/VPN-ONLY (operator decision 2026-11-08). Disowned from external-dns, so the public
# headlamp record and its TXT are DELETED and nothing republishes them; LAN/VPN clients keep
# resolving the name via PiHole's explicit entry for it (-> the Gateway VIP). The Cloudflare
# Tunnel also maps headlamp.${BASE_DOMAIN} to http_status:404 (platform/cloudflared/config.yaml).
# Same mechanism the renderer emits for `exposure: internal` services; asserted by
# conformance/tests/l0-static/internal-services-publish-no-public-dns (9c).
external-dns.alpha.kubernetes.io/controller: none
spec:
parentRefs:
- name: shared
@@ -24,7 +24,7 @@ spec:
chart:
spec:
chart: kube-prometheus-stack
version: ">=88.0.0 <89.0.0" # k8s-1.36 catch-up (po v0.93; CRDs pre-applied server-side, Helm won't update them). Renovate maintains
version: ">=93.1.0 <94.0.0" # k8s-1.36 catch-up (po v0.93; CRDs pre-applied server-side, Helm won't update them). Renovate maintains
sourceRef:
kind: HelmRepository
name: prometheus-community
+7 -4
View File
@@ -72,11 +72,14 @@ Both are deliberate, and both fail **visibly** rather than silently:
- **The nodes resolve the registry hostname to the WAN, so the TLS they fail to verify is
the router's — DIAGNOSED 2026-10-06, and fixed in this change.** containerd pulls as a
*node* process, so it resolves through the node's public upstreams
(`node_baseline_dns_servers` = the LAN gateway) rather than through Pi-hole. The public
answer for every name in this zone is a CNAME to `wan.home.keithjackson.dev` — the WAN
IP — which this router does not hairpin; it answers :443 with its own placeholder
(`node_baseline_dns_servers` = the LAN gateway) rather than through Pi-hole. At the time,
the public answer for every name in this zone was a CNAME to `wan.home.keithjackson.dev`
— the WAN IP — which this router does not hairpin; it answered :443 with its own placeholder
certificate `CN=%D` carrying **no extensions, therefore no SANs**, which is precisely
`x509: certificate is not valid for any names`. The Gateway, the wildcard certificate
`x509: certificate is not valid for any names`. (Since 2026-11-08 the public answer is
Cloudflare's edge via the Tunnel, not the WAN IP — see `platform/externaldns/README.md`.
The pin is still the right path: it keeps registry pulls on the LAN and off Cloudflare's
proxy.) The Gateway, the wildcard certificate
and the digest were never implicated. The fix is the pin this repository already uses
for the Keycloak issuer, for the identical reason: `gitea.{{ k3s_base_domain }}` is now
in `node_baseline_hosts_pins` — node configuration is Ansible's, declared in the
+54 -12
View File
@@ -42,8 +42,9 @@ no version string to match against a CVE, and no amplification to abuse.
That is the whole security argument, and it is why the exposed port carries no IP allowlist (which
would also have made roaming clients unusable — a phone's address changes with every cell tower).
Compare `apps/wave-0/pihole-dns.yaml`, which *does* pin `loadBalancerSourceRanges` to the LAN: a
resolver answers anybody, so it must be scoped by address. WireGuard answers nobody.
Compare `apps/wave-0/pihole-dns.yaml`, which *does* pin `loadBalancerSourceRanges` to the LAN and
this tunnel's client subnet: a resolver answers anybody, so it must be scoped by address. WireGuard
answers nobody.
What it does **not** protect against is a stolen device — anyone holding a device's private key
*is* that device. That is what revocation is for: delete the stanza, merge, done (~1 min).
@@ -55,9 +56,9 @@ What it does **not** protect against is a stolen device — anyone holding a dev
┌──────────────┐ udp/51820 ┌───────────────┐ forward ┌────────────────────────────┐
│ WireGuard app│─────────────▶│ WAN 51820 ──▶ │───────────▶│ wg0 10.90.0.1/24 │
│ 10.90.0.2/32 │ over the │ node IP:51820 │ │ decrypt, then FORWARD │
└──────────────┘ internet └───────────────┘ │ SNAT to the node's addr │
└──────────────┘ internet └───────────────┘ │ SNAT (LAN), eBPF LB (VIPs)│
└─────────────┬──────────────┘
│ as `host`/`remote-node`
│ translated per destination (below)
┌─────────────────────────────────┼──────────────────────┐
▼ ▼ ▼
LAN 192.168.2.0/24 Gateway VIP .20 (16 web UIs) WAN / internet
@@ -86,9 +87,21 @@ resolved, because `apps/wave-0/pihole-dns.yaml`'s ingress policy deliberately om
entity. A VPN that looks up and is useless is the worst available outcome, and nothing in the
manifests would have hinted at it.
In the **host** netns there is no pod identity to carry. Forwarded client traffic is SNATed to the
node's own address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` —
byte-for-byte what a LAN client looks like. Three further problems vanish with it:
In the **host** netns there is no pod identity to carry, which is what avoids the 403s. Clients do
**not**, however, all arrive SNATed to the node. Measured 2026-11-08: Cilium counts `wg0` among its
kube-proxy-replacement devices (`cilium-dbg status`: `eno1`, `wg0`, `tun0`), so what a packet looks
like depends on where it is going:
| destination | handled by | source the destination sees |
|---|---|---|
| LAN hosts, the internet | kernel FORWARD, then the iptables MASQUERADE in [`daemonset.yaml`](./daemonset.yaml) | the node's address |
| LoadBalancer VIPs (Gateway `.20`, PiHole `.23`, ...) | Cilium's eBPF load balancer, as the packet enters from `wg0`, **before** MASQUERADE | `10.90.0.x` at the VIP's source-range check, identity `world` |
The second row is why PiHole's `loadBalancerSourceRanges` has to list `${WG_CLIENT_CIDR}` as well as
the LAN: left LAN-only, tunnel DNS is predicted to die with `DROP_NOT_IN_SRC_RANGE`. That prediction
and the row's source column come from reading Cilium's datapath source; **no peer has ever completed
a handshake, so neither has been observed live.** The Gateway VIP carries no source ranges and needs
no change. Three further problems vanish with the host netns:
* `net.ipv4.ip_forward` is **already 1** on these nodes (`node_baseline` writes
`/etc/sysctl.d/99-k3s-cilium.conf`). It is *not* in the kubelet's safe-sysctl allowlist, so a
@@ -99,7 +112,9 @@ byte-for-byte what a LAN client looks like. Three further problems vanish with i
`conformance/tests/l0-static/l2-announced-vips-serve-from-any-node`) and no
`externalTrafficPolicy` trade-off to make at all.
* **No double encapsulation.** Client UDP lands on the node's own NIC, so the underlay is a flat
1500-byte path, not Cilium's 1450-byte VXLAN. That is what makes `MTU = 1420` simply correct.
1500-byte path, not Cilium's 1450-byte VXLAN. That is what makes `MTU = 1420` correct on the
server's `wg0`. It says nothing about a client's own path: `new-peer.sh` defaults client profiles
to `1280` (see "Troubleshooting" in `runbooks/vpn-remote-access.md`).
`hostNetwork` is *permitted* here, and not by oversight:
`platform/kyverno/policies/restrict-host-namespaces.yaml` matches `namespaceSelector:
@@ -163,6 +178,30 @@ internet path. To enforce less *server-side*, add filtering to the `PostUp` chai
before the accept rules) — the FORWARD chain there is the only enforcement point, since no
NetworkPolicy applies.
## Phone profiles: one tunnel for home, one for away
A phone that needs the LAN-only apps runs full tunnel (`AllowedIPs = 0.0.0.0/0`, `DNS = 192.168.2.23`)
everywhere, so it needs **two tunnels**, each its own peer with its own key and address:
| tunnel | peer | address | Endpoint | use when |
|---|---|---|---|---|
| `away` | `keith-phone` | `10.90.0.2` | `wan.home.keithjackson.dev:51820` | off the home network |
| `home` | `keith-phone-home` | `10.90.0.3` | `192.168.2.10:51820` | on the home Wi-Fi |
**`wan.` cannot work from inside the house.** It is an unproxied A record holding the house's WAN
address (`platform/externaldns/README.md`), and the router has no hairpin NAT, so a LAN client that
sends to it never comes back in through the port-forward. The Cloudflare Tunnel is no way round: it
carries HTTP(S) to the Gateway and WireGuard is UDP to a node, so `wan.` has to stay DNS-only and
`away` has to keep using it.
**`home` depends on `k3s-prod-1`.** Its Endpoint is that node's address, with no failover: if
prod-1 is down, `home` cannot handshake even though the other two listeners are healthy. The fix
is client-side: edit the tunnel's Endpoint to `192.168.2.11` or `.12`, since every node answers the
same keys. No server change and no new key is needed. The same dependency applies to `away` if the
WAN forward points at prod-1.
Android allows one active VPN at a time, so switch tunnels by hand when leaving or arriving.
## Operating it
* **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
@@ -215,10 +254,13 @@ right way; one was a defect in my own manifest.
handshake — and would be a real secret, forcing `peers.conf` into SOPS and costing the
reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
`wg-server-key` Secret.
* **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6
still reaches IPv6 sites *outside* the tunnel. Functionally harmless, a privacy leak if the point
was to hide the client's network. Fix is an `fd..` ULA on `wg0` plus `::/0` in the client's
`AllowedIPs`; not done because nothing here needs it.
* **No IPv6.** The server offers no IPv6 tunnel address and `new-peer.sh` emits only `0.0.0.0/0`.
What happens to a client's IPv6 then depends on the client. The **Android app does not leak it**:
its `GoBackend` skips `allowFamily()` when the config is one peer with a default route, so Android
blocks IPv6 for the VPN's duration and apps fall back to IPv4. Other clients are not covered by
that code; expect `wg-quick` on a dual-stack Linux host to leave IPv6 outside the tunnel. Fix is
an `fd..` ULA on `wg0` plus `::/0` in the client's `AllowedIPs`; not done because nothing here
needs it.
* **No in-cluster-service reach beyond what the LAN has.** Clients reach cluster apps the way a LAN
browser does — through the Gateway VIP. Pod IPs and `10.43.0.0/16` service IPs are not routed
into the tunnel.
+6 -3
View File
@@ -30,9 +30,12 @@
# `cluster` entity, so VPN clients would not have been able to resolve `*.${BASE_DOMAIN}` either.
# The VPN would have looked up and been useless, which is the worst failure shape available.
#
# In the host netns there is no pod identity to carry: forwarded client traffic is SNATed to the
# NODE's address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` —
# exactly what a LAN client looks like. Three further problems disappear with it:
# In the host netns there is no pod identity to carry, so a client never looks like a pod. What a
# destination sees depends on how the packet is handled. LAN hosts and the internet: forwarded,
# then MASQUERADEd to the NODE's address by the PostUp rule below. LoadBalancer VIPs: Cilium's load
# balancer takes the packet as it enters from wg0, before that rule, so the source is still
# 10.90.0.x with identity `world`, like a LAN browser (measured 2026-11-09 for PiHole .23 and Plex
# .24; see README). Three further problems disappear with it:
# * net.ipv4.ip_forward is ALREADY 1 on these nodes (node_baseline writes it to
# /etc/sysctl.d/99-k3s-cilium.conf), so no pod sysctl is needed — and `net.ipv4.ip_forward`
# is NOT in the kubelet's safe-sysctl allowlist, so a pod-netns version of this would have
+4
View File
@@ -90,3 +90,7 @@
[Peer]
PublicKey = pBRSHrRyG2hnsuZ6FvHfEzF8hOVTF5stHUb/7fW2Km8=
AllowedIPs = 10.90.0.2/32
# keith-phone-home — <owner>, enrolled 2026-11-08 (at-home profile, LAN Endpoint; no hairpin NAT)
[Peer]
PublicKey = AT8lBJHqrVkv8RL9n0qWDsh5nyl0e3qgrfTAaYKgBz0=
AllowedIPs = 10.90.0.3/32
+41 -11
View File
@@ -31,10 +31,29 @@ The operator decisions it encodes (2026-11-08):
policy admits neither the `kube-apiserver` nor the `host` entity. The L0 check
`cloudflared-ingress-rules` fails the gate if that ever changes.
**Nothing goes through the tunnel until DNS points at it.** Cloudflare routes a hostname through a
tunnel only when its record is a CNAME to `<UUID>.cfargotunnel.com`. Phase 1 (this component)
creates no such record; moving public DNS is a separate, later change. So in Phase 1 the alerts
firing means "the tunnel is not ready", not "users are down".
**Cloudflare routes a hostname through the tunnel only when its record is a proxied CNAME to
`<UUID>.cfargotunnel.com`.** Phase 1 created no such record.
**Phase 2 — DONE (2026-11-08, branch `feat/dns-via-cloudflare-tunnel`).** external-dns now publishes
every gateway name as a proxied CNAME to the tunnel: the shared Gateway's
`external-dns.alpha.kubernetes.io/target` is `<UUID>.cfargotunnel.com` and the HelmRelease sets
`--cloudflare-proxied` (see `platform/externaldns/README.md`). Plex's web UI and Headlamp are
disowned (`controller: none`), so their public records are deleted; unifi and crafty-mc stay DNS-only
A records (`cloudflare-proxied: "false"`); `wan.` stays for WireGuard. The L0 check
`internal-services-publish-no-public-dns` (9) pins all of this, including that the Gateway target's
UUID equals `tunnel:` in `config.yaml`. **So from Phase 2 on, these alerts firing DOES mean public
users are down** — every public HTTP(S) name depends on the connectors.
### After the Phase 2 merge — operator step (router)
Once `dig +short @1.1.1.1 <public name>` returns Cloudflare anycast addresses for the public names
(and external-dns logs show the UPDATEs), nothing legitimate reaches the WAN on 80/443 any more:
1. **Close** the router's WAN **tcp/80** and **tcp/443** port-forwards to the Gateway VIP.
2. **Keep** WAN **udp/51820** (WireGuard → `wan.home.keithjackson.dev`) and WAN **tcp/32400**
(Plex remote streaming via plex.tv; `runbooks/plex-remote-access.md`).
3. Verify from off-LAN (phone on cellular): a public name still loads (via the tunnel), and
`curl -m5 https://<WAN IP>/` now times out instead of answering.
Before the steps below are done, the Deployment CrashLoops (placeholder UUID and credentials) and
`CloudflaredTunnelDown` fires. That is expected, and it is contained: the `cloudflared` Kustomization
@@ -103,8 +122,8 @@ whole of what this Deployment needs; no `cert.pem` ever belongs in the pod.
- **Never run `cloudflared tunnel route dns`** — not for a real hostname, not for a test one.
**external-dns owns the `home.keithjackson.dev` records** (`platform/externaldns`, `policy: sync`);
a record created behind its back either fights it or is deleted by it, and a real hostname
routed that way is an unreviewed DNS flip. Moving records onto the tunnel is the separate Phase 2
change. The only record this runbook creates is the hand-made canary in step 6, and you delete it.
routed that way is an unreviewed DNS flip. Records reach the tunnel only through external-dns
(Phase 2, done). The only record this runbook creates is the hand-made canary in step 6, and you delete it.
- Escalate to Cloudflare status (https://www.cloudflarestatus.com) if every replica is
Running, egress shows no drops, and logs show edge connection errors on both QUIC and HTTP/2.
@@ -300,6 +319,9 @@ is deleted, which is why the last step is not optional.
```
OLD=$(sed -n 's/^tunnel: //p' platform/cloudflared/config.yaml)
sed -i "s/^tunnel: ${OLD}$/tunnel: <NEW UUID>/" platform/cloudflared/config.yaml
# Phase 2: the Gateway's external-dns target carries the SAME UUID. Change both in one commit;
# the L0 check internal-services-publish-no-public-dns (9a) fails the gate if they disagree.
sed -i "s/${OLD}\.cfargotunnel\.com/<NEW UUID>.cfargotunnel.com/" platform/gateway/gateway.yaml
```
3. **Swap the encrypted credentials** with *Enrolling* step 4 — the same `sops set`, then the
same `ok`-only verify, which now checks the JSON's `TunnelID` against the NEW UUID. Never
@@ -315,8 +337,9 @@ is deleted, which is why the last step is not optional.
cf tunnel delete <old name>
cf tunnel list # the old tunnel is gone
```
After the Phase 2 DNS flip, records point at `<UUID>.cfargotunnel.com`: confirm external-dns
has repointed every record to the NEW UUID before deleting, or those hostnames go dark.
Since Phase 2, records point at `<UUID>.cfargotunnel.com`: confirm external-dns has repointed
every record to the NEW UUID (its logs show an UPDATE per name) before deleting, or those
hostnames go dark.
Then replace `tunnel.json`/`cert.pem` in the password manager and delete them from disk.
## Rollback
@@ -330,8 +353,15 @@ is deleted, which is why the last step is not optional.
'flux --context k3s-prod suspend kustomization cloudflared &&
kubectl --context k3s-prod -n cloudflared scale deploy/cloudflared --replicas=0'
```
Cloudflare then answers tunnel hostnames with a 530/1033 error. Follow up in git, then
Cloudflare then answers tunnel hostnames with a 530/1033 error — **from the LAN too** for the
names PiHole does not list (apex, grafana, crafty, seerr; since 2026-11-09). Follow up in git, then
`flux resume kustomization cloudflared`.
- **After the DNS flip:** the fast path back is DNS, not the tunnel — restore the previous records
(that change's own runbook). Revoking the tunnel credential (`cf tunnel delete`, or rotating it
- **After the DNS flip (Phase 2):** the path back is a revert PR of `feat/dns-via-cloudflare-tunnel`
(Gateway target back to `wan.${BASE_DOMAIN}`, drop `--cloudflare-proxied`) — external-dns then
rewrites every record DNS-only to `wan.`. That only works if the router's WAN tcp/80+443 forwards
are **re-opened** first (they are closed after Phase 2). **Also restore PiHole's zone wildcard**
(`address=/${BASE_DOMAIN}/${GATEWAY_VIP}` in `catalog/pihole.yaml`, and drop the
`pihole-lan-names` L0 assertion (a)): since 2026-11-09 the apex, grafana, crafty and seerr are not
listed there and resolve publicly, and a public `wan.` answer does not hairpin from the LAN. Never edit the records by hand:
external-dns owns them and `policy: sync` reverts a hand edit. Revoking the tunnel credential (`cf tunnel delete`, or rotating it
in the Zero Trust dashboard) cuts every connector at once if the credential is believed leaked.
+17 -3
View File
@@ -11,6 +11,19 @@ env — note `ADVERTISE_IP` is deliberately NOT set; see "Why not the Gateway UR
**Related:** `runbooks/plex-quicksync.md` and `runbooks/plex-claim.md` — the other two Plex
procedures with a manual last mile.
## The Plex web UI is LAN/VPN-only (2026-11-08)
When public DNS moved to the Cloudflare Tunnel (`runbooks/cloudflare-tunnel.md`, Phase 2), the
operator kept **Plex off the tunnel**. `catalog/plex.yaml` is now `exposure: internal`, so the
`plex.home.keithjackson.dev` route carries `external-dns ... controller: none` and its public record
is deleted; the tunnel also 404s `plex` and `plex-vip`. On the LAN (and over WireGuard) the web UI
at `https://plex.home.keithjackson.dev` works exactly as before through PiHole's split-horizon answer.
**Remote streaming is unaffected** — it never used that name. It is the path below: plex.tv hands
clients the `plex.direct` URL (`ADVERTISE_IP`) for the L4 VIP `192.168.2.24`, reached through the
WAN **tcp/32400** forward. That forward must stay open when the WAN 80/443 forwards are closed after
the tunnel cut-over. Remote users use app.plex.tv, not our hostname.
## What was actually broken (measured 2026-10-03)
Plex was serving correctly and was reachable on the LAN. Two independent things stopped remote
@@ -44,9 +57,10 @@ either way — this repo has no such vantage point and must not pretend otherwis
router by hand, so it must never move on a reschedule.
While in the router, **check what else is bound to the WAN interface.** If remote management is
listening on 443 (see the certificate measurement above), move it or disable it — otherwise the
Gateway's 80/443 forward is shadowed and the Plex *web UI* over the internet stays broken even
though remote *streaming* works. These are two separate paths and they fail independently.
listening on 443 (see the certificate measurement above), move it or disable it. (Historical: this
used to shadow the Gateway's 80/443 forward for the Plex *web UI* over the internet. Since
2026-11-08 the web UI is LAN/VPN-only and public HTTP(S) goes through the Cloudflare Tunnel, so the
80/443 forwards are retired; only tcp/32400 matters here.)
Confirm the VIP is actually answering on the LAN first, so a failure later is unambiguous:
+1 -1
View File
@@ -74,7 +74,7 @@ Work through these in order; the first three are by far the most likely.
|---|---|---|
| `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
| `ImagePullBackOff`, digest `sha256:0000…` | (historical: before 2026-10-30 the committed digest was an all-zero placeholder) the image has never been published at the digest named | CI publishes it, not a person: check the latest `images` workflow run on `main` (it fails loudly if the `REGISTRY_PUSH_TOKEN` secret is missing; re-run it with `workflow_dispatch`), then merge Renovate's digest-bump PR, which updates **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) together — but read the row below first |
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and (before the 2026-11-08 Tunnel cutover; the public answer is now Cloudflare) got the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
| `ImagePullBackOff` with `401 Unauthorized` / `unauthorized` | the pull credential is wrong, absent or scoped without `read:package`. The `pm-agent-registry` Secret exists and is referenced; its *value* is what starts as a placeholder | check the key names only, never values: `kubectl -n pm-agent get secret pm-agent-registry -o go-template='{{range $k, $v := .data}}{{$k}}{{"\n"}}{{end}}'` should list `.dockerconfigjson`. Then confirm the token carries `read:package`, that its account can read the org's packages (a team with the Packages unit — see the `.example`), and that the registry key inside the JSON is the bare host `gitea.home.keithjackson.dev` — with a scheme or a port it matches nothing and the pull goes out anonymously, which looks identical to a missing Secret. Steps: `platform/pm-agent/pm-agent-registry.sops.yaml.example` |
| `GET /user` answers `403` / `token does not have at least one of required scope(s): [read:user]` at startup, then a CrashLoop | the token lacks `read:user` — Gitea 1.24.3 scopes the whole `/api/v1/user` group, and the dispatcher's first call is `GET /user` | mint a new token with all four scopes (`read:user`, `write:repository`, `write:issue`, `read:package`) and put it in **both** Secrets. Add `read:notification` if you want the cheap pull-request mention query (`platform/pm-agent/README.md`) |
| a worker Job is refused by admission (`pm-agent-job-shape`, `imagePullSecrets`) | `PM_WORKER_IMAGE` is a digest built before the template carried `imagePullSecrets` (PR #61), so its baked template lacks the entry admission requires | pin the agent-runner digest CI published after #61 merged (Renovate's bump PR); until then keep `PM_DRY_RUN=true` |
+98 -12
View File
@@ -252,7 +252,7 @@ cellular, with Wi-Fi off, and check all four — each proves a different hop:
|---|---|---|
| 1 | tunnel shows a recent handshake in the WireGuard app | the WAN forward + the server key + the peer stanza |
| 2 | `ping 192.168.2.1` (the router) | decrypt → FORWARD → LAN, and the SNAT return path |
| 3 | open `https://plex.home.keithjackson.dev` | PiHole DNS over the tunnel **and** the Gateway accepting a `host`-identity source — the specific thing a pod-networked VPN would have 403'd |
| 3 | open `https://plex.home.keithjackson.dev` | PiHole DNS over the tunnel **and** the Gateway admitting a tunnel client, which a pod-networked VPN would have 403'd. Expected source identity is `world`, like a LAN browser: measured 2026-11-09 on VIPs `.23` and `.24`, not yet observed on the Gateway's `.20` |
| 4 | visit an IP-echo site; it should show `136.51.115.252` | full-tunnel WAN egress out the house, not the carrier |
Then confirm server-side:
@@ -260,18 +260,97 @@ Then confirm server-side:
```bash
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show'
# a `latest handshake` under ~3 min, and non-zero transfer, on the expected peer.
# a `latest handshake` under ~3 min, and non-zero transfer, on the expected peer. `ds/wireguard`
# picks ONE pod; if it reads 0, ask the pod on the node holding the forward (see Troubleshooting).
```
**If check 3 fails while 2 passes**, the tunnel is fine and the problem is DNS or the Gateway, not
WireGuard. Check the client's `DNS = 192.168.2.23` and that PiHole is up.
**If 1 and 2 pass but large transfers hang**, it is MTU. The profile ships `MTU = 1420` (1500 − 60
bytes of WireGuard encapsulation); drop the client to `1280` and re-test.
**If 1 and 2 pass but pages or large transfers hang**, suspect the client MTU; see
Troubleshooting.
**If nothing handshakes at all**, you cannot tell a wrong forward from a wrong key from outside —
by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes
if the router is not delivering the packets, and a `handshake` with no completed session if it is.
**If nothing handshakes at all**, you cannot tell a wrong forward from a wrong key from the
device, by design. `wg show` cannot tell them apart either: it reads the same in both cases. The
node's conntrack table can; see Troubleshooting.
## Troubleshooting
Ask the pod on **the node carrying the traffic**: the node the WAN forward names for `away`,
`k3s-prod-1` for `home`. Sessions are per node, and the other two pods read `0`.
```bash
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n vpn-ingress get pods -o wide'
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n vpn-ingress exec <pod> -c wireguard -- wg show wg0 latest-handshakes'
# likewise: `wg show wg0 endpoints`, `wg show wg0 transfer`
```
**Use only those three subcommands to diagnose.** Never `wg showconf`, `wg show wg0 private-key` or
`wg show wg0 preshared-keys`: they print key material into your terminal and transcript.
| output | reading |
|---|---|
| `latest-handshakes`: `<public key> 0` | no handshake with this peer since the pod started |
| `endpoints`: `<public key> (none)` | the server has not accepted a packet from this peer since the pod started |
| `transfer`: `<public key> <rx> <tx>` | bytes, as the server counts them |
### The app shows TX, no RX and no "Latest handshake"
**Symptom.** The app's transmit counter climbs and receive stays at zero. On the server,
`endpoints` is `(none)` and `latest-handshakes` is `0` for that peer.
**Cause.** A key mismatch: the server's public key in the profile, or the device's public key in
`peers.conf`. WireGuard drops an initiation it cannot authenticate and sends nothing back, so
neither end logs an error. Measured 2026-11-09: a mistyped server public key in the phone's profile.
**Check.** First prove the packets arrive. The pod has no `conntrack` binary, but in `hostNetwork`
it can read the host's table:
```bash
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n vpn-ingress exec <pod> -c wireguard -- grep dport=51820 /proc/net/nf_conntrack'
```
- A line whose first tuple is `src=<the phone's public address> ... dport=51820`, marked
`[UNREPLIED]`: the datagrams reach the node and the server stays silent. It is the keys.
- No such line: the packets are not arriving. Look at the router forward, the DNS name, or the
device's network.
Then compare the keys **by copy and paste, never by eye and never retyped**: the server's public
half (the first command in Step 2's "Record it in your PASSWORD MANAGER") against the profile's
`PublicKey`, and the device's public key from the app against its stanza in `peers.conf`.
### Works on Wi-Fi, stalls on mobile data, handshake healthy
**Symptom.** `latest-handshakes` is recent and some traffic flows, but names or pages fail on
mobile data. The same device is fine on Wi-Fi.
**Cause.** Most likely a client MTU too large for the carrier's path: a tunnel MTU of `1420` means
outer packets of up to 1480 bytes. This is inferred from one before-and-after, on one phone and one
carrier (2026-11-09): `1420` failed and clearing the field fixed it. The path MTU itself was not
measured.
**Check.** In the Android WireGuard app, edit the tunnel, clear the `MTU` field or set it to `1280`,
save, then turn the tunnel off and on. A blank field means `1280` in the app's default userspace
backend (`GoBackend`: `getMtu().orElse(1280)`). On other clients set `MTU = 1280` in the profile.
`new-peer.sh` now generates `MTU = 1280` unless `WG_MTU` overrides it. A profile already on a device
does not change, so edit it by hand. If `1280` does not help, it is not the MTU: go back to
Step 6's check 3.
### The `away` profile cannot connect from home
**Symptom.** On the home network, `away` never handshakes.
**Cause.** Its Endpoint is `wan.home.keithjackson.dev`, the house's WAN address, and the router has
no hairpin NAT. WireGuard cannot use the Cloudflare Tunnel instead.
**Check.** Turn Wi-Fi off. If `away` handshakes on mobile data, the profile is fine: use the `home`
tunnel indoors, as described in
[Phone profiles: one tunnel for home, one for away](../platform/vpn-ingress/README.md#phone-profiles-one-tunnel-for-home-one-for-away).
## Revoking a device
@@ -293,10 +372,17 @@ top follow-up for this component.** Until then:
## Known limitations
* **No IPv6 in the tunnel.** A full-tunnel client with native IPv6 still reaches IPv6 sites outside
the tunnel. Functionally harmless; a privacy leak if hiding the client's network was the point.
* **Client IPs are not preserved** on the LAN side: everything arrives SNATed as the node's
address. A static route for `10.90.0.0/24` → the forwarded node, added on the router, would
preserve them — at the cost of another hand-maintained off-repo route.
* **No IPv6 in the tunnel.** What a client's IPv6 does then depends on the client. In the Android
app's default `GoBackend`, a config with one peer and a default route skips `allowFamily()`, so
Android should block IPv6 while the tunnel is up and apps fall back to IPv4. That is read from
upstream source, not observed on a device. A `--split` profile has no default route, so its IPv6
stays outside the tunnel. Expect `wg-quick` on a dual-stack Linux host to leave IPv6 outside too
(not tested).
* **Client IPs are only partly preserved.** LAN hosts and the internet see the node's address
(MASQUERADE). A static route for `10.90.0.0/24` → the forwarded node, added on the router, would
preserve them there, at the cost of another hand-maintained off-repo route. LoadBalancer VIPs are
different: Cilium handles the packet before MASQUERADE, so the source stays `10.90.0.x`, identity
`world` (BPF conntrack on prod-1, 2026-11-09, VIPs `.23` and `.24`). Any `loadBalancerSourceRanges`
must therefore list the tunnel subnet, as PiHole's do.
* **Rollouts briefly drop the forwarded node.** `maxUnavailable: 1`, so one node at a time; clients
re-handshake automatically (retry every 5s, rekey every 2 min).
+7 -9
View File
@@ -82,13 +82,10 @@ platform must render, and cites the requirement each mapping satisfies. Changing
all seven the moment it is set. Per-resource exclusion has exactly the blast radius of the
defect. The filter stays available as hardening once those seven are adjudicated — see
`platform/externaldns/helmrelease.yaml` and the register in the L0 check.
- `public` renders the record but the renderer does **not** yet set a target: the record still
resolves to the shared Gateway VIP, i.e. an RFC1918 address, so a `public` service is not in fact
publicly reachable. Fixing that is **not** a renderer change on v0.21.0 —
`source/gateway.go` reads `external-dns.alpha.kubernetes.io/target` only from the **Gateway** (and
a ListenerSet), never from an HTTPRoute, so a per-service target override is not expressible while
the public services share the `shared` Gateway. The open decision is recorded in
`platform/externaldns/helmrelease.yaml`.
- `public` renders the record but the renderer does **not** set a target: external-dns reads
`external-dns.alpha.kubernetes.io/target` only from the **Gateway** (and a ListenerSet), never from
an HTTPRoute, so the target is set once, on the `shared` Gateway — since 2026-11-08 the Cloudflare
Tunnel, proxied (`platform/gateway/gateway.yaml`, `platform/externaldns/README.md`).
- Exposure does **not** relax east-west default-deny (docs/09 §"Interaction with default-deny").
- **Requirements:** R-DNS-02/03 (docs/09), R-NET-04 (docs/03).
@@ -323,8 +320,9 @@ R-OBS-01 (the SLI must follow the rendered kind).
`public` service**. An `internal` service's route is rendered with ExternalDNS's controller
annotation set to `none`, which removes it from ExternalDNS's view entirely (see §`exposure`), so
its `<host>` exists **only** in the internal split-horizon answer PiHole serves, never in the
public zone. A `public` record's TARGET is still the Gateway VIP, not a routable address — open
operator decision, see §`exposure`.
public zone. A `public` record's TARGET is the Cloudflare Tunnel (a proxied CNAME, from the
shared Gateway's target annotation; `platform/externaldns/README.md`), and a `public` name is
not listed in PiHole unless it must stay on the LAN path.
- The `host` pattern is enforced to `<label>.home.keithjackson.dev` by the schema.
- **Requirements:** **R-TLS-01/02/03** (docs/08), **R-DNS-01/02/03** (docs/09).