Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
  jakman85andClaude Opus 5.5
|
f13d336630 |
No files matched your search
@@ -76,7 +76,7 @@ bug but the honest answer, because proofs are claims about a specific tree on a
|
||||
|---|---|
|
||||
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
|
||||
| 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; **now 3× mini-PC bare metal** (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13) |
|
||||
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the LAN resolver**, answering an explicit list of `${BASE_DOMAIN}` names (LAN-only apps, gitea/nextcloud/keycloak/auth, the L4 VIPs — everything else resolves publicly to Cloudflare, since 2026-11-09) on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
|
||||
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the authoritative LAN resolver** for `${BASE_DOMAIN}` on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
|
||||
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
|
||||
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled |
|
||||
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |
|
||||
|
||||
@@ -176,8 +176,8 @@ or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:
|
||||
|
||||
**UI access:** `https://unifi.home.keithjackson.dev:11443`. That name resolves to the VIP and not to
|
||||
the Gateway on purpose: both `platform/coredns/coredns-custom.yaml` (in-cluster) and
|
||||
`apps/wave-0/pihole.yaml` (LAN split-horizon, an explicit name list since 2026-11-09) point
|
||||
`unifi.${BASE_DOMAIN}` at `${UNIFI_L4_VIP}` rather than `${GATEWAY_VIP}`. `${BASE_DOMAIN}` on prod is the
|
||||
`apps/wave-0/pihole.yaml` (LAN split-horizon) carve `unifi.${BASE_DOMAIN}` out of the
|
||||
zone-wide `${GATEWAY_VIP}` answer and point it at `${UNIFI_L4_VIP}`. `${BASE_DOMAIN}` on prod is the
|
||||
apex `home.keithjackson.dev` (confirmed live in the `cluster-vars` ConfigMap) after the 2026-09-13
|
||||
flip, so this hostname is current, not a dev leftover.
|
||||
|
||||
|
||||
@@ -83,15 +83,25 @@
|
||||
# and would also fix the same latent defect in unifi and crafty; it belongs in its own change.
|
||||
#
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# loadBalancerSourceRanges — THE LAN RESTRICTION, ENFORCED IN THE DATAPATH.
|
||||
# loadBalancerSourceRanges — THE LAN + VPN RESTRICTION, ENFORCED IN THE DATAPATH.
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# Cilium enforces this list in eBPF on the ingress node, on the VIP, BEFORE load-balancing and
|
||||
# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} is dropped by the
|
||||
# datapath — it never reaches a backend and never reaches a network policy. This is what makes
|
||||
# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} and ${WG_CLIENT_CIDR} is
|
||||
# dropped by the datapath — it never reaches a backend and never reaches a network policy. This is what makes
|
||||
# "LAN clients only" a real, tested restriction rather than a comment: VERIFIED by narrowing the
|
||||
# range to a /24 that excludes the test client, after which the same query that had been answering
|
||||
# stopped being answered at all, and restoring it, after which it answered again.
|
||||
#
|
||||
# ${WG_CLIENT_CIDR} IS LISTED BECAUSE VPN CLIENTS REACH THIS CHECK UNTRANSLATED. wg0 is one of
|
||||
# Cilium's kube-proxy-replacement devices (measured 2026-11-08: `cilium-dbg status` lists eno1, wg0,
|
||||
# tun0), so a tunnel client's query is load-balanced as it enters from wg0, ahead of the iptables
|
||||
# MASQUERADE in platform/vpn-ingress/daemonset.yaml, and the source tested here is still 10.90.0.x
|
||||
# rather than the node's address. Without this entry a full-tunnel client's DNS is predicted to die
|
||||
# with DROP_NOT_IN_SRC_RANGE (derived from Cilium's source; NOT observed live, because no peer has
|
||||
# ever completed a handshake). The range is private and reachable only by a key in peers.conf, so
|
||||
# PiHole stays off the internet; conformance/tests/l0-static/pihole-dns-source-ranges fails the gate
|
||||
# if either entry goes missing or any range here stops being RFC1918.
|
||||
#
|
||||
# It is also the layer that survives the SNAT above. A CiliumNetworkPolicy cannot express this
|
||||
# restriction once the source has been rewritten: measured on this cluster, the forwarded query
|
||||
# arrives from the ingress node's internal router address and Cilium resolves that source to the
|
||||
@@ -113,6 +123,7 @@ spec:
|
||||
externalTrafficPolicy: Cluster
|
||||
loadBalancerSourceRanges:
|
||||
- ${LAN_CIDR}
|
||||
- ${WG_CLIENT_CIDR}
|
||||
selector:
|
||||
app.kubernetes.io/name: pihole
|
||||
ports:
|
||||
@@ -133,6 +144,7 @@ spec:
|
||||
externalTrafficPolicy: Cluster
|
||||
loadBalancerSourceRanges:
|
||||
- ${LAN_CIDR}
|
||||
- ${WG_CLIENT_CIDR}
|
||||
selector:
|
||||
app.kubernetes.io/name: pihole
|
||||
ports:
|
||||
@@ -158,7 +170,9 @@ spec:
|
||||
# "Policy denied ... identity world" drops in `cilium-dbg monitor`). So `world` is the only peer
|
||||
# selector that can describe LAN traffic that has been through the LoadBalancer, and the set of
|
||||
# addresses that can actually BE that traffic is fixed by loadBalancerSourceRanges before this
|
||||
# policy is consulted. Together: ${LAN_CIDR} in the datapath, udp/53 + tcp/53 here.
|
||||
# policy is consulted. Together: ${LAN_CIDR} + ${WG_CLIENT_CIDR} in the datapath, udp/53 + tcp/53
|
||||
# here. A VPN client delivered to a backend on its own ingress node keeps its 10.90.0.x source,
|
||||
# which Cilium also classifies as `world`, so this rule needs no new peer for it.
|
||||
#
|
||||
# This is NOT wider than the precedent it follows. unifi-allow-l4-ingress and crafty-allow-ingress
|
||||
# admit `fromEntities: [world, cluster, host]` on their L4 ports with NO source-range filter at all;
|
||||
@@ -187,7 +201,7 @@ spec:
|
||||
app.kubernetes.io/name: pihole
|
||||
ingress:
|
||||
- fromEntities:
|
||||
- world # LAN clients, post-LoadBalancer: see the block above for why not fromCIDR
|
||||
- world # LAN + VPN clients, post-LoadBalancer: see the block above for why not fromCIDR
|
||||
- host # this node's kubelet probes
|
||||
- remote-node # the node-to-node hop that forwards a query to wherever the pod runs
|
||||
toPorts:
|
||||
|
||||
@@ -73,7 +73,7 @@ spec:
|
||||
- name: FTLCONF_dns_rateLimit_count
|
||||
value: '0'
|
||||
- name: FTLCONF_misc_dnsmasq_lines
|
||||
value: local-ttl=60;address=/gitea.${BASE_DOMAIN}/${GATEWAY_VIP};address=/nextcloud.${BASE_DOMAIN}/${GATEWAY_VIP};address=/keycloak.${BASE_DOMAIN}/${GATEWAY_VIP};address=/auth.${BASE_DOMAIN}/${GATEWAY_VIP};address=/headlamp.${BASE_DOMAIN}/${GATEWAY_VIP};address=/jackett.${BASE_DOMAIN}/${GATEWAY_VIP};address=/pihole.${BASE_DOMAIN}/${GATEWAY_VIP};address=/plex.${BASE_DOMAIN}/${GATEWAY_VIP};address=/qbittorrent.${BASE_DOMAIN}/${GATEWAY_VIP};address=/radarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/semaphore.${BASE_DOMAIN}/${GATEWAY_VIP};address=/sonarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
|
||||
value: local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
|
||||
- name: FTLCONF_webserver_api_password
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
|
||||
+31
-58
@@ -85,7 +85,7 @@ env:
|
||||
# Default is LOCAL, which answers only clients on the same subnet as the listening interface. The
|
||||
# pod's interface is on the CLUSTER pod CIDR, so every LAN client would be refused. ALL is
|
||||
# required for a LoadBalancer-fronted resolver; what keeps PiHole from being an OPEN resolver is
|
||||
# the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}]` on the DNS Services,
|
||||
# the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}, ${WG_CLIENT_CIDR}]` on the DNS Services,
|
||||
# enforced by Cilium in the datapath before load-balancing, plus a CiliumNetworkPolicy that admits
|
||||
# nothing but udp/53 and tcp/53. See apps/wave-0/pihole-dns.yaml.
|
||||
- name: FTLCONF_dns_listeningMode
|
||||
@@ -108,61 +108,41 @@ env:
|
||||
# `loadBalancer.mode: dsr`, which preserves the client address.
|
||||
- name: FTLCONF_dns_rateLimit_count
|
||||
value: "0"
|
||||
# THE LAN ANSWERS — AN EXPLICIT LIST, NOT A ZONE WILDCARD (2026-11-09). A dnsmasq
|
||||
# `address=/<domain>/<ip>` entry answers that domain AND every subdomain of it (on LABEL
|
||||
# boundaries: `pihole.` does not match `pihole-dns.`), and dnsmasq resolves overlapping entries by
|
||||
# LONGEST MATCHING DOMAIN, so the order below is irrelevant. Every name NOT listed here is
|
||||
# FORWARDED to the upstreams above and gets the PUBLIC answer.
|
||||
# THE SPLIT-HORIZON ANSWERS. A dnsmasq `address=/<domain>/<ip>` entry answers that domain AND
|
||||
# every subdomain of it, and dnsmasq resolves overlapping entries by LONGEST MATCHING DOMAIN —
|
||||
# so the first line is a wildcard for the whole zone and each later line is a specific override.
|
||||
# PRECEDENCE IS THEREFORE STRUCTURAL, not order-dependent: `unifi.${BASE_DOMAIN}` matches both
|
||||
# line 1 and line 2, line 2 is the longer domain, line 2 wins. Rewriting these in any order
|
||||
# changes nothing.
|
||||
#
|
||||
# WHY NO WILDCARD ANY MORE. Until 2026-11-09 the first entry was `address=/${BASE_DOMAIN}/
|
||||
# ${GATEWAY_VIP}`, answering the whole zone. That made sense while every public record was a
|
||||
# DNS-only CNAME to `wan.${BASE_DOMAIN}` (no NAT hairpin on this router, so the LAN NEEDED a
|
||||
# private answer). Since PR #156 every public name is a PROXIED CNAME to the Cloudflare Tunnel
|
||||
# (platform/externaldns/README.md), which works identically from the LAN — so a name that is
|
||||
# public now SHOULD resolve publicly from the LAN too, and the LAN no longer depends on PiHole for
|
||||
# it: a client that bypasses PiHole (DoH browser, phone private DNS, the router's fallback) gets
|
||||
# the same working answer. The wildcard had also become actively misleading: it answered names
|
||||
# that exist nowhere (`wan.` itself resolved to the Gateway VIP on the LAN).
|
||||
# A NEWLY ONBOARDED SERVICE NEEDS NO CHANGE HERE. Every catalog ingressRoute host is
|
||||
# `<label>.${BASE_DOMAIN}` served by the shared Gateway, so the wildcard already answers it with
|
||||
# ${GATEWAY_VIP} — the same guarantee, and the same reasoning, as the `template IN A` block in
|
||||
# platform/coredns/coredns-custom.yaml. Add a line here ONLY for a service that does not live
|
||||
# behind the Gateway: an L4 LoadBalancer on its own pinned VIP. Those are enumerated from the
|
||||
# repo, not from memory — grep `external-dns.alpha.kubernetes.io/hostname` on a Service, which
|
||||
# is the annotation that means "this Service publishes its own hostname at its own address":
|
||||
# apps/vms/unifi/50-service-l4.yaml (unifi -> ${UNIFI_L4_VIP}) and
|
||||
# platform/crafty/service-l4.yaml (crafty-mc -> ${CRAFTY_L4_VIP}). Those two, plus PiHole's own
|
||||
# resolver address, are the complete set of exceptions today.
|
||||
#
|
||||
# WHAT IS LISTED, and why each — enforced by conformance/tests/l0-static/pihole-lan-names:
|
||||
# -> ${GATEWAY_VIP}, LAN-ONLY: every HTTPRoute disowned from public DNS
|
||||
# (`external-dns.alpha.kubernetes.io/controller: none`, i.e. catalog `exposure: internal`,
|
||||
# plus plex and headlamp) — jackett, pihole, plex, qbittorrent, radarr, semaphore, sonarr,
|
||||
# headlamp. These have NO public record and the tunnel 404s them, so THIS LINE IS THEIR ONLY
|
||||
# PATH: drop one and the app is unreachable everywhere. A NEWLY ONBOARDED `internal` SERVICE
|
||||
# MUST BE ADDED HERE (the L0 check fails until it is). A new `public` service needs nothing.
|
||||
# -> ${GATEWAY_VIP}, PUBLIC BUT KEPT ON THE LAN PATH: gitea (git push, LFS, container registry)
|
||||
# and nextcloud (uploads) — Cloudflare's proxy caps a request body at 100 MB and a response
|
||||
# at 100 s, which a large push or upload exceeds; keycloak and auth (oauth2-proxy) — so the
|
||||
# OIDC login of those two LAN-path apps stays on the same path as the apps themselves.
|
||||
# -> their own L4 VIPs: unifi (${UNIFI_L4_VIP}) and crafty-mc (${CRAFTY_L4_VIP}) are not behind
|
||||
# the Gateway (grep `external-dns.alpha.kubernetes.io/hostname` on a Service:
|
||||
# apps/vms/unifi/50-service-l4.yaml, platform/crafty/service-l4.yaml); their public records
|
||||
# are DNS-only A records to the same RFC1918 VIPs, which this router's rebind protection
|
||||
# strips, so the LAN answer is still required. pihole-dns (${PIHOLE_DNS_VIP}) is PiHole's own
|
||||
# resolver address; plex-vip (${PLEX_L4_VIP}) is below.
|
||||
# NOT listed, deliberately, so they resolve via Cloudflare from the LAN as from anywhere: the apex,
|
||||
# grafana, crafty, seerr. In-cluster (platform/coredns/coredns-custom.yaml) and node
|
||||
# (/etc/hosts pins, node_baseline_hosts_pins) resolution are separate and unchanged.
|
||||
#
|
||||
# AAAA: dnsmasq answers a AAAA query for an `address=`d name with NODATA (NOERROR, no answer),
|
||||
# so IPv6-capable clients fall back to the A record instead of getting NXDOMAIN — the same
|
||||
# behaviour coredns-custom hand-writes a `template IN AAAA` block for. Unlisted (proxied) names
|
||||
# now get Cloudflare's real A and AAAA answers.
|
||||
# AAAA needs nothing: dnsmasq answers a AAAA query for an `address=`d name with NODATA
|
||||
# (NOERROR, no answer), so IPv6-capable clients fall back to the A record instead of getting
|
||||
# NXDOMAIN — the same behaviour coredns-custom hand-writes a `template IN AAAA` block for.
|
||||
#
|
||||
# Array-valued FTLCONF variables are SEMICOLON-separated. A JSON-looking list is accepted and
|
||||
# then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal
|
||||
# characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid"
|
||||
# while resolution for the zone does not work. Confirmed both ways against the live image.
|
||||
#
|
||||
# plex-vip -> ${PLEX_L4_VIP} (added 2026-10-04) is the name of the tcp/32400 LoadBalancer
|
||||
# (apps/wave-0/plex-remote.yaml) and the domain of the certificate Plex serves there
|
||||
# (catalog/plex.yaml CERTDOMAIN / customCertificateDomain, Secret `plex-vip-tls`). It is NOT what
|
||||
# Plex advertises any more (ADVERTISE_IP is a resolver-independent plex.direct name since
|
||||
# 2026-10-05) and it has NO public record, so this entry is its only resolution — kept for
|
||||
# reaching the VIP by name with a matching certificate from a LAN shell. Measured 2026-10-04,
|
||||
# before it existed the zone wildcard answered it with ${GATEWAY_VIP}, where nothing listens on
|
||||
# 32400; without it now, the name is NXDOMAIN.
|
||||
# plex-vip -> ${PLEX_L4_VIP} (added 2026-10-04) is the LAN HALF of Plex's single advertised
|
||||
# hostname. Plex publishes ONE URL to plex.tv for both local and remote clients
|
||||
# (catalog/plex.yaml ADVERTISE_IP): on the LAN this answer points it at the tcp/32400
|
||||
# LoadBalancer, while Cloudflare answers the same name publicly with the WAN address, NATed to
|
||||
# the same VIP. WITHOUT THIS LINE the entry above answers it with ${GATEWAY_VIP} via the zone
|
||||
# wildcard, where NOTHING listens on 32400 — measured 2026-10-04, plex-vip.${BASE_DOMAIN}
|
||||
# resolved to the Gateway VIP and a connect to :32400 there returned http=000. That is a silent
|
||||
# failure: the Plex web UI on plex.${BASE_DOMAIN} keeps working through the Gateway on 443 while
|
||||
# every client that takes the ADVERTISED connection cannot reach the server at all.
|
||||
#
|
||||
# ── local-ttl=60 — THE SPLIT-HORIZON ANSWER MUST BE CACHEABLE ────────────────────────────────
|
||||
# MEASURED 2026-10-18, diagnosing "DNS flapping between the cluster ingress and the public IP":
|
||||
@@ -192,17 +172,10 @@ env:
|
||||
# cache expires, the router falls back, and the public answer latches again. That is a real
|
||||
# outage, and the fix for it is PiHole availability, not DNS caching.
|
||||
#
|
||||
# SINCE 2026-11-09 THE LATCH IS MOSTLY HARMLESS. The measurement above predates the tunnel: the
|
||||
# public answer was then `wan.`, unreachable from the LAN. Now the public answer for every
|
||||
# proxied name is Cloudflare, which works from the LAN, and those names are not listed below at
|
||||
# all. A latch still bites only the listed names: gitea/nextcloud/keycloak/auth fall back to
|
||||
# Cloudflare (working, but under its 100 MB / 100 s limits); the LAN-only names and the L4 VIPs
|
||||
# get NXDOMAIN or a stripped answer until the router re-queries PiHole.
|
||||
#
|
||||
# Applies to every `address=` reply below (dnsmasq: local-ttl governs replies made from
|
||||
# configuration, /etc/hosts and DHCP leases), so one directive covers every listed name.
|
||||
# configuration, /etc/hosts and DHCP leases), so one directive covers all five names.
|
||||
- name: FTLCONF_misc_dnsmasq_lines
|
||||
value: "local-ttl=60;address=/gitea.${BASE_DOMAIN}/${GATEWAY_VIP};address=/nextcloud.${BASE_DOMAIN}/${GATEWAY_VIP};address=/keycloak.${BASE_DOMAIN}/${GATEWAY_VIP};address=/auth.${BASE_DOMAIN}/${GATEWAY_VIP};address=/headlamp.${BASE_DOMAIN}/${GATEWAY_VIP};address=/jackett.${BASE_DOMAIN}/${GATEWAY_VIP};address=/pihole.${BASE_DOMAIN}/${GATEWAY_VIP};address=/plex.${BASE_DOMAIN}/${GATEWAY_VIP};address=/qbittorrent.${BASE_DOMAIN}/${GATEWAY_VIP};address=/radarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/semaphore.${BASE_DOMAIN}/${GATEWAY_VIP};address=/sonarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
|
||||
value: "local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
|
||||
# THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
|
||||
# apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
|
||||
# carries the full rationale and the measurement that found it.
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — PiHole's DNS frontends admit the LAN and the VPN, and nothing public
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# loadBalancerSourceRanges is the only thing keeping PiHole from being an open resolver, and it
|
||||
# must also carry the WireGuard client subnet, because Cilium load-balances tunnel traffic on wg0
|
||||
# before the iptables MASQUERADE (so the source it tests is still 10.90.0.x). check.sh's header
|
||||
# states the assertions and what they do not prove.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library + PyYAML);
|
||||
# conformance/run.sh --layer l0 runs it directly and this Chainsaw Test wraps the SAME script.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: pihole-dns-source-ranges
|
||||
labels:
|
||||
conformance.home.k3s/R-DNS-03: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-DNS-03"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE: ci/touched-requirements.sh maps apps/* to no R-DNS requirement, so a change
|
||||
# to these Services would otherwise select nothing.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Every PiHole LoadBalancer Service lists both the LAN and the WireGuard client subnet in
|
||||
loadBalancerSourceRanges, and every range resolves to RFC1918, so VPN clients can resolve
|
||||
and PiHole stays unreachable from the internet.
|
||||
spec:
|
||||
steps:
|
||||
- name: assert-pihole-source-ranges
|
||||
description: >-
|
||||
Collect the PiHole LoadBalancer Services, resolve each source range through cluster-vars,
|
||||
and assert both required ranges are present and every range is private.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,146 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — pihole-dns-source-ranges
|
||||
# Serves: R-DNS-03 (LAN split-horizon DNS — spec docs/09).
|
||||
#
|
||||
# THE BUG CLASS THIS CLOSES. PiHole's LoadBalancer frontends are the only thing stopping it being
|
||||
# an open resolver, and their `loadBalancerSourceRanges` is pulled in two opposite directions:
|
||||
# * Too WIDE, and the resolver is open: a public range, or 0.0.0.0/0, in this list is the whole
|
||||
# restriction gone, with nothing else in the path to catch it (the CiliumNetworkPolicy admits
|
||||
# `world` on :53, because that is the only peer that can describe post-LoadBalancer traffic).
|
||||
# * Too NARROW, and VPN DNS dies: wg0 is a Cilium kube-proxy-replacement device, so a tunnel
|
||||
# client's query is load-balanced with its 10.90.0.x source intact, ahead of the iptables
|
||||
# MASQUERADE. A LAN-only list is predicted to drop it as DROP_NOT_IN_SRC_RANGE, and the
|
||||
# tunnel would still report a healthy handshake.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every type=LoadBalancer Service in namespace `pihole` is collected and each source range
|
||||
# is resolved through cluster-vars (base value AND the prod overlay's replacement),
|
||||
# THEN (1) the frontends exist and cover both udp/53 and tcp/53;
|
||||
# (2) every frontend lists ${LAN_CIDR} and ${WG_CLIENT_CIDR};
|
||||
# (3) every range resolves to a valid IPv4 CIDR inside RFC1918 — so no 0.0.0.0/0, no public
|
||||
# range, and no unresolvable token that could hide one;
|
||||
# AND (4) the check is NON-VACUOUS: a real checkout and a floor on the assertion count.
|
||||
#
|
||||
# WHAT THIS DOES NOT PROVE: that a query over the tunnel is actually answered. That needs a
|
||||
# handshake from a real client; see platform/vpn-ingress/README.md.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
import ipaddress, re, subprocess, sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
REQUIRED_TOKENS = ("LAN_CIDR", "WG_CLIENT_CIDR")
|
||||
RFC1918 = [ipaddress.ip_network(n) for n in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16")]
|
||||
TOKEN = re.compile(r"^\$\{([A-Z0-9_]+)\}$")
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def docs_of(rel: str):
|
||||
try:
|
||||
return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace"))
|
||||
if isinstance(d, dict)]
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
|
||||
FILES = [p for p in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
|
||||
capture_output=True, text=True, check=True).stdout.split("\0")
|
||||
if p]
|
||||
YAML_FILES = [f for f in FILES if re.search(r"\.ya?ml$", f)]
|
||||
check(len(YAML_FILES) > 50, f"only {len(YAML_FILES)} YAML files seen — not a real checkout")
|
||||
|
||||
# ------------------------------------------------- every value a ${TOKEN} can take on this tree
|
||||
values: dict[str, set[str]] = {}
|
||||
for d in docs_of("clusters/base/cluster-vars.yaml"):
|
||||
if d.get("kind") == "ConfigMap":
|
||||
for k, v in (d.get("data") or {}).items():
|
||||
values.setdefault(k, set()).add(str(v))
|
||||
for d in docs_of("clusters/prod/kustomization.yaml"):
|
||||
for p in d.get("patches") or []:
|
||||
ops = p.get("patch") if isinstance(p, dict) else None
|
||||
ops = yaml.safe_load(ops) if isinstance(ops, str) else ops
|
||||
for op in ops if isinstance(ops, list) else []:
|
||||
m = re.fullmatch(r"/data/([A-Z0-9_]+)", str(op.get("path", "")))
|
||||
if m and op.get("op") in ("add", "replace"):
|
||||
values.setdefault(m.group(1), set()).add(str(op.get("value")))
|
||||
for tok in REQUIRED_TOKENS:
|
||||
check(values.get(tok), f"clusters/base/cluster-vars.yaml defines no {tok} (R-DNS-03)")
|
||||
|
||||
|
||||
def assert_private(where: str, cidr: str):
|
||||
try:
|
||||
net = ipaddress.ip_network(cidr, strict=False)
|
||||
except ValueError:
|
||||
check(False, f"{where}: {cidr!r} is not a valid CIDR (R-DNS-03)")
|
||||
return
|
||||
check(net.version == 4 and any(net.subnet_of(r) for r in RFC1918),
|
||||
f"{where}: {cidr} is not inside RFC1918, so it opens PiHole beyond the LAN and the VPN "
|
||||
"(R-DNS-03)")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------- the frontends
|
||||
frontends = []
|
||||
for rel in YAML_FILES:
|
||||
for d in docs_of(rel):
|
||||
md, spec = d.get("metadata") or {}, d.get("spec") or {}
|
||||
if d.get("kind") == "Service" and md.get("namespace") == "pihole" \
|
||||
and spec.get("type") == "LoadBalancer":
|
||||
frontends.append((rel, md.get("name"), spec))
|
||||
|
||||
protocols = {(p.get("protocol", "TCP"), p.get("port"))
|
||||
for _, _, spec in frontends for p in spec.get("ports") or []}
|
||||
for want in (("UDP", 53), ("TCP", 53)):
|
||||
check(want in protocols, f"no PiHole LoadBalancer Service serves {want[0]}/{want[1]} — this "
|
||||
"check's corpus has moved and must be re-derived (R-DNS-03)")
|
||||
|
||||
for rel, name, spec in frontends:
|
||||
where = f"{rel}: Service/{name}"
|
||||
ranges = spec.get("loadBalancerSourceRanges") or []
|
||||
check(ranges, f"{where} has no loadBalancerSourceRanges, so it answers anyone who can reach "
|
||||
"the VIP (R-DNS-03)")
|
||||
tokens = {m.group(1) for m in (TOKEN.match(str(r)) for r in ranges) if m}
|
||||
for tok in REQUIRED_TOKENS:
|
||||
check(tok in tokens, f"{where} does not list ${{{tok}}} in loadBalancerSourceRanges "
|
||||
"(R-DNS-03)")
|
||||
for r in ranges:
|
||||
m = TOKEN.match(str(r))
|
||||
if not m:
|
||||
assert_private(where, str(r))
|
||||
continue
|
||||
resolved = values.get(m.group(1))
|
||||
check(resolved, f"{where}: {r} is not defined in cluster-vars, so its value cannot be "
|
||||
"shown to be private (R-DNS-03)")
|
||||
for v in sorted(resolved or []):
|
||||
assert_private(f"{where} ({r})", v)
|
||||
|
||||
check(CHECKS >= 15, f"only {CHECKS} assertions ran — this check has gone vacuous (R-DNS-03)")
|
||||
|
||||
if FAILURES:
|
||||
for f in FAILURES:
|
||||
print(f"check FAIL: {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(f"check OK: pihole-dns-source-ranges — {CHECKS} assertions over "
|
||||
f"{len(frontends)} PiHole LoadBalancer Service(s)")
|
||||
PY
|
||||
@@ -1,45 +0,0 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — PiHole answers an explicit LAN name list, and no LAN-only name is left out of it
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# Since PR #156 every public name is a proxied CNAME to the Cloudflare Tunnel and works from the
|
||||
# LAN, so PiHole's zone wildcard was replaced by an explicit list (catalog/pihole.yaml). That makes
|
||||
# the list load-bearing: a route disowned from public DNS and absent from it resolves nowhere. This
|
||||
# test derives the required names from the tree (every `controller: none` HTTPRoute, every hostname
|
||||
# the tunnel 404s, the L4 Services' own VIPs) and asserts the list covers them, carries no
|
||||
# zone-wide entry, and pins gitea and nextcloud to the Gateway VIP. It proves itself non-vacuous by
|
||||
# mutation. Rationale and the honest NOT-proven list: check.sh.
|
||||
#
|
||||
# WHY SCOPE, NOT REQUIREMENTS: the property spans catalog/, apps/, platform/ and runbooks/; a new
|
||||
# internal service in any of them can break it, and ci/touched-requirements.sh maps apps/* to no
|
||||
# requirement at all, so a requirement-scoped selection would skip it on precisely those changes.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: pihole-lan-names
|
||||
labels:
|
||||
conformance.home.k3s/R-DNS-03: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-DNS-03"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "true"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
PiHole answers an explicit list of LAN names rather than the whole zone; every HTTPRoute
|
||||
disowned from public DNS has an entry to the Gateway VIP; every hostname the Cloudflare
|
||||
Tunnel 404s has an entry, with the L4 names on their own VIPs; and gitea and nextcloud stay
|
||||
on the LAN path, off Cloudflare's request-size and timeout limits.
|
||||
spec:
|
||||
steps:
|
||||
- name: assert-pihole-explicit-lan-names
|
||||
description: >-
|
||||
Parse FTLCONF_misc_dnsmasq_lines from catalog/pihole.yaml and the rendered
|
||||
apps/wave-0/pihole.yaml, derive the required names from every `controller: none` HTTPRoute,
|
||||
the http_status:404 hostnames in platform/cloudflared/config.yaml and the L4 Services'
|
||||
lbipam annotations, assert the list covers them and contains no zone-wide entry, and run
|
||||
the evaluator on mutated lists to prove it is not vacuous.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -1,269 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — pihole-lan-names
|
||||
# Serves: R-DNS-03 (LAN split-horizon answers), and decisions/2026-11-09-pihole-explicit-lan-names.md.
|
||||
#
|
||||
# WHY THIS EXISTS. Since PR #156 every PUBLIC name is a proxied CNAME to the Cloudflare Tunnel and
|
||||
# works from the LAN, so PiHole no longer answers the whole zone with a wildcard: it answers an
|
||||
# EXPLICIT list (catalog/pihole.yaml, FTLCONF_misc_dnsmasq_lines) and forwards everything else to
|
||||
# the public resolvers. That inverts the old failure mode. With the wildcard, a forgotten name still
|
||||
# resolved; now a forgotten LAN-ONLY name — a route disowned from public DNS, which the tunnel also
|
||||
# 404s — resolves NOWHERE, and the app is silently unreachable from every client. Nothing in the
|
||||
# cluster turns red when that happens. So the list is derived from the tree and asserted here.
|
||||
#
|
||||
# GIVEN the repository as authored,
|
||||
# THEN (0) the dnsmasq value parses (only `local-ttl=` and `address=/<domain>/<${VAR}>` entries,
|
||||
# every VAR is a cluster-var), the catalog value and the rendered apps/wave-0/pihole.yaml
|
||||
# value are identical, and the inventories below are non-empty (anti-vacuous-pass);
|
||||
# (a) NO entry covers the whole zone: no `address=/${BASE_DOMAIN}/`, no `address=/#/`, and no
|
||||
# entry for any parent of the zone (dnsmasq matches every subdomain of an entry);
|
||||
# (b) every HTTPRoute hostname disowned from public DNS
|
||||
# (`external-dns.alpha.kubernetes.io/controller: none`, anywhere in apps/ platform/
|
||||
# runbooks/) has an explicit entry to ${GATEWAY_VIP} — a disowned route with no
|
||||
# `hostnames:` is itself a FAIL, because it cannot be checked;
|
||||
# (c) every hostname platform/cloudflared/config.yaml maps to http_status:404 has an entry;
|
||||
# the L4 DNS-only names (derived: Services carrying both
|
||||
# `external-dns.alpha.kubernetes.io/hostname` and `lbipam.cilium.io/ips` — unifi,
|
||||
# crafty-mc) must map to THAT Service's VIP, not the Gateway;
|
||||
# (d) gitea and nextcloud are pinned to ${GATEWAY_VIP} (Cloudflare's 100 MB body / 100 s
|
||||
# limits break git push/LFS and large uploads);
|
||||
# AND (e) the evaluator is not vacuous: it is run against mutated copies of the real entry list
|
||||
# (zone wildcard added; a disowned host dropped; a 404'd host dropped; gitea repointed;
|
||||
# unifi repointed to the Gateway) and must flag each one.
|
||||
#
|
||||
# NOT proven (static): that the running PiHole pods carry this value (FTL applies FTLCONF_* at
|
||||
# start; the env change rolls the StatefulSet), that public resolution of the unlisted names
|
||||
# reaches Cloudflare, or the substituted IPs. Runtime check: `dig @${PIHOLE_DNS_VIP} <name>`.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
exec python3 - "$ROOT" <<'PY'
|
||||
import copy
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("CHECK FAIL (pihole-lan-names): PyYAML is required", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES, CHECKS = [], 0
|
||||
NAME = "pihole-lan-names"
|
||||
ZONE = "${BASE_DOMAIN}"
|
||||
GW = "${GATEWAY_VIP}"
|
||||
DISOWN = "external-dns.alpha.kubernetes.io/controller"
|
||||
EDNS_HOST = "external-dns.alpha.kubernetes.io/hostname"
|
||||
LB_IPS = "lbipam.cilium.io/ips"
|
||||
# Public, but must bypass Cloudflare's 100 MB request body / 100 s response limits.
|
||||
LAN_PINNED = ("gitea", "nextcloud")
|
||||
# Trees that hold applied manifests. conformance/ fixtures and .claude/ worktrees are not authored
|
||||
# surface (CLAUDE.md §0); decisions/ is prose.
|
||||
SCAN = ("apps", "platform", "runbooks")
|
||||
|
||||
|
||||
def check(cond, desc):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(desc)
|
||||
|
||||
|
||||
def docs(path):
|
||||
try:
|
||||
return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)]
|
||||
except yaml.YAMLError:
|
||||
return []
|
||||
|
||||
|
||||
def dnsmasq_value(objs):
|
||||
"""The FTLCONF_misc_dnsmasq_lines value from a list of docs (a ServiceDefinition or manifests)."""
|
||||
found = []
|
||||
for d in objs:
|
||||
envs = list(d.get("env") or [])
|
||||
tmpl = (((d.get("spec") or {}).get("template") or {}).get("spec") or {})
|
||||
for c in tmpl.get("containers") or []:
|
||||
envs += c.get("env") or []
|
||||
found += [e.get("value") for e in envs if e.get("name") == "FTLCONF_misc_dnsmasq_lines"]
|
||||
return found
|
||||
|
||||
|
||||
def die(msg):
|
||||
print(f"CHECK FAIL ({NAME}): {msg}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
CATALOG = ROOT / "catalog" / "pihole.yaml"
|
||||
RENDERED = ROOT / "apps" / "wave-0" / "pihole.yaml"
|
||||
CF = ROOT / "platform" / "cloudflared" / "config.yaml"
|
||||
for p in (CATALOG, RENDERED, CF):
|
||||
if not p.is_file():
|
||||
die(f"{p.relative_to(ROOT)} is absent — the surface this check guards moved; re-point or "
|
||||
"delete this check in the SAME change")
|
||||
|
||||
cat_vals = dnsmasq_value(docs(CATALOG))
|
||||
ren_vals = dnsmasq_value(docs(RENDERED))
|
||||
check(len(cat_vals) == 1, f"(0) catalog/pihole.yaml declares FTLCONF_misc_dnsmasq_lines exactly "
|
||||
f"once (found {len(cat_vals)})")
|
||||
check(len(ren_vals) == 1, f"(0) apps/wave-0/pihole.yaml carries FTLCONF_misc_dnsmasq_lines exactly "
|
||||
f"once (found {len(ren_vals)})")
|
||||
if not cat_vals:
|
||||
die("no dnsmasq value to check")
|
||||
check(cat_vals[:1] == ren_vals[:1],
|
||||
"(0) the rendered apps/wave-0/pihole.yaml dnsmasq value differs from catalog/pihole.yaml — "
|
||||
"re-render pihole (catalog/render.py --no-image-automation catalog/pihole.yaml)")
|
||||
|
||||
cv = next(d for d in docs(ROOT / "clusters" / "base" / "cluster-vars.yaml")
|
||||
if d.get("kind") == "ConfigMap")
|
||||
CLUSTER_VARS = set(cv["data"])
|
||||
BASE_DOMAIN_VALUE = cv["data"]["BASE_DOMAIN"]
|
||||
|
||||
ENTRY = re.compile(r"^address=/([^/]+)/(\$\{[A-Z0-9_]+\})$")
|
||||
|
||||
|
||||
def parse(value):
|
||||
"""Return (entries {domain: target}, problems)."""
|
||||
entries, problems = {}, []
|
||||
for raw in value.split(";"):
|
||||
line = raw.strip()
|
||||
if raw != line:
|
||||
problems.append(f"entry {raw!r} has surrounding whitespace (FTL writes it verbatim)")
|
||||
if not line:
|
||||
problems.append("empty entry (doubled or trailing `;`)")
|
||||
continue
|
||||
if re.fullmatch(r"local-ttl=\d+", line):
|
||||
continue
|
||||
m = ENTRY.match(line)
|
||||
if not m:
|
||||
problems.append(f"entry {line!r} is not `local-ttl=N` or `address=/<domain>/${{VAR}}` — "
|
||||
f"this check cannot reason about it")
|
||||
continue
|
||||
dom, tgt = m.groups()
|
||||
if tgt[2:-1] not in CLUSTER_VARS:
|
||||
problems.append(f"{dom} -> {tgt}: not a key of clusters/base/cluster-vars.yaml")
|
||||
if dom in entries and entries[dom] != tgt:
|
||||
problems.append(f"{dom} is listed twice with different targets")
|
||||
entries[dom] = tgt
|
||||
return entries, problems
|
||||
|
||||
|
||||
entries, problems = parse(cat_vals[0])
|
||||
for pr in problems:
|
||||
check(False, f"(0) {pr}")
|
||||
|
||||
# ── inventories, derived from the tree ──────────────────────────────────────────────────────────
|
||||
disowned, unnamed = set(), []
|
||||
l4 = {}
|
||||
for top in SCAN:
|
||||
for f in sorted((ROOT / top).rglob("*.yaml")):
|
||||
for d in docs(f):
|
||||
meta = d.get("metadata") or {}
|
||||
ann = meta.get("annotations") or {}
|
||||
if d.get("kind") == "HTTPRoute" and str(ann.get(DISOWN, "")).strip() == "none":
|
||||
hosts = (d.get("spec") or {}).get("hostnames") or []
|
||||
if not hosts:
|
||||
unnamed.append(f"{f.relative_to(ROOT)}:{meta.get('name')}")
|
||||
disowned.update(hosts)
|
||||
if d.get("kind") == "Service" and EDNS_HOST in ann and LB_IPS in ann:
|
||||
for h in str(ann[EDNS_HOST]).split(","):
|
||||
l4[h.strip()] = str(ann[LB_IPS]).strip()
|
||||
for u in unnamed:
|
||||
check(False, f"(b) disowned HTTPRoute {u} declares no `hostnames:` — it inherits a listener "
|
||||
f"hostname this check cannot see; name it explicitly")
|
||||
check(len(disowned) >= 5, f"(0) only {len(disowned)} disowned HTTPRoute hostnames found — the scan "
|
||||
f"is vacuous (the disown annotation was renamed?)")
|
||||
check(len(l4) >= 2, f"(0) only {len(l4)} L4 Services publishing their own hostname found (expected "
|
||||
f"unifi and crafty-mc) — the scan is vacuous")
|
||||
|
||||
cf = yaml.safe_load(CF.read_text())
|
||||
cf_404 = {r["hostname"] for r in cf.get("ingress") or []
|
||||
if r.get("hostname") and r.get("service") == "http_status:404"}
|
||||
check(len(cf_404) >= 5, f"(0) only {len(cf_404)} http_status:404 hostnames in the cloudflared "
|
||||
f"config — the scan is vacuous")
|
||||
|
||||
|
||||
def covers_zone(dom):
|
||||
"""True if an entry for `dom` answers every name in the zone (dom is the zone or a parent)."""
|
||||
if dom in ("#", "", "."):
|
||||
return True
|
||||
if dom == ZONE:
|
||||
return True
|
||||
# A literal parent (e.g. `keithjackson.dev`, `dev`) of the zone's substituted value.
|
||||
real = dom.replace(ZONE, BASE_DOMAIN_VALUE)
|
||||
return real == BASE_DOMAIN_VALUE or BASE_DOMAIN_VALUE.endswith("." + real)
|
||||
|
||||
|
||||
def evaluate(entries):
|
||||
bad = []
|
||||
# (a)
|
||||
for dom in entries:
|
||||
if covers_zone(dom):
|
||||
bad.append(f"(a) `address=/{dom}/` answers EVERY name in the zone — the wildcard is gone "
|
||||
f"on purpose (public names must resolve to Cloudflare from the LAN)")
|
||||
# (b)
|
||||
for h in sorted(disowned):
|
||||
if entries.get(h) != GW:
|
||||
bad.append(f"(b) {h} is disowned from public DNS but PiHole maps it to "
|
||||
f"{entries.get(h, 'NOTHING')} — expected {GW}; without it the app is "
|
||||
f"unreachable everywhere")
|
||||
# (c)
|
||||
for h in sorted(cf_404):
|
||||
if h in l4:
|
||||
if entries.get(h) != l4[h]:
|
||||
bad.append(f"(c) {h} is an L4 DNS-only name; PiHole maps it to "
|
||||
f"{entries.get(h, 'NOTHING')}, expected its own VIP {l4[h]}")
|
||||
elif h not in entries:
|
||||
bad.append(f"(c) the tunnel 404s {h} but PiHole has no entry for it — it would resolve "
|
||||
f"only publicly, and the tunnel refuses it")
|
||||
# (d)
|
||||
for n in LAN_PINNED:
|
||||
h = f"{n}.{ZONE}"
|
||||
if entries.get(h) != GW:
|
||||
bad.append(f"(d) {h} maps to {entries.get(h, 'NOTHING (public/Cloudflare)')}; it must be "
|
||||
f"pinned to {GW} — Cloudflare caps request bodies at 100 MB and responses at "
|
||||
f"100 s")
|
||||
return bad
|
||||
|
||||
|
||||
for v in evaluate(entries):
|
||||
check(False, v)
|
||||
|
||||
# (e) mutation self-test
|
||||
mutants = {}
|
||||
m = dict(entries); m[ZONE] = GW
|
||||
mutants["zone wildcard added"] = m
|
||||
m = dict(entries); m[BASE_DOMAIN_VALUE.split(".", 1)[1]] = GW
|
||||
mutants["parent-of-zone wildcard added"] = m
|
||||
m = dict(entries); m.pop(sorted(disowned)[0], None)
|
||||
mutants[f"disowned host {sorted(disowned)[0]} dropped"] = m
|
||||
pure_404 = sorted(h for h in cf_404 if h not in l4 and h not in disowned)
|
||||
if pure_404:
|
||||
m = dict(entries); m.pop(pure_404[0], None)
|
||||
mutants[f"404'd host {pure_404[0]} dropped"] = m
|
||||
m = dict(entries); m.pop(f"gitea.{ZONE}", None)
|
||||
mutants["gitea unpinned"] = m
|
||||
m = dict(entries); m[f"nextcloud.{ZONE}"] = "${UNIFI_L4_VIP}"
|
||||
mutants["nextcloud repointed"] = m
|
||||
if l4:
|
||||
h = sorted(l4)[0]
|
||||
m = dict(entries); m[h] = GW
|
||||
mutants[f"{h} repointed to the Gateway"] = m
|
||||
for name, mut in mutants.items():
|
||||
check(evaluate(mut), f"(e) self-test: the evaluator did NOT flag the mutant '{name}' — vacuous")
|
||||
check(len(mutants) >= 5, f"(e) only {len(mutants)} mutants built — the self-test degenerated")
|
||||
_, pr = parse("local-ttl=60;address=/x.${BASE_DOMAIN}/${NOT_A_CLUSTER_VAR}; address=/y/${GATEWAY_VIP}")
|
||||
check(len(pr) >= 2, "(e) self-test: the parser accepted an unknown VAR or a whitespace-padded entry")
|
||||
|
||||
if CHECKS < 15:
|
||||
FAILURES.append(f"only {CHECKS} assertions ran — the check degenerated")
|
||||
if FAILURES:
|
||||
print(f"CHECK FAIL ({NAME}): {len(FAILURES)} of {CHECKS} assertions failed", file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(f"PASS ({NAME}): {CHECKS} assertions; {len(entries)} PiHole names, {len(disowned)} disowned "
|
||||
f"routes, {len(cf_404)} tunnel-404 names, {len(l4)} L4 names")
|
||||
PY
|
||||
@@ -0,0 +1,103 @@
|
||||
---
|
||||
date: 2026-11-08
|
||||
branch: fix/wg-home-peer-pihole-sources
|
||||
title: Enrol keith-phone-home and admit the WireGuard subnet to PiHole's DNS frontends
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
- **`platform/vpn-ingress/peers.conf`**: new peer `keith-phone-home`, `10.90.0.3/32`, key generated
|
||||
on the phone. It is the `home` half of a two-tunnel profile (Endpoint `192.168.2.10:51820`),
|
||||
because the router has no hairpin NAT and `wan.` is unusable from inside the house.
|
||||
- **`apps/wave-0/pihole-dns.yaml`**: `${WG_CLIENT_CIDR}` added after `${LAN_CIDR}` on both
|
||||
`pihole-dns-udp` and `pihole-dns-tcp`. This file is hand-written, not rendered (`catalog/render.py`
|
||||
only lists it in `EXTRA_UNITS`), so it is the source of truth and `--emit-waves` was not run.
|
||||
`catalog/pihole.yaml` changes in one comment only, so nothing renders differently.
|
||||
- **New L0 guard `conformance/tests/l0-static/pihole-dns-source-ranges`** (whole-repo scope,
|
||||
R-DNS-03): every PiHole LoadBalancer Service lists both tokens, and every range resolves through
|
||||
cluster-vars (base and prod overlay) to RFC1918.
|
||||
- **`platform/vpn-ingress/README.md`**: corrected "clients arrive SNATed" (false for LoadBalancer
|
||||
VIPs) and the IPv6 claim (Android's `GoBackend` blocks IPv6 for a single-peer default route);
|
||||
documented the home/away pattern, why `wan.` fails from inside, and the prod-1 dependency.
|
||||
|
||||
Rollout on merge: the `wg-peers` ConfigMap hash changes (`m9kcckcb5t` -> `29mf6k867t`), so the
|
||||
DaemonSet rolls, `maxUnavailable: 1`, `maxSurge: 0`, one node at a time in an order the controller
|
||||
chooses. Each roll deletes and recreates that node's `wg0`, so an established session on that node
|
||||
drops and must re-handshake. `keith-phone` has no session today (see below), so nothing is
|
||||
interrupted in practice.
|
||||
|
||||
**The one risk worth watching:** `wg0` is in Cilium's auto-detected device list and `devices` is
|
||||
still unpinned. On 2026-10-06 a crash-looping `wg0` churned that list and all three nodes lost their
|
||||
main-table default route. A rollout is one delete/create per node, not a loop, so it should not
|
||||
repeat that, but after merge check `ip route get 1.1.1.1` on each node.
|
||||
|
||||
The Service change is an in-place update: same VIP, same Service objects, no pod restart.
|
||||
|
||||
Checked on the rest of the path and left unchanged:
|
||||
|
||||
- Gateway VIP `.20` (`gateway/cilium-gateway-shared`): no source ranges, live. Nothing needed.
|
||||
- `pihole-allow-lan-dns-ingress` admits `world` on :53, which covers an untranslated 10.90.0.x
|
||||
source. `platform/vpn-ingress/networkpolicy.yaml` does not apply to a hostNetwork pod.
|
||||
- No L0 check pinned these ranges to the LAN; none needed updating.
|
||||
|
||||
Flagged, not changed (out of scope):
|
||||
|
||||
- `runbooks/vpn-remote-access.md` "Known limitations" repeats both corrected claims, and step 3 of
|
||||
its verification table says the Gateway sees a `host`-identity source.
|
||||
- `platform/vpn-ingress/daemonset.yaml`'s header repeats the SNAT claim.
|
||||
- `clusters/base/cluster-vars.yaml`'s substitution-coverage comment does not list
|
||||
`WG_CLIENT_CIDR` under apps-wave-0. The variable is defined and substituted there regardless.
|
||||
- `peers.conf` still says "No peers enrolled yet" above the enrolled peers.
|
||||
- PiHole's NodePorts are not covered by source ranges (already acknowledged in `pihole-dns.yaml`).
|
||||
Tunnel clients can reach node addresses, so they can reach those too; they are enrolled devices.
|
||||
- Pinning Cilium's `devices` remains the open hardening for the churn risk above.
|
||||
|
||||
## What I verified
|
||||
|
||||
Live, read-only, 2026-11-08:
|
||||
|
||||
- `cilium-dbg status` on one agent: `KubeProxyReplacement: True [eno1 ..., wg0 10.90.0.1, tun0 ...]`
|
||||
and `Masquerading: BPF [eno1, wg0, tun0]`. So `wg0` really is a load-balancing device.
|
||||
- `pihole-dns-tcp` and `-udp` carry `ranges=[192.168.2.0/24]`; no other LoadBalancer Service has
|
||||
source ranges. Live `cluster-vars`: `WG_CLIENT_CIDR=10.90.0.0/24`, `LAN_CIDR=192.168.2.0/24`.
|
||||
- `wg show wg0 latest-handshakes` on all three pods: `keith-phone` = `0`.
|
||||
- `wan.home.keithjackson.dev` resolves publicly to a single A record (via 1.1.1.1).
|
||||
|
||||
Offline, in the pinned toolbox:
|
||||
|
||||
- `flux envsubst --strict` renders both Services to `[192.168.2.0/24, 10.90.0.0/24]`.
|
||||
- `kubectl kustomize platform/vpn-ingress`: `wg-peers-m9kcckcb5t` on the old `peers.conf`,
|
||||
`wg-peers-29mf6k867t` on the new one.
|
||||
- Guard mutation test, each restored afterwards. Baseline passes (22 assertions, 2 Services). All
|
||||
seven fail with rc 1 and a named message: WG range dropped from one Service; `0.0.0.0/0` added;
|
||||
`8.8.8.0/24` added; `WG_CLIENT_CIDR` made public in cluster-vars; prod overlay sets `LAN_CIDR` to
|
||||
`0.0.0.0/0`; ranges block removed; undefined token added.
|
||||
- The Android claim was read from upstream `GoBackend.java` (`wireguard-android`, master): the
|
||||
`allowFamily` calls are skipped when `sawDefaultRoute && config.getPeers().size() == 1`.
|
||||
- Gate (pinned toolbox, from this worktree, on the committed tree): **rc 3, UNDETERMINED, not
|
||||
green.** `gitea-settings` is the only stage counted as not run (no token locally). `conformance`
|
||||
(L2/L3) also skipped as the declared no-cluster gap. `spec-immutability`, `decisions-log`, `lint`,
|
||||
`l0` (including the new guard), `render-golden` and `l1` passed. `approval-policy` reported N/A
|
||||
offline; it will be red in CI until the operator overrides, because `conformance/**` is sensitive.
|
||||
- `git merge-tree` against `feat/pihole-explicit-lan-names` (#157): no conflicts.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **`DROP_NOT_IN_SRC_RANGE` was NOT observed live.** It is a prediction from Cilium's source. Only
|
||||
the `wg0` device membership was measured; the drop itself has never been seen.
|
||||
- **Full-tunnel behaviour was NOT observed live**, on either profile. **No handshake has ever been
|
||||
recorded for any peer**: `keith-phone` reads `0` on all three pods (a counter that resets when a
|
||||
pod restarts, last 22 days ago, so it covers only the time since; earlier entries here also
|
||||
record none).
|
||||
- That tunnel DNS works after this change, including the reply path from a backend on another node
|
||||
and from one on the ingress node. That needs the phone: `dig` over the tunnel, while watching
|
||||
`cilium-dbg monitor --type drop`.
|
||||
- That the Gateway serves tunnel clients. Its source is predicted to be `world`, as for a LAN
|
||||
browser, which is what the apps admit; not tested.
|
||||
- That the new peer's key is the one on the phone. I copied the public key from the brief.
|
||||
- That the rollout leaves node default routes intact. Reasoned from the 2026-10-06 incident being
|
||||
a loop, not measured.
|
||||
- That a LAN client is unaffected while the Service's source ranges are updated. Expected, not
|
||||
measured.
|
||||
- IPv6 behaviour on any non-Android client.
|
||||
- That the WAN forward targets prod-1: it is off-repo.
|
||||
@@ -1,97 +0,0 @@
|
||||
---
|
||||
date: 2026-11-09
|
||||
branch: feat/pihole-explicit-lan-names
|
||||
title: PiHole answers an explicit LAN name list instead of a zone wildcard (public names resolve to Cloudflare from the LAN)
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
**`catalog/pihole.yaml` → `apps/wave-0/pihole.yaml`.** `FTLCONF_misc_dnsmasq_lines` no longer carries
|
||||
`address=/${BASE_DOMAIN}/${GATEWAY_VIP}`. Every name not listed is now forwarded to 8.8.8.8/8.8.4.4 and
|
||||
gets the public answer. The final list (`local-ttl=60` kept):
|
||||
|
||||
| Name (`.${BASE_DOMAIN}`) | Answer | Why |
|
||||
|---|---|---|
|
||||
| jackett, pihole, plex, qbittorrent, radarr, semaphore, sonarr, headlamp | `${GATEWAY_VIP}` | LAN-only: every HTTPRoute with `external-dns.alpha.kubernetes.io/controller: none`, derived from the tree. No public record, tunnel 404s them; this entry is their only path |
|
||||
| gitea, nextcloud | `${GATEWAY_VIP}` | public, but Cloudflare's proxy caps bodies at 100 MB / responses at 100 s (git push, LFS, uploads) |
|
||||
| keycloak, auth | `${GATEWAY_VIP}` | keep the OIDC flow of the LAN-path apps on one path |
|
||||
| unifi | `${UNIFI_L4_VIP}` | L4 VIP, not behind the Gateway |
|
||||
| crafty-mc | `${CRAFTY_L4_VIP}` | L4 VIP, not behind the Gateway |
|
||||
| pihole-dns | `${PIHOLE_DNS_VIP}` | the resolver's own address |
|
||||
| plex-vip | `${PLEX_L4_VIP}` | kept, see below |
|
||||
|
||||
Not listed, so they resolve to Cloudflare from the LAN: the apex, grafana, crafty, seerr.
|
||||
|
||||
**The derived LAN-only set matches the brief's list exactly** (8 names; the tree was walked, not the
|
||||
list trusted). The `runbooks/serving` route has no `hostnames:` but is not disowned (it inherits the
|
||||
apex, public).
|
||||
|
||||
**plex-vip is kept.** Nothing advertises it any more (`ADVERTISE_IP` is a `plex.direct` name since
|
||||
2026-10-05), but it is not unused: it is Plex's `customCertificateDomain` (`CERTDOMAIN`, Secret
|
||||
`plex-vip-tls`), the name of the tcp/32400 VIP, and the 404'd name in the tunnel config. It has no
|
||||
public record, so this entry is its only resolution.
|
||||
|
||||
**New L0 check `conformance/tests/l0-static/pihole-lan-names`** (`scope: whole-repo`, so picked up by
|
||||
the runner's scope selection with no other wiring). Asserts: (a) no zone-wide entry (zone, `#`, or any
|
||||
parent of the zone); (b) every `controller: none` hostname → `${GATEWAY_VIP}`, and a disowned route
|
||||
with no `hostnames:` fails; (c) every tunnel-404 hostname has an entry, with the L4 names (derived
|
||||
from Services carrying both the external-dns hostname and an lbipam VIP) on their own VIP; (d) gitea
|
||||
and nextcloud → `${GATEWAY_VIP}`; plus catalog == rendered value and a built-in mutation self-test.
|
||||
|
||||
**Stale docs corrected:** `platform/pm-agent/README.md`, `runbooks/pm-agent.md`,
|
||||
`platform/externaldns/README.md`, `platform/cloudflared/README.md`, `platform/gateway/gateway.yaml`
|
||||
(comment), `platform/headlamp/httproute.yaml` (comment), `apps/vms/unifi/README.md`, `README.md`,
|
||||
`schemas/renderer-contract.md`; `runbooks/cloudflare-tunnel.md` gains the two consequences below.
|
||||
`platform/cloudflared/deployment.yaml`: the image entrypoint comment was wrong (comment-only).
|
||||
|
||||
## What I verified
|
||||
|
||||
- **Render:** the baseline `render.py --no-image-automation catalog/pihole.yaml` was byte-identical to
|
||||
the committed `apps/wave-0/pihole.yaml` before the change, and the file was replaced by that
|
||||
command's output after it; `--emit-waves` was not run.
|
||||
- **Mutation, against the real files (10 mutants, each rc 1 with the right assertion, each restored):**
|
||||
(a) zone wildcard restored; (b) headlamp and sonarr entries removed; (b) a disowned route with no
|
||||
`hostnames:`; (c) a new tunnel-404 host with no entry; (c) pihole-dns removed; (c) unifi pointed at
|
||||
the Gateway; (d) gitea removed; (d) nextcloud repointed; (0) rendered value drifted from the catalog.
|
||||
- **Rollout mechanics, measured live (read-only):** StatefulSet `updateStrategy` RollingUpdate
|
||||
partition 0, `podManagementPolicy: Parallel`, no `minReadySeconds`, 2 replicas, PDB minAvailable 1.
|
||||
The last rollout (2026-11-08 06:47–06:48Z) replaced pihole-1 then pihole-0: FTL logged "started" ~1 s
|
||||
after the container, the pod went Ready ~16 s after the container, and pihole-0 started ~9 s later.
|
||||
- **Expected LAN DNS blip:** no total outage; one replica answers throughout. Each of the two pod
|
||||
replacements can lose a few UDP queries for ~1–2 s while the terminating pod leaves the Service
|
||||
(clients retry). For roughly 30–60 s the two pods answer differently for the apex, grafana, crafty
|
||||
and seerr; both answers work. Cached answers age out in 60 s (`local-ttl`).
|
||||
- **Live DNS before the change:** @8.8.8.8 returns Cloudflare (104.21.51.91 / 172.67.178.27) for the
|
||||
apex, grafana, seerr, crafty and gitea, and nothing for jackett; @192.168.2.23 returns 192.168.2.20
|
||||
for all of them (the wildcard).
|
||||
- **Cloudflared image:** `skopeo inspect --config` on the pinned digest (amd64): Entrypoint
|
||||
`["cloudflared"]`, Cmd `["version"]`, User `65532:65532`.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **The post-merge behaviour.** Nothing here ran against a live PiHole with the new list; the
|
||||
expected answers are from dnsmasq semantics and the tree. After reconcile, check
|
||||
`dig @192.168.2.23` for one name per row, and grafana returning Cloudflare.
|
||||
- **Whether every LAN client path works via Cloudflare.** Measured 2xx/3xx through Cloudflare by the
|
||||
brief, not re-measured from a LAN client here.
|
||||
- **The FTL DNS-listening moment.** "FTL started" is a log line, not a successful query; the rollout
|
||||
timing above assumes DNS is up shortly after it.
|
||||
|
||||
**Accepted consequences, stated plainly:**
|
||||
- The apex, grafana, crafty and seerr now depend on the internet and the tunnel from the LAN too. An
|
||||
ISP or Cloudflare outage, or stopping the connectors, makes them unreachable from the LAN where they
|
||||
previously kept working. If grafana in particular should survive an internet outage, add it to the
|
||||
list (one entry; the check permits it).
|
||||
- The tunnel rollback (revert to `wan.`) now also requires restoring the wildcard; recorded in
|
||||
`runbooks/cloudflare-tunnel.md`.
|
||||
|
||||
**Flagged, not changed:**
|
||||
- The `pihole` container has no readiness probe; only the `oauth2-proxy` sidecar does, so "Ready" says
|
||||
nothing about DNS. The measured margin (FTL ~1 s, Ready ~16 s) makes this benign in a normal rollout
|
||||
but a slow FTL start would not hold the rollout back. A DNS readiness probe needs a schema/renderer
|
||||
decision.
|
||||
|
||||
**Stale statements left alone:** `platform/coredns/coredns-custom.yaml` header ("both resolve the whole
|
||||
zone ... by wildcard", brief said not to touch CoreDNS); `clusters/base/cluster-vars.yaml` comments
|
||||
(sensitive path); `schemas/renderer-contract.md` §`exposure` first bullets (manual edge port-forward
|
||||
wording); historical `infra/PROD-PROVISIONING.md`, `platform/externaldns/README.md` TODO list.
|
||||
@@ -17,8 +17,6 @@ outbound-only from the cluster (no router port forward, no hairpin, no split-DNS
|
||||
| `networkpolicy.yaml` | default-deny (single owner — not an app namespace) + Prometheus scrape allow + egress CNP: world 7844 tcp/udp and 443, the Gateway (`ingress` entity), and Gateway backends by `gateway-access=shared` minus the LAN-only namespaces. |
|
||||
| `servicemonitor.yaml`, `alerts.yaml` | Headless metrics Service + ServiceMonitor scraping :2000, and the two alerts, both written to fire (not go quiet) when their series vanish. |
|
||||
|
||||
Cloudflare sends a hostname through the tunnel only when its DNS record is a proxied CNAME to
|
||||
`<UUID>.cfargotunnel.com`; nothing in this directory creates one. Those records are published by
|
||||
external-dns from the shared Gateway's target annotation (Phase 2, PR #156; see
|
||||
`platform/externaldns/README.md`). LAN resolution: `catalog/pihole.yaml` (explicit names only since
|
||||
2026-11-09), so public names reach Cloudflare from the LAN too.
|
||||
**Phase 1 only.** This component changes no live traffic: Cloudflare sends a hostname through the
|
||||
tunnel only when its DNS record is a CNAME to `<UUID>.cfargotunnel.com`, and nothing here creates
|
||||
one. Moving DNS (external-dns, Gateway annotations, Pi-hole/CoreDNS split horizon) is a later change.
|
||||
@@ -70,9 +70,8 @@ spec:
|
||||
- name: cloudflared
|
||||
# Tag + digest: Renovate's `kubernetes` manager (renovate.json5, platform/**) bumps both.
|
||||
image: docker.io/cloudflare/cloudflared:2026.10.7@sha256:4955524aeaa379e154023d5a027c250f7b35041a2e7e950cf13f144bc9b9c5ba
|
||||
# The image ENTRYPOINT is `cloudflared` and its CMD is `version` (read from the pinned
|
||||
# digest's image config, 2026-11-09); these args replace that CMD. `no-autoupdate`, the
|
||||
# tunnel UUID, credentials path, metrics address and ingress rules all come from the file.
|
||||
# The image ENTRYPOINT is `cloudflared --no-autoupdate`; these replace its CMD. The tunnel
|
||||
# UUID, credentials path, metrics address and ingress rules all come from the file.
|
||||
args:
|
||||
- tunnel
|
||||
- --config
|
||||
|
||||
@@ -44,13 +44,8 @@ wan.home.keithjackson.dev A (DNS-only) <current WAN IP> <- cloudflare-ddn
|
||||
- `internal` routes (`controller: none`) still publish nothing. Plex's web UI and Headlamp joined that
|
||||
set on 2026-11-08 (LAN/VPN-only); the tunnel also 404s every such name (9c/9d).
|
||||
- Public answers are Cloudflare anycast **A and AAAA** — Cloudflare's edge, not an origin address.
|
||||
- **Pods** resolve the whole zone to the Gateway VIP via CoreDNS (split-horizon) and never use the
|
||||
public path. **LAN clients** use PiHole, which since 2026-11-09 answers an EXPLICIT list
|
||||
(`catalog/pihole.yaml`): the LAN-only names, plus gitea/nextcloud/keycloak/auth (kept off
|
||||
Cloudflare's 100 MB / 100 s limits and on one OIDC path), go to the Gateway VIP and the L4 names to
|
||||
their own VIPs; every other public name (apex, grafana, crafty, seerr) resolves to Cloudflare from
|
||||
the LAN exactly as from anywhere else, so no split-DNS is needed for them. Guard:
|
||||
`conformance/tests/l0-static/pihole-lan-names`.
|
||||
- LAN and pods resolve `*.home.keithjackson.dev` to the Gateway VIP via PiHole and CoreDNS
|
||||
(split-horizon) and never use the public path.
|
||||
- **Nodes resolve through public DNS** (`node_baseline_dns_servers`), so from a node the apiserver's
|
||||
OIDC issuer `keycloak.home.keithjackson.dev` now resolves to Cloudflare and its discovery/JWKS
|
||||
fetches go through the tunnel (publicly trusted edge cert; no `oidc-ca-file` is configured). The
|
||||
|
||||
@@ -30,10 +30,8 @@ metadata:
|
||||
# applies BEFORE it reads this target. Those same names map to `http_status:404` in the tunnel's
|
||||
# ingress rules, so even a hand-made record cannot reach them through the tunnel.
|
||||
#
|
||||
# In-cluster clients are unaffected: CoreDNS answers ${BASE_DOMAIN} with ${GATEWAY_VIP} directly
|
||||
# (split-horizon). LAN clients use PiHole's EXPLICIT list (catalog/pihole.yaml, since 2026-11-09):
|
||||
# LAN-only names and gitea/nextcloud/keycloak/auth get ${GATEWAY_VIP}; every other published
|
||||
# name takes this public path through Cloudflare, from the LAN as from anywhere.
|
||||
# LAN and in-cluster clients are unaffected: PiHole and CoreDNS answer ${BASE_DOMAIN} with
|
||||
# ${GATEWAY_VIP} directly (split-horizon), so they never take this public path.
|
||||
#
|
||||
# `wan.${BASE_DOMAIN}` (platform/externaldns/ddns.yaml) is NO LONGER a target of anything here;
|
||||
# it stays only as the WireGuard udp/51820 endpoint.
|
||||
|
||||
@@ -20,7 +20,7 @@ metadata:
|
||||
homelab.keithjackson.dev/auth-groups: platform-admins
|
||||
# LAN/VPN-ONLY (operator decision 2026-11-08). Disowned from external-dns, so the public
|
||||
# headlamp record and its TXT are DELETED and nothing republishes them; LAN/VPN clients keep
|
||||
# resolving the name via PiHole's explicit entry for it (-> the Gateway VIP). The Cloudflare
|
||||
# resolving the name via PiHole's split-horizon wildcard (-> the Gateway VIP). The Cloudflare
|
||||
# Tunnel also maps headlamp.${BASE_DOMAIN} to http_status:404 (platform/cloudflared/config.yaml).
|
||||
# Same mechanism the renderer emits for `exposure: internal` services; asserted by
|
||||
# conformance/tests/l0-static/internal-services-publish-no-public-dns (9c).
|
||||
|
||||
@@ -72,14 +72,11 @@ Both are deliberate, and both fail **visibly** rather than silently:
|
||||
- **The nodes resolve the registry hostname to the WAN, so the TLS they fail to verify is
|
||||
the router's — DIAGNOSED 2026-10-06, and fixed in this change.** containerd pulls as a
|
||||
*node* process, so it resolves through the node's public upstreams
|
||||
(`node_baseline_dns_servers` = the LAN gateway) rather than through Pi-hole. At the time,
|
||||
the public answer for every name in this zone was a CNAME to `wan.home.keithjackson.dev`
|
||||
— the WAN IP — which this router does not hairpin; it answered :443 with its own placeholder
|
||||
(`node_baseline_dns_servers` = the LAN gateway) rather than through Pi-hole. The public
|
||||
answer for every name in this zone is a CNAME to `wan.home.keithjackson.dev` — the WAN
|
||||
IP — which this router does not hairpin; it answers :443 with its own placeholder
|
||||
certificate `CN=%D` carrying **no extensions, therefore no SANs**, which is precisely
|
||||
`x509: certificate is not valid for any names`. (Since 2026-11-08 the public answer is
|
||||
Cloudflare's edge via the Tunnel, not the WAN IP — see `platform/externaldns/README.md`.
|
||||
The pin is still the right path: it keeps registry pulls on the LAN and off Cloudflare's
|
||||
proxy.) The Gateway, the wildcard certificate
|
||||
`x509: certificate is not valid for any names`. The Gateway, the wildcard certificate
|
||||
and the digest were never implicated. The fix is the pin this repository already uses
|
||||
for the Keycloak issuer, for the identical reason: `gitea.{{ k3s_base_domain }}` is now
|
||||
in `node_baseline_hosts_pins` — node configuration is Ansible's, declared in the
|
||||
|
||||
@@ -42,8 +42,9 @@ no version string to match against a CVE, and no amplification to abuse.
|
||||
|
||||
That is the whole security argument, and it is why the exposed port carries no IP allowlist (which
|
||||
would also have made roaming clients unusable — a phone's address changes with every cell tower).
|
||||
Compare `apps/wave-0/pihole-dns.yaml`, which *does* pin `loadBalancerSourceRanges` to the LAN: a
|
||||
resolver answers anybody, so it must be scoped by address. WireGuard answers nobody.
|
||||
Compare `apps/wave-0/pihole-dns.yaml`, which *does* pin `loadBalancerSourceRanges` to the LAN and
|
||||
this tunnel's client subnet: a resolver answers anybody, so it must be scoped by address. WireGuard
|
||||
answers nobody.
|
||||
|
||||
What it does **not** protect against is a stolen device — anyone holding a device's private key
|
||||
*is* that device. That is what revocation is for: delete the stanza, merge, done (~1 min).
|
||||
@@ -55,9 +56,9 @@ What it does **not** protect against is a stolen device — anyone holding a dev
|
||||
┌──────────────┐ udp/51820 ┌───────────────┐ forward ┌────────────────────────────┐
|
||||
│ WireGuard app│─────────────▶│ WAN 51820 ──▶ │───────────▶│ wg0 10.90.0.1/24 │
|
||||
│ 10.90.0.2/32 │ over the │ node IP:51820 │ │ decrypt, then FORWARD │
|
||||
└──────────────┘ internet └───────────────┘ │ SNAT to the node's addr │
|
||||
└──────────────┘ internet └───────────────┘ │ SNAT (LAN), eBPF LB (VIPs)│
|
||||
└─────────────┬──────────────┘
|
||||
│ as `host`/`remote-node`
|
||||
│ translated per destination (below)
|
||||
┌─────────────────────────────────┼──────────────────────┐
|
||||
▼ ▼ ▼
|
||||
LAN 192.168.2.0/24 Gateway VIP .20 (16 web UIs) WAN / internet
|
||||
@@ -86,9 +87,21 @@ resolved, because `apps/wave-0/pihole-dns.yaml`'s ingress policy deliberately om
|
||||
entity. A VPN that looks up and is useless is the worst available outcome, and nothing in the
|
||||
manifests would have hinted at it.
|
||||
|
||||
In the **host** netns there is no pod identity to carry. Forwarded client traffic is SNATed to the
|
||||
node's own address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` —
|
||||
byte-for-byte what a LAN client looks like. Three further problems vanish with it:
|
||||
In the **host** netns there is no pod identity to carry, which is what avoids the 403s. Clients do
|
||||
**not**, however, all arrive SNATed to the node. Measured 2026-11-08: Cilium counts `wg0` among its
|
||||
kube-proxy-replacement devices (`cilium-dbg status`: `eno1`, `wg0`, `tun0`), so what a packet looks
|
||||
like depends on where it is going:
|
||||
|
||||
| destination | handled by | source the destination sees |
|
||||
|---|---|---|
|
||||
| LAN hosts, the internet | kernel FORWARD, then the iptables MASQUERADE in [`daemonset.yaml`](./daemonset.yaml) | the node's address |
|
||||
| LoadBalancer VIPs (Gateway `.20`, PiHole `.23`, ...) | Cilium's eBPF load balancer, as the packet enters from `wg0`, **before** MASQUERADE | `10.90.0.x` at the VIP's source-range check, identity `world` |
|
||||
|
||||
The second row is why PiHole's `loadBalancerSourceRanges` has to list `${WG_CLIENT_CIDR}` as well as
|
||||
the LAN: left LAN-only, tunnel DNS is predicted to die with `DROP_NOT_IN_SRC_RANGE`. That prediction
|
||||
and the row's source column come from reading Cilium's datapath source; **no peer has ever completed
|
||||
a handshake, so neither has been observed live.** The Gateway VIP carries no source ranges and needs
|
||||
no change. Three further problems vanish with the host netns:
|
||||
|
||||
* `net.ipv4.ip_forward` is **already 1** on these nodes (`node_baseline` writes
|
||||
`/etc/sysctl.d/99-k3s-cilium.conf`). It is *not* in the kubelet's safe-sysctl allowlist, so a
|
||||
@@ -163,6 +176,30 @@ internet path. To enforce less *server-side*, add filtering to the `PostUp` chai
|
||||
before the accept rules) — the FORWARD chain there is the only enforcement point, since no
|
||||
NetworkPolicy applies.
|
||||
|
||||
## Phone profiles: one tunnel for home, one for away
|
||||
|
||||
A phone that needs the LAN-only apps runs full tunnel (`AllowedIPs = 0.0.0.0/0`, `DNS = 192.168.2.23`)
|
||||
everywhere, so it needs **two tunnels**, each its own peer with its own key and address:
|
||||
|
||||
| tunnel | peer | address | Endpoint | use when |
|
||||
|---|---|---|---|---|
|
||||
| `away` | `keith-phone` | `10.90.0.2` | `wan.home.keithjackson.dev:51820` | off the home network |
|
||||
| `home` | `keith-phone-home` | `10.90.0.3` | `192.168.2.10:51820` | on the home Wi-Fi |
|
||||
|
||||
**`wan.` cannot work from inside the house.** It is an unproxied A record holding the house's WAN
|
||||
address (`platform/externaldns/README.md`), and the router has no hairpin NAT, so a LAN client that
|
||||
sends to it never comes back in through the port-forward. The Cloudflare Tunnel is no way round: it
|
||||
carries HTTP(S) to the Gateway and WireGuard is UDP to a node, so `wan.` has to stay DNS-only and
|
||||
`away` has to keep using it.
|
||||
|
||||
**`home` depends on `k3s-prod-1`.** Its Endpoint is that node's address, with no failover: if
|
||||
prod-1 is down, `home` cannot handshake even though the other two listeners are healthy. The fix
|
||||
is client-side: edit the tunnel's Endpoint to `192.168.2.11` or `.12`, since every node answers the
|
||||
same keys. No server change and no new key is needed. The same dependency applies to `away` if the
|
||||
WAN forward points at prod-1.
|
||||
|
||||
Android allows one active VPN at a time, so switch tunnels by hand when leaving or arriving.
|
||||
|
||||
## Operating it
|
||||
|
||||
* **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
|
||||
@@ -215,10 +252,13 @@ right way; one was a defect in my own manifest.
|
||||
handshake — and would be a real secret, forcing `peers.conf` into SOPS and costing the
|
||||
reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
|
||||
`wg-server-key` Secret.
|
||||
* **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6
|
||||
still reaches IPv6 sites *outside* the tunnel. Functionally harmless, a privacy leak if the point
|
||||
was to hide the client's network. Fix is an `fd..` ULA on `wg0` plus `::/0` in the client's
|
||||
`AllowedIPs`; not done because nothing here needs it.
|
||||
* **No IPv6.** The server offers no IPv6 tunnel address and `new-peer.sh` emits only `0.0.0.0/0`.
|
||||
What happens to a client's IPv6 then depends on the client. The **Android app does not leak it**:
|
||||
its `GoBackend` skips `allowFamily()` when the config is one peer with a default route, so Android
|
||||
blocks IPv6 for the VPN's duration and apps fall back to IPv4. Other clients are not covered by
|
||||
that code; expect `wg-quick` on a dual-stack Linux host to leave IPv6 outside the tunnel. Fix is
|
||||
an `fd..` ULA on `wg0` plus `::/0` in the client's `AllowedIPs`; not done because nothing here
|
||||
needs it.
|
||||
* **No in-cluster-service reach beyond what the LAN has.** Clients reach cluster apps the way a LAN
|
||||
browser does — through the Gateway VIP. Pod IPs and `10.43.0.0/16` service IPs are not routed
|
||||
into the tunnel.
|
||||
@@ -90,3 +90,7 @@
|
||||
[Peer]
|
||||
PublicKey = pBRSHrRyG2hnsuZ6FvHfEzF8hOVTF5stHUb/7fW2Km8=
|
||||
AllowedIPs = 10.90.0.2/32
|
||||
# keith-phone-home — <owner>, enrolled 2026-11-08 (at-home profile, LAN Endpoint; no hairpin NAT)
|
||||
[Peer]
|
||||
PublicKey = AT8lBJHqrVkv8RL9n0qWDsh5nyl0e3qgrfTAaYKgBz0=
|
||||
AllowedIPs = 10.90.0.3/32
|
||||
@@ -353,15 +353,11 @@ is deleted, which is why the last step is not optional.
|
||||
'flux --context k3s-prod suspend kustomization cloudflared &&
|
||||
kubectl --context k3s-prod -n cloudflared scale deploy/cloudflared --replicas=0'
|
||||
```
|
||||
Cloudflare then answers tunnel hostnames with a 530/1033 error — **from the LAN too** for the
|
||||
names PiHole does not list (apex, grafana, crafty, seerr; since 2026-11-09). Follow up in git, then
|
||||
Cloudflare then answers tunnel hostnames with a 530/1033 error. Follow up in git, then
|
||||
`flux resume kustomization cloudflared`.
|
||||
- **After the DNS flip (Phase 2):** the path back is a revert PR of `feat/dns-via-cloudflare-tunnel`
|
||||
(Gateway target back to `wan.${BASE_DOMAIN}`, drop `--cloudflare-proxied`) — external-dns then
|
||||
rewrites every record DNS-only to `wan.`. That only works if the router's WAN tcp/80+443 forwards
|
||||
are **re-opened** first (they are closed after Phase 2). **Also restore PiHole's zone wildcard**
|
||||
(`address=/${BASE_DOMAIN}/${GATEWAY_VIP}` in `catalog/pihole.yaml`, and drop the
|
||||
`pihole-lan-names` L0 assertion (a)): since 2026-11-09 the apex, grafana, crafty and seerr are not
|
||||
listed there and resolve publicly, and a public `wan.` answer does not hairpin from the LAN. Never edit the records by hand:
|
||||
are **re-opened** first (they are closed after Phase 2). Never edit the records by hand:
|
||||
external-dns owns them and `policy: sync` reverts a hand edit. Revoking the tunnel credential (`cf tunnel delete`, or rotating it
|
||||
in the Zero Trust dashboard) cuts every connector at once if the credential is believed leaked.
|
||||
@@ -74,7 +74,7 @@ Work through these in order; the first three are by far the most likely.
|
||||
|---|---|---|
|
||||
| `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
|
||||
| `ImagePullBackOff`, digest `sha256:0000…` | (historical: before 2026-10-30 the committed digest was an all-zero placeholder) the image has never been published at the digest named | CI publishes it, not a person: check the latest `images` workflow run on `main` (it fails loudly if the `REGISTRY_PUSH_TOKEN` secret is missing; re-run it with `workflow_dispatch`), then merge Renovate's digest-bump PR, which updates **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) together — but read the row below first |
|
||||
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and (before the 2026-11-08 Tunnel cutover; the public answer is now Cloudflare) got the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
|
||||
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
|
||||
| `ImagePullBackOff` with `401 Unauthorized` / `unauthorized` | the pull credential is wrong, absent or scoped without `read:package`. The `pm-agent-registry` Secret exists and is referenced; its *value* is what starts as a placeholder | check the key names only, never values: `kubectl -n pm-agent get secret pm-agent-registry -o go-template='{{range $k, $v := .data}}{{$k}}{{"\n"}}{{end}}'` should list `.dockerconfigjson`. Then confirm the token carries `read:package`, that its account can read the org's packages (a team with the Packages unit — see the `.example`), and that the registry key inside the JSON is the bare host `gitea.home.keithjackson.dev` — with a scheme or a port it matches nothing and the pull goes out anonymously, which looks identical to a missing Secret. Steps: `platform/pm-agent/pm-agent-registry.sops.yaml.example` |
|
||||
| `GET /user` answers `403` / `token does not have at least one of required scope(s): [read:user]` at startup, then a CrashLoop | the token lacks `read:user` — Gitea 1.24.3 scopes the whole `/api/v1/user` group, and the dispatcher's first call is `GET /user` | mint a new token with all four scopes (`read:user`, `write:repository`, `write:issue`, `read:package`) and put it in **both** Secrets. Add `read:notification` if you want the cheap pull-request mention query (`platform/pm-agent/README.md`) |
|
||||
| a worker Job is refused by admission (`pm-agent-job-shape`, `imagePullSecrets`) | `PM_WORKER_IMAGE` is a digest built before the template carried `imagePullSecrets` (PR #61), so its baked template lacks the entry admission requires | pin the agent-runner digest CI published after #61 merged (Renovate's bump PR); until then keep `PM_DRY_RUN=true` |
|
||||
|
||||
@@ -82,10 +82,13 @@ platform must render, and cites the requirement each mapping satisfies. Changing
|
||||
all seven the moment it is set. Per-resource exclusion has exactly the blast radius of the
|
||||
defect. The filter stays available as hardening once those seven are adjudicated — see
|
||||
`platform/externaldns/helmrelease.yaml` and the register in the L0 check.
|
||||
- `public` renders the record but the renderer does **not** set a target: external-dns reads
|
||||
`external-dns.alpha.kubernetes.io/target` only from the **Gateway** (and a ListenerSet), never from
|
||||
an HTTPRoute, so the target is set once, on the `shared` Gateway — since 2026-11-08 the Cloudflare
|
||||
Tunnel, proxied (`platform/gateway/gateway.yaml`, `platform/externaldns/README.md`).
|
||||
- `public` renders the record but the renderer does **not** yet set a target: the record still
|
||||
resolves to the shared Gateway VIP, i.e. an RFC1918 address, so a `public` service is not in fact
|
||||
publicly reachable. Fixing that is **not** a renderer change on v0.21.0 —
|
||||
`source/gateway.go` reads `external-dns.alpha.kubernetes.io/target` only from the **Gateway** (and
|
||||
a ListenerSet), never from an HTTPRoute, so a per-service target override is not expressible while
|
||||
the public services share the `shared` Gateway. The open decision is recorded in
|
||||
`platform/externaldns/helmrelease.yaml`.
|
||||
- Exposure does **not** relax east-west default-deny (docs/09 §"Interaction with default-deny").
|
||||
- **Requirements:** R-DNS-02/03 (docs/09), R-NET-04 (docs/03).
|
||||
|
||||
@@ -320,9 +323,8 @@ R-OBS-01 (the SLI must follow the rendered kind).
|
||||
`public` service**. An `internal` service's route is rendered with ExternalDNS's controller
|
||||
annotation set to `none`, which removes it from ExternalDNS's view entirely (see §`exposure`), so
|
||||
its `<host>` exists **only** in the internal split-horizon answer PiHole serves, never in the
|
||||
public zone. A `public` record's TARGET is the Cloudflare Tunnel (a proxied CNAME, from the
|
||||
shared Gateway's target annotation; `platform/externaldns/README.md`), and a `public` name is
|
||||
not listed in PiHole unless it must stay on the LAN path.
|
||||
public zone. A `public` record's TARGET is still the Gateway VIP, not a routable address — open
|
||||
operator decision, see §`exposure`.
|
||||
- The `host` pattern is enforced to `<label>.home.keithjackson.dev` by the schema.
|
||||
- **Requirements:** **R-TLS-01/02/03** (docs/08), **R-DNS-01/02/03** (docs/09).
|
||||
|
||||
|
||||
Reference in new issue
Block a user