Files
home-network-k3s/decisions/adr-0016-public-ingress-cloudflare-tunnel.md
jakman85andClaude Opus 5.5 250fb54154 docs: bring the specification in line with the accepted implementation
The operator ruled the implementation repo's main (50c8719) the latest
and accepted state. Sync the spec to it.

Accepted changes are reflected: one bare-metal environment, TrueNAS bulk
storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway
with Cilium fail-closed, inbound WireGuard, per-app auth sidecars,
Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025;
ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended.

Where the implementation has simply not met a requirement, the
requirement stands and the shortfall is recorded in docs/16-known-gaps.md.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 22:46:00 -06:00

8.1 KiB

ADR-0016 — Public ingress: Cloudflare Tunnel (outbound-only) instead of edge port-forwarding

  • Status: Accepted
  • Date: 2026-11-08
  • Accepted by: operator ruling of 2026-11-08 that the implementation is the accepted state; effective on merge.
  • Serves: R-DNS-01, R-DNS-02, R-DNS-03, R-DNS-04, R-DNS-05, R-TLS-01, R-TLS-02, R-SEC-02, R-SEC-04
  • Supersedes: — (amends the ingress part of the OQ-1 resolution, "edge does NAT + port-forward only", in 00, 03 and 09. ADR-0002 (Gateway API on Cilium) and ADR-0006 (cert-manager + Let's Encrypt DNS-01) are unchanged: the Gateway still terminates origin TLS with the LE wildcard.)

Context

The OQ-1 resolution routed public traffic as: public DNS → WAN IP → edge router port-forward 80/443 → Cilium Gateway VIP. LAN clients were expected to avoid the WAN path through Pi-hole split-horizon DNS (R-DNS-03).

That design fails in this environment:

  • The ISP router (GFiber) cannot hairpin NAT. A LAN client that resolves a service to the WAN IP cannot reach it.
  • The same router advertises Google IPv6 DNS servers via Router Advertisement, and the operator cannot turn that off. LAN clients (Android in particular) therefore use a public resolver alongside, or instead of, Pi-hole. They get the WAN answer and fail.

So split-horizon DNS cannot be relied on to keep LAN clients off the WAN path. Correctness for public web apps has to come from the public answer itself working from the LAN.

Decision

Public HTTP(S) ingress goes through a Cloudflare Tunnel.

  • cloudflared runs in-cluster (a Flux-managed workload) and holds an outbound-only connection to Cloudflare. No inbound connection to the cluster is needed for public web traffic.
  • Public records for services declared public (catalog exposure: public, or a registered platform route: auth, Grafana, Crafty's web UI, the apex and its /runbooks) are proxied CNAMEs to <tunnel-uuid>.cfargotunnel.com, managed by ExternalDNS from declared service intent (R-DNS-02). No record is hand-edited.
  • The edge router's 80/443 forwards are closed after cutover.
  • Hostnames are unchanged: <service>.home.keithjackson.dev and the apex home.keithjackson.dev (R-DNS-01, R-C2-01).
  • Edge TLS is Cloudflare's: Advanced Certificate Manager / Total TLS, because Universal SSL covers only one subdomain level (*.keithjackson.dev) and not <svc>.home.keithjackson.dev. The edge certificate is publicly trusted and auto-renewed (R-TLS-01/02).
  • Origin TLS stays at the Gateway with the cert-manager Let's Encrypt wildcard (ADR-0006). cloudflared connects to the Gateway over HTTPS and validates the origin certificate, using the service hostname as SNI. It does not skip verification (noTLSVerify is not used).
  • The tunnel's ingress rules are a deny-list plus a zone wildcard. Each LAN/VPN-only hostname maps to http_status:404; the apex and *.home.keithjackson.dev then go to the Gateway; a catch-all http_status:404 closes the list for hosts outside the zone (R-DNS-05). A static conformance check keeps the deny-list and the set of unpublished routes identical in both directions.
  • The Kubernetes API is never reachable through the tunnel, and is never published by any other path. It is LAN/VPN-only (R-SEC-04).

Never published (LAN/VPN-only, exposure: internal): jackett, pihole, semaphore, qbittorrent, radarr, sonarr, headlamp (a front end to the Kubernetes API), and the non-HTTP L4 services unifi and crafty-mc (Minecraft). The tunnel refuses all of them with a 404. None has a public DNS record, except that unifi and crafty-mc hold DNS-only records at their LAN addresses, which are never proxied or tunnelled.

Plex stays off the tunnel. Cloudflare's terms restrict proxying video, so Plex remote streaming keeps using plex.tv + WAN tcp/32400 (an edge forward that stays). The plex.home web UI becomes LAN-only: refused at the tunnel, with no proxied record.

Unchanged:

  • WireGuard remote access: udp/51820, reached through the DNS-only (unproxied) wan. record, which cloudflare-ddns keeps current. The WAN IP therefore stays public. The tunnel does not hide the origin; it only removes 80/443 from the edge.
  • Pi-hole keeps serving the LAN, but split DNS is no longer required for public web apps to work. It keeps explicit local records for:
    • the internal-only names;
    • the L4 VIPs;
    • the names that must avoid Cloudflare's 100 MB request-body and 100 s response-timeout limits, or must keep working during an internet outage: gitea, nextcloud, and keycloak/auth.
  • In-cluster CoreDNS split horizon and the node /etc/hosts pins (the OIDC issuer and the Gitea registry) stay. Cluster-internal callers never go through Cloudflare.

Alternatives considered

  • Keep the port-forward and fix split horizon. Rejected: the router's RA-advertised Google IPv6 DNS cannot be disabled, so some LAN clients will always get the public answer, and the router cannot hairpin it.
  • Replace the ISP router (or put a router capable of hairpin NAT and RA control behind it). Rejected for now: it adds hardware and a non-IaC edge device, which contradicts the "dumb SOHO edge" premise of OQ-1, and the GFiber RA behaviour would still need to be contained. It can be reopened by a future ADR.
  • Block or override the Google DNS on the LAN (firewall DoH/DNS egress, or intercept port 53). Rejected: the SOHO edge has no such controls, and client-side DoH/hard-coded resolvers make it incomplete anyway.
  • Tailscale Funnel / ngrok / another reverse-tunnel provider. Rejected: Cloudflare is already the authoritative DNS and token holder (OQ-4), so its tunnel integrates with ExternalDNS and the existing scoped token with no new vendor.
  • Self-hosted reverse tunnel on a VPS (WireGuard + reverse proxy). Rejected: it adds a host to run, patch and secure, outside the cluster's GitOps model.

Consequences

  • Positive: LAN clients reach public web apps correctly whichever resolver answers. No NAT hairpin is needed. Inbound 80/443 is no longer open on the WAN IP. Publishing is an explicit, declared, per-hostname act (a public record), and LAN/VPN-only names 404 at the tunnel.
  • Dependence on Cloudflare. A Cloudflare or tunnel outage takes public access down. The LAN keeps working through Pi-hole's local records for gitea, nextcloud and keycloak/auth and the internal-only names. Public apps without a local record are unreachable from the LAN during such an outage too. Tunnel health becomes an alert condition (R-OBS-04).
  • Cloudflare sees plaintext. TLS terminates at Cloudflare's edge, so Cloudflare can read the request and response content of every tunnelled service. Re-encryption to the origin protects only the Cloudflare → cluster leg.
  • Upload and timeout limits: 100 MB request bodies and 100 s responses through the proxy. Large uploads (Nextcloud, Gitea pushes and registry blobs) must use the LAN path, which is why those names keep local records.
  • Client IP now arrives as CF-Connecting-IP / X-Forwarded-For from cloudflared, not as the TCP source. Logging, rate-limiting and any IP-based policy must read those headers, and trust them only from cloudflared.
  • The WAN IP stays public (WireGuard, Plex 32400). The tunnel is not origin cloaking.
  • TLS certificates: two publicly trusted certificates are in play per public name (Cloudflare edge
    • LE origin). Both auto-renew. ACM/Total TLS is a paid Cloudflare feature and an operator-held prerequisite.
  • Follow-ups (implementation repo): deploy cloudflared with a SOPS-encrypted tunnel credential; move ExternalDNS public records to proxied CNAMEs for public services only; add conformance tests for R-DNS-04/05 and R-SEC-04; alert on tunnel health; then close the 80/443 edge forwards (a documented manual edge step, like the original forwards).