Files
home-network-k3s/decisions/adr-0016-public-ingress-cloudflare-tunnel.md
jakman85andClaude Opus 5.5 250fb54154 docs: bring the specification in line with the accepted implementation
The operator ruled the implementation repo's main (50c8719) the latest
and accepted state. Sync the spec to it.

Accepted changes are reflected: one bare-metal environment, TrueNAS bulk
storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway
with Cilium fail-closed, inbound WireGuard, per-app auth sidecars,
Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025;
ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended.

Where the implementation has simply not met a requirement, the
requirement stands and the shortfall is recorded in docs/16-known-gaps.md.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 22:46:00 -06:00

119 lines
8.1 KiB
Markdown

# ADR-0016 — Public ingress: Cloudflare Tunnel (outbound-only) instead of edge port-forwarding
- Status: Accepted
- Date: 2026-11-08
- Accepted by: operator ruling of 2026-11-08 that the implementation is the accepted state; effective on merge.
- Serves: R-DNS-01, R-DNS-02, R-DNS-03, R-DNS-04, R-DNS-05, R-TLS-01, R-TLS-02, R-SEC-02, R-SEC-04
- Supersedes: — (amends the **ingress** part of the OQ-1 resolution, "edge does NAT + port-forward
only", in [00](../docs/00-vision-and-scope.md), [03](../docs/03-networking-isolation-and-egress.md)
and [09](../docs/09-dns-and-ingress.md). [ADR-0002](adr-0002-cni-cilium.md) (Gateway API on Cilium)
and [ADR-0006](adr-0006-tls-cert-manager-letsencrypt.md) (cert-manager + Let's Encrypt DNS-01) are
unchanged: the Gateway still terminates origin TLS with the LE wildcard.)
## Context
The OQ-1 resolution routed public traffic as: public DNS → WAN IP → **edge router port-forward
80/443** → Cilium Gateway VIP. LAN clients were expected to avoid the WAN path through **Pi-hole
split-horizon DNS** (R-DNS-03).
That design fails in this environment:
- The ISP router (GFiber) **cannot hairpin NAT**. A LAN client that resolves a service to the WAN IP
cannot reach it.
- The same router **advertises Google IPv6 DNS servers via Router Advertisement**, and the operator
cannot turn that off. LAN clients (Android in particular) therefore use a public resolver
alongside, or instead of, Pi-hole. They get the WAN answer and fail.
So split-horizon DNS cannot be *relied on* to keep LAN clients off the WAN path. Correctness for
public web apps has to come from the public answer itself working from the LAN.
## Decision
**Public HTTP(S) ingress goes through a Cloudflare Tunnel.**
- **cloudflared runs in-cluster** (a Flux-managed workload) and holds an **outbound-only**
connection to Cloudflare. No inbound connection to the cluster is needed for public web traffic.
- **Public records** for services declared public (catalog `exposure: public`, or a registered
platform route: `auth`, Grafana, Crafty's web UI, the apex and its `/runbooks`) are **proxied
CNAMEs to `<tunnel-uuid>.cfargotunnel.com`**, managed by **ExternalDNS** from declared service
intent (R-DNS-02). No record is hand-edited.
- **The edge router's 80/443 forwards are closed** after cutover.
- **Hostnames are unchanged:** `<service>.home.keithjackson.dev` and the apex
`home.keithjackson.dev` (R-DNS-01, R-C2-01).
- **Edge TLS** is Cloudflare's: **Advanced Certificate Manager / Total TLS**, because Universal SSL
covers only one subdomain level (`*.keithjackson.dev`) and not `<svc>.home.keithjackson.dev`. The
edge certificate is publicly trusted and auto-renewed (R-TLS-01/02).
- **Origin TLS stays at the Gateway** with the cert-manager Let's Encrypt wildcard (ADR-0006).
cloudflared connects to the Gateway over HTTPS and **validates the origin certificate**, using
the service hostname as SNI. It does not skip verification (`noTLSVerify` is not used).
- **The tunnel's ingress rules are a deny-list plus a zone wildcard.** Each LAN/VPN-only hostname
maps to `http_status:404`; the apex and `*.home.keithjackson.dev` then go to the Gateway; a
**catch-all `http_status:404`** closes the list for hosts outside the zone (R-DNS-05). A static
conformance check keeps the deny-list and the set of unpublished routes identical in both
directions.
- **The Kubernetes API is never reachable through the tunnel**, and is never published by any other
path. It is LAN/VPN-only (R-SEC-04).
**Never published (LAN/VPN-only, `exposure: internal`):** jackett, pihole, semaphore, qbittorrent,
radarr, sonarr, **headlamp** (a front end to the Kubernetes API), and the non-HTTP L4 services
**unifi** and **crafty-mc** (Minecraft). The tunnel refuses all of them with a 404. None has a
public DNS record, except that unifi and crafty-mc hold **DNS-only records at their LAN addresses**,
which are never proxied or tunnelled.
**Plex stays off the tunnel.** Cloudflare's terms restrict proxying video, so Plex remote streaming
keeps using **plex.tv + WAN tcp/32400** (an edge forward that stays). The `plex.home` web UI becomes
**LAN-only**: refused at the tunnel, with no proxied record.
**Unchanged:**
- **WireGuard remote access**: udp/51820, reached through the DNS-only (unproxied) `wan.` record,
which cloudflare-ddns keeps current. The WAN IP therefore **stays public**. The tunnel does not
hide the origin; it only removes 80/443 from the edge.
- **Pi-hole keeps serving the LAN**, but split DNS is no longer *required* for public web apps to
work. It keeps explicit local records for:
- the internal-only names;
- the L4 VIPs;
- the names that must avoid Cloudflare's **100 MB request-body** and **100 s response-timeout**
limits, or must keep working during an internet outage: **gitea**, **nextcloud**, and
**keycloak/auth**.
- **In-cluster CoreDNS split horizon** and the **node `/etc/hosts` pins** (the OIDC issuer and the
Gitea registry) stay. Cluster-internal callers never go through Cloudflare.
## Alternatives considered
- **Keep the port-forward and fix split horizon.** Rejected: the router's RA-advertised Google IPv6
DNS cannot be disabled, so some LAN clients will always get the public answer, and the router
cannot hairpin it.
- **Replace the ISP router (or put a router capable of hairpin NAT and RA control behind it).**
Rejected for now: it adds hardware and a non-IaC edge device, which contradicts the "dumb SOHO
edge" premise of OQ-1, and the GFiber RA behaviour would still need to be contained. It can be
reopened by a future ADR.
- **Block or override the Google DNS on the LAN** (firewall DoH/DNS egress, or intercept port 53).
Rejected: the SOHO edge has no such controls, and client-side DoH/hard-coded resolvers make it
incomplete anyway.
- **Tailscale Funnel / ngrok / another reverse-tunnel provider.** Rejected: Cloudflare is already the
authoritative DNS and token holder (OQ-4), so its tunnel integrates with ExternalDNS and the
existing scoped token with no new vendor.
- **Self-hosted reverse tunnel on a VPS** (WireGuard + reverse proxy). Rejected: it adds a host to
run, patch and secure, outside the cluster's GitOps model.
## Consequences
- **Positive:** LAN clients reach public web apps correctly whichever resolver answers. No NAT
hairpin is needed. Inbound 80/443 is no longer open on the WAN IP. Publishing is an explicit,
declared, per-hostname act (a public record), and LAN/VPN-only names 404 at the tunnel.
- **Dependence on Cloudflare.** A Cloudflare or tunnel outage takes public access down. The LAN keeps
working through Pi-hole's local records for gitea, nextcloud and keycloak/auth and the
internal-only names. Public apps without a local record are unreachable from the LAN during such an
outage too. Tunnel health becomes an alert condition (R-OBS-04).
- **Cloudflare sees plaintext.** TLS terminates at Cloudflare's edge, so Cloudflare can read the
request and response content of every tunnelled service. Re-encryption to the origin protects
only the Cloudflare → cluster leg.
- **Upload and timeout limits:** 100 MB request bodies and 100 s responses through the proxy. Large
uploads (Nextcloud, Gitea pushes and registry blobs) must use the LAN path, which is why those names
keep local records.
- **Client IP** now arrives as `CF-Connecting-IP` / `X-Forwarded-For` from cloudflared, not as the
TCP source. Logging, rate-limiting and any IP-based policy must read those headers, and trust them
only from cloudflared.
- **The WAN IP stays public** (WireGuard, Plex 32400). The tunnel is not origin cloaking.
- **TLS certificates:** two publicly trusted certificates are in play per public name (Cloudflare edge
+ LE origin). Both auto-renew. ACM/Total TLS is a paid Cloudflare feature and an operator-held
prerequisite.
- **Follow-ups (implementation repo):** deploy cloudflared with a SOPS-encrypted tunnel credential;
move ExternalDNS public records to proxied CNAMEs for `public` services only; add conformance tests
for R-DNS-04/05 and R-SEC-04; alert on tunnel health; then close the 80/443 edge forwards (a
documented manual edge step, like the original forwards).