ADR-0016 (Proposed): public ingress via Cloudflare Tunnel

Record the operator's 2026-11-08 decision to move public HTTP(S) ingress from edge port-forwarding of 80/443 to an outbound-only Cloudflare Tunnel (in-cluster cloudflared, proxied CNAMEs via ExternalDNS). Motivation: the GFiber router cannot hairpin NAT and advertises Google IPv6 DNS via RA, so LAN clients bypass Pi-hole split horizon and fail. Docs 00/01/03/07/08/09/13/14, GLOSSARY and definition-of-done brought into line. R-DNS-01..03 and R-TLS-01..03 wording unchanged. New requirements: R-DNS-04 (no 80/443 forward, resolver-independent LAN reachability), R-DNS-05 (only `public` services at the tunnel; 404 catch-all; no public record for `internal`), R-SEC-04 (Kubernetes API never publicly reachable). Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 12:38:16 -06:00
1 parent b633b7fb08
commit 28d0e0a0f2
@@ -57,3 +57,4 @@ serves. **Do not edit an accepted decision in place** — supersede it with a ne
 | [0013](adr-0013-vpn-provider-pia-gluetun-wireguard.md) | VPN provider: PIA over OpenVPN-UDP + port forwarding | Accepted — provider/protocol only (client+steering superseded by 0015) | R-NET-05/06/07 |
 | [0014](adr-0014-idp-keycloak.md) | Identity Provider: Keycloak (SSO via OIDC + forward-auth) | Accepted | R-IDP-01..06 |
 | [0015](adr-0015-vpn-egress-steering-node-tunnel-cilium.md) | VPN egress steering: node-level tunnel + Cilium Egress Gateway (host kill-switch) | Accepted | R-NET-05/06/07 |
+| [0016](adr-0016-public-ingress-cloudflare-tunnel.md) | Public ingress: Cloudflare Tunnel (outbound-only) instead of edge port-forwarding | Proposed | R-DNS-01..05, R-TLS-01/02, R-SEC-02/04 |
@@ -0,0 +1,112 @@
+# ADR-0016 — Public ingress: Cloudflare Tunnel (outbound-only) instead of edge port-forwarding
+- Status: Proposed (decided by the operator 2026-11-08; set Accepted on merge)
+- Date: 2026-11-08
+- Serves: R-DNS-01, R-DNS-02, R-DNS-03, R-DNS-04, R-DNS-05, R-TLS-01, R-TLS-02, R-SEC-02, R-SEC-04
+- Supersedes: — (amends the **ingress** part of the OQ-1 resolution, "edge does NAT + port-forward
+  only", in [00](../docs/00-vision-and-scope.md), [03](../docs/03-networking-isolation-and-egress.md)
+  and [09](../docs/09-dns-and-ingress.md). [ADR-0002](adr-0002-cni-cilium.md) (Gateway API on Cilium)
+  and [ADR-0006](adr-0006-tls-cert-manager-letsencrypt.md) (cert-manager + Let's Encrypt DNS-01) are
+  unchanged: the Gateway still terminates origin TLS with the LE wildcard.)
+
+## Context
+The OQ-1 resolution routed public traffic as: public DNS → WAN IP → **edge router port-forward
+80/443** → Cilium Gateway VIP. LAN clients were expected to avoid the WAN path through **Pi-hole
+split-horizon DNS** (R-DNS-03).
+
+That design fails in this environment:
+- The ISP router (GFiber) **cannot hairpin NAT**. A LAN client that resolves a service to the WAN IP
+  cannot reach it.
+- The same router **advertises Google IPv6 DNS servers via Router Advertisement**, and the operator
+  cannot turn that off. LAN clients (Android in particular) therefore use a public resolver
+  alongside, or instead of, Pi-hole. They get the WAN answer and fail.
+
+So split-horizon DNS cannot be *relied on* to keep LAN clients off the WAN path. Correctness for
+public web apps has to come from the public answer itself working from the LAN.
+
+## Decision
+**Public HTTP(S) ingress goes through a Cloudflare Tunnel.**
+
+- **cloudflared runs in-cluster** (a Flux-managed workload) and holds an **outbound-only**
+  connection to Cloudflare. No inbound connection to the cluster is needed for public web traffic.
+- **Public records** for `exposure: public` services are **proxied CNAMEs to
+  `<tunnel-uuid>.cfargotunnel.com`**, managed by **ExternalDNS** from declared service intent
+  (R-DNS-02). No record is hand-edited.
+- **The edge router's 80/443 forwards are closed** after cutover.
+- **Hostnames are unchanged:** `<service>.home.keithjackson.dev` and the apex
+  `home.keithjackson.dev` (R-DNS-01, R-C2-01).
+- **Edge TLS** is Cloudflare's: **Advanced Certificate Manager / Total TLS**, because Universal SSL
+  covers only one subdomain level (`*.keithjackson.dev`) and not `<svc>.home.keithjackson.dev`. The
+  edge certificate is publicly trusted and auto-renewed (R-TLS-01/02).
+- **Origin TLS stays at the Gateway** with the cert-manager Let's Encrypt wildcard (ADR-0006).
+  cloudflared connects to the Gateway over HTTPS and **validates the origin certificate**, using
+  the service hostname as SNI. It does not skip verification (`noTLSVerify` is not used).
+- **Only `exposure: public` services have a tunnel route.** The tunnel's ingress rules end in a
+  **catch-all `http_status:404`**, so any other hostname presented at the tunnel is refused
+  (R-DNS-05).
+- **The Kubernetes API is never reachable through the tunnel**, and is never published by any other
+  path. It is LAN/VPN-only (R-SEC-04).
+
+**Never published (LAN/VPN-only, `exposure: internal`):** jackett, pihole, semaphore, qbittorrent,
+radarr, sonarr, **headlamp** (a front end to the Kubernetes API), and the non-HTTP L4 services
+**unifi** and **crafty-mc** (Minecraft). They get no public DNS record and no tunnel route.
+
+**Plex stays off the tunnel.** Cloudflare's terms restrict proxying video, so Plex remote streaming
+keeps using **plex.tv + WAN tcp/32400** (an edge forward that stays). The `plex.home` web UI becomes
+**LAN-only**: no tunnel route and no proxied record.
+
+**Unchanged:**
+- **WireGuard remote access**: udp/51820, reached through the DNS-only (unproxied) `wan.` record,
+  which cloudflare-ddns keeps current. The WAN IP therefore **stays public**. The tunnel does not
+  hide the origin; it only removes 80/443 from the edge.
+- **Pi-hole keeps serving the LAN**, but split DNS is no longer *required* for public web apps to
+  work. It keeps explicit local records for:
+  - the internal-only names;
+  - the L4 VIPs;
+  - the names that must avoid Cloudflare's **100 MB request-body** and **100 s response-timeout**
+    limits, or must keep working during an internet outage: **gitea**, **nextcloud**, and
+    **keycloak/auth**.
+- **In-cluster CoreDNS split horizon** and the **node `/etc/hosts` pins** (the OIDC issuer and the
+  Gitea registry) stay. Cluster-internal callers never go through Cloudflare.
+
+## Alternatives considered
+- **Keep the port-forward and fix split horizon.** Rejected: the router's RA-advertised Google IPv6
+  DNS cannot be disabled, so some LAN clients will always get the public answer, and the router
+  cannot hairpin it.
+- **Replace the ISP router (or put a router capable of hairpin NAT and RA control behind it).**
+  Rejected for now: it adds hardware and a non-IaC edge device, which contradicts the "dumb SOHO
+  edge" premise of OQ-1, and the GFiber RA behaviour would still need to be contained. It can be
+  reopened by a future ADR.
+- **Block or override the Google DNS on the LAN** (firewall DoH/DNS egress, or intercept port 53).
+  Rejected: the SOHO edge has no such controls, and client-side DoH/hard-coded resolvers make it
+  incomplete anyway.
+- **Tailscale Funnel / ngrok / another reverse-tunnel provider.** Rejected: Cloudflare is already the
+  authoritative DNS and token holder (OQ-4), so its tunnel integrates with ExternalDNS and the
+  existing scoped token with no new vendor.
+- **Self-hosted reverse tunnel on a VPS** (WireGuard + reverse proxy). Rejected: it adds a host to
+  run, patch and secure, outside the cluster's GitOps model.
+
+## Consequences
+- **Positive:** LAN clients reach public web apps correctly whichever resolver answers. No NAT
+  hairpin is needed. Inbound 80/443 is no longer open on the WAN IP. Publishing is an explicit,
+  declared, per-hostname act (tunnel route + record), and anything else 404s at the edge.
+- **Dependence on Cloudflare.** A Cloudflare or tunnel outage takes public access down. The LAN keeps
+  working through Pi-hole's local records for gitea, nextcloud and keycloak/auth and the
+  internal-only names. Public apps without a local record are unreachable from the LAN during such an
+  outage too. Tunnel health becomes an alert condition (R-OBS-04).
+- **Cloudflare sees plaintext.** TLS terminates at Cloudflare's edge, so Cloudflare can read the
+  request and response content of every tunnelled service. Re-encryption to the origin protects
+  only the Cloudflare → cluster leg.
+- **Upload and timeout limits:** 100 MB request bodies and 100 s responses through the proxy. Large
+  uploads (Nextcloud, Gitea pushes and registry blobs) must use the LAN path, which is why those names
+  keep local records.
+- **Client IP** now arrives as `CF-Connecting-IP` / `X-Forwarded-For` from cloudflared, not as the
+  TCP source. Logging, rate-limiting and any IP-based policy must read those headers, and trust them
+  only from cloudflared.
+- **The WAN IP stays public** (WireGuard, Plex 32400). The tunnel is not origin cloaking.
+- **TLS certificates:** two publicly trusted certificates are in play per public name (Cloudflare edge
+  + LE origin). Both auto-renew. ACM/Total TLS is a paid Cloudflare feature and an operator-held
+  prerequisite.
+- **Follow-ups (implementation repo):** deploy cloudflared with a SOPS-encrypted tunnel credential;
+  move ExternalDNS public records to proxied CNAMEs for `public` services only; add conformance tests
+  for R-DNS-04/05 and R-SEC-04; alert on tunnel health; then close the 80/443 edge forwards (a
+  documented manual edge step, like the original forwards).