The operator ruled the implementation repo's main (50c8719) the latest and accepted state. Sync the spec to it. Accepted changes are reflected: one bare-metal environment, TrueNAS bulk storage, R2 off-cluster backup, CloudNativePG, single-node VPN gateway with Cilium fail-closed, inbound WireGuard, per-app auth sidecars, Renovate, the CI gate as approver, and the pm-agent. New ADR-0017..0025; ADR-0016 corrected and accepted; R-* IDs unchanged, eight amended. Where the implementation has simply not met a requirement, the requirement stands and the shortfall is recorded in docs/16-known-gaps.md. Co-Authored-By: Claude Opus 5.5 <[email protected]>
119 lines
8.1 KiB
Markdown
119 lines
8.1 KiB
Markdown
# ADR-0016 — Public ingress: Cloudflare Tunnel (outbound-only) instead of edge port-forwarding
|
|
- Status: Accepted
|
|
- Date: 2026-11-08
|
|
- Accepted by: operator ruling of 2026-11-08 that the implementation is the accepted state; effective on merge.
|
|
- Serves: R-DNS-01, R-DNS-02, R-DNS-03, R-DNS-04, R-DNS-05, R-TLS-01, R-TLS-02, R-SEC-02, R-SEC-04
|
|
- Supersedes: — (amends the **ingress** part of the OQ-1 resolution, "edge does NAT + port-forward
|
|
only", in [00](../docs/00-vision-and-scope.md), [03](../docs/03-networking-isolation-and-egress.md)
|
|
and [09](../docs/09-dns-and-ingress.md). [ADR-0002](adr-0002-cni-cilium.md) (Gateway API on Cilium)
|
|
and [ADR-0006](adr-0006-tls-cert-manager-letsencrypt.md) (cert-manager + Let's Encrypt DNS-01) are
|
|
unchanged: the Gateway still terminates origin TLS with the LE wildcard.)
|
|
|
|
## Context
|
|
The OQ-1 resolution routed public traffic as: public DNS → WAN IP → **edge router port-forward
|
|
80/443** → Cilium Gateway VIP. LAN clients were expected to avoid the WAN path through **Pi-hole
|
|
split-horizon DNS** (R-DNS-03).
|
|
|
|
That design fails in this environment:
|
|
- The ISP router (GFiber) **cannot hairpin NAT**. A LAN client that resolves a service to the WAN IP
|
|
cannot reach it.
|
|
- The same router **advertises Google IPv6 DNS servers via Router Advertisement**, and the operator
|
|
cannot turn that off. LAN clients (Android in particular) therefore use a public resolver
|
|
alongside, or instead of, Pi-hole. They get the WAN answer and fail.
|
|
|
|
So split-horizon DNS cannot be *relied on* to keep LAN clients off the WAN path. Correctness for
|
|
public web apps has to come from the public answer itself working from the LAN.
|
|
|
|
## Decision
|
|
**Public HTTP(S) ingress goes through a Cloudflare Tunnel.**
|
|
|
|
- **cloudflared runs in-cluster** (a Flux-managed workload) and holds an **outbound-only**
|
|
connection to Cloudflare. No inbound connection to the cluster is needed for public web traffic.
|
|
- **Public records** for services declared public (catalog `exposure: public`, or a registered
|
|
platform route: `auth`, Grafana, Crafty's web UI, the apex and its `/runbooks`) are **proxied
|
|
CNAMEs to `<tunnel-uuid>.cfargotunnel.com`**, managed by **ExternalDNS** from declared service
|
|
intent (R-DNS-02). No record is hand-edited.
|
|
- **The edge router's 80/443 forwards are closed** after cutover.
|
|
- **Hostnames are unchanged:** `<service>.home.keithjackson.dev` and the apex
|
|
`home.keithjackson.dev` (R-DNS-01, R-C2-01).
|
|
- **Edge TLS** is Cloudflare's: **Advanced Certificate Manager / Total TLS**, because Universal SSL
|
|
covers only one subdomain level (`*.keithjackson.dev`) and not `<svc>.home.keithjackson.dev`. The
|
|
edge certificate is publicly trusted and auto-renewed (R-TLS-01/02).
|
|
- **Origin TLS stays at the Gateway** with the cert-manager Let's Encrypt wildcard (ADR-0006).
|
|
cloudflared connects to the Gateway over HTTPS and **validates the origin certificate**, using
|
|
the service hostname as SNI. It does not skip verification (`noTLSVerify` is not used).
|
|
- **The tunnel's ingress rules are a deny-list plus a zone wildcard.** Each LAN/VPN-only hostname
|
|
maps to `http_status:404`; the apex and `*.home.keithjackson.dev` then go to the Gateway; a
|
|
**catch-all `http_status:404`** closes the list for hosts outside the zone (R-DNS-05). A static
|
|
conformance check keeps the deny-list and the set of unpublished routes identical in both
|
|
directions.
|
|
- **The Kubernetes API is never reachable through the tunnel**, and is never published by any other
|
|
path. It is LAN/VPN-only (R-SEC-04).
|
|
|
|
**Never published (LAN/VPN-only, `exposure: internal`):** jackett, pihole, semaphore, qbittorrent,
|
|
radarr, sonarr, **headlamp** (a front end to the Kubernetes API), and the non-HTTP L4 services
|
|
**unifi** and **crafty-mc** (Minecraft). The tunnel refuses all of them with a 404. None has a
|
|
public DNS record, except that unifi and crafty-mc hold **DNS-only records at their LAN addresses**,
|
|
which are never proxied or tunnelled.
|
|
|
|
**Plex stays off the tunnel.** Cloudflare's terms restrict proxying video, so Plex remote streaming
|
|
keeps using **plex.tv + WAN tcp/32400** (an edge forward that stays). The `plex.home` web UI becomes
|
|
**LAN-only**: refused at the tunnel, with no proxied record.
|
|
|
|
**Unchanged:**
|
|
- **WireGuard remote access**: udp/51820, reached through the DNS-only (unproxied) `wan.` record,
|
|
which cloudflare-ddns keeps current. The WAN IP therefore **stays public**. The tunnel does not
|
|
hide the origin; it only removes 80/443 from the edge.
|
|
- **Pi-hole keeps serving the LAN**, but split DNS is no longer *required* for public web apps to
|
|
work. It keeps explicit local records for:
|
|
- the internal-only names;
|
|
- the L4 VIPs;
|
|
- the names that must avoid Cloudflare's **100 MB request-body** and **100 s response-timeout**
|
|
limits, or must keep working during an internet outage: **gitea**, **nextcloud**, and
|
|
**keycloak/auth**.
|
|
- **In-cluster CoreDNS split horizon** and the **node `/etc/hosts` pins** (the OIDC issuer and the
|
|
Gitea registry) stay. Cluster-internal callers never go through Cloudflare.
|
|
|
|
## Alternatives considered
|
|
- **Keep the port-forward and fix split horizon.** Rejected: the router's RA-advertised Google IPv6
|
|
DNS cannot be disabled, so some LAN clients will always get the public answer, and the router
|
|
cannot hairpin it.
|
|
- **Replace the ISP router (or put a router capable of hairpin NAT and RA control behind it).**
|
|
Rejected for now: it adds hardware and a non-IaC edge device, which contradicts the "dumb SOHO
|
|
edge" premise of OQ-1, and the GFiber RA behaviour would still need to be contained. It can be
|
|
reopened by a future ADR.
|
|
- **Block or override the Google DNS on the LAN** (firewall DoH/DNS egress, or intercept port 53).
|
|
Rejected: the SOHO edge has no such controls, and client-side DoH/hard-coded resolvers make it
|
|
incomplete anyway.
|
|
- **Tailscale Funnel / ngrok / another reverse-tunnel provider.** Rejected: Cloudflare is already the
|
|
authoritative DNS and token holder (OQ-4), so its tunnel integrates with ExternalDNS and the
|
|
existing scoped token with no new vendor.
|
|
- **Self-hosted reverse tunnel on a VPS** (WireGuard + reverse proxy). Rejected: it adds a host to
|
|
run, patch and secure, outside the cluster's GitOps model.
|
|
|
|
## Consequences
|
|
- **Positive:** LAN clients reach public web apps correctly whichever resolver answers. No NAT
|
|
hairpin is needed. Inbound 80/443 is no longer open on the WAN IP. Publishing is an explicit,
|
|
declared, per-hostname act (a public record), and LAN/VPN-only names 404 at the tunnel.
|
|
- **Dependence on Cloudflare.** A Cloudflare or tunnel outage takes public access down. The LAN keeps
|
|
working through Pi-hole's local records for gitea, nextcloud and keycloak/auth and the
|
|
internal-only names. Public apps without a local record are unreachable from the LAN during such an
|
|
outage too. Tunnel health becomes an alert condition (R-OBS-04).
|
|
- **Cloudflare sees plaintext.** TLS terminates at Cloudflare's edge, so Cloudflare can read the
|
|
request and response content of every tunnelled service. Re-encryption to the origin protects
|
|
only the Cloudflare → cluster leg.
|
|
- **Upload and timeout limits:** 100 MB request bodies and 100 s responses through the proxy. Large
|
|
uploads (Nextcloud, Gitea pushes and registry blobs) must use the LAN path, which is why those names
|
|
keep local records.
|
|
- **Client IP** now arrives as `CF-Connecting-IP` / `X-Forwarded-For` from cloudflared, not as the
|
|
TCP source. Logging, rate-limiting and any IP-based policy must read those headers, and trust them
|
|
only from cloudflared.
|
|
- **The WAN IP stays public** (WireGuard, Plex 32400). The tunnel is not origin cloaking.
|
|
- **TLS certificates:** two publicly trusted certificates are in play per public name (Cloudflare edge
|
|
+ LE origin). Both auto-renew. ACM/Total TLS is a paid Cloudflare feature and an operator-held
|
|
prerequisite.
|
|
- **Follow-ups (implementation repo):** deploy cloudflared with a SOPS-encrypted tunnel credential;
|
|
move ExternalDNS public records to proxied CNAMEs for `public` services only; add conformance tests
|
|
for R-DNS-04/05 and R-SEC-04; alert on tunnel health; then close the 80/443 edge forwards (a
|
|
documented manual edge step, like the original forwards).
|