From 28d0e0a0f2962dead612f01b9b5dbef582281c15 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 8 Nov 2026 12:38:16 -0600 Subject: [PATCH] ADR-0016 (Proposed): public ingress via Cloudflare Tunnel Record the operator's 2026-11-08 decision to move public HTTP(S) ingress from edge port-forwarding of 80/443 to an outbound-only Cloudflare Tunnel (in-cluster cloudflared, proxied CNAMEs via ExternalDNS). Motivation: the GFiber router cannot hairpin NAT and advertises Google IPv6 DNS via RA, so LAN clients bypass Pi-hole split horizon and fail. Docs 00/01/03/07/08/09/13/14, GLOSSARY and definition-of-done brought into line. R-DNS-01..03 and R-TLS-01..03 wording unchanged. New requirements: R-DNS-04 (no 80/443 forward, resolver-independent LAN reachability), R-DNS-05 (only `public` services at the tunnel; 404 catch-all; no public record for `internal`), R-SEC-04 (Kubernetes API never publicly reachable). Co-Authored-By: Claude Opus 5.5 --- GLOSSARY.md | 7 +- agents/definition-of-done.md | 5 +- decisions/README.md | 1 + ...r-0016-public-ingress-cloudflare-tunnel.md | 112 ++++++++++++++++++ docs/00-vision-and-scope.md | 14 ++- docs/01-architecture-overview.md | 4 +- docs/03-networking-isolation-and-egress.md | 11 +- docs/07-security-and-secrets.md | 1 + docs/08-tls-and-certificates.md | 18 ++- docs/09-dns-and-ingress.md | 97 +++++++++++---- docs/13-service-catalog.md | 8 +- docs/14-testing-and-conformance.md | 5 + 12 files changed, 242 insertions(+), 41 deletions(-) create mode 100644 decisions/adr-0016-public-ingress-cloudflare-tunnel.md diff --git a/GLOSSARY.md b/GLOSSARY.md index bf3292e..43866fb 100644 --- a/GLOSSARY.md +++ b/GLOSSARY.md @@ -26,8 +26,11 @@ two things (e.g. "VM"), this file disambiguates. the port level (R-NET-01). - **Egress class** — a service's declared outbound path: `internet` (standard WAN) or `vpn` (commercial VPN tunnel). See [`docs/03-networking-isolation-and-egress.md`](docs/03-networking-isolation-and-egress.md). -- **Exposure class** — how a service is reachable: `internal` (LAN-only) or `public` (edge - port-forwarded). +- **Exposure class** — how a service is reachable: `internal` (LAN/VPN-only) or `public` (published + through the Cloudflare Tunnel, ADR-0016). +- **Cloudflare Tunnel** — the outbound-only connection from in-cluster cloudflared to Cloudflare that + carries all public HTTP(S) ingress; public records are proxied CNAMEs to `.cfargotunnel.com` + (ADR-0016). - **Kill-switch** — the guarantee that a `vpn`-class workload leaks nothing to the plain internet if the tunnel drops (R-NET-06); fail-closed. - **Multi-WAN** — multiple upstream internet uplinks with routing-rule-based selection; physical WAN diff --git a/agents/definition-of-done.md b/agents/definition-of-done.md index 8c05715..fb47753 100644 --- a/agents/definition-of-done.md +++ b/agents/definition-of-done.md @@ -53,7 +53,10 @@ screenshot, or "it reconciles" is **not** done (R-TEST-01..04, [`../docs/14-test ### DNS & ingress (R-DNS-*) - [ ] `.home.keithjackson.dev` record appears automatically and routes correctly. - [ ] `internal` is unreachable publicly; `public` is reachable + TLS-valid. -- [ ] Internal clients resolve to the internal Gateway (no public hairpin). +- [ ] Internal clients resolving via Pi-hole reach the internal Gateway; via a public resolver, a + `public` service still works through the tunnel (R-DNS-04). +- [ ] Non-`public` hostnames get 404 at the tunnel and have no public record (R-DNS-05); the + Kubernetes API is not publicly reachable (R-SEC-04). ### Automated updates (R-UPD-*) - [ ] Advances to a newer **stable** version automatically on schedule with **no human/AI edit**. diff --git a/decisions/README.md b/decisions/README.md index d779b20..596f5cc 100644 --- a/decisions/README.md +++ b/decisions/README.md @@ -57,3 +57,4 @@ serves. **Do not edit an accepted decision in place** — supersede it with a ne | [0013](adr-0013-vpn-provider-pia-gluetun-wireguard.md) | VPN provider: PIA over OpenVPN-UDP + port forwarding | Accepted — provider/protocol only (client+steering superseded by 0015) | R-NET-05/06/07 | | [0014](adr-0014-idp-keycloak.md) | Identity Provider: Keycloak (SSO via OIDC + forward-auth) | Accepted | R-IDP-01..06 | | [0015](adr-0015-vpn-egress-steering-node-tunnel-cilium.md) | VPN egress steering: node-level tunnel + Cilium Egress Gateway (host kill-switch) | Accepted | R-NET-05/06/07 | +| [0016](adr-0016-public-ingress-cloudflare-tunnel.md) | Public ingress: Cloudflare Tunnel (outbound-only) instead of edge port-forwarding | Proposed | R-DNS-01..05, R-TLS-01/02, R-SEC-02/04 | diff --git a/decisions/adr-0016-public-ingress-cloudflare-tunnel.md b/decisions/adr-0016-public-ingress-cloudflare-tunnel.md new file mode 100644 index 0000000..5785582 --- /dev/null +++ b/decisions/adr-0016-public-ingress-cloudflare-tunnel.md @@ -0,0 +1,112 @@ +# ADR-0016 — Public ingress: Cloudflare Tunnel (outbound-only) instead of edge port-forwarding +- Status: Proposed (decided by the operator 2026-11-08; set Accepted on merge) +- Date: 2026-11-08 +- Serves: R-DNS-01, R-DNS-02, R-DNS-03, R-DNS-04, R-DNS-05, R-TLS-01, R-TLS-02, R-SEC-02, R-SEC-04 +- Supersedes: — (amends the **ingress** part of the OQ-1 resolution, "edge does NAT + port-forward + only", in [00](../docs/00-vision-and-scope.md), [03](../docs/03-networking-isolation-and-egress.md) + and [09](../docs/09-dns-and-ingress.md). [ADR-0002](adr-0002-cni-cilium.md) (Gateway API on Cilium) + and [ADR-0006](adr-0006-tls-cert-manager-letsencrypt.md) (cert-manager + Let's Encrypt DNS-01) are + unchanged: the Gateway still terminates origin TLS with the LE wildcard.) + +## Context +The OQ-1 resolution routed public traffic as: public DNS → WAN IP → **edge router port-forward +80/443** → Cilium Gateway VIP. LAN clients were expected to avoid the WAN path through **Pi-hole +split-horizon DNS** (R-DNS-03). + +That design fails in this environment: +- The ISP router (GFiber) **cannot hairpin NAT**. A LAN client that resolves a service to the WAN IP + cannot reach it. +- The same router **advertises Google IPv6 DNS servers via Router Advertisement**, and the operator + cannot turn that off. LAN clients (Android in particular) therefore use a public resolver + alongside, or instead of, Pi-hole. They get the WAN answer and fail. + +So split-horizon DNS cannot be *relied on* to keep LAN clients off the WAN path. Correctness for +public web apps has to come from the public answer itself working from the LAN. + +## Decision +**Public HTTP(S) ingress goes through a Cloudflare Tunnel.** + +- **cloudflared runs in-cluster** (a Flux-managed workload) and holds an **outbound-only** + connection to Cloudflare. No inbound connection to the cluster is needed for public web traffic. +- **Public records** for `exposure: public` services are **proxied CNAMEs to + `.cfargotunnel.com`**, managed by **ExternalDNS** from declared service intent + (R-DNS-02). No record is hand-edited. +- **The edge router's 80/443 forwards are closed** after cutover. +- **Hostnames are unchanged:** `.home.keithjackson.dev` and the apex + `home.keithjackson.dev` (R-DNS-01, R-C2-01). +- **Edge TLS** is Cloudflare's: **Advanced Certificate Manager / Total TLS**, because Universal SSL + covers only one subdomain level (`*.keithjackson.dev`) and not `.home.keithjackson.dev`. The + edge certificate is publicly trusted and auto-renewed (R-TLS-01/02). +- **Origin TLS stays at the Gateway** with the cert-manager Let's Encrypt wildcard (ADR-0006). + cloudflared connects to the Gateway over HTTPS and **validates the origin certificate**, using + the service hostname as SNI. It does not skip verification (`noTLSVerify` is not used). +- **Only `exposure: public` services have a tunnel route.** The tunnel's ingress rules end in a + **catch-all `http_status:404`**, so any other hostname presented at the tunnel is refused + (R-DNS-05). +- **The Kubernetes API is never reachable through the tunnel**, and is never published by any other + path. It is LAN/VPN-only (R-SEC-04). + +**Never published (LAN/VPN-only, `exposure: internal`):** jackett, pihole, semaphore, qbittorrent, +radarr, sonarr, **headlamp** (a front end to the Kubernetes API), and the non-HTTP L4 services +**unifi** and **crafty-mc** (Minecraft). They get no public DNS record and no tunnel route. + +**Plex stays off the tunnel.** Cloudflare's terms restrict proxying video, so Plex remote streaming +keeps using **plex.tv + WAN tcp/32400** (an edge forward that stays). The `plex.home` web UI becomes +**LAN-only**: no tunnel route and no proxied record. + +**Unchanged:** +- **WireGuard remote access**: udp/51820, reached through the DNS-only (unproxied) `wan.` record, + which cloudflare-ddns keeps current. The WAN IP therefore **stays public**. The tunnel does not + hide the origin; it only removes 80/443 from the edge. +- **Pi-hole keeps serving the LAN**, but split DNS is no longer *required* for public web apps to + work. It keeps explicit local records for: + - the internal-only names; + - the L4 VIPs; + - the names that must avoid Cloudflare's **100 MB request-body** and **100 s response-timeout** + limits, or must keep working during an internet outage: **gitea**, **nextcloud**, and + **keycloak/auth**. +- **In-cluster CoreDNS split horizon** and the **node `/etc/hosts` pins** (the OIDC issuer and the + Gitea registry) stay. Cluster-internal callers never go through Cloudflare. + +## Alternatives considered +- **Keep the port-forward and fix split horizon.** Rejected: the router's RA-advertised Google IPv6 + DNS cannot be disabled, so some LAN clients will always get the public answer, and the router + cannot hairpin it. +- **Replace the ISP router (or put a router capable of hairpin NAT and RA control behind it).** + Rejected for now: it adds hardware and a non-IaC edge device, which contradicts the "dumb SOHO + edge" premise of OQ-1, and the GFiber RA behaviour would still need to be contained. It can be + reopened by a future ADR. +- **Block or override the Google DNS on the LAN** (firewall DoH/DNS egress, or intercept port 53). + Rejected: the SOHO edge has no such controls, and client-side DoH/hard-coded resolvers make it + incomplete anyway. +- **Tailscale Funnel / ngrok / another reverse-tunnel provider.** Rejected: Cloudflare is already the + authoritative DNS and token holder (OQ-4), so its tunnel integrates with ExternalDNS and the + existing scoped token with no new vendor. +- **Self-hosted reverse tunnel on a VPS** (WireGuard + reverse proxy). Rejected: it adds a host to + run, patch and secure, outside the cluster's GitOps model. + +## Consequences +- **Positive:** LAN clients reach public web apps correctly whichever resolver answers. No NAT + hairpin is needed. Inbound 80/443 is no longer open on the WAN IP. Publishing is an explicit, + declared, per-hostname act (tunnel route + record), and anything else 404s at the edge. +- **Dependence on Cloudflare.** A Cloudflare or tunnel outage takes public access down. The LAN keeps + working through Pi-hole's local records for gitea, nextcloud and keycloak/auth and the + internal-only names. Public apps without a local record are unreachable from the LAN during such an + outage too. Tunnel health becomes an alert condition (R-OBS-04). +- **Cloudflare sees plaintext.** TLS terminates at Cloudflare's edge, so Cloudflare can read the + request and response content of every tunnelled service. Re-encryption to the origin protects + only the Cloudflare → cluster leg. +- **Upload and timeout limits:** 100 MB request bodies and 100 s responses through the proxy. Large + uploads (Nextcloud, Gitea pushes and registry blobs) must use the LAN path, which is why those names + keep local records. +- **Client IP** now arrives as `CF-Connecting-IP` / `X-Forwarded-For` from cloudflared, not as the + TCP source. Logging, rate-limiting and any IP-based policy must read those headers, and trust them + only from cloudflared. +- **The WAN IP stays public** (WireGuard, Plex 32400). The tunnel is not origin cloaking. +- **TLS certificates:** two publicly trusted certificates are in play per public name (Cloudflare edge + + LE origin). Both auto-renew. ACM/Total TLS is a paid Cloudflare feature and an operator-held + prerequisite. +- **Follow-ups (implementation repo):** deploy cloudflared with a SOPS-encrypted tunnel credential; + move ExternalDNS public records to proxied CNAMEs for `public` services only; add conformance tests + for R-DNS-04/05 and R-SEC-04; alert on tunnel health; then close the 80/443 edge forwards (a + documented manual edge step, like the original forwards). diff --git a/docs/00-vision-and-scope.md b/docs/00-vision-and-scope.md index 100d08d..a580ec4 100644 --- a/docs/00-vision-and-scope.md +++ b/docs/00-vision-and-scope.md @@ -40,7 +40,9 @@ The platform is delivered in two stages: ## The domain & DNS model -- Public apex `*.keithjackson.dev` is available and its records resolve to the network's public IP. +- Public apex `*.keithjackson.dev` is available in Cloudflare DNS. `public` services under + `*.home.keithjackson.dev` are published through a **Cloudflare Tunnel** (proxied records, no inbound + 80/443 at the edge — ADR-0016); the network's public IP stays in use for WireGuard and Plex only. - All platform services live under `*.home.keithjackson.dev`. - Each service is reachable at `.home.keithjackson.dev` (e.g. `plex.home.keithjackson.dev`). - The **root** `home.keithjackson.dev` serves the **Command & Control (C2) center** UI. @@ -117,6 +119,12 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate. - **R-DNS-02** — DNS records MUST be managed automatically from declared service intent. - **R-DNS-03** — Internal clients MUST resolve service names correctly (split-horizon where public and internal answers differ). +- **R-DNS-04** — `public` HTTP(S) services MUST be reachable from the internet **without inbound + 80/443 port-forwarding at the edge**, and LAN clients MUST reach them correctly **whichever resolver + answers** (public or internal) — no dependency on NAT hairpin (ADR-0016). +- **R-DNS-05** — Only services declared `exposure: public` MAY be reachable through the public + ingress path. A request for any other hostname at that path MUST be refused (HTTP 404), and + `internal` services MUST have no public DNS record (ADR-0016). ### Observability & alerting — `R-OBS-*` - **R-OBS-01** — Services MUST be monitored (metrics, logs, health). @@ -137,6 +145,8 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate. - **R-SEC-02** — The platform MUST follow least-privilege for workload identities and network reach. - **R-SEC-03** — All the above properties MUST be demonstrable/auditable (an operator can prove isolation, egress path, cert validity, and backup integrity). +- **R-SEC-04** — The Kubernetes API MUST NOT be reachable from the public internet by any path (no + tunnel route, no edge forward, no public DNS record); it is LAN/VPN-only (ADR-0016). ### Identity & access (SSO) — `R-IDP-*` - **R-IDP-01** — The platform MUST provide a central Identity Provider (SSO) as the single user @@ -204,6 +214,8 @@ implementation. - **OQ-1 (WAN / edge)** — **RESOLVED**: one commercial WAN, no multi-ISP plans; dumb SOHO edge (no VLAN/VPN/services). All segmentation + VPN in-cluster; edge does NAT + port-forward only. Physical multi-WAN out of scope. See [`03`](03-networking-isolation-and-egress.md), [`09`](09-dns-and-ingress.md). + **Amended 2026-11-08 (ADR-0016):** public web ingress moved to an outbound-only **Cloudflare Tunnel**; + the edge no longer forwards 80/443. Its remaining forwards are WireGuard udp/51820 and Plex tcp/32400. - **OQ-2 (bulk media)** — **RESOLVED (deferred backing)**: Plex bulk media lives **outside the cluster**; the specific backing is **TBD later** and does not block the core platform. Plex's metadata DB still lives on replicated Longhorn. The **off-cluster DR backup target** (for app diff --git a/docs/01-architecture-overview.md b/docs/01-architecture-overview.md index b03fb78..818c61b 100644 --- a/docs/01-architecture-overview.md +++ b/docs/01-architecture-overview.md @@ -54,8 +54,8 @@ choice here without a superseding ADR.** │ dev/test: libvirt KVMs (OpenTofu + cloud-init + Ansible) │ │ prod: 3× mini-PC bare metal, local disks │ ├──────────────────────────────────────────────────────────────────────────┤ -│ Edge: router/firewall — multi-WAN uplinks, port-forward to Gateway, │ -│ VPN uplink(s), split-horizon DNS │ +│ Edge: SOHO router — single WAN, NAT; forwards only WireGuard + Plex │ +│ Public web ingress: Cloudflare Tunnel (outbound-only cloudflared, ADR-0016)│ └──────────────────────────────────────────────────────────────────────────┘ ``` diff --git a/docs/03-networking-isolation-and-egress.md b/docs/03-networking-isolation-and-egress.md index d194d2e..4025b02 100644 --- a/docs/03-networking-isolation-and-egress.md +++ b/docs/03-networking-isolation-and-egress.md @@ -82,8 +82,10 @@ as multi-ISP routing but as **two egress paths selected by routing rules**: route across; the `internet` path is simply the default route out. - **The edge router is a dumb SOHO device** — no VLANs, no VPN, no services, no API/IaC. Therefore **all segmentation and all VPN handling live in-cluster** (Cilium policy + the VPN gateway), not on - the edge. The edge's only jobs are NAT out and **port-forwarding 80/443 (and specific service - ports) to the ingress VIP** for `public` services — a documented manual step ([09](09-dns-and-ingress.md)). + the edge. The edge's only jobs are NAT out and a few **manual port-forwards that are not web + ingress**: WireGuard udp/51820 and Plex tcp/32400 ([09](09-dns-and-ingress.md)). Public web ingress + uses an outbound-only **Cloudflare Tunnel** from in-cluster cloudflared (ADR-0016), so the edge does + not forward 80/443. - The **service-facing contract is unchanged**: a service still just declares `egress: internet|vpn`. ## VPN egress model: node-level tunnel + Cilium Egress Gateway (R-NET-05/06 — ADR-0015; provider ADR-0013) @@ -133,7 +135,8 @@ closed (no packets to the plain internet), while `internet`-class is unaffected. North-south exposure is handled by **Gateway API implemented by Cilium**, with TLS from cert-manager and DNS from ExternalDNS. Detailed in [`09-dns-and-ingress.md`](09-dns-and-ingress.md). -Exposure is a declared property: `internal` (LAN-only) vs `public` (port-forwarded at the edge). +Exposure is a declared property: `internal` (LAN/VPN-only) vs `public` (published through the +Cloudflare Tunnel, ADR-0016). ## Observability of the network (supports R-SEC-03, R-OBS-04) - **Hubble** provides flow visibility to verify default-deny and troubleshoot allow rules. @@ -151,6 +154,8 @@ Exposure is a declared property: `internal` (LAN-only) vs `public` (port-forward ## Resolved - **OQ-1** — one commercial WAN, no multi-ISP plans; dumb SOHO edge (no VLAN/VPN/services). All segmentation + VPN in-cluster; edge does NAT + port-forward only. Multi-ISP routing out of scope. + Amended 2026-11-08 (ADR-0016): 80/443 are no longer forwarded; public web ingress is a Cloudflare + Tunnel. - **OQ-3** — **PIA over OpenVPN-UDP** (ADR-0013); egress is a **node-level host tunnel** steered by **Cilium Egress Gateway** with a host nftables kill-switch (ADR-0015), rules enforced outside the service. diff --git a/docs/07-security-and-secrets.md b/docs/07-security-and-secrets.md index 7ce6295..0ca36e1 100644 --- a/docs/07-security-and-secrets.md +++ b/docs/07-security-and-secrets.md @@ -51,6 +51,7 @@ Defense in depth with declarative, auditable controls: node(s) via Ansible vault** (the tunnel is host infra, not a cluster pod; ADR-0015), NOT a cluster SOPS Secret. - Keycloak admin credential and per-service **OIDC client secrets** ([15](15-identity-and-access.md)). - DNS provider API token (Cloudflare) for ACME DNS-01 + ExternalDNS. +- Cloudflare Tunnel credential/token for in-cluster cloudflared (SOPS-encrypted; ADR-0016). - SMTP credentials for Alertmanager email. - Service credentials (Nextcloud, Gitea admin, UniFi, Semaphore, Kasm, DB passwords). - Off-cluster backup target credentials (S3/NFS) for Velero/Longhorn. diff --git a/docs/08-tls-and-certificates.md b/docs/08-tls-and-certificates.md index 7757c3b..65a434c 100644 --- a/docs/08-tls-and-certificates.md +++ b/docs/08-tls-and-certificates.md @@ -1,6 +1,7 @@ # 08 — TLS & Certificates -Serves: **R-TLS-01/02/03**, **R-DNS-01**. Related: [ADR-0006 (cert-manager + Let's Encrypt)](../decisions/adr-0006-tls-cert-manager-letsencrypt.md). +Serves: **R-TLS-01/02/03**, **R-DNS-01**. Related: [ADR-0006 (cert-manager + Let's Encrypt)](../decisions/adr-0006-tls-cert-manager-letsencrypt.md), +[ADR-0016 (Cloudflare Tunnel public ingress)](../decisions/adr-0016-public-ingress-cloudflare-tunnel.md). ## Requirement recap @@ -30,8 +31,8 @@ services (R-TLS-01/02/03). ## DNS-01 challenge access (OQ-4 — RESOLVED: scoped Cloudflare API token) -Cloudflare already has a wildcard `*.keithjackson.dev` → public-IP record, which covers -*reachability* but not certificate issuance (ACME DNS-01 must **write `_acme-challenge` TXT records**, +Cloudflare already has a wildcard `*.keithjackson.dev` → public-IP record, which does not cover +certificate issuance (ACME DNS-01 must **write `_acme-challenge` TXT records**, and that wildcard does not match two-label-deep names like `plex.home.keithjackson.dev`). **Decision (OQ-4):** use a **scoped Cloudflare API token** — `Zone·DNS·Edit` limited to the @@ -55,8 +56,15 @@ DNS edit only) so its blast radius is bounded (R-SEC-02). Rotation = re-encrypt ## Where TLS terminates - **North-south**: at the **Gateway API** listener (implemented by Cilium, [09](09-dns-and-ingress.md)), - using the cert-manager-issued cert. Public services are reached via the edge router forwarding to - the Gateway; internal services resolve to the Gateway on the LAN. + using the cert-manager-issued cert. Internal services resolve to the Gateway on the LAN. +- **Public services via the Cloudflare Tunnel (ADR-0016)** are TLS twice: + - **Edge:** Cloudflare terminates the client's TLS with an ACM/Total TLS certificate. It is + publicly trusted and auto-renewed (R-TLS-01/02). Universal SSL is not enough, because it covers + only one subdomain level. + - **Origin:** cloudflared re-encrypts to the Gateway and **validates** the Let's Encrypt wildcard, + using the service hostname as SNI. Verification is never disabled. + + Cloudflare sees plaintext between the two legs, a trade-off accepted in ADR-0016. - Internal service-to-service traffic relies primarily on network policy for security; mTLS between workloads is **optional** future hardening (could be added via Cilium/service mesh) and is not a launch requirement — record as a possible future ADR. diff --git a/docs/09-dns-and-ingress.md b/docs/09-dns-and-ingress.md index d987f64..e44070b 100644 --- a/docs/09-dns-and-ingress.md +++ b/docs/09-dns-and-ingress.md @@ -1,14 +1,16 @@ # 09 — DNS & Ingress -Serves: **R-DNS-01/02/03**, **R-C2-01**, works with **R-TLS-***. Related: -[ADR-0002 (Cilium/Gateway API)](../decisions/adr-0002-cni-cilium.md). +Serves: **R-DNS-01/02/03/04/05**, **R-C2-01**, **R-SEC-04**, works with **R-TLS-***. Related: +[ADR-0002 (Cilium/Gateway API)](../decisions/adr-0002-cni-cilium.md), +[ADR-0016 (Cloudflare Tunnel public ingress)](../decisions/adr-0016-public-ingress-cloudflare-tunnel.md). ## Naming model - All services live at `.home.keithjackson.dev` (e.g. `plex.home.keithjackson.dev`). - The **root** `home.keithjackson.dev` serves the **Command & Control UI** ([12](12-command-center-ui.md)). -- Public apex `*.keithjackson.dev` resolves to the network's public IP; `*.home.keithjackson.dev` - is the platform's zone. +- `*.home.keithjackson.dev` is the platform's zone, in Cloudflare DNS. `public` services are + published as **proxied CNAMEs to the Cloudflare Tunnel** (`.cfargotunnel.com`, + ADR-0016), not as records pointing at the network's public IP. ## DNS automation (R-DNS-02) @@ -16,16 +18,32 @@ Serves: **R-DNS-01/02/03**, **R-C2-01**, works with **R-TLS-***. Related: authoritative DNS provider (Cloudflare, per OQ-4) automatically — a new service's DNS appears from its declared intent, no manual record editing. - Records for `*.home.keithjackson.dev` are created/updated/removed as services come and go. +- Public records are created **only for `exposure: public` services** (R-DNS-05). The DNS-only + (unproxied) `wan.` record for WireGuard is kept current by cloudflare-ddns, outside this flow. ## Split-horizon resolution (R-DNS-03) Internal and external answers differ: -- **External** (`*.home.keithjackson.dev` from the internet): resolves to the public IP; reaches - the edge router, which forwards `public`-class services to the in-cluster Gateway. -- **Internal** (LAN clients): resolves `.home.keithjackson.dev` to the **internal Gateway - address** directly, avoiding hairpinning through the public IP. -- **PiHole** (itself an onboarded service) provides internal LAN DNS and the split-horizon answers - for `.home.keithjackson.dev`; the trusted Let's Encrypt cert is served either way ([08](08-tls-and-certificates.md)). +- **External** (`.home.keithjackson.dev` from the internet): a `public` service resolves to + **Cloudflare's edge** (proxied CNAME to the tunnel). Cloudflare forwards the request down the + outbound-only tunnel to cloudflared in the cluster, which forwards it to the Gateway (ADR-0016). + `internal` services have no public answer. +- **Internal** (LAN clients): Pi-hole answers with the **internal Gateway address** (or the service's + L4 VIP) for the names it holds local records for. +- **Correctness does not depend on split horizon for `public` web apps** (R-DNS-04). The ISP router + cannot hairpin NAT and advertises a public IPv6 resolver via RA, so some LAN clients will get the + public answer. That answer goes through Cloudflare and works from the LAN too. +- **PiHole** (itself an onboarded service) provides internal LAN DNS. It keeps explicit local records + for: + - the `internal` names; + - the L4 VIPs; + - the names that must avoid Cloudflare's 100 MB body / 100 s timeout limits or must work during an + internet outage: **gitea**, **nextcloud**, **keycloak/auth**. + + A trusted certificate is served either way: Cloudflare's edge certificate on the tunnel path, the + Let's Encrypt wildcard on the LAN path ([08](08-tls-and-certificates.md)). +- **In-cluster** resolution keeps its own split horizon (CoreDNS) and the node `/etc/hosts` pins (OIDC + issuer, Gitea registry), so cluster-internal callers never traverse Cloudflare. > DNS is thus both a platform capability *and* a hosted service. Bootstrapping order matters: the > platform needs working DNS before PiHole is fully managed — handled by the bring-up sequence in @@ -40,19 +58,41 @@ Internal and external answers differ: - Each service contributes an **HTTPRoute** mapping its hostname to its Service — rendered from the ServiceDefinition, not hand-authored. - **Exposure class** is declared per service: - - `internal` — reachable only on the LAN (no edge port-forward; internal DNS only). - - `public` — additionally reachable from the internet via the edge router forwarding 80/443 to the - Gateway. + - `internal` — reachable only on the LAN/VPN (no tunnel route, no public record; internal DNS only). + - `public` — additionally reachable from the internet through the **Cloudflare Tunnel**: + cloudflared → the Gateway over HTTPS (ADR-0016). +- **Never `public`:** jackett, pihole, semaphore, qbittorrent, radarr, sonarr, **headlamp** (a front + end to the Kubernetes API), plex's web UI, and the non-HTTP L4 services unifi and crafty-mc. +- **The Kubernetes API is never published** — no tunnel route, no edge forward, no public record + (R-SEC-04). + +## Public ingress: Cloudflare Tunnel (ADR-0016) + +- **cloudflared** runs in-cluster as a Flux-managed workload and holds an **outbound-only** + connection to Cloudflare. No inbound 80/443 reaches the WAN IP. +- Its ingress rules list one route per `public` hostname → the Gateway (HTTPS, origin certificate + validated with the hostname as SNI). The rules end in a **catch-all `http_status:404`**, so an + unlisted hostname is refused at the tunnel (R-DNS-05). +- **Edge TLS** is Cloudflare ACM/Total TLS (Universal SSL does not cover two-label-deep names); + **origin TLS** is the Let's Encrypt wildcard at the Gateway ([08](08-tls-and-certificates.md)). +- **Limits and trade-offs:** + - 100 MB request bodies and 100 s responses through the proxy; + - Cloudflare terminates TLS and sees plaintext; + - a Cloudflare outage takes public access down (the LAN keeps working through Pi-hole's local + records); + - the client IP arrives as `CF-Connecting-IP` / `X-Forwarded-For`, not as the TCP source. ## Edge integration (single SOHO router — OQ-1 resolved) -- The environment has **one SOHO edge router**: no VLANs, no VPN, no services, no API/IaC. Its only - ingress job is **manual port-forwarding of 80/443 (and specific service ports, e.g. Plex 32400, - Minecraft 25565) to the Gateway VIP** for `public` services. This is a documented manual edge step - (the SOHO device has no automation) — the analogue of the old pfSense manual-config exception. - Confirm the router can port-forward (basic NAT; almost all can). +- The environment has **one SOHO edge router**: no VLANs, no VPN, no services, no API/IaC. It + **does not forward 80/443** (public web ingress is the tunnel, ADR-0016). Its only remaining + inbound forwards are **WireGuard udp/51820** (remote access, via the DNS-only `wan.` record) and + **Plex tcp/32400** (remote streaming via plex.tv; Cloudflare's terms restrict proxied video). This + is a documented manual edge step (the SOHO device has no automation), the analogue of the old + pfSense manual-config exception. The WAN IP therefore stays public. - No multi-WAN inbound rules — single WAN ([03](03-networking-isolation-and-egress.md)). -- `internal` services are never port-forwarded; they rely on internal DNS + Gateway on the LAN. +- `internal` services are never port-forwarded or tunnelled; they rely on internal DNS + Gateway (or + their L4 VIP) on the LAN, or WireGuard from outside. - **Ingress VIP mechanism — resolved: Cilium L2 announcement (ARP/NDP)** on the flat LAN. BGP is not available on a SOHO router, and a single flat subnet makes L2 announcement the natural choice. @@ -62,9 +102,10 @@ A wildcard `*.keithjackson.dev` matches only **one** label deep — it covers `h but **not** `plex.home.keithjackson.dev`. Consequences: - **`internal` services need no public record** — PiHole resolves `.home.keithjackson.dev` to the internal Gateway VIP (split-horizon). This is the majority of services. -- **`public` services** need a public record for `.home.keithjackson.dev`. Per OQ-4 (resolved), - **ExternalDNS manages these automatically** via the scoped Cloudflare API token — no manual record - editing (see [08](08-tls-and-certificates.md)). +- **`public` services** need a public record for `.home.keithjackson.dev`: a **proxied CNAME to + the tunnel** (ADR-0016). Per OQ-4 (resolved), **ExternalDNS manages these automatically** via the + scoped Cloudflare API token — no manual record editing (see [08](08-tls-and-certificates.md)). The + same depth limit is why edge TLS needs ACM/Total TLS rather than Universal SSL. ## Interaction with default-deny @@ -76,10 +117,18 @@ not let other in-cluster services reach it unless an explicit allow rule exists. - A new service's `.home.keithjackson.dev` record appears automatically and routes to it. - `internal` services are unreachable from the public internet; `public` services are reachable and TLS-valid. -- Internal clients resolve to the internal Gateway (no public hairpin) and get a trusted cert. +- Internal clients resolving through Pi-hole reach the internal Gateway for the names it holds local + records for, and get a trusted cert. +- A LAN client resolving a `public` name through a **public** resolver reaches it, TLS-valid + (R-DNS-04). The WAN IP does not answer on 80/443. +- A request at the tunnel for any hostname that is not a `public` service returns **404**, and no + `internal` service has a public DNS record (R-DNS-05). +- The Kubernetes API is unreachable from the internet: no tunnel route, no edge forward, no public + record (R-SEC-04). - The root `home.keithjackson.dev` serves the C2 UI. ## Resolved -- **OQ-1** — single SOHO edge (manual port-forward only); ingress VIP via **Cilium L2 announcement**. +- **OQ-1** — single SOHO edge; ingress VIP via **Cilium L2 announcement**. Public web ingress amended + 2026-11-08 to a **Cloudflare Tunnel** (ADR-0016); the edge keeps only udp/51820 and tcp/32400. - **OQ-4** — scoped **Cloudflare API token** drives ExternalDNS (records) and cert-manager DNS-01 (certs) automatically; see [08](08-tls-and-certificates.md). diff --git a/docs/13-service-catalog.md b/docs/13-service-catalog.md index fc5616c..050a509 100644 --- a/docs/13-service-catalog.md +++ b/docs/13-service-catalog.md @@ -19,7 +19,7 @@ kind: container | vm # PaaS (default) or IaaS via KubeVirt (R-PLAT source: image: # container: image ref # or vmTemplate: # vm: CDI source image -exposure: internal | public # LAN-only vs edge port-forwarded (R-DNS/R-NET) +exposure: internal | public # LAN/VPN-only vs Cloudflare Tunnel (R-DNS-04/05, ADR-0016) egress: internet | vpn # egress class (R-NET-05); vpn ⇒ kill-switch (R-NET-06) storage: - name: config @@ -138,8 +138,10 @@ Points the schema left ambiguous, now pinned so the JSON Schema, Kyverno policie - **DR-critical data** (Nextcloud, Gitea, Minecraft worlds, PiHole config, service DBs) MUST have off-cluster backups ([04](04-storage-and-state.md)); bulk media may be `reacquirable` per operator choice (OQ-2). -- **Public exposure** (Plex, Seerr, Nextcloud, Minecraft, Gitea) requires edge port-forwarding - ([09](09-dns-and-ingress.md)); everything else stays `internal`. +- **Public exposure** (Seerr, Nextcloud, Gitea, …) is through the **Cloudflare Tunnel** + ([09](09-dns-and-ingress.md), ADR-0016). Everything else stays `internal`. **Plex** remote + streaming uses plex.tv + an edge forward of tcp/32400 (not the tunnel), and its web UI is LAN-only. + **Minecraft (crafty-mc)** and other L4 services are LAN/VPN-only. - **Kasm** is resource-heavy and may be a candidate for `kind: vm` if container mode proves insufficient — the ServiceDefinition makes that a one-field change. - **Gitea is bootstrapping-critical.** The platform self-hosts its git and CI/CD, so Gitea is brought diff --git a/docs/14-testing-and-conformance.md b/docs/14-testing-and-conformance.md index 2ef5b5c..f8a1307 100644 --- a/docs/14-testing-and-conformance.md +++ b/docs/14-testing-and-conformance.md @@ -82,6 +82,11 @@ Each maps to requirement IDs; phrased as Given/When/Then intent. - **DNS/exposure (R-DNS-*):** *Given* a service, *When* resolved, *Then* the record exists and internal clients hit the internal Gateway; `internal` is unreachable publicly, `public` is reachable + TLS-valid. +- **Public ingress path (R-DNS-04/05, R-SEC-04):** *Given* the tunnel ingress config, *When* it is + rendered, *Then* every route is a `public` service, the last rule is `http_status:404`, and no rule + targets the Kubernetes API (L0). *When* a non-`public` hostname is requested at the tunnel, *Then* + it gets 404. *When* a `public` name is resolved via a public resolver from the LAN, *Then* it is + reachable + TLS-valid (L2). - **Hands-off update (R-UPD-01/02, L3):** *Given* an out-of-date service under a stable policy, *When* the scan interval elapses, *Then* it advances with no human/AI edit; *Given* a deliberately bad update, *Then* health-gating auto-rolls-back and an email alert fires; *Given* a pinned -- 2.54.0