Merge pull request 'ADR-0016 (Proposed): public ingress via Cloudflare Tunnel' (#1) from docs/adr-cloudflare-tunnel-ingress into main
Reviewed-on: jakman85/home-network-k3s#1
This commit was merged in pull request #1.
This commit is contained in:
jakman85 committed
commit
1ddc14b918
12 files changed
+242
-41
No files matched your search
+5
-2
@@ -26,8 +26,11 @@ two things (e.g. "VM"), this file disambiguates.
|
||||
the port level (R-NET-01).
|
||||
- **Egress class** — a service's declared outbound path: `internet` (standard WAN) or `vpn`
|
||||
(commercial VPN tunnel). See [`docs/03-networking-isolation-and-egress.md`](docs/03-networking-isolation-and-egress.md).
|
||||
- **Exposure class** — how a service is reachable: `internal` (LAN-only) or `public` (edge
|
||||
port-forwarded).
|
||||
- **Exposure class** — how a service is reachable: `internal` (LAN/VPN-only) or `public` (published
|
||||
through the Cloudflare Tunnel, ADR-0016).
|
||||
- **Cloudflare Tunnel** — the outbound-only connection from in-cluster cloudflared to Cloudflare that
|
||||
carries all public HTTP(S) ingress; public records are proxied CNAMEs to `<uuid>.cfargotunnel.com`
|
||||
(ADR-0016).
|
||||
- **Kill-switch** — the guarantee that a `vpn`-class workload leaks nothing to the plain internet if
|
||||
the tunnel drops (R-NET-06); fail-closed.
|
||||
- **Multi-WAN** — multiple upstream internet uplinks with routing-rule-based selection; physical WAN
|
||||
|
||||
@@ -53,7 +53,10 @@ screenshot, or "it reconciles" is **not** done (R-TEST-01..04, [`../docs/14-test
|
||||
### DNS & ingress (R-DNS-*)
|
||||
- [ ] `<service>.home.keithjackson.dev` record appears automatically and routes correctly.
|
||||
- [ ] `internal` is unreachable publicly; `public` is reachable + TLS-valid.
|
||||
- [ ] Internal clients resolve to the internal Gateway (no public hairpin).
|
||||
- [ ] Internal clients resolving via Pi-hole reach the internal Gateway; via a public resolver, a
|
||||
`public` service still works through the tunnel (R-DNS-04).
|
||||
- [ ] Non-`public` hostnames get 404 at the tunnel and have no public record (R-DNS-05); the
|
||||
Kubernetes API is not publicly reachable (R-SEC-04).
|
||||
|
||||
### Automated updates (R-UPD-*)
|
||||
- [ ] Advances to a newer **stable** version automatically on schedule with **no human/AI edit**.
|
||||
|
||||
@@ -57,3 +57,4 @@ serves. **Do not edit an accepted decision in place** — supersede it with a ne
|
||||
| [0013](adr-0013-vpn-provider-pia-gluetun-wireguard.md) | VPN provider: PIA over OpenVPN-UDP + port forwarding | Accepted — provider/protocol only (client+steering superseded by 0015) | R-NET-05/06/07 |
|
||||
| [0014](adr-0014-idp-keycloak.md) | Identity Provider: Keycloak (SSO via OIDC + forward-auth) | Accepted | R-IDP-01..06 |
|
||||
| [0015](adr-0015-vpn-egress-steering-node-tunnel-cilium.md) | VPN egress steering: node-level tunnel + Cilium Egress Gateway (host kill-switch) | Accepted | R-NET-05/06/07 |
|
||||
| [0016](adr-0016-public-ingress-cloudflare-tunnel.md) | Public ingress: Cloudflare Tunnel (outbound-only) instead of edge port-forwarding | Proposed | R-DNS-01..05, R-TLS-01/02, R-SEC-02/04 |
|
||||
@@ -0,0 +1,112 @@
|
||||
# ADR-0016 — Public ingress: Cloudflare Tunnel (outbound-only) instead of edge port-forwarding
|
||||
- Status: Proposed (decided by the operator 2026-11-08; set Accepted on merge)
|
||||
- Date: 2026-11-08
|
||||
- Serves: R-DNS-01, R-DNS-02, R-DNS-03, R-DNS-04, R-DNS-05, R-TLS-01, R-TLS-02, R-SEC-02, R-SEC-04
|
||||
- Supersedes: — (amends the **ingress** part of the OQ-1 resolution, "edge does NAT + port-forward
|
||||
only", in [00](../docs/00-vision-and-scope.md), [03](../docs/03-networking-isolation-and-egress.md)
|
||||
and [09](../docs/09-dns-and-ingress.md). [ADR-0002](adr-0002-cni-cilium.md) (Gateway API on Cilium)
|
||||
and [ADR-0006](adr-0006-tls-cert-manager-letsencrypt.md) (cert-manager + Let's Encrypt DNS-01) are
|
||||
unchanged: the Gateway still terminates origin TLS with the LE wildcard.)
|
||||
|
||||
## Context
|
||||
The OQ-1 resolution routed public traffic as: public DNS → WAN IP → **edge router port-forward
|
||||
80/443** → Cilium Gateway VIP. LAN clients were expected to avoid the WAN path through **Pi-hole
|
||||
split-horizon DNS** (R-DNS-03).
|
||||
|
||||
That design fails in this environment:
|
||||
- The ISP router (GFiber) **cannot hairpin NAT**. A LAN client that resolves a service to the WAN IP
|
||||
cannot reach it.
|
||||
- The same router **advertises Google IPv6 DNS servers via Router Advertisement**, and the operator
|
||||
cannot turn that off. LAN clients (Android in particular) therefore use a public resolver
|
||||
alongside, or instead of, Pi-hole. They get the WAN answer and fail.
|
||||
|
||||
So split-horizon DNS cannot be *relied on* to keep LAN clients off the WAN path. Correctness for
|
||||
public web apps has to come from the public answer itself working from the LAN.
|
||||
|
||||
## Decision
|
||||
**Public HTTP(S) ingress goes through a Cloudflare Tunnel.**
|
||||
|
||||
- **cloudflared runs in-cluster** (a Flux-managed workload) and holds an **outbound-only**
|
||||
connection to Cloudflare. No inbound connection to the cluster is needed for public web traffic.
|
||||
- **Public records** for `exposure: public` services are **proxied CNAMEs to
|
||||
`<tunnel-uuid>.cfargotunnel.com`**, managed by **ExternalDNS** from declared service intent
|
||||
(R-DNS-02). No record is hand-edited.
|
||||
- **The edge router's 80/443 forwards are closed** after cutover.
|
||||
- **Hostnames are unchanged:** `<service>.home.keithjackson.dev` and the apex
|
||||
`home.keithjackson.dev` (R-DNS-01, R-C2-01).
|
||||
- **Edge TLS** is Cloudflare's: **Advanced Certificate Manager / Total TLS**, because Universal SSL
|
||||
covers only one subdomain level (`*.keithjackson.dev`) and not `<svc>.home.keithjackson.dev`. The
|
||||
edge certificate is publicly trusted and auto-renewed (R-TLS-01/02).
|
||||
- **Origin TLS stays at the Gateway** with the cert-manager Let's Encrypt wildcard (ADR-0006).
|
||||
cloudflared connects to the Gateway over HTTPS and **validates the origin certificate**, using
|
||||
the service hostname as SNI. It does not skip verification (`noTLSVerify` is not used).
|
||||
- **Only `exposure: public` services have a tunnel route.** The tunnel's ingress rules end in a
|
||||
**catch-all `http_status:404`**, so any other hostname presented at the tunnel is refused
|
||||
(R-DNS-05).
|
||||
- **The Kubernetes API is never reachable through the tunnel**, and is never published by any other
|
||||
path. It is LAN/VPN-only (R-SEC-04).
|
||||
|
||||
**Never published (LAN/VPN-only, `exposure: internal`):** jackett, pihole, semaphore, qbittorrent,
|
||||
radarr, sonarr, **headlamp** (a front end to the Kubernetes API), and the non-HTTP L4 services
|
||||
**unifi** and **crafty-mc** (Minecraft). They get no public DNS record and no tunnel route.
|
||||
|
||||
**Plex stays off the tunnel.** Cloudflare's terms restrict proxying video, so Plex remote streaming
|
||||
keeps using **plex.tv + WAN tcp/32400** (an edge forward that stays). The `plex.home` web UI becomes
|
||||
**LAN-only**: no tunnel route and no proxied record.
|
||||
|
||||
**Unchanged:**
|
||||
- **WireGuard remote access**: udp/51820, reached through the DNS-only (unproxied) `wan.` record,
|
||||
which cloudflare-ddns keeps current. The WAN IP therefore **stays public**. The tunnel does not
|
||||
hide the origin; it only removes 80/443 from the edge.
|
||||
- **Pi-hole keeps serving the LAN**, but split DNS is no longer *required* for public web apps to
|
||||
work. It keeps explicit local records for:
|
||||
- the internal-only names;
|
||||
- the L4 VIPs;
|
||||
- the names that must avoid Cloudflare's **100 MB request-body** and **100 s response-timeout**
|
||||
limits, or must keep working during an internet outage: **gitea**, **nextcloud**, and
|
||||
**keycloak/auth**.
|
||||
- **In-cluster CoreDNS split horizon** and the **node `/etc/hosts` pins** (the OIDC issuer and the
|
||||
Gitea registry) stay. Cluster-internal callers never go through Cloudflare.
|
||||
|
||||
## Alternatives considered
|
||||
- **Keep the port-forward and fix split horizon.** Rejected: the router's RA-advertised Google IPv6
|
||||
DNS cannot be disabled, so some LAN clients will always get the public answer, and the router
|
||||
cannot hairpin it.
|
||||
- **Replace the ISP router (or put a router capable of hairpin NAT and RA control behind it).**
|
||||
Rejected for now: it adds hardware and a non-IaC edge device, which contradicts the "dumb SOHO
|
||||
edge" premise of OQ-1, and the GFiber RA behaviour would still need to be contained. It can be
|
||||
reopened by a future ADR.
|
||||
- **Block or override the Google DNS on the LAN** (firewall DoH/DNS egress, or intercept port 53).
|
||||
Rejected: the SOHO edge has no such controls, and client-side DoH/hard-coded resolvers make it
|
||||
incomplete anyway.
|
||||
- **Tailscale Funnel / ngrok / another reverse-tunnel provider.** Rejected: Cloudflare is already the
|
||||
authoritative DNS and token holder (OQ-4), so its tunnel integrates with ExternalDNS and the
|
||||
existing scoped token with no new vendor.
|
||||
- **Self-hosted reverse tunnel on a VPS** (WireGuard + reverse proxy). Rejected: it adds a host to
|
||||
run, patch and secure, outside the cluster's GitOps model.
|
||||
|
||||
## Consequences
|
||||
- **Positive:** LAN clients reach public web apps correctly whichever resolver answers. No NAT
|
||||
hairpin is needed. Inbound 80/443 is no longer open on the WAN IP. Publishing is an explicit,
|
||||
declared, per-hostname act (tunnel route + record), and anything else 404s at the edge.
|
||||
- **Dependence on Cloudflare.** A Cloudflare or tunnel outage takes public access down. The LAN keeps
|
||||
working through Pi-hole's local records for gitea, nextcloud and keycloak/auth and the
|
||||
internal-only names. Public apps without a local record are unreachable from the LAN during such an
|
||||
outage too. Tunnel health becomes an alert condition (R-OBS-04).
|
||||
- **Cloudflare sees plaintext.** TLS terminates at Cloudflare's edge, so Cloudflare can read the
|
||||
request and response content of every tunnelled service. Re-encryption to the origin protects
|
||||
only the Cloudflare → cluster leg.
|
||||
- **Upload and timeout limits:** 100 MB request bodies and 100 s responses through the proxy. Large
|
||||
uploads (Nextcloud, Gitea pushes and registry blobs) must use the LAN path, which is why those names
|
||||
keep local records.
|
||||
- **Client IP** now arrives as `CF-Connecting-IP` / `X-Forwarded-For` from cloudflared, not as the
|
||||
TCP source. Logging, rate-limiting and any IP-based policy must read those headers, and trust them
|
||||
only from cloudflared.
|
||||
- **The WAN IP stays public** (WireGuard, Plex 32400). The tunnel is not origin cloaking.
|
||||
- **TLS certificates:** two publicly trusted certificates are in play per public name (Cloudflare edge
|
||||
+ LE origin). Both auto-renew. ACM/Total TLS is a paid Cloudflare feature and an operator-held
|
||||
prerequisite.
|
||||
- **Follow-ups (implementation repo):** deploy cloudflared with a SOPS-encrypted tunnel credential;
|
||||
move ExternalDNS public records to proxied CNAMEs for `public` services only; add conformance tests
|
||||
for R-DNS-04/05 and R-SEC-04; alert on tunnel health; then close the 80/443 edge forwards (a
|
||||
documented manual edge step, like the original forwards).
|
||||
@@ -40,7 +40,9 @@ The platform is delivered in two stages:
|
||||
|
||||
## The domain & DNS model
|
||||
|
||||
- Public apex `*.keithjackson.dev` is available and its records resolve to the network's public IP.
|
||||
- Public apex `*.keithjackson.dev` is available in Cloudflare DNS. `public` services under
|
||||
`*.home.keithjackson.dev` are published through a **Cloudflare Tunnel** (proxied records, no inbound
|
||||
80/443 at the edge — ADR-0016); the network's public IP stays in use for WireGuard and Plex only.
|
||||
- All platform services live under `*.home.keithjackson.dev`.
|
||||
- Each service is reachable at `<service>.home.keithjackson.dev` (e.g. `plex.home.keithjackson.dev`).
|
||||
- The **root** `home.keithjackson.dev` serves the **Command & Control (C2) center** UI.
|
||||
@@ -117,6 +119,12 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
|
||||
- **R-DNS-02** — DNS records MUST be managed automatically from declared service intent.
|
||||
- **R-DNS-03** — Internal clients MUST resolve service names correctly (split-horizon where public
|
||||
and internal answers differ).
|
||||
- **R-DNS-04** — `public` HTTP(S) services MUST be reachable from the internet **without inbound
|
||||
80/443 port-forwarding at the edge**, and LAN clients MUST reach them correctly **whichever resolver
|
||||
answers** (public or internal) — no dependency on NAT hairpin (ADR-0016).
|
||||
- **R-DNS-05** — Only services declared `exposure: public` MAY be reachable through the public
|
||||
ingress path. A request for any other hostname at that path MUST be refused (HTTP 404), and
|
||||
`internal` services MUST have no public DNS record (ADR-0016).
|
||||
|
||||
### Observability & alerting — `R-OBS-*`
|
||||
- **R-OBS-01** — Services MUST be monitored (metrics, logs, health).
|
||||
@@ -137,6 +145,8 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
|
||||
- **R-SEC-02** — The platform MUST follow least-privilege for workload identities and network reach.
|
||||
- **R-SEC-03** — All the above properties MUST be demonstrable/auditable (an operator can prove
|
||||
isolation, egress path, cert validity, and backup integrity).
|
||||
- **R-SEC-04** — The Kubernetes API MUST NOT be reachable from the public internet by any path (no
|
||||
tunnel route, no edge forward, no public DNS record); it is LAN/VPN-only (ADR-0016).
|
||||
|
||||
### Identity & access (SSO) — `R-IDP-*`
|
||||
- **R-IDP-01** — The platform MUST provide a central Identity Provider (SSO) as the single user
|
||||
@@ -204,6 +214,8 @@ implementation.
|
||||
- **OQ-1 (WAN / edge)** — **RESOLVED**: one commercial WAN, no multi-ISP plans; dumb SOHO edge (no
|
||||
VLAN/VPN/services). All segmentation + VPN in-cluster; edge does NAT + port-forward only. Physical
|
||||
multi-WAN out of scope. See [`03`](03-networking-isolation-and-egress.md), [`09`](09-dns-and-ingress.md).
|
||||
**Amended 2026-11-08 (ADR-0016):** public web ingress moved to an outbound-only **Cloudflare Tunnel**;
|
||||
the edge no longer forwards 80/443. Its remaining forwards are WireGuard udp/51820 and Plex tcp/32400.
|
||||
- **OQ-2 (bulk media)** — **RESOLVED (deferred backing)**: Plex bulk media lives **outside the
|
||||
cluster**; the specific backing is **TBD later** and does not block the core platform. Plex's
|
||||
metadata DB still lives on replicated Longhorn. The **off-cluster DR backup target** (for app
|
||||
|
||||
@@ -54,8 +54,8 @@ choice here without a superseding ADR.**
|
||||
│ dev/test: libvirt KVMs (OpenTofu + cloud-init + Ansible) │
|
||||
│ prod: 3× mini-PC bare metal, local disks │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Edge: router/firewall — multi-WAN uplinks, port-forward to Gateway, │
|
||||
│ VPN uplink(s), split-horizon DNS │
|
||||
│ Edge: SOHO router — single WAN, NAT; forwards only WireGuard + Plex │
|
||||
│ Public web ingress: Cloudflare Tunnel (outbound-only cloudflared, ADR-0016)│
|
||||
└──────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
|
||||
@@ -82,8 +82,10 @@ as multi-ISP routing but as **two egress paths selected by routing rules**:
|
||||
route across; the `internet` path is simply the default route out.
|
||||
- **The edge router is a dumb SOHO device** — no VLANs, no VPN, no services, no API/IaC. Therefore
|
||||
**all segmentation and all VPN handling live in-cluster** (Cilium policy + the VPN gateway), not on
|
||||
the edge. The edge's only jobs are NAT out and **port-forwarding 80/443 (and specific service
|
||||
ports) to the ingress VIP** for `public` services — a documented manual step ([09](09-dns-and-ingress.md)).
|
||||
the edge. The edge's only jobs are NAT out and a few **manual port-forwards that are not web
|
||||
ingress**: WireGuard udp/51820 and Plex tcp/32400 ([09](09-dns-and-ingress.md)). Public web ingress
|
||||
uses an outbound-only **Cloudflare Tunnel** from in-cluster cloudflared (ADR-0016), so the edge does
|
||||
not forward 80/443.
|
||||
- The **service-facing contract is unchanged**: a service still just declares `egress: internet|vpn`.
|
||||
|
||||
## VPN egress model: node-level tunnel + Cilium Egress Gateway (R-NET-05/06 — ADR-0015; provider ADR-0013)
|
||||
@@ -133,7 +135,8 @@ closed (no packets to the plain internet), while `internet`-class is unaffected.
|
||||
|
||||
North-south exposure is handled by **Gateway API implemented by Cilium**, with TLS from
|
||||
cert-manager and DNS from ExternalDNS. Detailed in [`09-dns-and-ingress.md`](09-dns-and-ingress.md).
|
||||
Exposure is a declared property: `internal` (LAN-only) vs `public` (port-forwarded at the edge).
|
||||
Exposure is a declared property: `internal` (LAN/VPN-only) vs `public` (published through the
|
||||
Cloudflare Tunnel, ADR-0016).
|
||||
|
||||
## Observability of the network (supports R-SEC-03, R-OBS-04)
|
||||
- **Hubble** provides flow visibility to verify default-deny and troubleshoot allow rules.
|
||||
@@ -151,6 +154,8 @@ Exposure is a declared property: `internal` (LAN-only) vs `public` (port-forward
|
||||
## Resolved
|
||||
- **OQ-1** — one commercial WAN, no multi-ISP plans; dumb SOHO edge (no VLAN/VPN/services). All
|
||||
segmentation + VPN in-cluster; edge does NAT + port-forward only. Multi-ISP routing out of scope.
|
||||
Amended 2026-11-08 (ADR-0016): 80/443 are no longer forwarded; public web ingress is a Cloudflare
|
||||
Tunnel.
|
||||
- **OQ-3** — **PIA over OpenVPN-UDP** (ADR-0013); egress is a **node-level host tunnel** steered by **Cilium
|
||||
Egress Gateway** with a host nftables kill-switch (ADR-0015), rules enforced outside the service.
|
||||
|
||||
|
||||
@@ -51,6 +51,7 @@ Defense in depth with declarative, auditable controls:
|
||||
node(s) via Ansible vault** (the tunnel is host infra, not a cluster pod; ADR-0015), NOT a cluster SOPS Secret.
|
||||
- Keycloak admin credential and per-service **OIDC client secrets** ([15](15-identity-and-access.md)).
|
||||
- DNS provider API token (Cloudflare) for ACME DNS-01 + ExternalDNS.
|
||||
- Cloudflare Tunnel credential/token for in-cluster cloudflared (SOPS-encrypted; ADR-0016).
|
||||
- SMTP credentials for Alertmanager email.
|
||||
- Service credentials (Nextcloud, Gitea admin, UniFi, Semaphore, Kasm, DB passwords).
|
||||
- Off-cluster backup target credentials (S3/NFS) for Velero/Longhorn.
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
# 08 — TLS & Certificates
|
||||
|
||||
Serves: **R-TLS-01/02/03**, **R-DNS-01**. Related: [ADR-0006 (cert-manager + Let's Encrypt)](../decisions/adr-0006-tls-cert-manager-letsencrypt.md).
|
||||
Serves: **R-TLS-01/02/03**, **R-DNS-01**. Related: [ADR-0006 (cert-manager + Let's Encrypt)](../decisions/adr-0006-tls-cert-manager-letsencrypt.md),
|
||||
[ADR-0016 (Cloudflare Tunnel public ingress)](../decisions/adr-0016-public-ingress-cloudflare-tunnel.md).
|
||||
|
||||
## Requirement recap
|
||||
|
||||
@@ -30,8 +31,8 @@ services (R-TLS-01/02/03).
|
||||
|
||||
## DNS-01 challenge access (OQ-4 — RESOLVED: scoped Cloudflare API token)
|
||||
|
||||
Cloudflare already has a wildcard `*.keithjackson.dev` → public-IP record, which covers
|
||||
*reachability* but not certificate issuance (ACME DNS-01 must **write `_acme-challenge` TXT records**,
|
||||
Cloudflare already has a wildcard `*.keithjackson.dev` → public-IP record, which does not cover
|
||||
certificate issuance (ACME DNS-01 must **write `_acme-challenge` TXT records**,
|
||||
and that wildcard does not match two-label-deep names like `plex.home.keithjackson.dev`).
|
||||
|
||||
**Decision (OQ-4):** use a **scoped Cloudflare API token** — `Zone·DNS·Edit` limited to the
|
||||
@@ -55,8 +56,15 @@ DNS edit only) so its blast radius is bounded (R-SEC-02). Rotation = re-encrypt
|
||||
## Where TLS terminates
|
||||
|
||||
- **North-south**: at the **Gateway API** listener (implemented by Cilium, [09](09-dns-and-ingress.md)),
|
||||
using the cert-manager-issued cert. Public services are reached via the edge router forwarding to
|
||||
the Gateway; internal services resolve to the Gateway on the LAN.
|
||||
using the cert-manager-issued cert. Internal services resolve to the Gateway on the LAN.
|
||||
- **Public services via the Cloudflare Tunnel (ADR-0016)** are TLS twice:
|
||||
- **Edge:** Cloudflare terminates the client's TLS with an ACM/Total TLS certificate. It is
|
||||
publicly trusted and auto-renewed (R-TLS-01/02). Universal SSL is not enough, because it covers
|
||||
only one subdomain level.
|
||||
- **Origin:** cloudflared re-encrypts to the Gateway and **validates** the Let's Encrypt wildcard,
|
||||
using the service hostname as SNI. Verification is never disabled.
|
||||
|
||||
Cloudflare sees plaintext between the two legs, a trade-off accepted in ADR-0016.
|
||||
- Internal service-to-service traffic relies primarily on network policy for security; mTLS between
|
||||
workloads is **optional** future hardening (could be added via Cilium/service mesh) and is not a
|
||||
launch requirement — record as a possible future ADR.
|
||||
|
||||
+73
-24
@@ -1,14 +1,16 @@
|
||||
# 09 — DNS & Ingress
|
||||
|
||||
Serves: **R-DNS-01/02/03**, **R-C2-01**, works with **R-TLS-***. Related:
|
||||
[ADR-0002 (Cilium/Gateway API)](../decisions/adr-0002-cni-cilium.md).
|
||||
Serves: **R-DNS-01/02/03/04/05**, **R-C2-01**, **R-SEC-04**, works with **R-TLS-***. Related:
|
||||
[ADR-0002 (Cilium/Gateway API)](../decisions/adr-0002-cni-cilium.md),
|
||||
[ADR-0016 (Cloudflare Tunnel public ingress)](../decisions/adr-0016-public-ingress-cloudflare-tunnel.md).
|
||||
|
||||
## Naming model
|
||||
|
||||
- All services live at `<service>.home.keithjackson.dev` (e.g. `plex.home.keithjackson.dev`).
|
||||
- The **root** `home.keithjackson.dev` serves the **Command & Control UI** ([12](12-command-center-ui.md)).
|
||||
- Public apex `*.keithjackson.dev` resolves to the network's public IP; `*.home.keithjackson.dev`
|
||||
is the platform's zone.
|
||||
- `*.home.keithjackson.dev` is the platform's zone, in Cloudflare DNS. `public` services are
|
||||
published as **proxied CNAMEs to the Cloudflare Tunnel** (`<tunnel-uuid>.cfargotunnel.com`,
|
||||
ADR-0016), not as records pointing at the network's public IP.
|
||||
|
||||
## DNS automation (R-DNS-02)
|
||||
|
||||
@@ -16,16 +18,32 @@ Serves: **R-DNS-01/02/03**, **R-C2-01**, works with **R-TLS-***. Related:
|
||||
authoritative DNS provider (Cloudflare, per OQ-4) automatically — a new service's DNS appears from
|
||||
its declared intent, no manual record editing.
|
||||
- Records for `*.home.keithjackson.dev` are created/updated/removed as services come and go.
|
||||
- Public records are created **only for `exposure: public` services** (R-DNS-05). The DNS-only
|
||||
(unproxied) `wan.` record for WireGuard is kept current by cloudflare-ddns, outside this flow.
|
||||
|
||||
## Split-horizon resolution (R-DNS-03)
|
||||
|
||||
Internal and external answers differ:
|
||||
- **External** (`*.home.keithjackson.dev` from the internet): resolves to the public IP; reaches
|
||||
the edge router, which forwards `public`-class services to the in-cluster Gateway.
|
||||
- **Internal** (LAN clients): resolves `<service>.home.keithjackson.dev` to the **internal Gateway
|
||||
address** directly, avoiding hairpinning through the public IP.
|
||||
- **PiHole** (itself an onboarded service) provides internal LAN DNS and the split-horizon answers
|
||||
for `.home.keithjackson.dev`; the trusted Let's Encrypt cert is served either way ([08](08-tls-and-certificates.md)).
|
||||
- **External** (`<service>.home.keithjackson.dev` from the internet): a `public` service resolves to
|
||||
**Cloudflare's edge** (proxied CNAME to the tunnel). Cloudflare forwards the request down the
|
||||
outbound-only tunnel to cloudflared in the cluster, which forwards it to the Gateway (ADR-0016).
|
||||
`internal` services have no public answer.
|
||||
- **Internal** (LAN clients): Pi-hole answers with the **internal Gateway address** (or the service's
|
||||
L4 VIP) for the names it holds local records for.
|
||||
- **Correctness does not depend on split horizon for `public` web apps** (R-DNS-04). The ISP router
|
||||
cannot hairpin NAT and advertises a public IPv6 resolver via RA, so some LAN clients will get the
|
||||
public answer. That answer goes through Cloudflare and works from the LAN too.
|
||||
- **PiHole** (itself an onboarded service) provides internal LAN DNS. It keeps explicit local records
|
||||
for:
|
||||
- the `internal` names;
|
||||
- the L4 VIPs;
|
||||
- the names that must avoid Cloudflare's 100 MB body / 100 s timeout limits or must work during an
|
||||
internet outage: **gitea**, **nextcloud**, **keycloak/auth**.
|
||||
|
||||
A trusted certificate is served either way: Cloudflare's edge certificate on the tunnel path, the
|
||||
Let's Encrypt wildcard on the LAN path ([08](08-tls-and-certificates.md)).
|
||||
- **In-cluster** resolution keeps its own split horizon (CoreDNS) and the node `/etc/hosts` pins (OIDC
|
||||
issuer, Gitea registry), so cluster-internal callers never traverse Cloudflare.
|
||||
|
||||
> DNS is thus both a platform capability *and* a hosted service. Bootstrapping order matters: the
|
||||
> platform needs working DNS before PiHole is fully managed — handled by the bring-up sequence in
|
||||
@@ -40,19 +58,41 @@ Internal and external answers differ:
|
||||
- Each service contributes an **HTTPRoute** mapping its hostname to its Service — rendered from the
|
||||
ServiceDefinition, not hand-authored.
|
||||
- **Exposure class** is declared per service:
|
||||
- `internal` — reachable only on the LAN (no edge port-forward; internal DNS only).
|
||||
- `public` — additionally reachable from the internet via the edge router forwarding 80/443 to the
|
||||
Gateway.
|
||||
- `internal` — reachable only on the LAN/VPN (no tunnel route, no public record; internal DNS only).
|
||||
- `public` — additionally reachable from the internet through the **Cloudflare Tunnel**:
|
||||
cloudflared → the Gateway over HTTPS (ADR-0016).
|
||||
- **Never `public`:** jackett, pihole, semaphore, qbittorrent, radarr, sonarr, **headlamp** (a front
|
||||
end to the Kubernetes API), plex's web UI, and the non-HTTP L4 services unifi and crafty-mc.
|
||||
- **The Kubernetes API is never published** — no tunnel route, no edge forward, no public record
|
||||
(R-SEC-04).
|
||||
|
||||
## Public ingress: Cloudflare Tunnel (ADR-0016)
|
||||
|
||||
- **cloudflared** runs in-cluster as a Flux-managed workload and holds an **outbound-only**
|
||||
connection to Cloudflare. No inbound 80/443 reaches the WAN IP.
|
||||
- Its ingress rules list one route per `public` hostname → the Gateway (HTTPS, origin certificate
|
||||
validated with the hostname as SNI). The rules end in a **catch-all `http_status:404`**, so an
|
||||
unlisted hostname is refused at the tunnel (R-DNS-05).
|
||||
- **Edge TLS** is Cloudflare ACM/Total TLS (Universal SSL does not cover two-label-deep names);
|
||||
**origin TLS** is the Let's Encrypt wildcard at the Gateway ([08](08-tls-and-certificates.md)).
|
||||
- **Limits and trade-offs:**
|
||||
- 100 MB request bodies and 100 s responses through the proxy;
|
||||
- Cloudflare terminates TLS and sees plaintext;
|
||||
- a Cloudflare outage takes public access down (the LAN keeps working through Pi-hole's local
|
||||
records);
|
||||
- the client IP arrives as `CF-Connecting-IP` / `X-Forwarded-For`, not as the TCP source.
|
||||
|
||||
## Edge integration (single SOHO router — OQ-1 resolved)
|
||||
|
||||
- The environment has **one SOHO edge router**: no VLANs, no VPN, no services, no API/IaC. Its only
|
||||
ingress job is **manual port-forwarding of 80/443 (and specific service ports, e.g. Plex 32400,
|
||||
Minecraft 25565) to the Gateway VIP** for `public` services. This is a documented manual edge step
|
||||
(the SOHO device has no automation) — the analogue of the old pfSense manual-config exception.
|
||||
Confirm the router can port-forward (basic NAT; almost all can).
|
||||
- The environment has **one SOHO edge router**: no VLANs, no VPN, no services, no API/IaC. It
|
||||
**does not forward 80/443** (public web ingress is the tunnel, ADR-0016). Its only remaining
|
||||
inbound forwards are **WireGuard udp/51820** (remote access, via the DNS-only `wan.` record) and
|
||||
**Plex tcp/32400** (remote streaming via plex.tv; Cloudflare's terms restrict proxied video). This
|
||||
is a documented manual edge step (the SOHO device has no automation), the analogue of the old
|
||||
pfSense manual-config exception. The WAN IP therefore stays public.
|
||||
- No multi-WAN inbound rules — single WAN ([03](03-networking-isolation-and-egress.md)).
|
||||
- `internal` services are never port-forwarded; they rely on internal DNS + Gateway on the LAN.
|
||||
- `internal` services are never port-forwarded or tunnelled; they rely on internal DNS + Gateway (or
|
||||
their L4 VIP) on the LAN, or WireGuard from outside.
|
||||
- **Ingress VIP mechanism — resolved: Cilium L2 announcement (ARP/NDP)** on the flat LAN. BGP is not
|
||||
available on a SOHO router, and a single flat subnet makes L2 announcement the natural choice.
|
||||
|
||||
@@ -62,9 +102,10 @@ A wildcard `*.keithjackson.dev` matches only **one** label deep — it covers `h
|
||||
but **not** `plex.home.keithjackson.dev`. Consequences:
|
||||
- **`internal` services need no public record** — PiHole resolves `<svc>.home.keithjackson.dev` to
|
||||
the internal Gateway VIP (split-horizon). This is the majority of services.
|
||||
- **`public` services** need a public record for `<svc>.home.keithjackson.dev`. Per OQ-4 (resolved),
|
||||
**ExternalDNS manages these automatically** via the scoped Cloudflare API token — no manual record
|
||||
editing (see [08](08-tls-and-certificates.md)).
|
||||
- **`public` services** need a public record for `<svc>.home.keithjackson.dev`: a **proxied CNAME to
|
||||
the tunnel** (ADR-0016). Per OQ-4 (resolved), **ExternalDNS manages these automatically** via the
|
||||
scoped Cloudflare API token — no manual record editing (see [08](08-tls-and-certificates.md)). The
|
||||
same depth limit is why edge TLS needs ACM/Total TLS rather than Universal SSL.
|
||||
|
||||
## Interaction with default-deny
|
||||
|
||||
@@ -76,10 +117,18 @@ not let other in-cluster services reach it unless an explicit allow rule exists.
|
||||
- A new service's `<service>.home.keithjackson.dev` record appears automatically and routes to it.
|
||||
- `internal` services are unreachable from the public internet; `public` services are reachable and
|
||||
TLS-valid.
|
||||
- Internal clients resolve to the internal Gateway (no public hairpin) and get a trusted cert.
|
||||
- Internal clients resolving through Pi-hole reach the internal Gateway for the names it holds local
|
||||
records for, and get a trusted cert.
|
||||
- A LAN client resolving a `public` name through a **public** resolver reaches it, TLS-valid
|
||||
(R-DNS-04). The WAN IP does not answer on 80/443.
|
||||
- A request at the tunnel for any hostname that is not a `public` service returns **404**, and no
|
||||
`internal` service has a public DNS record (R-DNS-05).
|
||||
- The Kubernetes API is unreachable from the internet: no tunnel route, no edge forward, no public
|
||||
record (R-SEC-04).
|
||||
- The root `home.keithjackson.dev` serves the C2 UI.
|
||||
|
||||
## Resolved
|
||||
- **OQ-1** — single SOHO edge (manual port-forward only); ingress VIP via **Cilium L2 announcement**.
|
||||
- **OQ-1** — single SOHO edge; ingress VIP via **Cilium L2 announcement**. Public web ingress amended
|
||||
2026-11-08 to a **Cloudflare Tunnel** (ADR-0016); the edge keeps only udp/51820 and tcp/32400.
|
||||
- **OQ-4** — scoped **Cloudflare API token** drives ExternalDNS (records) and cert-manager DNS-01
|
||||
(certs) automatically; see [08](08-tls-and-certificates.md).
|
||||
@@ -19,7 +19,7 @@ kind: container | vm # PaaS (default) or IaaS via KubeVirt (R-PLAT
|
||||
source:
|
||||
image: <repo/image> # container: image ref
|
||||
# or vmTemplate: <image/source> # vm: CDI source image
|
||||
exposure: internal | public # LAN-only vs edge port-forwarded (R-DNS/R-NET)
|
||||
exposure: internal | public # LAN/VPN-only vs Cloudflare Tunnel (R-DNS-04/05, ADR-0016)
|
||||
egress: internet | vpn # egress class (R-NET-05); vpn ⇒ kill-switch (R-NET-06)
|
||||
storage:
|
||||
- name: config
|
||||
@@ -138,8 +138,10 @@ Points the schema left ambiguous, now pinned so the JSON Schema, Kyverno policie
|
||||
- **DR-critical data** (Nextcloud, Gitea, Minecraft worlds, PiHole config, service DBs) MUST have
|
||||
off-cluster backups ([04](04-storage-and-state.md)); bulk media may be `reacquirable` per operator
|
||||
choice (OQ-2).
|
||||
- **Public exposure** (Plex, Seerr, Nextcloud, Minecraft, Gitea) requires edge port-forwarding
|
||||
([09](09-dns-and-ingress.md)); everything else stays `internal`.
|
||||
- **Public exposure** (Seerr, Nextcloud, Gitea, …) is through the **Cloudflare Tunnel**
|
||||
([09](09-dns-and-ingress.md), ADR-0016). Everything else stays `internal`. **Plex** remote
|
||||
streaming uses plex.tv + an edge forward of tcp/32400 (not the tunnel), and its web UI is LAN-only.
|
||||
**Minecraft (crafty-mc)** and other L4 services are LAN/VPN-only.
|
||||
- **Kasm** is resource-heavy and may be a candidate for `kind: vm` if container mode proves
|
||||
insufficient — the ServiceDefinition makes that a one-field change.
|
||||
- **Gitea is bootstrapping-critical.** The platform self-hosts its git and CI/CD, so Gitea is brought
|
||||
|
||||
@@ -82,6 +82,11 @@ Each maps to requirement IDs; phrased as Given/When/Then intent.
|
||||
- **DNS/exposure (R-DNS-*):** *Given* a service, *When* resolved, *Then* the record exists and
|
||||
internal clients hit the internal Gateway; `internal` is unreachable publicly, `public` is
|
||||
reachable + TLS-valid.
|
||||
- **Public ingress path (R-DNS-04/05, R-SEC-04):** *Given* the tunnel ingress config, *When* it is
|
||||
rendered, *Then* every route is a `public` service, the last rule is `http_status:404`, and no rule
|
||||
targets the Kubernetes API (L0). *When* a non-`public` hostname is requested at the tunnel, *Then*
|
||||
it gets 404. *When* a `public` name is resolved via a public resolver from the LAN, *Then* it is
|
||||
reachable + TLS-valid (L2).
|
||||
- **Hands-off update (R-UPD-01/02, L3):** *Given* an out-of-date service under a stable policy,
|
||||
*When* the scan interval elapses, *Then* it advances with no human/AI edit; *Given* a deliberately
|
||||
bad update, *Then* health-gating auto-rolls-back and an email alert fires; *Given* a pinned
|
||||
|
||||
Reference in new issue
Block a user