feat(dns): publish public names as proxied CNAMEs to the Cloudflare Tunnel
Gateway target -> fc81938b-...cfargotunnel.com, --cloudflare-proxied globally, L4 service publishers (unifi, crafty-mc) pinned DNS-only, Plex web UI and Headlamp disowned (LAN/VPN-only). L0 internal-services-publish-no-public-dns gains assertion (9) tying the Gateway target to the tunnel UUID, requiring cloudflare-proxied=false on every service-source publisher, and keeping the tunnel's 404 list and the unpublished set consistent. L2 dns-exposure fixture opts out of proxying. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5.5 committed
1 parent
2e777e13ca
commit
92944486d1
16 files changed
+417
-88
No files matched your search
@@ -101,6 +101,11 @@ metadata:
|
||||
lbipam.cilium.io/ips: "${CRAFTY_L4_VIP}"
|
||||
# external-dns publishes A crafty-mc.home.keithjackson.dev -> the pinned VIP (Java server target).
|
||||
external-dns.alpha.kubernetes.io/hostname: "crafty-mc.${BASE_DOMAIN}"
|
||||
# DNS-ONLY, NEVER PROXIED (2026-11-08). external-dns runs with --cloudflare-proxied globally (the
|
||||
# gateway names point at the Cloudflare Tunnel); Cloudflare's proxy carries HTTP(S) only, so an
|
||||
# orange-clouded A for these L4 ports would be broken. Required by the L0 check
|
||||
# internal-services-publish-no-public-dns (9b) on every `service`-source publisher.
|
||||
external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Cluster
|
||||
|
||||
@@ -20,29 +20,44 @@ A wildcard `*.keithjackson.dev` matches only one label deep, so `<svc>.home.keit
|
||||
its own record. **`internal` services need NO public record** — PiHole resolves them to the internal
|
||||
Gateway VIP (split-horizon). ExternalDNS manages **`public`** services' records only.
|
||||
|
||||
## Public target + in-cluster DDNS (2026-10-03)
|
||||
## Public target: the Cloudflare Tunnel, proxied (2026-11-08)
|
||||
|
||||
Gateway-published records are **CNAMEs to `wan.home.keithjackson.dev`**, not the Gateway VIP:
|
||||
Gateway-published records are **proxied CNAMEs to the Cloudflare Tunnel**
|
||||
(`platform/cloudflared`), not the WAN IP and not the Gateway VIP:
|
||||
|
||||
```
|
||||
plex.home.keithjackson.dev CNAME wan.home.keithjackson.dev <- external-dns (Gateway `target` annotation)
|
||||
wan.home.keithjackson.dev A <current WAN IP> <- cloudflare-ddns (ddns.yaml), every 5m
|
||||
gitea.home.keithjackson.dev CNAME (proxied) fc81938b-5855-47f1-8a27-6286f56dda93.cfargotunnel.com <- external-dns
|
||||
unifi.home.keithjackson.dev A (DNS-only) 192.168.2.21 <- external-dns, `service` source (L4)
|
||||
crafty-mc.home... A (DNS-only) 192.168.2.22 <- external-dns, `service` source (L4)
|
||||
wan.home.keithjackson.dev A (DNS-only) <current WAN IP> <- cloudflare-ddns (ddns.yaml): WireGuard udp/51820 ONLY
|
||||
```
|
||||
|
||||
- The `target` annotation is on the **shared Gateway** (`platform/gateway/gateway.yaml`), so it
|
||||
applies to every route that publishes at all. `internal` routes (`controller: none`) still publish
|
||||
nothing.
|
||||
- `ddns.yaml` runs `favonia/cloudflare-ddns` in this namespace with the same `cloudflare-api-token`.
|
||||
It detects the WAN IP through Cloudflare's trace endpoint, so the pod **must not** carry
|
||||
`egress-class: vpn`.
|
||||
- The `target` annotation is on the **shared Gateway** (`platform/gateway/gateway.yaml`); external-dns
|
||||
reads it from the Gateway only, so it applies to every route that publishes at all. Its UUID must
|
||||
equal `tunnel:` in `platform/cloudflared/config.yaml` (L0 check
|
||||
`internal-services-publish-no-public-dns` (9a)).
|
||||
- `--cloudflare-proxied` (HelmRelease `extraArgs`) orange-clouds every record by default — the only
|
||||
practical lever, because v0.23.0 reads the per-object `cloudflare-proxied` annotation from the
|
||||
route/Service, never the Gateway. **Every `service`-source publisher must carry
|
||||
`external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"`** (L4 ports cannot ride the HTTP
|
||||
proxy); (9b) fails the gate otherwise.
|
||||
- `internal` routes (`controller: none`) still publish nothing. Plex's web UI and Headlamp joined that
|
||||
set on 2026-11-08 (LAN/VPN-only); the tunnel also 404s every such name (9c/9d).
|
||||
- Public answers are Cloudflare anycast **A and AAAA** — Cloudflare's edge, not an origin address.
|
||||
- LAN and pods resolve `*.home.keithjackson.dev` to the Gateway VIP via PiHole and CoreDNS
|
||||
(split-horizon) and never use the public path.
|
||||
- **Nodes resolve through public DNS** (`node_baseline_dns_servers`), so from a node, the apiserver's
|
||||
OIDC issuer `keycloak.home.keithjackson.dev` resolves to the WAN IP. kubectl SSO therefore relies on
|
||||
the router hairpinning (NAT reflection) WAN:443 back to the Gateway.
|
||||
- **Nodes resolve through public DNS** (`node_baseline_dns_servers`), so from a node the apiserver's
|
||||
OIDC issuer `keycloak.home.keithjackson.dev` now resolves to Cloudflare and its discovery/JWKS
|
||||
fetches go through the tunnel (publicly trusted edge cert; no `oidc-ca-file` is configured). The
|
||||
router NAT hairpin is no longer on that path.
|
||||
|
||||
Check: `dig +short @1.1.1.1 plex.home.keithjackson.dev` → `wan.home.keithjackson.dev.` then the WAN IP;
|
||||
`kubectl -n external-dns logs deploy/cloudflare-ddns`.
|
||||
Check: `dig +short @1.1.1.1 gitea.home.keithjackson.dev` → Cloudflare anycast addresses (no CNAME is
|
||||
visible for a proxied record); `kubectl -n external-dns logs deploy/external-dns`.
|
||||
|
||||
### History
|
||||
|
||||
2026-10-03 → 2026-11-08: records were DNS-only CNAMEs to `wan.home.keithjackson.dev`, with the router
|
||||
forwarding WAN tcp/80+443 to the Gateway VIP. Before 2026-10-03 they answered the Gateway VIP itself.
|
||||
|
||||
## Secrets
|
||||
|
||||
|
||||
@@ -2,9 +2,15 @@
|
||||
# In-cluster dynamic DNS for the home WAN (2026-10-03).
|
||||
#
|
||||
# Keeps ONE record, `wan.${BASE_DOMAIN}` (A, IPv4 only, not proxied), pointed at the home WAN's
|
||||
# current public IP. Every public name external-dns publishes from the shared Gateway is a CNAME to
|
||||
# it (platform/gateway/gateway.yaml `external-dns.alpha.kubernetes.io/target`), so a WAN IP change
|
||||
# moves all of them through this one record, with no Git change and no external-dns involvement.
|
||||
# current public IP.
|
||||
#
|
||||
# WIREGUARD-ONLY SINCE 2026-11-08. From 2026-10-03 every public name external-dns published from the
|
||||
# shared Gateway was a CNAME to this record. They are now proxied CNAMEs to the Cloudflare Tunnel
|
||||
# (platform/gateway/gateway.yaml `external-dns.alpha.kubernetes.io/target`), so NOTHING in the zone
|
||||
# points here any more. The record stays for exactly one consumer: the WireGuard VPN endpoint
|
||||
# (udp/51820, platform/vpn-ingress, infra/vpn-ingress/new-peer.sh), which cannot ride Cloudflare's
|
||||
# HTTP proxy and needs the real WAN address. It must stay DNS-only (`PROXIED` unset/false below) —
|
||||
# an orange-clouded A record would hand peers a Cloudflare anycast address that drops UDP.
|
||||
#
|
||||
# OWNERSHIP SPLIT WITH EXTERNAL-DNS. This record carries no external-dns TXT registry entry, so
|
||||
# external-dns (policy: sync, owner `prod`) never treats it as its own and never deletes it.
|
||||
@@ -64,7 +70,7 @@ spec:
|
||||
- name: IP6_PROVIDER
|
||||
value: "none" # IPv4 WAN only; leaves any AAAA alone
|
||||
- name: PROXIED
|
||||
value: "false" # DNS-only: Plex/Gitea/etc. terminate TLS on the Gateway
|
||||
value: "false" # DNS-only: WireGuard udp/51820 needs the real WAN IP
|
||||
- name: UPDATE_CRON
|
||||
value: "@every 5m"
|
||||
- name: DELETE_ON_STOP
|
||||
|
||||
@@ -107,29 +107,47 @@ spec:
|
||||
# what this instance can see or own; per-resource exclusion only ever REMOVES an object from
|
||||
# this instance's view, so it cannot make prod capable of deleting a record it does not own.
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# THE PUBLIC TARGET — DECIDED 2026-10-03 (operator): every Gateway-published name goes public
|
||||
# THE PUBLIC TARGET — THE CLOUDFLARE TUNNEL, PROXIED (2026-11-08; supersedes 2026-10-03)
|
||||
#
|
||||
# Until 2026-10-03 every gateway-sourced record answered with the Gateway VIP (192.168.2.20,
|
||||
# RFC1918), so nothing published here was reachable from the internet. Two facts constrained the
|
||||
# fix, and both still hold:
|
||||
# * `external-dns.alpha.kubernetes.io/target` is read in v0.21.0's gateway-httproute source ONLY
|
||||
# from the GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
|
||||
# gw.Annotations)`) and from a ListenerSet listener — NEVER from an HTTPRoute. A target on the
|
||||
# shared Gateway therefore moves EVERY gateway-sourced record at once.
|
||||
# * The home WAN IP is residential and can change, so a literal IP in Git would rot.
|
||||
# History: until 2026-10-03 every gateway-sourced record answered the Gateway VIP (RFC1918);
|
||||
# from 2026-10-03 they were CNAMEs to `wan.${BASE_DOMAIN}` (DNS-only, the home WAN IP, with the
|
||||
# router forwarding tcp/80+443 to the VIP). Both published an origin address of ours. Now:
|
||||
# * platform/gateway/gateway.yaml sets `target: <tunnel-uuid>.cfargotunnel.com` on the shared
|
||||
# Gateway. `target` is read in v0.23.0's gateway-httproute source ONLY from the GATEWAY
|
||||
# (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(gw.Annotations)`),
|
||||
# never from an HTTPRoute, so this one value moves every gateway-sourced record at once.
|
||||
# * `--cloudflare-proxied` below orange-clouds every record by default. It is GLOBAL because
|
||||
# that is the only practical lever: v0.23.0 reads the per-object `cloudflare-proxied`
|
||||
# annotation from the ROUTE (or Service), never the Gateway (source/gateway.go:
|
||||
# `annotations.ProviderSpecificAnnotations(annots)` with `annots := meta.Annotations` of the
|
||||
# route), so a Gateway-level opt-in does not exist. A CNAME to cfargotunnel.com MUST be
|
||||
# proxied — DNS-only it resolves to nothing routable — so "proxied by default" is the right
|
||||
# default for this zone's gateway names.
|
||||
# * The provider honours a per-object `"false"` over the global default (provider/cloudflare
|
||||
# shouldBeProxied: ParseBool of the annotation overrides proxiedByDefault). The key it reads
|
||||
# is `<annotationPrefix>cloudflare-proxied` (annotations.SetAnnotationPrefix re-derives
|
||||
# CloudflarePrefix), i.e. `external-dns.alpha.kubernetes.io/cloudflare-proxied` here.
|
||||
#
|
||||
# THE `service` SOURCE MUST OPT OUT. The L4 LoadBalancers (crafty-mc, unifi) publish an A record
|
||||
# at their own LB VIP; Cloudflare's proxy is HTTP(S)-only, so a proxied A for a game or UniFi
|
||||
# inform port is broken by construction. Each carries `cloudflare-proxied: "false"` and stays
|
||||
# DNS-only, exactly as before this change. conformance/tests/l0-static/
|
||||
# internal-services-publish-no-public-dns (assertion 9b) FAILS on any `service`-source publisher
|
||||
# without it, so a new L4 Service cannot silently become a proxied record.
|
||||
#
|
||||
# The operator's answers: (a) yes, ALL of them — the five `exposure: public` apps AND the admin
|
||||
# UIs (auth, crafty, grafana, headlamp, the apex), which sit behind Keycloak / oauth2-proxy;
|
||||
# (b) the address is kept current IN-CLUSTER. Implementation:
|
||||
# * platform/gateway/gateway.yaml sets `target: wan.${BASE_DOMAIN}` on the shared Gateway, so
|
||||
# each published name becomes a CNAME to that one hostname;
|
||||
# * platform/externaldns/ddns.yaml (favonia/cloudflare-ddns) keeps `wan.${BASE_DOMAIN}` an A
|
||||
# record at the current WAN IP. That record has no external-dns TXT registry entry, so this
|
||||
# instance never owns or garbage-collects it.
|
||||
# Internal services are unchanged: their routes carry `controller: none` (above), which is
|
||||
# checked BEFORE the target is read, so they still publish nothing. The `service` source (the L4
|
||||
# LoadBalancers crafty-mc, unifi) does not read the Gateway's annotation and still publishes its
|
||||
# LB VIP. Whether those should go public is a separate, still-open question.
|
||||
# checked BEFORE the target is read, so they still publish nothing. Plex's web UI and Headlamp
|
||||
# joined that set on 2026-11-08 (operator: LAN/VPN-only). `wan.${BASE_DOMAIN}` (ddns.yaml) has no
|
||||
# external-dns TXT registry entry and is untouched; it now serves only WireGuard udp/51820.
|
||||
#
|
||||
# THE RISK THIS ADDS, AND WHAT GUARDS IT. If the annotation prefix ever stops being honoured
|
||||
# again (the 2026-11-07 v0.23.0 incident), `target` is ignored and every name falls back to the
|
||||
# Gateway's own address — and with this flag it would be published as a PROXIED A record at
|
||||
# ${GATEWAY_VIP} (RFC1918; Cloudflare 1003/522s for every public name) while `controller: none`
|
||||
# is also ignored and internal names publish. Assertion (8) of the L0 check above ties the
|
||||
# prefix this tree writes to `values.annotationPrefix`; (8b) forbids overriding it here.
|
||||
extraArgs:
|
||||
- --cloudflare-proxied
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below.
|
||||
#
|
||||
|
||||
@@ -30,5 +30,9 @@ east-west default-deny (docs/09).
|
||||
|
||||
- **Phase 2 (WS-NET):** pin the L2-announced ingress VIP / IP pool (Cilium L2 announcement, docs/09);
|
||||
add the HTTP→HTTPS redirect route; confirm the cross-namespace ReferenceGrant pattern with WS-APPS.
|
||||
- **Public exposure** is a **manual edge port-forward** of 80/443 to the Gateway VIP (single SOHO
|
||||
router, docs/09) — a documented manual step, the analogue of the old pfSense exception.
|
||||
- **Public exposure** is via the **Cloudflare Tunnel** (2026-11-08, `platform/cloudflared`): the
|
||||
Gateway's `external-dns.alpha.kubernetes.io/target` is `<tunnel-uuid>.cfargotunnel.com`, so every
|
||||
published name is a proxied CNAME to the tunnel and cloudflared forwards to this Gateway's
|
||||
ClusterIP. No inbound WAN 80/443 forward is needed (the old manual edge port-forward is retired —
|
||||
runbooks/cloudflare-tunnel.md). LAN/VPN-only names (internal apps, plex, headlamp) carry
|
||||
`controller: none` on their route and are 404'd by the tunnel.
|
||||
@@ -8,22 +8,34 @@ metadata:
|
||||
name: shared
|
||||
namespace: gateway
|
||||
annotations:
|
||||
# PUBLIC DNS TARGET (2026-10-03, operator decision). Without this, external-dns publishes every
|
||||
# route on this Gateway with the Gateway's OWN address (${GATEWAY_VIP}, RFC1918), which is
|
||||
# unreachable from the internet. With it, every published `<svc>.${BASE_DOMAIN}` (and the apex)
|
||||
# becomes a CNAME to `wan.${BASE_DOMAIN}`, an A record that platform/externaldns/ddns.yaml keeps
|
||||
# pointed at the home WAN's current public IP. The WAN forwards 80/443 to ${GATEWAY_VIP}.
|
||||
# PUBLIC DNS TARGET = THE CLOUDFLARE TUNNEL (2026-11-08, operator decision; replaces the
|
||||
# 2026-10-03 `wan.${BASE_DOMAIN}` target). external-dns v0.23.0's gateway-httproute source reads
|
||||
# `target` from the GATEWAY only (source/gateway.go: `overrides: annotations.
|
||||
# TargetsFromTargetAnnotation(gw.Annotations)`) — never from an HTTPRoute — so this one value moves
|
||||
# EVERY published route on this Gateway at once. Each published `<svc>.${BASE_DOMAIN}` (and the
|
||||
# apex) becomes a CNAME to `<tunnel-uuid>.cfargotunnel.com`, and platform/externaldns/helmrelease.yaml
|
||||
# sets `--cloudflare-proxied` so the CNAME is orange-clouded: public clients resolve Cloudflare
|
||||
# anycast, Cloudflare terminates edge TLS (Total TLS) and hands the request to the in-cluster
|
||||
# cloudflared connectors (platform/cloudflared), which forward to this Gateway's ClusterIP. No
|
||||
# origin address of ours appears in public DNS, and no inbound WAN port is needed for HTTP(S).
|
||||
#
|
||||
# Scope is EVERY route on this Gateway that publishes at all — the operator chose to make the
|
||||
# admin UIs (auth, grafana, headlamp, crafty, apex) publicly resolvable alongside the five
|
||||
# `exposure: public` apps. `internal` services stay unpublished via their
|
||||
# `external-dns.alpha.kubernetes.io/controller: none` route annotation, which external-dns checks
|
||||
# BEFORE it reads this target. external-dns v0.21.0 reads `target` from the Gateway only, never
|
||||
# from an HTTPRoute — see platform/externaldns/helmrelease.yaml.
|
||||
# THE UUID IS A LITERAL, AND IT MUST EQUAL `tunnel:` IN platform/cloudflared/config.yaml. A CNAME
|
||||
# to any other <uuid>.cfargotunnel.com is a dead name (Cloudflare answers 1016). That equality is
|
||||
# asserted on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns
|
||||
# (assertion 9a). A literal rather than a cluster-var keeps this change off clusters/**.
|
||||
#
|
||||
# Scope: every route on this Gateway that publishes at all. `internal` services (and the
|
||||
# operator-designated LAN/VPN-only names: plex, headlamp) stay unpublished via their
|
||||
# `external-dns.alpha.kubernetes.io/controller: none` route annotation, which external-dns
|
||||
# applies BEFORE it reads this target. Those same names map to `http_status:404` in the tunnel's
|
||||
# ingress rules, so even a hand-made record cannot reach them through the tunnel.
|
||||
#
|
||||
# LAN and in-cluster clients are unaffected: PiHole and CoreDNS answer ${BASE_DOMAIN} with
|
||||
# ${GATEWAY_VIP} directly (split-horizon), so they never take this public path.
|
||||
external-dns.alpha.kubernetes.io/target: "wan.${BASE_DOMAIN}"
|
||||
#
|
||||
# `wan.${BASE_DOMAIN}` (platform/externaldns/ddns.yaml) is NO LONGER a target of anything here;
|
||||
# it stays only as the WireGuard udp/51820 endpoint.
|
||||
external-dns.alpha.kubernetes.io/target: "fc81938b-5855-47f1-8a27-6286f56dda93.cfargotunnel.com"
|
||||
spec:
|
||||
gatewayClassName: cilium
|
||||
listeners:
|
||||
|
||||
@@ -18,6 +18,13 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: oidc
|
||||
homelab.keithjackson.dev/auth-groups: platform-admins
|
||||
# LAN/VPN-ONLY (operator decision 2026-11-08). Disowned from external-dns, so the public
|
||||
# headlamp record and its TXT are DELETED and nothing republishes them; LAN/VPN clients keep
|
||||
# resolving the name via PiHole's split-horizon wildcard (-> the Gateway VIP). The Cloudflare
|
||||
# Tunnel also maps headlamp.${BASE_DOMAIN} to http_status:404 (platform/cloudflared/config.yaml).
|
||||
# Same mechanism the renderer emits for `exposure: internal` services; asserted by
|
||||
# conformance/tests/l0-static/internal-services-publish-no-public-dns (9c).
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
Reference in new issue
Block a user