From a1e808e697f36b589af00f68f100f03e6df272dc Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 3 Oct 2026 16:59:53 -0500 Subject: [PATCH] feat(dns): publish Gateway names to the WAN via in-cluster DDNS Gateway-published records now CNAME to wan.${BASE_DOMAIN}, which a favonia/cloudflare-ddns Deployment keeps at the home WAN's current public IP. Operator decision: every Gateway-published name goes public; internal services (controller: none) remain unpublished. Co-Authored-By: Claude Opus 5.5 --- decisions/2026-10-03-public-dns-wan-target.md | 73 ++++++++++++++++ platform/externaldns/README.md | 24 +++++ platform/externaldns/ddns.yaml | 87 +++++++++++++++++++ platform/externaldns/helmrelease.yaml | 51 +++++------ platform/externaldns/kustomization.yaml | 2 + platform/gateway/gateway.yaml | 17 ++++ 6 files changed, 224 insertions(+), 30 deletions(-) create mode 100644 decisions/2026-10-03-public-dns-wan-target.md create mode 100644 platform/externaldns/ddns.yaml diff --git a/decisions/2026-10-03-public-dns-wan-target.md b/decisions/2026-10-03-public-dns-wan-target.md new file mode 100644 index 0000000..dc3008d --- /dev/null +++ b/decisions/2026-10-03-public-dns-wan-target.md @@ -0,0 +1,73 @@ +--- +date: 2026-10-03 +branch: feat/public-dns-wan-target +title: Public DNS now points at the WAN — Gateway-published names CNAME to an in-cluster DDNS record +--- + +## What changed + +The operator closed the open "public-target gap" recorded in +`platform/externaldns/helmrelease.yaml`. The WAN NAT / port-forwarding (80/443 → Gateway VIP +192.168.2.20) was done by the operator outside this repo. + +Operator decisions (2026-10-03): + +1. **Scope: every Gateway-published name goes public.** That covers the five `exposure: public` + apps (plex, gitea, keycloak, nextcloud, seerr) and also the hand-authored admin UIs (auth, + crafty, grafana, headlamp, the `home.keithjackson.dev` apex). `internal` services still publish + nothing: their `controller: none` route annotation is checked before the target is read. +2. **The WAN IP is kept current in-cluster**, not hardcoded. + +Implementation: + +- `platform/gateway/gateway.yaml`: `external-dns.alpha.kubernetes.io/target: wan.${BASE_DOMAIN}` on + the shared Gateway. Every gateway-sourced record changes from `A 192.168.2.20` to + `CNAME wan.home.keithjackson.dev`. +- `platform/externaldns/ddns.yaml` (new): `favonia/cloudflare-ddns:1.17.1` Deployment in the + `external-dns` namespace. It reuses `cloudflare-api-token` and keeps `wan.${BASE_DOMAIN}` an + unproxied IPv4 A record at the WAN IP (detected via Cloudflare trace, checked every 5 min, + `DELETE_ON_STOP=false`). external-dns has no TXT registry entry for it, so it never owns or + deletes it. +- The helmrelease comment block and the README now describe the decision instead of the gap. + +Known consequence: **k3s nodes resolve through public DNS** (`node_baseline_dns_servers` = 8.8.8.8 / +1.1.1.1). The kube-apiserver's OIDC issuer `keycloak.home.keithjackson.dev` therefore resolves to the +WAN IP from the nodes, and kubectl SSO works only if the router hairpins WAN:443 back to the Gateway. +A LAN-side probe of `136.51.115.252:443` on 2026-10-03 returned a **self-signed placeholder cert**, +not the Gateway's Let's Encrypt wildcard. That suggests no hairpin. The cert-based admin kubeconfig +is unaffected. + +## What I verified + +- Live prod, 2026-10-03, before the change: Gateway `gateway/shared` ADDRESS `192.168.2.20`. + External-dns args include `--source=gateway-httproute --source=service --policy=sync + --txt-owner-id=prod --domain-filter=keithjackson.dev`, and it logs "All records are already up to + date". +- Public answers at 1.1.1.1: plex/gitea/keycloak/grafana/auth/apex → `192.168.2.20`; + `crafty-mc` → `.22` and `unifi` → `.21` (service source, not affected by this change); jackett → + none (internal, correctly unpublished). +- The authority (betty.ns.cloudflare.com) has no unprefixed TXT, AAAA or MX at `plex.` or the apex, + so the A→CNAME switch has no co-existing record to collide with. +- WAN IP from the workstation: `136.51.115.252`. +- No CiliumNetworkPolicy or NetworkPolicy in `external-dns`. The Kyverno label and default-deny + policies match only `app-namespace` namespaces. The vpn egress policy selects only + `egress-class: vpn` pods. +- Toolbox gate result: see the PR description (run after this commit). + +## What I did NOT verify + +- **The outside-in path.** I have no vantage point outside the LAN. Whether + `https://.home.keithjackson.dev` from the internet reaches the Gateway with the LE cert is + UNVERIFIED. +- **Hairpin / kubectl SSO after the flip.** Likely broken per the probe above. Fix candidates: a + node `/etc/hosts` pin for `keycloak.${BASE_DOMAIN}` → Gateway VIP via `node_baseline`, or router + NAT reflection. +- **The record-type flip itself**, until Flux reconciles: whether Cloudflare accepts external-dns's + create-CNAME in the same batch as the delete-A, or converges one interval later. +- `favonia/cloudflare-ddns` under this pod securityContext. The README warns that + `no-new-privileges` misbehaves on some Proxmox-style hosts. These are bare-metal nodes, but it is + untested until the pod runs. +- Whether the L4 service-source names (crafty-mc, unifi) should also go public. Not decided, not + changed. +- `conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh` header lines 39–40 + still describe the target as open. That is a sensitive path, so it is left for a follow-up. diff --git a/platform/externaldns/README.md b/platform/externaldns/README.md index 005b1ec..786400b 100644 --- a/platform/externaldns/README.md +++ b/platform/externaldns/README.md @@ -20,6 +20,30 @@ A wildcard `*.keithjackson.dev` matches only one label deep, so `.home.keit its own record. **`internal` services need NO public record** — PiHole resolves them to the internal Gateway VIP (split-horizon). ExternalDNS manages **`public`** services' records only. +## Public target + in-cluster DDNS (2026-10-03) + +Gateway-published records are **CNAMEs to `wan.home.keithjackson.dev`**, not the Gateway VIP: + +``` +plex.home.keithjackson.dev CNAME wan.home.keithjackson.dev <- external-dns (Gateway `target` annotation) +wan.home.keithjackson.dev A <- cloudflare-ddns (ddns.yaml), every 5m +``` + +- The `target` annotation is on the **shared Gateway** (`platform/gateway/gateway.yaml`), so it + applies to every route that publishes at all. `internal` routes (`controller: none`) still publish + nothing. +- `ddns.yaml` runs `favonia/cloudflare-ddns` in this namespace with the same `cloudflare-api-token`. + It detects the WAN IP through Cloudflare's trace endpoint, so the pod **must not** carry + `egress-class: vpn`. +- LAN and pods resolve `*.home.keithjackson.dev` to the Gateway VIP via PiHole and CoreDNS + (split-horizon) and never use the public path. +- **Nodes resolve through public DNS** (`node_baseline_dns_servers`), so from a node, the apiserver's + OIDC issuer `keycloak.home.keithjackson.dev` resolves to the WAN IP. kubectl SSO therefore relies on + the router hairpinning (NAT reflection) WAN:443 back to the Gateway. + +Check: `dig +short @1.1.1.1 plex.home.keithjackson.dev` → `wan.home.keithjackson.dev.` then the WAN IP; +`kubectl -n external-dns logs deploy/cloudflare-ddns`. + ## Secrets Cloudflare API token referenced **by name** (`cloudflare-api-token`, shared with cert-manager), diff --git a/platform/externaldns/ddns.yaml b/platform/externaldns/ddns.yaml new file mode 100644 index 0000000..ca796fa --- /dev/null +++ b/platform/externaldns/ddns.yaml @@ -0,0 +1,87 @@ +--- +# In-cluster dynamic DNS for the home WAN (2026-10-03). +# +# Keeps ONE record, `wan.${BASE_DOMAIN}` (A, IPv4 only, not proxied), pointed at the home WAN's +# current public IP. Every public name external-dns publishes from the shared Gateway is a CNAME to +# it (platform/gateway/gateway.yaml `external-dns.alpha.kubernetes.io/target`), so a WAN IP change +# moves all of them through this one record, with no Git change and no external-dns involvement. +# +# OWNERSHIP SPLIT WITH EXTERNAL-DNS. This record carries no external-dns TXT registry entry, so +# external-dns (policy: sync, owner `prod`) never treats it as its own and never deletes it. +# Conversely this updater manages only the names in DOMAINS and never touches external-dns's records. +# +# IP DETECTION. Default `cloudflare.trace`: an HTTPS request to Cloudflare's trace endpoint, which +# reports the address the request arrived from. Correct here only because this pod is NOT in the +# `vpn` egress class — the CiliumEgressGatewayPolicy (platform/vpn-egress) selects pods by +# `homelab.keithjackson.dev/egress-class: vpn` alone, cluster-wide. Labelling this pod `vpn` would +# publish the VPN exit IP instead of the WAN IP. +# +# TOKEN. Reuses this namespace's `cloudflare-api-token` (same Zone:DNS:Edit token as external-dns +# and cert-manager) — no new secret. +# +# TIER NOTE. This lives in the gating `platform` Flux Kustomization (wait: true) next to +# external-dns. If it ever crashloops, `platform` goes not-Ready and apps stop reconciling (no +# outage). Kept here rather than in its own non-gating Kustomization to avoid a clusters/** change. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: cloudflare-ddns + namespace: external-dns + labels: + app.kubernetes.io/name: cloudflare-ddns +spec: + replicas: 1 # exactly one writer for the record + revisionHistoryLimit: 3 + strategy: + type: Recreate # never two updaters at once + selector: + matchLabels: + app.kubernetes.io/name: cloudflare-ddns + template: + metadata: + labels: + app.kubernetes.io/name: cloudflare-ddns + spec: + automountServiceAccountToken: false + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + seccompProfile: + type: RuntimeDefault + containers: + - name: cloudflare-ddns + image: favonia/cloudflare-ddns:1.17.1 + imagePullPolicy: IfNotPresent + env: + - name: CLOUDFLARE_API_TOKEN + valueFrom: + secretKeyRef: + name: cloudflare-api-token + key: api-token + - name: DOMAINS + value: "wan.${BASE_DOMAIN}" + - name: IP6_PROVIDER + value: "none" # IPv4 WAN only; leaves any AAAA alone + - name: PROXIED + value: "false" # DNS-only: Plex/Gitea/etc. terminate TLS on the Gateway + - name: UPDATE_CRON + value: "@every 5m" + - name: DELETE_ON_STOP + value: "false" # a pod restart must not blank the record every CNAME depends on + - name: RECORD_COMMENT + value: "managed by k3s cloudflare-ddns (platform/externaldns/ddns.yaml)" + - name: EMOJI + value: "false" + resources: + requests: + cpu: 5m + memory: 16Mi + limits: + memory: 64Mi + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: + - ALL diff --git a/platform/externaldns/helmrelease.yaml b/platform/externaldns/helmrelease.yaml index 56e3581..62c5475 100644 --- a/platform/externaldns/helmrelease.yaml +++ b/platform/externaldns/helmrelease.yaml @@ -81,38 +81,29 @@ spec: # what this instance can see or own; per-resource exclusion only ever REMOVES an object from # this instance's view, so it cannot make prod capable of deleting a record it does not own. # --------------------------------------------------------------------------------------------- - # THE PUBLIC-TARGET GAP — STILL OPEN, AND NOT FIXABLE HERE ON v0.21.0 (operator decision) + # THE PUBLIC TARGET — DECIDED 2026-10-03 (operator): every Gateway-published name goes public # - # The five `exposure: public` services (plex, gitea, seerr, keycloak, nextcloud) publish records - # that point at 192.168.2.20 — the Gateway VIP, an unroutable RFC1918 address — so they are not - # actually reachable from the internet either. Their records are LEFT EXACTLY AS THEY ARE by - # this change, for two independent reasons: + # Until 2026-10-03 every gateway-sourced record answered with the Gateway VIP (192.168.2.20, + # RFC1918), so nothing published here was reachable from the internet. Two facts constrained the + # fix, and both still hold: + # * `external-dns.alpha.kubernetes.io/target` is read in v0.21.0's gateway-httproute source ONLY + # from the GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation( + # gw.Annotations)`) and from a ListenerSet listener — NEVER from an HTTPRoute. A target on the + # shared Gateway therefore moves EVERY gateway-sourced record at once. + # * The home WAN IP is residential and can change, so a literal IP in Git would rot. # - # 1. THE VALUE IS NOT ESTABLISHED. The one public IP this repo knows is 136.51.115.252 - # (clusters/prod/kustomization.yaml, where the zone's `*.keithjackson.dev` wildcard also - # points). It belongs to the LEGACY non-k3s stack and currently serves a placeholder - # certificate. Whether prod's public services belong behind it, behind a new forward, or - # are not meant to be public at all is an operator decision. Breaking five services' DNS - # on a guess is worse than leaving a known-wrong-but-stable state. - # 2. THE MECHANISM CANNOT BE SCOPED TO THEM. `external-dns.alpha.kubernetes.io/target` is a - # real override, but in v0.21.0's gateway-httproute source it is read ONLY from the - # GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation( - # gw.Annotations)`) and from a ListenerSet listener — NEVER from the HTTPRoute. So there is - # no per-service target override while every public service shares the `shared` Gateway. - # Annotating the Gateway would move EVERY gateway-sourced record, which after this change - # still includes auth, crafty, grafana, headlamp and the apex — i.e. it would publish a - # routable address for admin UIs. That is strictly worse than the status quo. - # - # SO THE DECISION THE OPERATOR OWNS IS TWO-PART, and no knob is added here until the first half - # is answered, because a knob that cannot take effect invites someone to set it and expect one: - # (a) WHAT address should the five public names answer with — 136.51.115.252 as-is, a new - # port-forward/WAN address, or nothing (drop `exposure: public` to `internal` instead)? - # (b) HOW should they be separated from the non-public names that share the Gateway? Options, - # roughly in ascending cost: a SECOND Gateway (or a ListenerSet) carrying the target - # annotation, with only the public services' routes attached; or publishing the five names - # from the `crd` source (DNSEndpoint objects the renderer emits for `public` services), - # which restores per-service targeting without a second Gateway. - # Only (b) is a manifest change; (a) is a fact about the home WAN that has to be checked. + # The operator's answers: (a) yes, ALL of them — the five `exposure: public` apps AND the admin + # UIs (auth, crafty, grafana, headlamp, the apex), which sit behind Keycloak / oauth2-proxy; + # (b) the address is kept current IN-CLUSTER. Implementation: + # * platform/gateway/gateway.yaml sets `target: wan.${BASE_DOMAIN}` on the shared Gateway, so + # each published name becomes a CNAME to that one hostname; + # * platform/externaldns/ddns.yaml (favonia/cloudflare-ddns) keeps `wan.${BASE_DOMAIN}` an A + # record at the current WAN IP. That record has no external-dns TXT registry entry, so this + # instance never owns or garbage-collects it. + # Internal services are unchanged: their routes carry `controller: none` (above), which is + # checked BEFORE the target is read, so they still publish nothing. The `service` source (the L4 + # LoadBalancers crafty-mc, unifi) does not read the Gateway's annotation and still publishes its + # LB VIP. Whether those should go public is a separate, still-open question. # --------------------------------------------------------------------------------------------- # CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below. # diff --git a/platform/externaldns/kustomization.yaml b/platform/externaldns/kustomization.yaml index 11cc387..e237df7 100644 --- a/platform/externaldns/kustomization.yaml +++ b/platform/externaldns/kustomization.yaml @@ -8,3 +8,5 @@ resources: # SOPS-encrypted Cloudflare token (env CF_API_TOKEN) — SAME token as cert-manager, second copy in # the external-dns namespace. Flux decrypts at reconcile. - cloudflare-api-token.sops.yaml + # DDNS for `wan.${BASE_DOMAIN}` — the CNAME target every Gateway-published record points at. + - ddns.yaml diff --git a/platform/gateway/gateway.yaml b/platform/gateway/gateway.yaml index d4f0589..391a2bc 100644 --- a/platform/gateway/gateway.yaml +++ b/platform/gateway/gateway.yaml @@ -7,6 +7,23 @@ kind: Gateway metadata: name: shared namespace: gateway + annotations: + # PUBLIC DNS TARGET (2026-10-03, operator decision). Without this, external-dns publishes every + # route on this Gateway with the Gateway's OWN address (${GATEWAY_VIP}, RFC1918), which is + # unreachable from the internet. With it, every published `.${BASE_DOMAIN}` (and the apex) + # becomes a CNAME to `wan.${BASE_DOMAIN}`, an A record that platform/externaldns/ddns.yaml keeps + # pointed at the home WAN's current public IP. The WAN forwards 80/443 to ${GATEWAY_VIP}. + # + # Scope is EVERY route on this Gateway that publishes at all — the operator chose to make the + # admin UIs (auth, grafana, headlamp, crafty, apex) publicly resolvable alongside the five + # `exposure: public` apps. `internal` services stay unpublished via their + # `external-dns.alpha.kubernetes.io/controller: none` route annotation, which external-dns checks + # BEFORE it reads this target. external-dns v0.21.0 reads `target` from the Gateway only, never + # from an HTTPRoute — see platform/externaldns/helmrelease.yaml. + # + # LAN and in-cluster clients are unaffected: PiHole and CoreDNS answer ${BASE_DOMAIN} with + # ${GATEWAY_VIP} directly (split-horizon), so they never take this public path. + external-dns.alpha.kubernetes.io/target: "wan.${BASE_DOMAIN}" spec: gatewayClassName: cilium listeners: -- 2.54.0