--- # ExternalDNS — publishes public DNS records for `public` services automatically (docs/09, R-DNS-02). # Watches Gateway/HTTPRoute (+Service); manages records in Cloudflare. Phase 0 STUB. apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: external-dns namespace: external-dns spec: interval: 30m chart: spec: chart: external-dns # chart 1.23.x = app v0.23.0. Floor >=1.23.0 because `annotationPrefix` below needs app >=0.22. # Renovate maintains — but READ THE APP RELEASE NOTES on every 0.x minor: 0.22 changed the # annotation prefix with no fallback and silently re-published every internal name (2026-11-07). version: ">=1.23.0 <1.24.0" sourceRef: kind: HelmRepository name: external-dns namespace: external-dns install: remediation: retries: 3 upgrade: remediation: retries: 3 values: provider: name: cloudflare # --------------------------------------------------------------------------------------------- # ANNOTATION PREFIX — PINNED TO THE LEGACY `alpha` PREFIX (2026-11-07). Do not remove. # # external-dns v0.22.0 (#6424) changed the default annotation prefix from # `external-dns.alpha.kubernetes.io/` to `external-dns.kubernetes.io/` "with no fallback". # EVERY external-dns annotation in this repo uses the alpha prefix: the Gateway's `target` # (platform/gateway/gateway.yaml), the renderer's `controller: none` on `internal` routes # (catalog/render.py -> apps/wave-*), and the L4 Services' `hostname` (crafty-mc, unifi). # Renovate PR #129 moved the chart range to `<1.24.0` (app v0.21.0 -> v0.23.0) and on the first # reconcile (2026-11-07T12:13Z) v0.23.0, reading only the GA prefix, saw none of them: # * `target` ignored -> every public name flipped CNAME wan. -> A 192.168.2.20 (RFC1918) # * `controller: none` ignored -> internal jackett, pihole, semaphore PUBLISHED (R-DNS-03 broken) # * `hostname` ignored -> unifi and crafty-mc A records DELETED # Logged as `action=CREATE ... type=A` / `action=DELETE ... type=CNAME` for every name at once. # # The chart's `annotationPrefix` renders `--annotation-prefix=`, which v0.23.0 applies to every # key it reads (source/annotations/annotations.go SetAnnotationPrefix: target, controller, # hostname, ttl, ...), restoring exactly the v0.21 behaviour. Chosen over # `--enable-legacy-annotation-prefix` (a dual-read migration aid upstream says will be removed) # and over migrating the annotations (renderer + goldens + conformance = sensitive paths, and a # flag-day across ~20 objects). Migrating to the GA prefix is the follow-up: change every # annotation AND this value in ONE commit, never one without the other. annotationPrefix: "external-dns.alpha.kubernetes.io/" # Watch Gateway API routes (Cilium Gateway) + Services. docs/09: records rendered from intent. # # BOTH SOURCES ARE UNFILTERED HERE, AND THAT IS DELIBERATE — see the WHICH RESOURCES block # below. Scoping is done PER RESOURCE by the renderer, not cluster-wide by this list. sources: - gateway-httproute - service # --------------------------------------------------------------------------------------------- # WHICH RESOURCES MAY PUBLISH — `internal` services no longer can (R-DNS-03, 2026-09-14) # # THE DEFECT. `sources` above admits EVERY HTTPRoute on the shared Gateway, and each one's # address is the Gateway VIP. docs/09 R-DNS-03 says `internal` services are unreachable from the # public internet and that Pi-hole serves the internal split-horizon answers; the renderer # contract said an `internal` service renders "no public DNS record". Neither sentence was # enforced by anything, so six `exposure: internal` services (jackett, pihole, qbittorrent, # radarr, semaphore, sonarr) published `.home.keithjackson.dev` into this PUBLIC # authoritative zone answering 192.168.2.20. That exposed the internal subnet, the Gateway VIP # and a complete service inventory to anyone who could guess a label, and made every one of # those names a DNS-rebind primitive — which is exactly why the SOHO router at 192.168.2.1 # strips RFC1918 answers and LAN clients need per-client dnsmasq overrides to reach their own # services. Pre-state captured at the AUTHORITY and committed: # .standup-dns/dns-20260914-pre-externaldns-scope.txt. # # THE MECHANISM — per-resource exclusion, emitted by catalog/render.py from `exposure`. # An `internal` service's rendered HTTPRoute carries # external-dns.alpha.kubernetes.io/controller: none # ExternalDNS v0.21.0 (the DEPLOYED app version; read off the live prod Deployment, image # registry.k8s.io/external-dns/external-dns:v0.21.0) skips any source object whose `controller` # annotation is present and != its own `dns-controller`: # source/annotations/processors.go IsControllerMismatch() # source/gateway.go Endpoints(): if annotations.IsControllerMismatch(meta, src.rtKind) { continue } # source/service.go Endpoints(): the identical check # The gateway check runs BEFORE any hostname is resolved, so such a route cannot become an # endpoint for any name it declares. PROVEN LIVE on prod before this landed, with a throwaway # hostname and no existing record at risk: an annotated route published nothing at all, and # ADDING the annotation to a route that already had a record produced # action=DELETE record= type=A and action=DELETE record=a- type=TXT # one ~60s reconcile later, and moved nothing else in the zone. # # WHY PER-RESOURCE AND NOT A CLUSTER-WIDE OPT-IN FILTER. The chart's `labelFilter` # (--label-filter) is the stronger default posture — publish only what is labelled — and # v0.21.0's own --help confirms BOTH configured sources support it ("currently supported by # source types crd, gateway-httproute, ... service and ambassador-host"). It is NOT set here # because its blast radius is not this defect's. Nine publishing objects in this repo are # hand-authored platform/apps manifests with no `exposure` field for the renderer to read, and # seven of them hold live records: auth (oauth2-proxy), crafty, crafty-mc, grafana, headlamp, # the apex (homepage), unifi. Setting an opt-in filter DELETES all seven on the next reconcile, # and whether each should be publicly resolvable at all is an OPERATOR decision per name, not a # renderer one. They are enumerated in the shrink-only register in # conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh, so the set # cannot grow silently — adjudicate them there, then this filter becomes the right hardening. # # DELIBERATELY UNTOUCHED: `policy: sync`, `txtOwnerId`, `domainFilters`. Nothing above widens # what this instance can see or own; per-resource exclusion only ever REMOVES an object from # this instance's view, so it cannot make prod capable of deleting a record it does not own. # --------------------------------------------------------------------------------------------- # THE PUBLIC TARGET — DECIDED 2026-10-03 (operator): every Gateway-published name goes public # # Until 2026-10-03 every gateway-sourced record answered with the Gateway VIP (192.168.2.20, # RFC1918), so nothing published here was reachable from the internet. Two facts constrained the # fix, and both still hold: # * `external-dns.alpha.kubernetes.io/target` is read in v0.21.0's gateway-httproute source ONLY # from the GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation( # gw.Annotations)`) and from a ListenerSet listener — NEVER from an HTTPRoute. A target on the # shared Gateway therefore moves EVERY gateway-sourced record at once. # * The home WAN IP is residential and can change, so a literal IP in Git would rot. # # The operator's answers: (a) yes, ALL of them — the five `exposure: public` apps AND the admin # UIs (auth, crafty, grafana, headlamp, the apex), which sit behind Keycloak / oauth2-proxy; # (b) the address is kept current IN-CLUSTER. Implementation: # * platform/gateway/gateway.yaml sets `target: wan.${BASE_DOMAIN}` on the shared Gateway, so # each published name becomes a CNAME to that one hostname; # * platform/externaldns/ddns.yaml (favonia/cloudflare-ddns) keeps `wan.${BASE_DOMAIN}` an A # record at the current WAN IP. That record has no external-dns TXT registry entry, so this # instance never owns or garbage-collects it. # Internal services are unchanged: their routes carry `controller: none` (above), which is # checked BEFORE the target is read, so they still publish nothing. The `service` source (the L4 # LoadBalancers crafty-mc, unifi) does not read the Gateway's annotation and still publishes its # LB VIP. Whether those should go public is a separate, still-open question. # --------------------------------------------------------------------------------------------- # CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below. # # On 2026-09-04 prod's external-dns DELETED EVERY ONE of dev's public records. Both clusters ran # `policy: sync` against the SAME Cloudflare zone, served the SAME hostnames, and — because # NEITHER set txtOwnerId — both inherited the chart default owner id `default`. Prod therefore # read dev's TXT registry entries as its own, found no local resource claiming those names, and # garbage-collected them. `policy: sync` is correct and stays; what was missing was identity. # Full write-up: infra/PROD-PROVISIONING.md §10. # # TWO INDEPENDENT GUARDS NOW, either of which alone prevents a recurrence: # 1. txtOwnerId — per-environment identity; a cluster only ever touches records it owns. # 2. domainFilters — per-environment name scope; prod cannot even see names outside its # own subtree. Defence in depth, added after the incident. # --------------------------------------------------------------------------------------------- # Which names this instance may manage at all. Cloudflare-provider caveat, empirically hit # 2026-08: this value is used for ZONE selection as well as record matching, and a filter that is # a strict subdomain of the zone dropped the parent zone entirely ("no hosted zone matching record # DNS Name") — so DEV keeps the zone apex `keithjackson.dev`, which is also what dev serves the # apex of. PROD narrows to `prod.home.keithjackson.dev`. If prod's first reconcile logs no # matching zone, that caveat still applies: prod creates NOTHING (fail-closed, never destructive) # and the fallback is to widen prod's EXTERNALDNS_DOMAIN_FILTER back to the zone apex, leaving # txtOwnerId as the sole guard. See clusters/base/cluster-vars.yaml. domainFilters: - "${EXTERNALDNS_DOMAIN_FILTER}" # The TXT-registry owner id. MUST be distinct per environment — this is the knob whose absence # caused the incident. Dev renders `default` (its live records are already registered under that # owner; changing it would orphan them and duplicate every record). Prod renders `prod`. txtOwnerId: "${EXTERNALDNS_TXT_OWNER_ID}" policy: sync # create + update + delete records as services come and go — but ONLY # records this owner id owns AND this domainFilter admits (see above). # Cloudflare API token: SOPS-encrypted secret, referenced BY NAME only (OQ-4, ADR-0007). env: - name: CF_API_TOKEN valueFrom: secretKeyRef: name: cloudflare-api-token key: api-token # R-DNS-02 PROVEN via the `service` source: the conformance/l2-behavioral/dns-exposure test # creates a LoadBalancer Service with an external-dns hostname and asserts the record auto-appears # in public DNS (then teardown + policy=sync removes it). gateway-httproute source wiring is the # same provider/zone path; a route-based test can follow when app routes exist.