From 267446c68f9220a329f8c0b66d39c4d2bb055364 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 7 Nov 2026 17:50:13 -0600 Subject: [PATCH] fix(dns): pin external-dns annotationPrefix to the alpha prefix external-dns v0.22 changed its default annotation prefix to external-dns.kubernetes.io/ with no fallback. Renovate #129 moved the chart to 1.23.0 (app v0.23.0), and at 12:13Z it ignored every alpha-prefixed annotation in this repo: the Gateway target (public names went to A 192.168.2.20), controller: none (jackett, pihole and semaphore got published), and the L4 Service hostnames (unifi and crafty-mc were deleted). Set annotationPrefix: external-dns.alpha.kubernetes.io/ to restore the v0.21 reading. Floor the chart at 1.23.0, because the flag needs app >= 0.22. Co-Authored-By: Claude Opus 5.5 --- .../2026-11-07-public-dns-wan-a-record.md | 99 +++++++++++++++++++ platform/externaldns/helmrelease.yaml | 28 +++++- 2 files changed, 126 insertions(+), 1 deletion(-) create mode 100644 decisions/2026-11-07-public-dns-wan-a-record.md diff --git a/decisions/2026-11-07-public-dns-wan-a-record.md b/decisions/2026-11-07-public-dns-wan-a-record.md new file mode 100644 index 0000000..2efe80b --- /dev/null +++ b/decisions/2026-11-07-public-dns-wan-a-record.md @@ -0,0 +1,99 @@ +--- +date: 2026-11-07 +branch: fix/public-dns-wan-a-record +title: external-dns v0.23 ignored every alpha-prefixed annotation — pin annotationPrefix back +--- + +## What changed + +`platform/externaldns/helmrelease.yaml`: + +- adds `values.annotationPrefix: "external-dns.alpha.kubernetes.io/"`. The chart renders it as + `--annotation-prefix=`. +- tightens the chart range from `<1.24.0` to `>=1.23.0 <1.24.0`, because that flag needs app + >= 0.22. +- adds a comment block explaining why. + +**Root cause.** Renovate PR #129 (merged 12:12:00Z) moved the chart from `>=1.21.0 <1.22.0` to +`<1.24.0`. Helm deployed chart 1.23.0 at 12:13:08Z, which moved the app from v0.21.0 to v0.23.0. +The upstream v0.22.0 release notes (#6424) say: "The default annotation prefix is now +`external-dns.kubernetes.io/` with no fallback". Every external-dns annotation in this repo uses +`external-dns.alpha.kubernetes.io/`. At 12:13:12Z v0.23.0 therefore: + +- **ignored the Gateway's `target`.** Every public name flipped from `CNAME + wan.home.keithjackson.dev` to `A 192.168.2.20`, an RFC1918 address served in public DNS. +- **ignored `controller: none`.** The internal names jackett, pihole and semaphore were published. + That breaks R-DNS-03. +- **ignored the L4 Services' `hostname`.** The unifi and crafty-mc A records were deleted. + +cloudflare-ddns (#123, v1.18.0) is not involved. It logs only `IPv4-enabled domains: +wan.home.keithjackson.dev` and has made no changes. + +**Why pin the prefix instead of migrating the annotations.** The pin is one value in one file, and +that file is not on the sensitive-path list. It restores the exact v0.21 reading of `target`, +`controller` and `hostname` together. A migration would touch `catalog/render.py`, every +`apps/wave-*` file, the hand-written platform manifests, and the conformance goldens and checks. +`conformance/**` needs an operator override. And a migration half-applied in either direction +reproduces this incident. + +I rejected `--enable-legacy-annotation-prefix`. Upstream calls it a migration aid that will be +removed. + +**Follow-up.** The L0 guard that the tree's annotation prefix matches this value is a separate PR +(`feat/l0-externaldns-annotation-prefix`), because it lives in `conformance/**`. A full migration +to the GA prefix remains open. It must change the annotations and this value in one commit. + +## What I verified + +- **Live HelmRelease history.** `1.23.0 0.23.0 lastDeployed 2026-11-07T12:13:08Z deployed`, and + before that `1.21.1 0.21.0 ... superseded`. The Deployment image is `external-dns:v0.23.0`. The + startup config logs `AnnotationPrefix:external-dns.kubernetes.io/ + EnableLegacyAnnotationPrefix:false`. +- **external-dns log, 12:13:12Z:** + - `action=CREATE ... type=A` and `a-` TXT for gitea, crafty, the apex, headlamp, auth, jackett, + nextcloud, keycloak, grafana, seerr, plex, pihole and semaphore. + - `action=DELETE ... type=CNAME` and `cname-` TXT for all of those except jackett, pihole and + semaphore. + - `action=DELETE ... type=A` for unifi and crafty-mc. +- **Before the upgrade.** Loki covers about 11-05 13:00Z to 12:13Z. The old pod logged no `Changing + record` line in that window. Its only error was one Cloudflare 502 on 11-05. +- **Gateway.** `gateway/shared` still carries `external-dns.alpha.kubernetes.io/target: + wan.home.keithjackson.dev`. +- **Public DNS.** At the authority (betty.ns.cloudflare.com), 1.1.1.1 and 8.8.8.8: + - plex, gitea, auth, the apex, jackett, pihole and semaphore answer A `192.168.2.20`. + - unifi and crafty-mc have no record. + - `wan.` answers A `136.51.115.252`. + - There is no AAAA for any of these names. + - The `a-plex` and `a-jackett` TXT records read `external-dns/owner=prod`. +- **Upstream v0.23.0 source.** + - `controller/execute.go` calls `annotations.SetAnnotationPrefix(cfg.AnnotationPrefix)`. + - `SetAnnotationPrefix` rebuilds `TargetKey`, `ControllerKey` and `HostnameKey`. + - `source/gateway.go` still reads `TargetsFromTargetAnnotation(gw.Annotations)`. + - Chart 1.23.0 `templates/deployment.yaml` renders `--annotation-prefix={{ + .Values.annotationPrefix }}`. +- **Phone.** Pi-hole 192.168.2.23 answers correctly: A `192.168.2.20` with TTL 60, and AAAA + NOERROR with no data. The router 192.168.2.1 answers `gitea.` with NOERROR and 0 answers, because + it strips RFC1918 results. The workstation uses only `nameserver 192.168.2.23` and has + `disable_ipv6 = 1`. +- **Toolbox gate.** See the PR description. + +## What I did NOT verify + +- **The fix live.** This depends on Flux applying it. Expected on the first reconcile, with + `policy: sync` and owner `prod`, all of these records being prod-owned: + - DELETE the A record and create CNAME `wan.` for the ten public names. + - DELETE the A and TXT records for jackett, pihole and semaphore. + - CREATE A for unifi (.21) and crafty-mc (.22). + This is UNVERIFIED until someone reads the external-dns log and checks with `dig + @betty.ns.cloudflare.com`. The A-to-CNAME type swap went through cleanly on 2026-10-03, and the + reverse went through today, so I expect it to converge. If it does not, see the manual list in + the PR. +- **That the chart's schema accepts `annotationPrefix` under Flux.** I did not render the + HelmRelease with helm here. The gate checks YAML and policy, not the chart render. +- **The phone.** Not investigated, by decision. Likely cause: the phone bypasses Pi-hole on the + client side. Candidates are IPv6 RA/RDNSS DNS from the router, Android Private DNS and iCloud + Private Relay. Answers it gets that way are rebind-stripped, or point to the WAN IP, which needs + hairpin NAT. Hairpin looked absent on 2026-10-03 (a self-signed cert on WAN:443). This is + router- or client-side, not in-repo. +- The AAAA records the operator saw on 8.8.8.8. I did not reproduce them. +- The cilium (#104) and pihole (#109) bumps were not examined beyond Pi-hole answering correctly. diff --git a/platform/externaldns/helmrelease.yaml b/platform/externaldns/helmrelease.yaml index 908e2b0..195f81d 100644 --- a/platform/externaldns/helmrelease.yaml +++ b/platform/externaldns/helmrelease.yaml @@ -11,7 +11,10 @@ spec: chart: spec: chart: external-dns - version: "<1.24.0" # k8s-1.36 catch-up (app v0.21; policy already set). Renovate maintains + # chart 1.23.x = app v0.23.0. Floor >=1.23.0 because `annotationPrefix` below needs app >=0.22. + # Renovate maintains — but READ THE APP RELEASE NOTES on every 0.x minor: 0.22 changed the + # annotation prefix with no fallback and silently re-published every internal name (2026-11-07). + version: ">=1.23.0 <1.24.0" sourceRef: kind: HelmRepository name: external-dns @@ -25,6 +28,29 @@ spec: values: provider: name: cloudflare + # --------------------------------------------------------------------------------------------- + # ANNOTATION PREFIX — PINNED TO THE LEGACY `alpha` PREFIX (2026-11-07). Do not remove. + # + # external-dns v0.22.0 (#6424) changed the default annotation prefix from + # `external-dns.alpha.kubernetes.io/` to `external-dns.kubernetes.io/` "with no fallback". + # EVERY external-dns annotation in this repo uses the alpha prefix: the Gateway's `target` + # (platform/gateway/gateway.yaml), the renderer's `controller: none` on `internal` routes + # (catalog/render.py -> apps/wave-*), and the L4 Services' `hostname` (crafty-mc, unifi). + # Renovate PR #129 moved the chart range to `<1.24.0` (app v0.21.0 -> v0.23.0) and on the first + # reconcile (2026-11-07T12:13Z) v0.23.0, reading only the GA prefix, saw none of them: + # * `target` ignored -> every public name flipped CNAME wan. -> A 192.168.2.20 (RFC1918) + # * `controller: none` ignored -> internal jackett, pihole, semaphore PUBLISHED (R-DNS-03 broken) + # * `hostname` ignored -> unifi and crafty-mc A records DELETED + # Logged as `action=CREATE ... type=A` / `action=DELETE ... type=CNAME` for every name at once. + # + # The chart's `annotationPrefix` renders `--annotation-prefix=`, which v0.23.0 applies to every + # key it reads (source/annotations/annotations.go SetAnnotationPrefix: target, controller, + # hostname, ttl, ...), restoring exactly the v0.21 behaviour. Chosen over + # `--enable-legacy-annotation-prefix` (a dual-read migration aid upstream says will be removed) + # and over migrating the annotations (renderer + goldens + conformance = sensitive paths, and a + # flag-day across ~20 objects). Migrating to the GA prefix is the follow-up: change every + # annotation AND this value in ONE commit, never one without the other. + annotationPrefix: "external-dns.alpha.kubernetes.io/" # Watch Gateway API routes (Cilium Gateway) + Services. docs/09: records rendered from intent. # # BOTH SOURCES ARE UNFILTERED HERE, AND THAT IS DELIBERATE — see the WHICH RESOURCES block