--- date: 2026-11-07 branch: fix/public-dns-wan-a-record title: external-dns v0.23 ignored every alpha-prefixed annotation — pin annotationPrefix back --- ## What changed `platform/externaldns/helmrelease.yaml`: - adds `values.annotationPrefix: "external-dns.alpha.kubernetes.io/"`. The chart renders it as `--annotation-prefix=`. - tightens the chart range from `<1.24.0` to `>=1.23.0 <1.24.0`, because that flag needs app >= 0.22. - adds a comment block explaining why. **Root cause.** Renovate PR #129 (merged 12:12:00Z) moved the chart from `>=1.21.0 <1.22.0` to `<1.24.0`. Helm deployed chart 1.23.0 at 12:13:08Z, which moved the app from v0.21.0 to v0.23.0. The upstream v0.22.0 release notes (#6424) say: "The default annotation prefix is now `external-dns.kubernetes.io/` with no fallback". Every external-dns annotation in this repo uses `external-dns.alpha.kubernetes.io/`. At 12:13:12Z v0.23.0 therefore: - **ignored the Gateway's `target`.** Every public name flipped from `CNAME wan.home.keithjackson.dev` to `A 192.168.2.20`, an RFC1918 address served in public DNS. - **ignored `controller: none`.** The internal names jackett, pihole and semaphore were published. That breaks R-DNS-03. - **ignored the L4 Services' `hostname`.** The unifi and crafty-mc A records were deleted. cloudflare-ddns (#123, v1.18.0) is not involved. It logs only `IPv4-enabled domains: wan.home.keithjackson.dev` and has made no changes. **Why pin the prefix instead of migrating the annotations.** The pin is one value in one file, and that file is not on the sensitive-path list. It restores the exact v0.21 reading of `target`, `controller` and `hostname` together. A migration would touch `catalog/render.py`, every `apps/wave-*` file, the hand-written platform manifests, and the conformance goldens and checks. `conformance/**` needs an operator override. And a migration half-applied in either direction reproduces this incident. I rejected `--enable-legacy-annotation-prefix`. Upstream calls it a migration aid that will be removed. **Follow-up.** The L0 guard that the tree's annotation prefix matches this value is a separate PR (`feat/l0-externaldns-annotation-prefix`), because it lives in `conformance/**`. A full migration to the GA prefix remains open. It must change the annotations and this value in one commit. ## What I verified - **Live HelmRelease history.** `1.23.0 0.23.0 lastDeployed 2026-11-07T12:13:08Z deployed`, and before that `1.21.1 0.21.0 ... superseded`. The Deployment image is `external-dns:v0.23.0`. The startup config logs `AnnotationPrefix:external-dns.kubernetes.io/ EnableLegacyAnnotationPrefix:false`. - **external-dns log, 12:13:12Z:** - `action=CREATE ... type=A` and `a-` TXT for gitea, crafty, the apex, headlamp, auth, jackett, nextcloud, keycloak, grafana, seerr, plex, pihole and semaphore. - `action=DELETE ... type=CNAME` and `cname-` TXT for all of those except jackett, pihole and semaphore. - `action=DELETE ... type=A` for unifi and crafty-mc. - **Before the upgrade.** Loki covers about 11-05 13:00Z to 12:13Z. The old pod logged no `Changing record` line in that window. Its only error was one Cloudflare 502 on 11-05. - **Gateway.** `gateway/shared` still carries `external-dns.alpha.kubernetes.io/target: wan.home.keithjackson.dev`. - **Public DNS.** At the authority (betty.ns.cloudflare.com), 1.1.1.1 and 8.8.8.8: - plex, gitea, auth, the apex, jackett, pihole and semaphore answer A `192.168.2.20`. - unifi and crafty-mc have no record. - `wan.` answers A `136.51.115.252`. - There is no AAAA for any of these names. - The `a-plex` and `a-jackett` TXT records read `external-dns/owner=prod`. - **Upstream v0.23.0 source.** - `controller/execute.go` calls `annotations.SetAnnotationPrefix(cfg.AnnotationPrefix)`. - `SetAnnotationPrefix` rebuilds `TargetKey`, `ControllerKey` and `HostnameKey`. - `source/gateway.go` still reads `TargetsFromTargetAnnotation(gw.Annotations)`. - Chart 1.23.0 `templates/deployment.yaml` renders `--annotation-prefix={{ .Values.annotationPrefix }}`. - **Phone.** Pi-hole 192.168.2.23 answers correctly: A `192.168.2.20` with TTL 60, and AAAA NOERROR with no data. The router 192.168.2.1 answers `gitea.` with NOERROR and 0 answers, because it strips RFC1918 results. The workstation uses only `nameserver 192.168.2.23` and has `disable_ipv6 = 1`. - **Toolbox gate.** See the PR description. ## What I did NOT verify - **The fix live.** This depends on Flux applying it. Expected on the first reconcile, with `policy: sync` and owner `prod`, all of these records being prod-owned: - DELETE the A record and create CNAME `wan.` for the ten public names. - DELETE the A and TXT records for jackett, pihole and semaphore. - CREATE A for unifi (.21) and crafty-mc (.22). This is UNVERIFIED until someone reads the external-dns log and checks with `dig @betty.ns.cloudflare.com`. The A-to-CNAME type swap went through cleanly on 2026-10-03, and the reverse went through today, so I expect it to converge. If it does not, see the manual list in the PR. - **That the chart's schema accepts `annotationPrefix` under Flux.** I did not render the HelmRelease with helm here. The gate checks YAML and policy, not the chart render. - **The phone.** Not investigated, by decision. Likely cause: the phone bypasses Pi-hole on the client side. Candidates are IPv6 RA/RDNSS DNS from the router, Android Private DNS and iCloud Private Relay. Answers it gets that way are rebind-stripped, or point to the WAN IP, which needs hairpin NAT. Hairpin looked absent on 2026-10-03 (a self-signed cert on WAN:443). This is router- or client-side, not in-repo. - The AAAA records the operator saw on 8.8.8.8. I did not reproduce them. - The cilium (#104) and pihole (#109) bumps were not examined beyond Pi-hole answering correctly.