home-network-k3s-impl/decisions/2026-11-07-public-dns-wan-a-record.md

date, branch, title

date	branch	title
2026-11-07	fix/public-dns-wan-a-record	external-dns v0.23 ignored every alpha-prefixed annotation — pin annotationPrefix back

What changed

platform/externaldns/helmrelease.yaml:

• adds values.annotationPrefix: "external-dns.alpha.kubernetes.io/". The chart renders it as --annotation-prefix=.
• tightens the chart range from <1.24.0 to >=1.23.0 <1.24.0, because that flag needs app

  = 0.22.

• adds a comment block explaining why.

Root cause. Renovate PR #129 (merged 12:12:00Z) moved the chart from >=1.21.0 <1.22.0 to <1.24.0. Helm deployed chart 1.23.0 at 12:13:08Z, which moved the app from v0.21.0 to v0.23.0. The upstream v0.22.0 release notes (#6424) say: "The default annotation prefix is now external-dns.kubernetes.io/ with no fallback". Every external-dns annotation in this repo uses external-dns.alpha.kubernetes.io/. At 12:13:12Z v0.23.0 therefore:

• ignored the Gateway's target. Every public name flipped from CNAME wan.home.keithjackson.dev to A 192.168.2.20, an RFC1918 address served in public DNS.
• ignored controller: none. The internal names jackett, pihole and semaphore were published. That breaks R-DNS-03.
• ignored the L4 Services' hostname. The unifi and crafty-mc A records were deleted.

cloudflare-ddns (#123, v1.18.0) is not involved. It logs only IPv4-enabled domains: wan.home.keithjackson.dev and has made no changes.

Why pin the prefix instead of migrating the annotations. The pin is one value in one file, and that file is not on the sensitive-path list. It restores the exact v0.21 reading of target, controller and hostname together. A migration would touch catalog/render.py, every apps/wave-* file, the hand-written platform manifests, and the conformance goldens and checks. conformance/** needs an operator override. And a migration half-applied in either direction reproduces this incident.

I rejected --enable-legacy-annotation-prefix. Upstream calls it a migration aid that will be removed.

Follow-up. The L0 guard that the tree's annotation prefix matches this value is a separate PR (feat/l0-externaldns-annotation-prefix), because it lives in conformance/**. A full migration to the GA prefix remains open. It must change the annotations and this value in one commit.

What I verified

• Live HelmRelease history. 1.23.0 0.23.0 lastDeployed 2026-11-07T12:13:08Z deployed, and before that 1.21.1 0.21.0 ... superseded. The Deployment image is external-dns:v0.23.0. The startup config logs AnnotationPrefix:external-dns.kubernetes.io/ EnableLegacyAnnotationPrefix:false.
• external-dns log, 12:13:12Z:
  • action=CREATE ... type=A and a- TXT for gitea, crafty, the apex, headlamp, auth, jackett, nextcloud, keycloak, grafana, seerr, plex, pihole and semaphore.
  • action=DELETE ... type=CNAME and cname- TXT for all of those except jackett, pihole and semaphore.
  • action=DELETE ... type=A for unifi and crafty-mc.
• Before the upgrade. Loki covers about 11-05 13:00Z to 12:13Z. The old pod logged no Changing record line in that window. Its only error was one Cloudflare 502 on 11-05.
• Gateway. gateway/shared still carries external-dns.alpha.kubernetes.io/target: wan.home.keithjackson.dev.
• Public DNS. At the authority (betty.ns.cloudflare.com), 1.1.1.1 and 8.8.8.8:
  • plex, gitea, auth, the apex, jackett, pihole and semaphore answer A 192.168.2.20.
  • unifi and crafty-mc have no record.
  • wan. answers A 136.51.115.252.
  • There is no AAAA for any of these names.
  • The a-plex and a-jackett TXT records read external-dns/owner=prod.
• Upstream v0.23.0 source.
  • controller/execute.go calls annotations.SetAnnotationPrefix(cfg.AnnotationPrefix).
  • SetAnnotationPrefix rebuilds TargetKey, ControllerKey and HostnameKey.
  • source/gateway.go still reads TargetsFromTargetAnnotation(gw.Annotations).
  • Chart 1.23.0 templates/deployment.yaml renders --annotation-prefix={{ .Values.annotationPrefix }}.
• Phone. Pi-hole 192.168.2.23 answers correctly: A 192.168.2.20 with TTL 60, and AAAA NOERROR with no data. The router 192.168.2.1 answers gitea. with NOERROR and 0 answers, because it strips RFC1918 results. The workstation uses only nameserver 192.168.2.23 and has disable_ipv6 = 1.
• Toolbox gate. See the PR description.

What I did NOT verify

• The fix live. This depends on Flux applying it. Expected on the first reconcile, with policy: sync and owner prod, all of these records being prod-owned:
  • DELETE the A record and create CNAME wan. for the ten public names.
  • DELETE the A and TXT records for jackett, pihole and semaphore.
  • CREATE A for unifi (.21) and crafty-mc (.22). This is UNVERIFIED until someone reads the external-dns log and checks with dig @betty.ns.cloudflare.com. The A-to-CNAME type swap went through cleanly on 2026-10-03, and the reverse went through today, so I expect it to converge. If it does not, see the manual list in the PR.
• That the chart's schema accepts annotationPrefix under Flux. I did not render the HelmRelease with helm here. The gate checks YAML and policy, not the chart render.
• The phone. Not investigated, by decision. Likely cause: the phone bypasses Pi-hole on the client side. Candidates are IPv6 RA/RDNSS DNS from the router, Android Private DNS and iCloud Private Relay. Answers it gets that way are rebind-stripped, or point to the WAN IP, which needs hairpin NAT. Hairpin looked absent on 2026-10-03 (a self-signed cert on WAN:443). This is router- or client-side, not in-repo.
• The AAAA records the operator saw on 8.8.8.8. I did not reproduce them.
• The cilium (#104) and pihole (#109) bumps were not examined beyond Pi-hole answering correctly.