home-network-k3s-impl/decisions/2026-11-07-public-dns-wan-a-record.md

---
date: 2026-11-07
branch: fix/public-dns-wan-a-record
title: external-dns v0.23 ignored every alpha-prefixed annotation — pin annotationPrefix back
---

## What changed

`platform/externaldns/helmrelease.yaml`:

- adds `values.annotationPrefix: "external-dns.alpha.kubernetes.io/"`. The chart renders it as
  `--annotation-prefix=`.
- tightens the chart range from `<1.24.0` to `>=1.23.0 <1.24.0`, because that flag needs app
  >= 0.22.
- adds a comment block explaining why.

**Root cause.** Renovate PR #129 (merged 12:12:00Z) moved the chart from `>=1.21.0 <1.22.0` to
`<1.24.0`. Helm deployed chart 1.23.0 at 12:13:08Z, which moved the app from v0.21.0 to v0.23.0.
The upstream v0.22.0 release notes (#6424) say: "The default annotation prefix is now
`external-dns.kubernetes.io/` with no fallback". Every external-dns annotation in this repo uses
`external-dns.alpha.kubernetes.io/`. At 12:13:12Z v0.23.0 therefore:

- **ignored the Gateway's `target`.** Every public name flipped from `CNAME
  wan.home.keithjackson.dev` to `A 192.168.2.20`, an RFC1918 address served in public DNS.
- **ignored `controller: none`.** The internal names jackett, pihole and semaphore were published.
  That breaks R-DNS-03.
- **ignored the L4 Services' `hostname`.** The unifi and crafty-mc A records were deleted.

cloudflare-ddns (#123, v1.18.0) is not involved. It logs only `IPv4-enabled domains:
wan.home.keithjackson.dev` and has made no changes.

**Why pin the prefix instead of migrating the annotations.** The pin is one value in one file, and
that file is not on the sensitive-path list. It restores the exact v0.21 reading of `target`,
`controller` and `hostname` together. A migration would touch `catalog/render.py`, every
`apps/wave-*` file, the hand-written platform manifests, and the conformance goldens and checks.
`conformance/**` needs an operator override. And a migration half-applied in either direction
reproduces this incident.

I rejected `--enable-legacy-annotation-prefix`. Upstream calls it a migration aid that will be
removed.

**Follow-up.** The L0 guard that the tree's annotation prefix matches this value is a separate PR
(`feat/l0-externaldns-annotation-prefix`), because it lives in `conformance/**`. A full migration
to the GA prefix remains open. It must change the annotations and this value in one commit.

## What I verified

- **Live HelmRelease history.** `1.23.0 0.23.0 lastDeployed 2026-11-07T12:13:08Z deployed`, and
  before that `1.21.1 0.21.0 ... superseded`. The Deployment image is `external-dns:v0.23.0`. The
  startup config logs `AnnotationPrefix:external-dns.kubernetes.io/
  EnableLegacyAnnotationPrefix:false`.
- **external-dns log, 12:13:12Z:**
  - `action=CREATE ... type=A` and `a-` TXT for gitea, crafty, the apex, headlamp, auth, jackett,
    nextcloud, keycloak, grafana, seerr, plex, pihole and semaphore.
  - `action=DELETE ... type=CNAME` and `cname-` TXT for all of those except jackett, pihole and
    semaphore.
  - `action=DELETE ... type=A` for unifi and crafty-mc.
- **Before the upgrade.** Loki covers about 11-05 13:00Z to 12:13Z. The old pod logged no `Changing
  record` line in that window. Its only error was one Cloudflare 502 on 11-05.
- **Gateway.** `gateway/shared` still carries `external-dns.alpha.kubernetes.io/target:
  wan.home.keithjackson.dev`.
- **Public DNS.** At the authority (betty.ns.cloudflare.com), 1.1.1.1 and 8.8.8.8:
  - plex, gitea, auth, the apex, jackett, pihole and semaphore answer A `192.168.2.20`.
  - unifi and crafty-mc have no record.
  - `wan.` answers A `136.51.115.252`.
  - There is no AAAA for any of these names.
  - The `a-plex` and `a-jackett` TXT records read `external-dns/owner=prod`.
- **Upstream v0.23.0 source.**
  - `controller/execute.go` calls `annotations.SetAnnotationPrefix(cfg.AnnotationPrefix)`.
  - `SetAnnotationPrefix` rebuilds `TargetKey`, `ControllerKey` and `HostnameKey`.
  - `source/gateway.go` still reads `TargetsFromTargetAnnotation(gw.Annotations)`.
  - Chart 1.23.0 `templates/deployment.yaml` renders `--annotation-prefix={{
    .Values.annotationPrefix }}`.
- **Phone.** Pi-hole 192.168.2.23 answers correctly: A `192.168.2.20` with TTL 60, and AAAA
  NOERROR with no data. The router 192.168.2.1 answers `gitea.` with NOERROR and 0 answers, because
  it strips RFC1918 results. The workstation uses only `nameserver 192.168.2.23` and has
  `disable_ipv6 = 1`.
- **Toolbox gate.** See the PR description.

## What I did NOT verify

- **The fix live.** This depends on Flux applying it. Expected on the first reconcile, with
  `policy: sync` and owner `prod`, all of these records being prod-owned:
  - DELETE the A record and create CNAME `wan.` for the ten public names.
  - DELETE the A and TXT records for jackett, pihole and semaphore.
  - CREATE A for unifi (.21) and crafty-mc (.22).
  This is UNVERIFIED until someone reads the external-dns log and checks with `dig
  @betty.ns.cloudflare.com`. The A-to-CNAME type swap went through cleanly on 2026-10-03, and the
  reverse went through today, so I expect it to converge. If it does not, see the manual list in
  the PR.
- **That the chart's schema accepts `annotationPrefix` under Flux.** I did not render the
  HelmRelease with helm here. The gate checks YAML and policy, not the chart render.
- **The phone.** Not investigated, by decision. Likely cause: the phone bypasses Pi-hole on the
  client side. Candidates are IPv6 RA/RDNSS DNS from the router, Android Private DNS and iCloud
  Private Relay. Answers it gets that way are rebind-stripped, or point to the WAN IP, which needs
  hairpin NAT. Hairpin looked absent on 2026-10-03 (a self-signed cert on WAN:443). This is
  router- or client-side, not in-repo.
- The AAAA records the operator saw on 8.8.8.8. I did not reproduce them.
- The cilium (#104) and pihole (#109) bumps were not examined beyond Pi-hole answering correctly.