2026-11-08 00:01:06 +00:00
2 changed files with 126 additions and 1 deletions
@@ -0,0 +1,99 @@
+---
+date: 2026-11-07
+branch: fix/public-dns-wan-a-record
+title: external-dns v0.23 ignored every alpha-prefixed annotation — pin annotationPrefix back
+---
+
+## What changed
+
+`platform/externaldns/helmrelease.yaml`:
+
+- adds `values.annotationPrefix: "external-dns.alpha.kubernetes.io/"`. The chart renders it as
+  `--annotation-prefix=`.
+- tightens the chart range from `<1.24.0` to `>=1.23.0 <1.24.0`, because that flag needs app
+  >= 0.22.
+- adds a comment block explaining why.
+
+**Root cause.** Renovate PR #129 (merged 12:12:00Z) moved the chart from `>=1.21.0 <1.22.0` to
+`<1.24.0`. Helm deployed chart 1.23.0 at 12:13:08Z, which moved the app from v0.21.0 to v0.23.0.
+The upstream v0.22.0 release notes (#6424) say: "The default annotation prefix is now
+`external-dns.kubernetes.io/` with no fallback". Every external-dns annotation in this repo uses
+`external-dns.alpha.kubernetes.io/`. At 12:13:12Z v0.23.0 therefore:
+
+- **ignored the Gateway's `target`.** Every public name flipped from `CNAME
+  wan.home.keithjackson.dev` to `A 192.168.2.20`, an RFC1918 address served in public DNS.
+- **ignored `controller: none`.** The internal names jackett, pihole and semaphore were published.
+  That breaks R-DNS-03.
+- **ignored the L4 Services' `hostname`.** The unifi and crafty-mc A records were deleted.
+
+cloudflare-ddns (#123, v1.18.0) is not involved. It logs only `IPv4-enabled domains:
+wan.home.keithjackson.dev` and has made no changes.
+
+**Why pin the prefix instead of migrating the annotations.** The pin is one value in one file, and
+that file is not on the sensitive-path list. It restores the exact v0.21 reading of `target`,
+`controller` and `hostname` together. A migration would touch `catalog/render.py`, every
+`apps/wave-*` file, the hand-written platform manifests, and the conformance goldens and checks.
+`conformance/**` needs an operator override. And a migration half-applied in either direction
+reproduces this incident.
+
+I rejected `--enable-legacy-annotation-prefix`. Upstream calls it a migration aid that will be
+removed.
+
+**Follow-up.** The L0 guard that the tree's annotation prefix matches this value is a separate PR
+(`feat/l0-externaldns-annotation-prefix`), because it lives in `conformance/**`. A full migration
+to the GA prefix remains open. It must change the annotations and this value in one commit.
+
+## What I verified
+
+- **Live HelmRelease history.** `1.23.0 0.23.0 lastDeployed 2026-11-07T12:13:08Z deployed`, and
+  before that `1.21.1 0.21.0 ... superseded`. The Deployment image is `external-dns:v0.23.0`. The
+  startup config logs `AnnotationPrefix:external-dns.kubernetes.io/
+  EnableLegacyAnnotationPrefix:false`.
+- **external-dns log, 12:13:12Z:**
+  - `action=CREATE ... type=A` and `a-` TXT for gitea, crafty, the apex, headlamp, auth, jackett,
+    nextcloud, keycloak, grafana, seerr, plex, pihole and semaphore.
+  - `action=DELETE ... type=CNAME` and `cname-` TXT for all of those except jackett, pihole and
+    semaphore.
+  - `action=DELETE ... type=A` for unifi and crafty-mc.
+- **Before the upgrade.** Loki covers about 11-05 13:00Z to 12:13Z. The old pod logged no `Changing
+  record` line in that window. Its only error was one Cloudflare 502 on 11-05.
+- **Gateway.** `gateway/shared` still carries `external-dns.alpha.kubernetes.io/target:
+  wan.home.keithjackson.dev`.
+- **Public DNS.** At the authority (betty.ns.cloudflare.com), 1.1.1.1 and 8.8.8.8:
+  - plex, gitea, auth, the apex, jackett, pihole and semaphore answer A `192.168.2.20`.
+  - unifi and crafty-mc have no record.
+  - `wan.` answers A `136.51.115.252`.
+  - There is no AAAA for any of these names.
+  - The `a-plex` and `a-jackett` TXT records read `external-dns/owner=prod`.
+- **Upstream v0.23.0 source.**
+  - `controller/execute.go` calls `annotations.SetAnnotationPrefix(cfg.AnnotationPrefix)`.
+  - `SetAnnotationPrefix` rebuilds `TargetKey`, `ControllerKey` and `HostnameKey`.
+  - `source/gateway.go` still reads `TargetsFromTargetAnnotation(gw.Annotations)`.
+  - Chart 1.23.0 `templates/deployment.yaml` renders `--annotation-prefix={{
+    .Values.annotationPrefix }}`.
+- **Phone.** Pi-hole 192.168.2.23 answers correctly: A `192.168.2.20` with TTL 60, and AAAA
+  NOERROR with no data. The router 192.168.2.1 answers `gitea.` with NOERROR and 0 answers, because
+  it strips RFC1918 results. The workstation uses only `nameserver 192.168.2.23` and has
+  `disable_ipv6 = 1`.
+- **Toolbox gate.** See the PR description.
+
+## What I did NOT verify
+
+- **The fix live.** This depends on Flux applying it. Expected on the first reconcile, with
+  `policy: sync` and owner `prod`, all of these records being prod-owned:
+  - DELETE the A record and create CNAME `wan.` for the ten public names.
+  - DELETE the A and TXT records for jackett, pihole and semaphore.
+  - CREATE A for unifi (.21) and crafty-mc (.22).
+  This is UNVERIFIED until someone reads the external-dns log and checks with `dig
+  @betty.ns.cloudflare.com`. The A-to-CNAME type swap went through cleanly on 2026-10-03, and the
+  reverse went through today, so I expect it to converge. If it does not, see the manual list in
+  the PR.
+- **That the chart's schema accepts `annotationPrefix` under Flux.** I did not render the
+  HelmRelease with helm here. The gate checks YAML and policy, not the chart render.
+- **The phone.** Not investigated, by decision. Likely cause: the phone bypasses Pi-hole on the
+  client side. Candidates are IPv6 RA/RDNSS DNS from the router, Android Private DNS and iCloud
+  Private Relay. Answers it gets that way are rebind-stripped, or point to the WAN IP, which needs
+  hairpin NAT. Hairpin looked absent on 2026-10-03 (a self-signed cert on WAN:443). This is
+  router- or client-side, not in-repo.
+- The AAAA records the operator saw on 8.8.8.8. I did not reproduce them.
+- The cilium (#104) and pihole (#109) bumps were not examined beyond Pi-hole answering correctly.
@@ -11,7 +11,10 @@ spec:
  chart:
    spec:
      chart: external-dns
-      version: "<1.24.0"   # k8s-1.36 catch-up (app v0.21; policy already set). Renovate maintains
+      # chart 1.23.x = app v0.23.0. Floor >=1.23.0 because `annotationPrefix` below needs app >=0.22.
+      # Renovate maintains — but READ THE APP RELEASE NOTES on every 0.x minor: 0.22 changed the
+      # annotation prefix with no fallback and silently re-published every internal name (2026-11-07).
+      version: ">=1.23.0 <1.24.0"
      sourceRef:
        kind: HelmRepository
        name: external-dns
@@ -25,6 +28,29 @@ spec:
  values:
    provider:
      name: cloudflare
+    # ---------------------------------------------------------------------------------------------
+    # ANNOTATION PREFIX — PINNED TO THE LEGACY `alpha` PREFIX (2026-11-07). Do not remove.
+    #
+    # external-dns v0.22.0 (#6424) changed the default annotation prefix from
+    # `external-dns.alpha.kubernetes.io/` to `external-dns.kubernetes.io/` "with no fallback".
+    # EVERY external-dns annotation in this repo uses the alpha prefix: the Gateway's `target`
+    # (platform/gateway/gateway.yaml), the renderer's `controller: none` on `internal` routes
+    # (catalog/render.py -> apps/wave-*), and the L4 Services' `hostname` (crafty-mc, unifi).
+    # Renovate PR #129 moved the chart range to `<1.24.0` (app v0.21.0 -> v0.23.0) and on the first
+    # reconcile (2026-11-07T12:13Z) v0.23.0, reading only the GA prefix, saw none of them:
+    #   * `target` ignored  -> every public name flipped CNAME wan.<base> -> A 192.168.2.20 (RFC1918)
+    #   * `controller: none` ignored -> internal jackett, pihole, semaphore PUBLISHED (R-DNS-03 broken)
+    #   * `hostname` ignored -> unifi and crafty-mc A records DELETED
+    # Logged as `action=CREATE ... type=A` / `action=DELETE ... type=CNAME` for every name at once.
+    #
+    # The chart's `annotationPrefix` renders `--annotation-prefix=`, which v0.23.0 applies to every
+    # key it reads (source/annotations/annotations.go SetAnnotationPrefix: target, controller,
+    # hostname, ttl, ...), restoring exactly the v0.21 behaviour. Chosen over
+    # `--enable-legacy-annotation-prefix` (a dual-read migration aid upstream says will be removed)
+    # and over migrating the annotations (renderer + goldens + conformance = sensitive paths, and a
+    # flag-day across ~20 objects). Migrating to the GA prefix is the follow-up: change every
+    # annotation AND this value in ONE commit, never one without the other.
+    annotationPrefix: "external-dns.alpha.kubernetes.io/"
    # Watch Gateway API routes (Cilium Gateway) + Services. docs/09: records rendered from intent.
    #
    # BOTH SOURCES ARE UNFILTERED HERE, AND THAT IS DELIBERATE — see the WHICH RESOURCES block