fix(dns): pin external-dns annotationPrefix — v0.23 ignored every alpha annotation #140

2026-11-07T23:54:19Z

gitea-admin commented

2026-11-07 23:54:19 +00:00

Problem

Since 2026-11-07 12:13:12Z, Cloudflare has answered the public names plex, gitea, auth, the apex, keycloak, grafana, seerr, nextcloud, headlamp and crafty with A 192.168.2.20 (RFC1918). They should be a CNAME to wan.home.keithjackson.dev. The internal names jackett, pihole and semaphore are now public, and the unifi and crafty-mc records were deleted.

Root cause

Renovate #129 moved the external-dns chart to <1.24.0. Helm deployed chart 1.23.0, which upgraded the app from v0.21.0 to v0.23.0. Upstream v0.22.0 (#6424): "The default annotation prefix is now external-dns.kubernetes.io/ with no fallback." Every external-dns annotation in this repo uses external-dns.alpha.kubernetes.io/. The live startup config shows AnnotationPrefix:external-dns.kubernetes.io/ EnableLegacyAnnotationPrefix:false, so external-dns stopped seeing:

the Gateway target, so public names fell back to the Gateway VIP;
controller: none, so the internal routes were published;
the L4 Service hostname, so unifi and crafty-mc were deleted.

The log at 12:13:12Z shows action=CREATE type=A and action=DELETE type=CNAME for every name. cloudflare-ddns (#123) is not involved: it is IPv4-only and only manages wan..

Fix

This PR sets annotationPrefix: "external-dns.alpha.kubernetes.io/". The chart renders it as --annotation-prefix=external-dns.alpha.kubernetes.io/, which I verified by rendering chart 1.23.0 with helm template. In v0.23.0, SetAnnotationPrefix rebuilds the keys for target, controller and hostname, which brings back the v0.21 behaviour. The PR also floors the chart range at >=1.23.0 <1.24.0.

I chose this over migrating the annotations to the GA prefix. The prefix change is one value in one non-sensitive file. A migration would touch the renderer, every apps/wave-* file, the platform manifests, and conformance goldens (a sensitive path), and a partial migration would recreate this incident.

The L0 guard that keeps the tree's prefix and this value in sync is in a separate PR, because it lives in conformance/** and needs operator review. That way this fix is not blocked.

Sensitive paths: none. This merges on a green gate.

After merge (expected self-heal, `policy: sync`, all records owner `prod`)

Name	Now	Expected after reconcile
plex, gitea, auth, home (apex), keycloak, grafana, seerr, nextcloud, headlamp, crafty	A 192.168.2.20 + TXT `a-<name>`	A deleted. CNAME `wan.home.keithjackson.dev` + TXT `cname-<name>` created
jackett, pihole, semaphore	A 192.168.2.20 + TXT `a-<name>`	A + TXT deleted (no record)
unifi, crafty-mc	none	A .21 / .22 + TXT `a-<name>` recreated (the pre-incident state)

If the external-dns log has not shown these changes within about 2 minutes of the HelmRelease upgrading, the operator should do the same changes by hand in Cloudflare:

delete the A records and their a- TXT for the 13 names above;
create the CNAMEs.

Gate (toolbox, worktree, all §3 workarounds, on commit `267446c`)

ci/gate.sh rc=3 (UNDETERMINED). Every stage that ran passed: spec-immutability, decisions-log, lint, l0, render-golden, l1, approval-policy.

Not run:

gitea-settings: expected, it needs the CI-only token.
conformance: L2/L3 for the touched requirement R-DNS-02 needs a cluster, and CLUSTER_AVAILABLE is unset in CI too.

So the DNS behaviour itself is not gate-proven. It has to be checked live after merge.

Not verified

The fix itself, live. Also the phone issue. Pi-hole 192.168.2.23 answers correctly, so the likely cause is on the client or router side: a resolver bypass (IPv6 RA DNS from the router, Private DNS or Private Relay) plus no hairpin NAT. Details are in decisions/2026-11-07-public-dns-wan-a-record.md.

🤖 Generated with Claude Code

## Problem Since 2026-11-07 12:13:12Z, Cloudflare has answered the public names plex, gitea, auth, the apex, keycloak, grafana, seerr, nextcloud, headlamp and crafty with **A 192.168.2.20** (RFC1918). They should be a CNAME to `wan.home.keithjackson.dev`. The **internal** names jackett, pihole and semaphore are now public, and the **unifi** and **crafty-mc** records were deleted. ## Root cause Renovate #129 moved the external-dns chart to `<1.24.0`. Helm deployed chart 1.23.0, which upgraded the app from v0.21.0 to **v0.23.0**. Upstream v0.22.0 (#6424): *"The default annotation prefix is now `external-dns.kubernetes.io/` with no fallback."* Every external-dns annotation in this repo uses `external-dns.alpha.kubernetes.io/`. The live startup config shows `AnnotationPrefix:external-dns.kubernetes.io/ EnableLegacyAnnotationPrefix:false`, so external-dns stopped seeing: - the Gateway `target`, so public names fell back to the Gateway VIP; - `controller: none`, so the internal routes were published; - the L4 Service `hostname`, so unifi and crafty-mc were deleted. The log at 12:13:12Z shows `action=CREATE type=A` and `action=DELETE type=CNAME` for every name. cloudflare-ddns (#123) is not involved: it is IPv4-only and only manages `wan.`. ## Fix This PR sets `annotationPrefix: "external-dns.alpha.kubernetes.io/"`. The chart renders it as `--annotation-prefix=external-dns.alpha.kubernetes.io/`, which I verified by rendering chart 1.23.0 with `helm template`. In v0.23.0, `SetAnnotationPrefix` rebuilds the keys for target, controller and hostname, which brings back the v0.21 behaviour. The PR also floors the chart range at `>=1.23.0 <1.24.0`. I chose this over migrating the annotations to the GA prefix. The prefix change is one value in one non-sensitive file. A migration would touch the renderer, every `apps/wave-*` file, the platform manifests, and conformance goldens (a sensitive path), and a partial migration would recreate this incident. The L0 guard that keeps the tree's prefix and this value in sync is in a separate PR, because it lives in `conformance/**` and needs operator review. That way this fix is not blocked. **Sensitive paths: none.** This merges on a green gate. ## After merge (expected self-heal, `policy: sync`, all records owner `prod`) | Name | Now | Expected after reconcile | |---|---|---| | plex, gitea, auth, home (apex), keycloak, grafana, seerr, nextcloud, headlamp, crafty | A 192.168.2.20 + TXT `a-<name>` | A deleted. CNAME `wan.home.keithjackson.dev` + TXT `cname-<name>` created | | jackett, pihole, semaphore | A 192.168.2.20 + TXT `a-<name>` | A + TXT deleted (no record) | | unifi, crafty-mc | none | A .21 / .22 + TXT `a-<name>` recreated (the pre-incident state) | If the external-dns log has not shown these changes within about 2 minutes of the HelmRelease upgrading, the operator should do the same changes by hand in Cloudflare: - **delete** the A records and their `a-` TXT for the 13 names above; - **create** the CNAMEs. ## Gate (toolbox, worktree, all §3 workarounds, on commit 267446c) `ci/gate.sh` **rc=3 (UNDETERMINED)**. Every stage that ran passed: spec-immutability, decisions-log, lint, l0, render-golden, l1, approval-policy. Not run: - `gitea-settings`: expected, it needs the CI-only token. - `conformance`: L2/L3 for the touched requirement R-DNS-02 needs a cluster, and `CLUSTER_AVAILABLE` is unset in CI too. So the DNS behaviour itself is **not** gate-proven. It has to be checked live after merge. ## Not verified The fix itself, live. Also the phone issue. Pi-hole 192.168.2.23 answers correctly, so the likely cause is on the client or router side: a resolver bypass (IPv6 RA DNS from the router, Private DNS or Private Relay) plus no hairpin NAT. Details are in `decisions/2026-11-07-public-dns-wan-a-record.md`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 1 commit 2026-11-07 23:54:19 +00:00

fix(dns): pin external-dns annotationPrefix to the alpha prefix

gate / gate (pull_request) Successful in 2m58s

Details

gate / conformance (pull_request) Skipped

Details

267446c68f

external-dns v0.22 changed its default annotation prefix to
external-dns.kubernetes.io/ with no fallback. Renovate #129 moved the chart
to 1.23.0 (app v0.23.0), and at 12:13Z it ignored every alpha-prefixed
annotation in this repo: the Gateway target (public names went to A
192.168.2.20), controller: none (jackett, pihole and semaphore got
published), and the L4 Service hostnames (unifi and crafty-mc were deleted).

Set annotationPrefix: external-dns.alpha.kubernetes.io/ to restore the v0.21
reading. Floor the chart at 1.23.0, because the flag needs app >= 0.22.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

gitea-admin referenced this pull request

2026-11-07 23:59:36 +00:00

feat(conformance): L0 guard — external-dns annotationPrefix must match the tree (stacked on #140) #141

jakman85 merged commit b0230ba736 into main

2026-11-08 00:01:06 +00:00

jakman85 referenced this issue from a commit

2026-11-08 00:08:30 +00:00

Merge pull request 'feat(conformance): L0 guard — external-dns annotationPrefix must match the tree (stacked on #140)' (#141) from feat/l0-externaldns-annotation-prefix into main

Sign in to join this conversation.