jakman85andClaude Opus 5.5
267446c68f
external-dns v0.22 changed its default annotation prefix to external-dns.kubernetes.io/ with no fallback. Renovate #129 moved the chart to 1.23.0 (app v0.23.0), and at 12:13Z it ignored every alpha-prefixed annotation in this repo: the Gateway target (public names went to A 192.168.2.20), controller: none (jackett, pihole and semaphore got published), and the L4 Service hostnames (unifi and crafty-mc were deleted). Set annotationPrefix: external-dns.alpha.kubernetes.io/ to restore the v0.21 reading. Floor the chart at 1.23.0, because the flag needs app >= 0.22. Co-Authored-By: Claude Opus 5.5 <[email protected]>
175 lines
12 KiB
YAML
175 lines
12 KiB
YAML
---
|
|
# ExternalDNS — publishes public DNS records for `public` services automatically (docs/09, R-DNS-02).
|
|
# Watches Gateway/HTTPRoute (+Service); manages records in Cloudflare. Phase 0 STUB.
|
|
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
kind: HelmRelease
|
|
metadata:
|
|
name: external-dns
|
|
namespace: external-dns
|
|
spec:
|
|
interval: 30m
|
|
chart:
|
|
spec:
|
|
chart: external-dns
|
|
# chart 1.23.x = app v0.23.0. Floor >=1.23.0 because `annotationPrefix` below needs app >=0.22.
|
|
# Renovate maintains — but READ THE APP RELEASE NOTES on every 0.x minor: 0.22 changed the
|
|
# annotation prefix with no fallback and silently re-published every internal name (2026-11-07).
|
|
version: ">=1.23.0 <1.24.0"
|
|
sourceRef:
|
|
kind: HelmRepository
|
|
name: external-dns
|
|
namespace: external-dns
|
|
install:
|
|
remediation:
|
|
retries: 3
|
|
upgrade:
|
|
remediation:
|
|
retries: 3
|
|
values:
|
|
provider:
|
|
name: cloudflare
|
|
# ---------------------------------------------------------------------------------------------
|
|
# ANNOTATION PREFIX — PINNED TO THE LEGACY `alpha` PREFIX (2026-11-07). Do not remove.
|
|
#
|
|
# external-dns v0.22.0 (#6424) changed the default annotation prefix from
|
|
# `external-dns.alpha.kubernetes.io/` to `external-dns.kubernetes.io/` "with no fallback".
|
|
# EVERY external-dns annotation in this repo uses the alpha prefix: the Gateway's `target`
|
|
# (platform/gateway/gateway.yaml), the renderer's `controller: none` on `internal` routes
|
|
# (catalog/render.py -> apps/wave-*), and the L4 Services' `hostname` (crafty-mc, unifi).
|
|
# Renovate PR #129 moved the chart range to `<1.24.0` (app v0.21.0 -> v0.23.0) and on the first
|
|
# reconcile (2026-11-07T12:13Z) v0.23.0, reading only the GA prefix, saw none of them:
|
|
# * `target` ignored -> every public name flipped CNAME wan.<base> -> A 192.168.2.20 (RFC1918)
|
|
# * `controller: none` ignored -> internal jackett, pihole, semaphore PUBLISHED (R-DNS-03 broken)
|
|
# * `hostname` ignored -> unifi and crafty-mc A records DELETED
|
|
# Logged as `action=CREATE ... type=A` / `action=DELETE ... type=CNAME` for every name at once.
|
|
#
|
|
# The chart's `annotationPrefix` renders `--annotation-prefix=`, which v0.23.0 applies to every
|
|
# key it reads (source/annotations/annotations.go SetAnnotationPrefix: target, controller,
|
|
# hostname, ttl, ...), restoring exactly the v0.21 behaviour. Chosen over
|
|
# `--enable-legacy-annotation-prefix` (a dual-read migration aid upstream says will be removed)
|
|
# and over migrating the annotations (renderer + goldens + conformance = sensitive paths, and a
|
|
# flag-day across ~20 objects). Migrating to the GA prefix is the follow-up: change every
|
|
# annotation AND this value in ONE commit, never one without the other.
|
|
annotationPrefix: "external-dns.alpha.kubernetes.io/"
|
|
# Watch Gateway API routes (Cilium Gateway) + Services. docs/09: records rendered from intent.
|
|
#
|
|
# BOTH SOURCES ARE UNFILTERED HERE, AND THAT IS DELIBERATE — see the WHICH RESOURCES block
|
|
# below. Scoping is done PER RESOURCE by the renderer, not cluster-wide by this list.
|
|
sources:
|
|
- gateway-httproute
|
|
- service
|
|
# ---------------------------------------------------------------------------------------------
|
|
# WHICH RESOURCES MAY PUBLISH — `internal` services no longer can (R-DNS-03, 2026-09-14)
|
|
#
|
|
# THE DEFECT. `sources` above admits EVERY HTTPRoute on the shared Gateway, and each one's
|
|
# address is the Gateway VIP. docs/09 R-DNS-03 says `internal` services are unreachable from the
|
|
# public internet and that Pi-hole serves the internal split-horizon answers; the renderer
|
|
# contract said an `internal` service renders "no public DNS record". Neither sentence was
|
|
# enforced by anything, so six `exposure: internal` services (jackett, pihole, qbittorrent,
|
|
# radarr, semaphore, sonarr) published `<name>.home.keithjackson.dev` into this PUBLIC
|
|
# authoritative zone answering 192.168.2.20. That exposed the internal subnet, the Gateway VIP
|
|
# and a complete service inventory to anyone who could guess a label, and made every one of
|
|
# those names a DNS-rebind primitive — which is exactly why the SOHO router at 192.168.2.1
|
|
# strips RFC1918 answers and LAN clients need per-client dnsmasq overrides to reach their own
|
|
# services. Pre-state captured at the AUTHORITY and committed:
|
|
# .standup-dns/dns-20260914-pre-externaldns-scope.txt.
|
|
#
|
|
# THE MECHANISM — per-resource exclusion, emitted by catalog/render.py from `exposure`.
|
|
# An `internal` service's rendered HTTPRoute carries
|
|
# external-dns.alpha.kubernetes.io/controller: none
|
|
# ExternalDNS v0.21.0 (the DEPLOYED app version; read off the live prod Deployment, image
|
|
# registry.k8s.io/external-dns/external-dns:v0.21.0) skips any source object whose `controller`
|
|
# annotation is present and != its own `dns-controller`:
|
|
# source/annotations/processors.go IsControllerMismatch()
|
|
# source/gateway.go Endpoints(): if annotations.IsControllerMismatch(meta, src.rtKind) { continue }
|
|
# source/service.go Endpoints(): the identical check
|
|
# The gateway check runs BEFORE any hostname is resolved, so such a route cannot become an
|
|
# endpoint for any name it declares. PROVEN LIVE on prod before this landed, with a throwaway
|
|
# hostname and no existing record at risk: an annotated route published nothing at all, and
|
|
# ADDING the annotation to a route that already had a record produced
|
|
# action=DELETE record=<name> type=A and action=DELETE record=a-<name> type=TXT
|
|
# one ~60s reconcile later, and moved nothing else in the zone.
|
|
#
|
|
# WHY PER-RESOURCE AND NOT A CLUSTER-WIDE OPT-IN FILTER. The chart's `labelFilter`
|
|
# (--label-filter) is the stronger default posture — publish only what is labelled — and
|
|
# v0.21.0's own --help confirms BOTH configured sources support it ("currently supported by
|
|
# source types crd, gateway-httproute, ... service and ambassador-host"). It is NOT set here
|
|
# because its blast radius is not this defect's. Nine publishing objects in this repo are
|
|
# hand-authored platform/apps manifests with no `exposure` field for the renderer to read, and
|
|
# seven of them hold live records: auth (oauth2-proxy), crafty, crafty-mc, grafana, headlamp,
|
|
# the apex (homepage), unifi. Setting an opt-in filter DELETES all seven on the next reconcile,
|
|
# and whether each should be publicly resolvable at all is an OPERATOR decision per name, not a
|
|
# renderer one. They are enumerated in the shrink-only register in
|
|
# conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh, so the set
|
|
# cannot grow silently — adjudicate them there, then this filter becomes the right hardening.
|
|
#
|
|
# DELIBERATELY UNTOUCHED: `policy: sync`, `txtOwnerId`, `domainFilters`. Nothing above widens
|
|
# what this instance can see or own; per-resource exclusion only ever REMOVES an object from
|
|
# this instance's view, so it cannot make prod capable of deleting a record it does not own.
|
|
# ---------------------------------------------------------------------------------------------
|
|
# THE PUBLIC TARGET — DECIDED 2026-10-03 (operator): every Gateway-published name goes public
|
|
#
|
|
# Until 2026-10-03 every gateway-sourced record answered with the Gateway VIP (192.168.2.20,
|
|
# RFC1918), so nothing published here was reachable from the internet. Two facts constrained the
|
|
# fix, and both still hold:
|
|
# * `external-dns.alpha.kubernetes.io/target` is read in v0.21.0's gateway-httproute source ONLY
|
|
# from the GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
|
|
# gw.Annotations)`) and from a ListenerSet listener — NEVER from an HTTPRoute. A target on the
|
|
# shared Gateway therefore moves EVERY gateway-sourced record at once.
|
|
# * The home WAN IP is residential and can change, so a literal IP in Git would rot.
|
|
#
|
|
# The operator's answers: (a) yes, ALL of them — the five `exposure: public` apps AND the admin
|
|
# UIs (auth, crafty, grafana, headlamp, the apex), which sit behind Keycloak / oauth2-proxy;
|
|
# (b) the address is kept current IN-CLUSTER. Implementation:
|
|
# * platform/gateway/gateway.yaml sets `target: wan.${BASE_DOMAIN}` on the shared Gateway, so
|
|
# each published name becomes a CNAME to that one hostname;
|
|
# * platform/externaldns/ddns.yaml (favonia/cloudflare-ddns) keeps `wan.${BASE_DOMAIN}` an A
|
|
# record at the current WAN IP. That record has no external-dns TXT registry entry, so this
|
|
# instance never owns or garbage-collects it.
|
|
# Internal services are unchanged: their routes carry `controller: none` (above), which is
|
|
# checked BEFORE the target is read, so they still publish nothing. The `service` source (the L4
|
|
# LoadBalancers crafty-mc, unifi) does not read the Gateway's annotation and still publishes its
|
|
# LB VIP. Whether those should go public is a separate, still-open question.
|
|
# ---------------------------------------------------------------------------------------------
|
|
# CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below.
|
|
#
|
|
# On 2026-09-04 prod's external-dns DELETED EVERY ONE of dev's public records. Both clusters ran
|
|
# `policy: sync` against the SAME Cloudflare zone, served the SAME hostnames, and — because
|
|
# NEITHER set txtOwnerId — both inherited the chart default owner id `default`. Prod therefore
|
|
# read dev's TXT registry entries as its own, found no local resource claiming those names, and
|
|
# garbage-collected them. `policy: sync` is correct and stays; what was missing was identity.
|
|
# Full write-up: infra/PROD-PROVISIONING.md §10.
|
|
#
|
|
# TWO INDEPENDENT GUARDS NOW, either of which alone prevents a recurrence:
|
|
# 1. txtOwnerId — per-environment identity; a cluster only ever touches records it owns.
|
|
# 2. domainFilters — per-environment name scope; prod cannot even see names outside its
|
|
# own subtree. Defence in depth, added after the incident.
|
|
# ---------------------------------------------------------------------------------------------
|
|
# Which names this instance may manage at all. Cloudflare-provider caveat, empirically hit
|
|
# 2026-08: this value is used for ZONE selection as well as record matching, and a filter that is
|
|
# a strict subdomain of the zone dropped the parent zone entirely ("no hosted zone matching record
|
|
# DNS Name") — so DEV keeps the zone apex `keithjackson.dev`, which is also what dev serves the
|
|
# apex of. PROD narrows to `prod.home.keithjackson.dev`. If prod's first reconcile logs no
|
|
# matching zone, that caveat still applies: prod creates NOTHING (fail-closed, never destructive)
|
|
# and the fallback is to widen prod's EXTERNALDNS_DOMAIN_FILTER back to the zone apex, leaving
|
|
# txtOwnerId as the sole guard. See clusters/base/cluster-vars.yaml.
|
|
domainFilters:
|
|
- "${EXTERNALDNS_DOMAIN_FILTER}"
|
|
# The TXT-registry owner id. MUST be distinct per environment — this is the knob whose absence
|
|
# caused the incident. Dev renders `default` (its live records are already registered under that
|
|
# owner; changing it would orphan them and duplicate every record). Prod renders `prod`.
|
|
txtOwnerId: "${EXTERNALDNS_TXT_OWNER_ID}"
|
|
policy: sync # create + update + delete records as services come and go — but ONLY
|
|
# records this owner id owns AND this domainFilter admits (see above).
|
|
# Cloudflare API token: SOPS-encrypted secret, referenced BY NAME only (OQ-4, ADR-0007).
|
|
env:
|
|
- name: CF_API_TOKEN
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: cloudflare-api-token
|
|
key: api-token
|
|
# R-DNS-02 PROVEN via the `service` source: the conformance/l2-behavioral/dns-exposure test
|
|
# creates a LoadBalancer Service with an external-dns hostname and asserts the record auto-appears
|
|
# in public DNS (then teardown + policy=sync removes it). gateway-httproute source wiring is the
|
|
# same provider/zone path; a route-based test can follow when app routes exist.
|