--- # ExternalDNS — publishes public DNS records for `public` services automatically (docs/09, R-DNS-02). # Watches Gateway/HTTPRoute (+Service); manages records in Cloudflare. Phase 0 STUB. apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: name: external-dns namespace: external-dns spec: interval: 30m chart: spec: chart: external-dns version: ">=1.21.0 <1.22.0" # k8s-1.36 catch-up (app v0.21; policy already set). Renovate maintains sourceRef: kind: HelmRepository name: external-dns namespace: external-dns install: remediation: retries: 3 upgrade: remediation: retries: 3 values: provider: name: cloudflare # Watch Gateway API routes (Cilium Gateway) + Services. docs/09: records rendered from intent. # # BOTH SOURCES ARE UNFILTERED HERE, AND THAT IS DELIBERATE — see the WHICH RESOURCES block # below. Scoping is done PER RESOURCE by the renderer, not cluster-wide by this list. sources: - gateway-httproute - service # --------------------------------------------------------------------------------------------- # WHICH RESOURCES MAY PUBLISH — `internal` services no longer can (R-DNS-03, 2026-09-14) # # THE DEFECT. `sources` above admits EVERY HTTPRoute on the shared Gateway, and each one's # address is the Gateway VIP. docs/09 R-DNS-03 says `internal` services are unreachable from the # public internet and that Pi-hole serves the internal split-horizon answers; the renderer # contract said an `internal` service renders "no public DNS record". Neither sentence was # enforced by anything, so six `exposure: internal` services (jackett, pihole, qbittorrent, # radarr, semaphore, sonarr) published `.home.keithjackson.dev` into this PUBLIC # authoritative zone answering 192.168.2.20. That exposed the internal subnet, the Gateway VIP # and a complete service inventory to anyone who could guess a label, and made every one of # those names a DNS-rebind primitive — which is exactly why the SOHO router at 192.168.2.1 # strips RFC1918 answers and LAN clients need per-client dnsmasq overrides to reach their own # services. Pre-state captured at the AUTHORITY and committed: # .standup-dns/dns-20260914-pre-externaldns-scope.txt. # # THE MECHANISM — per-resource exclusion, emitted by catalog/render.py from `exposure`. # An `internal` service's rendered HTTPRoute carries # external-dns.alpha.kubernetes.io/controller: none # ExternalDNS v0.21.0 (the DEPLOYED app version; read off the live prod Deployment, image # registry.k8s.io/external-dns/external-dns:v0.21.0) skips any source object whose `controller` # annotation is present and != its own `dns-controller`: # source/annotations/processors.go IsControllerMismatch() # source/gateway.go Endpoints(): if annotations.IsControllerMismatch(meta, src.rtKind) { continue } # source/service.go Endpoints(): the identical check # The gateway check runs BEFORE any hostname is resolved, so such a route cannot become an # endpoint for any name it declares. PROVEN LIVE on prod before this landed, with a throwaway # hostname and no existing record at risk: an annotated route published nothing at all, and # ADDING the annotation to a route that already had a record produced # action=DELETE record= type=A and action=DELETE record=a- type=TXT # one ~60s reconcile later, and moved nothing else in the zone. # # WHY PER-RESOURCE AND NOT A CLUSTER-WIDE OPT-IN FILTER. The chart's `labelFilter` # (--label-filter) is the stronger default posture — publish only what is labelled — and # v0.21.0's own --help confirms BOTH configured sources support it ("currently supported by # source types crd, gateway-httproute, ... service and ambassador-host"). It is NOT set here # because its blast radius is not this defect's. Nine publishing objects in this repo are # hand-authored platform/apps manifests with no `exposure` field for the renderer to read, and # seven of them hold live records: auth (oauth2-proxy), crafty, crafty-mc, grafana, headlamp, # the apex (homepage), unifi. Setting an opt-in filter DELETES all seven on the next reconcile, # and whether each should be publicly resolvable at all is an OPERATOR decision per name, not a # renderer one. They are enumerated in the shrink-only register in # conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh, so the set # cannot grow silently — adjudicate them there, then this filter becomes the right hardening. # # DELIBERATELY UNTOUCHED: `policy: sync`, `txtOwnerId`, `domainFilters`. Nothing above widens # what this instance can see or own; per-resource exclusion only ever REMOVES an object from # this instance's view, so it cannot make prod capable of deleting a record it does not own. # --------------------------------------------------------------------------------------------- # THE PUBLIC-TARGET GAP — STILL OPEN, AND NOT FIXABLE HERE ON v0.21.0 (operator decision) # # The five `exposure: public` services (plex, gitea, seerr, keycloak, nextcloud) publish records # that point at 192.168.2.20 — the Gateway VIP, an unroutable RFC1918 address — so they are not # actually reachable from the internet either. Their records are LEFT EXACTLY AS THEY ARE by # this change, for two independent reasons: # # 1. THE VALUE IS NOT ESTABLISHED. The one public IP this repo knows is 136.51.115.252 # (clusters/prod/kustomization.yaml, where the zone's `*.keithjackson.dev` wildcard also # points). It belongs to the LEGACY non-k3s stack and currently serves a placeholder # certificate. Whether prod's public services belong behind it, behind a new forward, or # are not meant to be public at all is an operator decision. Breaking five services' DNS # on a guess is worse than leaving a known-wrong-but-stable state. # 2. THE MECHANISM CANNOT BE SCOPED TO THEM. `external-dns.alpha.kubernetes.io/target` is a # real override, but in v0.21.0's gateway-httproute source it is read ONLY from the # GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation( # gw.Annotations)`) and from a ListenerSet listener — NEVER from the HTTPRoute. So there is # no per-service target override while every public service shares the `shared` Gateway. # Annotating the Gateway would move EVERY gateway-sourced record, which after this change # still includes auth, crafty, grafana, headlamp and the apex — i.e. it would publish a # routable address for admin UIs. That is strictly worse than the status quo. # # SO THE DECISION THE OPERATOR OWNS IS TWO-PART, and no knob is added here until the first half # is answered, because a knob that cannot take effect invites someone to set it and expect one: # (a) WHAT address should the five public names answer with — 136.51.115.252 as-is, a new # port-forward/WAN address, or nothing (drop `exposure: public` to `internal` instead)? # (b) HOW should they be separated from the non-public names that share the Gateway? Options, # roughly in ascending cost: a SECOND Gateway (or a ListenerSet) carrying the target # annotation, with only the public services' routes attached; or publishing the five names # from the `crd` source (DNSEndpoint objects the renderer emits for `public` services), # which restores per-service targeting without a second Gateway. # Only (b) is a manifest change; (a) is a fact about the home WAN that has to be checked. # --------------------------------------------------------------------------------------------- # CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below. # # On 2026-09-04 prod's external-dns DELETED EVERY ONE of dev's public records. Both clusters ran # `policy: sync` against the SAME Cloudflare zone, served the SAME hostnames, and — because # NEITHER set txtOwnerId — both inherited the chart default owner id `default`. Prod therefore # read dev's TXT registry entries as its own, found no local resource claiming those names, and # garbage-collected them. `policy: sync` is correct and stays; what was missing was identity. # Full write-up: infra/PROD-PROVISIONING.md §10. # # TWO INDEPENDENT GUARDS NOW, either of which alone prevents a recurrence: # 1. txtOwnerId — per-environment identity; a cluster only ever touches records it owns. # 2. domainFilters — per-environment name scope; prod cannot even see names outside its # own subtree. Defence in depth, added after the incident. # --------------------------------------------------------------------------------------------- # Which names this instance may manage at all. Cloudflare-provider caveat, empirically hit # 2026-08: this value is used for ZONE selection as well as record matching, and a filter that is # a strict subdomain of the zone dropped the parent zone entirely ("no hosted zone matching record # DNS Name") — so DEV keeps the zone apex `keithjackson.dev`, which is also what dev serves the # apex of. PROD narrows to `prod.home.keithjackson.dev`. If prod's first reconcile logs no # matching zone, that caveat still applies: prod creates NOTHING (fail-closed, never destructive) # and the fallback is to widen prod's EXTERNALDNS_DOMAIN_FILTER back to the zone apex, leaving # txtOwnerId as the sole guard. See clusters/base/cluster-vars.yaml. domainFilters: - "${EXTERNALDNS_DOMAIN_FILTER}" # The TXT-registry owner id. MUST be distinct per environment — this is the knob whose absence # caused the incident. Dev renders `default` (its live records are already registered under that # owner; changing it would orphan them and duplicate every record). Prod renders `prod`. txtOwnerId: "${EXTERNALDNS_TXT_OWNER_ID}" policy: sync # create + update + delete records as services come and go — but ONLY # records this owner id owns AND this domainFilter admits (see above). # Cloudflare API token: SOPS-encrypted secret, referenced BY NAME only (OQ-4, ADR-0007). env: - name: CF_API_TOKEN valueFrom: secretKeyRef: name: cloudflare-api-token key: api-token # R-DNS-02 PROVEN via the `service` source: the conformance/l2-behavioral/dns-exposure test # creates a LoadBalancer Service with an external-dns hostname and asserts the record auto-appears # in public DNS (then teardown + policy=sync removes it). gateway-httproute source wiring is the # same provider/zone path; a route-based test can follow when app routes exist.