jakman85andClaude Opus 5
4080065aa7
docs/09 R-DNS-03 and schemas/renderer-contract.md both said an `internal` service renders no public DNS record. Nothing enforced either sentence. ExternalDNS's `gateway-httproute` source admitted EVERY HTTPRoute on the shared Gateway, so six `exposure: internal` services — jackett, pihole, qbittorrent, radarr, semaphore, sonarr — published `<name>.home.keithjackson.dev` into the public authoritative Cloudflare zone answering with the Gateway VIP 192.168.2.20. That published the internal subnet, the Gateway VIP and a full service inventory to anyone who could guess a label, and made each of those names a DNS-rebind primitive — which is why the SOHO router strips RFC1918 answers and LAN clients need per-client dnsmasq overrides to reach their own services. THE MECHANISM, established against the DEPLOYED version rather than from memory. The live prod Deployment runs registry.k8s.io/external-dns/external-dns:v0.21.0. In v0.21.0, source/annotations/processors.go IsControllerMismatch() skips a source object whose `external-dns.alpha.kubernetes.io/controller` annotation is present and != its own `dns-controller`, and source/gateway.go applies that check inside Endpoints() BEFORE any hostname is resolved; source/service.go applies the identical check, covering the second configured source. catalog/render.py now emits that annotation on every HTTPRoute it renders for an `internal` service, derived from `exposure` alone — no per-service list — so a service that omits `exposure` inherits the internal default and is unpublishable too. Verified LIVE on prod on a throwaway hostname with no existing record at risk: the annotated route published nothing, and adding the annotation to a route that already had a record produced action=DELETE for both its A record and its `a-` TXT registry entry one ~60s reconcile later, moving nothing else in the zone. WHY PER-RESOURCE AND NOT A CLUSTER-WIDE OPT-IN FILTER. `labelFilter` is the stronger default posture and v0.21.0 supports it for both configured sources, but nine publishing objects here are hand-authored platform/apps manifests with no `exposure` field, seven of which hold live records (auth, crafty, crafty-mc, grafana, headlamp, the apex, unifi). An opt-in filter deletes all seven at once, and whether each should be publicly resolvable is an operator decision per name. Per-resource exclusion has exactly the blast radius of the defect; the seven are enumerated in a shrink-only register so the unadjudicated set cannot grow silently, and the filter stays available as hardening afterwards. `policy: sync`, `txtOwnerId` and `domainFilters` are UNTOUCHED. Per-resource exclusion only ever removes an object from this instance's view, so nothing here makes prod capable of deleting a record it does not own. WHAT THIS DELETES, and it is not reversible by a revert alone: 12 records — the A record and the `a-`-prefixed TXT registry entry for each of jackett, pihole, qbittorrent, radarr, semaphore and sonarr. The zone has no `*.home.keithjackson.dev` wildcard and `*.keithjackson.dev` does not match two labels, so each becomes a public NXDOMAIN with no fallback. The complete pre-state is captured at the authority and committed as a recovery baseline (.standup-dns/dns-20260914-pre-externaldns-scope.txt), with the restore procedure for both halves of a record. MERGE ORDER IS NOT OPTIONAL: feat/pihole-lan-dns-authority must land and be verified first. Removing these records while the LAN still depends on them, with the router stripping RFC1918 and no internal authority yet, breaks name resolution outright. The five `exposure: public` services are LEFT UNCHANGED and still answer with the Gateway VIP, so they remain publicly unreachable. That is two open operator decisions, both written up in platform/externaldns/helmrelease.yaml: what address they should answer with (136.51.115.252 is the legacy non-k3s stack serving a placeholder cert, not established as correct for this cluster), and how to separate them from the non-public names that share the Gateway — because in v0.21.0 the target annotation is read only from the GATEWAY and a ListenerSet, never from an HTTPRoute, so a per-service override is not expressible while they all attach to `shared`. No knob is added for it yet: one that cannot take effect invites someone to set it and expect one. Guard: conformance/tests/l0-static/internal-services-publish-no-public-dns, a whole-repo-scoped L0 static check (no ci/l0.sh stage — the whole-repo-invariants-wired meta-test forbids one). It asserts both directions (present on internal, absent on public), the `service`-source half, that the renderer DERIVES the annotation from `exposure` and cannot emit external-dns's own claiming value, proves the shipped renderer end-to-end on internal/public/omitted-field fixtures, and ratchets the non-catalog register. Proven to fail on the pre-fix tree (9 of 53 assertions, naming all six services) and pass after (56/56). Schema 1.6.0 -> 1.7.0: no field changes, a mapping change to `exposure`. Co-Authored-By: Claude Opus 5 <[email protected]>
158 lines
11 KiB
YAML
158 lines
11 KiB
YAML
---
|
|
# ExternalDNS — publishes public DNS records for `public` services automatically (docs/09, R-DNS-02).
|
|
# Watches Gateway/HTTPRoute (+Service); manages records in Cloudflare. Phase 0 STUB.
|
|
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
kind: HelmRelease
|
|
metadata:
|
|
name: external-dns
|
|
namespace: external-dns
|
|
spec:
|
|
interval: 30m
|
|
chart:
|
|
spec:
|
|
chart: external-dns
|
|
version: ">=1.21.0 <1.22.0" # k8s-1.36 catch-up (app v0.21; policy already set). Renovate maintains
|
|
sourceRef:
|
|
kind: HelmRepository
|
|
name: external-dns
|
|
namespace: external-dns
|
|
install:
|
|
remediation:
|
|
retries: 3
|
|
upgrade:
|
|
remediation:
|
|
retries: 3
|
|
values:
|
|
provider:
|
|
name: cloudflare
|
|
# Watch Gateway API routes (Cilium Gateway) + Services. docs/09: records rendered from intent.
|
|
#
|
|
# BOTH SOURCES ARE UNFILTERED HERE, AND THAT IS DELIBERATE — see the WHICH RESOURCES block
|
|
# below. Scoping is done PER RESOURCE by the renderer, not cluster-wide by this list.
|
|
sources:
|
|
- gateway-httproute
|
|
- service
|
|
# ---------------------------------------------------------------------------------------------
|
|
# WHICH RESOURCES MAY PUBLISH — `internal` services no longer can (R-DNS-03, 2026-09-14)
|
|
#
|
|
# THE DEFECT. `sources` above admits EVERY HTTPRoute on the shared Gateway, and each one's
|
|
# address is the Gateway VIP. docs/09 R-DNS-03 says `internal` services are unreachable from the
|
|
# public internet and that Pi-hole serves the internal split-horizon answers; the renderer
|
|
# contract said an `internal` service renders "no public DNS record". Neither sentence was
|
|
# enforced by anything, so six `exposure: internal` services (jackett, pihole, qbittorrent,
|
|
# radarr, semaphore, sonarr) published `<name>.home.keithjackson.dev` into this PUBLIC
|
|
# authoritative zone answering 192.168.2.20. That exposed the internal subnet, the Gateway VIP
|
|
# and a complete service inventory to anyone who could guess a label, and made every one of
|
|
# those names a DNS-rebind primitive — which is exactly why the SOHO router at 192.168.2.1
|
|
# strips RFC1918 answers and LAN clients need per-client dnsmasq overrides to reach their own
|
|
# services. Pre-state captured at the AUTHORITY and committed:
|
|
# .standup-dns/dns-20260914-pre-externaldns-scope.txt.
|
|
#
|
|
# THE MECHANISM — per-resource exclusion, emitted by catalog/render.py from `exposure`.
|
|
# An `internal` service's rendered HTTPRoute carries
|
|
# external-dns.alpha.kubernetes.io/controller: none
|
|
# ExternalDNS v0.21.0 (the DEPLOYED app version; read off the live prod Deployment, image
|
|
# registry.k8s.io/external-dns/external-dns:v0.21.0) skips any source object whose `controller`
|
|
# annotation is present and != its own `dns-controller`:
|
|
# source/annotations/processors.go IsControllerMismatch()
|
|
# source/gateway.go Endpoints(): if annotations.IsControllerMismatch(meta, src.rtKind) { continue }
|
|
# source/service.go Endpoints(): the identical check
|
|
# The gateway check runs BEFORE any hostname is resolved, so such a route cannot become an
|
|
# endpoint for any name it declares. PROVEN LIVE on prod before this landed, with a throwaway
|
|
# hostname and no existing record at risk: an annotated route published nothing at all, and
|
|
# ADDING the annotation to a route that already had a record produced
|
|
# action=DELETE record=<name> type=A and action=DELETE record=a-<name> type=TXT
|
|
# one ~60s reconcile later, and moved nothing else in the zone.
|
|
#
|
|
# WHY PER-RESOURCE AND NOT A CLUSTER-WIDE OPT-IN FILTER. The chart's `labelFilter`
|
|
# (--label-filter) is the stronger default posture — publish only what is labelled — and
|
|
# v0.21.0's own --help confirms BOTH configured sources support it ("currently supported by
|
|
# source types crd, gateway-httproute, ... service and ambassador-host"). It is NOT set here
|
|
# because its blast radius is not this defect's. Nine publishing objects in this repo are
|
|
# hand-authored platform/apps manifests with no `exposure` field for the renderer to read, and
|
|
# seven of them hold live records: auth (oauth2-proxy), crafty, crafty-mc, grafana, headlamp,
|
|
# the apex (homepage), unifi. Setting an opt-in filter DELETES all seven on the next reconcile,
|
|
# and whether each should be publicly resolvable at all is an OPERATOR decision per name, not a
|
|
# renderer one. They are enumerated in the shrink-only register in
|
|
# conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh, so the set
|
|
# cannot grow silently — adjudicate them there, then this filter becomes the right hardening.
|
|
#
|
|
# DELIBERATELY UNTOUCHED: `policy: sync`, `txtOwnerId`, `domainFilters`. Nothing above widens
|
|
# what this instance can see or own; per-resource exclusion only ever REMOVES an object from
|
|
# this instance's view, so it cannot make prod capable of deleting a record it does not own.
|
|
# ---------------------------------------------------------------------------------------------
|
|
# THE PUBLIC-TARGET GAP — STILL OPEN, AND NOT FIXABLE HERE ON v0.21.0 (operator decision)
|
|
#
|
|
# The five `exposure: public` services (plex, gitea, seerr, keycloak, nextcloud) publish records
|
|
# that point at 192.168.2.20 — the Gateway VIP, an unroutable RFC1918 address — so they are not
|
|
# actually reachable from the internet either. Their records are LEFT EXACTLY AS THEY ARE by
|
|
# this change, for two independent reasons:
|
|
#
|
|
# 1. THE VALUE IS NOT ESTABLISHED. The one public IP this repo knows is 136.51.115.252
|
|
# (clusters/prod/kustomization.yaml, where the zone's `*.keithjackson.dev` wildcard also
|
|
# points). It belongs to the LEGACY non-k3s stack and currently serves a placeholder
|
|
# certificate. Whether prod's public services belong behind it, behind a new forward, or
|
|
# are not meant to be public at all is an operator decision. Breaking five services' DNS
|
|
# on a guess is worse than leaving a known-wrong-but-stable state.
|
|
# 2. THE MECHANISM CANNOT BE SCOPED TO THEM. `external-dns.alpha.kubernetes.io/target` is a
|
|
# real override, but in v0.21.0's gateway-httproute source it is read ONLY from the
|
|
# GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
|
|
# gw.Annotations)`) and from a ListenerSet listener — NEVER from the HTTPRoute. So there is
|
|
# no per-service target override while every public service shares the `shared` Gateway.
|
|
# Annotating the Gateway would move EVERY gateway-sourced record, which after this change
|
|
# still includes auth, crafty, grafana, headlamp and the apex — i.e. it would publish a
|
|
# routable address for admin UIs. That is strictly worse than the status quo.
|
|
#
|
|
# SO THE DECISION THE OPERATOR OWNS IS TWO-PART, and no knob is added here until the first half
|
|
# is answered, because a knob that cannot take effect invites someone to set it and expect one:
|
|
# (a) WHAT address should the five public names answer with — 136.51.115.252 as-is, a new
|
|
# port-forward/WAN address, or nothing (drop `exposure: public` to `internal` instead)?
|
|
# (b) HOW should they be separated from the non-public names that share the Gateway? Options,
|
|
# roughly in ascending cost: a SECOND Gateway (or a ListenerSet) carrying the target
|
|
# annotation, with only the public services' routes attached; or publishing the five names
|
|
# from the `crd` source (DNSEndpoint objects the renderer emits for `public` services),
|
|
# which restores per-service targeting without a second Gateway.
|
|
# Only (b) is a manifest change; (a) is a fact about the home WAN that has to be checked.
|
|
# ---------------------------------------------------------------------------------------------
|
|
# CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below.
|
|
#
|
|
# On 2026-09-04 prod's external-dns DELETED EVERY ONE of dev's public records. Both clusters ran
|
|
# `policy: sync` against the SAME Cloudflare zone, served the SAME hostnames, and — because
|
|
# NEITHER set txtOwnerId — both inherited the chart default owner id `default`. Prod therefore
|
|
# read dev's TXT registry entries as its own, found no local resource claiming those names, and
|
|
# garbage-collected them. `policy: sync` is correct and stays; what was missing was identity.
|
|
# Full write-up: infra/PROD-PROVISIONING.md §10.
|
|
#
|
|
# TWO INDEPENDENT GUARDS NOW, either of which alone prevents a recurrence:
|
|
# 1. txtOwnerId — per-environment identity; a cluster only ever touches records it owns.
|
|
# 2. domainFilters — per-environment name scope; prod cannot even see names outside its
|
|
# own subtree. Defence in depth, added after the incident.
|
|
# ---------------------------------------------------------------------------------------------
|
|
# Which names this instance may manage at all. Cloudflare-provider caveat, empirically hit
|
|
# 2026-08: this value is used for ZONE selection as well as record matching, and a filter that is
|
|
# a strict subdomain of the zone dropped the parent zone entirely ("no hosted zone matching record
|
|
# DNS Name") — so DEV keeps the zone apex `keithjackson.dev`, which is also what dev serves the
|
|
# apex of. PROD narrows to `prod.home.keithjackson.dev`. If prod's first reconcile logs no
|
|
# matching zone, that caveat still applies: prod creates NOTHING (fail-closed, never destructive)
|
|
# and the fallback is to widen prod's EXTERNALDNS_DOMAIN_FILTER back to the zone apex, leaving
|
|
# txtOwnerId as the sole guard. See clusters/base/cluster-vars.yaml.
|
|
domainFilters:
|
|
- "${EXTERNALDNS_DOMAIN_FILTER}"
|
|
# The TXT-registry owner id. MUST be distinct per environment — this is the knob whose absence
|
|
# caused the incident. Dev renders `default` (its live records are already registered under that
|
|
# owner; changing it would orphan them and duplicate every record). Prod renders `prod`.
|
|
txtOwnerId: "${EXTERNALDNS_TXT_OWNER_ID}"
|
|
policy: sync # create + update + delete records as services come and go — but ONLY
|
|
# records this owner id owns AND this domainFilter admits (see above).
|
|
# Cloudflare API token: SOPS-encrypted secret, referenced BY NAME only (OQ-4, ADR-0007).
|
|
env:
|
|
- name: CF_API_TOKEN
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: cloudflare-api-token
|
|
key: api-token
|
|
# R-DNS-02 PROVEN via the `service` source: the conformance/l2-behavioral/dns-exposure test
|
|
# creates a LoadBalancer Service with an external-dns hostname and asserts the record auto-appears
|
|
# in public DNS (then teardown + policy=sync removes it). gateway-httproute source wiring is the
|
|
# same provider/zone path; a route-based test can follow when app routes exist.
|