diff --git a/.standup-dns/dns-20260914-pre-externaldns-scope.txt b/.standup-dns/dns-20260914-pre-externaldns-scope.txt new file mode 100644 index 0000000..9ec6594 --- /dev/null +++ b/.standup-dns/dns-20260914-pre-externaldns-scope.txt @@ -0,0 +1,97 @@ +# RECOVERY BASELINE — the COMPLETE Cloudflare record set for zone `keithjackson.dev` +# as it stood IMMEDIATELY BEFORE feat/externaldns-scope-public-only is applied. +# +# WHY THIS FILE EXISTS. The branch that adds it changes which resources ExternalDNS considers, +# and ExternalDNS runs `policy: sync` — a record whose source stops matching is DELETED, not +# merely left alone. That is a one-way, outward-facing operation against a live authoritative +# zone, so the pre-state is captured and committed BEFORE the change lands, and any record the +# reconcile removes can be restored by hand from this file. It is the same discipline as +# dns-before.txt / dns-verify.txt / dns-post-l2.txt beside it, applied to a destructive-capable +# change rather than to a migration. +# +# CAPTURED 2026-09-14 from the Cloudflare API (GET /zones//dns_records, all pages) using the +# SOPS-encrypted token in platform/externaldns/cloudflare-api-token.sops.yaml. Read at the +# AUTHORITY, not through a resolver: 1.1.1.1 served stale answers for minutes after the +# 2026-09-13 apex flip and must not be trusted for this. `dig` is on the host but NOT in the +# devcontainer, which has previously produced false "no records" readings — the API is the only +# reading that is both complete and authoritative. +# +# FORMAT (5 fields, `|`-separated) — wider than the sibling files on purpose: a restore needs the +# TTL and the proxy flag, not just the value. +# +# |||| +# +# ttl 1 = Cloudflare "Automatic". proxied false = DNS-only (grey cloud), which every record here +# is; a restore that flips one to proxied would put Cloudflare's edge in front of a LAN address. +# +# 39 records. 36 of them are ExternalDNS-managed (18 A + their 18 `a-`-prefixed TXT registry +# entries), every one registered `external-dns/owner=prod` — i.e. THIS cluster owns them, which +# is why this cluster's reconcile is able to delete them. The remaining 3 belong to the legacy +# non-k3s stack and carry NO TXT registry entry, so no owner id claims them and ExternalDNS +# cannot touch them regardless of what this branch does: +# +# *.keithjackson.dev -> 136.51.115.252 legacy stack (the zone wildcard) +# brickbreak.keithjackson.dev -> 20.236.242.14 unrelated, off-prem +# dev.brickbreak… -> 20.236.242.14 unrelated, off-prem +# +# NOTE there is NO `*.home.keithjackson.dev` wildcard in the zone any more (the comment in +# clusters/prod/kustomization.yaml still says the zone has both wildcards; it does not, as of +# this capture). A single-label wildcard does not match two labels, so `*.keithjackson.dev` does +# NOT cover `.home.keithjackson.dev`: when a specific `.home` record is removed, that +# name goes to NXDOMAIN publicly rather than falling back to the legacy IP. That is the intended +# end state for `internal` services and it is also precisely why Pi-hole must already be +# answering the LAN authoritatively before this branch merges. +# +# WHICH OF THESE THIS BRANCH DELETES — the 12 rows for the six `exposure: internal` catalog +# services (jackett, pihole, qbittorrent, radarr, semaphore, sonarr): each one's A record and its +# `a-`-prefixed TXT registry entry. Nothing else. Verified live on prod against the deployed +# v0.21.0 with a throwaway hostname before any of this was authored: adding the annotation to an +# HTTPRoute that already had a published record produced +# `action=DELETE record=... type=A` and `action=DELETE record=a-... type=TXT` one reconcile +# interval (~60s) later, and nothing else in the zone moved. +# +# TO RESTORE ONE ROW BY HAND (both halves, or ExternalDNS will not recognise the record as its +# own): recreate the A record with its value/ttl/proxied from this file, then recreate the +# matching `a-` TXT with its content VERBATIM — the `heritage=…,external-dns/owner=prod, +# external-dns/resource=…` string is what re-establishes ownership. An A record restored without +# its TXT is unowned and will not be reconciled; a TXT restored with a wrong `resource=` value +# points the registry at a source object that does not exist. +A|*.keithjackson.dev|136.51.115.252|1|false +A|auth.home.keithjackson.dev|192.168.2.20|1|false +A|brickbreak.keithjackson.dev|20.236.242.14|1|false +A|crafty-mc.home.keithjackson.dev|192.168.2.22|1|false +A|crafty.home.keithjackson.dev|192.168.2.20|1|false +A|dev.brickbreak.keithjackson.dev|20.236.242.14|1|false +A|gitea.home.keithjackson.dev|192.168.2.20|1|false +A|grafana.home.keithjackson.dev|192.168.2.20|1|false +A|headlamp.home.keithjackson.dev|192.168.2.20|1|false +A|home.keithjackson.dev|192.168.2.20|1|false +A|jackett.home.keithjackson.dev|192.168.2.20|1|false +A|keycloak.home.keithjackson.dev|192.168.2.20|1|false +A|nextcloud.home.keithjackson.dev|192.168.2.20|1|false +A|pihole.home.keithjackson.dev|192.168.2.20|1|false +A|plex.home.keithjackson.dev|192.168.2.20|1|false +A|qbittorrent.home.keithjackson.dev|192.168.2.20|1|false +A|radarr.home.keithjackson.dev|192.168.2.20|1|false +A|seerr.home.keithjackson.dev|192.168.2.20|1|false +A|semaphore.home.keithjackson.dev|192.168.2.20|1|false +A|sonarr.home.keithjackson.dev|192.168.2.20|1|false +A|unifi.home.keithjackson.dev|192.168.2.21|1|false +TXT|a-auth.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy"|1|false +TXT|a-crafty-mc.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp"|1|false +TXT|a-crafty.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty"|1|false +TXT|a-gitea.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000"|1|false +TXT|a-grafana.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana"|1|false +TXT|a-headlamp.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp"|1|false +TXT|a-home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage"|1|false +TXT|a-jackett.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117"|1|false +TXT|a-keycloak.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak"|1|false +TXT|a-nextcloud.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80"|1|false +TXT|a-pihole.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80"|1|false +TXT|a-plex.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400"|1|false +TXT|a-qbittorrent.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080"|1|false +TXT|a-radarr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878"|1|false +TXT|a-seerr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055"|1|false +TXT|a-semaphore.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000"|1|false +TXT|a-sonarr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989"|1|false +TXT|a-unifi.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp"|1|false diff --git a/apps/wave-0/jackett.yaml b/apps/wave-0/jackett.yaml index 96a4c5d..9d2574d 100644 --- a/apps/wave-0/jackett.yaml +++ b/apps/wave-0/jackett.yaml @@ -268,6 +268,7 @@ metadata: annotations: homelab.keithjackson.dev/auth-mode: forward-auth homelab.keithjackson.dev/auth-groups: media-users,platform-admins + external-dns.alpha.kubernetes.io/controller: none spec: parentRefs: - name: shared diff --git a/apps/wave-0/pihole.yaml b/apps/wave-0/pihole.yaml index edf537e..c883657 100644 --- a/apps/wave-0/pihole.yaml +++ b/apps/wave-0/pihole.yaml @@ -223,6 +223,7 @@ metadata: annotations: homelab.keithjackson.dev/auth-mode: forward-auth homelab.keithjackson.dev/auth-groups: platform-admins + external-dns.alpha.kubernetes.io/controller: none spec: parentRefs: - name: shared diff --git a/apps/wave-0/qbittorrent.yaml b/apps/wave-0/qbittorrent.yaml index bf0940f..6551fcf 100644 --- a/apps/wave-0/qbittorrent.yaml +++ b/apps/wave-0/qbittorrent.yaml @@ -344,6 +344,7 @@ metadata: annotations: homelab.keithjackson.dev/auth-mode: forward-auth homelab.keithjackson.dev/auth-groups: media-users,platform-admins + external-dns.alpha.kubernetes.io/controller: none spec: parentRefs: - name: shared diff --git a/apps/wave-1/radarr.yaml b/apps/wave-1/radarr.yaml index 6cd03b3..e84f565 100644 --- a/apps/wave-1/radarr.yaml +++ b/apps/wave-1/radarr.yaml @@ -348,6 +348,7 @@ metadata: annotations: homelab.keithjackson.dev/auth-mode: forward-auth homelab.keithjackson.dev/auth-groups: media-users,platform-admins + external-dns.alpha.kubernetes.io/controller: none spec: parentRefs: - name: shared diff --git a/apps/wave-1/semaphore.yaml b/apps/wave-1/semaphore.yaml index 9b6a820..d5b0722 100644 --- a/apps/wave-1/semaphore.yaml +++ b/apps/wave-1/semaphore.yaml @@ -227,6 +227,7 @@ metadata: annotations: homelab.keithjackson.dev/auth-mode: oidc homelab.keithjackson.dev/auth-groups: platform-admins + external-dns.alpha.kubernetes.io/controller: none spec: parentRefs: - name: shared diff --git a/apps/wave-1/sonarr.yaml b/apps/wave-1/sonarr.yaml index 84028d7..5b0d98b 100644 --- a/apps/wave-1/sonarr.yaml +++ b/apps/wave-1/sonarr.yaml @@ -348,6 +348,7 @@ metadata: annotations: homelab.keithjackson.dev/auth-mode: forward-auth homelab.keithjackson.dev/auth-groups: media-users,platform-admins + external-dns.alpha.kubernetes.io/controller: none spec: parentRefs: - name: shared diff --git a/catalog/render.py b/catalog/render.py index 4c12223..7b5bc48 100644 --- a/catalog/render.py +++ b/catalog/render.py @@ -80,6 +80,18 @@ BULK_NFS_SERVER = "${BULK_NFS_SERVER}" BULK_NFS_MOUNT_OPTIONS = ["nfsvers=4.1", "hard", "noatime", "timeo=600", "retrans=2"] GATEWAY_NAME = "shared" GATEWAY_NS = "gateway" +# ExternalDNS per-resource scoping (R-DNS-03; see _externaldns_route_annotations for the full WHY +# and for the v0.21.0 source citation). The KEY is external-dns's own `ControllerKey` +# (annotationPrefix + "controller"); external-dns skips any source object whose value is PRESENT +# and != `ControllerValue` ("dns-controller"), so the VALUE only has to be some other string. +# `none` is chosen because it reads as intent in `kubectl get httproute -o yaml` rather than as a +# typo — "no DNS controller owns this route" — and because the one value that must never be +# emitted here is external-dns's own, which a literal `none` cannot drift into. +EXTERNALDNS_CONTROLLER_KEY = "external-dns.alpha.kubernetes.io/controller" +EXTERNALDNS_CONTROLLER_DISOWN = "none" +# The value external-dns claims for itself. Held here ONLY so the renderer and the L0 guard can +# assert the disown value is not it; never emitted. +EXTERNALDNS_CONTROLLER_CLAIMED = "dns-controller" KP_RELEASE = "kube-prometheus-stack" # ServiceMonitor/PrometheusRule discovery label MONITORING_NS = "monitoring" # where Prometheus runs; the only admitted scrape source @@ -836,12 +848,15 @@ def r_in_cluster_client_policies(s: dict) -> list: def _route_meta(s: dict, route: dict) -> dict: + """HTTPRoute metadata: auth wiring hints PLUS the ExternalDNS scoping derived from + `exposure` (_externaldns_route_annotations — R-DNS-03: an `internal` service renders no + public DNS record, and now cannot).""" meta = { "name": f"{s['name']}-{route['port']}", "namespace": s["name"], "labels": {"app.kubernetes.io/name": s["name"]}, } - ann = _auth_annotations(s) + ann = {**_auth_annotations(s), **_externaldns_route_annotations(s)} if ann: meta["annotations"] = ann return meta @@ -1039,6 +1054,64 @@ def _auth_annotations(s: dict) -> dict: return {} +def _externaldns_route_annotations(s: dict) -> dict: + """ExternalDNS scoping, rendered FROM `exposure` (renderer-contract §exposure; R-DNS-03). + + `internal` renders NO public DNS record. That sentence was in the contract from the start and + nothing enforced it: ExternalDNS watched EVERY HTTPRoute the shared Gateway carried, so an + `internal` service's route published `.${BASE_DOMAIN}` into the PUBLIC authoritative + Cloudflare zone, answering with the Gateway VIP — an RFC1918 address. Six of them did + (jackett, pihole, qbittorrent, radarr, semaphore, sonarr), which published the internal + subnet, the Gateway VIP and a full service inventory to anyone who asked, and made every one + of those names a DNS-rebind primitive that the SOHO router strips on the way back in. + + THE MECHANISM, verified against the DEPLOYED version (chart >=1.21.0 <1.22.0, app v0.21.0 — + `registry.k8s.io/external-dns/external-dns:v0.21.0`, read off the live prod Deployment), not + from memory. external-dns v0.21.0 `source/annotations/processors.go`: + + func IsControllerMismatch(entity metav1.ObjectMetaAccessor, rType string) bool { + value, ok := entity.GetObjectMeta().GetAnnotations()[ControllerKey] + if ok && value != ControllerValue { ...; return true } + return false + } + + with `ControllerKey = "external-dns.alpha.kubernetes.io/controller"` and + `ControllerValue = "dns-controller"`. `source/gateway.go` calls it per route inside + `Endpoints()`, BEFORE any hostname is resolved: + + if annotations.IsControllerMismatch(meta, src.rtKind) { continue } + + so an HTTPRoute carrying any other value is not merely unpublished — it never becomes an + endpoint at all, for any hostname it names. `source/service.go` applies the identical check, + so the same annotation scopes the `service` source too if a rendered Service ever grows an + external-dns hostname. + + WHY THIS IS STRUCTURAL AND NOT INCIDENTAL. The annotation is derived HERE, from `exposure`, + with no per-service list anywhere: every HTTPRoute this renderer emits for an `internal` + service carries it, and ci/render-golden.sh proves `apps/` reproduces byte-for-byte from + `catalog/`, so an `internal` service cannot EXIST in the applied tree without it. Flipping + `exposure: internal` to `public` is the only edit that can publish a name, which is exactly + the operator-facing declaration R-DNS-03 is written in terms of. The L0 check + conformance/tests/l0-static/internal-services-publish-no-public-dns re-proves all of that on + every gate run, including that this function reads `exposure` rather than naming services. + + WHY NOT A CLUSTER-WIDE OPT-IN FILTER (`--label-filter`, chart `labelFilter`). It exists, and + v0.21.0's own `--help` names both sources in use as supporting it ("currently supported by + source types crd, gateway-httproute, ... service and ambassador-host"), so a "publish only + what is labelled" posture is available and is STRONGER by default. It is not taken here + because its blast radius is not this defect's: seven published names come from hand-authored + platform/apps manifests that carry no `exposure` field and that this renderer never sees + (auth/oauth2-proxy, crafty + crafty-mc, grafana, headlamp, the homepage apex, unifi). An + opt-in filter DELETES all seven the moment it is set, and whether each should stay published + is an operator decision, not a renderer one. Per-resource exclusion has exactly the blast + radius of the reported defect. The filter remains the available hardening once those seven + are adjudicated — see platform/externaldns/helmrelease.yaml. + """ + if s["exposure"] == "internal": + return {EXTERNALDNS_CONTROLLER_KEY: EXTERNALDNS_CONTROLLER_DISOWN} + return {} + + def r_servicemonitor(s: dict): """ServiceMonitor — rendered ONLY when the SD says WHERE metrics are served. @@ -2078,6 +2151,96 @@ def _selftest() -> int: "a non-exposed service must not render a gateway-ingress allow (least privilege)", ) + # 4b-dns) exposure -> ExternalDNS SCOPING on the HTTPRoute (R-DNS-03). `internal` renders no + # public DNS record and must be INCAPABLE of rendering one: its route carries + # external-dns's own controller annotation set to something other than `dns-controller`, + # which v0.21.0's gateway-httproute source (source/gateway.go -> + # annotations.IsControllerMismatch) uses to skip the route before any hostname is + # resolved. `public` must NOT carry it, or the five public names would vanish instead. + # Both directions are asserted here because only asserting the internal half would pass + # just as happily if the renderer stamped EVERY route and silently unpublished the zone. + dns_int_docs = render( + { + "name": "di", + "source": {"image": "a:1"}, + "exposure": "internal", + "ingressRoutes": [{"host": "di.home.keithjackson.dev", "port": 80}], + "auth": {"mode": "none"}, + } + ) + dns_int_routes = [d[1] for d in dns_int_docs if d[1]["kind"] == "HTTPRoute"] + check( + len(dns_int_routes) == 1, + "an internal exposed service must still render its HTTPRoute (LAN reachability is " + "unchanged — only the PUBLIC record is withdrawn)", + ) + if dns_int_routes: + iann = dns_int_routes[0]["metadata"].get("annotations", {}) + check( + iann.get(EXTERNALDNS_CONTROLLER_KEY) == EXTERNALDNS_CONTROLLER_DISOWN, + "an internal service's HTTPRoute must carry the ExternalDNS disown annotation", + ) + check( + iann.get(EXTERNALDNS_CONTROLLER_KEY) != EXTERNALDNS_CONTROLLER_CLAIMED, + "the disown value must never be external-dns's own `dns-controller` (that CLAIMS " + "the route rather than releasing it — the one value that inverts the guarantee)", + ) + dns_pub_docs = render( + { + "name": "dp", + "source": {"image": "a:1"}, + "exposure": "public", + "ingressRoutes": [{"host": "dp.home.keithjackson.dev", "port": 80}], + "auth": {"mode": "native"}, + } + ) + dns_pub_routes = [d[1] for d in dns_pub_docs if d[1]["kind"] == "HTTPRoute"] + check(len(dns_pub_routes) == 1, "a public service must render its HTTPRoute") + if dns_pub_routes: + check( + EXTERNALDNS_CONTROLLER_KEY + not in dns_pub_routes[0]["metadata"].get("annotations", {}), + "a public service's HTTPRoute must NOT carry the ExternalDNS disown annotation", + ) + # The DEFAULT posture is `internal` (renderer-contract "Default posture"), so a + # ServiceDefinition that simply omits `exposure` must also be unpublishable. Omitting the + # field is the likeliest way a new service arrives, and it is the case a per-service + # allow-list would miss. + dns_dflt_docs = render( + { + "name": "dd", + "source": {"image": "a:1"}, + "ingressRoutes": [{"host": "dd.home.keithjackson.dev", "port": 80}], + "auth": {"mode": "none"}, + } + ) + dns_dflt_route = next(d[1] for d in dns_dflt_docs if d[1]["kind"] == "HTTPRoute") + check( + dns_dflt_route["metadata"]["annotations"].get(EXTERNALDNS_CONTROLLER_KEY) + == EXTERNALDNS_CONTROLLER_DISOWN, + "a ServiceDefinition that OMITS `exposure` defaults to internal and must also be " + "unpublishable (default posture, renderer-contract)", + ) + # The scoping annotation must COEXIST with the auth hints rather than replace them: both are + # merged into one `annotations` map, and a forward-auth internal app is the overlap case. + dns_fa_docs = render( + { + "name": "df", + "source": {"image": "a:1"}, + "exposure": "internal", + "ingressRoutes": [{"host": "df.home.keithjackson.dev", "port": 80}], + "auth": {"mode": "forward-auth", "groups": ["platform-admins"]}, + } + ) + dns_fa_ann = next( + d[1] for d in dns_fa_docs if d[1]["kind"] == "HTTPRoute" + )["metadata"]["annotations"] + check( + dns_fa_ann.get(EXTERNALDNS_CONTROLLER_KEY) == EXTERNALDNS_CONTROLLER_DISOWN + and dns_fa_ann.get(f"{LBL}/auth-mode") == "forward-auth", + "DNS scoping and auth-mode hints must COEXIST on one internal forward-auth route", + ) + # 4c) inClusterClients -> one ADDITIVE, least-privilege ingress allow per entry that admits a # PLATFORM controller's NAMESPACE (not the reserved `ingress` identity, not `all`) to ONE # declared port. This is what keeps a cluster whose GitOps source is an in-cluster Service diff --git a/conformance/tests/l0-static/internal-services-publish-no-public-dns/chainsaw-test.yaml b/conformance/tests/l0-static/internal-services-publish-no-public-dns/chainsaw-test.yaml new file mode 100644 index 0000000..6d82639 --- /dev/null +++ b/conformance/tests/l0-static/internal-services-publish-no-public-dns/chainsaw-test.yaml @@ -0,0 +1,136 @@ +# ============================================================================= +# L0 STATIC — no service declared `internal` can produce a PUBLIC DNS record +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# +# THE DEFECT THIS CLOSES. docs/09 R-DNS-03 says `internal` services are unreachable from the +# public internet, that external names resolve to the PUBLIC IP, and that Pi-hole provides the +# internal split-horizon answers. schemas/renderer-contract.md §exposure says an `internal` +# service renders "no public DNS record". NOTHING ENFORCED EITHER SENTENCE. ExternalDNS +# (platform/externaldns/helmrelease.yaml, `sources: [gateway-httproute, service]`, +# `policy: sync`, provider cloudflare) considered EVERY HTTPRoute on the shared Gateway, so six +# `exposure: internal` services — jackett, pihole, qbittorrent, radarr, semaphore, sonarr — each +# published `.home.keithjackson.dev` into the PUBLIC authoritative Cloudflare zone, +# answering with the Gateway VIP 192.168.2.20. Verified at the authority on 2026-09-14; the +# pre-state is committed at .standup-dns/dns-20260914-pre-externaldns-scope.txt. +# +# Three things were wrong at once, and none of them was visible from inside the cluster: +# * the internal subnet, the Gateway VIP and a complete service inventory were publicly +# enumerable by anyone who could guess or brute-force a label; +# * a public name answering with an RFC1918 address IS a DNS-rebind primitive, which is why the +# SOHO router at 192.168.2.1 strips those answers and LAN clients need per-client dnsmasq +# overrides to reach their own services; +# * the guarantee was only ever prose. `exposure` was the operator's one declaration of intent +# and it had NO effect on DNS whatsoever, so "internal" meant nothing an attacker had to +# respect. +# +# WHY THE FIX NEEDS A GUARD AND NOT JUST A COMMIT. The mechanism is an annotation on a rendered +# HTTPRoute. An annotation is exactly the kind of thing that survives review by looking harmless +# and then disappears in a refactor of `_route_meta`, a hand-edit of apps/, or a new service added +# by copying a public one. Nothing about its ABSENCE is loud: the service keeps working on the LAN, +# the Kustomization stays Ready, and the only symptom is a public record nobody is looking at. The +# 2026-09-13 ServiceMonitor incident in this same tree is the precedent — twelve monitors pointed +# at a port that did not exist for weeks, silently, because the only thing asserting the wiring was +# prose. This test makes the DNS guarantee a gate condition instead. +# +# GIVEN every ServiceDefinition in catalog/ and every HTTPRoute and external-dns-annotated +# Service this repo authors, +# WHEN each catalog service's rendered routes are resolved in the applied tree (apps/), +# THEN (1) every route of an `exposure: internal` service carries ExternalDNS's own controller +# annotation set to something OTHER than `dns-controller`, which external-dns v0.21.0 +# uses to skip the source object outright (source/gateway.go -> +# annotations.IsControllerMismatch, checked before any hostname is resolved), so the +# route cannot become an endpoint for ANY name it declares; +# (2) NO route of an `exposure: public` service carries it — asserted explicitly, because a +# renderer that stamped every route would satisfy (1) perfectly while silently +# withdrawing the five public services' records, and `policy: sync` would delete them; +# (3) the same holds for the `service` source: an internal service's rendered Service may +# not carry an external-dns hostname annotation without the disown annotation beside it. +# Two sources are configured, so closing only the route half is half a guarantee; +# (4) the annotation is DERIVED FROM `exposure` in catalog/render.py and from nothing else — +# the emitting function must read `exposure`, must not name any catalog service, and +# must not be able to emit external-dns's own `dns-controller` value (the one value that +# CLAIMS a route rather than releasing it, inverting the guarantee while looking +# plausible). A per-service list is how twelve ServiceMonitors acquired one identical +# wrong port name in a single edit; +# (5) THE RENDERER IS PROVEN END-TO-END on synthetic ServiceDefinitions, by invoking the +# SHIPPED catalog/render.py as a subprocess: an `internal` fixture must yield the +# annotation, a `public` fixture must not, and a fixture that OMITS `exposure` must also +# yield it (the contract's default posture is `internal`, and omitting the field is the +# likeliest way a new service arrives). Without this the check would still pass if the +# renderer stopped working and someone hand-maintained apps/; +# (6) every HTTPRoute or annotated Service in the tree that the catalog does NOT render is +# DECLARED in KNOWN_NON_CATALOG_PUBLISHERS with what it publishes and why it is out of +# scope. Those are hand-authored platform/apps manifests with no `exposure` field for a +# renderer to read, and they are the seven names that REMAIN publicly advertised with an +# RFC1918 answer after this change. The register is shrink-only: an entry that stops +# matching a real object FAILS as stale, so the set of unadjudicated publishers cannot +# quietly grow, and adding a new one is a review event; +# AND (7) the check is LOAD-BEARING: this Test DECLARES `conformance.home.k3s/scope: whole-repo`, +# so ci/l0.sh Stage 3b's `--include-scope whole-repo` unions it into EVERY gate run, and +# ci/gate.sh blocks the merge when the l0 stage fails. It deliberately adds NO stage to +# ci/l0.sh — tests/l0-static/whole-repo-invariants-wired fails the gate if anyone does. +# +# WHY SCOPE, NOT REQUIREMENTS: ci/touched-requirements.sh's path map keys apps/* to NOTHING, and +# apps/ is exactly where the rendered routes live. A requirement-scoped selection would therefore +# filter this check out of the change most likely to break it — a hand-edit of a rendered route. +# Same reasoning as its whole-repo siblings. +# +# STATUS: EXECUTABLE. `check.sh` is an offline assertion script (PyYAML, as catalog/render.py and +# the sibling l0-static checks use); conformance/run.sh --layer l0 and ci/l0.sh run it directly, +# and this Chainsaw Test wraps the SAME script. +# +# NOT proven (honest gaps; this is static, and it runs with no cluster and no network): +# - That the record is actually GONE from Cloudflare. That is a live read against an external +# authority and belongs in L2 beside conformance/tests/l2-behavioral/dns-exposure, which owns +# the positive direction (a public name DOES appear). The negative direction was proven +# MANUALLY on prod against the deployed v0.21.0 before this landed — a throwaway HTTPRoute +# carrying the annotation produced no record at all, and adding the annotation to a route that +# already had one produced `action=DELETE` for both its A record and its `a-` TXT registry +# entry within one ~60s reconcile — but a manual proof is not a gate. +# - That an `internal` name is unreachable from the internet. No DNS record is necessary for +# reachability; a port-forward plus a hardcoded IP still reaches the Gateway. This check owns +# the NAME axis only. R-NET-04 / the edge port-forward posture owns the other. +# - Anything about the seven non-catalog publishers' own correctness. The register records WHAT +# each publishes so the claim is reviewable, which is all a static check can offer. +# - Anything about the five `public` services' TARGET. They still resolve to the Gateway VIP +# rather than a routable address; that is an open operator decision recorded in +# platform/externaldns/helmrelease.yaml, and asserting a target here would hard-code a guess. +# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06). +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: internal-services-publish-no-public-dns + labels: + conformance.home.k3s/R-DNS-02: "true" + conformance.home.k3s/R-DNS-03: "true" + annotations: + conformance.home.k3s/requirements: "R-DNS-02, R-DNS-03" + conformance.home.k3s/layer: "L0" + # WHOLE-REPO SCOPE — this invariant quantifies over every ServiceDefinition in catalog/, every + # HTTPRoute in apps/wave-* and platform/* and runbooks/, and every external-dns-annotated + # Service (apps/vms/unifi, platform/crafty). No single requirement path-maps to that set, and + # ci/touched-requirements.sh maps apps/* — where the rendered routes live — to no requirement + # at all, so a requirement-scoped selection would skip this check on precisely the changes + # that can break it. + conformance.home.k3s/scope: "whole-repo" + conformance.home.k3s/safety-critical: "true" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + A service the operator declared `internal` cannot produce a public DNS record: the + guarantee is derived from `exposure` by catalog/render.py, present on every rendered route + in the applied tree, absent from every public one, proven end-to-end on synthetic fixtures, + and the hand-authored manifests the catalog does not render are enumerated in a shrink-only + register so the unadjudicated set cannot grow silently. +spec: + steps: + - name: assert-no-internal-service-can-publish-a-public-dns-record + description: >- + Resolve every catalog service's rendered HTTPRoutes in apps/, assert the ExternalDNS + disown annotation on every internal one and on no public one, extend the same rule to the + `service` source, assert catalog/render.py derives the annotation from `exposure` rather + than from a service list, prove the shipped renderer on internal/public/default-posture + fixtures, and ratchet the register of hand-authored publishers the catalog does not render. + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh b/conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh new file mode 100755 index 0000000..9e3855b --- /dev/null +++ b/conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh @@ -0,0 +1,458 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 STATIC CHECK — no service declared `internal` can produce a PUBLIC DNS record. +# See chainsaw-test.yaml's header for the full WHY. In one paragraph: +# +# docs/09 R-DNS-03 and schemas/renderer-contract.md §exposure both say an `internal` service +# renders no public DNS record. Nothing enforced it: ExternalDNS watched every HTTPRoute on the +# shared Gateway, so six internal services published `.home.keithjackson.dev` into the +# public Cloudflare zone answering with the RFC1918 Gateway VIP (verified at the authority +# 2026-09-14; pre-state committed at .standup-dns/dns-20260914-pre-externaldns-scope.txt). The fix +# derives ExternalDNS's controller annotation from `exposure` in catalog/render.py; this check is +# what keeps it derived, present, and scoped. +# +# ASSERTIONS +# (0) the inventory is non-empty and the parse contract holds (anti-vacuous-pass); +# (1) every rendered route of an `internal` service carries the disown annotation; +# (2) NO rendered route of a `public` service carries it — a renderer stamping everything would +# satisfy (1) and silently delete the five public records under policy: sync; +# (3) the `service` source half: an internal service's Service may not carry an external-dns +# hostname annotation without the disown annotation beside it; +# (4) catalog/render.py DERIVES the annotation from `exposure` — the emitting function reads that +# field, names no catalog service, and cannot emit external-dns's own `dns-controller`; +# (5) the SHIPPED renderer is proven end-to-end on synthetic ServiceDefinitions (internal, +# public, and `exposure` OMITTED — the contract's default posture is internal); +# (6) every HTTPRoute / annotated Service the catalog does NOT render is declared in +# KNOWN_NON_CATALOG_PUBLISHERS — shrink-only, stale entries FAIL; +# (7) the check is LOAD-BEARING: whole-repo scope declared in chainsaw-test.yaml, honoured by +# ci/l0.sh's --include-scope union, and ci/gate.sh blocks on an l0 failure. NO hand-written +# ci/l0.sh stage (whole-repo-invariants-wired fails the gate if one appears). +# +# WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC, offline, cluster-free): +# - That the record is gone from Cloudflare. Live read; belongs beside +# conformance/tests/l2-behavioral/dns-exposure, which owns the positive direction. The negative +# direction was proven manually on prod against the deployed v0.21.0 (a route carrying the +# annotation produced no record; adding it to a route that had one produced action=DELETE for +# both the A and its `a-` TXT within ~60s) — but a manual proof is not a gate. +# - That an internal name is UNREACHABLE. A port-forward plus a literal IP needs no DNS record. +# This check owns the NAME axis only. +# - Anything about the five public services' TARGET. They still answer with the Gateway VIP; the +# routable-target decision is open and recorded in platform/externaldns/helmrelease.yaml. +# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06). +# +# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns +# into rc 3 = UNWIRED, never green, because this test declares the whole-repo scope) only if +# catalog/ or apps/ is absent — i.e. the subject tree has vanished, which is not a pass. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" + +# Honest SKIP (never green for a scope-declaring test) if the subject trees are not there at all. +if [ ! -d "${ROOT}/catalog" ] || [ ! -d "${ROOT}/apps" ] || [ ! -f "${ROOT}/catalog/render.py" ]; then + echo "check SKIPPED: catalog/, apps/ or catalog/render.py absent — nothing to audit" >&2 + exit 3 +fi + +cd "${ROOT}" +"${PYTHON}" - "${ROOT}" <<'PY' +"""Static audit: an `internal` ServiceDefinition cannot produce a public DNS record. + +PyYAML (the same dependency as catalog/render.py and the sibling l0-static checks) rather than +line-based parsing: an HTTPRoute's annotations and a ServiceDefinition's exposure are nested +structures, and a line-based reader of them is precisely the approximate parser that silently +stops matching — the failure class this check exists to prevent. +""" +from __future__ import annotations + +import re +import subprocess +import sys +import tempfile +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + print("check FAIL: PyYAML is required (same dependency as catalog/render.py)", + file=sys.stderr) + raise SystemExit(1) + +ROOT = Path(sys.argv[1]) +sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree + +FAILURES: list[str] = [] +CHECKS = 0 + +# external-dns's own constants (v0.21.0 source/annotations/annotations.go): +# ControllerKey = annotationPrefix + "controller" -> the key below +# ControllerValue = "dns-controller" -> the value that CLAIMS an object +# source/annotations/processors.go IsControllerMismatch skips a source object when the key is +# PRESENT and the value is NOT ControllerValue. So "disowned" == key present, value != claimed. +CONTROLLER_KEY = "external-dns.alpha.kubernetes.io/controller" +CONTROLLER_CLAIMED = "dns-controller" +HOSTNAME_KEY = "external-dns.alpha.kubernetes.io/hostname" + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(msg) + + +def read(p: Path) -> str: + try: + return p.read_text() + except (OSError, UnicodeDecodeError): + return "" + + +def docs_of(p: Path): + """Every YAML document in a file, tolerating the non-manifest files in the tree.""" + try: + return [d for d in yaml.safe_load_all(read(p)) if isinstance(d, dict)] + except yaml.YAMLError: + return [] + + +# ============================================================================================= +# REGISTER — KNOWN_NON_CATALOG_PUBLISHERS. Shrink-only. +# +# These are HAND-AUTHORED HTTPRoutes and Services that put a name into the same public Cloudflare +# zone but that catalog/render.py does not render, so there is no `exposure` field for the renderer +# to derive anything from and this check cannot adjudicate them. That is a real limit, not a pass — +# so each one is DECLARED, with the name it publishes and why it is out of this check's scope, and +# the register can only shrink. An entry naming an object that no longer exists FAILS as stale, so +# a deleted publisher cannot leave a lie behind; a NEW publisher fails until it is declared, so the +# unadjudicated set cannot grow without a review event. Same ratchet reasoning as +# KNOWN_EXTERNAL_MONITORS in servicemonitor-ports-resolve. +# +# EVERY ENTRY HERE IS A NAME THAT STILL ANSWERS WITH AN RFC1918 ADDRESS IN A PUBLIC ZONE. That is +# the honest state after this change, and it is deliberate: converting them is an operator +# decision per name (is grafana meant to be publicly resolvable at all?), not a renderer change, +# and a cluster-wide ExternalDNS opt-in filter — the available hardening, see +# platform/externaldns/helmrelease.yaml — would DELETE all of them at once the moment it is set. +# +# Keyed by ":/". +# ============================================================================================= +KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = { + "platform/oauth2-proxy/httproute.yaml:oauth2-proxy/oauth2-proxy": + "publishes auth.${BASE_DOMAIN} -> the Gateway VIP. The oauth2-proxy login endpoint is a " + "platform component, not a catalog service; it must be reachable by every browser that " + "any forward-auth app redirects, so whether it is PUBLICLY resolvable is coupled to " + "whether the five public services are — an operator decision, not a renderer one.", + "platform/keycloak/httproute.yaml:keycloak/keycloak": + "publishes keycloak.${BASE_DOMAIN} -> the Gateway VIP, and it is the route that actually " + "OWNS that record (the TXT registry entry reads resource=httproute/keycloak/keycloak, not " + "the catalog-rendered keycloak-8080 route, because external-dns merges duplicate " + "hostnames and this one won). Keycloak is `exposure: public` in the catalog, so the name " + "is meant to be public either way; it is listed because the PUBLISHING object is the " + "hand-authored platform route, which carries no exposure field.", + "platform/monitoring/grafana-httproute.yaml:monitoring/grafana": + "publishes grafana.${BASE_DOMAIN} -> the Gateway VIP. An admin dashboard; almost " + "certainly should not be publicly resolvable, but it is not a catalog service and " + "withdrawing it is an operator call with the same LAN-resolution prerequisite as the six " + "internal services (Pi-hole must answer first).", + "platform/headlamp/httproute.yaml:headlamp/headlamp": + "publishes headlamp.${BASE_DOMAIN} -> the Gateway VIP. A Kubernetes web UI with native " + "Keycloak OIDC; same adjudication as grafana.", + "platform/homepage/httproute.yaml:homepage/homepage": + "publishes the APEX ${BASE_DOMAIN} itself -> the Gateway VIP, via the https-apex " + "listener. The apex is not a per-service name and cannot be withdrawn by an exposure " + "declaration.", + "runbooks/serving/httproute.yaml:runbooks/runbooks": + "declares NO hostnames and inherits the https-apex listener's ${BASE_DOMAIN}, so it " + "contributes to the apex record rather than a name of its own (the apex TXT registry " + "entry names the homepage route). Deliberately un-authenticated — see the manifest's own " + "header for why gating runbooks on Keycloak is a loop with no exit.", + "platform/crafty/httproute.yaml:crafty/crafty": + "publishes crafty.${BASE_DOMAIN} -> the Gateway VIP (the Minecraft admin UI, gated by an " + "oauth2-proxy sidecar). Platform component, no exposure field.", + "platform/crafty/service-l4.yaml:crafty/crafty-l4-tcp": + "the `service` SOURCE, not a route: publishes crafty-mc.${BASE_DOMAIN} -> " + "${CRAFTY_L4_VIP} (192.168.2.22) for the raw Minecraft TCP/UDP game ports the HTTPS " + "Gateway cannot express. A game server is the one case where a public name may be " + "genuinely wanted; the address it answers with is still RFC1918.", + "apps/vms/unifi/50-service-l4.yaml:unifi/unifi-l4-tcp": + "the `service` source: publishes unifi.${BASE_DOMAIN} -> ${UNIFI_L4_VIP} " + "(192.168.2.21) for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored " + "under apps/vms (a VM, not a rendered container service); the catalog's only VM entry is " + "vm-demo, which declares no ingressRoutes and publishes nothing.", +} + +# ============================================================================================= +# (0) INVENTORY — the catalog, and every HTTPRoute / annotated Service in the tree. +# ============================================================================================= +CATALOG = ROOT / "catalog" +sds: dict[str, dict] = {} +for p in sorted(CATALOG.glob("*.yaml")): + for d in docs_of(p): + if "name" in d and "source" in d: + sds[d["name"]] = d + +check(len(sds) > 0, "inventory: catalog/ yielded no ServiceDefinition — parse contract broken") + +# The default posture is `internal` (renderer-contract "Default posture"), so an ABSENT field is +# internal, never "unclassified". Getting this wrong in the check would be the same bug as in the +# renderer: a new service that omits exposure would go unaudited. +exposure_of = {n: (sd.get("exposure") or "internal") for n, sd in sds.items()} +# (namespace, routename) -> service, for the routes the renderer emits: one per ingressRoute, +# named "-" in namespace "" (catalog/render.py _route_meta). +expected_routes: dict[tuple[str, str], str] = {} +for n, sd in sds.items(): + for r in sd.get("ingressRoutes") or []: + expected_routes[(n, f"{n}-{r['port']}")] = n + +internal_with_routes = sorted({s for (_, _), s in expected_routes.items() + if exposure_of[s] == "internal"}) +public_with_routes = sorted({s for (_, _), s in expected_routes.items() + if exposure_of[s] == "public"}) +check(len(internal_with_routes) > 0, + "inventory: no `internal` catalog service declares an ingressRoute — the subject set of " + "this check is empty, which means either the catalog changed shape or the parse is wrong") +check(len(public_with_routes) > 0, + "inventory: no `public` catalog service declares an ingressRoute — assertion (2) would be " + "vacuous, so the parse is presumed wrong rather than the repo") + +SCAN_DIRS = ["apps", "platform", "runbooks"] +routes: list[tuple[str, dict]] = [] # (relpath, HTTPRoute doc) +dns_services: list[tuple[str, dict]] = [] # (relpath, Service doc carrying an external-dns ann) +for sub in SCAN_DIRS: + base = ROOT / sub + if not base.is_dir(): + continue + for p in sorted(base.rglob("*.yaml")): + rel = p.relative_to(ROOT).as_posix() + for d in docs_of(p): + kind = d.get("kind") + md = d.get("metadata") or {} + ann = md.get("annotations") or {} + if kind == "HTTPRoute": + routes.append((rel, d)) + elif kind == "Service" and any( + a.startswith("external-dns.alpha.kubernetes.io/") for a in ann + ): + dns_services.append((rel, d)) + +check(len(routes) > 0, "inventory: no HTTPRoute found under apps/ platform/ runbooks/ — parse " + "contract broken (the tree is known to author several)") + + +def ident(d: dict) -> tuple[str, str]: + md = d.get("metadata") or {} + return (md.get("namespace") or "", md.get("name") or "") + + +def annotations_of(d: dict) -> dict: + return (d.get("metadata") or {}).get("annotations") or {} + + +def is_disowned(d: dict) -> bool: + """True iff external-dns v0.21.0 would SKIP this object: the controller annotation is present + and its value is not external-dns's own. Absent == published (the default), which is the whole + point — the annotation is the only thing standing between a route and the public zone.""" + v = annotations_of(d).get(CONTROLLER_KEY) + return v is not None and v != CONTROLLER_CLAIMED + + +# ============================================================================================= +# (1) + (2) every internal service's rendered route is disowned; no public one is. +# ============================================================================================= +by_ident: dict[tuple[str, str], list[tuple[str, dict]]] = {} +for rel, d in routes: + by_ident.setdefault(ident(d), []).append((rel, d)) + +for (ns, rname), svc in sorted(expected_routes.items()): + found = by_ident.get((ns, rname), []) + check(len(found) >= 1, + f"({svc}) the catalog declares an ingressRoute rendering HTTPRoute {ns}/{rname}, but no " + f"such HTTPRoute exists in the applied tree — apps/ is out of sync with catalog/ " + f"(ci/render-golden.sh owns the byte-level half of this)") + for rel, d in found: + if exposure_of[svc] == "internal": + v = annotations_of(d).get(CONTROLLER_KEY) + check( + v is not None, + f"({svc}) R-DNS-03 VIOLATED: {rel} HTTPRoute {ns}/{rname} belongs to an " + f"`exposure: internal` service but carries NO `{CONTROLLER_KEY}` annotation, so " + f"external-dns publishes {ns}. into the PUBLIC Cloudflare zone answering " + f"with the RFC1918 Gateway VIP. An internal service must render no public record.", + ) + check( + v != CONTROLLER_CLAIMED, + f"({svc}) {rel} HTTPRoute {ns}/{rname} sets `{CONTROLLER_KEY}: " + f"{CONTROLLER_CLAIMED}` — that is external-dns's OWN value, which CLAIMS the route " + f"rather than releasing it. It reads like the guard and is its exact inverse.", + ) + else: + check( + CONTROLLER_KEY not in annotations_of(d), + f"({svc}) {rel} HTTPRoute {ns}/{rname} belongs to an `exposure: " + f"{exposure_of[svc]}` service yet carries `{CONTROLLER_KEY}`. Under " + f"`policy: sync` that DELETES its live public record — the five public services " + f"must keep publishing.", + ) + +# ============================================================================================= +# (3) the `service` source half — an internal service's Service may not publish either. +# ============================================================================================= +for rel, d in dns_services: + ns, name = ident(d) + svc = ns if ns in sds else None + if svc and exposure_of[svc] == "internal": + check( + is_disowned(d), + f"({svc}) {rel} Service {ns}/{name} carries an external-dns annotation " + f"({sorted(a for a in annotations_of(d) if a.startswith('external-dns'))}) for an " + f"`exposure: internal` service without a disowning `{CONTROLLER_KEY}`. Both " + f"`gateway-httproute` AND `service` are configured sources, so closing only the route " + f"half is half a guarantee.", + ) + +# ============================================================================================= +# (4) the GENERATOR derives it from `exposure` — not from a list of service names. +# ============================================================================================= +render_py = ROOT / "catalog" / "render.py" +src = read(render_py) +check(bool(src), "catalog/render.py unreadable") + +emitter = re.search( + r"def _externaldns_route_annotations\(.*?\n(?=\n?def |\n?# =====)", src, re.S +) +check(emitter is not None, + "catalog/render.py must define `_externaldns_route_annotations` — the single place the " + "ExternalDNS scoping is derived. If it was renamed, rename it here too: this check has to " + "read the DERIVATION, not just the output, or a hardcoded per-service list would pass.") +if emitter: + body = emitter.group(0) + # The derivation must read `exposure`. A body that does not mention it cannot be deriving + # anything from the operator's declaration. + check('"exposure"' in body or "'exposure'" in body, + "`_externaldns_route_annotations` must read the `exposure` field — the guarantee is " + "that the OPERATOR'S DECLARATION decides, so `exposure` has to appear in the decision") + # and it must not name any catalog service: a per-service list is how twelve ServiceMonitors + # acquired one identical wrong port name in a single edit. Only the executable statements are + # inspected — the docstring legitimately names the affected services as evidence. + code = re.sub(r'"""(?:.|\n)*?"""', "", body, count=1) + named = sorted(n for n in sds if re.search(rf"""['"]{re.escape(n)}['"]""", code)) + check(not named, + f"`_externaldns_route_annotations` names catalog service(s) {named} in its CODE. The " + f"annotation must be derived from `exposure` alone; a per-service list silently omits " + f"the next service anyone adds.") + check(CONTROLLER_CLAIMED not in code, + f"`_externaldns_route_annotations` must never be able to emit `{CONTROLLER_CLAIMED}` — " + f"external-dns's own value CLAIMS the object instead of releasing it") + +# ============================================================================================= +# (5) THE SHIPPED RENDERER IS PROVEN on synthetic ServiceDefinitions. Three fixtures, because +# "internal is annotated" alone would also pass if the renderer annotated everything, and +# because the contract's DEFAULT posture is internal — the case a new service hits by +# omitting the field entirely, and the case a per-service list would miss. +# ============================================================================================= +FIXTURES = [ + ("fixture-internal", {"exposure": "internal"}, True, + "an `internal` fixture must render the disown annotation"), + ("fixture-public", {"exposure": "public"}, False, + "a `public` fixture must render NO disown annotation"), + ("fixture-default", {}, True, + "a fixture OMITTING `exposure` defaults to internal (renderer-contract default posture) and " + "must also render the disown annotation"), +] +with tempfile.TemporaryDirectory() as td: + for fname, extra, want, msg in FIXTURES: + sd = { + "name": fname, + "kind": "container", + "source": {"image": "example/app:1.0.0"}, + "egress": "internet", + "dependencies": [], + "ingressRoutes": [{"host": f"{fname}.home.keithjackson.dev", "port": 8080}], + # `native` avoids the oauth2-proxy sidecar path and the schema's groups requirement, + # keeping the fixture about DNS scoping and nothing else. A public service may not use + # auth.mode none (schema invariant since 1.1.0), so both fixtures use the same mode. + "auth": {"mode": "native"}, + "update": {"channel": "stable", "cadence": "0 4 * * 0", + "autoRollbackOnFailedHealth": True}, + "observability": {"metrics": True, "slis": ["up"]}, + "backup": {"policy": "reacquirable"}, + } + sd.update(extra) + f = Path(td) / f"{fname}.yaml" + f.write_text(yaml.safe_dump(sd, sort_keys=False)) + proc = subprocess.run( + [sys.executable, str(render_py), str(f), "--no-image-automation"], + capture_output=True, text=True, cwd=str(ROOT), + ) + if proc.returncode != 0: + check(False, f"fixture {fname}: the shipped renderer failed: " + f"{proc.stderr.strip()[:400]}") + continue + rendered = [d for d in yaml.safe_load_all(proc.stdout) + if isinstance(d, dict) and d.get("kind") == "HTTPRoute"] + check(len(rendered) == 1, f"fixture {fname}: expected exactly one rendered HTTPRoute") + if rendered: + got = is_disowned(rendered[0]) + check(got is want, f"fixture {fname}: {msg} (rendered annotations: " + f"{annotations_of(rendered[0])})") + +# ============================================================================================= +# (6) REGISTER RATCHET — every publisher the catalog does not render is declared, and every +# declaration still names a real object. +# ============================================================================================= +catalog_rendered = set(expected_routes) +seen_register_keys: set[str] = set() +for rel, d in routes + dns_services: + ns, name = ident(d) + if (ns, name) in catalog_rendered: + continue + key = f"{rel}:{ns}/{name}" + seen_register_keys.add(key) + check( + key in KNOWN_NON_CATALOG_PUBLISHERS, + f"UNDECLARED publisher {key}: a {d.get('kind')} the catalog does not render, so it has no " + f"`exposure` field this check can adjudicate, and it can put a name into the public " + f"Cloudflare zone. Add it to KNOWN_NON_CATALOG_PUBLISHERS with what it publishes and why " + f"it is out of scope — or render it from a ServiceDefinition instead.", + ) +stale = sorted(set(KNOWN_NON_CATALOG_PUBLISHERS) - seen_register_keys) +check(not stale, + f"STALE register entries {stale}: KNOWN_NON_CATALOG_PUBLISHERS is shrink-only, and these " + f"name no object in the tree any more. Delete them — a register that keeps entries for " + f"deleted objects stops being evidence of anything.") + +# ============================================================================================= +# (7) LOAD-BEARING — the scope is DECLARED here (so ci/l0.sh Stage 3b's --include-scope union +# always selects this test) and NO stage is hand-written into ci/l0.sh (the +# whole-repo-invariants-wired meta-test fails the gate if one is). +# ============================================================================================= +TEST_DIR = ROOT / "conformance/tests/l0-static/internal-services-publish-no-public-dns" +test_yaml = read(TEST_DIR / "chainsaw-test.yaml") +check(bool(test_yaml), "this test's own chainsaw-test.yaml is unreadable") +check( + re.search(r'^\s*conformance\.home\.k3s/scope:\s*"?whole-repo"?\s*$', test_yaml, re.M) + is not None, + "chainsaw-test.yaml must DECLARE `conformance.home.k3s/scope: \"whole-repo\"` on a real " + "mapping line — that declaration is the only thing that unions this check into every gate " + "run (ci/l0.sh Stage 3b). A scope named in prose is documentation, not a declaration.", +) +l0_sh = read(ROOT / "ci/l0.sh") +check( + "internal-services-publish-no-public-dns" not in l0_sh, + "ci/l0.sh must NOT name this test: a whole-repo invariant is selected by its DECLARED scope, " + "and the whole-repo-invariants-wired meta-test fails the gate on a hand-written stage.", +) + +# ============================================================================================= +if FAILURES: + for f in FAILURES: + print(f"check FAIL: {f}", file=sys.stderr) + print(f"check FAILED: {len(FAILURES)} of {CHECKS} assertions failed", file=sys.stderr) + raise SystemExit(1) +print(f"check OK: {CHECKS} assertions passed " + f"({len(sds)} ServiceDefinitions; {len(internal_with_routes)} internal + " + f"{len(public_with_routes)} public services with routes; {len(routes)} HTTPRoutes and " + f"{len(dns_services)} external-dns-annotated Services inventoried; " + f"{len(KNOWN_NON_CATALOG_PUBLISHERS)} declared non-catalog publishers)") +PY diff --git a/platform/externaldns/helmrelease.yaml b/platform/externaldns/helmrelease.yaml index 3a56c3b..56e3581 100644 --- a/platform/externaldns/helmrelease.yaml +++ b/platform/externaldns/helmrelease.yaml @@ -26,10 +26,94 @@ spec: provider: name: cloudflare # Watch Gateway API routes (Cilium Gateway) + Services. docs/09: records rendered from intent. + # + # BOTH SOURCES ARE UNFILTERED HERE, AND THAT IS DELIBERATE — see the WHICH RESOURCES block + # below. Scoping is done PER RESOURCE by the renderer, not cluster-wide by this list. sources: - gateway-httproute - service # --------------------------------------------------------------------------------------------- + # WHICH RESOURCES MAY PUBLISH — `internal` services no longer can (R-DNS-03, 2026-09-14) + # + # THE DEFECT. `sources` above admits EVERY HTTPRoute on the shared Gateway, and each one's + # address is the Gateway VIP. docs/09 R-DNS-03 says `internal` services are unreachable from the + # public internet and that Pi-hole serves the internal split-horizon answers; the renderer + # contract said an `internal` service renders "no public DNS record". Neither sentence was + # enforced by anything, so six `exposure: internal` services (jackett, pihole, qbittorrent, + # radarr, semaphore, sonarr) published `.home.keithjackson.dev` into this PUBLIC + # authoritative zone answering 192.168.2.20. That exposed the internal subnet, the Gateway VIP + # and a complete service inventory to anyone who could guess a label, and made every one of + # those names a DNS-rebind primitive — which is exactly why the SOHO router at 192.168.2.1 + # strips RFC1918 answers and LAN clients need per-client dnsmasq overrides to reach their own + # services. Pre-state captured at the AUTHORITY and committed: + # .standup-dns/dns-20260914-pre-externaldns-scope.txt. + # + # THE MECHANISM — per-resource exclusion, emitted by catalog/render.py from `exposure`. + # An `internal` service's rendered HTTPRoute carries + # external-dns.alpha.kubernetes.io/controller: none + # ExternalDNS v0.21.0 (the DEPLOYED app version; read off the live prod Deployment, image + # registry.k8s.io/external-dns/external-dns:v0.21.0) skips any source object whose `controller` + # annotation is present and != its own `dns-controller`: + # source/annotations/processors.go IsControllerMismatch() + # source/gateway.go Endpoints(): if annotations.IsControllerMismatch(meta, src.rtKind) { continue } + # source/service.go Endpoints(): the identical check + # The gateway check runs BEFORE any hostname is resolved, so such a route cannot become an + # endpoint for any name it declares. PROVEN LIVE on prod before this landed, with a throwaway + # hostname and no existing record at risk: an annotated route published nothing at all, and + # ADDING the annotation to a route that already had a record produced + # action=DELETE record= type=A and action=DELETE record=a- type=TXT + # one ~60s reconcile later, and moved nothing else in the zone. + # + # WHY PER-RESOURCE AND NOT A CLUSTER-WIDE OPT-IN FILTER. The chart's `labelFilter` + # (--label-filter) is the stronger default posture — publish only what is labelled — and + # v0.21.0's own --help confirms BOTH configured sources support it ("currently supported by + # source types crd, gateway-httproute, ... service and ambassador-host"). It is NOT set here + # because its blast radius is not this defect's. Nine publishing objects in this repo are + # hand-authored platform/apps manifests with no `exposure` field for the renderer to read, and + # seven of them hold live records: auth (oauth2-proxy), crafty, crafty-mc, grafana, headlamp, + # the apex (homepage), unifi. Setting an opt-in filter DELETES all seven on the next reconcile, + # and whether each should be publicly resolvable at all is an OPERATOR decision per name, not a + # renderer one. They are enumerated in the shrink-only register in + # conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh, so the set + # cannot grow silently — adjudicate them there, then this filter becomes the right hardening. + # + # DELIBERATELY UNTOUCHED: `policy: sync`, `txtOwnerId`, `domainFilters`. Nothing above widens + # what this instance can see or own; per-resource exclusion only ever REMOVES an object from + # this instance's view, so it cannot make prod capable of deleting a record it does not own. + # --------------------------------------------------------------------------------------------- + # THE PUBLIC-TARGET GAP — STILL OPEN, AND NOT FIXABLE HERE ON v0.21.0 (operator decision) + # + # The five `exposure: public` services (plex, gitea, seerr, keycloak, nextcloud) publish records + # that point at 192.168.2.20 — the Gateway VIP, an unroutable RFC1918 address — so they are not + # actually reachable from the internet either. Their records are LEFT EXACTLY AS THEY ARE by + # this change, for two independent reasons: + # + # 1. THE VALUE IS NOT ESTABLISHED. The one public IP this repo knows is 136.51.115.252 + # (clusters/prod/kustomization.yaml, where the zone's `*.keithjackson.dev` wildcard also + # points). It belongs to the LEGACY non-k3s stack and currently serves a placeholder + # certificate. Whether prod's public services belong behind it, behind a new forward, or + # are not meant to be public at all is an operator decision. Breaking five services' DNS + # on a guess is worse than leaving a known-wrong-but-stable state. + # 2. THE MECHANISM CANNOT BE SCOPED TO THEM. `external-dns.alpha.kubernetes.io/target` is a + # real override, but in v0.21.0's gateway-httproute source it is read ONLY from the + # GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation( + # gw.Annotations)`) and from a ListenerSet listener — NEVER from the HTTPRoute. So there is + # no per-service target override while every public service shares the `shared` Gateway. + # Annotating the Gateway would move EVERY gateway-sourced record, which after this change + # still includes auth, crafty, grafana, headlamp and the apex — i.e. it would publish a + # routable address for admin UIs. That is strictly worse than the status quo. + # + # SO THE DECISION THE OPERATOR OWNS IS TWO-PART, and no knob is added here until the first half + # is answered, because a knob that cannot take effect invites someone to set it and expect one: + # (a) WHAT address should the five public names answer with — 136.51.115.252 as-is, a new + # port-forward/WAN address, or nothing (drop `exposure: public` to `internal` instead)? + # (b) HOW should they be separated from the non-public names that share the Gateway? Options, + # roughly in ascending cost: a SECOND Gateway (or a ListenerSet) carrying the target + # annotation, with only the public services' routes attached; or publishing the five names + # from the `crd` source (DNSEndpoint objects the renderer emits for `public` services), + # which restores per-service targeting without a second Gateway. + # Only (b) is a manifest change; (a) is a fact about the home WAN that has to be checked. + # --------------------------------------------------------------------------------------------- # CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below. # # On 2026-09-04 prod's external-dns DELETED EVERY ONE of dev's public records. Both clusters ran diff --git a/schemas/renderer-contract.md b/schemas/renderer-contract.md index 713c1f7..9880401 100644 --- a/schemas/renderer-contract.md +++ b/schemas/renderer-contract.md @@ -53,6 +53,41 @@ platform must render, and cites the requirement each mapping satisfies. Changing ExternalDNS record for `.home.keithjackson.dev`. - `internal` renders **no public DNS record** and is never port-forwarded (PiHole resolves it to the internal Gateway VIP). +- **HOW "no public DNS record" IS MADE TRUE, since it was prose only until 2026-09-14.** Every + HTTPRoute rendered for an `internal` service carries ExternalDNS's own controller annotation with + a value other than the one ExternalDNS claims: + + external-dns.alpha.kubernetes.io/controller: none + + ExternalDNS v0.21.0 (`source/annotations/processors.go`) skips any source object whose + `controller` annotation is present and `!= "dns-controller"`, and `source/gateway.go` applies that + check **inside `Endpoints()` before any hostname is resolved** — so the route never becomes an + endpoint for any name it declares. `source/service.go` applies the identical check, which covers + the second configured source. It is derived **from `exposure` and nothing else** + (`catalog/render.py` `_externaldns_route_annotations`), so the operator's one declaration decides, + a service that OMITS `exposure` inherits the `internal` default and is likewise unpublishable, and + there is no per-service list to forget to extend. `ci/render-golden.sh` proves `apps/` reproduces + from `catalog/`, so an `internal` service cannot exist in the applied tree without it, and + `conformance/tests/l0-static/internal-services-publish-no-public-dns` re-proves both directions + (present on internal, **absent** on public) on every gate run. + - The route itself, the wildcard TLS and LAN reachability are **unchanged** — only the public + record is withdrawn. The internal split-horizon answer is Pi-hole's job (R-DNS-03), which is why + withdrawing these records requires Pi-hole to be answering the LAN authoritatively first: the + zone has no `*.home.keithjackson.dev` wildcard, and `*.keithjackson.dev` does not match two + labels, so a withdrawn name is a public NXDOMAIN with no fallback. + - Why not a cluster-wide ExternalDNS opt-in filter (`labelFilter`, which v0.21.0 supports for + both configured sources): seven published names come from hand-authored `platform/`/`apps/` + manifests that carry no `exposure` field for a renderer to read, and an opt-in filter deletes + all seven the moment it is set. Per-resource exclusion has exactly the blast radius of the + defect. The filter stays available as hardening once those seven are adjudicated — see + `platform/externaldns/helmrelease.yaml` and the register in the L0 check. +- `public` renders the record but the renderer does **not** yet set a target: the record still + resolves to the shared Gateway VIP, i.e. an RFC1918 address, so a `public` service is not in fact + publicly reachable. Fixing that is **not** a renderer change on v0.21.0 — + `source/gateway.go` reads `external-dns.alpha.kubernetes.io/target` only from the **Gateway** (and + a ListenerSet), never from an HTTPRoute, so a per-service target override is not expressible while + the public services share the `shared` Gateway. The open decision is recorded in + `platform/externaldns/helmrelease.yaml`. - Exposure does **not** relax east-west default-deny (docs/09 §"Interaction with default-deny"). - **Requirements:** R-DNS-02/03 (docs/09), R-NET-04 (docs/03). @@ -131,8 +166,12 @@ platform must render, and cites the requirement each mapping satisfies. Changing (Let's Encrypt via **DNS-01**, scoped Cloudflare token), so no per-service cert step — trusted, non-self-signed, auto-renewed, including for `internal` services. Per-service `Certificate` only where isolation warrants. -- DNS: **ExternalDNS** watches the HTTPRoute and manages the `` record (public IP for `public`, - internal Gateway VIP via PiHole split-horizon for `internal`). +- DNS: **ExternalDNS** watches the HTTPRoute and manages the `` record — but **only for a + `public` service**. An `internal` service's route is rendered with ExternalDNS's controller + annotation set to `none`, which removes it from ExternalDNS's view entirely (see §`exposure`), so + its `` exists **only** in the internal split-horizon answer PiHole serves, never in the + public zone. A `public` record's TARGET is still the Gateway VIP, not a routable address — open + operator decision, see §`exposure`. - The `host` pattern is enforced to `