Gateway-published records now CNAME to wan.${BASE_DOMAIN}, which a
favonia/cloudflare-ddns Deployment keeps at the home WAN's current public
IP. Operator decision: every Gateway-published name goes public; internal
services (controller: none) remain unpublished.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
docs/09 R-DNS-03 and schemas/renderer-contract.md both said an `internal`
service renders no public DNS record. Nothing enforced either sentence.
ExternalDNS's `gateway-httproute` source admitted EVERY HTTPRoute on the shared
Gateway, so six `exposure: internal` services — jackett, pihole, qbittorrent,
radarr, semaphore, sonarr — published `<name>.home.keithjackson.dev` into the
public authoritative Cloudflare zone answering with the Gateway VIP
192.168.2.20. That published the internal subnet, the Gateway VIP and a full
service inventory to anyone who could guess a label, and made each of those
names a DNS-rebind primitive — which is why the SOHO router strips RFC1918
answers and LAN clients need per-client dnsmasq overrides to reach their own
services.
THE MECHANISM, established against the DEPLOYED version rather than from
memory. The live prod Deployment runs
registry.k8s.io/external-dns/external-dns:v0.21.0. In v0.21.0,
source/annotations/processors.go IsControllerMismatch() skips a source object
whose `external-dns.alpha.kubernetes.io/controller` annotation is present and
!= its own `dns-controller`, and source/gateway.go applies that check inside
Endpoints() BEFORE any hostname is resolved; source/service.go applies the
identical check, covering the second configured source. catalog/render.py now
emits that annotation on every HTTPRoute it renders for an `internal` service,
derived from `exposure` alone — no per-service list — so a service that omits
`exposure` inherits the internal default and is unpublishable too. Verified
LIVE on prod on a throwaway hostname with no existing record at risk: the
annotated route published nothing, and adding the annotation to a route that
already had a record produced action=DELETE for both its A record and its `a-`
TXT registry entry one ~60s reconcile later, moving nothing else in the zone.
WHY PER-RESOURCE AND NOT A CLUSTER-WIDE OPT-IN FILTER. `labelFilter` is the
stronger default posture and v0.21.0 supports it for both configured sources,
but nine publishing objects here are hand-authored platform/apps manifests with
no `exposure` field, seven of which hold live records (auth, crafty, crafty-mc,
grafana, headlamp, the apex, unifi). An opt-in filter deletes all seven at
once, and whether each should be publicly resolvable is an operator decision
per name. Per-resource exclusion has exactly the blast radius of the defect;
the seven are enumerated in a shrink-only register so the unadjudicated set
cannot grow silently, and the filter stays available as hardening afterwards.
`policy: sync`, `txtOwnerId` and `domainFilters` are UNTOUCHED. Per-resource
exclusion only ever removes an object from this instance's view, so nothing
here makes prod capable of deleting a record it does not own.
WHAT THIS DELETES, and it is not reversible by a revert alone: 12 records — the
A record and the `a-`-prefixed TXT registry entry for each of jackett, pihole,
qbittorrent, radarr, semaphore and sonarr. The zone has no
`*.home.keithjackson.dev` wildcard and `*.keithjackson.dev` does not match two
labels, so each becomes a public NXDOMAIN with no fallback. The complete
pre-state is captured at the authority and committed as a recovery baseline
(.standup-dns/dns-20260914-pre-externaldns-scope.txt), with the restore
procedure for both halves of a record.
MERGE ORDER IS NOT OPTIONAL: feat/pihole-lan-dns-authority must land and be
verified first. Removing these records while the LAN still depends on them,
with the router stripping RFC1918 and no internal authority yet, breaks name
resolution outright.
The five `exposure: public` services are LEFT UNCHANGED and still answer with
the Gateway VIP, so they remain publicly unreachable. That is two open operator
decisions, both written up in platform/externaldns/helmrelease.yaml: what
address they should answer with (136.51.115.252 is the legacy non-k3s stack
serving a placeholder cert, not established as correct for this cluster), and
how to separate them from the non-public names that share the Gateway —
because in v0.21.0 the target annotation is read only from the GATEWAY and a
ListenerSet, never from an HTTPRoute, so a per-service override is not
expressible while they all attach to `shared`. No knob is added for it yet: one
that cannot take effect invites someone to set it and expect one.
Guard: conformance/tests/l0-static/internal-services-publish-no-public-dns, a
whole-repo-scoped L0 static check (no ci/l0.sh stage — the
whole-repo-invariants-wired meta-test forbids one). It asserts both directions
(present on internal, absent on public), the `service`-source half, that the
renderer DERIVES the annotation from `exposure` and cannot emit external-dns's
own claiming value, proves the shipped renderer end-to-end on
internal/public/omitted-field fixtures, and ratchets the non-catalog register.
Proven to fail on the pre-fix tree (9 of 53 assertions, naming all six
services) and pass after (56/56).
Schema 1.6.0 -> 1.7.0: no field changes, a mapping change to `exposure`.
Co-Authored-By: Claude Opus 5 <[email protected]>
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).
THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.
Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.
PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.
BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.
TWO THAT CANNOT BE TOKENIZED, flagged not forced:
- headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
anyway — client secrets are realm-scoped.
- kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
prod.home.
EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.
I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.
DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.
L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.
PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.
Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Phase-2 exit tests (were Phase-0 scaffolds), authored against the live platform:
- trusted-tls (R-TLS-01/03): in-cluster probe dials the gateway Service (env-agnostic
via --connect-to) with wildcard SNI and verifies the served cert CHAINS TO THE LE
STAGING ROOT (operator decision A; dev uses LE-staging per ADR-0006). Fixture embeds
the '(STAGING) Pretend Pear X1' root. Probes public + internal-only hostnames (R-TLS-03).
PASSES.
- dns-exposure (R-DNS-02): a LoadBalancer Service with an external-dns hostname; polls
public DNS until ExternalDNS auto-publishes the record; teardown + policy=sync removes it.
ExternalDNS fix (root cause of the DNS test): domain-filter was the platform SUBDOMAIN
home.keithjackson.dev, which filters OUT the actual Cloudflare zone keithjackson.dev
('no hosted zone matching record DNS Name' — record generated but skipped). Add a
DNS_ZONE cluster-var (keithjackson.dev) and set domainFilters to it. policy=sync +
txt-registry still only manage ExternalDNS-owned records (the pre-existing wildcard is
untouched). Resolves the file's WS-NET R-DNS-02 TODO.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
- Rename the committed encrypted secrets to canonical cloudflare-api-token.sops.yaml
(cert-manager + external-dns namespaces) and uncomment the resource lines in both
kustomizations, so cert-manager DNS-01 + ExternalDNS get the token at reconcile.
- Exempt **/*.sops.yaml from ci/yamllint.yaml: SOPS re-emits encrypted files with its
own indentation + long ENC[...] lines (never hand-edited), which was failing the lint
gate on main after the secrets were committed. Same treatment as **/vault.yml.
Both Secrets confirmed present + encrypted in the platform build; gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
renovate-bot
jakman85andClaude Opus 5.5