jakman85andClaude Opus 5
9137bbd0c3
*** DO NOT MERGE TO main WHILE THE OPERATOR IS AWAY. MERGING IS THE MIGRATION. ***
Operator simplified the architecture, inverting c8c1d29:
dev -> *.dev.home.keithjackson.dev (was the apex)
prod -> *.home.keithjackson.dev (the apex; no prod. subdomain, no CNAME layer)
WHY MERGING IS DANGEROUS, and this is the finding that matters most here:
clusters/dev/sync.yaml self-reconciles ./clusters/dev from `main` every 10 minutes. Merging
these files EXECUTES the cutover on the live dev cluster, unattended. And because the same
commit moves dev's GitRepository url to gitea.dev.home.keithjackson.dev, dev STRANDS — it
stops reconciling and cannot be fixed by pushing, because the broken thing IS the path Flux
reads pushes through. Banners added at the top of clusters/dev/kustomization.yaml and beside
the url itself. That url cannot be tokenized: the root Kustomization declares no
substituteFrom, so `gitea.${BASE_DOMAIN}` would ship the literal token.
Verified dev is currently safe: this work is NOT on origin/main, dev is reconciling
main@4809199, and its live BASE_DOMAIN is still home.keithjackson.dev. Deliberately NOT
pushed to the bootstrap Gitea either — prod reconciles from there, and prod on `prod.home.`
cannot collide with dev on the apex, so that is the safer interim for prod.
VALUES
BASE_DOMAIN TXT_OWNER_ID DOMAIN_FILTER
base dev.home.keithjackson.dev sentinel keithjackson.dev
dev dev.home.keithjackson.dev dev keithjackson.dev
prod home.keithjackson.dev prod keithjackson.dev
Ansible seam inverted too: role default k3s_base_domain = dev's subdomain, group_vars/k3s_prod
overrides to the apex.
dev's txtOwnerId moves `default` -> `dev`. The old value was FORCED (its live apex records
were already registered under `default`, so renaming would have orphaned them). On a new
suffix dev publishes a brand-new record set with no prior registry, so the id is chosen on
merits — and `default` has none: it is what you get by FORGETTING the flag, which is exactly
how both clusters came to share it and delete each other's records. Operationally it also
means dev's old apex records stay owned by a now-unclaimed id, so nothing deletes them
implicitly and vacating the apex becomes an explicit sequenceable step rather than a race
between two reconcile loops. PROD MUST NEVER BE GIVEN `default`: prod inherits the apex
NAMES, so `default` is the one value that would let it adopt and policy:sync-delete dev's
still-live apex records — the same incident with the roles swapped.
DOMAIN_FILTER stays the ZONE for both, with F-DNS1 written into all three files as a
do-not-retry: a subdomain filter drops the parent zone and publishes NOTHING.
VERIFICATION — the property changed, and so did the method. "Dev renders byte-identically" no
longer applies because dev is deliberately migrating. Instead: apply the INVERSE substitution
to the new render and demand byte-identity with the baseline. 145 changed line pairs, every
one classified: 143 hostname-move, 2 declared owner-id, ZERO unexplained, ZERO bare-apex
remnants. cilium and velero show 0 changes and provably contain no hostname, so they are not
blind spots. Prod: 0 occurrences of prod.home., 0 of dev.home., 143 apex names. Certs: dev
*.dev.home.keithjackson.dev, prod *.home.keithjackson.dev.
THREE positive controls, all fire — including one that proves the two checks are not
redundant: injecting a single un-moved hostname SURVIVES the inverse-substitution check and is
caught only by the independent bare-apex scan. That is why both exist.
Ansible probe (connection: local, zero SSH): dev nodes -> keycloak.dev.home..., prod nodes ->
keycloak.home.... Used roles: + --tags probe rather than include_vars, because include_vars
outranks group_vars and would have FALSELY shown prod inheriting dev's value.
Gates all PASS: l0, l1, render-golden (10 goldens), lint, substitution (254 assertions),
render.py --selftest, spec-immutability. Catalog ServiceDefinitions edited, never rendered apps/.
RUNBOOK rewritten in §10 + §10a with the ordering that avoids stranding: D0 snapshot escape
routes -> D1 break the automatic trigger -> D2 publish and PROVE gitea.dev.home + the new
wildcard cert WHILE dev still serves the apex -> D3 merge -> D4 release reconciliation
deliberately -> D5 Keycloak issuer + ~12 clients (headlamp's is SOPS, by hand) -> D6
one-server-at-a-time k3s restart + SSO re-issue (service restart only, NO reboots) -> D7
retire old apex records -> D8 verify the apex is empty AGAINST CLOUDFLARE, not pfSense —
the gate on everything after. Then P1 unsuspend prod.
ACME: prod's wildcard is EXACTLY the name set dev holds today, so prod's orders share dev's
5-per-week duplicate-certificate bucket. Order after D8, delete dev's old Certificate rather
than let it renew, rehearse on staging.
The 6 L2 chainsaw tests hardcoding the apex are enumerated but deliberately NOT edited — while
dev is live on the apex they are correct; they need per-environment parameterisation.
Serves: R-ENV-02, R-DNS-01/02/03, R-TLS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
83 lines
4.8 KiB
YAML
83 lines
4.8 KiB
YAML
---
|
|
# ExternalDNS — publishes public DNS records for `public` services automatically (docs/09, R-DNS-02).
|
|
# Watches Gateway/HTTPRoute (+Service); manages records in Cloudflare. Phase 0 STUB.
|
|
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
kind: HelmRelease
|
|
metadata:
|
|
name: external-dns
|
|
namespace: external-dns
|
|
spec:
|
|
interval: 30m
|
|
chart:
|
|
spec:
|
|
chart: external-dns
|
|
version: ">=1.21.0 <1.22.0" # k8s-1.36 catch-up (app v0.21; policy already set). Renovate maintains
|
|
sourceRef:
|
|
kind: HelmRepository
|
|
name: external-dns
|
|
namespace: external-dns
|
|
install:
|
|
remediation:
|
|
retries: 3
|
|
upgrade:
|
|
remediation:
|
|
retries: 3
|
|
values:
|
|
provider:
|
|
name: cloudflare
|
|
# Watch Gateway API routes (Cilium Gateway) + Services. docs/09: records rendered from intent.
|
|
sources:
|
|
- gateway-httproute
|
|
- service
|
|
# ---------------------------------------------------------------------------------------------
|
|
# CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below.
|
|
#
|
|
# On 2026-09-04 prod's external-dns DELETED EVERY ONE of dev's public records. Both clusters ran
|
|
# `policy: sync` against the SAME Cloudflare zone, served the SAME hostnames, and — because
|
|
# NEITHER set txtOwnerId — both inherited the chart default owner id `default`. Prod therefore
|
|
# read dev's TXT registry entries as its own, found no local resource claiming those names, and
|
|
# garbage-collected them. `policy: sync` is correct and stays; what was missing was identity.
|
|
# Full write-up: infra/PROD-PROVISIONING.md §10.
|
|
#
|
|
# THE TWO GUARDS THAT ACTUALLY PREVENT A RECURRENCE — note that neither is domainFilters:
|
|
# 1. txtOwnerId — per-environment identity; a cluster only ever touches records it owns.
|
|
# 2. DISJOINT HOSTNAME SETS — the two environments serve different suffixes, so they never
|
|
# describe the same record in the first place. dev = *.dev.home.keithjackson.dev,
|
|
# prod = *.home.keithjackson.dev (prod IS the apex; operator decision 2026-09-04).
|
|
# ---------------------------------------------------------------------------------------------
|
|
# Which names this instance may manage at all. BOTH ENVIRONMENTS RENDER THE CLOUDFLARE **ZONE**
|
|
# `keithjackson.dev` HERE, and this is NOT an oversight to be tightened.
|
|
#
|
|
# FINDING F-DNS1 (DECISIONS-LOG 2026-08-17): the Cloudflare provider uses this value for ZONE
|
|
# SELECTION as well as record matching, so a filter that is a STRICT SUBDOMAIN of the hosted zone
|
|
# matches NO zone — external-dns generates the record and then skips it with "no hosted zone
|
|
# matching record DNS Name", publishing NOTHING. This has been "improved" and reverted more than
|
|
# once (an agent narrowed prod to prod.home.keithjackson.dev on 2026-09-04; overridden in
|
|
# review). It is not defence in depth; it is a total outage of DNS publication. Isolation comes
|
|
# from the two guards above, which a domain filter cannot provide anyway — both environments live
|
|
# in the ONE zone regardless of what this says. See clusters/base/cluster-vars.yaml.
|
|
domainFilters:
|
|
- "${EXTERNALDNS_DOMAIN_FILTER}"
|
|
# The TXT-registry owner id. MUST be distinct per environment — this is the knob whose absence
|
|
# caused the incident. dev renders `dev`, prod renders `prod`, and the BASE default is a
|
|
# non-colliding sentinel so a new environment that forgets the knob owns nothing and can delete
|
|
# nothing. NOTHING renders `default` any more (the external-dns chart default, and the value both
|
|
# clusters silently shared): dev held it only because its live apex records were registered under
|
|
# it, and dev's move to *.dev.home.keithjackson.dev retired that constraint along with those
|
|
# records. Prod must NOT be given `default` even though it inherits the apex NAMES — that would
|
|
# hand prod ownership of dev's old apex records and let `policy: sync` below delete them.
|
|
txtOwnerId: "${EXTERNALDNS_TXT_OWNER_ID}"
|
|
policy: sync # create + update + delete records as services come and go — but ONLY
|
|
# records this owner id owns AND this domainFilter admits (see above).
|
|
# Cloudflare API token: SOPS-encrypted secret, referenced BY NAME only (OQ-4, ADR-0007).
|
|
env:
|
|
- name: CF_API_TOKEN
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: cloudflare-api-token
|
|
key: api-token
|
|
# R-DNS-02 PROVEN via the `service` source: the conformance/l2-behavioral/dns-exposure test
|
|
# creates a LoadBalancer Service with an external-dns hostname and asserts the record auto-appears
|
|
# in public DNS (then teardown + policy=sync removes it). gateway-httproute source wiring is the
|
|
# same provider/zone path; a route-based test can follow when app routes exist.
|