jakman85andClaude Opus 5
c8c1d2935d
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).
THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.
Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.
PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.
BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.
TWO THAT CANNOT BE TOKENIZED, flagged not forced:
- headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
anyway — client secrets are realm-scoped.
- kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
prod.home.
EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.
I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.
DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.
L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.
PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.
Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
99 lines
6.0 KiB
YAML
99 lines
6.0 KiB
YAML
---
|
|
# k3s_server role defaults (ADR-0001).
|
|
|
|
# Best-effort version pin (kept near the ci/versions kubectl 1.36.x line). Bump
|
|
# deliberately — a version change is a code change. Empty string => latest stable.
|
|
k3s_version: "v1.36.4+k3s1"
|
|
|
|
k3s_install_url: "https://get.k3s.io"
|
|
k3s_install_script: "/usr/local/bin/k3s-install.sh"
|
|
|
|
# Components disabled at install so Cilium + Gateway API own networking (ADR-0001/0002).
|
|
# flannel + kube-proxy + network-policy are disabled via dedicated flags below;
|
|
# traefik + servicelb via --disable.
|
|
# local-storage: k3s ships local-path as a DEFAULT StorageClass, which collides with Longhorn's
|
|
# default (two default SCs = ambiguous PVC binding). Longhorn (ADR-0003) is the cluster's storage,
|
|
# so disable k3s local-path entirely → `longhorn` is the sole default. (Effective on fresh installs;
|
|
# the k3s install task is idempotent, so this does not restart an existing cluster — see DECISIONS-LOG.)
|
|
k3s_disable: "traefik,servicelb,local-storage"
|
|
|
|
# --- k3s --data-dir (R-ENV-02) ------------------------------------------------------
|
|
# EVERYTHING k3s persists lives under this one directory: the embedded etcd database and
|
|
# WAL (server/db), the node token, the TLS CA and every issued cert, AND the containerd
|
|
# image store (agent/containerd) — which is by far the largest tenant.
|
|
#
|
|
# THE DEFAULT IS k3s'S OWN DEFAULT, restated explicitly rather than left implicit, so the
|
|
# LIVE dev cluster's install command is byte-for-byte what it already was: dev keeps
|
|
# /var/lib/rancher/k3s, where it also has a DEDICATED etcd device mounted at
|
|
# server/db (docs/plans/ssd-etcd-isolation.md). Changing this value on dev would move a
|
|
# running cluster's entire state directory out from under it and orphan that etcd mount.
|
|
#
|
|
# PROD OVERRIDES IT, in inventory/group_vars/k3s_prod/vars.yml — the same seam
|
|
# kube_vip_address / k3s_api_host / node_baseline_backports_kernel use, so a FOURTH prod
|
|
# node inherits it from group membership rather than from anyone remembering. It is set
|
|
# there as the CLUSTER-WIDE `k3s_data_dir` because TWO roles must agree on it: this role
|
|
# turns it into --data-dir, and roles/node_baseline creates the directory on the claimed
|
|
# data partition and derives etcd's location from it (node_baseline_k3s_data_dir defers to
|
|
# this same variable). One line moves both.
|
|
#
|
|
# WHY PROD MOVES IT (infra/PROD-PROVISIONING.md §2, revised 2026-09-04). Prod's /var is a
|
|
# separate partition of 11.8 GiB. Measured on the fully-deployed dev cluster, containerd
|
|
# alone is 21-25 GB per node and /var totals 58-70 GB — so prod would fill /var during its
|
|
# FIRST image pulls, and a full /var means etcd cannot write its WAL. That is cluster death,
|
|
# not degradation. Prod points --data-dir at /data/k3s on the claimed ~424 GB partition.
|
|
#
|
|
# NOT ALSO WRITTEN TO /etc/rancher/k3s/config.yaml, deliberately. --data-dir is consumed at
|
|
# INSTALL time (it is baked into the systemd unit's ExecStart); adding a `data-dir:` line to
|
|
# config.yaml would make that file `changed` on the LIVE dev cluster and fire the rolling
|
|
# Restart handler for a value that is not changing. The install flags are the correct and
|
|
# sufficient place, and prod has no k3s installed yet.
|
|
k3s_data_dir: "/var/lib/rancher/k3s"
|
|
|
|
# Set per-play: 'primary' (runs --cluster-init) or 'secondary' (joins the primary).
|
|
k3s_server_mode: "secondary"
|
|
|
|
# Subject Alternative Names baked into the kube-apiserver serving cert. Must include
|
|
# the kube-vip control-plane VIP (kube_vip_address) so TLS via the VIP validates,
|
|
# every server's real node IP (so the cert is valid whichever node answers behind
|
|
# the VIP), and 127.0.0.1 (local kubeconfig / health probes). k3s regenerates the
|
|
# serving cert on restart when this set changes. Deduplicated (the primary IP is
|
|
# also k3s_api_host).
|
|
#
|
|
# k3s_servers_group — NOT a literal group name — is what makes this role environment-
|
|
# agnostic (R-ENV-02): group_vars/all.yml sets it to `k3s_servers` (dev) and
|
|
# group_vars/k3s_prod/vars.yml to `k3s_prod`, so each environment SANs its own servers.
|
|
k3s_tls_sans: >-
|
|
{{ ([k3s_api_host, kube_vip_address, '127.0.0.1']
|
|
+ (groups[k3s_servers_group] | map('extract', hostvars, 'k3s_node_ip') | list))
|
|
| unique }}
|
|
|
|
# --- Keycloak OIDC issuer for kubectl SSO (R-IDP-03, R-ENV-02) -----------------------
|
|
# THE ANSIBLE-SIDE HALF OF ${BASE_DOMAIN}, and it cannot be the Flux-side half.
|
|
#
|
|
# The kube-apiserver's `oidc-issuer-url` is a k3s SERVER FLAG written into
|
|
# /etc/rancher/k3s/config.yaml by this role — it is NOT a Kubernetes manifest, so Flux never
|
|
# sees it and `postBuild.substituteFrom: cluster-vars` cannot reach it. The platform hostname
|
|
# suffix therefore exists as ONE FACT IN TWO PLACES, and they must agree:
|
|
#
|
|
# Flux side clusters/base/cluster-vars.yaml BASE_DOMAIN (per-overlay)
|
|
# Ansible side THIS variable k3s_base_domain (per-inventory-group)
|
|
#
|
|
# [parity] k3s_base_domain MUST equal the BASE_DOMAIN the SAME cluster's Flux overlay sets.
|
|
# If they disagree, the apiserver validates ID tokens against a DIFFERENT Keycloak than the
|
|
# one the cluster actually runs: every `kubectl` SSO login fails with an issuer mismatch,
|
|
# while the cert-based admin kubeconfig keeps working — so it looks like "SSO is broken",
|
|
# not like a config skew. The break-glass cert kubeconfig is why this is recoverable.
|
|
#
|
|
# THE DEFAULT IS THE DEV VALUE, restated explicitly rather than left implicit — the same
|
|
# convention as k3s_data_dir above and cluster-vars' GATEWAY_VIP / LONGHORN_DATA_PATH: the
|
|
# base default is the known-good LIVE value, so dev's rendered config.yaml is byte-for-byte
|
|
# what it already is and the Restart handler does NOT fire on a re-run.
|
|
#
|
|
# PROD OVERRIDES IT in inventory/group_vars/k3s_prod/vars.yml — the same inventory seam
|
|
# k3s_data_dir / k3s_api_host / kube_vip_address use, so a FOURTH prod server inherits it
|
|
# from group membership rather than from anyone remembering.
|
|
k3s_base_domain: "home.keithjackson.dev"
|
|
|
|
# Derived, so the realm path is stated once. Override k3s_base_domain, not this.
|
|
k3s_oidc_issuer_url: "https://keycloak.{{ k3s_base_domain }}/realms/homelab"
|