Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 6 Actions Packages Projects Releases Wiki Activity
522 Commits 140 Branches 0 Tags
fe45d0f5c510dd443e8191eee5b0da42f679157f
Commit Graph
8 Commits
Author SHA1 Message Date
jakman85andClaude Opus 5 9137bbd0c3 feat(dns): INVERT to operator's simpler layout — dev to subdomain, prod to apex 
*** DO NOT MERGE TO main WHILE THE OPERATOR IS AWAY. MERGING IS THE MIGRATION. ***

Operator simplified the architecture, inverting c8c1d29:
  dev  -> *.dev.home.keithjackson.dev        (was the apex)
  prod -> *.home.keithjackson.dev            (the apex; no prod. subdomain, no CNAME layer)

WHY MERGING IS DANGEROUS, and this is the finding that matters most here:
clusters/dev/sync.yaml self-reconciles ./clusters/dev from `main` every 10 minutes. Merging
these files EXECUTES the cutover on the live dev cluster, unattended. And because the same
commit moves dev's GitRepository url to gitea.dev.home.keithjackson.dev, dev STRANDS — it
stops reconciling and cannot be fixed by pushing, because the broken thing IS the path Flux
reads pushes through. Banners added at the top of clusters/dev/kustomization.yaml and beside
the url itself. That url cannot be tokenized: the root Kustomization declares no
substituteFrom, so `gitea.${BASE_DOMAIN}` would ship the literal token.

Verified dev is currently safe: this work is NOT on origin/main, dev is reconciling
main@4809199, and its live BASE_DOMAIN is still home.keithjackson.dev. Deliberately NOT
pushed to the bootstrap Gitea either — prod reconciles from there, and prod on `prod.home.`
cannot collide with dev on the apex, so that is the safer interim for prod.

VALUES
                BASE_DOMAIN                  TXT_OWNER_ID   DOMAIN_FILTER
  base          dev.home.keithjackson.dev    sentinel       keithjackson.dev
  dev           dev.home.keithjackson.dev    dev            keithjackson.dev
  prod          home.keithjackson.dev        prod           keithjackson.dev
Ansible seam inverted too: role default k3s_base_domain = dev's subdomain, group_vars/k3s_prod
overrides to the apex.

dev's txtOwnerId moves `default` -> `dev`. The old value was FORCED (its live apex records
were already registered under `default`, so renaming would have orphaned them). On a new
suffix dev publishes a brand-new record set with no prior registry, so the id is chosen on
merits — and `default` has none: it is what you get by FORGETTING the flag, which is exactly
how both clusters came to share it and delete each other's records. Operationally it also
means dev's old apex records stay owned by a now-unclaimed id, so nothing deletes them
implicitly and vacating the apex becomes an explicit sequenceable step rather than a race
between two reconcile loops. PROD MUST NEVER BE GIVEN `default`: prod inherits the apex
NAMES, so `default` is the one value that would let it adopt and policy:sync-delete dev's
still-live apex records — the same incident with the roles swapped.

DOMAIN_FILTER stays the ZONE for both, with F-DNS1 written into all three files as a
do-not-retry: a subdomain filter drops the parent zone and publishes NOTHING.

VERIFICATION — the property changed, and so did the method. "Dev renders byte-identically" no
longer applies because dev is deliberately migrating. Instead: apply the INVERSE substitution
to the new render and demand byte-identity with the baseline. 145 changed line pairs, every
one classified: 143 hostname-move, 2 declared owner-id, ZERO unexplained, ZERO bare-apex
remnants. cilium and velero show 0 changes and provably contain no hostname, so they are not
blind spots. Prod: 0 occurrences of prod.home., 0 of dev.home., 143 apex names. Certs: dev
*.dev.home.keithjackson.dev, prod *.home.keithjackson.dev.

THREE positive controls, all fire — including one that proves the two checks are not
redundant: injecting a single un-moved hostname SURVIVES the inverse-substitution check and is
caught only by the independent bare-apex scan. That is why both exist.

Ansible probe (connection: local, zero SSH): dev nodes -> keycloak.dev.home..., prod nodes ->
keycloak.home.... Used roles: + --tags probe rather than include_vars, because include_vars
outranks group_vars and would have FALSELY shown prod inheriting dev's value.

Gates all PASS: l0, l1, render-golden (10 goldens), lint, substitution (254 assertions),
render.py --selftest, spec-immutability. Catalog ServiceDefinitions edited, never rendered apps/.

RUNBOOK rewritten in §10 + §10a with the ordering that avoids stranding: D0 snapshot escape
routes -> D1 break the automatic trigger -> D2 publish and PROVE gitea.dev.home + the new
wildcard cert WHILE dev still serves the apex -> D3 merge -> D4 release reconciliation
deliberately -> D5 Keycloak issuer + ~12 clients (headlamp's is SOPS, by hand) -> D6
one-server-at-a-time k3s restart + SSO re-issue (service restart only, NO reboots) -> D7
retire old apex records -> D8 verify the apex is empty AGAINST CLOUDFLARE, not pfSense —
the gate on everything after. Then P1 unsuspend prod.

ACME: prod's wildcard is EXACTLY the name set dev holds today, so prod's orders share dev's
5-per-week duplicate-certificate bucket. Order after D8, delete dev's old Certificate rather
than let it renew, rehearse on staging.

The 6 L2 chainsaw tests hardcoding the apex are enumerated but deliberately NOT edited — while
dev is live on the apex they are correct; they need per-environment parameterisation.

Serves: R-ENV-02, R-DNS-01/02/03, R-TLS-01.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-04 23:08:50 -05:00
jakman85andClaude Opus 5 c8c1d2935d feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain
gate / gate (push) Has been cancelled
Details
gate / conformance (push) Has been cancelled
Details 
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).

THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.

Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.

PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.

BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.

TWO THAT CANNOT BE TOKENIZED, flagged not forced:
  - headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
    over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
    so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
    anyway — client secrets are realm-scoped.
  - kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
    reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
    overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
    prod.home.

EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.

I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.

DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.

L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.

PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.

Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-04 22:43:54 -05:00
jakman85andClaude Opus 4.8 19a368b237 Component catch-up batch 1: external-dns 1.21, oauth2-proxy 10.x, Alloy 1.x, CNPG in-place flag 
Co-Authored-By: Claude Opus 4.8 <[email protected]>
 2026-08-30 11:21:55 -05:00
jakman85andClaude Opus 4.8 20de78f5eb feat(conformance): author R-TLS + R-DNS L2 tests; fix ExternalDNS zone filter 
Phase-2 exit tests (were Phase-0 scaffolds), authored against the live platform:
- trusted-tls (R-TLS-01/03): in-cluster probe dials the gateway Service (env-agnostic
  via --connect-to) with wildcard SNI and verifies the served cert CHAINS TO THE LE
  STAGING ROOT (operator decision A; dev uses LE-staging per ADR-0006). Fixture embeds
  the '(STAGING) Pretend Pear X1' root. Probes public + internal-only hostnames (R-TLS-03).
  PASSES.
- dns-exposure (R-DNS-02): a LoadBalancer Service with an external-dns hostname; polls
  public DNS until ExternalDNS auto-publishes the record; teardown + policy=sync removes it.

ExternalDNS fix (root cause of the DNS test): domain-filter was the platform SUBDOMAIN
home.keithjackson.dev, which filters OUT the actual Cloudflare zone keithjackson.dev
('no hosted zone matching record DNS Name' — record generated but skipped). Add a
DNS_ZONE cluster-var (keithjackson.dev) and set domainFilters to it. policy=sync +
txt-registry still only manage ExternalDNS-owned records (the pre-existing wildcard is
untouched). Resolves the file's WS-NET R-DNS-02 TODO.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
 2026-08-17 08:24:51 -05:00
jakman85andClaude Opus 4.8 275ac466b2 wire Cloudflare DNS token secrets + exempt *.sops.yaml from yamllint 
- Rename the committed encrypted secrets to canonical cloudflare-api-token.sops.yaml
  (cert-manager + external-dns namespaces) and uncomment the resource lines in both
  kustomizations, so cert-manager DNS-01 + ExternalDNS get the token at reconcile.
- Exempt **/*.sops.yaml from ci/yamllint.yaml: SOPS re-emits encrypted files with its
  own indentation + long ENC[...] lines (never hand-edited), which was failing the lint
  gate on main after the secrets were committed. Same treatment as **/vault.yml.

Both Secrets confirmed present + encrypted in the platform build; gate PASS.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
 2026-08-17 00:35:45 -05:00
jakman85 06fa317063 Added fixes before token usage cutoff. Added required credentials for dns api for cert-manager and external-dns 2026-08-16 23:12:25 -05:00
jakman85andClaude Opus 4.8 a108d5ab98 correction: credential-forward + unified graph; activate SOPS (author-first) 
Operator directive: don't diverge architecture to avoid asking for credentials;
provision them upfront via SOPS and keep the existing (unified) graph.

Roadmap reconciliation: Phase 2 = Network/Storage/TLS/DNS/Ingress, so
cert-manager/externaldns/gateway (the credential-gated pieces) ARE Phase 2;
monitoring is Phase 4. Earlier A2/rollout scope was inverted by credential-avoidance.

Reverts (credential-driven): remove clusters/dev/rollout/ + the $patch:delete's;
restore unified base graph (dev = ../base + cluster-vars + SOPS decryption,
R-ENV-02 intact); restore monitoring files (Phase 4). Kept non-credential
improvements: tofu for_each/rejoin, Kyverno parametrization + webhook excludes.

platform/kustomization.yaml now phased (single shared list): Phase-2 set =
kyverno, cilium, longhorn, cert-manager, externaldns, gateway; later components
commented with phase markers.

SOPS activated: .sops.yaml at repo root; bootstrap/sops/secret-templates/*.yaml.example
for all secrets (cloudflare x2, grafana, alertmanager-smtp, keycloak, oauth2-proxy, pia);
bootstrap/sops/CREDENTIALS.md checklist; pre-wired commented cloudflare-api-token lines
in cert-manager + externaldns; filled non-secret ACME email. Agents never touch real
secret values (operator encrypts).

Validated: kubectl kustomize dev(unified)/prod/platform(Phase-2 only), templates inert,
make -C ci gate PASS.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
 2026-08-16 22:09:20 -05:00
jakman85andClaude Opus 4.8 af4f6c07ae T-0.1 Flux GitOps layout + dev/prod overlays + platform component skeletons (R-GITOPS-01/03, R-ENV-02, R-PLAT-01) 
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
 2026-08-16 00:08:31 -05:00