Commit Graph
10 Commits
Author SHA1 Message Date
renovate-bot 9c0292001e chore(deps): Update pihole/pihole Docker tag to v2026.10.1
gate / gate (pull_request) Successful in 2m57s
gate / conformance (pull_request) Has been skipped
2026-11-04 08:16:12 +00:00
jakman85andClaude Opus 5.5 183bca6897 feat(images): pin pihole, jackett and plex to the digests already running; Renovate follows them
gate / gate (pull_request) Successful in 2m57s
gate / conformance (pull_request) Has been skipped
pihole/pihole (no tag)               -> 2026.10.0               sha256:d3413471...
lscr.io/linuxserver/jackett:latest   -> v0.24.2865-ls7          sha256:4fcf3cc2...
plexinc/pms-docker:latest            -> 1.43.4.10903-e5521bd8c  sha256:e0ab2739...

Each tag was resolved against its registry and matched the pods' imageID, so the image is
unchanged (the pod template string does change, so Flux rolls the three workloads).

renovate.json5: regex versioning for jackett (-lsN) and plex (build-hex); jackett is looked
up on ghcr.io via overridePackageName, because lscr.io is capped at dockerMaxPages=20 pages
and jackett's 32k tags put the running one at #31,980 (dry run proposed nothing without it);
jackett weekly (13 builds in 12 days); pihole deliberately unheld, year rollover = major.
Stale "publishes only latest" prose fixed in renovate.json5 and three runbooks.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-31 10:38:07 -05:00
jakman85andClaude Opus 5 fda5c31a94 fix(pihole): make the split-horizon answer cacheable, or one blip latches the WAN IP for 5 minutes
gate / gate (pull_request) Successful in 1m56s
gate / conformance (pull_request) Has been skipped
Operator reported DNS flapping between the cluster ingress and the router's
public IP. PiHole is not at fault -- both replicas answer 192.168.2.20
consistently (6/6 via the VIP, 8/8 from a client). The bug is an asymmetry in
CACHEABILITY that we created. Measured:

  pihole  192.168.2.23 -> 192.168.2.20               TTL=0
  public  1.1.1.1      -> wan.home.keithjackson.dev  TTL=300
  router  192.168.2.1  -> wan....  TTL=153 -> 148 -> 143   (decrementing = caching)

dnsmasq answers `address=` entries with TTL 0 unless told otherwise. The router
forwards LAN queries to PiHole and caches the result, so the CORRECT answer is
uncacheable -- re-queried on every lookup, maximum exposure to any gap -- while
the WRONG answer caches for 300s. The instant the router falls back to upstream
(standard forwarder behaviour when the target does not answer) it latches the
WAN address for the whole LAN for five minutes, then expires, re-queries PiHole,
gets TTL 0, and is immediately re-exposed.

So ONE sub-second PiHole gap -- pod restart, L2 lease move, ARP flap, kured
reboot -- becomes FIVE MINUTES of the house resolving to the WAN IP. That
asymmetry, not the gap, is why it reads as "flapping".

Fix: local-ttl=60 prepended to FTLCONF_misc_dnsmasq_lines. dnsmasq applies
local-ttl to replies made from configuration, so one directive covers all five
address= names. 60s rides out a pod restart and a lease re-election while a real
VIP change still propagates inside a minute; these VIPs are pinned and static.

apps/wave-0/pihole.yaml regenerated with the canonical render command
(--all catalog -o apps --wave-layout --no-image-automation, NOT --emit-waves);
exactly two files changed and the rendered diff is the single local-ttl prefix.

UNAPPLIED. Does NOT remove the trigger: .23 still has two independently-elected
L2 leases (as do crafty .22 and unifi .21), which the operator has parked. If
the TTL is still 0 after merge, suspect FTL silently rejecting the passthrough
line -- see the handoff.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-18 02:56:49 -05:00
jakman85andClaude Opus 5 0aa1a0a78a feat(plex): one advertised hostname for local+remote, its own LE cert, and a LAN definition
gate / gate (pull_request) Successful in 1m49s
gate / conformance (pull_request) Has been skipped
Two symptoms after the publishing fix landed:

1. LAN clients were reported as Remote. Measured root cause:
   LanNetworksBandwidth was ABSENT, so Plex had no notion of a local
   network and classified everything WAN -- 468 requests logged
   "Allowed Network (WAN)", ZERO "(LAN)", including Seerr from inside the
   cluster. allowedNetworks only waives AUTH; it does not make a client
   local. Two settings, and only one has an env hook.
2. The advertised URL hard-coded both the VIP and the server's cert hash
   into a cluster-var -- two values nothing kept in sync.

Operator design: ONE hostname, plex-vip.${BASE_DOMAIN}, split-horizon --
PiHole answers it with ${PLEX_L4_VIP} on the LAN, Cloudflare answers it
publicly with the WAN address NATed to the same VIP.

- catalog/plex.yaml: ADVERTISE_IP -> https://plex-vip.${BASE_DOMAIN}:32400,
  plus a level-triggered postStart hook (pihole/qbittorrent idiom) that
  re-asserts LanNetworksBandwidth and the three customCertificate* prefs
  every start, converting PEM -> PKCS#12 so cert renewals need no extra part.
- apps/wave-0/plex-cert.yaml: cert-manager Certificate for exactly that
  name from letsencrypt-prod, so the advertised https URL is servable.
- schema 1.11.0: `optional` on secretKeyRef/configMapKeyRef. Load-bearing --
  the cert is issued INTO the plex namespace, which the rendered plex.yaml
  itself creates, so it lands in the same reconcile; hard refs would have
  held Plex in CreateContainerConfigError and, since apps-wave-0 is
  wait:true, stalled the whole app tier on a slow DNS01 challenge.
- catalog/pihole.yaml: the plex-vip split-horizon answer. Required --
  measured, the name resolved to the Gateway VIP where nothing listens
  on 32400 (http=000).
- PLEX_ADVERTISE_URL deleted; the hostname removes the cert-hash fragility.

Corrects my own earlier claim that a SAN match is REQUIRED: it is not
(httpsRequired=0, secureConnections defaults to Preferred, so clients can
fall back to cleartext). A matching cert keeps the remote leg ENCRYPTED.
Hence: obtain a real cert for the advertised name rather than pick the name
to suit Plex's built-in one.

UNVERIFIED, and stated plainly in the handoff: nothing here has run. Cert
issuance for this name is inferred from the wildcard's success, and whether
Plex accepts an OpenSSL-3 PKCS#12 bundle is untested -- the largest risk.

Touches clusters/** -> sensitive paths -> needs a non-author approval or the
operator's /approve-override; deliberately not self-posted (CLAUDE.md 1).

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-04 03:37:57 -05:00
jakman85andClaude Opus 5 028853c87c feat(pihole): two replicas on different nodes, and fix a PVC mounted where pihole never writes
gate / gate (pull_request) Successful in 1m47s
gate / conformance (pull_request) Has been skipped
Phase 2 of making LAN DNS survive a node failure. Pihole was one pod on one
node; measured 2026-10-04, losing that node meant roughly 6-10 minutes with no
LAN resolution (tolerationSeconds 300 + NotReady detection + a Longhorn RWO
force-detach), while losing the node holding the pihole-dns L2 lease -- a
DIFFERENT node -- meant tens of seconds. Two of three nodes were single points
of failure with wildly different blast radii.

Adds `availability.replicas` to the ServiceDefinition schema (1.10.0). Intent
only; the mechanism is render.py's, documented in the renderer contract. 1 or
absent renders a Deployment exactly as before -- verified byte-identical across
the whole apps/ tree. >1 renders a StatefulSet (a longhorn PVC is RWO, so a
multi-replica Deployment never schedules the second pod), per-replica
volumeClaimTemplates, REQUIRED hostname anti-affinity, podManagementPolicy
Parallel, a minAvailable:1 PodDisruptionBudget, and an SLI that follows the
rendered kind (a Deployment-shaped rule against a StatefulSet absent()s forever).

Pinned by ~20 new --selftest assertions rather than a conformance change.

ALSO FIXES A LIVE BUG. Pihole's config PVC was mounted at the renderer default
/data/config, but pihole writes to /etc/pihole. Measured: /etc/pihole (7.1M --
gravity.db, pihole.toml, query DB) was on the OVERLAY, while the 2Gi Longhorn
volume held lost+found and 24K of nothing. So every restart discarded all state
and re-downloaded 72k gravity domains, and backup.policy has been backing up an
empty volume while reporting success. Fixed here because otherwise the new
per-replica volumes would hold nothing while state stayed ephemeral -- and
because persisting gravity.db is what makes a restart cheap.

The /etc/pihole mount is safe on the image's own evidence: start.sh is PID 1 as
uid 0 and chowns the directory recursively (no fsGroup needed), and it calls
migrate_gravity which creates the database when absent.

The gate caught a real regression on the first run: the wave-0 alert unit test
fed pihole a Deployment-shaped series. Updated; promtool SUCCESS.

Touches no sensitive paths, so this one needs no override.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-04 01:34:52 -05:00
jakman85andClaude Opus 5 3b1d6ae115 feat(pihole): declare the admin password and adlists in git
gate / gate (pull_request) Successful in 1m47s
gate / conformance (pull_request) Has been skipped
Prerequisite for running more than one PiHole replica. A second replica with a
fresh PVC would have come up serving DNS and blocking nothing, with a different
admin password -- answers varying by which pod replied.

pihole.toml's own ### CHANGED markers gave the exact inventory: four settings
already came from env (declared), one came from the image, and exactly ONE
operator-set value lived only on the PVC -- webserver.api.pwhash, a one-way hash
nobody could recover. Separately the adlists were undeclared: one list and 72,234
gravity domains that nothing in this repo named.

Declares both:
- A SOPS secret feeding FTLCONF_webserver_api_password. webserver.api.password
  takes cleartext and FTL hashes it itself (verified: pihole setpassword IS
  setFTLConfigValue on that key), so the PVC's pwhash becomes derived state.
- A level-triggered postStart hook that reconciles the declared adlist set into
  gravity.db -- adds, re-enables, and disables (never deletes) undeclared lists,
  running pihole -g only when something moved. A file would not work: v6's
  adlists.list import is migration-only, guarded on the DB not existing.

The rendered hook was run in the live pod against throwaway copies of gravity.db,
exercising all four paths. A fresh DB built from the image's own schema template
has ZERO adlists, confirming this is load-bearing rather than cosmetic.

Password kept rather than set empty: the pihole Service still publishes port 80
past the oauth2-proxy, closed only by policy, and default-deny has drifted here
before.

Multi-replica is phase 2 and deliberately not in this branch -- it needs a schema
field, a StatefulSet (the config PVC is RWO), anti-affinity and a PDB.

Adds a *.sops.yaml -- a sensitive path -- so approval-policy will block pending
an operator /approve-override.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-04 01:07:30 -05:00
jakman85andClaude Opus 5 99560c2d22 feat(auth): declare API paths per service, so --api-route can ship at all
gate / gate (pull_request) Failing after 1m44s
gate / conformance (pull_request) Has been skipped
PR #28 fixed the cookie collision, the 5-minute session and the missing metrics
listener in one renderer function, and deliberately declined `--api-route`
because the config model could not express the difference between two apps.
This adds the hook it was missing and sets the flag only where a measurement
says to.

THE DEFECT. oauth2-proxy answers an unauthenticated request with a 302 to
Keycloak. Its AJAX guard would send 401 instead, but it fires only on an EXACT
`Accept: application/json`, which these apps' JavaScript does not consistently
send. So a browser XHR gets a 302, FOLLOWS IT INVISIBLY, has it satisfied
silently by the live SSO session, and resolves 200 carrying the app's HTML where
the JavaScript expected JSON. On a write it is worse: a browser following a 302
re-issues a POST as a GET and DROPS THE BODY, so the operator's change is
discarded and the UI reports success.

MEASURED, on two apps and two different paths -- which is the whole argument:
  jackett 2026-09-30: 33 x 302 vs 2 x 401 on /api/ paths, and 4 of 6 POSTs
          answered by the proxy and never reaching Jackett (upstream `-`).
  radarr  2026-10-01, 23h after PR #28 deployed (pod radarr-7d55cbffb9-42hmh,
          0 restarts): 232 unauthenticated requests in the pod's whole life and
          ALL 232 are the same call -- POST /signalr/messages/negotiate from a
          browser, 302, upstream `-`. A left-open tab whose session the cookie
          rename orphaned, polling forever, silently redirected every time and
          NEVER re-prompting. The reported symptom, still in production.

So radarr's broken path is /signalr/ and jackett's is /api/, and PIHOLE SERVES
NO /api/ AT ALL. One function renders all five sidecars, so the cheap fix is a
constant -- and a constant applies one app's measurement to four apps nobody
measured. `auth` was `additionalProperties: false`, so there was no per-app
hook. Adding one named, documented key is the change; the block stays closed.

  schemas/servicedefinition.schema.json  1.8.0 -> 1.9.0: optional
      `auth.apiRoutes`, a list (radarr needs two, and --api-route is natively
      repeatable) of anchored path regexes. Plus the shape invariant that a
      non-empty list requires mode: forward-auth, since no sidecar is rendered
      for oidc/native/none and declaring it there would be a SILENT NO-OP.
  catalog/render.py  emits one --api-route per entry, in order, and RAISES on
      an unanchored entry. The flag is SINGULAR -- verified against the pinned
      v7.8.1 binary's own --help; --api-routes does not exist and the proxy
      refuses to start with it.
  catalog/{jackett,radarr}.yaml  the two measured apps.
  catalog/{pihole,sonarr,qbittorrent}.yaml  comments recording that they get
      NOTHING, and why. Sonarr and qBittorrent were measured (0 unauthenticated
      requests in 23h) and left alone: that is evidence of no TRAFFIC, not of no
      need, and sonarr being nearly radarr's codebase is what makes the guess
      tempting and still a guess.
  apps/wave-0/jackett.yaml, apps/wave-1/radarr.yaml  regenerated. Only TWO
      goldens change; pihole/qbittorrent/sonarr are byte-identical to main,
      which is the per-app property proving itself on the real tree.

BREADTH IS THE DANGEROUS DIRECTION AND IT IS NOT SYMMETRIC. A missing
--api-route is a silent 302. A too-broad one answers a plain PAGE LOAD with
401, so a logged-out operator gets a bare error page and is never offered a
sign-in -- the service looks permanently broken. Each entry must therefore
match `^\^/[A-Za-z0-9._~/-]+(\.\*|\$)?$`: anchored at ^/ then literal path
characters. `^/`, `^/.*`, `.*`, `/api/` and `^/api/|^/` are all rejected, all
five provoked. Alternation is refused rather than supported because the flag is
repeatable, so one entry per route costs nothing.

NEW GATE CHECK: conformance/tests/l0-static/oauth2-proxy-api-routes-per-app
(whole-repo scope, so ci/l0.sh's --include-scope union runs it on every
invocation; no bespoke stage, which whole-repo-invariants-wired would fail).
It is a SIBLING of oauth2-proxy-cookie-names-distinct, not an extension of it:
that check's subject is cookie IDENTITY and shares neither inputs nor failure
mode with this, so bundling them would make its name a lie and let a future
api-route decision be expressed only by weakening a COOKIE check.

The check exists for one reason the other gates cannot cover: render-golden
compares the render to the goldens, so a RENDERER CONSTANT plus regenerated
goldens AGREE WITH EACH OTHER while both being wrong. Provoked: the regen exits
0 -- render-golden is happy -- and this check exits 1 with 15 failures naming
pihole, qbittorrent, radarr and sonarr individually. It also asserts the
anchoring pattern is the SAME in the schema and the renderer, because three
surfaces guard this and if they drift the loosest wins silently.

Tested in both directions on throwaway trees outside the repo: compliant tree
rc 0 / 43 assertions / 6 sidecars; pre-change tree rc 1 / 11 failures; and four
targeted mutations (hand-edited golden, widened regex, renderer constant,
deleted schema pattern) each firing their own assertion class and nothing else.

ALSO FOLDED IN (coordinator's request; defect is in the file I own). _selftest()
reached _cookie_name() through a bare `next(genexpr)` with NO default, so
deleting the --cookie-name emission -- the exact mutation that assertion exists
to catch -- raised StopIteration, which escaped _selftest() BEFORE its collected
`SELFTEST FAIL:` lines were printed. Right verdict, zero diagnostics, and it
masked every other assertion the same mutation broke (found by Desmond Oyelaran
while wiring the selftest into the render-golden stage; his PR #33 is not in this
base -- origin/main @ cdf3e14 drives the selftest from no stage, checked). Fixed
with `next(..., None)` plus an explicit check(), so a missing flag reports as a
NAMED assertion. Provoking it exposed three more instances of the same class in
the same block -- `side = fa_containers[1]`, `side["env"][0]`, and the fa_svc
lookup all raise rather than assert when the sidecar is absent -- so all four are
now None-tolerant, and the two fixtures added for --api-route were written that
way from the start. Measured on throwaway copies:
  --cookie-name deleted:  before rc 1 / 0 FAIL lines / StopIteration
                          after  rc 1 / 3 FAIL lines / no traceback
  sidecar not appended:   before rc 1 / 0 FAIL lines / IndexError
                          after  rc 1 / 20 FAIL lines / no traceback
Goldens are unchanged by this part; it touches _selftest() only.

NOT verified, in full in the handoff entry: no browser round-trip by anyone for
this flag -- that a 401 REPAIRS the operator's experience is reasoned from HTTP
semantics, not watched. `^/api/` on radarr is PR #30's measurement of radarr's
pre-fix pod, not mine; the current pod has zero /api/ traffic. Sonarr and
qBittorrent are UNMEASURED, not cleared. --cookie-refresh is still unexercised
(0 AuthSuccess on all five pods). Nothing was applied; the live pods are still
the pre-change spec.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-01 22:07:08 -05:00
jakman85andClaude Opus 5 fd43f332d6 feat(pihole): authoritative LAN DNS for the zone on udp/53 + tcp/53 at a pinned VIP
platform/coredns/coredns-custom.yaml has always said its split-horizon is for
PODS and deferred LAN clients to "PiHole split-horizon". PiHole served no DNS:
its Service was ClusterIP with ports 80 and 4180 only. The LAN half of R-DNS-03
that docs/09 and the README's Phase 2 row both claim did not exist, so LAN
clients ran on hand-maintained per-client dnsmasq files - one went stale after
the dev decommission and cost the operator sonarr.home.keithjackson.dev. The
SOHO router that owns 192.168.2.0/24 cannot cover for it either: it strips
RFC1918 answers for public domains as rebind protection and returns NOERROR
with no answer.

EXPOSURE (apps/wave-0/pihole-dns.yaml, hand-written, placed into wave 0 by
render.py's EXTRA_UNITS - the schema has no field for a LoadBalancer Service, a
pinned lbipam VIP, a sharing key, source ranges or UDP, the same reason
apps/vms/unifi and platform/crafty hand-write theirs):
  - two Services, udp/53 and tcp/53, sharing one pinned VIP
    ${PIHOLE_DNS_VIP} = 192.168.2.23 on prod, the next free address in the
    .20-.30 pool (pool status IPsUsed 3 of 11; .20 Gateway, .21 unifi, .22
    crafty; arping from a LAN host got no reply).
  - no external-dns hostname annotation: nothing public is created, and
    platform/externaldns/ is untouched.

ANSWERS (catalog/pihole.yaml env - declarative, no reconciler): PiHole v6 takes
every FTL setting as an FTLCONF_ environment variable and treats it as
env-forced, applied ahead of pihole.toml on every cold start and read-only in
the UI. That is a real declarative entry point, which the Keycloak and CNPG
CronJob reconcilers exist precisely because their state lacks, so a reconciler
here would be ceremony. misc.dnsmasq_lines carries a wildcard
address=/<zone>/<GATEWAY_VIP> plus one longer-domain override per L4 service
(unifi, crafty-mc, pihole-dns); dnsmasq resolves overlaps by longest matching
domain, so precedence is structural and a newly onboarded Gateway-fronted
service needs no DNS change. dns.upstreams is declared too - the live pod
carried hand-set upstreams that no manifest knew about and that every restart
discarded.

externalTrafficPolicy Cluster, NOT Local, and loadBalancerSourceRanges for the
LAN restriction. Measured on prod, Cilium 1.20.1: the L2 announcer does not
filter lease candidates by which nodes hold a backend - the two Services landed
on k3s-prod-3 and k3s-prod-2 while the pod ran on k3s-prod-1 - and it keeps
renewing after the pod moves away, so with Local a reschedule leaves LAN DNS
dead indefinitely. Cluster serves the VIP from any node; the LAN restriction
moves to loadBalancerSourceRanges, which Cilium enforces in eBPF on the VIP
before load-balancing (verified by narrowing the range to exclude the test
client and back). The cost is SNAT: forwarded queries reach PiHole from the
ingress node, so PiHole's per-client rate limit is disabled in the catalog or
the whole LAN would count as one client against its 1000-per-60s default.

Proven from a LAN host (192.168.2.132) against 192.168.2.23 with the pod and
both leases on three DIFFERENT nodes: sonarr/grafana/nextcloud/keycloak/apex
-> 192.168.2.20, unifi -> .21, crafty-mc -> .22, pihole-dns -> .23, recursion
(cloudflare.com, github.com, debian.org) intact over both udp and tcp, AAAA
NODATA, NXDOMAIN outside the zone, and still answering after a forced
prod-1 -> prod-2 reschedule.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-14 00:57:58 -05:00
jakman85andClaude Opus 5 c8c1d2935d feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).

THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.

Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.

PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.

BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.

TWO THAT CANNOT BE TOKENIZED, flagged not forced:
  - headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
    over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
    so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
    anyway — client secrets are realm-scoped.
  - kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
    reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
    overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
    prod.home.

EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.

I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.

DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.

L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.

PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.

Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-04 22:43:54 -05:00
jakman85andClaude Opus 4.8 eccc13bb3c T-0.3 ServiceDefinition schema + renderer contract + examples (R-GITOPS-02)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 00:08:30 -05:00