jakman85andClaude Opus 5
c8c1d2935d
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).
THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.
Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.
PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.
BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.
TWO THAT CANNOT BE TOKENIZED, flagged not forced:
- headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
anyway — client secrets are realm-scoped.
- kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
prod.home.
EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.
I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.
DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.
L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.
PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.
Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
95 lines
6.6 KiB
YAML
95 lines
6.6 KiB
YAML
# ServiceDefinition: plex
|
|
# Source of truth: spec docs/13-service-catalog.md, row 1 (the catalog's largest service).
|
|
# public | egress internet | config on Longhorn (metadata DB, DR-noncritical/reacquirable) |
|
|
# auth: native (Plex accounts, NOT Keycloak — docs/13 + docs/15 documented exception).
|
|
# Validates against schemas/servicedefinition.schema.json.
|
|
#
|
|
# BULK MEDIA (spec bulk-tier requirement, docs/04) — OPEN PREREQUISITE, intentionally NOT declared
|
|
# here yet (the requirement ID is cited in the WS-APPS report, not as a token here: this change does
|
|
# NOT exercise the bulk tier, so it must not be selected as a "touched" requirement whose in-cluster
|
|
# conformance test is still a Phase-1 gap).
|
|
# The Plex library belongs on the `bulk` tier (docs/04 / docs/13 cross-cutting note: metadata DB
|
|
# on replicated Longhorn, large library on bulk). The `bulk` StorageClass exists
|
|
# (platform/longhorn/storageclasses.yaml) and the schema+renderer both support `class: bulk`, but
|
|
# that class is a documented STUB: OQ-2 (external NFS/SMB backing) is UNRESOLVED and the NFS CSI
|
|
# driver is not installed, so a `bulk` PVC will not bind and would leave the Plex pod Pending.
|
|
# To onboard the running Plex shell now (compute + config + public exposure), the media volume is
|
|
# omitted. Once OQ-2 is resolved and `bulk` provisions, attach the library with a one-line add:
|
|
# storage:
|
|
# - name: media
|
|
# class: bulk # external mount; size omitted (sized outside the cluster)
|
|
# then re-render. See the WS-APPS report for the full bulk-media finding.
|
|
name: plex
|
|
kind: container
|
|
source:
|
|
image: plexinc/pms-docker:latest
|
|
# plexinc/pms-docker env. Rendered VERBATIM onto the app container by render.py (schema env list).
|
|
env:
|
|
# PLEX_UID/PLEX_GID: the image boots as root (s6 PID1), chowns /config to this uid/gid, then drops
|
|
# the "Plex Media Server" process to it. 1000 matches the Longhorn ownership convention the other
|
|
# stateful apps use (seerr/semaphore run as UID/GID 1000), keeping the /config PVC writable by the
|
|
# dropped plex user. (Plex has no podSecurityContext/fsGroup — the image manages ownership itself.)
|
|
- name: PLEX_UID
|
|
value: "1000"
|
|
- name: PLEX_GID
|
|
value: "1000"
|
|
# TZ: no other service in the repo sets a timezone (repo-wide grep: none), so use the safe default.
|
|
- name: TZ
|
|
value: Etc/UTC
|
|
# ALLOWED_NETWORKS marks these CIDRs as trusted LAN, so in-cluster clients (Seerr -> plex:32400) are
|
|
# treated as local and NOT pushed onto Plex's remote-auth/relay path. 10.0.0.0/8 covers the Cilium
|
|
# pod CIDR (10.0.0.0/16, CLUSTER_POD_CIDR — Seerr's pod is 10.0.x.x) and the k3s service net
|
|
# (10.43.0.0/16); 192.168.0.0/16 covers the home LAN.
|
|
- name: ALLOWED_NETWORKS
|
|
value: 10.0.0.0/8,192.168.0.0/16
|
|
# PLEX_CLAIM is deliberately NOT set here: a https://plex.tv/claim token has a ~4-min TTL, far
|
|
# shorter than the commit -> push -> Flux reconcile -> pod-roll cycle, so any token committed to git
|
|
# is already expired by the time the pod starts. Claiming is a one-time LIVE step (UI sign-in, or a
|
|
# single API POST) against the running server; the claim then persists on the /config PVC and
|
|
# survives pod rolls (including the roll this env change causes), so it never belongs in git.
|
|
# ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
|
|
# which URL to advertise to plex.tv for REMOTE clients. The only candidate is
|
|
# https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
|
|
# Gateway VIP (conformance/tests/l2-behavioral/dns-internal-resolution), and both its public
|
|
# reachability and whether the TLS-terminating Cilium Gateway proxies Plex's 32400 API correctly for
|
|
# remote access are UNVERIFIED. Advertising a wrong/internal-only URL breaks remote access for ALL
|
|
# clients, so per "prefer safety over completeness" it is left unset. FOLLOW-UP: once remote
|
|
# reachability through the Gateway is confirmed, set ADVERTISE_IP: "https://plex.home.keithjackson.dev:443".
|
|
exposure: public # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET)
|
|
egress: internet # normal WAN egress (docs/03; R-NET-05)
|
|
storage:
|
|
- name: config
|
|
size: 8Gi
|
|
class: longhorn # Plex DB/metadata: replicated app state on the 3-replica tier (docs/04; R-STATE-01/02)
|
|
mountPath: /config # plexinc/pms-docker writes its identity/claim + "Library" DB under /config, NOT the
|
|
# renderer default /data/config. Without this the PVC sat unused and Plex wrote to the
|
|
# container's EPHEMERAL layer -> unclaimed + all state lost on restart. The image inits as
|
|
# root (s6-svscan PID 1), chowns /config to the plex user, then drops -> no fsGroup needed
|
|
# (verified in-cluster: PID1=root, "Plex Media Server"=plex). Persisting /config here also
|
|
# makes the claim survive restarts once claimed. To auto-claim on first boot instead of the
|
|
# UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token
|
|
# (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`,
|
|
# so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
|
|
dependencies: [] # default-deny: reaches only DNS; clients/Seerr reach IT (docs/03; R-NET-01/02)
|
|
ingressRoutes:
|
|
- host: plex.${BASE_DOMAIN}
|
|
port: 32400 # Plex web UI + API served on 32400 (wildcard TLS at the Gateway; docs/08/09)
|
|
auth:
|
|
mode: native # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)
|
|
update:
|
|
channel: stable # R-UPD-01 (never pre-release)
|
|
range: ">=1.40.0" # acceptable versions (advisory; image-automation off during onboarding)
|
|
cadence: "0 4 * * 0" # weekly stable check/apply
|
|
autoRollbackOnFailedHealth: true # R-UPD-03
|
|
observability:
|
|
metrics: true
|
|
slis:
|
|
- up # R-OBS-01: service-down alert
|
|
backup:
|
|
# The config volume is on the longhorn (replicated, "irreplaceable") tier, so the L0 Kyverno
|
|
# backup policy (R-STATE-03) forbids `reacquirable` here — it must reference an off-cluster
|
|
# schedule. Plex's metadata DB (library index, thumbnails, watch state) is expensive to rebuild,
|
|
# so it is backed up off-cluster. The bulk library itself, once attached, is the `reacquirable`
|
|
# part (docs/04) and is NOT covered by this schedule.
|
|
policy: daily-offcluster # off-cluster backup of the longhorn config volume (docs/04; R-STATE-03)
|