INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).
THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.
Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.
PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.
BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.
TWO THAT CANNOT BE TOKENIZED, flagged not forced:
- headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
anyway — client secrets are realm-scoped.
- kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
prod.home.
EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.
I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.
DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.
L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.
PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.
Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
135 lines
7.7 KiB
YAML
135 lines
7.7 KiB
YAML
# ServiceDefinition: semaphore
|
|
# Source of truth: spec docs/13-service-catalog.md row 10; docs/15-identity-and-access.md; docs/04.
|
|
# internal Ansible UI | egress internet | config on Longhorn + external PostgreSQL | auth oidc.
|
|
#
|
|
# Semaphore (semaphoreui/semaphore) is a web UI for running Ansible/Terraform. It is a stateless
|
|
# app over (a) its /etc/semaphore config PVC on Longhorn and (b) an external PostgreSQL — the
|
|
# SECOND CNPG-DB app, reusing the pattern nextcloud established (apps/nextcloud-db.yaml). Its DB is
|
|
# a CloudNativePG Cluster in a dedicated namespace (apps/semaphore-db.yaml).
|
|
#
|
|
# DB WIRING (schema >=1.2.0 env): the app reaches its PostgreSQL cross-namespace at
|
|
# semaphore-db-rw.semaphore-db:5432 (the CNPG read-write Service). Its owner credentials are the
|
|
# SAME password held in two SOPS secrets — semaphore-db-app (basic-auth, ns semaphore-db; consumed
|
|
# by the CNPG Cluster's initdb) and semaphore-bootstrap (ns semaphore; consumed here as db-password).
|
|
# A Pod can only read Secrets in its OWN namespace, so the shared password is duplicated across the
|
|
# two encrypted files rather than replicated cross-namespace (documented in apps/semaphore-db.yaml).
|
|
#
|
|
# PERSISTENT ENCRYPTION KEY: SEMAPHORE_ACCESS_KEY_ENCRYPTION is a base64 32-byte AES key Semaphore
|
|
# uses to encrypt the SSH keys / access credentials it stores in the DB. It MUST be FIXED and stable
|
|
# across restarts — if it changes, every stored credential becomes undecryptable. It is pinned via
|
|
# the semaphore-bootstrap SOPS secret (like gitea's SECRET_KEY), never regenerated per boot.
|
|
# Validates against schemas/servicedefinition.schema.json.
|
|
name: semaphore
|
|
kind: container
|
|
source:
|
|
image: semaphoreui/semaphore:v2.19.10 # pinned recent stable tag (R-UPD-01); update.range governs advancement
|
|
env:
|
|
# --- PostgreSQL connection (the CNPG-DB pattern). DIALECT/HOST/PORT/DB/USER are literals; PASS is
|
|
# sourced from the in-namespace SOPS secret (the CNPG -app secret lives in ns semaphore-db,
|
|
# unreachable by a Pod secretKeyRef, so the same value is mirrored into semaphore-bootstrap). ---
|
|
- name: SEMAPHORE_DB_DIALECT
|
|
value: postgres
|
|
- name: SEMAPHORE_DB_HOST
|
|
value: semaphore-db-rw.semaphore-db.svc.cluster.local
|
|
- name: SEMAPHORE_DB_PORT
|
|
value: "5432"
|
|
- name: SEMAPHORE_DB
|
|
value: semaphore
|
|
- name: SEMAPHORE_DB_USER
|
|
value: semaphore
|
|
- name: SEMAPHORE_DB_PASS
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: semaphore-bootstrap
|
|
key: db-password
|
|
# --- Break-glass local admin (docs/15 availability caveat): created on first-boot so the operator
|
|
# is never locked out by an IdP outage. Username/password sourced from the SOPS secret; the
|
|
# display name + email are non-sensitive literals. ---
|
|
- name: SEMAPHORE_ADMIN
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: semaphore-bootstrap
|
|
key: admin-username
|
|
- name: SEMAPHORE_ADMIN_PASSWORD
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: semaphore-bootstrap
|
|
key: admin-password
|
|
- name: SEMAPHORE_ADMIN_NAME
|
|
value: Semaphore Admin
|
|
- name: SEMAPHORE_ADMIN_EMAIL
|
|
value: admin@${BASE_DOMAIN}
|
|
# --- Persistent access-key encryption key (see header): FIXED base64 32-byte key from SOPS so the
|
|
# SSH keys / credentials Semaphore stores in the DB stay decryptable across restarts. ---
|
|
- name: SEMAPHORE_ACCESS_KEY_ENCRYPTION
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: semaphore-bootstrap
|
|
key: access-key-encryption
|
|
# Because the Service is named `semaphore`, the kubelet injects a Docker-link-style
|
|
# SEMAPHORE_PORT=tcp://<svcIP>:3000 env var, which COLLIDES with Semaphore's OWN SEMAPHORE_PORT
|
|
# (its HTTP listen port) and crashes config validation ("field 'Port' is not valid: tcp://..."). An
|
|
# explicit env entry shadows the injected one (the kubelet gives user-defined env precedence over
|
|
# service links), pinning the listen port back to :3000 (matches containerPort/Service).
|
|
- name: SEMAPHORE_PORT
|
|
value: ":3000"
|
|
# --- Keycloak native OIDC SSO (auth.mode oidc; docs/15 "Native OIDC"; R-IDP-02/06). Semaphore
|
|
# v2.19.10 reads its OIDC providers from the SEMAPHORE_OIDC_PROVIDERS env as a JSON map
|
|
# (util ConfigType `oidc_providers`, env-tagged SEMAPHORE_OIDC_PROVIDERS). The value is a single
|
|
# JSON blob for the `keycloak` provider — provider_url is the homelab realm issuer (auto-discovery),
|
|
# client_id=semaphore, redirect_url=/api/auth/oidc/keycloak/redirect, claims map to the realm's
|
|
# token. It CONTAINS the confidential client secret, so the whole blob is held SOPS-encrypted in the
|
|
# semaphore-bootstrap Secret (key oidc-providers-json; same secret value SOPS-held in
|
|
# keycloak-realm-clients) and injected here by reference — never in cleartext. The name is NOT a
|
|
# kubelet service-link var (those are SEMAPHORE_SERVICE_HOST / SEMAPHORE_PORT_*), so no collision.
|
|
# The LOCAL SEMAPHORE_ADMIN above stays as the break-glass path (docs/15). ---
|
|
- name: SEMAPHORE_OIDC_PROVIDERS
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: semaphore-bootstrap
|
|
key: oidc-providers-json
|
|
# The semaphoreui/semaphore image runs as the non-root user UID 1001 / GID 0 (tini -> server-wrapper),
|
|
# and on boot the wrapper writes its generated config.json into the config dir. A mounted Longhorn PVC
|
|
# is root:root, so UID 1001 cannot write it -> the pod panics ("permission denied"). fsGroup=1001 makes
|
|
# the kubelet group-own the mounted PVC to 1001 and add it as a supplemental group so the process can
|
|
# write; runAsUser/runAsGroup pin the image's own identity, runAsNonRoot asserts it (R-STATE-01/02).
|
|
podSecurityContext:
|
|
runAsUser: 1001
|
|
runAsGroup: 0
|
|
runAsNonRoot: true
|
|
fsGroup: 1001
|
|
exposure: internal # LAN-only (internal DNS + Gateway on LAN); not edge-forwarded (docs/09; R-DNS/R-NET)
|
|
egress: internet # normal WAN egress (docs/03; R-NET-05) — clones repos, installs Galaxy roles
|
|
storage:
|
|
- name: config
|
|
size: 1Gi
|
|
class: longhorn # config (generated config.json, tmp inventories) on the 3-replica tier
|
|
mountPath: /etc/semaphore # Semaphore reads/writes its config under /etc/semaphore (R-STATE-02)
|
|
dependencies:
|
|
- to: semaphore-db # explicit allow: Semaphore -> its PostgreSQL (docs/13 graph; R-NET-02).
|
|
port: 5432 # Renders the egress half; the DB namespace mirrors keycloak-db (no
|
|
protocol: tcp # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
|
|
ingressRoutes:
|
|
- host: semaphore.${BASE_DOMAIN}
|
|
port: 3000 # Semaphore HTTP web/API; wildcard TLS terminates at the shared Gateway (docs/08/09)
|
|
auth:
|
|
mode: oidc # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)
|
|
groups:
|
|
- platform-admins # access scoped to an authorized group (R-IDP-06)
|
|
# WIRED (native OIDC): the Keycloak `semaphore` confidential client (platform/keycloak/keycloak-realm-
|
|
# import.yaml + live kcadm) is consumed via the SEMAPHORE_OIDC_PROVIDERS env (JSON blob held encrypted
|
|
# in the semaphore-bootstrap Secret, oidc-providers-json). The `groups` claim is exposed for
|
|
# authorization (R-IDP-06). Semaphore KEEPS its LOCAL admin as the break-glass path so an IdP outage
|
|
# never locks the operator out (docs/15 "Availability caveat"). No secret is embedded here.
|
|
update:
|
|
channel: stable # R-UPD-01
|
|
range: ">=2.19.0 <3.0.0" # stay on the 2.x stable line
|
|
cadence: "0 4 * * *"
|
|
autoRollbackOnFailedHealth: true # R-UPD-03
|
|
observability:
|
|
metrics: true
|
|
slis:
|
|
- up # automation UI down is a critical alert (docs/10; R-OBS-01)
|
|
backup:
|
|
policy: daily-offcluster # DR-critical: its DB holds automation credentials, backed up off-cluster (docs/13, docs/04; R-STATE-03)
|