jakman85andClaude Opus 5
c8c1d2935d
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).
THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.
Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.
PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.
BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.
TWO THAT CANNOT BE TOKENIZED, flagged not forced:
- headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
anyway — client secrets are realm-scoped.
- kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
prod.home.
EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.
I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.
DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.
L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.
PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.
Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
393 lines
18 KiB
JSON
393 lines
18 KiB
JSON
{
|
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
"$id": "https://home.keithjackson.dev/schemas/servicedefinition.schema.json",
|
|
"title": "ServiceDefinition",
|
|
"description": "Operator-facing onboarding contract for the home k3s + Longhorn platform. One ServiceDefinition per file under catalog/. Captures INTENT; the platform compiles it into concrete resources (see schemas/renderer-contract.md). Canonical source of truth: spec docs/13-service-catalog.md. Serves R-GITOPS-02; ties into R-NET-*, R-STATE-*, R-TLS-*, R-DNS-*, R-OBS-*, R-UPD-*, R-IDP-*. Schema version: 1.3.0 (adds optional pod-level podSecurityContext for the container kind — k8s PodSecurityContext emitted verbatim onto the pod spec; its fsGroup unblocks non-root images e.g. semaphore/UID-1001 writing a mounted Longhorn PVC). 1.2.0 added optional per-container env/envFrom/lifecycle for the container kind — k8s EnvVar/EnvFromSource/Lifecycle shapes emitted verbatim onto the app container; unblocks headless-install/DB-backed apps e.g. gitea/nextcloud/semaphore/unifi/minecraft. 1.1.0 added the shape invariant exposure:public => auth.mode!=none (docs/13 v1.1 / docs/15 / R-IDP-02).",
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"required": ["name", "source"],
|
|
"properties": {
|
|
"name": {
|
|
"type": "string",
|
|
"description": "DNS label -> <name>.${BASE_DOMAIN}, i.e. <name>.home.keithjackson.dev on dev and <name>.prod.home.keithjackson.dev on prod (docs/13, docs/09). Must be a valid RFC 1123 DNS label.",
|
|
"pattern": "^[a-z0-9]([-a-z0-9]*[a-z0-9])?$",
|
|
"minLength": 1,
|
|
"maxLength": 63
|
|
},
|
|
"kind": {
|
|
"type": "string",
|
|
"description": "PaaS container (default) or IaaS VM via KubeVirt (docs/13; R-PLAT-02/03).",
|
|
"enum": ["container", "vm"],
|
|
"default": "container"
|
|
},
|
|
"source": {
|
|
"type": "object",
|
|
"description": "Workload source. container => image; vm => vmTemplate (CDI source). Exactly one, matching kind.",
|
|
"additionalProperties": false,
|
|
"minProperties": 1,
|
|
"properties": {
|
|
"image": {
|
|
"type": "string",
|
|
"description": "Container image reference (repo/image[:tag]). Required when kind=container.",
|
|
"minLength": 1
|
|
},
|
|
"vmTemplate": {
|
|
"type": "string",
|
|
"description": "KubeVirt/CDI source image reference. Required when kind=vm.",
|
|
"minLength": 1
|
|
}
|
|
}
|
|
},
|
|
"env": {
|
|
"type": "array",
|
|
"description": "Optional per-container environment variables for the container kind (docs/13). Mirrors the k8s EnvVar shape and is emitted verbatim onto the app container: each entry is either a literal {name, value} or a {name, valueFrom: {secretKeyRef|configMapKeyRef}} sourced from a Secret/ConfigMap key. Absent => no env (existing apps render byte-identical). Ignored for kind=vm (VMs configure via cloud-init).",
|
|
"default": [],
|
|
"items": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"required": ["name"],
|
|
"properties": {
|
|
"name": {
|
|
"type": "string",
|
|
"description": "Environment variable name (C_IDENTIFIER).",
|
|
"pattern": "^[A-Za-z_][A-Za-z0-9_]*$"
|
|
},
|
|
"value": {
|
|
"type": "string",
|
|
"description": "Literal value. Mutually exclusive with valueFrom."
|
|
},
|
|
"valueFrom": {
|
|
"type": "object",
|
|
"description": "Source the value from a Secret or ConfigMap key (exactly one of secretKeyRef/configMapKeyRef).",
|
|
"additionalProperties": false,
|
|
"minProperties": 1,
|
|
"maxProperties": 1,
|
|
"properties": {
|
|
"secretKeyRef": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"required": ["name", "key"],
|
|
"properties": {
|
|
"name": { "type": "string", "minLength": 1 },
|
|
"key": { "type": "string", "minLength": 1 }
|
|
}
|
|
},
|
|
"configMapKeyRef": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"required": ["name", "key"],
|
|
"properties": {
|
|
"name": { "type": "string", "minLength": 1 },
|
|
"key": { "type": "string", "minLength": 1 }
|
|
}
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"oneOf": [
|
|
{ "required": ["value"], "not": { "required": ["valueFrom"] } },
|
|
{ "required": ["valueFrom"], "not": { "required": ["value"] } }
|
|
]
|
|
}
|
|
},
|
|
"envFrom": {
|
|
"type": "array",
|
|
"description": "Optional bulk env sources for the container kind. Mirrors the k8s EnvFromSource shape and is emitted verbatim onto the app container: each entry pulls ALL keys from a Secret or ConfigMap (exactly one of secretRef/configMapRef). Absent => no envFrom (existing apps render byte-identical). Ignored for kind=vm.",
|
|
"default": [],
|
|
"items": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"minProperties": 1,
|
|
"maxProperties": 1,
|
|
"properties": {
|
|
"secretRef": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"required": ["name"],
|
|
"properties": { "name": { "type": "string", "minLength": 1 } }
|
|
},
|
|
"configMapRef": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"required": ["name"],
|
|
"properties": { "name": { "type": "string", "minLength": 1 } }
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"lifecycle": {
|
|
"type": "object",
|
|
"description": "Optional container lifecycle hooks for the container kind (docs/13). Mirrors the k8s Lifecycle shape and is emitted verbatim onto the app container. Used for idempotent break-glass/bootstrap steps (e.g. gitea's postStart exec that seeds a local admin). Absent => no lifecycle (existing apps render byte-identical). Ignored for kind=vm.",
|
|
"additionalProperties": false,
|
|
"minProperties": 1,
|
|
"properties": {
|
|
"postStart": { "$ref": "#/$defs/execHandler" },
|
|
"preStop": { "$ref": "#/$defs/execHandler" }
|
|
}
|
|
},
|
|
"podSecurityContext": {
|
|
"type": "object",
|
|
"description": "Optional pod-level SecurityContext for the container kind (docs/13). Mirrors the k8s PodSecurityContext shape and is emitted verbatim onto the pod spec (spec.template.spec.securityContext). Its primary use is `fsGroup`, which makes a mounted Longhorn PVC group-writable by a non-root image (e.g. semaphore runs as UID 1001 and cannot otherwise write its config PVC). Absent => no pod securityContext (existing apps render byte-identical). Ignored for kind=vm.",
|
|
"additionalProperties": false,
|
|
"minProperties": 1,
|
|
"properties": {
|
|
"runAsUser": { "type": "integer", "description": "UID the container processes run as." },
|
|
"runAsGroup": { "type": "integer", "description": "Primary GID the container processes run as." },
|
|
"runAsNonRoot": { "type": "boolean", "description": "Require the container to run as a non-root UID (validated at admission)." },
|
|
"fsGroup": { "type": "integer", "description": "Supplemental group applied to mounted volumes so they are group-owned + group-writable (chowned by the kubelet), letting a non-root image write a PVC." },
|
|
"fsGroupChangePolicy": { "type": "string", "enum": ["Always", "OnRootMismatch"], "description": "When the kubelet recursively applies fsGroup ownership to a volume." },
|
|
"supplementalGroups": { "type": "array", "items": { "type": "integer" }, "description": "Additional GIDs added to the first process in each container." }
|
|
}
|
|
},
|
|
"exposure": {
|
|
"type": "string",
|
|
"description": "internal = LAN-only (internal DNS + Gateway on LAN); public = additionally edge port-forwarded 80/443 (docs/09; R-DNS/R-NET).",
|
|
"enum": ["internal", "public"],
|
|
"default": "internal"
|
|
},
|
|
"egress": {
|
|
"type": "string",
|
|
"description": "Egress class (docs/03; R-NET-05). internet = default WAN route; vpn = steered by Cilium Egress Gateway out the egress-gateway node's host PIA VPN tunnel, with a mandatory host kill-switch (ADR-0015; R-NET-06).",
|
|
"enum": ["internet", "vpn"],
|
|
"default": "internet"
|
|
},
|
|
"storage": {
|
|
"type": "array",
|
|
"description": "Persistent volumes. If it must survive a rebuild it lives here, never on ephemeral/hostPath (docs/04; R-STATE-01/02/04).",
|
|
"default": [],
|
|
"items": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"required": ["name", "class"],
|
|
"properties": {
|
|
"name": {
|
|
"type": "string",
|
|
"description": "Volume identifier within the service (e.g. config, media, db).",
|
|
"pattern": "^[a-z0-9]([-a-z0-9]*[a-z0-9])?$"
|
|
},
|
|
"size": {
|
|
"type": "string",
|
|
"description": "PVC request in Kubernetes quantity form (e.g. 5Gi). Required for the longhorn tier; may be omitted for the bulk tier (external mount, sized outside the cluster - docs/04).",
|
|
"pattern": "^[1-9][0-9]*(Mi|Gi|Ti)$"
|
|
},
|
|
"mountPath": {
|
|
"type": "string",
|
|
"description": "Optional explicit absolute container mount path for this volume. Default is /data/<name>. Set it when the app expects a FIXED on-disk layout the default cannot express (e.g. Gitea keeps all state directly under /data). Renderer honors this verbatim; omitting it preserves the /data/<name> default for back-compat (R-STATE-01).",
|
|
"pattern": "^/.*[^/]$"
|
|
},
|
|
"class": {
|
|
"type": "string",
|
|
"description": "longhorn = 3-replica block storage for app/config/DB state (R-STATE-02); bulk = large external media tier, NOT 3x replicated (R-STATE-04).",
|
|
"enum": ["longhorn", "bulk"],
|
|
"default": "longhorn"
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"dependencies": {
|
|
"type": "array",
|
|
"description": "Explicit east-west allow rules; everything else is denied (docs/03; R-NET-01/02). Empty => the service can reach only DNS.",
|
|
"default": [],
|
|
"items": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"required": ["to", "port"],
|
|
"properties": {
|
|
"to": {
|
|
"type": "string",
|
|
"description": "Destination service name (another catalog entry or a co-located component such as this service's DB).",
|
|
"minLength": 1
|
|
},
|
|
"port": {
|
|
"type": "integer",
|
|
"description": "Destination TCP/UDP port.",
|
|
"minimum": 1,
|
|
"maximum": 65535
|
|
},
|
|
"protocol": {
|
|
"type": "string",
|
|
"enum": ["tcp", "udp"],
|
|
"default": "tcp"
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"ingressRoutes": {
|
|
"type": "array",
|
|
"description": "Hostnames/ports to expose north-south; rendered to HTTPRoute + Certificate + DNS (docs/08, docs/09; R-DNS/R-TLS).",
|
|
"default": [],
|
|
"items": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"required": ["host", "port"],
|
|
"properties": {
|
|
"host": {
|
|
"type": "string",
|
|
"description": "Hostname under the platform zone, as a single DNS label plus the ${BASE_DOMAIN} token — e.g. 'gitea.${BASE_DOMAIN}' (docs/09). ${BASE_DOMAIN} is the per-environment hostname suffix from clusters/base/cluster-vars.yaml, resolved by Flux postBuild substitution at reconcile time (dev: home.keithjackson.dev, prod: prod.home.keithjackson.dev). AUTHOR THE TOKEN, NOT A LITERAL: a literal suffix is what let prod be authored to serve dev's public hostnames out of the shared Cloudflare zone, and prod's external-dns then deleted dev's records (infra/PROD-PROVISIONING.md §10). The literal 'home.keithjackson.dev' form stays accepted so the L1 render fixtures — which are renderer INPUTS, not reconciled manifests — keep validating unchanged.",
|
|
"pattern": "^[a-z0-9]([-a-z0-9]*[a-z0-9])?\\.(\\$\\{BASE_DOMAIN\\}|home\\.keithjackson\\.dev)$"
|
|
},
|
|
"port": {
|
|
"type": "integer",
|
|
"description": "Backing Service port the route targets.",
|
|
"minimum": 1,
|
|
"maximum": 65535
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"auth": {
|
|
"type": "object",
|
|
"description": "SSO integration (docs/15, ADR-0014; R-IDP-02/06). Default posture for any exposed UI is forward-auth unless it declares native OIDC (oidc) or a documented exception (native).",
|
|
"additionalProperties": false,
|
|
"required": ["mode"],
|
|
"properties": {
|
|
"mode": {
|
|
"type": "string",
|
|
"description": "oidc = app is a Keycloak OIDC client (own login); forward-auth = oauth2-proxy -> Keycloak at the Gateway; native = app manages its own auth (documented exception, e.g. Plex/Seerr, or Keycloak itself as the IdP); none = no auth (internal-only, discouraged).",
|
|
"enum": ["oidc", "forward-auth", "native", "none"],
|
|
"default": "forward-auth"
|
|
},
|
|
"groups": {
|
|
"type": "array",
|
|
"description": "Keycloak groups permitted access (R-IDP-06). Required and non-empty for oidc/forward-auth (least privilege: authenticated AND in an authorized group). N/A for native/none.",
|
|
"items": { "type": "string", "minLength": 1 },
|
|
"uniqueItems": true,
|
|
"default": []
|
|
}
|
|
}
|
|
},
|
|
"update": {
|
|
"type": "object",
|
|
"description": "Per-service automated-update policy (docs/11; R-UPD-01/02/03). The operator declares the policy once; an automation controller resolves it - no human/AI edits a tag.",
|
|
"additionalProperties": false,
|
|
"properties": {
|
|
"channel": {
|
|
"type": "string",
|
|
"description": "Release channel. stable only unless explicitly opted in (R-UPD-01).",
|
|
"default": "stable"
|
|
},
|
|
"range": {
|
|
"type": "string",
|
|
"description": "Acceptable versions: semver range or tag regex (feeds the Flux ImagePolicy)."
|
|
},
|
|
"cadence": {
|
|
"type": "string",
|
|
"description": "How often to check/apply (the schedule; scan interval)."
|
|
},
|
|
"autoRollbackOnFailedHealth": {
|
|
"type": "boolean",
|
|
"description": "Revert automatically on failed health gates (R-UPD-03).",
|
|
"default": true
|
|
},
|
|
"pin": {
|
|
"type": "string",
|
|
"description": "Operator freeze escape hatch: a fixed version that stops automation for this service."
|
|
}
|
|
}
|
|
},
|
|
"observability": {
|
|
"type": "object",
|
|
"description": "Observability intent; rendered to ServiceMonitor/PodMonitor + log scraping + dashboards + alerts (docs/10; R-OBS-01).",
|
|
"additionalProperties": false,
|
|
"properties": {
|
|
"metrics": {
|
|
"type": "boolean",
|
|
"description": "Whether the platform scrapes metrics for this service.",
|
|
"default": true
|
|
},
|
|
"slis": {
|
|
"type": "array",
|
|
"description": "Per-service alert intents (e.g. up). Each becomes at least one alert rule (R-OBS-01).",
|
|
"items": { "type": "string", "minLength": 1 },
|
|
"default": []
|
|
}
|
|
}
|
|
},
|
|
"backup": {
|
|
"type": "object",
|
|
"description": "Backup posture (docs/04; R-STATE-03). Irreplaceable state MUST be backed up off-cluster; bulk media may be 'reacquirable'.",
|
|
"additionalProperties": false,
|
|
"required": ["policy"],
|
|
"properties": {
|
|
"policy": {
|
|
"type": "string",
|
|
"description": "Reference to an off-cluster backup schedule, or the literal 'reacquirable' for bulk media that is not backed up 1:1.",
|
|
"minLength": 1
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"allOf": [
|
|
{
|
|
"$comment": "kind determines which source field is required (docs/13). Absent kind defaults to container -> image.",
|
|
"if": {
|
|
"required": ["kind"],
|
|
"properties": { "kind": { "const": "vm" } }
|
|
},
|
|
"then": {
|
|
"properties": { "source": { "required": ["vmTemplate"] } }
|
|
},
|
|
"else": {
|
|
"properties": { "source": { "required": ["image"] } }
|
|
}
|
|
},
|
|
{
|
|
"$comment": "oidc/forward-auth require at least one authorized group (R-IDP-06; docs/15 acceptance gate: restricted to the allowed group).",
|
|
"if": {
|
|
"required": ["auth"],
|
|
"properties": {
|
|
"auth": {
|
|
"required": ["mode"],
|
|
"properties": { "mode": { "enum": ["oidc", "forward-auth"] } }
|
|
}
|
|
}
|
|
},
|
|
"then": {
|
|
"properties": {
|
|
"auth": {
|
|
"required": ["groups"],
|
|
"properties": { "groups": { "minItems": 1 } }
|
|
}
|
|
}
|
|
}
|
|
},
|
|
{
|
|
"$comment": "public exposure must not be unauthenticated: an explicit auth.mode 'none' is disallowed on public services (docs/13 v1.1, docs/15, R-IDP-02). Absent auth defaults to forward-auth at render time, so it is permitted here.",
|
|
"if": {
|
|
"required": ["exposure"],
|
|
"properties": { "exposure": { "const": "public" } }
|
|
},
|
|
"then": {
|
|
"properties": {
|
|
"auth": {
|
|
"properties": { "mode": { "not": { "const": "none" } } }
|
|
}
|
|
}
|
|
}
|
|
}
|
|
],
|
|
"$defs": {
|
|
"execHandler": {
|
|
"type": "object",
|
|
"description": "A container lifecycle hook. Only the exec form is supported (a command run inside the container); mirrors k8s LifecycleHandler.exec.",
|
|
"additionalProperties": false,
|
|
"required": ["exec"],
|
|
"properties": {
|
|
"exec": {
|
|
"type": "object",
|
|
"additionalProperties": false,
|
|
"required": ["command"],
|
|
"properties": {
|
|
"command": {
|
|
"type": "array",
|
|
"description": "Command line to execute inside the container. Not run in a shell unless the command itself invokes one.",
|
|
"minItems": 1,
|
|
"items": { "type": "string" }
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|