Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity

feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain
gate / gate (push) Has been cancelled
Details
gate / conformance (push) Has been cancelled
Details

Browse Source 
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).

THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.

Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.

PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.

BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.

TWO THAT CANNOT BE TOKENIZED, flagged not forced:
  - headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
    over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
    so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
    anyway — client secrets are realm-scoped.
  - kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
    reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
    overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
    prod.home.

EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.

I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.

DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.

L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.

PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.

Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5 committed 2026-09-04 22:43:54 -05:00
1 parent 9b8979a427
commit c8c1d2935d
59 files changed
+627 -215
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

platform/coredns/coredns-custom.yaml
+2 -2
View File
@@ -29,9 +29,9 @@ metadata:
data:
# Filename `home.server` -> imported as a standalone server block for the whole zone.
home.server: |
home.keithjackson.dev:53 {
${BASE_DOMAIN}:53 {
errors
# Answer EVERY name in the zone (apex + all `<svc>.home.keithjackson.dev`) with the Gateway
# Answer EVERY name in the zone (apex + all `<svc>.${BASE_DOMAIN}`) with the Gateway
# VIP. The template plugin's default match is `.*`, so this covers current and FUTURE app
# hostnames without editing this file per service. `{{ .Name }}` echoes the queried FQDN.
template IN A {
platform/crafty/deployment.yaml
+4 -4
View File
@@ -125,7 +125,7 @@ spec:
- --http-address=0.0.0.0:4180
- --reverse-proxy=true
- --provider=oidc
- --oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab
- --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
# Gate the co-located Crafty UI. Crafty serves HTTPS with a SELF-SIGNED cert, so skip verify
# on this in-pod (127.0.0.1) hop; the browser-facing TLS is the trusted wildcard at the Gateway.
- --upstream=https://127.0.0.1:8443
@@ -138,9 +138,9 @@ spec:
- --allowed-group=platform-admins
# Single provider => 302 straight to Keycloak, no oauth2-proxy interstitial button.
- --skip-provider-button=true
- --redirect-url=https://crafty.home.keithjackson.dev/oauth2/callback
- --cookie-domain=.home.keithjackson.dev
- --whitelist-domain=.home.keithjackson.dev
- --redirect-url=https://crafty.${BASE_DOMAIN}/oauth2/callback
- --cookie-domain=.${BASE_DOMAIN}
- --whitelist-domain=.${BASE_DOMAIN}
- --cookie-secure=true
- --pass-access-token=true
- --set-xauthrequest=true
platform/crafty/httproute.yaml
+1 -1
View File
@@ -25,7 +25,7 @@ spec:
namespace: gateway
sectionName: https
hostnames:
- crafty.home.keithjackson.dev
- crafty.${BASE_DOMAIN}
rules:
- matches:
- path:
platform/crafty/service-l4.yaml
+1 -1
View File
@@ -42,7 +42,7 @@ metadata:
lbipam.cilium.io/sharing-key: "crafty"
lbipam.cilium.io/ips: "${CRAFTY_L4_VIP}"
# external-dns publishes A crafty-mc.home.keithjackson.dev -> the pinned VIP (Java server target).
external-dns.alpha.kubernetes.io/hostname: "crafty-mc.home.keithjackson.dev"
external-dns.alpha.kubernetes.io/hostname: "crafty-mc.${BASE_DOMAIN}"
spec:
type: LoadBalancer
externalTrafficPolicy: Local
platform/externaldns/helmrelease.yaml
+30 -8
View File
@@ -29,15 +29,37 @@ spec:
sources:
- gateway-httproute
- service
# domain-filter MUST be the actual Cloudflare ZONE (keithjackson.dev), not the platform subdomain
# (home.keithjackson.dev) — ExternalDNS filters the account's zones by this value, and a subdomain
# filter drops the parent zone → "no hosted zone matching record DNS Name" and nothing is created.
# Scope stays effectively home.keithjackson.dev because that's the only suffix services request;
# policy=sync + registry=txt only ever touch ExternalDNS-OWNED records, never the pre-existing
# wildcard or other keithjackson.dev records. (docs/09; resolves the WS-NET TODO below.)
# ---------------------------------------------------------------------------------------------
# CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below.
#
# On 2026-09-04 prod's external-dns DELETED EVERY ONE of dev's public records. Both clusters ran
# `policy: sync` against the SAME Cloudflare zone, served the SAME hostnames, and — because
# NEITHER set txtOwnerId — both inherited the chart default owner id `default`. Prod therefore
# read dev's TXT registry entries as its own, found no local resource claiming those names, and
# garbage-collected them. `policy: sync` is correct and stays; what was missing was identity.
# Full write-up: infra/PROD-PROVISIONING.md §10.
#
# TWO INDEPENDENT GUARDS NOW, either of which alone prevents a recurrence:
# 1. txtOwnerId — per-environment identity; a cluster only ever touches records it owns.
# 2. domainFilters — per-environment name scope; prod cannot even see names outside its
# own subtree. Defence in depth, added after the incident.
# ---------------------------------------------------------------------------------------------
# Which names this instance may manage at all. Cloudflare-provider caveat, empirically hit
# 2026-08: this value is used for ZONE selection as well as record matching, and a filter that is
# a strict subdomain of the zone dropped the parent zone entirely ("no hosted zone matching record
# DNS Name") — so DEV keeps the zone apex `keithjackson.dev`, which is also what dev serves the
# apex of. PROD narrows to `prod.home.keithjackson.dev`. If prod's first reconcile logs no
# matching zone, that caveat still applies: prod creates NOTHING (fail-closed, never destructive)
# and the fallback is to widen prod's EXTERNALDNS_DOMAIN_FILTER back to the zone apex, leaving
# txtOwnerId as the sole guard. See clusters/base/cluster-vars.yaml.
domainFilters:
- "${DNS_ZONE}"
policy: sync # create + update + delete records as services come and go
- "${EXTERNALDNS_DOMAIN_FILTER}"
# The TXT-registry owner id. MUST be distinct per environment — this is the knob whose absence
# caused the incident. Dev renders `default` (its live records are already registered under that
# owner; changing it would orphan them and duplicate every record). Prod renders `prod`.
txtOwnerId: "${EXTERNALDNS_TXT_OWNER_ID}"
policy: sync # create + update + delete records as services come and go — but ONLY
# records this owner id owns AND this domainFilter admits (see above).
# Cloudflare API token: SOPS-encrypted secret, referenced BY NAME only (OQ-4, ADR-0007).
env:
- name: CF_API_TOKEN
platform/gateway/gateway.yaml
+4 -4
View File
@@ -13,7 +13,7 @@ spec:
- name: https
protocol: HTTPS
port: 443
hostname: "*.home.keithjackson.dev"
hostname: "*.${BASE_DOMAIN}"
tls:
mode: Terminate
certificateRefs:
@@ -30,7 +30,7 @@ spec:
- name: http-redirect
protocol: HTTP
port: 80
hostname: "*.home.keithjackson.dev"
hostname: "*.${BASE_DOMAIN}"
allowedRoutes:
namespaces:
from: Selector
@@ -46,7 +46,7 @@ spec:
- name: https-apex
protocol: HTTPS
port: 443
hostname: "home.keithjackson.dev"
hostname: "${BASE_DOMAIN}"
tls:
mode: Terminate
certificateRefs:
@@ -61,7 +61,7 @@ spec:
- name: http-apex-redirect
protocol: HTTP
port: 80
hostname: "home.keithjackson.dev"
hostname: "${BASE_DOMAIN}"
allowedRoutes:
namespaces:
from: Selector
platform/gateway/wildcard-certificate.yaml
+2 -2
View File
@@ -10,8 +10,8 @@ metadata:
spec:
secretName: wildcard-home-tls # consumed by the Gateway HTTPS listener below
dnsNames:
- "*.home.keithjackson.dev"
- "home.keithjackson.dev" # apex for the C2 UI (docs/09)
- "*.${BASE_DOMAIN}"
- "${BASE_DOMAIN}" # apex for the C2 UI (docs/09)
issuerRef:
kind: ClusterIssuer
# ${ACME_CLUSTER_ISSUER}: dev -> letsencrypt-staging, prod -> letsencrypt-prod (cluster-vars).
platform/headlamp/httproute.yaml
+1 -1
View File
@@ -24,7 +24,7 @@ spec:
namespace: gateway
sectionName: https
hostnames:
- headlamp.home.keithjackson.dev
- headlamp.${BASE_DOMAIN}
rules:
- matches:
- path:
platform/homepage/configmap.yaml
+15 -15
View File
@@ -36,63 +36,63 @@ data:
---
- Platform:
- Headlamp:
href: https://headlamp.home.keithjackson.dev
href: https://headlamp.${BASE_DOMAIN}
description: Kubernetes dashboard (per-user OIDC RBAC)
icon: mdi-kubernetes
- Grafana:
href: https://grafana.home.keithjackson.dev
href: https://grafana.${BASE_DOMAIN}
description: Metrics, logs & dashboards (fronts Prometheus/Loki/Alertmanager)
icon: mdi-chart-areaspline
- Semaphore:
href: https://semaphore.home.keithjackson.dev
href: https://semaphore.${BASE_DOMAIN}
description: Ansible automation & CI
icon: mdi-cog-play
- Gitea:
href: https://gitea.home.keithjackson.dev
href: https://gitea.${BASE_DOMAIN}
description: Git hosting & Actions (GitOps source of truth)
icon: mdi-git
- Keycloak:
href: https://keycloak.home.keithjackson.dev
href: https://keycloak.${BASE_DOMAIN}
description: Identity provider (SSO / OIDC)
icon: mdi-account-key
- Media:
- Plex:
href: https://plex.home.keithjackson.dev
href: https://plex.${BASE_DOMAIN}
description: Media streaming
icon: mdi-plex
- Seerr:
href: https://seerr.home.keithjackson.dev
href: https://seerr.${BASE_DOMAIN}
description: Media requests
icon: mdi-play-box-multiple
- Sonarr:
href: https://sonarr.home.keithjackson.dev
href: https://sonarr.${BASE_DOMAIN}
description: TV library manager
icon: mdi-television-classic
- Radarr:
href: https://radarr.home.keithjackson.dev
href: https://radarr.${BASE_DOMAIN}
description: Movie library manager
icon: mdi-movie-open
- Jackett:
href: https://jackett.home.keithjackson.dev
href: https://jackett.${BASE_DOMAIN}
description: Indexer proxy
icon: mdi-magnify
- qBittorrent:
href: https://qbittorrent.home.keithjackson.dev
href: https://qbittorrent.${BASE_DOMAIN}
description: Download client (VPN egress)
icon: mdi-download-network
- Network & Infra:
- Pi-hole:
href: https://pihole.home.keithjackson.dev
href: https://pihole.${BASE_DOMAIN}
description: Network-wide DNS & ad-blocking
icon: mdi-shield-check
- UniFi:
href: https://unifi.home.keithjackson.dev:11443
href: https://unifi.${BASE_DOMAIN}:11443
description: Network controller (UOS)
icon: mdi-router-network
- Nextcloud:
href: https://nextcloud.home.keithjackson.dev
href: https://nextcloud.${BASE_DOMAIN}
description: Files, calendar & contacts
icon: mdi-cloud
@@ -104,7 +104,7 @@ data:
href: https://gitea.keithjackson.dev
- Auth portal:
- abbr: SSO
href: https://auth.home.keithjackson.dev
href: https://auth.${BASE_DOMAIN}
- Reference:
- Homepage docs:
- abbr: HP
platform/homepage/deployment.yaml
+1 -1
View File
@@ -54,7 +54,7 @@ spec:
# REQUIRED by Homepage >= 1.0 — requests whose Host header is not in this list get 403.
# The public apex host (served via the Gateway) + the pod IP (for kubelet probes).
- name: HOMEPAGE_ALLOWED_HOSTS
value: "home.keithjackson.dev,$(MY_POD_IP):3000"
value: "${BASE_DOMAIN},$(MY_POD_IP):3000"
resources:
requests:
cpu: 25m
platform/homepage/httproute.yaml
+1 -1
View File
@@ -28,7 +28,7 @@ spec:
namespace: gateway
sectionName: https-apex # the apex listener (wildcard `https` won't match the bare apex)
hostnames:
- home.keithjackson.dev
- ${BASE_DOMAIN}
rules:
- matches:
- path:
platform/keycloak/httproute.yaml
+1 -1
View File
@@ -16,7 +16,7 @@ spec:
namespace: gateway
sectionName: https
hostnames:
- keycloak.home.keithjackson.dev
- keycloak.${BASE_DOMAIN}
rules:
- matches:
- path:
platform/keycloak/keycloak-instance.yaml
+1 -1
View File
@@ -30,7 +30,7 @@ spec:
poolMaxSize: 20
# --- Exposure: TLS at the Gateway; Keycloak serves HTTP and trusts the proxy headers. ---
hostname:
hostname: https://keycloak.home.keithjackson.dev
hostname: https://keycloak.${BASE_DOMAIN}
http:
httpEnabled: true # TLS is terminated at the Gateway (edge), not in Keycloak
proxy:
platform/keycloak/keycloak-realm-import.yaml
+26 -26
View File
@@ -98,9 +98,9 @@ spec:
directAccessGrantsEnabled: false
secret: "$(env:GRAFANA_CLIENT_SECRET)"
redirectUris:
- https://grafana.home.keithjackson.dev/login/generic_oauth
- https://grafana.${BASE_DOMAIN}/login/generic_oauth
webOrigins:
- https://grafana.home.keithjackson.dev
- https://grafana.${BASE_DOMAIN}
defaultClientScopes:
- email
- profile
@@ -125,10 +125,10 @@ spec:
directAccessGrantsEnabled: false
secret: "$(env:OAUTH2_PROXY_CLIENT_SECRET)"
redirectUris:
- https://*.home.keithjackson.dev/oauth2/callback
- https://auth.home.keithjackson.dev/oauth2/callback
- https://*.${BASE_DOMAIN}/oauth2/callback
- https://auth.${BASE_DOMAIN}/oauth2/callback
webOrigins:
- https://*.home.keithjackson.dev
- https://*.${BASE_DOMAIN}
defaultClientScopes:
- email
- profile
@@ -209,9 +209,9 @@ spec:
directAccessGrantsEnabled: false
secret: "$(env:GITEA_CLIENT_SECRET)"
redirectUris:
- https://gitea.home.keithjackson.dev/user/oauth2/keycloak/callback
- https://gitea.${BASE_DOMAIN}/user/oauth2/keycloak/callback
webOrigins:
- https://gitea.home.keithjackson.dev
- https://gitea.${BASE_DOMAIN}
defaultClientScopes:
- email
- profile
@@ -240,9 +240,9 @@ spec:
directAccessGrantsEnabled: false
secret: "$(env:NEXTCLOUD_CLIENT_SECRET)"
redirectUris:
- https://nextcloud.home.keithjackson.dev/apps/user_oidc/code
- https://nextcloud.${BASE_DOMAIN}/apps/user_oidc/code
webOrigins:
- https://nextcloud.home.keithjackson.dev
- https://nextcloud.${BASE_DOMAIN}
# Post-logout redirect allow-list (#logout "redirect failed"). On sign-out user_oidc redirects the
# browser to Keycloak's end_session_endpoint with post_logout_redirect_uri=<Nextcloud base '/'> and
# client_id=nextcloud but NO id_token_hint (provider sendIdTokenHint=0), so Keycloak validates the
@@ -250,7 +250,7 @@ spec:
# "redirect failed". The wildcard covers the base URL and any logout landing path. Keycloak stores
# this as the client attribute `post.logout.redirect.uris` (## separates multiple values).
attributes:
post.logout.redirect.uris: https://nextcloud.home.keithjackson.dev/*
post.logout.redirect.uris: https://nextcloud.${BASE_DOMAIN}/*
defaultClientScopes:
- email
- profile
@@ -279,9 +279,9 @@ spec:
directAccessGrantsEnabled: false
secret: "$(env:SEMAPHORE_CLIENT_SECRET)"
redirectUris:
- https://semaphore.home.keithjackson.dev/api/auth/oidc/keycloak/redirect
- https://semaphore.${BASE_DOMAIN}/api/auth/oidc/keycloak/redirect
webOrigins:
- https://semaphore.home.keithjackson.dev
- https://semaphore.${BASE_DOMAIN}
defaultClientScopes:
- email
- profile
@@ -318,9 +318,9 @@ spec:
directAccessGrantsEnabled: false
secret: "$(env:HEADLAMP_CLIENT_SECRET)"
redirectUris:
- https://headlamp.home.keithjackson.dev/oidc-callback
- https://headlamp.${BASE_DOMAIN}/oidc-callback
webOrigins:
- https://headlamp.home.keithjackson.dev
- https://headlamp.${BASE_DOMAIN}
defaultClientScopes:
- email
- profile
@@ -363,9 +363,9 @@ spec:
directAccessGrantsEnabled: false
secret: "$(env:CRAFTY_CLIENT_SECRET)"
redirectUris:
- https://crafty.home.keithjackson.dev/oauth2/callback
- https://crafty.${BASE_DOMAIN}/oauth2/callback
webOrigins:
- https://crafty.home.keithjackson.dev
- https://crafty.${BASE_DOMAIN}
defaultClientScopes:
- email
- profile
@@ -398,9 +398,9 @@ spec:
directAccessGrantsEnabled: false
secret: "$(env:PIHOLE_CLIENT_SECRET)"
redirectUris:
- https://pihole.home.keithjackson.dev/oauth2/callback
- https://pihole.${BASE_DOMAIN}/oauth2/callback
webOrigins:
- https://pihole.home.keithjackson.dev
- https://pihole.${BASE_DOMAIN}
defaultClientScopes:
- email
- profile
@@ -433,9 +433,9 @@ spec:
directAccessGrantsEnabled: false
secret: "$(env:JACKETT_CLIENT_SECRET)"
redirectUris:
- https://jackett.home.keithjackson.dev/oauth2/callback
- https://jackett.${BASE_DOMAIN}/oauth2/callback
webOrigins:
- https://jackett.home.keithjackson.dev
- https://jackett.${BASE_DOMAIN}
defaultClientScopes:
- email
- profile
@@ -459,9 +459,9 @@ spec:
directAccessGrantsEnabled: false
secret: "$(env:QBITTORRENT_CLIENT_SECRET)"
redirectUris:
- https://qbittorrent.home.keithjackson.dev/oauth2/callback
- https://qbittorrent.${BASE_DOMAIN}/oauth2/callback
webOrigins:
- https://qbittorrent.home.keithjackson.dev
- https://qbittorrent.${BASE_DOMAIN}
defaultClientScopes:
- email
- profile
@@ -485,9 +485,9 @@ spec:
directAccessGrantsEnabled: false
secret: "$(env:RADARR_CLIENT_SECRET)"
redirectUris:
- https://radarr.home.keithjackson.dev/oauth2/callback
- https://radarr.${BASE_DOMAIN}/oauth2/callback
webOrigins:
- https://radarr.home.keithjackson.dev
- https://radarr.${BASE_DOMAIN}
defaultClientScopes:
- email
- profile
@@ -511,9 +511,9 @@ spec:
directAccessGrantsEnabled: false
secret: "$(env:SONARR_CLIENT_SECRET)"
redirectUris:
- https://sonarr.home.keithjackson.dev/oauth2/callback
- https://sonarr.${BASE_DOMAIN}/oauth2/callback
webOrigins:
- https://sonarr.home.keithjackson.dev
- https://sonarr.${BASE_DOMAIN}
defaultClientScopes:
- email
- profile
platform/kubevirt/alerts.yaml
+4 -4
View File
@@ -30,7 +30,7 @@ spec:
annotations:
summary: "KubeVirt VM(s) failing to launch in {{ $labels.namespace }}"
description: "One or more VirtualMachineInstances are in the Failed phase in namespace {{ $labels.namespace }} for 5m — a failed VM launch (image import, scheduling, or qemu error). Check `kubectl get vmi -n {{ $labels.namespace }}` and the virt-launcher pod events."
runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-launch-failed"
runbook_url: "https://${BASE_DOMAIN}/runbooks/kubevirt-launch-failed"
# Crash-loop: the VM's virt-launcher pod is restarting repeatedly (guest kernel panic, OOM, or a
# crashing qemu). >3 restarts in 15m is a crash-loop, not a one-off reboot.
- alert: KubeVirtVMCrashLooping
@@ -42,7 +42,7 @@ spec:
annotations:
summary: "KubeVirt VM crash-looping: {{ $labels.namespace }}/{{ $labels.pod }}"
description: "virt-launcher pod {{ $labels.namespace }}/{{ $labels.pod }} has restarted more than 3 times in 15m — the guest VM is crash-looping (panic / OOM / qemu crash)."
runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-crashloop"
runbook_url: "https://${BASE_DOMAIN}/runbooks/kubevirt-crashloop"
# Control plane down: no virt-controller or virt-api replica reports ready, so no VM lifecycle
# operations can proceed. The metrics are `kubevirt_virt_{controller,api}_ready_status` (per-pod
# 1=ready); sum()==0 means every replica is unready. The absent() clause also fires if the whole
@@ -59,7 +59,7 @@ spec:
annotations:
summary: "KubeVirt control plane degraded"
description: "The KubeVirt control plane (virt-controller/virt-api) is not ready or its metrics are absent for 10m — VM lifecycle operations (create/start/stop) will fail (docs/06)."
runbook_url: "https://home.keithjackson.dev/runbooks/kubevirt-control-plane"
runbook_url: "https://${BASE_DOMAIN}/runbooks/kubevirt-control-plane"
# CDI import stuck: a DataVolume import/clone still in flight after 30m blocks the VM it backs.
# kubevirt_cdi_import_pods_high_restart counts importer pods stuck restarting; a positive value
# sustained means an import is not progressing (metric name confirmed against CDI at bring-up).
@@ -71,4 +71,4 @@ spec:
annotations:
summary: "CDI DataVolume import stuck: {{ $labels.namespace }}/{{ $labels.name }}"
description: "A CDI DataVolume import/clone in {{ $labels.namespace }} has not completed after 30m — the VM disk it backs cannot start. Check the importer pod (often an egress/default-deny block on the image source, or a slow mirror)."
runbook_url: "https://home.keithjackson.dev/runbooks/cdi-import-stuck"
runbook_url: "https://${BASE_DOMAIN}/runbooks/cdi-import-stuck"
platform/monitoring/alerts.yaml
+26 -26
View File
@@ -38,7 +38,7 @@ spec:
annotations:
summary: "Node exporter down on {{ $labels.instance }}"
description: "Prometheus cannot scrape node-exporter on {{ $labels.instance }} for 5m — the node may be down or unreachable."
runbook_url: "https://home.keithjackson.dev/runbooks/node-down"
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-down"
- alert: NodeMemoryPressureHigh
expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) > 0.90
for: 10m
@@ -47,7 +47,7 @@ spec:
annotations:
summary: "Memory pressure high on {{ $labels.instance }}"
description: "Available memory below 10% for 10m on {{ $labels.instance }} — the ~22 GiB box is under pressure."
runbook_url: "https://home.keithjackson.dev/runbooks/node-memory"
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-memory"
- alert: NodeDiskFillingUp
expr: |
(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}
@@ -58,7 +58,7 @@ spec:
annotations:
summary: "Disk filling up on {{ $labels.instance }} ({{ $labels.mountpoint }})"
description: "Less than 10% free on {{ $labels.mountpoint }} of {{ $labels.instance }} for 15m."
runbook_url: "https://home.keithjackson.dev/runbooks/node-disk"
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-disk"
- alert: NodeCPUPressureHigh
# PSI "some" for CPU = fraction of wall-clock time at least one task was stalled waiting to
# run. <5% is healthy; sustained >15% means bursty workloads are queueing against the vCPU
@@ -71,7 +71,7 @@ spec:
annotations:
summary: "CPU pressure high on {{ $labels.instance }}"
description: "CPU PSI (some) above 15% for 15m on {{ $labels.instance }} — tasks are being delayed waiting for CPU (bursts exceeding the node's vCPU headroom). Add vCPUs or trim CPU-heavy workloads."
runbook_url: "https://home.keithjackson.dev/runbooks/node-cpu"
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-cpu"
- alert: NodeCPULoadHigh
# 5-min load average per core. Sustained >1 means the run queue exceeds the core count (more
# runnable/blocked tasks than CPUs). Core count = number of idle-mode cpu series per node.
@@ -86,7 +86,7 @@ spec:
annotations:
summary: "Load per core high on {{ $labels.instance }}"
description: "5-min load average per core above 1.0 for 15m on {{ $labels.instance }} — the node's run queue exceeds its core count."
runbook_url: "https://home.keithjackson.dev/runbooks/node-cpu"
runbook_url: "https://${BASE_DOMAIN}/runbooks/node-cpu"
# =====================================================================================
# Storage — Longhorn
@@ -113,7 +113,7 @@ spec:
annotations:
summary: "Longhorn volume degraded: {{ $labels.volume }}"
description: "Volume {{ $labels.volume }} has been degraded (replica rebuild / missing replica) for 10m."
runbook_url: "https://home.keithjackson.dev/runbooks/longhorn-degraded"
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-degraded"
- alert: LonghornVolumeFaulted
expr: longhorn_volume_robustness == 3
for: 1m
@@ -122,7 +122,7 @@ spec:
annotations:
summary: "Longhorn volume FAULTED: {{ $labels.volume }}"
description: "Volume {{ $labels.volume }} is faulted — data at risk / offline. Immediate attention required."
runbook_url: "https://home.keithjackson.dev/runbooks/longhorn-faulted"
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-faulted"
- alert: LonghornVolumeCapacityHigh
expr: (longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes) > 0.85
for: 30m
@@ -131,7 +131,7 @@ spec:
annotations:
summary: "Longhorn volume near capacity: {{ $labels.volume }}"
description: "Volume {{ $labels.volume }} is over 85% used for 30m."
runbook_url: "https://home.keithjackson.dev/runbooks/longhorn-capacity"
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
- alert: LonghornNodeStorageHigh
expr: (longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) > 0.85
for: 30m
@@ -140,7 +140,7 @@ spec:
annotations:
summary: "Longhorn node storage near capacity: {{ $labels.node }}"
description: "Longhorn node {{ $labels.node }} is over 85% of its storage capacity for 30m."
runbook_url: "https://home.keithjackson.dev/runbooks/longhorn-capacity"
runbook_url: "https://${BASE_DOMAIN}/runbooks/longhorn-capacity"
# =====================================================================================
# GitOps — Flux
@@ -169,7 +169,7 @@ spec:
annotations:
summary: "Flux resource not ready: {{ $labels.kind }}/{{ $labels.name }}"
description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has failed to reconcile (Ready=False) for 15m — GitOps drift/failure (R-GITOPS-03)."
runbook_url: "https://home.keithjackson.dev/runbooks/flux-reconcile"
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-reconcile"
- alert: FluxResourceSuspended
expr: |
max by (namespace, name, kind) (gotk_suspend_status) == 1
@@ -179,7 +179,7 @@ spec:
annotations:
summary: "Flux resource suspended: {{ $labels.kind }}/{{ $labels.name }}"
description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} has been suspended for 1h — reconciliation/drift-correction is off."
runbook_url: "https://home.keithjackson.dev/runbooks/flux-suspended"
runbook_url: "https://${BASE_DOMAIN}/runbooks/flux-suspended"
# =====================================================================================
# Certificates — cert-manager
@@ -205,7 +205,7 @@ spec:
annotations:
summary: "Certificate expiring soon: {{ $labels.name }}"
description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} expires in under 14 days."
runbook_url: "https://home.keithjackson.dev/runbooks/cert-expiry"
runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-expiry"
- alert: CertificateExpiryImminent
expr: (certmanager_certificate_expiration_timestamp_seconds - time()) < (3 * 24 * 3600)
for: 1h
@@ -214,7 +214,7 @@ spec:
annotations:
summary: "Certificate expiry imminent: {{ $labels.name }}"
description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} expires in under 3 days and has not renewed."
runbook_url: "https://home.keithjackson.dev/runbooks/cert-expiry"
runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-expiry"
- alert: CertificateNotReady
expr: certmanager_certificate_ready_status{condition="False"} == 1
for: 1h
@@ -223,7 +223,7 @@ spec:
annotations:
summary: "Certificate not ready / issuance failing: {{ $labels.name }}"
description: "Certificate {{ $labels.namespace }}/{{ $labels.name }} has been in a not-ready state for 1h (ACME issuance failure?)."
runbook_url: "https://home.keithjackson.dev/runbooks/cert-issuance"
runbook_url: "https://${BASE_DOMAIN}/runbooks/cert-issuance"
# =====================================================================================
# Networking — VPN / egress-gateway / denied flows (Hubble)
@@ -255,7 +255,7 @@ spec:
annotations:
summary: "Egress VPN tunnel down"
description: "The egress-gateway VPN tunnel is down (or its metric is absent) for 5m — outbound traffic could leak or stall (leak risk, docs/07)."
runbook_url: "https://home.keithjackson.dev/runbooks/vpn-down"
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-down"
# (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
# retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
# service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`
@@ -270,7 +270,7 @@ spec:
annotations:
summary: "Spike in policy-denied network flows"
description: "Policy-denied flow rate is elevated for 10m — a NetworkPolicy regression or an intrusion probe (security audit, docs/07)."
runbook_url: "https://home.keithjackson.dev/runbooks/denied-flows"
runbook_url: "https://${BASE_DOMAIN}/runbooks/denied-flows"
# =====================================================================================
# Backups — Velero
@@ -296,7 +296,7 @@ spec:
annotations:
summary: "Velero backup failed: schedule {{ $labels.schedule }}"
description: "One or more Velero backups failed in the last hour (R-STATE-03 — off-cluster DR at risk)."
runbook_url: "https://home.keithjackson.dev/runbooks/backup-failure"
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-failure"
- alert: VeleroBackupPartialFailure
expr: increase(velero_backup_partial_failure_total[1h]) > 0
for: 5m
@@ -305,7 +305,7 @@ spec:
annotations:
summary: "Velero backup partially failed"
description: "A Velero backup completed with partial failures in the last hour — some resources/volumes were skipped."
runbook_url: "https://home.keithjackson.dev/runbooks/backup-partial"
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-partial"
- alert: VeleroNoRecentBackup
expr: (time() - velero_backup_last_successful_timestamp{schedule!=""}) > (24 * 3600)
for: 1h
@@ -314,7 +314,7 @@ spec:
annotations:
summary: "No successful Velero backup in 24h: {{ $labels.schedule }}"
description: "Schedule {{ $labels.schedule }} has not produced a successful backup in over 24h (missed schedule)."
runbook_url: "https://home.keithjackson.dev/runbooks/backup-missed"
runbook_url: "https://${BASE_DOMAIN}/runbooks/backup-missed"
# =====================================================================================
# Identity — Keycloak (SSO is a critical dependency — docs/15)
@@ -345,7 +345,7 @@ spec:
annotations:
summary: "Keycloak IdP down"
description: "No ready Keycloak pod for 5m — SSO is unavailable and dependent services cannot authenticate (docs/15)."
runbook_url: "https://home.keithjackson.dev/runbooks/keycloak-down"
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-down"
- alert: KeycloakDatabaseDown
# Keycloak's PostgreSQL runs as a CloudNativePG Cluster `keycloak-db` (pods keycloak-db-N).
# DR-critical, Longhorn-backed. (CNPG also exports cnpg_* metrics via its PodMonitor.)
@@ -358,7 +358,7 @@ spec:
annotations:
summary: "Keycloak database unavailable"
description: "Keycloak's PostgreSQL is not ready for 5m — the IdP will fail (docs/15)."
runbook_url: "https://home.keithjackson.dev/runbooks/keycloak-db"
runbook_url: "https://${BASE_DOMAIN}/runbooks/keycloak-db"
# =====================================================================================
# The monitor's own capacity — don't let the monitor blind itself (docs/10)
@@ -386,7 +386,7 @@ spec:
annotations:
summary: "Monitoring PVC filling up: {{ $labels.persistentvolumeclaim }}"
description: "PVC {{ $labels.persistentvolumeclaim }} (Prometheus/Loki) is under 15% free — retention/PVC size may need raising before the monitor goes blind."
runbook_url: "https://home.keithjackson.dev/runbooks/monitoring-capacity"
runbook_url: "https://${BASE_DOMAIN}/runbooks/monitoring-capacity"
- alert: PrometheusTargetsMissing
expr: |
(count(up == 0) / count(up)) > 0.10
@@ -396,7 +396,7 @@ spec:
annotations:
summary: "More than 10% of Prometheus targets are down"
description: "Over 10% of scrape targets have been down for 15m — a broad monitoring blind spot."
runbook_url: "https://home.keithjackson.dev/runbooks/targets-down"
runbook_url: "https://${BASE_DOMAIN}/runbooks/targets-down"
# =====================================================================================
# Automated updates — Flux image-automation + hands-off update health (docs/11, ADR-0009)
@@ -430,7 +430,7 @@ spec:
annotations:
summary: "Flux image-update-automation failing: {{ $labels.name }}"
description: "ImageUpdateAutomation {{ $labels.namespace }}/{{ $labels.name }} cannot reconcile (git write-back) for 30m — stable image advances have stopped (R-UPD-02). Check the git write credential."
runbook_url: "https://home.keithjackson.dev/runbooks/image-automation"
runbook_url: "https://${BASE_DOMAIN}/runbooks/image-automation"
# An ImagePolicy/ImageRepository can't resolve the latest stable tag (registry rate-limit /
# auth / bad range) → that service stops receiving updates even though automation is healthy.
- alert: ImagePolicyStalled
@@ -444,7 +444,7 @@ spec:
annotations:
summary: "Image scan/policy stalled: {{ $labels.kind }}/{{ $labels.name }}"
description: "{{ $labels.kind }} {{ $labels.namespace }}/{{ $labels.name }} cannot resolve its stable tag for 1h (registry rate-limit / creds / range) — no longer receiving updates (R-UPD-01)."
runbook_url: "https://home.keithjackson.dev/runbooks/image-policy-stalled"
runbook_url: "https://${BASE_DOMAIN}/runbooks/image-policy-stalled"
# A bad update was health-gated and AUTO-ROLLED-BACK (or is stuck mid-remediation). helm-controller
# holds Ready=False while a failed upgrade remediates/rolls back to the last-good revision
# (HelmRelease.spec.upgrade.remediation + .spec.rollback). This is the R-UPD-03 safety rail firing.
@@ -461,4 +461,4 @@ spec:
annotations:
summary: "Update health-gate failed / rolled back: {{ $labels.name }}"
description: "HelmRelease {{ $labels.namespace }}/{{ $labels.name }} failed its health-gated upgrade and is rolling back to the last-good revision (bad-update auto-rollback — R-UPD-03). Consider an `update.pin` escape hatch (docs/11)."
runbook_url: "https://home.keithjackson.dev/runbooks/update-rollback"
runbook_url: "https://${BASE_DOMAIN}/runbooks/update-rollback"
platform/monitoring/grafana-httproute.yaml
+1 -1
View File
@@ -14,7 +14,7 @@ spec:
namespace: gateway
sectionName: https
hostnames:
- grafana.home.keithjackson.dev
- grafana.${BASE_DOMAIN}
rules:
- matches:
- path:
platform/monitoring/kube-prometheus-stack.yaml
+4 -4
View File
@@ -194,15 +194,15 @@ spec:
optional: true
grafana.ini:
server:
root_url: https://grafana.home.keithjackson.dev
root_url: https://grafana.${BASE_DOMAIN}
auth.generic_oauth:
enabled: true
name: Keycloak
client_id: grafana
scopes: "openid email profile"
auth_url: https://keycloak.home.keithjackson.dev/realms/homelab/protocol/openid-connect/auth
token_url: https://keycloak.home.keithjackson.dev/realms/homelab/protocol/openid-connect/token
api_url: https://keycloak.home.keithjackson.dev/realms/homelab/protocol/openid-connect/userinfo
auth_url: https://keycloak.${BASE_DOMAIN}/realms/homelab/protocol/openid-connect/auth
token_url: https://keycloak.${BASE_DOMAIN}/realms/homelab/protocol/openid-connect/token
api_url: https://keycloak.${BASE_DOMAIN}/realms/homelab/protocol/openid-connect/userinfo
login_attribute_path: preferred_username
email_attribute_path: email
# Map Keycloak groups (the `groups` claim) to Grafana roles (R-IDP-06): platform-admins ->
platform/oauth2-proxy/helmrelease.yaml
+4 -4
View File
@@ -32,7 +32,7 @@ spec:
existingSecret: oauth2-proxy-credentials
extraArgs:
provider: oidc
oidc-issuer-url: https://keycloak.home.keithjackson.dev/realms/homelab
oidc-issuer-url: https://keycloak.${BASE_DOMAIN}/realms/homelab
# Forward-auth (ext-auth) mode: no upstream app; return 202 for an authenticated session so the
# Gateway ext-authz can allow/deny. Auth headers are surfaced to the backend.
reverse-proxy: "true"
@@ -41,9 +41,9 @@ spec:
pass-access-token: "true"
email-domain: "*"
# Single sign-in/callback host for all forward-auth-protected services on the platform zone.
redirect-url: https://auth.home.keithjackson.dev/oauth2/callback
whitelist-domain: ".home.keithjackson.dev"
cookie-domain: ".home.keithjackson.dev"
redirect-url: https://auth.${BASE_DOMAIN}/oauth2/callback
whitelist-domain: ".${BASE_DOMAIN}"
cookie-domain: ".${BASE_DOMAIN}"
cookie-secure: "true"
# Expose group membership so per-service group authorization (R-IDP-06) can be enforced. The
# Keycloak `oauth2-proxy` client emits a `groups` claim (full.path=false => bare names like
platform/oauth2-proxy/httproute.yaml
+1 -1
View File
@@ -14,7 +14,7 @@ spec:
namespace: gateway
sectionName: https
hostnames:
- auth.home.keithjackson.dev
- auth.${BASE_DOMAIN}
rules:
- matches:
- path: