Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
  jakman85andClaude Opus 4.8
|
7a34590b19 |
No files matched your search
+31
-71
@@ -1,46 +1,19 @@
|
||||
# .gitea/workflows/gate.yml — the ENFORCED PR merge gate (Phase 6, WS-GITOSS).
|
||||
#
|
||||
# This is the LIVE gate wired to the in-cluster Gitea Actions runner (ns gitea-actions,
|
||||
# Docker executor + DinD). It invokes the AGGREGATE merge gate — `./ci/gate.sh`, which is
|
||||
# exactly what `make -C ci gate` runs, so CI and the devcontainer gate the same way
|
||||
# (R-TEST-02): the runner just installs the pinned tools from ci/versions.env first (and see
|
||||
# below for why the script, not `make`, is invoked here). It runs every stage in order —
|
||||
# spec-immutability, lint, Kyverno L0, render==golden, L1 offline render — and then applies the
|
||||
# R-TEST-03 decision, printing
|
||||
# "MERGE GATE: PASS" (0) / "BLOCKED" (1) / "UNDETERMINED" (3) / "NOT APPLICABLE" (0). It is the
|
||||
# single required status check on `main` (see branch protection). A non-zero exit fails the job,
|
||||
# so a BLOCKED gate blocks the merge.
|
||||
# Docker executor + DinD). It runs the STATIC layers of the conformance gate — the ones
|
||||
# docs/14 says run "without a cluster": spec-immutability (best-effort), lint, Kyverno L0,
|
||||
# the render==golden reproducibility check, and L1 offline render checks. It is the single
|
||||
# required status check on `main`
|
||||
# (see branch protection). `make -C ci lint l0 render-golden l1` here is byte-identical to a laptop run
|
||||
# (R-TEST-02) — the runner just installs the pinned tools from ci/versions.env first.
|
||||
#
|
||||
# WHY THIS STEP DECLARES GATE_STRICT_SKIPS AND TOLERATES EXIT 3 (2026-09-13). A SKIPPED stage is
|
||||
# never green, and since 2026-09-13 gate.sh refuses to print PASS when one did not run (it used to
|
||||
# print "— not green" and then "MERGE GATE: PASS", exit 0 — fail-open in the gate itself). Two
|
||||
# kinds of non-run must be told apart HERE, in the caller, because only the caller knows which
|
||||
# environment it is:
|
||||
# * a pinned tool missing or a seam unwired ON THIS RUNNER is a DEFECT — the toolchain is
|
||||
# installed for us two steps up. GATE_STRICT_SKIPS=true DECLARES that (no env sniffing, no
|
||||
# "does this look like CI?" heuristic) and makes those skips exit 1: untolerated, job fails.
|
||||
# * spec-immutability CANNOT be verified here at all: the spec repo is a separate repo and is
|
||||
# not checked out into this runner (SPEC_REPO_SLUG unwired; blueprint in merge-gate.yaml). It
|
||||
# is the ONLY skip that can still reach exit 3 under strict skips, which is what makes
|
||||
# `|| [ "$?" -eq 3 ]` a narrow, reviewed tolerance rather than a blanket one. The log still
|
||||
# says UNDETERMINED and names R-TEST-06 as unverified. Wire the spec checkout (or set
|
||||
# REQUIRE_SPEC_VERIFICATION=true) and this tolerance should be deleted.
|
||||
# Called DIRECTLY, not through `make`: GNU make reports any recipe failure as its own exit 2, so
|
||||
# `make -C ci gate || [ "$?" -eq 3 ]` would see 2 and fail the job. Same reason the
|
||||
# spec-immutability pre-step below calls the script directly.
|
||||
#
|
||||
# The cluster-gated L2/L3 conformance stage SKIPS honestly here (exit 3, surfaced not faked
|
||||
# — HARD RULE 4) because this runner declares no cluster: nothing here sets CLUSTER_AVAILABLE,
|
||||
# so gate.sh reports the touched requirements UNVERIFIED — and NAMES that declared gap in its
|
||||
# PASS headline — without blocking Phase-0/6 static work. L2/L3 run as an IN-CLUSTER Chainsaw Job (its own scoped SA; CI holds
|
||||
# no broad kubeconfig) driven by the separate `conformance` job below.
|
||||
#
|
||||
# NOTE (R-TEST-06 / gate-semantics): spec-immutability is ALSO surfaced as its own explicit
|
||||
# early step (below) so its result is a distinct, tolerant status in the Gitea UI; gate.sh
|
||||
# re-runs it as its first stage, which is the authoritative one. The gate step below is the
|
||||
# same script the laptop runs; the extra pre-step is a read-only git-diff check and is
|
||||
# purely additive. The full pipeline blueprint (spec-checkout, per-layer jobs, decision job)
|
||||
# is retained dormant in merge-gate.yaml.
|
||||
# The L2/L3 CONFORMANCE layer (behavioural + fault-injection, needs a live cluster) is
|
||||
# NOT wired yet — it is a deliberate design decision (a CI toolchain image + a scoped
|
||||
# in-cluster kubeconfig/RBAC for the runner). It is present below as an `if: false`
|
||||
# placeholder so the shape is visible and the wiring is a one-line flip once decided.
|
||||
# The full pipeline blueprint (spec-checkout, per-layer jobs, decision job) is retained
|
||||
# dormant in merge-gate.yaml.
|
||||
---
|
||||
name: gate
|
||||
|
||||
@@ -74,31 +47,19 @@ jobs:
|
||||
./.gitea/workflows/scripts/install-tools.sh l0
|
||||
./.gitea/workflows/scripts/install-tools.sh l1
|
||||
|
||||
# R-TEST-06: the spec is read-only to agents. An EXPLICIT early surfacing of the
|
||||
# spec-immutability check, so its result is a distinct, tolerant status line in the
|
||||
# Gitea UI before the (longer) aggregate gate log. The spec repo is NOT checked out
|
||||
# into CI here (separate repo; SPEC_REPO_SLUG not wired into this runner), so the guard
|
||||
# honestly SKIPS (exit 3) rather than faking green; a real drift/edit of a co-located
|
||||
# spec exits 1 and fails. Tolerated: exit 3 only. Call the script DIRECTLY, not via
|
||||
# `make spec-immutability`: GNU make masks a recipe's exit code as its own 2, so
|
||||
# `make spec-immutability || [ $? -eq 3 ]` would see 2 and fail. The AUTHORITATIVE
|
||||
# spec-immutability run is the FIRST stage of `make -C ci gate` below (gate.sh calls
|
||||
# the script directly, so it sees exit 3 → SKIP, not a mask); this pre-step is a cheap,
|
||||
# read-only git-diff duplicate that only adds early UI signal.
|
||||
# R-TEST-06: the spec is read-only to agents. The spec repo is NOT checked out into
|
||||
# CI here (it is a separate repo; SPEC_REPO_SLUG is not wired into this runner), so
|
||||
# the guard honestly SKIPS (exit 3) rather than faking green. A real drift/edit of a
|
||||
# co-located spec would exit 1 and fail. Tolerated: exit 3 only.
|
||||
# Call the script DIRECTLY, not via `make`: GNU make masks a recipe's exit code as
|
||||
# its own 2 on failure, so `make spec-immutability || [ $? -eq 3 ]` would see 2 and
|
||||
# fail. The script itself exits 3 on an honest skip (spec repo not checked out here),
|
||||
# which we tolerate; a real drift/edit exits 1 and fails the gate.
|
||||
- name: Spec immutability (R-TEST-06, skips if spec repo absent)
|
||||
run: ./ci/spec-immutability.sh || [ "$?" -eq 3 ]
|
||||
|
||||
# Aggregate merge gate: the same script a human runs in the devcontainer via
|
||||
# `make -C ci gate`. Runs spec-immutability, lint, Kyverno L0, render==golden and L1
|
||||
# render, then the R-TEST-03 decision. Exits non-zero on a BLOCKED gate, which fails this
|
||||
# job. The cluster-gated L2/L3 stage SKIPS here (no CLUSTER_AVAILABLE) — surfaced as
|
||||
# UNVERIFIED and NAMED in the PASS headline, never faked green. GATE_STRICT_SKIPS + the
|
||||
# tolerated exit 3 are explained in the file header; the tolerance covers exactly one
|
||||
# thing, the unverifiable spec repo, and nothing else can produce a 3 here.
|
||||
- name: Merge gate — ci/gate.sh (what `make -C ci gate` runs)
|
||||
env:
|
||||
GATE_STRICT_SKIPS: "true"
|
||||
run: ./ci/gate.sh || [ "$?" -eq 3 ]
|
||||
- name: Static gate — lint + Kyverno L0 + render==golden + L1 render
|
||||
run: make -C ci lint l0 render-golden l1
|
||||
|
||||
# NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4
|
||||
# ("not supported on GHES" — Gitea does not implement the artifact-v2 backend), which
|
||||
@@ -107,16 +68,15 @@ jobs:
|
||||
# Gitea-compatible @v3 action or Gitea's own artifact backend if retention is wanted.
|
||||
|
||||
# ───────────────────────────────────────────────────────────────────────────
|
||||
# PLACEHOLDER — NOT WIRED (deferred; concrete plan written, awaiting operator decision).
|
||||
# L2/L3 conformance (Chainsaw behavioural + fault-injection) needs a LIVE cluster and is
|
||||
# gated behind vars.CLUSTER_AVAILABLE + a narrow secrets.CONFORMANCE_TRIGGER_KUBECONFIG.
|
||||
# The intended model: an in-cluster Chainsaw Job (conformance/incluster/) under its OWN
|
||||
# scoped ServiceAccount — CI holds NO broad kubeconfig, only a trigger credential. The
|
||||
# blocker is a real privilege-escalation design question (a `create jobs` credential can
|
||||
# create a Job running as the privileged runner SA), plus a runner image + repo-clone +
|
||||
# results-collection that don't exist yet. Full analysis + step sequence:
|
||||
# docs/plans/l2-l3-conformance-ci-wiring.md
|
||||
# The blueprint job shape is retained dormant in merge-gate.yaml.
|
||||
# PLACEHOLDER — NOT WIRED (STOP-and-report item, WS-GITOSS Phase 6).
|
||||
# L2/L3 conformance (Chainsaw behavioural + fault-injection) needs a LIVE cluster.
|
||||
# Turning this on is a design decision with two open sub-decisions:
|
||||
# 1. Toolchain: build+push a pinned CI image to the Gitea container registry and
|
||||
# reference it as the job image, vs. install-in-job from ci/versions.env (slower).
|
||||
# 2. Cluster creds: how the runner reaches the cluster — the intended model is an
|
||||
# in-cluster Chainsaw Job driven by its OWN scoped ServiceAccount (CI holds only a
|
||||
# narrow trigger credential; the runner mounts NO SA token today). That SA + RBAC
|
||||
# + the trigger secret do not exist yet.
|
||||
# Kept `if: false` so it never runs (shows skipped, never fakes green — HARD RULE 4).
|
||||
# ───────────────────────────────────────────────────────────────────────────
|
||||
conformance:
|
||||
|
||||
@@ -39,21 +39,6 @@ install_chainsaw() {
|
||||
curl -fsSL "$url" | tar -xz -C "$bindir" chainsaw
|
||||
}
|
||||
|
||||
# promtool — the alert-rule UNIT-TEST engine the L0 check
|
||||
# conformance/tests/l0-static/alert-rules-unit-tested runs. Installed for the l0 stage because
|
||||
# without it that check exits 3 (SKIPPED, never green) on the runner and the alert unit tests
|
||||
# would be enforced nowhere — the exact "coverage that looks present" failure it exists to end.
|
||||
# Pinned from ci/versions.env to the Prometheus version prod runs (promtool IS the rule engine);
|
||||
# the check asserts `promtool --version` equals that pin, so a drifted install FAILS, not skips.
|
||||
install_promtool() {
|
||||
have promtool && return 0
|
||||
log "promtool ${PROMTOOL_VERSION} (from the prometheus release tarball)"
|
||||
local tarball="prometheus-${PROMTOOL_VERSION}.${os}-${arch}"
|
||||
curl -fsSL "https://github.com/prometheus/prometheus/releases/download/v${PROMTOOL_VERSION}/${tarball}.tar.gz" \
|
||||
| tar -xz -C "$bindir" --strip-components=1 "${tarball}/promtool"
|
||||
chmod +x "${bindir}/promtool"
|
||||
}
|
||||
|
||||
install_kubectl() {
|
||||
have kubectl && return 0
|
||||
log "kubectl ${KUBECTL_VERSION}"
|
||||
@@ -65,10 +50,10 @@ have() { command -v "$1" >/dev/null 2>&1; }
|
||||
|
||||
case "$stage" in
|
||||
lint) install_yamllint ;;
|
||||
l0) install_yamllint; install_check_jsonschema; install_kyverno; install_promtool ;; # schema (check-jsonschema) + Kyverno L0 + alert unit tests
|
||||
l0) install_yamllint; install_check_jsonschema; install_kyverno ;; # schema (check-jsonschema) + Kyverno L0
|
||||
l1) install_chainsaw ;; # L1 render via conformance runner (Chainsaw)
|
||||
conformance) install_chainsaw; install_kubectl ;; # trigger the in-cluster Job
|
||||
all) install_yamllint; install_check_jsonschema; install_kyverno; install_promtool; install_chainsaw; install_kubectl ;;
|
||||
all) install_yamllint; install_check_jsonschema; install_kyverno; install_chainsaw; install_kubectl ;;
|
||||
*) echo "unknown stage: $stage" >&2; exit 2 ;;
|
||||
esac
|
||||
log "stage '${stage}' tools ready"
|
||||
@@ -41,6 +41,3 @@ infra/tofu/.r2.env
|
||||
# ignoring it keeps `git add -A` from staging embedded worktree repos and keeps the lint
|
||||
# stage (which scans untracked files) from tripping on .claude/worktrees/.
|
||||
.claude/
|
||||
|
||||
# Vendored ansible collections (installed by ansible-galaxy for node re-provision; not tracked or linted)
|
||||
infra/ansible/collections/
|
||||
@@ -1,97 +0,0 @@
|
||||
# RECOVERY BASELINE — the COMPLETE Cloudflare record set for zone `keithjackson.dev`
|
||||
# as it stood IMMEDIATELY BEFORE feat/externaldns-scope-public-only is applied.
|
||||
#
|
||||
# WHY THIS FILE EXISTS. The branch that adds it changes which resources ExternalDNS considers,
|
||||
# and ExternalDNS runs `policy: sync` — a record whose source stops matching is DELETED, not
|
||||
# merely left alone. That is a one-way, outward-facing operation against a live authoritative
|
||||
# zone, so the pre-state is captured and committed BEFORE the change lands, and any record the
|
||||
# reconcile removes can be restored by hand from this file. It is the same discipline as
|
||||
# dns-before.txt / dns-verify.txt / dns-post-l2.txt beside it, applied to a destructive-capable
|
||||
# change rather than to a migration.
|
||||
#
|
||||
# CAPTURED 2026-09-14 from the Cloudflare API (GET /zones/<id>/dns_records, all pages) using the
|
||||
# SOPS-encrypted token in platform/externaldns/cloudflare-api-token.sops.yaml. Read at the
|
||||
# AUTHORITY, not through a resolver: 1.1.1.1 served stale answers for minutes after the
|
||||
# 2026-09-13 apex flip and must not be trusted for this. `dig` is on the host but NOT in the
|
||||
# devcontainer, which has previously produced false "no records" readings — the API is the only
|
||||
# reading that is both complete and authoritative.
|
||||
#
|
||||
# FORMAT (5 fields, `|`-separated) — wider than the sibling files on purpose: a restore needs the
|
||||
# TTL and the proxy flag, not just the value.
|
||||
#
|
||||
# <type>|<name>|<content>|<ttl>|<proxied>
|
||||
#
|
||||
# ttl 1 = Cloudflare "Automatic". proxied false = DNS-only (grey cloud), which every record here
|
||||
# is; a restore that flips one to proxied would put Cloudflare's edge in front of a LAN address.
|
||||
#
|
||||
# 39 records. 36 of them are ExternalDNS-managed (18 A + their 18 `a-`-prefixed TXT registry
|
||||
# entries), every one registered `external-dns/owner=prod` — i.e. THIS cluster owns them, which
|
||||
# is why this cluster's reconcile is able to delete them. The remaining 3 belong to the legacy
|
||||
# non-k3s stack and carry NO TXT registry entry, so no owner id claims them and ExternalDNS
|
||||
# cannot touch them regardless of what this branch does:
|
||||
#
|
||||
# *.keithjackson.dev -> 136.51.115.252 legacy stack (the zone wildcard)
|
||||
# brickbreak.keithjackson.dev -> 20.236.242.14 unrelated, off-prem
|
||||
# dev.brickbreak… -> 20.236.242.14 unrelated, off-prem
|
||||
#
|
||||
# NOTE there is NO `*.home.keithjackson.dev` wildcard in the zone any more (the comment in
|
||||
# clusters/prod/kustomization.yaml still says the zone has both wildcards; it does not, as of
|
||||
# this capture). A single-label wildcard does not match two labels, so `*.keithjackson.dev` does
|
||||
# NOT cover `<svc>.home.keithjackson.dev`: when a specific `<svc>.home` record is removed, that
|
||||
# name goes to NXDOMAIN publicly rather than falling back to the legacy IP. That is the intended
|
||||
# end state for `internal` services and it is also precisely why Pi-hole must already be
|
||||
# answering the LAN authoritatively before this branch merges.
|
||||
#
|
||||
# WHICH OF THESE THIS BRANCH DELETES — the 12 rows for the six `exposure: internal` catalog
|
||||
# services (jackett, pihole, qbittorrent, radarr, semaphore, sonarr): each one's A record and its
|
||||
# `a-`-prefixed TXT registry entry. Nothing else. Verified live on prod against the deployed
|
||||
# v0.21.0 with a throwaway hostname before any of this was authored: adding the annotation to an
|
||||
# HTTPRoute that already had a published record produced
|
||||
# `action=DELETE record=... type=A` and `action=DELETE record=a-... type=TXT` one reconcile
|
||||
# interval (~60s) later, and nothing else in the zone moved.
|
||||
#
|
||||
# TO RESTORE ONE ROW BY HAND (both halves, or ExternalDNS will not recognise the record as its
|
||||
# own): recreate the A record with its value/ttl/proxied from this file, then recreate the
|
||||
# matching `a-<name>` TXT with its content VERBATIM — the `heritage=…,external-dns/owner=prod,
|
||||
# external-dns/resource=…` string is what re-establishes ownership. An A record restored without
|
||||
# its TXT is unowned and will not be reconciled; a TXT restored with a wrong `resource=` value
|
||||
# points the registry at a source object that does not exist.
|
||||
A|*.keithjackson.dev|136.51.115.252|1|false
|
||||
A|auth.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|brickbreak.keithjackson.dev|20.236.242.14|1|false
|
||||
A|crafty-mc.home.keithjackson.dev|192.168.2.22|1|false
|
||||
A|crafty.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|dev.brickbreak.keithjackson.dev|20.236.242.14|1|false
|
||||
A|gitea.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|grafana.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|headlamp.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|jackett.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|keycloak.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|nextcloud.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|pihole.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|plex.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|qbittorrent.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|radarr.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|seerr.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|semaphore.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|sonarr.home.keithjackson.dev|192.168.2.20|1|false
|
||||
A|unifi.home.keithjackson.dev|192.168.2.21|1|false
|
||||
TXT|a-auth.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy"|1|false
|
||||
TXT|a-crafty-mc.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp"|1|false
|
||||
TXT|a-crafty.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty"|1|false
|
||||
TXT|a-gitea.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000"|1|false
|
||||
TXT|a-grafana.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana"|1|false
|
||||
TXT|a-headlamp.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp"|1|false
|
||||
TXT|a-home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage"|1|false
|
||||
TXT|a-jackett.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117"|1|false
|
||||
TXT|a-keycloak.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak"|1|false
|
||||
TXT|a-nextcloud.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80"|1|false
|
||||
TXT|a-pihole.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80"|1|false
|
||||
TXT|a-plex.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400"|1|false
|
||||
TXT|a-qbittorrent.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080"|1|false
|
||||
TXT|a-radarr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878"|1|false
|
||||
TXT|a-seerr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055"|1|false
|
||||
TXT|a-semaphore.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000"|1|false
|
||||
TXT|a-sonarr.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989"|1|false
|
||||
TXT|a-unifi.home.keithjackson.dev|"heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp"|1|false
|
||||
@@ -1,57 +0,0 @@
|
||||
A|*.keithjackson.dev|136.51.115.252
|
||||
A|auth.home.keithjackson.dev|192.168.1.53
|
||||
A|brickbreak.keithjackson.dev|20.236.242.14
|
||||
A|crafty-mc.home.keithjackson.dev|192.168.1.55
|
||||
A|crafty.home.keithjackson.dev|192.168.1.53
|
||||
A|dev.brickbreak.keithjackson.dev|20.236.242.14
|
||||
A|gitea.home.keithjackson.dev|192.168.1.53
|
||||
A|grafana.home.keithjackson.dev|192.168.1.53
|
||||
A|headlamp.home.keithjackson.dev|192.168.1.53
|
||||
A|home.keithjackson.dev|192.168.1.53
|
||||
A|jackett.home.keithjackson.dev|192.168.1.53
|
||||
A|keycloak.home.keithjackson.dev|192.168.1.53
|
||||
A|nextcloud.home.keithjackson.dev|192.168.1.53
|
||||
A|pihole.home.keithjackson.dev|192.168.1.53
|
||||
A|plex.home.keithjackson.dev|192.168.1.53
|
||||
A|qbittorrent.home.keithjackson.dev|192.168.1.53
|
||||
A|radarr.home.keithjackson.dev|192.168.1.53
|
||||
A|seerr.home.keithjackson.dev|192.168.1.53
|
||||
A|semaphore.home.keithjackson.dev|192.168.1.53
|
||||
A|sonarr.home.keithjackson.dev|192.168.1.53
|
||||
A|unifi.home.keithjackson.dev|192.168.1.54
|
||||
TXT|a-auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
|
||||
TXT|a-crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
|
||||
TXT|a-crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
|
||||
TXT|a-gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
|
||||
TXT|a-grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
|
||||
TXT|a-headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
|
||||
TXT|a-home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
|
||||
TXT|a-jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
|
||||
TXT|a-keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
|
||||
TXT|a-nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
|
||||
TXT|a-pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
|
||||
TXT|a-plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
|
||||
TXT|a-qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
|
||||
TXT|a-radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
|
||||
TXT|a-seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
|
||||
TXT|a-semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
|
||||
TXT|a-sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
|
||||
TXT|a-unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
|
||||
TXT|auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
|
||||
TXT|crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
|
||||
TXT|crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
|
||||
TXT|gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
|
||||
TXT|grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
|
||||
TXT|headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
|
||||
TXT|home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
|
||||
TXT|jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
|
||||
TXT|keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
|
||||
TXT|nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
|
||||
TXT|pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
|
||||
TXT|plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
|
||||
TXT|qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
|
||||
TXT|radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
|
||||
TXT|seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
|
||||
TXT|semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
|
||||
TXT|sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
|
||||
TXT|unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
|
||||
@@ -1,95 +0,0 @@
|
||||
A|*.keithjackson.dev|136.51.115.252
|
||||
A|auth.home.keithjackson.dev|192.168.1.53
|
||||
A|auth.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|brickbreak.keithjackson.dev|20.236.242.14
|
||||
A|conformance-dns.home.keithjackson.dev|192.168.1.56
|
||||
A|crafty-mc.home.keithjackson.dev|192.168.1.55
|
||||
A|crafty-mc.prod.home.keithjackson.dev|192.168.2.22
|
||||
A|crafty.home.keithjackson.dev|192.168.1.53
|
||||
A|crafty.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|dev.brickbreak.keithjackson.dev|20.236.242.14
|
||||
A|gitea.home.keithjackson.dev|192.168.1.53
|
||||
A|gitea.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|grafana.home.keithjackson.dev|192.168.1.53
|
||||
A|grafana.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|headlamp.home.keithjackson.dev|192.168.1.53
|
||||
A|headlamp.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|home.keithjackson.dev|192.168.1.53
|
||||
A|jackett.home.keithjackson.dev|192.168.1.53
|
||||
A|jackett.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|keycloak.home.keithjackson.dev|192.168.1.53
|
||||
A|keycloak.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|nextcloud.home.keithjackson.dev|192.168.1.53
|
||||
A|nextcloud.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|pihole.home.keithjackson.dev|192.168.1.53
|
||||
A|pihole.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|plex.home.keithjackson.dev|192.168.1.53
|
||||
A|plex.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|prod.home.keithjackson.dev|192.168.2.20
|
||||
A|qbittorrent.home.keithjackson.dev|192.168.1.53
|
||||
A|qbittorrent.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|radarr.home.keithjackson.dev|192.168.1.53
|
||||
A|radarr.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|seerr.home.keithjackson.dev|192.168.1.53
|
||||
A|seerr.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|semaphore.home.keithjackson.dev|192.168.1.53
|
||||
A|semaphore.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|sonarr.home.keithjackson.dev|192.168.1.53
|
||||
A|sonarr.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|unifi.home.keithjackson.dev|192.168.1.54
|
||||
A|unifi.prod.home.keithjackson.dev|192.168.2.21
|
||||
TXT|a-auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
|
||||
TXT|a-auth.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
|
||||
TXT|a-conformance-dns.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/chainsaw-growing-grouse/dns-probe-svc
|
||||
TXT|a-crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
|
||||
TXT|a-crafty-mc.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp
|
||||
TXT|a-crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
|
||||
TXT|a-crafty.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty
|
||||
TXT|a-gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
|
||||
TXT|a-gitea.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000
|
||||
TXT|a-grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
|
||||
TXT|a-grafana.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana
|
||||
TXT|a-headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
|
||||
TXT|a-headlamp.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp
|
||||
TXT|a-home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
|
||||
TXT|a-jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
|
||||
TXT|a-jackett.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117
|
||||
TXT|a-keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
|
||||
TXT|a-keycloak.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak
|
||||
TXT|a-nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
|
||||
TXT|a-nextcloud.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80
|
||||
TXT|a-pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
|
||||
TXT|a-pihole.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80
|
||||
TXT|a-plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
|
||||
TXT|a-plex.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400
|
||||
TXT|a-prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage
|
||||
TXT|a-qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
|
||||
TXT|a-qbittorrent.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
|
||||
TXT|a-radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
|
||||
TXT|a-radarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878
|
||||
TXT|a-seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
|
||||
TXT|a-seerr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055
|
||||
TXT|a-semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
|
||||
TXT|a-semaphore.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000
|
||||
TXT|a-sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
|
||||
TXT|a-sonarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989
|
||||
TXT|a-unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
|
||||
TXT|a-unifi.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp
|
||||
TXT|auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
|
||||
TXT|crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
|
||||
TXT|crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
|
||||
TXT|gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
|
||||
TXT|grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
|
||||
TXT|headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
|
||||
TXT|home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
|
||||
TXT|jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
|
||||
TXT|keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
|
||||
TXT|nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
|
||||
TXT|pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
|
||||
TXT|plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
|
||||
TXT|qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
|
||||
TXT|radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
|
||||
TXT|seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
|
||||
TXT|semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
|
||||
TXT|sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
|
||||
TXT|unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
|
||||
@@ -1,93 +0,0 @@
|
||||
A|*.keithjackson.dev|136.51.115.252
|
||||
A|auth.home.keithjackson.dev|192.168.1.53
|
||||
A|auth.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|brickbreak.keithjackson.dev|20.236.242.14
|
||||
A|crafty-mc.home.keithjackson.dev|192.168.1.55
|
||||
A|crafty-mc.prod.home.keithjackson.dev|192.168.2.22
|
||||
A|crafty.home.keithjackson.dev|192.168.1.53
|
||||
A|crafty.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|dev.brickbreak.keithjackson.dev|20.236.242.14
|
||||
A|gitea.home.keithjackson.dev|192.168.1.53
|
||||
A|gitea.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|grafana.home.keithjackson.dev|192.168.1.53
|
||||
A|grafana.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|headlamp.home.keithjackson.dev|192.168.1.53
|
||||
A|headlamp.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|home.keithjackson.dev|192.168.1.53
|
||||
A|jackett.home.keithjackson.dev|192.168.1.53
|
||||
A|jackett.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|keycloak.home.keithjackson.dev|192.168.1.53
|
||||
A|keycloak.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|nextcloud.home.keithjackson.dev|192.168.1.53
|
||||
A|nextcloud.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|pihole.home.keithjackson.dev|192.168.1.53
|
||||
A|pihole.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|plex.home.keithjackson.dev|192.168.1.53
|
||||
A|plex.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|prod.home.keithjackson.dev|192.168.2.20
|
||||
A|qbittorrent.home.keithjackson.dev|192.168.1.53
|
||||
A|qbittorrent.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|radarr.home.keithjackson.dev|192.168.1.53
|
||||
A|radarr.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|seerr.home.keithjackson.dev|192.168.1.53
|
||||
A|seerr.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|semaphore.home.keithjackson.dev|192.168.1.53
|
||||
A|semaphore.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|sonarr.home.keithjackson.dev|192.168.1.53
|
||||
A|sonarr.prod.home.keithjackson.dev|192.168.2.20
|
||||
A|unifi.home.keithjackson.dev|192.168.1.54
|
||||
A|unifi.prod.home.keithjackson.dev|192.168.2.21
|
||||
TXT|a-auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
|
||||
TXT|a-auth.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
|
||||
TXT|a-crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
|
||||
TXT|a-crafty-mc.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/crafty/crafty-l4-tcp
|
||||
TXT|a-crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
|
||||
TXT|a-crafty.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/crafty/crafty
|
||||
TXT|a-gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
|
||||
TXT|a-gitea.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/gitea/gitea-3000
|
||||
TXT|a-grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
|
||||
TXT|a-grafana.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/monitoring/grafana
|
||||
TXT|a-headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
|
||||
TXT|a-headlamp.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/headlamp/headlamp
|
||||
TXT|a-home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
|
||||
TXT|a-jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
|
||||
TXT|a-jackett.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/jackett/jackett-9117
|
||||
TXT|a-keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
|
||||
TXT|a-keycloak.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/keycloak/keycloak
|
||||
TXT|a-nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
|
||||
TXT|a-nextcloud.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/nextcloud/nextcloud-80
|
||||
TXT|a-pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
|
||||
TXT|a-pihole.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/pihole/pihole-80
|
||||
TXT|a-plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
|
||||
TXT|a-plex.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/plex/plex-32400
|
||||
TXT|a-prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/homepage/homepage
|
||||
TXT|a-qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
|
||||
TXT|a-qbittorrent.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
|
||||
TXT|a-radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
|
||||
TXT|a-radarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/radarr/radarr-7878
|
||||
TXT|a-seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
|
||||
TXT|a-seerr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/seerr/seerr-5055
|
||||
TXT|a-semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
|
||||
TXT|a-semaphore.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/semaphore/semaphore-3000
|
||||
TXT|a-sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
|
||||
TXT|a-sonarr.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=httproute/sonarr/sonarr-8989
|
||||
TXT|a-unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
|
||||
TXT|a-unifi.prod.home.keithjackson.dev|heritage=external-dns,external-dns/owner=prod,external-dns/resource=service/unifi/unifi-l4-tcp
|
||||
TXT|auth.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/oauth2-proxy/oauth2-proxy
|
||||
TXT|crafty-mc.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/crafty/crafty-l4-tcp
|
||||
TXT|crafty.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/crafty/crafty
|
||||
TXT|gitea.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitea/gitea-3000
|
||||
TXT|grafana.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/monitoring/grafana
|
||||
TXT|headlamp.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/headlamp/headlamp
|
||||
TXT|home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/homepage/homepage
|
||||
TXT|jackett.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/jackett/jackett-9117
|
||||
TXT|keycloak.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/keycloak/keycloak
|
||||
TXT|nextcloud.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/nextcloud/nextcloud-80
|
||||
TXT|pihole.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/pihole/pihole-80
|
||||
TXT|plex.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/plex/plex-32400
|
||||
TXT|qbittorrent.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/qbittorrent/qbittorrent-8080
|
||||
TXT|radarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/radarr/radarr-7878
|
||||
TXT|seerr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/seerr/seerr-5055
|
||||
TXT|semaphore.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/semaphore/semaphore-3000
|
||||
TXT|sonarr.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/sonarr/sonarr-8989
|
||||
TXT|unifi.home.keithjackson.dev|heritage=external-dns,external-dns/owner=default,external-dns/resource=service/unifi/unifi-l4-tcp
|
||||
@@ -1,117 +0,0 @@
|
||||
# CLAUDE.md — how work lands in this repo
|
||||
|
||||
Read this before doing anything durable. It is the delivery contract for AI coding
|
||||
sessions and agents. `DECISIONS-LOG.md`'s last entry is the current PM handoff and
|
||||
takes precedence on *what* to work on; this file governs *how* it lands.
|
||||
|
||||
## 1. Never push to `main`. Ever.
|
||||
|
||||
Every durable change reaches `main` exactly one way:
|
||||
|
||||
branch -> push the branch -> open a PR -> CI gate green -> human approval -> merge
|
||||
|
||||
There is no other path. Specifically:
|
||||
|
||||
- **Do not `git push origin main`.**
|
||||
- **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
|
||||
- **Do not merge your own PR.** A human approves and clicks merge.
|
||||
- **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
|
||||
anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
|
||||
|
||||
> **Many runbooks under `runbooks/` still say `git push origin main` IS the deploy.**
|
||||
> That wording is **stale** — it predates the PR flow. It describes what Flux does
|
||||
> (main is what the cluster follows), not what you are permitted to do. Honour this
|
||||
> file, not that line. Fixing those runbooks is tracked work.
|
||||
|
||||
## 2. Enforcement is currently WEAKER than this rule. Follow the rule anyway.
|
||||
|
||||
Verified against the live Gitea on 2026-09-29:
|
||||
|
||||
| Expected | Actual |
|
||||
|---|---|
|
||||
| `main` protected, direct push rejected | **No branch protection.** `protected: false`, `user_can_push: true` |
|
||||
| PR blocked until the gate passes | **`enable_status_check: false`**, `status_check_contexts: []` |
|
||||
| Gate runs on every PR | **Fixed 2026-09-29.** Runner `incluster-dind` is online; `gate / gate (pull_request)` runs and passes in ~1m30s–2m |
|
||||
| Approval required before merge | **`required_approvals: 0`** |
|
||||
|
||||
So nothing mechanically stops you from pushing to `main` right now. **That is a gap
|
||||
being closed, not permission.** `DECISIONS-LOG.md` (2026-08-31) claimed the gate was a
|
||||
"required status check" with a runner online; neither was true, which is how this went
|
||||
unnoticed for a month. The runner half is now genuinely fixed and observed green. The
|
||||
protection half is not — see `infra/gitea/`.
|
||||
|
||||
**Consequence for you:** CI now runs, but it does not yet *block* anything. Always run
|
||||
the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the
|
||||
strength of a check you have not seen report — and note that a **skipped** Gitea job
|
||||
reports as `success`, so a green tick is not by itself evidence that anything ran.
|
||||
|
||||
## 3. Gate before you push
|
||||
|
||||
./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'
|
||||
|
||||
Exit codes are a contract — do not flatten them:
|
||||
|
||||
| rc | Meaning |
|
||||
|---|---|
|
||||
| 0 | PASS |
|
||||
| 1 | BLOCKED — something failed |
|
||||
| 3 | UNDETERMINED — a stage did not run. **Never report this as green.** |
|
||||
|
||||
`spec-immutability` reaches rc 3 whenever the sibling spec repo is not checked out
|
||||
beside this one; that single skip is expected and tolerated. Any *other* skip is a
|
||||
real gap in your run.
|
||||
|
||||
**Never gate on the host.** `promtool` and `kustomize` are not on the host `$PATH`,
|
||||
so checks SKIP and you get a false rc 3. Only the pinned toolbox result counts.
|
||||
|
||||
### Running the gate from a git worktree
|
||||
|
||||
Two unrelated failures, both with known workarounds — put both in any agent brief:
|
||||
|
||||
1. **`dev.sh` refuses to start:** `infra/ansible/.cache/prod.kubeconfig` is untracked
|
||||
and exists only in the main checkout. Do **not** copy it (it is a credential).
|
||||
Use `DEV_KUBECONFIG=/dev/null` — the gate contacts no cluster, so L2/L3 scaffold-skip
|
||||
normally.
|
||||
2. **Every `git` call inside the container fails:** a worktree's `.git` is a pointer to a
|
||||
gitdir outside the mounted root. `dev.sh` takes no extra-mount flag, so inject one via
|
||||
the `ENGINE` hook — a shim that intercepts `run` and re-execs
|
||||
`podman run -v /path/to/main/.git:/path/to/main/.git:ro "$@"`. A traceback in
|
||||
`tracked_files()` is this bug, not a broken tree.
|
||||
|
||||
## 4. Branch naming
|
||||
|
||||
Use `feat/`, `fix/`, `docs/`, `chore/`. Note that `.gitea/workflows/gate.yml`'s **push**
|
||||
trigger lists `feature/**` — not `feat/**` — so branch pushes get no fast-feedback run.
|
||||
The `pull_request` trigger fires regardless of branch name, so the PR gate is unaffected.
|
||||
|
||||
## 5. Cluster access
|
||||
|
||||
`kubectl` does not work on the operator workstation; the toolchain lives only in the
|
||||
pinned container. Always name the context explicitly:
|
||||
|
||||
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
||||
|
||||
`dev.sh` merges both clusters and picks a current-context **per invocation**; that
|
||||
default has already changed once. Never rely on it. Do not run two `dev.sh`
|
||||
invocations concurrently.
|
||||
|
||||
**Mutating `kubectl` and `flux reconcile` are operator actions, not agent actions.**
|
||||
|
||||
## 6. Secrets
|
||||
|
||||
- **Never `sops -d` a file into a terminal, a log, or a transcript.**
|
||||
- **Never print a Secret's `.data`/`.stringData`** when checking whether it exists —
|
||||
list key names or object names only. A password has been leaked into a transcript
|
||||
this way before.
|
||||
- Encrypted files are `*.sops.yaml` or live under `secrets/`. The root `.sops.yaml` is
|
||||
the single creation-rules file; do not add a second one in a subtree.
|
||||
- A short-lived credential (e.g. a plex.tv claim token, ~minutes) cannot survive the PR
|
||||
flow. Do not design around committing one — read the expiry from the issuing page and
|
||||
treat a burnt token as free to retry.
|
||||
|
||||
## 7. Before you finish
|
||||
|
||||
Append a handoff entry to `DECISIONS-LOG.md` describing what you changed, what you
|
||||
verified, and what you did **not** verify. State gaps plainly — an honest
|
||||
"UNVERIFIED" is worth more than an optimistic "green", and this repo has already been
|
||||
burned by the difference.
|
||||
-3713
File diff suppressed because it is too large.
Load diff
@@ -38,56 +38,22 @@ ci/ the gate as scripts (lint · schema · Kyverno L0 · L1 render
|
||||
|
||||
## Status
|
||||
|
||||
Roadmap phases **0–6 are delivered**, and **Phase 7 (Production Readiness) is OPEN and in
|
||||
progress**. `clusters/prod` is no longer a design-only overlay: prod is the live cluster, serving the
|
||||
apex `home.keithjackson.dev`. The dev cluster was **decommissioned on 2026-09-12** (full migration:
|
||||
dev out, prod onto the apex — see the DECISIONS-LOG), so phases 0–6 are now exercised on prod
|
||||
hardware rather than on Proxmox VMs.
|
||||
|
||||
**Phase 7 exit criteria** — *prod runs the same config via the prod overlay; DR proven; monitoring +
|
||||
alerts live*:
|
||||
|
||||
| criterion | state |
|
||||
|---|---|
|
||||
| prod runs the prod overlay | **met** — 3× mini-PC bare metal, one Flux tree, prod overlay only |
|
||||
| monitoring + alerts live | **met 2026-09-13** — see the note below; this was not true before that date |
|
||||
| **DR proven** | **R-STATE-03 PROVEN on prod 2026-09-14** — off-cluster restore from Cloudflare R2. `R-STATE-02` (node-loss) is still dev-only |
|
||||
|
||||
**DR restore is proven on prod.** `R-STATE-03` reads `proven` in `--status` as of 2026-09-14 —
|
||||
the first requirement in this repo to reach that state. Both of its verifiers are recorded:
|
||||
`dr-restore` (L3, run `20260914T021505Z-24b8111`, sentinel `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK`)
|
||||
and `placeholder-guard-coupling` (L0). The L3 drill seeds a canary on the replicated `longhorn`
|
||||
class, backs the namespace and volume data up off-cluster, **destroys every in-cluster copy**
|
||||
(namespace, PVC and PV), restores into a clean namespace, and asserts the restored PVC binds a
|
||||
*different* PersistentVolume — so an in-cluster survival cannot pass as a restore.
|
||||
|
||||
**What remains for Phase 7:** `R-STATE-02` (`node-loss`) is still proven only on the decommissioned
|
||||
dev cluster. That drill takes a real node down on a live cluster, so it is a deliberate, scheduled
|
||||
exercise rather than something to run in passing.
|
||||
|
||||
Read `--coverage-only` and `--status` as two different questions; the earlier `PROVEN: 0` was not a
|
||||
bug but the honest answer, because proofs are claims about a specific tree on a specific cluster.
|
||||
Roadmap phases **0–6 are delivered and deployed** on the dev cluster; **Phase 7 (prod) is not
|
||||
open** — `clusters/prod` is a design-only overlay.
|
||||
|
||||
| Phase | Delivered |
|
||||
|---|---|
|
||||
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
|
||||
| 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; **now 3× mini-PC bare metal** (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13) |
|
||||
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the authoritative LAN resolver** for `${BASE_DOMAIN}` on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
|
||||
| 1 Cluster bring-up | 3-node k3s HA on **Proxmox VE** (`192.168.1.50/.51/.52`), Cilium, Flux, SOPS |
|
||||
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
|
||||
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
|
||||
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled |
|
||||
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero still blocked on OQ-BKP**; image-automation authored but disabled |
|
||||
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |
|
||||
| 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · **kubectl OIDC SSO with group→RBAC** · in-cluster Gitea Actions merge gate |
|
||||
| **7 Production readiness** | **IN PROGRESS** (DR restore proven 2026-09-14; node-loss outstanding). Prod live on bare metal serving the apex; dev decommissioned; CNPG HA 3 instances with primaries spread one per node; NFS bulk media + legacy Nextcloud mounted; VPN egress on prod. Alerting made real 2026-09-13 (Longhorn, cert-manager, Flux and twelve app SLIs had all been unable to fire). **Open: `node-loss` (R-STATE-02) re-proven on prod.** |
|
||||
|
||||
**Conformance: 51 / 51 requirements covered (100%)**, 0 gaps, 0 safety-critical gaps (closed
|
||||
2026-09-13; `R-C2-01`/`R-C2-02` were the last two). Regenerate with
|
||||
`./conformance/run.sh --coverage-only`; the full report is
|
||||
`conformance/_artifacts/coverage/coverage.md`.
|
||||
|
||||
Coverage is **not** proof. A requirement is *covered* when an executable test exists for it, and
|
||||
*PROVEN* only when a recorded run of that test is in `conformance/runs/ledger.jsonl` at the current
|
||||
SHA on the cluster it claims. Read both numbers: `--coverage-only` for the first, `--status` for the
|
||||
second.
|
||||
**Conformance: 36 / 51 requirements covered (70.6%)**, 15 gaps, 1 safety-critical gap
|
||||
(`R-STATE-02`, Longhorn node-loss). Regenerate with `./conformance/run.sh --coverage-only`;
|
||||
the full report is `conformance/_artifacts/coverage/coverage.md`.
|
||||
|
||||
The running narrative — decisions, root causes, incidents, follow-ups — is
|
||||
[`DECISIONS-LOG.md`](DECISIONS-LOG.md). It, not this table, is the authoritative record.
|
||||
|
||||
@@ -41,41 +41,10 @@ spec:
|
||||
# disk KubeVirt cannot migrate and eviction would hard-stop the guest.
|
||||
evictionStrategy: LiveMigrate
|
||||
domain:
|
||||
# =====================================================================================
|
||||
# FIRMWARE UUID == THE CLOUD-INIT INSTANCE-ID. THIS IS THE RE-RUN LEVER. READ BEFORE EDIT.
|
||||
# =====================================================================================
|
||||
# MEASURED on prod 2026-09-14, not assumed:
|
||||
# * the NoCloud seed KubeVirt builds for this VM carries
|
||||
# {"instance-id":"de5c3338-6d32-4956-89d1-12c8272b2c07","local-hostname":"unifi"}
|
||||
# (read out of noCloud.iso inside virt-launcher-unifi-gp594), and that value was
|
||||
# EXACTLY `spec.template.spec.domain.firmware.uuid`, which KubeVirt's mutating webhook
|
||||
# had stamped onto the VM OBJECT — so it is PERSISTENT, not per-VMI (the VMI's own uid
|
||||
# was b532fd7c-…, a different value). It is not a hash of the VM name either: the four
|
||||
# standard uuid5 namespaces over "unifi" / "unifi.unifi" all produce something else.
|
||||
# * cloud-init's NoCloud datasource decides "is this the same instance?" by comparing the
|
||||
# cached instance-id against the guest's DMI product_uuid — which IS this firmware UUID.
|
||||
#
|
||||
# CONSEQUENCE, and it is the whole reason the previous unattended attempt could never have
|
||||
# worked: EDITING `userData` BELOW AND RESTARTING THE VM CHANGES NOTHING. cloud-init sees an
|
||||
# unchanged instance-id, restores its CACHED datasource, never re-reads the seed, and skips
|
||||
# every per-instance module (write_files, runcmd). The new payload is silently ignored.
|
||||
#
|
||||
# So the UUID is pinned HERE, in git, and BUMPING IT IS THE SUPPORTED WAY to make the guest
|
||||
# process a new cloud-init payload: change the trailing hex, commit, restart the VM. Pinning
|
||||
# it also makes a rebuild reproducible — a fresh apply would otherwise take a random
|
||||
# webhook-generated UUID and give the guest a different identity every time.
|
||||
#
|
||||
# The value below is DELIBERATELY NOT the de5c3338-…c07 the live guest booted with: that
|
||||
# difference is what makes the next boot a NEW cloud-init instance and installs the
|
||||
# `install-uos` helper + its systemd unit. `serial` is pinned to the value the webhook
|
||||
# already stamped, so this change does not also churn the guest's DMI serial.
|
||||
firmware:
|
||||
uuid: "de5c3338-6d32-4956-89d1-12c8272b2c11"
|
||||
serial: "24bb12ac-d311-45f6-a75b-90a9e1b63e18"
|
||||
# FLAG (verify-at-bringup): Ubuntu 24.04 amd64 cloud images are hybrid BIOS+UEFI bootable,
|
||||
# so KubeVirt's default SeaBIOS SHOULD boot cleanly (unlike the Fedora containerdisk, which
|
||||
# is UEFI-only and needed an explicit efi block). RESOLVED at bring-up: boots as shipped;
|
||||
# the guest reports Ubuntu 24.04.4 LTS.
|
||||
# is UEFI-only and needed an explicit efi block). If the guest drops to emergency/no-boot,
|
||||
# add: firmware: { bootloader: { efi: { secureBoot: false } } }
|
||||
cpu:
|
||||
cores: 2
|
||||
memory:
|
||||
@@ -89,28 +58,9 @@ spec:
|
||||
- name: cloudinit
|
||||
disk:
|
||||
bus: virtio
|
||||
# The operator's installer URL (+ optional SSH key), delivered as a tiny read-only
|
||||
# iso9660 disk — see the `uosdata` volume near the bottom of this file.
|
||||
- name: uosdata
|
||||
disk:
|
||||
bus: virtio
|
||||
interfaces:
|
||||
# MAC PINNED 2026-09-18, and this is a root-cause fix, not cosmetics.
|
||||
# KubeVirt generates a fresh MAC for every VMI when none is pinned. cloud-init writes
|
||||
# /etc/netplan/50-cloud-init.yaml with a `match` on the MAC it saw, and it will NOT rewrite
|
||||
# that file while the NoCloud instance-id is unchanged — and the instance-id here is
|
||||
# domain.firmware.uuid, which is stable across restarts by design. So each VMI recreation
|
||||
# handed the guest a NEW MAC that the retained netplan did not match, leaving enp1s0
|
||||
# administratively DOWN with no address and no DHCP attempt.
|
||||
# That is exactly what the evidence shows for the boot of 2026-09-12: cloud-init's own
|
||||
# device table reports `enp1s0 | Up: False | Address: .`, no apt failures are logged
|
||||
# (because the per-instance modules were skipped), yet UOS is installed and
|
||||
# qemu-guest-agent runs — both of which required a working network on an EARLIER VMI.
|
||||
# Pinning the MAC to the value the current VMI already holds makes the guest's netplan
|
||||
# match permanently, so this cannot recur on the next recreation.
|
||||
- name: default
|
||||
masquerade: {}
|
||||
macAddress: a6:d7:c0:b3:72:fa
|
||||
# Declaring ports on a masquerade iface (a) restricts pod->VM forwarding to exactly
|
||||
# these and (b) surfaces them as containerPorts on the virt-launcher pod so the
|
||||
# LoadBalancer Services + CiliumNetworkPolicy ingress can target/match them.
|
||||
@@ -133,112 +83,32 @@ spec:
|
||||
- name: rootdisk
|
||||
dataVolume:
|
||||
name: unifi-rootdisk
|
||||
# =====================================================================================
|
||||
# THE OPERATOR'S ONE FIELD — SOPS Secret `unifi-uos-installer`, delivered as a DISK.
|
||||
# =====================================================================================
|
||||
# KubeVirt exposes a Secret as a small read-only iso9660 disk whose volume label we choose;
|
||||
# each Secret key becomes a file on it. The guest mounts it by label (LABEL=UOSDATA) and
|
||||
# reads `installer-url` plus an optional `console-password`. The Secret ships ENCRYPTED with
|
||||
# REPLACE-ME placeholders, and the guest-side script treats anything that is not an https
|
||||
# URL as "not provided yet" and exits 0 — so the VM boots and stays healthy until the
|
||||
# operator fills it in. See unifi-uos-installer.sops.yaml.example next to this file.
|
||||
#
|
||||
# `console-password` exists because THERE IS CURRENTLY NO WAY INTO THIS GUEST. The stock
|
||||
# Ubuntu cloud image ships a locked `ubuntu` account, this cloud-init never installed a key,
|
||||
# and SSH could not help anyway: the masquerade interface above forwards ONLY the declared
|
||||
# ports, and 22 is not one of them — probed from inside virt-launcher on 2026-09-14, the
|
||||
# guest answers RST on 22 exactly as it does on the UniFi ports. The serial console is the
|
||||
# only door, and a console needs a password. Set it and `virtctl -n unifi console unifi`
|
||||
# works; `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` already gives
|
||||
# read-only console output with no virtctl at all.
|
||||
#
|
||||
# WHY A SEPARATE DISK AND NOT `cloudInitNoCloud.secretRef` (which would also lift the
|
||||
# 2048-byte inline cap): secretRef moves the ENTIRE cloud-config into ciphertext. Two
|
||||
# concrete costs. (a) The shell below stops being reviewable in a diff and falls outside the
|
||||
# L0 dollar-collapse guard, which can only read plaintext. (b) Flux runs its postBuild
|
||||
# envsubst over the DECRYPTED resources, so the escape contract in
|
||||
# clusters/base/apps-vms.yaml would still govern text nobody can see in git — the exact
|
||||
# shape of the 2026-09-13 nextcloud-db incident. Keeping the script in plaintext and putting
|
||||
# ONLY the operator's two values behind SOPS keeps both properties.
|
||||
#
|
||||
# A CHANGED SECRET REACHES THE GUEST ON THE NEXT VMI START with NO firmware-UUID bump: the
|
||||
# disk is rebuilt from the Secret every time the VMI starts, and the systemd unit below
|
||||
# re-reads it on EVERY boot instead of relying on cloud-init (which will not re-run — see
|
||||
# the firmware block above). The UUID bump is a ONE-TIME step, to get that unit installed.
|
||||
- name: uosdata
|
||||
secret:
|
||||
secretName: unifi-uos-installer
|
||||
volumeLabel: UOSDATA
|
||||
- name: cloudinit
|
||||
cloudInitNoCloud:
|
||||
# INLINE, and it MUST stay <= 2048 bytes. That ceiling was MEASURED against the
|
||||
# DEPLOYED KubeVirt v1.9.0 rather than quoted: a server-side dry-run apply of an
|
||||
# otherwise-identical VM accepted 2048 bytes of userData and rejected 2049 with
|
||||
# "cloudInitNoCloud userdata exceeds 2048 byte limit. Should use UserDataSecretRef for
|
||||
# larger data." The payload below is ~1.5 KB; keep additions small.
|
||||
#
|
||||
# WHAT IT DOES: installs qemu-guest-agent (the KubeVirt AgentConnected signal, which is
|
||||
# also the proof cloud-init ran) and preps Podman for UOS, then writes `install-uos`
|
||||
# plus a systemd unit that runs it on EVERY boot. The unit — not cloud-init — is what
|
||||
# makes a later URL change take effect, because cloud-init will not re-run per-instance
|
||||
# modules while the instance-id is unchanged.
|
||||
#
|
||||
# `install-uos` IS IDEMPOTENT AND WILL NOT REINSTALL OVER A WORKING CONTROLLER. It bails
|
||||
# out the moment systemd knows a `uosserver.service` — which the live guest ALREADY has:
|
||||
# the serial console log for the current boot shows both `uosserver.service` and
|
||||
# `uosserver-updater.service` starting, and the root filesystem is 6.3 GB of 40 GB used.
|
||||
# UOS is installed here; see the README for what is actually wrong (the guest lost its
|
||||
# IP). A stamp file at /var/lib/uos-installed is the second, weaker guard for the case
|
||||
# where the unit name changes in some future UOS release.
|
||||
#
|
||||
# ENVSUBST: apps-vms declares postBuild.substituteFrom (for ${UNIFI_L4_VIP} in
|
||||
# 50-service-l4.yaml), so Flux's envsubst DOES run over this text. ${BASE_DOMAIN} below
|
||||
# is a deliberate substitution. Everything else is bare `$name` or `$(cmd)`, both of
|
||||
# which envsubst leaves untouched — verified against the live VMI, whose rendered
|
||||
# userData still carries its shell variables verbatim. There is deliberately NO
|
||||
# brace-form shell expansion left in this payload, so nothing here needs escaping; if
|
||||
# you add one, escape it per the contract in clusters/base/apps-vms.yaml, and NEVER
|
||||
# write an indexed expansion.
|
||||
# INLINE (kept < KubeVirt's 2048-byte userData limit — so NO SSH key here; add one back via
|
||||
# userDataSecretRef only if needed). qemu-guest-agent gives the KubeVirt AgentConnected
|
||||
# signal (proves cloud-init ran); install-uos resolves the latest UOS linux-x64 build from
|
||||
# Ubiquiti's PUBLIC firmware API (no auth) and installs it non-interactively (UI on :11443).
|
||||
# NOTE: apps-vms has NO postBuild.substituteFrom, so shell ${...} below is NOT envsubst'd.
|
||||
userData: |-
|
||||
#cloud-config
|
||||
hostname: unifi
|
||||
fqdn: unifi.${BASE_DOMAIN}
|
||||
fqdn: unifi.home.keithjackson.dev
|
||||
package_update: true
|
||||
package_upgrade: false
|
||||
packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl]
|
||||
packages: [qemu-guest-agent, podman, uidmap, slirp4netns, ca-certificates, curl, jq, cloud-guest-utils]
|
||||
write_files:
|
||||
- path: /usr/local/sbin/install-uos
|
||||
permissions: '0755'
|
||||
content: |
|
||||
#!/bin/bash
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
d=/run/uos; mkdir -p $d
|
||||
mountpoint -q $d || mount -o ro LABEL=UOSDATA $d
|
||||
p=$(tr -d '\r\n' < $d/console-password || true)
|
||||
case "$p" in ""|REPLACE-ME) ;; *) echo "ubuntu:$p" | chpasswd ;; esac
|
||||
systemctl cat uosserver.service >/dev/null 2>&1 && exit 0
|
||||
s=/var/lib/uos-installed
|
||||
[ -e $s ] && exit 0
|
||||
u=$(tr -d '\r\n' < $d/installer-url || true)
|
||||
case "$u" in https://*) ;; *) echo "UOS installer-url not set yet"; exit 0;; esac
|
||||
t=$(mktemp)
|
||||
curl -fL --retry 5 --retry-delay 10 -o $t "$u"
|
||||
chmod +x $t
|
||||
yes y | $t install || yes y | $t
|
||||
rm -f $t; date -Is > $s
|
||||
- path: /etc/systemd/system/uos-install.service
|
||||
content: |
|
||||
[Unit]
|
||||
Description=Install UniFi OS Server
|
||||
Wants=network-online.target
|
||||
After=network-online.target
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/sbin/install-uos
|
||||
StandardOutput=journal+console
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
if [ -z "${UOS_URL:-}" ]; then
|
||||
UOS_URL="$(curl -fsSL https://fw-update.ui.com/api/firmware-latest | jq -r '._embedded.firmware|map(select(.product=="unifi-os-server" and .platform=="linux-x64"))|sort_by(.version_major,.version_minor,.version_patch)|last._links.data.href')"
|
||||
fi
|
||||
[ -n "${UOS_URL}" ] && [ "${UOS_URL}" != "null" ] || { echo "no UOS URL"; exit 1; }
|
||||
t="$(mktemp)"; echo "downloading ${UOS_URL}"; curl -fL --retry 5 --retry-delay 10 -o "$t" "$UOS_URL"; chmod +x "$t"; echo "installing"; yes | "$t"; echo "done"
|
||||
runcmd:
|
||||
- [systemctl, enable, --now, qemu-guest-agent]
|
||||
- [systemctl, enable, --now, podman.socket]
|
||||
- [systemctl, enable, --now, uos-install.service]
|
||||
- [ systemctl, enable, --now, qemu-guest-agent ]
|
||||
- [ systemctl, enable, --now, podman.socket ]
|
||||
- [ bash, -lc, "/usr/local/sbin/install-uos 2>&1 | tee /var/log/uos-install.log > /dev/console" ]
|
||||
@@ -8,9 +8,8 @@
|
||||
# CANNOT express: raw TCP on non-HTTP ports (8080 inform, 6789), UDP at all (3478 STUN, 10001
|
||||
# discovery), or the device-facing protocols that never speak HTTP. UniFi device ADOPTION rides
|
||||
# exactly those L4 ports. So the controller is exposed as a type=LoadBalancer Service whose VIP
|
||||
# is pulled from the CiliumLoadBalancerIPPool `default-pool` (dev 192.168.1.53-.60 — the
|
||||
# CILIUM_LB_POOL_START/STOP cluster-vars knobs) and ARP-announced on the flat LAN by the existing
|
||||
# CiliumL2AnnouncementPolicy `default-l2` — the SAME L2/LB-IPAM
|
||||
# is pulled from the CiliumLoadBalancerIPPool `default-pool` (192.168.1.53-.60) and ARP-announced
|
||||
# on the flat LAN by the existing CiliumL2AnnouncementPolicy `default-l2` — the SAME L2/LB-IPAM
|
||||
# machinery the Gateway ingress VIP already uses (platform-config/cilium/*). No new Cilium config
|
||||
# is required; these Services just consume the pool.
|
||||
#
|
||||
@@ -21,67 +20,11 @@
|
||||
# k3s v1.31 — but the shared-key split is the lower-risk, Cilium-native pattern.)
|
||||
#
|
||||
# selector kubevirt.io/vm=unifi -> the virt-launcher pod; masquerade forwards the declared ports
|
||||
# into the guest.
|
||||
# into the guest. externalTrafficPolicy: Local preserves the LAN client source IP and keeps the
|
||||
# VIP pinned to the node running the VM (also what L2 announcement does).
|
||||
#
|
||||
# -------------------------------------------------------------------------------------------
|
||||
# externalTrafficPolicy: Cluster — CORRECTED 2026-09-14 after an 8-DAY LAN OUTAGE.
|
||||
# -------------------------------------------------------------------------------------------
|
||||
# This file used to say `Local`, justified as "preserves the LAN client source IP and keeps the
|
||||
# VIP pinned to the node running the VM (also what L2 announcement does)". BOTH HALVES OF THAT
|
||||
# WERE FALSE ON THIS CLUSTER, and the second one took the UniFi controller off the LAN entirely.
|
||||
#
|
||||
# 1) L2 ANNOUNCEMENT DOES NOT FOLLOW THE BACKEND. Cilium's L2 announcer elects one lease holder
|
||||
# per Service among the nodes matching the CiliumL2AnnouncementPolicy `default-l2` — which
|
||||
# matches ALL linux nodes — and it does NOT filter that set by which nodes hold a backend.
|
||||
# MEASURED on prod 2026-09-14: the unifi VM ran on k3s-prod-1 while BOTH
|
||||
# `cilium-l2announce-unifi-unifi-l4-tcp` and `-udp` were held by k3s-prod-2. With `Local`, an
|
||||
# announcing node that holds no backend has nowhere to send the packet and answers ICMP port
|
||||
# unreachable. 192.168.2.21:8443, :8080 and :11443 were dead from the LAN for up to 8 days
|
||||
# (the leases were 8 days old); the operator's controller UI and device inform path were down.
|
||||
# It does NOT self-heal: the lease holder keeps RENEWING after the backend moves, so only
|
||||
# deleting the Lease forces a re-election — and that re-election is random, so it can pick a
|
||||
# backend-less node again. See apps/wave-0/pihole-dns.yaml, which hit this first and is the
|
||||
# reference implementation this file now follows.
|
||||
#
|
||||
# 2) `Local` WAS NEVER PRESERVING THE CLIENT IP HERE ANYWAY — this is a KubeVirt VM. The
|
||||
# interface in 30-virtualmachine.yaml is `masquerade: {}`, which is KubeVirt's NAT binding: it
|
||||
# SNATs everything from the virt-launcher pod into the guest over a private link, so the UniFi
|
||||
# controller has always seen the pod-side masquerade gateway as the source, never the LAN
|
||||
# client. There is a SECOND translation below the Service, and it already destroyed the very
|
||||
# thing `Local` was being kept for. So the client-IP cost of this change is ZERO for UniFi:
|
||||
# it is giving up an address the guest could not see in the first place.
|
||||
#
|
||||
# Nothing UniFi depends on breaks: device identity in the adoption/inform flow is the device
|
||||
# MAC (and the device reports its own IP inside the inform payload), not the packet source,
|
||||
# and guest-portal authorization runs on the gateway appliance, not on this controller.
|
||||
#
|
||||
# `Cluster` makes EVERY node able to serve the VIP by forwarding to wherever the VM runs, which
|
||||
# is what makes this survive a reschedule and a rebuild.
|
||||
#
|
||||
# NO CHANGE IS NEEDED IN 40-networkpolicy.yaml, AND THAT WAS VERIFIED RATHER THAN ASSUMED.
|
||||
# Under `Cluster` the forwarded second hop is sourced from the ingress node and carries Cilium's
|
||||
# reserved `remote-node` identity. pihole-dns.yaml has to name `remote-node` explicitly in its
|
||||
# CiliumNetworkPolicy — but that is because its policy deliberately omits `cluster`.
|
||||
# unifi-allow-l4-ingress admits `cluster`, and Cilium's `cluster` entity SUBSUMES `remote-node`
|
||||
# (along with host/init/unmanaged). Confirmed on prod 2026-09-14 by dumping the endpoint's
|
||||
# datapath policy map (`cilium-dbg bpf policy get`) while the CNP was in its unmodified state:
|
||||
# `reserved:remote-node` was already an Allow on the L4 ports. Adding `remote-node` to that rule
|
||||
# would be a no-op, so it is deliberately NOT added here — do not "fix" it.
|
||||
#
|
||||
# NO loadBalancerSourceRanges HERE, DELIBERATELY — and this is where this file departs from
|
||||
# pihole-dns.yaml. PiHole added `loadBalancerSourceRanges: [${LAN_CIDR}]` because it was giving
|
||||
# up a restriction it actually had and genuinely needed to be LAN-only. These Services never had
|
||||
# a source restriction at all (unifi-allow-l4-ingress admits `world`), so `Cluster` removes no
|
||||
# protection and there is nothing to compensate for. Adding a LAN-only filter during an outage
|
||||
# fix would be a NEW restriction that could silently break a remote site's device informing over
|
||||
# WAN — trading this outage for a subtler one. If UniFi should be LAN-only, that is its own
|
||||
# change, made deliberately and tested against the device fleet.
|
||||
#
|
||||
# VIP: ${UNIFI_L4_VIP} (cluster-vars, R-ENV-02; dev 192.168.1.54, confirmed free, inside
|
||||
# default-pool and reserved in tofu — the Gateway already holds the pool's first IP, dev .53).
|
||||
# Substituted by the `apps-vms` Flux Kustomization, which carries postBuild.substituteFrom for
|
||||
# exactly this token; every shell $${...} in 30-virtualmachine.yaml's cloud-init is escaped as $${...}
|
||||
# accordingly. A new environment MUST set UNIFI_L4_VIP to a free address in ITS pool.
|
||||
# FLAG: confirm 192.168.1.54 is free, inside default-pool, and reserved in tofu (the Gateway
|
||||
# already holds one pool IP, typically .53). Adjust if it collides.
|
||||
#
|
||||
# REUSABLE PATTERN (the deferred `minecraft` :25565/L4 problem is the same shape): expose an
|
||||
# L4-only workload as a Cilium LoadBalancer on a pinned, L2-announced VIP with a per-app sharing
|
||||
@@ -96,13 +39,13 @@ metadata:
|
||||
app.kubernetes.io/name: unifi
|
||||
annotations:
|
||||
lbipam.cilium.io/sharing-key: "unifi"
|
||||
lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
|
||||
# external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP.
|
||||
lbipam.cilium.io/ips: "192.168.1.54"
|
||||
# external-dns (source: service) publishes A unifi.home.keithjackson.dev -> 192.168.1.54.
|
||||
# This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443.
|
||||
external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
|
||||
external-dns.alpha.kubernetes.io/hostname: "unifi.home.keithjackson.dev"
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Cluster
|
||||
externalTrafficPolicy: Local
|
||||
selector:
|
||||
kubevirt.io/vm: unifi
|
||||
ports:
|
||||
@@ -122,10 +65,10 @@ metadata:
|
||||
app.kubernetes.io/name: unifi
|
||||
annotations:
|
||||
lbipam.cilium.io/sharing-key: "unifi" # same key + same IP => shares the VIP with -tcp
|
||||
lbipam.cilium.io/ips: "${UNIFI_L4_VIP}"
|
||||
lbipam.cilium.io/ips: "192.168.1.54"
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Cluster
|
||||
externalTrafficPolicy: Local
|
||||
selector:
|
||||
kubevirt.io/vm: unifi
|
||||
ports:
|
||||
|
||||
+74
-202
@@ -1,48 +1,18 @@
|
||||
# UniFi Network controller (UniFi OS Server) — VM onboarding
|
||||
|
||||
> **LIVE — but the controller is unreachable, and NOT for the reason this file used to claim.**
|
||||
> Reconciled by the **non-gating `apps-vms`** Flux Kustomization (`clusters/base/apps-vms.yaml`,
|
||||
> `dependsOn: platform-config` + `platform-config-cdi`, `wait: false`). VMs are a separate lifecycle
|
||||
> from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import or install
|
||||
> cannot stall the app tier.
|
||||
> **LIVE — partially.** Reconciled by the **non-gating `apps-vms`** Flux Kustomization
|
||||
> (`clusters/base/apps-vms.yaml`, `dependsOn: platform-config`, `wait: false`). VMs are a separate
|
||||
> lifecycle from the rendered app waves, and UniFi has no in-cluster dependents, so a slow import
|
||||
> or install cannot stall the app tier.
|
||||
>
|
||||
> **Status (re-verified against live prod 2026-09-14):**
|
||||
> - `longhorn-vm` StorageClass — **live**; `unifi-rootdisk` PVC Bound, **40Gi, RWX, Block**.
|
||||
> - CDI import — **Succeeded** (100%).
|
||||
> - VM — **Running** on **`k3s-prod-1`**, guest agent connected, Ubuntu 24.04.4 LTS,
|
||||
> `LiveMigratable=True` / `StorageLiveMigratable=True`. VMI age 34h, VM 8d.
|
||||
> - L4 LoadBalancer VIP **`192.168.2.21`** — allocated and held on both Services.
|
||||
> - **UniFi OS Server IS INSTALLED.** This reverses the previous entry here. See below.
|
||||
> - **The guest has no IP address.** That — not a missing install — is why `:11443`, `:8443` and
|
||||
> `:8080` refuse connections.
|
||||
|
||||
## What is actually wrong (2026-09-14)
|
||||
|
||||
The earlier note in this file said "UOS is NOT installed; the guest is a bare, prepped Ubuntu".
|
||||
That was wrong, and acting on it would have meant reinstalling over a working controller.
|
||||
|
||||
Evidence, all read-only, from the live cluster:
|
||||
|
||||
| Claim | How it was checked | Result |
|
||||
|---|---|---|
|
||||
| UOS is installed | `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` | the current boot starts **`uosserver.service` — UniFi OS Server Service** and **`uosserver-updater.service`** |
|
||||
| …and it is a real install, not a stub | KubeVirt guest-agent `filesystemlist` subresource | `/` is **6.3 GB used of 40 GB** — a bare Ubuntu cloud image with our packages is ~2 GB |
|
||||
| the guest has no IP | `kubectl -n unifi get vmi unifi -o json` → `.status.interfaces` | `enp1s0`, `linkState: up`, **no `ipAddress` field at all**; the `kubectl get vmi` IP column is empty |
|
||||
| …confirmed at the link layer | `kubectl -n unifi exec virt-launcher-… -c compute -- ip neigh` | **`10.0.2.2 dev k6t-eth0 FAILED`** — the guest does not answer ARP for its masquerade address |
|
||||
| the ports are dead end-to-end | `/dev/tcp` probes from inside virt-launcher, to the pod IP **and** to `10.0.2.2` directly | 11443 / 8443 / 8080 / **22** all refused — i.e. the VIP and the CNP are exonerated |
|
||||
|
||||
So the controller is installed and its units start; nothing downstream of the guest's own network
|
||||
stack can work. The interface was fine at boot (the console log reaches `network-online.target` at
|
||||
~10s and cloud-init completes at 18.5s), so the address was lost afterwards — the obvious suspect is
|
||||
UOS taking over host networking, but **that has not been proven and cannot be, from outside the
|
||||
guest.** Diagnosing it needs a shell, which is the second problem below.
|
||||
|
||||
**There is currently no way into this guest.** The shipped cloud-init installs no SSH key and sets
|
||||
no password, so the stock `ubuntu` account is locked and a serial console reaches a login prompt
|
||||
nobody can answer. SSH would not rescue it either: the masquerade interface forwards **only** the
|
||||
ports declared in `30-virtualmachine.yaml`, and 22 is not one of them — the probe above shows the
|
||||
guest RSTs on 22 exactly as it does on 11443. Closing that hole is what `console-password` in the
|
||||
new Secret is for.
|
||||
> **Status (verified 2026-08-26):**
|
||||
> - `longhorn-vm` StorageClass — **live** (RWX / Block / migratable).
|
||||
> - CDI import — **Succeeded** (100%); `unifi-rootdisk` PVC Bound, 40Gi, RWX, `longhorn-vm`.
|
||||
> - VM — **Running**, guest agent connected, Ubuntu 24.04.4 LTS, live-migratable, on `k3s-dev-1`.
|
||||
> - L4 LoadBalancer VIP `192.168.1.54` — **allocated** on both Services.
|
||||
> - **UniFi OS Server itself is NOT installed.** Port `11443` does not answer from in-cluster
|
||||
> (the CNP explicitly permits `cluster` → `:11443`, so this is "nothing listening", not policy).
|
||||
> The guest is a bare, prepped Ubuntu. **This is the one remaining step** — see Open question 1.
|
||||
|
||||
## What this is
|
||||
|
||||
@@ -59,39 +29,24 @@ datastore** inside a Podman container stack — so **no separate MongoDB compani
|
||||
| `namespace.yaml` | `unifi` namespace: app-namespace label (Kyverno + default-deny apply), `gateway-access=shared`, backup annotation |
|
||||
| `10-default-deny.yaml` | R-NET-01 baseline: deny all, allow only DNS |
|
||||
| `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm` **RWX/Block** disk (**40Gi**). First real CDI http import on the cluster |
|
||||
| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, a **pinned firmware UUID** (= the cloud-init instance-id), the **UOSDATA secret disk**, and a minimal **inline** cloud-init that installs `install-uos` + a per-boot systemd unit |
|
||||
| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, minimal **inline** cloud-init (SSH key, qemu-guest-agent, Podman prep) |
|
||||
| `40-networkpolicy.yaml` | Additive CNP allows: L4 ingress ports, controller world+DNS egress, and CDI-importer egress |
|
||||
| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `${UNIFI_L4_VIP}` (prod **192.168.2.21**); external-dns A record |
|
||||
| `unifi-uos-installer.sops.yaml` | **SOPS-encrypted.** The operator's UOS installer URL + guest console password, mounted into the guest as a disk |
|
||||
| `unifi-uos-installer.sops.yaml.example` | Plaintext template + the operator hand-off steps (never applied, never encrypted) |
|
||||
| `50-service-l4.yaml` | Two `type=LoadBalancer` Services (TCP+UDP) sharing **one pinned Cilium VIP** `192.168.1.54`; external-dns A record |
|
||||
| `kustomization.yaml` | The reconciled resource list (referenced by `apps/vms/kustomization.yaml`) |
|
||||
|
||||
The optional L7 HTTPRoute was **dropped, and the file does not exist**: UOS serves HTTPS on
|
||||
`:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on this Cilium
|
||||
version. Browser access is via the L4 VIP. Re-add it if/when that lands.
|
||||
The optional L7 HTTPRoute (`90-httproute-optional.yaml`) was **dropped**, not merely disabled: UOS
|
||||
serves HTTPS on `:11443`, so a Gateway route needs a `BackendTLSPolicy` that is not confirmed on
|
||||
this Cilium version. Browser access is via the L4 VIP. Re-add it if/when that lands.
|
||||
|
||||
### Cloud-init delivery — three constraints, all measured on the deployed KubeVirt (v1.9.0)
|
||||
### Cloud-init delivery — two constraints learned the hard way
|
||||
|
||||
1. **Inline `userData` is capped at 2048 bytes.** MEASURED, not quoted: a server-side dry-run apply
|
||||
of an otherwise-identical VM accepted 2048 bytes and rejected 2049 with *"cloudInitNoCloud
|
||||
userdata exceeds 2048 byte limit. Should use UserDataSecretRef for larger data."* The shipped
|
||||
payload is ~1.45 KB. The API field for the escape hatch is spelled **`secretRef`** (the Go name
|
||||
is `UserDataSecretRef`); `userDataSecretRef` is not a YAML key and trying it will silently
|
||||
validate as an unknown field.
|
||||
2. **The firmware UUID is the cloud-init instance-id, and editing `userData` alone changes
|
||||
nothing.** The NoCloud seed's `instance-id` is exactly
|
||||
`spec.template.spec.domain.firmware.uuid`, which KubeVirt's webhook stamps onto the **VM object**
|
||||
(so it survives restarts), and cloud-init's NoCloud datasource compares it to the guest's DMI
|
||||
product UUID. Same id → cached datasource restored, new payload never read, `write_files` and
|
||||
`runcmd` skipped. This is why the previously shipped "unattended install" could never have taken
|
||||
effect on a second boot. The UUID is now **pinned in git**, with bumping it as the documented way
|
||||
to force a re-run. See the long comment in `30-virtualmachine.yaml`.
|
||||
3. **`apps-vms` DOES use `postBuild.substituteFrom`.** This file used to say it must not — that has
|
||||
been false since 2026-08-31, when `${UNIFI_L4_VIP}` landed in `50-service-l4.yaml`. Flux's
|
||||
envsubst therefore runs over the cloud-init payload, and the escape contract in
|
||||
`clusters/base/apps-vms.yaml` is in force. Bare `$name` and `$(cmd)` are left alone (verified
|
||||
against the live VMI's rendered userData); brace-form shell expansion must be escaped; an
|
||||
indexed expansion is a hard error that fails the whole Kustomization.
|
||||
1. **KubeVirt caps inline `userData` at 2048 bytes.** The original rich cloud-init (full Podman
|
||||
prep + `install-uos` helper) blew past it. A `userDataSecretRef` was tried, then reverted; the
|
||||
shipped config is a **minimal ~1.1 KB inline `userData`**. Keep additions small or move to a
|
||||
Secret deliberately.
|
||||
2. **`apps-vms` must NOT use `postBuild.substituteFrom`.** Flux variable substitution rewrote
|
||||
`$`-bearing text inside the cloud-init payload and **corrupted it**. `clusters/base/apps-vms.yaml`
|
||||
therefore carries no `substituteFrom` — unlike every app wave. Do not "restore" it for consistency.
|
||||
|
||||
### Node prerequisites this VM forced onto the cluster
|
||||
|
||||
@@ -101,31 +56,15 @@ version. Browser access is via the L4 VIP. Re-add it if/when that lands.
|
||||
- **`longhorn-scratch`** — a single-replica StorageClass for CDI scratch space; the default
|
||||
3-replica class made the import fail with insufficient storage.
|
||||
|
||||
## UOS install method (researched 2026-09-14)
|
||||
## UOS install method (researched)
|
||||
|
||||
- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`.
|
||||
Vendor/community minimum is 2 vCPU / 4 GB RAM / ~25 GB disk; we give **2 vCPU / 4 GiB / 40Gi**
|
||||
(the 40Gi figure is the DataVolume and the Bound PVC — an earlier "60 GB" claim here was wrong).
|
||||
Podman must be at `/usr/bin/podman`.
|
||||
- **Install:** obtain the download link from `ui.com/download/software/unifi-os-server` while signed
|
||||
in to a (free) Ubiquiti account, `chmod +x`, run **as root**. UI then on **`https://<host>:11443`**.
|
||||
- **Is the URL a short-lived signed link?** *Probably not.* The links look like
|
||||
`https://fw-download.ubnt.com/data/unifi-os-server/<hash>-linux-x64-<version>-<uuid>.<n>-x64` with
|
||||
**no signature query string**, and public community installers hard-code them. It is
|
||||
account-derived and per-release, not time-limited. It is kept in SOPS anyway — cheap, and it keeps
|
||||
account-derived material out of plaintext config — but do **not** build logic around expiry.
|
||||
- **Can it run unattended? Yes in practice, no officially — and this is the honest answer.**
|
||||
Ubiquiti documents no `--silent` flag, no answer file and no environment variables. What the
|
||||
ecosystem actually does is pipe confirmations in: the widely used community installer
|
||||
(`dgodibadze/UniFi_OS_Server`) runs `yes y | "$file" install`, and vendor-adjacent walkthroughs run
|
||||
either `./<binary> install` or a bare `sudo ./<binary>`. `install-uos` therefore does
|
||||
`yes y | $t install || yes y | $t` — the documented-by-practice form first, the bare form as a
|
||||
fallback for builds that take no subcommand. **This has not been executed end-to-end on our guest**
|
||||
(no URL has been supplied, and UOS is already installed here), so treat it as "established by
|
||||
convergent third-party practice", not as verified-by-us.
|
||||
- The previously shipped helper had two defects worth recording: it resolved the URL from
|
||||
`fw-update.ui.com/api/firmware-latest`, an endpoint nothing in this repo ever verified lists
|
||||
`unifi-os-server`, and it invoked the binary with **no `install` subcommand**.
|
||||
- **Host:** Ubuntu 24.04 LTS x86-64, systemd, **Podman 4.9.3+**, `slirp4netns` 1.2+, `uidmap`. Min
|
||||
2 GB RAM / 10 GB disk (we give 4 GB / 60 GB). Podman must be at `/usr/bin/podman`.
|
||||
- **Install:** download the signed installer from `fw-download.ubnt.com` (URL obtained from
|
||||
`ui.com/download/software/unifi-os-server`), `chmod +x`, run **as root**. UI then on
|
||||
**`https://<host>:11443`**.
|
||||
- **Confidence:** *High* on requirements, ports, and the download→chmod→run shape (multiple
|
||||
independent 2025–2026 sources agree). *Lower* on **fully unattended** install — see risks.
|
||||
|
||||
### Ports (device adoption needs the L4 ones — the L7 Gateway cannot express them)
|
||||
|
||||
@@ -139,143 +78,76 @@ version. Browser access is via the L4 VIP. Re-add it if/when that lands.
|
||||
| 3478 | UDP | **STUN** (device ⇄ controller) |
|
||||
| 10001 | UDP | **L2 device discovery** |
|
||||
|
||||
**22/TCP is deliberately absent**, and that is load-bearing: a masquerade interface forwards only
|
||||
the ports it declares, so SSH into this guest is impossible by construction. Use the serial console.
|
||||
|
||||
## L4 exposure design (the hard part)
|
||||
|
||||
The shared Gateway is L7 HTTPS only (`:443` host-routed). It **cannot** carry raw TCP `8080`/`6789`
|
||||
or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:
|
||||
|
||||
- **One VIP, two Services.** `unifi-l4-tcp` + `unifi-l4-udp` share a single LoadBalancer IP via
|
||||
Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned through `${UNIFI_L4_VIP}`
|
||||
(`lbipam.cilium.io/ips`) — **192.168.2.21 on prod**. The VIP comes from the existing
|
||||
`default-pool`, whose live prod block is **192.168.2.20–192.168.2.30** with **192.168.2.20 held by
|
||||
the shared Gateway** (`cilium-gateway-shared`), and is ARP-announced on the flat LAN by the
|
||||
existing `default-l2` CiliumL2AnnouncementPolicy — no new Cilium config. (The `192.168.1.53–.60`
|
||||
/ `.53` figures this section used to quote were **dev** addressing and are gone; dev has been
|
||||
decommissioned. Neighbours in the prod pool today: `.22` crafty, `.23` pihole-dns.)
|
||||
Cilium `lbipam.cilium.io/sharing-key: unifi`, pinned to **`192.168.1.54`** (`lbipam.cilium.io/ips`).
|
||||
The VIP is pulled from the existing `default-pool` (192.168.1.53–.60) and **ARP-announced on the
|
||||
flat LAN by the existing `default-l2` CiliumL2AnnouncementPolicy** — no new Cilium config.
|
||||
- `selector: kubevirt.io/vm=unifi` → the virt-launcher pod; the VM's `masquerade` iface forwards the
|
||||
declared ports into the guest.
|
||||
- **`externalTrafficPolicy: Cluster`, corrected 2026-09-14 after an 8-day LAN outage.** This file
|
||||
used to advertise `Local` "to preserve the LAN client source IP". Both halves were false here:
|
||||
Cilium's L2 announcer elects a lease holder without regard to which node holds a backend (the
|
||||
leases sat on `k3s-prod-2` while the VM ran on `k3s-prod-1`, so `Local` had nowhere to forward),
|
||||
and KubeVirt's `masquerade` binding had already NATed the client IP away before the guest ever saw
|
||||
it. The full reasoning — including why `40-networkpolicy.yaml` needs no `remote-node` entry and
|
||||
why no `loadBalancerSourceRanges` was added — is in `50-service-l4.yaml`.
|
||||
- **Device-adoption story:** point devices' **inform host** at the VIP
|
||||
(`http://192.168.2.21:8080/inform`) — via the controller's *Override inform host* setting,
|
||||
`set-inform` on the device, or DHCP option 43. Because the VIP is L2-announced on the devices' own
|
||||
subnet (**192.168.2.0/24**, `LAN_CIDR`), unicast inform + STUN reach it. **Limitation:**
|
||||
broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod masquerade boundary, so
|
||||
zero-touch "device just appears" adoption is unreliable — expect to use explicit inform. Devices
|
||||
on other VLANs must route to the VIP and be inform-pointed anyway.
|
||||
- **Reusable pattern:** L4-only workload → Cilium LoadBalancer on a pinned, L2-announced VIP with a
|
||||
per-app sharing key, A record via external-dns. `crafty` (`.22`) is the same shape and is live.
|
||||
declared ports into the guest. `externalTrafficPolicy: Local` preserves LAN client source IP.
|
||||
- **Device-adoption story:** point devices' **inform host** at the VIP (`http://192.168.1.54:8080/inform`)
|
||||
— via the controller's *Override inform host* setting, `set-inform` on the device, or DHCP option 43.
|
||||
Because the VIP is L2-announced on the devices' own subnet (192.168.1.0/24), unicast inform + STUN
|
||||
reach it. **Limitation:** broadcast-based L2 auto-discovery (10001/udp) does **not** cross the pod
|
||||
masquerade boundary, so zero-touch "device just appears" adoption is unreliable — expect to use
|
||||
explicit inform. Devices on other VLANs must route to the VIP and be inform-pointed anyway.
|
||||
- **Reusable pattern (== the deferred `minecraft` :25565 problem):** L4-only workload → Cilium
|
||||
LoadBalancer on a pinned, L2-announced VIP with a per-app sharing key, A record via external-dns.
|
||||
`minecraft` is the single-TCP-port special case of this multi-port TCP+UDP design.
|
||||
|
||||
**UI access:** `https://unifi.home.keithjackson.dev:11443`. That name resolves to the VIP and not to
|
||||
the Gateway on purpose: both `platform/coredns/coredns-custom.yaml` (in-cluster) and
|
||||
`apps/wave-0/pihole.yaml` (LAN split-horizon) carve `unifi.${BASE_DOMAIN}` out of the
|
||||
zone-wide `${GATEWAY_VIP}` answer and point it at `${UNIFI_L4_VIP}`. `${BASE_DOMAIN}` on prod is the
|
||||
apex `home.keithjackson.dev` (confirmed live in the `cluster-vars` ConfigMap) after the 2026-09-13
|
||||
flip, so this hostname is current, not a dev leftover.
|
||||
|
||||
## The operator's one field — installing UOS without a shell
|
||||
|
||||
`unifi-uos-installer.sops.yaml` is a SOPS-encrypted Secret with two keys. KubeVirt mounts it into
|
||||
the guest as a read-only iso9660 disk labelled `UOSDATA` (one file per key), and an in-guest
|
||||
`uos-install.service` reads it **on every boot**:
|
||||
|
||||
```bash
|
||||
sops apps/vms/unifi/unifi-uos-installer.sops.yaml # fill installer-url and/or console-password
|
||||
git commit && git push # merge to the branch prod's Flux tracks
|
||||
kubectl -n unifi delete vmi unifi # `running: true` recreates it; no virtctl needed
|
||||
kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1) \
|
||||
-c guest-console-log # watch the boot + the install, read-only
|
||||
```
|
||||
|
||||
Design notes that matter if you change this:
|
||||
|
||||
- **The Secret is a disk, not `cloudInitNoCloud.secretRef`.** `secretRef` would lift the 2048-byte
|
||||
cap but would move the whole cloud-config into ciphertext: unreviewable in diffs, invisible to the
|
||||
L0 dollar-collapse guard, and still subject to Flux's envsubst (which runs over **decrypted**
|
||||
resources) — the exact shape of the 2026-09-13 `nextcloud-db` incident. Script in plaintext,
|
||||
operator values in SOPS.
|
||||
- **A URL change needs no UUID bump.** The disk is rebuilt from the Secret on every VMI start and
|
||||
the systemd unit re-reads it every boot. Only a change to the cloud-init payload itself needs the
|
||||
firmware-UUID bump.
|
||||
- **`install-uos` will not reinstall over a working controller.** It exits as soon as systemd knows
|
||||
a `uosserver.service`, with `/var/lib/uos-installed` as a weaker second guard. Given the live
|
||||
guest already has UOS, this guard is the difference between "safe to merge" and "destroys the
|
||||
controller on next boot".
|
||||
- **`apps-vms` had no SOPS `decryption` stanza** — this is the first encrypted content under
|
||||
`apps/vms/`, so the stanza was added to both `clusters/dev/kustomization.yaml` and
|
||||
`clusters/prod/kustomization.yaml`. Without it kustomize hands Flux the raw SOPS document and the
|
||||
guest mounts a disk full of ciphertext, with the cluster reporting everything healthy.
|
||||
**UI access:** `https://unifi.home.keithjackson.dev:11443` (A record → VIP). The port-less
|
||||
`https://unifi.home.keithjackson.dev` via the Gateway is optional and needs a BackendTLSPolicy
|
||||
(UOS is HTTPS-only on 11443) — see `90-httproute-optional.yaml`.
|
||||
|
||||
## Open questions / risks
|
||||
|
||||
1. **The guest lost its IP (HIGH) — THE LIVE BLOCKER, and it needs the operator.** Diagnosis
|
||||
requires a shell, which requires `console-password` above plus one restart. First things to run
|
||||
on the console: `ip -4 a`, `networkctl status enp1s0`, `journalctl -u systemd-networkd`,
|
||||
`ls /etc/netplan`, and `systemctl status uosserver`. A plain restart may well restore DHCP on its
|
||||
own (the address survived the first ~34h of this boot's life), but that would hide the cause, so
|
||||
capture the state before rebooting a second time if you can.
|
||||
2. **Unattended install (MEDIUM) — mechanism shipped, end-to-end run NOT witnessed.** See the
|
||||
install-method section: no official silent flag exists, `yes y | <installer> install` is the
|
||||
convergent community practice, and it has not been executed on our guest. Treat the first real
|
||||
install as something to watch on the console log, not as a fire-and-forget.
|
||||
3. **CDI http import (LOW) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
|
||||
1. **Unattended install (HIGH) — STILL OPEN. This is the blocker.** The installer is a **signed,
|
||||
account-tied** URL that **rotates/expires** (cannot be committed) and is documented as
|
||||
**interactive** (no official silent flag). The shipped cloud-init had to be cut to ~1.1 KB for
|
||||
the inline `userData` limit, so the `install-uos` helper is **no longer baked into the guest**.
|
||||
**Decision needed:** operator runs the install by hand at bring-up (current de-facto answer), or
|
||||
invest in a verified silent path (Secret-delivered cloud-init + a fetched-at-boot URL).
|
||||
2. **CDI http import (MEDIUM) — RESOLVED in practice.** The import succeeded (first http/qcow2 CDI
|
||||
import on this cluster). Still no checksum on an http source, and the URL tracks a *moving*
|
||||
Ubuntu 24.04 build — pin a dated build dir if reproducibility matters. Needed two cluster-level
|
||||
fixes (`nonroot-devices`, `longhorn-scratch`) — see above.
|
||||
4. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
|
||||
5. **VIP `192.168.2.21` (LOW) — RESOLVED.** Allocated and held by both Services, inside the live
|
||||
prod pool `192.168.2.20–.30`, not colliding with the Gateway (`.20`), crafty (`.22`), pihole-dns
|
||||
(`.23`) or the node addresses (`.10/.11/.12`).
|
||||
6. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
|
||||
3. **Boot firmware (LOW) — RESOLVED.** Boots as shipped; running Ubuntu 24.04.4 LTS.
|
||||
4. **VIP `192.168.1.54` (LOW) — RESOLVED.** Allocated and held by both Services, inside the
|
||||
`192.168.1.53–.60` pool (`.53` is the shared Gateway).
|
||||
5. **Longhorn RWX prerequisite (was BLOCKING) — RESOLVED.** `longhorn-vm` is live (RWX / Block /
|
||||
migratable); the VMI reports `LiveMigratable=True` and `StorageLiveMigratable=True`.
|
||||
7. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
|
||||
6. **Licensing (LOW).** UOS self-host is free; a Ubiquiti account is needed only to fetch the
|
||||
installer and for optional remote access.
|
||||
8. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
|
||||
7. **Broadcast adoption (LOW, by design).** L2 auto-discovery (`10001/udp`) does not cross the pod
|
||||
masquerade boundary — see the L4 section. Expect explicit inform-host adoption.
|
||||
9. **No `virtctl` in the devcontainer (LOW).** Restart and read-only console both work with plain
|
||||
`kubectl` (above). An *interactive* console — the thing `console-password` unlocks — needs
|
||||
`virtctl` on the operator's path; it is a single static binary from the KubeVirt release matching
|
||||
the cluster's **v1.9.0**.
|
||||
|
||||
## Remaining bring-up
|
||||
|
||||
Steps 1–5 are **done**. The sequence is kept because it is also the recovery/rebuild runbook, so it
|
||||
is written for a from-scratch cluster; the prod addresses are the live ones.
|
||||
Steps 1–4 below are **done**. The sequence is kept because it is also the recovery/rebuild runbook.
|
||||
|
||||
1. ~~`longhorn-vm` StorageClass live; a free address in the LB pool.~~ **Done** (`192.168.2.21`).
|
||||
1. ~~`longhorn-vm` StorageClass live; `192.168.1.54` free.~~ **Done.**
|
||||
2. ~~Wire into a reconciled path~~ — **done**, via `apps/vms/` + the non-gating `apps-vms`
|
||||
Kustomization (not an app wave), including its SOPS `decryption` stanza in both overlays.
|
||||
Kustomization (not an app wave).
|
||||
3. ~~CDI import~~ — **done**: `kubectl -n unifi get dv unifi-rootdisk` → `Succeeded`.
|
||||
4. ~~VM + VIP~~ — **done**: `kubectl -n unifi get vmi unifi` → `Running`; both Services show
|
||||
`192.168.2.21`.
|
||||
5. ~~Install UOS in the guest~~ — **done** (it was already done when this file claimed otherwise;
|
||||
`uosserver.service` starts on every boot). On a REBUILD this is where the Secret does the work:
|
||||
fill `installer-url`, commit, and the guest installs itself on first boot.
|
||||
6. **→ NEXT, and it is the operator's:** fill `console-password`, restart the VM once, get on the
|
||||
console and find out why `enp1s0` has no address. Until that is fixed nothing below can pass.
|
||||
7. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.2.21:11443`),
|
||||
`192.168.1.54`.
|
||||
5. **→ NEXT: install UOS in the guest.** SSH in (the inline cloud-init installs the operator's key
|
||||
and `qemu-guest-agent`) or `virtctl console unifi`, fetch the signed installer URL from
|
||||
ui.com/download/software/unifi-os-server, then `chmod +x` and run it **as root**.
|
||||
6. Verify the UI: `https://unifi.home.keithjackson.dev:11443` (or `https://192.168.1.54:11443`),
|
||||
complete the first-run wizard.
|
||||
8. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.2.21` from a LAN client (pihole
|
||||
split-horizon) and from in-cluster (coredns-custom).
|
||||
9. Adopt a device: set its inform to `http://192.168.2.21:8080/inform`; confirm it appears + STUN
|
||||
7. Verify DNS: `dig unifi.home.keithjackson.dev` → `192.168.1.54` on the cluster LAN.
|
||||
8. Adopt a device: set its inform to `http://192.168.1.54:8080/inform`; confirm it appears + STUN
|
||||
connects (green). Test guest portal / throughput if used.
|
||||
10. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
|
||||
11. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
|
||||
9. Verify default-deny holds: only the declared L4 ports reachable; VM egress limited to world+DNS.
|
||||
10. (Optional) Re-add the L7 HTTPRoute once `BackendTLSPolicy` is confirmed on this Cilium version.
|
||||
|
||||
**Quick health check today:**
|
||||
```bash
|
||||
kubectl -n unifi get dv,vmi,svc
|
||||
kubectl -n unifi get vmi unifi -o jsonpath='{.status.guestOSInfo.prettyName}{"\n"}'
|
||||
# the two that would have caught the current failure:
|
||||
kubectl -n unifi get vmi unifi -o jsonpath='{.status.interfaces}{"\n"}' # expect an ipAddress
|
||||
kubectl -n unifi exec "$(kubectl -n unifi get pod -l kubevirt.io=virt-launcher -o name | head -1)" \
|
||||
-c compute -- ip neigh # expect 10.0.2.2 REACHABLE
|
||||
```
|
||||
@@ -11,11 +11,3 @@ resources:
|
||||
- 30-virtualmachine.yaml
|
||||
- 40-networkpolicy.yaml
|
||||
- 50-service-l4.yaml
|
||||
# SOPS-encrypted. Carries the operator's UOS installer URL (+ optional SSH key) and is mounted
|
||||
# into the guest as the UOSDATA disk by 30-virtualmachine.yaml. Decrypted by the `apps-vms` Flux
|
||||
# Kustomization, which clusters/dev and clusters/prod patch with the `decryption` stanza — the
|
||||
# FIRST encrypted content under apps/vms, so that patch was added with this file. The plaintext
|
||||
# template + the operator hand-off steps live in the `.example` sibling of the file below, which
|
||||
# is deliberately NOT named here: the L0 no-plaintext-secrets check fails any kustomization whose
|
||||
# text contains an example template's filename, on the assumption it is being applied.
|
||||
- unifi-uos-installer.sops.yaml
|
||||
@@ -1,29 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: unifi-uos-installer
|
||||
namespace: unifi
|
||||
type: Opaque
|
||||
stringData:
|
||||
installer-url: ENC[AES256_GCM,data:pZo/fRSnUmZj2w==,iv:tBWMAL0NwCURCmfzx3UAtryCovcR+Rmh4DdtUiCi4Ig=,tag:ZfLicc3pBWujPHKqfSNOag==,type:str]
|
||||
console-password: ENC[AES256_GCM,data:rQ6v6VjG4uoEShHic+zM,iv:nV7mTJFRsLvd67uZT+KyMrOhUVkERVv53LqsvcE5Qfs=,tag:9J/t2je48DRTVmppPqDfFQ==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB5M1RwWGhRKzNVenpzS25w
|
||||
WW4vNHFURy9hMjRGeW0wdUJSaHN4MDFhSHlBCm1udi9NOWJ3RGdvOVNnRG9ZUlp4
|
||||
TnVXRi90eFNSMjNVYVpUSmtFTURtU2MKLS0tIExPN2NRZW9CQXBKdUtCUkZSTXVO
|
||||
cE9IcTM2RG9oMmt2Y0xsRExzUjYwUlEK6vq56tnnYvruNcCgijRUsfi3IreXt7fk
|
||||
kgDevz3fOKkEJ3Y8+Mx3RBf3j6Za7L1lW7ft9TyTkBelRpj9bRD1Pg==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-09-19T02:47:13Z"
|
||||
mac: ENC[AES256_GCM,data:DVJZQlHHb0U/F8oQ2iwPSTCvzHNynGZFW4MSHzW+LmNt8ozA7n2/U9prhRdrq3a1MHlaoGAkOFMGoQdZRR83CtZyP6oFoLiAMsRvNjNMgkaT1E3D7dzhjxAiTb+vQk9O467bH3ofm1+I/Ak3VFrNdjTtWFkWltW7MHB8njgPRPM=,iv:Sve3Z6RHKSWcdcEZtphBBmhRxjhfJHu9Dus2vb6K4+0=,tag:OrsGBPJVF1jtOAXCWNXG2Q==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
@@ -1,71 +0,0 @@
|
||||
# =============================================================================
|
||||
# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
|
||||
# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
|
||||
#
|
||||
# WHAT THIS IS: the two values that (a) let the UniFi VM install UniFi OS Server by itself, with
|
||||
# nobody typing commands inside the guest, and (b) give the operator a way INTO that guest at all.
|
||||
# KubeVirt mounts this Secret into the VM as a tiny read-only iso9660 disk labelled `UOSDATA` (see
|
||||
# the `uosdata` volume in 30-virtualmachine.yaml); each key below becomes a file of the same name on
|
||||
# that disk, and the in-guest `uos-install.service` reads them on EVERY boot.
|
||||
#
|
||||
# READ THIS FIRST — UOS IS ALREADY INSTALLED ON THE LIVE PROD GUEST (verified 2026-09-14, see the
|
||||
# README "What is actually wrong"). `installer-url` is therefore NOT the fix for today's outage; it
|
||||
# is the rebuild path and the "the install never happened" path. `install-uos` refuses to run the
|
||||
# installer at all while systemd knows a `uosserver.service`, so filling this in cannot clobber a
|
||||
# working controller. `console-password` IS useful today: it is the only way to get a shell.
|
||||
#
|
||||
# WHY THE URL IS BEHIND SOPS: it is obtained through the operator's own (free) Ubiquiti account
|
||||
# session at ui.com/download/software/unifi-os-server, it identifies a specific build, and it is not
|
||||
# ours to redistribute. Treat it as account-derived material, not as public configuration.
|
||||
# HONEST CAVEAT, checked 2026-09-14: the resulting `https://fw-download.ubnt.com/data/
|
||||
# unifi-os-server/<hash>-linux-x64-<version>-<uuid>.<n>-x64` URL carries NO signature query string,
|
||||
# and public community install scripts hard-code these URLs — so it does not behave like a
|
||||
# short-lived signed link. It does change every release. Encrypting it costs nothing and keeps the
|
||||
# repo's "account-derived material is not config" line clean, but do NOT design around "it expires".
|
||||
#
|
||||
# HAND-OFF STEPS (operator) — this is the whole job:
|
||||
# (a) Sign in at https://ui.com/download/software/unifi-os-server, pick the Linux (x64) build, and
|
||||
# copy the download link (right-click -> Copy link address). Do NOT download it here; the guest
|
||||
# fetches it itself.
|
||||
# (b) Open the encrypted sibling in your editor:
|
||||
# sops apps/vms/unifi/unifi-uos-installer.sops.yaml
|
||||
# (c) Replace the two REPLACE-ME values. Either may be left as the placeholder — the guest skips a
|
||||
# placeholder rather than failing.
|
||||
# (d) Save/quit (sops re-encrypts in place), commit, push, merge to the branch prod tracks.
|
||||
# (e) Restart the VM once so the new cloud-init instance and the new disk are picked up:
|
||||
# kubectl -n unifi delete vmi unifi # `running: true` recreates it; no virtctl needed
|
||||
# Watch it come up WITHOUT virtctl:
|
||||
# kubectl -n unifi logs -f $(kubectl -n unifi get pod -l kubevirt.io=virt-launcher \
|
||||
# -o name | head -1) -c guest-console-log
|
||||
# (f) UI: https://192.168.2.21:11443 (prod) or https://unifi.home.keithjackson.dev:11443.
|
||||
#
|
||||
# IF YOU LATER NEED A NEW URL (new release, dead link): repeat (b)-(e). No firmware-UUID bump is
|
||||
# needed for that — the disk is rebuilt from this Secret on every VMI start and the systemd unit
|
||||
# re-reads it every boot. The UUID bump in 30-virtualmachine.yaml is only needed when the cloud-init
|
||||
# payload ITSELF changes.
|
||||
#
|
||||
# The file is already listed under `resources:` in apps/vms/unifi/kustomization.yaml, and `apps-vms`
|
||||
# is patched with the SOPS `decryption` stanza in clusters/dev + clusters/prod.
|
||||
# =============================================================================
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: unifi-uos-installer
|
||||
namespace: unifi
|
||||
type: Opaque
|
||||
stringData:
|
||||
# KEY NAMES ARE LOAD-BEARING: each key becomes a FILE of that exact name on the UOSDATA disk, and
|
||||
# /usr/local/sbin/install-uos reads them by path. Renaming a key breaks the install silently.
|
||||
#
|
||||
# The signed-in download link for UniFi OS Server, Linux x64. Anything that is not an https URL
|
||||
# (this placeholder included) is treated by the guest as "not provided yet": the unit logs
|
||||
# "UOS installer-url not set yet" and exits 0, so the VM stays healthy and simply does not install.
|
||||
installer-url: REPLACE-ME
|
||||
# Password set on the guest's `ubuntu` account on every boot, for `virtctl -n unifi console unifi`.
|
||||
# THIS IS THE ONLY DOOR INTO THE GUEST. The stock Ubuntu 24.04 cloud image ships `ubuntu` locked,
|
||||
# no key was ever installed, and an SSH key would not help: the VM's masquerade interface forwards
|
||||
# only the ports declared in 30-virtualmachine.yaml and 22 is not among them (probed from inside
|
||||
# virt-launcher, 2026-09-14 — the guest RSTs on 22 just as it does on 11443). Generate one with
|
||||
# openssl rand -base64 24
|
||||
# Leave it as REPLACE-ME to keep the account locked; the script skips the placeholder by name.
|
||||
console-password: REPLACE-ME
|
||||
+27
-32
@@ -90,9 +90,9 @@ spec:
|
||||
name: gitea-bootstrap
|
||||
key: secret-key
|
||||
- name: GITEA__server__DOMAIN
|
||||
value: gitea.${BASE_DOMAIN}
|
||||
value: gitea.home.keithjackson.dev
|
||||
- name: GITEA__server__ROOT_URL
|
||||
value: https://gitea.${BASE_DOMAIN}/
|
||||
value: https://gitea.home.keithjackson.dev/
|
||||
- name: GITEA__actions__ENABLED
|
||||
value: 'true'
|
||||
- name: GITEA__actions__DEFAULT_ACTIONS_URL
|
||||
@@ -127,8 +127,8 @@ spec:
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n i=$((i + 1))\n [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n n=0\n until su-exec git gitea admin user create \\\n --username \"$GITEA_ADMIN_USERNAME\" \\\n --password \"$GITEA_ADMIN_PASSWORD\" \\\n --email \"[email protected]\" \\\n --admin --must-change-password=false; do\n n=$((n + 1))\n [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n sleep 3\n done\n echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n d=0; disc_ok=0\n while [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1;\
|
||||
\ break\n fi\n d=$((d + 1)); sleep 2\n done\n [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n r=0; reg_ok=0\n while [ \"$r\" -lt 10 ]; do\n oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n if [ -n \"$oidc_id\" ]; then\n su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n else\n su-exec git gitea admin auth add-oauth \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n fi\n # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\n done\n if [ \"$reg_ok\" = 1 ]; then\n echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n else\n gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n fi\nfi\n# A postStart hook MUST exit 0 \u2014 a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but\n# it is made EXPLICIT so the guarantee does not depend on whatever statement happens to\n# be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,\n# the moment its OIDC guard was fixed and the block actually ran for the first time.\nexit 0\n"
|
||||
- "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n i=$((i + 1))\n [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n n=0\n until su-exec git gitea admin user create \\\n --username \"$GITEA_ADMIN_USERNAME\" \\\n --password \"$GITEA_ADMIN_PASSWORD\" \\\n --email \"[email protected]\" \\\n --admin --must-change-password=false; do\n n=$((n + 1))\n [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n sleep 3\n done\n echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\nif [ -z \"${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n d=0; disc_ok=0\n while [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1; break\n fi\n d=$((d + 1)); sleep 2\n done\n [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n r=0; reg_ok=0\n while [ \"$r\" -lt 10 ]; do\n oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n if [ -n \"$oidc_id\" ]; then\n su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n else\n su-exec git gitea admin auth add-oauth \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name\
|
||||
\ groups --admin-group platform-admins >/dev/null 2>&1 || true\n fi\n # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\n done\n if [ \"$reg_ok\" = 1 ]; then\n echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n else\n gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n fi\nfi\n"
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
@@ -170,27 +170,6 @@ spec:
|
||||
- port: '3000'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy in-cluster client ingress allow (R-NET-02): gitea-allow-flux-ingress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: gitea-allow-flux-ingress
|
||||
namespace: gitea
|
||||
labels:
|
||||
app.kubernetes.io/name: gitea
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: gitea
|
||||
ingress:
|
||||
- fromEndpoints:
|
||||
- matchLabels:
|
||||
k8s:io.kubernetes.pod.namespace: flux-system
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '3000'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): gitea-allow-internet-egress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
@@ -206,10 +185,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute gitea-3000 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
@@ -228,7 +203,7 @@ spec:
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- gitea.${BASE_DOMAIN}
|
||||
- gitea.home.keithjackson.dev
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
@@ -238,6 +213,26 @@ spec:
|
||||
- name: gitea
|
||||
port: 3000
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: gitea
|
||||
namespace: gitea
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: gitea
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: gitea
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- gitea
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
@@ -252,11 +247,11 @@ spec:
|
||||
- name: gitea.slis
|
||||
rules:
|
||||
- alert: gitea-down
|
||||
expr: kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"} == 0 or absent(kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"})
|
||||
expr: up{job="gitea"} == 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: gitea
|
||||
annotations:
|
||||
summary: gitea is down
|
||||
description: 'gitea: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
|
||||
description: 'gitea has been unreachable for 5m (SLI: up).'
|
||||
@@ -1,30 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: jackett-oauth2-proxy
|
||||
namespace: jackett
|
||||
type: Opaque
|
||||
stringData:
|
||||
client-id: ENC[AES256_GCM,data:n5h6NG0emA==,iv:d6lvZfgRq6+IKNpsSoRa5a38R2f2+tWBO+j9QKDThMY=,tag:P+cvQXlWR4aZs7a9/lVuMQ==,type:str]
|
||||
client-secret: ENC[AES256_GCM,data:WL5G7PuVTdHQoe7Zp2eiE9ppxSGE0+W2RMfISuwqD6JA9auG97ZYnPUjG8HNx7sosOvsmqeE9kONK23IqIeaQw==,iv:kSP10UKeDv/8BsgX36+uKwEY+LBovcicce1RkD14M4g=,tag:fEUsVuajWPxAmAo8Nnrrrg==,type:str]
|
||||
cookie-secret: ENC[AES256_GCM,data:WChW9YLhzG9Nwp8dCWTFtirufXxGk3Ep9HXpfqcNz69PXQW3RaMC4EJpw2U=,iv:f6obIerkog8Q2gfj1tVL259LOl5jj59VE29KnJwmCXQ=,tag:s0FwNtU6vffKgDcGUNUhCg==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBLWVptVHc4aFl0MXRWTHVP
|
||||
cC9qcVoyZUV2TGZnTEpHY3RLOU8vQnNNNm00ClZRejE2bzFaWXNSUmQ5MEZtV3VK
|
||||
djB2TXJqQjVUOE93QjN6U0lMVDBEbm8KLS0tIFFTZnB0c2pCZWZMZ084WEIxT2tW
|
||||
WUxxemJHdWRZbnBpL280dzlpMmhwSW8Kra0bkx3iJYQi+3rnpU/89fJbJHzQjKi2
|
||||
ZWNwYhLpULw1xNo3xbCm+PUp5f30kpyMKQld2qaLOZAqWI5vNl9Byw==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-08-31T13:41:37Z"
|
||||
mac: ENC[AES256_GCM,data:zU9nWCrSomAXAZ/Ve8U3yLHnoxhu8nkQ7523neJTGv810ONzAAZ7Gr8nFHR7bTqz2QwuGYD3P1ypdqYnKLM8ri2Zf1LDo/Su9ptnvQhUbfXKlRorcW6uAWsViCwvTM05wZNkdzZfza7tGwJQVTD6hc0cP+/7Y7JnpT5FEstK254=,iv:oggTRjcWqAJizA/a22+mtslkEWka8hUG0wudM8obO4g=,tag:i0TtNmqGe+b5t4iq8LpsqA==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
+28
-81
@@ -74,7 +74,7 @@ spec:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
homelab.keithjackson.dev/auth-groups: media-users
|
||||
spec:
|
||||
containers:
|
||||
- name: jackett
|
||||
@@ -83,72 +83,7 @@ spec:
|
||||
- containerPort: 9117
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /config
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
- --reverse-proxy=true
|
||||
- --provider=oidc
|
||||
- --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
|
||||
- --upstream=http://127.0.0.1:9117
|
||||
- --email-domain=*
|
||||
- --scope=openid email profile
|
||||
- --oidc-groups-claim=groups
|
||||
- --allowed-group=media-users
|
||||
- --allowed-group=platform-admins
|
||||
- --skip-provider-button=true
|
||||
- --redirect-url=https://jackett.${BASE_DOMAIN}/oauth2/callback
|
||||
- --cookie-domain=.${BASE_DOMAIN}
|
||||
- --whitelist-domain=.${BASE_DOMAIN}
|
||||
- --cookie-secure=true
|
||||
- --pass-access-token=true
|
||||
- --set-xauthrequest=true
|
||||
env:
|
||||
- name: OAUTH2_PROXY_CLIENT_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: jackett-oauth2-proxy
|
||||
key: client-id
|
||||
- name: OAUTH2_PROXY_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: jackett-oauth2-proxy
|
||||
key: client-secret
|
||||
- name: OAUTH2_PROXY_COOKIE_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: jackett-oauth2-proxy
|
||||
key: cookie-secret
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 4180
|
||||
protocol: TCP
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
mountPath: /data/config
|
||||
volumes:
|
||||
- name: config
|
||||
persistentVolumeClaim:
|
||||
@@ -169,9 +104,6 @@ spec:
|
||||
- name: port-9117
|
||||
port: 9117
|
||||
targetPort: 9117
|
||||
- name: authproxy
|
||||
port: 4180
|
||||
targetPort: 4180
|
||||
---
|
||||
# CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): jackett-allow-radarr-9117
|
||||
apiVersion: cilium.io/v2
|
||||
@@ -234,7 +166,7 @@ spec:
|
||||
- ingress
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '4180'
|
||||
- port: '9117'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): jackett-allow-internet-egress
|
||||
@@ -252,10 +184,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute jackett-9117 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
@@ -267,15 +195,14 @@ metadata:
|
||||
app.kubernetes.io/name: jackett
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
homelab.keithjackson.dev/auth-groups: media-users
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- jackett.${BASE_DOMAIN}
|
||||
- jackett.home.keithjackson.dev
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
@@ -283,7 +210,27 @@ spec:
|
||||
value: /
|
||||
backendRefs:
|
||||
- name: jackett
|
||||
port: 4180
|
||||
port: 9117
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: jackett
|
||||
namespace: jackett
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: jackett
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: jackett
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- jackett
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
@@ -299,11 +246,11 @@ spec:
|
||||
- name: jackett.slis
|
||||
rules:
|
||||
- alert: jackett-down
|
||||
expr: kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"} == 0 or absent(kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"})
|
||||
expr: up{job="jackett"} == 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: jackett
|
||||
annotations:
|
||||
summary: jackett is down
|
||||
description: 'jackett: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
|
||||
description: 'jackett has been unreachable for 5m (SLI: up).'
|
||||
@@ -8,21 +8,13 @@ kind: Kustomization
|
||||
resources:
|
||||
- gitea-bootstrap-secret.sops.yaml
|
||||
- gitea.yaml
|
||||
- jackett-oauth2-proxy-credentials.sops.yaml
|
||||
- jackett.yaml
|
||||
- nextcloud-db-app.sops.yaml
|
||||
- nextcloud-db-backup-credentials-prod.sops.yaml
|
||||
- nextcloud-db-backup-credentials.sops.yaml
|
||||
- nextcloud-db.yaml
|
||||
- pihole-dns.yaml
|
||||
- pihole-oauth2-proxy-credentials.sops.yaml
|
||||
- pihole.yaml
|
||||
- plex-claim-secret.sops.yaml
|
||||
- plex.yaml
|
||||
- qbittorrent-bootstrap-secret.sops.yaml
|
||||
- qbittorrent-oauth2-proxy-credentials.sops.yaml
|
||||
- qbittorrent.yaml
|
||||
- semaphore-db-app.sops.yaml
|
||||
- semaphore-db-backup-credentials-prod.sops.yaml
|
||||
- semaphore-db-backup-credentials.sops.yaml
|
||||
- semaphore-db.yaml
|
||||
@@ -1,29 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: nextcloud-db-backup-credentials-prod
|
||||
namespace: nextcloud-db
|
||||
type: Opaque
|
||||
stringData:
|
||||
ACCESS_KEY_ID: ENC[AES256_GCM,data:4lx6et4HcXDegulyIoa7jZR/lVgapGPEkukajfCKjvk=,iv:p6V7tizbikk55tOvpPOsPHTe/p52QLX5A5xrwuB14rA=,tag:oNJuipT/Lb6zynJ2K2Q71A==,type:str]
|
||||
ACCESS_SECRET_KEY: ENC[AES256_GCM,data:V5bHPoPeBOVyLp1n1Jilyso4i5P2aPSH0hrryfweg2q/SctWZxRbLBQ/FLzOigbJsAhlwOGu069mQPma6xjfdw==,iv:8u4UOvI8dC/cm6mEWDlyN5E3SfRZmwKY9uZY/DjPEsQ=,tag:tw7tGs29TmUvMPBCtCszAA==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBiSUt5N2k2N2lDU2ZsMy9m
|
||||
M1NFS0M2QjFsNFhnR0hCMzFyaTdiQWFZRFJZCmhBTWRXNmxLMGtsc3Rxd2FSMDZp
|
||||
aktjcTE2c1hmU1EzM3RWNksyb3Q1YVUKLS0tIDBFS1I5aDBER2cvcWtLZ3BXTEp5
|
||||
ZGI5ZmZIcStheUNQamxIVVVpc0ZMOHMKlE5aHewtHJpeU24F5sodc9CEm9KTjHQQ
|
||||
8ThEQ/8vyazL9V4ojlICxSHU/SHkHpD1eBaTKcAQd/LZzujIcJ+QfA==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-09-07T06:36:33Z"
|
||||
mac: ENC[AES256_GCM,data:8awz1Jh8t4zjDMyd93/v+6on9iiIaSi01293A7uYhxj9L1+g63zeO+DPmSn6+KbZvHdqU2A7up09H1NzfXAEdGuvVbB36cdGq0zKank3PUi8c7IT6qNuWydxRAcqLnbbD+c30Z6GB+gAuZHstNOe4Rgc4aJvMN6FuP6x6Z0870c=,iv:KGGKJq2QA5xztvoHs/SViXW72DYwT16i+W+dJBUeIJQ=,tag:3UpYqx2+RIK0QdMH/BSIpA==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
+13
-305
@@ -37,53 +37,16 @@ metadata:
|
||||
name: nextcloud-db
|
||||
namespace: nextcloud-db
|
||||
spec:
|
||||
# THREE instances — one primary + two hot standbys (2026-09-07), same as keycloak-db; see
|
||||
# platform/keycloak/postgres-cluster.yaml for the full rationale. Short version: Longhorn's 3-way
|
||||
# replication saves the DATA on node loss, not the SERVICE, and with `instances: 1` all three CNPG
|
||||
# primaries had drifted onto k3s-prod-2, so one node took Nextcloud down alongside SSO and
|
||||
# Semaphore. A streaming standby lets CNPG promote in seconds and repoint the -rw Service.
|
||||
# Gated on the Barman archive below actually working (ContinuousArchiving=True with a real
|
||||
# firstRecoverabilityPoint) — standbys replicate corruption, they are not a backup.
|
||||
#
|
||||
# Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
|
||||
# pg_basebackup from the running primary, which is neither restarted nor disconnected.
|
||||
instances: 3
|
||||
# Single Postgres instance on 3-replica Longhorn is acceptable at home scale (ADR-0014, same as
|
||||
# keycloak-db). The storage is already 3-way replicated by Longhorn; raise to 3 for Postgres-level
|
||||
# HA as a future hardening.
|
||||
instances: 1
|
||||
imageName: ghcr.io/cloudnative-pg/postgresql:16.4
|
||||
# Stamp app.kubernetes.io/name=nextcloud-db onto every generated pod/service so the rendered
|
||||
# nextcloud egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
|
||||
inheritedMetadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: nextcloud-db
|
||||
# HARD anti-affinity — the three instances MUST land on three DIFFERENT nodes; and roll the
|
||||
# primary by SWITCHOVER rather than CNPG's default in-place `restart`, which would delete and
|
||||
# recreate the primary pod. Full rationale in platform/keycloak/postgres-cluster.yaml. Short
|
||||
# version: CNPG's default `preferred` is a scheduler hint, so when CNPG recreates an instance
|
||||
# after a node loss it may stack two instances on one node — which is how `instances: 3` decays
|
||||
# back into the single-node failure domain this change removes. `required` makes it a constraint;
|
||||
# the cost is that an instance whose node is gone stays Pending, which is correct (2 of 3 still
|
||||
# serve). Satisfiable on both clusters: prod and dev each have 3 schedulable, untainted nodes.
|
||||
#
|
||||
# APPLYING THIS CHANGES THE POD SPEC, so CNPG rolls the instances: standbys first, then one
|
||||
# switchover. Expect a single brief connection reset for Nextcloud.
|
||||
affinity:
|
||||
enablePodAntiAffinity: true
|
||||
podAntiAffinityType: required
|
||||
primaryUpdateMethod: switchover
|
||||
# SYNCHRONOUS REPLICATION — quorum of ONE standby, PREFERRED durability (2026-09-07). Full
|
||||
# rationale in platform/keycloak/postgres-cluster.yaml. Short version: async replication loses the
|
||||
# last few transactions when a primary dies before its standbys catch up, and `method: any` +
|
||||
# `number: 1` makes COMMIT wait for one standby to confirm. dataDurability MUST stay `preferred`,
|
||||
# never `required` — `required` blocks writes outright when fewer than `number` healthy standbys
|
||||
# are available, which on a 3-node cluster sharing nodes with etcd and the CNI turns one degraded
|
||||
# standby into a total write outage. `preferred` relaxes to async instead.
|
||||
#
|
||||
# NON-DISRUPTIVE: compiles to `synchronous_standby_names`, a reload-only PostgreSQL parameter —
|
||||
# no restart, no switchover, no client disconnect. Land it after the anti-affinity roll.
|
||||
postgresql:
|
||||
synchronous:
|
||||
method: any
|
||||
number: 1
|
||||
dataDurability: preferred
|
||||
storage:
|
||||
storageClass: longhorn
|
||||
size: 5Gi
|
||||
@@ -105,38 +68,23 @@ spec:
|
||||
memory: 256Mi
|
||||
limits:
|
||||
memory: 512Mi
|
||||
# --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → object storage ---
|
||||
# --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → the on-Proxmox MinIO `cnpg` bucket ---
|
||||
# Continuous WAL archiving + periodic base backups = proper Postgres PITR (realises the SD's
|
||||
# backup.policy daily-offcluster). CNPG appends serverName (defaults to the Cluster name) to
|
||||
# destinationPath, so objects land under ${CNPG_BACKUP_PATH}/nextcloud-db/ — dev
|
||||
# s3://cnpg/nextcloud-db/ on the on-Proxmox MinIO, prod
|
||||
# s3://k3s-prod-backup/barman/nextcloud-db/ on Cloudflare R2. Both endpointURL and
|
||||
# destinationPath are cluster-vars knobs (R-ENV-02), substituted by `apps-wave-0`.
|
||||
# Credentials: PER-ENVIRONMENT (2026-09-07). dev reads nextcloud-db-backup-credentials (the
|
||||
# scoped MinIO key), prod reads nextcloud-db-backup-credentials-prod (the Cloudflare R2
|
||||
# token). BOTH Secrets are applied to BOTH clusters by apps/wave-0/kustomization.yaml (a
|
||||
# generated golden — render.py places the -prod sidecar in this DB's wave); the
|
||||
# cluster-vars knob ${BACKUP_CREDENTIALS_SUFFIX} (dev "" | prod "-prod") chooses which
|
||||
# one this Cluster reads. Both are in-namespace, which is all a pod can read. They were
|
||||
# ONE shared Secret until the split, so giving prod its R2 token would have left dev
|
||||
# signing MinIO requests with an R2 key id — 403 on every archive command, i.e. the
|
||||
# disk-full clock below rather than a merely missing backup.
|
||||
#
|
||||
# archive_mode is ON, so a FAILING archive command is a disk-full clock, not a degraded backup:
|
||||
# Postgres refuses to recycle a WAL segment it has not archived, and the pgdata PVC fills until
|
||||
# the primary PANICs. Prod demonstrated this from bootstrap to 2026-09-07 with a placeholder
|
||||
# endpoint. Treat ContinuousArchiving=False here as an availability incident.
|
||||
# backup.policy daily-offcluster). destinationPath is the bucket ROOT; CNPG appends serverName
|
||||
# (defaults to the Cluster name), so objects land under s3://cnpg/nextcloud-db/. Path-style HTTP.
|
||||
# Credentials: the scoped key from the in-namespace SOPS Secret nextcloud-db-backup-credentials
|
||||
# (same-ns only), kept in sync with platform/backup/minio-credentials.sops.yaml.
|
||||
backup:
|
||||
retentionPolicy: "30d"
|
||||
barmanObjectStore:
|
||||
destinationPath: ${CNPG_BACKUP_PATH}
|
||||
endpointURL: ${S3_BACKUP_ENDPOINT}
|
||||
destinationPath: s3://cnpg
|
||||
endpointURL: http://192.168.1.9:9000
|
||||
s3Credentials:
|
||||
accessKeyId:
|
||||
name: nextcloud-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
|
||||
name: nextcloud-db-backup-credentials
|
||||
key: ACCESS_KEY_ID
|
||||
secretAccessKey:
|
||||
name: nextcloud-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
|
||||
name: nextcloud-db-backup-credentials
|
||||
key: ACCESS_SECRET_KEY
|
||||
wal:
|
||||
compression: gzip
|
||||
@@ -158,243 +106,3 @@ spec:
|
||||
cluster:
|
||||
name: nextcloud-db
|
||||
method: barmanObjectStore
|
||||
---
|
||||
# =============================================================================
|
||||
# CNPG METRICS-EXPORTER CONNECT RECONCILER (R-OBS-01)
|
||||
#
|
||||
# THE BUG THIS CLOSES (true positive, verified live on prod 2026-09-12, and the
|
||||
# condition platform/cnpg/alerts.yaml already recorded on BOTH clusters 2026-09-06):
|
||||
# the `nextcloud` database's ACL had been narrowed out-of-band so that PUBLIC held
|
||||
# nothing, and CNPG's metrics exporter role therefore could not even CONNECT:
|
||||
#
|
||||
# nextcloud-db-1/-2/-3, every scrape:
|
||||
# FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)
|
||||
# DETAIL: User does not have CONNECT privilege.
|
||||
# msg="Error collecting user query" targetDatabase=nextcloud
|
||||
#
|
||||
# CONSEQUENCE: `cnpg_last_error` = 1 on all three instances and the ENTIRE
|
||||
# cnpg_pg_stat_* / pg_settings / pg_postmaster family MISSING for this database,
|
||||
# while the scrape target still reported up=1 — a blind spot that looks like health.
|
||||
# Measured before the fix: cnpg_pg_settings_setting had 765 series for keycloak and
|
||||
# 765 for semaphore-db and ZERO for nextcloud-db. Nextcloud's database had never
|
||||
# been monitored.
|
||||
#
|
||||
# ROOT CAUSE, AND WHY ONLY THIS DATABASE. All three Cluster specs are identical in
|
||||
# their bootstrap; the difference is entirely in the live ACL:
|
||||
#
|
||||
# nextcloud : datacl = {nextcloud=CTc/nextcloud} <-- explicit, no PUBLIC entry
|
||||
# keycloak : datacl = NULL
|
||||
# semaphore : datacl = NULL
|
||||
#
|
||||
# A NULL datacl means "PostgreSQL defaults", which include CONNECT and TEMP for
|
||||
# PUBLIC — so keycloak and semaphore let the exporter in only by inheriting PUBLIC's
|
||||
# implicit CONNECT. On `nextcloud` something ran the equivalent of
|
||||
# `REVOKE ALL ON DATABASE nextcloud FROM PUBLIC`, which MATERIALISES datacl and
|
||||
# drops the implicit PUBLIC entry with it. That REVOKE is not in this repo and not
|
||||
# in DECISIONS-LOG.md — it was applied out of band, which is exactly why a
|
||||
# git-declared fix is needed rather than another hand-run psql.
|
||||
#
|
||||
# WHY THE GRANT IS EXPLICIT RATHER THAN "RE-GRANT CONNECT TO PUBLIC": restoring
|
||||
# PUBLIC's CONNECT would undo whatever hardening that REVOKE intended and would
|
||||
# leave the exporter depending on an implicit default again — the precise fragility
|
||||
# that broke here and that keycloak/semaphore are still relying on. Naming the
|
||||
# exporter role makes the dependency explicit and survives a future REVOKE FROM
|
||||
# PUBLIC on any of the three databases.
|
||||
#
|
||||
# WHY A CronJob AND NOT A CNPG FIELD — CNPG 1.30.0's API GENUINELY CANNOT EXPRESS
|
||||
# THIS. Checked against the live CRDs on prod:
|
||||
# * `Database` (databases.postgresql.cnpg.io) has no database-ACL field at all.
|
||||
# `allowConnections` is the pg_database.datallowconn FLAG (open/closed to
|
||||
# everyone), not a per-role GRANT; its only grant/revoke verbs are the `usage`
|
||||
# lists on `fdws` and `servers`.
|
||||
# * `spec.managed.roles` and `DatabaseRole` manage ROLE ATTRIBUTES (login,
|
||||
# superuser, connectionLimit, ...) plus `inRoles` MEMBERSHIP. Membership is the
|
||||
# wrong tool: the only local role holding CONNECT here is the OWNER, so
|
||||
# `inRoles: [nextcloud]` would hand the exporter the owner's CREATE and TEMP on
|
||||
# the database and read/write on every application table — the opposite of
|
||||
# least privilege, to buy one CONNECT.
|
||||
# * `bootstrap.initdb.postInitApplicationSQL` runs ONCE, at bootstrap. It cannot
|
||||
# repair these three already-bootstrapped instances (the actual bug), and on a
|
||||
# FRESH cluster the grant is unnecessary anyway (datacl starts NULL). Adding it
|
||||
# would be a second mechanism that fixes nothing this one does not.
|
||||
# So the SQL has to be applied by something, and this repo already has the pattern
|
||||
# for state Flux cannot reach: the level-triggered CronJob reconciler in
|
||||
# platform/keycloak/client-secret-reconciler.yaml, whose reasoning transfers
|
||||
# verbatim — the drifted state lives inside PostgreSQL, not in a Kubernetes object,
|
||||
# so no admission or mutation policy can see it. A one-shot Job is deliberately NOT
|
||||
# used: a completed Job is never re-run, so a DB rebuild or a PITR restore would
|
||||
# silently reopen the blind spot. A loop converges instead.
|
||||
#
|
||||
# PRIVILEGE GRANTED — EXACTLY ONE, AND NOTHING ELSE:
|
||||
# GRANT CONNECT ON DATABASE nextcloud TO cnpg_metrics_exporter;
|
||||
# Resulting ACL entry is `cnpg_metrics_exporter=c/nextcloud` — the `c` is CONNECT;
|
||||
# there is no `C` (CREATE) and no `T` (TEMPORARY). No ownership, no superuser, no
|
||||
# write, no table or schema grants. Nothing further is needed because CNPG already
|
||||
# creates this role as a member of the built-in `pg_monitor` role (verified on all
|
||||
# three prod clusters), and pg_monitor already carries the read access every
|
||||
# collector query wants — they read pg_catalog views (pg_stat_database,
|
||||
# pg_stat_archiver, pg_settings, pg_postmaster) and no application tables. Proof
|
||||
# that CONNECT alone is sufficient: after this grant, `Error collecting user query`
|
||||
# went to ZERO on all three pods, i.e. every collector query now succeeds.
|
||||
#
|
||||
# IT RUNS AS THE DATABASE OWNER, NOT AS A SUPERUSER. `enableSuperuserAccess` is off
|
||||
# on this Cluster (there is no nextcloud-db-superuser Secret), and it stays off: the
|
||||
# owner of a database holds the grant option on that database's privileges, so
|
||||
# `nextcloud` can issue this GRANT itself. Verified live before committing, by
|
||||
# running the statement under `SET ROLE nextcloud`. Credentials come from the
|
||||
# in-namespace SOPS Secret nextcloud-db-app that already exists for the app.
|
||||
#
|
||||
# NETWORKING: nextcloud-db is intentionally NOT an app-namespace (see this file's
|
||||
# header), so Kyverno generates no default-deny baseline here and this same-namespace
|
||||
# pod -> nextcloud-db-rw:5432 connection needs no rendered allow.
|
||||
#
|
||||
# THE POD LABEL IS DELIBERATELY *NOT* app.kubernetes.io/name=nextcloud-db. That
|
||||
# label is what `inheritedMetadata` stamps on the CNPG instance pods and what the
|
||||
# rendered nextcloud egress CiliumNetworkPolicy selects; reusing it here would quietly
|
||||
# make this reconciler a permitted destination for the Nextcloud app. It gets its own
|
||||
# name so that selector keeps meaning "the database".
|
||||
apiVersion: batch/v1
|
||||
kind: CronJob
|
||||
metadata:
|
||||
name: nextcloud-db-exporter-grant
|
||||
namespace: nextcloud-db
|
||||
labels:
|
||||
app.kubernetes.io/name: nextcloud-db-exporter-grant
|
||||
app.kubernetes.io/part-of: nextcloud-db
|
||||
spec:
|
||||
# Every 15 minutes, matching the keycloak reconciler's cadence. A converged
|
||||
# database makes this one idempotent GRANT and exits; the cost is negligible, and
|
||||
# after a rebuild or PITR restore monitoring is blind for at most one interval
|
||||
# (well inside the alert's `for: 6h`).
|
||||
schedule: "*/15 * * * *"
|
||||
# A run still waiting on the primary must not be lapped by the next tick.
|
||||
concurrencyPolicy: Forbid
|
||||
startingDeadlineSeconds: 300
|
||||
successfulJobsHistoryLimit: 1
|
||||
failedJobsHistoryLimit: 3
|
||||
jobTemplate:
|
||||
spec:
|
||||
backoffLimit: 2
|
||||
activeDeadlineSeconds: 300
|
||||
ttlSecondsAfterFinished: 3600
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: nextcloud-db-exporter-grant
|
||||
app.kubernetes.io/part-of: nextcloud-db
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
# Talks only to PostgreSQL; its one input is a mounted Secret, so it needs
|
||||
# no Kubernetes API access at all (R-SEC-02, least privilege).
|
||||
automountServiceAccountToken: false
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
# uid/gid 26 is `postgres` in the CNPG PostgreSQL image.
|
||||
runAsUser: 26
|
||||
runAsGroup: 26
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: grant
|
||||
# The SAME image the instances above run, pinned to the same tag. psql
|
||||
# is already in it and it is already present on every node, so this
|
||||
# adds no new image and no new supply-chain surface. Keep the tag in
|
||||
# step with spec.imageName above when Renovate bumps it.
|
||||
image: ghcr.io/cloudnative-pg/postgresql:16.4
|
||||
imagePullPolicy: IfNotPresent
|
||||
command:
|
||||
- psql
|
||||
# -w never prompts; -X skips ~/.psqlrc; ON_ERROR_STOP makes the Job
|
||||
# FAIL LOUDLY rather than reporting success on a refused grant.
|
||||
- "-w"
|
||||
- "-X"
|
||||
- "-v"
|
||||
- "ON_ERROR_STOP=1"
|
||||
- "-f"
|
||||
- /etc/grant/grant.sql
|
||||
env:
|
||||
# libpq connection, assembled from the owner Secret. Target the -rw
|
||||
# Service: GRANT is a write and must reach the primary.
|
||||
- name: PGHOST
|
||||
value: nextcloud-db-rw.nextcloud-db.svc.cluster.local
|
||||
- name: PGPORT
|
||||
value: "5432"
|
||||
# Connecting to the application database itself is deliberate: the
|
||||
# owner has CONNECT on it, and a successful connection is the same
|
||||
# check the exporter was failing.
|
||||
- name: PGDATABASE
|
||||
value: nextcloud
|
||||
- name: PGUSER
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: nextcloud-db-app
|
||||
key: username
|
||||
# Read by libpq from the environment and never logged; psql echoes
|
||||
# no connection parameters.
|
||||
- name: PGPASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: nextcloud-db-app
|
||||
key: password
|
||||
- name: PGCONNECT_TIMEOUT
|
||||
value: "10"
|
||||
volumeMounts:
|
||||
- name: sql
|
||||
mountPath: /etc/grant
|
||||
readOnly: true
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 32Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
volumes:
|
||||
- name: sql
|
||||
configMap:
|
||||
name: nextcloud-db-exporter-grant
|
||||
defaultMode: 0444
|
||||
---
|
||||
# The desired state, as SQL. Idempotent: re-granting an existing privilege is a
|
||||
# no-op, so a converged run changes nothing. The DO block is the ASSERTION — it
|
||||
# turns "the grant did not take" into a failed Job instead of a silent success.
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: nextcloud-db-exporter-grant
|
||||
namespace: nextcloud-db
|
||||
labels:
|
||||
app.kubernetes.io/name: nextcloud-db-exporter-grant
|
||||
app.kubernetes.io/part-of: nextcloud-db
|
||||
data:
|
||||
grant.sql: |
|
||||
-- The ONE privilege the CNPG metrics exporter is missing on this database.
|
||||
-- CONNECT only: no CREATE, no TEMPORARY, no ownership, no table/schema grants.
|
||||
-- Read access for the collector's pg_catalog queries already comes from the
|
||||
-- built-in pg_monitor role, which CNPG grants this role when it creates it.
|
||||
GRANT CONNECT ON DATABASE nextcloud TO cnpg_metrics_exporter;
|
||||
|
||||
-- Fail the Job if the privilege is somehow still absent, so a broken
|
||||
-- reconciler surfaces as a failed Job rather than as a quiet blind spot.
|
||||
-- NO PL/pgSQL, AND NOT ONE DOLLAR SIGN IN THIS FILE — DELIBERATE, including in these
|
||||
-- comments. This assertion began life as a dollar-quoted DO block, which cannot survive
|
||||
-- being shipped this way: the ConfigMap is rendered through Flux's postBuild
|
||||
-- substitution, and envsubst collapses a doubled dollar into a single one. The block
|
||||
-- reached Postgres with its opening quote halved and every Flux-reconciled run died
|
||||
-- with a syntax error pointing at that character.
|
||||
-- It PASSED when hand-applied with kubectl, because that path runs no envsubst — so
|
||||
-- the defect was invisible until Flux owned the object, and three runs of Error pods
|
||||
-- were the first sign. Escaping by doubling again would work and would also be the
|
||||
-- next person's trap; plain SQL leaves substitution nothing to touch.
|
||||
--
|
||||
-- has_database_privilege() returns boolean; ::int is 1 when granted and 0 when not, so
|
||||
-- the division raises division_by_zero and ON_ERROR_STOP=1 fails the Job. The divisor
|
||||
-- is not a constant, so it cannot be folded away at plan time.
|
||||
\echo 'asserting cnpg_metrics_exporter has CONNECT on database nextcloud'
|
||||
SELECT 1 / has_database_privilege('cnpg_metrics_exporter', 'nextcloud', 'CONNECT')::int
|
||||
AS exporter_can_connect;
|
||||
@@ -1,196 +0,0 @@
|
||||
---
|
||||
# =================================================================================================
|
||||
# PiHole DNS EXPOSURE — the LAN half of split-horizon (R-DNS-01 / R-DNS-03), and NETWORK POLICY.
|
||||
# =================================================================================================
|
||||
# HAND-WRITTEN, NOT RENDERED. This file is a non-catalog app-tier unit anchored to the `pihole`
|
||||
# ServiceDefinition and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS
|
||||
# table (the same mechanism gitea-actions uses). It is hand-written because the ServiceDefinition
|
||||
# schema genuinely cannot express it: schemas/servicedefinition.schema.json is
|
||||
# `additionalProperties: false` and has NO field for a type=LoadBalancer Service, for a pinned
|
||||
# lbipam VIP, for a sharing key, for loadBalancerSourceRanges, or for UDP north-south exposure at
|
||||
# all. `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator and
|
||||
# cannot carry DNS. The two existing L4 workloads in this repo are hand-written for exactly this
|
||||
# reason and in exactly this shape — apps/vms/unifi/50-service-l4.yaml and
|
||||
# platform/crafty/service-l4.yaml. The ANSWERS PiHole gives are declared in the catalog
|
||||
# (catalog/pihole.yaml `env`), so the only thing living outside the catalog is the exposure the
|
||||
# schema has no word for.
|
||||
#
|
||||
# WHY THIS EXISTS AT ALL: platform/coredns/coredns-custom.yaml is explicit that its split-horizon is
|
||||
# for PODS and defers LAN clients to "PiHole split-horizon" — but PiHole served no DNS. Its Service
|
||||
# was ClusterIP with ports 80 and 4180 only. The LAN half of the split-horizon that docs/09
|
||||
# (R-DNS-03) and the README's Phase 2 row both claim simply did not exist, and LAN clients ran on
|
||||
# hand-maintained per-client dnsmasq files instead. Upstream cannot substitute: the SOHO router that
|
||||
# owns this LAN strips RFC1918 answers for public domains as rebind protection (NOERROR, ANSWER 0).
|
||||
#
|
||||
# VIP: ${PIHOLE_DNS_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.56, prod 192.168.2.23), PINNED via
|
||||
# lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
|
||||
# (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the
|
||||
# existing CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config
|
||||
# is needed; these Services just consume the pool. PINNING IS NOT OPTIONAL HERE: this address is
|
||||
# typed into every LAN client's resolver configuration (or handed out by the router's DHCP), so it
|
||||
# can never be allowed to move.
|
||||
#
|
||||
# ONE VIP, TWO SERVICES: a LoadBalancer Service is single-protocol-friendly, so udp/53 and tcp/53
|
||||
# split into two Services sharing one address via lbipam.cilium.io/sharing-key — the same
|
||||
# arrangement unifi and crafty use. BOTH PROTOCOLS ARE REQUIRED, not belt-and-braces: UDP is the
|
||||
# normal query path, and TCP is what a resolver falls back to for any answer that does not fit in a
|
||||
# UDP datagram and what some clients use unconditionally. A UDP-only resolver fails intermittently
|
||||
# and confusingly.
|
||||
#
|
||||
# NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
|
||||
# `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their address as a PUBLIC
|
||||
# Cloudflare record. This Service must not: `pihole-dns.${BASE_DOMAIN}` is answered by PiHole itself
|
||||
# (catalog/pihole.yaml's dnsmasq override), it is an RFC1918 address that is useless in public DNS,
|
||||
# and publishing a resolver's address in a public zone invites exactly the traffic the source-range
|
||||
# filter below exists to refuse. Internal name, internal answer, no public record — which is also
|
||||
# why this change touches nothing under platform/externaldns/.
|
||||
#
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# externalTrafficPolicy: Cluster — AND WHY IT IS NOT `Local`, WHICH IS WHAT unifi AND crafty USE.
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# `Local` is the obvious choice: it preserves the client source IP and, in theory, pins the VIP to
|
||||
# the node running the backend. The second half is NOT TRUE on this cluster, and that was measured,
|
||||
# not assumed (Cilium 1.20.1, prod, 2026-09-14):
|
||||
#
|
||||
# * Cilium's L2 announcer elects one lease holder per Service among the nodes matching the
|
||||
# CiliumL2AnnouncementPolicy's nodeSelector — `default-l2` matches ALL linux nodes — and it does
|
||||
# NOT filter that candidate set by which nodes hold a backend. Created fresh, the two Services
|
||||
# landed on DIFFERENT nodes (`cilium-l2announce-pihole-pihole-dns-tcp` -> k3s-prod-3,
|
||||
# `-udp` -> k3s-prod-2) while the pihole pod ran on k3s-prod-1.
|
||||
# * With `Local`, a node that announces the VIP but holds no backend has nowhere to send the
|
||||
# packet, and the query fails with ICMP port unreachable (dig: "connection refused"). On three
|
||||
# nodes with one replica that is a 1-in-3 chance of working, drawn independently per protocol.
|
||||
# * It does not self-heal. The lease holder keeps RENEWING after the pod moves away: a pod
|
||||
# rescheduled from k3s-prod-1 to k3s-prod-2 left the lease on k3s-prod-1, still renewing three
|
||||
# minutes later, with LAN resolution dead the whole time. Only deleting the Lease forced a
|
||||
# re-election — and that re-election is random, so it may pick a backend-less node again.
|
||||
#
|
||||
# A resolver that works one time in three and stays broken after any reschedule is not a resolver.
|
||||
# `Cluster` makes every node able to serve the VIP by forwarding to wherever the pod is, which is
|
||||
# what makes this survive a reschedule and a rebuild. VERIFIED in exactly the broken arrangement
|
||||
# above — leases on k3s-prod-2 and k3s-prod-3, pod on k3s-prod-1 — udp/53, tcp/53 and recursion all
|
||||
# answering.
|
||||
#
|
||||
# THE COST, STATED PLAINLY: Cilium runs in SNAT mode here (no `loadBalancer.mode: dsr` in
|
||||
# platform/cilium/helmrelease.yaml), so a query forwarded between nodes reaches PiHole with the
|
||||
# INGRESS NODE's address as its source, not the client's. PiHole's per-client query log, per-client
|
||||
# groups and per-client rate limiting therefore see one apparent client instead of the LAN. Two
|
||||
# consequences are handled: the rate limit is disabled in catalog/pihole.yaml (otherwise the whole
|
||||
# LAN counts as one client against PiHole's 1000-queries-per-60s default and PiHole starts
|
||||
# REFUSING), and the LAN restriction is enforced in the datapath rather than by source identity —
|
||||
# see loadBalancerSourceRanges below. The fix that would restore client IPs is `loadBalancer.mode:
|
||||
# dsr` on the Cilium HelmRelease, which is a platform-wide change with an Ansible [parity] contract
|
||||
# and would also fix the same latent defect in unifi and crafty; it belongs in its own change.
|
||||
#
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# loadBalancerSourceRanges — THE LAN RESTRICTION, ENFORCED IN THE DATAPATH.
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# Cilium enforces this list in eBPF on the ingress node, on the VIP, BEFORE load-balancing and
|
||||
# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} is dropped by the
|
||||
# datapath — it never reaches a backend and never reaches a network policy. This is what makes
|
||||
# "LAN clients only" a real, tested restriction rather than a comment: VERIFIED by narrowing the
|
||||
# range to a /24 that excludes the test client, after which the same query that had been answering
|
||||
# stopped being answered at all, and restoring it, after which it answered again.
|
||||
#
|
||||
# It is also the layer that survives the SNAT above. A CiliumNetworkPolicy cannot express this
|
||||
# restriction once the source has been rewritten: measured on this cluster, the forwarded query
|
||||
# arrives from the ingress node's internal router address and Cilium resolves that source to the
|
||||
# reserved `world` identity, which a `fromCIDR` clause does not match even when the CIDR covers the
|
||||
# address (tried: fromCIDR ${CLUSTER_POD_CIDR}, still dropped as `world`). So the peer restriction
|
||||
# lives here, where it is actually enforceable, and the policy below restricts the PORTS.
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: pihole-dns-udp
|
||||
namespace: pihole
|
||||
labels:
|
||||
app.kubernetes.io/name: pihole
|
||||
annotations:
|
||||
lbipam.cilium.io/sharing-key: "pihole-dns"
|
||||
lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}"
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Cluster
|
||||
loadBalancerSourceRanges:
|
||||
- ${LAN_CIDR}
|
||||
selector:
|
||||
app.kubernetes.io/name: pihole
|
||||
ports:
|
||||
- { name: dns-udp, port: 53, targetPort: 53, protocol: UDP }
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: pihole-dns-tcp
|
||||
namespace: pihole
|
||||
labels:
|
||||
app.kubernetes.io/name: pihole
|
||||
annotations:
|
||||
lbipam.cilium.io/sharing-key: "pihole-dns" # same key + same IP => shares the VIP with -udp
|
||||
lbipam.cilium.io/ips: "${PIHOLE_DNS_VIP}"
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Cluster
|
||||
loadBalancerSourceRanges:
|
||||
- ${LAN_CIDR}
|
||||
selector:
|
||||
app.kubernetes.io/name: pihole
|
||||
ports:
|
||||
- { name: dns-tcp, port: 53, targetPort: 53, protocol: TCP }
|
||||
---
|
||||
# =================================================================================================
|
||||
# NETWORK POLICY — a NEW ingress path from off-cluster, restricted to the two DNS ports.
|
||||
# =================================================================================================
|
||||
# The pihole namespace is default-deny on ingress (the baseline NetworkPolicy in the rendered
|
||||
# apps/wave-0/pihole.yaml: `ingress: []`). Cilium enforces the UNION of that NetworkPolicy and the
|
||||
# CiliumNetworkPolicies in the namespace, so this file ADDS one path and nothing else. Everything
|
||||
# undeclared stays denied, and the existing `pihole-allow-gateway-ingress` (Gateway -> :4180, the
|
||||
# oauth2-proxy gate in front of the admin UI) is untouched. PiHole's own :80 is NOT in this rule:
|
||||
# the admin UI stays reachable only through the Gateway and oauth2-proxy, and a LAN client that
|
||||
# talks to ${PIHOLE_DNS_VIP} on any port other than 53 gets nothing — there is no Service port for
|
||||
# it and no policy for it.
|
||||
#
|
||||
# PORTS ARE THE NARROWING HERE; THE LAN RESTRICTION IS THE SOURCE-RANGE FILTER ABOVE. The peer is
|
||||
# `world` for a mechanical reason, not a lazy one: with externalTrafficPolicy Cluster in Cilium's
|
||||
# SNAT mode a forwarded query arrives from the ingress node's internal router address, which Cilium
|
||||
# resolves to the reserved `world` identity — a `fromCIDR` clause does not match it, which was
|
||||
# measured (a fromCIDR of ${CLUSTER_POD_CIDR}, which contains that address, still produced
|
||||
# "Policy denied ... identity world" drops in `cilium-dbg monitor`). So `world` is the only peer
|
||||
# selector that can describe LAN traffic that has been through the LoadBalancer, and the set of
|
||||
# addresses that can actually BE that traffic is fixed by loadBalancerSourceRanges before this
|
||||
# policy is consulted. Together: ${LAN_CIDR} in the datapath, udp/53 + tcp/53 here.
|
||||
#
|
||||
# This is NOT wider than the precedent it follows. unifi-allow-l4-ingress and crafty-allow-ingress
|
||||
# admit `fromEntities: [world, cluster, host]` on their L4 ports with NO source-range filter at all;
|
||||
# this rule admits fewer entities (no `cluster` — in-cluster pods resolve through CoreDNS and none
|
||||
# needs PiHole on :53) on fewer ports, behind a filter those two do not have.
|
||||
#
|
||||
# WHAT REMAINS REACHABLE WITHOUT THE VIP, ACKNOWLEDGED: Cilium applies source ranges to the
|
||||
# LoadBalancer address, not to the NodePorts the same Services allocate, so a host that can route to
|
||||
# a node IP can still reach udp/53 and tcp/53 through the NodePort. On this LAN that is the same
|
||||
# population the source range admits, and nothing at the edge forwards 53 inward, so PiHole is not
|
||||
# an internet-facing open resolver. Say so out loud rather than implying the filter is total.
|
||||
#
|
||||
# `host` and `remote-node` are named explicitly because node-sourced traffic carries a reserved
|
||||
# Cilium identity that no CIDR or `world` selector matches: the kubelet's probes and the
|
||||
# node-to-node forwarding that IS the Cluster-policy datapath both need them.
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: pihole-allow-lan-dns-ingress
|
||||
namespace: pihole
|
||||
labels:
|
||||
app.kubernetes.io/name: pihole
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: pihole
|
||||
ingress:
|
||||
- fromEntities:
|
||||
- world # LAN clients, post-LoadBalancer: see the block above for why not fromCIDR
|
||||
- host # this node's kubelet probes
|
||||
- remote-node # the node-to-node hop that forwards a query to wherever the pod runs
|
||||
toPorts:
|
||||
- ports:
|
||||
- { port: "53", protocol: UDP }
|
||||
- { port: "53", protocol: TCP }
|
||||
@@ -1,30 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: pihole-oauth2-proxy
|
||||
namespace: pihole
|
||||
type: Opaque
|
||||
stringData:
|
||||
client-id: ENC[AES256_GCM,data:C6IXgQxi,iv:A9bWygwabOKIRi55jV9+TRU7eiZels3vqqLDTZ469aA=,tag:hC7sn/Xr4vQFkRjDLMgEmQ==,type:str]
|
||||
client-secret: ENC[AES256_GCM,data:50HEfMG+CoWXokS+IxQgCPQEt+EBCUontOc0GrTyauHuV/U7w9IPo83iJECO0QhTPiQIgD3j+F+YXfubCbTnuw==,iv:ejQxin0NMs0Ye0fhqfPuh8JmRrEimDrSAcZj28uS+GM=,tag:FVassLZPZMY9ry48ZjBgmw==,type:str]
|
||||
cookie-secret: ENC[AES256_GCM,data:nfTEg5DlIXCLI/hhHngyqLwsQh2SgXU21LaSkVnr91sTi4wODrEH4C7wvPY=,iv:WWXnHnFP2IqLrw1yXAlD0Sw+Vc1m4rxXIHT1u2UWH3o=,tag:pYZEuwPYH+b8S4kGtaWqEA==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBuTnJiWTVPR01aY2h3dkJI
|
||||
OTV0WTFQeHoyUGluVzV4NFpSZjd6bjlHZkE0ClZmTGgzQWo4VTROaUg5Wkd1TTVV
|
||||
cGRjZjJuMitPZC9OaVZUZzRGeEJmemcKLS0tIGRjUFd3SzFzeFBBQVRacjZWQmNH
|
||||
ZjllZTQxbzRWQXNiVllkc1hESGN4alEK/hFOtchsYMx09/WE5uaCe6JetUurzhhQ
|
||||
7imb8RIjh0cfP4PDT3dZhhPF+ehcV7ttslFkcyaqlG1F8Gk2jtN0pw==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-08-31T13:33:15Z"
|
||||
mac: ENC[AES256_GCM,data:Tse6fLhc08DxCAuZW5KSZ3M5OQUTlbeHnWiO3jIROUVLrBfbXCuDNWnQyOPjwVPqyzeAzC3APEKFzQ2S8rYjRyO08SJ4ZFPPUeRblcXgXHZ4Y/E4Renq7axk61VvFGtZteHWuQz9ss//Qt+KwPMDBGTivLCoVTOh2K01AXPFXdA=,iv:fyhkDLVN28umXDR4/f4ZzQuu3OG6QBkpUgf3E9rknKk=,tag:iNqor6cD9tJ6MsqAmmRP0Q==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
+25
-86
@@ -81,82 +81,9 @@ spec:
|
||||
image: pihole/pihole
|
||||
ports:
|
||||
- containerPort: 80
|
||||
env:
|
||||
- name: FTLCONF_dns_listeningMode
|
||||
value: ALL
|
||||
- name: FTLCONF_dns_upstreams
|
||||
value: 8.8.8.8;8.8.4.4
|
||||
- name: FTLCONF_dns_rateLimit_count
|
||||
value: '0'
|
||||
- name: FTLCONF_misc_dnsmasq_lines
|
||||
value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /data/config
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
- --reverse-proxy=true
|
||||
- --provider=oidc
|
||||
- --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
|
||||
- --upstream=http://127.0.0.1:80
|
||||
- --email-domain=*
|
||||
- --scope=openid email profile
|
||||
- --oidc-groups-claim=groups
|
||||
- --allowed-group=platform-admins
|
||||
- --skip-provider-button=true
|
||||
- --redirect-url=https://pihole.${BASE_DOMAIN}/oauth2/callback
|
||||
- --cookie-domain=.${BASE_DOMAIN}
|
||||
- --whitelist-domain=.${BASE_DOMAIN}
|
||||
- --cookie-secure=true
|
||||
- --pass-access-token=true
|
||||
- --set-xauthrequest=true
|
||||
env:
|
||||
- name: OAUTH2_PROXY_CLIENT_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: pihole-oauth2-proxy
|
||||
key: client-id
|
||||
- name: OAUTH2_PROXY_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: pihole-oauth2-proxy
|
||||
key: client-secret
|
||||
- name: OAUTH2_PROXY_COOKIE_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: pihole-oauth2-proxy
|
||||
key: cookie-secret
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 4180
|
||||
protocol: TCP
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
volumes:
|
||||
- name: config
|
||||
persistentVolumeClaim:
|
||||
@@ -177,9 +104,6 @@ spec:
|
||||
- name: port-80
|
||||
port: 80
|
||||
targetPort: 80
|
||||
- name: authproxy
|
||||
port: 4180
|
||||
targetPort: 4180
|
||||
---
|
||||
# CiliumNetworkPolicy gateway-ingress allow (R-NET-01/02): pihole-allow-gateway-ingress
|
||||
apiVersion: cilium.io/v2
|
||||
@@ -198,7 +122,7 @@ spec:
|
||||
- ingress
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '4180'
|
||||
- port: '80'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): pihole-allow-internet-egress
|
||||
@@ -216,10 +140,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute pihole-80 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
@@ -232,14 +152,13 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- pihole.${BASE_DOMAIN}
|
||||
- pihole.home.keithjackson.dev
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
@@ -247,7 +166,27 @@ spec:
|
||||
value: /
|
||||
backendRefs:
|
||||
- name: pihole
|
||||
port: 4180
|
||||
port: 80
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: pihole
|
||||
namespace: pihole
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: pihole
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: pihole
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- pihole
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
@@ -263,11 +202,11 @@ spec:
|
||||
- name: pihole.slis
|
||||
rules:
|
||||
- alert: pihole-down
|
||||
expr: kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"} == 0 or absent(kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"})
|
||||
expr: up{job="pihole"} == 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: pihole
|
||||
annotations:
|
||||
summary: pihole is down
|
||||
description: 'pihole: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
|
||||
description: 'pihole has been unreachable for 5m (SLI: up).'
|
||||
@@ -1,69 +0,0 @@
|
||||
# Plex plex.tv server-claim token (WS-APPS). SOPS-encrypted (age).
|
||||
# Consumed by catalog/plex.yaml as env PLEX_CLAIM via valueFrom.secretKeyRef.
|
||||
#
|
||||
# NOTE ON THESE COMMENTS: they carry NO dollar-brace expressions, deliberately. This file is applied
|
||||
# by the apps-wave-0 Flux Kustomization, which runs envsubst over the built output, and kustomize
|
||||
# preserves body comments into that stream (see conformance/tests/l0-static/flux-postbuild-
|
||||
# substitution). A braced token quoted from the upstream shell script would be a live hazard here,
|
||||
# not prose, so the upstream conditions below are described in words.
|
||||
#
|
||||
# THE VALUE SHIPPED WITH THIS FILE IS A PLACEHOLDER, DELIBERATELY — the literal string
|
||||
# `replace-me-with-a-fresh-plex-tv-claim-token`, not a credential. A https://plex.tv/claim token has
|
||||
# a ~4-minute TTL, far shorter than commit -> push -> Flux reconcile -> pod-roll, so no token
|
||||
# committed here is ever still valid when the pod starts. This file exists to make the Secret — and
|
||||
# therefore the env reference — ALWAYS RESOLVABLE IN THE FLUX GRAPH, not to carry a working token.
|
||||
# The real token is injected by the operator inside that 4-minute window: runbooks/plex-claim.md.
|
||||
#
|
||||
# WHY SHIPPING A PLACEHOLDER IS SAFE (verified against plexinc/pms-docker branch `master`, the
|
||||
# s6-overlay v2 script root/etc/cont-init.d/40-plex-first-run, cross-checked byte-identical against
|
||||
# the same file extracted from the running `plexinc/pms-docker:latest` image):
|
||||
# - line 106 guards the claim exchange on TWO conditions: PLEX_CLAIM non-empty AND the
|
||||
# PlexOnlineToken pref read out of /config empty. Once the server is claimed that block is
|
||||
# skipped entirely, so whatever this Secret holds is a NO-OP on every later start. Upstream
|
||||
# README: "If server is already logged in, this parameter is ignored."
|
||||
# - on a still-UNCLAIMED /config, an invalid or expired token makes plex.tv answer 403 with an
|
||||
# empty body. curl's exit status is never checked, the sed for the authentication-token element
|
||||
# yields the empty string, and the success branch at line 120 simply does not fire. There is no
|
||||
# else branch and no non-zero exit; the script has no `set -e` and ends by touching
|
||||
# /.firstRunComplete and echoing, so it exits 0. The server comes up UNCLAIMED and serving.
|
||||
# It does NOT crashloop.
|
||||
# - the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS, whose default is 0 ("continue silently even
|
||||
# if a cont-init script failed"), so even a hypothetical non-zero exit would not halt startup.
|
||||
# A MISSING Secret would NOT be safe: the ServiceDefinition schema's valueFrom.secretKeyRef shape
|
||||
# has no `optional` field, so an absent Secret is a hard CreateContainerConfigError. That is exactly
|
||||
# why this file is committed and listed in apps/wave-0/kustomization.yaml instead of being created
|
||||
# by hand at claim time.
|
||||
#
|
||||
# INJECTION / ROTATION (operator, not agents) — runbooks/plex-claim.md has the timed procedure:
|
||||
# infra/devcontainer/dev.sh run sops apps/wave-0/plex-claim-secret.sops.yaml
|
||||
# Never decrypt this file into a shell, a log, or a shared terminal.
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: plex-claim
|
||||
namespace: plex
|
||||
labels:
|
||||
app.kubernetes.io/name: plex
|
||||
type: Opaque
|
||||
stringData:
|
||||
PLEX_CLAIM: ENC[AES256_GCM,data:nLmzeUBWRhRjUakrGiGzQpQuqZNcIYvRH6M=,iv:z0on/dArsNejNLrp3V1nSiNLFzKcREhPuMO1634CwQw=,tag:Ih23CT1JYna/5lfYQUb1Dw==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2dWtqOVNQalZ2QlRxOXJa
|
||||
cDc1MTdtWENkdWIxS1JId3dIU1JDSmk0ODNrCk1YUzhFWXR3bUxyV2NNcmFOakdz
|
||||
THl1Q284dFJxTy9Rd1h0eXJJaVZtdmcKLS0tIERuN0ZFeW9zdi9aNFRYNWxmUS9s
|
||||
QUtYdHEvN042ajNoNktOLzlYQmQrc1kK4NHozMbXVzNGYbUBRbK6a9Uw/sVPAfe6
|
||||
RW5KWzZ4I/hloqjN3zzVISoSMieMkBGDnF5qZS5N6FqIc6/HrsDD8A==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-09-29T06:31:16Z"
|
||||
mac: ENC[AES256_GCM,data:62BmBAq45CD2g6LDyx+Ho4bTfrhJcaro/RB/iM+OaRc4a/D83MhjbcBL11Uz/O726inqhkvCKDJ/fFemoc50JDbtC79cYMaxrGJNjFM0sdckWrowPvh6ie7HHHUBRBaefl+gNfnc+pbvcncvRzwA4RyzcZse1wbxKfHN8FjEopE=,iv:h0cMShs3tIgnY6eruUYymYrcLFzQZLk/uXQJ/Jsd5ek=,tag:nZ9Pau7p4xWT1QCN1vv2vA==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
+23
-59
@@ -35,34 +35,6 @@ spec:
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
---
|
||||
# PersistentVolume plex-media - static external NFS (bulk tier)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: plex-media
|
||||
labels:
|
||||
app.kubernetes.io/name: plex
|
||||
spec:
|
||||
capacity:
|
||||
storage: 9Ti
|
||||
accessModes:
|
||||
- ReadOnlyMany
|
||||
persistentVolumeReclaimPolicy: Retain
|
||||
storageClassName: ''
|
||||
mountOptions:
|
||||
- nfsvers=4.1
|
||||
- hard
|
||||
- noatime
|
||||
- timeo=600
|
||||
- retrans=2
|
||||
nfs:
|
||||
server: ${BULK_NFS_SERVER}
|
||||
path: /mnt/Storage/plex
|
||||
readOnly: true
|
||||
claimRef:
|
||||
namespace: plex
|
||||
name: plex-media
|
||||
---
|
||||
# PVC plex-config (R-STATE)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
@@ -79,23 +51,6 @@ spec:
|
||||
requests:
|
||||
storage: 8Gi
|
||||
---
|
||||
# PVC plex-media (R-STATE)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: plex-media
|
||||
namespace: plex
|
||||
labels:
|
||||
app.kubernetes.io/name: plex
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadOnlyMany
|
||||
storageClassName: ''
|
||||
volumeName: plex-media
|
||||
resources:
|
||||
requests:
|
||||
storage: 9Ti
|
||||
---
|
||||
# Deployment (R-PLAT-02)
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
@@ -132,24 +87,13 @@ spec:
|
||||
value: Etc/UTC
|
||||
- name: ALLOWED_NETWORKS
|
||||
value: 10.0.0.0/8,192.168.0.0/16
|
||||
- name: PLEX_CLAIM
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: plex-claim
|
||||
key: PLEX_CLAIM
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /config
|
||||
- name: media
|
||||
mountPath: /media
|
||||
readOnly: true
|
||||
volumes:
|
||||
- name: config
|
||||
persistentVolumeClaim:
|
||||
claimName: plex-config
|
||||
- name: media
|
||||
persistentVolumeClaim:
|
||||
claimName: plex-media
|
||||
---
|
||||
# Service
|
||||
apiVersion: v1
|
||||
@@ -239,7 +183,7 @@ spec:
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- plex.${BASE_DOMAIN}
|
||||
- plex.home.keithjackson.dev
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
@@ -249,6 +193,26 @@ spec:
|
||||
- name: plex
|
||||
port: 32400
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: plex
|
||||
namespace: plex
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: plex
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: plex
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- plex
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
@@ -263,11 +227,11 @@ spec:
|
||||
- name: plex.slis
|
||||
rules:
|
||||
- alert: plex-down
|
||||
expr: kube_deployment_status_replicas_available{namespace="plex", deployment="plex"} == 0 or absent(kube_deployment_status_replicas_available{namespace="plex", deployment="plex"})
|
||||
expr: up{job="plex"} == 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: plex
|
||||
annotations:
|
||||
summary: plex is down
|
||||
description: 'plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
|
||||
description: 'plex has been unreachable for 5m (SLI: up).'
|
||||
@@ -1,60 +0,0 @@
|
||||
# qBittorrent DURABLE WebUI credential (WS-APPS). SOPS-encrypted (age), decrypted in-cluster by Flux.
|
||||
#
|
||||
# WHY THIS FILE EXISTS (defect, measured on prod 2026-09-13):
|
||||
# qBittorrent.conf on the config PVC carried NEITHER `WebUI\Username` NOR
|
||||
# `WebUI\Password_PBKDF2`. qBittorrent 5.x treats that as "no credential set" and generates a NEW
|
||||
# random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary
|
||||
# password is provided for this session: ..."). Consequences, both real: (a) Sonarr's and Radarr's
|
||||
# saved download-client credentials stop working at the next qbittorrent pod restart, silently
|
||||
# breaking Completed Download Handling; (b) a human operator has no stable login at all and must
|
||||
# read the pod log after every restart.
|
||||
#
|
||||
# HOW IT IS APPLIED: catalog/qbittorrent.yaml's app container reads BOTH keys below as env
|
||||
# (QBT_WEBUI_USERNAME / QBT_WEBUI_PASSWORD) and its `lifecycle.postStart` hook — the same idiom as
|
||||
# gitea's break-glass admin bootstrap — DERIVES the PBKDF2-HMAC-SHA512 digest qBittorrent stores
|
||||
# and asserts both keys into /config/qBittorrent/qBittorrent.conf on EVERY container start, then
|
||||
# verifies the WebUI actually accepts them. Level-triggered: a pod restart, a config-PVC restore
|
||||
# from an older backup, and a rebuild-from-git onto an EMPTY PVC all end with this SAME
|
||||
# credential, because the hook re-asserts it on every start rather than once.
|
||||
#
|
||||
# webui-username: the WebUI account name. qBittorrent has exactly ONE local account, and `admin` is
|
||||
# the name its own temporary-password path uses, so keeping it avoids inventing a second identity.
|
||||
# webui-password: the durable plaintext. It is ALSO what Sonarr's and Radarr's download-client
|
||||
# entries store in their own databases (app state on their config PVCs, NOT in git — see
|
||||
# DECISIONS-LOG), so rotating it here means re-pointing both arrs.
|
||||
# Deliberately alphanumeric: it travels through a urlencoded WebAPI login body and two arr JSON
|
||||
# payloads, and a shell/URL-special character in any of those is a bug waiting to happen.
|
||||
#
|
||||
# NEVER put a value in a comment: SOPS does NOT encrypt comments.
|
||||
# Recover with: infra/devcontainer/dev.sh run sops -d apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: qbittorrent-bootstrap
|
||||
namespace: qbittorrent
|
||||
labels:
|
||||
app.kubernetes.io/name: qbittorrent
|
||||
type: Opaque
|
||||
stringData:
|
||||
webui-username: ENC[AES256_GCM,data:P022nG0=,iv:MXcCXsD5hs7MAJ+OAYHkZ3pXep7XGPqeAExChoqna08=,tag:WE34mxtfQRgiwPRaDA12pQ==,type:str]
|
||||
webui-password: ENC[AES256_GCM,data:+Lx0Ny7SBZeF/CzQYm8umntw27ewyXS7unWObpl0fCg=,iv:JkAtublBMS7SJUutAFip1XjdmsKnGe+S2E9QDT4rCJk=,tag:MYptPVgoEUEpXe5kmg1xIg==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB1aWpUY04wZHoyWVdPbGU2
|
||||
TC9vaU5oYmhWSVh3YWVTdzkvU3Axdlh6QVFZCkV4MldaZXNyam5CejQyZStyczJ3
|
||||
UStFd0hRQmRsNGZLVEVTRnVjWXp6SVkKLS0tICtkWlRqVGN5OWYvVnZUaHYvbEZv
|
||||
VWtHMVdDU3hmdXo1TndLb05zQzJ0S28KfqlDBTocO0ZAKtH6ad/00EeVRqZ7WMwv
|
||||
FP2evd5vyn95QQICWhY/LH6EKbwVVlcIlL7i9l8K3kK4Ebdk54mAyw==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-09-13T05:23:23Z"
|
||||
mac: ENC[AES256_GCM,data:hQ7jIgFqpNcFY5H4AfMxuslC2R/2wGmGWtghemOmTUUipSdoGdbNjWGQhxzk40lFjw6Zj2b+buOpXpg70s4xr+4feusqa2TJ3+bxAuRWmuZpTQAU2OaO5v0SdDEs3gwKm2mA3JFo/e6POm/P/kf2GIECcyU1AciZQt6ws7UJmQc=,iv:6SF0ZAuzHQThD/FGbWhFfa4dXU8hyW8jWFiDmwgfFU8=,tag:31i8HM2s/mGeS4zvFxO53Q==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
@@ -1,30 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: qbittorrent-oauth2-proxy
|
||||
namespace: qbittorrent
|
||||
type: Opaque
|
||||
stringData:
|
||||
client-id: ENC[AES256_GCM,data:ytfVRDabEfRM8NI=,iv:ERtFy5iss10id/cGLMZDmIAazkYtFG/spkxRYkoT2pw=,tag:FzOepKeYT05BBrz6RlnACQ==,type:str]
|
||||
client-secret: ENC[AES256_GCM,data:klZzTdyAqQgG+1s074/0Bh5ybuHehiEo/RYceRGAGwuUrgCMb0AmYHCrE9gvtnPkSOBqWvFnPnorWkERc+V73g==,iv:nOUO/p+WnI5657I3Lm6zQAsumkscjF7J0fK0EVFHGo0=,tag:dF3AHYP4sxDChroRer9kMQ==,type:str]
|
||||
cookie-secret: ENC[AES256_GCM,data:jVpsZjYHRrPvblfsb2f9OuneqoZ0cwcTLKrPb8XJ5oves1PmHtN5njrkk4o=,iv:bNjUK/ar4nniR7AjJ9fsGVkKd8nxwphkptmhFaYh9Ek=,tag:2axf+AshSdIvvqwyIfhq6g==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAreWdPdUxiYXBxYWxZNlhy
|
||||
WHFTZmo1RW1McXJJTjlWeTBCd2Y4WVU2TVRNCnExbEQrQXZCS2RRaGxJQlQ5eHpY
|
||||
Z1UrZWlubUJpRC9YeEhJZW8zNVdLUVUKLS0tIGtmblFDRzk3ZC8wWVo2ZHJHWmEx
|
||||
OW9DRUpaSlFrbHNDTGYvQUdyamh2bWsK5VvV4cl1iNXYECnSk3fi1ZuAPKGN+n8B
|
||||
g51lT4gi03KA67ARzwVGVBfP5E1Cdmx4rJhi305tG+Ug99VyLBCUfQ==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-08-31T13:42:02Z"
|
||||
mac: ENC[AES256_GCM,data:7GdlUKnZ/ST7EOhfPB6I2e2wyVHWmZ2G554yBcZQzqoZZWZ1hn+lkosNJxWD7E0hRId6P320EKj+Sf0g2ur3Mna5zii1GKFJHGtDEoccKvJQRx2JMkZlEC/x6xwc4BeJNtvidECVRvKC2WdrByRMr9rrI8NfXpmWH4e7iz1Xac0=,iv:UqsNOPRwzT+Dt043ldxIubkAAkPr0/nvjxlgVPiXIbw=,tag:7Wex5Rng3Nwtr8wxkFJD9Q==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
+28
-157
@@ -35,33 +35,6 @@ spec:
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
---
|
||||
# PersistentVolume qbittorrent-downloads - static external NFS (bulk tier)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: qbittorrent-downloads
|
||||
labels:
|
||||
app.kubernetes.io/name: qbittorrent
|
||||
spec:
|
||||
capacity:
|
||||
storage: 9Ti
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
persistentVolumeReclaimPolicy: Retain
|
||||
storageClassName: ''
|
||||
mountOptions:
|
||||
- nfsvers=4.1
|
||||
- hard
|
||||
- noatime
|
||||
- timeo=600
|
||||
- retrans=2
|
||||
nfs:
|
||||
server: ${BULK_NFS_SERVER}
|
||||
path: /mnt/Storage/plex
|
||||
claimRef:
|
||||
namespace: qbittorrent
|
||||
name: qbittorrent-downloads
|
||||
---
|
||||
# PVC qbittorrent-config (R-STATE)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
@@ -78,23 +51,6 @@ spec:
|
||||
requests:
|
||||
storage: 5Gi
|
||||
---
|
||||
# PVC qbittorrent-downloads (R-STATE)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: qbittorrent-downloads
|
||||
namespace: qbittorrent
|
||||
labels:
|
||||
app.kubernetes.io/name: qbittorrent
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
storageClassName: ''
|
||||
volumeName: qbittorrent-downloads
|
||||
resources:
|
||||
requests:
|
||||
storage: 9Ti
|
||||
---
|
||||
# Deployment (R-PLAT-02)
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
@@ -118,117 +74,20 @@ spec:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
homelab.keithjackson.dev/auth-groups: media-users
|
||||
spec:
|
||||
containers:
|
||||
- name: qbittorrent
|
||||
image: lscr.io/linuxserver/qbittorrent:5.1.2
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
env:
|
||||
- name: PUID
|
||||
value: '1000'
|
||||
- name: PGID
|
||||
value: '3000'
|
||||
- name: QBT_WEBUI_USERNAME
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: qbittorrent-bootstrap
|
||||
key: webui-username
|
||||
- name: QBT_WEBUI_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: qbittorrent-bootstrap
|
||||
key: webui-password
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /config
|
||||
- name: downloads
|
||||
mountPath: /downloads
|
||||
subPath: downloads
|
||||
lifecycle:
|
||||
postStart:
|
||||
exec:
|
||||
command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the\n# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.\n#\n# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.\n# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over\n# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;\n# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE\n# CLUSTER \u2014 but only there:\n# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl\n# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during\n# development) keeps the doubled dollar, which bash reads as its own PID. Not hypothetical;\n# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`\n# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY\n# however it is run.\nset -u\nCONF=/config/qBittorrent/qBittorrent.conf\nSVC=/run/service/svc-qbittorrent\n# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this\n# ServiceDefinition \u2014 which is the single place the renderer takes the containerPort, the\n# Service port and the HTTPRoute backend port from \u2014 and the image exposes no WEBUI_PORT\n# env var for a lookup to read anyway.\n# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and\n# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,\n# so PORT became \"<pid>{WEBUI_PORT:-8080}\" \u2014 the brace never expanded, the digits never\n# matched. Every curl below then got an unparseable URL, returned nothing, and the hook\n# logged \"the WebUI rejected the durable\n# credential\" \u2014 having never actually asked it. That MISDIAGNOSIS is preserved in this\n# service's own marker log at 2026-09-13T05:35Z (three \"rejected\" lines, then GAVE UP,\n# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.\nPORT=8080\n# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` \u2014 so a hook\n# that only echoes is effectively silent (gitea's hits the same wall and writes a marker\n# file for it). Every outcome therefore also lands, timestamped, on the persistent config\n# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the\n# credential from git was asserted and accepted on this start.\nlog() {\n echo \"[qbt-webui-cred] $*\"\n printf '%s [qbt-webui-cred] %s\\n' \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\" \"$*\" \\\n >> /config/qbittorrent-webui-cred.log 2>/dev/null || true\n}\n\n# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC\n# before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of\n# racing to create one it would then not overwrite.\nn=0\nwhile [ ! -s \"$CONF\" ] && [ \"$n\" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done\n\nwaited=0\nattempt=0\nwhile [ \"$attempt\" -lt 3 ]; do\n attempt=$((attempt + 1))\n\n # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),\n # 10 = rewritten, anything else = could not write.\n python3 - \"$CONF\" <<'PY'\nimport base64, hashlib, hmac, os, stat, sys\n\nconf = sys.argv[1]\nuser = os.environ.get(\"QBT_WEBUI_USERNAME\", \"\")\npw = os.environ.get(\"QBT_WEBUI_PASSWORD\", \"\")\nif not user or not pw:\n sys.stderr.write(\"[qbt-webui-cred] credential env is empty - refusing to touch the conf\\n\")\n raise SystemExit(1)\n\n# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000\n# iterations, 16-byte random salt, 64-byte digest, stored as\n# WebUI\\Password_PBKDF2=\"@ByteArray(<base64 salt>:<base64 digest>)\"\n# Get any one of them wrong and the app simply rejects every login.\nITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16\nUKEY, PKEY, SECTION = \"WebUI\\\\Username\", \"WebUI\\\\Password_PBKDF2\"\
|
||||
, \"[Preferences]\"\n\ndef matches(raw):\n \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n v = raw.strip().strip('\"')\n if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n v = v[len(\"@ByteArray(\"):-1]\n try:\n salt_b64, digest_b64 = v.split(\":\", 1)\n salt = base64.b64decode(salt_b64, validate=True)\n digest = base64.b64decode(digest_b64, validate=True)\n except Exception:\n return False\n if not salt or not digest:\n return False\n return hmac.compare_digest(\n hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n lines = []\n\ndef value_of(key):\n for i, line in enumerate(lines):\n if line.startswith(key + \"=\"):\n return i, line.split(\"=\", 1)[1]\n return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n raise SystemExit(0) # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n at = out.index(SECTION) + 1\nelse:\n if out and out[-1].strip():\n out.append(\"\")\n out.append(SECTION)\n at = len(out)\nout[at:at] = desired\n\ntry:\n os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n tmp = conf + \".webui-cred.new\"\n with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n fh.write(\"\\n\".join(out) + \"\\n\")\n fh.flush()\n os.fsync(fh.fileno())\n # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n # cost the app the ability to save its own settings, so carry the ownership over.\n try:\n st = os.stat(conf)\n os.chown(tmp, st.st_uid, st.st_gid)\n os.chmod(tmp, stat.S_IMODE(st.st_mode))\n except FileNotFoundError:\n os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n int(os.environ.get(\"PGID\", \"1000\")))\n os.chmod(tmp, 0o664)\n os.replace(tmp, conf) # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n raise SystemExit(1)\nraise SystemExit(10)\nPY\n rc=$?\n\n if [ \"$rc\" = 0 ]; then\n log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n elif [ \"$rc\" = 10 ]; then\n log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n # have read the old, credential-less conf and be serving a random temporary password.\n # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n # Nothing to do when the service has not started yet - it will read the new file.\n # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the\n # 2026-09-13 failure spent the entire verification budget before the first login and\n # then fired attempts 2 and 3 with no wait left at all.\n sv=0\n while [ \"$sv\" -lt 60 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n sv=$((sv + 1)); sleep 1\n done\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"qbittorrent-nox was already running - restarting it to load the credential\"\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n sleep 3\n fi\n else\n log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n log \"minting a random password\
|
||||
\ every start and BOTH arrs will break. Fix and restart.\" >&2\n exit 0\n fi\n\n # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL\n # (shared ~120s budget across attempts), then spend exactly ONE login per attempt -\n # at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n #\n # \"THE API NEVER ANSWERED\" IS A DIFFERENT FAULT FROM \"THE PASSWORD WAS REJECTED\", and\n # conflating them is what made this hook lie once already (see the PORT comment). An\n # unreachable WebUI is reported as unreachable, and the hook stops rather than spending\n # its remaining login attempts to produce two more identical wrong diagnoses.\n api_ok=0\n while [ \"$waited\" -lt 120 ]; do\n code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n if [ -n \"$code\" ] && [ \"$code\" != \"000\" ]; then api_ok=1; break; fi\n waited=$((waited + 1)); sleep 1\n done\n if [ \"$api_ok\" != 1 ]; then\n log \"the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the\" >&2\n log \"credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was\" >&2\n log \"never submitted. Do not rotate the credential on the strength of this line.\" >&2\n exit 0\n fi\n body=$(curl -sS -m 10 \\\n --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n if [ \"$body\" = \"Ok.\" ]; then\n log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n exit 0\n fi\n log \"WebUI rejected the durable credential (login said '$body')\" >&2\n # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean\n # the running qbittorrent-nox is serving something it read from an EARLIER conf, so\n # rewriting the same bytes on the next attempt would change nothing. The remedy is to\n # make the process RE-READ the file, so restart the supervised service and let the next\n # attempt re-verify. Without this the retry loop could never change its own outcome.\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"restarting qbittorrent-nox so it re-reads $CONF, then re-verifying\" >&2\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\" >&2\n sleep 3\n fi\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n"
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
- --reverse-proxy=true
|
||||
- --provider=oidc
|
||||
- --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
|
||||
- --upstream=http://127.0.0.1:8080
|
||||
- --email-domain=*
|
||||
- --scope=openid email profile
|
||||
- --oidc-groups-claim=groups
|
||||
- --allowed-group=media-users
|
||||
- --allowed-group=platform-admins
|
||||
- --skip-provider-button=true
|
||||
- --redirect-url=https://qbittorrent.${BASE_DOMAIN}/oauth2/callback
|
||||
- --cookie-domain=.${BASE_DOMAIN}
|
||||
- --whitelist-domain=.${BASE_DOMAIN}
|
||||
- --cookie-secure=true
|
||||
- --pass-access-token=true
|
||||
- --set-xauthrequest=true
|
||||
env:
|
||||
- name: OAUTH2_PROXY_CLIENT_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: qbittorrent-oauth2-proxy
|
||||
key: client-id
|
||||
- name: OAUTH2_PROXY_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: qbittorrent-oauth2-proxy
|
||||
key: client-secret
|
||||
- name: OAUTH2_PROXY_COOKIE_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: qbittorrent-oauth2-proxy
|
||||
key: cookie-secret
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 4180
|
||||
protocol: TCP
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
securityContext:
|
||||
fsGroup: 3000
|
||||
mountPath: /data/config
|
||||
volumes:
|
||||
- name: config
|
||||
persistentVolumeClaim:
|
||||
claimName: qbittorrent-config
|
||||
- name: downloads
|
||||
persistentVolumeClaim:
|
||||
claimName: qbittorrent-downloads
|
||||
---
|
||||
# Service
|
||||
apiVersion: v1
|
||||
@@ -245,9 +104,6 @@ spec:
|
||||
- name: port-8080
|
||||
port: 8080
|
||||
targetPort: 8080
|
||||
- name: authproxy
|
||||
port: 4180
|
||||
targetPort: 4180
|
||||
---
|
||||
# CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): qbittorrent-allow-radarr-8080
|
||||
apiVersion: cilium.io/v2
|
||||
@@ -310,7 +166,7 @@ spec:
|
||||
- ingress
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '4180'
|
||||
- port: '8080'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): qbittorrent-allow-internet-egress
|
||||
@@ -328,10 +184,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute qbittorrent-8080 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
@@ -343,15 +195,14 @@ metadata:
|
||||
app.kubernetes.io/name: qbittorrent
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
homelab.keithjackson.dev/auth-groups: media-users
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- qbittorrent.${BASE_DOMAIN}
|
||||
- qbittorrent.home.keithjackson.dev
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
@@ -359,7 +210,27 @@ spec:
|
||||
value: /
|
||||
backendRefs:
|
||||
- name: qbittorrent
|
||||
port: 4180
|
||||
port: 8080
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: qbittorrent
|
||||
namespace: qbittorrent
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: qbittorrent
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: qbittorrent
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- qbittorrent
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
@@ -375,11 +246,11 @@ spec:
|
||||
- name: qbittorrent.slis
|
||||
rules:
|
||||
- alert: qbittorrent-down
|
||||
expr: kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"} == 0 or absent(kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"})
|
||||
expr: up{job="qbittorrent"} == 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: qbittorrent
|
||||
annotations:
|
||||
summary: qbittorrent is down
|
||||
description: 'qbittorrent: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
|
||||
description: 'qbittorrent has been unreachable for 5m (SLI: up).'
|
||||
@@ -1,29 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: semaphore-db-backup-credentials-prod
|
||||
namespace: semaphore-db
|
||||
type: Opaque
|
||||
stringData:
|
||||
ACCESS_KEY_ID: ENC[AES256_GCM,data:OhWEEYqrvmDB40NmiETgeUisBvaO9ocevCZAsurNijw=,iv:kEjOIg9NHbRYm+Ul1WrBAzb3gLbIv+ipiYCxFK0QoVc=,tag:KAB1JAIw0ATwbjDEKZVT9A==,type:str]
|
||||
ACCESS_SECRET_KEY: ENC[AES256_GCM,data:C0PNKv07LCYpFONLqd55sz+ioyU/i2YJUkbZ8OX/dCErM/DilqchyUAT5EqnT/r3YJmdaA1pX85S3fxlVVHkbg==,iv:4ha28b+xIebvX2JPITmtTovIe4vdmj2XfI3cgkUHH4c=,tag:xbMyflcHojytPzgaoOYcXQ==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZVTJ3cXR6cks3Y0RNS2hx
|
||||
K3pjZEduYUxmL0V3SmFUcU9XbGVHSVFITWh3Cjl1eHlTLytoOW5PaVNFVkJoeTBo
|
||||
T1FnRHNDVzJ6dWpuR09WaElvbWF4YU0KLS0tIEJoQ1VsM1cxRlg0R1dtNGdOenNn
|
||||
aVNjMjVBTytLdHlFdmZCS2tPelo1Y2MKYt1G5t+sDiHgcg+Q9ECBzJn2gS/paSEC
|
||||
7aoX60SBrPgHvnGls5lzc3Tin9nH4jWYl6Nu8DAvIE0YPXDFr5a0UA==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-09-07T06:36:33Z"
|
||||
mac: ENC[AES256_GCM,data:mhVP7tjdZK2FR2imvpJbTyT7BigvcIKEpLSnBS1oSITUirB8xZC7aHBfyFuIoKOrpOJgEe40nHUdDYCHRJtb4dLeVja8+7meVaQWzxC8BWVHsksW+73hsqhjAzlHeobx6Irbk4PAMqIY0/jnf4MHryjnRn3dUuwu9U1CqL30JbE=,iv:57oCcp6FW9EAIVKnYGtkaTrnbijyv54T+vmZxMbMqnw=,tag:agrRP/JLypU60KOeN0s/5w==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
@@ -38,53 +38,16 @@ metadata:
|
||||
name: semaphore-db
|
||||
namespace: semaphore-db
|
||||
spec:
|
||||
# THREE instances — one primary + two hot standbys (2026-09-07), same as keycloak-db /
|
||||
# nextcloud-db; see platform/keycloak/postgres-cluster.yaml for the full rationale. Short version:
|
||||
# Longhorn's 3-way replication saves the DATA on node loss, not the SERVICE, and with
|
||||
# `instances: 1` all three CNPG primaries had drifted onto k3s-prod-2, so one node took Semaphore
|
||||
# down alongside SSO and Nextcloud. A streaming standby lets CNPG promote in seconds and repoint
|
||||
# the -rw Service. Gated on the Barman archive below actually working (ContinuousArchiving=True
|
||||
# with a real firstRecoverabilityPoint) — standbys replicate corruption, they are not a backup.
|
||||
#
|
||||
# Scaling out is non-disruptive: CNPG creates -2 then -3 sequentially, each seeded by
|
||||
# pg_basebackup from the running primary, which is neither restarted nor disconnected.
|
||||
instances: 3
|
||||
# Single Postgres instance on 3-replica Longhorn is acceptable at home scale (ADR-0014, same as
|
||||
# keycloak-db / nextcloud-db). The storage is already 3-way replicated by Longhorn; raise to 3 for
|
||||
# Postgres-level HA as a future hardening.
|
||||
instances: 1
|
||||
imageName: ghcr.io/cloudnative-pg/postgresql:16.4
|
||||
# Stamp app.kubernetes.io/name=semaphore-db onto every generated pod/service so the rendered
|
||||
# semaphore egress CiliumNetworkPolicy's toEndpoints selector matches the primary pod (R-NET-02).
|
||||
inheritedMetadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: semaphore-db
|
||||
# HARD anti-affinity — the three instances MUST land on three DIFFERENT nodes; and roll the
|
||||
# primary by SWITCHOVER rather than CNPG's default in-place `restart`, which would delete and
|
||||
# recreate the primary pod. Full rationale in platform/keycloak/postgres-cluster.yaml. Short
|
||||
# version: CNPG's default `preferred` is a scheduler hint, so when CNPG recreates an instance
|
||||
# after a node loss it may stack two instances on one node — which is how `instances: 3` decays
|
||||
# back into the single-node failure domain this change removes. `required` makes it a constraint;
|
||||
# the cost is that an instance whose node is gone stays Pending, which is correct (2 of 3 still
|
||||
# serve). Satisfiable on both clusters: prod and dev each have 3 schedulable, untainted nodes.
|
||||
#
|
||||
# APPLYING THIS CHANGES THE POD SPEC, so CNPG rolls the instances: standbys first, then one
|
||||
# switchover. Expect a single brief connection reset for Semaphore.
|
||||
affinity:
|
||||
enablePodAntiAffinity: true
|
||||
podAntiAffinityType: required
|
||||
primaryUpdateMethod: switchover
|
||||
# SYNCHRONOUS REPLICATION — quorum of ONE standby, PREFERRED durability (2026-09-07). Full
|
||||
# rationale in platform/keycloak/postgres-cluster.yaml. Short version: async replication loses the
|
||||
# last few transactions when a primary dies before its standbys catch up, and `method: any` +
|
||||
# `number: 1` makes COMMIT wait for one standby to confirm. dataDurability MUST stay `preferred`,
|
||||
# never `required` — `required` blocks writes outright when fewer than `number` healthy standbys
|
||||
# are available, which on a 3-node cluster sharing nodes with etcd and the CNI turns one degraded
|
||||
# standby into a total write outage. `preferred` relaxes to async instead.
|
||||
#
|
||||
# NON-DISRUPTIVE: compiles to `synchronous_standby_names`, a reload-only PostgreSQL parameter —
|
||||
# no restart, no switchover, no client disconnect. Land it after the anti-affinity roll.
|
||||
postgresql:
|
||||
synchronous:
|
||||
method: any
|
||||
number: 1
|
||||
dataDurability: preferred
|
||||
storage:
|
||||
storageClass: longhorn
|
||||
size: 5Gi
|
||||
@@ -106,38 +69,23 @@ spec:
|
||||
memory: 256Mi
|
||||
limits:
|
||||
memory: 512Mi
|
||||
# --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → object storage ---
|
||||
# --- Off-cluster DR backup (R-STATE-03) — CNPG-native Barman → the on-Proxmox MinIO `cnpg` bucket ---
|
||||
# Continuous WAL archiving + periodic base backups = proper Postgres PITR (realises the SD's
|
||||
# backup.policy daily-offcluster). CNPG appends serverName (defaults to the Cluster name) to
|
||||
# destinationPath, so objects land under ${CNPG_BACKUP_PATH}/semaphore-db/ — dev
|
||||
# s3://cnpg/semaphore-db/ on the on-Proxmox MinIO, prod
|
||||
# s3://k3s-prod-backup/barman/semaphore-db/ on Cloudflare R2. Both endpointURL and
|
||||
# destinationPath are cluster-vars knobs (R-ENV-02), substituted by `apps-wave-0`.
|
||||
# Credentials: PER-ENVIRONMENT (2026-09-07). dev reads semaphore-db-backup-credentials (the
|
||||
# scoped MinIO key), prod reads semaphore-db-backup-credentials-prod (the Cloudflare R2
|
||||
# token). BOTH Secrets are applied to BOTH clusters by apps/wave-0/kustomization.yaml (a
|
||||
# generated golden — render.py places the -prod sidecar in this DB's wave); the
|
||||
# cluster-vars knob ${BACKUP_CREDENTIALS_SUFFIX} (dev "" | prod "-prod") chooses which
|
||||
# one this Cluster reads. Both are in-namespace, which is all a pod can read. They were
|
||||
# ONE shared Secret until the split, so giving prod its R2 token would have left dev
|
||||
# signing MinIO requests with an R2 key id — 403 on every archive command, i.e. the
|
||||
# disk-full clock below rather than a merely missing backup.
|
||||
#
|
||||
# archive_mode is ON, so a FAILING archive command is a disk-full clock, not a degraded backup:
|
||||
# Postgres refuses to recycle a WAL segment it has not archived, and the pgdata PVC fills until
|
||||
# the primary PANICs. Prod demonstrated this from bootstrap to 2026-09-07 with a placeholder
|
||||
# endpoint. Treat ContinuousArchiving=False here as an availability incident.
|
||||
# backup.policy daily-offcluster). destinationPath is the bucket ROOT; CNPG appends serverName
|
||||
# (defaults to the Cluster name), so objects land under s3://cnpg/semaphore-db/. Path-style HTTP.
|
||||
# Credentials: the scoped key from the in-namespace SOPS Secret semaphore-db-backup-credentials
|
||||
# (same-ns only), kept in sync with platform/backup/minio-credentials.sops.yaml.
|
||||
backup:
|
||||
retentionPolicy: "30d"
|
||||
barmanObjectStore:
|
||||
destinationPath: ${CNPG_BACKUP_PATH}
|
||||
endpointURL: ${S3_BACKUP_ENDPOINT}
|
||||
destinationPath: s3://cnpg
|
||||
endpointURL: http://192.168.1.9:9000
|
||||
s3Credentials:
|
||||
accessKeyId:
|
||||
name: semaphore-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
|
||||
name: semaphore-db-backup-credentials
|
||||
key: ACCESS_KEY_ID
|
||||
secretAccessKey:
|
||||
name: semaphore-db-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
|
||||
name: semaphore-db-backup-credentials
|
||||
key: ACCESS_SECRET_KEY
|
||||
wal:
|
||||
compression: gzip
|
||||
|
||||
@@ -1,267 +0,0 @@
|
||||
# promtool rule-unit tests for the wave-0 per-app `<app>-down` SLI rules.
|
||||
# Manifests under test: every PrometheusRule in apps/wave-0/ — gitea, jackett, pihole, plex,
|
||||
# qbittorrent (container/Deployment) and vm-demo (KubeVirt VMI). The harness discovers this file by
|
||||
# the SUITE convention (`<dir>/tests/*.test.yaml`) and flattens those RENDERED manifests, so these
|
||||
# tests run against the exact expressions Flux applies, never a copy.
|
||||
#
|
||||
# Run with: platform/monitoring/tests/run.sh apps/wave-0/tests/app-slis.test.yaml
|
||||
# (the merge gate runs the same harness via
|
||||
# conformance/tests/l0-static/alert-rules-unit-tested)
|
||||
#
|
||||
# WHY THESE EXIST. Until 2026-09-13 every one of these rules read `up{job="<app>"} == 0` while the
|
||||
# rendered ServiceMonitor named a Service port (`metrics`) that no rendered Service defined. The
|
||||
# monitor discovered ZERO targets, so no `up{job="<app>"}` series existed and the rule could not
|
||||
# fire — on prod, `count by (job) (up)` listed 24 jobs and not one of the twelve appeared.
|
||||
# `PrometheusTargetsMissing` could not see it either: a target that was never discovered has no `up`
|
||||
# series to be 0. The rules now read kube-state-metrics / kubevirt series that DO exist for these
|
||||
# workloads (verified on prod), and every rule below is pinned in BOTH directions:
|
||||
# * QUIET on the healthy state -> it is not permanently firing;
|
||||
# * FIRING when the workload has no ready replica -> it is not inert;
|
||||
# * FIRING when the signal is ABSENT -> the failure mode that produced this whole defect class
|
||||
# (a rule that goes quiet because its series vanished) is itself alerted on.
|
||||
#
|
||||
# CLOCK/`for` CONVENTION: sample interval 1m, series 21 samples long, evaluated at 10m — comfortably
|
||||
# past every rule's `for: 5m` and well inside Prometheus' 5m lookback.
|
||||
rule_files:
|
||||
- generated.rules.yaml
|
||||
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
# ===========================================================================================
|
||||
# 1. HEALTHY WAVE-0 (what prod looks like right now) -> every rule SILENT.
|
||||
# This is the "not permanently firing" half. It is also the test that catches a mis-spelled
|
||||
# label: get `deployment=` wrong and the `== 0` half matches nothing while the `absent()` half
|
||||
# matches everything, so this test goes red instead of the rule going quietly wrong.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: wave-0 healthy - every app has a ready replica, the demo VM runs - all quiet
|
||||
input_series:
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running", node="k3s-prod-1"}'
|
||||
values: '1+0x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: gitea-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: jackett-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: pihole-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: plex-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: qbittorrent-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: vm-demo-down
|
||||
exp_alerts: []
|
||||
|
||||
# ===========================================================================================
|
||||
# 2. EACH APP DOWN IN TURN (0 available replicas) -> ONLY that app's rule fires.
|
||||
# The per-app isolation matters: a rule that forgot its `namespace=`/`deployment=` matchers
|
||||
# would light up every app at once and be useless for routing.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: wave-0 plex has no available replica - only plex-down fires
|
||||
input_series:
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
|
||||
values: '0+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
|
||||
values: '1+0x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: plex-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: plex
|
||||
namespace: plex
|
||||
deployment: plex
|
||||
exp_annotations:
|
||||
summary: "plex is down"
|
||||
description: "plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
- eval_time: 10m
|
||||
alertname: gitea-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: jackett-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: pihole-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: qbittorrent-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: vm-demo-down
|
||||
exp_alerts: []
|
||||
# Before `for: 5m` has elapsed the rule must be PENDING, not firing — proof the 5m is real and
|
||||
# a single scrape blip does not page anyone.
|
||||
promql_expr_test:
|
||||
- expr: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"} == 0 or absent(kube_deployment_status_replicas_available{namespace="plex", deployment="plex"})'
|
||||
eval_time: 2m
|
||||
exp_samples:
|
||||
- labels: 'kube_deployment_status_replicas_available{deployment="plex", namespace="plex"}'
|
||||
value: 0
|
||||
|
||||
- interval: 1m
|
||||
name: wave-0 gitea/jackett/pihole/qbittorrent each down - each own rule fires
|
||||
input_series:
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
|
||||
values: '0+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"}'
|
||||
values: '0+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="pihole", deployment="pihole"}'
|
||||
values: '0+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"}'
|
||||
values: '0+0x20'
|
||||
- series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
|
||||
values: '1+0x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: gitea-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: gitea
|
||||
namespace: gitea
|
||||
deployment: gitea
|
||||
exp_annotations:
|
||||
summary: "gitea is down"
|
||||
description: "gitea: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
- eval_time: 10m
|
||||
alertname: jackett-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: jackett
|
||||
namespace: jackett
|
||||
deployment: jackett
|
||||
exp_annotations:
|
||||
summary: "jackett is down"
|
||||
description: "jackett: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
- eval_time: 10m
|
||||
alertname: pihole-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: pihole
|
||||
namespace: pihole
|
||||
deployment: pihole
|
||||
exp_annotations:
|
||||
summary: "pihole is down"
|
||||
description: "pihole: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
- eval_time: 10m
|
||||
alertname: qbittorrent-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: qbittorrent
|
||||
namespace: qbittorrent
|
||||
deployment: qbittorrent
|
||||
exp_annotations:
|
||||
summary: "qbittorrent is down"
|
||||
description: "qbittorrent: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
- eval_time: 10m
|
||||
alertname: plex-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: vm-demo-down
|
||||
exp_alerts: []
|
||||
|
||||
# ===========================================================================================
|
||||
# 3. THE SIGNAL VANISHES -> the rule FIRES. This is the defect class itself, asserted on.
|
||||
# The old `up{job=...} == 0` rule had exactly this shape of hole and was silent for it. Here
|
||||
# the Deployment series for plex is simply not in the input at all, and plex-down must still
|
||||
# fire (via the `absent()` half), with the labels the selector's equality matchers carry.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: wave-0 plex Deployment series ABSENT (never scraped / workload gone) - plex-down fires
|
||||
input_series:
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"}'
|
||||
values: '1+0x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: plex-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: plex
|
||||
namespace: plex
|
||||
deployment: plex
|
||||
exp_annotations:
|
||||
summary: "plex is down"
|
||||
description: "plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
- eval_time: 10m
|
||||
alertname: gitea-down
|
||||
exp_alerts: []
|
||||
|
||||
# ===========================================================================================
|
||||
# 4. THE VM: a VMI carries no replica count, so liveness is the EXISTENCE of a running-phase
|
||||
# series. Two ways to be down, both asserted: the VMI is gone entirely, and the VMI exists in
|
||||
# some OTHER phase. The second is the one a naive `kubevirt_vmi_info == 1` would miss, and it
|
||||
# is why the phase value is matched — LOWERCASE `running`, the value KubeVirt actually writes
|
||||
# (a capital-R matcher selects nothing and would be just as inert).
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: wave-0 demo VM has no running-phase VMI - vm-demo-down fires
|
||||
input_series:
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"}'
|
||||
values: '1+0x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: vm-demo-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: vm-demo
|
||||
namespace: vm-demo
|
||||
name: vm-demo
|
||||
phase: running
|
||||
exp_annotations:
|
||||
summary: "vm-demo is down"
|
||||
description: "vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info)."
|
||||
|
||||
- interval: 1m
|
||||
name: wave-0 demo VM exists but is Failed, not running - vm-demo-down still fires
|
||||
input_series:
|
||||
- series: 'kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="failed"}'
|
||||
values: '1+0x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: vm-demo-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: vm-demo
|
||||
namespace: vm-demo
|
||||
name: vm-demo
|
||||
phase: running
|
||||
exp_annotations:
|
||||
summary: "vm-demo is down"
|
||||
description: "vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info)."
|
||||
@@ -1,139 +0,0 @@
|
||||
---
|
||||
# Namespace (trust zone; default-deny + policies apply)
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: vm-demo
|
||||
labels:
|
||||
homelab.keithjackson.dev/app-namespace: 'true'
|
||||
annotations:
|
||||
homelab.keithjackson.dev/backup-policy: daily-offcluster
|
||||
---
|
||||
# Default-deny baseline (R-NET-01): deny all, allow only DNS
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: default-deny
|
||||
namespace: vm-demo
|
||||
labels:
|
||||
homelab.keithjackson.dev/baseline: default-deny
|
||||
spec:
|
||||
podSelector: {}
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress: []
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
---
|
||||
# CDI DataVolume (standalone; survives rebuild, R-STATE-01)
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
kind: DataVolume
|
||||
metadata:
|
||||
name: vm-demo-rootdisk
|
||||
namespace: vm-demo
|
||||
spec:
|
||||
source:
|
||||
registry:
|
||||
url: docker://quay.io/containerdisks/fedora:41
|
||||
storage:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
storageClassName: longhorn
|
||||
resources:
|
||||
requests:
|
||||
storage: 6Gi
|
||||
---
|
||||
# KubeVirt VirtualMachine (R-PLAT-03)
|
||||
apiVersion: kubevirt.io/v1
|
||||
kind: VirtualMachine
|
||||
metadata:
|
||||
name: vm-demo
|
||||
namespace: vm-demo
|
||||
labels:
|
||||
app.kubernetes.io/name: vm-demo
|
||||
spec:
|
||||
running: false
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
kubevirt.io/vm: vm-demo
|
||||
homelab.keithjackson.dev/egress-class: internet
|
||||
homelab.keithjackson.dev/auth-mode: none
|
||||
spec:
|
||||
domain:
|
||||
cpu:
|
||||
cores: 1
|
||||
memory:
|
||||
guest: 1536Mi
|
||||
devices:
|
||||
disks:
|
||||
- name: rootdisk
|
||||
disk:
|
||||
bus: virtio
|
||||
- name: cloudinit
|
||||
disk:
|
||||
bus: virtio
|
||||
interfaces:
|
||||
- name: default
|
||||
masquerade: {}
|
||||
resources:
|
||||
requests:
|
||||
memory: 1536Mi
|
||||
networks:
|
||||
- name: default
|
||||
pod: {}
|
||||
volumes:
|
||||
- name: rootdisk
|
||||
dataVolume:
|
||||
name: vm-demo-rootdisk
|
||||
- name: cloudinit
|
||||
cloudInitNoCloud:
|
||||
userData: "#cloud-config\npackage_update: true\npackages:\n - dnf-automatic\nwrite_files:\n - path: /etc/dnf/automatic.conf\n content: |\n [commands]\n upgrade_type = default\n random_sleep = 0\n download_updates = yes\n apply_updates = yes\nruncmd:\n - [ systemctl, enable, --now, dnf-automatic.timer ]\n"
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): vm-demo-allow-internet-egress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: vm-demo-allow-internet-egress
|
||||
namespace: vm-demo
|
||||
labels:
|
||||
app.kubernetes.io/name: vm-demo
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: vm-demo
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: vm-demo-slis
|
||||
namespace: vm-demo
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: vm-demo
|
||||
spec:
|
||||
groups:
|
||||
- name: vm-demo.slis
|
||||
rules:
|
||||
- alert: vm-demo-down
|
||||
expr: absent(kubevirt_vmi_info{namespace="vm-demo", name="vm-demo", phase="running"})
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: vm-demo
|
||||
annotations:
|
||||
summary: vm-demo is down
|
||||
description: 'vm-demo: no KubeVirt VMI in phase running for 5m (SLI: up, via kubevirt_vmi_info).'
|
||||
@@ -14,7 +14,7 @@ metadata:
|
||||
app.kubernetes.io/name: act-runner
|
||||
type: Opaque
|
||||
stringData:
|
||||
token: ENC[AES256_GCM,data:f574mAUkLvMGq0TYycwCd4tduMuBgQLiTg/140kuRDgFUYYm1ZoSjQ==,iv:Tn7gFWBLm2hHk/BbfVX7qIZegwZp9ah1cMou5i2O91E=,tag:TkdSiMI4sdCNn2JuN++Pig==,type:str]
|
||||
token: ENC[AES256_GCM,data:W48mEtBqGeukX13lMhrhrer+zB21cLVp0acy3m1Xy/oo89DqO8FX/Q==,iv:wfTUU7RiFLdEc1fmEFS9DfE2SmGCFLPK17m0IKspft0=,tag:UhoXA47p7386KHcYNbPzCg==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
@@ -30,8 +30,8 @@ sops:
|
||||
MDZLUE52QzJXbFlrNWxOcVhJbDNnY00KxCpFkbhhCuk4Fa8CC3C4+R9AcsUnaXQR
|
||||
UjAfwKFoXAneXUih7TTXMbR/6+VgQBww5Bmwpn7jVLnzP0N0pIZVTA==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-09-29T05:48:35Z"
|
||||
mac: ENC[AES256_GCM,data:nobqr794nrdRYFrd8Ct8bw9M8NtftJXawRGk9zrL+pTWahwlWrwXVtOAxMGEPVojz3WemsxTJESzmzy0gcT1QcrjpYTsF2MVHcr6+umk8q9gAh2enyxy/0QkYiSt6XjvRVqfqleLanEkeIq1FPf2zhSLSJaiw4/oZTfkCHD1/Zg=,iv:JxaP0htmGcvE4ZU1YAmGF4A6JkejV29NNygyMl1OKF8=,tag:G70NvcPyVuagXhugWoUfTw==,type:str]
|
||||
lastmodified: "2026-08-26T04:48:14Z"
|
||||
mac: ENC[AES256_GCM,data:b99/zkL332RIT3FYBa1sR3o9+pwlmrUgKvVHawbjWYSJZ7ERUfO+LXuhKJ0HKrp6i1JfkurZlRO27iBcfrBisAXS9KLaqmsCJf4kKfFc4skANrmKCzprrXUmZexr1uiE8eDUzjNCLNxSCZfnFR6d/AkMXJqi6Hsj9/O+wH2lWsc=,iv:3DIPszSdxK81ERO3o/nTs33qQYHaAKLNcGP4qlZViFc=,tag:/B6VKkzMUwh0byNUicS5wQ==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
@@ -103,69 +103,25 @@ spec:
|
||||
# ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ----
|
||||
- name: runner
|
||||
image: gitea/act_runner:0.2.11
|
||||
# TWO-STAGE BOUNDED WAIT on the DinD sidecar, then exec the image's normal
|
||||
# entrypoint (/sbin/tini -- /opt/act/run.sh), which registers-from-env on first
|
||||
# boot and runs `act_runner daemon`.
|
||||
#
|
||||
# STAGE 1 — the certs must EXIST: act_runner reads /certs/client/ca.pem at startup
|
||||
# and hard-exits if it is missing (a startup race against the sidecar).
|
||||
# STAGE 2 — the daemon must actually ACCEPT a connection: the cert FILE appearing is
|
||||
# necessary but NOT sufficient. dockerd writes the certs and only THEN begins
|
||||
# listening on :2376, so a runner that proceeds on the file alone still hard-exits
|
||||
# with "cannot ping the docker daemon at tcp://localhost:2376" — observed
|
||||
# crashlooping 11 times in ~8 minutes on a fresh pod after a node bounce.
|
||||
# Both loops are bounded (60 x 1s) and exit 1 on timeout, so a genuinely dead sidecar
|
||||
# surfaces as a CrashLoopBackOff with a clear message instead of hanging forever.
|
||||
#
|
||||
# ENVSUBST SAFETY — the BARE $i / $j below are CORRECT. Do not "tidy" them into the
|
||||
# braced form or the double-dollar form; both are wrong here, for different reasons.
|
||||
# clusters/base/apps-wave-1.yaml carries postBuild.substituteFrom cluster-vars, so
|
||||
# Flux envsubst runs over this ENTIRE manifest — YAML comments very much included.
|
||||
# Its contract (drone/envsubst; the same one spelled out in clusters/base/apps-vms.yaml,
|
||||
# and re-verified against this exact file with `flux envsubst`, flux v2.4.0) is:
|
||||
# - A bare $name is NOT a reference to envsubst: it passes through untouched, and
|
||||
# that holds even when the name IS set in cluster-vars. So $i and $j reach the
|
||||
# cluster verbatim and both 60s bounds below are real, enforced bounds.
|
||||
# - $((i + 1)) is likewise untouched: "(" cannot begin a variable name.
|
||||
# - A BRACED dollar-name IS a reference and gets replaced — with the empty string
|
||||
# when the name is unset. One of those here would make a bound read [ "" -ge 60 ],
|
||||
# which errors, short-circuits the &&, and silently turns the wait UNBOUNDED.
|
||||
# - A DOUBLE-dollar braced name is envsubst's escape and survives as the braced
|
||||
# form — still wrong for us, because on any path where envsubst does NOT run
|
||||
# (a plain `kustomize build` or `kubectl apply -k` of this directory) the shell
|
||||
# reads that leading double-dollar as its own PID, so a bound would compare a
|
||||
# string like 1234{i} against 60. The bare form is the only spelling that is
|
||||
# correct on BOTH paths.
|
||||
# - A braced name containing punctuation is a HARD ERROR ("unable to parse variable
|
||||
# name") that fails this whole Kustomization — and this Kustomization gates every
|
||||
# merge in the repo. So never write one anywhere in this file, INCLUDING in a
|
||||
# comment. That is why this note names the three forms in words rather than
|
||||
# showing them: an earlier draft of it spelled them out literally and broke
|
||||
# `flux envsubst` on the file. (Same trap the indexed-expansion warning in
|
||||
# clusters/base/apps-vms.yaml describes.)
|
||||
# Wait for the DinD sidecar to publish its TLS client certs into the shared
|
||||
# emptyDir before starting the daemon — act_runner reads /certs/client/ca.pem at
|
||||
# startup and hard-exits if it is missing (a startup race against the sidecar).
|
||||
# Then exec the image's normal entrypoint (/sbin/tini -- /opt/act/run.sh), which
|
||||
# registers-from-env on first boot and runs `act_runner daemon`.
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |
|
||||
# Stage 1: the sidecar's TLS client certs must appear in the shared emptyDir.
|
||||
i=0
|
||||
until [ -f /certs/client/ca.pem ]; do
|
||||
i=$((i + 1))
|
||||
[ "$i" -ge 60 ] && { echo "dind certs never appeared" >&2; exit 1; }
|
||||
echo "waiting for dind TLS certs ($i)"; sleep 1
|
||||
done
|
||||
# Stage 2: dockerd must actually accept a connection on :2376. busybox nc in this
|
||||
# Alpine-based image (gitea/act_runner:0.2.11) supports -z (scan) and -w (timeout).
|
||||
j=0
|
||||
until nc -z -w1 127.0.0.1 2376; do
|
||||
j=$((j + 1))
|
||||
[ "$j" -ge 60 ] && { echo "dind daemon never accepted a connection on 2376" >&2; exit 1; }
|
||||
echo "waiting for dind daemon on :2376 ($j)"; sleep 1
|
||||
done
|
||||
exec /sbin/tini -- /opt/act/run.sh
|
||||
env:
|
||||
- name: GITEA_INSTANCE_URL
|
||||
value: https://gitea.${BASE_DOMAIN}
|
||||
value: https://gitea.home.keithjackson.dev
|
||||
- name: GITEA_RUNNER_REGISTRATION_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
@@ -199,31 +155,6 @@ spec:
|
||||
limits:
|
||||
memory: 512Mi
|
||||
# ---- Docker-in-Docker: the job executor daemon (privileged) ----
|
||||
#
|
||||
# FOLLOW-UP (deliberately NOT bundled into the readiness fix above — own change):
|
||||
# Convert this container into a NATIVE SIDECAR and delete both shell wait-loops.
|
||||
# k3s is now v1.36, where sidecar containers are GA, so kubelet can enforce the
|
||||
# ordering that the loops above hand-roll — which also removes the envsubst hazard
|
||||
# from this file entirely (no shell, no $i/$j, nothing to escape or mis-escape).
|
||||
# Shape to build:
|
||||
# 1. Move this whole container into `initContainers:` as the FIRST entry.
|
||||
# 2. Add `restartPolicy: Always` to it. That single field is what makes an
|
||||
# initContainer a sidecar: it starts before the regular containers, keeps
|
||||
# running alongside them, and is terminated after them.
|
||||
# 3. Give it a `startupProbe` that succeeds only when dockerd accepts a TLS
|
||||
# connection — e.g. `exec` a `docker -H tcp://127.0.0.1:2376 --tlsverify
|
||||
# --tlscacert /certs/client/ca.pem --tlscert /certs/client/cert.pem
|
||||
# --tlskey /certs/client/key.pem info`. Do NOT use a bare `tcpSocket` probe:
|
||||
# it would re-open the same "port accepting but TLS/certs not ready" gap that
|
||||
# Stage 1 above exists to close. `periodSeconds: 1` + `failureThreshold: 60`
|
||||
# preserves today's 60s cap.
|
||||
# 4. kubelet then holds the `runner` container until that startupProbe passes, so
|
||||
# `runner` can drop its `command:` override completely and run the image's
|
||||
# default entrypoint.
|
||||
# Why it is a separate change: this Deployment is the ENFORCED MERGE GATE for this
|
||||
# entire repository. If the conversion is wrong, nothing can merge — so it must land
|
||||
# alone, where the failure is unambiguous and not entangled with anything else, and
|
||||
# only after a real workflow run has been observed green on the converted pod.
|
||||
- name: dind
|
||||
image: docker:27-dind
|
||||
securityContext:
|
||||
|
||||
@@ -1,208 +0,0 @@
|
||||
---
|
||||
# Namespace (trust zone; default-deny + policies apply)
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: keycloak
|
||||
labels:
|
||||
homelab.keithjackson.dev/app-namespace: 'true'
|
||||
gateway-access: shared
|
||||
annotations:
|
||||
homelab.keithjackson.dev/backup-policy: daily-offcluster
|
||||
---
|
||||
# Default-deny baseline (R-NET-01): deny all, allow only DNS
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: default-deny
|
||||
namespace: keycloak
|
||||
labels:
|
||||
homelab.keithjackson.dev/baseline: default-deny
|
||||
spec:
|
||||
podSelector: {}
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress: []
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
---
|
||||
# PVC keycloak-db (R-STATE)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: keycloak-db
|
||||
namespace: keycloak
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
storageClassName: longhorn
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi
|
||||
---
|
||||
# Deployment (R-PLAT-02)
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: keycloak
|
||||
namespace: keycloak
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
homelab.keithjackson.dev/egress-class: internet
|
||||
homelab.keithjackson.dev/auth-mode: native
|
||||
spec:
|
||||
containers:
|
||||
- name: keycloak
|
||||
image: quay.io/keycloak/keycloak
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
volumeMounts:
|
||||
- name: db
|
||||
mountPath: /data/db
|
||||
volumes:
|
||||
- name: db
|
||||
persistentVolumeClaim:
|
||||
claimName: keycloak-db
|
||||
---
|
||||
# Service
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: keycloak
|
||||
namespace: keycloak
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
spec:
|
||||
selector:
|
||||
app.kubernetes.io/name: keycloak
|
||||
ports:
|
||||
- name: port-8080
|
||||
port: 8080
|
||||
targetPort: 8080
|
||||
---
|
||||
# CiliumNetworkPolicy egress allow (R-NET-02): keycloak-allow-keycloak-db-5432
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: keycloak-allow-keycloak-db-5432
|
||||
namespace: keycloak
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
egress:
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
app.kubernetes.io/name: keycloak-db
|
||||
k8s:io.kubernetes.pod.namespace: keycloak-db
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '5432'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy gateway-ingress allow (R-NET-01/02): keycloak-allow-gateway-ingress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: keycloak-allow-gateway-ingress
|
||||
namespace: keycloak
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
ingress:
|
||||
- fromEntities:
|
||||
- ingress
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '8080'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): keycloak-allow-internet-egress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: keycloak-allow-internet-egress
|
||||
namespace: keycloak
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
---
|
||||
# HTTPRoute keycloak-8080 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
kind: HTTPRoute
|
||||
metadata:
|
||||
name: keycloak-8080
|
||||
namespace: keycloak
|
||||
labels:
|
||||
app.kubernetes.io/name: keycloak
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- keycloak.${BASE_DOMAIN}
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
type: PathPrefix
|
||||
value: /
|
||||
backendRefs:
|
||||
- name: keycloak
|
||||
port: 8080
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: keycloak-slis
|
||||
namespace: keycloak
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: keycloak
|
||||
spec:
|
||||
groups:
|
||||
- name: keycloak.slis
|
||||
rules:
|
||||
- alert: keycloak-down
|
||||
expr: kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"} == 0 or absent(kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"})
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: keycloak
|
||||
annotations:
|
||||
summary: keycloak is down
|
||||
description: 'keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready).'
|
||||
@@ -10,9 +10,7 @@ resources:
|
||||
- gitea-actions.yaml
|
||||
- nextcloud-bootstrap-secret.sops.yaml
|
||||
- nextcloud.yaml
|
||||
- radarr-oauth2-proxy-credentials.sops.yaml
|
||||
- radarr.yaml
|
||||
- semaphore-bootstrap-secret.sops.yaml
|
||||
- semaphore.yaml
|
||||
- sonarr-oauth2-proxy-credentials.sops.yaml
|
||||
- sonarr.yaml
|
||||
+28
-65
@@ -35,34 +35,6 @@ spec:
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
---
|
||||
# PersistentVolume nextcloud-legacy-archive - static external NFS (bulk tier)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: nextcloud-legacy-archive
|
||||
labels:
|
||||
app.kubernetes.io/name: nextcloud
|
||||
spec:
|
||||
capacity:
|
||||
storage: 270Gi
|
||||
accessModes:
|
||||
- ReadOnlyMany
|
||||
persistentVolumeReclaimPolicy: Retain
|
||||
storageClassName: ''
|
||||
mountOptions:
|
||||
- nfsvers=4.1
|
||||
- hard
|
||||
- noatime
|
||||
- timeo=600
|
||||
- retrans=2
|
||||
nfs:
|
||||
server: ${BULK_NFS_SERVER}
|
||||
path: /mnt/Storage/nextcloud
|
||||
readOnly: true
|
||||
claimRef:
|
||||
namespace: nextcloud
|
||||
name: nextcloud-legacy-archive
|
||||
---
|
||||
# PVC nextcloud-data (R-STATE)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
@@ -79,23 +51,6 @@ spec:
|
||||
requests:
|
||||
storage: 20Gi
|
||||
---
|
||||
# PVC nextcloud-legacy-archive (R-STATE)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: nextcloud-legacy-archive
|
||||
namespace: nextcloud
|
||||
labels:
|
||||
app.kubernetes.io/name: nextcloud
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadOnlyMany
|
||||
storageClassName: ''
|
||||
volumeName: nextcloud-legacy-archive
|
||||
resources:
|
||||
requests:
|
||||
storage: 270Gi
|
||||
---
|
||||
# Deployment (R-PLAT-02)
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
@@ -149,13 +104,13 @@ spec:
|
||||
name: nextcloud-bootstrap
|
||||
key: admin-password
|
||||
- name: NEXTCLOUD_TRUSTED_DOMAINS
|
||||
value: nextcloud.${BASE_DOMAIN}
|
||||
value: nextcloud.home.keithjackson.dev
|
||||
- name: OVERWRITEHOST
|
||||
value: nextcloud.${BASE_DOMAIN}
|
||||
value: nextcloud.home.keithjackson.dev
|
||||
- name: OVERWRITEPROTOCOL
|
||||
value: https
|
||||
- name: OVERWRITECLIURL
|
||||
value: https://nextcloud.${BASE_DOMAIN}
|
||||
value: https://nextcloud.home.keithjackson.dev
|
||||
- name: NEXTCLOUD_OIDC_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
@@ -164,26 +119,18 @@ spec:
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /var/www/html
|
||||
- name: legacy-archive
|
||||
mountPath: /mnt/legacy-nextcloud
|
||||
subPath: nextcloud
|
||||
readOnly: true
|
||||
lifecycle:
|
||||
postStart:
|
||||
exec:
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n# maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n# stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n i=$((i + 1))\n [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[nextcloud-oidc]\
|
||||
\ client secret unset; skipping OIDC (break-glass admin only)\"\n exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1; break\n fi\n d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n# HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n# single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n# form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n# occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n# (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n --clientid=nextcloud \\\n --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n --scope='openid email profile' \\\n --unique-uid=0 \\\n --mapping-uid=preferred_username \\\n --mapping-email=email \\\n --mapping-display-name=name \\\n --group-provisioning=1 \\\n --mapping-groups=groups\" >/dev/null 2>&1 || true\n if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n# NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n# login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n# built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n# start: occ group:adduser is a no-op for an existing member. A user is promotable only\n# AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n# rights on their next login + the following pod start (or run `occ group:adduser admin\n# <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n# passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n current_admins=$(parse_group admin)\n missing=0\n for uid in $(parse_group \"$ADMIN_SRC_GROUP\"\
|
||||
); do\n if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n continue\n fi\n missing=1\n if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n fi\n done\n [ \"$missing\" = 0 ] && break\n p=$((p + 1)); sleep 2\ndone\nif [ \"$p\" -ge 15 ]; then\n nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\nfi\n# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:\n# the final statement used to be `[ \"$p\" -ge 15 ] && nextcloud_oidc_fail ...`, so on the\n# SUCCESS path the test returned 1 and became the script's exit status. That was latent for\n# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard\n# exposed it immediately. Every failure path here logs loudly and writes a marker file and\n# deliberately does NOT crash the pod \u2014 the break-glass local admin must stay usable.\nexit 0\n"
|
||||
- "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n# maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n# stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n i=$((i + 1))\n [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\nif [ -z \"${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1; break\n fi\n d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n# HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n# single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n# form fed the value back through the wrapper's $*, which SPLIT it into extra\
|
||||
\ positionals ->\n# occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n# (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n --clientid=nextcloud \\\n --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n --scope='openid email profile' \\\n --unique-uid=0 \\\n --mapping-uid=preferred_username \\\n --mapping-email=email \\\n --mapping-display-name=name \\\n --group-provisioning=1 \\\n --mapping-groups=groups\" >/dev/null 2>&1 || true\n if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n# NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n# login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n# built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n# start: occ group:adduser is a no-op for an existing member. A user is promotable only\n# AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n# rights on their next login + the following pod start (or run `occ group:adduser admin\n# <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n# passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n current_admins=$(parse_group admin)\n missing=0\n for uid in $(parse_group \"$ADMIN_SRC_GROUP\"); do\n if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n continue\n fi\n missing=1\n if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n fi\n done\n [ \"$missing\" = 0 ] && break\n p=$((p + 1)); sleep 2\ndone\n[ \"$p\" -ge 15 ] && nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\n"
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: nextcloud-data
|
||||
- name: legacy-archive
|
||||
persistentVolumeClaim:
|
||||
claimName: nextcloud-legacy-archive
|
||||
---
|
||||
# Service
|
||||
apiVersion: v1
|
||||
@@ -258,10 +205,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute nextcloud-80 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
@@ -280,7 +223,7 @@ spec:
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- nextcloud.${BASE_DOMAIN}
|
||||
- nextcloud.home.keithjackson.dev
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
@@ -290,6 +233,26 @@ spec:
|
||||
- name: nextcloud
|
||||
port: 80
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: nextcloud
|
||||
namespace: nextcloud
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: nextcloud
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: nextcloud
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- nextcloud
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
@@ -304,11 +267,11 @@ spec:
|
||||
- name: nextcloud.slis
|
||||
rules:
|
||||
- alert: nextcloud-down
|
||||
expr: kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"} == 0 or absent(kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"})
|
||||
expr: up{job="nextcloud"} == 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: nextcloud
|
||||
annotations:
|
||||
summary: nextcloud is down
|
||||
description: 'nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
|
||||
description: 'nextcloud has been unreachable for 5m (SLI: up).'
|
||||
@@ -1,30 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: radarr-oauth2-proxy
|
||||
namespace: radarr
|
||||
type: Opaque
|
||||
stringData:
|
||||
client-id: ENC[AES256_GCM,data:vFLxqzxt,iv:6k0ZjKP8rvg56ZFPWEGdyj+FLgK+LBOE0pLBWfC0bjA=,tag:LiZb5BasJAEWM1c3I8vFTA==,type:str]
|
||||
client-secret: ENC[AES256_GCM,data:JzfHF8JF4YUxBxiC738N4UsMlcCOvxn4/AE2B4meZNGAZb2uhG4n51uEY8OQsbsM0Ni4qM2Wi17KuLDwAicfUA==,iv:A3AnUhYxw/IWfvfcZf6MgC5u37dMoNToIWxRG9AxEJQ=,tag:1OY2pWGTbCxPYYdaiBa/oA==,type:str]
|
||||
cookie-secret: ENC[AES256_GCM,data:LPt5nghLfoNvr7yBes8rLeEK3M6UVIO1h9POHMOAj8iZX3l2RJJa4w8GcSg=,iv:Q5+7zH8Tb1SWe92B0pxekfMgOXlBG8nixWn/ho4oJw4=,tag:D+41dBFPVTGj86xv5/GEng==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBHVG5MWVRyOVl1RWFpOExC
|
||||
SjZKcEd1ZlFnZVZ5NmtyUlQzRGI1b1o3dG04CkwrTUNjd3crM1VzaUdPczliSDY1
|
||||
a3dONnBFWk8yOW5kL2xES1RZbUN6L3cKLS0tIG9hWG9Cbm01RlMrcC9zMWkxeFgr
|
||||
ZnRGRUVTTS81SFRyenNsZUZEcG1TUm8K5XUGP/1NVsoaFcBZUER3/zMPTtCuVisT
|
||||
LrbSmTkIyVfzjN/vRpRH92b1VzENJvhGhRghhQaKY0rbG34i6yh3Vw==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-08-31T13:42:25Z"
|
||||
mac: ENC[AES256_GCM,data:TxdPz2zE6+ypgQG2b0Rp3VFiPL7LLls4NNkC7YQ/bVCLvyvaXdGsEWkiHJE+Po7hcESmZRt/NMS25mEXPMfvlSCltwFUTcrxoZDS0NI27kIzbtceTbxmznt/dMS/iZm21lCgA6Aag9ByLjWusPLkRKWhep+G+0G54qtfQzv3s80=,iv:44f0KikrpKpnSaLLyzTvnCfrwEXylRgVIlKMtSpMyNc=,tag:jDx43n0usDtbwzo8KzDBqw==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
+28
-139
@@ -35,33 +35,6 @@ spec:
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
---
|
||||
# PersistentVolume radarr-media - static external NFS (bulk tier)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: radarr-media
|
||||
labels:
|
||||
app.kubernetes.io/name: radarr
|
||||
spec:
|
||||
capacity:
|
||||
storage: 9Ti
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
persistentVolumeReclaimPolicy: Retain
|
||||
storageClassName: ''
|
||||
mountOptions:
|
||||
- nfsvers=4.1
|
||||
- hard
|
||||
- noatime
|
||||
- timeo=600
|
||||
- retrans=2
|
||||
nfs:
|
||||
server: ${BULK_NFS_SERVER}
|
||||
path: /mnt/Storage/plex
|
||||
claimRef:
|
||||
namespace: radarr
|
||||
name: radarr-media
|
||||
---
|
||||
# PVC radarr-config (R-STATE)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
@@ -78,23 +51,6 @@ spec:
|
||||
requests:
|
||||
storage: 5Gi
|
||||
---
|
||||
# PVC radarr-media (R-STATE)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: radarr-media
|
||||
namespace: radarr
|
||||
labels:
|
||||
app.kubernetes.io/name: radarr
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
storageClassName: ''
|
||||
volumeName: radarr-media
|
||||
resources:
|
||||
requests:
|
||||
storage: 9Ti
|
||||
---
|
||||
# Deployment (R-PLAT-02)
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
@@ -118,99 +74,20 @@ spec:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
homelab.keithjackson.dev/auth-groups: media-users
|
||||
spec:
|
||||
containers:
|
||||
- name: radarr
|
||||
image: lscr.io/linuxserver/radarr:5.21.1
|
||||
ports:
|
||||
- containerPort: 7878
|
||||
env:
|
||||
- name: PUID
|
||||
value: '1000'
|
||||
- name: PGID
|
||||
value: '3000'
|
||||
- name: TZ
|
||||
value: Etc/UTC
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /config
|
||||
- name: media
|
||||
mountPath: /media
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
- --reverse-proxy=true
|
||||
- --provider=oidc
|
||||
- --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
|
||||
- --upstream=http://127.0.0.1:7878
|
||||
- --email-domain=*
|
||||
- --scope=openid email profile
|
||||
- --oidc-groups-claim=groups
|
||||
- --allowed-group=media-users
|
||||
- --allowed-group=platform-admins
|
||||
- --skip-provider-button=true
|
||||
- --redirect-url=https://radarr.${BASE_DOMAIN}/oauth2/callback
|
||||
- --cookie-domain=.${BASE_DOMAIN}
|
||||
- --whitelist-domain=.${BASE_DOMAIN}
|
||||
- --cookie-secure=true
|
||||
- --pass-access-token=true
|
||||
- --set-xauthrequest=true
|
||||
env:
|
||||
- name: OAUTH2_PROXY_CLIENT_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: radarr-oauth2-proxy
|
||||
key: client-id
|
||||
- name: OAUTH2_PROXY_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: radarr-oauth2-proxy
|
||||
key: client-secret
|
||||
- name: OAUTH2_PROXY_COOKIE_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: radarr-oauth2-proxy
|
||||
key: cookie-secret
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 4180
|
||||
protocol: TCP
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
securityContext:
|
||||
fsGroup: 3000
|
||||
mountPath: /data/config
|
||||
volumes:
|
||||
- name: config
|
||||
persistentVolumeClaim:
|
||||
claimName: radarr-config
|
||||
- name: media
|
||||
persistentVolumeClaim:
|
||||
claimName: radarr-media
|
||||
---
|
||||
# Service
|
||||
apiVersion: v1
|
||||
@@ -227,9 +104,6 @@ spec:
|
||||
- name: port-7878
|
||||
port: 7878
|
||||
targetPort: 7878
|
||||
- name: authproxy
|
||||
port: 4180
|
||||
targetPort: 4180
|
||||
---
|
||||
# CiliumNetworkPolicy egress allow (R-NET-02): radarr-allow-qbittorrent-8080
|
||||
apiVersion: cilium.io/v2
|
||||
@@ -314,7 +188,7 @@ spec:
|
||||
- ingress
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '4180'
|
||||
- port: '7878'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): radarr-allow-internet-egress
|
||||
@@ -332,10 +206,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute radarr-7878 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
@@ -347,15 +217,14 @@ metadata:
|
||||
app.kubernetes.io/name: radarr
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
homelab.keithjackson.dev/auth-groups: media-users
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- radarr.${BASE_DOMAIN}
|
||||
- radarr.home.keithjackson.dev
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
@@ -363,7 +232,27 @@ spec:
|
||||
value: /
|
||||
backendRefs:
|
||||
- name: radarr
|
||||
port: 4180
|
||||
port: 7878
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: radarr
|
||||
namespace: radarr
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: radarr
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: radarr
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- radarr
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
@@ -379,11 +268,11 @@ spec:
|
||||
- name: radarr.slis
|
||||
rules:
|
||||
- alert: radarr-down
|
||||
expr: kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"})
|
||||
expr: up{job="radarr"} == 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: radarr
|
||||
annotations:
|
||||
summary: radarr is down
|
||||
description: 'radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
|
||||
description: 'radarr has been unreachable for 5m (SLI: up).'
|
||||
@@ -5,15 +5,6 @@
|
||||
# credential from its own namespace (a Pod cannot read a Secret in another namespace).
|
||||
# access-key-encryption: FIXED base64 32-byte AES key Semaphore uses to encrypt stored SSH keys /
|
||||
# credentials; MUST stay stable across restarts or stored secrets become undecryptable.
|
||||
# oidc-client-secret: the Keycloak `semaphore` CONFIDENTIAL client secret, and ONLY that. It MIRRORS
|
||||
# `semaphore-client-secret` in platform/keycloak/keycloak-realm-clients.sops.yaml (the realm import
|
||||
# injects that copy into the client), same cross-namespace mirror pattern as db-password above.
|
||||
# It replaced the former `oidc-providers-json` key (2026-09-07): that key held the WHOLE
|
||||
# SEMAPHORE_OIDC_PROVIDERS blob, which buried the environment-specific realm-issuer and redirect
|
||||
# URLs inside ciphertext where Flux's ${BASE_DOMAIN} postBuild substitution could not reach them,
|
||||
# so prod ran with dev's URLs. Those URLs are PUBLIC and now live in plaintext in
|
||||
# catalog/semaphore.yaml (rendered to apps/wave-1/semaphore.yaml); the kubelet splices THIS value
|
||||
# back into the blob via $(SEMAPHORE_OIDC_CLIENT_SECRET). Only the secret half is encrypted.
|
||||
# Recover: infra/devcontainer/dev.sh run sops -d apps/semaphore-bootstrap-secret.sops.yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
@@ -24,11 +15,11 @@ metadata:
|
||||
app.kubernetes.io/name: semaphore
|
||||
type: Opaque
|
||||
stringData:
|
||||
admin-username: ENC[AES256_GCM,data:ju3wVdjNzRUP/w==,iv:d/5ZxgQZ5bympu3gT2ck3DJy9iUPPBAtZYsJvD3To7I=,tag:0UtBaDVqNSK4RCJrQicchw==,type:str]
|
||||
admin-password: ENC[AES256_GCM,data:pxIwpgh3BoND2TI0mPTixOTJoictzT77dqtuRdMKgY/2WaW4wM1Ra6H6n/dxnyR2,iv:eYbFQGFYVJsnYGwn6cOI3BgAdzMcin9YggAhiuL+xqM=,tag:/sPWGv4gue0vjc2nMk7wWA==,type:str]
|
||||
db-password: ENC[AES256_GCM,data:hBFpWXdpk1R5a23Pntup03t56NBRPS+SZU8CrHVb/ypIaIpgf7kjYtVy6nhTF6u0,iv:eBzoQgH5ZhWWpfKZ6AQmwaSal5QaiuN9gfexQ7GsVFs=,tag:j2k8EwhKDlj/Ud6O39ioVQ==,type:str]
|
||||
access-key-encryption: ENC[AES256_GCM,data:SKtAV4HxJ9pNYDqUK/1uOdI1rtMji3pTfprI2rhQxFQyhc6dFlhqI+OUEcY=,iv:rKCP+FLidcOltOwmMMcO5M8tz4qiifIMI+W5soGO2M8=,tag:HaHzkX/JI+9wmcg2SD9GAQ==,type:str]
|
||||
oidc-client-secret: ENC[AES256_GCM,data:uCo7TNzbM2QKw9ie+xzyUG9MFKco1OAlwr5qPn/AGpNvo1YyXSfbZyyM0CLJ1NS821hGg5Zr/m3R/xpy84s3FQ==,iv:O8a4haskufpmorPaxOkJ1Q99Y5GTlzmDFJAsHS+irLk=,tag:VojJpA9jNMbNQmIe1Qr44w==,type:str]
|
||||
admin-username: ENC[AES256_GCM,data:uD/vO9A9MjerFg==,iv:i/NAqVD6rWg9oWWSGBrRkxzwcPp2IW1HLIKY0pdcPPg=,tag:r27tKLAJ+ydE7Ty3ccMNUw==,type:str]
|
||||
admin-password: ENC[AES256_GCM,data:544nWY1UmBnlsdPJAQZJMXRumu4W9X7UXZvZ6vHN/ArjRgLXKTArOe5JkT/nNrNw,iv:AC1DDbZj2HfpU19EjzA7T+WpIaR4nZc3PNN61y8bAzU=,tag:r28LFi5w6ZeCCaRaX2jRdg==,type:str]
|
||||
db-password: ENC[AES256_GCM,data:ySkdvaGRCW9fMQqChs5l8dY4zCG19i4HOEinRC48mgLuh9WOrjd16T6jwFgiwB55,iv:+1gztJ22HbwgQJiolZMMWOcKIsxrBJpnruJdtxdooOU=,tag:8CfJGwyPTjDGyr0CvalXMQ==,type:str]
|
||||
access-key-encryption: ENC[AES256_GCM,data:1YAMLo1nVkpJWZc+KBZ7B4ia+Y8P+xMyOhHMcEGGJOXd3o+WC5FqirLBVh4=,iv:j28ozeaDhGWDjoaxFOhn2t9YbYE7L3ah/Mfi+i2DWwY=,tag:czG9Wxc/7Nsuh1W5iATOww==,type:str]
|
||||
oidc-providers-json: ENC[AES256_GCM,data:90yt5mIVWLVI2KrlJWpxhL/BMDpEgRhv7+BL+mBRt79diRvU7pN9vtzhZprAKzcU1sEywlT4EwFlsvJpYyo5Qbkp+ZVknbyhAlGvHhJOBHukOPWgf4LignLerO2GzhXjOOzFqWfZ4EyUxNMWTbSe/xc0zIVHDDNQiYplGVRK+E/ukiI7sD/Tn2RrbrdvpC5GyK0NXWRsXn9ffhE3fCoiRHXagj9i/q+uUrQI5EKpPobrDUAGqvKoBXTl/wJ4phnWg9IqtCTE/c3yA7fLLz4j5dCtPZNSVPT4TKDx8Mdv8CfCgziVicjG2s+yHuWpsQn8lf6nvdJmpYPvIqTtLWdxFWuD8Vfo6Qbq2mqPbKR+KmDOGUnjl4gbZdL1SkE9JiqBUobz7UXeLXwYzxMIsinOfYZLcCk7alrD5Yndj9JDxDJZQ54sMMBfVbxDl94W1VHy03RxAR46KKAbPaxOP5wH4Q4z3SDceNwmuGCDRGChRXiHhN5X2X0GqYCbc8E6GdkIMcpmD80e26ydEmAjw7F89mCF4F/yMzJNiG82qaZIn8HeqgGA8urous9r9bYJNlitYTYQ,iv:jbfVdrrZ9fGiOxhQeoTJfFg1QwiOMkoQyaadiVfxX/w=,tag:JkepcRs34EuEgNb4buTTDQ==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
@@ -38,14 +29,14 @@ sops:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAyTzF6NTBtcFVWK1Vtc0I2
|
||||
VGdySHBsRGgvc1h0VzgvQ01xYVBJV3g2Q1NRCjJyZHVYZGtuamo1MFJHWlBwWkxt
|
||||
cHA4VkY1TkRpN3M4UTdMOWZTbFVGemMKLS0tIGVNdDBrWVJveU56VjVBU2JWdHF4
|
||||
RGtaYStXYUpONTZjZCtmM0NyZ3Rka1UKIQpmQai8dz+uRAsXxyeuGaRlRwsFqIPQ
|
||||
9AfrZJcslYwlaK3ptm1oqaCdrlxcMar2nI7thSqzbLe+e1nzA2z7vQ==
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBuRDlqRFJ0dk14UHluSGRP
|
||||
S1pESy9PWlpMakU0aTkzNVhqUGdtbWNEdTJjCm5RNG14azJHc28rdEJ4L0RzR0xO
|
||||
MGRocGZ1V1JZTENWQkN3VGZzamp2eTgKLS0tIGt6QkNNaW5hdTBJVHY0SVo5d3JX
|
||||
elp0SWxHcjg0RWF3aW9mWTJHZ0lhcUEKc0rU5VilAVDleCwqq4SOF+Y9sniu1Pka
|
||||
w/MGDA9THSB0b+YgWRWiX/KiimFqp0ZFODOJ9cHVwFGY71+VadPwOg==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-09-07T07:01:50Z"
|
||||
mac: ENC[AES256_GCM,data:Xutm1rQt57J4RILElD8FkNOsQT1HK5jZlPtqqLyZ/VqOoDwo+IS1wp7hROprlUvQC0bUXy79tRRIl5Q0CCOD+fvAGIdmade+vB7qXlYN6Q1441dGhuSTPvckso6FaB2ddRjKhKGLc/Z4bJoz9VaRNNxJNpWXIrENTg+I+QhNhlo=,iv:klJOrEQ6QN5dTGkQfGnbMbgCWsgWwZPb5YXDMeWeE34=,tag:fuJr4t3Tt3dT9u3/0SFo8Q==,type:str]
|
||||
lastmodified: "2026-08-27T22:12:32Z"
|
||||
mac: ENC[AES256_GCM,data:nHftJTGnTr0O5h8yGXwgz3eflggXJfIXM3m8n3bzDJE/P4nvSpZzFeMb+sc2on+AL+t+SeKrIl8blzm33XsQ1lkaY85Zm+bc5cnED1g04BMDVmmK3brh/8w50Bp2wPEVzb4I01MDzhhyIbOYg7pFvZAH3/VuqvM1t6XmfF1NrOA=,iv:4KBa9PH/dfdGwLC16RenZIVB6BtSoGo6x3qt3h1R43I=,tag:KULiA1klP/RZMNigkloGDA==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
+26
-13
@@ -110,7 +110,7 @@ spec:
|
||||
- name: SEMAPHORE_ADMIN_NAME
|
||||
value: Semaphore Admin
|
||||
- name: SEMAPHORE_ADMIN_EMAIL
|
||||
value: admin@${BASE_DOMAIN}
|
||||
value: admin@home.keithjackson.dev
|
||||
- name: SEMAPHORE_ACCESS_KEY_ENCRYPTION
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
@@ -118,13 +118,11 @@ spec:
|
||||
key: access-key-encryption
|
||||
- name: SEMAPHORE_PORT
|
||||
value: :3000
|
||||
- name: SEMAPHORE_OIDC_CLIENT_SECRET
|
||||
- name: SEMAPHORE_OIDC_PROVIDERS
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: semaphore-bootstrap
|
||||
key: oidc-client-secret
|
||||
- name: SEMAPHORE_OIDC_PROVIDERS
|
||||
value: '{"keycloak":{"display_name":"Keycloak","provider_url":"https://keycloak.${BASE_DOMAIN}/realms/homelab","client_id":"semaphore","client_secret":"$(SEMAPHORE_OIDC_CLIENT_SECRET)","redirect_url":"https://semaphore.${BASE_DOMAIN}/api/auth/oidc/keycloak/redirect","scopes":["openid","profile","email"],"username_claim":"preferred_username","email_claim":"email","name_claim":"name","order":1}}'
|
||||
key: oidc-providers-json
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /etc/semaphore
|
||||
@@ -211,10 +209,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute semaphore-3000 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
@@ -227,14 +221,13 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: oidc
|
||||
homelab.keithjackson.dev/auth-groups: platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- semaphore.${BASE_DOMAIN}
|
||||
- semaphore.home.keithjackson.dev
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
@@ -244,6 +237,26 @@ spec:
|
||||
- name: semaphore
|
||||
port: 3000
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: semaphore
|
||||
namespace: semaphore
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: semaphore
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: semaphore
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- semaphore
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
@@ -258,11 +271,11 @@ spec:
|
||||
- name: semaphore.slis
|
||||
rules:
|
||||
- alert: semaphore-down
|
||||
expr: kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"} == 0 or absent(kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"})
|
||||
expr: up{job="semaphore"} == 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: semaphore
|
||||
annotations:
|
||||
summary: semaphore is down
|
||||
description: 'semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
|
||||
description: 'semaphore has been unreachable for 5m (SLI: up).'
|
||||
@@ -1,30 +0,0 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: sonarr-oauth2-proxy
|
||||
namespace: sonarr
|
||||
type: Opaque
|
||||
stringData:
|
||||
client-id: ENC[AES256_GCM,data:JMYoJymH,iv:CM9HF71QZCeETzrOtkeTUs+BLTp/gGSExulXUP8yS/M=,tag:l73RaFQolO7+wZBo8nYCvw==,type:str]
|
||||
client-secret: ENC[AES256_GCM,data:Fq271KdbtKWKWOe1Sun6+SLv/kSBsvAeFUhZIq6J2DNPGM4E8dgc2YANITcpezcJR07PAbSlUcS0L/qF6K+YhQ==,iv:/mVD1uOkweyO26RMV3qRQmLdyxXJR4I7p+OjYiiuSMU=,tag:9ZOKye5Sp2NHU2RXH1Y+wQ==,type:str]
|
||||
cookie-secret: ENC[AES256_GCM,data:jMZZEi8fdh20BbPzCoVfwdS1g0/ma5n/QISU5A/gQCu/2Rkylt7mp798CUg=,iv:6IPqDNQyJHze2DUOQ93IwruWSge5nXjkVcQIOMfJSwY=,tag:cOKRcDKywV3PBOBrhoddRQ==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBoNU5NbnJnV0dKRnpIN2dw
|
||||
QmhBRWI4aEpWajRHVkxZcGxCRk1hMDBjSlRVCm91b0U2N08vNi8rSEFaVlRJazFL
|
||||
bllPcm9EMUgrQ3ZlUDlrdEZpZE8zNmMKLS0tIDU0ajdycXZrazlQb3RVRGQyY25h
|
||||
R3RDajltcmFvQWdLYWQ2Mjh0VkplaWsKJprnLNJYzMd0yHbxjsD8P66Wu+sefmaV
|
||||
nduEgVQY0t4EsYN6iKVDLMUAp+o3gXU8QuMuBnFZznrmBghfG+ZruQ==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-08-31T13:42:49Z"
|
||||
mac: ENC[AES256_GCM,data:ckVR9OyFvle0fLhBjHtJ6hH2St2CRJeu6abTCnqtxAfMDVXdbIkgHEesmN2kAgA7s3LkKJC2m7EFIpt3pO94VQNnZvC1KKD7qUSIfJN1DLVv78m8JbEQfVTJk/WgWy1ObxLriw5hDuMiEuOvKND+s6eqS5j6er4Q72lBsBpoEeY=,iv:uguThOQbR9Ks1qs1U0HO7EhUj1BT60fvM+3xVEC9hUU=,tag:m4P5nhlspCZyJRuOqffMog==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
+28
-139
@@ -35,33 +35,6 @@ spec:
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
---
|
||||
# PersistentVolume sonarr-media - static external NFS (bulk tier)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: sonarr-media
|
||||
labels:
|
||||
app.kubernetes.io/name: sonarr
|
||||
spec:
|
||||
capacity:
|
||||
storage: 9Ti
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
persistentVolumeReclaimPolicy: Retain
|
||||
storageClassName: ''
|
||||
mountOptions:
|
||||
- nfsvers=4.1
|
||||
- hard
|
||||
- noatime
|
||||
- timeo=600
|
||||
- retrans=2
|
||||
nfs:
|
||||
server: ${BULK_NFS_SERVER}
|
||||
path: /mnt/Storage/plex
|
||||
claimRef:
|
||||
namespace: sonarr
|
||||
name: sonarr-media
|
||||
---
|
||||
# PVC sonarr-config (R-STATE)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
@@ -78,23 +51,6 @@ spec:
|
||||
requests:
|
||||
storage: 5Gi
|
||||
---
|
||||
# PVC sonarr-media (R-STATE)
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: sonarr-media
|
||||
namespace: sonarr
|
||||
labels:
|
||||
app.kubernetes.io/name: sonarr
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
storageClassName: ''
|
||||
volumeName: sonarr-media
|
||||
resources:
|
||||
requests:
|
||||
storage: 9Ti
|
||||
---
|
||||
# Deployment (R-PLAT-02)
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
@@ -118,99 +74,20 @@ spec:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
homelab.keithjackson.dev/auth-groups: media-users
|
||||
spec:
|
||||
containers:
|
||||
- name: sonarr
|
||||
image: lscr.io/linuxserver/sonarr:4.0.15
|
||||
ports:
|
||||
- containerPort: 8989
|
||||
env:
|
||||
- name: PUID
|
||||
value: '1000'
|
||||
- name: PGID
|
||||
value: '3000'
|
||||
- name: TZ
|
||||
value: Etc/UTC
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /config
|
||||
- name: media
|
||||
mountPath: /media
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
- --reverse-proxy=true
|
||||
- --provider=oidc
|
||||
- --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
|
||||
- --upstream=http://127.0.0.1:8989
|
||||
- --email-domain=*
|
||||
- --scope=openid email profile
|
||||
- --oidc-groups-claim=groups
|
||||
- --allowed-group=media-users
|
||||
- --allowed-group=platform-admins
|
||||
- --skip-provider-button=true
|
||||
- --redirect-url=https://sonarr.${BASE_DOMAIN}/oauth2/callback
|
||||
- --cookie-domain=.${BASE_DOMAIN}
|
||||
- --whitelist-domain=.${BASE_DOMAIN}
|
||||
- --cookie-secure=true
|
||||
- --pass-access-token=true
|
||||
- --set-xauthrequest=true
|
||||
env:
|
||||
- name: OAUTH2_PROXY_CLIENT_ID
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: sonarr-oauth2-proxy
|
||||
key: client-id
|
||||
- name: OAUTH2_PROXY_CLIENT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: sonarr-oauth2-proxy
|
||||
key: client-secret
|
||||
- name: OAUTH2_PROXY_COOKIE_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: sonarr-oauth2-proxy
|
||||
key: cookie-secret
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 4180
|
||||
protocol: TCP
|
||||
resources:
|
||||
requests:
|
||||
cpu: 25m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ping
|
||||
port: http
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 15
|
||||
securityContext:
|
||||
fsGroup: 3000
|
||||
mountPath: /data/config
|
||||
volumes:
|
||||
- name: config
|
||||
persistentVolumeClaim:
|
||||
claimName: sonarr-config
|
||||
- name: media
|
||||
persistentVolumeClaim:
|
||||
claimName: sonarr-media
|
||||
---
|
||||
# Service
|
||||
apiVersion: v1
|
||||
@@ -227,9 +104,6 @@ spec:
|
||||
- name: port-8989
|
||||
port: 8989
|
||||
targetPort: 8989
|
||||
- name: authproxy
|
||||
port: 4180
|
||||
targetPort: 4180
|
||||
---
|
||||
# CiliumNetworkPolicy egress allow (R-NET-02): sonarr-allow-qbittorrent-8080
|
||||
apiVersion: cilium.io/v2
|
||||
@@ -314,7 +188,7 @@ spec:
|
||||
- ingress
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '4180'
|
||||
- port: '8989'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
|
||||
@@ -332,10 +206,6 @@ spec:
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
io.kubernetes.pod.namespace: keycloak
|
||||
app: keycloak
|
||||
---
|
||||
# HTTPRoute sonarr-8989 (R-TLS/R-DNS)
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
@@ -347,15 +217,14 @@ metadata:
|
||||
app.kubernetes.io/name: sonarr
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: forward-auth
|
||||
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
homelab.keithjackson.dev/auth-groups: media-users
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- sonarr.${BASE_DOMAIN}
|
||||
- sonarr.home.keithjackson.dev
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
@@ -363,7 +232,27 @@ spec:
|
||||
value: /
|
||||
backendRefs:
|
||||
- name: sonarr
|
||||
port: 4180
|
||||
port: 8989
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: sonarr
|
||||
namespace: sonarr
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: sonarr
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: sonarr
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- sonarr
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
@@ -379,11 +268,11 @@ spec:
|
||||
- name: sonarr.slis
|
||||
rules:
|
||||
- alert: sonarr-down
|
||||
expr: kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"})
|
||||
expr: up{job="sonarr"} == 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: sonarr
|
||||
annotations:
|
||||
summary: sonarr is down
|
||||
description: 'sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
|
||||
description: 'sonarr has been unreachable for 5m (SLI: up).'
|
||||
@@ -1,245 +0,0 @@
|
||||
# promtool rule-unit tests for the wave-1 per-app `<app>-down` SLI rules.
|
||||
# Manifests under test: every PrometheusRule in apps/wave-1/ — nextcloud, radarr, semaphore, sonarr
|
||||
# (container/Deployment) and keycloak (StatefulSet, see below). Discovered by the SUITE convention;
|
||||
# the flattened manifests are the RENDERED ones Flux applies.
|
||||
#
|
||||
# Run with: platform/monitoring/tests/run.sh apps/wave-1/tests/app-slis.test.yaml
|
||||
#
|
||||
# See apps/wave-0/tests/app-slis.test.yaml for the full history: these rules all read
|
||||
# `up{job="<app>"} == 0` until 2026-09-13 against a job no scrape target produced, and were
|
||||
# structurally inert on prod. Each is now pinned QUIET on the healthy state, FIRING on no-ready-
|
||||
# replica, and FIRING when the signal is ABSENT.
|
||||
#
|
||||
# KEYCLOAK IS THE INTERESTING ONE. Its server is a StatefulSet run by the Keycloak operator, NOT the
|
||||
# Deployment this ServiceDefinition would otherwise imply — so its rule reads
|
||||
# kube_statefulset_status_replicas_ready (catalog/keycloak.yaml declares
|
||||
# `observability.livenessWorkload: {kind: StatefulSet, name: keycloak}`). Had it been left to the
|
||||
# default, the `absent()` half would have matched forever and the alert would have been PERMANENTLY
|
||||
# FIRING, which is the other failure mode this repo has shipped to prod. Verified on prod
|
||||
# 2026-09-13: kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}
|
||||
# = 2, and the only Deployment in that namespace is keycloak-operator.
|
||||
rule_files:
|
||||
- generated.rules.yaml
|
||||
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
# ===========================================================================================
|
||||
# 1. HEALTHY WAVE-1 -> every rule SILENT ("not permanently firing").
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: wave-1 healthy - every app ready, keycloak StatefulSet at 2 replicas - all quiet
|
||||
input_series:
|
||||
- series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
|
||||
values: '2+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
|
||||
values: '1+0x20'
|
||||
# The namespace's OTHER workloads are present on purpose: keycloak-operator is a Deployment in
|
||||
# the keycloak namespace, and keycloak-db is a StatefulSet in it. A rule that matched the
|
||||
# namespace only (not the object) would be satisfied — or triggered — by these instead of by
|
||||
# the IdP itself. That is the exact masking KeycloakDown had to be fixed for.
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="keycloak", deployment="keycloak-operator"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak-db"}'
|
||||
values: '3+0x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: keycloak-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: nextcloud-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: radarr-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: semaphore-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: sonarr-down
|
||||
exp_alerts: []
|
||||
|
||||
# ===========================================================================================
|
||||
# 2. THE IdP ITSELF IS DOWN while its operator and DB are fine -> ONLY keycloak-down fires.
|
||||
# This is the anti-masking assertion: the neighbours in that namespace stay ready.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: wave-1 keycloak StatefulSet has no ready replica (operator + DB fine) - keycloak-down fires
|
||||
input_series:
|
||||
- series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
|
||||
values: '0+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="keycloak", deployment="keycloak-operator"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak-db"}'
|
||||
values: '3+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
|
||||
values: '1+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
|
||||
values: '1+0x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: keycloak-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: keycloak
|
||||
namespace: keycloak
|
||||
statefulset: keycloak
|
||||
exp_annotations:
|
||||
summary: "keycloak is down"
|
||||
description: "keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
|
||||
- eval_time: 10m
|
||||
alertname: nextcloud-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: radarr-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: semaphore-down
|
||||
exp_alerts: []
|
||||
- eval_time: 10m
|
||||
alertname: sonarr-down
|
||||
exp_alerts: []
|
||||
|
||||
# ===========================================================================================
|
||||
# 3. EACH DEPLOYMENT-BACKED APP DOWN -> its own rule fires, keycloak stays quiet.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: wave-1 nextcloud/radarr/semaphore/sonarr have no available replica - each own rule fires
|
||||
input_series:
|
||||
- series: 'kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"}'
|
||||
values: '2+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"}'
|
||||
values: '0+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"}'
|
||||
values: '0+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"}'
|
||||
values: '0+0x20'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"}'
|
||||
values: '0+0x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: nextcloud-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: nextcloud
|
||||
namespace: nextcloud
|
||||
deployment: nextcloud
|
||||
exp_annotations:
|
||||
summary: "nextcloud is down"
|
||||
description: "nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
- eval_time: 10m
|
||||
alertname: radarr-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: radarr
|
||||
namespace: radarr
|
||||
deployment: radarr
|
||||
exp_annotations:
|
||||
summary: "radarr is down"
|
||||
description: "radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
- eval_time: 10m
|
||||
alertname: semaphore-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: semaphore
|
||||
namespace: semaphore
|
||||
deployment: semaphore
|
||||
exp_annotations:
|
||||
summary: "semaphore is down"
|
||||
description: "semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
- eval_time: 10m
|
||||
alertname: sonarr-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: sonarr
|
||||
namespace: sonarr
|
||||
deployment: sonarr
|
||||
exp_annotations:
|
||||
summary: "sonarr is down"
|
||||
description: "sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
- eval_time: 10m
|
||||
alertname: keycloak-down
|
||||
exp_alerts: []
|
||||
|
||||
# ===========================================================================================
|
||||
# 4. THE SIGNAL VANISHES -> the rules FIRE. The defect class itself, asserted on: a rule that can
|
||||
# only fire on `== 0` goes SILENT when kube-state-metrics or the workload object disappears,
|
||||
# which is indistinguishable from healthy. Here NOTHING in wave-1 is scraped.
|
||||
# ===========================================================================================
|
||||
- interval: 1m
|
||||
name: wave-1 all liveness series ABSENT (kube-state-metrics gone) - every rule fires
|
||||
input_series:
|
||||
# One unrelated series so the test has an input at all; no wave-1 liveness series exists.
|
||||
- series: 'up{job="kube-state-metrics"}'
|
||||
values: '0+0x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: keycloak-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: keycloak
|
||||
namespace: keycloak
|
||||
statefulset: keycloak
|
||||
exp_annotations:
|
||||
summary: "keycloak is down"
|
||||
description: "keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
|
||||
- eval_time: 10m
|
||||
alertname: nextcloud-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: nextcloud
|
||||
namespace: nextcloud
|
||||
deployment: nextcloud
|
||||
exp_annotations:
|
||||
summary: "nextcloud is down"
|
||||
description: "nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
- eval_time: 10m
|
||||
alertname: radarr-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: radarr
|
||||
namespace: radarr
|
||||
deployment: radarr
|
||||
exp_annotations:
|
||||
summary: "radarr is down"
|
||||
description: "radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
- eval_time: 10m
|
||||
alertname: semaphore-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: semaphore
|
||||
namespace: semaphore
|
||||
deployment: semaphore
|
||||
exp_annotations:
|
||||
summary: "semaphore is down"
|
||||
description: "semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
- eval_time: 10m
|
||||
alertname: sonarr-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: sonarr
|
||||
namespace: sonarr
|
||||
deployment: sonarr
|
||||
exp_annotations:
|
||||
summary: "sonarr is down"
|
||||
description: "sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
+23
-3
@@ -223,7 +223,7 @@ spec:
|
||||
namespace: gateway
|
||||
sectionName: https
|
||||
hostnames:
|
||||
- seerr.${BASE_DOMAIN}
|
||||
- seerr.home.keithjackson.dev
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
@@ -233,6 +233,26 @@ spec:
|
||||
- name: seerr
|
||||
port: 5055
|
||||
---
|
||||
# ServiceMonitor (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: seerr
|
||||
namespace: seerr
|
||||
labels:
|
||||
release: kube-prometheus-stack
|
||||
app.kubernetes.io/name: seerr
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: seerr
|
||||
namespaceSelector:
|
||||
matchNames:
|
||||
- seerr
|
||||
endpoints:
|
||||
- port: metrics
|
||||
interval: 30s
|
||||
---
|
||||
# PrometheusRule SLI alerts (R-OBS-01)
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
@@ -247,11 +267,11 @@ spec:
|
||||
- name: seerr.slis
|
||||
rules:
|
||||
- alert: seerr-down
|
||||
expr: kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"})
|
||||
expr: up{job="seerr"} == 0
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
service: seerr
|
||||
annotations:
|
||||
summary: seerr is down
|
||||
description: 'seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'
|
||||
description: 'seerr has been unreachable for 5m (SLI: up).'
|
||||
@@ -1,69 +0,0 @@
|
||||
# promtool rule-unit tests for the wave-2 per-app `<app>-down` SLI rule (seerr).
|
||||
# Manifest under test: apps/wave-2/seerr.yaml. Discovered by the SUITE convention; the flattened
|
||||
# manifest is the RENDERED one Flux applies.
|
||||
#
|
||||
# Run with: platform/monitoring/tests/run.sh apps/wave-2/tests/app-slis.test.yaml
|
||||
#
|
||||
# seerr (Jellyseerr/Overseerr) exports NO Prometheus metrics of its own, so its `up` SLI could never
|
||||
# have been met by scraping the app — see apps/wave-0/tests/app-slis.test.yaml for the full history
|
||||
# of the inert `up{job="seerr"} == 0` rule this replaces. Pinned in both directions plus the
|
||||
# signal-vanishes case.
|
||||
rule_files:
|
||||
- generated.rules.yaml
|
||||
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
- interval: 1m
|
||||
name: wave-2 healthy - seerr has an available replica - quiet
|
||||
input_series:
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"}'
|
||||
values: '1+0x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: seerr-down
|
||||
exp_alerts: []
|
||||
|
||||
- interval: 1m
|
||||
name: wave-2 seerr has no available replica - seerr-down fires
|
||||
input_series:
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"}'
|
||||
values: '0+0x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: seerr-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: seerr
|
||||
namespace: seerr
|
||||
deployment: seerr
|
||||
exp_annotations:
|
||||
summary: "seerr is down"
|
||||
description: "seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
# Under `for: 5m` a 2m-old condition must NOT yet be a firing alert, but the expression must
|
||||
# already be selecting — the distinction an inert rule cannot make.
|
||||
promql_expr_test:
|
||||
- expr: 'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"})'
|
||||
eval_time: 2m
|
||||
exp_samples:
|
||||
- labels: 'kube_deployment_status_replicas_available{deployment="seerr", namespace="seerr"}'
|
||||
value: 0
|
||||
|
||||
- interval: 1m
|
||||
name: wave-2 seerr liveness series ABSENT - seerr-down still fires
|
||||
input_series:
|
||||
- series: 'up{job="kube-state-metrics"}'
|
||||
values: '1+0x20'
|
||||
alert_rule_test:
|
||||
- eval_time: 10m
|
||||
alertname: seerr-down
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
service: seerr
|
||||
namespace: seerr
|
||||
deployment: seerr
|
||||
exp_annotations:
|
||||
summary: "seerr is down"
|
||||
description: "seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
|
||||
@@ -11,76 +11,16 @@ Spec: [ADR-0004 (Flux)](../../../home-network-k3s/decisions/adr-0004-gitops-flux
|
||||
## Files
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. render+apply the Flux controllers (`components/`) → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. |
|
||||
| `components/kustomization.yaml` | **Customization of the Flux CONTROLLERS themselves.** Overlays `flux install --export` output; currently enables `StrictPostBuildSubstitutions` on kustomize-controller. See "Customizing the controllers" below. |
|
||||
| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. `flux install` → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. |
|
||||
| `patches/platform-sops-decryption.yaml` | Strategic-merge patch enabling `decryption.provider: sops` + `secretRef: sops-age` on the `platform` Kustomization (realises the base TODO). |
|
||||
| `patches/apps-sops-decryption.yaml` | Same, for the app-wave Kustomizations. There is **no single `apps` Kustomization** any more — the apps tier is now `apps-wave-0/1/2` + `apps-vms`, and SOPS decryption is layered on **per wave** (each wave Kustomization is patched by name in `clusters/dev`). |
|
||||
| `patches/apps-sops-decryption.yaml` | Same, for the `apps` Kustomization. |
|
||||
| `patches/flux-system-source-url.yaml` | Overrides the `flux-system` GitRepository `url` placeholder with the real bootstrap source (template — fill at bootstrap). |
|
||||
|
||||
## Customizing the controllers (`components/`) — and how it reaches a RUNNING cluster
|
||||
|
||||
Because this repo does not use `flux bootstrap`, the Flux controller Deployments are **not in git
|
||||
and not reconciled by Flux**. Verified on the live dev cluster (2026-09-01):
|
||||
|
||||
```
|
||||
$ kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}'
|
||||
{"app.kubernetes.io/component":"kustomize-controller","app.kubernetes.io/instance":"flux-system",
|
||||
"app.kubernetes.io/part-of":"flux","app.kubernetes.io/version":"v2.4.0","control-plane":"controller"}
|
||||
# no kustomize.toolkit.fluxcd.io/name label; deployment.kubernetes.io/revision: 1
|
||||
```
|
||||
|
||||
No `kustomize.toolkit.fluxcd.io/*` label => no Flux Kustomization owns it. **Nothing under
|
||||
`clusters/**` can change a controller flag.** So `bootstrap.sh` step 1 uses the mechanism Flux
|
||||
documents for hand-installed clusters ([Customize Flux
|
||||
manifests](https://fluxcd.io/flux/installation/#customize-flux-manifests)):
|
||||
|
||||
```
|
||||
flux install --export … -> <tmp>/gotk-components.yaml # regenerated from the pinned flux CLI
|
||||
components/kustomization.yaml -> <tmp>/kustomization.yaml # the only hand-authored part (in git)
|
||||
kubectl kustomize <tmp> | kubectl apply -f -
|
||||
```
|
||||
|
||||
Nothing is vendored into the repo, so the component manifests can never drift from `FLUX_VERSION`
|
||||
(`infra/devcontainer/versions.env`). Every future controller flag (vertical scaling, tolerations,
|
||||
more feature gates) is another entry in `components/kustomization.yaml`.
|
||||
|
||||
### Rolling this out to the ALREADY-RUNNING dev cluster (operator/PM step — NOT automatic)
|
||||
|
||||
> **A `git push` does not deliver this.** Merging the change reconciles `clusters/**` only; the
|
||||
> controller Deployment is outside Flux's inventory. Someone must run step 1.
|
||||
|
||||
Re-running `bootstrap.sh` is safe and idempotent, but if you want only step 1:
|
||||
|
||||
```bash
|
||||
# inside infra/devcontainer (pinned flux 2.4.0 / kubectl 1.36.4), from the repo root
|
||||
d=$(mktemp -d)
|
||||
flux install --export --namespace=flux-system \
|
||||
--components-extra=image-reflector-controller,image-automation-controller > "$d/gotk-components.yaml"
|
||||
cp bootstrap/flux/components/kustomization.yaml "$d/kustomization.yaml"
|
||||
kubectl kustomize "$d" | kubectl apply -f - # add --server-side --force-conflicts on a field-manager conflict
|
||||
kubectl -n flux-system rollout status deploy/kustomize-controller
|
||||
```
|
||||
|
||||
**Expect kustomize-controller to restart** (new arg => new pod template => new ReplicaSet). Any
|
||||
reconcile in flight is interrupted and retried by the new pod; Flux is level-triggered, so nothing
|
||||
is lost. Confirm afterwards:
|
||||
|
||||
```bash
|
||||
kubectl -n flux-system get deploy kustomize-controller \
|
||||
-o jsonpath='{.spec.template.spec.containers[0].args}' # must contain --feature-gates=StrictPostBuildSubstitutions=true
|
||||
flux get kustomizations -A # all 11 must return to Ready
|
||||
```
|
||||
|
||||
**Rollback** is the same command with `FLUX_INSTALL_PLAIN=1 bash bootstrap/flux/bootstrap.sh`, or
|
||||
simply `flux install --namespace=flux-system --components-extra=…` (drops the arg back off).
|
||||
|
||||
## Why `flux install` + apply, not `flux bootstrap git` (Phase 1)
|
||||
|
||||
The reconciliation graph is **already authored** in `clusters/base` (read-only to this worker):
|
||||
`flux-system.yaml` is the `flux-system` **GitRepository**; `platform.yaml`, `platform-config.yaml`
|
||||
and the app-wave Kustomizations (`apps-wave-0/1/2.yaml` + `apps-vms.yaml` — the single `apps.yaml`
|
||||
was retired) are the Flux **Kustomizations**; `clusters/dev` is the overlay that patches
|
||||
`cluster-vars` (R-ENV-02) and layers SOPS decryption onto each Kustomization by name.
|
||||
`flux-system.yaml` is the `flux-system` **GitRepository**; `platform.yaml`/`apps.yaml` are the Flux
|
||||
**Kustomizations**; `clusters/dev` is the overlay that patches `cluster-vars` (R-ENV-02).
|
||||
|
||||
`flux bootstrap git` **generates and owns its own** `gotk-sync` GitRepository + Kustomization named
|
||||
`flux-system` and commits them into the repo — which would **duplicate/conflict** with the authored
|
||||
@@ -101,9 +41,8 @@ tofu apply → cloud-init → Ansible (k3s + Cilium) [Infra worker, T-1
|
||||
3. bootstrap/flux/bootstrap.sh # flux install → verify sops-age → apply -k clusters/dev
|
||||
│
|
||||
▼
|
||||
Flux reconciles: GitRepository(flux-system) → Kustomization(platform) → Kustomization(platform-config)
|
||||
→ Kustomization(apps-wave-0 → apps-wave-1 → apps-wave-2) + Kustomization(apps-vms),
|
||||
with per-Kustomization SOPS decryption enabled (patches/) and cluster-vars overlay (R-ENV-02).
|
||||
Flux reconciles: GitRepository(flux-system) → Kustomization(platform) → Kustomization(apps),
|
||||
with SOPS decryption enabled (patches/) and cluster-vars overlay (R-ENV-02).
|
||||
```
|
||||
|
||||
## Interface note — for the Infra worker (T-1.3 Ansible)
|
||||
@@ -133,8 +72,7 @@ patches:
|
||||
- path: ../../bootstrap/flux/patches/apps-sops-decryption.yaml # SOPS on apps
|
||||
```
|
||||
|
||||
Alternatively the base TODOs in `clusters/base/platform.yaml` and each app-wave Kustomization
|
||||
(`apps-wave-0/1/2.yaml`, `apps-vms.yaml`) and the `url` in
|
||||
Alternatively the base TODOs in `clusters/base/{platform,apps}.yaml` and the `url` in
|
||||
`clusters/base/flux-system.yaml` can be filled in directly (same effect) — the clusters owner's call.
|
||||
Either way the `sops-age` `secretRef` name and `flux-system` source name are kept in lockstep with
|
||||
the authored base.
|
||||
|
||||
+14
-66
@@ -1,9 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint. ENVIRONMENT-AGNOSTIC: which overlay
|
||||
# it applies is FLUX_ENV_PATH (default clusters/dev). roles/flux_bootstrap exports that from
|
||||
# `flux_bootstrap_env_path`, overridden to clusters/prod in group_vars/k3s_prod/vars.yml — so the
|
||||
# same script bootstraps dev and prod with no fork (R-ENV-02).
|
||||
# bootstrap.sh — the SINGLE Flux + SOPS bootstrap entrypoint for the dev cluster.
|
||||
#
|
||||
# Phase 1, T-1.4 · ADR-0004 (Flux) · ADR-0007 (SOPS+age) · R-GITOPS-01 · R-SEC-01.
|
||||
# =============================================================================
|
||||
@@ -28,19 +25,16 @@
|
||||
# We do NOT use `flux bootstrap git` in Phase 1 because that command commits
|
||||
# and OWNS its own gotk-sync GitRepository+Kustomization named `flux-system`.
|
||||
# Our entrypoints are ALREADY authored in clusters/base (flux-system.yaml =
|
||||
# the GitRepository; platform.yaml + platform-config.yaml + the app-wave
|
||||
# Kustomizations apps-wave-0/1/2.yaml + apps-vms.yaml = the Kustomizations;
|
||||
# the single apps.yaml was retired). Letting
|
||||
# the GitRepository; platform.yaml/apps.yaml = the Kustomizations). Letting
|
||||
# bootstrap regenerate them would DUPLICATE/conflict. `flux install` +
|
||||
# applying the authored clusters/dev overlay is the reconciling path.
|
||||
# 2. sops-age Secret — the operator installs the age PRIVATE key out-of-band BEFORE reconcile so
|
||||
# decryption works on first pass (../sops/age-key-secret.md). This script
|
||||
# only VERIFIES it exists; it never handles key material.
|
||||
# 3. `kubectl apply -k ${FLUX_ENV_PATH}` — materialises the authored reconciliation graph:
|
||||
# 3. `kubectl apply -k clusters/dev` — materialises the authored reconciliation graph:
|
||||
# GitRepository `flux-system` (source) -> Kustomization `platform` ->
|
||||
# `platform-config` -> the app-wave Kustomizations `apps-wave-0/1/2`
|
||||
# (chained dependsOn) + `apps-vms`, with cluster-vars overlay
|
||||
# (R-ENV-02) and per-Kustomization SOPS decryption enabled via ./patches/.
|
||||
# Kustomization `apps` (dependsOn platform), with cluster-vars overlay
|
||||
# (R-ENV-02) and SOPS decryption enabled via the patches in ./patches/.
|
||||
#
|
||||
# PREREQUISITES the operator/PM resolves before this runs (see ../README.md "Handoff"):
|
||||
# - clusters/base/flux-system.yaml url placeholder is overridden — wire ./patches/flux-system-
|
||||
@@ -55,10 +49,7 @@
|
||||
set -euo pipefail
|
||||
|
||||
# --- config (env-overridable) ------------------------------------------------
|
||||
# Which environment overlay to apply. Dev is the DEFAULT (it was the only Phase-1 build target);
|
||||
# prod passes clusters/prod via roles/flux_bootstrap. Changing the default would silently retarget
|
||||
# every existing caller — override it, never edit it.
|
||||
FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}"
|
||||
FLUX_ENV_PATH="${FLUX_ENV_PATH:-clusters/dev}" # dev is the ONLY Phase-1 build target
|
||||
FLUX_NAMESPACE="${FLUX_NAMESPACE:-flux-system}"
|
||||
SOPS_SECRET_NAME="${SOPS_SECRET_NAME:-sops-age}"
|
||||
# The two Phase-4 image-automation controllers are installed alongside the default set so their CRDs
|
||||
@@ -68,66 +59,23 @@ FLUX_COMPONENTS_EXTRA="${FLUX_COMPONENTS_EXTRA:-image-reflector-controller,image
|
||||
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
repo_root="$(git -C "${script_dir}" rev-parse --show-toplevel 2>/dev/null || (cd "${script_dir}/../.." && pwd))"
|
||||
|
||||
echo "==> Flux + SOPS bootstrap — path=${FLUX_ENV_PATH}"
|
||||
echo "==> Flux + SOPS bootstrap (dev) — path=${FLUX_ENV_PATH}"
|
||||
|
||||
# --- 0. tooling present (from infra/devcontainer; versions pinned there) ------
|
||||
for bin in flux kubectl; do
|
||||
command -v "$bin" >/dev/null 2>&1 || { echo "ERROR: '$bin' not found (use infra/devcontainer)." >&2; exit 1; }
|
||||
done
|
||||
|
||||
# --- 1. install Flux controllers (rendered through ./components) --------------
|
||||
# --- 1. install Flux controllers ---------------------------------------------
|
||||
# Default controllers + the image-reflector/image-automation pair (Phase-4 hands-off updates,
|
||||
# ADR-0009). `--components-extra` regenerates flux-system with all controllers and registers the
|
||||
# image.toolkit.fluxcd.io CRDs before any reconcile — so platform/image-automation's CR applies cleanly.
|
||||
#
|
||||
# We do NOT run `flux install` directly any more. The controllers need CUSTOMIZING (currently:
|
||||
# --feature-gates=StrictPostBuildSubstitutions=true on kustomize-controller, so an undefined
|
||||
# postBuild variable FAILS the Kustomization instead of silently rendering an empty string — see
|
||||
# ./components/kustomization.yaml and the 2026-09-01 INCIDENT entry in DECISIONS-LOG.md).
|
||||
#
|
||||
# `flux install` has no flag for that, and because this repo does not use `flux bootstrap` the
|
||||
# controllers are not in git for a Flux Kustomization to patch. So we use the mechanism Flux
|
||||
# documents for hand-installed clusters ("Customize Flux manifests"):
|
||||
#
|
||||
# flux install --export > gotk-components.yaml # exact manifests `flux install` would apply
|
||||
# kustomize overlay = ./components/kustomization.yaml (checked in — the ONLY hand-authored part)
|
||||
# kubectl apply -k
|
||||
#
|
||||
# The export is regenerated from the PINNED flux CLI on every run (FLUX_VERSION,
|
||||
# infra/devcontainer/versions.env), so nothing is vendored and nothing can drift. The render dir is
|
||||
# a temp dir — the repo tree is never written to.
|
||||
#
|
||||
# ESCAPE HATCH: FLUX_INSTALL_PLAIN=1 falls back to a bare `flux install` (no customization, so NO
|
||||
# strict substitution). Use only to unblock a bring-up if the render path misbehaves, and say so.
|
||||
echo "==> [1/3] installing Flux controllers${FLUX_COMPONENTS_EXTRA:+ (+ extra: ${FLUX_COMPONENTS_EXTRA})}"
|
||||
flux_install_args=(--namespace="${FLUX_NAMESPACE}")
|
||||
[ -n "${FLUX_COMPONENTS_EXTRA}" ] && flux_install_args+=(--components-extra="${FLUX_COMPONENTS_EXTRA}")
|
||||
|
||||
if [ "${FLUX_INSTALL_PLAIN:-0}" = "1" ]; then
|
||||
echo " WARNING: FLUX_INSTALL_PLAIN=1 — controller customization SKIPPED."
|
||||
echo " StrictPostBuildSubstitutions will NOT be enabled; substitution fails OPEN."
|
||||
flux install "${flux_install_args[@]}"
|
||||
if [ -n "${FLUX_COMPONENTS_EXTRA}" ]; then
|
||||
echo "==> [1/3] flux install (default controllers + extra: ${FLUX_COMPONENTS_EXTRA})"
|
||||
flux install --namespace="${FLUX_NAMESPACE}" --components-extra="${FLUX_COMPONENTS_EXTRA}"
|
||||
else
|
||||
components_dir="${script_dir}/components"
|
||||
[ -f "${components_dir}/kustomization.yaml" ] || {
|
||||
echo "ERROR: ${components_dir}/kustomization.yaml missing — refusing to install uncustomized." >&2
|
||||
echo " (Override with FLUX_INSTALL_PLAIN=1 only if you accept fail-OPEN substitution.)" >&2
|
||||
exit 1
|
||||
}
|
||||
render_dir="$(mktemp -d)"
|
||||
trap 'rm -rf "${render_dir}"' EXIT
|
||||
echo " rendering flux install --export through ${components_dir}"
|
||||
flux install --export "${flux_install_args[@]}" > "${render_dir}/gotk-components.yaml"
|
||||
cp "${components_dir}/kustomization.yaml" "${render_dir}/kustomization.yaml"
|
||||
# Fail fast if the overlay cannot build (e.g. a patch target that no longer exists after a Flux
|
||||
# version bump) — better a hard stop than silently installing without the feature gate.
|
||||
kubectl kustomize "${render_dir}" > "${render_dir}/rendered.yaml"
|
||||
grep -q -- '--feature-gates=StrictPostBuildSubstitutions=true' "${render_dir}/rendered.yaml" || {
|
||||
echo "ERROR: the StrictPostBuildSubstitutions patch did not apply to the rendered manifests." >&2
|
||||
echo " Check ./components/kustomization.yaml against this flux CLI version." >&2
|
||||
exit 1
|
||||
}
|
||||
kubectl apply -f "${render_dir}/rendered.yaml"
|
||||
echo "==> [1/3] flux install (default controllers only)"
|
||||
flux install --namespace="${FLUX_NAMESPACE}"
|
||||
fi
|
||||
|
||||
# --- 2. verify the out-of-band age key Secret exists -------------------------
|
||||
@@ -140,7 +88,7 @@ if ! kubectl -n "${FLUX_NAMESPACE}" get secret "${SOPS_SECRET_NAME}" >/dev/null
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# --- 3. apply the authored environment entrypoint ----------------------------
|
||||
# --- 3. apply the authored dev entrypoint ------------------------------------
|
||||
echo "==> [3/3] kubectl apply -k ${FLUX_ENV_PATH}"
|
||||
kubectl apply -k "${repo_root}/${FLUX_ENV_PATH}"
|
||||
|
||||
|
||||
@@ -1,64 +0,0 @@
|
||||
---
|
||||
# Controller-level customization of the Flux CONTROLLERS themselves (the `flux-system` Deployments).
|
||||
#
|
||||
# WHY THIS DIRECTORY EXISTS
|
||||
# =========================
|
||||
# This repo does NOT use `flux bootstrap` (see ../README.md "Why `flux install` + apply"): the
|
||||
# reconciliation graph is authored by hand in clusters/base, and the controllers are installed
|
||||
# imperatively by `flux install`. Consequence, verified on the live dev cluster 2026-09-01:
|
||||
#
|
||||
# kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}'
|
||||
# -> NO kustomize.toolkit.fluxcd.io/name label, deployment.kubernetes.io/revision: 1
|
||||
#
|
||||
# i.e. **the controllers are not managed by Flux and are not in git.** There was therefore no
|
||||
# existing mechanism to customize them, and no amount of editing under clusters/** can reach them.
|
||||
#
|
||||
# This is the mechanism Flux documents for exactly that situation ("Customize Flux manifests" —
|
||||
# https://fluxcd.io/flux/installation/#customize-flux-manifests): export the component manifests
|
||||
# with `flux install --export`, overlay them with kustomize `patches:`, and apply the result.
|
||||
# `../bootstrap.sh` step 1 does that render — it copies THIS file next to a freshly exported
|
||||
# `gotk-components.yaml` in a temp dir and runs `kubectl apply -k`. Nothing is generated into the
|
||||
# repo, so the manifests can never drift from the pinned FLUX_VERSION
|
||||
# (infra/devcontainer/versions.env) the way a vendored copy would.
|
||||
#
|
||||
# Every future controller customization (vertical scaling, tolerations, extra feature gates,
|
||||
# --concurrent) belongs HERE, as another entry under `patches:`.
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
# Written by `bootstrap.sh` from `flux install --export ...` at render time. NOT checked in.
|
||||
- gotk-components.yaml
|
||||
patches:
|
||||
# -------------------------------------------------------------------------------------------
|
||||
# StrictPostBuildSubstitutions — make postBuild substitution FAIL CLOSED.
|
||||
#
|
||||
# Default behaviour is fail-OPEN: a `${VAR}` that is not defined in `cluster-vars` renders as the
|
||||
# EMPTY STRING, silently. On 2026-09-01 that shipped `k8sServiceHost: ""` into the Cilium
|
||||
# HelmRelease and put 2 of 3 agents into Init:Error (DECISIONS-LOG, INCIDENT entry). With this
|
||||
# gate the undefined variable FAILS the Kustomization instead, and Flux keeps the last-good state.
|
||||
#
|
||||
# PROVEN SAFE before enabling: every one of the 11 substituted paths was rendered with
|
||||
# `kubectl kustomize --load-restrictor LoadRestrictionsNone <path> | flux envsubst --strict` and
|
||||
# exited 0, with a negative control reproducing the incident. The permanent guard that keeps it
|
||||
# true is the L0 gate conformance/tests/l0-static/flux-postbuild-substitution/.
|
||||
#
|
||||
# NOTE the gate does NOT catch default-providing forms (`${VAR:-d}`, `${VAR:?msg}`, `${VAR:=v}`) —
|
||||
# Flux exempts them by design. That class is what the L0 gate covers; the two are complementary.
|
||||
#
|
||||
# This also pre-pays for the Flux 2.4.0 -> 2.9.x upgrade: strict substitution became DEFAULT-ON in
|
||||
# kustomize-controller v1.9.0, so the upgrade flips this behaviour whether or not we opt in.
|
||||
#
|
||||
# JSON-6902 `add ... /args/-` (append), NOT a strategic merge: a container's `args` is an atomic
|
||||
# list, so a strategic-merge patch would REPLACE every arg `flux install` generates
|
||||
# (--events-addr, --watch-all-namespaces, --log-level, --log-encoding, --enable-leader-election)
|
||||
# and silently drop any the CLI adds in a future version.
|
||||
- target:
|
||||
group: apps
|
||||
version: v1
|
||||
kind: Deployment
|
||||
name: kustomize-controller
|
||||
namespace: flux-system
|
||||
patch: |-
|
||||
- op: add
|
||||
path: /spec/template/spec/containers/0/args/-
|
||||
value: --feature-gates=StrictPostBuildSubstitutions=true
|
||||
@@ -1,30 +1,11 @@
|
||||
#!/usr/bin/env bash
|
||||
# bootstrap/gitea/bootstrap-gitea.sh — run a SINGLE Gitea container on the HOST (outside the
|
||||
# cluster) as the GitOps bootstrap source. While a cluster is built, Flux (in-cluster) reconciles
|
||||
# this repo from the host Gitea. This is the "local bootstrap source" that migrates to in-cluster
|
||||
# Gitea at cutover (docs/05 "Source-of-truth", CONTRIBUTING.md). Dev has already cut over; PROD is
|
||||
# using it now.
|
||||
# cluster) as the Phase-1 GitOps bootstrap source. While the cluster is built, Flux (in-cluster)
|
||||
# reconciles this repo from the host Gitea over the k3s-dev NAT gateway. This is the "local
|
||||
# bootstrap source" that migrates to in-cluster Gitea in Phase 6 (docs/05 "Source-of-truth",
|
||||
# CONTRIBUTING.md).
|
||||
#
|
||||
# HOW THE CLUSTER REACHES THE HOST — read this before setting HOST_NAT_IP.
|
||||
# It is an ADDRESS OF THIS HOST as seen from the cluster nodes, and it differs per environment:
|
||||
#
|
||||
# PROD (bare metal, current use): HOST_NAT_IP=192.168.2.132
|
||||
# The prod nodes (192.168.2.10/.11/.12) and this workstation are on the SAME LAN. There is
|
||||
# NO NAT hop of any kind, and NO firewall change was needed — verified 2026-09-04 from all
|
||||
# three nodes. The firewalld note further down is a DEV-ERA libvirt concern; it does not
|
||||
# apply to prod, so do not "fix" prod by opening zones.
|
||||
#
|
||||
# DEV (libvirt VMs, HISTORICAL): HOST_NAT_IP=10.73.0.1 was the k3s-dev libvirt NAT gateway.
|
||||
# *** THAT NETWORK IS RETIRED. 10.73.0.1 is stale and must not be reused. *** Dev's own
|
||||
# bootstrap Gitea was later reached on 192.168.2.103:3000 and is retained STOPPED as
|
||||
# recovery (see the comment on the GitRepository patch in clusters/dev/kustomization.yaml).
|
||||
#
|
||||
# WHO RUNS THIS: the OPERATOR, or an agent the operator has EXPLICITLY DIRECTED to stand up a
|
||||
# bootstrap source for a specific bring-up (as happened 2026-09-04 for prod). It is NOT part of
|
||||
# author-only work and must never be fired off casually or "to see if it works": it starts a
|
||||
# long-lived container, generates and writes admin credentials to $HOME, creates an org and repo,
|
||||
# and PUSHES THE CURRENT HEAD to that repo's `main` — which is the revision a cluster pointed at
|
||||
# it will then reconcile. Know which HEAD you are publishing before you run it.
|
||||
# Run by the OPERATOR at bring-up — NOT during author-only, NOT by any agent.
|
||||
set -euo pipefail
|
||||
ENGINE="${ENGINE:-podman}"
|
||||
GITEA_IMAGE="${GITEA_IMAGE:-docker.io/gitea/gitea:1.22}" # pinned; bump if needed
|
||||
@@ -35,24 +16,11 @@ SSH_PORT="${GITEA_SSH_PORT:-2222}"
|
||||
# subuid that can't write a host-owned dir, so Gitea fails to create /data/gitea/conf/app.ini.
|
||||
DATA_VOLUME="${GITEA_DATA_VOLUME:-gitea-bootstrap-data}"
|
||||
DATA_DIR="podman volume ${DATA_VOLUME}" # display label only (data lives in the named volume)
|
||||
# The address the CLUSTER uses to reach this host (see the header). ROOT_URL must match it so
|
||||
# The cluster VMs reach the host on the k3s-dev NAT gateway (10.73.0.1). ROOT_URL must match so
|
||||
# clone/redirect URLs are correct from inside the cluster.
|
||||
#
|
||||
# The DEFAULT IS DELIBERATELY UNSET — there is no correct default any more. The old default
|
||||
# (10.73.0.1) was the retired k3s-dev libvirt NAT gateway; leaving it in place would silently
|
||||
# publish a ROOT_URL no cluster can resolve, and Flux would report what looks like a network fault.
|
||||
# Pass it explicitly, e.g. for prod: HOST_NAT_IP=192.168.2.132 bash bootstrap-gitea.sh
|
||||
#
|
||||
# NOTE (DEV/libvirt ONLY, historical): firewalld's `libvirt` zone (owns virbrN) blocked
|
||||
# VM->host:3000 by default and had to be opened once with
|
||||
# NOTE: firewalld's `libvirt` zone (owns virbrN) blocks VM->host:3000 by default. Open it once:
|
||||
# sudo firewall-cmd --zone=libvirt --add-port=3000/tcp --permanent && sudo firewall-cmd --reload
|
||||
# PROD needs NO firewall change: same-LAN, no libvirt zone involved (verified 2026-09-04).
|
||||
if [ -z "${HOST_NAT_IP:-}" ]; then
|
||||
echo "ERROR: HOST_NAT_IP is not set — it is this host's address AS SEEN FROM THE CLUSTER." >&2
|
||||
echo " prod: HOST_NAT_IP=192.168.2.132 (same LAN as the nodes, no NAT, no firewall change)" >&2
|
||||
echo " There is no safe default: the old one (10.73.0.1) is the RETIRED libvirt NAT." >&2
|
||||
exit 1
|
||||
fi
|
||||
HOST_NAT_IP="${HOST_NAT_IP:-10.73.0.1}"
|
||||
ORG="${GITEA_ORG:-Home-Network}"
|
||||
REPO="${GITEA_REPO:-home-network-k3s-impl}"
|
||||
|
||||
@@ -107,17 +75,8 @@ cat <<EOF
|
||||
|
||||
Gitea bootstrap source ready (${DATA_DIR}).
|
||||
UI (host): ${BASE}/${ORG}/${REPO}
|
||||
Flux GitRepository (cluster): http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git
|
||||
(patch this url into the overlay being bootstrapped:
|
||||
clusters/prod today; clusters/dev has already cut over)
|
||||
Flux GitRepository (cluster): http://${HOST_NAT_IP}:${HTTP_PORT}/${ORG}/${REPO}.git (clusters/dev)
|
||||
Admin creds (host, 0600): ${ADMIN_CRED_FILE}
|
||||
Reachability: prod nodes reach ${HOST_NAT_IP}:${HTTP_PORT} directly on the LAN — no NAT, no
|
||||
firewall change (verified 2026-09-04). For libvirt-based clusters ONLY, firewalld's libvirt zone
|
||||
must allow VM->host:${HTTP_PORT} —
|
||||
Prerequisite: firewalld must allow VM->host:${HTTP_PORT} —
|
||||
ansible-playbook infra/ansible/playbooks/host-prereqs.yml --ask-become-pass
|
||||
REMINDER: the push above published the CURRENT HEAD as this repo's \`main\`. Whatever cluster points
|
||||
at this source will reconcile THAT revision.
|
||||
TEMPORARY BY DESIGN: this container is outside every cluster and is a single point of failure for
|
||||
any cluster reconciling from it. Cut over to in-cluster Gitea (docs/05); for prod the cutover
|
||||
procedure is written on the GitRepository patch in clusters/prod/kustomization.yaml.
|
||||
EOF
|
||||
+1
-11
@@ -10,7 +10,7 @@ policy, egress, TLS, DNS, monitoring, and update policy from it (see
|
||||
[`docs/13-service-catalog.md`](../../home-network-k3s/docs/13-service-catalog.md) (14-service table +
|
||||
SSO/egress/dependency intent).
|
||||
|
||||
## Current examples
|
||||
## Current examples (Phase 0)
|
||||
| File | docs/13 row | exposure | egress | auth | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| `pihole.yaml` | 6 | internal | internet | forward-auth | admin UI behind oauth2-proxy; config on Longhorn (DR-critical); also platform DNS |
|
||||
@@ -28,16 +28,6 @@ inherit every guarantee automatically.
|
||||
`auth.groups`; `native`/`none` are documented exceptions only.
|
||||
- Irreplaceable state (`class: longhorn` configs/DBs) must set `backup.policy` to an off-cluster
|
||||
schedule — never `reacquirable` (R-STATE-03). Only bulk media may be `reacquirable`.
|
||||
- **Shell braces must be escaped `$${...}`.** Anything rendered verbatim onto the workload — a
|
||||
`lifecycle` postStart script, an `env` value — lands under `apps/wave-<N>/`, and every
|
||||
`apps-wave-<N>` Flux Kustomization declares `postBuild.substituteFrom: cluster-vars`. Flux runs
|
||||
envsubst over the WHOLE manifest at reconcile time, so an unescaped `${VAR:-}` meant for the
|
||||
*shell* is replaced with the empty string before the container ever starts (this silently killed
|
||||
gitea's and nextcloud's OIDC registration for days). Bare `$VAR` and `$(cmd)` are left alone and
|
||||
need no escaping; a genuine environment knob is declared in `clusters/base/cluster-vars.yaml` and
|
||||
written bare `${KNOB}`. Full contract: [`../clusters/base/apps-vms.yaml`](../clusters/base/apps-vms.yaml).
|
||||
`render.py` REFUSES to emit a manifest that breaks this rule (Flux's *strict* substitution mode
|
||||
cannot: it deliberately exempts the `${V:-}` / `${V:=x}` / `${V:?}` default-operator forms).
|
||||
|
||||
## Validate before commit — single command, no cluster (R-TEST-02)
|
||||
```bash
|
||||
|
||||
+6
-52
@@ -24,9 +24,9 @@ env:
|
||||
name: gitea-bootstrap
|
||||
key: secret-key
|
||||
- name: GITEA__server__DOMAIN # correct public clone/redirect URLs for the TLS-terminated host
|
||||
value: gitea.${BASE_DOMAIN}
|
||||
value: gitea.home.keithjackson.dev
|
||||
- name: GITEA__server__ROOT_URL
|
||||
value: https://gitea.${BASE_DOMAIN}/
|
||||
value: https://gitea.home.keithjackson.dev/
|
||||
- name: GITEA__actions__ENABLED # Phase 6 CI/CD: declared (not implicit upstream default) so the
|
||||
value: "true" # in-cluster act_runner (ns gitea-actions) registers against it
|
||||
- name: GITEA__actions__DEFAULT_ACTIONS_URL # resolve `uses: <org>/<action>@ref` from github.com (cluster
|
||||
@@ -104,24 +104,12 @@ lifecycle:
|
||||
# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker
|
||||
# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays
|
||||
# usable). The verify-then-retry loop is the real guarantee against "exit 0 but nothing stuck".
|
||||
GITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
|
||||
GITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
|
||||
gitea_oidc_fail() {
|
||||
echo "[gitea-oidc] FAILED to register 'keycloak' login source: $1" >&2
|
||||
{ echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true
|
||||
}
|
||||
# NOTE the `$$` escape below — it is load-bearing. apps-wave-0 declares postBuild.
|
||||
# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE
|
||||
# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not
|
||||
# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z "" ]` — ALWAYS TRUE
|
||||
# — making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on
|
||||
# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was
|
||||
# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.
|
||||
# Flux STRICT substitution cannot catch this — it EXEMPTS default-operator forms by design.
|
||||
# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.
|
||||
# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping — which is why the
|
||||
# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.
|
||||
# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.
|
||||
if [ -z "$${GITEA_OIDC_CLIENT_SECRET:-}" ]; then
|
||||
if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then
|
||||
echo "[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)"
|
||||
else
|
||||
# (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).
|
||||
@@ -162,13 +150,7 @@ lifecycle:
|
||||
gitea_oidc_fail "add/update-oauth ran but the 'keycloak' source did not persist after retries"
|
||||
fi
|
||||
fi
|
||||
# A postStart hook MUST exit 0 — a non-zero exit makes the kubelet KILL the container
|
||||
# (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but
|
||||
# it is made EXPLICIT so the guarantee does not depend on whatever statement happens to
|
||||
# be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,
|
||||
# the moment its OIDC guard was fixed and the block actually ran for the first time.
|
||||
exit 0
|
||||
exposure: public # edge-forwarded git host at gitea.${BASE_DOMAIN} (docs/09; R-DNS/R-NET)
|
||||
exposure: public # edge-forwarded git host at gitea.home.keithjackson.dev (docs/09; R-DNS/R-NET)
|
||||
egress: internet # normal WAN egress (docs/03; R-NET-05)
|
||||
storage:
|
||||
- name: data
|
||||
@@ -178,36 +160,8 @@ storage:
|
||||
# the default /data/<name> would strand real state on the ephemeral rootfs (R-STATE-01)
|
||||
dependencies: [] # default-deny: reaches only DNS; git clients reach it north-south via the Gateway (R-NET-01/02)
|
||||
ingressRoutes:
|
||||
- host: gitea.${BASE_DOMAIN}
|
||||
- host: gitea.home.keithjackson.dev
|
||||
port: 3000 # Gitea HTTP web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
|
||||
# THIS IS WHAT KEEPS PROD'S GITOPS OUT OF A DEADLOCK — it is not an incidental allowance. Remove it
|
||||
# and prod cannot reconcile AT ALL, in a way that is very hard to diagnose from cold.
|
||||
#
|
||||
# Prod's Flux source is THIS SERVICE, not a public hostname:
|
||||
# clusters/prod/kustomization.yaml -> GitRepository.spec.url =
|
||||
# http://gitea.gitea.svc.cluster.local:3000/Home-Network/home-network-k3s-impl.git
|
||||
# chosen deliberately so prod's GitOps depends on NEITHER public DNS, Cloudflare, prod's own
|
||||
# Gateway, NOR ${BASE_DOMAIN} — a future domain change cannot sever prod from its own source. That
|
||||
# file documents the other half of this pair at length; the two must be read together.
|
||||
#
|
||||
# The only other ingress this entry renders is gitea-allow-gateway-ingress, which admits
|
||||
# `fromEntities: [ingress]` — the shared Gateway's Envoy proxy — and nothing else. Flux's
|
||||
# source-controller dials the ClusterIP DIRECTLY, so under the namespace default-deny (R-NET-01) its
|
||||
# clone was DROPPED (`wget` from source-controller timed out; srcReady=False, "unable to list
|
||||
# remote"). The deadlock that made this urgent: the HTTPRoute that would have served the new public
|
||||
# name after the 2026-09-13 apex flip lives in a DOWNSTREAM tier, which cannot apply while the
|
||||
# source is down — source down -> routes never rename -> source stays down. It was broken on
|
||||
# 2026-09-13 with a live `kubectl apply` of a hand-written gitea-allow-flux-ingress, i.e. with a
|
||||
# cluster object that existed OUTSIDE git. This declaration renders that object, so one
|
||||
# `kubectl delete` or one cluster rebuild can no longer put prod back in the deadlock.
|
||||
inClusterClients:
|
||||
# `name` is pinned to `flux` (rather than defaulting to the namespace) so the rendered policy name
|
||||
# equals the live object's, `gitea-allow-flux-ingress`, and Flux ADOPTS it in place: no
|
||||
# near-duplicate policy beside it, and no window in which prod's own source has no allow.
|
||||
- name: flux
|
||||
namespace: flux-system # matched by NAMESPACE alone — the flux controllers' pod labels belong
|
||||
# to upstream's manifests, not to this platform's contract
|
||||
port: 3000 # ONLY the Gitea HTTP port — the same single port the Gateway allow opens
|
||||
auth:
|
||||
mode: oidc # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC" set; R-IDP-02)
|
||||
groups:
|
||||
|
||||
+2
-22
@@ -17,40 +17,20 @@ name: jackett
|
||||
kind: container
|
||||
source:
|
||||
image: lscr.io/linuxserver/jackett:latest
|
||||
exposure: internal # LAN-only web UI at jackett.${BASE_DOMAIN} (docs/13 row 4; R-DNS/R-NET)
|
||||
exposure: internal # LAN-only web UI at jackett.home.keithjackson.dev (docs/13 row 4; R-DNS/R-NET)
|
||||
egress: vpn # indexer scraping via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
|
||||
storage:
|
||||
- name: config
|
||||
size: 2Gi
|
||||
class: longhorn # indexer definitions/API keys on 3-replica Longhorn (docs/04; R-STATE-02)
|
||||
mountPath: /config # THE APP WRITES /config, NOT the renderer default /data/config.
|
||||
# LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
|
||||
# /config), so without this the Longhorn PVC mounted at /data/config sat
|
||||
# holding nothing but lost+found while the app wrote its entire
|
||||
# configuration to the container's EPHEMERAL writable layer — destroyed on
|
||||
# every pod recreation: restart, reschedule, image bump. Measured in-cluster
|
||||
# 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
|
||||
# dev=2053 on the node's own disk with [Sentry asp config.xml logs].
|
||||
# This is the SAME failure catalog/plex.yaml:54 already records fixing.
|
||||
# The live /config contents were copied onto each PVC before this landed,
|
||||
# so API keys and credentials other apps reference were preserved.
|
||||
dependencies: [] # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
|
||||
ingressRoutes:
|
||||
- host: jackett.${BASE_DOMAIN}
|
||||
- host: jackett.home.keithjackson.dev
|
||||
port: 9117 # Jackett WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
|
||||
auth:
|
||||
mode: forward-auth # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)
|
||||
groups:
|
||||
- media-users # authenticated AND in an authorized group (R-IDP-06)
|
||||
# platform-admins is a SUPERSET, not a widening. --allowed-group is repeatable and OR-ed, so
|
||||
# listing both means "media consumers OR platform operators". Without it the media UIs are
|
||||
# reachable by NOBODY the moment media-users is empty: on prod the realm's only user is the
|
||||
# operator, who is in platform-admins only, so jackett/qbittorrent/radarr/sonarr answered an
|
||||
# immediate 403 at the oauth2-proxy callback (no login prompt, because the Keycloak SSO session
|
||||
# already existed) while pihole/crafty — which already gate on platform-admins — worked.
|
||||
# This grants no real privilege: platform-admins already maps to cluster-admin via
|
||||
# k8s-oidc-rbac.yaml (oidc:platform-admins), so a member can already exec into these pods.
|
||||
- platform-admins # operators of these services must not be locked out of them
|
||||
update:
|
||||
channel: stable # R-UPD-01
|
||||
range: ">=0.22.0" # stay on the Jackett 0.22.x stable line
|
||||
|
||||
+1
-11
@@ -18,7 +18,7 @@ dependencies:
|
||||
port: 5432
|
||||
protocol: tcp
|
||||
ingressRoutes:
|
||||
- host: keycloak.${BASE_DOMAIN}
|
||||
- host: keycloak.home.keithjackson.dev
|
||||
port: 8080 # HTTP listener; TLS terminates at the Gateway (docs/08, docs/09)
|
||||
auth:
|
||||
mode: native # it IS the IdP; own admin login is local + MFA (docs/15; ADR-0014). Documented exception.
|
||||
@@ -31,15 +31,5 @@ observability:
|
||||
metrics: true
|
||||
slis:
|
||||
- up # IdP down/unhealthy is a critical alert (docs/10; R-OBS-01)
|
||||
# WHAT "up" IS for Keycloak. The Keycloak OPERATOR runs the server as a StatefulSet named
|
||||
# `keycloak` (pods keycloak-0/1), not as the Deployment this ServiceDefinition would otherwise
|
||||
# imply — the operator owns the workload (platform/keycloak/keycloak-instance.yaml) and this SD
|
||||
# is the catalog's declaration of intent for it. Without this override the rendered SLI would
|
||||
# read a Deployment series that does not exist and its `absent()` half would fire FOREVER.
|
||||
# Verified on prod 2026-09-13: kube_statefulset_status_replicas_ready{namespace="keycloak",
|
||||
# statefulset="keycloak"} = 2, and the only Deployment in that namespace is keycloak-operator.
|
||||
livenessWorkload:
|
||||
kind: StatefulSet
|
||||
name: keycloak
|
||||
backup:
|
||||
policy: daily-offcluster # DR-critical: rebuilds from DB + git realm config (docs/15, docs/04; R-STATE-03)
|
||||
+7
-67
@@ -50,13 +50,13 @@ env:
|
||||
# the vhost answer for the public FQDN; OVERWRITE* make Nextcloud emit https:// self-URLs even
|
||||
# though it speaks plain HTTP behind the proxy. ---
|
||||
- name: NEXTCLOUD_TRUSTED_DOMAINS
|
||||
value: nextcloud.${BASE_DOMAIN}
|
||||
value: nextcloud.home.keithjackson.dev
|
||||
- name: OVERWRITEHOST
|
||||
value: nextcloud.${BASE_DOMAIN}
|
||||
value: nextcloud.home.keithjackson.dev
|
||||
- name: OVERWRITEPROTOCOL
|
||||
value: https
|
||||
- name: OVERWRITECLIURL
|
||||
value: https://nextcloud.${BASE_DOMAIN}
|
||||
value: https://nextcloud.home.keithjackson.dev
|
||||
# --- Keycloak native OIDC SSO (auth.mode oidc; docs/15 "Native OIDC"; R-IDP-02/06). The confidential
|
||||
# `nextcloud` realm client's secret is mirrored into this namespace's nextcloud-bootstrap Secret
|
||||
# (oidc-client-secret; same value SOPS-held in keycloak-realm-clients). The postStart hook reads it
|
||||
@@ -88,7 +88,7 @@ lifecycle:
|
||||
# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as
|
||||
# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.
|
||||
OCC() { su -s /bin/sh www-data -c "php /var/www/html/occ $*"; }
|
||||
NEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
|
||||
NEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration
|
||||
nextcloud_oidc_fail() {
|
||||
echo "[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1" >&2
|
||||
{ echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true
|
||||
@@ -111,19 +111,7 @@ lifecycle:
|
||||
# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.
|
||||
OCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \
|
||||
|| echo "[nextcloud-oidc] WARN: failed to set allow_local_remote_servers" >&2
|
||||
# NOTE the `$$` escape below — it is load-bearing. apps-wave-1 declares postBuild.
|
||||
# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE
|
||||
# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is
|
||||
# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z "" ]` — ALWAYS
|
||||
# TRUE — so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never
|
||||
# installing/enabling user_oidc nor registering the provider. SSO kept working only because
|
||||
# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with
|
||||
# NO OIDC at all. Flux STRICT substitution cannot catch this — it EXEMPTS default-operator
|
||||
# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion
|
||||
# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping — which is why
|
||||
# `--clientsecret-env` and the bare uses below always worked and only the guard broke.
|
||||
# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.
|
||||
if [ -z "$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then
|
||||
if [ -z "${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then
|
||||
echo "[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)"
|
||||
exit 0
|
||||
fi
|
||||
@@ -203,17 +191,7 @@ lifecycle:
|
||||
[ "$missing" = 0 ] && break
|
||||
p=$((p + 1)); sleep 2
|
||||
done
|
||||
if [ "$p" -ge 15 ]; then
|
||||
nextcloud_oidc_fail "could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries"
|
||||
fi
|
||||
# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container
|
||||
# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:
|
||||
# the final statement used to be `[ "$p" -ge 15 ] && nextcloud_oidc_fail ...`, so on the
|
||||
# SUCCESS path the test returned 1 and became the script's exit status. That was latent for
|
||||
# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard
|
||||
# exposed it immediately. Every failure path here logs loudly and writes a marker file and
|
||||
# deliberately does NOT crash the pod — the break-glass local admin must stay usable.
|
||||
exit 0
|
||||
[ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries"
|
||||
exposure: public # edge port-forward + public DNS (docs/09; R-DNS/R-NET)
|
||||
egress: internet # normal WAN egress (docs/03; R-NET-05) — app store, integrity checks
|
||||
storage:
|
||||
@@ -223,50 +201,12 @@ storage:
|
||||
mountPath: /var/www/html # 3-replica tier; irreplaceable, DR-critical (docs/04; R-STATE-02).
|
||||
# The whole html dir is persisted so config.php + user data survive a
|
||||
# rebuild (the default /data/<name> would strand them on ephemeral rootfs).
|
||||
# THE LEGACY NEXTCLOUD DATASET (Storage/nextcloud, 270 G) — MOUNTED READ-ONLY, DELIBERATELY, AND
|
||||
# NOT AS THIS INSTANCE'S DATA DIRECTORY. Read this before changing either decision.
|
||||
#
|
||||
# WHAT IS ACTUALLY IN THERE (measured 2026-09-07, not assumed):
|
||||
# /mnt/Storage/nextcloud/nextcloud/config 103 M
|
||||
# /mnt/Storage/nextcloud/nextcloud/config_old 3.4 G
|
||||
# /mnt/Storage/nextcloud/nextcloud/config_backup 267 G <- 99% of the dataset
|
||||
# There is NO `data/` directory. This is the OLD, non-k3s Nextcloud's configuration plus a backup
|
||||
# blob — an ARCHIVE, not a live user-data tree.
|
||||
#
|
||||
# WHY READ-ONLY. Two independent reasons, either sufficient:
|
||||
# 1. The old instance is STILL LIVE on it. The export is currently shared rw to 192.168.1.6 and
|
||||
# 192.168.1.5, and the operator has said the existing non-k3s prod must remain active. Two
|
||||
# Nextcloud instances writing one instance's config directory, each with its own database and
|
||||
# therefore its own oc_filecache, is a corruption scenario with no clean recovery.
|
||||
# 2. Nothing here needs writing. An archive is read.
|
||||
# "Do not grant more than needed" therefore lands on `ro`. If the operator's intent is in fact a
|
||||
# MIGRATION (read this data in, then decommission the old instance), that is a different and
|
||||
# larger change than a mount, and it should be a deliberate one — flip `readOnly` and update the
|
||||
# export at that point, not before. Enforced by the PV's nfs.readOnly, so it is a real `ro` mount.
|
||||
#
|
||||
# WHY NOT /var/www/html/data. That path is the LONGHORN PVC above and is this instance's own
|
||||
# state; mounting a foreign tree into it would have Nextcloud scan and index an archive it does
|
||||
# not own. A neutral path keeps the two completely separate. To expose it to users, add it as a
|
||||
# Nextcloud External Storage (Local) mount pointing at this path — an in-app, per-user decision.
|
||||
#
|
||||
# subPath skips the redundant single-child directory: the dataset root holds exactly one entry,
|
||||
# `nextcloud/`, so mounting that subtree puts config/ config_old/ config_backup/ at the top.
|
||||
#
|
||||
# UID: nothing to do. The tree is 0755 owned 1000:3000 and the mount is read-only, so www-data
|
||||
# (uid 33) can read and traverse it. (fsGroup would be a no-op on an NFS volume in any case.)
|
||||
- name: legacy-archive
|
||||
class: bulk
|
||||
export: /mnt/Storage/nextcloud # the ZFS dataset Storage/nextcloud, on ${BULK_NFS_SERVER}
|
||||
subPath: nextcloud
|
||||
mountPath: /mnt/legacy-nextcloud
|
||||
readOnly: true
|
||||
size: 270Gi # ADVISORY ONLY on a static NFS PV — the dataset's used size.
|
||||
dependencies:
|
||||
- to: nextcloud-db # explicit allow: Nextcloud -> its PostgreSQL (docs/13 graph; R-NET-02).
|
||||
port: 5432 # Renders the egress half; the DB namespace mirrors keycloak-db (no
|
||||
protocol: tcp # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
|
||||
ingressRoutes:
|
||||
- host: nextcloud.${BASE_DOMAIN}
|
||||
- host: nextcloud.home.keithjackson.dev
|
||||
port: 80 # apache HTTP; wildcard TLS terminates at the shared Gateway (docs/08/09)
|
||||
auth:
|
||||
mode: oidc # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)
|
||||
|
||||
+1
-88
@@ -13,95 +13,8 @@ storage:
|
||||
size: 2Gi
|
||||
class: longhorn # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
|
||||
dependencies: [] # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
# LAN SPLIT-HORIZON DNS — the half of R-DNS-03 that platform/coredns/coredns-custom.yaml defers to.
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
# coredns-custom answers the ${BASE_DOMAIN} zone for PODS ONLY and says so in its own header ("LAN
|
||||
# clients keep their own answers (PiHole split-horizon)"). Until now PiHole served no DNS at all —
|
||||
# its Service was ClusterIP :80/:4180, no port 53, no LoadBalancer — so the LAN half did not exist
|
||||
# and LAN clients depended on hand-maintained per-client dnsmasq files. One went stale after the dev
|
||||
# decommission and cost the operator sonarr.${BASE_DOMAIN}. Upstream cannot cover for it either: the
|
||||
# SOHO router that owns this LAN strips RFC1918 answers for public domains as rebind protection and
|
||||
# returns NOERROR / ANSWER 0. apps/wave-0/pihole-dns.yaml exposes udp/53 + tcp/53 on
|
||||
# ${PIHOLE_DNS_VIP}; these env vars are what make the ANSWERS declarative.
|
||||
#
|
||||
# WHY env AND NOT A RECONCILER: the repo's level-triggered CronJob reconcilers
|
||||
# (platform/keycloak/client-secret-reconciler.yaml, the CNPG grant reconciler in
|
||||
# apps/wave-0/nextcloud-db.yaml) exist because their state has NO declarative entry point — a
|
||||
# Keycloak client secret lives in Keycloak's own database and a Postgres GRANT lives in the
|
||||
# catalog, and nothing in a manifest can express either. PiHole v6 is not that case. Every FTL
|
||||
# setting has an `FTLCONF_<section>_<key>` environment variable, and a value supplied that way is
|
||||
# ENV-FORCED: FTL logs it as "forced through environment", applies it on every start ahead of
|
||||
# /etc/pihole/pihole.toml, and marks it read-only in the admin UI so a human cannot drift it. That
|
||||
# is strictly better than a reconciler — it is enforced at t=0 on a cold start rather than at the
|
||||
# next cron tick, it cannot be defeated by the UI, and it carries no extra RBAC, image or Job. A
|
||||
# CronJob here would be ceremony around a knob the app already exposes. (Verified on the live prod
|
||||
# image, PiHole Core v6.4.3 / FTL v6.7, 2026-09-14.)
|
||||
#
|
||||
# THE PVC IS NOT THE MECHANISM AND MUST NOT BE. The `config` volume below mounts at /data/config;
|
||||
# PiHole's actual state directory is /etc/pihole, which is EPHEMERAL in this Deployment. Anything
|
||||
# configured through the UI is lost on the next reschedule — which is exactly what happened to the
|
||||
# upstream servers: the live pod's pihole.toml carried a hand-set `upstreams` marked "### CHANGED"
|
||||
# that no manifest declared, so a restart would have left PiHole with upstreams = [] and no
|
||||
# recursion at all. Declaring upstreams here closes that too: PiHole must stay a usable GENERAL
|
||||
# resolver, not only authoritative for this zone, or pointing LAN clients at it breaks the internet.
|
||||
env:
|
||||
# Default is LOCAL, which answers only clients on the same subnet as the listening interface. The
|
||||
# pod's interface is on the CLUSTER pod CIDR, so every LAN client would be refused. ALL is
|
||||
# required for a LoadBalancer-fronted resolver; what keeps PiHole from being an OPEN resolver is
|
||||
# the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}]` on the DNS Services,
|
||||
# enforced by Cilium in the datapath before load-balancing, plus a CiliumNetworkPolicy that admits
|
||||
# nothing but udp/53 and tcp/53. See apps/wave-0/pihole-dns.yaml.
|
||||
- name: FTLCONF_dns_listeningMode
|
||||
value: "ALL"
|
||||
# Recursion for everything that is NOT this zone. Public resolvers on purpose, NOT the SOHO
|
||||
# router at the LAN gateway: that router is the box whose rebind protection strips RFC1918
|
||||
# answers, and forwarding to a device that may forward back here invites a loop.
|
||||
- name: FTLCONF_dns_upstreams
|
||||
value: "8.8.8.8;8.8.4.4"
|
||||
# Per-client query rate limit OFF (default: 1000 queries per 60s, then REFUSED). It has to be off
|
||||
# because of how the DNS Services are exposed: apps/wave-0/pihole-dns.yaml uses
|
||||
# externalTrafficPolicy Cluster — it must, or LAN resolution breaks whenever Cilium's L2
|
||||
# announcement lease and the pihole pod land on different nodes, which is most of the time — and
|
||||
# Cilium runs in SNAT mode here, so a query forwarded between nodes reaches PiHole carrying the
|
||||
# INGRESS NODE's address instead of the client's. Every LAN device therefore counts against ONE
|
||||
# apparent client's budget, and a normal evening of browsing from a handful of machines would trip
|
||||
# a limit meant to catch one misbehaving host. Leaving it at the default turns the whole LAN's DNS
|
||||
# off for 60 seconds at a time, intermittently, which is the worst possible failure shape. The
|
||||
# limit becomes meaningful again — and should be restored — if Cilium is ever switched to
|
||||
# `loadBalancer.mode: dsr`, which preserves the client address.
|
||||
- name: FTLCONF_dns_rateLimit_count
|
||||
value: "0"
|
||||
# THE SPLIT-HORIZON ANSWERS. A dnsmasq `address=/<domain>/<ip>` entry answers that domain AND
|
||||
# every subdomain of it, and dnsmasq resolves overlapping entries by LONGEST MATCHING DOMAIN —
|
||||
# so the first line is a wildcard for the whole zone and each later line is a specific override.
|
||||
# PRECEDENCE IS THEREFORE STRUCTURAL, not order-dependent: `unifi.${BASE_DOMAIN}` matches both
|
||||
# line 1 and line 2, line 2 is the longer domain, line 2 wins. Rewriting these in any order
|
||||
# changes nothing.
|
||||
#
|
||||
# A NEWLY ONBOARDED SERVICE NEEDS NO CHANGE HERE. Every catalog ingressRoute host is
|
||||
# `<label>.${BASE_DOMAIN}` served by the shared Gateway, so the wildcard already answers it with
|
||||
# ${GATEWAY_VIP} — the same guarantee, and the same reasoning, as the `template IN A` block in
|
||||
# platform/coredns/coredns-custom.yaml. Add a line here ONLY for a service that does not live
|
||||
# behind the Gateway: an L4 LoadBalancer on its own pinned VIP. Those are enumerated from the
|
||||
# repo, not from memory — grep `external-dns.alpha.kubernetes.io/hostname` on a Service, which
|
||||
# is the annotation that means "this Service publishes its own hostname at its own address":
|
||||
# apps/vms/unifi/50-service-l4.yaml (unifi -> ${UNIFI_L4_VIP}) and
|
||||
# platform/crafty/service-l4.yaml (crafty-mc -> ${CRAFTY_L4_VIP}). Those two, plus PiHole's own
|
||||
# resolver address, are the complete set of exceptions today.
|
||||
#
|
||||
# AAAA needs nothing: dnsmasq answers a AAAA query for an `address=`d name with NODATA
|
||||
# (NOERROR, no answer), so IPv6-capable clients fall back to the A record instead of getting
|
||||
# NXDOMAIN — the same behaviour coredns-custom hand-writes a `template IN AAAA` block for.
|
||||
#
|
||||
# Array-valued FTLCONF variables are SEMICOLON-separated. A JSON-looking list is accepted and
|
||||
# then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal
|
||||
# characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid"
|
||||
# while resolution for the zone does not work. Confirmed both ways against the live image.
|
||||
- name: FTLCONF_misc_dnsmasq_lines
|
||||
value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}"
|
||||
ingressRoutes:
|
||||
- host: pihole.${BASE_DOMAIN}
|
||||
- host: pihole.home.keithjackson.dev
|
||||
port: 80 # admin web UI, protected at the Gateway by forward-auth (docs/09)
|
||||
auth:
|
||||
mode: forward-auth # oauth2-proxy -> Keycloak; bare UI, no native OIDC (docs/15; R-IDP-02)
|
||||
|
||||
+25
-71
@@ -4,11 +4,21 @@
|
||||
# auth: native (Plex accounts, NOT Keycloak — docs/13 + docs/15 documented exception).
|
||||
# Validates against schemas/servicedefinition.schema.json.
|
||||
#
|
||||
# BULK MEDIA — NOW ATTACHED (docs/04 bulk tier; the long-standing OQ-2 prerequisite, resolved for
|
||||
# the STATIC case). The library is 8.33 T of ZFS on the Proxmox host at /mnt/Storage/plex, and it
|
||||
# already exists: nothing is provisioned, an existing dataset is mounted. See
|
||||
# docs/BULK-NFS-STORAGE.md for the layout, the export change and the uid analysis. (The bulk-tier
|
||||
# requirement ID is deliberately still not typed here — see that document's "gate" note.)
|
||||
# BULK MEDIA (spec bulk-tier requirement, docs/04) — OPEN PREREQUISITE, intentionally NOT declared
|
||||
# here yet (the requirement ID is cited in the WS-APPS report, not as a token here: this change does
|
||||
# NOT exercise the bulk tier, so it must not be selected as a "touched" requirement whose in-cluster
|
||||
# conformance test is still a Phase-1 gap).
|
||||
# The Plex library belongs on the `bulk` tier (docs/04 / docs/13 cross-cutting note: metadata DB
|
||||
# on replicated Longhorn, large library on bulk). The `bulk` StorageClass exists
|
||||
# (platform/longhorn/storageclasses.yaml) and the schema+renderer both support `class: bulk`, but
|
||||
# that class is a documented STUB: OQ-2 (external NFS/SMB backing) is UNRESOLVED and the NFS CSI
|
||||
# driver is not installed, so a `bulk` PVC will not bind and would leave the Plex pod Pending.
|
||||
# To onboard the running Plex shell now (compute + config + public exposure), the media volume is
|
||||
# omitted. Once OQ-2 is resolved and `bulk` provisions, attach the library with a one-line add:
|
||||
# storage:
|
||||
# - name: media
|
||||
# class: bulk # external mount; size omitted (sized outside the cluster)
|
||||
# then re-render. See the WS-APPS report for the full bulk-media finding.
|
||||
name: plex
|
||||
kind: container
|
||||
source:
|
||||
@@ -32,40 +42,11 @@ env:
|
||||
# (10.43.0.0/16); 192.168.0.0/16 covers the home LAN.
|
||||
- name: ALLOWED_NETWORKS
|
||||
value: 10.0.0.0/8,192.168.0.0/16
|
||||
# PLEX_CLAIM — the ONE-SHOT plex.tv server claim, sourced from the SOPS Secret `plex-claim`
|
||||
# (apps/wave-0/plex-claim-secret.sops.yaml, same wave => applied in the same reconcile as this
|
||||
# workload). Plex cannot create libraries over the mounted bulk share while the server is
|
||||
# UNCLAIMED, so this is wired; it was previously left out on the reasoning below, and only one
|
||||
# half of that reasoning survived contact with the upstream source.
|
||||
#
|
||||
# THE 4-MINUTE TTL IS STILL TRUE, and it is now the OPERATIONAL constraint rather than a reason to
|
||||
# abstain. A https://plex.tv/claim token expires in ~4 minutes — far shorter than commit -> push ->
|
||||
# Flux reconcile -> pod-roll if that cycle is left to its own timers. So the committed value is a
|
||||
# documented PLACEHOLDER and the live token is injected by the operator inside that window, with an
|
||||
# explicit `flux reconcile` to collapse the wait: runbooks/plex-claim.md is the timed procedure.
|
||||
#
|
||||
# ONE-SHOT, AND SAFE TO SIT HERE EXPIRED. Verified from plexinc/pms-docker's own first-run script
|
||||
# (branch `master`, root/etc/cont-init.d/40-plex-first-run, confirmed byte-identical to the file in
|
||||
# the running `:latest` image), NOT assumed:
|
||||
# - the exchange (line 106) is guarded on TWO conditions: PLEX_CLAIM non-empty AND the
|
||||
# PlexOnlineToken pref in /config empty. Once claimed, the block is SKIPPED — this env is a
|
||||
# no-op on every subsequent start, whatever it holds. (Upstream README: "If server is already
|
||||
# logged in, this parameter is ignored.") The claim itself persists on the /config PVC as before.
|
||||
# - an invalid/expired token on a still-unclaimed /config gets a 403 with an empty body; curl's
|
||||
# status is unchecked, the token `sed` yields "", the success branch at line 120 does not fire,
|
||||
# there is no `else` and no `exit 1`, and the script (no `set -e`) exits 0. It comes up UNCLAIMED and
|
||||
# SERVING — it does NOT crashloop. The image also leaves S6_BEHAVIOUR_IF_STAGE2_FAILS at its
|
||||
# default 0, which continues startup even when a cont-init script fails.
|
||||
# So the earlier "a placeholder claim Secret would crashloop the pod" note was WRONG about the
|
||||
# placeholder; it was right only about an ABSENT Secret. secretKeyRef here has no `optional` field,
|
||||
# so the Secret must always exist in the Flux graph — which is why it is committed, encrypted, and
|
||||
# listed in apps/wave-0/kustomization.yaml rather than hand-created at claim time. No schema change
|
||||
# is needed: `optional` would only matter if a missing Secret had to be tolerated, and it doesn't.
|
||||
- name: PLEX_CLAIM
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: plex-claim
|
||||
key: PLEX_CLAIM
|
||||
# PLEX_CLAIM is deliberately NOT set here: a https://plex.tv/claim token has a ~4-min TTL, far
|
||||
# shorter than the commit -> push -> Flux reconcile -> pod-roll cycle, so any token committed to git
|
||||
# is already expired by the time the pod starts. Claiming is a one-time LIVE step (UI sign-in, or a
|
||||
# single API POST) against the running server; the claim then persists on the /config PVC and
|
||||
# survives pod rolls (including the roll this env change causes), so it never belongs in git.
|
||||
# ADVERTISE_IP ("Custom server access URLs") is DEFERRED, not accidentally omitted. It tells Plex
|
||||
# which URL to advertise to plex.tv for REMOTE clients. The only candidate is
|
||||
# https://plex.home.keithjackson.dev, but that host resolves via split-horizon DNS to the INTERNAL
|
||||
@@ -85,40 +66,13 @@ storage:
|
||||
# container's EPHEMERAL layer -> unclaimed + all state lost on restart. The image inits as
|
||||
# root (s6-svscan PID 1), chowns /config to the plex user, then drops -> no fsGroup needed
|
||||
# (verified in-cluster: PID1=root, "Plex Media Server"=plex). Persisting /config here also
|
||||
# makes the claim survive restarts once claimed — which is what makes the PLEX_CLAIM env
|
||||
# above a ONE-SHOT: the token is exchanged for a PlexOnlineToken written here, and the
|
||||
# exchange is then skipped on every later start. (The older note here claimed a
|
||||
# placeholder claim Secret "would crashloop the pod"; upstream's first-run script says
|
||||
# otherwise — see the PLEX_CLAIM comment in `env` above. An ABSENT Secret is the hard
|
||||
# failure, and that is why one is committed.)
|
||||
# THE LIBRARY (docs/04 bulk tier). One READ-ONLY mount of the export ROOT, giving /media/movies,
|
||||
# /media/tv and /media/photos — the whole tree the layout actually has (`ls /mnt/Storage/plex`:
|
||||
# downloads/ movies/ photos/ tv/).
|
||||
#
|
||||
# WHY THE ROOT AND NOT THREE subPath MOUNTS. Nothing is gained by narrowing: the mount is `ro`,
|
||||
# so no subPath could remove a privilege that matters, while three mounts would triple the NFS
|
||||
# mounts per pod and mean a manifest change every time a top-level library is added.
|
||||
#
|
||||
# WHY READ-ONLY, when the export is `rw` and the other consumers write. Plex is a SERVER: it
|
||||
# indexes and streams. Every write on that tree belongs to Sonarr and Radarr. Marking it here
|
||||
# costs nothing and removes Plex from the set of things that can damage 8.33 T. Enforced by the
|
||||
# PV's nfs.readOnly (a real `ro` mount), not only by the volumeMount flag.
|
||||
#
|
||||
# UID: no change needed. The tree is 0755 owned 1000:3000 throughout, so it is world-readable and
|
||||
# world-traversable, and PLEX_UID/PLEX_GID above already make the process uid 1000 in any case.
|
||||
# (fsGroup would have been a no-op here regardless — the in-tree NFS plugin does no ownership
|
||||
# management. That is the trap the arrs' PUID/PGID exists to avoid; Plex never hits it.)
|
||||
- name: media
|
||||
class: bulk
|
||||
export: /mnt/Storage/plex # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER}
|
||||
mountPath: /media
|
||||
readOnly: true
|
||||
size: 9Ti # ADVISORY ONLY (nothing enforces a static NFS PV's capacity): the
|
||||
# dataset's 8.33 T used, rounded up, so `kubectl get pv` reports
|
||||
# something true. The real limit is the ZFS pool.
|
||||
# makes the claim survive restarts once claimed. To auto-claim on first boot instead of the
|
||||
# UI sign-in, the operator adds a PLEX_CLAIM env from a fresh https://plex.tv/claim token
|
||||
# (4-min TTL) — deliberately NOT wired here: the env secretKeyRef pattern has no `optional`,
|
||||
# so a placeholder/absent claim Secret would crashloop the pod. See operator hand-off.
|
||||
dependencies: [] # default-deny: reaches only DNS; clients/Seerr reach IT (docs/03; R-NET-01/02)
|
||||
ingressRoutes:
|
||||
- host: plex.${BASE_DOMAIN}
|
||||
- host: plex.home.keithjackson.dev
|
||||
port: 32400 # Plex web UI + API served on 32400 (wildcard TLS at the Gateway; docs/08/09)
|
||||
auth:
|
||||
mode: native # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)
|
||||
|
||||
+16
-418
@@ -4,18 +4,19 @@
|
||||
# internal download client | egress VPN + kill-switch | config on Longhorn | auth forward-auth
|
||||
# (bare web UI, no native OIDC).
|
||||
#
|
||||
# BULK DOWNLOADS TIER — DELIVERED 2026-09-12 (the deferral below is CLOSED; kept for the record).
|
||||
# This entry used to ship config-only, with a comment promising to attach the downloads volume "once
|
||||
# OQ-2 is resolved". OQ-2 was resolved for STATIC pre-existing datasets (schema 1.5.0 / renderer
|
||||
# `class: bulk` + `export` -> a static, pre-bound NFS PV), and plex/sonarr/radarr/nextcloud all took
|
||||
# their media mounts — but qBittorrent's was never added, so the DOWNLOAD CLIENT was the one member
|
||||
# of the chain with no media volume at all. Measured on prod 2026-09-12:
|
||||
# Session\DefaultSavePath=/downloads/ Session\TempPath=/downloads/incomplete/
|
||||
# Downloads\SavePath=/downloads/ `ls /downloads` -> No such file or directory
|
||||
# i.e. the app was configured to write a path that DID NOT EXIST, in a container with no volume there.
|
||||
# Anything it did manage to create would have been invisible to sonarr/radarr, lost on every pod
|
||||
# recreation, and filling the NODE's ephemeral disk in the meantime. The `downloads` entry under
|
||||
# storage: below closes that. See docs/BULK-NFS-STORAGE.md and the bulk-tier L0 conformance test.
|
||||
# BULK DOWNLOADS TIER — OPEN PREREQUISITE, intentionally NOT declared here (mirrors plex + sonarr/radarr):
|
||||
# docs/13 row 13 places qBittorrent's downloads/staging volume on the `bulk` tier (large, non-3x-
|
||||
# replicated media handoff to the arr stack). The `bulk` StorageClass exists
|
||||
# (platform/longhorn/storageclasses.yaml) and both the schema and renderer support `class: bulk`, BUT
|
||||
# it is a NON-PROVISIONING STUB pending OQ-2 (its NFS/SMB backing is REPLACE-ME and the NFS CSI driver
|
||||
# is not installed) — a `bulk` PVC would never bind and would leave the pod Pending. Declaring it now
|
||||
# would also pull the bulk-tier requirement into the change's "touched" set, whose in-cluster
|
||||
# conformance test is not yet scaffolded, blocking the merge gate. So the downloads volume is DEFERRED
|
||||
# and this onboarding ships config-only. Once OQ-2 is resolved and `bulk` provisions, attach downloads
|
||||
# with a one-line add under storage:
|
||||
# - name: downloads
|
||||
# class: bulk # external mount; size omitted (sized outside the cluster)
|
||||
# then re-render. See the WS-APPS report for the full bulk-downloads finding.
|
||||
#
|
||||
# VPN-class (docs/13 row 13; docs/03 "VPN-class set"): ALL of this pod's public-internet egress
|
||||
# (peer/tracker/DHT traffic) MUST leave via the node PIA tunnel, never the WAN. `egress: vpn` renders
|
||||
@@ -31,423 +32,20 @@ name: qbittorrent
|
||||
kind: container
|
||||
source:
|
||||
image: lscr.io/linuxserver/qbittorrent:5.1.2 # pinned stable tag (R-UPD-01); update.range governs advancement
|
||||
# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
|
||||
# LOAD-BEARING FOR THE BULK MOUNT, not cosmetic — exactly as on sonarr/radarr. Measured on prod
|
||||
# 2026-09-12 BEFORE this change: `id` inside the container reported the qbittorrent-nox process as
|
||||
# uid=911 gid=911, and /mnt/Storage/plex/downloads is 0755 owned 1000:1000. 0755 grants write to the
|
||||
# OWNER ONLY, so uid 911 would have mounted the export cleanly and then failed every write with
|
||||
# EACCES — the "mounts fine, cannot write" failure the sonarr entry warns about at length.
|
||||
env:
|
||||
- name: PUID
|
||||
value: "1000" # owner uid of /mnt/Storage/plex/downloads (and of every dir under the export)
|
||||
- name: PGID
|
||||
value: "3000" # the export's dominant group, matching sonarr/radarr: files this writes stay
|
||||
# readable to the EXISTING non-k3s consumers of the same dataset, and land
|
||||
# group-owned the same way the arrs' own writes do
|
||||
# --- THE DURABLE WebUI CREDENTIAL (consumed by the postStart hook below) -------------------
|
||||
# Measured on prod 2026-09-13: /config/qBittorrent/qBittorrent.conf held NEITHER `WebUI\Username`
|
||||
# NOR `WebUI\Password_PBKDF2`. qBittorrent 5.x reads that as "no credential configured" and mints a
|
||||
# NEW random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary
|
||||
# password is provided for this session: ..."). Two real consequences, not hypotheticals:
|
||||
# * Sonarr and Radarr store a download-client username/password in their OWN databases. Both
|
||||
# tested valid against the then-current temporary password; both would have started failing at
|
||||
# the NEXT qbittorrent pod restart, silently breaking Completed Download Handling (the arrs
|
||||
# report a failed client, not a failed import).
|
||||
# * The human operator had no stable login: every restart meant reading the pod log.
|
||||
# Provided as apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml (SOPS/age, decrypted by Flux) —
|
||||
# same shape and naming as gitea-bootstrap, so `render.py --emit-waves` places it in this wave.
|
||||
- name: QBT_WEBUI_USERNAME
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: qbittorrent-bootstrap
|
||||
key: webui-username
|
||||
- name: QBT_WEBUI_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: qbittorrent-bootstrap
|
||||
key: webui-password
|
||||
lifecycle:
|
||||
# LEVEL-TRIGGERED CREDENTIAL ASSERTION, run on EVERY container start.
|
||||
#
|
||||
# WHY A postStart HOOK AND NOT A CronJob RECONCILER: the repo's idiom for "state Flux cannot reach
|
||||
# directly" is the level-triggered CronJob (platform/keycloak/client-secret-reconciler.yaml,
|
||||
# apps/wave-0/nextcloud-db.yaml). That idiom's PRINCIPLE — desired state lives in git and is
|
||||
# RE-ASSERTED unconditionally, never set once — is exactly what is applied here, but the trigger is
|
||||
# the container start rather than a clock, because the failure this closes IS a container start:
|
||||
# * The credential is read by qbittorrent-nox ONCE, at process start, from a file on the config
|
||||
# PVC. A 15-minute cron would therefore leave a 0-15 minute window after every restart, PVC
|
||||
# restore or rebuild in which the app serves a random password and both arrs are broken. The
|
||||
# hook has no such window: it runs before/with the very start that would otherwise regenerate.
|
||||
# * A CronJob would also have to reach the conf from OUTSIDE the pod — mounting an RWO Longhorn
|
||||
# PVC that the app already holds, or `kubectl exec`-ing with RBAC it would need granting — and
|
||||
# then edit a file UNDER the running process, whose own save-on-exit can overwrite the edit.
|
||||
# In-container, before the service starts, none of that exists.
|
||||
# * It needs no ServiceAccount, no RBAC, no NetworkPolicy and no API token: strictly less
|
||||
# privilege than the CronJob alternative (R-SEC/least-privilege).
|
||||
# The gitea entry's postStart is the same pattern (idempotent bootstrap of a credential the app
|
||||
# keeps in its own state), and the schema documents this hook for exactly that use.
|
||||
#
|
||||
# AND WHY NOT AN initContainer, which is the genuinely tempting third option: it would run strictly
|
||||
# BEFORE qbittorrent-nox, removing the race the `s6-svc -r` below exists to cover, and its stdout
|
||||
# would reach `kubectl logs -c <init>` instead of the kubelet. Two reasons it loses here:
|
||||
# * It CANNOT do step (3) — the ground-truth login that proves the WebUI actually accepts the
|
||||
# credential — because the WebUI is not running yet. That assertion is the most valuable line
|
||||
# in this whole mechanism: it is what turns "we wrote a file" into "the credential works", and
|
||||
# writing the right PBKDF2 bytes is precisely the part that is easy to get subtly wrong.
|
||||
# * The ServiceDefinition schema (1.5.0) has `lifecycle` but NO initContainers field, so it would
|
||||
# mean a schema version bump plus a renderer change — a blast radius across every service's
|
||||
# golden — to buy something strictly weaker than what the hook already does.
|
||||
# The honest summary: postStart trades a short, self-healing start-up race (detected and closed by
|
||||
# the s6 restart + re-verify loop) for the ability to VERIFY. That is the right trade.
|
||||
#
|
||||
# DURABILITY, the whole point — all three paths end with the SAME credential:
|
||||
# pod restart -> conf on the PVC already holds it; the hook verifies and writes NOTHING.
|
||||
# config-PVC restore from an older backup -> restore requires the pod to be recreated, so the
|
||||
# hook runs and re-asserts over whatever the snapshot contained.
|
||||
# rebuild from git -> empty PVC; the image copies /defaults/qBittorrent.conf, the hook writes the
|
||||
# credential into it before the service is serving. Nothing outside git.
|
||||
#
|
||||
# WHAT IT DOES NOT DO: it never weakens authentication. `WebUI\AuthSubnetWhitelist` (bypassing auth
|
||||
# for the pod CIDR) was REJECTED: it leaves the human login broken, and makes qBittorrent's auth
|
||||
# depend on network policy staying correct forever. The WebUI keeps demanding a password; that
|
||||
# password is simply now known, fixed, and reproducible from git.
|
||||
postStart:
|
||||
exec:
|
||||
command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the
|
||||
# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.
|
||||
#
|
||||
# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.
|
||||
# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over
|
||||
# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;
|
||||
# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE
|
||||
# CLUSTER — but only there:
|
||||
# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl
|
||||
# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during
|
||||
# development) keeps the doubled dollar, which bash reads as its own PID. Not hypothetical;
|
||||
# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`
|
||||
# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY
|
||||
# however it is run.
|
||||
set -u
|
||||
CONF=/config/qBittorrent/qBittorrent.conf
|
||||
SVC=/run/service/svc-qbittorrent
|
||||
# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this
|
||||
# ServiceDefinition — which is the single place the renderer takes the containerPort, the
|
||||
# Service port and the HTTPRoute backend port from — and the image exposes no WEBUI_PORT
|
||||
# env var for a lookup to read anyway.
|
||||
# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and
|
||||
# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,
|
||||
# so PORT became "<pid>{WEBUI_PORT:-8080}" — the brace never expanded, the digits never
|
||||
# matched. Every curl below then got an unparseable URL, returned nothing, and the hook
|
||||
# logged "the WebUI rejected the durable
|
||||
# credential" — having never actually asked it. That MISDIAGNOSIS is preserved in this
|
||||
# service's own marker log at 2026-09-13T05:35Z (three "rejected" lines, then GAVE UP,
|
||||
# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.
|
||||
PORT=8080
|
||||
# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` — so a hook
|
||||
# that only echoes is effectively silent (gitea's hits the same wall and writes a marker
|
||||
# file for it). Every outcome therefore also lands, timestamped, on the persistent config
|
||||
# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the
|
||||
# credential from git was asserted and accepted on this start.
|
||||
log() {
|
||||
echo "[qbt-webui-cred] $*"
|
||||
printf '%s [qbt-webui-cred] %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" \
|
||||
>> /config/qbittorrent-webui-cred.log 2>/dev/null || true
|
||||
}
|
||||
|
||||
# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC
|
||||
# before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of
|
||||
# racing to create one it would then not overwrite.
|
||||
n=0
|
||||
while [ ! -s "$CONF" ] && [ "$n" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done
|
||||
|
||||
waited=0
|
||||
attempt=0
|
||||
while [ "$attempt" -lt 3 ]; do
|
||||
attempt=$((attempt + 1))
|
||||
|
||||
# (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),
|
||||
# 10 = rewritten, anything else = could not write.
|
||||
python3 - "$CONF" <<'PY'
|
||||
import base64, hashlib, hmac, os, stat, sys
|
||||
|
||||
conf = sys.argv[1]
|
||||
user = os.environ.get("QBT_WEBUI_USERNAME", "")
|
||||
pw = os.environ.get("QBT_WEBUI_PASSWORD", "")
|
||||
if not user or not pw:
|
||||
sys.stderr.write("[qbt-webui-cred] credential env is empty - refusing to touch the conf\n")
|
||||
raise SystemExit(1)
|
||||
|
||||
# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000
|
||||
# iterations, 16-byte random salt, 64-byte digest, stored as
|
||||
# WebUI\Password_PBKDF2="@ByteArray(<base64 salt>:<base64 digest>)"
|
||||
# Get any one of them wrong and the app simply rejects every login.
|
||||
ITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16
|
||||
UKEY, PKEY, SECTION = "WebUI\\Username", "WebUI\\Password_PBKDF2", "[Preferences]"
|
||||
|
||||
def matches(raw):
|
||||
"""True iff the stored digest is one OUR password produces (salt is read from it)."""
|
||||
v = raw.strip().strip('"')
|
||||
if v.startswith("@ByteArray(") and v.endswith(")"):
|
||||
v = v[len("@ByteArray("):-1]
|
||||
try:
|
||||
salt_b64, digest_b64 = v.split(":", 1)
|
||||
salt = base64.b64decode(salt_b64, validate=True)
|
||||
digest = base64.b64decode(digest_b64, validate=True)
|
||||
except Exception:
|
||||
return False
|
||||
if not salt or not digest:
|
||||
return False
|
||||
return hmac.compare_digest(
|
||||
hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, len(digest)), digest)
|
||||
|
||||
try:
|
||||
lines = open(conf, encoding="utf-8", errors="replace").read().splitlines()
|
||||
except FileNotFoundError:
|
||||
lines = []
|
||||
|
||||
def value_of(key):
|
||||
for i, line in enumerate(lines):
|
||||
if line.startswith(key + "="):
|
||||
return i, line.split("=", 1)[1]
|
||||
return -1, None
|
||||
|
||||
ui, uval = value_of(UKEY)
|
||||
pi, pval = value_of(PKEY)
|
||||
if uval == user and pval is not None and matches(pval):
|
||||
raise SystemExit(0) # converged - leave the file (and its salt) alone
|
||||
|
||||
salt = os.urandom(SALT_LEN)
|
||||
digest = hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, DIGEST_LEN)
|
||||
stored = base64.b64encode(salt).decode() + ":" + base64.b64encode(digest).decode()
|
||||
desired = [UKEY + "=" + user, PKEY + '="@ByteArray(' + stored + ')"']
|
||||
|
||||
out = [line for i, line in enumerate(lines) if i not in (ui, pi)]
|
||||
if SECTION in out:
|
||||
at = out.index(SECTION) + 1
|
||||
else:
|
||||
if out and out[-1].strip():
|
||||
out.append("")
|
||||
out.append(SECTION)
|
||||
at = len(out)
|
||||
out[at:at] = desired
|
||||
|
||||
try:
|
||||
os.makedirs(os.path.dirname(conf) or ".", exist_ok=True)
|
||||
tmp = conf + ".webui-cred.new"
|
||||
with open(tmp, "w", encoding="utf-8") as fh:
|
||||
fh.write("\n".join(out) + "\n")
|
||||
fh.flush()
|
||||
os.fsync(fh.fileno())
|
||||
# This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would
|
||||
# cost the app the ability to save its own settings, so carry the ownership over.
|
||||
try:
|
||||
st = os.stat(conf)
|
||||
os.chown(tmp, st.st_uid, st.st_gid)
|
||||
os.chmod(tmp, stat.S_IMODE(st.st_mode))
|
||||
except FileNotFoundError:
|
||||
os.chown(tmp, int(os.environ.get("PUID", "1000")),
|
||||
int(os.environ.get("PGID", "1000")))
|
||||
os.chmod(tmp, 0o664)
|
||||
os.replace(tmp, conf) # atomic: the app never reads a half-written conf
|
||||
except OSError as exc:
|
||||
sys.stderr.write("[qbt-webui-cred] cannot write %s: %s\n" % (conf, exc))
|
||||
raise SystemExit(1)
|
||||
raise SystemExit(10)
|
||||
PY
|
||||
rc=$?
|
||||
|
||||
if [ "$rc" = 0 ]; then
|
||||
log "conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)"
|
||||
elif [ "$rc" = 10 ]; then
|
||||
log "asserted WebUI Username + Password_PBKDF2 into $CONF"
|
||||
# postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY
|
||||
# have read the old, credential-less conf and be serving a random temporary password.
|
||||
# Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.
|
||||
# Nothing to do when the service has not started yet - it will read the new file.
|
||||
# Its own counter, NOT the shared `waited` API budget: sharing one counter is how the
|
||||
# 2026-09-13 failure spent the entire verification budget before the first login and
|
||||
# then fired attempts 2 and 3 with no wait left at all.
|
||||
sv=0
|
||||
while [ "$sv" -lt 60 ] && [ "$(s6-svstat -u "$SVC" 2>/dev/null)" != "true" ]; do
|
||||
sv=$((sv + 1)); sleep 1
|
||||
done
|
||||
if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
|
||||
log "qbittorrent-nox was already running - restarting it to load the credential"
|
||||
s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)"
|
||||
sleep 3
|
||||
fi
|
||||
else
|
||||
log "FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep" >&2
|
||||
log "minting a random password every start and BOTH arrs will break. Fix and restart." >&2
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL
|
||||
# (shared ~120s budget across attempts), then spend exactly ONE login per attempt -
|
||||
# at most 3 rejected logins, which stays under qBittorrent's default ban threshold.
|
||||
#
|
||||
# "THE API NEVER ANSWERED" IS A DIFFERENT FAULT FROM "THE PASSWORD WAS REJECTED", and
|
||||
# conflating them is what made this hook lie once already (see the PORT comment). An
|
||||
# unreachable WebUI is reported as unreachable, and the hook stops rather than spending
|
||||
# its remaining login attempts to produce two more identical wrong diagnoses.
|
||||
api_ok=0
|
||||
while [ "$waited" -lt 120 ]; do
|
||||
code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' "http://127.0.0.1:$PORT/" 2>/dev/null)
|
||||
if [ -n "$code" ] && [ "$code" != "000" ]; then api_ok=1; break; fi
|
||||
waited=$((waited + 1)); sleep 1
|
||||
done
|
||||
if [ "$api_ok" != 1 ]; then
|
||||
log "the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the" >&2
|
||||
log "credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was" >&2
|
||||
log "never submitted. Do not rotate the credential on the strength of this line." >&2
|
||||
exit 0
|
||||
fi
|
||||
body=$(curl -sS -m 10 \
|
||||
--data-urlencode "username=$QBT_WEBUI_USERNAME" \
|
||||
--data-urlencode "password=$QBT_WEBUI_PASSWORD" \
|
||||
"http://127.0.0.1:$PORT/api/v2/auth/login" 2>/dev/null)
|
||||
if [ "$body" = "Ok." ]; then
|
||||
log "VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'"
|
||||
exit 0
|
||||
fi
|
||||
log "WebUI rejected the durable credential (login said '$body')" >&2
|
||||
# A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean
|
||||
# the running qbittorrent-nox is serving something it read from an EARLIER conf, so
|
||||
# rewriting the same bytes on the next attempt would change nothing. The remedy is to
|
||||
# make the process RE-READ the file, so restart the supervised service and let the next
|
||||
# attempt re-verify. Without this the retry loop could never change its own outcome.
|
||||
if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
|
||||
log "restarting qbittorrent-nox so it re-reads $CONF, then re-verifying" >&2
|
||||
s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)" >&2
|
||||
sleep 3
|
||||
fi
|
||||
done
|
||||
|
||||
log "GAVE UP after $attempt attempts: the WebUI does not accept the credential from git." >&2
|
||||
# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container
|
||||
# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.
|
||||
# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the
|
||||
# app, and the arrs' own download-client health check surfaces a broken credential.
|
||||
exit 0
|
||||
podSecurityContext:
|
||||
# For the LONGHORN config PVC only, and for the same reason radarr/sonarr set it: fsGroup is a NO-OP
|
||||
# on the NFS volume (the in-tree plugin does no ownership management), but on a Longhorn filesystem
|
||||
# the kubelet applies it, so /config becomes group-writable by the now-uid-1000 process without
|
||||
# depending on the image chowning it. (This image DOES chown: init-qbittorrent-config runs
|
||||
# `lsiown -R abc:abc /config`. Belt and braces — the live /config was 911:911 before this change,
|
||||
# and losing write access to it would lose the WebUI password hash and the session state.)
|
||||
fsGroup: 3000
|
||||
exposure: internal # LAN-only web UI at qbittorrent.${BASE_DOMAIN} (docs/13 row 13; R-DNS/R-NET)
|
||||
exposure: internal # LAN-only web UI at qbittorrent.home.keithjackson.dev (docs/13 row 13; R-DNS/R-NET)
|
||||
egress: vpn # ALL public egress via node PIA tunnel + kill-switch (docs/03; ADR-0015; R-NET-05/06)
|
||||
storage:
|
||||
- name: config
|
||||
size: 5Gi
|
||||
class: longhorn # settings/categories/session on 3-replica Longhorn; DR-critical (docs/04; R-STATE-02)
|
||||
mountPath: /config # THE APP WRITES /config, NOT the renderer default /data/config.
|
||||
# LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
|
||||
# /config), so without this the Longhorn PVC mounted at /data/config sat
|
||||
# holding nothing but lost+found while the app wrote its entire
|
||||
# configuration to the container's EPHEMERAL writable layer — destroyed on
|
||||
# every pod recreation: restart, reschedule, image bump. Measured in-cluster
|
||||
# 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
|
||||
# dev=2053 on the node's own disk with [Sentry asp config.xml logs].
|
||||
# This is the SAME failure catalog/plex.yaml:54 already records fixing.
|
||||
# The live /config contents were copied onto each PVC before this landed,
|
||||
# so API keys and credentials other apps reference were preserved.
|
||||
# THE DOWNLOADS AREA — ONE mount of the SAME export sonarr/radarr/plex already consume
|
||||
# (192.168.2.102:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
|
||||
# /downloads, which is precisely where this app is ALREADY configured to write.
|
||||
#
|
||||
# WHY subPath: downloads AND NOT THE EXPORT ROOT AT /media (the shape sonarr/radarr use):
|
||||
# * LEAST PRIVILEGE, AND IT COSTS NOTHING HERE. A download client has no business writing the
|
||||
# LIBRARY. The arrs need the root mount because THEY hardlink downloads/ -> tv/ or movies/ and
|
||||
# link() is a same-MOUNT operation (EXDEV otherwise) — their smallest common ancestor genuinely
|
||||
# IS the export root. qBittorrent performs no such cross-directory link: it writes into the
|
||||
# downloads tree and nothing else, so narrowing to that subtree removes movies/, tv/ and photos/
|
||||
# from this pod's mount namespace without removing any operation it actually performs.
|
||||
# * HARDLINKS STILL WORK, because they are not done here. The arrs still see ONE mount containing
|
||||
# both /media/downloads and /media/tv, so their link() stays intra-mount. Proven live 2026-09-12:
|
||||
# a file created from this pod under /downloads was hardlinked from the sonarr pod between
|
||||
# /media/downloads/... and /media/tv/... with st_nlink 1 -> 2 and no EXDEV.
|
||||
# * IT IS ONE MOUNT, so the L0 test `bulk-tier-distinct-from-replicated` assertion (6) — no
|
||||
# workload mounts one export twice — holds. That assertion is about TWO mounts inside ONE pod;
|
||||
# a single subPath mount is not the EXDEV trap, two are.
|
||||
# * The narrowing is a CLIENT/kubelet guarantee, stated honestly: the NFS server still exports the
|
||||
# whole dataset to this node, so the containment is the mount namespace, not the server's ACLs —
|
||||
# the same class of caveat docs/BULK-NFS-STORAGE.md §5 records for plex's readOnly.
|
||||
#
|
||||
# NOT readOnly — this is the one workload whose whole job is WRITING here.
|
||||
#
|
||||
# KNOWN, DELIBERATE SIDE EFFECT ON SHARED STATE: the image's init-qbittorrent-config runs
|
||||
# `lsiown abc:abc /downloads` (non-recursive) whenever /downloads is a mount, so starting this pod
|
||||
# chgrps /mnt/Storage/plex/downloads from 1000:1000 to 1000:3000 on the SERVER. Benign and verified:
|
||||
# the directory is mode 0755, whose group bits (r-x) are identical to its other bits, so no consumer
|
||||
# — k3s or legacy — gains or loses any access. Recorded because it is a write to state shared with
|
||||
# non-cluster consumers, which should never be a surprise.
|
||||
#
|
||||
# PATH AGREEMENT WITH THE ARRS IS STILL *NOT* CLOSED — see DECISIONS-LOG 2026-09-12 "Remote Path
|
||||
# Mappings". This pod calls the finished file /downloads/<name>; sonarr/radarr call the same bytes
|
||||
# /media/downloads/<name>. Same inode, different absolute path, so Completed Download Handling needs
|
||||
# ONE Remote Path Mapping per arr (remote host `qbittorrent.qbittorrent.svc.cluster.local`, remote
|
||||
# path /downloads/, local path /media/downloads/).
|
||||
#
|
||||
# THOSE MAPPINGS ARE NOW CONFIGURED AND VERIFIED ON PROD (2026-09-12), together with the qBittorrent
|
||||
# download client itself — which did not exist in EITHER arr until then (both
|
||||
# `GET /api/v3/downloadclient` returned `[]`, so completed-download handling had never been wired up
|
||||
# at all). `downloadclient/testall` reports isValid=true in both, and Sonarr's queue was observed
|
||||
# reporting outputPath=/downloads/... with the mapping deleted and /media/downloads/... with it
|
||||
# present — same torrent, same client.
|
||||
#
|
||||
# BUT THEY LIVE IN THE ARRS' OWN DATABASES (app state on their config PVCs), NOT IN GIT: a rebuild
|
||||
# from git alone does NOT restore them and no gate here notices their absence. The alternative —
|
||||
# repointing qBittorrent's SavePath at /media/downloads and mounting this subPath there instead —
|
||||
# buys path agreement at the cost of editing THIS app's config PVC, which is also app state; it was
|
||||
# not chosen because it changes a running download client's save path, which the deferral above
|
||||
# already shows is easy to get wrong. It is nonetheless the RECOMMENDED end state (it makes the
|
||||
# mappings unnecessary and puts the mountPath half of the fix in git) and is awaiting an operator
|
||||
# decision, not a technical blocker.
|
||||
#
|
||||
# RELATED OPEN BLOCKER: this app has no durable WebUI credential (no WebUI\Username /
|
||||
# WebUI\Password_PBKDF2 in qBittorrent.conf), so v5 issues a NEW temporary `admin` password on every
|
||||
# start. The arrs' saved download-client password therefore goes stale on every pod restart. See the
|
||||
# same DECISIONS-LOG entry for the two fixes, neither of which was taken unilaterally.
|
||||
- name: downloads
|
||||
class: bulk
|
||||
export: /mnt/Storage/plex # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER} — the SAME export
|
||||
# plex (ro) and sonarr/radarr (rw) mount; one dataset, one set of inodes
|
||||
subPath: downloads # the handoff subtree ONLY: no movies/, no tv/, no photos/
|
||||
mountPath: /downloads # matches the app's EXISTING Session\DefaultSavePath / TempPath, so no
|
||||
# change to config-PVC app state is needed to make this take effect
|
||||
size: 9Ti # ADVISORY ONLY on a static NFS PV (the ZFS pool is the real limit): the
|
||||
# dataset's 8.4 T used rounded up, stated the same way plex/sonarr/radarr
|
||||
# state it so `kubectl get pv` reports something true
|
||||
dependencies: [] # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02).
|
||||
# STILL EMPTY WITH THE NFS MOUNT ABOVE, AND THAT IS CORRECT, NOT AN
|
||||
# OVERSIGHT. An NFS PersistentVolume is mounted by the NODE KERNEL in the
|
||||
# HOST netns, not by this pod: there is no pod-side socket to 2049, so no
|
||||
# CiliumNetworkPolicy can permit or deny it and none is needed. Verified by
|
||||
# construction — sonarr and radarr have mounted this same export read-write
|
||||
# for weeks and neither has any CNP naming the NFS server or port 2049
|
||||
# (grep apps/wave-1/sonarr.yaml: none). The SAME fact is what keeps the VPN
|
||||
# egress gateway off this traffic (its selector is a podSelector), and
|
||||
# 192.168.2.102 is inside the 192.168.0.0/16 excludedCIDR besides — see
|
||||
# platform/vpn-egress/egress-steering.yaml, "BULK NFS".
|
||||
dependencies: [] # default-deny: reaches only DNS; arr reach it north of its own egress (docs/03; R-NET-01/02)
|
||||
ingressRoutes:
|
||||
- host: qbittorrent.${BASE_DOMAIN}
|
||||
- host: qbittorrent.home.keithjackson.dev
|
||||
port: 8080 # qBittorrent WebUI; protected at the Gateway by forward-auth (docs/09, docs/15)
|
||||
auth:
|
||||
mode: forward-auth # oauth2-proxy -> Keycloak; bare UI, no usable native OIDC (docs/15; R-IDP-02)
|
||||
groups:
|
||||
- media-users # authenticated AND in an authorized group (R-IDP-06)
|
||||
# See catalog/jackett.yaml for the full rationale: --allowed-group is repeatable and OR-ed, so
|
||||
# platform-admins here is a superset, not a widening (that group already carries cluster-admin
|
||||
# via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
|
||||
# immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
|
||||
- platform-admins # operators of these services must not be locked out of them
|
||||
update:
|
||||
channel: stable # R-UPD-01
|
||||
range: ">=5.0.0 <6.0.0" # stay on the qBittorrent 5.x stable line
|
||||
|
||||
+7
-84
@@ -9,90 +9,18 @@ name: radarr
|
||||
kind: container
|
||||
source:
|
||||
image: lscr.io/linuxserver/radarr:5.21.1 # pinned stable v5 tag (R-UPD-01); update.range governs advancement
|
||||
# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
|
||||
# LOAD-BEARING for the bulk mount, not cosmetic: see the storage note below. The default is 911,
|
||||
# which cannot write a 1000:3000-owned 0755 tree.
|
||||
env:
|
||||
- name: PUID
|
||||
value: "1000" # owner uid of every directory under /mnt/Storage/plex
|
||||
- name: PGID
|
||||
value: "3000" # dominant group there; keeps files this writes readable to the
|
||||
# EXISTING non-k3s consumers of the same export, which stay live
|
||||
- name: TZ
|
||||
value: Etc/UTC # same safe default plex uses; the arrs schedule by local time
|
||||
podSecurityContext:
|
||||
# For the LONGHORN config PVC only. fsGroup does nothing on the NFS volume (the in-tree plugin
|
||||
# does no ownership management), but on a Longhorn filesystem the kubelet applies it, which is
|
||||
# what makes /config writable by the now-uid-1000 process WITHOUT relying on the image chowning
|
||||
# /config for itself when PUID changes. Matches PGID so the primary group owns the config too.
|
||||
fsGroup: 3000
|
||||
exposure: internal # LAN-only web UI at radarr.${BASE_DOMAIN} (docs/13 row 3; R-DNS/R-NET)
|
||||
exposure: internal # LAN-only web UI at radarr.home.keithjackson.dev (docs/13 row 3; R-DNS/R-NET)
|
||||
egress: vpn # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
|
||||
# kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
|
||||
storage:
|
||||
- name: config
|
||||
size: 5Gi
|
||||
class: longhorn # app DB / config on the 3-replica tier; DR-critical (docs/04; R-STATE-02)
|
||||
mountPath: /config # THE APP WRITES /config, NOT the renderer default /data/config.
|
||||
# LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
|
||||
# /config), so without this the Longhorn PVC mounted at /data/config sat
|
||||
# holding nothing but lost+found while the app wrote its entire
|
||||
# configuration to the container's EPHEMERAL writable layer — destroyed on
|
||||
# every pod recreation: restart, reschedule, image bump. Measured in-cluster
|
||||
# 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
|
||||
# dev=2053 on the node's own disk with [Sentry asp config.xml logs].
|
||||
# This is the SAME failure catalog/plex.yaml:54 already records fixing.
|
||||
# The live /config contents were copied onto each PVC before this landed,
|
||||
# so API keys and credentials other apps reference were preserved.
|
||||
# THE MEDIA TREE (docs/04 bulk tier) — ONE read-WRITE mount of the export ROOT at /media, giving
|
||||
# /media/movies (the library) and /media/downloads (the import source) UNDER THE SAME MOUNT.
|
||||
#
|
||||
# *** THE ROOT MOUNT IS THE POINT, NOT A CONVENIENCE. *** Radarr imports by HARDLINKING or
|
||||
# atomically renaming a finished download into the library. Both are same-FILESYSTEM operations,
|
||||
# and to the kernel "same filesystem" means SAME MOUNT. The layout on the host is:
|
||||
# /mnt/Storage/plex/movies/ the library
|
||||
# /mnt/Storage/plex/downloads/complete/radarr/ where finished releases land
|
||||
# — siblings under one dataset, so a hardlink between them is legal on the server. But mounting
|
||||
# them as two subPath volumes (subPath: movies at /media/movies, subPath: downloads elsewhere) creates
|
||||
# TWO mounts, and link() across mounts fails EXDEV no matter what the server thinks. Radarr then
|
||||
# silently degrades to copy-then-delete: every import rewrites the whole file over NFS and the
|
||||
# seeding copy is destroyed. Their smallest common ancestor IS the export root, so the root mount
|
||||
# is the MINIMAL mount that preserves hardlinks — least privilege bounded below by a hard
|
||||
# requirement, not a shortcut. (The cost is stated plainly: Radarr can also write /media/tv,
|
||||
# which is sonarr's. Narrowing that would cost hardlinks, which is a worse trade.)
|
||||
#
|
||||
# *** /media, NOT /data — AND THAT IS NOT A STYLE CHOICE. *** The config PVC above renders at the
|
||||
# renderer default /data/config. A media mount at /data would be mounted OVER it and MASK the
|
||||
# config volume entirely; the pod would come up with its app database shadowed by an 8 T media
|
||||
# tree. /media collides with nothing in a LinuxServer image and matches the path plex uses for
|
||||
# the same export, so when qbittorrent is eventually given this mount too, the arrs and the
|
||||
# download client will already agree on absolute paths — which is the other precondition for
|
||||
# hardlinked imports (otherwise every import needs a Remote Path Mapping).
|
||||
#
|
||||
# SEPARATE LATENT BUG, NOT FIXED HERE (deliberately — it changes live on-disk state): /data/config
|
||||
# is almost certainly the wrong place for this PVC in the first place. LinuxServer's arr images run
|
||||
# with `-data=/config`, so the app writes /config — the container's EPHEMERAL layer — while the
|
||||
# Longhorn PVC at /data/config sits unused. That is the EXACT failure plex's own mountPath comment
|
||||
# documents having already been bitten by. Fixing it means starting from an empty config volume, so
|
||||
# it belongs in its own change with the operator present.
|
||||
#
|
||||
# UID/GID — THE FAILURE THIS AVOIDS IS "MOUNTS FINE, CANNOT WRITE". The export is no_root_squash
|
||||
# + sec=sys, so the client's numeric uid is passed through verbatim and the server's ownership is
|
||||
# the whole story. The tree is 0755 owned 1000:3000 (some dirs 1000:1000), so ONLY uid 1000 can
|
||||
# write it. LinuxServer images default to PUID/PGID 911 — which would mount cleanly and then fail
|
||||
# every import with EACCES. The PUID/PGID set above make the process uid 1000, matching the owner.
|
||||
#
|
||||
# AND NOTE fsGroup CANNOT SUBSTITUTE FOR THAT. The in-tree NFS volume plugin performs no
|
||||
# ownership management, so podSecurityContext.fsGroup is a NO-OP on this volume — it is not the
|
||||
# lever here, and reaching for it is how this failure normally gets misdiagnosed. The fsGroup
|
||||
# that IS set above exists for the LONGHORN config PVC, so that changing PUID does not depend on
|
||||
# the image re-chowning /config for itself.
|
||||
- name: media
|
||||
class: bulk
|
||||
export: /mnt/Storage/plex # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER}
|
||||
mountPath: /media # => /media/movies (Root Folder) and /media/downloads (import source)
|
||||
size: 9Ti # ADVISORY ONLY on a static NFS PV: the dataset's 8.33 T used,
|
||||
# rounded up, so `kubectl get pv` reports something true.
|
||||
# NOTE (bulk-media PREREQUISITE): docs/13 row 3 places the media library / downloads volume on the
|
||||
# `bulk` tier (docs/04 bulk storage). The `bulk` StorageClass exists (platform/longhorn/storageclasses.yaml)
|
||||
# but is a NON-provisioning STUB pending OQ-2 (NFS/SMB backing = REPLACE-ME). Adding a `class: bulk`
|
||||
# volume now would leave the PVC (and therefore the pod) Pending. The media/downloads volume is
|
||||
# therefore DEFERRED until OQ-2 is resolved; config-only on longhorn keeps Radarr deployable today.
|
||||
dependencies:
|
||||
- to: qbittorrent # arr → download client (docs/13 dependency graph; R-NET-02 explicit allow)
|
||||
port: 8080
|
||||
@@ -101,17 +29,12 @@ dependencies:
|
||||
port: 9117
|
||||
protocol: tcp
|
||||
ingressRoutes:
|
||||
- host: radarr.${BASE_DOMAIN}
|
||||
- host: radarr.home.keithjackson.dev
|
||||
port: 7878 # Radarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
|
||||
auth:
|
||||
mode: forward-auth # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)
|
||||
groups:
|
||||
- media-users # authenticated AND in the media group (R-IDP-06)
|
||||
# See catalog/jackett.yaml for the full rationale: --allowed-group is repeatable and OR-ed, so
|
||||
# platform-admins here is a superset, not a widening (that group already carries cluster-admin
|
||||
# via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
|
||||
# immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
|
||||
- platform-admins # operators of these services must not be locked out of them
|
||||
update:
|
||||
channel: stable # R-UPD-01
|
||||
range: ">=5.0.0 <6.0.0" # stay on the Radarr v5 stable line
|
||||
|
||||
+46
-1389
File diff suppressed because it is too large.
Load diff
+1
-1
@@ -57,7 +57,7 @@ dependencies:
|
||||
port: 32400
|
||||
protocol: tcp
|
||||
ingressRoutes:
|
||||
- host: seerr.${BASE_DOMAIN}
|
||||
- host: seerr.home.keithjackson.dev
|
||||
port: 5055 # Overseerr web UI + API; wildcard TLS terminates at the shared Gateway (docs/08/09)
|
||||
auth:
|
||||
mode: native # Plex account auth — documented exception, NOT Keycloak (docs/13/15; R-IDP-02)
|
||||
|
||||
+13
-33
@@ -58,7 +58,7 @@ env:
|
||||
- name: SEMAPHORE_ADMIN_NAME
|
||||
value: Semaphore Admin
|
||||
- name: SEMAPHORE_ADMIN_EMAIL
|
||||
value: admin@${BASE_DOMAIN}
|
||||
value: admin@home.keithjackson.dev
|
||||
# --- Persistent access-key encryption key (see header): FIXED base64 32-byte key from SOPS so the
|
||||
# SSH keys / credentials Semaphore stores in the DB stay decryptable across restarts. ---
|
||||
- name: SEMAPHORE_ACCESS_KEY_ENCRYPTION
|
||||
@@ -78,34 +78,16 @@ env:
|
||||
# (util ConfigType `oidc_providers`, env-tagged SEMAPHORE_OIDC_PROVIDERS). The value is a single
|
||||
# JSON blob for the `keycloak` provider — provider_url is the homelab realm issuer (auto-discovery),
|
||||
# client_id=semaphore, redirect_url=/api/auth/oidc/keycloak/redirect, claims map to the realm's
|
||||
# token. The name is NOT a kubelet service-link var (those are SEMAPHORE_SERVICE_HOST /
|
||||
# SEMAPHORE_PORT_*), so no collision. The LOCAL SEMAPHORE_ADMIN above stays break-glass (docs/15).
|
||||
#
|
||||
# PER-ENVIRONMENT URLS (2026-09-07 fix). The blob USED to live whole inside the SOPS-encrypted
|
||||
# semaphore-bootstrap Secret (key `oidc-providers-json`) because it CONTAINS the confidential
|
||||
# client secret. That hid two environment-specific, NON-SECRET URLs — the realm issuer and the
|
||||
# redirect URI — inside ciphertext, where Flux's ${BASE_DOMAIN} postBuild substitution CANNOT
|
||||
# reach them. Both were frozen at the apex `home.keithjackson.dev`, so PROD (BASE_DOMAIN =
|
||||
# prod.home.keithjackson.dev) pointed its OIDC at DEV's Keycloak — unroutable from prod's
|
||||
# 192.168.2.0/24 (the identical root cause as the Headlamp login timeout; see
|
||||
# platform/headlamp/README.md). Dev was only ever "right" because dev IS the apex.
|
||||
#
|
||||
# THE SPLIT: the URLs are public, so the blob is now a PLAINTEXT literal here and gets
|
||||
# ${BASE_DOMAIN}-substituted per environment like every other hostname. Only the confidential
|
||||
# client secret stays SOPS-encrypted, in its own key (`oidc-client-secret`), and is spliced back
|
||||
# in by the KUBELET's dependent-env-var expansion: an env `value` may reference an EARLIER env
|
||||
# var of the same container as $(NAME), and secretKeyRef-sourced vars are eligible. So
|
||||
# SEMAPHORE_OIDC_CLIENT_SECRET MUST STAY IMMEDIATELY ABOVE — the kubelet resolves env in list
|
||||
# order, and a reference to a not-yet-defined name is left as the LITERAL text $(...). It is
|
||||
# fail-loud in the way that matters: if the Secret key is missing the Pod never starts
|
||||
# (CreateContainerConfigError), rather than silently authenticating against nothing. ---
|
||||
- name: SEMAPHORE_OIDC_CLIENT_SECRET
|
||||
# token. It CONTAINS the confidential client secret, so the whole blob is held SOPS-encrypted in the
|
||||
# semaphore-bootstrap Secret (key oidc-providers-json; same secret value SOPS-held in
|
||||
# keycloak-realm-clients) and injected here by reference — never in cleartext. The name is NOT a
|
||||
# kubelet service-link var (those are SEMAPHORE_SERVICE_HOST / SEMAPHORE_PORT_*), so no collision.
|
||||
# The LOCAL SEMAPHORE_ADMIN above stays as the break-glass path (docs/15). ---
|
||||
- name: SEMAPHORE_OIDC_PROVIDERS
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: semaphore-bootstrap
|
||||
key: oidc-client-secret
|
||||
- name: SEMAPHORE_OIDC_PROVIDERS
|
||||
value: '{"keycloak":{"display_name":"Keycloak","provider_url":"https://keycloak.${BASE_DOMAIN}/realms/homelab","client_id":"semaphore","client_secret":"$(SEMAPHORE_OIDC_CLIENT_SECRET)","redirect_url":"https://semaphore.${BASE_DOMAIN}/api/auth/oidc/keycloak/redirect","scopes":["openid","profile","email"],"username_claim":"preferred_username","email_claim":"email","name_claim":"name","order":1}}'
|
||||
key: oidc-providers-json
|
||||
# The semaphoreui/semaphore image runs as the non-root user UID 1001 / GID 0 (tini -> server-wrapper),
|
||||
# and on boot the wrapper writes its generated config.json into the config dir. A mounted Longhorn PVC
|
||||
# is root:root, so UID 1001 cannot write it -> the pod panics ("permission denied"). fsGroup=1001 makes
|
||||
@@ -128,19 +110,17 @@ dependencies:
|
||||
port: 5432 # Renders the egress half; the DB namespace mirrors keycloak-db (no
|
||||
protocol: tcp # default-deny), so Cilium's default-allow accepts the reciprocal ingress.
|
||||
ingressRoutes:
|
||||
- host: semaphore.${BASE_DOMAIN}
|
||||
- host: semaphore.home.keithjackson.dev
|
||||
port: 3000 # Semaphore HTTP web/API; wildcard TLS terminates at the shared Gateway (docs/08/09)
|
||||
auth:
|
||||
mode: oidc # native Keycloak OIDC client (docs/13 SSO row, docs/15 "Native OIDC"; R-IDP-02)
|
||||
groups:
|
||||
- platform-admins # access scoped to an authorized group (R-IDP-06)
|
||||
# WIRED (native OIDC): the Keycloak `semaphore` confidential client (platform/keycloak/keycloak-realm-
|
||||
# import.yaml + live kcadm) is consumed via the SEMAPHORE_OIDC_PROVIDERS env — a PLAINTEXT JSON blob
|
||||
# (see `env` above) whose per-environment URLs are ${BASE_DOMAIN}-substituted, with the confidential
|
||||
# client secret spliced in at runtime from the semaphore-bootstrap Secret key `oidc-client-secret`.
|
||||
# The `groups` claim is exposed for authorization (R-IDP-06). Semaphore KEEPS its LOCAL admin as the
|
||||
# break-glass path so an IdP outage never locks the operator out (docs/15 "Availability caveat").
|
||||
# No secret is embedded here.
|
||||
# import.yaml + live kcadm) is consumed via the SEMAPHORE_OIDC_PROVIDERS env (JSON blob held encrypted
|
||||
# in the semaphore-bootstrap Secret, oidc-providers-json). The `groups` claim is exposed for
|
||||
# authorization (R-IDP-06). Semaphore KEEPS its LOCAL admin as the break-glass path so an IdP outage
|
||||
# never locks the operator out (docs/15 "Availability caveat"). No secret is embedded here.
|
||||
update:
|
||||
channel: stable # R-UPD-01
|
||||
range: ">=2.19.0 <3.0.0" # stay on the 2.x stable line
|
||||
|
||||
+7
-84
@@ -9,90 +9,18 @@ name: sonarr
|
||||
kind: container
|
||||
source:
|
||||
image: lscr.io/linuxserver/sonarr:4.0.15 # pinned stable v4 tag (R-UPD-01); update.range governs advancement
|
||||
# LinuxServer images run their service as the user `abc`, whose uid/gid these two set. They are
|
||||
# LOAD-BEARING for the bulk mount, not cosmetic: see the storage note below. The default is 911,
|
||||
# which cannot write a 1000:3000-owned 0755 tree.
|
||||
env:
|
||||
- name: PUID
|
||||
value: "1000" # owner uid of every directory under /mnt/Storage/plex
|
||||
- name: PGID
|
||||
value: "3000" # dominant group there; keeps files this writes readable to the
|
||||
# EXISTING non-k3s consumers of the same export, which stay live
|
||||
- name: TZ
|
||||
value: Etc/UTC # same safe default plex uses; the arrs schedule by local time
|
||||
podSecurityContext:
|
||||
# For the LONGHORN config PVC only. fsGroup does nothing on the NFS volume (the in-tree plugin
|
||||
# does no ownership management), but on a Longhorn filesystem the kubelet applies it, which is
|
||||
# what makes /config writable by the now-uid-1000 process WITHOUT relying on the image chowning
|
||||
# /config for itself when PUID changes. Matches PGID so the primary group owns the config too.
|
||||
fsGroup: 3000
|
||||
exposure: internal # LAN-only web UI at sonarr.${BASE_DOMAIN} (docs/13 row 2; R-DNS/R-NET)
|
||||
exposure: internal # LAN-only web UI at sonarr.home.keithjackson.dev (docs/13 row 2; R-DNS/R-NET)
|
||||
egress: vpn # indexer scrape + downloader handoff steered via the shared VPN egress gateway,
|
||||
# kill-switch enforced OUTSIDE the workload (docs/03; ADR-0015; R-NET-05/06)
|
||||
storage:
|
||||
- name: config
|
||||
size: 5Gi
|
||||
class: longhorn # app DB / config on the 3-replica tier; DR-critical (docs/04; R-STATE-02)
|
||||
mountPath: /config # THE APP WRITES /config, NOT the renderer default /data/config.
|
||||
# LSIO images run with `-data=/config` (Jackett/qBittorrent likewise use
|
||||
# /config), so without this the Longhorn PVC mounted at /data/config sat
|
||||
# holding nothing but lost+found while the app wrote its entire
|
||||
# configuration to the container's EPHEMERAL writable layer — destroyed on
|
||||
# every pod recreation: restart, reschedule, image bump. Measured in-cluster
|
||||
# 2026-09-12: /data/config was dev=2080 with [lost+found], /config was
|
||||
# dev=2053 on the node's own disk with [Sentry asp config.xml logs].
|
||||
# This is the SAME failure catalog/plex.yaml:54 already records fixing.
|
||||
# The live /config contents were copied onto each PVC before this landed,
|
||||
# so API keys and credentials other apps reference were preserved.
|
||||
# THE MEDIA TREE (docs/04 bulk tier) — ONE read-WRITE mount of the export ROOT at /media, giving
|
||||
# /media/tv (the library) and /media/downloads (the import source) UNDER THE SAME MOUNT.
|
||||
#
|
||||
# *** THE ROOT MOUNT IS THE POINT, NOT A CONVENIENCE. *** Sonarr imports by HARDLINKING or
|
||||
# atomically renaming a finished download into the library. Both are same-FILESYSTEM operations,
|
||||
# and to the kernel "same filesystem" means SAME MOUNT. The layout on the host is:
|
||||
# /mnt/Storage/plex/tv/ the library
|
||||
# /mnt/Storage/plex/downloads/complete/tv-sonarr/ where finished releases land
|
||||
# — siblings under one dataset, so a hardlink between them is legal on the server. But mounting
|
||||
# them as two subPath volumes (subPath: tv at /media/tv, subPath: downloads elsewhere) creates
|
||||
# TWO mounts, and link() across mounts fails EXDEV no matter what the server thinks. Sonarr then
|
||||
# silently degrades to copy-then-delete: every import rewrites the whole file over NFS and the
|
||||
# seeding copy is destroyed. Their smallest common ancestor IS the export root, so the root mount
|
||||
# is the MINIMAL mount that preserves hardlinks — least privilege bounded below by a hard
|
||||
# requirement, not a shortcut. (The cost is stated plainly: Sonarr can also write /media/movies,
|
||||
# which is radarr's. Narrowing that would cost hardlinks, which is a worse trade.)
|
||||
#
|
||||
# *** /media, NOT /data — AND THAT IS NOT A STYLE CHOICE. *** The config PVC above renders at the
|
||||
# renderer default /data/config. A media mount at /data would be mounted OVER it and MASK the
|
||||
# config volume entirely; the pod would come up with its app database shadowed by an 8 T media
|
||||
# tree. /media collides with nothing in a LinuxServer image and matches the path plex uses for
|
||||
# the same export, so when qbittorrent is eventually given this mount too, the arrs and the
|
||||
# download client will already agree on absolute paths — which is the other precondition for
|
||||
# hardlinked imports (otherwise every import needs a Remote Path Mapping).
|
||||
#
|
||||
# SEPARATE LATENT BUG, NOT FIXED HERE (deliberately — it changes live on-disk state): /data/config
|
||||
# is almost certainly the wrong place for this PVC in the first place. LinuxServer's arr images run
|
||||
# with `-data=/config`, so the app writes /config — the container's EPHEMERAL layer — while the
|
||||
# Longhorn PVC at /data/config sits unused. That is the EXACT failure plex's own mountPath comment
|
||||
# documents having already been bitten by. Fixing it means starting from an empty config volume, so
|
||||
# it belongs in its own change with the operator present.
|
||||
#
|
||||
# UID/GID — THE FAILURE THIS AVOIDS IS "MOUNTS FINE, CANNOT WRITE". The export is no_root_squash
|
||||
# + sec=sys, so the client's numeric uid is passed through verbatim and the server's ownership is
|
||||
# the whole story. The tree is 0755 owned 1000:3000 (some dirs 1000:1000), so ONLY uid 1000 can
|
||||
# write it. LinuxServer images default to PUID/PGID 911 — which would mount cleanly and then fail
|
||||
# every import with EACCES. The PUID/PGID set above make the process uid 1000, matching the owner.
|
||||
#
|
||||
# AND NOTE fsGroup CANNOT SUBSTITUTE FOR THAT. The in-tree NFS volume plugin performs no
|
||||
# ownership management, so podSecurityContext.fsGroup is a NO-OP on this volume — it is not the
|
||||
# lever here, and reaching for it is how this failure normally gets misdiagnosed. The fsGroup
|
||||
# that IS set above exists for the LONGHORN config PVC, so that changing PUID does not depend on
|
||||
# the image re-chowning /config for itself.
|
||||
- name: media
|
||||
class: bulk
|
||||
export: /mnt/Storage/plex # the ZFS dataset Storage/plex, on ${BULK_NFS_SERVER}
|
||||
mountPath: /media # => /media/tv (Root Folder) and /media/downloads (import source)
|
||||
size: 9Ti # ADVISORY ONLY on a static NFS PV: the dataset's 8.33 T used,
|
||||
# rounded up, so `kubectl get pv` reports something true.
|
||||
# NOTE (bulk-media PREREQUISITE): docs/13 row 2 places the media library / downloads volume on the
|
||||
# `bulk` tier (docs/04 bulk storage). The `bulk` StorageClass exists (platform/longhorn/storageclasses.yaml)
|
||||
# but is a NON-provisioning STUB pending OQ-2 (NFS/SMB backing = REPLACE-ME). Adding a `class: bulk`
|
||||
# volume now would leave the PVC (and therefore the pod) Pending. The media/downloads volume is
|
||||
# therefore DEFERRED until OQ-2 is resolved; config-only on longhorn keeps Sonarr deployable today.
|
||||
dependencies:
|
||||
- to: qbittorrent # arr → download client (docs/13 dependency graph; R-NET-02 explicit allow)
|
||||
port: 8080
|
||||
@@ -101,17 +29,12 @@ dependencies:
|
||||
port: 9117
|
||||
protocol: tcp
|
||||
ingressRoutes:
|
||||
- host: sonarr.${BASE_DOMAIN}
|
||||
- host: sonarr.home.keithjackson.dev
|
||||
port: 8989 # Sonarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
|
||||
auth:
|
||||
mode: forward-auth # bare UI, no usable OIDC: oauth2-proxy → Keycloak at the Gateway (docs/15; R-IDP-02)
|
||||
groups:
|
||||
- media-users # authenticated AND in the media group (R-IDP-06)
|
||||
# See catalog/jackett.yaml for the full rationale: --allowed-group is repeatable and OR-ed, so
|
||||
# platform-admins here is a superset, not a widening (that group already carries cluster-admin
|
||||
# via k8s-oidc-rbac.yaml). Without it, an empty media-users locks every operator out with an
|
||||
# immediate 403 at the oauth2-proxy callback — observed on prod 2026-09-12.
|
||||
- platform-admins # operators of these services must not be locked out of them
|
||||
update:
|
||||
channel: stable # R-UPD-01
|
||||
range: ">=4.0.0 <5.0.0" # stay on the Sonarr v4 stable line
|
||||
|
||||
+3
-11
@@ -1,9 +1,8 @@
|
||||
# ci/Makefile — portable, single-command local equivalent of the Gitea Actions
|
||||
# merge gate (R-TEST-02: the gates are runnable TODAY, without Gitea).
|
||||
#
|
||||
# The LIVE Gitea Actions workflow (.gitea/workflows/gate.yml) invokes `make -C ci gate`
|
||||
# directly, so `make -C ci gate` locally IS the CI merge gate — byte-for-byte. Run from
|
||||
# anywhere:
|
||||
# The Gitea Actions workflow (.gitea/workflows/merge-gate.yaml) calls these exact
|
||||
# scripts, so `make -C ci gate` locally == the CI merge gate. Run from anywhere:
|
||||
# make -C ci gate
|
||||
#
|
||||
# Every stage is a thin wrapper over a ci/*.sh script; see ci/README.md.
|
||||
@@ -12,7 +11,7 @@ CI_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
|
||||
|
||||
.DEFAULT_GOAL := help
|
||||
|
||||
.PHONY: help spec-immutability lint schema l0 render-golden l1 conformance gate all touched versions tools test
|
||||
.PHONY: help spec-immutability lint schema l0 render-golden l1 conformance gate all touched versions tools
|
||||
|
||||
help: ## Show this help
|
||||
@awk 'BEGIN{FS=":.*##"} /^[a-zA-Z0-9_-]+:.*##/{printf " \033[36m%-14s\033[0m %s\n",$$1,$$2}' $(MAKEFILE_LIST)
|
||||
@@ -43,13 +42,6 @@ all: gate ## Alias for `gate`
|
||||
touched: ## Print the touched-requirement subset for the current branch vs main
|
||||
@"$(CI_DIR)touched-requirements.sh"
|
||||
|
||||
test: ## Run ci/ script unit tests (no cluster; e.g. touched-requirements content scan)
|
||||
@rc=0; for t in "$(CI_DIR)"tests/*.test.sh; do \
|
||||
echo "== $$t"; bash "$$t" || rc=1; \
|
||||
done; \
|
||||
if [ $$rc -ne 0 ]; then echo "ci/tests: FAILED" >&2; else echo "ci/tests: all suites passed"; fi; \
|
||||
exit $$rc
|
||||
|
||||
versions: ## Print pinned tool versions (R-TEST-02)
|
||||
@cat "$(CI_DIR)versions.env"
|
||||
|
||||
|
||||
+5
-122
@@ -20,117 +20,17 @@ Individual stages:
|
||||
|
||||
```bash
|
||||
make -C ci lint # yaml/lint (no cluster; live now)
|
||||
make -C ci l0 # schema + Kyverno + L0 static conformance (no cluster; live now)
|
||||
make -C ci l0 # ServiceDefinition schema + Kyverno (no cluster; live now)
|
||||
make -C ci l1 # L1 rendering checks (no cluster; live now)
|
||||
make -C ci conformance # L2/L3 in-cluster Chainsaw Job (SKIPS unless CLUSTER_AVAILABLE=true)
|
||||
make -C ci test # unit tests for these scripts themselves (ci/tests/*.test.sh)
|
||||
```
|
||||
|
||||
The `l0` stage has three sub-stages: ServiceDefinition schema validation, the Kyverno policy gate
|
||||
(`kyverno test`), and **`conformance/run.sh --layer l0 --include-scope whole-repo`** — the whole-repo
|
||||
L0 invariants a per-resource policy cannot express (secret hygiene across every tracked file,
|
||||
least-privilege RBAC, spec-immutability enforcement). See `conformance/tests/README.md`.
|
||||
|
||||
### `--include-scope whole-repo`: why the L0 sub-stage is not purely requirement-scoped
|
||||
|
||||
R-TEST-03 runs the conformance subset a change **touches**. That has one blind spot, and it is the
|
||||
worst-shaped kind: an invariant that quantifies over the whole tracked tree, or over subtrees the
|
||||
coarse path→requirement map keys to *nothing* (`apps/`, `infra/`, most of `platform/`), cannot be
|
||||
selected *by* the requirements a change touches — because **the change most likely to break it
|
||||
attributes no requirement at all**. Requirement-scoped selection then filters the check out of
|
||||
exactly that change.
|
||||
|
||||
This used to be handled by a hand-written "UNCONDITIONAL" stage per such check (3c subst-safety,
|
||||
3d keycloak-client-secrets, 3e bulk-tier): three near-identical rationales, three hardcoded
|
||||
`check.sh` paths, three copies of the same `rc 0/3/*` case block — and, worse, a mechanism that
|
||||
failed **silently**, because whoever added the next whole-repo invariant had to *remember* to write
|
||||
a stage. Those stages are gone. A test now **declares** `conformance.home.k3s/scope: "whole-repo"`
|
||||
and the runner unions every declaring test into the run. Adding the annotation is the only step.
|
||||
|
||||
`rc 3` is preserved exactly as those stages had it: a **SKIP** of a scope-declared check means the
|
||||
tree it audits has vanished, so the runner exits `3`, `ci/l0.sh` sets the gate's own `rc=3`, and the
|
||||
gate reports SKIPPED — **never a pass**. The mechanism is itself proven, end-to-end and falsifiably,
|
||||
by `conformance/tests/l0-static/whole-repo-invariants-wired`; `conformance/lib/common.sh` carries the
|
||||
full rationale and the incidents behind it.
|
||||
|
||||
## Exit-code contract
|
||||
|
||||
Every stage script uses: **`0` = pass · `1` = fail (blocks merge) · `3` = SKIPPED** (cluster-gated
|
||||
with no cluster, or an unwired seam / missing tool). A `3` is **never** counted as green — that is how
|
||||
the gate stays honest in Phase 0 (no fake green).
|
||||
|
||||
### The gate's OWN verdict (changed 2026-09-13 — read this if you consume the exit code)
|
||||
|
||||
**`MERGE GATE: PASS` + exit 0 means every stage that was supposed to run RAN and was GREEN.** It
|
||||
used to mean less than that: a FAILED stage set the block flag but a SKIPPED one only printed a
|
||||
warning, so the gate printed `l0: skipped (unwired seam / missing tool) — not green` and then
|
||||
`MERGE GATE: PASS`, exit 0. That happened for real twice — a missing `promtool` (the alert-rule
|
||||
unit tests) and a missing pinned `kyverno` CLI — and exit 0 is what automation reads as
|
||||
merge-approved. Fail-open in the one component whose whole job is to fail closed.
|
||||
|
||||
| verdict | exit | meaning |
|
||||
|---|---|---|
|
||||
| `MERGE GATE: PASS` | `0` | every stage that was supposed to run ran and was green; the R-TEST-03 subset was checked. Any capability the environment **declared** out of scope is **named in the headline** (see below), so a PASS never implies more than it verified. |
|
||||
| `MERGE GATE: BLOCKED` | `1` | a stage FAILED, or a stage did not run **where it must**, or the decision input is unusable. |
|
||||
| `MERGE GATE: UNDETERMINED` | `3` | nothing failed, but a stage did not run, so greenness was **not established**. Never a pass; non-zero so nothing can read it as approval. `3` is this repo's existing "not run" code, now spoken by the gate as a whole and not only by its stages. |
|
||||
| `MERGE GATE: NOT APPLICABLE` | `0` | there is no branch under gate at all (`HEAD` is on base — a push-to-`main` run). Stages ran; no subset was checked, and the headline says so. |
|
||||
|
||||
Precedence: BLOCKED > UNDETERMINED > NOT APPLICABLE > PASS.
|
||||
|
||||
**`make` masks the difference.** GNU make reports any recipe failure as its own exit `2`, so
|
||||
`make -C ci gate` collapses `1` and `3` into `2` (the headline still tells a human which it was). A
|
||||
caller that must distinguish them invokes `./ci/gate.sh` directly.
|
||||
|
||||
### Which skips block, and why — a skip is not one thing
|
||||
|
||||
| stage | a skip there means | verdict |
|
||||
|---|---|---|
|
||||
| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is installed for us by `.gitea/workflows/scripts/install-tools.sh`, so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
|
||||
| `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). |
|
||||
| `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite** `CLUSTER_AVAILABLE=true` is a defect and BLOCKS. |
|
||||
|
||||
That asymmetry is the whole rule: **declared-not-applicable may pass; could-not-run may not.**
|
||||
|
||||
`GATE_STRICT_SKIPS=true|false` decides strictness and is **declared, never sniffed** — `gate.yml`
|
||||
sets it on the gate step, because the workflow is the thing that knows it is CI. Unset, it falls
|
||||
back to `GITHUB_ACTIONS` / `CI` (set by the runner itself). A wrong answer in either direction
|
||||
cannot fail open: both branches are non-zero and neither is a PASS. Set it locally to rehearse CI
|
||||
strictness.
|
||||
|
||||
Locked down by `ci/tests/gate-decision.test.sh` (47 assertions over the decision table: stubbed
|
||||
stage outcomes against the real `gate.sh`; 32 of them fail against the pre-2026-09-13 gate).
|
||||
|
||||
### The touched-requirement subset can also be *undeterminable* (exit 3)
|
||||
|
||||
`touched-requirements.sh` is the input to the R-TEST-03 decision. It resolves a diff base in this
|
||||
order, **verifying each candidate exists** before using it:
|
||||
|
||||
```
|
||||
BASE_REF (explicit) > GITHUB_BASE_REF (Actions PR runs) > origin/main > main
|
||||
```
|
||||
|
||||
If no candidate resolves, if there is no merge base, if the diff command fails, or if **HEAD is not
|
||||
ahead of the base** (you are sitting on `main`, or the branch is already merged), it prints a loud
|
||||
banner, writes **nothing** to stdout, and exits **3** — plus a machine-readable reason code
|
||||
(`on-base` / `no-base` / `diff-failed`) to `$TOUCHED_REASON_FILE` when that variable names a path.
|
||||
|
||||
**Why this is not cosmetic.** Every one of those conditions used to degrade *silently* to an empty
|
||||
touched set and exit 0. An empty set means `conformance/run.sh` is invoked with no `--requirements`,
|
||||
which means its coverage `--require` gate never runs — so on 2026-08-31 the local gate printed
|
||||
`MERGE GATE: PASS` having checked nothing, while CI (correct base, 17 touched requirements) BLOCKED
|
||||
on real coverage debt. There is no silent narrowing left: the `HEAD~1` last resort and the two-dot
|
||||
diff fallback are both gone. Locked down by `ci/tests/touched-requirements-base.test.sh`.
|
||||
|
||||
`gate.sh` turns exit 3 into an **UNDECIDED** merge decision, never a pass:
|
||||
|
||||
| reason | headline | exit |
|
||||
|---|---|---|
|
||||
| `on-base` | `MERGE GATE: NOT APPLICABLE — no branch under gate` | 0 (a push-to-`main` CI run legitimately lands here; it just must not read as "my branch is green") |
|
||||
| `no-base`, `diff-failed` | `MERGE GATE: BLOCKED` | 1 (the environment cannot support a merge decision; refusing to decide must not look like deciding yes) |
|
||||
|
||||
A stage that needs the subset (`l0.sh`, `l1.sh`) still runs its FULL layer as a superset sanity net
|
||||
in that case — nothing goes untested — and says plainly that the *scoped* check did not happen.
|
||||
|
||||
## Reproducibility (R-TEST-02)
|
||||
|
||||
- **Version-pinned:** all tool versions live in [`versions.env`](versions.env) — the single source of
|
||||
@@ -143,21 +43,16 @@ in that case — nothing goes untested — and says plainly that the *scoped* ch
|
||||
|
||||
```
|
||||
ci/
|
||||
Makefile targets: lint · l0 · l1 · conformance · gate · touched · test · tools · versions
|
||||
Makefile targets: lint · l0 · l1 · conformance · gate · touched · tools · versions
|
||||
versions.env PINNED tool versions (R-TEST-02) — single source of truth
|
||||
lib.sh shared helpers: logging, tool checks, cluster detection, artifact paths
|
||||
yamllint.yaml lint config for the yaml/lint stage
|
||||
lint.sh Stage: yaml/lint
|
||||
l0.sh Stage: schema validation + Kyverno L0 gate + L0 static conformance
|
||||
l0.sh Stage: ServiceDefinition schema validation + Kyverno L0 policy gate
|
||||
l1.sh Stage: L1 rendering checks (Chainsaw L1 via the conformance runner)
|
||||
(harness: platform/monitoring/tests/). Catches the two alerting bugs no
|
||||
other stage can see — permanently-firing and structurally INERT rules.
|
||||
conformance.sh Stage: L2/L3 in-cluster Chainsaw Job (trigger + collect)
|
||||
touched-requirements.sh compute the R-* subset a change touches (drives the R-TEST-03 filter);
|
||||
exits 3 LOUDLY rather than guessing when the diff base is unreliable
|
||||
touched-requirements.sh compute the R-* subset a change touches (drives the R-TEST-03 filter)
|
||||
gate.sh run all stages + the merge decision (the single command)
|
||||
tests/*.test.sh unit tests for these scripts (`make -C ci test`), run against
|
||||
throwaway git repos via the REPO_ROOT override — no network, no cluster
|
||||
README.md this file
|
||||
```
|
||||
|
||||
@@ -172,7 +67,7 @@ below are this workstream's **assumptions** — flagged for the PM to verify at
|
||||
|---|---|---|---|---|
|
||||
| ServiceDefinition schema validator | WS-GITOPS · **T-0.3** (`schemas/`) | `SCHEMA_VALIDATE_CMD` | `schemas/validate.sh` | `validate.sh catalog/*.yaml` → exit 0/1; `--version` supported |
|
||||
| Kyverno L0 policies + tests | WS-TEST · **T-0.5** (`platform/kyverno/`) | `KYVERNO_POLICY_DIR` | `platform/kyverno` | contains `kyverno-test.yaml` files discoverable by `kyverno test <dir>` |
|
||||
| Conformance runner (Chainsaw) | WS-TEST · **T-0.4** (`conformance/`) | `CONFORMANCE_CMD` | `conformance/run.sh` | `run.sh --layer <l0\|l1\|l2,l3> [--requirements R-A,R-B]`; artifacts land in `conformance/_artifacts/` |
|
||||
| Conformance runner (Chainsaw) | WS-TEST · **T-0.4** (`conformance/`) | `CONFORMANCE_CMD` | `conformance/run.sh` | `run.sh --layer <l1\|l2,l3> [--requirements R-A,R-B] [--trigger in-cluster-job] --junit <file> --coverage <dir>` |
|
||||
| Catalog (ServiceDefinitions) | WS-APPS | `CATALOG_DIR` | `catalog` | `*.yaml` ServiceDefinition files |
|
||||
|
||||
Artifact locations (git-ignored): JUnit → `results/junit/*.xml`; coverage → `reports/coverage/**`.
|
||||
@@ -187,15 +82,3 @@ its owner delivers a runnable entrypoint.
|
||||
`CLUSTER_AVAILABLE=true` (env locally, `vars.CLUSTER_AVAILABLE` in Gitea) makes the L2/L3 stage run and
|
||||
be **required**; anything else makes it **SKIP** (Phase 0). `REQUIRE_CLUSTER_TESTS=true` (Phase 1+)
|
||||
turns any unverified cluster-needing touched requirement into a hard block.
|
||||
|
||||
All four switches are read through one `truthy()` in `lib.sh`, so they spell "on" identically
|
||||
(`true|1|yes|y|on`, any case) and — the direction that matters — **anything else is OFF**: a
|
||||
misspelled `CLUSTER_AVAILABLE=ture` leaves the capability undeclared instead of silently claiming
|
||||
it, which can only make the gate stricter about what it says it verified.
|
||||
|
||||
| switch | default | effect |
|
||||
|---|---|---|
|
||||
| `CLUSTER_AVAILABLE` | `false` | declares a cluster; L2/L3 then run and are required |
|
||||
| `REQUIRE_CLUSTER_TESTS` | `false` | Phase 1+: an unverified cluster-needing touched requirement BLOCKS |
|
||||
| `REQUIRE_SPEC_VERIFICATION` | `false` | an unverifiable spec repo BLOCKS instead of UNDETERMINED |
|
||||
| `GATE_STRICT_SKIPS` | `auto` | `true` = a missing tool / unwired seam BLOCKS (CI); `auto` follows `GITHUB_ACTIONS`/`CI` |
|
||||
+1
-5
@@ -47,11 +47,7 @@ fi
|
||||
# (conformance/lib/common.sh) now unions a comma/space list of layers, so this is a single
|
||||
# invocation with one aggregated report (no per-layer looping needed). Artifacts are then
|
||||
# COLLECTED from the fixed location below.
|
||||
# --include-scope: union the touched subset with every L2/L3 test DECLARING
|
||||
# `conformance.home.k3s/scope: whole-repo` (none today — wired once, here, so a future
|
||||
# whole-repo behavioural invariant needs only the annotation; see conformance/lib/common.sh).
|
||||
: "${WHOLE_REPO_SCOPE:=whole-repo}"
|
||||
"$CONFORMANCE_CMD" --layer l2,l3 --include-scope "$WHOLE_REPO_SCOPE" "${req_args[@]}"
|
||||
"$CONFORMANCE_CMD" --layer l2,l3 "${req_args[@]}"
|
||||
|
||||
CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts"
|
||||
if [ -f "${CONF_ART}/report.junit.xml" ]; then
|
||||
|
||||
+15
-216
@@ -7,75 +7,6 @@
|
||||
# skip (3) not run (cluster-gated stage w/ no cluster, or unwired seam) —
|
||||
# NOT counted as green (HARD RULE 4: don't fake green)
|
||||
#
|
||||
# ...AND THREE OUTCOMES FOR THE GATE ITSELF (2026-09-13). Until today a stage that
|
||||
# SKIPPED only printed a warning ("— not green") and the gate then printed
|
||||
# `MERGE GATE: PASS` and exited 0 anyway. That is fail-open in the one component whose
|
||||
# entire job is to fail closed: a pinned tool missing from the runner, or a seam silently
|
||||
# becoming unwired, produced exit 0 and read as merge-approved. Observed live:
|
||||
# `prom-rules: skipped (missing tool) — not green` followed by `MERGE GATE: PASS`, and
|
||||
# `l0: skipped (kyverno CLI not installed)` likewise. The gate's own vocabulary already
|
||||
# had the answer — the 2026-08-31 "local false green" introduced UNDECIDED for exactly
|
||||
# this shape of non-answer — the offline stages just did not use it.
|
||||
#
|
||||
# MERGE GATE: PASS exit 0 every stage that was SUPPOSED TO RUN ran and was
|
||||
# green, and the R-TEST-03 subset was checked.
|
||||
# MERGE GATE: BLOCKED exit 1 a stage FAILED, or a stage did not run WHERE IT
|
||||
# MUST (see the per-stage rules below), or the
|
||||
# decision input is unusable.
|
||||
# MERGE GATE: UNDETERMINED exit 3 nothing failed, but a stage did not run, so
|
||||
# greenness was NOT ESTABLISHED. Never a pass;
|
||||
# non-zero so no automation can read it as approval.
|
||||
# `3` is this repo's existing "not run / not
|
||||
# established" code (ci/lib.sh), now spoken by the
|
||||
# gate as a whole and not only by its stages.
|
||||
# MERGE GATE: NOT APPLICABLE exit 0 there is no branch under gate at all (HEAD is on
|
||||
# base) — a push-to-main CI run. Stages were green;
|
||||
# no subset was checked, and the headline says so.
|
||||
#
|
||||
# NOTE ON `make`: GNU make reports ANY recipe failure as its own exit 2, so
|
||||
# `make -C ci gate` collapses 1 and 3 into 2 — a human sees the headline and that is enough.
|
||||
# A caller that must tell BLOCKED from UNDETERMINED invokes ./ci/gate.sh directly; to tolerate
|
||||
# an UNDETERMINED run it writes `./ci/gate.sh || [ $? -eq 3 ]`, the same explicit idiom
|
||||
# .gitea/workflows/gate.yml already used for its spec-immutability pre-step and now uses for
|
||||
# the gate itself (with GATE_STRICT_SKIPS=true, so that tolerance can ONLY ever cover the
|
||||
# declared-environmental skip below — a missing tool on the runner is exit 1, untolerated).
|
||||
# TOLERATING A NON-ANSWER IS A VISIBLE, REVIEWED CHOICE IN THE CALLER. There is deliberately no
|
||||
# knob inside this script that turns UNDETERMINED into exit 0: that would be the fail-open this
|
||||
# change exists to remove, just relocated.
|
||||
#
|
||||
# WHICH SKIPS BLOCK, AND WHY (per stage — a skip is not one thing):
|
||||
#
|
||||
# lint / l0 / render-golden / l1 A skip here means a PINNED tool is absent or a wired
|
||||
# seam went unwired. On a CI runner the pinned toolchain is installed for us by
|
||||
# .gitea/workflows/scripts/install-tools.sh, so a skip there is a DEFECT, not an
|
||||
# environment: it BLOCKS. On a workstation a missing tool is a real local condition, so
|
||||
# it degrades to UNDETERMINED (still non-zero, still never a pass). See strict_skips()
|
||||
# in ci/lib.sh for how CI is detected and why misdetecting it cannot fail open.
|
||||
#
|
||||
# spec-immutability (R-TEST-06) A skip means the sibling spec repo is not checked out, so
|
||||
# the spec COULD NOT BE VERIFIED. Never a pass — unverified is unverified — so it denies a
|
||||
# PASS and yields UNDETERMINED. It is deliberately NOT upgraded to BLOCKED by CI strictness,
|
||||
# because on the live runner it is not a defect anyone can fix from inside this repo: the
|
||||
# workflow does not check the spec repo out at all (SPEC_REPO_SLUG is unwired — the blueprint
|
||||
# lives in merge-gate.yaml), and blocking would leave the only enforced status check
|
||||
# permanently red for a reason no PR could clear. So THIS is the one skip whose non-answer the
|
||||
# CALLER may knowingly tolerate, and gate.yml does so in one visible line. Locally the
|
||||
# remedy exists and the message names it: check the spec repo out beside this one, or point
|
||||
# SPEC_REPO at it (infra/devcontainer/dev.sh mounts it automatically when it is a sibling).
|
||||
# Set REQUIRE_SPEC_VERIFICATION=true (the counterpart of REQUIRE_CLUSTER_TESTS) to make it a
|
||||
# hard block once that checkout is wired.
|
||||
#
|
||||
# conformance (L2/L3) A skip when CLUSTER_AVAILABLE!=true is NOT a defect and
|
||||
# NOT an accident: it is the environment DECLARING that the capability is absent, which
|
||||
# is the documented Phase-0 behaviour. A declared-inapplicable stage was never "supposed
|
||||
# to run", so it does not deny a PASS — the touched requirements it would have covered
|
||||
# are still reported UNVERIFIED, and REQUIRE_CLUSTER_TESTS=true (Phase 1+) turns that
|
||||
# into a block. A skip DESPITE CLUSTER_AVAILABLE=true is a defect and blocks, exactly as
|
||||
# it already did. That asymmetry is the whole distinction this file now draws:
|
||||
# DECLARED-not-applicable may pass; COULD-NOT-RUN may not. A declared gap is still NAMED in
|
||||
# the PASS headline (see DECLARED_GAPS below), so a PASS never reads as "everything was
|
||||
# verified" when a layer was declared out of scope.
|
||||
#
|
||||
# Phase 0 policy (no cluster): L0/L1 stages (lint, schema, kyverno, L1 render)
|
||||
# MUST be green. L2/L3 SKIP. If the touched subset needs L2/L3 and no cluster is
|
||||
# available, the gate reports those requirements UNVERIFIED and — unless
|
||||
@@ -89,7 +20,6 @@ source "${CI_DIR}/lib.sh"
|
||||
set +e # lib.sh enables -e; the gate must run ALL stages and aggregate, not abort.
|
||||
|
||||
: "${REQUIRE_CLUSTER_TESTS:=false}"
|
||||
: "${REQUIRE_SPEC_VERIFICATION:=false}"
|
||||
|
||||
declare -A OUTCOME
|
||||
run_stage() {
|
||||
@@ -117,104 +47,28 @@ run_stage conformance "${CI_DIR}/conformance.sh"
|
||||
echo >&2
|
||||
log "==== gate decision ===="
|
||||
BLOCK=0
|
||||
# Stages that did NOT run and are tolerated here — but never as green. A non-empty list
|
||||
# means the gate could not establish greenness -> MERGE GATE: UNDETERMINED, exit 3.
|
||||
NOT_ESTABLISHED=()
|
||||
# Stages the environment DECLARED inapplicable (today: L2/L3 with no cluster declared). These
|
||||
# do not deny a PASS — nothing was supposed to run — but they are named IN the PASS headline, so
|
||||
# "PASS" is never mistaken for "everything was verified".
|
||||
DECLARED_GAPS=()
|
||||
|
||||
# --- R-TEST-03 input: the touched-requirement subset --------------------------
|
||||
# Computed ONCE here. ci/touched-requirements.sh exits 3 when it cannot establish a
|
||||
# reliable diff base; that used to degrade to an empty set, which silently disabled the
|
||||
# coverage `--require` gate and let the gate print "MERGE GATE: PASS" having checked
|
||||
# nothing (the 2026-08-31 local false green). An undeterminable subset is now surfaced
|
||||
# as an UNDECIDED merge decision, never as a pass.
|
||||
_reason_file="$(mktemp)"
|
||||
# NOTE: command substitution, NOT `mapfile < <(...)` — process substitution does not
|
||||
# propagate the child's exit status, and the whole point here is to SEE the exit 3.
|
||||
TOUCHED_OUT="$(TOUCHED_REASON_FILE="$_reason_file" "${CI_DIR}/touched-requirements.sh")"
|
||||
TOUCHED_RC=$?
|
||||
TOUCHED_REASON="$(cat "$_reason_file" 2>/dev/null || true)"
|
||||
rm -f "$_reason_file"
|
||||
TOUCHED=()
|
||||
[ -n "$TOUCHED_OUT" ] && mapfile -t TOUCHED <<< "$TOUCHED_OUT"
|
||||
UNDECIDED=0
|
||||
SUBSET_BROKEN=0
|
||||
if [ "$TOUCHED_RC" -eq 3 ]; then
|
||||
UNDECIDED=1
|
||||
echo >&2
|
||||
err "R-TEST-03 DECISION INPUT MISSING: the touched-requirement subset could not be"
|
||||
err " determined (${TOUCHED_REASON:-unknown}). The scoped coverage gate did NOT run."
|
||||
summary "- **R-TEST-03 subset UNDETERMINED** (\`${TOUCHED_REASON:-unknown}\`) — the scoped coverage gate did not run."
|
||||
elif [ "$TOUCHED_RC" -ne 0 ]; then
|
||||
# Any OTHER non-zero exit is a hard defect in the subset computation itself — an
|
||||
# incomplete path->requirement map (a changed path matches no row) or an invalid row
|
||||
# would report exactly this. The reasoning is taken from the unlanded
|
||||
# fix/touched-requirements-fail-closed (47068ee): gate.sh special-cased ONLY exit 3, so
|
||||
# any other non-zero exit left TOUCHED empty and UNDECIDED=0, and the decision below
|
||||
# printed "MERGE GATE: PASS" having applied no coverage gate at all. Unreachable on
|
||||
# `main` today (touched-requirements.sh exits only 0 or 3) and wired here so it is
|
||||
# already closed the moment a fail-closed path map makes exit 1 reachable. A subset that
|
||||
# cannot be computed is NOT an empty subset.
|
||||
SUBSET_BROKEN=1
|
||||
echo >&2
|
||||
err "R-TEST-03 DECISION INPUT BROKEN: touched-requirements.sh exited ${TOUCHED_RC}"
|
||||
err " (${TOUCHED_REASON:-unknown}). Its output above names the paths/rows to fix."
|
||||
summary "- **R-TEST-03 subset UNUSABLE** (\`${TOUCHED_REASON:-unknown}\`, exit ${TOUCHED_RC}) — the coverage gate could not be applied."
|
||||
elif [ "${#TOUCHED[@]}" -gt 0 ]; then
|
||||
log "touched requirements: ${TOUCHED[*]}"
|
||||
fi
|
||||
|
||||
# spec-immutability (R-TEST-06): a FAIL always blocks (the spec was modified/drifted).
|
||||
# A skip means the spec repo wasn't available to verify — UNVERIFIED, never green: it
|
||||
# denies a PASS (exit 3) and, with REQUIRE_SPEC_VERIFICATION=true, blocks outright. See the
|
||||
# file header for why this one is not upgraded to a block by CI detection alone.
|
||||
# A skip means the spec repo wasn't available to verify — surfaced, not green.
|
||||
case "${OUTCOME[spec-immutability]}" in
|
||||
fail) err "spec-immutability: FAILED -> blocks merge (R-TEST-06)"; BLOCK=1 ;;
|
||||
skip)
|
||||
if truthy "$REQUIRE_SPEC_VERIFICATION"; then
|
||||
err "spec-immutability: skipped (spec repo unavailable) and REQUIRE_SPEC_VERIFICATION=true -> blocks merge"
|
||||
summary "- spec-immutability: **SKIPPED** (spec repo unavailable) — blocks merge (REQUIRE_SPEC_VERIFICATION=true)"
|
||||
BLOCK=1
|
||||
else
|
||||
warn "spec-immutability: skipped (spec repo unavailable) — R-TEST-06 NOT VERIFIED, so"
|
||||
warn " this run cannot be a PASS. Check the spec repo out beside this one (or set"
|
||||
warn " SPEC_REPO) to verify it."
|
||||
summary "- spec-immutability: **SKIPPED** (spec repo unavailable — R-TEST-06 not verified; not a pass)"
|
||||
NOT_ESTABLISHED+=("spec-immutability")
|
||||
fi ;;
|
||||
skip) warn "spec-immutability: skipped (spec repo unavailable) — not verified" ;;
|
||||
pass) ok "spec-immutability: green" ;;
|
||||
esac
|
||||
|
||||
# L0/L1 stages: any failure blocks. A SKIP here means a pinned tool is missing or a seam is
|
||||
# unwired — NEVER green. In CI (strict) that is a defect and blocks; on a workstation it
|
||||
# degrades to UNDETERMINED, which is still non-zero and still not a pass. render-golden is a
|
||||
# L0/L1 stages: any failure blocks. A SKIP here means a seam is unwired — in
|
||||
# Phase 0 that is tolerated (placeholders), but surfaced. render-golden is a
|
||||
# no-cluster reproducibility gate (canonical render == committed apps/ goldens):
|
||||
# a FAIL means a golden drifted from its ServiceDefinition source — always blocks.
|
||||
for s in lint l0 render-golden l1; do
|
||||
case "${OUTCOME[$s]}" in
|
||||
fail) err "${s}: FAILED -> blocks merge"; BLOCK=1 ;;
|
||||
skip)
|
||||
if strict_skips; then
|
||||
err "${s}: skipped (unwired seam / missing tool) under STRICT skips, where the pinned"
|
||||
err " toolchain is installed for us (.gitea/workflows/scripts/install-tools.sh) and every"
|
||||
err " seam is on main — so this is a DEFECT, not an environment -> blocks merge"
|
||||
summary "- ${s}: **SKIPPED under strict skips** (unwired seam / missing tool) — blocks merge"
|
||||
BLOCK=1
|
||||
else
|
||||
warn "${s}: skipped (unwired seam / missing tool) — not green, so this run cannot be a PASS"
|
||||
summary "- ${s}: **SKIPPED** (unwired seam / missing tool) — not green"
|
||||
NOT_ESTABLISHED+=("$s")
|
||||
fi ;;
|
||||
skip) warn "${s}: skipped (unwired seam / missing tool) — not green" ;;
|
||||
pass) ok "${s}: green" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# L2/L3 conformance: cluster-gated. The ONLY stage whose non-execution can still be a PASS,
|
||||
# because CLUSTER_AVAILABLE!=true is the environment DECLARING the capability absent rather
|
||||
# than the stage failing to run where it could have — see the file header.
|
||||
# L2/L3 conformance: cluster-gated.
|
||||
case "${OUTCOME[conformance]}" in
|
||||
fail)
|
||||
err "conformance (L2/L3): FAILED -> blocks merge"; BLOCK=1 ;;
|
||||
@@ -223,86 +77,31 @@ case "${OUTCOME[conformance]}" in
|
||||
skip)
|
||||
if cluster_available; then
|
||||
err "conformance: skipped despite CLUSTER_AVAILABLE=true -> blocks merge"; BLOCK=1
|
||||
elif [ "$UNDECIDED" -eq 1 ]; then
|
||||
warn "conformance L2/L3 UNVERIFIED, and the touched subset is unknown — nothing certified."
|
||||
DECLARED_GAPS+=("conformance-l2-l3(no cluster declared)")
|
||||
else
|
||||
if [ "${#TOUCHED[@]}" -gt 0 ]; then
|
||||
reqs=("${TOUCHED[@]}")
|
||||
mapfile -t reqs < <("${CI_DIR}/touched-requirements.sh" || true)
|
||||
if [ "${#reqs[@]}" -gt 0 ]; then
|
||||
warn "conformance L2/L3 UNVERIFIED for touched requirements: ${reqs[*]}"
|
||||
warn " (no cluster — Phase 0). These are NOT certified green."
|
||||
summary "- **UNVERIFIED (no cluster):** ${reqs[*]}"
|
||||
if truthy "$REQUIRE_CLUSTER_TESTS"; then
|
||||
if [ "$REQUIRE_CLUSTER_TESTS" = "true" ]; then
|
||||
err "REQUIRE_CLUSTER_TESTS=true and cluster-needing requirements are unverified -> blocks merge"
|
||||
BLOCK=1
|
||||
else
|
||||
DECLARED_GAPS+=("conformance-l2-l3(no cluster declared; ${#TOUCHED[@]} touched reqs unverified)")
|
||||
fi
|
||||
else
|
||||
skip "conformance L2/L3 skipped; no cluster-needing requirements touched"
|
||||
DECLARED_GAPS+=("conformance-l2-l3(no cluster declared)")
|
||||
fi
|
||||
fi ;;
|
||||
esac
|
||||
|
||||
# An UNDECIDED subset is never a pass. Two outcomes, by reason:
|
||||
# on-base -> this checkout has no branch commits to gate (you are on main, or the
|
||||
# branch is already merged; a push-to-main CI run lands here). Legitimate,
|
||||
# so the job is not failed — but the headline says NOT APPLICABLE, never
|
||||
# PASS, so nobody reads it as "my branch is green".
|
||||
# anything -> the environment cannot support a merge decision (no base ref, broken
|
||||
# else diff). Refusing to decide must not look like deciding yes: BLOCK.
|
||||
if [ "$UNDECIDED" -eq 1 ] && [ "$TOUCHED_REASON" != "on-base" ]; then
|
||||
err "cannot compute the R-TEST-03 subset (${TOUCHED_REASON:-unknown}) -> blocks merge"
|
||||
BLOCK=1
|
||||
fi
|
||||
# An incomplete/invalid path map is not an environment problem to skip past: the change under
|
||||
# gate contains paths whose requirements are UNKNOWN, so no subset can be certified.
|
||||
if [ "$SUBSET_BROKEN" -eq 1 ]; then
|
||||
err "R-TEST-03 subset unusable (${TOUCHED_REASON:-unknown}) -> blocks merge"
|
||||
BLOCK=1
|
||||
fi
|
||||
|
||||
echo >&2
|
||||
# Precedence: BLOCKED (something is wrong) > UNDETERMINED (a stage did not run, so
|
||||
# greenness was not established) > NOT APPLICABLE (nothing to gate) > PASS. UNDETERMINED
|
||||
# outranks NOT APPLICABLE because "no branch under gate" AND "a stage did not run" is two
|
||||
# reasons this is not a green run, and the stronger, non-zero one must be the headline.
|
||||
if [ "$BLOCK" -ne 0 ]; then
|
||||
if [ "$BLOCK" -eq 0 ]; then
|
||||
ok "MERGE GATE: PASS (touched-requirement subset green for the layers that can run)"
|
||||
summary ""
|
||||
summary "**Result: PASS** — L0/L1 green. L2/L3 status per cluster availability above."
|
||||
exit 0
|
||||
else
|
||||
err "MERGE GATE: BLOCKED"
|
||||
summary ""
|
||||
summary "**Result: BLOCKED** — see failures above."
|
||||
exit 1
|
||||
elif [ "${#NOT_ESTABLISHED[@]}" -gt 0 ]; then
|
||||
err "MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED."
|
||||
err " Stages that did not run: ${NOT_ESTABLISHED[*]}"
|
||||
err " Nothing FAILED, and nothing is certified either: a stage that did not run is not"
|
||||
err " green (HARD RULE 4). This is exit 3, never 0 — no automation may read it as"
|
||||
err " merge-approved. Fix the environment (install the pinned tool / wire the seam /"
|
||||
err " check out the spec repo) and run again, or accept it KNOWINGLY and explicitly:"
|
||||
err " ./ci/gate.sh || [ \$? -eq 3 ]"
|
||||
summary ""
|
||||
summary "**Result: UNDETERMINED** — these stages did not run, so nothing is certified: \`${NOT_ESTABLISHED[*]}\`. Not a pass (exit 3)."
|
||||
exit 3
|
||||
elif [ "$UNDECIDED" -eq 1 ]; then
|
||||
warn "MERGE GATE: NOT APPLICABLE — no branch under gate (${TOUCHED_REASON})."
|
||||
warn " The stages above ran and did not fail, but NO touched-requirement subset was"
|
||||
warn " checked. This is NOT a green branch: check out the branch you mean to gate"
|
||||
warn " (or set BASE_REF) and run it again."
|
||||
summary ""
|
||||
summary "**Result: NOT APPLICABLE** — no branch under gate (\`${TOUCHED_REASON}\`); the R-TEST-03 subset was not checked."
|
||||
exit 0
|
||||
else
|
||||
ok "MERGE GATE: PASS — every stage that was supposed to run RAN and was GREEN, and the"
|
||||
ok " R-TEST-03 touched-requirement subset was checked."
|
||||
if [ "${#DECLARED_GAPS[@]}" -gt 0 ]; then
|
||||
ok " NOT verified here, because the environment DECLARED it out of scope (never a silent"
|
||||
ok " gap — Phase 0 by policy, see the header): ${DECLARED_GAPS[*]}"
|
||||
summary ""
|
||||
summary "**Result: PASS** — every stage that was supposed to run ran green. Declared out of scope (NOT verified): \`${DECLARED_GAPS[*]}\`."
|
||||
else
|
||||
summary ""
|
||||
summary "**Result: PASS** — every stage that was supposed to run ran green; nothing was declared out of scope."
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
@@ -7,21 +7,6 @@
|
||||
# Stage 3 L0 : `kyverno test` over platform/kyverno/ — default-deny baseline
|
||||
# present, no plaintext secrets, required labels/egress-class,
|
||||
# ServiceDefinition schema conformance (owner: T-0.5, ADR-0011).
|
||||
# Stage 3b L0 : conformance/run.sh --layer l0 — the WHOLE-REPO L0 invariants that
|
||||
# a per-resource policy cannot express (secret hygiene across every
|
||||
# tracked file, least-privilege RBAC, spec-immutability enforcement).
|
||||
# Selection = the TOUCHED requirement subset (R-TEST-03) UNION every L0
|
||||
# test DECLARING `conformance.home.k3s/scope: whole-repo`. That union
|
||||
# replaces what used to be a hand-written "UNCONDITIONAL" stage per
|
||||
# whole-repo check (3c subst-safety / 3d keycloak-client-secrets / 3e
|
||||
# bulk-tier): three near-identical 12-line rationales, three hardcoded
|
||||
# check.sh paths and three copies of the same rc 0/3/* case block, which
|
||||
# did not scale and — worse — failed SILENTLY, because whoever added the
|
||||
# next whole-repo invariant had to REMEMBER to hand-write a stage. Now the
|
||||
# test declares its own scope and the runner enforces it. The reasoning and
|
||||
# the incidents those stages recorded were moved INTO the tests they
|
||||
# describe (see each chainsaw-test.yaml's `scope` annotation) and the
|
||||
# general mechanism is documented in conformance/lib/common.sh.
|
||||
#
|
||||
# Integration seams are env-overridable (see ci/lib.sh + README). When an owning
|
||||
# subtree is still a placeholder, this reports SKIPPED (exit 3) for that sub-stage
|
||||
@@ -88,54 +73,6 @@ else
|
||||
rm -rf "$l0_render"
|
||||
fi
|
||||
|
||||
# ---- Stage 3b: L0 STATIC conformance checks ---------------------------------
|
||||
# docs/14's L0 row is "manifests/ServiceDefinitions obey invariants", which is wider
|
||||
# than what Kyverno alone can express over single resources: whole-repo invariants
|
||||
# (no plaintext secrets anywhere in git, least-privilege RBAC across every tracked
|
||||
# binding, the spec-immutability gate's own behaviour) live in the conformance
|
||||
# harness's tests/l0-static subtree and run with no cluster. This stage executes them
|
||||
# through the same single-command runner CI and a laptop both use (R-TEST-02).
|
||||
#
|
||||
# THE UNION, AND WHY IT IS NOT OPTIONAL (R-TEST-03's blind spot)
|
||||
# --------------------------------------------------------------
|
||||
# 3b scopes the conformance run to the TOUCHED requirements. That is right for an invariant
|
||||
# whose inputs live under the subtree its requirements map to. It is WRONG for one that
|
||||
# quantifies over the WHOLE tracked tree, or over subtrees (apps/, infra/, most of platform/)
|
||||
# that ci/touched-requirements.sh's coarse path map keys to NOTHING: for those, the change
|
||||
# most likely to break the invariant attributes no requirement at all, so requirement-scoped
|
||||
# selection filters the check out of EXACTLY that change. `--include-scope whole-repo` unions
|
||||
# in every L0 test that DECLARES that scope, so those run on every gate invocation.
|
||||
#
|
||||
# rc 3 IS NOT A PASS. The runner returns 3 when a scope-declared check SKIPPED (its subject
|
||||
# subtree is absent, or it has no check script): UNWIRED, not green, exactly as the deleted
|
||||
# per-check stages had it ("check SKIPPED (clusters/ absent) — not a pass"). It is mapped to
|
||||
# the gate's own rc 3 below, never to rc 0.
|
||||
: "${WHOLE_REPO_SCOPE:=whole-repo}"
|
||||
log "Stage: L0 static conformance checks (${CONFORMANCE_CMD} --layer l0 --include-scope ${WHOLE_REPO_SCOPE})"
|
||||
if ! entrypoint_ready "$CONFORMANCE_CMD"; then
|
||||
skip "conformance runner not present at '${CONFORMANCE_CMD}' — UNWIRED, not green"
|
||||
summary "- l0-static: **SKIPPED** (conformance runner missing)"
|
||||
if [ "$rc" -eq 0 ]; then rc=3; fi
|
||||
else
|
||||
load_touched_requirements
|
||||
l0_static_rc=0
|
||||
"$CONFORMANCE_CMD" --layer l0 --include-scope "$WHOLE_REPO_SCOPE" \
|
||||
${REQ_ARGS[@]+"${REQ_ARGS[@]}"} || l0_static_rc=$?
|
||||
case "$l0_static_rc" in
|
||||
0) ok "L0 static conformance checks passed"
|
||||
summary "- l0-static: **passed**" ;;
|
||||
3) skip "an L0 check declaring scope='${WHOLE_REPO_SCOPE}' SKIPPED — UNWIRED, not a pass"
|
||||
summary "- l0-static: **SKIPPED** (a whole-repo check is UNWIRED — not a pass)"
|
||||
if [ "$rc" -eq 0 ]; then rc=3; fi ;;
|
||||
*) err "L0 static conformance checks failed"
|
||||
summary "- l0-static: **FAILED**"
|
||||
rc=1 ;;
|
||||
esac
|
||||
# Per-check step-summary lines, so the gate output still NAMES what ran — the generated
|
||||
# replacement for the one hardcoded `summary` line each deleted stage carried.
|
||||
emit_offline_summary
|
||||
fi
|
||||
|
||||
case "$rc" in
|
||||
0) ok "L0 stage complete" ;;
|
||||
3) skip "L0 stage incomplete (unwired seam / missing tool) — not a pass" ;;
|
||||
|
||||
@@ -20,27 +20,19 @@ if ! entrypoint_ready "$CONFORMANCE_CMD"; then
|
||||
exit 3
|
||||
fi
|
||||
|
||||
# Touched-requirement subset (R-TEST-03). An empty OR undeterminable subset => run all
|
||||
# L1 as a superset sanity net, and SAY SO — see load_touched_requirements in ci/lib.sh.
|
||||
load_touched_requirements
|
||||
req_args=(${REQ_ARGS[@]+"${REQ_ARGS[@]}"})
|
||||
# Touched-requirement subset (R-TEST-03). Empty subset => run all L1 as a sanity net.
|
||||
mapfile -t reqs < <("${CI_DIR}/touched-requirements.sh" || true)
|
||||
req_args=()
|
||||
if [ "${#reqs[@]}" -gt 0 ]; then
|
||||
log "touched requirements: ${reqs[*]}"
|
||||
req_args=(--requirements "$(IFS=,; echo "${reqs[*]}")")
|
||||
fi
|
||||
|
||||
# RECONCILED (PM 2026-08-16): conformance/run.sh's real interface is `--layer <l1|l2|l3>`
|
||||
# [--requirements R-A,R-B]; it ALWAYS writes artifacts to a FIXED location
|
||||
# (conformance/_artifacts/) and does NOT accept --junit/--coverage. Invoke by layer +
|
||||
# touched subset, then COLLECT its artifacts into the CI artifact dirs.
|
||||
# --include-scope: union the touched subset with every L1 test DECLARING
|
||||
# `conformance.home.k3s/scope: whole-repo`. As of 2026-09-18 ALL THREE L1 tests declare it, so this
|
||||
# is load-bearing rather than a no-op: every L1 golden is checked on every gate run, whatever the
|
||||
# touched paths attribute. It was wired HERE first, ahead of any declaration, so picking up a new
|
||||
# whole-repo invariant needs only the annotation instead of someone remembering to hand-write a
|
||||
# stage — and that "someone must remember" is exactly how the 2026-09-04 external-dns merge gated
|
||||
# GREEN while leaving all three goldens stale (see the rationale in the Tests themselves).
|
||||
# rc 3 = a scope-declared check SKIPPED = UNWIRED, never a pass.
|
||||
: "${WHOLE_REPO_SCOPE:=whole-repo}"
|
||||
l1_rc=0
|
||||
"$CONFORMANCE_CMD" --layer l1 --include-scope "$WHOLE_REPO_SCOPE" \
|
||||
${req_args[@]+"${req_args[@]}"} || l1_rc=$?
|
||||
"$CONFORMANCE_CMD" --layer l1 "${req_args[@]}"
|
||||
|
||||
CONF_ART="$(dirname "$CONFORMANCE_CMD")/_artifacts"
|
||||
if [ -f "${CONF_ART}/report.junit.xml" ]; then
|
||||
@@ -50,16 +42,5 @@ if [ -d "${CONF_ART}/coverage" ]; then
|
||||
cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
|
||||
fi
|
||||
|
||||
case "$l1_rc" in
|
||||
0) ok "L1 rendering checks passed"
|
||||
summary "- l1-render: **passed**"
|
||||
emit_offline_summary ;;
|
||||
3) skip "an L1 check declaring scope='${WHOLE_REPO_SCOPE}' SKIPPED — UNWIRED, not a pass"
|
||||
summary "- l1-render: **SKIPPED** (a whole-repo check is UNWIRED — not a pass)"
|
||||
emit_offline_summary
|
||||
exit 3 ;;
|
||||
*) err "L1 rendering checks FAILED (rc=${l1_rc})"
|
||||
summary "- l1-render: **FAILED**"
|
||||
emit_offline_summary
|
||||
exit "$l1_rc" ;;
|
||||
esac
|
||||
ok "L1 rendering checks passed"
|
||||
summary "- l1-render: **passed**"
|
||||
@@ -10,12 +10,9 @@
|
||||
set -euo pipefail
|
||||
|
||||
# --- Paths -------------------------------------------------------------------
|
||||
# CI_DIR = this ci/ directory (always the real script location). REPO_ROOT = its
|
||||
# parent (impl repo root) by default, but OVERRIDABLE via the environment so the
|
||||
# scripts can be pointed at a throwaway git repo under test (see ci/tests/) without
|
||||
# copying them out of the tree. CI and normal runs leave it unset -> unchanged.
|
||||
# CI_DIR = this ci/ directory; REPO_ROOT = its parent (impl repo root).
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
: "${REPO_ROOT:=$(cd "${CI_DIR}/.." && pwd)}"
|
||||
REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "${CI_DIR}/versions.env"
|
||||
|
||||
@@ -72,123 +69,16 @@ entrypoint_ready() {
|
||||
return 1
|
||||
}
|
||||
|
||||
# truthy <value> — the ONE place the repo's "is this switch on" spelling lives, so
|
||||
# CLUSTER_AVAILABLE / REQUIRE_CLUSTER_TESTS / REQUIRE_SPEC_VERIFICATION / GATE_STRICT_SKIPS
|
||||
# all agree. Anything not listed is OFF, and that direction matters: a misspelled value
|
||||
# (CLUSTER_AVAILABLE=ture, REQUIRE_SPEC_VERIFICATION=yeah) leaves the guard OFF rather than
|
||||
# silently declaring a capability "available", so a typo can only make the gate stricter about
|
||||
# what it claims to have verified, never looser.
|
||||
truthy() {
|
||||
case "${1:-}" in
|
||||
true|TRUE|True|1|yes|YES|y|on|ON) return 0 ;;
|
||||
# --- Cluster detection (drives SKIP vs run for L2/L3) -------------------------
|
||||
# A cluster is 'available' only if explicitly declared. We NEVER auto-assume a
|
||||
# cluster in Phase 0 — silence means SKIP, not pass (HARD RULE 4).
|
||||
# CLUSTER_AVAILABLE=true -> L2/L3 run and are REQUIRED green
|
||||
# CLUSTER_AVAILABLE=false/unset -> L2/L3 SKIP (reported, never faked green)
|
||||
cluster_available() {
|
||||
case "${CLUSTER_AVAILABLE:-false}" in
|
||||
true|1|yes) return 0 ;;
|
||||
*) return 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# --- Cluster detection (drives SKIP vs run for L2/L3) -------------------------
|
||||
# A cluster is 'available' only if explicitly DECLARED. We NEVER auto-assume a
|
||||
# cluster in Phase 0 — silence means SKIP, not pass (HARD RULE 4).
|
||||
# CLUSTER_AVAILABLE=true -> L2/L3 run and are REQUIRED green
|
||||
# CLUSTER_AVAILABLE=false/unset -> L2/L3 SKIP (reported, never faked green)
|
||||
# That DECLARATION is what makes an L2/L3 skip different IN KIND from every other skip —
|
||||
# see "WHICH SKIPS BLOCK, AND WHY" in ci/gate.sh.
|
||||
cluster_available() { truthy "${CLUSTER_AVAILABLE:-false}"; }
|
||||
|
||||
# --- Skip strictness: is this a CI runner, or a workstation? -------------------
|
||||
# A SKIPPED stage is never green anywhere (HARD RULE 4), but its SEVERITY differs:
|
||||
#
|
||||
# CI the pinned toolchain is installed by .gitea/workflows/scripts/install-tools.sh
|
||||
# before the gate runs, and every seam is on `main`. A stage that skips here is a
|
||||
# DEFECT (a tool dropped from the installer, a seam that went unwired) -> BLOCK.
|
||||
# local a missing pinned tool is a real, ordinary local condition -> the gate reports
|
||||
# UNDETERMINED (exit 3): not green, not a pass, but distinguishable from a red.
|
||||
#
|
||||
# HOW STRICTNESS IS DECIDED — DECLARED first, detected only as a fallback:
|
||||
#
|
||||
# 1. GATE_STRICT_SKIPS=true|false is the AUTHORITY. .gitea/workflows/gate.yml sets `true` on
|
||||
# the gate step, so on the one CI runner that matters strictness is a DECLARATION IN THE
|
||||
# WORKFLOW, not an inference: the environment that KNOWS it is CI says so, and no
|
||||
# env-var sniffing can get it wrong.
|
||||
# 2. `auto` (the default — anywhere the caller said nothing) falls back to GITHUB_ACTIONS /
|
||||
# CI, which are set by the runner ITSELF (Gitea's act_runner, as well as GitHub). We do
|
||||
# NOT sniff hostnames, paths or tool availability: a heuristic that guesses "this looks
|
||||
# like CI" is exactly the unreliable inference this gate must not contain.
|
||||
#
|
||||
# WHY A WRONG ANSWER CANNOT FAIL OPEN: both branches are non-zero and NEITHER is a pass. A CI
|
||||
# run misread as local yields UNDETERMINED (exit 3) instead of BLOCKED (exit 1) — that still
|
||||
# fails the job unless its caller explicitly tolerates 3, which gate.yml does only knowingly
|
||||
# and only for the one declared-environmental skip (see its comment). A workstation misread as
|
||||
# CI yields BLOCKED instead of UNDETERMINED — harsher, never looser. Set GATE_STRICT_SKIPS=true
|
||||
# locally to rehearse CI strictness.
|
||||
: "${GATE_STRICT_SKIPS:=auto}"
|
||||
strict_skips() {
|
||||
case "$GATE_STRICT_SKIPS" in
|
||||
auto) truthy "${GITHUB_ACTIONS:-}" || truthy "${CI:-}" ;;
|
||||
*) truthy "$GATE_STRICT_SKIPS" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
mkartifacts() { mkdir -p "$JUNIT_DIR" "$COVERAGE_DIR"; }
|
||||
|
||||
# --- Offline (L0/L1) per-check step summary -----------------------------------
|
||||
# conformance/lib/run_l1_offline.py writes one Markdown line per check it executed to
|
||||
# conformance/_artifacts/offline-summary.md. Splice those into $GITHUB_STEP_SUMMARY so the
|
||||
# gate's output still NAMES every check that ran and its verdict.
|
||||
#
|
||||
# WHY THIS EXISTS: ci/l0.sh used to carry a hand-written stage per whole-repo check, each with
|
||||
# its own `summary "- subst-safety: **passed**"` line. Those stages are gone (the tests declare
|
||||
# `conformance.home.k3s/scope: whole-repo` instead and the runner unions them in), so the
|
||||
# naming is generated for EVERY offline check rather than for the three that had a stage.
|
||||
: "${OFFLINE_SUMMARY_FILE:=${CONFORMANCE_DIR}/_artifacts/offline-summary.md}"
|
||||
emit_offline_summary() {
|
||||
local f="${1:-$OFFLINE_SUMMARY_FILE}" line
|
||||
[ -f "$f" ] || return 0
|
||||
while IFS= read -r line; do
|
||||
[ -n "$line" ] && summary " $line"
|
||||
done < "$f"
|
||||
return 0
|
||||
}
|
||||
|
||||
# --- Touched-requirement subset (R-TEST-03 input) ----------------------------
|
||||
# Sets: TOUCHED_RC (0 ok / 3 not determinable), TOUCHED_REQS[] and REQ_ARGS[]
|
||||
# (ready to splice into a conformance/run.sh invocation).
|
||||
#
|
||||
# ci/touched-requirements.sh exits 3 — LOUDLY — when it cannot establish a reliable
|
||||
# diff base, instead of the old silent degradation to an empty set. An empty set
|
||||
# means run.sh gets no `--requirements`, which means its coverage `--require` gate
|
||||
# never runs; treating "could not determine" as "nothing touched" is precisely the
|
||||
# false green this helper exists to prevent. A stage still runs its FULL layer in
|
||||
# that case (a superset, so nothing goes untested), but says plainly that the
|
||||
# SCOPED check did not happen; ci/gate.sh turns that into an UNDECIDED decision.
|
||||
load_touched_requirements() {
|
||||
local out rc=0
|
||||
out="$("${CI_DIR}/touched-requirements.sh")" || rc=$?
|
||||
TOUCHED_RC=$rc
|
||||
TOUCHED_REQS=()
|
||||
# `if`, not `[ ... ] && ...`: under `set -e` a false AND-list is the caller's last
|
||||
# command status and would abort the stage script on an EMPTY touched set.
|
||||
if [ -n "$out" ]; then mapfile -t TOUCHED_REQS <<< "$out"; fi
|
||||
REQ_ARGS=()
|
||||
if [ "$rc" -eq 3 ]; then
|
||||
warn "touched-requirement subset UNDETERMINED (reason logged above)."
|
||||
warn " Running the FULL layer as a superset sanity net; the SCOPED R-TEST-03"
|
||||
warn " coverage check is NOT performed here — ci/gate.sh reports that as UNDECIDED."
|
||||
elif [ "$rc" -ne 0 ]; then
|
||||
# Any OTHER non-zero exit means the subset is UNUSABLE, not empty (today: a
|
||||
# path->requirement map that cannot classify a changed path, once that map fails closed).
|
||||
# The FULL layer still runs here as a superset — nothing goes untested — but this stage must
|
||||
# not claim the SCOPED check happened, and ci/gate.sh turns the same exit into a BLOCKED
|
||||
# merge decision. Wording carried over from the unlanded fix/touched-requirements-fail-closed
|
||||
# (47068ee), whose gate-side half this branch lands.
|
||||
warn "touched-requirements.sh exited ${rc} (subset UNUSABLE — see its output above)."
|
||||
warn " Running the FULL layer as a superset; the SCOPED R-TEST-03 check did NOT happen."
|
||||
warn " ci/gate.sh BLOCKS the merge on this: fix the path->requirement map in"
|
||||
warn " ci/touched-requirements.sh so every changed path is classified."
|
||||
elif [ "${#TOUCHED_REQS[@]}" -gt 0 ]; then
|
||||
log "touched requirements: ${TOUCHED_REQS[*]}"
|
||||
REQ_ARGS=(--requirements "$(IFS=,; echo "${TOUCHED_REQS[*]}")")
|
||||
else
|
||||
log "empty touched-requirement set (base WAS established) — running the full layer."
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
@@ -39,7 +39,6 @@ PYTHON="${PYTHON:-python3}"
|
||||
# DESIGN-waves.md §"Catalog services not yet onboarded"). They render as a
|
||||
# by-product of `--all catalog` but must NOT have a golden; the gate ignores
|
||||
# their rendered output. Onboarding one = add its golden and drop it here.
|
||||
#
|
||||
: "${RENDER_GOLDEN_NOT_ONBOARDED:=keycloak vm-demo}"
|
||||
|
||||
log "Stage: render==golden reproducibility (canonical render vs committed apps/)"
|
||||
|
||||
@@ -1,214 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
# =============================================================================
|
||||
# ci/render-keycloak-client-urls.py
|
||||
#
|
||||
# Render platform/keycloak/client-urls.yaml FROM platform/keycloak/keycloak-realm-import.yaml.
|
||||
#
|
||||
# WHY THIS EXISTS
|
||||
# ---------------
|
||||
# The Keycloak realm import is a ONE-SHOT that fails OPEN: once it rests at
|
||||
# Done=True / Started=False it never re-runs, so a redirect URI changed in git is
|
||||
# never applied to the realm. The client-secret half of that failure class is
|
||||
# already healed by a level-triggered CronJob (platform/keycloak/client-secret-
|
||||
# reconciler.yaml). This renderer supplies the DESIRED STATE that the same CronJob
|
||||
# needs to heal the URL half.
|
||||
#
|
||||
# THE CONSTRAINT THAT SHAPES IT: the reconciler pod runs with
|
||||
# automountServiceAccountToken: false. It has NO Kubernetes API access at all, by
|
||||
# design (R-SEC-02) — so it cannot read the KeycloakRealmImport CR to learn what
|
||||
# the realm is supposed to look like. Its inputs must be MOUNTED. Hence a
|
||||
# ConfigMap.
|
||||
#
|
||||
# A hand-authored ConfigMap would be a SECOND SOURCE OF TRUTH for every public
|
||||
# hostname in the realm, free to disagree with the realm import — and "two
|
||||
# declarations that can disagree" is the same shape as the bug this whole effort
|
||||
# is closing. So the ConfigMap is GENERATED from the realm import and the
|
||||
# generation is GATED: conformance/tests/l0-static/keycloak-client-secret-
|
||||
# reconciler/check.sh runs this script with --check on every merge, and the merge
|
||||
# fails if the committed file is not what this renderer produces. That is the same
|
||||
# render==golden idiom catalog/render.py + ci/render-golden.sh already use for
|
||||
# apps/.
|
||||
#
|
||||
# STRINGS ARE COPIED VERBATIM, NEVER RE-DERIVED. If a URL in the realm import is
|
||||
# written as a Flux substitution token (https://grafana.${BASE_DOMAIN}/...), the
|
||||
# token is carried through into the ConfigMap CHARACTER FOR CHARACTER. Both files
|
||||
# are rendered by the SAME Flux Kustomization (./platform-config, whose
|
||||
# postBuild.substituteFrom reads cluster-vars), so both are substituted in the
|
||||
# same pass with the same values and cannot disagree about what BASE_DOMAIN is.
|
||||
# This renderer never parses, splits, or reconstructs a hostname, and never reads
|
||||
# cluster-vars — so there is no second place for the domain to be decided.
|
||||
#
|
||||
# USAGE
|
||||
# python3 ci/render-keycloak-client-urls.py # print the rendered file
|
||||
# python3 ci/render-keycloak-client-urls.py --write # write it (regenerate)
|
||||
# python3 ci/render-keycloak-client-urls.py --check # exit 1 if it has drifted
|
||||
# =============================================================================
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import difflib
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("FATAL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
IMPORT_FILE = "platform/keycloak/keycloak-realm-import.yaml"
|
||||
OUT_FILE = "platform/keycloak/client-urls.yaml"
|
||||
DATA_KEY = "client-urls.json"
|
||||
CONFIGMAP_NAME = "keycloak-client-urls"
|
||||
NAMESPACE = "keycloak"
|
||||
|
||||
# The client fields this renderer exports, and therefore the ONLY fields the
|
||||
# reconciler may write. Every one of them is an environment-derived URL: it changes
|
||||
# when BASE_DOMAIN changes, and a stale value breaks OIDC login (a wrong
|
||||
# redirectUri) or silently widens trust (a stale wildcard left behind).
|
||||
#
|
||||
# DELIBERATELY ABSENT: `secret` (healed by the secret pass from the SOPS Secret,
|
||||
# and it must never appear in a ConfigMap), and every non-URL field —
|
||||
# protocolMappers, defaultClientScopes, attributes, flags. Those are not
|
||||
# environment-derived, so nothing re-renders them on a domain flip; keeping them
|
||||
# out keeps the reconciler's write surface as narrow as the problem.
|
||||
LIST_FIELDS = ("redirectUris", "webOrigins")
|
||||
STR_FIELDS = ("rootUrl", "baseUrl", "adminUrl")
|
||||
|
||||
|
||||
def render(root: Path) -> str:
|
||||
"""Return the full text of client-urls.yaml as it should be committed."""
|
||||
docs = list(yaml.safe_load_all((root / IMPORT_FILE).read_text()))
|
||||
imp = next((d for d in docs if d and d.get("kind") == "KeycloakRealmImport"), None)
|
||||
if imp is None:
|
||||
raise SystemExit("FATAL: no KeycloakRealmImport document in %s" % IMPORT_FILE)
|
||||
clients = (imp.get("spec") or {}).get("realm", {}).get("clients") or []
|
||||
if not clients:
|
||||
# Fail closed. Emitting an empty desired state would make the reconciler
|
||||
# believe every client is unmanaged and silently stop reconciling URLs.
|
||||
raise SystemExit("FATAL: %s declares zero clients — refusing to render an "
|
||||
"empty desired state" % IMPORT_FILE)
|
||||
|
||||
want: dict[str, dict] = {}
|
||||
for c in clients:
|
||||
cid = c.get("clientId")
|
||||
if not isinstance(cid, str) or not cid:
|
||||
raise SystemExit("FATAL: %s has a client with no clientId" % IMPORT_FILE)
|
||||
entry: dict[str, object] = {}
|
||||
for f in LIST_FIELDS:
|
||||
if f not in c:
|
||||
continue # not declared => NOT MANAGED (never forced empty)
|
||||
v = c[f]
|
||||
if not isinstance(v, list) or not all(isinstance(x, str) for x in v):
|
||||
raise SystemExit("FATAL: %s client %r field %s is not a list of strings"
|
||||
% (IMPORT_FILE, cid, f))
|
||||
entry[f] = list(v) # verbatim, order as authored
|
||||
for f in STR_FIELDS:
|
||||
if f not in c:
|
||||
continue
|
||||
v = c[f]
|
||||
if not isinstance(v, str):
|
||||
raise SystemExit("FATAL: %s client %r field %s is not a string"
|
||||
% (IMPORT_FILE, cid, f))
|
||||
entry[f] = v
|
||||
if entry:
|
||||
want[cid] = entry
|
||||
if not want:
|
||||
raise SystemExit("FATAL: no client in %s declares any of %s — refusing to render "
|
||||
"an empty desired state"
|
||||
% (IMPORT_FILE, ", ".join(LIST_FIELDS + STR_FIELDS)))
|
||||
|
||||
body = json.dumps(want, indent=2, sort_keys=True, ensure_ascii=False)
|
||||
indented = "\n".join((" " + ln) if ln else "" for ln in body.splitlines())
|
||||
return HEADER + indented + "\n"
|
||||
|
||||
|
||||
HEADER = """---
|
||||
# =============================================================================
|
||||
# GENERATED FILE — DO NOT EDIT BY HAND.
|
||||
#
|
||||
# source: platform/keycloak/keycloak-realm-import.yaml (spec.realm.clients)
|
||||
# regenerate: python3 ci/render-keycloak-client-urls.py --write
|
||||
# gated by: conformance/tests/l0-static/keycloak-client-secret-reconciler
|
||||
# (runs this renderer with --check on every merge)
|
||||
#
|
||||
# WHAT THIS IS: the DESIRED STATE for every environment-derived client URL in the
|
||||
# `homelab` realm — redirectUris, webOrigins, rootUrl, baseUrl, adminUrl. The
|
||||
# client-secret reconciler CronJob mounts it and makes the live realm match it.
|
||||
#
|
||||
# WHY IT IS A SEPARATE OBJECT: the realm import is a ONE-SHOT that never re-runs
|
||||
# once Done=True, so a redirect URI edited in git never reaches the realm. Prod's
|
||||
# stored redirect URIs were measured still pointing at the pre-migration hostname
|
||||
# while git declared the new one, and every OIDC login would have failed with a
|
||||
# redirect_uri mismatch. The reconciler is the level-triggered loop that closes
|
||||
# that; this file is what it reconciles TOWARDS.
|
||||
#
|
||||
# WHY IT IS GENERATED AND NOT AUTHORED: a hand-written copy of every public
|
||||
# hostname would be a second source of truth, free to disagree with the realm
|
||||
# import. It is rendered from the import and the render is gated, so the two
|
||||
# cannot drift. URL strings are copied VERBATIM — if the source uses a
|
||||
# ${BASE_DOMAIN} token it appears here unchanged, and both files are substituted
|
||||
# by the SAME Flux Kustomization (./platform-config), so the domain is decided in
|
||||
# exactly one place.
|
||||
#
|
||||
# A field that is NOT declared on a client in the realm import is simply ABSENT
|
||||
# here, and the reconciler leaves it alone. Absence means UNMANAGED, never
|
||||
# "force to empty".
|
||||
# =============================================================================
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: %s
|
||||
namespace: %s
|
||||
labels:
|
||||
app.kubernetes.io/name: %s
|
||||
app.kubernetes.io/part-of: keycloak
|
||||
data:
|
||||
%s: |
|
||||
""" % (CONFIGMAP_NAME, NAMESPACE, CONFIGMAP_NAME, DATA_KEY)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
g = ap.add_mutually_exclusive_group()
|
||||
g.add_argument("--write", action="store_true", help="write %s" % OUT_FILE)
|
||||
g.add_argument("--check", action="store_true",
|
||||
help="exit 1 if the committed file has drifted from the source")
|
||||
args = ap.parse_args()
|
||||
|
||||
want = render(ROOT)
|
||||
target = ROOT / OUT_FILE
|
||||
|
||||
if args.write:
|
||||
target.write_text(want, encoding="utf-8")
|
||||
print("wrote %s" % OUT_FILE)
|
||||
return 0
|
||||
|
||||
if args.check:
|
||||
if not target.is_file():
|
||||
print("CHECK FAIL: %s does not exist. Run: python3 "
|
||||
"ci/render-keycloak-client-urls.py --write" % OUT_FILE, file=sys.stderr)
|
||||
return 1
|
||||
have = target.read_text(encoding="utf-8")
|
||||
if have == want:
|
||||
print("client-urls.yaml is in sync with %s" % IMPORT_FILE)
|
||||
return 0
|
||||
print("CHECK FAIL: %s has DRIFTED from %s — the reconciler would converge the "
|
||||
"realm on a desired state the realm import does not declare.\n"
|
||||
"Regenerate with: python3 ci/render-keycloak-client-urls.py --write\n"
|
||||
% (OUT_FILE, IMPORT_FILE), file=sys.stderr)
|
||||
for line in difflib.unified_diff(have.splitlines(True), want.splitlines(True),
|
||||
fromfile="committed " + OUT_FILE,
|
||||
tofile="rendered from " + IMPORT_FILE):
|
||||
sys.stderr.write(line)
|
||||
return 1
|
||||
|
||||
sys.stdout.write(want)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,189 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/tests/gate-decision.test.sh — the merge gate's OWN decision table.
|
||||
#
|
||||
# WHY THIS SUITE EXISTS (2026-09-13). ci/gate.sh used to set BLOCK=1 for a FAILED stage but
|
||||
# only `warn` for a SKIPPED one, so it printed the words "— not green" and then concluded
|
||||
# `MERGE GATE: PASS` and exited 0. Observed live twice: `prom-rules` skipped for a missing
|
||||
# promtool, and `l0` skipped for a missing pinned kyverno CLI — both read by automation as
|
||||
# merge-approved. That is fail-open in the one component whose entire job is to fail closed,
|
||||
# and nothing tested it: every other ci/tests suite covers the INPUT (touched-requirements),
|
||||
# none covered the DECISION.
|
||||
#
|
||||
# WHAT IS UNDER TEST: the real ci/gate.sh and ci/lib.sh, copied into a throwaway CI_DIR beside
|
||||
# STUB stage scripts whose exit codes this suite dictates (STAGE_RC_<stage>) and a stub
|
||||
# touched-requirements.sh whose rc/reason it dictates (TR_RC / TR_REASON / TR_OUT). Stubbing the
|
||||
# stages is the point: this asserts how the gate REASONS about pass/fail/skip, independently of
|
||||
# what any stage happens to do on this machine — so it runs anywhere, with no tools, no cluster,
|
||||
# no network, in milliseconds.
|
||||
#
|
||||
# THE INVARIANT: `MERGE GATE: PASS` + exit 0 means every stage that was supposed to run RAN and
|
||||
# was GREEN. A stage that could not run is never a pass, and the exit code says so too.
|
||||
#
|
||||
# Run: ci/tests/gate-decision.test.sh (or `make -C ci test`)
|
||||
set -uo pipefail
|
||||
|
||||
REAL_CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
|
||||
PASS=0 FAIL=0
|
||||
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
|
||||
bad() { printf ' FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
|
||||
|
||||
TMP="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMP"' EXIT
|
||||
CI="${TMP}/ci"; mkdir -p "$CI"
|
||||
REPO="${TMP}/repo"; mkdir -p "$REPO"
|
||||
|
||||
cp "${REAL_CI_DIR}/gate.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env" "$CI/"
|
||||
|
||||
# One stub per stage. It exits with STAGE_RC_<stage-with-dashes-as-underscores>, default 0.
|
||||
for stage in spec-immutability lint l0 render-golden l1 conformance; do
|
||||
cat > "${CI}/${stage}.sh" <<'STUB'
|
||||
#!/usr/bin/env bash
|
||||
n="$(basename "$0" .sh)"; v="STAGE_RC_${n//-/_}"
|
||||
exit "${!v:-0}"
|
||||
STUB
|
||||
chmod +x "${CI}/${stage}.sh"
|
||||
done
|
||||
|
||||
# Stub decision input. TR_RC (default 0), TR_REASON (written to $TOUCHED_REASON_FILE like the
|
||||
# real script does), TR_OUT (the touched requirement list on stdout).
|
||||
cat > "${CI}/touched-requirements.sh" <<'STUB'
|
||||
#!/usr/bin/env bash
|
||||
if [ -n "${TOUCHED_REASON_FILE:-}" ]; then printf '%s\n' "${TR_REASON:-}" > "$TOUCHED_REASON_FILE"; fi
|
||||
[ -n "${TR_OUT:-}" ] && printf '%s\n' "$TR_OUT"
|
||||
exit "${TR_RC:-0}"
|
||||
STUB
|
||||
chmod +x "${CI}/touched-requirements.sh"
|
||||
|
||||
# run_gate <VAR=VAL>... — run the gate with a clean, explicit environment. Sets OUT and RC.
|
||||
# GATE_STRICT_SKIPS is pinned false unless a case says otherwise, so the suite never depends on
|
||||
# whether the machine running it happens to look like CI.
|
||||
OUT="" RC=0
|
||||
run_gate() {
|
||||
local summary_file="${TMP}/summary.md"; : > "$summary_file"
|
||||
OUT="$(env -i PATH="$PATH" HOME="$HOME" \
|
||||
REPO_ROOT="$REPO" GITHUB_STEP_SUMMARY="$summary_file" \
|
||||
GATE_STRICT_SKIPS=false TR_OUT="R-TEST-03" \
|
||||
"$@" bash "${CI}/gate.sh" 2>&1)"
|
||||
RC=$?
|
||||
SUMMARY="$(cat "$summary_file")"
|
||||
}
|
||||
|
||||
# assert_verdict <headline> <expected-rc> <label>
|
||||
assert_verdict() {
|
||||
local want="$1" want_rc="$2" label="$3"
|
||||
if ! grep -qF "MERGE GATE: ${want}" <<<"$OUT"; then
|
||||
bad "${label}: headline should be '${want}' — got: $(grep -oE 'MERGE GATE: [A-Z ]+' <<<"$OUT" | tail -1)"
|
||||
elif [ "$RC" -ne "$want_rc" ]; then
|
||||
bad "${label}: headline '${want}' correct but exit code ${RC} != ${want_rc}"
|
||||
else
|
||||
ok "${label} -> ${want}, exit ${want_rc}"
|
||||
fi
|
||||
}
|
||||
# The load-bearing negative: nothing may say PASS, and nothing may exit 0.
|
||||
assert_not_pass() {
|
||||
local label="$1"
|
||||
if grep -qF "MERGE GATE: PASS" <<<"$OUT"; then bad "${label}: printed 'MERGE GATE: PASS'"
|
||||
elif [ "$RC" -eq 0 ]; then bad "${label}: exited 0"
|
||||
else ok "${label}: not a PASS, non-zero exit (${RC})"; fi
|
||||
}
|
||||
|
||||
echo "== 1. the happy path: every stage green"
|
||||
run_gate
|
||||
assert_verdict PASS 0 "all stages green + a touched subset"
|
||||
grep -qF "nothing was declared out of scope" <<<"$SUMMARY" \
|
||||
&& ok "PASS summary states nothing was declared out of scope" \
|
||||
|| bad "PASS summary does not state the absence of declared gaps"
|
||||
|
||||
echo
|
||||
echo "== 2. a FAILED stage blocks (unchanged behaviour, locked down)"
|
||||
for s in lint l0 render-golden l1 conformance spec-immutability; do
|
||||
run_gate "STAGE_RC_${s//-/_}=1"
|
||||
assert_verdict BLOCKED 1 "${s} FAILED"
|
||||
done
|
||||
|
||||
echo
|
||||
echo "== 3. THE REGRESSION: a SKIPPED stage is not a PASS and not exit 0"
|
||||
for s in lint l0 render-golden l1; do
|
||||
run_gate "STAGE_RC_${s//-/_}=3"
|
||||
assert_not_pass "${s} SKIPPED (local)"
|
||||
assert_verdict UNDETERMINED 3 "${s} SKIPPED (local)"
|
||||
grep -qF "$s" <<<"$SUMMARY" && ok "${s}: the step summary names the stage that did not run" \
|
||||
|| bad "${s}: step summary does not name the skipped stage"
|
||||
done
|
||||
|
||||
echo
|
||||
echo "== 4. under STRICT skips (CI declares it) a could-not-run skip BLOCKS, never merely 3"
|
||||
for s in lint l0 render-golden l1; do
|
||||
run_gate GATE_STRICT_SKIPS=true "STAGE_RC_${s//-/_}=3"
|
||||
assert_verdict BLOCKED 1 "${s} SKIPPED with GATE_STRICT_SKIPS=true"
|
||||
done
|
||||
# ...which is what makes gate.yml's `|| [ $? -eq 3 ]` safe: in CI, exit 3 cannot be a missing
|
||||
# tool or an unwired seam. Only the declared-environmental skip below can produce it.
|
||||
run_gate GATE_STRICT_SKIPS=true STAGE_RC_l0=3 STAGE_RC_lint=3
|
||||
assert_verdict BLOCKED 1 "several stages SKIPPED under strict skips"
|
||||
|
||||
echo
|
||||
echo "== 5. spec-immutability: unverified is never green, but is not upgraded to a block by CI"
|
||||
run_gate STAGE_RC_spec_immutability=3
|
||||
assert_not_pass "spec-immutability SKIPPED"
|
||||
assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED (local)"
|
||||
run_gate GATE_STRICT_SKIPS=true STAGE_RC_spec_immutability=3
|
||||
assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED under strict skips (deliberate exception)"
|
||||
run_gate REQUIRE_SPEC_VERIFICATION=true STAGE_RC_spec_immutability=3
|
||||
assert_verdict BLOCKED 1 "spec-immutability SKIPPED + REQUIRE_SPEC_VERIFICATION=true"
|
||||
|
||||
echo
|
||||
echo "== 6. conformance L2/L3: a DECLARED-absent capability may still pass — and is NAMED"
|
||||
run_gate STAGE_RC_conformance=3
|
||||
assert_verdict PASS 0 "conformance SKIPPED, no cluster declared"
|
||||
grep -qF "conformance-l2-l3" <<<"$OUT" \
|
||||
&& ok "the PASS headline names the declared L2/L3 gap" \
|
||||
|| bad "the PASS headline hides the declared L2/L3 gap"
|
||||
grep -qF "conformance-l2-l3" <<<"$SUMMARY" \
|
||||
&& ok "the step summary names the declared L2/L3 gap" \
|
||||
|| bad "the step summary hides the declared L2/L3 gap"
|
||||
run_gate CLUSTER_AVAILABLE=true STAGE_RC_conformance=3
|
||||
assert_verdict BLOCKED 1 "conformance SKIPPED despite CLUSTER_AVAILABLE=true"
|
||||
run_gate REQUIRE_CLUSTER_TESTS=true STAGE_RC_conformance=3
|
||||
assert_verdict BLOCKED 1 "conformance SKIPPED + REQUIRE_CLUSTER_TESTS=true with a touched subset"
|
||||
# A misspelled switch must leave the guard OFF (truthy()), never half-on.
|
||||
run_gate CLUSTER_AVAILABLE=ture STAGE_RC_conformance=3
|
||||
assert_verdict PASS 0 "CLUSTER_AVAILABLE=ture is NOT 'available' (no cluster was declared)"
|
||||
run_gate CLUSTER_AVAILABLE=TRUE STAGE_RC_conformance=3
|
||||
assert_verdict BLOCKED 1 "CLUSTER_AVAILABLE=TRUE IS a declaration (skip then blocks)"
|
||||
|
||||
echo
|
||||
echo "== 7. the undetermined touched subset: unchanged (2026-08-31 false-green paths)"
|
||||
run_gate TR_RC=3 TR_REASON=on-base TR_OUT=
|
||||
assert_verdict "NOT APPLICABLE" 0 "subset undeterminable, reason on-base, all stages green"
|
||||
run_gate TR_RC=3 TR_REASON=no-base TR_OUT=
|
||||
assert_verdict BLOCKED 1 "subset undeterminable, reason no-base"
|
||||
run_gate TR_RC=3 TR_REASON=diff-failed TR_OUT=
|
||||
assert_verdict BLOCKED 1 "subset undeterminable, reason diff-failed"
|
||||
# Precedence: a stage that did not run outranks "nothing to gate" — the stronger, non-zero
|
||||
# headline must win, or a push-to-main run could hide a missing tool behind NOT APPLICABLE.
|
||||
run_gate TR_RC=3 TR_REASON=on-base TR_OUT= STAGE_RC_l1=3
|
||||
assert_not_pass "on-base AND l1 SKIPPED"
|
||||
assert_verdict UNDETERMINED 3 "on-base AND l1 SKIPPED (UNDETERMINED outranks NOT APPLICABLE)"
|
||||
|
||||
echo
|
||||
echo "== 8. an UNUSABLE subset blocks (carried over from 47068ee): any other non-zero exit"
|
||||
for rc in 1 2 4; do
|
||||
run_gate "TR_RC=${rc}" TR_REASON=unclassified-path TR_OUT=
|
||||
assert_verdict BLOCKED 1 "touched-requirements.sh exited ${rc} (subset unusable, not empty)"
|
||||
done
|
||||
run_gate TR_RC=1 TR_REASON=unclassified-path TR_OUT=
|
||||
grep -qF "UNUSABLE" <<<"$SUMMARY" && ok "the step summary calls the subset UNUSABLE" \
|
||||
|| bad "the step summary does not report an unusable subset"
|
||||
|
||||
echo
|
||||
echo "== 9. combinations: a FAILURE outranks a non-run, and the headline is the worst news"
|
||||
run_gate STAGE_RC_l0=1 STAGE_RC_l1=3
|
||||
assert_verdict BLOCKED 1 "l0 FAILED + l1 SKIPPED"
|
||||
run_gate STAGE_RC_spec_immutability=3 STAGE_RC_conformance=3
|
||||
assert_verdict UNDETERMINED 3 "spec-immutability SKIPPED + conformance declared-absent"
|
||||
|
||||
echo
|
||||
if [ "$FAIL" -eq 0 ]; then echo "gate-decision tests: PASS ($PASS assertions)"; exit 0
|
||||
else echo "gate-decision tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
|
||||
@@ -1,169 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/tests/touched-requirements-base.test.sh — regression tests for BASE-REF RESOLUTION
|
||||
# in ci/touched-requirements.sh.
|
||||
#
|
||||
# THE BUG THESE LOCK DOWN (2026-08-31, local/CI divergence):
|
||||
# `make -C ci gate` on a feature branch printed
|
||||
# "no changed files vs base; empty touched-requirement set"
|
||||
# and then "MERGE GATE: PASS" — while CI, with a correct base, computed 17 touched
|
||||
# requirements and BLOCKED on real coverage debt. Root cause: EVERY way of failing to
|
||||
# establish a diff base degraded silently to an EMPTY touched set and exit 0. An empty
|
||||
# set means conformance/run.sh is invoked without `--requirements`, which means its
|
||||
# coverage `--require` gate never runs — so the gate reported PASS having checked
|
||||
# nothing. A false green is worse than a red.
|
||||
#
|
||||
# The fix: an undeterminable base is a LOUD exit 3 (ci/README.md's SKIPPED code) with
|
||||
# EMPTY stdout and a machine-readable reason code, never exit 0. These tests assert
|
||||
# that for each way the base can be undeterminable, and assert the good paths still work.
|
||||
#
|
||||
# The tests point the real script at THROWAWAY git repos via the REPO_ROOT override
|
||||
# (ci/lib.sh honours a pre-set REPO_ROOT); the script runs from its true location so it
|
||||
# still sources the real lib.sh. No network, no cluster, and the working repo is never
|
||||
# touched.
|
||||
#
|
||||
# Run: ci/tests/touched-requirements-base.test.sh (or `make -C ci test`)
|
||||
set -uo pipefail # NOT -e: these tests deliberately run a script that exits non-zero.
|
||||
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
SCRIPT="${CI_DIR}/touched-requirements.sh"
|
||||
|
||||
PASS=0 FAIL=0
|
||||
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
|
||||
bad() { printf ' FAIL %s\n' "$*" >&2; FAIL=$((FAIL+1)); }
|
||||
|
||||
TMP="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMP"' EXIT
|
||||
|
||||
# newrepo <dir> — a throwaway repo with one baseline commit on `main`.
|
||||
newrepo() {
|
||||
local d="$1"
|
||||
mkdir -p "$d/notes"
|
||||
git -C "$d" init -q -b main
|
||||
git -C "$d" config user.email [email protected]
|
||||
git -C "$d" config user.name t
|
||||
git -C "$d" config commit.gpgsign false # throwaway: never sign (ignore host gpg config)
|
||||
echo baseline > "$d/notes/base.txt"
|
||||
git -C "$d" add -A && git -C "$d" commit -qm baseline
|
||||
}
|
||||
|
||||
# commit_touching <dir> <text> — a commit whose content declares an R-ID.
|
||||
commit_touching() {
|
||||
local d="$1" text="$2"
|
||||
printf 'name: %s\n' "$text" > "$d/notes/decl.yaml"
|
||||
git -C "$d" add -A && git -C "$d" commit -qm change
|
||||
}
|
||||
|
||||
# Runs the script in <dir>; sets RC, OUT (stdout), ERRTXT (stderr), REASON.
|
||||
run_in() {
|
||||
local d="$1"; shift
|
||||
local reason_file="${TMP}/reason.$$"
|
||||
: > "$reason_file"
|
||||
OUT="$(REPO_ROOT="$d" TOUCHED_REASON_FILE="$reason_file" "$@" "$SCRIPT" 2>"${TMP}/err.$$")"
|
||||
RC=$?
|
||||
ERRTXT="$(cat "${TMP}/err.$$")"
|
||||
REASON="$(cat "$reason_file")"
|
||||
}
|
||||
|
||||
# assert_refusal <label> <expected-reason>
|
||||
assert_refusal() {
|
||||
local label="$1" want="$2"
|
||||
[ "$RC" = 3 ] && ok "${label}: exits 3 (SKIPPED), not 0" || bad "${label}: rc=${RC}, expected 3"
|
||||
[ -z "$OUT" ] && ok "${label}: stdout is EMPTY (no silent requirement list)" \
|
||||
|| bad "${label}: stdout was '${OUT}'"
|
||||
[ "$REASON" = "$want" ] && ok "${label}: reason code '${want}'" \
|
||||
|| bad "${label}: reason '${REASON}', expected '${want}'"
|
||||
grep -q "NOT DETERMINABLE" <<<"$ERRTXT" && ok "${label}: announces the refusal LOUDLY on stderr" \
|
||||
|| bad "${label}: stderr lacked the loud banner"
|
||||
}
|
||||
|
||||
echo "== case 1: a branch ahead of local main (no remote) resolves and reports"
|
||||
R1="${TMP}/r1"; newrepo "$R1"
|
||||
git -C "$R1" checkout -q -b feature
|
||||
commit_touching "$R1" implements-R-ALPHA-01
|
||||
run_in "$R1"
|
||||
[ "$RC" = 0 ] && ok "ahead-of-main: exits 0" || bad "ahead-of-main: rc=${RC}"
|
||||
grep -qx "R-ALPHA-01" <<<"$OUT" && ok "ahead-of-main: emits the touched R-ID" \
|
||||
|| bad "ahead-of-main: output was '${OUT}'"
|
||||
grep -q "diff base: main" <<<"$ERRTXT" && ok "ahead-of-main: names the base it used" \
|
||||
|| bad "ahead-of-main: base not reported: ${ERRTXT}"
|
||||
|
||||
echo "== case 2: THE FALSE GREEN — HEAD is the base branch itself"
|
||||
# This is the state that produced 'no changed files vs base' + 'MERGE GATE: PASS'
|
||||
# locally while CI blocked. It must now refuse, not report an empty set.
|
||||
R2="${TMP}/r2"; newrepo "$R2"
|
||||
git -C "$R2" checkout -q -b feature
|
||||
commit_touching "$R2" implements-R-ALPHA-01
|
||||
git -C "$R2" checkout -q main # <- sitting on the base branch
|
||||
run_in "$R2"
|
||||
assert_refusal "on-base" "on-base"
|
||||
|
||||
echo "== case 3: a branch already merged into the base (HEAD is an ancestor)"
|
||||
R3="${TMP}/r3"; newrepo "$R3"
|
||||
git -C "$R3" checkout -q -b feature
|
||||
commit_touching "$R3" implements-R-ALPHA-01
|
||||
git -C "$R3" checkout -q main
|
||||
git -C "$R3" merge -q --no-edit feature
|
||||
git -C "$R3" checkout -q feature # feature is now behind main
|
||||
run_in "$R3"
|
||||
assert_refusal "already-merged" "on-base"
|
||||
|
||||
echo "== case 4: no base ref resolves at all (no main, no origin/main)"
|
||||
R4="${TMP}/r4"; newrepo "$R4"
|
||||
git -C "$R4" checkout -q -b work
|
||||
commit_touching "$R4" implements-R-ALPHA-01
|
||||
git -C "$R4" branch -q -D main
|
||||
run_in "$R4"
|
||||
assert_refusal "no-base-ref" "no-base"
|
||||
|
||||
echo "== case 5: a single-commit repo with no main (the removed HEAD~1 last resort)"
|
||||
# HEAD~1 does not exist here. The old code fell back to it, the diff failed, and the
|
||||
# empty result was reported as 'no changed files' + exit 0 — a silent false green.
|
||||
R5="${TMP}/r5"; mkdir -p "$R5/notes"
|
||||
git -C "$R5" init -q -b work
|
||||
git -C "$R5" config user.email [email protected]; git -C "$R5" config user.name t
|
||||
git -C "$R5" config commit.gpgsign false
|
||||
echo only > "$R5/notes/only.txt"; git -C "$R5" add -A; git -C "$R5" commit -qm only
|
||||
run_in "$R5"
|
||||
assert_refusal "single-commit-no-main" "no-base"
|
||||
|
||||
echo "== case 6: BASE_REF names a ref that does not exist"
|
||||
R6="${TMP}/r6"; newrepo "$R6"
|
||||
git -C "$R6" checkout -q -b feature
|
||||
commit_touching "$R6" implements-R-ALPHA-01
|
||||
run_in "$R6" env BASE_REF=refs/heads/nope
|
||||
assert_refusal "bogus-BASE_REF" "no-base"
|
||||
|
||||
echo "== case 7: GITHUB_BASE_REF set but absent from the checkout (shallow clone)"
|
||||
R7="${TMP}/r7"; newrepo "$R7"
|
||||
git -C "$R7" checkout -q -b feature
|
||||
commit_touching "$R7" implements-R-ALPHA-01
|
||||
run_in "$R7" env GITHUB_BASE_REF=release-9
|
||||
assert_refusal "missing-GITHUB_BASE_REF" "no-base"
|
||||
|
||||
echo "== case 8: origin/main is preferred over a stale local main, and staleness warns"
|
||||
R8="${TMP}/r8"; newrepo "$R8"
|
||||
UP="${TMP}/r8-origin.git"; git init -q --bare -b main "$UP"
|
||||
git -C "$R8" remote add origin "$UP"
|
||||
git -C "$R8" push -q origin main
|
||||
# advance origin/main beyond the local main ref, then branch off it
|
||||
git -C "$R8" checkout -q -b ahead
|
||||
echo more > "$R8/notes/more.txt"; git -C "$R8" add -A; git -C "$R8" commit -qm more
|
||||
git -C "$R8" push -q origin ahead:main
|
||||
git -C "$R8" fetch -q origin
|
||||
git -C "$R8" checkout -q -b feature origin/main
|
||||
commit_touching "$R8" implements-R-ALPHA-01
|
||||
run_in "$R8"
|
||||
[ "$RC" = 0 ] && ok "origin-preferred: exits 0" || bad "origin-preferred: rc=${RC}"
|
||||
grep -q "diff base: origin/main" <<<"$ERRTXT" && ok "origin-preferred: uses origin/main, not local main" \
|
||||
|| bad "origin-preferred: base line was: ${ERRTXT}"
|
||||
grep -q "may be stale" <<<"$ERRTXT" && ok "origin-preferred: warns that local main differs" \
|
||||
|| bad "origin-preferred: no staleness warning"
|
||||
grep -qx "R-ALPHA-01" <<<"$OUT" && ok "origin-preferred: emits the touched R-ID" \
|
||||
|| bad "origin-preferred: output was '${OUT}'"
|
||||
|
||||
echo
|
||||
if [ "$FAIL" -eq 0 ]; then
|
||||
echo "touched-requirements base-resolution tests: PASS (${PASS} assertions)"; exit 0
|
||||
fi
|
||||
echo "touched-requirements base-resolution tests: FAIL (${FAIL} failed, ${PASS} passed)" >&2
|
||||
exit 1
|
||||
@@ -1,164 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/tests/touched-requirements-test-attribution.test.sh — regression tests for DETECTION
|
||||
# RULE 3 in ci/touched-requirements.sh: a change anywhere under `conformance/tests/<test>/`
|
||||
# attributes THAT TEST'S OWN `conformance.home.k3s/R-*` labels.
|
||||
#
|
||||
# THE GAP THESE LOCK DOWN (2026-09-12). The coarse path map keys `conformance/*` to
|
||||
# R-TEST-01..04 — the requirements about the HARNESS — and says nothing about what a given
|
||||
# test PROVES. So weakening `conformance/tests/.../check.sh` attributed R-TEST-01..04 and
|
||||
# nothing else: the merge gate ran the harness's self-tests and NOT the weakened test's own
|
||||
# requirement subset. Weakening (or deleting) a test is the one change that most needs the
|
||||
# requirement it certifies to be re-verified, and it was the change least likely to select it.
|
||||
# The content scan cannot close this: it reads only ADDED/REMOVED lines, so it sees a change
|
||||
# to the LABEL BLOCK but not one to the test BODY — exactly the case that matters.
|
||||
#
|
||||
# Also covers the two path-map entries added in the same change (infra/* -> R-ENV-01, and
|
||||
# R-TEST-06 on .gitea/*|ci/*), which is how the two L0 checks that are DELIBERATELY not
|
||||
# whole-repo-scoped get selected by the paths that can break them.
|
||||
#
|
||||
# Same harness as the sibling suites: the real script, pointed at a THROWAWAY git repo via
|
||||
# the REPO_ROOT override. No network, no cluster, the working repo is never touched.
|
||||
#
|
||||
# Run: ci/tests/touched-requirements-test-attribution.test.sh (or `make -C ci test`)
|
||||
set -euo pipefail
|
||||
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
SCRIPT="${CI_DIR}/touched-requirements.sh"
|
||||
|
||||
PASS=0 FAIL=0
|
||||
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
|
||||
bad() { printf ' FAIL %s\n' "$*" >&2; FAIL=$((FAIL+1)); }
|
||||
|
||||
assert_present() {
|
||||
if grep -qxF "$1" <<<"$2"; then ok "$3"
|
||||
else bad "$3 (missing '$1' from: $(tr '\n' ' ' <<<"$2"))"; fi
|
||||
}
|
||||
assert_absent() {
|
||||
if grep -qxF "$1" <<<"$2"; then bad "$3 (found unexpected '$1' in: $(tr '\n' ' ' <<<"$2"))"
|
||||
else ok "$3"; fi
|
||||
}
|
||||
|
||||
TMP="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMP"' EXIT
|
||||
|
||||
git -C "$TMP" init -q -b main
|
||||
git -C "$TMP" config user.email [email protected]
|
||||
git -C "$TMP" config user.name t
|
||||
git -C "$TMP" config commit.gpgsign false
|
||||
|
||||
# --- Baseline: two conformance tests, each declaring its own requirement labels ----------
|
||||
# TEST A: a flat l0-static test. TEST B: a test with a NESTED fixture dir, to prove the
|
||||
# walk-up from an arbitrary changed file to the enclosing test dir.
|
||||
mkdir -p "$TMP/conformance/tests/l0-static/alpha" \
|
||||
"$TMP/conformance/tests/l2-behavioral/beta/fixtures" \
|
||||
"$TMP/conformance/lib" "$TMP/infra" "$TMP/ci"
|
||||
|
||||
cat > "$TMP/conformance/tests/l0-static/alpha/chainsaw-test.yaml" <<'YAML'
|
||||
# HEADER PROSE. This test used to also cover conformance.home.k3s/R-STALE-09: "true",
|
||||
# which is a REFERENCE in a comment, not a declaration, and must NOT be attributed.
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: alpha
|
||||
labels:
|
||||
conformance.home.k3s/R-ALPHA-01: "true"
|
||||
conformance.home.k3s/R-ALPHA-02: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/layer: "L0"
|
||||
spec: {}
|
||||
YAML
|
||||
# The check BODY deliberately contains NO R-ID text, so anything the scan finds for a
|
||||
# body-only edit came from rule 3 and not from the content scan.
|
||||
cat > "$TMP/conformance/tests/l0-static/alpha/check.sh" <<'SH'
|
||||
#!/usr/bin/env bash
|
||||
echo "assert something"
|
||||
SH
|
||||
|
||||
cat > "$TMP/conformance/tests/l2-behavioral/beta/chainsaw-test.yaml" <<'YAML'
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: beta
|
||||
labels:
|
||||
conformance.home.k3s/R-BETA-07: "true"
|
||||
spec: {}
|
||||
YAML
|
||||
echo "seed: 1" > "$TMP/conformance/tests/l2-behavioral/beta/fixtures/data.yaml"
|
||||
|
||||
echo "harness helper" > "$TMP/conformance/lib/helper.sh"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm baseline
|
||||
|
||||
# ========================================================================================
|
||||
echo "== case 1: a BODY-ONLY edit to a test attributes that test's own labels"
|
||||
git -C "$TMP" checkout -q -b body-edit
|
||||
cat > "$TMP/conformance/tests/l0-static/alpha/check.sh" <<'SH'
|
||||
#!/usr/bin/env bash
|
||||
echo "assert something weaker"
|
||||
SH
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm weaken
|
||||
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set:\n%s\n---\n' "$OUT"
|
||||
assert_present "R-ALPHA-01" "$OUT" "body-only test edit attributes the test's own R-ALPHA-01"
|
||||
assert_present "R-ALPHA-02" "$OUT" "body-only test edit attributes the test's own R-ALPHA-02"
|
||||
assert_present "R-TEST-03" "$OUT" "the conformance/* path map still applies (R-TEST-03)"
|
||||
assert_absent "R-STALE-09" "$OUT" "an R-ID cited only in the test's HEADER COMMENT is NOT attributed"
|
||||
assert_absent "R-BETA-07" "$OUT" "an UNRELATED test's labels are NOT attributed"
|
||||
|
||||
# ========================================================================================
|
||||
echo "== case 2: an edit to a NESTED file walks up to the enclosing test"
|
||||
git -C "$TMP" checkout -q -b nested-edit main
|
||||
echo "seed: 2" > "$TMP/conformance/tests/l2-behavioral/beta/fixtures/data.yaml"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm fixture
|
||||
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set:\n%s\n---\n' "$OUT"
|
||||
assert_present "R-BETA-07" "$OUT" "nested fixture edit attributes the enclosing test's R-BETA-07"
|
||||
assert_absent "R-ALPHA-01" "$OUT" "nested fixture edit does NOT attribute a sibling test's labels"
|
||||
|
||||
# ========================================================================================
|
||||
echo "== case 3: DELETING a test still attributes what it used to certify"
|
||||
# Removing the only verifier of a requirement must not be the cheapest way to make the gate
|
||||
# stop checking it — so the labels are read from the BASE when HEAD no longer has them.
|
||||
git -C "$TMP" checkout -q -b delete-test main
|
||||
git -C "$TMP" rm -rq "$TMP/conformance/tests/l0-static/alpha"
|
||||
git -C "$TMP" commit -qm "delete the alpha test"
|
||||
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set:\n%s\n---\n' "$OUT"
|
||||
assert_present "R-ALPHA-01" "$OUT" "deleting a test STILL attributes its R-ALPHA-01 (read from base)"
|
||||
assert_present "R-ALPHA-02" "$OUT" "deleting a test STILL attributes its R-ALPHA-02 (read from base)"
|
||||
|
||||
# ========================================================================================
|
||||
echo "== case 4: a change OUTSIDE conformance/tests/ gains no test labels"
|
||||
git -C "$TMP" checkout -q -b lib-edit main
|
||||
echo "harness helper v2" > "$TMP/conformance/lib/helper.sh"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm lib
|
||||
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set:\n%s\n---\n' "$OUT"
|
||||
assert_present "R-TEST-02" "$OUT" "a conformance/lib edit still attributes the harness requirements"
|
||||
assert_absent "R-ALPHA-01" "$OUT" "a conformance/lib edit attributes NO individual test's labels"
|
||||
assert_absent "R-BETA-07" "$OUT" "a conformance/lib edit attributes NO individual test's labels (B)"
|
||||
|
||||
# ========================================================================================
|
||||
echo "== case 5: path-map entries for the two DELIBERATELY path-scoped L0 checks"
|
||||
# infra/* -> R-ENV-01 (dev-nodes-provisioned-as-code reads nothing but infra/).
|
||||
git -C "$TMP" checkout -q -b infra-edit main
|
||||
printf 'resource "x" "y" {}\n' > "$TMP/infra/main.tf"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm infra
|
||||
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set:\n%s\n---\n' "$OUT"
|
||||
assert_present "R-ENV-01" "$OUT" "an infra/ edit attributes R-ENV-01 (was UNMAPPED: attributed nothing)"
|
||||
|
||||
# ci/* -> ... R-TEST-06 (spec-immutability-enforced reads nothing but ci/ + the spec repo).
|
||||
git -C "$TMP" checkout -q -b ci-edit main
|
||||
printf 'echo hello\n' > "$TMP/ci/thing.sh"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm ci
|
||||
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set:\n%s\n---\n' "$OUT"
|
||||
assert_present "R-TEST-06" "$OUT" "a ci/ edit attributes R-TEST-06 (the spec-immutability test's own ID)"
|
||||
assert_present "R-GITOPS-01" "$OUT" "a ci/ edit still attributes R-GITOPS-01"
|
||||
|
||||
echo
|
||||
if [ "$FAIL" -eq 0 ]; then
|
||||
echo "touched-requirements test-attribution tests: PASS (${PASS} assertions)"; exit 0
|
||||
fi
|
||||
echo "touched-requirements test-attribution tests: FAIL (${FAIL} failed, ${PASS} passed)" >&2
|
||||
exit 1
|
||||
@@ -1,233 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/tests/touched-requirements.test.sh — regression tests for the content scan in
|
||||
# ci/touched-requirements.sh, focused on rule 3: R-IDs cited only in a whole-line
|
||||
# YAML/shell `#` comment are PROSE (a reference), not a declaration, and must NOT be
|
||||
# attributed — while R-IDs on real code/spec lines still ARE.
|
||||
#
|
||||
# The tests point the real script at a THROWAWAY git repo via the REPO_ROOT override
|
||||
# (ci/lib.sh honours a pre-set REPO_ROOT); the script itself is run from its true
|
||||
# location so it still sources the real lib.sh. No network, no cluster.
|
||||
#
|
||||
# Run: ci/tests/touched-requirements.test.sh (or `make -C ci test`)
|
||||
set -euo pipefail
|
||||
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
SCRIPT="${CI_DIR}/touched-requirements.sh"
|
||||
|
||||
PASS=0 FAIL=0
|
||||
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
|
||||
bad() { printf ' FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
|
||||
|
||||
# assert_absent <needle> <haystack> <msg>
|
||||
assert_absent() {
|
||||
if grep -qxF "$1" <<<"$2"; then bad "$3 (found unexpected '$1' in: $(tr '\n' ' ' <<<"$2"))";
|
||||
else ok "$3"; fi
|
||||
}
|
||||
# assert_present <needle> <haystack> <msg>
|
||||
assert_present() {
|
||||
if grep -qxF "$1" <<<"$2"; then ok "$3";
|
||||
else bad "$3 (missing '$1' from: $(tr '\n' ' ' <<<"$2"))"; fi
|
||||
}
|
||||
|
||||
TMP="$(mktemp -d)"
|
||||
trap 'rm -rf "$TMP"' EXIT
|
||||
|
||||
git -C "$TMP" init -q -b main
|
||||
git -C "$TMP" config user.email [email protected]
|
||||
git -C "$TMP" config user.name t
|
||||
git -C "$TMP" config commit.gpgsign false # throwaway repo: never sign (ignore host gpg config)
|
||||
mkdir -p "$TMP/notes" # a path OUTSIDE path_requirements() so we isolate the content scan
|
||||
echo "baseline" > "$TMP/notes/base.txt"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm baseline
|
||||
|
||||
git -C "$TMP" checkout -q -b feature
|
||||
|
||||
# A YAML file: an R-ID in a whole-line comment (must be IGNORED) and an R-ID on a real
|
||||
# value line (must be DETECTED). Also an indented comment, to prove leading space is fine.
|
||||
cat > "$TMP/notes/example.yaml" <<'YAML'
|
||||
# This header explains R-DOC-01 and must NOT be attributed.
|
||||
kind: Thing
|
||||
metadata:
|
||||
name: implements-R-REAL-01
|
||||
spec:
|
||||
note: value # trailing inline comment mentioning R-INLINE-01 (conservatively still seen)
|
||||
# deeply indented comment about R-DOC-02 — also NOT attributed
|
||||
YAML
|
||||
|
||||
# A shell file: a `#` comment R-ID (IGNORED) plus a real command line R-ID (DETECTED).
|
||||
cat > "$TMP/notes/example.sh" <<'SH'
|
||||
#!/usr/bin/env bash
|
||||
# doc: this script relates to R-SHDOC-01 (comment only — ignored)
|
||||
echo "runs check for R-SHREAL-01"
|
||||
SH
|
||||
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm feature
|
||||
|
||||
OUT="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set:\n%s\n---\n' "$OUT"
|
||||
|
||||
# Comment-only citations are dropped (rule 3):
|
||||
assert_absent "R-DOC-01" "$OUT" "whole-line YAML comment R-ID is NOT attributed"
|
||||
assert_absent "R-DOC-02" "$OUT" "indented YAML comment R-ID is NOT attributed"
|
||||
assert_absent "R-SHDOC-01" "$OUT" "shell '#' comment R-ID is NOT attributed"
|
||||
# Real code/spec lines still count:
|
||||
assert_present "R-REAL-01" "$OUT" "R-ID on a real YAML value line IS attributed"
|
||||
assert_present "R-SHREAL-01" "$OUT" "R-ID on a real shell command line IS attributed"
|
||||
# Conservative: a trailing inline comment on a REAL line is still seen (only whole-line
|
||||
# comments are dropped) — documents the intentional boundary of the fix.
|
||||
assert_present "R-INLINE-01" "$OUT" "inline comment on a real line is (conservatively) still seen"
|
||||
|
||||
# Removed comment lines are also excluded, while removed real lines still count. The file
|
||||
# must EXIST at the base, so we diff the removal against the commit that added it.
|
||||
git -C "$TMP" checkout -q -b feature2 main
|
||||
cat > "$TMP/notes/todelete.yaml" <<'YAML'
|
||||
# will delete a comment citing R-DELDOC-01 and a real line citing R-DELREAL-01
|
||||
kind: Thing
|
||||
name: implements-R-DELREAL-01
|
||||
YAML
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm add
|
||||
ADD_SHA="$(git -C "$TMP" rev-parse HEAD)"
|
||||
git -C "$TMP" rm -q "$TMP/notes/todelete.yaml" && git -C "$TMP" commit -qm remove
|
||||
OUT2="$(REPO_ROOT="$TMP" BASE_REF="$ADD_SHA" "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (removal):\n%s\n---\n' "$OUT2"
|
||||
assert_absent "R-DELDOC-01" "$OUT2" "REMOVED whole-line comment R-ID is NOT attributed"
|
||||
assert_present "R-DELREAL-01" "$OUT2" "REMOVED real-line R-ID IS attributed"
|
||||
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# PATH MAP — bootstrap/* (added 2026-09-13)
|
||||
#
|
||||
# WHY THIS IS LOCKED DOWN. path_requirements() keys `bootstrap/*` to R-STATE-03 +
|
||||
# R-GITOPS-01, and until 2026-09-13 the comment justifying it described
|
||||
# `bootstrap/standup.sh` as a file that carries the placeholder guards. That file has never
|
||||
# been on `main` (it landed only on unmerged prod-standup branches), so the comment was
|
||||
# stale and the entry looked like dead weight someone would tidy away. It is not dead: it
|
||||
# is what SELECTS conformance/tests/l0-static/placeholder-guard-coupling for the change
|
||||
# that eventually lands a placeholder guard under bootstrap/ — and that test discovers its
|
||||
# guard by content rather than by path, so it activates on exactly that change. Removing
|
||||
# the entry would let the change introducing a guard be gated by nothing, which is the
|
||||
# 2026-09-05 class. These assertions make the entry's deletion a test failure rather than a
|
||||
# silent loss of selection.
|
||||
#
|
||||
# NOTE the R-ID content scan must not be what produces these: `notes/` is outside the path
|
||||
# map precisely so the other cases isolate the scan, and here the file content cites no
|
||||
# R-ID at all, so any R-ID in the output came from the PATH.
|
||||
git -C "$TMP" checkout -q -b feature3 main
|
||||
mkdir -p "$TMP/bootstrap/flux"
|
||||
cat > "$TMP/bootstrap/flux/bootstrap.sh" <<'SH'
|
||||
#!/usr/bin/env bash
|
||||
# a bootstrap-path change that cites no requirement id of its own
|
||||
echo "flux bootstrap"
|
||||
SH
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch bootstrap/"
|
||||
OUT3="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (bootstrap/ path map):\n%s\n---\n' "$OUT3"
|
||||
assert_present "R-STATE-03" "$OUT3" "bootstrap/* attributes R-STATE-03 (selects placeholder-guard-coupling)"
|
||||
assert_present "R-GITOPS-01" "$OUT3" "bootstrap/* attributes R-GITOPS-01 (it IS the GitOps bootstrap path)"
|
||||
|
||||
# And the neighbouring entry the same test leans on for its other half: clusters/* must
|
||||
# attribute R-ENV-02, which is what selects placeholder-guard-coupling's HALF A (the
|
||||
# authored-values sweep over clusters/<env> effective cluster-vars).
|
||||
git -C "$TMP" checkout -q -b feature4 main
|
||||
mkdir -p "$TMP/clusters/base"
|
||||
printf 'kind: ConfigMap\nmetadata:\n name: cluster-vars\n' > "$TMP/clusters/base/cluster-vars.yaml"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch clusters/"
|
||||
OUT4="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (clusters/ path map):\n%s\n---\n' "$OUT4"
|
||||
assert_present "R-ENV-02" "$OUT4" "clusters/* attributes R-ENV-02 (selects placeholder-guard-coupling HALF A)"
|
||||
assert_present "R-GITOPS-01" "$OUT4" "clusters/* attributes R-GITOPS-01"
|
||||
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# PATH MAP — apps/*, the C2 UI, and the platform assembly (added 2026-09-19)
|
||||
#
|
||||
# These arms closed the largest remaining fail-open hole in the map. Before them, 49 tracked
|
||||
# app manifests (apps/wave-0, -1, -2, apps/vms) plus platform/homepage, platform/headlamp,
|
||||
# platform/gateway, platform/kubevirt, platform/crafty, platform/coredns, the snapshot/backup
|
||||
# machinery and platform/kustomization.yaml all attributed NOTHING — so a change to any of them
|
||||
# selected only the whole-repo-scoped checks, and every requirement-scoped L2 behaviour test
|
||||
# (default-deny, egress class, auth mode, C2 surface) was filtered out of the merge gate.
|
||||
#
|
||||
# The C2 case was already documented as a known gap in changed_diff()'s rule-3 comment: the ONLY
|
||||
# route to R-C2-01/02 was rule 3, i.e. editing the C2 test itself.
|
||||
#
|
||||
# As with bootstrap/ above, each fixture cites NO R-ID of its own, so any R-ID in the output came
|
||||
# from the PATH and not from the content scan.
|
||||
|
||||
# apps/vms/* must win over apps/* — `case` takes the FIRST match, so arm ORDER is the property
|
||||
# under test here, not just the presence of both arms.
|
||||
git -C "$TMP" checkout -q -b feature-apps-vms main
|
||||
mkdir -p "$TMP/apps/vms/demo"
|
||||
printf 'kind: VirtualMachine\nmetadata:\n name: demo\n' > "$TMP/apps/vms/demo/vm.yaml"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/vms/"
|
||||
OUT_VMS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (apps/vms/ path map):\n%s\n---\n' "$OUT_VMS"
|
||||
assert_present "R-PLAT-03" "$OUT_VMS" "apps/vms/* attributes R-PLAT-03 (KubeVirt VM lifecycle)"
|
||||
assert_present "R-STATE-01" "$OUT_VMS" "apps/vms/* attributes R-STATE-01 (VM root/data volumes)"
|
||||
assert_absent "R-PLAT-02" "$OUT_VMS" "apps/vms/* does NOT fall through to the apps/* container arm (order holds)"
|
||||
|
||||
# A plain ServiceDefinition under apps/ gets the container-workload set.
|
||||
git -C "$TMP" checkout -q -b feature-apps main
|
||||
mkdir -p "$TMP/apps/wave-1"
|
||||
printf 'kind: ServiceDefinition\nmetadata:\n name: demo\n' > "$TMP/apps/wave-1/demo.yaml"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch apps/wave-1/"
|
||||
OUT_APPS="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (apps/ path map):\n%s\n---\n' "$OUT_APPS"
|
||||
assert_present "R-PLAT-02" "$OUT_APPS" "apps/* attributes R-PLAT-02 (container workload render)"
|
||||
assert_present "R-NET-01" "$OUT_APPS" "apps/* attributes R-NET-01 (default-deny applies to every app)"
|
||||
assert_present "R-NET-06" "$OUT_APPS" "apps/* attributes the egress-class requirements"
|
||||
assert_present "R-IDP-02" "$OUT_APPS" "apps/* attributes the auth-mode requirements"
|
||||
|
||||
# homepage / headlamp ARE the command-centre surface.
|
||||
git -C "$TMP" checkout -q -b feature-c2 main
|
||||
mkdir -p "$TMP/platform/headlamp"
|
||||
printf 'kind: Deployment\nmetadata:\n name: headlamp\n' > "$TMP/platform/headlamp/deployment.yaml"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/headlamp/"
|
||||
OUT_C2="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (C2 path map):\n%s\n---\n' "$OUT_C2"
|
||||
assert_present "R-C2-01" "$OUT_C2" "platform/headlamp/* attributes R-C2-01 (was reachable only by editing the C2 test)"
|
||||
assert_present "R-C2-02" "$OUT_C2" "platform/headlamp/* attributes R-C2-02"
|
||||
|
||||
# platform/kustomization.yaml is the assembly: one deleted line disables a whole component.
|
||||
git -C "$TMP" checkout -q -b feature-assembly main
|
||||
mkdir -p "$TMP/platform"
|
||||
printf 'resources:\n - cilium\n - longhorn\n' > "$TMP/platform/kustomization.yaml"
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "touch platform/kustomization.yaml"
|
||||
OUT_ASM="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (platform assembly):\n%s\n---\n' "$OUT_ASM"
|
||||
assert_present "R-GITOPS-01" "$OUT_ASM" "platform/kustomization.yaml attributes R-GITOPS-01 (component assembly)"
|
||||
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# THE ID PATTERN MUST ALLOW A DIGIT IN THE WORKSTREAM SEGMENT (added 2026-09-13)
|
||||
#
|
||||
# The content scan's pattern was `R-[A-Z]+-[0-9]+`, which CANNOT match `R-C2-01` or
|
||||
# `R-C2-02` — `[A-Z]+` consumes the `C` and then demands a `-` where the `2` is. Those are
|
||||
# the only two registry IDs whose workstream segment carries a digit, and the scan was blind
|
||||
# to BOTH, in the fail-OPEN direction: a change self-declaring a C2 requirement contributed
|
||||
# it to nobody, so the C2 conformance tests were filtered out of the merge gate. Found while
|
||||
# wiring conformance/tests/l0-static/c2-root-surface, which closes those two coverage gaps.
|
||||
#
|
||||
# `notes/` is outside path_requirements() on purpose, so the IDs below can only have come
|
||||
# from the CONTENT SCAN. A neighbouring all-letter ID is asserted too, so a regex that
|
||||
# somehow matched only the digit form would still fail here.
|
||||
git -C "$TMP" checkout -q -b feature5 main
|
||||
cat > "$TMP/notes/c2.yaml" <<'YAML'
|
||||
kind: Test
|
||||
metadata:
|
||||
labels:
|
||||
conformance.home.k3s/R-C2-01: "true"
|
||||
conformance.home.k3s/R-C2-02: "true"
|
||||
conformance.home.k3s/R-OBS-04: "true"
|
||||
YAML
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "declare R-C2-01 / R-C2-02"
|
||||
OUT5="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (digit in the workstream segment):\n%s\n---\n' "$OUT5"
|
||||
assert_present "R-C2-01" "$OUT5" "an R-ID with a DIGIT in the workstream segment IS attributed (R-C2-01)"
|
||||
assert_present "R-C2-02" "$OUT5" "the second such ID is attributed too (R-C2-02)"
|
||||
assert_present "R-OBS-04" "$OUT5" "an all-letter workstream segment still works (control)"
|
||||
# The pattern must not become so loose that a non-ID token is read as a requirement: a bare
|
||||
# numeric workstream is not a registry shape, and attributing one would WIDEN the gate on
|
||||
# noise (over-broad selection, never a false green, but noisy and misleading).
|
||||
assert_absent "R-1-01" "$OUT5" "a purely numeric workstream segment is not invented from thin air"
|
||||
|
||||
echo
|
||||
if [ "$FAIL" -eq 0 ]; then echo "touched-requirements tests: PASS ($PASS assertions)"; exit 0;
|
||||
else echo "touched-requirements tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
|
||||
+23
-334
@@ -21,110 +21,35 @@
|
||||
# the R-IDs that policies/ServiceDefinitions self-declare (docs/14 + ADR-0011).
|
||||
# Untouched lines and prose are NOT scanned; see changed_diff() for why.
|
||||
# 2. A coarse path->requirement fallback for changes that don't self-declare IDs.
|
||||
# 3. For a change anywhere under `conformance/tests/<test>/`: THAT TEST'S OWN requirement
|
||||
# labels, read from its chainsaw-test.yaml. See test_requirements_for_path().
|
||||
# The coarse map (2) NEVER overrides an explicit tag; it only widens coverage for
|
||||
# untagged changes so nothing silently escapes the gate. (3) is the same principle applied
|
||||
# to the tests themselves: editing or weakening a test must be gated by the requirements
|
||||
# that test CERTIFIES, not merely by R-TEST-0x.
|
||||
# untagged changes so nothing silently escapes the gate.
|
||||
set -euo pipefail
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "${CI_DIR}/lib.sh"
|
||||
|
||||
# ---------------------------------------------------------------------------------
|
||||
# BASE-REF RESOLUTION — and why it now REFUSES rather than guesses.
|
||||
#
|
||||
# BUG THIS FIXES (local/CI divergence, 2026-08-31): every failure to establish a
|
||||
# reliable diff base used to degrade SILENTLY to an EMPTY touched set, which main()
|
||||
# reported as "no changed files vs base" and exit 0. An empty set means the coverage
|
||||
# `--require` gate is never applied, so the merge gate printed "MERGE GATE: PASS"
|
||||
# having checked NOTHING — while CI, with a correct base, computed 17 touched
|
||||
# requirements and blocked. A false green is the worst outcome this project can
|
||||
# produce, so every one of those degradations is now a LOUD exit 3:
|
||||
#
|
||||
# * HEAD~1 last resort REMOVED. It silently narrowed the diff to the last commit
|
||||
# (and on a one-commit repo failed into an empty set).
|
||||
# * two-dot fallback REMOVED. It masked a failing three-dot diff as "no changes".
|
||||
# * unresolvable base exit 3 (was: HEAD~1 or empty).
|
||||
# * diff command fails exit 3 (was: empty set).
|
||||
# * HEAD not ahead of exit 3 (was: exit 0 with an empty set — this is the case
|
||||
# the base that produced the false green: running the gate while
|
||||
# sitting on `main`, or on a branch already merged).
|
||||
#
|
||||
# Exit 3 is the ci/README.md "SKIPPED / not-run" code: NOT a pass, NOT a fail. Callers
|
||||
# (ci/gate.sh) must surface it as an UNDECIDED merge decision. An honest, noisy skip
|
||||
# beats a quiet false pass.
|
||||
#
|
||||
# Order of preference: BASE_REF (explicit) > GITHUB_BASE_REF (Actions PR runs, target
|
||||
# branch) > origin/main > main. Each candidate is VERIFIED to resolve to a commit
|
||||
# before it is used, so a shallow checkout that lacks origin/main is reported, not
|
||||
# papered over.
|
||||
# Base ref to diff against. In Gitea/GitHub Actions PR runs this is the target
|
||||
# branch (main). Locally, default to main; override with BASE_REF.
|
||||
: "${BASE_REF:=}"
|
||||
BASE="" # resolved base ref
|
||||
BASE_SOURCE="" # where it came from (for the log line)
|
||||
BASE_PROBLEM="" # why resolution failed, when it did
|
||||
|
||||
rev() { git -C "$REPO_ROOT" rev-parse --verify -q "$1^{commit}" 2>/dev/null; }
|
||||
|
||||
# Loud, structured refusal. Prints to stderr only; stdout stays empty so a caller that
|
||||
# captures stdout gets NO requirement list rather than a silently-empty one.
|
||||
#
|
||||
# $1 is a machine-readable REASON CODE, also written to $TOUCHED_REASON_FILE when that
|
||||
# env var names a path. ci/gate.sh uses it to tell the two cases apart:
|
||||
# on-base HEAD is the base branch (or already merged) — nothing to gate. Legitimate
|
||||
# (a push-to-main CI run lands here), so the gate reports NOT APPLICABLE
|
||||
# rather than PASS, and does not fail the job.
|
||||
# no-base no reliable base ref exists in this checkout — the ENVIRONMENT is broken,
|
||||
# diff-failed or git could not produce the diff. A merge decision is impossible, so the
|
||||
# gate BLOCKS rather than guessing.
|
||||
undecidable() {
|
||||
local code="$1"; shift
|
||||
if [ -n "${TOUCHED_REASON_FILE:-}" ]; then printf '%s\n' "$code" > "$TOUCHED_REASON_FILE"; fi
|
||||
err "TOUCHED-REQUIREMENT SET NOT DETERMINABLE (${code})"
|
||||
err " reason: $*"
|
||||
err " consequence: the R-TEST-03 coverage gate CANNOT be applied to this checkout."
|
||||
err " This is exit 3 (SKIPPED / not-run), never a pass — see ci/README.md."
|
||||
err " Fix one of:"
|
||||
err " * check out the feature branch you mean to gate (not the base branch), or"
|
||||
err " * git fetch origin main (so origin/main exists and is current), or"
|
||||
err " * BASE_REF=<ref> make -C ci gate to name the base explicitly."
|
||||
exit 3
|
||||
}
|
||||
|
||||
resolve_base() {
|
||||
local cand
|
||||
if [ -n "$BASE_REF" ]; then
|
||||
if rev "$BASE_REF" >/dev/null; then BASE="$BASE_REF"; BASE_SOURCE="BASE_REF"; return 0; fi
|
||||
BASE_PROBLEM="BASE_REF='${BASE_REF}' does not resolve to a commit in ${REPO_ROOT}"
|
||||
return 1
|
||||
fi
|
||||
if [ -n "${GITHUB_BASE_REF:-}" ]; then
|
||||
for cand in "origin/${GITHUB_BASE_REF}" "${GITHUB_BASE_REF}"; do
|
||||
if rev "$cand" >/dev/null; then BASE="$cand"; BASE_SOURCE="GITHUB_BASE_REF"; return 0; fi
|
||||
done
|
||||
BASE_PROBLEM="GITHUB_BASE_REF='${GITHUB_BASE_REF}' is set, but neither 'origin/${GITHUB_BASE_REF}' nor '${GITHUB_BASE_REF}' exists here (shallow checkout? the workflow needs fetch-depth: 0)"
|
||||
return 1
|
||||
fi
|
||||
for cand in origin/main main; do
|
||||
if rev "$cand" >/dev/null; then BASE="$cand"; BASE_SOURCE="default"; return 0; fi
|
||||
done
|
||||
BASE_PROBLEM="no base ref resolves (tried origin/main, main) in ${REPO_ROOT}"
|
||||
return 1
|
||||
if [ -n "$BASE_REF" ]; then echo "$BASE_REF"; return; fi
|
||||
if [ -n "${GITHUB_BASE_REF:-}" ]; then echo "origin/${GITHUB_BASE_REF}"; return; fi
|
||||
if git -C "$REPO_ROOT" rev-parse --verify -q origin/main >/dev/null; then echo "origin/main"; return; fi
|
||||
if git -C "$REPO_ROOT" rev-parse --verify -q main >/dev/null; then echo "main"; return; fi
|
||||
echo "HEAD~1" # last resort
|
||||
}
|
||||
|
||||
# Both diff helpers assume BASE is already resolved and validated by main(). They return
|
||||
# git's exit status so the CALLER can distinguish "no changes" from "the diff failed" —
|
||||
# conflating those two is exactly what produced the false green.
|
||||
changed_files() {
|
||||
local base; base="$(resolve_base)"
|
||||
# Three-dot: changes on this branch since it diverged from base.
|
||||
git -C "$REPO_ROOT" diff --name-only "${BASE}...HEAD"
|
||||
if git -C "$REPO_ROOT" diff --name-only "${base}...HEAD" 2>/dev/null; then return; fi
|
||||
git -C "$REPO_ROOT" diff --name-only "${base}" 2>/dev/null || true
|
||||
}
|
||||
|
||||
# The ADDED/REMOVED lines of the change, for the R-ID content scan — excluding
|
||||
# documentation.
|
||||
#
|
||||
# THREE scoping rules, all learned from the gate blocking merges it should not have:
|
||||
# TWO scoping rules, both learned from the gate blocking merges it should not have:
|
||||
#
|
||||
# 1. Scan the DIFF, not whole file contents. Grepping whole files meant a change
|
||||
# inherited every requirement those files had EVER mentioned, on lines the author
|
||||
@@ -141,34 +66,14 @@ changed_files() {
|
||||
# that matter (conformance/, schemas/, catalog/, clusters/, ci/) are covered by
|
||||
# the path->requirement fallback regardless of file type.
|
||||
#
|
||||
# 3. Skip COMMENT-ONLY lines in non-Markdown files — rule 2's logic, applied to the
|
||||
# prose that lives INSIDE code files. A requirement ID cited in a YAML/shell `#`
|
||||
# comment is documentation, not a declaration, exactly like an ID in a `*.md` file.
|
||||
# Rule 2 drops whole Markdown files, but a `.gitea/workflows/*.yml` or `ci/*.sh`
|
||||
# HEADER that explains a requirement would otherwise re-attribute it. Concretely:
|
||||
# gate.yml's header cites R-TEST-06 (enforced by the spec-immutability STAGE, not by
|
||||
# any Chainsaw test, so a known-uncovered ID); before this rule, editing that comment
|
||||
# re-attributed R-TEST-06 and the coverage `--require` check false-blocked the merge —
|
||||
# the same self-referential trap rule 2 fixed for `*.md`. Applied in main()'s content
|
||||
# scan (see there): after the diff marker is stripped, a line whose first non-blank
|
||||
# character is `#` is dropped BEFORE the R-ID grep. Deliberately conservative — only
|
||||
# WHOLE-LINE comments are dropped; an R-ID on a real code/spec line is still detected
|
||||
# (authoritative declarations — test tags, policies, ServiceDefinitions — are never a
|
||||
# bare comment), so this only narrows PROSE, never a declaration.
|
||||
#
|
||||
# Removed lines count: deleting a test's requirement tag, or a policy's self-declared
|
||||
# ID, touches that requirement just as much as adding one.
|
||||
# `:(exclude)*.md` drops Markdown from the content scan (rule 2 above).
|
||||
# `:(exclude)ci/tests/` drops this dir's OWN unit-test fixtures: those files embed example
|
||||
# R-IDs as DATA (asserting the scan's behaviour), never as declarations — the same
|
||||
# "reference, not declaration" logic as rule 2. Their gate coverage is unaffected: any
|
||||
# ci/ change (ci/tests/ included) still attributes R-GITOPS-01/R-TEST-02/R-TEST-03 via the
|
||||
# path->requirement map below. NOTE: this excludes ci/tests/ only — conformance/tests/
|
||||
# carry AUTHORITATIVE `conformance.home.k3s/R-*` labels and are deliberately still scanned.
|
||||
# Three-dot to match changed_files(): changes on this branch since it diverged from base.
|
||||
# `:(exclude)*.md` drops Markdown from the content scan (rule 2 above). Three-dot to
|
||||
# match changed_files(): changes on this branch since it diverged from base.
|
||||
changed_diff() {
|
||||
local -a excl=(':(exclude)*.md' ':(exclude)ci/tests/')
|
||||
git -C "$REPO_ROOT" diff -U0 "${BASE}...HEAD" -- . "${excl[@]}"
|
||||
local base; base="$(resolve_base)"
|
||||
if git -C "$REPO_ROOT" diff -U0 "${base}...HEAD" -- . ':(exclude)*.md' 2>/dev/null; then return; fi
|
||||
git -C "$REPO_ROOT" diff -U0 "${base}" -- . ':(exclude)*.md' 2>/dev/null || true
|
||||
}
|
||||
|
||||
# Coarse path -> requirement map. Deliberately conservative; extend as subtrees
|
||||
@@ -177,23 +82,6 @@ path_requirements() {
|
||||
case "$1" in
|
||||
catalog/*) echo "R-GITOPS-02" ;;
|
||||
schemas/*) echo "R-GITOPS-02" ;;
|
||||
# apps/* was UNMAPPED until 2026-09-19, and it is the largest such hole in the map: 49 tracked
|
||||
# manifests (apps/wave-0, -1, -2, apps/vms) that describe every workload on the cluster. A change
|
||||
# adding or editing an app attributed NOTHING, so the merge gate ran only the whole-repo-scoped
|
||||
# checks and none of the requirement-scoped L2 behaviour tests — default-deny enforcement, egress
|
||||
# class enforcement, auth-mode enforcement — which are exactly what a new or edited app can
|
||||
# violate. Kyverno admission and the L0 egress-path check are real backstops, but "admission will
|
||||
# catch it at apply time" is post-merge, and the gate exists to catch it before.
|
||||
#
|
||||
# ORDER MATTERS: `case` takes the FIRST match, so apps/vms/* must precede apps/*.
|
||||
# apps/vms/ are KubeVirt VMs (R-PLAT-03 lifecycle) with root/data volumes (R-STATE-01), still
|
||||
# subject to default-deny (R-NET-01).
|
||||
apps/vms/*) echo "R-PLAT-03 R-NET-01 R-STATE-01" ;;
|
||||
# Everything else under apps/ is a ServiceDefinition rendered by catalog/render.py: a container
|
||||
# workload (R-PLAT-02) rendered from the catalog (R-GITOPS-02), carrying the egress-class
|
||||
# (R-NET-05/06/07) and auth-mode (R-IDP-02/R-IDP-06) labels that admission requires, under
|
||||
# default-deny (R-NET-01).
|
||||
apps/*) echo "R-PLAT-02 R-GITOPS-02 R-NET-01 R-NET-05 R-NET-06 R-NET-07 R-IDP-02 R-IDP-06" ;;
|
||||
platform/kyverno/*) echo "R-TEST-01 R-NET-01 R-SEC-01" ;;
|
||||
platform/cilium/*) echo "R-NET-01 R-NET-02 R-NET-05" ;;
|
||||
platform/longhorn/*) echo "R-STATE-01 R-STATE-02" ;;
|
||||
@@ -201,229 +89,30 @@ path_requirements() {
|
||||
platform/externaldns/*) echo "R-DNS-02" ;;
|
||||
platform/velero/*) echo "R-STATE-03" ;;
|
||||
platform/monitoring/*) echo "R-OBS-01 R-OBS-02" ;;
|
||||
# platform/cnpg was UNMAPPED until 2026-09-06. It is the PostgreSQL DR path for keycloak-db /
|
||||
# nextcloud-db / semaphore-db (barmanObjectStore WAL archiving + ScheduledBackup), so a change
|
||||
# there is an R-IDP-05 / R-STATE-03 change; it now also carries the platform-health alert rules
|
||||
# for that path (R-OBS-04). Without this line an edit under platform/cnpg/ contributed ZERO
|
||||
# requirements unless it happened to retype an R-ID on a non-comment line, and escaped the gate.
|
||||
platform/cnpg/*) echo "R-IDP-05 R-STATE-03 R-OBS-04" ;;
|
||||
# vpn-egress was UNMAPPED until 2026-09-06. platform/vpn-egress/egress-steering.yaml IS the
|
||||
# R-NET-04 datapath (the rule that selects between the two egress paths) and the R-NET-05/06/07
|
||||
# steering policy — it even self-declares those IDs in its `homelab.keithjackson.dev/requirements`
|
||||
# annotation. But the content scan only reads ADDED/REMOVED lines, so any edit that does not
|
||||
# happen to retype an R-ID (a selector change, an excludedCIDRs change, a comment) contributed
|
||||
# ZERO requirements and escaped the gate entirely. Mapped explicitly; the annotation still wins
|
||||
# when a diff does carry the tags.
|
||||
platform/vpn-egress/*) echo "R-NET-04 R-NET-05 R-NET-06 R-NET-07" ;;
|
||||
# Keycloak is the whole SSO surface: the realm import (declarative identity config),
|
||||
# the SOPS client secrets, and the reconciler that heals them. Added 2026-09-06 after
|
||||
# the client-secret incident, in which editing platform/keycloak/ attributed NO
|
||||
# requirement, so the L2 keycloak tests were filtered out of the merge gate's touched
|
||||
# subset — the change that could break every OIDC login was gated by nothing.
|
||||
# R-IDP-05 is in scope because platform/keycloak/ also carries the CNPG DB Cluster and
|
||||
# its off-cluster backup credential, not only the realm.
|
||||
platform/keycloak/*) echo "R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06" ;;
|
||||
platform/oauth2-proxy/*) echo "R-IDP-02 R-IDP-06" ;;
|
||||
platform/image-automation/*) echo "R-UPD-01 R-UPD-02" ;;
|
||||
# The C2 UI was UNMAPPED until 2026-09-19. This is the gap named explicitly in changed_diff()'s
|
||||
# rule-3 comment above: "path_requirements() has no arm for platform/homepage/* or
|
||||
# platform/headlamp/* either, so the ONLY route to R-C2-01/02 was rule 3 (editing the C2 test
|
||||
# itself)". homepage + headlamp ARE the command-centre surface (registry: R-C2-01/02 ->
|
||||
# docs/12-command-center-ui.md), so editing them must select the C2 tests.
|
||||
platform/homepage/*|platform/headlamp/*) echo "R-C2-01 R-C2-02" ;;
|
||||
# KubeVirt is the VM platform itself (R-PLAT-03); its virt-* components are also scrape targets.
|
||||
platform/kubevirt/*) echo "R-PLAT-03 R-OBS-01" ;;
|
||||
# The shared ingress Gateway: the L2-announced VIP every HTTP service answers on (R-DNS-01), the
|
||||
# TLS listeners and their certificates (R-TLS-01/02), under default-deny (R-NET-01).
|
||||
platform/gateway/*) echo "R-DNS-01 R-TLS-01 R-TLS-02 R-NET-01" ;;
|
||||
# CoreDNS is cluster DNS resolution (R-DNS-01).
|
||||
platform/coredns/*) echo "R-DNS-01" ;;
|
||||
# crafty is a first-class app living under platform/: a container workload with a Longhorn PVC and
|
||||
# its OWN L4 LoadBalancer VIP (Minecraft 25565 is not HTTP, so it does not use the Gateway).
|
||||
platform/crafty/*) echo "R-PLAT-02 R-NET-01 R-DNS-01 R-STATE-01" ;;
|
||||
# Snapshot machinery is the storage/DR path: CSI snapshots (R-STATE-01) feeding backup (R-STATE-03).
|
||||
platform/external-snapshotter/*|platform/backup/*) echo "R-STATE-01 R-STATE-03" ;;
|
||||
# runbooks/ IS the R-OBS-04 surface: alert-runbooks-resolvable asserts every alert's runbook_url
|
||||
# resolves to a served page, and runbooks/serving/ is the Deployment that serves them. That check
|
||||
# is whole-repo-scoped so it runs regardless, but the attribution should still be honest — a
|
||||
# runbook change is an observability change.
|
||||
runbooks/*) echo "R-OBS-04" ;;
|
||||
# platform-config/ is the env-specific configuration Flux substitutes into and applies.
|
||||
platform-config/*|platform-config-cdi/*) echo "R-ENV-02 R-GITOPS-01" ;;
|
||||
# platform/kustomization.yaml is the ASSEMBLY: the list of platform components Flux applies.
|
||||
# Deleting one line there disables an entire component (cilium, longhorn, velero...) while every
|
||||
# file it points at stays perfectly valid on disk, so nothing else in the map would attribute the
|
||||
# change. That is the most consequential single-line edit in the repo and it attributed nothing.
|
||||
platform/kustomization.yaml) echo "R-GITOPS-01" ;;
|
||||
# .sops.yaml holds the encryption rules (which paths must be encrypted, with which keys). A change
|
||||
# here can silently stop a new secret path from being encrypted, which is R-SEC-01's whole point.
|
||||
.sops.yaml) echo "R-SEC-01 R-GITOPS-01" ;;
|
||||
conformance/*) echo "R-TEST-01 R-TEST-02 R-TEST-03 R-TEST-04" ;;
|
||||
# infra/ was UNMAPPED until 2026-09-12 — the OpenTofu + Ansible that provision the dev
|
||||
# nodes. conformance/tests/l0-static/dev-nodes-provisioned-as-code reads NOTHING but
|
||||
# infra/, yet an infra/ edit attributed ZERO requirements, so the one check that asserts
|
||||
# the nodes are provisioned as code was filtered out of every change that could break it.
|
||||
# That test's invariant IS path-scoped (it is deliberately NOT whole-repo-scoped; see its
|
||||
# chainsaw-test.yaml), and the correct fix for a path-scoped invariant is a path-map entry.
|
||||
infra/*) echo "R-ENV-01" ;;
|
||||
clusters/*) echo "R-GITOPS-01 R-ENV-02" ;;
|
||||
# bootstrap/ was UNMAPPED until 2026-09-06. The entry STAYS, but its original rationale
|
||||
# was written against a file that is not here, and that is corrected on 2026-09-13:
|
||||
# it claimed "bootstrap/standup.sh carries the placeholder guards that decide whether
|
||||
# `velero` may be enabled against a real backup endpoint". bootstrap/standup.sh HAS NEVER
|
||||
# BEEN ON `main` — it was not deleted (no deletion commit exists anywhere), it landed only
|
||||
# on unmerged prod-standup branches. `git ls-tree main bootstrap/` lists README.md, flux/,
|
||||
# gitea/, sops/. So R-STATE-03 here is ANTICIPATORY, not descriptive.
|
||||
#
|
||||
# WHY IT IS STILL RIGHT. conformance/tests/l0-static/placeholder-guard-coupling DISCOVERS
|
||||
# its guard (any tracked *.sh defining `<name>_is_placeholder()`) instead of hardcoding a
|
||||
# path, so the moment a standup guard lands under bootstrap/ that test's coupling half
|
||||
# activates — and this entry is what selects the test for the change that lands it.
|
||||
# Deleting the entry would mean the change introducing a placeholder guard is gated by
|
||||
# nothing, which is the 2026-09-05 class (a prefix glob that stopped matching when the
|
||||
# value it guards changed, in a different file) left to reintroduce itself. R-GITOPS-01 is
|
||||
# descriptive today: bootstrap/flux/, gitea/ and sops/ ARE the GitOps bootstrap path.
|
||||
# Locked down by ci/tests/touched-requirements.test.sh's path-map section.
|
||||
bootstrap/*) echo "R-STATE-03 R-GITOPS-01" ;;
|
||||
# R-TEST-06 added 2026-09-12. The comment in changed_diff()'s rule 3 above records why it
|
||||
# was ABSENT: R-TEST-06 used to be enforced by the spec-immutability STAGE and carried by no
|
||||
# Chainsaw test, so attributing it made the coverage `--require` check false-block. It is now
|
||||
# carried by conformance/tests/l0-static/spec-immutability-enforced — whose entire input
|
||||
# surface is ci/ — so a ci/ change must attribute it, or the test that proves the
|
||||
# spec-immutability mechanism is still load-bearing runs on nothing that can break it.
|
||||
.gitea/*|ci/*) echo "R-GITOPS-01 R-TEST-02 R-TEST-03 R-TEST-06" ;;
|
||||
.gitea/*|ci/*) echo "R-GITOPS-01 R-TEST-02 R-TEST-03" ;;
|
||||
*) : ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------------
|
||||
# A CHANGED TEST ATTRIBUTES THE REQUIREMENTS IT CERTIFIES (detection rule 3).
|
||||
#
|
||||
# THE GAP THIS CLOSES (2026-09-12). The path map keys `conformance/*` to
|
||||
# R-TEST-01..04 — the requirements about the HARNESS. It says nothing about what a given
|
||||
# test PROVES. So editing `conformance/tests/l2-behavioral/tenant-isolation/check.sh` to
|
||||
# weaken or delete an assertion attributed R-TEST-01..04 and nothing else: the merge gate
|
||||
# then ran the harness's self-tests and NOT the weakened test's own requirement subset.
|
||||
# Weakening a test is the one change that most needs the requirement it certifies to be
|
||||
# re-verified, and it was the change least likely to select it.
|
||||
#
|
||||
# The `conformance.home.k3s/R-*: "true"` labels on a test ARE the authoritative declaration of
|
||||
# what it verifies (lib/coverage.py reads the same labels); this reads them directly. Note the
|
||||
# content scan in main() only sees ADDED/REMOVED lines, so it catches a change to the LABEL
|
||||
# BLOCK but not a change to the test BODY — which is exactly the case that matters here.
|
||||
#
|
||||
# WHY IT READS HEAD *OR* THE BASE: a change that DELETES a test, or deletes its
|
||||
# chainsaw-test.yaml, must still attribute what that test used to certify — otherwise removing
|
||||
# the only verifier of a requirement is the cheapest way to make the gate stop checking it.
|
||||
# HEAD first (the post-change truth), base as the fallback (the deleted case). Fail-closed:
|
||||
# this function can only ADD requirements, never remove one.
|
||||
: "${CONF_TEST_LABEL_PREFIX:=conformance.home.k3s}" # == CONF_LABEL_PREFIX, conformance/lib/common.sh
|
||||
|
||||
file_text_head_or_base() {
|
||||
local path="$1"
|
||||
git -C "$REPO_ROOT" show "HEAD:${path}" 2>/dev/null && return 0
|
||||
git -C "$REPO_ROOT" show "${BASE}:${path}" 2>/dev/null && return 0
|
||||
return 0
|
||||
}
|
||||
|
||||
# test_requirements_for_path <changed-path> -> space-separated R-IDs (possibly empty).
|
||||
# Walks UP from the changed file to the nearest ancestor directory carrying a
|
||||
# chainsaw-test.yaml, so it works for any file in a test dir (check.sh, mutations.py,
|
||||
# fixtures, the chainsaw-test.yaml itself) and for nested test layouts.
|
||||
test_requirements_for_path() {
|
||||
local f="$1" dir yaml text
|
||||
case "$f" in conformance/tests/*/*) ;; *) return 0 ;; esac
|
||||
dir="${f%/*}"
|
||||
while :; do
|
||||
case "$dir" in
|
||||
conformance/tests|conformance|.|"") return 0 ;;
|
||||
*/*) ;;
|
||||
*) return 0 ;;
|
||||
esac
|
||||
yaml="${dir}/chainsaw-test.yaml"
|
||||
text="$(file_text_head_or_base "$yaml")"
|
||||
if [ -n "$text" ]; then
|
||||
# Whole-line `#` comments are dropped first — same "reference, not declaration" rule the
|
||||
# content scan applies (changed_diff() rule 3): a label shown inside a header example is
|
||||
# documentation. Only a real YAML mapping line counts.
|
||||
printf '%s\n' "$text" \
|
||||
| grep -vE '^[[:space:]]*#' \
|
||||
| sed -nE "s#^[[:space:]]*${CONF_TEST_LABEL_PREFIX}/(R-[A-Z0-9]+-[0-9]+):[[:space:]]*[\"']?true.*#\1#p" \
|
||||
| tr '\n' ' '
|
||||
return 0
|
||||
fi
|
||||
dir="${dir%/*}"
|
||||
done
|
||||
}
|
||||
|
||||
main() {
|
||||
local files reqs="" rc=0 base_sha head_sha local_main merge_base
|
||||
|
||||
resolve_base || undecidable no-base "$BASE_PROBLEM"
|
||||
base_sha="$(rev "$BASE")" || undecidable no-base "resolved base '${BASE}' vanished"
|
||||
head_sha="$(rev HEAD)" || undecidable no-base "HEAD does not resolve to a commit (empty repo?)"
|
||||
log "diff base: ${BASE} (${BASE_SOURCE}) @ ${base_sha:0:12} ... HEAD @ ${head_sha:0:12}"
|
||||
|
||||
# Advisory: a never-fetched origin/main silently WIDENS the set (over-broad, never a
|
||||
# false green) — surfaced so an unexpectedly large touched set has an explanation.
|
||||
if [ "$BASE" = "origin/main" ] && local_main="$(rev main)" && [ -n "$local_main" ] \
|
||||
&& [ "$local_main" != "$base_sha" ]; then
|
||||
warn "local 'main' (${local_main:0:12}) != origin/main (${base_sha:0:12}) — if origin/main"
|
||||
warn " was not fetched recently the touched set may be stale: git fetch origin main"
|
||||
fi
|
||||
|
||||
merge_base="$(git -C "$REPO_ROOT" merge-base "$BASE" HEAD 2>/dev/null)" \
|
||||
|| undecidable no-base "no merge base between '${BASE}' and HEAD (unrelated histories, or a shallow clone)"
|
||||
if [ "$merge_base" = "$head_sha" ]; then
|
||||
undecidable on-base "HEAD is not ahead of '${BASE}' (merge-base == HEAD): this checkout contains no branch commits to gate. The empty touched set would be empty BY DEFAULT, not by verification."
|
||||
fi
|
||||
|
||||
files="$(changed_files)" || rc=$?
|
||||
[ "$rc" -eq 0 ] || undecidable diff-failed "git diff --name-only ${BASE}...HEAD failed (rc=${rc})"
|
||||
if [ -z "$files" ]; then
|
||||
warn "HEAD is ahead of ${BASE} but the three-dot diff is empty (self-cancelling commits?)"
|
||||
warn " -> genuinely empty touched-requirement set; base WAS established."
|
||||
exit 0
|
||||
fi
|
||||
local files reqs=""
|
||||
files="$(changed_files)"
|
||||
[ -n "$files" ] || { log "no changed files vs base; empty touched-requirement set"; exit 0; }
|
||||
|
||||
while IFS= read -r f; do
|
||||
[ -n "$f" ] || continue
|
||||
reqs+=" $(path_requirements "$f")"
|
||||
# Detection rule 3: a change under conformance/tests/<test>/ attributes THAT TEST'S own
|
||||
# requirement labels, so weakening a test is gated by what the test certifies.
|
||||
reqs+=" $(test_requirements_for_path "$f")"
|
||||
done <<< "$files"
|
||||
|
||||
# Self-declared R-IDs on the ADDED/REMOVED lines of the change (see changed_diff).
|
||||
#
|
||||
# THE ID PATTERN IS `R-[A-Z0-9]+-[0-9]+`, NOT `R-[A-Z]+-[0-9]+`. The WORKSTREAM segment may
|
||||
# contain a DIGIT: `R-C2-01` and `R-C2-02` (command & control UI) are the only two such IDs
|
||||
# in conformance/requirements/registry.yaml today, and the stricter pattern silently failed
|
||||
# to match BOTH of them — `[A-Z]+` consumes the `C` and then demands a `-` where the `2` is.
|
||||
# Every other consumer in the repo already allows the digit: test_requirements_for_path()
|
||||
# twenty lines above uses `R-[A-Z0-9]+-[0-9]+`, as do conformance/lib/coverage.py,
|
||||
# conformance/lib/status.py and conformance/lib/run_l1_offline.py. This grep was the lone
|
||||
# dissenter, and it dissented by UNDER-attributing — the fail-OPEN direction. A change that
|
||||
# retyped a `conformance.home.k3s/R-C2-01` tag, or a manifest self-declaring R-C2-02 from a
|
||||
# path the map keys elsewhere, contributed those IDs to NOBODY, so the C2 tests were
|
||||
# filtered out of the merge gate. Same defect class as the map's `*) : ;;` arm: a miss that
|
||||
# reads as "nothing touched". path_requirements() has no arm for platform/homepage/* or
|
||||
# platform/headlamp/* either, so before this fix the ONLY route to R-C2-01/02 was rule 3
|
||||
# (editing the C2 test itself). Regression-tested in ci/tests/touched-requirements.test.sh.
|
||||
# `^[+-]` keeps hunk headers (`@@`) and the `+++`/`---` file headers out of scope.
|
||||
# `sed 's/^[+-]//'` then strips the single-char diff marker (safe: -U0 emits one), so
|
||||
# what remains is the SOURCE line, and `grep -vE '^[[:space:]]*#'` drops whole-line
|
||||
# YAML/shell comments (rule 3 in changed_diff's header) BEFORE the R-ID grep — a
|
||||
# requirement cited only in a `#` comment is prose, not a declaration.
|
||||
local diff_txt
|
||||
diff_txt="$(changed_diff)" || rc=$?
|
||||
[ "$rc" -eq 0 ] || undecidable diff-failed "git diff -U0 ${BASE}...HEAD failed (rc=${rc})"
|
||||
reqs+=" $(printf '%s\n' "$diff_txt" \
|
||||
reqs+=" $(changed_diff \
|
||||
| grep -E '^[+-]' \
|
||||
| grep -vE '^(\+\+\+|---)' \
|
||||
| sed -E 's/^[+-]//' \
|
||||
| grep -vE '^[[:space:]]*#' \
|
||||
| grep -hoE 'R-[A-Z0-9]+-[0-9]+' \
|
||||
| grep -hoE 'R-[A-Z]+-[0-9]+' \
|
||||
| tr '\n' ' ' || true)"
|
||||
|
||||
# Normalize: split, unique, sort. Emit to stdout.
|
||||
|
||||
+2
-11
@@ -9,7 +9,7 @@
|
||||
# whatever platform/kyverno and conformance/ pin — see integration notes in README.
|
||||
|
||||
# L0 policy gate (Kyverno CLI) + ServiceDefinition validation (kyverno-json)
|
||||
KYVERNO_VERSION="1.19.0"
|
||||
KYVERNO_VERSION="1.13.4"
|
||||
KYVERNO_JSON_VERSION="0.0.3"
|
||||
|
||||
# L1/L2/L3 declarative runner (Chainsaw) — invoked via conformance/ entrypoint,
|
||||
@@ -18,18 +18,9 @@ CHAINSAW_VERSION="0.2.12"
|
||||
|
||||
# Lint / manifest tooling
|
||||
YAMLLINT_VERSION="1.35.1"
|
||||
KUBECTL_VERSION="1.36.4"
|
||||
KUBECTL_VERSION="1.31.4"
|
||||
YQ_VERSION="4.44.6"
|
||||
|
||||
# Alert-rule unit tests (promtool test rules) — the L0 check
|
||||
# conformance/tests/l0-static/alert-rules-unit-tested runs every
|
||||
# platform/**/*.promtool-test.yaml through it. MUST stay in lockstep with
|
||||
# infra/devcontainer/versions.env (which bakes it into the image) AND with the Prometheus
|
||||
# version running on prod, because promtool IS the rule engine: verified 2026-09-12 as
|
||||
# v3.14.0-distroless via `kubectl get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'`.
|
||||
# Shipped inside the prometheus release tarball, not as a separate release.
|
||||
PROMTOOL_VERSION="3.14.0"
|
||||
|
||||
# ServiceDefinition JSON-Schema validation (schemas/validate.sh). Pip-installed;
|
||||
# pinned here so CI and a laptop validate against the exact same validator.
|
||||
CHECK_JSONSCHEMA_VERSION="0.29.4"
|
||||
+2
-49
@@ -29,9 +29,8 @@ clusters/
|
||||
|
||||
## How the overlay model satisfies R-ENV-02
|
||||
|
||||
The base authors the reconciliation graph **once**. `platform.yaml` and the app-wave Kustomizations
|
||||
(`apps-wave-0/1/2.yaml` — the single `apps.yaml` was retired; `apps-vms.yaml` deliberately omits it,
|
||||
see below) declare `postBuild.substituteFrom` the `cluster-vars` ConfigMap. Platform and
|
||||
The base authors the reconciliation graph **once**. `platform.yaml` and `apps.yaml` (Flux
|
||||
`Kustomization`s) both declare `postBuild.substituteFrom` the `cluster-vars` ConfigMap. Platform and
|
||||
app manifests use `${TOKEN}` placeholders for every value that legitimately differs between
|
||||
environments (issuer, replica counts, storage sizes, retention, node placement).
|
||||
|
||||
@@ -40,31 +39,6 @@ they add no resources of their own and never copy/fork a base manifest. Promotio
|
||||
([`docs/02`](../../home-network-k3s/docs/02-environments-and-provisioning.md)) is a change to the
|
||||
prod overlay's variables, nothing else.
|
||||
|
||||
### LAN addressing is a knob, not a literal (fixed 2026-08-31)
|
||||
|
||||
Until 2026-08-31 the base manifests hardcoded the **dev** LAN: the k3s API host Cilium dials, the
|
||||
Cilium LB IP pool range, the pinned crafty/unifi L4 VIPs and the off-cluster S3 backup endpoint. That
|
||||
broke R-ENV-02 outright — `clusters/prod` would have rendered dev addresses (192.168.1.x) onto the
|
||||
prod LAN (192.168.2.x). They are now the `cluster-vars` "LAN addressing" block
|
||||
(`K3S_API_HOST`, `CILIUM_LB_POOL_START`/`_STOP`, `UNIFI_L4_VIP`, `CRAFTY_L4_VIP`,
|
||||
`S3_BACKUP_ENDPOINT`, alongside the pre-existing `GATEWAY_VIP`), set explicitly by BOTH overlays.
|
||||
|
||||
Two rules when touching that block:
|
||||
|
||||
1. **Substitution coverage.** A `${TOKEN}` only resolves in a path whose Flux Kustomization declares
|
||||
`postBuild.substituteFrom: cluster-vars`; anywhere else the literal string `${TOKEN}` ships to the
|
||||
cluster. `apps-vms` was the one path without it and now has it — which means every shell `${...}`
|
||||
in a VM manifest's cloud-init must be escaped `$${...}` (see `clusters/base/apps-vms.yaml` for the
|
||||
full envsubst escape contract).
|
||||
2. **`GATEWAY_VIP` must be inside `CILIUM_LB_POOL_START..STOP`,** and in practice equal to START: the
|
||||
`shared` Gateway pins no address, so Cilium LB-IPAM gives it the pool's first free IP while
|
||||
split-horizon CoreDNS answers with `${GATEWAY_VIP}`. A VIP outside the pool blackholes in-cluster
|
||||
hairpin traffic while the Gateway still looks healthy.
|
||||
|
||||
Values the operator has not decided are `TBD-PROD-*` placeholders, not guesses: prod inherits the
|
||||
DEV base defaults if an overlay op is simply omitted, so an invalid value that fails loudly is
|
||||
strictly safer than a plausible one that quietly comes up on the wrong network.
|
||||
|
||||
## How Flux is pointed at an environment
|
||||
|
||||
`flux bootstrap` (run by WS-INFRA in Phase 1) targets one environment's path per cluster:
|
||||
@@ -97,27 +71,6 @@ Kustomization: apps-wave-1 (./apps/wave-1, dependsOn apps-wave-0)
|
||||
Kustomization: apps-wave-N (./apps/wave-N, dependsOn apps-wave-N-1)
|
||||
```
|
||||
|
||||
### Second-order CRDs need a THIRD pass (`platform-config-cdi`, added 2026-09-05)
|
||||
|
||||
The controllers/configs split (`platform` → `platform-config`) assumes a CRD is registered by the
|
||||
controller **install**. CDI breaks that: `cdi-operator.yaml` registers only `cdis.cdi.kubevirt.io`,
|
||||
and cdi-operator creates `storageprofiles` / `datavolumes` / `datasources` **only after it has
|
||||
reconciled the `CDI` CR**. Since kustomize-controller server-side **dry-runs every object in a pass
|
||||
before applying any of them**, having the `CDI` CR and the CDI `StorageProfile`s in one pass was a
|
||||
**deadlock, not a race**: unknown kind → pass aborts → CR never applied → CRD never created →
|
||||
`platform-config` can never converge → every tier below it blocked. Invisible on dev (its CDI CRDs
|
||||
have existed for months); it cost 12h on prod's cold start.
|
||||
|
||||
So there is a third tier, `platform-config-cdi` (`./platform-config-cdi`, `dependsOn:
|
||||
platform-config`), and a rule:
|
||||
|
||||
> A CR belongs in `platform-config-cdi`, not `platform-config`, when the CRD it needs is created by a
|
||||
> **controller at runtime** rather than by a manifest in git.
|
||||
|
||||
`apps-vms` `dependsOn` it as well as `platform-config`, because CDI reads
|
||||
`StorageProfile/longhorn-vm` at DataVolume **provisioning** time — a VM disk provisioned before that
|
||||
profile exists is stamped RWO/Filesystem and is permanently not live-migratable.
|
||||
|
||||
Wave 0 `dependsOn` `platform-config` (transitively `platform`) so CNI/storage/TLS/DNS/gateway
|
||||
exist before any workload reconciles. The app tier is split into dependency-ordered WAVES
|
||||
(`catalog/render.py --emit-waves`): each `apps-wave-<N>` health-gates (wait:true) ONLY its own
|
||||
|
||||
@@ -4,11 +4,8 @@
|
||||
# NON-GATING (wait:false): unlike the app waves, this does NOT health-wait. A VM's first-boot image
|
||||
# import (CDI, minutes) + in-guest software install mean a VM can be intentionally long-not-Ready, so
|
||||
# health-gating it would wedge reconciliation for no benefit. VMs have no in-cluster dependents.
|
||||
# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass exist first, AND
|
||||
# platform-config-cdi so the CDI StorageProfiles do (2026-09-05: the StorageProfiles MOVED OUT of
|
||||
# platform-config into their own tier, so `dependsOn: platform-config` alone NO LONGER guarantees
|
||||
# them — see the dependsOn block). prune:true — this Kustomization solely owns the VM manifests (no
|
||||
# adoption concern).
|
||||
# dependsOn platform-config so KubeVirt/CDI + the longhorn-vm StorageClass + CDI StorageProfiles
|
||||
# exist first. prune:true — this Kustomization solely owns the VM manifests (no adoption concern).
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
@@ -22,47 +19,11 @@ spec:
|
||||
wait: false
|
||||
dependsOn:
|
||||
- name: platform-config
|
||||
# LOAD-BEARING, not decoration. apps/vms/unifi/20-datavolume.yaml provisions a DataVolume on
|
||||
# storageClassName `longhorn-vm`, and CDI reads StorageProfile/longhorn-vm AT PROVISIONING TIME to
|
||||
# decide the claim's accessModes/volumeMode. Without that profile's `claimPropertySets`
|
||||
# (ReadWriteMany + Block) the PVC is stamped RWO/Filesystem and the VM disk is PERMANENTLY not
|
||||
# live-migratable — a silent, unrecoverable-without-recreate loss. The profiles used to arrive
|
||||
# with platform-config; since 2026-09-05 they arrive with `platform-config-cdi`, so the edge has
|
||||
# to move with them. platform-config-cdi runs wait:true and StorageProfile has no status
|
||||
# conditions (kstatus -> Current on apply), so this edge cannot stall this tier.
|
||||
#
|
||||
# HONEST LIMIT: CDI provisions from the profile's STATUS, which its storageprofile-controller
|
||||
# derives from our spec. This edge proves the spec is applied, not that CDI has folded it into
|
||||
# status — that is a watch-triggered round-trip of seconds against a ~30s dependency requeue, so
|
||||
# it is a timing argument rather than a guarantee. See clusters/base/platform-config-cdi.yaml.
|
||||
#
|
||||
# This edge also makes the StorageProfile prune/recreate transition safe: platform-config's prune
|
||||
# deletes the two profiles for the ~30-60s before platform-config-cdi re-applies them, and this
|
||||
# dependsOn is what guarantees Flux cannot create a DataVolume inside that window.
|
||||
#
|
||||
# The platform-config edge above is kept although platform-config-cdi already dependsOn it: the
|
||||
# edges are a set, there is no cycle, and stating both means a future edit to platform-config-cdi
|
||||
# cannot silently drop this tier's KubeVirt/StorageClass gate.
|
||||
- name: platform-config-cdi
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./apps/vms
|
||||
# postBuild.substituteFrom ENABLED 2026-08-31 (R-ENV-02). It was previously omitted because Flux's
|
||||
# envsubst runs over the ENTIRE manifest — including VM cloud-init shell scripts — and would mangle
|
||||
# every `${...}`; the omission was safe only while no VM manifest referenced a cluster-var. It now
|
||||
# does: apps/vms/unifi/50-service-l4.yaml pins the UniFi L4 VIP from ${UNIFI_L4_VIP} instead of the
|
||||
# hardcoded dev-LAN literal that would have shipped 192.168.1.54 onto the PROD LAN.
|
||||
#
|
||||
# The escape contract the old comment prescribed is now in force and MUST be honoured by every VM
|
||||
# manifest under this path (verified with `flux envsubst`):
|
||||
# `${VAR}` -> substituted (empty when unset) => escape shell braces as `$${VAR}`, which
|
||||
# envsubst emits back as a literal `${VAR}`.
|
||||
# `$VAR` -> left alone by envsubst => bare shell vars need NO escaping.
|
||||
# `$(cmd)` -> left alone by envsubst => command substitution needs NO escaping.
|
||||
# `${ARR[0]}` (e.g. ${PIPESTATUS[0]}) -> HARD ERROR "missing closing brace", which fails this
|
||||
# whole Kustomization. NEVER use an indexed expansion here, escaped or not.
|
||||
postBuild:
|
||||
substituteFrom:
|
||||
- kind: ConfigMap
|
||||
name: cluster-vars
|
||||
# NO postBuild.substituteFrom: Flux's envsubst would run over the ENTIRE manifest, including the
|
||||
# VM cloud-init shell scripts, and mangle every `${...}` (e.g. ${UOS_URL}, ${PIPESTATUS[0]}) —
|
||||
# corrupting the in-guest install. VM manifests reference no cluster-vars, so substitution is
|
||||
# unneeded here. A future VM that needs a cluster-var must escape its shell `${...}` as `$${...}`.
|
||||
@@ -34,9 +34,6 @@ spec:
|
||||
substituteFrom:
|
||||
- kind: ConfigMap
|
||||
name: cluster-vars
|
||||
# SOPS decryption for the `cloud` INI credentials — velero-cloud.sops.yaml (dev's scoped MinIO key)
|
||||
# and velero-cloud-prod.sops.yaml (prod's Cloudflare R2 token) — is layered per environment by the
|
||||
# overlay (clusters/dev and clusters/prod patch THIS Kustomization by name), identical to how
|
||||
# headlamp/oauth2-proxy/crafty decrypt their own SOPS secrets. Both Secrets are applied on both
|
||||
# clusters; ${BACKUP_CREDENTIALS_SUFFIX} (substituted from cluster-vars above) decides which one the
|
||||
# velero Deployment actually mounts. See clusters/dev/kustomization.yaml.
|
||||
# SOPS decryption for velero-cloud.sops.yaml (the scoped MinIO `cloud` credential) is layered per
|
||||
# environment by the overlay (clusters/dev patches THIS Kustomization by name), identical to how
|
||||
# headlamp/oauth2-proxy/crafty decrypt their own SOPS secrets. See clusters/dev/kustomization.yaml.
|
||||
@@ -1,60 +0,0 @@
|
||||
---
|
||||
# Flux Kustomization: CILIUM (the CNI) — reconciles ./platform/cilium.
|
||||
# Serves R-PLAT-01 (platform components reconciled by GitOps) and R-GITOPS-01/03.
|
||||
#
|
||||
# WHY CILIUM HAS ITS OWN KUSTOMIZATION (split out of `platform`, 2026-09-01)
|
||||
#
|
||||
# This is the MIRROR IMAGE of the oauth2-proxy / headlamp splits, not a repeat of them. Those are
|
||||
# leaves that must NEVER gate a tier, so they left `platform` and run `wait: false` below it.
|
||||
# Cilium is the one component that legitimately SHOULD gate everything — but inside the shared
|
||||
# `platform` Kustomization its health was indistinguishable from eleven unrelated components:
|
||||
#
|
||||
# * `platform`'s Ready condition could not tell you "the CNI is down" from "longhorn is slow".
|
||||
# * The only emergency valve was `flux suspend kustomization platform` — freezing the ENTIRE
|
||||
# tier (storage, TLS, DNS, ingress, monitoring, Keycloak, KubeVirt) to stop touching the CNI.
|
||||
# `flux suspend kustomization cilium` now freezes exactly the CNI and nothing else, which is
|
||||
# the valve the 2026-09-01 substitution incident wanted and did not have.
|
||||
# * `platform`'s 10m interval + 5m timeout are wrong for a CNI: a 3-node DaemonSet rollout with
|
||||
# `wait: true` can exceed 5m, and re-rendering the CNI every 10m buys nothing.
|
||||
#
|
||||
# GATING IS PRESERVED, NOT WEAKENED: `wait: true` here plus `platform.dependsOn: [cilium]` means
|
||||
# nothing in the graph reconciles until the CNI is healthy — the same guarantee as before, now
|
||||
# expressed as an explicit edge instead of an accident of co-tenancy.
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: cilium
|
||||
namespace: flux-system
|
||||
spec:
|
||||
# A CNI does not need a 10m re-render cadence. A git push still reconciles immediately —
|
||||
# kustomize-controller is source-triggered; `interval` is only the drift-correction floor.
|
||||
interval: 1h
|
||||
retryInterval: 2m
|
||||
# 10m (not the 5m `platform` uses): a 3-node Cilium DaemonSet rollout under `wait: true`,
|
||||
# rolled ONE NODE AT A TIME (maxUnavailable: 1, P2), legitimately exceeds 5m.
|
||||
timeout: 10m
|
||||
prune: true
|
||||
# TRUE, deliberately: this is the ONE component whose health should gate the whole platform.
|
||||
wait: true
|
||||
# Same revision-gating reason `platform` has it (2026-09-01 incident): `cluster-vars` is APPLIED
|
||||
# by the root `flux-system` Kustomization, and this Kustomization SUBSTITUTES from it. `dependsOn`
|
||||
# is revision-gated (kustomize-controller v1.4.0 `checkDependencies`), so this cannot render
|
||||
# revision N of helmrelease.yaml against the revision N-1 `cluster-vars`. That race is exactly how
|
||||
# `k8sServiceHost: ""` once reached the Cilium HelmRelease and took down 2/3 agents.
|
||||
dependsOn:
|
||||
- name: flux-system # the ROOT Kustomization (sync.yaml) that applies cluster-vars
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./platform/cilium
|
||||
# MANDATORY, not decorative: platform/cilium/helmrelease.yaml references ${K3S_API_HOST}.
|
||||
# kustomize-controller now runs --feature-gates=StrictPostBuildSubstitutions=true, so without
|
||||
# this stanza an undefined ${K3S_API_HOST} FAILS this Kustomization outright (it no longer
|
||||
# renders empty). That is the intended fail-closed behaviour — but it means the stanza is
|
||||
# load-bearing for the CNI coming up at all. Do not remove it.
|
||||
postBuild:
|
||||
substituteFrom:
|
||||
- kind: ConfigMap
|
||||
name: cluster-vars
|
||||
# No `decryption` stanza: platform/cilium carries no SOPS-encrypted resource (unlike
|
||||
# oauth2-proxy/headlamp/crafty/velero, whose overlays patch one in by name).
|
||||
+10
-311
@@ -17,228 +17,20 @@ data:
|
||||
CLUSTER_ENV: "base" # dev | prod (overlays set this)
|
||||
|
||||
# --- DNS / TLS ---
|
||||
# THE platform hostname suffix. Every public hostname the cluster serves is `<svc>.${BASE_DOMAIN}`,
|
||||
# the wildcard cert is `*.${BASE_DOMAIN}`, coredns-custom answers the `${BASE_DOMAIN}` zone with
|
||||
# ${GATEWAY_VIP}, and every OIDC issuer/redirect URI is built from it.
|
||||
#
|
||||
# THIS KNOB WAS DEAD UNTIL 2026-09-04 — declared here but referenced by NO manifest, because every
|
||||
# hostname was hardcoded. That is what made the 2026-09-04 incident possible: prod and dev were
|
||||
# authored to serve the SAME hostnames out of the SAME Cloudflare zone, so prod's external-dns
|
||||
# (policy: sync, owner `default`) concluded it owned dev's public records and DELETED ALL OF THEM.
|
||||
# See infra/PROD-PROVISIONING.md §10. The fix is architectural: each environment gets its OWN
|
||||
# subdomain, so the two clusters can no longer describe the same name.
|
||||
#
|
||||
# dev -> home.keithjackson.dev (UNCHANGED — see below)
|
||||
# prod -> prod.home.keithjackson.dev
|
||||
#
|
||||
# DEV DELIBERATELY STAYS ON THE APEX `home.keithjackson.dev`. The target architecture puts dev on
|
||||
# `dev.home.keithjackson.dev` with apex CNAMEs `<svc>.home` -> `<svc>.prod.home`, but moving dev
|
||||
# relocates its Keycloak issuer, every OIDC client redirect URI, kubectl SSO, its wildcard cert and
|
||||
# the Gitea host that IS the GitOps source — on a live cluster. Deferred to phase 2
|
||||
# (infra/PROD-PROVISIONING.md §10). Base default is therefore the DEV value, the same convention as
|
||||
# GATEWAY_VIP / LONGHORN_DATA_PATH: the base default is the known-good working value, dev restates
|
||||
# it verbatim, and prod overrides it. Dev's rendered output is byte-identical to the pre-
|
||||
# tokenization hardcoded manifests (verified 2026-09-04, all 12 substituted paths).
|
||||
BASE_DOMAIN: "home.keithjackson.dev"
|
||||
|
||||
# ExternalDNS `--domain-filter` (platform/externaldns/helmrelease.yaml). SAFETY NET #2 behind
|
||||
# EXTERNALDNS_TXT_OWNER_ID: a record whose name falls outside this filter is not even considered,
|
||||
# so it cannot be created, updated or deleted regardless of TXT ownership.
|
||||
#
|
||||
# DEV KEEPS THE ZONE APEX `keithjackson.dev`, and must: dev serves the `home.keithjackson.dev`
|
||||
# apex itself, and the Cloudflare provider filters the ACCOUNT'S ZONES by this value — a filter
|
||||
# that is a strict subdomain of the zone dropped the parent zone outright ("no hosted zone matching
|
||||
# record DNS Name", DECISIONS-LOG 2026-08) on the version in use then. So dev's value is unchanged
|
||||
# from the retired `DNS_ZONE` knob this replaces, and dev renders byte-identically.
|
||||
#
|
||||
# PROD NARROWS TO ITS OWN SUBTREE `prod.home.keithjackson.dev`. VERIFY ON PROD'S FIRST RECONCILE
|
||||
# (infra/PROD-PROVISIONING.md §10): if the zone-lookup limitation above still applies to the pinned
|
||||
# chart, prod's external-dns will select NO zone and create NO records. That failure is FAIL-CLOSED
|
||||
# — it can never delete another environment's records — and the fallback is to set prod's value
|
||||
# back to "keithjackson.dev", leaving EXTERNALDNS_TXT_OWNER_ID as the sole (and sufficient) guard.
|
||||
EXTERNALDNS_DOMAIN_FILTER: "keithjackson.dev"
|
||||
|
||||
# ExternalDNS TXT registry owner id (`--txt-owner-id`). THE ROOT CAUSE OF THE 2026-09-04 INCIDENT:
|
||||
# neither cluster set it, so both inherited the chart default `default`, and ExternalDNS treats a
|
||||
# record whose TXT registry entry carries ITS OWN owner id as its own to reconcile — and with
|
||||
# `policy: sync`, to DELETE when no local resource claims it. Two clusters sharing an owner id on
|
||||
# one zone therefore garbage-collect each other. Each environment MUST have a distinct value.
|
||||
#
|
||||
# DEV MUST RENDER EXACTLY `default`. Dev's LIVE public records are already registered in Cloudflare
|
||||
# under owner `default`; changing dev's value would make dev's own external-dns stop recognising
|
||||
# them, and it would re-create duplicates alongside the orphaned originals. Dev's overlay pins it.
|
||||
#
|
||||
# THE BASE DEFAULT IS A NON-COLLIDING SENTINEL, deliberately NOT `default`. A new environment that
|
||||
# forgets this knob must not silently inherit dev's identity — that is precisely the incident. With
|
||||
# this sentinel such a cluster owns nothing, so it can DELETE nothing; it would publish under a
|
||||
# visibly wrong owner id (grep the TXT records) instead of destroying a live environment's DNS.
|
||||
EXTERNALDNS_TXT_OWNER_ID: "unset-see-cluster-vars"
|
||||
|
||||
BASE_DOMAIN: "home.keithjackson.dev" # platform hostname suffix (same both envs; split-horizon)
|
||||
DNS_ZONE: "keithjackson.dev" # the actual Cloudflare ZONE (BASE_DOMAIN is a subdomain of it);
|
||||
# ExternalDNS domain-filter must be the zone, not the subdomain
|
||||
ACME_CLUSTER_ISSUER: "letsencrypt-staging" # dev: staging | prod: letsencrypt-prod (ADR-0006)
|
||||
GATEWAY_VIP: "192.168.1.53" # in-cluster split-horizon target (R-DNS-01/03): the L2-announced
|
||||
# ingress VIP of the `shared` Gateway. MUST equal the FIRST address of
|
||||
# ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} below — the Gateway
|
||||
# takes the pool's first free IP, and coredns-custom answers
|
||||
# *.home.keithjackson.dev with THIS so pod -> public-hostname traffic
|
||||
# hairpins through the Gateway (correct SNI -> wildcard cert).
|
||||
# Base default = the DEV LAN VIP (was 10.73.0.240, an address on the
|
||||
# long-RETIRED libvirt network — corrected 2026-08-31 so an overlay that
|
||||
# forgets this knob at least lands on a REAL network). A prod overlay
|
||||
# MUST still set the prod LAN VIP alongside the prod L2 pool range.
|
||||
|
||||
# --- LAN addressing (R-ENV-02) — the ONE block every new environment must re-supply -------------
|
||||
# These were HARDCODED into base manifests until 2026-08-31, which broke the golden constraint:
|
||||
# `clusters/prod` would have rendered the DEV LAN (192.168.1.0/24) onto the prod LAN (192.168.2.0/24)
|
||||
# — a silent, cross-environment IP collision. They are knobs now. Base defaults ARE the dev values
|
||||
# (same convention as GATEWAY_VIP above: the base default is the known-good working value), so the
|
||||
# dev overlay restates them verbatim and a prod overlay MUST override EVERY knob in this block
|
||||
# before `flux bootstrap --path=clusters/prod`.
|
||||
#
|
||||
# Substitution coverage (verify before adding a knob here): every path that consumes these declares
|
||||
# postBuild.substituteFrom cluster-vars — platform (K3S_API_HOST), platform-config
|
||||
# (CILIUM_LB_POOL_*, and keycloak-db's S3_BACKUP_ENDPOINT + CNPG_BACKUP_PATH: the Cluster lives at
|
||||
# platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
|
||||
# tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
|
||||
# S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
|
||||
# GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers),
|
||||
# velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
|
||||
# VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
|
||||
# UNSUBSTITUTED path would ship the literal string into the cluster.
|
||||
#
|
||||
# ADDING A KNOB IS SAFE UNDER StrictPostBuildSubstitutions ONLY BECAUSE OF THE ROOT EDGE. An
|
||||
# UNDEFINED variable is a HARD Kustomization failure, so a manifest that references a knob the
|
||||
# live ConfigMap does not yet carry breaks its whole tier. `clusters/base/platform.yaml` gained
|
||||
# `dependsOn: [flux-system]` for exactly this: dependsOn is REVISION-gated, so no substituting
|
||||
# tier can reconcile revision N until the ROOT Kustomization has APPLIED revision N — i.e. until
|
||||
# the new cluster-vars key EXISTS. Every substituting consumer chains transitively through
|
||||
# `platform`, so one edge covers the graph, and a new knob plus its first reference may ship in
|
||||
# ONE commit. (A hand-run `kubectl apply -k clusters/<env>` bypasses that ordering entirely: apply
|
||||
# the overlay BEFORE the tier that references the knob, or the tier fails until it retries.)
|
||||
|
||||
# k3s API endpoint Cilium's kube-proxy replacement dials (platform/cilium/helmrelease.yaml).
|
||||
#
|
||||
# *** THIS IS THE CONTROL-PLANE VIP (kube-vip), NOT A NODE ADDRESS. *** Operator decision
|
||||
# 2026-09-06. It was each environment's PRIMARY NODE IP until then, which made the CNI's
|
||||
# control-plane path depend on one machine in a cluster whose entire purpose is HA: verified on
|
||||
# live prod, all three cilium-agents carried KUBERNETES_SERVICE_HOST=192.168.2.10 (k3s-prod-1's
|
||||
# own address) while kube-vip held 192.168.2.9. Losing that node leaves the datapath running off
|
||||
# Cilium's cache while NO service or policy update lands — a silent, unbounded staleness, not an
|
||||
# outage you would notice. A virtual address that follows the control plane is the fix, and it is
|
||||
# what kube-vip was stood up to provide.
|
||||
#
|
||||
# dev -> 192.168.1.49 (= kube_vip_address, group_vars/all.yml; base default, below)
|
||||
# prod -> 192.168.2.9 (= kube_vip_address, group_vars/k3s_prod/vars.yml)
|
||||
#
|
||||
# THE NAME IS NOW SLIGHTLY WRONG AND IS KEPT ANYWAY: renaming the knob would be a second,
|
||||
# unrelated change to every substituting Kustomization at the same time as a live CNI endpoint
|
||||
# move. Read it as "the API host Cilium dials", which is what it always meant.
|
||||
#
|
||||
# [parity] MUST equal the Ansible bootstrap's **k3s_apiserver_endpoint** for the SAME environment
|
||||
# (infra/ansible/inventory/group_vars/all.yml -> roles/cilium/templates/cilium-values.yaml.j2,
|
||||
# already inventory-parameterized). NOT k3s_api_host — that variable still exists and still holds
|
||||
# the FOUNDER's address, because the cold-bootstrap `--server` join needs a machine that is
|
||||
# already up before any VIP exists. The two are deliberately separate; see the TWO API ADDRESSES
|
||||
# block at the top of group_vars/all.yml.
|
||||
#
|
||||
# Wrong value = Cilium cannot reach the API server on adopt and the CNI breaks. Port stays 6443
|
||||
# in both envs, so it is NOT a knob.
|
||||
#
|
||||
# TLS needs nothing: k3s_tls_sans has always carried kube_vip_address alongside every node IP, so
|
||||
# both addresses are already in the serving cert (verified live 2026-09-06 — prod answers through
|
||||
# https://192.168.2.9:6443 under full CA verification). No cert reissue, no k3s restart.
|
||||
K3S_API_HOST: "192.168.1.49"
|
||||
|
||||
# CiliumLoadBalancerIPPool `default-pool` (platform-config/cilium/l2-ip-pool.yaml): the VIP range
|
||||
# L2-announced on the flat LAN. Must sit OUTSIDE the node IPs and any DHCP scope, and must match the
|
||||
# tofu reservation (cilium_l2_vip_range). INVARIANT: ${GATEWAY_VIP} must be INSIDE this range and in
|
||||
# practice EQUAL to START — the `shared` Gateway pins no address, so LB-IPAM hands it the pool's
|
||||
# first free IP. (⚠ the operator-supplied PROD values currently break this; see l2-ip-pool.yaml and
|
||||
# the flagged block in clusters/prod/kustomization.yaml.)
|
||||
CILIUM_LB_POOL_START: "192.168.1.53"
|
||||
CILIUM_LB_POOL_STOP: "192.168.1.60"
|
||||
|
||||
# Pinned VIPs carved out of that pool by name (lbipam.cilium.io/ips). These are L4 workloads the
|
||||
# HTTPS Gateway cannot express (raw TCP/UDP), so they hold their own stable address:
|
||||
UNIFI_L4_VIP: "192.168.1.54" # UniFi controller VM: inform/STUN/adoption (apps/vms/unifi)
|
||||
CRAFTY_L4_VIP: "192.168.1.55" # Crafty: Minecraft Java (TCP) + Bedrock (UDP) game ports
|
||||
# PiHole's DNS listener: udp/53 + tcp/53 for LAN CLIENTS (apps/wave-0/pihole-dns.yaml). It is a
|
||||
# THIRD pinned L4 VIP for the same reason as the two above — the `shared` Gateway is an L7 HTTPS
|
||||
# terminator and cannot express DNS at all (udp/53 especially). It must NOT be ${GATEWAY_VIP}:
|
||||
# LAN clients are pointed at this address as their resolver, and it has to stay put across
|
||||
# reschedules, so it is pinned by lbipam.cilium.io/ips rather than taken from the pool head.
|
||||
#
|
||||
# WHY A SEPARATE ADDRESS AND NOT THE GATEWAY VIP: every `<svc>.${BASE_DOMAIN}` name resolves TO
|
||||
# the Gateway VIP. Serving DNS from the same address would be legal but conflates "the thing that
|
||||
# answers where services live" with "a service", and it would put LAN resolution behind the
|
||||
# Gateway Service's own lifecycle. Keep them separate.
|
||||
#
|
||||
# dev -> 192.168.1.56 (base default; next free after crafty .55 in dev's .53-.60 pool)
|
||||
# prod -> 192.168.2.23 (next free after crafty .22 in prod's .20-.30 pool; see clusters/prod)
|
||||
PIHOLE_DNS_VIP: "192.168.1.56" # PiHole: authoritative LAN DNS for ${BASE_DOMAIN}
|
||||
|
||||
# The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
|
||||
# peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable
|
||||
# from `world` is an open resolver and an amplification reflector, so the allow is scoped to this
|
||||
# CIDR, not to Cilium's `world` entity the way the unifi/crafty game-port allows are. It is the
|
||||
# same /24 the node IPs and ${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP} come out of.
|
||||
#
|
||||
# dev -> 192.168.1.0/24 (Proxmox LAN; nodes .50/.51/.52)
|
||||
# prod -> 192.168.2.0/24 (prod LAN; nodes .10/.11/.12)
|
||||
LAN_CIDR: "192.168.1.0/24" # ingress scope for the PiHole DNS listener
|
||||
|
||||
# --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
|
||||
# THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media
|
||||
# library (Storage/plex, 8.33 T — plex, sonarr, radarr) and the legacy Nextcloud archive
|
||||
# (Storage/nextcloud, 270 G). The renderer emits it into each static NFS PersistentVolume's
|
||||
# `spec.nfs.server` (catalog/render.py, BULK_NFS_SERVER); the export PATH is per-dataset and is
|
||||
# declared in the ServiceDefinition, because the path is identical in every environment and only
|
||||
# the ADDRESS is not.
|
||||
#
|
||||
# WHY THIS IS A KNOB AND NOT A LITERAL — the two clusters are on OPPOSITE SIDES of pfSense.
|
||||
# The storage host has exactly one interface, vmbr0 192.168.1.9/24, on pfSense's LAN.
|
||||
# dev nodes are 192.168.1.40/.41/.42 — the SAME L2 segment. They reach it directly, and one
|
||||
# NFS export to them already exists on that host (/mnt/Storage/k8s-volumes in
|
||||
# /etc/exports), so this path is proven, not theoretical. Base default = the dev value,
|
||||
# per the convention of the LAN block above.
|
||||
# prod nodes are 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does NOT traverse that today;
|
||||
# 22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. Opening
|
||||
# the storage path is a PENDING OPERATOR DECISION with two candidate outcomes that give
|
||||
# DIFFERENT values here: 192.168.2.102 (another port-forward) or 192.168.1.9 (a route).
|
||||
# So prod holds a loud TBD-PROD-* placeholder — see clusters/prod/kustomization.yaml for the
|
||||
# blast radius of shipping it unset, which is NOT the usual "fails to apply".
|
||||
BULK_NFS_SERVER: "192.168.1.9"
|
||||
|
||||
# Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
|
||||
# Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,
|
||||
# nextcloud-db, semaphore-db) and Velero's backupStorageLocation s3Url. Full URL incl. scheme+port.
|
||||
# Prod needs its OWN target: a backup that lands on the dev LAN's MinIO is not off-cluster DR —
|
||||
# and, measured 2026-09-05 from a pod INSIDE prod, it CANNOT reach it anyway (prod is on the SOHO
|
||||
# network 192.168.2.0/24, which is pfSense's WAN, while this MinIO sits inside pfSense's LAN:
|
||||
# `curl http://192.168.1.9:9000/minio/health/live` timed out after 15s). Prod uses Cloudflare R2.
|
||||
S3_BACKUP_ENDPOINT: "http://192.168.1.9:9000"
|
||||
|
||||
# S3 region. Every S3 client REQUIRES one; MinIO ignores whatever it is; Cloudflare R2 demands
|
||||
# exactly `auto` (its `head-bucket` reports the physical hint `WNAM`, which is NOT a usable region
|
||||
# string). This was hardcoded `us-east-1` inside platform/velero/helmrelease.yaml — correct-by-
|
||||
# accident for MinIO and wrong for anything else, which is why it is a knob now. The base default
|
||||
# is dev's LIVE value, deliberately: dev's Velero has been signing requests with `us-east-1`
|
||||
# against that MinIO since bootstrap, and this repo's rule is that base == the known-good dev
|
||||
# value. Prod overrides to `auto`.
|
||||
S3_BACKUP_REGION: "us-east-1"
|
||||
GATEWAY_VIP: "10.73.0.240" # in-cluster split-horizon target (R-DNS-01/03): the L2-announced
|
||||
# ingress VIP of the `shared` Gateway (first IP of the Cilium
|
||||
# LB IP pool 10.73.0.240-.250, platform-config/cilium/l2-ip-pool.yaml).
|
||||
# coredns-custom answers *.home.keithjackson.dev with THIS so pod->
|
||||
# public-hostname traffic hairpins through the Gateway (correct SNI ->
|
||||
# wildcard cert). Base default = dev VIP; a prod overlay must set the
|
||||
# prod LAN VIP alongside the prod L2 pool range when hardware is built.
|
||||
|
||||
# --- storage (ADR-0003) ---
|
||||
LONGHORN_DEFAULT_REPLICA_COUNT: "3" # default StorageClass replica count (3 on 3 nodes)
|
||||
# Longhorn's defaultDataPath (platform/longhorn/helmrelease.yaml): the on-NODE directory every
|
||||
# replica is written into. It must be the path the NODE's own data disk is actually mounted at —
|
||||
# roles/node_baseline owns that half (node_baseline_longhorn_data_path), and the two are ONE
|
||||
# fact stated in two places, so [parity] holds per environment or replicas land on the OS disk.
|
||||
# Base default = the DEV value, same convention as GATEWAY_VIP / the LAN block: the known-good
|
||||
# working value. Dev's dedicated /dev/vdb IS mounted here and the LIVE cluster has 18 volumes
|
||||
# on it — changing dev's value would orphan every replica, so the dev overlay restates it
|
||||
# verbatim. PROD DIFFERS: its data partition serves TWO tenants (Longhorn plus k3s's whole data
|
||||
# dir, moved off the 11.8 GiB /var — infra/PROD-PROVISIONING.md §2), so it is mounted at the
|
||||
# neutral /data and Longhorn gets the /data/longhorn SUBDIRECTORY, not the mount root.
|
||||
LONGHORN_DATA_PATH: "/var/lib/longhorn"
|
||||
|
||||
# --- observability sizing (docs/10) ---
|
||||
MONITORING_PROMETHEUS_RETENTION: "7d" # dev short | prod long
|
||||
@@ -278,99 +70,6 @@ data:
|
||||
VELERO_BACKUP_SCHEDULE: "0 3 * * *" # cron; overlays may differ
|
||||
VELERO_RETENTION: "168h" # dev short | prod long
|
||||
|
||||
# ==========================================================================================
|
||||
# OBJECT-STORE TENANCY — the three knobs that keep two clusters and two backup tools from
|
||||
# writing into each other's data. THESE ARE SAFETY KNOBS, not preferences.
|
||||
# ==========================================================================================
|
||||
# Nothing about the object layout was per-environment before this: Velero's bucket was the
|
||||
# hardcoded literal `velero` with no prefix, and every CNPG Cluster used the hardcoded
|
||||
# `s3://cnpg` with barman's `serverName` defaulting to the CLUSTER NAME — and BOTH environments
|
||||
# have a keycloak-db, a nextcloud-db and a semaphore-db. Point two clusters at one bucket under
|
||||
# that layout and:
|
||||
# (a) each cluster's Velero ADOPTS the other's backups and TTL-DELETES them on retention; and
|
||||
# (b) two different Postgres servers stream WAL into ONE Barman prefix, which does not merely
|
||||
# mix files — it CORRUPTS THE CHAIN and destroys PITR for both. That is the worse half:
|
||||
# dev's DR restore is PROVEN (R-STATE-03) and this would silently un-prove it.
|
||||
#
|
||||
# WHY THERE IS NO SINGLE `S3_BACKUP_BUCKET`. The two consumers take differently-SHAPED values:
|
||||
# the Velero chart wants `bucket` and `prefix` as separate fields, while CNPG's
|
||||
# barmanObjectStore.destinationPath is ONE full s3:// URL whose first path segment IS the bucket.
|
||||
# CNPG's bucket therefore already lives inside CNPG_BACKUP_PATH; adding a second knob that had to
|
||||
# hold the same bucket name a second time would be a rotation waiting to drift — the exact class
|
||||
# of duplication this repo has been bitten by. Three knobs, each shaped for its own consumer, and
|
||||
# no two of them can contradict each other.
|
||||
#
|
||||
# BASE = DEV'S LIVE LAYOUT, BYTE-FOR-BYTE, AND MUST STAY THAT WAY. Dev's existing backups are at
|
||||
# `s3://velero/` (bucket root, no prefix) and `s3://cnpg/<clusterName>/` on that MinIO. Giving dev
|
||||
# a prefix, or renaming its buckets, would ORPHAN the very backups its proven restore depends on —
|
||||
# the same reason dev's EXTERNALDNS_TXT_OWNER_ID stays `default` and its LONGHORN_DATA_PATH stays
|
||||
# /var/lib/longhorn. Verified: with an EMPTY prefix the Velero chart omits `objectStorage.prefix`
|
||||
# from the rendered BackupStorageLocation entirely (helm template, chart 12.1.0), so dev's live
|
||||
# BSL is unchanged by this parameterization rather than merely equivalent to it.
|
||||
#
|
||||
# PROD, which has no history to preserve, gets ONE bucket with TWO disjoint top-level prefixes:
|
||||
#
|
||||
# s3://k3s-prod-backup/velero/… Velero (kopia repo + backup metadata)
|
||||
# s3://k3s-prod-backup/barman/<clusterName>/… CNPG WAL + base backups
|
||||
#
|
||||
# The operator created exactly one R2 bucket, so velero-vs-barman separation has to come from the
|
||||
# prefix rather than the bucket. The BUCKET ROOT IS DELIBERATELY LEFT EMPTY: if a prefix ever
|
||||
# renders wrong, its objects land somewhere obviously bogus instead of silently colliding with a
|
||||
# real tenant's data. Dev↔prod isolation is doubly independent — different endpoint AND different
|
||||
# bucket — so neither cluster can reach the other's objects even with a mis-substituted prefix.
|
||||
VELERO_BACKUP_BUCKET: "velero" # dev: the `velero` MinIO bucket | prod: k3s-prod-backup
|
||||
VELERO_BACKUP_PREFIX: "" # dev: bucket root (live layout) | prod: "velero"
|
||||
CNPG_BACKUP_PATH: "s3://cnpg" # dev: the `cnpg` MinIO bucket root | prod: s3://k3s-prod-backup/barman
|
||||
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
# WHICH CREDENTIAL SECRET THE BACKUP CONSUMERS READ (added 2026-09-07 — the per-env credential split)
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
# The three knobs above say WHERE the backups go. This one says WHICH KEY signs the requests, and
|
||||
# it exists because the endpoint moved per-environment while the credential did not.
|
||||
#
|
||||
# THE BUG IT FIXES. The four backup-credential Secrets — velero-cloud.sops.yaml and the three
|
||||
# <db>-backup-credentials.sops.yaml — are SHARED BASE manifests: platform/ and apps/ are rendered
|
||||
# by the same Flux tiers on both clusters, and cluster-vars substitution is the ONLY thing that
|
||||
# differs (R-ENV-02). So writing prod's R2 token into those four files did not GIVE prod a
|
||||
# credential, it REPLACED DEV'S: dev's Velero and all three dev CNPG Clusters would have started
|
||||
# signing MinIO requests with an R2 key id. For Velero that is a broken BSL; for CNPG it is worse,
|
||||
# because `archive_mode=on` means Postgres REFUSES TO RECYCLE a WAL segment it could not archive,
|
||||
# so a 403 loop is a monotonically filling pgdata PVC ending in a PANIC — the same failure prod
|
||||
# spent its whole bring-up in with a placeholder endpoint. One shared Secret also means rotating
|
||||
# either environment's key silently breaks the other; that is the structural reason for the split,
|
||||
# independent of dev's eventual decommission.
|
||||
#
|
||||
# THE MECHANISM. Each credential is now TWO Secret objects, both applied to BOTH clusters (they are
|
||||
# base resources; nothing here forks a manifest per environment). The consumer names one:
|
||||
#
|
||||
# velero: credentials.existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}
|
||||
# CNPG: s3Credentials.*.name: <db>-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
|
||||
#
|
||||
# ONE knob, not four, ON PURPOSE. Four independent secret-name knobs could be updated three-
|
||||
# quarters of the way, leaving one consumer pointed at the OTHER environment's key with nothing
|
||||
# rendering wrong — and StrictPostBuildSubstitutions cannot catch that, because it only fails on an
|
||||
# UNDEFINED variable, never on a defined-but-stale one. With a single suffix the four move together
|
||||
# or not at all. The cost is that the rendered Secret name is not greppable from the consumer; the
|
||||
# names are spelled out in full in each kustomization.yaml beside the resources.
|
||||
#
|
||||
# THE BASE DEFAULT IS THE EMPTY STRING, AND THAT IS LOAD-BEARING. It renders the UNSUFFIXED names,
|
||||
# which are the objects that already exist and that the live dev cluster is running. A default
|
||||
# naming a Secret that does not exist would be worse than the shared-Secret state it replaces:
|
||||
# Velero's pod would sit in ContainerCreating forever on a missing `existingSecret` volume, and a
|
||||
# CNPG Cluster whose s3Credentials Secret is absent stops archiving WAL — i.e. the disk-full clock
|
||||
# again — so an unresolvable default breaks the referencing cluster HARDER than a wrong key does.
|
||||
# Empty is also the value dev must carry, so base and dev agree, exactly like VELERO_BACKUP_PREFIX.
|
||||
#
|
||||
# An EMPTY VALUE IS SAFE UNDER StrictPostBuildSubstitutions: strict mode fails on a variable that is
|
||||
# not DEFINED, not on one defined as "". VELERO_BACKUP_PREFIX above has shipped empty since the
|
||||
# gate was enabled and proves it. Both overlays state this knob explicitly — dev "" and prod
|
||||
# "-prod" — so neither inherits it silently.
|
||||
#
|
||||
# ADDING A NEW BACKUP CREDENTIAL: create BOTH halves (`x-credentials.sops.yaml` and
|
||||
# `x-credentials-prod.sops.yaml`), list both in the kustomization, and reference
|
||||
# `x-credentials${BACKUP_CREDENTIALS_SUFFIX}`. Never a single shared Secret again.
|
||||
BACKUP_CREDENTIALS_SUFFIX: "" # dev: "" (MinIO creds) | prod: "-prod" (R2 creds)
|
||||
|
||||
# --- placement ---
|
||||
PLATFORM_NODE_SELECTOR_KEY: "kubernetes.io/os" # base no-op selector; prod may pin roles
|
||||
PLATFORM_NODE_SELECTOR_VALUE: "linux"
|
||||
@@ -34,12 +34,8 @@ spec:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./platform/homepage
|
||||
# LOAD-BEARING, not parity decoration (the comment here used to say the opposite). This component
|
||||
# references ${BASE_DOMAIN} in TWO places: httproute.yaml's apex hostname, and EVERY tile href in
|
||||
# configmap.yaml. Without this substitution the token is applied LITERALLY — the apex route binds
|
||||
# the host `${BASE_DOMAIN}`, which matches no Gateway listener, and every link on the operator's
|
||||
# front door points at a host that does not exist — while Flux still reports this Kustomization
|
||||
# Ready. It fails OPEN. Asserted by conformance/tests/l0-static/c2-root-surface (5).
|
||||
# Homepage references no ${VARS} today; the substitution is retained for parity with the other
|
||||
# tiers / future overlay knobs.
|
||||
postBuild:
|
||||
substituteFrom:
|
||||
- kind: ConfigMap
|
||||
|
||||
@@ -5,24 +5,8 @@ apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- flux-system.yaml # GitRepository (reconciliation source)
|
||||
# Cilium (the CNI) — its OWN Kustomization, split OUT of `platform` (2026-09-01). The mirror image
|
||||
# of the oauth2-proxy/headlamp splits: those are leaves that must never gate a tier (wait:false);
|
||||
# Cilium is the one component that legitimately SHOULD gate everything, so it keeps wait:true and
|
||||
# `platform` dependsOn it. Splitting it out buys a per-CNI Ready condition, CNI-appropriate
|
||||
# interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
|
||||
# freeze the other eleven platform components. See clusters/base/cilium.yaml.
|
||||
- cilium.yaml # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
|
||||
- platform.yaml # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
|
||||
- platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
|
||||
# SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any
|
||||
# manifest in git. cdi-operator.yaml registers only `cdis.cdi.kubevirt.io`; `storageprofiles` (and
|
||||
# datavolumes, datasources, …) appear only after cdi-operator reconciles the `CDI` CR that
|
||||
# platform-config applies. Because Flux dry-runs every object in a pass before applying any of them,
|
||||
# keeping the StorageProfiles in platform-config was a TRUE DEADLOCK (unknown kind -> pass aborts ->
|
||||
# CR never applied -> CRD never created -> tier never converges), invisible on dev and fatal on any
|
||||
# cold cluster — observed on prod 2026-09-05. This tier is the durable fix; `standup.sh crds` stays
|
||||
# as the belt-and-braces repair for clusters already wedged. See clusters/base/platform-config-cdi.yaml.
|
||||
- platform-config-cdi.yaml # Flux Kustomization -> ./platform-config-cdi (dependsOn platform-config)
|
||||
# App tier, split into dependency-ordered WAVES (was the single ./apps `apps` Kustomization).
|
||||
# Each apps-wave-<N> dependsOn wave N-1 (wave 0 dependsOn platform-config) and health-gates
|
||||
# (wait:true) ONLY its own wave, so a crashloop in a later wave can't stall an earlier one.
|
||||
@@ -50,13 +34,6 @@ resources:
|
||||
# reverse-proxy sidecar (Option B). POST-platform tier (dependsOn platform-config, wait:false); same
|
||||
# non-gating leaf-UI rationale as headlamp/homepage/oauth2-proxy (a UI must never gate the platform).
|
||||
- crafty.yaml # Flux Kustomization -> ./platform/crafty (dependsOn platform-config)
|
||||
# Alert runbooks — POST-platform tier (dependsOn platform-config, wait:false). Serves the 34
|
||||
# runbooks the alert rules' `runbook_url` annotations point at, at <apex>/runbooks/<name>; before
|
||||
# this tier existed that path was a 404 and every alert handed the operator a dead link. Same
|
||||
# non-gating leaf rationale as headlamp/homepage/crafty. Carries NO postBuild substitution on
|
||||
# purpose — it renders documentation containing `${...}` shell examples that envsubst would empty;
|
||||
# see clusters/base/runbooks.yaml.
|
||||
- runbooks.yaml # Flux Kustomization -> ./runbooks (dependsOn platform-config)
|
||||
# Velero backup controller — POST-platform tier (dependsOn platform-config, wait:false). A backup
|
||||
# controller must NEVER gate the platform chain; same non-gating rationale as headlamp/oauth2-proxy.
|
||||
# Carries velero-cloud.sops.yaml, so clusters/dev adds a SOPS decryption patch for it (by name).
|
||||
|
||||
@@ -1,135 +0,0 @@
|
||||
---
|
||||
# Flux Kustomization: SECOND-ORDER CDI CONFIGS — reconciles ./platform-config-cdi.
|
||||
# Serves R-GITOPS-01 (all desired state in git, reconciled by Flux) and R-ENV-02.
|
||||
#
|
||||
# WHY THIS TIER EXISTS — the durable fix for the 2026-09-05 prod CDI DEADLOCK
|
||||
#
|
||||
# `platform-config` applied the `CDI` CR and the CDI `StorageProfile`s in the SAME pass.
|
||||
# kustomize-controller server-side DRY-RUNS EVERY object before applying ANY of them, and
|
||||
# `storageprofiles.cdi.kubevirt.io` does not exist until cdi-operator has reconciled that `CDI` CR.
|
||||
# So the unknown kind aborted the whole tier → the CR was never applied → the CRD was never created →
|
||||
# platform-config could NEVER converge and every tier below it stayed blocked. Not a race: a
|
||||
# deadlock. cdi-operator was Available for 12h on prod with no CR. Invisible on dev (its CDI CRDs
|
||||
# have existed for months), FATAL on any cold cluster. See DECISIONS-LOG 2026-09-05.
|
||||
#
|
||||
# `bootstrap/standup.sh crds` pre-applies the `CDI` CR to break it by hand. That is a WORKAROUND and
|
||||
# stays as belt-and-braces for clusters already wedged; THIS Kustomization is the fix that makes the
|
||||
# graph order itself, so a cold cluster needs no human in the loop.
|
||||
#
|
||||
# THE ORDERING GUARANTEE, precisely: `dependsOn` is REVISION-gated, not merely Ready-gated (verified
|
||||
# against kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our
|
||||
# version): sharing a GitRepository with the dependency, this Kustomization cannot reconcile revision
|
||||
# N until `platform-config` has APPLIED revision N — and, because platform-config runs `wait: true`,
|
||||
# not until the `CDI` CR it applied is HEALTHY. cdi-operator creates the remaining CDI CRDs as part of
|
||||
# reaching that state, so by the time this tier's first dry-run runs, `StorageProfile` is a known
|
||||
# kind. One edge, no pre-apply.
|
||||
#
|
||||
# WHY NOT just move the CDI CR earlier, into `platform`? Because `platform` is the tier the CNI gates
|
||||
# and everything else waits on: putting CDI's health on that critical path is the mistake the
|
||||
# oauth2-proxy/headlamp splits were made to undo. The CR is correctly a platform-config CR; it is the
|
||||
# StorageProfiles that were in the wrong pass.
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: platform-config-cdi
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 10m
|
||||
# 2m (short, deliberately). kustomize-controller caches API discovery, so on a COLD cluster the
|
||||
# very first attempt here can still fail `no matches for kind "StorageProfile"` even though the CRD
|
||||
# now exists — a stale RESTMapper, not a missing CRD (the failure mode `standup.sh crds` warns
|
||||
# about). The controller resets the mapper and the retry succeeds, so the only cost of that stale
|
||||
# cache is one retryInterval. Keep it short.
|
||||
retryInterval: 2m
|
||||
# 2m is ample: this tier applies two tiny cluster-scoped CRs. There is no workload to roll out, no
|
||||
# image to pull and no controller to become Available, so a slow apply here means a real error, not
|
||||
# a slow rollout — failing fast surfaces it instead of burning 5m per attempt.
|
||||
timeout: 2m
|
||||
# TRUE, and this tier is the SOLE declarer of both StorageProfiles (checked repo-wide), so prune is
|
||||
# component-scoped: the stale set can only ever be a StorageProfile this file stopped rendering.
|
||||
# It is NOT the platform/cilium case — nothing here declares a shared Namespace, or any object a
|
||||
# second path also declares. See the transition note at the bottom of this file.
|
||||
prune: true
|
||||
# TRUE, but understand exactly what it buys and what it does NOT.
|
||||
#
|
||||
# CANNOT STALL: `StorageProfile`'s status type (CDI v1beta1 StorageProfileStatus: storageClass,
|
||||
# provisioner, claimPropertySets, cloneStrategy, dataImportCronSourceFormat) carries NO
|
||||
# `conditions` and NO `observedGeneration`, so kstatus's generic path reports it Current the moment
|
||||
# it is applied. `wait: true` therefore adds a Ready condition without adding a way to hang — which
|
||||
# is why it is safe to let `apps-vms` gate on it (contrast the headlamp gating stall).
|
||||
# If a resource that CAN be unhealthy is ever added to ./platform-config-cdi, re-derive this:
|
||||
# `apps-vms` waits on it.
|
||||
#
|
||||
# WHAT IT DOES NOT PROVE (honest gap): CDI reads the profile's **status**, not its spec, when it
|
||||
# provisions a claim (status.claimPropertySets / status.cloneStrategy), and CDI's
|
||||
# storageprofile-controller computes that status from our spec override plus inference. So Ready
|
||||
# here means "the spec override is accepted by the API server", not "CDI has folded it into
|
||||
# status". That reconcile is a watch-triggered round-trip of seconds, and the dependency requeue
|
||||
# below it is ~30s, so a DataVolume cannot realistically beat it — but it is a timing argument, not
|
||||
# a guarantee, and no `wait` setting can turn it into one (there is no condition to wait on).
|
||||
wait: true
|
||||
dependsOn:
|
||||
# platform-config applies the `CDI` CR and (wait:true) waits for it to be healthy — which is what
|
||||
# causes cdi-operator to create `storageprofiles.cdi.kubevirt.io`. Revision-gated, so this is an
|
||||
# ordering guarantee and not a hope. Transitively this also chains platform -> cilium ->
|
||||
# flux-system, so `cluster-vars` at revision N is applied before this renders revision N.
|
||||
- name: platform-config
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./platform-config-cdi
|
||||
# NO SOPS `decryption` stanza: ../platform/kubevirt/cdi-storageprofiles.yaml is two plain CRs with
|
||||
# no Secret and no *.sops.yaml, so neither overlay needs to patch one in (unlike
|
||||
# oauth2-proxy/headlamp/crafty/velero, which each carry an encrypted Secret). Nothing to add to
|
||||
# clusters/dev or clusters/prod for this tier.
|
||||
#
|
||||
# substituteFrom cluster-vars: the manifest currently contains NO `${TOKEN}`, so this stanza
|
||||
# substitutes nothing today. It is declared anyway for two concrete reasons, not for symmetry:
|
||||
# 1. COVERAGE. conformance/tests/l0-static/flux-postbuild-substitution/check.sh only audits paths
|
||||
# whose Kustomization declares substituteFrom. Omitting it would move
|
||||
# cdi-storageprofiles.yaml OUT of the audited set — a blind spot where a future `${TOKEN}`
|
||||
# could ship as a literal string, precisely the class of bug that check exists to stop.
|
||||
# 2. FAIL-CLOSED. kustomize-controller runs --feature-gates=StrictPostBuildSubstitutions=true, so
|
||||
# with this stanza an undefined token FAILS this Kustomization loudly instead of rendering
|
||||
# empty. Without the stanza there is no substitution at all and the literal ships silently.
|
||||
postBuild:
|
||||
substituteFrom:
|
||||
- kind: ConfigMap
|
||||
name: cluster-vars
|
||||
# ── FIRST RECONCILE AFTER MERGE: a TRANSIENT DELETE-THEN-RECREATE. READ BEFORE MERGING. ──────────
|
||||
# Moving cdi-storageprofiles.yaml out of ./platform-config drops both StorageProfiles from THAT
|
||||
# Kustomization's inventory, and platform-config has `prune: true` — so its next apply DELETES
|
||||
# StorageProfile/longhorn and StorageProfile/longhorn-vm, and this Kustomization recreates them on
|
||||
# the following reconcile (revision-gated: strictly after platform-config's apply, so there is no
|
||||
# dual-ownership fight — the objects are pruned before they are re-applied under this tier's
|
||||
# `kustomize.toolkit.fluxcd.io/name=platform-config-cdi` label).
|
||||
#
|
||||
# Why that is acceptable here — and it is NOT the platform/cilium case:
|
||||
# * Neither object is the sole (or any) declarer of a shared object. No Namespace, no ConfigMap,
|
||||
# nothing else in the repo declares a StorageProfile. The stale set is exactly two
|
||||
# cluster-scoped CDI CRs.
|
||||
# * The objects do not actually go missing: CDI's storageprofile-controller creates a
|
||||
# StorageProfile per StorageClass and recreates a deleted one, with an EMPTY `.spec` and
|
||||
# inferred `.status`. So the transient is "our .spec overrides are absent", not "the object is
|
||||
# absent".
|
||||
# * A StorageProfile is read at DataVolume/PVC PROVISIONING and CLONE time only. Existing PVCs,
|
||||
# existing DataVolumes and running VMs do not consult it again, so nothing already provisioned
|
||||
# is disturbed and no data path is touched.
|
||||
# * The only exposure is a DataVolume/clone STARTED inside the window: it would clone by `copy`
|
||||
# instead of `snapshot`, and on `longhorn-vm` be stamped RWO/Filesystem instead of RWX/Block —
|
||||
# permanently non-live-migratable for that disk. Flux cannot cause that, because `apps-vms` now
|
||||
# dependsOn THIS tier (clusters/base/apps-vms.yaml), so no Flux-managed DataVolume can be
|
||||
# created while the overrides are absent. A human running `virtctl`/`kubectl` in that minute
|
||||
# could.
|
||||
#
|
||||
# PRECONDITION THE OPERATOR MUST CHECK, because it is what bounds the window: the recreate cannot
|
||||
# happen until `platform-config` is Ready, and platform-config runs `wait: true` over the whole
|
||||
# config tier (ClusterIssuers, wildcard Certificate, Gateway, Keycloak, CNPG, KubeVirt, CDI…). On a
|
||||
# healthy cluster that is seconds and the window is ~30-60s (kustomize-controller's
|
||||
# --requeue-dependency default is 30s). If platform-config is NOT Ready when this merges, the
|
||||
# StorageProfiles are pruned and stay at CDI defaults until it recovers. So:
|
||||
# `flux get kustomization platform-config` must be Ready BEFORE merging.
|
||||
# If it is not, do the two-step the cilium move used instead: merge with `platform-config.prune`
|
||||
# temporarily set to `false` (the objects are then dropped from its inventory but NOT deleted, and
|
||||
# this tier ADOPTS them with zero window), verify the ownership label transferred, then restore
|
||||
# `prune: true` in a second commit.
|
||||
@@ -11,53 +11,8 @@ spec:
|
||||
interval: 10m
|
||||
retryInterval: 2m
|
||||
timeout: 5m
|
||||
# Cilium was moved OUT of this tier into its own Kustomization (clusters/base/cilium.yaml)
|
||||
# on 2026-09-01. `prune: true` is restored here now that the ownership transfer is VERIFIED
|
||||
# live: Namespace/kube-system, the Cilium HelmRelease and the HelmRepository all carry
|
||||
# `kustomize.toolkit.fluxcd.io/name=cilium`, this Kustomization's inventory no longer lists
|
||||
# any of the three, and the `cilium` Kustomization's inventory holds exactly those three.
|
||||
#
|
||||
# LESSON WORTH KEEPING: `platform/cilium/namespace.yaml` was the ONLY declaration of
|
||||
# Namespace/kube-system under ./platform. Moving that path out of a prune:true Kustomization
|
||||
# made the stale set `delete namespace kube-system` — coredns, metrics-server, local-path AND
|
||||
# Cilium. Before moving ANY platform/* component in future, check whether it is the sole
|
||||
# declarer of a shared Namespace; that turns a component-scoped prune into a cluster-scoped one.
|
||||
prune: true
|
||||
prune: true # drift correction: delete resources removed from git (R-GITOPS-03)
|
||||
wait: true # block `apps` until platform is healthy
|
||||
# ORDERING FIX (2026-09-01 incident: a substitution race that failed OPEN broke Cilium).
|
||||
#
|
||||
# `cluster-vars` — the ConfigMap every substituting Kustomization reads — is APPLIED by the root
|
||||
# `flux-system` Kustomization (clusters/<env>/sync.yaml), not by this one. Without an edge here,
|
||||
# `platform` could render a manifest from revision N against the revision N-1 `cluster-vars`, so a
|
||||
# NEWLY ADDED knob was undefined and (non-strict substitution) silently became the EMPTY STRING.
|
||||
# That is how `k8sServiceHost: ""` reached the Cilium HelmRelease.
|
||||
#
|
||||
# `dependsOn` is REVISION-gated, not merely Ready-gated (verified against
|
||||
# kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies`, our version):
|
||||
# when the dependency shares a GitRepository with the dependent, the dependent cannot reconcile
|
||||
# revision N until the dependency has APPLIED revision N. Applying revision N is exactly what
|
||||
# writes a newly-added `cluster-vars` knob — so this one edge closes the race.
|
||||
#
|
||||
# `platform` is the ONLY Kustomization in the graph with no `dependsOn`; every other substituting
|
||||
# consumer (platform-config, apps-wave-0/1/2, apps-vms, oauth2-proxy, headlamp, homepage, crafty,
|
||||
# velero) chains transitively through it. One edge therefore protects the whole graph.
|
||||
#
|
||||
# TRADE-OFF — READ THIS BEFORE YOU CURSE IT: the root is now a GATE. A malformed manifest
|
||||
# ANYWHERE under the root's path (clusters/<env>, which includes clusters/base) fails the root
|
||||
# Kustomization and therefore stops `platform` and EVERYTHING downstream, where before this edge
|
||||
# the rest of the graph would have kept reconciling. That is deliberate and fail-closed: a broken
|
||||
# entrypoint should stop the line, not let consumers render against a half-applied variable set.
|
||||
# No deadlock risk — the root runs `wait: false`, so it reaches Ready on APPLY success and never
|
||||
# health-gates its own children (see clusters/dev/sync.yaml).
|
||||
dependsOn:
|
||||
- name: flux-system # the ROOT Kustomization (sync.yaml) that applies cluster-vars
|
||||
# THE CNI GATES THE TIER. Cilium was split out of ./platform into its own Kustomization
|
||||
# (clusters/base/cilium.yaml, wait:true) so its health has its own Ready condition and its own
|
||||
# suspend valve. This edge preserves the ordering that co-tenancy used to provide implicitly:
|
||||
# nothing in `platform` — and therefore nothing anywhere downstream — reconciles until the CNI
|
||||
# is healthy. It is also revision-gated, which is what makes the prune-safety argument above
|
||||
# deterministic rather than a race.
|
||||
- name: cilium
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
|
||||
@@ -1,57 +0,0 @@
|
||||
---
|
||||
# Flux Kustomization: alert runbooks — reconciles ./runbooks.
|
||||
#
|
||||
# WHAT THIS SERVES. Every alert rule in this repo carries
|
||||
# `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations over 34 distinct names.
|
||||
# Until this tier existed that path returned 404 on the live cluster, so every alert that reached a
|
||||
# human handed them a dead link at the moment they needed help. This Kustomization renders the
|
||||
# markdown under ./runbooks into a ConfigMap and a two-replica nginx that serves it at that exact
|
||||
# prefix. See runbooks/README.md and runbooks/serving/httproute.yaml.
|
||||
#
|
||||
# POST-PLATFORM TIER (dependsOn platform-config), NON-GATING (wait:false) — the same placement and
|
||||
# the same reasoning as homepage/headlamp/oauth2-proxy. It needs the wildcard TLS cert, the shared
|
||||
# Gateway (including the `https-apex` listener) and split-horizon DNS, all of which arrive with
|
||||
# `platform` + `platform-config`; and a leaf web server must never be able to stall the platform
|
||||
# chain, which is the whole lesson of DECISIONS-LOG's "Headlamp gating stall". wait:false guarantees
|
||||
# its health can never gate anything even if something is later mistakenly pointed dependsOn it.
|
||||
#
|
||||
# NO postBuild.substituteFrom — THE ONE DELIBERATE DIVERGENCE FROM EVERY OTHER TIER, AND THE MOST
|
||||
# IMPORTANT LINE IN THIS FILE.
|
||||
# This tier renders the runbook PROSE into a ConfigMap, and runbooks are full of shell: `$POD`,
|
||||
# `$(kubectl ...)`, and — unavoidably, since they document these very annotations — the literal text
|
||||
# `${BASE_DOMAIN}`. Flux's postBuild substitution runs envsubst over the ENTIRE built stream, and it
|
||||
# FAILS OPEN: an undefined `${X}` becomes the empty string while the Kustomization still reports
|
||||
# Ready (DECISIONS-LOG 2026-09-01 — that is how `k8sServiceHost` was emptied and Cilium broke on 2
|
||||
# of 3 dev nodes). Enabled here it would silently mutilate the documentation: every braced expansion
|
||||
# in every command example blanked, in a file nobody re-reads after writing. Worse, the L0 check
|
||||
# `conformance/tests/l0-static/flux-postbuild-substitution` scans only files reachable through
|
||||
# kustomization `resources:`, not `configMapGenerator` inputs, so the damage would be invisible to
|
||||
# the gate as well.
|
||||
# The fix is to have nothing to substitute. The only knob this tier would have wanted is the apex
|
||||
# hostname, and runbooks/serving/httproute.yaml avoids needing it by declaring NO `hostnames:` and
|
||||
# inheriting the `https-apex` listener's own hostname instead. So this tier is
|
||||
# environment-independent by construction and R-ENV-02 ("authored once, promoted dev -> prod") holds
|
||||
# without substitution rather than because of it.
|
||||
# IF A FUTURE KNOB EVER GENUINELY NEEDS SUBSTITUTION HERE: do not add substituteFrom to this
|
||||
# Kustomization. Split the serving manifests into their own substituted tier and leave the content
|
||||
# ConfigMap in an unsubstituted one, or the runbooks lose their shell examples.
|
||||
#
|
||||
# NO SOPS decryption stanza: this tier carries no encrypted Secret (markdown plus five plain
|
||||
# manifests), so — like homepage — neither clusters/dev nor clusters/prod patches it.
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: runbooks
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 10m
|
||||
retryInterval: 2m
|
||||
timeout: 5m
|
||||
prune: true
|
||||
wait: false
|
||||
dependsOn:
|
||||
- name: platform-config # wildcard TLS + Gateway (incl. the https-apex listener) first
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./runbooks
|
||||
@@ -106,11 +106,8 @@ patches:
|
||||
secretRef:
|
||||
name: sops-age
|
||||
# velero (backup controller) has its own post-platform Flux Kustomization (clusters/base/backup.yaml).
|
||||
# It carries BOTH halves of the per-env `cloud` INI credential — velero-cloud.sops.yaml (dev's scoped
|
||||
# MinIO key) and velero-cloud-prod.sops.yaml (prod's R2 token) — so it needs its own SOPS decryption
|
||||
# stanza (same as headlamp/oauth2-proxy/crafty). Which one the Deployment mounts is chosen by
|
||||
# BACKUP_CREDENTIALS_SUFFIX in the cluster-vars patch below, not here. The `sops-age` Secret is
|
||||
# out-of-band (name only).
|
||||
# It carries velero-cloud.sops.yaml (the scoped MinIO `cloud` credential), so it needs its own SOPS
|
||||
# decryption stanza (same as headlamp/oauth2-proxy/crafty). The `sops-age` Secret is out-of-band (name only).
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
@@ -122,23 +119,6 @@ patches:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
# apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret
|
||||
# on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL,
|
||||
# which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize
|
||||
# would hand Flux the raw SOPS document and the VM would mount a disk full of ciphertext — the
|
||||
# failure is silent from the cluster's point of view (the Secret applies fine), so the stanza is
|
||||
# as load-bearing as the file itself. The `sops-age` Secret is out-of-band (name only).
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: apps-vms
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
- patch: |-
|
||||
apiVersion: source.toolkit.fluxcd.io/v1
|
||||
kind: GitRepository
|
||||
@@ -168,155 +148,16 @@ patches:
|
||||
- op: replace
|
||||
path: /data/ACME_CLUSTER_ISSUER
|
||||
value: "letsencrypt-prod"
|
||||
# --- DNS identity (R-DNS-01/02/03) -----------------------------------------------------
|
||||
# DEV STAYS ON THE APEX. Restated explicitly even though it equals the base default, exactly
|
||||
# like the LAN block and LONGHORN_DATA_PATH below: this is what the LIVE cluster is running.
|
||||
# Every one of dev's public DNS records, its `*.home.keithjackson.dev` wildcard cert, its
|
||||
# Keycloak issuer, every OIDC client redirect URI, kubectl SSO, and gitea.home.keithjackson.dev
|
||||
# (which IS this repo's GitOps source) are all on this exact suffix RIGHT NOW. Prod moved to
|
||||
# prod.home.keithjackson.dev instead (2026-09-04) — that alone ended the collision, so dev did
|
||||
# not have to move. Relocating dev to dev.home.keithjackson.dev is phase 2 and is a migration,
|
||||
# not a variable change: infra/PROD-PROVISIONING.md §10.
|
||||
- op: replace
|
||||
path: /data/BASE_DOMAIN
|
||||
value: "home.keithjackson.dev"
|
||||
# ExternalDNS scope. MUST stay the zone apex: dev serves the home.keithjackson.dev APEX itself,
|
||||
# and the Cloudflare provider selects zones with this value (a strict-subdomain filter dropped
|
||||
# the parent zone — DECISIONS-LOG 2026-08). Same value the retired DNS_ZONE knob held.
|
||||
- op: replace
|
||||
path: /data/EXTERNALDNS_DOMAIN_FILTER
|
||||
value: "keithjackson.dev"
|
||||
# DO NOT CHANGE THIS VALUE. Dev's LIVE public records are registered in Cloudflare's TXT
|
||||
# registry under owner `default` (the external-dns chart default, which is what both clusters
|
||||
# silently shared until the 2026-09-04 incident). Dev's external-dns identifies its own records
|
||||
# BY this string: any other value and it stops recognising every record it manages, then
|
||||
# re-creates duplicates beside the orphaned originals. Prod is `prod` — distinct, which is the
|
||||
# whole point. The base default is a non-colliding sentinel, so this line is load-bearing.
|
||||
- op: replace
|
||||
path: /data/EXTERNALDNS_TXT_OWNER_ID
|
||||
value: "default"
|
||||
# --- dev LAN addressing (192.168.1.0/24 on the Proxmox LAN) -----------------------------
|
||||
# EVERY knob in cluster-vars' "LAN addressing" block is restated here EXPLICITLY, even where it
|
||||
# equals the base default. These values are what the LIVE dev cluster is running right now, so
|
||||
# the substituted output is byte-identical to the pre-parameterization hardcoded manifests
|
||||
# (verified 2026-08-31). Never let this block drift from the live cluster.
|
||||
#
|
||||
# Split-horizon DNS target: the Cilium-L2 VIP of the `shared` Gateway on the Proxmox LAN
|
||||
# (= CILIUM_LB_POOL_START, the pool's first address). coredns-custom answers
|
||||
# (base default 10.73.0.240 was the old libvirt VIP). coredns-custom answers
|
||||
# *.home.keithjackson.dev with this so in-cluster OIDC clients reach Keycloak via the Gateway.
|
||||
- op: replace
|
||||
path: /data/GATEWAY_VIP
|
||||
value: "192.168.1.53"
|
||||
# k3s API endpoint Cilium's kube-proxy replacement dials — DEV'S CONTROL-PLANE VIP,
|
||||
# the address kube-vip floats, NOT the primary node .50.
|
||||
# [parity] equals the Ansible inventory's k3s_apiserver_endpoint for dev
|
||||
# (= kube_vip_address, group_vars/all.yml). See clusters/base/cluster-vars.yaml for why.
|
||||
#
|
||||
# *** THE ONE EXCEPTION TO "NEVER LET THIS BLOCK DRIFT FROM THE LIVE CLUSTER" ***
|
||||
# Every other value in this block is what dev is running right now. This one is not:
|
||||
# the live dev cilium-agents dial 192.168.1.50 (k3s-1). Changing it here is a
|
||||
# DELIBERATE, OPERATOR-APPROVED move (2026-09-06) and reconciling it ROLLS THE CILIUM
|
||||
# AGENT DAEMONSET on dev — one node at a time (updateStrategy.maxUnavailable: 1, kept
|
||||
# in parity for exactly this reason). Do not push this to dev-live casually, and not
|
||||
# while dev's DNS migration is mid-flight; the L2 datapath survives an agent restart
|
||||
# but a wrong value here is the 2026-09-01 incident. .49 is in dev's serving-cert SANs
|
||||
# already (k3s_tls_sans) and dev's own kubeconfig validates against it via
|
||||
# tls-server-name, so TLS is not the risk — the risk is reachability, and .49 is on the
|
||||
# same LAN as the nodes and ARP-announced by kube-vip on every server.
|
||||
- op: replace
|
||||
path: /data/K3S_API_HOST
|
||||
value: "192.168.1.49"
|
||||
# Cilium LB IP pool = the tofu reservation cilium_l2_vip_range .53-.60, outside the node IPs
|
||||
# (.50/.51/.52) and outside the DHCP scope. .53 Gateway, .54 unifi, .55 crafty; .56-.60 free.
|
||||
- op: replace
|
||||
path: /data/CILIUM_LB_POOL_START
|
||||
value: "192.168.1.53"
|
||||
- op: replace
|
||||
path: /data/CILIUM_LB_POOL_STOP
|
||||
value: "192.168.1.60"
|
||||
# Pinned L4 VIPs out of that pool (lbipam.cilium.io/ips). Changing either MOVES a live device-
|
||||
# facing endpoint: unifi is the inform/STUN target every adopted UniFi device holds.
|
||||
- op: replace
|
||||
path: /data/UNIFI_L4_VIP
|
||||
value: "192.168.1.54"
|
||||
- op: replace
|
||||
path: /data/CRAFTY_L4_VIP
|
||||
value: "192.168.1.55"
|
||||
# Bulk (external NFS) storage host: the Proxmox host itself, on the SAME LAN as dev's nodes
|
||||
# (192.168.1.40/.41/.42), which already hold an NFS export from it (/mnt/Storage/k8s-volumes).
|
||||
# Restated verbatim rather than inherited, per this overlay's "supply every knob" convention.
|
||||
# Dev is therefore the environment in which the bulk manifests can actually be VALIDATED
|
||||
# end-to-end before prod's network path exists — provided the additive `zfs set sharenfs=`
|
||||
# change described in docs/BULK-NFS-STORAGE.md is applied for the dev node addresses.
|
||||
- op: replace
|
||||
path: /data/BULK_NFS_SERVER
|
||||
value: "192.168.1.9"
|
||||
# Off-cluster backup target: the on-Proxmox MinIO (plain HTTP, path-style) that every CNPG
|
||||
# barmanObjectStore and Velero's backupStorageLocation write to.
|
||||
#
|
||||
# DEV IS DELIBERATELY STAYING ON THIS MinIO while prod goes to Cloudflare R2 (2026-09-07).
|
||||
# Prod could not use it at all — it cannot even route to it — but dev's move is a separate,
|
||||
# operator-gated decision, because R2 would start dev on an EMPTY prefix: Velero would write
|
||||
# a fresh kopia repository, CNPG would take new base backups, and the 30-day Barman window
|
||||
# would rebuild from zero. Dev's DR restore is the one PROVEN recovery path in this repo
|
||||
# (R-STATE-03); resetting its PITR window is not a side effect to take while fixing prod.
|
||||
- op: replace
|
||||
path: /data/S3_BACKUP_ENDPOINT
|
||||
value: "http://192.168.1.9:9000"
|
||||
# DO NOT CHANGE THESE FOUR. They are dev's LIVE object layout, restated verbatim for the same
|
||||
# reason as LONGHORN_DATA_PATH and EXTERNALDNS_TXT_OWNER_ID below/above: they equal the base
|
||||
# defaults today, and a silent drift in base would ORPHAN dev's existing backups — the ones
|
||||
# its proven restore depends on — without any manifest looking wrong.
|
||||
#
|
||||
# `us-east-1` is what dev's Velero has signed every request with since bootstrap; MinIO
|
||||
# ignores the value, but changing it is a needless risk against a live, working backup path.
|
||||
# An EMPTY VELERO_BACKUP_PREFIX means the bucket root, which is where dev's backups already
|
||||
# are — and the Velero chart omits `objectStorage.prefix` entirely when it is empty (verified
|
||||
# against chart 12.1.0), so dev's live BackupStorageLocation is byte-unchanged by this.
|
||||
- op: replace
|
||||
path: /data/S3_BACKUP_REGION
|
||||
value: "us-east-1"
|
||||
- op: replace
|
||||
path: /data/VELERO_BACKUP_BUCKET
|
||||
value: "velero"
|
||||
- op: replace
|
||||
path: /data/VELERO_BACKUP_PREFIX
|
||||
value: ""
|
||||
- op: replace
|
||||
path: /data/CNPG_BACKUP_PATH
|
||||
value: "s3://cnpg"
|
||||
# WHICH CREDENTIAL SIGNS THOSE REQUESTS. Empty suffix = the UNSUFFIXED Secrets
|
||||
# (velero-cloud-credentials, <db>-backup-credentials) — the scoped MinIO keys dev's Velero and
|
||||
# its three CNPG Clusters are running RIGHT NOW, byte-identical to what is on the cluster. Prod
|
||||
# renders "-prod" and reads the Cloudflare R2 token from the parallel `-prod` Secrets instead.
|
||||
#
|
||||
# RESTATED EXPLICITLY even though it equals the base default, for the same reason as
|
||||
# EXTERNALDNS_TXT_OWNER_ID and the four backup-layout knobs above: the four Secrets are SHARED
|
||||
# base manifests, so before this split prod's R2 token landed in the objects DEV reads. Dev
|
||||
# signing MinIO with an R2 key id is 403 on every barman-cloud-wal-archive, and archive_mode=on
|
||||
# turns that into a filling pgdata PVC — the disk-full clock, on the one cluster whose DR
|
||||
# restore is PROVEN (R-STATE-03). This line is what keeps dev on its own credential.
|
||||
#
|
||||
# DEV IS NOT BEING BACKED UP GOING FORWARD and is slated for decommission (operator,
|
||||
# 2026-09-07), so this is not protecting a future backup — it is protecting the EXISTING
|
||||
# backups and the WAL chain until dev is retired. Do not "simplify" it away while dev runs.
|
||||
- op: replace
|
||||
path: /data/BACKUP_CREDENTIALS_SUFFIX
|
||||
value: ""
|
||||
# dev mirrors prod's 3-node topology, so the replicated tier keeps 3 replicas.
|
||||
- op: replace
|
||||
path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
|
||||
value: "3"
|
||||
# DO NOT CHANGE THIS VALUE. The LIVE dev cluster has 18 Longhorn volumes whose replicas are
|
||||
# on disk under /var/lib/longhorn — the dedicated /dev/vdb ext4 disk roles/node_baseline
|
||||
# mounts there. Longhorn addresses a replica by dataPath; pointing defaultDataPath anywhere
|
||||
# else would orphan every one of them (they are not recoverable from the old path by
|
||||
# rescheduling). Restated explicitly even though it equals the base default, exactly like the
|
||||
# LAN block above: this is what the live cluster is running. Prod uses /data/longhorn because
|
||||
# its data partition also carries k3s's data dir — see clusters/prod and PROD-PROVISIONING §2.
|
||||
- op: replace
|
||||
path: /data/LONGHORN_DATA_PATH
|
||||
value: "/var/lib/longhorn"
|
||||
# smaller retention / volumes to keep the dev workstation footprint light.
|
||||
- op: replace
|
||||
path: /data/MONITORING_PROMETHEUS_RETENTION
|
||||
|
||||
@@ -5,255 +5,11 @@
|
||||
# When Phase 7 is opened, `flux bootstrap ... --path=clusters/prod` points the prod cluster's Flux
|
||||
# here. Like dev, this overlay adds NO resources — it ONLY patches cluster-vars with prod-scoped
|
||||
# values (R-ENV-02). Promotion to prod is a change to THESE variables, never a base edit.
|
||||
#
|
||||
# *** PHASE-7 BLOCKER — READ BEFORE BOOTSTRAPPING PROD ***
|
||||
# `clusters/base/platform.yaml` now carries `dependsOn: [flux-system]` (the 2026-09-01 substitution-
|
||||
# race fix). That names the ROOT Kustomization, which clusters/dev supplies as `dev/sync.yaml`
|
||||
# (path ./clusters/dev). THIS overlay has no sync.yaml yet, so as written a prod bootstrap would
|
||||
# leave `platform` stuck on "dependency 'flux-system/flux-system' is not ready" FOREVER and nothing
|
||||
# downstream would reconcile. Phase 7 MUST add `clusters/prod/sync.yaml` — a copy of dev/sync.yaml
|
||||
# with `path: ./clusters/prod` — as part of opening this overlay. (Adding it now is deliberately NOT
|
||||
# done here: prod is design-only and its root object is the prod bring-up owner's call.)
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
resources:
|
||||
- ../base
|
||||
- sync.yaml # self-managing root Kustomization -> ./clusters/prod (REQUIRED: see below)
|
||||
patches:
|
||||
# --- SOPS decryption (operator decision 2026-09-01: prod uses the SAME age key as dev) ---
|
||||
# Mirrors clusters/dev exactly, one stanza per Kustomization that carries encrypted Secrets.
|
||||
# The `sops-age` Secret is created OUT OF BAND on the prod cluster
|
||||
# (bootstrap/sops/install-age-key.sh) — referenced by NAME only, never inlined.
|
||||
# Add a stanza here whenever `render.py --emit-waves` introduces a new deepest wave.
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: platform
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: apps-wave-0
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: apps-wave-1
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: apps-wave-2
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: oauth2-proxy
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: headlamp
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: crafty
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: velero
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
# apps-vms (the KubeVirt VM tier, clusters/base/apps-vms.yaml) gained its FIRST encrypted Secret
|
||||
# on 2026-09-14: apps/vms/unifi/unifi-uos-installer.sops.yaml, the operator's UOS installer URL,
|
||||
# which 30-virtualmachine.yaml mounts into the guest as a disk. Without this stanza kustomize
|
||||
# would hand Flux the raw SOPS document and the VM would mount a disk full of ciphertext — the
|
||||
# failure is silent from the cluster's point of view (the Secret applies fine), so the stanza is
|
||||
# as load-bearing as the file itself. Mirrors clusters/dev. `sops-age` is out-of-band (name only).
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: apps-vms
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
# --- GitOps source (operator decision 2026-09-01: prod tracks the SAME BRANCH as dev) ---
|
||||
#
|
||||
# ############################################################################################
|
||||
# ### ###
|
||||
# ### *** T E M P O R A R Y B O O T S T R A P S O U R C E — NOT PERMANENT *** ###
|
||||
# ### ###
|
||||
# ### The url below is a Gitea container running ON THE OPERATOR'S WORKSTATION, outside ###
|
||||
# ### any cluster. It exists ONLY to get prod's first reconcile off the ground. It MUST ###
|
||||
# ### be cut over to the in-cluster URL once Gitea runs on prod. If you are reading this ###
|
||||
# ### and prod has been serving traffic for months, THE CUTOVER WAS NEVER DONE — that is ###
|
||||
# ### a live single point of failure, not a settled design. See "CUTOVER" below. ###
|
||||
# ### ###
|
||||
# ############################################################################################
|
||||
#
|
||||
# WHY THE IN-CLUSTER URL CANNOT BE USED YET (blocker found 2026-09-01, still true; the
|
||||
# unusable value was `https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git`).
|
||||
# Two independent reasons, both consequences of the split-horizon design:
|
||||
# 1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER. Prod syncing from it means prod's
|
||||
# GitOps source lives inside the very cluster slated for decommission. Gitea must move to
|
||||
# prod (or off-cluster) BEFORE dev is retired, or prod loses its source of truth.
|
||||
# 2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml
|
||||
# answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP}. On prod that is
|
||||
# 192.168.2.20 — PROD's own gateway, where no Gitea is listening. Prod's Flux would resolve
|
||||
# gitea.home.keithjackson.dev to itself and never reach a repository.
|
||||
#
|
||||
# WHAT THE TEMPORARY SOURCE IS (stood up 2026-09-04 on explicit operator direction):
|
||||
# bootstrap/gitea/bootstrap-gitea.sh with HOST_NAT_IP=192.168.2.132 — a podman container on the
|
||||
# operator's workstation, Gitea 1.22.6, listening on 192.168.2.132:3000. All three prod nodes
|
||||
# reach it DIRECTLY on the LAN: no libvirt NAT hop (that network is retired) and no firewall
|
||||
# change was needed. This is the same "local bootstrap source" pattern dev itself used before
|
||||
# its Phase-6 cutover (docs/05 "Source-of-truth").
|
||||
#
|
||||
# PLAIN HTTP, AND WHY THAT IS ACCEPTED HERE: the bootstrap Gitea has no TLS, so there is no cert
|
||||
# for Flux to trust and no caCertSecretRef to wire. Traffic is LAN-local between prod nodes and
|
||||
# the workstation, the repo carries NO plaintext secrets (everything sensitive is SOPS-encrypted,
|
||||
# .sops.yaml), and the window is the bring-up only. Do NOT extend this posture past cutover.
|
||||
#
|
||||
# NO secretRef, DELIBERATELY: the repo is PUBLIC on the bootstrap Gitea, so anonymous clone
|
||||
# works and Flux needs no git credentials. VERIFIED 2026-09-04 by unauthenticated request:
|
||||
# curl http://192.168.2.132:3000/api/v1/repos/Home-Network/home-network-k3s-impl -> "private":false
|
||||
# git ls-remote http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git -> refs/heads/main
|
||||
# If the GitRepository ever reports an auth failure that looks like a network problem, RE-CHECK
|
||||
# THAT FIRST: a private repo here fails exactly that way.
|
||||
#
|
||||
# *** CUTOVER — the exit condition, mirroring dev's Phase-6 cutover (docs/05) ***
|
||||
# WHEN: Gitea runs on prod (reconciled in-cluster) and answers gitea.home.keithjackson.dev
|
||||
# through prod's own Gateway VIP with a publicly-trusted cert.
|
||||
# DO: 1. push/mirror this repo into the prod in-cluster Gitea;
|
||||
# 2. change the url below to
|
||||
# https://gitea.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git
|
||||
# and DELETE this whole TEMPORARY block (dev's equivalent stanza, ~15 lines above in
|
||||
# clusters/dev/kustomization.yaml, is what "done" looks like);
|
||||
# 3. confirm `flux get sources git -n flux-system` reconciles the new revision;
|
||||
# 4. only THEN stop the workstation container. Keep it stopped-not-deleted as recovery,
|
||||
# exactly as dev retained its bootstrap Gitea.
|
||||
# UNTIL THEN: prod's reconciliation depends on a container on a workstation that may be
|
||||
# powered off. Flux keeps the last-reconciled state running, but NO new revision lands while
|
||||
# it is down. Tracked as an open blocker in infra/PROD-PROVISIONING.md §7.
|
||||
#
|
||||
# The BRANCH below is settled and correct: prod tracks `main`, the same branch as dev.
|
||||
# CONSEQUENCE, recorded deliberately: once prod is live, EVERY MERGE TO main RECONCILES BOTH
|
||||
# CLUSTERS. There is no promotion step and no soak window — dev and prod move together.
|
||||
# NOTE for bring-up: bootstrap-gitea.sh pushes the operator's CURRENT HEAD to the bootstrap
|
||||
# Gitea's `main`, whatever branch that is. Before bootstrapping prod, confirm that remote `main`
|
||||
# is the revision you intend prod to run (`git ls-remote` above, against local `main`).
|
||||
- patch: |-
|
||||
apiVersion: source.toolkit.fluxcd.io/v1
|
||||
kind: GitRepository
|
||||
metadata:
|
||||
name: flux-system
|
||||
namespace: flux-system
|
||||
spec:
|
||||
# CUTOVER DONE 2026-09-06 (operator direction: "move the repo to prod"). Prod now reconciles
|
||||
# from PROD'S OWN in-cluster Gitea. The workstation bootstrap container
|
||||
# (http://192.168.2.132:3000) is retired as prod's source — it was a podman container on the
|
||||
# operator's workstation holding a DHCP lease on WIFI, i.e. production's source of truth was a
|
||||
# laptop. It is kept only as a recovery path (bootstrap/gitea/bootstrap-gitea.sh).
|
||||
#
|
||||
# THIS LINE IS SELF-REFERENTIAL AND THAT IS THE WHOLE DIFFICULTY. The root Kustomization
|
||||
# renders ./clusters/prod, which contains this patch — so a live `kubectl patch` of
|
||||
# GitRepository.spec.url is REVERTED on the next reconcile unless the authored value moves
|
||||
# too. The cutover therefore had to be: (1) push the repo to prod's Gitea, (2) fast-forward
|
||||
# its `main` to the exact commit prod was already running so the source swap changed nothing
|
||||
# else, (3) live-patch the url to open a window, (4) land THIS commit through that window so
|
||||
# the authored value agrees. Dev hit the identical trap on 2026-09-05
|
||||
# (infra/PROD-PROVISIONING.md §10a "D0/D1 as executed").
|
||||
#
|
||||
# Verified before the switch: a throwaway pod INSIDE prod resolved this host through
|
||||
# in-cluster CoreDNS to the Gateway VIP and cloned over the public LE cert with no CA
|
||||
# supplied. The workstation being able to reach it proves nothing about pod egress.
|
||||
# MOVED WITH THE APEX FLIP (2026-09-13) — and it had to move in the SAME fetch.
|
||||
# This URL is a hardcoded literal, NOT templated on ${BASE_DOMAIN}, so the flip did not
|
||||
# carry it. But the flip DOES rename prod's own HTTPRoute hostnames, so:
|
||||
# - external-dns (policy=sync) deletes gitea.prod.home.keithjackson.dev, and
|
||||
# - platform/coredns/coredns-custom.yaml answers the whole ${BASE_DOMAIN} zone with
|
||||
# ${GATEWAY_VIP}, so in-cluster resolution moves from prod.home. to home.
|
||||
# Leaving this on the old name would therefore have severed prod from its OWN GitOps
|
||||
# source moments after the flip reconciled — self-reference, the same class of trap as
|
||||
# patching this url live and having the root Kustomization revert it. The fix has to land
|
||||
# THROUGH THE OLD SOURCE, which is why it is in the same push as the flip rather than
|
||||
# after it.
|
||||
# FOLLOW-UP worth doing: point this at the in-cluster Service
|
||||
# (gitea-http.gitea.svc.cluster.local) so prod's GitOps stops depending on public DNS and
|
||||
# on its own Gateway at all. Deferred: it needs the TLS/credential path rethought.
|
||||
# IN-CLUSTER SERVICE, NOT A PUBLIC HOSTNAME (2026-09-13). This is the durable fix for a
|
||||
# deadlock the apex flip walked straight into, and it is worth stating precisely.
|
||||
#
|
||||
# The url used to be a public name. The flip renamed prod's HTTPRoute hostnames, so the
|
||||
# name this GitRepository pointed at stopped serving — but the HTTPRoute that WOULD have
|
||||
# served the new name lives in a DOWNSTREAM tier, which cannot reconcile while the source
|
||||
# is broken. Source down -> routes never rename -> source stays down. Observed live:
|
||||
# `srcReady=False`, "unable to list remote for https://gitea.home.keithjackson.dev".
|
||||
#
|
||||
# Pointing at the Service breaks the cycle permanently: prod's GitOps no longer depends on
|
||||
# public DNS, on Cloudflare, on its own Gateway, or on ${BASE_DOMAIN} at all. A future
|
||||
# domain change cannot sever prod from its own source again.
|
||||
#
|
||||
# Plain http is deliberate and safe here: it is cluster-internal traffic to a ClusterIP,
|
||||
# TLS terminates at the Gateway for external clients only, the GitRepository has no
|
||||
# secretRef (the repo is cloned anonymously), and every secret in the tree is SOPS-encrypted
|
||||
# at rest regardless of transport.
|
||||
url: http://gitea.gitea.svc.cluster.local:3000/Home-Network/home-network-k3s-impl.git
|
||||
ref:
|
||||
branch: main
|
||||
- target:
|
||||
kind: ConfigMap
|
||||
name: cluster-vars
|
||||
@@ -265,97 +21,9 @@ patches:
|
||||
- op: replace
|
||||
path: /data/ACME_CLUSTER_ISSUER
|
||||
value: "letsencrypt-prod"
|
||||
# ==========================================================================================
|
||||
# PROD DNS IDENTITY (R-DNS-01/02/03) — the fix for the 2026-09-04 incident.
|
||||
# ==========================================================================================
|
||||
# PROD SERVES ITS OWN SUBDOMAIN. Until 2026-09-04 every hostname was HARDCODED to
|
||||
# home.keithjackson.dev in the base manifests, so prod was authored to serve the EXACT SAME
|
||||
# public names as the live dev cluster out of the SAME Cloudflare zone. Two consequences, one
|
||||
# of which already happened:
|
||||
# 1. HAPPENED: prod's external-dns (policy: sync, owner id `default` — the same implicit
|
||||
# default dev had) concluded it owned dev's records and DELETED EVERY ONE of them.
|
||||
# 2. QUEUED BEHIND IT: platform-config would have requested a `*.home.keithjackson.dev`
|
||||
# wildcard from letsencrypt-prod for a cert dev already holds, and both clusters would
|
||||
# have raced each other's `_acme-challenge` TXT records on that one zone.
|
||||
# Moving prod to its own suffix dissolves both: the two clusters can no longer name the same
|
||||
# record. This is what makes ACME_CLUSTER_ISSUER: letsencrypt-prod above safe here.
|
||||
#
|
||||
# dev -> home.keithjackson.dev (UNCHANGED — dev is live; phase 2 moves it)
|
||||
# prod -> prod.home.keithjackson.dev (this knob)
|
||||
#
|
||||
# CONSEQUENCE — prod's hostnames are NOT the apex names. Prod's Gitea is
|
||||
# gitea.prod.home.keithjackson.dev, its Keycloak keycloak.prod.home.keithjackson.dev, and so
|
||||
# on; the wildcard cert becomes `*.prod.home.keithjackson.dev`. The apex CNAMEs that make
|
||||
# `<svc>.home.keithjackson.dev` point at prod are PHASE 2, deliberately not done here — they
|
||||
# would take traffic off the live dev cluster. infra/PROD-PROVISIONING.md §10.
|
||||
#
|
||||
# [parity] MUST equal k3s_base_domain in infra/ansible/inventory/group_vars/k3s_prod/vars.yml.
|
||||
# The kube-apiserver's oidc-issuer-url is an INSTALL-TIME k3s flag, not a manifest, so it is
|
||||
# set Ansible-side from that variable and cannot read this one. If they disagree, kubectl SSO
|
||||
# trusts a different Keycloak than the one prod actually runs.
|
||||
- op: replace
|
||||
path: /data/BASE_DOMAIN
|
||||
# APEX AS OF 2026-09-13. Prod now serves home.keithjackson.dev directly; dev has been
|
||||
# decommissioned and released the name. Prerequisites completed and VERIFIED before this
|
||||
# flip, in this order:
|
||||
# 1. dev's 18 A records released by its own external-dns (policy=sync, sources deleted)
|
||||
# 2. dev's 18 LEGACY UNPREFIXED TXT registry entries deleted from Cloudflare by hand —
|
||||
# external-dns v0.21.0 removed only the `a-` prefixed form and then reported "All
|
||||
# records are already up to date", so it would NEVER have cleared them. They carried
|
||||
# `owner=default`, and prod runs `owner=prod`, so prod could not have claimed these
|
||||
# names while they existed. Confirmed gone at the AUTHORITY (betty.ns.cloudflare.com);
|
||||
# 1.1.1.1 served stale cache for minutes afterwards and must not be trusted here.
|
||||
# 3. dev's `*.home.keithjackson.dev` + apex Certificate deleted, so it stops renewing.
|
||||
# It was issued ~18d ago, so prod's request is #1 this week against LE's duplicate
|
||||
# limit of 5 per identical name set per week.
|
||||
# NOTE the zone has wildcards `*.keithjackson.dev` AND `*.home.keithjackson.dev` pointing
|
||||
# at the public IP 136.51.115.252 (legacy non-k3s prod). Dev's specific records were
|
||||
# SHADOWING that wildcard; prod's new specific records shadow it again. The legacy stack
|
||||
# lives on `*.keithjackson.dev` and is untouched by any of this.
|
||||
value: "home.keithjackson.dev"
|
||||
# SAFETY NET #2 (independent of the owner id below): narrow ExternalDNS's name scope to prod's
|
||||
# OWN subtree, so prod cannot create, update or delete any record outside it no matter what the
|
||||
# TXT registry says. VERIFY ON FIRST RECONCILE — the Cloudflare provider uses this value for
|
||||
# ZONE selection too, and on the version in use in 2026-08 a strict-subdomain filter dropped
|
||||
# the parent zone ("no hosted zone matching record DNS Name"). If prod's external-dns logs no
|
||||
# matching zone, it will create NOTHING — fail-closed, and it can still never delete dev's
|
||||
# records. Fallback in that case: set this to "keithjackson.dev" (dev's value) and rely on
|
||||
# EXTERNALDNS_TXT_OWNER_ID below, which is sufficient on its own.
|
||||
- op: replace
|
||||
path: /data/EXTERNALDNS_DOMAIN_FILTER
|
||||
# MUST be the Cloudflare ZONE, not prod's subdomain. FINDING F-DNS1 (DECISIONS-LOG,
|
||||
# 2026-08-17) proved a subdomain filter DROPS THE PARENT ZONE: ExternalDNS generated the
|
||||
# record then skipped it with "no hosted zone matching record DNS Name", so prod would
|
||||
# publish NOTHING. Isolation from dev does NOT come from this filter — it comes from
|
||||
# (1) prod only requesting *.prod.home.keithjackson.dev hostnames, and
|
||||
# (2) txtOwnerId: prod, so prod's registry never claims dev's `default`-owned records.
|
||||
# That pairing is what the 2026-09-04 incident needed and this filter cannot provide.
|
||||
value: "keithjackson.dev"
|
||||
# SAFETY NET #1 — THE ACTUAL ROOT CAUSE OF THE INCIDENT. Both clusters left this unset and so
|
||||
# both inherited the chart default `default`; prod then treated dev's TXT-registry entries as
|
||||
# its own and, under policy: sync, deleted the records behind them. This value MUST differ from
|
||||
# dev's (`default`, pinned in clusters/dev — dev's live records are registered under it and
|
||||
# cannot be renamed without orphaning them). `prod` is distinct, which is the entire fix.
|
||||
- op: replace
|
||||
path: /data/EXTERNALDNS_TXT_OWNER_ID
|
||||
value: "prod"
|
||||
- op: replace
|
||||
path: /data/LONGHORN_DEFAULT_REPLICA_COUNT
|
||||
value: "3"
|
||||
# Longhorn's on-node replica directory: a SUBDIRECTORY of the claimed data partition, not
|
||||
# the mount root. Prod's ~424 GB partition (LABEL=k3s-data) is mounted at the neutral /data
|
||||
# and serves TWO tenants — /data/longhorn here, and /data/k3s for k3s's entire data dir
|
||||
# (embedded etcd + the containerd image store), which had to move off the 11.8 GiB /var:
|
||||
# measured on dev, containerd alone is 21-25 GB per node, so prod would have filled /var on
|
||||
# its first image pulls and etcd could not have written its WAL. infra/PROD-PROVISIONING.md §2.
|
||||
#
|
||||
# [parity] MUST equal node_baseline_longhorn_data_path in
|
||||
# infra/ansible/inventory/group_vars/k3s_prod/vars.yml — Ansible creates and mounts the
|
||||
# directory, this tells Longhorn to use it. If they disagree, Longhorn writes replicas to a
|
||||
# path on the OS filesystem instead of the data partition and /var fills up.
|
||||
- op: replace
|
||||
path: /data/LONGHORN_DATA_PATH
|
||||
value: "/data/longhorn"
|
||||
# longer retention / larger volumes on prod NVMe.
|
||||
- op: replace
|
||||
path: /data/MONITORING_PROMETHEUS_RETENTION
|
||||
@@ -366,16 +34,8 @@ patches:
|
||||
# Size-cap below the 50Gi PVC (headroom for WAL + head + compaction scratch): prod may hit the
|
||||
# size cap before 30d if series volume is high, keeping the disk from filling either way.
|
||||
- op: replace
|
||||
# LOWERED 45GiB -> 36GiB on 2026-09-13. 45 of 50Gi left only 5 GiB, which is not headroom for
|
||||
# what the comment above already names: retentionSize counts ONLY persisted blocks, not the WAL,
|
||||
# not the head's in-flight segments, and not compaction scratch, and a large compaction can need
|
||||
# roughly the size of the blocks being merged. At 90% one bad compaction is ENOSPC, and ENOSPC on
|
||||
# /prometheus wedges every WAL commit: scrapes keep succeeding, nothing is stored, and every
|
||||
# absent()/==0 alert in the platform fires falsely at once. Costs NO history: the TSDB is 11.2 GiB
|
||||
# at 30d today, so 36GiB is ~3x current and 30d stays the binding limit. To keep more history,
|
||||
# grow the PVC rather than the fraction of it.
|
||||
path: /data/MONITORING_PROMETHEUS_RETENTION_SIZE
|
||||
value: "36GiB"
|
||||
value: "45GiB"
|
||||
- op: replace
|
||||
path: /data/MONITORING_LOKI_PVC_SIZE
|
||||
value: "50Gi"
|
||||
@@ -389,239 +49,8 @@ patches:
|
||||
- op: replace
|
||||
path: /data/VELERO_RETENTION
|
||||
value: "720h"
|
||||
# ==========================================================================================
|
||||
# PROD LAN ADDRESSING (192.168.2.0/24). Nodes: k3s-prod-1/2/3 = .10 / .11 / .12.
|
||||
# ==========================================================================================
|
||||
# These knobs were HARDCODED to the dev LAN inside base manifests until 2026-08-31; prod would
|
||||
# have come up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN. They are knobs now, and
|
||||
# this overlay supplies every one of them. Anything the operator has NOT yet decided stays an
|
||||
# INVALID `TBD-PROD-*` placeholder on purpose: base defaults are the DEV values, so an OMITTED
|
||||
# op means prod silently inherits a dev address — far worse than a failed apply. `TBD-PROD-*`
|
||||
# is not a valid IP/URL, so Cilium CRD validation / LB-IPAM / CNPG / Velero reject it LOUDLY.
|
||||
#
|
||||
# ------------------------------------------------------------------------------------------
|
||||
# ADDRESS PLAN (operator, 2026-08-31) — see infra/PROD-PROVISIONING.md §4 for the full table.
|
||||
# ------------------------------------------------------------------------------------------
|
||||
# nodes 192.168.2.10 / .11 / .12 (Ansible inventory)
|
||||
# control-plane VIP 192.168.2.9 (Ansible group_vars/k3s_prod — NOT here)
|
||||
# ingress Gateway VIP 192.168.2.20 (GATEWAY_VIP, below)
|
||||
# Cilium LB pool 192.168.2.20 - .30 (CILIUM_LB_POOL_*, below)
|
||||
#
|
||||
# CONTAINMENT INVARIANT SATISFIED: GATEWAY_VIP (.20) IS CILIUM_LB_POOL_START, mirroring dev
|
||||
# (.53 of .53-.60). This matters because the `shared` Gateway (platform/gateway/gateway.yaml)
|
||||
# pins NOTHING — no `lbipam.cilium.io/ips`, no spec.addresses (unlike platform/crafty and
|
||||
# apps/vms/unifi, which DO pin) — so Cilium LB-IPAM assigns it the FIRST FREE address of a
|
||||
# matching pool block. Had GATEWAY_VIP stayed outside every pool (the original .21-.30
|
||||
# proposal), the Gateway would have come up on .21 while split-horizon coredns-custom kept
|
||||
# answering *.home.keithjackson.dev with .20 — an address nothing announces — blackholing
|
||||
# in-cluster hairpin traffic (OIDC clients -> Keycloak/Grafana) while looking healthy.
|
||||
# KEEP GATEWAY_VIP == CILIUM_LB_POOL_START when changing either.
|
||||
# The control-plane VIP (.9) is deliberately OUTSIDE this pool: kube-vip floats it, and
|
||||
# LB-IPAM must never be able to hand it to a Service.
|
||||
- op: replace
|
||||
path: /data/GATEWAY_VIP
|
||||
value: "192.168.2.20"
|
||||
# Cilium LB IP pool, operator-supplied: 192.168.2.20-.30 (START == GATEWAY_VIP, see above).
|
||||
# Mirror this range into the prod tofu reservation (cilium_l2_vip_range) if prod ever gets one.
|
||||
- op: replace
|
||||
path: /data/CILIUM_LB_POOL_START
|
||||
value: "192.168.2.20"
|
||||
- op: replace
|
||||
path: /data/CILIUM_LB_POOL_STOP
|
||||
value: "192.168.2.30"
|
||||
# k3s API endpoint Cilium's kube-proxy replacement dials. This is the prod CONTROL-PLANE VIP
|
||||
# (kube-vip), NOT .20 (the ingress VIP) and NOT the single node address .10.
|
||||
#
|
||||
# *** DECIDED 2026-09-06 by the operator: the VIP, as intended. *** (This block previously
|
||||
# read "Not yet decided" while shipping .10 — the founder's own address — and that is what
|
||||
# the LIVE prod cluster is running: all three cilium-agents carry
|
||||
# KUBERNETES_SERVICE_HOST=192.168.2.10 while kube-vip holds .9, verified 2026-09-06. On a
|
||||
# 3-node HA cluster that makes the CNI's control-plane path depend on k3s-prod-1: lose it and
|
||||
# the datapath keeps running off Cilium's cache while no service or policy update lands, with
|
||||
# nothing looking broken. The VIP is a virtual address that reaches the cluster wherever the
|
||||
# control plane is, which is the whole reason kube-vip exists here.)
|
||||
#
|
||||
# It is owned Ansible-side by **k3s_apiserver_endpoint** (group_vars/all.yml, derived from
|
||||
# this environment's kube_vip_address = 192.168.2.9 in group_vars/k3s_prod/vars.yml) and MUST
|
||||
# equal that value — [parity] contract in platform/cilium/helmrelease.yaml. NOT k3s_api_host:
|
||||
# that variable still exists and still holds the FOUNDER's .10, because the cold-bootstrap
|
||||
# `--server` join needs a host that is up before any VIP is floated. Two addresses, two
|
||||
# variables, deliberately.
|
||||
#
|
||||
# .9 sits OUTSIDE the Cilium LB pool (.20-.30) and is not a node address, so kube-vip and
|
||||
# LB-IPAM can never be handed the same IP — asserted by
|
||||
# conformance/tests/l0-static/ha-control-plane-as-code.
|
||||
- op: replace
|
||||
path: /data/K3S_API_HOST
|
||||
value: "192.168.2.9"
|
||||
# --- still UNDECIDED -> loud placeholders (must be real before `flux bootstrap`) ------------
|
||||
# Pinned L4 VIPs, both carved out of the pool by lbipam.cilium.io/ips. Not yet assigned for
|
||||
# prod. Leaving the dev literals in place would hand a prod LoadBalancer a dev-LAN address, so
|
||||
# they stay placeholders; whatever is chosen MUST fall inside [POOL_START, POOL_STOP] and not
|
||||
# collide with the Gateway VIP or the node IPs (.10/.11/.12).
|
||||
- op: replace
|
||||
path: /data/UNIFI_L4_VIP
|
||||
value: "192.168.2.21"
|
||||
- op: replace
|
||||
path: /data/CRAFTY_L4_VIP
|
||||
value: "192.168.2.22"
|
||||
# PiHole's DNS listener VIP — udp/53 + tcp/53 for LAN clients (apps/wave-0/pihole-dns.yaml).
|
||||
# 192.168.2.23 is the NEXT FREE address in the prod pool 192.168.2.20-.30, verified live
|
||||
# 2026-09-14: the CiliumLoadBalancerIPPool `default-pool` reported IPsTotal 11 / IPsUsed 3 /
|
||||
# IPsAvailable 8, and the only three allocations are .20 (cilium-gateway-shared), .21
|
||||
# (unifi-l4-tcp/-udp) and .22 (crafty-l4-tcp/-udp). ARP-probed from a LAN host in the same
|
||||
# broadcast domain (arping 192.168.2.23, 3 probes, 0 responses), so nothing off-cluster holds
|
||||
# it either, and it is inside the pool block the operator already asserted sits outside the
|
||||
# SOHO router's DHCP scope.
|
||||
- op: replace
|
||||
path: /data/PIHOLE_DNS_VIP
|
||||
value: "192.168.2.23"
|
||||
# The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns —
|
||||
# NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path.
|
||||
# It scopes the ingress allow for PiHole's DNS listener to LAN clients only.
|
||||
- op: replace
|
||||
path: /data/LAN_CIDR
|
||||
value: "192.168.2.0/24"
|
||||
# ------------------------------------------------------------------------------------------
|
||||
# BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
|
||||
# ------------------------------------------------------------------------------------------
|
||||
# Backs every `class: bulk` volume: the 8.33 T media library (Storage/plex — plex, sonarr,
|
||||
# radarr) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
|
||||
#
|
||||
# THE STORAGE HOST IS NOT REACHABLE FROM PROD AT ALL RIGHT NOW. It has one interface,
|
||||
# vmbr0 192.168.1.9/24, on pfSense's LAN; prod's nodes are on 192.168.2.0/24, pfSense's WAN.
|
||||
# 22 and 8006 answer from prod only because pfSense port-forwards them via 192.168.2.102;
|
||||
# 2049 and 111 do not. Opening that path is an OPERATOR decision, and the two candidate
|
||||
# outcomes give DIFFERENT values here:
|
||||
# port-forward 192.168.2.102:2049 -> 192.168.1.9:2049 => "192.168.2.102"
|
||||
# a WAN->LAN route to 192.168.1.0/24 => "192.168.1.9"
|
||||
# Guessing either one authors a prod cluster that mounts nothing, or — worse, if the guess
|
||||
# ever became reachable — mounts something unintended. So it stays a placeholder.
|
||||
#
|
||||
# WHICH PORTS THE DECISION MUST OPEN: exactly ONE, tcp/2049. The PVs mount nfsvers=4.1, and
|
||||
# NFSv4 carries mount and lock in-protocol — no rpcbind/111, no mountd, no statd. Choosing v4
|
||||
# over v3 was in part this decision's cost: v3 would have needed four ports and three of them
|
||||
# on ephemeral numbers.
|
||||
#
|
||||
# WHAT THE EXPORT MUST ALLOW IS ALREADY SETTLED, and it is the same under BOTH outcomes.
|
||||
# Measured 2026-09-07 from this workstation (192.168.2.132) through the existing port-forward:
|
||||
# the storage host logged the connection as coming FROM 192.168.2.132, not from a pfSense LAN
|
||||
# address (`ss -tn` peer 192.168.2.132, and the host's only route off-LAN is its default via
|
||||
# 192.168.1.1). pfSense DNATs WITHOUT source NAT, so the NFS server will see prod's REAL node
|
||||
# addresses either way. The additive `zfs set sharenfs=` values are in docs/BULK-NFS-STORAGE.md.
|
||||
#
|
||||
# *** BLAST RADIUS — READ BEFORE MERGING THIS TO A BRANCH PROD RECONCILES. ***
|
||||
# A PersistentVolume's spec.nfs.server is an unvalidated string: the API server ACCEPTS this
|
||||
# placeholder, the PV is created and the PVC BINDS to it (binding is pure control-plane). The
|
||||
# failure surfaces only when a pod mounts — MountVolume.SetUp "Failed to resolve server
|
||||
# TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr and nextcloud would sit in
|
||||
# ContainerCreating on prod. Unlike the LAN knobs above, this placeholder does NOT fail at
|
||||
# apply time and there is no CRD validation to lean on. Set this value BEFORE prod reconciles
|
||||
# a revision containing the bulk volumes; the dev overlay renders a working value, which is
|
||||
# where these manifests are meant to be proven first.
|
||||
- op: replace
|
||||
path: /data/BULK_NFS_SERVER
|
||||
# 192.168.2.102 is pfSense's WAN address, which DNATs tcp/2049 through to the Proxmox
|
||||
# host's REAL address 192.168.1.9 on pfSense's LAN. Prod sits on the WAN side and cannot
|
||||
# reach 192.168.1.9 directly, so the cluster mounts the NAT address. Dev is on the LAN and
|
||||
# keeps the base value 192.168.1.9.
|
||||
#
|
||||
# PROVEN from k3s-prod-1 on 2026-09-12, not inferred:
|
||||
# - mount of 192.168.2.102:/mnt/Storage/plex negotiated nfs4 vers=4.1 proto=tcp
|
||||
# - a write as uid 1000 succeeded. Tested as 1000 and NOT root deliberately: the export
|
||||
# carries no_root_squash, so a root write would have passed and proved nothing, while
|
||||
# LSIO images default to PUID 911 and would have gone Ready then failed every import
|
||||
# with EACCES. The tree is 0755 1000:3000, so uid 1000 is the only writer.
|
||||
# - ln() across downloads/ -> tv/ returned link count 2 (same inode), so the arrs get
|
||||
# real atomic moves instead of silently degrading to copy-then-delete on EXDEV.
|
||||
# pfSense DNATs WITHOUT source NAT (the host logs prod's own node addresses as the peer),
|
||||
# so the ZFS export ACL names 192.168.2.10/.11/.12 directly rather than one translated IP.
|
||||
value: "192.168.2.102"
|
||||
# ==========================================================================================
|
||||
# OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05; bucket + token 2026-09-07).
|
||||
# ==========================================================================================
|
||||
# Dev's versitygw/MinIO (192.168.1.9:9000) sits on the DEV side of the network and is NOT an
|
||||
# option for prod, for two independent reasons:
|
||||
# 1. UNREACHABLE. Prod is on the SOHO network 192.168.2.0/24 — pfSense's WAN — while that
|
||||
# MinIO is inside pfSense's LAN. Measured from a pod INSIDE prod (the workstation proves
|
||||
# nothing about pod egress): `curl http://192.168.1.9:9000/minio/health/live` → connection
|
||||
# timed out after 15s. Reaching it would need a WAN→LAN rule that does not exist.
|
||||
# 2. NOT OFF-CLUSTER DR. It runs on the same Proxmox host as dev's VMs, so it already shares
|
||||
# a physical failure domain with dev. R2 is genuinely off-site, and its free egress means
|
||||
# restore REHEARSALS cost nothing — R-STATE-03 has no financial disincentive to stay proven.
|
||||
#
|
||||
# THIS UNBLOCKS AN ACTIVE INCIDENT, not just a roadmap item. All three prod CNPG Clusters ship
|
||||
# `archive_mode=on` with `endpointURL: TBD-PROD-S3-BACKUP-ENDPOINT`, so barman-cloud-wal-archive
|
||||
# exits 4 on every segment and Postgres — correctly — REFUSES TO RECYCLE WAL. Verified live
|
||||
# 2026-09-07: keycloak-db / nextcloud-db / semaphore-db all report
|
||||
# ContinuousArchiving=False/ContinuousArchivingFailing, and keycloak-db's 5 GiB PVC is at
|
||||
# 579 MiB used / 4360 MiB free and climbing monotonically. A disk-full PANIC there takes the
|
||||
# cluster's SSO database down, and with it kubectl OIDC, Grafana, Headlamp, Gitea and
|
||||
# forward-auth. Fixing the endpoint lets Postgres drain the whole backlog.
|
||||
#
|
||||
# ENDPOINT IS ACCOUNT-SCOPED, so it stays a prod-overlay value even though it is not a LAN
|
||||
# address: this is the operator's Cloudflare account id, confirmed from inside a prod pod
|
||||
# (`head-bucket` on k3s-prod-backup → BucketRegion WNAM; PUT and DELETE both succeed).
|
||||
# It is NOT put in base, because base is dev's live MinIO and dev is deliberately STAYING
|
||||
# there — moving dev to R2 would reset its PITR window and orphan the backups its PROVEN
|
||||
# restore depends on. Two environments, two targets, isolated by URL *and* by bucket.
|
||||
- op: replace
|
||||
path: /data/S3_BACKUP_ENDPOINT
|
||||
value: "https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com"
|
||||
# R2 requires the region string to be exactly `auto`. (`head-bucket` answers
|
||||
# `{"BucketRegion":"WNAM"}` — that is a PHYSICAL LOCATION HINT, not a signing region; using it
|
||||
# here produces AuthorizationHeaderMalformed.) Dev keeps the base `us-east-1` its live MinIO
|
||||
# has always been signed with.
|
||||
- op: replace
|
||||
path: /data/S3_BACKUP_REGION
|
||||
value: "auto"
|
||||
# ------------------------------------------------------------------------------------------
|
||||
# OBJECT TENANCY INSIDE THE ONE BUCKET. The operator created a SINGLE R2 bucket, so velero and
|
||||
# barman cannot be separated by bucket the way they are on dev's MinIO (`velero` and `cnpg`).
|
||||
# They are separated by disjoint TOP-LEVEL PREFIXES instead:
|
||||
#
|
||||
# s3://k3s-prod-backup/velero/… Velero: kopia repo + backup metadata
|
||||
# s3://k3s-prod-backup/barman/<clusterName>/… CNPG: WAL + base backups
|
||||
#
|
||||
# The bucket ROOT is left empty on purpose — a misrendered prefix then lands somewhere
|
||||
# obviously wrong instead of silently colliding with a live tenant. Velero's retention TTL
|
||||
# deletes objects under its own prefix only, so it can never reap a WAL segment; and barman's
|
||||
# per-cluster serverName subdirectory keeps the three Postgres servers' chains apart, which
|
||||
# matters because interleaving two servers' WAL into one prefix corrupts the chain outright.
|
||||
# Dev cannot be reached from any of these: different endpoint AND different bucket names.
|
||||
# ------------------------------------------------------------------------------------------
|
||||
- op: replace
|
||||
path: /data/VELERO_BACKUP_BUCKET
|
||||
value: "k3s-prod-backup"
|
||||
- op: replace
|
||||
path: /data/VELERO_BACKUP_PREFIX
|
||||
value: "velero"
|
||||
- op: replace
|
||||
path: /data/CNPG_BACKUP_PATH
|
||||
value: "s3://k3s-prod-backup/barman"
|
||||
# ------------------------------------------------------------------------------------------
|
||||
# WHICH CREDENTIAL SIGNS THE REQUESTS TO THAT BUCKET (per-env credential split, 2026-09-07).
|
||||
# ------------------------------------------------------------------------------------------
|
||||
# The knobs above moved the ENDPOINT and the OBJECT LAYOUT per environment. The credential did
|
||||
# not move with them, and it could not: the four backup-credential Secrets are SHARED BASE
|
||||
# manifests, applied to both clusters by the same Flux tiers. Writing the R2 token into them
|
||||
# gave prod a working key by TAKING DEV'S AWAY — dev would have signed MinIO requests with an
|
||||
# R2 key id, and under archive_mode=on that is a filling pgdata PVC, not a missing backup.
|
||||
#
|
||||
# So each credential is now two objects, and this suffix selects prod's:
|
||||
# velero-cloud-credentials-prod the R2 token in the AWS INI `cloud` shape
|
||||
# <db>-backup-credentials-prod the same token as ACCESS_KEY_ID / ACCESS_SECRET_KEY
|
||||
# for keycloak-db, nextcloud-db and semaphore-db. Both halves ship to both clusters (they are
|
||||
# base resources); ONLY this value decides which one each consumer reads. It is a SUFFIX rather
|
||||
# than four secret-name knobs deliberately — four could be updated three-quarters of the way,
|
||||
# leaving one consumer on dev's MinIO key with nothing rendering wrong, and strict-mode
|
||||
# substitution cannot catch a defined-but-stale variable.
|
||||
#
|
||||
# THE R2 TOKEN IS THE ONE PROVEN FROM INSIDE PROD (chore/r2-credentials 0a6a862): head-bucket on
|
||||
# k3s-prod-backup returns BucketRegion WNAM, and PUT and DELETE both succeed. Rotating it now
|
||||
# touches ONLY the four `-prod` files; dev's MinIO keys are untouched objects. That isolation is
|
||||
# the structural reason for the split, independent of dev's eventual decommission.
|
||||
- op: replace
|
||||
path: /data/BACKUP_CREDENTIALS_SUFFIX
|
||||
value: "-prod"
|
||||
# TODO(WS-INFRA, Phase 7): pin prod node placement once the bare-metal role labels exist.
|
||||
# TODO(WS-NET, Phase 7): set GATEWAY_VIP to the prod LAN ingress VIP (add a `replace` op here)
|
||||
# at the same time the prod Cilium LB IP pool range is supplied — the split-horizon
|
||||
# coredns-custom (platform/coredns) answers *.home.keithjackson.dev with ${GATEWAY_VIP}, so
|
||||
# the prod value must match the prod pool's Gateway address. Base default is the dev VIP.
|
||||
@@ -1,32 +0,0 @@
|
||||
---
|
||||
# Self-managing ROOT Kustomization for PROD (mirrors clusters/dev/sync.yaml).
|
||||
#
|
||||
# Reconciles ./clusters/prod from the GitRepository so the ENTIRE prod graph — the cluster-vars
|
||||
# ConfigMap, the GitRepository source, platform / platform-config / the app waves, and THIS object
|
||||
# itself — is applied from git on every push. dev/sync.yaml carried the note "a prod overlay needs
|
||||
# its own sync.yaml -> ./clusters/prod"; this is it.
|
||||
#
|
||||
# WHY IT IS NOW LOAD-BEARING, not just symmetry: `clusters/base/platform.yaml` gained
|
||||
# `dependsOn: [flux-system]` (2026-09-01) to close the post-build substitution race — a dependent
|
||||
# cannot reconcile revision N until the ROOT has APPLIED revision N. Without this object there is no
|
||||
# `flux-system` Kustomization on prod, so `platform` would block forever on a dependency that does
|
||||
# not exist, and NOTHING downstream would ever reconcile.
|
||||
#
|
||||
# `wait: false` is deliberate and is what keeps the dependsOn edge deadlock-free: the root reaches
|
||||
# Ready on apply success without health-gating its children (platform does its own wait/health
|
||||
# gating). Do not "harden" this to true.
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: flux-system
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 10m
|
||||
retryInterval: 2m
|
||||
timeout: 5m
|
||||
prune: true
|
||||
wait: false
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./clusters/prod
|
||||
@@ -5,25 +5,3 @@ _artifacts/
|
||||
|
||||
# Downloaded pinned tool binaries (run.sh may fetch chainsaw/kubectl here).
|
||||
.bin/
|
||||
|
||||
# Python bytecode from lib/ (status.py imports coverage.py); never committed.
|
||||
__pycache__/
|
||||
|
||||
# --- THE ONE EXCEPTION: the committed run ledger (runs/) -------------------------------
|
||||
# `runs/<UTC>-<short-sha>/{report.junit.xml,run.log}` + `runs/ledger.jsonl` are the
|
||||
# machine-readable proof-of-pass record that lib/status.py's docstring asks for. They are
|
||||
# SOURCE, not regenerated artifacts: a run's evidence cannot be reconstructed after the
|
||||
# fact, so it must survive in git.
|
||||
#
|
||||
# The negations are needed because the ROOT .gitignore ignores `*.junit.xml` globally
|
||||
# (and `coverage/`). A deeper .gitignore overrides a shallower one, and nothing ignores
|
||||
# the `conformance/runs/` directory itself, so re-including the files here works (git
|
||||
# cannot re-include a file whose PARENT DIRECTORY is excluded — that is why this must not
|
||||
# live under _artifacts/).
|
||||
#
|
||||
# The run LOG is the load-bearing half: Chainsaw's JUnit carries no stdout, and the L2/L3
|
||||
# tests self-skip GREEN, so the sentinel token in run.log is the only thing separating
|
||||
# "proven" from "skipped". Never commit a report.junit.xml without its run.log.
|
||||
!runs/
|
||||
!runs/**
|
||||
!runs/**/*.junit.xml
|
||||
@@ -18,14 +18,9 @@ behavioral / L3 fault-injection** (this harness).
|
||||
./run.sh --requirements R-NET-01,R-NET-06 # merge-gate subset (tagged)
|
||||
./run.sh --layer l2 # one layer: l1 | l2 | l3 | meta | all
|
||||
./run.sh --coverage-only # regenerate the coverage report (no cluster)
|
||||
./run.sh --status # derived requirement-STATUS report (no cluster)
|
||||
./run.sh --lint # structural-validate test defs (no cluster)
|
||||
./run.sh --strict-coverage # hygiene gate: fail on tag/annotation drift
|
||||
./run.sh --require-all # phase-exit gate: every requirement must be covered
|
||||
./run.sh --record # run AND capture the evidence (see "Proof of pass")
|
||||
./run.sh --layer l0 --include-scope whole-repo # union: the tagged subset PLUS every test
|
||||
# declaring `conformance.home.k3s/scope: whole-repo`
|
||||
./run.sh --layer l0 --print-selection # print the dirs the selection resolves to; run nothing
|
||||
```
|
||||
|
||||
**Gating is scoped, not "all 51 always"** (R-TEST-03). Three distinct checks:
|
||||
@@ -35,29 +30,6 @@ behavioral / L3 fault-injection** (this harness).
|
||||
- `--requirements R-…` — **merge gate**: selects the tagged tests to run **and** asserts
|
||||
those touched requirements are covered (a listed safety-critical one must have an L3 test).
|
||||
- `--require-all` — **phase-exit gate**: every registry requirement must be covered.
|
||||
- `--include-scope whole-repo` — **the escape hatch scoping needs**, because scoping has one
|
||||
blind spot and it is the worst-shaped kind. Some L0 invariants quantify over the WHOLE
|
||||
tracked tree (no plaintext secrets anywhere; no RBAC wildcard anywhere), or over subtrees
|
||||
the coarse path→requirement map keys to **nothing** (`apps/`, `infra/`, most of
|
||||
`platform/`). For those, *the change most likely to break the invariant attributes no
|
||||
requirement at all*, so requirement-scoped selection filters the check out of exactly that
|
||||
change. Such a test **declares** its scope:
|
||||
|
||||
```yaml
|
||||
metadata:
|
||||
annotations:
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
```
|
||||
|
||||
and `--include-scope whole-repo` unions every declaring test into the selection. The CI
|
||||
stages (`ci/l0.sh`, `ci/l1.sh`, `ci/conformance.sh`) all pass it, so **adding the
|
||||
annotation is the only step** — there is nothing to remember and nothing to hand-write.
|
||||
Two consequences, both deliberate: a declaring test always runs, and a **SKIP of a
|
||||
declaring test is UNWIRED, not green** (the runner exits 3, which `ci/l0.sh` turns into the
|
||||
gate's own rc 3 — never a pass). `tests/l0-static/whole-repo-invariants-wired` proves the
|
||||
mechanism end-to-end and falsifiably, so it cannot rot silently. This replaced a
|
||||
hand-written "UNCONDITIONAL" stage per check in `ci/l0.sh`; see `lib/common.sh` for the
|
||||
full rationale and the incidents behind it.
|
||||
- Documented out-of-scope gaps (e.g. `R-ENV-02` dev→prod promote and `R-GITOPS-01`
|
||||
state-in-git, owned by WS-INFRA/WS-GITOPS and keyed wholesale by the coarse path map)
|
||||
are exempted from the completeness gates via `--allow-gap`, wired in `run.sh` as
|
||||
@@ -65,33 +37,6 @@ behavioral / L3 fault-injection** (this harness).
|
||||
used to be here; it now has a real L3 test — `tests/l3-fault-injection/node-loss` — and
|
||||
is covered, not exempted.)
|
||||
|
||||
### Coverage vs status — two different questions
|
||||
|
||||
`coverage.md` answers **"does a requirement-tagged test EXIST for R-x?"** (R-TEST-04).
|
||||
`status.md` (`./run.sh --status`, generator `lib/status.py`) answers the operator's
|
||||
question — **"what is the real state of each requirement given what is actually in the
|
||||
repo?"** — deriving three orthogonal axes per requirement:
|
||||
|
||||
1. **implementation** — is there a tracked artifact naming the requirement?
|
||||
2. **reconciliation** — is that artifact reachable by walking `resources:` from
|
||||
`clusters/<env>/kustomization.yaml` and expanding every Flux `Kustomization`'s
|
||||
`spec.path`? This is what separates **authored** from **actually reconciled**: a
|
||||
commented-out `resources:` entry (e.g. `platform/image-automation`) is correctly
|
||||
reported as in-git-but-not-on-any-cluster.
|
||||
3. **verification** — is the tagged test EXECUTABLE, or a TODO scaffold? Coverage counts
|
||||
both as "covered"; status does not. TODO markers in a body always outrank an
|
||||
`executable: "true"` annotation, so ambiguity never upgrades a test.
|
||||
|
||||
**It deliberately refuses to derive proof-of-pass.** Nothing tracked in git records a test
|
||||
result — `_artifacts/` and `results/` are git-ignored and DECISIONS-LOG.md prose is not a
|
||||
signal — so every requirement's proof state is reported as `not-derivable` rather than
|
||||
defaulting to green. A status report that overstates completion is worse than none
|
||||
(`agents/definition-of-done.md`). Closing that gap needs a **committed, machine-readable
|
||||
run ledger**; until then the strongest claim the report makes is *wired and testable*.
|
||||
`--strict-coverage` also applies status's hygiene gate (registry↔spec drift,
|
||||
annotation/body conflicts, a safety-critical requirement whose only L3 test is a scaffold).
|
||||
Exit codes follow `ci/lib.sh`: `0` pass / `1` fail / `3` skipped — a skip is never green.
|
||||
|
||||
`run.sh` is the **only** entrypoint. Local runs and the in-cluster CI Chainsaw Job call
|
||||
the *same* script with the same config, so results are identical.
|
||||
|
||||
@@ -101,8 +46,6 @@ the *same* script with the same config, so results are identical.
|
||||
| `_artifacts/report.junit.xml` | JUnit — Chainsaw results, or a **skipped-scaffold** shim when no cluster | CI gate |
|
||||
| `_artifacts/coverage/coverage.md` | Human requirement-coverage report | PR reviewers / logs |
|
||||
| `_artifacts/coverage/coverage.json` | Machine coverage: `{requirement: [tests]}`, gaps, safety gaps | CI gate |
|
||||
| `_artifacts/status/status.md` | Human requirement-STATUS report derived from repo state | operator / PM |
|
||||
| `_artifacts/status/status.json` | Machine status: implementation / reconciliation / verification per requirement | tooling |
|
||||
|
||||
## Reproducibility contract (R-TEST-02 — what the harness guarantees)
|
||||
- **Single command.** `./run.sh` (or a tagged/layer subset) runs everything and emits the
|
||||
@@ -131,90 +74,13 @@ requirements/registry.yaml mirror of EVERY spec R-* ID (coverage enumerates fr
|
||||
lib/
|
||||
common.sh tagging convention + test discovery/selection helpers
|
||||
coverage.py requirement-coverage generator (stdlib only)
|
||||
status.py requirement-STATUS generator (stdlib only; --selftest)
|
||||
record_run.py run-ledger writer: PROOF-of-pass (stdlib only; --selftest)
|
||||
emit_skipped_junit.py Phase-0 skipped-JUnit shim (no cluster)
|
||||
runs/
|
||||
ledger.jsonl TRACKED append-only proof-of-pass ledger (one row per
|
||||
testcase per run) — the only source status.py trusts
|
||||
<UTC>-<short-sha>/ TRACKED per-run evidence: run.log + report.junit.xml
|
||||
tests/
|
||||
l1-rendering/ l2-behavioral/ l3-fault-injection/ meta/ (see tests/README.md)
|
||||
incluster/
|
||||
chainsaw-job.yaml rbac.yaml in-cluster L2/L3 Job + scoped test SA (ADR-0011)
|
||||
```
|
||||
|
||||
## Proof of pass — the run ledger (`runs/`)
|
||||
|
||||
`--status` used to report **PROVEN: 0** for all 51 requirements, and said so honestly:
|
||||
proof-of-pass was not recoverable from repo state. `_artifacts/` is gitignored and the
|
||||
root `.gitignore` kills `*.junit.xml` globally, so no run result ever reached git; the
|
||||
only record of a green run was DECISIONS-LOG.md prose. `runs/` is the fix `lib/status.py`
|
||||
asked for — **a committed, machine-readable run ledger**.
|
||||
|
||||
```bash
|
||||
./run.sh --record # runs normally, and captures the evidence
|
||||
# -> conformance/runs/<UTC>-<short-sha>/run.log (tee'd, carries the sentinels)
|
||||
# -> conformance/runs/<UTC>-<short-sha>/report.junit.xml (that run's JUnit)
|
||||
# -> conformance/runs/ledger.jsonl (one appended row per testcase)
|
||||
python3 lib/record_run.py --run-dir runs/<id> [--dry-run] # or record an existing run dir
|
||||
python3 lib/record_run.py --selftest # deterministic, no cluster
|
||||
```
|
||||
|
||||
**Recording is explicit, never implicit.** Plain `./run.sh` writes no ledger row: a run
|
||||
that silently records itself is how a bad run becomes a permanent false proof. **Commit
|
||||
the run dir and the ledger** — an uncommitted ledger proves nothing.
|
||||
|
||||
### Why a committed JUnit alone would be WORSE than nothing
|
||||
|
||||
The L2/L3 tests are runnable and **self-skipping**: when a precondition is missing (no
|
||||
`longhorn` StorageClass, no Velero CRDs, fewer than 3 Ready nodes) they echo a skip token
|
||||
and `exit 0`, and their `check:` accepts either outcome —
|
||||
|
||||
```yaml
|
||||
(contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')
|
||||
|| contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true
|
||||
```
|
||||
|
||||
So **a self-skip exits 0 and Chainsaw records `passed` exactly as a real run does**, and
|
||||
Chainsaw's JUnit carries no stdout — its `<testcase>` elements hold only `<properties>`.
|
||||
A committed JUnit on its own would therefore record `DR_RESTORE_SKIPPED_OK` as *proof of
|
||||
R-STATE-03*: fake green, and worse than no ledger because it would look authoritative.
|
||||
|
||||
**The sentinel token in `run.log` is the only thing that separates proven from skipped.**
|
||||
`record_run.py` derives each test's tokens from its own `chainsaw-test.yaml` (every
|
||||
`contains($stdout, 'TOKEN')`; `*_SKIP*_OK` is a skip token, the last non-skip token is the
|
||||
verdict token), so the table below is never hand-maintained:
|
||||
|
||||
| Test | pass token (proof) | skip token (never proof) |
|
||||
|---|---|---|
|
||||
| `l3/node-loss` | `NODE_LOSS_VOLUME_AVAILABLE_OK` | `NODE_LOSS_SKIPPED_OK` |
|
||||
| `l3/dr-restore` | `DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK` (+ `OFF_CLUSTER_TARGET_OK`, `BACKUPS_ARE_SCHEDULED_OK`) | `DR_RESTORE_SKIPPED_OK`, `DR_SCHEDULE_SKIPPED_OK` |
|
||||
| `l3/data-survival` | `DATA_SURVIVED_OK` | `DATA_SURVIVAL_SKIPPED_OK` |
|
||||
| `l3/vm-data-survival` | `VM_DATA_SURVIVED_OK` | `VM_DATA_SURVIVAL_SKIPPED_OK` |
|
||||
| `l3/hands-off-update` | `HANDS_OFF_UPDATE_OK` | `HANDS_OFF_UPDATE_SKIPPED_OK` |
|
||||
| `l2/vm-lifecycle` | `VM_LIFECYCLE_OK` | `VM_LIFECYCLE_SKIPPED_OK` |
|
||||
| `l2/vm-default-deny` | `VM_DEFAULT_DENY_OK` | `VM_DEFAULT_DENY_SKIPPED_OK` |
|
||||
|
||||
A row reaches `proof: "proven"` on **one** path: JUnit `passed`, **every** declared pass
|
||||
token present in that test's log lines, and **no** skip token anywhere in them. A skip
|
||||
token beats a pass token (worst wins). Anything else is `skipped`, `failed`,
|
||||
`junit-skipped`, `unknown-no-sentinel-declared` or `unknown-sentinel-absent` — all
|
||||
non-green. L0/L1 need no sentinel because `lib/run_l1_offline.py` already renders their
|
||||
skips as JUnit `<skipped/>`, never as a green pass.
|
||||
|
||||
### Staleness — a proof is a claim about a tree, not a badge
|
||||
|
||||
`--status` honours a `proven` row only while the run's `git_sha` is a **descendant of the
|
||||
last commit touching that requirement's implementing manifests or its verifying test**.
|
||||
Otherwise the row degrades to **`stale-proof`**, which is not green. A run captured from a
|
||||
dirty working tree is recorded as `<sha>-dirty`, which no ancestry check can satisfy — so
|
||||
it never counts. Re-run with `--record` to refresh.
|
||||
|
||||
A requirement reads `proven` only when **every** verifying test does; one skipped, failed
|
||||
or unrecorded test drags the whole requirement down. With an empty ledger `--status`
|
||||
behaves exactly as before: every requirement `not-derivable`, headline **PROVEN 0**.
|
||||
|
||||
## How CI invokes it
|
||||
CI (Gitea Actions, wired in T-0.2 / migrated in T-6.1b) calls **this same `run.sh`**:
|
||||
|
||||
|
||||
@@ -57,13 +57,6 @@ rules:
|
||||
verbs: [create, delete, get, list, watch, patch]
|
||||
- apiGroups: [""]
|
||||
resources: [pods, services, configmaps, secrets, persistentvolumeclaims]
|
||||
# LOAD-BEARING `get` ON CONFIGMAPS, BEYOND FIXTURE CREATION (R-ENV-02): the environment-
|
||||
# agnostic L2 tests (dns-exposure, dns-internal-resolution, gateway-ingress-reachable,
|
||||
# oidc-sso-grafana, trusted-tls) resolve the cluster's own BASE_DOMAIN / GATEWAY_VIP at run
|
||||
# time from `flux-system/cluster-vars` — the same ConfigMap Flux substitutes from — via
|
||||
# chainsaw `x_k8s_get($client, 'v1', 'ConfigMap', 'flux-system', 'cluster-vars')` bindings.
|
||||
# If this rule is ever narrowed to the test's own namespace, those bindings fail and the
|
||||
# tests error out. Narrow it only alongside a namespace-scoped read of that ConfigMap.
|
||||
verbs: [create, delete, get, list, watch, patch]
|
||||
- apiGroups: [""]
|
||||
resources: [pods/exec]
|
||||
@@ -71,22 +64,6 @@ rules:
|
||||
- apiGroups: [""]
|
||||
resources: [pods/log]
|
||||
verbs: [get] # read fault/probe Job logs on failure
|
||||
# ---- Storage-tier precondition gate (READ-ONLY). Every storage test self-skips on the
|
||||
# absence of the `longhorn` StorageClass rather than failing: data-survival (R-STATE-01),
|
||||
# node-loss (R-STATE-02), dr-restore (R-STATE-03), vm-data-survival / vm-lifecycle /
|
||||
# vm-default-deny (R-PLAT-03). Without this read the gate cannot tell "absent" from
|
||||
# "forbidden" and would skip for the wrong reason.
|
||||
- apiGroups: [storage.k8s.io]
|
||||
resources: [storageclasses]
|
||||
verbs: [get, list]
|
||||
# ---- Controller-readiness preconditions (READ-ONLY). dr-restore (R-STATE-03) skips cleanly
|
||||
# unless the Velero Deployment and its node-agent DaemonSet (kopia File System Backup, the
|
||||
# volume-DATA path) are actually ready — a backup path that is not up is an absent
|
||||
# precondition, not a failure. No write verbs: tests never scale or patch a platform
|
||||
# controller.
|
||||
- apiGroups: [apps]
|
||||
resources: [deployments, daemonsets]
|
||||
verbs: [get, list, watch]
|
||||
# ---- Probe / fault Jobs (R-TLS-01/03 tls-probe; R-NET-06 tunnel stop/start)
|
||||
# `create` cannot be constrained by resourceNames, and the fault Job names are
|
||||
# fixed per-test (tunnel-stop/tunnel-start/tunnel-start-finally); scoped by the
|
||||
@@ -126,29 +103,10 @@ rules:
|
||||
# resources: [imagepolicies, imagerepositories]
|
||||
# verbs: [get, patch]
|
||||
#
|
||||
# ---- R-STATE-03 (DR restore, tests/l3-fault-injection/dr-restore): the drill creates a
|
||||
# Velero Backup of its OWN scratch namespace, destroys that namespace + its PV, creates a
|
||||
# Restore into a fresh namespace, and (in `finally`) issues a DeleteBackupRequest so the
|
||||
# objects it wrote are removed from the off-cluster bucket again. `create` cannot be
|
||||
# constrained by resourceNames; the blast radius is bounded by the fact that every object
|
||||
# the test creates or deletes is named `<its-own-chainsaw-namespace>-dr-*`.
|
||||
- apiGroups: [velero.io]
|
||||
resources: [backups, restores, deletebackuprequests]
|
||||
verbs: [create, delete, get, list, watch]
|
||||
# Read-only evidence: the BSL is the off-cluster target whose Available phase + endpoint the
|
||||
# test asserts; schedules prove backups are SCHEDULED (not one-off); podvolumebackups /
|
||||
# podvolumerestores prove the kopia snapshot of the volume DATA reached/returned from the
|
||||
# remote repository, and are dumped on failure for diagnosis.
|
||||
- apiGroups: [velero.io]
|
||||
resources: [backupstoragelocations, schedules, podvolumebackups, podvolumerestores]
|
||||
verbs: [get, list, watch]
|
||||
# R-STATE-03 needs `delete` on the PV, not merely read: the proof that the restore came from
|
||||
# OFF-cluster is that no in-cluster copy of the data survived. The test asserts its own PV is
|
||||
# gone, and force-deletes it if a Retain policy or a stuck finalizer left it behind. It only
|
||||
# ever names the PV bound to the PVC it created itself.
|
||||
- apiGroups: [""]
|
||||
resources: [persistentvolumes]
|
||||
verbs: [get, list, delete]
|
||||
# ---- R-STATE-03 (DR restore): trigger a Velero restore
|
||||
# - apiGroups: [velero.io]
|
||||
# resources: [restores, backups]
|
||||
# verbs: [create, get, list, watch]
|
||||
#
|
||||
# ---- R-GITOPS-03 (drift): mutate a reconciled resource + observe Flux revert
|
||||
# - apiGroups: [kustomize.toolkit.fluxcd.io]
|
||||
|
||||
@@ -47,63 +47,7 @@ conf_dirs_for_requirements() {
|
||||
done < <(conf_all_test_files) | sort -u
|
||||
}
|
||||
|
||||
# --- DECLARED SCOPE: whole-repo invariants -------------------------------------------
|
||||
# A test may also declare the SCOPE of the invariant it asserts:
|
||||
#
|
||||
# metadata:
|
||||
# annotations:
|
||||
# conformance.home.k3s/scope: "whole-repo"
|
||||
#
|
||||
# WHY THIS EXISTS — the requirement-scoped-selection blind spot (R-TEST-03)
|
||||
# ------------------------------------------------------------------------
|
||||
# The merge gate runs the conformance subset tagged with the requirements a change
|
||||
# TOUCHES (ci/touched-requirements.sh -> `--requirements`). That is right for an invariant
|
||||
# whose input surface is the subtree its requirements map to. It is WRONG — silently, and in
|
||||
# the worst direction — for an invariant that quantifies over the WHOLE tracked tree, or over
|
||||
# subtrees (apps/, platform/<most>, infra/) that the coarse path->requirement map keys to
|
||||
# NOTHING. For those, the change most likely to break the invariant attributes no requirement
|
||||
# at all, so the requirement-scoped selection filters the check out of EXACTLY that change.
|
||||
# Three real incidents came out of that blind spot; each is recorded in the header of the
|
||||
# test that now guards it (flux-postbuild-substitution, keycloak-client-secret-reconciler,
|
||||
# bulk-tier-distinct-from-replicated).
|
||||
#
|
||||
# The repo's previous answer was a hand-written "UNCONDITIONAL" stage per such test in
|
||||
# ci/l0.sh — which does not scale and fails silently in the worst way: whoever adds the NEXT
|
||||
# whole-repo invariant must REMEMBER to hand-write a stage, and if they forget, the check runs
|
||||
# only when some unrelated requirement happens to be touched. The scope annotation replaces
|
||||
# that with a DECLARATION the runner enforces: `run.sh --include-scope whole-repo` unions
|
||||
# every test carrying it into the selection, so adding the annotation is the only step.
|
||||
#
|
||||
# TWO things follow from the annotation, and both are deliberate:
|
||||
# 1. the test is ALWAYS selected (union with the requirement-scoped subset), and
|
||||
# 2. a SKIP of that test is UNWIRED, NOT GREEN (lib/run_l1_offline.py --unwired-scope).
|
||||
# A whole-repo invariant that reports "nothing to audit" is reporting that the subtree
|
||||
# it audits has VANISHED, which is not a pass. This is the rc-3 semantics the bespoke
|
||||
# stages encoded ("check SKIPPED (clusters/ absent) — not a pass"), preserved verbatim.
|
||||
export CONF_SCOPE_ANNOTATION="${CONF_LABEL_PREFIX}/scope"
|
||||
|
||||
# Directories of tests DECLARING the given scope (e.g. "whole-repo"). The annotation is
|
||||
# matched on a real YAML mapping line only: a `#` comment cannot start with the key, so a
|
||||
# scope named in a test's header prose is documentation, not a declaration (the same
|
||||
# "reference, not declaration" rule ci/touched-requirements.sh applies to R-IDs).
|
||||
conf_dirs_for_scope() {
|
||||
local scope="$1" f
|
||||
[ -n "$scope" ] || return 0
|
||||
while IFS= read -r f; do
|
||||
[ -n "$f" ] || continue
|
||||
if grep -Eq "^[[:space:]]*${CONF_SCOPE_ANNOTATION}:[[:space:]]*[\"']?${scope}[\"']?[[:space:]]*$" "$f"; then
|
||||
dirname "$f"
|
||||
fi
|
||||
done < <(conf_all_test_files) | sort -u
|
||||
}
|
||||
|
||||
# Union of conf_dirs_for_scope over a comma/space separated list of scopes.
|
||||
conf_dirs_for_scopes() {
|
||||
local scopes="$1" s
|
||||
for s in ${scopes//,/ }; do conf_dirs_for_scope "$s"; done | sort -u
|
||||
}
|
||||
|
||||
# Directories for a given layer (subtree): l0 | l1 | l2 | l3 | meta | all.
|
||||
# Directories for a given layer (subtree): l1 | l2 | l3 | meta | all.
|
||||
# Accepts a comma/space separated LIST of layers (e.g. "l2,l3") and returns the UNION of
|
||||
# their test dirs, deduplicated. A single layer keeps its prior behavior (backward compatible);
|
||||
# this is what lets `run.sh --layer l2,l3` (the ci/conformance.sh L2/L3 stage) select both
|
||||
@@ -114,7 +58,6 @@ conf_dirs_for_layer() {
|
||||
for layer in ${layers//,/ }; do
|
||||
case "$layer" in
|
||||
all) roots=("${CONF_ROOT}/tests"); break ;;
|
||||
l0) roots+=("${CONF_ROOT}/tests/l0-static") ;;
|
||||
l1) roots+=("${CONF_ROOT}/tests/l1-rendering") ;;
|
||||
l2) roots+=("${CONF_ROOT}/tests/l2-behavioral") ;;
|
||||
l3) roots+=("${CONF_ROOT}/tests/l3-fault-injection") ;;
|
||||
|
||||
@@ -115,9 +115,7 @@ def parse_test(path: Path) -> dict:
|
||||
# Layer inferred from path as a fallback / cross-check.
|
||||
rel = str(path)
|
||||
if not layer:
|
||||
if "l0-static" in rel:
|
||||
layer = "L0"
|
||||
elif "l1-rendering" in rel:
|
||||
if "l1-rendering" in rel:
|
||||
layer = "L1"
|
||||
elif "l2-behavioral" in rel:
|
||||
layer = "L2"
|
||||
|
||||
@@ -33,8 +33,7 @@ def collect(path: Path) -> tuple[str, list[str], str]:
|
||||
reqs = sorted({m.group(1).upper() for m in
|
||||
(LABEL_LINE_RE.match(l) for l in text.splitlines()) if m})
|
||||
rel = str(path)
|
||||
layer = ("L0" if "l0-static" in rel
|
||||
else "L1" if "l1-rendering" in rel else "L2" if "l2-behavioral" in rel
|
||||
layer = ("L1" if "l1-rendering" in rel else "L2" if "l2-behavioral" in rel
|
||||
else "L3" if "l3-fault-injection" in rel else "META")
|
||||
return name, reqs, layer
|
||||
|
||||
|
||||
@@ -1,988 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Append a conformance RUN LEDGER row per testcase — the committed proof-of-pass record.
|
||||
|
||||
WHY THIS EXISTS
|
||||
---------------
|
||||
`lib/status.py` reports PROOF as `not-derivable` for every requirement, and names its own
|
||||
fix in its docstring: *"a committed, machine-readable run ledger"*. There was none, because
|
||||
`conformance/_artifacts/` is git-ignored and the root `.gitignore` kills `*.junit.xml` and
|
||||
`coverage/` globally, so nothing about a green run ever reached repo state. This tool is
|
||||
that ledger's writer: it reads one run's JUnit + run log out of `conformance/runs/<run-id>/`
|
||||
and appends one JSON object per testcase to `conformance/runs/ledger.jsonl`, which IS
|
||||
tracked (see conformance/.gitignore for the negations that let it survive).
|
||||
|
||||
THE TRAP THIS TOOL EXISTS TO AVOID — READ BEFORE CHANGING ANYTHING
|
||||
------------------------------------------------------------------
|
||||
The L2/L3 tests are RUNNABLE / SELF-SKIPPING: when a precondition is absent (no `longhorn`
|
||||
StorageClass, no Velero CRDs, <3 Ready nodes) they echo a skip token and `exit 0`, and their
|
||||
`check:` block is deliberately written to accept either outcome, e.g. tests/l3-fault-
|
||||
injection/dr-restore:
|
||||
|
||||
(contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')
|
||||
|| contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true
|
||||
|
||||
So a SELF-SKIP exits 0 and Chainsaw records `passed` EXACTLY as a real run does. And
|
||||
Chainsaw's JUnit carries no stdout — its `<testcase>` elements hold only `<properties>`
|
||||
(verified against the committed reports). Therefore:
|
||||
|
||||
*** A COMMITTED JUnit ALONE WOULD RECORD `DR_RESTORE_SKIPPED_OK` AS PROOF OF R-STATE-03. ***
|
||||
|
||||
AND THE SECOND TRAP, WHICH IS THE MIRROR OF THE FIRST
|
||||
------------------------------------------------------
|
||||
Chainsaw echoes each script's SOURCE into an indented `=== COMMAND` block before running
|
||||
it, so `echo NODE_LOSS_SKIPPED_OK` — the skip branch that was NOT taken — is present in
|
||||
the log of every run of that test. A section-blind grep therefore reads a skip token out
|
||||
of the source of every skip-gated test and can never record `proven` for any of them: the
|
||||
ledger fails closed so completely that it records nothing and proves nothing. Tokens are
|
||||
consequently read from SECTIONS (see classify_sections): a pass token only from
|
||||
`=== STDOUT`, a skip token from any section that is not script source. Skip still beats
|
||||
pass — within genuine output, which is the only place either was ever evidence.
|
||||
|
||||
That is the fake-green this project refuses, and it would be worse than no ledger at all
|
||||
because it would look authoritative. THE SENTINEL TOKEN IS THE ONLY THING THAT SEPARATES
|
||||
PROVEN FROM SKIPPED, AND IT EXISTS ONLY IN THE RUN LOG. Hence: this tool requires BOTH
|
||||
files, and for the Chainsaw layers it will NEVER emit `proof: "proven"` on a JUnit `passed`
|
||||
unless the log carries that test's own pass token and carries NO skip token.
|
||||
|
||||
HOW THE TOKENS ARE OBTAINED (derived, never hand-listed)
|
||||
--------------------------------------------------------
|
||||
Hard-coding a token table would rot the moment a test is edited. Instead every token is
|
||||
parsed out of the test's own `chainsaw-test.yaml`: each `contains($stdout, 'TOKEN')` in a
|
||||
`check:` block is a declared sentinel. Classification is by the project's naming convention
|
||||
(`*_SKIP*_OK` is a skip token, anything else is a pass token), and the TERMINAL pass token
|
||||
— the last non-skip token in file order — is the test's verdict token. Edit a test's tokens
|
||||
and this tool follows automatically; delete them and the test simply becomes unprovable
|
||||
rather than silently proven.
|
||||
|
||||
node-loss pass NODE_LOSS_VOLUME_AVAILABLE_OK skip NODE_LOSS_SKIPPED_OK
|
||||
dr-restore pass DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK skip DR_RESTORE_SKIPPED_OK
|
||||
(+ OFF_CLUSTER_TARGET_OK, BACKUPS_ARE_SCHEDULED_OK / DR_SCHEDULE_SKIPPED_OK)
|
||||
|
||||
WHY L0/L1 NEED NO SENTINEL (and why that is not a loophole)
|
||||
-----------------------------------------------------------
|
||||
The offline layers are not run by Chainsaw. `lib/run_l1_offline.py` shells out to each
|
||||
dir's `check.sh` / `render-check.sh` and maps exit 3 to a JUnit `<skipped/>` element and
|
||||
any other non-zero to `<failure>`. A self-skip there is ALREADY a JUnit skip, never a green
|
||||
pass, so the JUnit is a truthful record on its own. Only the Chainsaw layers (L2/L3/META)
|
||||
can express "skipped" as "passed", and only they require a sentinel — see
|
||||
SENTINEL_REQUIRED_LAYERS.
|
||||
|
||||
LEDGER ROW (conformance/runs/ledger.jsonl — append-only, one object per testcase per run)
|
||||
ts_utc run-dir timestamp, else the JUnit's mtime (when)
|
||||
git_sha `git rev-parse HEAD`, suffixed `-dirty` when tracked files were
|
||||
modified — a proof captured from a dirty tree is NOT honoured by
|
||||
status.py, because what ran is not what is committed (what)
|
||||
env --env / CONF_ENV (cluster overlay the run targeted)
|
||||
chainsaw_version CHAINSAW_VERSION from versions.env (the pin IS the truth)
|
||||
kube_version `kubectl version` server gitVersion, else "unknown"
|
||||
layer L0|L1|L2|L3|META, from the test's `layer` annotation / JUnit property
|
||||
test testcase name (== the test dir basename, asserted by convention)
|
||||
requirements[] the `conformance.home.k3s/R-*` labels on the test — the SAME
|
||||
authoritative source coverage.py uses, not the JUnit classname
|
||||
junit_result passed | failed | error | skipped
|
||||
sentinel the DECIDING token grepped from run.log, or null
|
||||
proof proven | skipped | failed | junit-skipped
|
||||
| unknown-no-sentinel-declared | unknown-sentinel-absent
|
||||
run_id the runs/<dir> name (provenance: which log backs this row)
|
||||
note one line saying WHY this row landed on that proof state
|
||||
|
||||
`proven` is emitted on exactly one path (see decide()); every other path is non-green.
|
||||
|
||||
RECORDING IS EXPLICIT, NEVER IMPLICIT
|
||||
-------------------------------------
|
||||
Invoke this tool by hand, or via `run.sh --record`. A run that silently records itself is
|
||||
how a bad run becomes a permanent false proof, so `run.sh` without `--record` writes no
|
||||
ledger row at all.
|
||||
|
||||
EXIT CODES (ci/lib.sh convention: 0 pass / 1 fail / 3 skipped-not-green)
|
||||
0 rows appended.
|
||||
1 a real failure: unparsable JUnit, unreadable test tree, unwritable ledger.
|
||||
3 the inputs a ledger row needs are absent (no report.junit.xml, or no run.log).
|
||||
NOTHING is recorded and this is NEVER green — a run we cannot evidence is not a run.
|
||||
|
||||
Pure standard library, matching the harness's no-pip-dependency posture (coverage.py,
|
||||
status.py). PyYAML lives in catalog/render.py; the harness does not depend on it.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import xml.etree.ElementTree as ET
|
||||
from pathlib import Path
|
||||
|
||||
_LIB = Path(__file__).resolve().parent
|
||||
if str(_LIB) not in sys.path:
|
||||
sys.path.insert(0, str(_LIB))
|
||||
# Importing a sibling would otherwise litter conformance/lib/__pycache__/ into the working
|
||||
# tree, and the lint stage scans untracked files. Same guard as status.py.
|
||||
sys.dont_write_bytecode = True
|
||||
import coverage as cov # noqa: E402 — reuse the registry/test parsers, don't fork them
|
||||
|
||||
CONF_ROOT = _LIB.parent
|
||||
REPO_ROOT = CONF_ROOT.parent
|
||||
DEFAULT_LEDGER = CONF_ROOT / "runs" / "ledger.jsonl"
|
||||
|
||||
# The ONLY layers exempt from needing a pass sentinel: the offline ones, whose skips
|
||||
# lib/run_l1_offline.py already renders as JUnit <skipped/> rather than as a green pass
|
||||
# (see docstring). Written as an EXEMPT set, not a REQUIRED set, so that an unrecognised
|
||||
# or new layer fails CLOSED — it needs a sentinel — instead of quietly becoming provable
|
||||
# by JUnit alone.
|
||||
SENTINEL_EXEMPT_LAYERS = frozenset({"L0", "L1"})
|
||||
|
||||
# Layer from the test's subtree. Authoritative over the annotation: coverage.py's layer
|
||||
# regex stops at the first non-[L/A/M/0-9] character, so `layer: "META"` parses as "M" —
|
||||
# harmless for its own report, but here a mis-read layer decides whether a sentinel is
|
||||
# required at all, so it is derived from the path exactly as emit_skipped_junit.py does.
|
||||
_LAYER_BY_SUBTREE = (("/l0-static/", "L0"), ("/l1-rendering/", "L1"),
|
||||
("/l2-behavioral/", "L2"), ("/l3-fault-injection/", "L3"),
|
||||
("/meta/", "META"))
|
||||
|
||||
|
||||
def layer_of(path: Path) -> str:
|
||||
rel = str(path)
|
||||
for frag, layer in _LAYER_BY_SUBTREE:
|
||||
if frag in rel:
|
||||
return layer
|
||||
return ""
|
||||
|
||||
# A declared sentinel: `contains($stdout, 'TOKEN')` inside a Chainsaw `check:` block.
|
||||
# Both quote styles are accepted; the repo uses single quotes throughout.
|
||||
_SENTINEL_RE = re.compile(r"contains\(\s*\$stdout\s*,\s*['\"]([A-Za-z0-9_]+)['\"]\s*\)")
|
||||
|
||||
# Naming convention for a SKIP token (tests/l3-fault-injection/*: NODE_LOSS_SKIPPED_OK,
|
||||
# DR_RESTORE_SKIPPED_OK, DR_SCHEDULE_SKIPPED_OK, VM_DATA_SURVIVAL_SKIPPED_OK, ...).
|
||||
# Deliberately broad: mis-classifying a PASS token as a skip only ever loses a proof;
|
||||
# mis-classifying a SKIP token as a pass would manufacture one.
|
||||
_SKIP_TOKEN_RE = re.compile(r"_SKIP")
|
||||
|
||||
# A Chainsaw log line header, as it really appears (Go `t.Log` prefixes the source site and
|
||||
# pads it with backspaces, and the fields are ANSI-coloured — both are stripped first):
|
||||
# l.go:52: | 23:51:18 | node-loss | volume-stays-available... | SCRIPT | LOG |
|
||||
# Field 2 is the test name; it is how output is attributed back to a testcase. The
|
||||
# `<file>.go:<line>:` prefix is optional so a redirected/plain log still attributes.
|
||||
_LOG_HEADER_RE = re.compile(
|
||||
r"^\s*(?:[\w./-]+\.go:\d+:\s*)?"
|
||||
r"\|\s*\d{1,2}:\d{2}:\d{2}(?:\.\d+)?\s*\|\s*([A-Za-z0-9][A-Za-z0-9._-]*)\s*\|")
|
||||
|
||||
# ANSI colour + the backspace padding Go writes ahead of a `t.Log` continuation. Stripped
|
||||
# before ANY parsing; neither ever appears at the head of a line, so indentation survives.
|
||||
_ANSI_RE = re.compile(r"\x1b\[[0-9;]*[A-Za-z]")
|
||||
|
||||
|
||||
def _strip_noise(line: str) -> str:
|
||||
return _ANSI_RE.sub("", line).replace("\b", "")
|
||||
|
||||
|
||||
# ===================================================================== output sections
|
||||
# THE BUG THIS SECTION MACHINERY EXISTS TO KILL — READ BEFORE CHANGING ANYTHING
|
||||
# -----------------------------------------------------------------------------
|
||||
# Chainsaw echoes the script it is ABOUT TO RUN, verbatim, in an indented `=== COMMAND`
|
||||
# block, and only afterwards prints what the script actually wrote in `=== STDOUT`. Every
|
||||
# skip-gated test contains its own skip token as SOURCE TEXT (`echo NODE_LOSS_SKIPPED_OK`),
|
||||
# so a raw whole-log grep finds that token in EVERY run of that test — including runs that
|
||||
# never took the skip branch. Combined with decide()'s (correct) rule that a skip token
|
||||
# outranks a pass token, a section-blind scan can never record `proven` for ANY test that
|
||||
# has a skip gate: the ledger fails closed so hard it records nothing. Observed on
|
||||
# runs/20260901T235117Z-e9a4b66, where NODE_LOSS_SKIPPED_OK appears twice as source at
|
||||
# lines 51/62 (`=== COMMAND`) and never in output, yet the run genuinely proved R-STATE-02.
|
||||
#
|
||||
# So tokens are attributed to SECTIONS, never to raw lines:
|
||||
# l.go:52: | 23:51:18 | node-loss | step | CMD | RUN | <- header, no section
|
||||
# === COMMAND <- marker (indent 8)
|
||||
# /usr/bin/sh -c set -euo pipefail <- source, indent >= 8
|
||||
# echo NODE_LOSS_SKIPPED_OK <- SOURCE, not output
|
||||
# l.go:52: | 23:52:42 | node-loss | step | SCRIPT | LOG | <- indent 4 => section ends
|
||||
# === STDOUT <- marker
|
||||
# NODE_LOSS_VOLUME_AVAILABLE_OK <- the REAL emission
|
||||
#
|
||||
# A marker is an INDENTED `=== NAME` on a line of its own. Go's own test markers
|
||||
# (`=== RUN chainsaw`, `=== PAUSE`, `=== CONT`) are at column 0 and carry a test name
|
||||
# after the verb, so they match neither the indent nor the end-of-line anchor; they are
|
||||
# additionally named in _GO_TEST_MARKERS so a future indented variant still cannot open a
|
||||
# section that would swallow the whole log.
|
||||
_SECTION_MARKER_RE = re.compile(r"^(\s+)=== ([A-Z][A-Z0-9_]*)\s*$")
|
||||
_GO_TEST_MARKERS = frozenset({"RUN", "PAUSE", "CONT"})
|
||||
|
||||
# Pseudo-section for a log that contains NO section markers at all (the plain/redirected
|
||||
# logs the earlier ledger rows were written from, and the selftest fixtures). Such a log
|
||||
# never echoes script source, so there is nothing to be confused by and the legacy
|
||||
# whole-log scan is exactly as safe as it was — narrowing is only needed where a COMMAND
|
||||
# echo exists. Lines OUTSIDE a section in a log that DOES use sections are attributed to
|
||||
# "" and are evidence for nothing.
|
||||
UNSECTIONED = "*"
|
||||
|
||||
# A pass token counts ONLY where Chainsaw's own `contains($stdout, ...)` check looks:
|
||||
# real stdout. Anything else — source echo, stderr, an unrecognised future section — can
|
||||
# never manufacture a proof.
|
||||
PASS_SECTIONS = frozenset({"STDOUT", UNSECTIONED})
|
||||
|
||||
# Sections that carry SOURCE rather than output. A skip token is honoured in every OTHER
|
||||
# section (STDOUT, STDERR, any section name added by a later Chainsaw release), because
|
||||
# over-reading a skip only ever COSTS a proof, while over-reading a pass would fabricate
|
||||
# one. Unknown section names therefore fail closed in both directions.
|
||||
SOURCE_SECTIONS = frozenset({"COMMAND"})
|
||||
|
||||
|
||||
def classify_sections(log_text: str) -> list[tuple[str, str]]:
|
||||
"""[(section, line)] for every line of a run log.
|
||||
|
||||
`section` is the enclosing `=== NAME` block, "" for a line inside no block, or
|
||||
UNSECTIONED for every line when the log declares no sections at all. A block runs from
|
||||
its marker until the first non-blank line indented LESS than the marker (blank lines do
|
||||
not end it) or until the next marker; the marker line itself is content of nothing.
|
||||
"""
|
||||
rows: list[tuple[str, str]] = []
|
||||
cur, cur_indent, sectioned = "", 0, False
|
||||
for raw in log_text.splitlines():
|
||||
line = _strip_noise(raw)
|
||||
m = _SECTION_MARKER_RE.match(line)
|
||||
if m and m.group(2) not in _GO_TEST_MARKERS:
|
||||
sectioned = True
|
||||
cur, cur_indent = m.group(2), len(m.group(1))
|
||||
rows.append(("", line))
|
||||
continue
|
||||
if cur:
|
||||
if not line.strip() or len(line) - len(line.lstrip()) >= cur_indent:
|
||||
rows.append((cur, line))
|
||||
continue
|
||||
cur = ""
|
||||
rows.append(("", line))
|
||||
if not sectioned:
|
||||
return [(UNSECTIONED, line) for _, line in rows]
|
||||
return rows
|
||||
|
||||
|
||||
def scan_texts(lines) -> tuple[str, str]:
|
||||
"""(text a PASS token may be read from, text a SKIP token may be read from).
|
||||
|
||||
Accepts either [(section, line)] pairs from classify_sections() or a plain list of
|
||||
strings, which is treated as an unsectioned log (legacy whole-log scanning).
|
||||
"""
|
||||
pass_lines, skip_lines = [], []
|
||||
for item in lines:
|
||||
section, line = item if isinstance(item, tuple) else (UNSECTIONED, item)
|
||||
if section in PASS_SECTIONS:
|
||||
pass_lines.append(line)
|
||||
if section and section not in SOURCE_SECTIONS:
|
||||
skip_lines.append(line)
|
||||
return "\n".join(pass_lines), "\n".join(skip_lines)
|
||||
|
||||
_RUN_ID_TS_RE = re.compile(r"^(\d{8}T\d{6}Z)-")
|
||||
|
||||
|
||||
# ========================================================================= sentinels
|
||||
def parse_sentinels(test_yaml: Path) -> dict:
|
||||
"""Declared sentinel tokens of one Chainsaw test, derived from the test file itself.
|
||||
|
||||
Returns {"pass": [...], "skip": [...], "terminal_pass": str|None}. `terminal_pass` is
|
||||
the LAST non-skip token in file order — the test's verdict token, i.e. the one its
|
||||
final `check:` block asserts (dr-restore -> DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK).
|
||||
A test with no `contains($stdout, ...)` assertion declares no sentinel and is reported
|
||||
unprovable rather than proven; see decide().
|
||||
"""
|
||||
try:
|
||||
text = test_yaml.read_text()
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return {"pass": [], "skip": [], "terminal_pass": None}
|
||||
toks = _SENTINEL_RE.findall(text)
|
||||
passes, skips = [], []
|
||||
for t in toks:
|
||||
(skips if _SKIP_TOKEN_RE.search(t) else passes).append(t)
|
||||
# De-duplicate, preserving order (dr-restore names DR_RESTORE_SKIPPED_OK twice).
|
||||
def uniq(xs: list[str]) -> list[str]:
|
||||
return list(dict.fromkeys(xs))
|
||||
passes, skips = uniq(passes), uniq(skips)
|
||||
return {"pass": passes, "skip": skips,
|
||||
"terminal_pass": passes[-1] if passes else None}
|
||||
|
||||
|
||||
def sentinel_map(tests_dir: Path) -> dict[str, dict]:
|
||||
"""{test-name: sentinels} for every test in the tree.
|
||||
|
||||
Keyed on the Chainsaw `metadata.name`, which is what lands in the JUnit `<testcase
|
||||
name=...>`. This repo keeps that equal to the test's directory basename; the mapping
|
||||
is by name, so it stays correct if that ever drifts.
|
||||
"""
|
||||
out: dict[str, dict] = {}
|
||||
for f in sorted(tests_dir.rglob("chainsaw-test.yaml")):
|
||||
meta = cov.parse_test(f)
|
||||
name = meta["name"] or f.parent.name
|
||||
s = parse_sentinels(f)
|
||||
s["dir"] = str(f.parent)
|
||||
s["layer"] = layer_of(f) or meta["layer"]
|
||||
s["requirements"] = meta["labels"]
|
||||
out[name] = s
|
||||
return out
|
||||
|
||||
|
||||
# =============================================================================== log
|
||||
def attribute_log(log_text: str, known: set[str]) -> tuple[dict[str, list[tuple[str, str]]],
|
||||
bool]:
|
||||
"""Split a run log into per-testcase buckets of (section, line) pairs.
|
||||
|
||||
Chainsaw prefixes its log lines with `| HH:MM:SS | <test> | <step> | ...` and prints a
|
||||
script's sections underneath that header, so the test name is carried by a state
|
||||
machine: a header naming a KNOWN test switches the current bucket, every following line
|
||||
lands in it. Requiring the name to be known is what stops run.sh's own `[run.sh]`
|
||||
chatter (and any other pipe-shaped line) from opening a bogus bucket.
|
||||
|
||||
Section labels come from classify_sections() and travel WITH the lines, so a bucket
|
||||
still knows which of its lines were script source and which were real output.
|
||||
|
||||
Returns (buckets, attributed) where `attributed` is False when the log contained no
|
||||
recognisable header at all — the signal for the degraded global scan in decide().
|
||||
"""
|
||||
buckets: dict[str, list[tuple[str, str]]] = {}
|
||||
cur: str | None = None
|
||||
attributed = False
|
||||
for section, line in classify_sections(log_text):
|
||||
m = _LOG_HEADER_RE.match(line)
|
||||
if m and m.group(1) in known:
|
||||
cur = m.group(1)
|
||||
attributed = True
|
||||
if cur is not None:
|
||||
buckets.setdefault(cur, []).append((section, line))
|
||||
return buckets, attributed
|
||||
|
||||
|
||||
def unique_tokens(smap: dict[str, dict]) -> set[str]:
|
||||
"""Tokens claimed by EXACTLY ONE test in the tree.
|
||||
|
||||
Only these may be used by the degraded global scan: `DNS_OK` is echoed by two different
|
||||
tests, so in an unattributable log it can never be evidence for either; but
|
||||
`NODE_LOSS_VOLUME_AVAILABLE_OK` names precisely one test and still can.
|
||||
"""
|
||||
owners: dict[str, set[str]] = {}
|
||||
for name, s in smap.items():
|
||||
for t in s["pass"] + s["skip"]:
|
||||
owners.setdefault(t, set()).add(name)
|
||||
return {t for t, o in owners.items() if len(o) == 1}
|
||||
|
||||
|
||||
# ============================================================================ decide
|
||||
def decide(layer: str, junit_result: str, sent: dict, lines: list[str],
|
||||
scope: str) -> tuple[str | None, str, str]:
|
||||
"""The ONE place a testcase becomes `proven` — or, far more often, does not.
|
||||
|
||||
Returns (sentinel, proof, note).
|
||||
|
||||
Ordering is worst-first ON PURPOSE, so that ambiguity can only ever cost a proof:
|
||||
1. JUnit failure/error -> failed (a red run is recorded as red)
|
||||
2. JUnit <skipped/> -> junit-skipped (never green; ci/lib.sh rule)
|
||||
3. ANY declared skip token seen -> skipped (BEATS a green JUnit and beats
|
||||
IN REAL OUTPUT the pass token: if any gated
|
||||
step self-skipped, the run did
|
||||
not prove the requirement)
|
||||
4. no sentinel declared -> proven only for L0/L1, whose skips are already
|
||||
JUnit skips; unknown-no-sentinel-declared for
|
||||
the Chainsaw layers
|
||||
5. EVERY declared pass token seen -> proven
|
||||
6. anything else -> unknown-sentinel-absent
|
||||
Step 5 requires ALL pass tokens, not just the terminal one: every Chainsaw step here
|
||||
asserts `pass || skip`, so a missing pass token means a step did not reach its proven
|
||||
branch, whatever the JUnit says.
|
||||
|
||||
Steps 3 and 5 read from SECTION-SCOPED text, never from raw lines: a pass token counts
|
||||
only in `=== STDOUT`, a skip token counts in any section that is not script source.
|
||||
That narrows WHERE a token is read from and changes nothing about what it means — skip
|
||||
still outranks pass, and it outranks it within genuine output, which is the only place
|
||||
either of them was ever evidence. `lines` may be [(section, line)] pairs or plain
|
||||
strings (an unsectioned log; see scan_texts / UNSECTIONED).
|
||||
"""
|
||||
if junit_result in ("failed", "error"):
|
||||
return (None, "failed", f"JUnit recorded {junit_result}")
|
||||
if junit_result == "skipped":
|
||||
return (None, "junit-skipped",
|
||||
"JUnit <skipped/> — the test did not execute; a skip is never green")
|
||||
|
||||
pass_text, skip_text = scan_texts(lines)
|
||||
seen_skip = [t for t in sent["skip"] if t in skip_text]
|
||||
if seen_skip:
|
||||
return (seen_skip[0], "skipped",
|
||||
f"run log OUTPUT carries the SELF-SKIP token {seen_skip[0]!r} — the test "
|
||||
f"exited 0 without exercising the behaviour, so this is NOT proof")
|
||||
|
||||
if not sent["pass"] and not sent["skip"]:
|
||||
if layer not in SENTINEL_EXEMPT_LAYERS:
|
||||
return (None, "unknown-no-sentinel-declared",
|
||||
f"{layer or 'unknown-layer'} test declares no contains($stdout, ...) "
|
||||
f"sentinel, so a green JUnit cannot be told apart from a self-skip — "
|
||||
f"refusing to call it proven")
|
||||
return (None, "proven",
|
||||
f"{layer} offline check exited 0; run_l1_offline.py renders its skips as "
|
||||
f"JUnit <skipped/>, so a green JUnit is truthful without a sentinel")
|
||||
|
||||
missing = [t for t in sent["pass"] if t not in pass_text]
|
||||
if not missing:
|
||||
return (sent["terminal_pass"], "proven",
|
||||
f"run log OUTPUT carries every declared pass token "
|
||||
f"({', '.join(sent['pass'])}) and no skip token [{scope}]")
|
||||
return (None, "unknown-sentinel-absent",
|
||||
f"run log OUTPUT lacks pass token(s) {', '.join(missing)} for this test "
|
||||
f"[{scope}] — refusing to infer proof from the JUnit alone")
|
||||
|
||||
|
||||
# ============================================================================= junit
|
||||
def junit_cases(path: Path) -> list[dict]:
|
||||
"""[{name, layer, requirements, result}] for every <testcase> in a JUnit report."""
|
||||
root = ET.parse(path).getroot()
|
||||
cases: list[dict] = []
|
||||
for c in root.iter("testcase"):
|
||||
if c.find("failure") is not None:
|
||||
result = "failed"
|
||||
elif c.find("error") is not None:
|
||||
result = "error"
|
||||
elif c.find("skipped") is not None:
|
||||
result = "skipped"
|
||||
else:
|
||||
result = "passed"
|
||||
props = {p.get("name"): (p.get("value") or "")
|
||||
for p in c.iter("property") if p.get("name")}
|
||||
reqs = sorted(set(cov.RID_RE.findall(props.get("requirements", ""))))
|
||||
layer = (props.get("layer") or "").upper()
|
||||
if not layer:
|
||||
layer = (c.get("classname") or "").split(":", 1)[0].upper()
|
||||
cases.append({"name": c.get("name") or "?", "layer": layer,
|
||||
"requirements": reqs, "result": result})
|
||||
return cases
|
||||
|
||||
|
||||
# =============================================================================== git
|
||||
def git_sha(repo: Path) -> str:
|
||||
"""HEAD, suffixed `-dirty` when TRACKED files differ from it.
|
||||
|
||||
Untracked files are deliberately ignored, matching status.py's `tracked_files()`
|
||||
posture (untracked scratch is not repo state) — and because a `--record` run creates
|
||||
untracked `runs/<id>/` files as it goes, which would otherwise mark every run dirty.
|
||||
A `-dirty` sha is not a valid rev, so status.py's ancestry check cannot resolve it and
|
||||
the proof degrades to `stale-proof`: correct, since what ran is not what is committed.
|
||||
"""
|
||||
try:
|
||||
sha = subprocess.run(["git", "-C", str(repo), "rev-parse", "HEAD"],
|
||||
capture_output=True, text=True, check=True).stdout.strip()
|
||||
except (subprocess.CalledProcessError, OSError):
|
||||
return "unknown"
|
||||
# EXCLUDE conformance/runs/. It is TRACKED, so `--record` writing ledger.jsonl dirtied the
|
||||
# tree and every LATER run in the same session was stamped `-dirty` — recording one test
|
||||
# poisoned the next, and check.sh refuses to honour a dirty proven row. That cost a second
|
||||
# live fault-injection run on prod to work around. The docstring above already states the
|
||||
# intent for the untracked runs/<id>/ half of this same artifact; the ledger is the tracked
|
||||
# half. Mirrors status.py's IMPL_EXCLUDE_PREFIXES, which lists this path for the same reason.
|
||||
dirty = subprocess.run(["git", "-C", str(repo), "diff", "--quiet", "HEAD", "--",
|
||||
".", ":(exclude)conformance/runs/"],
|
||||
capture_output=True).returncode != 0
|
||||
return f"{sha}-dirty" if dirty else sha
|
||||
|
||||
|
||||
def pinned_chainsaw_version(conf_root: Path) -> str:
|
||||
"""CHAINSAW_VERSION from versions.env — the pin is the single source of truth."""
|
||||
f = conf_root / "versions.env"
|
||||
if f.is_file():
|
||||
m = re.search(r'^CHAINSAW_VERSION="?([^"\n]+)"?', f.read_text(), re.M)
|
||||
if m:
|
||||
return m.group(1)
|
||||
return "unknown"
|
||||
|
||||
|
||||
def context_env() -> str:
|
||||
"""The env a run TARGETED, derived from the live kube context — never guessed.
|
||||
|
||||
WHY THIS IS NOT A DEFAULT STRING. `--env` used to fall back to the literal "dev"
|
||||
(`os.environ.get("CONF_ENV", "dev")`), and CONF_ENV is set nowhere in this repo, so EVERY
|
||||
recorded proof was stamped `env: dev` regardless of the cluster it ran against. That was
|
||||
harmless while dev was the only cluster. Dev was decommissioned 2026-09-12 and prod is now
|
||||
the only one, so on 2026-09-13 a genuine prod DR-restore proof was recorded as `env: dev` —
|
||||
a proof that lies about where it was proven is worse than no proof, because the ledger's
|
||||
whole purpose is to say what was demonstrated WHERE. Caught before it was committed.
|
||||
|
||||
Derived from `kubectl config current-context` (`k3s-prod` -> `prod`), which cannot drift
|
||||
from the cluster the run actually talked to. Undeterminable is "unknown", NEVER a cluster
|
||||
name: an unreadable context must not silently attribute a proof to an environment.
|
||||
"""
|
||||
try:
|
||||
p = subprocess.run(["kubectl", "config", "current-context"],
|
||||
capture_output=True, text=True, timeout=15)
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
return "unknown"
|
||||
if p.returncode != 0:
|
||||
return "unknown"
|
||||
ctx = (p.stdout or "").strip()
|
||||
if not ctx:
|
||||
return "unknown"
|
||||
# contexts in this repo are named k3s-<env>; anything else is recorded verbatim so an
|
||||
# unexpected context is visible in the ledger rather than being coerced into a known name.
|
||||
return ctx[len("k3s-"):] if ctx.startswith("k3s-") else ctx
|
||||
|
||||
|
||||
def server_kube_version() -> str:
|
||||
"""Server gitVersion via `kubectl version` (read-only), else "unknown"."""
|
||||
for args in (["kubectl", "version", "-o", "json"],
|
||||
["kubectl", "version", "--output=json"]):
|
||||
try:
|
||||
p = subprocess.run(args, capture_output=True, text=True, timeout=15)
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
continue
|
||||
if p.returncode != 0 or not p.stdout.strip():
|
||||
continue
|
||||
try:
|
||||
v = json.loads(p.stdout).get("serverVersion") or {}
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if v.get("gitVersion"):
|
||||
return v["gitVersion"]
|
||||
return "unknown"
|
||||
|
||||
|
||||
def run_timestamp(run_dir: Path, junit: Path) -> str:
|
||||
"""UTC stamp: the run-dir name's own `<UTC>-<sha>` prefix, else the JUnit's mtime."""
|
||||
m = _RUN_ID_TS_RE.match(run_dir.name)
|
||||
if m:
|
||||
t = m.group(1)
|
||||
return f"{t[0:4]}-{t[4:6]}-{t[6:8]}T{t[9:11]}:{t[11:13]}:{t[13:15]}Z"
|
||||
return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(junit.stat().st_mtime))
|
||||
|
||||
|
||||
# ============================================================================ record
|
||||
def build_rows(run_dir: Path, tests_dir: Path, env: str, chainsaw_version: str,
|
||||
kube_version: str, sha: str) -> list[dict]:
|
||||
junit = run_dir / "report.junit.xml"
|
||||
log = run_dir / "run.log"
|
||||
smap = sentinel_map(tests_dir)
|
||||
log_text = log.read_text(errors="replace")
|
||||
buckets, attributed = attribute_log(log_text, set(smap))
|
||||
uniq = unique_tokens(smap)
|
||||
all_pairs = classify_sections(log_text)
|
||||
sectioned = any(sec not in ("", UNSECTIONED) for sec, _ in all_pairs)
|
||||
section_note = ("tokens read from real-output sections only"
|
||||
if sectioned else "log declares no === sections; whole-log scan")
|
||||
ts = run_timestamp(run_dir, junit)
|
||||
|
||||
rows: list[dict] = []
|
||||
for case in junit_cases(junit):
|
||||
name = case["name"]
|
||||
sent = smap.get(name, {"pass": [], "skip": [], "terminal_pass": None,
|
||||
"layer": case["layer"], "requirements": []})
|
||||
layer = sent.get("layer") or case["layer"] or "?"
|
||||
# Requirements come from the test's LABELS (coverage.py's authoritative source);
|
||||
# the JUnit property is the fallback for a testcase with no matching test file.
|
||||
reqs = sent.get("requirements") or case["requirements"]
|
||||
|
||||
if name in buckets:
|
||||
lines, scope = buckets[name], f"log scoped to this testcase; {section_note}"
|
||||
elif not attributed:
|
||||
# Degraded mode: the log had no recognisable Chainsaw header (a redirected or
|
||||
# reformatted log). Fall back to a whole-log scan, but ONLY for tokens owned by
|
||||
# exactly one test, so a token another test emitted can never prove this one.
|
||||
lines = all_pairs
|
||||
sent = dict(sent,
|
||||
**{"pass": [t for t in sent["pass"] if t in uniq],
|
||||
"skip": [t for t in sent["skip"] if t in uniq]})
|
||||
sent["terminal_pass"] = sent["pass"][-1] if sent["pass"] else None
|
||||
scope = ("DEGRADED: unattributable log, globally-unique tokens only; "
|
||||
+ section_note)
|
||||
else:
|
||||
lines, scope = [], "no log lines attributed to this testcase"
|
||||
|
||||
sentinel, proof, note = decide(layer, case["result"], sent, lines, scope)
|
||||
rows.append({
|
||||
"ts_utc": ts,
|
||||
"git_sha": sha,
|
||||
"env": env,
|
||||
"chainsaw_version": chainsaw_version,
|
||||
"kube_version": kube_version,
|
||||
"layer": layer,
|
||||
"test": name,
|
||||
"requirements": reqs,
|
||||
"junit_result": case["result"],
|
||||
"sentinel": sentinel,
|
||||
"proof": proof,
|
||||
"run_id": run_dir.name,
|
||||
"note": note,
|
||||
})
|
||||
return rows
|
||||
|
||||
|
||||
def append_rows(ledger: Path, rows: list[dict]) -> None:
|
||||
"""Append-only, one compact JSON object per line. Never rewrites existing rows."""
|
||||
ledger.parent.mkdir(parents=True, exist_ok=True)
|
||||
with ledger.open("a", encoding="utf-8") as fh:
|
||||
for r in rows:
|
||||
fh.write(json.dumps(r, sort_keys=True) + "\n")
|
||||
|
||||
|
||||
# ========================================================================== selftest
|
||||
def _selftest() -> int:
|
||||
"""Deterministic, no-cluster self-test of every decision rule (house pattern:
|
||||
`status.py --selftest`, `catalog/render.py --selftest`). Throwaway tree only.
|
||||
|
||||
The load-bearing case is `SKIP sentinel does NOT produce proven`: it is the single
|
||||
assertion standing between this ledger and a permanent false proof of R-STATE-03.
|
||||
"""
|
||||
import tempfile
|
||||
ok = True
|
||||
|
||||
def check(label: str, cond: bool) -> None:
|
||||
nonlocal ok
|
||||
if not cond:
|
||||
sys.stderr.write(f"record_run.py selftest: FAIL — {label}\n")
|
||||
ok = False
|
||||
|
||||
# --- token derivation, against the dr-restore/node-loss shape ---------------------
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
root = Path(td)
|
||||
tdir = root / "tests" / "l3-fault-injection" / "dr-restore"
|
||||
tdir.mkdir(parents=True)
|
||||
(tdir / "chainsaw-test.yaml").write_text(
|
||||
"apiVersion: chainsaw.kyverno.io/v1alpha1\nkind: Test\nmetadata:\n"
|
||||
" name: dr-restore\n labels:\n"
|
||||
" conformance.home.k3s/R-STATE-03: \"true\"\n annotations:\n"
|
||||
" conformance.home.k3s/requirements: \"R-STATE-03\"\n"
|
||||
" conformance.home.k3s/layer: \"L3\"\nspec:\n steps:\n"
|
||||
" - script:\n content: 'echo OFF_CLUSTER_TARGET_OK'\n"
|
||||
" check:\n"
|
||||
" (contains($stdout, 'OFF_CLUSTER_TARGET_OK')"
|
||||
" || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true\n"
|
||||
" - script:\n content: 'echo x'\n check:\n"
|
||||
" (contains($stdout, 'DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK')"
|
||||
" || contains($stdout, 'DR_RESTORE_SKIPPED_OK')): true\n")
|
||||
s = parse_sentinels(tdir / "chainsaw-test.yaml")
|
||||
check("pass tokens derived from the test file",
|
||||
s["pass"] == ["OFF_CLUSTER_TARGET_OK",
|
||||
"DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"])
|
||||
check("skip token classified by the *_SKIP*_OK convention, de-duplicated",
|
||||
s["skip"] == ["DR_RESTORE_SKIPPED_OK"])
|
||||
check("terminal pass token is the LAST non-skip token",
|
||||
s["terminal_pass"] == "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK")
|
||||
|
||||
smap = sentinel_map(root / "tests")
|
||||
check("sentinel_map keys on metadata.name and carries layer + labels",
|
||||
smap["dr-restore"]["layer"] == "L3"
|
||||
and smap["dr-restore"]["requirements"] == ["R-STATE-03"])
|
||||
|
||||
# --- decide(): the SKIP-is-never-proven contract ----------------------------------
|
||||
l3 = {"pass": ["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"],
|
||||
"skip": ["DR_RESTORE_SKIPPED_OK"],
|
||||
"terminal_pass": "DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"}
|
||||
|
||||
# *** THE CASE THIS WHOLE FILE EXISTS FOR ***
|
||||
sent, proof, _ = decide("L3", "passed", l3, ["DR_RESTORE_SKIPPED_OK"], "t")
|
||||
check("SKIP sentinel on a GREEN JUnit is 'skipped', NEVER 'proven'",
|
||||
proof == "skipped" and sent == "DR_RESTORE_SKIPPED_OK")
|
||||
|
||||
_, proof, _ = decide("L3", "passed", l3,
|
||||
["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"], "t")
|
||||
check("pass sentinel on a green JUnit is 'proven'", proof == "proven")
|
||||
|
||||
_, proof, _ = decide("L3", "passed", l3,
|
||||
["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK",
|
||||
"DR_RESTORE_SKIPPED_OK"], "t")
|
||||
check("a skip token BEATS a pass token in the same log (worst wins)",
|
||||
proof == "skipped")
|
||||
|
||||
_, proof, _ = decide("L3", "passed", l3, ["nothing relevant here"], "t")
|
||||
check("green JUnit with no sentinel in the log is unknown, not proven",
|
||||
proof == "unknown-sentinel-absent")
|
||||
|
||||
_, proof, _ = decide("L3", "passed", l3, [], "t")
|
||||
check("green JUnit with NO log lines attributed is unknown, not proven",
|
||||
proof == "unknown-sentinel-absent")
|
||||
|
||||
_, proof, _ = decide("L3", "failed", l3,
|
||||
["DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK"], "t")
|
||||
check("a failing JUnit outranks a pass token in the log", proof == "failed")
|
||||
|
||||
_, proof, _ = decide("L3", "skipped", l3, [], "t")
|
||||
check("JUnit <skipped/> records junit-skipped, never green",
|
||||
proof == "junit-skipped")
|
||||
|
||||
multi = {"pass": ["A_OK", "B_OK"], "skip": [], "terminal_pass": "B_OK"}
|
||||
_, proof, _ = decide("L3", "passed", multi, ["B_OK"], "t")
|
||||
check("every declared pass token must be present, not just the terminal one",
|
||||
proof == "unknown-sentinel-absent")
|
||||
_, proof, _ = decide("L3", "passed", multi, ["A_OK", "B_OK"], "t")
|
||||
check("all pass tokens present == proven", proof == "proven")
|
||||
|
||||
none = {"pass": [], "skip": [], "terminal_pass": None}
|
||||
_, proof, _ = decide("L2", "passed", none, ["whatever"], "t")
|
||||
check("a Chainsaw test declaring no sentinel is NOT proven by a green JUnit",
|
||||
proof == "unknown-no-sentinel-declared")
|
||||
_, proof, _ = decide("L1", "passed", none, [], "t")
|
||||
check("L0/L1 offline checks are proven by JUnit (their skips are JUnit skips)",
|
||||
proof == "proven")
|
||||
_, proof, _ = decide("L0", "skipped", none, [], "t")
|
||||
check("an L0 JUnit skip is still never green", proof == "junit-skipped")
|
||||
_, proof, _ = decide("META", "passed", none, [], "t")
|
||||
check("META (a Chainsaw layer) needs a sentinel like L2/L3",
|
||||
proof == "unknown-no-sentinel-declared")
|
||||
_, proof, _ = decide("", "passed", none, [], "t")
|
||||
check("an unknown/blank layer fails CLOSED — sentinel required",
|
||||
proof == "unknown-no-sentinel-declared")
|
||||
|
||||
# --- SECTION SCOPING: the COMMAND echo is SOURCE, never evidence -------------------
|
||||
# Regression cases for the ledger's first real L3 run, which recorded a genuine PASS as
|
||||
# `skipped` because `echo NODE_LOSS_SKIPPED_OK` — the skip branch NOT taken — is echoed
|
||||
# inside `=== COMMAND` on every run of a skip-gated test.
|
||||
nl = {"pass": ["NODE_LOSS_VOLUME_AVAILABLE_OK"],
|
||||
"skip": ["NODE_LOSS_SKIPPED_OK"],
|
||||
"terminal_pass": "NODE_LOSS_VOLUME_AVAILABLE_OK"}
|
||||
_CMD = (" l.go:52: | 23:51:18 | node-loss | step | CMD | RUN |\n"
|
||||
" === COMMAND\n"
|
||||
" /usr/bin/sh -c set -euo pipefail\n"
|
||||
" if ! kubectl get storageclass longhorn >/dev/null 2>&1; then\n"
|
||||
" echo NODE_LOSS_SKIPPED_OK\n"
|
||||
"\n"
|
||||
" exit 0\n"
|
||||
" fi\n"
|
||||
" echo NODE_LOSS_VOLUME_AVAILABLE_OK\n"
|
||||
" l.go:52: | 23:52:42 | node-loss | step | SCRIPT | LOG |\n")
|
||||
|
||||
ran = classify_sections(_CMD + " === STDOUT\n"
|
||||
" ready nodes: 3\n"
|
||||
" NODE_LOSS_VOLUME_AVAILABLE_OK\n"
|
||||
" l.go:52: | 23:52:42 | node-loss | step | SCRIPT | DONE |\n")
|
||||
sent, proof, _ = decide("L3", "passed", nl, ran, "t")
|
||||
check("skip token ONLY in the COMMAND echo + pass token in STDOUT is PROVEN",
|
||||
proof == "proven" and sent == "NODE_LOSS_VOLUME_AVAILABLE_OK")
|
||||
|
||||
selfskipped = classify_sections(_CMD + " === STDOUT\n"
|
||||
" SKIP: 'longhorn' StorageClass absent\n"
|
||||
" NODE_LOSS_SKIPPED_OK\n")
|
||||
sent, proof, _ = decide("L3", "passed", nl, selfskipped, "t")
|
||||
check("skip token GENUINELY in STDOUT is still 'skipped' (the guarantee holds)",
|
||||
proof == "skipped" and sent == "NODE_LOSS_SKIPPED_OK")
|
||||
|
||||
neither = classify_sections(_CMD + " === STDOUT\n ready nodes: 3\n")
|
||||
_, proof, _ = decide("L3", "passed", nl, neither, "t")
|
||||
check("both tokens in COMMAND and neither in STDOUT is NOT proven",
|
||||
proof == "unknown-sentinel-absent")
|
||||
|
||||
both_out = classify_sections(" === STDOUT\n"
|
||||
" NODE_LOSS_VOLUME_AVAILABLE_OK\n"
|
||||
" NODE_LOSS_SKIPPED_OK\n")
|
||||
_, proof, _ = decide("L3", "passed", nl, both_out, "t")
|
||||
check("skip still BEATS pass when both are in genuine output", proof == "skipped")
|
||||
|
||||
stderr_skip = classify_sections(" === STDERR\n NODE_LOSS_SKIPPED_OK\n"
|
||||
" === STDOUT\n"
|
||||
" NODE_LOSS_VOLUME_AVAILABLE_OK\n")
|
||||
_, proof, _ = decide("L3", "passed", nl, stderr_skip, "t")
|
||||
check("a skip token in a non-source section (STDERR) still costs the proof",
|
||||
proof == "skipped")
|
||||
|
||||
stderr_pass = classify_sections(" === STDERR\n"
|
||||
" NODE_LOSS_VOLUME_AVAILABLE_OK\n")
|
||||
_, proof, _ = decide("L3", "passed", nl, stderr_pass, "t")
|
||||
check("a pass token OUTSIDE stdout cannot prove anything",
|
||||
proof == "unknown-sentinel-absent")
|
||||
|
||||
# --- classify_sections: markers, extents, and Go's own `=== RUN` --------------------
|
||||
gorows = classify_sections("=== RUN chainsaw\n=== PAUSE chainsaw\n"
|
||||
"=== CONT chainsaw/node-loss\n"
|
||||
" === COMMAND\n echo NODE_LOSS_SKIPPED_OK\n"
|
||||
" l.go:52: | 1:00:00 | node-loss | s | SCRIPT | DONE |\n"
|
||||
"--- PASS: chainsaw/node-loss (131.09s)\n")
|
||||
secs = [sec for sec, _ in gorows]
|
||||
check("Go's column-0 `=== RUN/PAUSE/CONT` never open a section",
|
||||
secs[:3] == ["", "", ""])
|
||||
check("a section covers its indented body and ends at a less-indented line",
|
||||
secs[3] == "" and secs[4] == "COMMAND" and secs[5] == "" and secs[6] == "")
|
||||
check("a log with no === markers stays wholly scannable (UNSECTIONED)",
|
||||
{sec for sec, _ in classify_sections("plain line\n another\n")} == {UNSECTIONED})
|
||||
|
||||
# --- the preserved run itself, as a fixture ----------------------------------------
|
||||
# runs/20260901T235117Z-e9a4b66 IS the bug: NODE_LOSS_SKIPPED_OK twice in `=== COMMAND`
|
||||
# (source), never in output; NODE_LOSS_VOLUME_AVAILABLE_OK once in `=== STDOUT`.
|
||||
fixture = CONF_ROOT / "runs" / "20260901T235117Z-e9a4b66"
|
||||
if (fixture / "run.log").is_file() and (fixture / "report.junit.xml").is_file():
|
||||
rows = build_rows(fixture, CONF_ROOT / "tests", "dev", "v0.2.12", "v1.36.4", "sha")
|
||||
row = next((r for r in rows if r["test"] == "node-loss"), None)
|
||||
check("preserved real L3 run records node-loss/R-STATE-02 as proven",
|
||||
row is not None and row["proof"] == "proven"
|
||||
and row["sentinel"] == "NODE_LOSS_VOLUME_AVAILABLE_OK"
|
||||
and row["requirements"] == ["R-STATE-02"])
|
||||
check("preserved real L3 run attributes (no DEGRADED fallback needed)",
|
||||
row is not None and "DEGRADED" not in row["note"])
|
||||
pairs = classify_sections((fixture / "run.log").read_text(errors="replace"))
|
||||
secs = {sec for sec, line in pairs if "NODE_LOSS_SKIPPED_OK" in line}
|
||||
check("every NODE_LOSS_SKIPPED_OK in the preserved log is COMMAND source",
|
||||
secs == {"COMMAND"})
|
||||
secs = {sec for sec, line in pairs
|
||||
if line.strip() == "NODE_LOSS_VOLUME_AVAILABLE_OK"}
|
||||
check("the real emission in the preserved log is in STDOUT", secs == {"STDOUT"})
|
||||
|
||||
# --- log attribution --------------------------------------------------------------
|
||||
log = ("[run.sh] Coverage report first\n"
|
||||
" | 20:41:32 | node-loss | step | SCRIPT | RUN |\n"
|
||||
" NODE_LOSS_SKIPPED_OK\n"
|
||||
" | 20:44:01 | dr-restore | step | SCRIPT | RUN |\n"
|
||||
" DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK\n")
|
||||
b, attributed = attribute_log(log, {"node-loss", "dr-restore"})
|
||||
check("log lines are attributed to the testcase whose header preceded them",
|
||||
attributed
|
||||
and any("NODE_LOSS_SKIPPED_OK" in l for _, l in b["node-loss"])
|
||||
and not any("NODE_LOSS_SKIPPED_OK" in l for _, l in b["dr-restore"]))
|
||||
check("a pass token does NOT leak across testcase boundaries",
|
||||
not any("DR_RESTORED" in l for _, l in b["node-loss"]))
|
||||
check("an unsectioned log keeps whole-log scanning (nothing narrowed away)",
|
||||
all(sec == UNSECTIONED for sec, _ in b["node-loss"]))
|
||||
|
||||
# Chainsaw's real headers carry a Go `t.Log` source prefix and ANSI colour; the earlier
|
||||
# regex matched neither, so every genuine Chainsaw log silently fell into DEGRADED mode.
|
||||
real_hdr = (" l.go:52: \x08\x08\x08| 23:51:18 | \x1b[32;1mnode-loss\x1b[0;22m | "
|
||||
"\x1b[32;1mstep\x1b[0;22m | SCRIPT | LOG |\n"
|
||||
" === STDOUT\n NODE_LOSS_VOLUME_AVAILABLE_OK\n")
|
||||
b2, attributed_real = attribute_log(real_hdr, {"node-loss"})
|
||||
check("a REAL Chainsaw header (go source prefix + ANSI) attributes, not degrades",
|
||||
attributed_real and "node-loss" in b2
|
||||
and ("STDOUT", " NODE_LOSS_VOLUME_AVAILABLE_OK") in b2["node-loss"])
|
||||
_, attributed2 = attribute_log("no chainsaw headers at all\n", {"node-loss"})
|
||||
check("an unattributable log is reported as such (degraded mode)", not attributed2)
|
||||
|
||||
smap2 = {"a": {"pass": ["DNS_OK", "A_ONLY_OK"], "skip": []},
|
||||
"b": {"pass": ["DNS_OK"], "skip": []}}
|
||||
check("only tokens owned by exactly one test may be used in degraded mode",
|
||||
unique_tokens(smap2) == {"A_ONLY_OK"})
|
||||
|
||||
# --- end-to-end: a self-skipped L3 run must land in the ledger as `skipped` --------
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
root = Path(td)
|
||||
tdir = root / "tests" / "l3-fault-injection" / "node-loss"
|
||||
tdir.mkdir(parents=True)
|
||||
(tdir / "chainsaw-test.yaml").write_text(
|
||||
"kind: Test\nmetadata:\n name: node-loss\n labels:\n"
|
||||
" conformance.home.k3s/R-STATE-02: \"true\"\n annotations:\n"
|
||||
" conformance.home.k3s/requirements: \"R-STATE-02\"\n"
|
||||
" conformance.home.k3s/layer: \"L3\"\nspec:\n steps:\n"
|
||||
" - script:\n content: 'echo hi'\n check:\n"
|
||||
" (contains($stdout, 'NODE_LOSS_VOLUME_AVAILABLE_OK')"
|
||||
" || contains($stdout, 'NODE_LOSS_SKIPPED_OK')): true\n")
|
||||
rd = root / "runs" / "20260901T120000Z-abc1234"
|
||||
rd.mkdir(parents=True)
|
||||
(rd / "report.junit.xml").write_text(
|
||||
'<testsuites><testsuite><testcase classname="L3:R-STATE-02" '
|
||||
'name="node-loss"><properties>'
|
||||
'<property name="requirements" value="R-STATE-02"/>'
|
||||
'<property name="layer" value="L3"/></properties></testcase>'
|
||||
"</testsuite></testsuites>")
|
||||
(rd / "run.log").write_text(
|
||||
" | 12:00:01 | node-loss | s | SCRIPT | RUN |\n"
|
||||
" SKIP: 'longhorn' StorageClass absent\n"
|
||||
" NODE_LOSS_SKIPPED_OK\n")
|
||||
rows = build_rows(rd, root / "tests", "dev", "v0.2.12", "v1.36.4", "abc1234")
|
||||
check("end-to-end: Chainsaw 'passed' + skip token => proof 'skipped'",
|
||||
len(rows) == 1 and rows[0]["junit_result"] == "passed"
|
||||
and rows[0]["proof"] == "skipped"
|
||||
and rows[0]["sentinel"] == "NODE_LOSS_SKIPPED_OK")
|
||||
check("end-to-end: row carries ts from the run-dir name and the R-ID label",
|
||||
rows[0]["ts_utc"] == "2026-09-01T12:00:00Z"
|
||||
and rows[0]["requirements"] == ["R-STATE-02"])
|
||||
|
||||
(rd / "run.log").write_text(
|
||||
" | 12:00:01 | node-loss | s | SCRIPT | RUN |\n"
|
||||
" NODE_LOSS_VOLUME_AVAILABLE_OK\n")
|
||||
rows = build_rows(rd, root / "tests", "dev", "v0.2.12", "v1.36.4", "abc1234")
|
||||
check("end-to-end: Chainsaw 'passed' + pass token => proof 'proven'",
|
||||
rows[0]["proof"] == "proven"
|
||||
and rows[0]["sentinel"] == "NODE_LOSS_VOLUME_AVAILABLE_OK")
|
||||
|
||||
ledger = root / "ledger.jsonl"
|
||||
append_rows(ledger, rows)
|
||||
append_rows(ledger, rows)
|
||||
check("ledger is append-only JSONL (one parsable object per line)",
|
||||
len(ledger.read_text().strip().splitlines()) == 2
|
||||
and json.loads(ledger.read_text().splitlines()[0])["test"] == "node-loss")
|
||||
|
||||
if not ok:
|
||||
return 1
|
||||
sys.stderr.write("record_run.py selftest: PASS\n")
|
||||
return 0
|
||||
|
||||
|
||||
# ============================================================================== main
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--run-dir", type=Path,
|
||||
help="conformance/runs/<UTC>-<sha>/ holding report.junit.xml + run.log")
|
||||
ap.add_argument("--ledger", type=Path, default=DEFAULT_LEDGER,
|
||||
help=f"append-only ledger (default {DEFAULT_LEDGER})")
|
||||
ap.add_argument("--tests", type=Path, default=CONF_ROOT / "tests")
|
||||
ap.add_argument("--repo", type=Path, default=REPO_ROOT)
|
||||
ap.add_argument("--env", default=os.environ.get("CONF_ENV") or context_env(),
|
||||
help="env the run targeted; derived from the kube context when "
|
||||
"unset (never defaulted to a cluster name)")
|
||||
ap.add_argument("--chainsaw-version", default=None,
|
||||
help="override the versions.env pin (recorded verbatim)")
|
||||
ap.add_argument("--kube-version", default=os.environ.get("CONF_KUBE_VERSION"),
|
||||
help="override the `kubectl version` probe (recorded verbatim)")
|
||||
ap.add_argument("--dry-run", action="store_true",
|
||||
help="print the rows that WOULD be appended; write nothing")
|
||||
ap.add_argument("--selftest", action="store_true", help="run the built-in self-test")
|
||||
args = ap.parse_args()
|
||||
|
||||
if args.selftest:
|
||||
return _selftest()
|
||||
if args.run_dir is None:
|
||||
sys.stderr.write("[record] --run-dir is required (or use --selftest)\n")
|
||||
return 1
|
||||
|
||||
run_dir = args.run_dir.resolve()
|
||||
junit, log = run_dir / "report.junit.xml", run_dir / "run.log"
|
||||
# BOTH files or nothing. The JUnit alone cannot tell a self-skip from a real pass, so
|
||||
# recording from it alone is exactly the false proof this tool exists to prevent.
|
||||
missing = [p.name for p in (junit, log) if not p.is_file()]
|
||||
if missing:
|
||||
sys.stderr.write(
|
||||
f"[record] {run_dir} is missing {', '.join(missing)} — nothing recorded. "
|
||||
f"The JUnit alone cannot distinguish a self-skip from a real pass "
|
||||
f"(see module docstring). SKIPPED (not green).\n")
|
||||
return 3
|
||||
|
||||
try:
|
||||
rows = build_rows(
|
||||
run_dir, args.tests, args.env,
|
||||
args.chainsaw_version or pinned_chainsaw_version(CONF_ROOT),
|
||||
args.kube_version or server_kube_version(),
|
||||
git_sha(args.repo))
|
||||
except ET.ParseError as e:
|
||||
sys.stderr.write(f"[record] {junit} is not parsable JUnit: {e}\n")
|
||||
return 1
|
||||
except OSError as e:
|
||||
sys.stderr.write(f"[record] cannot read the run inputs: {e}\n")
|
||||
return 1
|
||||
|
||||
if not rows:
|
||||
sys.stderr.write(f"[record] {junit} contains no <testcase> — nothing to record. "
|
||||
f"SKIPPED (not green).\n")
|
||||
return 3
|
||||
|
||||
for r in rows:
|
||||
print(f"[record] {r['layer']:4s} {r['proof']:28s} {r['test']} "
|
||||
f"({','.join(r['requirements']) or 'untagged'}) "
|
||||
f"junit={r['junit_result']} sentinel={r['sentinel'] or '—'}")
|
||||
counts: dict[str, int] = {}
|
||||
for r in rows:
|
||||
counts[r["proof"]] = counts.get(r["proof"], 0) + 1
|
||||
print(f"[record] {len(rows)} testcase(s): "
|
||||
+ ", ".join(f"{k}={v}" for k, v in sorted(counts.items())))
|
||||
|
||||
if args.dry_run:
|
||||
print("[record] --dry-run: ledger NOT written.")
|
||||
return 0
|
||||
try:
|
||||
append_rows(args.ledger, rows)
|
||||
except OSError as e:
|
||||
sys.stderr.write(f"[record] cannot append to {args.ledger}: {e}\n")
|
||||
return 1
|
||||
print(f"[record] appended {len(rows)} row(s) -> {args.ledger}")
|
||||
print("[record] COMMIT the run dir and the ledger; an uncommitted ledger proves nothing.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Loaded 100 of 433 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user